## Data Analytics

### Bootstrap Confidence Intervals in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### Bootstrap Confidence Intervals in Python

Here's a simple workflow, demonstration of bootstrap for modeling workflows. This should help you get started with this important data analytics method to evaluate and integrate uncertainty in any sample statistics or model.

#### Reporting Uncertainty and Significance

With confidence intervals and hypothesis testing we have the opportunity to report uncertainty and to report significance in our statistics. These methods are based on standard methods with their associated limitations and assumptions. For additional learning content to support learning about confidence intervals see [Confidence Intervals Lecture](https://youtu.be/oaXCcTWcU04) for more details and the [Confidence Intervals Interactive Demonstration](https://github.com/GeostatsGuy/PythonNumericalDemos/blob/master/Interactive_Confidence_Interval.ipynb) to play with and observe confidence intervals in real-time!

Also, for more information about statistical bootstrap see the [Bootstrap Lecture](https://youtu.be/oaXCcTWcU04) and [Bootstrap Workflow](https://git.io/fhgUW) for a general, empirical approach to assess uncertainty in statistics and models and [Bootstrap Interactive Demonstration](https://github.com/GeostatsGuy/PythonNumericalDemos/blob/master/Interactive_Bootstrap_Simple.ipynb).

This is a tutorial / demonstration of **Bootstrap Confidence Intervals in Python** for data analytics and data science workflows.  In Python, the SciPy package, specifically the [SciPy Stats Functions](https://docs.scipy.org/doc/scipy/reference/stats.html) provides excellent statistics tools.

#### Bootstrap

The uncertainty in an estimated population parameter from a sample, represented as a range, lower and upper bound, based on a specified probability interval known as the **confidence level**.

* one source of uncertainty is the paucity of data.
* do 200 or even less sample data provide a precise (and accurate estimate) of the mean? standard deviation? skew? 13th percentile / P13? 3rd central moment? experimental variogram? mutual information? Shannon entropy? etc.

Would it be useful to know the uncertainty due to limited sampling?
* what is the impact of uncertainty in the mean porosity e.g. 20%+/-2%?

**Bootstrap** is a method to assess the uncertainty in a sample statistic by repeated random sampling with replacement.

Assumptions
* sufficient, representative sampling, identical, idependent samples

Limitations
1. assumes the samples are representative
2. assumes stationarity
3. only accounts for uncertainty due to too few samples, e.g. no uncertainty due to changes away from data
4. does not account for boundary of area of interest
5. assumes the samples are independent
6. does not account for other local information sources

The Bootstrap Approach (Efron, 1982)

Statistical resampling procedure to calculate uncertainty in a calculated statistic from the data itself.
* Does this work?  Prove it to yourself, for uncertainty in the mean solution is standard error:

\begin{equation}
\sigma^2_\overline{x} = \frac{\sigma^2_s}{n}
\end{equation}

Extremely powerful - could calculate uncertainty in any statistic!  e.g. P13, skew etc.
* Would not be possible access general uncertainty in any statistic without bootstrap.
* Advanced forms account for spatial information and sampling strategy (game theory and Journel’s spatial bootstrap (1993).

Steps:

1. assemble a sample set, must be representative, reasonable to assume independence between samples

2. optional: build a cumulative distribution function (CDF)
    * may account for declustering weights, tail extrapolation
    * could use analogous data to support

3. For $\ell = 1, \ldots, L$ realizations, do the following:

    * For $i = \alpha, \ldots, n$ data, do the following:

        * Draw a random sample with replacement from the sample set or Monte Carlo simulate from the CDF (if available).

6. Calculate a realization of the sammary statistic of interest from the $n$ samples, e.g. $m^\ell$, $\sigma^2_{\ell}$. Return to 3 for another realization.

7. Compile and summarize the $L$ realizations of the statistic of interest.

* calculate and display the histogram or CDF to see the full distribution

* report the percentiles representing the lower and upper confidence intervals, e.g. for a confidence level of 95%, the 2.5 and 97.5 percentiles.

This is a very powerful method. Let's demonstrate a bunch of measures.

* when available, I compare to the distributions from the analytic expressions.

Let's demonstrate for a variety of parameters / statistics:

* mean / arithmetic average

* proportion

* interquartile range

* coefficient of variation

* correlation coefficient

#### Objective

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - sample_data_biased.csv at https://git.io/fh0CW

#### Load the required libraries

The following code loads the required libraries.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # plotting
import matplotlib                         # plotting
from matplotlib.ticker import (MultipleLocator, AutoMinorLocator) # control of axes ticks
plt.rc('axes', axisbelow=True)            # set axes and grids in the background for all plots
import seaborn as sns                     # plotting
from scipy import stats                   # summary statistics
import math                               # trig etc.
import scipy.signal as signal             # kernel for moving window calculation
import random                             # random sampling
from scipy.stats import gaussian_kde      # for PDF calculation
from scipy.stats import t                 # Student's t distribution for analytical solution
```

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
#os.chdir("c:/PGE383")                     # set the working directory
```

#### Set the Random Number Seed

For repeatability, set the random number seed.

* this ensures that the same random samples are drawn each run and everyone in the class will have the same results.

* change this seed number for a new set of random realizations

```python
seed = 73073
np.random.seed(seed = seed)
```

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object.

```python
#df = pd.read_csv('sample_data_biased.csv') # load our data table
df = pd.read_csv('https://raw.githubusercontent.com/GeostatsGuy/GeoDataSets/master/sample_data_biased.csv') # load the data from Dr. Pyrcz's GitHub repository
```

Let's drop some samples so that we increase the variations in bootstrap samples for our demonstration below.

```python
df = df.sample(frac = 0.2)                 # extract around 50 random samples to reduce the size of the dataset
print('Using ' + str(len(df)) + ' number of samples')
```

    Using 58 number of samples

Visualizing the DataFrame would be useful and we already learned about these methods in this [Working with Tabular Data Demo](https://git.io/fNgRW).

We can preview the DataFrame by printing a slice or by utilizing the 'head' DataFrame member function (with a nice and clean format, see below). With the slice we could look at any subset of the data table and with the head command, add parameter 'n=13' to see the first 13 rows of the dataset.

```python
df.head()                                  # display first 4 samples in the table as a preview
```

      X
      Y
      Facies
      Porosity
      Perm

      150
      20
      69
      0
      0.104484
      3.003225

      17
      400
      600
      1
      0.163054
      317.883581

      68
      640
      529
      1
      0.120547
      10.801778

      234
      490
      289
      1
      0.144525
      16.251107

      214
      470
      899
      1
      0.211944
      1302.937858

#### Summary Statistics for Tabular Data

The table includes X and Y coordinates (meters), Facies 1 and 0 (1 is sandstone and 0 interbedded sand and mudstone), Porosity (fraction), and permeability as Perm (mDarcy).

There are a lot of efficient methods to calculate summary statistics from tabular data in DataFrames. The describe command provides count, mean, minimum, maximum, and quartiles all in a nice data table. We use transpose just to flip the table so that features are on the rows and the statistics are on the columns.

```python
df.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      X
      58.0
      418.275862
      231.457153
      10.000000
      272.500000
      400.000000
      560.000000
      990.000000

      Y
      58.0
      493.224138
      287.393614
      39.000000
      279.000000
      499.500000
      676.500000
      989.000000

      Facies
      58.0
      0.810345
      0.395452
      0.000000
      1.000000
      1.000000
      1.000000
      1.000000

      Porosity
      58.0
      0.136501
      0.040473
      0.074349
      0.105662
      0.126192
      0.147067
      0.223661

      Perm
      58.0
      270.961923
      590.805185
      0.093252
      4.112400
      15.105651
      79.314434
      2372.383732

#### Visualizing Tabular Data with Location Maps

It is natural to set the x and y coordinate and feature ranges manually. e.g. do you want your color bar to go from 0.05887 to 0.24230 exactly? Also, let's pick a color map for display. I heard that plasma is known to be friendly to the color blind as the color and intensity vary together (hope I got that right, it was an interesting Twitter conversation started by Matt Hall from Agile if I recall correctly). We will assume a study area of 0 to 1,000m in x and y and omit any data outside this area.

```python
xmin = 0.0; xmax = 1000.0               # range of x values
ymin = 0.0; ymax = 1000.0               # range of y values
pormin = 0.05; pormax = 0.25;           # range of porosity values
permmin = 0.01; permmax = 10000
nx = 100; ny = 100; csize = 10.0
cmap = plt.cm.plasma                    # color map
cumul_prob = np.linspace(0.0,1.0,100)   # list of cumulative probabilities
```

Let's try out location maps, histograms and scatter plots.

```python
plt.subplot(231)
plt.scatter(df['X'],df['Y'],s = 40,c = df['Porosity'],cmap = plt.cm.inferno,linewidths = 0.3,edgecolor = 'black',alpha = 0.8,vmin = pormin,vmax = pormax)
plt.colorbar(); plt.xlabel('X (m)'); plt.ylabel('Y (m)'); plt.title('Porosity Location Map')

plt.subplot(234)
plt.hist(df['Porosity'],color = 'darkorange',alpha = 0.7,edgecolor='black',bins = np.linspace(pormin,pormax,int(len(df)/3)))
plt.xlabel('Porosity (fraction)'); plt.ylabel('Frequency'); plt.title('Porosity Histogram')
plt.grid(True, which='major',linewidth = 1.0); plt.grid(True, which='minor',linewidth = 0.2) # add y grids
plt.gca().tick_params(which='major',length=7); plt.gca().tick_params(which='minor', length=4)
plt.gca().xaxis.set_minor_locator(AutoMinorLocator()); plt.gca().yaxis.set_minor_locator(AutoMinorLocator()) # turn on minor ticks

plt.subplot(232)
plt.scatter(df['X'],df['Y'],s = 40,c = df['Perm'],cmap = plt.cm.inferno,linewidths = 0.3,edgecolor = 'black',alpha = 0.8,vmin = permmin,vmax = permmax,norm=matplotlib.colors.LogNorm())
plt.colorbar(); plt.xlabel('X (m)'); plt.ylabel('Y (m)'); plt.title('Permeability Location Map')

plt.subplot(235)
plt.hist(df['Perm'],color = 'darkorange',alpha = 0.7,edgecolor='black',bins=np.logspace(np.log10(permmin),np.log10(permmax),int(len(df)/3)))
#sns.kdeplot(x=df['Perm'],color = 'black',alpha = 0.2,levels = 1,log_scale = True,bw_adjust = 1)
plt.xlabel('Permeability (mD)'); plt.ylabel('Frequency'); plt.title('Permeability Histogram'); plt.xscale('log')
plt.grid(True, which='major',linewidth = 1.0); plt.grid(True, which='minor',linewidth = 0.2) # add y grids
plt.gca().tick_params(which='major',length=7); plt.gca().tick_params(which='minor', length=4)

plt.subplot(233)
plt.scatter(df['Porosity'],df['Perm'],s = 40,color = 'darkorange',alpha = 0.7,edgecolor='black')
#plt.contour(df['Porosity'],df['Perm'] Z, colors='black');
plt.ylabel('Permeability (mD)'); plt.xlabel('Porosity (fraction)'); plt.title('Permeability-Porosity Scatter Plot')
plt.yscale('log')
sns.kdeplot(x=df['Porosity'],y=df['Perm'],color = 'black',alpha = 0.2,levels = 4)

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=2.2, wspace=0.2, hspace=0.2)
plt.show()
```

### Bootstrap Method in Python

If you are new to bootstrap and Python, here's the most simple code possible for bootstrap.

* specify the number of bootstrap realizations, $L$
* declare a list to store the bootstrap realizations of the statistic of interest
* loop over L bootstrap realizations
    * n MCS, random samples with replacement for a new realization of the data
    * calculate the realization of the statistic from the realization of the data
* summarize the resulting uncertainty model, histogram, summary statistics etc.

#### Bootstrap of the Sample Mean, Arithmetic Average

Let's demonstrate bootstrap for uncertainty in the arithmetic average with a simple workflow.

```python
por_avg_real = []                      # declare an empty list to store the bootstrap realizations of the statistic
for k in range(0,L):                   # loop over the L bootstrap realizations
    samples = random.choices(df['Porosity'].values, k=len(df)) # n Monte Carlo simulations
    por_avg_real.append(np.average(samples)) # calculate the statistic of interest from the new bootstrap dataset
```

We will compare the bootstrap uncertainty in the sample arithmetic average distribution with the analytical expression:

\begin{equation}
\overline{x} \pm t_{\frac{\alpha}{2},n-1}\sqrt{\frac{s^2}{n}}
\end{equation}

The remaining code in this block is just a super cool set of plots with the results.

```python
# Bootstrap method for uncertainty in the sample mean
L = 1000                               # set the number of bootstrap realizations
por_avg_real = []                      # declare an empty list to store the bootstrap realizations of the statistic
for k in range(0,L):                   # loop over the L bootstrap realizations
    samples = random.choices(df['Porosity'].values, k=len(df)) # n Monte Carlo simulations
    por_avg_real.append(np.average(samples)) # calculate the statistic of interest from the new bootstrap dataset

# Analytical solution for the uncertainty in the sample mean, Student's t distribution with the correct mean and standard deviation
analytical = t.pdf(np.linspace(pormin,pormax,100), len(df)-1,loc=np.average(df['Porosity']),scale=math.sqrt(np.var(df['Porosity'].values)/len(df)))

# Plot of the original data and bootstap and analytical results
fig = plt.subplot(131)
plt.hist(df['Porosity'],color = 'darkorange',alpha = 0.8,edgecolor='black',linewidth=2,bins = np.linspace(pormin,pormax,int(len(df)/3)))
#plt.plot([np.average(df['Porosity']),np.average(df['Porosity'])],[0,100])
plt.axvline(x=np.average(df['Porosity']),linestyle="--",c='black')
plt.text(np.average(df['Porosity'])+0.005, 8.8, r'Average = ' + str(round(np.average(df['Porosity']),3)), fontsize=12)
plt.text(np.average(df['Porosity'])+0.005, 8.3, r'St.Dev. = ' + str(round(np.std(df['Porosity']),3)), fontsize=12)
plt.text(np.average(df['Porosity'])+0.005, 7.8, r'P10 = ' + str(round(np.percentile(df['Porosity'],10),3)), fontsize=12)
plt.text(np.average(df['Porosity'])+0.005, 7.3, r'P90 = ' + str(round(np.percentile(df['Porosity'],90),3)), fontsize=12)
plt.xlabel('Porosity (fraction)'); plt.ylabel('Frequency'); plt.title('Original Porosity Histogram')

plt.subplot(132)
plt.hist(por_avg_real,color = 'darkorange',alpha = 0.8,edgecolor = 'black',bins=np.linspace(pormin,pormax,100),label = 'bootstrap',density = True) # plot the distribution, could also calculate any summary statistics
plt.plot(np.linspace(pormin,pormax,100),analytical,color = 'black',linewidth=3,label = 'analytical',alpha=0.8)
plt.axvline(x=np.average(por_avg_real),linestyle="--",c='black')
plt.fill_between(np.linspace(pormin,pormax,100), 0, analytical, where = np.linspace(pormin,pormax,100) <= np.percentile(por_avg_real,10), facecolor='red', interpolate=True, alpha = 0.5)
plt.fill_between(np.linspace(pormin,pormax,100), 0, analytical, where = np.linspace(pormin,pormax,100) >= np.percentile(por_avg_real,90), facecolor='red', interpolate=True, alpha = 0.5)
plt.text(np.average(por_avg_real)+0.009, L*0.07, r'Average = ' + str(round(np.average(por_avg_real),3)), fontsize=12)
plt.text(np.average(por_avg_real)+0.009, L*0.066, r'St.Dev. = ' + str(round(np.std(por_avg_real),3)), fontsize=12)
plt.text(np.average(por_avg_real)+0.009, L*0.062, r'P90 = ' + str(round(np.percentile(por_avg_real,90),3)), fontsize=12)
plt.text(np.average(por_avg_real)+0.009, L*0.058, r'P10 = ' + str(round(np.percentile(por_avg_real,10),3)), fontsize=12)
plt.xlabel('Bootstrap Realizations and Analytical Uncertainty Distribution for Mean'); plt.ylabel('Frequency'); plt.title('Mean Bootstrap Realizations')
plt.legend(loc = 'upper left')

fig = plt.subplot(133)
plt.hist(df['Porosity'],color = 'darkorange',alpha = 0.8,edgecolor='black',linewidth=2,bins = np.linspace(pormin,pormax,int(len(df)/3)))
#plt.plot([np.average(df['Porosity']),np.average(df['Porosity'])],[0,100])
plt.axvline(x=np.average(df['Porosity']),c='black')
plt.axvline(x=np.percentile(por_avg_real,90),linestyle="--",c='black')
plt.axvline(x=np.percentile(por_avg_real,10),linestyle="--",c='black')
plt.text(np.percentile(por_avg_real,90)+0.009,8.8, r'Average = ' + str(round(np.average(por_avg_real),3)), fontsize=12)
plt.text(np.percentile(por_avg_real,90)+0.009,8.3, r'Average P90 = ' + str(round(np.percentile(por_avg_real,90),3)), fontsize=12)
plt.text(np.percentile(por_avg_real,90)+0.009,7.8, r'Average P10 = ' + str(round(np.percentile(por_avg_real,10),3)), fontsize=12)
plt.xlabel('Porosity (fraction)'); plt.ylabel('Frequency'); plt.title('Porosity Histogram with Uncertainty in Average')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

#### Bootstrap of the Proportion

Let's demonstrate another case for which we have the analytical solution.

We will compare the bootstrap results with the analytical expression:

\begin{equation}
\hat{p} \pm t_{\frac{\alpha}{2},n-1}\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}
\end{equation}

```python
L = 100000                               # set the number of bootstrap realizations
sand_prop_real = []                      # declare an empty list to store the bootstrap realizations of the statistic
shale_prop_real = []
for k in range(0,L):                     # loop over the L bootstrap realizations
    samples = random.choices(df['Facies'].values, k=len(df)) # n Monte Carlo simulations
    sand_prop_real.append(samples.count(1)/len(df)) # calculate the statistic of interest from the new bootstrap dataset
    shale_prop_real.append(samples.count(0)/len(df)) # calculate the statistic of interest from the new bootstrap dataset
sand_prop = np.sum(df['Facies'] == 1)/len(df); shale_prop = np.sum(df['Facies'] == 0)/len(df)

analytical_shale = t.pdf(np.linspace(0.0,1.0,100), len(df)-1,loc=shale_prop,scale=math.sqrt(shale_prop*(1.0-shale_prop)/len(df)))

fig = plt.subplot(121)
barlist = plt.bar(x=['Shale','Sand'],height = [1-sand_prop,sand_prop],color = 'gold',alpha = 0.8,edgecolor='black',linewidth=2)
barlist[0].set_color('brown'); barlist[0].set_edgecolor('black')
plt.text(-.3, shale_prop+0.02, r'Prop. Shale = ' + str(round(shale_prop,2)), fontsize=12)
plt.text(0.7, sand_prop+0.02, r'Prop. Sand = ' + str(round(sand_prop,2)), fontsize=12)
plt.xlabel('Rock Type / Facies'); plt.ylabel('Proportion'); plt.title('Sample Histogram')
plt.ylim([0,1]); plt.yticks(np.arange(0, 1.1, 0.1))

plt.subplot(122)
plt.hist(sand_prop_real,color = 'gold',alpha = 0.8,edgecolor = 'black',linewidth=2,bins=np.linspace(0.0,1.0,40),label = 'Sand',density = True) # plot the distribution, could also calculate any summary statistics
analytical = t.pdf(np.linspace(0.0,1.0,100), len(df)-1,loc=sand_prop,scale=math.sqrt(sand_prop*(1.0-sand_prop)/len(df)))
plt.plot(np.linspace(0.0,1.0,100),analytical,color = 'black',label = 'Analytical',alpha=0.8)
plt.axvline(x=sand_prop,linestyle="--",c='black')
plt.fill_between(np.linspace(0.0,1.0,100), 0, analytical, where = np.linspace(0.0,1.0,100) <= np.percentile(sand_prop_real,10), facecolor='red', interpolate=True, alpha = 0.8,label='Outside')
plt.fill_between(np.linspace(0.0,1.0,100), 0, analytical, where = np.linspace(0.0,1.0,100) >= np.percentile(sand_prop_real,90), facecolor='red', interpolate=True, alpha = 0.8)
plt.text(np.average(sand_prop_real)+0.04, 7.0, r'Prop. Exp = ' + str(round(sand_prop,2)), fontsize=8)
plt.text(np.average(sand_prop_real)+0.04, 6.4, r'Prop. P90 = ' + str(round(np.percentile(sand_prop_real,90),2)), fontsize=8)
plt.text(np.average(sand_prop_real)+0.04, 5.8, r'Prop. P10 = ' + str(round(np.percentile(sand_prop_real,10),2)), fontsize=8)

plt.hist(shale_prop_real,color = 'brown',alpha = 0.8,edgecolor = 'black',linewidth=2,bins=np.linspace(0.0,1.0,40),label = 'Shale',density = True) # plot the distribution, could also calculate any summary statistics
plt.plot(np.linspace(0.0,1.0,100),analytical_shale,color = 'black',alpha=0.8)
plt.axvline(x=shale_prop,linestyle="--",c='black')
plt.fill_between(np.linspace(0.0,1.0,100), 0, analytical_shale, where = np.linspace(0.0,1.0,100) <= np.percentile(shale_prop_real,10), facecolor='blue', interpolate=True, alpha = 0.8,label='Outside')
plt.fill_between(np.linspace(0.0,1.0,100), 0, analytical_shale, where = np.linspace(0.0,1.0,100) >= np.percentile(shale_prop_real,90), facecolor='blue', interpolate=True, alpha = 0.8)
plt.text(np.average(shale_prop_real)+0.04, 7.0, r'Prop. Exp = ' + str(round(shale_prop,2)), fontsize=8)
plt.text(np.average(shale_prop_real)+0.04, 6.4, r'Prop. P90 = ' + str(round(np.percentile(shale_prop_real,90),2)), fontsize=8)
plt.text(np.average(shale_prop_real)+0.04, 5.8, r'Prop. P10 = ' + str(round(np.percentile(shale_prop_real,10),2)), fontsize=8)

plt.xlabel('Rock Type / Facies Proportion (fraction)'); plt.ylabel('Frequency'); plt.title('Proportion Bootstrap Realizations and Analytical Uncertainty Distributions')
plt.legend(loc = 'upper center')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

#### Bootstrap of the Interquartile Range

To prove that we can bootstrap any statistic let's select a more complicated measure of dispersion, the interquartile range.

```python
L = 1000                               # set the number of bootstrap realizations
iqr_real = []                      # declare an empty list to store the bootstrap realizations of the statistic
for k in range(0,L):                   # loop over the L bootstrap realizations
    samples = random.choices(df['Porosity'].values, k=len(df)) # n Monte Carlo simulations
    iqr_real.append(np.percentile(samples,75) - np.percentile(samples,25)) # calculate the statistic of interest from the new bootstrap dataset

iqr = np.percentile(df['Porosity'],75) - np.percentile(df['Porosity'],25)

plt.subplot(111)
plt.hist(iqr_real,color = 'darkorange',alpha = 0.8,edgecolor = 'black',linewidth=2,bins=np.linspace(0.0,0.125,100),label = 'bootstrap',density = True) # plot the distribution, could also calculate any summary statistics
plt.axvline(x=iqr,c='black')
plt.axvline(x=np.percentile(iqr_real,10),linestyle="--",c='black')
plt.axvline(x=np.percentile(iqr_real,90),linestyle="--",c='black')
sns.kdeplot(x=iqr_real,color = 'black',alpha = 0.7,linewidth = 2,levels = 1,bw_adjust = 1)
plt.text(np.percentile(iqr_real,90)+0.009, L*0.04, r'Average = ' + str(round(np.average(iqr_real),3)), fontsize=12)
plt.text(np.percentile(iqr_real,90)+0.009, L*0.036, r'St.Dev. = ' + str(round(np.std(iqr_real),3)), fontsize=12)
plt.text(np.percentile(iqr_real,90)+0.009, L*0.032, r'P90 = ' + str(round(np.percentile(iqr_real,90),3)), fontsize=12)
plt.text(np.percentile(iqr_real,90)+0.009, L*0.028, r'P10 = ' + str(round(np.percentile(iqr_real,10),3)), fontsize=12)
plt.xlabel('Boostrap Realizations of Interquartile Range'); plt.ylabel('Frequency'); plt.title('Bootstrap Uncertainty Distribution of Interquartile Range')
plt.gca().grid(True, which='major',axis='both',linewidth = 1.0); plt.gca().grid(True, which='minor',axis='x',linewidth = 0.2) # add y grids
plt.gca().tick_params(which='major',length=7); plt.gca().tick_params(which='minor', length=4)
plt.gca().xaxis.set_minor_locator(AutoMinorLocator()); plt.gca().yaxis.set_minor_locator(AutoMinorLocator()) # turn on minor ticks

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

#### Bootstrap of the Coefficient of Variation

Here's another statistic that requires multiple measures from the each bootstrap realization of the data.

* this reinforces that we bootstrap dataset realizations and then calculate the statistic on this dataset realization

For the coefficient of variation we will:

* calculate a bootstrap realization of the dataset with $n$ samples with replacement
* calculate the mean and standard deviation from this bootstrapped realization of the dataset
* calculate a boostrap realization of the coefficient of variation as the standard deviation divided by the mean

Repeat this $L$ times and then evaluate the resulting distribution.

```python
L = 1000                               # set the number of bootstrap realizations
cv_real = []                      # declare an empty list to store the bootstrap realizations of the statistic

for k in range(0,L):                   # loop over the L bootstrap realizations
    samples = random.choices(df['Porosity'].values, k=len(df)) # n Monte Carlo simulations
    cv_real.append(np.std(samples)/np.average(samples)) # calculate the statistic of interest from the new bootstrap dataset

cv = np.std(df['Porosity'])/np.average(df['Porosity'])

plt.subplot(111)
plt.hist(cv_real,color = 'darkorange',alpha = 0.8,edgecolor = 'black',linewidth=2,bins=np.linspace(0.1,0.4,100),label = 'bootstrap',density = True) # plot the distribution, could also calculate any summary statistics
plt.axvline(x=cv,c='black')
plt.axvline(x=np.percentile(cv_real,10),linestyle="--",c='black')
plt.axvline(x=np.percentile(cv_real,90),linestyle="--",c='black')
sns.kdeplot(x=cv_real,color = 'black',alpha = 0.8,linewidth=2,levels = 1,bw_adjust = 1)
plt.text(np.percentile(cv_real,90)+0.009, 16, r'Average = ' + str(round(np.average(cv_real),3)), fontsize=12)
plt.text(np.percentile(cv_real,90)+0.009, 15, r'St.Dev. = ' + str(round(np.std(cv_real),3)), fontsize=12)
plt.text(np.percentile(cv_real,90)+0.009, 14, r'P90 = ' + str(round(np.percentile(cv_real,90),3)), fontsize=12)
plt.text(np.percentile(cv_real,90)+0.009, 13, r'P10 = ' + str(round(np.percentile(cv_real,10),3)), fontsize=12)
plt.xlabel('Boostrap Realizations of Coefficient of Variation'); plt.ylabel('Frequency'); plt.title('Bootstrap Uncertainty Distribution of Coefficient of Variation')
plt.gca().grid(True, which='major',axis='both',linewidth = 1.0); plt.gca().grid(True, which='minor',axis='x',linewidth = 0.2) # add y grids
plt.gca().tick_params(which='major',length=7); plt.gca().tick_params(which='minor', length=4)
plt.gca().xaxis.set_minor_locator(AutoMinorLocator()); plt.gca().yaxis.set_minor_locator(AutoMinorLocator()) # turn on minor ticks

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

#### Bootstrap of the Correlation Coefficient

Here's a statistic that requires us to work with multiple, paired features at once.
* this reinforces that we boostrap for a new realization of the dataset, a set of samples with all their features.

For the correlation coefficient we will:

* calculate a bootstrap realization of the dataset with $n$ samples with replacement as a new DataFrame with all features
* calculate the the correlation coefficient between 2 paired features

Repeat this $L$ times and then evaluate the resulting distribution.

```python
L = 1000                               # set the number of bootstrap realizations
corr_real = []                      # declare an empty list to store the bootstrap realizations of the statistic

for k in range(0,L):                   # loop over the L bootstrap realizations
    samples = df.sample(n=len(df),replace=True,random_state = seed + k) # n random samples with replacement as a new DataFrame
    corr_real.append(samples[['Porosity','Perm']].corr()['Porosity'][1]) # calculate the statistic of interest from the new bootstrap dataset

corr = df[['Porosity','Perm']].corr()['Porosity'][1]

plt.subplot(111)
plt.hist(corr_real,color = 'darkorange',alpha = 0.8,edgecolor = 'black',linewidth=2,bins=np.linspace(0.5,1.0,100),label = 'bootstrap',density = True) # plot the distribution, could also calculate any summary statistics
plt.axvline(x=corr,c='black')
plt.axvline(x=np.percentile(corr_real,10),linestyle="--",c='black')
plt.axvline(x=np.percentile(corr_real,90),linestyle="--",c='black')
sns.kdeplot(x=corr_real,color = 'black',linewidth=2,alpha = 0.8,levels = 1,bw_adjust = 1)
# plt.text(np.percentile(corr_real,90)+0.009, 16, r'Average = ' + str(round(np.average(corr_real),3)), fontsize=12)
# plt.text(np.percentile(corr_real,90)+0.009, 15, r'St.Dev. = ' + str(round(np.std(corr_real),3)), fontsize=12)
# plt.text(np.percentile(corr_real,90)+0.009, 14, r'P90 = ' + str(round(np.percentile(corr_real,90),3)), fontsize=12)
# plt.text(np.percentile(corr_real,90)+0.009, 13, r'P10 = ' + str(round(np.percentile(corr_real,10),3)), fontsize=12)
plt.xlabel('Boostrap Realizations of Correlation Coefficient'); plt.ylabel('Frequency'); plt.title('Bootstrap Uncertainty Distribution of Correlation Coefficient')
plt.gca().grid(True, which='major',axis='both',linewidth = 1.0); plt.gca().grid(True, which='minor',axis='x',linewidth = 0.2) # add y grids
plt.gca().tick_params(which='major',length=7); plt.gca().tick_params(which='minor', length=4)
plt.gca().xaxis.set_minor_locator(AutoMinorLocator()); plt.gca().yaxis.set_minor_locator(AutoMinorLocator()) # turn on minor ticks

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

#### Comments

This was a basic demonstration of bootstrap.

* you can use bootstrap to calculate uncertainty in any statistic!

* you are calculating a set of realizations of the statistic, representing uncertainty due to small sample size

* note the assumptions of bootstrap, including stationarity and representativity

* remember, get a dataset realization by bootstrap and then calculate the realization of the statistic from the dataset realization

* if your statistic has multiple inputs (e.g. P25 and P75), calculate each from the same bootstrap realization of the dataset.

I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)


```python
from IPython.core.display import HTML
HTML("<style>.container { width:95% !important; }</style>")
```

# Lecture 5: Example of using available software: scipy.optimize

# Optimization software
* When we want to optimize something, we do not of course need to start everything from scratch. It is good to know how algorithms work, but if the development of new algorithms is not the main point, then one can just use packages and libraries that have been premade.
* First, we have a look at some of the available software and, then, we have a closer look at scipy.optimize

## Have you used any optimization software before? Please share your experiences.

## Wolfram Alpha
* Free web version of Mathematica
* http://www.wolframalpha.com/
* Can perform either symbolic or numerical calculations
* Includes also some basic optimization

## Rosenbrock function
A non-convex function
$$
f(x) = (1-x_1)^2 +100(x_2-x_1^2)^2
$$
that has a global minimum in $x^*=(1,1)^T$ where $f(x^*)=0$. The minimum is located in a narrow, banana-shaped valley.

The coefficient of the second term can be adjusted but it does not affect the position of the global minimum. The Rosenbrock function is used to test optimization algorithms.

```python
def f_rosenb(x):
    return (1.0 -x[0])**2 + 100*(x[1] - x[0]**2)**2
```

## Matlab - Optimization toolbox
* Interactive environment for numerical computing
* Subroutines for unconstrained optimization:
  * fminbnd: find minimum of single-variable function on fixed interval
  * fminsearch: find minimum of unconstrained multivariable function using derivative-free method
  * fminunc: find minimum of unconstrained multivariable function using gradient-based method
* https://se.mathworks.com/products/optimization.html?s_tid=srchtitle_optimization_1
* Matlab codes for the subroutines can be found in the directory where Matlab is installed:
 ..\MATLAB\R2013a\toolbox\optim\optim\
* You can also use Octave (https://www.gnu.org/software/octave/) which is an open source software having compatibility with many Matlab scripts

# Optimization with scipy.optimize
In Python, there are multiple packages for optimization. At this lecture, we are going to take a look at *scipy.optimize* package.

## Starting up

When we want to study a package in Python, we can import it..

```python
from scipy.optimize import minimize
```

If we want to see the documentation, we can write the name of the package and two question marks and hit enter:

```python
minimize??
```

or

https://docs.scipy.org/doc/scipy/tutorial/optimize.html

## Optimization of multiple variables

Let us define again our friendly objective function:

```python
def f_simple(x):
    return (x[0] - 10.0)**2 + (x[1] + 5.0)**2+x[0]**2
```

### Method: `Nelder-Mead'

* Uses a simplex which is a shape consisting $n+1$ vertices in n dimensional space.

* E.g., a triangle in 2D, a tetrahedron in 3D, 5-cell pentachoron (https://en.wikipedia.org/wiki/5-cell). etc.

* Begin with $n+1$ arbitrarily selected points in nD (3 in 2D) to form the simplex.
* Follow the following steps in each iteration:
    * Sort the selected points ($f1<f2<f3$)
    * Reflect the worst point ($x_r$)
    * Extend ($x_e$)
    * Contract
    * Shrink
    * Check convergency

The documentation has the following to say:

    Method :ref:`Nelder-Mead <optimize.minimize-neldermead>` uses the
    Simplex algorithm [1]_, [2]_. This algorithm has been successful
    in many applications but other algorithms using the first and/or
    second derivatives information might be preferred for their better
    performances and robustness in general.
...
     References
    ----------
    .. [1] Nelder, J A, and R Mead. 1965. A Simplex Method for Function
        Minimization. The Computer Journal 7: 308-13.
    .. [2] Wright M H. 1996. Direct search methods: Once scorned, now
        respectable, in Numerical Analysis 1995: Proceedings of the 1995
        Dundee Biennial Conference in Numerical Analysis (Eds. D F
        Griffiths and G A Watson). Addison Wesley Longman, Harlow, UK.
        191-208.

```python
res = minimize(f_simple,[0,0],method='Nelder-Mead',
         options={'disp': True})
print(res.x)
res = minimize(f_simple,[0,0],method='Powell',
         options={'disp': True})
print(res.x)
```

    Optimization terminated successfully.
             Current function value: 50.000000
             Iterations: 99
             Function evaluations: 189
    [ 5.00003542 -4.99997315]
    Optimization terminated successfully.
             Current function value: 50.000000
             Iterations: 2
             Function evaluations: 99
    [ 5.         -5.00000004]

```python
print(type(res))
print(res)
print(res.message)
```

    <class 'scipy.optimize.optimize.OptimizeResult'>
       direc: array([[1., 0.],
           [0., 1.]])
         fun: 50.0
     message: 'Optimization terminated successfully.'
        nfev: 99
         nit: 2
      status: 0
     success: True
           x: array([ 5.        , -5.00000004])
    Optimization terminated successfully.

```python
res = minimize(f_rosenb,[-2.0,-10],method='Nelder-Mead',
         options={'disp': True})
```

    Optimization terminated successfully.
             Current function value: 0.000000
             Iterations: 155
             Function evaluations: 288

```python
print(res)
```

     final_simplex: (array([[1.00000755, 1.00001315],
           [1.00001942, 1.00004104],
           [0.99996779, 0.99993545]]), array([4.36964903e-10, 8.57224979e-10, 1.03919729e-09]))
               fun: 4.369649026757306e-10
           message: 'Optimization terminated successfully.'
              nfev: 288
               nit: 155
            status: 0
           success: True
                 x: array([1.00000755, 1.00001315])

### Method: Conjugate Gradient (`CG`)
* Idea is to improve convergence properties of steepest descent
* A search direction is a combination of the current search direction and a previous search direction
* Steps in directions that are **orthogonal**
* Memory consumption 𝑂(𝑛)
* Best suited for large scale problems

The documentation has the following to say:

    Method :ref:`CG <optimize.minimize-cg>` uses a nonlinear conjugate
    gradient algorithm by Polak and Ribiere, a variant of the
    Fletcher-Reeves method described in [5]_ pp.  120-122. Only the
    first derivatives are used.
...
   References
    ----------
...
    .. [5] Nocedal, J, and S J Wright. 2006. Numerical Optimization.
       Springer New York.

The Conjugate gradient method needs the gradient. The documentation has the following to say

    jac : bool or callable, optional
        Jacobian (gradient) of objective function. Only for CG, BFGS,
        Newton-CG, L-BFGS-B, TNC, SLSQP, dogleg, trust-ncg.
        If `jac` is a Boolean and is True, `fun` is assumed to return the
        gradient along with the objective function. If False, the
        gradient will be estimated numerically.
        `jac` can also be a callable returning the gradient of the
        objective. In this case, it must accept the same arguments as `fun`.

### Estimating the gradient numerically:

```python
res = minimize(f_simple, [0,0], method='CG', #Conjugate gradient method
               options={'disp': True})
print(res.x)
```

    Optimization terminated successfully.
             Current function value: 50.000000
             Iterations: 3
             Function evaluations: 21
             Gradient evaluations: 7
    [ 4.99999999 -5.00000006]

### Giving the gradient with ad

```python
import ad
res = minimize(f_simple, [0,0], method='CG', #Conjugate gradient method
               options={'disp': True}, jac=ad.gh(f_simple)[0])
print(res.x)
```

    Optimization terminated successfully.
             Current function value: 50.000000
             Iterations: 3
             Function evaluations: 7
             Gradient evaluations: 7
    [ 5. -5.]

### Newton-Conjugate-Gradient algorithm (method=`Newton-CG`)

Newton-CG method uses a Newton-CG algorithm [5] pp. 168 (also known as the truncated Newton method). It uses a CG method to the compute the search direction. See also *TNC* method for a box-constrained minimization with a similar algorithm.

   References
    ----------
    .. [5] Nocedal, J, and S J Wright. 2006. Numerical Optimization.
       Springer New York.

The Newton-CG algorithm needs the Jacobian and the Hessian. The documentation has the following to say:

    hess, hessp : callable, optional
        Hessian (matrix of second-order derivatives) of objective function or
        Hessian of objective function times an arbitrary vector p.  Only for
        Newton-CG, dogleg, trust-ncg.
        Only one of `hessp` or `hess` needs to be given.  If `hess` is
        provided, then `hessp` will be ignored.  If neither `hess` nor
        `hessp` is provided, then the Hessian product will be approximated
        using finite differences on `jac`. `hessp` must compute the Hessian
        times an arbitrary vector.

### Trying without reading the documentation

```python
import numpy as np
res = minimize(f_simple, [0,0], method='Newton-CG', #Newton-CG method
               options={'disp': True})
print(res.x)
```

### Giving the gradient

```python
import ad
res = minimize(f_simple, [0,0], method='Newton-CG', #Newton-CG method
               options={'disp': True},jac=ad.gh(f_simple)[0])
print(res.x)
```

    Optimization terminated successfully.
             Current function value: 50.000000
             Iterations: 7
             Function evaluations: 7
             Gradient evaluations: 15
             Hessian evaluations: 0
    [ 4.99999999 -4.99999999]

### Giving also the hessian

```python
import ad
res = minimize(f_simple, [0,0], method='Newton-CG', #Newton-CG method
               options={'disp': True},jac=ad.gh(f_simple)[0],
               hess=ad.gh(f_simple)[1])
print(res.x)
```

    Optimization terminated successfully.
             Current function value: 50.000000
             Iterations: 7
             Function evaluations: 7
             Gradient evaluations: 7
             Hessian evaluations: 7
    [ 5. -5.]

## Another example
$$
\begin{align}
\min \quad & (x_1-2)^4+(x_1 - 2x_2)^2\\
\text{s.t.}\quad &x_1,x_2\in\mathbb R
\end{align}
$$
Optimal solution clearly is $x^*=(2,1)^T$

```python
def f_simple2(x):
    return (x[0] - 2.0)**4 + (x[0] - 2.0*x[1])**2
```

```python
import numpy as np
x0 = np.array([-96,-2000])
res = minimize(f_simple2,x0,method='Nelder-Mead',
         options={'disp': True})
print(res.x)
res = minimize(f_simple2,x0,method='Powell',
         options={'disp': True})
print(res.x)
res = minimize(f_simple2, x0, method='CG', #Conjugate gradient method
               options={'disp': True}, jac=ad.gh(f_simple2)[0])
print(res.x)
res = minimize(f_simple2, x0, method='Newton-CG', #Newton-CG method
               options={'disp': True},jac=ad.gh(f_simple2)[0],
               hess=ad.gh(f_simple2)[1])
print(res.x)
```

    Optimization terminated successfully.
             Current function value: 0.000000
             Iterations: 79
             Function evaluations: 148
    [2.0000004  1.00000021]
    Optimization terminated successfully.
             Current function value: 0.000000
             Iterations: 29
             Function evaluations: 784
    [1.99999649 0.99999824]
    Optimization terminated successfully.
             Current function value: 0.000000
             Iterations: 14
             Function evaluations: 32
             Gradient evaluations: 32
    [2.00005457 1.00002728]
    Optimization terminated successfully.
             Current function value: 0.000000
             Iterations: 28
             Function evaluations: 30
             Gradient evaluations: 30
             Hessian evaluations: 28
    [1.99649493 0.99824742]

```python
# Add here optimization of the Rosenbrock function with gradient based algorithms
```

```python
x0 = np.array([-2.0,-10])
res = minimize(f_rosenb, x0, method='CG', #Conjugate gradient method
               options={'disp': True}, jac=ad.gh(f_rosenb)[0])
print(res.x)
res = minimize(f_rosenb, x0, method='Newton-CG', #Newton-CG method
               options={'disp': True},jac=ad.gh(f_rosenb)[0],
               hess=ad.gh(f_rosenb)[1])
print(res.x)
```

    Optimization terminated successfully.
             Current function value: 0.000000
             Iterations: 30
             Function evaluations: 71
             Gradient evaluations: 71
    [0.99999053 0.99998103]
    Optimization terminated successfully.
             Current function value: 0.000000
             Iterations: 31
             Function evaluations: 36
             Gradient evaluations: 36
             Hessian evaluations: 31
    [0.99999884 0.99999768]

## Line search

```python
def f_singlevar(x):
    return 2+(1-x)**2
```

```python
from scipy.optimize import minimize_scalar
minimize_scalar??
```

### Method: `Golden`

The documentation has the following to say:

    Method :ref:`Golden <optimize.minimize_scalar-golden>` uses the
    golden section search technique. It uses analog of the bisection
    method to decrease the bracketed interval. It is usually
    preferable to use the *Brent* method.

```python
minimize_scalar(f_singlevar,method='golden',tol=0.00001)
```

         fun: 2.0
        nfev: 30
         nit: 25
     success: True
           x: 1.0

### Method: `Brent`

The documentation has the following to say about the Brent method:

    Method *Brent* uses Brent's algorithm to find a local minimum.
    The algorithm uses inverse parabolic interpolation when possible to
    speed up convergence of the golden section method.

```python
minimize_scalar(f_singlevar,method='brent')
```

         fun: 2.0
        nfev: 8
         nit: 4
     success: True
           x: 0.99999998519


# Paralelismo en Julia

## 1. Introducción

Julia proporciona distintos mecanismos para paralelizar código. Algunas de las estrategias y desafíos para escribir algoritmos paralelos son los siguientes:

* Estrategias de paralelismo
     * SIMD
     * Multi-hilo
     * Tareas
     * Multiproceso
         * Memoria compartida
         * Memoria distribuida
     * Programación de GPU

* Desafíos de la computación paralela
     * Orden de ejecución
         * ejecución de fuera de orden de posibilidad
         * acceso y mutación simultáneos
     * Acceso y movimiento de datos
     * Código de acceso y movimiento
     * Adaptación adecuada de la estrategia de paralelismo a las capacidades de su máquina
     * Hacer coincidir adecuadamente la estrategia de paralelismo con el problema en cuestión

## ¿Qué es lo que está sucediendo con nuestras computadoras?

## Lo difícil de la computación paralela
   * No pensamos en paralelo
   * Aprendemos a escribir y razonar sobre programas en serie.
   * El deseo de paralelismo a menudo surge _después_ de haber escrito su algoritmo (¡y lo encontró demasiado lento!)

## Resumen:
   * Las arquitecturas computacionales actuales nos empujan hacia la programación paralela para un rendimiento máximo, ¡incluso si no estamos en un clúster!
   * Pero es difícil diseñar buenos algoritmos paralelos
   * Es difícil de expresar y razonar sobre esos algoritmos.

## 2. SIMD: El paralelismo que puede (a veces) suceder automáticamente

SIMD: Instrucción única, datos múltiples (Single Instruction Multiple Data)

**Nota:** También llamado confusamente vectorización

### Arquitectura

En lugar de calcular cuatro sumas secuencialmente:

\begin{align}
x_1 + y_1 &\rightarrow z_1 \\
x_2 + y_2 &\rightarrow z_2 \\
x_3 + y_3 &\rightarrow z_3 \\
x_4 + y_4 &\rightarrow z_4
\end{align}

Procesadores modernos tienen unidades de procesamiento vectorial que pueden hacer lo anterior a la vez:

$$
\left(\begin{array}{cc}
x_1 \\
x_2 \\
x_3 \\
x_4
\end{array}\right)
+
\left(\begin{array}{cc}
y_1 \\
y_2 \\
y_3 \\
y_4
\end{array}\right)
\rightarrow
\left(\begin{array}{cc}
z_1 \\
z_2 \\
z_3 \\
z_4
\end{array}\right)
$$

### ¿Cómo se logra?

```julia
using BenchmarkTools
```

```julia
A = rand(100_000)
function simplesum(A)
    result = zero(eltype(A))
    for i in eachindex(A)
        @inbounds result += A[i]
    end
    return result
end

simplesum(A)
```

    50154.62310354247

Como muchos lenguajes de programación modernos, Julia utiliza la verificación de límites ([_boundchecking_](https://docs.julialang.org/en/v1/devdocs/boundscheck/)) para garantizar la seguridad del programa al acceder a arreglos.
En bucles internos u otras situaciones críticas de rendimiento, es posible que se desee omitir estas comprobaciones de límites para mejorar el rendimiento en tiempo de ejecución.

En consecuencia, Julia incluye la macro `@inbounds(...)` para decirle al compilador que omita dichas comprobaciones de límites dentro del bloque dado.

```julia
@btime simplesum($A)
```

      143.232 μs (0 allocations: 0 bytes)

    50154.62310354247

¿ese tiempo es bueno?

```julia
@btime sum($A)
```

      17.611 μs (0 allocations: 0 bytes)

    50154.62310354205

Diseñamos una función más lenta que la suma prediseñada `sum()`, ¡y también estamos obteniendo una respuesta diferente! Veamos qué sucede con un flotante de 32 bits en lugar de uno de 64 bits. Cada elemento tiene la mitad del número de bits, por lo que también permite duplicar la longitud (para que el número total de bits procesados permanezca constante).

```julia
A32 = rand(Float32, length(A)*2)
@btime simplesum($A32)
@btime sum($A32)
```

      286.429 μs (0 allocations: 0 bytes)
      19.587 μs (0 allocations: 0 bytes)

    99875.3f0

¡Eso es aun peor! ¿Que está pasando aqui?
Estamos viendo múltiples diferencias en el desempeño: ¿quizás la suma incorporada de Julia está usando algún paralelismo?

Intentemos usar SIMD nosotros mismos:

```julia
function simdsum(A)
    result = zero(eltype(A))
    @simd for i in eachindex(A)
        @inbounds result += A[i]
    end
    return result
end
@btime simdsum($A)
@btime simdsum($A32)
```

      16.293 μs (0 allocations: 0 bytes)
      16.354 μs (0 allocations: 0 bytes)

    99875.305f0

¿Qué hizo y por qué no siempre usamos (usa Julia pues) `@simd` para cada bucle **for** automáticamente?

Veamos los resultados:

```julia
simplesum(A), simdsum(A), sum(A)
```

    (50008.443227500014, 50008.44322750009, 50008.443227500095)

```julia
simplesum(A32), simdsum(A32), sum(A32)
```

    (99940.69f0, 99940.2f0, 99940.19f0)

¿Por qué no son iguales?

Sin `@simd`, Julia está haciendo _exactamente_ lo que le dijimos que hiciera: está tomando cada elemento de nuestro arreglo y lo agrega a una gran pila secuencialmente. Nuestra respuesta es más pequeña de lo que la "suma" incorporada de Julia cree que es: eso es porque, como la pila se hace más grande, comenzamos a perder las partes inferiores de cada elemento que estamos sumando, ¡y esas pequeñas pérdidas comienzan a acumularse!

La macro `@simd` le dice a Julia que puede reorganizar las adiciones de punto flotante -
incluso si cambiara la respuesta. Dependiendo de su CPU, esto puede llevar a 2x o 4x
o incluso un paralelismo 8x. Básicamente, Julia está calculando sumas independientes para
los índices pares y los índices impares simultáneamente:

\begin{align}
odds &\leftarrow 0 \\
evens &\leftarrow 0 \\
\text{loop}&\ \text{odd}\ i: \\
    &\left(\begin{array}{cc}
odds \\
evens
\end{array}\right)
\leftarrow
\left(\begin{array}{cc}
odds \\
evens
\end{array}\right)
+
\left(\begin{array}{cc}
x_{i} \\
x_{i+1}
\end{array}\right) \\
total &\leftarrow evens + odds
\end{align}

En muchos casos, Julia puede y sabe que un bucle for puede ser vectorizado (SIMD-ed) y aprovechará esto por defecto.

```julia
B = rand(1:10, 100_000)
@btime simplesum($B)
@btime sum($B)
B32 = rand(Int32(1):Int32(10), length(B)*2)
@btime simplesum($B32)
@btime simdsum($B32)
```

      16.291 μs (0 allocations: 0 bytes)
      17.251 μs (0 allocations: 0 bytes)
      16.338 μs (0 allocations: 0 bytes)
      16.333 μs (0 allocations: 0 bytes)

    1101303

¿Cómo inspeccionamos que se está vectorizando?

```julia
@code_llvm simdsum(A32)
```

    ;  @ In[25]:2 within `simdsum'
    define float @julia_simdsum_18131(%jl_value_t addrspace(10)* nonnull align 16 dereferenceable(40)) {
    top:
    ;  @ In[25]:3 within `simdsum'
    ; ┌ @ simdloop.jl:69 within `macro expansion'
    ; │┌ @ abstractarray.jl:212 within `eachindex'
    ; ││┌ @ abstractarray.jl:95 within `axes1'
    ; │││┌ @ abstractarray.jl:75 within `axes'
    ; ││││┌ @ array.jl:155 within `size'
           %1 = addrspacecast %jl_value_t addrspace(10)* %0 to %jl_value_t addrspace(11)*
           %2 = bitcast %jl_value_t addrspace(11)* %1 to %jl_value_t addrspace(10)* addrspace(11)*
           %3 = getelementptr inbounds %jl_value_t addrspace(10)*, %jl_value_t addrspace(10)* addrspace(11)* %2, i64 3
           %4 = bitcast %jl_value_t addrspace(10)* addrspace(11)* %3 to i64 addrspace(11)*
           %5 = load i64, i64 addrspace(11)* %4, align 8
    ; ││││└
    ; ││││┌ @ tuple.jl:157 within `map'
    ; │││││┌ @ range.jl:320 within `OneTo' @ range.jl:311
    ; ││││││┌ @ promotion.jl:409 within `max'
             %6 = icmp sgt i64 %5, 0
             %7 = select i1 %6, i64 %5, i64 0
    ; │└└└└└└
    ; │ @ simdloop.jl:72 within `macro expansion'
       br i1 %6, label %L13.lr.ph, label %L30

    L13.lr.ph:                                        ; preds = %top
       %8 = bitcast %jl_value_t addrspace(11)* %1 to float addrspace(13)* addrspace(11)*
       %9 = load float addrspace(13)*, float addrspace(13)* addrspace(11)* %8, align 8
    ; │ @ simdloop.jl:75 within `macro expansion'
       %min.iters.check = icmp ult i64 %7, 32
       br i1 %min.iters.check, label %scalar.ph, label %vector.ph

    vector.ph:                                        ; preds = %L13.lr.ph
       %n.vec = and i64 %7, 9223372036854775776
       br label %vector.body

    vector.body:                                      ; preds = %vector.body, %vector.ph
    ; │ @ simdloop.jl:78 within `macro expansion'
    ; │┌ @ int.jl:53 within `+'
        %index = phi i64 [ 0, %vector.ph ], [ %index.next, %vector.body ]
        %vec.phi = phi <8 x float> [ zeroinitializer, %vector.ph ], [ %18, %vector.body ]
        %vec.phi10 = phi <8 x float> [ zeroinitializer, %vector.ph ], [ %19, %vector.body ]
        %vec.phi11 = phi <8 x float> [ zeroinitializer, %vector.ph ], [ %20, %vector.body ]
        %vec.phi12 = phi <8 x float> [ zeroinitializer, %vector.ph ], [ %21, %vector.body ]
    ; │└
    ; │ @ simdloop.jl:77 within `macro expansion' @ In[25]:4
    ; │┌ @ array.jl:788 within `getindex'
        %10 = getelementptr inbounds float, float addrspace(13)* %9, i64 %index
        %11 = bitcast float addrspace(13)* %10 to <8 x float> addrspace(13)*
        %wide.load = load <8 x float>, <8 x float> addrspace(13)* %11, align 4
        %12 = getelementptr inbounds float, float addrspace(13)* %10, i64 8
        %13 = bitcast float addrspace(13)* %12 to <8 x float> addrspace(13)*
        %wide.load13 = load <8 x float>, <8 x float> addrspace(13)* %13, align 4
        %14 = getelementptr inbounds float, float addrspace(13)* %10, i64 16
        %15 = bitcast float addrspace(13)* %14 to <8 x float> addrspace(13)*
        %wide.load14 = load <8 x float>, <8 x float> addrspace(13)* %15, align 4
        %16 = getelementptr inbounds float, float addrspace(13)* %10, i64 24
        %17 = bitcast float addrspace(13)* %16 to <8 x float> addrspace(13)*
        %wide.load15 = load <8 x float>, <8 x float> addrspace(13)* %17, align 4
    ; │└
    ; │┌ @ float.jl:400 within `+'
        %18 = fadd fast <8 x float> %vec.phi, %wide.load
        %19 = fadd fast <8 x float> %vec.phi10, %wide.load13
        %20 = fadd fast <8 x float> %vec.phi11, %wide.load14
        %21 = fadd fast <8 x float> %vec.phi12, %wide.load15
    ; │└
    ; │ @ simdloop.jl:78 within `macro expansion'
    ; │┌ @ int.jl:53 within `+'
        %index.next = add i64 %index, 32
        %22 = icmp eq i64 %index.next, %n.vec
        br i1 %22, label %middle.block, label %vector.body

    middle.block:                                     ; preds = %vector.body
    ; │└
    ; │ @ simdloop.jl:77 within `macro expansion' @ In[25]:4
    ; │┌ @ float.jl:400 within `+'
        %bin.rdx = fadd fast <8 x float> %19, %18
        %bin.rdx16 = fadd fast <8 x float> %20, %bin.rdx
        %bin.rdx17 = fadd fast <8 x float> %21, %bin.rdx16
        %rdx.shuf = shufflevector <8 x float> %bin.rdx17, <8 x float> undef, <8 x i32> <i32 4, i32 5, i32 6, i32 7, i32 undef, i32 undef, i32 undef, i32 undef>
        %bin.rdx18 = fadd fast <8 x float> %bin.rdx17, %rdx.shuf
        %rdx.shuf19 = shufflevector <8 x float> %bin.rdx18, <8 x float> undef, <8 x i32> <i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
        %bin.rdx20 = fadd fast <8 x float> %bin.rdx18, %rdx.shuf19
        %rdx.shuf21 = shufflevector <8 x float> %bin.rdx20, <8 x float> undef, <8 x i32> <i32 1, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
        %bin.rdx22 = fadd fast <8 x float> %bin.rdx20, %rdx.shuf21
        %23 = extractelement <8 x float> %bin.rdx22, i32 0
        %cmp.n = icmp eq i64 %7, %n.vec
    ; │└
    ; │ @ simdloop.jl:75 within `macro expansion'
       br i1 %cmp.n, label %L30, label %scalar.ph

    scalar.ph:                                        ; preds = %middle.block, %L13.lr.ph
       %bc.resume.val = phi i64 [ %n.vec, %middle.block ], [ 0, %L13.lr.ph ]
       %bc.merge.rdx = phi float [ %23, %middle.block ], [ 0.000000e+00, %L13.lr.ph ]
       br label %L13

    L13:                                              ; preds = %scalar.ph, %L13
       %value_phi16 = phi i64 [ %bc.resume.val, %scalar.ph ], [ %27, %L13 ]
       %value_phi5 = phi float [ %bc.merge.rdx, %scalar.ph ], [ %26, %L13 ]
    ; │ @ simdloop.jl:77 within `macro expansion' @ In[25]:4
    ; │┌ @ array.jl:788 within `getindex'
        %24 = getelementptr inbounds float, float addrspace(13)* %9, i64 %value_phi16
        %25 = load float, float addrspace(13)* %24, align 4
    ; │└
    ; │┌ @ float.jl:400 within `+'
        %26 = fadd fast float %value_phi5, %25
    ; │└
    ; │ @ simdloop.jl:78 within `macro expansion'
    ; │┌ @ int.jl:53 within `+'
        %27 = add nuw nsw i64 %value_phi16, 1
    ; │└
    ; │ @ simdloop.jl:75 within `macro expansion'
    ; │┌ @ int.jl:49 within `<'
        %28 = icmp ult i64 %27, %7
    ; │└
       br i1 %28, label %L13, label %L30

    L30:                                              ; preds = %L13, %middle.block, %top
       %value_phi2 = phi float [ 0.000000e+00, %top ], [ %26, %L13 ], [ %23, %middle.block ]
    ; └
    ;  @ In[25]:6 within `simdsum'
      ret float %value_phi2
    }

Entonces, ¿cuáles son los desafíos?:

- El mayor obstáculo es que tienes que convencer a Julia y LLVM de que puede usar instrucciones SIMD para tu algoritmo dado. Eso no siempre es posible.
- Hay muchas limitaciones de lo que se puede y no se puede vectorizar
- Es necesario pensar en las consecuencias de reordenar su algoritmo

## Resumen

SIMD:
- Explota el paralelismo integrado en un procesador
- Ideal para bucles internos pequeños (y estrechos)
- A menudo ocurre automáticamente si tienes cuidado
    - Siga las [mejores prácticas de rendimiento](https://docs.julialang.org/en/v1/manual/performance-tips/)
    - Usa `@inbounds` para cualquier acceso a un arreglo
    - Evita ramas o llamadas a funciones
- Dependiendo del procesador y los tipos involucrados, puede producir ganancias de 2-16x con una sobrecarga extraordinariamente pequeña

```julia

```

Curso Julia/1 Introduccion Julia/S8-OperacionesVectorizadas.ipynb


```python
from jupyter_to_medium._latex import is_latex_cell, replicate_alignment
from nbformat import read
```

```python
nb = read('/Users/jamisonm/dev/jupyter_to_medium/tests/notebooks/Test Latex.ipynb', as_version=4)
```

```python
md = [x for x in nb['cells'] if x['cell_type'] == 'markdown']
```

```python
lts = [x for x in md if is_latex_cell(x)]
```

```python
l = lts[1]['source']
```

```python
def format_multiline_latex(latex: list, align_offset: int = 4) -> str:

    # we have a multi-liner - we want to:
    # - remove the new lines, we will replace later
    lt = [x.replace("\n", "") for x in latex]
    # - remove latex new line \\ at end of lines
    lt = [x[:-2] if x[-2:] == r"\\" else x for x in lt]
    # - remove '$$' at top and bottom of list
    lt = [x for x in lt if x != "$$"]
    # - remove '$$' even if other latex on that line
    if "$$" in lt[0]:
        lt[0] = lt[0][2:]
    if "$$" in lt[-1]:
        lt[-1] = lt[-1][:-2]
    # - check if latex contains \begin and \end statements
    has_beg_end = ["\\begin" in x or "\\end" in x for x in lt]
    # - if so then we need to handle as mpl can't render
    if max(has_beg_end):
        # check if '{align}' is in the brackets
        has_align = ["align" in x for x in lt]
        # if aligns are present
        if max(has_align):
            # remove \begin{align } and \end{align}
            lt = [x for x in lt if x != "\\begin{align}"]
            lt = [x for x in lt if x != "\\end{align}"]
        else:
            # remove \begin{align } and \end{align}
            lt = [x for x in lt if "\\begin" in x]
            lt = [x for x in lt if "\\end" in x]

    return lt
```

```python
ll = format_multiline_latex(l.split('\n'))
```

```python
ll
```

    ['r^l_T &= ln(\\frac{S_T}{S_0}) ',
     '      &= ln(\\frac{S_T}{S_{T-1}}) + ln(\\frac{S_{T-1}}{S_{T-2}}) + \\ldots + ln(\\frac{S_1}{S_{0}}) ',
     '      &= r^l_{T-1} + r^l_{T-2} + \\ldots + r^l_0']


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, C. Cooper, G.F. Forsyth, A. Krishnan.

# Phugoid Motion

Welcome to [**"Practical Numerical Methods with Python!"**](http://openedx.seas.gwu.edu/courses/GW/MAE6286/2014_fall/about) This course is a collaborative, online, open education project, where we aim to give a foundation in scientific computing. The focus is on numerical solution of problems modeled by ordinary and partial differential equations.

This IPython Notebook introduces the problem we'll be studying in the **first module** of the course: the _phugoid model of glider flight_. We'll start with some background, explaining the physics, and working out the mathematical model.

First, we'll look at an idealized motion where there is no drag, resulting in a simple harmonic motion. We can plot some interesting trajectories that will pique your imagination. In the next notebook, you'll learn to numerically integrate the differential equation using Euler's method. But hang on ... first things first.

The term "phugoid" is used in aeronautics to refer to a motion pattern where an aircraft oscillates up and down —nose-up and climb, then nose-down and descend— around an equilibrium trajectory.  The aircraft oscillates in altitude, speed and pitch, with only small (neglected) variations in the angle of attack, as it repeatedly exchanges kinetic and potential energy.

A low-amplitude phugoid motion can be just a nuisance, as the aircraft does not exceed the stall angle of attack and nothing bad happens. But the mode can also be unstable leading to a stall or even a loop!

Look at this video showing a Cessna single-engine airplane in phugoid motion:

```python
from IPython.display import YouTubeVideo
YouTubeVideo('ysdU4mnRYdM')
```

That doesn't look too good! What's happening?

It can get a lot worse when an aircraft enters one of these modes that is unstable. For example, one of [NASA's Helios Solar Powered Aircraft](http://www.nasa.gov/centers/dryden/history/pastprojects/Helios/) prototype broke up in mid air due to extreme phugoid oscillations!

Helios was a proof-of-concept solar electric-powered flying wing that broke the world altitude record for a non-rocket-powered aircraft in August 2001. But in June 26, 2003, it broke something else. The aircraft entered phugoid motion after encountering turbulence near the Hawaiian Island of Kauai. The high speed in the oscillatory movement exceeded the design limits, and it ended up wrecked in the Pacific Ocean. Luckily, the Helios was remotely operated, and nobody got hurt.

## The physics of phugoids

The phugoid oscillation has the aircraft pitching up and down, as it decelerates and accelerates. The trajectory might look like a sinusoid, as in the figure below. The assumption is that the forward velocity of the aircraft, $v$, varies in such a way that the angle of attack remains (nearly) constant, which means that we can assume a constant lift coefficient.

####Figure 1. Trajectory of an aircraft in phugoid motion.

In the descending portion of the trajectory, the aircraft's velocity increases as it proceeds from a peak to the minimum height—gaining kinetic energy at the expense of potential energy. The contrary happens in the upward segment, as its velocity decreases there.

We measure the pitch angle (between the aircraft's longitudinal axis and the horizontal) as positive when the aircraft's nose is pointing up. In the portion of the trajectory below the center-line, where it curves upwards, the pitch angle $\theta$ is increasing: $\dot{\theta}>0$. And where the trajectory curves down, the pitch angle is decreasing: $\dot{\theta}<0$, as shown in the figure.

Let's remind ourselves of the forces affecting an aircraft in a downward glide. Look at the figure below: we show the flight path, the forces on the glider (no thrust), and the _glide angle_ or flight path angle, $\gamma$, between the flight path and the horizontal.

####Figure 2. Forces on a glider.

The force of lift, $L$ —created by the airflow around the wings— is perpendicular to the trajectory, and the force of drag, $D$, is parallel to the trajectory. Both forces are expressed in terms of coefficients of lift and drag, $C_L$ and $C_D$, respectively, that depend on the wing design and _angle of attack_—the angle between the wing chord and the flight path.

If you are not familiar with airplane aerodynamics, you might be getting confused with some terms here ... and all those angles! But be patient and look things up, if you need to. We're giving you a quick summary here.

Lift and drag are proportional to a surface area, $S$, and the dynamic pressure: $1/2 \rho v^2$, where $\rho$ is the density of air, and $v$ the forward velocity of the aircraft. The equations for lift and drag are:

$$\begin{eqnarray}
L &=& C_L S \times \frac{1}{2} \rho v^2 \\
D &=& C_D S \times \frac{1}{2} \rho v^2
\end{eqnarray}$$

If the glider were in equilibrium, the forces would balance each other. We can equate the forces in the directions perpendicular and parallel to the trajectory, as follows:

$$\begin{equation}
L = W \cos \gamma \quad \text{and} \quad D = W \sin \gamma
\end{equation}$$

where $W$ repesents the weight of the glider.

In the figure, we've drawn the angle $\gamma$ as the _glide angle_, formed between the direction of motion and the horizontal. We are not bothered with the _sign_ of the angle, because we draw a free-body diagram and take the direction of the forces into account in writing our balance equations. But later on, we will need to be careful with the sign of the angles. It can cause you a real headache to keep this straight, so be patient!

It looks like we've set this up to do a little bit of mathematics. Are you ready?

But before, a short glimpse of the history.

## Lanchester's Aerodonetics

"Phugoid theory" was first described by the British engineer Frederick W. Lanchester in _"Aerodonetics"_ (1909). This book is so old that it is now in the public domain, so you can actually download [from Google Books](http://books.google.com/books?id=6hxDAAAAIAAJ&dq=%22phugoid%20theory%20deals%20with%20the%20longitudinal%20stability%22&pg=PA37#v=onepage&q=%22phugoid%20theory%20deals%20with%20the%20longitudinal%20stability%22&f=false) a PDF file of a scan, or read it online.

Lanchester defines phugoid theory as the study of longitudinal stability of a flying machine (aerodone). He first considered the simplification where drag and moment of inertia are neglected. Then he included these effects, obtaining an equation of stability. In addition to describing many experiments by himself and others, Lanchester also reports on _"numerical work ... done by the aid of an ordinary 25-cm slide rule."_ Go figure!

### Ideal case of zero drag

In this section, we follow the derivation given by Milne-Thompson (1966), which we find a little bit easier than that of the original in "Aerodonetics"!

An aircraft flying in a steady, straight horizontal flight has a lift equal to its weight. The velocity in this condition is sometimes called _trim velocity_ ("trim" is what pilots do to set the controls to just stay in a steady flight). Let's use $v_t$ for the trim velocity, and from $L=W$ deduce that:

$$\begin{equation}
W = C_L S \times\frac{1}{2} \rho v_t^2
\end{equation}$$

The weight $W$ is constant for the aircraft, but the lift at any other flight condition depends on the flight speed, $v$. We can use the expression for the weight in terms of $v_t$ to obtain the ratio $L/W$ at any other flight velocity, as follows:

$$\begin{equation}
\frac{L}{W}= \frac{v^2}{v_t^2}
\end{equation}$$

Imagine that the aircraft experienced a little upset, a wind gust, and it finds itself off the "trim" level, in a curved path with an instantaneous angle $\theta$. In the sketch below, we exaggerate the curved trajectory of flight to help you visualize what we'll do next. The angle $\theta$ (using the same name as Milne-Thompson) is between the _trajectory_ and the horizontal, positive up.

####Figure 3. Curved trajectory of the aircraft going up.

A balance of forces now has to take into account that our reference frame is moving with the aircraft, in a rotating frame: we have a centrifugal force. The balance in the direction of lift is thus:

$$\begin{equation}
L- W \cos \theta = \frac{W}{g} \frac{v^2}{R}
\end{equation}$$

where $R$ is the radius of curvature of the trajectory, and $g$ the acceleration of gravity. Recall that the centrifugal acceleration is $v^2/R$. Rearrange this by dividing the equation by the weight, and use the expression we found for $L/W$, above. The following equation results:

$$\begin{equation}
\frac{v^2}{v_t^2}-\cos \theta = \frac{v^2}{g R}
\end{equation}$$

Recall that we simplified the problem assuming that there is no friction, which means that the total energy is constant (the lift does no work). If $z$ represents the depth below a reference horizontal line, the energy per unit mass is (kinetic plus potential energy):

$$\begin{equation}
\frac{1}{2}v^2-g z = \text{constant}
\end{equation}$$

To get rid of that pesky constant, we can choose the reference horizontal line at the level that makes the constant energy equal to zero, so $v^2 = 2 g z$. That helps us re-write the phugoid equation in terms of $z$ as follows:

$$\begin{equation}
\frac{z}{z_t}-\cos \theta = \frac{2z}{R}
\end{equation}$$

Let $ds$ represent a small arc-length of the trajectory. We can write

$$\begin{equation}
\frac{1}{R} = \frac{d\theta}{ds} \quad \text{and}\quad  \sin\theta = -\frac{dz}{ds}
\end{equation}$$

Employing the chain rule of calculus,

$$\begin{equation}
\frac{1}{R} = \frac{d\theta}{ds} = \frac{dz}{ds}\frac{d\theta}{dz} = -\sin \theta\frac{d\theta}{dz}
\end{equation}$$

Multiply the phugoid equation by $\frac{1}{2\sqrt{z}}$ to get:

$$\begin{equation}
\frac{\sqrt{z}}{2z_t} - \frac{\cos\theta}{2\sqrt{z}} = \frac{\sqrt{z}}{R}
\end{equation}$$

Substituting for $1/R$ on the right hand side and bringing the cosine term over to the right, we get:

$$\begin{equation}
\frac{\sqrt{z}}{2z_t} = \frac{\cos \theta}{2 \sqrt{z}} - \sqrt{z} \sin \theta \frac{d\theta}{dz}
\end{equation}$$

The right-hand-side is an exact derivative! We can rewrite it as:

$$\begin{equation}
\frac{d}{dz} \left(\sqrt{z}\cos\theta \right) = \frac{\sqrt{z}}{2z_t}
\end{equation}$$

Integrating this equation, we add an arbitrary constant, chosen as $C\sqrt{z_t}$ which (after dividing through by $\sqrt{z}$) gives:

$$\begin{equation}
\cos \theta = \frac{1}{3}\frac{z}{z_t} + C\sqrt{\frac{z_t}{z}}
\end{equation}$$

Taking the derivative of both sides of equation (15) and applying the relations from equation (10) yields:

$$\begin{equation}
\frac{z_t}{R} = \frac{1}{3} - \frac{C}{2}\sqrt{\frac{z_t^3}{z^3}}
\end{equation}$$

Make sure you have followed the derivation, and perhaps write it out on paper!

## Phugoid Curves

Equation (15) is non-linear, which usually means we are hard-pressed to write a clean expression for the variable of interest, $z$.  In fact, Lanchester himself said that he was unable to _"reduce this expression to a form suitable for co-ordinate plotting."_ If the great polymath couldn't do it, we can't either!

But Lanchester _was_ able to plot a suitable approximation of the phugoid flight path using what he called the "trammel" method.  If you're interested in seeing how he did it, his explanation begins on page [48 of Aerodonetics](http://books.google.com/books?id=6hxDAAAAIAAJ&pg=PA49&lpg=PA48&dq=aerodonetics+the+use+of+the+trammel&source=bl&ots=lB6EVKYQuT&sig=aVE2kiDWZoWftaWczMIrcYftMOs&hl=en&sa=X&ei=gTD_U82fGYjzgwT3moGwCQ&ved=0CCAQ6AEwAA#v=onepage&q=aerodonetics%20the%20use%20of%20the%20trammel&f=false).  It's a trip.

Lanchester used Equations (15) and (16) to solve for the constant $C$ and the radius of curvature $R$ and then iteratively plotted small arcs of the phugoid path. By hand.

We wrote a neat little code that duplicates the manual trammel method, but it might be a bit much for you to absorb in the first lesson.  If you want to look it over, you are more than welcome to.  If you are just starting with Python, skip it for the moment and we'll return to it at the end of this module.

###Plotting the flight path

As we mentioned, we wrote a Python code to reproduce programmatically what Lanchester did graphically. Here's a neat feature of IPython Notebooks: you can run external programs with the magical keyword ... wait for it ... `run`. And the jargon of IPython _is_ to call this "magic." In fact, there are a bunch of [magic functions](http://ipython.org/ipython-doc/dev/interactive/tutorial.html) that you will learn about. They will make you a happy camper.

Let's do it:

```python
%run phugoid.py
%matplotlib inline
```

This code cell loaded our simulated-trammel code, `phugoid.py`. The code defined a function for you in the background, called `plot_flight_path`, taking three inputs: $z_t$, $z$ and $\theta$.

Look again at Equation (15), where we take the positive square root. There are several possibilities, depending on the value that the constant $C$ takes.

* There are no physical solutions for $C>2/3$, because it would result in $\cos\theta>1$.

* If $C=2/3$, then the solution is a horizontal straight line, because $\cos\theta=1$, $\theta=0$ and $R=\infty$.

* Any value of $C$ for which $0 < C < \frac{2}{3}$ will produce "trochoidal"-like paths.  What does this look like? Let's use our custom function `plot_flight_path` to find out!

```python
#zt = 64, z = 16, theta=0
plot_flight_path(64, 16, 0)
```

Cool!  Note that the plot title tells us what the calculated value of $C$ was for our input conditions.  We have a value of $C$ between $0$ and $\frac{2}{3}$ and our path is trochoidal, like we announced it would be.

* For negative values of $C$, the resultant flight path consists of a series of loops.  Let's try it out!

```python
plot_flight_path(75,10,numpy.pi)
```

You can play around with the input values and see what kind of behavior results.  Just note that any value of $C > \frac{2}{3}$ will result in $\cos \theta > 1$, which doesn't exist.  Python will probably throw a few errors if you hit that condition, but just try again!

* The last case is $C = 0$.  Take another look at Equation (16) and plug in $C = 0$, what should happen? It looks like it will just reduce to

$$R = 3z_t$$

It's a constant radius of curvature!  In fact, this solution is a series of semi-circles, with a cusp between them. One way to force $C = 0$ that we can figure out from Equation (15), is to make:

$$z = 3z_t\ \ \ ,\ \ \  \theta = 0$$

```python
plot_flight_path(16,48,0.)
```

That looks an awful lot like a quarter circle.  And what's the radius of the arc?  It's $$r = 48 = 3z_t.$$

We can also get a semi-circle out of our simulated trammel by changing to another configuration where $C$ is (near) zero. Here's one example:

```python
plot_flight_path(64,16,-numpy.pi/2)
```

That is so nice. We have reproduced the trajectories that Lanchester found more than a hundred years ago, painstakingly drawing them by hand with a contraption called a "trammel." It must have taken him days!

Here is how the different phugoid curves are drawn in von Kármán's book, _Aerodynamics_ (1957). He never says _how_ he drew them, but we're guessing by hand, also. We did pretty good!

####Figure 4. Phugoid curves in von Kármán (1957).

In the next notebook of this series, we'll look at the differential equation that arises when you consider small perturbations on the horizontal phugoid, and we'll learn to numerically integrate that to get the flight paths.

## References

1. Lanchester, F. W. _Aerodonetics_, D. van Nostrand Company: New York, 1909. On the public domain. [Get it from Google Books](http://books.google.com/books?id=6hxDAAAAIAAJ&pg=PP1#v=onepage&q&f=false).

2. Milne-Thompson, L. M. _Theoretical Aerodynamics_, Dover 2012 reprint of the revised 1966 edition. [Read on Google Books](http://books.google.com/books?id=EMfCAgAAQBAJ&lpg=PP1&pg=PP1#v=onepage&q&f=false) (see section 18.5)

3. Sinha, N. K. and Ananthkrishnan, N. _Elementary Flight Dynamics with an introduction to Bifurcation and Continuation Methods_, CRC Press, 2013. [Read on Google Books](http://books.google.com/books?id=yXL6AQAAQBAJ&lpg=PP1&pg=PP1#v=onepage&q&f=false) (see chapter 5)

4. von Kármán, T. _Aerodynamics_, Dover 2004 reprint of the 1957 2nd edition. (see pages 149–151)

## About this course

This course is a collaborative project in open education. Three professors across the world are teaching connected courses, developing and reviewing course materials, and interacting with the community of learners that follow the course online. They are:

* Lorena A. Barba, the George Washington University, United States
* Carlos Jerez, Pontificia Universidad Católica de Chile
* Ian Hawke, Southampton University, United Kingdom

---

######The cell below loads the style of this notebook.

```python
# Execute this cell to load the notebook's style sheet, then ignore it
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
    margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
    font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
    font-family: "PT Mono";
    font-size: 90%;
}

</style>