#  CS109A Introduction to Data Science

## Lab 3:  Scikit-learn for Regression

**Harvard University**

**Fall 2018**

**Instructors:** Pavlos Protopapas and Kevin Rader

**Authors:** David Sondak, Will Claybaugh, Eleni Kaxiras

---

```python
# RUN THIS CELL TO PROPERLY HIGHLIGHT THE EXERCISES
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2018-CS109A/master/content/styles/cs109.css").text
HTML(styles)
```

---

# Table of Contents

 Array creation and reshape

 Some plotting

 Simple linear regression

 $k$-nearest neighbors

## Learning Goals

Overall description and goal for the lab.

By the end of this lab, you should be able to:
* Understand array reshaping
* Review how to make plots
* Feel comfortable with simple linear regression
* Feel comfortable with $k$ nearest neighbors

**This lab corresponds to lecture 4 and maps on to homework 2 (and beyond).**

```python
# import the necessary libraries
import warnings
warnings.filterwarnings('ignore')
%matplotlib inline
import numpy as np
import scipy as sp
import matplotlib as mpl
import matplotlib.cm as cm
import matplotlib.pyplot as plt
import pandas as pd
import time
pd.set_option('display.width', 500)
pd.set_option('display.max_columns', 100)
pd.set_option('display.notebook_repr_html', True)
import seaborn as sns
```

## Simple Linear Regression
Linear regression and its many extensions are a workhorse of the statistics and data science community, both in application and as a reference point for other models. Most of the major concepts in machine learning can be and often are discussed in terms of various linear regression models. Thus, this section will introduce you to building and fitting linear regression models and some of the process behind it, so that you can 1) fit models to data you encounter 2) experiment with different kinds of linear regression and observe their effects 3) see some of the technology that makes regression models work.

### Linear regression with a toy dataset
We first examine a toy problem, focusing our efforts on fitting a linear model to a small dataset with three observations.  Each observation consists of one predictor $x_i$ and one response $y_i$ for $i = 1, 2, 3$,

\begin{align*}
(x , y) = \{(x_1, y_1), (x_2, y_2), (x_3, y_3)\}.
\end{align*}

To be very concrete, let's set the values of the predictors and responses.

\begin{equation*}
(x , y) = \{(1, 2), (2, 2), (3, 4)\}
\end{equation*}

There is no line of the form $\beta_0 + \beta_1 x = y$ that passes through all three observations, since the data are not collinear. Thus our aim is to find the line that best fits these observations in the *least-squares sense*, as discussed in lecture.

Exercise (10 min)

* Make two numpy arrays out of this data, x_train and y_train
* Check the dimentions of these arrays
* Try to reshape them into a different shape
* Make points into a very simple scatterplot
* Make a better scatterplot

```python
# your code here
x_train = np.array([1,2,3])
y_train = np.array([2,3,6])
type(x_train)
```

    numpy.ndarray

```python
x_train.shape
```

    (3,)

```python
x_train = x_train.reshape(3,1)
x_train.shape
```

    (3, 1)

```python
xx = np.array([[1,3,5],[6,2,1]])
xx.shape
```

    (2, 3)

```python
xx = xx.reshape(3,-1)
xx
```

    array([[1, 3],
           [5, 6],
           [2, 1]])

```python
# make a simple scatterplot
x_train = np.array([1,2,3])
y_train = np.array([2,2,4])
plt.scatter(x_train,y_train)

# check dimensions
print(x_train.shape,y_train.shape)
```

```python
def nice_scatterplot(x, y, title):
    # font size
    f_size = 18

    # make the figure
    fig, ax = plt.subplots(1,1, figsize=(8,5)) # Create figure object

    # set axes limits to make the scale nice
    ax.set_xlim(np.min(x)-1, np.max(x) + 1)
    ax.set_ylim(np.min(y)-1, np.max(y) + 1)

    # adjust size of tickmarks in axes
    ax.tick_params(labelsize = f_size)

    # remove tick labels
    ax.tick_params(labelbottom=False,  bottom=False)

    # adjust size of axis label
    ax.set_xlabel(r'$x$', fontsize = f_size)
    ax.set_ylabel(r'$y$', fontsize = f_size)

    # set figure title label
    ax.set_title(title, fontsize = f_size)

    # you may set up grid with this
    ax.grid(True, lw=1.75, ls='--', alpha=0.15)

    # make actual plot (Notice the label argument!)
    #ax.scatter(x, y, label=r'$my points$')
    #ax.scatter(x, y, label='$my points$')
    ax.scatter(x, y, label=r'$my\,points$')
    ax.legend(loc='best', fontsize = f_size);

    return ax

nice_scatterplot(x_train, y_train, 'hello nice plot')
```

#### Formulae
Linear regression is special among the models we study beuase it can be solved explicitly. While most other models (and even some advanced versions of linear regression) must be solved itteratively, linear regression has a formula where you can simply plug in the data.

For the single predictor case it is:
    \begin{align}
      \beta_1 &= \frac{\sum_{i=1}^n{(x_i-\bar{x})(y_i-\bar{y})}}{\sum_{i=1}^n{(x_i-\bar{x})^2}}\\
      \beta_0 &= \bar{y} - \beta_1\bar{x}\
    \end{align}

Where $\bar{y}$ and $\bar{x}$ are the mean of the y values and the mean of the x values, respectively.

From the re-aranged second equation we can see that the best-fit line  passes through $(\bar{x},\bar{y})$, the center of mass of the data

From any of the first equations, we can see that the slope of the line has to do with whether or not an x value that is above/below the center of mass is typically paired with a y value that is likewise above/below, or typically paired with one that is opposite.

###  Building a model from scratch
In this part, we will solve the equations for simple linear regression and find the best fit solution to our toy problem.

The snippets of code below implement the linear regression equations on the observed predictors and responses, which we'll call the training data set.  Let's walk through the code.

We have to reshape our arrrays to 2D. We will see later why.

Exercise (5 min)

* make an array with shape (2,3)
* reshape it to a size that you want

```python
# your code here
xx = np.array([[1,2,3],[4,6,8]])
xxx = xx.reshape(-1,2)
xxx.shape
```

```python
# Reshape to be a proper 2D array
x_train = x_train.reshape(x_train.shape[0], 1)
y_train = y_train.reshape(y_train.shape[0], 1)

print(x_train.shape)
```

    (3, 1)

```python
# first, compute means
y_bar = np.mean(y_train)
x_bar = np.mean(x_train)

# build the two terms
numerator = np.sum( (x_train - x_bar)*(y_train - y_bar) )
denominator = np.sum((x_train - x_bar)**2)

print(numerator.shape, denominator.shape) #check shapes
```

    () ()

* Why the empty brackets? (The numerator and denominator are scalars, as expected.)

```python
#slope beta1
beta_1 = numerator/denominator

#intercept beta0
beta_0 = y_bar - beta_1*x_bar

print("The best-fit line is {0:3.2f} + {1:3.2f} * x".format(beta_0, beta_1))
print(f'The best fit is {beta_0}')
```

    The best-fit line is -0.33 + 2.00 * x
    The best fit is -0.3333333333333335

Exercise (5 min)

Turn the code from the above cells into a function called `simple_linear_regression_fit`, that inputs the training data and returns `beta0` and `beta1`.

To do this, copy and paste the code from the above cells below and adjust the code as needed, so that the training data becomes the input and the betas become the output.

```python
def simple_linear_regression_fit(x_train: np.ndarray, y_train: np.ndarray) -> np.ndarray:

    return
```

Check your function by calling it with the training data from above and printing out the beta values.

```python
def simple_linear_regression_fit(x_train: np.ndarray, y_train: np.ndarray) -> np.ndarray:
    """
    Inputs:
    x_train: a (num observations by 1) array holding the values of the predictor variable
    y_train: a (num observations by 1) array holding the values of the response variable

    Returns:
    beta_vals:  a (num_features by 1) array holding the intercept and slope coeficients
    """

    # Check input array sizes
    if len(x_train.shape) < 2:
        print("Reshaping features array.")
        x_train = x_train.reshape(x_train.shape[0], 1)

    if len(y_train.shape) < 2:
        print("Reshaping observations array.")
        y_train = y_train.reshape(y_train.shape[0], 1)

    # first, compute means
    y_bar = np.mean(y_train)
    x_bar = np.mean(x_train)

    # build the two terms
    numerator = np.sum( (x_train - x_bar)*(y_train - y_bar) )
    denominator = np.sum((x_train - x_bar)**2)

    #slope beta1
    beta_1 = numerator/denominator

    #intercept beta0
    beta_0 = y_bar - beta_1*x_bar

    return np.array([beta_0,beta_1])

```

* Let's run this function and see the coefficients

```python
x_train = np.array([1 ,2, 3])
y_train = np.array([2, 2, 4])

betas = simple_linear_regression_fit(x_train, y_train)

beta_0 = betas[0]
beta_1 = betas[1]

print("The best-fit line is {0:8.6f} + {1:8.6f} * x".format(beta_0, beta_1))
```

    Reshaping features array.
    Reshaping observations array.
    The best-fit line is 0.666667 + 1.000000 * x

Exercise (5 min)

* Do the values of `beta0` and `beta1` seem reasonable?
* Plot the training data using a scatter plot.
* Plot the best fit line with `beta0` and `beta1` together with the training data.

```python
fig_scat, ax_scat = plt.subplots(1,1, figsize=(10,6))

# Plot best-fit line
x_train = np.array([[1, 2, 3]]).T

best_fit = beta_0 + beta_1 * x_train

ax_scat.scatter(x_train, y_train, s=300, label='Training Data')
ax_scat.plot(x_train, best_fit, ls='--', label='Best Fit Line')

ax_scat.set_xlabel(r'$x_{train}$')
ax_scat.set_ylabel(r'$y$');
```

The values of `beta0` and `beta1` seem roughly reasonable.  They capture the positive correlation.  The line does appear to be trying to get as close as possible to all the points.

### Building a model with `statsmodels` and `sklearn`

Now that we can concretely fit the training data from scratch, let's learn two `python` packages to do it all for us:
* [statsmodels](http://www.statsmodels.org/stable/regression.html) and
* [scikit-learn (sklearn)](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html).

Our goal  is to show how to implement simple linear regression with these packages.  For an important sanity check, we compare the $\beta$ values from `statsmodels` and `sklearn` to the $\beta$ values that we found from above with our own implementation.

For the purposes of this lab, `statsmodels` and `sklearn` do the same thing.  More generally though, `statsmodels` tends to be easier for inference \[finding the values of the slope and intercept and dicussing uncertainty in those values\], whereas `sklearn` has machine-learning algorithms and is better for prediction \[guessing y values for a given x value\]. (Note that both packages make the same guesses, it's just a question of which activity they provide more support for.

**Note:** `statsmodels` and `sklearn` are different packages!  Unless we specify otherwise, you can use either one.

Below is the code for `statsmodels`.  `Statsmodels` does not by default include the column of ones in the $X$ matrix, so we include it manually with `sm.add_constant`.

```python
import statsmodels.api as sm
```

```python
# create the X matrix by appending a column of ones to x_train
X = sm.add_constant(x_train)

# this is the same matrix as in our scratch problem!
print(X)

# build the OLS model (ordinary least squares) from the training data
toyregr_sm = sm.OLS(y_train, X)

# do the fit and save regression info (parameters, etc) in results_sm
results_sm = toyregr_sm.fit()

# pull the beta parameters out from results_sm
beta0_sm = results_sm.params[0]
beta1_sm = results_sm.params[1]

print("The regression coefficients from the statsmodels package are: beta_0 = {0:8.6f} and beta_1 = {1:8.6f}".format(beta0_sm, beta1_sm))
```

    [[ 1.  1.]
     [ 1.  2.]
     [ 1.  3.]]
    The regression coefficients from the statsmodels package are: beta_0 = 0.666667 and beta_1 = 1.000000

Besides the beta parameters, `results_sm` contains a ton of other potentially useful information.

```python
import warnings
warnings.filterwarnings('ignore')
print(results_sm.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                      y   R-squared:                       0.750
    Model:                            OLS   Adj. R-squared:                  0.500
    Method:                 Least Squares   F-statistic:                     3.000
    Date:                Fri, 21 Sep 2018   Prob (F-statistic):              0.333
    Time:                        11:31:12   Log-Likelihood:                -2.0007
    No. Observations:                   3   AIC:                             8.001
    Df Residuals:                       1   BIC:                             6.199
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.6667      1.247      0.535      0.687     -15.181      16.514
    x1             1.0000      0.577      1.732      0.333      -6.336       8.336
    ==============================================================================
    Omnibus:                          nan   Durbin-Watson:                   3.000
    Prob(Omnibus):                    nan   Jarque-Bera (JB):                0.531
    Skew:                          -0.707   Prob(JB):                        0.767
    Kurtosis:                       1.500   Cond. No.                         6.79
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Now let's turn our attention to the `sklearn` library.

```python
from sklearn import linear_model
```

```python
# build the least squares model
toyregr = linear_model.LinearRegression()

# save regression info (parameters, etc) in results_skl
results = toyregr.fit(x_train, y_train)

# pull the beta parameters out from results_skl
beta0_skl = toyregr.intercept_
beta1_skl = toyregr.coef_[0]

print("The regression coefficients from the sklearn package are: beta_0 = {0:8.6f} and beta_1 = {1:8.6f}".format(beta0_skl, beta1_skl))
```

    The regression coefficients from the sklearn package are: beta_0 = 0.666667 and beta_1 = 1.000000

We should feel pretty good about ourselves now, and we're ready to move on to a real problem!

### The shape of things in `scikit-learn`
Before diving right in to a "real" problem, we really ought to discuss more of the details of `sklearn`.  We do this now.  Along the way, we'll import the real-world dataset.

`Scikit-learn` is the main `python` machine learning library. It consists of many learners which can learn models from data, as well as a lot of utility functions such as `train_test_split`. It can be used in `python` by the incantation `import sklearn`.

In scikit-learn, an **estimator** is a Python object that implements the methods fit(X, y) and predict(T)

Let's see the structure of `scikit-learn` needed to make these fits. `.fit` always takes two arguments:
```python
  estimator.fit(Xtrain, ytrain)
```
We will consider two estimators in this lab: `LinearRegression` and `KNeighborsRegressor`.

Critically, `Xtrain` must be in the form of an *array of arrays* (or a 2x2 array) with the inner arrays each corresponding to one sample, and whose elements correspond to the feature values for that sample (visuals coming in a moment).

`ytrain` on the other hand is a simple array of responses.  These are continuous for regression problems.

### Practice with `sklearn`
We begin by loading up the `mtcars` dataset and cleaning it up a little bit.

```python
import pandas as pd

#load mtcars
dfcars = pd.read_csv("../../data/mtcars.csv")
dfcars = dfcars.rename(columns={"Unnamed: 0":"car name"})
dfcars.head()
```

      car name
      mpg
      cyl
      disp
      hp
      drat
      wt
      qsec
      vs
      am
      gear
      carb

      0
      Mazda RX4
      21.0
      6
      160.0
      110
      3.90
      2.620
      16.46
      0
      1
      4
      4

      1
      Mazda RX4 Wag
      21.0
      6
      160.0
      110
      3.90
      2.875
      17.02
      0
      1
      4
      4

      2
      Datsun 710
      22.8
      4
      108.0
      93
      3.85
      2.320
      18.61
      1
      1
      4
      1

      3
      Hornet 4 Drive
      21.4
      6
      258.0
      110
      3.08
      3.215
      19.44
      1
      0
      3
      1

      4
      Hornet Sportabout
      18.7
      8
      360.0
      175
      3.15
      3.440
      17.02
      0
      0
      3
      2

Next, let's split the dataset into a training set and test set.

```python
# split into training set and testing set
from sklearn.model_selection import train_test_split

#set random_state to get the same split every time
traindf, testdf = train_test_split(dfcars, test_size=0.2, random_state=42)
```

```python
# testing set is around 20% of the total data; training set is around 80%
print("Shape of full dataset is: {0}".format(dfcars.shape))
print("Shape of training dataset is: {0}".format(traindf.shape))
print("Shape of test dataset is: {0}".format(testdf.shape))
```

    Shape of full dataset is: (32, 12)
    Shape of training dataset is: (25, 12)
    Shape of test dataset is: (7, 12)

Now we have training and test data.  We still need to select a predictor and a response from this dataset.  Keep in mind that we need to choose the predictor and response from both the training and test set.  You will do this in the exercises below.  However, we provide some starter code for you to get things going.

```python
# Extract the response variable that we're interested in
y_train = traindf.mpg
```

Notice the shape of `y_train`.

```python
np.shape(y_train)
```

    (25,)

Another way to see the shape is to use the shape method.

```python
y_train.shape
```

    (25,)

This is *not* an "array of arrays".  That's okay!  Remember, `sklearn` requires an array of arrays only for the predictor array!  You will have to pay close attention to this in the exercises later.

For now, let's discuss two ways out of this debacle.  All we'll do is get `y_train` to be an array of arrays.  This doesn't hurt anything because `sklearn` doesn't care too much about the shape of `y_train`.

First, let's reshape `y_train` to be an array of arrays using the `reshape` method.  We want the first dimension of `y_train` to be size $25$ and the second dimension to be size $1$.

```python
y_train_reshape = y_train.values.reshape(y_train.shape[0], 1)
```

```python
y_train_reshape.shape
```

    (25, 1)

Notice that `y_train.shape[0]` gives the size of the first dimension.

There's an even easier way to get the correct shape right from the beginning.

```python
y_train_reshape = traindf[['mpg']]
```

```python
y_train_reshape.shape
```

    (25, 1)

Finally, there is a nice shortcut to reshaping an array.  `numpy` can infer a dimension based on the other dimensions specified.

```python
y_train_reshape = y_train.values.reshape(-1,1)
y_train_reshape.shape
```

    (25, 1)

In this case, we said the second dimension should be size $1$.  Since the requirement of the `reshape()` method is that the requested dimensions be compatible, `numpy` decides the the first dimension must be size $25$.

What would the `.shape` return if we did `y_train.values.reshape(-1,5)`?

Okay, enough of that.  The whole reason we went through that whole process was to show you how to reshape your data into the correct format.

**IMPORTANT:** Remember that your response variable `ytrain` can be a vector but your predictor variable `xtrain` ***must*** be an array!

### Simple linear regression with automobile data
We will now use `sklearn` to predict automobile mileage per gallon (mpg) and evaluate these predictions. We already loaded the data and split them into a training set and a test set.

We need to choose the variables that we think will be good predictors for the dependent variable `mpg`.

Exercise (10 min)

* Pick one variable to use as a predictor for simple linear regression.  Create a markdown cell below and discuss your reasons.
* Justify your choice with some visualizations.
* Is there a second variable you'd like to use? For example, we're not doing multiple linear regression here, but if we were, is there another variable you'd like to include if we were using two predictors?

```python
# Your code here
```

```python
y_mpg = dfcars.mpg
x_wt = dfcars.wt
x_hp = dfcars.hp

fig_wt, ax_wt = plt.subplots(1,1, figsize=(10,6))
ax_wt.scatter(x_wt, y_mpg)
ax_wt.set_xlabel(r'Car Weight')
ax_wt.set_ylabel(r'Car MPG')

fig_hp, ax_hp = plt.subplots(1,1, figsize=(10,6))
ax_hp.scatter(x_hp, y_mpg)
ax_hp.set_xlabel(r'Car HP')
ax_hp.set_ylabel(r'Car MPG')

```

Exercise

* Use `sklearn` to fit the training data using simple linear regression.
* Use the model to make mpg predictions on the test set.
* Plot the data and the prediction.
* Print out the mean squared error for the training set and the test set and compare.

**Hints:**
* Use the following to perform the analysis:
```python
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
```

```python
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

dfcars = pd.read_csv("../../data/mtcars.csv")
dfcars = dfcars.rename(columns={"Unnamed: 0":"name"})

dfcars.head()
```

      name
      mpg
      cyl
      disp
      hp
      drat
      wt
      qsec
      vs
      am
      gear
      carb

      0
      Mazda RX4
      21.0
      6
      160.0
      110
      3.90
      2.620
      16.46
      0
      1
      4
      4

      1
      Mazda RX4 Wag
      21.0
      6
      160.0
      110
      3.90
      2.875
      17.02
      0
      1
      4
      4

      2
      Datsun 710
      22.8
      4
      108.0
      93
      3.85
      2.320
      18.61
      1
      1
      4
      1

      3
      Hornet 4 Drive
      21.4
      6
      258.0
      110
      3.08
      3.215
      19.44
      1
      0
      3
      1

      4
      Hornet Sportabout
      18.7
      8
      360.0
      175
      3.15
      3.440
      17.02
      0
      0
      3
      2

```python
traindf, testdf = train_test_split(dfcars, test_size=0.2, random_state=42)

y_train = np.array(traindf.mpg)
X_train = np.array(traindf.wt)
X_train = X_train.reshape(X_train.shape[0], 1)
```

```python
y_test = np.array(testdf.mpg)
X_test = np.array(testdf.wt)
X_test = X_test.reshape(X_test.shape[0], 1)

#create linear model
regression = LinearRegression()

#fit linear model
regression.fit(X_train, y_train)

predicted_y = regression.predict(X_test)

r2 = regression.score(X_test, y_test)
print(r2)
```

    0.6879761857596272

```python
reg = LinearRegression()
reg.fit(X_train,y_train)
reg.predict(X_test)
reg.coef_
reg.score(X_test, y_test)
reg.intercept_
```

    36.93731031351841

```python
print(regression.score(X_train, y_train))

print(mean_squared_error(predicted_y, y_test))
print(mean_squared_error(y_train, regression.predict(X_train)))

print('Coefficients: \n', regression.coef_[0], regression.intercept_)
```

    0.770137990979
    12.4759856599
    7.77369776639
    Coefficients:
     -5.33694140056 36.9373103135

```python
fig, ax = plt.subplots(1,1, figsize=(10,6))
ax.plot(y_test, predicted_y, 'o')
grid = np.linspace(np.min(dfcars.mpg), np.max(dfcars.mpg), 100)
ax.plot(grid, grid, color="black") # 45 degree line
ax.set_xlabel("actual y")
ax.set_ylabel("predicted y")

fig1, ax1 = plt.subplots(1,1, figsize=(10,6))
ax1.plot(dfcars.wt, dfcars.mpg, 'o')
xgrid = np.linspace(np.min(dfcars.wt), np.max(dfcars.wt), 100)
ax1.plot(xgrid, regression.predict(xgrid.reshape(100, 1)))
```

## $k$-nearest neighbors

Great, so we did a simple linear regression on the car data.

Now that you're familiar with `sklearn`, you're ready to do a KNN regression.  Let's use $5$ nearest neighbors.

```python
from sklearn.neighbors import KNeighborsRegressor
knnreg = KNeighborsRegressor(n_neighbors=5)
```

```python
knnreg.fit(X_train, y_train)
r2 = knnreg.score(X_test, y_test)
r2
```

    0.6992188728167332

Exercise

What is the $R^{2}$ score on the training set?

```python
# Your code here
```

```python
# solution
knnreg.score(X_train, y_train)
```

    0.8718092469742431

Lets vary the number of neighbors and see what we get.

```python
regdict = {}
# Do a bunch of KNN regressions
for k in [1, 2, 4, 6, 8, 10, 15]:
    knnreg = KNeighborsRegressor(n_neighbors=k)
    knnreg.fit(X_train, y_train)
    regdict[k] = knnreg # Store the regressors in a dictionary
```

```python
# Now let's plot it all
fig, ax = plt.subplots(1,1, figsize=(10,6))

ax.plot(dfcars.wt, dfcars.mpg, 'o', label="data")

xgrid = np.linspace(np.min(dfcars.wt), np.max(dfcars.wt), 100)
for k in [1, 2, 6, 10, 15]:
    predictions = regdict[k].predict(xgrid.reshape(100,1))
    if k in [1, 6, 15]:
        ax.plot(xgrid, predictions, label="{}-NN".format(k))

ax.legend();
```

Notice how the $1$-NN goes through every point on the training set but utterly fails elsewhere. Lets look at the scores on the training set.

```python
ks = range(1, 15) # Grid of k's
scores_train = [] # R2 scores
for k in ks:
    knnreg = KNeighborsRegressor(n_neighbors=k) # Create KNN model
    knnreg.fit(X_train, y_train) # Fit the model to training data
    score_train = knnreg.score(X_train, y_train) # Calculate R^2 score
    scores_train.append(score_train)

# Plot
fig, ax = plt.subplots(1,1, figsize=(12,8))
ax.plot(ks, scores_train,'o-')
ax.set_xlabel(r'$k$')
ax.set_ylabel(r'$R^{2}$')
```

Why do we get a perfect $R^2$ at k=1?

Exercise (5 min)

* Make the same plot as above on the *test* set.
* What is the best $k$?

```python
ks = range(1, 15) # Grid of k's
scores_test = [] # R2 scores
for k in ks:
    knnreg = KNeighborsRegressor(n_neighbors=k) # Create KNN model
    knnreg.fit(X_train, y_train) # Fit the model to training data
    score_test = knnreg.score(X_test, y_test) # Calculate R^2 score
    scores_test.append(score_test)

# Plot
fig, ax = plt.subplots(1,1, figsize=(12,8))
ax.plot(ks, scores_test,'o-', ms=12)
ax.set_xlabel(r'$k$')
ax.set_ylabel(r'$R^{2}$')
```

```python

```


```python
import os
import sys
import glob
import operator as op
import itertools as it
from functools import reduce, partial
import numpy as np
import pandas as pd
from pandas import DataFrame, Series
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_context("notebook", font_scale=1.5)
%matplotlib inline

from sympy import symbols, hessian, Function, N
```

# Algorithms for Optimization and Root Finding for Multivariate Problems

### Note: much of the following notes are taken from Nodecal, J., and S. J. Wright. "Numerical optimization." (2006). It is available online via the library.

## Convexity

A subset $A\subset \mathbb{R}^n$ is *convex* if for any two points $x,y\in A$, the line segment:

$$tx + (1-t)y \;\;\;\;\;\; t\in [0,1]$$
is also in $A$

A function $f:\mathbb{R}^n \rightarrow \mathbb{R}$ is *convex* if its domain $D$ is a convex set and for any two points $x,y\in D$, the graph of $f$ (a subset of $\mathbb{R}^{n+1})$ lies below the line:

$$tf(x) + (1-t)f(y)\;\;\;\;\;t\in [0,1]$$

i.e.
$$f(tx+(1-t)y) \leq tf(x) + (1-t)f(y)\;\;\;\;\;t\in [0,1]$$

#### Convexity guarantees that if an optimizer converges, it converges to the global minimum.

Luckily, we often encounter convex problems in statistics.

## Line Search Methods

There are essentially two classes of multivariate optimization methods. We'll cover line search methods, but refer the reader to Nodecal and Wright for discussion of 'trust region methods'. We should note that all of these methods require that we are 'close' to the minimum (maximum) we are seeking, and that 'noisy' functions or ill-behaved functions are beyond our scope.

A line search method is exactly as it sounds - we search on a line (in $n$ dimensional space) and try to find a minimum.  We start with an initial point, and use an iterative method:

$$x_{k+1} = x_k + \alpha_k p_k$$

where $\alpha_k$ is the *step size* and $p_k$ is the search direction. These are the critical choices that change the behavior of the search.

#### Step Size

Ideally, (given a choice of direction, $p_k$) we would want to minimize:

$$\varphi(\alpha) = f(x_k + \alpha p_k)$$

with respect to $\alpha$. This is usually computationally intensive, so in practice, a sequence of $\alpha$ candidates are generated, and then the 'best' is chosen according to some 'conditions'. We won't be going into detail regarding these. The important thing to know is that they ensure that $f$ decreases sufficiently, according to some conditions. Interested students should see Nodecal.

## Steepest Descent

In steepest descent, one chooses $p_k=\nabla f_k = \nabla f(x_k)$. It is so named, because the gradient points in the direction of steepest ascent, thus, $-\nabla f_k$ will point in the direction of steepest descent. We'll consider this method in its ideal case, that of a quadratic:

$$f(x) = \frac12 x^TQx - b^Tx$$

where $Q$ is positive-definite and symmetric. Note that:

$$\nabla f = Qx -b$$

so the minimum occurs at $x$ such that

$$Qx= b$$

Clearly, we can solve this easily, but let's walk through the algorithm and first find the (ideal) step length:

$$f(x_k - \alpha \nabla f_k) = \frac12\left(x_k - \alpha \nabla f_k\right)^TQ\left(x_k - \alpha \nabla f_k\right) - b^T \left(x_k - \alpha \nabla f_k\right) $$

If we differentiate this with respect to $\alpha$ and find the zero, we obtain:

$$\alpha_k = \frac{\nabla f_k^T\nabla f_k}{\nabla f_k^TQ\nabla f_k}$$

Thus,

$$x_{k+1} = x_k - \frac{\nabla f_k^T\nabla f_k}{\nabla f_k^TQ\nabla f_k} \nabla f_k$$

But we know that $\nabla f_k = Qx_k -b$, so we have a closed form solution for $x_{k+1}$. This allows us to compute an error bound. Again, details can be found in the text, but here is the result:

$$||x_{k+1} - x^*||_Q^2 \leq \left(\frac{\lambda_n - \lambda_1}{\lambda_n+\lambda_1}\right)^2 ||x_{k} - x^*||_Q^2$$

where $0<\lambda_1\leq ... \leq \lambda_n$ and $x^*$ denotes the minimizer.

Now, if $\lambda_1=...=\lambda_n = \lambda$, then $Q=\lambda I$, the algorithm converges in one step. Geometrically, the contours are ellipsoids, the value of $\frac{\lambda_n}{\lambda_1}$ elongates the axes and causes the steps to 'zig-zag'. Because of this, convergence slows as $\frac{\lambda_n}{\lambda_1}$ increases.

## Newton's Method

Newton's method is another line-search, and here

$$p_k = -H^{-1}\nabla f_k$$

Note that if the Hessian is not positive definite, this may not always be a descent direction.

In the neighborhood of a local minimum, the Hessian *will* be positive definite. Now, if $x_0$ is 'close enough' to the minimizer $x^*$, the step size $\alpha_k =1$ gives quadratic convergence.

The advantage of multiplying the gradient by the inverse of the Hessian is that the gradient is corrected for curvature, and the new direction points toward the minimum.

```python

#def Quad(x):
#    return (x[1:])*np.sin(x[:-1])**2.0)

#def DQuad(x,y):
#    return (np.array([np.cos(x)*np.sin(y)**2.0,2.0*np.sin(x)*np.cos(y)**2.0]))

def Quad(x):
    return ((x[1:])**2.0 + 5*(x[:-1])**2.0)

def DQuad(x,y):
    return (np.array([2.0*x,10.0*y]))
```

```python
x = np.linspace(-20,20, 100)
y = np.linspace(-20,20, 100)
X, Y = np.meshgrid(x, y)
Z = Quad(np.vstack([X.ravel(), Y.ravel()])).reshape((100,100))
Hinv=-np.array([[0.5,0],[0,0.1]])
```

```python
plt.figure(figsize=(12,4))
plt.subplot(121)

plt.contour(X,Y,Z);
plt.title("Steepest Descent");
step=-0.25
X0 = 10.0
Y0 = 1.0

Ngrad=Hinv.dot(DQuad(X0,Y0))

sgrad = step*DQuad(X0,Y0)
plt.quiver(X0,Y0,sgrad[0],sgrad[1],color='red',angles='xy',scale_units='xy',scale=1);
X1 = X0 + sgrad[0]
Y1 = Y0 + sgrad[1]
sgrad = step*DQuad(X1,Y1)
plt.quiver(X1,Y1,sgrad[0],sgrad[1],color='green',angles='xy',scale_units='xy',scale=1);
X2 = X1 + sgrad[0]
Y2 = Y1 + sgrad[1]
sgrad = step*DQuad(X2,Y2)
plt.quiver(X2,Y2,sgrad[0],sgrad[1],color='purple',angles='xy',scale_units='xy',scale=1);

plt.subplot(122)
plt.contour(X,Y,Z);
plt.title("Newton's Method")
plt.quiver(X0,Y0,Ngrad[0],Ngrad[1],color='purple',angles='xy',scale_units='xy',scale=1);

#Compute Hessian and plot again.

```

## Coordinate Descent

Another method is called 'coordinate' descent, and it involves searching along coordinate directions (cyclically), i.e.:

$$p_{mk} = e_{k} \;\;\;\;\;\; k=1,...,n$$

where $m$ is the number of steps.

The main advantage is that $\nabla f$ is not required. It can behave reasonably well, if coordinates are not tightly coupled.

### Newton CG Algorithm

Features:

* Minimizes a 'true' quadratic on $\mathbb{R}^n$ in $n$ steps
* Does NOT require storage or inversion of an $n \times n$ matrix.

We begin with $:\mathbb{R}^n\rightarrow \mathbb{R}$.  Take a quadratic approximation to $f$:

$$f(x) \approx \frac12 x^T H x + b^Tx + c$$

Note that in the neighborhood of a minimum, $H$ will be positive-definite (and symmetric). (If we are maximizing, just consider $-H$).

This reduces the optimization problem to finding the zeros of

$$Hx = -b$$

This is a linear problem, which is nice.  The dimension $n$ may be very large - which is not so nice.

#### General Inner Product

Recall the axiomatic definition of an inner product $<,>_A$:

* For any two vectors $v,w$ we have
$$<v,w>_A = <w,v>_A$$
* For any vector $v$
$$<v,v>_A \;\geq 0$$
   with equality $\iff$ $v=0$.
* For $c\in\mathbb{R}$ and $u,v,w\in\mathbb{R}^n$, we have
$$<cv+w,u> = c<v,u> + <w,u>$$

These properties are known as symmetric, positive definite and bilinear, respectively.

Fact: If we denote the standard inner product on $\mathbb{R}^n$ as $<,>$ (this is the 'dot product'), any symmetric, positive definite $n\times n$ matrix $A$ defines an inner product on $\mathbb{R}^n$ via:

$$<v,w>_A \; = <v,Aw> = v^TAw$$

Just as with the standard inner product, general inner products define for us a notion of 'orthogonality'. Recall that with respect to the standard product, 2 vectors are orthogonal if their product vanishes. The same applies to $<,>_A$:

$$<v,w>_A = 0 $$

means that $v$ and $w$ are orthogonal under the inner product induced by $A$.  Equivalently, if $v,w$ are orthogonal under $A$, we have:

$$v^TAw = 0$$

This is also called *conjugate* (thus the name of the method).

#### Conjugate Vectors

Suppose we have a set of $n$ vectors $p_1,...,p_n$ that are mutually conjugate.  These vectors form a basis of $\mathbb{R}^n$.  Getting back to the problem at hand, this means that our solution vector $x$ to the linear problem may be written as follows:

$$x = \sum\limits_{i=1}^n \alpha_i p_i$$

So, finding $x$ reduces to finding a conjugate basis and the coefficients for $x$ in that basis.

If we let $A=H$,note that:

$${p}_k^{T} {-b}={p}_k^{T} {A}{x}$$

and because $x = \sum\limits_{i=1}^n \alpha_i p_i$, we have:
$$p^TAx = \sum\limits_{i=1}^n \alpha_i p^TA p_i$$

we can solve for $\alpha_k$:

$$\alpha_k = \frac{{p}_k^{T}{(-b)}}{{p}_k^{T} {A}{p}_k} = -\frac{\langle {p}_k, {b}\rangle}{\,\,\,\langle {p}_k,  {p}_k\rangle_{A}} = -\frac{\langle{p}_k, {b}\rangle}{\,\,\,\|{p}_k\|_{A}^2}.$$

Now, all we need are the $p_k$'s.

A nice initial guess would be the gradient at some initial point $x_1$.  So, we set $p_1 = \nabla f(x_1)$. Then set:

$$x_2 = x_1 + \alpha_1p_1$$

This should look familiar.  In fact, it is gradient descent.  For $p_2$, we want $p_1$ and $p_2$ to be conjugate (under $A$).  That just means orthogonal under the inner product induced by $A$.  We set

$$p_2 = \nabla f(x_2) - \frac{p_1^TA\nabla f(x_2)}{{p}_1^{T}{A}{p}_1} {p}_1$$

I.e. We take the gradient at $x_1$ and subtract its projection onto $p_1$.  This is the same as Gram-Schmidt orthogonalization.

The $k^{th}$ conjugate vector is:

$$p_{k} = \nabla f(x_k) - \sum\limits_{i=1}^{k-1}\frac{p_i^T A \nabla f(x_k)}{p_i^TAp_i} p_i$$

The 'trick' is that in general, we do not need all $n$ conjugate vectors. In fact, it turns out that $\nabla f(x_k) = b-Ax_k$ is conjugate to all the $p_i$ for $i=1,...,k-2$. Therefore, we need only the last term in the sum.

Convergence rate is dependent on sparsity and condition number of $A$.  Worst case is $n^2$.

### BFGS - Broyden–Fletcher–Goldfarb–Shanno

BFGS is a 'quasi' Newton method of optimization.  Such methods are variants of the Newton method, where the Hessian $H$ is replaced by some approximation.  We we wish to solve the equation:

$$B_k{p}_k = -\nabla f({x}_k)$$

for $p_k$. This gives our search direction, and the next candidate point is given by:

$$x_{k+1} = x_k + \alpha_k p_k$$.

where $\alpha_k$ is a step size.

At each step, we require that the new approximate $H$ meets the secant condition:

$$B_{k+1}(x_{k+1}-x_k) = \nabla f(x_{k+1}) -\nabla f(x_k)$$

There is a unique, rank one update that satisfies the above:

$$B_{k+1} = B_k + c_k v_kv_k^T$$

where

$$ c_k = -\frac{1}{\left(B_k(x_{k+1}-x_k) - (\nabla f(x_{k+1})-\nabla f(x_k)\right)^T (x_{k+1}-x_k) }$$

and

$$v_k = B_k(x_{k+1}-x_k) - (\nabla f(x_{k+1})-\nabla f(x_k))$$

Note that the update does NOT preserve positive definiteness if $c_k<0$.  In this case, there are several options for the rank one correction, but we will not address them here.  Instead, we will describe the BFGS method, which almost always guarantees a positive-definite correction.  Specifically:

$$B_{k+1} = B_k + b_k g_k g_k^T + c_k B_k d_k d_k^TB_k$$

where we have introduced the shorthand:

$$g_k = \nabla f(x_{k+1}) - \nabla f(x_k) \;\;\;\;\;\;\;\ \mathrm{ and }\;\;\;\;\;\;\; d_k = x_{k+1} - x_k$$

If we set:

$$b_k = \frac{1}{g_k^Td_k} \;\;\;\;\; \mathrm{ and } \;\;\;\;\; c_k = \frac{1}{d_k^TB_kd_k}$$

we satisfy the secant condition.

### Nelder-Mead Simplex

While Newton's method is considered a 'second order method' (requires the second derivative), and quasi-Newton methods are first order (require only first derivatives), Nelder-Mead is a zero-order method.  I.e. NM requires only the function itself - no derivatives.

For $f:\mathbb{R}^n\rightarrow \mathbb{R}$, the algorithm computes the values of the function on a simplex of dimension $n$, constructed from $n+1$ vertices.  For a univariate function, the simplex is a line segment.  In two dimensions, the simplex is a triangle, in 3D, a tetrahedral solid, and so on.

The algorithm begins with $n+1$ starting points and then the follwing steps are repeated until convergence:

* Compute the function at each of the points
* Sort the function values so that
$$f(x_1)\leq ...\leq f(x_{n+1})$$
* Compute the centroid $x_c$ of the n-dimensional region defined by $x_1,...,x_n$
* Reflect $x_{n+1}$ about the centroid to get $x_r$
$$x_r = x_c + \alpha (x_c - x_{n+1})$$
* Create a new simplex according to the following rules:
    - If $f(x_1)\leq f(x_r) < f(x_n)$, replace $x_{n+1}$ with $x_r$
    - If $f(x_r)<f(x_1)$, expand the simplex through $x_r$:
    $$x_e = x_c + \gamma (x_c - x_{n+1})$$
      If $f(x_e)<f(x_r)$, replace $x_{n+1}$ with $x_e$, otherwise, replace $x_{n+1}$ with $x_r$
    - If $f({x}_{r}) \geq f({x}_{n})$, compute $x_p = x_c + \rho(x_c - x_{n+1})$.  If $f({x}_{p}) < f({x}_{n+1})$, replace $x_{n+1}$ with $x_p$
    - If all else fails, replace *all* points except $x_1$ according to
    $$x_i = {x}_{1} + \sigma({x}_{i} - {x}_{1})$$

The default values of $\alpha, \gamma,\rho$ and $\sigma$ in scipy are not listed in the documentation, nor are they inputs to the function.

### Powell's Method

Powell's method is another derivative-free optimization method that is similar to conjugate-gradient.  The algorithm steps are as follows:

Begin with a point $p_0$ (an initial guess) and a set of vectors $\xi_1,...,\xi_n$, initially the standard basis of $\mathbb{R}^n$.

- Compute for $i=1,...,n$, find $\lambda_i$ that minimizes $f(p_{i-1} +\lambda_i \xi_i)$ and set $p_i = p_{i-1} + \lambda_i\xi_i$
- For $i=1,...,n-1$, replace $\xi_{i}$ with $\xi_{i+1}$ and then replace $\xi_n$ with $p_n - p_0$
- Choose $\lambda$ so that $f(p_0 + \lambda(p_n-p_0)$ is minimum and replace $p_0$ with $p_0 + \lambda(p_n-p_0)$

Essentially, the algorithm performs line searches and tries to find fruitful directions to search.

## Solvers

### Levenberg-Marquardt (Damped Least Squares)

Recall the least squares problem:

Given a set of data points $(x_i, y_i)$ where $x_i$'s are independent variables (in $\mathbb{R}^n$ and the $y_i$'s are response variables (in $\mathbb{R}$), find the parameter values of $\beta$ for the model $f(x;\beta)$ so that

$$S(\beta) = \sum\limits_{i=1}^m \left(y_i - f(x_i;\beta)\right)^2$$
is minimized.

If we were to use Newton's method, our update step would look like:

$$\beta_{k+1} = \beta_k - H^{-1}\nabla S(\beta_k)$$

Gradient descent, on the other hand, would yield:

$$\beta_{k+1} = \beta_k - \gamma\nabla S(\beta_k)$$

Levenberg-Marquardt adaptively switches between Newton's method and gradient descent.

$$\beta_{k+1} = \beta_k - (H + \lambda I)^{-1}\nabla S(\beta_k)$$

When $\lambda$ is small, the update is essentially Newton-Gauss, while for $\lambda$ large, the update is gradient descent.

### Newton-Krylov

The notion of a Krylov space comes from the Cayley-Hamilton theorem (CH).  CH states that a matrix $A$ satisfies its characteristic polynomial.  A direct corollary is that $A^{-1}$ may be written as a linear combination of powers of the matrix (where the highest power is $n-1$).

The Krylov space of order $r$ generated by an $n\times n$ matrix $A$ and an $n$-dimensional vector $b$ is given by:

$$\mathcal{K}_r(A,b) = \operatorname{span} \, \{ b, Ab, A^2b, \ldots, A^{r-1}b \}$$

These are actually the subspaces spanned by the conjugate vectors we mentioned in Newton-CG, so, technically speaking, Newton-CG is a Krylov method.

Now, the scipy.optimize newton-krylov solver is what is known as a 'Jacobian Free Newton Krylov'.  It is a very efficient algorithm for solving *large* $n\times n$ non-linear systems.  We won't go into detail of the algorithm's steps, as this is really more applicable to problems in physics and non-linear dynamics.

## GLM Estimation and IRLS

Recall generalized linear models are models with the following components:

* A linear predictor $\eta = X\beta$
* A response variable with distribution in the exponential family
* An invertible 'link' function $g$ such that
$$E(Y) = \mu = g^{-1}(\eta)$$

We may write the log-likelihood:

$$\ell(\eta) = \sum\limits_{i=1}^m (y_i \log(\eta_i) + (\eta_i - y_i)\log(1-\eta_i) $$

where $\eta_i = \eta(x_i,\beta)$.

Differentiating, we obtain:

$$\frac{\partial L}{\partial \beta} = \frac{\partial \eta}{\partial \beta}^T\frac{\partial L}{\partial \eta} = 0$$

Written slightly differently than we have in the previous sections, the Newton update to find $\beta$ would be:

$$-\frac{\partial^2 L}{\partial \beta \beta^T} \left(\beta_{k+1} -\beta_k\right) = \frac{\partial \eta}{\partial \beta}^T\frac{\partial L}{\partial \eta}$$

Now, if we compute:

$$-\frac{\partial^2 L}{\partial \beta \beta^T} = \sum \frac{\partial L}{\partial \eta_i}\frac{\partial^2 \eta_i}{\partial \beta \beta^T} - \frac{\partial \eta}{\partial \beta}^T \frac{\partial^2 L}{\partial \eta \eta^T}  \frac{\partial \eta}{\partial \beta}$$

Taking expected values on the right hand side and noting:

$$E\left(\frac{\partial L}{\partial \eta_i} \right) = 0$$

and

$$E\left(-\frac{\partial^2 L}{\partial \eta \eta^T} \right) = E\left(\frac{\partial L}{\partial \eta}\frac{\partial L}{\partial \eta}^T\right) \equiv A$$

So if we replace the Hessian in Newton's method with its expected value, we obtain:

$$\frac{\partial \eta}{\partial \beta}^TA\frac{\partial \eta}{\partial \beta}\left(\beta_{k+1} -\beta_k\right) = \frac{\partial \eta}{\partial \beta}^T\frac{\partial L}{\partial \eta} $$

Now, these actually have the form of the normal equations for a weighted least squares problem.

$$\min_{\beta_{k+1}}\left(A^{-1}\frac{\partial L}{\partial \eta} + \frac{\partial \eta}{\partial \beta}\left(\beta_{k+1} -\beta_k\right)\right)^T A \left(A^{-1}\frac{\partial L}{\partial \eta} + \frac{\partial \eta}{\partial \beta}\left(\beta_{k+1} -\beta_k\right)\right)$$

$A$ is a weight matrix, and changes with iteration - thus this technique is *iteratively reweighted least squares*.

### Constrained Optimization and Lagrange Multipliers

Often, we want to optimize a function subject to a constraint or multiple constraints.  The most common analytical technique for this is called 'Lagrange multipliers'.  The theory is based on the following:

If we wish to optimize a function $f(x,y)$ subject to the constraint $g(x,y)=c$, we are really looking for points at which the gradient of $f$ and the gradient of $g$ are in the same direction.  This amounts to:

$$\nabla_{(x,y)}f = \lambda \nabla_{(x,y)}g$$

(often, this is written with a (-) sign in front of $\lambda$). The 2-d problem above defines two equations in three unknowns.  The original constraint, $g(x,y)=c$ yields a third equation.  Additional constraints are handled by finding:

$$\nabla_{(x,y)}f = \lambda_1 \nabla_{(x,y)}g_1 + ... + \lambda_k \nabla_{(x,y)}g_k$$

The generalization to functions on $\mathbb{R}^n$ is also trivial:

$$\nabla_{x}f = \lambda \nabla_{x}g$$

```python

```


# Flow rate waveform transformation

Arjan Geers

An artery's flow rate waveform (FRW) can be characterized in many ways. Three common descriptors are the heart rate (HR), pulsatility index (PI), and time-averaged flow rate (QTA). In this notebook, a FRW is linearly transformed to generate new FRWs with specified values for these descriptors.

In [this work](http://dx.doi.org/10.1016/j.jbiomech.2013.09.033), we used FRW transformation to test how each descriptor in isolation affects the hemodynamics:

Geers AJ, Larrabide I, Morales HG, Frangi AF. Approximating hemodynamics of cerebral aneurysms with steady flow simulations. Journal of Biomechanics, 47(1):178–185, 2014.

## Preamble

```python
%matplotlib inline
```

```python
import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from ipywidgets import interact, IntSlider, FloatSlider
```

## Data

We start by reading the FRW of a healthy volunteer's internal carotid artery (ICA). This data was acquired with PC-MRA by Cebral et al. as part of [this study](http://dx.doi.org/10.1023/B:ENGI.0000007977.02652.02).

The 99 unique flow rate values (in ml/s) are uniformly distributed in time. Corresponding time values will be added during the transformation to account for the heart rate.

```python
frw = pd.read_csv(os.path.join('data', 'frw.csv'))
```

```python
pd.set_option('display.max_rows', 10)
frw
```

      flowrate (ml/s)

      0
      2.100451

      1
      2.613188

      2
      4.132279

      3
      6.301960

      4
      8.448246

      ...
      ...

      94
      2.621831

      95
      2.547649

      96
      2.466270

      97
      2.355295

      98
      2.179521

99 rows × 1 columns

The physiological ranges over which to vary the FRW descriptors are defined as follows:

```python
def qcebral(radius=2e-3):
    """Use relationship from Cebral et al. Physiol Meas, 2008,
    to return a flow rate in ml/s for a given radius in meters.

    """
    area = np.pi * radius**2
    return 48.21 * (1e4 * area)**1.84

frw_descriptors = pd.DataFrame(index=['lower',
                                      'baseline',
                                      'upper'],
                               columns=['HR (bpm)',
                                        'PI (-)',
                                        'QTA (ml/s)'])

frw_descriptors['HR (bpm)'] = [52, 68, 84]
frw_descriptors['PI (-)'] = [0.58, 0.92, 1.26]
frw_descriptors['QTA (ml/s)'] = [0.73*qcebral(), qcebral(), 1.27*qcebral()]
frw_descriptors
```

      HR (bpm)
      PI (-)
      QTA (ml/s)

      lower
      52
      0.58
      0.774473

      baseline
      68
      0.92
      1.060922

      upper
      84
      1.26
      1.347370

This dataframe corresponds to Table 1 of [our paper](http://dx.doi.org/10.1016/j.jbiomech.2013.09.033).

For HR and PI, the three values are defined as:
* baseline = mean
* lower = mean - 2 SD
* upper = mean + 2 SD

where the mean and standard deviation (SD) are taken from [Ford et al.](http://dx.doi.org/10.1088/0967-3334/26/4/013) and [Hoi et al.](http://dx.doi.org/10.1088/0967-3334/31/3/002).

Values for QTA were derived from another [paper](http://dx.doi.org/10.1088/0967-3334/29/5/005) by Cebral et al. in which they experimentally determined the relationship between the cross sectional area and time-averaged flow rate of ICAs and vertebral arteries (VAs):

\begin{equation}
  \textrm{Qcebral} = 48.21 \, A^{1.84}
\end{equation}

where A is the cross sectional area in cm$^2$ and Qcebral is in ml/s.

Qcebral was taken as baseline value for QTA. The lower and upper values were 27% below and above the baseline, respectively. The 27% variation is the average relative error between prediction and measurement, derived from Cebral et al.'s paper.

To get some actual QTA values to work with, we assume an artery with a circular cross section and a radius of 2 mm, which is about the size of an ICA.

## Transformation

Following Eq. 1 of our paper, the FRW transformation from $\textrm{Q}^0(t)$ to $\textrm{Q}(t)$ is given by:

\begin{equation}
  \textrm{Q}(t) = a \, \textrm{Q}^0(c t) + b
\end{equation}

where

\begin{equation}
  a = \frac{\textrm{QTA}}{\textrm{QTA}^0} \frac{\textrm{PI}}{\textrm{PI}^0},%
  \qquad b = \textrm{QTA} \left(1 - \frac{\textrm{PI}}{\textrm{PI}^0}\right),%
  \qquad c = \frac{\textrm{HR}}{\textrm{HR}^0}
\end{equation}

We implement this as `transform_frw`, which takes as input:
* `frw_in`, i.e. $\textrm{Q}^0(t)$
* `hr_out`, i.e. $\textrm{HR}$
* `pi_out`, i.e. $\textrm{PI}$
* `qta_out`, i.e. $\textrm{QTA}$

```python
def transform_frw(frw_in, hr_out, pi_out, qta_out):
    """Linearly transform a flow rate waveform (frw_in) to generate one with
    specified heart rate (hr_out), pulsatility index (pi_out), and
    time-averaged flow rate (qta_out).

    The input flow rate waveform is a pandas DataFrame with a single column of
    unique flow rate values uniformly distributed in time.

    The output flow rate waveform is a pandas DataFrame with a time and a flow
    rate column.

    ----------------------------------------------------------------------------
    Time-averaged values are calculated by numerical integration using the
    trapezoidal rule.

    T is period
    var[t] is variable as function of time
    n is number of trapezoid areas
    h = T/n is width of each trapezoid
    varta = time-averaged value of variable

    varta = 1/T * Integrate[var[t], {t, 0, T}]
          = 1/T * h * (0.5*var[t_0] + var[t_1] + ... + 0.5*var[t_n])
          = 1/n * (0.5*var[t_0] + var[t_1] + ... + 0.5*var[t_n])
          = 1/n * (var[t_0] + ... + var[t_n-1])
          = mean(var[t_0], ..., var[t_n-1])

    In the last step, we used the property of a periodic function that
    var[t_0] = var[t_n].
    ----------------------------------------------------------------------------

    """
    hr_in = 60
    qta_in = np.mean(frw_in)
    pi_in = (np.amax(frw_in) - np.amin(frw_in)) / qta_in

    qta_ratio = qta_out / qta_in
    pi_ratio = pi_out / pi_in
    hr_ratio = hr_out / hr_in

    a = qta_ratio * pi_ratio
    b = qta_out * (1 - pi_ratio)
    c = hr_ratio  # inverse of cardiac period

    frw_out = a * frw_in + b

    numberoftimesteps = len(frw_out)
    t = [timestep / (c * numberoftimesteps) for timestep in range(numberoftimesteps)]
    frw_out.insert(0, 'time (s)', pd.Series(t))

    return frw_out
```

## Interactive plot

To demo how the three descriptors change the shape of the FRW, we plot the transformed FRW and provide sliders to interactively adjust the descriptor values.

```python
def plot_frw(hr_out, pi_out, q_out):
    frw_out = transform_frw(frw, hr_out, pi_out, q_out)
    frw_baseline = transform_frw(frw, *frw_descriptors.ix['baseline'].tolist())
    fig, ax = plt.subplots()
    ax.plot(frw_out['time (s)'], frw_out['flowrate (ml/s)'], c='black', lw=2, )
    ax.plot(frw_baseline['time (s)'], frw_baseline['flowrate (ml/s)'], c='black', ls=':', lw=2, )
    ax.set_xlabel('Time (s)', fontsize=18)
    ax.set_ylabel('Flow rate (ml/s)', fontsize=18)
    ax.set_xlim(0, 1)
    ax.set_ylim(0, 2)
    ax.tick_params(axis='both', which='major', labelsize=14)
    plt.show()
```

```python
hr_slider = IntSlider(description='HR (bpm)',
                      value=frw_descriptors['HR (bpm)']['baseline'],
                      min=frw_descriptors['HR (bpm)']['lower'],
                      max=frw_descriptors['HR (bpm)']['upper'],
                      step=1)
pi_slider = FloatSlider(description='PI (-)',
                        value=frw_descriptors['PI (-)']['baseline'],
                        min=frw_descriptors['PI (-)']['lower'],
                        max=frw_descriptors['PI (-)']['upper'],
                        step=0.01)
q_slider = FloatSlider(description='QTA (ml/s)',
                       value=round(frw_descriptors['QTA (ml/s)']['baseline'], 2),
                       min=round(frw_descriptors['QTA (ml/s)']['lower'], 2),
                       max=round(frw_descriptors['QTA (ml/s)']['upper'], 2),
                       step=0.01)
interact(plot_frw, hr_out=hr_slider, pi_out=pi_slider, q_out=q_slider);
```

```python

```


# $H_{\rm SS}$, up to and including third post-Newtonian order

## This notebook constructs the spin-spin coupling terms in the Hamiltonian up to 3 post-Newtonian order

**Notebook Status:**  Validated

**Validation Notes:** All expressions in this notebook were transcribed twice by hand on separate occasions, and expressions were corrected as needed to ensure consistency with published PN expressions. In addition, this tutorial notebook has been confirmed to be self-consistent with its corresponding NRPy+ module, as documented [below](#code_validation). **Additional validation tests may have been performed, but are as yet, undocumented.**

## Author: Zach Etienne

### This notebook exists as the following Python module:
1. [PN_Hamiltonian_SS.py](../../edit/NRPyPN/PN_Hamiltonian_SS.py)

### This notebook & corresponding Python module depend on the following NRPy+/NRPyPN Python modules:
1. [indexedexp.py](../../edit/indexedexp.py): [**documentation+tutorial**](../Tutorial-Indexed_Expressions.ipynb)
1. [NRPyPN_shortcuts.py](../../edit/NRPyPN/NRPyPN_shortcuts.py): [**documentation**](NRPyPN_shortcuts.ipynb)

# Table of Contents
$$\label{toc}$$

1. Part 1: [$H_{S_1,S_2,{\rm 2PN}}+H_{S_1^2,{\rm 2PN}}+H_{S_2^2,{\rm 2PN}}$](#twopn), as summarized in [Buonanno, Chen, and Damour (2006)](https://arxiv.org/abs/gr-qc/0508067) (see references therein for sources)
1. Part 2: [$H_{S_1,S_2,{\rm 3PN}}$](#s1s2threepn), as derived by [Steinhoff, Hergt, and Schäfer (2008a)](https://arxiv.org/abs/0712.1716)
1. Part 3: [$H_{S_1^2,{\rm 3PN}}+H_{S_2^2,{\rm 3PN}}$](#s1squaredthreepn), as derived in [Steinhoff, Hergt, and Schäfer (2008b)](https://arxiv.org/abs/0809.2200)
1. Part 4: [Validation against second transcription and corresponding Python module](#code_validation)
1. Part 5: [LaTeX PDF output](#latex_pdf_output): $\LaTeX$ PDF Output

# Part 1: $H_{\rm SS, 2PN}$, as summarized in [Buonanno, Chen, and Damour (2006)](https://arxiv.org/abs/gr-qc/0508067) (see references therein for sources) \[Back to [top](#toc)\]
$$\label{twopn}$$

As described in the [nonspinning Hamiltonian notebook](PN-Hamiltonian-Nonspinning.ipynb), the basic physical system assumes two point particles of mass $m_1$ and $m_2$ with corresponding momentum vectors $\mathbf{P}_1$ and $\mathbf{P}_2$, and displacement vectors $\mathbf{X}_1$ and $\mathbf{X}_2$ with respect to the center of mass. Here we also consider the spin vectors of each point mass $\mathbf{S}_1$ and $\mathbf{S}_2$, respectively.

[Steinhoff, Hergt, and Schäfer (2008a)](https://arxiv.org/abs/0712.1716) and [Steinhoff, Hergt, and Schäfer (2008b)](https://arxiv.org/abs/0809.2200) adopt the additional notation
\begin{align}
\mathbf{r}_{12} &= (\mathbf{X}_1-\mathbf{X}_2)\\
r_{12} = r_{21} &= |\mathbf{r}_{12}|\\
\mathbf{n}_{12} &= \frac{\mathbf{r}_{12}}{r_{12}},
\end{align}
and when the numbers in subscripts are flipped, the particles are interchanged.

The complete $H_{\rm SO, 1.5PN}$ expression is constructed in Eqs. 2.18 and 2.19 [Buonanno, Chen, and Damour (2006)](https://arxiv.org/abs/gr-qc/0508067):
\begin{align}
\mu &= \frac{m_1 m_2}{m_1+m_2}\\
\mathbf{S_0} &= \left(1+\frac{m_2}{m_1}\right) \mathbf{S_1} + \left(1+\frac{m_1}{m_2}\right) \mathbf{S_2}\\
H_{SS,\rm 2PN} = H_{S_1,S_2,{\rm 2PN}}+H_{S_1^2,{\rm 2PN}}+H_{S_2^2,{\rm 2PN}} &= \frac{1}{2 q^3} \frac{\mu}{M} \left[3(\mathbf{S_0}\cdot\mathbf{n})^2-\mathbf{S_0}^2\right]\\
\end{align}

```python
# Step 0: Add NRPy's directory to the path
# https://stackoverflow.com/questions/16780014/import-file-from-parent-directory
import os,sys                    # Standard Python modules for multiplatform OS-level functions
nrpy_dir_path = os.path.join("..")
if nrpy_dir_path not in sys.path:
    sys.path.append(nrpy_dir_path)
import indexedexp as ixp                   # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support
from NRPyPN_shortcuts import div,dot,cross # NRPyPN: shortcuts for e.g., vector operations

# 2PN spin-spin term, from Eqs. 2.18 and 2.19 of
#      Buonanno, Chen, and Damour (2006):
#     https://arxiv.org/abs/gr-qc/0508067
def f_H_SS_2PN(m1,m2, S1U,S2U, nU, q):
    S0U = ixp.zerorank1()
    for i in range(3):
        S0U[i] = (1 + m2/m1)*S1U[i] + (1 + m1/m2)*S2U[i]
    global H_SS_2PN
    mu = m1*m2 / (m1 + m2)
    H_SS_2PN = mu/(m1 + m2) * (3*dot(S0U,nU)**2 - dot(S0U,S0U)) / (2*q**3)
```

```python
# Second version, for validation purposes only.
def f_H_SS_2PNv2(m1,m2, S1U,S2U, nU, q):
    S_0U = ixp.zerorank1()
    for i in range(3):
        S_0U[i] = (1 + m2/m1)*S1U[i] + (1 + m1/m2)*S2U[i]
    mu = m1*m2 / (m1+m2)
    global H_SS_2PNv2
    H_SS_2PNv2 = div(1,2)*mu/(m1+m2)*( 3*dot(S_0U,nU)**2 - dot(S_0U,S_0U) )/q**3
```

# Part 2: $H_{S_1,S_2,{\rm 3PN}}$, as derived by [Steinhoff, Hergt, and Schäfer (2008a)](https://arxiv.org/abs/0712.1716) \[Back to [top](#toc)\]
$$\label{s1s2threepn}$$

To reduce possibility of copying error, equations are taken directly from the arXiv LaTeX source code of Eq 2.11 in [Steinhoff, Hergt, and Schäfer (2008a)](https://arxiv.org/abs/0712.1716), and only mildly formatted to (1) improve presentation in Jupyter notebooks and (2) to ensure some degree of consistency in notation across different terms in other Hamiltonian notebooks:

\begin{align}
	H_{S_1,S_2, 3PN} &=
		\frac{1}{2 m_1 m_2 r_{1 2}^3} [
			\tfrac{3}{2} ((\mathbf{p}_1 \times \mathbf{S}_1) \cdot \mathbf{n}_{1 2}) ((\mathbf{p}_2 \times \mathbf{S}_2) \cdot \mathbf{n}_{1 2})
			+ 6 ((\mathbf{p}_2 \times \mathbf{S}_1) \cdot \mathbf{n}_{1 2}) ((\mathbf{p}_1 \times \mathbf{S}_2) \cdot \mathbf{n}_{1 2}) \\
&			- 15 (\mathbf{S}_1 \cdot \mathbf{n}_{1 2}) (\mathbf{S}_2 \cdot \mathbf{n}_{1 2}) (\mathbf{p}_1 \cdot \mathbf{n}_{1 2}) (\mathbf{p}_2 \cdot \mathbf{n}_{1 2})
			- 3 (\mathbf{S}_1 \cdot \mathbf{n}_{1 2}) (\mathbf{S}_2 \cdot \mathbf{n}_{1 2}) (\mathbf{p}_1 \cdot \mathbf{p}_2) \\
&           + 3 (\mathbf{S}_1 \cdot \mathbf{p}_2) (\mathbf{S}_2 \cdot \mathbf{n}_{1 2}) (\mathbf{p}_1 \cdot \mathbf{n}_{1 2})
			+ 3 (\mathbf{S}_2 \cdot \mathbf{p}_1) (\mathbf{S}_1 \cdot \mathbf{n}_{1 2}) (\mathbf{p}_2 \cdot \mathbf{n}_{1 2}) + 3 (\mathbf{S}_1 \cdot \mathbf{p}_1) (\mathbf{S}_2 \cdot \mathbf{n}_{1 2}) (\mathbf{p}_2 \cdot \mathbf{n}_{1 2}) \\
&			+ 3 (\mathbf{S}_2 \cdot \mathbf{p}_2) (\mathbf{S}_1 \cdot \mathbf{n}_{1 2}) (\mathbf{p}_1 \cdot \mathbf{n}_{1 2}) - \tfrac{1}{2} (\mathbf{S}_1 \cdot \mathbf{p}_2) (\mathbf{S}_2 \cdot \mathbf{p}_1)
			+ (\mathbf{S}_1 \cdot \mathbf{p}_1) (\mathbf{S}_2 \cdot \mathbf{p}_2) \\
&            - 3 (\mathbf{S}_1 \cdot \mathbf{S}_2) (\mathbf{p}_1 \cdot \mathbf{n}_{1 2}) (\mathbf{p}_2 \cdot \mathbf{n}_{1 2}) + \tfrac{1}{2} (\mathbf{S}_1 \cdot \mathbf{S}_2) (\mathbf{p}_1 \cdot \mathbf{p}_2)
		] \\
&		+ \frac{3}{2 m_1^2 r_{1 2}^3} [
			- ((\mathbf{p}_1 \times \mathbf{S}_1) \cdot \mathbf{n}_{1 2}) ((\mathbf{p}_1 \times \mathbf{S}_2) \cdot \mathbf{n}_{1 2})
			+ (\mathbf{S}_1 \cdot \mathbf{S}_2) (\mathbf{p}_1 \cdot \mathbf{n}_{1 2})^2 - (\mathbf{S}_1 \cdot \mathbf{n}_{1 2}) (\mathbf{S}_2 \cdot \mathbf{p}_1) (\mathbf{p}_1 \cdot \mathbf{n}_{1 2})
		] \\
&		+ \frac{3}{2 m_2^2 r_{1 2}^3} [
			- ((\mathbf{p}_2 \times \mathbf{S}_2) \cdot \mathbf{n}_{1 2}) ((\mathbf{p}_2 \times \mathbf{S}_1) \cdot \mathbf{n}_{1 2})
			+ (\mathbf{S}_1 \cdot \mathbf{S}_2) (\mathbf{p}_2 \cdot \mathbf{n}_{1 2})^2 - (\mathbf{S}_2 \cdot \mathbf{n}_{1 2}) (\mathbf{S}_1 \cdot \mathbf{p}_2) (\mathbf{p}_2 \cdot \mathbf{n}_{1 2})
		] \\
&		+ \frac{6 ( m_1 + m_2 )}{r_{1 2}^4} [ (\mathbf{S}_1 \cdot \mathbf{S}_2) - 2 (\mathbf{S}_1 \cdot \mathbf{n}_{1 2}) (\mathbf{S}_2 \cdot \mathbf{n}_{1 2}) ] \,,
\end{align}

```python
# 3PN spin-spin S_1,S_2 coupling term, from Eq. 2.11 of
#       Steinhoff, Hergt, and Schäfer (2008a)
#         https://arxiv.org/abs/0712.1716
def f_H_SS_S1S2_3PN(m1,m2, n12U, S1U,S2U, p1U,p2U, r12):
    global H_SS_S1S2_3PN
    H_SS_S1S2_3PN = (+div(3,2)*(dot(cross(p1U,S1U),n12U)*dot(cross(p2U,S2U),n12U))
                     +       6*(dot(cross(p2U,S1U),n12U)*dot(cross(p1U,S2U),n12U))
                     -15*dot(S1U,n12U)*dot(S2U,n12U)*dot(p1U,n12U)*dot(p2U,n12U)
                     -3*dot(S1U,n12U)*dot(S2U,n12U)*dot(p1U,p2U)
                     +3*dot(S1U,p2U)*dot(S2U,n12U)*dot(p1U,n12U)
                     +3*dot(S2U,p1U)*dot(S1U,n12U)*dot(p2U,n12U)
                     +3*dot(S1U,p1U)*dot(S2U,n12U)*dot(p2U,n12U)
                     +3*dot(S2U,p2U)*dot(S1U,n12U)*dot(p1U,n12U)
                     -div(1,2)*dot(S1U,p2U)*dot(S2U,p1U)
                     +dot(S1U,p1U)*dot(S2U,p2U)
                     -3*dot(S1U,S2U)*dot(p1U,n12U)*dot(p2U,n12U)
                     +div(1,2)*dot(S1U,S2U)*dot(p1U,p2U))/(2*m1*m2*r12**3)
    H_SS_S1S2_3PN+= (-dot(cross(p1U,S1U),n12U)*dot(cross(p1U,S2U),n12U)
                     +dot(S1U,S2U)*dot(p1U,n12U)**2
                     -dot(S1U,n12U)*dot(S2U,p1U)*dot(p1U,n12U))*3/(2*m1**2*r12**3)
    H_SS_S1S2_3PN+= (-dot(cross(p2U,S2U),n12U)*dot(cross(p2U,S1U),n12U)
                     +dot(S1U,S2U)*dot(p2U,n12U)**2
                     -dot(S2U,n12U)*dot(S1U,p1U)*dot(p2U,n12U))*3/(2*m2**2*r12**3)
    H_SS_S1S2_3PN+= (+dot(S1U,S2U)-2*dot(S1U,n12U)*dot(S2U,n12U))*6*(m1+m2)/r12**4
```

```python
# Second version, for validation purposes only.
def f_H_SS_S1S2_3PNv2(m1,m2, n12U, S1U,S2U, p1U,p2U, q):
    def SHS2008a_HS1S2_3PNv2_pt1(m1,m2, n12U, S1U,S2U, p1U,p2U, q):
        Hpt1 = ( +div(3,2)*(dot(cross(p1U,S1U),n12U)*dot(cross(p2U,S2U),n12U)) # line 1
                 +6 *dot(cross(p2U,S1U),n12U)*dot(cross(p1U,S2U),n12U)         # line 1
                 -15*dot(S1U,n12U)*dot(S2U,n12U)*dot(p1U,n12U)*dot(p2U,n12U)   # line 2
                 -3*dot(S1U,n12U)*dot(S2U,n12U)*dot(p1U,p2U)                   # line 2
                 +3*dot(S1U,p2U)*dot(S2U,n12U)*dot(p1U,n12U) # line 3
                 +3*dot(S2U,p1U)*dot(S1U,n12U)*dot(p2U,n12U) # line 3
                 +3*dot(S1U,p1U)*dot(S2U,n12U)*dot(p2U,n12U) # line 3
                 +3*dot(S2U,p2U)*dot(S1U,n12U)*dot(p1U,n12U) # line 4
                 -div(1,2)*dot(S1U,p2U)*dot(S2U,p1U)         # line 4
                 +dot(S1U,p1U)*dot(S2U,p2U)                  # line 4
                 -3*dot(S1U,S2U)*dot(p1U,n12U)*dot(p2U,n12U)            # line 5
                 +div(1,2)*dot(S1U,S2U)*dot(p1U,p2U) )/(2*m1*m2*q**3)   # line 5
        return Hpt1
    def SHS2008a_HS1S2_3PNv2_pt2(m1,m2, n12U, S1U,S2U, p1U,p2U, q):
        Hpt2 = ( -dot(cross(p1U,S1U),n12U)*dot(cross(p1U,S2U),n12U)                # line 6
                 +dot(S1U,S2U)*dot(p1U,n12U)**2                                    # line 6
                 -dot(S1U,n12U)*dot(S2U,p1U)*dot(p1U,n12U) )*div(3,2)/(m1**2*q**3) # line 6
        return Hpt2
    def SHS2008a_HS1S2_3PNv2_pt3(m1,m2, n12U, S1U,S2U, p1U,p2U, q):
        Hpt3 = ( -dot(cross(p2U,S2U),n12U)*dot(cross(p2U,S1U),n12U)                # line 7
                 +dot(S1U,S2U)*dot(p2U,n12U)**2                                    # line 7
                 -dot(S2U,n12U)*dot(S1U,p1U)*dot(p2U,n12U) )*div(3,2)/(m2**2*q**3) # line 7
        return Hpt3
    def SHS2008a_HS1S2_3PNv2_pt4(m1,m2, n12U, S1U,S2U, p1U,p2U, q):
        Hpt4 = ( dot(S1U,S2U) - 2*dot(S1U,n12U)*dot(S2U,n12U) ) * 6*(m1+m2)/q**4   # line 8
        return Hpt4
    global H_SS_S1S2_3PNv2
    H_SS_S1S2_3PNv2 = ( +SHS2008a_HS1S2_3PNv2_pt1(m1,m2, n12U, S1U,S2U, p1U,p2U, q)
                        +SHS2008a_HS1S2_3PNv2_pt2(m1,m2, n12U, S1U,S2U, p1U,p2U, q)
                        +SHS2008a_HS1S2_3PNv2_pt3(m1,m2, n12U, S1U,S2U, p1U,p2U, q)
                        +SHS2008a_HS1S2_3PNv2_pt4(m1,m2, n12U, S1U,S2U, p1U,p2U, q) )
```

# Part 3: $H_{S_1^2,{\rm 3PN}}+H_{S_2^2,{\rm 3PN}}$, as derived in [Steinhoff, Hergt, and Schäfer (2008b)](https://arxiv.org/abs/0809.2200) \[Back to [top](#toc)\]
$$\label{s1squaredthreepn}$$

To reduce possibility of copying error, equations are taken directly from the arXiv LaTeX source code of Eq 9 in [Steinhoff, Hergt, and Schäfer (2008b)](https://arxiv.org/abs/0809.2200), and only mildly formatted to (1) improve presentation in Jupyter notebooks and (2) to ensure some degree of consistency in notation across different terms in other Hamiltonian notebooks:

\begin{align}
H_{S_1^2,{\rm 3PN}}+H_{S_2^2,{\rm 3PN}}&=
\frac{1}{r_{12}^3}\bigg[
	\frac{m_{2}}{4m_{1}^3}\left(\mathbf{P}_{1}\cdot\mathbf{S}_{1}\right)^2
	+\frac{3m_{2}}{8m_{1}^3}\left(\mathbf{P}_{1}\cdot\mathbf{n}_{12}\right)^{2}\mathbf{S}_{1}^{2}
	-\frac{3m_{2}}{8m_{1}^3}\mathbf{P}_{1}^{2}\left(\mathbf{S}_{1}\cdot\mathbf{n}_{12}\right)^2 \\
&    -\frac{3m_{2}}{4m_{1}^3}\left(\mathbf{P}_{1}\cdot\mathbf{n}_{12}\right)\left(\mathbf{S}_{1}\cdot\mathbf{n}_{12}\right)\left(\mathbf{P}_{1}\cdot\mathbf{S}_{1}\right)
	-\frac{3}{4m_{1}m_{2}}\mathbf{P}_{2}^{2}\mathbf{S}_{1}^{2}\\
&	+\frac{9}{4m_{1}m_{2}}\mathbf{P}_{2}^{2}\left(\mathbf{S}_{1}\cdot\mathbf{n}_{12}\right)^2
	+\frac{3}{4m_{1}^2}\left(\mathbf{P}_{1}\cdot\mathbf{P}_{2}\right)\mathbf{S}_{1}^2
	-\frac{9}{4m_{1}^2}\left(\mathbf{P}_{1}\cdot\mathbf{P}_{2}\right)\left(\mathbf{S}_{1}\cdot\mathbf{n}_{12}\right)^2 \\
&	-\frac{3}{2m_{1}^2}\left(\mathbf{P}_{1}\cdot\mathbf{n}_{12}\right)\left(\mathbf{P}_{2}\cdot\mathbf{S}_{1}\right)\left(\mathbf{S}_{1}\cdot\mathbf{n}_{12}\right)
	+\frac{3}{m_{1}^2}\left(\mathbf{P}_{2}\cdot\mathbf{n}_{12}\right)\left(\mathbf{P}_{1}\cdot\mathbf{S}_{1}\right)\left(\mathbf{S}_{1}\cdot\mathbf{n}_{12}\right) \\
&	+\frac{3}{4m_{1}^2}\left(\mathbf{P}_{1}\cdot\mathbf{n}_{12}\right)\left(\mathbf{P}_{2}\cdot\mathbf{n}_{12}\right)\mathbf{S}_{1}^2
	-\frac{15}{4m_{1}^2}\left(\mathbf{P}_{1}\cdot\mathbf{n}_{12}\right)\left(\mathbf{P}_{2}\cdot\mathbf{n}_{12}\right)\left(\mathbf{S}_{1}\cdot\mathbf{n}_{12}\right)^2\bigg] \\
& - \frac{G^2 m_2}{r_{12}^4} \bigg[
	\frac{9}{2} (\mathbf{S}_1 \cdot \mathbf{n}_{12})^2 - \frac{5}{2} \mathbf{S}_1^2
	+ \frac{7 m_2}{m_1} (\mathbf{S}_1 \cdot \mathbf{n}_{12})^2
	- \frac{3 m_2}{m_1} \mathbf{S}_1^2 \bigg]
 + (1\leftrightarrow2)\,.
\end{align}

```python
# 3PN spin-orbit coupling term, from Eq. 9 of
#    Steinhoff, Hergt, and Schäfer (2008b)
#       https://arxiv.org/abs/0809.2200
def f_H_SS_S1sq_S2sq_3PN(m1,m2, n12U,n21U, S1U,S2U, p1U,p2U, r12):
    def f_H_SS_particle(m1,m2, n12U, S1U,S2U, p1U,p2U, r12):
        H_SS_S1sq_S2sq_3PN_particle = (
            +  m2/(4*m1**3)*dot(p1U,S1U)**2
            +3*m2/(8*m1**3)*dot(p1U,n12U)**2*dot(S1U,S1U)
            -3*m2/(8*m1**3)*dot(p1U,p1U)*dot(S1U,n12U)**2
            -3*m2/(4*m1**3)*dot(p1U,n12U)*dot(S1U,n12U)*dot(p1U,S1U)
            -3/(4*m1*m2)*dot(p2U,p2U)*dot(S1U,S1U)
            +9/(4*m1*m2)*dot(p2U,p2U)*dot(S1U,n12U)**2
            +3/(4*m1**2)*dot(p1U,p2U)*dot(S1U,S1U)
            -9/(4*m1**2)*dot(p1U,p2U)*dot(S1U,n12U)**2
            -3/(2*m1**2)*dot(p1U,n12U)*dot(p2U,S1U)*dot(S1U,n12U)
            +3/(m1**2)  *dot(p2U,n12U)*dot(p1U,S1U)*dot(S1U,n12U)
            +3/(4*m1**2)*dot(p1U,n12U)*dot(p2U,n12U)*dot(S1U,S1U)
            -15/(4*m1**2)*dot(p1U,n12U)*dot(p2U,n12U)*dot(S1U,n12U)**2)/r12**3
        H_SS_S1sq_S2sq_3PN_particle+= -(+div(9,2)*dot(S1U,n12U)**2
                                         -div(5,2)*dot(S1U,S1U)
                                         +7*m2/m1*dot(S1U,n12U)**2
                                         -3*m2/m1*dot(S1U,S1U))*m2/r12**4
        return H_SS_S1sq_S2sq_3PN_particle
    global H_SS_S1sq_S2sq_3PN
    H_SS_S1sq_S2sq_3PN = (+f_H_SS_particle(m1,m2, n12U, S1U,S2U, p1U,p2U, r12)
                          +f_H_SS_particle(m2,m1, n21U, S2U,S1U, p2U,p1U, r12))
```

```python
# Second version, transcribed on a separate occasion. For validation purposes only.
def f_H_SS_S1sq_S2sq_3PNv2(m1,m2, n12U,n21U, S1U,S2U, p1U,p2U, q):
    def SHS2008b_HSsq_3PNv2_pt(m1,m2, n12U, S1U, p1U,p2U, q):
        H = ( +div(1,4)*m2/m1**3*dot(p1U,S1U)**2
              +div(3,8)*m2/m1**3*dot(p1U,n12U)**2*dot(S1U,S1U) # line 1
              -div(3,8)*m2/m1**3*dot(p1U,p1U)*dot(S1U,n12U)**2                                     # line 1
              -div(3,4)*m2/m1**3*dot(p1U,n12U)*dot(S1U,n12U)*dot(p1U,S1U) # line 2
              -div(3,4)/(m1*m2) *dot(p2U,p2U)*dot(S1U,S1U)                # line 2
              +div(9,4)/(m1*m2) *dot(p2U,p2U)*dot(S1U,n12U)**2 # line 3
              +div(3,4)/m1**2   *dot(p1U,p2U)*dot(S1U,S1U)     # line 3
              -div(9,4)/m1**2   *dot(p1U,p2U)*dot(S1U,n12U)**2 # line 3
              -div(3,2)/m1**2   *dot(p1U,n12U)*dot(p2U,S1U)*dot(S1U,n12U) # line 4
              +       3/m1**2   *dot(p2U,n12U)*dot(p1U,S1U)*dot(S1U,n12U) # line 4
              +div(3,4)/m1**2   *dot(p1U,n12U)*dot(p2U,n12U)*dot(S1U,S1U)            # line 5
              -div(15,4)/m1**2  *dot(p1U,n12U)*dot(p2U,n12U)*dot(S1U,n12U)**2 )/q**3 \
            -( +div(9,2)*dot(S1U,n12U)**2          # line 6
               -div(5,2)*dot(S1U,S1U)              # line 6
               +       7*m2/m1*dot(S1U,n12U)**2    # line 6
               -       3*m2/m1*dot(S1U,S1U) )*m2/q**4 # line 6
        return H

    global H_SS_S1sq_S2sq_3PNv2
    H_SS_S1sq_S2sq_3PNv2 = ( +SHS2008b_HSsq_3PNv2_pt(m1,m2, n12U, S1U, p1U,p2U, q)   # S_1^2 term
                             +SHS2008b_HSsq_3PNv2_pt(m2,m1, n21U, S2U, p2U,p1U, q) ) # S_2^2 term
```

# Part 4: Validation against second transcription and corresponding Python module \[Back to [top](#toc)\]
$$\label{code_validation}$$

As a code validation check, we verify agreement between
* the SymPy expressions transcribed from the cited published work on two separate occasions, and
* the SymPy expressions generated in this notebook, and the corresponding Python module.

```python
import sympy as sp               # SymPy: The Python computer algebra package upon which NRPy+ depends
from NRPyPN_shortcuts import m1,m2, nU,n12U,n21U, S1U,S2U, p1U,p2U, q # NRPyPN: Import needed input variables

f_H_SS_2PN(m1,m2, S1U,S2U, nU, q)
f_H_SS_S1S2_3PN(     m1,m2, n12U,      S1U,S2U, p1U,p2U, q)
f_H_SS_S1sq_S2sq_3PN(m1,m2, n12U,n21U, S1U,S2U, p1U,p2U, q)

def error(varname):
    print("ERROR: When comparing Python module & notebook, "+varname+" was found not to match.")
    sys.exit(1)

# Validation against second transcription of the expressions:
f_H_SS_2PNv2(m1,m2, S1U,S2U, nU, q)
f_H_SS_S1S2_3PNv2(     m1,m2, n12U,      S1U,S2U, p1U,p2U, q)
f_H_SS_S1sq_S2sq_3PNv2(m1,m2, n12U,n21U, S1U,S2U, p1U,p2U, q)

if sp.simplify(H_SS_2PN           - H_SS_2PNv2)           != 0: error("H_SS_2PNv2")
if sp.simplify(H_SS_S1S2_3PN      - H_SS_S1S2_3PNv2)      != 0: error("H_SS_S1S2_3PNv2")
if sp.simplify(H_SS_S1sq_S2sq_3PN - H_SS_S1sq_S2sq_3PNv2) != 0: error("H_SS_S1sq_S2sq_3PNv2")

# Validation against corresponding Python module:
import PN_Hamiltonian_SS as HSS
HSS.f_H_SS_2PN(m1,m2, S1U,S2U, nU, q)
HSS.f_H_SS_S1S2_3PN(     m1,m2, n12U,      S1U,S2U, p1U,p2U, q)
HSS.f_H_SS_S1sq_S2sq_3PN(m1,m2, n12U,n21U, S1U,S2U, p1U,p2U, q)

if sp.simplify(H_SS_2PN           - HSS.H_SS_2PN)           != 0: error("H_SS_2PN")
if sp.simplify(H_SS_S1S2_3PN      - HSS.H_SS_S1S2_3PN)      != 0: error("H_SS_S1S2_3PN")
if sp.simplify(H_SS_S1sq_S2sq_3PN - HSS.H_SS_S1sq_S2sq_3PN) != 0: error("H_SS_S1sq_S2sq_3PN")
print("ALL TESTS PASS")
```

    ALL TESTS PASS

# Part 5: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[PN-Hamiltonian-Spin-Spin.pdf](PN-Hamiltonian-Spin-Spin.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
import os,sys                    # Standard Python modules for multiplatform OS-level functions
nrpy_dir_path = os.path.join("..")
if nrpy_dir_path not in sys.path:
    sys.path.append(nrpy_dir_path)
import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("PN-Hamiltonian-Spin-Spin",location_of_template_file=os.path.join(".."))
```

    Created PN-Hamiltonian-Spin-Spin.tex, and compiled LaTeX file to PDF file
        PN-Hamiltonian-Spin-Spin.pdf


# Eq. (4.17) and (4.18)

Equation (4.17) is

\begin{equation}
    \overline{\phi}^{(k,\alpha,\beta)}_m = \gamma^{(k,\alpha,\beta)}_{m} \phi^{(k/2, \alpha+{k}/{2}, \beta+{k}/{2})}_{m}, \label{eq:phiover}
\end{equation}

where

\begin{equation}
\gamma^{(k,\alpha,\beta)}_{n} = \frac{\psi^{(k/2,\alpha,\beta)}_{n+k} g^{(\alpha, \beta)}_{n+k}h^{({k}/{2},\alpha+{k}/{2}, \beta+{k}/{2})}_{n+{k}/{2}}}{g^{(\alpha+{k}/{2},\beta+{k}/{2})}_{n+{k}/{2}}h^{(k,\alpha,\beta)}_{n+k}}.
\end{equation}

Equation (4.17) is derived using

\begin{equation}
    \partial^k P^{(\alpha, \beta)}_n = \psi^{(k,\alpha,\beta)}_{n} P^{(\alpha+k,\beta+k)}_{n-k},
    \label{eq:derP}
\end{equation}

where $\psi^{(k,\alpha,\beta)}_{n}$ is given in Eq. (2.8). Using

\begin{equation}
    Q^{(\alpha,\beta)}_n(x) = g_n^{(\alpha,\beta)} P^{(\alpha,\beta)}_n(x),
    \label{eq:Qspec}
\end{equation}

we get

\begin{equation}
    \partial^k Q^{(\alpha, \beta)}_n = \frac{g^{(\alpha,\beta)}_n }{g^{(\alpha+k,\beta+k)}_{n-k}} \psi^{(k,\alpha,\beta)}_{n} Q^{(\alpha+k,\beta+k)}_{n-k}.
    \label{eq:derQ}
\end{equation}

The same equation for $k/2$ is

\begin{equation}
    \partial^{k/2} Q^{(\alpha, \beta)}_{n} = \frac{g^{(\alpha,\beta)}_n }{g^{(\alpha+k/2,\beta+k/2)}_{n-k/2}} \psi^{(k/2,\alpha,\beta)}_{n} Q^{(\alpha+k/2,\beta+k/2)}_{n-k/2},
    \label{eq:derQ2}
\end{equation}

Take $\partial^{k/2}$ of this equation to obtain

\begin{equation}
    \partial^{k} Q^{(\alpha, \beta)}_{n} = \frac{g^{(\alpha,\beta)}_n }{g^{(\alpha+k/2,\beta+k/2)}_{n-k/2}} \psi^{(k/2,\alpha,\beta)}_{n} \partial^{k/2}Q^{(\alpha+k/2,\beta+k/2)}_{n-k/2},
    \label{eq:derQ3}
\end{equation}

and shift the index to $n+k$ to obtain

\begin{equation}
    \partial^{k} Q^{(\alpha, \beta)}_{n+k} = \frac{g^{(\alpha,\beta)}_{n+k} }{g^{(\alpha+k/2,\beta+k/2)}_{n+k/2}} \psi^{(k/2,\alpha,\beta)}_{n+k} \partial^{k/2}Q^{(\alpha+k/2,\beta+k/2)}_{n+k/2},
    \label{eq:derQ4}
\end{equation}

Now we want to find $\gamma^{(k,\alpha,\beta)}$ such that

\begin{equation}
    \overline{\phi}^{(k,\alpha,\beta)}_m = \gamma^{(k,\alpha,\beta)}_{m} \phi^{(k/2, \alpha+{k}/{2}, \beta+{k}/{2})}_{m},
\end{equation}

where

\begin{equation}
\overline{\phi}^{(k,\alpha,\beta)}_m = \frac{(1-x^2)^{k/2}}{h^{(k,\alpha,\beta)}_{n+k}}\partial^k Q^{(\alpha,\beta)}_{n+k}
\end{equation}

and

\begin{equation}
\phi^{(k/2,\alpha+k/2,\beta+k/2)}_n = \frac{(1-x^2)^{k/2}}{h^{(k/2,\alpha+k/2,\beta+k/2)}_{n+k/2}}\partial^{k/2} Q^{(\alpha+k/2,\beta+k/2)}_{n+k/2}
\end{equation}

We get

\begin{equation}
\gamma^{(k,\alpha,\beta)}_n = \frac{\frac{(1-x^2)^{k/2}}{h^{(k,\alpha,\beta)}_{n+k}}\partial^k Q^{(\alpha,\beta)}_{n+k}}{\frac{(1-x^2)^{k/2}}{h^{(k/2,\alpha+k/2,\beta+k/2)}_{n+k/2}}\partial^{k/2} Q^{(\alpha,\beta)}_{n+k/2}}
\end{equation}

The $(1-x^2)^{k/2}$ cancel out

\begin{equation}
\gamma^{(k,\alpha,\beta)}_n = \frac{{h^{(k/2,\alpha+k/2,\beta+k/2)}_{n+k/2}}}{h^{(k,\alpha,\beta)}_{n+k}} \frac{ \partial^k Q^{(\alpha,\beta)}_{n+k} }{\partial^{k/2}Q^{(\alpha+k/2,\beta+k/2)}_{n+k/2}}
\end{equation}

and we insert for $\frac{\partial^k Q^{(\alpha,\beta)}_{n+k}}{\partial^{k/2}Q^{(\alpha+k/2,\beta+k/2)}_{n+k/2}}$ to get

\begin{equation}
\gamma^{(k,\alpha,\beta)}_{n} = \frac{\psi^{(k/2,\alpha,\beta)}_{n+k} g^{(\alpha, \beta)}_{n+k}h^{({k}/{2},\alpha+{k}/{2}, \beta+{k}/{2})}_{n+{k}/{2}}}{g^{(\alpha+{k}/{2},\beta+{k}/{2})}_{n+{k}/{2}}h^{(k,\alpha,\beta)}_{n+k}}.
\end{equation}

Note that the two scaling functions $g^{(\alpha, \beta)}_{n+k}$ and $g^{(\alpha+{k}/{2},\beta+{k}/{2})}_{n+{k}/{2}}$ represents the specialized polynomials $Q^{(\alpha,\beta)}_{n+k}$ and $Q^{(\alpha+k/2,\beta+k/2)}_{n+k/2}$, that not necessarily use the same scaling function $g$. For example, Chebyshev polynomials of first and second kind use slightly different scaling functions.

```python
from shenfun.jacobi.recursions import half, alfa, beta, psi, cn, un, h, n, sp

def gamma(k, alf, bet, gna, gnb):
    """Return Eq. (12)

    Parameters
    ----------
    k : int
    alf : number
    bet : number
    gna : The scaling function of the trial space
    gnb : The scaling function of the test space
    """
    return sp.simplify(psi(alf, bet, n + k, k//2)
                       *gna(alf, bet, n + k)
                       *h(alf + k//2, bet + k//2, n + k//2, k//2, gnb)
                       /(gnb(alf + k//2, bet + k//2, n + k//2) * h(alf, bet, n + k, k, gna)))

```

Let the trial space be based on Chebyshev polynomials of the first kind, whereas the test space makes use of Chebyshev polynomials of the second kind. For these to bases the scaling function is

$$
g_n^{(-1/2,-1/2)} = \frac{1}{P^{(-1/2,-1/2)}_{n}(1)}
$$

and

$$
g_n^{(1/2,1/2)} = \frac{n+1}{P^{(1/2,1/2)}_{n}(1)}
$$

respectively. These two functions are imported under names `cn` and `un`, respectively. See [jacobi.recursions.py](https://github.com/spectralDNS/shenfun/blob/master/shenfun/jacobi/recursions.py).

We get $\gamma^{(2,-1/2,-1/2)}_n$ to be

```python
gam = gamma(2, -half, -half, cn, un)
gam
```

$\displaystyle \frac{1}{n + 2}$