# Regression in Python
***
This is a very quick run-through of some basic statistical concepts, adapted from [Lab 4 in Harvard's CS109](https://github.com/cs109/2015lab4) course. Please feel free to try the original lab if you're feeling ambitious :-) The CS109 git repository also has the solutions if you're stuck.

* Linear Regression Models
* Prediction using linear regression

Linear regression is used to model and predict continuous outcomes with normal random errors. There are nearly an infinite number of different types of regression models and each regression model is typically defined by the distribution of the prediction errors (called "residuals") of the type of data. Logistic regression is used to model binary outcomes whereas Poisson regression is used to predict counts. In this exercise, we'll see some examples of linear regression as well as Train-test splits.

The packages we'll cover are: `statsmodels`, `seaborn`, and `scikit-learn`. While we don't explicitly teach `statsmodels` and `seaborn` in the Springboard workshop, those are great libraries to know.
***

***

```python
# special IPython command to prepare the notebook for matplotlib and other libraries
%matplotlib inline

import numpy as np
import pandas as pd
import scipy.stats as stats
import matplotlib.pyplot as plt
import sklearn

import seaborn as sns

# special matplotlib argument for improved plots
from matplotlib import rcParams
sns.set_style("whitegrid")
sns.set_context("poster")

```

***
# Part 1: Introduction to Linear Regression
### Purpose of linear regression
***

 Given a dataset containing predictor variables $X$ and outcome/response variable $Y$, linear regression can be used to:

 Build a predictive model to predict future values of $\hat{Y}$, using new data $X^*$ where $Y$ is unknown.

 Model the strength of the relationship between each independent variable $X_i$ and $Y$

 Many times, only a subset of independent variables $X_i$ will have a linear relationship with $Y$

 Need to figure out which $X_i$ contributes most information to predict $Y$

It is in many cases, the first pass prediction algorithm for continuous outcomes.

### A Brief Mathematical Recap
***

[Linear Regression](http://en.wikipedia.org/wiki/Linear_regression) is a method to model the relationship between a set of independent variables $X$ (also knowns as explanatory variables, features, predictors) and a dependent variable $Y$.  This method assumes the relationship between each predictor $X$ is **linearly** related to the dependent variable $Y$. The most basic linear regression model contains one independent variable $X$, we'll call this the simple model.

$$ Y = \beta_0 + \beta_1 X + \epsilon$$

where $\epsilon$ is considered as an unobservable random variable that adds noise to the linear relationship. In linear regression, $\epsilon$ is assumed to be normally distributed with a mean of 0. In other words, what this means is that on average, if we know $Y$, a roughly equal number of predictions $\hat{Y}$ will be above $Y$ and others will be below $Y$. That is, on average, the error is zero. The residuals, $\epsilon$ are also assumed to be "i.i.d.": independently and identically distributed. Independence means that the residuals are not correlated -- the residual from one prediction has no effect on the residual from another prediction. Correlated errors are common in time series analysis and spatial analyses.

* $\beta_0$ is the intercept of the linear model and represents the average of $Y$ when all independent variables $X$ are set to 0.

* $\beta_1$ is the slope of the line associated with the regression model and represents the average effect of a one-unit increase in $X$ on $Y$.

* Back to the simple model. The model in linear regression is the *conditional mean* of $Y$ given the values in $X$ is expressed a linear function.

$$ y = f(x) = E(Y | X = x)$$

http://www.learner.org/courses/againstallodds/about/glossary.html

* The goal is to estimate the coefficients (e.g. $\beta_0$ and $\beta_1$). We represent the estimates of the coefficients with a "hat" on top of the letter.

$$ \hat{\beta}_0, \hat{\beta}_1 $$

* Once we estimate the coefficients $\hat{\beta}_0$ and $\hat{\beta}_1$, we can use these to predict new values of $Y$ given new data $X$.

$$\hat{y} = \hat{\beta}_0 + \hat{\beta}_1 x_1$$

* Multiple linear regression is when you have more than one independent variable and the estimation involves matrices
    * $X_1$, $X_2$, $X_3$, $\ldots$

* How do you estimate the coefficients?
    * There are many ways to fit a linear regression model
    * The method called **least squares** is the most common methods
    * We will discuss least squares

$$ Y = \beta_0 + \beta_1 X_1 + \ldots + \beta_p X_p + \epsilon$$

### Estimating $\hat\beta$: Least squares
***
[Least squares](http://en.wikipedia.org/wiki/Least_squares) is a method that can estimate the coefficients of a linear model by minimizing the squared residuals:

$$ \mathscr{L} = \sum_{i=1}^N \epsilon_i = \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2  = \sum_{i=1}^N \left(y_i - \left(\beta_0 + \beta_1 x_i\right)\right)^2 $$

where $N$ is the number of observations and $\epsilon$ represents a residual or error, ACTUAL - PREDICTED.

#### Estimating the intercept $\hat{\beta_0}$ for the simple linear model

We want to minimize the squared residuals and solve for $\hat{\beta_0}$ so we take the partial derivative of $\mathscr{L}$ with respect to $\hat{\beta_0}$

$
\begin{align}
\frac{\partial \mathscr{L}}{\partial \hat{\beta_0}} &= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \epsilon^2 \\
&= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2 \\
&= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right)^2 \\
&= -2 \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right) \hspace{25mm} \mbox{(by chain rule)} \\
&= -2 \sum_{i=1}^N y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i \\
&= -2 \left[ \left( \sum_{i=1}^N y_i \right) - n \hat{\beta_0} - \hat{\beta}_1 \left( \sum_{i=1}^N x_i
\right) \right] \\
& 2 \left[ n \hat{\beta}_0 + \hat{\beta}_1 \sum_{i=1}^N x_i - \sum_{i=1}^N y_i \right] = 0 \hspace{20mm} \mbox{(Set equal to 0 and solve for $\hat{\beta}_0$)} \\
& n \hat{\beta}_0 + \hat{\beta}_1 \sum_{i=1}^N x_i - \sum{i=1}^N y_i = 0 \\
& n \hat{\beta}_0 = \sum_{i=1}^N y_i - \hat{\beta}_1 \sum_{i=1}^N x_i \\
& \hat{\beta}_0 = \frac{\sum_{i=1}^N y_i - \hat{\beta}_1 \sum_{i=1}^N x_i}{n} \\
& \hat{\beta}_0 = \frac{\sum_{i=1}^N y_i}{n} - \hat{\beta}_1 \frac{\sum_{i=1}^N x_i}{n} \\
& \boxed{\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}}
\end{align}
$

Using this new information, we can compute the estimate for $\hat{\beta}_1$ by taking the partial derivative of $\mathscr{L}$ with respect to $\hat{\beta}_1$.

$
\begin{align}
\frac{\partial \mathscr{L}}{\partial \hat{\beta_1}} &= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \epsilon^2 \\
&= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2 \\
&= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right)^2 \\
&= 2 \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right) \left( -x_i \right) \hspace{25mm}\mbox{(by chain rule)} \\
&= -2 \sum_{i=1}^N x_i \left( y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i \right) \\
&= -2 \sum_{i=1}^N x_i y_i - \hat{\beta}_0 x_i - \hat{\beta}_1 x_i^2 \\
&= -2 \sum_{i=1}^N x_i y_i - \left( \bar{y} - \hat{\beta}_1 \bar{x} \right) x_i - \hat{\beta}_1 x_i^2 \\
&= -2 \sum_{i=1}^N x_i y_i - \bar{y}x_i + \hat{\beta}_1\bar{x}x_i - \hat{\beta}_1 x_i^2 \\
&= -2 \left[ \sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i + \hat{\beta}_1\bar{x} - \hat{\beta}_1 x_i^2 \right] \\
&= -2 \left[ \hat{\beta}_1 \left\{ \bar{x} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i^2 \right\} + \left\{ \sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i \right\}\right] \\
& 2 \left[ \hat{\beta}_1 \left\{ \sum_{i=1}^N x_i^2 - \bar{x} \sum_{i=1}^N x_i \right\} + \left\{ \bar{y} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i y_i \right\} \right] = 0 \\
& \hat{\beta}_1 = \frac{-\left( \bar{y} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i y_i \right)}{\sum_{i=1}^N x_i^2 - \bar{x}\sum_{i=1}^N x_i} \\
&= \frac{\sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i}{\sum_{i=1}^N x_i^2 - \bar{x} \sum_{i=1}^N x_i} \\
& \boxed{\hat{\beta}_1 = \frac{\sum_{i=1}^N x_i y_i - \bar{x}\bar{y}n}{\sum_{i=1}^N x_i^2 - n \bar{x}^2}}
\end{align}
$

The solution can be written in compact matrix notation as

$$\hat\beta =  (X^T X)^{-1}X^T Y$$

We wanted to show you this in case you remember linear algebra, in order for this solution to exist we need $X^T X$ to be invertible. Of course this requires a few extra assumptions, $X$ must be full rank so that $X^T X$ is invertible, etc. Basically, $X^T X$ is full rank if all rows and columns are linearly independent. This has a loose relationship to variables and observations being independent respective. **This is important for us because this means that having redundant features in our regression models will lead to poorly fitting (and unstable) models.** We'll see an implementation of this in the extra linear regression example.

***
# Part 2: Exploratory Data Analysis for Linear Relationships

The [Boston Housing data set](https://archive.ics.uci.edu/ml/datasets/Housing) contains information about the housing values in suburbs of Boston.  This dataset was originally taken from the StatLib library which is maintained at Carnegie Mellon University and is now available on the UCI Machine Learning Repository.

## Load the Boston Housing data set from `sklearn`
***

This data set is available in the [sklearn](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html#sklearn.datasets.load_boston) python module which is how we will access it today.

```python
from sklearn.datasets import load_boston
import pandas as pd

boston = load_boston()
```

```python
type(boston)
```

    sklearn.utils.Bunch

```python
boston.keys()
```

    ['data', 'feature_names', 'DESCR', 'target']

```python
boston.data.shape
```

    (506L, 13L)

```python
# Print column names
print(boston.feature_names)
```

    ['CRIM' 'ZN' 'INDUS' 'CHAS' 'NOX' 'RM' 'AGE' 'DIS' 'RAD' 'TAX' 'PTRATIO'
     'B' 'LSTAT']

```python
# Print description of Boston housing data set
print(boston.DESCR)
```

    Boston House Prices dataset
    ===========================

    Notes
    ------
    Data Set Characteristics:

        :Number of Instances: 506

        :Number of Attributes: 13 numeric/categorical predictive

        :Median Value (attribute 14) is usually the target

        :Attribute Information (in order):
            - CRIM     per capita crime rate by town
            - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.
            - INDUS    proportion of non-retail business acres per town
            - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
            - NOX      nitric oxides concentration (parts per 10 million)
            - RM       average number of rooms per dwelling
            - AGE      proportion of owner-occupied units built prior to 1940
            - DIS      weighted distances to five Boston employment centres
            - RAD      index of accessibility to radial highways
            - TAX      full-value property-tax rate per $10,000
            - PTRATIO  pupil-teacher ratio by town
            - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town
            - LSTAT    % lower status of the population
            - MEDV     Median value of owner-occupied homes in $1000's

        :Missing Attribute Values: None

        :Creator: Harrison, D. and Rubinfeld, D.L.

    This is a copy of UCI ML housing dataset.
    http://archive.ics.uci.edu/ml/datasets/Housing

    This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.

    The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic
    prices and the demand for clean air', J. Environ. Economics & Management,
    vol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics
    ...', Wiley, 1980.   N.B. Various transformations are used in the table on
    pages 244-261 of the latter.

    The Boston house-price data has been used in many machine learning papers that address regression
    problems.

    **References**

       - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.
       - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.
       - many more! (see http://archive.ics.uci.edu/ml/datasets/Housing)

Now let's explore the data set itself.

```python
bos = pd.DataFrame(boston.data)
bos.head()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33

There are no column names in the DataFrame. Let's add those.

```python
bos.columns = boston.feature_names
bos.head()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33

Now we have a pandas DataFrame called `bos` containing all the data we want to use to predict Boston Housing prices.  Let's create a variable called `PRICE` which will contain the prices. This information is contained in the `target` data.

```python
print(boston.target.shape)
```

    (506L,)

```python
bos['PRICE'] = boston.target
bos.head()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      PRICE

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98
      24.0

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14
      21.6

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03
      34.7

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94
      33.4

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33
      36.2

## EDA and Summary Statistics
***

Let's explore this data set.  First we use `describe()` to get basic summary statistics for each of the columns.

```python
bos.describe()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      PRICE

      count
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000

      mean
      3.593761
      11.363636
      11.136779
      0.069170
      0.554695
      6.284634
      68.574901
      3.795043
      9.549407
      408.237154
      18.455534
      356.674032
      12.653063
      22.532806

      std
      8.596783
      23.322453
      6.860353
      0.253994
      0.115878
      0.702617
      28.148861
      2.105710
      8.707259
      168.537116
      2.164946
      91.294864
      7.141062
      9.197104

      min
      0.006320
      0.000000
      0.460000
      0.000000
      0.385000
      3.561000
      2.900000
      1.129600
      1.000000
      187.000000
      12.600000
      0.320000
      1.730000
      5.000000

      25%
      0.082045
      0.000000
      5.190000
      0.000000
      0.449000
      5.885500
      45.025000
      2.100175
      4.000000
      279.000000
      17.400000
      375.377500
      6.950000
      17.025000

      50%
      0.256510
      0.000000
      9.690000
      0.000000
      0.538000
      6.208500
      77.500000
      3.207450
      5.000000
      330.000000
      19.050000
      391.440000
      11.360000
      21.200000

      75%
      3.647423
      12.500000
      18.100000
      0.000000
      0.624000
      6.623500
      94.075000
      5.188425
      24.000000
      666.000000
      20.200000
      396.225000
      16.955000
      25.000000

      max
      88.976200
      100.000000
      27.740000
      1.000000
      0.871000
      8.780000
      100.000000
      12.126500
      24.000000
      711.000000
      22.000000
      396.900000
      37.970000
      50.000000

### Scatterplots
***

Let's look at some scatter plots for three variables: 'CRIM' (per capita crime rate), 'RM' (number of rooms) and 'PTRATIO' (pupil-to-teacher ratio in schools).

```python
plt.scatter(bos.CRIM, bos.PRICE)
plt.xlabel("Per capita crime rate by town (CRIM)")
plt.ylabel("Housing Price")
plt.title("Relationship between CRIM and Price")
```

Part 2 Checkup Exercise Set I

Exercise: What kind of relationship do you see? e.g. positive, negative?  linear? non-linear? Is there anything else strange or interesting about the data? What about outliers?

Exercise: Create scatter plots between *RM* and *PRICE*, and *PTRATIO* and *PRICE*. Label your axes appropriately using human readable labels. Tell a story about what you see.

Exercise: What are some other numeric variables of interest? Why do you think they are interesting? Plot scatterplots with these variables and *PRICE* (house price) and tell a story about what you see.

### The scatterplot, Housing Price v. Per capita crime rate by town (CRIM), shows 3 different trends:
1. When the crime rate is very low (~ 0 ), it does not correlate to house price
2. When the crime rate is between 5 and 20, houseing price and crime rate are negativly correlated (with a few outliers for very expensive houses)
3. Once the crime rate exceeds 20, housing price becomes independent of crime rate again

```python
# your turn: scatter plot between *RM* and *PRICE*
plt.scatter(bos.RM, bos.PRICE)
plt.xlabel("Average Number of Rooms (RM)")
plt.ylabel("Housing Price")
plt.title("Relationship between RM and Price")
```

#### For houses with 5 or more rooms, the higher the number of rooms the higher the house price will be for most houses.  This makes sense, the more people that can live in a house is the more expensive it is. However, for the most expensive homes there is no relationship between price and number of rooms.  This maybe due to the fact that recently haveing open layout houses, one large room instead of many smaller rooms, has become more popular.  Thus, the same number of people are living in a 5 room as an 8 room.  Or perhaps, people without large families are less concerned about the number of rooms in their house, and have larger disposable incomes.  So they are willing to pay the same price ( ~50) for a 5 room as they would pay for an 8 room house.

```python
# your turn: scatter plot between *PTRATIO* and *PRICE*
plt.scatter(bos.PTRATIO, bos.PRICE)
plt.xlabel("Pupil-teacher Ratio by Town (PTRATIO)")
plt.ylabel("Housing Price")
plt.title("Relationship between PTRATIO and Price")
```

#### There is no clear relationship between housing price and the pupil-teacher ratio by town, but there does seem to be a lot of data points when the PTRATIO of 14.7, 18, 20.1, and 21. This plot seems to show that large towns (lots of data) have schools with different priced homes, but similar PTRATIO.

```python
# your turn: create some other scatter plots
g = sns.PairGrid(bos, x_vars = ['ZN', 'INDUS', 'NOX', 'AGE','DIS','TAX','B','LSTAT'], y_vars = ['PRICE'])
g.map(plt.scatter)
```

#### Using Seaborn's pairgrid to plot multiple independent variables and price, it seems that LSTAT ( % lower status of the population) has a near-linear relationship with price.

```python
plt.scatter(bos.LSTAT, bos.PRICE)
plt.xlabel("% lower status of the population (LSTAT)")
plt.ylabel("Housing Price")
plt.title("Relationship between LSTAT and Price")
```

#### Looking closer at the relationship between LSTAT and Price, it seems that at 10% and above the higher the LSTAT the lower the housing price.

### Scatterplots using Seaborn
***

[Seaborn](https://stanford.edu/~mwaskom/software/seaborn/) is a cool Python plotting library built on top of matplotlib. It provides convenient syntax and shortcuts for many common types of plots, along with better-looking defaults.

We can also use [seaborn regplot](https://stanford.edu/~mwaskom/software/seaborn/tutorial/regression.html#functions-to-draw-linear-regression-models) for the scatterplot above. This provides automatic linear regression fits (useful for data exploration later on). Here's one example below.

```python
sns.regplot(y="PRICE", x="RM", data=bos, fit_reg = True)
```

### Histograms
***

```python
plt.hist(np.log(bos.CRIM))
plt.title("CRIM")
plt.xlabel("Crime rate per capita")
plt.ylabel("Frequencey")
plt.show()
```

Part 2 Checkup Exercise Set II

Exercise: In the above histogram, we took the logarithm of the crime rate per capita. Repeat this histogram without taking the log. What was the purpose of taking the log? What do we gain by making this transformation? What do you now notice about this variable that is not obvious without making the transformation?

Exercise: Plot the histogram for *RM* and *PTRATIO* against each other, along with the two variables you picked in the previous section. We are looking for correlations in predictors here.

```python
#your turn
plt.hist(bos.CRIM)
plt.title("CRIM")
plt.xlabel("Crime rate per capita")
plt.ylabel("Frequencey")
plt.show()
```

#### The purpose of using the log function is to scale down very large numbers.  Using the log transformation makes it obvious how much of the data resides between 0-1 (all negative log values), where that information is completely obscured when looking at the un-transformed data.

```python
import matplotlib.gridspec as gridspec
plt.style.use('seaborn-whitegrid')
```

```python
# your turn: create some other scatter plots
fig = plt.figure(figsize=(10,7))

gs = gridspec.GridSpec(2, 2)
ax_rm = fig.add_subplot(gs[0,0])
ax_ptratio = fig.add_subplot(gs[0,1])
ax_lstat = fig.add_subplot(gs[1,0])
ax_price = fig.add_subplot(gs[1,1])

ax_rm.hist(bos.RM)
ax_rm.set_ylabel("Frequencey")
ax_rm.set_xlabel('Average number of rooms per dwelling')

ax_ptratio.hist(bos.PTRATIO)
ax_ptratio.set_ylabel("Frequencey")
ax_ptratio.set_xlabel("pupil-teacher ratio by town")

ax_lstat.hist(bos.LSTAT)
ax_lstat.set_ylabel("Frequencey")
ax_lstat.set_xlabel('% lower status of the population')

ax_price.hist(bos.PRICE)
ax_price.set_ylabel("Frequencey")
ax_price.set_xlabel('Price of house')

fig.tight_layout()
```

```python

```

## Part 3: Linear Regression with Boston Housing Data Example
***

Here,

$Y$ = boston housing prices (called "target" data in python, and referred to as the dependent variable or response variable)

and

$X$ = all the other features (or independent variables, predictors or explanatory variables)

which we will use to fit a linear regression model and predict Boston housing prices. We will use the least-squares method to estimate the coefficients.

We'll use two ways of fitting a linear regression. We recommend the first but the second is also powerful in its features.

### Fitting Linear Regression using `statsmodels`
***
[Statsmodels](http://statsmodels.sourceforge.net/) is a great Python library for a lot of basic and inferential statistics. It also provides basic regression functions using an R-like syntax, so it's commonly used by statisticians. While we don't cover statsmodels officially in the Data Science Intensive workshop, it's a good library to have in your toolbox. Here's a quick example of what you could do with it. The version of least-squares we will use in statsmodels is called *ordinary least-squares (OLS)*. There are many other versions of least-squares such as [partial least squares (PLS)](https://en.wikipedia.org/wiki/Partial_least_squares_regression) and [weighted least squares (WLS)](https://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares).

```python
# Import regression modules
import statsmodels.api as sm
from statsmodels.formula.api import ols
```

    C:\Anaconda2\lib\site-packages\statsmodels\compat\pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.
      from pandas.core import datetools

```python
# statsmodels works nicely with pandas dataframes
# The thing inside the "quotes" is called a formula, a bit on that below
m = ols('PRICE ~ RM',bos).fit()
print(m.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.484
    Model:                            OLS   Adj. R-squared:                  0.483
    Method:                 Least Squares   F-statistic:                     471.8
    Date:                Wed, 27 Dec 2017   Prob (F-statistic):           2.49e-74
    Time:                        23:13:01   Log-Likelihood:                -1673.1
    No. Observations:                 506   AIC:                             3350.
    Df Residuals:                     504   BIC:                             3359.
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept    -34.6706      2.650    -13.084      0.000     -39.877     -29.465
    RM             9.1021      0.419     21.722      0.000       8.279       9.925
    ==============================================================================
    Omnibus:                      102.585   Durbin-Watson:                   0.684
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              612.449
    Skew:                           0.726   Prob(JB):                    1.02e-133
    Kurtosis:                       8.190   Cond. No.                         58.4
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

#### Interpreting coefficients

There is a ton of information in this output. But we'll concentrate on the coefficient table (middle table). We can interpret the `RM` coefficient (9.1021) by first noticing that the p-value (under `P>|t|`) is so small, basically zero. This means that the number of rooms, `RM`, is a statisticall significant predictor of `PRICE`. The regression coefficient for `RM` of 9.1021 means that *on average, each additional room is associated with an increase of $\$9,100$ in house price net of the other variables*. The confidence interval gives us a range of plausible values for this average change, about ($\$8,279, \$9,925$), definitely not chump change.

In general, the $\hat{\beta_i}, i > 0$ can be interpreted as the following: "A one unit increase in $x_i$ is associated with, on average, a $\hat{\beta_i}$ increase/decrease in $y$ net of all other variables."

On the other hand, the interpretation for the intercept, $\hat{\beta}_0$ is the average of $y$ given that all of the independent variables $x_i$ are 0.

####  `statsmodels` formulas
***
This formula notation will seem familiar to `R` users, but will take some getting used to for people coming from other languages or are new to statistics.

The formula gives instruction for a general structure for a regression call. For `statsmodels` (`ols` or `logit`) calls you need to have a Pandas dataframe with column names that you will add to your formula. In the below example you need a pandas data frame that includes the columns named (`Outcome`, `X1`,`X2`, ...), but you don't need to build a new dataframe for every regression. Use the same dataframe with all these things in it. The structure is very simple:

`Outcome ~ X1`

But of course we want to to be able to handle more complex models, for example multiple regression is doone like this:

`Outcome ~ X1 + X2 + X3`

In general, a formula for an OLS multiple linear regression is

`Y ~ X1 + X2 + ... + Xp`

This is the very basic structure but it should be enough to get you through the homework. Things can get much more complex. You can force statsmodels to treat variables as categorical with the `C()` function, call numpy functions to transform data such as `np.log` for extremely-skewed data, or fit a model without an intercept by including `- 1` in the formula. For a quick run-down of further uses see the `statsmodels` [help page](http://statsmodels.sourceforge.net/devel/example_formulas.html).

Let's see how our model actually fit our data. We can see below that there is a ceiling effect, we should probably look into that. Also, for large values of $Y$ we get underpredictions, most predictions are below the 45-degree gridlines.

Part 3 Checkup Exercise Set I

Exercise: Create a scatterplot between the predicted prices, available in `m.fittedvalues` (where `m` is the fitted model) and the original prices. How does the plot look? Do you notice anything interesting or weird in the plot? Comment on what you see.

```python
# your turn
plt.scatter(m.fittedvalues, bos.PRICE)
plt.xlabel('Predicted Prices')
plt.ylabel('Original Prices')
plt.title("Original Price vs. Predicted Price")
plt.tight_layout()
```

This plot shows that for the model isn't good at predicting houses with very low or very high prices

### Fitting Linear Regression using `sklearn`

```python
bos.head()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      PRICE

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98
      24.0

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14
      21.6

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03
      34.7

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94
      33.4

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33
      36.2

```python
from sklearn.linear_model import LinearRegression
X = bos.drop('PRICE', axis = 1)

# This creates a LinearRegression object
lm = LinearRegression()
lm
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)

#### What can you do with a LinearRegression object?
***
Check out the scikit-learn [docs here](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html). We have listed the main functions here. Most machine learning models in scikit-learn follow this same API of fitting a model with `fit`, making predictions with `predict` and the appropriate scoring function `score` for each model.

Main functions | Description
--- | ---
`lm.fit()` | Fit a linear model
`lm.predict()` | Predict Y using the linear model with estimated coefficients
`lm.score()` | Returns the coefficient of determination (R^2). *A measure of how well observed outcomes are replicated by the model, as the proportion of total variation of outcomes explained by the model*

#### What output can you get?

```python
# Look inside lm object
# lm.<tab>
```

Output | Description
--- | ---
`lm.coef_` | Estimated coefficients
`lm.intercept_` | Estimated intercept

### Fit a linear model
***

The `lm.fit()` function estimates the coefficients the linear regression using least squares.

```python
# Use all 13 predictors to fit linear regression model
print(lm.fit(X, bos.PRICE))
print(lm.intercept_)
print(lm.coef_)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)
    36.4911032804
    [ -1.07170557e-01   4.63952195e-02   2.08602395e-02   2.68856140e+00
      -1.77957587e+01   3.80475246e+00   7.51061703e-04  -1.47575880e+00
       3.05655038e-01  -1.23293463e-02  -9.53463555e-01   9.39251272e-03
      -5.25466633e-01]

Part 3 Checkup Exercise Set II

Exercise: How would you change the model to not fit an intercept term? Would you recommend not having an intercept? Why or why not? For more information on why to include or exclude an intercept, look [here](https://online.stat.psu.edu/~ajw13/stat501/SpecialTopics/Reg_thru_origin.pdf).

Exercise: One of the assumptions of the linear model is that the residuals must be i.i.d. (independently and identically distributed). To satisfy this, is it enough that the residuals are normally distributed? Explain your answer.

Exercise: True or false. To use linear regression, $Y$ must be normally distributed. Explain your answer.

The lm model above can be fit without an intercept term by simply including the parameter, fit_intercept=False. Looking at the different X features, I think the model will perform better without an intercept term.  However, the best way to determine the better model is to compare them.

```python
# plot model with an intercept term
lm.fit(X, bos.PRICE)
y_predict = lm.predict(X)
plt.scatter(y_predict, bos.PRICE)
```

```python
lm.intercept_
```

    36.491103280363404

```python
# plot model without an intercept term
lm2 = LinearRegression(fit_intercept=False)
lm2.fit(X, bos.PRICE)
y_predict2 = lm2.predict(X)
plt.scatter(y_predict2, bos.PRICE)
```

```python
lm2.intercept_
```

    0.0

Based on the above plots, there doesn't seem to be much difference between having a y-intercept or not. However, when you look at the actual numbers, you can see that the intercepts are really different, b=36.5 vs b=0

### Estimated intercept and coefficients

Let's look at the estimated coefficients from the linear model using `1m.intercept_` and `lm.coef_`.

After we have fit our linear regression model using the least squares method, we want to see what are the estimates of our coefficients $\beta_0$, $\beta_1$, ..., $\beta_{13}$:

$$ \hat{\beta}_0, \hat{\beta}_1, \ldots, \hat{\beta}_{13} $$

```python
print('Estimated intercept coefficient: {}'.format(lm.intercept_))
```

    Estimated intercept coefficient: 36.4911032804

```python
print('Number of coefficients: {}'.format(len(lm.coef_)))
```

    Number of coefficients: 13

```python
# The coefficients
pd.DataFrame({'features': X.columns, 'estimatedCoefficients': lm.coef_})[['features', 'estimatedCoefficients']]
```

      features
      estimatedCoefficients

      0
      CRIM
      -0.107171

      1
      ZN
      0.046395

      2
      INDUS
      0.020860

      3
      CHAS
      2.688561

      4
      NOX
      -17.795759

      5
      RM
      3.804752

      6
      AGE
      0.000751

      7
      DIS
      -1.475759

      8
      RAD
      0.305655

      9
      TAX
      -0.012329

      10
      PTRATIO
      -0.953464

      11
      B
      0.009393

      12
      LSTAT
      -0.525467

### Predict Prices

We can calculate the predicted prices ($\hat{Y}_i$) using `lm.predict`.

$$ \hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_1 + \ldots \hat{\beta}_{13} X_{13} $$

```python
# first five predicted prices
lm.predict(X)[0:5]
```

    array([ 30.00821269,  25.0298606 ,  30.5702317 ,  28.60814055,  27.94288232])

Part 3 Checkup Exercise Set III

Exercise: Histogram: Plot a histogram of all the predicted prices. Write a story about what you see. Describe the shape, center and spread of the distribution. Are there any outliers? What might be the reason for them? Should we do anything special with them?

Exercise: Scatterplot: Let's plot the true prices compared to the predicted prices to see they disagree (we did this with `statsmodels` before).

Exercise: We have looked at fitting a linear model in both `statsmodels` and `scikit-learn`. What are the advantages and disadvantages of each based on your exploration? Based on the information provided by both packages, what advantage does `statsmodels` provide?

```python
# your turn
plt.hist(lm.predict(X))
plt.xlabel("Predicted values of house prices")
plt.ylabel("Frequencey")
plt.show()
```

The above histogram has a gaussian shape, with the center at 20. The data ranges from 0-45, with what might be an outlier at 0. Having a house priced at 0 seems odd. Perhaps it was an error, regardless it should be removed.  A house valued at 0, given the median price is 20, does not seem very useful.

```python
# plot the predicted vs. actual
plt.scatter(bos.PRICE, lm.predict(X), c= 'black')
plt.title('Predicted Price vs Actual Price')
plt.xlabel('Actual House Price')
plt.ylabel('Predicted House Price')

```

Based on the scatter plots, sklearn's prediction maybe slightly better than StatsModel.  However, statsmodel gives a better summary and is slightly easier to use.

```python

```

### Evaluating the Model: Sum-of-Squares

The partitioning of the sum-of-squares shows the variance in the predictions explained by the model and the variance that is attributed to error.

$$TSS = ESS + RSS$$

#### Residual Sum-of-Squares (aka $RSS$)

The residual sum-of-squares is one of the basic ways of quantifying how much error exists in the fitted model. We will revisit this in a bit.

$$ RSS = \sum_{i=1}^N r_i^2 = \sum_{i=1}^N \left(y_i - \left(\beta_0 + \beta_1 x_i\right)\right)^2 $$

```python
print(np.sum((bos.PRICE - lm.predict(X)) ** 2))
```

    11080.2762841

#### Explained Sum-of-Squares (aka $ESS$)

The explained sum-of-squares measures the variance explained by the regression model.

$$ESS = \sum_{i=1}^N \left( \hat{y}_i - \bar{y} \right)^2 = \sum_{i=1}^N \left( \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) - \bar{y} \right)^2$$

```python
print(np.sum(lm.predict(X) - np.mean(bos.PRICE)) ** 2)
```

    8.69056631064e-23

### Evaluating the Model: The Coefficient of Determination ($R^2$)

The coefficient of determination, $R^2$, tells us the percentage of the variance in the response variable $Y$ that can be explained by the linear regression model.

$$ R^2 = \frac{ESS}{TSS} $$

The $R^2$ value is one of the most common metrics that people use in describing the quality of a model, but it is important to note that *$R^2$ increases artificially as a side-effect of increasing the number of independent variables.* While $R^2$ is reported in almost all statistical packages, another metric called the *adjusted $R^2$* is also provided as it takes into account the number of variables in the model, and can sometimes even be used for non-linear regression models!

$$R_{adj}^2 = 1 - \left( 1 - R^2 \right) \frac{N - 1}{N - K - 1} = R^2 - \left( 1 - R^2 \right) \frac{K}{N - K - 1} = 1 - \frac{\frac{RSS}{DF_R}}{\frac{TSS}{DF_T}}$$

where $N$ is the number of observations, $K$ is the number of variables, $DF_R = N - K - 1$ is the degrees of freedom associated with the residual error and $DF_T = N - 1$ is the degrees of the freedom of the total error.

### Evaluating the Model: Mean Squared Error and the $F$-Statistic
***
The mean squared errors are just the *averages* of the sum-of-squares errors over their respective degrees of freedom.

$$MSE = \frac{ESS}{K}$$
$$MSR = \frac{RSS}{N-K-1}$$

**Remember: ** Notation may vary across resources particularly the use of *R* and *E* in *RSS/ESS* and *MSR/MSE*. In some resources, E = explained and R = residual. In other resources, E = error and R = regression (explained). **This is a very important distinction that requires looking at the formula to determine which naming scheme is being used.**

Given the MSR and MSE, we can now determine whether or not the entire model we just fit is even statistically significant. We use an $F$-test for this. The null hypothesis is that all of the $\beta$ coefficients are zero, that is, none of them have any effect on $Y$. The alternative is that *at least one* $\beta$ coefficient is nonzero, but it doesn't tell us which one in a multiple regression:

$$H_0: \beta_i = 0, \mbox{for all $i$} \\
H_A: \beta_i > 0, \mbox{for some $i$}$$

$$F = \frac{MSR}{MSE} = \left( \frac{R^2}{1 - R^2} \right) \left( \frac{N - K - 1}{K} \right)$$

Once we compute the $F$-statistic, we can use the $F$-distribution with $N-K$ and $K-1$ degrees of degrees of freedom to get a p-value.

**Warning!** The $F$-statistic mentioned in this section is NOT the same as the F1-measure or F1-value discused in Unit 7.

Part 3 Checkup Exercise Set IV

Let's look at the relationship between `PTRATIO` and housing price.

Exercise: Try fitting a linear regression model using only the 'PTRATIO' (pupil-teacher ratio by town) and interpret the intercept and the coefficients.

Exercise: Calculate (or extract) the $R^2$ value. What does it tell you?

Exercise: Compute the $F$-statistic. What does it tell you?

Exercise: Take a close look at the $F$-statistic and the $t$-statistic for the regression coefficient. What relationship do you notice? Note that this relationship only applies in *simple* linear regression models.

```python
# Fit a linear regression model PTRATIO vs. PRICE
m = ols('PRICE ~ PTRATIO',bos).fit()
print(m.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.258
    Model:                            OLS   Adj. R-squared:                  0.256
    Method:                 Least Squares   F-statistic:                     175.1
    Date:                Wed, 27 Dec 2017   Prob (F-statistic):           1.61e-34
    Time:                        23:13:04   Log-Likelihood:                -1764.8
    No. Observations:                 506   AIC:                             3534.
    Df Residuals:                     504   BIC:                             3542.
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept     62.3446      3.029     20.581      0.000      56.393      68.296
    PTRATIO       -2.1572      0.163    -13.233      0.000      -2.477      -1.837
    ==============================================================================
    Omnibus:                       92.924   Durbin-Watson:                   0.725
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              191.444
    Skew:                           1.001   Prob(JB):                     2.68e-42
    Kurtosis:                       5.252   Cond. No.                         160.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

The coefficient for PTRATIO is -2.16, which means that the larger the average pupil-teacher ratio, the lower the house price.  The R^2 value 0.25, which means the PTRATIO and price have a small linear relationship. The F-statistic, which is used to assess multiple coefficients, is 175.1,  with a corresponding probability of approx. 0 . So we can reject the null hypothesis, and accept that at least one feature is non-zero.  Since there is only one feature in this regression, that means the PTRATIO coefficient is non-zero. Both the F-statistic and the t-test have the same p-value, this makes sense because with only one feature they are assessing the same thing.

```python

```

Part 3 Checkup Exercise Set V

Fit a linear regression model using three independent variables

 'CRIM' (per capita crime rate by town)

 'RM' (average number of rooms per dwelling)

 'PTRATIO' (pupil-teacher ratio by town)

Exercise: Compute or extract the $F$-statistic. What does it tell you about the model?

Exercise: Compute or extract the $R^2$ statistic. What does it tell you about the model?

Exercise: Which variables in the model are significant in predicting house price? Write a story that interprets the coefficients.

```python
m = ols('PRICE ~ CRIM + RM +PTRATIO',bos).fit()
print(m.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.593
    Model:                            OLS   Adj. R-squared:                  0.591
    Method:                 Least Squares   F-statistic:                     244.2
    Date:                Wed, 27 Dec 2017   Prob (F-statistic):           1.09e-97
    Time:                        23:13:04   Log-Likelihood:                -1612.6
    No. Observations:                 506   AIC:                             3233.
    Df Residuals:                     502   BIC:                             3250.
    Df Model:                           3
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept     -3.3066      4.038     -0.819      0.413     -11.240       4.627
    CRIM          -0.2021      0.032     -6.301      0.000      -0.265      -0.139
    RM             7.3816      0.402     18.360      0.000       6.592       8.171
    PTRATIO       -1.0742      0.133     -8.081      0.000      -1.335      -0.813
    ==============================================================================
    Omnibus:                      233.249   Durbin-Watson:                   0.828
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):             2002.374
    Skew:                           1.802   Prob(JB):                         0.00
    Kurtosis:                      12.054   Cond. No.                         311.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

The F-statistic is 244.5, with a corresponding probability of approx. 0, which means that at least one coefficent is non-zero. The R^2 statistic is 0.59, which means there is a linear relastionships with all of the features and price. All of the feature have a significant p-value, but the crime rate coefficent is very small, so it contributes to the prediction the least.

## Part 4: Comparing Models

During modeling, there will be times when we want to compare models to see which one is more predictive or fits the data better. There are many ways to compare models, but we will focus on two.

### The $F$-Statistic Revisited

The $F$-statistic can also be used to compare two *nested* models, that is, two models trained on the same dataset where one of the models contains a *subset* of the variables of the other model. The *full* model contains $K$ variables and the *reduced* model contains a subset of these $K$ variables. This allows us to add additional variables to a base model and then test if adding the variables helped the model fit.

$$F = \frac{\left( \frac{RSS_{reduced} - RSS_{full}}{DF_{reduced} - DF_{full}} \right)}{\left( \frac{RSS_{full}}{DF_{full}} \right)}$$

where $DF_x = N - K_x - 1$ where $K_x$ is the number of variables in model $x$.

### Akaike Information Criterion (AIC)

Another statistic for comparing two models is AIC, which is based on the likelihood function and takes into account the number of variables in the model.

$$AIC = 2 K - 2 \log_e{L}$$

where $L$ is the likelihood of the model. AIC is meaningless in the absolute sense, and is only meaningful when compared to AIC values from other models. Lower values of AIC indicate better fitting models.

`statsmodels` provides the AIC in its output.

Part 4 Checkup Exercises

Exercise: Find another variable (or two) to add to the model we built in Part 3. Compute the $F$-test comparing the two models as well as the AIC. Which model is better?

```python
m = ols('PRICE ~ CHAS + TAX + DIS',bos).fit()
print(m.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.245
    Model:                            OLS   Adj. R-squared:                  0.241
    Method:                 Least Squares   F-statistic:                     54.35
    Date:                Wed, 27 Dec 2017   Prob (F-statistic):           1.94e-30
    Time:                        23:13:04   Log-Likelihood:                -1769.1
    No. Observations:                 506   AIC:                             3546.
    Df Residuals:                     502   BIC:                             3563.
    Df Model:                           3
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept     31.6929      1.637     19.363      0.000      28.477      34.909
    CHAS           5.8616      1.419      4.131      0.000       3.074       8.649
    TAX           -0.0245      0.003     -9.728      0.000      -0.029      -0.020
    DIS            0.1141      0.202      0.564      0.573      -0.284       0.512
    ==============================================================================
    Omnibus:                      152.128   Durbin-Watson:                   0.707
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              376.497
    Skew:                           1.524   Prob(JB):                     1.76e-82
    Kurtosis:                       5.927   Cond. No.                     2.12e+03
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
    [2] The condition number is large, 2.12e+03. This might indicate that there are
    strong multicollinearity or other numerical problems.

The above model has:
1. An R^2 statistic is 0.24 , which means there is a very slight linear relastionships with all of the features and price.
2. Only two of the features have a significant p-value
3. Of the significant features, only one has a decent coefficent value
4. The AIC is slightly larger than the previous model

The F-statistic for both models have a corresponding probability of approx. 0, which means that at least one coefficent is non-zero. However, given all of the information I would say the previous model is the best model

## Part 5: Evaluating the Model via Model Assumptions and Other Issues
***
Linear regression makes several assumptions. It is always best to check that these assumptions are valid after fitting a linear regression model.

**Linearity**. The dependent variable $Y$ is a linear combination of the regression coefficients and the independent variables $X$. This can be verified with a scatterplot of each $X$ vs. $Y$ and plotting correlations among $X$. Nonlinearity can sometimes be resolved by [transforming](https://onlinecourses.science.psu.edu/stat501/node/318) one or more independent variables, the dependent variable, or both. In other cases, a [generalized linear model](https://en.wikipedia.org/wiki/Generalized_linear_model) or a [nonlinear model](https://en.wikipedia.org/wiki/Nonlinear_regression) may be warranted.

**Constant standard deviation**. The SD of the dependent variable $Y$ should be constant for different values of X. We can check this by plotting each $X$ against $Y$ and verifying that there is no "funnel" shape showing data points fanning out as $X$ increases or decreases. Some techniques for dealing with non-constant variance include weighted least squares (WLS), [robust standard errors](https://en.wikipedia.org/wiki/Heteroscedasticity-consistent_standard_errors), or variance stabilizing transformations.

 **Normal distribution for errors**.  The $\epsilon$ term we discussed at the beginning are assumed to be normally distributed. This can be verified with a fitted values vs. residuals plot and verifying that there is no pattern, and with a quantile plot.
  $$ \epsilon_i \sim N(0, \sigma^2)$$
Sometimes the distributions of responses $Y$ may not be normally distributed at any given value of $X$.  e.g. skewed positively or negatively.

 **Independent errors**.  The observations are assumed to be obtained independently.

e.g. Observations across time may be correlated

There are some other issues that are important investigate with linear regression models.

**Correlated Predictors:** Care should be taken to make sure that the independent variables in a regression model are not too highly correlated. Correlated predictors typically do not majorly affect prediction, but do inflate standard errors of coefficients making interpretation unreliable. Common solutions are dropping the least important variables involved in the correlations, using regularlization, or, when many predictors are highly correlated, considering a dimension reduction technique such as principal component analysis (PCA).

**Influential Points:** Data points that have undue influence on the regression model. These points can be high leverage points or outliers. Such points are typically removed and the regression model rerun.

Part 5 Checkup Exercises

Take the reduced model from Part 3 to answer the following exercises. Take a look at [this blog post](http://mpastell.com/2013/04/19/python_regression/) for more information on using statsmodels to construct these plots.

Exercise: Construct a fitted values versus residuals plot. What does the plot tell you? Are there any violations of the model assumptions?

Exercise: Construct a quantile plot of the residuals. What does the plot tell you?

Exercise: What are some advantages and disadvantages of the fitted vs. residual and quantile plot compared to each other?

Exercise: Identify any outliers (if any) in your model and write a story describing what these outliers might represent.

Exercise: Construct a leverage plot and identify high leverage points in the model. Write a story explaining possible reasons for the high leverage points.

Exercise: Remove the outliers and high leverage points from your model and run the regression again. How do the results change?

```python
# Fit a linear regression model PTRATIO vs. PRICE
linear_model = ols('PRICE ~ PTRATIO',bos).fit()
print(linear_model.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.258
    Model:                            OLS   Adj. R-squared:                  0.256
    Method:                 Least Squares   F-statistic:                     175.1
    Date:                Wed, 27 Dec 2017   Prob (F-statistic):           1.61e-34
    Time:                        23:13:04   Log-Likelihood:                -1764.8
    No. Observations:                 506   AIC:                             3534.
    Df Residuals:                     504   BIC:                             3542.
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept     62.3446      3.029     20.581      0.000      56.393      68.296
    PTRATIO       -2.1572      0.163    -13.233      0.000      -2.477      -1.837
    ==============================================================================
    Omnibus:                       92.924   Durbin-Watson:                   0.725
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              191.444
    Skew:                           1.001   Prob(JB):                     2.68e-42
    Kurtosis:                       5.252   Cond. No.                         160.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

#### Plot the residuals vs. fitted data

```python
ypred = linear_model.predict(bos.PTRATIO)
```

```python
err_dict = {'PRICE':bos['PRICE'].copy(), 'ypred':ypred}
```

```python
err_df = pd.DataFrame(err_dict)
```

```python
err_df['resid'] = err_df['PRICE']- err_df.ypred
```

```python
plt.scatter(err_df.PRICE, err_df.resid)
plt.xlabel('Fitted Data')
plt.ylabel('Residuals')
plt.title('Residuals vs. Fitted Data')
```

The residual vs. fitted values plot is not random, which indicates the relationship between price and the pupil-teacher ratio by town is not linear.

#### Quantile plot of the residuals

```python
import scipy.stats as stats
import pylab
```

```python
stats.probplot(err_df.resid, dist='norm',plot = pylab)
pylab.show()
```

The deviation from the red line at the tail of the residual-quantile plot suggests there maybe outlier values in the dataset.

#### The residual-fitted plot can be used to determine if a linear model is approprate, where a quantile plot is used to determine the assumption of normally distributed data.

### Identifying outliers

```python
third_quartile = np.percentile(bos.PTRATIO, 75)
first_quartile = np.percentile(bos.PTRATIO, 25)
IQR = third_quartile - first_quartile
high_outlier = third_quartile + (1.5 * IQR)
low_outlier = first_quartile - (1.5 * IQR)
```

```python
# Remove outliers
mask = (bos.PTRATIO > low_outlier) & (bos.PTRATIO < high_outlier)
no_outliers = bos[mask]
```

```python
number_outliers = len(bos)-len(no_outliers)
print(number_outliers)
```

    15

There are 15 outliers in the dataset. These points represent houses where the price is either lower or higher than expected based on the pupil-to-teacher ratio

#### Leverage plot

```python
from statsmodels.graphics.regressionplots import *
```

```python
fig, ax = plt.subplots(figsize=(14,8))
ax.set_xlim([0.0, 0.02])
ax.set_ylim([-3, 4.5])
fig = sm.graphics.influence_plot(linear_model, ax=ax, criterion="cooks")

```

It looks like the house represented by 267, 262, and 283 have a large Cook's distance, which means it is probably an influential outlier. Below is the same linear regression, but with outliers and the high leverage points removed from the data.

```python
drop_Hleverage = no_outliers.drop(no_outliers.index[[267,262,283]])
```

```python
# Fit a linear regression model without outliers
modified_model = ols('PRICE ~ PTRATIO',drop_Hleverage).fit()
print(modified_model.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.202
    Model:                            OLS   Adj. R-squared:                  0.200
    Method:                 Least Squares   F-statistic:                     123.0
    Date:                Wed, 27 Dec 2017   Prob (F-statistic):           1.25e-25
    Time:                        23:13:07   Log-Likelihood:                -1699.0
    No. Observations:                 488   AIC:                             3402.
    Df Residuals:                     486   BIC:                             3410.
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept     59.7743      3.423     17.461      0.000      53.048      66.500
    PTRATIO       -2.0252      0.183    -11.089      0.000      -2.384      -1.666
    ==============================================================================
    Omnibus:                       99.867   Durbin-Watson:                   0.704
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              219.218
    Skew:                           1.077   Prob(JB):                     2.50e-48
    Kurtosis:                       5.479   Cond. No.                         180.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

#### Removing the outliers and high leverage points didn't really improve the linear model. The residuals (shown below) are still not randomly distributed, so it makes sense that model didn't improve, as the Pupil-to-teacher ratio is still not linearly related to price.

```python
ypred = modified_model.predict(drop_Hleverage.PTRATIO)
resid = drop_Hleverage['PRICE']- ypred
```

```python
plt.scatter(drop_Hleverage.PRICE, resid)
plt.xlabel('Fitted Data')
plt.ylabel('Residuals')
plt.title('Residuals vs. Fitted Data')
```

```python

```


# Vascular, Extracellular and Restricted Diffusion for Cytometry in Tumors (VERDICT)

Panagiotaki et al. (2014) proposed a multi-compartment model called VERDICT to characterize the composition of tumorous tissues. VERDICT models the diffusion in tumor cells, the extra-cellular space and surrounding bloodvessels as a restricted Sphere, an isotropic Gaussian Ball and a Stick compartment, respectively. VERDICT's design is as follows:
\begin{equation}
 E_{\textrm{VERDICT}}= \underbrace{f_{\textrm{Tumor}}\overbrace{E(D|\lambda_{\textrm{intra}})}^{\textrm{Sphere}}}_{\textrm{Tumor Cells}} + \underbrace{f_{\textrm{extra}}\overbrace{E(\cdot|\lambda_{iso})}^{\textrm{Ball}}}_{\textrm{Hindered Extra-Cellular}}+\underbrace{f_{blood}\overbrace{E(\lambda_\parallel, \boldsymbol{\mu})}^{\textrm{Stick}}}_{\textrm{Vascular}}
\end{equation}
where $D$ is the sphere's diameter. VERDICT uses the Gaussian Phase approximation to model the sphere *(Balinov et al. 1993)*, which accounts to changes in gradient pulse duration $\delta$ and separation $\Delta$. Furthermore, some particular parameter constraints are imposed:
- The intra-cellular (intra-spherical) diffusivity is fixed to 0.9e-9 m^2/s
- The extra-cellular Gaussian diffusivity is also fixed to 0.9e-9 m^2/s
- The optimization range for $\lambda_\parallel$ of the Vascular Stick is set between (3.05-10)e-9 m^2/s

We can define the VERDICT model in the following lines of code:

# Using Dmipy to set up the VERDICT model

We first load the necessary modules to import the Sphere, Ball, and Stick models.

```python
from dmipy.signal_models import sphere_models, cylinder_models, gaussian_models
```

The Sphere model must be initiated with VERDICT's setting for intra-spherical diffusivity, while the Ball and Stick model are regularly imported.

```python
sphere = sphere_models.S4SphereGaussianPhaseApproximation(diffusion_constant=0.9e-9)
ball = gaussian_models.G1Ball()
stick = cylinder_models.C1Stick()
```

We then assemble the three models into a multi-compartment model:

```python
from dmipy.core.modeling_framework import MultiCompartmentModel
verdict_mod = MultiCompartmentModel(models=[sphere, ball, stick])
verdict_mod.parameter_names
```

    We highly recommend installing pathos to take advantage of multicore processing.

    ['G1Ball_1_lambda_iso',
     'S4SphereGaussianPhaseApproximation_1_diameter',
     'C1Stick_1_mu',
     'C1Stick_1_lambda_par',
     'partial_volume_0',
     'partial_volume_1',
     'partial_volume_2']

Visualize the model:

```python
from IPython.display import Image
verdict_mod.visualize_model_setup(view=False, cleanup=False, with_parameters=True)
Image('Model Setup.png')
```

We then fix the Ball's diffusivity, and adjust the Stick's optimization range for $\lambda_\parallel$  for the optimization.

```python
verdict_mod.set_fixed_parameter('G1Ball_1_lambda_iso', 0.9e-9)
verdict_mod.set_parameter_optimization_bounds('C1Stick_1_lambda_par', [3.05e-9, 10e-9])
```

We are now ready to fit the data.

# Fitting VERDICT to UCL Example Data

As illustration data we use the voxel of VERDICT data that is freely available at the UCL website at http://camino.cs.ucl.ac.uk/index.php?n=Tutorials.VERDICTcol. We prepared the script to easily import the acquisition scheme and example data from there:

```python
from dmipy.data import saved_data
scheme, data = saved_data.panagiotaki_verdict()
scheme.print_acquisition_info
```

    Acquisition scheme summary

    total number of measurements: 220
    number of b0 measurements: 49
    number of DWI shells: 41

    shell_index |# of DWIs |bvalue [s/mm^2] |gradient strength [mT/m] |delta [ms] |Delta[ms] |TE[ms]
    0           |19        |20              |0                        |3.0        |10.0      |17.0
    1           |3         |148             |160                      |3.0        |10.0      |17.0
    2           |3         |231             |200                      |3.0        |10.0      |17.0
    3           |3         |333             |240                      |3.0        |10.0      |17.0
    4           |3         |454             |280                      |3.0        |10.0      |17.0
    5           |3         |593             |320                      |3.0        |10.0      |17.0
    6           |3         |751             |360                      |3.0        |10.0      |17.0
    7           |3         |927             |400                      |3.0        |10.0      |17.0
    8           |12        |4               |0                        |3.0        |20.0      |27.0
    9           |3         |78              |80                       |3.0        |20.0      |27.0
    10          |3         |176             |120                      |3.0        |20.0      |27.0
    11          |3         |313             |160                      |3.0        |20.0      |27.0
    12          |3         |489             |200                      |3.0        |20.0      |27.0
    13          |3         |704             |240                      |3.0        |20.0      |27.0
    14          |3         |959             |280                      |3.0        |20.0      |27.0
    15          |3         |1253            |320                      |3.0        |20.0      |27.0
    16          |3         |1585            |360                      |3.0        |20.0      |27.0
    17          |9         |0               |0                        |3.0        |30.0      |37.0
    18          |3         |119             |80                       |3.0        |30.0      |37.0
    19          |3         |268             |120                      |3.0        |30.0      |37.0
    20          |3         |478             |160                      |3.0        |30.0      |37.0
    21          |3         |747             |200                      |3.0        |30.0      |37.0
    22          |3         |1075            |240                      |3.0        |30.0      |37.0
    23          |3         |1464            |280                      |3.0        |30.0      |37.0
    24          |3         |1912            |320                      |3.0        |30.0      |37.0
    25          |3         |2420            |360                      |3.0        |30.0      |37.0
    26          |3         |2988            |400                      |3.0        |30.0      |37.0
    27          |13        |9               |0                        |3.0        |40.0      |47.0
    28          |3         |160             |80                       |3.0        |40.0      |47.0
    29          |3         |361             |120                      |3.0        |40.0      |47.0
    30          |3         |643             |160                      |3.0        |40.0      |47.0
    31          |3         |1004            |200                      |3.0        |40.0      |47.0
    32          |3         |1446            |240                      |3.0        |40.0      |47.0
    33          |3         |1969            |280                      |3.0        |40.0      |47.0
    34          |3         |2572            |320                      |3.0        |40.0      |47.0
    35          |3         |3255            |360                      |3.0        |40.0      |47.0
    36          |3         |4018            |400                      |3.0        |40.0      |47.0
    37          |6         |0               |0                        |3.5        |8.0       |15.0
    38          |42        |958             |400                      |3.5        |8.0       |15.0
    39          |2         |0               |0                        |10.0       |30.0      |44.0
    40          |3         |305             |40                       |10.0       |30.0      |44.0
    41          |3         |2748            |120                      |10.0       |30.0      |44.0
    42          |3         |0               |0                        |10.0       |40.0      |47.0
    43          |3         |419             |40                       |10.0       |40.0      |47.0
    44          |3         |1679            |80                       |10.0       |40.0      |47.0
    45          |3         |3778            |120                      |10.0       |40.0      |47.0

We can see that the VERDICT acquisition scheme is very particular, having many shells with different diffusion times and often only 3 perpendicular measurements per shell, with the exception of one DTI shell with 42 measurements.

To fit the data we use the MIX algorithm *(Farooq et al. 2016)*, which is efficient for finding the global minimum in models with many compartments. We set parallel processing to False since there is only one voxel.

```python
verdict_fit = verdict_mod.fit(scheme, data, solver='mix', use_parallel_processing=False)
```

    Setup MIX optimizer in 8.10623168945e-06 seconds
    Fitting of 1 voxels complete in 2.82182407379 seconds.
    Average of 2.82182407379 seconds per voxel.

# Visualize results

To illustrate that Dmipy's VERDICT implementation is correct, we show that we can produce a very similar signal fitting plots as the one shown on the UCL website. On the left we show their original graph, and on the right we show Dmipy's predicted signal together with the measured signal attenuation.

```python
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import matplotlib.image as mpimg
img=mpimg.imread("http://camino.cs.ucl.ac.uk/uploads/Tutorials/VCLSsynth.png")

mask_nonzero = scheme.gradient_strengths>0.
G_nonzero = scheme.gradient_strengths[mask_nonzero]
Delta_nonzero = scheme.Delta[mask_nonzero]
delta_nonzero = scheme.delta[mask_nonzero]

predicted_data = verdict_fit.predict()[0]
predicted_data_nonzero = predicted_data[mask_nonzero]
data_nonzero = data[mask_nonzero]

fig, axs = plt.subplots(1, 2, figsize=[30, 10])
axs = axs.ravel()
axs[0].imshow(img)
axs[0].set_title('UCL Verdict Results', fontsize=30)
axs[0].axis('off')

for delta_, Delta_ in np.unique(np.c_[scheme.shell_delta, scheme.shell_Delta], axis=0):
    mask = np.all([Delta_nonzero == Delta_, delta_nonzero == delta_], axis=0)
    axs[1].plot(G_nonzero[mask], predicted_data_nonzero[mask])
    axs[1].scatter(G_nonzero[mask], data_nonzero[mask], s=3., marker='o')
axs[1].set_title('Dmipy Verdict Results', fontsize=30)
axs[1].set_xlabel('G(T/m)', fontsize=20)
axs[1].set_ylabel('S', fontsize=20);

```

While we didn't exactly duplicate their graph style, we can see that both the plots and signal fitting are very similar between Dmipy's and UCL's implementation.

Showing the fitting parameters, we can also see that Dmipy's estimate of the tumor cell diameter is 1.54e-5m, i.e. 15.4$\mu m$, which falls exactly within the ranges that Panagiotaki et al. reports.

```python
verdict_fit.fitted_parameters
```

    {'C1Stick_1_lambda_par': array([6.5832855e-09]),
     'C1Stick_1_mu': array([[1.01286177, 0.6053996 ]]),
     'S4SphereGaussianPhaseApproximation_1_diameter': array([1.54027035e-05]),
     'partial_volume_0': array([0.76131121]),
     'partial_volume_1': array([0.09743701]),
     'partial_volume_2': array([0.14125178])}

## References
- Panagiotaki, Eletheria, et al. "Noninvasive quantification of solid tumor microstructure using VERDICT MRI." Cancer research 74.7 (2014): 1902-1912.
- Balinov, Balin, et al. "The NMR self-diffusion method applied to restricted diffusion. Simulation of echo attenuation from molecules in spheres and between planes." Journal of Magnetic Resonance, Series A 104.1 (1993): 17-25.
- Farooq, Hamza, et al. "Microstructure Imaging of Crossing (MIX) White Matter Fibers from diffusion MRI." Nature Scientific reports 6 (2016).


# Optimal period grid and optimal duration grid for transit searches

This tutorial explains the optimal period and duration grid, as implemented by the Transit Least Squares algorithm and explained in detail in our paper [(Hippke & Heller 2019)](https://arxiv.org/abs/1901.02015).

For transit searches with the Box Least Squares (BLS, [Kovács+ 2002](http://adsabs.harvard.edu/abs/2002A%26A...391..369K)) algorithm, the most commonly used grid is linear in duration and period, and searches the grid space fully. Roughly, individual searches can be visualized in the 2D plane of period and duration:

```python
import numpy
import matplotlib.pyplot as plt
import matplotlib.patches as patches
from matplotlib import rcParams; rcParams["figure.dpi"] = 150
ax = plt.gca()
rect = patches.Rectangle((0.5, 0.01), 980, 0.92, linewidth=1, edgecolor='r', facecolor='red')
ax.add_patch(rect)
for x in numpy.linspace(0.5, 980, 20):
    for y in numpy.linspace(0.02, 0.92, 20):
        plt.plot(x, y, color='black', marker='o', markersize=1)
ax.set_xlabel(r'Period (days)')
ax.set_ylabel(r'Transit duration (days)')
ax.set_xlim(0, 1000)
ax.set_ylim(0, 1);
```

## Period grid
This filled linear period-duration grid is simple, but it has a major disadvantage. For a reasonable sensitivity over all trial periods, short periods require a much denser sampling compared to long periods. As an example, thousands of periods need to be sampled for $0.5<P<1$ day. Periods between 365 and 366 days, however, require only a few trials to achieve similar sensitivity. A linear grid which is sufficiently dense for
short periods will thus be computationally inefficient at longer periods.

[Ofir (2014)](https://ui.adsabs.harvard.edu/#abs/2014A&A...561A.138O/abstract) derived the optimal number of test frequencies as

\begin{equation}
N_{\rm freq,,opt} = \left( f_{\rm max}^{1/3} - f_{\rm min}^{1/3} + \frac{A}{3} \right) \frac{3}{A}
\end{equation}

with

\begin{equation}
A=
\frac{(2\pi)^{2/3}}{\pi }
\frac{R_{\rm s}}{(GM_{\rm s})^{1/3}}
\frac{1}{S \times OS} \ ,
\end{equation}

where $G$ is the gravitational constant, $S$ is the time span of the data set, and $OS$ is the oversampling parameter to be chosen between 2 and 5 to ensure that the SDE peak is not missed between trial frequencies (or periods). The minimum and maximum trial orbital frequencies can be found at $f_{\rm min}=2/S$ (or $f_{\rm min}=3/S$ if three transits are required) and at the most short-period (high-frequency) circumstellar orbit, the Roche limit, $f_{\rm max}= \sqrt{GM_{\rm s}/(3R_{\rm s})^3} / (2\pi)$. Strictly speaking, the Roche limit depends on the density ($\rho_{\rm p}$) of the planet, and the term $3{\,}R_{\rm s}$ for our expression of $f_{\rm max}$ assumes the most pessimistic case of an extremely low-density fluid-like planet with $\rho_{\rm p} \sim 1{\,}{\rm g{\,}cm}^{-3}$, which can be compared to Jupiter's mean density of $1.33{\,} {\rm g{\,}cm}^{-3}$. Our TLS implementation generates an array of evenly spaced orbital frequencies with $N_{\rm freq,opt}$ constant steps between $f_{\rm min}$ and $f_{\rm max}$ and then computes the (non-uniform) trial orbital periods as the inverse of this frequency grid.

Since computational speed is a key concern for TLS, we illustrate the resulting number of trial periods in the following Figure, using three different time spans $S$ of a hypothetical light curve for different stellar masses (and radii, assuming main-sequence mass-radius relation).

```python
from transitleastsquares import period_grid
import numpy
import matplotlib.pyplot as plt

# Main sequence stars with masses and radii
R = numpy.array([0.13, 0.32, 0.63, 0.74, 0.85, 0.93, 1, 1.05, 1.2, 1.3])
M = numpy.array([0.1,  0.21, 0.47, 0.69, 0.78, 0.93, 1, 1.1,  1.3, 1.7])

p10d = []
p100d = []
p1000d = []
for i in range(len(R)):
    p10d.append(len(period_grid(R[i], M[i], 10)))
    p100d.append(len(period_grid(R[i], M[i], 100)))
    p1000d.append(len(period_grid(R[i], M[i], 1000)))
plt.plot(M, p10d, color='black', lw=0.5)
plt.plot(M, p100d, color='black', lw=0.5)
plt.plot(M, p1000d, color='black', lw=0.5)

plt.text(0.75, 1.2*10**5, '1000 days')
plt.text(0.75, 10**4, '100 days')
plt.text(0.75, 0.7*10**3, '10 days')

plt.yscale('log')
plt.xlim(0, 1.7)
plt.ylabel(r'Number of trial periods')
plt.xlabel(r'Stellar mass ($M_{\odot}$)');
```

We find that an extension of the light curve by a certain factor -- here ten between the three example curves -- increases the number of trial periods by the same factor. This plot warns us of the large number of trial periods that need to be examined for planets around low-mass stars, with $N_{\rm freq, opt}$ reaching values of up to almost one million for a light curve with 1000 d of continuous observations of a very-low-mass star. This feature is inherent to both TLS and BLS.

## Duration grid
Regarding the transit duration, we now show that a linear grid is not optimal.

Transit searches using the BLS algorithm usually operate with trial transit
durations $T_{14}$ that span
$0.00125-0.07\times$ [(Petigura+ 2013)](https://ui.adsabs.harvard.edu/#abs/2013ApJ...770...69P/abstract), $0.01-0.1\times$ [Giacobbe 2012+](https://ui.adsabs.harvard.edu/#abs/2012MNRAS.424.3101G/abstract), or $0.001-0.2\times$ [(Sanchis-Ojeda 2014+](https://ui.adsabs.harvard.edu/#abs/2014ApJ...787...47S/abstract), [Aigrain 2016+)](https://ui.adsabs.harvard.edu/#abs/2016MNRAS.459.2408A/abstract) the orbital period.

For circular orbits, the maximum transit duration is $T_{14,{\rm max}}~=~2(R_{\rm s}+R_{\rm p})/v_{\rm p}$, where $v_{\rm p}$ is the planet's average orbital velocity during the transit. Shorter transit durations are possible if the planetary transit path is not across the stellar diameter. We then have

\begin{equation}
T_{14,{\rm max}} = \frac{2,(R_{\rm s}+R_{\rm p})}{v_{\rm p}} = \frac{2,(R_{\rm s}+R_{\rm p})}{2\pi a / P}
\end{equation}

In the limit of the star being much more massive than the planet, Kepler's third law becomes

\begin{align}
\left(\frac{2\pi}{P}\right)^2 \ a^3 \approx \ & G M_{\rm s} \
\Leftrightarrow \hspace{1.8cm} a \approx \ & \left( G M_{\rm s} \left( \frac{P}{2\pi} \right)^2 \right)^{1/3} \ .
\end{align}

We insert the second into the first Equation and obtain

\begin{align}
T_{14,{\rm max}} = \ & \frac{(R_{\rm s}+R_{\rm p}) , P}{\pi} \left( \frac{1}{G M_{\rm s}} \left( \frac{2\pi}{P} \right)^2 \right)^{1/3} \
= \ & (R_{\rm s}+R_{\rm p}) \left( \frac{4P}{\pi G M_{\rm s}} \right)^{1/3} \ .
\end{align}

We now plot this equation for $R_{\rm p}=2,R_{\rm J}$ ($R_{\rm J}$ being Jupiter's radius) orbiting an A5 star ($M_{\rm s}=2.1,M_\odot, R_{\rm s}=1.7,R_\odot$) to maximize the effect of a very large planet on the transit duration. To embrace the physically plausible search space, we also show a main-sequence M8 red dwarf star ($M_{\rm s}=0.1{\,}M_\odot,{\,} R_{\rm s}=0.13{\,}R_\odot$) and a sun-like star, both with a small planet ($R_{\rm p}={\,}R_{\rm \oplus}$).

```python
from numpy import pi

# astrophysical constants
SECONDS_PER_DAY = 86400
G = 6.67408e-11  # gravitational constant [m^3 / kg / s^2]
R_sun = 695508000  # radius of the Sun [m]
R_jup = 69911000  # radius of Jupiter [m]
M_sun = 1.989 * 10 ** 30  # mass of the Sun [kg]

# Maximum fractional transit duration ever observed is 0.117
# for Kepler-1368 b (as of Oct 2018), so we set upper_limit=0.12
# Long fractional transit durations are computationally expensive
# following a quadratic relation. If applicable, use a different value.
# Longer transits can still be found, but at decreasing sensitivity
FRACTIONAL_TRANSIT_DURATION_MAX = 0.12

def T14(R_s, M_s, P, upper_limit=FRACTIONAL_TRANSIT_DURATION_MAX, small_planet=False):
    """Input:  Stellar radius and mass [in solar units]; planetary period [in days]
       Output: Maximum planetary transit duration T_14max [in fraction of period P]"""

    P = P * SECONDS_PER_DAY
    R_s = R_sun * R_s
    M_s = M_sun * M_s

    if small_planet:  # small planet assumption: planet size negligible
        T14 = R_s * ((4 * P) / (pi * G * M_s)) ** (1 / 3)
    else:  # planet size 2 R_jup
        T14 = (R_s + 2 * R_jup) * ((4 * P) / (pi * G * M_s)) ** (1 / 3)

    result = T14 / P

    # Assert all return values (float or in array) are <= upper_limit
    if type(result) is float:
        result = min(result, upper_limit)
    else:
        result[result > upper_limit] = upper_limit
    return result

def make_plot():
    # M8, G2, A5
    periods = numpy.geomspace(0.8, 1000, 2)
    plt.plot(periods, T14(R_s=1.7, M_s=2.1, P=periods), linestyle='dashed', color='red')
    periods = numpy.geomspace(0.5, 1000, 2)
    plt.plot(periods, T14(R_s=1, M_s=1, P=periods, small_planet=True), linestyle='dashed', color='red')
    periods = numpy.geomspace(0.1, 1000, 2)
    plt.plot(periods, T14(R_s=0.13, M_s=0.1, P=periods, small_planet=True), linestyle='dashed', color='red')

    plt.text(200, 0.0004, 'M8')
    plt.text(500, 0.001, 'G2')
    plt.text(500, 0.002, 'A5')

    plt.xlabel(r'Period (days)')
    plt.ylabel(r'Transit duration ($T_{\rm 14}/P$)')
    plt.xscale('log')
    plt.yscale('log')
    plt.xlim(0.2, 1000)
    plt.ylim(0.0003, 0.2)

make_plot();
```

We now add to the plot all known transiting exoplanets:

```python
make_plot()
def add_data():
    data = numpy.genfromtxt(
        'data_per_t14.csv',
        dtype='f8, f8',
        names = ['period', 't14'])
    plt.scatter(data['period'], data['t14'], s=10, alpha=0.5, edgecolor='black');
add_data()
```

We see that a significant amount of planets is actually located above the uppermost
of these lines, which can be attributed to bloated super-Jovian planets and/or to planets
transiting slightly evolved stars, for example. The absence of planets in the lower
left part of the diagram could be partly astrophysical in nature and interpreted as a
trace of planet formation and evolution (e.g. the absence of ultra-short period planets
around M dwarfs). That said, planets can naturally have transit durations that are
arbitrarily shorter than $T_{14,\rm{max}}$, e.g. on eccentric or inclined orbits.
The empty space can nevertheless be explained with a detection bias against planets
with transit durations of just a few minutes, e.g. $\sim{\,}15{\,}$min for $P=1{\,}$d and
$T_{14}/P=10^{-2}$.

In order to compensate for planets transiting evolved stars as well as for planets on
eccentric orbits (in eccentric cases the average orbital velocity in-transit
can be smaller or larger and the resulting transit duration can be larger or smaller
than in the circular case.) and other astrophysical effects that are potentially hard
to predict, we use the Figure to derive empirical estimates for the maximum
and minimum values of $d$ to be searched. We parameterize the upper limit via
the previous Equation and using $M_{\rm s}=1{\,}M_\odot$ and $R_{\rm s}=3.5{\,}R_\odot$,
and we parameterize the lower limit of $T_{14}$ using $M_{\rm s}=1{\,}M_\odot$ and
$R_{\rm s}=0.184\,R_\odot$. Note that these two parameterizations
do not correspond to any particular or even physically plausible star. The motivation
behind this parameterization is entirely empirical with the
aim of embracing all known transiting exoplanets.

```python
make_plot()
add_data()

# Upper limits
periods = numpy.geomspace(3.3, 1000, 2)
plt.plot(periods, T14(R_s=3.5, M_s=1, P=periods), color='red')
plt.plot((0.01, 3.3), (0.12, 0.12), color='red')

# Bottom border
periods = numpy.geomspace(0.1, 1000, 2)
plt.plot(periods, T14(R_s=0.184, M_s=1, P=periods, small_planet=True), color='red')
plt.plot(periods, T14(R_s=0.13, M_s=0.1, P=periods, small_planet=True), linestyle='dashed', color='red');
```

These empirical limits for the transit durations to be searched with TLS are
shown with inclined solid lines and their parameterization
was intentionally chosen to encompass all known transiting exoplanets. The horizontal
cutoff at $T_{14}/P \sim~0.12$ is a global threshold. This is the default
region in the $T_{14}$-$(P)$ diagram to be tested with our implementation of the TLS
algorithm. That said, the user can define their own range of transit durations to be tested.

tutorials/09 Optimal period grid and optimal duration grid.ipynb