# Regression in Python

***
This is a very quick run-through of some basic statistical concepts, adapted from [Lab 4 in Harvard's CS109](https://github.com/cs109/2015lab4) course. Please feel free to try the original lab if you're feeling ambitious :-) The CS109 git repository also has the solutions if you're stuck.

* Linear Regression Models
* Prediction using linear regression

Linear regression is used to model and predict continuous outcomes with normal random errors. There are nearly an infinite number of different types of regression models and each regression model is typically defined by the distribution of the prediction errors (called "residuals") of the type of data. Logistic regression is used to model binary outcomes whereas Poisson regression is used to predict counts. In this exercise, we'll see some examples of linear regression as well as Train-test splits.

The packages we'll cover are: `statsmodels`, `seaborn`, and `scikit-learn`. While we don't explicitly teach `statsmodels` and `seaborn` in the Springboard workshop, those are great libraries to know.
***

***

```python
# special IPython command to prepare the notebook for matplotlib and other libraries
%matplotlib inline

import numpy as np
import pandas as pd
import scipy.stats as stats
import matplotlib.pyplot as plt
import sklearn

import seaborn as sns

# special matplotlib argument for improved plots
from matplotlib import rcParams
sns.set_style("whitegrid")
sns.set_context("poster")

```

***
# Part 1: Introduction to Linear Regression
### Purpose of linear regression
***

 Given a dataset containing predictor variables $X$ and outcome/response variable $Y$, linear regression can be used to:

 Build a predictive model to predict future values of $\hat{Y}$, using new data $X^*$ where $Y$ is unknown.

 Model the strength of the relationship between each independent variable $X_i$ and $Y$

 Many times, only a subset of independent variables $X_i$ will have a linear relationship with $Y$

 Need to figure out which $X_i$ contributes most information to predict $Y$

It is in many cases, the first pass prediction algorithm for continuous outcomes.

### A Brief Mathematical Recap
***

[Linear Regression](http://en.wikipedia.org/wiki/Linear_regression) is a method to model the relationship between a set of independent variables $X$ (also knowns as explanatory variables, features, predictors) and a dependent variable $Y$.  This method assumes the relationship between each predictor $X$ is **linearly** related to the dependent variable $Y$. The most basic linear regression model contains one independent variable $X$, we'll call this the simple model.

$$ Y = \beta_0 + \beta_1 X + \epsilon$$

where $\epsilon$ is considered as an unobservable random variable that adds noise to the linear relationship. In linear regression, $\epsilon$ is assumed to be normally distributed with a mean of 0. In other words, what this means is that on average, if we know $Y$, a roughly equal number of predictions $\hat{Y}$ will be above $Y$ and others will be below $Y$. That is, on average, the error is zero. The residuals, $\epsilon$ are also assumed to be "i.i.d.": independently and identically distributed. Independence means that the residuals are not correlated -- the residual from one prediction has no effect on the residual from another prediction. Correlated errors are common in time series analysis and spatial analyses.

* $\beta_0$ is the intercept of the linear model and represents the average of $Y$ when all independent variables $X$ are set to 0.

* $\beta_1$ is the slope of the line associated with the regression model and represents the average effect of a one-unit increase in $X$ on $Y$.

* Back to the simple model. The model in linear regression is the *conditional mean* of $Y$ given the values in $X$ is expressed a linear function.

$$ y = f(x) = E(Y | X = x)$$

*Image from http://www.learner.org/courses/againstallodds/about/glossary.html. Note this image uses $\alpha$ and $\beta$ instead of $\beta_0$ and $\beta_1$.*

* The goal is to estimate the coefficients (e.g. $\beta_0$ and $\beta_1$). We represent the estimates of the coefficients with a "hat" on top of the letter.

$$ \hat{\beta}_0, \hat{\beta}_1 $$

* Once we estimate the coefficients $\hat{\beta}_0$ and $\hat{\beta}_1$, we can use these to predict new values of $Y$ given new data $X$.

$$\hat{y} = \hat{\beta}_0 + \hat{\beta}_1 x_1$$

* Multiple linear regression is when you have more than one independent variable and the estimation involves matrices
    * $X_1$, $X_2$, $X_3$, $\ldots$

* How do you estimate the coefficients?
    * There are many ways to fit a linear regression model
    * The method called **least squares** is the most common methods
    * We will discuss least squares

$$ Y = \beta_0 + \beta_1 X_1 + \ldots + \beta_p X_p + \epsilon$$

### Estimating $\hat\beta$: Least squares
***
[Least squares](http://en.wikipedia.org/wiki/Least_squares) is a method that can estimate the coefficients of a linear model by minimizing the squared residuals:

$$ \mathscr{L} = \sum_{i=1}^N \epsilon_i^2 = \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2  = \sum_{i=1}^N \left(y_i - \left(\beta_0 + \beta_1 x_i\right)\right)^2 $$

where $N$ is the number of observations and $\epsilon$ represents a residual or error, ACTUAL - PREDICTED.

#### Estimating the intercept $\hat{\beta_0}$ for the simple linear model

We want to minimize the squared residuals and solve for $\hat{\beta_0}$ so we take the partial derivative of $\mathscr{L}$ with respect to $\hat{\beta_0}$

$
\begin{align}
\frac{\partial \mathscr{L}}{\partial \hat{\beta_0}} &= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \epsilon^2 \\
&= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2 \\
&= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right)^2 \\
&= -2 \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right) \hspace{25mm} \mbox{(by chain rule)} \\
&= -2 \sum_{i=1}^N (y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i) \\
&= -2 \left[ \left( \sum_{i=1}^N y_i \right) - N \hat{\beta_0} - \hat{\beta}_1 \left( \sum_{i=1}^N x_i
\right) \right] \\
& 2 \left[ N \hat{\beta}_0 + \hat{\beta}_1 \sum_{i=1}^N x_i - \sum_{i=1}^N y_i \right] = 0 \hspace{20mm} \mbox{(Set equal to 0 and solve for $\hat{\beta}_0$)} \\
& N \hat{\beta}_0 + \hat{\beta}_1 \sum_{i=1}^N x_i - \sum_{i=1}^N y_i = 0 \\
& N \hat{\beta}_0 = \sum_{i=1}^N y_i - \hat{\beta}_1 \sum_{i=1}^N x_i \\
& \hat{\beta}_0 = \frac{\sum_{i=1}^N y_i - \hat{\beta}_1 \sum_{i=1}^N x_i}{N} \\
& \hat{\beta}_0 = \frac{\sum_{i=1}^N y_i}{N} - \hat{\beta}_1 \frac{\sum_{i=1}^N x_i}{N} \\
& \boxed{\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}}
\end{align}
$

Using this new information, we can compute the estimate for $\hat{\beta}_1$ by taking the partial derivative of $\mathscr{L}$ with respect to $\hat{\beta}_1$.

$
\begin{align}
\frac{\partial \mathscr{L}}{\partial \hat{\beta_1}} &= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \epsilon^2 \\
&= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2 \\
&= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right)^2 \\
&= 2 \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right) \left( -x_i \right) \hspace{25mm}\mbox{(by chain rule)} \\
&= -2 \sum_{i=1}^N x_i \left( y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i \right) \\
&= -2 \sum_{i=1}^N x_i (y_i - \hat{\beta}_0 x_i - \hat{\beta}_1 x_i^2) \\
&= -2 \sum_{i=1}^N x_i (y_i - \left( \bar{y} - \hat{\beta}_1 \bar{x} \right) x_i - \hat{\beta}_1 x_i^2) \\
&= -2 \sum_{i=1}^N (x_i y_i - \bar{y}x_i + \hat{\beta}_1\bar{x}x_i - \hat{\beta}_1 x_i^2) \\
&= -2 \left[ \sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i + \hat{\beta}_1\bar{x}\sum_{i=1}^N x_i - \hat{\beta}_1 \sum_{i=1}^N x_i^2 \right] \\
&= -2 \left[ \hat{\beta}_1 \left\{ \bar{x} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i^2 \right\} + \left\{ \sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i \right\}\right] \\
& 2 \left[ \hat{\beta}_1 \left\{ \sum_{i=1}^N x_i^2 - \bar{x} \sum_{i=1}^N x_i \right\} + \left\{ \bar{y} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i y_i \right\} \right] = 0 \\
& \hat{\beta}_1 = \frac{-\left( \bar{y} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i y_i \right)}{\sum_{i=1}^N x_i^2 - \bar{x}\sum_{i=1}^N x_i} \\
&= \frac{\sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i}{\sum_{i=1}^N x_i^2 - \bar{x} \sum_{i=1}^N x_i} \\
& \boxed{\hat{\beta}_1 = \frac{\sum_{i=1}^N x_i y_i - \bar{x}\bar{y}n}{\sum_{i=1}^N x_i^2 - n \bar{x}^2}}
\end{align}
$

The solution can be written in compact matrix notation as

$$\hat\beta =  (X^T X)^{-1}X^T Y$$

We wanted to show you this in case you remember linear algebra, in order for this solution to exist we need $X^T X$ to be invertible. Of course this requires a few extra assumptions, $X$ must be full rank so that $X^T X$ is invertible, etc. Basically, $X^T X$ is full rank if all rows and columns are linearly independent. This has a loose relationship to variables and observations being independent respective. **This is important for us because this means that having redundant features in our regression models will lead to poorly fitting (and unstable) models.** We'll see an implementation of this in the extra linear regression example.

***
# Part 2: Exploratory Data Analysis for Linear Relationships

The [Boston Housing data set](https://archive.ics.uci.edu/ml/datasets/Housing) contains information about the housing values in suburbs of Boston.  This dataset was originally taken from the StatLib library which is maintained at Carnegie Mellon University and is now available on the UCI Machine Learning Repository.

## Load the Boston Housing data set from `sklearn`
***

This data set is available in the [sklearn](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html#sklearn.datasets.load_boston) python module which is how we will access it today.

```python
from sklearn.datasets import load_boston
import pandas as pd

boston = load_boston()
```

```python
boston.keys()
```

    dict_keys(['data', 'target', 'feature_names', 'DESCR', 'filename'])

```python
boston.data.shape
```

    (506, 13)

```python
# Print column names
print(boston.feature_names)
```

    ['CRIM' 'ZN' 'INDUS' 'CHAS' 'NOX' 'RM' 'AGE' 'DIS' 'RAD' 'TAX' 'PTRATIO'
     'B' 'LSTAT']

```python
# Print description of Boston housing data set
print(boston.DESCR)
```

    .. _boston_dataset:

    Boston house prices dataset
    ---------------------------

    **Data Set Characteristics:**

        :Number of Instances: 506

        :Number of Attributes: 13 numeric/categorical predictive. Median Value (attribute 14) is usually the target.

        :Attribute Information (in order):
            - CRIM     per capita crime rate by town
            - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.
            - INDUS    proportion of non-retail business acres per town
            - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
            - NOX      nitric oxides concentration (parts per 10 million)
            - RM       average number of rooms per dwelling
            - AGE      proportion of owner-occupied units built prior to 1940
            - DIS      weighted distances to five Boston employment centres
            - RAD      index of accessibility to radial highways
            - TAX      full-value property-tax rate per $10,000
            - PTRATIO  pupil-teacher ratio by town
            - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town
            - LSTAT    % lower status of the population
            - MEDV     Median value of owner-occupied homes in $1000's

        :Missing Attribute Values: None

        :Creator: Harrison, D. and Rubinfeld, D.L.

    This is a copy of UCI ML housing dataset.
    https://archive.ics.uci.edu/ml/machine-learning-databases/housing/

    This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.

    The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic
    prices and the demand for clean air', J. Environ. Economics & Management,
    vol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics
    ...', Wiley, 1980.   N.B. Various transformations are used in the table on
    pages 244-261 of the latter.

    The Boston house-price data has been used in many machine learning papers that address regression
    problems.

    .. topic:: References

       - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.
       - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.

Now let's explore the data set itself.

```python
bos = pd.DataFrame(boston.data)
bos.head()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33

There are no column names in the DataFrame. Let's add those.

```python
bos.columns = boston.feature_names
bos.head()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33

Now we have a pandas DataFrame called `bos` containing all the data we want to use to predict Boston Housing prices.  Let's create a variable called `PRICE` which will contain the prices. This information is contained in the `target` data.

```python
print(boston.target.shape)
```

    (506,)

```python
bos['PRICE'] = boston.target
bos.head()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      PRICE

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98
      24.0

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14
      21.6

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03
      34.7

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94
      33.4

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33
      36.2

## EDA and Summary Statistics
***

Let's explore this data set.  First we use `describe()` to get basic summary statistics for each of the columns.

```python
bos.describe()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      PRICE

      count
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000

      mean
      3.613524
      11.363636
      11.136779
      0.069170
      0.554695
      6.284634
      68.574901
      3.795043
      9.549407
      408.237154
      18.455534
      356.674032
      12.653063
      22.532806

      std
      8.601545
      23.322453
      6.860353
      0.253994
      0.115878
      0.702617
      28.148861
      2.105710
      8.707259
      168.537116
      2.164946
      91.294864
      7.141062
      9.197104

      min
      0.006320
      0.000000
      0.460000
      0.000000
      0.385000
      3.561000
      2.900000
      1.129600
      1.000000
      187.000000
      12.600000
      0.320000
      1.730000
      5.000000

      25%
      0.082045
      0.000000
      5.190000
      0.000000
      0.449000
      5.885500
      45.025000
      2.100175
      4.000000
      279.000000
      17.400000
      375.377500
      6.950000
      17.025000

      50%
      0.256510
      0.000000
      9.690000
      0.000000
      0.538000
      6.208500
      77.500000
      3.207450
      5.000000
      330.000000
      19.050000
      391.440000
      11.360000
      21.200000

      75%
      3.677083
      12.500000
      18.100000
      0.000000
      0.624000
      6.623500
      94.075000
      5.188425
      24.000000
      666.000000
      20.200000
      396.225000
      16.955000
      25.000000

      max
      88.976200
      100.000000
      27.740000
      1.000000
      0.871000
      8.780000
      100.000000
      12.126500
      24.000000
      711.000000
      22.000000
      396.900000
      37.970000
      50.000000

### Scatterplots
***

Let's look at some scatter plots for three variables: 'CRIM' (per capita crime rate), 'RM' (number of rooms) and 'PTRATIO' (pupil-to-teacher ratio in schools).

```python
plt.scatter(bos.CRIM, bos.PRICE)
plt.xlabel("Per capita crime rate by town (CRIM)")
plt.ylabel("Housing Price")
plt.title("Relationship between CRIM and Price")
```

Part 2 Checkup Exercise Set I

Exercise: What kind of relationship do you see? e.g. positive, negative?  linear? non-linear? Is there anything else strange or interesting about the data? What about outliers?

Exercise: Create scatter plots between *RM* and *PRICE*, and *PTRATIO* and *PRICE*. Label your axes appropriately using human readable labels. Tell a story about what you see.

Exercise: What are some other numeric variables of interest? Why do you think they are interesting? Plot scatterplots with these variables and *PRICE* (house price) and tell a story about what you see.

## Relationship Between Crime Rate and Price
The relationship between crime rate per capita and housing price appears to be negative and non-linear.  There also appears to be a wide range of house values among areas where the crime rate is at or near 0%.  The highest-priced houses also appear to have a crime rate range of approximately 0-10%.

```python
# your turn: scatter plot between *RM* and *PRICE*
plt.scatter(bos.RM, bos.PRICE)
plt.xlabel("Average number of rooms per dwelling (RM)")
plt.ylabel("Housing Price")
plt.title("Relationship between RM and Price")
```

### Relationship Between Number of Rooms Per Dwelling and Price
The relationship between number of rooms and price appears to be positive and linear, other than a few outliers at various points in the data.

```python
# your turn: scatter plot between *PTRATIO* and *PRICE*
plt.scatter(bos.PTRATIO, bos.PRICE)
plt.xlabel("Pupil-Teacher Ratio by town (PTRATIO)")
plt.ylabel("Housing Price")
plt.title("Relationship between PTRATIO and Price")
```

### Relationship Between Pupil-Teacher Ratio and Price
The relationship between pupil-teacher relationship and price appears to be slightly-negative and slightly linear.  The data appears to have a widely distributed, though, we a trend of very low prices when the pupil-teacher ratio is at or around 20%.

```python
# your turn: create some other scatter plots
plt.scatter(bos.INDUS, bos.PRICE)
plt.xlabel("Proportion of non-retail business by town (INDUS)")
plt.ylabel("Housing Price")
plt.title("Relationship between INDUS and Price")
```

### Relationship Between Non-Retail Business Proportion by Town and Price
This relationship appears to be negative and non-linear.  There appears to be a number of high-valued housing in areas where there is very little industrial property and few data points of high value housing in high-industry towns.

```python
plt.scatter(bos.RAD, bos.PRICE)
plt.xlabel("Index of accessibility to radial highways (RAD)")
plt.ylabel("Housing Price")
plt.title("Relationship between RAD and Price")
```

### Relationship Between Accessibility to Radial Highways and Price
The relationship between these variables does not appear to be correlated positively or negatively.  There appear to be a bunch of houses with very low indexes of access to radial highway and then a small subset with high indexes.  The only possible correlation is with the set with high indexes of access, which appear to all have lower prices except for one outlier.

### Scatterplots using Seaborn
***

[Seaborn](https://stanford.edu/~mwaskom/software/seaborn/) is a cool Python plotting library built on top of matplotlib. It provides convenient syntax and shortcuts for many common types of plots, along with better-looking defaults.

We can also use [seaborn regplot](https://stanford.edu/~mwaskom/software/seaborn/tutorial/regression.html#functions-to-draw-linear-regression-models) for the scatterplot above. This provides automatic linear regression fits (useful for data exploration later on). Here's one example below.

```python
sns.regplot(y="PRICE", x="RM", data=bos, fit_reg = True)
```

### Histograms
***

```python
plt.hist(np.log(bos.CRIM))
plt.title("CRIM")
plt.xlabel("Crime rate per capita")
plt.ylabel("Frequencey")
plt.show()
```

Part 2 Checkup Exercise Set II

Exercise: In the above histogram, we took the logarithm of the crime rate per capita. Repeat this histogram without taking the log. What was the purpose of taking the log? What do we gain by making this transformation? What do you now notice about this variable that is not obvious without making the transformation?

Exercise: Plot the histogram for *RM* and *PTRATIO* against each other, along with the two variables you picked in the previous section. We are looking for correlations in predictors here.

```python
# Crime Histogram without log
plt.hist(bos.CRIM)
plt.title("CRIM")
plt.xlabel("Crime rate per capita")
plt.ylabel("Frequencey")
plt.show()
```

The logarithm histogram of the crime rate data provides us with much more of an understanding of the actual distribution of that variable.  We can see from the logarithm that the distribution appears to bimodal, with local means at or around -2.5 and 2.5.  We cannot observe the actual distribution of the data in the non-logarithm historgram.

```python
# RM scatter plots
plt.figure(figsize=(20,20))
plt.subplot(2,2,1)
plt.scatter(bos.RM, bos.PTRATIO)
plt.xlabel("RM")
plt.ylabel("PTRATIO")
plt.title("RM/PTRATIO Relationship")

plt.subplot(2,2,2)
plt.scatter(bos.RM, bos.INDUS)
plt.xlabel("RM")
plt.ylabel("INDUS")
plt.title("RM/INDUS Relationship")

plt.subplot(2,2,3)
plt.scatter(bos.RM, bos.RAD)
plt.xlabel("RM")
plt.ylabel("RAD")
plt.title("RM/RAD Relationship")
```

```python
# PTRATIO scatter plots
plt.figure(figsize=(20,20))
plt.subplot(2,2,1)
plt.scatter(bos.PTRATIO, bos.RM)
plt.xlabel("PTRATIO")
plt.ylabel("RM")
plt.title("PTRATIO/RM Relationship")

plt.subplot(2,2,2)
plt.scatter(bos.PTRATIO, bos.INDUS)
plt.xlabel("PTRATIO")
plt.ylabel("INDUS")
plt.title("PTRATIO/INDUS Relationship")

plt.subplot(2,2,3)
plt.scatter(bos.PTRATIO, bos.RAD)
plt.xlabel("PTRATIO")
plt.ylabel("RAD")
plt.title("PTRATIO/RAD Relationship")
```

## Part 3: Linear Regression with Boston Housing Data Example
***

Here,

$Y$ = boston housing prices (called "target" data in python, and referred to as the dependent variable or response variable)

and

$X$ = all the other features (or independent variables, predictors or explanatory variables)

which we will use to fit a linear regression model and predict Boston housing prices. We will use the least-squares method to estimate the coefficients.

We'll use two ways of fitting a linear regression. We recommend the first but the second is also powerful in its features.

### Fitting Linear Regression using `statsmodels`
***
[Statsmodels](http://statsmodels.sourceforge.net/) is a great Python library for a lot of basic and inferential statistics. It also provides basic regression functions using an R-like syntax, so it's commonly used by statisticians. While we don't cover statsmodels officially in the Data Science Intensive workshop, it's a good library to have in your toolbox. Here's a quick example of what you could do with it. The version of least-squares we will use in statsmodels is called *ordinary least-squares (OLS)*. There are many other versions of least-squares such as [partial least squares (PLS)](https://en.wikipedia.org/wiki/Partial_least_squares_regression) and [weighted least squares (WLS)](https://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares).

```python
# Import regression modules
import statsmodels.api as sm
from statsmodels.formula.api import ols
```

```python
# statsmodels works nicely with pandas dataframes
# The thing inside the "quotes" is called a formula, a bit on that below
m = ols('PRICE ~ RM',bos).fit()
print(m.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.484
    Model:                            OLS   Adj. R-squared:                  0.483
    Method:                 Least Squares   F-statistic:                     471.8
    Date:                Mon, 04 Nov 2019   Prob (F-statistic):           2.49e-74
    Time:                        20:04:29   Log-Likelihood:                -1673.1
    No. Observations:                 506   AIC:                             3350.
    Df Residuals:                     504   BIC:                             3359.
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept    -34.6706      2.650    -13.084      0.000     -39.877     -29.465
    RM             9.1021      0.419     21.722      0.000       8.279       9.925
    ==============================================================================
    Omnibus:                      102.585   Durbin-Watson:                   0.684
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              612.449
    Skew:                           0.726   Prob(JB):                    1.02e-133
    Kurtosis:                       8.190   Cond. No.                         58.4
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

#### Interpreting coefficients

There is a ton of information in this output. But we'll concentrate on the coefficient table (middle table). We can interpret the `RM` coefficient (9.1021) by first noticing that the p-value (under `P>|t|`) is so small, basically zero. This means that the number of rooms, `RM`, is a statisticall significant predictor of `PRICE`. The regression coefficient for `RM` of 9.1021 means that *on average, each additional room is associated with an increase of $\$9,100$ in house price net of the other variables*. The confidence interval gives us a range of plausible values for this average change, about ($\$8,279, \$9,925$), definitely not chump change.

In general, the $\hat{\beta_i}, i > 0$ can be interpreted as the following: "A one unit increase in $x_i$ is associated with, on average, a $\hat{\beta_i}$ increase/decrease in $y$ net of all other variables."

On the other hand, the interpretation for the intercept, $\hat{\beta}_0$ is the average of $y$ given that all of the independent variables $x_i$ are 0.

####  `statsmodels` formulas
***
This formula notation will seem familiar to `R` users, but will take some getting used to for people coming from other languages or are new to statistics.

The formula gives instruction for a general structure for a regression call. For `statsmodels` (`ols` or `logit`) calls you need to have a Pandas dataframe with column names that you will add to your formula. In the below example you need a pandas data frame that includes the columns named (`Outcome`, `X1`,`X2`, ...), but you don't need to build a new dataframe for every regression. Use the same dataframe with all these things in it. The structure is very simple:

`Outcome ~ X1`

But of course we want to to be able to handle more complex models, for example multiple regression is doone like this:

`Outcome ~ X1 + X2 + X3`

In general, a formula for an OLS multiple linear regression is

`Y ~ X1 + X2 + ... + Xp`

This is the very basic structure but it should be enough to get you through the homework. Things can get much more complex. You can force statsmodels to treat variables as categorical with the `C()` function, call numpy functions to transform data such as `np.log` for extremely-skewed data, or fit a model without an intercept by including `- 1` in the formula. For a quick run-down of further uses see the `statsmodels` [help page](http://statsmodels.sourceforge.net/devel/example_formulas.html).

Let's see how our model actually fit our data. We can see below that there is a ceiling effect, we should probably look into that. Also, for large values of $Y$ we get underpredictions, most predictions are below the 45-degree gridlines.

Part 3 Checkup Exercise Set I

Exercise: Create a scatterplot between the predicted prices, available in `m.fittedvalues` (where `m` is the fitted model) and the original prices. How does the plot look? Do you notice anything interesting or weird in the plot? Comment on what you see.

```python
plt.scatter(bos.PRICE, m.fittedvalues)
plt.xlabel("Original Prices")
plt.ylabel("Predicted Prices")
plt.title("Relationship Between Original and Predicted Prices")
```

### Fitting Linear Regression using `sklearn`

```python
from sklearn.linear_model import LinearRegression
X = bos.drop('PRICE', axis = 1)

# This creates a LinearRegression object
lm = LinearRegression()
lm
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,
             normalize=False)

#### What can you do with a LinearRegression object?
***
Check out the scikit-learn [docs here](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html). We have listed the main functions here. Most machine learning models in scikit-learn follow this same API of fitting a model with `fit`, making predictions with `predict` and the appropriate scoring function `score` for each model.

Main functions | Description
--- | ---
`lm.fit()` | Fit a linear model
`lm.predit()` | Predict Y using the linear model with estimated coefficients
`lm.score()` | Returns the coefficient of determination (R^2). *A measure of how well observed outcomes are replicated by the model, as the proportion of total variation of outcomes explained by the model*

#### What output can you get?

```python
# Look inside lm object
# lm.<tab>
```

Output | Description
--- | ---
`lm.coef_` | Estimated coefficients
`lm.intercept_` | Estimated intercept

### Fit a linear model
***

The `lm.fit()` function estimates the coefficients the linear regression using least squares.

```python
# Use all 13 predictors to fit linear regression model
lm.fit(X, bos.PRICE)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,
             normalize=False)

Part 3 Checkup Exercise Set II

Exercise: How would you change the model to not fit an intercept term? Would you recommend not having an intercept? Why or why not? For more information on why to include or exclude an intercept, look [here](https://stats.idre.ucla.edu/other/mult-pkg/faq/general/faq-what-is-regression-through-the-origin/).

Exercise: One of the assumptions of the linear model is that the residuals must be i.i.d. (independently and identically distributed). To satisfy this, is it enough that the residuals are normally distributed? Explain your answer.

Exercise: True or false. To use linear regression, $Y$ must be normally distributed. Explain your answer.

### Answers to above questions
You can change the model to not fit an intercept term by using the keyword argument `fit_intercept=False` when creating th `LinearRegression` object.  I would recommend using an intercept unless you have a compelling reason not to (such as that the Y value should equal 0 if all of the features also equal 0).

Verifying that all of the residuals in a model are normally distributed would satisfy that they are identically distributed, but not that they are independently distributed.  You would still need to investigate the independence of each residual.  For example, if you divided a categorical feature into dummy variables, you would need to eliminate one to maintain their independence (otherwise, totalling all of the variable columns would always equal 1 for each data point).

False, I do not believe that the Y values must be normally distributed.

### Estimated intercept and coefficients

Let's look at the estimated coefficients from the linear model using `1m.intercept_` and `lm.coef_`.

After we have fit our linear regression model using the least squares method, we want to see what are the estimates of our coefficients $\beta_0$, $\beta_1$, ..., $\beta_{13}$:

$$ \hat{\beta}_0, \hat{\beta}_1, \ldots, \hat{\beta}_{13} $$

```python
print('Estimated intercept coefficient: {}'.format(lm.intercept_))
```

    Estimated intercept coefficient: 36.459488385089735

```python
print('Number of coefficients: {}'.format(len(lm.coef_)))
```

    Number of coefficients: 13

```python
# The coefficients
pd.DataFrame({'features': X.columns, 'estimatedCoefficients': lm.coef_})[['features', 'estimatedCoefficients']]
```

      features
      estimatedCoefficients

      0
      CRIM
      -0.108011

      1
      ZN
      0.046420

      2
      INDUS
      0.020559

      3
      CHAS
      2.686734

      4
      NOX
      -17.766611

      5
      RM
      3.809865

      6
      AGE
      0.000692

      7
      DIS
      -1.475567

      8
      RAD
      0.306049

      9
      TAX
      -0.012335

      10
      PTRATIO
      -0.952747

      11
      B
      0.009312

      12
      LSTAT
      -0.524758

### Predict Prices

We can calculate the predicted prices ($\hat{Y}_i$) using `lm.predict`.

$$ \hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_1 + \ldots \hat{\beta}_{13} X_{13} $$

```python
# first five predicted prices
lm.predict(X)[0:5]
```

    array([ 30.00384338,  25.02556238,  30.56759672,  28.60703649,  27.94352423])

Part 3 Checkup Exercise Set III

Exercise: Histogram: Plot a histogram of all the predicted prices. Write a story about what you see. Describe the shape, center and spread of the distribution. Are there any outliers? What might be the reason for them? Should we do anything special with them?

Exercise: Scatterplot: Let's plot the true prices compared to the predicted prices to see they disagree (we did this with `statsmodels` before).

Exercise: We have looked at fitting a linear model in both `statsmodels` and `scikit-learn`. What are the advantages and disadvantages of each based on your exploration? Based on the information provided by both packages, what advantage does `statsmodels` provide?

```python
# Plotting histogram of predicted prices
plt.hist(lm.predict(X))
plt.title("Predicted Housing Prices")
plt.xlabel("Predicted Prices")
plt.ylabel("Frequencey")
plt.show()
```

The distribution appears to be a slightly-skewed left normal distribution, with the majority of housing pries falling at our around 20.  There appear to be a few outliers at either end of the spectrum.  The houses with predicted price of almost zero are a bit concerning and warrant further investigation.

```python
# Plotting scatterplot of predicted v. true prices
plt.scatter(bos.PRICE, lm.predict(X))
plt.xlabel("True Prices")
plt.ylabel("Predicted Prices")
plt.title("Relationship Between True and Predicted Prices")
```

Overall, it appears that the model provides a slightly lowers price prediction than the true price.  There also are some outliers at each price point, most dramatically underpredicting the value of the most expensive properties.

### Evaluating the Model: Sum-of-Squares

The partitioning of the sum-of-squares shows the variance in the predictions explained by the model and the variance that is attributed to error.

$$TSS = ESS + RSS$$

#### Residual Sum-of-Squares (aka $RSS$)

The residual sum-of-squares is one of the basic ways of quantifying how much error exists in the fitted model. We will revisit this in a bit.

$$ RSS = \sum_{i=1}^N r_i^2 = \sum_{i=1}^N \left(y_i - \left(\beta_0 + \beta_1 x_i\right)\right)^2 $$

```python
print(np.sum((bos.PRICE - lm.predict(X)) ** 2))
```

    11078.784578

#### Explained Sum-of-Squares (aka $ESS$)

The explained sum-of-squares measures the variance explained by the regression model.

$$ESS = \sum_{i=1}^N \left( \hat{y}_i - \bar{y} \right)^2 = \sum_{i=1}^N \left( \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) - \bar{y} \right)^2$$

```python
print(np.sum((lm.predict(X) - np.mean(bos.PRICE)) ** 2))
```

    31637.5108371

### Evaluating the Model: The Coefficient of Determination ($R^2$)

The coefficient of determination, $R^2$, tells us the percentage of the variance in the response variable $Y$ that can be explained by the linear regression model.

$$ R^2 = \frac{ESS}{TSS} $$

The $R^2$ value is one of the most common metrics that people use in describing the quality of a model, but it is important to note that *$R^2$ increases artificially as a side-effect of increasing the number of independent variables.* While $R^2$ is reported in almost all statistical packages, another metric called the *adjusted $R^2$* is also provided as it takes into account the number of variables in the model, and can sometimes even be used for non-linear regression models!

$$R_{adj}^2 = 1 - \left( 1 - R^2 \right) \frac{N - 1}{N - K - 1} = R^2 - \left( 1 - R^2 \right) \frac{K}{N - K - 1} = 1 - \frac{\frac{RSS}{DF_R}}{\frac{TSS}{DF_T}}$$

where $N$ is the number of observations, $K$ is the number of variables, $DF_R = N - K - 1$ is the degrees of freedom associated with the residual error and $DF_T = N - 1$ is the degrees of the freedom of the total error.

### Evaluating the Model: Mean Squared Error and the $F$-Statistic
***
The mean squared errors are just the *averages* of the sum-of-squares errors over their respective degrees of freedom.

$$MSE = \frac{RSS}{N-K-1}$$

$$MSR = \frac{ESS}{K}$$

**Remember:** Notation may vary across resources particularly the use of $R$ and $E$ in $RSS/ESS$ and $MSR/MSE$. In some resources, E = explained and R = residual. In other resources, E = error and R = regression (explained). **This is a very important distinction that requires looking at the formula to determine which naming scheme is being used.**

Given the MSR and MSE, we can now determine whether or not the entire model we just fit is even statistically significant. We use an $F$-test for this. The null hypothesis is that all of the $\beta$ coefficients are zero, that is, none of them have any effect on $Y$. The alternative is that *at least one* $\beta$ coefficient is nonzero, but it doesn't tell us which one in a multiple regression:

$$H_0: \beta_i = 0, \mbox{for all $i$} \\
H_A: \beta_i > 0, \mbox{for some $i$}$$

$$F = \frac{MSR}{MSE} = \left( \frac{R^2}{1 - R^2} \right) \left( \frac{N - K - 1}{K} \right)$$

Once we compute the $F$-statistic, we can use the $F$-distribution with $N-K$ and $K-1$ degrees of degrees of freedom to get a p-value.

**Warning!** The $F$-statistic mentioned in this section is NOT the same as the F1-measure or F1-value discused in Unit 7.

Part 3 Checkup Exercise Set IV

Let's look at the relationship between `PTRATIO` and housing price.

Exercise: Try fitting a linear regression model using only the 'PTRATIO' (pupil-teacher ratio by town) and interpret the intercept and the coefficients.

Exercise: Calculate (or extract) the $R^2$ value. What does it tell you?

Exercise: Compute the $F$-statistic. What does it tell you?

Exercise: Take a close look at the $F$-statistic and the $t$-statistic for the regression coefficient. What relationship do you notice? Note that this relationship only applies in *simple* linear regression models.

```python
# fitting linear regression model for PTRATIO
lm_ptratio = LinearRegression()
lm_ptratio.fit(X[['PTRATIO']], bos.PRICE)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,
             normalize=False)

```python
print('The intercept is {}'.format(lm_ptratio.intercept_))
print('The PTRATIO coefficient is {}'.format(lm_ptratio.coef_))
```

    The intercept is 62.344627474832706
    The PTRATIO coefficient is [-2.1571753]

```python
# calculate the R-squared value
RSS = np.sum((bos.PRICE - lm_ptratio.predict(X[['PTRATIO']])) ** 2)
ESS = np.sum((lm_ptratio.predict(X[['PTRATIO']]) - np.mean(bos.PRICE)) ** 2)
R_squared = ESS / (ESS + RSS)
print(R_squared)
```

    0.257847318009

```python
lm_ptratio.score(X[['PTRATIO']], bos.PRICE)
```

    0.257847318009223

The R-squared value indicates that approximately 25% of the variance in price is attributable to the PTRATIO variable.

```python
# calculate the F-statistic
MSE = RSS / (len(bos.PRICE) - 1)
MSR = ESS
F = MSR / MSE
print(F)
```

    175.452974508

With such a high F-statistic value, it is highly likely that the PTRATIO variable is statistically significant to price.

Part 3 Checkup Exercise Set V

Fit a linear regression model using three independent variables

 'CRIM' (per capita crime rate by town)

 'RM' (average number of rooms per dwelling)

 'PTRATIO' (pupil-teacher ratio by town)

Exercise: Compute or extract the $F$-statistic. What does it tell you about the model?

Exercise: Compute or extract the $R^2$ statistic. What does it tell you about the model?

Exercise: Which variables in the model are significant in predicting house price? Write a story that interprets the coefficients.

```python
# fitting linear regression model for CRIM, RM, and PTRATIO
X_three = X[['CRIM', 'RM', 'PTRATIO']]
lm_three = LinearRegression()
lm_three.fit(X_three, bos.PRICE)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,
             normalize=False)

```python
# calculate the R-squared value
RSS = np.sum((bos.PRICE - lm_three.predict(X_three)) ** 2)
ESS = np.sum((lm_three.predict(X_three) - np.mean(bos.PRICE)) ** 2)
R_squared = ESS / (ESS + RSS)
print(R_squared)
```

    0.594341294072

```python
lm_three.score(X_three, bos.PRICE)
```

    0.59434129407234726

The R-squared value indicates that approximately 59% of the variance in price is attributable to the three variables in our model.

```python
# calculate the F-statistic
MSE = RSS / (len(bos.PRICE) - len(X_three.columns) - 1)
MSR = ESS / len(X_three.columns)
F = MSR / MSE
print(F)
```

    245.164490301

With such a high F-statistic value, it is highly likely that the variables in the model are statistically significant to price.

```python
# displaying the coefficient values
pd.DataFrame({'features': X_three.columns, 'estimatedCoefficients': lm_three.coef_})[['features', 'estimatedCoefficients']]
```

      features
      estimatedCoefficients

      0
      CRIM
      -0.204961

      1
      RM
      7.380411

      2
      PTRATIO
      -1.069546

For every unit of CRIM, we can expect it to reduce the price by approximately 2050 dollars.  For every unit of RM, we can expect it to increase the house price by approximately 73800 dollars.  For every unit of PTRATIO, we can expect it to reduce the price by approximately 10700 dollars.

## Part 4: Comparing Models

During modeling, there will be times when we want to compare models to see which one is more predictive or fits the data better. There are many ways to compare models, but we will focus on two.

### The $F$-Statistic Revisited

The $F$-statistic can also be used to compare two *nested* models, that is, two models trained on the same dataset where one of the models contains a *subset* of the variables of the other model. The *full* model contains $K$ variables and the *reduced* model contains a subset of these $K$ variables. This allows us to add additional variables to a base model and then test if adding the variables helped the model fit.

$$F = \frac{\left( \frac{RSS_{reduced} - RSS_{full}}{DF_{reduced} - DF_{full}} \right)}{\left( \frac{RSS_{full}}{DF_{full}} \right)}$$

where $DF_x = N - K_x - 1$ where $K_x$ is the number of variables in model $x$.

### Akaike Information Criterion (AIC)

Another statistic for comparing two models is AIC, which is based on the likelihood function and takes into account the number of variables in the model.

$$AIC = 2 K - 2 \log_e{L}$$

where $L$ is the likelihood of the model. AIC is meaningless in the absolute sense, and is only meaningful when compared to AIC values from other models. Lower values of AIC indicate better fitting models.

`statsmodels` provides the AIC in its output.

Part 4 Checkup Exercises

Exercise: Find another variable (or two) to add to the model we built in Part 3. Compute the $F$-test comparing the two models as well as the AIC. Which model is better?

```python
# use statsmodel to build the new model
m = ols('PRICE ~ RM + PTRATIO + CRIM + INDUS + RAD',bos).fit()
print(m.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.606
    Model:                            OLS   Adj. R-squared:                  0.602
    Method:                 Least Squares   F-statistic:                     153.5
    Date:                Mon, 04 Nov 2019   Prob (F-statistic):           1.38e-98
    Time:                        20:24:42   Log-Likelihood:                -1604.9
    No. Observations:                 506   AIC:                             3222.
    Df Residuals:                     500   BIC:                             3247.
    Df Model:                           5
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept     -0.6608      4.178     -0.158      0.874      -8.870       7.548
    RM             6.9380      0.417     16.631      0.000       6.118       7.758
    PTRATIO       -0.9740      0.141     -6.884      0.000      -1.252      -0.696
    CRIM          -0.1732      0.039     -4.466      0.000      -0.249      -0.097
    INDUS         -0.1768      0.050     -3.544      0.000      -0.275      -0.079
    RAD            0.0169      0.046      0.369      0.712      -0.073       0.107
    ==============================================================================
    Omnibus:                      264.291   Durbin-Watson:                   0.809
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):             2546.594
    Skew:                           2.072   Prob(JB):                         0.00
    Kurtosis:                      13.179   Cond. No.                         421.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

From the results above, it appears that the intercept is not significant.  We can re-run the original three feature model and the new five feature model without the intercept to see what we get.

```python
# original three feature model
m_three = ols('PRICE ~ RM + PTRATIO + CRIM - 1',bos).fit()
print(m.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.944
    Model:                            OLS   Adj. R-squared:                  0.943
    Method:                 Least Squares   F-statistic:                     1682.
    Date:                Mon, 04 Nov 2019   Prob (F-statistic):          2.05e-310
    Time:                        20:54:26   Log-Likelihood:                -1604.9
    No. Observations:                 506   AIC:                             3220.
    Df Residuals:                     501   BIC:                             3241.
    Df Model:                           5
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    RM             6.8836      0.237     29.104      0.000       6.419       7.348
    PTRATIO       -0.9908      0.093    -10.690      0.000      -1.173      -0.809
    CRIM          -0.1736      0.039     -4.492      0.000      -0.250      -0.098
    INDUS         -0.1788      0.048     -3.700      0.000      -0.274      -0.084
    RAD            0.0188      0.044      0.426      0.670      -0.068       0.105
    ==============================================================================
    Omnibus:                      263.894   Durbin-Watson:                   0.808
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):             2528.947
    Skew:                           2.070   Prob(JB):                         0.00
    Kurtosis:                      13.139   Cond. No.                         25.4
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

```python
# new five feature model
m = ols('PRICE ~ RM + PTRATIO + CRIM + INDUS + RAD - 1',bos).fit()
print(m.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.944
    Model:                            OLS   Adj. R-squared:                  0.943
    Method:                 Least Squares   F-statistic:                     1682.
    Date:                Mon, 04 Nov 2019   Prob (F-statistic):          2.05e-310
    Time:                        20:54:29   Log-Likelihood:                -1604.9
    No. Observations:                 506   AIC:                             3220.
    Df Residuals:                     501   BIC:                             3241.
    Df Model:                           5
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    RM             6.8836      0.237     29.104      0.000       6.419       7.348
    PTRATIO       -0.9908      0.093    -10.690      0.000      -1.173      -0.809
    CRIM          -0.1736      0.039     -4.492      0.000      -0.250      -0.098
    INDUS         -0.1788      0.048     -3.700      0.000      -0.274      -0.084
    RAD            0.0188      0.044      0.426      0.670      -0.068       0.105
    ==============================================================================
    Omnibus:                      263.894   Durbin-Watson:                   0.808
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):             2528.947
    Skew:                           2.070   Prob(JB):                         0.00
    Kurtosis:                      13.139   Cond. No.                         25.4
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

In comparing the two models, it appears that the addition of the INDUS variable slightly improved the model's fit, increasing the r-squared value from 0.942 to 0.944 and reducing the AIC from 3231 to 3220.

## Part 5: Evaluating the Model via Model Assumptions and Other Issues
***
Linear regression makes several assumptions. It is always best to check that these assumptions are valid after fitting a linear regression model.

**Linearity**. The dependent variable $Y$ is a linear combination of the regression coefficients and the independent variables $X$. This can be verified with a scatterplot of each $X$ vs. $Y$ and plotting correlations among $X$. Nonlinearity can sometimes be resolved by [transforming](https://onlinecourses.science.psu.edu/stat501/node/318) one or more independent variables, the dependent variable, or both. In other cases, a [generalized linear model](https://en.wikipedia.org/wiki/Generalized_linear_model) or a [nonlinear model](https://en.wikipedia.org/wiki/Nonlinear_regression) may be warranted.

**Constant standard deviation**. The SD of the dependent variable $Y$ should be constant for different values of X. We can check this by plotting each $X$ against $Y$ and verifying that there is no "funnel" shape showing data points fanning out as $X$ increases or decreases. Some techniques for dealing with non-constant variance include weighted least squares (WLS), [robust standard errors](https://en.wikipedia.org/wiki/Heteroscedasticity-consistent_standard_errors), or variance stabilizing transformations.

 **Normal distribution for errors**.  The $\epsilon$ term we discussed at the beginning are assumed to be normally distributed. This can be verified with a fitted values vs. residuals plot and verifying that there is no pattern, and with a quantile plot.
  $$ \epsilon_i \sim N(0, \sigma^2)$$
Sometimes the distributions of responses $Y$ may not be normally distributed at any given value of $X$.  e.g. skewed positively or negatively.

 **Independent errors**.  The observations are assumed to be obtained independently.

e.g. Observations across time may be correlated

There are some other issues that are important investigate with linear regression models.

**Correlated Predictors:** Care should be taken to make sure that the independent variables in a regression model are not too highly correlated. Correlated predictors typically do not majorly affect prediction, but do inflate standard errors of coefficients making interpretation unreliable. Common solutions are dropping the least important variables involved in the correlations, using regularlization, or, when many predictors are highly correlated, considering a dimension reduction technique such as principal component analysis (PCA).

**Influential Points:** Data points that have undue influence on the regression model. These points can be high leverage points or outliers. Such points are typically removed and the regression model rerun.

Part 5 Checkup Exercises

Take the reduced model from Part 3 to answer the following exercises. Take a look at [this blog post](http://mpastell.com/2013/04/19/python_regression/) for more information on using statsmodels to construct these plots.

Exercise: Construct a fitted values versus residuals plot. What does the plot tell you? Are there any violations of the model assumptions?

Exercise: Construct a quantile plot of the residuals. What does the plot tell you?

Exercise: What are some advantages and disadvantages of the fitted vs. residual and quantile plot compared to each other?

Exercise: Identify any outliers (if any) in your model and write a story describing what these outliers might represent.

Exercise: Construct a leverage plot and identify high leverage points in the model. Write a story explaining possible reasons for the high leverage points.

Exercise: Remove the outliers and high leverage points from your model and run the regression again. How do the results change?

```python
# create the model
m = ols('PRICE ~ RM',bos).fit()
print(m.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.484
    Model:                            OLS   Adj. R-squared:                  0.483
    Method:                 Least Squares   F-statistic:                     471.8
    Date:                Mon, 04 Nov 2019   Prob (F-statistic):           2.49e-74
    Time:                        21:00:27   Log-Likelihood:                -1673.1
    No. Observations:                 506   AIC:                             3350.
    Df Residuals:                     504   BIC:                             3359.
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept    -34.6706      2.650    -13.084      0.000     -39.877     -29.465
    RM             9.1021      0.419     21.722      0.000       8.279       9.925
    ==============================================================================
    Omnibus:                      102.585   Durbin-Watson:                   0.684
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              612.449
    Skew:                           0.726   Prob(JB):                    1.02e-133
    Kurtosis:                       8.190   Cond. No.                         58.4
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

```python
# plot the residuals and fitted values
sns.set()
plt.scatter(bos['RM'], bos['PRICE'])
plt.plot(bos['RM'], m.fittedvalues, color='red')
plt.xlabel("Average number of rooms per dwelling (RM)")
plt.ylabel("Housing Price")
plt.title("Relationship between RM and Price")
plt.legend(['Data', 'Fitted Model'], loc='lower right')
```

```python
# quantile plot of residuals
plt.hist(m.fittedvalues)
plt.ylabel('Count')
plt.xlabel('residuals')
```

```python
# plot a leverage plot
from statsmodels.graphics.regressionplots import *
plot_leverage_resid2(m)
```

```python
bos.loc[364:375]
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      PRICE

      364
      3.47428
      0.0
      18.1
      1.0
      0.718
      8.780
      82.9
      1.9047
      24.0
      666.0
      20.2
      354.55
      5.29
      21.9

      365
      4.55587
      0.0
      18.1
      0.0
      0.718
      3.561
      87.9
      1.6132
      24.0
      666.0
      20.2
      354.70
      7.12
      27.5

      366
      3.69695
      0.0
      18.1
      0.0
      0.718
      4.963
      91.4
      1.7523
      24.0
      666.0
      20.2
      316.03
      14.00
      21.9

      367
      13.52220
      0.0
      18.1
      0.0
      0.631
      3.863
      100.0
      1.5106
      24.0
      666.0
      20.2
      131.42
      13.33
      23.1

      368
      4.89822
      0.0
      18.1
      0.0
      0.631
      4.970
      100.0
      1.3325
      24.0
      666.0
      20.2
      375.52
      3.26
      50.0

      369
      5.66998
      0.0
      18.1
      1.0
      0.631
      6.683
      96.8
      1.3567
      24.0
      666.0
      20.2
      375.33
      3.73
      50.0

      370
      6.53876
      0.0
      18.1
      1.0
      0.631
      7.016
      97.5
      1.2024
      24.0
      666.0
      20.2
      392.05
      2.96
      50.0

      371
      9.23230
      0.0
      18.1
      0.0
      0.631
      6.216
      100.0
      1.1691
      24.0
      666.0
      20.2
      366.15
      9.53
      50.0

      372
      8.26725
      0.0
      18.1
      1.0
      0.668
      5.875
      89.6
      1.1296
      24.0
      666.0
      20.2
      347.88
      8.88
      50.0

      373
      11.10810
      0.0
      18.1
      0.0
      0.668
      4.906
      100.0
      1.1742
      24.0
      666.0
      20.2
      396.90
      34.77
      13.8

      374
      18.49820
      0.0
      18.1
      0.0
      0.668
      4.138
      100.0
      1.1370
      24.0
      666.0
      20.2
      396.90
      37.97
      13.8

      375
      19.60910
      0.0
      18.1
      0.0
      0.671
      7.313
      97.9
      1.3163
      24.0
      666.0
      20.2
      396.90
      13.44
      15.0

It appears that the data points at index 364:372 are outliers.  Perhaps it is a high-price neighborhood with unique characteristics.  Let's remove these datapoints from the dataset and re-run the model.

```python
# remove outliers from dataset
bos_no_out = bos.drop([364, 365, 366, 367, 368, 369, 370, 371, 372])
bos_no_out.shape
```

    (497, 14)

```python
# create the no outliers model
m_no_out = ols('PRICE ~ RM',bos_no_out).fit()
print(m.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.606
    Model:                            OLS   Adj. R-squared:                  0.605
    Method:                 Least Squares   F-statistic:                     761.1
    Date:                Mon, 04 Nov 2019   Prob (F-statistic):          3.67e-102
    Time:                        22:01:23   Log-Likelihood:                -1557.2
    No. Observations:                 497   AIC:                             3118.
    Df Residuals:                     495   BIC:                             3127.
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept    -42.0954      2.346    -17.947      0.000     -46.704     -37.487
    RM            10.2230      0.371     27.589      0.000       9.495      10.951
    ==============================================================================
    Omnibus:                       39.804   Durbin-Watson:                   0.722
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):               55.966
    Skew:                          -0.602   Prob(JB):                     7.03e-13
    Kurtosis:                       4.119   Cond. No.                         61.0
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

We can see immediately that the no outlier model has improved significantly.  The R-squared value has increased from 0.484 to 0.606, the F-statistic has increased from 471.8 to 761.1, and the AIC has reduced from 3350 to 3118.  Let's observe the graphs for the no outlier model:

```python
# plot the residuals and fitted values for the no outlier model
sns.set()
plt.scatter(bos_no_out['RM'], bos_no_out['PRICE'])
plt.plot(bos_no_out['RM'], m_no_out.fittedvalues, color='red')
plt.xlabel("Average number of rooms per dwelling (RM)")
plt.ylabel("Housing Price")
plt.title("Relationship between RM and Price")
plt.legend(['Data', 'Fitted Model'], loc='lower right')
```

```python
# quantile plot of residuals for no outlier model
plt.hist(m_no_out.fittedvalues)
plt.ylabel('Count')
plt.xlabel('residuals')
```

```python
# leverage plot for no outlier model
plot_leverage_resid2(m_no_out)
```

The graphs demonstrate that the new model has a much more consistent dataset, though the leverage plot shows that further datapoints could potentially be investigated as additional outliers that we may want to remove.

```python

```


# Lecture 1 - Introduction to Uncertainty Quantification

> Ignorance is preferable to error and he is less remote from the truth who believes nothing than he who believes what is wrong.
Thomas Jefferson (1781)

## Objectives
+ To tell the difference between **aleatory** and **epistemic** uncertainties.
+ To define **predictive modeling**.
+ To get accustomed with the processes of **verification**, **calibration**, and **validation** of a computer code.
+ To use **probability theory** to represent both aleatory and epistemic uncertainties.

## Readings

Before coming to class, please read the following:

+ [Oden, Moser, Ghattas, Computer Predictions with Quantified Uncertainty, Part I](http://www.siam.org/pdf/news/1842.pdf)

+ [Oden, Moser, Ghattas, Computer Predictions with Quantified Uncertainty, Part II](http://www.siam.org/pdf/news/1857.pdf)

## Definitions

We are not going to make a big effort to be consistent about the use of the following terms, since their precise meaning is still under debate.

### Uncertainty
In general, we are uncertain about a logical proposition if we do not know whether it is true or false.
In particular, we can be uncertain about:
+ the value of a model parameter
+ the mathematical form of a model
+ the initial conditions of a ordinary differntial equations
+ the boundary conditions of a partial differential equation
+ the value of an experimental measurment we are about to perform
+ etc.

Uncertainty may be *aleatory* or *epistemic*. Aleatory uncertainty is associated with inherent system randomness. Epistemic uncertainty is associated with lack of knowledge. If you think too hard, the distinction between the two becomes philosophical. We are not going to push this too hard. Fortunately, our approach (the Bayesian approach) treats both uncertainty on an equal footing.

### Predictive Modeling
*Predictive modeling* is the process of assigning error bars to the predictions of computational models which rigorously quantify the effect of all (ideally) associated uncertainties. This quantified uncertainty can be used to assess the risk of making decisions based on these model predictions.

### Verification
Verification is the process of *convincing yourself* that a computer code is actually solving the right mathematical equations. Even though it is impossible to be 100% sure that a complicated computer code works as intended, it is easy to verify that it does not.

### Calibration
Calibration is the process of fitting the parameters of a computer code to experimental data. It is most important problem of uncertainty quantification. The difficulties arise from the experimental noise, insufficient data (observing part of the system), model discrepancy and more.

### Validation
Validation is the process of comparing the model predictions to experimental data not used in the calibration process.

## Example: Catalytic Conversion of Nitrate to Nitrogen

This is Example 3.1 of [(Tsilifis, 2014)](http://arxiv.org/abs/1410.5522).

Consider the catalytic
conversion of nitrate ($\mbox{NO}_3^-$) to nitrogen ($\mbox{N}_2$) and other
by-products by electrochemical means.
The mechanism that is followed is complex and not well understood.
The experiment of \cite{katsounaros} confirmed the
production of nitrogen ($\mbox{N}_2$), ammonia
($\mbox{NH}_3$), and nitrous oxide ($\mbox{N}_2\mbox{O}$) as final products
of the reaction, as well as the intermediate production of nitrite ($\mbox{NO}_2^-$).
The data are reproduced in [Comma-separated values](https://en.wikipedia.org/wiki/Comma-separated_values) (CSV) and stored in
[data/catalysis.csv](data/catalysis.csv).
The time is measured in minutes and the conentrations are measured in $\mbox{mmol}\cdot\mbox{L}^{-1}$.
Let's load the data into this notebook using the [Pandas](http://pandas.pydata.org) Python module:

```python
import pandas as pd
catalysis_data = pd.read_csv('../data/catalysis.csv', index_col=0)
catalysis_data
```

      NO3
      NO2
      N2
      NH3
      N2O

      Time

      0
      500.00
      0.00
      0.00
      0.00
      0.00

      30
      250.95
      107.32
      18.51
      3.33
      4.98

      60
      123.66
      132.33
      74.85
      7.34
      20.14

      90
      84.47
      98.81
      166.19
      13.14
      42.10

      120
      30.24
      38.74
      249.78
      19.54
      55.98

      150
      27.94
      10.42
      292.32
      24.07
      60.65

      180
      13.54
      6.11
      309.50
      27.26
      62.54

Let's visualize the data using [Matplotlib](http://matplotlib.org):

```python
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
%matplotlib inline
catalysis_data.plot()
```

The theory of catalytic reactions guarantees that the total mass must be conserved.
However, this is not the case in our dataset:

```python
catalysis_data.sum(axis=1)
```

    Time
    0      500.00
    30     385.09
    60     358.32
    90     404.71
    120    394.28
    150    415.40
    180    418.95
    dtype: float64

This inconsistency suggests the existence of an intermediate unobserved reaction product X.
[(Katsounaros, 2012)](http://www.sciencedirect.com/science/article/pii/S0013468612005208) suggested that the following reaction path shown in the following figure.

The dynamical system associated with the reaction is:
$$
\begin{array}{cc}
\frac{d \left[\mbox{NO}_3^-\right]}{dt} &= -k_1\left[\mbox{NO}_3^-\right], \\
\frac{d\left[\mbox{NO}_2^-\right]}{dt} &= k_1\left[\mbox{NO}_3^-\right] - (k_2 + k_4 +
k_5)[\mbox{NO}_2^-], \\
\frac{d \left[\mbox{X}\right]}{dt} &= k_2 \left[\mbox{NO}_2^-\right] - k_3 [X],\\
\frac{d \left[\mbox{N}_2\right]}{dt} &= k_3 \left[\mbox{X}\right], \\
\frac{d \left[\mbox{NH}_3\right]}{dt} &= k_4 \left[\mbox{NO}_2^-\right],\\
\frac{d \left[\mbox{N}_2O\right]}{dt} &= k_5 \left[\mbox{NO}_2^-\right],
\end{array}
$$
where $[\cdot]$ denotes the concentration of a quantity, and
$k_i > 0$, $i=1,...5$ are the *kinetic rate constants*.

### Questions

1) Assume that you are a chemical engineer and that you are assigned the task of designing a reactor for the conversion of nitrate to nitrogen. Before you start designing, you collect on information in an attempt to characterize your state of knowledge about the problem. How many different sources of uncertainty can you think of?

2) Which of these uncertainties would you characterize as aleatoric uncertainties and which as epistemic?

3) Is the distinction between aleatory and epistemic uncertainties always clear cut?

### Computational Model

We will develop a generic computational model for the solution of dynamical systems and we will use it to study the catalysis problem. The code relies on the [Fourth-order Runge-Kutta method](https://en.wikipedia.org/wiki/Runge–Kutta_methods) and is a modified copy of [http://www.math-cs.gordon.edu/courses/ma342/python/diffeq.py](http://www.math-cs.gordon.edu/courses/ma342/python/diffeq.py) developed by Jonathan Senning. The code solves:

$$
\begin{array}{ccc}
\dot{\mathbf{y}} &=& f(\mathbf{y}, t),\\
\mathbf{y}(0) &=& \mathbf{y}_0.
\end{array}
$$

```python
import numpy as np
def rk45( f, y0, t, args=() ):
    """Fourth-order Runge-Kutta method with error estimate.

    USAGE:
        y = rk45(f, x0, t, args=())

    INPUT:
        f     - function of x and t equal to dx/dt.  x may be multivalued,
                in which case it should a list or a NumPy array.  In this
                case f must return a NumPy array with the same dimension
                as x.
        y0    - the initial condition(s).  Specifies the value of x when
                t = t[0].  Can be either a scalar or a list or NumPy array
                if a system of equations is being solved.
        t     - list or NumPy array of t values to compute solution at.
                t[0] is the the initial condition point, and the difference
                h=t[i+1]-t[i] determines the step size h.
        args  - any other parameters of the function f.

    OUTPUT:
        y     - NumPy array containing solution values corresponding to each
                entry in t array.  If a system is being solved, x will be
                an array of arrays.

    NOTES:
        This version is based on the algorithm presented in "Numerical
        Mathematics and Computing" 6th Edition, by Cheney and Kincaid,
        Brooks-Cole, 2008.
    """

    # Coefficients used to compute the independent variable argument of f

    c20  =   2.500000000000000e-01  #  1/4
    c30  =   3.750000000000000e-01  #  3/8
    c40  =   9.230769230769231e-01  #  12/13
    c50  =   1.000000000000000e+00  #  1
    c60  =   5.000000000000000e-01  #  1/2

    # Coefficients used to compute the dependent variable argument of f

    c21 =   2.500000000000000e-01  #  1/4
    c31 =   9.375000000000000e-02  #  3/32
    c32 =   2.812500000000000e-01  #  9/32
    c41 =   8.793809740555303e-01  #  1932/2197
    c42 =  -3.277196176604461e+00  # -7200/2197
    c43 =   3.320892125625853e+00  #  7296/2197
    c51 =   2.032407407407407e+00  #  439/216
    c52 =  -8.000000000000000e+00  # -8
    c53 =   7.173489278752436e+00  #  3680/513
    c54 =  -2.058966861598441e-01  # -845/4104
    c61 =  -2.962962962962963e-01  # -8/27
    c62 =   2.000000000000000e+00  #  2
    c63 =  -1.381676413255361e+00  # -3544/2565
    c64 =   4.529727095516569e-01  #  1859/4104
    c65 =  -2.750000000000000e-01  # -11/40

    # Coefficients used to compute 4th order RK estimate

    a1  =   1.157407407407407e-01  #  25/216
    a2  =   0.000000000000000e-00  #  0
    a3  =   5.489278752436647e-01  #  1408/2565
    a4  =   5.353313840155945e-01  #  2197/4104
    a5  =  -2.000000000000000e-01  # -1/5

    b1  =   1.185185185185185e-01  #  16.0/135.0
    b2  =   0.000000000000000e-00  #  0
    b3  =   5.189863547758284e-01  #  6656.0/12825.0
    b4  =   5.061314903420167e-01  #  28561.0/56430.0
    b5  =  -1.800000000000000e-01  # -9.0/50.0
    b6  =   3.636363636363636e-02  #  2.0/55.0

    n = len( t )
    y = np.array( [ y0 ] * n )
    for i in xrange( n - 1 ):
        h = t[i+1] - t[i]
        k1 = h * f( y[i], t[i], *args )
        k2 = h * f( y[i] + c21 * k1, t[i] + c20 * h, *args )
        k3 = h * f( y[i] + c31 * k1 + c32 * k2, t[i] + c30 * h, *args )
        # BUG: The ``-`` in the equation below should be a ``+``.
        k4 = h * f( y[i] - c41 * k1 + c42 * k2 + c43 * k3, t[i] + c40 * h, *args )
        k5 = h * f( y[i] + c51 * k1 + c52 * k2 + c53 * k3 + c54 * k4, \
                        t[i] + h, *args )
        k6 = h * f( \
            y[i] + c61 * k1 + c62 * k2 + c63 * k3 + c64 * k4 + c65 * k5, \
            t[i] + c60 * h, *args )

        y[i+1] = y[i] + a1 * k1 + a3 * k3 + a4 * k4 + a5 * k5
        y5 = y[i] + b1 * k1 + b3 * k3 + b4 * k4 + b5 * k5 + b6 * k6

    return y
```

### Code Verification
It is obvious that even a small typo in the code above will have a significant impact in the solution. How do we know that the code actually solves the right equation? Well, we can't... However, we can convince ourselves about it. We need to find a case of a dynamical system with a known solution. Consider the ODE governing a forced harmonic oscillator:
$$
\begin{array}{ccl}
\ddot{y} &=& -k y + A\sin(\omega t),\\
y(0) &=& 0,\\
\dot{y}(0) &=& 1.
\end{array}
$$
This can be solved analytically. Let's use [SymPy](http://docs.sympy.org/dev/index.html) to solve it:

```python
import sympy
from sympy import init_printing
init_printing()
from sympy import Function, dsolve, Eq, Derivative, sin, symbols
from sympy.abc import t
t, k, A, omega = sympy.symbols('t k A omega', real = True, positive = True)
y = Function('y')
generic_solution = dsolve(Derivative(y(t), t, t) + k * y(t) - A * sin(omega * t), y(t))
generic_solution
```

```python
# Apply the initial conditions
constants = sympy.solve((generic_solution.rhs.subs(t, 0), generic_solution.rhs.diff(t).subs(t, 0) - 1), ('C1', 'C2'))
solution = generic_solution.rhs.subs(constants)
solution
```

```python
# Turn this into a Python function that we can actually evaluate
from sympy.utilities.autowrap import ufuncify
y_true = ufuncify([t, k, A, omega], solution)
```

To solve this initial value problem numerically using ``rk45``, we must first turn it into a system. This is done by setting $y_1 = y$ and $y_2 = \dot{y_1}$:
$$
\begin{array}{ccl}
\dot{y}_1 &=& y_2,\\
\dot{y}_2 &=& -k y_1 + A\sin(\omega t),\\
y_1(0) &=& 0,\\
y_2(0) &=& 1.
\end{array}
$$
Thus $\mathbf{f}(\mathbf{y}, t) := \mathbf{f}(y_1, y_2, t;k, A, \omega) = (y_2, -k y_1 + A \sin(\omega t))$.

```python
def f(y, t, k, A, omega):
    """
    RHS of the ODE we want to solve.

    k, A, and omega are model parameters.
    """
    return np.array([y[1], -k * y[0] + A * np.sin(omega * t)])

def plot_ode_sol(T = 30., k = 0.5, A = 1., omega = 2., nt = 10):
    """
    Compare the numerical solution to the analytical one.
    """
    t = np.linspace(0, T, int(nt))
    y = rk45(f, (0., 1.), t, args=(k, A, omega))
    plt.plot(t, y[:, 0], label='Numerical solution')
    plt.plot(t, y_true(t, k, A, omega), '--', label='Analytical solution')
    plt.legend()
```

Use the [interactive widget](https://github.com/ipython/ipywidgets) below to verify that our code acutally solves the right mathematical equations.

### Questions
4) Can you verify (using the widget below) that our code solves the right mathematical equation?

5) If you can't, skim through the code of ``rk45`` to find a BUG and fix it. It should be fairly easy. The rerun the cell defininig the solver and play with the widget again.

5) What is the parameter that controls the accuracy of the solution?

```python
from ipywidgets import interactive
interactive(plot_ode_sol, T = (20, 100), k = [0., 1.], A = [0., 2.], omega = [0., 2], nt = [10, 100])
```

## Calibrating the Catalysis Model to the Experimental Data

Now that we are certain that our generic ODE solver works, let us use it to develop a solver for the catalysis model. All, we need to do is define the right hand side of Eq. (\ref{eq:kinetic_model}):

```python
def f_catalysis(y, t, kappa):
    rhs = np.zeros((6,))
    rhs[0] = -kappa[0] * y[0]
    rhs[1] = kappa[0] * y[0] - (kappa[1] + kappa[3] + kappa[4]) * y[1]
    rhs[2] = kappa[1] * y[1] - kappa[2] * y[2]
    rhs[3] = kappa[2] * y[2]
    rhs[4] = kappa[3] * y[1]
    rhs[5] = kappa[4] * y[1]
    return rhs
```

Let's try to calibrate the parameters of the model to the data, manually. Because the parameters are two small, let us work with the transformed version:

$$
\xi_i = \log\left(\frac{k_i}{180}\right).
$$

```python
def compare_model_to_data(xi1 = 1.359, xi2 = 1.657, xi3 = 1.347, xi4 = -.162, xi5 = -1.009):
    """
    Compare the model predictions to the data.
    """
    t = np.linspace(0, 180, 100)
    kappa = np.exp([xi1, xi2, xi3, xi4, xi5]) / 180.
    y = rk45(f_catalysis, (500., 0., 0., 0., 0., 0.), t, args=(kappa,))
    fig, ax = plt.subplots(figsize=(10, 10))
    catalysis_data.plot(ax=ax, style='s')
    ax.plot(t, y[:, 0], color=sns.color_palette()[0], label='Model NO3-')
    ax.plot(t, y[:, 1], color=sns.color_palette()[1], label='Model NO2-')
    ax.plot(t, y[:, 2], color=sns.color_palette()[5], label='Model X')
    ax.plot(t, y[:, 3], color=sns.color_palette()[2], label='Model N2')
    ax.plot(t, y[:, 4], color=sns.color_palette()[3], label='Model NH3')
    ax.plot(t, y[:, 5], color=sns.color_palette()[4], label='Model N2O')
    plt.legend()

interactive(compare_model_to_data, xi1 = (-2, 2, 0.05), xi2 = (-2, 2, 0.05), xi3 = (-2, 2, 0.05),
                                   xi4 = (-2, 2, 0.05), xi5 = (-2, 2, 0.05) )
```

This is the calibration problem.

### Questions

1) Obviously, you do not want to be calibrating models by hand. Can you think of a *natural* way to calibrate a model?

2) No matter what we do, we cannot really match the data to the model exactly? List at least two reasons why this is the case?

### Uncertainty Propagation

As discussed in Question 2 above, there various reasons why a model cannot be calibrated perfectly. Some of these are:

+ lack of data;
+ the existence of measurement noise;
+ the fact that the model is just not perfect.

Ignoring for the moment the possibility that the moment is just bluntly wrong, we see that the lack of data or the presence of noise will induce some uncertainty in the values of the calibrated parameters. We are going to represent uncertainty on parameters by assigning a probability density on them. There are systematic ways of estimating the uncertainty induced because of the calibration process, but this will not concern us now.
For the moment, assume that somebody told us that the uncertainty in the scaled parameters $\xi_i$ of the model is as follows:

| Variable | Value |
|---------|------------------|
| $\xi_1$ |$1.35\pm 0.05$ |
| $\xi_2$ |$1.65\pm 0.08$   |
| $\xi_3$ |$1.34\pm 0.11$ |
| $\xi_4$ |$-0.16\pm 0.16$ |
| $\xi_5$ |$-3.84\pm 0.20$ |

But what does this information actually mean? As we will discuss in the following lectures, this information can be used to assign a probability density on each one of these parameters, say $p(\xi_i)$, that *models* our state of knowledge about them. For example, let us assume that our state of knowledge about $\xi_1$ is given by a Gaussian probability density:

$$
p(\xi_1) = \mathcal{N}(\xi_1|\mu_1=1.35, \sigma^2 = 0.05^2),
$$

which we can visualize as follows:

```python
import scipy.stats
from scipy.stats import norm
xi1 = np.linspace(-0, 2, 200)
plt.plot(xi1, norm.pdf(xi1, loc=1.35, scale=0.05))
```

This means that we do not beleive that the value of the parameter can be less than 1.0 or greater than 1.6. Note that, we are deliberately trying to avoid the use of the term "random". There is nothing random in our example. Probability models a state of knowledge.

How does this uncertainty propagate throw the model? We will study this question with a simple numerical experiment. We are going to assign Gaussian probability densities on all the $\xi_i$'s, sample them a few times, and run our catalysis model for each one.

```python
def plot_samples(mu1 = 1.359, sig1=0.055,
                mu2 = 1.657, sig2=0.086,
                mu3 = 1.347, sig3=0.118,
                mu4 = -.162, sig4=0.167,
                mu5 = -1.009, sig5=0.368,
                num_samples=10):
    """
    Take a few samples of the model to study uncertainty propagation.
    """
    fig, ax = plt.subplots(figsize=(10, 10))
    catalysis_data.plot(ax=ax, style='s')
    t = np.linspace(0, 180, 100)
    for i in xrange(num_samples):
        xi1 = norm.rvs(loc=mu1, scale=sig1)
        xi2 = norm.rvs(loc=mu2, scale=sig2)
        xi3 = norm.rvs(loc=mu3, scale=sig3)
        xi4 = norm.rvs(loc=mu4, scale=sig4)
        xi5 = norm.rvs(loc=mu5, scale=sig5)
        kappa = np.exp([xi1, xi2, xi3, xi4, xi5]) / 180.
        y = rk45(f_catalysis, (500., 0., 0., 0., 0., 0.), t, args=(kappa,))
        ax.plot(t, y[:, 0], linewidth=0.5, color=sns.color_palette()[0])#, label='Model NO3-')
        ax.plot(t, y[:, 1], linewidth=0.5, color=sns.color_palette()[1])#, label='Model NO2-')
        ax.plot(t, y[:, 2], linewidth=0.5, color=sns.color_palette()[5])#, label='Model X')
        ax.plot(t, y[:, 3], linewidth=0.5, color=sns.color_palette()[2])#, label='Model N2')
        ax.plot(t, y[:, 4], linewidth=0.5, color=sns.color_palette()[3])#, label='Model NH3')
        ax.plot(t, y[:, 5], linewidth=0.5, color=sns.color_palette()[4])#, label='Model N2O')
    plt.legend()

interactive(plot_samples, mu1 = (-2, 2, 0.05), sig1=(0.02, 0.4, 0.01),
                                   mu2 = (-2, 2, 0.05), sig2=(0.02, 0.4, 0.01),
                                   mu3 = (-2, 2, 0.05), sig3=(0.02, 0.4, 0.01),
                                   mu4 = (-2, 2, 0.05), sig4=(0.02, 0.4, 0.01),
                                   mu5 = (-2, 2, 0.05), sig5=(0.02, 0.4, 0.01))
```

### Questions

1) It was pretty easy to propagate uncertainty through this simple model. When would you face difficulties with such a program?

2) Can you come up with any idea of accelerating the uncertainty propagation process?

# References

[1] Katsounaros I, Dortsiou M, Polatides C et al., ``_Reaction pathways in the electrochemical reduction of nitrate on tin_'', Electrochimica Acta, vol. 71, number , pp. 270--276,  2012.


# Resposta a Degrau Circuito RLC paralelo

Jupyter Notebook desenvolvido por [Gustavo S.S.](https://github.com/GSimas)

Consideremos o circuito RLC em paralelo, mostrado na Figura 8.22.

Aplicando a LKC ao nó superior para t > 0:

\begin{align}
{\Large \frac{v}{R} + i + C \frac{dv}{dt}}
\end{align}

Porém:

\begin{align}
{\Large v = L \frac{di}{dt}}
\end{align}

Assim, substituindo v e reorganizando a equação, temos:

\begin{align}
{\Large \frac{d^2 i}{dt^2} + \frac{1}{RC} \frac{di}{dt} + \frac{i}{LC} = \frac{I_s}{LC}}
\end{align}

A solução completa para a Equação consiste na resposta transiente
it(t) e da resposta de estado estável iss; ou seja:

\begin{align}
{\Large i(t) = i_t(t) + i_{ss}(t)}
\end{align}

A resposta transiente é a resposta natural (regime transitório). A resposta de estado estável é a resposta forçada (regime permanente). Assim:

\begin{align}
{\Large i(t) = I_s + A_1 e^{s_1 t} + A_2 e^{s_2 t} \space \space Superamortecido}
\\
\\{\Large i(t) = I_s + (A_1 + A_2t)e^{-\alpha t} \space \space Amortecimento \space Critico}
\\
\\{\Large i(t) = I_s + (A_1 cos(\omega_d t) + A_2 sin(\omega_d t))e^{-\alpha t} \space \space Subamortecido}
\end{align}

Onde:

\begin{align}
{\Large \alpha = \frac{1}{2RC}}
\\
\\{\Large \omega_0 = \frac{1}{\sqrt{LC}}}
\\
\\{\Large \omega_d = \sqrt{\omega_0^2 - \alpha^2}}
\\
\\{\Large s_{1,2} = -\alpha \pm \sqrt{\alpha^2 - \omega_0^2}}
\end{align}

De forma alternativa, a resposta completa para qualquer variável x(t) pode ser encontrada diretamente:

\begin{align}
\\{\Large x(t) = x_{ss}(t) + x_t(t)}
\end{align}

**Exemplo 8.23**

No circuito da Figura 8.23, determine i(t) e iR(t) para t > 0.

```python
print("Exemplo 8.8")

from sympy import *

m = 10**(-3)
C = 8*m
L = 20
Is = 4
Vs = 30

t = symbols('t')
A1 = symbols('A1')
A2 = symbols('A2')

def sqrt(x,root=2): #definicao de funcao para calculo de raiz quadrada
    y = x**(1/root)
    return y

#Para t < 0
i0 = Is
v0 = Vs*20/(20 + 20)

print("i(0):",i0,"A")
print("v(0):",v0,"V")

#Para t > 0
R = 20*20/(20 + 20) #Req paralelo

def rlc_paralelo(R,L,C):
    alpha = 1/(2*R*C)
    omega0 = 1/sqrt(L*C)

    print("Alpha:",alpha,"Np/s")
    print("Omega0:",omega0,"rad/s")

    s1 = -alpha + sqrt(alpha**2 - omega0**2)
    s2 = -alpha - sqrt(alpha**2 - omega0**2)
    omegad = sqrt(omega0**2 - alpha**2)

    if alpha > omega0:
        resposta = "Superamortecido"
        i = Is + A1*exp(s1*t) + A2*exp(s2*t)

    elif alpha == omega0:
        resposta = "Amortecimento Crítico"
        i = Is + (A1 + A2*t)*exp(-alpha*t)

    else:
        resposta = "Subamortecido"
        i = Is + (A1*cos(omegad*t) + A2*sin(omegad*t))*exp(-alpha*t)

    print("Tipo de resposta:",resposta)
    print("i(t):",i,"A")
    print("i(0):",i.subs(t,0),"A")
    print("di(0)/dt:",diff(i,t).subs(t,0))

    return alpha, omega0, omegad, resposta, s1, s2, i

alpha, omega0, omegad, resposta, s1, s2, i = rlc_paralelo(R,L,C)

#i0 = A1 + A2 + 4 = 4
    #A1 = -A2

print("di(0)/dt:",v0/L,"A/s")

#di(0)/dt = -0.52A1 - 11.98A2 =
    #0.52A2 - 11.98A2 = 0.75
A_2 = 0.75/(0.52 - 11.98)
A_1 = -A_2

print("Constante A1:",A_1)
print("Constante A2:",A_2)

i = i.subs(A1,A_1).subs(A2,A_2)

print("i(t)",i,"A")

vl = L*diff(i,t)
ir = vl/20

print("ir(t):",ir,"A")
```

    Exemplo 8.8
    i(0): 4 A
    v(0): 15.0 V
    Alpha: 6.25 Np/s
    Omega0: 2.5 rad/s
    Tipo de resposta: Superamortecido
    i(t): A1*exp(-0.5217803813052*t) + A2*exp(-11.9782196186948*t) + 4 A
    i(0): A1 + A2 + 4 A
    di(0)/dt: -0.5217803813052*A1 - 11.9782196186948*A2
    di(0)/dt: 0.75 A/s
    Constante A1: 0.06544502617801047
    Constante A2: -0.06544502617801047
    i(t) 4 - 0.0654450261780105*exp(-11.9782196186948*t) + 0.0654450261780105*exp(-0.5217803813052*t) A
    ir(t): 0.78391489651144*exp(-11.9782196186948*t) - 0.0341479307136911*exp(-0.5217803813052*t) A

**Problema Prático 8.8**

Determine i(t) e v(t) para t > 0 no circuito da Figura 8.24.

```python
print("Problema Prático 8.8")

C = 0.2
L = 20
Is = 10

#Para t < 0
i0 = 0
v0 = oo

print("i(0):",i0,"A")
print("v(0):",v0,"V")

#Para t = inf
i_f = Is
v_f = 0

#Para t > 0
R = oo

alpha, omega0, omegad, resposta, s1, s2, i = rlc_paralelo(R,L,C)

#i0 = A1 + 10 = 0
A_1 = -10

#di(0)/dt = vl(0)/L
print("di(0)/t:",0/L,"A/s")
A_2 = 0

i = i.subs(A1,A_1).subs(A2,A_2)

print("i(t):",i,"A")

vl = L*diff(i,t)

print("v(t):",vl,"V")
```

    Problema Prático 8.8
    i(0): 0 A
    v(0): oo V
    Alpha: 0 Np/s
    Omega0: 0.5 rad/s
    Tipo de resposta: Subamortecido
    i(t): A2*sin(0.5*t) - 10*cos(0.5*t) + 10 A
    i(0): 0 A
    di(0)/dt: 0.5*A2
    di(0)/t: 0.0 A/s
    i(t): -10*cos(0.5*t) + 10 A
    v(t): 100.0*sin(0.5*t) V

Aula 16 - Resposta a Degrau RLC Paralelo.ipynb