# Regression in Python

***
This is a very quick run-through of some basic statistical concepts, adapted from [Lab 4 in Harvard's CS109](https://github.com/cs109/2015lab4) course. Please feel free to try the original lab if you're feeling ambitious :-) The CS109 git repository also has the solutions if you're stuck.

* Linear Regression Models
* Prediction using linear regression

Linear regression is used to model and predict continuous outcomes with normal random errors. There are nearly an infinite number of different types of regression models and each regression model is typically defined by the distribution of the prediction errors (called "residuals") of the type of data. Logistic regression is used to model binary outcomes whereas Poisson regression is used to predict counts. In this exercise, we'll see some examples of linear regression as well as Train-test splits.

The packages we'll cover are: `statsmodels`, `seaborn`, and `scikit-learn`. While we don't explicitly teach `statsmodels` and `seaborn` in the Springboard workshop, those are great libraries to know.
***

***

```python
# special IPython command to prepare the notebook for matplotlib and other libraries
%matplotlib inline

import numpy as np
import pandas as pd
import scipy.stats as stats
import matplotlib.pyplot as plt
import sklearn

import seaborn as sns

# special matplotlib argument for improved plots
from matplotlib import rcParams
sns.set_style("whitegrid")
sns.set_context("poster")

```

***
# Part 1: Introduction to Linear Regression
### Purpose of linear regression
***

 Given a dataset containing predictor variables $X$ and outcome/response variable $Y$, linear regression can be used to:

 Build a predictive model to predict future values of $\hat{Y}$, using new data $X^*$ where $Y$ is unknown.

 Model the strength of the relationship between each independent variable $X_i$ and $Y$

 Many times, only a subset of independent variables $X_i$ will have a linear relationship with $Y$

 Need to figure out which $X_i$ contributes most information to predict $Y$

It is in many cases, the first pass prediction algorithm for continuous outcomes.

### A Brief Mathematical Recap
***

[Linear Regression](http://en.wikipedia.org/wiki/Linear_regression) is a method to model the relationship between a set of independent variables $X$ (also knowns as explanatory variables, features, predictors) and a dependent variable $Y$.  This method assumes the relationship between each predictor $X$ is **linearly** related to the dependent variable $Y$. The most basic linear regression model contains one independent variable $X$, we'll call this the simple model.

$$ Y = \beta_0 + \beta_1 X + \epsilon$$

where $\epsilon$ is considered as an unobservable random variable that adds noise to the linear relationship. In linear regression, $\epsilon$ is assumed to be normally distributed with a mean of 0. In other words, what this means is that on average, if we know $Y$, a roughly equal number of predictions $\hat{Y}$ will be above $Y$ and others will be below $Y$. That is, on average, the error is zero. The residuals, $\epsilon$ are also assumed to be "i.i.d.": independently and identically distributed. Independence means that the residuals are not correlated -- the residual from one prediction has no effect on the residual from another prediction. Correlated errors are common in time series analysis and spatial analyses.

* $\beta_0$ is the intercept of the linear model and represents the average of $Y$ when all independent variables $X$ are set to 0.

* $\beta_1$ is the slope of the line associated with the regression model and represents the average effect of a one-unit increase in $X$ on $Y$.

* Back to the simple model. The model in linear regression is the *conditional mean* of $Y$ given the values in $X$ is expressed a linear function.

$$ y = f(x) = E(Y | X = x)$$

http://www.learner.org/courses/againstallodds/about/glossary.html

* The goal is to estimate the coefficients (e.g. $\beta_0$ and $\beta_1$). We represent the estimates of the coefficients with a "hat" on top of the letter.

$$ \hat{\beta}_0, \hat{\beta}_1 $$

* Once we estimate the coefficients $\hat{\beta}_0$ and $\hat{\beta}_1$, we can use these to predict new values of $Y$ given new data $X$.

$$\hat{y} = \hat{\beta}_0 + \hat{\beta}_1 x_1$$

* Multiple linear regression is when you have more than one independent variable and the estimation involves matrices
    * $X_1$, $X_2$, $X_3$, $\ldots$

* How do you estimate the coefficients?
    * There are many ways to fit a linear regression model
    * The method called **least squares** is the most common methods
    * We will discuss least squares

$$ Y = \beta_0 + \beta_1 X_1 + \ldots + \beta_p X_p + \epsilon$$

### Estimating $\hat\beta$: Least squares
***
[Least squares](http://en.wikipedia.org/wiki/Least_squares) is a method that can estimate the coefficients of a linear model by minimizing the squared residuals:

$$ \mathscr{L} = \sum_{i=1}^N \epsilon_i = \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2  = \sum_{i=1}^N \left(y_i - \left(\beta_0 + \beta_1 x_i\right)\right)^2 $$

where $N$ is the number of observations and $\epsilon$ represents a residual or error, ACTUAL - PREDICTED.

#### Estimating the intercept $\hat{\beta_0}$ for the simple linear model

We want to minimize the squared residuals and solve for $\hat{\beta_0}$ so we take the partial derivative of $\mathscr{L}$ with respect to $\hat{\beta_0}$

$
\begin{align}
\frac{\partial \mathscr{L}}{\partial \hat{\beta_0}} &= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \epsilon^2 \\
&= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2 \\
&= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right)^2 \\
&= -2 \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right) \hspace{25mm} \mbox{(by chain rule)} \\
&= -2 \sum_{i=1}^N y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i \\
&= -2 \left[ \left( \sum_{i=1}^N y_i \right) - n \hat{\beta_0} - \hat{\beta}_1 \left( \sum_{i=1}^N x_i
\right) \right] \\
& 2 \left[ n \hat{\beta}_0 + \hat{\beta}_1 \sum_{i=1}^N x_i - \sum_{i=1}^N y_i \right] = 0 \hspace{20mm} \mbox{(Set equal to 0 and solve for $\hat{\beta}_0$)} \\
& n \hat{\beta}_0 + \hat{\beta}_1 \sum_{i=1}^N x_i - \sum{i=1}^N y_i = 0 \\
& n \hat{\beta}_0 = \sum_{i=1}^N y_i - \hat{\beta}_1 \sum_{i=1}^N x_i \\
& \hat{\beta}_0 = \frac{\sum_{i=1}^N y_i - \hat{\beta}_1 \sum_{i=1}^N x_i}{n} \\
& \hat{\beta}_0 = \frac{\sum_{i=1}^N y_i}{n} - \hat{\beta}_1 \frac{\sum_{i=1}^N x_i}{n} \\
& \boxed{\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}}
\end{align}
$

Using this new information, we can compute the estimate for $\hat{\beta}_1$ by taking the partial derivative of $\mathscr{L}$ with respect to $\hat{\beta}_1$.

$
\begin{align}
\frac{\partial \mathscr{L}}{\partial \hat{\beta_1}} &= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \epsilon^2 \\
&= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2 \\
&= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right)^2 \\
&= 2 \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right) \left( -x_i \right) \hspace{25mm}\mbox{(by chain rule)} \\
&= -2 \sum_{i=1}^N x_i \left( y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i \right) \\
&= -2 \sum_{i=1}^N x_i y_i - \hat{\beta}_0 x_i - \hat{\beta}_1 x_i^2 \\
&= -2 \sum_{i=1}^N x_i y_i - \left( \bar{y} - \hat{\beta}_1 \bar{x} \right) x_i - \hat{\beta}_1 x_i^2 \\
&= -2 \sum_{i=1}^N x_i y_i - \bar{y}x_i + \hat{\beta}_1\bar{x}x_i - \hat{\beta}_1 x_i^2 \\
&= -2 \left[ \sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i + \hat{\beta}_1\bar{x} - \hat{\beta}_1 x_i^2 \right] \\
&= -2 \left[ \hat{\beta}_1 \left\{ \bar{x} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i^2 \right\} + \left\{ \sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i \right\}\right] \\
& 2 \left[ \hat{\beta}_1 \left\{ \sum_{i=1}^N x_i^2 - \bar{x} \sum_{i=1}^N x_i \right\} + \left\{ \bar{y} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i y_i \right\} \right] = 0 \\
& \hat{\beta}_1 = \frac{-\left( \bar{y} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i y_i \right)}{\sum_{i=1}^N x_i^2 - \bar{x}\sum_{i=1}^N x_i} \\
&= \frac{\sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i}{\sum_{i=1}^N x_i^2 - \bar{x} \sum_{i=1}^N x_i} \\
& \boxed{\hat{\beta}_1 = \frac{\sum_{i=1}^N x_i y_i - \bar{x}\bar{y}n}{\sum_{i=1}^N x_i^2 - n \bar{x}^2}}
\end{align}
$

The solution can be written in compact matrix notation as

$$\hat\beta =  (X^T X)^{-1}X^T Y$$

We wanted to show you this in case you remember linear algebra, in order for this solution to exist we need $X^T X$ to be invertible. Of course this requires a few extra assumptions, $X$ must be full rank so that $X^T X$ is invertible, etc. Basically, $X^T X$ is full rank if all rows and columns are linearly independent. This has a loose relationship to variables and observations being independent respective. **This is important for us because this means that having redundant features in our regression models will lead to poorly fitting (and unstable) models.** We'll see an implementation of this in the extra linear regression example.

***
# Part 2: Exploratory Data Analysis for Linear Relationships

The [Boston Housing data set](https://archive.ics.uci.edu/ml/datasets/Housing) contains information about the housing values in suburbs of Boston.  This dataset was originally taken from the StatLib library which is maintained at Carnegie Mellon University and is now available on the UCI Machine Learning Repository.

## Load the Boston Housing data set from `sklearn`
***

This data set is available in the [sklearn](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html#sklearn.datasets.load_boston) python module which is how we will access it today.

```python
from sklearn.datasets import load_boston
import pandas as pd

boston = load_boston()
```

```python
boston.keys()
```

    dict_keys(['data', 'target', 'feature_names', 'DESCR'])

```python
boston.data.shape
```

    (506, 13)

```python
# Print column names
print(boston.feature_names)
```

    ['CRIM' 'ZN' 'INDUS' 'CHAS' 'NOX' 'RM' 'AGE' 'DIS' 'RAD' 'TAX' 'PTRATIO'
     'B' 'LSTAT']

```python
# Print description of Boston housing data set
print(boston.DESCR)
```

    Boston House Prices dataset
    ===========================

    Notes
    ------
    Data Set Characteristics:

        :Number of Instances: 506

        :Number of Attributes: 13 numeric/categorical predictive

        :Median Value (attribute 14) is usually the target

        :Attribute Information (in order):
            - CRIM     per capita crime rate by town
            - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.
            - INDUS    proportion of non-retail business acres per town
            - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
            - NOX      nitric oxides concentration (parts per 10 million)
            - RM       average number of rooms per dwelling
            - AGE      proportion of owner-occupied units built prior to 1940
            - DIS      weighted distances to five Boston employment centres
            - RAD      index of accessibility to radial highways
            - TAX      full-value property-tax rate per $10,000
            - PTRATIO  pupil-teacher ratio by town
            - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town
            - LSTAT    % lower status of the population
            - MEDV     Median value of owner-occupied homes in $1000's

        :Missing Attribute Values: None

        :Creator: Harrison, D. and Rubinfeld, D.L.

    This is a copy of UCI ML housing dataset.
    http://archive.ics.uci.edu/ml/datasets/Housing

    This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.

    The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic
    prices and the demand for clean air', J. Environ. Economics & Management,
    vol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics
    ...', Wiley, 1980.   N.B. Various transformations are used in the table on
    pages 244-261 of the latter.

    The Boston house-price data has been used in many machine learning papers that address regression
    problems.

    **References**

       - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.
       - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.
       - many more! (see http://archive.ics.uci.edu/ml/datasets/Housing)

Now let's explore the data set itself.

```python
bos = pd.DataFrame(boston.data)
bos.head()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33

There are no column names in the DataFrame. Let's add those.

```python
bos.columns = boston.feature_names
bos.head()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33

Now we have a pandas DataFrame called `bos` containing all the data we want to use to predict Boston Housing prices.  Let's create a variable called `PRICE` which will contain the prices. This information is contained in the `target` data.

```python
print(boston.target.shape)
```

    (506,)

```python
bos['PRICE'] = boston.target
bos.head()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      PRICE

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98
      24.0

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14
      21.6

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03
      34.7

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94
      33.4

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33
      36.2

## EDA and Summary Statistics
***

Let's explore this data set.  First we use `describe()` to get basic summary statistics for each of the columns.

```python
bos.describe()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      PRICE

      count
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000

      mean
      3.593761
      11.363636
      11.136779
      0.069170
      0.554695
      6.284634
      68.574901
      3.795043
      9.549407
      408.237154
      18.455534
      356.674032
      12.653063
      22.532806

      std
      8.596783
      23.322453
      6.860353
      0.253994
      0.115878
      0.702617
      28.148861
      2.105710
      8.707259
      168.537116
      2.164946
      91.294864
      7.141062
      9.197104

      min
      0.006320
      0.000000
      0.460000
      0.000000
      0.385000
      3.561000
      2.900000
      1.129600
      1.000000
      187.000000
      12.600000
      0.320000
      1.730000
      5.000000

      25%
      0.082045
      0.000000
      5.190000
      0.000000
      0.449000
      5.885500
      45.025000
      2.100175
      4.000000
      279.000000
      17.400000
      375.377500
      6.950000
      17.025000

      50%
      0.256510
      0.000000
      9.690000
      0.000000
      0.538000
      6.208500
      77.500000
      3.207450
      5.000000
      330.000000
      19.050000
      391.440000
      11.360000
      21.200000

      75%
      3.647423
      12.500000
      18.100000
      0.000000
      0.624000
      6.623500
      94.075000
      5.188425
      24.000000
      666.000000
      20.200000
      396.225000
      16.955000
      25.000000

      max
      88.976200
      100.000000
      27.740000
      1.000000
      0.871000
      8.780000
      100.000000
      12.126500
      24.000000
      711.000000
      22.000000
      396.900000
      37.970000
      50.000000

### Scatterplots
***

Let's look at some scatter plots for three variables: 'CRIM' (per capita crime rate), 'RM' (number of rooms) and 'PTRATIO' (pupil-to-teacher ratio in schools).

```python
plt.scatter(bos.CRIM, bos.PRICE)
plt.xlabel("Per capita crime rate by town (CRIM)")
plt.ylabel("Housing Price")
plt.title("Relationship between CRIM and Price")
```

Part 2 Checkup Exercise Set I

Exercise: What kind of relationship do you see? e.g. positive, negative?  linear? non-linear? Is there anything else strange or interesting about the data? What about outliers?

Exercise: Create scatter plots between *RM* and *PRICE*, and *PTRATIO* and *PRICE*. Label your axes appropriately using human readable labels. Tell a story about what you see.

Exercise: What are some other numeric variables of interest? Why do you think they are interesting? Plot scatterplots with these variables and *PRICE* (house price) and tell a story about what you see.

The **CRIM vs Price** plot above shows a negative correlation between housing prices and crime rate. The downtrend appears to show that lower housing prices in Boston suburbs suffer from a much higher crime rate. On the other hand, it is also possible to notice that neightbourhoods with the most expensive houses have a lower crime rate. Also, there might be a few outliers in the plot responsible for the highest crime rates.

The **RM vs Price** plot below on the left, shows a strong positive correlation between number of rooms and housing price. It makes sense for houses with more rooms to have a higher cost so this relationship was expected. The value of the land and the neightbourhood can also play an important factor on the final price of the house but this relationship is not shown here. There is a good linear relationship between these two variables and a linear regression model to predict housing price based on number of rooms could be a good decision.

The **PTRATIO vs Price** plot below on the right, seems to show a weak negative correlation between the pupil-teacher ratio and housing price. We can see that there are a lot of occurences with a PTRATIO of ~20. Most of them are for *mid-to-low* housing prices but a significant few also appear for the highest housing prices. Another interesting observation is that houses in the *mid-to-high* price range have the lowest PTRATIO. This means that the more expensive a house is, the more attention from a teacher a pupil might receive.

```python
# karl
sns.set() # initialize seaborn style for matplotlib
plt.figure(figsize=(15,5))

# your turn: scatter plot between *RM* and *PRICE*
plt.subplot(1, 2, 1)
plt.scatter(bos.RM, bos.PRICE, s=15, alpha=0.3)
plt.xlabel("average number of rooms per dwelling (RM)")
plt.ylabel("Housing Price")
plt.title("Relationship between RM and Price")

# your turn: scatter plot between *PTRATIO* and *PRICE*
plt.subplot(1, 2, 2)
plt.scatter(bos.PTRATIO, bos.PRICE, s=15, alpha=0.3)
plt.xlabel("pupil-teacher ratio by town (PTRATIO)")
plt.ylabel("Housing Price")
plt.title("Relationship between PTRATIO and Price")

plt.tight_layout()
plt.show()
```

The **DIS vs Price** plot below shows that the majority of the houses are located between 4 units of distance from one of the five Boston employment centres. It is also possible to notice that most cheap houses are actually located near employment centres. This plot shows a positive correlation between distance of a house from an employment centre and housing price.

```python
# karl
# your turn: create some other scatter plots
plt.figure(figsize=(7.5,5))

plt.scatter(bos.DIS, bos.PRICE, s=15, alpha=0.3)
plt.xlabel("weighted distances to five Boston employment centres")
plt.ylabel("Housing Price")
plt.title("Relationship between DIS and Price")
plt.show()
```

### Scatterplots using Seaborn
***

[Seaborn](https://stanford.edu/~mwaskom/software/seaborn/) is a cool Python plotting library built on top of matplotlib. It provides convenient syntax and shortcuts for many common types of plots, along with better-looking defaults.

We can also use [seaborn regplot](https://stanford.edu/~mwaskom/software/seaborn/tutorial/regression.html#functions-to-draw-linear-regression-models) for the scatterplot above. This provides automatic linear regression fits (useful for data exploration later on). Here's one example below.

```python
sns.regplot(y="PRICE", x="RM", data=bos, fit_reg = True)
```

### Histograms
***

```python
plt.hist(np.log(bos.CRIM))
plt.title("CRIM")
plt.xlabel("Crime rate per capita")
plt.ylabel("Frequencey")
plt.show()
```

Part 2 Checkup Exercise Set II

Exercise: In the above histogram, we took the logarithm of the crime rate per capita. Repeat this histogram without taking the log. What was the purpose of taking the log? What do we gain by making this transformation? What do you now notice about this variable that is not obvious without making the transformation?

Exercise: Plot histograms for *RM* and *PTRATIO*, along with the two variables you picked in the previous section.

The shape of this histogram without the logarithm is asymmetric: the distribution doesn’t show any kind of symmetry. The original plot below clearly shows a right-skewed distribution with a long tail in the positive direction. Using a log transformation is a common technique in linear regression to make the shape of right-skewed distributions look more normal. In linear and logistic regression, it's essential that the relationship between input and output variables is approximately linear and that the input variables approximate a normal distribution. The log transformation is a great tool in the CRIM case to help make this input variable meet these requirements.

```python
# karl
plt.hist(bos.CRIM)
plt.title("CRIM")
plt.xlabel("Crime rate per capita")
plt.ylabel("Frequencey")
plt.show()
```

## Part 3: Linear Regression with Boston Housing Data Example
***

Here,

$Y$ = boston housing prices (called "target" data in python, and referred to as the dependent variable or response variable)

and

$X$ = all the other features (or independent variables, predictors or explanatory variables)

which we will use to fit a linear regression model and predict Boston housing prices. We will use the least-squares method to estimate the coefficients.

We'll use two ways of fitting a linear regression. We recommend the first but the second is also powerful in its features.

### Fitting Linear Regression using `statsmodels`
***
[Statsmodels](http://statsmodels.sourceforge.net/) is a great Python library for a lot of basic and inferential statistics. It also provides basic regression functions using an R-like syntax, so it's commonly used by statisticians. While we don't cover statsmodels officially in the Data Science Intensive workshop, it's a good library to have in your toolbox. Here's a quick example of what you could do with it. The version of least-squares we will use in statsmodels is called *ordinary least-squares (OLS)*. There are many other versions of least-squares such as [partial least squares (PLS)](https://en.wikipedia.org/wiki/Partial_least_squares_regression) and [weighted least squares (WLS)](https://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares).

```python
# Import regression modules
import statsmodels.api as sm
from statsmodels.formula.api import ols
```

    C:\Users\brcod\Anaconda3\lib\site-packages\statsmodels\compat\pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.
      from pandas.core import datetools

```python
# statsmodels works nicely with pandas dataframes
# The thing inside the "quotes" is called a formula, a bit on that below
m = ols('PRICE ~ RM',bos).fit()
print(m.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.484
    Model:                            OLS   Adj. R-squared:                  0.483
    Method:                 Least Squares   F-statistic:                     471.8
    Date:                Sat, 12 May 2018   Prob (F-statistic):           2.49e-74
    Time:                        17:50:16   Log-Likelihood:                -1673.1
    No. Observations:                 506   AIC:                             3350.
    Df Residuals:                     504   BIC:                             3359.
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept    -34.6706      2.650    -13.084      0.000     -39.877     -29.465
    RM             9.1021      0.419     21.722      0.000       8.279       9.925
    ==============================================================================
    Omnibus:                      102.585   Durbin-Watson:                   0.684
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              612.449
    Skew:                           0.726   Prob(JB):                    1.02e-133
    Kurtosis:                       8.190   Cond. No.                         58.4
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

#### Interpreting coefficients

There is a ton of information in this output. But we'll concentrate on the coefficient table (middle table). We can interpret the `RM` coefficient (9.1021) by first noticing that the p-value (under `P>|t|`) is so small, basically zero. This means that the number of rooms, `RM`, is a statisticall significant predictor of `PRICE`. The regression coefficient for `RM` of 9.1021 means that *on average, each additional room is associated with an increase of $\$9,100$ in house price net of the other variables*. The confidence interval gives us a range of plausible values for this average change, about ($\$8,279, \$9,925$), definitely not chump change.

In general, the $\hat{\beta_i}, i > 0$ can be interpreted as the following: "A one unit increase in $x_i$ is associated with, on average, a $\hat{\beta_i}$ increase/decrease in $y$ net of all other variables."

On the other hand, the interpretation for the intercept, $\hat{\beta}_0$ is the average of $y$ given that all of the independent variables $x_i$ are 0.

####  `statsmodels` formulas
***
This formula notation will seem familiar to `R` users, but will take some getting used to for people coming from other languages or are new to statistics.

The formula gives instruction for a general structure for a regression call. For `statsmodels` (`ols` or `logit`) calls you need to have a Pandas dataframe with column names that you will add to your formula. In the below example you need a pandas data frame that includes the columns named (`Outcome`, `X1`,`X2`, ...), but you don't need to build a new dataframe for every regression. Use the same dataframe with all these things in it. The structure is very simple:

`Outcome ~ X1`

But of course we want to to be able to handle more complex models, for example multiple regression is doone like this:

`Outcome ~ X1 + X2 + X3`

In general, a formula for an OLS multiple linear regression is

`Y ~ X1 + X2 + ... + Xp`

This is the very basic structure but it should be enough to get you through the homework. Things can get much more complex. You can force statsmodels to treat variables as categorical with the `C()` function, call numpy functions to transform data such as `np.log` for extremely-skewed data, or fit a model without an intercept by including `- 1` in the formula. For a quick run-down of further uses see the `statsmodels` [help page](http://statsmodels.sourceforge.net/devel/example_formulas.html).

Let's see how our model actually fit our data. We can see below that there is a ceiling effect, we should probably look into that. Also, for large values of $Y$ we get underpredictions, most predictions are below the 45-degree gridlines.

Part 3 Checkup Exercise Set I

Exercise: Create a scatterplot between the predicted prices, available in `m.fittedvalues` (where `m` is the fitted model) and the original prices. How does the plot look? Do you notice anything interesting or weird in the plot? Comment on what you see.

The plot below shows that there is a linear relationship between predicted and original housing prices. This is to be expected since the best scenario for our model is for it to make a price prediction that has the exact same value as the original housing price: if the original price is 20 we should wish for our model to also predict 20 and so on. We can also see a large cluster of predictions centered around 18-24. This could mean that the model performs best predicting house prices within this interval or that most prices in the original dataset lie within this interval. More investigation is required before a conclusion should be made.

```python
# karl
plt.figure(figsize=(7.5,5))

plt.scatter(m.fittedvalues, bos.PRICE, s=15, alpha=0.3)
plt.xlabel("Predicted Price")
plt.ylabel("Original Price")
plt.title("Relationship between Predicted and Original Housing Price")
plt.show()
```

### Fitting Linear Regression using `sklearn`

```python
from sklearn.linear_model import LinearRegression
X = bos.drop('PRICE', axis = 1)

# This creates a LinearRegression object
lm = LinearRegression()
lm
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)

#### What can you do with a LinearRegression object?
***
Check out the scikit-learn [docs here](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html). We have listed the main functions here. Most machine learning models in scikit-learn follow this same API of fitting a model with `fit`, making predictions with `predict` and the appropriate scoring function `score` for each model.

Main functions | Description
--- | ---
`lm.fit()` | Fit a linear model
`lm.predit()` | Predict Y using the linear model with estimated coefficients
`lm.score()` | Returns the coefficient of determination (R^2). *A measure of how well observed outcomes are replicated by the model, as the proportion of total variation of outcomes explained by the model*

#### What output can you get?

```python
# karl
# Look inside lm object
print(lm.get_params())
```

    {'copy_X': True, 'fit_intercept': True, 'n_jobs': 1, 'normalize': False}

Output | Description
--- | ---
`lm.coef_` | Estimated coefficients
`lm.intercept_` | Estimated intercept

### Fit a linear model
***

The `lm.fit()` function estimates the coefficients the linear regression using least squares.

```python
# Use all 13 predictors to fit linear regression model
lm.fit(X, bos.PRICE)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)

Part 3 Checkup Exercise Set II

Exercise: How would you change the model to not fit an intercept term? Would you recommend not having an intercept? Why or why not? For more information on why to include or exclude an intercept, look [here](https://online.stat.psu.edu/~ajw13/stat501/SpecialTopics/Reg_thru_origin.pdf).

Exercise: One of the assumptions of the linear model is that the residuals must be i.i.d. (independently and identically distributed). To satisfy this, is it enough that the residuals are normally distributed? Explain your answer.

Exercise: True or false. To use linear regression, $Y$ must be normally distributed. Explain your answer.

> **Exercise**: How would you change the model to not fit an intercept term? Would you recommend not having an intercept? Why or why not?

To make a model not fit an intercept term you should drop the y-intercept term ($\beta_0$) in the equation below:
$$ Y = \beta_0 + \beta_1 X + \epsilon$$
where: $\beta_0$ is the intercept, $\beta_1$ is the slope and $\epsilon$ is the residual.

Dropping the intercept and making the regression line pass through the origin (RTO), can be seen as a special case of
OLS since the absence of the intercept term is actually a simplification:

$$ Y = \beta_1 X + \epsilon$$

The decision to do such a thing must be done with caution. There are special circunstances in which this would be appropriate, here are some examples:

- When transformations of the OLS model are needed to correct violations of the Gauss–Markov assumptions;
- When there are strong a priori reasons for believing that Y = 0 when x = 0;

However, when you are not sure if it's appropriate to including an intercept, there are several diagnostic devices that can provide guidance. One of them is the OLS regression followed by a test of the Null Hypothesis `Η0 : β0 = 0` using the **Student’s T statistic** to determine if the intercept is significant. Another way is to run the regression *with* and *without* an intercept, compare the standard errors to see if OLS or RTO provides a better fit.

> **Exercise**: One of the assumptions of the linear model is that the residuals must be i.i.d. (independently and identically distributed). To satisfy this, is it enough that the residuals are normally distributed? Explain your answer.

**Yes**! If the residuals are normally distributed that means that the Y values that generated them are also normally distributed, and this is a very important fact. On the other hand, residuals don't have to be independent. None of the four principal assumptions that justify the use of linear regression models mention this need.

> **Exercise**: True or false. To use linear regression, Y must be normally distributed. Explain your answer.

**False**! This is a common misconception that results from a simplified explanation or misinterpretion of the model equation:
$$Y = \beta_0 + \beta_1X + \epsilon$ where $\epsilon \sim \text{N}(0, \sigma^2_\epsilon)$$

Which can be rewritten as:
$$Y|X \sim \text{N}(\beta_0 + \beta_1X, \sigma^2_\epsilon)$$

Which means, *"Y, conditional on X, is normally distributed with a mean of the predicted values and some variance"*. Since this condition is more difficult to understand and explain, the belief that *"Y must be normally distributed"* has spreaded.

### Estimated intercept and coefficients

Let's look at the estimated coefficients from the linear model using `1m.intercept_` and `lm.coef_`.

After we have fit our linear regression model using the least squares method, we want to see what are the estimates of our coefficients $\beta_0$, $\beta_1$, ..., $\beta_{13}$:

$$ \hat{\beta}_0, \hat{\beta}_1, \ldots, \hat{\beta}_{13} $$

```python
print('Estimated intercept coefficient: {}'.format(lm.intercept_))
```

    Estimated intercept coefficient: 36.491103280364015

```python
print('Number of coefficients: {}'.format(len(lm.coef_)))
```

    Number of coefficients: 13

```python
# The coefficients
pd.DataFrame({'features': X.columns, 'estimatedCoefficients': lm.coef_})[['features', 'estimatedCoefficients']]
```

      features
      estimatedCoefficients

      0
      CRIM
      -0.107171

      1
      ZN
      0.046395

      2
      INDUS
      0.020860

      3
      CHAS
      2.688561

      4
      NOX
      -17.795759

      5
      RM
      3.804752

      6
      AGE
      0.000751

      7
      DIS
      -1.475759

      8
      RAD
      0.305655

      9
      TAX
      -0.012329

      10
      PTRATIO
      -0.953464

      11
      B
      0.009393

      12
      LSTAT
      -0.525467

### Predict Prices

We can calculate the predicted prices ($\hat{Y}_i$) using `lm.predict`.

$$ \hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_1 + \ldots \hat{\beta}_{13} X_{13} $$

```python
# first five predicted prices
lm.predict(X)[0:5]
```

    array([ 30.00821269,  25.0298606 ,  30.5702317 ,  28.60814055,  27.94288232])

Part 3 Checkup Exercise Set III

Exercise: Histogram: Plot a histogram of all the predicted prices. Write a story about what you see. Describe the shape, center and spread of the distribution. Are there any outliers? What might be the reason for them? Should we do anything special with them?

Exercise: Scatterplot: Let's plot the true prices compared to the predicted prices to see they disagree (we did this with `statsmodels` before).

Exercise: We have looked at fitting a linear model in both `statsmodels` and `scikit-learn`. What are the advantages and disadvantages of each based on your exploration? Based on the information provided by both packages, what advantage does `statsmodels` provide?

> **Exercise**: Histogram: Plot a histogram of all the predicted prices. Write a story about what you see. Describe the shape, center and spread of the distribution. Are there any outliers? What might be the reason for them? Should we do anything special with them?

```python
# karl: ecdf() returns a one-dimensional array of measurements
def ecdf(data):
    n = len(data)                # number of data points: n
    x = np.sort(data)            # x-data for the ECDF: x
    y = np.arange(1, n+1/n) / n  # y-data for the ECDF: y
    return x, y
```

```python
y_pred    = lm.predict(X)
pred_mean = np.mean(y_pred)
pred_median  = np.median(y_pred)
pred_std  = np.std(y_pred)

print('>>> shape          =', y_pred.shape)
print('>>> mean           =', pred_mean)
print('>>> median         =', pred_median)
print('>>> std dev        =', pred_std)
print('>>> 95% percentile =', np.percentile(y_pred, [2.5, 97.5]))

### Simulate a larger dataset with the same mean and std deviation to plot ECDF and later PDF
samples = np.random.normal(pred_mean, pred_std, size=5000)
x, y    = ecdf(y_pred)
x_theor, y_theor = ecdf(samples)

plt.figure(figsize=(16,10))

### Plot ECDF
plt.subplot(2, 2, 1)
plt.plot(x_theor, y_theor)                                   # predicted prices
plt.plot(x, y, marker='.', linestyle='none')  # original data
plt.axvline(np.mean(bos.PRICE), color='r', linestyle='--', linewidth=1, label='mean')
plt.xlabel('Predicted Prices')
plt.ylabel('CDF')
plt.legend(loc='upper left')

### Plot PDF
plt.subplot(2, 2, 2)
plt.hist(y_pred, bins=10, normed=True, histtype='step', linewidth=1)
plt.axvline(np.mean(y_pred), color='r', linestyle='--', linewidth=1, label='mean')
plt.xlabel('Predicted Prices')
plt.ylabel('PDF')
plt.legend(loc='upper left')

plt.subplot(2, 2, 3)
plt.boxplot(y_pred, 0, 'rs', 0)
plt.xlabel('Predicted Prices')

plt.tight_layout()
plt.show()
```

```python
print('>>> Prediction < 0   =', y_pred[y_pred <= 0])
print('>>> Prediction located at index =', np.nonzero(y_pred <= 0))
print('>>> X row @ index 414:')
display(X.iloc[[414]])
```

    >>> Prediction < 0   = [-4.28300934]
    >>> Prediction located at index = (array([414], dtype=int64),)
    >>> X row @ index 414:

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      414
      45.7461
      0.0
      18.1
      0.0
      0.693
      4.519
      100.0
      1.6582
      24.0
      666.0
      20.2
      88.27
      36.98

The histogram of the predicted prices resembles a normal distribution and the ECDF also suggests that the data is normally distributed. The mean (22.53) and median (22.12) values are close enough to support this idea.

The boxplot reveal a few outliers on the extremities but only one of them is really concerning. Its negative value of **-4.28** (located at index **414**) implicates that the house is not only free, but the seller is also giving (4.28 * 1000) USD away for anyone who would like to keep the house. Of course, this value doesn't make any sense. The `X` row that is responsible for this prediction is displayed above. We can see that LSTAT has a value of **36.98**. This value is so high that in fact, that the instructions below shows that there's only one other observation (index 374) in the entire dataset that has a LSTAT higher than this:

```python
print('>>> Indexes and values for all rows where LSTAT > 35:')
print(X.LSTAT[X.LSTAT > 35])

# print the model coefficient for LSTAT:
pd.DataFrame({'features': X.columns, 'estimatedCoefficients': lm.coef_})[['features', 'estimatedCoefficients']].iloc[[12]]
```

    >>> Indexes and values for all rows where LSTAT > 35:
    374    37.97
    414    36.98
    Name: LSTAT, dtype: float64

      features
      estimatedCoefficients

      12
      LSTAT
      -0.525467

This high LSTAT value plays an important part during the computation of the prediction when it is multiplied by **-0.525467**, the model's LSTAT coefficient, resulting in a big negative term of **-19.4317** in the equation. I speculate if the LSTAT value could be having a big influence on the prediction calculation. However, its dangerous from a scientific and moral standpoint to simply discard this row because of it's negative impact on the prediction. Anyway, an alternative solution is to normalize/scale the data to see if it improves the prediction score or even use a better method for computing linear regression.

> **Exercise**: Scatterplot: Let's plot the true prices compared to the predicted prices to see they disagree (we did this with statsmodels before).

```python
# karl
plt.figure(figsize=(8,5))

plt.scatter(y_pred, bos.PRICE, s=15, alpha=0.3)
plt.xlabel("Predicted Price")
plt.ylabel("Original Price")
plt.title("Predicted Price vs Original Housing Price")
plt.show()
```

> **Exercise**: We have looked at fitting a linear model in both `statsmodels` and `scikit-learn`. What are the advantages and disadvantages of each based on your exploration? Based on the information provided by both packages, what advantage does `statsmodels` provide?

`statsmodels` offers `summary()`, among other sophisticated tools, which provides a nice table with several interesting pieces of information like r-squared, f-statistic, coefficients used by the model, p-value, standard errors, 95% confidence intervals and much more. `statsmodels` seem to offer a great variety of tools to deal with linear regression models, mixed linear models, regression with discrete dependent variables, among others.

`scikit-learn` on the other hand, offers an easy way to use data structured in DataFrames for linear regression, logistic regression, clustering and other machine learning techniques. Plus, it offers tools that assist on hyperparameter tuning, helping you quickly find the best values.

### Evaluating the Model: Sum-of-Squares

The partitioning of the sum-of-squares shows the variance in the predictions explained by the model and the variance that is attributed to error.

$$TSS = ESS + RSS$$

#### Residual Sum-of-Squares (aka $RSS$)

The residual sum-of-squares is one of the basic ways of quantifying how much error exists in the fitted model. We will revisit this in a bit.

$$ RSS = \sum_{i=1}^N r_i^2 = \sum_{i=1}^N \left(y_i - \left(\beta_0 + \beta_1 x_i\right)\right)^2 $$

```python
print(np.sum((bos.PRICE - lm.predict(X)) ** 2))
```

    11080.276284149872

#### Explained Sum-of-Squares (aka $ESS$)

The explained sum-of-squares measures the variance explained by the regression model.

$$ESS = \sum_{i=1}^N \left( \hat{y}_i - \bar{y} \right)^2 = \sum_{i=1}^N \left( \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) - \bar{y} \right)^2$$

```python
print(np.sum(lm.predict(X) - np.mean(bos.PRICE)) ** 2)
```

    5.80189651524e-23

### Evaluating the Model: The Coefficient of Determination ($R^2$)

The coefficient of determination, $R^2$, tells us the percentage of the variance in the response variable $Y$ that can be explained by the linear regression model.

$$ R^2 = \frac{ESS}{TSS} $$

The $R^2$ value is one of the most common metrics that people use in describing the quality of a model, but it is important to note that *$R^2$ increases artificially as a side-effect of increasing the number of independent variables.* While $R^2$ is reported in almost all statistical packages, another metric called the *adjusted $R^2$* is also provided as it takes into account the number of variables in the model, and can sometimes even be used for non-linear regression models!

$$R_{adj}^2 = 1 - \left( 1 - R^2 \right) \frac{N - 1}{N - K - 1} = R^2 - \left( 1 - R^2 \right) \frac{K}{N - K - 1} = 1 - \frac{\frac{RSS}{DF_R}}{\frac{TSS}{DF_T}}$$

where $N$ is the number of observations, $K$ is the number of variables, $DF_R = N - K - 1$ is the degrees of freedom associated with the residual error and $DF_T = N - 1$ is the degrees of the freedom of the total error.

### Evaluating the Model: Mean Squared Error and the $F$-Statistic
***
The mean squared errors are just the *averages* of the sum-of-squares errors over their respective degrees of freedom.

$$MSE = \frac{ESS}{K}$$
$$MSR = \frac{RSS}{N-K-1}$$

**Remember: ** Notation may vary across resources particularly the use of *R* and *E* in *RSS/ESS* and *MSR/MSE*. In some resources, E = explained and R = residual. In other resources, E = error and R = regression (explained). **This is a very important distinction that requires looking at the formula to determine which naming scheme is being used.**

Given the MSR and MSE, we can now determine whether or not the entire model we just fit is even statistically significant. We use an $F$-test for this. The null hypothesis is that all of the $\beta$ coefficients are zero, that is, none of them have any effect on $Y$. The alternative is that *at least one* $\beta$ coefficient is nonzero, but it doesn't tell us which one in a multiple regression:

$$H_0: \beta_i = 0, \mbox{for all $i$} \\
H_A: \beta_i > 0, \mbox{for some $i$}$$

$$F = \frac{MSR}{MSE} = \left( \frac{R^2}{1 - R^2} \right) \left( \frac{N - K - 1}{K} \right)$$

Once we compute the $F$-statistic, we can use the $F$-distribution with $N-K$ and $K-1$ degrees of degrees of freedom to get a p-value.

**Warning!** The $F$-statistic mentioned in this section is NOT the same as the F1-measure or F1-value discused in Unit 7.

Part 3 Checkup Exercise Set IV

Let's look at the relationship between `PTRATIO` and housing price.

Exercise: Make a scatterplot of `PTRATIO` and housing price. Tell a story about the relationship between the variables.

Exercise: Try fitting a linear regression model using only the 'PTRATIO' (pupil-teacher ratio by town) and interpret the intercept and the coefficients.

Exercise: Calculate (or extract) the $R^2$ value. What does it tell you?

Exercise: Compute the $F$-statistic. What does it tell you?

Exercise: Take a close look at the $F$-statistic and the $t$-statistic for the regression coefficient. What relationship do you notice? Note that this relationship only applies in *simple* linear regression models.

> **Exercise**: Make a scatterplot of PTRATIO and housing price. Tell a story about the relationship between the variables.

`PTRATIO` and `PRICE` seem to follow a negative linear relationship where cheaper houses have a higher pupil-teacher ratio. More expensive houses have a lower pupil-teacher ratio, meaning a teacher has less students to give attention to, a fact that could lead to a better education.

```python
# karl: scatterplot of PTRATIO and housing price.
plt.figure(figsize=(8,5))
plt.subplot(1, 1, 1)

plt.scatter(bos.PTRATIO, bos.PRICE, s=15, alpha=0.3, color='blue')
plt.xlabel("pupil-teacher ratio by town (PTRATIO)")
plt.ylabel("Housing Price")
plt.title("Relationship between PTRATIO and Price")
plt.show()
```

> **Exercise**: Try fitting a linear regression model using only the 'PTRATIO' (pupil-teacher ratio by town) and interpret the intercept and the coefficients.

The `intercept` (or `y-intercept`) of **62.34** defines the starting point of the linear regression line in the vertical axis. For educational purposes, we created an array of X values that goes from 0 to 29 just to be able to draw a (red) line that starts at the origin of the X-axis and allow us to have a better visualization of the implications of the `intercept` and the `coefficient` of the model. Thus, the regression line starts at **(0, 62.34)** and every increment in the X-axis causes a change of **-2.15** in the Y-axis. This particular combination of values mean that the height of the line will decrease while the length of the line increases. This combination of values characterizes a negative relationship between PTRATIO and PRICE. It's important to note that the model coefficient(s) is what causes the **slope** of the line. The line plotted in *green* is the original linear regression line: it passes through all the X values and overlaps momentaneously with the (artificial) *red* line.

```python
# karl: linear regression model using only the 'PTRATIO'
X = bos.PTRATIO.values.reshape(-1, 1)
y = bos.PRICE.values.reshape(-1, 1)

# Fit the model to the data
reg = LinearRegression()
reg.fit(X, y)
y_pred = reg.predict(X)
print('>>> Intercept    =', reg.intercept_, '\n>>> Coefficients =', reg.coef_)

# Setup plot size and plot data points
plt.figure(figsize=(8,5))
plt.subplot(1, 1, 1)
plt.scatter(X, y, s=15, alpha=0.3, color='blue')

# plot and calc by hand the regression line, starting from x = 0 to x = 29
x_values = np.arange(0, 30)
y_values = (reg.coef_ * x_values + reg.intercept_).reshape(-1) # Calculate Y = m*X + b
plt.plot(x_values, y_values, color='red', linewidth=1)

# plot original regression line as green
plt.plot(X, y_pred, color='green', linewidth=3)
plt.xlim((0, 30))
plt.ylim((0, 65))
plt.show()
```

> **Exercise**: Calculate (or extract) the $R^2$ value. What does it tell you?

R-squared is a measure of how close the data are to the fitted regression line. It's value always stays between 0.0 and 1.0, and the higher the R-squared, the better the model fits the data. In this case, the value shows that the model explains only **25.8%** of the variability of the response data around its mean. We can clearly see that most data points are far from the regression line which makes me think that this is likely not a good model.

```python
# karl
r_sq = reg.score(X, y)
print(">>> R^2 = {}".format(r_sq))
```

    >>> R^2 = 0.257847318009223

> **Exercise:** Compute the  $F$-statistic. What does it tell you?

The Hypothesis for F-Test for significance can be constructed as:

- **H0**: the fit of IOM and the current model is same. That is, the data can be modeled accurately by setting regression coefficients to zero. (H0 = 0)
- **HA**: the fit of IOM is significantly less compared to our current model.That is, at least one of the regression coefficients should be non-zero to make the model significantly better. (HA > 0)

Observation: a model with zero predictor variables (coefficients) is also called *Intercept Only Model (IOM)*. In this model, only the *intercept* has a non-null value.

The **p-value** of the F-statistic is computed below and its much smaller than the significance value (0.05): **0.000000000000000111022302463**. Thus, we must reject the Null Hypothesis in favor of the Alternative Hypothesis: the coefficients of the model do improve the fit of the model.

```python
# karl: compute F-test by hand
n = len(X)              # No. Observations
k = 1 + 1               # (no of coefficient variables + 1 for intercept)
dfn = k - 1             # Degrees of Freedom Numerator   = (k-1)
dfd = n - k             # Degrees of Freedom Denominator = (n-k)
mse_model = ( r_sq / dfn )
mse_resid = ( (1-r_sq) / dfd )
F = mse_model / mse_resid
print('>>> F-test  = ', F)

# Subtracting scipy.stats.f.cdf() by one we get the probability in the tail, which represents
# the probability of seeing an F-statistics more extreme than the one observed.
import scipy
print('>>> P-value = ', 1.0 - scipy.stats.f.cdf(F, dfn, dfd))

# Compare my calculation with the output of statsmodel summary F-statistic:
X = bos.PTRATIO
y = bos.PRICE
data = pd.DataFrame({'x': X, 'y': y})
model = ols("y ~ x", data).fit()
#print(model.summary())
```

    >>> F-test  =  175.105542876
    >>> P-value =  1.11022302463e-16

> **Exercise**: Take a close look at the  $F$-statistic and the  $t$-statistic for the regression coefficient. What relationship do you notice? Note that this relationship only applies in simple linear regression models.

In the coefficients table below, `t` is the value of the *t-statistic* for testing whether the corresponding regression coefficient is different from 0. `P>|t|` represents the probability of a test statistic at least as unusual as the one  obtained if *H0* were true. In this case, the regressor of `x` is significant at `α = 0.05` because it's corresponding *p-value* of **0.000** is smaller than 0.05, thus, rejecting the Null Hypothesis: when `x` is larger than 0 it's value is relevant for the model.

In other words, the predictors don't fail individual t-tests. This means that their values are important! Also, the overall regression given by the F-test makes it clear that the predictors are jointly informative.

```python
# karl
print(model.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                      y   R-squared:                       0.258
    Model:                            OLS   Adj. R-squared:                  0.256
    Method:                 Least Squares   F-statistic:                     175.1
    Date:                Sat, 12 May 2018   Prob (F-statistic):           1.61e-34
    Time:                        17:50:20   Log-Likelihood:                -1764.8
    No. Observations:                 506   AIC:                             3534.
    Df Residuals:                     504   BIC:                             3542.
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept     62.3446      3.029     20.581      0.000      56.393      68.296
    x             -2.1572      0.163    -13.233      0.000      -2.477      -1.837
    ==============================================================================
    Omnibus:                       92.924   Durbin-Watson:                   0.725
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              191.444
    Skew:                           1.001   Prob(JB):                     2.68e-42
    Kurtosis:                       5.252   Cond. No.                         160.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Part 3 Checkup Exercise Set V

Fit a linear regression model using three independent variables

 'CRIM' (per capita crime rate by town)

 'RM' (average number of rooms per dwelling)

 'PTRATIO' (pupil-teacher ratio by town)

Exercise: Compute or extract the $F$-statistic. What does it tell you about the model?

Exercise: Compute or extract the $R^2$ statistic. What does it tell you about the model?

Exercise: Which variables in the model are significant in predicting house price? Write a story that interprets the coefficients.

```python
# karl: multivariate linear regression with CRIM, RM and PTRATIO
X = bos[ ['CRIM', 'RM', 'PTRATIO'] ]
y = bos.PRICE.values.reshape(-1, 1)

# Fit the model to the data
reg = LinearRegression()
reg.fit(X, y)
y_pred = reg.predict(X)
print('>>> Intercept    =', reg.intercept_, '\n>>> Coefficients =', reg.coef_)

# Setup plot size and plot data points
plt.figure(figsize=(8,5))
plt.subplot(1, 1, 1)
plt.scatter(y, y_pred, s=15, alpha=0.3, color='blue')
plt.plot([y.min(), y.max()], [y.min(), y.max()], 'r-', lw=1)
plt.title('Actual Price vs Predicted Price')
plt.xlabel('Actual Price')
plt.ylabel('Predicted Price')
plt.show()
```

> **Exercise**: Compute or extract the $F$-statistic. What does it tell you about the model?

The $F$-statistic of 244.22 has a *p-value* of **0.000000000000000111022302463** and therefore much smaller then the significance level `α = 0.05`. This value sucessfully rejects the Null Hypothesis (*H0*) in favor of the alternative: `x` contains at least one variable that is relevant to the model.

```python
# karl: compute F-test by hand
X = bos[ ['CRIM', 'RM', 'PTRATIO'] ]
y = bos.PRICE.values.reshape(-1, 1)
r_sq = reg.score(X, y)

n = len(X)              # No. Observations
k = 3 + 1               # (no of coefficient variables + 1 for intercept)
dfn = k - 1             # Df Model or Degrees of Freedom Numerator   = (k-1)
dfd = n - k             # Df Residuals or Degrees of Freedom Denominator = (n-k)
mse_model = ( r_sq / dfn )
mse_resid = ( (1-r_sq) / dfd )
F = mse_model / mse_resid
print('>>> F-test  = ', F)
print('>>> P-value = ', 1.0 - scipy.stats.f.cdf(F, dfn, dfd))

# Compare my calculation with the output of statsmodel summary F-statistic:
X = bos[ ['CRIM', 'RM', 'PTRATIO'] ]
X = sm.add_constant(X)
y = bos['PRICE']
model = sm.OLS(endog=y, exog=X).fit()
#print(model.summary())
```

    >>> F-test  =  244.223870239
    >>> P-value =  1.11022302463e-16

> Exercise: Compute or extract the  $R^2$ statistic. What does it tell you about the model?

The $R^2$ statistic shows that the model explains 59.3% of the variability of the response data around its mean. This value is encouraging since its a measure of how well the model fits the data.

```python
# karl
print('>>> R^2', r_sq)
#print(model.summary())
```

    >>> R^2 0.593414155114

> Exercise: Which variables in the model are significant in predicting house price? Write a story that interprets the coefficients.

- `const`: is the intercept (the model's response variable). A p-value of `0.413` falls below the significant threshold and indicates that this coefficient is not significant to the model.
- `CRIM`: a p-value of `0.000` indicates that this coefficient is significant to the model.
- `RM`:  a p-value of `0.000` indicates that this coefficient is significant to the model.
- `PTRATIO`: a p-value of `0.000` indicates that this coefficient is significant to the model.

```python
# karl
print(model.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.593
    Model:                            OLS   Adj. R-squared:                  0.591
    Method:                 Least Squares   F-statistic:                     244.2
    Date:                Sat, 12 May 2018   Prob (F-statistic):           1.09e-97
    Time:                        17:50:21   Log-Likelihood:                -1612.6
    No. Observations:                 506   AIC:                             3233.
    Df Residuals:                     502   BIC:                             3250.
    Df Model:                           3
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const         -3.3066      4.038     -0.819      0.413     -11.240       4.627
    CRIM          -0.2021      0.032     -6.301      0.000      -0.265      -0.139
    RM             7.3816      0.402     18.360      0.000       6.592       8.171
    PTRATIO       -1.0742      0.133     -8.081      0.000      -1.335      -0.813
    ==============================================================================
    Omnibus:                      233.249   Durbin-Watson:                   0.828
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):             2002.374
    Skew:                           1.802   Prob(JB):                         0.00
    Kurtosis:                      12.054   Cond. No.                         311.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

## Part 4: Comparing Models

During modeling, there will be times when we want to compare models to see which one is more predictive or fits the data better. There are many ways to compare models, but we will focus on two.

### The $F$-Statistic Revisited

The $F$-statistic can also be used to compare two *nested* models, that is, two models trained on the same dataset where one of the models contains a *subset* of the variables of the other model. The *full* model contains $K$ variables and the *reduced* model contains a subset of these $K$ variables. This allows us to add additional variables to a base model and then test if adding the variables helped the model fit.

$$F = \frac{\left( \frac{RSS_{full} - RSS_{reduced}}{K_{full} - K_{reduced}} \right)}{\left( \frac{RSS_{reduced}}{N - K_{reduced}} \right)}$$

### Akaike Information Criterion (AIC)

Another statistic for comparing two models is AIC, which is based on the likelihood function and takes into account the number of variables in the model.

$$AIC = 2 K - 2 \log_e{L}$$

where $L$ is the likelihood of the model. AIC is meaningless in the absolute sense, and is only meaningful when compared to AIC values from other models. Lower values of AIC indicate better fitting models.

`statsmodels` provides the AIC in its output.

Part 4 Checkup Exercises

Exercise: Find another variable (or two) to add to the model we built in Part 3. Compute the $F$-test comparing the two models as well as the AIC. Which model is better?

Below there's a heatmap showing the correlation between the different features. It's possible to see that `ZN` and `B` have a good correlation with `PRICE`. On the new test below, these 2 variables were added to the model.

The **$F$-statistic** of **163** has a *p-value* smaller than the significance level `α = 0.05`. This tell us that this group of variables are jointly significant! The new test also has a slightly higher **$R^2$ statistic** of **62%** (the previous was 59.3%). Another confirmation that this new model is better than the previous comes from the **AIC**: the decrease to **3203** (was 3233) also indicates a better model quality.

```python
# karl
plt.figure(figsize=(6, 6))
sns.heatmap(bos.corr(), square=True, cmap='RdYlGn')
plt.plot()
plt.show()

X = bos[ ['CRIM', 'RM', 'PTRATIO', 'ZN', 'B'] ]
X = sm.add_constant(X)
y = bos['PRICE']
model = sm.OLS(endog=y, exog=X).fit()
#print(model.summary())

# print just the relevant parts, not all the summary
A = np.identity(len(model.params))
A = A[1:,:]
print('>>> F-test =', round(model.f_test(A).fvalue.flatten()[0], 1), ' p-value =', model.f_test(A).pvalue)
print('>>> R^2    =', model.rsquared)
print('>>> AIC    =', model.aic)
```

## Part 5: Evaluating the Model via Model Assumptions and Other Issues
***
Linear regression makes several assumptions. It is always best to check that these assumptions are valid after fitting a linear regression model.

**Linearity**. The dependent variable $Y$ is a linear combination of the regression coefficients and the independent variables $X$. This can be verified with a scatterplot of each $X$ vs. $Y$ and plotting correlations among $X$. Nonlinearity can sometimes be resolved by [transforming](https://onlinecourses.science.psu.edu/stat501/node/318) one or more independent variables, the dependent variable, or both. In other cases, a [generalized linear model](https://en.wikipedia.org/wiki/Generalized_linear_model) or a [nonlinear model](https://en.wikipedia.org/wiki/Nonlinear_regression) may be warranted.

**Constant standard deviation**. The SD of the dependent variable $Y$ should be constant for different values of X. We can check this by plotting each $X$ against $Y$ and verifying that there is no "funnel" shape showing data points fanning out as $X$ increases or decreases. Some techniques for dealing with non-constant variance include weighted least squares (WLS), [robust standard errors](https://en.wikipedia.org/wiki/Heteroscedasticity-consistent_standard_errors), or variance stabilizing transformations.

 **Normal distribution for errors**.  The $\epsilon$ term we discussed at the beginning are assumed to be normally distributed. This can be verified with a fitted values vs. residuals plot and verifying that there is no pattern, and with a quantile plot.
  $$ \epsilon_i \sim N(0, \sigma^2)$$
Sometimes the distributions of responses $Y$ may not be normally distributed at any given value of $X$.  e.g. skewed positively or negatively.

 **Independent errors**.  The observations are assumed to be obtained independently.

e.g. Observations across time may be correlated

There are some other issues that are important investigate with linear regression models.

**Correlated Predictors:** Care should be taken to make sure that the independent variables in a regression model are not too highly correlated. Correlated predictors typically do not majorly affect prediction, but do inflate standard errors of coefficients making interpretation unreliable. Common solutions are dropping the least important variables involved in the correlations, using regularlization, or, when many predictors are highly correlated, considering a dimension reduction technique such as principal component analysis (PCA).

**Influential Points:** Data points that have undue influence on the regression model. These points can be high leverage points or outliers. Such points are typically removed and the regression model rerun.

Part 5 Checkup Exercises

Take the reduced model from Part 3 to answer the following exercises. Take a look at [this blog post](http://mpastell.com/2013/04/19/python_regression/) for more information on using statsmodels to construct these plots.

Exercise: Construct a fitted values versus residuals plot. What does the plot tell you? Are there any violations of the model assumptions?

Exercise: Construct a quantile plot of the residuals. What does the plot tell you?

Exercise: What are some advantages and disadvantages of the fitted vs. residual and quantile plot compared to each other?

Exercise: Identify any outliers (if any) in your model and write a story describing what these outliers might represent.

Exercise: Construct a leverage plot and identify high leverage points in the model. Write a story explaining possible reasons for the high leverage points.

Exercise: Remove the outliers and high leverage points from your model and run the regression again. How do the results change?

> Exercise: Construct a fitted values versus residuals plot. What does the plot tell you? Are there any violations of the model assumptions?

A residual is the difference between the actual value `y` and the predicted value `ŷ`. The diagnostic plot displayed below (on the left), is meant to show if residuals have non-linear patterns. It draws a scatterplot of fitted values against residuals, with a *"locally weighted scatterplot smoothing (lowess)"* regression line (in red) showing any apparent trend. For plotting purposes, fitted values are passed as `x` and the dependent variable as `y`. After that, `sns.residplot()` calculates the residuals as `residuals = observed - predicted` and defines these values for the `Y` axis. From the plot, we can see that this is not a good model because:

- The data shows a curvilinear trend or pattern;
- On the histogram, the distribution of the residuals is not symmetrical;

```python
# karl
X = bos[ ['CRIM', 'RM', 'PTRATIO'] ]
y = bos.PRICE.values.reshape(-1, 1)

# Fit the model to the data
reg = LinearRegression()
reg.fit(X, y)
y_pred = reg.predict(X)

# setup plot size
plt.figure(figsize=(15, 5))

# Plot Fitted vs Residuals
plt.subplot(1, 2, 1)
sns.residplot(y_pred.flatten(), bos.PRICE, lowess=True, scatter_kws={'s':20, 'alpha': 0.3}, color='green', line_kws={'color':'red'})
plt.title('Fitted vs Residuals')
plt.xlabel('Fitted') # Prediction
plt.ylabel('Residuals') # Dependent Variable (PRICE)

### Plot residuals PDF
#residuals = bos.PRICE - y_pred.flatten() # residuals = observed - predicted
residuals = model.resid

plt.subplot(1, 2, 2)
plt.hist(residuals, bins=20, normed=True, histtype='step', linewidth=1, color='green')
plt.axvline(np.mean(residuals), color='r', linestyle='--', linewidth=1, label='mean')
plt.title('Residuals Distribution')
plt.xlabel('Residuals')
plt.ylabel('PDF')
plt.legend(loc='upper left')

plt.tight_layout()
plt.show()
```

> Exercise: Construct a quantile plot of the residuals. What does the plot tell you?

This type of plot shows how well the distribution of residuals fit the normal distribution. The image below show a plot of the standardized (z-score) residuals against the theoretical normal quantiles. Anything quite off the diagonal lines may be a concern for further investigation: do residuals follow a straight line well or do they deviate severely? In this case, most of the residuals follow a straight line but there are a few of them that are distant and ultimatelly, not good for the model.

```python
# karl:
fig = plt.figure(figsize = (8, 5))
fig.clf()
ax = fig.gca()

pp = sm.ProbPlot(residuals, fit=True)
pp.qqplot(line='45', alpha=0.3, color='green', lw=1, ax=ax)
plt.title('QQ plot');
plt.show()
```

> Exercise: What are some advantages and disadvantages of the fitted vs. residual and quantile plot compared to each other?

The **fitted vs. residual plot**:

- The residuals are the part of the dependent variable that the model couldn't explain and are the best estimate of the error term from the regression model;
- It makes it easy to detect if there is a trend or pattern in the residuals, facilitating the identification of non-linear relationships;
- It helps to spot outliers.

The **quantile plot**:

- Similar to histograms, show how residuals are distributed: this particular U-shape indicates the distribution is right-skewed;
- This plot helps to detect shifts in location, shifts in scale, changes in symmetry, and the presence of outliers.

> Exercise: Identify any outliers (if any) in your model and write a story describing what these outliers might represent.

The quantile plot reveals that residuals with a value higher than 20 are most certainly outliers. On the DataFrame displayed below it's possible to see the original CRIM, RM, RATIO data that created for those 7 outliers. It's also possible to see the original PRICE (`y`) for that data and the predicted PRICE (`y_hat`). Analysing the DataFrame makes it clear that  our predictions for these observations were far smaller than they should be, causing the residuals to be high.

```python
#print('>>> List residuals > 20:\n', residuals[residuals > 20])
resid_idx = residuals[residuals > 20].index
diag_df = X.iloc[resid_idx].copy() # copy all the rows from X that generated those residuals

### Build DataFrame to inspect the original data that generated high residual values after prediction
diag_df['y'] = bos.PRICE.iloc[resid_idx].copy()
diag_df['y_hat'] = y_pred[resid_idx]
diag_df['residual'] = residuals[resid_idx]
display(diag_df)
```

      CRIM
      RM
      PTRATIO
      y
      y_hat
      residual

      365
      4.55587
      3.561
      20.2
      27.5
      0.359783
      26.667465

      367
      13.52220
      3.863
      20.2
      23.1
      0.776834
      25.056254

      368
      4.89822
      4.970
      20.2
      50.0
      10.691218
      38.701619

      369
      5.66998
      6.683
      20.2
      50.0
      23.179862
      26.454131

      370
      6.53876
      7.016
      20.2
      50.0
      25.462335
      23.889699

      371
      9.23230
      6.216
      20.2
      50.0
      19.012691
      30.460936

      372
      8.26725
      5.875
      20.2
      50.0
      16.690622
      33.096002

> Exercise: Construct a leverage plot and identify high leverage points in the model. Write a story explaining possible reasons for the high leverage points.

Leverage is a measure of how far an observation on the predictor variable is from the **mean** of that predictor variable. This helps identify observations that have an unusually large influence on the estimation of the predicted value of the response variable (PRICE). The plot also helps us identify outliers: price predictions that have residuals greater than **2** or less than **-2**. These outliers are easily identified on the left plot below, located inside the *red* rectangular areas.

Some of these outliers have very small (nearly zero) leverage values while other like 365, 367 and 364 are quite significant. Although they are several outlying observations, not all of them have a significant influence on the estimation of the regression model. **On the other hand**, the observations within the *green* rectangular area are cases of much higher than average leverage. From all these, observation 380 has the largest leverage but it's not considered an outlier.

```python
# karl: compute leverage values from the statsmodels OLS() we executed earlier
influence = model.get_influence()
leverage = influence.hat_matrix_diag

plt.figure(figsize = (16, 6))

### Draw standard leverage plot
ax = plt.subplot(1, 2, 1)
sns.regplot(leverage, model.resid_pearson, fit_reg=False, scatter_kws={'s':25, 'alpha': 0.7}, ax = ax)
ax.fill_between([0,0.045], 2, 7.5, facecolor='red', alpha = .20, interpolate=True)
ax.fill_between([0,0.045], -2, -3.5, facecolor='red', alpha = .20, interpolate=True)
ax.fill_between([0.055,0.265], 2, -2, facecolor='green', alpha = .20, interpolate=True)
plt.title('Leverage vs Price Residuals')
plt.xlabel('Leverage')
plt.ylabel('Price residuals')

### Draw influence plot (leverage significance)
from statsmodels.graphics.regressionplots import *
ax = plt.subplot(1, 2, 2)
influence_plot(model, ax=ax, size = 6)
plt.show()
```

> Exercise: Remove the outliers and high leverage points from your model and run the regression again. How do the results change?

**Yes!** The new model, without the outliers and without the high leverage points, has an improved $R^2$ score of **0.97** (was 0.62) and an improved AIC of **2767.6** (was 3203.3). From the plot below, we can see that the new model still has outliers however they are less harmful than before.

```python
print('>>> X shape =', X.shape, '(before)')
print('>>> y shape =', y.shape, '(before)')

### Remove high leverage pts (green area)
high_leverage_idx = np.where(leverage > 0.05)
X = X.drop(X.index[high_leverage_idx])   # erase pts from X
y = np.delete(y, high_leverage_idx)      # erase pts from y
print('>> Erased', len(high_leverage_idx[0]), 'observations with high leverage.')

### Remove outliers!
# First, use standardized pearson residuals from statsmodel to be able to locate outliers using +2 and -2 values (Y)
norm_resid = model.resid_pearson
outliers_idx = np.where((norm_resid > 2) | (norm_resid < -2))
X = X.drop(X.index[outliers_idx])       # erase pts from X
y = np.delete(y, outliers_idx)          # erase pts from y
print('>> Erased', len(outliers_idx[0]), 'outlier residuals.')

print('>>> X shape =', X.shape, '(after)')
print('>>> y shape =', y.shape, '(after)')
```

    >>> X shape = (506, 3) (before)
    >>> y shape = (506, 1) (before)
    >> Erased 5 observations with high leverage.
    >> Erased 20 outlier residuals.
    >>> X shape = (481, 3) (after)
    >>> y shape = (481,) (after)

```python
model = sm.OLS(endog=y, exog=X).fit()
influence = model.get_influence()
leverage = influence.hat_matrix_diag
#print(model.summary())

# print just the relevant parts, not all the summary
A = np.identity(len(model.params))
A = A[1:,:]
print('>>> F-test =', round(model.f_test(A).fvalue.flatten()[0], 1), ' p-value =', model.f_test(A).pvalue)
print('>>> R^2    =', model.rsquared)
print('>>> AIC    =', model.aic)

plt.figure(figsize = (16, 6))

### Draw standard leverage plot
ax = plt.subplot(1, 2, 1)
sns.regplot(leverage, model.resid_pearson, fit_reg=False, scatter_kws={'s':25, 'alpha': 0.7}, ax = ax)
ax.fill_between([0,0.045], 2, 7.5, facecolor='red', alpha = .20, interpolate=True)
ax.fill_between([0,0.045], -2, -3.5, facecolor='red', alpha = .20, interpolate=True)
ax.fill_between([0.055,0.265], 2, -2, facecolor='green', alpha = .20, interpolate=True)
plt.title('Leverage vs Price Residuals')
plt.xlabel('Leverage')
plt.ylabel('Price residuals')

plt.show()
```

```python

```


# Statistiek

## Week 1

### Hoorcollege

* Doorlezen van 1.1.1 t/m 1.1.3

#### 1.1.4 Variabelen & 1.1.5 Indeling in schalen

Statistiek $\rightarrow$ data-analyse

Om dit te kunnen doen gaan we _data verzamelen_, dit kan zijn:
* Bestaande gegevens (nadelen: incompleet, teveel informatie, incorrect)
* Interviews (enquete)
* Observeren (metingen)

We kunnen een _begrip_ zoals bijv. opleidingsniveau omzetten naar een _variabele_ (wel of niet meetbaar maken = open rationaliseren)

** Vb: Opleidingsniveau**

Om het opleidingsniveau meetbaar te maken kunnen we antwoord alternatieven bedenken.

1. Basischool
2. VMBO k/b / LBO
3. VMBO t/g / MAVO
4. HAVO / MBO
5. VWO
6. HBO
7. WO

#### Continue variabele

Een continue variabele is op een schaal. Bijvoorbeeld:

* Leeftijd (meetbaar op jaar/dag/uur/minuut/...)
* Temperatuur
* Lengte

#### Discrete variabele

Een discrete variabele zijn alle losse punten. Bijvoorbeeld:

* Kleuren (rood/geel/blauw)
* Opleidingsniveau
* Geslacht (m/v)
* Leeftijdsklassen

#### Klassificeren

Klassificeren houdt in dat we variabelen gaan samenvoegen. Op deze manier kun je van een continue variabel een discrete variabel maken.

#### Meetniveau's

Gegevens kunnen op verschillende niveau's worden gemeten:

1. Nominaal
2. Ordinaal (laag naar hoog)
3. Interval
4. Ratio

$\begin{array}{l l l} \text{Meetniveau} & \text{Kenmerk} & \text{Voorbeeld} \\ \hline (1) \text{ Nominaal} & \text{Onderscheidbaar} & \text{Geslacht, kleur auto} \\ (2) \text{ Ordinaal} & \text{+ Ordening} & \text{Opleidingsniveau, tevredenheid, leeftijdsklasse, rapportcijfer} \\ (3) \text{ Interval} & \text{+ Verschil (heeft betekenis)} & \text{Temp C/F} \\ (4) \text{ Ratio} & \text{+ Verhouding in betekenis, absoluut nulpunt} & \text{Temp K, gewicht, lengte, toetsscore} \end{array}$

#### 1.1.2 Frequentieverdeling

Dit kunnen we doen door te _klassificeren_ wat inhoudt dat we schaalpunten samenvoegen.

Hiermee veranderd het meetniveau van de data:

$\begin{cases}\begin{aligned} \text{Interval/Ratio} &\rightarrow \text{Ordinaal/Nominaal} \\ \text{Ordinaal} &\rightarrow \text{Ordinaal/Nominaal} \\ \text{Nominaal} &\rightarrow \text{Nominaal} \end{aligned}\end{cases}$

#### Relatiefrequentie

Dit kunnen we berekenen met de formule:

$\text{Relatiefrequentie} = \frac{\text{Absolute frequentie}}{\text{Totaal # waarnemingen}}$

Een voorbeeld om te zien welk $\%$ van de populatie/steekproef vrouwen zijn kunnen we de formule als volgt invullen:

$\text{Relatiefrequentie} = \frac{\text{# vrouwen}}{\text{# mensen}}$

#### Populatie

Om de relatiefrequentie van een populatie te berekenen gebruiken we:

$\text{Relatiefrequentie} = \frac{f}{N}$

Om alle klassen bij elkaar op te tellen gebruiken we:

$N = f_1 + f_2, + f_3 + \ldots + f_k$

Waarbij $k$ de klassen zijn.

#### Steekproef

Om de relatiefrequentie van een steekproef te berekenen gebruiken we:

$\text{Relatiefrequentie} = \frac{f}{n}$

Let hierop de kleine $n$.

Om alle klassen bij elkaar op te tellen gebruiken we:

$n = f_1+ f_2 + f_3 + \ldots + f_k$

Waarbij $k$ de klassen zijn.

#### Percentage

Om het percentage te berekenen gebruiken we:

$\text{Percentage} = \frac{f}{n} \times 100\% \qquad (\text{steekproef})$

#### Klassen

De klassen hebben een _bovengrens_, en een _ondergrens_.

#### Klassenmidden

Om het klassenmidden te berekenen gebruiken we:

$\text{Klassenmidden} = \frac{\text{Ondergrens} + \text{Bovengrens}}{2}$

#### Klassenbreedte

Om de klassenbreedte te berekenen gebruiken we:

$\text{Klassenbreedte} = \text{Bovengrens} - \text{Ondergrens}$

#### 1.5.2 Histogram & 1.5.4 Frequentiepolygon

Het verschil tussen een histogram en een staafdiagram is:

* De ruimte tussen de staven (deze is $0$ bij een histogram)
* Histogram gebruikt een interval of ratio.

Let er op dat de staven in een histogram klassengrenzen staat en dat de frequentiepolygon op het klassenmidden komt.

Bij een cumulatieve frequentiepolygon komen de punten wel weer op de klassengrenzen.

Let er goed op dat bij het opstellen van de klassenbreedte alle punten binnen de klassen vallen en dat alle klassenbreedten even groot zijn. Als de frequentieverdeling bijvoorbeeld $10 - < 15$ is dan is de ondergrends $9.5$ en de bovengrens $14.5$. Op deze manier hebben we altijd een klassenbreedte van $5$.

Zodra er _sprongetjes_ in de gegevens zitten dan spreken we van een _discrete_ variabel.

## Week 2

### Hoorcollege

Op het tentamen kan er worden gevraagd naar 1 van de volgende 2 situaties:

**Gelijke klassenbreedte**

y-as: frequentie of frequentiedichtheid per eenheid (bijv. per €100)

**Ongelijk klassenbreedte**

Hier moet je gebruik maken van de frequentiedichtheid (zie opgave 1.1, figuur 1.5, blz. 44)

#### 1.3 Grafische voorstellingen

**Lijndiagram**

Wordt gebruik om de ontwikkeling van een variabele in de tijd weer te geven (x=tijd, y=variabele).

Voor de y-as kunne we ook een logaritmische schaalverderen (ipv lineair) gebruik maar dit hoeven we nog niet te leren.

Een andere variant van een lijndiagram is een gestapeld lijndiagram (vb 1.11, fig 1.13, blz. 32)

**Spreidingsdiagram (scatter plot)**

Wordt gebruikt als de x en y allebeide een variabele zijn. Hierin is x de onafhankelijke (oorzaak) variabele en is y de afhankelijke (gevolg) variabele. Dus $y=f(x)$.

Meetniveau: 2x interval/ratio/ordinaal met veel schaalpunten.

Dit diagram is handig om te zien of de variabelen positief/negatief of helemaal geen correlatie (samenhang) hebben.

#### 1.4 Diverse diagrammen

**Staafdiagrammen**

Meetniveau: nominaal of ordinaal (met weinig schaalpunten)

**Cirkeldiagram**

Niet gebruiken bij meer dan 5 schaalpunten.

Meetniveau: nominaal of ordinaal.

$100\%$ van de waarnemingen zijn $360^\circ$. Let er dus op dat het getekende percentage de correcte hoek heeft... Let ook op de oppervlakte (als een gegeven 2 maal zo groot is dan moet de oppervlakte van dat cirkeldiagram ook twee keer zo groot zijn als de andere...)

* Oppervlakte cirkel: $\pi R^2$
* Omtrek cirkel: $2\pi R$

#### 1.5 Stamdiagrammen (leaf plot)

Meetniveau: Ordinaal of hoger

Begin met het zoeken van de $\text{Min}$, en de $\text{Max}$ waarde in de data. Daarna kunnen we deze rangschikken.

Ook kunnen we een gesplitst stamdiagram maken, dit is handig als er heel veel waarnemingen in het hogere/lagere segment van een stam zit.

Tot slot is het mogelijk om een tweezijdig stamdiagram te maken. En uiteraard ook een gesplitst tweezijdig stamdiagram...

**Pareto diagram**

Dit is een staafdiagram + een cumulatieve frequentiepolygon.

1. Frequentie rangschikken vanaf groot naar klein
2. Bereken $F$.
3. Bereken $F$ in $\%$.
4. Teken paretodiagram.

Voorbeeld 1.26, blz. 49

* 80/20 regel

## Week 3

### Centrummaten voor ongegroepeerde waarnemingsuitkomsten

#### Rekenkundig gemiddelde ($\bar{x}$)

Het rekenkundig gemiddelde kunnen we berekenen bij gegevens met het meetniveau: interval of ratio. De aanduiding hiervoor is $\bar{x}$.

Door alle waarnemingen bij elkaar op te tellen en dit te delen door het aantal waarnemingen krijgen we het rekenkundig gemiddelde.

$$\bar{x} = \frac{\sum\limits^{n}_{x=i}X_i}{n}$$

#### Gewogen rekenkundig gemiddelde  ($\bar{x}$)

Het gewogen rekenkundig gemiddelde kunnen we gebruiken om bepaalde waarnemingen zwaarder mee te laten tellen. Dit kan je bijvoorbeeld gebruiken om het gemiddelde van een aantal toetsen uit te rekenen waarbij elke toets een ander gewicht kan hebben.

Alle waarnemingen worden vermenigvuldigd met het gewicht en daarvan de som wordt weer gedeeld met de som van alle gewichten.

$$\bar{x} = \frac{\sum w_i\cdot x_i}{\sum w_i}$$

#### Mediaan ($x_{me}$)

Om de mediaan te bepalen moet eerst de data worden gerangschikt. De mediaan wordt bepaald op $1$ variabele. Het meetniveau is ordinaal. Dit wordt soms gebruikt om een beter gemiddelde te laten zien wanneer er op de uiteindes van de waarnemingen hele hoge frequenties zitten. Dit geeft een vertekend beeld in het rekenkundig gemiddelde.

Om de mediaan te bepalen nadat alle data is gerangschikt nemen we de middelste waarneming. Als er een even aantal waarnemingen zijn dan tellen we de middelste twee bij elkaar op en delen we dit door $2$.

#### Modus ($x_{mo}$)

De modus is het getal dat het vaakste voorkomt. Dit mag maar $1$ getal zijn. Als er meerdere getallen zijn die evenvaak voorkomen dan is de modus niet gedefinieerd.

De modus kan worden bepaald op gegevens met het nominale meetniveau.

### Centrummaten voor gegroepeerde waarnemingsuitkomsten

#### Rekenkundig gemiddelde

We kunnen het rekenkundig gemiddelde wel berekenen voor gegroepeerde waarnemingsuitkomsten, echter is dit een schatting van het gemiddelde.

Om dit doen nemen we de som van alle klassen maal de frequentie van de klasse en delen we dit door het aantal waarnemingen.

$$\bar{x}=\frac{ \sum\limits^k_{i=1}f_i \cdot m_i}{n}$$

Hier is $f$ de frequentie van de klasse, $m$ het klassenmidden van de klasse en $n$ het totaal aantal waarnemingen.

#### Mediaan ($x_{me}$)

Om de mediaan te bepalen voor gegroepeerde waarneminguitkomsten gebruiken we de volgende formule:

$$ x_{me} = L + \big( r -\frac{1}{2} \big) \frac{b}{f}$$

Hier is $L$ de benedengrens van de mediale klasse, $r$ het rangnummer van de waarneming binnen de klasse, $b$ de breedte van de mediale klassen en $f$ het totaal aantal waarnemingen in de mediale klasse.

#### Modus ($x_{mo}$)

Om de modus te bepalen nemen we de klasse met de hoogste frequentiedichtheid. Let er op dat je niet de klasse met de hoogste frequentie gebruikt!

### Spreidingsmaten bij ongegroepeerde waarnemingsuitkomsten

#### Spreidingsbreedte

Om de spreidingsbreedte (range) nemen we de hoogste waarnemingsuitkomt en trekken we hier de laagste waarnemingsuitkomst van af.

#### Gemiddelde absolute afwijking

Om de gemiddele absolute afwijking (G.A.A.) te bepalen gebruiken we de volgende formule:

$$\text{GAA} = \frac{ \sum\limits^n_{i=n}(\mid x_i - \bar{x} \mid)}{n}$$

#### Variantie

Dit kunnen we toepassen op data met het meetniveau interval of ratio. Let er op of we dit voor de populatie doen of voor de steekproef.

$$s^2 = \frac{\sum\limits^n_{i=1}(x_i-\bar{x})^2}{n-1} \quad \text{(steekproef)} \qquad \sigma^2 = \frac{\sum\limits^n_{i=1}(x_i - \mu)^2}{N} \quad \text{(populatie)}$$

Waarbij de rekenkundige gemiddelden als volgt zijn bepaald:

$$\bar{x} = \frac{\sum\limits^n_{i=1}x_i}{n} \quad \text{(steekproef)} \qquad \mu = \frac{\sum\limits^n_{i=1}x_i}{N} \quad \text{(populatie)}$$

#### Standaard deviatie

Omdat we bij de variantie een ander meeteenheid krijgen, namelijk $x^2$ tot de macht, kunnen we de oorspronkelijke meeteenheid terug krijgen de wortel te nemen van de variantie. Dit wordt dan de standaard deviate genoemd.

$$s = \sqrt{\frac{\sum\limits^n_{i=1}(x_i-\bar{x})^2}{n-1}} \quad \text{(steekproef)} \qquad \sigma = \sqrt{\frac{\sum\limits^n_{i=1}(x_i - \mu)^2}{N}} \quad \text{(populatie)}$$

### Spreidingsmaten bij gegroepeerde waarnemingsuitkomsten

Om bij gegroepeerde waarnemingsuitkomsten de centrummaten te bereken is het voldoende om te laten zien hoe dit voor de standaard deviate wordt gedaan. Omdat de standaard deviate alle bovenstaande centrummaten bevat is het simpel om te zien hoe deze zijn aangepast voor gegroepeerde waarnemingsuitkomsten.

$$s = \sqrt{\frac{\sum\limits^n_{i=1}f_i(m_i-\bar{x})^2}{n-1}} \quad \text{(steekproef)} \qquad \sigma = \sqrt{\frac{\sum\limits^n_{i=1}f_i(m_i - \mu)^2}{N}} \quad \text{(populatie)}$$

Waarbij $m_i$ het klassenmidden is en $f_i$ de frequentie van die klasse. Let er ook op dat $\bar{x}$ en $\mu$ nu met het rekenkundig gemiddelde voor gegroepeerde waarnemingsuitkomsten moet worden berekend.

## Week 4

**Kwartielen**

Er zijn 4 kwartielen ($Q1,Q2,Q3,Q4$), $Q2$ is hetzelfde als $x_{me}$.

Kwartielen worden op dezelfde manier berekend als de mediaan. Hiervoor wordt er wel naar een ander middenpunt gezocht. Om $Q1$ te krijgen zoeken we de middelste waarneming tussen de eerste waarde en $Q2$. Voor $Q3$ doen we dit ook maar dan tussen $Q2$ en de laatste waarneming.

**Decilen**

Er zijn in totaal 9 decilen; $D1,D2,\ldots,D9$.

**Percentilen**

Er zijn in totaal 99 percentilen: $P1, P2, \ldots, P99$. De laatste wordt niet mee gerekend.

**Halve kwartiel afstand (HK)**

Om de halve kwartiel afstrand te bereken gebruiken we onderstaande formule:

$\text{HK}=\frac{Q3-Q1}{2}$

**Inter-kwartiel range (IQR)**

Om de inter-kwartiel range te bereken gebruiken we $\text{IQR}=Q3-Q1$.

** Boxplot **

Om een boxplot te maken gebruiken we de onderstaande stappen:

1. Waarnemingen rangschikken
2. Bepaal die minimale waarde
3. Bepaal de maximale waarde
4. Bepaal de mediaan ($Q2$)
5. Bepaal $Q1, Q3$

Nu kunnen we de boxplot tekenen. Als er **geen** rekening wordt gehouden met uitschieters dan is het een _simpele_ boxplot.

Als we **wel** rekening houden met uitschieters dan hebben we een _gewone_ boxplot. In dit geval mogen de lijnstukken van min/max niet langer zijn dan $1\frac{1}{2}\text{IQR}$. Alle waarnemingen die hierbuiten vallen worden met een kruisje aangegeven. Vervolgens wordt de eerst volgende waarneming gebruikt die binnen deze range valt als min/max.

## Week 5

**Meetkundig gemiddelde**

Dit wordt gebruikt om het gemiddelde te berekenen bij verhoudingsgetallen (zoals bijv. groeipercentages). Dit alleen van toepassing bij ongegroepeerde waarnemingen.

$$\bar{x}_{mk} = \left(\prod x_i \right)^{\frac{1}{n}} = \sqrt[n]{\prod\limits^n_{i=1}x_i} = \sqrt[n]{x_1 \cdot x_2 \cdot \ldots \cdot x_n}$$

**Harmonisch gemiddelde**

Dit wordt gebruikt om gemiddelde te berekenen bij samengestelde grootheden (bijv. snelheid). Ook dit is alleen van toepassing bij ongegroepeerde waarnemingen.

$$\bar{x}_h = \frac{n}{\cfrac{1}{x_1} + \cfrac{1}{x_2} + \ldots + \cfrac{1}{x_n}}$$

**Populatie & steekproef**

De gemiddelden zijn als volgt gedefinieerd:

$$ \bar{x} = \cfrac{ \sum\limits^n_{i=1}x_i }{n} \quad(\text{Sample}) \qquad \qquad \mu = \frac{ \sum\limits^n_{i=1} x_i}{n} \quad(\text{Population})$$

En de variantie:

$$ s^2 = \cfrac{ \sum\limits^n_{i=1}(x_i - \bar{x})^2 }{n-1} \quad (\text{Sample}) \qquad \qquad \sigma^2 = \cfrac{ \sum\limits^N_{i=1}(x_i - \mu)^2}{N} \quad (\text{Population})$$

Waarbij:
* $\bar{x} =$ steekproefgemiddelde
* $\mu =$ populatiegemiddelde
* $s =$ steekproefstandaardafwijking
* $\sigma =$ populatiestandaardafwijking
* $n =$ steekproefomvang
* $N =$ populatieomvang

**Eigenschappen van $\bar{x}$ en $s^2$**

Laten we eerste twee definities bewijzen voor sommen.

**Definitie 1:**

$\sum\limits^n_{i=1}(x_i+b) = nb + \sum\limits^n_{i=1} x_i$

Bewijs:

$\begin{align} \sum\limits^n_{i=1}(x_i+b) &= x_1+b + x_2+b + \ldots + x_n+b \\ &= (x_1 + x_2 + \ldots + x_n) + (b + b + \ldots + b) \\ &= (x_1 + x_2 + \ldots +x_n) + (nb) \\ &= nb \sum\limits^n_{i=1}x_i\end{align}$

**Definitie 2:**

$\sum\limits^n_{i=1}(bx_i) = b \cdot \sum\limits^n_{i=1} x_i$

Bewijs:

$\begin{align} \sum\limits^n_{i=1} (bx_i) &= bx_1 + bx_2 + \ldots + bx_n \\ &= b(x_1 + x_2 + \ldots + x_n) \\ &= b \cdot \sum\limits^n_{i=1} x_i \end{align}$

Als we een lineaire transformatie ($y=ax+b$) toepassen op alle waarnemingen $x_i$ dan wordt het nieuwe gemiddelde $\bar{x}_{nieuw} = a\cdot \bar{x}_{oud} + b$.

Bewijs:

De nieuwe $x_i^1 = ax_i+b$, dus:

$\begin{align} \bar{x}_{nieuw} &= \frac{ \sum\limits^n_{i=1} (x_i^1) }{n} \\ &= \frac{ \sum\limits^n_{i=1} \left(ax_i+b\right) }{n} \quad &(x_i^1 = ax_i+b) \\ &= \frac{ nb + a \sum\limits^n_{i=1}x_i}{n} \quad &\left(\text{Def. 1}, \text{Def. 2}, \frac{nb}{n} = b\right) \\ &= b + a \left( \frac{\sum\limits^n_{i=1}x_i}{n}\right) \\ &= a\bar{x}_{oud} +b\end{align}$

Als we een lineaire transformatie toepassen op alle waarnemingen $x_i$ dan wordt de nieuwe variantie $s^2_{nieuw}=a^2\cdot s^2_{oud}$.

Bewijs:

Als $x_i^1 = ax_i +b$ en $\bar{x}_{nieuw} = a\bar{x}_{oud}+b$, dan:

$\begin{align} s^2_{nieuw} &= \frac{ \sum\limits^n_{i=1}(x_i^1-\bar{x}_{nieuw})^2 }{n-1} \\ &= \frac{\sum\limits^n_{i=1}((ax_i+b)-(a\bar{x}_{oud}+b))^2}{n-1} \quad &(\text{Replace  } x_i^1, \text{Replace  } \bar{x}_{nieuw}) \\ &= \frac{\sum\limits^n_{i=1}(ax_i - a\bar{x}_{oud})^2}{n-1} \quad &(\text{Cancel b}) \\ &= \frac{\sum\limits^n_{i=1}a^2(x_i - \bar{x}_{oud})^2}{n-1} \quad &(\text{Factor a}) \\ &= a^2 \cdot \frac{\sum\limits^n_{i=1} (x_i - \bar{x}_{oud} )^2 }{n-1} \quad &(\text{Def. 2}) \\ s^2_{nieuw} &= a^2s^2_{oud} \end{align}$

De standaardafwijking is te verkrijgen door $\sqrt{s^2_{nieuw}} = \sqrt{a^2s^2_{oud}}$, ofterwel $s_{nieuw} = |a|\cdot s_{oud}$.

**Alternatieve formule voor $s^2$**

Todo: add proof

$$s^2 = \cfrac{\sum\limits^n_{i=1}x_i^2}{n-1} - \cfrac{\left(\sum\limits^n_{i=1} x_i\right)^2}{n(n-1)}$$

#### Bayesiaanse gemiddelde

Om het Bayesiaanse gemiddelde te berekenen gebruiken we de volgende formule:

$\bar{X}(j)_{Bayes}=\cfrac{C \times m + \sum X_i}{C + n}$

Waarbij:

* $j$ is item
* $C$ is arbitraire wegingsfactor
* $m$ is gemiddelde van alle waarnemingen (gewogen gemiddelde)
* $X_i$ is waarneming vermenigvuldigd met score

```python

```