<center>

## [mlcourse.ai](mlcourse.ai) – Open Machine Learning Course
### <center> Author: Artem Kuznetsov, ODS Slack te

## <center> Exploring TED Talks

**Research plan**
    - Dataset and features description
    - Exploratory data analysis
    - Visual analysis of the features
    - Patterns, insights, pecularities of data
    - Data preprocessing
    - Metric selection
    - Feature engineering and description
    - Cross-validation, hyperparameter tuning
    - Validation and learning curves
    - Prediction for hold-out set
    - Model selection
    - Conclusions

### Part 1. Dataset and features description

TED is the conference organizer, which holds events were people from different areas can have a public talk of important ideas. Last years TED had significantly grown in popularity due video and audio recordings publications of talks.

The dataset was collected by Rounak Banik and stored to Kaggle https://www.kaggle.com/rounakbanik/ted-talks/. It's not sure how it was collected by web scrapping or by TED api (now closed). Data contains talks before September 21st, 2017.

Data set constists of two files:

ted_main.csv - metadata about talks and speakers

- comments- The number of first level comments made on the talk (number)
- description - A blurb of what the talk is about (string)
- duration - The duration of the talk in seconds (number)
- event - The TED/TEDx event where the talk took place (string)
- film_date - The Unix timestamp of the filming (date in unix time format)
- languages - The number of languages in which the talk is available (number)
- main_speaker - The first named speaker of the talk (string)
- name - The official name of the TED Talk. Includes the title and the speaker. (string)
- num_speaker - The number of speakers in the talk (number)
- published_date - The Unix timestamp for the publication of the talk on TED.com (date in unix time format)
- ratings - A stringified dictionary of the various ratings given to the talk (inspiring, fascinating, jaw dropping, etc.) (json)
- related_talks - A list of dictionaries of recommended talks to watch next (json)
- speaker_occupation - The occupation of the main speaker (string)
- tags - The themes associated with the talk (list)
- title - The title of the talk (string)
- url - The URL of the talk (string)
- views - The number of views on the talk (number)

transcripts.csv - talk transcripts

- transcript - The official English transcript of the talk. (string)
- url - The URL of the talk (string)

Target of this project is to to research how can be predicted count of views.

```python
import re
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import statsmodels.api as sm
import seaborn as sns
import scipy.stats

from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split, TimeSeriesSplit, GridSearchCV, learning_curve
from sklearn.model_selection import cross_val_score
from sklearn.metrics import mean_squared_error, mean_absolute_error
from sklearn.linear_model import LinearRegression, Ridge

DATA_PATH = '../data/'

# Set up seeds
RANDOM_SEED = 42
np.random.seed(RANDOM_SEED)

%matplotlib inline
```

```python
plt.rcParams['figure.figsize'] = 12., 9.
```

### Part 2. Exploratory data analysis

```python
# Load data
df_ted_main = pd.read_csv(DATA_PATH + 'ted_main.csv.zip')
df_ted_transcripts = pd.read_csv(DATA_PATH + 'transcripts.csv.zip')
```

```python
df_ted_main.info()
```

```python
df_ted_transcripts.info()
```

The datasets contains different count of records, so probably there are fewer transcripts then talks.

#### Duplicates check

```python
df_ted_main[df_ted_main.duplicated()]
```

```python
df_ted_transcripts[df_ted_transcripts.duplicated()]
```

Ok, we've got some in df_ted_transcripts, let's remove them.

```python
df_ted_transcripts = df_ted_transcripts.drop_duplicates()
```

#### Merge datasets

```python
df_ted_main.shape, df_ted_transcripts.shape
```

```python
df_ted = pd.merge(df_ted_main, df_ted_transcripts, how='left', on='url')
df_ted.shape
```

```python
df_ted.columns
```

```python
df_ted.head()
```

```python
DATE_COLUMNS = 'film_date', 'published_date'
for column in DATE_COLUMNS:
    df_ted[column] = pd.to_datetime(df_ted[column], unit='s')
```

```python
df_ted.info()
```

#### Missing values

Looks like we have transcript for almost all talks but also have missing values. Also some values of speaker_ocupation is missing.

#### Recheck NA's

```python
for column in df_ted.columns:
    na_count = df_ted[column].isna().sum()
    if na_count > 0:
        print('%s : %s' % (column, na_count))
```

#### Common numerics stats

```python
df_ted.describe()
```

```python
df_ted.median()
```

### description

```python
df_ted['description'].nunique(), len(df_ted['description'])
```

```python
df_ted['description'].str.len().describe()
```

```python
df_ted['duration'].values[:100]
```

Each talk has an unique description.

### event

```python
df_ted['event'].value_counts()
```

We had different type of events here whith the most popular TED2014 event. We can see TED and TEDx events, and some events differen from it. Let's investigate a little more.

```python
sorted(df_ted['event'].unique())
```

```python
sorted(df_ted[df_ted['event'].str.startswith('TEDx')]['event'].unique())
```

```python
sorted(df_ted[df_ted['event'].str.startswith('TEDx') == False]['event'].unique())
```

We can add some feature to distinct different types of events.

```python
def get_event_type(event):
    '''
    Returns type of event
    '''
    if not 'TED' in event:
        return 'NOT_TED'
    elif event.startswith('TEDx'):
        return 'TEDx'
    elif event.startswith('TED@'):
        return 'TED@'
    elif re.fullmatch('TED\d{4}', event) is not None:
        return 'TED_YEAR'
    else:
        return event.split()[0]
```

```python
df_ted['event'].apply(get_event_type).value_counts()
```

Wikipedia has some additional info on different conference types https://en.wikipedia.org/wiki/TED_(conference)

```python
df_ted.columns
```

#### film_date

```python
df_ted['film_date'].describe()
```

Some talks have filming date year as early as 1972. Let's try to find some more.

```python
df_ted[df_ted['film_date'] < '2000-01-01']
```

Can be seen that there are three talks, that are not from TED and filmed before 1992.

#### languages

```python
df_ted['languages'].describe(), df_ted['languages'].median()
```

Interesting, some of the talks has language count equal to zero. Let's investigate a little bit.

```python
df_ted[df_ted['languages'] == 0]
```

```python
df_ted[df_ted['languages'] == 0]['url'].values[:10]
```

Most of those are art perfomances, but not all. Also for those records there is no transcript.

#### main_speaker

```python
df_ted['main_speaker'].value_counts()
```

```python
df_ted['main_speaker'].value_counts().describe()
```

Most of people have a talk at TED events only once.

```python
df_ted['main_speaker'].str.len().describe()
```

```python
df_ted[df_ted['main_speaker'].str.len() > 20]
```

In case of long main_speaker field we can suspect more then one speaker.

#### name

```python
df_ted['name'].nunique()
```

Every talk has an unique name.

```python
df_ted['name'].str.len().describe()
```

#### num_speaker

```python
df_ted['num_speaker'].describe()
```

Most of people present their talks alone.

#### published_date

```python
df_ted['published_date'].describe()
```

```python
df_ted[df_ted['event'].str.startswith('TED')]['film_date'].min()
```

Fist published_date is 2006-06-27, but first filmed talk was on 1984-02-02. So it may be interesting to have a look at timespan between the filming and the publication.

```python
(df_ted['published_date'] - df_ted['film_date']).describe()
```

```python
(df_ted['published_date'] - df_ted['film_date']).median()
```

```python
df_ted[(df_ted['published_date'] - df_ted['film_date']).dt.total_seconds() < 0]
```

Interesting, looks like we have some mistakes in data. The records above are where published_date is earlier then film_date which is the case.

#### ratings

It's a rating from TED site. TED asks people to describe video (talk) in three words. Count simply means amount of people who choosen the category.
We will not use the field because it's closely linked with our target variable "views".
More views video has more people rated it.

```python
df_ted['ratings'].values[0]
```

```python
df_ted['ratings'].values[1]
```

```python
df_ted['ratings'].values[2]
```

```python
df_ted['ratings'].values[3]
```

#### related_talks

We will not use the field in research due to it complexity for analysis.

```python
df_ted['related_talks'].values[0]
```

#### speaker_occupation

```python
df_ted['speaker_occupation'].value_counts()
```

```python
df_ted['speaker_occupation'].str.len().describe()
```

```python
df_ted[df_ted['speaker_occupation'].str.len() > 50]['speaker_occupation']
```

The most popular occupations are  from arts, business, journalism, architecture and psychology.
Some of people describe themself with a lot of different occupation types. Count of occupations could be a feature later.

#### tags

```python
df_ted['tags'].values[:5]
```

```python
df_ted['tags'] = df_ted['tags'].apply(lambda x: eval(x))
```

```python
df_ted['tags'].values.reshape(-1,1)
```

```python
type(df_ted['tags'].values[0])
```

```python
# Some code to flatten list of tags
df_ted['tags'].apply(pd.Series).reset_index().melt(id_vars='index').value.dropna().value_counts()
```

#### title

```python
df_ted['title'].nunique()
```

Every talk has his own title.

```python
df_ted['title'].str.len().describe()
```

```python
df_ted['title'].values[:5]
```

Looks like title + main_speaker = name

```python
df_ted[['name', 'main_speaker', 'title']].head()
```

#### url

```python
df_ted['url'].nunique()
```

```python
df_ted['url'].values[:5]
```

```python
sum(df_ted['url'].str.endswith('\n'))
```

Every url ends with '\n', so it could be cleaned.

```python
df_ted['url'] = df_ted['url'].str.strip()
```

```python
df_ted['url'].apply(lambda s: s.split('/')[0]).value_counts()
```

```python
df_ted['url'].apply(lambda s: s.split('/')[2]).value_counts()
```

```python
df_ted['url'].apply(lambda s: s.split('/')[3]).value_counts()
```

All urls are 'https://www.ted.com/talks/name_of_talk' so we could omit the field without consequences.

#### views

'views' is our target variable. We also need to check normality of it distribution.

```python
df_ted['views'].describe()
```

Doesn't look normal distributed. Let's check via plots and stat tests.

```python
df_ted['views'].hist(bins=100);
```

```python
scipy.stats.normaltest(df_ted['views'])
```

```python
scipy.stats.shapiro(df_ted['views'])
```

```python
sm.qqplot(df_ted['views'], line='s');
```

```python
scipy.stats.normaltest(np.log(df_ted['views']))
```

```python
scipy.stats.shapiro(np.log(df_ted['views']))
```

```python
sm.qqplot(np.log(df_ted['views']), line='s');
```

```python
np.log(df_ted['views']).hist(bins=100);
```

```python
alpha = 0.001
p = scipy.stats.shapiro(np.log(np.log(df_ted['views'])))[1]

if p < alpha:  # null hypothesis: x comes from a normal distribution
    print("The null hypothesis can be rejected")
else:
    print("The null hypothesis cannot be rejected")
```

It doesnt't looks like we get normal distibution after applying logarithm, but it looks much closer to it. So we will assume that our target variable has normal distribution.

```python
df_ted.columns
```

```python
df_ted['target'] = np.log(df_ted['views'])
```

#### transcripts

```python
df_ted['transcript'].nunique(), len(df_ted['transcript']), sum(df_ted['transcript'].isna())
```

Not every talk has transcipt and each transcript in unique.

```python
df_ted['transcript'].str.len().describe()
```

```python
df_ted[df_ted['transcript'].str.len() < 200]['transcript'].values
```

Ok, looks like some transcript are from music.

### Part 3. Visual analysis of the features

```python
df_ted.columns
```

```python
df_ted.drop('views', axis=1, inplace=True)
```

```python
df_ted.drop('related_talks', axis=1, inplace=True)
```

```python
df_ted.drop('comments', axis=1, inplace=True)
```

```python
# Make separate dataframe for data preparation for plotting
df_plot = df_ted.copy()
df_plot['film_date_unix'] = df_ted['film_date'].astype(int)
df_plot['published_date_unix'] = df_ted['published_date'].astype(int)
```

```python

```

```python
%%time

sns.pairplot(df_plot, diag_kind="kde", markers="+",
    plot_kws=dict(s=50, edgecolor="b", linewidth=1),
    diag_kws=dict(shade=True));
```

Clearly visible correlation between number of languages and views count.

```python
df_ted.columns
```

```python
df_plot.corr(method='pearson')
```

```python
sns.heatmap(df_plot.corr(method='pearson').abs(), annot=True)
```

```python
df_plot.corr(method='spearman')
```

```python
sns.heatmap(df_plot.corr(method='spearman').abs(), annot=True)
```

```python
plt.plot(df_plot['published_date_unix'])
```

So, data is sorted by published_date

```python
plt.plot(df_plot['target'])
```

```python
plt.plot(df_plot['published_date_unix'], df_plot['target'])
```

```python
sns.countplot(df_ted['event']);
```

```python
plt.plot(df_plot.groupby(by='event')['target'].mean().sort_values(ascending=False), 'o-');
```

Target mean variable looks like near normally disributed related to event name.

```python
sns.countplot(df_ted['main_speaker']);
```

```python
plt.plot(df_plot.groupby(by='main_speaker')['target'].mean().sort_values(ascending=False), 'o-');
```

The normality of disribution also holds for speaker name.

```python
sns.countplot(df_ted['speaker_occupation']);
```

```python
plt.plot(df_plot.groupby(by='speaker_occupation')['target'].mean().sort_values(ascending=False), 'o-');
```

speaker_occupation also looks normaly distributed.

### Part 4. Patterns, insights, pecularities of data

From previous analysis we have following observations:
- 'view' variable is not normally distributed according to tests. For linear model it's should be more adequate to use normally distributed target variable, so we apply logarithm to it. It doesn't makes distribution normal but now it looks closer to it.
- Our new target variable highly correlated with language count. May be it's due the fact that the most popular talks are moe often translated to more languages. Because we are doing correlation analysis we cann't say it for sure without additional data. But it can be useful to omit language variable.
- Also we have strong correlation between published and film date, so we need to exclude one of them for more accurate predicictions.
- We clearly see different kind of events, so may be useful to add additional feature with event type information
- Url and name just redunant because contains information also available from other fields
- Data is sorted by published date, so we can use TimeSeriesSplit to not get catched by data leak. Also it's fine because we are interested in feature prediction so we don't need to sort data.
- We have some errors in data, when published_date is smaller than film_date, but as we are more interested in date of publication and published_date and film_date are highly correlated we will exclude film_date

### Part 5. Data preprocessing

We will do different type of preprocessing for different type of columns:
- numeric columns will be scaled using StandardScaler
- text columns will be converter to lower case and after that vectorized using TfIdfVectorizer
- categorial variables will be factorized (similar to label encoding) and then transformed using OneHotEncoder
- empty values in transcript field will be filled with 'na' string
- date will be converted to unix time and used as numeric
- tags arrays will be converted to string and used as string column

```python
df_ted.columns
```

```python
# Only leave features filtred by assumptions from previous part
X = df_ted[['description', 'duration', 'event', 'languages',
       'main_speaker', 'num_speaker', 'published_date',
       'speaker_occupation', 'tags', 'title', 'transcript']].copy()
y = df_ted['target'].copy()

NUMERIC_COLUMNS = ['duration', 'languages']
DATE_COLUMNS = ['published_date']

# We will convert 'tags' column to string
TEXT_COLUMNS = ['description', 'tags', 'title', 'transcript']

CATEGORICAL_COLUMNS = ['event',
       'main_speaker', 'speaker_occupation', 'num_speaker']

# We will convert published_date back to unix time and use it like numeric column
for c in DATE_COLUMNS:
    X[c] = X[c].astype(int)

# We will use data columns simply as numeric_column, so
NUMERIC_COLUMNS += DATE_COLUMNS

# StandardScaler will convert fields to float64 with warning, so we will do it before
for c in NUMERIC_COLUMNS:
    X[c] = X[c].astype(float)

# Convert tags to string
X['tags'] = X['tags'].apply(lambda tags: ' '.join(tags))

X['transcript'] = X['transcript'].fillna('na')

# Convert all text columns to lower case
for c in TEXT_COLUMNS:
    X[c] = X[c].str.lower()

# Factorize categorical_columns (similar to LabelEncoding)
for c in CATEGORICAL_COLUMNS:
    X[c] = X[c].factorize()[0]
```

```python
X.head()
```

```python
preprocessing = ColumnTransformer(transformers=[
    ('ohe', OneHotEncoder(categories='auto', handle_unknown='ignore'), CATEGORICAL_COLUMNS),
    ('scaler', StandardScaler(), NUMERIC_COLUMNS),
    ('tfidf_0', TfidfVectorizer(), TEXT_COLUMNS[0]),
    ('tfidf_1', TfidfVectorizer(), TEXT_COLUMNS[1]),
    ('tfidf_2', TfidfVectorizer(), TEXT_COLUMNS[2]),
    ('tfidf_3', TfidfVectorizer(), TEXT_COLUMNS[3]),
])
```

```python
# It's crucial not sort splits, because we want to predict future (so no future data should be in train set)
# We don't need seed here because shuffle is disabled
X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size = 0.3, shuffle=False)
```

```python
X_train.shape, X_valid.shape, y_train.shape, y_valid.shape
```

### Part 6.  Metric selection

For regression task there are two most popular metrics - RMSE and MAE.

$
\begin{align}
RMSE = \sqrt{\frac{1}{n}\sum_{j=1}^{n}{(\hat{y} - y_j)^2}}
\end{align}
$

$
\begin{align}
MAE = \frac{1}{n}\sum_{j=1}^{n}{\lvert\hat{y} - y_j\rvert}
\end{align}
$

RMSE put higher weight on the bigger errors in predictions.
RMSE has tendency to increase more then MAE with bigger sample size.
In our case bigger errors should not be threated in special way.
MAE is more easy to interpretate, especially as we have log transformation of initial target variable, so exp(MAE) could be viewed as multiplicator of true value of the original variable.

So we will go with MAE.

### Part 7. Feature engineering and description

Let's try Ridge from sklearn.

```python
%%time
model_ridge = Pipeline(
    steps=[
        ('preprocessing', preprocessing),
        ('ridge', Ridge(random_state=RANDOM_SEED))
    ]
)

cv = GridSearchCV(model_ridge, param_grid={}, scoring='neg_mean_absolute_error', cv=TimeSeriesSplit(n_splits=5),
                 return_train_score=True, verbose=3)
cv.fit(X_train, y_train)
```

```python
cv.best_score_
```

We will use it as baseline for future.

Let's construct new features:
- len of transcript (because target can depends on how much time presenter speaks)
- event type, because it's some events are clearly more popular (like TED events vs regional TEDx events)
- published_date hour, month, dayofweek

```python
# Only leave features filtred by assumptions from previous part
X = df_ted[['description', 'duration', 'event', 'languages',
       'main_speaker', 'num_speaker', 'published_date',
       'speaker_occupation', 'tags', 'title', 'transcript']].copy()
y = df_ted['target'].copy()

X['transcript'] = X['transcript'].fillna('na')
X['transcript_len'] = X['transcript'].str.len()
X['event_type'] = X['event'].apply(get_event_type)
X['published_hour'] = X['published_date'].dt.hour
X['published_month'] = X['published_date'].dt.month
X['published_dayofweek'] = X['published_date'].dt.dayofweek

NUMERIC_COLUMNS = ['duration', 'languages',
                   'transcript_len'
                  ]
DATE_COLUMNS = ['published_date']

# We will convert 'tags' column to string
TEXT_COLUMNS = ['description', 'tags', 'title', 'transcript']

CATEGORICAL_COLUMNS = ['event',
       'main_speaker', 'speaker_occupation', 'num_speaker',
                       'event_type',
                       'published_hour',
                       'published_month',
                       'published_dayofweek'
                      ]

# We will convert published_date back to unix time and use it like numeric column
for c in DATE_COLUMNS:
    X[c] = X[c].astype(int)

# We will use data columns simply as numeric_column, so
NUMERIC_COLUMNS += DATE_COLUMNS

# Convert tags to string
X['tags'] = X['tags'].apply(lambda tags: ' '.join(tags))

# StandardScaler will convert fields to float64 with warning, so we will do it before
for c in NUMERIC_COLUMNS:
    X[c] = X[c].astype(float)

# Convert all text columns to lower case
for c in TEXT_COLUMNS:
    X[c] = X[c].str.lower()

# Factorize categorical_columns (similar to LabelEncoding)
for c in CATEGORICAL_COLUMNS:
    X[c] = X[c].factorize()[0]

X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size = 0.3, shuffle=False)
```

We will test new features it one by one, using ColumnTransformer propery - it will drop columns, not mentnioned in transformers list.

#### Let's try to exclude published_date

```python
%%time
NUMERIC_COLUMNS = ['duration', 'languages']

CATEGORICAL_COLUMNS = ['event',
       'main_speaker', 'speaker_occupation', 'num_speaker']

preprocessing = ColumnTransformer(transformers=[
    ('ohe', OneHotEncoder(categories='auto', handle_unknown='ignore'), CATEGORICAL_COLUMNS),
    ('scaler', StandardScaler(), NUMERIC_COLUMNS),
    ('tfidf_0', TfidfVectorizer(), TEXT_COLUMNS[0]),
    ('tfidf_1', TfidfVectorizer(), TEXT_COLUMNS[1]),
    ('tfidf_2', TfidfVectorizer(), TEXT_COLUMNS[2]),
    ('tfidf_3', TfidfVectorizer(), TEXT_COLUMNS[3]),
])

model_ridge = Pipeline(
    steps=[
        ('preprocessing', preprocessing),
        ('ridge', Ridge(random_state=RANDOM_SEED))
    ]
)

cv = GridSearchCV(model_ridge, param_grid={}, scoring='neg_mean_absolute_error', cv=TimeSeriesSplit(n_splits=5),
                 return_train_score=True, verbose=3)
cv.fit(X_train, y_train)
```

```python
cv.best_score_
```

We have some improvement in score, let's continue.

#### transcript_len

```python
%%time
NUMERIC_COLUMNS = ['duration', 'languages', 'transcript_len'
                  ]

CATEGORICAL_COLUMNS = ['event',
       'main_speaker', 'speaker_occupation', 'num_speaker']

preprocessing = ColumnTransformer(transformers=[
    ('ohe', OneHotEncoder(categories='auto', handle_unknown='ignore'), CATEGORICAL_COLUMNS),
    ('scaler', StandardScaler(), NUMERIC_COLUMNS),
    ('tfidf_0', TfidfVectorizer(), TEXT_COLUMNS[0]),
    ('tfidf_1', TfidfVectorizer(), TEXT_COLUMNS[1]),
    ('tfidf_2', TfidfVectorizer(), TEXT_COLUMNS[2]),
    ('tfidf_3', TfidfVectorizer(), TEXT_COLUMNS[3]),
])

model_ridge = Pipeline(
    steps=[
        ('preprocessing', preprocessing),
        ('ridge', Ridge(random_state=RANDOM_SEED))
    ]
)

cv = GridSearchCV(model_ridge, param_grid={}, scoring='neg_mean_absolute_error', cv=TimeSeriesSplit(n_splits=5),
                 return_train_score=True, verbose=3)
cv.fit(X_train, y_train)
```

```python
cv.best_score_
```

Previous value was -0.4767736055960914, so we have some small imporvement.

#### event_type

```python
%%time
NUMERIC_COLUMNS = ['duration', 'languages', 'transcript_len']

CATEGORICAL_COLUMNS = [
       'main_speaker', 'speaker_occupation', 'num_speaker', 'event_type']

preprocessing = ColumnTransformer(transformers=[
    ('ohe', OneHotEncoder(categories='auto', handle_unknown='ignore'), CATEGORICAL_COLUMNS),
    ('scaler', StandardScaler(), NUMERIC_COLUMNS),
    ('tfidf_0', TfidfVectorizer(), TEXT_COLUMNS[0]),
    ('tfidf_1', TfidfVectorizer(), TEXT_COLUMNS[1]),
    ('tfidf_2', TfidfVectorizer(), TEXT_COLUMNS[2]),
    ('tfidf_3', TfidfVectorizer(), TEXT_COLUMNS[3]),
])

model_ridge = Pipeline(
    steps=[
        ('preprocessing', preprocessing),
        ('ridge', Ridge(random_state=RANDOM_SEED))
    ]
)

cv = GridSearchCV(model_ridge, param_grid={}, scoring='neg_mean_absolute_error', cv=TimeSeriesSplit(n_splits=5),
                 return_train_score=True, verbose=3)
cv.fit(X_train, y_train)
```

```python
cv.best_score_
```

This is our new best crossval score

#### published_hour

```python
%%time
NUMERIC_COLUMNS = ['duration', 'languages', 'transcript_len'
                  ]

CATEGORICAL_COLUMNS = ['event_type',
       'main_speaker', 'speaker_occupation', 'num_speaker', 'published_hour']

preprocessing = ColumnTransformer(transformers=[
    ('ohe', OneHotEncoder(categories='auto', handle_unknown='ignore'), CATEGORICAL_COLUMNS),
    ('scaler', StandardScaler(), NUMERIC_COLUMNS),
    ('tfidf_0', TfidfVectorizer(), TEXT_COLUMNS[0]),
    ('tfidf_1', TfidfVectorizer(), TEXT_COLUMNS[1]),
    ('tfidf_2', TfidfVectorizer(), TEXT_COLUMNS[2]),
    ('tfidf_3', TfidfVectorizer(), TEXT_COLUMNS[3]),
])

model_ridge = Pipeline(
    steps=[
        ('preprocessing', preprocessing),
        ('ridge', Ridge(random_state=RANDOM_SEED))
    ]
)

cv = GridSearchCV(model_ridge, param_grid={}, scoring='neg_mean_absolute_error', cv=TimeSeriesSplit(n_splits=5),
                 return_train_score=True, verbose=3)
cv.fit(X_train, y_train)
```

```python
cv.best_score_
```

Not improvement of the best score

#### published_month

```python
%%time
NUMERIC_COLUMNS = ['duration', 'languages', 'transcript_len'
                  ]

CATEGORICAL_COLUMNS = ['event_type',
       'main_speaker', 'speaker_occupation', 'num_speaker', 'published_month']

preprocessing = ColumnTransformer(transformers=[
    ('ohe', OneHotEncoder(categories='auto', handle_unknown='ignore'), CATEGORICAL_COLUMNS),
    ('scaler', StandardScaler(), NUMERIC_COLUMNS),
    ('tfidf_0', TfidfVectorizer(), TEXT_COLUMNS[0]),
    ('tfidf_1', TfidfVectorizer(), TEXT_COLUMNS[1]),
    ('tfidf_2', TfidfVectorizer(), TEXT_COLUMNS[2]),
    ('tfidf_3', TfidfVectorizer(), TEXT_COLUMNS[3]),
])

model_ridge = Pipeline(
    steps=[
        ('preprocessing', preprocessing),
        ('ridge', Ridge(random_state=RANDOM_SEED))
    ]
)

cv = GridSearchCV(model_ridge, param_grid={}, scoring='neg_mean_absolute_error', cv=TimeSeriesSplit(n_splits=5),
                 return_train_score=True, verbose=3)
cv.fit(X_train, y_train)
```

```python
cv.best_score_
```

Not improvement of score

#### published_dayofweek

```python
%%time
NUMERIC_COLUMNS = ['duration', 'languages', 'transcript_len'
                  ]

CATEGORICAL_COLUMNS = ['event_type',
       'main_speaker', 'speaker_occupation', 'num_speaker', 'published_dayofweek']

preprocessing = ColumnTransformer(transformers=[
    ('ohe', OneHotEncoder(categories='auto', handle_unknown='ignore'), CATEGORICAL_COLUMNS),
    ('scaler', StandardScaler(), NUMERIC_COLUMNS),
    ('tfidf_0', TfidfVectorizer(), TEXT_COLUMNS[0]),
    ('tfidf_1', TfidfVectorizer(), TEXT_COLUMNS[1]),
    ('tfidf_2', TfidfVectorizer(), TEXT_COLUMNS[2]),
    ('tfidf_3', TfidfVectorizer(), TEXT_COLUMNS[3]),
])

model_ridge = Pipeline(
    steps=[
        ('preprocessing', preprocessing),
        ('ridge', Ridge(random_state=RANDOM_SEED))
    ]
)

cv = GridSearchCV(model_ridge, param_grid={}, scoring='neg_mean_absolute_error', cv=TimeSeriesSplit(n_splits=5),
                 return_train_score=True, verbose=3)
cv.fit(X_train, y_train)
```

```python
cv.best_score_
```

Not improvement of the score

#### Conclusion on feature engineering

We just found two new useful features:
- event_type instead of event
- transcript_len

Cross validation on Ridge shows score improvement with both of them.

### Part 8. Cross-validation, hyperparameter tuning

We will use features we already found and selected.

```python
%%time

X = df_ted[['description', 'duration', 'event', 'languages',
       'main_speaker', 'num_speaker',
       'speaker_occupation', 'tags', 'title', 'transcript']].copy()
y = df_ted['target'].copy()

X['transcript'] = X['transcript'].fillna('na')
X['transcript_len'] = X['transcript'].str.len()
X['event_type'] = X['event'].apply(get_event_type)
X.drop('event', axis=1, inplace=True)

NUMERIC_COLUMNS = ['duration', 'languages', 'transcript_len']

CATEGORICAL_COLUMNS = ['main_speaker', 'speaker_occupation', 'num_speaker', 'event_type']

# We will convert 'tags' column to string
TEXT_COLUMNS = ['description', 'tags', 'title', 'transcript']

# Convert tags to string
X['tags'] = X['tags'].apply(lambda tags: ' '.join(tags))

# StandardScaler will convert fields to float64 with warning, so we will do it before
for c in NUMERIC_COLUMNS:
    X[c] = X[c].astype(float)

# Convert all text columns to lower case
for c in TEXT_COLUMNS:
    X[c] = X[c].str.lower()

# Factorize categorical_columns (similar to LabelEncoding)
for c in CATEGORICAL_COLUMNS:
    X[c] = X[c].factorize()[0]

X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size = 0.3, shuffle=False)
```

```python
X_train.shape, X_valid.shape, y_train.shape, y_valid.shape
```

```python
preprocessing = ColumnTransformer(transformers=[
    ('ohe', OneHotEncoder(categories='auto', handle_unknown='ignore'), CATEGORICAL_COLUMNS),
    ('scaler', StandardScaler(), NUMERIC_COLUMNS),
    ('tfidf_0', TfidfVectorizer(), TEXT_COLUMNS[0]),
    ('tfidf_1', TfidfVectorizer(), TEXT_COLUMNS[1]),
    ('tfidf_2', TfidfVectorizer(), TEXT_COLUMNS[2]),
    ('tfidf_3', TfidfVectorizer(), TEXT_COLUMNS[3]),
])
```

Let's tune alpha (l1 regularization) for Ridge.

```python
model_ridge = Pipeline(
    steps=[
        ('preprocessing', preprocessing),
        ('ridge', Ridge(random_state=RANDOM_SEED))
    ]
)

params = {

    'ridge__alpha' : np.logspace(-2, 5, num=8)
}

cv = GridSearchCV(model_ridge, param_grid=params, scoring='neg_mean_absolute_error', cv=TimeSeriesSplit(n_splits=5),
                 return_train_score=True, verbose=3)
cv.fit(X_train, y_train)
```

```python
cv.best_score_
```

```python
cv.best_params_
```

```python
def plot_param_tuning(params, param_name, cv, x_scale_log=False):

    plt.plot(params[param_name], cv.cv_results_['mean_train_score'], 'o-', label='train')
    plt.plot(params[param_name], cv.cv_results_['mean_test_score'], 'o-', label='test')

    plt.fill_between(params[param_name],
                     cv.cv_results_['mean_train_score'] - cv.cv_results_['std_train_score'],
                     cv.cv_results_['mean_train_score'] + cv.cv_results_['std_train_score'],
                     alpha=0.2
                    )
    plt.fill_between(params[param_name],
                     cv.cv_results_['mean_test_score'] - cv.cv_results_['std_test_score'],
                     cv.cv_results_['mean_test_score'] + cv.cv_results_['std_test_score'],
                     alpha=0.2
                    )
    if x_scale_log:
        plt.xscale('log')

    plt.legend();
```

```python
plot_param_tuning(params, 'ridge__alpha', cv, x_scale_log=True)
plt.xlabel('alpha')
plt.ylabel('neg_mean_absolute_error')
plt.title('Ridge alpha tuning');
```

It rather difficult to select good alpha value, because of wide range in standard deviation and different sample sizes due to TimeSeriesSplit. But we can consider alpha=10^2 as good guess because here is minimal difference between train and test sample.

```python
model_lgb = Pipeline(
    steps=[
        ('preprocessing', preprocessing),
        ('lgb', LGBMRegressor(random_state=RANDOM_SEED))
    ]
)

params = {
}

cv = GridSearchCV(model_lgb, param_grid=params, scoring='neg_mean_absolute_error', cv=TimeSeriesSplit(n_splits=5),
                 return_train_score=True, verbose=3)
cv.fit(X_train, y_train)
```

```python
cv.best_score_
```

```python
%%time

model_lgb = Pipeline(
    steps=[
        ('preprocessing', preprocessing),
        ('lgb', LGBMRegressor(random_state=RANDOM_SEED))
    ]
)

params = {
    'lgb__max_depth': [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]
}

cv = GridSearchCV(model_lgb, param_grid=params, scoring='neg_mean_absolute_error', cv=TimeSeriesSplit(n_splits=5),
                 return_train_score=True, verbose=3)
cv.fit(X_train, y_train)
```

```python
cv.best_score_
```

```python
cv.best_params_
```

```python
plot_param_tuning(params, 'lgb__max_depth', cv)
plt.xlabel('max_depth')
plt.ylabel('neg_mean_absolute_error')
plt.title('Lgb max_depth tuning');
```

Definitely we can't say something about the best max_depth for lgbm regression. We will tune n_estimators.

```python
model_lgb = Pipeline(
    steps=[
        ('preprocessing', preprocessing),
        ('lgb', LGBMRegressor(random_state=RANDOM_SEED))
    ]
)

params = {
    'lgb__n_estimators': [10,20,30,40,50, 60, 70, 80, 90, 100, 150, 200, 300, 400, 500, 600, 700]
}

cv = GridSearchCV(model_lgb, param_grid=params, scoring='neg_mean_absolute_error', cv=TimeSeriesSplit(n_splits=5),
                 return_train_score=True, verbose=3)
cv.fit(X_train, y_train)
```

```python
cv.best_score_, cv.best_params_
```

```python
plot_param_tuning(params, 'lgb__n_estimators', cv)
plt.xlabel('n_estimators')
plt.ylabel('neg_mean_absolute_error')
plt.title('Lgb n_estimators tuning');
```

```python
model_lgb = Pipeline(
    steps=[
        ('preprocessing', preprocessing),
        ('lgb', LGBMRegressor(random_state=RANDOM_SEED))
    ]
)

params = {
    'lgb__n_estimators': [30],
    'lgb__num_leaves':np.linspace(10,51, num=10, dtype=int)
}

cv = GridSearchCV(model_lgb, param_grid=params, scoring='neg_mean_absolute_error', cv=TimeSeriesSplit(n_splits=5),
                 return_train_score=True, verbose=3)
cv.fit(X_train, y_train)
```

```python
cv.best_score_, cv.best_params_
```

```python
plot_param_tuning(params, 'lgb__num_leaves', cv)
plt.xlabel('num_leaves')
plt.ylabel('neg_mean_absolute_error')
plt.title('Lgb num_leaves tuning');
```

Looks like we haven't any visible success in lgbm tuning so we can use only 'lgb__n_estimators': 30 as parameter.

#### Conclusion

Our params as result of hyperparameter tuning:
- Ridge - alpha: 10 (but 100 looks better because of smaller distance between train and test)
- Lgbm regression - n_estimators: 30

### Part 9. Validation and learning curves

#### Ridge

```python
X_train.shape, y_train.shape
```

```python
%%time

model_ridge = Pipeline(
    steps=[
        ('preprocessing', preprocessing),
        ('ridge', Ridge(random_state=RANDOM_SEED, alpha=100))
    ]
)

train_sizes, train_scores, test_scores = \
    learning_curve(model_ridge, X_train, y_train,
                   cv=TimeSeriesSplit(n_splits=5), scoring='neg_mean_absolute_error', random_state=RANDOM_SEED)
```

```python
def plot_learning_curve(train_sizes, train_scores, test_scores):
    train_scores_mean = np.mean(train_scores, axis=1)
    train_scores_std = np.std(train_scores, axis=1)
    test_scores_mean = np.mean(test_scores, axis=1)
    test_scores_std = np.std(test_scores, axis=1)

    plt.fill_between(train_sizes, train_scores_mean - train_scores_std,
                     train_scores_mean + train_scores_std, alpha=0.1,
                     color="r")
    plt.fill_between(train_sizes, test_scores_mean - test_scores_std,
                     test_scores_mean + test_scores_std, alpha=0.1, color="g")
    plt.plot(train_sizes, train_scores_mean, 'o-', color="r",
             label="Training score")
    plt.plot(train_sizes, test_scores_mean, 'o-', color="g",
             label="Cross-validation score")

    plt.legend(loc="best")
```

```python
plot_learning_curve(train_sizes, train_scores, test_scores)
plt.xlabel('train_sizes')
plt.ylabel('neg_mean_absolute_error')
plt.title('Learning curve Ridge');
```

#### Lgbm regressor

```python
%%time

model_lgb = Pipeline(
    steps=[
        ('preprocessing', preprocessing),
        ('lgb', LGBMRegressor(random_state=RANDOM_SEED, n_estimators=30))
    ]
)

train_sizes, train_scores, test_scores = \
    learning_curve(model_lgb, X_train, y_train,
                   cv=TimeSeriesSplit(n_splits=5), scoring='neg_mean_absolute_error', random_state=RANDOM_SEED)
```

```python
plot_learning_curve(train_sizes, train_scores, test_scores)
plt.xlabel('train_sizes')
plt.ylabel('neg_mean_absolute_error')
plt.title('Learning curve Ridge');
```

LGBRegressor tends towards overfitting, while for Ridge train and validation scores tends to look closer to each other.

### Part 10. Prediction for hold-out set

Let's check our models on hold-out set. Hold-out set was produced from all data and consist of last 30% data sorted by time.

#### Ridge

```python
model_ridge.fit(X_train, y_train)
```

```python
ridge_mae_valid = mean_absolute_error(y_valid, model_ridge.predict(X_valid))
ridge_mae_valid
```

#### Lgbm regressor

```python
model_lgb.fit(X_train, y_train)
```

```python
lgb_mae_valid = mean_absolute_error(y_valid, model_lgb.predict(X_valid))
lgb_mae_valid
```

### Part 11. Model selection

Let's recheck cross_val_score for models.

```python
%%time

ridge_cv_score = cross_val_score(model_ridge, X_train, y_train, scoring='neg_mean_absolute_error',
                                 cv=TimeSeriesSplit(n_splits=5))
lgb_cv_score = cross_val_score(model_lgb, X_train, y_train, scoring='neg_mean_absolute_error',
                                 cv=TimeSeriesSplit(n_splits=5))
```

```python
ridge_cv_score.mean(), lgb_cv_score.mean()
```

```python
pd.DataFrame(index=['Ridge', 'LGBRegressor'], data = [
    [ridge_mae_valid, -ridge_cv_score.mean()],
    [lgb_mae_valid, -lgb_cv_score.mean()],
    ], columns = ['valid', 'cv_score'])
```

We've got some mess with result. No model looks like the clean winner. But on the learning plots Ridge looks more sustainable. So probably we should choose Ridge as basic model for futher research.

### Part 12. Conclusions

We've made some initial research on TED Talks dataset. 'Views' variable wasn't normally distributed so we used logarithm of it.

After parameter tuning and model selection both Ridge and LGBM regressor was able to get about 0.48 MAE on cross-validation. Despite of fact that on hold=out set LGBM outperforms Ridge, Ridge looks better on cross-validation.

The model can be useful for research on predicting TED talk popularity measured in views.

Ways to impove and futher develope the model:
- Normalization of text
- Turning Tf-IDF ngramm_range for text fields
- Usage of PCA before LGBM
- Try to get more data (more new data should be available)
- More precisely model tuning
- Research on how model will perform without 'language' variable

```python

```


# Regression in Python

***
This is a very quick run-through of some basic statistical concepts, adapted from [Lab 4 in Harvard's CS109](https://github.com/cs109/2015lab4) course. Please feel free to try the original lab if you're feeling ambitious :-) The CS109 git repository also has the solutions if you're stuck.

* Linear Regression Models
* Prediction using linear regression

Linear regression is used to model and predict continuous outcomes with normal random errors. There are nearly an infinite number of different types of regression models and each regression model is typically defined by the distribution of the prediction errors (called "residuals") of the type of data. Logistic regression is used to model binary outcomes whereas Poisson regression is used to predict counts. In this exercise, we'll see some examples of linear regression as well as Train-test splits.

The packages we'll cover are: `statsmodels`, `seaborn`, and `scikit-learn`. While we don't explicitly teach `statsmodels` and `seaborn` in the Springboard workshop, those are great libraries to know.
***

***

```python
# special IPython command to prepare the notebook for matplotlib and other libraries
%matplotlib inline

import numpy as np
import pandas as pd
import scipy.stats as stats
import matplotlib.pyplot as plt
import sklearn

import seaborn as sns

# special matplotlib argument for improved plots
from matplotlib import rcParams
sns.set_style("whitegrid")
sns.set_context("poster")

```

***
# Part 1: Introduction to Linear Regression
### Purpose of linear regression
***

 Given a dataset containing predictor variables $X$ and outcome/response variable $Y$, linear regression can be used to:

 Build a predictive model to predict future values of $\hat{Y}$, using new data $X^*$ where $Y$ is unknown.

 Model the strength of the relationship between each independent variable $X_i$ and $Y$

 Many times, only a subset of independent variables $X_i$ will have a linear relationship with $Y$

 Need to figure out which $X_i$ contributes most information to predict $Y$

It is in many cases, the first pass prediction algorithm for continuous outcomes.

### A Brief Mathematical Recap
***

[Linear Regression](http://en.wikipedia.org/wiki/Linear_regression) is a method to model the relationship between a set of independent variables $X$ (also knowns as explanatory variables, features, predictors) and a dependent variable $Y$.  This method assumes the relationship between each predictor $X$ is **linearly** related to the dependent variable $Y$. The most basic linear regression model contains one independent variable $X$, we'll call this the simple model.

$$ Y = \beta_0 + \beta_1 X + \epsilon$$

where $\epsilon$ is considered as an unobservable random variable that adds noise to the linear relationship. In linear regression, $\epsilon$ is assumed to be normally distributed with a mean of 0. In other words, what this means is that on average, if we know $Y$, a roughly equal number of predictions $\hat{Y}$ will be above $Y$ and others will be below $Y$. That is, on average, the error is zero. The residuals, $\epsilon$ are also assumed to be "i.i.d.": independently and identically distributed. Independence means that the residuals are not correlated -- the residual from one prediction has no effect on the residual from another prediction. Correlated errors are common in time series analysis and spatial analyses.

* $\beta_0$ is the intercept of the linear model and represents the average of $Y$ when all independent variables $X$ are set to 0.

* $\beta_1$ is the slope of the line associated with the regression model and represents the average effect of a one-unit increase in $X$ on $Y$.

* Back to the simple model. The model in linear regression is the *conditional mean* of $Y$ given the values in $X$ is expressed a linear function.

$$ y = f(x) = E(Y | X = x)$$

http://www.learner.org/courses/againstallodds/about/glossary.html

* The goal is to estimate the coefficients (e.g. $\beta_0$ and $\beta_1$). We represent the estimates of the coefficients with a "hat" on top of the letter.

$$ \hat{\beta}_0, \hat{\beta}_1 $$

* Once we estimate the coefficients $\hat{\beta}_0$ and $\hat{\beta}_1$, we can use these to predict new values of $Y$ given new data $X$.

$$\hat{y} = \hat{\beta}_0 + \hat{\beta}_1 x_1$$

* Multiple linear regression is when you have more than one independent variable and the estimation involves matrices
    * $X_1$, $X_2$, $X_3$, $\ldots$

* How do you estimate the coefficients?
    * There are many ways to fit a linear regression model
    * The method called **least squares** is the most common methods
    * We will discuss least squares

$$ Y = \beta_0 + \beta_1 X_1 + \ldots + \beta_p X_p + \epsilon$$

### Estimating $\hat\beta$: Least squares
***
[Least squares](http://en.wikipedia.org/wiki/Least_squares) is a method that can estimate the coefficients of a linear model by minimizing the squared residuals:

$$ \mathscr{L} = \sum_{i=1}^N \epsilon_i = \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2  = \sum_{i=1}^N \left(y_i - \left(\beta_0 + \beta_1 x_i\right)\right)^2 $$

where $N$ is the number of observations and $\epsilon$ represents a residual or error, ACTUAL - PREDICTED.

#### Estimating the intercept $\hat{\beta_0}$ for the simple linear model

We want to minimize the squared residuals and solve for $\hat{\beta_0}$ so we take the partial derivative of $\mathscr{L}$ with respect to $\hat{\beta_0}$

$
\begin{align}
\frac{\partial \mathscr{L}}{\partial \hat{\beta_0}} &= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \epsilon^2 \\
&= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2 \\
&= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right)^2 \\
&= -2 \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right) \hspace{25mm} \mbox{(by chain rule)} \\
&= -2 \sum_{i=1}^N y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i \\
&= -2 \left[ \left( \sum_{i=1}^N y_i \right) - n \hat{\beta_0} - \hat{\beta}_1 \left( \sum_{i=1}^N x_i
\right) \right] \\
& 2 \left[ n \hat{\beta}_0 + \hat{\beta}_1 \sum_{i=1}^N x_i - \sum_{i=1}^N y_i \right] = 0 \hspace{20mm} \mbox{(Set equal to 0 and solve for $\hat{\beta}_0$)} \\
& n \hat{\beta}_0 + \hat{\beta}_1 \sum_{i=1}^N x_i - \sum{i=1}^N y_i = 0 \\
& n \hat{\beta}_0 = \sum_{i=1}^N y_i - \hat{\beta}_1 \sum_{i=1}^N x_i \\
& \hat{\beta}_0 = \frac{\sum_{i=1}^N y_i - \hat{\beta}_1 \sum_{i=1}^N x_i}{n} \\
& \hat{\beta}_0 = \frac{\sum_{i=1}^N y_i}{n} - \hat{\beta}_1 \frac{\sum_{i=1}^N x_i}{n} \\
& \boxed{\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}}
\end{align}
$

Using this new information, we can compute the estimate for $\hat{\beta}_1$ by taking the partial derivative of $\mathscr{L}$ with respect to $\hat{\beta}_1$.

$
\begin{align}
\frac{\partial \mathscr{L}}{\partial \hat{\beta_1}} &= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \epsilon^2 \\
&= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2 \\
&= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right)^2 \\
&= 2 \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right) \left( -x_i \right) \hspace{25mm}\mbox{(by chain rule)} \\
&= -2 \sum_{i=1}^N x_i \left( y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i \right) \\
&= -2 \sum_{i=1}^N x_i y_i - \hat{\beta}_0 x_i - \hat{\beta}_1 x_i^2 \\
&= -2 \sum_{i=1}^N x_i y_i - \left( \bar{y} - \hat{\beta}_1 \bar{x} \right) x_i - \hat{\beta}_1 x_i^2 \\
&= -2 \sum_{i=1}^N x_i y_i - \bar{y}x_i + \hat{\beta}_1\bar{x}x_i - \hat{\beta}_1 x_i^2 \\
&= -2 \left[ \sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i + \hat{\beta}_1\bar{x} - \hat{\beta}_1 x_i^2 \right] \\
&= -2 \left[ \hat{\beta}_1 \left\{ \bar{x} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i^2 \right\} + \left\{ \sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i \right\}\right] \\
& 2 \left[ \hat{\beta}_1 \left\{ \sum_{i=1}^N x_i^2 - \bar{x} \sum_{i=1}^N x_i \right\} + \left\{ \bar{y} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i y_i \right\} \right] = 0 \\
& \hat{\beta}_1 = \frac{-\left( \bar{y} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i y_i \right)}{\sum_{i=1}^N x_i^2 - \bar{x}\sum_{i=1}^N x_i} \\
&= \frac{\sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i}{\sum_{i=1}^N x_i^2 - \bar{x} \sum_{i=1}^N x_i} \\
& \boxed{\hat{\beta}_1 = \frac{\sum_{i=1}^N x_i y_i - \bar{x}\bar{y}n}{\sum_{i=1}^N x_i^2 - n \bar{x}^2}}
\end{align}
$

The solution can be written in compact matrix notation as

$$\hat\beta =  (X^T X)^{-1}X^T Y$$

We wanted to show you this in case you remember linear algebra, in order for this solution to exist we need $X^T X$ to be invertible. Of course this requires a few extra assumptions, $X$ must be full rank so that $X^T X$ is invertible, etc. Basically, $X^T X$ is full rank if all rows and columns are linearly independent. This has a loose relationship to variables and observations being independent respective. **This is important for us because this means that having redundant features in our regression models will lead to poorly fitting (and unstable) models.** We'll see an implementation of this in the extra linear regression example.

***
# Part 2: Exploratory Data Analysis for Linear Relationships

The [Boston Housing data set](https://archive.ics.uci.edu/ml/datasets/Housing) contains information about the housing values in suburbs of Boston.  This dataset was originally taken from the StatLib library which is maintained at Carnegie Mellon University and is now available on the UCI Machine Learning Repository.

## Load the Boston Housing data set from `sklearn`
***

This data set is available in the [sklearn](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html#sklearn.datasets.load_boston) python module which is how we will access it today.

```python
from sklearn.datasets import load_boston
import pandas as pd

boston = load_boston()
```

```python
boston.keys()
```

    dict_keys(['data', 'target', 'feature_names', 'DESCR'])

```python
boston.data.shape
```

    (506, 13)

```python
# Print column names
print(boston.feature_names)
```

    ['CRIM' 'ZN' 'INDUS' 'CHAS' 'NOX' 'RM' 'AGE' 'DIS' 'RAD' 'TAX' 'PTRATIO'
     'B' 'LSTAT']

```python
# Print description of Boston housing data set
print(boston.DESCR)
```

    Boston House Prices dataset
    ===========================

    Notes
    ------
    Data Set Characteristics:

        :Number of Instances: 506

        :Number of Attributes: 13 numeric/categorical predictive

        :Median Value (attribute 14) is usually the target

        :Attribute Information (in order):
            - CRIM     per capita crime rate by town
            - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.
            - INDUS    proportion of non-retail business acres per town
            - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
            - NOX      nitric oxides concentration (parts per 10 million)
            - RM       average number of rooms per dwelling
            - AGE      proportion of owner-occupied units built prior to 1940
            - DIS      weighted distances to five Boston employment centres
            - RAD      index of accessibility to radial highways
            - TAX      full-value property-tax rate per $10,000
            - PTRATIO  pupil-teacher ratio by town
            - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town
            - LSTAT    % lower status of the population
            - MEDV     Median value of owner-occupied homes in $1000's

        :Missing Attribute Values: None

        :Creator: Harrison, D. and Rubinfeld, D.L.

    This is a copy of UCI ML housing dataset.
    http://archive.ics.uci.edu/ml/datasets/Housing

    This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.

    The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic
    prices and the demand for clean air', J. Environ. Economics & Management,
    vol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics
    ...', Wiley, 1980.   N.B. Various transformations are used in the table on
    pages 244-261 of the latter.

    The Boston house-price data has been used in many machine learning papers that address regression
    problems.

    **References**

       - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.
       - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.
       - many more! (see http://archive.ics.uci.edu/ml/datasets/Housing)

Now let's explore the data set itself.

```python
bos = pd.DataFrame(boston.data)
bos.head()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33

There are no column names in the DataFrame. Let's add those.

```python
bos.columns = boston.feature_names
bos.head()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33

Now we have a pandas DataFrame called `bos` containing all the data we want to use to predict Boston Housing prices.  Let's create a variable called `PRICE` which will contain the prices. This information is contained in the `target` data.

```python
print(boston.target.shape)
```

    (506,)

```python
bos['PRICE'] = boston.target
bos.head()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      PRICE

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98
      24.0

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14
      21.6

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03
      34.7

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94
      33.4

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33
      36.2

## EDA and Summary Statistics
***

Let's explore this data set.  First we use `describe()` to get basic summary statistics for each of the columns.

```python
bos.describe()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      PRICE

      count
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000

      mean
      3.593761
      11.363636
      11.136779
      0.069170
      0.554695
      6.284634
      68.574901
      3.795043
      9.549407
      408.237154
      18.455534
      356.674032
      12.653063
      22.532806

      std
      8.596783
      23.322453
      6.860353
      0.253994
      0.115878
      0.702617
      28.148861
      2.105710
      8.707259
      168.537116
      2.164946
      91.294864
      7.141062
      9.197104

      min
      0.006320
      0.000000
      0.460000
      0.000000
      0.385000
      3.561000
      2.900000
      1.129600
      1.000000
      187.000000
      12.600000
      0.320000
      1.730000
      5.000000

      25%
      0.082045
      0.000000
      5.190000
      0.000000
      0.449000
      5.885500
      45.025000
      2.100175
      4.000000
      279.000000
      17.400000
      375.377500
      6.950000
      17.025000

      50%
      0.256510
      0.000000
      9.690000
      0.000000
      0.538000
      6.208500
      77.500000
      3.207450
      5.000000
      330.000000
      19.050000
      391.440000
      11.360000
      21.200000

      75%
      3.647423
      12.500000
      18.100000
      0.000000
      0.624000
      6.623500
      94.075000
      5.188425
      24.000000
      666.000000
      20.200000
      396.225000
      16.955000
      25.000000

      max
      88.976200
      100.000000
      27.740000
      1.000000
      0.871000
      8.780000
      100.000000
      12.126500
      24.000000
      711.000000
      22.000000
      396.900000
      37.970000
      50.000000

### Scatterplots
***

Let's look at some scatter plots for three variables: 'CRIM' (per capita crime rate), 'RM' (number of rooms) and 'PTRATIO' (pupil-to-teacher ratio in schools).

```python
plt.scatter(bos.CRIM, bos.PRICE)
plt.xlabel("Per capita crime rate by town (CRIM)")
plt.ylabel("Housing Price")
plt.title("Relationship between CRIM and Price")
```

Part 2 Checkup Exercise Set I

Exercise: What kind of relationship do you see? e.g. positive, negative?  linear? non-linear? Is there anything else strange or interesting about the data? What about outliers?

Exercise: Create scatter plots between *RM* and *PRICE*, and *PTRATIO* and *PRICE*. Label your axes appropriately using human readable labels. Tell a story about what you see.

Exercise: What are some other numeric variables of interest? Why do you think they are interesting? Plot scatterplots with these variables and *PRICE* (house price) and tell a story about what you see.

```python
# your turn: describe relationship
slope = np.polyfit(bos.CRIM, bos.PRICE, 1)[0]
print('The slope of the line of best fit is {0:0.3f}'.format(slope))
```

    The slope of the line of best fit is -0.413

The slope of the linear fit to this scatter plot is negative, which indicates that the relationship is negative. It does not appear to be linear in nature though, since the drop off is very steep near zero and levels out in the upper range of the x axis, which approximates something more quadratic.

There also seems to be many data points closely bunched together near the zero of the x axis, which possibly indicates that there is some additional feature or better pair of features to compare to get a better view of the relationship between housing price and crime per capita.

```python
# your turn: scatter plot between *RM* and *PRICE*
plt.scatter(bos.RM, bos.PRICE)
plt.xlabel("Average Number of Rooms  (RM)")
plt.ylabel("Housing Price")
plt.title("Relationship between RM and Price")

plt.show()
```

```python
# your turn: scatter plot between *PTRATIO* and *PRICE*
plt.scatter(bos.PTRATIO, bos.PRICE)
plt.xlabel("Pupil-Teacher Ratio (PTRATIO)")
plt.ylabel("Housing Price")
plt.title("Relationship between PTRATIO and Price")

plt.show()
```

```python
# your turn: create some other scatter plots
plt.scatter(bos.AGE, bos.PRICE)
plt.xlabel("Proportion of units built before 1940(AGE)")
plt.ylabel("Housing Price")
plt.title("Relationship between AGE and Price")

plt.show()

plt.scatter(bos.NOX, bos.PRICE)
plt.xlabel("Nitric Oxides Concentration (NOX)")
plt.ylabel("Housing Price")
plt.title("Relationship between NOX and Price")

plt.show()
```

### Scatterplots using Seaborn
***

[Seaborn](https://stanford.edu/~mwaskom/software/seaborn/) is a cool Python plotting library built on top of matplotlib. It provides convenient syntax and shortcuts for many common types of plots, along with better-looking defaults.

We can also use [seaborn regplot](https://stanford.edu/~mwaskom/software/seaborn/tutorial/regression.html#functions-to-draw-linear-regression-models) for the scatterplot above. This provides automatic linear regression fits (useful for data exploration later on). Here's one example below.

```python
sns.regplot(y="PRICE", x="RM", data=bos, fit_reg = True)
```

### Histograms
***

```python
plt.hist(np.log(bos.CRIM))
plt.title("CRIM")
plt.xlabel("Crime rate per capita")
plt.ylabel("Frequencey")
plt.show()
```

Part 2 Checkup Exercise Set II

Exercise: In the above histogram, we took the logarithm of the crime rate per capita. Repeat this histogram without taking the log. What was the purpose of taking the log? What do we gain by making this transformation? What do you now notice about this variable that is not obvious without making the transformation?

Exercise: Plot the histogram for *RM* and *PTRATIO* against each other, along with the two variables you picked in the previous section. We are looking for correlations in predictors here.

```python
#your turn
plt.hist(bos.CRIM)
plt.title("CRIM")
plt.xlabel("Crime rate per capita")
plt.ylabel("Frequencey")
plt.show()
```

Using log transformation in the first histogram produced a distribution that was closer to normal (perhaps bimodal in this case), compared to the non-transformed histogram distribution.

```python
plt.subplots(figsize=(12, 6))
plt.subplot(1, 2, 1)
plt.hist(bos.RM)
plt.title("RM")
plt.xlabel("Average Number of Rooms")
plt.ylabel("Frequency")

plt.subplot(1, 2, 2)
plt.hist(bos.PTRATIO)
plt.title("PTRATIO")
plt.xlabel("Pupil-Teacher Ratio")
plt.ylabel("Frequency")

plt.show()
```

```python
plt.subplots(figsize=(12, 6))
plt.subplot(1, 2, 1)
plt.hist(bos.AGE)
plt.title("AGE")
plt.xlabel("Proportion of units built before 1940")
plt.ylabel("Frequency")

plt.subplot(1, 2, 2)
plt.hist(bos.NOX)
plt.title("NOX")
plt.xlabel("Nitric Oxides Concentration")
plt.ylabel("Frequency")

plt.show()
```

## Part 3: Linear Regression with Boston Housing Data Example
***

Here,

$Y$ = boston housing prices (called "target" data in python, and referred to as the dependent variable or response variable)

and

$X$ = all the other features (or independent variables, predictors or explanatory variables)

which we will use to fit a linear regression model and predict Boston housing prices. We will use the least-squares method to estimate the coefficients.

We'll use two ways of fitting a linear regression. We recommend the first but the second is also powerful in its features.

### Fitting Linear Regression using `statsmodels`
***
[Statsmodels](http://statsmodels.sourceforge.net/) is a great Python library for a lot of basic and inferential statistics. It also provides basic regression functions using an R-like syntax, so it's commonly used by statisticians. While we don't cover statsmodels officially in the Data Science Intensive workshop, it's a good library to have in your toolbox. Here's a quick example of what you could do with it. The version of least-squares we will use in statsmodels is called *ordinary least-squares (OLS)*. There are many other versions of least-squares such as [partial least squares (PLS)](https://en.wikipedia.org/wiki/Partial_least_squares_regression) and [weighted least squares (WLS)](https://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares).

```python
# Import regression modules
import statsmodels.api as sm
from statsmodels.formula.api import ols
```

    C:\Users\301339\AppData\Local\Continuum\anaconda3\lib\site-packages\statsmodels\compat\pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.
      from pandas.core import datetools

```python
# statsmodels works nicely with pandas dataframes
# The thing inside the "quotes" is called a formula, a bit on that below
m = ols('PRICE ~ RM',bos).fit()
print(m.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.484
    Model:                            OLS   Adj. R-squared:                  0.483
    Method:                 Least Squares   F-statistic:                     471.8
    Date:                Fri, 05 Jan 2018   Prob (F-statistic):           2.49e-74
    Time:                        17:15:21   Log-Likelihood:                -1673.1
    No. Observations:                 506   AIC:                             3350.
    Df Residuals:                     504   BIC:                             3359.
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept    -34.6706      2.650    -13.084      0.000     -39.877     -29.465
    RM             9.1021      0.419     21.722      0.000       8.279       9.925
    ==============================================================================
    Omnibus:                      102.585   Durbin-Watson:                   0.684
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              612.449
    Skew:                           0.726   Prob(JB):                    1.02e-133
    Kurtosis:                       8.190   Cond. No.                         58.4
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

#### Interpreting coefficients

There is a ton of information in this output. But we'll concentrate on the coefficient table (middle table). We can interpret the `RM` coefficient (9.1021) by first noticing that the p-value (under `P>|t|`) is so small, basically zero. This means that the number of rooms, `RM`, is a statisticall significant predictor of `PRICE`. The regression coefficient for `RM` of 9.1021 means that *on average, each additional room is associated with an increase of $\$9,100$ in house price net of the other variables*. The confidence interval gives us a range of plausible values for this average change, about ($\$8,279, \$9,925$), definitely not chump change.

In general, the $\hat{\beta_i}, i > 0$ can be interpreted as the following: "A one unit increase in $x_i$ is associated with, on average, a $\hat{\beta_i}$ increase/decrease in $y$ net of all other variables."

On the other hand, the interpretation for the intercept, $\hat{\beta}_0$ is the average of $y$ given that all of the independent variables $x_i$ are 0.

####  `statsmodels` formulas
***
This formula notation will seem familiar to `R` users, but will take some getting used to for people coming from other languages or are new to statistics.

The formula gives instruction for a general structure for a regression call. For `statsmodels` (`ols` or `logit`) calls you need to have a Pandas dataframe with column names that you will add to your formula. In the below example you need a pandas data frame that includes the columns named (`Outcome`, `X1`,`X2`, ...), but you don't need to build a new dataframe for every regression. Use the same dataframe with all these things in it. The structure is very simple:

`Outcome ~ X1`

But of course we want to to be able to handle more complex models, for example multiple regression is doone like this:

`Outcome ~ X1 + X2 + X3`

In general, a formula for an OLS multiple linear regression is

`Y ~ X1 + X2 + ... + Xp`

This is the very basic structure but it should be enough to get you through the homework. Things can get much more complex. You can force statsmodels to treat variables as categorical with the `C()` function, call numpy functions to transform data such as `np.log` for extremely-skewed data, or fit a model without an intercept by including `- 1` in the formula. For a quick run-down of further uses see the `statsmodels` [help page](http://statsmodels.sourceforge.net/devel/example_formulas.html).

Let's see how our model actually fit our data. We can see below that there is a ceiling effect, we should probably look into that. Also, for large values of $Y$ we get underpredictions, most predictions are below the 45-degree gridlines.

Part 3 Checkup Exercise Set I

Exercise: Create a scatterplot between the predicted prices, available in `m.fittedvalues` (where `m` is the fitted model) and the original prices. How does the plot look? Do you notice anything interesting or weird in the plot? Comment on what you see.

```python
# your turn
plt.scatter(m.fittedvalues, bos.PRICE)
plt.plot([0, 50], [0, 50], 'k-', lw=4)
plt.xlabel("Predicted Housing Price")
plt.ylabel("True Housing Price")
plt.title("Relationship between Predicted and True Housing Prices")

plt.show()
```

The model seems to do adequately well in the midrange, but we can see that in the lower range of prices the model tends to overpredict, while in the high price ranges the model returns undervalued prices.

### Fitting Linear Regression using `sklearn`

```python
from sklearn.linear_model import LinearRegression
X = bos.drop('PRICE', axis = 1)

# This creates a LinearRegression object
lm = LinearRegression()
lm
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)

#### What can you do with a LinearRegression object?
***
Check out the scikit-learn [docs here](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html). We have listed the main functions here. Most machine learning models in scikit-learn follow this same API of fitting a model with `fit`, making predictions with `predict` and the appropriate scoring function `score` for each model.

Main functions | Description
--- | ---
`lm.fit()` | Fit a linear model
`lm.predit()` | Predict Y using the linear model with estimated coefficients
`lm.score()` | Returns the coefficient of determination (R^2). *A measure of how well observed outcomes are replicated by the model, as the proportion of total variation of outcomes explained by the model*

#### What output can you get?

```python
# Look inside lm object
# lm.<tab>
```

Output | Description
--- | ---
`lm.coef_` | Estimated coefficients
`lm.intercept_` | Estimated intercept

### Fit a linear model
***

The `lm.fit()` function estimates the coefficients the linear regression using least squares.

```python
# Use all 13 predictors to fit linear regression model
lm.fit(X, bos.PRICE)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)

Part 3 Checkup Exercise Set II

Exercise: How would you change the model to not fit an intercept term? Would you recommend not having an intercept? Why or why not? For more information on why to include or exclude an intercept, look [here](https://online.stat.psu.edu/~ajw13/stat501/SpecialTopics/Reg_thru_origin.pdf).

Exercise: One of the assumptions of the linear model is that the residuals must be i.i.d. (independently and identically distributed). To satisfy this, is it enough that the residuals are normally distributed? Explain your answer.

Exercise: True or false. To use linear regression, $Y$ must be normally distributed. Explain your answer.

```python
# your turn
```

In order to fit a model without an intercept term I would simply change the linear regression option 'fit_intercept' to 'False'. For this dataset it may not be useful to fit an intercept, since there wouldn't reasonably be any housing with a 0 price point and points nearing the zero will probably level off or have outlier behavior.

In order for residuals be i.i.d. it is not enough that residuals are normally distributed. In addition, there cannot be correlations between residuals, hence independence.

Y is not required to be normally distributed in order to use linear regression. Even if the distribution of Y is skewed, a line can be fit through areas of more dense and less densely populated regions of the feature space. However, a normal distribution of Y aids in the fitting of the model and the weight of each point on the regression line.

### Estimated intercept and coefficients

Let's look at the estimated coefficients from the linear model using `1m.intercept_` and `lm.coef_`.

After we have fit our linear regression model using the least squares method, we want to see what are the estimates of our coefficients $\beta_0$, $\beta_1$, ..., $\beta_{13}$:

$$ \hat{\beta}_0, \hat{\beta}_1, \ldots, \hat{\beta}_{13} $$

```python
print('Estimated intercept coefficient: {}'.format(lm.intercept_))
```

    Estimated intercept coefficient: 36.491103280363404

```python
print('Number of coefficients: {}'.format(len(lm.coef_)))
```

    Number of coefficients: 13

```python
# The coefficients
pd.DataFrame({'features': X.columns, 'estimatedCoefficients': lm.coef_})[['features', 'estimatedCoefficients']]
```

      features
      estimatedCoefficients

      0
      CRIM
      -0.107171

      1
      ZN
      0.046395

      2
      INDUS
      0.020860

      3
      CHAS
      2.688561

      4
      NOX
      -17.795759

      5
      RM
      3.804752

      6
      AGE
      0.000751

      7
      DIS
      -1.475759

      8
      RAD
      0.305655

      9
      TAX
      -0.012329

      10
      PTRATIO
      -0.953464

      11
      B
      0.009393

      12
      LSTAT
      -0.525467

### Predict Prices

We can calculate the predicted prices ($\hat{Y}_i$) using `lm.predict`.

$$ \hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_1 + \ldots \hat{\beta}_{13} X_{13} $$

```python
# first five predicted prices
lm.predict(X)[0:5]
```

    array([ 30.00821269,  25.0298606 ,  30.5702317 ,  28.60814055,  27.94288232])

Part 3 Checkup Exercise Set III

Exercise: Histogram: Plot a histogram of all the predicted prices. Write a story about what you see. Describe the shape, center and spread of the distribution. Are there any outliers? What might be the reason for them? Should we do anything special with them?

Exercise: Scatterplot: Let's plot the true prices compared to the predicted prices to see they disagree (we did this with `statsmodels` before).

Exercise: We have looked at fitting a linear model in both `statsmodels` and `scikit-learn`. What are the advantages and disadvantages of each based on your exploration? Based on the information provided by both packages, what advantage does `statsmodels` provide?

```python
# your turn
preds = lm.predict(X)
plt.hist(preds)
plt.xlabel('Predicted House Prices')
plt.ylabel('Frequency')
plt.title('Distribution of Predicted Housing Prices')

plt.show()
```

```python
plt.scatter(preds, bos.PRICE)
plt.plot([0, 50], [0, 50], 'k-', lw=4)
plt.xlabel('Predicted House Prices')
plt.ylabel('True House Prices')
plt.title('Predicted vs. True Housing Prices')

plt.show()
```

The statsmodel provided more information and metrics on the fit and prediction of data upfront, though the sklearn model seems more user-friendly and provided a seeminlgly more accurate prediction at a glance.

### Evaluating the Model: Sum-of-Squares

The partitioning of the sum-of-squares shows the variance in the predictions explained by the model and the variance that is attributed to error.

$$TSS = ESS + RSS$$

#### Residual Sum-of-Squares (aka $RSS$)

The residual sum-of-squares is one of the basic ways of quantifying how much error exists in the fitted model. We will revisit this in a bit.

$$ RSS = \sum_{i=1}^N r_i^2 = \sum_{i=1}^N \left(y_i - \left(\beta_0 + \beta_1 x_i\right)\right)^2 $$

```python
print(np.sum((bos.PRICE - lm.predict(X)) ** 2))
```

    11080.276284149868

#### Explained Sum-of-Squares (aka $ESS$)

The explained sum-of-squares measures the variance explained by the regression model.

$$ESS = \sum_{i=1}^N \left( \hat{y}_i - \bar{y} \right)^2 = \sum_{i=1}^N \left( \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) - \bar{y} \right)^2$$

```python
print(np.sum(lm.predict(X) - np.mean(bos.PRICE)) ** 2)
```

    8.69056631064e-23

### Evaluating the Model: The Coefficient of Determination ($R^2$)

The coefficient of determination, $R^2$, tells us the percentage of the variance in the response variable $Y$ that can be explained by the linear regression model.

$$ R^2 = \frac{ESS}{TSS} $$

The $R^2$ value is one of the most common metrics that people use in describing the quality of a model, but it is important to note that *$R^2$ increases artificially as a side-effect of increasing the number of independent variables.* While $R^2$ is reported in almost all statistical packages, another metric called the *adjusted $R^2$* is also provided as it takes into account the number of variables in the model, and can sometimes even be used for non-linear regression models!

$$R_{adj}^2 = 1 - \left( 1 - R^2 \right) \frac{N - 1}{N - K - 1} = R^2 - \left( 1 - R^2 \right) \frac{K}{N - K - 1} = 1 - \frac{\frac{RSS}{DF_R}}{\frac{TSS}{DF_T}}$$

where $N$ is the number of observations, $K$ is the number of variables, $DF_R = N - K - 1$ is the degrees of freedom associated with the residual error and $DF_T = N - 1$ is the degrees of the freedom of the total error.

### Evaluating the Model: Mean Squared Error and the $F$-Statistic
***
The mean squared errors are just the *averages* of the sum-of-squares errors over their respective degrees of freedom.

$$MSE = \frac{ESS}{K}$$
$$MSR = \frac{RSS}{N-K-1}$$

**Remember: ** Notation may vary across resources particularly the use of *R* and *E* in *RSS/ESS* and *MSR/MSE*. In some resources, E = explained and R = residual. In other resources, E = error and R = regression (explained). **This is a very important distinction that requires looking at the formula to determine which naming scheme is being used.**

Given the MSR and MSE, we can now determine whether or not the entire model we just fit is even statistically significant. We use an $F$-test for this. The null hypothesis is that all of the $\beta$ coefficients are zero, that is, none of them have any effect on $Y$. The alternative is that *at least one* $\beta$ coefficient is nonzero, but it doesn't tell us which one in a multiple regression:

$$H_0: \beta_i = 0, \mbox{for all $i$} \\
H_A: \beta_i > 0, \mbox{for some $i$}$$

$$F = \frac{MSR}{MSE} = \left( \frac{R^2}{1 - R^2} \right) \left( \frac{N - K - 1}{K} \right)$$

Once we compute the $F$-statistic, we can use the $F$-distribution with $N-K$ and $K-1$ degrees of degrees of freedom to get a p-value.

**Warning!** The $F$-statistic mentioned in this section is NOT the same as the F1-measure or F1-value discused in Unit 7.

Part 3 Checkup Exercise Set IV

Let's look at the relationship between `PTRATIO` and housing price.

Exercise: Try fitting a linear regression model using only the 'PTRATIO' (pupil-teacher ratio by town) and interpret the intercept and the coefficients.

Exercise: Calculate (or extract) the $R^2$ value. What does it tell you?

Exercise: Compute the $F$-statistic. What does it tell you?

Exercise: Take a close look at the $F$-statistic and the $t$-statistic for the regression coefficient. What relationship do you notice? Note that this relationship only applies in *simple* linear regression models.

```python
ptratio_m = ols('PRICE ~ PTRATIO',bos).fit()
print(ptratio_m.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.258
    Model:                            OLS   Adj. R-squared:                  0.256
    Method:                 Least Squares   F-statistic:                     175.1
    Date:                Fri, 05 Jan 2018   Prob (F-statistic):           1.61e-34
    Time:                        18:08:41   Log-Likelihood:                -1764.8
    No. Observations:                 506   AIC:                             3534.
    Df Residuals:                     504   BIC:                             3542.
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept     62.3446      3.029     20.581      0.000      56.393      68.296
    PTRATIO       -2.1572      0.163    -13.233      0.000      -2.477      -1.837
    ==============================================================================
    Omnibus:                       92.924   Durbin-Watson:                   0.725
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              191.444
    Skew:                           1.001   Prob(JB):                     2.68e-42
    Kurtosis:                       5.252   Cond. No.                         160.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

The coefficient here explains that PTRATIO is negatively correlated with housing prices, which means that as the pupil-teacher ratio of a town increases, the housing prices are generally decreasing. The intercept suggests a base "starting point" or about $62k before taking into account any information on the pupil-teacher ratio.

The R^2 value here suggests that a model of housing prices based only on the PTRATIO predictor explains about 26% of the variance in the data.

The F-statistic here is ~175, and should be the same as the t statistic because F statistic is a calculated combination metric of individual t statistics for each of the predictor features used.

Part 3 Checkup Exercise Set V

Fit a linear regression model using three independent variables

 'CRIM' (per capita crime rate by town)

 'RM' (average number of rooms per dwelling)

 'PTRATIO' (pupil-teacher ratio by town)

Exercise: Compute or extract the $F$-statistic. What does it tell you about the model?

Exercise: Compute or extract the $R^2$ statistic. What does it tell you about the model?

Exercise: Which variables in the model are significant in predicting house price? Write a story that interprets the coefficients.

```python
# your turn
multi_m = ols('PRICE ~ CRIM + RM + PTRATIO',bos).fit()
print(multi_m.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.593
    Model:                            OLS   Adj. R-squared:                  0.591
    Method:                 Least Squares   F-statistic:                     244.2
    Date:                Fri, 05 Jan 2018   Prob (F-statistic):           1.09e-97
    Time:                        18:24:03   Log-Likelihood:                -1612.6
    No. Observations:                 506   AIC:                             3233.
    Df Residuals:                     502   BIC:                             3250.
    Df Model:                           3
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept     -3.3066      4.038     -0.819      0.413     -11.240       4.627
    CRIM          -0.2021      0.032     -6.301      0.000      -0.265      -0.139
    RM             7.3816      0.402     18.360      0.000       6.592       8.171
    PTRATIO       -1.0742      0.133     -8.081      0.000      -1.335      -0.813
    ==============================================================================
    Omnibus:                      233.249   Durbin-Watson:                   0.828
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):             2002.374
    Skew:                           1.802   Prob(JB):                         0.00
    Kurtosis:                      12.054   Cond. No.                         311.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Given by the model summary above, the F-statistic computed here is about 244, which explains more of the variance than the simple regression model used previously.

The R^2 statistic here is 0.593, which is again another indicator of the improvement in fitting with the inclusion of these 3 variables rather than the single feature used before, which has an r^2 value of approximately 0.256

From the model summary, it seems that the feature 'RM', which is the average number of rooms per unit of housing, is the strongest predictor in this set.

## Part 4: Comparing Models

During modeling, there will be times when we want to compare models to see which one is more predictive or fits the data better. There are many ways to compare models, but we will focus on two.

### The $F$-Statistic Revisited

The $F$-statistic can also be used to compare two *nested* models, that is, two models trained on the same dataset where one of the models contains a *subset* of the variables of the other model. The *full* model contains $K$ variables and the *reduced* model contains a subset of these $K$ variables. This allows us to add additional variables to a base model and then test if adding the variables helped the model fit.

$$F = \frac{\left( \frac{RSS_{reduced} - RSS_{full}}{DF_{reduced} - DF_{full}} \right)}{\left( \frac{RSS_{full}}{DF_{full}} \right)}$$

where $DF_x = N - K_x - 1$ where $K_x$ is the number of variables in model $x$.

### Akaike Information Criterion (AIC)

Another statistic for comparing two models is AIC, which is based on the likelihood function and takes into account the number of variables in the model.

$$AIC = 2 K - 2 \log_e{L}$$

where $L$ is the likelihood of the model. AIC is meaningless in the absolute sense, and is only meaningful when compared to AIC values from other models. Lower values of AIC indicate better fitting models.

`statsmodels` provides the AIC in its output.

Part 4 Checkup Exercises

Exercise: Find another variable (or two) to add to the model we built in Part 3. Compute the $F$-test comparing the two models as well as the AIC. Which model is better?

```python
multi5_m = ols('PRICE ~ CRIM + RM + PTRATIO + AGE + NOX',bos).fit()
print(multi5_m.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.616
    Model:                            OLS   Adj. R-squared:                  0.612
    Method:                 Least Squares   F-statistic:                     160.3
    Date:                Fri, 05 Jan 2018   Prob (F-statistic):          1.92e-101
    Time:                        18:33:09   Log-Likelihood:                -1598.2
    No. Observations:                 506   AIC:                             3208.
    Df Residuals:                     500   BIC:                             3234.
    Df Model:                           5
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept      5.9649      4.449      1.341      0.181      -2.777      14.707
    CRIM          -0.1353      0.034     -4.025      0.000      -0.201      -0.069
    RM             6.9133      0.402     17.196      0.000       6.123       7.703
    PTRATIO       -1.0493      0.131     -7.982      0.000      -1.308      -0.791
    AGE           -0.0127      0.014     -0.937      0.349      -0.039       0.014
    NOX          -11.1027      3.400     -3.265      0.001     -17.783      -4.423
    ==============================================================================
    Omnibus:                      289.292   Durbin-Watson:                   0.796
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):             3125.608
    Skew:                           2.290   Prob(JB):                         0.00
    Kurtosis:                      14.281   Cond. No.                     1.46e+03
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
    [2] The condition number is large, 1.46e+03. This might indicate that there are
    strong multicollinearity or other numerical problems.

Adding in the NOX and AGE features, it seems that there is no gain in performance of the model, which suggests that the features added do not add much enough extra information to offset the increase in complexity that the introduce into the model.

## Part 5: Evaluating the Model via Model Assumptions and Other Issues
***
Linear regression makes several assumptions. It is always best to check that these assumptions are valid after fitting a linear regression model.

**Linearity**. The dependent variable $Y$ is a linear combination of the regression coefficients and the independent variables $X$. This can be verified with a scatterplot of each $X$ vs. $Y$ and plotting correlations among $X$. Nonlinearity can sometimes be resolved by [transforming](https://onlinecourses.science.psu.edu/stat501/node/318) one or more independent variables, the dependent variable, or both. In other cases, a [generalized linear model](https://en.wikipedia.org/wiki/Generalized_linear_model) or a [nonlinear model](https://en.wikipedia.org/wiki/Nonlinear_regression) may be warranted.

**Constant standard deviation**. The SD of the dependent variable $Y$ should be constant for different values of X. We can check this by plotting each $X$ against $Y$ and verifying that there is no "funnel" shape showing data points fanning out as $X$ increases or decreases. Some techniques for dealing with non-constant variance include weighted least squares (WLS), [robust standard errors](https://en.wikipedia.org/wiki/Heteroscedasticity-consistent_standard_errors), or variance stabilizing transformations.

 **Normal distribution for errors**.  The $\epsilon$ term we discussed at the beginning are assumed to be normally distributed. This can be verified with a fitted values vs. residuals plot and verifying that there is no pattern, and with a quantile plot.
  $$ \epsilon_i \sim N(0, \sigma^2)$$
Sometimes the distributions of responses $Y$ may not be normally distributed at any given value of $X$.  e.g. skewed positively or negatively.

 **Independent errors**.  The observations are assumed to be obtained independently.

e.g. Observations across time may be correlated

There are some other issues that are important investigate with linear regression models.

**Correlated Predictors:** Care should be taken to make sure that the independent variables in a regression model are not too highly correlated. Correlated predictors typically do not majorly affect prediction, but do inflate standard errors of coefficients making interpretation unreliable. Common solutions are dropping the least important variables involved in the correlations, using regularlization, or, when many predictors are highly correlated, considering a dimension reduction technique such as principal component analysis (PCA).

**Influential Points:** Data points that have undue influence on the regression model. These points can be high leverage points or outliers. Such points are typically removed and the regression model rerun.

Part 5 Checkup Exercises

Take the reduced model from Part 3 to answer the following exercises. Take a look at [this blog post](http://mpastell.com/2013/04/19/python_regression/) for more information on using statsmodels to construct these plots.

Exercise: Construct a fitted values versus residuals plot. What does the plot tell you? Are there any violations of the model assumptions?

Exercise: Construct a quantile plot of the residuals. What does the plot tell you?

Exercise: What are some advantages and disadvantages of the fitted vs. residual and quantile plot compared to each other?

Exercise: Identify any outliers (if any) in your model and write a story describing what these outliers might represent.

Exercise: Construct a leverage plot and identify high leverage points in the model. Write a story explaining possible reasons for the high leverage points.

Exercise: Remove the outliers and high leverage points from your model and run the regression again. How do the results change?

```python
# Your turn.
res = multi_m.resid
plt.scatter(multi_m.fittedvalues, res)
plt.show()

plt.hist(multi_m.get_influence().resid_studentized_internal)
plt.ylabel('Count')
plt.xlabel('Normalized residuals')
plt.show()
```

The residual errors shown an approximate normal distribution of the residual as calculated by the RSS, with the exception of a few outliers.

```python
from statsmodels.graphics.gofplots import ProbPlot

QQ = ProbPlot(multi_m.get_influence().resid_studentized_internal)
quantile_plot = QQ.qqplot(line='45', alpha=0.5)

quantile_plot.axes[0].set_title('Normal Q-Q')
quantile_plot.axes[0].set_xlabel('Theoretical Quantiles')
quantile_plot.axes[0].set_ylabel('Standardized Residuals');
```

We can see from the quantile plot above that the residuals are mostly linear, except in the high range where they deviate significantly, which may be due to outlier behavior.

The fitted plot of predicted vs. True prices we constructed earlier has the main advantage of being clearly interpretable, although the quantile plot we have constructed is a clearer metric for evaluation of our model if one takes the time to learn the meaning.

The outliers in the high range most likely can be attributed to some housing units that are extremely expensive due to some other features not taken into account in the 3 features we have included such as historical importance of the residence or some uniquely valuable location within a town that makes it more valuable than would otherwise be predicted.

```python
from statsmodels.graphics.regressionplots import *

plot_leverage_resid2(multi_m)
plt.show()
```

From the plot above, there seems to be a point of high leverage at 380, but there are also points notable higher than the rest shown at 405 and 418. This may be the point where the outlier behavior is strongly influencing the model performance.

```python
model_leverage = multi_m.get_influence().hat_matrix_diag
plt.scatter(range(len(model_leverage)), model_leverage)
plt.title('Alternative View of Leverage vs. Residuals')
plt.xlabel('Index of Residuals')
plt.ylabel('Leverage')

plt.show()
```

```python
# Refit model without outliers
bos_dropped = bos.drop(bos.index[[380,398,404,405,410,414,418,427]])
multi_m_mod = ols('PRICE ~ CRIM + RM + PTRATIO',bos_dropped).fit()
print(multi_m_mod.summary())

#plot leverage
plot_leverage_resid2(multi_m_mod)
plt.show()
```

By dropping a few of the outliers with very high leverage in the model, we were able to slightly increase the model performance, as gauged by the F-statistic, AIC and BIC metrics.

Mini-Project-5 - Linear Regression using Boston Housing Dataset/Mini_Project_Linear_Regression.ipynb