```python
!pip install -U gif
```

    Requirement already up-to-date: gif in /home/avani/anaconda3/lib/python3.8/site-packages (3.0.0)
    Requirement already satisfied, skipping upgrade: Pillow>=7.1.2 in /home/avani/anaconda3/lib/python3.8/site-packages (from gif) (7.2.0)

```python
import io, os, sys, types
from IPython import get_ipython
from nbformat import read
from IPython.core.interactiveshell import InteractiveShell

def find_notebook(fullname, path=None):
    """find a notebook, given its fully qualified name and an optional path

    This turns "foo.bar" into "foo/bar.ipynb"
    and tries turning "Foo_Bar" into "Foo Bar" if Foo_Bar
    does not exist.
    """
    name = fullname.rsplit('.', 1)[-1]
    if not path:
        path = ['']
    for d in path:
        nb_path = os.path.join(d, name + ".ipynb")
        #print('searching: %s'%nb_path)
        if os.path.isfile(nb_path):
            return nb_path
        # let import Notebook_Name find "Notebook Name.ipynb"
        nb_path = nb_path.replace("_", " ")
        #print('searching: %s' % nb_path)
        if os.path.isfile(nb_path):
            return nb_path

class NotebookLoader(object):
    """Module Loader for Jupyter Notebooks"""
    def __init__(self, path=None):
        self.shell = InteractiveShell.instance()
        self.path = path

    def load_module(self, fullname):
        """import a notebook as a module"""
        path = find_notebook(fullname, self.path)

        print ("importing Jupyter notebook from %s" % path)

        # load the notebook object
        with io.open(path, 'r', encoding='utf-8') as f:
            nb = read(f, 4)

        # create the module and add it to sys.modules
        # if name in sys.modules:
        #    return sys.modules[name]
        mod = types.ModuleType(fullname)
        mod.__file__ = path
        mod.__loader__ = self
        mod.__dict__['get_ipython'] = get_ipython
        sys.modules[fullname] = mod

        # extra work to ensure that magics that would affect the user_ns
        # actually affect the notebook module's ns
        save_user_ns = self.shell.user_ns
        self.shell.user_ns = mod.__dict__

        #print('Found %d cells'%len(nb.cells))
        try:
          for cell in nb.cells:
            if cell.cell_type == 'code':
                # transform the input to executable Python
                code = self.shell.input_transformer_manager.transform_cell(cell.source)
                # run the code in themodule
                exec(code, mod.__dict__)
        finally:
            self.shell.user_ns = save_user_ns
        return mod

class NotebookFinder(object):
    """Module finder that locates Jupyter Notebooks"""
    def __init__(self):
        self.loaders = {}

    def find_module(self, fullname, path=None):
        nb_path = find_notebook(fullname, path)
        if not nb_path:
            return

        key = path
        if path:
            # lists aren't hashable
            key = os.path.sep.join(path)

        if key not in self.loaders:
            self.loaders[key] = NotebookLoader(path)
        return self.loaders[key]

#  register the NotebookFinder with sys.meta_path
print('running importer')
sys.meta_path.append(NotebookFinder())
```

    running importer

# Linear Discriminant Analysis

Also, known as Fisher's LDA, the principle idea behind the algorithm is to reduce the d-dimensional data $\mathbf{x \in R^{d}}$ and map it to one dimension by finding a direction that best separates the projected data $\mathbf{w^{\top}x}$.

$\textbf{Goal:}$ Find a direction that best separates projected data $\mathbf{w^{\top}x}$.

Let's consider a two-class problem:

\begin{align}
\mu_0 &= \frac{1}{n_0} \sum_{\mathbf{i:y_i = 0}} \mathbf{x_i} &
\mu_1 &= \frac{1}{n_1} \sum_{\mathbf{i:y_i = 1}} \mathbf{x_i}
\end{align}

We would like to:

1.   Minimize the within class variance.
2.   Maximize the distance between projected class means.

The distance between projected class means is:

\begin{align}
(\mathbf{w^{\top} \mu_0 - w^{\top} \mu_1})^2 &= (\mathbf{w^{\top} \mu_0 - w^{\top} \mu_1})^{\top} (\mathbf{w^{\top} \mu_0 - w^{\top} \mu_1})\\
&= (\mu_0 - \mu_1)^{\top} \mathbf{w w^{\top}} (\mu_0 - \mu_1)\\
&= \mathbf{w^{\top}} (\mu_0 - \mu_1)^{\top} (\mu_0 - \mu_1) \mathbf{w}\\
&= \mathbf{w^{\top}} S_B \mathbf{w}
\end{align}

where $S_B$ is the between-class variance.

Minimizing the within-class variance is equivalent to minimizing the sum of all individual within-class variances. Thus, the within-class variance is:

\begin{align}
\mathbf{w^{\top} \Sigma_0 w + w^{\top} \Sigma_1 w} &= \mathbf{w^{\top} (\Sigma_0 + \Sigma_1) w}\\
&= \mathbf{w^{\top}} S_W \mathbf{w}
\end{align}

where $S_W$ is the within-class variance.

Hence, our optimization goal becomes:

\begin{equation}
\max_\mathbf{w} \frac{\mathbf{w^{\top}} S_B \mathbf{w}}{\mathbf{w^{\top}} S_W \mathbf{w}}
\end{equation}

Note that $\mathbf{w}$ is unbounded. We can convert the objective function into a constrained optimization goal using Lagrange multipliers.

Our optimization will be subject to the constraint, $\mathbf{w^{\top}} S_W \mathbf{w} = 1$.

\begin{equation}
\phi(\mathbf{w}, \lambda) = \mathbf{w^{\top}} S_B \mathbf{w} - \lambda(\mathbf{w^{\top}} S_W \mathbf{w} - 1)
\end{equation}

Differentiating w.r.t $\mathbf{w}$ we get:

\begin{align}
\frac{\partial \phi}{\partial \mathbf{w}} = 2 \cdot S_B \cdot \mathbf{w} - \lambda \cdot 2 \cdot S_W \cdot \mathbf{w} &= 0\\
S_B \mathbf{w} &= \lambda \cdot S_W \cdot \mathbf{w}
\end{align}

Therefore, the generalized eigenvector problem is:

\begin{equation}
S_W^{-1} S_B \mathbf{w} = \lambda \mathbf{w}
\end{equation}

where $\lambda$ and $\mathbf{w}$ are the eigenvalues and eigenvectors of $S_W^{-1} S_B$ respectively.

A simplified solution for two-class problem is:

\begin{equation}
S_W^{-1} (\mu_0 - \mu_1) \propto \mathbf{w}
\end{equation}

where $S_B = \mu_0 - \mu_1$.

## LDA - Binary classification on Partially separable dataset

```python
from sklearn.datasets import make_classification
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns; sns.set();
import pandas as pd
from sklearn.model_selection import train_test_split
from numpy import pi

from utils import get_accuracy, get_prediction, plot_decision_boundary, generate_gifs
from utils import plot_2D_input_datapoints, signum, multi_class_signum, normalize
```

    importing Jupyter notebook from utils.ipynb

```python
plt.figure(figsize=(8,8));

np.random.seed(20)

# Generating a dataset
X, y = make_classification(n_samples=1000, n_features=2, n_redundant=0, n_informative=1, n_clusters_per_class=1, flip_y=-1)

# Absorbing weight b of the hyperplane
b_ones = np.ones((len(X), 1))
X = np.hstack((X, b_ones))

y = y.reshape((-1, 1))
```

    <Figure size 576x576 with 0 Axes>

```python
plot_2D_input_datapoints(X[:, :2], y)
```

```python
# Train-Val-Test split
X_train, X_test, Y_train, Y_test = train_test_split(X, y, shuffle=True, test_size = 0.2)

print("Training dataset: ", X_train.shape)
print("Test dataset: ", X_test.shape)
```

    Training dataset:  (800, 3)
    Test dataset:  (200, 3)

```python
# Normalizing X_train and absorbing weight b of the hyperplane
X_normalized_train = normalize(X_train[:, :2])

b_ones = np.ones((len(X_normalized_train), 1))
X_normalized_train = np.hstack((X_normalized_train, b_ones))
```

```python
# Calculating covariance of an input matrix
def calc_cov_matrix(X_input):
  n_samples = np.shape(X_input)[0]
  cov_matrix = np.array((1 / (n_samples-1)) * (X_input - X_input.mean(axis=0)).T.dot(X_input - X_input.mean(axis=0)))

  return cov_matrix
```

Before we dive in the train method, let us look at the pseudo code.

Given a dataset $D = (x_i, y_i)_{i=1}^n$ and class labels $i = 0, 1$.

1.  Collecting class-specific dataset. $D_i = \{x_j | y_j = c_i; j = 1, 2, ...., n\}$
2.  Calcualte class-specific means, $\mu_i = mean(D_i)$.
3.  Calculate between class scatter matrix, $S_B = (\mu_0 - \mu_1)(\mu_0 - \mu_1)^{\top}$
4.  Center class-specific dataset using class-specific means, $Z_i = D_i - 1_{n_i} \mu_i^{\top}$
5.  Compute class-specific scatter matrices, $S_i = Z_i^{\top} Z_i$
6.  Compute within scatter matrix, $S_W = S_0 + S_1$
7.  Compute dominant eigenvector, $\lambda_1, w = eigen(S_W^{-1} S_B)$

```python
def train(X_train, y_train):

  """Train method for LDA.

  Parameters
  -----------
  X_train: ndarray (num_examples(rows) vs num_features(columns))
   Input dataset which LDA will use to obtain optimal weights during training

  y_train: ndarray (num_examples(rows) vs class_labels(columns))
  """

  # Collecting all class 0 and class 1 into separate variables
  class_X0 = X_train[np.argwhere(y_train == 0)[:, 0]]
  class_X1 = X_train[np.argwhere(y_train == 1)[:, 0]]

  # Getting number of examples in each class
  num_class_X0_samples = np.shape(class_X0)[0]
  num_class_X1_samples = np.shape(class_X1)[0]

  # Computing class mean for each label and calculating the difference between them.
  class_X0_mean = class_X0.mean(0)
  class_X1_mean = class_X1.mean(0)
  class_mean_diff = class_X1_mean - class_X0_mean
  class_mean_diff = class_mean_diff.reshape((-1, 1))
  SB = np.dot(class_mean_diff, class_mean_diff.T)

  # Calculating covariance matrix
  cov_mat_class_X0 = calc_cov_matrix(class_X0)
  cov_mat_class_X1 = calc_cov_matrix(class_X1)
  #SW = num_class_X0_samples * cov_mat_class_X0 + num_class_X0_samples * cov_mat_class_X1
  SW = cov_mat_class_X0 + cov_mat_class_X1

  eigvals, eigvecs = np.linalg.eig(np.linalg.pinv(SW).dot(SB))

  # Getting the eigenvectors with the maximum eigenvalue.
  idx = eigvals.argsort()[::-1]
  eigvals = eigvals[idx][:1]
  weights = np.atleast_1d(eigvecs[:, idx])[:, :1]

  return weights
```

```python
trained_weights = train(X_normalized_train, Y_train)
```

```python
# Predict on test set
num_test_samples = np.shape(Y_test)[0]

y_test_predicted = np.dot(X_test, trained_weights)
y_test_predicted[y_test_predicted >= 0] = 1
y_test_predicted[y_test_predicted < 0] = 0

y_test_predicted = y_test_predicted.reshape((-1, 1))
Y_test = Y_test.reshape((-1, 1))

# Getting misclassfied points on test set
miscls_test_points = np.unique(np.argwhere(y_test_predicted != Y_test)[:, 0])
accuracy = 1-(len(miscls_test_points)/num_test_samples)
print("Accuracy: ", accuracy*100)
```

    Accuracy:  93.5

```python
plot_decision_boundary(X_test, Y_test, trained_weights, dataset_type='test', class_label_01_form='on', model_type='LinearDA')
```

```python

```


```python
# remove_cell
import sys
sys.path.insert(0, '/home/jovyan/ros/')

%load_ext autoreload
%autoreload 2
```

```python
# remove_cell
import itertools as it
import logging
import operator

import altair as A
import arviz as az
from bambi import Model
import dscontrib.wbeard as wb
import matplotlib.pyplot as plt
import numpy as np
import numpy.random as nr
import pandas as pd
import scipy.stats as st
from numba import njit
import toolz.curried as z
import seaborn as sns

from ros.utils.common import hstack, vstack, plot_wrap, drop_outliers, disable_logging
from ros.utils import bootstrap as bs, plot as plu

disable_logging(["numba", "arviz", "pymc3", "bambi", "numexpr"])  #

str_concat = z.compose("-".join, z.map(str))
lmap = z.comp(list, map)
plt.rcParams["font.size"] = 17

p = lambda: None
p.__dict__.update(
    dict(
        zip(
            "hide_output hide_input collapse_hide collapse_show remove_cell".split(),
            range(10),
        )
    )
)
```

    WARNING (theano.configdefaults): install mkl with `conda install mkl-service`: No module named 'mkl'

    Warning, $BQ_CREDS not found in environment

    /opt/conda/lib/python3.7/site-packages/sklearn/utils/deprecation.py:143: FutureWarning: The sklearn.preprocessing.data module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.preprocessing. Anything that cannot be imported from sklearn.preprocessing is now part of the private API.
      warnings.warn(message, FutureWarning)

There's an interplay between sample size, effect size, and the sensitivity of an experiment to detect changes that we spend time thinking about at Mozilla. All else equal, it's usually preferable to enroll a smaller sample size so long as it's sufficient to pick up the signal of the treatment. Among other reasons, this helps reduce the likelihood of different experiments interacting with each other. But there are ways to increase the resolution of experimental analysis without having to increase the population size, and this efficient frontier is a good place to aim for.

Some techniques I've tried lately are using [blocking](https://en.wikipedia.org/wiki/Blocking_(statistics)) and pre-treatment predictors as useful ways to get more precise estimates for free, without the need of a larger study population. This post simulates experimental data to demonstrate the improvement in precision that you can get with these.

## The Setup

The example here is a study that measures an improvement in startup times of a new feature. This is a metric we pay quite a bit of attention to in the platform area, and are obviously interested in features that can reduce startup times. The study population in this example has a distribution of startup times, but on average Windows 7 users have longer times than Windows 10 users.[1]

The basic idea with blocking is that if 2 groups in the population have significantly different outcomes, independent of the treatment variable, you can get a more precise estimate of the treatment effect by modeling these groups separately.
Intuitively, if the 2 groups have significantly different outcomes even before the treatment is applied, this difference will contribute to a higher variance in the estimate when it comes time to measure the size of the treatment effect. The variable that determines the grouping needs to be independent of the treatment assignment, so using Windows 7 as a blocking factor would be a good choice, as our feature doesn't do anything preposterous like upgrade the OS once the client enrolls.

The second idea is to use pre-treatment variables as a predictor. In this case, it involves looking at the startup time before enrollment, and seeing how much this changes on average for the treatment group once they get the feature. This works if a
client's pre-treatment startup time $t_{pre}$ is more informative of the post-treatment startup time $t_{post}$ than merely knowing the OS version, and it's safe to assume here that $t_{post}$ and the OS are conditionally independent given $t_{pre}$.

As with many metrics we use, the log of the startup time more closely follows the distributions we're used to. For this simulation we'll set the log of the first_paint time to follow a gamma distribution, with the mean time increased for Windows 7 users.[2] For users in the treatment group, we'll add a noisy log(.9)  (=-.105) to the distribution, which translates to roughly a 10% decrease in startup times on the linear scale.[3] After the simulation, we'll look at how much of an improvement you get with the estimates when using a randomized block design. The formulas describing the simulation are

\begin{align}
fp_{baseline} & \sim \mathrm{Gamma}(4 + \mathbb 1_{win7} \cdot \mu_{win}) \\
w_{treat} & \sim \mathcal N (\mathbb 1_{treat} \cdot \mu_{treat}, \sigma_{treat}) \\
\log(first\_paint) & = fp_{baseline} + w_{treat}
\end{align}

```python
# hide_input
@njit
def seed(n):
    nr.seed(n)

@njit
def randn(mu, sig, size=1):
    return nr.randn(size) * sig + mu
```

```python
# collapse_show
WIN7_FACT = 1.2
TREAT_MU = np.log(.9)
TREAT_SD = .15

@njit
def gen_log_first_paint_pre_post(win7, treat, size=1):
    pre = nr.gamma(4 + WIN7_FACT * win7, 1, size=size)
    return np.concatenate((pre, pre + randn(TREAT_MU * treat, TREAT_SD, size=size)))
```

```python
# collapse_hide
n_each = 10_000
n_win_7 = {0: n_each, 1: n_each}
seed(0)

def add_columns(df):
    pre_post = pd.DataFrame(
        [
            gen_log_first_paint_pre_post(win7, treat=treat)
            # gen_pre_post(win7, win7_fact=WIN7_FACT, treat=treat, treat_fact=TREAT_FACT,)
            for win7, treat in df[["win7", "treat"]].itertuples(index=False)
        ],
        columns=["lpre", "lpost"],
    ).assign(
        pre=lambda df: np.exp(df.lpre),
        post=lambda df: np.exp(df.lpost),
    )
    df = hstack([df, pre_post])
    df = (
        df.assign(os=lambda df: df.win7.map({0: "win10", 1: "win7"}))
        .reset_index(drop=0)
        .rename(columns={"index": "id"})
    )
    df["demo"] = [
        str_concat(tup)
        for tup in df[["treat", "os"]]
        .assign(treat=lambda df: df.treat.map({1: "treat", 0: "control"}))
        .itertuples(index=False)
    ]
    return df

def create_test_pop(n_each=50):
    data_dct = [
        {"win7": win7, "treat": treat}
        for win7 in (0, 1)
        for treat in (0, 1)
        for _ in range(n_win_7[win7])
    ]
    df_ = pd.DataFrame(data_dct)
    df = df_.pipe(add_columns)
    return df

def stack_pre_post(df):
    """
    demo[graphic] is concatenated values for
    ["treat", "os"]
    demo2 is for ["treat", "os", "pre"]
    """
    dfs = (
        df.set_index(["id", "win7", "os", "treat", "demo"])
        .stack()
        .reset_index(drop=0)
        .rename(columns={"level_5": "pre", 0: "y"})
        .assign(
            demo2=lambda df: lmap(
                str_concat, df[["demo", "pre"]].itertuples(index=False)
            )
        )
        .assign(pre=lambda df: (df.pre == "pre").astype(int))
    )
    return dfs

treat_i2name = {0: "control", 1: "treat"}
df = create_test_pop().assign(branch=lambda df: df.treat.map(treat_i2name))
dfs = stack_pre_post(df.drop(["lpre", "lpost", "branch"], axis=1)).assign(
    branch=lambda df: df.treat.map(treat_i2name)
)
```

The data is in the following format:

```python
# remove_cell
df.sample(frac=1, random_state=0).drop_duplicates(["win7", "treat"]).drop(
    ["demo", "win7", "lpre", "lpost", 'treat'], axis=1
).reset_index(drop=1)
```

      id
      pre
      post
      os
      branch

      0
      12836
      47.077870
      33.275930
      win10
      treat

      1
      4214
      16.080462
      14.603434
      win10
      control

      2
      31403
      174.809855
      157.946772
      win7
      treat

      3
      27440
      161.269030
      185.037511
      win7
      control

```python
# collapse_hide
dfs.sample(frac=1, random_state=0).drop_duplicates(["win7", "treat", "pre"]).drop(
    ["demo", "win7", 'treat', 'demo2'], axis=1
).reset_index(drop=1)[:6]
```

      id
      os
      pre
      y
      branch

      0
      3694
      win10
      0
      1047.670220
      control

      1
      15330
      win10
      0
      7.301414
      treat

      2
      31595
      win7
      0
      481.718039
      treat

      3
      14683
      win10
      1
      10.478407
      treat

      4
      32253
      win7
      1
      98.373985
      treat

      5
      26217
      win7
      1
      53.140637
      control

and can be split into 4 groups based on the OS version (`win7` or `win10`) and study branch (`treat` or`control`). Each client (identified by an integer `id` value) has a measurement before and after enrolling into the experiment, denoted by a binary flag in the `pre` column. This pre- and post-measurement will help get more a more sensitive estimate of the treatment effect.

The following histograms show the distributions for pre- and post-enrollment measurements:

```python
# collapse_hide
axs, spi = wb.plot_utils.mk_sublots(nrows=1, ncols=2, figsize=(16, 5), sharex=False)

for pre, pdf in dfs.groupby("pre"):
    spi.n
    plt.title(["Pre-enrollment", "Post-enrollment"][pre])
    for branch, bdf in pdf.groupby("branch"):
        lbins = np.logspace(0, np.log(bdf.y.max()), 100)
        plt.hist(bdf.y, bins=lbins, density=1, alpha=0.5, label=branch)

    plt.legend()
    plt.xscale("log")

    if not pre:
        plt.ylabel("Density")
    plt.xlabel("Simulated first paint (ms)")
```

These are mostly overlapping, and show roughly gamma distributed times on the log scale. Notice that while most of the density is less than $10^4=10,000$ ms (~96% to be more precise), the scale extends to around $10^{19}$, indicating some serious outliers.

The plot below shows the quantile ranges covering the middle 50% (thick lines) and middle 95% (thin lines) of times, with the median represented with the dot for the different slices of data.

```python
# collapse_hide
plu.plot_groupby(dfs, 'demo2', 'y', quants=[50, 95])
plt.xscale('log')
```

This plot gives a finer view of the difference between the distributions. Each pair for the control branch (the first four lines) should have identical values, but for the treatment groups, the `post` measurements should be slightly less than the `post` measurements. Note that the Windows 7 distributions are significantly longer, and that this difference in OS times is larger than the difference in treatment.

## Difference in means

```python
# remove_cell
dfs_pre = dfs.query("pre == 1")
dfs_pre.groupby("branch").y.mean()
```

    branch
    control     9317.992199
    treat      34222.844459
    Name: y, dtype: float64

A common way of evaluating the effect of a treatment is by looking at the difference in the mean of each group, with bootstrapping to measure the uncertainty of the statistic. (Comparing branches throughout the distribution can give a lot more insight, but comparing means keeps the illustration simpler). So a standard way to compare the difference between branches would be to just take the mean of the `post` measurements for each branch (which are representative of the times while the experiment is running), take the difference between these, and bootstrap this statistic.

```python
# collapse_hide
def bootstrap_stat_diff(
    df, gbcol, ycol, stat_fn=np.mean, n_reps=10_000, comp=operator.sub
):
    """
    `gbcol` needs to be col of 0 and 1's, designating
    the groups being compared.
    """
    control = df[ycol][df[gbcol] == 0].values
    treat = df[ycol][df[gbcol] == 1].values
    size = len(control)
    stats = []

    for _ in range(n_reps):
        ac = nr.choice(control, size=size, replace=True)
        at = nr.choice(treat, size=size, replace=True)
        #         stats.append(stat_fn(at) - stat_fn(ac))
        stats.append(comp(stat_fn(at), stat_fn(ac)))

    return np.array(stats)

def uplift(a, b):
    return a / b - 1

def plot_bs(diffs, log=False, title=None, pe=1, ax=None):
    diffs_p = plot_wrap(diffs)
    plu.plot_groupby(
        diffs_p,
        "g",
        "y",
    )
    plt.yticks([0], ["Post-enrollment" if pe else ""])
    if log:
        plt.xscale("log")
    plt.title(title)

dfs_post = dfs.query("pre == 0").assign(yl=lambda df: df.y.pipe(np.log))
```

```python
# remove_cell
diffs = bootstrap_stat_diff(dfs_post, "treat", "y")
diffs_out_rm = bootstrap_stat_diff(dfs_post.pipe(drop_outliers), "treat", "y")
diffs_log = bootstrap_stat_diff(dfs_post, "treat", "yl")
diffs_gmean = bootstrap_stat_diff(dfs_post, "treat", "y", stat_fn=st.gmean)
ul_out_rm = bootstrap_stat_diff(dfs_post.pipe(drop_outliers), 'treat', 'y', comp=uplift)
ul = bootstrap_stat_diff(dfs_post, 'treat', 'y', comp=uplift)
```

A first thing to note is that doing this is a Bad Idea with log-normal distributed data. The following show both the bootstrapped differences in means and bootstrapped uplift (% increase in times of the treatment over the control group):

```python
# hide_input
axs, spi = wb.plot_utils.mk_sublots(nrows=2, ncols=1, figsize=(8, 8), sharex=False)

plot_bs(diffs, log=1, title='Difference in means', ax=spi.n)
plot_bs(ul * 100, log=0, title='% Increase in treatment', pe=1, ax=spi.n)
plt.tight_layout()
```

Instead of showing a _decrease_ in times from the treatment group, these estimates show an _increase_ in the times. And an increase on the order of 200%! This goes back to the histograms above that showed outliers way higher than the median time. These bootstrapped estimates above are essentially dominated by a handful of outliers; the top 5 are all from the treatment group:

```python
# hide_input
dfs_post.sort_values('y', ascending=False)[:5][['branch', 'y']][::-1].reset_index(drop=1)
```

      branch
      y

      0
      treat
      3.222020e+07

      1
      treat
      3.610928e+07

      2
      treat
      4.309360e+07

      3
      treat
      5.817939e+07

      4
      treat
      3.184504e+08

When the outliers (arbitrarily chosen to be the top .01%) are removed, the estimates look a bit more sane:

```python
# hide_input
axs, spi = wb.plot_utils.mk_sublots(nrows=2, ncols=1, figsize=(8, 10))

plot_bs(diffs_out_rm, log=0, title='Difference in means', pe=0, ax=spi.n)

plot_bs(ul_out_rm, log=0, title='% Increase in treatment', pe=0, ax=spi.n)
plt.suptitle("Outliers removed");
# plt.tight_layout();
```

In fact, the ~10% decrease we programmed into the simulation is now showing up as a plausible value in the plot below, but these estimates also give a relatively high likelihood of there being no effect (that is, 0 is within the 50% CI).

Taking the log and bootstrapping those means, however, similarly allows for a 10% decrease in startup times, but this time rules out a positive effect:

```python
plot_bs(diffs_log, log=0, title='Difference in mean of log(startup time)', pe=0)
```

Note that this is very similar to taking the geometric mean (after taking the mean of the log, you'd just need to exponentiate the result), which I often prefer as a more robust statistic on this kind of data, for exactly this reason.

## Linear regression as a difference in means

```python
# remove_cell
fit_kwa = dict(samples=1000, chains=4, backend='pymc3', cores=4, target_accept=.8)

data = dfs.assign(yl=lambda df: df.y.pipe(np.log))
```

Another way to accomplish the difference in means from above is with Bayesian linear regression. I'm using [Bambi](https://github.com/bambinos/bambi) with [pymc3](https://github.com/pymc-devs/pymc3/) for easy integration with python, and we'll be operating on the log from here on out due to the stability issues shown above. Bambi allow r-like formula syntax, and we can start off with the simplest regression: `log(y) ~ treat`, where `treat` is 1 for clients in the treatment group and 0 otherwise. When run on the post-enrollment data, the coefficient for `treat` gives a similar estimate as the bootstrapped mean does:

```python
# collapse_show
def fit_regression(df):
    return Model(df).fit("yl ~ treat", **fit_kwa)
```

```python
# remove_cell
fit = fit_regression(data.query("pre == 0").copy())
```

  <progress value='8000' class='' max='8000' style='width:300px; height:20px; vertical-align: middle;'></progress>
  100.00% [8000/8000 00:27<00:00 Sampling 4 chains, 10 divergences]

    There were 8 divergences after tuning. Increase `target_accept` or reparameterize.
    There were 2 divergences after tuning. Increase `target_accept` or reparameterize.
    /opt/conda/lib/python3.7/site-packages/arviz/data/io_pymc3.py:89: FutureWarning: Using `from_pymc3` without the model will be deprecated in a future release. Not using the model will return less accurate and less useful results. Make sure you use the model argument or call from_pymc3 within a model context.
      FutureWarning,

```python
# hide_input
posterior_treatment = plot_wrap(fit.posterior.treat.values.ravel())

post_plus_boot = vstack([
    plot_wrap(diffs_log).assign(g='bootstrapped mean'),
    posterior_treatment.assign(g='linear model posterior')
])

plu.plot_groupby(post_plus_boot, 'g', 'y')
yo, yi = plt.ylim()
plt.vlines([TREAT_MU, TREAT_MU], yo, yi, linestyles='dashed');
```

I hadn't realized this, but it makes sense in hindsight; if the value of the indicator variable is 1, then the slope (aka the coefficient) is going to be equivalent to the mean of the difference between the two groups.

## Regression with Blocking and baseline

Now we get what we came here for. The model to use OS version as a blocking variable will be `log(y) ~ treat + win7`, and the formula that adds pretreatment will be `log(y_post) ~ log(y_pre) + treat + win7`.

```python
def fit_block_regression(df):
    return Model(df.copy()).fit("log_post_startup ~ treat + win7", **fit_kwa)

def fit_block_baseline_regression(df):
    return Model(df.copy()).fit("log_post_startup ~ log_baseline_startup + treat + win7", **fit_kwa)
```

```python
# remove_cell
df_pre_post = df.assign(log_baseline_startup=lambda df: df.pre.pipe(np.log), log_post_startup=lambda df: df.post.pipe(np.log))
fit_block = fit_block_regression(df_pre_post)
posterior_treatment_block = plot_wrap(fit_block.posterior.treat.values.ravel())
```

  <progress value='8000' class='' max='8000' style='width:300px; height:20px; vertical-align: middle;'></progress>
  100.00% [8000/8000 00:48<00:00 Sampling 4 chains, 0 divergences]

    /opt/conda/lib/python3.7/site-packages/arviz/data/io_pymc3.py:89: FutureWarning: Using `from_pymc3` without the model will be deprecated in a future release. Not using the model will return less accurate and less useful results. Make sure you use the model argument or call from_pymc3 within a model context.
      FutureWarning,

```python
# remove_cell
fit_block_baseline = fit_block_baseline_regression(df_pre_post)
posterior_treatment_block_bl = plot_wrap(fit_block_baseline.posterior.treat.values.ravel())
```

  <progress value='8000' class='' max='8000' style='width:300px; height:20px; vertical-align: middle;'></progress>
  100.00% [8000/8000 01:30<00:00 Sampling 4 chains, 0 divergences]

    /opt/conda/lib/python3.7/site-packages/arviz/data/io_pymc3.py:89: FutureWarning: Using `from_pymc3` without the model will be deprecated in a future release. Not using the model will return less accurate and less useful results. Make sure you use the model argument or call from_pymc3 within a model context.
      FutureWarning,

The estimated treatment effects of all the models run so far are below. The model with blocking shows only a slight improvement; the extent of the 95% credible interval is slightly, but visibly less than those of the previous models, and the median estimate moved a tiny bit toward the true effect (shown with the dashed line).

```python
# hide_input
plt.figure(figsize=(12, 5))
post_boot_block = vstack([
    posterior_treatment.assign(g='2- linear model posterior'),
    posterior_treatment_block.assign(g='3- linear model posterior:\nblocked'),
    posterior_treatment_block_bl.assign(g='4- linear model posterior:\nblocked + baseline'),
    plot_wrap(diffs_log).assign(g='1- bootstrapped mean'),
])

plu.plot_groupby(post_boot_block, 'g', 'y', quants=[50, 95], label='_no_label_')
yo, yi = plt.ylim()
plt.vlines([TREAT_MU, TREAT_MU], yo, yi, linestyles='dashed', label='true effect')
plt.legend();
```

But the real difference was when the pre-treatment effects were included. The estimate with the baseline predictor honed right in on the true effect, and the credible intervals tightened to a fraction of what the raw log mean estimates would have been. In this case, the baseline measurements increased both the accuracy and precision of the estimate remarkably.

## Summary and Caveats

This post hopefully makes clear the benefits of including the baseline measure, but a contrived example like this requires some caveats. Two major ones to be mindful of are the magnitude and consistency of the effect size across variables.  A 10% effect size is very large among the experimental results I've seen, so this example isn't representative in that respect (though this may highlight the importance of including baseline measurements!). I also added a constant 10% improvement to all clients in the treatment branch, regardless of their baseline measurement. From the actual data I've seen, though, the effect is very non-linear, partly due to a regression to the mean effect, and partly due to other factors.

Nevertheless, including the baseline measurement enables more precise estimates, especially when there's an imbalance in the experiment branches. If there had been more Windows 7 users in the treatment branch through some fluke of randomization, then a naive comparison of the means could underestimate the effect size if baselines aren't taken into account.

### TLDR highlights

- When working with data that appears to be log-distributed, looking at the difference in mean can give you noisy results
- When comparing 2 groups, modeling a group using an indicator variable as a linear predictor is equivalent to modeling the difference in means of the group
- Block variables and baseline measurements are a good way to improve the estimates of the effect of a treatment

```python

```


#  CS109A/STAT121A/AC209A/CSCIE109A  Introduction to Data Science

## Lab 3: plotting, K-NN Regression, Simple Linear Regression

**Harvard University**

**Fall 2019**

**Instructors:** Pavlos Protopapas, Kevin Rader, and Chris Tanner

**Material prepared by**: David Sondak, Will Claybaugh, Pavlos Protopapas, and Eleni Kaxiras.

---

```python
#RUN THIS CELL
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2018-CS109A/master/content/styles/cs109.css").text
HTML(styles)
```

## Learning Goals

By the end of this lab, you should be able to:
* Review `numpy` including 2-D arrays and understand array reshaping
* Use `matplotlib` to make plots
* Feel comfortable with simple linear regression
* Feel comfortable with $k$ nearest neighbors

**This lab corresponds to lectures 4 and 5 and maps on to homework 2 and beyond.**

## Table of Contents

####  HIGHLIGHTS FROM PRE-LAB

* [1 - Review of numpy](#first-bullet)
* [2 - Intro to matplotlib plus more ](#second-bullet)

####  LAB 3 MATERIAL

* [3 - Simple Linear Regression](#third-bullet)
* [4 - Building a model with `statsmodels` and `sklearn`](#fourth-bullet)
* [5 - Example: Simple linear regression with automobile data](#fifth-bullet)
* [6 - $k$Nearest Neighbors](#sixth-bullet)

```python
import numpy as np
import scipy as sp
import matplotlib as mpl
import matplotlib.cm as cm
import matplotlib.pyplot as plt
import pandas as pd
import time
pd.set_option('display.width', 500)
pd.set_option('display.max_columns', 100)
pd.set_option('display.notebook_repr_html', True)
#import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
# Displays the plots for us.
%matplotlib inline
```

```python
# Use this as a variable to load solutions: %load PATHTOSOLUTIONS/exercise1.py. It will be substituted in the code
# so do not worry if it disappears after you run the cell.
PATHTOSOLUTIONS = 'solutions'
```

## 1 - Review of  the  `numpy` Python library

In lab1 we learned about the `numpy` library [(documentation)](http://www.numpy.org/) and its fast array structure, called the `numpy array`.

```python
# import numpy
import numpy as np
```

```python
# make an array
my_array = np.array([1,4,9,16])
my_array
```

    array([ 1,  4,  9, 16])

```python
print(f'Size of my array: {my_array.size}, or length of my array: {len(my_array)}')
print (f'Shape of my array: {my_array.shape}')
```

    Size of my array: 4, or length of my array: 4
    Shape of my array: (4,)

#### Notice the way the shape appears in numpy arrays

- For a 1D array, .shape returns a tuple with 1 element (n,)
- For a 2D array, .shape returns a tuple with 2 elements (n,m)
- For a 3D array, .shape returns a tuple with 3 elements (n,m,p)

```python
# How to reshape a 1D array to a 2D
my_array.reshape(2,-1)
```

    array([[ 1,  4],
           [ 9, 16]])

Numpy arrays support the same operations as lists! Below we slice and iterate.

```python
print("array[2:4]:", my_array[2:4]) # A slice of the array

# Iterate over the array
for ele in my_array:
    print("element:", ele)
```

    array[2:4]: [ 9 16]
    element: 1
    element: 4
    element: 9
    element: 16

Remember `numpy` gains a lot of its efficiency from being **strongly typed** (all elements are of the same type, such as integer or floating point). If the elements of an array are of a different type, `numpy` will force them into the same type (the longest in terms of bytes)

```python
mixed = np.array([1, 2.3, 'eleni', True])
print(type(1), type(2.3), type('eleni'), type(True))
mixed # all elements will become strings
```

    <class 'int'> <class 'float'> <class 'str'> <class 'bool'>

    array(['1', '2.3', 'eleni', 'True'],
          dtype='<U32')

Next, we push ahead to two-dimensional arrays and begin to dive into some of the deeper aspects of `numpy`.

```python
# create a 2d-array by handing a list of lists
my_array2d = np.array([ [1, 2, 3, 4],
                        [5, 6, 7, 8],
                        [9, 10, 11, 12]
])

my_array2d
```

    array([[ 1,  2,  3,  4],
           [ 5,  6,  7,  8],
           [ 9, 10, 11, 12]])

### Array Slicing (a reminder...)

Numpy arrays can be sliced, and can be iterated over with loops.  Below is a schematic illustrating slicing two-dimensional arrays.

Notice that the list slicing syntax still works!
`array[2:,3]` says "in the array, get rows 2 through the end, column 3]"
`array[3,:]` says "in the array, get row 3, all columns".

### Pandas Slicing (a reminder...)

`.iloc` is by position (position is unique), `.loc` is by label (label is not unique)

```python
# import cast dataframe
cast = pd.read_csv('../data/cast.csv', encoding='utf_8')
cast.head()
```

      title
      year
      name
      type
      character
      n

      0
      Closet Monster
      2015
      Buffy #1
      actor
      Buffy 4
      31.0

      1
      Suuri illusioni
      1985
      Homo $
      actor
      Guests
      22.0

      2
      Battle of the Sexes
      2017
      $hutter
      actor
      Bobby Riggs Fan
      10.0

      3
      Secret in Their Eyes
      2015
      $hutter
      actor
      2002 Dodger Fan
      NaN

      4
      Steve Jobs
      2015
      $hutter
      actor
      1988 Opera House Patron
      NaN

```python
# get me rows 10 to 13 (python slicing style : exclusive of end)
cast.iloc[10:13]
```

      title
      year
      name
      type
      character
      n

      10
      When the Man Went South
      2014
      Taipaleti 'Atu'ake
      actor
      Two Palms - Ua'i Paame
      8.0

      11
      Little Angel (Angelita)
      2015
      Michael 'babeepower' Viera
      actor
      Chico
      9.0

      12
      Mixing Nia
      1998
      Michael 'babeepower' Viera
      actor
      Rapper
      NaN

```python
# get me columns 0 to 2 but all rows - use head()
cast.iloc[:, 0:2].head()
```

      title
      year

      0
      Closet Monster
      2015

      1
      Suuri illusioni
      1985

      2
      Battle of the Sexes
      2017

      3
      Secret in Their Eyes
      2015

      4
      Steve Jobs
      2015

```python
# get me rows 10 to 13 AND only columns 0 to 2
cast.iloc[10:13, 0:2]
```

      title
      year

      10
      When the Man Went South
      2014

      11
      Little Angel (Angelita)
      2015

      12
      Mixing Nia
      1998

```python
# COMPARE: get me rows 10 to 13 (pandas slicing style : inclusive of end)
cast.loc[10:13]
```

      title
      year
      name
      type
      character
      n

      10
      When the Man Went South
      2014
      Taipaleti 'Atu'ake
      actor
      Two Palms - Ua'i Paame
      8.0

      11
      Little Angel (Angelita)
      2015
      Michael 'babeepower' Viera
      actor
      Chico
      9.0

      12
      Mixing Nia
      1998
      Michael 'babeepower' Viera
      actor
      Rapper
      NaN

      13
      The Replacements
      2000
      Steven 'Bear'Boyd
      actor
      Defensive Tackle - Washington Sentinels
      NaN

```python
# give me columns 'year' and 'type' by label but only for rows 5 to 10
cast.loc[5:10,['year','type']]
```

      year
      type

      5
      2015
      actor

      6
      2015
      actor

      7
      2009
      actor

      8
      2014
      actor

      9
      2014
      actor

      10
      2014
      actor

```python

```

## 2 - Plotting with matplotlib and beyond

`matplotlib` is a very powerful `python` library for making scientific plots.

We will not focus too much on the internal aspects of `matplotlib` in today's lab. There are many excellent tutorials out there for `matplotlib`.  For example,
* [`matplotlib` homepage](https://matplotlib.org/)
* [`matplotlib` tutorial](https://github.com/matplotlib/AnatomyOfMatplotlib)

Conveying your findings convincingly is an absolutely crucial part of any analysis. Therefore, you must be able to write well and make compelling visuals.  Creating informative visuals is an involved process and we won't cover that in this lab.  However, part of creating informative data visualizations means generating *readable* figures.  If people can't read your figures or have a difficult time interpreting them, they won't understand the results of your work.  Here are some non-negotiable commandments for any plot:
* Label $x$ and $y$ axes
* Axes labels should be informative
* Axes labels should be large enough to read
* Make tick labels large enough
* Include a legend if necessary
* Include a title if necessary
* Use appropriate line widths
* Use different line styles for different lines on the plot
* Use different markers for different lines

There are other important elements, but that list should get you started on your way.

We will work with `matplotlib` and `seaborn` for plotting in this class.  `matplotlib` is a very powerful `python` library for making scientific plots.  `seaborn` is a little more specialized in that it was developed for statistical data visualization.  We will cover some `seaborn` later in class. In the meantime you can look at the [seaborn documentation](https://seaborn.pydata.org)

First, let's generate some data.

#### Let's plot some functions

We will use the following three functions to make some plots:

* Logistic function:
  \begin{align*}
    f\left(z\right) = \dfrac{1}{1 + be^{-az}}
  \end{align*}
  where $a$ and $b$ are parameters.
* Hyperbolic tangent:
  \begin{align*}
    g\left(z\right) = b\tanh\left(az\right) + c
  \end{align*}
  where $a$, $b$, and $c$ are parameters.
* Rectified Linear Unit:
  \begin{align*}
    h\left(z\right) =
    \left\{
      \begin{array}{lr}
        z, \quad z > 0 \\
        \epsilon z, \quad z\leq 0
      \end{array}
    \right.
  \end{align*}
  where $\epsilon < 0$ is a small, positive parameter.

You are given the code for the first two functions.  Notice that $z$ is passed in as a `numpy` array and that the functions are returned as `numpy` arrays.  Parameters are passed in as floats.

You should write a function to compute the rectified linear unit.  The input should be a `numpy` array for $z$ and a positive float for $\epsilon$.

```python
import numpy as np

def logistic(z: np.ndarray, a: float, b: float) -> np.ndarray:
    """ Compute logistic function
      Inputs:
         a: exponential parameter
         b: exponential prefactor
         z: numpy array; domain
      Outputs:
         f: numpy array of floats, logistic function
    """

    den = 1.0 + b * np.exp(-a * z)
    return 1.0 / den

def stretch_tanh(z: np.ndarray, a: float, b: float, c: float) -> np.ndarray:
    """ Compute stretched hyperbolic tangent
      Inputs:
         a: horizontal stretch parameter (a>1 implies a horizontal squish)
         b: vertical stretch parameter
         c: vertical shift parameter
         z: numpy array; domain
      Outputs:
         g: numpy array of floats, stretched tanh
    """
    return b * np.tanh(a * z) + c

def relu(z: np.ndarray, eps: float = 0.01) -> np.ndarray:
    """ Compute rectificed linear unit
      Inputs:
         eps: small positive parameter
         z: numpy array; domain
      Outputs:
         h: numpy array; relu
    """
    return np.fmax(z, eps * z)
```

Now let's make some plots.  First, let's just warm up and plot the logistic function.

```python
x = np.linspace(-5.0, 5.0, 100) # Equally spaced grid of 100 pts between -5 and 5

f = logistic(x, 1.0, 1.0) # Generate data
```

```python
plt.plot(x, f)
plt.xlabel('x')
plt.ylabel('f')
plt.title('Logistic Function')
plt.grid(True)
```

#### Figures with subplots

Let's start thinking about the plots as objects. We have the `figure` object which is like a matrix of smaller plots named `axes`. You can use array notation when handling it.

```python
fig, ax = plt.subplots(1,1) # Get figure and axes objects

ax.plot(x, f) # Make a plot

# Create some labels
ax.set_xlabel('x')
ax.set_ylabel('f')
ax.set_title('Logistic Function')

# Grid
ax.grid(True)
```

Wow, it's *exactly* the same plot!  Notice, however, the use of `ax.set_xlabel()` instead of `plt.xlabel()`.  The difference is tiny, but you should be aware of it.  I will use this plotting syntax from now on.

What else do we need to do to make this figure better?  Here are some options:
* Make labels bigger!
* Make line fatter
* Make tick mark labels bigger
* Make the grid less pronounced
* Make figure bigger

Let's get to it.

```python
fig, ax = plt.subplots(1,1, figsize=(10,6)) # Make figure bigger

# Make line plot
ax.plot(x, f, lw=4)

# Update ticklabel size
ax.tick_params(labelsize=24)

# Make labels
ax.set_xlabel(r'$x$', fontsize=24) # Use TeX for mathematical rendering
ax.set_ylabel(r'$f(x)$', fontsize=24) # Use TeX for mathematical rendering
ax.set_title('Logistic Function', fontsize=24)

ax.grid(True, lw=1.5, ls='--', alpha=0.75)
```

Notice:
* `lw` stands for `linewidth`.  We could also write `ax.plot(x, f, linewidth=4)`
* `ls` stands for `linestyle`.
* `alpha` stands for transparency.

The only thing remaining to do is to change the $x$ limits.  Clearly these should go from $-5$ to $5$.

```python
#fig.savefig('logistic.png')

# Put this in a markdown cell and uncomment this to check what you saved.
#
```

#### Resources
If you want to see all the styles available, please take a look at the documentation.
* [Line styles](https://matplotlib.org/2.0.1/api/lines_api.html#matplotlib.lines.Line2D.set_linestyle)
* [Marker styles](https://matplotlib.org/2.0.1/api/markers_api.html#module-matplotlib.markers)
* [Everything you could ever want](https://matplotlib.org/2.0.1/api/lines_api.html#matplotlib.lines.Line2D.set_marker)

We haven't discussed it yet, but you can also put a legend on a figure.  You'll do that in the next exercise.  Here are some additional resources:
* [Legend](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.legend.html)
* [Grid](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.grid.html)

`ax.legend(loc='best', fontsize=24);`

Exercise

Do the following:
* Make a figure with the logistic function, hyperbolic tangent, and rectified linear unit.
* Use different line styles for each plot
* Put a legend on your figure

Here's an example of a figure:

```python
# your code here

# First get the data
f = logistic(x, 2.0, 1.0)
g = stretch_tanh(x, 2.0, 0.5, 0.5)
h = relu(x)

fig, ax = plt.subplots(1,1, figsize=(10,6)) # Create figure object

# Make actual plots
# (Notice the label argument!)
ax.plot(x, f, lw=4, ls='-', label=r'$L(x;1)$')
ax.plot(x, g, lw=4, ls='--', label=r'$\tanh(2x)$')
ax.plot(x, h, lw=4, ls='-.', label=r'$relu(x; 0.01)$')

# Make the tick labels readable
ax.tick_params(labelsize=24)

# Set axes limits to make the scale nice
ax.set_xlim(x.min(), x.max())
ax.set_ylim(h.min(), 1.1)

# Make readable labels
ax.set_xlabel(r'$x$', fontsize=24)
ax.set_ylabel(r'$h(x)$', fontsize=24)
ax.set_title('Activation Functions', fontsize=24)

# Set up grid
ax.grid(True, lw=1.75, ls='--', alpha=0.75)

# Put legend on figure
ax.legend(loc='best', fontsize=24);

fig.savefig('../images/nice_plots.png')
```

Exercise

These figures look nice in the plot and it makes sense for comparison. Now let's put the 3 different figures in separate plots.

* Make a separate plot for each figure and line them up on the same row.

```python
# your code here

```

```python
# %load solutions/three_subplots.py
```

Exercise

* Make a grid of 2 x 3 separate plots, 3 will be empty. Just plot the functions and do not worry about cosmetics. We just want you ro see the functionality.

```python
# your code here

```

```python
# %load solutions/six_subplots.py
```

## 3 - Simple Linear Regression

Linear regression and its many extensions are a workhorse of the statistics and data science community, both in application and as a reference point for other models. Most of the major concepts in machine learning can be and often are discussed in terms of various linear regression models. Thus, this section will introduce you to building and fitting linear regression models and some of the process behind it, so that you can 1) fit models to data you encounter 2) experiment with different kinds of linear regression and observe their effects 3) see some of the technology that makes regression models work.

### Linear regression with a toy dataset
We first examine a toy problem, focusing our efforts on fitting a linear model to a small dataset with three observations.  Each observation consists of one predictor $x_i$ and one response $y_i$ for $i = 1, 2, 3$,

\begin{align*}
(x , y) = \{(x_1, y_1), (x_2, y_2), (x_3, y_3)\}.
\end{align*}

To be very concrete, let's set the values of the predictors and responses.

\begin{equation*}
(x , y) = \{(1, 2), (2, 2), (3, 4)\}
\end{equation*}

There is no line of the form $\beta_0 + \beta_1 x = y$ that passes through all three observations, since the data are not collinear. Thus our aim is to find the line that best fits these observations in the *least-squares sense*, as discussed in lecture.

Exercise (for home)

* Make two numpy arrays out of this data, x_train and y_train
* Check the dimentions of these arrays
* Try to reshape them into a different shape
* Make points into a very simple scatterplot
* Make a better scatterplot

```python
# your code here
```

```python
# solution
x_train = np.array([1,2,3])
y_train = np.array([2,3,6])
type(x_train)
```

    numpy.ndarray

```python
x_train.shape
```

    (3,)

```python
x_train = x_train.reshape(3,1)
x_train.shape
```

    (3, 1)

```python
xx = np.array([[1,3,5],[6,2,1]])
xx.shape
```

    (2, 3)

```python
xx = xx.reshape(3,-1)
xx
```

    array([[1, 3],
           [5, 6],
           [2, 1]])

```python
# %load solutions/simple_scatterplot.py
# Make a simple scatterplot
plt.scatter(x_train,y_train)

# check dimensions
print(x_train.shape,y_train.shape)

```

```python
# %load solutions/nice_scatterplot.py
def nice_scatterplot(x, y, title):
    # font size
    f_size = 18

    # make the figure
    fig, ax = plt.subplots(1,1, figsize=(8,5)) # Create figure object

    # set axes limits to make the scale nice
    ax.set_xlim(np.min(x)-1, np.max(x) + 1)
    ax.set_ylim(np.min(y)-1, np.max(y) + 1)

    # adjust size of tickmarks in axes
    ax.tick_params(labelsize = f_size)

    # remove tick labels
    ax.tick_params(labelbottom=False,  bottom=False)

    # adjust size of axis label
    ax.set_xlabel(r'$x$', fontsize = f_size)
    ax.set_ylabel(r'$y$', fontsize = f_size)

    # set figure title label
    ax.set_title(title, fontsize = f_size)

    # you may set up grid with this
    ax.grid(True, lw=1.75, ls='--', alpha=0.15)

    # make actual plot (Notice the label argument!)
    #ax.scatter(x, y, label=r'$my points$')
    #ax.scatter(x, y, label='$my points$')
    ax.scatter(x, y, label=r'$my\,points$')
    ax.legend(loc='best', fontsize = f_size);

    return ax

nice_scatterplot(x_train, y_train, 'hello nice plot')

```

#### Formulae
Linear regression is special among the models we study because it can be solved explicitly. While most other models (and even some advanced versions of linear regression) must be solved itteratively, linear regression has a formula where you can simply plug in the data.

For the single predictor case it is:
    \begin{align}
      \beta_1 &= \frac{\sum_{i=1}^n{(x_i-\bar{x})(y_i-\bar{y})}}{\sum_{i=1}^n{(x_i-\bar{x})^2}}\\
      \beta_0 &= \bar{y} - \beta_1\bar{x}\
    \end{align}

Where $\bar{y}$ and $\bar{x}$ are the mean of the y values and the mean of the x values, respectively.

From the re-aranged second equation we can see that the best-fit line  passes through $(\bar{x},\bar{y})$, the center of mass of the data

From any of the first equations, we can see that the slope of the line has to do with whether or not an x value that is above/below the center of mass is typically paired with a y value that is likewise above/below, or typically paired with one that is opposite.

###  Building a model from scratch
In this part, we will solve the equations for simple linear regression and find the best fit solution to our toy problem.

The snippets of code below implement the linear regression equations on the observed predictors and responses, which we'll call the training data set.  Let's walk through the code.

We have to reshape our arrrays to 2D. We will see later why.

Exercise

* make an array with shape (2,3)
* reshape it to a size that you want

```python
# your code here
xx = np.array([[1,2,3],[4,6,8]])
xxx = xx.reshape(-1,2)
xxx.shape
```

    (3, 2)

```python
# Reshape to be a proper 2D array
x_train = x_train.reshape(x_train.shape[0], 1)
y_train = y_train.reshape(y_train.shape[0], 1)

print(x_train.shape)
```

    (3, 1)

```python
# first, compute means
y_bar = np.mean(y_train)
x_bar = np.mean(x_train)

# build the two terms
numerator = np.sum( (x_train - x_bar)*(y_train - y_bar) )
denominator = np.sum((x_train - x_bar)**2)

print(numerator.shape, denominator.shape) #check shapes
```

    () ()

* Why the empty brackets? (The numerator and denominator are scalars, as expected.)

```python
#slope beta1
beta_1 = numerator/denominator

#intercept beta0
beta_0 = y_bar - beta_1*x_bar

print("The best-fit line is {0:3.2f} + {1:3.2f} * x".format(beta_0, beta_1))
print(f'The best fit is {beta_0}')
```

    The best-fit line is -0.33 + 2.00 * x
    The best fit is -0.3333333333333335

Exercise

Turn the code from the above cells into a function called `simple_linear_regression_fit`, that inputs the training data and returns `beta0` and `beta1`.

To do this, copy and paste the code from the above cells below and adjust the code as needed, so that the training data becomes the input and the betas become the output.

```python
def simple_linear_regression_fit(x_train: np.ndarray, y_train: np.ndarray) -> np.ndarray:

    return
```

Check your function by calling it with the training data from above and printing out the beta values.

```python
# Your code here
```

```python
# %load solutions/simple_linear_regression_fit.py
def simple_linear_regression_fit(x_train: np.ndarray, y_train: np.ndarray) -> np.ndarray:
    """
    Inputs:
    x_train: a (num observations by 1) array holding the values of the predictor variable
    y_train: a (num observations by 1) array holding the values of the response variable

    Returns:
    beta_vals:  a (num_features by 1) array holding the intercept and slope coeficients
    """

    # Check input array sizes
    if len(x_train.shape) < 2:
        print("Reshaping features array.")
        x_train = x_train.reshape(x_train.shape[0], 1)

    if len(y_train.shape) < 2:
        print("Reshaping observations array.")
        y_train = y_train.reshape(y_train.shape[0], 1)

    # first, compute means
    y_bar = np.mean(y_train)
    x_bar = np.mean(x_train)

    # build the two terms
    numerator = np.sum( (x_train - x_bar)*(y_train - y_bar) )
    denominator = np.sum((x_train - x_bar)**2)

    #slope beta1
    beta_1 = numerator/denominator

    #intercept beta0
    beta_0 = y_bar - beta_1*x_bar

    return np.array([beta_0,beta_1])

```

* Let's run this function and see the coefficients

```python
x_train = np.array([1 ,2, 3])
y_train = np.array([2, 2, 4])

betas = simple_linear_regression_fit(x_train, y_train)

beta_0 = betas[0]
beta_1 = betas[1]

print("The best-fit line is {0:8.6f} + {1:8.6f} * x".format(beta_0, beta_1))
```

    Reshaping features array.
    Reshaping observations array.
    The best-fit line is 0.666667 + 1.000000 * x

Exercise

* Do the values of `beta0` and `beta1` seem reasonable?
* Plot the training data using a scatter plot.
* Plot the best fit line with `beta0` and `beta1` together with the training data.

```python
# Your code here
```

```python
# %load solutions/best_fit_scatterplot.py
fig_scat, ax_scat = plt.subplots(1,1, figsize=(10,6))

# Plot best-fit line
x_train = np.array([[1, 2, 3]]).T

best_fit = beta_0 + beta_1 * x_train

ax_scat.scatter(x_train, y_train, s=300, label='Training Data')
ax_scat.plot(x_train, best_fit, ls='--', label='Best Fit Line')

ax_scat.set_xlabel(r'$x_{train}$')
ax_scat.set_ylabel(r'$y$');

```

The values of `beta0` and `beta1` seem roughly reasonable.  They capture the positive correlation.  The line does appear to be trying to get as close as possible to all the points.

## 4 - Building a model with `statsmodels` and `sklearn`

Now that we can concretely fit the training data from scratch, let's learn two `python` packages to do it all for us:
* [statsmodels](http://www.statsmodels.org/stable/regression.html) and
* [scikit-learn (sklearn)](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html).

Our goal  is to show how to implement simple linear regression with these packages.  For an important sanity check, we compare the $\beta$ values from `statsmodels` and `sklearn` to the $\beta$ values that we found from above with our own implementation.

For the purposes of this lab, `statsmodels` and `sklearn` do the same thing.  More generally though, `statsmodels` tends to be easier for inference \[finding the values of the slope and intercept and dicussing uncertainty in those values\], whereas `sklearn` has machine-learning algorithms and is better for prediction \[guessing y values for a given x value\]. (Note that both packages make the same guesses, it's just a question of which activity they provide more support for.

**Note:** `statsmodels` and `sklearn` are different packages!  Unless we specify otherwise, you can use either one.

Below is the code for `statsmodels`.  `Statsmodels` does not by default include the column of ones in the $X$ matrix, so we include it manually with `sm.add_constant`.

```python
import statsmodels.api as sm
```

```python
# create the X matrix by appending a column of ones to x_train
X = sm.add_constant(x_train)

# this is the same matrix as in our scratch problem!
print(X)

# build the OLS model (ordinary least squares) from the training data
print(X.shape)
print(y_train.shape)
toyregr_sm = sm.OLS(y_train, X)

# do the fit and save regression info (parameters, etc) in results_sm
results_sm = toyregr_sm.fit()

# pull the beta parameters out from results_sm
beta0_sm = results_sm.params[0]
beta1_sm = results_sm.params[1]

print(f'The regression coef from statsmodels are: beta_0 = {beta0_sm:8.6f} and beta_1 = {beta1_sm:8.6f}')
```

    [[ 1.  1.]
     [ 1.  2.]
     [ 1.  3.]]
    (3, 2)
    (3,)
    The regression coef from statsmodels are: beta_0 = 0.666667 and beta_1 = 1.000000

Besides the beta parameters, `results_sm` contains a ton of other potentially useful information.

```python
import warnings
warnings.filterwarnings('ignore')
print(results_sm.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                      y   R-squared:                       0.750
    Model:                            OLS   Adj. R-squared:                  0.500
    Method:                 Least Squares   F-statistic:                     3.000
    Date:                Mon, 23 Sep 2019   Prob (F-statistic):              0.333
    Time:                        21:30:57   Log-Likelihood:                -2.0007
    No. Observations:                   3   AIC:                             8.001
    Df Residuals:                       1   BIC:                             6.199
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.6667      1.247      0.535      0.687     -15.181      16.514
    x1             1.0000      0.577      1.732      0.333      -6.336       8.336
    ==============================================================================
    Omnibus:                          nan   Durbin-Watson:                   3.000
    Prob(Omnibus):                    nan   Jarque-Bera (JB):                0.531
    Skew:                          -0.707   Prob(JB):                        0.767
    Kurtosis:                       1.500   Cond. No.                         6.79
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Now let's turn our attention to the `sklearn` library.

```python
from sklearn import linear_model
```

```python
# build the least squares model
toyregr = linear_model.LinearRegression()

# save regression info (parameters, etc) in results_skl
results = toyregr.fit(x_train, y_train)

# pull the beta parameters out from results_skl
beta0_skl = toyregr.intercept_
beta1_skl = toyregr.coef_[0]

print("The regression coefficients from the sklearn package are: beta_0 = {0:8.6f} and beta_1 = {1:8.6f}".format(beta0_skl, beta1_skl))
```

    The regression coefficients from the sklearn package are: beta_0 = 0.666667 and beta_1 = 1.000000

We should feel pretty good about ourselves now, and we're ready to move on to a real problem!

### The `scikit-learn` library and the shape of things

Before diving into a "real" problem, let's discuss more of the details of `sklearn`.

`Scikit-learn` is the main `Python` machine learning library. It consists of many learners which can learn models from data, as well as a lot of utility functions such as `train_test_split()`.

Use the following to add the library into your code:

```python
import sklearn
```

In `scikit-learn`, an **estimator** is a Python object that implements the methods `fit(X, y)` and `predict(T)`

Let's see the structure of `scikit-learn` needed to make these fits. `fit()` always takes two arguments:
```python
estimator.fit(Xtrain, ytrain)
```
We will consider two estimators in this lab: `LinearRegression` and `KNeighborsRegressor`.

It is very important to understand that `Xtrain` must be in the form of a **2x2 array** with each row corresponding to one sample, and each column corresponding to the feature values for that sample.

`ytrain` on the other hand is a simple array of responses.  These are continuous for regression problems.

### Practice with `sklearn` and a real dataset
We begin by loading up the `mtcars` dataset. This data was extracted from the 1974 Motor Trend US magazine, and comprises of fuel consumption and 10 aspects of automobile design and performance for 32 automobiles (1973–74 models). We will load this data to a dataframe with 32 observations on 11 (numeric) variables. Here is an explanation of the features:

- `mpg` is Miles/(US) gallon
- `cyl` is Number of cylinders,
- `disp` is	Displacement (cu.in.),
- `hp` is	Gross horsepower,
- `drat` is	Rear axle ratio,
- `wt` is the Weight (1000 lbs),
- `qsec` is 1/4 mile time,
- `vs` is Engine (0 = V-shaped, 1 = straight),
- `am` is Transmission (0 = automatic, 1 = manual),
- `gear` is the Number of forward gears,
- `carb` is	Number of carburetors.

```python
import pandas as pd

#load mtcars
dfcars = pd.read_csv("../data/mtcars.csv")
dfcars.head()
```

      Unnamed: 0
      mpg
      cyl
      disp
      hp
      drat
      wt
      qsec
      vs
      am
      gear
      carb

      0
      Mazda RX4
      21.0
      6
      160.0
      110
      3.90
      2.620
      16.46
      0
      1
      4
      4

      1
      Mazda RX4 Wag
      21.0
      6
      160.0
      110
      3.90
      2.875
      17.02
      0
      1
      4
      4

      2
      Datsun 710
      22.8
      4
      108.0
      93
      3.85
      2.320
      18.61
      1
      1
      4
      1

      3
      Hornet 4 Drive
      21.4
      6
      258.0
      110
      3.08
      3.215
      19.44
      1
      0
      3
      1

      4
      Hornet Sportabout
      18.7
      8
      360.0
      175
      3.15
      3.440
      17.02
      0
      0
      3
      2

```python
# Fix the column title
dfcars = dfcars.rename(columns={"Unnamed: 0":"car name"})
dfcars.head()
```

      car name
      mpg
      cyl
      disp
      hp
      drat
      wt
      qsec
      vs
      am
      gear
      carb

      0
      Mazda RX4
      21.0
      6
      160.0
      110
      3.90
      2.620
      16.46
      0
      1
      4
      4

      1
      Mazda RX4 Wag
      21.0
      6
      160.0
      110
      3.90
      2.875
      17.02
      0
      1
      4
      4

      2
      Datsun 710
      22.8
      4
      108.0
      93
      3.85
      2.320
      18.61
      1
      1
      4
      1

      3
      Hornet 4 Drive
      21.4
      6
      258.0
      110
      3.08
      3.215
      19.44
      1
      0
      3
      1

      4
      Hornet Sportabout
      18.7
      8
      360.0
      175
      3.15
      3.440
      17.02
      0
      0
      3
      2

```python
dfcars.shape
```

    (32, 12)

#### Searching for values: how many cars have 4 gears?

```python
len(dfcars[dfcars.gear == 4].drop_duplicates(subset='car name', keep='first'))
```

    12

Next, let's split the dataset into a training set and test set.

```python
# split into training set and testing set
from sklearn.model_selection import train_test_split

#set random_state to get the same split every time
traindf, testdf = train_test_split(dfcars, test_size=0.2, random_state=42)
```

```python
# testing set is around 20% of the total data; training set is around 80%
print("Shape of full dataset is: {0}".format(dfcars.shape))
print("Shape of training dataset is: {0}".format(traindf.shape))
print("Shape of test dataset is: {0}".format(testdf.shape))
```

    Shape of full dataset is: (32, 12)
    Shape of training dataset is: (25, 12)
    Shape of test dataset is: (7, 12)

Now we have training and test data.  We still need to select a predictor and a response from this dataset.  Keep in mind that we need to choose the predictor and response from both the training and test set.  You will do this in the exercises below.  However, we provide some starter code for you to get things going.

```python
traindf.head()
```

      car name
      mpg
      cyl
      disp
      hp
      drat
      wt
      qsec
      vs
      am
      gear
      carb

      25
      Fiat X1-9
      27.3
      4
      79.0
      66
      4.08
      1.935
      18.90
      1
      1
      4
      1

      12
      Merc 450SL
      17.3
      8
      275.8
      180
      3.07
      3.730
      17.60
      0
      0
      3
      3

      0
      Mazda RX4
      21.0
      6
      160.0
      110
      3.90
      2.620
      16.46
      0
      1
      4
      4

      4
      Hornet Sportabout
      18.7
      8
      360.0
      175
      3.15
      3.440
      17.02
      0
      0
      3
      2

      16
      Chrysler Imperial
      14.7
      8
      440.0
      230
      3.23
      5.345
      17.42
      0
      0
      3
      4

```python
# Extract the response variable that we're interested in
y_train = traindf.mpg
y_train
```

    25    27.3
    12    17.3
    0     21.0
    4     18.7
    16    14.7
    5     18.1
    13    15.2
    11    16.4
    23    13.3
    1     21.0
    2     22.8
    26    26.0
    3     21.4
    21    15.5
    27    30.4
    22    15.2
    18    30.4
    31    21.4
    20    21.5
    7     24.4
    10    17.8
    14    10.4
    28    15.8
    19    33.9
    6     14.3
    Name: mpg, dtype: float64

Exercise

Use slicing to get the same vector `y_train`

----

Now, notice the shape of `y_train`.

```python
y_train.shape, type(y_train)
```

    ((25,), pandas.core.series.Series)

### Array reshape
This is a 1D array as should be the case with the **Y** array.  Remember, `sklearn` requires a 2D array only for the predictor array.  You will have to pay close attention to this in the exercises later. `Sklearn` doesn't care too much about the shape of `y_train`.

The whole reason we went through that whole process was to show you how to reshape your data into the correct format.

**IMPORTANT:** Remember that your response variable `ytrain` can be a vector but your predictor variable `xtrain` ***must*** be an array!

## 5 - Example: Simple linear regression with automobile data
We will now use `sklearn` to predict automobile mileage per gallon (mpg) and evaluate these predictions. We already loaded the data and split them into a training set and a test set.

We need to choose the variables that we think will be good predictors for the dependent variable `mpg`.

Exercise in pairs

* Pick one variable to use as a predictor for simple linear regression.  Discuss your reasons with the person next to you.
* Justify your choice with some visualizations.
* Is there a second variable you'd like to use? For example, we're not doing multiple linear regression here, but if we were, is there another variable you'd like to include if we were using two predictors?

```python
x_wt = dfcars.wt
x_wt.shape
```

```python
# Your code here

```

```python
# %load solutions/cars_simple_EDA.py
```

Exercise

* Use `sklearn` to fit the training data using simple linear regression.
* Use the model to make mpg predictions on the test set.
* Plot the data and the prediction.
* Print out the mean squared error for the training set and the test set and compare.

```python
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

dfcars = pd.read_csv("../data/mtcars.csv")
dfcars = dfcars.rename(columns={"Unnamed: 0":"name"})

dfcars.head()
```

      name
      mpg
      cyl
      disp
      hp
      drat
      wt
      qsec
      vs
      am
      gear
      carb

      0
      Mazda RX4
      21.0
      6
      160.0
      110
      3.90
      2.620
      16.46
      0
      1
      4
      4

      1
      Mazda RX4 Wag
      21.0
      6
      160.0
      110
      3.90
      2.875
      17.02
      0
      1
      4
      4

      2
      Datsun 710
      22.8
      4
      108.0
      93
      3.85
      2.320
      18.61
      1
      1
      4
      1

      3
      Hornet 4 Drive
      21.4
      6
      258.0
      110
      3.08
      3.215
      19.44
      1
      0
      3
      1

      4
      Hornet Sportabout
      18.7
      8
      360.0
      175
      3.15
      3.440
      17.02
      0
      0
      3
      2

```python
traindf, testdf = train_test_split(dfcars, test_size=0.2, random_state=42)

y_train = np.array(traindf.mpg)
X_train = np.array(traindf.wt)
X_train = X_train.reshape(X_train.shape[0], 1)
```

```python
y_test = np.array(testdf.mpg)
X_test = np.array(testdf.wt)
X_test = X_test.reshape(X_test.shape[0], 1)
```

```python
# Let's take another look at our data
dfcars.head()
```

      name
      mpg
      cyl
      disp
      hp
      drat
      wt
      qsec
      vs
      am
      gear
      carb

      0
      Mazda RX4
      21.0
      6
      160.0
      110
      3.90
      2.620
      16.46
      0
      1
      4
      4

      1
      Mazda RX4 Wag
      21.0
      6
      160.0
      110
      3.90
      2.875
      17.02
      0
      1
      4
      4

      2
      Datsun 710
      22.8
      4
      108.0
      93
      3.85
      2.320
      18.61
      1
      1
      4
      1

      3
      Hornet 4 Drive
      21.4
      6
      258.0
      110
      3.08
      3.215
      19.44
      1
      0
      3
      1

      4
      Hornet Sportabout
      18.7
      8
      360.0
      175
      3.15
      3.440
      17.02
      0
      0
      3
      2

```python
# And out train and test sets
y_train.shape, X_train.shape
```

    ((25,), (25, 1))

```python
y_test.shape, X_test.shape
```

    ((7,), (7, 1))

```python
#create linear model
regression = LinearRegression()

#fit linear model
regression.fit(X_train, y_train)

predicted_y = regression.predict(X_test)

r2 = regression.score(X_test, y_test)
print(f'R^2 = {r2:.5}')
```

    R^2 = 0.68798

```python
print(regression.score(X_train, y_train))

print(mean_squared_error(predicted_y, y_test))
print(mean_squared_error(y_train, regression.predict(X_train)))

print('Coefficients: \n', regression.coef_[0], regression.intercept_)
```

    0.770137990979
    12.4759856599
    7.77369776639
    Coefficients:
     -5.33694140056 36.9373103135

```python
fig, ax = plt.subplots(1,1, figsize=(10,6))
ax.plot(y_test, predicted_y, 'o')
grid = np.linspace(np.min(dfcars.mpg), np.max(dfcars.mpg), 100)
ax.plot(grid, grid, color="black") # 45 degree line
ax.set_xlabel("actual y")
ax.set_ylabel("predicted y")

fig1, ax1 = plt.subplots(1,1, figsize=(10,6))
ax1.plot(dfcars.wt, dfcars.mpg, 'o')
xgrid = np.linspace(np.min(dfcars.wt), np.max(dfcars.wt), 100)
ax1.plot(xgrid, regression.predict(xgrid.reshape(100, 1)))
```

## 6 - $k$-nearest neighbors

Now that you're familiar with `sklearn`, you're ready to do a KNN regression.

Sklearn's regressor is called `sklearn.neighbors.KNeighborsRegressor`. Its main parameter is the `number of nearest neighbors`. There are other parameters such as the distance metric (default for 2 order is the Euclidean distance). For a list of all the parameters see the [Sklearn kNN Regressor Documentation](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsRegressor.html).

Let's use $5$ nearest neighbors.

```python
# Import the library
from sklearn.neighbors import KNeighborsRegressor
```

```python
# Set number of neighbors
k = 5
knnreg = KNeighborsRegressor(n_neighbors=k)
```

```python
# Fit the regressor - make sure your numpy arrays are the right shape
knnreg.fit(X_train, y_train)

# Evaluate the outcome on the train set using R^2
r2_train = knnreg.score(X_train, y_train)

# Print results
print(f'kNN model with {k} neighbors gives R^2 on the train set: {r2_train:.5}')
```

```python
knnreg.predict(X_test)
```

Exercise

Calculate and print the $R^{2}$ score on the test set

```python
# Your code here
```

Not so good? Lets vary the number of neighbors and see what we get.

```python
# Make our lives easy by storing the different regressors in a dictionary
regdict = {}

# Make our lives easier by entering the k values from a list
k_list = [1, 2, 4, 15]

# Do a bunch of KNN regressions
for k in k_list:
    knnreg = KNeighborsRegressor(n_neighbors=k)
    knnreg.fit(X_train, y_train)
    # Store the regressors in a dictionary
    regdict[k] = knnreg

# Print the dictionary to see what we have
regdict
```

Now let's plot all the k values in same plot.

```python
fig, ax = plt.subplots(1,1, figsize=(10,6))

ax.plot(dfcars.wt, dfcars.mpg, 'o', label="data")

xgrid = np.linspace(np.min(dfcars.wt), np.max(dfcars.wt), 100)

# let's unpack the dictionary to its elements (items) which is the k and Regressor
for k, regressor in regdict.items():
    predictions = regressor.predict(xgrid.reshape(-1,1))
    ax.plot(xgrid, predictions, label="{}-NN".format(k))

ax.legend();
```

Exercise

Explain what you see in the graph. **Hint** Notice how the $1$-NN goes through every point on the training set but utterly fails elsewhere.

Lets look at the scores on the training set.

```python
ks = range(1, 15) # Grid of k's
scores_train = [] # R2 scores
for k in ks:
    # Create KNN model
    knnreg = KNeighborsRegressor(n_neighbors=k)

    # Fit the model to training data
    knnreg.fit(X_train, y_train)

    # Calculate R^2 score
    score_train = knnreg.score(X_train, y_train)
    scores_train.append(score_train)

# Plot
fig, ax = plt.subplots(1,1, figsize=(12,8))
ax.plot(ks, scores_train,'o-')
ax.set_xlabel(r'$k$')
ax.set_ylabel(r'$R^{2}$')
```

Exercise

* Why do we get a perfect $R^2$ at k=1 for the training set?
* Make the same plot as above on the *test* set.
* What is the best $k$?

```python
# Your code here

```

```python
# %load solutions/knn_regression.py
```

```python
# solution to previous exercise
r2_test = knnreg.score(X_test, y_test)
print(f'kNN model with {k} neighbors gives R^2 on the test set: {r2_test:.5}')
```


```python
!pip install qiskit
```

    Collecting qiskit
      Downloading https://files.pythonhosted.org/packages/8b/b6/9e38d181beb9dd07748cb36eb37ba5dfd43c34e7e1cab0afcd15e949c306/qiskit-0.23.2.tar.gz
    Collecting qiskit-terra==0.16.1
      Downloading https://files.pythonhosted.org/packages/66/91/813c0b986fa953d09ff58c74a7948bf9d52c64497319746bb4a1bb0e7f3a/qiskit_terra-0.16.1-cp36-cp36m-manylinux2010_x86_64.whl (8.5MB)
         |████████████████████████████████| 8.5MB 6.2MB/s
    Collecting qiskit-aer==0.7.2
      Downloading https://files.pythonhosted.org/packages/9b/5a/caa85b889de9aa4250fff68e76ad82fd2916f63b1eb3cced1c5419efe215/qiskit_aer-0.7.2-cp36-cp36m-manylinux2010_x86_64.whl (17.6MB)
         |████████████████████████████████| 17.6MB 257kB/s
    Collecting qiskit-ibmq-provider==0.11.1
      Downloading https://files.pythonhosted.org/packages/32/b9/f99bec4fdc4dec234d8a85a8da378750441a203c614be35353f5e8738316/qiskit_ibmq_provider-0.11.1-py3-none-any.whl (195kB)
         |████████████████████████████████| 204kB 41.0MB/s
    Collecting qiskit-ignis==0.5.1
      Downloading https://files.pythonhosted.org/packages/f7/42/5e151db90ebeb0d08d0f799f72c1922cded76065b03c4ca4fc11ec0085ce/qiskit_ignis-0.5.1-py3-none-any.whl (204kB)
         |████████████████████████████████| 204kB 47.2MB/s
    Collecting qiskit-aqua==0.8.1
      Downloading https://files.pythonhosted.org/packages/4d/5f/41abad4b0375effe79a51a20130da6c18bb2101c257366aab0f300197e53/qiskit_aqua-0.8.1-py3-none-any.whl (2.1MB)
         |████████████████████████████████| 2.1MB 31.8MB/s
    Requirement already satisfied: psutil>=5 in /usr/local/lib/python3.6/dist-packages (from qiskit-terra==0.16.1->qiskit) (5.4.8)
    Collecting python-constraint>=1.4
      Downloading https://files.pythonhosted.org/packages/37/8b/5f1bc2734ca611943e1d6733ee244238679f6410a10cd45ede55a61a8402/python-constraint-1.4.0.tar.bz2
    Requirement already satisfied: dill>=0.3 in /usr/local/lib/python3.6/dist-packages (from qiskit-terra==0.16.1->qiskit) (0.3.3)
    Collecting contextvars>=2.4; python_version < "3.7"
      Downloading https://files.pythonhosted.org/packages/83/96/55b82d9f13763be9d672622e1b8106c85acb83edd7cc2fa5bc67cd9877e9/contextvars-2.4.tar.gz
    Collecting retworkx>=0.5.0
      Downloading https://files.pythonhosted.org/packages/b7/21/5d4498cedb23927a31173176975e2162e0d6a2b80846ef254dd5500e10da/retworkx-0.7.2-cp36-cp36m-manylinux2010_x86_64.whl (1.3MB)
         |████████████████████████████████| 1.3MB 38.6MB/s
    Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.6/dist-packages (from qiskit-terra==0.16.1->qiskit) (2.6.0)
    Collecting sympy>=1.3
      Downloading https://files.pythonhosted.org/packages/ff/69/b16fc81b939d3efdd0b552f2e3e54f7fa1423d0c320cced2e69e675dde26/sympy-1.7.1-py3-none-any.whl (5.9MB)
         |████████████████████████████████| 5.9MB 39.7MB/s
    Requirement already satisfied: networkx>=2.2 in /usr/local/lib/python3.6/dist-packages (from qiskit-terra==0.16.1->qiskit) (2.5)
    Collecting ply>=3.10
      Downloading https://files.pythonhosted.org/packages/a3/58/35da89ee790598a0700ea49b2a66594140f44dec458c07e8e3d4979137fc/ply-3.11-py2.py3-none-any.whl (49kB)
         |████████████████████████████████| 51kB 5.7MB/s
    Requirement already satisfied: scipy>=1.4 in /usr/local/lib/python3.6/dist-packages (from qiskit-terra==0.16.1->qiskit) (1.4.1)
    Collecting fastjsonschema>=2.10
      Downloading https://files.pythonhosted.org/packages/c7/47/e860ab658dc2da844e6fdb76b12f6b3e14fbcb62fdd04f3e1c572bca20e1/fastjsonschema-2.14.5-py3-none-any.whl
    Requirement already satisfied: python-dateutil>=2.8.0 in /usr/local/lib/python3.6/dist-packages (from qiskit-terra==0.16.1->qiskit) (2.8.1)
    Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.6/dist-packages (from qiskit-terra==0.16.1->qiskit) (1.19.5)
    Requirement already satisfied: cython>=0.27.1 in /usr/local/lib/python3.6/dist-packages (from qiskit-aer==0.7.2->qiskit) (0.29.21)
    Collecting pybind11>=2.4
      Downloading https://files.pythonhosted.org/packages/00/84/fc9dc13ee536ba5e6b8fd10ce368fea5b738fe394c3b296cde7c9b144a92/pybind11-2.6.1-py2.py3-none-any.whl (188kB)
         |████████████████████████████████| 194kB 49.9MB/s
    Requirement already satisfied: urllib3>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from qiskit-ibmq-provider==0.11.1->qiskit) (1.24.3)
    Collecting requests-ntlm>=1.1.0
      Downloading https://files.pythonhosted.org/packages/03/4b/8b9a1afde8072c4d5710d9fa91433d504325821b038e00237dc8d6d833dc/requests_ntlm-1.1.0-py2.py3-none-any.whl
    Requirement already satisfied: nest-asyncio!=1.1.0,>=1.0.0 in /usr/local/lib/python3.6/dist-packages (from qiskit-ibmq-provider==0.11.1->qiskit) (1.4.3)
    Collecting websockets>=8
      Downloading https://files.pythonhosted.org/packages/bb/d9/856af84843912e2853b1b6e898ac8b802989fcf9ecf8e8445a1da263bf3b/websockets-8.1-cp36-cp36m-manylinux2010_x86_64.whl (78kB)
         |████████████████████████████████| 81kB 6.8MB/s
    Requirement already satisfied: requests>=2.19 in /usr/local/lib/python3.6/dist-packages (from qiskit-ibmq-provider==0.11.1->qiskit) (2.23.0)
    Requirement already satisfied: scikit-learn>=0.17 in /usr/local/lib/python3.6/dist-packages (from qiskit-ignis==0.5.1->qiskit) (0.22.2.post1)
    Requirement already satisfied: setuptools>=40.1.0 in /usr/local/lib/python3.6/dist-packages (from qiskit-ignis==0.5.1->qiskit) (51.3.3)
    Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from qiskit-aqua==0.8.1->qiskit) (1.1.5)
    Requirement already satisfied: fastdtw in /usr/local/lib/python3.6/dist-packages (from qiskit-aqua==0.8.1->qiskit) (0.3.4)
    Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from qiskit-aqua==0.8.1->qiskit) (2.10.0)
    Collecting docplex
      Downloading https://files.pythonhosted.org/packages/6e/53/7d85a2e2b8559d3dc08ffde370c0f57189ffd9e35493acf1f719f6ac5e0d/docplex-2.19.202.tar.gz (607kB)
         |████████████████████████████████| 614kB 41.7MB/s
    Collecting dlx
      Downloading https://files.pythonhosted.org/packages/54/c0/b8fb5bb727e983b6f5251433ef941b48f38c65bb0bd6ec509e9185bcd406/dlx-1.0.4.tar.gz
    Collecting quandl
      Downloading https://files.pythonhosted.org/packages/c2/58/9f0e69d836045e3865d263e9ed49f42b23a58526fdabb30f74c430baee3f/Quandl-3.6.0-py2.py3-none-any.whl
    Collecting yfinance
      Downloading https://files.pythonhosted.org/packages/7a/e8/b9d7104d3a4bf39924799067592d9e59119fcfc900a425a12e80a3123ec8/yfinance-0.1.55.tar.gz
    Collecting immutables>=0.9
      Downloading https://files.pythonhosted.org/packages/99/e0/ea6fd4697120327d26773b5a84853f897a68e33d3f9376b00a8ff96e4f63/immutables-0.14-cp36-cp36m-manylinux1_x86_64.whl (98kB)
         |████████████████████████████████| 102kB 9.6MB/s
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.6/dist-packages (from sympy>=1.3->qiskit-terra==0.16.1->qiskit) (1.1.0)
    Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx>=2.2->qiskit-terra==0.16.1->qiskit) (4.4.2)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.8.0->qiskit-terra==0.16.1->qiskit) (1.15.0)
    Collecting cryptography>=1.3
      Downloading https://files.pythonhosted.org/packages/c9/de/7054df0620b5411ba45480f0261e1fb66a53f3db31b28e3aa52c026e72d9/cryptography-3.3.1-cp36-abi3-manylinux2010_x86_64.whl (2.6MB)
         |████████████████████████████████| 2.6MB 37.1MB/s
    Collecting ntlm-auth>=1.0.2
      Downloading https://files.pythonhosted.org/packages/ff/84/97c550164b54942b0e908c31ef09d9469f3ba4cd7332a671e2125732f63b/ntlm_auth-1.5.0-py2.py3-none-any.whl
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.11.1->qiskit) (3.0.4)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.11.1->qiskit) (2.10)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.11.1->qiskit) (2020.12.5)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn>=0.17->qiskit-ignis==0.5.1->qiskit) (1.0.0)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->qiskit-aqua==0.8.1->qiskit) (2018.9)
    Collecting inflection>=0.3.1
      Downloading https://files.pythonhosted.org/packages/59/91/aa6bde563e0085a02a435aa99b49ef75b0a4b062635e606dab23ce18d720/inflection-0.5.1-py2.py3-none-any.whl
    Requirement already satisfied: more-itertools in /usr/local/lib/python3.6/dist-packages (from quandl->qiskit-aqua==0.8.1->qiskit) (8.6.0)
    Requirement already satisfied: multitasking>=0.0.7 in /usr/local/lib/python3.6/dist-packages (from yfinance->qiskit-aqua==0.8.1->qiskit) (0.0.9)
    Collecting lxml>=4.5.1
      Downloading https://files.pythonhosted.org/packages/bd/78/56a7c88a57d0d14945472535d0df9fb4bbad7d34ede658ec7961635c790e/lxml-4.6.2-cp36-cp36m-manylinux1_x86_64.whl (5.5MB)
         |████████████████████████████████| 5.5MB 34.2MB/s
    Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.6/dist-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.11.1->qiskit) (1.14.4)
    Requirement already satisfied: pycparser in /usr/local/lib/python3.6/dist-packages (from cffi>=1.12->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.11.1->qiskit) (2.20)
    Building wheels for collected packages: qiskit, python-constraint, contextvars, docplex, dlx, yfinance
      Building wheel for qiskit (setup.py) ... done
      Created wheel for qiskit: filename=qiskit-0.23.2-cp36-none-any.whl size=2876 sha256=c9a02df53b4a696d02a5851e762d9e8e176c4cfe01811330b400753471a1cb92
      Stored in directory: /root/.cache/pip/wheels/c2/62/3e/d5aeec3b19ba0e2c0c10292747d0bd4489b34e3f3385a908d0
      Building wheel for python-constraint (setup.py) ... done
      Created wheel for python-constraint: filename=python_constraint-1.4.0-py2.py3-none-any.whl size=24079 sha256=1ac4d4feb0cbb4d23e212294dd3be9e8d6860533df89606b2a7c32b8d99b4009
      Stored in directory: /root/.cache/pip/wheels/34/31/15/7b070b25d0a549d20ce2e9fe6d727471c2c61ef904720fd40c
      Building wheel for contextvars (setup.py) ... done
      Created wheel for contextvars: filename=contextvars-2.4-cp36-none-any.whl size=7667 sha256=827b8d8a536651a2bb57676bedde044a71349557ad0cdb05417db5a18240ba7d
      Stored in directory: /root/.cache/pip/wheels/a5/7d/68/1ebae2668bda2228686e3c1cf16f2c2384cea6e9334ad5f6de
      Building wheel for docplex (setup.py) ... done
      Created wheel for docplex: filename=docplex-2.19.202-cp36-none-any.whl size=670938 sha256=6cbe644f395e8cd9471e01d3f3678f92b67fa0a0d0eb2e738829b351eaf0daef
      Stored in directory: /root/.cache/pip/wheels/33/bb/64/5dd51006e730eb1227b8033d1f9e2246b806d9985a70c90d99
      Building wheel for dlx (setup.py) ... done
      Created wheel for dlx: filename=dlx-1.0.4-cp36-none-any.whl size=5711 sha256=4aa2a2f7f199db8d9b547d1236cf7d3637a893bd5fffbef71a80a2f04cf87ce9
      Stored in directory: /root/.cache/pip/wheels/bb/ba/15/fdd0deb104df3254912998150ba9245668db06b00af5912d1a
      Building wheel for yfinance (setup.py) ... done
      Created wheel for yfinance: filename=yfinance-0.1.55-py2.py3-none-any.whl size=22616 sha256=f4b56b7fb152f67dc6efd73d80b7f02d9fc1f88ab9e9766dd06686b5e41b4231
      Stored in directory: /root/.cache/pip/wheels/04/98/cc/2702a4242d60bdc14f48b4557c427ded1fe92aedf257d4565c
    Successfully built qiskit python-constraint contextvars docplex dlx yfinance
    Installing collected packages: python-constraint, immutables, contextvars, retworkx, sympy, ply, fastjsonschema, qiskit-terra, pybind11, qiskit-aer, cryptography, ntlm-auth, requests-ntlm, websockets, qiskit-ibmq-provider, qiskit-ignis, docplex, dlx, inflection, quandl, lxml, yfinance, qiskit-aqua, qiskit
      Found existing installation: sympy 1.1.1
        Uninstalling sympy-1.1.1:
          Successfully uninstalled sympy-1.1.1
      Found existing installation: lxml 4.2.6
        Uninstalling lxml-4.2.6:
          Successfully uninstalled lxml-4.2.6
    Successfully installed contextvars-2.4 cryptography-3.3.1 dlx-1.0.4 docplex-2.19.202 fastjsonschema-2.14.5 immutables-0.14 inflection-0.5.1 lxml-4.6.2 ntlm-auth-1.5.0 ply-3.11 pybind11-2.6.1 python-constraint-1.4.0 qiskit-0.23.2 qiskit-aer-0.7.2 qiskit-aqua-0.8.1 qiskit-ibmq-provider-0.11.1 qiskit-ignis-0.5.1 qiskit-terra-0.16.1 quandl-3.6.0 requests-ntlm-1.1.0 retworkx-0.7.2 sympy-1.7.1 websockets-8.1 yfinance-0.1.55

```python
import qiskit
```

# Lab 1

In this lab you will be reconstructing circuits from given input and output states using Qiskit.
$$\newcommand{\bra}[1]{\left\langle{#1}\right|}$$
$$\newcommand{\ket}[1]{\left|{#1}\right\rangle}$$

## Example

This is an example for how to build a cirucit using Qiskit. This goes over how to create a circuit, and apply basic gates to the qubits.

### Circuit Creation

```python
# Allocate a 3 Qubit Quantum Register
qrex = qiskit.QuantumRegister(3)
# Allocate a 3 Qubit Register
crex = qiskit.ClassicalRegister(3)
# Create a Quantum Circuit that has 1 3-Qubit Quantum Register and 1 3-Bit Classical Register
qcex = qiskit.QuantumCircuit(qrex, crex)
```

### Gates

```python
qcex.x(qrex[0]) # Apply an X gate to the first qubit.
qcex.z(qrex[1]) # Apply a Z gate to the second qubit.
qcex.h(qrex[2]) # Apply an H gate to the third qubit.
qcex.cx(qrex[0], qrex[1]) # Apply a CNOT gate where the control is the first qubit and the target is the second.
qcex.cx(qrex[1], qrex[0]) # Apply a CNOT gate where the control is the second qubit and the target is the first.
qcex.swap(qrex[1], qrex[2]) # Apply a SWAP gate between the second and third qubits.  Order does not matter.

qcex.measure(qrex, crex) # Applies a measurement gate across all the qubits to classical registers

# To look at the circuit, use .draw(), this will generate a string that you
# can use to look at the construction.
qcex.draw()
```

      ┌───┐     ┌───┐   ┌─┐
q0_0: ┤ X ├──■──┤ X ├───┤M├──────
      ├───┤┌─┴─┐└─┬─┘   └╥┘┌─┐
q0_1: ┤ Z ├┤ X ├──■───X──╫─┤M├───
      ├───┤└───┘      │  ║ └╥┘┌─┐
q0_2: ┤ H ├───────────X──╫──╫─┤M├
      └───┘              ║  ║ └╥┘
c0: 3/═══════════════════╩══╩══╩═
                         0  1  2

## Pre-Exercise 1

Recreate the following circuit in Qiskit. Don't worry about spacing, just get the gates in the right order.
```
                ┌───┐
q0_0: ──■───────┤ H ├──X───────
      ┌─┴─┐┌───┐└───┘  |  ┌───┐
q0_1: ┤ X ├┤ X ├───────|──┤ X ├
      └───┘└─┬─┘┌───┐  |  └───┘
q0_2: ───────■──┤ Z ├──X───────
                └───┘

```

```python
# Circuit set up
qrpre = qiskit.QuantumRegister(3)
qcpre = qiskit.QuantumCircuit(qrpre)
```

## Exercise 1

Starting in state $\ket{0}$ create a circuit that generates a $\frac{\sqrt{2}}{2}\ket{0} - \frac{\sqrt{2}}{2}\ket{1}$ state, or $\ket{-}$.

```python
# Circuit set up
qr1 = qiskit.QuantumRegister(1)
qc1 = qiskit.QuantumCircuit(qr1)
```

## Exercise 2

```python
# Circuit set up
qr2 = qiskit.QuantumRegister(2)
qc2 = qiskit.QuantumCircuit(qr2)
```

Starting in state $\alpha\ket{00} + \beta\ket{11}$ create a circuit that generates a $\frac{\sqrt{2} \beta}{2}\ket{00} - \frac{\sqrt{2} \alpha}{2}\ket{01} + \frac{\sqrt{2} \beta}{2}\ket{10} + \frac{\sqrt{2} \alpha}{2}\ket{11}$ state.

```python
# Put your circuit here
```

## Exercise 3

```python
# Circuit set up
qr3 = qiskit.QuantumRegister(2)
qc3 = qiskit.QuantumCircuit(qr3)
```

create a circuit that starting in state $\alpha\ket{00}$ generates a $-\frac{\alpha\sqrt{2}}{2}\ket{10} + \frac{\alpha\sqrt{2}}{2}\ket{11}$ state and starting in state $\alpha\ket{11}$ generates a $\frac{\alpha\sqrt{2}}{2}\ket{00} - \frac{\alpha\sqrt{2}}{2}\ket{01}$ state.

**Hint:** You will need to use a CNOT Gate.

```python
# Put your circuit here
```

## Exercise 4

```python
# Circuit set up
qr4 = qiskit.QuantumRegister(2)
qc4 = qiskit.QuantumCircuit(qr4)
```

Create a circuit that:

Starting in state $\alpha\ket{00}$ generates a $\frac{\alpha}{2}\ket{00} - \frac{\alpha}{2}\ket{01} - \frac{\alpha}{2}\ket{10} + \frac{\alpha}{2}\ket{11}$ state.

Starting in state $\alpha\ket{01}$ generates a $\frac{\alpha}{2}\ket{00} - \frac{\alpha}{2}\ket{01} + \frac{\alpha}{2}\ket{10} - \frac{\alpha}{2}\ket{11}$ state.

Starting in state $\alpha\ket{10}$ generates a $\frac{\alpha}{2}\ket{00}
+ \frac{\alpha}{2}\ket{01} - \frac{\alpha}{2}\ket{10} - \frac{\alpha}{2}\ket{11}$ state.

Starting in state $\alpha\ket{11}$ generates a $\frac{\alpha}{2}\ket{00} + \frac{\alpha}{2}\ket{01} + \frac{\alpha}{2}\ket{10} + \frac{\alpha}{2}\ket{11}$ state.

```python
# Put your circuit here
```

```python

```