_Lambda School Data Science — Linear Models_

# Understanding Linear Regression

#### Objectives
- understand how ordinary least squares regression minimizes the sum of squared errors
- understand how linear algebra can solve ordinary least squares regression
- get and interpret coefficients of a linear model
- visualize a line of best fit in 2D, and hyperplane in 3D
- use regression metrics: MSE, RMSE, MAE, R^2

#### Extra Links
- [Statistics 101: Simple Linear Regression](https://www.youtube.com/watch?v=ZkjP5RJLQF4) (20 minute video)
- [_An Introduction to Statistical Learning_](http://www-bcf.usc.edu/~gareth/ISL/ISLR%20Seventh%20Printing.pdf), Chapter 3.1, Simple Linear Regression, & 3.2, Multiple Linear Regression
- Priceonomics, [The Discovery of Statistical Regression](https://priceonomics.com/the-discovery-of-statistical-regression/)
- Priceonomics, [Why the Father of Modern Statistics Didn’t Believe Smoking Caused Cancer](https://priceonomics.com/why-the-father-of-modern-statistics-didnt-believe/)
- Harvard Business Review, [When to Act on a Correlation, and When Not To](https://hbr.org/2014/03/when-to-act-on-a-correlation-and-when-not-to)
- [xkcd 552: Correlation](https://www.explainxkcd.com/wiki/index.php/552:_Correlation)
- [xkcd 1725: Linear Regression](https://www.explainxkcd.com/wiki/index.php/1725:_Linear_Regression)

## What is Linear Regression?

[Linear Regression](https://en.wikipedia.org/wiki/Linear_regression) is a statistical model that seeks to describe the relationship between some y variable and one or more x variables.

In the simplest case, linear regression seeks to fit a straight line through a cloud of points. This line is referred to as the "regression line" or "line of best fit." This line tries to summarize the relationship between our X and Y in a way that enables us to use the equation for that line to make predictions.

```python
%matplotlib inline
import seaborn as sns
x = [10,  8, 13,  9, 11, 14,  6,  4, 12,  7,  5]
y = [ 8,  6,  7,  8,  8,  9,  7,  4, 10,  4,  5]
sns.regplot(x, y);
```

### Synonyms for "y variable"
- Dependent Variable
- Response Variable
- Outcome Variable
- Predicted Variable
- Measured Variable
- Explained Variable
- Label
- Target

### Synonyms for "x variable"
- Independent Variable
- Explanatory Variable
- Regressor
- Covariate
- Feature

## Simple Linear Regresion

#### Making Predictions

Say that we were trying to create a model that captured the relationship between temperature outside and ice cream sales. In Machine Learning our goal is often different that of other flavors of Linear Regression Analysis, because we're trying to fit a model to this data with the intention of making **predictions** on new data (in the future) that we don't have yet.

#### What are we trying to predict?

So if we had measured ice cream sales and the temprature outside on 11 different days, at the end of our modeling **what would be the thing that we would want to predict? - Ice Cream Sales or Temperature?**

We would probably want to be measuring temperature with the intention of using that to **forecast** ice cream sales. If we were able to successfully forecast ice cream sales from temperature, this might help us know beforehand how much ice cream to make or how many cones to buy or on which days to open our store, etc. Being able to make predictions accurately has a lot of business implications. This is why making accurate predictions is so valuable (And in large part is why data scientists are paid so well).

#### y variable intuition

We want the thing that we're trying to predict to serve as our **y** variable. This is why it's sometimes called the "predicted variable." We call it the "dependent" variable because our prediction for how much ice cream we're going to sell "depends" on the temperature outside.

#### x variable intuition

All other variables that we use to predict our y variable (we're going to start off just using one) we call our **x** variables. These are called our "independent" variables because they don't *depend* on y, they "explain" y. Hence they are also referred to as our "explanatory" variables.

## Example: Predict presidential election voting

#### Douglas Hibbs, [Background Information on the ‘Bread and Peace’ Model of Voting in Postwar US Presidential Elections](https://douglas-hibbs.com/background-information-on-bread-and-peace-voting-in-us-presidential-elections/)

> Aggregate two-party vote shares going to candidates of the party holding the presidency during the postwar era are well explained by just two fundamental determinants:

> (1) Positively by weighted-average growth of per capita real disposable personal income over the term.
> (2) Negatively by cumulative US military fatalities (scaled to population) owing to unprovoked, hostile deployments of American armed forces in foreign wars.

#### Data sources
- 1952-2012: Douglas Hibbs, [2014 lecture at Deakin University Melbourne](http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf), Slide 40
- 2016, Vote Share: [The American Presidency Project](https://www.presidency.ucsb.edu/statistics/elections)
- 2016, Recent Growth in Personal Incomes: [The 2016 election economy: the "Bread and Peace" model final forecast](https://angrybearblog.com/2016/11/the-2016-election-economy-the-bread-and-peace-model-final-forecast.html)
- 2016, US Military Fatalities: Assumption that Afghanistan War fatalities in 2012-16 occured at the same rate as 2008-12

```python
import pandas as pd

columns = ['year','inc_party_candidate','other_candidate','inc_party_vote']

data = [[1952,"Stevenson","Eisenhower",44.6],
        [1956,"Eisenhower","Stevenson",57.76],
        [1960,"Nixon","Kennedy",49.91],
        [1964,"Johnson","Goldwater",61.34],
        [1968,"Humphrey","Nixon",49.60],
        [1972,"Nixon","McGovern",61.79],
        [1976,"Ford","Carter",48.95],
        [1980,"Carter","Reagan",44.70],
        [1984,"Reagan","Mondale",59.17],
        [1988,"Bush, Sr.","Dukakis",53.94],
        [1992,"Bush, Sr.","Clinton",46.55],
        [1996,"Clinton","Dole",54.74],
        [2000,"Gore","Bush, Jr.",50.27],
        [2004,"Bush, Jr.","Kerry",51.24],
        [2008,"McCain","Obama",46.32],
        [2012,"Obama","Romney",52.00],
        [2016,"Clinton","Trump",48.2]]

votes = pd.DataFrame(data=data, columns=columns)
```

```python
columns = ['year','income_growth']

data = [[1952,2.40],
        [1956,2.89],
        [1960, .85],
        [1964,4.21],
        [1968,3.02],
        [1972,3.62],
        [1976,1.08],
        [1980,-.39],
        [1984,3.86],
        [1988,2.27],
        [1992, .38],
        [1996,1.04],
        [2000,2.36],
        [2004,1.72],
        [2008, .10],
        [2012, .95],
        [2016, .10]]

growth = pd.DataFrame(data=data, columns=columns)
```

```python
"""
Fatalities denotes the cumulative number of American military
fatalities per millions of US population the in Korea, Vietnam,
Iraq and Afghanistan wars during the presidential terms
preceding the 1952, 1964, 1968, 1976 and 2004, 2008 and
2012 elections.

http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf
"""

columns = ['year','fatal_per_mil']

data = [[1952,190],
        [1956,  0],
        [1960,  0],
        [1964,  1],
        [1968,146],
        [1972,  0],
        [1976,  2],
        [1980,  0],
        [1984,  0],
        [1988,  0],
        [1992,  0],
        [1996,  0],
        [2000,  0],
        [2004,  4],
        [2008, 14],
        [2012,  5],
        [2016,  5]]

deaths = pd.DataFrame(data=data, columns=columns)
```

```python
df = votes.merge(growth).merge(deaths)
```

### Plot univariate correlations
[Seaborn tutorial: Visualizing linear relationships](https://seaborn.pydata.org/tutorial/regression.html)

```python
%matplotlib inline
import seaborn as sns

target = 'inc_party_vote'
features = ['income_growth',
            'fatal_per_mil']

for feature in features:
    sns.lmplot(x=feature, y=target, data=df)
```

We can see from the scatterplot that these data points seem to follow a somewhat linear relationship for the "Average Recent Growth in Personal Incomes" feature. This means that we could probably summarize their relationship well by fitting a line of best fit to these points. Lets do it.

## The Equation for a Line

A common equation for a line is:

\begin{align}
y = mx + b
\end{align}

Where $m$ is the slope of our line and $b$ is the y-intercept.

If we want to plot a line through our cloud of points we figure out what these two values should be. Linear Regression seeks to **estimate** the slope and intercept values that describe a line that best fits the data points.

## The Anatomy of Linear Regression

- Intercept: The $b$ value in our line equation $y=mx+b$
- Slope: The $m$ value in our line equation $y=mx+b$. These two values together define our regression line.
- $\hat{y}$ : A prediction
- Line of Best Fit (Regression Line)
- Predicted (fitted) Values: Points on our regression line
- Observed Values: Points from our dataset
- Error: The distance between predicted and observed values.

Ordinary Least Squares Regression is a way to solve for $m$ and $b$.

Let's start by seeing what would happen if we just guessed and checked some values for $m$ and $b$.

What's the line of "best" fit look like? What's the error?

```python
# TODO

x = df['income_growth'];
y = df['inc_party_vote']

m = 0;
b = y.mean()
y_pred = m*x + b
y_pred
```

    0     51.828235
    1     51.828235
    2     51.828235
    3     51.828235
    4     51.828235
    5     51.828235
    6     51.828235
    7     51.828235
    8     51.828235
    9     51.828235
    10    51.828235
    11    51.828235
    12    51.828235
    13    51.828235
    14    51.828235
    15    51.828235
    16    51.828235
    Name: income_growth, dtype: float64

```python
import matplotlib.pyplot as plt;
from sklearn.metrics import mean_absolute_error, r2_score;

def plot_preds(x, y, y_pred):
    plt.scatter(x, y, label='y_true');
    plt.plot(x, y_pred, label='y_pred');
    plt.legend();

    mae = mean_absolute_error(y, y_pred);
    r2 = r2_score(y, y_pred);
    print(f'Mean Abosulte Error: {mae}');
    print(f'R^2 Score: {r2}');
```

```python
plot_preds(x, y, y_pred);
```

```python
m = 4.1;
b = 45;
y_pred = m*x + b;
plot_preds(x, y, y_pred);
plt.title("Guessing LR Values");
```

## R Squared:  $R^2$

One final attribute of linear regressions that we're going to talk about today is a measure of goodness of fit known as $R^2$ or R-squared. $R^2$ is a statistical measure of how close the data are fitted to our regression line. A helpful interpretation for the $R^2$ is the percentage of the dependent variable that is explained by the model.

In other words, the $R^2$ is the percentage of y that is explained by the x variables included in the model. For this reason the $R^2$ is also known as the "coefficient of determination," because it explains how much of y is explained (or determined) by our x varaibles. We won't go into the calculation of $R^2$ today, just know that a higher $R^2$ percentage is nearly always better and indicates a model that fits the data more closely.

## Residual Error

The residual error is the distance between points in our dataset and our regression line.

```python
def regression_residuals(df, feature, target, m, b):
    x = df[feature]
    y = df[target]
    y_pred = m*x + b

    plt.scatter(x, y, label='y_true')
    plt.plot(x, y_pred, label='y_pred')
    plt.legend()

    # Plot residual errors
    for x, y1, y2 in zip(x, y, y_pred):
        plt.plot((x, x), (y1, y2), color='grey')

    mae = mean_absolute_error(y, y_pred)
    r2 = r2_score(y, y_pred)
    print('Mean Absolute Error:', mae)
    print('R^2:', r2)

regression_residuals(df, feature='income_growth',
                     target='inc_party_vote', m=3, b=46)
```

```python
from ipywidgets import interact, fixed

interact(regression_residuals,
         df=fixed(df),
         feature=fixed('income_growth'),
         target=fixed('inc_party_vote'),
         m=(-10,10,0.5),
         b=(40,60,0.5));
```

    interactive(children=(FloatSlider(value=0.0, description='m', max=10.0, min=-10.0, step=0.5), FloatSlider(valu…

## More Formal Notation

We have talked about a line of regression being represented like a regular line $y=mx+b$ but as we get to more complicated versions we're going to need to extend this equation. So lets establish the proper terminology.

**X** - Independent Variable, predictor variable, explanatory variable, regressor, covariate

**Y** - Response variable, predicted variable, measured vairable, explained variable, outcome variable

$\beta_0$ - "Beta Naught" or "Beta Zero", the intercept value. This is how much of y would exist if X were zero. This is sometimes represented by the letter "a" but I hate that. So it's "Beta 0" during my lecture.

$\beta_1$ - "Beta One" The primary coefficient of interest. This values is the slope of the line that is estimated by "minimizing the sum of the squared errors/residuals" - We'll get to that.

$\epsilon$ - "Epsilon" The "error term", random noise, things outside of our model that affect y.

## Minimizing the Sum of the Squared Error

The most common method of estimating our $\beta$ parameters  is what's known as "Ordinary Least Squares" (OLS). (There are different methods of arriving at a line of best fit). OLS estimates the parameters that minimize the squared distance between each point in our dataset and our line of best fit.

\begin{align}
SSE = \sum(y_i - \hat{y})^2
\end{align}

```python
from matplotlib.patches import Rectangle
import numpy as np
from sklearn.metrics import mean_squared_error

def regression_squared_errors(df, feature, target, m, b):
    x = df[feature]
    y = df[target]
    y_pred = m*x + b

    fig = plt.figure(figsize=(7,7))
    ax = plt.axes()
    ax.scatter(x, y, label='y_true')
    ax.plot(x, y_pred, label='y_pred')
    ax.legend()

    # Plot square errors
    xmin, xmax = ax.get_xlim()
    ymin, ymax = ax.get_ylim()
    scale = (xmax-xmin)/(ymax-ymin)
    for x, y1, y2 in zip(x, y, y_pred):
        bottom_left = (x, min(y1, y2))
        height = abs(y1 - y2)
        width = height * scale
        ax.add_patch(Rectangle(xy=bottom_left, width=width, height=height, alpha=0.1))

    mse = mean_squared_error(y, y_pred)
    rmse = np.sqrt(mse)
    mae = mean_absolute_error(y, y_pred)
    r2 = r2_score(y, y_pred)
    print('Mean Squared Error:', mse)
    print('Root Mean Squared Error:', rmse)
    print('Mean Absolute Error:', mae)
    print('R^2:', r2)

regression_squared_errors(df, feature='income_growth',
           target='inc_party_vote', m=3, b=46)
```

```python
interact(regression_squared_errors,
         df=fixed(df),
         feature=fixed('income_growth'),
         target=fixed('inc_party_vote'),
         m=(-10,10,0.5),
         b=(40,60,0.5));
```

    interactive(children=(FloatSlider(value=0.0, description='m', max=10.0, min=-10.0, step=0.5), FloatSlider(valu…

## Hypotheses

```python
b = 46
ms = np.arange(-10,10,0.5)
sses = []
feature = 'income_growth'

for m in ms:
    predictions = m * df[feature] + b
    errors = predictions - df[target]
    square_errors = errors ** 2
    sse = square_errors.sum()
    sses.append(sse)

hypotheses = pd.DataFrame({'Slope': ms})
hypotheses['Intercept'] = b
hypotheses['Sum of Square Errors'] = sses

hypotheses
```

      Slope
      Intercept
      Sum of Square Errors

      0
      -10.0
      46
      15215.58740

      1
      -9.5
      46
      14095.52845

      2
      -9.0
      46
      13018.88500

      3
      -8.5
      46
      11985.65705

      4
      -8.0
      46
      10995.84460

      5
      -7.5
      46
      10049.44765

      6
      -7.0
      46
      9146.46620

      7
      -6.5
      46
      8286.90025

      8
      -6.0
      46
      7470.74980

      9
      -5.5
      46
      6698.01485

      10
      -5.0
      46
      5968.69540

      11
      -4.5
      46
      5282.79145

      12
      -4.0
      46
      4640.30300

      13
      -3.5
      46
      4041.23005

      14
      -3.0
      46
      3485.57260

      15
      -2.5
      46
      2973.33065

      16
      -2.0
      46
      2504.50420

      17
      -1.5
      46
      2079.09325

      18
      -1.0
      46
      1697.09780

      19
      -0.5
      46
      1358.51785

      20
      0.0
      46
      1063.35340

      21
      0.5
      46
      811.60445

      22
      1.0
      46
      603.27100

      23
      1.5
      46
      438.35305

      24
      2.0
      46
      316.85060

      25
      2.5
      46
      238.76365

      26
      3.0
      46
      204.09220

      27
      3.5
      46
      212.83625

      28
      4.0
      46
      264.99580

      29
      4.5
      46
      360.57085

      30
      5.0
      46
      499.56140

      31
      5.5
      46
      681.96745

      32
      6.0
      46
      907.78900

      33
      6.5
      46
      1177.02605

      34
      7.0
      46
      1489.67860

      35
      7.5
      46
      1845.74665

      36
      8.0
      46
      2245.23020

      37
      8.5
      46
      2688.12925

      38
      9.0
      46
      3174.44380

      39
      9.5
      46
      3704.17385

```python
hypotheses.plot(x='Slope', y='Sum of Square Errors',
                title=f'Intercept={b}');
```

## Scikit-learn

#### Jake VanderPlas, [Python Data Science Handbook, Chapter 5.2, Introducing Scikit-Learn](https://jakevdp.github.io/PythonDataScienceHandbook/05.02-introducing-scikit-learn.html), Scikit-Learn's Estimator API

> Most commonly, the steps in using the Scikit-Learn estimator API are as follows (we will step through a handful of detailed examples in the sections that follow).

> 1. Choose a class of model by importing the appropriate estimator class from Scikit-Learn.
> 2. Choose model hyperparameters by instantiating this class with desired values.
> 3. Arrange data into a features matrix and target vector following the discussion above.
> 4. Fit the model to your data by calling the `fit()` method of the model instance.
> 5. Apply the Model to new data: For supervised learning, often we predict labels for unknown data using the `predict()` method.

```python
# TODO
```

```python
from sklearn.linear_model import LinearRegression;

model = LinearRegression();
features = ['income_growth'];
target = 'inc_party_vote';

x = df[features];
y = df[target];

model.fit(x, y);
```

```python
model.intercept_, model.coef_;
```

```python
model.predict([[1]]);
```

```python
model.predict([[4]]);
```

```python
model.predict([[2]]) - model.predict([[1]]);
```

```python
x
```

      income_growth

      0
      2.40

      1
      2.89

      2
      0.85

      3
      4.21

      4
      3.02

      5
      3.62

      6
      1.08

      7
      -0.39

      8
      3.86

      9
      2.27

      10
      0.38

      11
      1.04

      12
      2.36

      13
      1.72

      14
      0.10

      15
      0.95

      16
      0.10

```python
m = model.coef_[0];
b = model.intercept_;

y_pred = m*x + b;
```

```python
y_pred = model.predict(x);
plot_preds(x, y, y_pred);
```

## Linear Algebra!

The same result that is found by minimizing the sum of the squared errors can be also found through a linear algebra process known as the "Least Squares Solution:"

\begin{align}
\hat{\beta} = (X^{T}X)^{-1}X^{T}y
\end{align}

Before we can work with this equation in its linear algebra form we have to understand how to set up the matrices that are involved in this equation.

### The $\beta$ vector

The $\beta$ vector represents all the parameters that we are trying to estimate, our $y$ vector and $X$ matrix values are full of data from our dataset. The $\beta$ vector holds the variables that we are solving for: $\beta_0$ and $\beta_1$

Now that we have all of the necessary parts we can set them up in the following equation:

\begin{align}
y = X \beta + \epsilon
\end{align}

Since our $\epsilon$ value represents **random** error we can assume that it will equal zero on average.

\begin{align}
y = X \beta
\end{align}

The objective now is to isolate the $\beta$ matrix. We can do this by pre-multiplying both sides by "X transpose" $X^{T}$.

\begin{align}
X^{T}y =  X^{T}X \beta
\end{align}

Since anything times its transpose will result in a square matrix, if that matrix is then an invertible matrix, then we should be able to multiply both sides by its inverse to remove it from the right hand side. (We'll talk tomorrow about situations that could lead to $X^{T}X$ not being invertible.)

\begin{align}
(X^{T}X)^{-1}X^{T}y =  (X^{T}X)^{-1}X^{T}X \beta
\end{align}

Since any matrix multiplied by its inverse results in the identity matrix, and anything multiplied by the identity matrix is itself, we are left with only $\beta$ on the right hand side:

\begin{align}
(X^{T}X)^{-1}X^{T}y = \hat{\beta}
\end{align}

We will now call it "beta hat" $\hat{\beta}$ because it now represents our estimated values for $\beta_0$ and $\beta_1$

### Lets calculate our $\beta$ coefficients with numpy!

```python
df[features].values.T
```

    array([[ 2.4 ,  2.89,  0.85,  4.21,  3.02,  3.62,  1.08, -0.39,  3.86,
             2.27,  0.38,  1.04,  2.36,  1.72,  0.1 ,  0.95,  0.1 ]])

```python
from statsmodels.api import add_constant;
```

```python
x = add_constant(df[feature].values)
print('x')
print(x)

y = df[target].values[:, np.newaxis]
print('y')
print(y)

x_transpose = x.T
print('x Transpose')
print(x_transpose)

x_transpose_x = x_transpose @ x
print('x Transpose x')
print(x_transpose_x)

x_transpose_x_inverse = np.linalg.inv(x_transpose_x)
print('x Transpose x Inverse')
print(x_transpose_x_inverse)

x_transpose_y = x_transpose @ y
print('x Transpose y')
print(x_transpose_y)

beta_hat = x_transpose_x_inverse @ x_transpose_y
print('Beta Hat')
print(beta_hat)
```

    x
    [[ 1.    2.4 ]
     [ 1.    2.89]
     [ 1.    0.85]
     [ 1.    4.21]
     [ 1.    3.02]
     [ 1.    3.62]
     [ 1.    1.08]
     [ 1.   -0.39]
     [ 1.    3.86]
     [ 1.    2.27]
     [ 1.    0.38]
     [ 1.    1.04]
     [ 1.    2.36]
     [ 1.    1.72]
     [ 1.    0.1 ]
     [ 1.    0.95]
     [ 1.    0.1 ]]
    y
    [[44.6 ]
     [57.76]
     [49.91]
     [61.34]
     [49.6 ]
     [61.79]
     [48.95]
     [44.7 ]
     [59.17]
     [53.94]
     [46.55]
     [54.74]
     [50.27]
     [51.24]
     [46.32]
     [52.  ]
     [48.2 ]]
    x Transpose
    [[ 1.    1.    1.    1.    1.    1.    1.    1.    1.    1.    1.    1.
       1.    1.    1.    1.    1.  ]
     [ 2.4   2.89  0.85  4.21  3.02  3.62  1.08 -0.39  3.86  2.27  0.38  1.04
       2.36  1.72  0.1   0.95  0.1 ]]
    x Transpose x
    [[17.    30.46 ]
     [30.46  86.831]]
    x Transpose x Inverse
    [[ 0.15835959 -0.05555197]
     [-0.05555197  0.03100405]]
    x Transpose y
    [[ 881.08  ]
     [1674.6167]]
    Beta Hat
    [[46.49920976]
     [ 2.97417709]]

# Multiple Regression

Simple or bivariate linear regression involves a single $x$ variable and a single $y$ variable. However, we can have many $x$ variables. A linear regression model that involves multiple x variables is known as **Multiple** Regression (NOT MULTIVARIATE).

\begin{align}
y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + ... + \beta_n X_n + \epsilon
\end{align}

```python
# TODO

model = LinearRegression();
features = ['income_growth', 'fatal_per_mil'];

target = 'inc_party_vote';

x = df[features];
y = df[target];

model.fit(x, y)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,
             normalize=False)

```python
pd.Series(model.coef_, features)
```

    income_growth    3.406214
    fatal_per_mil   -0.053752
    dtype: float64

## Visualize hyperplane of best fit in 3D

```python
# https://stackoverflow.com/a/47230966
# Plotly notebook mode with google colaboratory
# You need to define this function
# And call it in each offline plotting cell

def configure_plotly_browser_state():
    import IPython
    display(IPython.core.display.HTML('''

        '''))
```

```python
import itertools
import plotly.graph_objs as go
from plotly.offline import init_notebook_mode, iplot
init_notebook_mode(connected=True)

def viz3D(fitted_model, X, features, target='', num=100):
    """
    Visualize model predictions in 3D, for regression model fit on 2 features

    Parameters
    ----------
    fitted_model : scikit-learn model, already fitted
    X : pandas dataframe, which was used to fit model
    features : list of strings, column names of the 2 features
    target : string, name of target
    num : int, number of grid points for each feature

    References
    ----------
    https://plot.ly/python/3d-charts/
    """
    feature1, feature2 = features
    min1, max1 = X[feature1].min(), X[feature1].max()
    min2, max2 = X[feature2].min(), X[feature2].max()
    x1 = np.linspace(min1, max1, num)
    x2 = np.linspace(min2, max2, num)
    combos = list(itertools.product(x1, x2))
    Z = fitted_model.predict(combos).reshape(num, num)

    configure_plotly_browser_state()
    data = [go.Surface(x=x1, y=x2, z=Z)]
    layout = go.Layout(
        scene={'xaxis': {'title': feature1, 'range': [min1,max1], 'showticklabels': True},
               'yaxis': {'title': feature2, 'range': [min2,max2], 'showticklabels': True},
               'zaxis': {'title': target, 'showticklabels': True}},
    )
    fig = go.Figure(data=data, layout=layout)
    iplot(fig)
```

```python
# TODO
```

## Dimensionality in Linear Regression

Muliple Regression is simply an extension of the bivariate case. The reason why we see the bivariate case demonstrated so often is simply because it's easier to graph and all of the intuition from the bivariate case is the same as we keep on adding explanatory variables.

As we increase the number of $x$ values in our model we are simply fitting a n-1-dimensional plane to an n-dimensional cloud of points within an n-dimensional hypercube.

## Interpreting Coefficients

One of Linear Regression's strengths is that the parameters of the model (coefficients) are readily interpretable and useful. Not only do they describe the relationship between x and y but they put a number on just how much x is associated with y. We should be careful to not speak about this relationshiop in terms of causality because these coefficients are in fact correlative measures. We would need a host of additional techniques in order to estimate a causal effect using linear regression (econometrics).

\begin{align}
\hat{\beta} = \frac{Cov(x,y)}{Var(y)}
\end{align}

## Why is Linear Regression so Important?

### Popularity

Linear Regression is an extremely popular technique that every data scientist **needs** to understand. It's not the most advanced technique and there are supervised learning techniques that will obtain a higher accuracy, but where it lacks in accuracy it makes up for it in interpretability and simplicity.

### Interpretability

Few other models possess coefficients that are so directly linked to their variables with a such a clear interpretation. Tomorrow we're going to learn about ways to make them even easier to interpret.

### Simplicity

A linear regression model can be communicated just by writing out its equation. It's kind of incredible that such high dimensional relationships can be described from just a linear combination of variables and coefficients.

# Assignment
- Continue to predict New York City apartment rents. This is your last assignment with this dataset.
- You may select any number of features. You are encouraged to engineer new features.
- Get and plot your model's coefficients.
- Report your Root Mean Squared Error, Mean Absolute Error, and R^2 Score, for your Train and Test sets. Share your scores with your cohort on Slack!
- Fit a model with 2 features, and visualize the plane of best fit in 3D.
- Commit your notebook to your fork of the repo.

## Stretch Goals

Study more about Linear Regression. Here are two helpful links. If you find more links, share your favorites with your cohort on Slack.

1. Watch this 20 minute video that just hit 1 million views: Brandon Foltz, Statistics 101: Simple Linear Regression (https://www.youtube.com/watch?v=ZkjP5RJLQF4)
2. Skim _An Introduction to Statistical Learning_, Chapter 3.1, Simple Linear Regression, & Chapter 3.2, Multiple Linear Regression (http://www-bcf.usc.edu/~gareth/ISL/ISLR%20Seventh%20Printing.pdf)

In your 3D visualization, can you include the actual datapoints, like in [this notebook](https://nbviewer.jupyter.org/urls/s3.amazonaws.com/datarobotblog/notebooks/multiple_regression_in_python.ipynb)? Can you also include the residual lines from the datapoints to the plane of the best fit, like in _An Introduction to Statistical Learning?_ This would be hard to do, but awesome!

Can you get creative with feature engineering? Share with your cohort on Slack. We mentioned some feature ideas at the end of last lesson, but didn't demonstrate how to engineer them. So here are some example solutions:

```python
# Does apartment have a non-empty description?
df['description'] = df['description'].str.strip().fillna('')
df['has_description'] = df['description'] != ''

# How long is the description?
df['description_length'] = df['description'].str.len()

# How many total perks does each apartment have?
perk_cols = ['elevator', 'cats_allowed', 'hardwood_floors', 'dogs_allowed',
             'doorman', 'dishwasher', 'no_fee', 'laundry_in_building',
             'fitness_center', 'pre-war', 'laundry_in_unit', 'roof_deck',
             'outdoor_space', 'dining_room', 'high_speed_internet', 'balcony',
             'swimming_pool', 'new_construction', 'exclusive', 'terrace',
             'loft', 'garden_patio', 'common_outdoor_space',
             'wheelchair_access']
df['perk_count'] = df[perk_cols].sum(axis=1)

# Are pets allowed?
df['pets_allowed'] = (df['cats_allowed']==1) | (df['dogs_allowed']==1)
```

```python
LOCAL = '../data/nyc/nyc-rent-2016.csv'
WEB = 'https://raw.githubusercontent.com/LambdaSchool/DS-Unit-2-Linear-Models/master/data/nyc/nyc-rent-2016.csv'

import pandas as pd
df = pd.read_csv(LOCAL)
assert df.shape == (48300, 34)
df.head()
```

      bathrooms
      bedrooms
      created
      description
      display_address
      latitude
      longitude
      price
      street_address
      interest_level
      ...
      high_speed_internet
      balcony
      swimming_pool
      new_construction
      exclusive
      terrace
      loft
      garden_patio
      common_outdoor_space
      wheelchair_access

      0
      1.5
      3
      2016-06-24 07:54:24
      A Brand New 3 Bedroom 1.5 bath ApartmentEnjoy ...
      Metropolitan Avenue
      40.7145
      -73.9425
      3000
      792 Metropolitan Avenue
      medium
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      1
      1.0
      2
      2016-06-12 12:19:27

      Columbus Avenue
      40.7947
      -73.9667
      5465
      808 Columbus Avenue
      low
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      2
      1.0
      1
      2016-04-17 03:26:41
      Top Top West Village location, beautiful Pre-w...
      W 13 Street
      40.7388
      -74.0018
      2850
      241 W 13 Street
      high
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      3
      1.0
      1
      2016-04-18 02:22:02
      Building Amenities - Garage - Garden - fitness...
      East 49th Street
      40.7539
      -73.9677
      3275
      333 East 49th Street
      low
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      4
      1.0
      4
      2016-04-28 01:32:41
      Beautifully renovated 3 bedroom flex 4 bedroom...
      West 143rd Street
      40.8241
      -73.9493
      3350
      500 West 143rd Street
      low
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

5 rows × 34 columns

```python
df['created'] = pd.to_datetime(df['created'], infer_datetime_format=True);
df['month'] = df['created'].dt.month;
df.head()
```

      bathrooms
      bedrooms
      created
      description
      display_address
      latitude
      longitude
      price
      street_address
      interest_level
      ...
      balcony
      swimming_pool
      new_construction
      exclusive
      terrace
      loft
      garden_patio
      common_outdoor_space
      wheelchair_access
      month

      0
      1.5
      3
      2016-06-24 07:54:24
      A Brand New 3 Bedroom 1.5 bath ApartmentEnjoy ...
      Metropolitan Avenue
      40.7145
      -73.9425
      3000
      792 Metropolitan Avenue
      medium
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      6

      1
      1.0
      2
      2016-06-12 12:19:27

      Columbus Avenue
      40.7947
      -73.9667
      5465
      808 Columbus Avenue
      low
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      6

      2
      1.0
      1
      2016-04-17 03:26:41
      Top Top West Village location, beautiful Pre-w...
      W 13 Street
      40.7388
      -74.0018
      2850
      241 W 13 Street
      high
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      4

      3
      1.0
      1
      2016-04-18 02:22:02
      Building Amenities - Garage - Garden - fitness...
      East 49th Street
      40.7539
      -73.9677
      3275
      333 East 49th Street
      low
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      4

      4
      1.0
      4
      2016-04-28 01:32:41
      Beautifully renovated 3 bedroom flex 4 bedroom...
      West 143rd Street
      40.8241
      -73.9493
      3350
      500 West 143rd Street
      low
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      4

5 rows × 35 columns

```python
train = df[df['month'] < 6];
test = df[df['month'] == 6];
assert df.shape[0] == train.shape[0] + test.shape[0]
```

```python
df.columns
```

    Index(['bathrooms', 'bedrooms', 'created', 'description', 'display_address',
           'latitude', 'longitude', 'price', 'street_address', 'interest_level',
           'elevator', 'cats_allowed', 'hardwood_floors', 'dogs_allowed',
           'doorman', 'dishwasher', 'no_fee', 'laundry_in_building',
           'fitness_center', 'pre-war', 'laundry_in_unit', 'roof_deck',
           'outdoor_space', 'dining_room', 'high_speed_internet', 'balcony',
           'swimming_pool', 'new_construction', 'exclusive', 'terrace', 'loft',
           'garden_patio', 'common_outdoor_space', 'wheelchair_access'],
          dtype='object')

```python
main_customer_requirements = ['bathrooms', 'bedrooms', 'dishwasher', 'laundry_in_building', 'laundry_in_unit', 'high_speed_internet'];
target = 'price';

x_train = train[main_customer_requirements];
y_train = train[target];

x_test = test[main_customer_requirements];
y_test = test[target];
```

```python
customerModel = LinearRegression();
customerModel.fit(x_train, y_train)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,
             normalize=False)

```python
y_pred = customerModel.predict(x_test);
y_pred.shape
```

    (16785,)

```python
customerModel.coef_
```

    array([1421.83475493,  408.66550573,  237.13122324,   59.02091715,
            378.02911758,  -13.25821541])

```python
customerModel.intercept_
```

    977.3088601415961

```python
from sklearn.metrics import mean_squared_error;
from sklearn.metrics import mean_absolute_error;
from sklearn.metrics import r2_score;
import numpy as np

sqrt = np.sqrt(mean_squared_error(y_test, y_pred))
mae = mean_absolute_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
rmse, mae, r2
```

    (741.3110985694503, 741.3110985694503, 0.47948851127734315)


# Lab 7: Qubit Spectroscopy

In this lab, you will take what you learned about the interactions between qubits and resonators to perform transmon spectroscopy with the pulse simulator.

### Installing Necessary Packages
Before we begin, you will need to install some prerequisites into your environment. Run the cell below to complete these installations. At the end, the cell outputs will be cleared.

```python
!pip install -U -r grading_tools/requirements.txt

from IPython.display import clear_output
clear_output()
```

      Installing backend dependencies: finished with status 'done'
        Preparing wheel metadata: started
        Preparing wheel metadata: finished with status 'done'
    Requirement already up-to-date: qiskit==0.19 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from -r grading_tools/requirements.txt (line 1)) (0.19.0)
    Requirement already up-to-date: scipy==1.5.1 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from -r grading_tools/requirements.txt (line 2)) (1.5.1)
    Requirement already up-to-date: sympy==1.6.1 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from -r grading_tools/requirements.txt (line 3)) (1.6.1)
    Requirement already satisfied, skipping upgrade: cython>=0.27.1 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-aer==0.6.0->-r grading_tools/requirements.txt (line 4)) (0.29.21)
    Requirement already satisfied, skipping upgrade: numpy>=1.16.3; python_version > "3.5" in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-aer==0.6.0->-r grading_tools/requirements.txt (line 4)) (1.19.0)
    Requirement already satisfied, skipping upgrade: pybind11>=2.4 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-aer==0.6.0->-r grading_tools/requirements.txt (line 4)) (2.5.0)
    Requirement already satisfied, skipping upgrade: qiskit-terra>=0.12.0 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-aer==0.6.0->-r grading_tools/requirements.txt (line 4)) (0.14.0)
    Requirement already satisfied, skipping upgrade: qiskit-aqua==0.7.0 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (0.7.0)
    Requirement already satisfied, skipping upgrade: qiskit-ibmq-provider==0.7.0 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (0.7.0)
    Requirement already satisfied, skipping upgrade: qiskit-ignis==0.3.0 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (0.3.0)
    Requirement already satisfied, skipping upgrade: mpmath>=0.19 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from sympy==1.6.1->-r grading_tools/requirements.txt (line 3)) (1.1.0)
    Requirement already satisfied, skipping upgrade: fastjsonschema>=2.10 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-terra>=0.12.0->qiskit-aer==0.6.0->-r grading_tools/requirements.txt (line 4)) (2.14.4)
    Requirement already satisfied, skipping upgrade: python-constraint>=1.4 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-terra>=0.12.0->qiskit-aer==0.6.0->-r grading_tools/requirements.txt (line 4)) (1.4.0)
    Requirement already satisfied, skipping upgrade: marshmallow-polyfield<6,>=5.7 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-terra>=0.12.0->qiskit-aer==0.6.0->-r grading_tools/requirements.txt (line 4)) (5.9)
    Requirement already satisfied, skipping upgrade: networkx>=2.2; python_version > "3.5" in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-terra>=0.12.0->qiskit-aer==0.6.0->-r grading_tools/requirements.txt (line 4)) (2.4)
    Requirement already satisfied, skipping upgrade: retworkx>=0.3.2 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-terra>=0.12.0->qiskit-aer==0.6.0->-r grading_tools/requirements.txt (line 4)) (0.4.0)
    Requirement already satisfied, skipping upgrade: psutil>=5 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-terra>=0.12.0->qiskit-aer==0.6.0->-r grading_tools/requirements.txt (line 4)) (5.7.2)
    Requirement already satisfied, skipping upgrade: dill>=0.3 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-terra>=0.12.0->qiskit-aer==0.6.0->-r grading_tools/requirements.txt (line 4)) (0.3.2)
    Requirement already satisfied, skipping upgrade: ply>=3.10 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-terra>=0.12.0->qiskit-aer==0.6.0->-r grading_tools/requirements.txt (line 4)) (3.11)
    Requirement already satisfied, skipping upgrade: jsonschema>=2.6 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-terra>=0.12.0->qiskit-aer==0.6.0->-r grading_tools/requirements.txt (line 4)) (3.2.0)
    Requirement already satisfied, skipping upgrade: python-dateutil>=2.8.0 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-terra>=0.12.0->qiskit-aer==0.6.0->-r grading_tools/requirements.txt (line 4)) (2.8.1)
    Requirement already satisfied, skipping upgrade: marshmallow<4,>=3 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-terra>=0.12.0->qiskit-aer==0.6.0->-r grading_tools/requirements.txt (line 4)) (3.7.1)

      -- The C compiler identification is unknown
      CMake Error at CMakeLists.txt:3 (ENABLE_LANGUAGE):
        The CMAKE_C_COMPILER:

          cl

        is not a full path and was not found in the PATH.

        To use the NMake generator with Visual C++, cmake must be run from a shell
        that can use the compiler cl from the command line.  This environment is
        unable to invoke the cl compiler.  To fix this problem, run cmake from the
        Visual Studio Command Prompt (vcvarsall.bat).

        Tell CMake where to find the compiler by setting either the environment
        variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to
        the compiler, or to the compiler name if it is in the PATH.

      -- Configuring incomplete, errors occurred!
      See also "C:/Users/Hernan/AppData/Local/Temp/pip-req-build-7yy62sb7/_cmake_test_compile/build/CMakeFiles/CMakeOutput.log".
      See also "C:/Users/Hernan/AppData/Local/Temp/pip-req-build-7yy62sb7/_cmake_test_compile/build/CMakeFiles/CMakeError.log".
      Not searching for unused variables given on the command line.
      -- The C compiler identification is unknown
      CMake Error at CMakeLists.txt:3 (ENABLE_LANGUAGE):
        The CMAKE_C_COMPILER:

          cl

        is not a full path and was not found in the PATH.

        To use the JOM generator with Visual C++, cmake must be run from a shell
        that can use the compiler cl from the command line.  This environment is
        unable to invoke the cl compiler.  To fix this problem, run cmake from the
        Visual Studio Command Prompt (vcvarsall.bat).

        Tell CMake where to find the compiler by setting either the environment
        variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to
        the compiler, or to the compiler name if it is in the PATH.

      -- Configuring incomplete, errors occurred!
      See also "C:/Users/Hernan/AppData/Local/Temp/pip-req-build-7yy62sb7/_cmake_test_compile/build/CMakeFiles/CMakeOutput.log".
      See also "C:/Users/Hernan/AppData/Local/Temp/pip-req-build-7yy62sb7/_cmake_test_compile/build/CMakeFiles/CMakeError.log".

      --------------------------------------------------------------------------------
      -- Trying "Ninja (Visual Studio 15 2017 Win64 v141)" generator
      --------------------------------
      ---------------------------
      ----------------------
      -----------------
      ------------
      -------
      --
      --
      -------
      ------------

    Requirement already satisfied, skipping upgrade: fastdtw in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (0.3.4)
    Requirement already satisfied, skipping upgrade: setuptools>=40.1.0 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (49.2.0.post20200714)
    Requirement already satisfied, skipping upgrade: dlx in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (1.0.4)
    Requirement already satisfied, skipping upgrade: h5py in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (2.10.0)
    Requirement already satisfied, skipping upgrade: scikit-learn>=0.20.0 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (0.23.1)
    Requirement already satisfied, skipping upgrade: quandl in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (3.5.1)
    Requirement already satisfied, skipping upgrade: docplex in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (2.15.194)
    Requirement already satisfied, skipping upgrade: websockets<8,>=7 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-ibmq-provider==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (7.0)
    Requirement already satisfied, skipping upgrade: urllib3>=1.21.1 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-ibmq-provider==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (1.25.9)
    Requirement already satisfied, skipping upgrade: requests-ntlm>=1.1.0 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-ibmq-provider==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (1.1.0)
    Requirement already satisfied, skipping upgrade: requests>=2.19 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-ibmq-provider==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (2.24.0)
    Requirement already satisfied, skipping upgrade: nest-asyncio!=1.1.0,>=1.0.0 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from qiskit-ibmq-provider==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (1.4.0)
    Requirement already satisfied, skipping upgrade: decorator>=4.3.0 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from networkx>=2.2; python_version > "3.5"->qiskit-terra>=0.12.0->qiskit-aer==0.6.0->-r grading_tools/requirements.txt (line 4)) (4.4.2)
    Requirement already satisfied, skipping upgrade: pyrsistent>=0.14.0 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from jsonschema>=2.6->qiskit-terra>=0.12.0->qiskit-aer==0.6.0->-r grading_tools/requirements.txt (line 4)) (0.16.0)
    Requirement already satisfied, skipping upgrade: attrs>=17.4.0 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from jsonschema>=2.6->qiskit-terra>=0.12.0->qiskit-aer==0.6.0->-r grading_tools/requirements.txt (line 4)) (19.3.0)
    Requirement already satisfied, skipping upgrade: six>=1.11.0 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from jsonschema>=2.6->qiskit-terra>=0.12.0->qiskit-aer==0.6.0->-r grading_tools/requirements.txt (line 4)) (1.15.0)
    Requirement already satisfied, skipping upgrade: threadpoolctl>=2.0.0 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from scikit-learn>=0.20.0->qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (2.1.0)
    Requirement already satisfied, skipping upgrade: joblib>=0.11 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from scikit-learn>=0.20.0->qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (0.16.0)
    Requirement already satisfied, skipping upgrade: inflection>=0.3.1 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from quandl->qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (0.5.0)
    Requirement already satisfied, skipping upgrade: pandas>=0.14 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from quandl->qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (1.0.5)
    Requirement already satisfied, skipping upgrade: more-itertools in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from quandl->qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (8.4.0)
    Requirement already satisfied, skipping upgrade: cryptography>=1.3 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (2.9.2)
    Requirement already satisfied, skipping upgrade: ntlm-auth>=1.0.2 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (1.5.0)
    Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from requests>=2.19->qiskit-ibmq-provider==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (3.0.4)
    Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from requests>=2.19->qiskit-ibmq-provider==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (2020.6.20)
    Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from requests>=2.19->qiskit-ibmq-provider==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (2.10)
    Requirement already satisfied, skipping upgrade: pytz>=2017.2 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from pandas>=0.14->quandl->qiskit-aqua==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (2020.1)
    Requirement already satisfied, skipping upgrade: cffi!=1.11.3,>=1.8 in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (1.14.0)
    Requirement already satisfied, skipping upgrade: pycparser in c:\users\hernan\anaconda3\envs\qiskit\lib\site-packages (from cffi!=1.11.3,>=1.8->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.7.0->qiskit==0.19->-r grading_tools/requirements.txt (line 1)) (2.20)
    Building wheels for collected packages: qiskit-aer
      Building wheel for qiskit-aer (PEP 517): started
      Building wheel for qiskit-aer (PEP 517): finished with status 'error'
    Failed to build qiskit-aer

      -----------------
      ----------------------
      ---------------------------
      --------------------------------
      -- Trying "Ninja (Visual Studio 15 2017 Win64 v141)" generator - failure
      --------------------------------------------------------------------------------

      --------------------------------------------------------------------------------
      -- Trying "Visual Studio 15 2017 Win64 v141" generator
      --------------------------------
      ---------------------------
      ----------------------
      -----------------
      ------------
      -------
      --
      --
      -------
      ------------
      -----------------
      ----------------------
      ---------------------------
      --------------------------------
      -- Trying "Visual Studio 15 2017 Win64 v141" generator - failure
      --------------------------------------------------------------------------------

      --------------------------------------------------------------------------------
      -- Trying "NMake Makefiles (Visual Studio 15 2017 Win64 v141)" generator
      --------------------------------
      ---------------------------
      ----------------------
      -----------------
      ------------
      -------
      --
      --
      -------
      ------------
      -----------------
      ----------------------
      ---------------------------
      --------------------------------
      -- Trying "NMake Makefiles (Visual Studio 15 2017 Win64 v141)" generator - failure
      --------------------------------------------------------------------------------

      --------------------------------------------------------------------------------
      -- Trying "NMake Makefiles JOM (Visual Studio 15 2017 Win64 v141)" generator
      --------------------------------
      ---------------------------
      ----------------------
      -----------------
      ------------
      -------
      --
      --
      -------
      ------------
      -----------------
      ----------------------
      ---------------------------
      --------------------------------
      -- Trying "NMake Makefiles JOM (Visual Studio 15 2017 Win64 v141)" generator - failure
      --------------------------------------------------------------------------------

      ********************************************************************************
      scikit-build could not get a working generator for your system. Aborting build.

      Building windows wheels for Python 3.8 requires Microsoft Visual Studio 2017.
      Get it with "Visual Studio 2017":

        https://visualstudio.microsoft.com/vs/

      ********************************************************************************
      ----------------------------------------
      ERROR: Failed building wheel for qiskit-aer

## Simulating the Transmon as a Duffing Oscillator

As you learned in Lecture 6, the transmon can be understood as a Duffing oscillator specified by a frequency $\nu$, anharmonicity $\alpha$, and drive strength $r$, which results in the Hamiltonian
$$
   \hat{H}_{\rm Duff}/\hbar = 2\pi\nu a^\dagger a + \pi \alpha a^\dagger a(a^\dagger a - 1) + 2 \pi r (a + a^\dagger) \times D(t),
$$

where $D(t)$ is the signal on the drive channel for the qubit, and $a^\dagger$ and $a$ are, respectively, the creation and annihilation operators for the qubit. Note that the drive strength $r$ sets the scaling of the control term, with $D(t)$ assumed to be a complex and unitless number satisfying $|D(t)| \leq 1$.

## Qiskit Pulse Overview

As a brief overview, Qiskit Pulse schedules (experiments) consist of Instructions (i.e., Play) acting on Channels (i.e., the drive channel). Here is a summary table of available Instructions and Channels:

For more detail, this table summarizes the interaction of the channels with the actual quantum hardware:

However, we find it is more instructive to begin with guided programming in Pulse. Below you will learn how to create pulses, schedules, and run experiments on a simulator. These lessons can be immediately applied to actual pulse-enabled quantum hardware, in particular [`ibmq_armonk`](https://www.ibm.com/blogs/research/2019/12/qiskit-openpulse/).

## Let's get started!

In most of the cells below, nothing needs to be modified. **However, you will need to execute the cells by pressing `shift+Enter` in each code block**. In order to keep things tidy and focus on the important aspects of Qiskit Pulse, the following cells make use of methods from the `helper` module. For the gory details, please refer back to the [Lab 7 notebook](lab7-jc-spect-readout.ipynb). Just as in Lab 6, before coming to the discussion of **Sideband Modulation**, the following code blocks

- create backend pulse simulator and instantiate the transmon as a Duffing oscillator of frequency $\sim 5$ GHz
- import libraries for numerics and visualization, and define helpful constants
- create the channels for the pulse schedule and define measurment schedule (we will only work with the drive channel)

```python
# our backend is the Pulse Simulator
from resources import helper
from qiskit.providers.aer import PulseSimulator
backend_sim = PulseSimulator()

# sample duration for pulse instructions
dt = 1e-9

# create the model
duffing_model = helper.get_transmon(dt)

# get qubit frequency from Duffing model
qubit_lo_freq = duffing_model.hamiltonian.get_qubit_lo_from_drift()
```

    ERROR: Could not build wheels for qiskit-aer which use PEP 517 and cannot be installed directly

```python
import numpy as np

# visualization tools
import matplotlib.pyplot as plt
plt.style.use('dark_background')

# unit conversion factors -> all backend properties returned in SI (Hz, sec, etc)
GHz = 1.0e9 # Gigahertz
MHz = 1.0e6 # Megahertz
kHz = 1.0e3 # kilohertz
us = 1.0e-6 # microseconds
ns = 1.0e-9 # nanoseconds
```

### Instantiate channels and create measurement schedule

We will use the same measurement schedule throughout, whereas the drive schedules will vary. This must be built for the simulator, for a real backend we can ask for its default measurement pulse.

```python
from qiskit import pulse
from qiskit.pulse import Play, Acquire
from qiskit.pulse.pulse_lib import GaussianSquare

# qubit to be used throughout the notebook
qubit = 0

### Collect the necessary channels
drive_chan = pulse.DriveChannel(qubit)
meas_chan = pulse.MeasureChannel(qubit)
acq_chan = pulse.AcquireChannel(qubit)

# Construct a measurement schedule and add it to an InstructionScheduleMap
meas_samples = 1200
meas_pulse = GaussianSquare(duration=meas_samples, amp=0.025, sigma=4, width=1150)
measure_sched = Play(meas_pulse, meas_chan) | Acquire(meas_samples, acq_chan, pulse.MemorySlot(qubit))

inst_map = pulse.InstructionScheduleMap()
inst_map.add('measure', [qubit], measure_sched)

# save the measurement/acquire pulse for later
measure = inst_map.get('measure', qubits=[qubit])
```

## Sideband Modulation

Unlike the case of running on an actual device, with the simulator we can only set the (local) oscillator frequency of the drive, $f_{\rm LO}$, to a single value. In order to sweep frequencies to perform spectroscopy, we use a trick called *sideband modulation*, where we modulate our spectroscopy pulse by a sideband frequency $f_{\rm SB}$ so that the pulse applied to the qubit is of (radio) frequency

$$ f_{\rm RF} = f_{\rm LO} + f_{\rm SB}. $$

This is achieved by multiplying each sample amplitude by a complex exponential

$$ d_j^{\rm SB} = \sum_j e^{2\pi f_{\rm SB} t_j} d_j $$

but we will tuck the details away in the `helper` module. The important thing is that we must apply the sideband for each pulse in order to change its frequency.

Now, instead of `assemble`'ing a single schedule with an array of schedule LO's as, we will create a schedule of the same pulse *sidebanded* by an array of sideband frequecies at a fixed LO frequency. Since we are now considering a transmon, we have multiple energy levels we can perform spectroscopy on. We will being with spectroscopy of the $|0\rangle \to |1\rangle$ transition, which is the one used as the qubit, often called the *computational basis*.

```python
from qiskit.pulse import pulse_lib

# the same spect pulse used in every schedule
drive_amp = 0.9
drive_sigma = 16
drive_duration = 128
spec_pulse = pulse_lib.gaussian(duration=drive_duration, amp=drive_amp,
                                sigma=drive_sigma, name=f"Spec drive amplitude = {drive_amp}")

# Construct an np array of the frequencies for our experiment
spec_freqs_GHz = np.arange(5.0, 5.2, 0.005)

# Create the base schedule
# Start with drive pulse acting on the drive channel
spec_schedules = []
for freq in spec_freqs_GHz:
    sb_spec_pulse = helper.apply_sideband(spec_pulse, qubit_lo_freq[0]-freq*GHz, dt)

    spec_schedule = pulse.Schedule(name='SB Frequency = {}'.format(freq))
    spec_schedule += Play(sb_spec_pulse, drive_chan)
    # The left shift `<<` is special syntax meaning to shift the start time of the schedule by some duration
    spec_schedule += measure << spec_schedule.duration
    spec_schedules.append(spec_schedule)
```

```python
spec_schedules[0].draw()
```

```python
from qiskit import assemble

# assemble the schedules into a Qobj
spec01_qobj = assemble(**helper.get_params('spec01', globals()))
```

```python
# run the simulation
spec01_result = backend_sim.run(spec01_qobj, duffing_model).result()
```

```python
# retrieve the data from the experiment
spec01_values = helper.get_values_from_result(spec01_result, qubit)
```

We will fit the spectroscopy signal to a Lorentzian function of the form

$$ \frac{AB}{\pi[(f-f_{01})^2 + B^2]} + C $$

to find the qubit frequency $f_{01}$.

```python
fit_params, y_fit = helper.fit_lorentzian(spec_freqs_GHz, spec01_values, [5, 5, 1, 0])

f01 = fit_params[1]

plt.scatter(spec_freqs_GHz, np.real(spec01_values), color='white') # plot real part of sweep values
plt.plot(spec_freqs_GHz, y_fit, color='red')
plt.xlim([min(spec_freqs_GHz), max(spec_freqs_GHz)])

plt.xlabel("Frequency [GHz]")
plt.ylabel("Measured Signal [a.u.]")
plt.show()

print("01 Spectroscopy yields %f GHz"%f01)
```

# Exercise 1: Spectroscopy of 1->2 transition

In order to observe the transition between the $|1\rangle$ and $|2\rangle$ states of the transmon, we must apply an $X_\pi$ pulse to transition the qubit from $|0\rangle$ to $|1\rangle$ first. Because we are using the simulator, we must first define our $X_\pi$ pulse from the Rabi experiment in Lab 6.

```python
x180_amp = 0.629070 #from lab 6 Rabi experiment

x_pulse = pulse_lib.gaussian(duration=drive_duration,
                             amp=x180_amp,
                             sigma=drive_sigma,
                             name='x_pulse')
```

The anharmonicity of our transmon qubits is typically around $-300$ MHz, so we will sweep around that value.

```python
anharmonicity_guess_GHz = -0.3

def build_spec12_pulse_schedule(freq):
    sb12_spec_pulse = helper.apply_sideband(spec_pulse, (freq + anharmonicity_guess_GHz)*GHz, dt)

    ### create a 12 spectroscopy pulse schedule spec12_schedule (already done)
    ### play an x pulse on the drive channel
    ### play sidebanded spec pulse on the drive channel
    ### add measurement pulse to schedule

    spec12_schedule = pulse.Schedule()

    ### WRITE YOUR CODE BETWEEN THESE LINES - START
    spec12_schedule += Play(x_pulse, drive_chan)
    spec12_schedule += Play(sb12_spec_pulse, drive_chan)
    spec12_schedule += measure << spec12_schedule.duration
    ### WRITE YOUR CODE BETWEEN THESE LINES - END

    return spec12_schedule
```

```python
sb_freqs_GHz = np.arange(-.1, .1, 0.005) # sweep +/- 100 MHz around guess

# now vary the sideband frequency for each spec pulse
spec_schedules = []
for freq in sb_freqs_GHz:
    spec_schedules.append(build_spec12_pulse_schedule(freq))
```

```python
spec_schedules[0].draw()
```

```python
# assemble the schedules into a Qobj
spec12_qobj = assemble(**helper.get_params('spec12', globals()))
answer1 = spec12_qobj
```

```python
# run the simulation
spec12_result = backend_sim.run(spec12_qobj, duffing_model).result()
```

```python
# retrieve the data from the experiment
spec12_values = helper.get_values_from_result(spec12_result, qubit)
```

We will again fit the spectroscopy signal to a Lorentzian function of the form

$$ \frac{AB}{\pi[(f-f_{12})^2 + B^2]} + C $$

to find the frequency of the $|1\rangle \to |2\rangle$ transition $f_{12}$.

```python
anharm_offset = qubit_lo_freq[0]/GHz + anharmonicity_guess_GHz

fit_params, y_fit = helper.fit_lorentzian(anharm_offset + sb_freqs_GHz, spec12_values, [5, 4.5, .1, 3])

f12 = fit_params[1]

plt.scatter(anharm_offset + sb_freqs_GHz, np.real(spec12_values), color='white') # plot real part of sweep values
plt.plot(anharm_offset + sb_freqs_GHz, y_fit, color='red')
plt.xlim([anharm_offset + min(sb_freqs_GHz), anharm_offset + max(sb_freqs_GHz)])

plt.xlabel("Frequency [GHz]")
plt.ylabel("Measured Signal [a.u.]")
plt.show()

print("12 Spectroscopy yields %f GHz"%f12)
print("Measured transmon anharmonicity is %f MHz"%((f12-f01)*GHz/MHz))
```

**Help us improve our educational tools by submitting your code**<br>
If you would like to help us learn how to improve our educational materials and offerings, you can opt in to send us a copy of your Jupyter notebook. By executing the cell below, you consent to sending us the code in your Jupyter notebook. All of the personal information will be anonymized.

```python
from IPython.display import display, Javascript;display(Javascript('IPython.notebook.save_checkpoint();'));
from grading_tools import send_code;send_code('ex1.ipynb')
```

# Additional Resources

- The Qiskit textbook sections that cover this material are
  - [Circuit Quantum Electrodynamics](https://qiskit.org/textbook/ch-quantum-hardware/cQED-JC-SW.html)
  - [Accessing Higher Energy States](https://qiskit.org/textbook/ch-quantum-hardware/accessing_higher_energy_states.html)

- Watch the videos
  - [Quantum Coding with Lauren Capelluto](https://www.youtube.com/watch?v=ZvipHRY-URs)
  - ["Qiskit Pulse: Programming Quantum Computers Through the Cloud with Pulses"](https://www.youtube.com/watch?v=V_as5PufUiU) webinar at CQT by yours truly


#  CS109B Data Science 2: Advanced Topics in Data Science

## Lab 4 - Bayesian Analysis

**Harvard University**

**Spring 2020**

**Instructors:** Mark Glickman, Pavlos Protopapas, and Chris Tanner

**Lab Instructors:** Chris Tanner and Eleni Angelaki Kaxiras

**Content:** Eleni Angelaki Kaxiras

---

```python
## RUN THIS CELL TO PROPERLY HIGHLIGHT THE EXERCISES
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2019-CS109B/master/content/styles/cs109.css").text
HTML(styles)
```

```python
import pymc3 as pm
from pymc3 import summary
```

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as stats
import pandas as pd
%matplotlib inline

import warnings
warnings.filterwarnings('ignore')
```

```python
print('Running on PyMC3 v{}'.format(pm.__version__))
```

    Running on PyMC3 v3.8

```javascript
%%javascript
IPython.OutputArea.auto_scroll_threshold = 20000;
```

    <IPython.core.display.Javascript object>

## Learning Objectives

By the end of this lab, you should be able to:
* Understand how probability distributions work.
* Apply Bayes Rule in calculating probabilities.
* Understand how to apply Bayesian analysis using PyMC3
* Avoid getting fired when talking to your Bayesian employer.

**This lab corresponds to Lectures 6, 7, and 8, and maps to Homework 3.**

## Table of Contents

1. The Bayesian Way of Thinking or Is this a Fair Coin?
2. [Intro to `pyMC3`](#pymc3).
3. [Bayesian Linear Regression](#blr).
4. [Try this at Home: Example on Mining Disasters](#no4).

## 1. The Bayesian way of Thinking

```
Here is my state of knowledge about the situation. Here is some data, I am now going to revise my state of knowledge.
```

Table Exercise: Discuss the statement above with your table mates and make sure everyone understands what it means and what constitutes Bayesian way of thinking. Finally, count the Bayesians among you.

### A. Bayes Rule

\begin{equation}
\label{eq:bayes}
P(A|\textbf{B}) = \frac{P(\textbf{B} |A) P(A) }{P(\textbf{B})}
\end{equation}

$P(A|\textbf{B})$ is the **posterior** distribution, prob(hypothesis | data)

$P(\textbf{B} |A)$ is the **likelihood** function, how probable is my data **B** for different values of the parameters

$P(A)$ is the marginal probability to observe the data, called the **prior**, this captures our belief about the data before observing it.

$P(\textbf{B})$ is the marginal distribution (sometimes called marginal likelihood)

Table Exercise: Solve the Monty Hall Paradox using Bayes Rule.

You are invited to play a game. There are 3 doors behind **one** of which are the keys to a brand new red Tesla. There is a goat behind each of the other two.

You are asked to pick one door, and let's say you pick **Door1**. The host knows where the keys are. Of the two remaining closed doors, he will always open the door that has a goat behind it. He'll say "I will do you a favor and open **Door2**". So he opens Door2 inside which there is, of course, a goat. He now asks you, do you want to open the initial Door you chose or change to **Door3**? Generally, in this game, when you are presented with this choice should you swap the doors?

**Initial Steps:**
- Start by defining the `events` of this probabilities game. One definition is:

    - $A_i$: car is behind door $i$

    - $B_i$ host opens door $i$

$i\in[1,2,3]$

- In more math terms, the question is: is the probability that the price is behind **Door 1** higher than the probability that the price is behind **Door2**, given that an event **has occured**?

### B. Bayes Rule written with Probability Distributions

We have data that we believe come from an underlying distribution of unknown parameters. If we find those parameters, we know everything about the process that generated this data and we can make inferences (create new data).

\begin{equation}
\label{eq:bayes}
P(\theta|\textbf{D}) = \frac{P(\textbf{D} |\theta) P(\theta) }{P(\textbf{D})}
\end{equation}

#### But what is $\theta \;$?

$\theta$ is an unknown yet fixed set of parameters. In Bayesian inference we express our belief about what $\theta$ might be and instead of trying to guess $\theta$ exactly, we look for its **probability distribution**. What that means is that we are looking for the **parameters** of that distribution. For example, for a Poisson distribution our $\theta$ is only $\lambda$. In a normal distribution, our $\theta$ is often just $\mu$ and $\sigma$.

### C. A review of Common Probability Distributions

#### Discrete Distributions

The random variable has a **probability mass function (pmf)** which measures the probability that our random variable will take a specific value $y$, denoted $P(Y=y)$.

- **Bernoulli** (binary outcome, success has probability $\theta$, $one$ trial):
$
P(Y=k) =  \theta^k(1-\theta)^{1-k}
$

- **Binomial** (binary outcome, success has probability $\theta$, $n$ trials):
\begin{equation}
P(Y=k) =  {{n}\choose{k}} \cdot \theta^k(1-\theta)^{n-k}
\end{equation}

*Note*: Binomial(1,$p$) = Bernouli($p$)

- **Negative Binomial**

- **Poisson** (counts independent events occurring at a rate)
\begin{equation}
P\left( Y=y|\lambda \right) = \frac{{e^{ - \lambda } \lambda ^y }}{{y!}}
\end{equation}
y = 0,1,2,...

- **Discrete Uniform**

- **Categorical, or Multinulli** (random variables can take any of K possible categories, each having its own probability; this is a generalization of the Bernoulli distribution for a discrete variable with more than two possible outcomes, such as the roll of a die)

- **Dirichlet-multinomial** (a generalization of the beta distribution for many variables)

#### Continuous Distributions

The random variable has a **probability density function (pdf)**.
- **Uniform** (variable equally likely to be near each value in interval $(a,b)$)
\begin{equation}
P(X = x) = \frac{1}{b - a}
\end{equation}
anywhere within the interval $(a, b)$, and zero elsewhere.

- **Normal** (a.k.a. Gaussian)
\begin{equation}
X \sim  \mathcal{N}(\mu,\,\sigma^{2})
\end{equation}

    A Normal distribution can be parameterized either in terms of precision $\tau$ or standard deviation ($\sigma^{2}$. The link between the two is given by
\begin{equation}
\tau = \frac{1}{\sigma^{2}}
\end{equation}
 - Mean $\mu$
 - Variance $\frac{1}{\tau}$ or $\sigma^{2}$
 - Parameters: `mu: float`, `sigma: float` or `tau: float`

- **Beta** (variable ($\theta$) taking on values in the interval $[0,1]$, and parametrized by two positive parameters, $\alpha$ and $\beta$ that control the shape of the distribution.

*Note:*Beta is a good distribution to use for priors (beliefs) because its range is $[0,1]$ which is the natural range for a probability and because we can model a wide range of functions by changing the $\alpha$ and $\beta$ parameters.

\begin{equation}
\label{eq:beta}
P(\theta) = \frac{1}{B(\alpha, \beta)} {\theta}^{\alpha - 1} (1 - \theta)^{\beta - 1} \propto {\theta}^{\alpha - 1} (1 - \theta)^{\beta - 1}
\end{equation}

where the normalisation constant, $B$, is a beta function of $\alpha$ and $\beta$,

\begin{equation}
B(\alpha, \beta) = \int_{t=0}^1 t^{\alpha - 1} (1 - t)^{\beta - 1} dt.
\end{equation}

- **Exponential**

- **Gamma**

 #### Code Resources:
 - Statistical Distributions in numpy/scipy: [scipy.stats](https://docs.scipy.org/doc/scipy/reference/stats.html)
 - Statistical Distributions in pyMC3: [distributions in PyMC3](https://docs.pymc.io/api/distributions.html) (we will see those below).

Exercise: Plot a Discrete variable

Change the value of $\mu$ in the Poisson PMF and see how the plot changes. Remember that the y-axis in a discrete probability distribution shows the probability of the random variable having a specific value in the x-axis.

\begin{equation}
P\left( X=k \right) = \frac{{e^{ - \mu } \mu ^k }}{{k!}}
\end{equation}

**stats.poisson.pmf(x, mu)** $\mu$(mu) is our $\theta$ in this case.

```python
plt.style.use('seaborn-darkgrid')
x = np.arange(0, 30)
for m in [0.5, 3, 8]:
    pmf = stats.poisson.pmf(x, m)
    plt.plot(x, pmf, 'o', alpha=0.5, label='$\mu$ = {}'.format(m))
plt.xlabel('random variable', fontsize=12)
plt.ylabel('probability', fontsize=12)
plt.legend(loc=1)
plt.ylim=(-0.1)
plt.show()
```

```python
# same for binomial
plt.style.use('seaborn-darkgrid')
x = np.arange(0, 22)
ns = [10, 17]
ps = [0.5, 0.7]
for n, p in zip(ns, ps):
    pmf = stats.binom.pmf(x, n, p)
    plt.plot(x, pmf, 'o', alpha=0.5, label='n = {}, p = {}'.format(n, p))
plt.xlabel('x', fontsize=14)
plt.ylabel('f(x)', fontsize=14)
plt.legend(loc=1)
plt.show()
```

```python
# discrete uniform
plt.style.use('seaborn-darkgrid')
ls = [0]
us = [3] # watch out, this number can only be integer!
for l, u in zip(ls, us):
    x = np.arange(l, u+1)
    pmf = [1.0 / (u - l + 1)] * len(x)
    plt.plot(x, pmf, '-o', label='lower = {}, upper = {}'.format(l, u))
plt.xlabel('x', fontsize=12)
plt.ylabel('probability P(x)', fontsize=12)
plt.legend(loc=1)
plt.show()
```

Exercise: Plot a continuous variable

Change the value of $\mu$ in the Uniform PDF and see how the plot changes.

Remember that the y-axis in a continuous probability distribution does not shows the actual probability of the random variable having a specific value in the x-axis because that probability is zero!. Instead, to see the probability that the variable is within a small margin we look at the integral below the curve of the PDF.

The uniform is often used as a noninformative prior.

```
Uniform - numpy.random.uniform(a=0.0, b=1.0, size)
```

$\alpha$ and $\beta$ are our parameters. `size` is how many tries to perform.
Our $\theta$ is basically the combination of the parameters a,b. We can also call it
\begin{equation}
\mu = (a+b)/2
\end{equation}

```python
from scipy.stats import uniform

r = uniform.rvs(size=1000)
plt.plot(r, uniform.pdf(r),'r-', lw=5, alpha=0.6, label='uniform pdf')
plt.hist(r, density=True, histtype='stepfilled', alpha=0.2)
plt.ylabel(r'probability density')
plt.xlabel(f'random variable')
plt.legend(loc='best', frameon=False)
plt.show()
```

```python
from scipy.stats import beta

alphas = [0.5, 1.5, 3.0]
betas = [0.5, 1.5, 3.0]
x = np.linspace(0, 1, 1000)
colors = ['red', 'green', 'blue']

fig, ax = plt.subplots(figsize=(8, 5))

for a, b, colors in zip(alphas, betas, colors):
    dist = beta(a, b)
    plt.plot(x, dist.pdf(x), c=colors,
             label=f'a={a}, b={b}')

ax.set_ylim(0, 3)

ax.set_xlabel(r'$\theta$')
ax.set_ylabel(r'$p(\theta|\alpha,\beta)$')
ax.set_title('Beta Distribution')

ax.legend(loc='best')
fig.show();
```

```python
plt.style.use('seaborn-darkgrid')
x = np.linspace(-5, 5, 1000)
mus = [0., 0., 0., -2.]
sigmas = [0.4, 1., 2., 0.4]
for mu, sigma in zip(mus, sigmas):
    pdf = stats.norm.pdf(x, mu, sigma)
    plt.plot(x, pdf, label=r'$\mu$ = '+ f'{mu},' + r'$\sigma$ = ' + f'{sigma}')
plt.xlabel('random variable', fontsize=12)
plt.ylabel('probability density', fontsize=12)
plt.legend(loc=1)
plt.show()
```

```python
plt.style.use('seaborn-darkgrid')
x = np.linspace(-5, 5, 1000)
mus = [0., 0., 0., -2.] # mean
sigmas = [0.4, 1., 2., 0.4] # std
for mu, sigma in zip(mus, sigmas):
      plt.plot(x, uniform.pdf(x, mu, sigma), lw=5, alpha=0.4, \
               label=r'$\mu$ = '+ f'{mu},' + r'$\sigma$ = ' + f'{sigma}')
plt.xlabel('random variable', fontsize=12)
plt.ylabel('probability density', fontsize=12)
plt.legend(loc=1)
plt.show()
```

### D. Is this a Fair Coin?

We do not want to promote gambling but let's say you visit the casino in **Monte Carlo**. You want to test your theory that casinos are dubious places where coins have been manipulated to have a larger probability for tails. So you will try to estimate how fair a coin is based on 100 flips.

You begin by flipping the coin. You get either Heads ($H$) or Tails ($T$) as our observed data and want to see if your posterior probabilities change as you obtain more data, that is, more coin flips. A nice way to visualize this is to plot the posterior probabilities as we observe more flips (data).

We will be using Bayes rule. $\textbf{D}$ is our data.

\begin{equation}
\label{eq:bayes}
P(\theta|\textbf{D}) = \frac{P(\textbf{D} |\theta) P(\theta) }{P(\textbf{D})}
\end{equation}

In the case of a coin toss when we observe $k$ heads in $n$ tosses:
\begin{equation}
\label{eq:bayes}
P(\theta|\textbf{k}) = Beta(\alpha + \textbf{k}, \beta + n - \textbf{k})
\end{equation}

we can say that $\alpha$ and $\beta$ play the roles of a "prior number of heads" and "prior number of tails".

```python
# play with the priors - here we manually set them but we could be sampling from a separate Beta
trials = np.array([0, 1, 3, 5, 10, 15, 20, 100, 200, 300])
heads = np.array([0, 1, 2, 4, 8, 10, 10, 50, 180, 150])
x = np.linspace(0, 1, 100)

# for simplicity we set a,b=1

plt.figure(figsize=(10,8))
for k, N in enumerate(trials):
    sx = plt.subplot(len(trials)/2, 2, k+1)
    posterior = stats.beta.pdf(x, 1 + heads[k], 1 + trials[k] - heads[k])
    plt.plot(x, posterior, alpha = 0.5, label=f'{trials[k]} tosses\n {heads[k]} heads');
    plt.fill_between(x, 0, posterior, color="#348ABD", alpha=0.4)
    plt.legend(loc='upper left', fontsize=10)
    plt.legend()
    plt.autoscale(tight=True)

plt.suptitle("Posterior probabilities for coin flips", fontsize=15);
plt.tight_layout()
plt.subplots_adjust(top=0.88)
```

 [Top](#top)

## 2. Introduction to `pyMC3`

PyMC3 is a Python library for programming Bayesian analysis, and more specifically, data creation, model definition, model fitting, and posterior analysis. It uses the concept of a `model` which contains assigned parametric statistical distributions to unknown quantities in the model. Within models we define random variables and their distributions. A distribution requires at least a `name` argument, and other `parameters` that define it. You may also use the `logp()` method in the model to build the model log-likelihood function. We define and fit the model.

PyMC3 includes a comprehensive set of pre-defined statistical distributions that can be used as model building blocks. Although they are not meant to be used outside of a `model`, you can invoke them by using the prefix `pm`, as in `pm.Normal`.

#### Markov Chain Monte Carlo (MCMC) Simulations

PyMC3 uses the **No-U-Turn Sampler (NUTS)** and the **Random Walk Metropolis**, two Markov chain Monte Carlo (MCMC) algorithms for sampling in posterior space. Monte Carlo gets into the name because when we sample in posterior space, we choose our next move via a pseudo-random process. NUTS is a sophisticated algorithm that can handle a large number of unknown (albeit continuous) variables.

```python
with pm.Model() as model:
    z = pm.Normal('z', mu=0., sigma=5.)
    x = pm.Normal('x', mu=z, sigma=1., observed=5.)
print(x.logp({'z': 2.5}))
print(z.random(10, 100)[:10])
```

    -4.043938533204672
    [ 8.01444447  1.23006802  1.15969301 -0.93252273  5.54960832 -2.59564893
      2.21650584 -6.0796205  -2.26857049  2.37727101]

**References**:

- *Salvatier J, Wiecki TV, Fonnesbeck C. 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science 2:e55* [(https://doi.org/10.7717/peerj-cs.55)](https://doi.org/10.7717/peerj-cs.55)
- [Distributions in PyMC3](https://docs.pymc.io/api/distributions.html)
- [More Details on Distributions](https://docs.pymc.io/developer_guide.html)

Information about PyMC3 functions including descriptions of distributions, sampling methods, and other functions, is available via the `help` command.

```python
#help(pm.Poisson)
```

 [Top](#top)

## 3. Bayesian Linear Regression

Let's say we want to predict outcomes Y as normally distributed observations with an expected value $mu$ that is a linear function of two predictor variables, $\bf{x}_1$ and $\bf{x}_2$.

\begin{equation}
\mu = \alpha + \beta_1 \bf{x}_1 + \beta_2 x_2
\end{equation}

\begin{equation}
Y \sim  \mathcal{N}(\mu,\,\sigma^{2})
\end{equation}

where $\sigma^2$ represents the measurement error.

In this example, we will use $\sigma^2 = 10$

We also choose the parameters as normal distributions:

\begin{eqnarray}
\alpha \sim  \mathcal{N}(0,\,10) \\
\beta_i \sim  \mathcal{N}(0,\,10) \\
\sigma^2 \sim  |\mathcal{N}(0,\,10)|
\end{eqnarray}

We will artificially create the data to predict on. We will then see if our model predicts them correctly.

```python
# Initialize random number generator
np.random.seed(123)

# True parameter values
alpha, sigma = 1, 1
beta = [1, 2.5]

# Size of dataset
size = 100

# Predictor variable
X1 = np.linspace(0, 1, size)
X2 = np.linspace(0,.2, size)

# Simulate outcome variable
Y = alpha + beta[0]*X1 + beta[1]*X2 + np.random.randn(size)*sigma

fig, ax = plt.subplots(1,2, figsize=(10,6), sharex=True)
ax[0].scatter(X1,Y)
ax[1].scatter(X2,Y)
ax[0].set_xlabel(r'$x_1$', fontsize=14)
ax[0].set_ylabel(r'$Y$', fontsize=14)
ax[1].set_xlabel(r'$x_2$', fontsize=14)
ax[1].set_ylabel(r'$Y$', fontsize=14)
```

```python
from pymc3 import Model, Normal, HalfNormal

basic_model = Model()

with basic_model:

    # Priors for unknown model parameters, specifically create stochastic random variables
    # with Normal prior distributions for the regression coefficients,
    # and a half-normal distribution for the standard deviation of the observations, σ.
    alpha = Normal('alpha', mu=0, sd=10)
    beta = Normal('beta', mu=0, sd=10, shape=2)
    sigma = HalfNormal('sigma', sd=1)

    # Expected value of outcome - posterior
    mu = alpha + beta[0]*X1 + beta[1]*X2

    # Likelihood (sampling distribution) of observations
    Y_obs = Normal('Y_obs', mu=mu, sd=sigma, observed=Y)
```

```python
# model fitting with sampling
from pymc3 import NUTS, sample, find_MAP
from scipy import optimize

with basic_model:

    # obtain starting values via MAP
    start = find_MAP(fmin=optimize.fmin_powell)

    # instantiate sampler
    step = NUTS(scaling=start)

    # draw 2000 posterior samples
    trace = sample(2000, step, start=start)
```

    logp = -164.5:   5%|▌         | 270/5000 [00:00<00:01, 2401.04it/s]

    Optimization terminated successfully.
             Current function value: 164.496957
             Iterations: 6
             Function evaluations: 271

    logp = -164.5:   5%|▌         | 271/5000 [00:00<00:12, 392.25it/s]
    Multiprocess sampling (2 chains in 2 jobs)
    NUTS: [sigma, beta, alpha]
    Sampling 2 chains, 0 divergences: 100%|██████████| 5000/5000 [00:07<00:00, 632.32draws/s]
    The number of effective samples is smaller than 10% for some parameters.

```python
from pymc3 import traceplot

traceplot(trace);
```

```python
results = pm.summary(trace,
                     var_names=['alpha', 'beta', 'sigma'])
results
```

This linear regression example is from the original paper on PyMC3: *Salvatier J, Wiecki TV, Fonnesbeck C. 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science 2:e55 https://doi.org/10.7717/peerj-cs.55*

 [Top](#top)

## 4. Try this at Home: Example on Mining Disasters
We will go over the classical `mining disasters from 1851 to 1962` dataset.

This example is from the [pyMC3 Docs](https://docs.pymc.io/notebooks/getting_started.html).

```python
import pandas as pd
disaster_data = pd.Series([4, 5, 4, 0, 1, 4, 3, 4, 0, 6, 3, 3, 4, 0, 2, 6,
                           3, 3, 5, 4, 5, 3, 1, 4, 4, 1, 5, 5, 3, 4, 2, 5,
                           2, 2, 3, 4, 2, 1, 3, np.nan, 2, 1, 1, 1, 1, 3, 0, 0,
                           1, 0, 1, 1, 0, 0, 3, 1, 0, 3, 2, 2, 0, 1, 1, 1,
                           0, 1, 0, 1, 0, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2,
                           3, 3, 1, np.nan, 2, 1, 1, 1, 1, 2, 4, 2, 0, 0, 1, 4,
                           0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1])
fontsize = 12
years = np.arange(1851, 1962)
plt.figure(figsize=(10,5))
#plt.scatter(years, disaster_data);
plt.bar(years, disaster_data)
plt.ylabel('Disaster count', size=fontsize)
plt.xlabel('Year', size=fontsize);
plt.title('Was there a Turning Point in Mining disasters from 1851 to 1962?', size=15);
```

#### Building the model

**Step1:** We choose the probability model for our experiment. Occurrences of disasters in the time series is thought to follow a **Poisson** process with a large **rate** parameter in the early part of the time series, and from one with a smaller **rate** in the later part. We are interested in locating the change point in the series, which perhaps is related to changes in mining safety regulations.

```
disasters = pm.Poisson('disasters', rate, observed=disaster_data)
```

We have two rates, `early_rate` if $t<=s$, and `late_rate` if $t>s$, where $s$ is the year the switch was made (a.k.a. the `switchpoint`).

**Step2:** Choose a prior distributions of the two rates, what we believe the rates were before we observed the data, and the switchpoint. We choose Exponential.
```
early_rate = pm.Exponential('early_rate', 1)
```

The parameters of this model are:

**Note:** Watch for missing values. Missing values are handled transparently by passing a MaskedArray or a pandas.DataFrame. Behind the scenes, another random variable, disasters.missing_values is created to model the missing values. If you pass a np.array with missing values you will get an error.

```python
with pm.Model() as disaster_model:

    # discrete
    switchpoint = pm.DiscreteUniform('switchpoint', lower=years.min(), upper=years.max(), testval=1900)

    # Priors for pre- and post-switch rates number of disasters
    early_rate = pm.Exponential('early_rate', 1)
    late_rate = pm.Exponential('late_rate', 1)

    # our theta - allocate appropriate Poisson rates to years before and after current
    # switch is an `if` statement in puMC3
    rate = pm.math.switch(switchpoint >= years, early_rate, late_rate)

    # our observed data as a likelihood function of the `rate` parameters
    # shows how we think our data is distributed
    disasters = pm.Poisson('disasters', rate, observed=disaster_data)
```

#### Model Fitting

```python
# there are defaults but we can also more explicitly set the sampling algorithms
with disaster_model:

    # for continuous variables
    step1 = pm.NUTS([early_rate, late_rate])

    # for discrete variables
    step2 = pm.Metropolis([switchpoint, disasters.missing_values[0]] )

    trace = pm.sample(10000, step=[step1, step2])
    # try different number of samples
    #trace = pm.sample(5000, step=[step1, step2])
```

#### Posterior Analysis

On the left side plots we notice that our early rate is between 2.5 and 3.5 disasters a year. In the late period it seems to be between 0.6 and 1.2 so definitely lower.

The right side plots show the samples we drew to come to our conclusion.

```python
pm.traceplot(trace, ['early_rate', 'late_rate', 'switchpoint'], figsize=(20,10));
```

```python
results = pm.summary(trace,
                     var_names=['early_rate', 'late_rate', 'switchpoint'])
results
```


```python
from IPython.core.display import HTML
with open("style/notebook.css", "r") as f:
    s = f"<style>{f.read()}</style>"
HTML(s)
```

```python
from bokeh.plotting import figure, ColumnDataSource
from bokeh.io import show, output_notebook, push_notebook
from bokeh.layouts import row
import subprocess
import swarming
import numpy as np
import pandas as pd

from dask.distributed import Client, LocalCluster, progress
import dask.bag
import dask.dataframe

output_notebook()
```

Predicting the behavior of swarms

Modelling interacting particles

*by Andreas Roth*

**2021-08-23**

## The model

* How do swarms of birds or fish align themselves?
* Who decides about collective movement?

J. Carrillo, A. Klar, S. Martin, and S. Tiwari, [Self-propelled interacting particle systems with
roosting force](https://www.worldscientific.com/doi/abs/10.1142/S0218202510004684). *Mathematical Models and Methods in Applied Sciences, 20(supp01), 2010*

Carrillo, J.A.; Klar, A.; Roth, A., [Single to double mill small noise transition via semi-lagrangian finite volume methods](https://www.intlpress.com/site/pub/pages/journals/items/cms/content/vols/0014/0004/a012/), *Communications in mathematical sciences 14 (2016), No.4, pp.1111-1136, ISSN: 1539-6746*

## Basic ideas

We look at the behavior of $n$ point masses
* position $x_i$ and velocity $v_i$, where $i = 1,...,n$

## Basic ideas

We look at the behavior of $n$ particles
* position $x_i$ and velocity $v_i$, where $i = 1,...,n$
* interaction force between pairs of particles depends on distance
    * if to close: **repulsion**

## Basic ideas

We look at the behavior of $n$ particles
* position $x_i$ and velocity $v_i$, where $i = 1,...,n$
* interaction force between pairs of particles depends on distance
    * if to close: **repulsion**
    * if to far: **attraction**
* forces balance each other
* pairwise interactions on one particle **superpose** each other
* evolution over time follows **Newton's second law**

## Model equations

System of ordinary differential equations:
$$
\begin{align}
dx_i =& v_i \cdot dt\\
dv_i =& (v_i(\alpha-\beta|v_i|) \\
&- \frac{1}{n}\sum_{i\neq j}F(|r_{ij}|) \frac{r_{ij}}{|r_{ij}|}) \cdot dt
\end{align}
$$

* $2n$ equations
* $r_{ij} = x_i-x_j$ connection vector between particles $i$ and $j$
* Computation effort goes with $n^2$ due to summing pairwise forces!

### Force term

One possibility to achieve what we want is

$$
F(r) = \frac{C_a}{l_a}\cdot \exp(-\frac{r}{l_a})-\frac{C_r}{l_r}\cdot \exp(-\frac{r}{l_r})
$$

```python
r = np.linspace(7, 200, 200)
f = figure(title="Force over distance", plot_width=1500, plot_height=200)
f.line(x=r, y=swarming.force_of_distance(r), line_width=3)
show(f)
```

## How to compute

* all code is published under MIT License at https://github.com/scherbertlemon/swarming
* ``swarming`` python package
* [Bokeh](https://bokeh.org) is used for all visualizations
* [Dask](https://dask.org) is used for parallel computing

## Generating some particles

```python
swarm = swarming.InitialCondition(condition="circular", n_particles=300)
show(row(swarm.plot(plot_width=600, plot_height=600), swarm.plot_density(plot_width=800, plot_height=600, size=3)))
```

## Letting the particles move

```python
nh = show(swarm.plot(plot_width=600, plot_height=600), notebook_handle=True)
```

```python
swarm.evolve(0.1, 10).update_cds()
push_notebook(handle=nh)
```

### More interactive

```python
proc = subprocess.Popen(["bokeh", "serve", "--port", "5006", "../bokeh_app/swarm.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
```

More interactively with [Bokeh app](http://localhost:5006)!
* Javascript frontend connected to a python backend
* Python runs the model
* Sliders change parameters while model is running

```python
import psutil
parentproc = psutil.Process(proc.pid)
for subproc in parentproc.children(recursive=True):
    subproc.kill()
parentproc.kill()
```

## Investigating steady states

Apparently, there is only a finite set of "equilibria" for this system, depending on the initial condition.

### All velocities aligned

```python
swarm.n = 400
show(row(swarm.set_initial("square").plot_density(size=5),
        swarm.set_initial("square").record_for_time(120, 0.5, 2, lr=1).plot_density(size=3)))
```

##  Investigating steady states

Apparently, there is only a finite set of "equilibria" for this system, depending on the initial condition.

### Random velocities

```python
show(row(swarm.set_initial("randomspeed").plot_density(size=5),
        swarm.set_initial("randomspeed").record_for_time(120, 0.5, 2, lr=1).plot_density(size=2)))
```

## Parameter study

* To check if the "donut" equilibrium always occurs for a given initial condition (random velocity), we run a parameter study
* We need to sample model parameters randomly and cover the complete parameter space: **Latin Hypercube Sampling**
* We use Dask to run the models for each parameter sample in parallel

## Parameter sampling

Model has parameters attraction strength $C_a$, attraction range $l_a$, repulsion strength $C_r$, repulsion range $l_r$.

```python
params = swarming.Parameters([("la", 100, 40, 160), ("lr", 2, 0.5, 3.5), ("Cr", 30, 5, 35)])
params
```

```python
pd.DataFrame(params.sampling_dicts(n_samples=10))
```

## Dask cluster and client

```python
cluster = LocalCluster(n_workers=4)
client = Client(cluster)
cluster
```

* There are backends to run Dask cluster on all kinds of systems (IBM LSF, Hadoop, ...)
* We run a ``LocalCluster`` on one CPU
* Dask client connects to cluster over the network (really distributed computing!)
* [Cluster dashboard](http://localhost:8787) shows cluster load etc.

### Run and store model for every of those parameter sets

We build a Dask *task graph*. No calculation happens here.

```python
def calc(dct):
    return (dct, swarming.InitialCondition(condition="randomspeed", n_particles=400).record_for_time(100, 0.5, 2, **dct).history.to_dict(orient="records"))
def explode(params, result):
    return [{**params, **r} for r in result]

dag = dask.bag.from_sequence(params.sampling_dicts(n_samples=24)) \
    .map(calc) \
    .starmap(explode) \
    .flatten() \
    .to_dataframe() \
    .to_parquet("results.parq", engine="pyarrow", overwrite=True, compute=False)
```

### Visualizing and computing the task graph

```python
dag.visualize()
```

```python
future = client.compute(dag)  # send the graph to the workers for computing
# progress(future)  # does not display correctly in slideshow
```

### Evaluate stored results

We have written **all particle positions / velocities** at defined **time steps** with the **used parameter sample**:

```python
data = dask.dataframe.read_parquet("results.parq")
data.head(150, npartitions=2)
```

### Evaluate stored results

* We want to plot the end densities, so select records with the maximum computed time
* Note, that at no time the dataset is completely in memory!
* When calling ``compute`` on a DAG, the result is actually transferred into local memory as ``pandas.DataFrame``

```python
data.groupby(params.names).agg({"time": "max"}).compute().head(3)
```

```python
final_states = data.loc[abs(data["time"]-100.0)<1e-12, :].compute()
final_states.head(3)
```

## Density plots of final states

Apparently, this parameterset delivers donuts in varying shapes.

```python
show(swarming.get_density_plots(final_states, param_names=params.names, ncols=4, size=3))
```

### Tidy up

We close the Dask cluster and client:

```python
client.close()
cluster.close()
```

Thanks for your attention

Time for your questions!