# Regression in Python

***
This is a very quick run-through of some basic statistical concepts, adapted from [Lab 4 in Harvard's CS109](https://github.com/cs109/2015lab4) course. Please feel free to try the original lab if you're feeling ambitious :-) The CS109 git repository also has the solutions if you're stuck.

* Linear Regression Models
* Prediction using linear regression

Linear regression is used to model and predict continuous outcomes with normal random errors. There are nearly an infinite number of different types of regression models and each regression model is typically defined by the distribution of the prediction errors (called "residuals") of the type of data. Logistic regression is used to model binary outcomes whereas Poisson regression is used to predict counts. In this exercise, we'll see some examples of linear regression as well as Train-test splits.

The packages we'll cover are: `statsmodels`, `seaborn`, and `scikit-learn`. While we don't explicitly teach `statsmodels` and `seaborn` in the Springboard workshop, those are great libraries to know.
***

***

```python
# special IPython command to prepare the notebook for matplotlib and other libraries
%matplotlib inline

import numpy as np
import pandas as pd
import scipy.stats as stats
import matplotlib.pyplot as plt
import sklearn

import seaborn as sns

# special matplotlib argument for improved plots
from matplotlib import rcParams
sns.set_style("whitegrid")
sns.set_context("poster")

```

***
# Part 1: Introduction to Linear Regression
### Purpose of linear regression
***

 Given a dataset containing predictor variables $X$ and outcome/response variable $Y$, linear regression can be used to:

 Build a predictive model to predict future values of $\hat{Y}$, using new data $X^*$ where $Y$ is unknown.

 Model the strength of the relationship between each independent variable $X_i$ and $Y$

 Many times, only a subset of independent variables $X_i$ will have a linear relationship with $Y$

 Need to figure out which $X_i$ contributes most information to predict $Y$

It is in many cases, the first pass prediction algorithm for continuous outcomes.

### A Brief Mathematical Recap
***

[Linear Regression](http://en.wikipedia.org/wiki/Linear_regression) is a method to model the relationship between a set of independent variables $X$ (also knowns as explanatory variables, features, predictors) and a dependent variable $Y$.  This method assumes the relationship between each predictor $X$ is **linearly** related to the dependent variable $Y$. The most basic linear regression model contains one independent variable $X$, we'll call this the simple model.

$$ Y = \beta_0 + \beta_1 X + \epsilon$$

where $\epsilon$ is considered as an unobservable random variable that adds noise to the linear relationship. In linear regression, $\epsilon$ is assumed to be normally distributed with a mean of 0. In other words, what this means is that on average, if we know $Y$, a roughly equal number of predictions $\hat{Y}$ will be above $Y$ and others will be below $Y$. That is, on average, the error is zero. The residuals, $\epsilon$ are also assumed to be "i.i.d.": independently and identically distributed. Independence means that the residuals are not correlated -- the residual from one prediction has no effect on the residual from another prediction. Correlated errors are common in time series analysis and spatial analyses.

* $\beta_0$ is the intercept of the linear model and represents the average of $Y$ when all independent variables $X$ are set to 0.

* $\beta_1$ is the slope of the line associated with the regression model and represents the average effect of a one-unit increase in $X$ on $Y$.

* Back to the simple model. The model in linear regression is the *conditional mean* of $Y$ given the values in $X$ is expressed a linear function.

$$ y = f(x) = E(Y | X = x)$$

http://www.learner.org/courses/againstallodds/about/glossary.html

* The goal is to estimate the coefficients (e.g. $\beta_0$ and $\beta_1$). We represent the estimates of the coefficients with a "hat" on top of the letter.

$$ \hat{\beta}_0, \hat{\beta}_1 $$

* Once we estimate the coefficients $\hat{\beta}_0$ and $\hat{\beta}_1$, we can use these to predict new values of $Y$ given new data $X$.

$$\hat{y} = \hat{\beta}_0 + \hat{\beta}_1 x_1$$

* Multiple linear regression is when you have more than one independent variable and the estimation involves matrices
    * $X_1$, $X_2$, $X_3$, $\ldots$

* How do you estimate the coefficients?
    * There are many ways to fit a linear regression model
    * The method called **least squares** is the most common methods
    * We will discuss least squares

$$ Y = \beta_0 + \beta_1 X_1 + \ldots + \beta_p X_p + \epsilon$$

### Estimating $\hat\beta$: Least squares
***
[Least squares](http://en.wikipedia.org/wiki/Least_squares) is a method that can estimate the coefficients of a linear model by minimizing the squared residuals:

$$ \mathscr{L} = \sum_{i=1}^N \epsilon_i = \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2  = \sum_{i=1}^N \left(y_i - \left(\beta_0 + \beta_1 x_i\right)\right)^2 $$

where $N$ is the number of observations and $\epsilon$ represents a residual or error, ACTUAL - PREDICTED.

#### Estimating the intercept $\hat{\beta_0}$ for the simple linear model

We want to minimize the squared residuals and solve for $\hat{\beta_0}$ so we take the partial derivative of $\mathscr{L}$ with respect to $\hat{\beta_0}$

$
\begin{align}
\frac{\partial \mathscr{L}}{\partial \hat{\beta_0}} &= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \epsilon^2 \\
&= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2 \\
&= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right)^2 \\
&= -2 \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right) \hspace{25mm} \mbox{(by chain rule)} \\
&= -2 \sum_{i=1}^N y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i \\
&= -2 \left[ \left( \sum_{i=1}^N y_i \right) - n \hat{\beta_0} - \hat{\beta}_1 \left( \sum_{i=1}^N x_i
\right) \right] \\
& 2 \left[ n \hat{\beta}_0 + \hat{\beta}_1 \sum_{i=1}^N x_i - \sum_{i=1}^N y_i \right] = 0 \hspace{20mm} \mbox{(Set equal to 0 and solve for $\hat{\beta}_0$)} \\
& n \hat{\beta}_0 + \hat{\beta}_1 \sum_{i=1}^N x_i - \sum{i=1}^N y_i = 0 \\
& n \hat{\beta}_0 = \sum_{i=1}^N y_i - \hat{\beta}_1 \sum_{i=1}^N x_i \\
& \hat{\beta}_0 = \frac{\sum_{i=1}^N y_i - \hat{\beta}_1 \sum_{i=1}^N x_i}{n} \\
& \hat{\beta}_0 = \frac{\sum_{i=1}^N y_i}{n} - \hat{\beta}_1 \frac{\sum_{i=1}^N x_i}{n} \\
& \boxed{\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}}
\end{align}
$

Using this new information, we can compute the estimate for $\hat{\beta}_1$ by taking the partial derivative of $\mathscr{L}$ with respect to $\hat{\beta}_1$.

$
\begin{align}
\frac{\partial \mathscr{L}}{\partial \hat{\beta_1}} &= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \epsilon^2 \\
&= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2 \\
&= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right)^2 \\
&= 2 \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right) \left( -x_i \right) \hspace{25mm}\mbox{(by chain rule)} \\
&= -2 \sum_{i=1}^N x_i \left( y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i \right) \\
&= -2 \sum_{i=1}^N x_i y_i - \hat{\beta}_0 x_i - \hat{\beta}_1 x_i^2 \\
&= -2 \sum_{i=1}^N x_i y_i - \left( \bar{y} - \hat{\beta}_1 \bar{x} \right) x_i - \hat{\beta}_1 x_i^2 \\
&= -2 \sum_{i=1}^N x_i y_i - \bar{y}x_i + \hat{\beta}_1\bar{x}x_i - \hat{\beta}_1 x_i^2 \\
&= -2 \left[ \sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i + \hat{\beta}_1\bar{x} - \hat{\beta}_1 x_i^2 \right] \\
&= -2 \left[ \hat{\beta}_1 \left\{ \bar{x} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i^2 \right\} + \left\{ \sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i \right\}\right] \\
& 2 \left[ \hat{\beta}_1 \left\{ \sum_{i=1}^N x_i^2 - \bar{x} \sum_{i=1}^N x_i \right\} + \left\{ \bar{y} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i y_i \right\} \right] = 0 \\
& \hat{\beta}_1 = \frac{-\left( \bar{y} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i y_i \right)}{\sum_{i=1}^N x_i^2 - \bar{x}\sum_{i=1}^N x_i} \\
&= \frac{\sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i}{\sum_{i=1}^N x_i^2 - \bar{x} \sum_{i=1}^N x_i} \\
& \boxed{\hat{\beta}_1 = \frac{\sum_{i=1}^N x_i y_i - \bar{x}\bar{y}n}{\sum_{i=1}^N x_i^2 - n \bar{x}^2}}
\end{align}
$

The solution can be written in compact matrix notation as

$$\hat\beta =  (X^T X)^{-1}X^T Y$$

We wanted to show you this in case you remember linear algebra, in order for this solution to exist we need $X^T X$ to be invertible. Of course this requires a few extra assumptions, $X$ must be full rank so that $X^T X$ is invertible, etc. Basically, $X^T X$ is full rank if all rows and columns are linearly independent. This has a loose relationship to variables and observations being independent respective. **This is important for us because this means that having redundant features in our regression models will lead to poorly fitting (and unstable) models.** We'll see an implementation of this in the extra linear regression example.

***
# Part 2: Exploratory Data Analysis for Linear Relationships

The [Boston Housing data set](https://archive.ics.uci.edu/ml/datasets/Housing) contains information about the housing values in suburbs of Boston.  This dataset was originally taken from the StatLib library which is maintained at Carnegie Mellon University and is now available on the UCI Machine Learning Repository.

## Load the Boston Housing data set from `sklearn`
***

This data set is available in the [sklearn](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html#sklearn.datasets.load_boston) python module which is how we will access it today.

```python
from sklearn.datasets import load_boston
import pandas as pd

boston = load_boston()
```

```python
boston.keys()
```

    ['data', 'feature_names', 'DESCR', 'target']

```python
boston.data.shape
```

    (506, 13)

```python
# Print column names
print(boston.feature_names)
```

    ['CRIM' 'ZN' 'INDUS' 'CHAS' 'NOX' 'RM' 'AGE' 'DIS' 'RAD' 'TAX' 'PTRATIO'
     'B' 'LSTAT']

```python
# Print description of Boston housing data set
print(boston.DESCR)
```

    Boston House Prices dataset
    ===========================

    Notes
    ------
    Data Set Characteristics:

        :Number of Instances: 506

        :Number of Attributes: 13 numeric/categorical predictive

        :Median Value (attribute 14) is usually the target

        :Attribute Information (in order):
            - CRIM     per capita crime rate by town
            - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.
            - INDUS    proportion of non-retail business acres per town
            - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
            - NOX      nitric oxides concentration (parts per 10 million)
            - RM       average number of rooms per dwelling
            - AGE      proportion of owner-occupied units built prior to 1940
            - DIS      weighted distances to five Boston employment centres
            - RAD      index of accessibility to radial highways
            - TAX      full-value property-tax rate per $10,000
            - PTRATIO  pupil-teacher ratio by town
            - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town
            - LSTAT    % lower status of the population
            - MEDV     Median value of owner-occupied homes in $1000's

        :Missing Attribute Values: None

        :Creator: Harrison, D. and Rubinfeld, D.L.

    This is a copy of UCI ML housing dataset.
    http://archive.ics.uci.edu/ml/datasets/Housing

    This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.

    The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic
    prices and the demand for clean air', J. Environ. Economics & Management,
    vol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics
    ...', Wiley, 1980.   N.B. Various transformations are used in the table on
    pages 244-261 of the latter.

    The Boston house-price data has been used in many machine learning papers that address regression
    problems.

    **References**

       - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.
       - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.
       - many more! (see http://archive.ics.uci.edu/ml/datasets/Housing)

Now let's explore the data set itself.

```python
bos = pd.DataFrame(boston.data)
bos.head()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33

There are no column names in the DataFrame. Let's add those.

```python
bos.columns = boston.feature_names
bos.head()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33

Now we have a pandas DataFrame called `bos` containing all the data we want to use to predict Boston Housing prices.  Let's create a variable called `PRICE` which will contain the prices. This information is contained in the `target` data.

```python
print(boston.target.shape)
```

    (506,)

```python
bos['PRICE'] = boston.target
bos.head()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      PRICE

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98
      24.0

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14
      21.6

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03
      34.7

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94
      33.4

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33
      36.2

## EDA and Summary Statistics
***

Let's explore this data set.  First we use `describe()` to get basic summary statistics for each of the columns.

```python
bos.describe()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      PRICE

      count
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000

      mean
      3.593761
      11.363636
      11.136779
      0.069170
      0.554695
      6.284634
      68.574901
      3.795043
      9.549407
      408.237154
      18.455534
      356.674032
      12.653063
      22.532806

      std
      8.596783
      23.322453
      6.860353
      0.253994
      0.115878
      0.702617
      28.148861
      2.105710
      8.707259
      168.537116
      2.164946
      91.294864
      7.141062
      9.197104

      min
      0.006320
      0.000000
      0.460000
      0.000000
      0.385000
      3.561000
      2.900000
      1.129600
      1.000000
      187.000000
      12.600000
      0.320000
      1.730000
      5.000000

      25%
      0.082045
      0.000000
      5.190000
      0.000000
      0.449000
      5.885500
      45.025000
      2.100175
      4.000000
      279.000000
      17.400000
      375.377500
      6.950000
      17.025000

      50%
      0.256510
      0.000000
      9.690000
      0.000000
      0.538000
      6.208500
      77.500000
      3.207450
      5.000000
      330.000000
      19.050000
      391.440000
      11.360000
      21.200000

      75%
      3.647423
      12.500000
      18.100000
      0.000000
      0.624000
      6.623500
      94.075000
      5.188425
      24.000000
      666.000000
      20.200000
      396.225000
      16.955000
      25.000000

      max
      88.976200
      100.000000
      27.740000
      1.000000
      0.871000
      8.780000
      100.000000
      12.126500
      24.000000
      711.000000
      22.000000
      396.900000
      37.970000
      50.000000

### Scatterplots
***

Let's look at some scatter plots for three variables: 'CRIM' (per capita crime rate), 'RM' (number of rooms) and 'PTRATIO' (pupil-to-teacher ratio in schools).

```python
plt.scatter(bos.CRIM, bos.PRICE)
plt.xlabel("Per capita crime rate by town (CRIM)")
plt.ylabel("Housing Price")
plt.title("Relationship between CRIM and Price")
```

Part 2 Checkup Exercise Set I

Exercise: What kind of relationship do you see? e.g. positive, negative?  linear? non-linear? Is there anything else strange or interesting about the data? What about outliers?

Exercise: Create scatter plots between *RM* and *PRICE*, and *PTRATIO* and *PRICE*. Label your axes appropriately using human readable labels. Tell a story about what you see.

Exercise: What are some other numeric variables of interest? Why do you think they are interesting? Plot scatterplots with these variables and *PRICE* (house price) and tell a story about what you see.

```python
# your turn: describe relationship
# There is a negative relationship between CRIM and Housing price.
# It is not linear, instead it appears related to log of housing price.
# What is also unusual about the data is that the housing price appears capped at 50.

```

```python
# your turn: scatter plot between *RM* and *PRICE*
plt.scatter(bos.RM, bos.PRICE)
plt.xlabel("Average number of rooms")
plt.ylabel("Housing Price")
plt.title("Relationship between Num Rooms and Price")

```

```python
# your turn: scatter plot between *PTRATIO* and *PRICE*
plt.scatter(bos.PTRATIO, bos.PRICE)
plt.xlabel("Pupil-teacher ratio")
plt.ylabel("Housing Price")
plt.title("Relationship between Pupil-teacher Ratio and Price")
```

```python
# your turn: create some other scatter plots
plt.figure(0)
plt.scatter(bos.DIS, bos.PRICE)
plt.xlabel("Distance to work centers(DIS)")
plt.ylabel("Housing Price")
plt.title("Relationship between Distance to work and Price")

plt.figure(1)
plt.scatter(bos.NOX, bos.PRICE)
plt.xlabel("Nitrous Oxide Level (NOX)")
plt.ylabel("Housing Price")
plt.title("Relationship between NOX Polution and Price")
```

### Scatterplots using Seaborn
***

[Seaborn](https://stanford.edu/~mwaskom/software/seaborn/) is a cool Python plotting library built on top of matplotlib. It provides convenient syntax and shortcuts for many common types of plots, along with better-looking defaults.

We can also use [seaborn regplot](https://stanford.edu/~mwaskom/software/seaborn/tutorial/regression.html#functions-to-draw-linear-regression-models) for the scatterplot above. This provides automatic linear regression fits (useful for data exploration later on). Here's one example below.

```python
sns.regplot(y="PRICE", x="RM", data=bos, fit_reg = True)
```

### Histograms
***

```python
plt.hist(np.log(bos.CRIM))
plt.title("CRIM")
plt.xlabel("Crime rate per capita")
plt.ylabel("Frequency")
plt.show()
```

Part 2 Checkup Exercise Set II

Exercise: In the above histogram, we took the logarithm of the crime rate per capita. Repeat this histogram without taking the log. What was the purpose of taking the log? What do we gain by making this transformation? What do you now notice about this variable that is not obvious without making the transformation?

Exercise: Plot the histogram for *RM* and *PTRATIO* against each other, along with the two variables you picked in the previous section. We are looking for correlations in predictors here.

```python
#your turn
plt.hist(bos.CRIM)
plt.title("CRIM")
plt.xlabel("Crime rate per capita")
plt.ylabel("Frequency")
plt.show()

# we took the log so that the large value of CRIM in a few regions didn't cause the x-axis to
# cover as wide a range, so that we can better see the pattern in the smaller values of CRIM.

# in the log of CRIM we notice that there are two peaks, one at log = about -2.5 and one at
# log = about 2.

```

```python
with sns.axes_style("white"):
    sns.jointplot(y="PTRATIO", x="RM", data=bos, kind="hex", color="k");
```

## Part 3: Linear Regression with Boston Housing Data Example
***

Here,

$Y$ = boston housing prices (called "target" data in python, and referred to as the dependent variable or response variable)

and

$X$ = all the other features (or independent variables, predictors or explanatory variables)

which we will use to fit a linear regression model and predict Boston housing prices. We will use the least-squares method to estimate the coefficients.

We'll use two ways of fitting a linear regression. We recommend the first but the second is also powerful in its features.

### Fitting Linear Regression using `statsmodels`
***
[Statsmodels](http://statsmodels.sourceforge.net/) is a great Python library for a lot of basic and inferential statistics. It also provides basic regression functions using an R-like syntax, so it's commonly used by statisticians. While we don't cover statsmodels officially in the Data Science Intensive workshop, it's a good library to have in your toolbox. Here's a quick example of what you could do with it. The version of least-squares we will use in statsmodels is called *ordinary least-squares (OLS)*. There are many other versions of least-squares such as [partial least squares (PLS)](https://en.wikipedia.org/wiki/Partial_least_squares_regression) and [weighted least squares (WLS)](https://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares).

```python
# Import regression modules
import statsmodels.api as sm
from statsmodels.formula.api import ols
```

```python
# statsmodels works nicely with pandas dataframes
# The thing inside the "quotes" is called a formula, a bit on that below
m = ols('PRICE ~ RM',bos).fit()
print(m.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.484
    Model:                            OLS   Adj. R-squared:                  0.483
    Method:                 Least Squares   F-statistic:                     471.8
    Date:                Sun, 08 Oct 2017   Prob (F-statistic):           2.49e-74
    Time:                        18:14:38   Log-Likelihood:                -1673.1
    No. Observations:                 506   AIC:                             3350.
    Df Residuals:                     504   BIC:                             3359.
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept    -34.6706      2.650    -13.084      0.000     -39.877     -29.465
    RM             9.1021      0.419     21.722      0.000       8.279       9.925
    ==============================================================================
    Omnibus:                      102.585   Durbin-Watson:                   0.684
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              612.449
    Skew:                           0.726   Prob(JB):                    1.02e-133
    Kurtosis:                       8.190   Cond. No.                         58.4
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

#### Interpreting coefficients

There is a ton of information in this output. But we'll concentrate on the coefficient table (middle table). We can interpret the `RM` coefficient (9.1021) by first noticing that the p-value (under `P>|t|`) is so small, basically zero. This means that the number of rooms, `RM`, is a statisticall significant predictor of `PRICE`. The regression coefficient for `RM` of 9.1021 means that *on average, each additional room is associated with an increase of $\$9,100$ in house price net of the other variables*. The confidence interval gives us a range of plausible values for this average change, about ($\$8,279, \$9,925$), definitely not chump change.

In general, the $\hat{\beta_i}, i > 0$ can be interpreted as the following: "A one unit increase in $x_i$ is associated with, on average, a $\hat{\beta_i}$ increase/decrease in $y$ net of all other variables."

On the other hand, the interpretation for the intercept, $\hat{\beta}_0$ is the average of $y$ given that all of the independent variables $x_i$ are 0.

####  `statsmodels` formulas
***
This formula notation will seem familiar to `R` users, but will take some getting used to for people coming from other languages or are new to statistics.

The formula gives instruction for a general structure for a regression call. For `statsmodels` (`ols` or `logit`) calls you need to have a Pandas dataframe with column names that you will add to your formula. In the below example you need a pandas data frame that includes the columns named (`Outcome`, `X1`,`X2`, ...), but you don't need to build a new dataframe for every regression. Use the same dataframe with all these things in it. The structure is very simple:

`Outcome ~ X1`

But of course we want to to be able to handle more complex models, for example multiple regression is doone like this:

`Outcome ~ X1 + X2 + X3`

In general, a formula for an OLS multiple linear regression is

`Y ~ X1 + X2 + ... + Xp`

This is the very basic structure but it should be enough to get you through the homework. Things can get much more complex. You can force statsmodels to treat variables as categorical with the `C()` function, call numpy functions to transform data such as `np.log` for extremely-skewed data, or fit a model without an intercept by including `- 1` in the formula. For a quick run-down of further uses see the `statsmodels` [help page](http://statsmodels.sourceforge.net/devel/example_formulas.html).

Let's see how our model actually fit our data. We can see below that there is a ceiling effect, we should probably look into that. Also, for large values of $Y$ we get underpredictions, most predictions are below the 45-degree gridlines.

Part 3 Checkup Exercise Set I

Exercise: Create a scatterplot between the predicted prices, available in `m.fittedvalues` (where `m` is the fitted model) and the original prices. How does the plot look? Do you notice anything interesting or weird in the plot? Comment on what you see.

```python
# your turn
plt.scatter(bos.PRICE, m.fittedvalues)
plt.xlabel("Original Values")
plt.ylabel("Fitted Values")
plt.title("Relationship between Original values and fitted values")

# All of the original values which were at 50 differ greatly from their predicted values.
# Also, there are outliners that probably reflect incorrect values for price or # of rooms.
```

### Fitting Linear Regression using `sklearn`

```python
from sklearn.linear_model import LinearRegression
X = bos.drop('PRICE', axis = 1)

# This creates a LinearRegression object
lm = LinearRegression()
lm
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)

#### What can you do with a LinearRegression object?
***
Check out the scikit-learn [docs here](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html). We have listed the main functions here. Most machine learning models in scikit-learn follow this same API of fitting a model with `fit`, making predictions with `predict` and the appropriate scoring function `score` for each model.

Main functions | Description
--- | ---
`lm.fit()` | Fit a linear model
`lm.predit()` | Predict Y using the linear model with estimated coefficients
`lm.score()` | Returns the coefficient of determination (R^2). *A measure of how well observed outcomes are replicated by the model, as the proportion of total variation of outcomes explained by the model*

#### What output can you get?

```python
# Look inside lm object
# lm.<tab>
```

Output | Description
--- | ---
`lm.coef_` | Estimated coefficients
`lm.intercept_` | Estimated intercept

### Fit a linear model
***

The `lm.fit()` function estimates the coefficients the linear regression using least squares.

```python
# Use all 13 predictors to fit linear regression model
lm.fit(X, bos.PRICE)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)

Part 3 Checkup Exercise Set II

Exercise: How would you change the model to not fit an intercept term? Would you recommend not having an intercept? Why or why not? For more information on why to include or exclude an intercept, look [here](https://online.stat.psu.edu/~ajw13/stat501/SpecialTopics/Reg_thru_origin.pdf).

Exercise: One of the assumptions of the linear model is that the residuals must be i.i.d. (independently and identically distributed). To satisfy this, is it enough that the residuals are normally distributed? Explain your answer.

Exercise: True or false. To use linear regression, $Y$ must be normally distributed. Explain your answer.

```python
# your turn
# To avoid fitting an intercept, call lm = LinearRegression(fit_intercept = False)
# based on the reading, I would recommend against fitting an intercept, since a house
# with 0 rooms would clearly not have a value.

# It is not enough that the residuals be normally distributed, as that says nothing about them
# being independent of each other.

# False:  using linear regression does not require that the Y values are normally distributed,
# but requires that the residuals from the fitting are normally distributed.  There are no
# assumptions made about the distributions of independent or dependent variables. The residual
# variance, that is, the part of the observed variance in the outcome that we cannot explain
# by knowing the predictors, is the variation around the expected value that is predicted by
# the model. If you use least-squares estimation of your model, you implicitly assume that
# this residual variation is normally distributed.  See
# https://www.researchgate.net/post/Do_we_need_normal_distribution_of_dependent_variable_when_working_with_ordinary_least_squares_or_other_linear_regression_method
```

### Estimated intercept and coefficients

Let's look at the estimated coefficients from the linear model using `1m.intercept_` and `lm.coef_`.

After we have fit our linear regression model using the least squares method, we want to see what are the estimates of our coefficients $\beta_0$, $\beta_1$, ..., $\beta_{13}$:

$$ \hat{\beta}_0, \hat{\beta}_1, \ldots, \hat{\beta}_{13} $$

```python
print('Estimated intercept coefficient: {}'.format(lm.intercept_))
```

    Estimated intercept coefficient: 36.4911032804

```python
print('Number of coefficients: {}'.format(len(lm.coef_)))
```

    Number of coefficients: 13

```python
# The coefficients
pd.DataFrame({'features': X.columns, 'estimatedCoefficients': lm.coef_})[['features', 'estimatedCoefficients']]
```

      features
      estimatedCoefficients

      0
      CRIM
      -0.107171

      1
      ZN
      0.046395

      2
      INDUS
      0.020860

      3
      CHAS
      2.688561

      4
      NOX
      -17.795759

      5
      RM
      3.804752

      6
      AGE
      0.000751

      7
      DIS
      -1.475759

      8
      RAD
      0.305655

      9
      TAX
      -0.012329

      10
      PTRATIO
      -0.953464

      11
      B
      0.009393

      12
      LSTAT
      -0.525467

### Predict Prices

We can calculate the predicted prices ($\hat{Y}_i$) using `lm.predict`.

$$ \hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_1 + \ldots \hat{\beta}_{13} X_{13} $$

```python
# first five predicted prices
lm.predict(X)[0:5]
```

    array([ 30.00821269,  25.0298606 ,  30.5702317 ,  28.60814055,  27.94288232])

Part 3 Checkup Exercise Set III

Exercise: Histogram: Plot a histogram of all the predicted prices. Write a story about what you see. Describe the shape, center and spread of the distribution. Are there any outliers? What might be the reason for them? Should we do anything special with them?

Exercise: Scatterplot: Let's plot the true prices compared to the predicted prices to see they disagree (we did this with `statsmodels` before).

Exercise: We have looked at fitting a linear model in both `statsmodels` and `scikit-learn`. What are the advantages and disadvantages of each based on your exploration? Based on the information provided by both packages, what advantage does `statsmodels` provide?

```python
# your turn
plt.hist(lm.predict(X), bins=20)
plt.title("Predicted Prices")
plt.xlabel("Price")
plt.ylabel("Frequency")
plt.show()
```

```python
# The distribution of the predicted prices is close to a normal with a center at 20.  There tend to be outliers
# at the upper and lower price values.  I believe that these are due to the prevalence of the capped 50 in the
# input, plus data errors in the input (the most likely would be incorrect measurement of NOX, since that has
# the largest coefficient.)

# to address this, I would delete all records where the price is 50, and any where the NOX value is greatly out
# of the normal range for it.

plt.scatter(bos.PRICE, lm.predict(X))
plt.xlabel("Original Values")
plt.ylabel("Fitted Values")
plt.title("Relationship between Original values and fitted values")
```

```python
# Advantages of Statsmodels are:
# * ability to carry out weighted regression
# * ability to describe the regression problem using R-like formulas
# * more complete set of summary stats

# Disadvantages of Statsmodels are:
# * from what I have read: The biggest disadvantage of statsmodels lies in the
#   fact that it is still a very new package. The quantity and quality of documentation
#   available is very poor.

# Advantages of Scikit are:
# * ease of use as its API matches other Scikit objects including ML code.
# * simpler API
# * easy to ask for regression with no intercept

# Disadvantages of Scikit are:
# * doesn't provide as many "interesting"-in-depth statistical models.

# Conclusion:  statsmodels is more useful for detailed statistical analysis, if you know
# the literature and can use the scattered documentation.
```

### Evaluating the Model: Sum-of-Squares

The partitioning of the sum-of-squares shows the variance in the predictions explained by the model and the variance that is attributed to error.

$$TSS = ESS + RSS$$

#### Residual Sum-of-Squares (aka $RSS$)

The residual sum-of-squares is one of the basic ways of quantifying how much error exists in the fitted model. We will revisit this in a bit.

$$ RSS = \sum_{i=1}^N r_i^2 = \sum_{i=1}^N \left(y_i - \left(\beta_0 + \beta_1 x_i\right)\right)^2 $$

```python
print(np.sum((bos.PRICE - lm.predict(X)) ** 2))
```

    11080.2762841

#### Explained Sum-of-Squares (aka $ESS$)

The explained sum-of-squares measures the variance explained by the regression model.

$$ESS = \sum_{i=1}^N \left( \hat{y}_i - \bar{y} \right)^2 = \sum_{i=1}^N \left( \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) - \bar{y} \right)^2$$

```python
print(np.sum(lm.predict(X) - np.mean(bos.PRICE)) ** 2)
```

    5.16987882846e-24

### Evaluating the Model: The Coefficient of Determination ($R^2$)

The coefficient of determination, $R^2$, tells us the percentage of the variance in the response variable $Y$ that can be explained by the linear regression model.

$$ R^2 = \frac{ESS}{TSS} $$

The $R^2$ value is one of the most common metrics that people use in describing the quality of a model, but it is important to note that *$R^2$ increases artificially as a side-effect of increasing the number of independent variables.* While $R^2$ is reported in almost all statistical packages, another metric called the *adjusted $R^2$* is also provided as it takes into account the number of variables in the model, and can sometimes even be used for non-linear regression models!

$$R_{adj}^2 = 1 - \left( 1 - R^2 \right) \frac{N - 1}{N - K - 1} = R^2 - \left( 1 - R^2 \right) \frac{K}{N - K - 1} = 1 - \frac{\frac{RSS}{DF_R}}{\frac{TSS}{DF_T}}$$

where $N$ is the number of observations, $K$ is the number of variables, $DF_R = N - K - 1$ is the degrees of freedom associated with the residual error and $DF_T = N - 1$ is the degrees of the freedom of the total error.

### Evaluating the Model: Mean Squared Error and the $F$-Statistic
***
The mean squared errors are just the *averages* of the sum-of-squares errors over their respective degrees of freedom.

$$MSE = \frac{ESS}{K}$$
$$MSR = \frac{RSS}{N-K-1}$$

**Remember: ** Notation may vary across resources particularly the use of *R* and *E* in *RSS/ESS* and *MSR/MSE*. In some resources, E = explained and R = residual. In other resources, E = error and R = regression (explained). **This is a very important distinction that requires looking at the formula to determine which naming scheme is being used.**

Given the MSR and MSE, we can now determine whether or not the entire model we just fit is even statistically significant. We use an $F$-test for this. The null hypothesis is that all of the $\beta$ coefficients are zero, that is, none of them have any effect on $Y$. The alternative is that *at least one* $\beta$ coefficient is nonzero, but it doesn't tell us which one in a multiple regression:

$$H_0: \beta_i = 0, \mbox{for all $i$} \\
H_A: \beta_i > 0, \mbox{for some $i$}$$

$$F = \frac{MSR}{MSE} = \left( \frac{R^2}{1 - R^2} \right) \left( \frac{N - K - 1}{K} \right)$$

Once we compute the $F$-statistic, we can use the $F$-distribution with $N-K$ and $K-1$ degrees of degrees of freedom to get a p-value.

**Warning!** The $F$-statistic mentioned in this section is NOT the same as the F1-measure or F1-value discused in Unit 7.

Part 3 Checkup Exercise Set IV

Let's look at the relationship between `PTRATIO` and housing price.

Exercise: Try fitting a linear regression model using only the 'PTRATIO' (pupil-teacher ratio by town) and interpret the intercept and the coefficients.

Exercise: Calculate (or extract) the $R^2$ value. What does it tell you?

Exercise: Compute the $F$-statistic. What does it tell you?

Exercise: Take a close look at the $F$-statistic and the $t$-statistic for the regression coefficient. What relationship do you notice? Note that this relationship only applies in *simple* linear regression models.

```python
# your turn
m = ols('PRICE ~ PTRATIO',bos).fit()
print(m.rsquared)
print(m.rsquared_adj)

# the R2 value of .25 shows us that the fit is not that great.
```

    0.257847318009
    0.256374792847

```python
# your turn
f = (m.rsquared / (1 - m.rsquared)) * ((m.nobs - 1 - 1) / 1)
print("The F-value is %f" % f)

# F-value is large, indicating that the model does not support the data well.

r = [0,1]
T_test = m.t_test(r)
t_value = T_test.tvalue[0][0]
print("The t-value is %f" % t_value)
print("The t-value squared is %f" % (t_value * t_value))

# the relationship between the F-statistic and t-statistic is f is equal to t squared.
```

    The F-value is 175.105543
    The t-value is -13.232745
    The t-value squared is 175.105543

Part 3 Checkup Exercise Set V

Fit a linear regression model using three independent variables

 'CRIM' (per capita crime rate by town)

 'RM' (average number of rooms per dwelling)

 'PTRATIO' (pupil-teacher ratio by town)

Exercise: Compute or extract the $F$-statistic. What does it tell you about the model?

Exercise: Compute or extract the $R^2$ statistic. What does it tell you about the model?

Exercise: Which variables in the model are significant in predicting house price? Write a story that interprets the coefficients.

```python
# your turn
m = ols('PRICE ~ CRIM + RM + PTRATIO',bos).fit()
f = (m.rsquared / (1 - m.rsquared)) * ((m.nobs - 1 - 1) / 1)
print("The F-value is %f" % f)
print("The R-squared is %f" % m.rsquared)
print("The AIC value is %f" % m.aic)

# the F-value and R-squared values indicate a better fit.

print("The parameters are")
print(m.params)

# the strongest variable is RM
```

    The F-value is 735.590621
    The R-squared is 0.593414
    The AIC value is 3233.100274
    The parameters are
    Intercept   -3.306617
    CRIM        -0.202110
    RM           7.381567
    PTRATIO     -1.074187
    dtype: float64

## Part 4: Comparing Models

During modeling, there will be times when we want to compare models to see which one is more predictive or fits the data better. There are many ways to compare models, but we will focus on two.

### The $F$-Statistic Revisited

The $F$-statistic can also be used to compare two *nested* models, that is, two models trained on the same dataset where one of the models contains a *subset* of the variables of the other model. The *full* model contains $K$ variables and the *reduced* model contains a subset of these $K$ variables. This allows us to add additional variables to a base model and then test if adding the variables helped the model fit.

$$F = \frac{\left( \frac{RSS_{reduced} - RSS_{full}}{DF_{reduced} - DF_{full}} \right)}{\left( \frac{RSS_{full}}{DF_{full}} \right)}$$

where $DF_x = N - K_x - 1$ where $K_x$ is the number of variables in model $x$.

### Akaike Information Criterion (AIC)

Another statistic for comparing two models is AIC, which is based on the likelihood function and takes into account the number of variables in the model.

$$AIC = 2 K - 2 \log_e{L}$$

where $L$ is the likelihood of the model. AIC is meaningless in the absolute sense, and is only meaningful when compared to AIC values from other models. Lower values of AIC indicate better fitting models.

`statsmodels` provides the AIC in its output.

Part 4 Checkup Exercises

Exercise: Find another variable (or two) to add to the model we built in Part 3. Compute the $F$-test comparing the two models as well as the AIC. Which model is better?

```python
# your turn
m = ols('PRICE ~ CRIM + RM + PTRATIO + CHAS + NOX',bos).fit()
f = (m.rsquared / (1 - m.rsquared)) * ((m.nobs - 1 - 1) / 1)
print("The F-value is %f" % f)
print("The R-squared is %f" % m.rsquared)
print("The AIC value is %f" % m.aic)

# the F-value and R-squared values indicate a better fit.

print("The parameters are")
print(m.params)

# the strongest variable is NOX
```

    The F-value is 845.818470
    The R-squared is 0.626616
    The AIC value is 3193.994673
    The parameters are
    Intercept     7.244026
    CRIM         -0.127250
    RM            6.772290
    PTRATIO      -1.024814
    CHAS          3.950277
    NOX         -14.737869
    dtype: float64

## Part 5: Evaluating the Model via Model Assumptions and Other Issues
***
Linear regression makes several assumptions. It is always best to check that these assumptions are valid after fitting a linear regression model.

**Linearity**. The dependent variable $Y$ is a linear combination of the regression coefficients and the independent variables $X$. This can be verified with a scatterplot of each $X$ vs. $Y$ and plotting correlations among $X$. Nonlinearity can sometimes be resolved by [transforming](https://onlinecourses.science.psu.edu/stat501/node/318) one or more independent variables, the dependent variable, or both. In other cases, a [generalized linear model](https://en.wikipedia.org/wiki/Generalized_linear_model) or a [nonlinear model](https://en.wikipedia.org/wiki/Nonlinear_regression) may be warranted.

**Constant standard deviation**. The SD of the dependent variable $Y$ should be constant for different values of X. We can check this by plotting each $X$ against $Y$ and verifying that there is no "funnel" shape showing data points fanning out as $X$ increases or decreases. Some techniques for dealing with non-constant variance include weighted least squares (WLS), [robust standard errors](https://en.wikipedia.org/wiki/Heteroscedasticity-consistent_standard_errors), or variance stabilizing transformations.

 **Normal distribution for errors**.  The $\epsilon$ term we discussed at the beginning are assumed to be normally distributed. This can be verified with a fitted values vs. residuals plot and verifying that there is no pattern, and with a quantile plot.
  $$ \epsilon_i \sim N(0, \sigma^2)$$
Sometimes the distributions of responses $Y$ may not be normally distributed at any given value of $X$.  e.g. skewed positively or negatively.

 **Independent errors**.  The observations are assumed to be obtained independently.

e.g. Observations across time may be correlated

There are some other issues that are important investigate with linear regression models.

**Correlated Predictors:** Care should be taken to make sure that the independent variables in a regression model are not too highly correlated. Correlated predictors typically do not majorly affect prediction, but do inflate standard errors of coefficients making interpretation unreliable. Common solutions are dropping the least important variables involved in the correlations, using regularlization, or, when many predictors are highly correlated, considering a dimension reduction technique such as principal component analysis (PCA).

**Influential Points:** Data points that have undue influence on the regression model. These points can be high leverage points or outliers. Such points are typically removed and the regression model rerun.

Part 5 Checkup Exercises

Take the reduced model from Part 3 to answer the following exercises. Take a look at [this blog post](http://mpastell.com/2013/04/19/python_regression/) for more information on using statsmodels to construct these plots.

Exercise: Construct a fitted values versus residuals plot. What does the plot tell you? Are there any violations of the model assumptions?

Exercise: Construct a quantile plot of the residuals. What does the plot tell you?

Exercise: What are some advantages and disadvantages of the fitted vs. residual and quantile plot compared to each other?

Exercise: Identify any outliers (if any) in your model and write a story describing what these outliers might represent.

Exercise: Construct a leverage plot and identify high leverage points in the model. Write a story explaining possible reasons for the high leverage points.

Exercise: Remove the outliers and high leverage points from your model and run the regression again. How do the results change?

```python
# Your turn.
m = ols('PRICE ~ PTRATIO',bos).fit()
print(m.rsquared)
print(m.rsquared_adj)

plt.scatter(m.fittedvalues, m.resid)
plt.xlabel("Fitted values")
plt.ylabel("Residuals")
plt.title("Relationship between Fitted values an Residuals")
```

```python
# the residuals appear normally distributed, so I don't think there are voluations of the model assumptions.
```

```python
# your turn
res = m.resid # residuals
fig = sm.qqplot(res)
plt.show()
```

```python
# your turn
# The Q-Q plot is helpful because it shows if the two distributions being
# compared are similar.  If they are, the points in the Q–Q plot will
# approximately lie on the line y = x

# However, the fitted vs. actual chart shows the specific values, so you can determine
# where you might flag values as outliers.

# The outliers are those which have a price of 50, meaning that they have been capped.
# Another set of outliers is for ZN which only has a meaningful value for some of the records.
```

```python
fig, ax = plt.subplots(figsize=(20,8))
fig = sm.graphics.influence_plot(m, ax=ax, criterion="cooks")
```

```python
# some documentation is at http://www.statsmodels.org/dev/examples/notebooks/generated/regression_plots.html
```


## Lab 9-1 : XOR을 위한 텐스플로우 딥넷트웍

```python
import tensorflow as tf
import numpy as np
```

    /anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      from ._conv import register_converters as _register_converters

### XOR data set

| A | B | X |
| - | - | - |
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |

#### Boolean Expression
$$
X = A \oplus B
$$

#### Logic Diagram Symbol

[[이미지 출처]](https://mathphysics.tistory.com/579)

간단한 데이터셋이므로 따로 입력받지않고 **numpy array** 사용

```python
x_data = np.array(
    [
        [0, 0],
        [0, 1],
        [1, 0],
        [1, 1],
    ],
    dtype=np.float32
)
y_data = np.array(
    [
        [0],
        [1],
        [1],
        [0],
    ],
    dtype=np.float32
)
```

### Logistic Regression으로 해결하기
**0**인지 **1**인지 예측하는 **Logistic Regression**사용

$$X$$, $$Y$$에 맞게 **Weight**과 **Bias**를 결정해야한다.

**Weigth**을 정할때에는 **크기**가 **중요**하다.

입력값이 $$X_1$$, $$X_2$$로 2개고 출력값이 $$Y$$하나이기 때문에

**Weight**의 크기는 **[2, 1]**의 크기다.

**Bias**는 항상 **출력값의 갯수**와 같으므로 **[1]**이다.

```python
X = tf.placeholder(tf.float32)
Y = tf.placeholder(tf.float32)
W = tf.Variable(tf.random_normal([2, 1]), name='weight')
b = tf.Variable(tf.random_normal([1]), name="bias")
```

### Hypothesis
**Sigmoid** 함수 사용

\begin{align} Sigmoid = \frac{1}{1+e^{-x}} \end{align}

**행렬 곱셉** 후 **Sigmoid Function**에 넣는다.

```python
hypothesis = tf.sigmoid(tf.matmul(X, W) + b)
```

### Cost Function
**Logistic Regression**의 **Cost Function**은 아래와 같다.

$$
C(H(x), y) =
-Y * log(H(x)
-(1 - Y) * log(1 - H(X))
$$

구해진 Cost를 가지고 **경사하강법**을 사용해 학습을 진행한다.

```python
cost = -tf.reduce_mean(Y * tf.log(hypothesis) + (1 - Y) * tf.log(1 - hypothesis))
train = tf.train.GradientDescentOptimizer(learning_rate=0.01).minimize(cost)
```

### Training
위에 작성한 **Hypothesis**와 **Cost Function**을 사용해 학습을 진행

`cast`함수를 사용해 **Hypothesis**의 값이 **0.5**보다 클경우 **True**로

**0.5**보다 작을경우 **False**로 값을 바꾸어준다.

또한 예측값과 결과값을 비교해 `cast`함수를 사용한 **결과의 평균**을 구해 **정확도**를 계산한다.

```python
predicted = tf.cast(hypothesis > 0.5, dtype=tf.float32)
accuracy = tf.reduce_mean(tf.cast(tf.equal(predicted, Y), dtype=tf.float32))

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    for step in range(10001):
        _, cost_val, w_val = sess.run(
            [train, cost, W], feed_dict={X: x_data, Y: y_data}
        )

        if step % 1000 == 0:
            print(step, cost_val, w_val)

    h, p, a = sess.run(
        [hypothesis, predicted, accuracy], feed_dict={X: x_data, Y: y_data}
    )

    print(f"\nHypothesis:\n{h} \nPredicted:\n{p} \nAccuracy:\n{a}")
```

    0 0.99409974 [[-1.081909 ]
     [ 2.3671274]]
    1000 0.75658506 [[-0.85761964]
     [ 1.1723609 ]]
    2000 0.7131047 [[-0.46316972]
     [ 0.65583754]]
    3000 0.699124 [[-0.23715353]
     [ 0.36725613]]
    4000 0.6949206 [[-0.11818436]
     [ 0.20615283]]
    5000 0.69367814 [[-0.05713516]
     [ 0.11657296]]
    6000 0.69330937 [[-0.02641575]
     [ 0.06656651]]
    7000 0.6931982 [[-0.01132161]
     [ 0.03844175]]
    8000 0.6931639 [[-0.00415682]
     [ 0.02247488]]
    9000 0.69315296 [[-0.00093976]
     [ 0.01331245]]
    10000 0.6931492 [[0.00036489]
     [0.00799207]]

    Hypothesis:
    [[0.4987609 ]
     [0.50075895]
     [0.49885216]
     [0.5008502 ]]
    Predicted:
    [[0.]
     [1.]
     [0.]
     [1.]]
    Accuracy:
    0.5

모델이 정확하고 문제가 없음에도 불구하고 정확도가 높지않다.

정확도를 올리기 위해서 **Neural Network**를 사용하면 된다.

### Using Neural Network
여러개의 Layer를 사용하는 **NN**을 사용

layer1의 결과물을 hypothesis에 넣어 한번더 학습시킨다.

```python
W1 = tf.Variable(tf.random_normal([2, 2]), name='weight1')
b1 = tf.Variable(tf.random_normal([2]), name='bias1')
layer1 = tf.sigmoid(tf.matmul(X, W1) + b1)

W2 = tf.Variable(tf.random_normal([2, 1]), name='weight2')
b2 = tf.Variable(tf.random_normal([1]), name='bias2')
hypothesis = tf.sigmoid(tf.matmul(layer1, W2) + b2)
```

이때 **weight의 크기**를 잘 정해주어야 한다.

layer1의 입력값은 $$X_1$$, $$X_2$$로 **2개**이고 출력값의 갯수는

임의로 결정하면 되므로 $$W_1$$의 크기는 **[2, 2]**로 정했다.

출력값의 갯수를 2개로 결정했으니 **bias**의 크기는 **[2]**가 된다.

layer1을 입력으로 받는 다른 layer의 입력값의 개수는

**layer1의 입력값**인 **2개**로 같고 출력값은

$$ \bar{Y} $$이므로 **1개**이기 때문에 **weight**의 크기는 **[2. 1]**이다.

**bias**의 크기는 출력값이 **1개**이기 때문에 **[1]**이다.

따라서 결정된 Layer들의 **Weigth**과 **Bias**의 크기는 아래와 같다.

|        | Weight | Bias |
| ------ | ------ | ---- |
| Layer1 | [2, 2] | [2]  |
| Layer2 | [2, 1] | [1]  |

### Training
2개의 Layer를 겹친 **Neural Network**를 이용해 한번 더 학습

```python
cost = -tf.reduce_mean(Y * tf.log(hypothesis) + (1 - Y) * tf.log(1 - hypothesis))
train = tf.train.GradientDescentOptimizer(learning_rate=0.1).minimize(cost)

predicted = tf.cast(hypothesis > 0.5, dtype=tf.float32)
accuracy = tf.reduce_mean(tf.cast(tf.equal(predicted, Y), dtype=tf.float32))

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    for step in range(10001):
        _, cost_val = sess.run(
            [train, cost], feed_dict={X: x_data, Y: y_data}
        )

        if step % 1000 == 0:
            print(step, cost_val)

    h, p, a = sess.run(
        [hypothesis, predicted, accuracy], feed_dict={X: x_data, Y: y_data}
    )

    print(f"\nHypothesis:\n{h} \nPredicted:\n{p} \nAccuracy:\n{a}")
```

    0 0.82048273
    1000 0.67851186
    2000 0.606001
    3000 0.4774453
    4000 0.1442676
    5000 0.06338048
    6000 0.039486427
    7000 0.028429154
    8000 0.022119436
    9000 0.01806068
    10000 0.015238974

    Hypothesis:
    [[0.01475588]
     [0.9810414 ]
     [0.985857  ]
     [0.01261569]]
    Predicted:
    [[0.]
     [1.]
     [1.]
     [0.]]
    Accuracy:
    1.0

같은 **Hypothesis**와 **Cost Function**을 사용하였으나,

2개의 Layer를 사용한 것만으로 **모든 값을 예측**하는데 성공하였다.

### 전체적인 코드

```python
X = tf.placeholder(tf.float32, [None, 2])
Y = tf.placeholder(tf.float32, [None, 1])

W1 = tf.Variable(tf.random_normal([2, 2]), name='weight1')
b1 = tf.Variable(tf.random_normal([2]), name='bias1')
layer1 = tf.sigmoid(tf.matmul(X, W1) + b1)

W2 = tf.Variable(tf.random_normal([2, 1]), name='weight2')
b2 = tf.Variable(tf.random_normal([1]), name='bias2')
hypothesis = tf.sigmoid(tf.matmul(layer1, W2) + b2)

cost = -tf.reduce_mean(Y * tf.log(hypothesis) + (1 - Y) * tf.log(1 - hypothesis))
train = tf.train.GradientDescentOptimizer(learning_rate=0.1).minimize(cost)

predicted = tf.cast(hypothesis > 0.5, dtype=tf.float32)
accuracy = tf.reduce_mean(tf.cast(tf.equal(predicted, Y), dtype=tf.float32))

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    for step in range(10001):
        _, cost_val = sess.run(
            [train, cost], feed_dict={X: x_data, Y: y_data}
        )

        if step % 1000 == 0:
            print(step, cost_val)

    h, p, a = sess.run(
        [hypothesis, predicted, accuracy], feed_dict={X: x_data, Y: y_data}
    )

    print(f"\nHypothesis:\n{h} \nPredicted:\n{p} \nAccuracy:\n{a}")
```

    0 1.0340692
    1000 0.68017256
    2000 0.5653977
    3000 0.24299939
    4000 0.09485774
    5000 0.053028427
    6000 0.035822317
    7000 0.026766824
    8000 0.021257266
    9000 0.017577857
    10000 0.014957345

    Hypothesis:
    [[0.01222286]
     [0.98380554]
     [0.9837938 ]
     [0.01474641]]
    Predicted:
    [[0.]
     [1.]
     [1.]
     [0.]]
    Accuracy:
    1.0

### Deep Neural Network로 XOR 해결하기

```python
X = tf.placeholder(tf.float32, [None, 2])
Y = tf.placeholder(tf.float32, [None, 1])

W1 = tf.Variable(tf.random_normal([2, 10]), name='weight1')
b1 = tf.Variable(tf.random_normal([10]), name='bias1')
layer1 = tf.sigmoid(tf.matmul(X, W1) + b1)

W2 = tf.Variable(tf.random_normal([10, 10]), name='weight2')
b2 = tf.Variable(tf.random_normal([10]), name='bias2')
layer2 = tf.sigmoid(tf.matmul(layer1, W2) + b2)

W3 = tf.Variable(tf.random_normal([10, 10]), name='weight3')
b3 = tf.Variable(tf.random_normal([10]), name='bias3')
layer3 = tf.sigmoid(tf.matmul(layer2, W3) + b3)

W4 = tf.Variable(tf.random_normal([10, 1]), name='weight4')
b4 = tf.Variable(tf.random_normal([1]), name='bias4')
hypothesis = tf.sigmoid(tf.matmul(layer3, W4) + b4)

cost = -tf.reduce_mean(Y * tf.log(hypothesis) + (1 - Y) * tf.log(1 - hypothesis))
train = tf.train.GradientDescentOptimizer(learning_rate=0.1).minimize(cost)

predicted = tf.cast(hypothesis > 0.5, dtype=tf.float32)
accuracy = tf.reduce_mean(tf.cast(tf.equal(predicted, Y), dtype=tf.float32))

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    for step in range(10001):
        _, cost_val = sess.run(
            [train, cost], feed_dict={X: x_data, Y: y_data}
        )

        if step % 1000 == 0:
            print(step, cost_val)

    h, p, a = sess.run(
        [hypothesis, predicted, accuracy], feed_dict={X: x_data, Y: y_data}
    )

    print("\nHypothesis: ", h, "\nPredicted: ", p, "\nAccuracy: ", a)
```

    0 0.786564
    1000 0.21167177
    2000 0.030031722
    3000 0.013181003
    4000 0.008059783
    5000 0.005704046
    6000 0.0043751714
    7000 0.0035298648
    8000 0.0029479803
    9000 0.0025245028
    10000 0.0022032897

    Hypothesis:  [[0.00249692]
     [0.99716216]
     [0.998467  ]
     [0.00193409]]
    Predicted:  [[0.]
     [1.]
     [1.]
     [0.]]
    Accuracy:  1.0

**Hypothesis**의 값을 보면 0이 되어야하는 값은

0에 더 가까워졌고 1이 되어야하는 값은 1에 더 가까워졌다.

여러개의 Layer를 사용해 **Deep**하게 학습한 결과

**정확도**가 더 **높아**지게 되었다.


TensorFlow Quantum example, adapted from
[*TensorFlow Quantum: A Software Framework for Quantum Machine Learning*](http://arxiv.org/abs/2003.02989)

Loading of libraries and initialization

```python
import random

import cirq
from cirq.contrib.svg import SVGCircuit
import matplotlib.pyplot as plt
import numpy as np
import sympy
import tensorflow as tf
import tensorflow_quantum as tfq

if not tf.config.list_physical_devices('GPU'):
    print("Warning: GPU was not found, so simulations can be very slow")
```

TensorFlow Quantum example

```python
def generate_dataset(
        qubit, theta_a, theta_b, num_samples):
    q_data = []
    labels = []
    blob_size = abs(theta_a - theta_b) / 5
    for _ in range(num_samples):
        coin = random.random()
        spread_x, spread_y = np.random.uniform(-blob_size, blob_size, 2)
        if coin < 0.5:
            label = [1, 0]
            angle = theta_a + spread_y
        else:
            label = [0, 1]
            angle = theta_b + spread_y
        labels.append(label)
        q_data.append(cirq.Circuit(
            cirq.Ry(rads=-angle)(qubit),
            cirq.Rx(rads=-spread_x)(qubit)
        ))
    return tfq.convert_to_tensor(q_data), np.array(labels)

# Dataset generation
qubit = cirq.GridQubit(0, 0)
theta_a = 1
theta_b = 4
num_samples = 200
q_data, labels = generate_dataset(qubit, theta_a, theta_b, num_samples)

# Quantum parametric model
theta = sympy.Symbol("theta")
q_model = cirq.Circuit(cirq.Ry(rads=theta)(qubit))
q_data_input = tf.keras.Input(shape=(), dtype=tf.dtypes.string)
expectation = tfq.layers.PQC(q_model, cirq.Z(qubit))
expectation_output = expectation(q_data_input)

classifier = tf.keras.layers.Dense(2, activation=tf.keras.activations.softmax)
classifier_output = classifier(expectation_output)
model = tf.keras.Model(inputs=q_data_input, outputs=classifier_output)

optimizer = tf.keras.optimizers.Adam(learning_rate=0.1)

loss = tf.keras.losses.CategoricalCrossentropy()
model.compile(optimizer=optimizer, loss=loss)
history = model.fit(x=q_data, y=labels, epochs=50)

test_data, _ = generate_dataset(qubit, theta_a, theta_b, 1)
p = model.predict(test_data)[0]
print(f"prob(a)={p[0]:.4f}, prob(b)={p[1]:.4f}")
```

    Epoch 1/50
    7/7 [==============================] - 1s 5ms/step - loss: 0.7619
    Epoch 2/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.3277
    Epoch 3/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.1140
    Epoch 4/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0576
    Epoch 5/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0306
    Epoch 6/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.0201
    Epoch 7/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0157
    Epoch 8/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.0130
    Epoch 9/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0111
    Epoch 10/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0095
    Epoch 11/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0088
    Epoch 12/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0080
    Epoch 13/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.0074
    Epoch 14/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.0068
    Epoch 15/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.0063
    Epoch 16/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.0058
    Epoch 17/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0056
    Epoch 18/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0050
    Epoch 19/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.0047
    Epoch 20/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0046
    Epoch 21/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0043
    Epoch 22/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0040
    Epoch 23/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0038
    Epoch 24/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0036
    Epoch 25/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0035
    Epoch 26/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.0034
    Epoch 27/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0032
    Epoch 28/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0030
    Epoch 29/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.0030
    Epoch 30/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.0027
    Epoch 31/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0026
    Epoch 32/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.0026
    Epoch 33/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0024
    Epoch 34/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0024
    Epoch 35/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.0023
    Epoch 36/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0022
    Epoch 37/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.0021
    Epoch 38/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.0020
    Epoch 39/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0020
    Epoch 40/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.0020
    Epoch 41/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0018
    Epoch 42/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0018
    Epoch 43/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.0017
    Epoch 44/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.0017
    Epoch 45/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.0016
    Epoch 46/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0016
    Epoch 47/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0015
    Epoch 48/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.0015
    Epoch 49/50
    7/7 [==============================] - 0s 4ms/step - loss: 0.0014
    Epoch 50/50
    7/7 [==============================] - 0s 5ms/step - loss: 0.0014
    prob(a)=0.0007, prob(b)=0.9993


```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import fsolve
from scipy.integrate import quad
import itertools
import copy
from image_matrix_helper import compute_master_list, imshow_list, rgb_map, color_to_rgb, list_to_matrix
import random
import time

nb_start = time.time()
```

## Simulation for the Simple Particle Binding Model

In this notebook, we simulate a thermal system of particles of various colors binding onto a grid. We have $R$ different types of particles and particle of type $j$ has $n_j$ copies in the system. Particles can exist both on and off the grid and particles of type $j$ have a binding affinity of $\gamma_j$ to the grid. Here we want to use simulations of this system to affirm analytical calculations of the average number of bound particles.

### Numerical representations of analytical work

#### System Partition Function

The partition function for this system is

\begin{equation}
W_{\boldsymbol{n}}(\{\gamma_i\}) = \frac{1}{2\pi i}  \oint \frac{dz}{z} \,e^{z}\prod_{i=1}^{R}\left(\gamma_i + \frac{1}{z}\right)^{n_i},
\label{eq:Wdef0}
\end{equation}

*It would be convenient to represent this result numerically and use it to compute the exact expression for $\langle k \rangle$, but Python's numerical complex integration abilities are limited.*

```python
# gamma function definition
gamma_func = lambda E0, Ev, T: 4*np.sqrt(2)*np.exp(E0/T)*(Ev/T)**(3/2)
```

```python
## test values
R = 50
E0s = 4.15*np.ones(R)
Evs = 1e-3*np.ones(R)
Nelems = np.ones(R)
T0 = 0.2
```

#### Thermal limits of disorder

At what temperature, will all the ligands in the be be attached to a receptor. $\langle k \rangle$ representing the number of ligands bound to receptors, at what temperature does the system settle into the macrostate  $\langle \ell_{\text{tot}}\rangle = N$?"

We found that this temperature $\beta_c = 1/k_BT_c$ is defined implicitly by

$$ 1 = \sum_{j=1}^R n_j \gamma_{j}^{-1} $$

```python
# implicit thermal constraint

def ligand_bind_constr(beta, E0s, Evs, Ns):

    gammas_ = gamma_func(E0s, Evs, beta**(-1))

    F =  1-np.sum(Ns*gammas_**(-1))

    return F

# critical temperature
kBTcrit = lambda E0s, Evs, Ns: 1.0/fsolve(ligand_bind_constr, x0 = 0.5, args = (E0s, Evs, Ns))[0]

# Test function
kBTcrit(E0s, Evs, Nelems)
```

    0.3749141536398384

#### Equations of Large $N$ approximation

The order parameter can be approximated as
\begin{align}
\langle k \rangle & =   \sum_{j=1}^R \frac{n_j \gamma_j \bar{z}}{\gamma_j \bar{z} +1},
\end{align}
where $\bar{z}$ is defined from
\begin{equation}
\bar{z} = \sum_{j=1}^{R} \frac{n_j}{\bar{z} \gamma_j+1}.
\end{equation}

#### Analytical Approximation

```python
# function that determines xbar
def zbar_func(z, T, E0s, Evs, Ns):

    F = np.ones(2)
    R = len(Ns)

    gammas_ = gamma_func(E0s, Evs, T)

    F = z- np.sum(Ns/(z*gammas_+1))

    return F

# function that determs k_avg for large N
def avg_k_approx(T, E0s,  Evs, Ns):

    R = len(Ns)
    zbar = fsolve(zbar_func, x0 = (110), args = (T, E0s, Evs, Ns))

    gammas_ = gamma_func(E0s, Evs, T)

    k_js = Ns*zbar*gammas_/(zbar*gammas_+1)

    return np.sum(k_js)

```

#### Example analytical plot

```python
# parameters for the integral
np.random.seed(42)

# temperature vals
Tvals = np.linspace(0.1, 3.0, 50)

R = 50
sigma_E = 1.5
E0bar = 6.15
E0s = np.random.randn(R)*sigma_E+E0bar
Nelems = np.random.randint(1,10,R)
Evs = np.ones(R)*0.001
avg_k_approx_vals = [avg_k_approx(T, E0s, Evs, Nelems)/np.sum(Nelems) for T in Tvals]

## plotting order parameters
ax = plt.subplot(111)
ax.plot(Tvals, avg_k_approx_vals, label = r"$\langle k \rangle/N$")
ax.plot(Tvals, np.ones(len(Tvals)), 'k--')

ax.set_ylabel(r'$\langle k \rangle$', fontsize = 18)
ax.set_xlabel(r'$k_BT$', fontsize = 18, labelpad = 10.5)
ax.grid(alpha = 0.5)
ax.axvline(x = kBTcrit(E0s, Evs, Nelems), color = 'r', linestyle = '--')
# Hide the right and top spines
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)

# Only show ticks on the left and bottom spines
ax.yaxis.set_ticks_position('left')
ax.xaxis.set_ticks_position('bottom')
```

### Metropolis Hastings simulation code

#### Microstate transitions

```python
## dissociation operator
def trans_dissoc(free_objs, bound_objs):

        # indices of non-empty
        indxs = [i for i, x in enumerate(bound_objs) if x != "-"]

        # random choice for bound object
        random_indx = random.choice(indxs)

        ## new state vector
        free_objs_new = copy.deepcopy(free_objs)
        bound_objs_new = copy.deepcopy(bound_objs)

        # putting empty slot
        bound_objs_new[random_indx] = '-'
        # appending previously bound object to free objects
        free_objs_new.append(bound_objs[random_indx])

        return free_objs_new, bound_objs_new

## association operator
def trans_assoc(free_objs, bound_objs):

        # random element to associate
        elem = random.choice(free_objs)

        # indices of empty spaces
        indxs = [i for i, x in enumerate(bound_objs) if x == "-"]

        # random choice for empty space
        random_indx = random.choice(indxs)

        ## new state vector
        free_objs_new = copy.deepcopy(free_objs)
        bound_objs_new = copy.deepcopy(bound_objs)

        ## state
        free_objs_new.remove(elem)
        bound_objs_new[random_indx] = elem

        return free_objs_new, bound_objs_new
```

#### Logarithm of Botlzmann factor

The logarithm of the Botlzmann factor for a microstate (i.e., the temperature normalized negative energy of the microstate) is defined as

\begin{equation}
\beta E(\boldsymbol{k}) = \sum_{i=1}^R k_i \ln \gamma_i.
\label{eq:sim_en}
\end{equation}

```python
def log_boltz(free_objs, bound_objs, mstr_vec,  gammas, name_key):

    elem_set = list(set(mstr_vec))
    count_dict = dict()
    for elem in elem_set:
        count_dict[elem] = bound_objs.count(elem)

    bind_log_factor = 0
    for elem in elem_set:
        key = name_key[elem]
        bind_log_factor += count_dict[elem]*np.log(gammas[key])

    return bind_log_factor
```

#### Checking logarithm of Boltzmann factor definition

```python
# defining name key
name_key0 = dict()
key_list = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', ]
for j in range(len(key_list)):
    name_key0[key_list[j]] = j

# random energies
np.random.seed(2)
q1 = np.random.rand(10)

# sample master list
orig_free = ['F', 'G', 'H', 'I', 'J']
orig_bound = ['A', 'B', 'C', 'D', 'E', '-','-','-','-','-']
sample_master = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J']

print('Energy for original ordering:', log_boltz(orig_free, orig_bound, sample_master, np.exp(-q1), name_key0))
print('Checking energy value:', -np.sum([q1[k] for k in range(len(sample_master)) if sample_master[k] in orig_bound]))
print('Number of bound elements:', np.sum([1 for elem in orig_bound if elem!='-']))
print('-----')
new_free, new_bound = trans_dissoc(orig_free, orig_bound)
print('Energy after switching first and last values:', log_boltz(new_free, new_bound, sample_master, np.exp(-q1), name_key0))
print('Checking energy value:', -np.sum([q1[k] for k in range(len(sample_master)) if sample_master[k] in new_bound]))
print('Number of bound elements:',  np.sum([1 for elem in new_bound if elem!='-']))
```

    Energy for original ordering: -1.8672738065543701
    Checking energy value: -1.86727380655437
    Number of bound elements: 5
    -----
    Energy after switching first and last values: -1.4312789044123664
    Checking energy value: -1.4312789044123664
    Number of bound elements: 4

#### Metropolis Hastings algorithm

```python
### Metropolis Monte Carlo Algorithm

## loads uniform random sampling
runif = np.random.rand

def met_assembly_grid(Niter, free_objs, bound_objs, mstr_vec, gammas, name_key):
    '''
    #################################################################
    # function to sample using Metropolis
    #
    # n_iter:  number of iterations
    # initial_state: initial state for the start position for our chain
    # gamma: energy cost for incorrect component
    # temp: temperature
    ##################################################################
    '''

    # Initialize state values
    free_objs_vals = [0]*(Niter+1)
    bound_objs_vals = [0]*(Niter+1)

    # Set initial values
    free_objs_vals[0] = free_objs[:]
    bound_objs_vals[0] = bound_objs[:]

    # Initialize acceptance counts
    # We can use this to tune our number of steps
    accepted = 0

    for i in range(Niter):

        # get current monomer and dimer states
        current_free_objs = copy.deepcopy(free_objs_vals[i])
        current_bound_objs = copy.deepcopy(bound_objs_vals[i])

        N_free = len(current_free_objs)
        N_bound = len(current_bound_objs)-len(current_free_objs)

        u_trans = runif()

        if u_trans < 1/2: #first type of transition; monomer association

            if N_free < 1:

                log_alpha = np.log(1e-15)

            else:
                # proposed new monomer and dimer states
                new_free_objs, new_bound_objs = trans_assoc(current_free_objs, current_bound_objs)

                # transition elements
                log_init = log_boltz(current_free_objs, current_bound_objs, mstr_vec, gammas, name_key)
                log_final = log_boltz(new_free_objs, new_bound_objs, mstr_vec, gammas, name_key)

                # testing weight
                num = N_free*N_free
                den = N_bound+1

                # Log-acceptance rate
                log_alpha = log_final-log_init+np.log(num/den)

        elif 1/2 <= u_trans : #second type of transition; bound monomer dissociation

            if N_bound <1:

                log_alpha = np.log(1e-15)

            else:

                # proposed new monomer and dimer states
                new_free_objs, new_bound_objs = trans_dissoc(current_free_objs, current_bound_objs)

                # transition elements
                log_init = log_boltz(current_free_objs, current_bound_objs, mstr_vec,  gammas, name_key)
                log_final = log_boltz(new_free_objs, new_bound_objs, mstr_vec, gammas, name_key)

                # testing weight
                num = N_bound
                den = (N_free+1)*(N_free+1)

                # Log-acceptance rate
                log_alpha = log_final-log_init+np.log(num/den)

        # Sample a uniform random variate
        u = runif()

        # Test proposed value
        if np.log(u) < log_alpha:
            # Accept
            free_objs_vals[i+1] = new_free_objs
            bound_objs_vals[i+1] = new_bound_objs

            #log_current_prob = log_proposed_prob
            accepted += 1
        else:
            # Stay put
            free_objs_vals[i+1] = free_objs_vals[i]
            bound_objs_vals[i+1] = bound_objs_vals[i]

    # return our samples and the number of accepted steps
    return free_objs_vals, bound_objs_vals, accepted
```

#### Computing microstate average from simiulations

```python
def avg_k(bound_objs_vals, Nmc):

    """
    Microstate average of number of bound objects
    We only consider microstates near the end of theh chain to ensure
    that the system has equilibrated
    """

    length = int(Nmc/50)

    ls = [0]*length
    ls = np.array(ls)
    for k in range(length):
        ls[k] = len(bound_objs_vals[Nmc-length+k]) - bound_objs_vals[Nmc-length+k].count('-')

    return(np.mean(ls))
```

#### Image grid for completely correct configuration

```python
# defining master_list
master_list =compute_master_list()
# testing plot
imshow_list(master_list)
# defining Nelems
Nelems = np.zeros(8)
key_list = list(rgb_map.keys())[:-1]
name_key_ = dict()
for j in range(len(key_list)):
    name_key_[key_list[j]] = j
    Nelems[j] = master_list.count(key_list[j])
```

```python
# displaying copy-number counts of the various elements
Nelems
```

    array([ 9.,  9., 10.,  5.,  7.,  6.,  3., 51.])

#### Simulating system

```python
## Generate lf for each temperature from .03 to 2.0 in npoints steps

t0 = time.time()

# number of steps for MC algortihm
Nmc = 10000

# binding energy parameters
R = 8
E0_bar, sigma_E = 6.0, 2.0
E0s = np.random.randn(R)*sigma_E+E0_bar
Evs = np.ones(R)*0.001

# initial monomer and dimer states;
# system in microstate of all correct dimers
random.seed(0)
free_objs_0 = []
bound_objs_0 = random.sample(master_list, len(master_list))
mstr_vec = copy.deepcopy(master_list)

# temperature limits
Tmin = .05
Tmax = 3.0

npoints = 15 #number of temperature values
navg = 5 # number of times we run simulation at each temperature; 50 in paper
temp_vals = np.linspace(Tmin, Tmax, npoints).tolist()

# list of dimer values
sim_k_vals = [0]*npoints

# accepted list
accepted_list = [0]*npoints

# saved list for plotting
saved_list = dict()

for k in range(npoints):

    fin_k_vals = [0]*navg
    fin_accepted = [0]*navg

    for j in range(navg):

        # make copy of initial monomer and dimer states
        free_objs_copy = copy.deepcopy(free_objs_0)
        bound_objs_copy = copy.deepcopy(bound_objs_0)

        # defining helper functions
        gammas_ = gamma_func(E0s, Evs, temp_vals[k])

        # metroplois generator
        free_list, bound_list, accepted = met_assembly_grid(Nmc,
                                                            free_objs_copy,
                                                            bound_objs_copy,
                                                            mstr_vec,
                                                            gammas_,
                                                            name_key_)

        # averaging final states to compute observables
        fin_k_vals[j] = avg_k(bound_list, Nmc)
        fin_accepted[j] = accepted

    # saving every 5 temperatures
    if (k+1)%5 ==0 or k ==0:
        saved_list[k] = ['white' if x=='-' else x for x in bound_list[-1]]

    # averaging over computed equilibrium averages
    sim_k_vals[k] = np.mean(np.array(fin_k_vals))
    accepted_list[k] = np.mean(np.array(fin_accepted))

    t_prelim = time.time()
    print("Temperature Run:",str(k+1),"; Current Time:", round(t_prelim-t0,2),"secs")

t1 = time.time()
print("Total Simulation Run Time:",t1-t0,"secs")
```

    Temperature Run: 1 ; Current Time: 5.93 secs
    Temperature Run: 2 ; Current Time: 10.95 secs
    Temperature Run: 3 ; Current Time: 19.29 secs
    Temperature Run: 4 ; Current Time: 29.63 secs
    Temperature Run: 5 ; Current Time: 41.04 secs
    Temperature Run: 6 ; Current Time: 54.38 secs
    Temperature Run: 7 ; Current Time: 67.18 secs
    Temperature Run: 8 ; Current Time: 80.79 secs
    Temperature Run: 9 ; Current Time: 95.19 secs
    Temperature Run: 10 ; Current Time: 109.09 secs
    Temperature Run: 11 ; Current Time: 123.55 secs
    Temperature Run: 12 ; Current Time: 139.58 secs
    Temperature Run: 13 ; Current Time: 154.84 secs
    Temperature Run: 14 ; Current Time: 169.51 secs
    Temperature Run: 15 ; Current Time: 184.19 secs
    Total Simulation Run Time: 184.18926906585693 secs

#### Simulated image grid at various temperatures

```python
# figure parameters
rows, cols, idx = 2, 2, 0
fig, axes = plt.subplots(nrows=rows, ncols=cols, figsize=(9,9))

# list of keys for saved snapshots of image
img_key_list = list(saved_list.keys())

for i in range(rows):
    for j in range(cols):
        if idx < 4:
            axes[i, j].imshow(color_to_rgb(list_to_matrix(saved_list[img_key_list[idx]])))
            ax = plt.gca()

            # making ticks invisible
            axes[i, j].set_xticks([])
            axes[i, j].set_yticks([])

            # Minor ticks
            axes[i, j].set_xticks(np.arange(-0.5, 11, 1), minor=True)
            axes[i, j].set_yticks(np.arange(-0.5, 10, 1), minor=True)
            axes[i, j].tick_params(axis='y', colors='red')

            # labeling images
            itimes = 'i'*(1+idx) if idx<3 else 'iv'

            # Gridlines based on minor ticks
            axes[i, j].grid(which='minor', color='w', linestyle='-', linewidth=3)
            axes[i, j].set_title(fr'({itimes})  $k_BT = {round(temp_vals[img_key_list[idx]],2)}$', fontsize = 18, y = -.2)

            # making spines invisible
            axes[i, j].spines['right'].set_visible(False)
            axes[i, j].spines['top'].set_visible(False)
            axes[i, j].spines['left'].set_visible(False)
            axes[i, j].spines['bottom'].set_visible(False)

            idx +=1

# plt.savefig('binding_grid_assembly_grid_plots.png', bbox_inches='tight', format = 'png')
plt.show()
```

#### Comparing analytical and simulation results

```python
plt.figure(figsize = (7,5))
ax = plt.subplot(111)

# plot of simulation results
plt.plot(temp_vals,np.array(sim_k_vals)/np.sum(Nelems),
        label = f'Simulation',
        markersize = 7.5,
       marker = 'D',
       linestyle = '')

# plot of large N analytical results
k_avg_approx_vals = [avg_k_approx(T, E0s, Evs, Nelems)/np.sum(Nelems) for T in Tvals]
plt.plot(Tvals, k_avg_approx_vals, label = f'Large $N$',  markersize = 4.0, )

# plot of critical temperature
plt.axvline(x = kBTcrit(E0s, Evs, Nelems), color = 'k',  linestyle = '-.')

plt.legend(loc = 'best', fontsize = 12)
# plot formatting
ax.set_xlabel(r'$k_B T$', fontsize = 18)
plt.xlim([-0.01,3.2])
plt.ylim([0,1.1])
plt.ylabel(r'$\langle k \rangle/N$', fontsize = 18, rotation =90)
# plt.yaxis.set_label_coords(-0.1,.5)
plt.grid(alpha = 0.45)

# Hide the right and top spines
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)

# Only show ticks on the left and bottom spines
ax.yaxis.set_ticks_position('left')
ax.xaxis.set_ticks_position('bottom')

# increase label size
ax.tick_params(axis='both', which='major', labelsize=12)
ax.tick_params(axis='both', which='minor', labelsize=12)

ax.text(kBTcrit(E0s, Evs, Nelems)-.2, 0.25, r'$k_BT_{bind}$', color='black', fontsize = 15,
        bbox=dict(facecolor='white', edgecolor='none', pad=5.0))

for i in range(4):
    ax.text(temp_vals[img_key_list[i]]+.05, sim_k_vals[img_key_list[i]]/np.sum(Nelems)+.05,'('+'i'*(1+i)+')' if i<3 else '(iv)', fontsize = 14 )

# plt.savefig(f'simple_binding_model.png', bbox_inches='tight', format = 'png')
plt.show()
```

```python
print('Total Notebook Runtime: %.3f mins' % ((time.time()-nb_start)/60))
```

    Total Notebook Runtime: 3.096 mins


```python
%load_ext autoreload
%autoreload 2
%matplotlib inline
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

```python
import os.path
import pickle
import numpy as np
import pandas as pd
import seaborn as sns
from scipy.stats import norm
import matplotlib.pyplot as plt
from GenDataBallDrop1 import gen_data, plot_data
from sepia.SepiaModel import SepiaModel
from sepia.SepiaData import SepiaData
import sepia.SepiaPlot as SepiaPlot
from sepia.SepiaPredict import SepiaEmulatorPrediction
```

```javascript
%%javascript
MathJax.Hub.Config({
    TeX: { equationNumbers: { autoNumber: "AMS" } }
});
```

    <IPython.core.display.Javascript object>

```python
load_model = 0
save_model = 0
model_file = 'ball_drop_1_noD_model'
```

## Ball Drop with Varying Radii
Consider dropping balls of various radii $R$ from a tower at heights 5, 10, 15, and 20 meters. Each dropped ball experiment produces vector output in the form of a height-time curve, i.e., a curve showing time as a function of the current height $t=t(h)$.

Rather than performing these experiments, we generate synthetic experimental data from:
\begin{equation}
\frac{d^2h}{dt^2} = g - \frac{C}{R}\bigg(\frac{dh}{dt}\bigg)^2
\end{equation}
where $g$ is the acceleration due to gravity, $C$ is the coefficient of drag, and R is the radius of the ball. We will refer to data generated from equation (1) as system observations. To illustrate the modeling of discrepency between simulator and reality, we will generate simulated data from:
\begin{equation}
\frac{d^2h}{dt^2} = g - \frac{C}{R}\frac{dh}{dt}
\end{equation}
Note the simulated data will be systematically biased from the experimental data as it is generated with a linear velocity term $(dh/dt)$ rather than a squared velocity term. We will explicitly model this discrepecy.

These differential equations can be solved for height $h$ as a function of time $t$, ball radius $R$, and drag $C$. Observations will be the inverse: time recorded as a function of height, radius, and drag which are calculated by an optimizer from the scipy library in Python.

Acceleration due to gravity $g$ is assumed to be known, but the coefficient of drag $C$ is not, and is the target of inference. That is, the goal is to find the posterior distribution of $C$ that corresponds to the Sepia statistical model fit to all available data.

The first step is to compute system observations from (1) and simulated data from (2). As is standard practice in the study of computer codes a (space-filling) Latin hypercube (LH) design is used to efficiently collect simulator data over the domain of model parameters, in this case $(R,C)$.

Suppose we have data from $n=3$ experiments, one each for balls of radius $R={0.1,0.2,0.4}$ meters. Each experiment produces a curve of drop times made of four height-time pairs. The experimental heights are $h_e={5,10,15,20}$ meters. We assume a zero mean, normally distributed observation error $\epsilon \sim N(0,et)$ (IS THIS TRUE?). We fix $C=0.1*4\pi/3$, the coefficient of drag for a sphere, and $g=9.8 \; m/s^2$.

```python
et = 0.01                      # observation error
data_dict = gen_data(et)

# field data
R = data_dict['R']             # radii of balls   .1,.2,.4 (m)
h_field = data_dict['h_field'] # observed heights 5,10,15,20 (m)
y_field = data_dict['y_field'] # observed times
```

    generating data with C =  0.023873241463784306

For simulation design points, we use a scaled latin hypercube to select $m=25\;(R,C)$ pairs. We then scale the design so $R\in[0.05,0.45]$ to better match experimental data and $C\in[0.05,0.25]$ is chosen with some hindsight. The simulator will be run at these 25 design points and will return a time curve at 16 evenly spaced heights between 1.5 and 25 meters.

```python
# sim data
sim_design = data_dict['sim_design']

# to generate a different design
#import pyDOE
#sim_design = pyDOE.lhs(2,data_dict['m'])*np.array([.4,.2])+.05
#data_dict['sim_design'] = sim_design

R_sim = sim_design[:,0]
C_sim = sim_design[:,1]
h_sim = data_dict['h_sim']
y_sim = data_dict['y_sim']
```

See invertH.py for details on solving and inverting equations (1) and (2) to obtain experimental and simulated data. Below we show, for each Radius R, the physical reality, defined by equation (1), the noisy observations, and all 25 simulation realizations. For each R, the 3 simulations with design points nearest to R are shown as red, green, and blue lines.

```python
# see GenDataBallDrop1.py for ploting details
plot_data(data_dict)
```

# Preparing the data for Sepia
To use Sepia, we must package our data into a SepiaData object. In this example, the known inputs to the simulator are simply the vector of radii R_sim. We pass this into x_sim as a column vector. Similarly, C_sim is passed into t_sim as a column vector and is the parameter to be calibrated. We also pass in y_sim, the simulated time-height curves, and h_sim the heights associated with those times in y_sim. For the observed data, x_obs get the experimental radii R, and y_obs gets the experimental time-height curves generated from (1). Finally we pass in the heights at which the experimental time observations were taken, y_ind_obs = h_field.

```python
data = SepiaData(x_sim = np.reshape(R_sim,(len(R_sim),1)),
                 t_sim = np.reshape(C_sim,(len(C_sim),1)),
                 y_sim = y_sim, y_ind_sim = h_sim,
                 x_obs = np.reshape(R,(len(R),1)), y_obs = y_field, y_ind_obs=h_field)
```

### Transforming x, t, and y
Sepia required that the inputs $x,t$ lie in the interval $[0,1]^{p+q}$, and the responses $y_{sim},y_{obs}$ be $N(0,1)$.

```python
data.transform_xt()
data.standardize_y()
```

### Generate K basis
Sepia models multivariate observations and responses using a linear basis. These *principal components*, or scaled eigenvectors, are computed by the singular value decomposition.

```python
data.create_K_basis(2)
fig_sim, fig_obs = SepiaPlot.plot_K_basis(data)
```

We can visually check that our principal component weights are reasonable. In an ideal setting, weights would follow a standard normal distribution.

```python
fig_uw = SepiaPlot.plot_K_weights(data)
fig_uw_pairs = SepiaPlot.plot_u_w_pairs(data)
```

# Model initialization and MCMC
Now that our data is appropiately transformed, and out bases set, the model can be initialized and the posterior distributions of parameters sampled via Markov chain Monte Carlo (MCMC). Before performing MCMC, we call tune_step_sizes() to optimize proposal widths for each parameter.

```python
model = SepiaModel(data)
nsamp = 1000
if os.path.isfile(model_file+'.pkl') and load_model:
    model.restore_model_info(cachefile_name)
else:
    model.tune_step_sizes(50, 20)
    model.do_mcmc(1000)
    if save_model: model.save_model_info(cachefile_name)
```

    Warning: No D basis, proceeding with a no-discrepancy model.
    To use a D basis (normally recommended), call data.create_D_basis on your SepiaData object.
    Starting tune_step_sizes...
    Default step sizes:
    theta
    [[0.2]]
    betaU
    [[0.1 0.1]
     [0.1 0.1]]
    lamUz
    [[5. 5.]]
    lamWs
    [[100. 100.]]
    lamWOs
    [[100.]]
    lamOs
    [[877.96349747]]

    Step size tuning: 100%|██████████| 50/50 [00:04<00:00, 10.36it/s]

    Done with tune_step_size.
    Selected step sizes:
    theta
    [[0.22103611]]
    betaU
    [[0.57615966 2.22782773]
     [0.48810537 2.09129974]]
    lamUz
    [[0.7802294  2.03052135]]
    lamWs
    [[1445.40834612 5931.99312077]]
    lamWOs
    [[19342.75933036]]
    lamOs
    [[392.82342367]]

    MCMC sampling: 100%|██████████| 1000/1000 [00:04<00:00, 235.71it/s]

```python
model = SepiaModel(data)
nsamp = 1000
if os.path.isfile(model_file+'.pkl') and load_model:
    model.restore_model_info(cachefile_name)
else:
    model.tune_step_sizes(50, 20)
    model.do_mcmc(1000)
    if save_model: model.save_model_info(cachefile_name)
```

    Step size tuning:   0%|          | 0/50 [00:00<?, ?it/s]

    Warning: No D basis, proceeding with a no-discrepancy model.
    To use a D basis (normally recommended), call data.create_D_basis on your SepiaData object.
    Starting tune_step_sizes...
    Default step sizes:
    theta
    [[0.2]]
    betaU
    [[0.1 0.1]
     [0.1 0.1]]
    lamUz
    [[5. 5.]]
    lamWs
    [[100. 100.]]
    lamWOs
    [[100.]]
    lamOs
    [[712.11465609]]

    Step size tuning: 100%|██████████| 50/50 [00:04<00:00, 10.28it/s]
    MCMC sampling:   0%|          | 0/1000 [00:00<?, ?it/s]

    Done with tune_step_size.
    Selected step sizes:
    theta
    [[0.24223259]]
    betaU
    [[0.55342594 3.30341965]
     [0.46243984 1.78460551]]
    lamUz
    [[0.83169357 2.21217804]]
    lamWs
    [[1690.36808084 4475.91506232]]
    lamWOs
    [[16533.32931086]]
    lamOs
    [[490.9969195]]

    MCMC sampling: 100%|██████████| 1000/1000 [00:03<00:00, 255.97it/s]

```python
samples_dict = model.get_samples()
theta_names = ['C']
theta_pairs = SepiaPlot.theta_pairs(samples_dict,theta_names)
```

```python
mcmc_trace = SepiaPlot.mcmc_trace(samples_dict,theta_names)
```

```python
p_stats = SepiaPlot.param_stats(samples_dict,theta_names=theta_names,q1=.05,q2=.95,digits=4)
print(p_stats)
```

                        mean         sd  0.05 quantile  0.95 quantile
    C                 0.4091     0.0315         0.3510         0.4574
    betaU_1           8.8929     2.6138         5.6977        14.2231
    betaU_2           0.6795     0.2875         0.2988         1.1866
    betaU_3           3.8355     2.3656         1.5171         7.7850
    betaU_4           3.2546     2.2230         1.2336         6.6352
    lamUz_1           0.4918     0.1697         0.3073         0.8319
    lamUz_2           0.8097     0.2663         0.4510         1.3479
    lamWs_1         455.5997   326.4388       131.4904      1151.2552
    lamWs_2         935.2040   522.5637       228.2085      1922.4866
    lamWOs        28433.6470  2112.3813     24924.2425     32294.2816
    lamOs           110.1587    63.9493        31.6353       230.6232
    logPost        1690.9607     2.2664      1686.9577      1694.0968
    theta_native      0.1330     0.0060         0.1218         0.1423

```python
acf = SepiaPlot.plot_acf(model,50)
```

# Assessing emulator adequacy
It is important to examine diagnostics to understand the quality of the emulator in reproducing the simulator output at new, untried settings. The following are usful ways to asses the quality of the emulator fit.

### Principal components

### Parameters in the Gaussian Process Fit
Using the MCMC draws of the spacial dependence parameters $\beta$, $\rho=exp\{-\beta/4\}$ can be calculated to show the values on the more interpretable $[0,1]$ scale. The values of $\rho$ give us information about the dependence of the simulation output on each input parameter $x$ and $\theta$. We show boxplots of the posterior draws for $\rho$ for each $x$ and $\theta$. and for each principal component.
When $\rho$ is near 1, it suggests that particular component of the simulator output in linear in that dimention. As $\rho$ goes smaller, nonlinear activity is associated with that input. In other words larger values of $\rho$ indicate more smoothness. Additionally, smaller values of $\rho$ indicate increasing flexibility in the model response. Very small values could indicate overfitting. Thus, if any of the boxplots show values very near to zero, the model is suspect. The boxplots below are for correlation parameters associated with the K-basis. Discrepency basis correlation parameters could be similarly assesed.

```python
rho_box = SepiaPlot.rho_box_plots(model,labels=[r'$R_{ball}$','C'])
```

----
## Predictions of model output from the full model

First, predictions showing just the emulator response, using
SepiaEmulatorPredict

```python
model.verbose=True
x_pred=data.obs_data.x
n_pred=x_pred.shape[0]
pred_samples=model.get_samples(nburn=int(.1*nsamp),effectivesamples=True)
pred=SepiaEmulatorPrediction(x_pred=x_pred, samples=pred_samples, model=model)#, t_pred=t_pred)
```

    Max effective sample size over thetas: 270
    Total samples: 1000

```python
predw=pred.get_w()
plt.figure()
plt.plot(np.mean(predw,0).T,'--o')
plt.xlabel('PCA weights w')
plt.ylabel('w value for each predicted x \n mean over samples')
plt.show()
```

```python
predystd=pred.get_y(std=True)
plt.figure()
plt.plot(model.data.sim_data.y_ind,np.mean(predystd,0).T)
plt.xlabel('native data space y')
plt.ylabel('standardized predicted y for each predicted x \n mean over samples')
plt.show()
```

```python
predy=pred.get_y()
plt.figure()
plt.plot(model.data.sim_data.y_ind,np.mean(predy,0).T)
plt.xlabel('native data space y ')
plt.ylabel('native (original) predicted value for each x_pred \n mean over samples')
plt.show()
```

```python
plt.figure()
plt.plot(model.data.sim_data.y_ind,np.mean(predy[:,0,:],0).T,label='R=.1')
plt.plot(model.data.sim_data.y_ind,np.mean(predy[:,1,:],0).T,label='R=.2')
plt.plot(model.data.sim_data.y_ind,np.mean(predy[:,2,:],0).T,label='R=.4')
plt.plot(model.data.sim_data.y_ind,np.quantile(predy[:,0,:],[0.05, 0.95],axis=0).T,'k:')
plt.plot(model.data.sim_data.y_ind,np.quantile(predy[:,1,:],[0.05, 0.95],axis=0).T,'k:')
plt.plot(model.data.sim_data.y_ind,np.quantile(predy[:,2,:],[0.05, 0.95],axis=0).T,'k:')
plt.xlabel('native data space y ')
plt.ylabel('native (original) mean and (0.05,0.95) quantiles')
plt.legend()
plt.show()
```

### Summary plot of predicted uncertainty for an observation.
The 5th prediction done corresponds to the observation in the model, with xpred=0.5.

```python
predysim=pred.get_y()
colors = ('r','g','b')
for R_idx in range(3):
    plt.figure()
    plt.plot(model.data.obs_data.y_ind,model.data.obs_data.y[R_idx,:],marker='o',color=colors[R_idx])
    plt.plot(model.data.sim_data.y_ind,np.mean(predysim[:,R_idx,:],0).T,'--',c='lightgreen')
    plt.plot(model.data.sim_data.y_ind,np.quantile(predysim[:,R_idx,:],.05,0),'k:')
    plt.plot(model.data.sim_data.y_ind,np.quantile(predysim[:,R_idx,:],.95,0),'k:')
    plt.xlabel('native scale (obs grid)')
    plt.ylabel('native scale response')
    plt.title('Observation and model prediction')
    plt.legend(['obs R=.1','pred mean','pred q5/q95'])
    plt.ylim(0,4.5)
    plt.show()
```