# Theano tensor 模块：操作符和逐元素操作

## 操作符

```python
import theano
from theano import tensor as T
```

    Using gpu device 1: Tesla C2075 (CNMeM is disabled)

`tensor` 类型支持很多基本的操作：

```python
# 两个整形三维张量

a, b = T.itensor3("a"), T.itensor3("b")
```

### 算术操作

```python
print theano.pp(a + 3)      # T.add(a, 3) -> itensor3
print theano.pp(3 - a)      # T.sub(3, a)
print theano.pp(a * 3.5)    # T.mul(a, 3.5) -> ftensor3 or dtensor3 (depending on casting)
print theano.pp(2.2 / a)    # T.truediv(2.2, a)
print theano.pp(2.2 // a)   # T.intdiv(2.2, a)
print theano.pp(2.2**a)     # T.pow(2.2, a)
print theano.pp(b % a)      # T.mod(b, a)
```

    (a + TensorConstant{3})
    (TensorConstant{3} - a)
    (a * TensorConstant{3.5})
    (TensorConstant{2.20000004768} / a)
    (TensorConstant{2.20000004768} // a)
    (TensorConstant{2.20000004768} ** a)
    mod(b, a)

### 比特操作

```python
print theano.pp(a & b)      # T.and_(a,b)    bitwise and (alias T.bitwise_and)
print theano.pp(a ^ 1)      # T.xor(a,1)     bitwise xor (alias T.bitwise_xor)
print theano.pp(a | b)      # T.or_(a,b)     bitwise or (alias T.bitwise_or)
print theano.pp(~a)         # T.invert(a)    bitwise invert (alias T.bitwise_not)
```

    and_(a, b)
    xor(a, TensorConstant{1})
    or_(a, b)
    invert(a)

### 原地操作

`Theano` 不支持原地操作如 `+=` 等，`Theano` 的图优化解构会自动决定是否使用原地操作。如果需要更新变量的值，可以考虑使用共享变量 `theano.shared`。

## 逐元素操作

### 类型转换

`T.cast(x, dtype)` 用于类型转换：

```python
x = T.matrix()
x_as_int = T.cast(x, 'int32')
```

`T.cast(x, dtype)` 的机制与 `numpy.asarray(x, dtype)` 的机制类似，只有 `dtype` 不同时才会创建新的变量：

```python
print x_as_int is x
print T.cast(x, theano.config.floatX) is x
```

    False
    True

复数取实部，虚部，角度，模：

- `T.real(a)`
- `T.imag(a)`
- `T.angle(a)`
- `T.abs_(a)`

### 比较

`Theano` 的比较操作也是逐元素的：

- `T.lt(a, b)` : <
- `T.gt(a, b)` : >
- `T.le(a, b)` : <=
- `T.ge(a, b)` : >=
- `T.eq(a, b)` : ==
- `T.neq(a, b)` : !=

`Theano` 中没有 `bool` 类型，所有的 `bool` 类型都用 `int8` 表示。

```python
x, y = T.dmatrices('x','y')

print theano.pp(T.le(x, y))
```

    le(x, y)

除此之外，还有另一些与 `numpy` 类似的用法：

- `T.isnan(a)` : 是否 NAN
- `T.isinf(a)` : 是否 INF
- `T.isclose(a, b)` ：浮点数是否接近
- `T.allclose(a, b)` ：浮点数是否很接近

### 条件

`T.switch(cond, ift, iff)` 选择 `ift (if ture)` 和 `iff (if false)`。

`T.where(cond, ift, iff)` 与 `switch` 一致。

`T.clip(x, min, max)` 低于 `min` 的部分变成 `min`，超过 `max` 的部分变成 `max`。

### 数学操作

```python
a, b = T.matrices("a", "b")

print theano.pp(T.maximum(a, b))  # max(a, b)
print theano.pp(T.minimum(a, b))  # min(a, b)

print theano.pp(T.neg(a)) # -a
print theano.pp(T.inv(a)) # 1.0/a

print theano.pp(T.exp(a))
print theano.pp(T.log(a)), theano.pp(T.log2(a)), theano.pp(T.log10(a))       # log10(a)

print theano.pp(T.sgn(a))       # sgn(a)
print theano.pp(T.floor(a))     # floor(a)
print theano.pp(T.ceil(a))      # ceil(a)
print theano.pp(T.round(a))     # round(a)
print theano.pp(T.iround(a))    # iround(a)

print theano.pp(T.sqr(a))   # sqr(a)
print theano.pp(T.sqrt(a))  # sqrt(a)

print theano.pp(T.cos(a)), theano.pp(T.sin(a)), theano.pp(T.tan(a))
print theano.pp(T.cosh(a)), theano.pp(T.sinh(a)), theano.pp(T.tanh(a))         # tan(a)

print theano.pp(T.erf(a)), theano.pp(T.erfc(a)) # erf(a), erfc(a)
print theano.pp(T.erfinv(a)), theano.pp(T.erfcinv(a))

print theano.pp(T.gamma(a))    # gamma(a)
print theano.pp(T.gammaln(a))  # log(gamma(a))
print theano.pp(T.psi(a))      # digamma(a)
```

    maximum(a, b)
    minimum(a, b)
    (-a)
    inv(a)
    exp(a)
    log(a) log2(a) log10(a)
    sgn(a)
    floor(a)
    ceil(a)
    round_half_away_from_zero(a)
    int64(round_half_away_from_zero(a))
    sqr(a)
    sqrt(a)
    cos(a) sin(a) tan(a)
    cosh(a) sinh(a) tanh(a)
    erf(a) erfc(a)
    erfinv(a) erfcinv(a)
    gamma(a)
    gammaln(a)
    psi(a)

其中 `erf, erfc` 定义如下：
https://en.wikipedia.org/wiki/Error_function

$$
\operatorname{erf}(x) = \frac{2}{\sqrt\pi} \int_0^x e^{-t^2} dt
$$
$$
\begin{align}
             \operatorname{erfc}(x) & = 1-\operatorname{erf}(x) \\
                                    & = \frac{2}{\sqrt\pi} \int_x^{\infty} e^{-t^2}\,\mathrm dt \\
                                    & = e^{-x^2} \operatorname{erfcx}(x)
\end{align}
$$

`erfinv, erfcinv` 为其反函数：1
https://en.wikipedia.org/wiki/Error_function#Inverse_functions

### Broadcasting

图示如上。

## 线性代数

矩阵乘法：`T.dot(x, y)`

向量外积：`T.outer(x, y)`

张量乘法：`tensordot(a, b, axes=2)`

`axes` 参数表示 `a` `b` 对应要去掉的维度。

```python
import numpy as np

a = np.random.random((2,3,4))
b = np.random.random((5,6,4,3))

#tensordot
c = np.tensordot(a, b, [[1,2],[3,2]])

#loop replicating tensordot
a0, a1, a2 = a.shape
b0, b1, _, _ = b.shape
cloop = np.zeros((a0,b0,b1))

#loop over non-summed indices -- these exist
#in the tensor product.
for i in range(a0):
    for j in range(b0):
        for k in range(b1):
            #loop over summed indices -- these don't exist
            #in the tensor product.
            for l in range(a1):
                for m in range(a2):
                    cloop[i,j,k] += a[i,l,m] * b[j,k,m,l]

assert np.allclose(c, cloop)

print a.shape, b.shape
print c.shape
```

    (2, 3, 4) (5, 6, 4, 3)
    (2, 5, 6)


# Regression in Python

***
This is a very quick run-through of some basic statistical concepts, adapted from [Lab 4 in Harvard's CS109](https://github.com/cs109/2015lab4) course. Please feel free to try the original lab if you're feeling ambitious :-) The CS109 git repository also has the solutions if you're stuck.

* Linear Regression Models
* Prediction using linear regression

Linear regression is used to model and predict continuous outcomes with normal random errors. There are nearly an infinite number of different types of regression models and each regression model is typically defined by the distribution of the prediction errors (called "residuals") of the type of data. Logistic regression is used to model binary outcomes whereas Poisson regression is used to predict counts. In this exercise, we'll see some examples of linear regression as well as Train-test splits.

The packages we'll cover are: `statsmodels`, `seaborn`, and `scikit-learn`. While we don't explicitly teach `statsmodels` and `seaborn` in the Springboard workshop, those are great libraries to know.
***

***

```python
# special IPython command to prepare the notebook for matplotlib and other libraries
%matplotlib inline

import numpy as np
import pandas as pd
import scipy.stats as stats
import matplotlib.pyplot as plt
import sklearn

import seaborn as sns

# special matplotlib argument for improved plots
from matplotlib import rcParams
sns.set_style("whitegrid")
sns.set_context("poster")

```

***
# Part 1: Introduction to Linear Regression
### Purpose of linear regression
***

 Given a dataset containing predictor variables $X$ and outcome/response variable $Y$, linear regression can be used to:

 Build a predictive model to predict future values of $\hat{Y}$, using new data $X^*$ where $Y$ is unknown.

 Model the strength of the relationship between each independent variable $X_i$ and $Y$

 Many times, only a subset of independent variables $X_i$ will have a linear relationship with $Y$

 Need to figure out which $X_i$ contributes most information to predict $Y$

It is in many cases, the first pass prediction algorithm for continuous outcomes.

### A Brief Mathematical Recap
***

[Linear Regression](http://en.wikipedia.org/wiki/Linear_regression) is a method to model the relationship between a set of independent variables $X$ (also knowns as explanatory variables, features, predictors) and a dependent variable $Y$.  This method assumes the relationship between each predictor $X$ is **linearly** related to the dependent variable $Y$. The most basic linear regression model contains one independent variable $X$, we'll call this the simple model.

$$ Y = \beta_0 + \beta_1 X + \epsilon$$

where $\epsilon$ is considered as an unobservable random variable that adds noise to the linear relationship. In linear regression, $\epsilon$ is assumed to be normally distributed with a mean of 0. In other words, what this means is that on average, if we know $Y$, a roughly equal number of predictions $\hat{Y}$ will be above $Y$ and others will be below $Y$. That is, on average, the error is zero. The residuals, $\epsilon$ are also assumed to be "i.i.d.": independently and identically distributed. Independence means that the residuals are not correlated -- the residual from one prediction has no effect on the residual from another prediction. Correlated errors are common in time series analysis and spatial analyses.

* $\beta_0$ is the intercept of the linear model and represents the average of $Y$ when all independent variables $X$ are set to 0.

* $\beta_1$ is the slope of the line associated with the regression model and represents the average effect of a one-unit increase in $X$ on $Y$.

* Back to the simple model. The model in linear regression is the *conditional mean* of $Y$ given the values in $X$ is expressed a linear function.

$$ y = f(x) = E(Y | X = x)$$

*Image from http://www.learner.org/courses/againstallodds/about/glossary.html. Note this image uses $\alpha$ and $\beta$ instead of $\beta_0$ and $\beta_1$.*

* The goal is to estimate the coefficients (e.g. $\beta_0$ and $\beta_1$). We represent the estimates of the coefficients with a "hat" on top of the letter.

$$ \hat{\beta}_0, \hat{\beta}_1 $$

* Once we estimate the coefficients $\hat{\beta}_0$ and $\hat{\beta}_1$, we can use these to predict new values of $Y$ given new data $X$.

$$\hat{y} = \hat{\beta}_0 + \hat{\beta}_1 x_1$$

* Multiple linear regression is when you have more than one independent variable and the estimation involves matrices
    * $X_1$, $X_2$, $X_3$, $\ldots$

* How do you estimate the coefficients?
    * There are many ways to fit a linear regression model
    * The method called **least squares** is the most common methods
    * We will discuss least squares

$$ Y = \beta_0 + \beta_1 X_1 + \ldots + \beta_p X_p + \epsilon$$

### Estimating $\hat\beta$: Least squares
***
[Least squares](http://en.wikipedia.org/wiki/Least_squares) is a method that can estimate the coefficients of a linear model by minimizing the squared residuals:

$$ \mathscr{L} = \sum_{i=1}^N \epsilon_i^2 = \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2  = \sum_{i=1}^N \left(y_i - \left(\beta_0 + \beta_1 x_i\right)\right)^2 $$

where $N$ is the number of observations and $\epsilon$ represents a residual or error, ACTUAL - PREDICTED.

#### Estimating the intercept $\hat{\beta_0}$ for the simple linear model

We want to minimize the squared residuals and solve for $\hat{\beta_0}$ so we take the partial derivative of $\mathscr{L}$ with respect to $\hat{\beta_0}$

$
\begin{align}
\frac{\partial \mathscr{L}}{\partial \hat{\beta_0}} &= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \epsilon^2 \\
&= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2 \\
&= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right)^2 \\
&= -2 \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right) \hspace{25mm} \mbox{(by chain rule)} \\
&= -2 \sum_{i=1}^N (y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i) \\
&= -2 \left[ \left( \sum_{i=1}^N y_i \right) - N \hat{\beta_0} - \hat{\beta}_1 \left( \sum_{i=1}^N x_i
\right) \right] \\
& 2 \left[ N \hat{\beta}_0 + \hat{\beta}_1 \sum_{i=1}^N x_i - \sum_{i=1}^N y_i \right] = 0 \hspace{20mm} \mbox{(Set equal to 0 and solve for $\hat{\beta}_0$)} \\
& N \hat{\beta}_0 + \hat{\beta}_1 \sum_{i=1}^N x_i - \sum_{i=1}^N y_i = 0 \\
& N \hat{\beta}_0 = \sum_{i=1}^N y_i - \hat{\beta}_1 \sum_{i=1}^N x_i \\
& \hat{\beta}_0 = \frac{\sum_{i=1}^N y_i - \hat{\beta}_1 \sum_{i=1}^N x_i}{N} \\
& \hat{\beta}_0 = \frac{\sum_{i=1}^N y_i}{N} - \hat{\beta}_1 \frac{\sum_{i=1}^N x_i}{N} \\
& \boxed{\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}}
\end{align}
$

Using this new information, we can compute the estimate for $\hat{\beta}_1$ by taking the partial derivative of $\mathscr{L}$ with respect to $\hat{\beta}_1$.

$
\begin{align}
\frac{\partial \mathscr{L}}{\partial \hat{\beta_1}} &= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \epsilon^2 \\
&= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2 \\
&= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right)^2 \\
&= 2 \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right) \left( -x_i \right) \hspace{25mm}\mbox{(by chain rule)} \\
&= -2 \sum_{i=1}^N x_i \left( y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i \right) \\
&= -2 \sum_{i=1}^N x_i (y_i - \hat{\beta}_0 x_i - \hat{\beta}_1 x_i^2) \\
&= -2 \sum_{i=1}^N x_i (y_i - \left( \bar{y} - \hat{\beta}_1 \bar{x} \right) x_i - \hat{\beta}_1 x_i^2) \\
&= -2 \sum_{i=1}^N (x_i y_i - \bar{y}x_i + \hat{\beta}_1\bar{x}x_i - \hat{\beta}_1 x_i^2) \\
&= -2 \left[ \sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i + \hat{\beta}_1\bar{x}\sum_{i=1}^N x_i - \hat{\beta}_1 \sum_{i=1}^N x_i^2 \right] \\
&= -2 \left[ \hat{\beta}_1 \left\{ \bar{x} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i^2 \right\} + \left\{ \sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i \right\}\right] \\
& 2 \left[ \hat{\beta}_1 \left\{ \sum_{i=1}^N x_i^2 - \bar{x} \sum_{i=1}^N x_i \right\} + \left\{ \bar{y} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i y_i \right\} \right] = 0 \\
& \hat{\beta}_1 = \frac{-\left( \bar{y} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i y_i \right)}{\sum_{i=1}^N x_i^2 - \bar{x}\sum_{i=1}^N x_i} \\
&= \frac{\sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i}{\sum_{i=1}^N x_i^2 - \bar{x} \sum_{i=1}^N x_i} \\
& \boxed{\hat{\beta}_1 = \frac{\sum_{i=1}^N x_i y_i - \bar{x}\bar{y}n}{\sum_{i=1}^N x_i^2 - n \bar{x}^2}}
\end{align}
$

The solution can be written in compact matrix notation as

$$\hat\beta =  (X^T X)^{-1}X^T Y$$

We wanted to show you this in case you remember linear algebra, in order for this solution to exist we need $X^T X$ to be invertible. Of course this requires a few extra assumptions, $X$ must be full rank so that $X^T X$ is invertible, etc. Basically, $X^T X$ is full rank if all rows and columns are linearly independent. This has a loose relationship to variables and observations being independent respective. **This is important for us because this means that having redundant features in our regression models will lead to poorly fitting (and unstable) models.** We'll see an implementation of this in the extra linear regression example.

***
# Part 2: Exploratory Data Analysis for Linear Relationships

The [Boston Housing data set](https://archive.ics.uci.edu/ml/datasets/Housing) contains information about the housing values in suburbs of Boston.  This dataset was originally taken from the StatLib library which is maintained at Carnegie Mellon University and is now available on the UCI Machine Learning Repository.

## Load the Boston Housing data set from `sklearn`
***

This data set is available in the [sklearn](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html#sklearn.datasets.load_boston) python module which is how we will access it today.

```python
from sklearn.datasets import load_boston
import pandas as pd

boston = load_boston()
```

```python
boston.keys()
```

    dict_keys(['data', 'target', 'feature_names', 'DESCR', 'filename'])

```python
boston.data.shape
```

    (506, 13)

```python
# Print column names
print(boston.feature_names)
```

    ['CRIM' 'ZN' 'INDUS' 'CHAS' 'NOX' 'RM' 'AGE' 'DIS' 'RAD' 'TAX' 'PTRATIO'
     'B' 'LSTAT']

```python
# Print description of Boston housing data set
print(boston.DESCR)
```

    .. _boston_dataset:

    Boston house prices dataset
    ---------------------------

    **Data Set Characteristics:**

        :Number of Instances: 506

        :Number of Attributes: 13 numeric/categorical predictive. Median Value (attribute 14) is usually the target.

        :Attribute Information (in order):
            - CRIM     per capita crime rate by town
            - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.
            - INDUS    proportion of non-retail business acres per town
            - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
            - NOX      nitric oxides concentration (parts per 10 million)
            - RM       average number of rooms per dwelling
            - AGE      proportion of owner-occupied units built prior to 1940
            - DIS      weighted distances to five Boston employment centres
            - RAD      index of accessibility to radial highways
            - TAX      full-value property-tax rate per $10,000
            - PTRATIO  pupil-teacher ratio by town
            - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town
            - LSTAT    % lower status of the population
            - MEDV     Median value of owner-occupied homes in $1000's

        :Missing Attribute Values: None

        :Creator: Harrison, D. and Rubinfeld, D.L.

    This is a copy of UCI ML housing dataset.
    https://archive.ics.uci.edu/ml/machine-learning-databases/housing/

    This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.

    The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic
    prices and the demand for clean air', J. Environ. Economics & Management,
    vol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics
    ...', Wiley, 1980.   N.B. Various transformations are used in the table on
    pages 244-261 of the latter.

    The Boston house-price data has been used in many machine learning papers that address regression
    problems.

    .. topic:: References

       - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.
       - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.

Now let's explore the data set itself.

```python
bos = pd.DataFrame(boston.data)
bos.head()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33

There are no column names in the DataFrame. Let's add those.

```python
bos.columns = boston.feature_names
bos.head()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33

Now we have a pandas DataFrame called `bos` containing all the data we want to use to predict Boston Housing prices.  Let's create a variable called `PRICE` which will contain the prices. This information is contained in the `target` data.

```python
print(boston.target.shape)
```

    (506,)

```python
bos['PRICE'] = boston.target
bos.head()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      PRICE

      0
      0.00632
      18.0
      2.31
      0.0
      0.538
      6.575
      65.2
      4.0900
      1.0
      296.0
      15.3
      396.90
      4.98
      24.0

      1
      0.02731
      0.0
      7.07
      0.0
      0.469
      6.421
      78.9
      4.9671
      2.0
      242.0
      17.8
      396.90
      9.14
      21.6

      2
      0.02729
      0.0
      7.07
      0.0
      0.469
      7.185
      61.1
      4.9671
      2.0
      242.0
      17.8
      392.83
      4.03
      34.7

      3
      0.03237
      0.0
      2.18
      0.0
      0.458
      6.998
      45.8
      6.0622
      3.0
      222.0
      18.7
      394.63
      2.94
      33.4

      4
      0.06905
      0.0
      2.18
      0.0
      0.458
      7.147
      54.2
      6.0622
      3.0
      222.0
      18.7
      396.90
      5.33
      36.2

## EDA and Summary Statistics
***

Let's explore this data set.  First we use `describe()` to get basic summary statistics for each of the columns.

```python
bos.describe()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT
      PRICE

      count
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000
      506.000000

      mean
      3.613524
      11.363636
      11.136779
      0.069170
      0.554695
      6.284634
      68.574901
      3.795043
      9.549407
      408.237154
      18.455534
      356.674032
      12.653063
      22.532806

      std
      8.601545
      23.322453
      6.860353
      0.253994
      0.115878
      0.702617
      28.148861
      2.105710
      8.707259
      168.537116
      2.164946
      91.294864
      7.141062
      9.197104

      min
      0.006320
      0.000000
      0.460000
      0.000000
      0.385000
      3.561000
      2.900000
      1.129600
      1.000000
      187.000000
      12.600000
      0.320000
      1.730000
      5.000000

      25%
      0.082045
      0.000000
      5.190000
      0.000000
      0.449000
      5.885500
      45.025000
      2.100175
      4.000000
      279.000000
      17.400000
      375.377500
      6.950000
      17.025000

      50%
      0.256510
      0.000000
      9.690000
      0.000000
      0.538000
      6.208500
      77.500000
      3.207450
      5.000000
      330.000000
      19.050000
      391.440000
      11.360000
      21.200000

      75%
      3.677083
      12.500000
      18.100000
      0.000000
      0.624000
      6.623500
      94.075000
      5.188425
      24.000000
      666.000000
      20.200000
      396.225000
      16.955000
      25.000000

      max
      88.976200
      100.000000
      27.740000
      1.000000
      0.871000
      8.780000
      100.000000
      12.126500
      24.000000
      711.000000
      22.000000
      396.900000
      37.970000
      50.000000

### Scatterplots
***

Let's look at some scatter plots for three variables: 'CRIM' (per capita crime rate), 'RM' (number of rooms) and 'PTRATIO' (pupil-to-teacher ratio in schools).

```python
plt.scatter(bos.CRIM, bos.PRICE, s=10)
plt.xlabel("Per capita crime rate by town (CRIM)")
plt.ylabel("Housing Price")
plt.title("Relationship between CRIM and Price")
```

Part 2 Checkup Exercise Set I

Exercise: What kind of relationship do you see? e.g. positive, negative?  linear? non-linear? Is there anything else strange or interesting about the data? What about outliers?

Exercise: Create scatter plots between *RM* and *PRICE*, and *PTRATIO* and *PRICE*. Label your axes appropriately using human readable labels. Tell a story about what you see.

Exercise: What are some other numeric variables of interest? Why do you think they are interesting? Plot scatterplots with these variables and *PRICE* (house price) and tell a story about what you see.

-- your turn: describe relationship

It seems clear that there is a negative relationship between crime rate and price. It might be better fitted with a curve but, in general, it's fairly linear. It is somewhat interesting that houses at the bottom of the pricing scale have a large range of crime ratios.

```python
# your turn: scatter plot between *RM* and *PRICE*
plt.scatter(bos.RM, bos.PRICE, s=12)
plt.xlabel('Number of Rooms')
plt.ylabel('Housing Price')
plt.title('Relationship between number of rooms and price')
plt.show()
```

```python
# your turn: scatter plot between *PTRATIO* and *PRICE*
plt.scatter(bos.PTRATIO, bos.PRICE, s=12)
plt.xlabel('Pupil-Teacher ratio')
plt.ylabel('Housing Price')
plt.title('Relationship between PTRATIO and PRICE')
```

```python
# your turn: create some other scatter plots
plt.scatter(bos.B, bos.PRICE, s=12)
plt.xlabel('Proportion of blacks')
plt.ylabel('Housing Price')

```

```python
plt.scatter(bos.NOX, bos.PRICE, s=12)
plt.xlabel('Nitric Oxide Level')
plt.ylabel('Housing Price')
```

Nitric oxide levels do seem to have an effect on pricing. Or it may be that areas without facilities that produce NOX are naturally more appealing, which correlates to price.

### Scatterplots using Seaborn
***

[Seaborn](https://stanford.edu/~mwaskom/software/seaborn/) is a cool Python plotting library built on top of matplotlib. It provides convenient syntax and shortcuts for many common types of plots, along with better-looking defaults.

We can also use [seaborn regplot](https://stanford.edu/~mwaskom/software/seaborn/tutorial/regression.html#functions-to-draw-linear-regression-models) for the scatterplot above. This provides automatic linear regression fits (useful for data exploration later on). Here's one example below.

```python
_ = sns.regplot(y="PRICE", x="RM", data=bos, fit_reg = True)
```

### Histograms
***

```python
plt.hist(np.log(bos.CRIM))
plt.title("CRIM")
plt.xlabel("Crime rate per capita")
plt.ylabel("Frequencey")
plt.show()
```

Part 2 Checkup Exercise Set II

Exercise: In the above histogram, we took the logarithm of the crime rate per capita. Repeat this histogram without taking the log. What was the purpose of taking the log? What do we gain by making this transformation? What do you now notice about this variable that is not obvious without making the transformation?

Exercise: Plot the histogram for *RM* and *PTRATIO* against each other, along with the two variables you picked in the previous section. We are looking for correlations in predictors here.

```python
#your turn
#histogram without the log
plt.hist(bos.CRIM)
plt.title("CRIM")
plt.xlabel("Crime rate per capita")
plt.ylabel("Frequencey")
plt.show()
```

```python
plt.subplot(2,2,1)
plt.hist(np.log(bos.RM))
plt.xlabel('RM', fontsize=14)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.title('Room Count', fontsize=14)

plt.subplot(2,2,2)
plt.hist(np.log(bos.PTRATIO))
plt.xlabel('PTRATIO', fontsize=14)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.title('Pupil-Teacher Ratio', fontsize=14)

plt.subplot(2,2,3)
plt.hist(bos.B)
plt.xlabel('B', fontsize=14)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.title('Proportion of Blacks', fontsize=14)

plt.subplot(2,2,4)
plt.hist(np.log(bos.NOX))
plt.xlabel('NOX', fontsize=14)
plt.xticks(fontsize=14, rotation=60)
plt.yticks(fontsize=14)
plt.title('Nitric Oxide Level', fontsize=14)

plt.tight_layout()
```

## Part 3: Linear Regression with Boston Housing Data Example
***

Here,

$Y$ = boston housing prices (called "target" data in python, and referred to as the dependent variable or response variable)

and

$X$ = all the other features (or independent variables, predictors or explanatory variables)

which we will use to fit a linear regression model and predict Boston housing prices. We will use the least-squares method to estimate the coefficients.

We'll use two ways of fitting a linear regression. We recommend the first but the second is also powerful in its features.

### Fitting Linear Regression using `statsmodels`
***
[Statsmodels](http://statsmodels.sourceforge.net/) is a great Python library for a lot of basic and inferential statistics. It also provides basic regression functions using an R-like syntax, so it's commonly used by statisticians. While we don't cover statsmodels officially in the Data Science Intensive workshop, it's a good library to have in your toolbox. Here's a quick example of what you could do with it. The version of least-squares we will use in statsmodels is called *ordinary least-squares (OLS)*. There are many other versions of least-squares such as [partial least squares (PLS)](https://en.wikipedia.org/wiki/Partial_least_squares_regression) and [weighted least squares (WLS)](https://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares).

```python
# Import regression modules
import statsmodels.api as sm
from statsmodels.formula.api import ols
```

```python
# statsmodels works nicely with pandas dataframes
# The thing inside the "quotes" is called a formula, a bit on that below
m = ols('PRICE ~ RM',bos).fit()
print(m.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.484
    Model:                            OLS   Adj. R-squared:                  0.483
    Method:                 Least Squares   F-statistic:                     471.8
    Date:                Sat, 08 Jun 2019   Prob (F-statistic):           2.49e-74
    Time:                        11:10:27   Log-Likelihood:                -1673.1
    No. Observations:                 506   AIC:                             3350.
    Df Residuals:                     504   BIC:                             3359.
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept    -34.6706      2.650    -13.084      0.000     -39.877     -29.465
    RM             9.1021      0.419     21.722      0.000       8.279       9.925
    ==============================================================================
    Omnibus:                      102.585   Durbin-Watson:                   0.684
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              612.449
    Skew:                           0.726   Prob(JB):                    1.02e-133
    Kurtosis:                       8.190   Cond. No.                         58.4
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

#### Interpreting coefficients

There is a ton of information in this output. But we'll concentrate on the coefficient table (middle table). We can interpret the `RM` coefficient (9.1021) by first noticing that the p-value (under `P>|t|`) is so small, basically zero. This means that the number of rooms, `RM`, is a statistically significant predictor of `PRICE`. The regression coefficient for `RM` of 9.1021 means that *on average, each additional room is associated with an increase of $\$9,100$ in house price net of the other variables*. The confidence interval gives us a range of plausible values for this average change, about ($\$8,279, \$9,925$), definitely not chump change.

In general, the $\hat{\beta_i}, i > 0$ can be interpreted as the following: "A one unit increase in $x_i$ is associated with, on average, a $\hat{\beta_i}$ increase/decrease in $y$ net of all other variables."

On the other hand, the interpretation for the intercept, $\hat{\beta}_0$ is the average of $y$ given that all of the independent variables $x_i$ are 0.

####  `statsmodels` formulas
***
This formula notation will seem familiar to `R` users, but will take some getting used to for people coming from other languages or are new to statistics.

The formula gives instruction for a general structure for a regression call. For `statsmodels` (`ols` or `logit`) calls you need to have a Pandas dataframe with column names that you will add to your formula. In the below example you need a pandas data frame that includes the columns named (`Outcome`, `X1`,`X2`, ...), but you don't need to build a new dataframe for every regression. Use the same dataframe with all these things in it. The structure is very simple:

`Outcome ~ X1`

But of course we want to to be able to handle more complex models, for example multiple regression is done like this:

`Outcome ~ X1 + X2 + X3`

In general, a formula for an OLS multiple linear regression is

`Y ~ X1 + X2 + ... + Xp`

This is the very basic structure but it should be enough to get you through the homework. Things can get much more complex. You can force statsmodels to treat variables as categorical with the `C()` function, call numpy functions to transform data such as `np.log` for extremely-skewed data, or fit a model without an intercept by including `- 1` in the formula. For a quick run-down of further uses see the `statsmodels` [help page](http://statsmodels.sourceforge.net/devel/example_formulas.html).

Let's see how our model actually fit our data. We can see below that there is a ceiling effect, we should probably look into that. Also, for large values of $Y$ we get underpredictions, most predictions are below the 45-degree gridlines.

Part 3 Checkup Exercise Set I

Exercise: Create a scatterplot between the predicted prices, available in `m.fittedvalues` (where `m` is the fitted model) and the original prices. How does the plot look? Do you notice anything interesting or weird in the plot? Comment on what you see.

```python
plt.scatter(m.fittedvalues, bos.PRICE, s=12)
plt.xlabel('Predicted Prices')
plt.ylabel('Actual Prices')
plt.title('Predicted vs. Actual')
```

For the most part, the original prices and the predicted prices correlate to each other. Most of the observations that are predicted in the $20k range actually concentrate in a similar area on the Y-axis. However, there are a few observations where the actual price is much higher that the predicted price, and there is one very noticeable example of the opposite.

One final curiosity, there looks to be one house with a predicted negative value, which is odd, especially given that the actual price is above the median.

### Fitting Linear Regression using `sklearn`

```python
from sklearn.linear_model import LinearRegression
X = bos.drop('PRICE', axis = 1)

# This creates a LinearRegression object
lm = LinearRegression()
lm
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,
             normalize=False)

#### What can you do with a LinearRegression object?
***
Check out the scikit-learn [docs here](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html). We have listed the main functions here. Most machine learning models in scikit-learn follow this same API of fitting a model with `fit`, making predictions with `predict` and the appropriate scoring function `score` for each model.

Main functions | Description
--- | ---
`lm.fit()` | Fit a linear model
`lm.predit()` | Predict Y using the linear model with estimated coefficients
`lm.score()` | Returns the coefficient of determination (R^2). *A measure of how well observed outcomes are replicated by the model, as the proportion of total variation of outcomes explained by the model*

#### What output can you get?

```python
# Look inside lm object
# lm.<tab>
```

Output | Description
--- | ---
`lm.coef_` | Estimated coefficients
`lm.intercept_` | Estimated intercept

### Fit a linear model
***

The `lm.fit()` function estimates the coefficients the linear regression using least squares.

```python
# Use all 13 predictors to fit linear regression model
lm.fit(X, bos.PRICE)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,
             normalize=False)

Part 3 Checkup Exercise Set II

Exercise: How would you change the model to not fit an intercept term? Would you recommend not having an intercept? Why or why not? For more information on why to include or exclude an intercept, look [here](https://stats.idre.ucla.edu/other/mult-pkg/faq/general/faq-what-is-regression-through-the-origin/).

Exercise: One of the assumptions of the linear model is that the residuals must be i.i.d. (independently and identically distributed). To satisfy this, is it enough that the residuals are normally distributed? Explain your answer.

Exercise: True or false. To use linear regression, $Y$ must be normally distributed. Explain your answer.

\# your turn

If we wanted to not fit an intercept, we could set the fit_intercept argument when calling LinearRegression() to =False. However, if the model predicts that there is a constant or base value of Y given all X variables to be zero, it seems we would want to capture that information. Looking at the housing example, even an empty plot of land has value regardless of the fact that there are zero rooms. It would be useful to be able to predict the value of that plot.

While normally distributed residuals is ideal, it's not the only consideration. For one thing, the sample size might be too low to indicate normality, but doesn't guarantee that the residuals are not i.i.d. As for Y, the dependent variable does not need to be normally distributed. The distribution can be bi-modal, or otherwise, and still 'fit the line.'

### Estimated intercept and coefficients

Let's look at the estimated coefficients from the linear model using `1m.intercept_` and `lm.coef_`.

After we have fit our linear regression model using the least squares method, we want to see what are the estimates of our coefficients $\beta_0$, $\beta_1$, ..., $\beta_{13}$:

$$ \hat{\beta}_0, \hat{\beta}_1, \ldots, \hat{\beta}_{13} $$

```python
print('Estimated intercept coefficient: {}'.format(lm.intercept_))
```

    Estimated intercept coefficient: 36.45948838509015

```python
print('Number of coefficients: {}'.format(len(lm.coef_)))
```

    Number of coefficients: 13

```python
# The coefficients
coef_df = pd.DataFrame({'features': X.columns, 'estimatedCoefficients': lm.coef_})[['features', 'estimatedCoefficients']]
coef_df
```

      features
      estimatedCoefficients

      0
      CRIM
      -0.108011

      1
      ZN
      0.046420

      2
      INDUS
      0.020559

      3
      CHAS
      2.686734

      4
      NOX
      -17.766611

      5
      RM
      3.809865

      6
      AGE
      0.000692

      7
      DIS
      -1.475567

      8
      RAD
      0.306049

      9
      TAX
      -0.012335

      10
      PTRATIO
      -0.952747

      11
      B
      0.009312

      12
      LSTAT
      -0.524758

### Predict Prices

We can calculate the predicted prices ($\hat{Y}_i$) using `lm.predict`.

$$ \hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_1 + \ldots \hat{\beta}_{13} X_{13} $$

```python
# first five predicted prices
lm.predict(X)[0:5]
```

    array([30.00384338, 25.02556238, 30.56759672, 28.60703649, 27.94352423])

Part 3 Checkup Exercise Set III

Exercise: Histogram: Plot a histogram of all the predicted prices. Write a story about what you see. Describe the shape, center and spread of the distribution. Are there any outliers? What might be the reason for them? Should we do anything special with them?

Exercise: Scatterplot: Let's plot the true prices compared to the predicted prices to see if they disagree (we did this with `statsmodels` before).

Exercise: We have looked at fitting a linear model in both `statsmodels` and `scikit-learn`. What are the advantages and disadvantages of each based on your exploration? Based on the information provided by both packages, what advantage does `statsmodels` provide?

```python
# your turn

#histogram of predicted prices
_ = plt.hist(lm.predict(X))
_ = plt.xlabel('Predicted prices')
_ = plt.title('Predicted home values')
```

```python
_ = plt.scatter(lm.predict(X), bos.PRICE, s=12)
_ = plt.xlabel('Predicted Prices')
_ = plt.ylabel('Actual Prices')
```

### Evaluating the Model: Sum-of-Squares

The partitioning of the sum-of-squares shows the variance in the predictions explained by the model and the variance that is attributed to error.

$$TSS = ESS + RSS$$

#### Residual Sum-of-Squares (aka $RSS$)

The residual sum-of-squares is one of the basic ways of quantifying how much error exists in the fitted model. We will revisit this in a bit.

$$ RSS = \sum_{i=1}^N r_i^2 = \sum_{i=1}^N \left(y_i - \left(\beta_0 + \beta_1 x_i\right)\right)^2 $$

```python
print(np.sum((bos.PRICE - lm.predict(X)) ** 2))
```

    11078.784577954977

#### Explained Sum-of-Squares (aka $ESS$)

The explained sum-of-squares measures the variance explained by the regression model.

$$ESS = \sum_{i=1}^N \left( \hat{y}_i - \bar{y} \right)^2 = \sum_{i=1}^N \left( \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) - \bar{y} \right)^2$$

```python
print(np.sum((lm.predict(X) - np.mean(bos.PRICE)) ** 2))
```

    31637.510837065056

### Evaluating the Model: The Coefficient of Determination ($R^2$)

The coefficient of determination, $R^2$, tells us the percentage of the variance in the response variable $Y$ that can be explained by the linear regression model.

$$ R^2 = \frac{ESS}{TSS} $$

The $R^2$ value is one of the most common metrics that people use in describing the quality of a model, but it is important to note that *$R^2$ increases artificially as a side-effect of increasing the number of independent variables.* While $R^2$ is reported in almost all statistical packages, another metric called the *adjusted $R^2$* is also provided as it takes into account the number of variables in the model, and can sometimes even be used for non-linear regression models!

$$R_{adj}^2 = 1 - \left( 1 - R^2 \right) \frac{N - 1}{N - K - 1} = R^2 - \left( 1 - R^2 \right) \frac{K}{N - K - 1} = 1 - \frac{\frac{RSS}{DF_R}}{\frac{TSS}{DF_T}}$$

where $N$ is the number of observations, $K$ is the number of variables, $DF_R = N - K - 1$ is the degrees of freedom associated with the residual error and $DF_T = N - 1$ is the degrees of the freedom of the total error.

### Evaluating the Model: Mean Squared Error and the $F$-Statistic
***
The mean squared errors are just the *averages* of the sum-of-squares errors over their respective degrees of freedom.

$$MSE = \frac{RSS}{N-K-1}$$

$$MSR = \frac{ESS}{K}$$

**Remember:** Notation may vary across resources particularly the use of $R$ and $E$ in $RSS/ESS$ and $MSR/MSE$. In some resources, E = explained and R = residual. In other resources, E = error and R = regression (explained). **This is a very important distinction that requires looking at the formula to determine which naming scheme is being used.**

Given the MSR and MSE, we can now determine whether or not the entire model we just fit is even statistically significant. We use an $F$-test for this. The null hypothesis is that all of the $\beta$ coefficients are zero, that is, none of them have any effect on $Y$. The alternative is that *at least one* $\beta$ coefficient is nonzero, but it doesn't tell us which one in a multiple regression:

$$H_0: \beta_i = 0, \mbox{for all $i$} \\
H_A: \beta_i > 0, \mbox{for some $i$}$$

$$F = \frac{MSR}{MSE} = \left( \frac{R^2}{1 - R^2} \right) \left( \frac{N - K - 1}{K} \right)$$

Once we compute the $F$-statistic, we can use the $F$-distribution with $N-K$ and $K-1$ degrees of degrees of freedom to get a p-value.

**Warning!** The $F$-statistic mentioned in this section is NOT the same as the F1-measure or F1-value discused in Unit 7.

Part 3 Checkup Exercise Set IV

Let's look at the relationship between `PTRATIO` and housing price.

Exercise: Try fitting a linear regression model using only the 'PTRATIO' (pupil-teacher ratio by town) and interpret the intercept and the coefficients.

Exercise: Calculate (or extract) the $R^2$ value. What does it tell you?

Exercise: Compute the $F$-statistic. What does it tell you?

Exercise: Take a close look at the $F$-statistic and the $t$-statistic for the regression coefficient. What relationship do you notice? Note that this relationship only applies in *simple* linear regression models.

```python
# your turn

#reshape the PTRATIO data and instatiate the regression model
X = np.array(bos['PTRATIO']).reshape(-1,1)
pt_reg = LinearRegression()

#fit and predict the model, extracting the first 5 observations
pt_reg.fit(X, bos.PRICE)
pt_reg.predict(X)[:5]

```

    array([29.33984545, 23.9469072 , 23.9469072 , 22.00544944, 22.00544944])

```python
print('Intercept is {}'.format(pt_reg.intercept_))
```

    Intercept is 62.34462747483266

```python
print('Estimated coefficient is {}'.format(pt_reg.coef_))
```

    Estimated coefficient is [-2.1571753]

```python
#The R2 value is fairly low, indicating a weak predictor
print('The R2 value is {}'.format(pt_reg.score(X, bos.PRICE)))
```

    The R2 value is 0.257847318009223

```python
#use the summary to find the F-statistic
f = ols('PRICE ~ PTRATIO',bos).fit()
print(f.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  PRICE   R-squared:                       0.258
    Model:                            OLS   Adj. R-squared:                  0.256
    Method:                 Least Squares   F-statistic:                     175.1
    Date:                Tue, 07 May 2019   Prob (F-statistic):           1.61e-34
    Time:                        19:39:12   Log-Likelihood:                -1764.8
    No. Observations:                 506   AIC:                             3534.
    Df Residuals:                     504   BIC:                             3542.
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept     62.3446      3.029     20.581      0.000      56.393      68.296
    PTRATIO       -2.1572      0.163    -13.233      0.000      -2.477      -1.837
    ==============================================================================
    Omnibus:                       92.924   Durbin-Watson:                   0.725
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              191.444
    Skew:                           1.001   Prob(JB):                     2.68e-42
    Kurtosis:                       5.252   Cond. No.                         160.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Based on this summary and the previous summary from the 'Rooms' model, the t-statistic and the F-statistic seem to be correlated.

Part 3 Checkup Exercise Set V

Fit a linear regression model using three independent variables

 'CRIM' (per capita crime rate by town)

 'RM' (average number of rooms per dwelling)

 'PTRATIO' (pupil-teacher ratio by town)

Exercise: Compute or extract the $F$-statistic. What does it tell you about the model?

Exercise: Compute or extract the $R^2$ statistic. What does it tell you about the model?

Exercise: Which variables in the model are significant in predicting house price? Write a story that interprets the coefficients.

```python
# your turn
from sklearn.metrics import r2_score

X = bos[['CRIM', 'RM', 'PTRATIO']]
reg_3 = LinearRegression()

#Fit the three variables against price
reg_3.fit(X, bos.PRICE)
r2 = r2_score(bos.PRICE, reg_3.predict(X))

r2
```

    0.5943412940723471

```python
X = bos[['CRIM', 'RM', 'PTRATIO']]
f_3 = ols('PRICE ~ X', bos).fit()
f_3.summary()
```

OLS Regression Results

  Dep. Variable:	          PRICE	        R-squared:         	    0.594

  Model:	                   OLS	         Adj. R-squared:    	    0.592

  Method:	             Least Squares	    F-statistic:       	    245.2

  Date:	             Sat, 08 Jun 2019	   Prob (F-statistic):	 6.15e-98

  Time:	                 11:43:17	       Log-Likelihood:    	  -1612.0

  No. Observations:	         506	        AIC:               	    3232.

  Df Residuals:	             502	        BIC:               	    3249.

  Df Model:	                   3

  Covariance Type:	      nonrobust

      	         coef	     std err	      t	      P>|t|	  [0.025	    0.975]

  Intercept	    -3.3707	     4.034	    -0.836	  0.404	   -11.296	     4.555

  X[0]	         -0.2050	     0.032	    -6.399	  0.000	    -0.268	    -0.142

  X[1]	          7.3804	     0.402	    18.382	  0.000	     6.592	     8.169

  X[2]	         -1.0695	     0.133	    -8.051	  0.000	    -1.331	    -0.809

  Omnibus:	       234.656	   Durbin-Watson:     	    0.830

  Prob(Omnibus):	  0.000	    Jarque-Bera (JB):  	 2020.689

  Skew:	           1.815	    Prob(JB):          	     0.00

  Kurtosis:	      12.092	    Cond. No.          	     311.

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

The model performs better with multiple predictor variables.

```python
#Re-view the coefficients table
coef_df
```

      features
      estimatedCoefficients

      0
      CRIM
      -0.108011

      1
      ZN
      0.046420

      2
      INDUS
      0.020559

      3
      CHAS
      2.686734

      4
      NOX
      -17.766611

      5
      RM
      3.809865

      6
      AGE
      0.000692

      7
      DIS
      -1.475567

      8
      RAD
      0.306049

      9
      TAX
      -0.012335

      10
      PTRATIO
      -0.952747

      11
      B
      0.009312

      12
      LSTAT
      -0.524758

The strongest coefficients appear to be RM and NOX. Even though NOX is negative, that information is still useful.

## Part 4: Comparing Models

During modeling, there will be times when we want to compare models to see which one is more predictive or fits the data better. There are many ways to compare models, but we will focus on two.

### The $F$-Statistic Revisited

The $F$-statistic can also be used to compare two *nested* models, that is, two models trained on the same dataset where one of the models contains a *subset* of the variables of the other model. The *full* model contains $K$ variables and the *reduced* model contains a subset of these $K$ variables. This allows us to add additional variables to a base model and then test if adding the variables helped the model fit.

$$F = \frac{\left( \frac{RSS_{reduced} - RSS_{full}}{DF_{reduced} - DF_{full}} \right)}{\left( \frac{RSS_{full}}{DF_{full}} \right)}$$

where $DF_x = N - K_x - 1$ where $K_x$ is the number of variables in model $x$.

### Akaike Information Criterion (AIC)

Another statistic for comparing two models is AIC, which is based on the likelihood function and takes into account the number of variables in the model.

$$AIC = 2 K - 2 \log_e{L}$$

where $L$ is the likelihood of the model. AIC is meaningless in the absolute sense, and is only meaningful when compared to AIC values from other models. Lower values of AIC indicate better fitting models.

`statsmodels` provides the AIC in its output.

Part 4 Checkup Exercises

Exercise: Find another variable (or two) to add to the model we built in Part 3. Compute the $F$-test comparing the two models as well as the AIC. Which model is better?

```python
X = bos[['CRIM', 'RM', 'PTRATIO', 'NOX', 'CHAS']]
f_5 = ols('PRICE ~ X', bos).fit()
f_5.summary()
```

OLS Regression Results

  Dep. Variable:	          PRICE	        R-squared:         	    0.627

  Model:	                   OLS	         Adj. R-squared:    	    0.623

  Method:	             Least Squares	    F-statistic:       	    168.1

  Date:	             Sat, 08 Jun 2019	   Prob (F-statistic):	 1.24e-104

  Time:	                 13:01:31	       Log-Likelihood:    	  -1590.7

  No. Observations:	         506	        AIC:               	    3193.

  Df Residuals:	             500	        BIC:               	    3219.

  Df Model:	                   5

  Covariance Type:	      nonrobust

      	         coef	     std err	      t	      P>|t|	  [0.025	    0.975]

  Intercept	     7.1186	     4.306	     1.653	  0.099	    -1.341	    15.579

  X[0]	         -0.1298	     0.033	    -3.906	  0.000	    -0.195	    -0.065

  X[1]	          6.7755	     0.397	    17.055	  0.000	     5.995	     7.556

  X[2]	         -1.0218	     0.128	    -7.973	  0.000	    -1.274	    -0.770

  X[3]	        -14.6251	     2.489	    -5.876	  0.000	   -19.515	    -9.735

  X[4]	          3.9403	     1.009	     3.903	  0.000	     1.957	     5.924

  Omnibus:	       267.737	   Durbin-Watson:     	    0.889

  Prob(Omnibus):	  0.000	    Jarque-Bera (JB):  	 2583.103

  Skew:	           2.107	    Prob(JB):          	     0.00

  Kurtosis:	      13.235	    Cond. No.          	     360.

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

With more variables the F-statistic is lower, but R-squared and AIC indicate a better fit.

## Part 5: Evaluating the Model via Model Assumptions and Other Issues
***
Linear regression makes several assumptions. It is always best to check that these assumptions are valid after fitting a linear regression model.

**Linearity**. The dependent variable $Y$ is a linear combination of the regression coefficients and the independent variables $X$. This can be verified with a scatterplot of each $X$ vs. $Y$ and plotting correlations among $X$. Nonlinearity can sometimes be resolved by [transforming](https://onlinecourses.science.psu.edu/stat501/node/318) one or more independent variables, the dependent variable, or both. In other cases, a [generalized linear model](https://en.wikipedia.org/wiki/Generalized_linear_model) or a [nonlinear model](https://en.wikipedia.org/wiki/Nonlinear_regression) may be warranted.

**Constant standard deviation**. The SD of the dependent variable $Y$ should be constant for different values of X. We can check this by plotting each $X$ against $Y$ and verifying that there is no "funnel" shape showing data points fanning out as $X$ increases or decreases. Some techniques for dealing with non-constant variance include weighted least squares (WLS), [robust standard errors](https://en.wikipedia.org/wiki/Heteroscedasticity-consistent_standard_errors), or variance stabilizing transformations.

 **Normal distribution for errors**.  The $\epsilon$ term we discussed at the beginning are assumed to be normally distributed. This can be verified with a fitted values vs. residuals plot and verifying that there is no pattern, and with a quantile plot.
  $$ \epsilon_i \sim N(0, \sigma^2)$$
Sometimes the distributions of responses $Y$ may not be normally distributed at any given value of $X$.  e.g. skewed positively or negatively.

 **Independent errors**.  The observations are assumed to be obtained independently.

e.g. Observations across time may be correlated

There are some other issues that are important investigate with linear regression models.

**Correlated Predictors:** Care should be taken to make sure that the independent variables in a regression model are not too highly correlated. Correlated predictors typically do not majorly affect prediction, but do inflate standard errors of coefficients making interpretation unreliable. Common solutions are dropping the least important variables involved in the correlations, using regularlization, or, when many predictors are highly correlated, considering a dimension reduction technique such as principal component analysis (PCA).

**Influential Points:** Data points that have undue influence on the regression model. These points can be high leverage points or outliers. Such points are typically removed and the regression model rerun.

Part 5 Checkup Exercises

Take the reduced model from Part 3 to answer the following exercises. Take a look at [this blog post](http://mpastell.com/2013/04/19/python_regression/) for more information on using statsmodels to construct these plots.

Exercise: Construct a fitted values versus residuals plot. What does the plot tell you? Are there any violations of the model assumptions?

Exercise: Construct a quantile plot of the residuals. What does the plot tell you?

Exercise: What are some advantages and disadvantages of the fitted vs. residual and quantile plot compared to each other?

Exercise: Identify any outliers (if any) in your model and write a story describing what these outliers might represent.

Exercise: Construct a leverage plot and identify high leverage points in the model. Write a story explaining possible reasons for the high leverage points.

Exercise: Remove the outliers and high leverage points from your model and run the regression again. How do the results change?

```python
# Your turn.
X_3 = bos[['CRIM', 'RM', 'PTRATIO']]
f_3 = ols('PRICE ~ X_3', bos).fit()
f_3.summary()
```

OLS Regression Results

  Dep. Variable:	          PRICE	        R-squared:         	    0.594

  Model:	                   OLS	         Adj. R-squared:    	    0.592

  Method:	             Least Squares	    F-statistic:       	    245.2

  Date:	             Sat, 08 Jun 2019	   Prob (F-statistic):	 6.15e-98

  Time:	                 13:48:14	       Log-Likelihood:    	  -1612.0

  No. Observations:	         506	        AIC:               	    3232.

  Df Residuals:	             502	        BIC:               	    3249.

  Df Model:	                   3

  Covariance Type:	      nonrobust

      	         coef	     std err	      t	      P>|t|	  [0.025	    0.975]

  Intercept	    -3.3707	     4.034	    -0.836	  0.404	   -11.296	     4.555

  X_3[0]	       -0.2050	     0.032	    -6.399	  0.000	    -0.268	    -0.142

  X_3[1]	        7.3804	     0.402	    18.382	  0.000	     6.592	     8.169

  X_3[2]	       -1.0695	     0.133	    -8.051	  0.000	    -1.331	    -0.809

  Omnibus:	       234.656	   Durbin-Watson:     	    0.830

  Prob(Omnibus):	  0.000	    Jarque-Bera (JB):  	 2020.689

  Skew:	           1.815	    Prob(JB):          	     0.00

  Kurtosis:	      12.092	    Cond. No.          	     311.

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

```python
_ = plt.hist(f_3.resid)
_ = plt.title('Residual Histogram', fontsize=16)
```

```python
import scipy.stats as stats

stats.probplot(f_3.resid, plot=plt)
plt.show()
```

```python
from statsmodels.graphics.regressionplots import *

plot_leverage_resid2(f_3)
```

```python
bos_adj = bos.drop(bos.index[[380,418, 405, 367, 373, 365, 373, 369, 371, 372, 368]])

res_adj = ols('PRICE ~ CRIM + RM + PTRATIO', bos_adj).fit()
res_adj.summary()
```

OLS Regression Results

  Dep. Variable:	          PRICE	        R-squared:         	    0.707

  Model:	                   OLS	         Adj. R-squared:    	    0.705

  Method:	             Least Squares	    F-statistic:       	    395.6

  Date:	             Sat, 08 Jun 2019	   Prob (F-statistic):	 1.13e-130

  Time:	                 17:13:47	       Log-Likelihood:    	  -1481.5

  No. Observations:	         496	        AIC:               	    2971.

  Df Residuals:	             492	        BIC:               	    2988.

  Df Model:	                   3

  Covariance Type:	      nonrobust

      	         coef	     std err	      t	      P>|t|	  [0.025	    0.975]

  Intercept	    -7.5249	     3.375	    -2.230	  0.026	   -14.156	    -0.894

  CRIM	         -0.2524	     0.036	    -6.994	  0.000	    -0.323	    -0.182

  RM	            8.0731	     0.345	    23.405	  0.000	     7.395	     8.751

  PTRATIO	      -1.0942	     0.110	    -9.940	  0.000	    -1.310	    -0.878

  Omnibus:	       40.219	   Durbin-Watson:     	    1.064

  Prob(Omnibus):	  0.000	   Jarque-Bera (JB):  	  112.969

  Skew:	           0.363	   Prob(JB):          	 2.94e-25

  Kurtosis:	       5.222	   Cond. No.          	     312.

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Removing the leverage outliers helped improve R-squared and AIC


# Plotting with Matplotlib

## Prepare for action

```
import numpy as np
import scipy as sp
import sympy
import pylab as plb

# Pylab combines the pyplot functionality (for plotting) with the numpy
# functionality (for mathematics and for working with arrays) in a single namespace
# aims to provide a closer MATLAB feel (the easy way). Note that his approach
# should only be used when doing some interactive quick and dirty data inspection.
# DO NOT USE THIS FOR SCRIPTS
#from pylab import *

# the convienient Matplotib plotting interface pyplot (the tidy/right way)
# use this for building scripts. The examples here will all use pyplot.
import matplotlib.pyplot as plt

# for using the matplotlib API directly (the hard and verbose way)
# use this when building applications, and/or backends
import matplotlib as mpl
```

How would you like the IPython notebook show your plots? In order to use the
matplotlib IPython magic youre IPython notebook should be launched as

    ipython notebook --matplotlib=inline

Make plots appear as a pop up window, chose the backend: 'gtk', 'inline', 'osx', 'qt', 'qt4', 'tk', 'wx'

    %matplotlib qt

or inline the notebook (no panning, zooming through the plot). Not working in IPython 0.x

    %matplotib inline

```
# activate pop up plots
#%matplotlib qt
# or change to inline plots
%matplotlib inline
```

### Matplotlib documentation

Finding your own way (aka RTFM). Hint: there is search box available!

* http://matplotlib.org/contents.html

The Matplotlib API docs:

* http://matplotlib.org/api/index.html

Pyplot, object oriented plotting:

* http://matplotlib.org/api/pyplot_api.html
* http://matplotlib.org/api/pyplot_summary.html

Extensive gallery with examples:

* http://matplotlib.org/gallery.html

### Tutorials for those who want to start playing

If reading manuals is too much for you, there is a very good tutorial available here:

* http://nbviewer.ipython.org/github/jrjohansson/scientific-python-lectures/blob/master/Lecture-4-Matplotlib.ipynb

Note that this tutorial uses

    from pylab import *

which is usually not adviced in more advanced script environments. When using

    import matplotlib.pyplot as plt

you need to preceed all plotting commands as used in the above tutorial with

    plt.

Give me more!

[EuroScipy 2012 Matlotlib tutorial](http://www.loria.fr/~rougier/teaching/matplotlib/). Note that here the author uses ```from pylab import * ```. When using ```import matplotliblib.pyplot as plt``` the plotting commands need to be proceeded with ```plt.```

## Plotting template starting point

```
# some sample data
x = np.arange(-10,10,0.1)
```

To change the default plot configuration values.

```
page_width_cm = 13
dpi = 200
inch = 2.54 # inch in cm
# setting global plot configuration using the RC configuration style
plt.rc('font', family='serif')
plt.rc('xtick', labelsize=12) # tick labels
plt.rc('ytick', labelsize=12) # tick labels
plt.rc('axes', labelsize=20)  # axes labels
# If you don’t need LaTeX, don’t use it. It is slower to plot, and text
# looks just fine without. If you need it, e.g. for symbols, then use it.
#plt.rc('text', usetex=True) #<- P-E: Doesn't work on my Mac
```

```
# create a figure instance, note that figure size is given in inches!
fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(8,6))
# set the big title (note aligment relative to figure)
fig.suptitle("suptitle 16, figure alignment", fontsize=16)

# actual plotting
ax.plot(x, x**2, label="label 12")

# set axes title (note aligment relative to axes)
ax.set_title("title 14, axes alignment", fontsize=14)

# axes labels
ax.set_xlabel('xlabel 12')
ax.set_ylabel(r'$y_{\alpha}$ 12', fontsize=8)

# legend
ax.legend(fontsize=12, loc="best")

# saving the figure in different formats
fig.savefig('figure-%03i.png' % dpi, dpi=dpi)
fig.savefig('figure.svg')
fig.savefig('figure.eps')
```

```
# following steps are only relevant when using figures as pop up windows (with %matplotlib qt)
# to update a figure with has been modified
fig.canvas.draw()
# show a figure
fig.show()
```

    C:\Anaconda\lib\site-packages\matplotlib\figure.py:371: UserWarning: matplotlib is currently using a non-GUI backend, so cannot show the figure
      "matplotlib is currently using a non-GUI backend, "

## Exercise

The current section is about you trying to figure out how to do several plotting features. You should use the previously mentioned resources to find how to do that. In many cases, google is your friend!

* add a grid to the plot

```
plt.plot(x,x**2)
#Write code to show grid in plot here
axx = plt.gca();
axx.grid(b=True,which='major',color='r',linestyle='-',linewidth=0.25)
```

* change the location of the legend to different places

```
plt.plot(x,x**2, label="label 12")
plt.legend(fontsize=12, loc="best")
plt.legend(fontsize=12, loc=9)
```

* find a way to control the line type and color, marker type and color, control the frequency of the marks (`markevery`). See plot options at: http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.plot

```
#plt.plot(x,x**2, 'o-')
plt.plot(x,x**2,'o-',color='r',linestyle='-',linewidth=0.75,markersize=8,markevery=10)
```

* add different sub-plots

```
fig, ax = plt.subplots(nrows=3, ncols=1, sharex=True)
ax1=ax[0].plot(x,x**2)
fontsizze=9;
ax[1].plot(x,x**7)
ax[2].plot(x,x**4)
for i in range(0, 2):
    ax[i]
    axx = plt.gca();
    for tick in axx.yaxis.get_major_ticks():
        tick.label.set_fontsize(fontsizze)

#for (tx, ty) in zip(axx.xaxis.get_major_ticks(), axx.yaxis.get_major_ticks()):
#    tx.label.set_fontsize(4)
#    ty.label.set_fontsize(14)
figg = plt.figure(figsize = (5.5, 4.3))
#figg.subplots_adjust(hspace = 0.005)
sal_ax = plt.subplot(3,1,1)
sal_ax.plot(x,x**2)
sal_ax = plt.subplot(3,1,2)
sal_ax.plot(x,x**7)
sal_ax = plt.subplot(3,1,3)
sal_ax.plot(x,x**4)

for i in range(2, 4):
    plt.subplot(3,1,i)
    axx = plt.gca();
    for tick in axx.yaxis.get_major_ticks():
        tick.label1.set_fontsize(fontsizze)
```

* size the figure such that when included on an A4 page the fonts are given in their true size

```

fig, ax = plt.subplots(nrows=2, ncols=1, sharex=True)
ax[0].plot(x,x**2)
ax[1].plot(x,x**7)
f = plt.gcf()
f.set_size_inches(8.27,11.69)
```

* make a contour plot

```
X, Y = np.meshgrid(x,x)
Z = X**2+Y
plt.contourf(X,Y,Z)
plt.colorbar()
```

* use twinx() to create a second axis on the right for the second plot

```
ax =plt.plot(x,x**2)
axx=plt.plot(x,x**4, 'r')
plt.twinx()
```

* add horizontal and vertical lines using axvline(), axhline()

```
plt.plot(x,x**2,'r')
plt.axvline(5)
plt.axhline(75)
```

* autoformat dates for nice printing on the x-axis using fig.autofmt_xdate()

```
import datetime
dates = np.array([datetime.datetime.now() + datetime.timedelta(days=i) for i in xrange(24)])
fig, ax = plt.subplots(nrows=1, ncols=1)
#for i in range(1,24):
#    print dates[i]
ax.plot(dates,xrange(24),'o')
plt.gcf().autofmt_xdate()
```

## Advanced exercises

We are going to play a bit with regression

* Create a vector x of equally spaced number between $x \in [0, 5\pi]$ of 1000 points (keyword: linspace)

```
xx=np.array(np.linspace(0,5*np.pi,1000))
```

* create a vector y, so that y=sin(x) with some random noise

```
#print xx.shape()
print len(xx)
y=np.sin(xx)+np.random.uniform(-1, 1, size=len(xx))
```

    1000

* plot it like this:

```
plt.plot(xx,y,'bo',markersize=8,markevery=3)
plt.plot(xx,np.sin(xx),'k-',linewidth=4,label="y=sin(x)")
plt.legend(fontsize=12,loc='best')
```

Try to do a polynomial fit on y(x) with different polynomial degree (Use numpy.polyfit to obtain coefficients)

Plot it like this (use np.poly1d(coef)(x) to plot polynomials)

```
#a= [i for i in range (0,10)];
#b= [i for i in range (0,len(xx))];
z=np.zeros((10,10));
z1=np.zeros((len(xx),10));
for i in range(0,10):
    #print i
    z[0:i+1,i]=np.polyfit(xx, y, i)

```

```
fig = plt.figure(1)
ax = fig.add_subplot(111)
ax.plot(xx,y,'bo',markersize=8,markevery=2)
ax.plot(xx,np.sin(xx),'k-',linewidth=5,label="y=sin(x)")
for i in range (0,10):
    ax.plot(xx,np.poly1d(z[0:i+1,i])(xx),linewidth=2,label=('deg='+str(i)))

handles, labels = ax.get_legend_handles_labels()
lgd = ax.legend(handles, labels, fontsize=12,loc='upper left', bbox_to_anchor=(1.02, 1), borderaxespad=0)
#ax.legend(fontsize=11,loc='upper right')

```

```

```

```

```


# The VDDG model forced by the Jin-Timmermann ENSO model

## Preamble

This notebook comes as a supplementary material to the article

* Vannitsem, S., Demaeyer, J., and Ghil, M. (2021). Extratropical low‐frequency variability with ENSO forcing: A reduced‐order coupled model study. *Journal of Advances in Modeling Earth Systems*, e2021MS002530. [https://doi.org/10.1029/2021MS002530](https://doi.org/10.1029/2021MS002530).

and details the code of the model used to perform the computations needed to obtain the results therein.

It consists of the [qgs model](https://github.com/Climdyn/qgs) in the MAOOAM-VDDG configuration to model the extra-tropics, and forced by the Jin-Timmerman model which models ENSO. More information on the components can be found in the articles:

* Vannitsem, S., Demaeyer, J., De Cruz, L., & Ghil, M. (2015). Low-frequency variability and heat transport in a low-order nonlinear coupled ocean–atmosphere model. *Physica D: Nonlinear Phenomena*, **309**, 71-85. [doi:10.1016/j.physd.2015.07.006](https://doi.org/10.1016/j.physd.2015.07.006)
* De Cruz, L., Demaeyer, J. and Vannitsem, S. (2016). The Modular Arbitrary-Order Ocean-Atmosphere Model: MAOOAM v1.0, *Geosci. Model Dev.*, **9**, 2793-2808. [doi:10.5194/gmd-9-2793-2016](https://doi.org/10.5194/gmd-9-2793-2016)
* Demaeyer J., De Cruz, L. and Vannitsem, S. , (2020). qgs: A flexible Python framework of reduced-order multiscale climate models. *Journal of Open Source Software*, **5**(56), 2597, [doi:10.21105/joss.02597](https://doi.org/10.21105/joss.02597).
* Timmermann, A., Jin, F. F., & Abshagen, J. (2003). A nonlinear theory for El Niño bursting. *Journal of the atmospheric sciences*, **60**(1), 152-165, [doi:10.1175/1520-0469(2003)060%3C0152:ANTFEN%3E2.0.CO;2](https://doi.org/10.1175/1520-0469(2003)060%3C0152:ANTFEN%3E2.0.CO;2).
* Roberts, A., Guckenheimer, J., Widiasih, E., Timmermann, A., & Jones, C. K. (2016). Mixed-mode oscillations of El Nino–southern oscillation. *Journal of Atmospheric Sciences*, **73**(4), 1755-1766, [doi:10.1175/JAS-D-15-0191.1](https://doi.org/10.1175/JAS-D-15-0191.1).

In this notebook, we define the forced model and integrate it to get several trajectories.

## Requirements

The present notebook was tested on a computer with

    12 recent cpu cores,
    16 Gb of RAM,

## Installation

To install this notebook, you must first [install the qgs model](https://qgs.readthedocs.io/en/latest/files/general_information.html#installation) and test that it works properly on your machine. Follow the instructions in the model documentation. Then you must move the present notebook in the notebooks directory and run it with jupyter-notebook. In the qgs model main folder, do:

    conda activate qgs
    cd notebooks
    jupyter-notebook

It will open a page in a web browser where you can open and then run the present notebook.

We now proceed to the code itself:

## Modules import

First, setting of the path and loading of some modules

```python
import sys, os
```

```python
sys.path.extend([os.path.abspath('../')])
```

```python
import numpy as np
from numba import njit
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

Initializing the random number generator (for reproducibility). -- Disable if needed.

```python
np.random.seed(210217)
```

Importing the model's modules

```python
from qgs.params.params import QgParams
from qgs.integrators.integrator import RungeKuttaIntegrator
from qgs.functions.tendencies import create_tendencies
```

## Systems definition

General parameters

```python
# Time step parameters
dt = 0.1
# Saving the model state n steps
write_steps = 100

number_of_trajectories = 2
```

### VDDG (using qgs)

Setting some model parameters

```python
# Model parameters instantiation
model_parameters = QgParams()

## VDDG model selection:
# Mode truncation at the wavenumber 2 in both x and y spatial
# coordinates for the atmosphere
model_parameters.set_atmospheric_channel_fourier_modes(2, 2)
# Mode truncation at the wavenumber 2 in the x and at the
# wavenumber 4 in the y spatial coordinates for the ocean
model_parameters.set_oceanic_basin_fourier_modes(2, 4)
```

```python
# Setting MAOOAM parameters according to the publication
model_parameters.set_params({'kd': 0.0290, 'kdp': 0.0290, 'n': 1.5, 'r': 1.e-7,
                             'h': 100.0, 'd': 1.5e-7, 'time_unit': 'years'})
model_parameters.atemperature_params.set_params({'hlambda': 15.06})
model_parameters.gotemperature_params.set_params({'gamma': 4.0e8})
```

Setting the short-wave radiation component $C_{\text{a},1}$ and $C_{\text{o},1}$ as in the publication:

```python
model_parameters.atemperature_params.set_insolation(310./4, 0)
model_parameters.gotemperature_params.set_insolation(310, 0)
```

Printing the VDDG model's parameters

```python
model_parameters.print_params()
```

    Qgs v0.2.5 parameters summary
    =============================

    General Parameters:
    'time_unit': years,
    'rr': 287.058  [J][kg^-1][K^-1]  (gas constant of dry air),
    'sb': 5.67e-08  [J][m^-2][s^-1][K^-4]  (Stefan-Boltzmann constant),

    Scale Parameters:
    'scale': 5000000.0  [m]  (characteristic space scale (L*pi)),
    'f0': 0.0001032  [s^-1]  (Coriolis parameter at the middle of the domain),
    'n': 1.5    (aspect ratio (n = 2 L_y / L_x)),
    'rra': 6370000.0  [m]  (earth radius),
    'phi0_npi': 0.25    (latitude expressed in fraction of pi),
    'deltap': 50000.0  [Pa]  (pressure difference between the two atmospheric layers),

    Atmospheric Parameters:
    'kd': 0.029  [nondim]  (atmosphere bottom friction coefficient),
    'kdp': 0.029  [nondim]  (atmosphere internal friction coefficient),
    'sigma': 0.2  [nondim]  (static stability of the atmosphere),

    Atmospheric Temperature Parameters:
    'gamma': 10000000.0  [J][m^-2][K^-1]  (specific heat capacity of the atmosphere),
    'C[1]': 77.5  [W][m^-2]  (spectral component 1 of the short-wave radiation of the atmosphere),
    'C[2]': 0.0  [W][m^-2]  (spectral component 2 of the short-wave radiation of the atmosphere),
    'C[3]': 0.0  [W][m^-2]  (spectral component 3 of the short-wave radiation of the atmosphere),
    'C[4]': 0.0  [W][m^-2]  (spectral component 4 of the short-wave radiation of the atmosphere),
    'C[5]': 0.0  [W][m^-2]  (spectral component 5 of the short-wave radiation of the atmosphere),
    'C[6]': 0.0  [W][m^-2]  (spectral component 6 of the short-wave radiation of the atmosphere),
    'C[7]': 0.0  [W][m^-2]  (spectral component 7 of the short-wave radiation of the atmosphere),
    'C[8]': 0.0  [W][m^-2]  (spectral component 8 of the short-wave radiation of the atmosphere),
    'C[9]': 0.0  [W][m^-2]  (spectral component 9 of the short-wave radiation of the atmosphere),
    'C[10]': 0.0  [W][m^-2]  (spectral component 10 of the short-wave radiation of the atmosphere),
    'eps': 0.76    (emissivity coefficient for the grey-body atmosphere),
    'T0': 270.0  [K]  (stationary solution for the 0-th order atmospheric temperature),
    'sc': 1.0    (ratio of surface to atmosphere temperature),
    'hlambda': 15.06  [W][m^-2][K^-1]  (sensible+turbulent heat exchange between ocean and atmosphere),

    Oceanic Parameters:
    'gp': 0.031  [m][s^-2]  (reduced gravity),
    'r': 1e-07  [s^-1]  (frictional coefficient at the bottom of the ocean),
    'h': 100.0  [m]  (depth of the water layer of the ocean),
    'd': 1.5e-07  [s^-1]  (strength of the ocean-atmosphere mechanical coupling),

    Oceanic Temperature Parameters:
    'gamma': 400000000.0  [J][m^-2][K^-1]  (specific heat capacity of the ocean),
    'C[1]': 310.0  [W][m^-2]  (spectral component 1 of the short-wave radiation of the ocean),
    'C[2]': 0.0  [W][m^-2]  (spectral component 2 of the short-wave radiation of the ocean),
    'C[3]': 0.0  [W][m^-2]  (spectral component 3 of the short-wave radiation of the ocean),
    'C[4]': 0.0  [W][m^-2]  (spectral component 4 of the short-wave radiation of the ocean),
    'C[5]': 0.0  [W][m^-2]  (spectral component 5 of the short-wave radiation of the ocean),
    'C[6]': 0.0  [W][m^-2]  (spectral component 6 of the short-wave radiation of the ocean),
    'C[7]': 0.0  [W][m^-2]  (spectral component 7 of the short-wave radiation of the ocean),
    'C[8]': 0.0  [W][m^-2]  (spectral component 8 of the short-wave radiation of the ocean),
    'C[9]': 0.0  [W][m^-2]  (spectral component 9 of the short-wave radiation of the ocean),
    'C[10]': 0.0  [W][m^-2]  (spectral component 10 of the short-wave radiation of the ocean),
    'T0': 285.0  [K]  (stationary solution for the 0-th order oceanic temperature),

Creating the VDDG (qgs) tendencies function

```python
%%time
f_vddg, Df_vddg = create_tendencies(model_parameters)
```

    CPU times: user 4.88 s, sys: 630 ms, total: 5.51 s
    Wall time: 4.78 s

### Jin-Timmermann ENSO model

The nondimensional equations governing the ENSO module are given by:
$
\begin{align}
\frac{dx}{dt} = & \, \rho \delta (x^2-a x) + s x (x+y+c-c \tanh(x+z)), \\
\frac{dy}{dt} = & \, -\rho \delta (a y+x^2), \\
\frac{dz}{dt} = & \, \delta (k-z-\frac{x}{2}).
\end{align}
$

Here $x, y, z$ and $t$ are dimensionless, $x$ is the temperature difference between the eastern and western basins of the Tropical Pacific, $y$ the western basin's temperature anomaly with respect to a reference value, and $z$ the western basin's thermocline depth anomaly.

Setting first some model parameters

```python
# Periodic solutions
#delta = 0.00028058
#rho = 0.3224
#c = 2.3952
#k = 0.4032
#a = 6.8927
#s = 0.0010691

# Chaotic solutions
delta = 0.0002803
rho = 0.29016
c = 2.3952
k = 0.4032
a = 7.658609809
s = 0.001069075
```

Creating the tendencies function using a [Numba](https://numba.pydata.org/) jitted function

```python
@njit
def f_JT(t, x):
    dxdt = rho * delta * (x[0]**2 - a * x[0]) + s * x[0] * (x[0] + x[1] + c - c * np.tanh(x[2] + x[0]))
    dydt = -rho * delta * (a * x[1] + x[0]**2)
    dzdt = delta * (k - x[2] - x[0] / 2)

    return np.array([dxdt, dydt, dzdt])

```

#### Integrating the ENSO model to get averages

Defining an integrator

```python
integrator = RungeKuttaIntegrator()
integrator.set_func(f_JT)
```

First initial conditions for the ENSO model

```python
ic_enso = np.array([-2.8439, -0.62, 1.48])
```

```python
%%time
integrator.integrate(0., 10000000., dt, ic=ic_enso, write_steps=write_steps)
time, traj = integrator.get_trajectories()
# new initial condition
ic_enso = traj[:,-1]
# average
mean_enso = np.mean(traj, axis=-1)
```

    CPU times: user 211 ms, sys: 575 ms, total: 786 ms
    Wall time: 1min 59s

### Forced VDDG-ENSO model

The forced VDDG-ENSO model is obtained by modifying the tendency $f_1$ of the first component $\psi_{{\rm a}, 1}$ of the atmospheric streamfunction of VDDG:

$$
\frac{\mathrm{d}\psi_{{\rm a},1}}{\mathrm{d}t} = f_1(\psi_{{\rm a}, 1}, \theta_{{\rm a}, 1}) + g\, (x+y)'.
$$

where the parameter $g$ scales the intensity of the tropical forcing. The variable $(x + y)'$ is the Tropical eastern basin temperature anomaly of the ENSO model.

To simplify the integration, the full system of equations is constructed by concatenating together the equations of motion of both models, to give:
$$
\frac{\mathrm{d}\boldsymbol{X}}{\mathrm{d}t} = \boldsymbol{F}(\boldsymbol{X})
$$

Hence, we first set the coupling parameters:

```python
g = 0.03
enso_temperature_scaling = 2.8182
ndim = model_parameters.ndim
vddg_temperature_scaling = (model_parameters.scale_params.L * model_parameters.scale_params.f0)**2 / model_parameters.rr
enso_to_vddg_temperature_scaling = enso_temperature_scaling / vddg_temperature_scaling
mean_east_basin_enso = mean_enso[0] + mean_enso[1]
```

and then create the tendencies function of the full system using a [Numba](https://numba.pydata.org/) jitted function:

```python
@njit
def f_forced(t, x):
    tend_vddg = f_vddg(t, x[:ndim])
    tend_JT = f_JT(t, x[ndim:])
    tend_vddg[0] = tend_vddg[0] + g * enso_to_vddg_temperature_scaling * ((x[ndim] + x[ndim+1]) - mean_east_basin_enso)
    return np.concatenate((tend_vddg, tend_JT))
```

**The function** `f_forced` **hence defined is the tendencies** $\boldsymbol{F}(\boldsymbol{X})$ **of the ENSO forced VDDG model used in the publication above.**

## Time integration

#### Integrating the forced model

Start random initial conditions for the VDDG model and integrate over a transient time to obtain initial conditions on the attractor(s)

```python
%%time
## Reuse the integrator
integrator.set_func(f_forced)

## Might take several minutes, depending on your cpu computational power.
ic = np.random.rand(number_of_trajectories, model_parameters.ndim+3)*0.1
ic[:, ndim:] = ic_enso + np.zeros((number_of_trajectories, 3))
# Integration loop (to integrate a very long time)
for i in range(10):
    integrator.integrate(0., 1000000., dt, ic=ic, write_steps=0)
    time, ic = integrator.get_trajectories()
```

    CPU times: user 365 ms, sys: 1.21 s, total: 1.58 s
    Wall time: 8min 14s

Now integrate to obtain trajectories on the attractor(s)

```python
%%time
integrator.integrate(0., 1000000., dt, ic=ic, write_steps=write_steps)
time, traj = integrator.get_trajectories()

# Integration loop (to integrate a very long time)
for i in range(49):
    integrator.integrate(time[-1], time[-1]+1000000., dt, ic=traj[...,-1], write_steps=write_steps)
    time_new, traj_new = integrator.get_trajectories()
    time = np.concatenate((time, time_new[1:]))
    traj = np.concatenate((traj, traj_new[..., 1:]), axis=-1)

```

    CPU times: user 8.11 s, sys: 20.7 s, total: 28.8 s
    Wall time: 43min 12s

#### Plotting the result of the integration in 3D and 2D

3D plot of the variables $\psi_{{\rm o}, 2}$, $\theta_{{\rm o}, 2}$ and $\psi_{{\rm a}, 1}$ of the first trajectory

```python
varx = 21
vary = 29
varz = 0

fig = plt.figure(figsize=(10, 8))
axi = plt.axes(projection='3d')

axi.scatter(traj[0, varx], traj[0, vary], traj[0, varz], s=0.2);

axi.set_xlabel('$'+model_parameters.latex_var_string[varx]+'$')
axi.set_ylabel('$'+model_parameters.latex_var_string[vary]+'$')
axi.set_zlabel('$'+model_parameters.latex_var_string[varz]+'$');
```

2D plot of the variables $\psi_{{\rm o}, 2}$ and $\theta_{{\rm o}, 2}$ of the first and second trajectories

```python
varx = 21
vary = 29
plt.figure(figsize=(10, 8))

plt.plot(traj[:, varx].T, traj[:, vary].T, marker='o', ms=0.1, ls='')

plt.xlabel('$'+model_parameters.latex_var_string[varx]+'$')
plt.ylabel('$'+model_parameters.latex_var_string[vary]+'$');
```

Plot of the variables $\theta_{{\rm a}, 1}$, $\psi_{{\rm o}, 2}$, $\theta_{{\rm o}, 2}$ and $(x+y)$ versus the time for the first and second trajectories

```python
var1 = 10
var2 = 21
var3 = 29
plt.figure(figsize=(15, 32))

plt.subplot(4,1,1)
for nt in range(number_of_trajectories):
    plt.plot(model_parameters.dimensional_time*time, traj[nt, var1])
plt.xlabel('time (years)')
plt.ylabel('$'+model_parameters.latex_var_string[var1]+'$');

plt.subplot(4,1,2)
for nt in range(number_of_trajectories):
    plt.plot(model_parameters.dimensional_time*time, traj[nt, var2])
plt.xlabel('time (years)')
plt.ylabel('$'+model_parameters.latex_var_string[var2]+'$');

plt.subplot(4,1,3)
for nt in range(number_of_trajectories):
    plt.plot(model_parameters.dimensional_time*time, traj[nt, var3])
plt.xlabel('time (years)')
plt.ylabel('$'+model_parameters.latex_var_string[var3]+'$');

plt.subplot(4,1,4)
plt.plot(model_parameters.dimensional_time*time, traj[0, 36] + traj[0, 37])
plt.xlabel('time (years)')
plt.ylabel('$x + y$');
```


```python
import sympy
from phasor.utilities.ipynb.displays import *
from phasor.utilities.ipynb.ipy_sympy import *
import scipy.linalg

import numpy.testing as np_test
import declarative as decl
from declarative.bunch import (
    DeepBunch,
)

#import numpy as np

from phasor import system
from phasor import readouts
from phasor import optics
from phasor.optics.nonlinear_crystal import NonlinearCrystal
from phasor.utilities.print import pprint

from phasor.optics.models.KTP_test_stand import OPOTestStandResonant
```

    Populating the interactive namespace from numpy and matplotlib
    Sympy version:  1.0

```python
db = DeepBunch()
db.test.ktp.length.val = np.linspace(0, .12, 100)
db.test.ktp.N_ode = 10
db.test.include_AC = True
db.test.F_PM.frequency.val = 1
db.environment.F_AC.order = 1
db.test.generateF_PM.amplitude = 0#np.linspace(0, .3, 20)
sys = system.BGSystem(
    ctree = db,
)
sys.own.test = OPOTestStandResonant()
db = sys.ctree_shadow()
#print(sys.test.ktp.ctree_as_yaml())
```

```python
#print(sys.test.DC_R.DC_readout)
db.test.F_PM.frequency.val
#print(sys.test.DC_G.DC_readout)
```

```python
axB = mplfigB(Nrows=1)
test = sys.test
axB.ax0.plot(test.ktp.length_mm.val, test.DC_R.DC_readout, color = 'red')
axB.ax0.plot(test.ktp.length_mm.val, test.DC_G.DC_readout, color = 'green')
axB.ax0.plot(test.ktp.length_mm.val, test.DC_R.DC_readout + test.DC_G.DC_readout, color = 'black')
#axB.ax0.plot(test.ktp.length_mm.val, 1 * np.tanh(.200 * test.ktp.length_mm.val)**2, ls = '--', color = 'blue')
#axB.ax0.set_xscale('log')
#axB.ax0.set_ylim(0, 1.1)
```

```python
axB = mplfigB(Nrows=1)
test = sys.test
axB.ax0.plot(test.ktp.length_mm.val, test.AC_R.AC_CSD_IQ.real[0,0]**.5, color = 'red')
axB.ax0.plot(test.ktp.length_mm.val, test.AC_R.AC_CSD_IQ.real[1,1]**.5, color = 'blue')
axB.ax0.plot(test.ktp.length_mm.val, test.AC_R.AC_CSD_ellipse.max**.5, color = 'orange', ls = '--')
axB.ax0.plot(test.ktp.length_mm.val, test.AC_R.AC_CSD_ellipse.min**.5, color = 'purple', ls = '--')
axB.ax0.plot(test.ktp.length_mm.val, test.AC_R.AC_CSD_ellipse.min**.25 * test.AC_R.AC_CSD_ellipse.max**.25, color = 'black', ls = '--')
axB.ax0.axhline(1./2.)
axB.ax0.axvline(.11)
axB.ax0.set_yscale('log_zoom')
#axB.ax0.set_ylim(0, 1.1)
```

```python
db = DeepBunch()
db.test.ktp.length.val = .11
#db.test.ktp.length.val = np.linspace(0, .2, 100)
db.test.ktp.N_ode = 1
db.test.ktp.solution_order = 4
db.test.include_AC = True
db.environment.F_AC.order = 1
db.test.F_PM.frequency.val = 1
db.test.generateF_PM.amplitude = np.linspace(0, .3, 20)
sys = system.BGSystem(
    ctree = db,
)
sys.own.test = OPOTestStandResonant()
db = sys.ctree_shadow()
#print(sys.test.ktp.ctree_as_yaml())

axB = mplfigB(Nrows=1)
test = sys.test
#axB.ax0.plot(test.generateF_PM.amplitude.real, test.AC_R.AC_CSD_IQ.real[0,0]**.5, color = 'red')
axB.ax0.plot(test.generateF_PM.amplitude.real, test.AC_R.AC_CSD_IQ.real[1,1]**.5, color = 'blue')
#axB.ax0.plot(test.generateF_PM.amplitude.real, test.AC_R.AC_CSD_ellipse.max**.5, color = 'orange', ls = '--')
axB.ax0.plot(test.generateF_PM.amplitude.real, test.AC_R.AC_CSD_ellipse.min**.5, color = 'purple', ls = '--')
axB.ax0.plot(test.generateF_PM.amplitude.real,
             (test.AC_R.AC_CSD_ellipse.min[0] +
             test.generateF_PM.amplitude.real**2/(np.pi**2) * test.AC_R.AC_CSD_ellipse.max[0])**.5,
             color = 'cyan', ls = '-')
#axB.ax0.plot(test.generateF_PM.amplitude, test.AC_R.AC_CSD_ellipse.min**.25 * test.AC_R.AC_CSD_ellipse.max**.25, color = 'black', ls = '--')
#axB.ax0.set_yscale('log_zoom')
#axB.ax0.plot(test.ktp.length_mm, test.DC_G.DC_readout, color = 'green')
#axB.ax0.plot(test.ktp.length_mm, test.DC_R.DC_readout + test.DC_G.DC_readout, color = 'black')
#axB.ax0.plot(test.ktp.length_mm, 1 * np.tanh(.100 * test.ktp.length_mm.val)**2, color = 'blue')
#axB.ax0.set_ylim(0, 1.1)
```

```python
db = DeepBunch()
db.test.ktp.length.val = .11
#db.test.ktp.length.val = np.linspace(0, .2, 100)
db.test.ktp.N_ode = 1
db.test.ktp.solution_order = 4
db.test.include_AC = True
db.environment.F_AC.order = 1
db.test.F_PM.frequency.val = 1

db.test.generateF_PM.amplitude = 0
db.test.generateF_PMRead.amplitude = np.linspace(0, .3, 20)/2
sys = system.BGSystem(
    ctree = db,
)
sys.own.test = OPOTestStandResonant()
db = sys.ctree_shadow()
#print(sys.test.ktp.ctree_as_yaml())

axB = mplfigB(Nrows=1)
test = sys.test
#axB.ax0.plot(test.generateF_PM.amplitude.real, test.AC_R.AC_CSD_IQ.real[0,0]**.5, color = 'red')
axB.ax0.plot(2 * test.generateF_PMRead.amplitude.real, test.AC_R.AC_CSD_IQ.real[1,1]**.5, color = 'blue')
#axB.ax0.plot(test.generateF_PM.amplitude.real, test.AC_R.AC_CSD_ellipse.max**.5, color = 'orange', ls = '--')
axB.ax0.plot(2 * test.generateF_PMRead.amplitude.real, test.AC_R.AC_CSD_ellipse.min**.5, color = 'purple', ls = '--')
axB.ax0.plot(2 * test.generateF_PMRead.amplitude.real,
             (test.AC_R.AC_CSD_ellipse.min[0] +
             4 * test.generateF_PMRead.amplitude.real**2/(np.pi**2) * test.AC_R.AC_CSD_ellipse.max[0])**.5,
             color = 'cyan', ls = '-')
#axB.ax0.plot(test.generateF_PM.amplitude, test.AC_R.AC_CSD_ellipse.min**.25 * test.AC_R.AC_CSD_ellipse.max**.25, color = 'black', ls = '--')
#axB.ax0.set_yscale('log_zoom')
#axB.ax0.plot(test.ktp.length_mm, test.DC_G.DC_readout, color = 'green')
#axB.ax0.plot(test.ktp.length_mm, test.DC_R.DC_readout + test.DC_G.DC_readout, color = 'black')
#axB.ax0.plot(test.ktp.length_mm, 1 * np.tanh(.100 * test.ktp.length_mm.val)**2, color = 'blue')
#axB.ax0.set_ylim(0, 1.1)
```

```python
db = DeepBunch()
db.test.ktp.length.val = .11
#db.test.ktp.length.val = np.linspace(0, .2, 100)
db.test.ktp.N_ode = 1
db.test.ktp.solution_order = 6
db.test.include_AC = True
db.environment.F_AC.order = 1
db.test.F_PM.frequency.val = 1

db.test.generateF_PM.amplitude = np.linspace(0, .3, 20)
db.test.generateF_PMRead.amplitude = np.linspace(0, .3, 20)/2
sys = system.BGSystem(
    ctree = db,
)
sys.own.test = OPOTestStandResonant()
db = sys.ctree_shadow()
#print(sys.test.ktp.ctree_as_yaml())

axB = mplfigB(Nrows=1)
test = sys.test
#axB.ax0.plot(test.generateF_PM.amplitude.real, test.AC_R.AC_CSD_IQ.real[0,0]**.5, color = 'red')
axB.ax0.plot(2 * test.generateF_PMRead.amplitude.real, test.AC_R.AC_CSD_IQ.real[1,1]**.5, color = 'blue')
#axB.ax0.plot(test.generateF_PM.amplitude.real, test.AC_R.AC_CSD_ellipse.max**.5, color = 'orange', ls = '--')
axB.ax0.plot(2 * test.generateF_PMRead.amplitude.real, test.AC_R.AC_CSD_ellipse.min**.5, color = 'purple', ls = '--')
#axB.ax0.plot(2 * test.generateF_PMRead.amplitude.real,
#             (test.AC_R.AC_CSD_ellipse.min[0] +
#             4 * test.generateF_PMRead.amplitude.real**2/(np.pi**2) * test.AC_R.AC_CSD_ellipse.max[0])**.5,
#             color = 'cyan', ls = '-')
```

```python
#_ = sys.test.full_noise_matrix()
```

```python

```

```python
test.AC_R.AC_CSD_ellipse.min
```

```python
1/4
```

```python
1/.06
```

```python
.3 + 1
```

```python
db = DeepBunch()
db.test.ktp.length.val = .11
#db.test.ktp.length.val = np.linspace(0, .2, 100)
db.test.ktp.N_ode = 1
db.test.ktp.solution_order = 4
db.test.include_AC = True
db.environment.F_AC.order = 1
db.test.F_PM.frequency.val = 1
db.test.PSLG.phase.val = np.linspace(0, 360, 100)
db.test.generateF_PM.amplitude = 0 #np.linspace(0, .3, 20)
sys = system.BGSystem(
    ctree = db,
)
sys.own.test = OPOTestStandResonant()
db = sys.ctree_shadow()
#print(sys.test.ktp.ctree_as_yaml())
```

```python
axB = mplfigB(Nrows=2)
test = sys.test
axB.ax0.plot(test.PSLG.phase_deg.val, test.AC_R.AC_CSD_IQ.real[0,0], color = 'red')
axB.ax0.plot(test.PSLG.phase_deg.val, test.AC_R.AC_CSD_IQ.real[1,1], color = 'blue')
axB.ax0.plot(test.PSLG.phase_deg.val, test.AC_R.AC_CSD_ellipse.max, color = 'orange', ls = '--')
axB.ax0.plot(test.PSLG.phase_deg.val, test.AC_R.AC_CSD_ellipse.min, color = 'purple', ls = '--')
axB.ax1.plot(test.PSLG.phase_deg.val, test.AC_R.AC_CSD_ellipse.deg, color = 'green', ls = '-')
#axB.ax0.plot(test.generateF_PM.amplitude, test.AC_R.AC_CSD_ellipse.min**.25 * test.AC_R.AC_CSD_ellipse.max**.25, color = 'black', ls = '--')
axB.ax0.set_yscale('log_zoom')
#axB.ax0.plot(test.ktp.length_mm, test.DC_G.DC_readout, color = 'green')
#axB.ax0.plot(test.ktp.length_mm, test.DC_R.DC_readout + test.DC_G.DC_readout, color = 'black')
#axB.ax0.plot(test.ktp.length_mm, 1 * np.tanh(.100 * test.ktp.length_mm.val)**2, color = 'blue')
#axB.ax0.set_ylim(0, 1.1)
```

```python

db.test.PSLG.phase.units
```

```python

```