# Chapter 6 Linear Model Selection and Regularization

- Recall the linear model
\begin{equation}Y=\beta_{0}+\beta_{1} X_{1}+\cdots+\beta_{p} X_{p}+\epsilon\end{equation}
- In the lectures that follow, we consider some approaches for extending the linear model framework. In the lectures covering Chapter 7 of the text, we generalize the linear model in order to accommodate non-linear, but still additive, relationships.
- In the lectures covering Chapter 8 we consider even more general non-linear models.
- Despite its simplicity, the linear model has distinct advantages in terms of its interpretability and often shows good predictive performance.
- Hence we discuss in this lecture some ways in which the simple linear model can be improved, by replacing ordinary least squares fitting with some alternative fitting procedures.

### Why consider alternatives to least squares?

- Prediction Accuracy: especially when p > n, to control the variance.
- Model Interpretability: By removing irrelevant features, that is, by setting the corresponding coefficient estimates to zero we can obtain a model that is more easily interpreted. We will present some approaches for automatically performing feature selection.

### Three classes of methods

- Subset Selection. We identify a subset of the p predictors
that we believe to be related to the response. We then fit a
model using least squares on the reduced set of variables.
- Shrinkage. We fit a model involving all p predictors, but
the estimated coefficients are shrunken towards zero
relative to the least squares estimates. This shrinkage (also
known as regularization) has the effect of reducing variance
and can also perform variable selection.
- Dimension Reduction. We project the p predictors into a
M-dimensional subspace, where M < p. This is achieved by
computing M different linear combinations, or projections,
of the variables. Then these M projections are used as
predictors to fit a linear regression model by least squares.

### Subset selection

#### Best Subset Selection
1. Let $\mathcal{M}_{0}$ denote the null model, which contains no predictors. This model simply predicts the sample mean for each observation.
2. For $k=1,2, \ldots p:$
(a) Fit all $\left(\begin{array}{l}p \\ k\end{array}\right)$ models that contain exactly $k$ predictors.
(b) Pick the best among these $\left(\begin{array}{l}p \\ k\end{array}\right)$ models, and call it $\mathcal{M}_{k} .$ Here best is defined as having the smallest RSS, or equivalently largest $R^{2}$.
3. Select a single best model from among $\mathcal{M}_{0}, \ldots, \mathcal{M}_{p}$ using cross-validated prediction error, $C_{p}(\mathrm{AIC}), \mathrm{BIC},$ or adjusted $R^{2}$

```python
# Example - Credit data set
#load modules
import pandas as pd
from sklearn.linear_model import LinearRegression
from mlxtend.feature_selection import ExhaustiveFeatureSelector as EFS
import numpy as np
import matplotlib.pyplot as plt
from mlxtend.feature_selection import SequentialFeatureSelector as sfs
from sklearn.model_selection import train_test_split
```

```python
#load credit card dataset
credit = pd.read_csv('data/Credit.csv', index_col =0)
credit.dtypes
```

    Limit         int64
    Rating        int64
    Cards         int64
    Age           int64
    Education     int64
    Gender       object
    Student      object
    Married      object
    Ethnicity    object
    Balance       int64
    dtype: object

```python
#mark the predictors and respond variables
X= credit.iloc[:, :-1]
y= credit.Balance
```

```python
#convert the categorical variables
credit_dummies = pd.get_dummies(credit[['Gender', 'Student', 'Married', 'Ethnicity']])
credit =credit.drop(credit[['Gender', 'Student', 'Married', 'Ethnicity']],axis=1)
credit_dummies =credit_dummies.drop(['Gender_Female', 'Student_No', 'Married_No', 'Ethnicity_African American'],axis=1)
credit_df= pd.concat([credit, credit_dummies], axis=1)
credit_df.info
```

    <bound method DataFrame.info of          Limit  Rating  Cards  Age  Education  Balance  Gender_ Male  \
    Income
    14.891    3606     283      2   34         11      333             1
    106.025   6645     483      3   82         15      903             0
    104.593   7075     514      4   71         11      580             1
    148.924   9504     681      3   36         11      964             0
    55.882    4897     357      2   68         16      331             1
    ...        ...     ...    ...  ...        ...      ...           ...
    12.096    4100     307      3   32         13      560             1
    13.364    3838     296      5   65         17      480             1
    57.872    4171     321      5   67         12      138             0
    37.728    2525     192      1   44         13        0             1
    18.701    5524     415      5   64          7      966             0

             Student_Yes  Married_Yes  Ethnicity_Asian  Ethnicity_Caucasian
    Income
    14.891             0            1                0                    1
    106.025            1            1                1                    0
    104.593            0            0                1                    0
    148.924            0            0                1                    0
    55.882             0            1                0                    1
    ...              ...          ...              ...                  ...
    12.096             0            1                0                    1
    13.364             0            0                0                    0
    57.872             0            1                0                    1
    37.728             0            1                0                    1
    18.701             0            0                1                    0

    [400 rows x 11 columns]>

```python
#mark the predictors and respond variables
X= credit_df.loc[:, credit_df.columns != 'Balance']
y= credit_df.Balance
```

```python
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

def fit_ols(X,y):
    ols = LinearRegression()
    ols_model =ols.fit(X,y)
    RSS= round(mean_squared_error(y,ols_model.predict(X)) * len(y),2)
    R_square = round(ols_model.score(X,y),2)
    return RSS, R_square
```

```python
fit_ols(X,y)
```

    (14617892.35, 0.83)

```python
import itertools

k = 11
RSS_list, R_squared_list, feature_list = [],[], []
numb_features = []

#Looping over k = 1 to k = 11 features in X
for k in range(1,len(X.columns) + 1):
    #Looping over all possible combinations: from 11 choose k
    for combo in itertools.combinations(X.columns,k):
        tmp_result = fit_ols(X[list(combo)],y)   #Store temp result b
        RSS_list.append(tmp_result[0])                  #Append lists
        R_squared_list.append(tmp_result[1])
        feature_list.append(combo)
        numb_features.append(len(combo))

#Store in DataFrame
df = pd.DataFrame({'numb_features': numb_features,'RSS': RSS_list, 'R_squared':R_squared_list,'features':feature_list})
```

```python
df['min_RSS'] = df.groupby('numb_features')['RSS'].transform(min)
df['max_R_squared'] = df.groupby('numb_features')['R_squared'].transform(max)
```

```python
fig = plt.figure(figsize = (16,6))
ax = fig.add_subplot(1, 2, 1)

ax.scatter(df.numb_features,df.RSS, alpha = .2, color = 'darkblue' )
ax.set_xlabel('# Features')
ax.set_ylabel('RSS')
ax.set_title('RSS - Best subset selection')
ax.plot(df.numb_features,df.min_RSS,color = 'r', label = 'Best subset')
ax.legend()

ax = fig.add_subplot(1, 2, 2)
ax.scatter(df.numb_features,df.R_squared, alpha = .2, color = 'darkblue' )
ax.plot(df.numb_features,df.max_R_squared,color = 'r', label = 'Best subset')
ax.set_xlabel('# Features')
ax.set_ylabel('R squared')
ax.set_title('R_squared - Best subset selection')
ax.legend()

plt.show()
```

For each possible model containing a subset of the ten predictors in the Credit data set, the RSS and R2 are displayed. The red frontier tracks the best model for a given number of predictors, according to RSS and R2. Though the data set contains only ten predictors, the x-axis ranges from 1 to 11, since one of the variables is categorical and takes on three values, leading to the creation of two dummy variables.
Although we have presented best subset selection here for least squares regression, the same ideas apply to other types of models, such as logistic regression. The deviance, negative two times the maximized log-likelihood, plays the role of RSS for a broader class of models.

### Stepwise Selection

- For computational reasons, best subset selection cannot be
applied with very large p. Why not?
- Best subset selection may also suffer from statistical
problems when p is large: larger the search space, the
higher the chance of finding models that look good on the
training data, even though they might not have any
predictive power on future data.
- Thus an enormous search space can lead to overfitting and
high variance of the coefficient estimates.
- For both of these reasons, stepwise methods, which explore
a far more restricted set of models, are attractive
alternatives to best subset selection.

### Forward Stepwise Selection
- Forward stepwise selection begins with a model containing
no predictors, and then adds predictors to the model,
one-at-a-time, until all of the predictors are in the model.
- In particular, at each step the variable that gives the
greatest additional improvement to the fit is added to the
model.
1. Let $\mathcal{M}_{0}$ denote the null model, which contains no predictors.
2. For $k=0, \ldots, p-1:$
2.1 Consider all $p-k$ models that augment the predictors in $\mathcal{M}_{k}$ with one additional predictor.
2.2 Choose the best among these $p-k$ models, and call it $\mathcal{M}_{k+1} .$ Here best is defined as having smallest RSS or highest $R^{2}$
3. Select a single best model from among $\mathcal{M}_{0}, \ldots, \mathcal{M}_{p}$ using cross-validated prediction error, $C_{p}(\mathrm{AIC}), \mathrm{BIC},$ or adjusted $R^{2}$.

- Computational advantage over best subset selection is
clear.
- It is not guaranteed to find the best possible model out of
all $2^{p}$ models containing subsets of the p predictors. Why
not? Give an example.

```python
# one variable
ols = LinearRegression()
sfs1 = sfs(ols,
           k_features=1,
           forward=True,
           floating=False,
           verbose=2,
           scoring='neg_mean_squared_error',
           cv=5,
           n_jobs=-1)
sfs1.fit(X, y)

```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    2.5s finished

    [2020-04-22 16:39:18] Features: 1/1 -- score: -54293.176584293506

    SequentialFeatureSelector(clone_estimator=True, cv=5,
                              estimator=LinearRegression(copy_X=True,
                                                         fit_intercept=True,
                                                         n_jobs=None,
                                                         normalize=False),
                              fixed_features=None, floating=False, forward=True,
                              k_features=1, n_jobs=-1, pre_dispatch='2*n_jobs',
                              scoring='neg_mean_squared_error', verbose=2)

```python
# two variables
sfs2 = sfs(ols,
           k_features=2,
           forward=True,
           floating=False,
           verbose=2,
           scoring='neg_mean_squared_error',
           cv=5,
           n_jobs=-1)
sfs2.fit(X, y)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.0s finished

    [2020-04-22 16:39:20] Features: 1/2 -- score: -54293.176584293506[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
    [Parallel(n_jobs=-1)]: Done   7 out of   9 | elapsed:    0.0s remaining:    0.0s
    [Parallel(n_jobs=-1)]: Done   9 out of   9 | elapsed:    0.0s finished

    [2020-04-22 16:39:20] Features: 2/2 -- score: -40104.361174004574

    SequentialFeatureSelector(clone_estimator=True, cv=5,
                              estimator=LinearRegression(copy_X=True,
                                                         fit_intercept=True,
                                                         n_jobs=None,
                                                         normalize=False),
                              fixed_features=None, floating=False, forward=True,
                              k_features=2, n_jobs=-1, pre_dispatch='2*n_jobs',
                              scoring='neg_mean_squared_error', verbose=2)

```python
# three variable
sfs3 = sfs(ols, k_features =3, forward =True, floating =False, verbose= 2, scoring = 'neg_mean_squared_error', cv=5, n_jobs=-1)
sfs3.fit(X,y)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.0s finished

    [2020-04-22 16:39:21] Features: 1/3 -- score: -54293.176584293506[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
    [Parallel(n_jobs=-1)]: Done   7 out of   9 | elapsed:    0.0s remaining:    0.0s
    [Parallel(n_jobs=-1)]: Done   9 out of   9 | elapsed:    0.0s finished

    [2020-04-22 16:39:21] Features: 2/3 -- score: -40104.361174004574[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
    [Parallel(n_jobs=-1)]: Done   6 out of   8 | elapsed:    0.0s remaining:    0.0s
    [Parallel(n_jobs=-1)]: Done   8 out of   8 | elapsed:    0.0s finished

    [2020-04-22 16:39:21] Features: 3/3 -- score: -38706.395570382505

    SequentialFeatureSelector(clone_estimator=True, cv=5,
                              estimator=LinearRegression(copy_X=True,
                                                         fit_intercept=True,
                                                         n_jobs=None,
                                                         normalize=False),
                              fixed_features=None, floating=False, forward=True,
                              k_features=3, n_jobs=-1, pre_dispatch='2*n_jobs',
                              scoring='neg_mean_squared_error', verbose=2)

```python
# four variables
sfs4 = sfs(ols, k_features =4, forward =True, floating =False, verbose= 2, scoring = 'neg_mean_squared_error', cv=5, n_jobs=-1)
sfs4.fit(X,y)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.0s finished

    [2020-04-22 16:39:21] Features: 1/4 -- score: -54293.176584293506[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
    [Parallel(n_jobs=-1)]: Done   7 out of   9 | elapsed:    0.0s remaining:    0.0s
    [Parallel(n_jobs=-1)]: Done   9 out of   9 | elapsed:    0.0s finished

    [2020-04-22 16:39:21] Features: 2/4 -- score: -40104.361174004574[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
    [Parallel(n_jobs=-1)]: Done   6 out of   8 | elapsed:    0.0s remaining:    0.0s
    [Parallel(n_jobs=-1)]: Done   8 out of   8 | elapsed:    0.0s finished

    [2020-04-22 16:39:22] Features: 3/4 -- score: -38706.395570382505[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.
    [Parallel(n_jobs=-1)]: Done   4 out of   7 | elapsed:    0.0s remaining:    0.0s
    [Parallel(n_jobs=-1)]: Done   7 out of   7 | elapsed:    0.0s finished

    [2020-04-22 16:39:22] Features: 4/4 -- score: -38285.92694490571

    SequentialFeatureSelector(clone_estimator=True, cv=5,
                              estimator=LinearRegression(copy_X=True,
                                                         fit_intercept=True,
                                                         n_jobs=None,
                                                         normalize=False),
                              fixed_features=None, floating=False, forward=True,
                              k_features=4, n_jobs=-1, pre_dispatch='2*n_jobs',
                              scoring='neg_mean_squared_error', verbose=2)

```python
dat= [['one variable', sfs1.k_feature_names_],['two variables', sfs2.k_feature_names_], ['three variables', sfs3.k_feature_names_], ['four variables', sfs4.k_feature_names_]]

dat_df = pd.DataFrame(dat, columns=['Number of Variables','Forward stepwise Selection'])

print(dat_df)
```

      Number of Variables         Forward stepwise Selection
    0        one variable                          (Rating,)
    1       two variables              (Rating, Student_Yes)
    2     three variables         (Rating, Age, Student_Yes)
    3      four variables  (Rating, Cards, Age, Student_Yes)

The first four selected models for best subset selection and
forward stepwise selection on the Credit data set. The first
three models are identical but the fourth models differ.

### Backward Subset Selection

- Like forward stepwise selection, backward stepwise selection
provides an efficient alternative to best subset selection.
- However, unlike forward stepwise selection, it begins with
the full least squares model containing all p predictors, and
then iteratively removes the least useful predictor,
one-at-a-time.

1. Let $\mathcal{M}_{p}$ denote the full model, which contains all $p$ predictors.
2. For $k=p, p-1, \ldots, 1:$
2.1 Consider all $k$ models that contain all but one of the predictors in $\mathcal{M}_{k},$ for a total of $k-1$ predictors.
2.2 Choose the best among these $k$ models, and call it $\mathcal{M}_{k-1}$ Here best is defined as having smallest RSS or highest $R^{2}$.
3. Select a single best model from among $\mathcal{M}_{0}, \ldots, \mathcal{M}_{p}$ using cross-validated prediction error, $C_{p}(\mathrm{AIC}), \mathrm{BIC},$ or adjusted $R^{2}$

- Like forward stepwise selection, the backward selection
approach searches through only 1 + p(p + 1)=2 models, and
so can be applied in settings where p is too large to apply
best subset selection
- Like forward stepwise selection, backward stepwise
selection is not guaranteed to yield the best model
containing a subset of the p predictors.
- Backward selection requires that the number of samples n
is larger than the number of variables p (so that the full
model can be fit). In contrast, forward stepwise can be
used even when n < p, and so is the only viable subset
method when p is very large.

### Choosing the optimal model

- The model containing all of the predictors will always have
the smallest RSS and the largest R2, since these quantities
are related to the training error.
- We wish to choose a model with low test error, not a model
with low training error. Recall that training error is usually
a poor estimate of test error.
- Therefore, RSS and R2 are not suitable for selecting the
best model among a collection of models with dierent
numbers of predictors.

### Estimating test error: two approaches

- We can indirectly estimate test error by making an
adjustment to the training error to account for the bias due
to overfitting.
- We can directly estimate the test error, using either a
validation set approach or a cross-validation approach, as
discussed in previous lectures.

#### Adjusting training error

- These techniques adjust the training error for the model
size, and can be used to select among a set of models with
different numbers of variables.
- The next figures displays Cp, BIC, and adjusted R2 for the
best model of each size produced by best subset selection
on the Credit data set.

#### Mallow's $C_{p}$ :
$$
C_{p}=\frac{1}{n}\left(\mathrm{RSS}+2 d \hat{\sigma}^{2}\right)
$$
where $d$ is the total # of parameters used and $\hat{\sigma}^{2}$ is an estimate of the variance of the error $\epsilon$ associated with each response measurement.
- The $A I C$ criterion is defined for a large class of models fit by maximum likelihood:
$$
\mathrm{AIC}=-2 \log L+2 \cdot d
$$
where $L$ is the maximized value of the likelihood function for the estimated model.
: In the case of the linear model with Gaussian errors, maximum likelihood and least squares are the same thing, and $C_{p}$ and AIC are equivalent.

#### BIC
$$
\mathrm{BIC}=\frac{1}{n}\left(\mathrm{RSS}+\log (n) d \hat{\sigma}^{2}\right)
$$
- Like $C_{p},$ the BIC will tend to take on a small value for a model with a low test error, and so generally we select the model that has the lowest BIC value.
" Notice that BIC replaces the $2 d \hat{\sigma}^{2}$ used by $C_{p}$ with a $\log (n) d \hat{\sigma}^{2}$ term, where $n$ is the number of observations.
- since $\log n>2$ for any $n>7,$ the BIC statistic generally places a heavier penalty on models with many variables, and hence results in the selection of smaller models than $C_{p} .$ See Figure on slide 19

#### Adjusted $R^{2}$
- For a least squares model with $d$ variables, the adjusted $R^{2}$ statistic is calculated as
$$
\text { Adjusted } R^{2}=1-\frac{\mathrm{RSS} /(n-d-1)}{\operatorname{TSS} /(n-1)}
$$
where TSS is the total sum of squares.
\bullet Unlike $C_{p}, \mathrm{AIC},$ and $\mathrm{BIC},$ for which a small value indicates a model with a low test error, a large value of adjusted $R^{2}$ indicates a model with a small test error.
- Maximizing the adjusted $R^{2}$ is equivalent to minimizing $\frac{\mathrm{RSS}}{n-d-1} \cdot$ While RSS always decreases as the number of variables in the model increases, $\frac{\mathrm{RSS}}{n-d-1}$ may increase or decrease, due to the presence of $d$ in the denominator.
- Unlike the $R^{2}$ statistic, the adjusted $R^{2}$ statistic pays $a$ price for the inclusion of unnecessary variables in the model.

#### Validation and Cross-Validation
。 Each of the procedures returns a sequence of models $\mathcal{M}_{k}$ indexed by model size $k=0,1,2, \ldots . .$ Our job here is to select $\hat{k} .$ Once selected, we will return model $\mathcal{M}_{\hat{k}}$
・ We compute the validation set error or the cross-validation error for each model $\mathcal{M}_{k}$ under consideration, and then select the $k$ for which the resulting estimated test error is smallest.
- This procedure has an advantage relative to AIC, BIC, $C_{p}$ and adjusted $R^{2},$ in that it provides a direct estimate of the test error, and doesn't require an estimate of the error variance $\sigma^{2}$
It can also be used in a wider range of model selection tasks, even in cases where it is hard to pinpoint the model degrees of freedom (e.g. the number of predictors in the model) or hard to estimate the error variance $\sigma^{2}$

#### Ridge regression and Lasso
- The subset selection methods use least squares to fit a
linear model that contains a subset of the predictors.
- As an alternative, we can fit a model containing all p
predictors using a technique that constrains or regularizes
the coefficient estimates, or equivalently, that shrinks the
coefficient estimates towards zero.
- It may not be immediately obvious why such a constraint
should improve the fit, but it turns out that shrinking the
coefficient estimates can significantly reduce their variance.

#### Ridge Regression
。 Recall that the least squares fitting procedure estimates $\beta_{0}, \beta_{1}, \ldots, \beta_{p}$ using the values that minimize
$$
\mathrm{RSS}=\sum_{i=1}^{n}\left(y_{i}-\beta_{0}-\sum_{j=1}^{p} \beta_{j} x_{i j}\right)^{2}
$$
In contrast, the ridge regression coefficient estimates $\hat{\beta}^{R}$ are the values that minimize
$\sum_{i=1}^{n}\left(y_{i}-\beta_{0}-\sum_{j=1}^{p} \beta_{j} x_{i j}\right)^{2}+\lambda \sum_{j=1}^{p} \beta_{j}^{2}=\mathrm{RSS}+\lambda \sum_{j=1}^{p} \beta_{j}^{2}$
where $\lambda \geq 0$ is a tuning parameter, to be determined separately.
- As with least squares, ridge regression seeks coefficient
estimates that fit the data well, by making the RSS small.
- However, the second term, $\lambda \sum_{j=1}^{p}$, called a shrinkage
penalty, is small when $\beta_{0}, \beta_{1}, \ldots, \beta_{p}$ are close to zero, and so it
has the effect of shrinking the estimates of $\beta_{j}$ towards zero.
- The tuning parameter $\lambda$ serves to control the relative
impact of these two terms on the regression coefficient
estimates.
- Selecting a good value for $\lambda$ is critical; cross-validation is
used for this.

```python
# Credit Card Example
import pandas as pd
import numpy as np
from numpy import linalg
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.preprocessing import scale, StandardScaler
from sklearn import model_selection
from sklearn.linear_model import LinearRegression, Ridge, RidgeCV, Lasso, LassoCV
from sklearn.decomposition import PCA
from sklearn.cross_decomposition import PLSRegression
from sklearn.model_selection import KFold, cross_val_score
from sklearn.metrics import mean_squared_error

import itertools
from itertools import combinations

%matplotlib inline
plt.style.use('seaborn-white')
```

```python
df_credit = pd.read_csv('Data/Credit.csv', index_col=0)
df_credit["Gender"] = df_credit["Gender"].astype('category')
df_credit["Student"] = df_credit["Student"].astype('category')
df_credit["Married"] = df_credit["Married"].astype('category')
df_credit["Ethnicity"] = df_credit["Ethnicity"].astype('category')
df_credit.info()
```

    <class 'pandas.core.frame.DataFrame'>
    Float64Index: 400 entries, 14.890999999999998 to 18.701
    Data columns (total 10 columns):
     #   Column     Non-Null Count  Dtype
    ---  ------     --------------  -----
     0   Limit      400 non-null    int64
     1   Rating     400 non-null    int64
     2   Cards      400 non-null    int64
     3   Age        400 non-null    int64
     4   Education  400 non-null    int64
     5   Gender     400 non-null    category
     6   Student    400 non-null    category
     7   Married    400 non-null    category
     8   Ethnicity  400 non-null    category
     9   Balance    400 non-null    int64
    dtypes: category(4), int64(6)
    memory usage: 23.8 KB

```python
y = df_credit.Balance

X = df_credit[df_credit.columns.difference(['Balance'])]
# Use K-1 columns for K categories so Yes/No uses only one column 0/1
X = pd.get_dummies(X, drop_first=True)
X_scaled = scale(X)
X.head(3)
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Age</th>
      <th>Cards</th>
      <th>Education</th>
      <th>Limit</th>
      <th>Rating</th>
      <th>Ethnicity_Asian</th>
      <th>Ethnicity_Caucasian</th>
      <th>Gender_Female</th>
      <th>Married_Yes</th>
      <th>Student_Yes</th>
    </tr>
    <tr>
      <th>Income</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>14.891</th>
      <td>34</td>
      <td>2</td>
      <td>11</td>
      <td>3606</td>
      <td>283</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
    </tr>
    <tr>
      <th>106.025</th>
      <td>82</td>
      <td>3</td>
      <td>15</td>
      <td>6645</td>
      <td>483</td>
      <td>1</td>
      <td>0</td>
      <td>1</td>
      <td>1</td>
      <td>1</td>
    </tr>
    <tr>
      <th>104.593</th>
      <td>71</td>
      <td>4</td>
      <td>11</td>
      <td>7075</td>
      <td>514</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
    </tr>
  </tbody>
</table>
</div>

```python
n_lambdas = 200
lambdas = np.logspace(5, -2, n_lambdas)

coefs = []
scores = []
for lam in lambdas:
    ridge = Ridge(alpha=lam)
    # it's very important to scale the data for Ridge regression
    ridge.fit(X_scaled, y)
    coefs.append(ridge.coef_)
    scores.append(ridge.score(X_scaled, y))
```

```python
fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(15,4))

# get the four largest (in abs value) coefficient positions
ind = np.argpartition(np.abs(coefs[-1]), -4)[-4:]

# firt plot
ax1.plot(lambdas, coefs)
ax1.set_xscale('log')
ax1.set_xlabel('lambda')
ax1.set_ylabel('Standardized Coefficients')
ax1.legend(np.array(ax1.get_lines())[ind], X.columns[ind]);

# second plot
no_ridge_norm = linalg.norm(coefs[-1])
norm_coefs = linalg.norm(coefs/no_ridge_norm, axis=1)
ax2.plot(norm_coefs, coefs)
ax2.set_xlabel('Normalized Ridge coefficients')
ax2.set_ylabel('Standardized Coefficients')
ax2.legend(np.array(ax2.get_lines())[ind], X.columns[ind]); #delimiter

# third
ax3.plot(lambdas, scores)
ax3.set_xscale('log')
ax3.set_xlabel('lambda')
ax3.set_ylabel('R^2')
ax3.set_title('R^2 as a function of regularization');

```

- In the left-hand panel, each curve corresponds to the ridge regression coefficient estimate for one of the ten variables, plotted as a function of $\lambda$
- The right-hand panel displays the same ridge coefficient estimates as the left-hand panel, but instead of displaying $\lambda$ on the $x$ -axis, we now display $\left\|\hat{\beta}_{\lambda}^{R}\right\|_{2} /\|\hat{\beta}\|_{2},$ where $\hat{\beta}$ denotes the vector of least squares coefficient estimates.
- The notation $\|\beta\|_{2}$ denotes the $\ell_{2}$ norm (pronounced "ell $\left.2^{\prime \prime}\right)$ of a vector, and is defined as $\|\beta\|_{2}=\sqrt{\sum_{j=1}^{p} \beta_{j}^{2}}$

#### Scaling of predictors

- The standard least squares coefficient estimates are scale equivariant: multiplying $X_{j}$ by a constant $c$ simply leads to a scaling of the least squares coefficient estimates by a factor of $1 / c .$ In other words, regardless of how the $j$ th predictor is scaled, $X_{j} \hat{\beta}_{j}$ will remain the same.
In contrast, the ridge regression coefficient estimates can change substantially when multiplying a given predictor by a constant, due to the sum of squared coefficients term in the penalty part of the ridge regression objective function.
- Therefore, it is best to apply ridge regression after standardizing the predictors, using the formula
$$
\tilde{x}_{i j}=\frac{x_{i j}}{\sqrt{\frac{1}{n} \sum_{i=1}^{n}\left(x_{i j}-\bar{x}_{j}\right)^{2}}}
$$

### Lasso
- Ridge regression does have one obvious disadvantage:
unlike subset selection, which will generally select models that involve just a subset of the variables, ridge regression will include all $p$ predictors in the final model
- The Lasso is a relatively recent alternative to ridge regression that overcomes this disadvantage. The lasso coefficients, $\hat{\beta}_{\lambda}^{L},$ minimize the quantity
$\sum_{i=1}^{n}\left(y_{i}-\beta_{0}-\sum_{j=1}^{p} \beta_{j} x_{i j}\right)^{2}+\lambda \sum_{j=1}^{p}\left|\beta_{j}\right|=\mathrm{RSS}+\lambda \sum_{j=1}^{p}\left|\beta_{j}\right|$
- In statistical parlance, the lasso uses an $\ell_{1}$ (pronounced
"ell 1") penalty instead of an $\ell_{2}$ penalty. The $\ell_{1}$ norm of a coefficient vector $\beta$ is given by $\|\beta\|_{1}=\sum\left|\beta_{j}\right|$
- As with ridge regression, the lasso shrinks the coefficient
estimates towards zero.
- However, in the case of the lasso, the $l_{1}$ penalty has the
efect of forcing some of the coefficient estimates to be
exactly equal to zero when the tuning parameter $\lambda$ is
sufficiently large.
- Hence, much like best subset selection, the lasso performs
variable selection.
- We say that the lasso yields sparse models | that is,
models that involve only a subset of the variables.
- As in ridge regression, selecting a good value of $\lambda$ for the
lasso is critical; cross-validation is again the method of
choice.

```python
n_lambdas = 200
lambdas = np.logspace(3, 0.5, n_lambdas)

coefs = []
scores = []
for lam in lambdas:
    lasso = Lasso(alpha=lam)
    lasso.fit(X_scaled, y)
    coefs.append(lasso.coef_)
    scores.append(lasso.score(X_scaled, y));

fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(15,4))

# get the four largest (in abs value) coefficient positions
ind = np.argpartition(np.abs(coefs[-1]), -4)[-4:]

# firt plot
ax1.plot(lambdas, coefs)
ax1.set_xscale('log')
ax1.set_xlabel('lambda')
ax1.set_ylabel('Standardized Coefficients')
ax1.legend(np.array(ax1.get_lines())[ind], X.columns[ind]);

# second plot
no_lasso_norm = linalg.norm(coefs[-1], ord=1)
norm_coefs = linalg.norm(coefs/no_lasso_norm, axis=1, ord=1)
ax2.plot(norm_coefs, coefs)
ax2.set_xlabel('Normalized Lasso coefficients')
ax2.set_ylabel('Standardized Coefficients')
ax2.legend(np.array(ax2.get_lines())[ind], X.columns[ind]);

# third
ax3.plot(lambdas, scores)
ax3.set_xscale('log')
ax3.set_xlabel('lambda')
ax3.set_ylabel('R^2')
ax3.set_title('R^2 as a function of regularization');

# the values for lambda are different here than in the text
# most likely the function that is minimized is a bit different in scikit-learn and R
# there's probably a factor of N or 2N (N=len(y)) difference.
```

Why is it that the lasso, unlike ridge regression, results in coefficient estimates that are exactly equal to zero?

One can show that the lasso and ridge regression coefficient estimates solve the problems
$$
\underset{\beta}{\operatorname{minimize}} \sum_{i=1}^{n}\left(y_{i}-\beta_{0}-\sum_{j=1}^{p} \beta_{j} x_{i j}\right)^{2} \quad \text { subject to } \quad \sum_{j=1}^{p}\left|\beta_{j}\right| \leq s
$$
and
$$
\underset{\beta}{\operatorname{minimize}} \sum_{i=1}^{n}\left(y_{i}-\beta_{0}-\sum_{j=1}^{p} \beta_{j} x_{i j}\right)^{2} \quad \text { subject to } \quad \sum_{j=1}^{p} \beta_{j}^{2} \leq s
$$
respectively.

#### Conclusions
- These two examples illustrate that neither ridge regression
nor the lasso will universally dominate the other.
- In general, one might expect the lasso to perform better
when the response is a function of only a relatively small
number of predictors.
- However, the number of predictors that is related to the
response is never known a priori for real data sets.
- A technique such as cross-validation can be used in order
to determine which approach is better on a particular data
set.

#### Selecting the Tuning Parameter for Ridge Regression and Lasso
- As for subset selection, for ridge regression and lasso we
require a method to determine which of the models under
consideration is best.
- That is, we require a method selecting a value for the
tuning parameter $\lambda$ or equivalently, the value of the
constraint s.
- Cross-validation provides a simple way to tackle this
problem. We choose a grid of $\lambda$ values, and compute the
cross-validation error rate for each value of $\lambda$.
- We then select the tuning parameter value for which the
cross-validation error is smallest.
- Finally, the model is re-fit using all of the available
observations and the selected value of the tuning
parameter.

```python
lambdas = np.logspace(0.5, -4, 100)
ridgeCV = RidgeCV(alphas=lambdas, store_cv_values=True)
ridgeCV.fit(X_scaled, y)
MSE_alphas = np.mean(ridgeCV.cv_values_, axis=0)

fig, ax = plt.subplots(1, 1, figsize=(5,4))
ax.plot(lambdas, MSE_alphas)
ax.axvline(ridgeCV.alpha_, color='k', linestyle='--')
ax.set_xscale('log')
ax.set_xlabel('lambda')
ax.set_ylabel('CV MSE');
```

#### Dimension Reduction Methods
- The methods that we have discussed so far in this chapter
have involved fitting linear regression models, via least
squares or a shrunken approach, using the original
predictors, $X_{0}, X_{1}, \ldots, X_{p}$.
- We now explore a class of approaches that transform the
predictors and then fit a least squares model using the
transformed variables. We will refer to these techniques as
dimension reduction methods.

#### Dimension Reduction Methods: details
- Let $Z_{1}, Z_{2}, \ldots, Z_{M}$ represent $M<p$ linear combinations of our original $p$ predictors. That is,
$Z_{m}=\sum_{j=1}^{p} \phi_{m j} X_{j}$
for some constants $\phi_{m 1}, \ldots, \phi_{m p}$
- We can then fit the linear regression model,
$$
y_{i}=\theta_{0}+\sum_{m=1}^{M} \theta_{m} z_{i m}+\epsilon_{i}, \quad i=1, \ldots, n
$$
using ordinary least squares.
- Note that in model (2), the regression coefficients are given by $\theta_{0}, \theta_{1}, \ldots, \theta_{M} .$ If the constants $\phi_{m 1}, \ldots, \phi_{m p}$ are chosen wisely, then such dimension reduction approaches can often outperform OLS regression

- Notice that from definition (1),
$\sum_{m=1}^{M} \theta_{m} z_{i m}=\sum_{m=1}^{M} \theta_{m} \sum_{j=1}^{p} \phi_{m j} x_{i j}=\sum_{j=1}^{p} \sum_{m=1}^{M} \theta_{m} \phi_{m j} x_{i j}=\sum_{j=1}^{p} \beta_{j} x_{i j}$
where $\beta_{j}=\sum_{m=1}^{M} \theta_{m} \phi_{m j}$
- Hence model (2) can be thought of as a special case of the original linear regression model.
- Dimension reduction serves to constrain the estimated $\beta_{j}$ coefficients, since now they must take the form (3)
- Can win in the bias-variance tradeoff.

#### Principal Components Regression
- Here we apply principal components analysis (PCA)
(discussed in Chapter 10 of the text) to define the linear
combinations of the predictors, for use in our regression.
- The first principal component is that (normalized) linear
combination of the variables with the largest variance.
- The second principal component has largest variance,
subject to being uncorrelated with the first.
- Hence with many correlated original variables, we replace
them with a small set of principal components that capture
their joint variation.

```python
components = list(range(1, X_scaled.shape[1]+1))
coefs = []
scores = []
for comp in components:
    pca = PCA(n_components=comp)
    X_pca_comp = pca.fit_transform(X_scaled)
    # use the first comp components
    #X_pca_comp = X_pca[:, 0:comp+1]
    linear = LinearRegression(fit_intercept=False)
    linear.fit(X_pca_comp, y)
    coefs.append(linear.coef_)
    scores.append(linear.score(X_pca_comp, y));

coefs = np.array(list(itertools.zip_longest(*coefs, fillvalue=0))).T

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10,4))

# get the four largest (in abs value) coefficient positions
ind = np.argpartition(np.abs(coefs[-1]), -4)[-4:]

# firt plot
ax1.plot(components, coefs)
ax1.set_xlabel('Number of PCA components')
ax1.set_ylabel('Standardized Coefficients')
ax1.legend(np.array(ax1.get_lines())[ind], X.columns[ind])

# third
ax2.plot(components, scores)
ax2.set_xlabel('Number of PCA components')
ax2.set_ylabel('R^2')
ax2.set_title('R^2 as a function of regularization');

```

#### Partial Least Squares
- PCR identifies linear combinations, or directions, that best
represent the predictors $X_{0}, X_{1}, \ldots, X_{p}$.
- These directions are identified in an unsupervised way, since
the response Y is not used to help determine the principal
component directions.
- That is, the response does not supervise the identification
of the principal components.
- Consequently, PCR suffers from a potentially serious
drawback: there is no guarantee that the directions that
best explain the predictors will also be the best directions
to use for predicting the response.
- Like PCR, PLS is a dimension reduction method, which
first identifies a new set of features $Z_{0}, Z_{1}, \ldots, Z_{p}$ that are
linear combinations of the original features, and then fits a
linear model via OLS using these M new features.
- But unlike PCR, PLS identifies these new features in a
supervised way - that is, it makes use of the response Y in
order to identify new features that not only approximate
the old features well, but also that are related to the
response.
- Roughly speaking, the PLS approach attempts to find
directions that help explain both the response and the
predictors.
- After standardizing the $p$ predictors, PLS computes the first direction $Z_{1}$ by setting each $\phi_{1 j}$ in (1) equal to the coefficient from the simple linear regression of $Y$ onto $X_{j}$
one can show that this coefficient is proportional to the correlation between $Y$ and $X_{j}$
- Hence, in computing $Z_{1}=\sum_{j=1}^{p} \phi_{1 j} X_{j},$ PLS places the highest weight on the variables that are most strongly related to the response.
- Subsequent directions are found by taking residuals and then repeating the above prescription.

### Summary
- Model selection methods are an essential tool for data
analysis, especially for big datasets involving many
predictors.
- Research into methods that give sparsity, such as the lasso
is an especially hot area.
- Later, we will return to sparsity in more detail, and will
describe related approaches such as the elastic net.

### End of Chapter 6

```python
# Exercise Problems
```


# Measuring qubits in roqoqo

This notebook is designed to demonstrate the use of measurements in roqoqo. We will look at several examples of measuring qubits, from single and multi-qubit registers. To learn about the effect of measurement, we will look at the state vectors before and after measurement.

```Rust
:dep roqoqo = "0.9.0"
:dep roqoqo-quest = "0.2.0"
:dep num-complex = "0.4"
```

```Rust
extern crate num_complex;
extern crate roqoqo;
extern crate roqoqo_quest;

use roqoqo::{Circuit, operations,prelude::*};
use roqoqo_quest::Backend;
use std::collections::HashMap;
```

## Measuring a single qubit

Here we first prepare the qubit in a superposition state,
\begin{equation}
|+ \rangle = \frac{1}{\sqrt{2}} \big ( |0 \rangle + |1 \rangle \big ).
\end{equation}
We look at the state after preparation, then do a measurement in the Z basis, and finally look again at the state after measurement.

We see that the state after measurement has been projected into the state either $|0>$ or $|1>$, consistently with the measurement outcome. Running this code many times should result in a random distribution of 'True' and 'False' outcomes.

```Rust
let mut state_init = Circuit::new();
state_init += operations::Hadamard::new(0); //prepare |+> state

// write state before measuring to readout register 'psi_in'
let mut read_input = Circuit::new();
read_input += operations::DefinitionComplex::new("psi_in".to_string(), 2, true);
read_input += operations::PragmaGetStateVector::new("psi_in".to_string(), Some(Circuit::new()));

// measure qubit in Z basis and write result to classical register 'M1'
let mut meas_circ = Circuit::new();
meas_circ += operations::DefinitionBit::new("M1".to_string(), 1, true);
meas_circ += operations::MeasureQubit::new(0, "M1".to_string(), 0);

// write state after measuring to readout register 'psi_out'
let mut read_output = Circuit::new();
read_output += operations::DefinitionComplex::new("psi_out".to_string(), 2, true);
read_output += operations::PragmaGetStateVector::new("psi_out".to_string(), Some(Circuit::new()));

// put each step of the circuit together
let circuit = state_init + read_input + meas_circ + read_output;

// run the circuit and collect output
let mut backend = roqoqo_quest::Backend::new(1);
let result_of_run = backend.run_circuit(&circuit);
let (result_bit_registers, _result_float_registers, result_complex_registers)
= result_of_run.unwrap();

println!("Input state:{:?}",result_complex_registers["psi_in"]);
println!("Measurement result:{:?}",result_bit_registers["M1"]);
println!("State after measurement:{:?} \n",result_complex_registers["psi_out"]);

```

    Input state:[[Complex { re: 0.7071067811865475, im: 0.0 }, Complex { re: 0.7071067811865475, im: 0.0 }]]
    Measurement result:[[true]]
    State after measurement:[[Complex { re: 0.0, im: 0.0 }, Complex { re: 0.9999999999999998, im: 0.0 }]]

## Measuring a single qubit in the X basis

Instead of measuring in the Z basis, we can measure the qubit in the X basis by performing a Hadamard operator before the measurement.

This time we see that the measurement result is always 'False', since we are measuring the $|+ \rangle$ state in the X basis, and it is an X eigenvector of the X operator.

```Rust
let number_of_qubits: usize = 3;

let mut state_init = Circuit::new();
state_init += operations::PauliX::new(1);
state_init += operations::Hadamard::new(0);
state_init += operations::CNOT::new(0, 1);
state_init += operations::CNOT::new(0, 2);
state_init += operations::SGate::new(0);

// write state before measuring to readout register 'psi_in'
let mut read_input = Circuit::new();
read_input += operations::DefinitionComplex::new("psi_in".to_string(), 2usize.pow(number_of_qubits as u32),  true);
read_input += operations::PragmaGetStateVector::new("psi_in".to_string(), Some(Circuit::new()));

// measure qubits in Z basis and write result to classical register 'M1M2M3'
let mut meas_circ = Circuit::new();
meas_circ += operations::DefinitionBit::new("M1M2M3".to_string(), 3, true);
meas_circ += operations::MeasureQubit::new(0, "M1M2M3".to_string(), 0);
meas_circ += operations::MeasureQubit::new(1, "M1M2M3".to_string(), 1);
meas_circ += operations::MeasureQubit::new(2, "M1M2M3".to_string(), 2);

// write state after measuring to readout register 'psi_out'
let mut read_output = Circuit::new();
read_output += operations::DefinitionComplex::new("psi_out".to_string(), 2usize.pow(number_of_qubits as u32),true );
read_output += operations::PragmaGetStateVector::new("psi_out".to_string(), Some(Circuit::new()));

let mut circuit = state_init + read_input + meas_circ + read_output;

// run the circuit and collect output
let backend = roqoqo_quest::Backend::new(number_of_qubits);
let result_of_run = backend.run_circuit(&circuit);
let (result_bit_registers,_result_float_registers,result_complex_registers)
= result_of_run.unwrap_or_default();

println!("Input state:{:?}",result_complex_registers["psi_in"]);
println!("Measurement result:{:?}",result_bit_registers["M1M2M3"]);
println!("State after measurement:{:?} \n",result_complex_registers["psi_out"]);
```

    Input state:[[Complex { re: 0.0, im: 0.0 }, Complex { re: 0.0, im: 0.0 }, Complex { re: 0.7071067811865475, im: 0.0 }, Complex { re: 0.0, im: 0.0 }, Complex { re: 0.0, im: 0.0 }, Complex { re: 0.0, im: 0.7071067811865475 }, Complex { re: 0.0, im: 0.0 }, Complex { re: 0.0, im: 0.0 }]]
    Measurement result:[[true, false, true]]
    State after measurement:[[Complex { re: 0.0, im: 0.0 }, Complex { re: 0.0, im: 0.0 }, Complex { re: 0.0, im: 0.0 }, Complex { re: 0.0, im: 0.0 }, Complex { re: 0.0, im: 0.0 }, Complex { re: 0.0, im: 1.0 }, Complex { re: 0.0, im: 0.0 }, Complex { re: 0.0, im: 0.0 }]]

## Measuring a multi-qubit register

Here we first prepare a multi-qubit register and demonstrate how it is possible to measure the entire register. As an example we prepare the multi-qubit register in the state,
\begin{equation}
|\psi \rangle = \frac{1}{\sqrt{2}} |010 \rangle + \frac{i}{\sqrt{2}} |101 \rangle.
\end{equation}

After preparation we read out the simulated state, before measurement. Next we measure each qubit of the state, and finally we readout out the post-measurement state.

```Rust
let number_of_qubits: usize = 3;

let mut state_init = Circuit::new();
state_init += operations::PauliX::new(1);
state_init += operations::Hadamard::new(0);
state_init += operations::CNOT::new(0, 1);
state_init += operations::CNOT::new(0, 2);
state_init += operations::SGate::new(0);

// write state before measuring to readout register 'psi_in'
let mut read_input = Circuit::new();
read_input += operations::DefinitionComplex::new("psi_in".to_string(), 2usize.pow(number_of_qubits as u32),  true);
read_input += operations::PragmaGetStateVector::new("psi_in".to_string(), Some(Circuit::new()));

// measure qubit in Z basis and write result to classical register 'M1'
let mut meas_circ = Circuit::new();
meas_circ += operations::DefinitionBit::new("M1".to_string(), 1, true);
meas_circ += operations::MeasureQubit::new(0, "M1".to_string(), 0);

// write state after measuring to readout register 'psi_out'
let mut read_output = Circuit::new();
read_output += operations::DefinitionComplex::new("psi_out".to_string(), 2usize.pow(number_of_qubits as u32),true );
read_output += operations::PragmaGetStateVector::new("psi_out".to_string(), Some(Circuit::new()));

let circuit = state_init + read_input + meas_circ + read_output;

// run the circuit and collect output
let backend = roqoqo_quest::Backend::new(number_of_qubits);
let result_of_run = backend.run_circuit(&circuit);
let (result_bit_registers,_result_float_registers,result_complex_registers)
= result_of_run.unwrap_or_default();

println!("Input state:{:?}",result_complex_registers["psi_in"]);
println!("Measurement result:{:?}",result_bit_registers["M1"]);
println!("State after measurement:{:?}",result_complex_registers["psi_out"]);
```

    Input state:[[Complex { re: 0.0, im: 0.0 }, Complex { re: 0.0, im: 0.0 }, Complex { re: 0.7071067811865475, im: 0.0 }, Complex { re: 0.0, im: 0.0 }, Complex { re: 0.0, im: 0.0 }, Complex { re: 0.0, im: 0.7071067811865475 }, Complex { re: 0.0, im: 0.0 }, Complex { re: 0.0, im: 0.0 }]]
    Measurement result:[[true]]
    State after measurement:[[Complex { re: 0.0, im: 0.0 }, Complex { re: 0.0, im: 0.0 }, Complex { re: 0.0, im: 0.0 }, Complex { re: 0.0, im: 0.0 }, Complex { re: 0.0, im: 0.0 }, Complex { re: 0.0, im: 0.9999999999999998 }, Complex { re: 0.0, im: 0.0 }, Complex { re: 0.0, im: 0.0 }]]


```python
%matplotlib inline
```

## ``torch.autograd`` による自動微分

ニューラルネットワークを学習する際、最もよく使われるアルゴリズムは**back propagation**です。
このアルゴリズムでは、パラメータ（モデルの重み）はこのアルゴリズムでは、パラメータ（モデルの重み）は、与えられたパラメータに対する損失関数の**勾配**に応じて調整されます。
**gradient**に応じてパラメータ（モデルの重み）を調整します。

この勾配を計算するために、PyTorchは`torch.autogradat`という微分エンジンを内蔵しています。
このエンジンは、任意の勾配の自動計算をサポートします。

入力 `x`、パラメータ `w` および `b` と、何らかの損失関数を持つ最も単純な1層のニューラルネットワークを考えてみましょう。
これは次のようにPyTorchで定義できます。

```python
import torch

x = torch.ones(5)  # input tensor
y = torch.zeros(3)  # expected output
w = torch.randn(5, 3, requires_grad=True)
b = torch.randn(3, requires_grad=True)
z = torch.matmul(x, w) + b
loss = torch.nn.functional.binary_cross_entropy_with_logits(z, y)
```

## テンソルと関数と計算グラフ

このコードは、以下の**計算グラフ**を定義しています。

このネットワークでは、`w`と`b`は**パラメータ**であり、これを最適化する必要があります。
そのため、これらの変数に対する損失関数の勾配を計算する必要があります。
これを行うためにはテンソルの`requires_grad`プロパティを設定します。

> **Note:** `requires_grad` の値は，テンソルの作成時に設定することもできますし、後で `x.requirees_grad_(True)` メソッドを使って設定することもできます。

計算グラフを構築するためにテンソルに適用する関数は実際にはクラス `Function` のオブジェクトです。
このオブジェクトは、関数を *forward* 方向に計算する方法と、その微分を *backward* 方向に微分計算する方法を知っています。
逆伝搬関数への参照は、テンソルの `grad_fn` プロパティに格納されています。
`Function`の詳細については[ドキュメント](https://pytorch.org/docs/stable/autograd.html#function)を参照してください。

```python
print('Gradient function for z =', z.grad_fn)
print('Gradient function for loss =', loss.grad_fn)
```

    Gradient function for z = <AddBackward0 object at 0x12bc2c8e0>
    Gradient function for loss = <BinaryCrossEntropyWithLogitsBackward object at 0x12bc2c5e0>

## 勾配計算

ニューラルネットワークのパラメータの重みを最適化するためにはパラメータに対する損失関数の導関数を計算する必要があります。
具体的にはある固定値の`x`と`y`の下で$\frac{\partial loss}{\partial w}$と$\frac{\partial loss}{\partial b}$が必要です。
これらの導関数を計算するために、`loss.backward()`を呼び出し、`w.grad`と`b.grad`から値を取得します。

```python
loss.backward()
print(w.grad)
print(b.grad)
```

    tensor([[0.0424, 0.0851, 0.3282],
            [0.0424, 0.0851, 0.3282],
            [0.0424, 0.0851, 0.3282],
            [0.0424, 0.0851, 0.3282],
            [0.0424, 0.0851, 0.3282]])
    tensor([0.0424, 0.0851, 0.3282])

> Note: 計算グラフのリーフノードのうち`requires_grad` プロパティが `True` に設定されているノードに対してのみ`grad` プロパティを取得することができます。
グラフ内の他のすべてのノードでは勾配を利用することはできません。さらにパフォーマンス上の理由から`backward`を用いた勾配計算は1つのグラフに対して1回しか行うことができません。
もし同じグラフに対して複数回の`backward`呼び出しを行う必要がある場合には`retain_graph=True`を`backward`呼び出しに渡す必要があります。

## 勾配トラッキングの無効化

デフォルトでは`requires_grad=True`となっているすべてのテンソルはその計算履歴を追跡し、勾配計算をサポートします。
しかし勾配計算が必要ない場合もあります。例えば、モデルを学習した後そのモデルをある入力データに適用したい場合、つまり例えばモデルをトレーニングした後、入力データに適用したい場合、つまり、ネットワークを介して*forward* 計算だけを行いたい場合などです。
追跡計算を止めるには計算コードを`torch.no_grad()`ブロックで囲むことで計算を停止することができます。

```python
z = torch.matmul(x, w) + b
print(z.requires_grad)

with torch.no_grad():
    z = torch.matmul(x, w) + b
print(z.requires_grad)
```

    True
    False

同じ結果を得るためのもう一つの方法は，テンソルに`detach()`メソッドを使うことです。

```python
z = torch.matmul(x, w) + b
z_det = z.detach()
print(z_det.requires_grad)
```

    False

勾配トラッキングを無効にしたい理由はいくつかあります。

  - ニューラルネットワークのいくつかのパラメータを **凍結パラメータ** にする。これは[学習済みネットワークの微調整](https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html)に一般的なシナリオです。
  - フォワードパスのみを行っている場合に**計算を高速化**する。なぜならばテンソルの計算は勾配を追跡しない方が効率的だからです。

計算機上のグラフの詳細
----------------------------
概念的には、autogradはデータ（テンソル）と、実行されたすべての演算（およびその結果としての新しいテンソル）を[関数](https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function)からなる有向非巡回グラフ（DAG）に記録します。
このDAGでは、葉が入力テンソル、根が出力テンソルです。このグラフを根から葉へとたどっていくと連鎖律を使って自動的に勾配を計算することができます。

フォワードパスでは、autogradは2つのことを同時に行います。

- 要求された演算を実行して結果のテンソルを計算する
- DAGの中で演算の*勾配関数*を維持する。

バックワードパスは、DAGのルートで`.backward()`が呼ばれたときに起動します。
`autograd` は次に

- 各 `.grad_fn` から勾配を計算します。
- それらをそれぞれのテンソルの `.grad` 属性に蓄積する。
- 連鎖律を利用して、リーフテンソルにまで伝搬します。

**PyTorchではDAGは動的なものです**。

重要なことは、グラフがゼロから再作成されるということです。
`.backward()`を呼び出すたびに、autogradは新しいグラフの作成を開始します。
これこそが、モデルで制御フロー文を使用できる理由です。
必要に応じて、反復ごとに形状、サイズ、操作を変更することができます。

## Optional reading: ヤコビアンとベクトル勾配の積

多くの場合スカラーの損失関数があり、いくつかのパラメータに対する勾配を計算する必要があります。
しかし，場合によっては出力関数が任意のベクトルである場合があります。
このような場合、PyTorchは実際の勾配ではなく、いわゆる **ヤコビアン積** を計算することができます。

ベクトル関数 $\vec{y}=f(\vec{x})$ に対してここで、$\vec{x}=\langle x_1,\dots,x_n\rangle$ と $\vec{y}=\langle y_1,\dots,y_m\rangle$ とすると、その勾配は
次の式で与えられます *ヤコビアン行列*:

$\begin{align}\begin{align}J=\left(\begin{array}{ccc}
      \frac{\partial y_{1}}{\partial x_{1}} & \cdots & \frac{\partial y_{1}}{\partial x_{n}}\\
      \vdots & \ddots & \vdots\\
      \frac{\partial y_{m}}{\partial x_{1}} & \cdots & \frac{\partial y_{m}}{\partial x_{n}}
      \end{array}\right)\end{align}\end{align}$

ヤコビアン行列を計算する代わりに、PyTorchではヤコビアン積 **$v^T\cdot J$** を計算します。
これは引数に $v^T\cdot J$ を指定して `backward` を呼び出すことで実現できます。
$v$ のサイズは元のテンソルのサイズと同じでなければなりません．
積を計算したい元のテンソルのサイズと同じである必要があります。

```python
inp = torch.eye(5, requires_grad=True)
out = (inp + 1).pow(2)
out.backward(torch.ones_like(inp), retain_graph=True)
print("First call\n", inp.grad)
out.backward(torch.ones_like(inp), retain_graph=True)
print("\nSecond call\n", inp.grad)
inp.grad.zero_()
out.backward(torch.ones_like(inp), retain_graph=True)
print("\nCall after zeroing gradients\n", inp.grad)
```

    First call
     tensor([[4., 2., 2., 2., 2.],
            [2., 4., 2., 2., 2.],
            [2., 2., 4., 2., 2.],
            [2., 2., 2., 4., 2.],
            [2., 2., 2., 2., 4.]])

    Second call
     tensor([[8., 4., 4., 4., 4.],
            [4., 8., 4., 4., 4.],
            [4., 4., 8., 4., 4.],
            [4., 4., 4., 8., 4.],
            [4., 4., 4., 4., 8.]])

    Call after zeroing gradients
     tensor([[4., 2., 2., 2., 2.],
            [2., 4., 2., 2., 2.],
            [2., 2., 4., 2., 2.],
            [2., 2., 2., 4., 2.],
            [2., 2., 2., 2., 4.]])

同じ引数で2回目に `backward` を呼び出すと、勾配の値が異なることに注意してください。
これは以下の理由によるものです。
`backward`伝搬を行う際、PyTorchは**グラデーションを蓄積**します。すなわち、計算グラフの全てのリーフノードの `grad` プロパティに計算された勾配の値が追加されます。
もしも 適切な勾配を計算する必要があるならばプロパティをゼロにしておく必要があります。
実際のトレーニングでは、*optimizer* がこれを行う手助けをしてくれます。

> **Note:** 以前はパラメータなしで `backward()` 関数を呼び出していました。
これは`backward(torch.tensor(1.0))`を呼んでいるのと同じで、ニューラルネットワーク学習時の損失のようなスカラー値の関数の場合、勾配を計算するのに便利な方法です。

# 自分の知識を確認する

1. torch.autogradエンジンの目的は、以下の通りです。

- モデルの精度を自動的に評価
  > 正しくは、モデルの最適化時に自動的に勾配を計算します。
- モデルの内部レイヤー構造を自動的に最適化
  > 正しくは、モデルの最適化時に自動的に勾配を計算します。
- モデルの構築に使用するデータセットを自動的に最適化する
  > 正しくは、モデルの最適化時に自動的に勾配を計算します。
- モデルの最適化における勾配の自動計算
  > 正解！


# [Polytropic TOV](https://en.wikipedia.org/wiki/Tolman%E2%80%93Oppenheimer%E2%80%93Volkoff_equation) Initial Data

## Authors: Phil Chang & Zach Etienne
### Formatting improvements courtesy Brandon Clark

## This module sets up initial data for a [TOV](https://en.wikipedia.org/wiki/Tolman%E2%80%93Oppenheimer%E2%80%93Volkoff_equation) star in *spherical, isotropic coordinates*

**This module has been validated to exhibit convergence to zero of the Hamiltonian constraint violation at the expected order to the exact solution (see [start-to-finish TOV module](Tutorial-Start_to_Finish-BSSNCurvilinear-Setting_up_TOV_initial_data.ipynb) for full test). Note that convergence at the surface of the star is lower order due to the sharp drop to zero in $T^{\mu\nu}$.**

### NRPy+ Source Code for this module: [TOV/TOV_Solver.py](../edit/TOV/TOV_Solver.py)

# Table of Contents
$$\label{toc}$$

The module is organized as follows:

1. [Step 1](#initializenrpy): Initialize core Python/NRPy+ modules
1. [Step 2](#tov): The TOV Equations
1. [Step 3](#code_validation): Code Validation against TOV/TOV_Solver NRPy+ module
1. [Step 4](#latex_pdf_output): Output this module to $\LaTeX$-formatted PDF

# Step 1: Initialize core Python/NRPy+ modules \[Back to [top](#toc)\]
$$\label{initializenrpy}$$

```python
# Step 1: Import needed Python/NRPy+ modules
import numpy as np
import scipy.integrate as si
import math
import sys
```

# Step 2: The TOV equations \[Back to [top](#toc)\]
$$\label{tov}$$

The [TOV line element](https://en.wikipedia.org/wiki/Tolman%E2%80%93Oppenheimer%E2%80%93Volkoff_equation) in terms of the *Schwarzschild coordinate* $r$ is written (in the $-+++$ form):
$$
ds^2 = - c^2 e^\nu dt^2 + \left(1 - \frac{2Gm}{rc^2}\right)^{-1} dr^2 + r^2 d\Omega^2,
$$
where $m(r)$ is the mass-energy enclosed at a given $r$, and is equal to the total star's mass outside the stellar radius $r=R$.

In terms of the *isotropic coordinate* $\bar{r}$ with $G=c=1$ (i.e., the coordinate system and units we'd prefer to use), the ($-+++$ form) line element is written:
$$
ds^2 = - e^{\nu} dt^2 + e^{4\phi} \left(d\bar{r}^2 + \bar{r}^2 d\Omega^2\right),
$$
where $\phi$ here is the *conformal factor*.

Setting components of the above line element equal to one another, we get (in $G=c=1$ units):

\begin{align}
r^2 &= e^{4\phi} \bar{r}^2 \implies e^{4\phi} = \frac{r^2}{\bar{r}^2} \\
\left(1 - \frac{2m}{r}\right)^{-1} dr^2 &= e^{4\phi} d\bar{r}^2 \\
\implies \frac{d\bar{r}(r)}{dr} &= \left(1 - \frac{2m}{r} \right)^{-1/2} \frac{\bar{r}(r)}{r}.
\end{align}

The TOV equations provide radial ODEs for the pressure and $\nu$ (from [the Wikipedia article on the TOV solution](https://en.wikipedia.org/wiki/Tolman%E2%80%93Oppenheimer%E2%80%93Volkoff_equation)):

\begin{align}
\frac{dP}{dr} &= - \frac{1}{r} \left( \frac{\rho + P}{2} \right) \left(\frac{2 m}{r} + 8 \pi r^2 P\right) \left(1 - \frac{2 m}{r}\right)^{-1} \\
\frac{d \nu}{d r} &= \frac{1}{r}\left(1 - \frac{2 m}{r}\right)^{-1} \left(\frac{2 m}{r} + 8 \pi r^2 P\right) \\
\end{align}

Assuming a polytropic equation of state, which relates the pressure $P$ to the baryonic rest-mass density $\rho_B$,

$$
P(\rho_B) = K \rho_B^\Gamma,
$$
the specific internal energy will be given by
$$
\epsilon = \frac{P}{\rho_B (\Gamma - 1)},
$$

so the total mass-energy density $\rho$ is given by
$$
\rho = \rho_B (1 + \epsilon).
$$

Given this, the mass-energy $m(r)$ density is the solution to the ODE:
$$
\frac{m(r)}{dr} = 4\pi r^2 \rho(r)
$$

Thus the full set of ODEs that need to be solved is given by

\begin{align}
\frac{dP}{dr} &= - \frac{1}{r} \left( \frac{\rho + P}{2} \right) \left(\frac{2 m}{r} + 8 \pi r^2 P\right) \left(1 - \frac{2 m}{r}\right)^{-1} \\
\frac{d \nu}{d r} &= \frac{1}{r}\left(1 - \frac{2 m}{r}\right)^{-1} \left(\frac{2 m}{r} + 8 \pi r^2 P\right) \\
\frac{m(r)}{dr} &= 4\pi r^2 \rho(r) \\
\frac{d\bar{r}(r)}{dr} &= \left(1 - \frac{2m}{r} \right)^{-1/2} \frac{\bar{r}(r)}{r}.
\end{align}

The following code solves these equations, and was largely written by Phil Chang.

```python
# Step 2: The TOV equations

## TOV SOLVER FOR SIMPLE POLYTROPES.
## Authors: Phil Chang, Zachariah B. Etienne

# Full documentation for this module may be found in the NRPy+ tutorial Jupyter notebook:
#  Tutorial-Start_to_Finish-BSSNCurvilinear-Setting_up_TOV_initial_data.ipynb

# Inputs:
# * Output data file name
# * rho_baryon_central, the central density of the TOV star.
# * n, the polytropic equation of state index. n=1 models cold, degenerate neutron star matter.
# * K_Polytrope, the polytropic constant.
# * Verbose output toggle (default = True)

# Output: An initial data file (default file name = "outputTOVpolytrope.txt") that well
#         samples the (spherically symmetric) solution both inside and outside the star.
#         It is up to the initial data module to perform the 1D interpolation to generate
#         the solution at arbitrary radius. The file has the following columns:
# Column 1: Schwarzschild radius
# Column 2: rho(r), *total* mass-energy density (as opposed to baryonic rest-mass density)
# Column 3: P(r), Pressure
# Column 4: m(r), mass enclosed
# Column 5: e^{nu(r)}, g_{tt}(r)
# Column 6: e^{4 phi(r)}, conformal factor g_{rr}(r)
# Column 7: rbar(r), Isotropic radius

# rbar refers to the isotropic radius, and
# R_Schw refers to the Schwarzschild radius

n = 1.
rho_baryon_central = 0.129285

P0 = 1. # ZACH NOTES: CHANGED FROM 100.
gamma = 1. + 1./n
gam1 = gamma - 1.

def TOV_pressure( rho_baryon) :
    return P0*rho_baryon**gamma

def TOV_rhs(r_Schw, y) :
# In \tilde units
#
    P    = y[0]
    m    = y[1]
    nu   = y[2]
    rbar = y[3]

    dPdrSchw    = 0.
    drbardrSchw = 0.

    rho_baryon = (P/P0)**(1./gamma)
    rho = rho_baryon + P/gam1 # rho is the *total* mass-energy density!
    if( r_Schw < 1e-4 or m <= 0.):
        m = 4*math.pi/3. * rho*r_Schw**3
        dPdrSchw = -(rho + P)*(4.*math.pi/3.*r_Schw*rho + 4.*math.pi*r_Schw*P)/(1.-8.*math.pi*rho*r_Schw*r_Schw)
        drbardrSchw = 1./(1. - 8.*math.pi*rho*r_Schw*r_Schw)**0.5
    else:
        dPdrSchw = -(rho + P)*(m + 4.*math.pi*r_Schw**3*P)/(r_Schw*r_Schw*(1.-2.*m/r_Schw))
        drbardrSchw = 1./(1. - 2.*m/r_Schw)**0.5*rbar/r_Schw

    dmdrSchw  =  4.*math.pi*r_Schw*r_Schw*rho
    dnudrSchw = -2./(P + rho)*dPdrSchw
    return [dPdrSchw, dmdrSchw, dnudrSchw, drbardrSchw]

def integrateStar( P, dumpData = False):
    integrator = si.ode(TOV_rhs).set_integrator('dop853')
    y0 = [P, 0., 0., 0.]
    integrator.set_initial_value(y0,0.)
    dr_Schw = 1e-5
    P = y0[0]

    PArr      = []
    r_SchwArr = []
    mArr      = []
    nuArr     = []
    rbarArr   = []

    r_Schw = 0.

    while integrator.successful() and P > 1e-9*y0[0] :
        P, m, nu, rbar = integrator.integrate(r_Schw + dr_Schw)
        r_Schw = integrator.t

        dPdrSchw, dmdrSchw, dnudrSchw, drbardrSchw = TOV_rhs( r_Schw+dr_Schw, [P,m,nu,rbar])
        dr_Schw = 0.1*min(abs(P/dPdrSchw), abs(m/dmdrSchw))
        dr_Schw = min(dr_Schw, 1e-2)
        PArr.append(P)
        r_SchwArr.append(r_Schw)
        mArr.append(m)
        nuArr.append(nu)
        rbarArr.append(rbar)

    M = mArr[-1]
    R_Schw = r_SchwArr[-1]

    # Apply integration constant to ensure rbar is continuous across TOV surface
    for ii in range(len(rbarArr)):
        rbarArr[ii] *= 0.5*(np.sqrt(R_Schw*(R_Schw - 2.0*M)) + R_Schw - M) / rbarArr[-1]

    nuArr_np = np.array(nuArr)
    # Rescale solution to nu so that it satisfies BC: exp(nu(R))=exp(nutilde-nu(r=R)) * (1 - 2m(R)/R)
    #   Thus, nu(R) = (nutilde - nu(r=R)) + log(1 - 2*m(R)/R)
    nuArr_np = nuArr_np - nuArr_np[-1] + math.log(1.-2.*mArr[-1]/r_SchwArr[-1])

    r_SchwArrExtend_np = 10.**(np.arange(0.01,5.0,0.01))*r_SchwArr[-1]

    r_SchwArr.extend(r_SchwArrExtend_np)
    mArr.extend(r_SchwArrExtend_np*0. + M)
    PArr.extend(r_SchwArrExtend_np*0.)
    exp2phiArr_np = np.append( np.exp(nuArr_np), 1. - 2.*M/r_SchwArrExtend_np)
    nuArr.extend(np.log(1. - 2.*M/r_SchwArrExtend_np))
    rbarArr.extend( 0.5*(np.sqrt(r_SchwArrExtend_np**2 - 2.*M*r_SchwArrExtend_np) + r_SchwArrExtend_np - M) )

    # Appending to a Python array does what one would reasonably expect.
    #   Appending to a numpy array allocates space for a new array with size+1,
    #   then copies the data over... over and over... super inefficient.
    r_SchwArr_np          = np.array(r_SchwArr)
    PArr_np               = np.array(PArr)
    rho_baryonArr_np      = (PArr_np/P0)**(1./gamma)
    mArr_np               = np.array(mArr)
    rbarArr_np            = np.array(rbarArr)
    confFactor_exp4phi_np = (r_SchwArr_np/rbarArr_np)**2

    # Compute the *total* mass-energy density (as opposed to the *baryonic* mass density)
    rhoArr_np = []
    for i in range(len(rho_baryonArr_np)):
        rhoArr_np.append(rho_baryonArr_np[i] + PArr_np[i]/(gamma - 1.))

    print(len(r_SchwArr_np),len(rhoArr_np),len(PArr_np),len(mArr_np),len(exp2phiArr_np))
    # Special thanks to Leonardo Werneck for pointing out this issue with zip()
    if sys.version_info[0] < 3:
        np.savetxt("outputTOVpolytrope.txt", zip(r_SchwArr_np,rhoArr_np,PArr_np,mArr_np,exp2phiArr_np,confFactor_exp4phi_np,rbarArr_np),
                   fmt="%.15e")
    else:
        np.savetxt("outputTOVpolytrope.txt", list(zip(r_SchwArr_np,rhoArr_np,PArr_np,mArr_np,exp2phiArr_np,confFactor_exp4phi_np,rbarArr_np)),
                   fmt="%.15e")

    return R_Schw, M

R_Schw_TOV,M_TOV = integrateStar(TOV_pressure(rho_baryon_central), True)
print("Just generated a TOV star with R_Schw = "+str(R_Schw_TOV)+" , M = "+str(M_TOV)+" , M/R_Schw = "+str(M_TOV/R_Schw_TOV)+" .")
```

    1051 1051 1051 1051 1051
    Just generated a TOV star with R_Schw = 0.9565681425227097 , M = 0.14050303285288188 , M/R_Schw = 0.1468824086931645 .

# Step 3: Code Validation against TOV.TOV_Solver NRPy+ module \[Back to [top](#toc)\]
$$\label{code_validation}$$

Here, as a code validation check, we verify agreement in the SymPy expressions for these TOV initial data between

1. this tutorial and
2. the NRPy+ [TOV/TOV_Solver](../edit/TOV/TOV_Solver.py) module.

```python
# Step 3: Code Validation against TOV.TOV_Solver module

import filecmp

import TOV.TOV_Solver as TOV
TOV.TOV_Solver("outputTOVpolytrope-validation.txt",rho_baryon_central=0.129285, n_Polytrope=1.0, K_Polytrope=1.0,
               verbose = False)

if filecmp.cmp('outputTOVpolytrope.txt',
               'outputTOVpolytrope-validation.txt') == False:
    print("ERROR: TOV initial data test FAILED!")
    exit(1)
else:
    print("TOV initial data test PASSED.")
```

    TOV initial data test PASSED.

# Step 4: Output this module to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-ADM_Initial_Data-TOV](Tutorial-ADM_Initial_Data-TOV.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
!jupyter nbconvert --to latex --template latex_nrpy_style.tplx Tutorial-ADM_Initial_Data-TOV.ipynb
!pdflatex -interaction=batchmode Tutorial-ADM_Initial_Data-TOV.tex
!pdflatex -interaction=batchmode Tutorial-ADM_Initial_Data-TOV.tex
!pdflatex -interaction=batchmode Tutorial-ADM_Initial_Data-TOV.tex
!rm -f Tut*.out Tut*.aux Tut*.log
```

    [NbConvertApp] Converting notebook Tutorial-ADM_Initial_Data-TOV.ipynb to latex
    [NbConvertApp] Writing 47047 bytes to Tutorial-ADM_Initial_Data-TOV.tex
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode


## The argument in "Why most published research findings are false"

I spent some time trying to understand the argument in this paper:

* Ioannidis, John PA. 2005. “Why most published research findings are false.” *PLoS medicine* 2 (8): e124.

These papers were useful for understanding the argument:

* Goodman, Steven, and Sander Greenland. 2007. “Assessing the unreliability of the medical literature: a response to ‘why most published research findings are false.’” *Johns Hopkins University, Dept. of Biostatistics Working Papers*.
* Kass, Robert E., and Adrian E. Raftery. 1995. “Bayes factors.” *Journal of the american statistical association* 90 (430): 773–795.
* Wacholder, Sholom, Stephen Chanock, Montserrat Garcia-Closas, Nathaniel Rothman, and others. 2004. “Assessing the probability that a positive report is false: an approach for molecular epidemiology studies.” *Journal of the National Cancer Institute* 96 (6): 434–442.

## "The practice of science is profoundly broken".  Discuss? - no - model and test!

The point that Ioannnides makes is:

We know that the scientific process is flawed in a variety of ways.  We assume that these flaws do not have a large effect on the outcome.  But, if we model some of the flaws, we see that their effect can be catastrophic, in the sense that a large proportion of scientific findings are likely to be wrong.

We scientists commit ourselves to rational thinking.  In this case, rational thinking is asking, "how likely is it that we are getting the answers wrong"?.  We have to ask this question in a rational way.  This is what Ioannidis sets out to do in this paper.

## Different ways of phrasing the argument

The basis of Ioannidis' argument comes from [Wacholder et al 2004](http://jnci.oxfordjournals.org/content/96/6/434.long) (see appendix table 1).  [Goodman and Greenland 2007](http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0040168) explain Ioannidis in terms of Bayes theorem.

Both Ioannidis and Goodman & Greenland use odds ratios rather than probability values in their exposition.  I found it easier to think in terms of probabilities.

## Some terms

We've done an experiment, and we have conducted a statistical test:

* $H_A$ - alternative hypothesis
* $H_0$ - null hypothesis
* $\alpha$ : false positive rate - probability for test to reject $H_0$ when $H_0$ is true ($H_A$ is false)
* $\beta$ : false negative rate - probability for test to accept $H_0$ when $H_A$ is true ($H_0$ is false)
* $1 - \beta$ : power - probability we will reject $H_0$ if $H_A$ is true ($H_0$ is false)

Let's say that the test can either be "significant" (test gives $p \le \alpha$) or "not significant" ($p > \alpha$).  Denote "test is significant" by $T_S$, "test not significant" by $T_N$.

Before we did the experiment there were two possibilities - $H_A$ is true, or $H_0$ is true.  After we have four possibilities:

* $H_A \land T_S$ : $H_A$ is true, test is significant;
* $H_A \land T_N$ : $H_A$ is true, test is not significant;
* $H_0 \land T_S$ : $H_0$ is true ($H_A$ is false) - test is significant;
* $H_0 \land T_N$ : $H_0$ is true ($H_A$ is false) - test is not significant.

## What does a "significant" statistical test result tell us?

In this section we work up slowly to Ioannidis table 1.

First we need to load functions for symbolic mathematics from the Sympy library:

```
from sympy import symbols, Eq, solve, simplify, lambdify, init_printing, latex
init_printing(use_latex=True, order='old')
```

Define a function to display the probability tables:

```
from IPython.display import HTML
# Code to make HTML for a probability table
def association_table(assocs, title):
    latexed = {'title': title}
    for key, value in assocs.items():
        latexed[key] = latex(value)
    latexed['s_total'] = latex(assocs['t_s'] + assocs['f_s'])
    latexed['ns_total'] = latex(assocs['t_ns'] + assocs['f_ns'])
    return """<h3>{title}</h3>
              <TABLE><TR><TH><TH>$T_S$<TH>$T_N$
              <TR><TH>$H_A$<TD>${t_s}$<TD>${t_ns}$
              <TR><TH>$H_0$<TD>${f_s}$<TD>${f_ns}$
              <TR><TH>Total<TD>${s_total}$<TD>${ns_total}$
              </TABLE>""".format(**latexed)
```

If we do not take any prior probabilities into account, then we have the following probabilities:

```
alpha, beta = symbols(r'\alpha, \beta')
assoc = dict(t_s = 1 - beta, # H_A true, test significant = true positives
             t_ns = beta, # true, not significant = false negatives
             f_s = alpha, # false, significant = false positives
             f_ns = 1 - alpha) # false, not sigificant = true negatives
HTML(association_table(assoc, 'Not considering prior'))
```

Not considering prior

$T_S$$T_N$

$H_A$$1 - \beta$$\beta$

$H_0$$\alpha$$1 - \alpha$

Total$1 + \alpha - \beta$$1 + \beta - \alpha$

$\newcommand{Frac}[2]{\frac{\displaystyle #1}{\displaystyle #2}}$

Some new terms:

* $Pr(H_A)$ - prior probability of $H_A$ - probability of $H_A$ before the experiment was conducted.
* $Pr(H_0)$ - prior probability of $H_0$ = $1 - Pr(H_A)$ - probability of null hypothesis before experiment conducted

We are interested in updating the probability of $H_A$ and $H_0$ as a result of a test on some collected data.  This updated probability is $Pr(H_A | T)$ - the probability of $H_A$ given the test  result $T$. $Pr(H_A | T)$ is called the *posterior* probability because it is the probability after the test result.

The test result $T$ is assumed to have arisen under either $H_A$ or $H_0$.

$Pr(T) = Pr(T | H_A) Pr(H_A) + Pr(T | H_0) Pr(H_0)$

Also the probability of a *signficant* result of the test $T_S$ is from the same formula:

$Pr(T_S) = Pr(T_S | H_A) Pr(H_A) + Pr(T_S | H_0) Pr(H_0)$

(From Kass & Rafferty 1995)

Remembering [Bayes theorem](http://en.wikipedia.org/wiki/Bayes'_theorem#Derivation):

$P(A | B) = \Frac{P(B | A) P(A)}{P(B)}$

Bayes theorem gives:

$P(H_A | T) = \Frac{Pr(T | H_A) Pr(H_A)}{Pr(T)} = \Frac{Pr(T | H_A) Pr(H_A)}{Pr(T | H_A) Pr(H_A) + Pr(T | H_0) Pr(H_0)}$

Consider only the test result $T_S$ (the test is signficant).  What is the posterior probability of $H_A$ given that the test is signficant?

$P(H_A | T_S) = \Frac{Pr(T_S | H_A) Pr(H_A)}{Pr(T_S | H_A) Pr(H_A) + Pr(T_S | H_0) Pr(H_0)}$

We have $Pr(T_S | H_A)$, $Pr(T_S | H_0)$ from the first column of the table above. Substituting into the equation:

$P(H_A | T_S) = \Frac{(1 - \beta) Pr(H_A)}{(1 - \beta) Pr(H_A) + \alpha Pr(H_0)}$

To make this a little less cluttered, define:

$\pi := Pr(H_A)$

So

$1 - \pi = Pr(H_0)$

and:

$P(H_A | T_S) = \Frac{(1 - \beta) \pi}{(1 - \beta) \pi + \alpha (1 - \pi)}$

Let's put that formula into Sympy for later:

```
pi = symbols(r'\pi')
post_prob = (1 - beta) * pi / ((1 - beta) * pi + alpha * (1 - pi))
post_prob
```

A table shows the new probabilities that take the prior into account:

```
assoc = dict(t_s = pi * (1 - beta),
             t_ns = pi * beta,
             f_s = (1 - pi) * alpha,
             f_ns = (1 - pi) * (1 - alpha))
HTML(association_table(assoc, r'Considering prior $\pi := Pr(H_A)$'))
```

Considering prior $\pi := Pr(H_A)$

$T_S$$T_N$

$H_A$$\pi \left(1 - \beta\right)$$\beta \pi$

$H_0$$\alpha \left(1 - \pi\right)$$\left(1 - \alpha\right) \left(1 - \pi\right)$

Total$\alpha \left(1 - \pi\right) + \pi \left(1 - \beta\right)$$\beta \pi + \left(1 - \alpha\right) \left(1 - \pi\right)$

This table is equivalent to Ioannidis table 1. The first column of the table gives the probabilities in the case we're interested in, of $T_S$.  The posterior probability is the first row, first column - $Pr(T_S | H_A)$, divided by the total row, first column - $Pr(T_S)$.

Ioannidis uses "positive predictive value" (PPV) for the posterior probability $P(H_A | T_S)$.  Goodman and Greenland complain, reasonably enough, that "positive predictive value" is a confusing new term for a standard concept.

Ioannidis also prefers his equations in terms of $R$ - the *prior odds ratio*.  $R := \Frac{Pr(H_A)}{Pr(H_0)}$. Also (from $\pi := Pr(H_A)$ and $Pr(H_0) = 1 - Pr(H_A)$): $R = \Frac{\pi}{1 - \pi}$.

Ioannidis' formula for PPV is $\Frac{(1 - \beta) R}{R - \beta R + \alpha}$.  This is the same as our formula above, only rephrased in terms of $R$:

```
R = pi / (1 - pi)
ppv = (1 - beta) * R / (R - beta * R + alpha)
# Is this the same as our formula above?
simplify(ppv - post_prob) == 0
```

    True

The posterior probability is our estimate of whether $H_A$ is true, given our prior knowledge $Pr(H_A) = \pi$ combined with the new information from the test result.

## What is a finding that is likely to be true?

A finding that is likely to be true is one for which the posterior probability $Pr(H_A | T_S) > 0.5$.  That is, the likelihood of the tested hypothesis, given the reported significant test result, is greater than $0.5$

## Whether a finding is likely to be true depends on the power of the experiment

Assume that $\alpha = 0.05$ (standard significance threshold for null hypothesis test).
First we load the plotting and numerical libraries.

```
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
```

Let's have a look at the posterior probability as a function of power and prior probability:

```
# Make symbolic expression into a function we can evaluate
post_prob_func = lambdify((alpha, beta, pi), post_prob)
# Make set of pi values for x axis of plot
pi_vals = np.linspace(0, 1, 100)
for power in (0.8, 0.5, 0.2):
    beta_val = 1 - power
    plt.plot(pi_vals, post_prob_func(0.05, beta_val, pi_vals), label='power={0}'.format(power))
plt.xlabel('Prior probability $Pr(H_A)$')
plt.ylabel('Posterior probability $Pr(H_A | T_S)$')
plt.legend()
plt.title("Posterior probability for different priors and power levels")
```

The posterior probability depends on the power.  If the power is low and $H_A$ is true, the likelihood of getting a significant test result is small.  Assuming $\pi = Pr(H_A) = 0.5$, our posterior probability is given by $\Frac{(1 - \beta)}{(1 - \beta) + \alpha}$.  As the chance of finding a true positive $= 1-\beta$ drops towards the chance of finding a false negative $= \alpha$, our confidence in the truth of the significant result must drop too.

The posterior likelihood also depends on the prior. When the prior $Pr(H_A)$ drops then we become more wary of the (apriori more unlikely) true positive compared to the (apriori more likely) false positive.

As you can see from the figure.  When power is 0.2, and the prior probability is less than around 0.2, then even if there is a significant test result, the null is still more likely than the $H_A$ (posterior < 0.5).

## Quantifying the effect of bias

Working scientists know that working scientists have many sources of bias in data collection and analysis.

We tend to assume that the effect of this bias is relatively minor.  Is this true?

Ioannidis quantifies bias with a parameter $u$.  $u$ is the proportion of not-significant findings that become significant as a result of bias.  Put another way, the effect of bias is the result of taking the second column in the probablity table above (the not-significant findings) and multiplying by $u$. We add this effect to the first column (significant findings) and subtract from the second column (not-significant findings). Before applying the priors, this looks like:

```
u = symbols('u')
bias_assoc_noprior = dict(t_s = (1 - beta) + u * beta,
                          t_ns = beta - u * beta,
                          f_s = alpha + u * (1 - alpha),
                          f_ns = (1 - alpha) - u * (1 - alpha))
HTML(association_table(bias_assoc_noprior, r'Bias $u$ not considering prior'))
```

Bias $u$ not considering prior

$T_S$$T_N$

$H_A$$1 - \beta + \beta u$$\beta - \beta u$

$H_0$$\alpha + u \left(1 - \alpha\right)$$1 - \alpha - u \left(1 - \alpha\right)$

Total$1 + \alpha - \beta + \beta u + u \left(1 - \alpha\right)$$1 + \beta - \alpha - \beta u - u \left(1 - \alpha\right)$

After applying the prior:

```
bias_assoc = bias_assoc_noprior.copy()
bias_assoc['t_s'] *= pi
bias_assoc['t_ns'] *= pi
bias_assoc['f_s'] *= 1 - pi
bias_assoc['f_ns'] *= 1 - pi
HTML(association_table(bias_assoc, r'Bias $u$ with prior $\pi := Pr(H_A)$'))
```

Bias $u$ with prior $\pi := Pr(H_A)$

$T_S$$T_N$

$H_A$$\pi \left(1 - \beta + \beta u\right)$$\pi \left(\beta - \beta u\right)$

$H_0$$\left(1 - \pi\right) \left(\alpha + u \left(1 - \alpha\right)\right)$$\left(1 - \pi\right) \left(1 - \alpha - u \left(1 - \alpha\right)\right)$

Total$\pi \left(1 - \beta + \beta u\right) + \left(1 - \pi\right) \left(\alpha + u \left(1 - \alpha\right)\right)$$\pi \left(\beta - \beta u\right) + \left(1 - \pi\right) \left(1 - \alpha - u \left(1 - \alpha\right)\right)$

The first cell in the table is $Pr(T_S | H_A) Pr(H_A)$.  The total for the first column gives $Pr(T_S)$.  Therefore the posterior probability $Pr(H_A | T_S)$ is:

```
post_prob_bias = bias_assoc['t_s'] / (bias_assoc['t_s'] + bias_assoc['f_s'])
post_prob_bias
```

```
# Same as Ioannidis formulation?
# This from Ioannidies 2005:
ppv_bias = ((1 - beta) * R + u * beta * R) / (R + alpha - beta * R + u - u * alpha + u * beta * R)
# Is this the same as our formula above?
simplify(ppv_bias - post_prob_bias) == 0
```

    True

What effect does bias have on the posterior probabilities?

```
# Formula as a function we can evaluate
post_prob_bias_func = lambdify((alpha, beta, pi, u), post_prob_bias)
pi_vals = np.linspace(0, 1, 100)
fig, axes = plt.subplots(3, 1, figsize=(8, 16))
for i, power in enumerate((0.8, 0.5, 0.2)):
    beta_val = 1 - power
    for bias in (0.05, 0.2, 0.5, 0.8):
        pp_vals = post_prob_bias_func(0.05, beta_val, pi_vals, bias)
        axes[i].plot(pi_vals, pp_vals, label='u={0}'.format(bias))
        axes[i].set_ylabel('Posterior probability $Pr(H_A | T_S)$')
    axes[i].set_title('Power = {0}'.format(power))
    axes[-1].set_xlabel('Prior probability $Pr(H_A)$')
axes[-1].legend()
```

As we'd expect, as bias increases to 1, the result of the experiment has less and less effect on our posterior estimate.  If the analysis was entirely biased, then our posterior estimate is unchanged from the prior (diagonal line on the graph).

## The effect of multiple studies

Ioannidis makes the point that when a field is particularly fashionable, there may be many research groups working on the same question.

Given publication bias for positive findings, it is possible that only positive research findings will be published.  If $n$ research groups have done the same experiment, then the probability that *all* the $n$ studies will be not significant, given $H_A$ is true, is $\beta^n$. Conversly the probability that there is at least one positive finding in the $n$ tests is $1 - \beta^n$.  Similarly the probability that all $n$ studies will be not significant, given $H_0$, is $(1 - \alpha)^n$.  The probability of at least one false positive is $1 - (1 - \alpha)^n$.

The probability table (without the priors) is:

```
n = symbols('n')
multi_assoc_noprior = dict(t_s = (1 - beta ** n),
                          t_ns = beta ** n,
                          f_s = 1 - (1 - alpha) ** n,
                          f_ns = (1 - alpha) ** n)
HTML(association_table(multi_assoc_noprior,
                       r'$n$ replications with positive publication bias, not considering prior'))
```

$n$ replications with positive publication bias, not considering prior

$T_S$$T_N$

$H_A$$1 - \beta^{n}$$\beta^{n}$

$H_0$$1 - \left(1 - \alpha\right)^{n}$$\left(1 - \alpha\right)^{n}$

Total$2 - \beta^{n} - \left(1 - \alpha\right)^{n}$$\beta^{n} + \left(1 - \alpha\right)^{n}$

Considering the prior:

```
multi_assoc = multi_assoc_noprior.copy()
multi_assoc['t_s'] *= pi
multi_assoc['t_ns'] *= pi
multi_assoc['f_s'] *= 1 - pi
multi_assoc['f_ns'] *= 1 - pi
HTML(association_table(multi_assoc,
                       r'$n$ replications with positive publication bias, with prior $\pi$'))
```

$n$ replications with positive publication bias, with prior $\pi$

$T_S$$T_N$

$H_A$$\pi \left(1 - \beta^{n}\right)$$\pi \beta^{n}$

$H_0$$\left(1 - \pi\right) \left(1 - \left(1 - \alpha\right)^{n}\right)$$\left(1 - \alpha\right)^{n} \left(1 - \pi\right)$

Total$\pi \left(1 - \beta^{n}\right) + \left(1 - \pi\right) \left(1 - \left(1 - \alpha\right)^{n}\right)$$\pi \beta^{n} + \left(1 - \alpha\right)^{n} \left(1 - \pi\right)$

Giving posterior probability of:

```
post_prob_multi = multi_assoc['t_s'] / (multi_assoc['t_s'] + multi_assoc['f_s'])
post_prob_multi
```

```
# Formula from Ioannidies 2005:
ppv_multi = R * (1 - beta ** n) / (R + 1 - (1 - alpha) ** n - R * beta ** n)
# Is this the same as our formula above?
simplify(ppv_multi - post_prob_multi) == 0
```

    True

```
# Formula as a function we can evaluate
post_prob_multi_func = lambdify((alpha, beta, pi, n), post_prob_multi)
pi_vals = np.linspace(0, 1, 100)
fig, axes = plt.subplots(3, 1, figsize=(8, 16))
for i, power in enumerate((0.8, 0.5, 0.2)):
    beta_val = 1 - power
    for n_studies in (1, 5, 10, 50):
        pp_vals = post_prob_multi_func(0.05, beta_val, pi_vals, n_studies)
        axes[i].plot(pi_vals, pp_vals, label='n={0}'.format(n_studies))
        axes[i].set_ylabel('Posterior probability $Pr(H_A | T_S)$')
    axes[i].set_title('Power = {0}'.format(power))
axes[-1].set_xlabel('Prior probability $Pr(H_A)$')
axes[-1].legend()
```

## Putting it together

Considering analysis bias and positive publication bias together:

```
multi_bias_assoc_noprior = dict(t_s = (1 - beta ** n) + u * beta ** n,
                                t_ns = beta ** n - u * beta ** n,
                                f_s = 1 - (1 - alpha) ** n + u * (1 - alpha) ** n,
                                f_ns = (1 - alpha) ** n - u * (1 - alpha)**n)
HTML(association_table(multi_bias_assoc_noprior,
                       r'$n$ replications, $u$ analysis bias, not considering prior'))
```

$n$ replications, $u$ analysis bias, not considering prior

$T_S$$T_N$

$H_A$$1 - \beta^{n} + u \beta^{n}$$\beta^{n} - u \beta^{n}$

$H_0$$1 - \left(1 - \alpha\right)^{n} + u \left(1 - \alpha\right)^{n}$$\left(1 - \alpha\right)^{n} - u \left(1 - \alpha\right)^{n}$

Total$2 - \beta^{n} - \left(1 - \alpha\right)^{n} + u \beta^{n} + u \left(1 - \alpha\right)^{n}$$\beta^{n} + \left(1 - \alpha\right)^{n} - u \beta^{n} - u \left(1 - \alpha\right)^{n}$

```
multi_bias_assoc = multi_bias_assoc_noprior.copy()
multi_bias_assoc['t_s'] *= pi
multi_bias_assoc['t_ns'] *= pi
multi_bias_assoc['f_s'] *= 1 - pi
multi_bias_assoc['f_ns'] *= 1 - pi
HTML(association_table(multi_bias_assoc,
                       r'$n$ replications, $u$ analysis bias, with prior'))
```

$n$ replications, $u$ analysis bias, with prior

$T_S$$T_N$

$H_A$$\pi \left(1 - \beta^{n} + u \beta^{n}\right)$$\pi \left(\beta^{n} - u \beta^{n}\right)$

$H_0$$\left(1 - \pi\right) \left(1 - \left(1 - \alpha\right)^{n} + u \left(1 - \alpha\right)^{n}\right)$$\left(1 - \pi\right) \left(\left(1 - \alpha\right)^{n} - u \left(1 - \alpha\right)^{n}\right)$

Total$\pi \left(1 - \beta^{n} + u \beta^{n}\right) + \left(1 - \pi\right) \left(1 - \left(1 - \alpha\right)^{n} + u \left(1 - \alpha\right)^{n}\right)$$\pi \left(\beta^{n} - u \beta^{n}\right) + \left(1 - \pi\right) \left(\left(1 - \alpha\right)^{n} - u \left(1 - \alpha\right)^{n}\right)$

```
post_prob_multi_bias = multi_bias_assoc['t_s'] / (multi_bias_assoc['t_s'] + multi_bias_assoc['f_s'])
post_prob_multi_bias
```

Let's assume that two groups are doing more or less the same study, and only the positive study publishes ($n = 2$).  There is an analysis bias of 10% ($u= 0.1$).  We take the power from the Button et al estimate for neuroimaging studies = 0.08.  Therefore $\beta = 1 - 0.08 = 0.92$:

* Button, Katherine S., John PA Ioannidis, Claire Mokrysz, Brian A. Nosek, Jonathan Flint, Emma SJ Robinson, and Marcus R. Munafò. 2013. “Power failure: why small sample size undermines the reliability of neuroscience.” *Nature Reviews Neuroscience*.

```
pp_mb_func = lambdify((alpha, beta, pi, u, n), post_prob_multi_bias)
pp_vals_nobias = pp_mb_func(0.05, 0.92, pi_vals, 0, 1)
pp_vals_bias = pp_mb_func(0.05, 0.92, pi_vals, 0.1, 2)
plt.plot(pi_vals, pp_vals_nobias, label='no analysis or publication bias')
plt.plot(pi_vals, pp_vals_bias, label='with analysis and publication bias')
plt.plot(pi_vals, pi_vals, 'r:', label='$T_S$ not informative')
plt.ylabel('Posterior probability $Pr(H_A | T_S)$')
plt.xlabel('Prior probability $Pr(H_A)$')
plt.legend()
```

This graph tells us that, for a study with average power in neuroimaging, with some mild analysis bias and positive publication bias, the significant finding $T_S$ does not change our posterior very much from our prior.

If we do some study with an hypothesis that is suitably unlikely apriori - say $Pr(H_A) = 0.25$ - then our posterior probability is:

```
pp_mb_func(0.05, 0.92, 0.25, 0.1, 2)
```

What if the result was significant at $p < 0.01$?:

```
pp_mb_func(0.01, 0.92, 0.25, 0.1, 2)
```

So, even if our result is significant at $p < 0.01$, the probability that $H_A$ is correct is still less than $0.5$.


```python
from kanren import run, var, fact
import kanren.assoccomm as la

# 足し算(add)と掛け算（mul)
# addとmulはルールの名前なだけ
add = 'addition'
mul = 'multiplication'

# 足し算、掛け算は交換法則（commutative)、結合法則(associative)を持つ事をfactを使って宣言する
# 交換法則とは、入れ替えても結果が変わらない事。足し算も掛け算も入れ替えても答えは変わらない
# 結合法則とは、カッコの位置を変えても変わらない事。足し算だけの式、掛け算だけの式はカッコの位置が変わっても答えは変わらない
# 交換法則も結合法則も文字列をどう扱うかといったルールでも使える
fact(la.commutative, mul)
fact(la.commutative, add)
fact(la.associative, mul)
fact(la.associative, add)

a, b, c = var('a'), var('b'), var('c')

expression_orig = (add, (mul, 3, -2), (mul, (add, 1, (mul, 2, 3)), -1))

expression1 = (add, (mul, (add, 1, (mul, 2, a)), b), (mul, 3, c))
expression2 = (add, (mul, 3, c), (mul, b, (add, (mul, 2, a), 1)))
expression3 = (add, (add, (mul, (mul, 2, a), b), b), (mul, 3, c))
```

この式を例に

$$
  expression\_orig = 3 \times (-2) + (1 + 2 \times 3) \times (-1)
$$

変数で置き換えた式を照合する

$$
\begin{align}
  &expression1 = (1 + 2 \times a) \times b + 3 \times c\\
  &expression2 = c \times 3 + b \times (2 \times a + 1)\\
  &expression3 = (((2 \times a) \times b) + b) + 3 \times c
 \end{align}
$$

expression1〜3は数学的に同じ式

```python
print(run(0, (a, b, c), la.eq_assoccomm(expression1, expression_orig)))
print(run(0, (a, b, c), la.eq_assoccomm(expression2, expression_orig)))
print(run(0, (a, b, c), la.eq_assoccomm(expression3, expression_orig)))
```

    ((3, -1, -2),)
    ((3, -1, -2),)
    ()

1番目と2番目の式は一緒。３番目は構造的に異なる（らしい）

run(答えの個数, （欲しい答えの変数）,

分配法則を定義していないから？？

`eq_assoccomm`が分からん。。。