# Linear Regression

# Simple Linear Regression

Running a SLR in Python is fairly simple once you know how to use the relevant functions. What might be confusing is that there exist several packages which provide functions for linear regression. We will use functions from the `statsmodels` (sub-)package. Other packages such as e.g. `scikit-learn` have linear regression functions too, but what makes `statsmodels` stand out from other packages is its broad set of auxiliary functions for regression diagnostics. As usual we start by importing the packages needed for our task.

```python
# Load relevant packages
%matplotlib inline
import numpy as np
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
plt.rcParams['font.size'] = 14
```

As toy data we will use the 'Advertising' data set introduced in the chapter 4 of the script. The data is taken from James et al. (2013). A copy is provided on the book's website where we will download it from.

```python
# From Advertising data set read cols 2:4
url = 'https://www.statlearning.com/s/Advertising.csv'
ad = pd.read_csv(url, sep=',', usecols=(np.arange(1, 5, 1)))
print(ad.head())
```

          TV  radio  newspaper  sales
    0  230.1   37.8       69.2   22.1
    1   44.5   39.3       45.1   10.4
    2   17.2   45.9       69.3    9.3
    3  151.5   41.3       58.5   18.5
    4  180.8   10.8       58.4   12.9

Next we run a linear regression of `TV` on `sales` to calculate the coefficients and print a summary output.

```python
# Run regression and calculate fit
reg = sm.OLS(ad.sales, exog=sm.add_constant(ad.TV)).fit()
# Alternatively: reg = sm.OLS(ad.sales, sm.add_constant(ad.TV)).fit()
print(reg.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  sales   R-squared:                       0.612
    Model:                            OLS   Adj. R-squared:                  0.610
    Method:                 Least Squares   F-statistic:                     312.1
    Date:                Fri, 19 Feb 2021   Prob (F-statistic):           1.47e-42
    Time:                        00:01:06   Log-Likelihood:                -519.05
    No. Observations:                 200   AIC:                             1042.
    Df Residuals:                     198   BIC:                             1049.
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          7.0326      0.458     15.360      0.000       6.130       7.935
    TV             0.0475      0.003     17.668      0.000       0.042       0.053
    ==============================================================================
    Omnibus:                        0.531   Durbin-Watson:                   1.935
    Prob(Omnibus):                  0.767   Jarque-Bera (JB):                0.669
    Skew:                          -0.089   Prob(JB):                        0.716
    Kurtosis:                       2.779   Cond. No.                         338.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

**Side note:** If you are an `R` guy and prefer their syntax, you could import the statsmodels.formula.api subpackage and run something like:

> `from statsmodels.formula.api import ols
reg = ols("Sales ~ TV", data=ad).fit()
reg.summary()`

Intercept will automatically be calculated in the above setting.

Both p-values for intercept ($\hat{\beta}_0$) and slope ($\hat{\beta}_1$) are smaller than any reasonable significance level and thus we can reject the null hypothesis that either of the coefficients is zero (or irrelevant).

Instead of printing the whole summary, we could also access each of the three summary as follows:

```python
reg.summary().tables[1]
```

    	       coef	     std err	      t	      P>|t|	  [0.025	    0.975]

  const	     7.0326	     0.458	    15.360	  0.000	     6.130	     7.935

  TV	        0.0475	     0.003	    17.668	  0.000	     0.042	     0.053

## Plotting the Fit

The `sm.OLS()` function calculates all kind of regression-related results. All this information is attached to the `reg` object. For example to plot the data we would need the model's fitted values. These can be accessed by combining the regression variable/object with the attribute `.fittedvalues` as in `reg.fittedvalues`. In below plot it is shown how this can be of use. We plot the data and fit using the standard plotting functions.

```python
# Plot scatter & lm
plt.figure(figsize=(12, 8))
plt.scatter(ad.TV, ad.sales, marker='.', label='Sample') # Training data
plt.plot(ad.TV, reg.fittedvalues, c='k', label='Fit')  # Linear fit
plt.ylabel('Sales')
plt.xlabel('TV')
plt.legend();
```

Let us plot residuals versus fitted values to do some visual regression diagnostics:

```python
plt.figure(figsize=(12, 8))
plt.scatter(ad.TV, reg.resid)
plt.axhline(y=0, c='k')  #Black horizontal line at 0
plt.xlabel('TV (fitted)')
plt.ylabel('Residuals');
```

The above two plots are just standard `matplotlib` plots serving the purpose of visual diagnostics. Beyond that, the `statsmodel` package has separate built-in plotting functions suited for visual regression diagnostics. We will not discuss them here but if interested you'll find plenty of sources on the web (e.g. [here](http://www.statsmodels.org/dev/examples/notebooks/generated/regression_plots.html), [here](http://www.statsmodels.org/stable/graphics.html) or [here](http://mpastell.com/2013/04/19/python_regression/))

## Accessing the Output

The `reg` object contains a ton of information which you can all access. To see what there is, type `reg.` and press tab. Two examples are shown below. Notice that some are attributes (like `x.shape`, `x.size`) and do not need paranthesis to call them. Others are methods (similar to `.sum()`, `.min()`) and require parenthesis.

```python
# Regression coefficients
print(reg.params, '\n')
print(reg.resid.head())
```

    const    7.032594
    TV       0.047537
    dtype: float64

    0    4.129225
    1    1.252026
    2    1.449776
    3    4.265605
    4   -2.727218
    dtype: float64

# Confidence Intervals & Hypthesis Testing

The 95%-confidence interval (CI) is printed in the summary above. If one wishes to calculate it for a different significance level (`alpha`), it is done as follows:

```python
# 99% CI (alpha = 1%) based on t-distribution
reg.conf_int(alpha=0.01)
```

      0
      1

      const
      5.841796
      8.223391

      TV
      0.040539
      0.054535

The regression summary provides $t$-statistic and p-value for the null hypothesis $H_0: \hat{\beta}_j = 0$, $H_a: \hat{\beta}_j \neq 0$. You can call the resulting $t$-statistic and p-value with its attributes.

```python
print(reg.tvalues, '\n')
print(reg.pvalues)
```

    const    15.360275
    TV       17.667626
    dtype: float64

    const    1.406300e-35
    TV       1.467390e-42
    dtype: float64

If you wish to test a different null hypothesis, e.g. $H_0: \hat{\beta}_{TV} = 0.054$ vs. $H_1: \hat{\beta}_{TV} \neq 0.054$ use the following code:

```python
reg.t_test('TV=0.054')
```

    <class 'statsmodels.stats.contrast.ContrastResults'>
                                 Test for Constraints
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    c0             0.0475      0.003     -2.402      0.017       0.042       0.053
    ==============================================================================

As far as I know, `Statsmodels` does not provide a function to calculate 'greater than' or 'smaller than' alternative hypothesis. Reason being: because with symmetric distributions, the one-sided test can be derived from the two-sided test. A one-sided p-value is just half of the two-sided p-value. This means that given p and $t$ values from a two-tailed test, you would reject the null hypothesis of a greater-than test when p/2 < alpha and $t$ > 0, and of a less-than test when p/2 < alpha and $t$ < 0.

## Coefficient of Determination

The $R^2$ measure, or "coefficient of determination", displays the proportion of the variability in $y$ that is well explained by the regression fit. It is defined as

$$\begin{equation}
R^2 = \frac{TSS - SSR}{TSS} = 1 - \frac{SSR}{TSS}
\end{equation}$$

where TSS is the *total sum of squares*, defined as $TSS = \sum (y_i - \bar{y})^2$, and SSR is the *sum of squared residuals*, given by $SSR = \sum (y_i - \hat{y}_i)^2$.

It is easy to call the $R^2$ value from the regression object `reg` as the following line shows.

```python
# R squared measure
reg.rsquared
```

    0.611875050850071

## Regression Diagnostics
### Test of Homoskedasticity

In general we assume a constant variance of the error term (homoskedasticity; $Var(\epsilon_i) = \sigma^2$ for $i = 1, \ldots, N$). From the residuals vs. fitted plot we have to question this assumption. To test it mathematically, you can run a heteroskedasticity test. The stats package offers several test options; the more common ones are White's or the one from Breusch-Pagan. See [here for more details](http://www.statsmodels.org/dev/diagnostic.html) on tests on heteroskedasticity.

Below the White test is applied as an example. The parameter 'reg.model.exog' simply contains the matrix X (here it is a [200 x 2] matrix with constant 1 in first column and values for TV in second). The output should become more clear when you check the function's help page (use `?sm.stats.diagnostic.het_white`).

The null hypothesis is that the error variance does **not** depend on X, thus is homoskedastic. Based on the large f-statistic value we can gently reject the null-hypothesis that the error variance is homoskedastic.

```python
# Test for heteroskedasticity with White test
wht = sm.stats.diagnostic.het_white(resid=reg.resid, exog=reg.model.exog)
print('f-statistic: {0:>19.4f} \n'
      'p-value for f-statistic: {1:>7.4f}'.format(wht[2], wht[3]))
```

    f-statistic:             35.1683
    p-value for f-statistic:  0.0000

If you wish to run tests with heteroskedastistic robust standard errors you can either access the `reg` object's robust standard errors (`reg.HC0_se`, `reg.HC1_se`, `reg.HC2_se`, `reg.HC3_se`) or, more conveniently, directly define the covariance estimator (e.g. 'HC3' as below) when you generate the object in the first place. Below example shows how you can do this. See also [here](http://www.statsmodels.org/dev/generated/statsmodels.regression.linear_model.RegressionResults.html) or [here](https://stackoverflow.com/questions/30553838/getting-statsmodels-to-use-heteroskedasticity-corrected-standard-errors-in-coeff) for some further information.

```python
regRobust = sm.OLS(ad.sales, exog=sm.add_constant(ad.TV)).fit(cov_type='HC3')
print(regRobust.HC3_se, '\n')
print(reg.HC3_se)
```

    const    0.336426
    TV       0.002890
    dtype: float64

    const    0.336426
    TV       0.002890
    dtype: float64

### Other Relevant Checks

The `statsmodels` package offers many more functions to run regression diagnostics, e.g. checks for autocorrelation, non-linearity, normality of residuals etc. These functions are applicable to both simple as well as multiple linear regression models. There's a short [Jupyther notebook](http://www.statsmodels.org/dev/examples/notebooks/generated/regression_diagnostics.html) detailing some of the options.

## Application: Stock Beta

A stock beta measures the systematic risk of a security, the tendency of a security to respond to swings in the broad market. Typically, a large, well diversified index is taken as a proxy for the market portfolio (e.g. S&P500, Euro Stoxx 50, SPI, etc.). There are different ways to calculate a stock beta. We will show the regression approach, where a stock's beta is the slope of the following linear regression:

$$\begin{equation}
r - r_f = \alpha + \beta(r_M - r_f) + e
\end{equation}$$

Let us look into Geberit's stock beta. As a proxy for the market portfolio we use the Swiss market index (SMI). The risk free rate is set to $r_f=0$, which is a fairly reasonable approach in light of [the Swiss national bank's (SNB) interest rates](https://data.snb.ch/de/topics/ziredev#!/cube/zimoma) for the past eight years (at the time of this writing in 2021). We will work with monthly returns for the past five years (60 months) - though other approaches (e.g. last 24 monthly returns, weekly returns for last 2 years, etc.) are reasonable choices too. The stock and SMI data we will load from a csv that was sourced through a financial data provider.

```python
df = pd.read_csv('Data/SMIDataMonthly.csv', sep=',',
                 parse_dates=['Date'], dayfirst=True,
                 index_col=['Date'])
df.head()
```

      ABBN
      ADEN
      BAER
      CFR
      CSGN
      GEBN
      GIVN
      LHN
      LONN
      NESN
      ...
      ROG
      SCMN
      SGSN
      SLHN
      SREN
      UBSG
      UHR
      ZURN
      SIK
      SMI

      Date

      2018-01-31
      25.95
      76.60
      63.9
      89.32
      17.96
      441.0
      2240.0
      57.00
      258.9
      80.42
      ...
      229.45
      508.4
      2503
      349.5
      91.80
      18.90
      426.5
      305.9
      8065
      9335.40

      2017-12-29
      26.12
      74.55
      59.6
      88.30
      17.40
      429.1
      2252.0
      54.95
      263.3
      83.80
      ...
      246.50
      518.5
      2541
      345.0
      91.25
      17.94
      397.4
      296.6
      7740
      9381.87

      2017-11-30
      25.19
      74.40
      57.8
      84.70
      16.66
      427.8
      2238.0
      53.80
      257.1
      84.25
      ...
      248.20
      519.0
      2432
      330.2
      92.20
      16.99
      358.7
      297.2
      7600
      9318.77

      2017-10-31
      26.08
      79.15
      59.0
      92.00
      15.73
      451.6
      2228.0
      56.35
      265.0
      83.90
      ...
      230.50
      504.0
      2464
      346.8
      93.85
      16.98
      391.0
      304.5
      7385
      9242.18

      2017-09-29
      23.94
      75.40
      57.3
      88.50
      15.33
      458.1
      2107.0
      56.60
      254.0
      81.10
      ...
      247.20
      496.2
      2323
      341.1
      87.70
      16.55
      402.8
      295.4
      7205
      9157.46

5 rows × 21 columns

The dataframe `df` contains monthly closing prices on all SMI-stocks (incl. SMI index) with a date index in descending order. Let us create a separate Pandas dataframe with the returns of the past 60 months (`dfRets`) for both GEBN as well as the index.

```python
# Calculate returns and assign to variable dfRets
dfRets = pd.DataFrame()
dfRets['GEBNrets'] = np.log(df['GEBN'] / df['GEBN'].shift(-1))
dfRets['SMIrets'] = np.log(df['SMI'] / df['SMI'].shift(-1))
print(dfRets.head())
```

                GEBNrets   SMIrets
    Date
    2018-01-31  0.027355 -0.004965
    2017-12-29  0.003034  0.006748
    2017-11-30 -0.054141  0.008253
    2017-10-31 -0.014291  0.009209
    2017-09-29  0.044640  0.025662

Having done that, we are already set to run the regression and print the results.

```python
# Set observation period (last 60 monthly returns)
months = 60

# Create OLS object, run regression and calculate fit
regBeta = sm.OLS(endog=dfRets.iloc[:months, 0],
                 exog=sm.add_constant(dfRets.iloc[:months, 1])).fit()

# Show table on coefficients
print(regBeta.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:               GEBNrets   R-squared:                       0.311
    Model:                            OLS   Adj. R-squared:                  0.300
    Method:                 Least Squares   F-statistic:                     26.23
    Date:                Fri, 19 Feb 2021   Prob (F-statistic):           3.62e-06
    Time:                        00:01:07   Log-Likelihood:                 107.89
    No. Observations:                  60   AIC:                            -211.8
    Df Residuals:                      58   BIC:                            -207.6
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.0086      0.005      1.617      0.111      -0.002       0.019
    SMIrets        0.9156      0.179      5.121      0.000       0.558       1.273
    ==============================================================================
    Omnibus:                        4.808   Durbin-Watson:                   2.225
    Prob(Omnibus):                  0.090   Jarque-Bera (JB):                4.204
    Skew:                           0.405   Prob(JB):                        0.122
    Kurtosis:                       4.013   Cond. No.                         34.0
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Based on the regression output we have no reason to reject the null of Geberit's stock beta being equal to zero. The $R^2$ measure, though, shows that only a small portion of the variation in Geberits monthly returns is explained by SMI's monthly returns.

```python
# Get relevant information
beta = regBeta.params['SMIrets']
alpha = regBeta.params["const"]
rsqr = regBeta.rsquared

# Plot scatter & lm; add text with alpha, beta , R2
plt.figure(figsize=(12, 8))
plt.scatter(dfRets.iloc[:months, 1],
            dfRets.iloc[:months, 0],
            marker='.', label='Monthly Returns')
plt.plot(dfRets.iloc[:months, 1], regBeta.fittedvalues, c='k', label='Fit')
plt.gca().set_aspect('equal')
plt.gca().set_xlim(-0.1, 0.15)
plt.ylabel('Geberit Monthly Returns')
plt.xlabel('SMI Monthly Returns')
plt.legend(loc='lower right')
plt.text(-0.08, 0.13, 'Beta: {0: .2f}'.format(beta))
plt.text(-0.08, 0.12, 'Alpha: {0: .2f}'.format(alpha))
plt.text(-0.08, 0.11, 'R^2: {0: .2f}'.format(rsqr));
```

What we calculated above is often referred to as the raw beta. The beta value of a stock has been found to be on average closer to the mean value of 1.0, the beta of an average-systematic-risk portfolio, than to the value of the raw beta (Pinto et al. (2016)). This is why data providers such as Bloomberg publish the adjusted beta as first introduced by Blume (1971), which is calculated as

$$ \text{Adjusted beta} = 2/3 \cdot \text{raw beta} + 1/3 \cdot 1$$

Now, let us assume we are given the task to investigate whether a beta indeed regresses to 1 over time. For that we could, as a starting point, assess a stock's rolling beta over the past years. Note that this is just an example of use. Computationally it would be much faster to calculate the stock beta via the covariance/variance formula.

```python
def rollingBeta(df, window=60):

    '''Calculates the running beta of a stock.

    Parameters
    ==========
    df : [n x 2] pandas dataframe with log-returns for
         stock and market portfolio. Index should be
         datetime series.
    window : rolling window with default value 60 [optional]

    Returns
    =======
    rb : Pandas dataframe with (backward-looking) rolling beta.
    '''

    # Drop NA rows from df
    df = df.dropna()
    # Set up empty results array
    res = np.empty(len(df) - window + 1)

    # Loop through df
    for i in range(0, len(df)):

        # As long as remaining subset is >= window, we proceed
        if (len(df) - i) >= window:

            # Subset df
            sub = df.iloc[i:window+i, :]
            # Run Regression
            model = sm.OLS(endog=sub.iloc[:, 0],
                           exog=sm.add_constant(sub.iloc[:, 1])).fit()
            # Read out beta coefficient
            res[i] = model.params[1]

    # Format output to dataframe
    rb = pd.DataFrame(data=res, index=df.index[:(len(df)-window+1)])
    rb.columns = ['RollingBeta']

    return rb
```

Now we are ready to call the function. This time we use the last two years of monthly returns. Thus we set `window=24` to overwrite the default value of 60.

```python
# Call function and save output to 'rollBeta'
rollBeta = rollingBeta(df=dfRets, window=24)
```

Though this is far away from a thorough analysis, plotting the results shows that at least in Geberit's case, there is indeed some truth to the assessment, that the beta exhibits some reversion to the market beta value of 1.

```python
# Import 'mdates' library to format dates in x-axis
import matplotlib.dates as mdates

# Plot rolling beta
fig, ax = plt.subplots(1, figsize=(12, 8))
ax.plot(rollBeta, label='Geberit Rolling Beta')
ax.axhline(y=1, c='gray', linestyle=':')  # Horizontal line
ax.legend(fontsize=12)
ax.xaxis.set_major_locator(mdates.MonthLocator(interval=6))
ax.xaxis.set_major_formatter(mdates.DateFormatter('%m-%Y'))
fig.autofmt_xdate();  # Autorotate x-axis for readability
```

# Multiple Linear Regression

## Estimating the Regression Coefficients

Simple linear regression serves well to introduce the concept and to build a good understanding. However, in reality we often have to work with more than one predictor. In the advertising data set for example we had not only data on TV advertising spendings but also on radio newspaper. It thus makes sense to extend the simple to a multiple linear regression model.

We again use the Advertising data set to see how this is done in Python. The same functions from the `statsmodels` package apply to multiple linear regression. We run the following regression

$$\text{sales} = \beta_0 + \beta_1 \text{TV} + \beta_2 \text{radio} + \beta_3 \text{newspaper} + \epsilon$$

```python
# Assign features and response to X and y
y = ad.sales
X = ad[['TV', 'radio', 'newspaper']]
X = sm.add_constant(X)
```

```python
# Run regression and print summary
mlReg = sm.OLS(endog=y, exog=X).fit()
print(mlReg.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  sales   R-squared:                       0.897
    Model:                            OLS   Adj. R-squared:                  0.896
    Method:                 Least Squares   F-statistic:                     570.3
    Date:                Fri, 19 Feb 2021   Prob (F-statistic):           1.58e-96
    Time:                        00:01:09   Log-Likelihood:                -386.18
    No. Observations:                 200   AIC:                             780.4
    Df Residuals:                     196   BIC:                             793.6
    Df Model:                           3
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          2.9389      0.312      9.422      0.000       2.324       3.554
    TV             0.0458      0.001     32.809      0.000       0.043       0.049
    radio          0.1885      0.009     21.893      0.000       0.172       0.206
    newspaper     -0.0010      0.006     -0.177      0.860      -0.013       0.011
    ==============================================================================
    Omnibus:                       60.414   Durbin-Watson:                   2.084
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              151.241
    Skew:                          -1.327   Prob(JB):                     1.44e-33
    Kurtosis:                       6.332   Cond. No.                         454.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

The coefficient for radio (0.1885) tells us, that - holding all other factors fixed - an additional 1'000 dollars in radio advertising spendings will boost the product's sales by 188.5 units.

Again: If you are an `R` guy and prefer their syntax, you could import the statsmodels.formula.api subpackage and run something like:

> `from statsmodels.formula.api import ols
mlReg = ols("Sales ~ TV + radio + newspaper", data=ad).fit()
mlReg.summary()`

## Hypothesis Tests

Again the summary above provides $t$-statistic and p-value for each individual regression coefficient. As was the case for the simple linear regression, the underlying null hypothesis is that each parameter is zero ($H_0: \beta_{j,\, H_0} = 0$). For TV and Radio we reject the null even at the 1% significance level. However, given the large p-value for Newspaper we fail to reject the null for $\beta_{\text{Newspaper}} = 0$ at any reasonable level. Thus we can conclude that leaving Newspaper data out might be a reasonable option. If other null hypothesis' ought to be tested, we can use the same command as shown above.

```python
# t-test on H0: beta(TV) = 0.0475
mlReg.t_test('TV=0.0475')
```

    <class 'statsmodels.stats.contrast.ContrastResults'>
                                 Test for Constraints
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    c0             0.0458      0.001     -1.244      0.215       0.043       0.049
    ==============================================================================

The output shows that we fail to reject the null hypothesis that $\beta_{TV} = 0.0475$.

Beyond the element-wise hypothesis tests the regression summary also provides **F-statistic** (and the corresponding p-value) on the combined hypothesis that

$$\begin{align}
H_0&: \quad \beta_j = \beta_1, \beta_2, \ldots \beta_p = 0 \\
H_a&: \quad \beta_j \neq 0 \text{ for at least one $j$}
\end{align}$$

On the basis of the corresponding p-value (i.e. 1.58e-96) we can reject the null at any reasonable significance level. Should we be interested in assessing a particular hypothesis, say

$$\begin{align}
H_0&: \quad \beta_{TV} = \beta_{\text{Radio}} = 0.1 \\
H_a&: \quad \beta_{TV} \neq \beta_{\text{Radio}} \neq 0.1
\end{align}$$
we use the `.f_test()` method.

```python
# Test H0: beta(radio) = beta(newspaper) = 0.1
mlReg.f_test('radio = newspaper = 0.1')
```

    <class 'statsmodels.stats.contrast.ContrastResults'>
    <F test: F=array([[158.25546736]]), p=1.23080057155986e-41, df_denom=196, df_num=2>

See the [documentation page](http://www.statsmodels.org/dev/generated/statsmodels.regression.linear_model.RegressionResults.f_test.html) for further examples on how this function can be used.

### Coefficient of Determination

The $R^2$ measure for the MLR is the same as for the SLR. However, in the case of MLR it has one drawback: the value will always increase when more explanatory variables are added to the model - even if those variables are only weakly associated with the response. To make good on this disadvantage a modificated measure is often used: **adjusted $R^2$**.

$$\begin{equation}
R^2_{adj} = 1 - (1-R^2) \frac{n-1}{n-p-1}
\end{equation}$$

To get this measure in Python, simply use the OLS object and call the `.rsquareed_adj` attribute.

```python
mlReg.rsquared_adj
```

    0.8956373316204668

## Application: Factor Models

### Fama-French Three Factor Model

We will apply the concept of multiple linear regression in the context of Fama-French's three factor model (Fama and French (1993)). Their model follows Ross' arbitrage pricinge theory which postulates that excess returns are linearly related to a set of systematic risk factors (Ross et al. (1973)). The factors can be returns on other assets, such as the market portfolio, or any other variable (e.g. interest rates, inflation, consumption growth, market sentiment, hedging demands etc.. Fama-French empirically discovered three factors to capture the systematic risk: (A) firm size, (B) book-to-market ratio (B/M) and (C) market risk. To quantify their findings, Fama-French constructed zero-net-investment factor portfolios capturing the systematic risk on firm size (factor is labeled 'small minus big' (SMB) and is constructed by going long on small and short on big size stocks) and B/M (labeled 'high minus low' (HML), i.e. going long on high B/M, short on low B/M stocks). The sensitivity of individual stocks to the three factors is then given by the estimated coefficients of a multiple linear regression. As a group the three factors combine for the total risk premium.

The expected excess return $R_{it}$ of asset $i$ at time $t$ in the Fama-French three-factor model is described by

$$\begin{equation}
R_{it} = \alpha_i + \beta_i^{M} (r_{M,t} - r_{f,t}) + \beta_i^{SMB} SMB_t + \beta_i^{HML} HML_t + \epsilon_{it}
\end{equation}$$

The above Fama-French factors are calculated on a monthly basis and published on [Kenneth R. French's website](http://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html). There you will also find information on the methodology of the model and lots of other possible factors besides the three we look into here. To run this regression in Python we use a shortcut. The `pandas_datareader` package is capable of loading the data without having to download a txt or csv file in a separate step.

We will calculate the factor beta for all 20 SMI stocks. For that we use Fama-French's 'European 3 Factors' data. Following our 'Stock Beta' example from above and for the sake of simplicity, the risk free rate for Switzerland is again assumed to be zero (wrt. $R_{it}$).

### Prepare Data for Fama-French Model
We start by importing the `pandas_datareader.data` package and defining some key parameter.

```python
import pandas_datareader as web

# Define obs. period, start & enddate
months = 60
startdate = '2012-06-01'
enddate   = '2017-12-31'
```

    C:\Anaconda\lib\site-packages\pandas_datareader\compat\__init__.py:7: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
      from pandas.util.testing import assert_frame_equal

** Side Note: ** If you want to know what data is available (and their labels), you can run the `get_available_datasets()` function.

> `from pandas_datareader.famafrench import get_available_datasets
get_available_datasets()`

or simply check [Kenneth R. French's website](http://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html)

```python
# Load FF factors
data = web.DataReader('Europe_3_Factors', data_source='famafrench',
                      start=startdate, end=enddate)
```

Variable `data` is a dictionary with three entries: monhtly data, annual data and a description. We select the monthly rates in dictionary 0 and format the data.

```python
# Select monthly data
ff = data[0]

# Sort data in descending order
ff = ff.sort_index(ascending=False)

# Convert returns to decimal percentages
ff = ff/100
print(ff.head(3))
```

             Mkt-RF     SMB     HML      RF
    Date
    2017-12  0.0147  0.0155  0.0050  0.0009
    2017-11 -0.0001 -0.0038  0.0129  0.0008
    2017-10  0.0059 -0.0096 -0.0013  0.0009

Next we calculate the log-returns of all SMI stocks. The share prices are taken from dataframe `df` which we loaded above.

```python
shsRets = np.log(df / df.shift(-1))
shsRets = shsRets['2017-12-31':'2012-06-01']
shsRets = shsRets.iloc[:, :-1]  # We exclude the last column (with SMI data)
shsRets.head(3)
```

      ABBN
      ADEN
      BAER
      CFR
      CSGN
      GEBN
      GIVN
      LHN
      LONN
      NESN
      NOVN
      ROG
      SCMN
      SGSN
      SLHN
      SREN
      UBSG
      UHR
      ZURN
      SIK

      Date

      2017-12-29
      0.036254
      0.002014
      0.030667
      0.041625
      0.043460
      0.003034
      0.006236
      0.021150
      0.023829
      -0.005356
      -0.021609
      -0.006873
      -0.000964
      0.043844
      0.043846
      -0.010357
      0.054408
      0.102457
      -0.002021
      0.018253

      2017-11-30
      -0.034722
      -0.061889
      -0.020549
      -0.082673
      0.057441
      -0.054141
      0.004478
      -0.046309
      -0.030265
      0.004163
      0.024040
      0.073984
      0.029328
      -0.013072
      -0.049050
      -0.017738
      0.000589
      -0.086221
      -0.024266
      0.028697

      2017-10-31
      0.085618
      0.048538
      0.029237
      0.038786
      0.025758
      -0.014291
      0.055839
      -0.004427
      0.042396
      0.033943
      -0.008480
      -0.069947
      0.015597
      0.058927
      0.016573
      0.067776
      0.025650
      -0.029733
      0.030341
      0.024676

### Calculate Fama-French Coefficients

In order to run the regression we need to have matching indices. Fama-French's index is 'yyyy-mm' while our dataframe with returns has format 'yyyy-mm-dd'. Since we know that the length of both dataframes is equal, we can simply overwrite the index of one of the dataframes.

```python
# Create matching indices
ff.index = shsRets.index
```

We are now in a position to run the multiple linear regression. We will again use the past 60 months. From the Fama French set we just need the first three columns. Column 4 is the risk free rate which we do not use.

```python
# Add constant to matrix for alphas (=intercept)
X = sm.add_constant(ff.iloc[:months, :3])
```

```python
# Assign ticker to variable
tickers = shsRets.columns
```

```python
# Create results matrix to paste beta factors
res = np.empty(shape=(5, len(tickers)))
```

```python
# Run regression for each ticker
for i in range(0, len(tickers)):

    # Select returns of share i
    sub = shsRets.iloc[:months, i]
    # Run regression
    model = sm.OLS(endog=sub, exog=X).fit()
    # Paste beta factors to 'res' matrix
    res[0:4, i] = model.params
    res[4, i] = model.rsquared_adj
```

```python
# Format output to dataframe
ff3f = pd.DataFrame(data=res, index=['Alpha', 'BetaMkt', 'BetaSMB', 'BetaHML', 'R2_adj'])
ff3f.columns = tickers
```

And here are our factors:

```python
ff3f
```

      ABBN
      ADEN
      BAER
      CFR
      CSGN
      GEBN
      GIVN
      LHN
      LONN
      NESN
      NOVN
      ROG
      SCMN
      SGSN
      SLHN
      SREN
      UBSG
      UHR
      ZURN
      SIK

      Alpha
      -0.000906
      -0.001914
      0.005105
      -0.002015
      -0.011465
      0.011036
      0.013076
      -0.011962
      0.022730
      0.003479
      0.003800
      0.002710
      0.005065
      0.000988
      0.010169
      0.005657
      -0.002030
      -0.009120
      0.002555
      0.018161

      BetaMkt
      0.693293
      0.845470
      0.625330
      0.912283
      0.937398
      0.464959
      0.501213
      1.176732
      0.803426
      0.509535
      0.503625
      0.448914
      0.345137
      0.519122
      0.481308
      0.260073
      0.731883
      0.926187
      0.416506
      0.660433

      BetaSMB
      0.279470
      0.693865
      0.041670
      -0.395494
      0.345019
      -0.517202
      -0.700818
      0.099676
      0.078710
      -0.423302
      -0.418416
      -0.324102
      -0.809450
      -0.296927
      0.898115
      -0.570457
      0.007746
      -0.141485
      -0.645586
      -0.413393

      BetaHML
      0.140517
      0.423802
      0.674928
      0.284910
      1.700849
      -0.157164
      -0.524454
      0.331936
      -0.206294
      -0.576125
      -0.359140
      -0.289612
      -0.207932
      -0.411242
      0.542230
      -0.284763
      1.046295
      0.297058
      0.039949
      0.006759

      R2_adj
      0.187997
      0.336439
      0.178910
      0.294399
      0.356713
      0.103455
      0.223414
      0.432733
      0.213924
      0.312858
      0.164864
      0.107853
      0.165120
      0.100622
      0.155472
      0.042623
      0.295308
      0.216747
      0.099964
      0.169966

```python
# Transpose matrix (.T) and display stats summary
print(ff3f.T.describe())
```

               Alpha    BetaMkt    BetaSMB    BetaHML     R2_adj
    count  20.000000  20.000000  20.000000  20.000000  20.000000
    mean    0.003256   0.638141  -0.160618   0.123625   0.207969
    std     0.008962   0.235797   0.461006   0.566272   0.101621
    min    -0.011962   0.260073  -0.809450  -0.576125   0.042623
    25%    -0.001940   0.477221  -0.446777  -0.285975   0.143567
    50%     0.003095   0.572226  -0.310515   0.023354   0.183454
    75%     0.006785   0.813937   0.083951   0.354903   0.294626
    max     0.022730   1.176732   0.898115   1.700849   0.432733

### Hedge a Portolio

Now that we have all the factors, let's assume we want to build a portfolio with all 20 SMI stocks that maximizes the Sharpe ratio (SR). As a further condition we want to limit our exposure to the SMB factor to, let's say, $\beta_p^{SMB} = 0$. How would we allocate our investment under these conditions? In mathematical terms we have the following optimization problem:

$$\begin{equation}
\max_{w_i} SR = \frac{\mathbb{E}[r_p] - r_f}{\sigma_p} \qquad s.t. \qquad
\begin{cases}
\sum w_i &= 1 \\
\beta_p^{SMB} &= 0
\end{cases}
\end{equation}$$

Usually, to calculate the expected return $\mathbb{E}[r_p]$, historical returns are taken. For our case here, we will take the expected returns given by our Fama-French 3 Factor model (denoted $\mathbf{R_{ff}}$). The portfolio variance $\sigma_p$ however, we estimate using historical data. Alternatively one could think of taking the SMI volatility index value as proxy. But this is only approximately true because we will not have the same weights per stock as the SMI and this thus might be questionable. With that we have

$$\begin{equation}
\max_{w_i} SR = \frac{\mathbf{w}^T \left(r_f + \mathbf{\alpha} + \mathbf{\beta}^{M} (r_{M} - r_{f}) + \mathbf{\beta}^{SMB} SMB + \mathbf{\beta}^{HML} HML \right) - r_f}{\mathbf{w}^T \mathbf{\Sigma}\mathbf{w}} \qquad s.t. \qquad
\begin{cases}
\sum w_i &= 1 \\
\beta_p^{SMB} &= 0
\end{cases}
\end{equation}$$

Python can solve this problem numerically. We first set the stage by defining a auxiliary function `pfStats` that returns the expected portfolio return, volatility and Sharpe ratio given a vector of weights. Note that the function also makes use of other data like monthly returns and the riskfree rate as previously defined (which is set to 0) but only weights are a function input value. This is necessary for the optimization function.

```python
# Define rf and (equally spread) start weights
rf = 0
wghts = np.repeat(1. / len(tickers), len(tickers))

# Expected stock returns based on ff3f model
expShsRets = rf + ff3f.T.Alpha + \
    ff3f.T.BetaMkt * ff['Mkt-RF'].mean() + \
    ff3f.T.BetaSMB * ff.SMB.mean() + \
    ff3f.T.BetaHML * ff.HML.mean()
```

```python
def pfStats(weights):
    '''Returns basic measures for a portfolio

    Parameters
    ==========
    weights : array-like
        weights for different securities in portfolio

    Returns
    =======
    expPfRet : float
        weighted, annualized expected portfolio return based on ff3f model
    pfVol : float
        historical annualized portfolio volatility
    SR : float
        portfolio Sharpe ratio for given riskfree rate
    '''

    expPfRet = np.sum(weights * expShsRets) * 12
    pfVol    = np.sqrt(np.dot(weights.T, np.dot(shsRets.cov() * 12, weights)))
    SR       = (expPfRet - rf) / pfVol

    return np.array([expPfRet, pfVol, SR])
```

Python's scipy package has a sublibrary for constrained optimization problems. We will use the minimize function and minimize the negative value of the Sharpe ratio (which is obviously equal to maximizing the SR)

```python
import scipy.optimize as sco

def minSR(wghts):
    return -pfStats(wghts)[2]
```

Our constraints are as stated above: $\sum w_i = 1$, $\beta_P^{SMB} = 0$. Additionally we set bounds for the weights such that short/long position are allowed but only up to 100% per share ($w_i \in [-1, 1]\; \forall i \in [1, 2, \ldots, n]$).

```python
# Constraints and bounds
constr = [{'type': 'eq', 'fun': lambda x: np.sum(x) - 1},
          {'type': 'eq', 'fun': lambda x: np.sum(x * ff3f.T.BetaSMB) - 0}]
bnds   = tuple((-1,1) for x in range(len(tickers)))
```

```python
# Minimization function
optPf = sco.minimize(minSR, x0=wghts, method='SLSQP', bounds=bnds, constraints=constr)
```

```python
# Check if conditions are actually met
print('Sum of weights:  ', np.sum(optPf['x']))
print('Beta SMB factor: ', np.sum(optPf['x'] * ff3f.T.BetaSMB))
```

    Sum of weights:   1.0
    Beta SMB factor:  1.3223311334797927e-12

```python
# Calculate portfolio stats given optimal weights
rsltsOptPf = pfStats(optPf['x'])

# Format weights into dataframe with Tickers as heading
optWghts = pd.DataFrame(data=optPf['x'], index=tickers)
optWghts.columns = ['optimalWghts']

# Print results
print('Portfolio return:     ', str(rsltsOptPf[0]))
print('Portfolio volatility: ', str(rsltsOptPf[1]))
print('Portfolio SR:         ', str(rsltsOptPf[2]), '\n')
print(str(optWghts))
```

    Portfolio return:      0.6625784538904684
    Portfolio volatility:  0.2005679728729619
    Portfolio SR:          3.3035107470031626

          optimalWghts
    ABBN     -0.054099
    ADEN     -0.257005
    BAER      0.099357
    CFR       0.263547
    CSGN     -0.191775
    GEBN      0.078464
    GIVN      0.606703
    LHN      -0.465790
    LONN      0.525160
    NESN      0.233943
    NOVN      0.392504
    ROG      -0.679709
    SCMN     -0.118646
    SGSN     -0.266730
    SLHN      0.739888
    SREN     -0.049775
    UBSG      0.078083
    UHR      -0.280616
    ZURN     -0.466523
    SIK       0.813018

# Further Ressources

In writing this notebook, many ressources were consulted. For internet ressources the links are provided within the textflow above and will therefore not be listed again. Beyond these links, the following ressources were consulted and are recommended as further reading on the discussed topics:

* Blume, Marshall E., 1971, On the Assessment of Risk, *The Journal of Finance* 26, 1-10.
* Fama, Eugene F, and Kenneth R French, 1993, Common risk factors in the returns on stocks and bonds, *Journal of Financial Economics* 33, 3–56.
* Hilpisch, Yves, 2019, Python for Finance (O'Reilly Media, Sebastopol, CA).
* James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani, 2013, *An Introduction to Statistical Learning: With Applications in R* (Springer Science & Business Media, New York, NY).
* Müller, Andreas C., and Sarah Guido, 2017, *Introduction to Machine Learning with Python* (O’Reilly Media, Sebastopol, CA).
* Pinto, Jerald E., Elaine Henry, Thomas R. Robinson, and John D. Stowe, 2016, *Equity Asset Valuation* (MIT Press, Cambridge, MA).
* Ross, Stephen A., et al., 1973, Return, risk and arbitrage (Rodney L. White Center for Financial Research, The Wharton School, University of Pennyslvania).
*   Sheppard, Kevin, 2017, Introduction to Python for Econometrics, Statistics and Data Analysis from Website https://www.kevinsheppard.com/images/b/b3/Python_introduction-2016.pdf, 07/07/2017.
* Wooldridge, Jeffrey M, 2015, *Introductory Econometrics: A modern approach* (Cengage Learning, Boston, MA).


# Voltage deviation Minimization via PandaModels.jl
### PandaModels.jl: Interfacing PowerModels with pandapower

This tutorial describes how to run the Voltage Deviation minimization via [PandaModels.jl](https://e2niee.github.io/PandaModels.jl/dev/).
Voltage Deviation minimization is an optimization problem which is exclusively defined in [PandaModels.jl](https://e2niee.github.io/PandaModels.jl/dev/) and
needs the extra user defined parameters from pandapower.

The general mathematical model of VD is defined as follows:

\begin{align}
& \underset{\mathcal{X} = [q, ...]}{\text{minimize}}
& &  \sum_{i\in \mathcal{BI}} [v_i - setpoint\_v]^2 \\
& \text{subject to}
& & g(\mathcal{X})=0 \\
& & & h(\mathcal{X}) \leq 0
\end{align}

where $v_{i}$ is the voltage variable of bus $i$ in $\mathcal{BI}$ which denotes the set of buses located at the DSO-TSO interfaces. The $g(\mathcal{X})$ and $h(\mathcal{X})$, denote equality and inequality constraints, respectively. The $\mathcal{X}$ denotes the set of variables decisions, such as reactive power, $q$.

### Let's get started

So here is an example of how it works. First, we create cigre grid with pv and wind DERs(distribution energy resource) from pandapower's network database.

```python
import pandapower as pp
import pandapower.networks as nw
from copy import deepcopy

net = nw.create_cigre_network_mv(with_der="pv_wind")
```

then we need to run powerflow to get the initial values:

```python
net.sgen.p_mw = net.sgen.p_mw * 8
net.sgen.sn_mva = net.sgen.sn_mva * 8
pp.runpp(net)
```

Let's have a look at the grid we created with pandapower plotting module:

```python
import pandapower.plotting as plot
%matplotlib inline
plot.simple_plot(net)
```

let's keep a copy of the net for further comparision:

```python
net_org = deepcopy(net)
```

Although VD is not a function of PowerModels and is exclusively modeled in Pandamodels, PandaModels similar to PowerModels follows the data structures from [InfrastructureModels.jl](https://github.com/lanl-ansi/InfrastructureModels.jl).
 In this data structure DERs are not defined separately unlike pandapower,
 then the uncontrollable load and sgens are converted to load and controllable elements are converted to the generator.

Accordingly, we need to set loads as uncontrollable elements and set DERs (sgens) as controllable elements.

```python
net.load['controllable'] = False
net.sgen['controllable'] = True
```

now, lets set the optimization boundaries:

```python
# lower and upper bounds for buses
net.bus["max_vm_pu"] = 1.1
net.bus["min_vm_pu"] = 0.9

# lower and upper bounds for external grid
net.ext_grid["max_q_mvar"] = 10000.0
net.ext_grid["min_q_mvar"] = -10000.0
net.ext_grid["max_p_mw"] = 10000.0
net.ext_grid["min_p_mw"] = -10000.0
```

in VD we need to keep the active power value for DER and generators constant to the calculated value from power flow.

```python
# lower and upper bounds for DERs
net.sgen["max_p_mw"] = net.sgen.p_mw.values
net.sgen["min_p_mw"] = net.sgen.p_mw.values
net.sgen["max_q_mvar"] = net.sgen.p_mw.values * 0.328
net.sgen["min_q_mvar"] = -net.sgen.p_mw.values * 0.328

# lower and upper bounds for generators
net.gen["max_p_mw"] = net.gen.p_mw.values
net.gen["min_p_mw"] = net.gen.p_mw.values
net.gen["max_q_mvar"] = 10000.0
net.gen["min_q_mvar"] = -10000.0
```

Let's set a high upper bound for line and transformers to avoid congestion issue:

```python
# lower and upper bounds for lines
net.trafo["max_loading_percent"] = 500.0
net.line["max_loading_percent"] = 500.0
```

I also we could set costs for sgens, gens and external grids in order to not get warnings during data correctness in julia side.

```python
for idx in net.sgen.index:
    pp.create_poly_cost(net, idx, "sgen", 1.0)
for idx in net.gen.index:
    pp.create_poly_cost(net, idx, "gen", 1.0)
for idx in net.ext_grid.index:
    pp.create_poly_cost(net, idx, "ext_grid", 1.0)
```

Finally, lets add our user defined parameter "setpoint_v". at first we need to add extra column called "pm_param/setpoint_v" in bus data,
then we set values for buses contain DERs(sgens):

```python
net.bus["pm_param/setpoint_v"] = None
net.bus["pm_param/setpoint_v"].loc[net.sgen.bus] = 0.99
```

Now lets run an VD through PandaModels and look at the results (Note that the first time the function is called, Julia is started in the background, which may take some time):

```python
pp.runpm_vd(net)
```

    the json file for convertet net is stored in:  C:\Users\e2n037\AppData\Local\Temp\pp_to_pm_r2m2vooy.json
    the json file for convertet net is deleted from  C:\Users\e2n037\AppData\Local\Temp\pp_to_pm_r2m2vooy.json

Also, there more parameters and options that you can add as input while calling the Optimization Problem from PandaModles:

| parameter | description | type | default |
| :--- | :--- | :---: | :--- |
| correct_pm_network_data | checks if network data is correct. If not tries to correct it | bool | True |
| silence | Suppresses information and warning messages output by PowerModels | bool | True |
| pm_model | PowerModels.jl model to use | str | "ACPPowerModel" |
| pm_solver | "main" solver| str | "ipopt" |
| pm_mip_solver | mixed integer solver| str | "cbc" |
| pm_nl_solver | nonlinear solver| str | "ipopt" |
| pm_tol | default desired convergence tolerance for solver to use | float | 1e-8 |
| pm_log_level | solver log level in power models | int | 0 |
| delete_buffer_file | If True, the .json file used by PandaModels will be deleted after optimization. | bool | True |

lets check the result values for voltage and reactive power at buses contains DERs before and after VD:

```python
# results from power flow
net_org.res_bus.vm_pu[net_org.sgen.bus]
```

    3     1.039473
    4     1.038293
    5     1.037487
    6     1.036603
    8     1.053060
    9     1.052446
    10    1.051725
    11    1.051589
    7     1.075642
    Name: vm_pu, dtype: float64

```python
# results from VD
net.res_bus.vm_pu[net.sgen.bus]
```

    3     0.984793
    4     0.983774
    5     0.983079
    6     0.982361
    8     0.992054
    9     0.991521
    10    0.990934
    11    0.990804
    7     1.006539
    Name: vm_pu, dtype: float64

as exected the results are close to the defined setpoints, and as we set the active power constants we expect to get different results from power flow for reactive power:

```python
# results from power flow
net_org.res_sgen.q_mvar
```

    0    0.0
    1    0.0
    2    0.0
    3    0.0
    4    0.0
    5    0.0
    6    0.0
    7    0.0
    8    0.0
    Name: q_mvar, dtype: float64

```python
# results from VD
net.res_sgen.q_mvar

```

    0    0.052467
    1    0.052470
    2    0.078711
    3    0.078712
    4    0.078694
    5    0.078693
    6    0.104932
    7    0.026212
    8   -3.135065
    Name: q_mvar, dtype: float64


```
c=4
dx=0.2
sigma=1
```

```
dt = (sigma*dx)/c
print(dt)
```

    0.05

```
import sympy
import numpy
%matplotlib inline
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
from sympy import init_printing
from sympy.utilities.lambdify import lambdify
init_printing()
```

```
x = sympy.symbols('x')
```

```
f=((sympy.cos(x)**2)*(sympy.sin(x)**3))/(4*(x**5)*sympy.exp(x))
f
```

```
fprime=f.diff(x)
fprime
```

```
fprime.subs(x,2.2).evalf()
```

```

```


```python
import sympy
```

# Mersenne Prime (메르센 소수)

### 메르센?
- 마린 메르센(1588-1648)은 프랑스 수도승으로 페르마, 파스칼, 갈릴레오 등과 동시대를 산 인물
- 당시에는 수학 저널 같은 것이 없어서 메르센은 수학에서의 발견을 받아 정리하고 배포하는 역할(clearing house)을 했다고 함
- 물론 본인도 철학자이면서 수학자

### 소수를 공식으로 표현하고자 하는 노력
- 무한대의 소수가 있음을 증명할 수 있는 간단한 방법
- $x^{2} + x + 41$이 대표적

```python
[i for i in range(1, 50) if not sympy.ntheory.isprime(i * i + i + 41)]

```

    [40, 41, 44, 49]

1~39까지는 참이지만 40부터 성립하지 않는 결과가 나오기 시작

### Mersenne numbers and primes
- 메르센도 공식을 하나 만들었다. $M_{n} = 2^{n} - 1$
- 이 수를 Mersenne numbers라고 불렀고 이 중 소수인 숫자가 Mersenne prime이라고 불린다
- $n$이 소수가 아니면 $M_{n}$도 소수가 아니다
- $n$이 소수면?

```python
primes = [sympy.prime(n) for n in range(1, 9)]
[(n, 2 ** n - 1, sympy.ntheory.isprime(2 ** n - 1)) for n in primes]
```

    [(2, 3, True),
     (3, 7, True),
     (5, 31, True),
     (7, 127, True),
     (11, 2047, False),
     (13, 8191, True),
     (17, 131071, True),
     (19, 524287, True)]

- 11을 제외하고 19까지 공식이 성립한 걸 본 메르센은 19 이상의 소수에 대해 조사를 하기 시작 ~~수도승은 시간이 많았겠...~~
- $M_{31}$, $M_{67}$, $M_{127}$, $M_{257}$이 소수라고 결론

```python
[(n, 2 ** n - 1, sympy.ntheory.isprime(2 ** n - 1)) for n in [31, 67, 127, 257]]
```

    [(31, 2147483647, True),
     (67, 147573952589676412927, False),
     (127, 170141183460469231731687303715884105727, True),
     (257,
      231584178474632390847141970017375815706539969331281128078915168015826259279871,
      False)]

- 메르센이 어떻게 이 숫자 찾은 과정은 알려지지 않음
- $M_{31}$만 해도 10자리 숫자로 아주 큰 숫자 ~~당시 기술로는 ...~~

### 그 후
- 1750년 오일러 형님이 $M_{31}$은 소수가 맞다고 증명
- 1876년 Edouard Lucas가 $M_{67}$이 소수가 아니라고 발표했지만 어떤 수로 소인수분해되는지는 밝히지 못함
- 이후 Derrick Lehmer라는 사람이 Lucas의 방법을 간결하게 개선

```python
def lucas(p):
    U = 4
    Mp = 2 ** p - 1
    for _ in range(p - 2):
        U = (U * U - 2) % Mp
    return U == 0
```

```python
[lucas(p) for p in (31, 67, 127, 257)]
```

    [True, False, True, False]

- 된다! 근데 왜 되는거지? ~~저는 몰라요~~
- 그 뒤로도 $M_{61}$, $M_{89}$, $M_{107}$이 메르센 소수임이 밝혀짐
- 결론적으로 메르센 형님의 찍기는 반타작: $M_{31}$(O), $M_{67}$(X), $M_{127}$(O), $M_{257}$(X) ~~많은 수학자들을 낚아 이걸 증명하게 만든 것은 큰 그림~~

### $M_{67}$의 소인수 분해
- Lucas의 방법으로 $M_{67}$이 소수가 아닌 건 알았지만 이 수는 어떤 수들의 곱인지는 밝혀지지 않은 상황
- 1903년, 수학자 Frank Nelson Cole(1861-1926)은 American Mathematical Society에서 'On the Factorisation of Large Numbers'라는 강의를 하게 되는데...
- 아무말 하지 않고 $2^{67}$을 계산하기 시작, 그리고 1을 신중하게(carefully) 뺀다
- 이 숫자는 무려

```python
m67 = 2**67 - 1
f'{m67:_}'
```

    '147_573_952_589_676_412_927'

- 그리고 다른 칠판에 193,707,721와 761,838,257,287를 쓰고 곱하기 시작

```python
193_707_721 * 761_838_257_287
```

    147573952589676412927

- 아무 말 없이 계산을 끝냈고, Cole은 기립 박수를 받음
- Cole이 후에 밝히길 이걸 찾아내는데 매주 일요일을 투자했고 3년이 걸렸다고 함

```python
sympy.ntheory.isprime(2**67 - 1)
```

    False

```python
%time
print(sympy.factorint(2**67 - 1))
```

    Wall time: 0 ns
    {193707721: 1, 761838257287: 1}

- 컴퓨터가 발전하면서 그 뒤로 많은 메르센 소수가 발견되었음
- [GIMPS(Great Internet Mersenne Prime Search) 프로젝트](https://www.mersenne.org/)가 있고 Prime95라는 소프트웨어를 제공
- 이 소프트웨어는 CPU & GPU의 성능을 최대로 활용하기 때문에 오버클럭 후 안정성 테스트에 사용하기도 함
- 2018년 1월 3일 50번째 메르센 소수가 발견됨 : $2^{77,232,917}-1$
- 23,249,425 digits
- 발견 당시 인류가 알고 있는 가장 큰 소수
- 50번째 메르센 발견 시점에서 **증명된 가장 큰 메르센 소수**는 $M_{37156667}$로 45번째 메르센 소수 (발견된 건 8년 전)

- 2018년 12월 7일에 51번째 메르센 소수 발견 : $2^{82,589,933}-1$
- 24,862,048 digits
- Volunteer인 Patrick Laroche(from Ocala, Florida)가 발견했는데, 소수를 찾겠다는 의지가 아니라 무료 스트레스 테스트로 활용했던 것
- 51번째 메르센 소수 발견 시점에서 47번째 메르신 소수까지 공식 증명됨 (발견된 건 9년 전)


```julia
using DifferentialEquations
using Plots
using LaTeXStrings
pgfplotsx()
```

    Plots.PGFPlotsXBackend()

Consider the simple reaction:

\begin{align}
    A &\longrightarrow B\\
\end{align}

The reaction is elementary, and B acts as a solvent for A and the transition state. We will consider it in a few different solvent environments.

```julia
function batch(du, u, p, t)
    MAR = p["MAR"]
    PAR = p["PAR"]

    k10, V = PAR

    NA  = u[:,1]
    NB  = u[:,2]

    NT  = NA + NB

    XA   = NA./NT
    XB   = NB./NT

    #For A in solvent
    A12A = MAR[1]

    #For B in solvent
    A12B = MAR[2]

    #For Transition State 1 in solvent
    A12TS1 = MAR[3]

    gammaA  = exp.(XB.^2*A12A)
    gammaB  = exp.(XA.^2*A12B)

    gammaTS1 = exp.(XB.^2*A12TS1)

    r1 = k10*gammaA./gammaTS1.*XA.*NT/V

    aA = gammaA.*XA

    RA  = -r1[1]
    RB  =  r1[1]

    du[1] = RA*V
    du[2] = RB*V
    return du, r1, aA
end
```

    batch (generic function with 1 method)

```julia
k10  = 1
V    = 1

NA0  = 1
NB0  = 0

var0     = [NA0 NB0]
span     = (0.0, 1000.0);
```

```julia
#Solvate transition state relative to reactants
MARSET1 = zeros(3,3)
MARSET1[:,1] = [0.0,  0.0,  0.0] #no solvation
MARSET1[:,2] = [0.0,  0.0,  2.0] #destabilize TS1
MARSET1[:,3] = [0.0,  0.0, -2.0] #stabilize TS1

tfine  = 10 .^(range(-3.0, stop = log10(maximum(span)), length = 1000))
Xout   = zeros(length(tfine), size(MARSET1, 2))
r1out  = zeros(length(tfine), size(MARSET1, 2))
NAout  = zeros(length(tfine), size(MARSET1, 2))
NBout  = zeros(length(tfine), size(MARSET1, 2))
aAout  = zeros(length(tfine), size(MARSET1, 2))

for i = 1:size(MARSET1, 2)
    p0   = Dict("MAR" => MARSET1[:,i], "PAR" => [k10, V])
    prob = ODEProblem(batch, var0, span, p0)
    sol  = solve(prob, Rodas5(), abstol = 1e-18, reltol = 1e-18)
    solf = sol(tfine)
    NA   = solf[1,:]
    NB   = solf[2,:]
    NT   = NA + NB
    ext  = NB/NA0
    dut, rt1, aAt = batch([0., 0.], [NA NB], p0, tfine)

    Xout[:,i]  = ext
    r1out[:,i] = rt1
    NAout[:,i] = NA
    NBout[:,i] = NB
    aAout[:,i] = aAt
end

```

```julia
#Create Plot 1
plt1 = plot(
    xscale  = :log10,
    xlim    = (1e-6, 1),
    xminorticks = 4,
    ylim = (0.0, 1.0),
    yticks = (0.0:0.25:1.0),
    #yticks = (-2.0:1.0:2.0, [@sprintf("%1.1f", x) for x in -2.0:1.0:2.0]),
    yminorticks = 4,
    tick_direction = :out,
    grid = :off,
    guidefontsize = 26,
    tickfontsize = 20,
    xlabel  = "time",
    ylabel  = "Conversion",
    fontfamily = "Times Roman",
    legendfontsize = 16,
    legendfonthalign = :left,
    legend  = :bottomright,
    title = "(b)",
    titlelocation = :left,
    titlefontsize = 26
)

plt1 = plot!(plt1, tfine/maximum(tfine), Xout[:,3],  ls = :dash,       lc = :blue,  lw = 1.00, label = "Stabilize TS")
plt1 = plot!(plt1, tfine/maximum(tfine), Xout[:,1],  ls = :solid,      lc = :black, lw = 0.75, label = "Ideal Solution")
plt1 = plot!(plt1, tfine/maximum(tfine), Xout[:,2],  ls = :dashdotdot, lc = :red,   lw = 1.00, label = "Destabilize TS")
display(plt1)
savefig(plt1, "CS1b1.pdf")
```

```julia
idx1 = r1out[:,1] .>= 0
idx2 = r1out[:,2] .>= 0
idx3 = r1out[:,3] .>= 0

plt2 = plot(
    xscale  = :log10,
    xlim    = (1e-6, 1),
    xminorticks = 4,
    ylim = (0.0, 1.0),
    yticks = (0.0:0.25:1.0),
    #yticks = (-2.0:1.0:2.0, [@sprintf("%1.1f", x) for x in -2.0:1.0:2.0]),
    yminorticks = 4,
    tick_direction = :out,
    grid = :off,
    guidefontsize = 26,
    tickfontsize = 20,
    xlabel  = "time",
    ylabel  = "rate",
    fontfamily = "Times Roman",
    legendfontsize = 16,
    legendfonthalign = :left,
    legend  = :bottomleft,
    title = "(a)",
    titlelocation = :left,
    titlefontsize = 26
)

plt2 = plot!(plt2, tfine[idx2]/maximum(tfine), r1out[:,2][idx2], ls = :dashdotdot, lc = :red,   lw = 1.00, label = "Destabilize TS")
plt2 = plot!(plt2, tfine[idx1]/maximum(tfine), r1out[:,1][idx1], ls = :solid,      lc = :black, lw = 0.75, label = "Ideal Solution")
plt2 = plot!(plt2, tfine[idx3]/maximum(tfine), r1out[:,3][idx3], ls = :dash,       lc = :blue,  lw = 1.00, label = "Stabilize TS")
display(plt2)
savefig(plt2, "CS1b2.pdf")
```

```julia
idx1 = Xout[:,1] .< 0.9
idx2 = Xout[:,2] .< 0.9
idx3 = Xout[:,3] .< 0.9

plt3 = plot(
    xlim    = (0.0, 0.006),
    xminorticks = 4,
    ylim = (0.0, 2.5),
    yminorticks = 4,
    tick_direction = :out,
    grid = :off,
    guidefontsize = 26,
    tickfontsize = 20,
    xlabel  = "time",
    ylabel  = L"-\textrm{ln}(1 - X_A)",
    fontfamily = "Times Roman",
    legendfontsize = 16,
    legendfonthalign = :left,
    legend  = :bottomright,
    title = "(c)",
    titlelocation = :left,
    titlefontsize = 26
)

plt3 = plot!(plt3, tfine[idx3]/maximum(tfine), -log.(1 .- Xout[:,3][idx3]), ls = :dash,       lc = :blue,  lw = 1.00, label = "Stabilize TS")
plt3 = plot!(plt3, tfine[idx1]/maximum(tfine), -log.(1 .- Xout[:,1][idx1]), ls = :solid,      lc = :black, lw = 0.75, label = "Ideal Solution")
plt3 = plot!(plt3, tfine[idx2]/maximum(tfine), -log.(1 .- Xout[:,2][idx2]), ls = :dashdotdot, lc = :red,   lw = 1.00, label = "Destabilize TS")

display(plt3)
savefig(plt3, "CS1b3.pdf")
```

```julia
plt4 = plot(
    xlim    = (0.0, 1.0),
    xticks  = 0.0:0.25:1.0,
    xminorticks = 4,
    ylim = (0.0, 1.0),
    yticks = 0.0:0.25:1.0,
    yminorticks = 4,
    tick_direction = :out,
    grid = :off,
    guidefontsize = 26,
    tickfontsize = 20,
    xlabel  = "Conversion",
    ylabel  = "rate",
    fontfamily = "Times Roman",
    legendfontsize = 16,
    legendfonthalign = :left,
    legend  = :bottomleft,
    title = "(d)",
    titlelocation = :left,
    titlefontsize = 26
)

plt4 = plot!(plt4, Xout[:,2], r1out[:,2], ls = :dashdotdot, lc = :red,   lw = 1.00, label = "Destabilize TS")
plt4 = plot!(plt4, Xout[:,1], r1out[:,1], ls = :solid,      lc = :black, lw = 0.75, label = "Ideal Solution")
plt4 = plot!(plt4, Xout[:,3], r1out[:,3], ls = :dash,       lc = :blue,  lw = 1.00, label = "Stabilize TS")
display(plt4)
savefig(plt4, "CS1b4.pdf")
```

```julia
#Solvate transition state relative to reactants
MARSET1 = zeros(3,3)
MARSET1[:,1] = [0.0,   0.0,  0.0] #no solvation
MARSET1[:,2] = [2.0,   0.0,  0.0] #destabilize TS1
MARSET1[:,3] = [-2.0,  0.0,  0.0] #stabilize TS1

tfine  = 10 .^(range(-3.0, stop = log10(maximum(span)), length = 1000))
Xout   = zeros(length(tfine), size(MARSET1, 2))
r1out2  = zeros(length(tfine), size(MARSET1, 2))
NAout  = zeros(length(tfine), size(MARSET1, 2))
NBout  = zeros(length(tfine), size(MARSET1, 2))
aAout2  = zeros(length(tfine), size(MARSET1, 2))

for i = 1:size(MARSET1, 2)
    p0   = Dict("MAR" => MARSET1[:,i], "PAR" => [k10, V])
    prob = ODEProblem(batch, var0, span, p0)
    sol  = solve(prob, Rodas5(), abstol = 1e-18, reltol = 1e-18)
    solf = sol(tfine)
    NA   = solf[1,:]
    NB   = solf[2,:]
    NT   = NA + NB
    ext  = NB/NA0
    dut, rt1, aAt = batch([0., 0.], [NA NB], p0, tfine)

    Xout[:,i]  = ext
    r1out2[:,i] = rt1
    NAout[:,i] = NA
    NBout[:,i] = NB
    aAout2[:,i] = aAt
end

```

```julia
plt5 = plot(
    xlim    = (0.0, 1.0),
    xticks  = 0.0:0.25:1.0,
    xminorticks = 4,
    ylim = (0.0, 1.0),
    yticks = 0.0:0.25:1.0,
    yminorticks = 4,
    tick_direction = :out,
    grid = :off,
    guidefontsize = 26,
    tickfontsize = 20,
    xlabel  = "Activity of A",
    ylabel  = "rate",
    fontfamily = "Times Roman",
    legendfontsize = 16,
    legendfonthalign = :left,
    legend  = :bottomright,
    title = "(b)",
    titlelocation = :left,
    titlefontsize = 26
)

plt5 = plot!(plt5, aAout[:,3], r1out[:,3], ls = :dash,       lc = :blue,  lw = 1.00, label = "Stabilize TS")
plt5 = plot!(plt5, aAout[:,1], r1out[:,1], ls = :solid,      lc = :black, lw = 0.75, label = "Ideal Solution")
plt5 = plot!(plt5, aAout[:,2], r1out[:,2], ls = :dashdotdot, lc = :red,   lw = 1.00, label = "Destabilize TS")

display(plt5)
savefig(plt5, "CS1b5.pdf")
```

```julia
plt6 = plot(
    xlim    = (0.0, 1.0),
    xticks  = 0.0:0.25:1.0,
    xminorticks = 4,
    ylim = (0.0, 1.0),
    yticks = 0.0:0.25:1.0,
    yminorticks = 4,
    tick_direction = :out,
    grid = :off,
    guidefontsize = 26,
    tickfontsize = 20,
    xlabel  = "Activity of A",
    ylabel  = "rate",
    fontfamily = "Times Roman",
    legendfontsize = 16,
    legendfonthalign = :left,
    legend  = :bottomright,
    title = "(a)",
    titlelocation = :left,
    titlefontsize = 26
)

plt6 = plot!(plt6, aAout2[:,3], r1out2[:,3], ls = :dash,       lc = :blue,  lw = 1.00, label = "Stabilize Reactant")
plt6 = plot!(plt6, aAout2[:,1], r1out2[:,1], ls = :solid,      lc = :black, lw = 0.75, label = "Ideal Solution")
plt6 = plot!(plt6, aAout2[:,2], r1out2[:,2], ls = :dashdotdot, lc = :red,   lw = 1.00, label = "Destabilize Reactant")

display(plt6)
savefig(plt6, "CS1b6.pdf")
```