# Linear Regression

# Simple Linear Regression

Running a SLR in Python is fairly simple once you know how to use the relevant functions. What might be confusing is that there exist several packages which provide functions for linear regression. We will use functions from the `statsmodels` (sub-)package. Other packages such as e.g. `scikit-learn` have linear regression functions too, but what makes `statsmodels` stand out from other packages is its broad set of auxiliary functions for regression diagnostics. As usual we start by importing the packages needed for our task.

```python
# Load relevant packages
%matplotlib inline
import numpy as np
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
plt.rcParams['font.size'] = 14
```

    C:\Anaconda\lib\site-packages\statsmodels\compat\pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.
      from pandas.core import datetools

As toy data we will use the 'Advertising' data set in this section, introduced in the previous chapter. The data is taken from James et al. (2013). A copy is provided on the book's website where we will download it from.

```python
# From Advertising data set read cols 2:4
url = 'http://www-bcf.usc.edu/~gareth/ISL/Advertising.csv'
ad = pd.read_csv(url, sep=',', usecols=(np.arange(1, 5, 1)))
print(ad.head())
```

          TV  radio  newspaper  sales
    0  230.1   37.8       69.2   22.1
    1   44.5   39.3       45.1   10.4
    2   17.2   45.9       69.3    9.3
    3  151.5   41.3       58.5   18.5
    4  180.8   10.8       58.4   12.9

Next we run a linear regression to calculate the coefficients and print a summary output.

```python
# Run regression and calculate fit
reg = sm.OLS(ad.sales, exog=sm.add_constant(ad.TV)).fit()
# Alternatively: reg = sm.OLS(ad.sales, sm.add_constant(ad.TV)).fit()
print(reg.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  sales   R-squared:                       0.612
    Model:                            OLS   Adj. R-squared:                  0.610
    Method:                 Least Squares   F-statistic:                     312.1
    Date:                Sun, 11 Feb 2018   Prob (F-statistic):           1.47e-42
    Time:                        20:34:34   Log-Likelihood:                -519.05
    No. Observations:                 200   AIC:                             1042.
    Df Residuals:                     198   BIC:                             1049.
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          7.0326      0.458     15.360      0.000       6.130       7.935
    TV             0.0475      0.003     17.668      0.000       0.042       0.053
    ==============================================================================
    Omnibus:                        0.531   Durbin-Watson:                   1.935
    Prob(Omnibus):                  0.767   Jarque-Bera (JB):                0.669
    Skew:                          -0.089   Prob(JB):                        0.716
    Kurtosis:                       2.779   Cond. No.                         338.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

**Side note:** If you are the `R` guy and prefer their syntax, you could import the statsmodels.formula.api subpackage and run something like:

> `from statsmodels.formula.api import ols
reg = ols("Sales ~ TV", data=ad).fit()
reg.summary()`

Intercept will automatically be calculated in the above setting.

Both p-values for intercept ($\hat{\beta}_0$) and slope ($\hat{\beta}_1$) are smaller than any reasonable significance level and thus we can reject the null hypothesis that either of the coefficients is zero (or irrelevant).

Instead of printing the whole summary, we could also access each of the three summary as follows:

```python
reg.summary().tables[1]
```

    	       coef	     std err	      t	      P>|t|	  [0.025	    0.975]

  const	     7.0326	     0.458	    15.360	  0.000	     6.130	     7.935

  TV	        0.0475	     0.003	    17.668	  0.000	     0.042	     0.053

## Plotting the Fit

Many details were calculated by calling the `sm.OLS` function. All information is attached to the `reg` object. For example to plot the data we would need the model's fitted values. These can be accessed by combining the regression variable/object with the attribute `.fittedvalues` as in `reg.fittedvalues`. In below plot it is shown how this can be of use. We plot the data and fit using the standard plotting functions.

```python
# Plot scatter & lm
plt.figure(figsize=(12,8))
plt.scatter(ad.TV, ad.sales, marker='.', label='Sample')
plt.plot(ad.TV, reg.fittedvalues, c='k', label='Fit')
plt.ylabel('Sales')
plt.xlabel('TV')
plt.legend();
```

Let us plot residuals versus fitted values to do some visual regression diagnostics:

```python
plt.figure(figsize=(12, 8))
plt.scatter(ad.TV, reg.resid)
plt.axhline(y=0, c='k')
plt.xlabel('TV (fitted)')
plt.ylabel('Residuals');
```

The above two plots are just standard `matplotlib` plots serving the purpose of visual diagnostics. Beyond that, the `statsmodel` package has separate built-in plotting functions suited for visual regression diagnostics. We will not discuss them here but if interested you'll find plenty of sources on the web (e.g. [here](http://www.statsmodels.org/dev/examples/notebooks/generated/regression_plots.html), [here](http://www.statsmodels.org/stable/graphics.html) or [here](http://mpastell.com/2013/04/19/python_regression/))

## Accessing the Output

The `reg` object contains a ton of information which you can all access. To see what there is, type `reg.` and press tab. Two examples are shown below. Notice that some are attributes (like `x.shape`, `x.size`) and don not need paranthesis to call them. Others are methods (similar to `.sum()`, `.min()`) and require parenthesis.

```python
# Regression coefficients
print(reg.params, '\n')
print(reg.resid.head())
```

    const    7.032594
    TV       0.047537
    dtype: float64

    0    4.129225
    1    1.252026
    2    1.449776
    3    4.265605
    4   -2.727218
    dtype: float64

# Confidence Intervals & Hypthesis Testing

The 95%-confidence interval (CI) is printed in the summary above. If one wishes to calculate it for a different significance level (`alpha`), it is done as follows:

```python
# 99% CI (alpha = 1%) based on t-distribution
reg.conf_int(alpha=0.01)
```

      0
      1

      const
      5.841796
      8.223391

      TV
      0.040539
      0.054535

The regression summary provides $t$-statistic and p-value for the null hypothesis $H_0: \hat{\beta}_j = 0$, $H_a: \hat{\beta}_j \neq 0$. You can call the resulting $t$-statistic and p-value with its attributes.

```python
print(reg.tvalues, '\n')
print(reg.pvalues)
```

    const    15.360275
    TV       17.667626
    dtype: float64

    const    1.406300e-35
    TV       1.467390e-42
    dtype: float64

If you wish to test a different null hypothesis, e.g. $H_0: \hat{\beta}_{TV} = 0.054$ vs. $H_1: \hat{\beta}_{TV} \neq 0.054$ use the following code:

```python
reg.t_test('TV=0.054')
```

    <class 'statsmodels.stats.contrast.ContrastResults'>
                                 Test for Constraints
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    c0             0.0475      0.003     -2.402      0.017       0.042       0.053
    ==============================================================================

As far as I know, `Statsmodels` does not provide a function to calculate 'greater than' or 'smaller than' alternative hypothesis. Reason being: because with symmetric distributions, the one-sided test can be backed out from the two-sided test. A one-sided p-value is just half of the two-sided p-value. This means that given p and $t$ values from a two-tailed test, you would reject the null hypothesis of a greater-than test when p/2 < alpha and $t$ > 0, and of a less-than test when p/2 < alpha and $t$ < 0.

## Coefficient of Determination

The $R^2$ measure, or "coefficient of determination", displays the proportion of the variability in $y$ that is well explained by the regression fit. It is defined as

$$\begin{equation}
R^2 = \frac{TSS - SSR}{TSS} = 1 - \frac{SSR}{TSS}
\end{equation}$$

where TSS is the *total sum of squares*, defined as $TSS = \sum (y_i - \bar{y})^2$, and SSR is the *sum of squared residuals*, given by $SSR = \sum (y_i - \hat{y}_i)^2$.

It is easy to call the $R^2$ value from the regression object `reg` as the following line shows.

```python
# R squared measure
reg.rsquared
```

    0.61187505085007099

## Regression Diagnostics
### Test of Homoskedasticity

In general we assume a constant variance of the error term (homoskedasticity; $Var(\epsilon_i) = \sigma^2$ for $i = 1, \ldots, N$). From the residuals vs. fitted plot we have to question this assumption. To test it mathematically, you can run a heteroskedasticity test. The stats package offers several test options; the more common ones are White's or the one from Breusch-Pagan. See [here for more details](http://www.statsmodels.org/dev/diagnostic.html) on tests on heteroskedasticity.

Below the White test is applied as an example. The parameter 'reg.model.exog' simply contains the X matrix (here a 200x2 matrix with constant 1 in first column and values for TV in second). Instead of calling the `reg` object, we could also use `Xconst` from above. The output becomes clear when you check the function's help page (use `?sm.stats.diagnostic.het_white`).

The null hypothesis is that the error variance does **not** depend on x, thus is homoskedastic. Based on the large f-statistic value we can gently reject the null-hypothesis that the error variance is homoskedastic.

```python
# Test for heteroskedasticity with White test
wht = sm.stats.diagnostic.het_white(resid=reg.resid, exog=reg.model.exog)
print('f-statistic: {0:>19.4f} \n'
      'p-value for f-statistic: {1:>7.4f}'.format(wht[2], wht[3]))
```

    f-statistic:             35.1683
    p-value for f-statistic:  0.0000

If you wish to run tests with heteroskedastistic robust standard errors you can either access the `reg` object's robust standard errors (`reg.HC0_se`, `reg.HC1_se`, `reg.HC2_se`, `reg.HC3_se`) or, more conveniently, run directly define the covariance estimator (e.g. 'HC3' as below) when you generate the object in the first place. Below example shows how you can do this. See also [here](http://www.statsmodels.org/dev/generated/statsmodels.regression.linear_model.RegressionResults.html) or [here](https://stackoverflow.com/questions/30553838/getting-statsmodels-to-use-heteroskedasticity-corrected-standard-errors-in-coeff) for some further information.

```python
regRobust = sm.OLS(ad.sales, exog=sm.add_constant(ad.TV)).fit(cov_type='HC3')
print(regRobust.HC3_se, '\n')
print(reg.HC3_se)
```

    const    0.336426
    TV       0.002890
    dtype: float64

    const    0.336426
    TV       0.002890
    dtype: float64

### Other Relevant Checks

The `statsmodels` package offers many more functions to run regression diagnostics, e.g. checks for autocorrelation, non-linearity, normality of residuals etc. These functions are applicable to both simple as well as multiple linear regression models. There's a short [Jupyther notebook](http://www.statsmodels.org/dev/examples/notebooks/generated/regression_diagnostics.html) detailing some of the options.

## Application: Stock Beta

A stock beta measures the systematic risk of a security, the tendency of a security to respond to swings in the broad market. Typically, a large, well diversified index is taken as a proxy for the market portfolio (e.g. S&P500, Euro Stoxx 50, SPI, etc.). There are different ways to calculate a stock beta. We will show the regression approach, where a stock's beta is the slope of the following linear regression:

$$\begin{equation}
r - r_f = \alpha + \beta(r_M - r_f) + e
\end{equation}$$

Let us look into Geberit's stock beta. As a proxy for the market portfolio we use the Swiss market index (SMI). The risk free rate is set to $r_f=0$, which is a fairly reasonable approach in light of [the Swiss national bank's (SNB) interest rates](https://data.snb.ch/de/topics/ziredev#!/cube/zimoma) for the past five years. We will work with monthly returns for the past five years (60 months) - though other approaches (e.g. last 24 monthly returns, weekly returns for last 2 years, etc.) are reasonable choices too. The data we will load from a csv that was sourced through a financial data provider.

```python
df = pd.read_csv('Data/SMIDataMonthly.csv', sep=',',
                 parse_dates=['Date'], dayfirst=True,
                 index_col=['Date'])
df.head()
```

      ABBN
      ADEN
      BAER
      CFR
      CSGN
      GEBN
      GIVN
      LHN
      LONN
      NESN
      ...
      ROG
      SCMN
      SGSN
      SLHN
      SREN
      UBSG
      UHR
      ZURN
      SIK
      SMI

      Date

      2018-01-31
      25.95
      76.60
      63.9
      89.32
      17.96
      441.0
      2240.0
      57.00
      258.9
      80.42
      ...
      229.45
      508.4
      2503
      349.5
      91.80
      18.90
      426.5
      305.9
      8065
      9335.40

      2017-12-29
      26.12
      74.55
      59.6
      88.30
      17.40
      429.1
      2252.0
      54.95
      263.3
      83.80
      ...
      246.50
      518.5
      2541
      345.0
      91.25
      17.94
      397.4
      296.6
      7740
      9381.87

      2017-11-30
      25.19
      74.40
      57.8
      84.70
      16.66
      427.8
      2238.0
      53.80
      257.1
      84.25
      ...
      248.20
      519.0
      2432
      330.2
      92.20
      16.99
      358.7
      297.2
      7600
      9318.77

      2017-10-31
      26.08
      79.15
      59.0
      92.00
      15.73
      451.6
      2228.0
      56.35
      265.0
      83.90
      ...
      230.50
      504.0
      2464
      346.8
      93.85
      16.98
      391.0
      304.5
      7385
      9242.18

      2017-09-29
      23.94
      75.40
      57.3
      88.50
      15.33
      458.1
      2107.0
      56.60
      254.0
      81.10
      ...
      247.20
      496.2
      2323
      341.1
      87.70
      16.55
      402.8
      295.4
      7205
      9157.46

5 rows × 21 columns

The dataframe `df` contains monthly closing prices on all SMI-stocks (incl. SMI index) with a date index in descending order. Let us create a separate Pandas dataframe with the returns of the past 60 months (`dfRets`).

```python
# Calculate returns and assign to variable dfRets
dfRets = pd.DataFrame()
dfRets['GEBNrets'] = np.log(df['GEBN'] / df['GEBN'].shift(-1))
dfRets['SMIrets'] = np.log(df['SMI'] / df['SMI'].shift(-1))
print(dfRets.head())
```

                GEBNrets   SMIrets
    Date
    2018-01-31  0.027355 -0.004965
    2017-12-29  0.003034  0.006748
    2017-11-30 -0.054141  0.008253
    2017-10-31 -0.014291  0.009209
    2017-09-29  0.044640  0.025662

Having done that, we are already set to run the regression and print the results.

```python
# Set observation period (last 60 monthly returns)
months = 60

# Create OLS object, run regression and calculate fit
regBeta = sm.OLS(endog=dfRets.iloc[:months, 0],
                 exog=sm.add_constant(dfRets.iloc[:months, 1])).fit()

# Show table on coefficients
print(regBeta.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:               GEBNrets   R-squared:                       0.311
    Model:                            OLS   Adj. R-squared:                  0.300
    Method:                 Least Squares   F-statistic:                     26.23
    Date:                Sun, 11 Feb 2018   Prob (F-statistic):           3.62e-06
    Time:                        20:34:35   Log-Likelihood:                 107.89
    No. Observations:                  60   AIC:                            -211.8
    Df Residuals:                      58   BIC:                            -207.6
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          0.0086      0.005      1.617      0.111      -0.002       0.019
    SMIrets        0.9156      0.179      5.121      0.000       0.558       1.273
    ==============================================================================
    Omnibus:                        4.808   Durbin-Watson:                   2.225
    Prob(Omnibus):                  0.090   Jarque-Bera (JB):                4.204
    Skew:                           0.405   Prob(JB):                        0.122
    Kurtosis:                       4.013   Cond. No.                         34.0
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Based on the regression output we have no reason to reject the null of Geberit's stock beta being equal to zero. The $R^2$ measure, though, shows that only a small amount of the variation in Geberits monthly returns is explained by SMI's monthly returns.

```python
# Get relevant information
beta = regBeta.params['SMIrets']
alpha = regBeta.params["const"]
rsqr = regBeta.rsquared

# Plot scatter & lm; add text with alpha, beta , R2
plt.figure(figsize=(12, 8))
plt.scatter(dfRets.iloc[:months, 0],
            dfRets.iloc[:months, 1],
            marker='.', label='Monthly Returns')
plt.plot(dfRets.iloc[:months, 1], regBeta.fittedvalues, c='k', label='Fit')
plt.axis('equal')
plt.ylabel('Geberit Monthly Returns')
plt.xlabel('SMI Monthly Returns')
plt.legend(loc='lower right')
plt.text(-0.07, 0.06, 'Beta: {0: .2f}'.format(beta))
plt.text(-0.07, 0.05, 'Alpha: {0: .2f}'.format(alpha))
plt.text(-0.07, 0.04, 'R^2: {0: .2f}'.format(rsqr));
```

What we calculated above is often referred to as the raw beta. The beta value of a stock has been found to be on average closer to the mean value of 1.0, the beta of an average-systematic-risk portfolio, than to the value of the raw beta (Pinto et al. (2016)). This is why data providers such as Bloomberg publish the adjusted beta as first introduced by Blume (1971), which is calculated as

$$ \text{Adjusted beta} = 2/3 \cdot \text{raw beta} + 1/3 \cdot 1$$

Now, let us assume we are given the task to investigate whether a beta indeed regresses to 1 over time. For that we could, as a starting point, assess a stock's rolling beta over the past years. Note that this is just an example of use. Computationally it would be much faster to calculate the stock beta via the covariance/variance formula.

```python
def rollingBeta(df, window=60):

    '''Calculates the running beta of a stock.

    Parameters
    ==========
    df : [n x 2] pandas dataframe with log-returns for
        stock and market portfolio. Index should be
        datetime series.
    window : rolling window with default value 60 [optional]

    Returns
    =======
    rb : Pandas dataframe with (backward-looking) rolling beta.
    '''

    # Drop NA rows from df
    df = df.dropna()
    # Set up empty results array
    res = np.empty(len(df) - window + 1)

    # Loop through df
    for i in range(0, len(df)):

        # As long as remaining subset is >= window, we proceed
        if (len(df) - i) >= window:

            # Subset df
            sub = df.iloc[i:window+i, :]
            # Run Regression
            model = sm.OLS(endog=sub.iloc[:, 0],
                           exog=sm.add_constant(sub.iloc[:, 1])).fit()
            # Read out beta coefficient
            res[i] = model.params[1]

    # Format output to dataframe
    rb = pd.DataFrame(data=res, index=df.index[:(len(df)-window+1)])
    rb.columns = ['RollingBeta']

    return rb
```

Now we are ready to call the function. This time we use the last two years of monthly returns. Thus we set `window=24` to overwrite the default value of 60.

```python
# Call function and save output to 'rollBeta'
rollBeta = rollingBeta(df=dfRets, window=24)
```

```python
# Import 'mdates' library to format dates in x-axis
import matplotlib.dates as mdates

# Plot rolling beta
fig, ax = plt.subplots(1, figsize=(12, 8))
ax.plot(rollBeta, label='Geberit Rolling Beta')
ax.axhline(y=1, c='gray', linestyle=':')  # Horizontal line
ax.legend(fontsize=12)
ax.xaxis.set_major_locator(mdates.MonthLocator(interval=6))
ax.xaxis.set_major_formatter(mdates.DateFormatter('%m-%Y'))
fig.autofmt_xdate();  # Autorotate x-axis for readability
plt.savefig('C:/Users/Ben Zimmermann/Dropbox/Uni/MA/ML Seminar/LaTeX_Script/graphics/0205_RollingBetaPlot.pdf')
```

Though this is far away from a thorough analysis, plotting the results shows that at least in Geberit's case, there is indeed some truth to the assessment, that the beta exhibits some reversion to the market beta value of 1.

# Multiple Linear Regression

## Estimating the Regression Coefficients

Simple linear regression serves well to introduce the concept and to build a good understanding. However, in reality we often have to work with more than one predictor. In the advertising data set for example we had not only data on TV advertising spendings but also on radio newspaper. It thus makes sense to extend the simple to a multiple linear regression model.

We again use the Advertising data set to see how this is done in Python. The same functions from the `statsmodels` package apply to multiple linear regression. We run the following regression

$$\text{sales} = \beta_0 + \beta_1 \text{TV} + \beta_2 \text{radio} + \beta_3 \text{newspaper} + \epsilon$$

```python
# Assign features and response to X and y
y = ad.sales
X = ad[['TV', 'radio', 'newspaper']]
X = sm.add_constant(X)
```

```python
# Run regression and print summary
mlReg = sm.OLS(endog=y, exog=X).fit()
print(mlReg.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                  sales   R-squared:                       0.897
    Model:                            OLS   Adj. R-squared:                  0.896
    Method:                 Least Squares   F-statistic:                     570.3
    Date:                Sun, 11 Feb 2018   Prob (F-statistic):           1.58e-96
    Time:                        20:34:36   Log-Likelihood:                -386.18
    No. Observations:                 200   AIC:                             780.4
    Df Residuals:                     196   BIC:                             793.6
    Df Model:                           3
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          2.9389      0.312      9.422      0.000       2.324       3.554
    TV             0.0458      0.001     32.809      0.000       0.043       0.049
    radio          0.1885      0.009     21.893      0.000       0.172       0.206
    newspaper     -0.0010      0.006     -0.177      0.860      -0.013       0.011
    ==============================================================================
    Omnibus:                       60.414   Durbin-Watson:                   2.084
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              151.241
    Skew:                          -1.327   Prob(JB):                     1.44e-33
    Kurtosis:                       6.332   Cond. No.                         454.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

The coefficient for radio (0.1885) tells us, that - holding all other factors fixed - an additional 1'000 dollars in radio advertising spendings will boost the product's sales by 188.5 units.

Again: If you are an `R` guy and prefer their syntax, you could import the statsmodels.formula.api subpackage and run something like:

> `from statsmodels.formula.api import ols
mlReg = ols("Sales ~ TV + radio + newspaper", data=ad).fit()
mlReg.summary()`

## Hypothesis Tests

Again the summary above provides $t$-statistic and p-value for each individual regression coefficient. As was the case for the simple linear regression, the underlying null hypothesis is that each parameter is zero ($H_0: \beta_{j,\, H_0} = 0$). For TV and Radio we reject the null even at the 1% significance level. However, given the large p-value for Newspaper we fail to reject the null for $\beta_{\text{Newspaper}} = 0$ at any reasonable level. Thus we can conclude that leaving Newspaper data out might be a reasonable option. If other null hypothesis' ought to be tested, we can use the same command as shown above.

```python
# t-test on H0: beta(TV) = 0.0475
mlReg.t_test('TV=0.0475')
```

    <class 'statsmodels.stats.contrast.ContrastResults'>
                                 Test for Constraints
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    c0             0.0458      0.001     -1.244      0.215       0.043       0.049
    ==============================================================================

The output shows that we fail to reject the null hypothesis that $\beta_{TV} = 0.0475$.

Beyond the element-wise hypothesis tests the regression summary also provides the **F-statistic** (and the corresponding p-value) on the combined hypothesis that

$$\begin{align}
H_0&: \quad \beta_j = \beta_1, \beta_2, \ldots \beta_p = 0 \\
H_a&: \quad \beta_j \neq 0 \text{ for at least one $j$}
\end{align}$$

On the basis of the corresponding p-value (i.e. 1.58e-96) we can reject the null at any reasonable significance level. Should we be interested in assessing a particular hypothesis, say

$$\begin{align}
H_0&: \quad \beta_{TV} = \beta_{\text{Radio}} \\
H_a&: \quad \beta_{TV} \neq \beta_{\text{Radio}}
\end{align}$$
we use the `.f_test()` method.

```python
# Test H0: beta(radio) = beta(newspaper) = 0
mlReg.f_test('const = radio = newspaper = 0.1')
```

    <class 'statsmodels.stats.contrast.ContrastResults'>
    <F test: F=array([[ 126.9935335]]), p=1.0186277412639764e-45, df_denom=196, df_num=3>

See the [documentation page](http://www.statsmodels.org/dev/generated/statsmodels.regression.linear_model.RegressionResults.f_test.html) for further examples on how this function can be used.

### Coefficient of Determination

The $R^2$ measure for the MLR is the same as for the SLR. However, in the case of MLR it has one drawback: the value will always increase when more explanatory variables are added to the model - even if those variables are only weakly associated with the response. To make good on this disadvantage a modificated measure is often used: **adjusted $R^2$**.

$$\begin{equation}
R^2_{adj} = 1 - (1-R^2) \frac{n-1}{n-p-1}
\end{equation}$$

To get this measure in Python, simply use the OLS object and call the `.rsquareed_adj` attribute.

```python
mlReg.f_test('TV=radio=newspaper=0')
```

    <class 'statsmodels.stats.contrast.ContrastResults'>
    <F test: F=array([[ 570.27070366]]), p=1.575227256093874e-96, df_denom=196, df_num=3>

## Application: Factor Models

### Fama-French Three Factor Model

We will apply the concept of multiple linear regression in the context of Fama-French's three factor model (Fama and French (1993)). Their model follows Ross' arbitrage pricinge theory which postulates that excess returns are linearly related to a set of systematic risk factors (Ross et al. (1973)). The factors can be returns on other assets, such as the market portfolio, or any other variable (e.g. interest rates, inflation, consumption growth, market sentiment, hedging demands etc.. Fama-French empirically discovered three factors to capture the systematic risk: firm size, book-to-market ratio (B/M) and market risk. To quantify their findings, Fama-French constructed zero-net-investment factor portfolios capturing the systematic risk on firm size ('small minus big' (SMB), i.e. long on small and short on big size stocks) and B/M ('high minus low' (HML), i.e. long on high B/M, short on low B/M stocks). For details on how these portfolios are constructed see e.g. Bodie et al. (2014). The third factor, market risk, is simply the excess return on a well diversified market portfolio.

With that, the sensitivity of individual stocks to the three factors is given by the estimated coefficients of a multiple linear regression. As a group they predict the total risk premium. The expected excess return $R_{it}$ of asset $i$ at time $t$ in the Fama-French three-factor model is described by

$$\begin{equation}
R_{it} = \alpha_i + \beta_i^{M} (r_{M,t} - r_{f,t}) + \beta_i^{SMB} SMB_t + \beta_i^{HML} HML_t + \epsilon_{it}
\end{equation}$$

The above Fama-French factors are calculated on a monthly basis and published on [Kenneth R. French's website](http://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html). There you will also find information on the methodology of the model and lots of other possible factors. To run this regression in Python we use a shortcut. The `pandas_datareader` package is capable of loading the data without having to download a txt or csv file in a separate step.

We will calculate the factor beta for all 20 SMI stocks. For that we use Fama-French's 'European 3 Factors' data. Following our 'Stock Beta' example from above, the risk free rate for Switzerland is again assumed to be zero over the past 5 years.

### Prepare Data for Fama-French Model
We start by importing the `pandas_datareader.data` package and defining some key parameter.

```python
import pandas_datareader as web

# Define obs. period, start & enddate
months = 60
startdate = '2012-06-01'
enddate   = '2017-12-31'
```

** Side Note: ** If you want to know what data is available (and their labels), you can run the `get_available_datasets()` function.

> `from pandas_datareader.famafrench import get_available_datasets
get_available_datasets()`

or simply check [Kenneth R. French's website](http://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html)

```python
# Load FF factors
data = web.DataReader('Europe_3_Factors', data_source='famafrench',
                      start=startdate, end=enddate)
```

Variable `data` is a dictionary with three entries: monhtly data, annual data and a description. We select the monthly rates in dictionary 0 and format the data.

```python
# Select monthly data
ff = data[0]

# Sort data in descending order
ff = ff.sort_index(ascending=False)

# Convert returns to decimal percentages
ff = ff/100
print(ff.head(3))
```

             Mkt-RF     SMB     HML      RF
    Date
    2017-12  0.0147  0.0156  0.0046  0.0009
    2017-11 -0.0002 -0.0040  0.0108  0.0008
    2017-10  0.0060 -0.0090 -0.0020  0.0009

Next we calculate the log-returns of all SMI stocks. The share prices are taken from dataframe `df` which we loaded above.

```python
shsRets = np.log(df / df.shift(-1))
shsRets = shsRets['2017-12-31':'2012-06-01']
shsRets = shsRets.iloc[:, :-1]  # We exclude last column (with SMI data)
shsRets.head(3)
```

      ABBN
      ADEN
      BAER
      CFR
      CSGN
      GEBN
      GIVN
      LHN
      LONN
      NESN
      NOVN
      ROG
      SCMN
      SGSN
      SLHN
      SREN
      UBSG
      UHR
      ZURN
      SIK

      Date

      2017-12-29
      0.036254
      0.002014
      0.030667
      0.041625
      0.043460
      0.003034
      0.006236
      0.021150
      0.023829
      -0.005356
      -0.021609
      -0.006873
      -0.000964
      0.043844
      0.043846
      -0.010357
      0.054408
      0.102457
      -0.002021
      0.018253

      2017-11-30
      -0.034722
      -0.061889
      -0.020549
      -0.082673
      0.057441
      -0.054141
      0.004478
      -0.046309
      -0.030265
      0.004163
      0.024040
      0.073984
      0.029328
      -0.013072
      -0.049050
      -0.017738
      0.000589
      -0.086221
      -0.024266
      0.028697

      2017-10-31
      0.085618
      0.048538
      0.029237
      0.038786
      0.025758
      -0.014291
      0.055839
      -0.004427
      0.042396
      0.033943
      -0.008480
      -0.069947
      0.015597
      0.058927
      0.016573
      0.067776
      0.025650
      -0.029733
      0.030341
      0.024676

### Calculate Fama-French Coefficients

In order to run the regression we need to have matching indices. Fama-French's index is 'yyyy-mm' while our dataframe with returns has format 'yyyy-mm-dd'. Since we know that the length of both dataframes is equal, we can simply overwrite the index of one of the dataframes.

```python
# Create matching indices
ff.index = shsRets.index
```

We are now in a position to run the multiple linear regression. We will again use the past 60 months. From the Fama French set we just need the first three columns. Column 4 is the risk free rate which we do not use.

```python
# Add constant to matrix for alphas (=intercept)
X = sm.add_constant(ff.iloc[:months, :3])
```

```python
# Assign ticker to variable
tickers = shsRets.columns
```

```python
# Create results matrix to paste beta factors
res = np.empty(shape=(5, len(tickers)))
```

```python
# Run regression for each ticker
for i in range(0, len(tickers)):

    # Select returns of share i
    sub = shsRets.iloc[:months, i]
    # Run regression
    model = sm.OLS(endog=sub, exog=X).fit()
    # Paste beta factors to 'res' matrix
    res[0:4, i] = model.params
    res[4, i] = model.rsquared_adj
```

```python
# Format output to dataframe
ff3f = pd.DataFrame(data=res, index=['Alpha', 'BetaMkt', 'BetaSMB', 'BetaHML', 'R2_adj'])
ff3f.columns = tickers
```

And here are our factors:

```python
ff3f
```

      ABBN
      ADEN
      BAER
      CFR
      CSGN
      GEBN
      GIVN
      LHN
      LONN
      NESN
      NOVN
      ROG
      SCMN
      SGSN
      SLHN
      SREN
      UBSG
      UHR
      ZURN
      SIK

      Alpha
      -0.001025
      -0.001959
      0.005041
      -0.001922
      -0.011472
      0.011149
      0.013058
      -0.011969
      0.022697
      0.003409
      0.003714
      0.002670
      0.005087
      0.000832
      0.010098
      0.005727
      -0.002099
      -0.009045
      0.002611
      0.018352

      BetaMkt
      0.702400
      0.848991
      0.645388
      0.915350
      0.960051
      0.457179
      0.500261
      1.177939
      0.804210
      0.505555
      0.504823
      0.449106
      0.340536
      0.522752
      0.485764
      0.255057
      0.750614
      0.929691
      0.415227
      0.653636

      BetaSMB
      0.293945
      0.703264
      0.048114
      -0.401858
      0.369227
      -0.524710
      -0.699341
      0.114422
      0.077635
      -0.411104
      -0.403297
      -0.317893
      -0.795906
      -0.273266
      0.911996
      -0.577227
      0.034922
      -0.149020
      -0.640211
      -0.437440

      BetaHML
      0.122113
      0.444320
      0.628217
      0.273710
      1.694172
      -0.138601
      -0.551688
      0.353747
      -0.210786
      -0.582850
      -0.378915
      -0.306853
      -0.204705
      -0.438750
      0.569497
      -0.287004
      1.019220
      0.288909
      0.040235
      0.023124

      R2_adj
      0.188357
      0.339020
      0.173241
      0.293771
      0.354041
      0.103003
      0.228293
      0.434920
      0.214598
      0.310364
      0.165969
      0.108944
      0.160359
      0.102738
      0.159904
      0.043001
      0.289225
      0.216220
      0.099337
      0.171639

```python
# Transpose matrix (.T) and display stats summary
print(ff3f.T.describe())
```

               Alpha    BetaMkt    BetaSMB    BetaHML     R2_adj
    count  20.000000  20.000000  20.000000  20.000000  20.000000
    mean    0.003248   0.641227  -0.153888   0.117856   0.207847
    std     0.008978   0.239535   0.465744   0.567100   0.101348
    min    -0.011969   0.255057  -0.795906  -0.582850   0.043001
    25%    -0.001931   0.478618  -0.459258  -0.291966   0.147164
    50%     0.003040   0.584070  -0.295580   0.031680   0.180799
    75%     0.006820   0.815405   0.086832   0.376390   0.290361
    max     0.022697   1.177939   0.911996   1.694172   0.434920

### Hedge a Portolio

Now that we have all the factors, let's assume we want to build a portfolio with all 20 SMI stocks that maximizes the Sharpe ratio (SR). As a further condition we want to limit our exposure to the SMB factor to, let's say, $\beta_p^{SMB} = 0$. How would we allocate our investment under these conditions? In mathematical terms we have the following optimization problem:

$$\begin{equation}
\max_{w_i} SR = \frac{\mathbb{E}[r_p] - r_f}{\sigma_p} \qquad s.t. \qquad
\begin{cases}
\sum w_i &= 1 \\
\beta_p^{SMB} &= 0
\end{cases}
\end{equation}$$

Usually, to calculate the expected return $\mathbb{E}[r_p]$, historical returns are taken. For our case here, we will take the expected returns given by our Fama-French 3 Factor model (denoted $\mathbf{R_{ff}}$). The portfolio variance $\sigma_p$ however, we estimate using historical data. Alternatively one could think of taking the SMI volatility index value as proxy. But this is only approximately true because we will not have the same weights per stock as the SMI and this thus might be a questionable. With that we have

$$\begin{equation}
\max_{w_i} SR = \frac{\mathbf{w}^T \left(r_f + \mathbf{\alpha} + \mathbf{\beta}^{M} (r_{M} - r_{f}) + \mathbf{\beta}^{SMB} SMB + \mathbf{\beta}^{HML} HML \right) - r_f}{\mathbf{w}^T \mathbf{\Sigma}\mathbf{w}} \qquad s.t. \qquad
\begin{cases}
\sum w_i &= 1 \\
\beta_p^{SMB} &= 0
\end{cases}
\end{equation}$$

Python can solve this problem numerically. We first set the stage by defining a auxiliary function `pfStats` that returns the expected portfolio return, volatility and Sharpe ratio given a vector of weights. Note that the function also makes use of other data like monthly returns and the riskfree rate as previously defined (which is again set to 0) but only weights are a function input value. This is necessary for the optimization function.

```python
# Define rf and (equally spread) start weights
rf = 0
wghts = np.repeat(1. / len(tickers), len(tickers))

# Expected stock returns based on ff3f model
expShsRets = rf + ff3f.T.Alpha + \
    ff3f.T.BetaMkt * ff['Mkt-RF'].mean() + \
    ff3f.T.BetaSMB * ff.SMB.mean() + \
    ff3f.T.BetaHML * ff.HML.mean()
```

```python
def pfStats(weights):
    '''Returns basic measures for a portfolio

    Parameters
    ==========
    weights : array-like
        weights for different securities in portfolio

    Returns
    =======
    expPfRet : float
        weighted, annualized expected portfolio return based on ff3f model
    pfVol : float
        historical annualized portfolio volatility
    SR : float
        portfolio Sharpe ratio for given riskfree rate
    '''

    expPfRet = np.sum(weights * expShsRets) * 12
    pfVol    = np.sqrt(np.dot(weights.T, np.dot(shsRets.cov() * 12, weights)))
    SR       = (expPfRet - rf) / pfVol

    return np.array([expPfRet, pfVol, SR])
```

Python's scipy package has a sublibrary for constrained optimization problems. We will use the minimize function and minimize the negative value of the Sharpe ratio (which is obviously equal to maximizing the SR)

```python
import scipy.optimize as sco

def minSR(wghts):
    return -pfStats(wghts)[2]
```

Our constraints are as stated above: $\sum w_i = 1$, $\beta_P^{SMB} = 0$. Additionally we set bounds for the weights such that short/long position are allowed but only up to 100% per share ($w_i \in [-1, 1]\; \forall i \in [1, 2, \ldots, n]$).

```python
# Constraints and bounds
constr = [{'type': 'eq', 'fun': lambda x: np.sum(x) - 1},
          {'type': 'eq', 'fun': lambda x: np.sum(x * ff3f.T.BetaSMB) - 0}]
bnds   = tuple((-1,1) for x in range(len(tickers)))
```

```python
# Minimization function
optPf = sco.minimize(minSR, x0=wghts, method='SLSQP', bounds=bnds, constraints=constr)
```

```python
# Check if conditions are actually met
print('Sum of weights:  ', np.sum(optPf['x']))
print('Beta SMB factor: ', np.sum(optPf['x'] * ff3f.T.BetaSMB))
```

    Sum of weights:   1.0
    Beta SMB factor:  6.940115149234316e-12

```python
# Calculate portfolio stats given optimal weights
rsltsOptPf = pfStats(optPf['x'])

# Format weights into dataframe with Tickers as heading
optWghts = pd.DataFrame(data=optPf['x'], index=tickers)
optWghts.columns = ['optimalWghts']

# Print results
print('Portfolio return:     ', str(rsltsOptPf[0]))
print('Portfolio volatility: ', str(rsltsOptPf[1]))
print('Portfolio SR:         ', str(rsltsOptPf[2]), '\n')
print(str(optWghts))
```

    Portfolio return:      0.66978301121
    Portfolio volatility:  0.202898675338
    Portfolio SR:          3.30107138499

          optimalWghts
    ABBN     -0.051382
    ADEN     -0.255488
    BAER      0.099283
    CFR       0.263565
    CSGN     -0.191812
    GEBN      0.075917
    GIVN      0.609015
    LHN      -0.471116
    LONN      0.533256
    NESN      0.232630
    NOVN      0.395617
    ROG      -0.690954
    SCMN     -0.121551
    SGSN     -0.269585
    SLHN      0.755819
    SREN     -0.055902
    UBSG      0.076473
    UHR      -0.282089
    ZURN     -0.474419
    SIK       0.822725

# Further Ressources

In writing this notebook, many ressources were consulted. For internet ressources the links are provided within the textflow above and will therefore not be listed again. Beyond these links, the following ressources were consulted and are recommended as further reading on the discussed topics:

* Blume, Marshall E., 1971, On the Assessment of Risk, *The Journal of Finance* 26, 1-10.
* Fama, Eugene F, and Kenneth R French, 1993, Common risk factors in the returns on stocks and bonds, *Journal of Financial Economics* 33, 3–56.
* Hilpisch, Yves, 2015, Python for Finance (O'Reilly Media, Sebastopol, CA).
* James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani, 2013, *An Introduction to Statistical Learning: With Applications in R* (Springer Science & Business Media, New York, NY).
* Müller, Andreas C., and Sarah Guido, 2017, *Introduction to Machine Learning with Python* (O’Reilly Media, Sebastopol, CA).
* Pinto, Jerald E., Elaine Henry, Thomas R. Robinson, and John D. Stowe, 2016, *Equity Asset Valuation* (MIT Press, Cambridge, MA).
* Ross, Stephen A., et al., 1973, Return, risk and arbitrage (Rodney L. White Center for Financial Research, The Wharton School, University of Pennyslvania).
*   Sheppard, Kevin, 2017, Introduction to Python for Econometrics, Statistics and Data Analysis from Website https://www.kevinsheppard.com/images/b/b3/Python_introduction-2016.pdf, 07/07/2017.
* Wooldridge, Jeffrey M, 2015, *Introductory Econometrics: A modern approach* (Cengage Learning, Boston, MA).


# Complementary Filter

Convert continuous transfer function into discrete

```
import scipy.signal as sig
from math import sqrt

dt = 0.01
ki = 0.1**2
kp = sqrt(2)*0.1

num = [kp, ki]
den = [1, 0.]
sig.cont2discrete(sys=(num,den), dt=0.1)
```

    (array([[ 0.14142136, -0.14042136]]), array([ 1., -1.]), 0.1)

```
a = 5
num = [a, 0]
den = [a, 1]
sig.cont2discrete(sys=(num,den), dt=0.1)
```

    (array([[ 1., -1.]]), array([ 1.        , -0.98019867]), 0.1)

```
from sympy import *
from sympy.abc import *
a = Symbol('a', positive=True)
foo = (s/(s+a)).rewrite()
print(foo)
inverse_laplace_transform(1/((s+a)), s, t)
```

    s/(a + s)

    exp(-a*t)*Heaviside(t)

```

```

```

```

```

```

```

```

```

```

```

```

```

```

```

```


# Lecture 28: Sum of a random number of random variables; Inequalities (Cauchy-Schwarz, Jensen, Markov, Chebyshev)

## Stat 110, Prof. Joe Blitzstein, Harvard University

----

## Sum of Random Number of Random Variables

Let's say there's a store. At this store, there is a random number of customers. Each customer chooses to spend a random amount of money at the store. What can we say about the expected total amount spent at the store? What about its variance?

* $N$ is the number of customers (maybe Poisson? it doesn't matter at this point)
* $X_j$ be the amount customer $j$ spends
* $X_j$ has mean $\mu$ and variance $\sigma^2$
* assume that $N, \, X_1, \, N_2, \dots$ are independent

Find the mean and variance of $X = \sum_{j=1}^{N} X_j$.

Be careful, because $N$ is itself a random variable, and so we are summing up a _random number of random variables_. At this point, $N$ could be any distribution, perhaps Poisson, but we can put that question aside for now. Let's see if we can find some statistics about $X$ in terms of random variable $N$.

### $\mathbb{E}(X)$

Now, if we blindly tried to apply linearity, we might have:

\begin{align}
  \mathbb{E}(X) &=? \, N \, \mu &\quad \text{... no, this cannot be right}
\end{align}

But this is a category error, since the right-hand side of that equation is a _value_, while the left-hand side is a _random variable_. However, it does give us a clue: all we need to do is find a way to treat $N$ with a _value_, which we can do by thinking conditionally.

There are two ways we can use to find $\mathbb{E}(X)$:

\begin{align}
  &\text{...conditioning on } N \text{, the long way} \\
  \mathbb{E}(X) &= \sum_{n=0}^{\infty} \mathbb{E}(X|N=n) \, P(N=n) &\quad \text{conditioning on }N \\
  &= \sum_{n=0}^{\infty} \mu \, n \, P(N=n) &\quad \text{by linearity, independence} \\
  &= \mu \, \sum_{n=0}^{\infty} n \, P(N=n) &\quad \text{but by definition of }\mathbb{E} \\
  &= \boxed{\mu \, \mathbb{E}(N)} \\\\
  \\
  & \text{... or using Adam's Law} \\
  \mathbb{E}(X) &= \mathbb{E}\left( \mathbb{E}(X|N) \right) &\quad \text{Adam's Law} \\
  &= \mathbb{E}(\mu \, N) &\quad \text{treating } N \text{ as constant, and by linearity} \\
  &= \boxed{\mu \, \mathbb{E}(N)}
\end{align}

Note that unlike the 2-envelope paradox covered in lectures 25 and 26, we _can_ forget about the condition $N=n$ _because of the independence of $X_j$_. That fact allows us to use linearity in the second step of finding $\mathbb{E}(X)$ the long way by conditioning on $N$.

### $\operatorname{Var}(X)$

Here, we will just apply EVvE's Law:

\begin{align}
  \operatorname{Var}(X) &= \mathbb{E}\left( \operatorname{Var}(X|N) \right) + \operatorname{Var}\left( \mathbb{E}(X|N) \right) \quad \text{and working on each part} \\\\
  \mathbb{E}\left( \operatorname{Var}(X|N) \right) &= \mathbb{E}(N \sigma^2) \quad \text{treating } N \text{ as fixed, independence} \\
  &= \sigma^2 \, \mathbb{E}(N) \\
  \\
  \operatorname{Var}\left( \mathbb{E}(X|N) \right) &= \operatorname{Var}(\mu \, N) \quad \text{} \\
  &= \mu^2 \, \operatorname{Var}(N) \\\\
  \Rightarrow \operatorname{Var}(X) &= \boxed{ \sigma^2 \, \mathbb{E}(N) + \mu^2 \, \operatorname{Var}(N) }
\end{align}

Idiot-checking that result, we confirm that

* $\mathbb{E}(X) = \mu \, \mathbb{E}(N)$ means that the average total sales is the average per-customer spending amount times the average number of customers
* $\sigma^2$ is in terms of money-squared; $\mu$ is in terms of money, so $\operatorname{Var}(X)$ is in terms of money-squared
* both $\mathbb{E}(N)$ and $\operatorname{Var}(N)$ are dimensionless values ("people" are not units in this case)
* if we knew the MGF of each $X_j$, then we could condition on $N$ to treat it as fixed in order to get the MGF of $X$ (multiply by $n$)

## Statistical Inequality

In the field of law, it is a lot easier to stand in court to use statistical inequalities rather than approximations.

Basically, approximations could be painted to be subjective. If you use a Poisson approximation and had to explain yourself in court, you might be hard-pressed to justify why you think your approximation is good. "Good" is subject to interpretation; we would claim that an approximation is good if it is close to the truth. But a lawyer might then grill on that, asking you "Is there some accepted standard to closeness? Do you know how close your approximation is?"

Now if you knew that, then you would already *know* the actual distribution and you wouldn't be using an approximation in the first place!

...

That sort of argument probably would not fly very well in front of the judge.

But inequalities are pretty much unassailable, as they provide a _definite fact about something that is random_.

### (1) Cauchy-Schwarz Inequality: $\lvert \mathbb{E}(XY) \rvert \le \sqrt{\mathbb{E}(X^2)(Y^2)}$

Recall that $\mathbb{E}(XY)$ is an analog of the dot-product for random variables $X,Y$.

Recall the definition of a dot-product in linear algebra.

\begin{align}
  \textbf{a} \cdot \textbf{b} &= \lVert \mathbf{a} \rVert \lVert \mathbf{b} \rVert \mathbf{cos}\theta \\
  \\
  \mathbb{E}(XY) &\le \sqrt{\mathbb{E}(X^2)(Y^2)}
\end{align}

Note that $X,Y$ are uncorrelated, then by definition $\mathbb{E}(XY) = \mathbb{E}(X)\mathbb{E}(Y)$, and so using the Cauchy-Schwarz Inequality is really unnecessary.

The Cauchy-Schwarz Inequality really comes in handy when the random variables $X,Y$ are correlated. In such a case where $X,Y$ are correlated and you are interested in $\mathbb{E}(XY)$, then without using a 2D LOTUS or calculating the distribution of $XY$, we can still find an upper bound to $\mathbb{E}(XY)$.

#### Statistical Interpretation

The interpretation is easiest to see if we let $X,Y$ have mean 0.

Then:
\begin{align}
  \lvert \operatorname{Corr}(X,Y) \rvert &= \lvert \frac{\operatorname{Cov}(X,Y)}{\sqrt{\operatorname{Var}(X)} \sqrt{\operatorname{Var}(Y)}} \rvert \\
  &= \lvert \frac{\mathbb{E}(XY) - \mathbb{E}(X)\mathbb{E}(Y)}{\sqrt{\mathbb{E}(X^2)-(\mathbb{E}(X))^2}\sqrt{\mathbb{E}(Y^2)-(\mathbb{E}(Y))^2}} \rvert \\
  &= \lvert \frac{\mathbb{E}(XY)}{\left(\mathbb{E}(X^2) \mathbb{E}(Y^2)\right)^{\frac{1}{2}}} \rvert \le 1
\end{align}

... and the left-hand side of the above equation is *exactly* the Cauchy-Schwarz Inequality. Note that this would hold for any value of mean for $X,Y$.

So for statistics, the Cauchy-Schwarz Inequality really just means that the correlation of two random variables will be between -1 and 1. For providing upper bounds to $\mathbb{E}(XY)$, the Cauchy-Schwarz Inequality's strengths are simplicity and generality.

### (2) Jensen's Inequality: if $g$ is a convex function, then $\mathbb{E} g(X) \ge g(\mathbb{E}X)$

\begin{align}
  \mathbb{E}g(x) &\le g(\mathbb{E}X) &\quad \text{if }g(x) \text{ is concave} \\\\
  \mathbb{E}h(x) &\ge h(\mathbb{E}X) &\quad \text{if }h(x) \text{ is convex}
\end{align}

This tells you when $g$ is a convex function which way the inequality goes.

Now, a convex function is such that $g^{\prime\prime}(x) \ge 0$ (if $g^{\prime\prime}(x)$ exists). Another way of describing it is that if you pick any two points on $g(x)$, the line segment they make will lay above $g(x)$.

```python
import matplotlib
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline

plt.xkcd()

x = np.linspace(-10, 10, 1000)
y = x**2
x2 = [-4,7]
y2 = [16,49]

_, ax = plt.subplots(figsize=(12,8))
ax.axhline(y=0, color='k')
ax.axvline(x=0, color='k')

ax.set_xticks([])
ax.set_yticks([])

plt.text(2.0, 80.0, '$g(x)=x^2$,\n$g^{\prime\prime}(x)=2$', color='xkcd:azure')
plt.text(-5.5, 10.0, 'A')
plt.text(7.5, 42.5, 'B')

ax.plot(x, y, 'xkcd:azure', lw=4, alpha=0.6)
ax.plot(x2,y2,'xkcd:orange', linestyle='--', lw=4, alpha=0.6)

plt.suptitle(r'Convexity: $\overline{AB}$ lies above $g(x)$ for all points $A,B$')

plt.show()
```

```python
x = np.linspace(0.001, 5, 1000)
y = 1/x
x2 = np.linspace(0.001,5,1000)
y2 = np.log(x2)

_, ax = plt.subplots(figsize=(12,8))
ax.axhline(y=0, color='k')
ax.axvline(x=0, color='k')

ax.set_xticks([])
ax.set_yticks([])
ax.set_xlim([-2,5])
ax.set_ylim([-2,5])
ax.text(0.85, 2.2, (r'$\mathbb{E}(\frac{1}{X}) \geq \mathbb{E}(X)$' '\n$X$ is positive, $g(X)$ is convex'), color='xkcd:azure')
ax.text(0.85, -1.2, '$\mathbb{E}(\ln X) \leq \ln \mathbb{E}(X)$\n$h(X)$ is concave, inequality flips', color='xkcd:orange')
ax.plot(x, y, 'xkcd:azure', lw=4, alpha=0.6)
ax.plot(x2, y2, 'xkcd:orange', lw=4, alpha=0.6)

plt.suptitle('Examples: Jensen\'s Inequality')

plt.show()
```

#### Proof

We will try a graphical approach to prove Jensen's Inequality.

```python
x = np.linspace(-10, 10, 1000)
y = x**2

f = 1 + 2*(x - 1)

_, ax = plt.subplots(figsize=(12,8))
ax.axhline(y=0, color='k')
ax.axvline(x=0, color='k')

ax.text(1.9, 2.2, ('$y = a + bx$'), color='xkcd:orange')
ax.text(1.22, 0.95, ('($x, g(x))$'))

ax.set_xticks([])
ax.set_yticks([])
ax.set_xticks([])
ax.set_yticks([])
ax.set_xlim([-2,4])
ax.set_ylim([-2,4])
ax.plot(x, y, 'xkcd:azure', lw=4, alpha=0.6)
ax.plot(x, f, 'xkcd:orange', linestyle='--', lw=4, alpha=0.6)

plt.suptitle(r'Line $y$ tangent to $g(x)$')

plt.show()
```

Referring to the figure above...

\begin{align}
  g(x) &\geq a + bx &\quad g(x) \text{ lies above the tangent } y = a + bx \\
  \Rightarrow g(X) &\geq a + bX &\quad \text{inequality applies to r.v. as well} \\
  \mathbb{E}\left( g(X) \right) &\geq \mathbb{E}\left( a + bX \right) &\quad \mathbb{E} \text{ as operator} \\\\
  \mathbb{E}\left( a + bX \right) &= a + b \, \mathbb{E}(X) \\
  &= a + b \, \mu &\quad \text{let } \mu = \mathbb{E}(X) \\
  &= g(\mu) \\
  &= g\left(\mathbb{E}(X)\right) \\\\
  \therefore \mathbb{E}\left( g(X) \right) &\geq g\left(\mathbb{E}(X)\right) &\quad \blacksquare
\end{align}

It is also possible to use Taylor expansions to prove Jensen's Inequality, but that is something for another time.

### (3) Markov's Inequality: $P(|X| \geq a) \leq \frac{\mathbb{E}|X|}{a}$ for any $a \gt 0$

The strength of Markov's Inequality is not in its accuracy of _approximation_, but rather in it simplicity and generality.

#### Proof

We will use the fundamental bridge to convert the probability of an event to the expected value of the indicator of said event.

\begin{align}
  I_{|X| \geq a} &\geq |X| &\quad \text{always true for } I_{|X| \geq a} = 0 \text{ or } I_{|X| \geq a} = 1 \\
  a \, \mathbb{E} \, I_{|X| \geq a} \geq \mathbb{E} \, |X| \\
  \mathbb{E} \, I_{|X| \geq a} \geq \frac{\mathbb{E} \, |X|}{a}
\end{align}

#### Intuition

Suppose that we have 100 people.

Q1: Is it possible that at least 95% of the people are younger than the mean age of the group?

A1: Yes, if one of those people is really, really old, then that person's age can raise the mean age above a large portion of group.

Q2: Is it possible that at least 50% of the people are older than *twice* the mean age?

A2: No, since that 50% will raise the mean age beyond that!

### (4) Chebyshev's Inequality: $P(|X-\mu| \leq \frac{Var(X)}{a^2}$ for $\mu = \mathbb{E}X, a \gt 0$

An alternate way of putting Chebyshev's Inequality is

\begin{align}
  P\left(|X-\mu| > c \, SD(X) \right) \leq \frac{1}{c^2} &\quad \text{for } c > 0 \\
\end{align}

Consider this: the probability that $X$ is more than 2 standard deviations from its mean is at most $\frac{1}{4}$. Recall the Normal distribution and the 68,95,98.7 percent rule? Compared to that rule, Chebyshev's Inequality provides a cruder but still correct relation.

#### Proof

\begin{align}
  P(|X-\mu| \ge a) &= P\left( (X-\mu)^2 \ge a^2 \right) \\
  &\leq \frac{\mathbb{E}(X-\mu)^2}{a^2} &\quad \text{by Markov's Inequality} \\
  &\leq \frac{Var(X)}{a^2} &\quad \blacksquare
\end{align}

----

View [Lecture 28: Inequalities | Statistics 110](http://bit.ly/2CzCOoM) on YouTube.


```
from config import setup, import_and_display_fnc

setup()
%matplotlib widget
```

```
import numpy as np
import matplotlib.pyplot as plt
from common import (
    gauss_rule,
    qbx_matrix,
    symbolic_eval,
    qbx_setup,
    double_layer_matrix,
    PanelSurface,
    panelize_symbolic_surface,
    build_panel_interp_matrix,
)
import sympy as sp
```

```
%matplotlib inline
```

```
import_and_display_fnc('common', 'refine_panels')
import_and_display_fnc('common', 'stage1_refine')
```

```
qx, qw = gauss_rule(16)
t = sp.var("t")

sym_obs_surf = (t, -t * 1000, 0 * t)
sym_src_surf = (t, t * 0, (t + 1) * -0.5)
src_panels = np.array([[-1, 1]])
src_surf = panelize_symbolic_surface(
    *sym_src_surf, src_panels, qx, qw
)

control_points = np.array([(0, 0, 2, 0.5)])
obs_surf = stage1_refine(
    sym_obs_surf, (qx, qw), other_surfaces=[src_surf], control_points=control_points
)
```

```
%matplotlib widget
plt.figure()
plt.plot(obs_surf.pts[obs_surf.panel_start_idxs,0], obs_surf.pts[obs_surf.panel_start_idxs,1], 'k-*')
plt.xlim([-25,25])
plt.show()
```

```
from common import qbx_panel_setup, build_interp_matrix, build_interpolator

expansions = qbx_panel_setup(obs_surf, direction=1, p=10)
```

```
import_and_display_fnc('common', 'build_panel_interp_matrix')
import_and_display_fnc('common', 'stage2_refine')
```

```
%matplotlib inline
```

```
stage2_surf = stage2_refine(src_surf, expansions)
```

```
%matplotlib widget
plt.figure()
plt.plot(stage2_surf.pts[stage2_surf.panel_start_idxs,0], stage2_surf.pts[stage2_surf.panel_start_idxs,1], 'k-*')
plt.plot(expansions.pts[:,0], expansions.pts[:,1], 'r*')
plt.axis('equal')
plt.xlim([-1,1])
plt.ylim([-1,0])
plt.show()
```

```
t = sp.var("t")
theta = sp.pi + sp.pi * t
F = 0.98
u = F * sp.cos(theta)
v = F * sp.sin(theta)
x = 0.5 * (
    sp.sqrt(2 + 2 * u * sp.sqrt(2) + u ** 2 - v ** 2)
    - sp.sqrt(2 - 2 * u * sp.sqrt(2) + u ** 2 - v ** 2)
)
y = 0.5 * (
    sp.sqrt(2 + 2 * v * sp.sqrt(2) - u ** 2 + v ** 2)
    - sp.sqrt(2 - 2 * v * sp.sqrt(2) - u ** 2 + v ** 2)
)
x = (1.0 / F) * x * 100000
y = (1.0 / F) * y * 20000 - 20000
```

```
rounded_corner_box = stage1_refine((t, x, y), (qx, qw), control_points = [(0,0,10000,5000)], max_radius_ratio=10.0)
```

```
%matplotlib inline
plt.figure()
plt.plot(
    rounded_corner_box.pts[rounded_corner_box.panel_start_idxs, 0],
    rounded_corner_box.pts[rounded_corner_box.panel_start_idxs, 1],
    "k-*",
)
plt.axis("equal")
plt.show()
```

```
box_expansions = qbx_panel_setup(rounded_corner_box, direction=1, p=10)
```

```
stage2_box = stage2_refine(rounded_corner_box, box_expansions)
print(stage2_box.n_panels)
plt.figure()
plt.plot(
    stage2_box.pts[stage2_box.panel_start_idxs, 0],
    stage2_box.pts[stage2_box.panel_start_idxs, 1],
    "k-*",
)
plt.plot(box_expansions.pts[:,0], box_expansions.pts[:,1], 'r*')
plt.axis("equal")
plt.show()
```

```

```


# Music Machine Learning - Gaussian Mixture Models

### Author: Philippe Esling (esling@ircam.fr)

In this course we will cover
1. Unsupervised learning and [clustering](#clustering).
2. Motivating the need for [latent variable](#latent) models.
2. Approaching the problem naïvely with the [k-Means](#kmeans) algorithm.

We begin with the standard imports

```python
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns; sns.set()
import numpy as np
from helper_plot import hdr_plot_style
hdr_plot_style()
```

## Defining mixture models

We have seen previously how to fit a Gaussian, so we could apply the same reasoning and try to fit a single multivariate Gaussian to our dataset
$$
\begin{align}
p(\mathbf{x} \vert \theta) &= \mathcal{N}\left(\mathbf{x}\vert \mu, \Sigma\right) \\
\theta &= \{\mu, \Sigma\}
\end{align}
$$
However, we can quickly see that our data contains several groups, and the single Gaussian seems largely not adapted.

```python
from helper_plot import fit_multivariate_gaussian
from sklearn.datasets.samples_generator import make_blobs
X, y_true = make_blobs(n_samples=400, centers=4, cluster_std=0.60, random_state=0)
X = X[:, ::-1];
fit_multivariate_gaussian(X);
```

Instead of using a single Gaussian, we could use several Gaussians, for instance, if we use three Gaussians, we have the following model
$$
\begin{align}
p(\mathbf{x} \vert \theta) &= \pi_{1} \mathcal{N}\left(\mathbf{x}\vert \mu_{1}, \Sigma_{1}\right) \\
&+ \pi_{2} \mathcal{N}\left(\mathbf{x}\vert \mu_{2}, \Sigma_{2}\right)\\
&+ \pi_{3} \mathcal{N}\left(\mathbf{x}\vert \mu_{3}, \Sigma_{3}\right)\\
&+ \pi_{4} \mathcal{N}\left(\mathbf{x}\vert \mu_{4}, \Sigma_{4}\right)\\
\end{align}
$$
So now the parameters are both the parameters of each Gaussian, along with the probabilities of belonging to each Gaussian
$$
\theta = \{\pi_{1}, \pi_{2}, \pi_{3}, \pi_{4}, \mu_{1}, \mu_{2}, \mu_{3}, \mu_{4}, \Sigma_{1}, \Sigma_{2}, \Sigma_{3}, \Sigma_{4}\}
$$
Note that if we solve this model, called a _Gaussian Mixture Model_ (GMM), we can find the probabilities of each point belonging to one of the corresponding cluster

```python
from helper_plot import fit_gaussian_mixture
fit_gaussian_mixture(X);
```

Compared to a single Gaussian, a GMM has obviously a higher flexibility, but also a larger number of parameters (which makes it harder to optimize). Generally speaking, the full distribution of a GMM is defined as follows
$$
p(\mathbf{x} \vert \theta) = \sum_{c} \pi_{c} \mathcal{N}\left(\mathbf{x}\vert \mu_{c}, \Sigma_{c}\right)
$$

We can see that any *mixture model*, is defined through two main components.

**Base distribution**: $p(\mathbf{x}\vert z=c, \theta) = \mathcal{N}\left(\mathbf{x}\vert \mu_{c}, \Sigma_{c}\right)$

**Mixing probabilities** $p(z=c \vert \theta) = \pi_{c}$

### Optimizing this model

If we want to optimize our probabilistic model, we have to solve a typical *maximum likelihood* problem (considering that points $x_{i}$ are independent)
$$
\underset{\theta}{\text{max. }} p(x \vert \theta) = \prod_{i=1}^{N} p(x_{i} \vert \theta)
$$
In the case of a GMM, we could try to replace our full model inside this problem, which leads to a rather complicated formulation.
$$
\underset{\theta}{\text{max. }} \prod_{i=1}^{N} \sum_{c} \pi_{c} \mathcal{N}\left(x_{i}\vert \mu_{c}, \Sigma_{c}\right)
$$

Furthermore, we have to include the following constraints
$$
\sum_{c} \pi_{c} = 1; \pi_{c} \geq 0, \forall c \\
\Sigma_{c} \succcurlyeq 0
$$

Although, we could rely on optimization, we need to incorporate hard to follow constraints, which leads to slow and unstable training. Here, the reasoning is way easier if we introduce a latent variable

### Introducing a latent variable

We introduce the latent variable $z$, which _explains_ a given datapoint. In our case, this variable tells us _from which group_ the point was generated.

In our case, we can simply say that our latent variable corresponds to the mixing probabilities
$$
\begin{align}
p(z=c\vert\theta) &= \pi_{c} \\
p(\mathbf{x}\vert z=c,\theta) &= \mathcal{N}\left(\mathbf{x}\vert \mu_{c}, \Sigma_{c}\right)
\end{align}
$$

Therefore, our original model can be retrieved as a marginalization of the latent variable, as
$$
p(\mathbf{x} \vert \theta) = \sum_{c} p(\mathbf{x} \vert z=c,\theta) p(z=c\vert\theta)
$$

#### Estimating parameters

Now in order to optimize the parameters, we should need to optimize both the $\theta$ and $z$ variables. If we knew the cluster belonging (latent) variable $z$, then the optimization would be easy, since we know that
$$
p(\mathbf{x}\vert z=c, \theta) = \mathcal{N}\left(\mathbf{x}\vert \mu_{c}, \Sigma_{c}\right)
$$
And computing the parameters of this Gaussian is pretty easy, as we simply need to update the values depending on the corresponding points (weighted mean)
$$
\mu_{c} = \frac{\sum_{i} p(z=i\vert \mathbf{x}_{i}, \theta) \bx_{i}}{\sum_{i} p(z=i\vert \mathbf{x}_{i}, \theta)}
$$

Now in order to optimize the source variable $z$, the optimization would be easy if we knew the parameters as we have
$$
p(z = c \vert \mathbf{x}, \theta) = \frac{p(\mathbf{x} \vert z = c, \theta) p(z = c \vert \theta)}{Z}
$$

## Expectation-Maximization

As we have seen in the slides, a natural alternating solution emerges in the form of the **Expectation-Maximization** algorithm
1. Start with a random set of Gaussians parameters $\theta^{(0)}$
2. Repeat until convergence
    1. *E-step*: for each point, find weights encoding the probability of membership in each cluster by computing $p(z=c \vert \mathbf{x}_{i}, \theta^{(t)})$
    2. *M-step*: for each cluster, update its location, normalization, and shape based on *all* data points, making use of the weights, hence updating Gaussian parameters $\theta^{(t+1)}$

The result of this is that each cluster is associated not with a hard-edged sphere, but with a smooth Gaussian model.

### Deriving the E-Step
As discussed in the slides, we want to minimize the gap between our lower bound and the true distribution, which can be written as
$$
\mathcal{D} = \log p(\mathbf{x} \vert \theta) - \mathcal{L}(\theta\vert q)
$$
Developing this expression, we find out that
$$
\mathcal{D} = \sum_{i=1}^{N} \mathcal{D}_{KL} \left[ q(z_{i}) \Vert p(z_{i} \vert \mathbf{x}_{i}, \theta) \right]
$$
Hence, we need to minimize the KL divergence, which amounts to setting the approximate function as the posterior distribution
$$
q(z_{i}) = p(z_{i} \vert \mathbf{x}_{i}, \theta)
$$

### Deriving the M-step

Based on the approximation that we want to maximize
$$
\mathcal{L}(\theta, q) = \sum_{i=1}^{N} \sum_{c} q(t_{i}=c) \log\frac{p(x_{i}, t_{i}=c \vert \theta)}{q(t_{i}=c)}
$$
We have seen that expending this maximization led to
$$
\mathcal{L}(\theta, q) = \mathbb{E}_{q} \log p(x,t\vert \theta) + const
$$
Note that the fonction is usually concave, which makes it easy to optimize

## Using GMM with `scikit-learn`

By using the EM algorithm, a GMM will find a mixture of multi-dimensional Gaussian probability distributions that best model any input dataset, which can elegantly solve our clustering problem.

```python
from sklearn.mixture import GaussianMixture
gmm = GaussianMixture(n_components=4).fit(X)
labels = gmm.predict(X)
plt.figure(figsize=(10,8))
plt.scatter(X[:, 0], X[:, 1], c=labels, s=40, cmap='magma', edgecolor='w');
```

But because GMM contains a probabilistic model under the hood, it is also possible to find probabilistic cluster assignments—in Scikit-Learn this is done using the ``predict_proba`` method.
This returns a matrix of size ``[n_samples, n_clusters]`` which measures the probability that any point belongs to the given cluster:

```python
probs = gmm.predict_proba(X)
print(probs[:5].round(3))
```

We can visualize this uncertainty by, for example, making the size of each point proportional to the certainty of its prediction; looking at the following figure, we can see that it is precisely the points at the boundaries between clusters that reflect this uncertainty of cluster assignment:

```python
size = 100 * (probs.max(1) ** 2)  # square emphasizes differences
plt.figure(figsize=(10,8))
plt.scatter(X[:, 0], X[:, 1], c=labels, cmap='magma', s=size, edgecolor='w');
```

With this in place, we can take a look at what the four-component GMM gives us for our initial data:

```python
from helper_plot import plot_gmm
gmm = GaussianMixture(n_components=4, random_state=42)
plot_gmm(gmm, X)
```

Similarly, we can use the GMM approach to fit our stretched dataset. Here, we already see that allowing for a full covariance the model will fit even very oblong, stretched-out clusters (as opposed to the *hard* version of kMeans)

```python
rng = np.random.RandomState(13)
X_stretched = np.dot(X, rng.randn(2, 2))
gmm = GaussianMixture(n_components=4, covariance_type='full', random_state=42)
plot_gmm(gmm, X_stretched)
```

This makes clear that GMM addresses the two main practical issues with *k*-means encountered before.

## Implementing EM for GMM

Now, we will derive and implement formulas for the GMM ourselves. To do so, we will use samples from a Gaussian mixture model with unknown mean, variance, and priors. We will also have to derive both the E-step and M-step for our selected model.

### Reminder

We recall that the EM algorithm is a coordinate descent optimization of a lower bound
$$
\underset{\theta}{\text{argmax }} \mathcal{L}(\theta, q) = \underset{\theta}{\text{argmax }} \int q(\mathbf{z}) \log\frac{p(\mathbf{x}, \mathbf{z} \vert \theta)}{q(\mathbf{z})}d\mathbf{z}
$$

**E-step**: In the E-step, we use a lower-bounding approximation to find the latent variables $\mathbf{z}$, while using the current parameters estimate $\theta^{(t)}$
$$
\underset{q}{\text{argmax }} \mathcal{L}(\theta, q) \Leftrightarrow \underset{q}{\text{argmin }} \mathcal{D}_{KL} \left[q(\mathbf{z}) \Vert p(\mathbf{z}\vert\mathbf{x}, \theta) \right]
$$

We have seen that the optimal solution is $q(\mathbf{z}) = p(\mathbf{z}\vert\mathbf{x}, \theta)$, which means that we can safely use the conditional probability for this step.

**M-step**: In the M-step, we use our updated knowledge on the latent variable to update the parameters. Hence, we have to optimize
$$
\underset{\theta}{\text{argmax }} \mathcal{L}(\theta, q) \Leftrightarrow \underset{\theta}{\text{argmax }} \mathbb{E}_{q(\mathbf{z})}\log p(\mathbf{x},\mathbf{z} | \theta)
$$

For our GMM model, $\theta$ is a set of parameters that consists of mean vectors $\mu_c$, covariance matrices $\Sigma_c$ and priors $\pi_c$ for each component. The *latent variable* $z$ are indices of components to which each data point is assigned, i.e. $z_i$  is the cluster index for object $\mathbf{x}_i$. Therefore, the joint distribution of our GMM can be written as follows

$$\log p(\mathbf{z}, X \mid \theta) =  \sum\limits_{i=1}^N \log p(z_i, x_i \mid \theta) = \sum\limits_{i=1}^N \sum\limits_{c=1}^C q(z_i = c) \log \left (\pi_c \, \mathcal{N}(x_i \mid \mu_c, \Sigma_c)\right)
$$
where $\mathcal{N}(x_i \mid \mu_c, \Sigma_c) = \frac{1}{\sqrt{(2\pi)^n|\boldsymbol\Sigma_c|}}
\exp\left(-\frac{1}{2}({x}-{\mu_c})^T{\boldsymbol\Sigma_c}^{-1}({x}-{\mu_c})
\right)$ is the probability density function (pdf) of the normal distribution.

### E-step
In this step we need to estimate the posterior distribution over the latent variables with fixed values of parameters: $q_i(z_i) = p(z_i \mid \mathbf{x}_i, \theta)$. We assume that $z_i$ equals to the cluster index of the true component of the $\mathbf{x}_i$ object. To do so we need to compute $\pi_{ic} = p(z_i = c \mid \mathbf{x}_i, \theta)$. Note that $\sum\limits_{c=1}^C\pi_{ic}=1$.

***
**Exercise**

1. Implement E-step for GMM using template below.

***

```python
from scipy.stats import multivariate_normal

def E_step(X, pi, mu, sigma):
    """
    Performs E-step on GMM model
    Each input is numpy array:
    X: (N x d), data points
    pi: (C), mixture component weights
    mu: (C x d), mixture component means
    sigma: (C x d x d), mixture component covariance matrices

    Returns:
    gamma: (N x C), probabilities of clusters for objects
    """
    N = X.shape[0] # number of objects
    C = pi.shape[0] # number of clusters
    d = mu.shape[1] # dimension of each object
    pi_update = np.zeros((N, C)) # distribution q(T)

    ######################
    # YOUR CODE GOES HERE
    ######################

    return pi_update
```

```python
pi0 = np.random.randn(4, X.shape[0])
mu0 = np.random.randn(4, 2)
sigma0 = np.zeros((4, 2, 2))
for i in range(4):
    sigma0[i] = np.eye(2, 2)
pi = E_step(X, pi0, mu0, sigma0)
```

### M-step

In M-step we need to maximize $\mathbb{E}_{q(T)}\log p(X,T | \theta)$ with respect to $\theta$. In our model this means that we need to find optimal values of $\pi$, $\mu$, $\Sigma$. To do so, you need to compute the derivatives and
set them to zero. You should start by deriving formulas for $\mu$ as it is the easiest part. Then move on to $\Sigma$. Here it is crucial to optimize function w.r.t. to $\Lambda = \Sigma^{-1}$ and then inverse obtained result. Finaly, to compute $\pi$, you will need Lagrange Multipliers technique to satisfy constraint $\sum\limits_{i=1}^{n}\pi_i = 1$.

Important note: You will need to compute derivatives of scalars with respect to matrices. To refresh this technique from previous courses, see  wiki article about it . Main formulas of matrix derivatives can be found in Chapter 2 of The Matrix Cookbook. For example, there you may find that $\frac{\partial}{\partial A}\log |A| = A^{-T}$.

Task 2: Implement M-step for GMM using template below.

```python
def M_step(X, pi_update):
    """
    Performs M-step on GMM model
    Each input is numpy array:
    X: (N x d), data points
    gamma: (N x C), distribution q(T)

    Returns:
    pi: (C)
    mu: (C x d)
    sigma: (C x d x d)
    """
    N = X.shape[0] # number of objects
    C = pi_update.shape[1] # number of clusters
    d = X.shape[1] # dimension of each object

    ######################
    # YOUR CODE GOES HERE
    ######################

    return pi, mu, sigma
```

```python
pi_update = E_step(X, pi0, mu0, sigma0)
pi, mu, sigma = M_step(X, pi_update)
```

### Loss function

Finally, we need some function to track convergence. We will use variational lower bound $\mathcal{L}$ for this purpose. We will stop our EM iterations when $\mathcal{L}$ will saturate. Usually, you will need only about 10-20 iterations to converge. It is also useful to check that this function never decreases during training. If it does, you have a bug in your code.

Task 3: Implement a function that will compute $\mathcal{L}$ using template below.

$$\mathcal{L} = \sum_{i=1}^{N} \sum_{c=1}^{C} q(t_i =c) (\log \pi_c + \log f_{\!\mathcal{N}}(x_i \mid \mu_c, \Sigma_c)) - \sum_{i=1}^{N} \sum_{c=1}^{K} q(t_i =c) \log q(t_i =c)$$

```python
def compute_vlb(X, pi, mu, sigma, pi_update):
    """
    Each input is numpy array:
    X: (N x d), data points
    gamma: (N x C), distribution q(T)
    pi: (C)
    mu: (C x d)
    sigma: (C x d x d)

    Returns value of variational lower bound
    """
    N = X.shape[0] # number of objects
    C = pi_update.shape[1] # number of clusters
    d = X.shape[1] # dimension of each object

    ######################
    # YOUR CODE GOES HERE
    ######################

    return loss
```

```python
pi, mu, sigma = pi0, mu0, sigma0
pi_update = E_step(X, pi, mu, sigma)
pi, mu, sigma = M_step(X, pi_update)
loss = compute_vlb(X, pi, mu, sigma, pi_update)
```

### Bringing it all together

Now that we have E step, M step and VLB, we can implement the training loop. We will initialize values of $\pi$, $\mu$ and $\Sigma$ to some random numbers, train until $\mathcal{L}$ stops changing, and return the resulting points. We also know that the EM algorithm converges to local optima. To find a better local optima, we will restart the algorithm multiple times from different (random) starting positions. Each training trial should stop either when maximum number of iterations is reached or when relative improvement is smaller than given tolerance ($|\frac{\mathcal{L}_i-\mathcal{L}_{i-1}}{\mathcal{L}_{i-1}}| \le \text{rtol}$).

Remember, that initial (random) values of $\pi$ that you generate must be non-negative and sum up to 1. Also, $\Sigma$ matrices must be symmetric and positive semi-definite. If you don't know how to generate those matrices, you can use $\Sigma=I$ as initialization.

You will also sometimes get numerical errors because of component collapsing. The easiest way to deal with this problems is to restart the procedure.

Task 4: Implement training procedure

```python
import math
def train_EM(X, C, rtol=1e-3, max_iter=100, restarts=10):
    '''
    Starts with random initialization *restarts* times
    Runs optimization until saturation with *rtol* reached
    or *max_iter* iterations were made.

    X: (N, d), data points
    C: int, number of clusters
    '''
    N = X.shape[0] # number of objects
    d = X.shape[1] # dimension of each object
    best_loss = None
    best_pi = None
    best_mu = None
    best_sigma = None

    for _ in range(restarts):
        try:
            ######################
            # YOUR CODE GOES HERE
            ######################

        except np.linalg.LinAlgError:
            print("Singular matrix: components collapsed")
            pass

    return best_loss, best_pi, best_mu, best_sigma
```

```python
best_loss, best_pi, best_mu, best_sigma = train_EM(X, 4)
```

If you implemented all the steps correctly, your algorithm should converge in about 20 iterations. Let's plot the clusters to see it. We will assign a cluster label as the most probable cluster index. This can be found using a matrix $\gamma$ computed on last E-step.

```python
gamma = E_step(X, best_pi, best_mu, best_sigma)
labels = gamma.argmax(axis=1)
colors = np.array([(31, 119, 180), (255, 127, 14), (44, 160, 44), (50, 50, 50)]) / 255.
plt.figure(figsize=(10, 8))
plt.scatter(X[:, 0], X[:, 1], c=colors[labels], s=50, edgecolors='w')
```

## GMM as *Density Estimation*

Though GMM is often categorized as a clustering algorithm, fundamentally it is an algorithm for *density estimation*.
That is to say, the result of a GMM fit to some data is technically not a clustering model, but a generative probabilistic model describing the distribution of the data.

As an example, consider some data generated from Scikit-Learn's ``make_moons`` function, which we saw in [In Depth: K-Means Clustering](05.11-K-Means.ipynb):

```python
from sklearn.datasets import make_moons
Xmoon, ymoon = make_moons(200, noise=.05, random_state=0)
plt.figure(figsize=(10, 8))
plt.scatter(Xmoon[:, 0], Xmoon[:, 1], edgecolor='w', s=50);
```

If we try to fit this with a two-component GMM viewed as a clustering model, the results are not particularly useful:

```python
gmm2 = GaussianMixture(n_components=2, covariance_type='full', random_state=0)
plot_gmm(gmm2, Xmoon)
```

But if we instead use many more components and ignore the cluster labels, we find a fit that is much closer to the input data:

```python
gmm16 = GaussianMixture(n_components=16, covariance_type='full', random_state=0)
plot_gmm(gmm16, Xmoon, label=False)
```

Here the mixture of 16 Gaussians serves not to find separated clusters of data, but rather to model the overall *distribution* of the input data.
This is a generative model of the distribution, meaning that the GMM gives us the recipe to generate new random data distributed similarly to our input.
For example, here are 400 new points drawn from this 16-component GMM fit to our original data:

```python
Xnew = gmm16.sample(400)[0]
plt.figure(figsize=(10, 8))
plt.scatter(Xnew[:, 0], Xnew[:, 1], s=40, edgecolor='w');
```

GMM is convenient as a flexible means of modeling an arbitrary multi-dimensional distribution of data.

### How many components?

The fact that GMM is a generative model gives us a natural means of determining the optimal number of components for a given dataset.
A generative model is inherently a probability distribution for the dataset, and so we can simply evaluate the *likelihood* of the data under the model, using cross-validation to avoid over-fitting.
Another means of correcting for over-fitting is to adjust the model likelihoods using some analytic criterion such as the [Akaike information criterion (AIC)](https://en.wikipedia.org/wiki/Akaike_information_criterion) or the [Bayesian information criterion (BIC)](https://en.wikipedia.org/wiki/Bayesian_information_criterion).
Scikit-Learn's ``GMM`` estimator actually includes built-in methods that compute both of these, and so it is very easy to operate on this approach.

Let's look at the AIC and BIC as a function as the number of GMM components for our moon dataset:

```python
n_components = np.arange(1, 21)
models = [GaussianMixture(n, covariance_type='full', random_state=0).fit(Xmoon)
          for n in n_components]

plt.figure(figsize=(10, 8))
plt.plot(n_components, [m.bic(Xmoon) for m in models], label='BIC')
plt.plot(n_components, [m.aic(Xmoon) for m in models], label='AIC')
plt.legend(loc='best')
plt.xlabel('n_components');
```

The optimal number of clusters is the value that minimizes the AIC or BIC, depending on which approximation we wish to use. The AIC tells us that our choice of 16 components above was probably too many: around 8-12 components would have been a better choice.
As is typical with this sort of problem, the BIC recommends a simpler model.

Notice the important point: this choice of number of components measures how well GMM works *as a density estimator*, not how well it works *as a clustering algorithm*.
I'd encourage you to think of GMM primarily as a density estimator, and use it for clustering only when warranted within simple datasets.


# Vanilla RNNs, GRUs and the `scan` function

In this notebook, you will learn how to define the forward method for vanilla RNNs and GRUs. Additionally, you will see how to define and use the function `scan` to compute forward propagation for RNNs.

By completing this notebook, you will:

- Be able to define the forward method for vanilla RNNs and GRUs
- Be able to define the `scan` function to perform forward propagation for RNNs
- Understand how forward propagation is implemented for RNNs.

```python
import numpy as np
from numpy import random
from time import perf_counter
```

An implementation of the `sigmoid` function is provided below so you can use it in this notebook.

```python
def sigmoid(x): # Sigmoid function
    return 1.0 / (1.0 + np.exp(-x))
```

# Part 1: Forward method for vanilla RNNs and GRUs

In this part of the notebook, you'll see the implementation of the forward method for a vanilla RNN and you'll implement that same method for a GRU. For this excersice you'll use a set of random weights and variables with the following dimensions:

- Embedding size (`emb`) : 128
- Hidden state size (`h_dim`) : (16,1)

The weights `w_` and biases `b_` are initialized with dimensions (`h_dim`, `emb + h_dim`) and (`h_dim`, 1). We expect the hidden state `h_t` to be a column vector with size (`h_dim`,1) and the initial hidden state `h_0` is a vector of zeros.

```python
random.seed(10)                 # Random seed, so your results match ours
emb = 128                       # Embedding size
T = 256                         # Number of variables in the sequences
h_dim = 16                      # Hidden state dimension
h_0 = np.zeros((h_dim, 1))      # Initial hidden state
# Random initialization of weights and biases
w1 = random.standard_normal((h_dim, emb+h_dim))
w2 = random.standard_normal((h_dim, emb+h_dim))
w3 = random.standard_normal((h_dim, emb+h_dim))
b1 = random.standard_normal((h_dim, 1))
b2 = random.standard_normal((h_dim, 1))
b3 = random.standard_normal((h_dim, 1))
X = random.standard_normal((T, emb,1))
#需要shape 是(256, 128,1 ), 不能是(256, 128),
# 第一个input是  X[0] dim (128,1), h_0是 shape 是(128, 1), 可以concatenate,
# 如果X shape是(256, 128), X[0] dim (128,）, error
print(X.shape)
weights = [w1, w2, w3, b1, b2, b3]
```

    (256, 128, 1)

## 1.1 Forward method for vanilla RNNs

The vanilla RNN cell is quite straight forward. Its most general structure is presented in the next figure:

As you saw in the lecture videos, the computations made in a vanilla RNN cell are equivalent to the following equations:

\begin{equation}
h^{<t>}=g(W_{h}[h^{<t-1>},x^{<t>}] + b_h)
\label{eq: htRNN}
\end{equation}

\begin{equation}
\hat{y}^{<t>}=g(W_{yh}h^{<t>} + b_y)
\label{eq: ytRNN}
\end{equation}

where $[h^{<t-1>},x^{<t>}]$ means that $h^{<t-1>}$ and $x^{<t>}$ are concatenated together. In the next cell we provide the implementation of the forward method for a vanilla RNN.

```python
def forward_V_RNN(inputs, weights): # Forward propagation for a a single vanilla RNN cell
    x, h_t = inputs

    # weights.
    wh, _, _, bh, _, _ = weights

    # new hidden state
    h_t = np.dot(wh, np.concatenate([h_t, x])) + bh #np.concatenate default axis =0, 每行后面加
    h_t = sigmoid(h_t)

    return h_t, h_t
```

As you can see, we omitted the computation of $\hat{y}^{<t>}$. This was done for the sake of simplicity, so you can focus on the way that hidden states are updated here and in the GRU cell.

## 1.2 Forward method for GRUs

A GRU cell have more computations than the ones that vanilla RNNs have. You can see this visually in the following diagram:

As you saw in the lecture videos, GRUs have relevance $\Gamma_r$ and update $\Gamma_u$ gates that control how the hidden state $h^{<t>}$ is updated on every time step. With these gates, GRUs are capable of keeping relevant information in the hidden state even for long sequences. The equations needed for the forward method in GRUs are provided below:

\begin{equation}
\Gamma_r=\sigma{(W_r[h^{<t-1>}, x^{<t>}]+b_r)}
\end{equation}

\begin{equation}
\Gamma_u=\sigma{(W_u[h^{<t-1>}, x^{<t>}]+b_u)}
\end{equation}

\begin{equation}
c^{<t>}=\tanh{(W_h[\Gamma_r*h^{<t-1>},x^{<t>}]+b_h)}
\end{equation}

\begin{equation}
h^{<t>}=\Gamma_u*c^{<t>}+(1-\Gamma_u)*h^{<t-1>}
\end{equation}

In the next cell, please implement the forward method for a GRU cell by computing the update `u` and relevance `r` gates, and the candidate hidden state `c`.

```python
def forward_GRU(inputs, weights): # Forward propagation for a single GRU cell
    x, h_t = inputs
    print(x.shape)
    print(h_t.shape)
    # weights.
    wu, wr, wc, bu, br, bc = weights

    # Update gate
    ### START CODE HERE (1-2 lINES) ###
    u = np.dot(wu, np.concatenate([h_t, x])) + bu
    u = sigmoid(u)
    ### END CODE HERE ###

    # Relevance gate
    ### START CODE HERE (1-2 lINES) ###
    r = np.dot(wr, np.concatenate([h_t, x])) + br
    r = sigmoid(r)
    ### END CODE HERE ###

    # Candidate hidden state
    ### START CODE HERE (1-2 lINES) ###
    c = np.dot(wc, np.concatenate([r * h_t, x])) + bc
    c = np.tanh(c)
    ### END CODE HERE ###

    # New Hidden state h_t
    h_t = u* c + (1 - u)* h_t
    return h_t, h_t
```

Run the following cell to check your implementation.

```python
forward_GRU([X[1],h_0], weights)[0]
```

    (128, 1)
    (16, 1)

    array([[ 9.77779014e-01],
           [-9.97986240e-01],
           [-5.19958083e-01],
           [-9.99999886e-01],
           [-9.99707004e-01],
           [-3.02197037e-04],
           [-9.58733503e-01],
           [ 2.10804828e-02],
           [ 9.77365398e-05],
           [ 9.99833090e-01],
           [ 1.63200940e-08],
           [ 8.51874303e-01],
           [ 5.21399924e-02],
           [ 2.15495959e-02],
           [ 9.99878828e-01],
           [ 9.77165472e-01]])

Expected output:

array([[ 9.77779014e-01],
       [-9.97986240e-01],
       [-5.19958083e-01],
       [-9.99999886e-01],
       [-9.99707004e-01],
       [-3.02197037e-04],
       [-9.58733503e-01],
       [ 2.10804828e-02],
       [ 9.77365398e-05],
       [ 9.99833090e-01],
       [ 1.63200940e-08],
       [ 8.51874303e-01],
       [ 5.21399924e-02],
       [ 2.15495959e-02],
       [ 9.99878828e-01],
       [ 9.77165472e-01]])

# Part 2: Implementation of the `scan` function

In the lectures you saw how the `scan` function is used for forward propagation in RNNs. It takes as inputs:

- `fn` : the function to be called recurrently (i.e. `forward_GRU`)
- `elems` : the list of inputs for each time step (`X`)
- `weights` : the parameters needed to compute `fn`
- `h_0` : the initial hidden state

`scan` goes through all the elements `x` in `elems`, calls the function `fn` with arguments ([`x`, `h_t`],`weights`), stores the computed hidden state `h_t` and appends the result to a list `ys`. Complete the following cell by calling `fn` with arguments ([`x`, `h_t`],`weights`).

```python
def scan(fn, elems, weights, h_0=None): # Forward propagation for RNNs
    h_t = h_0
    ys = []
    for x in elems:
        ### START CODE HERE (1 lINE) ###
        y, h_t = fn([x, h_t], weights)
        ### END CODE HERE ###
        ys.append(y)
    return ys, h_t
```

# Part 3: Comparison between vanilla RNNs and GRUs

You have already seen how forward propagation is computed for vanilla RNNs and GRUs. As a quick recap, you need to have a forward method for the recurrent cell and a function like `scan` to go through all the elements from a sequence using a forward method. You saw that GRUs performed more computations than vanilla RNNs, and you can check that they have 3 times more parameters. In the next two cells, we compute forward propagation for a sequence with 256 time steps (`T`) for an RNN and a GRU with the same hidden state `h_t` size (`h_dim`=16).

```python
# vanilla RNNs
tic = perf_counter()
ys, h_T = scan(forward_V_RNN, X, weights, h_0)
toc = perf_counter()
RNN_time=(toc-tic)*1000
print (f"It took {RNN_time:.2f}ms to run the forward method for the vanilla RNN.")
```

    It took 2.67ms to run the forward method for the vanilla RNN.

```python
# GRUs
tic = perf_counter()
ys, h_T = scan(forward_GRU, X, weights, h_0)
toc = perf_counter()
GRU_time=(toc-tic)*1000
print (f"It took {GRU_time:.2f}ms to run the forward method for the GRU.")
```

    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    (128, 1)
    (16, 1)
    It took 53.07ms to run the forward method for the GRU.

As you were told in the lectures, GRUs take more time to compute (However, sometimes, although a rare occurrence, Vanilla RNNs take more time. Can you figure out what might cause this ?). This means that training and prediction would take more time for a GRU than for a vanilla RNN. However, GRUs allow you to propagate relevant information even for long sequences, so when selecting an architecture for NLP you should assess the tradeoff between computational time and performance.

Congratulations! Now you know how the forward method is implemented for vanilla RNNs and GRUs, and you know how the scan function provides an abstraction for forward propagation in RNNs.

Natural Language Processing with Sequence Models/week2-Vanilla RNNs, GRUs and the scan function.ipynb