# Problema de selección de portafolio con preferencias media-varianza

En la clase pasada hablamos acerca de:
- preferencias,
- funciones de utilidad,
- la actitud de los inversionistas de cara al riesgo,
- la aversión al riesgo, entre otros.

Todas ellas son piezas que necesitamos para responder la pregunta de ¿cómo un inversionista toma la decisión óptima de selección de portafolio?

En esta clase al fin estamos listos para ensamblar estos conceptos y escribir el problema de selección de portafolios.

En el camino aprenderemos acerca del concepto de **utilidad esperada**, que nos permite trabajar con incertidumbre en el modelado económico (una de las ideas más importantes en economía). Esta idea tiene más de 60 años, y básicamente dice que los individuos, cuando están de cara a incertidumbre, maximizan el valor esperado de su utilidad (solo cierto si somos homo economicus).

Además del concepto de utilidad esperada, aprenderemos acerca de **preferencias media-varianza**. Es decir, supondremos que los inversionistas toman decisiones basados en un tipo particular de preferencias.

Con lo anterior, estableceremos el problema de selección de portafolios.

**Objetivos:**

- ¿Qué es utilidad esperada?
- ¿Qué son las preferencias media-varianza?
- Funciones de utilidad media-varianza.
- Enunciado y solución del problema básico de selección de portafolio.

*Referencia:*
- Notas del curso "Portfolio Selection and Risk Management", Rice University, disponible en Coursera.
___

## 1. Utilidad esperada
- Básicamente combina las probabilidades de los resultados con cómo los inversionistas se sienten con dichos resultados.
- En otras palabras, la utilidad esperada multiplica la probabilidad de suceso de un evento con la utilidad que genera dicho evento.

Recordemos que las *funciones de utilidad* permiten a los inversionistas expresar cómo se sienten con los resultados, especialmente en los malos ratos.

Entonces la *utilidad esperada* es una herramienta que nos permite cuantificar cómo nos sentimos en nuestros malos momentos económicos, capturando el riesgo con la probabilidad de ocurrencia de dichos malos momentos.

Dado este marco de trabajo, cualquier decisión se puede escribir como la maximización de la utilidad esperada:
\begin{align}
\max_{\theta} & \quad E[U(W)], \\
\end{align}
mediante la escogencia de cierta variable $\theta$ (gastos, planes de ahorro, compra de activos, planes de producción, etc.).

Para nuestros propósitos, la variable de decisión serán los pesos o ponderaciones del portafolio.

Adicionalmente, en el contexto de la decisión de distribución de la riqueza entre activos, el problema de maximización tendrá comúnmente las siguientes restricciones:
- universo de inversión,
- posición en los activos dados.

**Ejemplo.**

Supongamos que un inversionista debe determinar la composición óptima de su portafolio, que contiene activos y bonos. Supongamos que son los únicos instrumentos disponibles.

Sean:
- $w_s$: peso o ponderación de activos en el portafolio,
- $w_b$: peso o ponderación de bonos en el portafolio,
- $r_s$: rendimiento de los activos, y
- $r_b$: rendimiento de los bonos.

De manera que podemos escribir el problema de selección de portafolios como la maximización de la utilidad esperade de nuestra riqueza futura, la cual dependerá de nuestros rendimientos:

\begin{align}
\max_{w_s,w_b} &\quad E[U(W)]\\
\text{s. a.}   &\quad W=W_0(1+w_sr_s+w_br_b)\\
               &\quad w_s+w_b=1\\
               &\quad w_s,w_b\geq0
\end{align}

Preguntas:
- ¿Qué significan las restricciones?
- Ya que tenemos planteado este problema básico, ¿qué haría falta para empezar a resolverlo?
___

## 2. Preferencias media-varianza

### 2.1. Utilidad media-varianza

Entonces, ¿qué funciones de utilidad deberíamos de usar en este problema de selección de portafolios?

- La respuesta es: **preferencias media-varianza**.
- Éstas serán representadas en términos de funciones de utilidad como: **utilidad media-varianza**.

Usamos la *utilidad media-varianza* en el problema de selección de portafolios dado que ésta decribe el "trade-off" entre riesgo y rendimiento que enfrentan los inversionistas. La *utilidad media-varianza* está dada por la siguiente expresión:

$$U=E[r_p]-\frac{1}{2}\gamma\sigma_p^2,$$

donde
- $E[r_p]$ es el rendimiento esperado del portafolio,
- $\sigma_p^2$ es la varianza del portafolio, y
- $\gamma$ es el coeficiente de aversión al riesgo.

#### Intuición acerca de la función de utilidad media-varianza:
- Sólo se preocupa por medias :) y varianzas :(.
- Incrementa con: rendimiento esperado del portafolio.
- Decrece con: varianza del portafolio.
- Malos tiempos: rendimientos son bajos y las volatilidades son altas.
- Conecta bastante bien con la teoría moderna de portafolios, la cual caracteriza los rendimientos con medias y varianzas únicamente.
- Criticada por su limitación: supone que los inversionistas sólo se preocupan por medias y varianzas.

### 2.2. Curvas de indiferencia

*¿Recuerdan las curvas de nivel que se ven en cálculo de varias variables?*
- Bien, acá nos servirán para representar la utilidad media-varianza gráficamente.
- En el contexto de utilidad media-varianza, las curvas de nivel se llaman **curvas de indiferencia**.

Dados ciertos niveles de utilidad $U_1>U_2>U_3$, las curvas de indiferencia relativas a estos niveles de utilidad, son los lugares geométricos en el espacio de rendimiento esperado vs. volatilidad representados por las siguientes expresiones

$$U_1=E[r_p]-\frac{1}{2}\gamma\sigma_p^2\Rightarrow E[r_p]=\frac{1}{2}\gamma\sigma_p^2+U_1,$$

$$U_2=E[r_p]-\frac{1}{2}\gamma\sigma_p^2\Rightarrow E[r_p]=\frac{1}{2}\gamma\sigma_p^2+U_2,$$

$$U_3=E[r_p]-\frac{1}{2}\gamma\sigma_p^2\Rightarrow E[r_p]=\frac{1}{2}\gamma\sigma_p^2+U_3.$$

**Gráficamente**

```python
# Importar numpy y pyplot
import numpy as np
from matplotlib import pyplot as plt
%matplotlib inline
```

```python
# Coeficiente de aversión al riesgo (entre 1 y 10 comúnmente)
g = 5
# Niveles de utilidad
U1, U2, U3 = 0.21, 0.14, 0.07
# Vector de volatilidades (sugerido 1%-60%)
sp = np.linspace(0.01, 0.6, 50)
# Curvas de indiferencia
Erp1 = 0.5 * g * sp**2 + U1
Erp2 = 0.5 * g * sp**2 + U2
Erp3 = 0.5 * g * sp**2 + U3
```

```python
# Gráfica
plt.figure(figsize=(6, 4))
plt.plot(sp, Erp1, label=f'Nivel de utilidad $U_1$={U1}')
plt.plot(sp, Erp2, label=f'Nivel de utilidad $U_2$={U2}')
plt.plot(sp, Erp3, label=f'Nivel de utilidad $U_3$={U3}')
plt.grid()
plt.legend(loc='best')
plt.xlabel('Volatilidad $\sigma$')
plt.ylabel('Rendimiento esperado $E[r]$')
```

Bueno, ¿y porqué se llaman curvas de indiferencia?, ¿qué representa una cuva de indiferencia?

- Porque sobre una misma curva el nivel de utilidad es el mismo (es indiferente).
- Son todas las combinaciones de riesgo y rendimiento que producen un mismo nivel de utilidad.

Volviendo al problema de selección de portafolios, queremos la utilidad más alta.
- ¿Cuál de las anteriores curvas de indiferencia corresponde a la utilidad más alta?
- Intuitivamente, ¿porqué?
- Curvas de indiferencia para niveles de utilidad más altos, estarán...

Notamos además que las anteriores curvas de indiferencia son *paralelas* una con otra. Claro, las dibujamos con el mismo coeficiente de aversión al riesgo.

$$\frac{d E[r_p]}{d\sigma_p}=\frac{d }{d\sigma_p}\left[\frac{1}{2}\gamma\sigma_p^2+U\right]=\gamma\sigma_p.$$

¿Cómo cambian estas curvas para coeficientes de aversión al riesgo más altos?

```python
# Coeficientes de aversión al riesgo (entre 1 y 10 comúnmente)
g1, g2, g3 = 10, 7, 4
# Nivel de utilidad
U = 0.07
# Vector de volatilidades (sugerido 1%-60%)
sp = np.linspace(0.01, 0.6, 50)
# Curvas de indiferencia
Erp1 = 0.5 * g1 * sp**2 + U
Erp2 = 0.5 * g2 * sp**2 + U
Erp3 = 0.5 * g3 * sp**2 + U
```

```python
# Gráfica
plt.figure(figsize=(6, 4))
plt.plot(sp, Erp1, label=f'Coeficiente de aversión al riesgo $\gamma_1$={g1}')
plt.plot(sp, Erp2, label=f'Coeficiente de aversión al riesgo $\gamma_2$={g2}')
plt.plot(sp, Erp3, label=f'Coeficiente de aversión al riesgo $\gamma_3$={g3}')
plt.grid()
plt.axvline(x=0.5, ls='--', color='grey')
plt.axhline(y=0.5, ls='--', color='grey')
plt.legend(loc='best')
plt.xlabel('Volatilidad $\sigma$')
plt.ylabel('Rendimiento esperado $E[r]$')
```

¿Cómo interpretamos las anteriores gráficas?, ¿qué pasa con las personas más aversas al riesgo?
- Se puede ver de dos maneras: para un mismo nivel de rendimiento esperado, una persona más aversa al riesgo soporta un nivel menor de riesgo; equivalentemente, para un mismo nivel de riesgo, una persona más aversa al riesgo requerirá un nivel de rendimiento esperado más alto.

Con todo lo anterior, el problema de selección de portafolios se puede plantear como *encontrar la curva de indeferencia más alta dado el conjunto de oportunidades de inversión y restricciones*.

## 3. Problema de selección de portafolios: una ilustración

Ahora ilustraremos el problema de selección de portafolios con algunos datos.
- Por ahora solo queremos ilustrar gráficamente cómo se resuelve este problema. Trabajar en la intuición.
- En las siguientes dos clases nos enfocaremos en cómo resolverlo analíticamente.

Acá tenemos el rendimiento medio anual y la volatilidad para dos instrumentos usando datos de EU: instrumentos de deuda (bonos) y acciones. Supondremos que el inversionista solo puede invertir en estas dos clases de instrumentos.

```python
# Importamos pandas
import pandas as pd
```

```python
# Datos
data = pd.DataFrame(index=['Stocks','Bonds', 'CorrSB'], columns=['Mean', 'Std'])
data['Mean'] = [0.119, 0.0591, 0.113]
data['Std'] = [0.1915, 0.0833, None]
data
```

      Mean
      Std

      Stocks
      0.1190
      0.1915

      Bonds
      0.0591
      0.0833

      CorrSB
      0.1130
      NaN

Entonces, ¿cuál es la distribución de riqueza óptima?, o más bien, ¿cuál es la composición óptima del portafolio para un inversionista dado su nivel de aversión al riesgo?

**Primero.** Recordamos que, para dos activos, podemos trazar la frontera de mínima varianza tomando todas las posibles combinaciones de los dos activos.

De nuevo, sean:
- $w_s=w$: peso o ponderación de activos en el portafolio,
- $w_b=1-w$: peso o ponderación de bonos en el portafolio,
- $r_s$: rendimiento de los activos, y
- $r_b$: rendimiento de los bonos.

Entonces

$$E[r_p]=wE[r_{s}]+(1-w)E[r_b]$$

$$\sigma_p^2=w^2\sigma_{s}^2+(1-w)^2\sigma_b^2+2w(1-w)\rho_{s,b}\sigma_s\sigma_b$$

```python
# Vector de w variando entre 0 y 1 con n pasos
N = 101
w = np.linspace(0, 1, N)
# Rendimientos esperados individuales
Ers = data.loc['Stocks', 'Mean']
Erb = data.loc['Bonds', 'Mean']
# Volatilidades individuales
ss = data.loc['Stocks', 'Std']
sb = data.loc['Bonds', 'Std']
# Correlacion
rsb = data.loc['CorrSB', 'Mean']
Ers, Erb, ss, sb, rsb
```

    (0.119, 0.0591, 0.1915, 0.0833, 0.113)

```python
# Crear un DataFrame cuyas columnas sean rendimiento
# y volatilidad del portafolio para cada una de las w
# generadas
portafolios = pd.DataFrame({'Media': w * Ers + (1 - w) * Erb,
                            'Vol': ((w * ss)**2 + ((1 - w) * sb)**2 + 2 * w * (1 - w) * ss * sb * rsb)**0.5
                           },
                           index = w
                          )
portafolios.index.name = 'w'
portafolios.head()
```

      Media
      Vol

      w

      0.00
      0.059100
      0.083300

      0.01
      0.059699
      0.082705

      0.02
      0.060298
      0.082155

      0.03
      0.060897
      0.081650

      0.04
      0.061496
      0.081191

```python
# Gráfica
plt.figure(figsize=(6, 4))
plt.plot(portafolios['Vol'], portafolios['Media'], label='portafolios')
plt.plot(ss, Ers, 'og', ms=10, label='Stocks')
plt.plot(sb, Erb, 'or', ms=10, label='Bonds')
plt.xlabel('Volatilidad $\sigma$')
plt.ylabel('Rendimiento esperado $E[r]$')
plt.legend(loc='best')
plt.grid()
```

**Segundo.** Graficamos en la misma ventana, curvas de indiferencia.

```python
# Niveles de utilidad
U1, U2, U3 = 0.04, 0.045, 0.0512
# Coeficiente de aversión al riesgo
g = 7
# Curvas de indiferencia
sp = np.linspace(0.08, 0.18, 50)
Erp1 = 0.5 * g * sp**2 + U1
Erp2 = 0.5 * g * sp**2 + U2
Erp3 = 0.5 * g * sp**2 + U3
```

```python
# Gráfica
plt.figure(figsize=(6, 4))
plt.plot(portafolios['Vol'], portafolios['Media'], label='portafolios')
plt.plot(sp, Erp1, label=f'Nivel de utilidad $U_1$={U1}')
plt.plot(sp, Erp2, label=f'Nivel de utilidad $U_2$={U2}')
plt.plot(sp, Erp3, label=f'Nivel de utilidad $U_3$={U3}')
plt.plot(ss, Ers, 'og', ms=10, label='Stocks')
plt.plot(sb, Erb, 'or', ms=10, label='Bonds')
plt.xlabel('Volatilidad $\sigma$')
plt.ylabel('Rendimiento esperado $E[r]$')
plt.legend(loc='best')
plt.grid()
```

**Tercero.** La elección óptima está dada por la curva de indiferencia para el nivel de utilidad más alto que es tangente a la frontera media-varianza.
- Claramente, esta selección depende del coeficiente de aversión al riesgo.

```python
# Gráfica con zoom
# Gráfica
plt.figure(figsize=(6, 4))
plt.plot(portafolios['Vol'], portafolios['Media'], label='portafolios')
plt.plot(sp, Erp1, label=f'Nivel de utilidad $U_1$={U1}')
plt.plot(sp, Erp2, label=f'Nivel de utilidad $U_2$={U2}')
plt.plot(sp, Erp3, label=f'Nivel de utilidad $U_3$={U3}')
plt.plot(ss, Ers, 'og', ms=10, label='Stocks')
plt.plot(sb, Erb, 'or', ms=10, label='Bonds')
plt.xlabel('Volatilidad $\sigma$')
plt.ylabel('Rendimiento esperado $E[r]$')
plt.legend(loc='best')
plt.grid()
plt.axis([0.08, 0.10, 0.07, 0.09])
```

# Anuncios parroquiales

## 1. Quiz la siguiente clase.
## 2. Un par de artículos del WSJ y el NYT que discuten herramientas disponibles para la medición de su propia tolerancia al riesgo:
- [Artículo 1](https://www.nytimes.com/2016/02/13/your-money/as-stocks-fall-its-time-to-measure-your-risk-tolerance.html)
- [Artículo 2](https://www.wsj.com/articles/check-your-tolerance-for-investment-risk-now-before-markets-sag-1405619939)

Created with Jupyter by Esteban Jiménez Rodríguez.


```python
import numpy as np
import pandas as pd
import linearsolve as ls
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Class 16: Real and Nominal Business Cycle Data

In this notebook, we will briefly explore US macroeconomic data suggesting that, contrary to the assumptions of most RBC models, there is in fact a relationship between real and nominal quantities over the business cycle. We will estimate an aggregate supply equation for the US and a monetary policy rule for the FOMC.

## Data

The file `business_cycle_data_actual_trend_cycle.csv`, available at https://github.com/letsgoexploring/econ126/raw/master/Data/Csv/business_cycle_data_actual_trend_cycle.csv, contains actual and trend data for real GDP per capita, real consumption per capita, real investment per capita, real physical capital per capita, TFP, hours per capita, the rea money supply (M2), (nominal) interest rate on 3-month T-bills, the PCE inflation rate, and the unemployment rate; each at quarterly frequency. The GDP, consumption, investment, capital, and money supply data are in terms of 2012 dollars. Hours is measured as an index with the value in October 2012 set to 100.

```python
# Read business_cycle_data_actual_trend.csv into a Pandas DataFrame with the first column set as the index and parse_dates=True
data = pd.read_csv('https://raw.githubusercontent.com/letsgoexploring/econ126/master/Data/Csv/business_cycle_data_actual_trend_cycle.csv',index_col=0,parse_dates=True)

# Print the last five rows of the data
data.tail()
```

      gdp
      gdp_trend
      gdp_cycle
      consumption
      consumption_trend
      consumption_cycle
      investment
      investment_trend
      investment_cycle
      hours
      ...
      real_m2_cycle
      t_bill_3mo
      t_bill_3mo_trend
      t_bill_3mo_cycle
      pce_inflation
      pce_inflation_trend
      pce_inflation_cycle
      unemployment
      unemployment_trend
      unemployment_cycle

      2019-07-01
      73.783132
      72.272855
      0.020682
      50.165309
      49.019365
      0.023108
      12.878614
      12.525037
      0.027838
      105.344295
      ...
      -0.041774
      0.019800
      0.014278
      0.005522
      0.039688
      0.018943
      0.020745
      0.036000
      0.052215
      -0.016215

      2019-10-01
      74.049483
      72.347117
      0.023258
      50.254751
      49.058110
      0.024100
      12.709561
      12.519089
      0.015100
      105.428767
      ...
      -0.039929
      0.015767
      0.014497
      0.001270
      0.040008
      0.015080
      0.024928
      0.036000
      0.053943
      -0.017943

      2020-01-01
      73.222987
      72.412037
      0.011137
      49.397299
      49.090095
      0.006238
      12.483490
      12.510535
      -0.002164
      103.941972
      ...
      -0.030253
      0.011100
      0.014671
      -0.003571
      0.018833
      0.011056
      0.007777
      0.038000
      0.055803
      -0.017803

      2020-04-01
      66.535376
      72.471686
      -0.085462
      44.643027
      49.118421
      -0.095536
      10.664014
      12.500674
      -0.158908
      90.210557
      ...
      0.085672
      0.001433
      0.014820
      -0.013386
      -0.096568
      0.006943
      -0.103511
      0.130667
      0.057744
      0.072922

      2020-07-01
      71.371904
      72.530653
      -0.016105
      48.552336
      49.146390
      -0.012161
      12.434136
      12.490785
      -0.004546
      97.421248
      ...
      0.105134
      0.001133
      0.014960
      -0.013826
      -0.016645
      0.002819
      -0.019464
      0.088000
      0.059703
      0.028297

5 rows × 30 columns

### Exercise: GDP and Inflation

Construct a plot of the cyclical components of GDP and inflation.

```python
# Construct plot
plt.plot(data['pce_inflation_cycle']*100,alpha=0.75,lw=3,label='Inflation')
plt.plot(data['gdp_cycle']*100,c='r',alpha=0.5,label='GDP')

plt.ylabel('Percent')
plt.grid()

# Place legend to right of figure. PROVIDED
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
```

### Exercise: GDP and the 3-Month T-Bill Rate

Construct a plot of the cyclical components of GDP and the 3-month T-bill rate.

```python
# Construct plot
plt.plot(data['t_bill_3mo_cycle']*100,alpha=0.75,lw=3,label='3mo T-bill')
plt.plot(data['gdp_cycle']*100,c='r',alpha=0.5,label='GDP')

plt.ylabel('Percent')
plt.grid()

# Place legend to right of figure. PROVIDED
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
```

### Correlations Between GDP, Inflation, and 3-Month T-Bill Rate

Compute the coefficients of corrrelation between GDP, inflation, and the 3-month T-bill rate.

```python
data[['gdp_cycle','pce_inflation_cycle','t_bill_3mo_cycle']].corr()
```

      gdp_cycle
      pce_inflation_cycle
      t_bill_3mo_cycle

      gdp_cycle
      1.000000
      0.64085
      0.415906

      pce_inflation_cycle
      0.640850
      1.00000
      0.266250

      t_bill_3mo_cycle
      0.415906
      0.26625
      1.000000

Strong (but not perfect!) correlations between GDP and inflation and GDP and the T-bill rate suggest link between nominal and real quantities over the business cycle that should be exaplined by business cycle theory.

## Aggregate Supply

Contemporary business cycle models for monetary policy emphasize the importance of an upward-sloping aggreagate supply (AS) curve: a positive relationship between the cyclical components of output and inflation. We will uncover evidence of an upward-sloping AS curve for the US by estimating the following equation:

\begin{align}
\hat{\pi}_t & = \beta \hat{y}_t + \epsilon_t,
\end{align}

where $\hat{\pi}_t$ is the cyclical component of inflation, $\hat{y}_t$ is the cyclical component of real GDP for the US, and $\epsilon_t$ is the resudual of the regression. We do not estimate a constant because the data are mean 0.

Note that estimation exercise is subject to an endogeneity bias because output and inflation are in fact determined simultaneously in equilbrium. However, while the results we obtain my not be robust, they will be suggestive and that's enough for now.

First, make a scatter plot of inflation against GDP to verify that there appears to be a relationship worth estimating

```python
# Create figure and axis
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

# Scatter plot with poitn size of 50, opacity 0.25, and label of 'actual'
ax.scatter(data['gdp_cycle']*100,data['pce_inflation_cycle']*100,s=50,alpha=0.25,label='actual')

# Set x-axis limits to [-10,6]
ax.set_xlim([-10,6])

# Axis labels and title
ax.set_title('Output and inflation for US')
ax.set_xlabel('GDP - % from trend')
ax.set_ylabel('inflation - % from trend')

ax.grid()
```

### OLS Regression

```python
# Import statsmodels.api
import statsmodels.api as sm
```

```python
# Define variable 'X' to be equal to the cyclical component of real GDP
X = data['gdp_cycle']

# Define variable 'Y' to be equal to the cyclical component of inflation
Y = data['pce_inflation_cycle']

# Initialze the OLS model
model = sm.OLS(Y,X)

# Fit the OLS model
results = model.fit()

# Display estimation results
print(results.summary())
```

                                     OLS Regression Results
    ========================================================================================
    Dep. Variable:     pce_inflation_cycle   R-squared (uncentered):                   0.411
    Model:                             OLS   Adj. R-squared (uncentered):              0.408
    Method:                  Least Squares   F-statistic:                              171.4
    Date:                 Thu, 25 Feb 2021   Prob (F-statistic):                    4.46e-30
    Time:                         16:20:29   Log-Likelihood:                          751.96
    No. Observations:                  247   AIC:                                     -1502.
    Df Residuals:                      246   BIC:                                     -1498.
    Df Model:                            1
    Covariance Type:             nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    gdp_cycle      0.6215      0.047     13.093      0.000       0.528       0.715
    ==============================================================================
    Omnibus:                       24.126   Durbin-Watson:                   0.680
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):               45.558
    Skew:                          -0.523   Prob(JB):                     1.28e-10
    Kurtosis:                       4.826   Cond. No.                         1.00
    ==============================================================================

    Notes:
    [1] R² is computed without centering (uncentered) since the model does not contain a constant.
    [2] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Apparently, a one percent increase in GDP relative to trend corresponds with a 0.6 percent increase in the inflation rate.

### Plot results

Construct a scatter plot of the underelying data with the predicted values from the OLS regression.

```python
# Create variable 'x_values' as an array from -10 to 6
x_values = np.linspace(-10,6,4)

# Create variable 'y_values' using estimated value for beta
y_values = results.params['gdp_cycle']*x_values

# Create figure and axis
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

# Scatter plot with poitn size of 50, opacity 0.25, and label of 'actual'
ax.scatter(data['gdp_cycle']*100,data['pce_inflation_cycle']*100,s=50,alpha=0.25,label='actual')

# Plot estimated relationship as a red line with label of 'fitted'
ax.plot(x_values,y_values,'r',label='fitted')

# Set x-axis limits to [-10,6]
ax.set_xlim([-10,6])

# Axis labels and title
ax.set_title('Output and inflation')
ax.set_xlabel('GDP - % from trend')
ax.set_ylabel('inflation - % from trend')

# Legend in the lower right corner with scatterpoints=1
ax.legend(scatterpoints=1,loc='lower right')

# Grid
ax.grid()
```

## Taylor's Monetary Policy Rule

Central banks in many countries set monetary policy with at least some autonomy. In the US, the Federal Reserve has broad discretion to decide how to manage its federal funds rate target in response to changes in the economy. The Federal Open Market Committee (FOMC), the monetary policymaking body within the Fed, comprises members of the Board of Governors and a rotating group of presidents from five of the regional Federal Reserve Banks.

The members of the FOMC can vote to set policy how they like and the composition of the FOMC changes regularly. So it might seem that modeling the decisionmaking of the FOMC would not be fruitful. But in fact, evidence shows that the FOMC does behave in a remarkably systematic way.

In his article "Discretion versus policy rules in practice" in the December 1993 issue of the *Carnegie-Rochester Conference Series on Public Policy*, John Taylor shows that monetary policy in the US could be approximated by a simple linear equation. (link to article: https://www.sciencedirect.com/science/article/pii/016722319390009L). His equation is:

\begin{align}
\text{Federal funds rate} & = \text{Inflation} + 0.5\cdot \text{Output gap} + 0.5\cdot(\text{Inflation - 2}) + 2,
\end{align}

where *inflation* is the percent change in the GDP deflator over the previous year and *output gap* is the percent deviation of GDP from a "target rate". Taylor assumed a target rate consistent with a constant growth GDP path of 2.2%. Note that the coefficients on inflation sum to 1.5.

You can see how well Taylor's rule fit the data as of 1993 in this figure from his paper:

Figure 1. Taylor's rule and the actual federal funds rate. Source: Taylor (1993)

Taylor doesn't estimate his equation precisely and it's remarkable that such a simple linear equation would even come close to matching policy. In this example, we will gather data to replicate Taylor's rule and estimate his rule using OLS.

### Federal Funds Rate

The federal funds are is the interest rate on overnight interbank loans. The FOMC sets a taget for the fed funds rate and implements monetary policy so that the actual (or *effective*) federal unds rate equals the FOMS's target. The effective federal funds rate has FRED series ID `FEDFUNDS` and is available here: https://fred.stlouisfed.org/series/FEDFUNDS. Notice that there are 62 lines of text *before* the line starting with "DATE".

```python
# Import federal funds rate data from FRED. Use arguments: sep='\s+',skiprows=62,index_col=0,parse_dates=True
fed_funds = pd.read_csv('https://fred.stlouisfed.org/data/FEDFUNDS.txt',sep='\s+',skiprows=62,index_col=0,parse_dates=True)

# Set fed_funds equal to 'VALUE' column of fed_funds
fed_funds = fed_funds['VALUE']

# Use .resample('QS').mean() method of 'fed_funds' to convert the fed funds data from monthdy to quarterly
fed_funds = fed_funds.resample('QS').mean()

# Plot the federal funds rate. PROVIDED
plt.plot(fed_funds)
```

# Inflation

Taylor computes the inflation rate as the percent change in the GDP deflator from one year prior:

\begin{align}
\text{Inflation}_t = \frac{\text{GDP deflator}_t - \text{GDP deflator}_{t-4}}{\text{GDP deflator}_{t-4}}\cdot 100
\end{align}

The GDP deflator has FRED series ID `GDPDEF` and is available here: https://fred.stlouisfed.org/series/GDPDEF. Notice that there are 15 lines of text *before* the line starting with "DATE".

```python
# Import GDP deflator data from FRED. PROVIDED
deflator = pd.read_csv('https://fred.stlouisfed.org/data/GDPDEF.txt',sep='\s+',skiprows=15,index_col=0,parse_dates=True)

# Set deflator equal to 'VALUE' column of deflator. PROVIDED
deflator = deflator['VALUE']

# Compute the inflation rate
inflation = (deflator - deflator.shift(4))/deflator.shift(4)*100

# Plot the inflation rate data. PROVIDED
plt.plot(inflation)
```

### Output gap

The output gap is measured as the percent difference of real GDP from the CBO's estimate of the potential real GDP:

\begin{align}
\text{Output gap} & = \left(\frac{\text{Real GDP}-\text{Real potential GDP}}{\text{Real potential GDP}}\right)\cdot 100
\end{align}

Real GDP has FRED series ID `GDPC1` and is available here: https://fred.stlouisfed.org/series/GDPC1. Notice that there are 17 lines of text *before* the line starting with "DATE".

```python
# Import real GDP data from FRED. PROVIDED
gdp_actual = pd.read_csv('https://fred.stlouisfed.org/data/GDPC1.txt',sep='\s+',skiprows=17,index_col=0,parse_dates=True)

# Set gdp_actual equal to 'VALUE' column of gdp_actual. PROVIDED
gdp_actual = gdp_actual['VALUE']
```

Potential real GDP has FRED series ID `GDPPOT` and is available here: https://fred.stlouisfed.org/series/GDPPOT. Notice that there are 12 lines of text *before* the line starting with "DATE".

```python
# Import potential real GDP data from FRED. PROVIDED
gdp_potential = pd.read_csv('https://fred.stlouisfed.org/data/GDPPOT.txt',sep='\s+',skiprows=12,index_col=0,parse_dates=True)

# Set gdp_potential equal to 'VALUE' column of gdp_potential. PROVIDED
gdp_potential = gdp_potential['VALUE']
```

```python
# Plot the actual and potential real GDP data with a legend. CELL PROVIDED
plt.plot(gdp_actual,label='actual')
plt.plot(gdp_potential,label='potential')
plt.legend(loc='lower right')
plt.title('Real GDP')
```

### Combine Data into a DataFrame

```python
# Create variable 'df' that is a DataFrame storing fed funds, inflation, actual and potential real GDP
df = pd.DataFrame({
    'fed_funds':fed_funds,
    'inflation':inflation,
    'gdp_actual':gdp_actual,
    'gdp_potential':gdp_potential
})

# Drop missing values from 'df'
df = df.dropna()

df
```

      fed_funds
      inflation
      gdp_actual
      gdp_potential

      DATE

      1954-07-01
      1.030000
      0.725639
      2682.601
      2721.074302

      1954-10-01
      0.986667
      0.813962
      2735.091
      2738.310469

      1955-01-01
      1.343333
      0.962662
      2813.212
      2755.016766

      1955-04-01
      1.500000
      1.291564
      2858.988
      2772.864269

      1955-07-01
      1.940000
      1.866209
      2897.598
      2790.460690

      ...
      ...
      ...
      ...
      ...

      2019-10-01
      1.643333
      1.628577
      19253.959
      19065.580000

      2020-01-01
      1.260000
      1.786868
      19010.848
      19153.980000

      2020-04-01
      0.060000
      0.602813
      17302.511
      19242.040000

      2020-07-01
      0.093333
      1.163235
      18596.521
      19327.310000

      2020-10-01
      0.090000
      1.279327
      18783.902
      19414.160000

266 rows × 4 columns

```python
# Construct an output gap column
df['output_gap'] = (df['gdp_actual'] - df['gdp_potential'])/df['gdp_potential']*100
df['output_gap'].plot()
```

Recall Taylor's rule:

\begin{align}
\text{Federal funds rate} & = \text{Inflation} + 0.5\cdot \text{Output gap} + 0.5\cdot(\text{Inflation - 2}) + 2
\end{align}

```python
# Construct a column called 'taylor' that implements Taylor's rule
df['taylor'] = df['inflation'] + 0.5*df['output_gap'] + 0.5*(df['inflation']-2)+2
```

```python
# Plot actual federal funds rate and the federal funds rate implied by Taylor's rule. CELL PROVIDED
plt.plot(df['fed_funds'],label='actual')
plt.plot(df['taylor'],label='Taylor')
plt.title('Federal Funds Rate')
plt.legend()
```

### OLS Model of the FOMC's Monetary Policy Rule

Following the format suggested by Taylor's rule, we'll estimate the following monetary policy rule for the US:

\begin{align}
\hat{i}_t & = \bar{\imath} + \phi_{\pi}\pi_t + \phi_{y} y_t + \epsilon_t
\end{align}

where $\pi_t$ is the percent change in the GDP deflator over the previous year and $y_t$ is the output gap measured as the percent difference of real GDP from the CBO's estimate of the potential real GDP. $\phi_{\pi}$ is the weight that the FOMC places on inflation in the rule and $\phi_{y}$ is the weight that the central bank places on the output gap. $\epsilon_t$ is the residual of the regression.

```python
# Create variable 'X' with columns inflation, output and a constant
X = sm.add_constant(df[['inflation','output_gap']])

# Create variable 'Y' equal to the federal funds rate
Y = df['fed_funds']

# Initialize OLS model
model = sm.OLS(Y,X)

# Fit OLS model
results = model.fit()

# Print regression results
print(results.summary2())
```

                     Results: Ordinary least squares
    ==================================================================
    Model:              OLS              Adj. R-squared:     0.574
    Dependent Variable: fed_funds        AIC:                1210.4505
    Date:               2021-02-25 16:20 BIC:                1221.2010
    No. Observations:   266              Log-Likelihood:     -602.23
    Df Model:           2                F-statistic:        179.7
    Df Residuals:       263              Prob (F-statistic): 6.26e-50
    R-squared:          0.577            Scale:              5.4820
    --------------------------------------------------------------------
                  Coef.    Std.Err.      t      P>|t|    [0.025   0.975]
    --------------------------------------------------------------------
    const         0.9364     0.2533    3.6967   0.0003   0.4376   1.4351
    inflation     1.2204     0.0646   18.8783   0.0000   1.0931   1.3477
    output_gap    0.1235     0.0593    2.0838   0.0381   0.0068   0.2403
    ------------------------------------------------------------------
    Omnibus:               2.584        Durbin-Watson:           0.127
    Prob(Omnibus):         0.275        Jarque-Bera (JB):        2.613
    Skew:                  0.237        Prob(JB):                0.271
    Kurtosis:              2.891        Condition No.:           7
    ==================================================================

The estimated constant in the regression is about 0.94 compared with 1 in Taylor's rule. The estimated coefficnets on inflation and the output gap are 1.22 and 0.12 compared with 1.5 and 0.5 in Taylor's rule. All three parameter estimates are remarkably close to Taylor's proposal. It's amazing that his rule still preforms will after almost 30 years.

Note, Taylor doens't argue that his rule is *optimal* in the sense that it is the best way for the FOMC to meet it's objectives. But there is evidence that it is actually very close to an optimal rule and therefore evidence that the FOMC does in fact tend to make good policy decisions.

```python
# Add column 'fed_funds_predicted' to df equal to predicted fed funds rate values from regression
df['fed_funds_predicted'] = results.predict()
```

```python
# Plot actual federal funds rate and the predicted federal funds. CELL PROVIDED
plt.plot(df['fed_funds'],label='actual')
plt.plot(df['fed_funds_predicted'],label='predicted')
plt.title('Federal Funds Rate')
plt.legend()
```


```python
from itertools import product

import numpy as np

from matplotlib import pyplot as plt, colors, lines, ticker
from matplotlib.patches import Rectangle

import sympy as smp
```

```python
plt.rc('text', usetex=True)
plt.rc('text.latex', preamble=r'\usepackage{amsmath}\usepackage{bm}')
plt.rcParams['figure.figsize'] = [3.25, 2.5]
plt.rcParams['figure.dpi'] = 150
plt.rcParams['font.family'] = 'Times New Roman'
plt.rcParams['font.size'] = 8
plt.rcParams['axes.titlesize'] = 'small'
plt.rcParams['axes.titlepad'] = 3
plt.rcParams['xtick.labelsize'] = 'x-small'
plt.rcParams['ytick.labelsize'] = plt.rcParams['xtick.labelsize']
plt.rcParams['legend.fontsize'] = 6
plt.rcParams['legend.handlelength'] = 1.5
plt.rcParams['lines.markersize'] = 4
plt.rcParams['lines.linewidth'] = 0.7
plt.rcParams['patch.linewidth'] = 0.7
plt.rcParams['hatch.linewidth'] = 0.7
plt.rcParams['axes.linewidth'] = 0.6
plt.rcParams['grid.linewidth'] = 0.6
plt.rcParams['xtick.major.width'] = 0.6
plt.rcParams['xtick.minor.width'] = 0.4
plt.rcParams['ytick.major.width'] = plt.rcParams['xtick.major.width']
plt.rcParams['ytick.minor.width'] = plt.rcParams['xtick.minor.width']
color_cycle = ['#003366', '#990000', '#669900']
#color_cycle = plt.rcParams['axes.prop_cycle'].by_key()['color']
```

```python
risk_versus_k = np.load('risk_versus_k.npz')
gammas = risk_versus_k['gammas']
ks = risk_versus_k['ks']
risks_ensemble = risk_versus_k['risks_ensemble']
risks_ensemble_eta = risk_versus_k['risks_ensemble_eta']
risks_ridge = risk_versus_k['risks_ridge']

fig, ax = plt.subplots()

for t, gamma in enumerate(gammas):
    plt.loglog(ks, risks_ensemble[t, :, :].mean(0), color=color_cycle[t], label=r'$\gamma=%g$' % gamma)
    plt.plot(ks, risks_ensemble_eta[t, :, :].mean(0), ':', color=color_cycle[t])
    plt.plot(ks, [risks_ridge[t]] * len(ks), '--', color=color_cycle[t])

plt.xlabel(r'$k$')
plt.ylabel(r'$R_{\alpha, \eta, k}^\mathrm{ens}$')
plt.legend()
plt.tight_layout()
plt.savefig('risk_versus_k.pdf')
plt.show()
```

```python
risk_versus_sigma = np.load('risk_versus_sigma.npz')
sigmas = risk_versus_sigma['sigmas']
ks = risk_versus_sigma['ks']
risks_ensemble = risk_versus_sigma['risks_ensemble']
risks_ridge = risk_versus_sigma['risks_ridge']

risk_versus_sigma_optimal_alpha = np.load('risk_versus_sigma_optimal_alpha.npz')
risks_ensemble_optimal_alpha = risk_versus_sigma_optimal_alpha['risks_ensemble']

plt.figure(figsize=(3.25, 4.5))
plt.subplot(2, 1, 1)

for j, k in enumerate(ks):
    plt.semilogx(sigmas, risks_ensemble[:, :, j].mean(1), color=color_cycle[j], label='$k=%d$' % k)
plt.plot(sigmas, risks_ridge, '--', color='black')

plt.ylim(0.48, 1.15)
plt.ylabel(r'$R_{\alpha, \eta, k}^\mathrm{ens}$')
plt.legend(loc='lower right')

plt.subplot(2, 1, 2)

for j, k in enumerate(ks):
    plt.semilogx(sigmas, risks_ensemble_optimal_alpha[:, :, j].mean(1), color=color_cycle[j], label='$k=%d$' % k)
plt.plot(sigmas, risks_ridge, '--', color='black')

plt.ylim(0.48, 1.15)
plt.xlabel(r'$\sigma$')
plt.ylabel(r'$R_{\alpha, \eta, k}^\mathrm{ens}$')
plt.legend(loc='lower right')

plt.tight_layout()
plt.savefig('risk_versus_sigma.pdf')
plt.show()
```

```python
risk_versus_k = np.load('risk_versus_sigma_optimal_alpha.npz')
sigmas = risk_versus_k['sigmas']
ks = risk_versus_k['ks']
risks_ensemble = risk_versus_k['risks_ensemble']
risks_ridge = risk_versus_k['risks_ridge']

fig, ax = plt.subplots()

for j, k in enumerate(ks):
    plt.semilogx(sigmas, risks_ensemble[:, :, j].mean(1), color=color_cycle[j], label='$k=%d$' % k)
plt.plot(sigmas, risks_ridge, '--', color='black')

plt.ylim(0.48, 1.15)
plt.xlabel(r'$\sigma$')
plt.ylabel(r'$R_{\alpha, \eta, k}^\mathrm{ens}$')
plt.legend(loc='lower right')

formatter = ticker.FuncFormatter(lambda x, _: '%.1f' % x)
ax.yaxis.set_major_formatter(formatter)
ax.yaxis.set_minor_formatter(formatter)

plt.tight_layout()
plt.savefig('risk_versus_sigma_optimal_alpha.pdf')
plt.show()
```

```python
fig, ax = plt.subplots()

n, p = 12, 15

rows = [(1, 2), (4, 2), (8, 1), (10, 2)]
cols = [(1, 1), (4, 1), (6, 3), (14, 1)]

int_edgecolor = color_cycle[0]
int_facecolor = '#bbccff'
int_hatch = 'XXX'
col_edgecolor = color_cycle[2]
col_facecolor = '#eeffbb'
col_hatch = '\\\\\\'
row_edgecolor = color_cycle[1]
row_facecolor = '#ffccbb'
row_hatch = '///'

for y, height in rows:
    ax.add_patch(Rectangle((0, y), p, height, edgecolor=row_edgecolor, facecolor=row_facecolor, hatch=row_hatch))
    ax.add_patch(Rectangle((-3, y), 1, height, edgecolor=row_edgecolor, facecolor=row_facecolor, hatch=row_hatch))

for x, width in cols:
    ax.add_patch(Rectangle((x, 0), width, n, edgecolor=col_edgecolor, facecolor=col_facecolor, hatch=col_hatch))
    ax.add_patch(Rectangle((p + 1, (n + p) / 2 - x - width), 1, width, edgecolor=col_edgecolor, facecolor=col_facecolor, hatch=col_hatch))

for (x, width), (y, height) in product(cols, rows):
    ax.add_patch(Rectangle((x, y), width, height, edgecolor=int_edgecolor, facecolor=int_facecolor, hatch=int_hatch))

ax.add_patch(Rectangle((0, 0), p, n, fill=False, linewidth=1.0))
ax.add_patch(Rectangle((-3, 0), 1, n, fill=False, linewidth=1.0))
ax.add_patch(Rectangle((p+1, -(p - n) / 2), 1, p, fill=False, linewidth=1.0))

plt.text(-2.5, n + 1, r'${\mathbf{y}}$', fontsize=10, horizontalalignment='center')
plt.text(-1, n / 2, r'$\approx$', fontsize=10, horizontalalignment='center', verticalalignment='center')
plt.text(p / 2, n + 1, r'${\mathbf{X}}$', fontsize=10, horizontalalignment='center')
plt.text(p + 1.5, (n + p) / 2 + 1, r'${\widehat{\bm\beta}^{(i)}}$', fontsize=10, horizontalalignment='center')

plt.xlim(-3.1, p+2.1)
plt.ylim(-(p - n) / 2 - 0.1, (n + p) / 2 + 0.1)

plt.axis('off')
plt.tight_layout()

plt.savefig('subsampling.pdf')

plt.show()
```

```python
def ensemble_risk_ij(intersection_size, complement_intersection_size, gamma, sigma, eta=1):

    return complement_intersection_size * (1 + intersection_size * gamma / (eta - intersection_size * gamma)) + intersection_size * gamma * sigma**2 / (eta - intersection_size * gamma)

def ensemble_risk(alpha, gamma, sigma, k=None, eta=1):

    if k is None:
        return ensemble_risk_ij(alpha**2, (1 - alpha)**2, gamma, sigma, eta=1)
    else:
        return (1 - 1/k) * ensemble_risk_ij(alpha**2, (1 - alpha)**2, gamma, sigma, eta=1) + 1/k * ensemble_risk_ij(alpha, 1-alpha, gamma, sigma, eta=eta)

def ensemble_optimal_alpha(gamma, sigma, k=None, eta=None):

    if k is None:

        a = 1
        b = - (1 / gamma + 1 + sigma**2)
        c = 1 / gamma
        alpha = (-b - np.sqrt(b**2 - 4 * a * c)) / (2 * a)

        return alpha

    else:
        if eta is None:
            eta = 1

        alpha = smp.symbols('a', real=True)
        r = ensemble_risk(alpha, gamma, sigma, k, eta)
        drdalpha = smp.diff(r, alpha)
        numerator, _ = smp.fraction(smp.simplify(drdalpha))
        coeffs = smp.Poly(numerator).coeffs()
        roots = [np.abs(r) for r in np.roots(coeffs) if np.angle(r) < 1e-2 and 0 <= np.abs(r) <= 1/gamma]
        roots += [0, 1/gamma - 1e-6]

        return min(roots, key=lambda x: ensemble_risk(x, gamma, sigma, k, eta))

def ridge_optimal_risk(gamma, sigma):

    dobriban = 1 / 2 * (1 + (gamma - 1) / gamma / sigma**2 + np.sqrt((1 - (gamma - 1) / gamma / sigma**2)**2 + 4 / sigma**2))
    return (dobriban - 1) * sigma**2

def mu_scaled_risk(alpha, gamma, sigma, mu=1):

    return mu**2 * ensemble_risk(alpha, gamma, sigma) + (1 - mu)**2 + 2 * mu * (1 - mu) * (1 - alpha)

def optimal_mu(alpha, gamma, sigma):

    mu = alpha / (ensemble_risk(alpha, gamma, sigma) + 2*alpha - 1)
    return mu

def optimal_mu_scaled_risk(alpha, gamma, sigma):

    return mu_scaled_risk(alpha, gamma, sigma, optimal_mu(alpha, gamma, sigma))

sigmas = np.logspace(-1, 1)
gamma = 0.5

risks_ridge = [ridge_optimal_risk(gamma, sigma) for sigma in sigmas]
risks_alpha_half = []
risks_alpha_half_mu = []
risks_alpha_double = []
risks_alpha_double_mu = []
risks_half = []
risks_half_mu = []

for sigma in sigmas:

    alpha = ensemble_optimal_alpha(gamma, sigma)
    risks_alpha_half.append(ensemble_risk(alpha / 2, gamma, sigma))
    risks_alpha_half_mu.append(optimal_mu_scaled_risk(alpha / 2, gamma, sigma))
    risks_alpha_double.append(ensemble_risk(min(1, 2*alpha), gamma, sigma))
    risks_alpha_double_mu.append(optimal_mu_scaled_risk(min(1, 2*alpha), gamma, sigma))
    risks_half.append(ensemble_risk(1 / 2, gamma, sigma))
    risks_half_mu.append(optimal_mu_scaled_risk(1 / 2, gamma, sigma))

fig, ax = plt.subplots()

plt.semilogx(sigmas, risks_alpha_half, color=color_cycle[0], label=r'$\alpha=\alpha_*/2$')
plt.semilogx(sigmas, risks_alpha_half_mu, ':', color=color_cycle[0])
#plt.semilogx(sigmas, risks_alpha_double, color=color_cycle[1], label=r'$\alpha=\min\{1, 2\alpha_*\}$')
#plt.semilogx(sigmas, risks_alpha_double_mu, ':', color=color_cycle[1])
plt.semilogx(sigmas, risks_half, color=color_cycle[1], label=r'$\alpha=1/2$')
plt.semilogx(sigmas, risks_half_mu, ':', color=color_cycle[1])
plt.plot(sigmas, risks_ridge, '--', color='black')

plt.ylim(0, 1.15)
plt.xlabel(r'$\sigma$')
plt.ylabel(r'$R_{\alpha,\mu}^\mathrm{ens}$')
plt.legend(loc='lower right')

formatter = ticker.FuncFormatter(lambda x, _: '%.1f' % x)
ax.yaxis.set_major_formatter(formatter)
ax.yaxis.set_minor_formatter(formatter)

plt.tight_layout()
plt.savefig('risk_versus_sigma_tuned_mu.pdf')
plt.show()
```

```python

```


```python
%matplotlib inline
DEFAULT_FIGSIZE = (12, 8)

import itertools
import sys

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style('darkgrid')

sys.path.append('..')
from antlia.record import load_records
from antlia import plot_braking as braking

%load_ext autoreload
%autoreload 2

import matplotlib as mpl
mpl.rcParams['figure.figsize'] = DEFAULT_FIGSIZE
```

```python
records = load_records(sync=True)
```

    Unable to determine range 2
    Unable to determine range 1

```python
import IPython.display

def animate_braking(rid, tid, xlim, ylim, speedup=1, **kwargs):
    record = records[rid]
    trial = record.trial[tid]

    m, _, _, _ = braking.get_metrics(trial.data)

    t0 = m['braking starttime']
    t1 = m['braking endtime']

    frames = record.lidar.frame(lambda t: (t >= t0 - 1) & (t < t1 + 1))

    c = sns.color_palette('Paired', 10)[1::2]
    colors = [c[2]
              if (t > t0) and (t < t1) else c[1]
              for t in frames.time]

    return frames.animate(xlim, ylim, speedup, colors, **kwargs)

def display_animation(animation):
    plt.close(animation._fig)
    IPython.display.display(IPython.display.HTML(ani.to_html5_video()))

for rid, tid in [[0, 6]]:
#for rid, tid in itertools.product(range(3, 4), range(1)):
    record = records[rid]
    trial = record.trial[tid]

    ani = animate_braking(rid, tid,
                          xlim=(0, 50),
                          ylim=(0, 2),
                          speedup=0.5,
                          plot_kwargs={'marker': 'o'})
    ani._fig.get_axes()[0].set_title('rider {} trial {:02d}'.format(rid, tid))
    display_animation(ani)
```

```python
## calculate forward length
# We make some simplifications in the bicycle geometry
# (namely ignoring changes in roll and pitch, and mostly
# assuming yaw to be zero) as they will be small as we
# are near the nominal orientation.

bb_to_front_hub = 0.655 # m
front_wheel_radius = 0.359 # m
head_tube_angle = (90 - 72) * np.pi/180 # rad TODO: measure this

import sympy
import sympy.physics.vector as sv

yaw, steer = sympy.symbols('yaw steer')
N = sv.ReferenceFrame('N')
A = N.orientnew('A', 'axis', [yaw, N.z])
B = A.orientnew('B', 'axis', [head_tube_angle, A.y])
C = B.orientnew('C', 'axis', [steer, B.z])

# vector pointing from front hub center to most forward
# point on front wheel
n = sv.cross(C.y, N.z).normalize()

o = sv.Point('o')
p = o.locatenew('p', bb_to_front_hub*A.x + front_wheel_radius*n)

d = sv.dot(p.pos_from(o), N.x)

def forward_length(steer_angle, yaw_angle = 0):
    # distance from bottom bracket to front hub,
    # projected onto rear frame body fixed x-axis
    return d.subs({yaw: yaw_angle, steer: steer_angle})

print(forward_length(0))
print(forward_length(15*np.pi/180))
print(forward_length(-15*np.pi/180))
print(forward_length(30*np.pi/180))
print(forward_length(-30*np.pi/180))
```

    1.01400000000000
    1.00288180754931
    1.00288180754931
    0.969681991356316
    0.969681991356316

```python
plt.close('all')
for rid, tid in [[0, 2]]:
    record = records[rid]
    trial = record.trial[tid]
    try:
        m, _, _, _ = braking.get_metrics(trial.data)
    except TypeError:
        continue
    print('rider {}, trial {}'.format(rid, tid))

    index1 = record.lidar.frame_index(m['braking starttime'][0])

    xlim = (0, 50)
    ylim = (0.5, 1.5)
    fig, ax = plt.subplots()

    x, y = record.lidar[index1 - 1].cartesian(xlim=xlim, ylim=ylim)
    ax.plot(x.compressed(), y.compressed(), linestyle=' ', marker='.')
    x, y = record.lidar[index1].cartesian(xlim=xlim, ylim=ylim)
    ax.plot(x.compressed(), y.compressed(), linestyle=' ', marker='.')

    ax.set_xlabel('position [m]')
    ax.set_ylabel('position [m]')
    ax.set_xlim((0, 50))
    ax.set_ylim(ylim)

    rider_x = x[np.where((x > 10) & (x < 48))]
    print('average rider x: {}'.format(rider_x.mean()))

    # from bottom bracket to front wheel center
    # projected on inertia x-axis
    dtc1 = 50 - rider_x.mean() - 0.580
    print('trial dtc, fl = .580: {}'.format(dtc1))

    steer_angle = np.interp(m['braking starttime'][0],
                            trial.data.time,
                            trial.data['steer angle'])
    dtc2 = 50 - rider_x.mean() - forward_length(steer_angle)
    print('trial dtc, fl func: {}'.format(dtc2))
plt.show()
```

```python
def calculate_dtc(rid, tid):
    record = records[rid]
    trial = record.trial[tid]
    m, _, _, _ = braking.get_metrics(trial.data)

    index = record.lidar.frame_index(m['braking starttime'][0])
    index = [-1 + index, index]

    xlim = (0, 50)
    ylim = (0.5, 1.5)

    dtc = []
    for i in index:
        x, _ = record.lidar[[i]].cartesian(xlim=xlim, ylim=ylim)
        x = x.compressed()
        rider_x = x[np.where((x > 10) & (x < 48))]

        steer_angle = np.interp(record.lidar[i].time[0],
                                trial.data.time,
                                trial.data['steer angle'])

        dtc.append(50 - rider_x.mean() - forward_length(steer_angle))

    dtc = np.array(dtc).astype(np.float64)
    t = record.lidar[np.array(index)].time.flatten()
    return np.interp(m['braking starttime'][0], t, dtc), m

calculate_dtc(0, 2)
```

    (6.0975533337432655,
     array([ (-1.72348435,  715.84898323, -0.74410543,   2.76832297e-44,  0.0994677,  6.07819461,  1.952244,  8.70374047,  411.786928,  413.739172, 55, [2212, 2456], 0, 0, 0)],
           dtype=[('linregress slope', '<f8'), ('linregress intercept', '<f8'), ('linregress r-value', '<f8'), ('linregress p-value', '<f8'), ('linregress stderr', '<f8'), ('starting velocity', '<f8'), ('braking duration', '<f8'), ('braking distance', '<f8'), ('braking starttime', '<f8'), ('braking endtime', '<f8'), ('window size', '<i8'), ('braking range', '<i8', (2,)), ('lockup ranges', '<i8'), ('rider id', '<i8'), ('trial id', '<i8')]))

```python
import pandas as pd

def get_dataframe(records):
    """Define function to get metrics for lidar data.
    """
    metrics = []
    dtc = []
    for rid, tid in itertools.product(range(4), range(15)):
        try:
            x, m = calculate_dtc(rid, tid)
        except TypeError:
            print('skipping {}, {}'.format(rid, tid))
            continue

        dtc.append(x)

        m['rider id'] = rid
        m['trial id'] = tid
        metrics.append(m)

    metrics = np.concatenate(metrics)
    # exclude non-scalar fields
    names = [dtype[0]
             for dtype in metrics.dtype.fields.items()
             if dtype[1][0].shape == ()]

    df = pd.DataFrame(metrics[names])
    df['distance-to-collision'] = dtc

    f = lambda row: row['distance-to-collision']/row['starting velocity']
    df['time-to-collision'] = df.apply(f, axis=1)
    return df

df = get_dataframe(records)
df.to_pickle('braking_ttc.p.gz')
# DTC/TTC calculations will be 'nan' if not detected by lidar
# lidar only has a guaranteed detection range of 30 m
df
```

    skipping 0, 0
    skipping 0, 1

    /Users/oliver/miniconda3/envs/dev/lib/python3.5/site-packages/ipykernel_launcher.py:22: RuntimeWarning: Mean of empty slice.
    /Users/oliver/miniconda3/envs/dev/lib/python3.5/site-packages/numpy/core/_methods.py:80: RuntimeWarning: invalid value encountered in double_scalars
      ret = ret.dtype.type(ret / rcount)

      linregress stderr
      braking starttime
      braking endtime
      linregress p-value
      linregress intercept
      linregress slope
      braking duration
      trial id
      window size
      linregress r-value
      lockup ranges
      starting velocity
      braking distance
      rider id
      distance-to-collision
      time-to-collision

      0
      0.099468
      411.786928
      413.739172
      2.768323e-44
      715.848983
      -1.723484
      1.952244
      2
      55
      -0.744105
      0
      6.078195
      8.703740
      0
      6.097553
      1.003185

      1
      0.514137
      493.557209
      494.157285
      8.387193e-03
      691.587816
      -1.393157
      0.600076
      3
      55
      -0.302307
      0
      3.937731
      2.140694
      0
      NaN
      NaN

      2
      0.118824
      571.154965
      572.667154
      5.695965e-23
      773.491082
      -1.345448
      1.512189
      4
      55
      -0.637766
      0
      4.857473
      6.070381
      0
      7.685615
      1.582225

      3
      0.213894
      677.698360
      678.858507
      2.569676e-21
      1630.390552
      -2.397210
      1.160147
      5
      55
      -0.683830
      0
      5.533575
      5.122003
      0
      5.334500
      0.964024

      4
      0.130077
      766.029467
      767.317630
      4.054087e-11
      710.990535
      -0.922756
      1.288163
      6
      55
      -0.490318
      0
      4.031114
      4.557585
      0
      NaN
      NaN

      5
      0.199042
      846.439577
      847.655731
      1.672730e-15
      1504.730026
      -1.771827
      1.216154
      7
      55
      -0.587936
      0
      4.699092
      4.753165
      0
      5.562428
      1.183724

      6
      0.119038
      910.143588
      911.839800
      1.107465e-34
      1618.459859
      -1.771589
      1.696212
      8
      55
      -0.716459
      0
      5.855901
      7.729463
      0
      8.116734
      1.386078

      7
      0.432361
      1020.986526
      1021.778623
      9.763750e-05
      1798.833210
      -1.757588
      0.792097
      9
      55
      -0.381527
      0
      4.146971
      2.901759
      0
      4.766233
      1.149329

      8
      0.272392
      1090.427257
      1091.723418
      7.108176e-09
      1821.773991
      -1.665790
      1.296161
      10
      55
      -0.435266
      0
      4.927042
      5.537081
      0
      5.782857
      1.173697

      9
      0.131949
      1152.203024
      1154.227278
      4.058286e-25
      1766.450561
      -1.527542
      2.024254
      11
      55
      -0.589990
      0
      6.070815
      9.849418
      0
      7.363308
      1.212903

      10
      0.590566
      1227.340468
      1228.124568
      9.067728e-06
      3403.827194
      -2.769317
      0.784100
      12
      55
      -0.431701
      0
      4.393109
      3.015768
      0
      3.324807
      0.756823

      11
      0.227213
      1296.417151
      1297.841334
      1.777402e-07
      1606.978394
      -1.235655
      1.424183
      13
      55
      -0.379297
      0
      4.911338
      5.944619
      0
      6.410373
      1.305219

      12
      0.221304
      1358.652981
      1360.045157
      1.134258e-32
      4476.042180
      -3.289115
      1.392176
      14
      55
      -0.749813
      0
      7.264251
      6.941725
      0
      6.967701
      0.959177

      13
      0.372370
      149.144789
      150.308935
      2.130479e-03
      178.284159
      -1.165539
      1.164146
      0
      55
      -0.256599
      0
      4.468970
      4.390792
      1
      5.851486
      1.309359

      14
      0.315063
      267.399657
      268.519797
      2.374467e-11
      618.441244
      -2.291882
      1.120140
      1
      55
      -0.526470
      0
      5.332766
      4.826932
      1
      5.867699
      1.100310

      15
      0.247502
      345.050421
      346.462602
      7.241352e-16
      768.100148
      -2.208759
      1.412181
      2
      55
      -0.565956
      0
      5.820894
      6.224143
      1
      6.047911
      1.039000

      16
      0.521148
      431.805332
      432.653438
      9.615722e-03
      597.924467
      -1.374868
      0.848106
      3
      55
      -0.250448
      0
      4.084662
      3.109326
      1
      3.841801
      0.940543

      17
      0.105052
      511.564360
      513.884651
      7.957886e-25
      614.627484
      -1.189345
      2.320291
      4
      55
      -0.554965
      0
      6.257176
      11.186423
      1
      10.939693
      1.748343

      18
      0.267451
      618.606801
      619.966989
      5.918754e-09
      1020.617117
      -1.640769
      1.360188
      5
      55
      -0.427813
      0
      5.095139
      6.135208
      1
      5.991892
      1.176002

      19
      0.303229
      701.457234
      702.481363
      3.583644e-04
      784.351425
      -1.112454
      1.024129
      6
      55
      -0.310662
      0
      3.922852
      3.526178
      1
      4.685507
      1.194414

      20
      0.159852
      780.423163
      782.223391
      4.134105e-18
      1187.822777
      -1.514316
      1.800228
      7
      55
      -0.535680
      0
      5.694162
      8.375723
      1
      7.702101
      1.352631

      21
      0.181183
      904.446757
      906.014955
      2.957331e-11
      1161.410129
      -1.278428
      1.568198
      8
      55
      -0.451911
      0
      4.762297
      6.489323
      1
      7.379062
      1.549476

      22
      0.451207
      1041.463986
      1042.336094
      2.854776e-04
      1767.360226
      -1.692859
      0.872108
      9
      55
      -0.340969
      0
      3.978648
      3.113868
      1
      3.719035
      0.934748

      23
      0.131998
      1120.657943
      1122.730203
      1.139927e-18
      1417.486938
      -1.259601
      2.072260
      10
      55
      -0.511491
      0
      5.860524
      9.532710
      1
      8.631915
      1.472891

      24
      0.178199
      1198.235696
      1199.795896
      3.484063e-15
      1833.545261
      -1.525589
      1.560200
      11
      55
      -0.524629
      0
      5.052453
      6.771815
      1
      6.786150
      1.343140

      25
      0.426958
      1296.765087
      1297.569187
      4.772477e-05
      2360.607031
      -1.817068
      0.804100
      12
      55
      -0.394954
      0
      4.095955
      2.867340
      1
      4.115599
      1.004796

      26
      0.080633
      1368.302085
      1370.866405
      1.159383e-30
      1422.442075
      -1.035140
      2.564320
      13
      55
      -0.584857
      0
      5.688769
      12.129617
      1
      11.712218
      2.058832

      27
      0.213803
      1450.108366
      1451.532548
      2.380192e-09
      1956.712171
      -1.345778
      1.424182
      14
      55
      -0.428662
      0
      4.920177
      6.023610
      1
      6.134057
      1.246715

      28
      0.192734
      67.937108
      69.477294
      2.184056e-12
      104.074664
      -1.451001
      1.540186
      0
      55
      -0.484273
      0
      5.250042
      6.746693
      2
      7.678202
      1.462503

      29
      0.459561
      159.108565
      159.932668
      6.778942e-03
      206.168884
      -1.270426
      0.824103
      1
      55
      -0.265221
      0
      4.140467
      2.892420
      2
      NaN
      NaN

      30
      0.186733
      240.858843
      242.387040
      3.175743e-20
      472.449027
      -1.935900
      1.528197
      2
      55
      -0.602094
      0
      5.756837
      7.169094
      2
      6.853391
      1.190479

      31
      0.184803
      327.793768
      329.481980
      6.320371e-17
      558.665530
      -1.685668
      1.688212
      3
      55
      -0.533608
      0
      5.759884
      7.919973
      2
      7.918953
      1.374846

      32
      0.117980
      388.025342
      390.185616
      2.197673e-24
      522.255291
      -1.330083
      2.160274
      4
      55
      -0.567176
      0
      6.033922
      10.180508
      2
      11.340628
      1.879479

      33
      0.460033
      460.818498
      461.354565
      1.989741e-04
      839.788339
      -1.814344
      0.536067
      5
      55
      -0.439426
      0
      3.632680
      1.725589
      2
      4.638613
      1.276912

      34
      0.123615
      523.414369
      525.470628
      2.590222e-21
      678.380714
      -1.284263
      2.056259
      6
      55
      -0.545340
      0
      5.959406
      9.990297
      2
      9.548608
      1.602275

      35
      0.283712
      595.659453
      596.819600
      1.985175e-08
      1010.307303
      -1.687670
      1.160147
      7
      55
      -0.445380
      0
      4.712639
      4.700794
      2
      5.249717
      1.113965

      36
      0.237073
      666.012298
      667.372468
      9.150321e-09
      960.560949
      -1.434223
      1.360170
      8
      55
      -0.422943
      0
      5.274323
      5.951734
      2
      6.443613
      1.221695

      37
      0.143553
      725.987840
      728.004092
      7.698399e-21
      1075.893731
      -1.472820
      2.016252
      9
      55
      -0.544331
      0
      6.195570
      10.403022
      2
      9.697881
      1.565293

      38
      0.160362
      792.288173
      793.992385
      7.577289e-20
      1290.271951
      -1.621243
      1.704212
      10
      55
      -0.571253
      0
      5.585574
      7.496743
      2
      7.334811
      1.313170

      39
      0.368513
      861.340855
      862.108949
      2.970089e-04
      1196.854970
      -1.384924
      0.768094
      11
      55
      -0.361420
      0
      3.821096
      2.635392
      2
      4.625546
      1.210529

      40
      0.621001
      937.198392
      937.798466
      2.872655e-04
      2221.722279
      -2.366124
      0.600074
      12
      55
      -0.407285
      0
      4.009770
      2.091206
      2
      4.683763
      1.168088

      41
      0.090844
      1002.855650
      1005.279956
      2.040731e-39
      1398.661282
      -1.388128
      2.424306
      13
      55
      -0.660943
      0
      6.113808
      11.847343
      2
      11.258227
      1.841443

      42
      0.104427
      1063.131230
      1065.155483
      4.680409e-24
      1254.729751
      -1.175198
      2.024253
      14
      55
      -0.579102
      0
      4.894521
      8.402625
      2
      8.338216
      1.703581

      43
      0.088299
      200.234093
      202.194340
      1.588852e-47
      329.475777
      -1.613547
      1.960247
      0
      55
      -0.760791
      0
      6.213133
      9.423451
      3
      10.671649
      1.717595

      44
      0.377837
      290.552450
      291.232535
      2.714446e-06
      557.072111
      -1.903188
      0.680085
      1
      55
      -0.483859
      0
      3.848757
      2.345546
      3
      3.392364
      0.881418

      45
      0.162949
      372.902804
      374.262975
      2.418834e-11
      439.577530
      -1.166514
      1.360171
      2
      55
      -0.483472
      0
      4.516728
      5.152085
      3
      6.312491
      1.397580

      46
      0.128711
      446.240024
      447.784214
      9.314305e-43
      1035.378559
      -2.305830
      1.544190
      3
      55
      -0.791776
      0
      5.800934
      7.172419
      3
      8.631196
      1.487898

      47
      0.192970
      523.161695
      524.369850
      3.227420e-12
      771.154586
      -1.464629
      1.208155
      4
      55
      -0.528038
      0
      4.810463
      4.871084
      3
      5.296799
      1.101100

      48
      0.411237
      606.516175
      607.252267
      3.406169e-02
      540.559473
      -0.885047
      0.736092
      5
      55
      -0.221236
      0
      3.892019
      2.531203
      3
      3.507543
      0.901214

      49
      0.645426
      701.688144
      702.288220
      2.535772e-06
      2316.762848
      -3.295095
      0.600076
      6
      55
      -0.512936
      0
      4.479492
      2.187171
      3
      2.614892
      0.583747

      50
      0.135457
      775.569433
      777.009615
      1.436463e-23
      1225.343357
      -1.573038
      1.440182
      7
      55
      -0.656544
      0
      5.242828
      6.064177
      3
      6.605595
      1.259930

      51
      0.241547
      849.714755
      850.610868
      1.089151e-13
      1747.392063
      -2.050825
      0.896113
      8
      55
      -0.629199
      0
      4.720554
      3.456587
      3
      5.106999
      1.081864

      52
      0.289432
      958.456427
      959.152515
      2.809079e-06
      1396.529880
      -1.452917
      0.696088
      9
      55
      -0.478195
      0
      3.818448
      2.412781
      3
      3.940970
      1.032087

      53
      0.159471
      1035.458110
      1036.570250
      2.661685e-17
      1609.419085
      -1.549688
      1.112140
      10
      55
      -0.638778
      0
      4.617103
      4.359397
      3
      5.472000
      1.185159

      54
      0.154867
      1107.235135
      1108.499294
      1.926171e-32
      2602.650230
      -2.345220
      1.264159
      11
      55
      -0.771458
      0
      5.681426
      5.636111
      3
      6.804707
      1.197711

      55
      0.242277
      1189.645494
      1190.477600
      2.394909e-17
      2956.201500
      -2.480941
      0.832106
      12
      55
      -0.711977
      0
      4.459765
      3.102752
      3
      4.550239
      1.020287

      56
      0.382559
      1273.521044
      1274.121119
      3.178336e-04
      1845.580331
      -1.446112
      0.600075
      13
      55
      -0.404598
      0
      3.837834
      2.095830
      3
      2.564490
      0.668213

      57
      0.116723
      1342.429710
      1343.949900
      3.627982e-35
      2419.708028
      -1.798177
      1.520190
      14
      55
      -0.746988
      0
      5.966686
      6.712599
      3
      7.548745
      1.265149

```python
plt.close('all')
fig, ax = plt.subplots()
sns.swarmplot(x='rider id', y='starting velocity', data=df, ax=ax)
ax.set_title('rider vs starting velocity (braking event)')

fig, ax = plt.subplots()
sns.swarmplot(x='rider id', y='linregress slope', data=df, ax=ax)
ax.set_title('rider vs acceleration (braking event)')

fig, ax = plt.subplots()
sns.swarmplot(x='rider id', y='distance-to-collision', data=df, ax=ax)
ax.set_title('rider vs DTC')

fig, ax = plt.subplots()
sns.swarmplot(x='rider id', y='time-to-collision', data=df, ax=ax)
ax.set_title('rider vs TTC')
plt.show()
```

```python
from antlia.plotdf import plotjoint

colors = sns.color_palette()

plt.close('all')
g = plotjoint('starting velocity', 'distance-to-collision',
              df[np.isfinite(df['time-to-collision'])],
              ('rider id', colors))
g.fig.set_size_inches(DEFAULT_FIGSIZE)
g.set_axis_labels('velocity [m/s]', 'distance-to-collision [m]')

g = plotjoint('starting velocity', 'time-to-collision',
              df[np.isfinite(df['time-to-collision'])],
              ('rider id', colors))
g.fig.set_size_inches(DEFAULT_FIGSIZE)
g.set_axis_labels('velocity [m/s]', 'time-to-collision [s]')
plt.show()
```