# Valor Presente y Sensibilidad OIS

**Definición:** una curva cupón cero es una colección ordenada de plazos y tasas de interés donde cada una de las tasas es una  tasa adecuada para traer a valor presente un flujo de caja al plazo que corresponde a la tasa.

Consideremos un instrumento financiero cuyo valor depende de una curva cupón cero. Por ejemplo, un swap de tasa de interés. Matemáticamente, esta dependencia se expresa como:

$$
V = V\left(z_1,\ldots,z_n;\alpha\right)
$$

donde $z_1,\ldots,z_N$ son los valores de las tasas de la curva y $\alpha$ representa un vector de parámetros adicionales (como la tasa de cupón en un bono o un tipo de cambio en un swap de monedas).

**Definición:** la cantidad $\Delta_i$ de $V$ respecto a $z_i$ se define como:

$$
\Delta_i=\frac{\partial V}{\partial z_i}
$$

**Definición:** la sensibilidad de $V$ respecto a un cambio $\delta z_i$ (positivo o negativo) en la tasa $z_i$ se define como:

$$
S\left(V,z_i,\delta_i\right)=V\left(z_1,...,z_i+\delta_i,\ldots,z_n\right)-V\left(z_1,\ldots,z_i,\ldots,z_n\right)
$$

Cuando se busca conocer la sensibilidad a un cambio $\delta z_i$  en la tasa $z_i$, donde $\delta z_i$ representa una magnitud sin signo, se utiliza la siguiente definición:

**Definición:** la sensibilidad de $V$ respecto a un cambio de magnitud $\delta z_i$ en la tasa $z_i$ se define como:

$$
\overline{S}\left(V,z_i,\delta_i\right)=\frac{V\left(z_1,\ldots ,z_i+\delta_i,\ldots,z_n\right)-V\left(z_1,\ldots,z_i-\delta_i,...,z_n\right)}{2}
$$

En ocasiones, cuando se busca conocer la sensibilidad a un movimiento de magnitud pequeña, resulta conveniente realizar la siguiente aproximación:

$$
\overline{S}\left(V,z_i,\delta_i\right)\approx\Delta_i \cdot\delta_i
$$

este es el caso, cuando se busca conocer esta sensibilidad para un elevado número de operaciones. Es importante considerar que se puede asegurar que esta aproximación es válida cuando la función $V$ satisface los requerimientos de regularidad necesarios para aplicar el [Teorema de Taylor](https://en.wikipedia.org/wiki/Taylor%27s_theorem).

## Configuración

### Librerías

```python
from finrisk import QC_Financial_3 as Qcf
from functools import partial
from enum import Enum
import pandas as pd
import my_functions as my
```

### Variables Globales

```python
class BusCal(Enum):
    NY = 1
    SCL = 2
```

```python
def get_cal(code: BusCal) -> Qcf.BusinessCalendar:
    """
    Retorna un calendario NY o SCL.
    """
    if code == BusCal.NY:
        cal = Qcf.BusinessCalendar(Qcf.QCDate(1, 1, 2020), 20)
        for agno in range(2020, 2071):
            f = Qcf.QCDate(12, 10, agno)
            if f.week_day() == Qcf.WeekDay.SAT:
                cal.add_holiday(Qcf.QCDate(14, 10, agno))
            elif f.week_day() == Qcf.WeekDay.SUN:
                cal.add_holiday(Qcf.QCDate(13, 10, agno))
            elif f.week_day() == Qcf.WeekDay.MON:
                cal.add_holiday(Qcf.QCDate(12, 10, agno))
            elif f.week_day() == Qcf.WeekDay.TUE:
                cal.add_holiday(Qcf.QCDate(11, 10, agno))
            elif f.week_day() == Qcf.WeekDay.WED:
                cal.add_holiday(Qcf.QCDate(10, 10, agno))
            elif f.week_day() == Qcf.WeekDay.THU:
                cal.add_holiday(Qcf.QCDate(9, 10, agno))
            else:
                cal.add_holiday(Qcf.QCDate(8, 10, agno))
        cal.add_holiday(Qcf.QCDate(15, 2, 2021))

    if code == BusCal.SCL:
        cal = Qcf.BusinessCalendar(Qcf.QCDate(1, 1, 2020), 20)
        for agno in range(2020, 2071):
            cal.add_holiday(Qcf.QCDate(1, 1, agno))
            cal.add_holiday(Qcf.QCDate(18, 9, agno))
            cal.add_holiday(Qcf.QCDate(19, 9, agno))
            cal.add_holiday(Qcf.QCDate(25, 12, agno))

    return cal
```

```python
get_cal(BusCal.NY)
```

    <finrisk.QC_Financial_3.BusinessCalendar at 0x7f88aa515688>

```python
frmt = {
    'tasa': '{:.6%}',
    'df': '{:.6%}',
    'valor_tasa': '{:.4%}',
    'spread': '{:.4%}',
    'nominal': '{:,.0f}',
    'interes': '{:,.0f}',
    'amortizacion': '{:,.0f}',
    'flujo': '{:,.4f}',
}
```

```python
class TypeOis(Enum):
    SOFR = 1
    ICP = 2
```

```python
type_ois_template = {
    TypeOis.SOFR: {
        'currency': Qcf.QCUSD(),
        'periodicity': Qcf.Tenor('1Y'),
        'stub_period': Qcf.StubPeriod.SHORTFRONT,
        'settlement_lag': 0,
        'calendar': BusCal.NY,
        'bus_adj_rule': Qcf.BusyAdjRules.MODFOLLOW,
        'amort_is_cashflow': True,
        'fixed_rate': Qcf.QCInterestRate(0.0, Qcf.QCAct360(), Qcf.QCLinearWf()),
    },
    TypeOis.ICP: {
        'currency': Qcf.QCCLP(),
        'periodicity': Qcf.Tenor('6M'),
        'stub_period': Qcf.StubPeriod.SHORTFRONT,
        'settlement_lag': 0,
        'calendar': BusCal.SCL,
        'bus_adj_rule': Qcf.BusyAdjRules.MODFOLLOW,
        'amort_is_cashflow': True,
        'fixed_rate': Qcf.QCInterestRate(0.0, Qcf.QCAct360(), Qcf.QCLinearWf()),
    }
}
```

## Construye Curva Cero Cupón

Se importa la data de una curva cupón cero construida con cotizaciones SOFR.

```python
df_curva = pd.read_excel('data/20201012_built_sofr_zero.xlsx')
```

```python
def get_curve_from_dataframe(yf: Qcf.QCYearFraction, wf: Qcf.QCWealthFactor,
                             df_curva: pd.DataFrame) -> Qcf.ZeroCouponCurve:
    """
    Retorna un objeto Qcf.ZeroCouponCurve. Esta función requiere que `df_curva` tenga una columna
    de nombre 'plazo' y una columna de nombre 'tasa'. Se usa interpolación lineal en la curva que
    se retorna.
    """
    plazos = Qcf.long_vec()
    tasas = Qcf.double_vec()
    for row in df_curva.itertuples():
        plazos.append(row.plazo)
        tasas.append(row.tasa)
    curva = Qcf.QCCurve(plazos, tasas)
    curva = Qcf.QCLinearInterpolator(curva)
    tipo_tasa = Qcf.QCInterestRate(0.0, yf, wf)
    curva = Qcf.ZeroCouponCurve(curva, tipo_tasa)
    return curva
```

```python
df_curva.head().style.format(frmt)
```

        	        plazo	        tasa	        df

                        0
                        1
                        0.081111%
                        99.999778%

                        1
                        7
                        0.084051%
                        99.998388%

                        2
                        14
                        0.077967%
                        99.997010%

                        3
                        21
                        0.077358%
                        99.995549%

                        4
                        33
                        0.078067%
                        99.992942%

```python
zcc = get_curve_from_dataframe(Qcf.QCAct365(),Qcf.QCContinousWf(), df_curva)
```

Algunos métodos del objeto`zcc`.

```python
plazo = 900
print(f"Tasa a {plazo} días es igual a {zcc.get_rate_at(plazo):.4%}")
print(f"Factor de descuento a {plazo} días es igual a {zcc.get_discount_factor_at(plazo):.6%}")
```

    Tasa a 900 días es igual a 0.0652%
    Factor de descuento a 900 días es igual a 99.839331%

## Valorización

```python
def get_ois_using_template(
    template,
    type_ois: TypeOis,
    rp: Qcf.RecPay,
    notional: float,
    start_date: Qcf.QCDate,
    tenor: Qcf.Tenor,
    fixed_rate_value: float,
    spread: float,
    gearing: float
):
    """
    """
    template_dict = template[type_ois]
    meses = tenor.get_years() * 12 + tenor.get_months()
    end_date = start_date.add_months(meses)
    template_dict['fixed_rate'].set_value(fixed_rate_value)
    es_bono = False

    # Construye la pata fija
    fixed_rate_leg = Qcf.LegFactory.build_bullet_fixed_rate_leg(
        rp,
        start_date,
        end_date,
        template_dict['bus_adj_rule'],
        template_dict['periodicity'],
        template_dict['stub_period'],
        get_cal(template_dict['calendar']),
        template_dict['settlement_lag'],
        notional,
        template_dict['amort_is_cashflow'],
        template_dict['fixed_rate'],
        template_dict['currency'],
        es_bono)

    # Construye la pata ois
    rp = Qcf.RecPay.PAY if rp == Qcf.RecPay.RECEIVE else Qcf.RecPay.RECEIVE
    icp_clp_leg = Qcf.LegFactory.build_bullet_icp_clp2_leg(
        rp,
        start_date,
        end_date,
        template_dict['bus_adj_rule'],
        template_dict['periodicity'],
        template_dict['stub_period'],
        get_cal(template_dict['calendar']),
        template_dict['settlement_lag'],
        notional,
        template_dict['amort_is_cashflow'],
        spread,
        gearing,
        True
    )

    for i in range(icp_clp_leg.size()):
        cshflw = icp_clp_leg.get_cashflow_at(i)
        cshflw.set_start_date_icp(1.0)
        cshflw.set_end_date_icp(1.0)

    return (fixed_rate_leg, icp_clp_leg)
```

### Operación Ejemplo

```python
op = get_ois_using_template(
    type_ois_template,
    TypeOis.SOFR,
    Qcf.RecPay.RECEIVE,
    10000000,
    Qcf.QCDate(14, 10, 2020),
    Qcf.Tenor('2Y'),
    .01,
    0.0,
    1.0
)
op
```

    (<finrisk.QC_Financial_3.Leg at 0x7f88a65a67e8>,
     <finrisk.QC_Financial_3.Leg at 0x7f88a65a6a58>)

#### Digresión: `functools.partial`

Supongamos que estoy en una situación en la que sólo quiero construir OIS de *SOFR*. Me gustaría no tener que repetir los argumentos `type_ois_template` y `TypeOis.SOFR` cada vez que llamo la función `get_ois_using_template`. Puedo definir una nueva función de la siguiente forma:

```python
get_ois_sofr = partial(get_ois_using_template, type_ois_template, TypeOis.SOFR)
```

Con esta nueva función, `get_ois_sofr`, ahora puedo construir la operación `op` de la siguiente forma:

```python
op = get_ois_sofr(
    Qcf.RecPay.RECEIVE,
    10000000,
    Qcf.QCDate(14, 10, 2020),
    Qcf.Tenor('5Y'),
    .01,
    0.0,
    1.0
)
```

#### Continuamos (fin digresión ...)

```python
my.leg_as_dataframe(op[0], my.TipoPata.FIJA).style.format(frmt)
```

        	        fecha_inicial	        fecha_final	        fecha_pago	        nominal	        amortizacion	        interes	        amort_es_flujo	        flujo	        moneda	        valor_tasa	        tipo_tasa

                        0
                        2020-10-14
                        2021-10-14
                        2021-10-14
                        10,000,000
                        0
                        101,389
                        True
                        101,388.8889
                        USD
                        1.0000%
                        LinAct360

                        1
                        2021-10-14
                        2022-10-14
                        2022-10-14
                        10,000,000
                        0
                        101,389
                        True
                        101,388.8889
                        USD
                        1.0000%
                        LinAct360

                        2
                        2022-10-14
                        2023-10-16
                        2023-10-16
                        10,000,000
                        0
                        101,944
                        True
                        101,944.4444
                        USD
                        1.0000%
                        LinAct360

                        3
                        2023-10-16
                        2024-10-15
                        2024-10-15
                        10,000,000
                        0
                        101,389
                        True
                        101,388.8889
                        USD
                        1.0000%
                        LinAct360

                        4
                        2024-10-15
                        2025-10-14
                        2025-10-14
                        10,000,000
                        10,000,000
                        101,111
                        True
                        10,101,111.1111
                        USD
                        1.0000%
                        LinAct360

```python
my.leg_as_dataframe(op[0], my.TipoPata.FIJA).to_excel('ejemplo_valorizacion.xlsx')
```

```python
my.leg_as_dataframe(op[1], my.TipoPata.ICPCLP).style.format(frmt)
```

        	        fecha_inicial	        fecha_final	        fecha_pago	        nominal	        amortizacion	        amort_es_flujo	        flujo	        moneda	        icp_inicial	        icp_final	        valor_tasa	        interes	        spread	        gearing	        tipo_tasa

                        0
                        2020-10-14
                        2021-10-14
                        2021-10-14
                        -10,000,000
                        0
                        True
                        0.0000
                        CLP
                        1.000000
                        1.000000
                        0.0000%
                        -0
                        0.0000%
                        1.000000
                        LinAct360

                        1
                        2021-10-14
                        2022-10-14
                        2022-10-14
                        -10,000,000
                        0
                        True
                        0.0000
                        CLP
                        1.000000
                        1.000000
                        0.0000%
                        -0
                        0.0000%
                        1.000000
                        LinAct360

                        2
                        2022-10-14
                        2023-10-16
                        2023-10-16
                        -10,000,000
                        0
                        True
                        0.0000
                        CLP
                        1.000000
                        1.000000
                        0.0000%
                        -0
                        0.0000%
                        1.000000
                        LinAct360

                        3
                        2023-10-16
                        2024-10-15
                        2024-10-15
                        -10,000,000
                        0
                        True
                        0.0000
                        CLP
                        1.000000
                        1.000000
                        0.0000%
                        -0
                        0.0000%
                        1.000000
                        LinAct360

                        4
                        2024-10-15
                        2025-10-14
                        2025-10-14
                        -10,000,000
                        -10,000,000
                        True
                        -10,000,000.0000
                        CLP
                        1.000000
                        1.000000
                        0.0000%
                        -0
                        0.0000%
                        1.000000
                        LinAct360

#### Valor Presente Pata Fija

```python
vp = Qcf.PresentValue()
```

```python
fecha_val = Qcf.QCDate(14, 10, 2020)
```

```python
vp_fija = vp.pv(fecha_val, op[0], zcc)
print(f'El valor presente de la pata fija es: USD {vp_fija:,.4f}')
```

    El valor presente de la pata fija es: USD 10,408,804.8068

**Ejercicio:** Replique el valor de la pata fija utilizando los flujos y los factores de descuento que obtiene de la curva `zcc`.

```python
# Se dan de alta las fechas finales de ambos cupones
fecha1 = Qcf.QCDate(14, 10, 2021)
fecha2 = Qcf.QCDate(14, 10, 2022)

# O de forma más automática:
fecha1 = op[0].get_cashflow_at(0).get_settlement_date()
fecha2 = op[0].get_cashflow_at(1).get_settlement_date()

# Se calcula el número de días entre la fecha de valorización (fecha_val) y las fechas
# finales de ambos cupones.
plazo1 = fecha_val.day_diff(fecha1)
plazo2 = fecha_val.day_diff(fecha2)

# Utilizando la curva zcc se calculan los df a esos plazos
df1 = zcc.get_discount_factor_at(plazo1)
df2 = zcc.get_discount_factor_at(plazo2)

# Se obtienen los flujos totales (interés y amortización) de ambos cupones
flujo1 = op[0].get_cashflow_at(0).amount()
flujo2 = op[0].get_cashflow_at(1).amount()

# Finalmente, se calcula el valor presente como el producto (escalar) entre los df y los flujos.
check_vp = df1 * flujo1 + df2 * flujo2

# Se muestra el resultado.
print(f'El valor presente a mano es: {check_vp:,.8f}')
```

    El valor presente a mano es: 202,591.56249166

#### Valor Presente Pata Flotante

```python
fwd = Qcf.ForwardRates()
```

```python
print(f'VP: {vp.pv(fecha_val, op[1], zcc):,.2f}')
```

    VP: -9,903,483.66

```python
print(f'{df2 * 10000000:,.2f}')
```

    9,988,654.87

```python
fwd.set_rates_icp_clp_leg(fecha_val, 1.0, op[1], zcc)
```

```python
my.leg_as_dataframe(op[1], my.TipoPata.ICPCLP).style.format(frmt)
```

        	        fecha_inicial	        fecha_final	        fecha_pago	        nominal	        amortizacion	        amort_es_flujo	        flujo	        moneda	        icp_inicial	        icp_final	        valor_tasa	        interes	        spread	        gearing	        tipo_tasa

                        0
                        2020-10-14
                        2021-10-14
                        2021-10-14
                        -10,000,000
                        0
                        True
                        -7,026.2451
                        CLP
                        1.000000
                        1.000703
                        0.0700%
                        -7,097
                        0.0000%
                        1.000000
                        LinAct360

                        1
                        2021-10-14
                        2022-10-14
                        2022-10-14
                        -10,000,000
                        0
                        True
                        -4,328.7268
                        CLP
                        1.000703
                        1.001136
                        0.0400%
                        -4,056
                        0.0000%
                        1.000000
                        LinAct360

                        2
                        2022-10-14
                        2023-10-16
                        2023-10-16
                        -10,000,000
                        0
                        True
                        -11,198.7676
                        CLP
                        1.001136
                        1.002257
                        0.1100%
                        -11,214
                        0.0000%
                        1.000000
                        LinAct360

                        3
                        2023-10-16
                        2024-10-15
                        2024-10-15
                        -10,000,000
                        0
                        True
                        -27,864.3756
                        CLP
                        1.002257
                        1.005050
                        0.2700%
                        -27,375
                        0.0000%
                        1.000000
                        LinAct360

                        4
                        2024-10-15
                        2025-10-14
                        2025-10-14
                        -10,000,000
                        -10,000,000
                        True
                        -10,046,724.2505
                        CLP
                        1.005050
                        1.009746
                        0.4600%
                        -46,511
                        0.0000%
                        1.000000
                        LinAct360

```python
vp_flot = vp.pv(fecha_val, op[1], zcc)
print(f'El valor presente de la pata flotante es: USD {vp_flot:,.2f}')
```

    El valor presente de la pata flotante es: USD -10,000,000.00

```python
c = op[1].get_cashflow_at(0)
```

¿Porqué sabíamos que tenía que dar 10,000,000.00? **Hint**: la respuesta está en la construcción de la curva cero OIS.

## Sensibilidad

En `QC_Financial_3` al calcular el valor presente, también se calculan las derivadas del valor presente respecto a cada uno de los vértices de la curva.

### Pata Fija

```python
vp.pv(fecha_val, op[0], zcc)
der = vp.get_derivatives()
```

Con esas derivadas, se puede calcular la sensibilidad a cada vértice de la curva cupón cero para un movimiento de 1 punto básico.

```python
print(type(der))
```

    <class 'finrisk.QC_Financial_3.double_vec'>

```python
delta = .0001
total = 0
for i, d in enumerate(der):
    total += d * delta
    print(f"Sensibilidad en {i}: {d * delta: 0,.2f}")
print(f"Sensibilidad total: {total:,.2f}")
```

    Sensibilidad en 0:  0.00
    Sensibilidad en 1:  0.00
    Sensibilidad en 2:  0.00
    Sensibilidad en 3:  0.00
    Sensibilidad en 4:  0.00
    Sensibilidad en 5:  0.00
    Sensibilidad en 6:  0.00
    Sensibilidad en 7:  0.00
    Sensibilidad en 8:  0.00
    Sensibilidad en 9:  0.00
    Sensibilidad en 10:  0.00
    Sensibilidad en 11:  0.00
    Sensibilidad en 12:  0.00
    Sensibilidad en 13:  0.00
    Sensibilidad en 14:  0.00
    Sensibilidad en 15: -10.12
    Sensibilidad en 16:  0.00
    Sensibilidad en 17: -20.24
    Sensibilidad en 18: -30.55
    Sensibilidad en 19: -40.36
    Sensibilidad en 20: -4,994.91
    Sensibilidad en 21:  0.00
    Sensibilidad en 22:  0.00
    Sensibilidad en 23:  0.00
    Sensibilidad en 24:  0.00
    Sensibilidad en 25:  0.00
    Sensibilidad en 26:  0.00
    Sensibilidad en 27:  0.00
    Sensibilidad en 28:  0.00
    Sensibilidad en 29:  0.00
    Sensibilidad en 30:  0.00
    Sensibilidad en 31:  0.00
    Sensibilidad en 32:  0.00
    Sensibilidad total: -5,096.19

**Ejercicio:** Verifique por diferencias finitas centrales la sensibilidad en el vértice 17. La aproximación de la derivada por diferencias finitas centrales es:

$$
\begin{equation}
f^{\prime}\left(x\right)\approx\frac{f\left(x+h\right)-f\left(x-h\right)}{2h}
\end{equation}
$$

### Pata Flotante

```python
vp.pv(fecha_val, op[1], zcc)
der = vp.get_derivatives()
```

```python
total = 0
for i, d in enumerate(der):
    total += d * delta
    print(f"Sensibilidad en {i}: {d * delta: 0,.2f}")
print(f"Sensibilidad total: {total:,.2f}")
```

    Sensibilidad en 0:  0.00
    Sensibilidad en 1:  0.00
    Sensibilidad en 2:  0.00
    Sensibilidad en 3:  0.00
    Sensibilidad en 4:  0.00
    Sensibilidad en 5:  0.00
    Sensibilidad en 6:  0.00
    Sensibilidad en 7:  0.00
    Sensibilidad en 8:  0.00
    Sensibilidad en 9:  0.00
    Sensibilidad en 10:  0.00
    Sensibilidad en 11:  0.00
    Sensibilidad en 12:  0.00
    Sensibilidad en 13:  0.00
    Sensibilidad en 14:  0.00
    Sensibilidad en 15:  0.70
    Sensibilidad en 16:  0.00
    Sensibilidad en 17:  0.86
    Sensibilidad en 18:  3.35
    Sensibilidad en 19:  11.08
    Sensibilidad en 20:  4,967.99
    Sensibilidad en 21:  0.00
    Sensibilidad en 22:  0.00
    Sensibilidad en 23:  0.00
    Sensibilidad en 24:  0.00
    Sensibilidad en 25:  0.00
    Sensibilidad en 26:  0.00
    Sensibilidad en 27:  0.00
    Sensibilidad en 28:  0.00
    Sensibilidad en 29:  0.00
    Sensibilidad en 30:  0.00
    Sensibilidad en 31:  0.00
    Sensibilidad en 32:  0.00
    Sensibilidad total: 4,983.99

La estructura es la misma que para una pata fija, lo que indica que se debe también incluir la sensibilidad a la curva de proyección.

```python
import numpy as np
result = []
for i in range(op[1].size()):
    # Se obtiene cada uno de los flujos
    cshflw = op[1].get_cashflow_at(i)

    # Se obtiene el gradiente de los flujos respecto a la curva de proyección
    amt_der = cshflw.get_amount_derivatives()

    # Se obtiene el factor de descuento que corresponde al flujo
    df = zcc.get_discount_factor_at(fecha_val.day_diff(cshflw.get_settlement_date()))

    # Con el gradiente y el factor de descuento se obtiene la sensibilidad
    amt_sens = [da_dz * delta * df for da_dz in amt_der]

    # Se almacena la sensibilidad
    if len(amt_sens) > 0:
        result.append(np.array(amt_sens))

# Vector donde se almacena la sensibilidad total por vértice
total = result[0] * 0

# Esta es una suma vectorial componente a componente
for r in result:
    total += r

# Se muestra el resultado
for i in range(len(total)):
    print(f"Sensibilidad en {i}: {total[i]:0,.2f}")

print(f"Sensibilidad de proyección: {sum(total):,.2f} USD")
```

    Sensibilidad en 0: -0.00
    Sensibilidad en 1: -0.00
    Sensibilidad en 2: -0.00
    Sensibilidad en 3: -0.00
    Sensibilidad en 4: -0.00
    Sensibilidad en 5: -0.00
    Sensibilidad en 6: -0.00
    Sensibilidad en 7: -0.00
    Sensibilidad en 8: -0.00
    Sensibilidad en 9: -0.00
    Sensibilidad en 10: -0.00
    Sensibilidad en 11: -0.00
    Sensibilidad en 12: -0.00
    Sensibilidad en 13: -0.00
    Sensibilidad en 14: -0.00
    Sensibilidad en 15: -0.70
    Sensibilidad en 16: -0.00
    Sensibilidad en 17: -0.86
    Sensibilidad en 18: -3.35
    Sensibilidad en 19: -11.08
    Sensibilidad en 20: -4,967.99
    Sensibilidad en 21: -0.00
    Sensibilidad en 22: -0.00
    Sensibilidad en 23: -0.00
    Sensibilidad en 24: -0.00
    Sensibilidad en 25: -0.00
    Sensibilidad en 26: -0.00
    Sensibilidad en 27: -0.00
    Sensibilidad en 28: -0.00
    Sensibilidad en 29: -0.00
    Sensibilidad en 30: -0.00
    Sensibilidad en 31: -0.00
    Sensibilidad en 32: -0.00
    Sensibilidad de proyección: -4,983.99 USD

**Ejercicio:** Ambas sensibilidades se cancelan. ¿Porqué?


```python
import sympy

import adaptive_trapezint as p1
import sinesum1 as p2
import centered_diff as p3
import find_primes as p4
```

# Homework 1
## Austin Ayers
### 2/12/2016

### Excercises Completed:
    Exercise 3.8 (adaptive_trapezint.py)
    Exercise 3.15 (sinesum1.py)
    Exercise 3.18 (centered_diff.py)
    Exercise 3.20 (find_primes.py)

## Adaptive trapezint
    Program that takes takes a function an approximates its integral using the trapezoidal rule. As well, the program uses a varying n (number of subdivisions) to acquire a certain accuracy.

```python
p1.outputInformation()
```

    Function	Error			Estimated n
    Cos [0,Pi]	0.00226972593728	508.367284751
    Sin [0,Pi]	8.94076871116e-06	508.367284751
    Sin [0,Pi/2]	0.00642962323093	508.367284751

## Sine Summation
    Program that compares a piecewise function: See 2.png
    and the function that can approximate it: See 1.png

```python
p2.outputTable()
```

    n	alpha		Value of S		Value of f	Value of f - S

    1	0.01	0.0795270998917		1		0.920472900108
    1	0.25	-0.424413181578		1		1.42441318158
    1	0.49	0.0795270998917		1		0.920472900108

    3	0.01	0.235663207245		1		0.764336792755
    3	0.25	-0.351656636165		1		1.35165663616
    3	0.49	0.235663207245		1		0.764336792755

    5	0.01	0.385248413145		1		0.614751586855
    5	0.25	-0.32593462516		1		1.32593462516
    5	0.49	0.385248413145		1		0.614751586855

    10	0.01	0.711980829915		1		0.288019170085
    10	0.25	-0.244361467333		1		1.24436146733
    10	0.49	0.711980829915		1		0.288019170085

    30	0.01	1.05156392494		1		-0.0515639249442
    30	0.25	-0.262974151619		1		1.26297415162
    30	0.49	1.05156392494		1		-0.0515639249442

    100	0.01	0.870356367282		1		0.129643632718
    100	0.25	-0.270088038931		1		1.27008803893
    100	0.49	0.870356367282		1		0.129643632718

## Centered Diff
    Approximates a derivative of a function f(x) using an approximation formula

```python
p3.application()
```

    error of e^x @ (x=0): 1.66667499921e-05
    error of e^(-2x^2) @ (x=0): 0.0
    error of cos(x) @ (x=2pi): 0
    error of ln(x) @ (x=1): 1.00003333533348

## Find Prime Numbers
    Uses the Sieve of Eratostenes to produce prime numbers

```python
p4.output()
```

    2
    3
    5
    7
    11
    13
    17
    19
    23
    29
    31
    37
    41
    43
    47
    53
    59
    61
    67
    71

```python

```


```python
# Preliminary functions for 2D finite element approximation

import matplotlib.pyplot as plt
import numpy as np
import sympy as sym

def mesh(nx, ny, x=[0,1], y=[0,1], diagonal='right'):
    """
    Return a 2D finite element mesh on a rectangle with
    extend x and y in the x and y directions.
    nx and ny are the divisions in the x and y directions.
    Return vertices and cells (local to global vertex number mapping).
    """
    if len(x) == 2:
        if nx is None:
            raise ValueError('box: interval in x %s, no nx set' % x)
        x = np.linspace(x[0], x[1], nx+1)
    else:
        nx = len(x)-1
    if len(y) == 2:
        if nx is None:
            raise ValueError('box: interval in y %s, no ny set' % y)
        y = np.linspace(y[0], y[1], ny+1)
    else:
        ny = len(y)-1

    if diagonal is None:
        vertices = np.zeros(((nx+1)*(ny+1), 2), dtype=np.float)
        cells = np.zeros((nx*ny, 4), dtype=np.int)
    elif diagonal == 'crossed':
        vertices = np.zeros(((nx+1)*(ny+1) + nx*ny, 2), dtype=np.float)
        cells = np.zeros((4*nx*ny, 3), dtype=np.int)
    else:
        vertices = np.zeros(((nx+1)*(ny+1), 2), dtype=np.float)
        cells = np.zeros((2*nx*ny, 3), dtype=np.int)

    vertex = 0
    for iy in range(ny+1):
        for ix in range(nx+1):
            vertices[vertex,:] = x[ix], y[iy]
            vertex += 1

    if diagonal == 'crossed':
        for iy in range(ny):
            for ix in range(nx):
                x_mid = 0.5*(x[ix+1] + x[ix])
                y_mid = 0.5*(y[iy+1] + y[iy])
                vertices[vertex,:] = x_mid, y_mid
                vertex += 1

    cell = 0
    if diagonal is None:
        # Quadrilateral elements
        for iy in range(ny):
            for ix in range(nx):
                v0 = iy*(nx + 1) + ix
                v1 = v0 + 1
                v2 = v0 + nx+1
                v3 = v1 + nx+1
                cells[cell,:] = v0, v1, v3, v2;  cell += 1

    elif diagonal == 'crossed':
        for iy in range(ny):
            for ix in range(nx):
                v0 = iy*(nx+1) + ix
                v1 = v0 + 1
                v2 = v0 + (nx+1)
                v3 = v1 + (nx+1)
                vmid = (nx+1)*(ny+1) + iy*nx + ix

                # Note that v0 < v1 < v2 < v3 < vmid.
                cells[cell,:] = v0, v1, vmid;  cell += 1
                cells[cell,:] = v0, v2, vmid;  cell += 1
                cells[cell,:] = v1, v3, vmid;  cell += 1
                cells[cell,:] = v2, v3, vmid;  cell += 1

    else:
        local_diagonal = diagonal
        # Set up alternating diagonal
        for iy in range(ny):
            if diagonal == "right/left":
                if iy % 2 == 0:
                    local_diagonal = "right"
                else:
                    local_diagonal = "left"

            if diagonal == "left/right":
                if iy % 2 == 0:
                    local_diagonal = "left"
                else:
                    local_diagonal = "right"
            for ix in range(nx):
                v0 = iy*(nx + 1) + ix
                v1 = v0 + 1
                v2 = v0 + nx+1
                v3 = v1 + nx+1

                if local_diagonal == "left":
                    cells[cell,:] = v0, v1, v2;  cell += 1
                    cells[cell,:] = v1, v2, v3;  cell += 1
                    if diagonal == "right/left" or diagonal == "left/right":
                        local_diagonal = "right"
                else:
                    cells[cell,:] = v0, v1, v3;  cell += 1
                    cells[cell,:] = v0, v2, v3;  cell += 1
                    if diagonal == "right/left" or diagonal == "left/right":
                        local_diagonal = "left"
    return vertices, cells

def plot_mesh(vertices, cells, materials=None, plotfile='tmp.png'):
    cell_vertex_coordinates = []
    for e in range(cells.shape[0]):
        local_vertex_numbers = cells[e,:]
        local_coordinates = vertices[local_vertex_numbers,:]
        cell_vertex_coordinates.append(local_coordinates)
    import matplotlib.cm as cm
    import matplotlib.collections as collections
    import matplotlib.pyplot as plt
    col = collections.PolyCollection(cell_vertex_coordinates)
    if materials is not None:
        col.set_array(materials)
        #col.set_cmap(cm.jet)
        #col.set_cmap(cm.gray_r)
        col.set_cmap(cm.hot_r)
    fig = plt.figure()
    ax = fig.gca()
    ax.add_collection(col)
    xmin, xmax = vertices[:,0].min(), vertices[:,0].max()
    ymin, ymax = vertices[:,1].min(), vertices[:,1].max()
    ax.set_xlim(xmin, xmax)
    ax.set_ylim(ymin, ymax)
    ax.set_aspect('equal')
    plt.savefig(plotfile + '.png')
    plt.savefig(plotfile + '.pdf')
    plt.show()

def demo():
    nx = 8; ny = 20

    vertices, cells = mesh(nx, ny, x=[1,2], y=[0,1], diagonal='crossed')
    Theta = np.pi/2
    x = vertices[:,0]*np.cos(Theta*vertices[:,1])
    y = vertices[:,0]*np.sin(Theta*vertices[:,1])
    vertices[:,0] = x
    vertices[:,1] = y
    plot_mesh(vertices, cells, materials=np.zeros(cells.shape[0], dtype=np.int), plotfile='tmp_circle')
    # FIXME: resulting plot is empty
    import sys
    sys.exit(0)

    nx = 4; ny = 3
    for diagonal in None, 'right', 'left', 'right/left', 'left/right', 'crossed':
        vertices, cells = mesh(nx, ny, x=[0,3], y=[0,1], diagonal=diagonal)
        plot_mesh(vertices, cells, materials=np.zeros(cells.shape[0], dtype=np.int), plotfile=str('tmp_' + str(diagonal)).replace('/', '_'))

if __name__ == '__main__':
    demo()

```

```python

```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/EXAMPLES/FINITE_ELEMENTS/INTRO/SRC/21_FE_APPROX2D.ipynb


# Homework 2

**For exercises in the week 20-25.11.19**

**Points: 6 + 1b**

Please solve the problems at home and bring to class a [declaration form](http://ii.uni.wroc.pl/~jmi/Dydaktyka/misc/kupony-klasyczne.pdf) to indicate which problems you are willing to present on the blackboard.

### Declartation

| Exercise || 1 | 2 | 3 | 4 |  5  | 6 | 7 | 8 |
|----------||---|---|---|---| --- |---|---|---|
| Points   || 0 | 1 | 0 | 1 | 0.5 | 0 | 1 | 0 |

## Problem 1 [1p]

Let $(x^{(i)},y^{(i)})$ be a data sample with $x^{(i)}\in\mathbb{R}^D$, $y^{(i)}\in\mathbb{R}$. Let $\Theta \in\mathbb{R}^D$ a parameter vector.

Find the closed form solution $\Theta^*$ to

$$
\min_\Theta \left(\frac{1}{2}\sum_i (\Theta^Tx^{(i)} - y^{(i)})^2 + \frac{\lambda}{2}\sum_{d=1}^D \Theta_d^2\right).
$$
### Ans:
$\Theta = (X^T X+\lambda I)^{-1}X^T Y$

## Problem 2 [1p]
Let $v\in\mathbb{R}^D$ be a vector. Define the gradient of $f(v)\in\mathbb{R}$ with respect to $v$ to be $\frac{\partial f}{\partial v} = \left[\frac{\partial f(v)}{\partial v_1}, \frac{\partial f(v)}{\partial v_2}, ..., \frac{\partial f(v)}{\partial v_D}\right]$

Find the following functions' gradients with respect to vector $[x, y, z]^T$:
1. $f_1([x, y, z]^T) = x + y$
2. $f_2([x, y, z]^T) = xy$
3. $f_3([x, y, z]^T) = x^2y^2$
4. $f_4([x, y, z]^T) = (x + y)^2$
5. $f_5([x, y, z]^T) = x^4 + x^2 y z + x y^2 z + z^4$
6. $f_6([x, y, z]^T) = e^{x + 2y}$
7. $f_7([x, y, z]^T) = \frac{1}{x y^2}$
8. $f_8([x, y, z]^T) = ax + by + c$
9. $f_9([x, y, z]^T) = \tanh(ax + by + c)$

### Ans:
1. [1, 1, 0]
2. [y, x, 0]
3. [$2y^2x$, $2x^2y$, 0]
4. [$2x + 2y$, $2x + 2y$, 0]
5. [$4x^3 +2yzx +y^2z$, $2xzy + zx^2$, $3x^4 + x^2y + xy^2$]
6. [$e^{x + 2y}$, $2e^{x + 2y}$, 0]
7. [$-\frac{1}{y^2x^2}$, $-\frac{2}{xy^3}$, 0]
8. [$a$, $b$, 0]
9. [$a*sech^2(ax + by + c)$, $b*sech^2(ax + by + c)$, 0]

## Problem 3 [0.5p]

Find the following functions' gradients or Jacobians with respect to vector $\mathbf{x}$, where $\mathbf{x}, \mathbf{b} \in \mathbb{R}^{n}$, $\mathbf{W} \in \mathbb{R}^{n \times n}$:

1. $\mathbf{W} \mathbf{x} + \mathbf{b}$
2. $\mathbf{x}^T \mathbf{W} \mathbf{x}$,

### Ans:
1. $\mathbf{W}$
2. $x^T(W^T + W)$

## Problem 4 [1p]

Find the derivative of
$-\log(S(\mathbf{x})_j)$, where $S$ is the
    softmax function
    (https://en.wikipedia.org/wiki/Softmax_function) and we are
    interested in the derivative over the $j$-th output of the
    Softmax.

### Ans:
$$
(-log(S(x_j)))' = - \frac{1}{S(x_j)} * S'(x_j) = - \frac{1}{S(x_j)} * (\frac{e^{x_j}}{\sum_i e^{x_i}})' =
$$

$$
= - \frac{1}{S(x_j)} * \frac{(e^{x_j})' * (\sum_i e^{x_i}) - e^{x_j} * (\sum_i e^{x_i})'}{(\sum_i e^{x_i})^2} =
$$

$$
= - \frac{\sum_i e^{x_i}}{e^{x_j}} * \frac{e^{x_j} * \sum_i e^{x_i} - e^{x_j} * e^{x_j}}{(\sum_i e^{x_i})^2} =
$$

$$
= \frac{\sum_i e^{x_i} - e^{x_j}}{\sum_i e^{x_i}} = S(x_j) - 1
$$

## Problem 5 [0.5p]

Consider a dataset with 400 examples of class C1 and 400 of class C2.
Let tree A have 2 leaves with class distributions:

| Tree A   | C1    | C2  |
|----------|-------|-----|
| Leaf 1   | 100   | 300 |
| Leaf 2   | 300   | 100 |

and let tree B have 2 leaves with class distribution:

| Tree B   | C1    | C2  |
|----------|-------|-----|
| Leaf 1   | 200   | 400 |
| Leaf 2   | 200   |   0 |

What is the misclassification rate for both trees? Which tree is more pure according to Gini or Infogain?

### Ans:

```python
# misclassification rate:
1/2*1/4 + 1/2 * 1/4, 1/2 * 1/2 + 1/2 * 0
```

    (0.25, 0.25)

```python
import pandas as pd
import numpy as np
def entropy(counts):
    _sum = np.sum(counts.values, axis=0)
    return -np.sum(counts/_sum * np.log2(counts/_sum), axis=0)

def gini(counts):
#     print(type(counts))
#     print(counts)
    _sum = np.sum(counts.values, axis=0)
    return np.sum(counts/_sum * (1 - counts/_sum), axis=0)

def mean_err_rate(counts):
    _sum = np.sum(counts.values, axis=0)
    return np.min(counts/_sum, axis=0)

data1 = pd.DataFrame.from_dict({"c1": [100,300], "c2": [300,100]}).transpose()
data2 = pd.DataFrame.from_dict({"c1": [200,200], "c2": [400,0]}).transpose()

np.average(gini(data1)), np.average(gini(data2), weights=[2/3, 1/3])
```

    (0.375, 0.2962962962962963)

```python
np.average(entropy(data1)), np.average(entropy(data2), weights=[2/3, 1/3])
```

    /Users/michal/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:5: RuntimeWarning: divide by zero encountered in log2
      """

    (0.8112781244591328, 0.612197222702993)

## Problem 6 [1p]

Consider regresion problem, with $M$ predictors $h_m(x)$ trained to aproximate a target $y$. Define the error to be $\epsilon_m(x) = h_m(x) - y$.

Suppose you train $M$ independent classifiers with average least squares error
$$
E_{AV} = \frac{1}{M}\sum_{m=1}^M \mathbb{E}_{x}[\epsilon_m(x)^2].
$$

Further assume that the errors have zero mean and are uncorrelated:
$$
\mathbb{E}_{x}[\epsilon_m(x)] = 0\qquad\text{ and }\qquad\mathbb{E}_{x}[\epsilon_m(x)\epsilon_l(x)] = 0\text{ for } m \neq l
$$

Let the mean predictor be
$$
h_M(x) = \frac{1}{M}h_m(x).
$$

What is the average error of $h_M(x)$?

## Problem 7 [1p]

Suppose you work on a binary classification problem and train 3 weak classifiers. You combine their prediction by voting.

Can the training error rate of the voting ensemble smaller that the error rate of the individual weak predictors? Can it be larger? Show an example or prove infeasibility.

```python
import numpy as np
import matplotlib.pyplot as plt
A = np.arange(1,7)
B = np.arange(5,11)
C = np.hstack([np.arange(1,5),[7,8]])
Res = np.arange(1, 11)
ResC = ['g'] * 8  + ['r'] * 2
plt.scatter(A, A*0+1)
plt.scatter(B, B*0+1.5)
plt.scatter(C, C*0+2, c='purple')
plt.scatter(Res, Res*0, c=ResC )

```

### 3*60% -> 80%

```python
import numpy as np
import matplotlib.pyplot as plt
A = np.arange(1,7)
B = np.hstack([np.arange(1,5),[9,10]])
C = np.hstack([np.arange(1,5),[7,8]])
Res = np.arange(1, 11)
ResC = ['g'] * 4  + ['r'] * 6
plt.scatter(A, A*0+1)
plt.scatter(B, B*0+1.5)
plt.scatter(C, C*0+2, c='purple')
plt.scatter(Res, Res*0, c=ResC )

```

### 3*60% -> 40%

## Problem 8 [1 bonus point]

While on a walk, you notice that a locomotive has the serial number 50. Assuming that all locomotives used by PKP (the Polish railroad operator) are numbered using consecutive natural numbers, what is your estimate of $N$ the total number of locomotives operated by PKP?

Tell why the Maximum Likelihood principle may not yield satisfactory results.

Use the Bayesian approach to find the posterior distribution over
  the number of locomotives. Then compute the expected count of
  locomotives. For the prior use the power law:
  \begin{equation}
  p(N) =  \frac{1}{N^\alpha}\frac{1}{\zeta(\alpha,1)},
  \end{equation}
  where the $\zeta(s,q)=\sum_{n=0}^{\infty}\frac{1}{(q+n)^s}$ is the
  Hurwitz Zeta function
  (https://en.wikipedia.org/wiki/Hurwitz_zeta_function)
  available in Python as `scipy.special.zeta`. The use of the
  power law is motivated by the observation that the frequency of
  occurrence of a company is inversely proportional to its size (see
  also: R.L. Axtell, Zipf distribution of US firm sizes
  https://www.sciencemag.org/content/293/5536/1818).

  How would your estimate change after seeing 5 locomotives, with the
  biggest serial number among them being 50?

  **Note**: During the Second World War, a similar problem was
  encountered while trying to estimate the total German tank
  production from the serial numbers of captured machines. The
  statistical estimates were the most precise!


# Overlapping Generations Model (OLG)
### The Usefulness of the Model
The key ingredient of the OLG model is exogenous population turnover. Individuals in the population live for two periods, one in which they work and save, and the second in which they are 'old', do not work, but rather live off their savings.
The model can be used to assess various systems of 'social security'; is there a place for the government in providing for the older generation?
In the following, we describe the basic model setup, and go through how to solve the model analytically. A reader who is familiar with the OLG model may choose to simple skip these next steps and go straight to our coded implementation - our model setup is entirely standard, and as presented e.g. in Macro III at KU.

### Model Setup
Time is discrete and infinite, $t = 0,1,2,...$.
Population grows at a constant rate, n:
    $$ L_t = L_{t-1}(1+n)$$
This implies, that in every period t, there are $L_t$ 'young' (working) individuals, and $L_{t-1}$ 'old' (non-working) individuals.  Agents derive utility from consumption in each of the two periods they are alive:
    $$U_t = u(c_{1t})+\frac{1}{1+\rho} \cdot u(c_{2t+1})$$
Where $c_{1t}$ is consumption of the young at time t, $c_{2t+1}$ is consumption of old at time $t+1$ and $\rho$ is a discount factor.
Let $r_{t+1}$ denote the interest rate between $t$ and $t+1$. Then individuals are budget constrained in each period of their live as follows:
    $$c_{1t} + s_{t} = w_t$$
    $$c_{2t+1} = (1+r_{t+1})\cdot s_t$$
Substituting for s, we get the following life-time constraint:
    $$c_{1+t} + \frac{c_{2t+1}}{1+r_{t+1}} = w_t$$

In every period t, agents born at time t solve the following problem:
    $$\max_{c_{1t},c_{2t+1}} u(c_{1t})+\frac{1}{1+\rho} u(c_{2t+1})$$
Subject to the constraint from above:
    $$c_{1+t} + \frac{c_{2t+1}}{1+r_{t+1}} = w_t$$

Production is assumed to take place with CRS technology, competitive markets and profit-maximizing firms. This yields:
    $$r_t = f'(k_t)$$
    $$w_t = f(k_t)-f'(k_t)k_t$$
Where $k_t = \frac{K_t}{L_t}$. Recall, that $L_t$ refers to the *working population* at time t.

### Solving the Household Problem
The household problem is a standard optimization problem subject to a constraint. We set up the lagrangian:
    $$ L(c_{1t},c_{2t+1},\lambda) = u(c_{1t}) + \frac{1}{1+\rho} u(c_{2t+1}) + \lambda[w_t-c_{1t}-\frac{c_{2t+1}}{1+r_{t+1}}]$$
Differentiate wrt. consumption (in each period):
    $$\frac{\partial L}{\partial c_{1t}} = u'(c_{1t}) -\lambda $$
    $$\frac{\partial L}{\partial c_{2t+1}} = \frac{1}{1+\rho} u'(c_{2t+1}) -\frac{\lambda}{1+r_{t+1}} $$
Equate to zero (first order conditions) and substitute for the lagrange multiplier $\lambda$ to obtain the Euler Equation:
    $$u'(c_{1t}) = \frac{1+r_{t+1}}{1+\rho} u'(c_{2t+1})$$

### Characterizing Optimal Savings
Substitute the budget constrains into the Euler Equation to get:
    $$u'(w_t-s_t) = \frac{1+r_{t+1}}{1+\rho} u'((1+r_{t+1})s_t)$$
This implicitly defines optimal savings as a function of wage and interest rate, i.e. $s(w_t,r_{t+1})$

### Law of Motion for Capital
Knowing the individual decisions allow us to aggregate economy-wide. The aggregate capital stock at time $t+1$ is equal to aggregate savings in time $t$, aggregate dissaving in time t, and un-depreciated capital carried over from time t.
    $$K_{t+1} = S_t -K_t+(1-\delta)K_t$$
    $$\Rightarrow K_{t+1} = S_t - \delta K_t$$
        $$\Rightarrow k_{t+1}(1+n) = s_t-\delta k_t$$

Substituting for savings, substituting for wage and interest rate, and assuming $\delta = 0$:
    $$k_{t+1} (1+n) = s(f(k_t)-k_tf'(k_t),f'(k_{t+1})$$
Which implicitly defines the law of motion for aggregate capital per worker.

```python
import numpy as np
import sympy as sp
import matplotlib.pyplot as plt

from modelproject import *
```

```python
sp.init_printing()
```

We solve the OLG model using sympy, thus we begin by setting up symbols for the variables we use

```python
rho = sp.symbols('rho')
delta = sp.symbols('delta')
n = sp.symbols('n')

c1 = sp.symbols('c_{1t}')
c2 = sp.symbols('c_{2t+1}')

r = sp.symbols('r_{t+1}')
w = sp.symbols('w_t')

s = sp.symbols('s_t')
k = sp.symbols('k_t')
kp1 = sp.symbols('k_{t+1}')

lmbda = sp.symbols('lambda')
```

next we define the standard log utility, and a placeholder that keeps the code general for the derivations. By using `sp.Function` for the utility we can then substitute in the specific functional form at a laster stage.

```python
u_undetermined = sp.Function('u')

def log_u(c):
    return sp.log(c)

def U(c1, c2, rho, u):
    return u(c1) + 1 / (1 + rho) * u(c2)

U(c1, c2, rho, u = u_undetermined)
```

We define budget constraints and calculate the intertemporal constraint

```python
period_1_budget = sp.Eq(w, c1 + s)
period_2_budget = sp.Eq(c2, (1+r)*s)

period_1_budget, period_2_budget
```

```python
inter_budget = period_1_budget.subs(s, sp.solve(period_2_budget, s)[0])
inter_budget
```

next we set up the lagrangian and calculate the relevant first derivatives

```python
lagrangian = U(c1,c2,rho, u = u_undetermined) - lmbda *  (inter_budget.rhs - w)
lagrangian
```

```python
dc1    = sp.diff(lagrangian, c1)
dc2    = sp.diff(lagrangian, c2)
dlmbda = sp.diff(lagrangian, lmbda)

dc1, dc2, dlmbda
```

Calculate euler equation

Finally we call our `make_euler_equation` function with the derivatives of the lagrangian as inputs. We the nsubstitute in the within period budget constraints to gen an exproession in terms of savings wages and interest rate.

```python
euler_eq = make_euler_equation(dc1, dc2, u = u_undetermined, c1 = c1, lmbda = lmbda)
euler_eq
```

```python
euler_eq = euler_eq.subs(c1, w - s).subs(c2, period_2_budget.rhs)
euler_eq
```

Implicitly the euler equation determines $s$, now we move to determine the evolution of captial in the economy.

This is also a good spot to set the actual functional form of $u$, we use a $log$ function here as it is analytically solveable, and explore using the square root further down.

```python
euler_eq = euler_eq.replace(u_undetermined, log_u).doit()
euler_eq
```

Next setup for firms. We show the aggregate production function but the work with the per capita normalized version to calculate expressions for the equilibrium interest rates and wages.

```python
Y = sp.symbols('Y_t')
K = sp.symbols('K_t')
L = sp.symbols('L_t')

y = sp.symbols('y_t')

alpha = sp.symbols('alpha')

production_function = sp.Eq(
    Y,
    K**alpha * L**(1-alpha)
)
production_function

```

```python
norm_prod_func = sp.Eq(y, k**alpha)
norm_prod_func
```

Define interest rate and wage

```python
interest_rate = sp.Eq(r, sp.Derivative(norm_prod_func.rhs, k)).doit()
wage = sp.Eq(w, norm_prod_func.rhs - k*sp.Derivative(norm_prod_func.rhs, k)).doit()

interest_rate, wage
```

Finally from a standard capital depreciation identity we derive a first expression for the evolution of capital in the economy.

```python
evolution = sp.Eq(kp1, 1/(1+n)* (s - delta*k) )
evolution
```

Using the euler equation and the capital evolution path we can derive the transition path of the economy

```python
savings_of_k = sp.solve(euler_eq, s)[0].subs(w, wage.rhs)

# Only substitute
if r in savings_of_k.atoms():
    savings_of_k = savings_of_k.subs(r, interest_rate.rhs)

transition_eq = evolution.subs(s, savings_of_k)
transition_eq
```

In the following section we show visualizations of the capital accumulation path for various parametrizations, including varying $\alpha$ and $\n$.

```python
_transition_func = sp.lambdify((k, rho, delta, n, alpha, r), transition_eq.rhs)

def transition_func(k, alpha, rho, delta, n, r = 0):
    return _transition_func(k, rho, delta, n , alpha, r)
```

```python
_equilibrium = sp.lambdify(
    (alpha, n, rho, delta),
    sp.solve(transition_eq.subs(kp1, k), k)[0]
            )

def equilibrium(alpha, n, rho, delta):
    return _equilibrium(alpha, n, rho, delta)
```

Visualizing variation in $\alpha$

```python
xr = np.linspace(0,2,1000)
_r = 0.05
_d = 0.02
_n = 0.02

for _a in np.linspace(0.1, 0.5, 5):

    res = [transition_func(k = x, alpha = _a, rho = _r, delta = _d, n = _n) for x in xr]
    kstar = equilibrium(alpha = _a, rho = _r, n = _n, delta = _d)
    plt.plot(xr, res, color = 'blue', alpha = 1)
    plt.scatter([kstar], [kstar], color = 'red')
    plt.annotate(f'$\\alpha$={round(_a,2)}', xy= (kstar + 0.01, kstar - 0.02))

plt.plot(xr, xr, color = 'black')
plt.xlim(0,0.5)
plt.ylim(0,0.5)
plt.xlabel('$k_t$')
plt.ylabel('$k_{t+1}$')
plt.title('Transition curves for varying $\\alpha$')
```

Visualizing variation in $n$

```python
xr = np.linspace(0,2,1000)
_r = 0.05
_d = 0.02
_a = 0.2
#_n = 0.02

for _n in np.linspace(0, 1, 5):

    res = [transition_func(k = x, alpha = _a, rho = _r, delta = _d, n = _n) for x in xr]
    kstar = equilibrium(alpha = _a, rho = _r, n = _n, delta = _d)
    plt.plot(xr, res, color = 'blue', alpha = 1)
    plt.scatter([kstar], [kstar], color = 'red')
    plt.annotate(f'$n$={round(_n,2)}', xy= (kstar + 0.01, kstar - 0.02))

plt.plot(xr, xr, color = 'black')
plt.xlim(0,0.5)
plt.ylim(0,0.5)
plt.xlabel('$k_t$')
plt.ylabel('$k_{t+1}$')
plt.title('Transition curves for varying $n$')
```

Visualizing the convergence to st.st.

```python
k_ = 0.0001
xr = range(10)
xr2 = np.linspace(0,1,1000)

_r = 0.05
_d = 0.05
_a = 0.5
_n = 0

out = list()

for _ in xr:
    k_ = transition_func(k = k_, alpha = _a, rho = _r, delta = _d, n = _n)
    out.append(k_)

```

```python
res = [transition_func(k = x, alpha = _a, rho = _r, delta = _d, n = _n) for x in xr2]

plt.plot(xr, xr, color = 'black')
plt.plot(xr2, res, color = 'blue', alpha = 1)
plt.step(out[:-1], out[1:], where = 'post', color = 'red', linestyle = '--', alpha = 0.8)
plt.scatter(out[:-1], out[1:], color = 'red', alpha = 0.8)

plt.xlabel('$k_t$')
plt.ylabel('$k_{t+1}$')
plt.xlim(0,0.1)
plt.ylim(0,0.1)
```

## Part 2 - numerical solution with sqrt utility

In the section below we first show that there is no analytical solution to the OLG model when the within period utility is given by $u(c)=\sqrt(c)$. We then show graphically that even though no analytical solution exists, it can be found numerically, and we iteratively identify the aproximate value of $k^*$ with square root utility.

The first steps are identical to the ones covered above.

```python
def sqrt_u(c):
    return sp.sqrt(c)
```

```python
euler_eq = make_euler_equation(dc1, dc2, u = u_undetermined, c1 = c1, lmbda = lmbda)
euler_eq = euler_eq.subs(c1, w - s).subs(c2, period_2_budget.rhs)
euler_eq = euler_eq.replace(u_undetermined, sqrt_u).doit()
euler_eq
```

```python
savings_of_k = sp.solve(euler_eq, s)[0].subs(w, wage.rhs)
savings_of_k
```

```python
savings_of_k = savings_of_k.subs(r, interest_rate.rhs)
savings_of_k
```

```python
transition_eq = evolution.subs(s, savings_of_k)
transition_eq
```

```python
_tf2 = sp.lambdify((k, alpha, rho, delta, n), transition_eq.rhs)
def tf2(k, alpha, rho, delta, n):
    return _tf2(k, alpha, rho, delta, n)

```

Notice now the analytical solution is empty.

```python
sp.solve(transition_eq.subs(kp1, k), k)
```

Here we compare the steady state of a model with log utility and one with square root utility, notice how the change in utility drastically changes the steady state level of capital.

```python
k_ = 0.0001
k_2 = 0.0001

xr = range(10)
xr2 = np.linspace(0,1,1000)

_r = 0.05
_d = 0.05
_a = 0.5
_n = 0

out = list()
out2 = list()

for _ in xr:
    k_ = tf2(k = k_, alpha = _a, rho = _r, delta = _d, n = _n)
    k_2 = transition_func(k = k_2, alpha = _a, rho = _r, delta = _d, n = _n)

    out.append(k_)
    out2.append(k_2)

res = [tf2(k = x, alpha = _a, rho = _r, delta = _d, n = _n) for x in xr2]
res2 = [transition_func(k = x, alpha = _a, rho = _r, delta = _d, n = _n) for x in xr2]

plt.plot(xr, xr, color = 'black')
plt.plot(xr2, res, color = 'blue', alpha = 1, label = 'sqrt')
plt.plot(xr2, res2, color = 'navy', alpha = 1, label = 'log')

plt.step(out[:-1], out[1:], where = 'post', color = 'red', linestyle = '--', alpha = 0.8)
plt.scatter(out[:-1], out[1:], color = 'red', alpha = 0.8)

plt.step(out2[:-1], out2[1:], where = 'post', color = 'red', linestyle = '--', alpha = 0.8)
plt.scatter(out2[:-1], out2[1:], color = 'red', alpha = 0.8)

plt.xlabel('$k_t$')
plt.ylabel('$k_{t+1}$')
plt.xlim(0,0.2)
plt.ylim(0,0.2)

plt.legend()

```

Finally using the simple convergence algorithm shown in the figure we can approximate the steady state level of capital in an economy characterized by square root utility.

```python
k_ = 0.0001
for _ in range(10000):
    k_ = tf2(k = k_, alpha = _a, rho = _r, delta = _d, n = _n)

print(k_)
```

    0.10949660035925014


# Session 12: Gaussian Mixture Models and the EM algorithm

------------------------------------------------------
*Introduction to Data Science & Machine Learning*

*Pablo M. Olmos olmos@tsc.uc3m.es*

------------------------------------------------------

[Mixture Models](http://www.stat.cmu.edu/~cshalizi/uADA/12/lectures/ch20.pdf) are a special class of probabilistic generative models that can be used for both **density estimation** and **clustering**. Given a set of multidimensional observations, mixture models assume that observed variables are correlated because they arise from a hidden common "cause" or **latent variable** that is discrete (it can only take a finite number of values).

Fitting the parameters of Mixture Models is particularly challenging, as we deal with non-convex problems characterized by multiple modes. The **Expectation-Maximization (EM)** algorithm provides a simple iterative solution that guarantees to converge to a **local mode**.

In this notebook, we will present the EM algorithm to fit a Mixture of Gaussian distributions or [**Gaussian Mixture Models** (GMM)](https://brilliant.org/wiki/gaussian-mixture-model/).

Recommended bibliography:
- Chapter 09 of Pattern Recognition and Machine Learning, Christopher Bishop, 2006
- Chapter 11 of Machine Learning, a Probabilistic Perspective, Kevin Murply 2012

## Mixture Models

We say that a distribution $p(\mathbf{x})$ is a **mixture of $K$ component** distributions
$p_{1}(\mathbf{x}, p_{2}(\mathbf{x}),\ldots,p_{K}(\mathbf{x})$ if

\begin{align}
p(\mathbf{x})=\sum_{k=1}^{K} \pi_k p_{k}(\mathbf{x}),
\end{align}
with $\pi_k$ being the **mixing weights**, $\pi_k>0$, $\sum_k \pi_k =1$. An equivalent way to understad a Mixture Model is to think that every sample $\mathbf{x}$ is generated by first sample a discrete R.V. $z\in\{1,\ldots,K\}$ with probabilities $\pi_1,\ldots,\pi_K$, and then generate a sample $\mathbf{x}$ from $p_k(\mathbf{x})$. Namely,
\begin{align}
p(\mathbf{x})=\int_{z} p(\mathbf{x},z) dz = \sum_{k=1}^{K} \pi_k p_{k}(\mathbf{x}),
\end{align}
where
\begin{align}
p(\mathbf{x},z) = \prod_{k=1}^{K} \left(\pi_k ~p_{k}(\mathbf{x})\right)^{\mathbb{1} [z==k]}
\end{align}

### Mixture of Gaussians

The most wiely used mixture model is the GMM. In this model, each component distribution in the mixture is a multivariate Gaussian with mean $\mathbf{\mu}_k$ and covariance matrix $\mathbf{\Sigma}_k$:
\begin{align}
p_{k}(\mathbf{x}) = \sum_{k=1}^{K} \pi_k \mathcal{N}(\mathbf{x}|\mathbf{\mu}_k,\mathbf{\Sigma}_k)
\end{align}

Lets generate a toy dataset of samples coming from a GMM.

```python
import matplotlib.pyplot as plt
import numpy as np
from IPython.display import Image
import matplotlib.mlab as mlab
from mpl_toolkits.mplot3d import Axes3D
from scipy.stats import multivariate_normal as mvn

%matplotlib inline
```

```python
mus = np.array([[0,4], [-2,0], [6,-2]])
sigmas = np.array([   [[3, 0], [0, 0.5]], [[1,0],[0,2]], [[3,0],[0,1]] ])
pis = np.array([0.3, 0.4, 0.3])

# Lets plot the pdf contour plot
intervals = 200
x = np.linspace(-6, 12, intervals)
y = np.linspace(-6, 12, intervals)

X,Y = np.meshgrid(x,y)

xys = np.vstack([X.ravel(), Y.ravel()]).T

Z = np.zeros(len(xys))
for pi, mu, sigma in zip(pis, mus, sigmas):
    Z += pi*mvn(mu, sigma).pdf(xys)

Z = Z.reshape([intervals,intervals])
plt.contour(X, Y, Z, 20, cmap='RdGy')
plt.grid()
plt.title('A GMM contour plot')
plt.xlabel('x')
plt.ylabel('y')
```

```python
# And the pdf surfplace plot
fig = plt.figure(figsize=(12,8))
ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(X, Y, Z, cmap='RdGy')
plt.title('Gaussian Mxixture Model');
```

```python
# Create a dataset of samples of the above GMM

N = 250

X_samples= np.concatenate([np.random.multivariate_normal(mu, sigma, int(pi*N))
                    for pi, mu, sigma in zip(pis, mus, sigmas)])
```

```python
plt.scatter(X_samples[:,0],X_samples[:,1])
plt.grid()
plt.xlabel('x')
plt.ylabel('y')
```

## Learning a GMM from samples with the EM algorithm

Imagine we are given a dataset $\{\mathbf{x}^{(i)}\}$, $i=1,\ldots,N$, and we want to fit a GMM to explain this data set. For a given number $K$ of components, our goal is to find $\mathbf{\theta}\triangleq[\mathbf{\pi},\mathbf{\mu}_1,\ldots,\mathbf{\mu}_K,\mathbf{\Sigma}_1,\ldots,\mathbf{\Sigma}_K]$ to maximize the **log-likelihood of the observed data**:

\begin{align}
\ell(\mathbf{\theta})=\sum_{i=1}^{N} \log \left[\sum_{z^{(i)}} p(\mathbf{x}^{(i)},z^{(i)}|\mathbf{\theta}\right]
\end{align}

Unfortunately this is hard to optimize, since the log cannot be pushed inside the sum. The **Expectation Maximization (EM)** algorithm gets around this problem as follows. Define the **complete data log likelihood** to be

\begin{align}
\ell_c(\mathbf{\theta})=\sum_{i=1}^{N} \log  p(\mathbf{x}^{(i)},z^{(i)}|\mathbf{\theta})
\end{align}

$\ell_c(\mathbf{\theta})$ cannot be computed **since $z_i$ is unknown!**. Given our **current estimate** to $\mathbf{\theta}$, denoted by $\mathbf{\theta}_{t-1}$, we  define the **expected complete data log-likelihood** as follows:

\begin{align}
\mathcal{Q}(\mathbf{\theta},\mathbf{\theta}_{t-1})=\int_{\mathbf{z}} \ell_c(\mathbf{\theta}) ~~  p(\mathbf{z}|\mathbf{X},\mathbf{\theta}_{t-1}) = \sum_{i=1}^{N} \mathbb{E}_{z^{(i)}\sim p(z^{(i)}|\mathbf{x}^{(i)},\mathbf{\theta}_{t-1})}\left[\log p(\mathbf{x}^{(i)},z^{(i)})\right]
\end{align}

Computing $\mathcal{Q}(\mathbf{\theta},\mathbf{\theta}_{t-1})$ is referred as the **E-step** of the algorithm. In the **M-step**, we optimize the $\mathcal{Q}(\mathbf{\theta},\mathbf{\theta}_{t-1})$ w.r.t. $\mathbf{\theta}$:

\begin{align}
\mathbf{\theta}^t = \arg\max_{\mathbf{\theta}} \mathcal{Q}(\mathbf{\theta},\mathbf{\theta}_{t-1})
\end{align}

It can be proved that **the EM algorithm monotonically increases the log likelihood of the observed data** $\ell(\mathbf{\theta})$.

Lets visualize the [EM-implementation for GMMs provided by scitkit-learn](http://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html#sklearn.mixture.GaussianMixture), then we will particularize the above equations.

```python
#Plots
from sklearn import mixture

K = 3

gmm = mixture.GaussianMixture(n_components=K,covariance_type='full')

gmm.fit(X_samples)

# True means vs Estimated means (they can be in a different order!)
print('Component Mean comparison')
print(gmm.means_)
print(mus)
print('\n')

# True weights vs Estimated weights (they can be in a different order!)
print('Component weight comparison')
print(gmm.weights_)
print(pis)
print('\n')

# Loglikelihood
print('Train loglikelihood')
print(np.mean(gmm.score_samples(X_samples)))

gmm_means = gmm.means_
gmm_covs = gmm.covariances_
gmm_pis = gmm.weights_

Zgmm = np.zeros(len(xys))
for pi, mu, sigma in zip(gmm_pis, gmm_means, gmm_covs):
    Zgmm += pi*mvn(mu, sigma).pdf(xys)

Zgmm = Zgmm.reshape([intervals,intervals])
plt.contour(X, Y, Zgmm, 20, cmap='RdGy')
plt.scatter(X_samples[:,0],X_samples[:,1])
plt.grid()
plt.xlabel('x')
plt.ylabel('y')

```

## EM update equations for the GMM

Now that we have seen how to use the EM algorithm, let's take a closer look to the GMM EM updates.

### E-step

The expected complete data log likelihood is given by

\begin{align}
\mathcal{Q}(\mathbf{\theta},\mathbf{\theta}_{t-1})&= \sum_{i=1}^{N} \mathbb{E}_{z^{(i)}\sim p(z^{(i)}|\mathbf{x}^{(i)},\mathbf{\theta}_{t-1})}\left[\log p(\mathbf{x}^{(i)},z^{(i)})\right]=\sum_{i=1}^{N} \mathbb{E}_{z^{(i)}\sim p(z^{(i)}|\mathbf{x}^{(i)},\mathbf{\theta}_{t-1})}\left[\log \prod_{k=1}^{K} \left(\pi_k ~ \mathcal{N}(\mathbf{x}^{(i)}|\mathbf{\mu}_k,\mathbf{\Sigma}_k)\right)^{\mathbb{1} [z==k]} \right] \\\\
&= \sum_{i=1}^{N}\sum_{k=1}^K p(z^{(i)}=k|\mathbf{x}^{(i)},\mathbf{\theta}_{t-1}) \left[\log \pi_k +\log \mathcal{N}(\mathbf{x}^{(i)}|\mathbf{\mu}_k,\mathbf{\Sigma}_k)\right],
\end{align}
where, using Baye's rule,
\begin{align}
p(z^{(i)}=k|\mathbf{x}^{(i)},\mathbf{\theta}_{t-1}) \triangleq r_{ik} = \frac{\pi_{(k,t-1)} \mathcal{N}(\mathbf{x}^{(i)}|\mathbf{\mu}_{(k,t-1)},\mathbf{\Sigma}_{(k,t-1)})}{\sum_{q=1}^K \pi_{(q,t-1)} \mathcal{N}(\mathbf{x}^{(i)}|\mathbf{\mu}_{(q,t-1)},\mathbf{\Sigma}_{(q,t-1)})}
\end{align}

$r_{ik}$ is known as the responsibility that cluster $k$ takes for data point $i$ (given $\mathbf{\theta}_{t-1}$)

### M-step

In the M step, we optimize $\mathcal{Q}$ w.r.t. $\pi_k,\mathbf{\mu}_k,\mathbf{\Sigma}_k$, $k=1,\ldots,K$.

#### Optimization of $\pi_k$

It is easy to see that
\begin{align}
\pi_k = \frac{1}{N} \sum_{i=1}^N r_{ik}=\frac{r_k}{N},
\end{align}
where $r_k=\sum_{i=1}^N r_{ik}$ is the weighted number of points assigned to cluster $k$.

#### Optimization of $\mathbf{\mu}_k$, $\mathbf{\Sigma}_k$

We look at the parts of $\mathcal{Q}$ that depend on $\mathbf{\mu}_k$ and $\mathbf{\Sigma}_k$
\begin{align}
f(\mathbf{\mu}_k,\mathbf{\Sigma}_k) = \frac{-1}{2}\sum_{i=1}^N r_{ik} \left[\log |\mathbf{\Sigma}_k| + (\mathbf{x}^{(i)}-\mathbf{\mu}_k)^T\mathbf{\Sigma}^{-1}_k(\mathbf{x}^{(i)}-\mathbf{\mu}_k)\right]
\end{align}

and one can show that the new parameter estimates are given by (See Section 9.2.2 of Bishop's Book for details)

\begin{align}
\mathbf{\mu}_k &= \frac{\sum_{i=1}^N r_{ik}\mathbf{x}^{(i)}}{r_k} \\ \\
\mathbf{\mu}_k &= \frac{\sum_{i=1}^N r_{ik}\mathbf{x}^{(i)}(\mathbf{x}^{(i)})^T}{r_k}-\mathbf{\mu}_k\mathbf{\mu}_k^T
\end{align}

Thus, the EM for Gaussian Mixture Models is **very easy to implement!**.

## Model Comparison

An important question that typically arises when fitting a generative probabilistic model is how to select the right number of components. This problem is known as **model selection**.

A generative model is inherently a probability distribution for the dataset, and so we can simply evaluate the likelihood of the data under the model, **using cross-validation to avoid over-fitting.** Namely, we evaluate the model log-likelihood for a validation set not used for training.

Alternatively, the optimal Bayesian approach is to pick the model with the largest **marginal likelihood**:

\begin{align}
K^* = \arg\max_{K} p(\mathbf{X}|K) = \arg\max_{K} \int_{\mathbf{\theta}} p(\mathbf{X}|\mathbf{\theta},K) p(\mathbf{\theta}|K) d\mathbf{\theta}
\end{align}

In general, computing the marginal likelihood is quite difficult. Assuming a that prior distribution $p(\mathbf{\theta}|K)$ is Gaussian and very broad, we can approximate this term by the so-called **Bayessian Information Criterion** (BIC)
\begin{align}
-\log p(\mathbf{X}|K) \approx \frac{K\log N}{2}-p(\mathbf{X}|\mathbf{\theta}_{MAP},K)
\end{align}

where $\mathbf{\theta}_{MAP}$ is the mode of $p(\mathbf{\theta}|\mathbf{X},K)$, which is what we approximate with the EM (assuming again a very broad prior that does not weight much if the number of data points is large enough). **See Section 4.4.1. of Bishop's book for more details**.

Lets plot the BIC criterion for different values of $K$ in the above example

```python
K_list = range(1,10)
BIC_list = []

for k in K_list:
    gmm = mixture.GaussianMixture(n_components=k,covariance_type='full').fit(X_samples)
    BIC_list.append(gmm.bic(X_samples))

plt.plot(K_list,BIC_list)
plt.xlabel('Number of components')
plt.ylabel('BIC')
plt.grid()
```

Note that the minimum is attained at the right number of components

# GMM for a real dataset

Lets run the method for the [Wine Dataset](https://archive.ics.uci.edu/ml/datasets/wine). These data are the results of a chemical analysis of wines grown in the same region in Italy but derived from three different cultivars. The analysis determined the quantities of 13 constituents found in each of the **three types of wines**.

The attributes are explained [here](https://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.names)

```python
from numpy import genfromtxt

wine_data = genfromtxt('wine.csv', delimiter=',')

print('Dataset size is',wine_data.shape)

#Lets normalize
mean = np.mean(wine_data)
var = np.var(wine_data)

wine_data -= mean
wine_data /= var

```

    Dataset size is (6497, 13)

```python
K_list = range(1,10)
BIC_list = []

for k in K_list:
    gmm = mixture.GaussianMixture(n_components=k,covariance_type='full').fit(wine_data)
    BIC_list.append(gmm.bic(wine_data))

plt.plot(K_list,BIC_list)
plt.xlabel('Number of components')
plt.ylabel('BIC')
plt.grid()
```

```python

```

Notebooks/Session 12 GMMs and EM/.ipynb_checkpoints/Session_12_GMMs_and_the_EM_algorithm-checkpoint.ipynb


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Class 12: Introduction to the `linearsolve` Module

In general, dynamic stochastic general equilibrium (DSGE) models do not admit analytic (i.e., pencil-and-paper) solutions and they are time-consuming to work with. The `linearsolve` module approximates, solves, and simulates DSGE models and therefore makes DSGE models easier to use.

## Installing `linearsolve`

`linearsolve` is not included in the Anaconda Python installation and so before you can import it, you need to download and install the `linearsolve` package from PyPI, the Python Package Index. In Windows, open the Anaconda Prompt and in Mac, open the Terminal and run the following commmand:

    pip install linearsolve

You only have to install the package once.

```python
# Import the linearsolve under the 'ls' namespace
import linearsolve as ls
```

## Example: A One-Equation Model of TFP

Consider the following AR(1) specification for $\log$ TFP:

\begin{align}
\log A_{t+1} & = \rho \log A_t + \epsilon_{t+1},
\end{align}

where $\epsilon_{t+1} \sim \mathcal{N}(0,\sigma^2)$. The goal is to simulate

\begin{align}
\log (A_{t}/\bar{A}) & \approx \frac{A_t - \bar{A}}{\bar{A}},
\end{align}

where $\bar{A}$ is the nonstochastic (i.e., $\epsilon_t = 0$) steady state value of $A_t$. Let's compute the model simulation with `linearsolve`. To do this we need to do several things:

1. Create a Pandas series that stores the names of the parameters of the model.
2. Define a function that returns the equilibrium conditions of the model solved for zero.
3. Initialize an instance of the `linearsolve.model` class
4. Compute and input the steady state of the model.
5. Approximate and solve the model.
6. Compute simulations of the model.

Use the following values for the simulation:

| $\rho$ | $\sigma$ |
|--------|----------|
| 0.95   | 0.01    |

**Step 1:** Create a variable called `parameters` that stores parameter values as a Pandas Series.

```python
# Create a variable called 'parameters' that stores the model parameter values in a Pandas Series
parameters = pd.Series()
parameters['rho'] = .95

# Print stored parameter values
print(parameters)
```

    rho    0.95
    dtype: float64

```python
# Create a variable called 'sigma' that stores the value of sigma
sigma = 0.01**0.5
```

```python
# Create variable called 'var_names' that stores the variable names in a list with state variables ordered first
var_names = ['a']

# Create variable called 'shock_names' that stores an exogenous shock name for each state variable.
shock_names = ['e_a']
```

**Step 2:** Define a function called `equilibrium_equations` that evaluates the equilibrium
equations of model solved for zero. The function should accept three arguments:

1. `variables_forward`: Values of $t+1$-dated variables
2. `variables_current`: Values of $t$-dated variables
3. `parameters`: Pandas Series with the parameters for the model

The function should return a NumPy array of the model's equilibrium conditions solved for zero.

```python
# Define a function that evaluates the equilibrium conditions of the model solved for zero. PROVIDED
def equilibrium_equations(variables_forward,variables_current,parameters):

    # Create a variable called 'p' that storres the model's parameters. PROVIDED.
    p = parameters

    # Create variable called 'cur' that stores the values of current (date t) variables. PROVIDED.
    cur = variables_current

    # Create variable called 'fwd' that stores the values of one-period-ahead (date t+1) variables. PROVIDED.
    fwd = variables_forward

    # Create variable called 'tfp_proc' that returns the law of motion for TFP solved for zero (exogenous shock excluded)
    tfp_proc = p.rho*np.log(cur.a) - np.log(fwd.a)

    # Return equilibrium conditions stacked in a numpy array. PROVIDED.
    return np.array([
        tfp_proc
        ])
```

**Step 3:** Initialize a model instance using the `ls.model()` function. The

1. `equations`: Name of function that stores the equilibrium conditions
2. `n_states`: Number of *state* variables (i.e., variables that are *predetermined*)
3. `var_names`: List of the names of the endogenous variables
4. `shock_names`: List of the names of the exogenous shocks
5. `parameters`: Pandas Series of parameter values

```python
# Initialize the model into a variable named 'ar1_model' using the ls.model() function
ar1_model = ls.model(equations = equilibrium_equations,
                     n_states=1,
                     var_names=var_names,
                     shock_names=shock_names,
                     parameters = parameters)
```

**Step 4:** Set the steady state of the model. Either use the `.compute_ss()` method which requires an initial guess of what the steady state is. Or set the steady state `.ss` attribute directly.

```python
# Compute the steady state numerically using .compute_ss() method of ar1_model
ar1_model.compute_ss([1])
print(ar1_model.ss)
```

    a    1.0
    dtype: float64

```python
# Set the .ss attribute of ar1_model directly
ar1_model.ss = pd.Series({'a':1})
```

**Step 5:** Compute a transform the model into a log-linear approximation around the nonstochastic steady state. Then rewrite the equilibrium conditions so that all endogenous variables are expressed as linear functions of state variables and exogenous shocks.

```python
# Appproximate and solve using the .approximate_and_solve() method of ar1_model
ar1_model.approximate_and_solve()
```

**Step 6:** Simulate the model using one of the following methods:

1. `.impulse()`: Compute impulse responses to a one-time shock to each exogenous variables. Results stored in `.irs` attribute.
2. `.stoch_sim()`: Compute stochastic simulation. Results stored in `.simulated` attribute.

First, we'll compute an impulse response simulation. Let's consider the effect of a one time shock to $\epsilon$ of 0.01 in period 5. Simulate 41 periods.

```python
# Compute impulse response of a to a one-time shock to e_a
ar1_model.impulse(T=41,t0=5,shocks=[0.01])
```

The impulse response simulations are stored in the `.irs` attribute as a dictionary with keys equal to the names of the exogenous shocks.

```python
# Print ar1_model.irs
print(ar1_model.irs)
```

    {'e_a':      e_a         a
    0   0.00  0.000000
    1   0.00  0.000000
    2   0.00  0.000000
    3   0.00  0.000000
    4   0.00  0.000000
    5   0.01  0.010000
    6   0.00  0.009500
    7   0.00  0.009025
    8   0.00  0.008574
    9   0.00  0.008145
    10  0.00  0.007738
    11  0.00  0.007351
    12  0.00  0.006983
    13  0.00  0.006634
    14  0.00  0.006302
    15  0.00  0.005987
    16  0.00  0.005688
    17  0.00  0.005404
    18  0.00  0.005133
    19  0.00  0.004877
    20  0.00  0.004633
    21  0.00  0.004401
    22  0.00  0.004181
    23  0.00  0.003972
    24  0.00  0.003774
    25  0.00  0.003585
    26  0.00  0.003406
    27  0.00  0.003235
    28  0.00  0.003074
    29  0.00  0.002920
    30  0.00  0.002774
    31  0.00  0.002635
    32  0.00  0.002503
    33  0.00  0.002378
    34  0.00  0.002259
    35  0.00  0.002146
    36  0.00  0.002039
    37  0.00  0.001937
    38  0.00  0.001840
    39  0.00  0.001748
    40  0.00  0.001661}

Let's look at the first 10 rows of the `'e_a'` element of `ar1_model.irs`.

```python
# Print first 10 rows of the element in ar1_model.irs that corresponds to the shock to TFP
print(ar1_model.irs['e_a'].head())
```

       e_a    a
    0  0.0  0.0
    1  0.0  0.0
    2  0.0  0.0
    3  0.0  0.0
    4  0.0  0.0

```python
# Plot simulated impulse response to e_a
ar1_model.irs['e_a'][['e_a','a']].plot(lw=3,alpha=0.5,grid=True).legend(loc='upper right',ncol=2)
```

Next, we'll use the `.stoch_sim()` method to compute a stochastic simulation. The method takes arguments:

1. `seed`: Seed of NumPy RNG. (Optional)
2. `T`: Number of periods to simulate
3. `covMat`: Covariance matrix for exogenous shock process

The simulation should be for 201 periods.

```python
# Compute stochastic simulation
ar1_model.stoch_sim(seed=126,T=201,cov_mat= [[sigma**2]])

# Print the first 10 rows of `model.simulated`
print(ar1_model.simulated.head(10))
```

            e_a         a
    0 -0.060064  0.033436
    1  0.076834  0.108598
    2 -0.065005  0.038163
    3 -0.004405  0.031850
    4 -0.008192  0.022066
    5  0.030705  0.051668
    6  0.062828  0.111912
    7  0.014509  0.120825
    8  0.029805  0.144589
    9 -0.085171  0.052188

The stochastic simulations are stored in the `.stoch_sim` attribute as a Pandas `DataFrame`.

```python
# Plot the stochastic simulation
ar1_model.simulated[['e_a','a']].plot(lw=3,alpha=0.5,grid=True).legend(loc='upper right',ncol=2)
```

## Example 2: The Stochastic Solow Growth Model Revisited

Now consider the following system of equations:

\begin{align}
Y_t & = A_t K_t^{\alpha} \\
K_{t+1} & = sY_t + (1-\delta) K_t\\
\log A_{t+1} & = \rho \log A_t + \epsilon_{t+1}
\end{align}

where $\epsilon_{t+1} \sim \mathcal{N}(0,\sigma^2)$. Let's simulate the model with `linearsolve`. Before proceding, let's also go ahead and rewrite the model with all variables moved to the lefthand side of the equations:

\begin{align}
0 & = A_t K_t^{\alpha} - Y_t \\
0 & = sY_t + (1-\delta) K_t - K_{t+1}\\
0 & = \rho \log A_t + \epsilon_{t+1} - \log A_{t+1}
\end{align}

Capital and TFP are called *state variables* because they're $t+1$ values are predetermined. Output is called a *costate* or *control* variable. Note that the model as 3 endogenous variables with 2 state variables.

Use the following values for the simulation:

| $\rho$ | $\sigma$ | $s$ | $\alpha$ | $\delta $ | $T$  |
|--------|----------|-----|----------|-----------|------|
| 0.75   | 0.006    | 0.1 | 0.35     |  0.025    | 201  |

### Initialization, Approximation, and Solution

The next several cells initialize the model in `linearsolve` and then approximate and solve it.

```python
# Create a variable called 'parameters' that stores the model parameter values in a Pandas Series
parameters = pd.Series()
parameters['rho'] = .75
parameters['alpha'] = 0.35
parameters['delta'] = 0.025
parameters['s'] = 0.1

# Print the model's parameters
print(parameters)
```

    rho      0.750
    alpha    0.350
    delta    0.025
    s        0.100
    dtype: float64

```python
# Create a variable called 'sigma' that stores the value of sigma
sigma = 0.006
```

```python
# Create variable called 'var_names' that stores the variable names in a list with state variables ordered first
var_names = ['a','k','y']

# Create variable called 'shock_names' that stores an exogenous shock name for each state variable.
shock_names = ['e_a','e_k']
```

```python
# Define a function that evaluates the equilibrium conditions of the model solved for zero. PROVIDED
def equilibrium_equations(variables_forward,variables_current,parameters):

    # Parameters. PROVIDED
    p = parameters

    # Current variables. PROVIDED
    cur = variables_current

    # Forward variables. PROVIDED
    fwd = variables_forward

    # Production function
    production_function = cur.a*cur.k**p.alpha - cur.y

    # Capital evolution
    capital_evolution = p.s*cur.a*cur.k**p.alpha + (1 - p.delta)*cur.k - fwd.k

    # Exogenous tfp
    tfp_process = p.rho*np.log(cur.a) - np.log(fwd.a)

    # Stack equilibrium conditions into a numpy array
    return np.array([
        production_function,
        capital_evolution,
        tfp_process
        ])
```

Next, initialize the model.

```python
# Initialize the model into a variable named 'solow_model'
solow_model = ls.model(equations = equilibrium_equations,
                 n_states=2,
                 var_names=var_names,
                 shock_names=shock_names,
                 parameters = parameters)
```

```python
# Compute the steady state numerically using .compute_ss() method of solow_model
guess = [1,4,1]
solow_model.compute_ss(guess)

# Print the computed steady state
print(solow_model.ss)
```

    a    1.000000
    k    8.438129
    y    2.109532
    dtype: float64

```python
# Find the log-linear approximation around the non-stochastic steady state and solve using .approximate_and_solve() method of solow_model
solow_model.approximate_and_solve()
```

### A Few Details About the Approximation (Optional)

The previous step constructs a log-linear approximation of the model and then solves for the endogenous variables as functions of the state variables and exogenous shocks only.

View the approximated model by calling the `.approximated()` method.

```python
# Print the log-linear approximation to the models's equilibrium conditions
print(solow_model.approximated())
```

    Log-linear equilibrium conditions:

                 0 = -2.1095·a[t]-0.7383·k[t]+2.1095·y[t]

    -8.4381·k[t+1] = -0.211·a[t]-8.301·k[t]

           -a[t+1] = -0.75·a[t]

Each variable represents the log-deviation from the steady state of the respective variable in our model. For example, the variable `y[t]` means $\log(Y_t) - \log(Y)$ in terms of the stochastic Solow model. But how do these equations relate to the original model?

The first equation appears to be:

\begin{align}
0 &= -2.1095\cdot a_t-0.7383\cdot k_t + 2.1095\cdot y_t
\end{align}

Note that dividing by 2.1095 and solving for $y_t$ yields:

\begin{align}
y_t &= a_t +0.3499\cdot k_t,
\end{align}

so the coefficient on $k_t$ appears to be close to $\alpha=0.35$. We can derive this linear equation directly. First, start with the production function:

\begin{align}
Y_t &= A_t K_t^{\alpha}.
\end{align}

Then divide both sides by steady state output:

\begin{align}
\frac{Y_t}{Y} &= \frac{A_t K_t^{\alpha}}{AK^{\alpha}} \, = \, \frac{A_t}{A}\frac{K_t^{\alpha}}{K^{\alpha}}.
\end{align}

Then, take the log of both sides:
\begin{align}
\log\left(\frac{Y_t}{Y} \right)&= \log\left(\frac{A_t}{A}\right) + \alpha\log\left(\frac{K_t}{K}\right).
\end{align}

finally, letting $y_t = \log(Y_t/Y)$, k_t = $\log(K_t/K)$, and $a_t = \log(A_t/A)$, we have:

\begin{align}
y_t &= a_t + \alpha k_t.
\end{align}

However, understanding this process isn't as important as being able to interpret the graphs and statistics that compute using the output of `linearsolve`.

### A Few Details About the Solution (Optional)

It's also worth seeing what it means for a model to be *solved*. After `linearsolve` computes the log-linear approximation to the model, it solves for each endogenous variable as a function of state variables only. View the solved model by calling the `.solved()` method.

```python
# Print the solved model
print(solow_model.solved())
```

    Solution to the log-linear system:

    a[t+1] = 0.75·a[t]+e_a[t+1]

    k[t+1] = 0.025·a[t]+0.9837·k[t]+e_k[t+1]

      y[t] = a[t]+0.35·k[t]

### Impulse Responses

Compute a 41 period impulse responses of the model's variables to a 0.01 unit shock to TFP in period 5.

```python
# Compute impulse responses
solow_model.impulse(T=41,t0=5,shocks=[0.01,0])

# Print the first 10 rows of the computed impulse responses.
print(solow_model.irs['e_a'].head(10))
```

        e_a         a         k         y
    0  0.00  0.000000  0.000000  0.000000
    1  0.00  0.000000  0.000000  0.000000
    2  0.00  0.000000  0.000000  0.000000
    3  0.00  0.000000  0.000000  0.000000
    4  0.00  0.000000  0.000000  0.000000
    5  0.01  0.010000  0.000000  0.010000
    6  0.00  0.007500  0.000250  0.007588
    7  0.00  0.005625  0.000433  0.005777
    8  0.00  0.004219  0.000567  0.004417
    9  0.00  0.003164  0.000663  0.003396

```python
# Plot the computed impulse responses to a TFP shock
fig = plt.figure(figsize=(12,4))

ax1 = fig.add_subplot(1,2,1)
solow_model.irs['e_a'][['a','y','k']].plot(lw=3,alpha=0.5,grid=True,ax = ax1).legend(loc='upper right',ncol=2)
ax1.set_title('Endogenous Variables')

ax2 = fig.add_subplot(1,2,2)
solow_model.irs['e_a'][['e_a']].plot(lw=3,alpha=0.5,grid=True,ax = ax2).legend(loc='upper right',ncol=2)
ax2.set_title('TFP Shock Process')

fig.tight_layout()
```

### Stochastic Simulation

Compute a 201 period stochastic simulation of the model's variables. Set the variance of $\epsilon_t$ to $\sigma^2$ and the variance of the shock to capital to 0 so that the covariance matrix for the shock process is:

\begin{align}
\text{Covariance matrix} & = \left[\begin{array}{cc} \sigma^2 & 0\\ 0 & 0\end{array} \right]
\end{align}

```python
# Compute stochastic simulation and print the simulated values.
solow_model.stoch_sim(seed=126,T=201,cov_mat= [[sigma**2,0],[0,0]])

# Print first 10 rows of model.simulated
print(solow_model.simulated.head(10))
```

            e_a  e_k         a         k         y
    0  0.001484  0.0  0.003862  0.000649  0.004089
    1  0.001427  0.0  0.004323  0.000735  0.004580
    2  0.003155  0.0  0.006397  0.000831  0.006688
    3  0.011305  0.0  0.016103  0.000978  0.016445
    4 -0.001778  0.0  0.010299  0.001364  0.010777
    5 -0.013211  0.0 -0.005486  0.001600 -0.004927
    6  0.004054  0.0 -0.000060  0.001437  0.000442
    7  0.006153  0.0  0.006108  0.001412  0.006602
    8 -0.001336  0.0  0.003245  0.001541  0.003785
    9 -0.006801  0.0 -0.004367  0.001598 -0.003808

```python
# Plot the computed stochastic simulation
fig = plt.figure(figsize=(12,4))

ax1 = fig.add_subplot(1,2,1)
solow_model.simulated[['a','y','k']].plot(lw=3,alpha=0.5,grid=True,ax = ax1).legend(loc='upper right',ncol=3)
ax1.set_title('Endogenous Variables')

ax2 = fig.add_subplot(1,2,2)
solow_model.simulated[['e_a']].plot(lw=3,alpha=0.5,grid=True,ax = ax2).legend(loc='upper right',ncol=2)
ax2.set_title('TFP Shock Process')

fig.tight_layout()
```

```python
# Compute standard deviations of simulated TFP, output, and capital
solow_model.simulated[['a','y','k']].std()
```

    a    0.009478
    y    0.009484
    k    0.001385
    dtype: float64

```python
# Compute correlation coefficients of simulated TFP, output, and capital
solow_model.simulated[['a','y','k']].corr()
```

      a
      y
      k

      a
      1.000000
      0.998694
      -0.012365

      y
      0.998694
      1.000000
      0.038745

      k
      -0.012365
      0.038745
      1.000000


```python
%load_ext autoreload
%autoreload 2
import numpy as np
import scipy

import matplotlib.pyplot as plt
import matplotlib.ticker as ticker
import pandas as pd
import seaborn as sns
from pylab import cm

import os
import sys
import re
import sympy

import cmath
import matplotlib.patches as patches
from matplotlib import colors as m2colors

import plotly.graph_objects as go

from MMfxns import *

cwd = os.getcwd()
sns.set(style="ticks", font_scale=1.5)

mcolors = dict(m2colors.BASE_COLORS, **m2colors.CSS4_COLORS)

saveall = True
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

```python
## SI Figure 1
### plot the turnover and self reinforcement terms here w.r.t. x

res = 100
tau = 2
alpha = 1.
n = 3
x_space = np.linspace(0,2.5,res)
turnover = -x_space/tau
reinforce = alpha * x_space ** n / (x_space **n + 1)

fig, ax = plt.subplots(1,1)
ax2 = ax.twinx()
ax.plot(x_space, turnover, color='r')
ax.set_ylabel('turnover')
ax.yaxis.label.set_color('r')
ax.tick_params(axis='y', colors='r')

ax2.plot(x_space, reinforce, color='g')
ax2.set_ylabel('reinforcement')
ax2.yaxis.label.set_color('g')
ax2.tick_params(axis='y', colors='g')
# ax2.spines['right']
ax.set_xlabel('x')
```

```python
## SI fig 2?

from scipy.signal import argrelextrema

res = 500
m_space = np.linspace(.1, 1.2, res)
a_space = np.linspace(0.5, 1.5, int(res/1))
x_space = np.linspace(0, 3, res)

m0 = 1.
tau = 1.

x_c = x_crit(n)
a_c = alpha_crit(n, tau)
m_c = m_crit_over_m0(n, tau)

print('a_c = %f, x_c = %f, m_c = %f' % (a_c, x_c, m_c))

Uma = np.zeros((len(m_space),len(a_space)))
nummins = np.zeros((len(m_space),len(a_space)))

x_guess = np.linspace(np.amin(x_space), np.amax(x_space), 15)

for ai, aa in enumerate(a_space):
    for mi, mm in enumerate(m_space):
        xoutlist = []
        for xi, xx in enumerate(x_guess):

            xout = scipy.optimize.fsolve(x_equil, xx, args=(mm, m0, tau, aa, 3), xtol=1e-10)[0]
            xout = np.round(xout, 6)
            xoutlist.append(xout)

        xoutlist = np.unique(xoutlist)
        x_stiff = 1-np.exp(-mm/m0)
        Uma[mi,ai] = U(x_stiff, xoutlist, aa, tau, 3)

print(Uma)
# lenmins = []
# for ai, aa in enumerate(a_space):
#     for mi, mm in enumerate(m_space):
#         Uvec = U(x_stiff, x_space, aa, tau, 3)
#         mins = Uvec[argrelextrema(np.diff(Uvec), np.less)[0]]
#         if len(mins) not in lenmins:
#             lenmins.append(len(mins))

# print(lenmins)
#         nummins[mi,ai]
mm, aa = np.meshgrid(m_space,a_space)

fig, ax = plt.subplots(1,1,figsize=(6,6))

cf = ax.contourf(aa, mm, Uma, levels=100, cmap=cm.hot)
cb = plt.colorbar(cf, ax=ax, format='%.2f')
ax.scatter(a_c, m_c)
```

```python

a_space = np.linspace(0,2,100)

m1list = [m1n3(ai, 3) for ai in a_space]
m2list = [m2n3(ai, 3) for ai in a_space]
# plt.plot(a_space, m1list)
# plt.plot(a_space, m2list)

plt.plot(np.arange(len(deltaVlist)), deltaVlist)
plt.show()
```


# 2.5 Local Methods in High Dimensions

**Mean squared error for estimating f(0):**

*Assume that the relationship between X and Y is: $Y = f(X)$*

$$
\begin{align}
 \text{MSE}(x_0) & = E_\tau[f(x_0) - \hat{y_0}]^2\\
 & = E_\tau[(f(x_0) - E_\tau(\hat{y_0})) + (E_\tau(\hat{y_0}) - \hat{y_0})]^2\\
 & = E_\tau[(E_\tau(\hat{y_0}) - \hat{y_0})^2  + 2(f(x_0) - E_\tau(\hat{y_0}))(E_\tau(\hat{y_0}) - \hat{y_0})+ (f(x_0) - E_\tau(\hat{y_0}))^2]\\
 & = E_\tau[(E_\tau(\hat{y_0}) - \hat{y_0})^2] + E_\tau[(E_\tau(\hat{y_0}) - f(x_0))^2]\\
 & = E_\tau[\hat{y_0} - E_\tau(\hat{y_0})]^2 + [E_\tau(\hat{y_0}) - f(x_0)]^2\\
 & = Var_\tau(\hat{y_0}) + Bias^2(\hat{y_0})
\end{align}
$$

We have broken down the MSE into two components: variance and squared bias. Such decomposition is always possible and is known as *the bias-variance decomposition*.

*(2.26) Suppose that the relationship between Y and X is linear with some noise:*

$$Y = X^T\beta + \varepsilon $$

where $\varepsilon \sim N(0, \sigma^2)$ and we fit the model by least squares to the training data. For a test point $x_0$ we have $\hat{y_0}=x_0^T\hat{\beta}$ which can be written as $\hat{y_0} = x_0^T\beta + \sum_{i=1}^N {l_i(x_0)\varepsilon_i}$ where $l_i(x_0)$ is the $i$th element of $\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}x_0$

Proof:

$$
\begin{equation}
 \hat{\beta}=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y} \\
 \hat{\beta}=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T(\mathbf{X}\beta + \varepsilon) \\
 \hat{\beta}=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T(\mathbf{X}\beta + \varepsilon) \\
 \hat{\beta}=\beta + (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\varepsilon \\
\end{equation}
$$

and by plugging $\hat{B}$ into the linear model:

$$
\begin{equation}
 \hat{y_0} = x_0^T(\beta+(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\varepsilon) \\
 \hat{y_0} = x_0^T\beta+x_0^T(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\varepsilon
\end{equation}
$$

we can get $\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}x_0$ from $(x_0^T(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T)^T$ by using two matrix properties:

1. $(\mathbf{AB})^T=\mathbf{B}^T\mathbf{A}^T$

2. $(\mathbf{A}^{-1})^T = (\mathbf{A}^T)^{-1}$

Under this model the least square estimates are unbiased, so the expected prediction error will be:
$$
\begin{align}
\text{EPE}(x_0) & = E_{y_0|x_0}E_\tau(y_0-\hat{y_0})^2\\
& = \text{Var}(y_0|x_0) + Var_\tau(\hat{y_0}) + \text{Bias}^2(\hat{y_0})\\
& = \sigma^2 + E_{\tau}x_0^T(\mathbf{X}^T\mathbf{X})^{-1}x_0\sigma^2 + 0^2
\end{align}
$$

*Proof*:
$$
\begin{align}
\text{EPE}(x_0) & = E_{y_0|x_0}E_\tau(y_0-\hat{y_0})^2\\
& = E_{y_0|x_0}E_\tau((y_0 - f(x_0)) + (f(x_0) - \hat{y_0}))^2\\
& = E_{y_0|x_0}E_\tau(y_0 - f(x_0))^2 + 2E_{y_0|x_0}E_\tau(y_0 - f(x_0))(f(x_0) - \hat{y_0}) + E_{y_0|x_0}E_\tau(f(x_0) - \hat{y_0})^2\\
& = U_1 + U_2 + U_3
\end{align}
$$

There are three components $U_1$, $U_2$, $U_3$ and we're going to expand them as well.

$U_1 = E_{y_0|x_0}E_\tau(y_0 - f(x_0))^2 = E_{y_0|x_0}(y_0-f(x_0))^2 = \sigma^2$

*Note*: $f(x_0) = E_{y_0|x_0}(y_0) $

$U_2 = 2E_{y_0|x_0}E_\tau(y_0 - f(x_0))(f(x_0) - \hat{y_0}) = 0$

*Note*: $E_{y_0|x_0}(y_0-f(x_0)) = 0$

*$U_3$*:

$$
\begin{align}
U_3 & = E_{y_0|x_0}E_\tau(f(x_0) - \hat{y_0})^2\\
& = E_{y_0|x_0}E_\tau((\hat{y_0} - E_\tau(\hat{y_0})) + (E_\tau(\hat{y_0}) - f(x_0)))^2\\
& = E_{y_0|x_0}E_\tau(\hat{y_0} - E_\tau(\hat{y_0}))^2 + 2E_{y_0|x_0}E_\tau[(\hat{y_0} - E_\tau(\hat{y_0}))(E_\tau(\hat{y_0}) - f(x_0))] + E_{y_0|x_0}E_\tau(E_\tau(\hat{y_0}) - f(x_0))^2\\
& = E_\tau(\hat{y_0} - E_\tau(\hat{y_0}))^2 + (E_\tau(\hat{y_0}) - f(x_0))^2\\
& = \text{Var}_\tau(\hat{y_0}) + \text{Bias}_\tau^2(\hat{y_0})
\end{align}
$$

Finally if we sum all $U_i$ we get: $$\text{EPE}(x_0) = U_1+U_2+U_3 = \sigma^2 + 0 + (\text{Var}_\tau(\hat{y_0}) + \text{Bias}_\tau^2(\hat{y_0}))$$

$E_\tau(\hat{y_0}) = E_\tau(x_0^T\beta + \sum_{i=1}^N {l_i(x_0)\varepsilon_i})=x_0^T\beta + E(\sum_{i=1}^N {l_i(x_0)\varepsilon_i}) = x_0^T\beta + 0$ thus $\text{Bias}_\tau{\hat{y_0}} = 0$

(2.27) and we can find variance:
$$
\begin{align}
\text{Var}_\tau(\hat{y_0}) & = E_\tau(\hat{y_0} - E_\tau(\hat{y_0})) ^ 2\\
& = E_\tau(x_0^T(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\varepsilon)\\
& = E_\tau(x_0^T(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\varepsilon\varepsilon^T\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}x_0)
\end{align}
$$

where $\varepsilon\varepsilon^T=\sigma^2\mathbf{I}_n$, so we can simplify further:
$$\text{Var}_\tau(\hat{y_0}) = \sigma^2x_0^{T}E_\tau[(\mathbf{X}^T\mathbf{X})^{-1})]x_0$$

(2.28) if N is large and $\tau$ were selected at random, and assuming E(X) = 0, then $\mathbf{X}^T\mathbf{X}$->$NCov(\mathbf{X})$.

*Proof*: By definition of covariance $\text{Cov}(X) = E[(X-E(X))(X-E(X))^T] = E(XX^T) = \frac{\mathbf{X}^T\mathbf{X}}{N}$

and we can derive that:
$$
\begin{align}
E_{x_0}\text{EPE}(x_0) & = E_{x_0}x_0^{T}\text{Cov}^{-1}(X)x_0\sigma^2/N+\sigma^2\\
& = \text{trace}[\text{Cov}^{-1}(X)\text{Cov}(x_0)]\sigma^2/N+\sigma^2\\
& = \sigma^2(p/N)+\sigma^2
\end{align}
$$

**FIGURE 2.9**

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

simulations = 10000
sample_size = 500

def least_square_error(x_0, y_0, train_x, train_y):
    X = np.c_[np.ones((len(train_x), 1)), train_x]
    beta = np.linalg.lstsq(X, train_y, rcond = None)[0]
    return (np.dot(np.array([1, *x_0]), beta) - y_0) ** 2

# 1-nearest neighbor error
def nn_error(x_0, y_0, train_x, train_y):
    X = (train_x * train_x).sum(axis=1)
    return (y_0 - train_y[X.argmin()]) ** 2

cubic_epe_ratio = []
linear_epe_ratio = []

for p in range(1, 11):
    least_square_epe = [0, 0]
    nn_epe = [0, 0]
    for _ in range(simulations):
        error = np.random.standard_normal(sample_size)
        train_x = np.random.uniform(-1, 1, size=(sample_size, p))
        train_y = [train_x[:, 0] + error,
                   0.5 * (train_x[:, 0] + 1) ** 3 + error]
        x_0 = np.zeros(p)
        y_0 = [np.random.standard_normal(),
               0.5 + np.random.standard_normal()]
        for i in range(2):
            least_square_epe[i] += least_square_error(x_0, y_0[i],
                                                      train_x, train_y[i])
            nn_epe[i] += nn_error(x_0, y_0[i],
                                  train_x, train_y[i])
    for i in range(2):
        least_square_epe[i] /= simulations
        nn_epe[i] /= simulations
    linear_epe_ratio.append(nn_epe[0] / least_square_epe[0])
    cubic_epe_ratio.append(nn_epe[1] / least_square_epe[1])

# plot
fig = plt.figure(1, figsize = (8, 8))
axes = fig.add_subplot(1, 1, 1)

axes.set_title("Expected Prediction Error of 1NN vs. OLS")

axes.plot(np.arange(1, 11), linear_epe_ratio, '-o',
          color = 'orange', label = "Linear")

axes.plot(np.arange(1, 11), cubic_epe_ratio, '-o',
          color = 'blue', label = "Cubic")

axes.legend()
axes.set_xlabel("Dimension")
axes.set_ylabel("EPE Ration")
plt.show()

```