## A Demonstration of the Harmenberg (2021) Aggregation Method

   - ["Aggregating heterogeneous-agent models with permanent income shocks"](https://doi.org/10.1016/j.jedc.2021.104185)

## Authors: [Christopher D. Carroll](http://www.econ2.jhu.edu/people/ccarroll/), [Mateo Velásquez-Giraldo](https://mv77.github.io/)

`# Set Up the Computational Environment: (in JupyterLab, click the dots)`

```python
# Preliminaries
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
from copy import copy, deepcopy

from HARK.distribution import calc_expectation
from HARK.ConsumptionSaving.ConsIndShockModel import (
    IndShockConsumerType,init_idiosyncratic_shocks
)
```

# Description of the problem

$\newcommand{\pLvl}{\mathbf{p}}$
$\newcommand{\mLvl}{\mathbf{m}}$
$\newcommand{\mNrm}{m}$
$\newcommand{\CLvl}{\mathbf{C}}$
$\newcommand{\MLvl}{\mathbf{M}}$
$\newcommand{\CLvlest}{\widehat{\CLvl}}$
$\newcommand{\MLvlest}{\widehat{\MLvl}}$
$\newcommand{\mpLvlDstn}{\mu}$
$\newcommand{\mWgtDstnMarg}{\tilde{\mu}^{m}}$
$\newcommand{\PermGroFac}{\pmb{\Phi}}$
$\newcommand{\PermShk}{\pmb{\Psi}}$
$\newcommand{\def}{:=}$
$\newcommand{\kernel}{\Lambda}$
$\newcommand{\pShkNeutDstn}{\tilde{f}_{\PermShk}}$
$\newcommand{\Ex}{\mathbb{E}}$
$\newcommand{\cFunc}{\mathrm{c}}$
$\newcommand{\Rfree}{\mathsf{R}}$

Macroeconomic models with heterogeneous agents sometimes incorporate a microeconomic income process with a permanent component ($\pLvl_t$) that follows a geometric random walk. To find an aggregate characteristic of these economies such as aggregate consumption $\CLvl_t$, one must integrate over permanent income (and all the other relevant state variables):
\begin{equation*}
\CLvl_t = \int_{\pLvl} \int_{\mLvl} \mathrm{c}(\mLvl,\pLvl) \times f_t(\mLvl,\pLvl) \, d \mLvl\, d\pLvl,
\end{equation*}
where $\mLvl$ denotes any other state variables that consumption might depend on, $\cFunc(\cdot,\cdot)$ is the individual consumption function, and $f_t(\cdot,\cdot)$ is the joint density function of permanent income and the other state variables at time $t$.

Under the usual assumption of Constant Relative Risk Aversion utility and standard assumptions about the budget constraint, [such models are homothetic](https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#The-Problem-Can-Be-Normalized-By-Permanent-Income). This means that for a state variable $\mLvl$ one can solve for a normalized policy function $\cFunc(\cdot)$ such that
\begin{equation*}
    \mathrm{c}(\mLvl,\pLvl) = \mathrm{c}\left(\mLvl/\pLvl\right)\times \pLvl
\end{equation*}

In practice, this implies that one can defined a normalized state vector $\mNrm = \mLvl/\pLvl$ and solve for the normalized policy function. This eliminates one dimension of the optimization problem problem, $\pLvl$.

While convenient for the solution of the agents' optimization problem, homotheticity has not simplified our aggregation calculations as we still have

\begin{equation*}
\begin{split}
\CLvl_t =& \int \int \cFunc(\mLvl,\pLvl) \times f_t(\mLvl,\pLvl) \, d\mLvl\, d\pLvl\\
=& \int \int \cFunc\left(\frac{1}{\pLvl}\times \mLvl\right)\times \pLvl \times f_t(\mLvl,\pLvl) \, d\mLvl\, d\pLvl,
\end{split}
\end{equation*}

which depends on $\pLvl$.

To further complicate matters, we usually do not have analytical expressions for $\cFunc(\cdot)$ or $f_t(\mLvl,\pLvl)$. What we often do in practice is to simulate a population $I$ of agents for a large number of periods $T$ using the model's policy functions and transition equations. The result is a set of observations $\{\mLvl_{i,t},\pLvl_{i,t}\}_{i\in I, 0\leq t\leq T}$ which we then use to approximate
\begin{equation*}
\CLvl_t \approx \frac{1}{|I|}\sum_{i \in I} \cFunc\left(\mLvl_{i,t}/\pLvl_{i,t}\right)\times \pLvl_{i,t}.
\end{equation*}

At least two features of the previous strategy are unpleasant:
- We have to simulate the distribution of permanent income, even though the model's solution does not depend on it.
- As a geometric random walk, permanent income might have an unbounded distribution. Since $\pLvl_{i,t}$ appears multiplicatively in our approximation, agents with high permanent incomes will be the most important in determining levels of aggregate variables. Therefore, it is important for our simulated population to achieve a good approximation of the distribution of permanent income among the small number of agents with very high permanent income, which will require us to use many agents (large $I$, requiring considerable computational resources).

[Harmenberg (2021)](https://www.sciencedirect.com/science/article/pii/S0165188921001202?via%3Dihub) solves both problems. His solution constructs a distribution $\tilde{f}(\cdot)$ of the normalized state vector that he calls **the permanent-income-weighted distribution** and which has the convenient property that
\begin{equation*}
\begin{split}
\CLvl_t =& \int \int \cFunc\left(\frac{1}{\pLvl}\times \mLvl\right)\times \pLvl \times f_t(\mLvl,\pLvl) \, d\mLvl\, d\pLvl\\
=& \int \cFunc\left(\mNrm\right) \times \tilde{f}(\mNrm) \, d\mNrm.
\end{split}
\end{equation*}

Therefore, his solution allows us to calculate aggregate variables without the need to keep track of the distribution of permanent income. Additionally, the method eliminates the issue of a small number of agents in the tail having an outsized influence in our approximation and this makes it much more precise.

This notebook briefly describes Harmenberg's method and demonstrates its implementation in the HARK toolkit.

# Description of the method

To illustrate Harmenberg's idea, consider a [buffer stock saving](https://econ-ark.github.io/BufferStockTheory) model in which:
- The individual agent's problem has two state variables:
    - Market resources $\mLvl_{i,t}$.
    - Permanent income $\pLvl_{i,t}$.

- The agent's problem is homothetic in permanent income, so that we can define $m_t = \mLvl_t/\pLvl_t$ and find a normalized policy function $\cFunc(\cdot)$ such that
\begin{equation*}
\cFunc(\mNrm) \times \pLvl_t = \cFunc(\mLvl_t, \pLvl_t) \,\,\qquad \forall(\mLvl_t, \pLvl_t)
\end{equation*}
where $\cFunc(\cdot,\cdot)$ is the optimal consumption function.

- $\pLvl_t$ evolves according to $$\pLvl_{t+1} = \PermGroFac \PermShk_{t+1} \pLvl_t,$$ where $\PermShk_{t+1}$ is a shock with density function $f_{\PermShk}(\cdot)$ satisfying $\Ex_t[\PermShk_{t+1}] = 1$.

To compute aggregate consumption $\CLvl_t$ in this model, we would follow the approach from above
\begin{equation*}
\CLvl_t = \int \int \cFunc(\mNrm)\times\pLvl \times \mpLvlDstn_t(\mNrm,\pLvl) \, d\mNrm \, d\pLvl,
\end{equation*}
where $\mpLvlDstn_t(\mNrm,\pLvl)$ is the measure of agents with normalized resources $\mNrm$ and permanent income $\pLvl$.

## First insight

The first of Harmenberg's insights is that the previous integral can be rearranged as
\begin{equation*}
\CLvl_t = \int_{\mNrm} \cFunc(\mNrm)\left(\int \pLvl \times \mpLvlDstn_t(\mNrm,\pLvl) \, d\pLvl\right) \, d\mNrm.
\end{equation*}
The inner integral, $\int_{\pLvl} \pLvl \times \mpLvlDstn_t(\mNrm,\pLvl) \, d\pLvl$, is a function of $\mNrm$ and it measures *the total amount of permanent income accruing to agents with normalized market resources of* $\mNrm$. De-trending this object by the deterministic component of growth in permanent income $\PermGroFac$, Harmenberg defines the *permanent-income-weighted distribution* $\mWgtDstnMarg(\cdot)$ as

\begin{equation*}
\mWgtDstnMarg_{t}(\mNrm) \def \PermGroFac^{-t}\int_{\pLvl} \pLvl \times \mpLvlDstn_t(\mNrm,\pLvl) \, d\pLvl.
\end{equation*}

The definition allows us to rewrite
\begin{equation}\label{eq:aggC}
\CLvl_{t} = \PermGroFac^t \int_{m} \cFunc(\mNrm) \times \mWgtDstnMarg_t(\mNrm) \, dm.
\end{equation}

There are no computational advances yet: We have merely hidden the joint distribution of $(\mNrm,\pLvl)$ inside the $\mWgtDstnMarg$ object we have defined. This helps us notice that $\mWgtDstnMarg$ is the only object besides the solution that we need in order to compute aggregate consumption. But we still have no practial way of computing or approximating $\mWgtDstnMarg$.

## Second insight

Harmenberg's second insight produces a simple way of generating simulated counterparts of $\mWgtDstnMarg$ without having to simulate permanent incomes.

We start with the density function of $\mNrm_{t+1}$ given $\mNrm_t$ and $\PermShk_{t+1}$, $\kernel(\mNrm_{t+1}|\mNrm_t,\PermShk_{t+1})$. This density will depend on the model's transition equations and draws of random variables like transitory shocks to income in $t+1$ or random returns to savings between $t$ and $t+1$. If we can simulate those things, then we can sample from $\kernel(\cdot|\mNrm_t,\PermShk_{t+1})$.

Harmenberg shows that
\begin{equation*}\label{eq:transition}
\texttt{transition:    }\mWgtDstnMarg_{t+1}(\mNrm_{t+1}) = \int \kernel(\mNrm_{t+1}|\mNrm_t, \PermShk_t) \pShkNeutDstn(\PermShk_{t+1}) \mWgtDstnMarg_t(\mNrm_t)\, d\mNrm_t\, d\PermShk_{t+1},
\end{equation*}
where $\pShkNeutDstn$ is an altered density function for the permanent income shocks $\PermShk$, which he calls the *permanent-income-neutral* measure, and which relates to the original density $f_{\PermShk}$ through $$\pShkNeutDstn(\PermShk_{t+1})\def \PermShk_{t+1}f_{\PermShk}(\PermShk_{t+1})\,\,\, \forall \PermShk_{t+1}.$$

What's remarkable about this equation is that it gives us a way to obtain a distribution $\mWgtDstnMarg_{t+1}$ from $\mWgtDstnMarg_t$:
- Start with a population whose $\mNrm$ is distributed according to $\mWgtDstnMarg_t$.
- Give that population permanent income shocks with distribution $\pShkNeutDstn$.
- Apply the transition equations and other shocks of the model to obtain $\mNrm_{t+1}$ from $\mNrm_{t}$ and $\PermShk_{t+1}$ for every agent.
- The distribution of $\mNrm$ across the resulting population will be $\mWgtDstnMarg_{t+1}$.

Notice that the only change in these steps from what how we would usually simulate the model is that we now draw permanent income shocks from $\pShkNeutDstn$ instead of $f_{\PermShk}$. Therefore, with this procedure we can approximate $\mWgtDstnMarg_t$ and compute aggregates using formulas like the equation `transition`, all without tracking permanent income and with few changes to the code we use to simulate the model.

# Harmenberg's method in HARK

Harmenberg's method for simulating under the permanent-income-neutral measure is available in [HARK's `IndShockConsumerType` class](https://github.com/econ-ark/HARK/blob/master/HARK/ConsumptionSaving/ConsIndShockModel.py) and the (many) models that inherit its income process, such as [`PortfolioConsumerType`](https://github.com/econ-ark/HARK/blob/master/HARK/ConsumptionSaving/ConsPortfolioModel.py).

As the cell below illustrates, using Harmenberg's method in [HARK](https://github.com/econ-ark/HARK) simply requires setting an agent's property `agent.neutral_measure = True` and then computing the discrete approximation to the income process. After these steps, `agent.simulate` will simulate the model using Harmenberg's permanent-income-neutral measure.

`# Implementation in HARK:`

#### Farther down in the notebook, code like this solves the standard model:

```python
# Create a population with the default parametrization

popn = IndShockConsumerType(**params)

# Specify which variables to track in the simulation
popn.track_vars=[
    'mNrm',  # mLvl normalized by permanent income (mLvl = market resources)
    'cNrm',  # cLvl normalized by permanent income (cLvl = consumption)
    'pLvl']  # pLvl: permanent income

popn.cycles = 0  # No life cycles -- an infinite horizon

# Solve for the consumption function
popn.solve()

# Simulate under the base measure
popn.initialize_sim()
popn.simulate()
```

#### Later, code like this simulates using the permanent-income-neutral measure
```python
# Harmenberg permanent-income-neutral simulation

# Make a clone of the population weighted solution
ntrl = deepcopy(popn)

# Change the income process to use the neutral measure

ntrl.neutral_measure = True
ntrl.update_income_process()

# Simulate
ntrl.initialize_sim()
ntrl.simulate()
```

All we had to do differently to simulate using the permanent-income-neutral measure was to set the agent's property `neutral_measure=True`.

This is implemented when the function `update_income_process` re-constructs the agent's income process. The specific lines that achieve the change of measure in HARK are in [this link](https://github.com/econ-ark/HARK/blob/760df611a6ec2ff147d00b7d866dbab6fc4e18a1/HARK/ConsumptionSaving/ConsIndShockModel.py#L2734-L2735), or reproduced here:

```python
if self.neutral_measure == True:
    PermShkDstn_t.pmf = PermShkDstn_t.X*PermShkDstn_t.pmf
```

Simple!

# The eficiency gain from using Harmenberg's method

To demonstrate the gain in efficiency from using Harmenberg's method, we will set up the following experiment.

Consider an economy populated by [Buffer-Stock](https://econ-ark.github.io/BufferStockTheory/) savers, whose individual-level state variables are market resources $\mLvl_t$ and permanent income $\pLvl_t$. Such agents have a [homothetic consumption function](https://econ-ark.github.io/BufferStockTheory/#The-Problem-Can-Be-Normalized-By-Permanent-Income), so that we can define normalized market resources $\mNrm_t \def \mLvl_t / \pLvl_t$, solve for a normalized consumption function $\cFunc(\cdot)$, and express the consumption function as $\cFunc(\mLvl,\pLvl) = \cFunc(\mNrm)\times\pLvl$.

Assume further that mortality, impatience, and permanent income growth are such that the economy converges to stable joint distribution of $\mNrm$ and $\pLvl$ characterized by the density function $f(\cdot,\cdot)$. Under these conditions, define the stable level of aggregate market resources and consumption as
\begin{equation}
    \MLvl \def \int \int \mNrm \times \pLvl \times f(\mNrm, \pLvl)\,d\mNrm \,d\pLvl, \,\,\,    \CLvl \def \int \int \cFunc(\mNrm) \times \pLvl \times f(\mNrm, \pLvl)\,d\mNrm \,d\pLvl.
\end{equation}

If we could simulate the economy with a continuum of agents we would find that, over time, our estimate of aggregate market resources $\MLvlest_t$ would converge to $\MLvl$ and $\CLvlest_t$ would converge to $\CLvl$. Therefore, if we computed our aggregate estimates at different periods in time we would find them to be close:
\begin{equation*}
    \MLvlest_t \approx \MLvlest_{t+n} \approx \MLvl \,\,
    \text{and} \,\,
    \CLvlest_t \approx \CLvlest_{t+n} \approx \CLvl, \,\,
    \text{for } n>0 \text{ and } t \text{ large enough}.
\end{equation*}

In practice, however, we rely on approximations using a finite number of agents $I$. Our estimates of aggregate market resources and consumption at time $t$ are
\begin{equation}
\MLvlest_t \def \frac{1}{I} \sum_{i=1}^{I} m_{i,t}\times\pLvl_{i,t}, \,\,\, \CLvlest_t \def \frac{1}{I} \sum_{i=1}^{I} \cFunc(m_{i,t})\times\pLvl_{i,t},
\end{equation}

under the basic simulation strategy or

\begin{equation}
\MLvlest_t \def \frac{1}{I} \sum_{i=1}^{I} \tilde{m}_{i,t}, \,\,\, \CLvlest_t \def \frac{1}{I} \sum_{i=1}^{I} \cFunc(\tilde{m}_{i,t}),
\end{equation}

if we use Harmenberg's method to simulate the distribution of normalized market resources under the permanent-income neutral measure.

If we do not use enough agents, our distributions of agents over state variables will be noisy at approximating their continuous counterparts. Additionally, they will depend on the sequences of shocks that the agents receive. With a finite sample, the stochasticity of the draws will cause fluctuations in $\MLvlest_t$ and $\CLvlest_t$. Therefore an informal way to measure the precision of our approximations is to examine the amplitude of these fluctuations.

First, some setup.
1. Simulate the economy for a sufficiently long "burn in" time $T_0$.
2. Sample our aggregate estimates at regular intervals after $T_0$. Letting the sampling times be $\mathcal{T}\def \{T_0 + \Delta t\times n\}_{n=0,1,...,N}$, obtain $\{\MLvlest_t\}_{t\in\mathcal{T}}$ and $\{\CLvlest_t\}_{t\in\mathcal{T}}$.
3. Compute the variance of approximation samples $\text{Var}\left(\{\MLvlest_t\}_{t\in\mathcal{T}}\right)$ and $\text{Var}\left(\{\CLvlest_t\}_{t\in\mathcal{T}}\right)$.
    - Other measures of uncertainty (like standard deviation) could also be computed
    - But variance is the natural choice [because it is closely related to expected welfare](http://www.econ2.jhu.edu/people/ccarroll/papers/candcwithstickye/#Utility-Costs-Of-Sticky-Expectations)

We will now perform exactly this exercise, examining the fluctuations in aggregates when they are approximated using the basic simulation strategy and Harmenberg's permanent-income-neutral measure. Since each approximation can be made arbitrarily good by increasing the number of agents it uses, we will examine the variances of aggregates for various sample sizes.

`# Setup computational environment:`

```python
# How long to run the economies without sampling? T_0
# Because we start the population at mBalLvl which turns out to be close
# to MBalLvl so we don't need a long burn in period
burn_in = 200
# Fixed intervals between sampling aggregates, Δt
sample_every = 1 # periods - increase this if worried about serial correlation
# How many times to sample the aggregates? n
n_sample = 200 # times; minimum

# Create a vector with all the times at which we'll sample
sample_periods_lvl = \
    np.arange(start = burn_in,
              stop  = burn_in + \
                      sample_every * n_sample,
              step  = sample_every, dtype = int)
# Corresponding periods when object is first difference not level
sample_periods_dff = \
    np.arange(start = burn_in,
              stop  = burn_in + \
                      sample_every * n_sample-1, # 1 fewer diff
              step  = sample_every, dtype = int)

# Maximum number of agents that we will use for our approximations
max_agents = 100000
# Minimum number of agents for comparing methods in plots
min_agents = 100
```

`# Define tool to calculate summary statistics:`

```python
# Now create a function that takes HARK's simulation output
# and computes all the summary statistics we need

def sumstats(sims, sample_periods):
    # sims will be an array in the shape of [economy].history elements
    # Columns are different agents and rows are different times.

    # Subset the times at which we'll sample and transpose.
    samples_lvl = pd.DataFrame(sims[sample_periods,].T)

    # Get averages over agents. This will tell us what our
    # aggregate estimate would be if we had each possible sim size
    avgs_lvl = samples_lvl.expanding(1).mean()

    # Now get the mean and standard deviations across time with
    # every number of agents
    mean_lvl = avgs_lvl.mean(axis = 1)
    vars_lvl = avgs_lvl.std(axis = 1)**2

    # Also return the full sample on the last simulation period
    return {'mean_lvl': mean_lvl, 'vars_lvl': vars_lvl,
            'dist_last': sims[-1,]}
```

We now configure and solve a buffer-stock agent with a default parametrization.

```python
# Create and solve agent

popn = IndShockConsumerType(**init_idiosyncratic_shocks)

# Modify default parameters
popn.T_sim = max(sample_periods_lvl)+1
popn.AgentCount = max_agents
popn.track_vars = ['mNrm', 'cNrm', 'pLvl']
popn.LivPrb=[1.0]
popn.cycles = 0

# Solve (but do not yet simulate)
popn.solve()
```

    (quietly=False and messaging_level < logging.WARNING, so some model information is provided below):

    The model has the following parameter values:

    	DiscFac: 0.96
    	LivPrb: 0.98
    	CRRA: 2.0
    	Rfree: 1.03
    	PermGroFac: 1.01
    	IncShkDstn: <HARK.distribution.DiscreteDistribution object at 0x7fdb787b2e50>
    	PermShkDstn: <HARK.distribution.DiscreteDistribution object at 0x7fdb4136f760>
    	TranShkDstn: <HARK.distribution.DiscreteDistribution object at 0x7fdb78785970>
    	BoroCnstArt: 0.0
    	MaxKinks: None
    	solverType: HARK
    	solveMethod: EGM
    	eventTiming: EOP
    	horizon: infinite
    	args: ()

    The model's transition equations are:

    chosen_to_next_choice step:
    	kNrm = aNrm
    	RNrm = Rfree / (PermGroFac * PermShk)
    	bNrm = kNrm * RNrm
    	yNrm = TranShk
    	mNrm = bNrm + yNrm

    choice_to_chosen step:
    	aNrm = mNrm - cNrm

    Because messaging_level is >= logging.INFO, infinite horizon conditions are reported below:

    The Absolute Patience Factor, APFac=0.98439 satisfies the Absolute Impatience Condition (AIC), APFac < 1:
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#AIC

    The Finite Human Wealth Factor, FHWFac=0.98058 satisfies the Finite Human Wealth Condition (FHWC), FHWFac < 1:
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#FHWC

    The Return Patience Factor, RPFac=0.95572 satisfies the Return Impatience Condition (RIC), RPFac < 1:
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#RPFacDefn

    The Normalized Growth Patience Factor GPFacNrm, GPFacNrm=0.98379 satisfies the Normalized Growth Impatience Condition (GICNrm), GPFacNrm < 1:
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#GICNrmDefn

    The Szeidl Growth Patience Factor GPFacSdl=0.97921 satisfies the Szeidl Growth Impatience Condition (GICSdl), GPFacSdl < 1:
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#GICSdlDefn

    The Growth Patience Factor, GPF=0.97464 satisfies the Growth Impatience Condition (GICRaw), GPFac < 1:
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#GICRawDefn

    The Harmenberg Growth Patience Factor GPFacHrm=0.97010 satisfies the Harmenberg Growth Impatience Condition (GICHrm), GPFacHrm < 1:
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#GICHrmDefn

    The Mortality Adjusted Growth Patience Factor, GPFacLiv=0.95515 satisfies the Mortality Adjusted Growth Impatience Condition (GICLiv):
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#GPFacLivDefn

    The Modigliani Mortality Adjusted Growth Patience Factor, GPFacLivMod=0.93605 satisfies the Mortality Adjusted Growth Impatience Condition (GICLivMod):
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#GPFacLivModDefn

    The Finite Value of Autarky Factor, VAFac=0.92198 satisfies the Finite Value of Autarky Condition, VAFac < 1:
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#FVAC
    Some parameter has changed since last update.
    Storing calculated consequences for grid etc.

    .
    ..............................................................................................................................................................................................................

    (quietly=False and messaging_level < logging.WARNING, so some model information is provided below):

    The model has the following parameter values:

    	DiscFac: 0.96
    	LivPrb: 1.0
    	CRRA: 2.0
    	Rfree: 1.03
    	PermGroFac: 1.01
    	IncShkDstn: <HARK.distribution.DiscreteDistribution object at 0x7fdb41381a90>
    	PermShkDstn: <HARK.distribution.DiscreteDistribution object at 0x7fdb413815e0>
    	TranShkDstn: <HARK.distribution.DiscreteDistribution object at 0x7fdb4136ff40>
    	BoroCnstArt: 0.0
    	MaxKinks: None
    	solverType: HARK
    	solveMethod: EGM
    	eventTiming: EOP
    	horizon: infinite
    	args: ()

    The model's transition equations are:

    chosen_to_next_choice step:
    	kNrm = aNrm
    	RNrm = Rfree / (PermGroFac * PermShk)
    	bNrm = kNrm * RNrm
    	yNrm = TranShk
    	mNrm = bNrm + yNrm

    choice_to_chosen step:
    	aNrm = mNrm - cNrm

    Because messaging_level is >= logging.INFO, infinite horizon conditions are reported below:

    The Absolute Patience Factor, APFac=0.99438 satisfies the Absolute Impatience Condition (AIC), APFac < 1:
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#AIC

        Because the APFac < 1,  the absolute amount of consumption is expected to fall over time.

    The Finite Human Wealth Factor, FHWFac=0.98058 satisfies the Finite Human Wealth Condition (FHWC), FHWFac < 1:
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#FHWC

        Therefore, the limiting consumption function is not c(m)=Infinity.

        Human wealth normalized by permanent income is 51.50000.

    The Return Patience Factor, RPFac=0.96542 satisfies the Return Impatience Condition (RIC), RPFac < 1:
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#RPFacDefn

        Therefore, the limiting consumption function is not c(m)=0 for all m

    The Normalized Growth Patience Factor GPFacNrm, GPFacNrm=0.99378 satisfies the Normalized Growth Impatience Condition (GICNrm), GPFacNrm < 1:
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#GICNrmDefn

        Therefore, a target level of the individual market resources ratio m exists.

    The Szeidl Growth Patience Factor GPFacSdl=0.98915 satisfies the Szeidl Growth Impatience Condition (GICSdl), GPFacSdl < 1:
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#GICSdlDefn

        Therefore, invariant distributions of ratio variables like m exist.

    The Growth Patience Factor, GPF=0.98454 satisfies the Growth Impatience Condition (GICRaw), GPFac < 1:
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#GICRawDefn

        Therefore, for a perfect foresight consumer, the ratio of individual wealth to permanent income is expected to fall indefinitely.

    The Harmenberg Growth Patience Factor GPFacHrm=0.97995 satisfies the Harmenberg Growth Impatience Condition (GICHrm), GPFacHrm < 1:
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#GICHrmDefn

        Therefore, a target level of the individual market resources ratio m exists under the permanent-income-neutral measure.

    The Mortality Adjusted Growth Patience Factor, GPFacLiv=0.98454 satisfies the Mortality Adjusted Growth Impatience Condition (GICLiv):
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#GPFacLivDefn

        Therefore, a target level of the ratio of aggregate market resources to aggregate permanent income exists.

    The Modigliani Mortality Adjusted Growth Patience Factor, GPFacLivMod=0.98454 satisfies the Mortality Adjusted Growth Impatience Condition (GICLivMod):
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#GPFacLivModDefn

        Therefore, a target level of the ratio of aggregate market resources to aggregate permanent income exists.

    The Finite Value of Autarky Factor, VAFac=0.96000 satisfies the Finite Value of Autarky Condition, VAFac < 1:
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#FVAC

        Therefore, a nondegenerate solution exists if the RIC also holds.

    ..

Under the basic simulation strategy, we have to de-normalize market resources and consumption multiplying them by permanent income. Only then we construct our statistics of interest.

Note that our time-sampling strategy requires that, after enough time has passed, the economy settles on a stable distribution of its agents across states. How can we know this will be the case? [Szeidl (2013)](http://www.personal.ceu.hu/staff/Adam_Szeidl/papers/invariant.pdf) and [Harmenberg (2021)](https://www.sciencedirect.com/science/article/pii/S0165188921001202?via%3Dihub) provide conditions that can give us some reassurance.$\newcommand{\Rfree}{\mathsf{R}}$

1. [Szeidl (2013)](http://www.personal.ceu.hu/staff/Adam_Szeidl/papers/invariant.pdf) shows that if $$\log \left[\frac{(\Rfree\beta)^{1/\rho}}{\PermGroFac}
\right] < \Ex[\log \PermShk],$$ then there is a stable invariant distribution of normalized market resources $\mNrm$.
2. [Harmenberg (2021)](https://www.sciencedirect.com/science/article/pii/S0165188921001202?via%3Dihub) repurposes the Szeidl proof to argue that if the same condition is satisfied when the expectation is taken with respect to the permanent-income-neutral measure ($\pShkNeutDstn$), then there is a stable invariant permanent-income-weighted distribution ($\mWgtDstnMarg$)

We now check both conditions with our parametrization.

```python
# Two versions for different HARK versions
try: # This works with HARK 2.0 pre-alpha
    Bilt = popn.solution[0].Bilt
    APFac = Bilt.APFac
    GPFacRaw = Bilt.GPFacRaw
    soln=popn.solution[0]
    soln.check_GICSdl(soln,quietly=False)
    soln.check_GICHrm(soln,quietly=False)
except: # This one for HARK 0.12 or later
    APFac = popn.thorn
    GPFacRaw = popn.GPFRaw
    e_log_PermShk_popn = calc_expectation(popn.PermShkDstn[0], func = lambda x:   np.log(x))
    e_log_PermShk_ntrl = calc_expectation(popn.PermShkDstn[0], func = lambda x: x*np.log(x))
    szeidl_cond = np.log(GPFacRaw) < e_log_PermShk_popn
    harmen_cond = np.log(GPFacRaw) < e_log_PermShk_ntrl
    if szeidl_cond:
        print("Szeidl's condition is satisfied, there is a stable invariant distribution of normalized market resources")
    else:
        print("Warning: Szeidl's condition is not satisfied")
    if harmen_cond:
        print("Harmenberg's condition is satisfied, there is a stable invariant permanent-income-weighted distribution")
    else:
        print("Warning: Harmenberg's condition is not satisfied")
```

    The Szeidl Growth Patience Factor GPFacSdl=0.98915 satisfies the Szeidl Growth Impatience Condition (GICSdl), GPFacSdl < 1:
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#GICSdlDefn

        Therefore, invariant distributions of ratio variables like m exist.

    The Harmenberg Growth Patience Factor GPFacHrm=0.97995 satisfies the Harmenberg Growth Impatience Condition (GICHrm), GPFacHrm < 1:
        https://econ-ark.github.io/BufferStockTheory/BufferStockTheory3.html#GICHrmDefn

        Therefore, a target level of the individual market resources ratio m exists under the permanent-income-neutral measure.

Knowing that the conditions are satisfied, we are ready to perform our experiments.

First, we simulate using the traditional approach.

```python
# Base simulation

# Start assets at m balanced growth (levels) point
try: # Accommodate syntax for old and new versions of HARK
    Bilt=popn.solution[0].Bilt
    popn.aNrmInitMean = np.log(Bilt.mBalLvl-1)
except:
    popn.aNrmInitMean = np.log(popn.solution[0].mNrmStE-1)

popn.aNrmInitStd = 0.

popn.initialize_sim()
popn.simulate()

# Retrieve history
mNrm_popn = popn.history['mNrm']
mLvl_popn = popn.history['mNrm'] * popn.history['pLvl']
cLvl_popn = popn.history['cNrm'] * popn.history['pLvl']
```

Update and simulate using Harmenberg's strategy. This time, not multiplying by permanent income.

```python
# Harmenberg permanent income neutral simulation

# Start by duplicating the previous setup
ntrl = deepcopy(popn)

# Recompute income process to use neutral measure
ntrl.neutral_measure = True
ntrl.update_income_process()

ntrl.initialize_sim()
ntrl.simulate()

# Retrieve history
cLvl_ntrl = ntrl.history['cNrm']
mLvl_ntrl = ntrl.history['mNrm']
```

# Now Compare the Variances of Simulated Outcomes

Harmenberg (2021) and Szeidl (2013) prove that with an infinite population size, models of this kind will have constant and identical growth rates of aggregate consumption, market resources, and noncapital income.

A method of comparing the efficiency of the two methods is therefore to calculate the variance of the simulated aggregate variables, and see how many agents must be simulated using each of them in order to achieve a given variance.  (An infinite number of agents would be required to achieve zero variance).

The plots below show the (logs of) the estimated variances for the two methods as a function of the (logs of) the number of agents.

```python
# Plots

# Construct aggregate levels and growth rates
## Cumulate levels and divide by number of agents to get average
CAvg_popn=np.cumsum(cLvl_popn, axis=1)/np.arange(1, max_agents+1)
CAvg_ntrl=np.cumsum(cLvl_ntrl, axis=1)/np.arange(1, max_agents+1)
MAvg_popn=np.cumsum(mLvl_popn, axis=1)/np.arange(1, max_agents+1)
MAvg_ntrl=np.cumsum(mLvl_ntrl, axis=1)/np.arange(1, max_agents+1)
## First difference the logs to get aggregate growth rates
CGro_popn=np.diff(np.log(CAvg_popn).T).T
CGro_ntrl=np.diff(np.log(CAvg_ntrl).T).T
MGro_popn=np.diff(np.log(MAvg_popn).T).T
MGro_ntrl=np.diff(np.log(MAvg_ntrl).T).T
## Calculate statistics for them
CGro_popn_stats=sumstats(CGro_popn, sample_periods_dff)
CGro_ntrl_stats=sumstats(CGro_ntrl, sample_periods_dff)
MGro_popn_stats=sumstats(MGro_popn, sample_periods_dff)
MGro_ntrl_stats=sumstats(MGro_ntrl, sample_periods_dff)

# Count the agents
nagents = np.arange(1,max_agents+1,1)

# Plot
fig, axs = plt.subplots(2, figsize=(10, 7), constrained_layout=True)
fig.suptitle('Variances of Aggregate Growth Rates', fontsize=16)
axs[0].plot(nagents[min_agents:], np.array(CGro_popn_stats['vars_lvl'])[min_agents:], label='Base')
axs[0].plot(nagents[min_agents:], np.array(CGro_ntrl_stats['vars_lvl'])[min_agents:], label='Perm. Inc. Neutral')
axs[0].set_yscale('log')
axs[0].set_xscale('log')
axs[0].set_title('Consumption', fontsize=14)
axs[0].set_ylabel(r'$\Delta \log \left(\{\hat{C}_t\}_{t\in\mathcal{T}}\right)$', fontsize = 14)
axs[0].set_xlabel('Number of Agents', fontsize=10)
axs[0].grid()
axs[0].legend(fontsize=12)

axs[1].plot(nagents[min_agents:], np.array(MGro_popn_stats['vars_lvl'])[min_agents:], label = 'Base')
axs[1].plot(nagents[min_agents:], np.array(MGro_ntrl_stats['vars_lvl'])[min_agents:], label = 'Perm. Inc. Neutral')
axs[1].set_yscale('log')
axs[1].set_xscale('log')
axs[1].set_title('Market Resources', fontsize=14)
axs[1].set_ylabel(r'$\Delta \log \left(\{\hat{M}_t\}_{t\in\mathcal{T}}\right)$', fontsize = 14)
axs[1].set_xlabel('Number of Agents', fontsize=10)
axs[1].grid()
axs[1].legend(fontsize=12)

plt.show()
```

# Harmenberg's Method Produces Large Gains in Efficiency

The number of agents required to achieve a given variance is revealed by choosing that variance and then finding the points on the horizontal axis that correspond to the two methods.

The upper variance plot shows that the efficiency gains are very large for consumption: The horizontal gap between the two loci is generally more than two orders of magnitude.  That is, Harmenberg's method requires less than **one-hundredth** as many agents as the standard method would require for a given precision.  Alternatively, for a given number of agents it is typically more than 10 times as precise.

The improvement in variance is smaller for market resources, likely because in a buffer stock model the point of consumers' actions is to use assets to absorb shocks.  But even for $\MLvl$, the Harmenberg method attains any given level of precision ($\text{var}\left(\{\MLvlest_t\}_{t\in\mathcal{T}}\right)$) with roughly **one tenth** of the agents needed by the standard method to achieve that same level.

Of course, these results apply only to the particular configuration of parameter values that is the default in the HARK toolkit (but which were chosen long before Harmenberg developed his method).  The degree of improvement will vary depending on the calibration -- for example, if the magnitude of permanent shocks is small or zero, the method will yield little or no improvement.

```python
# Execute the line below to see that there's little drift from mBalLvl as starting point
# (after setting burn_in to zero above).  This means burn_in does not need to be large:

# plt.plot(np.arange(1,len(np.mean(MAvg_ntrl,axis=1))+1),np.mean(MAvg_ntrl,axis=1).T)
```


# New features v0.3.3

[1. Classical-quantum maps and mixed circuits](#1.-Classical-quantum-maps-and-mixed-circuits)

- `discopy.quantum.cqmap` implements Bob and Aleks' classical-quantum maps.
- Now `discopy.quantum.circuit` diagrams have two generating objects: `bit` and `qubit`.
- New boxes `Discard`, `Measure` and `ClassicalGate` can be simulated with `cqmap` or sent to `pytket`.

[2. ZX diagrams and PyZX interface](#2.-ZX-diagrams-and-PyZX-interface)

- `discopy.quantum.zx` implements diagrams with spiders, swaps and Hadamard boxes.
- `to_pyzx` and `from_pyzx` methods can be used to turn diagrams into graphs, simplify then back.

[3. Parametrised diagrams, formal sums and automatic gradients](#3.-Parametrised-diagrams,-formal-sums-and-automatic-gradients)

- We can use `sympy.Symbols` as variables in our diagrams (tensor, circuit or ZX).
- We can take formal sums of diagrams. `TensorFunctor` sends formal sums to concrete sums.
- Given a diagram (tensor, circuit or ZX) with a free variable, we can compute its gradient as a sum.

[4. Learning functors, diagrammatically](#4.-Learning-functors,-diagrammatically)

- We can use automatic gradients to learn functors (classical and/or quantum) from data.

## 1. Classical-quantum maps and mixed circuits

```python
from discopy.quantum import *

circuit = H @ X >> CX >> Measure() @ Id(qubit)
circuit.draw()
```

```python
circuit.eval()
```

    CQMap(dom=Q(Dim(2, 2)), cod=C(Dim(2)) @ Q(Dim(2)), array=[0. +0.j, 0. +0.j, 0. +0.j, ..., 0.5+0.j, 0. +0.j, 0. +0.j])

```python
circuit.init_and_discard().draw()
```

```python
circuit.init_and_discard().eval()
```

    CQMap(dom=CQ(), cod=C(Dim(2)), array=[0.5+0.j, 0. +0.j])

```python
circuit.to_tk()
```

    tk.Circuit(2, 1).H(0).X(1).CX(0, 1).Measure(0, 0)

```python
from pytket.extensions.qiskit import AerBackend
backend = AerBackend()
circuit.eval(backend)
```

    Tensor(dom=Dim(1), cod=Dim(2), array=[0.52832031, 0.47167969])

```python
postprocess = ClassicalGate('postprocess', 2, 0, data=[1, 0, 0, 0])
postprocessed_circuit = Ket(0, 0) >> H @ X >> CX >> Measure() @ Measure() >> postprocess
postprocessed_circuit.draw(aspect='auto')
```

```python
postprocessed_circuit.to_tk()
```

    tk.Circuit(2, 2).H(0).X(1).CX(0, 1).Measure(0, 0).Measure(1, 1).post_process(ClassicalGate('postprocess', bit @ bit, Ty(), data=[1, 0, 0, 0]))

```python
postprocessed_circuit.eval(backend)
```

    Tensor(dom=Dim(1), cod=Dim(1), array=[0.])

## 2. ZX diagrams and PyZX interface

```python
from discopy.quantum.zx import *
from pyzx import draw

bialgebra = Z(1, 2, .25) @ Z(1, 2, .75) >> Id(1) @ SWAP @ Id(1) >> X(2, 1) @ X(2, 1, .5)
bialgebra.draw(aspect='equal')
draw(bialgebra.to_pyzx())
```

```python
from pyzx import generate, simplify

graph = generate.cliffordT(2, 5)
print("From DisCoPy:")
Diagram.from_pyzx(graph).draw()

print("To PyZX:")
draw(graph)
simplify.full_reduce(graph)
draw(graph)

print("And back!")
Diagram.from_pyzx(graph).draw()
```

## 3. Parametrised diagrams, formal sums and automatic gradients

```python
from sympy.abc import phi
from discopy.quantum import *
from discopy import drawing

circuit = sqrt(2) @ Ket(0, 0) >> H @ Rx(phi) >> CX >> Bra(0, 1)
drawing.equation(circuit, circuit.subs(phi, .5), symbol="|-->")
```

```python
gradient = (circuit >> circuit[::-1]).grad(phi)
gradient.draw(draw_type_labels=False)
```

```python
import numpy as np

x = np.arange(0, 1, 0.05)
y = np.array([circuit.lambdify(phi)(i).measure() for i in x])
dy = np.array([gradient.lambdify(phi)(i).eval().array.real for i in x])
```

```python
from matplotlib import pyplot as plt

plt.subplot(2, 1, 1)
plt.plot(x, y)
plt.ylabel("Amplitude")

plt.subplot(2, 1, 2)
plt.plot(x, dy)
plt.ylabel("Gradient")
```

## 4. Learning functors, diagrammatically

```python
from discopy import *

s, n = Ty('s'), Ty('n')
Alice, loves, Bob = Word("Alice", n), Word("loves", n.r @ s @ n.l), Word("Bob", n)

grammar = Cup(n, n.r) @ Id(s) @ Cup(n.l, n)
parsing = {
    "{} {} {}".format(subj, verb, obj): subj @ verb @ obj >> grammar
    for subj in [Alice, Bob] for verb in [loves] for obj in [Alice, Bob]}
pregroup.draw(parsing["Alice loves Bob"], aspect='equal')

print("Our favorite toy dataset:")
corpus = {
    "{} {} {}".format(subj, verb, obj): int(obj != subj)
    for subj in [Alice, Bob] for verb in [loves] for obj in [Alice, Bob]}
for sentence, scalar in corpus.items():
    print("'{}' is {}.".format(sentence, "true" if scalar else "false"))
```

```python
from sympy import symbols

parameters = symbols("a0 a1 b0 b1 c00 c01 c10 c11")

F = TensorFunctor(
    ob={s: 1, n: 2},
    ar={Alice: symbols("a0 a1"),
        Bob: symbols("b0 b1"),
        loves: symbols("c00 c01 c10 c11")})
gradient = F(parsing["Alice loves Bob"]).grad(parameters[0])
gradient
```

    Tensor(dom=Dim(1), cod=Dim(1), array=[1.0*b0*c00 + 1.0*b1*c01])

```python
gradient.subs(list(zip(parameters, 8 * [0])))
```

    Tensor(dom=Dim(1), cod=Dim(1), array=[0])

```python
from discopy.quantum import *

gates = {
    Alice: ClassicalGate('Alice', 0, 1, symbols("a0 a1")),
    Bob: ClassicalGate('Bob', 0, 1, symbols("b0 b1")),
    loves: ClassicalGate('loves', 0, 2, symbols("c00 c01 c10 c11"))}

F = CircuitFunctor(ob={s: Ty(), n: bit}, ar=gates)
F(parsing["Alice loves Bob"]).draw()
```

```python
F(parsing["Alice loves Alice"]).grad(symbols("a0")).draw()
```


# Selección óptima de portafolios I

En la clase pasada vimos que:
- La LAC describe las posibles selecciones de riesgo-rendimiento entre un activo libre de riesgo y un activo riesgoso.
- Su pendiente es igual al radio de Sharpe del activo riesgoso.
- La asignación óptima de capital para cualquier inversionista es el punto tangente de la curva de indiferencia del inversionista con la LAC (depende de las preferencias particulares - aversión al riesgo).

Para todo lo anterior, supusimos que ya teníamos el portafolio óptimo (activo riesgoso).

En el siguiente análisis:

**Objetivos:**
- ¿Cuál es el portafolio óptimo de activos riesgosos?
 - ¿Cuál es el mejor portafolio de activos riesgosos?
 - Es un portafolio eficiente en media-varianza.
- Problema: dado un conjunto de activos riesgosos, ¿cómo construimos la mejor combinación?

*Referencia:*
- Notas del curso "Portfolio Selection and Risk Management", Rice University, disponible en Coursera.
___

## 1. Maximizando el radio de Sharpe

### ¿Qué pasa si tenemos dos activos riesgosos?

Cuando tenemos dos o más activos riesgosos, tenemos disponibles diferentes LAC. ¿Qué significan sus pendientes?

 Ver en el tablero.

Pregunta:
- ¿Qué es lo que se quiere?

**Conclusión:**
- El mejor portafolio de activos no depende de las preferencias individuales, y por tanto va a ser igual para todos.
- Dicho mejor portafolio maximiza el radio de Sharpe.
- A este portafolio lo llamaremos el portafolio eficiente en media-varianza (EMV)

**Idea principal: el portafolio óptimo de activos riesgosos es independiente de las preferencias del inversionista.**
- El portafolio EMV determina el portafolio óptimo de activos riesgosos.
- Todos tendremos el mismo portafolio de activos riesgosos (EMV), y lo combinaremos con el activo libre de reisgo de acuerdo con las preferencias de cada uno de nosotros (aversión al riesgo).
- La LAC combinando el activo libre de riesgo y el portafolio EMV, se vuelve el conjunto de portafolios eficientes.

Entonces, se deben seguir los siguientes pasos:
1. Crear la frontera media-varianza.
2. Encontrar el portafolio que maximize el radio de Sharpe (portafolio EMV).
3. Construir la frontera eficiente (LAC) del punto $(0,r_f)$ al punto $(\sigma_s,E[r_s])$ del portafolio EMV.
4. Combinar de acuerdo a sus preferencias.
___

## 2. Solución analítica del portafolio EMV: caso con dos activos.

Queremos solucionar el siguiente problema:

\begin{align}
\max_{w_1,w_2} &\quad \frac{E[r_p]-r_f}{\sigma_p}\\
\text{s.a.}    &\quad E[r_p]=w_1E[r_1]+w_2E[r_2]\\
               &\quad \sigma_p=\sqrt{w_1^2\sigma_1^2+w_2^2\sigma_2^2+2w_1w_2\rho_{12}\sigma_1\sigma_2}\\
               &\quad w_1+w_2=1, \quad w_1,w_2\geq0
\end{align}

el cual es equivalente a

\begin{align}
\max_{w_1} &\quad \frac{w_1E[r_1]+(1-w_1)E[r_2]-r_f}{\sqrt{w_1^2\sigma_1^2+(1-w_1)^2\sigma_2^2+2w_1(1-w_1)\rho_{12}\sigma_1\sigma_2}}\\
\text{s.a.}    &\quad 0\leq w_1\leq1
\end{align}

**Actividad.**
El anterior es un problema de maximizar una función de una variable en un dominio cerrado. No debaría representar dificultad.

Encontrar la solución analítica a este problema.

Quien primero lo haga, y salga a explicarlo al tablero, le subo alguna tarea o quiz a 100.

Deben llegar a:

$$w_{1,EMV}=\frac{(E[r_1]-r_f)\sigma_2^2-(E[r_2]-r_f)\sigma_{12}}{(E[r_2]-r_f)\sigma_1^2+(E[r_1]-r_f)\sigma_2^2-((E[r_1]-r_f)+(E[r_2]-r_f))\sigma_{12}}.$$

Si nadie lo ha hecho en 30 min., procederé a hacerlo yo.

**Nota:**
- así como obtuvimos una expresión para el peso del portafolio de mínima varianza con dos activos, obtenemos una expresión para el peso del portafolio Eficiente en Media-Varianza.
- Estas actividades son sin duda un buen ejercicio, y se pueden replicar usando técnicas de varias variables (multiplicadores de Lagrange) cuando se tengan más de dos activos.
- Sin embargo, la complejidad del problema crece considerablemente con el número de variables, y la solución analítica deja de ser viable cuando mencionamos que un portafolio bien diversificado consta aproximadamente de 50-60 activos.
- En esos casos, este problema se soluciona con rutinas numéricas que hagan la optimización por nosotros.
- Por eso, les enseño cómo resolver este problema con optimizadores numéricos, porque son una solución viable y escalable a más variables.

## 3. Ejemplo ilustrativo.

Retomamos el ejemplo de mercados de acciones en los países integrantes del $G5$: EU, RU, Francia, Alemania y Japón.

```python
# Importamos pandas y numpy

```

```python
# Resumen en base anual de rendimientos esperados y volatilidades
annual_ret_summ = pd.DataFrame(columns=['EU', 'RU', 'Francia', 'Alemania', 'Japon'], index=['Media', 'Volatilidad'])
annual_ret_summ.loc['Media'] = np.array([0.1355, 0.1589, 0.1519, 0.1435, 0.1497])
annual_ret_summ.loc['Volatilidad'] = np.array([0.1535, 0.2430, 0.2324, 0.2038, 0.2298])

annual_ret_summ.round(4)
```

```python
# Matriz de correlación
corr = pd.DataFrame(data= np.array([[1.0000, 0.5003, 0.4398, 0.3681, 0.2663],
                                    [0.5003, 1.0000, 0.5420, 0.4265, 0.3581],
                                    [0.4398, 0.5420, 1.0000, 0.6032, 0.3923],
                                    [0.3681, 0.4265, 0.6032, 1.0000, 0.3663],
                                    [0.2663, 0.3581, 0.3923, 0.3663, 1.0000]]),
                    columns=annual_ret_summ.columns, index=annual_ret_summ.columns)
corr.round(4)
```

Supondremos, además, que la tasa libre de riesgo es $r_f=5\%$.

```python
# Tasa libre de riesgo

```

Entonces, supondremos que tenemos disponibles los activos correspondientes a los mercados de acciones de EU y Japón, y en adición el activo libre de riesgo.

#### 1. Construir la frontera de mínima varianza

```python
# Vector de w variando entre 0 y 1 con n pasos

# Rendimientos esperados individuales
# Activo1: EU, Activo2:Japon

# Volatilidades individuales

# Correlacion

# Covarianza

```

```python
# DataFrame de portafolios:
# 1. Índice: i
# 2. Columnas 1-2: w, 1-w
# 3. Columnas 3-4: E[r], sigma
# 4. Columna 5: Sharpe ratio

```

```python

```

```python
# Importar librerías de gráficos

```

```python
# Gráfica de dispersión de puntos coloreando
# de acuerdo a RS

```

#### 2. Encontrar el portafolio que maximiza el radio de Sharpe (EMV)

Primero, encontramos este portafolio con la fórmula que obtuvimos:

$$w_{1,EMV}=\frac{(E[r_1]-r_f)\sigma_2^2-(E[r_2]-r_f)\sigma_{12}}{(E[r_2]-r_f)\sigma_1^2+(E[r_1]-r_f)\sigma_2^2-((E[r_1]-r_f)+(E[r_2]-r_f))\sigma_{12}}.$$

```python
# Fórmula que obtuvimos

```

Ahora sí, con la función scipy.optimize.minimize

```python
# Importar la función minimize del módulo optimize

```

```python
# Función objetivo (-RS)

```

```python
# Dato inicial

# Cotas de las variables

```

```python
# Optimización numérica

# Resultado

```

```python

```

Con lo anterior, podemos obtener datos de rendimiento esperado y volatilidad del portafolio EMV

```python
# Rendimiento esperado y volatilidad del portafolio EMV

```

```python

```

```python
# Gráfica de dispersión de puntos coloreando
# de acuerdo a SR, y portafolio EMV

```

#### 3. Construir LAC

Ahora, dibujamos la LAC, combinando el portafolio EMV con el activo libre de riesgo:

```python
# Vector de wp variando entre 0 y 1.5 con n pasos

```

```python
# DataFrame de CAL:
# 1. Índice: i
# 2. Columnas 1-2: wp, wrf
# 3. Columnas 3-4: E[r], sigma
# 4. Columna 5: Sharpe ratio

```

```python

```

```python
# Gráfica de dispersión de puntos coloreando
# de acuerdo a SR, portafolio EMV y LAC

```

#### 4. Combinación óptima de acuerdo a preferencias

Con los datos anteriores, y la caracterización de aversión al riesgo, se escoge la combinación óptima entre el portafolio EMV y el activo libre de riesgo de acuerdo a:

$$w^\ast=\frac{E[r_s-r_f]}{\gamma\sigma_s^2}.$$

```python
# Para gamma=7

```

```python

```

```python

```

```python
# Ponderaciones finales: índice de EU, índice de Japón, tasa libre de riesgo

```

Created with Jupyter by Esteban Jiménez Rodríguez.


## CCNSS 2018 Module 5: Whole-Brain Dynamics and Cognition
# Tutorial 4: Kuramoto model (II)

*Please execute the cell bellow in order to initialize the notebook environment*

```
!if [ ! -d data ]; then git clone https://github.com/ccnss/ccnss2018_students; \
                        cp -rf ccnss2018_students/module5/4_kuramoto model_2/data ./; fi
```

```
import matplotlib.pyplot as plt    # import matplotlib
import numpy as np                 # import numpy
import math                        # import basic math functions
import random                      # import basic random number generator functions
import csv                         # import CSV(Comma Separated Values) file reading and writing
import scipy as sp                 # import scipy
from scipy import sparse           # import sparse module from scipy
from scipy import signal           # import signal module from scipy

import os                          # import basic os functions
import time                        # import time to measure real time
import collections                 # import collections
import networkx as nx              # import networkx

data_folder = 'data'

print('Available data files:\n'+'\n'.join(sorted(os.listdir(data_folder))))

data_file_1 = os.path.join(data_folder, 'gong_net_list.txt')
data_file_2 = os.path.join(data_folder, 'macaque_net_list.txt')
```

```
net_gong_ut = [ row for row in csv.reader(open(data_file_1,'r'),delimiter='\t')]
net_gong_ut = np.array(net_gong_ut).astype(int)
net_gong_lt = net_gong_ut[:,(1,0,2)]
net_gong = np.concatenate( (net_gong_ut, net_gong_lt), axis=0)

net_row_gong = net_gong[:,0]-1
net_column_gong = net_gong[:,1]-1
net_value_gong = net_gong[:,2]
net_coord_gong = np.column_stack((net_row_gong,net_column_gong))

```

# Objectives

In this notebook, we add time-delay to the Kuramoto model, to account for the delay in transfer between brain areas. We then compute the phase-lead/lag realtionship between each pair of oscillators, which may give us clues on the information transfer between different areas of the brain. Next, we will construct functional connectivity from a Kuramoto model on network. We can compare the resulting functional connectivity with the underlying structural network.

## Background

1. There is inherent time delay in information transfer between different areas of the brain, which will likely be proportional to the length of the neural pathways (ex: fiber bundles) connecting different areas. We will add such time delay to our Kuramoto model. The time-delay Kuramoto model exhibits stable patterns of phase-lead/lag relationships between nodes, which may give us clues on the information flow pattern of the brain network.

2. Kuramoto model on a network generates a system of multiple time series. We can naturally construct a functional connectivity from the time series. This functional connectivity will resemble underlying structural connectivity (network structure) when the coupling strength $K$ of the system has a critical value.

**EXERCISE 1: Kuramoto model with time delay and phase delay, and equivalence between them **

To make Kuramoto model account for the delays between the nodes, we can add a time delay term to the model.

$$ \frac {d \theta_i} {dt} = \omega_i + \frac{K}{N} \sum_{j=1}^{N} A_{ij} \sin( \theta_j(t-\tau) - \theta_i(t)),  \quad    i=1,...N. $$

If the delay is short enough (less than the period of the oscillation), the time delay simply will act as a phase delay term:

$$ \frac {d \theta_i} {dt} = \omega_i + \frac{K}{N} \sum_{j=1}^{N} A_{ij} \sin( \theta_j(t) - \beta - \theta_i(t)),  \quad    i=1,...N. $$

• Make codes for above equations.

• Then repeat the exercises from previous tutorial with above time-delay equation with $ \tau = 10 ms $ and  phase-delay equation with $ \beta = 2\pi/10 $: using a Gong network of human brain, compute, plot, and print $ \langle r \rangle _t $ for $ K\in\{0, 3, ..., 33\} $ for each equations. Remember, to compare the results from above two equations, one must use same initial conditions for the $\theta_i $.

• Generate a time series $ \theta_i $ from the models with sufficiently large $ K $, such that  $ \langle r \rangle _t $  is high enough but not close to one (larger than 0.4 but less than 0.8: $ K $ ~ 30 would be a good candidate number to start guessing).

```
# code for computing r
def compute_r(theta):

    len_t = theta.shape[0]
    len_x = theta.shape[1]
    r_t_all = 1/len_x*np.absolute(np.sum( math.cos(theta) +  math.sin(theta)*1j ) , axis=1 )
    t_t_avg = np.mean(r_t_all)

    psi = np.angle( ( math.cos(theta_1[k+1])+math.cos(theta_2[k+1]) ) + ( math.sin(theta_1[k+1])+ math.sin(theta_2[k+1]) )*1j )

```

```
# code for time averaging r
def compute_r_tavg(r):

    r_half = r[int(0.5*len(r)):len(r)-1]
    r_tavg = np.mean(r_half)
    return r_tavg
```

```
#insert your code here for the time delayed Kuramoto model

# time delayed kuramoto model code for general network
def simulate_k_mat_time_delay(N,K,W,tau,theta_ini,t,net_coord,net_row,net_column):

    dt = t[1]-t[0]
    theta = np.zeros((len(t),N))
    r = np.zeros(len(t))
    psi = np.zeros(len(t))

    theta[0,:] = theta_ini
    r[0] = 1/N*np.absolute( np.sum(np.cos(theta[0,:]))  + np.sum(np.sin(theta[0,:]))*1j )
    psi[0] = np.angle(  np.sum(np.cos(theta[0,:])) + np.sum(np.sin(theta[0,:]))*1j )

    for k in range(0,tau):
        theta[k+1,:] = theta[k,:] + dt*( W )
        r[k+1] = 1/N*np.absolute( np.sum(np.cos(theta[k+1,:]))  + np.sum(np.sin(theta[k+1,:]))*1j )
        psi[k+1] = np.angle(  np.sum(np.cos(theta[k+1,:])) + np.sum(np.sin(theta[k+1,:]))*1j )

    for k in range(tau,len(t)-1):

        mat_size =  np.amax(net_coord)+1
        term_sin = np.sin( theta[k-tau,net_row] - theta[k,net_column] )
        term_mat = sp.sparse.coo_matrix((term_sin, (net_row,net_column)), shape=(mat_size,mat_size))
        term_mat_csc = sp.sparse.csc_matrix(term_mat)
        term_sum = term_mat_csc.sum(axis=0)

        theta[k+1,:] = theta[k,:] + dt*( W + K* term_sum )
        r[k+1] = 1/N*np.absolute( np.sum(np.cos(theta[k+1,:]))  + np.sum(np.sin(theta[k+1,:]))*1j )
        psi[k+1] = np.angle(  np.sum(np.cos(theta[k+1,:])) + np.sum(np.sin(theta[k+1,:]))*1j )

    return theta,r,psi
```

```
#insert your code here for the phase delayed Kuramoto model

# phase delayed kuramoto model code for general network
def simulate_k_mat_phase_delay(N,K,W,beta,theta_ini,t,net_coord,net_row,net_column):

    dt = t[1]-t[0]
    theta = np.zeros((len(t),N))
    r = np.zeros(len(t))
    psi = np.zeros(len(t))

    theta[0,:] = theta_ini
    r[0] = 1/N*np.absolute( np.sum(np.cos(theta[0,:]))  + np.sum(np.sin(theta[0,:]))*1j )
    psi[0] = np.angle(  np.sum(np.cos(theta[0,:])) + np.sum(np.sin(theta[0,:]))*1j )

    for k in range(len(t)-1):

        # finish your code by filling in here

        term_sum =

        theta[k+1,:] = theta[k,:] + dt*( W + K* term_sum )
        r[k+1] = 1/N*np.absolute( np.sum(np.cos(theta[k+1,:]))  + np.sum(np.sin(theta[k+1,:]))*1j )
        psi[k+1] = np.angle(  np.sum(np.cos(theta[k+1,:])) + np.sum(np.sin(theta[k+1,:]))*1j )

    return theta,r,psi
```

```
# code for comparison of two types of model (time delay and phase delay) on Gong network

# number of oscillators
N_gong = 78

# time delay
tau = 10 # in units of dt

# phase delay
beta = 2*np.pi/10

# initial frequency
W = 10*2*np.pi

# time
dt = 0.001
t = np.arange(0,10,dt)

# initial theta
random.seed()

theta_ini = np.random.rand(N_gong)*2*np.pi
theta_ini[1] = theta_ini[0] + 0.5*np.pi
theta_ini[2] = theta_ini[0] -0.5*np.pi

Ks = np.arange(0,22,2)
rs_pd = np.zeros(len(Ks))
rs_td = np.zeros(len(Ks))

theta_ini_td=theta_ini
theta_ini_pd=theta_ini

for k in range(len(Ks)):
    # what time is it now?
    rt0 = time.time()

    # core code
    theta_td,r_td,psi_td = simulate_k_mat_time_delay(N_gong,Ks[k]*(1/N_gong),W,tau,theta_ini_td,t,net_coord_gong,net_row_gong,net_column_gong)
    rs_td[k] = compute_r_tavg(r_td)
    theta_ini_td

    theta_pd,r_pd,psi_pd = simulate_k_mat_phase_delay(N_gong,Ks[k]*(1/N_gong),W,beta,theta_ini_pd,t,net_coord_gong,net_row_gong,net_column_gong)
    rs_pd[k] = compute_r_tavg(r_pd)
    theta_ini_pd

    # what time is it now and how long did it take?
    rt1 = time.time()
    print('time spent for calculation:{}s'.format(rt1-rt0))
```

```
# plot

plt.figure()
plt.plot(Ks,rs_td,'C0s')
plt.plot(Ks,rs_pd,'rd')
plt.xlabel('K')
plt.ylabel('< r >')
plt.title('Gong network')
plt.show()

print("r from time delay K-model:", rs_td)
print("r from phase delay K-model:", rs_pd)
```

** EXPECTED OUTPUT **

```
r from time delay K-model: [0.11049591 0.1577925  0.24157359 0.28125095 0.29806161 0.31194672
 0.33116812 0.36152939 0.39519595 0.42499737 0.45220356]
r from phase delay K-model: [0.11049591 0.15785225 0.24187163 0.28134417 0.29959361 0.31640776
 0.33795266 0.36941168 0.40366517 0.43349274 0.46021181]
```

```
# code to generate two types of Kuramoto model with sufficiently large coupling strength of 30

K_lg= 30

theta_td_lg,r_td_lg,psi_td_lg = simulate_k_mat_time_delay(N_gong,K_lg*(1/N_gong),W,tau,theta_ini,t,net_coord_gong,net_row_gong,net_column_gong)
rs_td_lg= compute_r_tavg(r_td_lg)
print("r from time delay K-model:", rs_td_lg)

theta_pd_lg,r_pd_lg,psi_pd_lg = simulate_k_mat_phase_delay(N_gong,K_lg*(1/N_gong),W,beta,theta_ini,t,net_coord_gong,net_row_gong,net_column_gong)
rs_pd_lg= compute_r_tavg(r_pd_lg)
print("r from phase delay K-model:", rs_pd_lg)
```

** EXPECTED OUTPUT **

```
r from time delay K-model: 0.4966947824741849
r from phase delay K-model: 0.5035572264579842
```

**EXERCISE 2: Phase difference between oscillators, and *corr* between degree and phase of each node **

we can calculate phase differences $\Delta \theta$ between each pair of oscillators:

$$ \Delta \theta_{ij} = \theta_i - \theta_j .$$

Or, we can rewrite $\Delta \theta$ in another way:

$$ \Delta \theta_{ij} = \arg( e^{i (\theta_i - \theta_j)} ),$$

which means we take the angle of $ e^{i (\theta_i - \theta_j)} $ from the real axis. We will follow the second representation for our computation.

This phase difference may give a valuable information about the "directionality" between pair of oscillators.

• First, from the phase-delay (or time-delay) time series we generated, calculate the $\theta_{ij}$ between all pairs of time series, and build a phase-difference matrix. The resulting matrix will be anti-symmetric. Use only the latter half of time series in the process.

• From the phase-difference matrix, we can compute the average phase-difference for each node. Calculate the row-sum of the matrix:

$$ \theta_i = 1/N \sum_{j=1}^{N} \theta_{ij},$$

then we can have a vector of averaged phase-differences, each element of the vector corresponding for each node.

• Finally, do a scatterplot of the average phase-difference and degree for each node. Is there any pattern between these two quantities? Calculate the Pearson correlation coefficient between these two vectors.

```
# insert your code for calculating phase difference

def phase_diff_mat(theta):

    # theta must has dimension TxN, where T is the length of time points and N is the number of nodes

    N_len = theta.shape[1]

    PDiff_mat= np.zeros((N_len,N_len))
    for ch1 in range(N_len):
        for ch2 in range(ch1+1,N_len):
            PDiff=theta[:,ch1]-theta[:,ch2]
            PDiff_exp = np.angle( np.exp(1j*PDiff) )
            PDiff_exp_mean = np.mean(PDiff_exp)
            PDiff_mat[ch1,ch2] = PDiff_exp_mean
            PDiff_mat[ch2,ch1] = -1*PDiff_exp_mean

    PDiff_mean = np.mean(PDiff_mat,axis=1)

    #alternative code
    #arr = np.array([np.roll(theta, i, axis=1) for i in range(N_len)])
    #PDiff_mat = theta[None, :] - arr
    #PDiff_mean = PDiff_mat.mean(1)

    return PDiff_mean,PDiff_mat
```

```
# calculate the phase difference

theta_pd_mean,theta_pd_mat = phase_diff_mat(theta_pd[-int(theta_pd.shape[0]/2):,:])

# calculate the degree of each node of gong network

net_gong_size =  np.amax(net_coord_gong)+1
net_gong_mat = sp.sparse.coo_matrix((net_value_gong, (net_row_gong,net_column_gong)), shape=(net_gong_size,net_gong_size) )
net_gong_mat_csc = sp.sparse.csc_matrix(net_gong_mat)
net_gong_degree = np.sum(net_gong_mat_csc,axis=0).T

# plot
plt.figure()
plt.plot(net_gong_degree,theta_pd_mean,'C0s')
plt.xlabel('K')
plt.ylabel('theta')
plt.title('theta vs K')
plt.show()

# corr
corr_degree_phase = np.corrcoef(net_gong_degree,theta_pd_mean)
print("corr. of degree and phase of each node:", corr_degree_phase[0,1])
```

**EXPECTED OUTPUT**

```
corr. of degree and phase of each node: -0.5463676447067697
```

**EXERCISE 3 (optional, possible for mini-project) : constructing functional connectivity from the oscillators via calculating $ PLI $ between them**

We begin with the previous phase-delayed Kuramoto equations:

$$ \frac {d \theta_i} {dt} = \omega_i + \frac{K}{N} \sum_{j=1}^{N} A_{ji} \sin( \theta_j - \theta_i - \beta),  \quad    i=1,...N, $$

where $A_{ji}$ is the connectivity between two nodes $i$ and $j$ (with value of either 1 or 0).*italicized text*

• Using a Gong network of human brain, give initial phase $\theta_i$ randomly, and let coupling strength $K$ vary from 0.1 to 1 by increment of 0.1.

• At each coupling strength, compute the average $ \langle r \rangle _t $ (choose only the latter-half of time series).

• At each coupling strength, measure $ PLI_{ij} $ between the time series. Again, choose only the latter-half of time series. Ultimately, construct a $ PLI $ matrix containing all the $ PLI_{ij} $ between possible pairs. The resulting matrix will be symmetric.

• Compute the average $ PLI $ for each strength $K$ (sum of all $ PLI_{ij} $s of the $ PLI $ matrix).  Plot the average $ \langle r \rangle _t $ and average $ PLI $ for each coupling strength $K$. What kind of patterns do you observe from the changes of  $ \langle r \rangle _t $  and  average $ PLI $ with respect to the changes in $K$?

```

```

**EXERCISE 4 (optional, possible for mini-project) : Calculating $corr$ between structural connectivity network and functional connectivity network**

With the $ PLI $ matrix we built in exercise 3, we can compare against structural connectivity.

• First, give a threshold of 0.3, and binarize the connectivity matrix. If the matrix element is larger than 0.3, it will turn into value of 1, otherwise 0. In the end, we will have one connectivity matrix per one value of coupling strength $ K $.

• After constructing the connectivity matrix, compute the degree for each node. In the due process, we will have a vector consisting of the degree for each node, per one value of coupling strength $ K $.

• For Gong network, we can also compute a vector of degrees for each node.

• Finally, compute the Pearson correlation $ c $ between the degree vector from the functional connecvtivity, and the degree vector from Gong network. In the graph of Exercise 4, add a plot of the correalation $ c $ as a function of time. At which coupling strength $ K $ does the correlation $ c $ has largest value? At this $ K $, what is the value of $ r(t) $ and average $ corr $? We will call this coupling strength the critical strength $ Kc $.

```

```

**EXERCISE 5 : Runge-Kutta method (optional, possible for mini-project)**

Remerber in the first module how we learned to compute differential equations via Euler method? In the Wilson-Cowan model, we discretized $dt$ into $\Delta t$. To requote:

\begin{align}
\frac{dE}{dt} = f(E,I,t) \to \frac{E(t+\Delta t)- E(t)}{\Delta t}&= f(E,I,t) \hspace{5 mm}\text{ and }\hspace{5mm} \\ \frac{dI}{dt} = g(E,I,t) \to \frac{I(t+\Delta t)-I(t)}{\Delta t} &=g(E,I,t),\\
\end{align}

henceforth equating the future trajectory of $E(t)$  and $I(t)$ in the following way:

\begin{align}
E(t+\Delta t) = &E(t) + \Delta t f(E,I,t) \hspace{5 mm}\text{ and }\hspace{5mm}\ \\
I(t+\Delta t) = &I(t) + \Delta t g(E,I,t). \\
\end{align}

 Euler method has served us well, but in order to simulate more complex differential equations, more refined method is needed.

 If we inspect Euler method closely, there is a inherent asymmetry in the right side of the equation. We only evaluate the value of $f(E,I,t)$ or $g(E,I,t)$ at time $t$. Therefore, we are predicting the (future) value of $E(t+\Delta t)$ solely based on the values at the current moment, $t$.

Based on this observation, we can generalize Euler method. Here we introduce (2nd order) Runge-Kutta method, which is exactly a generalization of Euler method towards resolving this asymmetry. With Runge-Kutta method, we begin with writing Wilson-Cowan equation as the following:

$$ \frac{E(t+\Delta t)- E(t)}{\Delta t} = \frac{1}{2} \{ f(E(t),I(t),t)+ f(E(t+\Delta t), I(t+\Delta t),t+\Delta t) \},$$

Now, as an approximation, we write:

\begin{align}
\tilde E(t+\Delta t) &= E(t) + \Delta t f(E(t),I(t),t) \\
E(t+\Delta t) &= E(t) + \frac{\Delta t}{2} \{f(E(t),I(t),t) +f(\tilde E(t+\Delta t),\tilde I(t+\Delta t),t+\Delta t) \}, \\
\end{align}

and similarly for the $I$.

• Write an expression of Runge-Kutta method applied to the following Kuramoto model with two nodes:

$$ \frac {d \theta_1(t)} {dt} = \omega_1 + \frac{K}{N} \sin( \theta_2(t) - \theta_1(t)),  \quad  \\  \frac {d \theta_2(t)} {dt} = \omega_2 + \frac{K}{N} \sin( \theta_1(t) - \theta_2(t)) .$$

• Write the code for above equation. The way one wants to do it, is to define a function for R-K2 method and put $f(t)$ as an input to the function. Compare the result is similar with the Euler method of Tutorial #3, exercise 2.

```
def euler( f, x0, t ):
    """ AUTHOR:
        Jonathan Senning <jonathan.senning@gordon.edu>
        Gordon College
        Based Octave functions written in the spring of 1999
        Python version: March 2008, October 2008
    """
    """Euler's method to solve x' = f(x,t) with x(t[0]) = x0.

    USAGE:
        x = euler(f, x0, t)

    INPUT:
        f     - function of x and t equal to dx/dt.  x may be multivalued,
                in which case it should a list or a NumPy array.  In this
                case f must return a NumPy array with the same dimension
                as x.
        x0    - the initial condition(s).  Specifies the value of x when
                t = t[0].  Can be either a scalar or a list or NumPy array
                if a system of equations is being solved.
        t     - list or NumPy array of t values to compute solution at.
                t[0] is the the initial condition point, and the difference
                h=t[i+1]-t[i] determines the step size h.

    OUTPUT:
        x     - NumPy array containing solution values corresponding to each
                entry in t array.  If a system is being solved, x will be
                an array of arrays.
    """

    n = len( t )
    x = np.array( [x0] * n )
    for i in range( n - 1 ):
        x[i+1] = x[i] + ( t[i+1] - t[i] ) * f( x[i], t[i] )

    return x
```

```
# new code for 2nd order Runge-Kutta method

def rk2( f, x0, t ):
    """ AUTHOR:
        Jonathan Senning <jonathan.senning@gordon.edu>
        Gordon College
        Based Octave functions written in the spring of 1999
        Python version: March 2008, October 2008
    """
    """Heun's method to solve x' = f(x,t) with x(t[0]) = x0.

    USAGE:
        x = rk2(f, x0, t)

    INPUT:
        f     - function of x and t equal to dx/dt.  x may be multivalued,
                in which case it should a list or a NumPy array.  In this
                case f must return a NumPy array with the same dimension
                as x.
        x0    - the initial condition(s).  Specifies the value of x when
                t = t[0].  Can be either a scalar or a list or NumPy array
                if a system of equations is being solved.
        t     - list or NumPy array of t values to compute solution at.
                t[0] is the the initial condition point, and the difference
                h=t[i+1]-t[i] determines the step size h.

    OUTPUT:
        x     - NumPy array containing solution values corresponding to each
                entry in t array.  If a system is being solved, x will be
                an array of arrays.
    """
    n = len( t )
    x = np.array( [x0] * n )
    #print(x.shape)
    for i in range( n - 1 ):
        h = t[i+1] - t[i]
        k1 = h * f( x[i], t[i] )
        k2 = h * f( x[i] + k1, t[i+1] )
        x[i+1] = x[i] + ( k1 + k2 ) / 2.0

    return x
```

```
def f( x, t ):
        return x * np.sin( t )

a, b = (0.0, 15.0)
x0 = -1.0

n = 151
t = np.linspace( a, b, n )
h = t[1] - t[0];

# compute various numerical solutions
x_euler = euler( f, x0, t )
x_heun = rk2( f, x0, t )

# compute true solution values in equal spaced and unequally spaced cases
x = -np.exp( 1.0 - np.cos( t ) )

plt.figure()
#    figure( 1 )
plt.subplot( 2, 1, 1 )
plt.plot( t, x_euler, 'b-o', t, x_heun, 'g-o' ,t, x, 'r-*'  )
plt.xlabel( '$t$' )
plt.ylabel( '$x$' )
plt.title( 'Solutions of $dx/dt = x \sin t$, $x(0)=-1$ ($h = %4.2f$)' % h )
plt.legend( ( 'Euler', 'Heun' ,'true'), loc='lower left' )

#    figure( 2 )
plt.subplot( 2, 1, 2 )
plt.plot( t, (x_euler - x), 'b-o', t, (x_heun - x), 'g-o' )
plt.xlabel( '$t$' )
plt.ylabel( '$x - x^*$' )
plt.title( 'Errors in sol. of $dx/dt = x \sin t$, $x(0)=-1$ ($h = %4.2f$)' % h )
plt.legend( ( 'Euler', 'Heun' ), loc='upper left' )

plt.tight_layout(pad=0.4, w_pad=0.5, h_pad=1.0)
plt.show()
```


$$ \LaTeX \text{ command declarations here.}
\newcommand{\R}{\mathbb{R}}
\renewcommand{\vec}[1]{\mathbf{#1}}
$$

```python
%matplotlib inline
from Lec05 import *
```

# EECS 445: Introduction to Machine Learning
## Lecture 05:  Linear Regression II
* Instructor:  **Jacob Abernethy** and **Jia Deng**
* Date:  September 21, 2016

*Lecture Exposition Credit: Benjamin Bray and Zhe Du*

## Outline for this Lecture
- Overfitting
- Regularized Least Squares
- Maximum Likelihood Interpretation of Linear Regression

## Reading List
- Suggested:
    - **[PRML]**, §3.2: The Bias-Variance Decomposition
    - **[PRML]**, §3.3: Bayesian Linear Regression
- Optional:
    - **[MLAPP]**, Chapter 7: Linear Regression

> In this lecture, we will first look at how degree of linear regression and sample dataset size will cause *overfitting* in linear regression. To deal with overfitting, *regularized least squares* will be introduced.  Finally, we will show regular linear regression and regularized linear regression can be interpreted from probablistic perspective each using *maximum likelihood estimation* and *maximum a posteriori estimation*.

## Overfitting

### Overfitting: Degree of Linear Regression

```python
regression_overfitting_degree(degree0=0, degree1=3,degree2=9,degree3=12)
```

> **Remark**
>    - In the above plots, we try to predict the true sinusoidal curve hidden in the data with polynomial degrees 0, 3, 9 and 12.
>    - In the first plot (degree=0), we only get a horizontal line. The learned curve can neither fit the training data nor match true curve. This is called **Underfitting**.
>    - In the second plot (degree=3), the predicted plot fits both data and true curve perfectly. This is a good degree for our setting.
>    - In the third (degree=9) and fourth (degree=12) plots, as the degree increases, the traning data are fitted better but the learned curve deviates from the true curve further. This is called **Overfitting**.
>    - Explanations of why degree could impact the predicted curve will come in the **Remark** later.

### Overfitting: Dataset Size

```python
regression_overfitting_datasetsize(size0 = 13, size1 = 50, size2 = 100, size3 = 500)
```

> **Remark**
>   - In the above plots, we try to predict the true sinusoidal curve hidden in the data with polynomial degree 12 and training dataset size 13, 50, 100 and 500.
>   - In the first plot (size=13), **overfitting** occurs as we have seen just now.
>   - Comparing the four plots altogether, we could see as the size increases, overfitting diminishes. And although more and more data points cannot be fitted by the learned curve (training error is becoming higher), but it matches the true curve better (test error will be smaller).
>   - Explanations of why training data size impact the predicted curve will come in the **Remark** later.

### Overfitting: Overall Performance

* On the left plot, we fix the dataset size and vary the polynomial degree
* On the right plot, we fix the polynomial degree and vary the dataset size

```python
regression_overfitting_curve()
```

> **Remark**
>   - The plot below is the **root mean squared error (RMSE)** of training dataset and test dataset with respect to different degree and dataset size.
>   - NOTE: For simplicity of presentation, we divided the dataset into training set and test set. However, it's not legitimate to find the optimal hyperparameter based on the test set. We will talk about legitimate ways of doing this when we cover model selection and cross-validation.
>   - Combining the last 10 plots, we could have:
>   - **Degree**
>       - When degree is really small, the regressor is not powerful enough (i.e. degree is small) to learn the underlying true model. At this time, underfitting occurs. Both training error and test error are high.
>       - As degree increases, regressor become more powerful enough to roughly learn the true model but not that powerful to also take the noise into considerations. At this time, underfitting reduces. Both training error and test error will be smaller.
>       - As degree further increases, regressor become so powerful that it could fit most of the data in training dataset perfectly. And since the data are noisy, the learned model actually deviates from the true model. At this time, overfitting occurs. Training error could becomes very small (even 0 sometimes), while test error increases.
>   - **Dataset Size**
>       - When training dataset size is small, a powerful regressor (i.e. degree is high) can fit every single sample in training dataset. Therefore, noise is also considered. This is equivalent to the ending zone of the plot (left) with respect degree.
>       - As training dataset size increases, since the power of regressor is finite (degree is fixed), regressor starts to fail to fit every single sample. Instead it seeks to learn a curve such that samples will fall on both sides equally. Since the noise are assumed to 0 mean Gaussian noise which is symmetric with respect to 0, this is actually close to the underlying true curve.
>       - The training error and test error will converge to 0.3 which is the variance of noise (you could check this by examining the Python code). This is not a coincidence and can be derived. We will cover this later when we study **bias-variance tradeoff**
>   - $$\text{RMSE} = \sqrt{\frac{1}{N} \sum_{n=1}^{N} (y_n-t_n)^2}$$

### Rule of Thumb to Choose the Degree
- For a small number of datapoints, use a low degree
    - Otherwise, the model will overfit!
- As you obtain more data, you can gradually increase the degree
    - Add more features to represent more data
    - **Warning:**  Your model is still limited by the finite amount of data available.  The optimal model for finite data cannot be an infinite-dimensional polynomial!)
- Use **regularization** to control model complexity.

## Regularized Linear Regression

### Coefficients of Overfitting

- Before we move to regularized linear regression, let's first look at what happened to the coefficients $\vec{w}$ when there is overfitting.

```python
regression_overfitting_coeffs()
```

      M=0 (Underfitting)
      M=3 (Good)
      M=9 (Overfitting)
      M=12 (Overfitting)

      w_0
      1.52097
      7.39357
      0.0480902
      0.070104

      w_1

      -65.5216
      -1.37193
      -2.761237

      w_2

      120.601
      16.7429
      47.783685

      w_3

      24.0947
      -114.303
      -477.641436

      w_4

      476.559
      3049.898178

      w_5

      -1232.81
      -12989.706091

      w_6

      1910.46
      37403.890882

      w_7

      -1639.33
      -72227.191816

      w_8

      658.622
      90733.895634

      w_9

      4.95618
      -69700.637614

      w_10

      28897.440780

      w_11

      -4666.324657

      w_12

      4.516949

> **Remark**
>    - The table above corresponds to the coefficients (multiplied by 100 for better visualization) for different degrees.
>    - We could see that when overfitting occurs, we get some really crazy and large numbers!
>    - So one intuition to handle overfitting is to *penalize* large coefficients.

### Regularized Least Squares: Objective Function

- Recall the objective function we minimizes in last lecture is
$$
E(\vec{w}) = \frac12 \sum_{n=1}^N \left( \vec{w}^T \phi(\vec{x}_n) - t_n \right)^2
$$

- To penalize the large coefficients, we will add one penalization/regularization term to it and minimize them altogether.
$$
E(\vec{w}) = \underbrace{ \frac12 \sum_{n=1}^N \left( \vec{w}^T \phi(\vec{x}_n) - t_n \right)^2 }_{E_D(\vec{w})}+ \underbrace{\boxed{\frac{\lambda}{2} \left \| \vec{w} \right \|^2}}_{E_W(\vec{w})}
$$
of which $E_D(\vec{w})$ represents the term of sum of squared errors and $E_W(\vec{w})$ is the regularization term.

- $\lambda$ is the regularization coefficient.
- If $\lambda$ is large, $E_{\vec{W}}(\vec{w})$ will dominate the objective function. As a result we will focus more on minimizing $E_W(\vec{w})$ and the resulting solution $\vec{w}$ tends to have smaller norm and the $E_D(\vec{w})$ term will be larger.

### Regularized Least Squares: Derivation
- Based on what we have derived in last lecture, we could write the objective function as
$$
\begin{align}
E(\vec{w})
&= \frac12 \sum_{n=1}^N \left( \vec{w}^T \phi(\vec{x}_n) - t_n \right)^2 + \frac{\lambda}{2} \left \| \vec{w} \right \|^2 \\
&= \frac12 \vec{w}^T \Phi^T \Phi \vec{w} - \vec{t}^T \Phi \vec{w} + \frac12 \vec{t}^T \vec{t} + \frac{\lambda}{2}\vec{w}^T\vec{w}
\end{align}
$$

- The gradient is
$$
\begin{align}
\nabla_\vec{w} E(\vec{w})
&= \Phi^T \Phi \vec{w} - \Phi^T \vec{t} + \lambda \vec{w}\\
&= (\Phi^T \Phi + \lambda I)\vec{w} - \Phi^T \vec{t}
\end{align}
$$

- Setting the gradient to 0, we will get the solution
$$
\boxed{ \hat{\vec{w}}=(\Phi^T \Phi + \lambda I)^{-1} \Phi^T \vec{t} }
$$

- In the solution to ordinary least squares which is $\hat{\vec{w} }=(\Phi^T \Phi)^{-1} \Phi^T \vec{t}$, we cannot guarantee $\Phi^T \Phi$ is invertible. But in regularized least squares, if $\lambda > 0$, $\Phi^T \Phi + \lambda I$ is always invertible.

### Regularized Least Squares:  Different Norms
- The $\ell^p$ norm of a vector $\vec{x}$ is defined as
$$
\left \| \vec{x} \right \|_p = (\sum_{j=1}^{M} |x_j|^p)^\frac{1}{p}
$$

- For the regularized least squares above, we used $\ell^2$ norm. We could also use other $\ell^p$ norms for different regularizers and the objective function becomes
$$
E(\vec{w}) = \frac12 \sum_{n=1}^N \left( \vec{w}^T \phi(\vec{x}_n) - t_n \right)^2 + \frac{\lambda}{2} \left \| \vec{w} \right \|_p^p
$$

(Contour of Different p-norms)

- **Lasso** regularization ($\ell^1$ regularization) tends to generate sparser solutions than **ridge** regression ($\ell^2$ regularization)

(Image taken from [here](http://gerardnico.com/wiki/data_mining/lasso))

> **Remark**
>    - RSS is residual of sum of squares, which is the sum of squared errors $E_D(\vec{w})$ we use.
>    - This plot is to illustrate intuitively why lasso has sparser solution than ridge regression.
>    - Our objective function is
$$
E(\vec{w}) = \frac12 \sum_{n=1}^N \left( \vec{w}^T \phi(\vec{x}_n) - t_n \right)^2 + \frac{\lambda}{2} \left \| \vec{w} \right \|_p^p
$$
To illustrate, lets look at an *equivalent* constrained problem. (Not exactly equivalent, just for illustration purpose)
$$
\begin{aligned}
& {\text{minimize}} &  & \frac12 \sum_{n=1}^N \left( \vec{w}^T \phi(\vec{x}_n) - t_n \right)^2\\
& \text{subject to} &  & \frac{\lambda}{2} \left \| \vec{w} \right \|_p^p \leq C
\end{aligned}
$$
>    - The objective function $\frac12 \sum_{n=1}^N \left( \vec{w}^T \phi(\vec{x}_n) - t_n \right)^2$ is the red contours in the plots. The optimal solution without the constraint is $\hat{\vec{w}}_{OLS}$, which is just the solution to ordinary least squares. The farther we are away from $\hat{\vec{w}}_{OLS}$ the larger objective function will be.
>    - The feasible area that satisfying the constraint is the cyan area in the plots. The solution must fall in these areas. For lasso, it's a diamond; for ridge regression, it's a circle.
>    - As we increase the value of objective function, the contour will expand. The first time the contour *touch* the feasible cyan area, the touching point would be the optimal solution. This is because our solution should both be as close to $\hat{\vec{w}}_{OLS}$ as possible and fall in the feasible area.
>    - Since lasso has diamond area with four corners, the contours tend to touch the corner first. And since the corner is on the axis where coordinate has at least one zero component, this guarantees the sparsity of solution. On the contrary, the circle area in ridge regression cannot give us this property.

### Regularized Least Squares: Example

```python
regression_regularization_plot()
```

> **Remark**
>    - In the second plot, we can see that after the regularization term is added, the learned curve looks much more like the true curve than the learnd curve without regularization in the first plot.
>    - However, in the third plot, when the coefficient $\lambda$ for regularization is too large, the minimization will mostly focus on minimizing $\left \| \vec{w} \right \|$, thus deviating from the true curve in a great deal. (We will see the coefficients $\vec{w}$) are really small for this case in next slide)
>    - In the fourth plot, as we increase $\lambda$, the training error is monotonically increasing because we are far away from the task of minimizing sum of squared error $E_D(\vec{w})$. As for the the test error, it has minimum value near $\lambda=1$, which is the blancing point in the tradeoff between minimizing $E_D(\vec{w})$ and minimizing regularization term $E_W(\vec{w})$.

### Regularized Least Squares: Coefficients
- Let's look at how the coefficients change after we add regularization

```python
regression_regularization_coeff()
```

      lambda=0
      lambda=exp^1
      lambda=exp^10

      w_0
      -16.965621
      10.877126
      0.012736

      w_1
      -1292.159565
      11.986821
      0.018990

      w_2
      5229.010114
      13.100179
      0.031047

      w_3
      -7982.812459
      12.480543
      0.049577

      w_4
      6463.732308
      7.476729
      0.070690

      w_5
      -3091.570578
      -2.806526
      0.076215

      w_6
      910.209395
      -9.075007
      0.031334

      w_7
      -165.687039
      5.979848
      -0.055185

      w_8
      18.028087
      -1.485710
      0.012029

      w_9
      -1.062826
      0.167608
      -0.000419

      w_10
      0.025676
      -0.007200
      -0.000045

> **Remark**
>    - From the table above, we can see the regularization term has effectively constrained those huge coefficients.
>    - However, when $\lambda$ is large ($\lambda = e^{10}$), the coefficients are "over-regularized".

### Regularized Least Squares:  Summary

- Simple modification of linear regression
- $\ell^2$ Regularization controls the tradeoff between *fitting error* and *complexity*.
    - Small $\ell^2$ regularization results in complex models, but with risk of overfitting
    - Large $\ell^2$ regularization results in simple models, but with risk of underfitting
- It is important to find an optimal regularization that *balances* between the two

## Probablistic Interpretation of Least Squares Regression
- We have showed derived the solution to least squares regression by minimizing objective function. Now we will provide a probablistic perspective. Specifically, we will show the solution to **regular least squares** is just the **maximum likelihood** estimate of $\vec{w}$ and the solution to **regularized least squares** is the **Maximum a Posteriori** estimate.

### Some Background
- Gaussian Distribution
$$
\mathcal{N}(x, \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left[ \frac{(x-\mu)^2}{2\sigma^2} \right]
$$

- **Maximum Likelihood Estimation** and **Maximum a Posteriori Estimation (MAP)**
    - For distribution $t \sim p(t|\theta)$. $\theta$ is some unknown parameter (like mean or variance) to be estimated.
    - Given observation $\vec{t} = (t_1, t_2, \dots, t_N)$,
        - The Maximum Likelihood Estimator is
        $$
        \theta_{ML} = \arg \max \prod_{n=1}^N p(t_n | \theta)
        $$
        - If we have some prior knowledge about $\theta$, the MAP estimator is
        $$
        \theta_{MAP} = \arg \max \prod_{n=1}^N p(\theta | t_n) \quad (\text{Posteriori Probability of } \theta)
        $$

### Maximum Likelihood Estimator $\vec{w}_{ML}$

- We assume the **signal+noise** model of single data $(\vec{x}, t)$ is
    $$
    \begin{gather}
    t = \vec{w}^T \phi(\vec{x}) + \epsilon \\
    \epsilon \sim \mathcal{N}(0, \beta^{-1})
    \end{gather}
    $$
    of which $\vec{w}^T \phi(\vec{x})$ is the true model, $\epsilon$ is the perturbation/randomness.

- Since $\vec{w}^T \phi(\vec{x})$ is deterministic/non-random, we have
    $$
    t \sim \mathcal{N}(\vec{w}^T \phi(\vec{x}), \beta^{-1})
    $$

- The **likelihood function** of $t$ is just **probability density function (PDF)** of $t$
    $$
    p(t|\vec{x},\vec{w},\beta) = \mathcal{N}(t|\vec{w}^T \phi(\vec{x}),\beta^{-1})
    $$

- For inputs $\mathcal{X}=(\vec{x}_1, \dots, \vec{x}_n)$ and target values $\vec{t}=(t_1,\dots,t_n)$, the data likelihood is
    $$
    p(\vec{t}|\mathcal{X},\vec{w},\beta)
    = \prod_{n=1}^N p(t_n|\vec{x}_n,\vec{w},\beta)
    = \prod_{n=1}^N \mathcal{N}(t_n|\vec{w}^T\phi(\vec{x}_n),\beta^{-1})
    $$

- **Notation Clarification**
    - $p(t|x,w,\beta)$ it the PDF of $t$ whose distribution is parameterized by $x,\vec{w},\beta$.
    - $\mathcal{N}(\vec{w}^T \phi(\vec{x}), \beta^{-1})$ is Gaussian distribution with **mean** $\vec{w}^T \phi(\vec{x})$ and **variance** $\beta^{-1}$.
    - $\mathcal{N}(t|\vec{w}^T \phi(\vec{x}),\beta^{-1})$ is the PDF of $\vec{t}$ which has Gaussian distribution $\mathcal{N}(\vec{w}^T \phi(\vec{x}), \beta^{-1})$

### Maximum Likelihood Estimator $\vec{w}_{ML}$
- **Main Idea of Maximum Likelihood Estimate**
    - Given $\{ \vec{x}_n, t_n \}_{n=1}^N$, we want to find $\vec{w}_{ML}$ that maximizes data likelihood function
        $$
        \vec{w}_{ML}
        =\arg \max p(\vec{t}|\mathcal{X},\vec{w},\beta)
        =\arg \max \prod_{n=1}^N \mathcal{N}(t_n|\vec{w}^T\phi(\vec{x}_n),\beta^{-1})
        $$
        and by derivation we will show $\vec{w}_{ML}$ is equivalent to the least squares solution $\hat{\vec{w}} = \Phi^\dagger \vec{t}$.

- **Intuition about Maximum Likelihood Estimation**
    - Finding maximum likelihood estimate $\vec{w}_{ML} = \arg \max p(\vec{t}|\mathcal{X},\vec{w},\beta)$ is just finding the parameter $\vec{w}$ under which for data $\mathcal{X}=(\vec{x}_1, \dots, \vec{x}_n)$, observed $\vec{t}=(t_1,\dots,t_n)$ is the most likely result to be generated among all possible $\vec{t}$.

### Maximum Likelihood Estimator $\vec{w}_{ML}$: Derivation
- Single data likelihood is
    $$
    p(t_n|\vec{x}_n,\vec{w},\beta)
    = \mathcal{N}(t_n|\vec{w}^T\phi(\vec{x}_n),\beta^{-1})
    = \frac{1}{\sqrt{2 \pi \beta^{-1}}} \exp \left \{ - \frac{1}{2 \beta^{-1}} (t_n - \vec{w}^T \phi(x_n))^2 \right \}
    $$

- Single data log-likelihood is
    $$
    \ln p(t_n|\vec{x}_n,\vec{w},\beta) = - \frac12 \ln 2 \pi \beta^{-1} - \frac{\beta}{2} (\vec{w}^T \phi(x_n) - t_n)^2
    $$
    We use logarithm because maximizer of $f(x)$ is the same as maximizer of $\log f(x)$. Logarithm can convert product to summation which makes life easier.

- Complete data log-likelohood is
    $$
    \begin{align}
    \ln p(\vec{t}|\mathcal{X},\vec{w},\beta)
    &= \ln \left[ \prod_{n=1}^N p(t_n|\vec{x}_n,\vec{w},\beta) \right] = \sum_{n=1}^N \ln p(t_n|\vec{x}_n,\vec{w},\beta) \\
    &= \sum_{n=1}^N \left[ - \frac12 \ln 2 \pi \beta^{-1} - \frac{\beta}{2} (\vec{w}^T \phi(x_n) - t_n)^2 \right]
    \end{align}
    $$

- Maximum likelihood estimate $\vec{w}_{ML}$ is
    $$
    \begin{align}
    \vec{w}_{ML}
    &= \underset{\vec{w}}{\arg \max} \ln p(\vec{t}|\mathcal{X},\vec{w},\beta) \\
    &= \underset{\vec{w}}{\arg \max}  \sum_{n=1}^N \left[ - \frac12 \ln 2 \pi \beta^{-1} - \frac{\beta}{2} (\vec{w}^T \phi(x_n) - t_n)^2 \right] \\
    &= \underset{\vec{w}}{\arg \max}  \sum_{n=1}^N \left[ - \frac{\beta}{2} (\vec{w}^T \phi(x_n) - t_n)^2 \right] \\
    &= \underset{\vec{w}}{\arg \min}  \sum_{n=1}^N \left[(\vec{w}^T \phi(x_n) - t_n)^2 \right]
    \end{align}
    $$

- Familiar? Recall the objective function we minimized in least squares is $E(\vec{w}) = \frac12 \sum_{n=1}^N \left( \vec{w}^T \phi(\vec{x}_n) - t_n \right)^2$, so we could conclude that
    $$
    \boxed{\vec{w}_{ML}  = \hat{\vec{w}}_{LS} = \Phi^\dagger \vec{t}}
    $$

### MAP Estimator $\vec{w}_{MAP}$
- The **MAP estimator** is obtained by
    $$
    \begin{align}
    \vec{w}_{MAP}
    &= \arg \max p(\vec{w}|\vec{t}, \mathcal{X},\beta) & & (\text{Posteriori Probability})\\
    &= \arg \max \frac{p(\vec{w}, \vec{t}, \mathcal{X},\beta)}{p(\mathcal{X}, t, \beta)} \\
    &= \arg \max \frac{p(\vec{t}|\vec{w}, \mathcal{X},\beta) p(\vec{w}, \mathcal{X}, \beta)}{p(\mathcal{X}, t, \beta)} \\
    &= \arg \max p(\vec{t}|\vec{w}, \mathcal{X},\beta) p(\vec{w}, \mathcal{X}, \beta) & & (p(X, t, \beta) \text{ is irrelevant to} \ \vec{w})\\
    &= \arg \max p(\vec{t}|\vec{w}, \mathcal{X},\beta) p(\vec{w}) p(\mathcal{X}) p(\beta) & & (\text{Independence}) \\
    &= \arg \max p(\vec{t}|\vec{w}, \mathcal{X},\beta) p(\vec{w}) & & (\text{Likelihood} \times \text{Prior})
    \end{align}
    $$
    We are just using **Bayes Theorem** for the above steps.
- The only difference from ML estimator is we have an extra term of PDF of $\vec{w}$. This is the **prior belief** of $\vec{w}$. Here, we assume,
    $$
    \vec{w} \sim \mathcal{N}(\vec{0}, \alpha^{-1}I)
    $$

- ML vs. MAP
    - Maximum Likelihood: We know **nothing** about $\vec{w}$ initially and every $\vec{w}$ are equally likelihood
    - Maximum a Posteriori: We know **something** about about $\vec{w}$ initially and certain $\vec{w}$ are more likely (depending on **prior** $p(\vec{w})$). In another way, $\vec{w}$ are weighted.
- Assumption $\vec{w} \sim \mathcal{N}(\vec{0}, \alpha^{-1}I)$ makes sense because
    - In **regularized least squares**
        - We already know large coefficient $\vec{w}$ that may lead to overfitting should be avoided.
        - When we increase the regularization coefficient $\lambda$, the smaller $\left \| \vec{w} \right \|$ will be.
    - When use $\vec{w} \sim \mathcal{N}(\vec{0}, \alpha^{-1}I)$
        - $\vec{w} \sim \mathcal{N}(\vec{0}, \alpha^{-1}I)$ encodes the assumption that $\vec{w}$ with a smaller norm $\left \| \vec{w} \right \|$ is more "likely" than a $\vec{w}$ with a bigger norm.
        - When we increase $\alpha$, variance is smaller, small $\left \| \vec{w} \right \|$ will be much more likely

### MAP Estimator $\vec{w}_{MAP}$: Derivation
- $\vec{w} \sim \mathcal{N}(\vec{0}, \alpha^{-1}I)$ is **multivariate Gaussian** which has PDF
    $$
    p(\vec{w}) = \frac{1}{\left( \sqrt{2 \pi \alpha^{-1}} \right)^N} \exp \left \{ -\frac{1}{2 \alpha^{-1}} \sum_{n=1}^N w_n^2 \right \}
    $$

- So the MAP estimator is
    $$
    \begin{align}
     \vec{w}_{MAP}
     &= \underset{\vec{w}}{\arg \max} \ p(\vec{t}|\vec{w}, \mathcal{X},\beta) p(\vec{w}) = \underset{\vec{w}}{\arg \max} \left[\ln p(\vec{t}|\vec{w}, \mathcal{X},\beta) + \ln p(\vec{w}) \right] \\
     &= \underset{\vec{w}}{\arg \min} \left[ \sum_{n=1}^N \frac{\beta}{2} (\vec{w}^T \phi(x_n) - t_n)^2  +  \frac{\alpha}{2} \sum_{n=1}^N w_n^2 \right] \\
     &= \underset{\vec{w}}{\arg \min} \left[ \sum_{n=1}^N \frac12 (\vec{w}^T \phi(x_n) - t_n)^2  +  \frac12 \frac{\alpha}{\beta} \left \| \vec{w} \right \|^2 \right]
    \end{align}
    $$

- Exactly the objective in regularized least squares! So
$$
\boxed{ \vec{w}_{MAP} = \hat{\vec{w}}=\left(\Phi^T \Phi + \frac{\alpha}{\beta} I\right)^{-1} \Phi^T \vec{t} }
$$

> ** Remark **
>    - Of the above expression, $\frac{\alpha}{\beta}$ corresponds to the regularization coefficient $\lambda$ we used in previous regulazied least squares.
>    - **Priors**: Represent prior beliefs about acceptable values for model parameters.
>    - Example: In linear regression, $\ell^2$ regularization can be interpreted as placing a Gaussian Prior on the regression coefficients.
>    - All statistical models and machine learning algorithms make assumptions.
>        - All reasoning is based on implicit assumptions.
>        - A Bayesian will tell you that his prior is a way of explicitly stating those assumptions.
>    - This can all get very philosophical, but...
>        - Bayesian reasoning is best seen as a useful tool.
>        - Many concepts in machine learning have Bayesian interpretations.
>            - Choice of loss / error function, regularization, etc.
>    - For a fully Bayesian take on machine learning, check out the **Murphy** textbook:

>    - We will cover more about Bayesian reasoning when we move to **Bayesian Linear Regression**, a linear regression that is used for streaming data.