# Design of Digital Filters

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Design of Non-Recursive Filters using the Window Method

The design of non-recursive filters with a finite impulse response (FIR) is a frequent task in practical applications. The designed filter should approximate a desired frequency response as close as possible. First, the design of causal filters is considered. For many applications the resulting filter should have a linear phase characteristic since this results in a constant (and thus frequency independent) group delay. We therefore specialize the design to causal linear-phase filters.

### Causal Filters

Let's assume that the desired frequency characteristics of the filter are given by the frequency response (i.e. the DTFT spectrum) $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. The corresponding impulse response is computed by its inverse discrete-time Fourier transform (IDTFT)

\begin{equation}
h_\text{d}[k] = \frac{1}{2 \pi} \int\limits_{- \pi}^{\pi} H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \, \mathrm{e}^{\,\mathrm{j}\,\Omega\,k} \; \mathrm{d}\Omega
\end{equation}

In the general case, $h_\text{d}[k]$ will not be a causal FIR. The [Paley-Wiener theorem](https://en.wikipedia.org/wiki/Paley%E2%80%93Wiener_theorem) states, that a causal system $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ can only have zeros at single frequencies. This is not the case for idealized filters, like e.g. the [ideal low-pass filter](https://en.wikipedia.org/wiki/Low-pass_filter#Ideal_and_real_filters). The basic idea of the window method is to truncate the impulse response $h_\text{d}[k]$ in order to derive a causal FIR filter. This can be achieved by applying a window $w[k]$ of finite length $N$ to $h_\text{d}[k]$

\begin{equation}
h[k] = h_\text{d}[k] \cdot w[k]
\end{equation}

where $h[k]$ denotes the impulse response of the designed filter. Its frequency response $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is given by the multiplication theorem of the discrete-time Fourier transform (DTFT)

\begin{equation}
H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{1}{2 \pi} \; H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \circledast W(\mathrm{e}^{\,\mathrm{j}\,\Omega})
\end{equation}

where $W(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ denotes the DTFT of the window function $w[k]$. The frequency response $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of the filter is given as the periodic convolution of the desired frequency response $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and the frequency response of the window function $W(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. The frequency response $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is equal to the desired frequency response $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ only if $W(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = 2 \pi \cdot \delta(\Omega)$. This would require that $w[k] = 1$ for $k = -\infty, \dots, \infty$. Hence for a window $w[k]$ of finite length deviations from the desired frequency response are to be expected.

In order to investigate the effect of truncation on the frequency response $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$, a particular window is considered. A straightforward choice is the rectangular window $w[k] = \text{rect}_N[k]$ of length $N$. Its DTFT is given as

\begin{equation}
W(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \mathrm{e}^{-\mathrm{j} \, \Omega \,\frac{N-1}{2}} \cdot \frac{\sin(\frac{N \,\Omega}{2})}{\sin(\frac{\Omega}{2})}
\end{equation}

The frequency-domain properties of the rectangular window have already been discussed for the [leakage effect](../spectral_analysis_deterministic_signals/leakage_effect.ipynb). The rectangular window features a narrow main lobe at the cost of relative high sidelobe level. The main lobe gets narrower with increasing length $N$. The convolution of the desired frequency response with the frequency response of the window function effectively results in smoothing and ringing. While the main lobe will smooth discontinuities of the desired transfer function, the sidelobes result in undesirable ringing effects. The latter can be alleviated by using other window functions. Note that typical [window functions](../spectral_analysis_deterministic_signals/window_functions.ipynb) decay towards their ends and are symmetric with respect to their center. This may cause problems for desired impulse responses with large values towards their ends.

#### Example: Causal approximation of ideal low-pass

The design of an ideal low-pass filter using the window technique is illustrated in the following example. For $|\Omega| < \pi$ the transfer function of the ideal low-pass is given as

\begin{equation}
H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \begin{cases}
1 & \text{for } |\Omega| \leq \Omega_\text{c} \\
0 & \text{otherwise}
\end{cases}
\end{equation}

where $\Omega_\text{c}$ denotes the corner frequency of the low-pass. An inverse DTFT of the desired transfer function yields

\begin{equation}
h_\text{d}[k] = \frac{\Omega_\text{c}}{\pi} \cdot \text{si}[\Omega_\text{c} \, k]
\end{equation}

The impulse response $h_\text{d}[k]$ is not causal nor FIR. In order to derive a causal and FIR approximation a rectangular window $w[k]$ of length $N$ is applied

\begin{equation}
h[k] = h_\text{d}[k] \cdot \text{rect}_N[k]
\end{equation}

The resulting magnitude and phase response is computed numerically in the following.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

N = 32  # length of filter
Omc = np.pi/2

# compute impulse response
k = np.arange(N)
hd = Omc/np.pi * np.sinc(k*Omc/np.pi)
# windowing
w = np.ones(N)
h = hd * w

# frequency response
Om, H = sig.freqz(h)

# plot impulse response
plt.figure(figsize=(10, 3))
plt.stem(h)
plt.title('Impulse response')
plt.xlabel(r'$k$')
plt.ylabel(r'$h[k]$')
# plot magnitude responses
plt.figure(figsize=(10, 3))
plt.plot([0, Omc, Omc], [0, 0, -100], 'r--')
plt.plot(Om, 20 * np.log10(abs(H)))
plt.title('Magnitude response')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$|H(e^{j \Omega})|$ in dB')
plt.axis([0, np.pi, -20, 3])
plt.grid()
# plot phase responses
plt.figure(figsize=(10, 3))
plt.plot(Om, np.unwrap(np.angle(H)))
plt.title('Phase')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$\varphi (\Omega)$ in rad')
plt.grid()
```

**Exercises**

* Does the resulting filter have the desired phase?
* Increase the length `N` of the filter. What changes?

### Zero-Phase Filters

Above results show that an ideal-low pass cannot be realized very well with the window method. The reason is that an ideal-low pass has zero-phase, as most of the idealized filters.

Lets assume a general zero-phase filter with transfer function $H_d(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = A(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ with amplitude $A(\mathrm{e}^{\,\mathrm{j}\,\Omega})  \in \mathbb{R}$. Its impulse response $h_d[k] = \mathcal{F}_*^{-1} \{ H_d(e^{j \Omega} \}$ is conjugate complex symmetric

\begin{equation}
h_d[k] = h_d^*[-k]
\end{equation}

due to the symmetry relations of the DTFT. Hence, a transfer function with zero-phase cannot be realized by a causal non-recursive filter. This observation motivates to replace the zero-phase by a linear-phase in such situations. This is illustrated in the following.

### Causal Linear-Phase Filters

The design of non-recursive FIR filters with a linear phase is often desired due to their constant group delay. Let's assume a system with generalized linear phase. For $|\Omega| < \pi$ its transfer function is given as

\begin{equation}
H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot \mathrm{e}^{-\mathrm{j} \alpha \Omega + \mathrm{j} \beta}
\end{equation}

where $A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \in \mathbb{R}$ denotes the amplitude of the filter, $\alpha$ its linear phase and $\beta$ a constant phase offset. Such a system can be decomposed into two cascaded systems: a zero-phase system with transfer function $A(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and an all-pass with phase $\varphi(\Omega) = - \alpha \Omega + \beta$. The linear phase term $- \alpha \Omega$ results in the desired constant group delay $t_g(\Omega) = \alpha$.

The impulse response $h[k]$ of a linear-phase system shows a specific symmetry which can be deduced from the symmetry relations of the DTFT for odd/even symmetry of $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ as

\begin{equation}
h[k] = \pm h[N-1-k]
\end{equation}

for $k=0, 1, \dots, N-1$ where $N \in \mathbb{N}$ denotes the length of the (finite) impulse response. The transfer function of a linear phase filter is given by its DTFT

\begin{equation}
H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sum_{k=0}^{N-1} h[k] \, \mathrm{e}^{\,-\mathrm{j}\,\Omega\,k}
\end{equation}

Introducing the symmetry relations of the impulse response $h[k]$ into the DTFT and comparing the result with above definition of a generalized linear phase system reveals four different types of linear-phase systems. These can be discriminated with respect to their phase and magnitude characteristics

| Type | Length $N$ | Impulse Response $h[k]\;$ | Constant Group Delay $\alpha$ in Samples| Constant Phase $\beta$ | Transfer Function $A(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ |
| :---:  | :---: | :---: | :---:| :---: | :---: |
| 1 | odd | $h[k] = h[N-1-k]$ | $\alpha = \frac{N-1}{2} \in \mathbb{N}$ | $\beta = \{0, \pi\}$ | $A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = A(\mathrm{e}^{-\,\mathrm{j}\,\Omega})$, all filter characteristics|
| 2 | even| $h[k] = h[N-1-k]$ | $\alpha = \frac{N-1}{2} \notin \mathbb{N}$ | $\beta = \{0, \pi\}$ | $A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = A(\mathrm{e}^{-\,\mathrm{j}\,\Omega})$, $A(\mathrm{e}^{\,\mathrm{j}\,\pi}) = 0$, only lowpass or bandpass|
| 3 | odd | $h[k] = -h[N-1-k]$ | $\alpha = \frac{N-1}{2} \in \mathbb{N}$ | $\beta = \{ \frac{\pi}{2}, \frac{3 \pi}{2} \} $ | $A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = - A(\mathrm{e}^{-\,\mathrm{j}\,\Omega})$, $A(\mathrm{e}^{\,\mathrm{j}\,0}) = A(\mathrm{e}^{\,\mathrm{j}\,\pi}) =  0$, only bandpass|
| 4 | even | $h[k] = -h[N-1-k]$ | $\alpha = \frac{N-1}{2} \notin \mathbb{N}$ | $\beta = \{ \frac{\pi}{2}, \frac{3 \pi}{2} \} $ | $A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = - A(\mathrm{e}^{-\,\mathrm{j}\,\Omega})$, $A(\mathrm{e}^{\,\mathrm{j}\,0}) = 0$, only highpass or bandpass|

These relations have to be considered in the design of a causal linear phase filter. Depending on the desired frequency characteristics $A(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ the suitable type is chosen. The odd/even length $N$ of the filter and the phase (or group delay) is chosen accordingly for the design of the filter. It is also abovious that a filter with zero-phase $\alpha = 0$, e.g. an ideal low-pass, would result in $N=1$.

#### Example: Causal linear-phase approximation of ideal low-pass

We aim at the design of a causal linear-phase low-pass using the window technique. According to the previous example, the desired frequency response has an even symmetry $A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = A(\mathrm{e}^{\,-\mathrm{j}\,\Omega})$ with $A(\mathrm{e}^{\mathrm{j}\,0}) = 1$. This could be realized by a filter of type 1 or 2. We choose type 1, since the resulting filter then exhibits an integer group delay of $t_g(\Omega) = \frac{N-1}{2}$ samples. Consequently the length of the filter $N$ has to be odd.

The impulse response $h_\text{d}[k]$ is given by the inverse DTFT of $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ as

\begin{equation}
h_\text{d}[k] = \frac{\Omega_\text{c}}{\pi} \cdot \text{si} \left[ \Omega_\text{c} \left(k-\frac{N-1}{2} \right) \right]
\end{equation}

The impulse response fulfills the desired symmetry for $k=0,1, \dots, N-1$. A causal and FIR approximation is obtained by applying a window function of length $N$ to the impulse response $h_\text{d}[k]$

\begin{equation}
h[k] = h_\text{d}[k] \cdot w[k]
\end{equation}

Note that the window function $w[k]$ also has to fulfill the desired symmetries.

As already outlined, the chosen window determines the properties of the transfer function $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. The [spectral properties of commonly applied windows](../spectral_analysis_deterministic_signals/window_functions.ipynb) have been discussed previously. The width of the main lobe will generally influence the  smoothing of the desired transfer function $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$, while the sidelobes influence the typical ringing artifacts. This is illustrated in the following.

```python
N = 33  # length of filter
Omc = np.pi/2

# compute impulse response
k = np.arange(N)
hd = Omc/np.pi * np.sinc((k-(N-1)/2)*Omc/np.pi)
# windowing
w1 = np.ones(N)
w2 = np.blackman(N)
h1 = hd * w1
h2 = hd * w2

# frequency responses
Om, H1 = sig.freqz(h1)
Om, H2 = sig.freqz(h2)

# plot impulse response
plt.figure(figsize=(10, 3))
plt.stem(h1)
plt.title('Impulse response (rectangular window)')
plt.xlabel(r'$k$')
plt.ylabel(r'$h[k]$')
# plot magnitude responses
plt.figure(figsize=(10, 3))
plt.plot([0, Omc, Omc], [0, 0, -300], 'r--')
plt.plot(Om, 20 * np.log10(abs(H1)), label='rectangular window')
plt.plot(Om, 20 * np.log10(abs(H2)), label='Blackmann window')
plt.title('Magnitude response')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$|H(e^{j \Omega})|$ in dB')
plt.axis([0, np.pi, -120, 3])
plt.legend(loc=3)
plt.grid()
# plot phase responses
plt.figure(figsize=(10, 3))
plt.plot(Om, np.unwrap(np.angle(H1)), label='rectangular window')
plt.plot(Om, np.unwrap(np.angle(H2)), label='Blackmann window')
plt.title('Phase')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$\varphi (\Omega)$ in rad')
plt.legend(loc=3)
plt.grid()
```

**Exercises**

* Does the impulse response fulfill the required symmetries for a type 1 filter?
* Can you explain the differences between the magnitude responses $|H(\mathrm{e}^{\,\mathrm{j}\,\Omega})|$ for the different window functions?
* What happens if you increase the length `N` of the filter?

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2017*.


# Real Business Cycle (RBC) model
This introductory notebook demonstrates the basics of the Real Business Cycle (RBC) model used in economics

The materials and equations are from Chapter 2 of the following book:
- Costa, C. (2018). Understanding DSGE models: Theory and Applications. Vernon Press.

This book is brilliant and I highly recommend to anyone interested in studying the DSGE modeling regime used in macroeconomics.

# Table of contents

- [1. Household problem](#1.-Household-problem)
    - 1.1 Model with two goods
        - 1.1.1 Python example
        - 1.1.2 Marginal rate of substitution (MRS)
    - 1.2 Budget constraint
    - 1.3 Optimal choice
        - 1.3.1 Maximization & constraint
        - 1.3.2 Optimization: Lagrangian
        - 1.3.3 Substitution and income effects
- [2. Dynamic structure of consumption savings](#2.-Dynamic-structure-of-consumption-savings)
    - 2.1 Two-period example
    - 2.2 Optimal intertemporal choice
        - 2.2.1 Maximization & constraint
        - 2.2.2 Optimization: Lagrangian
    - 2.3 Input markets
        - 2.3.1 Maximization & constraint
        - 2.3.2 Optimization: Lagrangian
- [3. The RBC model](#3.-The-RBC-model)
    - 3.1 Model assumptions
    - 3.2 Households
        - 3.2.1 Household assumptions
        - 3.2.2 Maximization
        - 3.2.3 Optimization: Lagrangian
   - 3.3 Firms
       - 3.3.1 Firms assumptions
       - 3.3.2 Cobb-Douglas
       - 3.3.3 Productivity shock
       - 3.3.4 Marginal cost calculation
  - 3.4 Equilibrium conditions
      - 3.4.1 Structure of the model
      - 3.4.2 Steady state
- [4. Solve steady state](#4.-Solve-steady-state)
    - 4.1 Set parameters names and values
    - 4.2 Solve model
- [5. Log-linearization](#5.-Log-linearization)

```python
#Import libraries
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
plt.style.use('seaborn-darkgrid')
```

# 1.  Household problem

## 1.1 Model with two goods

Two factors that provide utility are consumption (C) and leaisure

$$U(C, \ \ \text{leisure})$$

Both treated as goods

Utility function has following properties:
1. Strictly increasing, $\frac{\partial u}{\partial C}>0$ and $\frac{\partial u}{\partial \text{leisure}}>0$
2. Diminishing marginal returns, $\frac{\partial^2 u}{\partial C^2}<0$ and $\frac{\partial u^2}{\partial \text{leisure}^2}<0$

### 1.1.1 Python example: Model with two goods
Code reference: https://datascience.quantecon.org/scientific/optimization.html

```python
def U(A, B, alpha=1/3):
    return B**alpha * A**(1-alpha)
def U_b(A,B, alpha=1/3):
    return alpha*A**(alpha-1)*B**(1-alpha)
def U_a(A,B, alpha=1/3):
    return (1-alpha)*A**(alpha)*B**(-alpha)

def U_aa(A,B, alpha=1/3):
    return A**alpha*(alpha-1)*alpha*B**(-alpha-1)

fig, ax = plt.subplots()
B = 1.5
A = np.linspace(1, 10, 100)
ax.plot(A, U(A, B), label = "U(B=1.5, A)")
ax.plot(A, U_a(A,B), label=r'$\partial u/ \partial A$')
ax.plot(A, U_aa(A,B), label=r'$\partial^2 u/ \partial A^2$')
ax.set_xlabel("A")
ax.set_ylabel("U(B=1.5, A)")
plt.legend();
```

### 1.1.2 Marginal rate of substitution (MRS)

- Measures how many units of a good one will give up in exchange for another
    - The slope on the graph is the MRS

$$MRS_{X,Y}= -\frac{\partial Y}{\partial X}\Bigg|_{u=U_1}=-\frac{MU_X}{MU_Y}\Bigg|_{u=U_i}$$

#### 1.1.2.1 Python example: Plotting indifference curves
- Code reference: https://datascience.quantecon.org/scientific/optimization.html

```python
A = np.linspace(1, 10, 100)
B = np.linspace(1, 20, 100).reshape((100, 1))

def U(A, B, alpha=1/3):
    return B**alpha * A**(1-alpha)

def A_indifference(B, ubar, alpha=1/3):
    return ubar**(1/(1-alpha)) * B**(-alpha/(1-alpha))

def plot_indifference_curves(ax, alpha=1/3):
    ubar = np.arange(1, 11, 2)
    ax.plot(B, A_indifference(B, ubar, alpha))
    ax.legend([r"$\bar{U}$" + " = {}".format(i) for i in ubar])
    ax.set_xlabel("B")
    ax.set_ylabel(r"$A(B, \bar{U}$)")

fig, ax = plt.subplots()
plot_indifference_curves(ax)
```

## 1.2 Budget constraint

### 1.2.1 Python example: Budget constraint

```python
def consum(max_work, wage, price, leisure):
    c = (max_work*wage)/price-(wage/price)*leisure
    return c
```

```python
max_work, wage, price, leisure = [70, 30, 10,30]
consum = consum(max_work,wage,price,leisure)
print('Total consumption: {}'.format(consum))
```

    Total consumption: 120.0

### 1.2.2 Graph: Budget constraint

```python
def budget_constraint_graph(wage, price, hours_worked):
    vert_intercept = (hours_worked*wage)/price
    slope = -(wage/price)
    line = []
    for hours in range(hours_worked):
        line.append(vert_intercept+slope*hours)
    plt.plot(line)

for wage in [10,20]:
    budget_constraint_graph(wage, 5, 40)

plt.axhline(0, color='black')
plt.axvline(0, color='black')
plt.xlabel('Leisure')
plt.ylabel(r'Consumption: $\frac{W}{P}$')
plt.text(20, 90, r"Slope=$-\frac{W}{P}$", size=15);
```

## 1.3 Optimal choice

### 1.3.1 Maxzimiation & constraint
$$
\max_{c, L}  u(c, L)\\
\text{s.t.} \ Pc=WL \\
L = u(c, \ L)-\lambda(Pc-WL)
$$

### 1.3.2 Optimization: Lagrangian

### Supply of labor
$$\frac{\partial u/\partial L}{\partial u /\partial c}= - \frac{W}{P}\tag{2.1.1}$$

`Household problem:`

To maximize utility an individual purchase gods that deplates their total income equating to
- The physical rate of tradeoff between any two goods (MRS) and the rate at which a good can be exchanged for another on the market.

`Optimal result of the problem of the household:`

The optimal consumption is the point which is on the highest indifference curve AND is within the individual’s budget constraint.

### 1.3.3 Substitution and income effects

The effects of changes in real wages on optimal leisure choice has two components:
1. a substitution effect
2. a income effect.

- In the consumption-leisure model:
    - The substitution effect of higher real wages leads an indvidual to work more
        - Because the opportunity cost of leisure has risen.
    - On the other hand, the income effect of higher real wages leads an individual to choose more leisure

# 2. Dynamic structure of consumption savings

$$u(c_1, c_2, c_3, \cdots) = u(c_1)+\beta u(c_2)+\beta^2 u(c_3)$$

`Intertemporal discount factor:` $\beta<1$

## 2.1 Two-period example
$$P_1 c_1 + A_1 = R A_0+Y_1 \\
P_2 c_2 + A_2 = R A_1+Y_2 \\
R= 1+r$$

Savings:
$$A_1-A_0=S_1 = (R-1)A_0+Y_1-P_1c_1$$

Solving budget constraint 1 and substitute:
$$P_2c_2 = R[RA_0+Y_1-P_1c_1]+Y_2$$

Divide both sides by R:
$$P_1c_1+\frac{P_2c_2}{R} = Y_1+\frac{Y_2}{R}+RA_0$$

- RHS: Discounted intertemporal resource
- LHS: Discounted intertemporal consumption

Solve for $c_2$:
$$c_2 = \Big[\Big(\frac{R}{P_2}\Big)Y_1+\frac{Y_2}{P_2}\Big]-\Big[\frac{P_1R}{P_2}\Big]c_1$$

### 2.1.1. Graph: Intertemporal budget constraint

```python
def intertemporal_bc(R=1.01, p2=10, p1=5, Y1=30, Y2=40, c1=5):
    vert_int = (R/p2)*Y1+(Y2/p2)
    slope = -((p1*R)/p2)*c1
    eq = vert_int+slope
    return eq
```

```python
line1 = [ ]
line2 = []
for consum_period_1 in range(10):
    line1.append(intertemporal_bc(c1=consum_period_1))
    line2.append(intertemporal_bc(R=1.2,c1=consum_period_1))

plt.plot(line1, label="R=1.01")
plt.plot(line2, label = "R=1.2")

plt.xlabel("Consumption: period 1")
plt.ylabel("Consumption: period 2")
plt.legend();
```

## 2.2 Optimal intertemporal choice

### 2.2.1 Maximization & constraint
$$u(c_1, c_2) = u(c_1)+\beta u(c_2)$$

$$
\max_{c1,c2, A_1}  u(c1)+\beta u(c_2)\\
\text{s.t.}\\
\ P_1c_1+A_1= RA_0+Y_1 \\
P_2c_2= RA_1+Y_2
$$

### 2.2.2 Optimization: Lagrangian
$$
L = u(c_1)+\beta u(c_2)-\lambda_1[P_1c_1+A_1-RA_0-Y_1]-\lambda_2[P_2c_2-RA_1-Y_2]\\
\frac{L}{c_1} = \frac{\partial u}{\partial c_1}-\lambda_1P_1=0\\
\frac{L}{c_2} = \beta\frac{\partial u}{\partial c_2}-\lambda_2P_2=0\\
\frac{L}{A_1} = -\lambda_1+\lambda_2R=0\\
\lambda_1 = \frac{\partial u/\partial c_1}{P_1}
, \ \ \lambda_2 = \beta\frac{\partial u/\partial c_2}{P_2}\\
\text{Define} \ \  \pi_2= \frac{P_2}{P_1} $$

#### 2.2.2.1 Euler equation:
$$-\frac{\partial u/\partial c_1}{\beta\partial u/\partial c_2}=-\frac{R}{\pi_2}$$

- The Euler equation relates to consumption for both periods (MRS c1-c2) with the relative price of intertemporal consumption
- On the other hand, the budget constraint’s slope determines the premium, R, for saving more.
    - High values for β lead to indifference curves having low slopes.

- `Intertemporal substitution effect:` change in relative price which motivates households to increase future consumption in relation to the present

## 2.3 Input markets
- Firms represent the agents that acquire inputs, while households are those that supply them

### 2.3.1 Maximization & constraint
Demand inputs:
$$Y = f(K^d, L^d) \\
\max_{K^d,L^d} = PY-WL^d-RK^d$$

### 2.3.2 Optimization: Lagrangian
$$
P\frac{\partial Y}{\partial L^d}-W=0\\
P\frac{\partial Y}{\partial K^d}-R=0
$$

#### Demand for inputs:

Marginal productivity of Labor = Real marginal cost of labor
$$
\frac{\partial Y}{\partial L^d}=\frac{W}{P}\\
$$

Marginal productivity of capital = Real marginal cost of capital
$$
\frac{\partial Y}{\partial K^d}=\frac{R}{P}\\
$$

#### Relative demand for inputs:

Marginal rate of technical substitution (MRTS)= Economic rate of substitution
$$
\frac{\frac{\partial Y}{\partial L^d}}{\frac{\partial Y}{\partial K^d}}=-\frac{W}{R}
$$

`MRTS:` Indicates the additional units of capital that can offset one less unit of labor

Agents make decisions using the marginal rate of substitution between goods and their relative prices by:
1. Households consumption-leisure tradeoff analyzing the relative price between these goods (real wages).
      - Intertemporal choice is the tradeoff is between consumption today and in the future
          - The relative price is the nominal interest rate.
2. Firms make a similar decision when determining the combination of labor and capital by analyzing the relative prices of their inputs.

# 3. The RBC model

Structural model of the economy is presented and solved, step by step.

1. Present agents (households and firms)
2. Equilibrium conditions are shown.
3. The state is found and the equations that make up the model’s equilibrium are log-linearized.

## 3.1 Model assumptions:
- Assumption 2.2.1. The economy is closed, with no government or financial sector.
- Assumption 2.2.2. This economy does not have a currency (Barter economy).
- Assumption 2.2.3. Adjustment costs are not present.

## 3.2 Households

### 3.2.1 Household assumptions
- Assumption 2.2.4. The model is made up of a unitary set of households indexed by j ∈ [0, 1] whose goal is to maximize intertemporal welfare function.
    - A utility function is used, additively separable into consumption (C) and labor (L).
- Assumption 2.2.5. Consumption is intertemporally additively separable (no habit formation).
- Assumption 2.2.6. Population growth does not exist.
- Assumption 2.2.7. The labor market is of perfect competition (no wage rigidity).

### 3.2.2 Maximization & Constraint
#### Representative household optimal welfare function
$$\max_{C_{j,t},L_{j,t},K_{j,t+1}} E_t \sum^{\infty}_{t=0} \beta^t\bigg(\frac{C_{j,t}^{1-\sigma}}{1-\sigma}- \frac{L_{j,t}^{1+\varphi}}{1+\varphi}\bigg)\tag{2.1}\\
$$
Where:
- $E_t=\text{expectations operator}$
- $C=\text{Consumption}$
- $L=\text{Hours worked}$
- $\sigma= \text{relative risk aversion}$
- $\varphi= \text{marginal disutility of labor supply}$

#### Intermporal budget constraint:
$$P_t(C_{j,t}+I_{j,t})= W_tL_{j,t}+R_tK_{j,t}+\Pi_t\tag{2.2}$$

Where:
- P=price
- I=investment
- W= wages
- K= capital stock
- R= return on capital
- $\Pi$= frims profit (dividends)

#### Capital accumulation over time:
$$K_{j,t+1} = (1-\delta)K_{j,t}+I_{j,t}\tag{2.3}$$

### 3.2.3 Optimization: Lagrangian

#### Solve problem of the household using 2.1,2.2, 2.3
$$
L = E_t \sum^{\infty}_{t=0} \beta^t\bigg(\frac{C_{j,t}^{1-\sigma}}{1-\sigma}- \frac{L_{j,t}^{1+\varphi}}{1+\varphi}\bigg)\\-\lambda_{j,t}[P_tC_{j,t}+P_tK_{j,t+1}-P_t(1-\delta)K_{j,t}-W_tL_{j,t}-R_tK_{j,t}-\Pi_t]\tag{2.4}
$$

Solve F.O.C:
$$
\frac{\partial L}{\partial C_{j,t}}=C_{j,t}^{-\sigma}- \lambda_{j,t}P_t=0\tag{2.5}$$
$$\frac{\partial L}{\partial L_{j,t}}= -L^{\varphi}_{j,t}+\lambda_{j,t} W_t=0\tag{2.6}$$
$$
\frac{\partial L}{\partial K_{j,t+1}}= -\lambda_{j,t} P_t +\beta E_t\lambda_{j,t+1}[(1-\delta) E_tP_{t+1}E_tR_{t+1}]=0 \tag{2.7}
$$

#### Household labor supply equation
Consumption-leisure MRS = Consumption-leisure relative price

$$
C_{j,t}^\sigma L^\varphi_{j,t} = \frac{W_t}{P_t}\tag{2.8}
$$

#### Euler equation Plug 2.5 into 2.7
$$-C_{j,t}^{-\sigma} + E_t \Big\{\frac{C_{j,t+1}^{-\sigma}}{P_{t+1}}[(1-\delta)P_{t+1}+R_{t+1}]
\Big\}=0\\
\Big(\frac{E_t C_{j,t+1}}{C_{j,t}}\Big)^\sigma = \beta\Big[(1-\delta)+ E_t\frac{R_{t+1}}{P_{t+1}}\Big]
\tag{2.9}$$

- The previous equation determines the household’s savings decision
    -  They compare the utility of consuming an additional amount today compared to in the future

Assume that $\beta=1, \ \delta=1$
$$-E_t\Big(\frac{1}{\pi_{t+1}}\frac{ C_{j,t+1}}{C_{j,t}}\Big)^\sigma = -E_t\frac{r_{t+1}}{\pi_{t+1}}$$

Where:
- Real rate of return on capital: $E_tr_{t+1}=E_t\frac{R_{t+1}}{P_{t+1}}$

The household problem has two choices:
1. Intratemporal choice: choose between consumption and leisure
2. Intertemporal choice: choose between present and future consumption

## 3.3 Firms

### 3.3.1 Firms assumptions
- Assumption 2.2.8. Firms indexed by j that maximize profit observing a structure of perfect competition (Zero profits)
- Assumption 2.2.9. Productivity growth is ignored.

### 3.3.2 Cobb-Douglas

$Y_{j,t}= A_tK^\alpha_{j,t}L^{1-\alpha}_{j,t}\tag{2.10}$

Where:
- $A_t$= productivity
- $Y_t$= product
- $\sigma$= elasticity of production w.r.t. capital

- Production function must have following properties (similar to household utility function):
- Strictly increasing (FK > 0 and FL > 0)
- Strictly concave (FK K < 0 e FLL < 0)
- Constant returns to scale
- Satisfy Inada conditions

#### 3.3.2.1 Max profit function

$$\max_{L_{j,t},K_{j,t}}\Pi_{j,t} =  A_t K_{j,t}^\sigma L_{j,t}^{1-\sigma}P_{j,t}-W_tL_{j,t}-R_tK_{j,t}\tag{2.11}$$

#### 3.3.2.2 Optimization lagrangian:
Demand for capital: MCK=MPK
$$\frac{R_t}{P_{j,t}}=\alpha \frac{Y_{j,t}}{K_{j,t}}\tag{2.14}$$
Demand for labor: MCL=MPL
$$\frac{W_t}{P_{j,t}}=(1-\alpha)\frac{Y_{j,t}}{L_{j,t}}\tag{2.15}
$$

### 3.3.3 Productivity shock

$$log A_t=(1-\rho_a)logA_{ss}+\rho_A logA_{t-1}+\varepsilon_t\tag{2.16}$$

Where:
- $A_{ss}$= value of productivity at steady state
- $\rho_{A}$= autoregressive paramater of productiity

### 3.3.4 Marginal cost calculation
- RBC approach: Price level= Marginal cost
- Combine 2.14 & 2.15

MRTS=ERS

$$-\frac{W_t}{R_t} = - \frac{(1-\alpha)K_{j,t}}{\alpha L_{j,t}}$$

-  RHS: marginal rate of technical substitution
    - The rate at which labor can be replaced by capital while maintaining a constant level of production.
- LHS: economic rate of substitution,
    - The rate at which labor can be replaced by capital while maintaining the same cost.

Rearranging:
$$ L_{j,t}= \frac{1-\alpha}{\alpha}-\frac{R_t}{W_t}K_{j,t}\tag{2.17}$$

Substitue 2.17 into production function 2.10
$$Y_{j,t}= A_t K^\alpha_{j,t}\Big[\frac{1-\alpha}{\alpha}-\frac{R_t}{W_t}K_{j,t}\Big]^{1-\alpha}\\
K_{j,t}= \frac{Y_{j,t}}{A_t} \Big[\frac{\alpha}{1-\alpha}\frac{W_t}{R_t}\Big]^{1-\alpha}\tag{2.18}$$

Substitute 2.18 in 2.17:

$$
L_{j,t} = \frac{A_t}{Y_{j,t}} \Big[\frac{\alpha}{1-\alpha}\frac{W_t}{R_t}\Big]^{-\alpha}
$$

#### Total cost function:
$$TC_{j,t} = W_tL_{j,t}+R_tK_{j,t}$$

Substitute 2.18 and 2.19 into total cost function

$$
TC_{j,t} = \frac{Y_{j,t}}{A_t}\Big(\frac{W_t}{1-\alpha}\Big)^{1-\alpha}\Big(\frac{R_t}{\alpha}\Big)^\alpha
$$

#### Marginal cost:

$$
MC_{j,t} = \frac{1}{A_t}\Big(\frac{W_t}{1-\alpha}\Big)^{1-\alpha}\Big(\frac{R_t}{\alpha}\Big)^\alpha\tag{2.20}
$$

General price level: marginal cost depends solely on productivity and the prices of the factors of production

$$
P_{t} = \frac{1}{A_t}\Big(\frac{W_t}{1-\alpha}\Big)^{1-\alpha}\Big(\frac{R_t}{\alpha}\Big)^\alpha \tag{2.21}
$$

## 3.4 Equilibrium conditions

We have now described the behavior of firms and households, now we examine the interaction between them in order to determine macroeconomic equilibrium.

Aiming to max utility:
1. Households decide how much to:

- consume (C)
- invest (I)
- labor supply (L)

2. Firms decide how much to (Y) produce:

- Using available technology
- Choosing the factors of production (capital and labor)

Both households and firms take prices as given

The model’s equilibrium consists of the following blocks:

1. A price system: W , R and P
2. An endowment of values for goods and inputs Y , C, I , L and K
3. A production-possibility frontier where aggregate supply= aggregate demand

### 3.4.1 Structure of the model

$$
\begin{align}
C_t^\sigma L_t^\varphi & = \frac{W_t}{P_t} & \text{Labor supply, 2.8} \\
\Big(\frac{E_t C_{j,t+1}}{C_{j,t}}\Big)^\sigma &= \beta\Big[(1-\delta)+ E_t\frac{R_{t+1}}{P_{t+1}}\Big]& \text{Euler, 2.9} \\
K_{t+1} & = (1-\delta)K_t + I_t& \text{Law of motion of capital, 2.3}\\
y_t & = A_t K_t^\alpha L_t^{1 - \alpha} & \text{Production function, 2.10} \\
K_t & =\alpha\frac{Y_t}{\frac{R_t}{P_t}} & \text{Demand for capital, 2.14} \\
L_t & =(1-\alpha)\frac{Y_t}{\frac{W_t}{P_t}} & \text{Demand for labor, 2.15} \\
P_t & =\frac{1}{A_t}\Big(\frac{W_t}{1-\alpha}\Big)^{1-\alpha}\Big(\frac{R_t}{\alpha}\Big)^\alpha & \text{Price level=MC, 2.21}\\
Y_t & =C_t+I_t & \text{Equilibrium condition}\\
logA_t & =(1-\rho_a)logA_{ss}+\rho_A logA_{t-1}+\varepsilon_t& \text{Productivity shock, 2.16}
\end{align}
$$

#### 3.4.1.1 Households
$$C_{ss}^\sigma L_{ss}^\varphi = \frac{W_{ss}}{P_{ss}}  \tag{Labor supply, 2.23}$$
$$1 = \beta\Big[(1-\delta)+ E_t\frac{R_{ss}}{P_{ss}}\Big] \tag{Euler, 2.24}$$
$$I_{ss}=\delta K_{ss}\tag{2.25}$$

#### 3.4.1.2 Firms
$$K_{ss}=\alpha \frac{Y_{ss}}{R_{ss}/P_{ss}}\tag{2.26}$$
$$L_{ss}=(1-\alpha)\frac{Y_{ss}}{W_{ss}/P_{ss}}\tag{2.27}$$
$$Y_{ss}= K_{ss}^\alpha L_{ss}^{1-\alpha}\tag{2.28}$$
$$P_{ss}=\Big(\frac{W_{ss}}{1-\alpha}\Big)^{1-\alpha}\Big(\frac{R_{ss}}{\alpha}\Big)^\alpha\tag{2.29}$$
#### 3.4.1.3 Equilibrium condition
$$Y_{ss} = C_{ss}+I_{ss}\tag{2.30}$$

Variables from 3.4.1.1, 3.4.1.2 and 3.4.1.3 are used to determine eight endogenous variables of the steady state

### 3.4.2 Steady state

#### Steady state
- Some endogenous variables have their steady state values previously determined (exogenously).
    - For example productivity: the source of RBC models’ shocks at the steady state
    - The next step is to remove the variables’ time indicators which yields the structure of the model

$$
A_{ss} = 1 \\
P_{ss} = 1 \\
$$

$$R_{ss} = P_{ss}\Big[\frac{1}{\beta}-(1-\delta)\Big] \tag{2.31 from: 2.24, Euler}$$

$$W_{ss} = (1-\delta)P_{ss}^{\frac{1}{1-\alpha}}\Big(\frac{\alpha}{R_{ss}}\Big)^{\frac{\alpha}{1-\alpha}} \tag{2.32 from: 2.21, Price level}$$

$$Y_{ss} = \Big(\frac{R_{ss}}{R_{ss}-\delta\alpha}\Big)^{\frac{\sigma}{\sigma+\varphi}}+\Bigg[\frac{W_{ss}}{P_{ss}}\Big(\frac{\frac{W_{ss}}{P_{ss}}}{1-\alpha}\Big)^{\varphi}\Bigg]^{\frac{1}{\sigma+\varphi}}\tag{2.35 from: 2.30 with 2.33 and 2.34 }$$

$$I_{ss}=\Big(\frac{\delta\alpha}{R_{ss}}\Big)Y_{ss} \tag{2.34 from: 2.26 with 2.25}$$

$$C_{ss}= \frac{1}{Y_{ss}^{\frac{\varphi}{\sigma}}}\Big[(1-\alpha)^{-\varphi}\Big(\frac{W_{ss}}{P_{ss}}\Big)^{1+\varphi}\Big]^{1/\sigma}\tag{2.33 from: 2.23, Labor supply with 2.27}$$

$$
K_{ss}=\alpha\frac{Y_{ss}}{R_{ss}/P_{ss}}\tag{from 2.26}
$$
$$
L_{ss}=(1-\alpha)\frac{Y_{ss}}{W_{ss}/P_{ss}}\tag{from 2.27}
$$

### 3.4.3 Steady state chart

# 4. Solve steady state

## 4.1 Set parameters names and values

```python
parameter_names = [
    'relative risk',
    'marginal disutility of labor',
    'elasticity of production w.r.t K',
    'discount factor',
    'depreciation rate',
    'price'
]
# Setup fixed parameters
parameters = pd.DataFrame({
    'name': parameter_names,
    'value': [2, 1.5, 0.35, 0.985, 0.025, 1]
})
parameters.T
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>0</th>
      <th>1</th>
      <th>2</th>
      <th>3</th>
      <th>4</th>
      <th>5</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>name</th>
      <td>relative risk</td>
      <td>marginal disutility of labor</td>
      <td>elasticity of production w.r.t K</td>
      <td>discount factor</td>
      <td>depreciation rate</td>
      <td>price</td>
    </tr>
    <tr>
      <th>value</th>
      <td>2</td>
      <td>1.5</td>
      <td>0.35</td>
      <td>0.985</td>
      <td>0.025</td>
      <td>1</td>
    </tr>
  </tbody>
</table>
</div>

## 4.2 Solve model

### 4.2.1 Create model function

```python
class steady(object):
    def __init__(self, params=None):
        # Model dimensions
        self.k_params = 6

        # Initialize parameters
        if params is not None:
            self.update(params)

    def update(self, params):
        self.risk = params[0]
        self.disutility_labor = params[1]
        self.elasticity_capital = params[2]
        self.discount_factor = params[3]
        self.depreciation = params[4]
        self.price = params[5]

    def steady_state(self):
        rates = self.price*((1/self.discount_factor)-(1-self.depreciation))

        wages = (1-self.elasticity_capital)*self.price**(1/(1-self.elasticity_capital))*(self.elasticity_capital/rates)**(self.elasticity_capital/(1-self.elasticity_capital))

        output_1 = ((rates)/(rates-self.depreciation*self.elasticity_capital))**((self.risk)/(self.risk+self.disutility_labor))
        output_2 = ((wages/self.price)*((wages/self.price)/(1-self.elasticity_capital))**(self.disutility_labor))**((1/(self.risk+self.disutility_labor)))
        output = output_1*output_2

        investment = (self.depreciation*self.elasticity_capital/rates)*output

        cons_1 = (1/(output**(self.disutility_labor/self.risk)))
        cons_2 = (((1-self.elasticity_capital)**(-self.disutility_labor))*(wages/self.price)**(1+self.disutility_labor))**(1/self.risk)
        consumption = cons_1*cons_2

        capital = self.elasticity_capital*((output)/(rates/self.price))

        labor = (1-self.elasticity_capital)*((output)/(wages/self.price))

        return (rates, wages, output, investment, consumption, capital, labor)
```

### 4.2.2 Steady state solution

```python
RBC_solution = steady(parameters['value'])
Steady_state_solution = RBC_solution.steady_state()
```

### 4.2.3 Print results
- Solution matches table 2.3 Costa pg. 47

```python
variable_names = [
    'Interest rates', 'Wages', 'Output',
    'Investment', 'Consumption', 'Capital', 'Labor'
]
steady_state = pd.DataFrame({
    'Solution': Steady_state_solution,
}, index=variable_names)

round(steady_state.T,3)
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Interest rates</th>
      <th>Wages</th>
      <th>Output</th>
      <th>Investment</th>
      <th>Consumption</th>
      <th>Capital</th>
      <th>Labor</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>Solution</th>
      <td>0.04</td>
      <td>2.084</td>
      <td>2.338</td>
      <td>0.508</td>
      <td>1.829</td>
      <td>20.338</td>
      <td>0.729</td>
    </tr>
  </tbody>
</table>
</div>

# 5. Log-linearization

- When the model is very simple, it is possible to find an approxima- tion of the policy function by recursively solving the value function. On the other hand, linear models are often easier to solve. The prob- lem is converting a non-linear model to a sufficiently adequate lin- ear approximation such that its solution helps in the understand- ing of the underlying non-linear system’s behavior.

### Uhlig's rule

$$\tilde{X}_t= logX-logX_{ss}\tag{2.36}\\
e^{\tilde X_t+a \tilde{Y}_t} = 1+\tilde{X_t}+a \tilde{Y_t}$$

## 5.1 Log-linearize steady state equations
### 5.1.1 Labor supply

$$C_{t}^\sigma L_{t}^\varphi = \frac{W_{t}}{P_{t}}$$

Replace $X_t= X_{ss}e^{\tilde{X_t}}$

$$C_{ss}^\sigma L_{ss}^\varphi e^{\sigma \tilde{C_t}+\varphi\tilde{L_t}} = \frac{W_{ss}}{P_{ss}} e^{\tilde{w_t}-\tilde{P_t}} $$

Use Uhlig's rule 2.36

$$C_{ss}^\sigma L_{ss}^\varphi (1+\sigma \tilde{C_t}+\varphi\tilde{L_t}) = \frac{W_{ss}}{P_{ss}} (1+\tilde{w_t}-\tilde{P_t})$$

Given steady state of $C_{ss}^\sigma L_{ss}^\varphi =\frac{W_{ss}}{P_{ss}}$

$$\sigma \tilde{C_t}+\varphi\tilde{L_t} = \tilde{W_t}-\tilde{P_t}\tag{2.39}$$

### 5.1.2 Euler equation for consumption

$$\frac{\sigma}{\beta}(E_t \tilde{C_{t+1}}-\tilde{C_t}) = \frac{R_{ss}}{P_{ss}} E_t(\tilde{R_{t+1}}-\tilde{P_{t+1}})\tag{2.40}$$

### 5.1.3 Return on capital

$$\tilde{R_t}-\tilde{P_t}= \tilde{Y_t}-\tilde{K_t}\tag{2.41}$$

### 5.1.4 Wage levels

$$\tilde{W_t}-\tilde{P_t}= \tilde{Y_t}-\tilde{L_t}\tag{2.42}$$

### 5.1.5 Production function
$$\tilde{Y_t}= \tilde{A_t}+\alpha\tilde{K_t}+(1-\alpha)\tilde{L_t}\tag{2.43}$$

### 5.1.6 Law of motion of capital

$$\tilde{K_{t+1}}=(1-\delta)\tilde{K_t}+\delta\tilde{I_t}\tag{2.44}$$

### 5.1.7 Equilibrium condition

$$Y_{tt}\tilde{Y_{t}}=C_{ss}\tilde{C_t}+I_{ss}\tilde{I_t}\tag{2.45}$$

### 5.1.8 Technological shock

$$Y_{tt}\tilde{Y=A_{t}}= \rho_A \tilde{A_{t-1}}+\varepsilon_t\tag{2.46}$$

## 5.2 Structure of log-linear model

$$
\begin{align}
C_t^\sigma L_t^\varphi & = \frac{W_t}{P_t} & \Rightarrow \ \ &  \sigma\tilde{C_t}+\varphi\tilde{L_t} = \tilde{W_t}-\tilde{P_t}  & \ \  \text{Labor supply} \\
\Big(\frac{E_t C_{j,t+1}}{C_{j,t}}\Big)^\sigma &= \beta\Big[(1-\delta)+ E_t\frac{R_{t+1}}{P_{t+1}}\Big]&\Rightarrow & \frac{\sigma}{\beta}(E_t \tilde{C_{t+1}}-\tilde{C_t}) = \frac{R_{ss}}{P_{ss}} E_t(\tilde{R_{t+1}}-\tilde{P_{t+1}}) & \text{Euler}\\
K_{t+1} & = (1-\delta)K_t + I_t& \Rightarrow \ \ & \tilde{R_t}-\tilde{P_t}= \tilde{Y_t}-\tilde{K_t}& \text{Law of motion of capital}\\
y_t & = A_t K_t^\alpha L_t^{1 - \alpha} & \Rightarrow \ \ & \tilde{Y_t}= \tilde{A_t}+\alpha\tilde{K_t}+(1-\alpha)\tilde{L_t}& \text{Production function} \\
K_t & =\alpha\frac{Y_t}{\frac{R_t}{P_t}} & \Rightarrow \ \ & \tilde{K_{t+1}}=(1-\delta)\tilde{K_t}+\delta\tilde{I_t}& \text{Demand for capital} \\
Y_t & =C_t+I_t  & \Rightarrow \ \ &  Y_{tt}\tilde{Y_{t}}=C_{ss}\tilde{C_t}+I_{ss}\tilde{I_t}  & \text{Equilibrium condition}\\
logA_t & =(1-\rho_a)logA_{ss}+\rho_A logA_{t-1}+\varepsilon_t& \Rightarrow \ \ & Y_{tt}\tilde{Y=A_{t}}= \rho_A \tilde{A_{t-1}}+\varepsilon_t& \text{Productivity shock}
\end{align}
$$

## 5.3 Productivity shock

-  In short, positive productivity shocks increase the consumption variables (C and I), demand for inputs (L and K) and the prices of these factors of production ( W and R).

# References:
- Costa, C. (2018). Understanding DSGE models: Theory and Applications. Vernon Press.
- https://datascience.quantecon.org/scientific/optimization.html


# Introduction to the Harmonic Oscillator

*Note:* Much of this is adapted/copied from https://flothesof.github.io/harmonic-oscillator-three-methods-solution.html

This week week we are going to begin studying molecular dynamics, which uses classical mechanics to study molecular systems. Our "hydrogen atom" in this section will be the 1D harmomic oscillator.

The harmonic oscillator is a system that, when displaced from its equilibrium position, experiences a restoring force F proportional to the displacement x:

$$F=-kx$$

The potential energy of this system is

$$V = {1 \over 2}k{x^2}$$

These are sometime rewritten as

$$ F=- \omega_0^2 m x, \text{ }  V(x) = {1 \over 2}  \omega_0^2 m {x^2}$$

Where $\omega_0  = \sqrt {{k \over m}} $

In classical mechanics, our goal is to determine the equations of motion, $x(t),y(t)$, that describe our system.

In this notebook we will use sympy to solve an second order, ordinary differential equation.

## 1. Solving differential equations with sympy

Soliving differential equations can be tough, and there is not always a set plan on how to proceed. Luckily for us, the harmonic osscillator is the classic second order diffferential eqations.

Consider the following second order differential equation

$$ay(t)''+by(t)'=c$$

where $y(t)'' = {{{d^2}y} \over {dt^2}}$, and $y(t)' = {{{d}y} \over {dt}}$

We can rewrite this as a homogeneous linear differential equations

$$ay(t)''+by(t)'-c=0$$

The goal here is to find $y(t)$, similar to our classical mechanics problems. Lets use sympy to solve this equation

### Second order ordinary differential equation

First we import the sympy library

```python
import sympy as sym
```

Next we initialize pretty printing

```python
sym.init_printing()
```

Next we will set our symbols

```python
t,a,b,c=sym.symbols("t,a,b,c")
```

Now for somehting new. We can define functions using `sym.Function("f")`

```python
y=sym.Function("y")
y(t)
```

Now, If I want to define a first or second derivative, I can use `sym.diff`

```python
sym.diff(y(t),(t,1)),sym.diff(y(t),(t,2))
```

My differential equation can be written as follows

```python
dfeq=a*sym.diff(y(t),(t,2))+b*sym.diff(y(t),(t,1))-c
dfeq
```

```python
sol = sym.dsolve(dfeq)
sol
```

The two constants $C_1$ and $C_2$ can be determined by setting boundry conditions.
First, we can set the condition $y(t=0)=y_0$

The next intial condition we will set is $y'(t=0)=v_0$

To setup the equality we want to solve, we are using `sym.Eq`. This function sets up an equaility between a lhs aand rhs of an equation

```python
# sym.Eq example
alpha,beta=sym.symbols("alpha,beta")
sym.Eq(alpha+2,beta)
```

Back to the actual problem

```python
y0,v0=sym.symbols("y_0,v_0")
ics=[sym.Eq(sol.args[1].subs(t, 0), y0),
     sym.Eq(sol.args[1].diff(t).subs(t, 0), v0)]
ics
```

We can use this result to first solve for $C_2$ and then solve for $C_1$.
Or we can use sympy to solve this for us.

```python
solved_ics=sym.solve(ics)
solved_ics
```

Substitute the result back into $y(t)$

```python
full_sol = sol.subs(solved_ics[0])
full_sol
```

We can plot this result too. Assume that $a,b,c=1$ and that the starting conditions are $y_0=0,v_0=0$

We will use two sample problems:

* case 1 : initial position is nonzero and initial velocity is zero
* case 2 : initial position is zero and initialvelocity is nonzero

```python
# Print plots
%matplotlib inline
```

#### Initial velocity set to zero

```python
case1 = sym.simplify(full_sol.subs({y0:0, v0:0, a:1, b:1, c:1}))
case1
```

```python
sym.plot(case1.rhs)
sym.plot(case1.rhs,(t,-2,2))
```

#### Initial velocity set to one

```python
case2 = sym.simplify(full_sol.subs({y0:0, v0:1, a:1, b:1, c:1}))
case2
```

```python
sym.plot(case2.lhs,(t,-2,2))
```

## Calculate the phase space

As we will see in lecture, the state of our classical systems are defined as points in phase space, a hyperspace defined by ${{\bf{r}}^N},{{\bf{p}}^N}$. We will convert our sympy expression into a numerical function so that we can plot the path of $y(t)$ in phase space $y,y'$.

```python
case1
```

```python
# Import numpy library
import numpy as np

# Make numerical functions out of symbolic expressions
yfunc=sym.lambdify(t,case1.rhs,'numpy')
vfunc=sym.lambdify(t,case1.rhs.diff(t),'numpy')

# Make list of numbers
tlst=np.linspace(-2,2,100)

# Import pyplot
import matplotlib
import matplotlib.pyplot as plt
# Make plot
plt.plot(yfunc(tlst),vfunc(tlst))
plt.xlabel('$y$')
plt.ylabel("$y'$")
plt.show()
```

### Exercise 1.1

Change the initial starting conditions and see how that changes the plots. Make three different plots with different starting conditions

```python
# Your code here
# Import numpy library
import numpy as np

# Make numerical functions out of symbolic expressions
case1 = sym.simplify(full_sol.subs({y0:2, v0:1, a:1, b:1, c:1}))
case2 = sym.simplify(full_sol.subs({y0:1, v0:4, a:1, b:1, c:1}))

case1
sym.plot(case1.rhs,(t,-2,2))
case2
sym.plot(case2.lhs,(t,-2,2))
```

## 2. Harmonic oscillator

Applying the harmonic oscillator force to Newton's second law leads to the following second order differential equation

$$ F = m a $$

$$ F= - \omega_0^2 m x $$

$$ a = - \omega_0^2 x $$

$$ x(t)'' = - \omega_0^2 x $$

The final expression can be rearranged into a second order homogenous differential equation, and can be solved using the methods we used above

Your goal is determine and plot the equations of motion of a 1D harmomnic oscillator

### Exercise 2.1

1. Use the methodology above to determine the equations of motion $x(t), v(t)$ for a harmonic ocillator
1. Solve for any constants by using the following initial conditions: $x(0)=x_0, v(0)=v_0$
1. Show expressions for and plot the equations of motion for the following cases:
    1. $x(0)=0, v(0)=0$
    1. $x(0)=0, v(0)>0$
    1. $x(0)>0, v(0)=0$
    1. $x(0)<0, v(0)=0$
1. Plot the phasespace diagram for the harmonic oscillator

```python
# Your code here
omega0, t=sym.symbols("omega_0,t")
x=sym.Function("x")
x(t), omega0
```

```python
dfeq=sym.diff(x(t),t,2)+omega0**2*x(t)
dfeq
```

```python
sol=sym.dsolve(dfeq)
sol
```

```python
x0,v0=sym.symbols("x_0,v_0")
ics=[sym.Eq(sol.args[1].subs(t,0),x0),
    sym.Eq(sol.args[1].diff(t).subs(t,0),v0)]
ics
```

```python
solved_ics=sym.solve(ics)
solved_ics
```

```python
full_sol = sol.subs(solved_ics[0])
full_sol
```

```python
case1=sym.simplify(full_sol.subs({x0:1, v0:1}))
case1
```

```python
case_a = sym.simplify(full_sol.subs({x0:0, v0:0,omega0:1}))
case_a
```

```python
case_b = sym.simplify(full_sol.subs({x0:0, v0:1,omega0:1}))
case_b
```

```python
case_c = sym.simplify(full_sol.subs({x0:1, v0:0,omega0:1}))
case_c
```

```python
case_d = sym.simplify(full_sol.subs({x0:-1, v0:0,omega0:1}))
case_d
```

```python

```

```python

```


**IMPORTANT NOTICE**

Before submitting, **please**, make sure that your notebook runs **without errors** in Python 3.6
and **reproduces your solution as intended**, when you **Restart the Kernel and re-run the whole
notebook**!
You will be severely penalized if you notebook does not run.

Whereever applicable your solution will be graded based on the **plots**, generated by
**your code** on **TA's** computer.

# Home Assignment -- 5

Please, write your implementation within the designated blocks:
```python
...
### BEGIN Solution

# >>> your solution here <<<

### END Solution
...
```

Write your theoretical derivations within such blocks:
```markdown
**BEGIN Solution**

<!-- >>> your derivation here <<< -->

**END Solution**
```

## $\LaTeX$ in Jupyter
Jupyter has constantly improving $\LaTeX$ support. Below are the basic methods to
write **neat, tidy, and well typeset** equations in your notebooks:
* to write an **inline** equation use
```markdown
$ you latex equation here $
```
* to write an equation, that is **displayed on a separate line** use
```markdown
$$ you latex equation here $$
```
* to write a **block of equations** use
```markdown
\begin{align}
    left-hand-side
        &= right-hand-side on line 1
        \\
        &= right-hand-side on line 2
        \\
        &= right-hand-side on the last line
\end{align}
```
The **ampersand** (`&`) aligns the equations horizontally and the **double backslash**
(`\\`) creates a new line.

# Part 2 (19 + 10  pt.): Clustering

```python
import numpy as np

%matplotlib inline
import matplotlib.pyplot as plt
from sklearn.mixture import GaussianMixture
from sklearn.cluster import MiniBatchKMeans
from sklearn.cluster import SpectralClustering
from sklearn.cluster import KMeans
```

## Task 1 (11 + 4 pt.): Practice with Different Clustering Algorithms

In this task you are asked to try different clustering algorithms from sklearn on the provided data

```python
points, labels = np.loadtxt('data/clustering.txt'), np.loadtxt('data/clustering_labels.txt')
labels = labels.astype(int)
print("True number of clusters is {}".format(np.max(labels)))

plt.scatter(points[:,0], points[:,1], c=labels)
plt.show()
```

Try **KMeans, Gaussian Mixture** and **two** more clustering algorithms from sklearn.
To evaluate clustering performance use **two** clustering metrics: *silhouette score*
and *adjusted mutual information*.

For each algorithm **your task** is to
1. **(4 pt.)** Try to find the parameters leading to the best performance for `n_clusters=true_number_of_clusters`; **use these parameters hereafter**;
2. **(4 pt.)** Apply the algorithm with the true number of clusters and at least two **other settings** for the number of clusters: a smaller and a larger number than the true one;
3. **(3 pt.)** For each number of clusters **vizualize the clustering result**, calculate the **clustering metrics** and plot them;

**Bonus**
You get extra **+4 pt.** if you produce all your plots in one place and make as few
visualizations as possible (counting subplots). This means that you should depict the
comparison in a clear and comprehensible, yet concise manner.

**HINT** If plots cloud somehow overlay
one another and not sit side by side or in different cells...

```python
### BEGIN Solution
from sklearn import metrics

#CorrectNumber
y_predKM = KMeans(n_clusters=15, random_state=5).fit_predict(points)
y_predGM = GaussianMixture(n_components=15, random_state=5).fit(points).predict(points)
y_predMbkm = MiniBatchKMeans(n_clusters=15, random_state=5).fit_predict(points)
y_predSP = SpectralClustering(n_clusters=15, random_state=5).fit_predict(points)

sil_km = metrics.silhouette_score(points, y_predKM, metric='euclidean')
sil_gm = metrics.silhouette_score(points, y_predGM, metric='euclidean')
sil_Mbkm = metrics.silhouette_score(points, y_predMbkm, metric='euclidean')
sil_sp = metrics.silhouette_score(points, y_predSP, metric='euclidean')

mi_km = metrics.adjusted_mutual_info_score(labels, y_predKM)
mi_gm = metrics.adjusted_mutual_info_score(labels, y_predGM)
mi_Mbkm = metrics.adjusted_mutual_info_score(labels, y_predMbkm)
mi_sp = metrics.adjusted_mutual_info_score(labels, y_predSP)

#PLOT
f, ((ax11,ax12,ax13,ax14),(ax21,ax22,ax23,ax24),(ax31,ax32,ax33,ax34)) = plt.subplots(3,4,figsize=(22,17))

ax11.set_title("Kmeans")
ax11.scatter(points[:, 0], points[:, 1], c=y_predKM)
ax11.set_ylabel("15 clusters")
ax11.set_xlabel("Sil: "+str(sil_km)+" MI: "+str(mi_km))

ax12.set_title("GaussianMixture")
ax12.scatter(points[:, 0], points[:, 1], c=y_predGM)
ax12.set_xlabel("Sil: "+str(sil_gm)+" MI: "+str(mi_gm))

ax13.set_title("MiniBatchKMeans")
ax13.scatter(points[:, 0], points[:, 1], c=y_predMbkm)
ax13.set_xlabel("Sil: "+str(sil_Mbkm)+" MI: "+str(mi_Mbkm))

ax14.set_title("SpectralClustering")
ax14.scatter(points[:, 0], points[:, 1], c=y_predSP)
ax14.set_xlabel("Sil: "+str(sil_sp)+" MI: "+str(mi_sp))

#Low Number of clusters
y_predKM = KMeans(n_clusters=5, random_state=5).fit_predict(points)
y_predGM = GaussianMixture(n_components=5, random_state=5).fit(points).predict(points)
y_predMbkm = MiniBatchKMeans(n_clusters=5, random_state=5).fit_predict(points)
y_predSP = SpectralClustering(n_clusters=5, random_state=5).fit_predict(points)

sil_km = metrics.silhouette_score(points, y_predKM, metric='euclidean')
sil_gm = metrics.silhouette_score(points, y_predGM, metric='euclidean')
sil_Mbkm = metrics.silhouette_score(points, y_predMbkm, metric='euclidean')
sil_sp = metrics.silhouette_score(points, y_predSP, metric='euclidean')

mi_km = metrics.adjusted_mutual_info_score(labels, y_predKM)
mi_gm = metrics.adjusted_mutual_info_score(labels, y_predGM)
mi_Mbkm = metrics.adjusted_mutual_info_score(labels, y_predMbkm)
mi_sp = metrics.adjusted_mutual_info_score(labels, y_predSP)

#PLOT
ax21.set_title("Kmeans")
ax21.scatter(points[:, 0], points[:, 1], c=y_predKM)
ax21.set_ylabel("5 clusters")
ax21.set_xlabel("Sil: "+str(sil_km)+" MI: "+str(mi_km))

ax22.set_title("GaussianMixture")
ax22.scatter(points[:, 0], points[:, 1], c=y_predGM)
ax22.set_xlabel("Sil: "+str(sil_gm)+" MI: "+str(mi_gm))

ax23.set_title("MiniBatchKMeans")
ax23.scatter(points[:, 0], points[:, 1], c=y_predMbkm)
ax23.set_xlabel("Sil: "+str(sil_Mbkm)+" MI: "+str(mi_Mbkm))

ax24.set_title("SpectralClustering")
ax24.scatter(points[:, 0], points[:, 1], c=y_predSP)
ax24.set_xlabel("Sil: "+str(sil_sp)+" MI: "+str(mi_sp))

#High Number of clusters
y_predKM = KMeans(n_clusters=25, random_state=5).fit_predict(points)
y_predGM = GaussianMixture(n_components=25, random_state=5).fit(points).predict(points)
y_predMbkm = MiniBatchKMeans(n_clusters=25, random_state=5).fit_predict(points)
y_predSP = SpectralClustering(n_clusters=25, random_state=5).fit_predict(points)

sil_km = metrics.silhouette_score(points, y_predKM, metric='euclidean')
sil_gm = metrics.silhouette_score(points, y_predGM, metric='euclidean')
sil_Mbkm = metrics.silhouette_score(points, y_predMbkm, metric='euclidean')
sil_sp = metrics.silhouette_score(points, y_predSP, metric='euclidean')

mi_km = metrics.adjusted_mutual_info_score(labels, y_predKM)
mi_gm = metrics.adjusted_mutual_info_score(labels, y_predGM)
mi_Mbkm = metrics.adjusted_mutual_info_score(labels, y_predMbkm)
mi_sp = metrics.adjusted_mutual_info_score(labels, y_predSP)

#PLOT
ax31.set_title("Kmeans")
ax31.scatter(points[:, 0], points[:, 1], c=y_predKM)
ax31.set_ylabel("25 clusters")
ax31.set_xlabel("Sil: "+str(sil_km)+" MI: "+str(mi_km))

ax32.set_title("GaussianMixture")
ax32.scatter(points[:, 0], points[:, 1], c=y_predGM)
ax32.set_xlabel("Sil: "+str(sil_gm)+" MI: "+str(mi_gm))

ax33.set_title("MiniBatchKMeans")
ax33.scatter(points[:, 0], points[:, 1], c=y_predMbkm)
ax33.set_xlabel("Sil: "+str(sil_Mbkm)+" MI: "+str(mi_Mbkm))

ax34.set_title("SpectralClustering")
ax34.scatter(points[:, 0], points[:, 1], c=y_predSP)
ax34.set_xlabel("Sil: "+str(sil_sp)+" MI: "+str(mi_sp))

### END Solution
```

## Task 2 (8 + 6 pt.): Finding the Number of Clusters with Bootstrap

For a chosen algorithm find the best number of clusters using bootstrap.

So first based on your observations made at previous task, choose one algorithm assuming **you do not have true labels and do not know the true number of clusters**. Justify your choice. **(1 pt.)**

**BEGIN Solution**

Spectral clustering produces very noisy structure, so it is inappropriate. Kmeans and miniBatch KMeans are very geometrically sensitive. Therefore it is better to use Gaussian mixtures, with this we successfuly can derive clusters of number close to the original one. If we go far from it, then either silhouette coefficient or MI will suffer.

**END Solution**

**Your task** is to
1. **(2 pt.)** For each bootstrap sample perform clusterization and calculate **silhouette score**;
2. **(3 pt.)** Estimate variance of the metric and construct normal 95% **confidence intervals**;
3. **(2 pt.)** **Plot the metric** with the corresponding confidence intervals and chose the best number of clusters.

**Bonus** You may get extra **+2 pt.** if **in addition to** the silhouette score you use
**another metric** and **justify your choice of the algorithm**.
If you choose the [Gap statistic](https://web.stanford.edu/~hastie/Papers/gap.pdf),
*correctly implement and apply* it, you get extra **+4 pt.**.

```python
from tqdm import tqdm_notebook

### BEGIN Solution

cluster_ns = np.arange(5,25)
sample_number = 100

#Optimize weighted sum  1/2 Sil + 1/2 MI
bestScore = -20
bestScoreSil = -20

myMetricMeans=np.zeros(cluster_ns.shape[0])
myMetricStds=np.zeros(cluster_ns.shape[0])
silMetricMeans=np.zeros(cluster_ns.shape[0])
silMetricStds=np.zeros(cluster_ns.shape[0])

for i_cn, n_clusters in tqdm_notebook(enumerate(cluster_ns)):
    # >>> your code here <<<
    myMetrics=np.zeros(sample_number)
    silMetrics=np.zeros(sample_number)
    for sample_index in tqdm_notebook(range(sample_number)):

        idx = np.random.choice(range(points.shape[0]), points.shape[0], replace=True)

        y_predGM = GaussianMixture(n_components=n_clusters, random_state=5).fit(points[idx,:]).predict(points[idx,:])
        sil_gm = metrics.silhouette_score(points[idx,:], y_predGM, metric='euclidean')
        mi_gm = metrics.adjusted_mutual_info_score(labels[idx], y_predGM)

        silMetrics[sample_index]=sil_gm
        myMetrics[sample_index]=sil_gm*0.5 + mi_gm*0.5

    myMetricMean = np.mean(myMetrics)
    myMetricStd = np.std(myMetrics)

    silMetricMean = np.mean(silMetrics)
    silMetricStd = np.std(silMetrics)

    if(myMetricMean > bestScore):
            bestScore=myMetricMean
            bestNumber=n_clusters

    if(silMetricMean > bestScoreSil):
            bestScoreSil=silMetricMean
            bestNumberSil=n_clusters

    myMetricMeans[i_cn]=myMetricMean
    myMetricStds[i_cn]=myMetricStd

    silMetricMeans[i_cn]=silMetricMean
    silMetricStds[i_cn]=silMetricStd

# >>> your code here <<<

```

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

```python
f,(ax1,ax2) = plt.subplots(1,2,figsize=(21,6))

index = cluster_ns
bar_width = 0.6

opacity = 0.4
error_config = {'ecolor': '0.3'}

ax1.bar(index, myMetricMeans, bar_width,
                 alpha=opacity,
                 color='b',
                 yerr=myMetricStds,
                 error_kw=error_config)

ax1.set_xlabel('Number of clusters',fontsize=16)
ax1.set_ylabel('CombinedMetric Scores',fontsize=16)
f.suptitle('Bootstrap Scores for Clustering',fontsize=20)
ticks=ax1.set_xticks(index) #to silence the output
ax1.tick_params(axis='both', which='major', labelsize=14)

ax2.bar(index, silMetricMeans, bar_width,
                 alpha=opacity,
                 color='b',
                 yerr=silMetricStds,
                 error_kw=error_config)

ax2.set_xlabel('Number of clusters',fontsize=16)
ax2.set_ylabel('Silhouette Scores',fontsize=16)
ticks=ax2.set_xticks(index) #to silence the output
ax2.tick_params(axis='both', which='major', labelsize=14)

### END Solution
```

The interesting thing is that both metrics suggest to choose 15 as optimal number of clusters, however silhouette selector is more uncertain about that. Anyway, the optimum for number of clusters will be nearby 15 if we use combined metrics or silhouette coefficient for model selection.


Linear-Algebra(一)
# 1 预备知识
## 1.1 基本概念
### - 标量
标量表示单独的一个数，通常用斜体小写字母表示，如 $s\in \mathbb R, n \in \mathbb N$。
### - 向量
向量是一个一维数组，这些数字有一定的顺序，可以通过下标获取对应的值，通常用粗体小写字母表示：$\mathbf{x} \in \mathbb R^n$，表示元素取实数，且有 $n$ 个元素，它的元素用对应的斜体小写字母加下标表示：$x_1$。一般我们将向量写成列向量的形式：

$$
x = \begin{bmatrix}
x_1 \\
x_2 \\
\cdots \\
x_n
\end{bmatrix}
$$

有时候我们需要得到一个向量的子集，例如得到第 $1,3,6$ 个元素，那么我们可以令集合 $S = \{1,3,6\}$，然后用 $\mathbf x_{S}$ 来表示这个子集。

另外，我们用 $-$ 的下标表示去集合的补：$\mathbf x_{-1}$ 表示除了 $x_1$ 之外 $\mathbf x$ 中的所有元素，$\mathbf x_{-S}$ 表示除了 $x_1, x_3, x_6$ 之外 $\mathbf x$ 中的所有元素。
### - 矩阵
矩阵是一个二维数组，每个元素的下标有两个，通常用大写粗体字母表示：$\mathbf A \in \mathbb R^{m\times n}$ 表示取实值的 $m$ 行 $n$ 列矩阵，其元素用对应的小写斜体字母加下标表示：$A_{1,1}, A_{m,n}$。

另外，我们用 $:$ 下标表示矩阵的一行或者一列：$\mathbf A_{i,:}$ 第 $i$ 行，$\mathbf A_{:,j}$ 第 $j$ 列。

矩阵可以写成这样的形式：

$$
\begin{bmatrix}
A_{1,1} & A_{1,2} \\
A_{2,1} & A_{2,2} \\
\end{bmatrix}
$$

有时候我们需要对矩阵进行逐元素操作，例如将函数 $f$ 应用到 $\bf A$ 的所有元素上，此时我们用 $f({\bf A})_{i,j}$ 表示。
### - 张量
张量是超过二维的数组，我们用 $\sf A$ 表示张量，${\sf A}_{i,j,k}$ 表示其元素（三维张量情况下）。
### - 矩阵转置
矩阵的转置操作相当于沿着对角线翻转，定义为：

$$
\mathbf A^\top_{i,j} = \mathbf A_{i,j}
$$

矩阵转置的转置等于矩阵本身：

$$\bf \left(A^\top\right)^\top=A$$

转置将矩阵的形状从 $m\times n$ 变成了 $n \times m$。

向量可以看成是只有一列的矩阵，因此也可以进行转置操作，有时候为了方便，我们可以使用行向量加转置的操作，如：$\mathbf x=[x_1,x_2,x_3]^\top$

标量也可以看出是一行一列的矩阵，其转置等于自己：$a^\top=a$。
### - 矩阵加法和数乘
加法即逐元素相加，要求两个矩阵的形状一样：

$$
\mathbf{C = A+B}, C_{i,j}=A_{i,j}+B_{i,j}
$$

数乘即一个标量与矩阵相乘：

$$
\mathbf D = a \cdot \mathbf B + c, D_{i,j}=a \cdot B_{i,j}+c
$$

有时候我们允许使用一个矩阵和向量相加的形式，得到一个矩阵，把 $\bf b$ 加到了 $C$ 的每一行上，本质上是构造了一个将 $\bf b$ 按行复制的一个新矩阵，这种机制叫做 `broadcasting`：

$$
\mathbf {C = A + b}, C_{i,j} = A_{i,j} + b_{j}
$$

## 1.2 矩阵乘法
对于矩阵$A_{m \times n}$和矩阵$B_{n \times p}$，$A \times B = C$，其中$C$为$m \times p$的矩阵，其每一个元素,$$C_{ij}=\sum_{k=1}^{n}a_{ik} \times b_{kj}$$在`numpy`中使用`dot`运算符来进行数学上矩阵乘法运算，而普通的`*`运算则是$A,B$矩阵对应位置元素的乘积。
### 矩阵乘法性质
矩阵乘法满足结合律（`associative`）和分配律（`distributive`）：

$$
\begin{align}
\bf A(B+C)&=\bf AB+AC \\
\bf A(BC)&=\bf (AB)C
\end{align}
$$

矩阵乘法通常是不可交换的：

$$
\bf AB \neq BA
$$

但是向量内积是可交换的：

$$
\bf x^\top y = y^\top x
$$

矩阵乘法的转置形式如下：

$$
\bf (AB)^\top = B^\top A^\top
$$

利用这个式子和标量转置等于其本身，我们马上得到内积是可交换的结论：

$$
\bf x^\top y = (x^\top y)^\top = y^\top x
$$

## 1.3 单位矩阵和逆
### 单位矩阵
为了引入矩阵的逆，我们需要先定义单位矩阵：单位矩阵乘以任意一个向量等于这个向量本身。记 $\mathbf I_n$ 为保持 $n$ 维向量不变的单位矩阵，即：

$$
\mathbf I_n \in \mathbb R^{n\times n}, \forall \mathbf x \in \mathbb R^{n}, \mathbf I_n \mathbf{x=x}
$$

单位矩阵的形式十分简单，所有的对角元素都为 $1$，其他元素都为 $0$，如：

$$
\mathbf I_3
\begin{bmatrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1 \\
\end{bmatrix}
$$
### 矩阵的逆
矩阵 $\bf A$ 的逆记作 $\mathbf A^{-1}$，定义为一个矩阵使得

$$
\mathbf A^{-1} \mathbf A = \mathbf I_n
$$

如果 $\mathbf A^{-1}$ 存在，那么线性方程组 $\bf Ax=b$ 的解为：

$$
\mathbf A^{-1}\mathbf{Ax} = \mathbf I_n \mathbf x = \mathbf x = \mathbf A^{-1}\mathbf b
$$

##  1.4 向量和矩阵相乘
+ 向量右乘
$\Bigg[v_1\ v_2\ v_3\Bigg]\begin{bmatrix}3\\4\\5\end{bmatrix}=3v_1+4v_2+5v_3$

向量$v_1,v_2,v_3$分别代表列向量，当$v_1,v_2,v_3$组成的矩阵右乘一个列向量，则相当于使用右边的列向量中的数值对$v_1,v_2,v_3$进行线性组合。
+ 向量左乘
$\begin{bmatrix}1&2&7\end{bmatrix}\begin{bmatrix} row_1\\row_2\\row_3\end{bmatrix}=1row_1+2row_2+7row3$

向量$row_1,row_2,row_3$分别代表了行向量，当$row_1,row_2,row_3$组成的的矩阵左乘一个行向量，则相当于使用左边行向量中的数值对$row_1,row_2,row_3$进行线性组合。

## 1.5 置换矩阵
由1.2的向量和矩阵相乘后的分析，当若干个向量拼接在一起组成矩阵的时候，将会更多的向量的线性组合。$$\begin{bmatrix} a & b \\ c & d \end{bmatrix}\begin{bmatrix}0 & 1 \\ 1 & 0 \end{bmatrix}=\begin{bmatrix}b & a \\ d & c \end{bmatrix}$$

将矩阵乘法拆分如下：

$\begin{bmatrix} b \\ d \end{bmatrix} = \begin{bmatrix} a & b \\ c & d \end{bmatrix} \begin{bmatrix}0 \\ 1 \end{bmatrix}=0 \times \begin{bmatrix} a \\ c \end{bmatrix} + 1 \times \begin{bmatrix} b \\ d \end{bmatrix}$

$\begin{bmatrix} a \\ c \end{bmatrix} = \begin{bmatrix} a & b \\ c & d \end{bmatrix} \begin{bmatrix}1 \\ 0 \end{bmatrix}=1 \times \begin{bmatrix} a \\ c \end{bmatrix} + 0 \times \begin{bmatrix} b \\ d \end{bmatrix}$
其等价效果为**交换矩阵的两列**。
同理，当左乘一个矩阵的时候
$$\begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix}\begin{bmatrix}a & b \\ c & d \end{bmatrix}=\begin{bmatrix}c & d \\ a & b \end{bmatrix}$$
其等价效果为**交换矩阵的两行**。

置换矩阵定义：
当一个矩阵的系数为0或者1的方阵，其每一行和每一列上*恰好*只有一个1，其余的为0，那么该矩阵则为置换矩阵。

# 1.6 消元矩阵
通过矩阵行之间的运算，使得矩阵达到精简的程度，比如：
$$\begin{bmatrix}\underline{1}&2&1\\3&8&1\\0&4&1\end{bmatrix}\xrightarrow{row_2-3row_1}\begin{bmatrix}\underline{1}&2&1\\0&2&-2\\0&4&1\end{bmatrix} \xrightarrow{row_3-2row_1}\begin{bmatrix}\underline{1}&2&1\\0&\underline{2}&-2\\0&0&\underline{5}\end{bmatrix}$$

每一个行之间的运算可以看成矩阵右乘一个矩阵:
$row_2-3row_1 \Rightarrow E_{12}=\begin{bmatrix}1 & 0 & 0 \\ -3 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}$

$row_3-2row_1 \Rightarrow E_{23}= \begin{bmatrix}1 & 0 & 0 \\ 0 & 1 & 0 \\ -2 & 0 & 1 \end{bmatrix}$
综上所述：
$$E_{23}(E_{12}A)=(E_{23}E_{12})A=U$$,其中$U$为上三角矩阵，进一步来讲$$(E_{23}E_{12})A=U \Rightarrow A=(E_{23}E_{12})^{-1}U=LU$$
这就是矩阵的LU分解，其中$L$为下三角矩阵。有时矩阵的主元**pivot**为零，那么就需要进行交换有关的行，即在矩阵A右乘一个置换矩阵P，则$PA=LU$

# 1.7 矩阵求逆
如果矩阵A为$n$阶方阵，那么$AA^{-1}=I$,称$A^{-1}$为逆矩阵，矩阵求逆的方法: $$[ A | I ] \Rightarrow [I | A^{-1}]$$ 在矩阵A的右边写上相同阶的单位阵，形成一个$n \times 2n$的矩阵，通过行变换，将左边矩阵A转换成单位矩阵的时候，矩阵右边的单位阵将会转换为A的逆。
**证明**
对于矩阵A而言，每一部行操作都会被右边的矩阵记录下来，当矩阵A转换成单位阵的时候，相当于左乘了一个$A^{-1}$,所以$$A^{-1}[A|I]=[A^{-1}A | A^{-1}I]=[I|A^{-1}]$$