```python
import holoviews as hv
hv.extension('bokeh')
hv.opts.defaults(hv.opts.Curve(width=500),
                 hv.opts.Image(width=500, colorbar=True, cmap='Viridis'))
```

```python
import numpy as np
import scipy.signal
import scipy.fft
from IPython.display import Audio
```

# Diseño de sistemas y filtros IIR

Un filtro FIR de buena calidad puede requerir una gran cantidad de coeficientes

Es posible implementar filtros más eficientes usando **recursividad**. Esta es la base de los filtros de respuesta al impulso infinita o IIR que veremos en esta lección

## Definición de un sistema IIR

Generalizando el sistema FIR para incluir versiones pasadas de la salida y asumiendo $a[0] = 1$ llegamos a

$$
\begin{align}
y[n] &= b[0] x[n] + b[1] x[n-1] + b[2] x[n-2] + \ldots + b[L] x[n-L]  \nonumber \\
& - a[1] y[n-1] - a[2] y[n-2] - \ldots - a[M] y[n-M] \nonumber \\
&= \sum_{l=0}^{L} b[l] x[n-l] - \sum_{m=1}^{M} a[m] y[n-m]  \nonumber  \\
\sum_{m=0}^{M} a[m] y[n-m] &= \sum_{l=0}^{L} b[l] x[n-l] \nonumber \\
(a * y)[n] &= (b * x)[n], \nonumber
\end{align}
$$

es decir dos convoluciones discretas que definen una **ecuación de diferencias**

Este tipo de sistema se conoce como
- sistema *infinite impulse response* (IIR)
- sistema *auto-regresive moving average* (ARMA)
    - autoregresivo de orden M: incluye valores pasados de la salida
    - media movil de orden L+1: pondera el valor presente y pasados de la entrada

Podemos ver el sistema IIR como una generalización del sistema FIR. El caso particular del sistema FIR se recupera si

$a[m] = 0$ para $m=[1, \ldots, M]$

### Respuesta en frecuencia del sistema IIR

Aplicando la transformada de Fourier convertimos las convoluciones en multiplicaciones y encontramos la respuesta en frecuencia como

$$
\begin{align}
\text{DFT}_N[(a * y)[n]] &= \text{DFT}_N[(b * x)[n]] \nonumber \\
A[k] Y[k] &= B[k] X[k] \nonumber \\
H[k] = \frac{Y[k]}{X[k]} &= \frac{B[k]}{A[k]} = \frac{ \sum_{l=0}^L b[l]e^{-j \frac{2\pi}{N} nl} }{ \sum_{m=0}^M a[m]e^{-j \frac{2\pi}{N} mk}} \nonumber
\end{align}
$$

que existe siempre que $A[k] \neq 0$.

La respuesta en frecuencia también suele expresarse como

$$
H[k] = K \frac{ \prod_{l=1}^L (e^{j \frac{2\pi}{N} k}- \beta[l]) }{ \prod_{m=1}^M (e^{j \frac{2\pi}{N} k}- \alpha[m])}
$$

donde

- $K$ se llama **ganancia**
- las raices del polinomio del numerador $\alpha$ se llaman conjuntamente **ceros**
- las raices del polinomio del denominador $\beta$ se llaman conjuntamente **polos**

### Ejemplo de respuesta al impulso de un sistema IIR

Consideremos el siguiente sistema IIR

$$
\begin{align}
y[n] &= (1-\gamma) x[n] + \gamma y[n-1] \nonumber \\
y[n] - \gamma y[n-1] &= (1-\gamma) x[n] \nonumber
\end{align}
$$

Los coeficientes del sistema son

$a[0] = 1$, $a[1] = -\gamma$ y $b[0] = (1-\gamma)$

Es decir que es AR de orden 1 y  MA de orden 1

¿Cúal es su respuesta al impulso? Asumiendo $y[n]=0, n<0$, tenemos que

$$
\begin{matrix}
n & \delta[n] & y[n] \\
-2 & 0 & 0 \\
-1 & 0 & 0 \\
0 & 1 & (1-\gamma) \\
1 & 0 & \gamma(1-\gamma) \\
2 & 0 & \gamma^2(1-\gamma)  \\
3 & 0 & \gamma^3(1-\gamma) \\
4 & 0 & \gamma^4(1-\gamma) \\
\end{matrix}
$$

¿Cómo cambia la respuesta al impulso con distintos valores de $\gamma$? ¿Qué pasa si $\gamma \geq 1$?

Respondamos estas preguntas visualizando la respuesta al impulso de este sistema con la función `scipy.signal.dimpulse`

```python
# Valores de gamma que probaremos:
gamma = [-1.5, -1, -0.5, 0.5, 1., 1.5]

p = []
for g in gamma:
    t, y = scipy.signal.dimpulse(([1-g, 0], [1,-g], 1), x0=0, n=30)
    p.append(hv.Curve((t, y[0][:, 0]), label=f"gamma={g}"))

hv.Layout(p).cols(3).opts(hv.opts.Curve(width=250, height=200, axiswise=True))
```

De las figuras podemos ver que:

- Para $\gamma < 0$ (primera fila) los coeficientes del sistema son alternantes en signo
- Para $|\gamma| < 1$ los coeficientes del sistema tienden a cero
- Para $|\gamma| > 1$ los coeficientes del sistema divergen y tienen a infinito

:::{warning}

A diferencia de un sistema FIR, el sistema IIR puede tener configuraciones inestables en que los coeficientes crecen o decrecen infinitamente

:::

Por otro lado consideremos el sistema anterior y asumamos que $|\gamma|<1$, desenrollando tenemos que

$$
\begin{align}
y[0] &= (1-\gamma) x[0] \nonumber \\
y[1] &= (1-\gamma) (x[1] + \gamma x[0]) \nonumber \\
y[2] &= (1-\gamma) (x[2] + \gamma x[1] + \gamma^2 x[0]) \nonumber \\
y[3] &= (1-\gamma) (x[3] + \gamma x[2] + \gamma^2 x[1] + \gamma^3 x[0]) \nonumber \\
y[4] &= (1-\gamma) (x[4] + \gamma x[3] + \gamma^2 x[2] + \gamma^3 x[1]  + \gamma^4 x[0]) \nonumber \\
y[5] &= \ldots \nonumber
\end{align}
$$

:::{note}

Con un sistema IIR de pocos coeficientes podemos representar un sistema FIR considerablemente más grande

:::

En el ejemplo anterior, si escogemos $\gamma$ tal que $\gamma^{20 }\approx 0$ entonces aproximamos un sistema FIR de orden 20 con tan sólo 3 coeficientes

### Ejemplo de respuesta en frecuencia de un sistema IIR

Para el sistema del ejemplo anterior su respuesta en frecuencia es

$$
\begin{align}
Y[k] &= (1-\gamma) X[k] + \gamma Y[k] e^{-j \frac{2\pi}{N} k} \nonumber \\
H[k] = \frac{Y[k]}{X[k]} &= \frac{1-\gamma}{1 - \gamma e^{-j \frac{2\pi}{N} k} }  \nonumber
\end{align}
$$

que en notación de polos y ceros se escribe como

$$
H[k] =  (1-\gamma)\frac{e^{j \frac{2\pi}{N} k} - 0}{e^{j \frac{2\pi}{N} k} - \gamma  }
$$

es decir que tiene un cero en $0$, un polo en $\gamma$ y una ganancia de $(1-\gamma)$

Para entender mejor este sistema estudiemos la magnitud de $|H[k]|$ para $\gamma < 1$

$$
\begin{align}
| H[k]| &= \frac{|1-\gamma|}{|1 - \gamma e^{-j \frac{2\pi}{N} k}|}  \nonumber \\
&=  \frac{1-\gamma}{\sqrt{1 - 2\gamma \cos(\frac{2\pi}{N} k) + \gamma^2}} \nonumber
\end{align}
$$

¿Cómo se ve $|H[k]|$? ¿Qué función cumple este sistema?

```python
k = np.arange(-24, 25)/50
Hk = lambda gamma, k : (1-gamma)/np.sqrt(1 - 2*gamma*np.cos(2.0*np.pi*k) + gamma**2)
```

```python
p = []
for gamma in [0.25, 0.5, 0.75]:
    p.append(hv.Curve((k, Hk(gamma, k)), 'Frecuencia', 'Respuesta', label=f'gamma={gamma}'))

hv.Overlay(p)
```

:::{note}

Este sistema atenua las frecuencias altas, es decir que actua como un filtro pasa bajos

:::

## Diseño de filtros IIR simples

Los filtros IIR más simples son los de un polo y un cero, es decir filtros de primer orden

$$
H[k] = \frac{b[0] + b[1] e^{-j \frac{2\pi}{N} k}}{1 + a[1] e^{-j \frac{2\pi}{N} k}}  = K\frac{e^{j \frac{2\pi}{N} k} - \beta}{e^{j \frac{2\pi}{N} k} - \alpha  }
$$

donde podemos reconocer

- $b[0]=K$
- $\beta = - b[1] \cdot K$
- $\alpha=-a[1]$

Definimos la frecuencia de corte $f_c$ como aquella frecuencia en la que el filtro alcanza una atenuación de 0.7 (-3 dB). Haciendo la equivalencia con el ejemplo anterior tenemos que $\gamma = e^{-2\pi f_c}$

### Receta para un filtro pasa bajo IIR con frecuencia de corte $f_c$

Asignamos

- $b[0] = 1 - e^{-2\pi f_c}$
- $b[1] = 0$
- $a[1] = -e^{-2\pi f_c}$

Lo que resulta en la siguiente respuesta en frecuencia

$$
H[k] = \frac{1-e^{-2\pi f_c}}{1 - e^{-2\pi f_c} e^{-j \frac{2\pi}{N} k}} = (1-e^{-2\pi f_c}) \frac{(e^{j \frac{2\pi}{N} k}- 0)}{(e^{j \frac{2\pi}{N} k} - e^{-2\pi f_c} )}
$$

Es decir un cero en $0$, un polo en $e^{-2\pi f_c}$ y ganancia $1-e^{-2\pi f_c}$

### Receta para un filtro pasa alto IIR con frecuencia de corte $f_c$

Asignamos

- $b[0] = (1 + e^{-2\pi f_c})/2$
- $b[1] = -(1 + e^{-2\pi f_c})/2$
- $a[1] = -e^{-2\pi f_c}$

Lo que resulta en la siguiente respuesta en frecuencia

$$
H[k] = \frac{1+e^{-2\pi f_c}}{2} \frac{(e^{j \frac{2\pi}{N} k} - 1)}{(e^{j \frac{2\pi}{N} k} - e^{-2\pi f_c})}
$$

Es decir un cero en $1$, un polo en $e^{-2\pi f_c}$ y ganancia $\frac{1+e^{-2\pi f_c}}{2}$

### Aplicar un filtro a una señal con scipy

Para filtrar una señal unidimensional con un filtro IIR (sin variar la fase de la señal) podemos utilizar la función

```python
    scipy.signal.filtfilt(b, # Coeficientes del numerador
                          a, # Coeficientes del denominador
                          x, # Señal a filtrar
                          ...
                         )
```

Los siguientes ejemplos muestran un señal de tipo pulso rectangular filtrada con sistemas IIR de primer orden pasa bajo y pasa-alto diseñados con las recetas mostradas anteriormente

```python
n = np.arange(0, 500)
x = 0.5 + 0.5*scipy.signal.square((n)/(2.*np.pi*5), duty=0.3)
```

```python
def iir_low_pass(signal, fc):
    gamma = np.exp(-2*np.pi*(fc))
    b, a = [(1-gamma), 0], [1, -gamma]
    return scipy.signal.filtfilt(b, a, signal)

y = {}
for fc in [0.05, 0.02, 0.01]:
    y[fc] = iir_low_pass(x, fc)
```

```python
px = hv.Curve((n, x))
py = []
for fc, y_ in y.items():
    py.append(hv.Curve((n, y_), label=f'fc={fc}'))

hv.Layout([px, hv.Overlay(py)]).cols(1).opts(hv.opts.Curve(height=200))
```

```python
def iir_high_pass(signal, fc):
    gamma = np.exp(-2*np.pi*(fc))
    b, a = [(1+gamma)/2, -(1+gamma)/2], [1, -gamma]
    return scipy.signal.filtfilt(b, a, signal)

y = {}
for fc in [0.01, 0.02, 0.05]:
    y[fc] = iir_high_pass(x, fc)
```

```python
px = hv.Curve((n, x))
py = []
for fc, y_ in y.items():
    py.append(hv.Curve((n, y_), label=f'fc={fc}'))

hv.Layout([px, hv.Overlay(py)]).cols(1).opts(hv.opts.Curve(height=200))
```

:::{note}

El filtro pasa-bajos suaviza los cambios de los pulsos rectangulares. El filtro pasa-altos elimina las zonas constantes y resalta los cambios de la señal.

:::

## Diseño de filtros IIR de segundo orden

Los filtros IIR de segundo orden o **biquad** tienen dos polos y dos ceros.

Su respuesta en frecuencia es

$$
H[k] = \frac{b[0] + b[1] W_N^k + b[2] W_N^{2k}}{1 + a[1] W_N^k + a[2] W_N^{2k}} = K \frac{(W_N^{-k} - \beta_1) (W_N^{-k} - \beta_2)}{(W_N^{-k} - \alpha_1)(W_N^{-k} - \alpha_2)},
$$

donde $W_N = e^{-j \frac{2 \pi}{N}}$ y la relación entreo coeficientes y polos/ceros es:

$$
b[0] = K, \quad b[1] = -K (\beta_1 + \beta_2), \quad b[2]= K \beta_1\beta_2
$$

$$
a[1] = - (\alpha_1 + \alpha_2), \quad a[2]=\alpha_1 \alpha_2
$$

Con arquitecturas de segundo orden se pueden crear filtros pasabanda y rechaza banda

## Diseño de filtros IIR de orden mayor

Para crear los coeficientes de filtro IIR de orden mayor podemos usar la función

```python
scipy.signal.iirfilter(N, # Orden del filtro
                       Wn,  # Frecuencias de corte (normalizadas en [0,1])
                       fs, # Frecuencia de muestreo
                       btype='bandpass', # Tipo de filtro: 'bandpass', 'lowpass', 'highpass', 'bandstop'
                       ftype='butter', # Familia del filtro: 'butter', 'ellip', 'cheby1', 'cheby2', 'bessel'
                       output='ba', # Retornar coeficientes
                       ...
                       )
```

El filtro Butterworth es óptimo en el sentido de tener la banda de paso lo más plana posible.

Otros filtros se diseñaron con otras consideraciones.

Los filtros IIR digitales están basados en los filtros IIR analógicos.

Observe como al aumentar el orden el filtro pasabajo IIR comienza a cortar de forma más abrupta

```python
Hk = {}
for order in [1, 2, 5, 20]:
    b, a = scipy.signal.iirfilter(N=order, Wn=0.2, fs=1,
                                  ftype='butter', btype='lowpass', output='ba')
    freq, response = scipy.signal.freqz(b, a, fs=1)
    Hk[order] = np.abs(response)
```

```python
p = []
for order, response in Hk.items():
    p.append(hv.Curve((freq, response), 'Frecuencia', 'Respuesta', label=f'orden={order}'))
hv.Overlay(p)
```

## Comparación de la respuesta en frecuencia de filtros FIR e IIR del orden equivalente

Comparemos la respuesta en frecuencia de un filtro IIR y otro FIR ambos pasa-bajo con 20 coeficientes

```python
Fs = 1
fc = 0.25
h = scipy.signal.firwin(numtaps=20, cutoff=fc, pass_zero=True, window='hann', fs=Fs)
b, a = scipy.signal.iirfilter(N=9, Wn=fc, fs=Fs, ftype='butter', btype='lowpass')
display(len(h), len(b)+len(a))

freq_fir, response_fir = scipy.signal.freqz(h, 1, fs=Fs)
freq_iir, response_iir = scipy.signal.freqz(b, a, fs=Fs)
```

```python
p1 = hv.Curve((freq_fir, np.abs(response_fir)), 'Frecuencia', 'Respuesta', label='FIR')
p2 = hv.Curve((freq_iir, np.abs(response_iir)), 'Frecuencia', 'Respuesta', label='IIR')
hv.Overlay([p1, p2])*hv.VLine(fc).opts(color='k', alpha=0.5)
```

La linea negra marca la ubicación de la frecuencia de corte

:::{note}

El filtro IIR es mucho más abrupto, es decir filtra mejor, que el filtro FIR equivalente

:::

Una desventaja del filtro IIR es que por definición introduce una desfase no constante en la señal de salida

```python
freq_fir, delay_fir = scipy.signal.group_delay(system=(h, 1), fs=Fs)
freq_iir, delay_iir = scipy.signal.group_delay(system=(b, a), fs=Fs)
```

```python
p1 = hv.Curve((freq_fir, delay_fir), 'Frecuencia', 'Desfase', label='FIR')
p2 = hv.Curve((freq_iir, delay_iir), 'Frecuencia', 'Desfase', label='IIR')
hv.Overlay([p1, p2])*hv.VLine(fc).opts(color='k', alpha=0.5)
```

¿Cómo se ve una señal filtrada donde se preserva la fase versus una donde no se preserva la fase?

Consideremos la señal rectangular anterior y apliquemos un filtro pasa-bajo IIR de orden 1

Esta vez compararemos el filtro con la función `scipy.signal.lfilter` y la función `scipy.signal.filtfilt`. La primera no preserva la fase mientras que la segunda si lo hace

```python
Fs = 1
fc = 0.01
n = np.arange(0, 500)
x = 0.5 + 0.5*scipy.signal.square((n)/(2.*np.pi*5), duty=0.3)

b, a = scipy.signal.iirfilter(N=1, Wn=fc, fs=Fs, ftype='butter', btype='lowpass')
# No se preserva la fase
y_lfilter = scipy.signal.lfilter(b, a, x)
# Se preserva la fase
y_filtfilt = scipy.signal.filtfilt(b, a, x)
```

```python
px = hv.Curve((n, x), 'Tiempo', 'Entrada')
py = []
py.append(hv.Curve((n, y_filtfilt), 'Tiempo', 'Salida', label=f'Fase constante'))
py.append(hv.Curve((n, y_lfilter), 'Tiempo', 'Salida', label=f'Fase no constante'))

hv.Layout([px, hv.Overlay(py)]).cols(1).opts(hv.opts.Curve(height=200))
```

:::{note}

En el caso donde no se preserva la fase podemos notar que la señal de salida está desplazada con respecto a la original. Además los cambios tienen una transición asimétrica

:::

La función `scipy.signal.filtfilt` "arregla" el problema del desfase filtrando la señal dos veces. La primera vez se filtra hacia adelante en el tiempo y la segunda vez hacia atrás. Por ende no se puede aplicar en un escenario de tipo *streaming* donde los datos van llegando de forma causal.

En una aplicación causal donde se necesite preservar la fase debemos usar un filtro FIR.

## Apéndice: Efectos de audio con filtros IIR

El siguiente ejemplo muestra como implementar el conocido filtro Wah-wah usando un sistema IIR

Este es un filtro pasabanda modulado con ancho de pasada fijo $f_b$ [Hz] y una frecuencia central variable $f_c$ [Hz], donde La frecuencia central se modula con una onda lenta

Se modela como el siguiente sistema **IIR**

$$
H[k] = \frac{(1+c)W_N^{2k} -(1+c) }{W_N^{2k} + d(1-c)W_N^k -c}
$$

donde

$$
d=-\cos(2\pi f_c/f_s)
$$

y

$$
c = \frac{\tan(\pi f_b/f_s) -1}{\tan(2\pi f_b /f_s)+1}
$$

Veamos como modifica este filtro una señal de audio

```python
import librosa
data, fs = librosa.load("../../data/DPSAU.ogg")
Audio(data, rate=fs)
```

```python
data_wah = []
zi = np.zeros(shape=(2,))
# Parámetros fijos del filtro
fb, Nw = 200, 5
c  = (np.tan(np.pi*fb/fs) - 1.)/(np.tan(2*np.pi*fb/fs) +1)
# Filtramos una ventana de la señal moviendo lentamente fc
for k in range(len(data)//Nw):
    # Cálculo de la frecuencia central
    fc = 500 + 2000*(np.cos(2.0*np.pi*k*30./fs) +1)/2
    d = -np.cos(2*np.pi*fc/fs)
    # Coeficientes del filtro
    b, a = [(1+c), 0, -(1+c)], [1, d*(1-c), -c]
    # Filtramos, usando el filtrado anterior como borde (zi)
    data2, zi = scipy.signal.lfilter(b, a, data[k*Nw:(k+1)*Nw], zi=zi)
    # Guardamos
    data_wah.append(data2)
```

```python
Audio(np.hstack(data_wah), rate=int(fs))
```

Si quieres profundizar en el tema de los filtros IIR aplicados a efectos de audio recomiendo: https://www.ee.columbia.edu/~ronw/adst-spring2010/lectures/lecture2.pdf

```python

```


# Estimating text loss in Middle Dutch chivalric epics

This English-language, Python notebook accompanies the following publication:

> Mike Kestemont and Folgert Karsdorp, "Het Atlantis van de Middelnederlandse ridderepiek. Een schatting van het tekstverlies met methodes uit de ecodiversiteit". *Spiegel der letteren* (2020).

All figures and numbers were prepared using the code below. Future updates of the code and data will be managed in an open [Github repository](https://github.com/mikekestemont/chivalric_diversity). The code block below loads all (third-party) packages and modules necessary to run the module. These can be installed from the file `requirements.txt`:

    pip install -r requirements.txt

```python
from functools import partial
from itertools import product

import numpy as np
np.random.seed(12345)
from scipy.special import erfinv
import pandas as pd

import matplotlib.pyplot as plt
plt.style.use("tufte.mplstyle")
plt.rcParams["text.usetex"] = False
%matplotlib inline

import scipy.stats as stats
from scipy.special import gammaln
```

## Data

We load the data from the spreadsheet file `mnl.xlsx`:

```python
mnl = pd.read_excel('mnl.xlsx', header=None, names=('text', 'witness'))
mnl.head(10)
```

      text
      witness

      0
      Aiol (1)
      H1

      1
      Aiol (2)
      H2

      2
      Alexanders geesten
      München, Bayerische Staatsbibliotheek, Cod. ge...

      3
      Alexanders geesten
      H3

      4
      Alexanders geesten
      H4

      5
      Alexanders geesten
      H5

      6
      Alexanders geesten
      H6

      7
      Arturs doet
      Den Haag, KB, 129 A 10, fol. 201r-238r

      8
      Arturs doet
      Rijksarchief te Antwerpen, Sint-Catharinakapit...

      9
      Aubri de Borgengoen
      H7

We are only interested in the count data, i.e. the number of witnesses per text (the technical term is "abundance data").

```python
mnl.groupby('text').size().sort_values(ascending=False).head()
```

    text
    Historie van Troyen       16
    Roman van Limborch        10
    Madelgijs                 10
    Karel ende Elegast         7
    Parthonopeus van Bloys     6
    dtype: int64

The counts per text can be plotted as follows:

```python
fig, ax = plt.subplots(figsize=(10,18))
mnl.groupby('text').size().sort_values(ascending=True).plot.barh(ax=ax);
ax.set(xlabel='aantal handschriften', ylabel='',
       title='Distributie van de (ons bekende) ridderepische teksten over tekstgetuigen')
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)
plt.savefig('output/Fig1.jpeg', dpi=300, transparent=True)
```

Yet a different perspective is to list the size of the frequency bins that we can distinguish within the manuscript counts:

```python
types = mnl.groupby('text').size().sort_values(ascending=False).value_counts().sort_index()
types = types.to_frame(name='aantal teksten')
types['aantal handschriften'] = types.index
types.to_excel('output/Tab1.xlsx')
types
```

      aantal teksten
      aantal handschriften

      1
      44
      1

      2
      13
      2

      3
      6
      3

      4
      3
      4

      5
      3
      5

      6
      1
      6

      7
      1
      7

      10
      2
      10

      16
      1
      16

Finally, we define the auxiliary function `species_richness` to count the number of unique texts in the data (i.e. the number of non-zero counts):

```python
def species_richness(counts):
    return np.sum(counts > 0)

print('# unique texts:', species_richness(mnl.groupby('text').size()))
print('# witnesses:', len(mnl))
```

    # unique texts: 74
    # witnesses: 164

## Jackknife

The following function computes the first-order Jackknife estimate, on the basis of the abundance data in our data frame, as well as a confidence interval (.95 be default). This approach is detailed in the following paper:

> K. Burnham & W. Overton, "Robust Estimation of Population Size When Capture Probabilities Vary Among Animals". *Ecology* (1979), 927-936.

```python
def jackknife(data, conf_lvl=0.95):
    jack_stat = species_richness(data)

    x = np.array(sum([[i] * c for i, c in enumerate(data, 1)], []))
    index = np.arange(x.shape[0])

    vals = []
    for i in range(x.shape[0]):
        t = x[index != i]
        vals.append(species_richness(np.bincount(t)))

    mean_jack_stat = np.mean(vals)
    bias = (x.shape[0] - 1) * (mean_jack_stat - jack_stat)

    estimate = jack_stat - bias

    std_err = np.sqrt(
        (x.shape[0] - 1) *
        np.mean((mean_jack_stat - vals) *
                (mean_jack_stat - vals), axis=0)
    )

    z_score = np.sqrt(2.0) * erfinv(conf_lvl)
    conf_interval = estimate + z_score * np.array((-std_err, std_err))

    return estimate, std_err, conf_interval

results = jackknife(mnl.groupby('text').size())
print('jackknife-estimate (order=1):', results[0], results[-1])
```

    jackknife-estimate (order=1): 117.73170731707278 [106.64468284 128.8187318 ]

This implementation is verbose and uses an explicit `for`-loop, which iteratively leaves out observations and tracks the drops in diversity that follow from this operation. In the code blocks below we show that the same estimate can also be obtained in a fully analytical fashion. First we calculate the frequency counts for each unique text:

```python
num_per_text = mnl.groupby('text').size()
num_per_text
```

    text
    Aiol (1)                 1
    Aiol (2)                 1
    Alexanderroman           1
    Alexanders geesten       5
    Arturs doet              2
                            ..
    Van den bere Wisselau    1
    Walewein ende Keye       1
    Willem van Oringen       1
    Willem van Oringen II    1
    Wrake van Ragisel        3
    Length: 74, dtype: int64

Next, we store the species richness (the number of unique texts) in $t$:

```python
t = species_richness(num_per_text)
t
```

    74

Then we set $s$ to the number of texts that are only attested in a single witness:

```python
s = sum(num_per_text == 1)
s
```

    44

Only the $s$ texts that occur once will affect the species richness during the iterative Jackknife procedure. We can therefore predict that we will obtain the following deviations when applying the bootstrap:

```python
mu = (((t - s) * t) + (s * (t - 1))) / t
mu
```

    73.4054054054054

That means that we can calculate the bias as follows:

```python
bias = (t - 1) * (mu - t)
bias
```

    -43.405405405405546

To account for this bias, we can subtract it from the original species richness in the observed data:

```python
t - bias
```

    117.40540540540555

## Simple example

```python
counts = [5, 4, 3, 3, 1, 1, 1, 1, 1]
names = 'ABCDEFGHI'
data = zip(counts, names)
df = pd.DataFrame(zip(names, counts), columns=('naam', 'mss'))
df.to_excel('output/Tab2.xlsx')
df
```

      naam
      mss

      0
      A
      5

      1
      B
      4

      2
      C
      3

      3
      D
      3

      4
      E
      1

      5
      F
      1

      6
      G
      1

      7
      H
      1

      8
      I
      1

```python
print('total # of witnesses:', df['mss'].sum())
```

    total # of witnesses: 20

```python
species_richness(df['mss'])
```

    9

```python
jackknife(df['mss'])
```

    (13.75, 1.8874586088176875, array([10.0506491, 17.4493509]))

```python
data = np.array(df['mss'])
x = np.array(sum([[i]*c for i, c in enumerate(data, 1)], []))
tradition = [names[i - 1] for i in x]
print(tradition)
```

    ['A', 'A', 'A', 'A', 'A', 'B', 'B', 'B', 'B', 'C', 'C', 'C', 'D', 'D', 'D', 'E', 'F', 'G', 'H', 'I']

```python
bootstrap = []
for i in range(len(tradition)):
    tradition_ = [tradition[j] for j in range(len(tradition)) if i != j]

    bootstrap.append((
        (i + 1), tradition[i], ''.join(tradition_),
        len(set(tradition_)), len(set(tradition_)) - len(set(tradition))))
```

```python
df = pd.DataFrame(bootstrap, columns=('iteration', 'leftout', 'imputed tradition', 'richness', 'error'))
df.to_excel('output/Tab3.xlsx')
df
```

      iteration
      leftout
      imputed tradition
      richness
      error

      0
      1
      A
      AAAABBBBCCCDDDEFGHI
      9
      0

      1
      2
      A
      AAAABBBBCCCDDDEFGHI
      9
      0

      2
      3
      A
      AAAABBBBCCCDDDEFGHI
      9
      0

      3
      4
      A
      AAAABBBBCCCDDDEFGHI
      9
      0

      4
      5
      A
      AAAABBBBCCCDDDEFGHI
      9
      0

      5
      6
      B
      AAAAABBBCCCDDDEFGHI
      9
      0

      6
      7
      B
      AAAAABBBCCCDDDEFGHI
      9
      0

      7
      8
      B
      AAAAABBBCCCDDDEFGHI
      9
      0

      8
      9
      B
      AAAAABBBCCCDDDEFGHI
      9
      0

      9
      10
      C
      AAAAABBBBCCDDDEFGHI
      9
      0

      10
      11
      C
      AAAAABBBBCCDDDEFGHI
      9
      0

      11
      12
      C
      AAAAABBBBCCDDDEFGHI
      9
      0

      12
      13
      D
      AAAAABBBBCCCDDEFGHI
      9
      0

      13
      14
      D
      AAAAABBBBCCCDDEFGHI
      9
      0

      14
      15
      D
      AAAAABBBBCCCDDEFGHI
      9
      0

      15
      16
      E
      AAAAABBBBCCCDDDFGHI
      8
      -1

      16
      17
      F
      AAAAABBBBCCCDDDEGHI
      8
      -1

      17
      18
      G
      AAAAABBBBCCCDDDEFHI
      8
      -1

      18
      19
      H
      AAAAABBBBCCCDDDEFGI
      8
      -1

      19
      20
      I
      AAAAABBBBCCCDDDEFGH
      8
      -1

```python
mean_estimate = np.mean(df['richness'])
print('Average estimate:', mean_estimate)
print('Bias:', mean_estimate - 9)
```

    Average estimate: 8.75
    Bias: -0.25

```python
bias = 19 * (mean_estimate - 9)
bias
```

    -4.75

```python
corrected = 9 - bias
corrected
```

    13.75

```python
conf_lvl = .95

std_err = np.sqrt(
    19 * np.mean((mean_estimate - df['richness']) *
                 (mean_estimate - df['richness']), axis=0))

z_score = np.sqrt(2.0) * erfinv(conf_lvl)
conf_interval = corrected + z_score * np.array((-std_err, std_err))
conf_interval
```

    array([10.0506491, 17.4493509])

## Chao1

In the paper we eventually opt for the more recent, non-parametric formula "Chao1", which is described in this paper:

> A. Chao & L. Jost, ‘Estimating diversity and entropy profiles via discovery rates of new species". *Methods in Ecology and Evolution* (2015), 873-882.

Because we have "doubletons" in our data, we use can the following formula, where:
- $\hat{f_0}$ is the (theoretical) number of non-observed species/texts;
- $f_1$ is the number of species/texts attested exactly once ("singletons");
- $f_2$ is the number of species/texts attested exactly twice ("doubletons");
- $n$ is the total number of individuals/manuscripts in the observed data.

\begin{equation}
\hat{f_0} = \frac{(n - 1)}{n} \frac{f_1^2}{2f_2}
\end{equation}

The code block below returns the full, theoretical species richness as etimated by Chao1, i.e. it adds the estimated $\hat{f_0}$ to the species richness that was observed in the sample:

```python
def chao_richness(x):
    x, n = x[x > 0], x.sum()
    t = x.shape[0]
    f1, f2 = (x == 1).sum(), (x == 2).sum()
    return t + (n - 1) / n * ((f1 ** 2 / 2 / f2) if f2 > 0 else (f1 * (f1 - 1) / 2))
```

If we apply this function to our data, we obtain an even higher (but arguably more realistic) estimate of the loss in textual diversity for this literature. Note, however, that this estimate is still a theoretical *minimum estimate*, since the original loss could still be higher.

```python
chao_richness(num_per_text)
```

    148.00750469043152

Instead of reporting just this number, we apply a bootstrapped procedure in which we sample from the material using a multinomial distribution (see the Appendix Chao and Jost, 2015) and apply Chao1 to the resulting samples. This procedure allows us to calculate a .95 confidence interval for this value.

```python
def bt_prob(x):
    x, n = x[x > 0], x.sum()
    f1, f2 = (x == 1).sum(), (x == 2).sum()
    C = 1 - f1 / n * (((n - 1) * f1 / ((n - 1) * f1 + 2 * f2)) if f2 > 0 else
                      ((n - 1) * (f1 - 1) / ((n - 1) * (f1 - 1) + 2)) if f1 > 0 else
                      0)
    W = (1 - C) / np.sum(x / n * (1 - x / n) ** n)
    p = x / n * (1 - W * (1 - x / n) ** n)
    f0 = np.ceil(((n - 1) / n * f1 ** 2 / (2 * f2)) if f2 > 0 else
                 ((n - 1) / n * f1 * (f1 - 1) / 2))
    p0 = (1 - C) / f0
    p = np.hstack((p, np.array([p0 for i in np.arange(f0)])))
    return p

def bootstrap(x, n_iter=1000, conf=.95):
    # define a multinomial probability distribution
    # for the bootstrap procedure to sample from:
    p, n = bt_prob(x), x.sum()
    data_bt = np.random.multinomial(n, p, n_iter)

    pro = np.array([chao_richness(row) for row in data_bt])

    pro_mean = pro.mean(0)

    lci_pro = -np.quantile(pro, (1 - conf) / 2, axis=0) + pro_mean
    uci_pro = np.quantile(pro, 1 - (1 - conf) / 2, axis=0) - pro_mean

    sd_pro = np.std(pro, axis=0)

    pro = pro_mean - pro
    return (lci_pro, uci_pro, sd_pro, pro)
```

```python
def chao_estimate(x, n_iter=1000, conf=0.95):
    pro = chao_richness(x)
    (lci_pro, uci_pro, sd_pro, bt_pro) = bootstrap(x, n_iter=n_iter, conf=conf)
    lci_pro, uci_pro = pro - lci_pro, pro + uci_pro
    bt_pro = pro - bt_pro
    return (lci_pro, uci_pro, bt_pro, pro)
```

The following block applies this bootstrapped procedure to obtain the final estimates:

```python
lci_pro, uci_pro, bt_pro, pro = chao_estimate(num_per_text, n_iter=10000)
print('pro:', pro)
print('lci_pro:', lci_pro)
print('uci_pro:', uci_pro)
```

    pro: 148.00750469043152
    lci_pro: 106.21863495939421
    uci_pro: 219.01578019221017

The array `bt_pro` contains the estimates that were collected during the bootstrap (1,000 iterations by default). Below, we plot the distribution of these numbers using a rainplot:

```python
import ptitprince as pt

fig, ax = plt.subplots(figsize=(8, 6))

d = list([(x, 'bootstrap') for x in bt_pro])
bt = pd.DataFrame(d, columns=('bootstrap', 'type'))

pt.RainCloud(
    data=bt, x="type", y="bootstrap", ax=ax,
    orient="h", alpha=.8, bw=.2, rain_alpha=.3, palette="Greys"
)
ax.axvline(pro, c='black', ls='--')
ax.axvline(uci_pro, c='darkgrey', ls='--')
ax.axvline(lci_pro, c='darkgrey', ls='--')
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)
ax.spines['left'].set_visible(False)
ax.set_yticks([])
ax.set_ylabel('')
plt.savefig('output/Fig2.png', dpi=300, transparent=True)
```

The idea that there were at least 100 texts is not completely unlikely, but it is a very
conservative estimate, at the very bottom of the probability continuum. The estimate of ~148 manuscripts (or more) is much more plausible, which would mean that *at least half of
the chivalric texts have been lost*.  Just as 100 is an extremely optimistic
estimate, ~219 is the most pessimistic estimate: in that
case, only a third of the ever available chivalric epics would have been persisted through
time, which is quite a dramatic, but not entirely unrealistic figure.

## Species accumulation curve

In what preceded, we have investigated how many unique texts may have been lost, or, more positively, how many unique texts we may have not yet seen. In this concluding section, we investigate how many texts should be retrieved before we arrive at this diversity estimate. This new estimate provides us with information about the total population size, i.e. the total number of text witnesses. We follow Hsieh, Ma and Chao (2016) to compute this estimate using "Rarefaction Extrapolation". For details about this method, see:

> Hsieh, Ma and Chao (2016): iNEXT: an R package for rarefaction and extrapolation ofspecies diversity. *Methods in Ecology and Evolution*, 7, 1451–1456.

```python
def bootstrap_re(x, fn=chao_richness, n_iter=1000, conf=.95):
    # define a multinomial probability distribution
    # for the bootstrap procedure to sample from:
    p, n = bt_prob(x), x.sum()
    data_bt = np.random.multinomial(n, p, n_iter)

    Dq = fn(x)

    pro = np.array([fn(row) for row in data_bt])

    error = stats.norm.ppf(1 - (1 - conf) / 2) * np.std(pro, 0)
    lci_pro = Dq - error
    uci_pro = Dq + error

    sd_pro = np.std(pro, axis=0)

    return (lci_pro, uci_pro, sd_pro, Dq, )

def rarefaction_extrapolation(x, max_steps):
    x, n = x[x > 0], x.sum()
    def _sub(m):
        if m <= n:
            return np.sum(1 - np.array(
                [np.exp(gammaln(n - i + 1) + gammaln(n - m + 1) -
                        gammaln(n - i - m + 1) - gammaln(n + 1)) if i <= (n - m) else
                 0 for i in x]))
        else:
            S = (x > 0).sum()
            f1, f2 = (x == 1).sum(), (x == 2).sum()
            f0 = ((n - 1) / n * f1 * (f1 - 1) / 2) if f2 == 0 else ((n - 1) / n * f1**2 / 2 / f2)
            A = n * f0 / (n * f0 + f1)
            return S if f1 == 0 else (S + f0 * (1 - A**(m - n)))
    return np.array([_sub(mi) for mi in range(1, max_steps)])
```

```python
counts = np.bincount(mnl.groupby('text').size())[1:] # ignore zero
x = np.array(sum([[i] * c for i, c in enumerate(counts, 1)], []))
```

Here too we use a bootstrap method with 100 samples:

```python
max_steps = 1000
lci_pro, uci_pro, sd_pro, Dq = bootstrap_re(
    x,
    fn=partial(rarefaction_extrapolation, max_steps=max_steps),
    n_iter=100
)
```

```python
steps = np.arange(1, max_steps)
interpolated = np.arange(1, max_steps) < x.sum()

fig, ax = plt.subplots(figsize=(8, 6))
ax.plot(steps[interpolated], Dq[interpolated], color='C0')
ax.plot(x.sum(), Dq[x.sum() - 1], 'o')
ax.plot(steps[~interpolated], Dq[~interpolated], '--', color='C0')
ax.fill_between(steps, lci_pro, uci_pro, alpha=0.3)
ax.grid()
ax.set(xlabel='# handschriften', ylabel='# teksten', title='Species Accumulation Curve')
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)

plt.savefig('output/Fig3.png', dpi=300, transparent=True)
```

The graph clearly show that a significant part of the text witnesses has been lost (or has not yet been found). It is only at around 600 witnesses (and more), that the curve slowly flattens towards its asymptote.


This notebook demonstrates the functionality of the `quantecon.models.solow` module. The code was developed with funding from the Scottish Graduate Programme in Economics (SGPE) and the Scottish Institute for Research in Economics (SIRE). The code has found widespread use in  my own research and teaching and it is my fervent hope that others will adapt and contribute to the development of these notebooks for use in their own research and teaching.

```python
%matplotlib inline
```

```python
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import sympy as sym

import quantecon as qe
import solowpy
```

# 0. Motivation

The typical economics graduate student places great faith in the analytical mathematical tools that he or she was taught as an undergraduate. In particular this student is likely under the impression that virtually all economic models have closed-form solutions. At best the typical student believes that if he or she were to encounter an economic model without a closed-form solution, then simplifying assumptions could be made that would render the model analytically tractable without sacrificing important economic content.

The typical economics student is, of course, wrong about general existence of closed-form solutions to economic models. In fact the opposite is true: most economic models, particular dynamic, non-linear models with meaningful constraints (i.e., most any *interesting* model) will fail to have an analytic solution. I the objective of this notebook is to demonstrate this fact and thereby motivate the use of numerical methods in economics more generally using the [Solow model of economic growth](http://faculty.smu.edu/tosang/pdf/Solow_1956.pdf).

Economics graduate students are very familiar with the Solow growth model. For many students, the Solow model will have been one of the first macroeconomic models taught to them as undergraduates. Indeed, Greg Mankiw's [*Macroeconomics*](http://www.macmillanhighered.com/Catalog/product/macroeconomics-eighthedition-mankiw), the dominant macroeconomics textbook for first and second year undergraduates, devotes two full chapters to motivating and deriving the Solow model. The first few chapters of David Romer's [*Advanced Macroeconomics*](http://highered.mheducation.com/sites/0073511374/index.html), one of the most widely used final year undergraduate and first-year graduate macroeconomics textbook, are also devoted to the Solow growth model and its descendants.

## 0.1 The basic Solow growth model
The Solow model can be reduced down to a single non-linear differential equation and associated initial condition describing the time evolution of capital stock (per unit effective labor), $k(t)$.

$$ \dot{k}(t) = sf(k(t)) - (n + g + \delta)k(t),\ k(t) = k_0 \tag {0.1.1} $$

The parameter $0 < s < 1$ is the fraction of output invested and the parameters $n, g, \delta$ are the rates of population growth, technological progress, and depreciation of physical capital. The intensive form of the production function $f$ is assumed to be to be strictly concave with

$$ f(0) = 0,\ lim_{k\rightarrow 0}\ f' = \infty,\ lim_{k\rightarrow \infty}\ f' = 0. \tag{0.1.2} $$

A common choice for the function $f$ which satisfies the above conditions is known as the Cobb-Douglas production function.

$$ f(k(t)) = k(t)^{\alpha} $$

Assuming a Cobb-Douglas functional form for $f$ also makes the model analytically tractable (and thus contributes to the typical economics student's belief that all such models "must" have an analytic solution). [Sato 1963](http://www.jstor.org/stable/2296026) showed that the solution to the model under the assumption of Cobb-Douglas production is

$$ k(t) = \Bigg[\bigg(\frac{s}{n+g+\delta}\bigg)\bigg(1 - e^{-(n+g+\delta)(1-\alpha)t}\bigg)+ k_0^{1-\alpha}e^{-(n+g+\delta)(1-\alpha)t}\Bigg]^{\frac{1}{1-\alpha}}. \tag{0.1.3} $$

A notable property of the Solow model with Cobb-Douglas production is that the model predicts that the shares of real income going to capital and labor should be constant. Denoting capital's share of income as $\alpha_K(k)$, the model predicts that

$$ \alpha_K(k(t)) \equiv \frac{\partial \ln f(k(t))}{\partial \ln k(t)} = \alpha \tag{0.1.4} $$

Note that the prediction is that factor shares are constant along both the balanced growth path *and* during the disequilibrium transient (i.e., the period in which $k(t)$ is varying).  We can test this implication of the model using data from the newest version of the [Penn World Tables (PWT)](http://www.rug.nl/research/ggdc/data/penn-world-table).

```python
import pypwt
pwt = pypwt.load_pwt_data()
```

```python
fig, ax = plt.subplots(1, 1, figsize=(8,6))

for ctry in pwt.major_axis:
    tmp_data = pwt.major_xs(ctry)
    tmp_data.labsh.plot(color='gray', alpha=0.5)

# plot some specific countries
pwt.major_xs('USA').labsh.plot(color='blue', ax=ax, label='USA')
pwt.major_xs('IND').labsh.plot(color='green', ax=ax, label='IND')
pwt.major_xs('CHN').labsh.plot(color='orange', ax=ax, label='CHN')

# plot global average
avg_labor_share = pwt.labsh.mean(axis=0)
avg_labor_share.plot(color='r', ax=ax)

ax.set_title("Labor's share has been far from constant!",
             fontsize=20, family='serif')
ax.set_xlabel('Year', family='serif', fontsize=15)
ax.set_ylabel('Labor share of income', family='serif', fontsize=15)
ax.set_ylim(0, 1)

plt.show()
```

From the above figure it is clear that the prediction of constant factor shares is strongly at odds with the empirical data for most countries. Labor's share of real GDP has been declining, on average, for much of the post-war period. For many countries, such as India, China, and South Korea, the fall in labor's share has been dramatic. Note also that the observed trends in factor shares are inconsistent with an economy being on its long-run balanced growth path.

## 0.2 A more general Solow growth model

While the data clearly reject the Solow model with Cobb-Douglas production, they are *not* inconsistent with the Solow model in general. A simple generalization of the Cobb-Douglas production function, known as the constant elasticity of substitution (CES) function:

$$ f(k(t)) = \bigg[\alpha k(t)^{\rho} + (1-\alpha)\bigg]^{\frac{1}{\rho}} $$

where $-\infty < \rho < 1$ is the elasticity of substitution between capital and effective labor in production is capable of generating the variable factor shares observed in the data. Note that

$$ \lim_{\rho\rightarrow 0} f(k(t)) = k(t)^{\alpha} $$

and thus the CES production function nests the Cobb-Douglas functional form as a special case.

To see that the CES production function also generates variable factor shares note that

$$ \alpha_K(k(t)) \equiv \frac{\partial \ln f(k(t))}{\partial \ln k(t)} = \frac{\alpha k(t)^{\rho}}{\alpha k(t)^{\rho} + (1 - \alpha)} $$

which varies with $k(t)$.

This seemingly simple generalization of the Cobb-Douglas production function, which is necessary in order for the Solow model generate variable factor share, an economically important feature of the post-war growth experience in most countries, renders the Solow model analytically intractable. To make progress solving a Solow growth model with CES production one needs to resort to computational methods.

# 1. Creating an instance of the `solow.Model` class

We begin by creating an instance of the `solow.Model` class in the IPython notebook. As always, it is a good idea to read the docstrings...

```python
solowpy.Model?
```

 From the docsting we see that in order to create an instance of the model we need to specify two primitives: the extensive production function, $F$, and a dictionary of model parameters.

```python
# define model variables
A, K, L = sym.symbols('A, K, L')

# define production parameters
alpha, sigma = sym.symbols('alpha, sigma')

# specify some production function
rho = (sigma - 1) / sigma
ces_output = (alpha * K**rho + (1 - alpha) * (A * L)**rho)**(1 / rho)
```

```python
# define model parameters
ces_params = {'A0': 1.0, 'L0': 1.0, 'g': 0.02, 'n': 0.03, 's': 0.15,
              'delta': 0.05, 'alpha': 0.33, 'sigma': 0.95}

# create an instance of the solow.Model class
ces_model = solowpy.CESModel(params=ces_params)
```

More details on on how to create instances of the `solow.Model` class can be found in the **Getting started** notebook in the [solowPy](https://github.com/davidrpugh/solowPy) repository.

# 2. Finding the steady state

Traditionally, most analysis of the Solow model focuses almost excusively on the long run steady state of the model. Recall that the steady state of the Solow model is the value of capital stock (per unit effective labor) that solves

$$ 0 = sf(k^*) - (g + n + \delta)k^*. \tag{2.0.1} $$

In words: in the long-run, capital stock (per unit effective labor) converges to the value that balances actual investment,
$$sf(k),$$
with effective depreciation,

$$(g + n + \delta)k.$$

Given the assumption made about the aggregate production technology, $F$, and its intensive form, $f$, there is always a unique value $k^* >0$ satisfying equation 2.0.1.

Although it is trivial to derive an analytic expression for the long-run equilibrium of the Solow model for most intensive production functions, the Solow model serves as a good illustrative case for various numerical methods for solving non-linear equations.

The `solowpy.Model.find_steady_state` method provides a simple interface to the various 1D root finding routines available in `scipy.optimize` and uses them to solve the non-linear equation 2.0.1. To see the list of currently supported methods, check out the docstring for the `Model.find_steady_state` method...

```python
solowpy.Model.find_steady_state?
```

```python
k_star, result = ces_model.find_steady_state(1e-6, 1e6, method='bisect', full_output=True)
```

```python
print("The steady-state value is {}".format(k_star))
print("Did the bisection algorithm coverge? {}".format(result.converged))
```

    The steady-state value is 1.82583173149
    Did the bisection algorithm coverge? True

More details on on how to the various methods of the `solow.Model` class for finding the model's steady state can be found in the accompanying **Finding the steady state** notebook in the [solowPy](https://github.com/davidrpugh/solowPy) repository.

# 3. Graphical analysis using Matplotlib and IPython widgets

Graphical analysis is an important pedagogic and research technique for understanding the behavior of the Solow (or really any!) model and as such the `solow.Model` class has a number of useful, built-in plotting methods.

## Static example: the classic Solow diagram

```python
fig, ax = plt.subplots(1, 1, figsize=(8,6))
ces_model.plot_solow_diagram(ax)
fig.show()
```

## Interactive example: the classic Solow diagram

```python
from IPython.html.widgets import fixed, interact, FloatSliderWidget
```

```python
# wrap the static plotting code in a function
def interactive_solow_diagram(model, **params):
    """Interactive widget for the Solow diagram."""
    fig, ax = plt.subplots(1, 1, figsize=(8, 6))
    model.plot_solow_diagram(ax, Nk=1000, **params)

# define some widgets for the various parameters
eps = 1e-2
technology_progress_widget = FloatSliderWidget(min=-0.05, max=0.05, step=eps, value=0.02)
population_growth_widget = FloatSliderWidget(min=-0.05, max=0.05, step=eps, value=0.02)
savings_widget = FloatSliderWidget(min=eps, max=1-eps, step=eps, value=0.5)
output_elasticity_widget = FloatSliderWidget(min=eps, max=1.0, step=eps, value=0.5)
depreciation_widget = FloatSliderWidget(min=eps, max=1-eps, step=eps, value=0.5)
elasticity_substitution_widget = FloatSliderWidget(min=eps, max=10.0, step=0.01, value=1.0+eps)

# create the widget!
interact(interactive_solow_diagram,
         model=fixed(ces_model),
         g=technology_progress_widget,
         n=population_growth_widget,
         s=savings_widget,
         alpha=output_elasticity_widget,
         delta=depreciation_widget,
         sigma=elasticity_substitution_widget,
         )
```

There are number of additional plotting methods available (all of which can be turned into interactive plots using IPython widgets). See the **Graphical analysis** notebook in the [solowPy](https://github.com/davidrpugh/solowPy) repository.

# 4. Solving the Solow model

Solving the Solow model requires efficiently and accurately approximating the solution to a non-linear ordinary differential equation (ODE) with a given initial condition (i.e., an non-linear initial value problem).

## 4.1 Solow model as an initial value problem

The Solow model with can be formulated as an initial value problem (IVP) as follows.

$$ \dot{k}(t) = sf(k(t)) - (g + n + \delta)k(t),\ t\ge t_0,\ k(t_0) = k_0 \tag{4.1.0} $$

The `quantecon` library has its own module `quantecon.ivp` for solving initial value problems of this form using [finite difference methods](http://en.wikipedia.org/wiki/Finite_difference_method). Upon creation of our instance of the `solow.Model` class, an instance of the `quantecon.ivp.IVP` class was created and stored as an attribute of our model...

```python
ces_model.ivp?
```

...meaning that we can solve this initial value problem by applying the `solve` method of the `ivp` attribute!

```python
ces_model.ivp.solve?
```

```python
# need to specify some initial conditions
t0, k0 = 0.0, 0.5
numeric_soln = ces_model.ivp.solve(t0, k0, T=100, integrator='dopri5')
```

We can plot the finite-difference approximation of the solution as follows...

```python
fig, ax = plt.subplots(1, 1, figsize=(8,6))

# plot the finite-difference-approximation
ax.plot(numeric_soln[:,0], numeric_soln[:,1], 'bo', markersize=3.0)

# equilibrium value of capital stock (per unit effective labor)
k_star = ces_model.steady_state
ax.axhline(k_star, linestyle='dashed', color='k', label='$k^*$')

# axes, labels, title, etc
ax.set_xlabel('Time, $t$', fontsize=15, family='serif')
ax.set_ylabel('$k(t)$', rotation='horizontal', fontsize=20, family='serif')
ax.set_title('Finite-difference approximation',
             fontsize=20, family='serif')
ax.legend(loc=0, frameon=False, bbox_to_anchor=(1.0, 1.0))
ax.grid('on')

plt.show()
```

Finite-difference methods only return a discrete approximation to the continuous function $k(t)$. To get a continouous approximation of the solution we can combined finite-difference methods with [B-spline interpolation](http://en.wikipedia.org/wiki/B-spline) using the `interpolate` method of the `ivp` attribute.

```python
ces_model.ivp.interpolate?
```

```python
# interpolate!
ti = np.linspace(0, 100, 1000)
interpolated_soln = ces_model.ivp.interpolate(numeric_soln, ti, k=3)
```

We can graphically compare the discrete, finite-difference approximation with the continuous, B-spline approximation as follows.

```python
fig, ax = plt.subplots(1, 1, figsize=(8,6))

# plot the interpolated and finite difference approximations
ax.plot(ti, interpolated_soln[:,1], 'r-')
ax.plot(numeric_soln[:,0], numeric_soln[:,1], 'bo', markersize=3.0)

# equilibrium value of capital stock (per unit effective labor)
k_star = ces_model.steady_state
ax.axhline(k_star, linestyle='dashed', color='k', label='$k^*$')

# axes, labels, title, etc
ax.set_xlabel('Time, $t$', fontsize=15, family='serif')
ax.set_ylabel('$k(t)$', rotation='horizontal', fontsize=20, family='serif')
ax.set_title('B-spline approximation of the solution',
             fontsize=20, family='serif')
ax.legend(loc=0, frameon=False, bbox_to_anchor=(1.0, 1.0))

plt.show()
```

## Accuracy of our numerical methods

When doing numerical work it is important to understand the accuracy of the methods that you are using to approximate the solution to your model. Typically one assesses the accuracy of a solution method by computing and evaluatin some residual function:

$$ R(k; \theta) = sf(\hat{k}(\theta)) - (g + n + \delta)\hat{k}(\theta) - \dot{\hat{k}}(\theta) $$

where $\hat{k}(\theta)$ is out computed solution to the original differential equation.  We can assess the accuracy of our finite-difference methods by plotting the residual function for our approximate solution using the `compute_residual` method of the `ivp` attribute.

```python
# compute the residual...
ti = np.linspace(0, 100, 1000)
residual = ces_model.ivp.compute_residual(numeric_soln, ti, k=3)
```

We can then plot the residual as follows. Our approximation is accurate so long as the residual is everywhere "close" to zero.

```python
# extract the raw residuals
capital_residual = residual[:, 1]

# typically, normalize residual by the level of the variable
norm_capital_residual = np.abs(capital_residual) / interpolated_soln[:,1]

# create the plot
fig = plt.figure(figsize=(8, 6))
plt.plot(interpolated_soln[:,1], norm_capital_residual, 'b-', label='$k(t)$')
plt.axhline(np.finfo('float').eps, linestyle='dashed', color='k', label='Machine eps')
plt.xlabel('Capital (per unit effective labor), $k$', fontsize=15, family='serif')
plt.ylim(1e-16, 1)
plt.ylabel('Residuals (normalized)', fontsize=15, family='serif')
plt.yscale('log')
plt.title('Residual', fontsize=20, family='serif')
plt.legend(loc=0, frameon=False, bbox_to_anchor=(1.0,1.0))
plt.show()
```

For more details behind the numerical methods used in this section see the the **Solving the Solow model** notebook in the [solowPy](https://github.com/davidrpugh/solowPy) repository.

# 5. Impulse response functions

Impulse response functions (IRFs) are a standard tool for analyzing the short run dynamics of dynamic macroeconomic models, such as the Solow growth model, in response to an exogenous shock. The `solow.impulse_response.ImpulseResponse` class has several attributes and methods for generating and analyzing impulse response functions.

### Example: Impact of a change in the savings rate
One can analyze the impact of a doubling of the savings rate on model variables as follows.

```python
# 50% increase in the current savings rate...
ces_model.irf.impulse = {'s': 1.5 * ces_model.params['s']}

# in efficiency units...
ces_model.irf.kind = 'efficiency_units'
```

```python
fig, ax = plt.subplots(1, 1, figsize=(8,6))
ces_model.irf.plot_impulse_response(ax, variable='output')
plt.show()
```

### Example: Interactive impulse reponse functions
Using IPython widgets makes it extremely easy to analyze the various impulse response functions.

```python
def interactive_impulse_response(model, shock, param, variable, kind, log_scale):
    """Interactive impulse response plotting tool."""
    # specify the impulse response
    model.irf.impulse = {param: shock * model.params[param]}
    model.irf.kind = kind

    # create the plot
    fig, ax = plt.subplots(1, 1, figsize=(8,6))
    model.irf.plot_impulse_response(ax, variable=variable, log=log_scale)

irf_widget = interact(interactive_impulse_response,
                      model=fixed(ces_model),
                      shock = FloatSliderWidget(min=0.1, max=5.0, step=0.1, value=0.5),
                      param = ['g', 'n', 's', 'alpha', 'delta' , 'sigma'],
                      variable=['capital', 'output', 'consumption', 'investment'],
                      kind=['efficiency_units', 'per_capita', 'levels'],
                      log_scale=False,
                      )
```

For more details and examples see the accompanying **Impulse response function** notebook in the [solowPy](https://github.com/davidrpugh/solowPy) repository.

# 6. The Solow model, declining labor's share, and secular stagnation

Recently there has been much discussion about the reasons for the [Elsby (2013)](http://www.brookings.edu/~/media/Projects/BPEA/Fall%202013/2013b_elsby_labor_share.pdf) has a nice paper that looks at the decline of labor share in the U.S.; as well as much debat about whether or not developed economes are experiencing some sort of [secular stagnation](http://www.economist.com/blogs/buttonwood/2014/11/secular-stagnation) more generally.

```python
def awesome_interactive_plot(model, iso3_code, **params):
    """Interactive widget for the my awesome plot."""

    # extract the relevant data
    tmp_data = pwt.major_xs(iso3_code)
    actual_labor_share = tmp_data.labsh.values
    actual_capital_share = 1 - tmp_data.labsh

    output = tmp_data.rgdpna
    capital = tmp_data.rkna
    labor = tmp_data.emp

    # need to update params
    model.params.update(params)

    # get new initial condition
    implied_technology = model.evaluate_solow_residual(output, capital, labor)
    k0 = tmp_data.rkna[0] / (implied_technology[0] * labor[0])

    # finite difference approximation
    T = actual_labor_share.size
    soln = model.ivp.solve(t0, k0, T=T, integrator='dopri5')

    # get predicted labor share
    predicted_capital_share = model.evaluate_output_elasticity(soln[:,1])
    predicted_labor_share = 1 - predicted_capital_share

    # get predicted output per unit labor
    predicted_intensive_output = model.evaluate_intensive_output(soln[:,1])
    technology = implied_technology[0] * np.exp(ces_model.params['g'] * soln[:,0])
    predicted_output_per_unit_labor = predicted_intensive_output * technology

    # make the plots!
    fig, axes = plt.subplots(1, 2, figsize=(12,6))
    axes[0].plot(soln[:,0], predicted_labor_share, 'b')
    axes[0].plot(soln[:,0], predicted_capital_share, 'g')
    axes[0].plot(actual_labor_share)
    axes[0].plot(actual_capital_share)
    axes[0].set_xlabel('Time, $t$', fontsize=15, family='serif')
    axes[0].set_ylim(0, 1)
    axes[0].set_title('Labor share of income in {}'.format(iso3_code),
                      fontsize=20, family='serif')
    axes[0].legend(loc=0, frameon=False, bbox_to_anchor=(1.0, 1.0))

    axes[1].set_xlabel('Time, $t$', fontsize=15, family='serif')
    axes[1].set_title('Growth rate of Y/L in {}'.format(iso3_code),
                      fontsize=20, family='serif')
    axes[1].legend(loc=0, frameon=False, bbox_to_anchor=(1.0, 1.0))
    axes[1].plot(soln[1:,0], np.diff(np.log(predicted_output_per_unit_labor)),
                 'b', markersize=3.0)
    axes[1].plot(np.log(output / labor).diff().values)

# define some widgets for the various parameters
technology_progress_widget = FloatSliderWidget(min=-0.05, max=0.05, step=5e-3, value=0.01)
population_growth_widget = FloatSliderWidget(min=-0.05, max=0.05, step=5e-3, value=0.01)
savings_widget = FloatSliderWidget(min=eps, max=1-eps, step=5e-3, value=0.2)
output_elasticity_widget = FloatSliderWidget(min=eps, max=1.0, step=5e-3, value=0.15)
depreciation_widget = FloatSliderWidget(min=eps, max=1-eps, step=5e-3, value=0.02)
elasticity_substitution_widget = FloatSliderWidget(min=eps, max=10.0, step=1e-2, value=2.0+eps)

# create the widget!
interact(awesome_interactive_plot,
         model=fixed(ces_model),
         iso3_code='USA',
         g=technology_progress_widget,
         n=population_growth_widget,
         s=savings_widget,
         alpha=output_elasticity_widget,
         delta=depreciation_widget,
         sigma=elasticity_substitution_widget,
         )
```

```python

```


# solve_ivp gives "wobbly" results?

## Purpose
* solve_ivp with RK45 gives some wobbly results can this be improved?

## Methodology
* Run solve_ivp with various settings.
* compare the accelerations.

## Results
Describe and comment the most important results.

## Setup

```python
# %load imports.py
## Local packages:

%matplotlib inline
%load_ext autoreload
%autoreload 2
%config Completer.use_jedi = False  ## (To fix autocomplete)

## External packages:
import pandas as pd
pd.options.display.max_rows = 999
pd.options.display.max_columns = 999
pd.set_option("display.max_columns", None)

import numpy as np
import os
import matplotlib.pyplot as plt
#if os.name == 'nt':
#    plt.style.use('presentation.mplstyle')  # Windows

import plotly.express as px
import plotly.graph_objects as go

import seaborn as sns
import sympy as sp
from sympy.physics.mechanics import (dynamicsymbols, ReferenceFrame,
                                      Particle, Point)
from sympy.physics.vector.printing import vpprint, vlatex
from IPython.display import display, Math, Latex
from src.substitute_dynamic_symbols import run, lambdify

import pyro

import sklearn
import pykalman
from statsmodels.sandbox.regression.predstd import wls_prediction_std
import statsmodels.api as sm

from scipy.integrate import solve_ivp

## Local packages:
from src.data import mdl

from src.symbols import *
from src.parameters import *
import src.symbols as symbols
from src import prime_system
from src.models import regression
from src.visualization.regression import show_pred
from src.visualization.plot import track_plot

## Load models:
# (Uncomment these for faster loading):
import src.models.vmm_simple_nonlinear  as vmm
from src.data.case_1 import ship_parameters, df_parameters, ps, ship_parameters_prime
from src.data.transform import transform_to_ship
```

## Ship parameters

```python
ship_parameters
```

## Brix parameters

```python
mask = df_parameters['prime'].notnull()
index = df_parameters.loc[mask,'prime'].index
coefficients=vmm.simulator.get_all_coefficients(sympy_symbols=False)
missing_coefficients = set(coefficients) - set(index)
missing_coefficients
```

```python
mask = df_parameters['prime'].notnull()
df_parameters.loc[mask,'prime']
```

## Simulate data

```python
parameters=df_parameters['prime'].copy()

t_ = np.linspace(0,70,1000)
df_ = pd.DataFrame(index=t_)

df_['u'] = 2
df_['v'] = 0
df_['r'] = 0
df_['x0'] = 0
df_['y0'] = 0
df_['psi'] = 0
df_['U'] = np.sqrt(df_['u']**2 + df_['v']**2)
df_['beta'] = -np.arctan2(df_['v'],df_['u'])
df_['thrust'] = 50

df_['delta'] = 0
df_.loc[10:,'delta'] = np.deg2rad(20)

results = {}
for method in ['RK45','Radau','BDF','RK23','DOP853','LSODA']:

    result = vmm.simulator.simulate(df_=df_, parameters=parameters, ship_parameters=ship_parameters,
                                control_keys=['delta','thrust'], primed_parameters=True,
                                prime_system=ps, method=method)
    results[method] = result

```

```python
results=pd.Series(results)
```

## Compare

```python
fig,ax=plt.subplots()
for method,result in results.items():
    result.result.plot(y='u1d', label=method, ax=ax);

ax.set_ylim(results['RK45'].result['u1d'].min(), results['RK45'].result['u1d'].max())
```

```python
fig,ax=plt.subplots()
for method,result in results.loc[['Radau','BDF','LSODA']].items():
    result.result.plot(y='u1d', label=method, ax=ax);

ax.set_ylim(results['RK45'].result['u1d'].min(), results['RK45'].result['u1d'].max())
```

```python
x,y,z = sp.symbols('x y z')
M = sp.Matrix([sp.sin(x) + y, sp.cos(y) + x, z])
M
```

```python
M.jacobian([x, y, z])
```

```python
eq_acceleration = vmm.simulator.acceleartion_eq.subs([(X_qs,vmm.X_qs_eq.rhs),
                                    (Y_qs,vmm.Y_qs_eq.rhs),
                                    (N_qs,vmm.N_qs_eq.rhs),
                                   ])
subs = {value:key for key,value in p.items()}
eq_acceleration = eq_acceleration.subs(subs)

jac = eq_acceleration.jacobian([u,v,r])
jac_lambda=lambdify(jac)
```

```python
jac_lambda
```

```python

```


```python
clean_up = True
%run StdPackages.ipynb
d['gams'] = os.path.join(d['CGE'],'gams')
```

## Test A: Production version 2
Investigate how nested CES and normalized CES sectors work. This goes through identical steps for:
* CES1: A simple CES sector.
* CES2: A nested CES sector (2 nests).
* CES1_norm: Normalized CES sector.
* CES2_norm: Nested and normalized CES sector.
* CET1: A simple CET sector.
* CET2: A nested CET sector.
* CET1_norm: A normalized CET sector.
* CET2_norm: A nested, normalized CET sector.
* CES_CET: A nested CES, CET sector (CES to a intermediate good, that is split into two outputs CET).
* CES_CET_norm: A nested CES, CET normalized sector (same as CES_CET, but CET sector has normalized technology).
* CESCET_norm: A nested, normalized CES,CET sector (same as CES_CET, but both sectors have normalized tech).
* FunkyTree: Mixes normalized input and output trees: Z1 is an input that is split into an output Y1 and an intermediate good X1. Similarly, Z2 is split into output Y2 and intermediate good X2. Finally, X1 and X2 are combined to a single output Y3.
* ExtremeFunk: Mixes normalized input and output trees to highlight the role of choosing the right $\mu$s to keep exogenous when calibrating.

## Initialize modules:

*Global settings:*

```python
glob = CGE_globals.SmallOpen(kwargs_vals = {'t': range(1,3)})
```

*Init nesting structures:*

```python
FunnyName_NS = {n+'_input': n for n in ('Y1','Y2')}
data_str = os.path.join(d['data'], 'Nestings.xlsx')
read_trees = {'CES1': {'CES1': {'f': 'CES'}},
              'CES2': {'CES2': {'f':'CES'}},
              'CES1_norm': {'CES1': {'f':'CES_norm'}},
              'CES2_norm': {'CES2': {'f':'CES_norm'}},
              'CET1': {'CET1': {'f': 'CET'}},
              'CET2': {'CET2': {'f': 'CET'}},
              'CET1_norm': {'CET1': {'f': 'CET_norm'}},
              'CET2_norm': {'CET2': {'f': 'CET_norm'}},
              'CES_CET': {'CES1': {'f': 'CES'}, 'CET1': {'f': 'CET'}},
              'CES_CET_norm': {'CES1': {'f':'CES'}, 'CET1': {'f':'CET_norm'}},
              'CESCET_norm': {'CES1': {'f':'CES_norm'}, 'CET1': {'f': 'CET_norm'}},
              'FunkyTree': {'FunkyTree_CET': {'f':'CET_norm'}, 'FunkyTree_CES': {'f':'CES_norm'}},
              'ExtremeFunk': {'ExtremeF1': {'f': 'CES_norm'}, 'ExtremeF2': {'f':'CET_norm'}},
              'FunnyName': {'FunnyNameInp': {'f':'CES_norm'}, 'FunnyNameOut': {'f':'CET_norm'}}}
Trees = {k: NestingTree.AggTree_from_data(data_str, read_trees = read_trees[k], name = k)(namespace=FunnyName_NS) for k in read_trees} # init trees
```

*Init:*

```python
ws = gams.GamsWorkspace(working_directory=d['work'])
Ps = {k: CGE_Production.Production(tree = Trees[k], glob = glob, ns = {}, s_kwargs = {'ws': ws}) for k in Trees}
```

#### Calibration subsets

The default calibration method relies on endogenizing all $\mu$ parameters and in turn exogenize all $qD$ variables in the nesting tree. This method has to be adjusted in a way that depends on (1) whether or not the model features multiple outputs per sector and (2) if there are nests with scale-preserving technologies.

*NB: We ignore the time index for now.*

##### Without scale-preserving technologies

In the simple case, all $\mu$ are endogenized and all $qD$ are exogenized at some level specified by input-output data. Note that even if we know the cost price on ouptuts, $pS[s,n]$, we would leave this variable endogenous. The reason is that for a sector with constant returns to scale (CRS) technology, we have that:

$$\begin{align}
    \sum_{n\in inputs} qD[s,n]pD[s,n] = \sum_{n\in outputs} qS[s,n] pS[s,n].
\end{align}$$

All variables on the left-hand-side (LHS) is exogenous when calibration, and so is the supplied quantities $qS[s,n]$. Thus, for the system of equations to be square, we need to leave $pS[s,n]$ endogenous (alternatively, we could remove one of the price index equations from the system when we calibrate). If input-output data is consistent, note that the solution will automatically be the in data.

Note that we still allow for multiple outputs to be produced. Usually, we would distinguish between cost-prices $pS[s,n]$ and equilibrium prices $p[n]$ and assume a price equation in the style of

$$\begin{align}
    p[n] = (1+m[s])(pS[s,n]+\tau[s,n])+\Gamma[s,n]),
\end{align}$$

where $m[s]$ is a sector-specific mark-up, $\tau[s,n]$ is a unit tax, and $\Gamma[s,n]$ is a function capturing e.g. adjustment costs of investments. Note that this equation holds for all (outputs, sector)-combinations. If we observe $p[n],\tau[s,n]$ and can compute $\Gamma[s,n]$ from the model, this identifies $m[s]$ and all but one $pS[s,n]$; the last one ($pS$) is endogenous.

##### Scale-preserving technologies

When we have scale-preserving technologies, we have to think more carefully about what variables are endogenized/exogenized when calibrating. Consider for instance the simple case with $Y$ being produced by $X_1,X_2$ using some scale-preserving technology. The system of equations read (ignoring taxes+markups etc).:

$$\begin{align}
    p_Y Y &= p_1X_1+p_2X_2 \\
    X_1 &= \dfrac{F_1(p_1,p_2,p_Y; \mu_1)}{F_1(p_1,p_2,p_Y; \mu_1)+F_2(p_1,p_2,p_Y; \mu_2)}Y \\
    X_2 &= \dfrac{F_2(p_1,p_2,p_Y; \mu_2)}{F_1(p_1,p_2,p_Y; \mu_1)+F_2(p_1,p_2,p_Y; \mu_2)}Y,
\end{align}$$

where $F_1,F_2$ are some price functions with share parameters $\mu_1,\mu_2$.  In baseline mode, this system is square in $p_Y,X_1,X_2$ taking $Y,p_1,p_2,\mu_1,\mu_2$ as given. In calibration mode we would usually exogenize $X_1,X_2$ and endogenize $\mu_1,\mu_2$. Given exogenous variables, we can always choose a set of $\mu_1,\mu_2$ to induce values $F_1,F_2$, thus, we are essentially solving for $F_1,F_2,p_Y$. Note that solving for $F_1,F_2$ in demand functions yield

$$\begin{align}
    F_1\left(1-\dfrac{X_1}{Y}\right) &= \dfrac{X_1}{Y}F_2 \\
    F_2\left(1-\dfrac{X_2}{Y}\right) &= \dfrac{X_2}{Y}F_1
\end{align}$$

Using that $1-X_1/Y = X_2/Y$ and $1-X_2/Y=X_1/Y$ note that the two conditions are linearly dependent, i.e. that they identify the same restriction:

$$\begin{align}
    F_1 &= \dfrac{X_1}{X_2} F_2 \\
    F_2 &= \dfrac{X_2}{X_1} F_1.
\end{align}$$

Naturally, with a scale-preserving function, a nest with $N$ share parameters only identify $N-1$ variables; the final one is residually determined by $X_N = Y-\sum_{i=1}^{N-1}X_i$. This causality means that we need to keep one of the $\mu$ fixed in calibration mode in scale-preserving nests, and keep the corresponding quantity endogenous.

##### Identifying the right $\mu$s to keep exogenous/quantities to keep endogenous

As it turns out, simply choosing one random element in $\mu$ for all scale-preserving nests to keep exogenous (and endogenize corresponding quantity), does not always work. In particular, this can be a problem if we have nodes in the nesting tree that are simultaneously a branch in a scale-preserving input tree and another scale-preserving output tree. The nesting tree 'FunkyTree' and 'ExtremeFunk' are examples of this. In this case, we could potentially, randomly, pick the element $qD[s,n]$ to be the endogenous element for two trees, thus exogenizing two $\mu$ elements, but only endogenizing one quantity. Thus, we go through a couple of steps to make sure that does not happen

##### Identifying calibration subsets, the general case:

The ```Production``` class includes an algorithm that ensures that the right $\mu$s are kept exogenous in calibration mode, the right quantities $(qD/qS)$ are kept/made endogenous. Importantly, note that the subset ```exomu``` identifies the same number of elements as ```endo_qS``` and ```endo_qD``` combined.

```python
n = 'ExtremeFunk'
p = Ps[n]
t = Trees[n]
```

*Four elements are kept exogenous:*

```python
p.get('exomu')
```

    MultiIndex([('s1', 'X1', 'Z2'),
                ('s1', 'Y1', 'Z1'),
                ('s1', 'Y2', 'X3'),
                ('s1', 'Y3', 'Z3')],
               names=['s', 'n', 'nn'])

*One supply element and three demand elements are endogenized:*

```python
p.get('endo_qS')
```

    MultiIndex([('s1', 'Y3')],
               names=['s', 'n'])

```python
p.get('endo_qD')
```

    MultiIndex([('s1', 'X1'),
                ('s1', 'X3'),
                ('s1', 'Z1')],
               names=['s', 'n'])

#### Test baseline mode

*Init states:*

```python
[P.compile(initDB=True) for P in Ps.values()];
```

*Write text:*

```python
[P.write() for P in Ps.values()];
```

*Run models (with same workspace):*

```python
Ms = {k: Ps[k].run(exportTo=d['work'], ws=ws,**{'cns': 'CONOPT4'}) for k in Ps}
```

#### Test calibration mode:

*Change state:*

```python
[setattr(p.s,'state','C') for p in Ps.values()];
```

*Update database to baseline solution:*

```python
[setattr(Ps[k].s,'db',Ms[k].out_db) for k in Ps];
```

*Write:*

```python
[P.write() for P in Ps.values()];
```

*Re-run:*

```python
Ms = {k: Ps[k].run(exportTo=d['work'], ws=ws,**{'cns': 'CONOPT4'}) for k in Ps}
```