# The VDDG model forced by the Jin-Timmermann ENSO model : computation of the first Lyapunov exponent

## Preamble

This notebook comes as a supplementary material to the article

* Vannitsem, S., Demaeyer, J., and Ghil, M. (2021). Extratropical low‐frequency variability with ENSO forcing: A reduced‐order coupled model study. *Journal of Advances in Modeling Earth Systems*, e2021MS002530. [https://doi.org/10.1029/2021MS002530](https://doi.org/10.1029/2021MS002530).

and details the code of the model used to perform the computations needed to obtain the results therein.

It consists of the [qgs model](https://github.com/Climdyn/qgs) in the MAOOAM-VDDG configuration to model the extra-tropics, and forced by the Jin-Timmerman model which models ENSO. More information on the components can be found in the articles:

* Vannitsem, S., Demaeyer, J., De Cruz, L., & Ghil, M. (2015). Low-frequency variability and heat transport in a low-order nonlinear coupled ocean–atmosphere model. *Physica D: Nonlinear Phenomena*, **309**, 71-85. [doi:10.1016/j.physd.2015.07.006](https://doi.org/10.1016/j.physd.2015.07.006)
* De Cruz, L., Demaeyer, J. and Vannitsem, S. (2016). The Modular Arbitrary-Order Ocean-Atmosphere Model: MAOOAM v1.0, *Geosci. Model Dev.*, **9**, 2793-2808. [doi:10.5194/gmd-9-2793-2016](https://doi.org/10.5194/gmd-9-2793-2016)
* Demaeyer J., De Cruz, L. and Vannitsem, S. , (2020). qgs: A flexible Python framework of reduced-order multiscale climate models. *Journal of Open Source Software*, **5**(56), 2597, [doi:10.21105/joss.02597](https://doi.org/10.21105/joss.02597).
* Timmermann, A., Jin, F. F., & Abshagen, J. (2003). A nonlinear theory for El Niño bursting. *Journal of the atmospheric sciences*, **60**(1), 152-165, [doi:10.1175/1520-0469(2003)060%3C0152:ANTFEN%3E2.0.CO;2](https://doi.org/10.1175/1520-0469(2003)060%3C0152:ANTFEN%3E2.0.CO;2).
* Roberts, A., Guckenheimer, J., Widiasih, E., Timmermann, A., & Jones, C. K. (2016). Mixed-mode oscillations of El Nino–southern oscillation. *Journal of Atmospheric Sciences*, **73**(4), 1755-1766, [doi:10.1175/JAS-D-15-0191.1](https://doi.org/10.1175/JAS-D-15-0191.1).

In this notebook, we define the forced model and integrate it to get several trajectories.

## Requirements

The present notebook was tested on a computer with

    12 recent cpu cores,
    16 Gb of RAM,

## Installation

To install this notebook, you must first [install the qgs model](https://qgs.readthedocs.io/en/latest/files/general_information.html#installation) and test that it works properly on your machine. Follow the instructions in the model documentation. Then you must move the present notebook in the notebooks directory and run it with jupyter-notebook. In the qgs model main folder, do:

    conda activate qgs
    cd notebooks
    jupyter-notebook

It will open a page in a web browser where you can open and then run the present notebook.

We now proceed to the code itself:

## Modules import

First, setting of the path and loading of some modules

```python
import sys, os
```

```python
sys.path.extend([os.path.abspath('../')])
```

```python
import numpy as np
from numba import njit
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

Initializing the random number generator (for reproducibility). -- Disable if needed.

```python
np.random.seed(210217)
```

Importing the model's modules

```python
from qgs.params.params import QgParams
from qgs.integrators.integrator import RungeKuttaIntegrator
from qgs.integrators.integrate import integrate_runge_kutta_tgls
from qgs.functions.tendencies import create_tendencies
```

## Systems definition

General parameters

```python
# Time step parameters
dt = 0.1
# Saving the model state n steps
write_steps = 100

number_of_trajectories = 2
```

### VDDG (using qgs)

Setting some model parameters

```python
# Model parameters instantiation
model_parameters = QgParams()

## VDDG model selection:
# Mode truncation at the wavenumber 2 in both x and y spatial
# coordinates for the atmosphere
model_parameters.set_atmospheric_channel_fourier_modes(2, 2)
# Mode truncation at the wavenumber 2 in the x and at the
# wavenumber 4 in the y spatial coordinates for the ocean
model_parameters.set_oceanic_basin_fourier_modes(2, 4)
```

```python
# Setting MAOOAM parameters according to the publication
model_parameters.set_params({'kd': 0.0290, 'kdp': 0.0290, 'n': 1.5, 'r': 1.e-7,
                             'h': 100.0, 'd': 1.5e-7, 'time_unit': 'years'})
model_parameters.atemperature_params.set_params({'hlambda': 15.06})
model_parameters.gotemperature_params.set_params({'gamma': 4.0e8})
```

Setting the short-wave radiation component $C_{\text{a},1}$ and $C_{\text{o},1}$ as in the publication:

```python
model_parameters.atemperature_params.set_insolation(310./4, 0)
model_parameters.gotemperature_params.set_insolation(310, 0)
```

Printing the VDDG model's parameters

```python
model_parameters.print_params()
```

    Qgs v0.2.5 parameters summary
    =============================

    General Parameters:
    'time_unit': years,
    'rr': 287.058  [J][kg^-1][K^-1]  (gas constant of dry air),
    'sb': 5.67e-08  [J][m^-2][s^-1][K^-4]  (Stefan-Boltzmann constant),

    Scale Parameters:
    'scale': 5000000.0  [m]  (characteristic space scale (L*pi)),
    'f0': 0.0001032  [s^-1]  (Coriolis parameter at the middle of the domain),
    'n': 1.5    (aspect ratio (n = 2 L_y / L_x)),
    'rra': 6370000.0  [m]  (earth radius),
    'phi0_npi': 0.25    (latitude expressed in fraction of pi),
    'deltap': 50000.0  [Pa]  (pressure difference between the two atmospheric layers),

    Atmospheric Parameters:
    'kd': 0.029  [nondim]  (atmosphere bottom friction coefficient),
    'kdp': 0.029  [nondim]  (atmosphere internal friction coefficient),
    'sigma': 0.2  [nondim]  (static stability of the atmosphere),

    Atmospheric Temperature Parameters:
    'gamma': 10000000.0  [J][m^-2][K^-1]  (specific heat capacity of the atmosphere),
    'C[1]': 77.5  [W][m^-2]  (spectral component 1 of the short-wave radiation of the atmosphere),
    'C[2]': 0.0  [W][m^-2]  (spectral component 2 of the short-wave radiation of the atmosphere),
    'C[3]': 0.0  [W][m^-2]  (spectral component 3 of the short-wave radiation of the atmosphere),
    'C[4]': 0.0  [W][m^-2]  (spectral component 4 of the short-wave radiation of the atmosphere),
    'C[5]': 0.0  [W][m^-2]  (spectral component 5 of the short-wave radiation of the atmosphere),
    'C[6]': 0.0  [W][m^-2]  (spectral component 6 of the short-wave radiation of the atmosphere),
    'C[7]': 0.0  [W][m^-2]  (spectral component 7 of the short-wave radiation of the atmosphere),
    'C[8]': 0.0  [W][m^-2]  (spectral component 8 of the short-wave radiation of the atmosphere),
    'C[9]': 0.0  [W][m^-2]  (spectral component 9 of the short-wave radiation of the atmosphere),
    'C[10]': 0.0  [W][m^-2]  (spectral component 10 of the short-wave radiation of the atmosphere),
    'eps': 0.76    (emissivity coefficient for the grey-body atmosphere),
    'T0': 270.0  [K]  (stationary solution for the 0-th order atmospheric temperature),
    'sc': 1.0    (ratio of surface to atmosphere temperature),
    'hlambda': 15.06  [W][m^-2][K^-1]  (sensible+turbulent heat exchange between ocean and atmosphere),

    Oceanic Parameters:
    'gp': 0.031  [m][s^-2]  (reduced gravity),
    'r': 1e-07  [s^-1]  (frictional coefficient at the bottom of the ocean),
    'h': 100.0  [m]  (depth of the water layer of the ocean),
    'd': 1.5e-07  [s^-1]  (strength of the ocean-atmosphere mechanical coupling),

    Oceanic Temperature Parameters:
    'gamma': 400000000.0  [J][m^-2][K^-1]  (specific heat capacity of the ocean),
    'C[1]': 310.0  [W][m^-2]  (spectral component 1 of the short-wave radiation of the ocean),
    'C[2]': 0.0  [W][m^-2]  (spectral component 2 of the short-wave radiation of the ocean),
    'C[3]': 0.0  [W][m^-2]  (spectral component 3 of the short-wave radiation of the ocean),
    'C[4]': 0.0  [W][m^-2]  (spectral component 4 of the short-wave radiation of the ocean),
    'C[5]': 0.0  [W][m^-2]  (spectral component 5 of the short-wave radiation of the ocean),
    'C[6]': 0.0  [W][m^-2]  (spectral component 6 of the short-wave radiation of the ocean),
    'C[7]': 0.0  [W][m^-2]  (spectral component 7 of the short-wave radiation of the ocean),
    'C[8]': 0.0  [W][m^-2]  (spectral component 8 of the short-wave radiation of the ocean),
    'C[9]': 0.0  [W][m^-2]  (spectral component 9 of the short-wave radiation of the ocean),
    'C[10]': 0.0  [W][m^-2]  (spectral component 10 of the short-wave radiation of the ocean),
    'T0': 285.0  [K]  (stationary solution for the 0-th order oceanic temperature),

Creating the VDDG (qgs) tendencies function

```python
%%time
f_vddg, Df_vddg = create_tendencies(model_parameters)
```

    CPU times: user 5.27 s, sys: 523 ms, total: 5.79 s
    Wall time: 5.15 s

### Jin-Timmermann ENSO model

The nondimensional equations governing the ENSO module are given by:
$
\begin{align}
\frac{dx}{dt} = & \, \rho \delta (x^2-a x) + s x (x+y+c-c \tanh(x+z)), \\
\frac{dy}{dt} = & \, -\rho \delta (a y+x^2), \\
\frac{dz}{dt} = & \, \delta (k-z-\frac{x}{2}).
\end{align}
$

Here $x, y, z$ and $t$ are dimensionless, $x$ is the temperature difference between the eastern and western basins of the Tropical Pacific, $y$ the western basin's temperature anomaly with respect to a reference value, and $z$ the western basin's thermocline depth anomaly.

Setting first some model parameters

```python
# Periodic solutions
#delta = 0.00028058
#rho = 0.3224
#c = 2.3952
#k = 0.4032
#a = 6.8927
#s = 0.0010691

# Chaotic solutions
delta = 0.0002803
rho = 0.29016
c = 2.3952
k = 0.4032
a = 7.658609809
s = 0.001069075
```

Creating the tendencies function using a [Numba](https://numba.pydata.org/) jitted function

```python
@njit
def f_JT(t, x):
    dxdt = rho * delta * (x[0]**2 - a * x[0]) + s * x[0] * (x[0] + x[1] + c - c * np.tanh(x[2] + x[0]))
    dydt = -rho * delta * (a * x[1] + x[0]**2)
    dzdt = delta * (k - x[2] - x[0] / 2)

    return np.array([dxdt, dydt, dzdt])

```

#### Integrating the ENSO model to get averages

Defining an integrator

```python
integrator = RungeKuttaIntegrator()
integrator.set_func(f_JT)
```

First initial conditions for the ENSO model

```python
ic_enso = np.array([-2.8439, -0.62, 1.48])
```

```python
%%time
integrator.integrate(0., 10000000., dt, ic=ic_enso, write_steps=write_steps)
time, traj = integrator.get_trajectories()
# new initial condition
ic_enso = traj[:,-1]
# average
mean_enso = np.mean(traj, axis=-1)
```

    CPU times: user 206 ms, sys: 717 ms, total: 923 ms
    Wall time: 2min 10s

### Forced VDDG-ENSO model

The forced VDDG-ENSO model is obtained by modifying the tendency $f_1$ of the first component $\psi_{{\rm a}, 1}$ of the atmospheric streamfunction of VDDG:

$$
\frac{\mathrm{d}\psi_{{\rm a},1}}{\mathrm{d}t} = f_1(\psi_{{\rm a}, 1}, \theta_{{\rm a}, 1}) + g\, (x+y)'.
$$

where the parameter $g$ scales the intensity of the tropical forcing. The variable $(x + y)'$ is the Tropical eastern basin temperature anomaly of the ENSO model.

To simplify the integration, the full system of equations is constructed by concatenating together the equations of motion of both models, to give:
$$
\frac{\mathrm{d}\boldsymbol{X}}{\mathrm{d}t} = \boldsymbol{F}(\boldsymbol{X})
$$

Hence, we first set the coupling parameters:

```python
g = 0.03
enso_temperature_scaling = 2.8182
ndim = model_parameters.ndim
vddg_temperature_scaling = (model_parameters.scale_params.L * model_parameters.scale_params.f0)**2 / model_parameters.rr
enso_to_vddg_temperature_scaling = enso_temperature_scaling / vddg_temperature_scaling
mean_east_basin_enso = mean_enso[0] + mean_enso[1]
```

and then create the tendencies function of the full system using a [Numba](https://numba.pydata.org/) jitted function:

```python
@njit
def f_forced(t, x):
    tend_vddg = f_vddg(t, x[:ndim])
    tend_JT = f_JT(t, x[ndim:])
    tend_vddg[0] = tend_vddg[0] + g * enso_to_vddg_temperature_scaling * ((x[ndim] + x[ndim+1]) - mean_east_basin_enso)
    return np.concatenate((tend_vddg, tend_JT))
```

**The function** `f_forced` **hence defined is the tendencies** $\boldsymbol{F}(\boldsymbol{X})$ **of the ENSO forced VDDG model used in the publication above.**

The corresponding Jacobian matrix of the VDDG model alone is also defined, to compute the corresponding first Lyapunov exponent.

```python
@njit
def Df_forced(t, x):
    J_vddg = Df_vddg(t, x[:ndim])
    J_forced = np.zeros((ndim+3, ndim+3))
    J_forced[:ndim, :ndim] = J_vddg
    return J_forced

```

## Time integration

#### Integrating the forced model

Start random initial conditions for the VDDG model and integrate over a transient time to obtain initial conditions on the attractor(s)

```python
%%time
## Reuse the integrator
integrator.set_func(f_forced)

## Might take several minutes, depending on your cpu computational power.
ic = np.random.rand(number_of_trajectories, model_parameters.ndim+3)*0.1
ic[:, ndim:] = ic_enso + np.zeros((number_of_trajectories, 3))
# Integration loop (to integrate a very long time)
for i in range(10):
    integrator.integrate(0., 1000000., dt, ic=ic, write_steps=0)
    time, ic = integrator.get_trajectories()
```

    CPU times: user 397 ms, sys: 1.38 s, total: 1.77 s
    Wall time: 8min 54s

```python
integrator.terminate()
```

## Integrate and compute the Lyapunov exponents

Setting some parameters for the computation of the first Lyapunov exponent

```python
integration_time = 10000000.
lyap_dt = 0.5
number_of_steps = int(integration_time/lyap_dt)
record_every_nsteps = 20
```

We compute now the first Lyapunov exponent by evolving a single vector (per trajectory) with the VDDG tangent linear model. This vector will align quickly with the first (most unstable) Lyapunov direction. We renormalize this vector every `lyap_dt`, and its growth over time gives us the first local Lyapunov exponent. The average of this exponent over time gives the first Lyapunov exponent:

```python
%%time

X = ic
# Starts with a set of random normalized vectors
DX = np.random.rand(number_of_trajectories, ndim+3)
DX = (DX.T / np.sqrt((DX**2).sum(axis=-1))).T
t = 0
time = list()
traj = list()
exponents = list()
cumulative_exponents = np.zeros(number_of_trajectories)
for i in range(number_of_steps):
    # Integration with the tangent linear model of the VDDG model alone
    tn, Xn, DXn = integrate_runge_kutta_tgls(f_forced, Df_forced, t, t+lyap_dt, dt, ic=X, tg_ic=DX, write_steps=0)
    amplitude = np.sqrt((DXn**2).sum(axis=-1))
    local_exponent = np.log(amplitude) / lyap_dt
    cumulative_exponents += local_exponent
    DX = (DXn.T / amplitude).T
    # Records the result every n steps:
    X = Xn
    if i % record_every_nsteps == 0:
        traj.append(Xn)
    if i % record_every_nsteps == 0:
        exponents.append(local_exponent)
    t = tn
    if i % record_every_nsteps == 0:
        time.append(t)

averaged_exponents = cumulative_exponents / number_of_steps
```

    CPU times: user 1h 22min 18s, sys: 5.79 s, total: 1h 22min 24s
    Wall time: 1h 22min 38s

```python
exponents = np.moveaxis(np.array(exponents), 0, -1)
traj = np.moveaxis(np.array(traj), 0, -1)
time = np.array(time)
```

Averaged first Lyapunov exponent:

```python
averaged_exponents
```

    array([0.02064013, 0.01722321])

Plot of the variables $\theta_{{\rm a}, 1}$, $\psi_{{\rm o}, 2}$, $\theta_{{\rm o}, 2}$, the local first Lyapunov exponent $\sigma_1$ and its cumulative average, and finally $(x+y)$, all versus the time, for the first and second trajectories

```python
var1 = 10
var2 = 21
exp_range = slice(0,1)
colors = ['tab:orange', 'tab:blue']
plt.figure(figsize=(15, 40))

plt.subplot(5,1,1)
for nt in range(number_of_trajectories):
    plt.plot(model_parameters.dimensional_time*time, traj[nt, var1], color=colors[nt], label='PBA '+str(nt+1))
plt.xlabel('time (years)')
plt.ylabel('$'+model_parameters.latex_var_string[var1]+'$');
plt.legend();

plt.subplot(5,1,2)
for nt in range(number_of_trajectories):
    plt.plot(model_parameters.dimensional_time*time, traj[nt, var2], color=colors[nt], label='PBA '+str(nt+1))
plt.xlabel('time (years)')
plt.ylabel('$'+model_parameters.latex_var_string[var2]+'$');
plt.legend();

plt.subplot(5,1,3)
for nt in range(number_of_trajectories):
    plt.plot(model_parameters.dimensional_time*time, exponents[nt], color=colors[nt], label='PBA '+str(nt+1))
plt.xlabel('time (years)')
plt.ylabel('Local Lyapunov exponents $\sigma_1$');
plt.legend();

plt.subplot(5,1,4)
for nt in range(number_of_trajectories):
    plt.plot(model_parameters.dimensional_time*time, np.cumsum(exponents[nt]) / (np.arange(len(exponents[nt]))+1.), color=colors[nt], label='PBA '+str(nt+1))
plt.xlabel('time (years)')
plt.ylabel('Averaged Lyapunov exponents $\sigma_1$');
plt.legend();

plt.subplot(5,1,5)
plt.plot(model_parameters.dimensional_time*time, traj[0, 36] + traj[0, 37])
plt.xlabel('time (years)')
plt.ylabel('$x + y$');
```


## Helmholtz energy conversion of cubics

As in Michelsen's book, a generalized volume-translated cubic EOS can be given the common form:
$$ p = \frac{RT}{v-b}-\frac{a(T)}{(v+\Delta_1b)(v+\Delta_2b)} $$

$$ p = \frac{RT}{v+c-b}-\frac{a(T)}{(v+c+\Delta_1b)(v+c+\Delta_2b)} $$

```python
from __future__ import division
from sympy import *
from IPython.display import display, Math, Latex
from IPython.core.display import display_html
init_session(quiet=True)
init_printing()
```

    IPython console for SymPy 0.7.6.1 (Python 2.7.10-32-bit) (ground types: python)

```python
rho_r,rho,tau,delta,T_r = symbols('rho_r,rho,tau,delta,T_r', positive = True, real = True)
R,Delta_1,Delta_2,T,v,c = symbols('R,Delta_1,Delta_2,T,v,c', positive = True, real = True)
x_i,x_j,x_k = symbols('x_i,x_j,x_k', positive=True, real = True)
a = symbols('a', cls=Function, positive = True, real = True)(tau)
b = symbols('b', positive = True, real = True)
```

```python
normal_cubic = True # True for no volume translation
cubic = 'SRK'
```

```python
p = R*T/(v+c-b)-a/((v+c+Delta_1*b)*(v+c+Delta_2*b))
if cubic == 'SRK':
    p = p.subs(Delta_1,1).subs(Delta_2,0)
elif cubic == 'VdW':
    p = p.subs(Delta_1,0).subs(Delta_2,0)
elif cubic == 'PR':
    p = p.subs(Delta_1,1+sqrt(2)).subs(Delta_2,1-sqrt(2))
Math('p = '+latex(p))
```

$$p = \frac{R T}{- b + c + v} - \frac{a{\left (\tau \right )}}{\left(c + v\right) \left(b + c + v\right)}$$

```python
Z = p*v/(R*T)
Math('Z = '+latex(Z))
```

$$Z = \frac{v}{R T} \left(\frac{R T}{- b + c + v} - \frac{a{\left (\tau \right )}}{\left(c + v\right) \left(b + c + v\right)}\right)$$

```python
Z = Z.replace(v, 1/rho)
display(Math('Z = '+latex(Z)))
Z = Z.replace(T, T_r/tau)
Z = Z.replace(rho, delta*rho_r)
display(Math('Z = '+latex(Z)))
dalphar_dDelta = (Z-1)/delta
dalphar_dDelta = expand(simplify(dalphar_dDelta))
Math('\\frac{d\\alpha^r}{d\delta}|\\tau = ' + latex(dalphar_dDelta))
```

$$Z = \frac{1}{R T \rho} \left(\frac{R T}{- b + c + \frac{1}{\rho}} - \frac{a{\left (\tau \right )}}{\left(c + \frac{1}{\rho}\right) \left(b + c + \frac{1}{\rho}\right)}\right)$$

$$Z = \frac{\tau}{R T_{r} \delta \rho_{r}} \left(\frac{R T_{r}}{\tau \left(- b + c + \frac{1}{\delta \rho_{r}}\right)} - \frac{a{\left (\tau \right )}}{\left(c + \frac{1}{\delta \rho_{r}}\right) \left(b + c + \frac{1}{\delta \rho_{r}}\right)}\right)$$

$$\frac{d\alpha^r}{d\delta}|\tau = - \frac{\tau a{\left (\tau \right )}}{R T_{r} b c \delta^{2} \rho_{r} + R T_{r} b \delta + R T_{r} c^{2} \delta^{2} \rho_{r} + 2 R T_{r} c \delta + \frac{R T_{r}}{\rho_{r}}} + \frac{\tau}{- b \delta^{2} \rho_{r} \tau + c \delta^{2} \rho_{r} \tau + \delta \tau} - \frac{1}{\delta}$$

```python
antideriv_pieces = 0
for arg in expand(simplify(dalphar_dDelta))._args:
    antideriv_pieces += integrate(arg,delta)

antideriv_pieces = simplify(antideriv_pieces)
alphar = antideriv_pieces - antideriv_pieces.subs(delta,0)

display(Math('\\alpha^r = ' + latex(simplify(alphar))))
display(Math('\\lim_{c \\to 0}\\alpha^r = ' + latex(simplify(limit(alphar,c,0)))))
```

$$\alpha^r = \frac{1}{R T_{r} b} \left(R T_{r} b \log{\left (- \frac{1}{\rho_{r} \left(b - c\right)} \right )} - R T_{r} b \log{\left (\frac{\delta \rho_{r} \left(b - c\right) - 1}{\rho_{r} \left(b - c\right)} \right )} + a{\left (\tau \right )} \log{\left (\frac{c^{\tau} \left(b + c \delta \rho_{r} \left(b + c\right) + c\right)^{\tau}}{\left(c \left(\delta \rho_{r} \left(b + c\right) + 1\right)\right)^{\tau} \left(b + c\right)^{\tau}} \right )}\right)$$

$$\lim_{c \to 0}\alpha^r = - \log{\left (b \delta \rho_{r} - 1 \right )} + i \pi - \frac{\tau a{\left (\tau \right )}}{R T_{r} b} \log{\left (b \delta \rho_{r} + 1 \right )}$$

```python
# Maybe turn off volume translation
if normal_cubic:
    alphar = limit(alphar,c,0)
```

```python
def format_deriv(arg, itau, idel, RHS):
    """
    A function for giving a nice latex representation of
    the partial derivative in question
    """
    if itau+idel == 1:
        numexp = ''
    else:
        numexp = '^{{{s:d}}}'.format(s=itau+idel)

    if itau == 0:
        tau = ''
    elif itau == 1:
        tau = '\\partial \\tau'
    else:
        tau = '\\partial \\tau^{{{s:d}}}'.format(s=itau)

    if idel == 0:
        delta = ''
    elif idel == 1:
        delta = '\\partial \\delta'
    else:
        delta = '\\partial \\delta^{{{s:d}}}'.format(s=idel)

    temp = '\\frac{{\\partial{{{numexp:s}}} {arg:s}}}{{{{{tau:s}}}{{{delta:s}}}}} = '
    if itau == 0 and idel == 0:
        as_latex = arg +' = ' + latex(RHS)
    else:
        as_latex = temp.format(**locals()) + latex(RHS)
    return Math(as_latex), as_latex
```

```python
for deriv_count in range(1,5):
    for dtau in range(deriv_count+1):
        ddelta = deriv_count-dtau
        #print dtau, ddelta
        mth, tex = format_deriv('\\alpha^r', dtau, ddelta,
                             diff(diff(alphar,tau,dtau),delta,ddelta))
        #print tex
        display(mth)
```

$$\frac{\partial{} \alpha^r}{{}{\partial \delta}} = \frac{1}{R T_{r} b} \left(- \frac{R T_{r} b}{\delta - \frac{1}{b \rho_{r}}} - \frac{b \rho_{r} \tau a{\left (\tau \right )}}{b \delta \rho_{r} + 1}\right)$$

$$\frac{\partial{} \alpha^r}{{\partial \tau}{}} = \frac{1}{R T_{r} b} \left(- \tau \log{\left (b \delta \rho_{r} + 1 \right )} \frac{d}{d \tau} a{\left (\tau \right )} - a{\left (\tau \right )} \log{\left (b \delta \rho_{r} + 1 \right )}\right)$$

$$\frac{\partial{^{2}} \alpha^r}{{}{\partial \delta^{2}}} = \frac{1}{R T_{r}} \left(\frac{R T_{r}}{\left(\delta - \frac{1}{b \rho_{r}}\right)^{2}} + \frac{b \rho_{r}^{2} \tau a{\left (\tau \right )}}{\left(b \delta \rho_{r} + 1\right)^{2}}\right)$$

$$\frac{\partial{^{2}} \alpha^r}{{\partial \tau}{\partial \delta}} = \frac{1}{R T_{r} b} \left(- \frac{b \rho_{r} \tau \frac{d}{d \tau} a{\left (\tau \right )}}{b \delta \rho_{r} + 1} - \frac{b \rho_{r} a{\left (\tau \right )}}{b \delta \rho_{r} + 1}\right)$$

$$\frac{\partial{^{2}} \alpha^r}{{\partial \tau^{2}}{}} = - \frac{1}{R T_{r} b} \left(\tau \frac{d^{2}}{d \tau^{2}}  a{\left (\tau \right )} + 2 \frac{d}{d \tau} a{\left (\tau \right )}\right) \log{\left (b \delta \rho_{r} + 1 \right )}$$

$$\frac{\partial{^{3}} \alpha^r}{{}{\partial \delta^{3}}} = - \frac{1}{R T_{r}} \left(\frac{2 R T_{r}}{\left(\delta - \frac{1}{b \rho_{r}}\right)^{3}} + \frac{2 b^{2} \rho_{r}^{3} \tau a{\left (\tau \right )}}{\left(b \delta \rho_{r} + 1\right)^{3}}\right)$$

$$\frac{\partial{^{3}} \alpha^r}{{\partial \tau}{\partial \delta^{2}}} = \frac{b \rho_{r}^{2} \left(\tau \frac{d}{d \tau} a{\left (\tau \right )} + a{\left (\tau \right )}\right)}{R T_{r} \left(b \delta \rho_{r} + 1\right)^{2}}$$

$$\frac{\partial{^{3}} \alpha^r}{{\partial \tau^{2}}{\partial \delta}} = - \frac{\rho_{r}}{R T_{r} \left(b \delta \rho_{r} + 1\right)} \left(\tau \frac{d^{2}}{d \tau^{2}}  a{\left (\tau \right )} + 2 \frac{d}{d \tau} a{\left (\tau \right )}\right)$$

$$\frac{\partial{^{3}} \alpha^r}{{\partial \tau^{3}}{}} = - \frac{1}{R T_{r} b} \left(\tau \frac{d^{3}}{d \tau^{3}}  a{\left (\tau \right )} + 3 \frac{d^{2}}{d \tau^{2}}  a{\left (\tau \right )}\right) \log{\left (b \delta \rho_{r} + 1 \right )}$$

$$\frac{\partial{^{4}} \alpha^r}{{}{\partial \delta^{4}}} = \frac{1}{R T_{r}} \left(\frac{6 R T_{r}}{\left(\delta - \frac{1}{b \rho_{r}}\right)^{4}} + \frac{6 b^{3} \rho_{r}^{4} \tau a{\left (\tau \right )}}{\left(b \delta \rho_{r} + 1\right)^{4}}\right)$$

$$\frac{\partial{^{4}} \alpha^r}{{\partial \tau}{\partial \delta^{3}}} = - \frac{2 b^{2} \rho_{r}^{3}}{R T_{r} \left(b \delta \rho_{r} + 1\right)^{3}} \left(\tau \frac{d}{d \tau} a{\left (\tau \right )} + a{\left (\tau \right )}\right)$$

$$\frac{\partial{^{4}} \alpha^r}{{\partial \tau^{2}}{\partial \delta^{2}}} = \frac{b \rho_{r}^{2}}{R T_{r} \left(b \delta \rho_{r} + 1\right)^{2}} \left(\tau \frac{d^{2}}{d \tau^{2}}  a{\left (\tau \right )} + 2 \frac{d}{d \tau} a{\left (\tau \right )}\right)$$

$$\frac{\partial{^{4}} \alpha^r}{{\partial \tau^{3}}{\partial \delta}} = - \frac{\rho_{r}}{R T_{r} \left(b \delta \rho_{r} + 1\right)} \left(\tau \frac{d^{3}}{d \tau^{3}}  a{\left (\tau \right )} + 3 \frac{d^{2}}{d \tau^{2}}  a{\left (\tau \right )}\right)$$

$$\frac{\partial{^{4}} \alpha^r}{{\partial \tau^{4}}{}} = - \frac{1}{R T_{r} b} \left(\tau \frac{d^{4}}{d \tau^{4}}  a{\left (\tau \right )} + 4 \frac{d^{3}}{d \tau^{3}}  a{\left (\tau \right )}\right) \log{\left (b \delta \rho_{r} + 1 \right )}$$

Derivatives of $a_i(\tau)$
==============

```python
a0,omega,kappa = symbols('a0,omega,kappa')
ai=a0*(1+kappa*(1-sqrt(1/tau)))**2

for diff_count in range(0,5):
    mth, tex = format_deriv('a_i',diff_count, 0, diff(ai, tau, diff_count))
    display(mth)
```

$$a_i = a_{0} \left(\kappa \left(1 - \frac{1}{\sqrt{\tau}}\right) + 1\right)^{2}$$

$$\frac{\partial{} a_i}{{\partial \tau}{}} = \frac{a_{0} \kappa}{\tau^{\frac{3}{2}}} \left(\kappa \left(1 - \frac{1}{\sqrt{\tau}}\right) + 1\right)$$

$$\frac{\partial{^{2}} a_i}{{\partial \tau^{2}}{}} = \frac{a_{0} \kappa}{2} \left(\frac{\kappa}{\tau^{3}} - \frac{1}{\tau^{\frac{5}{2}}} \left(3 \kappa \left(1 - \frac{1}{\sqrt{\tau}}\right) + 3\right)\right)$$

$$\frac{\partial{^{3}} a_i}{{\partial \tau^{3}}{}} = \frac{3 a_{0}}{4} \kappa \left(- \frac{3 \kappa}{\tau^{4}} + \frac{1}{\tau^{\frac{7}{2}}} \left(5 \kappa \left(1 - \frac{1}{\sqrt{\tau}}\right) + 5\right)\right)$$

$$\frac{\partial{^{4}} a_i}{{\partial \tau^{4}}{}} = \frac{3 a_{0}}{8} \kappa \left(\frac{29 \kappa}{\tau^{5}} - \frac{1}{\tau^{\frac{9}{2}}} \left(35 \kappa \left(1 - \frac{1}{\sqrt{\tau}}\right) + 35\right)\right)$$


# 数据降维

小说《三体》中的终极武器是降维打击，这里的概念是相近的。

常用的算法| 简述
:-|-:
主成分分析PCA|转换后样本方差最大的线性降维方式
线性判别分析LDA|类似PCA，但对带标签数据效果更好
等距映射Isomap|降维后依然保持近邻点间结构不变的流形学习方法
局部线性嵌入LLE|假设样本在局部空间满足线性关系的流形学习方法
谱聚类|先用Laplacian Eigenmaps进行流形学习降维，再聚类的方法
t-SNE|用概率建模的流形学习方法

### 主成分分析

求样本最大方差的方法：

- 特征值
- 奇异值分解

```python
%matplotlib inline
import matplotlib.pyplot as plt
from sklearn import decomposition
from sklearn import datasets

iris = datasets.load_iris()
X = iris.data
y = iris.target

plt.subplot(1,2,1)
for name, label, m in [('Setosa', 0, '<'), ('Versicolour', 1, 'o'), ('Virginica', 2, '>')]:
    plt.scatter(X[y==label, 0], X[y==label, 1], label=name, marker=m)

plt.legend()
plt.title("data shape: %s "%(X.shape,))
pca = decomposition.PCA(n_components=2)
pca.fit(X)
X = pca.transform(X)

plt.subplot(1,2,2)
for name, label, m in [('Setosa', 0, '<'), ('Versicolour', 1, 'o'), ('Virginica', 2, '>')]:
    plt.scatter(X[y==label, 0], X[y==label, 1], label=name, marker=m)

plt.legend()
plt.title("data shape: %s "%(X.shape,))

plt.show()
```

### 线性判别分析

相比PCA，多考虑了每种标签样本数据集内的分布情况，即“最大化类间样本的方差，最小化类内样本的方差”。因此它们算法差别在于，PCA计算所有数据协方差矩阵的特征值/特征向量，而LDA计算的是类内协方差矩阵的逆与类间协方差矩阵乘积的特征值/特征向量。

LDA缺点是降维后的最大维数既受特征数量，也受原始数据维度影响，PCA仅受原始数据维度影响。

**在sklearn中LinearDiscriminantAnalysis既是一个监督分类器，也是一个降维器。**

```python
%matplotlib inline
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn import datasets
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA

iris = datasets.load_iris()
X = iris.data
y = iris.target

plt.subplot(131)
pca = PCA(n_components=2).fit(X)
X_pca = pca.transform(X)

for name, label, m in [('Setosa', 0, '<'), ('Versicolour', 1, 'o'), ('Virginica', 2, '>')]:
    plt.scatter(X_pca[y==label, 0], X_pca[y==label, 1], label=name, marker=m)

plt.legend()
plt.title("PCA")

plt.subplot(132)
lda_svd = LDA(n_components=2, solver="svd").fit(X,y)
X_lda_svd = lda_svd.transform(X)

for name, label, m in [('Setosa', 0, '<'), ('Versicolour', 1, 'o'), ('Virginica', 2, '>')]:
    plt.scatter(X_lda_svd[y==label, 0], X_lda_svd[y==label, 1], label=name, marker=m)

plt.legend()
plt.title("LDA-svd")

plt.subplot(133)
lda = LDA(n_components=2, solver="eigen").fit(X,y)
X_lda = lda.transform(X)

for name, label, m in [('Setosa', 0, '<'), ('Versicolour', 1, 'o'), ('Virginica', 2, '>')]:
    plt.scatter(X_lda[y==label, 0], X_lda[y==label, 1], label=name, marker=m)

plt.legend()
plt.title("LDA-eigen")

plt.show()
```

如果将上面的LDA降维策略从svd换成eigen，下图将这些点在折线图中，做对比

很明显，可以发现，数据分布还是比较统一的，差别是延坐标轴做了平移

```python
fig=plt.figure(figsize=(8,8))
plt.plot(X_lda_svd)
plt.plot(X_lda)
```

降维之后，无法从视觉感知降维的效果时，可以计算方差比，若总和小于1，说明丢失了部分信息。一般可以用“降维保留了百分之多少的方差比”形容降维效果。

```python
print(pca.explained_variance_ratio_)
print(lda_svd.explained_variance_ratio_)
print(lda.explained_variance_ratio_)
```

    [0.92461872 0.05306648]
    [0.9912126 0.0087874]
    [0.9912126 0.0087874]

### 流形学习之Isomap

流形学习是非线性降维的主要方法。流形（manifold）是局部具有欧几里得性质的空间，通常嵌入比该局部空间更高维的外围欧几里得空间中。从英语含义也可引申出，Manifold= Many + fold(小平面)

适用性： 流形学习也适用于PCA等要求的高维空间本身有信息冗余的场景；流形的基础是局部的欧几里得空间，所以需要有足够密集的数据样本让算法检测到局部空间（现实场景中的测地线）；流形学习一般对噪声比较敏感。

**流形学习的核心步骤**

1. 用某种方式表达高维和低维空间样本之间的亲缘性（Isomap中是相似度，LLE中是近邻权重矩阵，LE中是拉普拉斯矩阵）
2. 以亲缘性尽量不变为优化目标，用拉格朗日乘子或特征值分解求得低维映射

```python
%matplotlib inline
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn import datasets
from sklearn.manifold import Isomap

iris = datasets.load_iris()
X, y = iris.data, iris.target

plt.subplot(221)
X_pca = PCA(n_components=2).fit_transform(X)
for name, label, m in [('Setosa', 0, '<'), ('Versicolour', 1, 'o'), ('Virginica', 2, '>')]:
    plt.scatter(X_pca[y==label, 0], X_pca[y==label, 1], label=name, marker=m)
plt.legend()
plt.title("PCA")

for idx, neighbor in enumerate([2,20,100]):
    plt.subplot(222+idx)
    isomap = Isomap(n_components=2, n_neighbors=neighbor)
    X_isomap = isomap.fit_transform(X)

    for name, label, m in [('Setosa', 0, '<'), ('Versicolour', 1, 'o'), ('Virginica', 2, '>')]:
        plt.scatter(X_isomap[y==label, 0], X_isomap[y==label, 1], label=name, marker=m)
    #plt.legend()
    plt.title("Isomap (n_neighbors=%d)"%neighbor)
plt.tight_layout()
plt.show()
```

### 局部线性嵌入（Locally Linear Embedding，LLE）

是一种以保持每个样本与邻近样本之间距离作为降维目标的流形学习方法。

步骤：
1. 选择近邻
2. 建立权重系数矩阵（表达每个样本与邻近样本的关系）
3. 降维（降维后尽量保持前面的权重矩阵）

#### 权重系数矩阵

基本假设是每个样本可以通过近邻结点的线性组合近似的表示。权重系数矩阵中的每个元素值
$w_{ij}$用于表示用样本$j$生成样本$i$时使用的权值，即
$$^{\bar{x}_i}=\sum_{j \in N_i}w_{ij}x_j$$

其中$N_i$是所有邻近点的集合，可以用“最近的k个点”这样的策略定义。$x_j$是第j个真实
的原始特征向量，$^{\bar{x}_i}$是第i个通过近邻点恢复出的特征向量。

#### 寻找最优权重

尽管理想情况下，$^{\bar{x}_i}=x_i$，但是一个样本$x_i$是否真的能从邻近点恢复，
取决于自身的分布情况。所以LLE首先找出使$^{\bar{x}_i}$最接近$x_i$的一组权重系数，优化目标
如下：
$$min\sum_{i}||x_i-^{\bar{x}_i}||^2$$
再加上权重归一化要求，$\sum_{j\in N_i}w_{ij}=1$，可用拉格朗日乘子法求解。

#### 降维

最小化优化目标：
$$J(Y)=\sum_{i=1..n}||y_i-\sum_{j\in N_i}w_{ij}x_j||^2$$

最后上式可转换：
$$J(Y)=trace(Y^T(I-W)^T(I-W)Y)$$(trace是矩阵迹函数，即矩阵对角线元素之和)，
另有归一化条件$Y^TY=nI$，也可用拉格朗日乘子法求解。

#### 用特征值分解求优化问题

最后一步降维时常用特征值分解代替拉格朗日乘子法。因为两个方法最终是一致的，特征值计算更快。

### 拉普拉斯映射(Laplacian Eigenmaps,LE)

与LLE区别在，LLE尽量保持权重系数矩阵进行降维，LE围绕保持拉普拉斯矩阵进行降维。

#### 拉普拉斯矩阵

它是图论中表示无向有权图结构的矩阵。定义如下：

$$L=D-W$$
其中D是度矩阵(对角线元素是该点所有边的权值和，即$d_{ii}=\sum w_{ij}$，其他元素是0)，W是邻接权重矩阵。

#### 降维优化函数

$$J(Y)=\sum_{i,j\in N}||y_i-y_j||^2w_{ij}$$
其中N是所有样本集合，$||y_i-y_j||^2$可以看成两个样本的距离度量，所以该目标函数直观意义是：
- 两样本间原始权重越大，则降维后越应该使两个样本距离变小
- 样本间权值越低，则降维后对整体目标影响越小，也就是说可以离得更远

整理目标函数发现：

$$J(Y)=\sum_{i,j\in N}||y_i-y_j||^2w_{ij}=\sum_{i,j\in N}y_i^2w_{ij}+\sum_{i,j\in N}y_j^2w_{ij}-2\sum_{i,j\in N}2y_iy_jw_{ij}$$

由于$d_{ii}=\sum w_{ij}$，所以上式最终变成：

$$
\begin{align}
J(Y)&=\sum_{i,j\in N}y_i^2w_{ij}+\sum_{i,j\in N}y_j^2w_{ij}-2\sum_{i,j\in N}2y_iy_jw_{ij}\\
&=trace(2Y^T(D-W)Y)\\
&=trace(2Y^TLY)
\end{align}
$$

其中Y是降维后所有样本坐标值，L就是原始样本的拉普拉斯矩阵。

#### 优化函数求解

为防止过拟合，即降维后所有样本有相同的坐标值，这是最小目标。必须加入约束条件$Y^TLY=Y$。这里也用特征值分解就好，因为现在优化目标是$min(trace(2Y^TLY))$，因此只要对L进行特征值分解并取最小的d个特征值对应的特征向量（d是降维后的维度），就可以求得Y。

```python
%matplotlib inline
import matplotlib.pyplot as plt
from sklearn import datasets
from sklearn.manifold import LocallyLinearEmbedding as LLE
from sklearn.manifold import Isomap
from sklearn.manifold import SpectralEmbedding as SE

iris = datasets.load_iris()
X, y = iris.data, iris.target

plt.subplot(221)
X_isomap = Isomap(n_components=100).fit_transform(X)
for name, label, m in [('Setosa', 0, '<'), ('Versicolour', 1, 'o'), ('Virginica', 2, '>')]:
    plt.scatter(X_isomap[y==label, 0], X_isomap[y==label, 1], label=name, marker=m)
plt.legend()
plt.title("Isomap")

for idx, neighbor in enumerate([2,20,100]):
    plt.subplot(222+idx)
    lle = LLE(n_components=2, n_neighbors=neighbor)
    X_lle = lle.fit_transform(X)

    for name, label, m in [('Setosa', 0, '<'), ('Versicolour', 1, 'o'), ('Virginica', 2, '>')]:
        plt.scatter(X_lle[y==label, 0], X_lle[y==label, 1], label=name, marker=m)
    #plt.legend()
    plt.title("LLE (n_neighbors=%d)"%neighbor)
plt.tight_layout()
plt.show()
```

```python
plt.subplot(221)
X_isomap = Isomap(n_components=2).fit_transform(X)
for name, label, m in [('Setosa', 0, '<'), ('Versicolour', 1, 'o'), ('Virginica', 2, '>')]:
    plt.scatter(X_isomap[y==label, 0], X_isomap[y==label, 1], label=name, marker=m)
plt.legend()
plt.title("Isomap")

for idx, neighbor in enumerate([2,20,100]):
    plt.subplot(222+idx)
    se = SE(n_components=2, n_neighbors=neighbor)
    X_se = se.fit_transform(X)

    for name, label, m in [('Setosa', 0, '<'), ('Versicolour', 1, 'o'), ('Virginica', 2, '>')]:
        plt.scatter(X_se[y==label, 0], X_se[y==label, 1], label=name, marker=m)
    #plt.legend()
    plt.title("SE (n_neighbors=%d)"%neighbor)
plt.tight_layout()
plt.show()
```

### 谱聚类

明显是一种聚类模型，相比K-means，计算代价低、适应非凸数据集等特点，特别适应于高维数据
的聚类。

#### 原理

放到这里，没有写在聚类那里，是因为它和拉普拉斯映射有密切关系。

$$谱聚类 = 拉普拉斯映射 + K-means 聚类$$

设目标聚类分组数是k，谱聚类就是对样本执行LE降维到K维后执行centers=k的K-means聚类。

```python
%matplotlib inline
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import numpy as np
from sklearn.cluster import SpectralClustering
from sklearn import datasets

iris = datasets.load_iris()

x=iris.data[:,:3]
y=iris.target
result=SpectralClustering(n_clusters=3, eigen_solver='arpack',gamma=0.01,
                      n_neighbors=100, assign_labels='discretize').fit_predict(x)

fig=plt.figure(figsize=(12,8))
ax = Axes3D(fig, elev=-150, azim=110)

ax.scatter(x[:,0],x[:,1],x[:,2],c=result,cmap=plt.cm.tab20, edgecolor='k',s=40)
ax.scatter(x[:,0],x[:,1],x[:,2],c=y,cmap=plt.cm.tab20, edgecolor='k',s=40)
ax.set_title('谱聚类')
ax.set_xlabel('花萼长度')
ax.w_xaxis.set_ticklabels([])
ax.set_ylabel('花萼宽度')
ax.w_yaxis.set_ticklabels([])
ax.set_zlabel('花瓣长度')
ax.w_zaxis.set_ticklabels([])

#ax.scatter(sc.fit_predict(x),c=y,cmap=plt.cm.Set1, edgecolor='b',s=140)
if(result[0]!=y[0]):
    for index,val in enumerate(result):
        if val==0:
            result[index]=2
        elif val==2:
            result[index]=0

print(result,y,sep='\n')
true_partion_sum=0;
for r,l in zip(result,y):
    if(r==l):
        true_partion_sum += 1

print("精度是：%.2f%%" % (true_partion_sum/len(y)), true_partion_sum,len(y))
print("邻接权重矩阵：\n",sc.affinity_matrix_)
plt.show()
```

### 流形学习之t-SNE

t分布随机近邻嵌入，从降维效果来看，是一个流形学习的终极武器，缺点是训练时间较长。

t-SNE假设高维空间符合高斯分布，低维空间符合t分布，以两个分布的Kullback-Leiber散度
(简称KL散度)最小为优化目标。

为啥用t分布，高斯分布在中心极限定理中很是受用，原因是流形学习的**拥挤问题**。

#### 拥挤问题

前面一直假设高维空间样本降维依然尽量保持亲缘性或等距，，这几乎是流形学习的出发点，
但这个实际上很难达到。

t分布从某种程度上缓解了拥挤问题。
1. 只有一个超参数——自由度，这里固定为1
2. 形态上与高斯分布类似，只是二维表现图像的两侧尾部比高斯分布高。自由度越低，
两侧尾部越高；自由度取∞，t分布演变为标准高斯分布。

```python
%matplotlib inline
import numpy as np
from scipy.stats import norm
from scipy.stats import t
import matplotlib.pyplot as plt

print('比较t-分布与标准正态分布')
x = np.linspace( -3, 3, 100)
plt.plot(x, t.pdf(x,1), label='df=1')
plt.plot(x, t.pdf(x,2), label='df=20')
plt.plot(x, t.pdf(x,100), label = 'df=100')
plt.plot( x[::5], norm.pdf(x[::5]),'kx', label='normal')
plt.legend()
plt.show()
```

### 综合对比

以一个三维空间中的一个S状流形降到两维为目标，分别使用PCA、LDA、MDS、Isomap、LLE、LE及t-SNE。

```python
%matplotlib inline
from time import time
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib.ticker import NullFormatter
from sklearn import manifold, datasets
from sklearn.decomposition import PCA
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA

Axes3D

n_points = 1000
X, color = datasets.samples_generator.make_s_curve(n_points, random_state=0)
n_neighbors = 10
n_components = 2
fig = plt.figure(figsize=(15,8))
ax = fig.add_subplot(251, projection='3d')
ax.scatter(X[:, 0], X[:, 2], c=color, cmap=plt.cm.Spectral)
ax.view_init(24, -52) #(z平面仰角，xy的平面方位角)

methods = ['standard', 'ltsa', 'hessian']
labels = ['LLE', 'LTSA','Hession LLE']

for i, method in enumerate(methods):
    t0 = time()
    Y = manifold.LocallyLinearEmbedding(n_neighbors, n_components, eigen_solver='auto',
                                        method=method).fit_transform(X)
    t1 = time()
    print("%s: %.2g sec" % (methods[i], t1-t0))
    ax = fig.add_subplot(252+i)

    plt.scatter(Y[:, 0], Y[:, 1], c=color, cmap=plt.cm.Spectral)
    plt.title("%s (%.2g sec)" % (labels[i], t1-t0))
    ax.xaxis.set_major_formatter(NullFormatter())
    ax.yaxis.set_major_formatter(NullFormatter())
    plt.axis('tight')

#  初始化六种降维模型
estimators = [(manifold.Isomap(n_neighbors, n_components), "Isomap"),
              (manifold.MDS(n_components, max_iter=100, n_init=1), "MDS"),
             (manifold.SpectralEmbedding(n_components=n_components,n_neighbors=n_neighbors),
                                        "Laplace Eigenmaps"),
             (manifold.TSNE(n_components=n_components, init='pca', random_state=0),"t-SNE"),
             (PCA(n_components), "PCA"),
             (LDA(n_components=n_components), "LDA")]

# 训练并显示降维结果
for idx, (estimator_obj, estimator_name) in enumerate(estimators):
    t0 = time()
    if estimator_name =='LDA':
        Y = estimator_obj.fit_transform(X, (color).astype(int))
    else:
        Y = estimator_obj.fit_transform(X)
    t1 = time()
    print("%s: %.2g sec" % (estimator_name, t1-t0))
    ax = fig.add_subplot(2,5,5+idx)
    plt.scatter(Y[:, 0], Y[:, 1], c=color, cmap=plt.cm.Spectral)
    plt.title("%s (%.2g sec)" % (estimator_name, t1-t0))
    ax.xaxis.set_major_formatter(NullFormatter())
    ax.yaxis.set_major_formatter(NullFormatter())
    plt.axis('tight')

plt.show()
```


```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats
import scipy as sp
from mpl_toolkits.mplot3d import Axes3D
import statsmodels.api as sm
import linregfunc as lr
```

    E:\Anaconda\lib\site-packages\statsmodels\compat\pandas.py:65: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
      from pandas import Int64Index as NumericIndex

#  Common Distributions in Hypothesis Testing

Here we will have a quick refresh of distributions that are commonly used in hypothesis testing.

##  The Normal Distributions

Normal Distribution is the most important one among all, here we provide a graphic reminder of bivariate normal distribution. Please check out my notebooks of linear algebra, there is a whole chapter devoted for normal distribution.

For your reference, the pdf of multivariate normal distribution is

$$
p(\boldsymbol{x} ; \mu, \Sigma)=\frac{1}{(2 \pi)^{n / 2}|\Sigma|^{1 / 2}} \exp \left(-\frac{1}{2}(x-\mu)^{T} \Sigma^{-1}(x-\mu)\right) \tag{1}\label{1}
$$

```python
x = np.linspace(-10,10,500)
y = np.linspace(-10,10,500)
X,Y = np.meshgrid(x,y)

pos = np.array([X.flatten(),Y.flatten()]).T # two columns matrix

fig = plt.figure(figsize=(14,14))

#########################
ax = fig.add_subplot(221, projection='3d')

mu_x = 0
mu_y = 0

sigma_x = 3
sigma_y = 15
rho = 3

rv = sp.stats.multivariate_normal([mu_x, mu_y], [[sigma_x, rho], [rho, sigma_y]]) # frozen distribution
ax.plot_surface(X, Y, rv.pdf(pos).reshape(500,500),cmap='coolwarm')

ax = fig.add_subplot(223)
ax.contour(rv.pdf(pos).reshape(500,500))
string1 = r'$\sigma_x = %.1f$, $\sigma_y = %.1f$, $\rho = %.1f$'% (sigma_x, sigma_y, rho)
ax.annotate(text = string1, xy=(0.2, 0.05), xycoords='axes fraction', color = 'r')

####

mu_x = 0
mu_y = 0

sigma_x = 10
sigma_y = 4
rho = 0

rv = sp.stats.multivariate_normal([mu_x, mu_y], [[sigma_x, rho], [rho, sigma_y]]) # frozen distribution
ax = fig.add_subplot(222, projection='3d')
ax.plot_surface(X, Y, rv.pdf(pos).reshape(500,500),cmap='coolwarm')

ax = fig.add_subplot(224)
ax.contour(rv.pdf(pos).reshape(500,500))
string2 = r'$\sigma_x = %.1f$, $\sigma_y = %.1f$, $\rho = %.1f$'% (sigma_x, sigma_y, rho)
ax.annotate(text = string2, xy=(0.2, 0.05), xycoords='axes fraction', color = 'r')
#########################

plt.show()
```

Keep in your mind: _any linear combination of normally distributed variables is yet a normal distribution_.

##  The Chi-Squared Distribution

If an $n$-random vector follows $iid$ normal distribution, $\boldsymbol{z}\sim N(\boldsymbol{0}, \mathbf{I})$. Then the random variable

$$
y = \boldsymbol{z}^T\boldsymbol{z} = \sum_{i=i}^n z_i^2
$$

is said to follow the **chi-squared distribution** with $n$ degrees of freedom. Denoted as

$$
y\sim\chi^2(n)
$$

The mean is

$$\mathrm{E}(y)=\sum_{i=1}^{m} \mathrm{E}\left(z_{i}^{2}\right)=\sum_{i=1}^{m} 1=m$$

And the variance is

$$\begin{aligned}
\operatorname{Var}(y) &=\sum_{i=1}^{m} \operatorname{Var}\left(z_{i}^{2}\right)=m \mathrm{E}\left(\left(z_{i}^{2}-1\right)^{2}\right) \\
&=m \mathrm{E}\left(z_{i}^{4}-2 z_{i}^{2}+1\right)=m(3-2+1)=2 m
\end{aligned}$$

As $n$ increases, the probability density function of $\chi^2$ approaches the $N(m, 2m)$. Here is the graphic demonstration shows how $\chi^2$ distribution changes as d.o.f. rises.

```python
fig, ax = plt.subplots(figsize = (9, 9))
x = np.linspace(0, 50, 1000)
for i in range(1, 14):
    chi_pdf = sp.stats.chi2.pdf(x, i)
    ax.plot(x, chi_pdf, lw = 3, label = '$\chi^2 (%.0d)$'%i)
    ax.legend(fontsize = 12)
ax.axis([0, 20, 0, .6])
plt.show()
```

###  Quadratic Form of $\chi^2$ Distribution

If an $n$-random vector $\boldsymbol{y} \sim N(\boldsymbol{\mu}, \Sigma)$ then

$$
(\boldsymbol{y} - \boldsymbol{\mu})^T\Sigma^{-1}(\boldsymbol{y}-\boldsymbol{\mu})\sim \chi^2(n)
$$

If $\boldsymbol{y} \sim N(\boldsymbol{0}, \Sigma)$ simplifies the expression

$$
\boldsymbol{y}^T\Sigma^{-1}\boldsymbol{y}\sim \chi^2(n)
$$

We will show why that holds by using diagonal decomposition. Since the $\Sigma$ is symmetric, it is orthogonally diagonalizable,

$$
\Sigma = QDQ^T
$$

where

$$
D=\left[\begin{array}{ccccc}
\lambda_{1} & 0 & 0 & \ldots & 0 \\
0 & \lambda_{2} & 0 & \ldots & 0 \\
\vdots & \vdots & \vdots & & \vdots \\
0 & 0 & 0 & \ldots & \lambda_{n}
\end{array}\right]
$$

$\lambda$s are eigenvalues. And $Q^{-1} = Q^T$, $Q$ holds all the eigenvectors of $\Sigma$ which are mutually perpendicular.

Denote $D^*$ as

$$
D^* =
\left[\begin{array}{ccccc}
\frac{1}{\sqrt{\lambda_{1}}} & 0 & 0 & \ldots & 0 \\
0 & \frac{1}{\sqrt{\lambda_{2}}} & 0 & \ldots & 0 \\
\vdots & \vdots & \vdots & & \vdots \\
0 & 0 & 0 & \ldots & \frac{1}{\sqrt{\lambda_{n}}}
\end{array}\right]
$$

Let the matrix $H = QD^*Q^T$, since $H$ is also symmetric

$$
HH^T= QD^*Q^TQD^*Q^T= Q^TD^*D^*Q = QD^{-1}Q^T =\Sigma^{-1}
$$

Furthermore

$$
H\Sigma H^T = QD^*Q^T\Sigma QD^*Q^T = QD^*Q^TQDQ^T  QD^*Q^T = QD^*DD^*Q^T = QQ^T = I
$$

Back to the results from above, we set $\boldsymbol{z} = H^T (\boldsymbol{y}-\boldsymbol{\mu})$, which is standard normal distribution since

$$
E(\boldsymbol{z})= H^TE(\boldsymbol{y}-\boldsymbol{\mu})=\boldsymbol{0}\\
\text{Var}(\boldsymbol{z}) = H\text{Var}(\boldsymbol{y}-\boldsymbol{\mu})H^T =H\Sigma H^T = I
$$

Back to where we started

$$
(\boldsymbol{y}-\boldsymbol{\mu})^T\Sigma^{-1}(\boldsymbol{y}-\boldsymbol{\mu}) = (\boldsymbol{y}-\boldsymbol{\mu})^THH^T (\boldsymbol{y}-\boldsymbol{\mu}) = (H^T (\boldsymbol{y}-\boldsymbol{\mu}))^T(H^T (\boldsymbol{y}-\boldsymbol{\mu})) = \boldsymbol{z}^T\boldsymbol{z}
$$

here we proved that $(\boldsymbol{y}-\boldsymbol{\mu})^T\Sigma^{-1}(\boldsymbol{y}-\boldsymbol{\mu}) \sim \chi^2(n)$.

More details of proof in this page.

##  The Student’s $t$ Distribution

If $z\sim N(0, 1)$ and $y\sim \chi^2(m)$, and $z$ and $y$ are independent, then

$$
t = \frac{z}{\sqrt{y/m}}
$$

follows the **Student's t distribution** with $m$ d.o.f.

Here is the plot of $t$-distribution, note that $t(1)$ is called **Cauchy distribution** which has no moments at all, because integral does not converge due to fat tails.

```python
fig, ax = plt.subplots(figsize = (12, 7))
x = np.linspace(-5, 5, 1000)
for i in range(1, 6):
    chi_pdf = sp.stats.t.pdf(x, i)
    if i == 1:
        ax.plot(x, chi_pdf, lw = 3, label = 'Cauchy', ls = '--')
        continue
    else:
        ax.plot(x, chi_pdf, lw = 3, label = '$t (%.0d)$'%i)
        ax.legend(fontsize = 12)
ax.axis([-5, 5, 0, .4])
ax.set_title("Student's $t$ Distribution", size = 18)
plt.show()
```

As $m \rightarrow \infty$, $t(m)\rightarrow N(0, 1)$, in the limit process, the tails of $t$ distribution will diminish, and becoming thinner.

##  The $F$ Distribution

If $y_1$ and $y_2$ are independent random variables distributed as $\chi^2(m_1)$ and $\chi^2(m_2)$, then the random variable

$$
F = \frac{y_1/m_1}{y_2/m_2}
$$

follows the **$F$ distribution**, denoted $ F(m_1, m_2)$.

```python
x = np.linspace(.001, 5, 100)
fig, ax = plt.subplots(figsize = (12, 7))

df1 = 10
df2 = 2
f_pdf = sp.stats.f.pdf(x, dfn = df1, dfd = df2)
ax.plot(x, f_pdf, lw =3, label = '$df_1 = %.d, df_2 = %.d$' %(df1, df2))

df1 = 2
df2 = 10
f_pdf = sp.stats.f.pdf(x, dfn = df1, dfd = df2)
ax.plot(x, f_pdf, lw =3, label = '$df_1 = %.d, df_2 = %.d$ '%(df1, df2))

df1 = 8
df2 = 15
f_pdf = sp.stats.f.pdf(x, dfn = df1, dfd = df2)
ax.plot(x, f_pdf, lw =3, color = 'red', label = '$df_1 = %.d, df_2 = %.d$' %(df1, df2))

df1 = 15
df2 = 8
f_pdf = sp.stats.f.pdf(x, dfn = df1, dfd = df2)
ax.plot(x, f_pdf, lw =3, label = '$df_1 = %.d, df_2 = %.d$ '%(df1, df2))

ax.legend(fontsize = 15)
ax.axis([0, 4, 0, .8])
ax.set_title("$F$ Distribution", size = 18)
plt.show()
```

# Single Restriction

Any linear restriction, such as $\beta_1 = 5$, $\beta_1 =2{\beta_2}$ can be tested, however, no loss of generality if linear restriction of $\beta_2=0$ is demonstrated.

We first take a look at single restriction, see how FWL regression can help to construct statistic tests.

The regression model is

$$
\boldsymbol{y} = \boldsymbol{X}_1\boldsymbol{\beta}_1+\beta_2\boldsymbol{x}_2+\boldsymbol{u}
$$

where $\boldsymbol{x}_2$ is an $n$-vector, whereas $\boldsymbol{X}_1$ is an $n\times (m-1)$ matrix.

Project $\boldsymbol{X}_1$ off $\boldsymbol{x}_2$, the FWL regression is

$$
\boldsymbol{M}_1\boldsymbol{y} = \beta_2\boldsymbol{M}_1\boldsymbol{x}_2 + \boldsymbol{M}_1\boldsymbol{u}
$$

Applying OLS estimate and variance formula,

$$
\hat{\beta}_2 = [(\boldsymbol{M}_1\boldsymbol{x}_2)^T\boldsymbol{M}_1\boldsymbol{x}_2]^{-1}(\boldsymbol{M}_1\boldsymbol{x}_2)^T\boldsymbol{M}_1\boldsymbol{y}=(\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{x}_2)^{-1}\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{y}=\frac{\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{y}}{\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{x}_2}\\
\text{Var}(\hat{\beta}_2) = \sigma^2 [(\boldsymbol{M}_1\boldsymbol{x}_2)^T(\boldsymbol{M}_1\boldsymbol{x}_2)]^{-1} =  \sigma^2(\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{x}_2)^{-1}
$$

If null hypothesis is $\beta_2^0=0$, also assuming that $\sigma^2$ is known. Construct a $z$-statistic

$$
z_{\beta_2} = \frac{\hat{\beta}_2}
{\sqrt{\text{Var}(\hat{\beta}_2)}}=\frac{\frac{\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{y}}{\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{x}_2}}{ \sigma(\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{x}_2)^{-\frac{1}{2}}}
= \frac{\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{y}}{\sigma(\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{x}_2)^{\frac{1}{2}}}
$$

However, $\sigma$ is not likely to be known, hence we replace it with $s$, the least square standard error estimator. Recall that

$$
s^2 =\frac{1}{n-k} \sum_{t=1}^n u_t^2 = \frac{\boldsymbol{u}^T\boldsymbol{u}}{n-k}= \frac{(\boldsymbol{M_X y})^T\boldsymbol{M_X y}}{n-k}=\frac{\boldsymbol{y}^T\boldsymbol{M_X y}}{n-k}
$$

Replace the $\sigma$ in $z_{\beta_2}$, we obtain $t$-statistic

$$
t_{\beta_2} =  \frac{\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{y}}{s(\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{x}_2)^{\frac{1}{2}}} = \left(\frac{\boldsymbol{y}^T\boldsymbol{M_X y}}{n-k}\right)^{-\frac{1}{2}}\frac{\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{y}}{(\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{x}_2)^{\frac{1}{2}}}
$$

Of course we can also show it indeed follows the $t$ distribution by using the definition that the ratio of standard normal variable to a $\chi^2$ variable. However, this is unnecessary for us.

#  Multiple Restrictions

Multiple restrictions test can be formulated as following

$$
H_0:\quad \boldsymbol{y} = \boldsymbol{X}_1\boldsymbol{\beta_1} + \boldsymbol{u}\\
H_1:\quad \boldsymbol{y} = \boldsymbol{X}_1\boldsymbol{\beta_1} +\boldsymbol{X}_2\boldsymbol{\beta_2}+ \boldsymbol{u}
$$

$H_0$ is restricted model, $H_1$ is unrestricted. And we denote restricted residual sum of squares as $\text{RRSS}$, and unrestricted residual sum of squares as $\text{URSS}$, the test statistic is

$$
F_{\beta_2}= \frac{(\text{RRSS}-\text{URSS})/r}{\text{URSS}/(n-k)}
$$

where $r=k_2$, the number of restrictions on $\beta_2$.

Using FWL regression with $\boldsymbol{M}_1$ and $\text{TSS = ESS + RSS}$,

$$
\boldsymbol{M}_1\boldsymbol{y}=\boldsymbol{M}_1\boldsymbol{X}_2\boldsymbol{\beta}_2+\boldsymbol{u} \tag{FWL regression}
$$

\begin{align}
\text{URSS}& =(\boldsymbol{M_1y})^T\boldsymbol{M_1y}- \underbrace{\left[\boldsymbol{M}_1\boldsymbol{X}_2[(\boldsymbol{M}_1\boldsymbol{X}_2)^T\boldsymbol{M}_1\boldsymbol{X}_2]^{-1}(\boldsymbol{M}_1\boldsymbol{X}_2)^T\boldsymbol{y}\right]^T}_{\text{projection matrix}\ P}\boldsymbol{y}\\
 &= \boldsymbol{y}^T\boldsymbol{M}_1\boldsymbol{y} -\boldsymbol{y}^T\boldsymbol{M}_1\boldsymbol{X}_2(\boldsymbol{X}_2^T\boldsymbol{M}_1\boldsymbol{X}_2)^{-1}\boldsymbol{X}^T_2\boldsymbol{M}_1 \boldsymbol{y} = \boldsymbol{y}^T\boldsymbol{M_X}\boldsymbol{y}
\end{align}

$$
\text{RRSS} = (\boldsymbol{M}_1y)^T\boldsymbol{M}_1y = \boldsymbol{y}^T\boldsymbol{M}_1\boldsymbol{y}
$$

Therefore
$$
\text{RRSS}-\text{URSS}=\boldsymbol{y}^T\boldsymbol{M}_1\boldsymbol{X}_2(\boldsymbol{X}_2^T\boldsymbol{M}_1\boldsymbol{X}_2)^{-1}\boldsymbol{X}^T_2\boldsymbol{M}_1 \boldsymbol{y}
$$

We have all parts of $F$ statistic, combine them

$$
F_{\boldsymbol{\beta}_{2}}=\frac{\boldsymbol{y}^{T} \boldsymbol{M}_{1} \boldsymbol{X}_{2}\left(\boldsymbol{X}_{2}^{T} \boldsymbol{M}_{1} \boldsymbol{X}_{2}\right)^{-1} \boldsymbol{X}_{2}^{T} \boldsymbol{M}_{1} \boldsymbol{y} / r}{\boldsymbol{y}^{T} \boldsymbol{M}_{\boldsymbol{X}} \boldsymbol{y} /(n-k)}
$$

In contrast, $t$ statistic will be
$$
t_{\beta_{2}}=\sqrt{\frac{\boldsymbol{y}^{\top} \boldsymbol{M}_{1} \boldsymbol{x}_{2}\left(\boldsymbol{x}_{2}^{\top} \boldsymbol{M}_{1} \boldsymbol{x}_{2}\right)^{-1} \boldsymbol{x}_{2}^{\top} \boldsymbol{M}_{1} \boldsymbol{y}}{\boldsymbol{y}^{\top} \boldsymbol{M}_{\boldsymbol{X}} \boldsymbol{y} /(n-k)}}
$$

To test the equality of two parameter vectors, we modify the $F$ test as
$$
F_{\gamma}=\frac{\left(\mathrm{RRSS}-\mathrm{RSS}_{1}-\mathrm{RSS}_{2}\right) / k}{\left(\mathrm{RSS}_{1}+\mathrm{RSS}_{2}\right) /(n-2 k)}
$$
for example, the sample being divided into two subsamples, to compare the stability of two subsamples, this so-called **Chow test** is a common practice.

#  Asymptotic Theory

**Asymptotic Theory** is concerned with the distribution of estimators and test statistics as the sample size $n$ tends to infinity.

##  Law of Large Numbers

The widely-known **Law of Large Numbers** ($\text{LLN}$) takes the form
$$
\bar{x} = \frac{1}{n}\sum_{t=1}^nx_t
$$
where $x_t$ are independent variables with its own bounded variance $\sigma_t^2$ and a common mean $\mu$. $\text{LLN}$ tells that as $n\rightarrow \infty$, then $\bar{x} \rightarrow \mu$.

The **Fundamental Theorem of Statistics** can be proved with $\text{LLN}$.

An **empirical distribution function** ($\text{EDF}$) can be expressed as
$$
\hat{F}(x) \equiv \frac{1}{n} \sum_{t=1}^{n} I\left(x_{t} \leq x\right)
$$
where $I(\cdot)$ is an **indicator function**, which takes value $1$ when its argument is true, otherwise $0$. To prove the Fundamental Theorem of Statistics, we invoke the $\text{LLN}$, expand the expectation

\begin{aligned}
\mathrm{E}\left(I\left(x_{t} \leq x\right)\right) &=0 \cdot \operatorname{Pr}\left(I\left(x_{t} \leq x\right)=0\right)+1 \cdot \operatorname{Pr}\left(I\left(x_{t} \leq x\right)=1\right) \\
&=\operatorname{Pr}\left(I\left(x_{t} \leq x\right)=1\right)=\operatorname{Pr}\left(x_{t} \leq x\right)=F(x)
\end{aligned}

It turns out that $\hat{F}(x)$ is a consistent estimator of $F(x)$.

##  Central Limit Theorems

The **Central Limit Theorems** ($\text{CLT}$) tells that $\frac{1}{n}$ times the sum of $n$ centered random variables will approximately follow a normal distribution when $n$ is sufficiently large. The most well $\text{CLT}$ is **Lindeberg-Lévy** $\text{CLT}$, the quantity
$$
z \equiv \frac{1}{\sqrt{n}} \sum_{t=1}^{n} \frac{x_{t}-\mu}{\sigma}
$$
is **asymptotically distributed** as $N(0,1)$.

We won't bother to prove them, but they are the implicit theoretical foundation when we are discussing **simulation-based tests**.

#  Simulated $P$ Values

```python
t_stats_real_model = betahat.T[0]/np.sqrt(cov_betahat_prin_diag)
print('t statistics: {}'.format(t_stats))
p_values_real_model = 2*(1-sp.stats.t.cdf(t_stats, df=n-k, loc=0, scale=1))
print('p_values: {}'.format(p_values))
```

    t statistics: [1.75043396 2.79293141 2.73485916]
    p_values: [0.25890776 0.00241733 0.07849986]

Above will be our imaginary real data.

```python

```

```python

```

```python

```

```python
indicator_array = np.array([0, 0, 0])
sim_rounds = 5000
for i in range(sim_rounds):
    u = lr.gen_u(n)
    y = X@beta + 15*u
    betahat = lr.ols(y, X)
    resid = y - X@betahat
    cov_betahat, cov_betahat_prin_diag = lr.cov_beta_hat(resid, k, X)
    t_stats_sim = betahat.T[0]/np.sqrt(cov_betahat_prin_diag)
    indicator = t_stats_sim - t_stats_real_model
    indicator[indicator < 0] = 0
    indicator[indicator > 0] = 1
    indicator_array = np.vstack([indicator_array, indicator])
indicator_array = indicator_array[1:]
```

```python
np.sum(indicator_array, axis = 0)/sim_rounds
```

    array([0.6924, 0.288 , 0.3778])

```python
indicator
```

    array([1., 1., 0.])

```python

```

```python
comparison = p_values - A
comparison
```

    array([-0.04109224,  0.00141733,  0.02849986])

```python
comparison[comparison<0] = 0
```

```python
comparison[comparison>0] = 1
```

```python
np.vstack([comparison, comparison1])
```

    array([[0., 1., 1.],
           [1., 2., 3.]])

```python

```

Mathematics/Statistics/0.0 Basic_Statistics_With_Python/Chapter 3 - Hypothesis Test.ipynb


# This notebook shows how to extract a model from a Latex document and simulate the mode.

## Why specify a model in Latex?

Sometime the **implementation** of a model in software don't match the **specification** of the model in
the text in which the model is presented. It can be a challenge to make sure that the specification is
updated in order to reflect changes made in the implementation.

By extracting the model from a Latex script which describes and specify the model a one can always be sure that simulations reflect the model as described in the paper.

Also the author is forced to make a complete specification of the model, else it won't run.

## The Economic Credit Loss model
This jupyter notebook is inspired by IMF working paper (WP/20/111) The Expected Credit Loss Modeling from a Top-Down Stress Testing Perspective by Marco Gross, Dimitrios Laliotis, Mindaugas Leika, Pavel Lukyantsau. The working paper and the associated material is located https://www.imf.org/en/Publications/WP/Issues/2020/07/03/Expected-Credit-Loss-Modeling-from-a-Top-Down-Stress-Testing-Perspective-49545

from the abstract of the paper:
> The objective of this paper is to present an integrated tool suite for IFRS 9- and CECL compatible
estimation in top-down solvency stress tests. The tool suite serves as an
illustration for institutions wishing to include accounting-based approaches for credit risk
modeling in top-down stress tests.

This is a jupyter notebook build with the purpose of illustrating the conversion of a model from Latex to ModelFlow. The purpose is testing so take very much care.

## Import libraries

```python
%load_ext autoreload
%autoreload 2
import pandas as pd
from IPython.core.display import HTML,Markdown

from modelclass import model
import modeljupytermagic

# some useful stuf
model.widescreen()
pd.set_option('display.max_rows', None, 'display.max_columns', 10, 'precision', 2)
sortdf = lambda df: df[sorted([c for c in df.columns])]
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

## Write a latex script
The model consists of the equations and the lists

The jupyter magic command **%%latexflow** will extract the model. then it will  transform the equations to **ModelFlow** equations and finaly it will create a modelflow **model** instance.

The **model** instance will be able to solve the model.

```python
%%latexflow ecl

### Loans can be in 3 stages
Loans can be in 3 stages, s1,s2 and s3.
New loans will be generated and loans will mature.

Two lists of stages are defined:

List $stage=\{s1, s2,s3\}$

List $stage\_from=\{s1, s2,s3\}$

### A share of the loans in each stage wil transition to the same or another stage in the next time frame:

\begin{equation}
\label{eq:loanfromto}
loan\_transition\_from\_to^{stage\_from,stage}_{t} = loan^{stage\_from}_{t-1} \times TR^{stage\_from,stage}_{t}
\end{equation}

\begin{equation}
\label{eq:transition}
loan\_transition\_to^{stage}_{t}  = \underbrace{\sum_{stage\_from}(loan\_transition\_from\_to^{stage\_from,stage}_{t})}_{Transition}
\end{equation}

### A share of the loans in each stage will mature another share will have to be written off

\begin{equation}
\label{eq:maturing}
loan\_maturing^{stage}_{t}  = M^{stage}_{t} \times loan^{stage}_{t-1}
\end{equation}

\begin{equation}
\label{eq:writeoff}
loan\_writeoff^{stage}_{t}  = WRO^{stage}_{t} \times loan^{stage}_{t-1}
\end{equation}

### So loans in a stage will reflect the inflow and outflow

\begin{equation}
\label{eq:E}
loan^{stage}_{t}  = \underbrace{loan\_transition\_to^{stage}_{t} }_{Transition}
-\underbrace{loan\_maturing^{stage}_{t}}_{Maturing}
-\underbrace{loan\_writeoff^{stage}_{t}}_{Writeoff}
+\underbrace{loan\_new^{stage}_{t}}_{New loans}
\end{equation}

\begin{equation}
\label{eq:new}
loan\_new^{stage}_{t}  = new^{stage}_{t} \times loan^{stage}_{t-1}
\end{equation}

\begin{equation}
\label{eq:g}
loan\_total_{t}  =  \sum_{stage}(loan^{stage}_{t})
\end{equation}

### New loans are only generated in stage 1.

\begin{equation}
\label{eq:E2}
new\_s1_{t} = \frac{loan\_growth_{t} \times loan\_total_{t-1} +
                \sum_{stage}((M^{stage}_{t}+WRO^{stage}_{t})\times loan^{stage}_{t-1})}{(loan\_s1_{t-1})}
\end{equation}

### Performing Loans

\begin{equation}
\label{eq:Performing}
loan\_performing_{t} = loan\_s1_{t}+loan\_s2_{t}
\end{equation}

### Cure

\begin{equation}
\label{eq:cure}
loan\_cure = loan\_transition\_from\_to\_s3\_s1+loan\_transition\_from\_to\_s3\_s2
\end{equation}

### Probability of default (PD)

The point in time PD is the fraction of loans in stage s1 and s2 going into stage s3
\begin{equation}
\label{eq:PDPIT}
PD\_pit= \frac{loan\_transition\_from\_to\_s1\_s3+loan\_transition\_from\_to\_s2\_s3}{loan\_s1+loan\_s2}
\end{equation}

The Troug The  Cycle PD is a slow mowing average of the Point in time PD. The

\begin{equation}
\label{eq:PDTTC}
PD\_TTC = logit^{-1}(logit(PD\_TTC(-1)) + alfa \times \Delta{PD\_pit})
\end{equation}

### And we can specify the dynamic of the transition matrix, based on Z score

Let $\Phi$ be the normal cumulative distribution $\frac{1}{\sqrt{2\pi}}
\int_{-\infty}^x e^{
-\frac{t^2}{2}}dt$

\begin{equation}
\label{eq:tr}
TR^{stage\_from,stage}_{t} = \Phi{\left(\frac{bound\_upper^{stage\_from,stage}-\sqrt{\rho}\times Z_{t}}{\sqrt{1-\rho}}\right)}-\Phi{\left(\frac{bound\_lower^{stage\_from,stage}-\sqrt{\rho}\times Z_{t}}{\sqrt{1-\rho}}\right)}
\end{equation}

```

# Now creating the model **ecl**

## The model

### Loans can be in 3 stages
Loans can be in 3 stages, s1,s2 and s3.
New loans will be generated and loans will mature.

Two lists of stages are defined:

List $stage=\{s1, s2,s3\}$

List $stage\_from=\{s1, s2,s3\}$

### A share of the loans in each stage wil transition to the same or another stage in the next time frame:

\begin{equation}
\label{eq:loanfromto}
loan\_transition\_from\_to^{stage\_from,stage}_{t} = loan^{stage\_from}_{t-1} \times TR^{stage\_from,stage}_{t}
\end{equation}

\begin{equation}
\label{eq:transition}
loan\_transition\_to^{stage}_{t}  = \underbrace{\sum_{stage\_from}(loan\_transition\_from\_to^{stage\_from,stage}_{t})}_{Transition}
\end{equation}

### A share of the loans in each stage will mature another share will have to be written off

\begin{equation}
\label{eq:maturing}
loan\_maturing^{stage}_{t}  = M^{stage}_{t} \times loan^{stage}_{t-1}
\end{equation}

\begin{equation}
\label{eq:writeoff}
loan\_writeoff^{stage}_{t}  = WRO^{stage}_{t} \times loan^{stage}_{t-1}
\end{equation}

### So loans in a stage will reflect the inflow and outflow

\begin{equation}
\label{eq:E}
loan^{stage}_{t}  = \underbrace{loan\_transition\_to^{stage}_{t} }_{Transition}
-\underbrace{loan\_maturing^{stage}_{t}}_{Maturing}
-\underbrace{loan\_writeoff^{stage}_{t}}_{Writeoff}
+\underbrace{loan\_new^{stage}_{t}}_{New loans}
\end{equation}

\begin{equation}
\label{eq:new}
loan\_new^{stage}_{t}  = new^{stage}_{t} \times loan^{stage}_{t-1}
\end{equation}

\begin{equation}
\label{eq:g}
loan\_total_{t}  =  \sum_{stage}(loan^{stage}_{t})
\end{equation}

### New loans are only generated in stage 1.

\begin{equation}
\label{eq:E2}
new\_s1_{t} = \frac{loan\_growth_{t} \times loan\_total_{t-1} +
                \sum_{stage}((M^{stage}_{t}+WRO^{stage}_{t})\times loan^{stage}_{t-1})}{(loan\_s1_{t-1})}
\end{equation}

### Performing Loans

\begin{equation}
\label{eq:Performing}
loan\_performing_{t} = loan\_s1_{t}+loan\_s2_{t}
\end{equation}

### Cure

\begin{equation}
\label{eq:cure}
loan\_cure = loan\_transition\_from\_to\_s3\_s1+loan\_transition\_from\_to\_s3\_s2
\end{equation}

### Probability of default (PD)

The point in time PD is the fraction of loans in stage s1 and s2 going into stage s3
\begin{equation}
\label{eq:PDPIT}
PD\_pit= \frac{loan\_transition\_from\_to\_s1\_s3+loan\_transition\_from\_to\_s2\_s3}{loan\_s1+loan\_s2}
\end{equation}

The Troug The  Cycle PD is a slow mowing average of the Point in time PD. The

\begin{equation}
\label{eq:PDTTC}
PD\_TTC = logit^{-1}(logit(PD\_TTC(-1)) + alfa \times \Delta{PD\_pit})
\end{equation}

### And we can specify the dynamic of the transition matrix, based on Z score

Let $\Phi$ be the normal cumulative distribution $\frac{1}{\sqrt{2\pi}}
\int_{-\infty}^x e^{
-\frac{t^2}{2}}dt$

\begin{equation}
\label{eq:tr}
TR^{stage\_from,stage}_{t} = \Phi{\left(\frac{bound\_upper^{stage\_from,stage}-\sqrt{\rho}\times Z_{t}}{\sqrt{1-\rho}}\right)}-\Phi{\left(\frac{bound\_lower^{stage\_from,stage}-\sqrt{\rho}\times Z_{t}}{\sqrt{1-\rho}}\right)}
\end{equation}

## The equations in Macro business logic language can be inspected

```python
print(ecl.equations_original)
```

    Do stage_from $ Do stage $
     Frml loanfromto loan_transition_from_to_{stage_from}_{stage}  = loan_{stage_from}(-1)*TR_{stage_from}_{stage} $
    enddo $ enddo $
    Do stage $
     Frml transition loan_transition_to_{stage}   = sum(stage_from,loan_transition_from_to_{stage_from}_{stage}) $
    enddo $
    Do stage $
     Frml maturing loan_maturing_{stage}   = M_{stage}*loan_{stage}(-1) $
    enddo $
    Do stage $
     Frml writeoff loan_writeoff_{stage}   = WRO_{stage}*loan_{stage}(-1) $
    enddo $
    Do stage $
     Frml E loan_{stage}   = loan_transition_to_{stage}-loan_maturing_{stage}-loan_writeoff_{stage}+loan_new_{stage} $
    enddo $
    Do stage $
     Frml new loan_new_{stage}   = new_{stage}*loan_{stage}(-1) $
    enddo $
    Frml g loan_total   = sum(stage,loan_{stage}) $
    Frml E2 new_s1  = ((loan_growth*loan_total(-1)+sum(stage,(M_{stage}+WRO_{stage})*loan_{stage}(-1)))/((loan_s1(-1)))) $
    Frml Performing loan_performing  = loan_s1+loan_s2 $
    Frml cure loan_cure  = loan_transition_from_to_s3_s1+loan_transition_from_to_s3_s2 $
    Frml PDPIT PD_pit = ((loan_transition_from_to_s1_s3+loan_transition_from_to_s2_s3)/(loan_s1+loan_s2)) $
    Frml PDTTC PD_TTC  = logit_inverse(logit(PD_TTC(-1))+alfa*diff(PD_pit)) $
    Do stage_from $ Do stage $
     Frml tr TR_{stage_from}_{stage}  = NORM.CDF((((bound_upper_{stage_from}_{stage}-sqrt(rho)*Z)/(sqrt(1-rho)))))-NORM.CDF((((bound_lower_{stage_from}_{stage}-sqrt(rho)*Z)/(sqrt(1-rho))))) $
    enddo $ enddo $
    LIST STAGE =  STAGE : S1  S2 S3$
    LIST STAGE_FROM =  STAGE_FROM : S1  S2 S3$

### The equations in business logic language can be inspected

```python
print(ecl.equations)
```

    FRML LOANFROMTO LOAN_TRANSITION_FROM_TO_S1_S1  = LOAN_S1(-1)*TR_S1_S1 $
    FRML LOANFROMTO LOAN_TRANSITION_FROM_TO_S1_S2  = LOAN_S1(-1)*TR_S1_S2 $
    FRML LOANFROMTO LOAN_TRANSITION_FROM_TO_S1_S3  = LOAN_S1(-1)*TR_S1_S3 $
    FRML LOANFROMTO LOAN_TRANSITION_FROM_TO_S2_S1  = LOAN_S2(-1)*TR_S2_S1 $
    FRML LOANFROMTO LOAN_TRANSITION_FROM_TO_S2_S2  = LOAN_S2(-1)*TR_S2_S2 $
    FRML LOANFROMTO LOAN_TRANSITION_FROM_TO_S2_S3  = LOAN_S2(-1)*TR_S2_S3 $
    FRML LOANFROMTO LOAN_TRANSITION_FROM_TO_S3_S1  = LOAN_S3(-1)*TR_S3_S1 $
    FRML LOANFROMTO LOAN_TRANSITION_FROM_TO_S3_S2  = LOAN_S3(-1)*TR_S3_S2 $
    FRML LOANFROMTO LOAN_TRANSITION_FROM_TO_S3_S3  = LOAN_S3(-1)*TR_S3_S3 $
    FRML TRANSITION LOAN_TRANSITION_TO_S1   = (LOAN_TRANSITION_FROM_TO_S1_S1+LOAN_TRANSITION_FROM_TO_S2_S1+LOAN_TRANSITION_FROM_TO_S3_S1) $
    FRML TRANSITION LOAN_TRANSITION_TO_S2   = (LOAN_TRANSITION_FROM_TO_S1_S2+LOAN_TRANSITION_FROM_TO_S2_S2+LOAN_TRANSITION_FROM_TO_S3_S2) $
    FRML TRANSITION LOAN_TRANSITION_TO_S3   = (LOAN_TRANSITION_FROM_TO_S1_S3+LOAN_TRANSITION_FROM_TO_S2_S3+LOAN_TRANSITION_FROM_TO_S3_S3) $
    FRML MATURING LOAN_MATURING_S1   = M_S1*LOAN_S1(-1) $
    FRML MATURING LOAN_MATURING_S2   = M_S2*LOAN_S2(-1) $
    FRML MATURING LOAN_MATURING_S3   = M_S3*LOAN_S3(-1) $
    FRML WRITEOFF LOAN_WRITEOFF_S1   = WRO_S1*LOAN_S1(-1) $
    FRML WRITEOFF LOAN_WRITEOFF_S2   = WRO_S2*LOAN_S2(-1) $
    FRML WRITEOFF LOAN_WRITEOFF_S3   = WRO_S3*LOAN_S3(-1) $
    FRML E LOAN_S1   = LOAN_TRANSITION_TO_S1-LOAN_MATURING_S1-LOAN_WRITEOFF_S1+LOAN_NEW_S1 $
    FRML E LOAN_S2   = LOAN_TRANSITION_TO_S2-LOAN_MATURING_S2-LOAN_WRITEOFF_S2+LOAN_NEW_S2 $
    FRML E LOAN_S3   = LOAN_TRANSITION_TO_S3-LOAN_MATURING_S3-LOAN_WRITEOFF_S3+LOAN_NEW_S3 $
    FRML NEW LOAN_NEW_S1   = NEW_S1*LOAN_S1(-1) $
    FRML NEW LOAN_NEW_S2   = NEW_S2*LOAN_S2(-1) $
    FRML NEW LOAN_NEW_S3   = NEW_S3*LOAN_S3(-1) $
    FRML G LOAN_TOTAL   = (LOAN_S1+LOAN_S2+LOAN_S3) $
    FRML E2 NEW_S1  = ((LOAN_GROWTH*LOAN_TOTAL(-1)+((M_S1+WRO_S1)*LOAN_S1(-1)+(M_S2+WRO_S2)*LOAN_S2(-1)+(M_S3+WRO_S3)*LOAN_S3(-1)))/((LOAN_S1(-1)))) $
    FRML PERFORMING LOAN_PERFORMING  = LOAN_S1+LOAN_S2 $
    FRML CURE LOAN_CURE  = LOAN_TRANSITION_FROM_TO_S3_S1+LOAN_TRANSITION_FROM_TO_S3_S2 $
    FRML PDPIT PD_PIT = ((LOAN_TRANSITION_FROM_TO_S1_S3+LOAN_TRANSITION_FROM_TO_S2_S3)/(LOAN_S1+LOAN_S2)) $
    FRML PDTTC PD_TTC  = LOGIT_INVERSE(LOGIT(PD_TTC(-1))+ALFA*((PD_PIT)-(PD_PIT(-1)))) $
    FRML TR TR_S1_S1  = NORM.CDF((((BOUND_UPPER_S1_S1-SQRT(RHO)*Z)/(SQRT(1-RHO)))))-NORM.CDF((((BOUND_LOWER_S1_S1-SQRT(RHO)*Z)/(SQRT(1-RHO))))) $
    FRML TR TR_S1_S2  = NORM.CDF((((BOUND_UPPER_S1_S2-SQRT(RHO)*Z)/(SQRT(1-RHO)))))-NORM.CDF((((BOUND_LOWER_S1_S2-SQRT(RHO)*Z)/(SQRT(1-RHO))))) $
    FRML TR TR_S1_S3  = NORM.CDF((((BOUND_UPPER_S1_S3-SQRT(RHO)*Z)/(SQRT(1-RHO)))))-NORM.CDF((((BOUND_LOWER_S1_S3-SQRT(RHO)*Z)/(SQRT(1-RHO))))) $
    FRML TR TR_S2_S1  = NORM.CDF((((BOUND_UPPER_S2_S1-SQRT(RHO)*Z)/(SQRT(1-RHO)))))-NORM.CDF((((BOUND_LOWER_S2_S1-SQRT(RHO)*Z)/(SQRT(1-RHO))))) $
    FRML TR TR_S2_S2  = NORM.CDF((((BOUND_UPPER_S2_S2-SQRT(RHO)*Z)/(SQRT(1-RHO)))))-NORM.CDF((((BOUND_LOWER_S2_S2-SQRT(RHO)*Z)/(SQRT(1-RHO))))) $
    FRML TR TR_S2_S3  = NORM.CDF((((BOUND_UPPER_S2_S3-SQRT(RHO)*Z)/(SQRT(1-RHO)))))-NORM.CDF((((BOUND_LOWER_S2_S3-SQRT(RHO)*Z)/(SQRT(1-RHO))))) $
    FRML TR TR_S3_S1  = NORM.CDF((((BOUND_UPPER_S3_S1-SQRT(RHO)*Z)/(SQRT(1-RHO)))))-NORM.CDF((((BOUND_LOWER_S3_S1-SQRT(RHO)*Z)/(SQRT(1-RHO))))) $
    FRML TR TR_S3_S2  = NORM.CDF((((BOUND_UPPER_S3_S2-SQRT(RHO)*Z)/(SQRT(1-RHO)))))-NORM.CDF((((BOUND_LOWER_S3_S2-SQRT(RHO)*Z)/(SQRT(1-RHO))))) $
    FRML TR TR_S3_S3  = NORM.CDF((((BOUND_UPPER_S3_S3-SQRT(RHO)*Z)/(SQRT(1-RHO)))))-NORM.CDF((((BOUND_LOWER_S3_S3-SQRT(RHO)*Z)/(SQRT(1-RHO))))) $
    LIST STAGE =  STAGE : S1  S2 S3$
    LIST STAGE_FROM =  STAGE_FROM : S1  S2 S3$

### The model structure can be inspected

```python
ecl.drawmodel(sink='LOAN_TOTAL',HR=0,pdf=0,att=0,size=(12,12))
```

## load data
The data is copy-pasted from the excel sheet
### Load data common for baseline and adverse

```python
%%dataframe T0tr noshow  prefix=TR_  periods=1 melt
	S1	S2	S3
S1	89%	9%	2%
S2	15%	79%	6%
S3	2%	24%	74%
```

```python
%%dataframe  startvalues  show  periods=1
     pd_pit lgd_pit
        1.6%   20%
```

      PD_PIT
      LGD_PIT

      index

      2021
      0.02
      0.2

```python
%%dataframe  Loan_t0  show periods=1 melt
S1	S2	S3 total
loan 500	180	30 710

```

      S1
      S2
      S3
      TOTAL

      LOAN
      500.0
      180.0
      30.0
      710.0

      var_name
      LOAN_S1
      LOAN_S2
      LOAN_S3
      LOAN_TOTAL

      index

      2021
      500.0
      180.0
      30.0
      710.0

```python
%%dataframe upper_bin prefix=bound_upper_ nshow periods=7 melt
	S1	S2	S3
S1	10000	-1,34	-2,33
S2	10000	1,08	-1,64
S3	10000	1,88	0,58

```

```python
%%dataframe lower_bin prefix=bound_lower_ show periods=7 melt
	S1	S2	S3
S1	-1,34	-2,33	-10000
S2	1,08	-1,64	-10000
S3	1,88	0,58	-10000
```

      S1
      S2
      S3

      S1
      -1.34
      -2.33
      -10000.0

      S2
      1.08
      -1.64
      -10000.0

      S3
      1.88
      0.58
      -10000.0

      var_name
      BOUND_LOWER_S1_S1
      BOUND_LOWER_S2_S1
      BOUND_LOWER_S3_S1
      BOUND_LOWER_S1_S2
      BOUND_LOWER_S2_S2
      BOUND_LOWER_S3_S2
      BOUND_LOWER_S1_S3
      BOUND_LOWER_S2_S3
      BOUND_LOWER_S3_S3

      index

      2021
      -1.34
      1.08
      1.88
      -2.33
      -1.64
      0.58
      -10000.0
      -10000.0
      -10000.0

      2022
      -1.34
      1.08
      1.88
      -2.33
      -1.64
      0.58
      -10000.0
      -10000.0
      -10000.0

      2023
      -1.34
      1.08
      1.88
      -2.33
      -1.64
      0.58
      -10000.0
      -10000.0
      -10000.0

      2024
      -1.34
      1.08
      1.88
      -2.33
      -1.64
      0.58
      -10000.0
      -10000.0
      -10000.0

      2025
      -1.34
      1.08
      1.88
      -2.33
      -1.64
      0.58
      -10000.0
      -10000.0
      -10000.0

      2026
      -1.34
      1.08
      1.88
      -2.33
      -1.64
      0.58
      -10000.0
      -10000.0
      -10000.0

      2027
      -1.34
      1.08
      1.88
      -2.33
      -1.64
      0.58
      -10000.0
      -10000.0
      -10000.0

```python
%%dataframe parameters show periods=7
rho
0.2
```

      RHO

      index

      2021
      0.2

      2022
      0.2

      2023
      0.2

      2024
      0.2

      2025
      0.2

      2026
      0.2

      2027
      0.2

### Create the Static dataframe, common to scenarios

```python
staticdf = pd.concat([T0tr_melted,Loan_t0_melted,startvalues,upper_bin_melted,
                      lower_bin_melted,parameters],axis=1)
HTML(staticdf.T.style.render().replace('nan',''))
```

        index	        2021	        2022	        2023	        2024	        2025	        2026	        2027

                        TR_S1_S1
                        0.89

                        TR_S2_S1
                        0.15

                        TR_S3_S1
                        0.02

                        TR_S1_S2
                        0.09

                        TR_S2_S2
                        0.79

                        TR_S3_S2
                        0.24

                        TR_S1_S3
                        0.02

                        TR_S2_S3
                        0.06

                        TR_S3_S3
                        0.74

                        LOAN_S1
                        500.00

                        LOAN_S2
                        180.00

                        LOAN_S3
                        30.00

                        LOAN_TOTAL
                        710.00

                        PD_PIT
                        0.02

                        LGD_PIT
                        0.20

                        BOUND_UPPER_S1_S1
                        10000.00
                        10000.00
                        10000.00
                        10000.00
                        10000.00
                        10000.00
                        10000.00

                        BOUND_UPPER_S2_S1
                        10000.00
                        10000.00
                        10000.00
                        10000.00
                        10000.00
                        10000.00
                        10000.00

                        BOUND_UPPER_S3_S1
                        10000.00
                        10000.00
                        10000.00
                        10000.00
                        10000.00
                        10000.00
                        10000.00

                        BOUND_UPPER_S1_S2
                        -1.34
                        -1.34
                        -1.34
                        -1.34
                        -1.34
                        -1.34
                        -1.34

                        BOUND_UPPER_S2_S2
                        1.08
                        1.08
                        1.08
                        1.08
                        1.08
                        1.08
                        1.08

                        BOUND_UPPER_S3_S2
                        1.88
                        1.88
                        1.88
                        1.88
                        1.88
                        1.88
                        1.88

                        BOUND_UPPER_S1_S3
                        -2.33
                        -2.33
                        -2.33
                        -2.33
                        -2.33
                        -2.33
                        -2.33

                        BOUND_UPPER_S2_S3
                        -1.64
                        -1.64
                        -1.64
                        -1.64
                        -1.64
                        -1.64
                        -1.64

                        BOUND_UPPER_S3_S3
                        0.58
                        0.58
                        0.58
                        0.58
                        0.58
                        0.58
                        0.58

                        BOUND_LOWER_S1_S1
                        -1.34
                        -1.34
                        -1.34
                        -1.34
                        -1.34
                        -1.34
                        -1.34

                        BOUND_LOWER_S2_S1
                        1.08
                        1.08
                        1.08
                        1.08
                        1.08
                        1.08
                        1.08

                        BOUND_LOWER_S3_S1
                        1.88
                        1.88
                        1.88
                        1.88
                        1.88
                        1.88
                        1.88

                        BOUND_LOWER_S1_S2
                        -2.33
                        -2.33
                        -2.33
                        -2.33
                        -2.33
                        -2.33
                        -2.33

                        BOUND_LOWER_S2_S2
                        -1.64
                        -1.64
                        -1.64
                        -1.64
                        -1.64
                        -1.64
                        -1.64

                        BOUND_LOWER_S3_S2
                        0.58
                        0.58
                        0.58
                        0.58
                        0.58
                        0.58
                        0.58

                        BOUND_LOWER_S1_S3
                        -10000.00
                        -10000.00
                        -10000.00
                        -10000.00
                        -10000.00
                        -10000.00
                        -10000.00

                        BOUND_LOWER_S2_S3
                        -10000.00
                        -10000.00
                        -10000.00
                        -10000.00
                        -10000.00
                        -10000.00
                        -10000.00

                        BOUND_LOWER_S3_S3
                        -10000.00
                        -10000.00
                        -10000.00
                        -10000.00
                        -10000.00
                        -10000.00
                        -10000.00

                        RHO
                        0.20
                        0.20
                        0.20
                        0.20
                        0.20
                        0.20
                        0.20

### Load data specific for the scenarios

```python
%%dataframe  inf_baseline  nshow t periods=7 melt
    m     wro
s1  5%   0
s2  3.8%   0
s3  0    7.5%
```

```python
%%dataframe  inf_adverse  nshow t periods=7 melt
    m     wro
s1  3.8%   0
s2  2.5%   0
s3  0    6.3%
```

```python
%%dataframe projection_baseline nshow
Z  loan_growth
 0          0.01
-0,47	    0.01
-0,42	    0.01
-0,38	    0.01
-0,36	    0.01
-0,34	    0.01
-0,33	    0.01
```

```python
%%dataframe projection_adverse nshow
 Z        loan_growth
 0          0.01
  -0,65     -0.01
  -0,84     -0.008
   -0,99    -0.006
    -0,69   -0.004
    -0,39   -0.002
    -0,24   -0.0
```

### Create a dataframe for baseline and adverse scenario

```python
baseupdate    = pd.concat([inf_baseline_melted, projection_baseline],axis=1).pipe(sortdf)
adverseupdate =  pd.concat([inf_adverse_melted, projection_adverse],axis=1).pipe(sortdf)
getnotnul = lambda x: x.loc[:,(x!=0.0).any(axis=0)]

display(Markdown('## Baseline scenario'))
display(baseupdate.pipe(getnotnul))
display(Markdown('## Adverse scenario'))
display(adverseupdate.pipe(getnotnul))
```

## Baseline scenario

      LOAN_GROWTH
      M_S1
      M_S2
      WRO_S3
      Z

      index

      2021
      0.01
      0.05
      0.04
      0.07
      0.00

      2022
      0.01
      0.05
      0.04
      0.07
      -0.47

      2023
      0.01
      0.05
      0.04
      0.07
      -0.42

      2024
      0.01
      0.05
      0.04
      0.07
      -0.38

      2025
      0.01
      0.05
      0.04
      0.07
      -0.36

      2026
      0.01
      0.05
      0.04
      0.07
      -0.34

      2027
      0.01
      0.05
      0.04
      0.07
      -0.33

## Adverse scenario

      LOAN_GROWTH
      M_S1
      M_S2
      WRO_S3
      Z

      index

      2021
      1.00e-02
      0.04
      0.03
      0.06
      0.00

      2022
      -1.00e-02
      0.04
      0.03
      0.06
      -0.65

      2023
      -8.00e-03
      0.04
      0.03
      0.06
      -0.84

      2024
      -6.00e-03
      0.04
      0.03
      0.06
      -0.99

      2025
      -4.00e-03
      0.04
      0.03
      0.06
      -0.69

      2026
      -2.00e-03
      0.04
      0.03
      0.06
      -0.39

      2027
      -0.00e+00
      0.04
      0.03
      0.06
      -0.24

```python
baseline = pd.concat([staticdf, inf_baseline_melted, projection_baseline],axis=1)
adverse  = pd.concat([staticdf, inf_adverse_melted, projection_adverse],axis=1)
baseline.index = pd.period_range(start=2021,freq = 'Y',periods=7)
adverse.index  = pd.period_range(start=2021,freq = 'Y',periods=7)

display(baseline.pipe(sortdf).T)
```

      2021
      2022
      2023
      2024
      2025
      2026
      2027

      BOUND_LOWER_S1_S1
      -1.34
      -1.34
      -1.34
      -1.34
      -1.34
      -1.34
      -1.34

      BOUND_LOWER_S1_S2
      -2.33
      -2.33
      -2.33
      -2.33
      -2.33
      -2.33
      -2.33

      BOUND_LOWER_S1_S3
      -10000.00
      -10000.00
      -10000.00
      -10000.00
      -10000.00
      -10000.00
      -10000.00

      BOUND_LOWER_S2_S1
      1.08
      1.08
      1.08
      1.08
      1.08
      1.08
      1.08

      BOUND_LOWER_S2_S2
      -1.64
      -1.64
      -1.64
      -1.64
      -1.64
      -1.64
      -1.64

      BOUND_LOWER_S2_S3
      -10000.00
      -10000.00
      -10000.00
      -10000.00
      -10000.00
      -10000.00
      -10000.00

      BOUND_LOWER_S3_S1
      1.88
      1.88
      1.88
      1.88
      1.88
      1.88
      1.88

      BOUND_LOWER_S3_S2
      0.58
      0.58
      0.58
      0.58
      0.58
      0.58
      0.58

      BOUND_LOWER_S3_S3
      -10000.00
      -10000.00
      -10000.00
      -10000.00
      -10000.00
      -10000.00
      -10000.00

      BOUND_UPPER_S1_S1
      10000.00
      10000.00
      10000.00
      10000.00
      10000.00
      10000.00
      10000.00

      BOUND_UPPER_S1_S2
      -1.34
      -1.34
      -1.34
      -1.34
      -1.34
      -1.34
      -1.34

      BOUND_UPPER_S1_S3
      -2.33
      -2.33
      -2.33
      -2.33
      -2.33
      -2.33
      -2.33

      BOUND_UPPER_S2_S1
      10000.00
      10000.00
      10000.00
      10000.00
      10000.00
      10000.00
      10000.00

      BOUND_UPPER_S2_S2
      1.08
      1.08
      1.08
      1.08
      1.08
      1.08
      1.08

      BOUND_UPPER_S2_S3
      -1.64
      -1.64
      -1.64
      -1.64
      -1.64
      -1.64
      -1.64

      BOUND_UPPER_S3_S1
      10000.00
      10000.00
      10000.00
      10000.00
      10000.00
      10000.00
      10000.00

      BOUND_UPPER_S3_S2
      1.88
      1.88
      1.88
      1.88
      1.88
      1.88
      1.88

      BOUND_UPPER_S3_S3
      0.58
      0.58
      0.58
      0.58
      0.58
      0.58
      0.58

      LGD_PIT
      0.20
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      LOAN_GROWTH
      0.01
      0.01
      0.01
      0.01
      0.01
      0.01
      0.01

      LOAN_S1
      500.00
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      LOAN_S2
      180.00
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      LOAN_S3
      30.00
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      LOAN_TOTAL
      710.00
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      M_S1
      0.05
      0.05
      0.05
      0.05
      0.05
      0.05
      0.05

      M_S2
      0.04
      0.04
      0.04
      0.04
      0.04
      0.04
      0.04

      M_S3
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      PD_PIT
      0.02
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      RHO
      0.20
      0.20
      0.20
      0.20
      0.20
      0.20
      0.20

      TR_S1_S1
      0.89
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      TR_S1_S2
      0.09
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      TR_S1_S3
      0.02
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      TR_S2_S1
      0.15
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      TR_S2_S2
      0.79
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      TR_S2_S3
      0.06
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      TR_S3_S1
      0.02
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      TR_S3_S2
      0.24
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      TR_S3_S3
      0.74
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      WRO_S1
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      WRO_S2
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      WRO_S3
      0.07
      0.07
      0.07
      0.07
      0.07
      0.07
      0.07

      Z
      0.00
      -0.47
      -0.42
      -0.38
      -0.36
      -0.34
      -0.33

## Run baseline

```python
base_result  = ecl(baseline,keep='Baseline')
```

    Will start calculating: testmodel
    2022  solved
    2023  solved
    2024  solved
    2025  solved
    2026  solved
    2027  solved
    testmodel calculated

### Save model and baseline results

```python
ecl.modeldump('ecl.pcim')
```

## Run adverse

```python
adverse_result = ecl(adverse,keep = 'Adverse')
```

    Will start calculating: testmodel
    2022  solved
    2023  solved
    2024  solved
    2025  solved
    2026  solved
    2027  solved
    testmodel calculated

## Inspect Results

```python
with ecl.set_smpl('2021','2027'):
    ecl.keep_plot('loan_total',showtype='growth',legend=False);
```

```python
ecl.keep_plot('tr_*',showtype='level',legend=False);
```

```python

```

Examples/Economic Credit Loss/ECL Z setup.ipynb