# Integración Numérica

Python en Jupyter Notebook

Creado por Giancarlo Ortiz para el curso de Métodos Numéricos

## Solución de integrales por análisis numérico
la integración numérica constituye una amplia gama de algoritmos para calcular el valor numérico de una integral definida

## Agenda
1. Integración
1. Funciones primitivas
1. Integración numérica
1. Métodos del rectángulo
1. Método del trapecio
1. Regla Simpson

```python
# Importar módulos al cuaderno de jupyter
import math as m
import numpy as np
import pylab as plt
from sympy import *
from sympy.utilities.lambdify import lambdify, implemented_function

# Definir e incluir nuevas funciones al cuaderno
def _error_relativo(valor_real, valor_aproximado):
    error = 100 * abs(valor_real - valor_aproximado) / valor_real
    return error

# Declarar una variable simbólica
t, k, v = symbols('t k v')
```

```python
x = symbols('x')
fx = (x-1)*(x-2)*(x-3)
f1 = expand((x-1)*(x-2)*(x-3))
dfx = Derivative(fx, x).doit()
dfx1 = solve(dfx, x)
x1 = N(dfx1[0])
x2 = N(dfx1[1])
display(fx)
display(f1)
display(simplify(dfx))
print(f"El máximo es igual a: -> {x1:7.3f}")
print(f"El mínimo es igual a: -> {x2:7.3f}")
```

$\displaystyle \left(x - 3\right) \left(x - 2\right) \left(x - 1\right)$

$\displaystyle x^{3} - 6 x^{2} + 11 x - 6$

$\displaystyle 3 x^{2} - 12 x + 11$

    El máximo es igual a: ->   1.423
    El mínimo es igual a: ->   2.577

## 1. Integración
---
En un sistema de coordenadas cartesianas, la integral de una función $\color{#a78a4d}{f(x)}$ representa el área limitada por la gráfica de la función y el eje de las abscisas $\color{#a78a4d}{x}$ en un intervalo $\color{#a78a4d}{(a,b)}$, agregando el signo positivo cuando la función toma valores positivos o está definida por encima del eje $\color{#a78a4d}{x}$ y agregando el signo negativo cuando la función toma valores negativos o está definida por debajo del eje $\color{#a78a4d}{x}$.

\begin{equation*}
\tag{1} \int_{a}^{b} f(x) \,dx \\
\end{equation*}

Resolver esta integral consiste en calcular el área total bajo la función $\color{#a78a4d}{f(x)}$ en un intervalo definido, del [teorema fundamental del calculo](https://es.wikipedia.org/wiki/Teorema_fundamental_del_c%C3%A1lculo) se obtiene la solución exacta evaluando la función primitiva $\color{#a78a4d}{F(x)}$ en los limites de la integral definida.

\begin{equation*}
\tag{1} \int_{a}^{b} f(x) \,dx = F(b) - F(a) \\
\end{equation*}

```python
# Definir una integral
Lim_a = 4
Lim_b = 8
ft = (4/125) * t**3 - (6/25) * t**2 + (19/5)
Ft = Integral(ft, t)
Ft_ab = Integral(ft, (t, Lim_a, Lim_b))

# Evaluar la integral
Rt = Ft.doit()
Rt_ab = Ft_ab.doit()

# Salida estándar
print(f"Sea la función f(t):")
display(ft)
print(f"La función primitiva de f(t) es evaluada como:")
display(Eq(Ft, Rt))
print(f"La integral de f(t) en (a,b) es evaluada como:")
display(Ft_ab, Rt_ab)

# Función y valor exacto
F1 = lambdify(t, ft)
V1 = float(Rt_ab)
```

    Sea la función f(t):

$\displaystyle 0.032 t^{3} - 0.24 t^{2} + 3.8$

    La función primitiva de f(t) es evaluada como:

$\displaystyle \int \left(0.032 t^{3} - 0.24 t^{2} + 3.8\right)\, dt = 0.008 t^{4} - 0.08 t^{3} + 3.8 t$

    La integral de f(t) en (a,b) es evaluada como:

$\displaystyle \int\limits_{4}^{8} \left(0.032 t^{3} - 0.24 t^{2} + 3.8\right)\, dt$

$\displaystyle 10.08$

## 2. Funciones primitivas
---
Las funciones primitivas constituyen un método exacto para solucionar integrales, existen diferentes técnicas que usadas de forma combinada permiten calcular una primitiva de funciones exponenciales, racionales y trigonométricas pero no existe un algoritmo determinista que permita expresar la primitiva de todas las funciones incluyendo casos en los que la primitiva de funciones elementales no es una función elemental o no existe.

```python
# Definir una integral
fk = (1/2)*k - cos(2*k) - 2
Fk = Integral(fk, k)
Fk_ab = Integral(fk, (k, Lim_a, Lim_b))

# Evaluar la integral
Rk = Fk.doit()
Rk_ab1 = Fk_ab.doit()
Rk_ab = N(Rk_ab1)

# Salida estándar
print(f"Sea la función f(e):")
display(fk)
print(f"La función primitiva de f(e) es evaluada como:")
display(Eq(Fk, Rk))
print(f"La integral de f(e) en (a,b) es evaluada como:")
display(Fk_ab, Rk_ab)

# Función y valor exacto
F2 = lambdify(k, fk)
V2 = float(Rk_ab)
```

    Sea la función f(e):

$\displaystyle 0.5 k - \cos{\left(2 k \right)} - 2$

    La función primitiva de f(e) es evaluada como:

$\displaystyle \int \left(0.5 k - \cos{\left(2 k \right)} - 2\right)\, dk = 0.25 k^{2} - 2 k - \frac{\sin{\left(2 k \right)}}{2}$

    La integral de f(e) en (a,b) es evaluada como:

$\displaystyle \int\limits_{4}^{8} \left(0.5 k - \cos{\left(2 k \right)} - 2\right)\, dk$

$\displaystyle 4.63863078164422$

## 2. Integración Numérica
---
La integración numérica es una técnica para aproximar una integral definida, cuando no existe una anti-derivada elemental, como en el caso de la [función gaussiana](https://es.wikipedia.org/wiki/Funci%C3%B3n_gaussiana), que se usa frecuentemente en estadística, probabilidad y teoría del error.

\begin{equation*}
f(x) = a \cdot e^{-\frac{1}{2} \left(\frac{x-b}{c}\right)^2}
\end{equation*}

Teniendo en cuenta que el problema básico considerado por la integración numérica es calcular una solución aproximada a la integral definida; se tiene que el problema se reduce a calcular el área total bajo la función $\color{#a78a4d}{f(x)}$.

### Ejemplo: Ruido blanco Gaussiano
---
En telecomunicaciones y redes informáticas, los canales de comunicación pueden verse afectados por distintas fuentes de ruido que proceden de muchas fuentes naturales; como las vibraciones térmicas de los átomos en los conductores que ocasionan [ruido térmico](https://es.wikipedia.org/wiki/Ruido_de_Johnson-Nyquist) (conocido como ruido de Johnson-Nyquist), el movimiento de cargas discretas que ocasiona [ruido de disparo](https://es.wikipedia.org/wiki/Ruido_de_disparo) (o de cargas discretas) y la [radiación de cuerpo negro](https://es.wikipedia.org/wiki/Radiaci%C3%B3n_de_cuerpo_negro) en equilibrio termodinámico.

El teorema del límite central de la teoría de la probabilidad indica que la suma de muchos procesos aleatorios tenderá a tener una distribución denominada gaussiana o normal; es por esto que el modelo de ruido blanco gaussiano, [AWGN](https://en.wikipedia.org/wiki/Additive_white_Gaussian_noise); se utiliza a menudo como un modelo de canal en el que el único impedimento para la comunicación es una adición lineal de banda ancha o ruido blanco con una densidad espectral constante (expresada en vatios por hertz de ancho de banda) y una distribución de amplitud gaussiana.

El Ruido gaussiano blanco aditivo, será aquel cuya [función de densidad de probabilidad](https://es.wikipedia.org/wiki/Funci%C3%B3n_de_densidad_de_probabilidad) responde a una distribución normal.

```python
# Función Gaussiana
def _funcion_gauss(a, μ, σ, x):
    y = a * np.e**(-1/2 * ((x-μ)/σ)**2)
    return y

# Características del ruido
σ = 7e-3    # Desviación o voltaje eficaz del ruido (V_RMS)
μ = 0       # Media del ruido

# Amplitud de la función de probabilidad
a = (2 * m.pi * σ**2)**(-2)

# Función de probabilidad
P = lambda x: _funcion_gauss(a, μ, σ, x)

# Gráficas
voltaje = np.linspace(-4*σ, 4*σ, 512)
tiempo  = np.linspace(0, 10, 1024)
ruido   = np.random.default_rng().normal(μ, σ, 1024)

# Figuras
fig = plt.figure(figsize=(16,4))
ax = [0, 1]
ax[0] = plt.subplot2grid((1, 4), (0, 0))
ax[1] = plt.subplot2grid((1, 4), (0, 1), colspan=3)
fig.suptitle('Ruido Blanco Gaussiano'.upper(), fontsize=14, fontweight="bold")

# Gráficas 1
ax[0].axhline(y = 0, linewidth=1.0, color="#ccc")
ax[0].plot(P(voltaje), voltaje, label="$P(v)$")
ax[0].set_ylim([-0.03, 0.03])
ax[0].set_title("Función probabilidad de ruido", fontsize=8)
ax[0].set_xlabel("Probabilidad")
ax[0].set_ylabel("Voltaje")
ax[0].legend()

# Gráficas 2
ax[1].axhline(y = 0, linewidth=1, color="#ccc")
ax[1].plot(tiempo, ruido, label="$(x)$")
ax[1].set_ylim([-0.03, 0.03])
ax[1].set_title("Señal de ruido", fontsize=8)
ax[1].set_xlabel("tiempo")
ax[1].legend()

plt.show()
```

## 3. Métodos del rectángulo
---
En el análisis numérico, los algoritmos del rectángulo son útiles para resolver numéricamente la integral definida; considerando como una aproximación del área total, la suma de las áreas de $\color{#a78a4d}{n}$ rectángulos.

\begin{align}
\tag{2} A_i &= h_i \cdot b_i \\
\tag{3} A &= \sum h_i \cdot b_i \\
\tag{4} b &= \Delta x \\
\tag{5} A &= \Delta x \sum h_i \\
\end{align}

### a. Suma de Riemann por la izquierda
La suma de Riemann por la izquierda es uno de los algoritmos del rectángulo para resolver numéricamente la integral definida; considerando la altura de cada rectángulo como la función evaluada en el lado izquierdo de ese intervalo.

\begin{align}
\tag{6} A &= \Delta x \sum_{i=0}^{n} f(a + i \cdot \Delta x) \\
\tag{7} \Delta x &= \frac{b-a}{n}  \\
\end{align}

### b. Suma de Riemann por la derecha
La suma de Riemann por la derecha es uno de los algoritmos del rectángulo para resolver numéricamente la integral definida; considerando la altura de cada rectángulo como la función evaluada en el lado derecho de ese intervalo.

\begin{align}
\tag{8} A &= \Delta x \sum_{i=0}^{n} f(b - i \cdot \Delta x) \\
\tag{9} \Delta x &= \frac{b-a}{n}  \\
\end{align}

### c. Método del punto medio
En el análisis numérico, el método del punto medio es uno de los algoritmos del rectángulo para resolver numéricamente la integral definida; considerando la altura de cada rectángulo como la función evaluada en el punto medio de ese intervalo.

\begin{align}
\tag{10} A &= \Delta x \sum_{i=0}^{n} f(a + \frac{2i+1}{2} \cdot \Delta x) \\
\tag{11} \Delta x &= \frac{b-a}{n}  \\
\end{align}

```python
# Defino los métodos de suma de rectángulos - izquierda
def _Riemman_izq(Func, limA, limB, numI):
    """ Método de la Bisección para encontrar raíces en un intervalo.

        ## Parámetros:
            Func (function) : función que depende de una variable.
            limA (function) : derivada de la función.
            limB (int)      : semilla de la solución.
            numI (int)      : número máximo de iteraciones.

        ## Devoluciones:
            Area (float)       : valor de area encontrado.
    """
    Δx = (limB - limA) / numI
    Alturas = 0
    for i in range(numI):
        Alturas += Func(limA + i * Δx)

    Area = Δx * Alturas
    return Area

# Defino los métodos de suma de rectángulos - izquierda
def _Riemman_der(Func, limA, limB, numI):
    """ Método de la Bisección para encontrar raíces en un intervalo.

        ## Parámetros:
            Func (function) : función que depende de una variable.
            limA (function) : derivada de la función.
            limB (int)      : semilla de la solución.
            numI (int)      : número máximo de iteraciones.

        ## Devoluciones:
            Area (float)       : valor de area encontrado.
    """
    Δx = (limB - limA) / numI
    Alturas = 0
    for i in range(numI):
        Alturas += Func(limB - i * Δx)

    Area = Δx * Alturas
    return Area

# Defino los métodos de suma de rectángulos - izquierda
def _Riemman_med(Func, limA, limB, numI):
    """ Método de la Bisección para encontrar raíces en un intervalo.

        ## Parámetros:
            Func (function) : función que depende de una variable.
            limA (function) : derivada de la función.
            limB (int)      : semilla de la solución.
            numI (int)      : número máximo de iteraciones.

        ## Devoluciones:
            Area (float)       : valor de area encontrado.
    """
    Δx = (limB - limA) / numI
    Alturas = 0
    for i in range(numI):
        Alturas += Func(limA + (2*i + 1)/2 * Δx)

    Area = Δx * Alturas
    return Area
```

## 4. Método del trapecio
---
En el análisis numérico, el método del trapecio es un algoritmo para resolver numéricamente la integral definida; considerando el área total debajo de la función como la suma de las áreas de $\color{#a78a4d}{n}$ trapecios rectos.

\begin{align}
\tag{12} A_i &= \frac{b1_i + b2_i}{2} \cdot h_i \\
\tag{13} A &= \frac{1}{2} \sum h_i \cdot (b1_i + b2_i) \\
\tag{14} h &= \Delta x \\
\tag{15} A &= \frac{ \Delta x }{2} \sum (b1_i + b2_i) \\
\tag{16} A &= \frac{ \Delta x }{2} \sum f(a + i \cdot \Delta x) + f(a + \Delta x + i \cdot \Delta x) \\
\end{align}

```python
# Defino el métodos del trapecio
def _trapecio(Func, limA, limB, numI):
    """ Método de la Bisección para encontrar raíces en un intervalo.

        ## Parámetros:
            Func (function) : función que depende de una variable.
            limA (function) : derivada de la función.
            limB (int)      : semilla de la solución.
            numI (int)      : número máximo de iteraciones.

        ## Devoluciones:
            Area (float)       : valor de area encontrado.
    """
    Δx = (limB - limA) / numI
    Bases = 0
    for i in range(numI):
        Bases += Func(limA + i * Δx) + Func(limA + Δx + i * Δx)

    Area = Δx/2 * Bases
    return Area
```

## 5. Reglas Simpson
---
En el análisis numérico, la regla de Simpson es un algoritmo para resolver numéricamente la integral definida; considerando el área total debajo de la función como la suma de las áreas de $\color{#a78a4d}{n}$ polinomios mas simples.

### a. Ecuaciones lineales
Se aproxima la integral usando $\color{#a78a4d}{n}$ polinomios de orden dos $\color{#a78a4d}{P_2(x)}$, evaluando la función en tres puntos según lo define el [polinomio de Lagrange](https://es.wikipedia.org/wiki/Interpolaci%C3%B3n_polin%C3%B3mica_de_Lagrange) del mismo orden.

\begin{align}
\tag{17} P_2(x) &= x2 + a_1 x + a_o \\
\tag{18} l_2(x) &= f(x_o) \frac{x-x_1}{x_o-x_1} \cdot \frac{x-x_2}{x_o-x_2} + f(x_1) \frac{x-x_o}{x_1-x_o} \cdot \frac{x-x_2}{x_1-x_2} + f(x_2)\frac{x-x_o}{x_2-x_o} \cdot \frac{x-x_1}{x_2-x_1} \\
\tag{19} \int_{a}^{b} f(x) \,dx &= \int_{a}^{b} l_2(x) \,dx + E(f) \\

\tag{13} A &= \frac{1}{2} \sum h_i \cdot (b1_i + b2_i) \\

\tag{16} A &= \frac{b-a}{6} \left( f(a) + 4 f(\frac{a+b}{2}) + f(b) \right) \\

\tag{16} A &= \frac{\Delta x}{6} \left( f(a + i \cdot \Delta x) + 4 f(a + \frac{(2i+1)}{2} \Delta x) + f(a + \Delta x + i \cdot \Delta x) \right) \\
\end{align}

```python
# Defino el métodos de la regla simpson
def _simpson(Func, limA, limB, numI):
    """ Método de la Bisección para encontrar raíces en un intervalo.

        ## Parámetros:
            Func (function) : función que depende de una variable.
            limA (function) : derivada de la función.
            limB (int)      : semilla de la solución.
            numI (int)      : número máximo de iteraciones.

        ## Devoluciones:
            Area (float)       : valor de area encontrado.
    """
    Δx = (limB - limA) / numI
    Bases = 0
    for i in range(numI):
        Bases += Func(limA + i * Δx) + 4 * Func(limA + (2*i + 1)/2 * Δx) + Func(limA + Δx + i * Δx)

    Area = Δx/6 * Bases
    return Area
```

```python
# Funciones de prueba
F1 = lambda x: (4/125) * x**3 - (6/25) * x**2 + (59/15)
F2 = lambda x: (1/2)*x - np.cos(2*x) - 2
F3 = lambda x: _funcion_gauss(1, 5, 1, x)

# Vectores de prueba
x = np.linspace(0, 10, 512, endpoint=True)

# Gráficas
#-------------------
fig, ax = plt.subplots(1, 3)
fig.set_size_inches(16, 4)
fig.suptitle('Funciones de Prueba'.upper(), fontsize=14, fontweight="bold")

# Gráficas 1
ax[0].axhline(y = 0, linewidth=1.0, color="#ccc")
ax[0].axhline(y = 5, linewidth=1.0, color="#ccc")
ax[0].axvline(x = Lim_a, linewidth=1.0, color="#ccc")
ax[0].axvline(x = Lim_b, linewidth=1.0, color="#ccc")
ax[0].plot(x, F1(x), label="$F_1(x)$")
ax[0].fill_between(x, F1(x), where=(F1(x) > 0) & (x > Lim_a) & (x < Lim_b), facecolor='#FEE')
ax[0].set_xlim([2, 9])
ax[0].set_ylim([-1, 6])
ax[0].set_title("Función Polinómica", fontsize=8)
ax[0].set_xlabel("Variable independiente")
ax[0].set_ylabel("variable dependiente")
ax[0].legend()

# Gráficas 2
ax[1].axhline(y = 0, linewidth=1, color="#ccc")
ax[1].axhline(y = 3, linewidth=1.0, color="#ccc")
ax[1].axvline(x = Lim_a, linewidth=1.0, color="#ccc")
ax[1].axvline(x = Lim_b, linewidth=1.0, color="#ccc")
ax[1].plot(x, F2(x), label="$F_3(x)$")
ax[1].fill_between(x, F2(x), where=(F2(x) > 0) & (x > Lim_a) & (x < Lim_b), facecolor='#FEE')
ax[1].set_xlim([2, 9])
ax[1].set_ylim([-1, 6])
ax[1].set_title("Función sinusoide", fontsize=8)
ax[1].set_xlabel("Variable independiente")
ax[1].legend()

# Gráficas 3
ax[2].axhline(y = 0, linewidth=1, color="#ccc")
ax[2].axhline(y = 1, linewidth=1.0, color="#ccc")
ax[2].axvline(x = Lim_a, linewidth=1.0, color="#ccc")
ax[2].axvline(x = Lim_b, linewidth=1.0, color="#ccc")
ax[2].plot(x, F3(x), label="$F_2(x)$")
ax[2].fill_between(x, F3(x), where=(F3(x) > 0) & (x > Lim_a) & (x < Lim_b), facecolor='#FEE')
ax[2].set_xlim([0, 10])
ax[2].set_ylim([-1, 9])
ax[2].set_title("Función Gaussiana", fontsize=8)
ax[2].set_xlabel("Variable independiente")
ax[2].legend()

plt.show()
```

```python
# Métodos de integración numéricos
I1, I2, I3 = _Riemman_izq(F1, Lim_a, Lim_b, 5), _Riemman_izq(F2, Lim_a, Lim_b, 5), _Riemman_izq(F3, Lim_a, Lim_b, 5)
D1, D2, D3 = _Riemman_der(F1, Lim_a, Lim_b, 5), _Riemman_der(F2, Lim_a, Lim_b, 5), _Riemman_der(F3, Lim_a, Lim_b, 5)
M1, M2, M3 = _Riemman_med(F1, Lim_a, Lim_b, 5), _Riemman_med(F2, Lim_a, Lim_b, 5), _Riemman_med(F3, Lim_a, Lim_b, 5)
T1, T2, T3 =    _trapecio(F1, Lim_a, Lim_b, 5),    _trapecio(F2, Lim_a, Lim_b, 5),    _trapecio(F3, Lim_a, Lim_b, 5)
S1, S2, S3 =     _simpson(F1, Lim_a, Lim_b, 5),     _simpson(F2, Lim_a, Lim_b, 5),     _simpson(F3, Lim_a, Lim_b, 5)

# Error relativo de los métodos de integración numéricos
er_I1, er_D1, er_M1, er_T1, er_S1 = _error_relativo(V1, I1), _error_relativo(V1, D1), _error_relativo(V1, M1), _error_relativo(V1, T1), _error_relativo(V1, S1)
er_I2, er_D2, er_M2, er_T2, er_S2 = _error_relativo(V2, I2), _error_relativo(V2, D2), _error_relativo(V2, M2), _error_relativo(V2, T2), _error_relativo(V2, S2)

# Salida estándar
print(f"Método de integración               : {'F1':>7} : {'ER1':>9} : {'F2':>7} : {'ER2':>9} : {'F3':>7} :")
print(f"-------------------------------------------------------------------------------------------")
print(f"Area de Riemann por la izquierda    : {I1:7.3f} : {er_I1:7.1f} % : {I2:7.1f} : {er_I2:7.1f} % : {I3:7.3f} :")
print(f"Area de Riemann por la derecha      : {D1:7.3f} : {er_D1:7.1f} % : {D2:7.1f} : {er_D2:7.1f} % : {D3:7.3f} :")
print(f"Area por el método del punto medio  : {M1:7.3f} : {er_M1:7.1f} % : {M2:7.1f} : {er_M2:7.1f} % : {M3:7.3f} :")
print(f"Area por el método del trapecio     : {T1:7.3f} : {er_T1:7.1f} % : {T2:7.1f} : {er_T2:7.1f} % : {T3:7.3f} :")
print(f"Area por el método del Simpson      : {S1:7.3f} : {er_S1:7.1f} % : {S2:7.1f} : {er_S2:7.1f} % : {S3:7.3f} :")
```

    Método de integración               :      F1 :       ER1 :      F2 :       ER2 :      F3 :
    -------------------------------------------------------------------------------------------
    Area de Riemann por la izquierda    :   9.630 :     4.5 % :     3.4 :    27.3 % :   2.309 :
    Area de Riemann por la derecha      :  11.883 :    17.9 % :     5.6 :    21.2 % :   1.833 :
    Area por el método del punto medio  :  10.542 :     4.6 % :     4.7 :     1.6 % :   2.123 :
    Area por el método del trapecio     :  10.757 :     6.7 % :     4.5 :     3.1 % :   2.071 :
    Area por el método del Simpson      :  10.613 :     5.3 % :     4.6 :     0.0 % :   2.106 :

---
## Mas Recursos

- [Integración numérica](https://es.wikipedia.org/wiki/Integraci%C3%B3n_num%C3%A9rica) (Wikipedia)
- [Suma de Riemann](https://es.wikipedia.org/wiki/Suma_de_Riemann) (Wikipedia)
- [Regla del trapecio](https://es.wikipedia.org/wiki/Regla_del_trapecio) (Wikipedia)
- [Regla de Simpson](https://es.wikipedia.org/wiki/Regla_de_Simpson) (Wikipedia)


```python
# stan implementation
import pystan
%pylab inline
from scipy.special import polygamma as pg
```

    Populating the interactive namespace from numpy and matplotlib

    Bad key "axes.color_cycle" on line 250 in
    /home/matus/Desktop/matustools/matplotlibrc.
    You probably need to get an updated matplotlibrc file from
    https://github.com/matplotlib/matplotlib/blob/v3.2.1/matplotlibrc.template
    or from the matplotlib source distribution

# Notation

$Y$ generic random variable

$U$ latent random variable

$V$ residual random variable

$X$ predictor
### Parameters
$\eta$ and $\nu$ generic parameters

$\mu=E[Y]$ mean parameter

$\gamma=E[\log Y]$ geometric mean parameter

$\sigma^2=E[(Y-\mu)^2]$ standard deviation parameter

$Y=\alpha+U$ shift parameter

$Y= U/\theta$ scale parameter

$Y= U \lambda$ inverse-scale (rate) parameter

$Y=e^{-\tau} U$ log-scale parameter

$Y=U^\kappa$ exponent parameter

$Y=f(U,\rho)$ shape parameter

$Y=\alpha + \beta X$ linear predictor

$\psi$ digamma function

$\pi$ pi number

$\phi$ measurement scale

$\delta$ dirac function

$\zeta,\epsilon,\varepsilon,\vartheta,\iota,\xi,\varpi,\varrho,\varsigma,\varphi,\chi,\omega$

## Gamma distribution

Paremeters $\eta$ and $\nu$ are orthogonal if

$$\operatorname{E}_Y
\left[
 \frac{\partial \log f(Y;\eta,\nu)}{\partial\eta \ \partial\nu}
 \right]=0$$

The probability density function of Gamma distribution parametrized by shape parameter $\rho$ and scale parameter $\theta$ is
$$f(Y=y;\rho,\theta)=\frac{1}{\Gamma(\rho) \theta^\rho} y^{\rho - 1} e^{-\frac{y}{\theta}}$$

with Fisher information

$$I_{\rho \theta} =  \begin{pmatrix}
\psi'(\rho) & \theta^{-1} \\
\theta^{-1} & \rho \theta^{-2} \end{pmatrix} $$

Consider parametrization in terms of logarithm of geometric mean $\gamma=E[\log Y]=\psi(\rho)+\log \theta$ and log-scale $\tau=\log(\theta)$, where $\psi$ is the digamma function. Then the logarithm of density function parametrized by $\gamma$ and $\tau$ is

$$\log f(Y=y;\gamma,\tau)=-\log{\Gamma(\omega(\gamma-\tau)) -\tau \omega(\gamma-\tau) + (\omega(\gamma-\tau)-1)\log y- y e^{-\tau}}$$

where we use $\omega$ to label the inverse digamma function. By $\omega'(y)$ $\omega''(y)$ and  we denote the first and second derivative of inverse digamma function with respect to $y$. Next, we compute the first derivative of the log-density with respect to $\gamma$:

$$\begin{align} \frac{\partial}{\partial\gamma}\log f(Y;\gamma,\tau) &= -\psi(\omega(\gamma-\tau)) \omega'(\gamma-\tau)-\tau \omega'(\gamma-\tau) + \omega'(\gamma-\tau) \log y \\
&= -(\gamma-\tau) \omega'(\gamma-\tau)-\tau \omega'(\gamma-\tau) + \omega'(\gamma-\tau) \log y \\
&= (\log y - \gamma)\omega'(\gamma -\tau)\end{align}$$

Next we obtain derivative with respect to $\gamma$ and $\tau$:

$$\begin{align} \frac{\partial}{\partial\gamma \partial\tau}\log f(Y;\gamma,\tau) &= \frac{\partial}{\partial\tau}\left[(\log y - \gamma)\omega'(\gamma -\tau)\right]\\
&= (\gamma-\log y)\omega''(\gamma-\tau)
\end{align}$$

Finally, compute the expectation
$$\begin{align} \operatorname{E}_Y
\left[
 \frac{\partial \log f(Y;\tau,\gamma)}{\partial\tau\ \partial\gamma}
 \right]&= \operatorname{E}\left[\omega''(\gamma-\tau)(\gamma-\log y)\right] \\
 &=\omega''(\gamma-\tau)(\gamma-\operatorname{E}[\log y])\\
 &=\omega''(\gamma-\tau)(\gamma-\gamma)\\
 &=0
\end{align}$$

Note that $\operatorname{E}[\log y]$ is the logarithm of geometric mean and hence $\operatorname{E}[\log y]=\gamma$

$$I_{\gamma \tau} =  \begin{pmatrix}
\omega'(\gamma-\tau) & 0\\
0 & \omega(\gamma-\tau)-\omega'(\gamma-\tau)\end{pmatrix} $$

$$I_{\rho \tau} =  \begin{pmatrix}
\psi'(\rho) & 1 \\
1 & \rho \end{pmatrix} $$

$$I_{\rho, \tau+\log \rho} =  \begin{pmatrix}
\psi'(\rho)-1/\rho & 0 \\
0 & \rho \end{pmatrix} $$

$$I_{\psi(\rho), \tau} =  \begin{pmatrix}
\psi'(\rho)^{-1} & \psi'(\rho)^{-1} \\
\psi'(\rho)^{-1} & \rho \end{pmatrix} $$

$$I_{\psi(\rho)+\tau, \tau} =  \begin{pmatrix}
\psi'(\rho)^{-1} & 0 \\
0& \rho-\psi'(\rho)^{-1} \end{pmatrix} $$

```python
model = """
data {
    int<lower=0> N; //nr subjects
    real<lower=0> k;
    real<lower=0> t;
}generated quantities{
    real<lower=0> y;
    y=gamma_rng(k,1/t);
}
"""
smGammaGen = pystan.StanModel(model_code=model)
```

    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_56e71184937d3586a66635dbe41869eb NOW.

```python
model = """
data {
    int<lower=0> N; //nr subjects
    real<lower=0> y[N];
}parameters{
    real<lower=0> k;
    real<lower=0> t;
}model{
    for (n in 1:N)
        y[n]~gamma(k,1/t);
}
"""
smGamma = pystan.StanModel(model_code=model)
```

    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_b8b2a451eabaf76e034fac0e9eb791d9 NOW.

```python
N=1000
fit=smGammaGen.sampling(data={'N':N,'k':10,'t':np.exp(-10)},
    chains=1,n_jobs=1,seed=1,thin=1,iter=N,warmup=0,algorithm="Fixed_param")
w=fit.extract()
y=w['y']
print(y.shape)

fit=smGamma.sampling(data={'N':N,'y':y},
    chains=4,n_jobs=4,seed=1,thin=2,iter=2000,warmup=1000)
print(fit)
w=fit.extract()
t=np.log(w['t'])
g=pg(0,w['k'])+t
w=fit.extract()
plt.plot(g,t,'.')
np.corrcoef(g,t)[0,1]
```

```python
invgammafun='''functions{
    vector invdigamma(vector x){
        vector[num_elements(x)] y; vector[num_elements(x)] L;
        for (i in 1:num_elements(x)){
            if (x[i]==digamma(1)){
                y[i]=1;
            }else{ if (x[i]>=-2.22){
                y[i]=(exp(x[i])+0.5);
            }else{
                y[i]=1/(x[i]-digamma(1));
        }}}
        L=digamma(y)-x;
        while (min(L)>10^-12){
            y=y-L ./trigamma(y);
            L=digamma(y)-x;
        }
        return y;}
    real invdigammaR(real x){
        real y; real L;
        if (x==digamma(1)){
            y=1;
        }else{ if (x>=-2.22){
            y=(exp(x)+0.5);
        }else{
            y=1/(x-digamma(1));
        }}
        L=digamma(y)-x;

        while (abs(L)>1e-5){
            y=y-L ./trigamma(y);
            L=digamma(y)-x;
        }
        return y;
    }} '''
model = """
data {
    int<lower=0> N; //nr subjects
    real<lower=0> y[N];
}parameters{
    real<lower=-100,upper=100> g;
    real<lower=-100,upper=100> t;
}transformed parameters{
    real k;
    k=invdigammaR(g-t);
}model{
    for (n in 1:N)
        y[n]~gamma(k,exp(-t));
}
"""
smGammaGeom = pystan.StanModel(model_code=invgammafun+model)
```

    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_73d022c146ed724ba4b8072e7a8ef172 NOW.

```python
N=10
fit=smGammaGen.sampling(data={'N':N,'k':1,'t':np.exp(0)},
    chains=1,n_jobs=1,seed=1,thin=1,iter=N,warmup=0,algorithm="Fixed_param")
w=fit.extract()
y=w['y']

fit=smGammaGeom.sampling(data={'N':N,'y':y},
    chains=4,n_jobs=4,seed=2,thin=1,iter=500,warmup=200)
#control={'adapt_delta':0.99})
print(fit)
```

```python
w=fit.extract()
#plt.plot(pg(0,w['k']),w['g']-w['t'],'.')
#np.max(np.abs(pg(0,w['k'])-w['g']+w['t']))
plt.plot(w['g'],w['t'],'.')
```

# Hierarchical parameter recovery

```python
model = """
data {
    int<lower=0> N; //nr subjects
    int<lower=0> M;
    real gm;
    real gs;
    real t;
}generated quantities{
    real g[N];
    real<lower=0> y[N,M];
    for (n in 1:N){
        g[n]=normal_rng(gm,gs);
        for (m in 1:M){
            y[n,m]=gamma_rng(invdigammaR(g[n]-t),exp(t));
}}}
"""
smGammaGen = pystan.StanModel(model_code=invgammafun+model)
```

    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_0f1c4cd5a7a2dd5bfc3c53aaa88ff7a5 NOW.

```python
N=10;M=20
fit=smGammaGen.sampling(data={'N':N,'M':M,'gm':5,'gs':2,'t':1},
    chains=4,n_jobs=4,seed=1,thin=1,iter=30,warmup=0,algorithm="Fixed_param")
w=fit.extract()
y=w['y'][0,:,:]
print(y.shape)
```

    (10, 20)

```python
model = """
data {
    int<lower=0> N; //nr subjects
    int<lower=0> M;
    real<lower=0> y[N,M];
}parameters{
    real g[N];
    real gm;
    real<lower=0> gs;
    real t;
}model{
    for (n in 1:N){
        g[n]~normal(gm,gs);
        for (m in 1:M){
            y[n,m]~gamma(invdigammaR(g[n]-t),exp(t));
}}}
"""
smGamma = pystan.StanModel(model_code=invgammafun+model)
```

    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_f6a8758085290b91fe0fd36fc35263d5 NOW.

```python
fit=smGamma.sampling(data={'N':N,'M':M,'y':y},
    chains=4,n_jobs=4,seed=2,thin=1,iter=1000,warmup=500)
print(fit)
```

```python
%pylab inline
plt.plot(w['gm'])
```

## Weibull distribution
$$f(y)=\frac{\kappa}{y}\left(\frac{y}{\theta}\right)^{\kappa}e^{-\left(\frac{y}{\theta} \right)^\kappa}$$

$$I_{\theta \kappa} = \begin{pmatrix}
\frac{\kappa^2}{\theta^2} & -\frac{\psi(2)}{\theta}\\
. & \frac{1}{\kappa^2}\left(\psi'(1)+\psi(2)^2\right)\end{pmatrix} $$

$E[\log Y]= \log \theta + \psi(1)/\kappa$

$E[Y^s]=\theta^s \Gamma(1+s/\kappa)$

$E[Y^\kappa]=\theta^\kappa $

$\mathrm{Var}[\log Y]=\psi'(1)/\kappa^2$

$E[(Y/\theta)^\kappa]=1$

$\mathrm{Var}[(Y/\theta)^\kappa]=1$

$E[\log (Y/\theta^\kappa)]= \psi(1)$

$E[\log^2 (Y/\theta^\kappa)]= \psi'(1)+\psi(1)^2$

$E[(Y/\theta)^\kappa \log (Y/\theta)^\kappa ]= \psi(2)= \psi(1)+1$

$E[(Y/\theta)^\kappa \log^2(Y/\theta)^\kappa ]= \psi'(2)+\psi(2)^2$

$$I_{\tau \kappa} = \begin{pmatrix}
\kappa^2 & - \psi(2)\\
. & \frac{1}{\kappa^2}\left(\psi'(1)+\psi(2)^2\right)\end{pmatrix} $$
$\tau=\log \theta$

$r_{\tau \kappa}=\psi(2)/\sqrt{\psi'(1)+\psi(2)^2}=0.31$

This is orthogonal parametrization

$$\kappa= \frac{1}{\xi-H \tau}$$
$$\xi=\frac{1}{\kappa}+H \tau $$

$H=\frac{\psi(2)}{\psi'(1)+\psi(2)^2}=0.232$

$$I_{\tau \xi} = \frac{H}{(\xi-\tau)^{2}} \begin{pmatrix}
\left(1+\frac{\psi(2)^2}{\psi'(1)}\right)^{-1} &0 \\
. & \left(1+\frac{\psi'(1)}{\psi(2)^2}\right)^{-1} \end{pmatrix} $$

$$I_{\tau \kappa} = \begin{pmatrix}
\kappa^2 & - \psi(2)\\
. & \frac{1}{\kappa^2}\left(\psi'(1)+\psi(2)^2\right)\end{pmatrix} $$

$$I_{\tau,1/\kappa} =\kappa^{2} \begin{pmatrix}
1 & \psi(2)\\
. & \psi'(1)+\psi(2)^2\end{pmatrix} $$

$$I_{\tau,1/\kappa-H\tau} =\kappa^{2} \begin{pmatrix}
1-\psi(2) H& 0\\
. & \psi'(1)+\psi(2)^2\end{pmatrix} = \kappa^{2} \begin{pmatrix}
0.902 & 0\\
. & 1.824\end{pmatrix}$$

$$I_{\tau,H\kappa} =\begin{pmatrix}
\kappa^{2} & \psi(2) H\\
. & \kappa^{-2} \psi(2) H\end{pmatrix} $$

$$I_{\tau,1/(H\kappa)} =\kappa^{2} H^2 \begin{pmatrix}
1 & \psi(2) H\\
. & \psi(2) H\end{pmatrix} $$

$$I_{\tau,1/(H\kappa)+\tau} =\kappa^{2} H^2 \begin{pmatrix}
1-\psi(2) H& 0\\
. & \psi(2) H\end{pmatrix} \\= \kappa^{2} H^2
\begin{pmatrix}
\left(1+\frac{\psi(2)^2}{\psi'(1)}\right)^{-1} &0 \\
. & \left(1+\frac{\psi'(1)}{\psi(2)^2}\right)^{-1} \end{pmatrix}= \kappa^{2} H^2 \begin{pmatrix}
0.902 & 0\\
. & 0.098\end{pmatrix}$$
$$I_{\tau,\epsilon} =(\epsilon-\tau)^2\begin{pmatrix}
\left(1+\frac{\psi(2)^2}{\psi'(1)}\right)^{-1} &0 \\
. & \left(1+\frac{\psi'(1)}{\psi(2)^2}\right)^{-1} \end{pmatrix}$$

Orthogonal from Cox and Reid (1987)

$\epsilon= \exp(\log \theta + \psi(2)/\kappa)=\exp(1/\kappa)\exp(E[\log Y])=\exp E[(Y/\theta)^\kappa \log Y]$

$\theta= \epsilon \exp(-\psi(2)/\kappa)$

```python
1/H
```

    4.313501020391736

$$J_{a/H,b}=\begin{pmatrix} H &0 \\0 & 1 \end{pmatrix}$$

$$J^T \begin{pmatrix}  H^{-2} A &  H^{-1} B \\  H^{-1} B & C \end{pmatrix} J= \begin{pmatrix}  A &B \\B & C\end{pmatrix}$$

$H=B/A$
$$J^T \begin{pmatrix}   A &   B \\  B & C \end{pmatrix} J= \begin{pmatrix}  B^2/A &B^2/A \\B^2/A & C\end{pmatrix}$$

$$J_{a+b,b}=\begin{pmatrix}  1 &-1 \\0 & 1 \end{pmatrix}$$

$$J^T\begin{pmatrix}  A &A \\A & B \end{pmatrix} J= \begin{pmatrix}  A &0 \\0 & B-A\end{pmatrix}$$

$$J_{\log a,b}=\begin{pmatrix}  e^a &0 \\0 & 1 \end{pmatrix}$$

$$J_{\log a,b}^T \begin{pmatrix}  e^{-2a} A & e^{-a} B \\e^{-a} B & C \end{pmatrix} J_{\log a,b}= \begin{pmatrix}  A &B \\B & C\end{pmatrix}$$

$$J_{e^a,b}=\begin{pmatrix}  1/a &0 \\0 & 1 \end{pmatrix}$$

$$J^T \begin{pmatrix}  a^2 A & a B \\ a B & C \end{pmatrix} J= \begin{pmatrix}  A &B \\B & C\end{pmatrix}$$

$$J_{a^{-1},b}=\begin{pmatrix}  -a^{2} &0 \\0 & 1 \end{pmatrix}$$

$$J^T \begin{pmatrix}  a^{-4} A & a^{-2} B \\ a^{-2} B & C \end{pmatrix} J= \begin{pmatrix}  A &-B \\-B & C\end{pmatrix}$$

```python
pg(1,1)
```

    array(1.64493407)

#### old stuff

$$\mathrm{Cov}(\gamma,\phi)=J_{12}J_{11}\frac{\kappa^2}{\theta^2}+J_{21}J_{22}\frac{1}{\kappa^2}\left(\psi'(2)+\psi(2)^2+1\right)-\frac{\psi(2)}{\theta}(J_{12}J_{11}+J_{21}J_{22}+J_{21}J_{12}+J_{11}J_{22})$$

$\theta=e^\phi$

$J_{11}=\frac{\partial \theta}{\partial \phi}=e^\phi=\theta$

$J_{12}=\frac{\partial \theta}{\partial \gamma}=0$

$$\mathrm{Cov}(\gamma,\phi)=J_{21}J_{22}\frac{1}{\kappa^2}\left(\psi'(2)+\psi(2)^2+1\right)-\frac{\psi(2)}{J_{11}}(J_{21}J_{22}+J_{11}J_{22})$$

$$\mathrm{Cov}(\gamma,\phi)=J_{22}\left(J_{21}\frac{1}{\kappa^2}\left(\psi'(2)+\psi(2)^2+1\right)-\psi(2)(J_{21}/J_{11}+1)\right)\\
= J_{21}J_{22}\frac{\psi(2)}{\kappa^2}\left(
\frac{\psi'(2)+\psi(2)^2+1}{\psi(2)}-\kappa^2\left(\frac{\partial \phi}{\partial \kappa}+e^{-\phi}\right)\right)\\
= J_{21}J_{22}\psi(2)\left(
\frac{\psi'(2)+\psi(2)^2+1}{\kappa^2\psi(2)}-e^{-\phi}-\frac{\partial \phi}{\partial \kappa}\right)
$$

$\gamma=-\phi- \frac{\psi'(2)+\psi(2)^2+1}{\kappa \psi(2)}$

$\frac{\partial \gamma}{\partial \kappa}= -\frac{\psi'(2)+\psi(2)^2+1}{\kappa^2 \psi(2)}$

$\kappa=-\frac{\psi'(2)+\psi(2)^2+1}{(\gamma+\phi) \psi(2)}$

$\frac{\partial \kappa}{\partial \phi}= \frac{\psi'(2)+\psi(2)^2+1}{(\gamma+\phi)^2 \psi(2)}$

$$\mathrm{Cov}(\gamma,\phi)= J_{21}J_{22}\psi(2)\left(
\frac{(\gamma+\phi)^2 \psi(2)}{\psi'(2)+\psi(2)^2+1}-e^{-\phi}- \frac{(\gamma+\phi)^2 \psi(2)}{\psi'(2)+\psi(2)^2+1} \right)
$$

$c \mathrm{Ei}(\frac{c}{\kappa})-e^\frac{c}{\kappa}(e^{-\phi}+\kappa)=k$

```python

model = """
data {
    int<lower=0> N; //nr subjects
    vector<lower=0>[N] y;
}parameters {
    real<lower=0> k;
    real<lower=0> t;
}model {
  y~weibull(k,t);
}
"""
smWeibull = pystan.StanModel(model_code=model)

model = """
data {
    int<lower=0> N; //nr subjects
    vector<lower=0>[N] y;
}parameters {
    real t;
    real e;
}model {
  y~weibull(4.313501020391736/(e-t),exp(t));
}
"""
smWeibullE = pystan.StanModel(model_code=model)
```

    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_dd507d30f0dad57573a6224c4c01ef6c NOW.

```python
model = """
data {
    int<lower=0> N;
    int<lower=0> M;
    vector<lower=0>[M] y[N];
}parameters {
    real lnk[N];
    real lnt[N];
    real km;real tm;
    real<lower=0> ks;
    real<lower=0> ts;
}model {
    lnk~normal(km,ks);
    lnt~normal(tm,ts);
    for (n in 1:N)
        y[n]~weibull(exp(lnk[n]),exp(lnt[n]));
}
"""
#smWeibullH = pystan.StanModel(model_code=model)

model = """
data {
    int<lower=0> N;
    int<lower=0> M;
    vector<lower=0>[M] y[N];
}parameters {
    real<lower=0> lne[N];
    real lnt[N];
    real em;real tm;
    real<lower=0> es;
    real<lower=0> ts;
}model {
    lne~normal(em,es);
    lnt~normal(tm,ts);
    for (n in 1:N)
        y[n]~weibull(4.313501020391736/(lne[n]),exp(lnt[n]));
}
"""
smWeibullEH = pystan.StanModel(model_code=model)
```

    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_3d99857cd395ed26afb9bdf6f6f112d9 NOW.

```python
print(polygamma(0,1))
print(polygamma(0,2))
print(polygamma(1,1))
print(polygamma(1,2))
print(polygamma(1,1)**2)
print(polygamma(1,2)**2)
```

    -0.5772156649015329
    0.42278433509846713
    1.6449340668482266
    0.6449340668482266
    2.7058080842778462
    0.4159399505813928

```python
ts=[-10,-1,0,1,10]
k=1
for t in ts:
    plt.subplot(2,3,k);k+=1
    e=np.linspace(-10,10,101)
    plt.plot(e,4.313501020391736/(e-t))
```

```python

```

```python
fit.get_adaptation_info()
```

    ['# Adaptation terminated\n# Step size = 0.889853\n# Diagonal elements of inverse mass matrix:\n# 0.0492589, 0.380148\n',
     '# Adaptation terminated\n# Step size = 0.791963\n# Diagonal elements of inverse mass matrix:\n# 0.0610601, 0.670636\n',
     '# Adaptation terminated\n# Step size = 0.964206\n# Diagonal elements of inverse mass matrix:\n# 0.0462998, 0.622056\n',
     '# Adaptation terminated\n# Step size = 0.79833\n# Diagonal elements of inverse mass matrix:\n# 0.0554586, 0.539706\n',
     '# Adaptation terminated\n# Step size = 0.840287\n# Diagonal elements of inverse mass matrix:\n# 0.0620329, 0.380415\n',
     '# Adaptation terminated\n# Step size = 0.708001\n# Diagonal elements of inverse mass matrix:\n# 0.0556485, 0.39183\n']

```python
from scipy import stats
def prs(x):
    ts= x.rsplit('\n#')
    out=[ts[1].rsplit('=')[1]]
    out.extend(ts[3][:-2].rsplit(','))
    return out
def computeConvergence(ms,data,reps=50):
    from time import time
    D=[[],[]]
    R=[[],[]]
    for sd in range(reps):
        print(sd)
        for m in range(len(ms)):
            sm=ms[m]
            t0=time()
            try:
                fit=sm.sampling(data=data,chains=6,n_jobs=6,
                        seed=1,thin=1,iter=1000,warmup=500)
                D[m].append(time()-t0)
                nfo=list(map(prs,fit.get_adaptation_info()) )
                R[m].append(nfo)
            except:
                D[m].append(np.nan)
                R[m].append(np.zeros((6,3))*np.nan)
    D=np.array(D)
    #R=np.float32(R)
    print(np.mean(D,1))
    return D, R

t=-1;e=1
k=4.313501020391736/(e-t)
print('k= ',k)

temp={'y':stats.weibull_min.rvs(k,0,np.exp(t),size=100),'N':100}
#D,R=computeConvergence([smWeibull, smWeibullE])
```

    k=  2.156750510195868

# Hierarchical weibull

```python
N=20
M=50
e=np.random.randn(N)*1+2
t=np.random.randn(N)*1+1
#t=-1;e=1
k=4.313501020391736/(np.abs(e-t))

#print('k= ',k)
data={'y':stats.weibull_min.rvs(k,0,np.exp(t),size=(M,N)).T,'N':N,'M':M}
ms=[smWeibullH, smWeibullEH]
D,R=computeConvergence(ms,data,reps=50)
```

    0
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    [2.06550846 3.78643272]

```python
D
```

    array([[1.03845024, 0.94294357, 1.01578379, 0.99120307, 1.02925539,
            1.01027203, 1.05439496, 1.0273838 , 1.03560948, 1.03058553,
            1.05739832, 1.05801606, 1.06082344, 1.05408597, 1.07381892,
            1.04979825, 1.0801661 , 1.05875039, 1.0076158 , 1.02997637,
            1.04445195, 0.97521567, 1.12239099, 0.96830678, 1.11979413,
            0.96245241, 1.05582952, 0.98207235, 1.06741858, 0.98894548,
            1.03653193, 0.9653337 , 1.05480242, 1.03396964, 1.0364089 ,
            0.97402525, 1.01951122, 1.06045532, 1.08821511, 1.0336051 ,
            1.05263734, 1.06267476, 1.11014342, 1.03312802, 1.05125666,
            1.01969242, 0.99545646, 1.03181911, 1.04430819, 1.05982757],
           [       nan,        nan,        nan,        nan,        nan,
                   nan,        nan,        nan,        nan,        nan,
                   nan,        nan,        nan,        nan,        nan,
                   nan,        nan,        nan,        nan,        nan,
                   nan,        nan,        nan,        nan,        nan,
                   nan,        nan,        nan,        nan,        nan,
                   nan,        nan,        nan,        nan,        nan,
                   nan,        nan,        nan,        nan,        nan,
                   nan,        nan,        nan,        nan,        nan,
                   nan,        nan,        nan,        nan,        nan]])

# Information matrix generalized gamma

$$f(y)=\frac{\kappa}{y \Gamma(\rho)}\left(\frac{y}{\theta}\right)^{\kappa \rho}e^{-\left(\frac{y}{\theta} \right)^\kappa}$$

$$\log f(y)=\log \kappa- \log y -\log \Gamma(\rho) +\kappa \rho \log y - \kappa \rho \log \theta -\left(\frac{y}{\theta} \right)^\kappa$$

$$I_{\rho \theta \kappa} = \begin{pmatrix} \psi'(\rho) & \frac{\kappa}{\theta} &- \frac{\psi(\rho)}{\kappa} \\
. & \frac{\rho \kappa^2}{\theta^2} & -\frac{\rho}{\theta}\psi(\rho+1)\\
. & . & \frac{\rho}{\kappa^2}\left(\psi'(\rho+1)+\psi(\rho+1)^2+\frac{1}{\rho}\right)\end{pmatrix} $$

$\rho (\psi'(\rho+1)+\psi(\rho+1)^2+\frac{1}{k})= \rho \psi'(\rho)+\rho \psi(\rho)^2 + 2\psi(\rho) +1$

$E[\log Y]= \log \theta + \psi(\rho)/\kappa$

$E[Y^s]=\theta^s \Gamma(\rho+s/\kappa)/\Gamma(\rho)$

$E[Y^\kappa]=\theta^\kappa \rho$

$E[Y^\kappa \log Y ]=\theta^\kappa \rho (\log \theta + \psi(\rho+1)/\kappa)= \theta^\kappa (\rho \log \theta + \rho \psi(\rho)/\kappa+1/\kappa)$

$E[\log^2 Y]= \log^2 \theta + 2 \log \theta \psi(\rho)/\kappa+(\psi'(\rho)+\psi(\rho)^2)/\kappa^2$

$E[Y^\kappa \log^2 Y]= \theta^\kappa \rho (\log^2 \theta + 2 \log \theta \psi(\rho+1)/\kappa+(\psi'(\rho+1)+\psi(\rho+1)^2)/\kappa^2)$

$E[Y^{2\kappa} \log^2 Y]= \theta^{2\kappa} (\rho+1) (\log^2 \theta + 2 \log \theta \psi(\rho+2)/\kappa+(\psi'(\rho+2)+\psi(\rho+2)^2)/\kappa^2)$

$\mathrm{Var}[\log Y]=\psi'(\rho)/\kappa^2$

$E[(Y/\theta)^\kappa]=\rho$

$\mathrm{Var}[(Y/\theta)^\kappa]=\rho$

$E[\log (Y/\theta)^\kappa]= \psi(\rho)$

$E[\log^2 (Y/\theta)^\kappa]= \psi'(\rho)+\psi(\rho)^2$

$E[(Y/\theta)^\kappa \log (Y/\theta)^\kappa ]= \rho \psi(\rho+1)= \rho \psi(\rho)+1$

$E[(Y/\theta)^\kappa \log^2(Y/\theta)^\kappa ]= \rho (\psi'(\rho+1)+\psi(\rho+1)^2)$

$$I_{\rho \tau \kappa} = \begin{pmatrix} \psi'(\rho) & \kappa &- \frac{\psi(\rho)}{\kappa} \\
. & \rho \kappa^2 & -\rho\psi(\rho+1)\\
. & . & \frac{\rho}{\kappa^2}\left(\psi'(\rho+1)+\psi(\rho+1)^2+\frac{1}{\rho}\right)\end{pmatrix} $$

$$I_{\rho, \tau, \log \kappa} = \begin{pmatrix} \psi'(\rho) & \kappa &- \psi(\rho) \\
. & \rho \kappa^2 & -\kappa\rho\psi(\rho+1)\\
. & . & \rho \left(\psi'(\rho+1)+\psi(\rho+1)^2+\frac{1}{\rho}\right)\end{pmatrix} $$

$$I_{\rho \tau,1/\kappa} = \begin{pmatrix} \psi'(\rho) & \kappa &- \kappa\psi(\rho) \\
. & \rho \kappa^2 & -\kappa^2 \rho A\\
. & . & \kappa^2 \rho B\end{pmatrix} $$

$$I_{\rho \tau,B/(A \kappa)} = \begin{pmatrix} \psi'(\rho) & \kappa &- \kappa\psi(\rho)A/B \\
. & \rho \kappa^2 & -\kappa^2 \rho A^2/B\\
. & . & \kappa^2 \rho A^2/B\end{pmatrix} $$

$$I_{\rho \tau,B/(A \kappa)-\tau} = \begin{pmatrix} \psi'(\rho) & \kappa-\kappa\psi(\rho)A/B &- \kappa\psi(\rho)A/B \\
. & \rho \kappa^2 & 0\\
. & . & \kappa^2 \rho A^2/B-\rho \kappa^2\end{pmatrix} $$

A=\psi(\rho+1)
B=\left(\psi'(\rho+1)+\psi(\rho+1)^2+\frac{1}{\rho}\right)

$\gamma=\tau+\psi(\rho)/\kappa$

$\rho=\omega(\kappa(\gamma-\tau))=\omega$

$$J=\begin{pmatrix}\kappa \omega' &-\kappa \omega' & (\gamma-\tau)\omega'\\ 0&1 &0 \\ 0& 0& 1 \end{pmatrix}$$
$$I_{\gamma \tau \kappa} = J^T\begin{pmatrix} \frac{1}{\omega'} & \kappa &-(\gamma-\tau) \\
. & \omega \kappa^2 & -(\gamma-\tau)\omega-1\\
. & . & \frac{R}{\kappa^2}\end{pmatrix} J $$

$$I_{\gamma \tau \kappa} = \begin{pmatrix} \kappa^2\omega' &0&0 \\
. & \kappa^2(\omega -\omega')& (\gamma-\tau)(\kappa\omega'-\omega)-1\\
. & . & \frac{R}{\kappa^2}-(\gamma-\tau)^2\omega'\end{pmatrix} $$

with $R=\frac{\omega}{\omega'} +\omega \kappa^2 (\gamma-\tau)^2 + 2\kappa (\gamma-\tau)+1$

## Simplyfied Gamma

$$f(y;\rho)=\frac{ y^{\rho-1} e^{-y}}{\Gamma(\rho)}$$

$$\log f(y;\rho)=\rho \log y -\log y -y-\log \Gamma(\rho)$$

$\Gamma(z+1) = \int_0^\infty x^{z} e^{-x}\, dx$

$\Gamma(z+1)/\Gamma(z)=z$

$\frac{d^n}{dx^n}\Gamma(x) = \int_0^\infty t^{x-1} e^{-t} (\ln t)^n \, dt$

$\psi(x)=\log(\Gamma(x))'=\Gamma'(x)/\Gamma(x)$

$E[Y]= \int_0^\infty y^{\rho} e^{-y}\, dy / \Gamma(\rho)=    \Gamma(\rho+1)/ \Gamma(\rho)=\rho$

$E[Y^s]=\Gamma(\rho+s)/ \Gamma(\rho)$

$\mathrm{Var}[Y]=E[Y^2]-E[Y]^2=\rho(\rho+1)-\rho^2=\rho$

$E[\log Y]=\Gamma'(\rho)/\Gamma(\rho)=\psi(\rho)$

$E[Y \log Y]=\Gamma'(\rho+1)/\Gamma(\rho)= \rho \psi(\rho+1)= \rho \psi(\rho)+1$

$E[1/Y]=    \Gamma(\rho-1)/ \Gamma(\rho)=1/(\rho-1)$

$\mathrm{Var}[1/Y]=E[Y^2]-E[Y]^2=\frac{1}{(\rho-2)(\rho-1)^2}$

$E[\log^2 Y]=\Gamma''(\rho)/\Gamma(\rho)=\psi'(\rho)+\psi(\rho)^2$

use $\psi'(x)=(\Gamma'(x)/\Gamma(x))'=\Gamma''(x)/\Gamma(x)-(\Gamma'(x)/\Gamma(x))^2$

$E[Y \log^2 Y]=\Gamma''(\rho+1)/\Gamma(\rho)=\rho(\psi'(\rho+1)+\psi(\rho+1)^2)=\rho\psi'(\rho)+\rho\psi(\rho)^2+2\psi(\rho)$

# Gengamma with $\theta=1$

$$f(y)=\frac{\kappa}{y \Gamma(\rho)}y^{\kappa \rho} e^{-y^\kappa}$$

$$\log f(y)=\log \kappa- \log y -\log \Gamma(\rho) +\kappa \rho \log y  -y^\kappa$$

$$I_{\rho \kappa} = \begin{pmatrix} \psi'(\rho) & - \frac{\psi(\rho)}{\kappa} \\
. & \frac{\rho}{\kappa^2}\left(\psi'(\rho+1)+\psi(\rho+1)^2+\frac{1}{\rho}\right)\end{pmatrix} $$

$$I_{\rho \log\kappa} = \begin{pmatrix} \psi'(\rho) & - \psi(\rho) \\
. & \rho\psi'(\rho)+\rho\psi(\rho)^2+2\psi(\rho)+1\end{pmatrix} $$

$\gamma=\psi(\rho)/\kappa$

$\rho=\omega(\gamma \kappa)$

$1=d \psi(\omega(\gamma))/d \gamma$

$$I_{\gamma \kappa} = \begin{pmatrix} \kappa^2 \omega(\gamma\kappa) & 0 \\
. & \frac{\omega(\gamma\kappa)}{\kappa\omega'(\gamma\kappa)}+ \omega(\gamma\kappa)\gamma^2\kappa+2\gamma+\frac{1}{\kappa^2}\end{pmatrix} $$

$$I_{\gamma \kappa} = \begin{pmatrix} \kappa^2 \omega(\gamma\kappa) & 0 \\
. & \kappa^{-1}E[Y\log^2 Y]+\frac{1}{\kappa^2}\end{pmatrix} $$

TODO check the last result by transformation of $I_{\rho \kappa}$

orthogonal with

```python
import pystan
ggcode='''functions{
//' Naive implementation of the generalized Gamma density.
//' @param x Value to evaluate density at.
//' @param alpha Shape parameter.
//' @param beta Scale parameter.
//' @param nu Tail parameter.
real gengamma_pdf(real x, real k, real b, real q) {
  real d;
  d = q/(b*tgamma(k))*(x/b)^(k*q-1) * exp(-(x/b)^q);
  return d;
}

real gengamma_lpdf(real x, real k, real b, real q) {
  real d;
  d = log(q) - log(b) - lgamma(k) +
    (k*q-1)*(log(x) - log(b)) - (x/b)^q;
  return d;
}

real generalized_gamma_cdf(real x, real k, real b, real q) {
  real d;
  d = gamma_p(k, (x/b)^q);
  return d;
}

real generalized_gamma_lcdf(real x, real k, real b, real q) {
  real d;
  d = log(generalized_gamma_cdf(x, k, b, q));
  return d;
}}'''

model = """
data {
    int<lower=0> N; //nr subjects
    vector<lower=0>[N] yLT;
}parameters {
    real k;
    //real b;
    real q;
}model {
    for (n in 1:N)
        yLT[n]~gengamma(exp(k),exp(0),exp(q));
}
"""
smGengamma = pystan.StanModel(model_code=ggcode+model)
```

    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_50f847b44dc31effb899e4ef907a37a6 NOW.

```python
from scipy import stats
x=np.linspace(0,10,101)[1:]
#k,q,0,b
k=2;b=1;q=3;
plt.plot(x,stats.gengamma.pdf(x,k,q,0,b))
temp={'yLT':stats.gengamma.rvs(k,q,0,b,size=100),'N':100}

fit=smGengamma.sampling(data=temp,chains=6,n_jobs=6,
        seed=1,thin=1,iter=10000,warmup=500)
print(fit)
w=fit.extract()
p=np.exp(w['k'])
#b=np.exp(w['b'])
H=(pg(1,p+1)+np.square(pg(0,p+1))+1/p)/pg(0,p+1)
e=H/np.exp(w['q'])+1
plt.figure()
plt.plot(p,e,'.')
np.corrcoef(p,e)[0,1]
```

```python
from scipy.special import gamma, digamma,polygamma
plt.figure(figsize=(12,4))
g=np.log(b)+digamma(k)/q
c=(polygamma(1,k+1)+polygamma(0,k+1)**2+1/k)*q/polygamma(0,k+1)+np.log(b)
q1=g
q2=np.log(b)
q3=c
#*np.exp(-a)+q2
plt.subplot(1,3,1)
plt.plot(q1,q2,'.')
plt.title(np.corrcoef(q1,q2)[0,1])
plt.subplot(1,3,2)
plt.plot(q1,q3,'.')
plt.title(np.corrcoef(q1,q3)[0,1])
plt.ylim([-1000,1000])
plt.subplot(1,3,3)
plt.plot(q2,q3,'.')
plt.title(np.corrcoef(q2,q3)[0,1]);
plt.ylim([-50,50])
```

# Beta distribution

Parameters $\alpha$ and $\beta$ are orthogonal if

$$\operatorname{E}_X
\left[
 \frac{\partial \log f(X;\alpha,\beta)}{\partial\alpha \ \partial\beta}
 \right]=0$$

The probability density function of Beta distribution parametrized by shape parameters $\alpha$ and $\beta$ is
$$f(X=x;\alpha,\beta)=\frac{ x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha,\beta)}$$

Consider parametrization in terms of logarithm of geometric mean $E[\log X]=\gamma=\psi(\alpha)-\psi(\alpha+\beta)$ and the logarithm of geometric mean of $1-X$: $E[\log (1-X)]=\phi=\psi(\beta)-\psi(\alpha+\beta)$

Then the fisher information matrix of the distribution parametrized by shape parameters is

$$I_{\alpha,\beta}=\begin{pmatrix}\psi'(\alpha)-\psi'(\alpha+\beta) & -\psi'(\alpha+\beta)\\
-\psi'(\alpha+\beta) & \psi'(\beta)-\psi'(\alpha+\beta)
\end{pmatrix}$$

Fisher information matrix when parametrized by $\gamma$ and $\phi$ is

$$I_{\gamma,\phi}=J^\mathrm{T} I_{\alpha,\beta} J$$

Where $J$ is the Jacobian matrix defined as

$$J=\begin{pmatrix}\frac{\partial \alpha}{\partial \gamma} & \frac{\partial \alpha}{\partial \phi}\\
\frac{\partial \beta}{\partial \gamma} & \frac{\partial \beta}{\partial \phi}
\end{pmatrix}$$

Note that $I_{\alpha,\beta}$ can be written as:

$$I_{\alpha,\beta}=\begin{pmatrix}\frac{\partial \gamma}{\partial \alpha} & \frac{\partial \phi}{\partial \alpha} \\ \frac{\partial \gamma}{\partial \beta}  & \frac{\partial \phi}{\partial \beta}
\end{pmatrix}$$

$$\mathrm{Cov}(\gamma,\phi)=J_{12}J_{11}\psi'(\alpha)+J_{21}J_{22}\psi'(\beta)-\psi'(\alpha+\beta)(J_{12}J_{11}+J_{21}J_{22}+J_{21}J_{12}+J_{11}J_{22})$$

$\gamma=\psi(\alpha)-\psi(\alpha+\beta)$

$\phi=\psi(\beta)-\psi(\alpha+\beta)$

$\gamma-\phi=\psi(\alpha)-\psi(\beta)$

$\alpha=\omega(\phi-\psi(\beta))-\beta$

$\beta=\omega(\gamma-\psi(\alpha))-\alpha$

$$\gamma=\frac{\partial \log \mathrm{B}(\alpha,\beta)}{\partial \alpha}=\frac{\partial \log \Gamma(\alpha)}{\partial \alpha}-\frac{\partial \log \Gamma(\alpha+\beta)}{\partial \alpha}$$

$$\phi=\frac{\partial \log \mathrm{B}(\alpha,\beta)}{\partial \beta}=\frac{\partial \log \Gamma(\beta)}{\partial \beta}-\frac{\partial \log \Gamma(\alpha+\beta)}{\partial \beta}$$

$\psi'(\alpha)=\psi'(\alpha+\beta)\frac{\partial \beta}{\partial \alpha}  -\frac{1}{J_{11}}$

$\psi'(\beta)=\psi'(\alpha+\beta)\frac{\partial \alpha}{\partial \beta}  -\frac{1}{J_{22}}$

$I_{\alpha,\beta}=\begin{pmatrix}A+C & C \\ C & B+C
\end{pmatrix}$

$J^{-1}=\begin{pmatrix}A+C & C \\ C & B+C \end{pmatrix}$

$I_{\gamma,\phi}=J^\mathrm{T} I_{\alpha,\beta} J= J^\mathrm{T} J^{-1} J=J$

$$J=\frac{1}{AB+BC+AC}\begin{pmatrix}B+C & -C \\ -C & A+C \end{pmatrix}
= \begin{pmatrix}\frac{1}{A+\frac{BC}{B+C}} & -\frac{1}{A+B+\frac{AB}{C}} \\ -\frac{1}{A+B+\frac{AB}{C}} & \frac{1}{B+\frac{AC}{A+C}} \end{pmatrix}$$

$$J_{11}=(A+C)^{-1}$$
$$J_{12}=J_{21}= C^{-1}$$
$$J_{22}=-(B+C)^{-1}$$

$$\frac{J_{11}J_{22}}{J_{12}J_{21}}=1$$

$$\frac{-C^2}{(A+C)(B+C)}=1$$

$$\mathrm{Cov}(\gamma,\phi)=J_{12}J_{11}A+J_{21}J_{22}B+C(J_{12}J_{11}+J_{21}J_{22}+J_{21}J_{12}+J_{11}J_{22})$$

$$\mathrm{Cov}(\gamma,\phi)=\frac{A}{C(A+C)}-\frac{B}{C(B+C)}+\frac{1}{A+C}-\frac{1}{B+C} +\frac{1}{C} +\frac{1}{C}\frac{-C^2}{(A+C)(B+C)}
= \frac{1}{C}\left(\frac{A}{A+C}-\frac{B}{B+C}+\frac{C}{A+C}-\frac{C}{B+C} +1 +1\right)
= \frac{2}{C}$$

```python
import pystan

model = """
data {
    int<lower=0> N; //nr subjects
    vector<lower=0>[N] yLT;
}parameters {
    real a;
    real b;
}model {
    for (n in 1:N)
        yLT[n]~beta(exp(a),exp(b));
}
"""
smBeta = pystan.StanModel(model_code=model)
```

    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_bd01f563b8035622229a042f898734bd NOW.

```python
from scipy import stats
x=np.linspace(0,1,101)[1:]
plt.plot(x,stats.beta.pdf(x,4,15,0,1))
temp={'yLT':stats.beta.rvs(4,15,0,1,size=100),'N':100}
fit=smBeta.sampling(data=temp,chains=6,n_jobs=6,
        seed=1,thin=4,iter=55000,warmup=5000)
print(fit)
w=fit.extract()
a=np.exp(w['a'])
b=np.exp(w['b'])
```

```python
from scipy.special import gamma, digamma,polygamma,beta
plt.figure(figsize=(12,12))
gA=digamma(a)-digamma(a+b)
gB=digamma(b)-digamma(a+b)
tA=polygamma(1,a)-polygamma(1,a+b)

var=a*b/np.square(a+b)/(a+b+1)
ex=a/(a+b)
q1=ex
q2=var

#q2=g
#k=np.exp(a)
#l=np.exp(b)
#q1=np.log(np.square(k)*digamma(2)+digamma(1))/(2*digamma(2))-g/(polygamma(1,1)+1)

plt.plot(q1,q2,'.')
#plt.ylim([0,1])
#plt.xlim([0,1])
np.corrcoef(q1,q2)[0,1]
```

## Wald distribution Fisher information

$$f(x)=\frac{\alpha}{\sigma \sqrt{2 \pi x^3}}\exp\left(-\frac{(\nu x-\alpha)^2}{2 \sigma^2 x}\right)$$

$E[X]=\alpha/\nu$

$E[1/X]=\nu/\alpha +\sigma^2/\alpha^2$

$$I_{\alpha \sigma \nu} = \begin{pmatrix} \frac{2}{\alpha^2}+\frac{\nu}{\sigma^2 \alpha} & \frac{2}{\sigma \alpha} & \frac{1}{\sigma}\\
. & \frac{1}{\sigma^2} &0\\
. & . & \frac{\alpha}{\sigma^2 \nu}\end{pmatrix} $$

$$I_{\log \alpha,\log \sigma \nu} = \begin{pmatrix} 2 \sigma+\frac{\nu \alpha}{\sigma} & 2 & \frac{1}{\sigma}\\
. & 1 &0\\
. & . & \frac{\alpha}{\sigma^2 \nu}\end{pmatrix} $$

```python
1/(1+pg(0,2)**2/pg(1,1))
```

    0.9019858038517234

```python
pg(0,2)
```

    array(0.42278434)

```python
from scipy.special import gamma
```

```python
gamma(1)
```

    1.0

```python

```


```python
%load_ext autoreload
%autoreload 2

%matplotlib inline
```

```python
#export
from exp.nb_06 import *
```

## ConvNet

Let's get the data and training interface from where we left in the last notebook.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=5899)

```python
x_train,y_train,x_valid,y_valid = get_data()

x_train,x_valid = normalize_to(x_train,x_valid)
train_ds,valid_ds = Dataset(x_train, y_train),Dataset(x_valid, y_valid)

nh,bs = 50,512
c = y_train.max().item()+1
loss_func = F.cross_entropy

data = DataBunch(*get_dls(train_ds, valid_ds, bs), c)
```

```python
mnist_view = view_tfm(1,28,28)
cbfs = [Recorder,
        partial(AvgStatsCallback,accuracy),
        CudaCallback,
        partial(BatchTransformXCallback, mnist_view)]
```

```python
nfs = [8,16,32,64,64]
```

```python
learn,run = get_learn_run(nfs, data, 0.4, conv_layer, cbs=cbfs)
```

```python
get_learn_run??
```

```python
%time run.fit(2, learn)
```

    train: [2.04338578125, tensor(0.3068, device='cuda:0')]
    valid: [0.88439150390625, tensor(0.6807, device='cuda:0')]
    train: [0.4270001171875, tensor(0.8614, device='cuda:0')]
    valid: [0.1689892578125, tensor(0.9470, device='cuda:0')]
    CPU times: user 4.14 s, sys: 1.1 s, total: 5.24 s
    Wall time: 4.85 s

## Batchnorm

### Custom

Let's start by building our own `BatchNorm` layer from scratch.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=6018)

```python
class BatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        # NB: pytorch bn mom is opposite of what you'd expect
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))
        self.register_buffer('vars',  torch.ones(1,nf,1,1))
        self.register_buffer('means', torch.zeros(1,nf,1,1))

    def update_stats(self, x):
        m = x.mean((0,2,3), keepdim=True)
        v = x.var ((0,2,3), keepdim=True)
        self.means.lerp_(m, self.mom)
        self.vars.lerp_ (v, self.mom)
        return m,v

    def forward(self, x):
        if self.training:
            with torch.no_grad(): m,v = self.update_stats(x)
        else: m,v = self.means,self.vars
        x = (x-m) / (v+self.eps).sqrt()
        return x*self.mults + self.adds
```

```python
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    # No bias needed if using bn
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(BatchNorm(nf))
    return nn.Sequential(*layers)
```

```python
#export
def init_cnn_(m, f):
    if isinstance(m, nn.Conv2d):
        f(m.weight, a=0.1)
        if getattr(m, 'bias', None) is not None: m.bias.data.zero_()
    for l in m.children(): init_cnn_(l, f)

def init_cnn(m, uniform=False):
    f = init.kaiming_uniform_ if uniform else init.kaiming_normal_
    init_cnn_(m, f)

def get_learn_run(nfs, data, lr, layer, cbs=None, opt_func=None, uniform=False, **kwargs):
    model = get_cnn_model(data, nfs, layer, **kwargs)
    init_cnn(model, uniform=uniform)
    return get_runner(model, data, lr=lr, cbs=cbs, opt_func=opt_func)
```

We can then use it in training and see how it helps keep the activations means to 0 and the std to 1.

```python
learn,run = get_learn_run(nfs, data, 0.9, conv_layer, cbs=cbfs)
```

```python
with Hooks(learn.model, append_stats) as hooks:
    run.fit(1, learn)
    fig,(ax0,ax1) = plt.subplots(1,2, figsize=(10,4))
    for h in hooks[:-1]:
        ms,ss = h.stats
        ax0.plot(ms[:10])
        ax1.plot(ss[:10])
        h.remove()
    plt.legend(range(6));

    fig,(ax0,ax1) = plt.subplots(1,2, figsize=(10,4))
    for h in hooks[:-1]:
        ms,ss = h.stats
        ax0.plot(ms)
        ax1.plot(ss)
```

```python
learn,run = get_learn_run(nfs, data, 1.0, conv_layer, cbs=cbfs)
```

```python
%time run.fit(3, learn)
```

    train: [0.27697853515625, tensor(0.9140, device='cuda:0')]
    valid: [0.15755572509765625, tensor(0.9502, device='cuda:0')]
    train: [0.091682783203125, tensor(0.9712, device='cuda:0')]
    valid: [0.12868045654296875, tensor(0.9622, device='cuda:0')]
    train: [0.0663201953125, tensor(0.9792, device='cuda:0')]
    valid: [0.1041472900390625, tensor(0.9675, device='cuda:0')]
    CPU times: user 4.42 s, sys: 412 ms, total: 4.83 s
    Wall time: 4.26 s

### Builtin batchnorm

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=6679)

```python
#export
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(nn.BatchNorm2d(nf, eps=1e-5, momentum=0.1))
    return nn.Sequential(*layers)
```

```python
learn,run = get_learn_run(nfs, data, 1., conv_layer, cbs=cbfs)
```

```python
%time run.fit(3, learn)
```

    train: [0.219562109375, tensor(0.9325, device='cuda:0')]
    valid: [0.08801719360351562, tensor(0.9727, device='cuda:0')]
    train: [0.0662408203125, tensor(0.9795, device='cuda:0')]
    valid: [0.065371728515625, tensor(0.9804, device='cuda:0')]
    train: [0.046659765625, tensor(0.9860, device='cuda:0')]
    valid: [0.06360169067382812, tensor(0.9819, device='cuda:0')]
    CPU times: user 3.87 s, sys: 460 ms, total: 4.33 s
    Wall time: 3.81 s

### With scheduler

Now let's add the usual warm-up/annealing.

```python
sched = combine_scheds([0.3, 0.7], [sched_lin(0.6, 2.), sched_lin(2., 0.1)])
```

```python
learn,run = get_learn_run(nfs, data, 0.9, conv_layer, cbs=cbfs
                          +[partial(ParamScheduler,'lr', sched)])
```

```python
run.fit(8, learn)
```

    train: [0.25703958984375, tensor(0.9224, device='cuda:0')]
    valid: [0.14331890869140626, tensor(0.9532, device='cuda:0')]
    train: [0.085831865234375, tensor(0.9725, device='cuda:0')]
    valid: [0.0797807373046875, tensor(0.9772, device='cuda:0')]
    train: [0.059824091796875, tensor(0.9807, device='cuda:0')]
    valid: [0.07554669189453125, tensor(0.9780, device='cuda:0')]
    train: [0.033681728515625, tensor(0.9893, device='cuda:0')]
    valid: [0.060228680419921875, tensor(0.9843, device='cuda:0')]
    train: [0.02097315185546875, tensor(0.9936, device='cuda:0')]
    valid: [0.05780220947265625, tensor(0.9835, device='cuda:0')]
    train: [0.011302626953125, tensor(0.9970, device='cuda:0')]
    valid: [0.04739341125488281, tensor(0.9867, device='cuda:0')]
    train: [0.006279501342773437, tensor(0.9987, device='cuda:0')]
    valid: [0.04542775268554688, tensor(0.9876, device='cuda:0')]
    train: [0.003952417602539063, tensor(0.9996, device='cuda:0')]
    valid: [0.04444574584960938, tensor(0.9876, device='cuda:0')]

## More norms

### Layer norm

From [the paper](https://arxiv.org/abs/1607.06450): "*batch normalization cannot be applied to online learning tasks or to extremely large distributed models where the minibatches have to be small*".

General equation for a norm layer with learnable affine:

$$y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta$$

The difference with BatchNorm is
1. we don't keep a moving average
2. we don't average over the batches dimension but over the hidden dimension, so it's independent of the batch size

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=6717)

```python
class LayerNorm(nn.Module):
    __constants__ = ['eps']
    def __init__(self, eps=1e-5):
        super().__init__()
        self.eps = eps
        self.mult = nn.Parameter(tensor(1.))
        self.add  = nn.Parameter(tensor(0.))

    def forward(self, x):
        m = x.mean((1,2,3), keepdim=True)
        v = x.var ((1,2,3), keepdim=True)
        x = (x-m) / ((v+self.eps).sqrt())
        return x*self.mult + self.add
```

```python
def conv_ln(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=True),
              GeneralRelu(**kwargs)]
    if bn: layers.append(LayerNorm())
    return nn.Sequential(*layers)
```

```python
learn,run = get_learn_run(nfs, data, 0.8, conv_ln, cbs=cbfs)
```

```python
%time run.fit(3, learn)
```

    train: [0.4715575390625, tensor(0.8531, device='cuda:0')]
    valid: [0.11122366943359376, tensor(0.9674, device='cuda:0')]
    train: [0.09401033203125, tensor(0.9712, device='cuda:0')]
    valid: [0.07507783813476562, tensor(0.9785, device='cuda:0')]
    train: [0.0614589013671875, tensor(0.9806, device='cuda:0')]
    valid: [0.06611319580078125, tensor(0.9817, device='cuda:0')]
    CPU times: user 4.79 s, sys: 692 ms, total: 5.48 s
    Wall time: 4.96 s

*Thought experiment*: can this distinguish foggy days from sunny days (assuming you're using it before the first conv)?

### Instance norm

From [the paper](https://arxiv.org/abs/1607.08022):

The key difference between **contrast** and batch normalization is that the latter applies the normalization to a  whole batch of images instead for single ones:

\begin{equation}\label{eq:bnorm}
    y_{tijk} =  \frac{x_{tijk} - \mu_{i}}{\sqrt{\sigma_i^2 + \epsilon}},
    \quad
    \mu_i = \frac{1}{HWT}\sum_{t=1}^T\sum_{l=1}^W \sum_{m=1}^H x_{tilm},
    \quad
    \sigma_i^2 = \frac{1}{HWT}\sum_{t=1}^T\sum_{l=1}^W \sum_{m=1}^H (x_{tilm} - mu_i)^2.
\end{equation}

In order to combine the effects of instance-specific normalization and batch normalization, we propose to replace the latter by the *instance normalization* (also known as *contrast normalization*) layer:

\begin{equation}\label{eq:inorm}
    y_{tijk} =  \frac{x_{tijk} - \mu_{ti}}{\sqrt{\sigma_{ti}^2 + \epsilon}},
    \quad
    \mu_{ti} = \frac{1}{HW}\sum_{l=1}^W \sum_{m=1}^H x_{tilm},
    \quad
    \sigma_{ti}^2 = \frac{1}{HW}\sum_{l=1}^W \sum_{m=1}^H (x_{tilm} - mu_{ti})^2.
\end{equation}

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=7114)

```python
class InstanceNorm(nn.Module):
    __constants__ = ['eps']
    def __init__(self, nf, eps=1e-0):
        super().__init__()
        self.eps = eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))

    def forward(self, x):
        m = x.mean((2,3), keepdim=True)
        v = x.var ((2,3), keepdim=True)
        res = (x-m) / ((v+self.eps).sqrt())
        return res*self.mults + self.adds
```

```python
def conv_in(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=True),
              GeneralRelu(**kwargs)]
    if bn: layers.append(InstanceNorm(nf))
    return nn.Sequential(*layers)
```

```python
learn,run = get_learn_run(nfs, data, 0.1, conv_in, cbs=cbfs)
```

```python
%time run.fit(3, learn)
```

    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    CPU times: user 5.04 s, sys: 624 ms, total: 5.66 s
    Wall time: 5.09 s

*Question*: why can't this classify anything?

Lost in all those norms? The authors from the [group norm paper](https://arxiv.org/pdf/1803.08494.pdf) have you covered:

### Group norm

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=7213)

*From the PyTorch docs:*

`GroupNorm(num_groups, num_channels, eps=1e-5, affine=True)`

The input channels are separated into `num_groups` groups, each containing
``num_channels / num_groups`` channels. The mean and standard-deviation are calculated
separately over the each group. $\gamma$ and $\beta$ are learnable
per-channel affine transform parameter vectorss of size `num_channels` if
`affine` is ``True``.

This layer uses statistics computed from input data in both training and
evaluation modes.

Args:
-    num_groups (int): number of groups to separate the channels into
-    num_channels (int): number of channels expected in input
-    eps: a value added to the denominator for numerical stability. Default: 1e-5
-    affine: a boolean value that when set to ``True``, this module
        has learnable per-channel affine parameters initialized to ones (for weights)
        and zeros (for biases). Default: ``True``.

Shape:
- Input: `(N, num_channels, *)`
- Output: `(N, num_channels, *)` (same shape as input)

Examples::

    >>> input = torch.randn(20, 6, 10, 10)
    >>> # Separate 6 channels into 3 groups
    >>> m = nn.GroupNorm(3, 6)
    >>> # Separate 6 channels into 6 groups (equivalent with InstanceNorm)
    >>> m = nn.GroupNorm(6, 6)
    >>> # Put all 6 channels into a single group (equivalent with LayerNorm)
    >>> m = nn.GroupNorm(1, 6)
    >>> # Activating the module
    >>> output = m(input)

## Fix small batch sizes

### What's the problem?

When we compute the statistics (mean and std) for a BatchNorm Layer on a small batch, it is possible that we get a standard deviation very close to 0. because there aren't many samples (the variance of one thing is 0. since it's equal to its mean).

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=7304)

```python
data = DataBunch(*get_dls(train_ds, valid_ds, 2), c)
```

```python
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(nn.BatchNorm2d(nf, eps=1e-5, momentum=0.1))
    return nn.Sequential(*layers)
```

```python
learn,run = get_learn_run(nfs, data, 0.4, conv_layer, cbs=cbfs)
```

```python
%time run.fit(1, learn)
```

    train: [2.35244171875, tensor(0.1834, device='cuda:0')]
    valid: [397195.2896, tensor(0.2484, device='cuda:0')]
    CPU times: user 1min 8s, sys: 1.3 s, total: 1min 10s
    Wall time: 1min 10s

### Running Batch Norm

To solve this problem we introduce a Running BatchNorm that uses smoother running mean and variance for the mean and std.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=7516)

```python
class RunningBatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds = nn.Parameter(torch.zeros(nf,1,1))
        self.register_buffer('sums', torch.zeros(1,nf,1,1))
        self.register_buffer('sqrs', torch.zeros(1,nf,1,1))
        self.register_buffer('batch', tensor(0.))
        self.register_buffer('count', tensor(0.))
        self.register_buffer('step', tensor(0.))
        self.register_buffer('dbias', tensor(0.))

    def update_stats(self, x):
        bs,nc,*_ = x.shape
        self.sums.detach_()
        self.sqrs.detach_()
        dims = (0,2,3)
        s = x.sum(dims, keepdim=True)
        ss = (x*x).sum(dims, keepdim=True)
        c = self.count.new_tensor(x.numel()/nc)
        mom1 = 1 - (1-self.mom)/math.sqrt(bs-1)
        self.mom1 = self.dbias.new_tensor(mom1)
        self.sums.lerp_(s, self.mom1)
        self.sqrs.lerp_(ss, self.mom1)
        self.count.lerp_(c, self.mom1)
        self.dbias = self.dbias*(1-self.mom1) + self.mom1
        self.batch += bs
        self.step += 1

    def forward(self, x):
        if self.training: self.update_stats(x)
        sums = self.sums
        sqrs = self.sqrs
        c = self.count
        if self.step<100:
            sums = sums / self.dbias
            sqrs = sqrs / self.dbias
            c    = c    / self.dbias
        means = sums/c
        vars = (sqrs/c).sub_(means*means)
        if bool(self.batch < 20): vars.clamp_min_(0.01)
        x = (x-means).div_((vars.add_(self.eps)).sqrt())
        return x.mul_(self.mults).add_(self.adds)
```

```python
def conv_rbn(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(RunningBatchNorm(nf))
    return nn.Sequential(*layers)
```

```python
learn,run = get_learn_run(nfs, data, 0.4, conv_rbn, cbs=cbfs)
```

```python
%time run.fit(1, learn)
```

    train: [0.34816828125, tensor(0.9095, device='cuda:0')]
    valid: [0.1711124267578125, tensor(0.9688, device='cuda:0')]
    CPU times: user 2min 53s, sys: 2.1 s, total: 2min 55s
    Wall time: 2min 58s

This solves the small batch size issue!

### What can we do in a single epoch?

Now let's see with a decent batch size what result we can get.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=8068)

```python
data = DataBunch(*get_dls(train_ds, valid_ds, 32), c)
```

```python
learn,run = get_learn_run(nfs, data, 0.9, conv_rbn, cbs=cbfs
                          +[partial(ParamScheduler,'lr', sched_lin(1., 0.2))])
```

```python
%time run.fit(1, learn)
```

    train: [0.1522569921875, tensor(0.9527, device='cuda:0')]
    valid: [0.140905810546875, tensor(0.9663, device='cuda:0')]
    CPU times: user 11.4 s, sys: 120 ms, total: 11.6 s
    Wall time: 11.5 s

## Export

```python
nb_auto_export()
```

```python

```


## Mesh refinement study using 5PTI

Next, we tested our code on a real biomolecule - bovine pancreatic trypsin inhibitor (PDB code 5PTI).
We ran this experiment on a single CPU node of Pegasus, and the raw result files are located at `/repro-pack/runs/5PTI_convergence`.
This notebook demonstrates how we generated all results presented in section 3.3.
We computed its solvation energy using 5 meshes with the element density ranging from 1 to 16 (see table below).

#### Table: Mesh size and element density of 5 meshes used in the grid refinement study on 5PTI .

| number of elements | element density (# elem / sq Angstrom) |
|:------------------:|:---------------:|
|        3032        |        1        |
|        6196        |        2        |
|        12512       |        4        |
|        25204       |        8        |
|        50596       |        16       |

Other parameters are the same as the previous mesh refinement study using a spherical molecule.

```python
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
from bempp_pbs.postprocess import PLOT_PARAMS, get_df
```

```python
plt.rcParams.update(PLOT_PARAMS)   # update plot style
```

**load results**

Similar to the previous study, we obtained three sets of results:
- `direct`: direct formulation with a block-diagonal preconditioner
- `derivative`: derivative formulation with a mass-lumping preconditioner
- `derivative_mass_matrix`: derivative formulation with a mass-matrix preconditioner

```python
direct_df = get_df('../runs/5PTI_convergence/direct/', formulation='direct', skip4=True)
direct_df
```

      t_total_assembly
      t_total_gmres
      num_iter
      e_solv [kcal/Mol]
      memory [GB]
      t_fmm_init
      t_singular_assembler
      t_assemble_sparse
      t_assembly_other
      t_singular_correction
      t_laplace
      t_helmholtz
      t_avg_laplace
      t_avg_helmholtz
      t_gmres_other

      num_elem

      3032
      9.255245
      41.141969
      44
      -405.298901
      1.245852
      1.994944
      2.517442
      1.489074
      3.253786
      2.509389
      4.010676
      34.101236
      0.021797
      0.185333
      0.520669

      6196
      6.873639
      55.456878
      48
      -352.498510
      1.298352
      1.984072
      0.604215
      1.515710
      2.769642
      4.833908
      6.902839
      42.929911
      0.034514
      0.214650
      0.790220

      12512
      8.224664
      87.453475
      53
      -331.053002
      1.497760
      2.529693
      0.797550
      1.513400
      3.384022
      8.860051
      12.914149
      64.242070
      0.058701
      0.292009
      1.437205

      25204
      11.171512
      171.576826
      72
      -321.430106
      1.694780
      2.761671
      1.381049
      1.528130
      5.500661
      24.296032
      29.508209
      114.710456
      0.098361
      0.382368
      3.062129

      50596
      17.939818
      301.352296
      81
      -317.397593
      2.596536
      3.672400
      2.966928
      1.650310
      9.650179
      35.566245
      64.952653
      194.572055
      0.193311
      0.579083
      6.261343

```python
derivative_df = get_df('../runs/5PTI_convergence/derivative_ex/', formulation='derivative', skip4=True)
derivative_df
```

      t_total_assembly
      t_total_gmres
      num_iter
      e_solv [kcal/Mol]
      memory [GB]
      t_fmm_init
      t_singular_assembler
      t_assemble_sparse
      t_assembly_other
      t_singular_correction
      t_laplace
      t_helmholtz
      t_avg_laplace
      t_avg_helmholtz
      t_gmres_other

      num_elem

      3032
      11.467519
      53.837476
      20
      -389.044489
      1.268228
      3.063636
      1.123780
      2.73863
      4.541472
      3.047618
      4.048155
      46.063559
      0.023001
      0.190345
      0.678145

      6196
      12.328889
      65.472754
      19
      -349.148460
      1.360512
      2.073360
      1.063001
      2.88178
      6.310748
      4.657528
      6.035649
      53.222862
      0.035926
      0.230402
      1.556715

      12512
      16.094679
      92.094376
      19
      -330.440148
      1.611068
      3.515644
      1.685500
      2.79936
      8.094174
      8.467109
      10.176532
      71.267346
      0.060575
      0.308517
      2.183389

      25204
      20.313739
      121.308250
      18
      -321.330782
      1.881372
      3.067885
      2.120460
      2.79688
      12.328514
      13.368390
      16.634562
      87.766347
      0.103966
      0.398938
      3.538950

      50596
      32.908272
      195.402694
      18
      -317.373973
      2.913360
      3.941171
      4.261869
      3.05930
      21.645932
      22.829768
      31.914846
      133.534401
      0.199468
      0.606975
      7.123678

```python
derivative_mass_matrix_df = get_df('../runs/5PTI_convergence/derivative_ex_mass_matrix/', formulation='derivative', skip4=True)
derivative_mass_matrix_df
```

      t_total_assembly
      t_total_gmres
      num_iter
      e_solv [kcal/Mol]
      memory [GB]
      t_fmm_init
      t_singular_assembler
      t_assemble_sparse
      t_assembly_other
      t_singular_correction
      t_laplace
      t_helmholtz
      t_avg_laplace
      t_avg_helmholtz
      t_gmres_other

      num_elem

      3032
      8.878015
      38.294068
      14
      -389.044497
      1.266776
      1.927307
      0.594318
      2.69806
      3.658330
      2.184614
      2.939802
      32.401477
      0.022967
      0.184099
      0.768174

      6196
      10.256916
      50.961094
      15
      -349.148450
      1.381168
      2.041324
      0.730325
      2.68155
      4.803717
      4.241707
      4.847207
      40.772995
      0.035641
      0.218037
      1.099185

      12512
      14.382855
      73.263648
      16
      -330.440136
      1.610184
      3.419241
      1.042230
      2.73290
      7.188484
      6.526180
      8.560723
      56.647088
      0.059449
      0.286096
      1.529658

      25204
      19.547795
      112.650256
      17
      -321.330778
      1.934784
      2.833449
      1.917480
      2.83897
      11.957896
      13.460070
      15.276854
      80.377198
      0.100506
      0.384580
      3.536134

      50596
      32.397936
      179.180266
      17
      -317.373969
      3.027228
      3.722781
      4.091687
      3.05140
      21.532068
      19.940254
      30.031492
      122.258945
      0.197576
      0.584971
      6.949574

The numbers of iteration show that the mass-lumping preconditioner is an effective substitution for the mass-matrix preconditioner for our study, only adding a few more iterations. Therefore, we only reported the results from using the latter in our manuscript.

**Compare with an approximate solution using Richardson extrapolation**

Since an analytical solution is not available for this geometry, the reference values for error estimation come from Richardson extrapolation using the middle three values:
$$
\begin{equation}
\bar{f}=\frac{f_{1} f_{3}-f_{2}^{2}}{f_{1}-2 f_{2}+f_{3}}
\end{equation}
$$

```python
def richardson_extrapolation(f1, f2, f3):
    return (f1*f3 - f2**2) / (f3 - 2*f2 + f1)
```

```python
e_solv = direct_df['e_solv [kcal/Mol]'].values
e_solv_exact = richardson_extrapolation(*e_solv[-2:0:-1])
print('extrapolated solution using direct formulation: ', e_solv_exact)
rel_error_direct = np.abs((e_solv-e_solv_exact)/e_solv_exact)
e_solv = derivative_df['e_solv [kcal/Mol]'].values
e_solv_exact = richardson_extrapolation(*e_solv[-2:0:-1])
print('extrapolated solution using derivative formulation: ', e_solv_exact)
rel_error_derivative = np.abs((e_solv-e_solv_exact)/e_solv_exact)
```

    extrapolated solution using direct formulation:  -313.59764676311323
    extrapolated solution using derivative formulation:  -312.68602853438085

**Relative error of the solvation energy using the direct formulation**

```python
pd.DataFrame({'number of elements': direct_df.index, 'relative error (%)': rel_error_direct*100}).set_index('number of elements')
```

      relative error (%)

      number of elements

      3032
      29.241691

      6196
      12.404705

      12512
      5.566163

      25204
      2.497614

      50596
      1.211727

**Relative error of the solvation energy using the derivative formulation**

```python
pd.DataFrame({'number of elements': derivative_df.index, 'relative error (%)': rel_error_derivative*100}).set_index('number of elements')
```

      relative error (%)

      number of elements

      3032
      24.420170

      6196
      11.661036

      12512
      5.677938

      25204
      2.764675

      50596
      1.499250

The figure below shows that the error of computing solvation energy of 5PTI converges linearly with respect to N, for both direct and derivative formulation.

#### Figure: Mesh convergence study of the solvation energy of bovine pancreatic trypsin inhibitor (PDB code 5PTI), using both direct formulation and derivative formulation. The error is with respect to the extrapolated solution using Richardson extrapolation.

```python
fig = plt.figure(figsize=(3, 2))
ax = fig.add_subplot(111)
N = direct_df.index.values
N_ = np.array((1e3, 1e5))
asymp = N[2] * rel_error_direct[2] / N_
ax.loglog(N, rel_error_direct, linestyle='', marker='o', fillstyle='none', label='direct', color='black')
ax.loglog(N, rel_error_derivative, linestyle='', marker='+', fillstyle='none', label='derivative', color='black')
ax.loglog(N_, asymp, linestyle='--', color='#7f7f7f')
ax.grid(which="both")
ax.set_xlabel('number of elements')
ax.set_ylabel('relative error')
ax.legend()

loc = (3*N[-2]+N[-1])/4
text_loc = np.array((loc, 1.2*N[-2]*rel_error_direct[-2]/loc))
ax.text(text_loc[0], text_loc[1],r'N$^{-1}$', fontsize=8, rotation=-30,rotation_mode='anchor')
plt.tight_layout()
# plt.savefig('../../tex/figs/5PTI_convergence.pdf', dpi=300);
```