# Introducción al aprendizaje profundo (Deep learning) con PyTorch

 En este notebook, se le presentará
PyTorch
, un framework para construir y entrenar redes neuronales. PyTorch en muchos sentidos se comporta como los array y matrices de Numpy. Estos arrays y matrices Numpy, después de todo, son solo
Tensores.
PyTorch toma estos tensores y simplifica el envio de datos a las GPU para el procesamiento más rápido necesario al entrenar redes neuronales. También proporciona un módulo que calcula automáticamente los gradientes (¡para backpropagation!) Y otro módulo específicamente para construir redes neuronales (Neural Networks). En conjunto, PyTorch termina siendo más afin con Python y Numpy/Scipy stack en comparación con TensorFlow y otros frameworks.

## Redes Neuronales - Neural Networks

El aprendizaje profundo (Deep Learning) se basa en redes neuronales artificiales que han existido de alguna forma desde finales de la década de 1950. Las redes se construyen a partir de partes individuales que se aproximan a las neuronas, normalmente llamadas unidades o simplemente "neuronas". Cada unidad tiene una cierta cantidad de entradas ponderadas. Estas entradas ponderadas se suman (una combinación lineal) y luego se pasan a través de una función de activación para obtener la salida de la unidad.

Matemáticamente esto se ve así:

$$
\begin{align}
y &= f(w_1 x_1 + w_2 x_2 + b) \\
y &= f\left(\sum_i w_i x_i +b \right)
\end{align}
$$
Con vectores, este es el producto-punto interno de dos vectores:

$$
h = \begin{bmatrix}
x_1 \, x_2 \cdots  x_n
\end{bmatrix}
\cdot
\begin{bmatrix}
           w_1 \\
           w_2 \\
           \vdots \\
           w_n
\end{bmatrix}
$$

## Tensors

Los cálculos de redes neuronales son solo un montón de operaciones de álgebra lineal en *tensores*, una generalización de matrices. Un vector es un tensor unidimensional, una matriz es un tensor bidimensional, una matriz con tres índices es un tensor tridimensional (imágenes en color RGB, por ejemplo). La estructura de datos fundamental para las redes neuronales son los tensores y PyTorch (así como casi todos los demás marcos de aprendizaje profundo) se basa en tensores.

Con los conceptos básicos cubiertos, es hora de explorar cómo podemos usar PyTorch para construir una red neuronal simple.

```python
# primero, importamos PyTorch
import torch
```

```python
# creamos la funcion de activacion
def activation(x):
    """ Sigmoid activation function

        Arguments
        ---------
        x: torch.Tensor
    """
    return 1/(1+torch.exp(-x))
```

```python
### creamos algunos datos
torch.manual_seed(7) # Asignamos un valor aleatorio para que el analosis se torne predecible

# Features son 5 variables aleatorias normales
features = torch.randn((1, 5))
# pesos verdaderos para los datos usando de nuevo variables aleatorias
weights = torch.randn_like(features)
# se defina el sesgo
bias = torch.randn((1, 1))
```

```python
features
```

    tensor([[-0.1468,  0.7861,  0.9468, -1.1143,  1.6908]])

Arriba generamos datos que podemos usar para obtener el resultado de nuestra red simple. Todo esto es simplemente aleatorio por ahora, en el futuro comenzaremos a usar datos normales. Pasando por cada línea relevante:

`features = torch.randn((1, 5))` creamos un tensor con forma `(1, 5)`, una fila y cinco columnas, que contiene valores distribuidos aleatoriamente de acuerdo con la distribución normal con una media de cero y una desviación estándar de uno.

`weights = torch.randn_like(features)` creamos otro tensor con la misma forma que `features`, nuevamente conteniendo valores de una distribución normal.

por ultimo, `bias = torch.randn((1, 1))` crea un valor único a partir de una distribución normal.

Los tensores de PyTorch se pueden sumar, multiplicar, restar, etc., al igual que las matrices Numpy. En general, usaremos tensores PyTorch de la misma manera que usariamos matrices en Numpy. Sin embargo, vienen con algunos beneficios interesantes, como la aceleración de la GPU, que veremos más adelante. Por ahora, utilicemos los datos generados para calcular la salida de esta simple red de una sola capa.

> ** Ejercicio **: Calcule la salida de la red con las características de entrada "features", ponderaciones "weights" y sesgo "bias". Similar a Numpy, PyTorch tiene un [`torch.sum()`](https://pytorch.org/docs/stable/torch.html#torch.sum) función, así como un método `.sum ()` sobre tensores, para tomar sumas. Utilice la función "activación" definida anteriormente como función de activación.

```python
## Calculemos la salida de esta red usando las caracteristicas(features), los pesos(weights) y el el sesgo(bias) del tensor.

print(torch.sum(features))
print(torch.sum(weights))
print(torch.sum(bias))
print()
print(activation(features.sum()))
print(activation(weights.sum()))
print(activation(bias.sum()))
print()
```

    tensor(2.1626)
    tensor(-1.5621)
    tensor(0.3177)

    tensor(0.8968)
    tensor(0.1733)
    tensor(0.5788)

```python
## Solucion
y = activation(torch.sum(features * weights) + bias)
y = activation((features * weights).sum() + bias)
```

```python
bias.shape
```

    torch.Size([1, 1])

```python
## Solution

y = activation(torch.mm(features, weights.view(5,1)) + bias)
```

Podemos hacer la multiplicación y la suma en la misma operación usando una multiplicación de matrices. En general, se busca usar multiplicaciones de matrices, ya que son más eficientes y aceleradas al usar bibliotecas modernas y computación de alto rendimiento en GPU.

Aquí, queremos hacer una multiplicación matricial de las características y los pesos. Para esto podemos usar [`torch.mm()`](https://pytorch.org/docs/stable/torch.html#torch.mm) or [`torch.matmul()`](https://pytorch.org/docs/stable/torch.html#torch.matmul) que es algo más complicado y soporta la radiodifusión. Si nosotros intentamos hacerlo con `features` y` weights` tal como están, obtendremos un error
```python
>> torch.mm(features, weights)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-13-15d592eb5279> in <module>()
----> 1 torch.mm(features, weights)

RuntimeError: size mismatch, m1: [1 x 5], m2: [1 x 5] at /Users/soumith/minicondabuild3/conda-bld/pytorch_1524590658547/work/aten/src/TH/generic/THTensorMath.c:2033
```

A medida que construye redes neuronales en cualquier marco, verá esto a menudo. Muy a menudo. Lo que sucede aquí es que nuestros tensores no tienen las formas correctas para realizar una multiplicación de matrices. Recuerde que para las multiplicaciones de matrices, el número de columnas en el primer tensor debe ser igual al número de filas en la segunda columna. Tanto las "características" como los "pesos" tienen la misma forma, "(1, 5)". Esto significa que debemos cambiar la forma de los "pesos" para que funcione la multiplicación de matrices.

** Nota: ** Para ver la forma de un tensor llamado `tensor`, usa` tensor.shape`. Si está creando redes neuronales, utilizará este método con frecuencia.

Aquí hay algunas opciones: [`weights.reshape()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.reshape), [`weights.resize_()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.resize_), y [`weights.view()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view).

* `weights.reshape(a, b)` devolverá un nuevo tensor con los mismos datos que "pesos" con tamaño "(a, b)" a veces, ya veces un clon, ya que copia los datos a otra parte de la memoria.
* `weights.resize_(a, b)` devuelve el mismo tensor con una forma diferente. Sin embargo, si la nueva forma da como resultado menos elementos que el tensor original, algunos elementos se eliminarán del tensor (pero no de la memoria). Si la nueva forma da como resultado más elementos que el tensor original, los elementos nuevos no se inicializarán en la memoria. Aquí debo señalar que el guión bajo al final del método denota que este método se realiza ** en el lugar **. Aquí hay un gran hilo del foro de [read more about in-place operations](https://discuss.pytorch.org/t/what-is-in-place-operation/16244) en PyTorch.
* `weights.view(a, b)` devolverá un nuevo tensor con los mismos datos que "weights" con tamaño "(a, b)".

Normalmente ese `.view()`, pero cualquiera de los tres métodos funcionará para esto. Entonces, ahora podemos cambiar la forma de "weights" para tener cinco filas y una columna con algo como "weights.view(5, 1)".

> ** Ejercicio **: Calcula la salida de nuestra pequeña red usando la multiplicación de matrices.

```python
## Calculate the output of this network using matrix multiplication
features
weights = weights.view(5,1)
activation(torch.mm(features, weights) + bias)
```

    tensor([[ 0.1595]])

### Stack them up!

Así es como se puede calcular la salida de una sola neurona. El verdadero poder de este algoritmo ocurre cuando comienzas a apilar estas unidades individuales en capas y pilas de capas, en una red de neuronas. La salida de una capa de neuronas se convierte en la entrada para la siguiente capa. Con múltiples unidades de entrada y unidades de salida, ahora necesitamos expresar los pesos como una matriz.

La primera capa que se muestra en la parte inferior aquí son las entradas, se conoce como ** capa de entrada **. La capa intermedia se llama ** capa oculta ** y la capa final (a la derecha) es la ** capa de salida **. Podemos expresar esta red matemáticamente con matrices nuevamente y usar la multiplicación de matrices para obtener combinaciones lineales para cada unidad en una operación. Por ejemplo, la capa oculta ($h_1 $ y $h_2 $ aquí) se puede calcular

$$
\vec{h} = [h_1 \, h_2] =
\begin{bmatrix}
x_1 \, x_2 \cdots \, x_n
\end{bmatrix}
\cdot
\begin{bmatrix}
           w_{11} & w_{12} \\
           w_{21} &w_{22} \\
           \vdots &\vdots \\
           w_{n1} &w_{n2}
\end{bmatrix}
$$

La salida para esta pequeña red se encuentra tratando la capa oculta como entradas para la unidad de salida. La salida de la red se expresa simplemente

$$
y =  f_2 \! \left(\, f_1 \! \left(\vec{x} \, \mathbf{W_1}\right) \mathbf{W_2} \right)
$$

```python
### Generamos datos
torch.manual_seed(7) # Asignamos un valor aleatorio para que el analosis se torne predecible

# Features son 3 variables aleatorias normales
features = torch.randn((1, 3))

# Define los valores de cada capa in la red
n_input = features.shape[1]     # Number of input units, must match number of input features
n_hidden = 2                    # Number of hidden units
n_output = 1                    # Number of output units

# Pesos para las entradas de la capa oculta
W1 = torch.randn(n_input, n_hidden)
# Pesos de la capa oculta a la capa de salida
W2 = torch.randn(n_hidden, n_output)

# definimos los términos de sesgo para capas ocultas y de salida
B1 = torch.randn((1, n_hidden))
B2 = torch.randn((1, n_output))
```

> **Exercise:** Calculate the output for this multi-layer network using the weights `W1` & `W2`, and the biases, `B1` & `B2`.

```python
## Your solution here
#calculate firts hidden layer
x = activation(torch.mm(features, W1) + B1)
#calculate lastone hidden layer
activation(torch.mm(x,W2) + B2)
```

    tensor([[ 0.3171]])

Si hizo esto correctamente, debería ver la salida `tensor ([[0.3171]])`.

El número de unidades ocultas es un parámetro de la red, a menudo llamado ** hiperparámetro ** para diferenciarlo de los parámetros de ponderaciones y sesgos. Como verá más adelante, cuando analicemos el entrenamiento de una red neuronal, cuantas más unidades ocultas tenga una red y más capas, mejor podrá aprender de los datos y hacer predicciones precisas.

## Numpy to Torch and back

Sección de bonificación especial! PyTorch tiene una gran característica para convertir entre matrices Numpy y tensores Torch. Para crear un tensor a partir de una matriz Numpy, use `torch.from_numpy()`. Para convertir un tensor en una matriz Numpy, use el método `.numpy ()`.

```python
import numpy as np
a = np.random.rand(4,3)
a
```

    array([[ 0.67074982,  0.90466332,  0.46128531],
           [ 0.51230911,  0.18835729,  0.43471674],
           [ 0.22338983,  0.76382028,  0.67405823],
           [ 0.82932055,  0.65622593,  0.0371518 ]])

```python
b = torch.from_numpy(a)
b
```

    tensor([[ 0.6707,  0.9047,  0.4613],
            [ 0.5123,  0.1884,  0.4347],
            [ 0.2234,  0.7638,  0.6741],
            [ 0.8293,  0.6562,  0.0372]], dtype=torch.float64)

```python
b.numpy()
```

    array([[ 0.67074982,  0.90466332,  0.46128531],
           [ 0.51230911,  0.18835729,  0.43471674],
           [ 0.22338983,  0.76382028,  0.67405823],
           [ 0.82932055,  0.65622593,  0.0371518 ]])

La memoria se comparte entre la matriz Numpy y el tensor Torch, por lo que si cambia los valores en el lugar de un objeto, el otro también cambiará.

```python
# Multiply PyTorch Tensor by 2, in place
b.mul_(4)
```

    tensor([[ 2.6830,  3.6187,  1.8451],
            [ 2.0492,  0.7534,  1.7389],
            [ 0.8936,  3.0553,  2.6962],
            [ 3.3173,  2.6249,  0.1486]], dtype=torch.float64)

```python
# Numpy array matches new values from Tensor
a
```

    array([[ 2.6829993 ,  3.61865328,  1.84514124],
           [ 2.04923646,  0.75342916,  1.73886697],
           [ 0.89355934,  3.05528112,  2.69623291],
           [ 3.31728218,  2.62490371,  0.1486072 ]])

Python/PyTorch/Part 1 - Tensors in PyTorch  Spanish (Exercises).ipynb


## 4.4 微积分

事实上，数学公式的范畴极为广泛，我们所熟知的大学数学课程中，微积分、线性代数、概率论与数理统计中数学表达式的符号系统均大不相同。本节将主要介绍如何使用LaTeX对微积分中的数学表达式进行书写和编译。

### 4.4.1 极限

求极限是整个微积分中的基石，例如$\lim_{x\to 2}x^{2}$对应的LaTeX代码为`$\lim_{x\to 2}x^{2}$`。

【**例1**】书写以下求极限的表达式：

$$\lim_{x\to-\infty}\frac{3x^{2}-2}{3x-2x^{2}}=\lim_{x\to-\infty}\frac{x^{2}\left(3-\frac{2}{x^{2}}\right)}{x^{2}\left(\frac{3}{x}-2\right)}=\lim_{x\to-\infty}\frac{3-\frac{2}{x^{2}}}{\frac{3}{x}-2}=-\frac{3}{2}$$

```tex
\documentclass[12pt]{article}
\begin{document}

$$\lim_{x\to-\infty}\frac{3x^{2}-2}{3x-2x^{2}}=\lim_{x\to-\infty}\frac{x^{2}\left(3-\frac{2}{x^{2}}\right)}{x^{2}\left(\frac{3}{x}-2\right)}=\lim_{x\to-\infty}\frac{3-\frac{2}{x^{2}}}{\frac{3}{x}-2}=-\frac{3}{2}$$

\end{document}
```

【**例2**】书写极限$\lim_{\Delta t\to0}\frac{s(t+\Delta t)+s(t)}{\Delta t}$和$\displaystyle{\lim_{\Delta t\to0}\frac{s(t+\Delta t)+s(t)}{\Delta t}}$。

```tex
\documentclass[12pt]{article}
\begin{document}

$\lim_{\Delta t\to0}\frac{s(t+\Delta t)+s(t)}{\Delta t}$ \& $\displaystyle{\lim_{\Delta t\to0}\frac{s(t+\Delta t)+s(t)}{\Delta t}}$

\end{document}
```

### 4.4.2 导数

在微积分中，给定函数$f(x)$后，我们能够将其导数定义为

$$f^\prime(a)=\lim_{x\to a}\frac{f(x)-f(a)}{x-a}$$

用LaTeX书写这条公式为`$$f^\prime(a)=\lim_{x\to a}\frac{f(x)-f(a)}{x-a}$$`，有时候，为了让分数的形式在直观上不显得过大，可以用`$$f^\prime(a)=\lim\limits_{x\to a}\frac{f(x)-f(a)}{x-a}$$`，其中，`\lim`和`\limits`两个命令需要配合使用。

需要注意的是，`f^\prime(x)`中的`\prime`命令是标准写法，有时候也可以写作`f'(x)`。

【**例3**】使用`\prime`命令书写导数的定义$f^\prime(x)=\lim_{\Delta x\to 0}\frac{f(x+\Delta x)-f(x)}{\Delta x}$。

```tex
\documentclass[12pt]{article}
\begin{document}

$$f^\prime(x)=\lim_{\Delta x\to 0}\frac{f(x+\Delta x)-f(x)}{\Delta x}$$

\end{document}
```

【**例4**】书写函数$f(x)=3x^{5}+2x^{3}+1$的导数$f^\prime(x)=15x^{4}+6x^{2}$。

```tex
\documentclass[12pt]{article}
\begin{document}

$$f^\prime(x)=15x^{4}+6x^{2}$$

\end{document}
```

微分在微积分中举足轻重，`\mathrm{d}`为微分符号$\mathrm{d}$的命令，一般而言，微分的标准写法为$\frac{\mathrm{d}^{n}}{\mathrm{d}x^{n}}f(x)$。

【**例5**】书写微分$\frac{\mathrm{d}}{\mathrm{d}x}f(x)=15x^{4}+6x^{2}$、$\frac{\mathrm{d}^{2}}{\mathrm{d}^{2}x}f(x)=60x^{3}+12x$。

```tex
\documentclass[12pt]{article}
\begin{document}

$$\frac{\mathrm{d}}{\mathrm{d}x}f(x)=15x^{4}+6x^{2}$$
$$\frac{\mathrm{d}^{2}}{\mathrm{d}^{2}x}f(x)=60x^{3}+12x$$

\end{document}
```

在微积分中，偏微分符号$\partial$的命令为`\partial`，对于任意函数$f(x,y)$，偏微分的标准写法为$\frac{\partial^{n}}{\partial x^{n}}f(x,y)$或$\frac{\partial^{n}}{\partial y^{n}}f(x,y)$。

【**例6**】书写函数$f(x,y)=3x^{5}y^{2}+2x^{3}y+1$的偏微分$\frac{\partial}{\partial x}f(x,y)=15x^{4}y^{2}+6x^{2}y$和$\frac{\partial}{\partial y}f(x,y)=6x^{5}y+2x^{3}$。

```tex
\documentclass[12pt]{article}
\begin{document}

$$\frac{\partial}{\partial x}f(x,y)=15x^{4}y^{2}+6x^{2}y$$
$$\frac{\partial}{\partial y}f(x,y)=6x^{5}y+2x^{3}$$

\end{document}
```

【**例7**】书写偏导数$z=\mu\,\frac{\partial y}{\partial x}\bigg|_{x=0}$。

```tex
\documentclass[12pt]{article}
\begin{document}

$$z=\mu\,\frac{\partial y}{\partial x}\bigg|_{x=0}$$

\end{document}
```

### 4.4.3 积分

积分的标准写法为$\int_{a}^{b}f(x)\,\mathrm{d}x$，代码为`\int_{a}^{b}f(x)\,\mathrm{d}x`，其中，`\int`表示积分，是英文单词integral的缩写形式，使用`\,`的目的是引入一个空格。

【**例8**】书写积分$\int\frac{\mathrm{d}x}{\sqrt{a^{2}+x^{2}}}=\frac{1}{a}\arcsin\left(\frac{x}{a}\right)+C$和$\int\tan^{2}x\,\mathrm{d}x=\tan x-x+C$。

```tex
\documentclass[12pt]{article}
\begin{document}

$$\int\frac{\mathrm{d}x}{\sqrt{a^{2}+x^{2}}}=\frac{1}{a}\arcsin\left(\frac{x}{a}\right)+C$$
$$\int\tan^{2}x\,\mathrm{d}x=\tan x-x+C$$

\end{document}
```

【**例9**】书写积分$\int_{a}^{b}\left[\lambda_{1}f_{1}(x)+\lambda_{2}f_{2}(x)\right]\,\mathrm{d}x=\lambda_{1}\int_{a}^{b}f_{1}(x)\,\mathrm{d}x+\lambda_{2}\int_{a}^{b}f_{2}(x)\,\mathrm{d}x$和$\int_{a}^{b}f(x)\,\mathrm{d}x=\int_{a}^{c}f(x)\,\mathrm{d}x+\int_{c}^{b}f(x)\,\mathrm{d}x$。

```tex
\documentclass[12pt]{article}
\begin{document}

$$\int_{a}^{b}\left[\lambda_{1}f_{1}(x)+\lambda_{2}f_{2}(x)\right]\,\mathrm{d}x=\lambda_{1}\int_{a}^{b}f_{1}(x)\,\mathrm{d}x+\lambda_{2}\int_{a}^{b}f_{2}(x)\,\mathrm{d}x$$
$$\int_{a}^{b}f(x)\,\mathrm{d}x=\int_{a}^{c}f(x)\,\mathrm{d}x+\int_{c}^{b}f(x)\,\mathrm{d}x$$

\end{document}
```

【**例10**】书写积分

\begin{equation}
\begin{aligned}
V &=2\pi\int_{0}^{1} x\left[1-(x-1)^{2}\right]\,\mathrm{d}x \\
&=2\pi\int_{0}^{2}\left[-x^{3}+2 x^{2}\right]\,\mathrm{d}x \\
&=2\pi\left[-\frac{1}{4} x^{4}+\frac{2}{3} x^{3}\right]_{0}^{2} \\
&=8\pi/3
\end{aligned}
\end{equation}

```tex
\documentclass[12pt]{article}
\begin{document}

\begin{equation}
\begin{aligned}
V&=2\pi\int_{0}^{1} x\left\{1-(x-1)^{2}\right\}\,\mathrm{d}x \\
&=2\pi\int_{0}^{2}\left\{-x^{3}+2 x^{2}\right\}\,\mathrm{d}x \\
&=2\pi\left[-\frac{1}{4} x^{4}+\frac{2}{3} x^{3}\right]_{0}^{2} \\
&=8\pi/3
\end{aligned}
\end{equation}

\end{document}
```

上述介绍的都是一重积分，在微积分课程中，还有二重积分、三重积分等，对于一重积分，LaTeX提供的基本命令为`\int`，二重积分为`\iint`，三重积分为`\iiint`、四重积分为`\iiiint`，当积分为五重或以上时，一般使用`\idotsint`，即$\idotsint$。

【**例11**】书写积分$\iint\limits_{D}f(x,y)\,\mathrm{d}\sigma$和$\iiint\limits_{\Omega}\left(x^{2}+y^{2}+z^{2}\right)\,\mathrm{d}v$。

```tex
\documentclass[12pt]{article}
\begin{document}

$$\iint\limits_{D}f(x,y)\,\mathrm{d}\sigma$$
$$\iiint\limits_{\Omega}\left(x^{2}+y^{2}+z^{2}\right)\,\mathrm{d}v$$

\end{document}
```

在积分中，有一种特殊的积分符号是在标准的积分符号上加上一个圈，可用来表示计算曲线曲面积分，即$\oint_{C}f(x)\,\mathrm{d}x+g(y)\,\mathrm{d}y$，代码为`\oint_{C}f(x)\,\mathrm{d}x+g(y)\,\mathrm{d}y`。

### 练习题

> 打开LaTeX在线系统[https://www.overleaf.com](https://www.overleaf.com/project)或本地安装好的LaTeX编辑器，创建名为LaTeX_practice的项目，并同时新建一个以`.tex`为拓展名的源文件，完成以下几道练习题。

[1] 书写泰勒展开式

\begin{equation}
\begin{aligned}
f\left(x\right)=&\frac{f\left(x_{0}\right)}{0!}+\frac{f'\left(x_{0}\right)}{1!}\left(x-x_{0}\right)^{2} \\
&+\cdots+\frac{f^{\left(n\right)}\left(x_{0}\right)}{n!}\left(x-x_{0}\right)^{n}+R_{n}\left(x\right)
\end{aligned}
\end{equation}

```tex
\documentclass[12pt]{article}
\begin{document}

\begin{equation}
\begin{aligned}
% 在此处书写公式
\end{aligned}
\end{equation}

\end{document}
```

【回放】[**4.3 希腊字母**](https://nbviewer.jupyter.org/github/xinychen/latex-cookbook/blob/main/chapter-4/section3.ipynb)

【继续】[**4.5 线性代数**](https://nbviewer.jupyter.org/github/xinychen/latex-cookbook/blob/main/chapter-4/section5.ipynb)

### License

This work is released under the MIT license.


# Welfare Costs of Short-Run Price Volatility

```python
import os
import numpy as np
from scipy import stats
import pandas as pd
import math
import matplotlib as mpl
import matplotlib.pyplot as plt
import ipywidgets as widgets
from ipywidgets import interact, interact_manual
%matplotlib inline
plt.style.use('seaborn-whitegrid')
mpl.style.use('seaborn')
prop_cycle = plt.rcParams["axes.prop_cycle"]
colors = prop_cycle.by_key()["color"]
```

## **1: Setup**

We consider the incredibly simple static maximization problem:

$$\begin{align}
    U &= \max_{C,E} C^{\alpha}E^{1-\alpha}, \\
    \text{s.t. }I&= C+pE,
\end{align}$$
with the standard simple solution
$$\begin{align}
    C &= \alpha I \\
    E &= (1-\alpha)\dfrac{I}{p}
\end{align}$$

Consider instead the case where $E$ is split into a number of hours $h\in\lbrace 1,...,H\rbrace$ with hourly prices $p_h$. Augment the utility function to accomodate bliss point preferences / price volatility aversion. As outlined in the paper this implies the form
$$\begin{align}
    U = \max_{C,E} C^{\alpha}E^{1-\alpha}+u^{hour}(E_1,...,E_H)+u^{h,scale}(E),
\end{align}$$
with $E=\sum_h E_h$ and $p = \sum_h p_hE_h/E$.

The first order conditions are now on the form:
$$\begin{align}
\lambda &= \alpha \left(\dfrac{C}{E}\right)^{\alpha-1} \tag{1}\\
p_h \lambda &= (1-\alpha)\left(\dfrac{C}{E}\right)^{\alpha}+\dfrac{\partial u^{hour}}{\partial E_h}+\dfrac{\partial u^{h,scale}}{\partial E} \tag{2}
\end{align}$$

Combination of (2) for hours $h,j$ implies
$$\begin{align}
    \dfrac{\partial u^{hour}}{\partial E_h} - \dfrac{\partial u^{hour}}{\partial E_j} = \lambda(p_h-p_j). \tag{3}
\end{align}$$

## **2: Logit example**

The logit-like example applies the functional forms (notation from paper adopted):

$$\begin{align}
    u_h &=  \dfrac{1}{\sigma}\ln\left(\tilde{u}_h\right) \\
    \tilde{u}_h&= \exp\left(\sigma\lambda(p_h-p_t)\right) \\
    \chi_h(\tilde{u}_h) &= \underline{D}+\dfrac{(1-\underline{D})(\overline{D}-\underline{D})}{1-\underline{D}+(\overline{D}-1)\tilde{u}_h}
\end{align}$$

Using the interpretation of the bliss point as $E_t^*$ (the level in absence of price volatility), the implied utility function becomes (notation in the paper):

$$\begin{align}
    u_h^{hour} = \dfrac{1}{\sigma}\left[E_h\ln\left(\dfrac{1-\underline{D}}{\overline{D}-1}\right)-(E_h-\underline{E}_h)\ln\left(E_h-\underline{E}_h\right)-\left(\overline{E}_h-E_h\right)\ln\left(\overline{E}_h-E_h\right)\right]
\end{align}$$

### **2.1: Parameter values**

Set parameter values

```python
par = {'I': 100,
       'alpha': 0.5,
       'eps': 0.00001} # Added small value for numerical stability
```

Add random sample of prices (here random normal draws) and habit shares $g_h\in(0,1)$ (normalized truncated normal draws):

```python
np.var(par['gh'])
```

    7.3238795540792615e-09

```python
par['gh']
```

    array([1.84513965e-04, 2.94051929e-05, 1.65670066e-04, ...,
           1.91401811e-04, 1.24090995e-04, 1.15029137e-04])

```python
par['ph'] = np.random.normal(100,20,8760)
rs_g = stats.truncnorm.rvs(0,100,size=8760)
par['gh'] = rs_g/sum(rs_g)
```

Logit parameters:

```python
par['Upper']= 2
par['Lower']= 0
par['sigma']= 10
```

Add functional forms:

```python
par['chi_h'] = lambda x: par['Lower']+((1-par['Lower'])*(par['Upper']-par['Lower']))/(1-par['Lower']+(par['Upper']-1)*x)
par['utilde_h'] = lambda lambda_,ph,pt: np.exp(par['sigma']*lambda_*(ph-pt))
```

*NB: We can easily verify that the $\chi_h$ function exhibits a fixed-point around 1*

```python
par['chi_h'](1)
```

    1.0

### **2.2: Solution without flexible demand**

We compute $p^*,E^*,C^*,\lambda^*$ under assumption of no price volatility:

```python
sol = {}
sol['pstar'] = sum(par['gh']*par['ph'])
sol['Estar'] = (1-par['alpha'])*par['I']/sol['pstar']
sol['Cstar'] = par['alpha']*par['I']
sol['lambda_star'] = par['alpha']*(sol['Cstar']/sol['Estar'])**(par['alpha']-1)
sol['Ehstar'] = par['gh']*sol['Estar']
sol['Ehlower'] = par['gh']*par['Lower']*sol['Estar']-par['eps']
sol['Ehupper'] = par['gh']*par['Upper']*sol['Estar']+par['eps']
```

Scale of disutility from $u^{hour}$ term in absence of price variation:

```python
sol['uhour_star'] = (1/par['sigma'])*sum(sol['Ehstar']*np.log((1-par['Lower'])/(par['Upper']-1))-(sol['Ehstar']-sol['Ehlower'])*np.log(sol['Ehstar']-sol['Ehlower'])-(sol['Ehupper']-sol['Ehstar'])*np.log(sol['Ehupper']-sol['Ehstar']))
```

### **2.3: Solution with flexible demand**

Logit shares (as a function of lambda):

```python
sol['sh_f'] = lambda lambda_: par['gh'] * par['chi_h'](par['utilde_h'](lambda_,par['ph'],sol['pstar']))
sol['Eh_f'] = lambda lambda_: sol['sh_f'](lambda_)*sol['Estar']
```

Compare endogeonus and exogenous shares (with $\lambda$ fixed at value without price volatility)

```python
pd.DataFrame({'Endogenous shares': pd.Series(sol['sh_f'](sol['lambda_star'])).sort_values().reset_index(drop=True),
              'Exogenous shares': pd.Series(par['gh']).sort_values().reset_index(drop=True)}).plot();
```

Utility as a function of $\lambda$:

```python
sol['uhour_function'] = lambda lambda_: (1/par['sigma'])*sum(sol['Eh_f'](lambda_)*np.log((1-par['Lower'])/(par['Upper']-1))-(sol['Eh_f'](lambda_)-sol['Ehlower'])*np.log(sol['Eh_f'](lambda_)-sol['Ehlower'])-(sol['Ehupper']-sol['Eh_f'](lambda_))*np.log(sol['Ehupper']-sol['Eh_f'](lambda_)))
```

Value in the case of $\lambda^*$ (as a benchmark):

```python
data = pd.DataFrame({'Upper bound': sol['Ehupper'],
              'Actual E': sol['Eh_f'](sol['lambda_star'])})
data['diff'] = data['Upper bound']-data['Actual E']
```

Utility as a function of $\lambda$, including the normalization around zero:

```python
sol['uhour_sum'] = lambda lambda_: sol['uhour_function'](lambda_)-sol['uhour_star']
```

### **2.4: Illustrating the cost function**

**1: Plot the utility as a function of $\lambda$**

```python
lambdagrid = np.linspace(0,1,100)
```

```python
sol['uhour_lambdagrid'] = np.array([sol['uhour_sum'](i) for i in lambdagrid])
plt.plot(sol['uhour_lambdagrid']);
```

With $\lambda$ being the shadow-value of income, however, we can check that the *total* effect on welfare is always positive (otherwise the household simply wouldn't react)

```python
# What's the income the household saves from adjusting to price fluctuations:
sol['shares_lambdagrid'] = np.array([sum(sol['sh_f'](i)*par['ph'])-sol['pstar']*sol['Estar'] for i in lambdagrid])
# What's the effect on utility:
sol['uincome_lambdagrid'] = sol['shares_lambdagrid']*lambdagrid
```

**2: Compare welfare with and without flexible demand**

Without flexible prices:

```python
sol['Cstar']**(par['alpha'])*sol['Estar']**(1-par['alpha'])
```

    5.004835319766224

Assume $\lambda=\lambda^*$. The utility in this case:

```python
E_year_lambdastar = sum(sol['Eh_f'](sol['lambda_star']))
C_year_lambdastar = par['I']-sum(sol['Eh_f'](sol['lambda_star'])*par['ph'])
```

```python
C_year_lambdastar**(par['alpha'])*E_year_lambdastar**(1-par['alpha'])+sol['uhour_sum'](sol['lambda_star'])
```

    5.341557116754611


This notebook is part of https://github.com/AudioSceneDescriptionFormat/splines, see also https://splines.readthedocs.io/.

[back to overview](hermite.ipynb)

# Non-Uniform Cubic Hermite Splines

We have already derived
[uniform cubic Hermite splines](hermite-uniform.ipynb),
where the parameter $t$ ranges from $0$ to $1$.

When we want to use *non-uniform* cubic Hermite splines,
and therefore arbitrary ranges from $t_i$ to $t_{i+1}$,
we have (at least) two possibilities:

* Do the same derivations as in the *uniform* case,
  except when we previously evaluated an expression
  at the parameter value $t=0$,
  we now evaluate it at the value $t=t_i$.
  Of course we do the same with $t = 1 \to t = t_{i+1}$.

* Re-scale the *non-uniform* parameter using
  $t \to \frac{t - t_i}{t_{i+1} - t_i}$
  (which makes the new parameter go from $0$ to $1$)
  and then simply use the results from the *uniform* case.

The first approach leads to more complicated expressions
in the basis matrix and the basis polynomials,
but it has the advantage that the parameter value doesn't have to be re-scaled
each time when evaluating the spline for a given parameter
(which *might* be slightly more efficient).

The second approach has the problem that it doesn't actually work correctly,
but we will see that we can make a slight adjustment to fix that problem
(spoiler alert: we will have to multiply the tangent vectors by $\Delta_i$).

The class
[splines.CubicHermite](../python-module/splines.rst#splines.CubicHermite)
is implemented using the second approach
(because its parent class
[splines.Monomial](../python-module/splines.rst#splines.Monomial)
also uses the re-scaling approach).

We show the second approach here,
but the first approach can be done very similarly,
with only very few changed steps.
The appropriate changes are mentioned below.

```python
import sympy as sp
sp.init_printing(order='grevlex')
```

```python
from utility import NamedExpression, NamedMatrix
```

To simplify the indices in the following derivation,
we are looking at the fifth polynomial segment $\boldsymbol{p}_4(t)$
from $\boldsymbol{x}_4$ to $\boldsymbol{x}_5$, where $t_4 \le t \le t_5$.
The results will be easily generalizable to an arbitrary
polynomial segment $\boldsymbol{p}_i(t)$
from $\boldsymbol{x}_i$ to $\boldsymbol{x}_{i+1}$.

```python
t, t4, t5 = sp.symbols('t t4:6')
```

```python
coefficients = sp.Matrix(sp.symbols('a:dbm4')[::-1])
b_monomial = sp.Matrix([t**3, t**2, t, 1]).T
b_monomial.dot(coefficients)
```

We use the humble cubic polynomial (with monomial basis)
to represent our curve segment $\boldsymbol{p}_4(t)$,
but we re-scale the parameter
to map $t_4 \to 0$ and $t_5 \to 1$:

```python
p4 = NamedExpression('pbm4', _.subs(t, (t - t4) / (t5 - t4)))
```

If you don't want to do the re-scaling,
simply un-comment the next line!

```python
#p4 = NamedExpression('pbm4', b_monomial.dot(coefficients))
```

Either way, this is our polynomial segment ...

```python
p4
```

... and it's derivative/velocity/tangent vectors:

```python
pd4 = p4.diff(t)
pd4
```

The next steps are very similar to what we did in the
[uniform case](hermite-uniform.ipynb),
except that we use $t_4$ and $t_5$ instead of
$0$ and $1$, respectively.

```python
x4 = p4.evaluated_at(t, t4).with_name('xbm4')
x5 = p4.evaluated_at(t, t5).with_name('xbm5')
xd4 = pd4.evaluated_at(t, t4).with_name('xdotbm4')
xd5 = pd4.evaluated_at(t, t5).factor().with_name('xdotbm5')
```

To simplify things,
we define a new symbol $\Delta_4 = t_5 - t_4$,
representing the duration of the current segment.
However, we only use this for simplifying the display,
further calculations are still carried out with $t_i$.

```python
delta = {
    t5 - t4: sp.Symbol('Delta4'),
}
```

```python
display(x4, x5, xd4.subs(delta), xd5.subs(delta))
```

## Basis Matrix

```python
M_H = NamedMatrix(r'{M_{\text{H},4}}', 4, 4)
```

```python
control_values_H = NamedMatrix(
    sp.Matrix([x4.name, x5.name, xd4.name, xd5.name]),
    M_H.name.I * coefficients)
control_values_H
```

```python
substitutions = x4, x5, xd4, xd5
```

```python
control_values_H.subs_symbols(*substitutions).subs(delta)
```

```python
M_H.I = sp.Matrix([
    [expr.expand().coeff(c) for c in coefficients]
    for expr in control_values_H.subs_symbols(*substitutions).name])
M_H.I.subs(delta)
```

```python
print(_.expr)
```

```python
M_H.factor().subs(delta)
```

```python
print(_.expr)
```

## Basis Polynomials

```python
b_H = NamedMatrix(r'{b_{\text{H},4}}', b_monomial * M_H.expr)
b_H.factor().subs(delta).simplify().T
```

Those are the *non-uniform* Hermite basis functions.
Not surprisingly, they are different for each segment,
because generally the values $\Delta_i$ are different
in the non-uniform case.

## Example Plot

To quickly check whether the matrix $M_{H,4}$ does what we expect,
let's plot an example segment:

```python
import numpy as np
```

If we use the same API as for the other splines,
we can reuse the helper functions for plotting
from [helper.py](helper.py):

```python
from helper import plot_spline_2d, plot_tangents_2d
```

The following code re-scales the parameter with
`t = (t - begin) / (end - begin)`.
If you did *not* re-scale $t$ in the derivation above,
you'll have to remove this line.

```python
class HermiteSegment:

    def __init__(self, control_values, begin, end):
        array = sp.lambdify([t4, t5], M_H.expr)(begin, end)
        self.coeffs = array @ control_values
        self.grid = begin, end

    def evaluate(self, t):
        t = np.expand_dims(t, -1)
        begin, end = self.grid
        # If you derived M_H without re-scaling t, remove the following line:
        t = (t - begin) / (end - begin)
        return t**[3, 2, 1, 0] @ self.coeffs
```

```python
vertices = [0, 0], [5, 1]
tangents = [2, 3], [0, -2]
```

We can simulate the *uniform* case
by specifying a parameter range from $0$ to $1$:

```python
s1 = HermiteSegment([*vertices, *tangents], 0, 1)
```

```python
plot_spline_2d(s1, chords=False)
plot_tangents_2d(tangents, vertices)
```

But other ranges should work as well:

```python
s2 = HermiteSegment([*vertices, *tangents], 2.1, 5.5)
```

```python
plot_spline_2d(s2, chords=False)
plot_tangents_2d(tangents, vertices)
```

## Utilizing the Uniform Basis Matrix

If you did *not* re-scale $t$ in the beginning of the derivation,
you can use the matrix $M_{H,i}$ to calculate the monomial coefficients
of each segment (as shown in the example code above) and be done with it.
The following simplification does only apply
if you *did* re-scale $t$.

If you *did* re-scale $t$,
the basis matrix and the basis polynomials
will look very similar to the
[uniform case](hermite-uniform.ipynb),
but they are not quite the same.
This means that simply re-scaling the parameter
is not enough to correctly use the *uniform* results
for implementing *non-uniform* Hermite splines.

However, we can see that the only difference is
that the components associated with
$\dot{\boldsymbol{x}}_4$ and
$\dot{\boldsymbol{x}}_5$ are simply multiplied by $\Delta_4$.
That means if we re-scale the parameter *and*
multiply the given tangent vectors by $\Delta_i$,
we can indeed use the *uniform* workflow.

Just to make sure we are actually telling the truth, let's check that
the control values with scaled tangent vectors ...

```python
control_values_H_scaled = sp.Matrix([x4.name,
                                     x5.name,
                                     (t5 - t4) * xd4.name,
                                     (t5 - t4) * xd5.name])
control_values_H_scaled.subs(delta)
```

... really lead to the same result
as when using the *uniform* basis matrix:

```python
sp.Eq(
    sp.simplify(M_H.expr * control_values_H.name),
    sp.simplify(sp.Matrix([[ 2, -2,  1,  1],
                           [-3,  3, -2, -1],
                           [ 0,  0,  1,  0],
                           [ 1,  0,  0,  0]]) * control_values_H_scaled))
```

The following line will fail if you did *not* rescale $t$:

```python
assert _ == True
```

Long story short,
to implement a *non-uniform* cubic Hermite spline segment,
we can simply re-scale the parameter to a range from $0$ to $1$
(by substituting $t \to \frac{t - t_i}{t_{i+1} - t_i}$),
multiply both given tangent vectors by $\Delta_i = t_{i+1} - t_i$
and then simply use the implementation of
the *uniform* cubic Hermite spline segment.

Another way of looking at this is
to consider the *uniform* polynomial segment $\boldsymbol{u}_i(t)$
and its tangent vector (i.e. first derivative) $\boldsymbol{u}'_i(t)$.
If we want to know the tangent vector after substituting
$t \to \frac{t - t_i}{\Delta_i}$,
we have to use the [chain rule](https://en.wikipedia.org/wiki/Chain_rule)
(with the inner derivative being $\frac{1}{\Delta_i}$):

\begin{equation*}
\frac{d}{dt} \boldsymbol{u}_i\!\left(\frac{t-t_i}{\Delta_i}\right)
=
\frac{1}{\Delta_i} \boldsymbol{u}'_i\!\left(\frac{t-t_i}{\Delta_i}\right).
\end{equation*}

This means the tangent vectors have been shrunk by $\Delta_i$!
If we want to maintain the original lengths of our tangent vectors,
we can simply scale them by $\Delta_i$ beforehand.


# Introduction

In this note we provide an introduction to the creation of a rich class of Gaussian
processes constructed by the following two step procedure

1. Specify a Gaussian source function for a linear PDE
2. Solve the PDE to produce a new Gaussian process

We will discuss the basics of the theory, and show how to implement these
models in `Tenfem`, both to analyticaly solve for the covariance matrix, and
to to solve these models as part of ensemble variational layers in the style
of `tensorflow_probability` layers.

First some basic imports

```python
import numpy as np
import tensorflow as tf
import tensorflow_probability as tfp
tf.keras.backend.set_floatx('float64')
import tenfem
import matplotlib.pyplot as plt
from typing import Callable
from matplotlib.collections import LineCollection
from matplotlib.tri import Triangulation
from tenfem.layers import (AssembleStiffnessMatrix, AssembleLoadVector, SolveDirichletProblem,
                           TriangleMeshInterpolator)

tf_dtype = tf.float64
tfk = tf.keras
tfkl = tf.keras.layers
tfpl = tfp.layers
tfd = tfp.distributions

tf.random.set_seed(248)
np.set_printoptions(precision=3, suppress=True)
num_inducing_points = 50
```

# Background

Solving a linear PDE like

$$
    \mathcal{L}u(x) = f(x),
$$

can be thought of as an operator which maps the source term $f(x)$ to a new function
$u(x)$, this operator is linear and so properties, such as Gaussianity, are preserved.
In this note we explore a little bit more of this feature, in particular it allows
us to regard PDEs as a special form of *mechanistically inspired smoothing kernel*, and
and so make their use much natural for people in the statistical and machine learning
communities who are more likely to be familiar with Gaussian processes.

Formally we might write this convolution as

$$
    u(x) = \mathcal{L}^{-1}f(x),
$$

then if $f(x) \sim \mathcal{GP}(0, k_f(x, x'))$, we have

$$
    u(x) \sim \mathcal{GP}(0, \mathcal{L}^{-1} \otimes \mathcal{L}^{-\top} [k] (x, x') ),
$$

where $\mathcal{L}^{-1} \otimes \mathcal{L}^{-\top}$ is some appropriate notion of
the tensor product of two operators, but what does this all mean? This notebook aims
to demonstrate the basic approach to constructing interesting GP models by using the
functionality of `Tenfem` to explore and build these features.

## Green's functions, kernels and Sobolev spaces

The *Green's function*, say $k_x(y)$, of a linear PDE is a solution to the problem

$$
    \mathcal{L} k_x(y) = \delta(x-y), \qquad y \in \Omega
$$

where $\delta(x-y)$ is the Dirac delta function. Knowing this solution for the Dirac delta source
allows one to obtain the solution for a General source by the convolution

$$
    u(x) = \int_{\Omega} k_x(y) f(y) dy
$$

Lets make this clearer with an example! We shall take our domain to be $\Omega = [0, 1]$ and our
operator will be the *negative Laplacian* $\mathcal{L} = -\Delta = -\frac{\partial^2}{\partial x}$.
We shall also apply zero Dirichlet boundary conditions, then the Green's function is a solution to
the problem

$$
\begin{align}
    &-\frac{\partial^2}{\partial x^2} k_y(x) = \delta_y(x) \\
    & k_y(0) = k_y(1) = 0
\end{align}
$$

then one can integrate this problem twice, applying the boundary condintions and so check
that the Greens function for this problem is given by

$$
    k(x, y) = \begin{cases} x(1-y) \qquad &\text{if } x \leq y \\
    y(1-x) \quad &\text{otherwise.}
    \end{cases}
$$

Which we could also recognise as the covariance operator of the Brownian bridge on $[0, 1]$.
This kernel is infact the reproducing kernel of the reproducing kernel Hilbert space (RKHS)

$$
\mathcal{H}_{0}^{1}([0, 1]) = \{
f \; : \; [0, 1] \rightarrow \mathbb{R} \; : \; f \text{ is absolutely continuous, }
f(0) = f(1) = 0, \, f' \in L^2[0, 1] \}
$$

this is a Sobolev space and is a fundamental part of the functional analytic approach to
studying partial differential equations, but has also had important methodological
applications in modern Gaussian process theory. See [(1)](paulsen) for more on this space.

Returning to the problem with general source term, we can obtain a solution by
the convolution against the kernel

$$
\begin{align}
   u(x) &= \left(-\frac{\partial^2}{\partial x^2}\right)^{-1} f(x)\\
   &=\int_{0}^{x}x(1-y) f(y) dy + \int_{x}^{1}y(1-x)f(y) dy.
\end{align}
$$

```python
def build_interval_mesh_and_element(n_nodes):
    """ Build an `IntervalMesh` on the interval [0, 1] and the reference element. """
    a, b = (0, 1)
    nodes = np.linspace(a, b, n_nodes)[:, np.newaxis]
    elements = np.column_stack((np.arange(0, n_nodes-1), np.arange(1, n_nodes)))
    boundary_elements = [[0, ], [n_nodes-1, ]]
    ref_element = tenfem.reference_elements.IntervalElement(degree=1, dtype=tf.float64)
    return tenfem.mesh.IntervalMesh(nodes, elements, boundary_elements), ref_element

def bb_kernel(x, y):
    """ Covariance function of the Brownian bridge. """
    x = np.atleast_1d(x)
    y = np.atleast_1d(y)
    return tf.squeeze(tf.where(x <= y, x*(1-y), y*(1-x)))
```

## Solution by numerical quadrature

We saw above that we can obtain by an integral convolution, so our first approach
is to use a numerical integration to perform this integration. The `ReferenceElement`
objects in `tenfem` specify a procedure for performing numerical quadrature over an
element

```python
def f(x):
    return 1.

n_nodes = 25
mesh, element = build_interval_mesh_and_element(n_nodes)

xx = np.linspace(0., 1.)
vals = []
for x in xx:
    vals.append(element.quadrature(lambda y: f(y) * bb_kernel(x, y), mesh))

fig, ax = plt.subplots(figsize=(8, 4))
ax.plot(xx, vals, label='Quadrature approximation')
ax.plot(xx, .5*xx*(1-xx), 'o', markeredgecolor='k', label='Exact solution, $u(x) = x(1-x)/2$')
ax.legend()
ax.grid()
plt.show()
```

```python
xx = mesh.nodes[:, 0].numpy().copy()
x1 = xx[:, None]
x2 = xx[None, :]
k = bb_kernel(x1, x2)
kinv = np.zeros_like(k)
kinv[1:-1, 1:-1] = np.linalg.inv(k[1:-1, 1:-1])
fig, axes = plt.subplots(ncols=2)
axes[0].imshow(k)
axes[1].imshow(kinv)

titles = ['Brownian bridge covariance',
          'Brownian bridge precision', ]

for i, ax in enumerate(axes.ravel()):
    for method in [ax.set_xticks, ax.set_yticks]:
        method([])
    ax.set_title(titles[i])
plt.show()
```

## Using `tenfem` to repeat the above

We have written the solution above formally as

$$
    u(x) = \mathcal{L}^{-1} f(x),
$$

but the FEM allows us to approximate the nodal coefficients of $u_h(x) = \sum_{i} (\mathbf{u})_i\varphi_i(x)$
as

$$
   \mathbf{A}\mathbf{u} = \mathbf{f}
$$

so the inverse of this matrix $\mathbf{A}$, should act on vectors in a similar way to the convolution
integral above. We can see this by noting that the stiffness matrix

$$
    (\mathbf{A})_{ij} = (\mathbf{K}^{-1})_{ij}
$$

where $\mathbf{K}$ is the Gram matrix of the Brownian bridge covariance function. This equivalence is
checked below. Note that because of the degeneracy on the boundary where the solution is specified by
the Dirichlet boundary condition we actually only check these matrices on the interior, which corresponds
to the free boundary nodes.

In the code below we show how this model can be solved using our FEM implementation

```python
def solve_poisson_problem(diffusion_coefficient: Callable,
                          source: Callable) -> tf.Tensor:
    """ Function to solve the Poisson problem.

    Args:
        diffusion_coefficient: A callable giving the diffusion
          coefficient of the PDE problem.
        source: A callable giving the source term of the PDE
          problem.

    Returns:
        u: A float `Tensor` giving the solution of the poisson problem, will
          will have shape `[:, mesh.n_nodes, 1]` with the first dimension
          representing batches of the problem.
    """
    A = AssembleStiffnessMatrix(
        diffusion_coefficient = diffusion_coefficient,
        reference_element=element)(mesh.get_tensor_repr())

    b = AssembleLoadVector(
        source=source,
        reference_element=element)(mesh.get_tensor_repr())

    Aint = A[:, 1:-1, 1:-1]
    bint = b[:, 1:-1]

    Aint = tf.linalg.LinearOperatorFullMatrix(Aint)
    uint = Aint.solve(bint)

    batch_size = tf.shape(uint)[0]
    indices = tf.concat((tf.repeat(tf.range(batch_size), repeats=mesh.n_nodes-2)[..., tf.newaxis],
                     tf.tile(tf.range(1, mesh.n_nodes-1), multiples=[batch_size, ])[..., tf.newaxis]), axis=-1)

    return tf.scatter_nd(indices,
                         tf.reshape(uint, [-1]),
                         shape=[batch_size, mesh.n_nodes, ])[..., tf.newaxis]

u = solve_poisson_problem(lambda x: tf.ones_like(x[..., 0]),
                          lambda x: tf.ones_like(x[..., 0]))

# assert(np.allclose(Aint, kinv[1:-1, 1:-1]))
```

```python
fig, ax = plt.subplots(figsize=(8, 4))

ax.plot(xx, .5*xx*(1-xx), '-', label='Exact solution, $u(x) = x(1-x)/2$')
ax.plot(mesh.nodes[:, 0], u[0, :, 0], 'X',
        markeredgecolor='k',
        markersize=10,
        label='FEM Nodal Solution')

ax.legend()
ax.grid()
plt.show()
```

## GP sources

The above demonstrates the solution of the FEM using the integral
convolution, we are more interested though in the case where
the source, $f(x)$, is a Gaussian process.

Our simple `solve_poisson_problem` above requires that the source
function be callable, since it will be evaluated at the points
`element.get_quadrature_nodes(mesh)`, so we need to create a GP
object that is callable. The most natural way of doing this is
through the *Gaussian process inducing point* method. We assume
that the reader is already familiar with these techniques, but
for more information on these methods see [[Titsias 2009]](link) or
[[Hensman]](link). We shall use the
`VariationalGaussianProcess` layer from the [`layers`](link_to_mod)
module of the `TensorflowProbability` package. On first sight this
might look a little clunky, but essentially we

1. Wrap `Kernel` object inside of a `keras.layers.Layer` object to hold parameters of the kernel.
2. This layer gets passed into the `__init__` method of the `VariationalGussianProcess` layer,
along with methods to initialize the inducing variables.
3. The layer is a callable which when called with a `Tensor` representing a
set of index points $\mathbf{X}$ will return the *distribution* defined by
the finite dimensional distributions of the Gaussian process evaluated at these
index points, that is the output of the layer
```
q_f = vgp_layer(X)
```
is the distribution
$$
    q(\mathbf{f}) = \int p(\mathbf{f} \mid \mathbf{u} ) q(\mathbf{u} \mid \mathbf{Z}) d\mathbf{u},
$$
where $\mathbf{f} = (f(\mathbf{x}_1), \ldots, f(\mathbf{x}_N))^{\top}$ for $\mathbf{x}_i \in \mathbf{X}$,
and $\mathbf{Z}$ is the set of inducing points.

```python
class MaternFiveHalvesProvider(tf.keras.layers.Layer):
    """ Layer which provides a MaternFiveHalves kernel. """
    @property
    def kernel(self):
        return tfp.math.psd_kernels.MaternFiveHalves(
            length_scale=0.01*tf.ones([1], dtype=tf_dtype))
```

```python
num_inducing_points = 5

z = np.linspace(0, 1, num_inducing_points)[:, None]

# initialize q(u) at the prior
kernel_provider = MaternFiveHalvesProvider()
kzz = kernel_provider.kernel.matrix(z, z)
scale = tf.linalg.cholesky(kzz).numpy()

vgp_layer = tfp.layers.VariationalGaussianProcess(
    num_inducing_points=num_inducing_points,
    kernel_provider=kernel_provider,
    event_shape=[1],
    inducing_index_points_initializer=tf.constant_initializer(z),
    variational_inducing_observations_scale_initializer=tf.constant_initializer(scale),
    jitter=1e-4)
```

```python
fig, ax = plt.subplots(figsize=(8, 4))

num_samples = 30
u = solve_poisson_problem(lambda x: .1 * tf.ones_like(x[..., 0]),
                          source = lambda x: vgp_layer(x).sample(num_samples))

ax.plot(mesh.nodes[:, 0], u[..., 0].numpy().T, 'C3-', alpha=0.3)

ax.set_title('Samples from GP-forced Poisson problem')
ax.grid()
plt.show()
```

The above gives us the necessary tenchiques to simulate solutions of the
GP by sampling from the inducing point representation. In practice this
is probably the most frequent use case when we allow for more complex
models, or non-linear observation models. However, we might be interested
in deriving a representation of the process above, and not just the ability
to draw samples from it.

Formally we wrote the kernel above as

$$
  k_u(x, x') = \mathcal{L}^{-1} \otimes \mathcal{L}^{-\top} [k_f] (x, x')
$$

where $k_f$ is the kernel of the source function. For our example above
this is the double integral

$$
    k_u(x, x') = \int_{\Omega \times \Omega} k_{BB}(x, y) k_{BB}(x', y') k_f(y, y') dy dy'
$$

or using the FEM approach we can get the distribution of the nodal coefficients from

$$
    \mathbf{K}_u = \mathbf{A}^{-1} \mathbf{K}_{f} \mathbf{A}^{-\top}
$$

where $\mathbf{K}_f$ is the covariance matrix of the *load vector* -- note this
isn't the Gram matrix of $k_f$, instead it has entries of the form

$$
(\mathbf{K}_f)_{ij} = \int_{\Omega \times \Omega} \varphi_{i} (x) \varphi_j(x') k_{f}(x, x') dxdx'.
$$

We should also note that while in the case of the single integrals used to assemble the stiffness
and mass matrix the small common support between the nodal basis functions leads to a high degree
of sparsity, this is no longer the case for the double integral which will in general be dense.

# More general model

To arrive at the analytical solution for the kernel in the above example several things had to go right; we needed a sufficiently simple domain, a simple diffusion coefficient etc. In practice these conditions are unlikely to hold -- we want to model processes on complex, interesting domains, and we want to allow for complex and varying diffusion coefficients, and so we will typically have to solve the problem numerically, fortunately this is exactly the kind of problem we have set up to handle.

Below we consider the following PDE

$$
    -\epsilon \Delta u(x) + \mathbf{w}^{\top} u(x) = f(x),
$$

where $f(x) \sim GP(0, k(x, x'))$, and $\mathbf{w}$ is a vector in $\mathbb{R}^2$. This equation can be
interpretted as the stationary solution of an environment with a constant input source of contaminants
described by $f(x)$, andthese contaminants then diffuse at the rate of $\epsilon$, and are transported
in the direction of $\mathbf{w}$.

```python
class MaternFiveHalvesProvider(tf.keras.layers.Layer):
    """ Layer which provides a MaternFiveHalves kernel. """
    @property
    def kernel(self):
        return tfp.math.psd_kernels.MaternFiveHalves(
            length_scale=0.2*tf.ones([1], dtype=tf_dtype))

def build_gp_layer(mesh):
    """ Simple function to build an SVGP layer. """
    # select the initial inducing points uniformly from the mesh node
    z_indices = np.random.choice(mesh.interior_node_indices.numpy(),
                                 size=num_inducing_points,
                                 replace=False)
    z = mesh.nodes.numpy()[z_indices]

    # initialize q(u) at the prior
    kernel_provider = MaternFiveHalvesProvider(dtype=tf.float64)
    kzz = kernel_provider.kernel.matrix(z, z)
    scale = tf.linalg.cholesky(kzz).numpy()

    gp_layer = tfp.layers.VariationalGaussianProcess(
        num_inducing_points=num_inducing_points,
        kernel_provider=kernel_provider,
        event_shape=[1],
        inducing_index_points_initializer=tf.constant_initializer(z),
        variational_inducing_observations_scale_initializer=tf.constant_initializer(scale),
        jitter=1e-4)
    return gp_layer
```

```python
mesh = tenfem.mesh.examples.star(0.1, dtype=tf_dtype)
low_res_mesh = tenfem.mesh.examples.star(0.25, dtype=tf_dtype)
element = tenfem.reference_elements.TriangleElement(degree=1, dtype=tf_dtype)
```

```python
inputs = tfkl.Input(2)
f = build_gp_layer(mesh)(inputs)
source = tfk.Model(inputs=inputs, outputs=f, name='source')
# source.summary()
```

    WARNING:tensorflow:From /Users/danieltait/anaconda3/envs/tenfem/lib/python3.7/site-packages/tensorflow/python/ops/linalg/linear_operator_lower_triangular.py:158: calling LinearOperator.__init__ (from tensorflow.python.ops.linalg.linear_operator) with graph_parents is deprecated and will be removed in a future version.
    Instructions for updating:
    Do not pass `graph_parents`.  They will  no longer be used.

    /Users/danieltait/anaconda3/envs/tenfem/lib/python3.7/site-packages/tensorflow_probability/python/distributions/gaussian_process.py:311: UserWarning: Unable to detect statically whether the number of index_points is 1. As a result, defaulting to treating the marginal GP at `index_points` as a multivariate Gaussian. This makes some methods, like `cdf` unavailable.
      'Unable to detect statically whether the number of index_points is '

```python
gp_pde = tenfem.layers.LinearEllipticOperator(
    diffusion_coefficient=lambda x: .1*tf.ones_like(x[..., 0]),
    source=source,
    transport_vector_field = lambda x: (tf.ones_like(x)[..., tf.newaxis]
                                        * tf.constant([-0.3, -1.], dtype=tf_dtype)),
    boundary_values=lambda x: tf.ones_like(x[..., 0, tf.newaxis]),
    reference_element=element)

tf.random.set_seed(123)
u = gp_pde(mesh.get_tensor_repr())

tf.random.set_seed(123)
f = source(mesh.nodes)

fig, axes = plt.subplots(ncols=3, figsize=(12, 4))

axes[0].tricontourf(mesh._triang, f[:, 0], levels=20, cmap='Spectral')

axes[1].triplot(mesh._triang, alpha=0.5)
np.random.seed(123)
xs = mesh.nodes.numpy()[np.random.choice(mesh.n_nodes, size=150, replace=False)]
axes[1].quiver(xs[:, 0], xs[:, 1], -0.3*np.ones_like(xs[:, 0]), -1*np.ones_like(xs[:, 1]),
               scale=0.07,
               scale_units='dots')

axes[2].tricontourf(mesh._triang, u[:, 0], levels=20, cmap='inferno')

for ax in axes:
    ax.add_collection(LineCollection(tf.gather(mesh.nodes, mesh.boundary_elements),
                      colors='k', label='$\partial \Omega$'))
    ax.set_aspect('equal'); ax.axis('off')
    ax.set_xlim((-1.75, 1.75))
    ax.set_ylim((ax.get_xlim()))

plt.show()
```

# Introduction

In the [Notebook](hello_poisson.ipynb) we introduced the basic process of
solving the Poisson problem with constant source. This seemed to work just fine,
but giving the prior existence of FEM solving software packages why have we gone
to all the additional work of implementing these methods inside of the Tensorflow
ecosystem? In this notebook we being to make this abit clearer by considering a
much more interesting class of models obtained by replacing the relatively
boring source of the previous examples with the tensorflow implementation of the
`GaussianProcess` class. That is we shall consider the solution of PDE systems of the
form

$$
\begin{align}
    -\nabla ( a(x) \nabla u(x) ) &= f(x) \qquad x \in \Omega \\
    u(x) &= g(x) \qquad x \in \partial\Omega \\
\end{align}
$$

where the source term, $f(x)$, and the boundary condition $g(x)$, are both
given independent Gaussian process priors.

$$
\begin{align}
    f(x) &\sim \mathcal{GP}(0, k_f), \\
    g(x) &\sim \mathcal{GP}(0, k_g)
\end{align}
$$

where the source function is now a mean zero Gaussian process with kernel function $k$.
We show that it is simple to put together this model, and for the cost of implementing
the FEM solver we have gained access to the full set of statistical and machine learning
models implemented in TensorFlow

### Domain creation
Before proceeding to define this part of the model, lets create our problem domain
$\Omega$, again we will use a simple example mesh which is the unit square
$\Omega = [0, 1] \times [0, 1]$. We will also rescale this domain to give our
GP variables some "wiggle" room.

```python
mesh = tenfem.mesh.examples.square(20, 20, dtype=tf_dtype)
# rescale the mesh a bit
mesh = tenfem.mesh.TriangleMesh(3 * mesh.nodes,
                                mesh.elements,
                                mesh.boundary_elements)
mesh.add_matplotlib_tri()
```

# Creating the GP model

## The kernel function
In this example we shall take the kernel function of our Gaussian process to be popular
exponentiated quadratic, or radial basis function (RBF) kernel

$$
    k(x, x') = \alpha^2 \exp\left(-\frac{1}{2} \sum_{d=1}^D (x_d - x'_d ) / \ell_d^2 \right)
$$

where $\alpha$ is the *amplitude* parameter, and $\ell$ is the *length-scale* parameter.

We want to implement the GP model inside of the `tf.keras` functional API using the layers
created by the `tensorflow_probability` team. To do so we are going to need a layer whose
job is provide the kernel function to the Gaussian process. This layers job is to hold the
model parameters, and simple layer class to do this job taken from
the [tfp examples]() is given below

```python
class RBFKernelFn(tf.keras.layers.Layer):
    def __init__(self, **kwargs):
        super(RBFKernelFn, self).__init__(**kwargs)
        dtype = kwargs.get('dtype', None)

        self._amplitude = self.add_weight(
                initializer=tf.constant_initializer(0),
                dtype=dtype,
                name='amplitude')

        self._length_scale = self.add_weight(
                initializer=tf.constant_initializer(0.5),
                dtype=dtype,
                name='length_scale')

    @property
    def kernel(self):
        return tfp.math.psd_kernels.ExponentiatedQuadratic(
          amplitude=tf.nn.softplus(0.1 * self._amplitude),
          length_scale=tf.nn.softplus(0.1 * self._length_scale))
```

## `VariationalGaussianProcess`

Read more about these process at ... first we are just going to define a function
to build these layers, and then give a bit more of a recap of what is going on

```python
def build_gp_layer(kernel_fn='RBF'):
    """ Simple function to quickly build a SVGP layer. """
    if kernel_fn != 'RBF':
        raise NotImplementError('Add more kernel functions!')

    # select the initial inducing points uniformly from the mesh node
    z_indices = np.random.choice(mesh.interior_node_indices.numpy(),
                                 size=num_inducing_points,
                                 replace=False)
    z = mesh.nodes.numpy()[z_indices]

    # initialize q(u) at the prior
    kernel_provider = RBFKernelFn(dtype=tf.float64)
    kzz = kernel_provider.kernel.matrix(z, z)
    scale = tf.linalg.cholesky(kzz).numpy()

    gp_layer = tfp.layers.VariationalGaussianProcess(
        num_inducing_points=num_inducing_points,
        kernel_provider=RBFKernelFn(),
        event_shape=[1],
        inducing_index_points_initializer=tf.constant_initializer(z),
        variational_inducing_observations_scale_initializer=tf.constant_initializer(scale),
        jitter=1e-4)
    return gp_layer
```

A Gaussian process determines the distribution of finite samples
$\mathbf{f} = (f(\mathbf{x}_1), \ldots, f(\mathbf{x}_n) )$, a sparse
variational Gaussian process introduces an additional set of
*inducing variables* denoted by $\mathbf{u}$ with distribution
$q(\mathbf{u})$ and then approximates the distribution over the
complete set of $\mathbb{f}$ by the factorised

$$
    p(\mathbf{f}, \mathbf{u}) =
    p(\mathbf{f} \mid \mathbf{u}) q(\mathbf{u}).
$$

For Gaussian observations the parameters of $q(\mathbf{u})$ can
be determined analytically. For more complex models these are
instead freely optimised, in which case we specify

$$
    q(\mathbf{u}) = \mathcal{N}(\mathbf{u} \mid \mathbf{m}, \mathbf{LL}^{\top})
$$

where $\mathbf{L}$ is a lower-triangular matrix. For
`num_inducing_points = M` this is a total of
$M + M(M+1)/2$ parameters.

In the examples below we use the
*inducing points variables* which are additional samples of the
process $f(\mathbf{x})$ drawn at locations
$\mathbf{Z} = \{ \mathbf{z}_m \}_{m=1}^M$. This gives the
following set of parameters

* $\mathbf{Z}$ -- the `inducing_point_locations`
* $\mathbf{m}$ -- the `variational_inducing_observations_loc`
* $\mathbf{L}$ -- the `variational_inducing_observations_scale`

this variables can be accessed by passing a variable through the GP
and querying the output. This is a bit sketchy, the name of the
object returned by a GP layer is a mess of transposes and transformed
distribution which is a little hard to parse, but if we print the
variables we see they are what we would expect

```python
inputs = tfkl.Input(2)
gp_layer = build_gp_layer()
f = gp_layer(inputs)
for item in f.variables:
    print(item.name)
```

    /Users/danieltait/anaconda3/envs/tenfem/lib/python3.7/site-packages/tensorflow_probability/python/distributions/gaussian_process.py:311: UserWarning: Unable to detect statically whether the number of index_points is 1. As a result, defaulting to treating the marginal GP at `index_points` as a multivariate Gaussian. This makes some methods, like `cdf` unavailable.
      'Unable to detect statically whether the number of index_points is '

    variational_gaussian_process_2/inducing_index_points:0
    variational_gaussian_process_2/variational_inducing_observations_loc:0
    variational_gaussian_process_2/variational_inducing_observations_scale:0
    variational_gaussian_process_2/mean:0
    variational_gaussian_process_2/observation_noise_variance:0
    amplitude:0
    length_scale:0

What we want is a `tfpl` layer which will return a
`tfd.distributions.VariationalGaussianProcess` object.

We are going to initialise the locations of the inducing points at the a
selection of randomly sampled nodes on the mesh, and the distribution
$q(\mathbf{u})$ at the prior

We are also going to add a further layer transforming our inputs to the parameters
of the kernel function
```
    Input ---> Dense(2) ---> VariationalGaussianProcess
```
This layer does simple rescaling of the parameters, but obviously one is
free to extend this idea adding more and more layers to warp the input space
which will be points in $\Omega$. Clearly this approach to model building can be
easily adapted to join multiple spaces etc.

We leave the exploration of the full possibilities for the reader and future notebooks
and for now just implement the simple model shown above. We can put the source
together as a full model and use the `.summary()` method to check everything is
working as expected.

```python
inputs = tfkl.Input(2)
# x = tfkl.Dense(2, use_bias=False)(inputs)  # scales inputs (like a length_scale)
f = build_gp_layer('RBF')(inputs)
source = tfk.Model(inputs=inputs, outputs=f, name='source')
source.summary()
```

    Model: "source"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    input_3 (InputLayer)         [(None, 2)]               0
    _________________________________________________________________
    variational_gaussian_process ((None, 1), (None, 1))    44
    =================================================================
    Total params: 44
    Trainable params: 44
    Non-trainable params: 0
    _________________________________________________________________

# Implementing the FEM model

We have the following steps
1. Define the source model
2. Define the Dirichlet boundary condition model
3. Define the diffusion coefficient model
4. Combine

```python
element = tenfem.reference_elements.TriangleElement(degree=2, dtype=tf_dtype)
quad_mesh = tenfem.mesh.triangle.convert_linear_to_quadratic(mesh)

diffusion_coefficient = lambda x: tf.ones_like(x[..., 0])[..., tf.newaxis] # tfkl.Dense(1, activation='softplus')
```

### The boundary condition $u_D$

We are also going to assign a GP to the values of $u$ on the boundary, that
is

$$
    u_{D}(x) \sim GP(0, k_D(x, x'))
$$

and as with the source model we make this using the `VariationalGaussianProcessLayer`

```python
# GP model to give the boundary values
inputs = tfkl.Input(2)
f = build_gp_layer('RBF')(inputs)
ud = tfk.Model(inputs=inputs, outputs=f, name='boundary')
ud.summary()
```

    Model: "boundary"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    input_4 (InputLayer)         [(None, 2)]               0
    _________________________________________________________________
    variational_gaussian_process ((None, 1), (None, 1))    44
    =================================================================
    Total params: 44
    Trainable params: 44
    Non-trainable params: 0
    _________________________________________________________________

### The full latent GP PDE

```python
mesh_provider = tenfem.layers.MeshProvider(quad_mesh, element)

ud = lambda x: tf.zeros_like(x[..., 0])[..., tf.newaxis]

gp_pde = tenfem.layers.LinearEllipticOperator(
    diffusion_coefficient=diffusion_coefficient,
    source=source,
    boundary_values=ud,
    reference_element=element)

interp = tenfem.layers.TriangleMeshInterpolator(
    tri_finder = mesh._triang.get_trifinder(),
    reference_element=element)

# create the model using the tf.keras functional API
inputs = tfkl.Input(shape=(2, ))
mesh_tensor_repr = mesh_provider(inputs)

u = gp_pde(mesh_tensor_repr)  # solution of the model at the tensor nodes
outputs = interp([inputs, u, mesh_tensor_repr])

gp_pde_model = tfk.Model(inputs=inputs, outputs=outputs)
gp_pde_model.summary()
```

    Model: "model"
    __________________________________________________________________________________________________
    Layer (type)                    Output Shape         Param #     Connected to
    ==================================================================================================
    input_5 (InputLayer)            [(None, 2)]          0
    __________________________________________________________________________________________________
    mesh_provider (MeshProvider)    ((1521, 2), (722, 6) 0           input_5[0][0]
    __________________________________________________________________________________________________
    linear_elliptic_opeartor (Linea (1521, 1)            44          mesh_provider[0][0]
                                                                     mesh_provider[0][1]
                                                                     mesh_provider[0][2]
                                                                     mesh_provider[0][3]
    __________________________________________________________________________________________________
    trimesh_interpolate (TriangleMe None                 0           input_5[0][0]
                                                                     linear_elliptic_opeartor[0][0]
                                                                     mesh_provider[0][0]
                                                                     mesh_provider[0][1]
                                                                     mesh_provider[0][2]
                                                                     mesh_provider[0][3]
    ==================================================================================================
    Total params: 44
    Trainable params: 44
    Non-trainable params: 0
    __________________________________________________________________________________________________

lets sample some realisations of this model to see what sort of patterns we can realise

```python
xx = np.linspace(0., np.max(mesh.nodes[:, 0]))
yy = np.linspace(0., np.max(mesh.nodes[:, 1]))
xx, yy = np.meshgrid(xx, yy)
points = np.column_stack((xx.ravel(), yy.ravel()))

fig, axes = plt.subplots(ncols=3, figsize=(10, 5))

for i in range(3):
    #u = interp([points, source(quad_mesh.get_tensor_repr()), quad_mesh.get_tensor_repr()])
    #u = u.numpy().reshape(xx.shape)
    u = gp_pde_model(points).numpy().reshape(xx.shape)
    axes[i].contourf(xx, yy, u, levels=20)
for ax in axes.flatten():
    ax.set_aspect('equal')
```

```python
assert(False)
```

### references

[1] Paulsen, V., & Raghupathi, M. (2016). An Introduction to the Theory of Reproducing Kernel Hilbert Spaces (Cambridge Studies in Advanced Mathematics). Cambridge: Cambridge University Press. doi:10.1017/CBO9781316219232

```python

```


# Límites e continuidade

Ademais de representar graficamente funcións dunha ou varias variables, o módulo **Sympy** pode empregarse para obter as singularidades dunha función, o seu dominio e tamén o seu rango. Como veremos nesta práctica, tamén poderemos calcular límites nunha ou en varias variables, e comprobar se unha función é continua nun punto dado. Nesta práctica revisaremos as funcións básicas para poder facer este tipo de cálculos.

### Obxectivos:

- Diferenciar unha expresión dunha función en Sympy
- Definir funcións definidas a cachos
- Explorar o dominio, singularidades e rango dunha función
- Cálculo de límites nunha e varias variables
- Análise da continuidade dunha función

Para dispoñer do módulo **Sympy** e importalo para o resto do guión de prácticas, usaremos:

```python
import sympy as sp
```

## Expresións e funcións en **Sympy**
Ata o de agora, nas prácticas anteriores estivemos a traballar con expresións matemáticas que se gardaban en obxectos do módulo **Sympy**. Nembargantes, aínda non tiñamos usado funcións. Para ver as diferencias entre unha expresión e a súa función asociada imos a ilustrar como avaliar funcións e expresións en **Sympy** cun exemplo: a función $F:(x,y)\to x(2x+y)$ e a expresión $f=x(2x+y)$:

```python
x, y = sp.symbols('x y', real=True) # define as variables simbólicas x e y
f = x*(2*x+y) # Isto é unha expresión
display(f)
print('Valor de f(2,1)=',f.subs({x:2,y:1})) # Avaliar a expresión f usando "subs"
```

$\displaystyle x \left(2 x + y\right)$

    Valor de f(2,1)= 10

No caso das funcións, a avaliación é moito máis sinxela xa que esta se realizaría como se fose unha función máis como o seno, coseno, exponencial, etc. Para empregar funcións en **Sympy**, débese empregar a función `sp.Lambda`:

```python
F = sp.Lambda((x,y),f) # Isto é unha función Lambda
display(F)
print('Valor de F(2,1)=',F(2,1)) # Avaliación da función mediante os seus argumentos
print(F(x,y)==f)
```

$\displaystyle \left( \left( x, \  y\right) \mapsto x \left(2 x + y\right) \right)$

    Valor de F(2,1)= 10
    True

As funcións tamén se poden vir dadas por expresións que están **definidas a cachos**, tendo en conta distintas expresións que serán avaliadas se se cumpren certas condicións. O módulo **Sympy** irá avaliando cada unha das tuplas que aparecen como argumentos (de esquerda a dereita) ata que atope unha no que a condición sexa certa. Por exemplo:

```python
f = sp.Piecewise((1/(x*y), (x>0) & (y>0)),(1, x<0), (0, True))
F = sp.Lambda((x,y), f)
display(F)
```

$\displaystyle \left( \left( x, \  y\right) \mapsto \begin{cases} \frac{1}{x y} & \text{for}\: x > 0 \wedge y > 0 \\1 & \text{for}\: x < 0 \\0 & \text{otherwise} \end{cases} \right)$

## Dominio e rango dunha función de varias variables
Para calcular o dominio dunha función pódese primeiramente calcular as singularidades dunha función dada. Na implementación do módulo **Sympy** (versión 1.4.0) unicamente está dispoñible esta cálculo para funcións de tipo racional e con dependecia nunha única variable. Tendo isto en conta, o dominio da función virá dado polo conxunto de puntos do espazo ao que se lle deben quitar os puntos onde se atopan as singularidades. Para o caso de varias variables, este análise pódese facer unicamente con respecto a unha das variables (e tomando un valor fixo para o resto):

```python
f=sp.Lambda((x,y), x/((x+y)*y))
display(sp.calculus.singularities(f(x,y), x))
display(sp.calculus.singularities(f(x,y), y))
```

$\displaystyle \left\{- y\right\} \setminus \left\{0\right\}$

$\displaystyle \left\{0, - x\right\}$

Do mesmo xeito ao que ocorre coas singularidades, **Sympy** soamente é capaz de calcular o rango de funcións dunha única variable. Aínda que isto supón unha limitación, é posible explorar o rango dunha función de varias variables restrinxindo os valores dos argumentos a certos planos cartesianos e analizando o rango das funcións resultantes (dunha variable):

```python
x, y, z = sp.symbols('x y z', real=True) # define as variables simbólicas x, y, z
f=sp.Lambda((x,y,z), x*z/((x+y)*y))
Rx = sp.calculus.util.function_range(f(x,1,1), x, sp.Reals)
Ry = sp.calculus.util.function_range(f(1,y,1), y, sp.Reals)
Rz = sp.calculus.util.function_range(f(1,1,z), z, sp.Reals)
display(Rx)
display(Ry)
display(Rz)
```

$\displaystyle \left(-\infty, 1\right) \cup \left(1, \infty\right)$

$\displaystyle \left(-\infty, -4\right] \cup \left(0, \infty\right)$

$\displaystyle \left(-\infty, \infty\right)$

### **Exercicio 5.1**
Calcula as singularidades e o rango da función $f(x,y)=\displaystyle\frac{y+x}{y-x^3}$.

```python
# O TEU CÓDIGO AQUÍ
```

## Límites
O módulo **Sympy** soamente dispón de ferramentas para cálcular os límites que depende dunha única variable. A pesar desta limitación, é posible calcular límites en varias variables usando as técnicas que xa revisamos nas clases de pizarra como son o uso de camiños por rectas $y=mx$, os límites (re)iterados, ou o cambio a coordenadas polares.

### Límites de expresións nunha variable
Os límites de expresións nunha variable poden calcularse usando a función `sp.limit`. O mesmo comando tamén permite o cálculo de límites laterais. Comprobemos o seu uso cunha función dunha variable moi sinxela (que non posúe límite):
$$
f(x)=
\begin{cases}
+1 & \text{se } x>0,\\
-1 & \text{se } x\le 0.
\end{cases}
$$

```python
x = sp.Symbol('x', real=True) # define a variable simbólica x
f = sp.Piecewise((1, x>0), (-1, True))
F = sp.Lambda(x, f)
display(F)
```

$\displaystyle \left( x \mapsto \begin{cases} 1 & \text{for}\: x > 0 \\-1 & \text{otherwise} \end{cases} \right)$

Neste caso, o límite $\displaystyle\lim_{x\to 0}f(x)$ non existe xa que os límite laterais non coinciden. A pesar disto, obtense:

```python
display(sp.limit(F(x),x,0)) # Por defecto usa o valor do límite pola dereita
display(sp.limit(F(x),x,0,dir='+')) # límite pola dereita
display(sp.limit(F(x),x,0,dir='-')) # límite pola esquerda
```

$\displaystyle 1$

$\displaystyle 1$

$\displaystyle -1$

Do mesmo xeito a como se definen os límite cando $x$ tende a un valor finito $a$, tamén se poden calcular os valores cando $x\to+\infty$ ou $x\to-\infty$. O valor de $\infty$ está representado en **Sympy** por `sp.oo`. No caso da función $f(x)=e^x$, se obtén:

```python
display(sp.limit(sp.exp(x),x,-sp.oo))
display(sp.limit(sp.exp(x),x,sp.oo))
```

$\displaystyle 0$

$\displaystyle \infty$

### Límites de expresións en varias variables
Como xa revisamos nas clases de pizarra, o cálculo de límites en varias variables é máis complexo xa que debemos analizar todos os camiños posibles para poder asegurar que o límite existe e coincide cun valor dado. Por exemplo, no caso da función $f(x,y)=\frac{x^2y}{x^5+y^2}$ podemos ter valores diferentes segundo o camiño escollido para chegar ao punto $(0,0)$:

```python
x, y = sp.symbols('x y', real=True) # define a variable simbólica x e y
f = sp.Lambda((x,y), x**2*y/(x**5+y**2)) # agora temos definida a función simbólica f(x)
sp.limit(f(x,y),x,0) # límite fixando o valor de y
```

$\displaystyle 0$

```python
sp.limit(f(x,y),y,0) # límite fixando o valor de x
```

$\displaystyle 0$

```python
m = sp.Symbol('m', real=True)
display(f(x,m*x))
sp.limit(f(x,m*x),x,0) # límites polas rectas de pendente m
```

$\displaystyle \frac{m x^{3}}{m^{2} x^{2} + x^{5}}$

$\displaystyle 0$

Nembargantes, se se emprega o camiño $y=x^3$ obtense un valor do límite diferentes de cero, do que se conclúe que o límite non existe:

```python
sp.limit(f(x,x**3),x,0)
```

$\displaystyle 1$

### Límites (re)iterados
Os límites (re)iterados é outra ferramenta máis para tratar de analizar o que ocorre co límite de funcións de varias variables. Posto que estos límite iterados son unha sucesión de cálculos de límites tendo en conta unha soa variable, este procedemento pode implementarse en **Sympy**. Por exemplo, se consideramos $f(x,y)= \displaystyle \frac{x^2-y^2}{x^2+y^2}$, resulta

```python
f = sp.Lambda((x,y),(x**2-y**2)/(x**2+y**2))
g = sp.limit(f(x,y),x,0)
sp.limit(g,y,0)
```

$\displaystyle -1$

```python
g = sp.limit(f(x,y),y,0)
sp.limit(g,x,0)
```

$\displaystyle 1$

e polo tanto pódese concluir que non existe o límite, xa que os valores non coinciden.

### Casos onde non existe límites
Como xa vimos nos exemplos analizados en clase de pizarra, existen multitude de exemplos onde os límites non existe, como é o caso de $\displaystyle\lim_{(x,y)\to(0,0)}f(x,y)$ con
$$
f(x,y)=
\begin{cases}
\frac{xy}{x^2+y^2} & \text{se } x^2+y^2>0,\\
0 & \text{se } (x,y)=(0,0).
\end{cases}
$$

```python
f = sp.Piecewise((x*y/(x**2+y**2), x**2+y**2>0), (0, True))
F = sp.Lambda((x,y), f)
display(F)
# Límite iterado primeiro en x e despois en y
G = sp.limit(F(x,y),x,0)
display(G)
sp.limit(G,y,0)
```

$\displaystyle \left( \left( x, \  y\right) \mapsto \begin{cases} \frac{x y}{x^{2} + y^{2}} & \text{for}\: x^{2} + y^{2} > 0 \\0 & \text{otherwise} \end{cases} \right)$

$\displaystyle 0$

$\displaystyle 0$

Tendo en conta o límite sobre a recta $y=mx$, resulta:

```python
display(F(x,m*x))
sp.limit(F(x,m*x),x,0)
```

$\displaystyle \begin{cases} \frac{m x^{2}}{m^{2} x^{2} + x^{2}} & \text{for}\: m^{2} x^{2} + x^{2} > 0 \\0 & \text{otherwise} \end{cases}$

$\displaystyle \frac{m}{m^{2} + 1}$

Polo que podemos asegurar que o límite non existe xa que o valor do límite dependería de $m$. Un último exemplo, onde os límites iterados non existe, pero o límite si existe é o seguinte:

```python
f = sp.Piecewise((y, x > 0), (-y, True))
F = sp.Lambda((x,y), f)
display(F)
# Límite iterado primeiro en x e despois en y
G = sp.limit(F(x,y),x,0)
sp.limit(G,y,0)
```

$\displaystyle \left( \left( x, \  y\right) \mapsto \begin{cases} y & \text{for}\: x > 0 \\- y & \text{otherwise} \end{cases} \right)$

$\displaystyle 0$

```python
# Límite iterado primeiro en y e despois en x
G = sp.limit(F(x,y),y,0) # ERRO: Sympy non é capaz de calcular este límite
```

### Límites con coordenadas polares
Un dos recursos que usaremos con máis frecuencia para calcular límites en varias variables será o uso de coordenadas polares (no plano) ou coordenadas esféricas (no espazo tridimensional), para calcular límites do tipo $\displaystyle\lim_{(x,y)\to(0,0)}f(x,y)$ substitutindo $x=r\cos\theta$ e $y=r\sin\theta$. Por exemplo:

```python
r = sp.Symbol('r', nonnegative=True)
theta = sp.Symbol('theta', real=True)
f = (x**3+2*x*y**2)/(x**2+y**2)
F = sp.Lambda((x,y), f)
display(F)
display(F(r*sp.cos(theta), r*sp.sin(theta)))
sp.limit(F(r*sp.cos(theta), r*sp.sin(theta)),r,0)
```

$\displaystyle \left( \left( x, \  y\right) \mapsto \frac{x^{3} + 2 x y^{2}}{x^{2} + y^{2}} \right)$

$\displaystyle \frac{2 r^{3} \sin^{2}{\left(\theta \right)} \cos{\left(\theta \right)} + r^{3} \cos^{3}{\left(\theta \right)}}{r^{2} \sin^{2}{\left(\theta \right)} + r^{2} \cos^{2}{\left(\theta \right)}}$

$\displaystyle 0$

### **Exercicio 5.2**

Dada a función $f(x,y)=\displaystyle\frac{x-y^3}{x+y^3}$ calcula os seguintes límites cando $(x,y)\to (0,0)$:
- Límites sobre as rectas $y=mx$
- Límites iterados
- Límite usando coordenadas polares

```python
# O TEU CÓDIGO AQUÍ
sp.calculus.
```

## Análise da continuidade de varias variables

No módulo de **Sympy** existe unha función que calcular o dominio de continuidade (o conxunto de puntos onde a funciín é continua) dunha función dunha única variable. No caso de varias variables, o que se pode analizar é a continuidade dunha función con respecto ao resto de variables fixas. A función  empregar é `sp.calculus.util.continuous_domain(f(x,y), x, sp.Reals)`

```python
f=sp.Lambda((x,y), x/((x+y)*y))
Ix = sp.calculus.util.continuous_domain(f(x,y), x, sp.Reals)
Iy = sp.calculus.util.continuous_domain(f(x,y), y, sp.Reals)
display(Ix)
display(Iy)
```

$\displaystyle \mathbb{R} \setminus \left\{- y\right\} \cup \left\{- y\right\} \setminus \left\{0\right\}$

$\displaystyle \left(-\infty, 0\right) \cup \left(0, \infty\right)$

Ademáis, para analizar a continuidade de $F$ nun punto $(a,b)$, basta con comprobar que
$$
F(a,b)=\lim_{(x,y)\to(a,b)}F(x,y).
$$
Por exemplo, no caso dunha función dunha única variable:

```python
# Dominio de continuidade da función valor absoluto
F = sp.Lambda((x), sp.Abs(x))
Ix = sp.calculus.util.continuous_domain(F(x), x, sp.Reals)
display(Ix)

# Comprobación da continuidade definida a cachos
f = sp.Piecewise((x, x > 0), (-x, True))
F = sp.Lambda((x), f)
print('A función F é continua en x=0:', sp.limit(F(x),x,0)==F(0))
Ix = sp.calculus.util.continuous_domain(F(x), x, sp.Reals) # ERRO: non están soportadas as Piecewise functions!

```

### **Exercicio 5.3**
Analiza a continuidade da función
$$
f(x,y)=
\begin{cases}
\displaystyle\frac{1}{xy} & \text{se }x\neq 0, y\neq 0,\\
1 & \text{noutro caso},
\end{cases}
$$
en calquera punto $(x,y)$ do plano

```python
# O TEU CÓDIGO AQUÍ
```


# Models of the solid-liquid interface double layer

In order to understand lubrication better, we simulate thin layers of lubricant on a metallic surface, solvated in water.
Different structures of lubricant films are created by varying parameters like their concentration and the charge of the surface.
The lubricant is somewhat solvable in water, thus parts of the film will diffuse into the bulk water.
Lubricant molecules are charged, and their distribution is roughly exponential.

## The Poisson-Bolzmann distribution

As simplification, we  first create a solution of ions (Na+, purple; Cl-, green) in water (not shown).

Close to the positively charged metallic surface, the electric potential (red) will be highest, falling off exponentially when further away.
This potential attracts negatively charged Chlorine ions, and pushes positively charged Natrium ions away, resulting in a higher (lower) concentration of Clorine (Natrium) near the surface.

The Poission-Bolzmann approach fails for high potentials near the charged layer. Ions are predicted to be located abitrarily close to the surface, which is inconsistent with a finite ion radius.
This shortcoming is addressed by the Stern model.

## The Stern Model

Very close to the surface, a layer of ions is located. In this region, the potential decreases linerarly. In the rest of the liquid phase, the Poisson-Bolzmann solution holds.

```python
import numpy as np
import scipy.constants as sc
import matplotlib.pyplot as plt

# Sampling and plotting
from generate_structure import generate_structure
from generate_structure import plot_dist
from generate_structure import get_histogram
from IPython.display import Image

# Physical distribution generation
from poisson_bolzmann_distribution import debye
from poisson_bolzmann_distribution import gamma
from poisson_bolzmann_distribution import charge_density
from poisson_bolzmann_distribution import potential

np.random.seed(74)
```

The Poisson-Bolzmann equation:

$
\begin{align}
\rho_{Na^+}(x) &= \rho_{Na^+}(\infty) e^{-e \Psi(x)/k_B T}\\
\Psi(x) &= \frac{2k_B T}{e} \log\Big(\frac{1 + \gamma e^{-\kappa x}}{1- \gamma e^{-\kappa x}}\Big)
        \approx \frac{4k_B T}{e} \gamma e^{-\kappa x} \\
\gamma &= \tanh(\frac{e\Psi(0)}{4k_B T})\\
\kappa &= 1/\lambda_D\\
\lambda_D &= \Big(\frac{\epsilon \epsilon_0 k_B T}{\sum_{i} \rho_i(\infty) e^2 z_i^2} \Big)^\frac{1}{2} [m^{-1}]
\end{align}
$

## Poisson-Bolzmann Charge density $\rho(x)$

```python
z = np.linspace(0, 100*10**-9, 2000)
density = sc.Avogadro * 1000 * 10**-4
rho = {'Na': density, 'Cl':density}
charge = {'Na': 1, 'Cl': -1}
pot_0 = 0.05  # Breaks if > 1

psi = [potential(location=loc, rho_bulk=rho, charge=charge, surface_potential=pot_0) for loc in z]

rho_na = np.array([charge_density(location=loc, rho_bulk=rho, charge=charge, surface_potential=pot_0, species='Na') for loc in z])
rho_cl = np.array([charge_density(location=loc, rho_bulk=rho, charge=charge, surface_potential=pot_0, species='Cl') for loc in z])

deb = debye(rho_bulk=rho, charge=charge) * 10**9

fig, ax1 = plt.subplots(figsize=[16,5])
ax1.set_xlabel('z [nm]')
ax1.plot(z*10**9, psi, marker='', color='red', label='Potential', linewidth=1, linestyle='--')
ax1.set_ylabel('Potential')
ax1.axvline(x=deb, label='Debye Length', color='orange')

ax2 = ax1.twinx()
ax2.plot(z*10**9, [density]*len(z), label='Bulk concentration', color='grey')
ax2.plot(z*10**9, rho_na, marker='', color='green', label='Na+ ions')
ax2.plot(z*10**9, rho_cl, marker='', color='blue', label='Cl- ions')
ax2.set_ylabel('Density')

#fig.legend(loc='center')
ax2.legend(loc='best', fontsize=15)
ax1.legend(loc='upper center', fontsize=15)
fig.tight_layout()
plt.show()
```

The charge density behaves as expected, it interpolates between low (high) concentration and the bulk concentration within the first few debye lengths.

## Sampling
Now let's see if we can just plug our new distribution in our existing framework.

First, we need to convert the physical distribution to the format we were using so far:

```python
def wrap_distribution(x, species):
    """Wrapper for na+ ions."""
    density = sc.Avogadro * 1000 * 10**-4
    rho = {'Na': density, 'Cl':density}
    charge = {'Na': 1, 'Cl': -1}
    pot_0 = 0.05  # Breaks if > 1

    def call_distri(loc):
        distri = charge_density(location=loc, rho_bulk=rho,
                                charge=charge, surface_potential=pot_0, species=species)
        return float(distri)

    if not np.isscalar(x):
        y = []
        for i in range(0, len(x)):
            val = call_distri(x[i])

            # Normalize to be 1 at x=0
            val /= call_distri(0)
            # Scale distribution to have values in [0, 0.1] for ease of sampling
            val /= 10
            y += [val]
        return np.array(y)

    # If we have only a point estimate
    val = call_distri(x)
    # Normalize to be 1 at x=0
    val /= call_distri(0)
    # Scale distribution to have values in [0, 0.1] for ease of sampling
    val /= 10
    return val

def cl_distribution(x):
    return wrap_distribution(x, species='Cl')

def na_distribution(x):
    return wrap_distribution(x, species='Na')
```

Now we can sample the distribution of our $Na^+$ ions in z-direction.

```python
x = 50 * 10**-9
z = 100 * 10**-9
box = np.array([x, x, z])
sample_size = 1000

struc = generate_structure(distribution=na_distribution, box=box, atom_count=sample_size)
histx, histy, histz = get_histogram(struc, box=box, n_bins=51)
plot_dist(histz, 'Distribution of Na+ ions in z-direction', reference_distribution=na_distribution)
plt.show()

struc = generate_structure(distribution=cl_distribution, box=box, atom_count=sample_size)
histx, histy, histz = get_histogram(struc, box=box, n_bins=51)
plot_dist(histz, 'Distribution of Cl- ions in z-direction', reference_distribution=cl_distribution)
plt.show()
```

## Write to file
To visualize our structure, we export it to the .xyz file format, which is basically

```
ATOM_NUMBER
Optional comment
atom_type x y z
atom_type x y z
```

Avogadro expects x, y, z to be in units of $10^{-9}~m$, so we convert our salt "solution" to this unit.

Also, we use fewer atoms, just 100 for the 100x50x50 box - this will increase fluctuations with respect to the correct distribution though.

```python
from generate_structure import concat_names_structs
from generate_structure import export_named_struc

cl_struc = generate_structure(distribution=cl_distribution, box=box, atom_count=100)
na_struc = generate_structure(distribution=na_distribution, box=box, atom_count=100)

concat_list = concat_names_structs(struc_list=[cl_struc, na_struc], name_list=['Cl', 'Na'])
rescaled_list = []
for line in concat_list:
    name, x, y, z = line
    x = float(x) * 10**9
    y = float(y) * 10**9
    z = float(z) * 10**9
    rescaled_list += [[name, x, y, z]]

rescaled_list = np.array(rescaled_list)
export_named_struc(rescaled_list)

histx, histy, histz = get_histogram(cl_struc, box=box, n_bins=21)
plot_dist(histz, 'Distribution of Cl- ions in z-direction', reference_distribution=cl_distribution)

histx, histy, histz = get_histogram(na_struc, box=box, n_bins=21)
plot_dist(histz, 'Distribution of Na+ ions in z-direction', reference_distribution=na_distribution)
```


# Lecture 24
# Gamma Distribution, Poisson Processes

```python
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import gamma
%matplotlib inline

plt.xkcd()
None
```

## Stirling's Approximation to the Factorial

\begin{align}
  n! &= n \, (n-1) \, (n -1) \, \dots \, 1 \\
     &\sim \sqrt{2 \pi n} \, \left( \frac{n}{e} \right)^{n}
\end{align}

where $\sim$ means that the ratio of the two number converges to 1 as $n$ approaches $\infty$.

This is fine when we are talking about $n \in \{1,2,3,\dots\}$, but have you ever wondered what $\pi!$ is?

For that, we need the Gamma function.

----

## Gamma Function

Just as an aside, the Beta and Gamma functions are closely related. The Gamma function should be amongst the Top 10 Functions of All Time (if there was ever such a list).

\begin{align}
  \Gamma(a) &= \int_{0}^{\infty} x^a \, e^{-x} \, \frac{dx}{x} \quad \text{for any real }a \gt 0 \\
  &= \int_{0}^{\infty} x^{a-1} \, e^{-x} \, dx \quad \text{(alternately)}
\end{align}

Note that if $a$ approaches 0 from the right, the $\frac{dx}{x}$ is what drives that integral since $x^0 = 1$ and $e^{-0}=1$.

But $\int \frac{dx}{x} = log(x)$ which blows up to $-\infty$, which is why we restrict $a \gt 0$.

### Properties of the Gamma Function

1. $\Gamma(n) = (n-1)!$ where $n \in \mathbb{Z}^{+}$

1. $\Gamma(x+1) = x \, \Gamma(x)$

1. $\Gamma(\frac{1}{2}) = \sqrt{\pi}$

----

## Gamma Distribution

How would we derive a PDF that is based on the Gamma distribution?

_By normalizing the Gamma function._

\begin{align}
  1 &= \int_{0}^{\infty} c \, x^a \, e^{-x} \, \frac{dx}{x} \\
  &= \int_{0}^{\infty} \frac{1}{\Gamma(a)} \, x^a \, e^{-x} \, \frac{dx}{x}
\end{align}

And so the PDF for a Gamma distribution would be

\begin{align}
 Gamma(a,1) &= \frac{1}{\Gamma(a)} \, x^a \, e^{-x} \, \frac{dx}{x} \quad \text{for } x \gt 0 \\\\
 \\
 \text{More generally } Y &\sim Gamma(a, \lambda) \\
 \text{Let } Y &= \frac{X}{\lambda} \text{, where } X \sim Gamma(a,1) \\
 \rightarrow y &= \frac{x}{\lambda} \\
 x &= \lambda \, y \\
 \frac{dx}{dy} &= \lambda \\
 \\
 \Rightarrow f_Y(y) &= f_X(x) \, \frac{dx}{dy} \quad \text{ transforming } X \text{ to } Y \\
 &= \frac{1}{\Gamma(a)} \, (\lambda y)^a \, e^{-\lambda y} \, \frac{1}{\lambda y} \, \lambda \\
 &= \frac{1}{\Gamma(a)} \, (\lambda y)^a \, e^{-\lambda y} \, \frac{1}{y} \quad \text{for } y \gt 0
\end{align}

```python
alpha_values  = [1.,2.,3.,4.,5.,9.,7.5,0.5]
lambda_values = [2.,2.,2.,1.,0.5,1.,1.,0.5]
x = np.linspace(0, 20, 1000)

# plot the distributions
fig, ax = plt.subplots(figsize=(12, 6))

for a,l in zip(alpha_values, lambda_values):
    ax.plot(x, gamma.pdf(x, a, scale=l), lw=4, alpha=0.6, label=r'$\alpha$={}, $\lambda$={}'.format(a,l))

plt.xlim((0,20.0))
plt.ylim((0,0.5))
plt.legend()
ax.title.set_text(r'Examples ofGamma($\alpha , \lambda$)')
plt.show()
```

----

## Connecting the Gamma and Exponential, Poisson Distributions

### Poisson Processes

Recall Poisson processes, where the number of events happening in a certain time period $t$ is such that

\begin{align}
  N_t &= \text{number of events occuring up to time }t \\
      &\sim Pois(\lambda, t)
\end{align}

where the number of events occuring in disjoint time intervals are independent.

Now earlier, we discussed how time $t$ is exponential. Consider $t_1$, the time until we observe the very first event.

\begin{align}
  P(t_1 \gt t) &= P(N_t = 0) &\quad \text{by definition} \\
  &= \frac{\lambda^0 \, e^{\lambda t}}{0!} \\
  &= e^{\lambda t} \\
  &= 1 - (1-e^{\lambda t}) &\quad \text{1 - CDF of } Expo(\lambda) \\\\
  \\
  \Rightarrow &\text{time until first event } \sim Expo(\lambda)
\end{align}

And so using this argument along with the memorylessness property, all of the other times between subsequent events $t_2, t_3, \dots , t_n$ are also $Expo(\lambda)$

### Analogies: Geometric $\rightarrow$ Negative binomial (discrete); Exponential $\rightarrow$ Gamma (continuous)

So the interarrival time for events in a Poisson process are i.i.d. $Expo(\lambda)$.

But what if we want to know $t_n$, the time of the $n^{th}$ arrival?

\begin{align}
  T_n &= \sum_{i=1}^{n} X_1, X_2, \dots , X_n &\quad \text{where } X_i \text{ is i.i.d. } Expo(\lambda) \\\\
  &\sim Gamma(n, \lambda) &\quad \text{ assuming } n \text{ is an integer}
\end{align}

Recall the relation between the geometric and negative binomial distributions:
* In geometric $Geom(p)$, we count discrete time until $1^{st}$ success
* In negative binomial $NB(n, p)$, we count discrete time until $n^{th}$ success ($\sum_{i=1}^{n} X_i \text{ where } X_i \text{ i.i.d., } X_i \sim Geom(\lambda)$ )

Well, there's is something analogous between the exponential and Gamma distributions:
* In exponential $Expo(\lambda)$, we count continuous time until $1^{st}$ success
* In Gamma, we count continuous time until $n^{th}$ success ($\sum_{i=1}^{n}  X_i \text{ where } X_i \text{ i.i.d, } X_i \sim Expo(\lambda)$ )

----

## Moments of Gamma

#### $X \sim Gamma(\alpha, 1)$

Let $X \sim Gamma(\alpha,1)$

Directly using LOTUS:

\begin{align}
  \mathbb{E}(X^c) &= \int_{0}^{\infty} \frac{1}{\Gamma(a)} \,x^c \, x^{\alpha} \, e^{-x} \, \frac{dx}{x} \\
  &= \int_{0}^{\infty} \frac{1}{\Gamma(\alpha)} \, x^{a+c} \, e^{-x} \, \frac{dx}{x} \\
  &= \frac{\Gamma(\alpha+c)}{\Gamma(\alpha)} &\quad \text{, where } \alpha+x \gt 0 \\\\
  \\
  1^{st} \text{ moment (mean) of } Gamma(n,1) &= \mathbb{E}(X) \\
  &= \frac{\Gamma(\alpha+1)}{\Gamma(\alpha)} \\
  &= \frac{\alpha \, \Gamma(\alpha)}{\Gamma(\alpha)} \\
  &= \boxed{ \alpha } \\\\
  \\
  2^{nd} \text{ moment } Gamma(n,1) &= \mathbb{E}(X^2) \\
  &= \frac{\Gamma(\alpha+2)}{\Gamma(\alpha)} \\
  &= \frac{(\alpha+1) \, \Gamma(\alpha+1)}{\Gamma(\alpha)} \\
  &= \frac{(\alpha+1)\alpha \, \Gamma(\alpha)}{\Gamma(\alpha)} \\
  &= \alpha^2 + \alpha \\
  \\
  \Rightarrow \text{Var of } Gamma(n,1) &=  \mathbb{E}(X^2) - \left( \mathbb{E}(X)^2 \right) \\
  &= \alpha^2 + \alpha - \alpha^2 \\
  &= \boxed{ \alpha  }
\end{align}

Let $X \sim Gamma(\alpha, \lambda)$

Applying what we know from the case of above and using LOTUS:

\begin{align}
  \mathbb{E}(X^c) &= \int_{0}^{\infty} \frac{1}{\Gamma(\alpha)} \,x^c \, (\lambda x)^{\alpha} \, e^{-\lambda x} \, \frac{dx}{\lambda x} \\
  &= \int_{0}^{\infty} \frac{1}{\Gamma(\alpha)} \, x^{\alpha+c} \, \lambda^{\alpha} \, e^{-\lambda x} \, \frac{dx}{\lambda \, x} \\
  &= \int_{0}^{\infty} \frac{1}{\Gamma(\alpha)} \, (\lambda \, x)^{\alpha+c} \, e^{-\lambda x} \, \frac{dx}{\lambda \, x} \, \frac{1}{\lambda^c} \\
  &= \frac{\Gamma(\alpha+c)}{\Gamma(\alpha) \, \lambda^c} &\quad \text{, where } \alpha+x \gt 0 \\\\
  \\
  1^{st} \text{ moment (mean) of } Gamma(\alpha,\lambda) &= \mathbb{E}(X) \\
  &= \frac{\Gamma(\alpha+1)}{\Gamma(\alpha) \, \lambda} \\
  &= \frac{\alpha \, \Gamma(\alpha)}{\Gamma(\alpha) \, \lambda} \\
  &= \boxed{ \frac{\alpha}{\lambda} } \\\\
  \\
  2^{nd} \text{ moment } Gamma(\alpha,\lambda) &= \mathbb{E}(X^2) \\
  &= \frac{\Gamma(\alpha+2)}{\Gamma(\alpha) \, \lambda^2} \\
  &= \frac{(\alpha+1) \, \Gamma(\alpha+1)}{\Gamma(\alpha) \, \lambda^2} \\
  &= \frac{(\alpha+1)\alpha \, \Gamma(\alpha)}{\Gamma(\alpha) \, \lambda^2} \\
  &= \frac{(\alpha+1)\alpha}{\lambda^2} \\
  \\
  \Rightarrow \text{Var of } Gamma(\alpha,\lambda) &=  \mathbb{E}(X^2) - \left( \mathbb{E}(X)^2 \right) \\
  &= \frac{(\alpha+1)\alpha}{\lambda^2} - \left(  \frac{\alpha}{\lambda} \right)^2 \\
  &= \frac{\alpha + \alpha^2 - \alpha^2}{\lambda^2} \\
  &= \boxed{  \frac{\alpha}{\lambda^2} }
\end{align}