# Linear Regression线性回归原理

## 1. 原理

### 1.1 问题引入

#### 1.1.1 一元线性回归

```python
from matplotlib import pyplot as plt
%matplotlib inline

x = [2, 4]
y = [5, 9]

plt.figure(figsize=[10, 5])
plt.xlabel("x")
plt.ylabel("y")
plt.xlim((0, 5))
plt.ylim((0, 10))
plt.scatter(x, y, s=16)
plt.plot([0, 5], [1, 11], color="darkorange")
plt.show()
```

已知点A(2, 7)和点B(4, 9)在直线L上，求直线L的线性函数：$f(x) = wx + b$ 的参数 $w$ 和 $b$

将点A、B坐标代入直线函数，解方程组
$ \left\{
\begin{aligned}
2w+b=5 \\
4w+b=9
\end{aligned}
\right.
$，得$w=2, b=1$

**问题升级**

样本数据存在噪声 $\epsilon$ ：
$f(x) = wx+b$
$y - f(x) = \epsilon$，如何求解？

```python
import numpy as np
x = np.arange(1, 11, 0.01)
noise = np.random.randn(1000)
y = 2 * x + 1 + noise
plt.figure(figsize=[10, 5])
plt.xlabel("x")
plt.ylabel("y")
plt.xlim((0, 12))
plt.ylim((0, 26))
plt.scatter(x, y, s=12)
plt.show()
```

#### 1.1.2 多元线性回归

$f(x_i)=w_{1}x_{i1} + w_{2}x_{i2} + ... + w_{d}x_{id} + b$

向量形式：$f(x_i) = W^T x_i + b$

e.g: $f_{颜值}(x) = 0.3·x_{身高} + 0.7·x_{五官} + 0.5·x_{胸围} + 0.6·x_{衣品} +\ ...\ + 0.6·x_{气质} + 5$

### 1.2 求解

#### 1.2.1 最小二乘法

找出最佳参数，使得数据与实际数据之间误差的平方和为最小。
即 **残差平方和的总和最小化**
**几何意义**： 使预测值和真实值的欧式距离之和最小

##### 1.2.1.1 问题定义

给定数据集$D=\left\{\left(\boldsymbol{x}_{1}, y_{1}\right),\left(\boldsymbol{x}_{2}, y_{2}\right), \ldots,\left(\boldsymbol{x}_{m}, y_{m}\right)\right\}$，其中$\boldsymbol{x}_{i}=\left(x_{i 1}\right. \left.x_{i 2} ; \ldots ; x_{i d}\right), y_{i} \in \mathbb{R}$。
线性回归试图学得： $f\left(x_{i}\right)=w x_{i}+b$，使得 $f\left(x_{i}\right) \simeq y_{i}$

##### 1.2.1.2 基本求解

采用SSE（Sum of Squares Error）计算预测值 $f(x_i)$ 和 $y_i$ 之间的误差

**一、 目标函数（损失函数Loss function、代价函数cost function）**:
> $\begin{aligned} J(w, b) = \sum_{i=1}^{m}\left(f\left(x_{i}\right)-y_{i}\right)^{2} \end{aligned}$
> 求使得目标函数 $J(w, b)$ 取最小值的 $w$ 、 $b$
> $\begin{aligned} (w^{*}, b^{*}) &=\underset{(w, b)}{\arg \min } \sum_{i=1}^{m}\left(f\left(x_{i}\right)-y_{i}\right)^{2} \\ &=\underset{(w, b)}{\arg \min } \sum_{i=1}^{m}\left(y_{i} - w x_{i}-b\right)^{2} \end{aligned}$

**二、 对 $w$ 求偏导** ：
$\begin{aligned} \frac{\partial J({w}, {b})}{\partial {w}} &=\frac{\partial}{\partial {w}}\left[\sum_{{i}=1}^{{m}}\left({y}_{{i}}-{w} {x}_{{i}}-{b}\right)^{2}\right] \\ &=\sum_{{i}=1}^{{m}} \frac{\partial}{\partial {w}}\left[\left({y}_{{i}}-{w} {x}_{{i}}-{b}\right)^{2}\right] \\ &=\sum_{{i}=1}^{{m}}\left[2 \cdot\left({y}_{{i}}-{w} {x}_{{i}}-{b}\right) \cdot\left(-{x}_{{i}}\right)\right] \\ &=\sum_{{i}=1}^{{m}}\left[2 \cdot\left({w} {x}_{{i}}^{2}-{y}_{{i}} {x}_{{i}}+{b} {x}_{{i}}\right)\right] \\ &=2 \cdot\left({w} \sum_{{i}=1}^{{m}} {x}_{{i}}^{2}-\sum_{{i}=1}^{{m}} {y}_{{i}} {x}_{{i}}+{b} \sum_{{i}=1}^{{m}} {x}_{{i}}\right) \\ &=2\left({w} \sum_{{i}=1}^{{m}} {x}_{{i}}^{2}-\sum_{{i}=1}^{{m}}\left({y}_{{i}}-{b}\right) {x}_{{i}}\right)  \end{aligned}$

**三、 对 $b$ 求偏导** ：
$\begin{aligned} \frac{\partial {J}{({w}, {b})}}{\partial {b}} &=\frac{\partial}{\partial {b}}\left[\sum_{{i}=1}^{{m}}\left({y}_{{i}}-{w} {x}_{{i}}-{b}\right)^{2}\right] \\ &=\sum_{{i}=1}^{{m}} \frac{\partial}{\partial {b}}\left[\left({y}_{{i}}-{w} {x}_{{i}}-{b}\right)^{2}\right] \\ &=\sum_{{i}=1}^{{m}}\left[2 \cdot\left({y}_{{i}}-{w} {x}_{{i}}-{b}\right) \cdot(-1)\right] \\ &=\sum_{{i}=1}^{{m}}\left[2 \cdot\left({b}-{y}_{{i}}+{w} {x}_{{i}}\right)\right] \\ &=2 \cdot\left[\sum_{{i}=1}^{{m}} {b}-\sum_{{i}=1}^{{m}} {y}_{{i}}+\sum_{{i}=1}^{{m}} {w} {x}_{{i}}\right] \\ &=2\left({mb}-\sum_{{i}=1}^{{m}}\left({y}_{{i}}-{w} {x}_{{i}}\right)\right) \end{aligned}$

**四、 令$\begin{aligned} \frac{\partial J(w, b)}{\partial w} = 0 \end{aligned}$** ：

$\begin{aligned} 0={w} \sum_{{i}=1}^{{m}} {x}_{{i}}^{2}-\sum_{{i}=1}^{{m}}\left({y}_{{i}}-{b}\right) {x}_{{i}} \end{aligned}$
$\begin{aligned} {w} \sum_{{i}=1}^{{m}} {x}_{{i}}^{2}=\sum_{{i}=1}^{{m}} {y}_{{i}} {x}_{{i}}-\sum_{{i}=1}^{{m}} {bx}_{{i}} \end{aligned}$

> 令$\begin{aligned} \frac{\partial J(w, b)}{\partial b}=0 \end{aligned}$ 得：
> $\begin{aligned} {b}=\frac{1}{{m}} \sum_{{i}=1}^{{m}}\left({y}_{{i}}-{w} {x}_{{i}}\right) \end{aligned}$
> 因为 $\begin{aligned} \frac{1}{{m}} \sum_{{i}=1}^{{m}} {y}_{{i}}=\overline{{y}}, \frac{1}{{m}} \sum_{{i}=1}^{{m}} {x}_{{i}}=\overline{{x}} \end{aligned}$
> 所以 $\begin{aligned} {b}=\overline{{y}}-{w} \overline{{x}} \end{aligned}$， 代入上式得：

$\begin{aligned} {w} \sum_{{i}=1}^{{m}} {x}_{{i}}^{2}=\sum_{{i}=1}^{{m}} {y}_{{i}} {x}_{{i}}-\sum_{{i}=1}^{{m}}(\overline{{y}}-{w} \overline{{x}}) {x}_{{i}} \end{aligned}$
$\begin{aligned} {w} \sum_{{i}=1}^{{m}} {x}_{{i}}^{2}=\sum_{{i}=1}^{{m}} {y}_{{i}} {x}_{{i}}-\overline{{y}} \sum_{{i}=1}^{{m}} {x}_{{i}}+{w} \overline{{x}} \sum_{{i}=1}^{{m}} {x}_{{i}} \end{aligned}$
$\begin{aligned} {w}\left(\sum_{{i}=1} {x}_{{i}}^{2}-\overline{{x}} \sum_{{i}=1}^{{m}} {x}_{{i}}\right)=\sum_{{i}=1}^{{m}} {y}_{{i}} {x}_{{i}}-\overline{{y}} \sum_{{i}=1}^{{m}} {x}_{{i}} \end{aligned}$
$\begin{aligned} {w}=\frac{\sum_{{i}=1}^{{m}} {y}_{{i}} {x}_{{i}}-\overline{{y}} \sum_{{i}=1}^{{m}} {x}_{{i}}}{\sum_{{i}=1}^{{m}} {x}_{{i}}^{2}-\overline{{x}} \sum_{{i}=1}^{{m}} {x}_{{i}}} \end{aligned}$

> 由于 $\begin{aligned}
\bar{y} \sum_{i=1}^{m} x_{i}=\frac{1}{m} \sum_{i=1}^{m} y_{i} \sum_{i=1}^{m} x_{i}=\bar{x} \sum_{i=1}^{m} y_{i}
\end{aligned}$，
> $\begin{aligned}
\overline{{x}} \sum_{{i}=1}^{{m}} {x}_{{i}}=\frac{1}{{m}} \sum_{{i}=1}^{{m}} {x}_{{i}} \sum_{{i}=1}^{{m}} {x}_{{i}}=\frac{1}{{m}}\left(\sum_{{i}=1}^{{m}} {x}_{{i}}\right)^{2}
\end{aligned}$
代入上式得

求得 $w$ 和 $b$ 最优解的**闭式解**（closed-form）：

$\begin{aligned}
{w^*}=\frac{\sum_{{i}=1}^{{m}} {y}_{{i}}\left({x}_{{i}}-\overline{{x}}\right)}{\sum_{{i}=1}^{{m}} {x}_{{i}}^{2}-\frac{1}{{m}}\left(\sum_{{i}=1}^{{m}} {x}_{{i}}\right)^{2}}
\end{aligned}$

$\begin{aligned} {b^*}=\frac{1}{{m}} \sum_{{i}=1}^{{m}}\left({y}_{{i}}-{w} {x}_{{i}}\right) \end{aligned}$

##### 1.2.1.3 向量求解

将 $b$ 纳入 $w$，记作$\hat{\boldsymbol{w}}$ :

$\begin{aligned}
\hat{\boldsymbol{w}}=(\boldsymbol{w } ; b)
\end{aligned}$

即 $f(x_i) = x_{i1}w_1 + x_{i2}w_2 +\ ...\ + x_{id}w_d + 1 · w_{d+1}$

将 $x$ 也写成向量形式，并扩展一列填充为 $1$ ，记作 $X$ :

$\begin{equation}
\mathbf{X}=\left(\begin{array}{ccccc}
x_{11} & x_{12} & \ldots & x_{1 d} & 1 \\
x_{21} & x_{22} & \ldots & x_{2 d} & 1 \\
\vdots & \vdots & \ddots & \vdots & \vdots \\
x_{m 1} & x_{m 2} & \ldots & x_{m d} & 1
\end{array}\right)=\left(\begin{array}{cc}
x_{1}^{\mathrm{T}} & 1 \\
x_{2}^{\mathrm{T}} & 1 \\
\vdots & \vdots \\
x_{m}^{\mathrm{T}} & 1
\end{array}\right)
\end{equation}$

$\begin{aligned}
\hat{\boldsymbol{w}}^{*}=\underset{\hat{w}}{\arg \min }(\boldsymbol{y}-\mathbf{X} \hat{\boldsymbol{w}})^{\mathrm{T}}(\boldsymbol{y}-\mathbf{X} \hat{\boldsymbol{w}})
\end{aligned}$

$\begin{aligned} J(\hat{\boldsymbol{w}}) = (\boldsymbol{y}-\mathbf{X} \hat{\boldsymbol{w}})^{\mathrm{T}}(\boldsymbol{y}-\mathbf{X} \hat{\boldsymbol{w}}) \end{aligned}$

将上式展开，得：
$\begin{aligned}
J({\hat{w}})=\boldsymbol{y}^{\mathrm{T}} \boldsymbol{y}-\boldsymbol{y}^{\mathrm{T}} \mathbf{X} \hat{\boldsymbol{w}}-\hat{\boldsymbol{w}}^{\mathrm{T}} \mathbf{X}^{\mathrm{T}} \boldsymbol{y}+\hat{\boldsymbol{w}}^{\mathrm{T}} \mathbf{X}^{\mathrm{T}} \mathbf{X} \hat{\boldsymbol{w}}
\end{aligned}$

对 $\hat{w}$ 求导，得：
$\begin{aligned}
\frac{\partial \mathrm{J}({\hat{\boldsymbol{w}})}}{\partial \hat{\boldsymbol{w}}}=\frac{\partial \boldsymbol{y}^{\mathrm{T}} \boldsymbol{y}}{\partial \hat{\boldsymbol{w}}}-\frac{\partial \boldsymbol{y}^{\mathrm{T}} \mathbf{X} \hat{\boldsymbol{w}}}{\partial \hat{\boldsymbol{w}}}-\frac{\partial \hat{\boldsymbol{w}}^{\mathrm{T}} \mathbf{X}^{\mathrm{T}} \boldsymbol{y}}{\partial \hat{\boldsymbol{w}}}+\frac{\partial \hat{\boldsymbol{w}}^{\mathrm{T}} \mathbf{X}^{\mathrm{T}} \mathbf{X} \hat{\boldsymbol{w}}}{\partial \hat{\boldsymbol{w}}}
\end{aligned}$

由矩阵微分公式：
$\begin{aligned}
\frac{\partial \boldsymbol{a}^{\mathrm{T}} \boldsymbol{x}}{\partial \boldsymbol{x}}=\frac{\partial \boldsymbol{x}^{\mathrm{T}} \boldsymbol{a}}{\partial \boldsymbol{x}}=\boldsymbol{a}, \frac{\partial \boldsymbol{x}^{\mathrm{T}} \mathbf{A} \boldsymbol{x}}{\partial \boldsymbol{x}}=\left(\mathbf{A}+\mathbf{A}^{\mathrm{T}}\right) \boldsymbol{x}
\end{aligned}$
可得：
$\begin{aligned}
\frac{\partial \mathrm{J}({\hat{\boldsymbol{w}})}}{\partial \hat{\boldsymbol{w}}}=0-\mathbf{X}^{\mathrm{T}} \boldsymbol{y}-\mathbf{X}^{\mathrm{T}} \boldsymbol{y}+\left(\mathbf{X}^{\mathrm{T}} \mathbf{X}+\mathbf{X}^{\mathrm{T}} \mathbf{X}\right) \hat{\boldsymbol{w}}
\end{aligned}$
$\begin{aligned}
\frac{\partial \mathrm{J}({\hat{\boldsymbol{w}})}}{\partial \hat{\boldsymbol{w}}}=2 \mathbf{X}^{\mathrm{T}}(\mathbf{X} \hat{\boldsymbol{w}}-\boldsymbol{y})
\end{aligned}$

令$\begin{aligned} \frac{\partial \mathrm{J}({\hat{\boldsymbol{w}})}}{\partial \hat{\boldsymbol{w}}} = 0 \end{aligned}$

解得 $\begin{aligned}
\hat{\boldsymbol{w}}^{*}=\left(\mathbf{X}^{\mathrm{T}} \mathbf{X}\right)^{-1} \mathbf{X}^{\mathrm{T}} \boldsymbol{y}
\end{aligned}$

#### 1.2.2 梯度下降法Gradient Descent

##### 1.2.2.1 梯度和参数更新

**梯度**：关于目标函数 $J(w, b)$ 的偏导数组成的向量，即$\begin{aligned}(\frac{\partial J({w}, {b})}{\partial {w}}, \frac{\partial J({w}, {b})}{\partial {b}} )\end{aligned}$
$\begin{aligned}
J(w, b) = \sum_{{i}=1}^{{m}}(wx_i + b - y_i)^2
\end{aligned}$

$\begin{aligned}
\frac{\partial J({w}, {b})}{\partial {w}} = 2\sum_{{i}=1}^{m}(wx_i + b - y_i)x_i
\end{aligned}$

$\begin{aligned}
\frac{\partial J({w}, {b})}{\partial {b}} = 2\sum_{{i}=1}^{m}(wx_i + b - y_i)
\end{aligned}$

用梯度去**更新参数** $w$ 和 $b$

$\begin{aligned}
w \leftarrow w-\eta \frac{\partial J(w, b)}{\partial w}
\end{aligned}$

$\begin{aligned}
b \leftarrow b-\eta \frac{\partial J(w, b)}{\partial b}
\end{aligned}$

其中 $\eta$ 是学习率（learning rate），需要手动设置，通常取 $0.01, 0.001, 0.0001, ...$

##### 1.2.2.2 问题

* 为什么是减去梯度（乘以学习率），不是加？
> $ J(w) = 3(w - 3)^2 +1 = 3w^2 - 18w + 28$
>
> $ \begin{aligned}
\frac{\partial J({w})}{\partial {w}} = 6w - 18
\end{aligned} $
>
> 已知在 $w=3$ 时 $J(w)$ 取得最小值 $1$
>
> 点 $(0, 28)$ 处的导数为 $-18$
>
> $w$ 要从 $0$ 增大到 $3$ ，就得加一个正数（减一个负数），即 $0 - (-18)*\eta$

```python
plt.figure(figsize=[5, 7])
w = np.arange(-3, 10, 0.01)
j = np.square(w - 3) * 3 + 1
plt.xlabel("w")
plt.ylabel("J(w)")
plt.xlim(-3, 10)
plt.ylim(0, 100)
plt.plot(w, j)

plt.show()
```

* 什么时候通过加上梯度值（乘以学习率）更新参数？
> 目标函数要求极大值时，通过加上梯度值（乘以学习率）更新参数
> 这种方法叫 **梯度上升法**。

* 为什么要乘以学习率？
梯度向量 $(1, 2)$ 和 $(10, 20)$ 表示同一个方向

##### 1.2.2.3 各种梯度下降的方法

* 批梯度下降BGD（Batch Gradient Descent）

    * 将整个训练集计算梯度，更新参数，多次迭代。
    * 计算慢，计算量随训练集的增大而增大，计算冗余。
    * 新数据加入，需要重新计算，更新模型。
    * 对于凸函数可以收敛到全局极小值，对于非凸函数可以收敛到局部极小值。

* 随机梯度下降SGD（Stochastic Gradient Descent）

    * 从训练集中**随机**抽出（有放回的）一条数据，计算梯度，更新参数，多次迭代。
    * 计算快，更新参数快。
    * 随机采样代表全训练集，新数据加入无需重新计算整个数据集。
    * 噪声大，并不是每次迭代都向着整体最优化方向。虽然训练速度快，但是准确度下降，并不是全局最优。
    * 更新比较频繁，会造成 $loss$ 有严重的震荡。

* 小批量梯度下降MBGD（Mini-Batch Gradient Descent）
小批量随机梯度下降 Mini-batch Stochastic Gradient Descent

    * 从训练集中**随机**抽出（有放回的）长度为 $batch\_size$ 的Mini-Batch样本代表全样本， 计算梯度，更新参数，多次迭代
    * 可以降低参数更新时的方差，收敛更稳定

##### 1.2.2.4 超参数

* 只控制模型的训练过程
* 需要手动设置
* **超参数调优**：多次调整超参数的组合，获得使模型训练效果最好的超参数

    * learning_rate
    * batch_size
    * epochs

**名词解释**

* batch_size: 每次从训练集随机抽取的mini-batch的样本数

* step（Iteration）: 用一个mini-batch求梯度 更新一次参数，叫做one step(1步)

* epoch: 将训练集的全部数据完整训练一次叫做一个epoch

* steps_per_epoch: 一个epoch需要经过的step数。$\begin{aligned} steps\_per\_epoch = \frac{sample\_length} {batch\_size} \end{aligned}$

#### 1.2.3 问题

 * 什么时候用闭式解， 什么时候用梯度下降法 ？

     *  一般特征数比较大，$X^T X$ 不可逆的时候
     * 数据集较大，较复杂的时候，闭式解矩阵乘法和矩阵求逆运算量大，梯度下降法更快

### 1.3 Lasso(索套回归)和Ridge(岭回归)

#### 1.3.1 Overfitting 过拟合

##### 1.3.1.1 问题描述

过度拟合训练集，模型只在训练集表现很好，在测试集表现很糟。
出现原因：
* 数据过少
* 模型过复杂

```python
x = np.arange(1, 11, 0.01)
noise = np.random.randn(1000)
top_x = x[noise>2]
top_y = 2 * top_x + 1 + noise[noise>2]

# y = 2 * x + 1 + noise

plt.figure(figsize=[10, 5])
plt.xlabel("x")
plt.ylabel("y")
plt.xlim((0, 12))
plt.ylim((0, 26))
plt.scatter(top_x, top_y, s=12, c="darkorange")
plt.show()
```

```python
x = x[noise <= 2]
y = 2 * x + 1 + noise[noise<=2]

plt.figure(figsize=[10, 5])
plt.xlabel("x")
plt.ylabel("y")
plt.xlim((0, 12))
plt.ylim((0, 26))
plt.scatter(top_x, top_y, s=12, c="darkorange")
plt.scatter(x, y, s=12)
plt.show()
```

##### 1.3.1.2 抑制过拟合的方法
* 增加数据
* 降维（特征选择、特征提取）
* Regularize正则化
* 降低模型复杂度
* ~~集成学习~~
* ~~Dropout~~

#### 1.3.2 Regularize 正则化

原来的代价函数上加一项正则项（惩罚项penalty）与惩罚率 $\lambda$ 的乘积: $\lambda P(w)$ ，即：

$\begin{aligned} J(w) = \sum_{i=1}^{m}\left(f\left(x_{i}\right)-y_{i}\right)^{2} + P(w) \end{aligned}$

$\begin{aligned}
w \leftarrow w-\eta \frac{\partial J(w)}{\partial w}
\end{aligned}$

##### 1.3.2.1 范数

$L_p$ 范数：
$\begin{align*}
\| w \|_{p}=\sqrt[p]{\sum_{i}\left|w_{i}\right|^{p}} = \left(\left|w_{1}\right|^{p}+\left|w_{2}\right|^{p}+\cdots+\left|w_{n}\right|^{p}\right)^{\frac{1}{p}}
\end{align*}$

$L_2$ 范数：
$\begin{align*}
\| w \|_{2}=\sqrt[2]{\sum_{i} w_i^2} =\left(\left|w_{1}\right|^{2}+\left|w_{2}\right|^{2}+\left|w_{3}\right|^{2}+\cdots+\left|w_{n}\right|^{2}\right)^{1 / 2}
\end{align*}$

$L_1$ 范数：
$\begin{align*}
\| w \|_{1}={\sum_{i} |w_i|} =\left|w_{1}\right|+\left|w_{2}\right|+\left|w_{3}\right|+\cdots+\left|w_{n}\right|
\end{align*}$

##### 1.3.2.2 正则化
L1正则(Lasso Regression)： $\begin{aligned} J(w, b) = \sum_{i=1}^{m}\left(f\left(x_{i}\right)-y_{i}\right)^{2} + ||w||_1 \end{aligned}$
L2正则(Ridge Regression)： $\begin{aligned} J(w, b) = \sum_{i=1}^{m}\left(f\left(x_{i}\right)-y_{i}\right)^{2} + ||w||_2^2 \end{aligned}$

**结论**
* 原 loss function 加上 $L_1$ 正则项就是Lasso回归（索套回归）， 加上$L_2$ 正则项就是Ridge回归（岭回归）
* Lasso和Ridge都可以一定程度抑制过拟合。
* 岭回归用的多一点，效果好一点，且方便求解。 Lasso不方便求解

#### 1.3.3 Ridge求解

Ridge Regression 岭回归，也叫权值衰减。

$\begin{aligned}
J(w, b) &= \sum_{i=1}^{m}\left(w^T x_{i}-y_{i}\right)^{2} + \lambda ||w||_2^2 \\ &= \sum_{i=1}^{m}\left(w^T x_{i}-y_{i}\right)^{2} + \lambda w^T w \\ &= \left(w^T X^T-y^T\right)(X w-y)+\lambda w^T w \\ &= w^T X^T X w -2w^T X^T y + y^T y + \lambda w^T w \\ &= w^T\left(X^T X+\lambda I\right) w-2 w^T X^T y^T y
\end{aligned}$

$\begin{align}
\hat{w} &=\arg \min _{w} J(w) \\ &= w^T\left(X^T X+\lambda I\right) w-2 w^T X^T y^T y
\end{align}$

$\begin{aligned}
\frac{\partial J(w)}{\partial w} = 2\left(X^T X+\lambda I\right) w-2 X^T y=0
\end{aligned}$

解得 $\begin{align}
\hat{w^*}=\left(X^T X+\lambda I\right)^{-1} X^T Y
\end{align}$

#### 1.3.4 岭回归的特点

* 可以抑制过拟合。
* 引入单位矩阵，半逆矩阵加上单位矩阵，**一定可逆**。

#### 1.3.5 问题

* 1. 岭回归的“岭”指的是什么？
> 岭回归中引入了单位矩阵，单位矩阵也叫对角矩阵。
> 单位矩阵是一个方阵，对角线上的值全为 $1$ , 其他值全为 $0$。
> 对角线就像山岭一样，岭回归因此得名。

* 2. 正则化为什么能够抑制过拟合？
> **正则化的本质是给参数加上约束。**
>
> 如何求带约束的优化问题？
    >> $\begin{aligned} J(w) = \sum_{i=1}^{m}\left(f\left(x_{i}\right)-y_{i}\right)^{2} \end{aligned}$
    >>
    >> $s.t: ||w||_2 ^2 < m $
    >>
    >> 构造拉格朗日函数：
    >>  $\begin{aligned} L(w) = \sum_{i=1}^{m}\left(f\left(x_{i}\right)-y_{i}\right)^{2} + \lambda (||w||_2^2 - m) \end{aligned}$

* 3. 为什么L1正则会产生稀疏解？
> $w = (1.2, 0, 0, 3.2, 0, 0, 6.3, 0)$
>>
> * 从解空间形状的角度看：
>>
>>
>> 假设只有两个特征，且没有偏置 $b$ 。
>> 给 $w$ 加上约束： $w_1^2 + w_2^2 \leq m$
> * 贝叶斯先验角度
>> L1正则化相当于对模型参数 $w$ 引入了**拉普拉斯先验分布**，L2正则化引入了**高斯先验分布**。
>>
>> 而**拉普拉斯先验使参数为0的概率更大**。
>>
>>
>>
>>
>> 高斯分布(Gaussian distribution)：均值为 $\mu$ ，方差为 $\sigma$ ，记作：
$\begin{align}
X \sim N\left(\mu, \sigma^{2}\right)
\end{align}$
>>
>> 概率密度函数：
>> $\begin{align}
f(x)=\frac{1}{\sigma \sqrt{2 \pi}} e^{-\frac{(x-\mu)^{2}}{2 \sigma^{2}}}
\end{align}$
>>
>>
>>
>>
>> 拉普拉斯分布(Laplace distribution): 均值为 $\mu$ ，b > 0 是尺度参数
>>
>> 概率密度函数：
>> $\begin{align}
f(x \mid \mu, b)=\frac{1}{2 b} e^ {-\frac{|x-\mu|}{b}}
\end{align}$

* 4. Lasso有哪些求解方式？
>

#### 1.3.6 L1 vs L2

##### L1
* 产生稀疏解，可以做特征选择
* 梯度分情况讨论，不能直接得到闭式解，也不能用梯度下降求解

##### L2
* 抑制过拟合效果更好
* 可以直接得到闭式解，且**一定可以求逆**
* 可以用梯度下降法求解

### 1.4 概率视角看线性回归和岭回归

#### 1.4.1 极大似然估计MLE（Maximum Likelihood Estimate）

$\begin{align}
y=f(w)+\varepsilon
\end{align}$

$\begin{align}
\varepsilon \sim N\left(0,\sigma^{2}\right)
\end{align}$

$\begin{align}
f(w)=w^{\top} X
\end{align}$

$\begin{align}
y=w^{\top} X+\varepsilon
\end{align}$

$\begin{align}
y \mid X ; w \sim N\left(w^{\top} X, \varepsilon ^2\right)
\end{align}$

$\begin{align} P(y|X;w) = \frac{1}{\sigma \sqrt{2 \pi}} e^{-\frac{(y-w^{\top} X)^{2}}{2 \sigma^{2}}} \end{align}$

写出似然函数：
$\begin{aligned}
L(w) &= log\left( P(y|X; w) \right)
\\ &= log \prod_{i=1}^{m}P(y_i|x_i; w)
\\ &= \sum_{i=1}^{m} log P(y_i|x_i; w)
\\ &= \sum_{i=1}^{m} (log \frac{1}{\sigma \sqrt{2 \pi}}  - {\frac{(y_i - w^{\top} x_i)^{2}}{2 \sigma^{2}}})
\end{aligned}$

$\begin{align}
\hat{w} &= \underset{w}{\arg \max} L(w) \\
&= \underset{w}{\arg \max} \sum_{i=1}^{m} (log \frac{1}{\sigma \sqrt{2 \pi}}  - {\frac{(y_i-w^{\top} x_i)^{2}}{2 \sigma^{2}}}) \\
&= \underset{w}{\arg \max} \sum_{i=1}^{m} -\frac{1}{2 \sigma^{2}}\left(y_{i}-w^{\top} x_{i}\right)^{2} \\
&= \underset{w}{\arg \min} \sum_{i=1}^{m} \left(y_{i}-w^{\top} x_{i}\right)^{2}
\end{align}$

##### 结论
**最小二乘估计LSE等价于噪声服从高斯分布的MLE**

#### 1.4.2 贝叶斯角度 最大后验概率估计MAP（maximum a posteriori probability estimate）

Ridge

$\begin{align}
y=f(w)+\varepsilon
\end{align}$

$\begin{align}
\varepsilon \sim N\left(0,\sigma^{2}\right)
\end{align}$

$\begin{align}
y \mid X ; w \sim N\left(w^{\top} X, \varepsilon ^2\right)
\end{align}$

$\begin{align} P(y \mid w) = \frac{1}{\sigma \sqrt{2 \pi}} e^{-\frac{(y-w^{\top} X)^{2}}{2 \sigma^{2}}} \end{align}$

$\begin{align}
w \sim N\left(0, \sigma_{0}^{2}\right)
\end{align}$

$\begin{align} P(w) = \frac{1}{\sigma_0 \sqrt{2 \pi}} e^{-\frac{w^{2}}{2 \sigma_0 ^{2}}} \end{align}$

$\begin{align}
P(w \mid y) &= \frac{P(y|w) \cdot P(w)}{P(y)}
\end{align}$

MAP：

$\begin{align}
\begin{aligned}
\hat{w} &=\underset{w}{\arg \max}\  log P(w \mid y) \\
&=\underset{w}{\arg \max} \ log P(y \mid w) \cdot P(w) \\
&=\underset{w}{\arg \max} \ log \prod_{i=1}^{m} P(y_i \mid w) \cdot P(w) \\
&=\underset{w}{\arg \max} \sum_{i=1}^{m} \ log\ P(y_i \mid w) + log\ P(w) \\
&=\underset{w}{\arg \max} \sum_{i=1}^{m} \ log \left(\frac{1}{\sigma \sqrt{2 \pi}} \cdot \frac{1}{\sigma_0 \sqrt{2 \pi}}\right) + ({-\frac{(y_i-w^{\top} x_i)^{2}}{2 \sigma^{2}}} {-\frac{w^{2}}{2 \sigma_0 ^{2}}}) \\
&=\underset{w}{\arg \max} \sum_{i=1}^{m}  ({-\frac{(y_i-w^{\top} x_i)^{2}}{2 \sigma^{2}}} {-\frac{w^{2}}{2 \sigma_0 ^{2}}}) \\
&=\underset{w}{\arg \min} \sum_{i=1}^{m}  ({\frac{(y_i-w^{\top} x_i)^{2}}{2 \sigma^{2}}} + {\frac{w^{2}}{2 \sigma_0 ^{2}}}) \\
&=\underset{w}{\arg \min} \sum_{i=1}^{m}  ({(y_i-w^{\top} x_i)^{2}} + {\frac{2 \sigma^{2}}{2 \sigma_0 ^{2}}} \cdot {w^{2}})
\end{aligned}
\end{align}$

$\begin{align}
\hat{w}_{MAP} = \underset{w}{\arg \min} \sum_{i=1}^{m}  ({(y_i-w^{\top} x_i)^{2}} + {\frac{2 \sigma^{2}}{2 \sigma_0 ^{2}}} \cdot {w^{2}})
\end{align}$

##### 结论
**L2正则化的最小二乘估计LSE等价于 噪声服从高斯分布、参数w先验也服从高斯分布的MAP**
**同理可推，L1正则化的最小二乘估计LSE等价于 噪声服从高斯分布、参数w先验服从拉普拉斯分布的MAP**

## 2. 算法复杂度

### 2.1 时间复杂度

#### 2.1.1 线性回归闭式解

 $X$ 为 $ m \times (d + 1)$ 的矩阵

$\begin{aligned}
\hat{\boldsymbol{w}}^{*}=\left(\mathbf{X}^{\mathrm{T}} \mathbf{X}\right)^{-1} \mathbf{X}^{\mathrm{T}} \boldsymbol{y}
\end{aligned}$

|操作|input|output|时间复杂度|
|-|:-:|:-:|:-:|
|矩阵乘法|(n, m) * (m, p) |(n, p)|$O(nmp)$|
|求逆矩阵|(n, n)| (n, n)|$O(n^3)$|

$ O(m \cdot (d+1)^2) = O(m \cdot (d+1)^2) + O((d+1)^3) + O((d+1)^2 \cdot m) + O((d+1) \cdot m) $

#### 2.1.3 Ridge闭式解

 $\begin{align}
\hat{w}=\left(X^T X+\lambda I\right)^{-1} X^T Y
\end{align}$

$ O(m \cdot (d+1)^2) = O(m \cdot (d+1)^2) +  O((d+1) ^ 2) + O((d+1)^3) + O((d+1)^2 \cdot m) $

### 2.2 空间复杂度

#### 2.2.1 线性回归闭式解

$O(m \cdot (d+1)) = O(X) + O(y) + O(w) =  O(m \cdot (d+1)) + O(m) + O(d+1)$

#### 2.2.2 Ridge闭式解

$O(m \cdot (d+1)) = O(X) + O(y) + O(w) + O(I) =  O(m \cdot (d+1)) + O(m) + O(d+1) + O((d+1)^2)$

## 3. 优缺点及其他特点

### 3.1 优点
1. 实现简单
2. 效果稳定

### 3.2 缺点：
1. 计算复杂度高
2. 只能实现线性拟合

### 3.3 其他
1. 属于监督学习
2. 是一种线性模型
3. 是一种参数模型

## 4. 优化空间

### 4.1 多项式回归

$\begin {aligned}
f(x) = a x^2 + b x + c
\end {aligned}$

```python
def show_func_points(func, x_start=0, x_stop=10, x_step=0.01):
    x = np.arange(x_start, x_stop, x_step)
    y = func(x)
    plt.figure(figsize=[10, 5])
    plt.xlabel("x")
    plt.ylabel("y")
    plt.scatter(x, y, s=12)
    plt.show()

show_func_points(func=lambda x: np.square(x) - 10 * x - 1, x_step=0.1)
```

### 4.2 局部加权线性回归 Locally Weighted Linear Regression，LWLR

#### 4.2.1 欠拟合underfitting

模型在训练集上拟合的不好，在测试集上也拟合的不好。

#### 4.2.2 原理

预测过程，给每个点一个权值，离待预测点越近的点权值越大，优先优化权值大的点。
LWLR使用“核”来对附近的点赋予更高的权重，一般采用高斯核（Gaussian kernel，也叫径向基函数核 Radial basis function kernel，RBF kernel）计算，
公式： $\begin{align}
w(i, i)=\exp \left(\frac{||x^{(i)}-x||_{2} ^ 2}{-2 k^{2}}\right)
\end{align}$

#### 4.2.3 求解

$
\begin{align}
J(a, b)= \sum_{i=1}^{N} w_{i}\left(y_{i}-a x_{i}-b\right)^{2}
\end{align}
$

$\begin{align}
\hat{a}^*=\left( X ^{ T } W X \right)^{-1} X ^{ T } W y
\end{align}$

## 总结

1. 最小二乘法
    > * 使残差平方和最小
    > * 几何意义：使预测值和真实值的欧式距离之和最小
    > * 目标函数：$\begin{aligned} J(w, b) = \sum_{i=1}^{m}\left(y_{i} - w x_{i}-b\right)^{2} \end{aligned}$
    > * $\begin{aligned} (w^{*}, b^{*}) =\underset{(w, b)}{\arg \min } \sum_{i=1}^{m}\left(y_{i} - w x_{i}-b\right)^{2} \end{aligned}$
    >
    > * 闭式解：$\begin{aligned}
{w^*}=\frac{\sum_{{i}=1}^{{m}} {y}_{{i}}\left({x}_{{i}}-\overline{{x}}\right)}{\sum_{{i}=1}^{{m}} {x}_{{i}}^{2}-\frac{1}{{m}}\left(\sum_{{i}=1}^{{m}} {x}_{{i}}\right)^{2}}
\end{aligned}$
> $\begin{aligned} {b^*}=\frac{1}{{m}} \sum_{{i}=1}^{{m}}\left({y}_{{i}}-{w} {x}_{{i}}\right) \end{aligned}$
    >
    > * 矩阵形式，将 $b$ 纳入 $w$ 记作 $\hat{w}$ ，将 $X$ 添加一列，全部填充为 $1$
    > * 矩阵形式闭式解：$\begin{aligned}
\hat{\boldsymbol{w}}^{*}=\left(\mathbf{X}^{\mathrm{T}} \mathbf{X}\right)^{-1} \mathbf{X}^{\mathrm{T}} \boldsymbol{y}
\end{aligned}$

2. 梯度下降法：
    > * 梯度：关于目标函数  𝐽(𝑤,𝑏)  的偏导数组成的向量
    > * 目标函数：$\begin{aligned}
J(w, b) = \sum_{{i}=1}^{{m}}(wx_i + b - y_i)^2
\end{aligned}$
    > * 梯度：$\begin{aligned}
\frac{\partial J({w}, {b})}{\partial {w}} = 2\sum_{{i}=1}^{m}(wx_i + b - y_i)x_i
\end{aligned}$
    > $\begin{aligned}
\frac{\partial J({w}, {b})}{\partial {b}} = 2\sum_{{i}=1}^{m}(wx_i + b - y_i)
\end{aligned}$
    > * 更新梯度：$\begin{aligned}
w \leftarrow w-\eta \frac{\partial J(w, b)}{\partial w}
\end{aligned}$
    >
    > $\begin{aligned}
b \leftarrow b-\eta \frac{\partial J(w, b)}{\partial b}
\end{aligned}$
    >
    > * BGD、SGD、MBGD

3. 正则化：
    > * 正则化公式：$\begin{aligned}
J(w) = L(w) + \lambda P(w)
\end{aligned}$
    > $L(w)$是Loss function，$P(w)$ 是penalty function
    >
    > * 范数公式：$\begin{align*}
\| w \|_{p}=\sqrt[p]{\sum_{i}\left|w_{i}\right|^{p}} = \left(\left|w_{1}\right|^{p}+\left|w_{2}\right|^{p}+\cdots+\left|w_{n}\right|^{p}\right)^{\frac{1}{p}}
\end{align*}$
    > * Lasso索套回归: $P(w) = ||w||_1$
    > * Ridge岭回归（权值衰减）：  $P(w) = ||w||_2 ^ 2$
    >
    >
    > * Ridge目标函数：$\begin{aligned}
J(w, b) &= \sum_{i=1}^{m}\left(w^T x_{i}-y_{i}\right)^{2} + \lambda ||w||_2^2 \\ &= w^T\left(X^T X+\lambda I\right) w-2 w^T X^T y^T y
\end{aligned}$
    > * Ridge矩阵形式闭式解：$\begin{align}
\hat{w^*}=\left(X^T X+\lambda I\right)^{-1} X^T Y
\end{align}$
    >
    > * Lasso vs. Ridge：
    > Lasso不能求闭式解也不能直接梯度下降法求解，但是Lasso可以产生稀疏解，可用于特征选择；
    > Ridge都可以，且一定可逆，且效果更好。
    >
    > * 为什么正则化能够抑制过拟合？
    > 正则化的本质是给参数加上约束。降低了模型的复杂度。
    > * 为什么l1正则可以产生稀疏解？
    > 答：从解空间的形状看，l2的解空间是圆形，l1的解空间是多边形，多边形的解空间更容易在尖角处与等位线碰撞出稀疏解。从贝叶斯先验看，l2的参数w服从高斯分布，l1的参数w服从拉普拉斯分布，拉普拉斯分布在0点处是一个尖峰，所以参数w取值为0的可能性更高。

4. 概率视角：
    > * 线性回归 极大似然估计：最小二乘估计LSE等价于噪声服从高斯分布的MLE
    > * Lasso、Ridge贝叶斯视角 最大后验估计：
    > L2正则化的最小二乘估计LSE等价于 噪声服从高斯分布、参数w先验也服从高斯分布的MAP
    > L1正则化的最小二乘估计LSE等价于 噪声服从高斯分布、参数w先验服从拉普拉斯分布的MAP

5. 算法复杂度
    > * $X$ 为 $ m \times (d + 1)$ 的矩阵
    > * 时间复杂度（闭式解，线性回归和Ridge同）：
    > $ O(m(d+1)^2)$
    > * 空间复杂度（闭式解，线性回归和Ridge同）：
    > $ O(m(d+1))$

5. 优化
    > * 多项式回归
    >> * 转化为特征工程问题
    > * 局部加权线性回归LWLR
    >>  * 求距离一般采用高斯核：$\begin{align}
w(i, i)=\exp \left(\frac{||x^{(i)}-x||_{2} ^ 2}{-2 k^{2}}\right)
\end{align}$
    >> * 目标函数：$
\begin{align}
J(a, b)= \sum_{i=1}^{N} w_{i}\left(y_{i}-a x_{i}-b\right)^{2}
\end{align}
$
    >> * 向量形式解： $\begin{align}
\hat{a}^*=\left( X ^{ T } W X \right)^{-1} X ^{ T } W y
\end{align}$

6. 名词
    > 过拟合、欠拟合
    > 梯度、学习率learning_rate、超参数、epochs、mini-batch、batch-size、step、正则系数

## 相关链接

 2.2 sklearn中使用线性回归、岭回归、Lasso、SGD、多项式回归
 2.3 numpy实现线性回归、岭回归、SGD回归、LWLR
 2.4 TensorFlow2实现线性回归、岭回归、SGD回归、LWLR
 2.5 PyTorch1实现线性回归、岭回归、SGD回归、LWLR

## 参考文献

[1] [M]Peter Harrington[美]. 机器学习实战. 北京: 人民邮电出版社, 2013.6.
[2] [M]周志华. 机器学习. 北京: 清华大学出版社, 2016.1.
[3] [M]诸葛越, 葫芦娃. 百面机器学习. 北京: 人民邮电出版社, 2018.8.1.

https://www.experfy.com/blog/gradient-descent/
https://datawhalechina.github.io/pumpkin-book/#/chapter3/chapter3
https://blog.csdn.net/pxhdky/article/details/82960659
https://www.cnblogs.com/guoyaohua/p/8542554.html
https://zh.wikipedia.org/wiki/最小二乘法
https://zh.wikipedia.org/wiki/正态分布
https://zh.wikipedia.org/wiki/拉普拉斯分布
https://github.com/dragen1860/Deep-Learning-with-TensorFlow-book


```python
import ambulance_game as abg
import numpy as np
import math
import matplotlib.pyplot as plt
import functools
import sympy as sym
```

# Blocking time formula (recursive approach)
- **From steady state probabilities**
- **For class 2 patients (ambulance patients)**

```python
abg.markov.visualise_ambulance_markov_chain(2, 3, 6, 2)
```

## Blocking States
$$ S_b = \{ (u,v) \in S \; | \; u > 0 \} $$

```python
def is_blocking_state(state):
    return state[0] > 0
```

## Accepting States
$$
S_A^A =
\begin{cases}
\{(u,v) \in S \; | \; u < M \}  & \text{if } T \leq N\\
\{(u,v) \in S \; | \; v < N \}  & \text{otherwise}
\end{cases}
$$

```python
def is_accepting_state(state, threshold, system_capacity, buffer_capacity):
    condition = (
        (state[0] < buffer_capacity)
        if (threshold <= system_capacity)
        else (state[1] < system_capacity)
    )
    return condition

def is_accepting_state_others(state, system_capacity):
    return state[1] < system_capacity
```

## Expected sojourn time in a state (ignoring any arrivals)
$$
c(u,v) =
    \frac{1}{min(v,C) \mu}
$$

```python
def expected_sojourn_time_in_markov_state(state, num_of_servers, mu):
    return 1 / (min(state[1], num_of_servers) * mu)
```

## Recursive blocking time for each state $(u,v)$
$$
b(u,v) =
\begin{cases}
    0, & \textbf{if } (u,v) \notin S_b \\
    c(u,v) + p_s b(u-1, v) + p_o b(u, v+1), & \textbf{if } u > 0 \textbf{ and } v = T \\
    c(u,v) + p_s b(u, v-1) + p_o b(u, v+1), & \textbf{otherwise} \\
\end{cases}
$$

$$
p_s = \frac{\mu}{\lambda_1 + \mu} \\
p_o = \frac{\lambda_1}{\lambda_1 + \mu}
$$

```python
def prob_service(lambda_1, mu):
    return mu / (lambda_1   + mu)

def prob_other_arrival(lambda_1, mu):
    return lambda_1 / (lambda_1 + mu)

def get_recursive_blocking_time(
    state, lambda_1, mu, num_of_servers, threshold, system_capacity, buffer_capacity
):

    if not is_blocking_state(state):
        return 0

    if state[0] >= 1 and state[1] == threshold:
        service_state = (state[0] - 1, state[1])
    else:
        service_state = (state[0], state[1] - 1)

    return "STUCK IN AN INFINITE RECURSION"

    others_arrival_state = (state[0], state[1] + 1) if state[1] < system_capacity else 0
    print(state)
    time_blocked = expected_sojourn_time_in_markov_state(
        state, num_of_servers, mu
    ) + prob_service(
        lambda_1, mu
    ) * get_recursive_blocking_time(
        service_state, lambda_1, mu, num_of_servers, threshold, system_capacity, buffer_capacity
    ) + prob_other_arrival(
        lambda_1, mu
    ) * get_recursive_blocking_time(
        others_arrival_state, lambda_1, mu, num_of_servers, threshold, system_capacity, buffer_capacity
    )

    return time_blocked
```

# Closer look at the problem

```python
abg.markov.visualise_ambulance_markov_chain(2, 3, 6, 2)
```

## Recursive blocking time for each state $(u,v)$
$$
b(u,v) =
\begin{cases}
    0, & \textbf{if } (u,v) \notin S_b \\
    c(u,v) + p_s b(u-1, v) + p_o b(u, v+1), & \textbf{if } u > 0 \textbf{ and } v = T \\
    c(u,v) + p_s b(u, v-1) + p_o b(u, v+1), & \textbf{otherwise} \\
\end{cases}
$$

$$
p_s = \frac{\mu}{\lambda_1 + \mu} \\
p_o = \frac{\lambda_1}{\lambda_1 + \mu}
$$

$$
b(1,3) = c(1,3) + p_s b(0,3) + p_o b(1,4) \\
b(1,4) = c(1,4) + p_s b(1,3) + p_o b(1,5) \\
b(1,5) = c(1,5) + p_s b(1,4) + p_o b(1,6) \\
b(1,6) = c(1,6) + b(1,5)
$$

$$
b(2,3) = c(2,3) + p_s b(1,3) + p_o b(2,4) \\
b(2,4) = c(2,4) + p_s b(2,3) + p_o b(1,5) \\
b(2,5) = c(2,5) + p_s b(2,4) + p_o b(1,6) \\
b(2,6) = c(2,6) + b(2,5)
$$

# *Recursive approach may be more complex to model - different approach should be used*

```python

```

```python

```

```python
lambda_2 = 1
lambda_1 = 1
mu = 2
num_of_servers = 2
threshold = 3
system_capacity = 6
buffer_capacity = 2
```

```python
prob_service(lambda_1, mu)
```

    0.6666666666666666

```python
prob_other_arrival(lambda_1, mu)
```

    0.3333333333333333

```python
get_recursive_blocking_time((1,3), lambda_1, mu, num_of_servers, threshold, system_capacity, buffer_capacity)
```

    'Stuck in an infinite loop'

## Mean blocking time
$$ B = \frac{\sum_{(u,v) \in S_A} \pi_{(u,v)} \; b(u,v)}{\sum_{(u,v) \in S_A} \pi_{(u,v)}} $$

```python
def mean_blocking_time_formula(
    all_states,
    pi,
    lambda_1,
    mu,
    num_of_servers,
    threshold,
    system_capacity,
    buffer_capacity,
):
    mean_blocking_time = 0
    prob_accept_ambulance = 0
    for u, v in all_states:
        if is_accepting_state((u, v), threshold, system_capacity, buffer_capacity):
            arriving_state = (u + 1, v) if v >= threshold else (u, v + 1)
            current_state_block = get_recursive_blocking_time(
                arriving_state,
                lambda_1,
                mu,
                num_of_servers,
                threshold,
                system_capacity,
                buffer_capacity,
            )
            mean_blocking_time += current_state_block * pi[u, v]
            prob_accept_ambulance += pi[u, v]

    # prob_accept_others = np.sum([pi[state] for state in all_states if is_accepting_state_others(state, system_capacity)])
    # others_rate = (prob_accept_others) / ((prob_accept_ambulance) + (prob_accept_others))
    # ambulance_rate = (prob_accept_ambulance) / ((prob_accept_ambulance) + (prob_accept_others))
    # return mean_blocking_time, prob_accept_ambulance, prob_accept_others, ambulance_rate, others_rate

    return mean_blocking_time / prob_accept_ambulance

```

# Comparing simulation and Markov chains

### 1. Function that gets blocking time from simulation (already calculated)
### 2. Function that calculates blocking time using state probabilities of Markov chains

```python
def get_mean_blocking_time_markov(
    lambda_2,
    lambda_1,
    mu,
    num_of_servers,
    threshold,
    system_capacity,
    buffer_capacity,
):
    transition_matrix = abg.markov.get_transition_matrix(
        lambda_2,
        lambda_1,
        mu,
        num_of_servers,
        threshold,
        system_capacity,
        buffer_capacity,
    )
    pi = abg.markov.get_steady_state_algebraically(
        transition_matrix, algebraic_function=np.linalg.solve
    )
    all_states = abg.markov.build_states(threshold=threshold, system_capacity=system_capacity, buffer_capacity=buffer_capacity)
    state_probabilities = abg.markov.get_markov_state_probabilities(
        pi, all_states, output=np.ndarray
    )

    mean_blocking_time = mean_blocking_time_formula(
        all_states=all_states,
        pi=state_probabilities,
        mu=mu,
        num_of_servers=num_of_servers,
        threshold=threshold,
        system_capacity=system_capacity,
        buffer_capacity=buffer_capacity,
    )

    return mean_blocking_time

```

### 3. Function that calculates blocking time using state probability of simulation

```python
def get_mean_blocking_time_simulation(
    lambda_2,
    lambda_1,
    mu,
    num_of_servers,
    threshold,
    system_capacity,
    buffer_capacity,
    seed_num=None,
    num_of_trials=10,
    runtime=2000
):
    state_probabilities = abg.simulation.get_average_simulated_state_probabilities(
        lambda_2,
        lambda_1,
        mu,
        num_of_servers,
        threshold,
        system_capacity,
        buffer_capacity,
        seed_num=seed_num,
        num_of_trials=num_of_trials,
        runtime=runtime
    )

    all_states = [
        (u, v)
        for v in range(state_probabilities.shape[1])
        for u in range(state_probabilities.shape[0])
        if state_probabilities[u, v] > 0
    ]

    mean_blocking_time = mean_blocking_time_formula(
        all_states=all_states,
        pi=state_probabilities,
        mu=mu,
        num_of_servers=num_of_servers,
        threshold=threshold,
        system_capacity=system_capacity,
        buffer_capacity=buffer_capacity,
    )

    return mean_blocking_time

```

```python

```

```python

```

```python

```

```python

```

```python

```

## Playing around

```python
lambda_2 = 1
lambda_1 = 1
mu = 2
num_of_servers = 1
threshold = 1
system_capacity = 2
buffer_capacity = 1
```

```python
abg.markov.visualise_ambulance_markov_chain(num_of_servers, threshold, system_capacity, buffer_capacity)
```

```python
get_mean_blocking_time_simulation(
    lambda_2=lambda_2,
    lambda_1=lambda_1,
    mu=mu,
    num_of_servers=num_of_servers,
    threshold=threshold,
    system_capacity=system_capacity,
    buffer_capacity=buffer_capacity,
    seed_num=0,
    num_of_trials=10,
    runtime=2000
)
```

    0.3531281614188685

```python
get_mean_blocking_time_markov(
    lambda_2=lambda_2,
    lambda_1=lambda_1,
    mu=mu,
    num_of_servers=num_of_servers,
    threshold=threshold,
    system_capacity=system_capacity,
    buffer_capacity=buffer_capacity
)
```

    0.35714285714285715

```python
times = abg.simulation.get_multiple_runs_results(
            lambda_2=lambda_2,
            lambda_1=lambda_1,
            mu=mu,
            num_of_servers=num_of_servers,
            threshold=threshold,
            system_capacity=system_capacity,
            buffer_capacity=buffer_capacity,
            num_of_trials=10,
            seed_num=None,
            runtime=1000,
            patient_type="ambulance",
            warm_up_time=100
        )
simulation_blocking_times = [np.mean(b.blocking_times) for b in times]
np.mean(simulation_blocking_times)
```

    0.5078234203227561

```python

```

```python
transition_matrix = abg.markov.get_transition_matrix(
    lambda_2,
    lambda_1,
    mu,
    num_of_servers,
    threshold,
    system_capacity,
    buffer_capacity,
)
pi = abg.markov.get_steady_state_algebraically(
    transition_matrix, algebraic_function=np.linalg.solve
)
all_states = abg.markov.build_states(threshold=threshold, system_capacity=system_capacity, buffer_capacity=buffer_capacity)
state_probabilities = abg.markov.get_markov_state_probabilities(
    pi, all_states, output=np.ndarray
)
state_probabilities
```

    array([[0.28571429, 0.28571429, 0.0952381 ],
           [       nan, 0.19047619, 0.14285714]])

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python
def get_plot_comparing_times(
    lambda_2,
    lambda_1,
    mu,
    num_of_servers,
    threshold,
    num_of_trials,
    seed_num,
    runtime,
    system_capacity,
    buffer_capacity,
    plot_over="lamda_a",
    max_parameter_value=1,
    accuracy=None,
):
    all_times_sim = []
    all_mean_times_sim = []
    all_mean_times_markov = []

    if accuracy == None or accuracy <= 1:
        accuracy = 5

    if plot_over == "lambda_2":
        range_space = np.linspace(lambda_2, max_parameter_value, accuracy)
    elif plot_over == "lambda_1":
        range_space = np.linspace(lambda_1, max_parameter_value, accuracy)
    elif plot_over == "mu":
        range_space = np.linspace(mu, max_parameter_value, accuracy)
    elif plot_over == "num_of_servers":
        range_space = np.linspace(num_of_servers, max_parameter_value, accuracy)
    elif plot_over == "threshold":
        range_space = np.linspace(threshold, max_parameter_value, accuracy)
    elif plot_over == "system_capacity":
        range_space = np.linspace(system_capacity, max_parameter_value, accuracy)
    elif plot_over == "buffer_capacity":
        range_space = np.linspace(buffer_capacity, max_parameter_value, accuracy)
    else:
        return "Arguement plot_over invalid"

    for parameter in range_space:
        if plot_over == "lambda_2":
            lambda_2 = parameter
        elif plot_over == "lambda_1":
            lambda_1 = parameter
        elif plot_over == "mu":
            mu = parameter
        elif plot_over == "num_of_servers":
            num_of_servers = parameter
        elif plot_over == "threshold":
            threshold = parameter
        elif plot_over == "system_capacity":
            system_capacity = parameter
        elif plot_over == "buffer_capacity":
            buffer_capacity = parameter

        num_of_servers = int(num_of_servers)
        threshold = int(threshold)
        system_capacity = int(system_capacity)
        buffer_capacity = int(buffer_capacity)

        times = abg.simulation.get_multiple_runs_results(
            lambda_2=lambda_2,
            lambda_1=lambda_1,
            mu=mu,
            num_of_servers=num_of_servers,
            threshold=threshold,
            system_capacity=system_capacity,
            buffer_capacity=buffer_capacity,
            num_of_trials=num_of_trials,
            seed_num=seed_num,
            runtime=runtime,
            patient_type="both",
            warm_up_time=100
        )
        simulation_blocking_times = [np.mean(b.blocking_times) for b in times]
        mean_blocking_time_sim = get_mean_blocking_time_simulation(
            lambda_2=lambda_2,
            lambda_1=lambda_1,
            mu=mu,
            num_of_servers=num_of_servers,
            threshold=threshold,
            system_capacity=system_capacity,
            buffer_capacity=buffer_capacity,
            num_of_trials=num_of_trials,
            seed_num=seed_num,
            runtime=runtime
        )

        mean_blocking_time_markov = get_mean_blocking_time_markov(
            lambda_2=lambda_2,
            lambda_1=lambda_1,
            mu=mu,
            num_of_servers=num_of_servers,
            threshold=threshold,
            system_capacity=system_capacity,
            buffer_capacity=buffer_capacity
        )

        all_times_sim.append(simulation_blocking_times)
        all_mean_times_sim.append(mean_blocking_time_sim)
        all_mean_times_markov.append(mean_blocking_time_markov)

    diff = (range_space[1] - range_space[0]) / 2

    plt.figure(figsize=(20, 10))
    plt.plot(range_space, all_mean_times_sim, label="Simulation State probabilities")
    plt.plot(range_space, all_mean_times_markov, label="Markov State probabilities")
    plt.violinplot(
        all_times_sim,
        positions=range_space,
        widths=diff,
        showmeans=True,
        showmedians=False,
    )
    title = (
        "lambda_2="
        + str(lambda_2)
        + ", lambda_1="
        + str(lambda_1)
        + ", mu="
        + str(mu)
        + ", C="
        + str(num_of_servers)
        + ", T="
        + str(threshold)
        + ", N="
        + str(system_capacity)
        + ", M="
        + str(buffer_capacity)
    )
    plt.title(title)
    plt.xlabel(plot_over)
    plt.ylabel("Blocking time")
    plt.legend()
    plt.show()
    return range_space, all_mean_times_sim, all_mean_times_markov, all_times_sim

```

```python

```

```python

```

# Plots

```python
lambda_2 = 1
lambda_1 = 1
mu = 2
num_of_servers = 1
threshold = 1
system_capacity = 2
buffer_capacity = 1

num_of_trials = 10
seed_num = None
runtime = 1000
accuracy = 5
```

```python
plt.figure(figsize=(20, 10))
abg.get_heatmaps(
    lambda_2=lambda_2,
    lambda_1=lambda_1,
    mu=mu,
    num_of_servers=num_of_servers,
    threshold=threshold,
    system_capacity=system_capacity,
    buffer_capacity=buffer_capacity,
    seed_num=seed_num,
    runtime=runtime,
    num_of_trials=num_of_trials,
)

```

```python
plot_over = "lambda_2"
max_parameter_value = 2
x_axis, mean_sim, mean_markov, all_sim = get_plot_comparing_times(
    lambda_2=lambda_2,
    lambda_1=lambda_1,
    mu=mu,
    num_of_servers=num_of_servers,
    threshold=threshold,
    num_of_trials=num_of_trials,
    seed_num=seed_num,
    runtime=runtime,
    system_capacity=system_capacity,
    buffer_capacity=buffer_capacity,
    plot_over=plot_over,
    max_parameter_value=max_parameter_value,
    accuracy=accuracy,
)

```

```python
plot_over = "lambda_1"
max_parameter_value = 2
x_axis, mean_sim, mean_markov, all_sim = get_plot_comparing_times(
    lambda_2=lambda_2,
    lambda_1=lambda_1,
    mu=mu,
    num_of_servers=num_of_servers,
    threshold=threshold,
    num_of_trials=num_of_trials,
    seed_num=seed_num,
    runtime=runtime,
    system_capacity=system_capacity,
    buffer_capacity=buffer_capacity,
    plot_over=plot_over,
    max_parameter_value=max_parameter_value,
    accuracy=accuracy,
)
```

```python
plot_over = "mu"
max_parameter_value = 4
x_axis, mean_sim, mean_markov, all_sim = get_plot_comparing_times(
    lambda_2=lambda_2,
    lambda_1=lambda_1,
    mu=mu,
    num_of_servers=num_of_servers,
    threshold=threshold,
    num_of_trials=num_of_trials,
    seed_num=seed_num,
    runtime=runtime,
    system_capacity=system_capacity,
    buffer_capacity=buffer_capacity,
    plot_over=plot_over,
    max_parameter_value=max_parameter_value,
    accuracy=accuracy,
)
```

```python
plot_over = "num_of_servers"
max_parameter_value = 5
x_axis, mean_sim, mean_markov, all_sim = get_plot_comparing_times(
    lambda_2=lambda_2,
    lambda_1=lambda_1,
    mu=mu,
    num_of_servers=num_of_servers,
    threshold=threshold,
    num_of_trials=num_of_trials,
    seed_num=seed_num,
    runtime=runtime,
    system_capacity=5,
    buffer_capacity=buffer_capacity,
    plot_over=plot_over,
    max_parameter_value=max_parameter_value,
    accuracy=accuracy,
)
```

```python
plot_over = "threshold"
max_parameter_value = 5
x_axis, mean_sim, mean_markov, all_sim = get_plot_comparing_times(
    lambda_2=lambda_2,
    lambda_1=lambda_1,
    mu=mu,
    num_of_servers=num_of_servers,
    threshold=threshold,
    num_of_trials=num_of_trials,
    seed_num=seed_num,
    runtime=runtime,
    system_capacity=system_capacity,
    buffer_capacity=buffer_capacity,
    plot_over=plot_over,
    max_parameter_value=max_parameter_value,
    accuracy=accuracy,
)
```

```python
plot_over = "system_capacity"
max_parameter_value = 10
x_axis, mean_sim, mean_markov, all_sim = get_plot_comparing_times(
    lambda_2=lambda_2,
    lambda_1=lambda_1,
    mu=mu,
    num_of_servers=num_of_servers,
    threshold=threshold,
    num_of_trials=num_of_trials,
    seed_num=seed_num,
    runtime=runtime,
    system_capacity=system_capacity,
    buffer_capacity=buffer_capacity,
    plot_over=plot_over,
    max_parameter_value=max_parameter_value,
    accuracy=accuracy,
)
```

```python
plot_over = "buffer_capacity"
max_parameter_value = 10
x_axis, mean_sim, mean_markov, all_sim = get_plot_comparing_times(
    lambda_2=lambda_2,
    lambda_1=lambda_1,
    mu=mu,
    num_of_servers=num_of_servers,
    threshold=threshold,
    num_of_trials=num_of_trials,
    seed_num=seed_num,
    runtime=runtime,
    system_capacity=system_capacity,
    buffer_capacity=buffer_capacity,
    plot_over=plot_over,
    max_parameter_value=max_parameter_value,
    accuracy=accuracy,
)
```

```python

```

```python

```

```python

```


# On the Fundamental Resonant Mode of Inhomogeneous Soild Deposits

This notebook details the calculations and the generation of results in "On the fundamental resonant mode of inhomogeneous soil deposits". The authors recommend using this notebook as the paper is being read. The references to equations and figures correspond to those in the text.

_Joaquin Garcia-Suarez, 2019 (All rights reserved)_

## Import python packages

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sym
import mpmath as mp
import cmath
import itertools
import array as arr
from scipy import special
from scipy.special import jv as besselJ
from scipy.special import yv as besselY
from scipy.optimize import fsolve
from math import pi as Pi
from cmath import sqrt as sqrt
```

Use inline magic to display plots on notebook

```python
%matplotlib inline
```

## Auxiliary Parameters

Parameters necessary to evaluate the exact response corresponding the "generalized-parabola distribution" (Mylonakis et. al 2013):

```python
def beta(alpha):
    return alpha**0.5
```

```python
def b(alpha,n):
    return beta(alpha)**(1/n)
```

```python
def q(alpha,n):
    return 1.0-b(alpha,n)
```

```python
def psi(n):
    return (1-2*n)/2
```

```python
def ell(n):
    return 2(1-n)
```

```python
def nu(n):
    return (2*n-1)/2/(1-n)
```

```python
def lambd(alpha,n,r,delta):
    auxval=r/sqrt(1+1j*delta)/(1-n)/(1-b(alpha,n))
    return auxval
```

Shear-wave velocity distribution ("generalized parabola"), eq.(3) in the text:

```python
def f(eta,n,alpha):
    return (1+(beta(alpha)**(1/n)-1)*eta)**n
```

## Previous and new results used in the paper (for generalized parabola distribution)

### Exact solutions for generalized parabola (Rovithis et al. 2011)

Base-to-top transfer function, eq.(7):

```python
def TFR(alpha,n,r,delta):
    aux_value=2/Pi*(b(alpha,n)**(psi(n)-1+n))/lambd(alpha,n,r,delta)*\
    (besselJ(nu(n)+1,lambd(alpha,n,r,delta)*b(alpha,n)**(1-n))*\
    besselY(nu(n),lambd(alpha,n,r,delta))-\
    besselY(nu(n)+1,lambd(alpha,n,r,delta)*b(alpha,n)**(1-n))*\
    besselJ(nu(n),lambd(alpha,n,r,delta)))**(-1)
    return aux_value
```

Function to find the exact fundamental frequency

```python
def findR(alpha,n):
    aux_val=lambda r: abs(besselJ(nu(n)+1,lambd(alpha,n,r,0)*b(alpha,n)**(1-n))*\
    besselY(nu(n),lambd(alpha,n,r,0))-\
    besselY(nu(n)+1,lambd(alpha,n,r,0)*b(alpha,n)**(1-n))*\
    besselJ(nu(n),lambd(alpha,n,r,0)))
    return float(fsolve(aux_val,1))
```

Vertical displacement evolution across stratum, eq(5):

```python
def Uexact(alpha,n,r,delta,eta):
    aux_value=((b(alpha,n)+q(alpha,n)*(1-eta))**psi(n))*\
    (besselJ(nu(n)+1,lambd(alpha,n,r,delta)*b(alpha,n)**(1-n))*\
    besselY(nu(n),lambd(alpha,n,r,delta)*(b(alpha,n)+q(alpha,n)*(1-eta))**(1-n))-\
    besselY(nu(n)+1,lambd(alpha,n,r,delta)*b(alpha,n)**(1-n))*\
    besselJ(nu(n),lambd(alpha,n,r,delta)*(b(alpha,n)+q(alpha,n)*(1-eta))**(1-n)))*\
    (besselJ(nu(n)+1,lambd(alpha,n,r,delta)*b(alpha,n)**(1-n))*\
    besselY(nu(n),lambd(alpha,n,r,delta))-\
    besselY(nu(n)+1,lambd(alpha,n,r,delta)*b(alpha,n)**(1-n))*\
    besselJ(nu(n),lambd(alpha,n,r,delta)))**(-1)
    return aux_value
```

### Estimate of the fundamental frequency applying Rayleigh Quotient (Mylonakis et al. 2011)

This function returns the value of the Rayleigh Quotient, eq.(4):

```python
def RayEst(alpha,n):
    aux_value2=(15/2*(-1-3*n-2*n**2+(1+2*n)*(3+2*n)*alpha**(1/2/n)-\
    (1+n)*(3+2*n)*alpha**(1/n)+alpha**(1+3/2/n))/(1+n)/(1+2*n)/(3+2*n)\
    /(-1+alpha**(1/2/n))**3)**(1/2)
    return aux_value2
```

### Results based on asymptotic behavior of exact solution (Garcia-Suarez, Seylabi and Asimaki, 2019)

Base-to-top transfer function, eq.(8):

```python
def Asy_TR(alpha,n,r,delta):
    aux_value=(alpha**(-1/4))/np.cos((1-alpha**((1-n)/2/n))/(1-n)/(1-b(alpha,n))*r/sqrt(1+1j*DE))
    return aux_value
```

Corresponding value of fundamental frequency

```python
def AsyEst(alpha,n):
    aux_value2=(1-n)*(1-beta(alpha)**(1/n))/(1-beta(alpha)**((1-n)/n))
    return aux_value2
```

### New results deriving from the resonance argument

Estimate of the displacement field, eq.(15):

```python
def Uest(alpha,n,r,delta,eta):
    auxV=1-(r**2)/delta*1j*((1-2*b(alpha,n)+2*n*(-1+b(alpha,n))+(-1+b(alpha,n)*\
    (2+2*n*(-1+eta)-eta)-2*n*(-1+eta)+eta)*(1+(b(alpha,n)-1)*eta)\
    **(1-2*n))/(2*(-1+n)*(-1+2*n)*(-1+b(alpha,n))**2))
    return auxV
```

Dichotomy for displacement at resonance, eq.(18) (converted to total displacements):

```python
def Uest2(delta):
    auxV=1-1/delta*1j
    return auxV
```

The new estimate of the fundamental frequency based on the resonance argument, eq.(20b):

```python
def IntegralEst(alpha,n):
    aux_value1=(Pi/8*(alpha-2*n*alpha+2*(-1+n)*alpha**(1+1/2/n)+alpha**(1/n))/\
               ((-1+n)*(-1+2*n)*alpha*(-1+b(alpha,n))**2))**(-1/2)
    return aux_value1
```

## Generate Figure 4 and 5: comparison of magnitude and phase of displacement field at resonance, assuming inhomogeneity given by gen. parabola
Fix the first set of values to input:

```python
NN=0.1
BETA=0.1
DE=0.1
ALPHA=BETA**2
#find the corresponding frequency first
RR=findR(ALPHA,NN)
```

### Figure 4

```python
y=np.linspace(1,0)
plt.figure(figsize=(8,8))
plt.tight_layout()
plt.subplot(1,2,1)
plt.plot(abs(Uexact(ALPHA,NN,RR,DE,y)),y,'r',\
         abs(Uest(ALPHA,NN,RayEst(ALPHA,NN),DE,y)),y,'gray',\
         abs(np.cos(Pi/2/sqrt(1+1j*DE)*(1-y))/np.cos(Pi/2/sqrt(1+1j*DE))),y,'b',\
         abs(Uest2(DE))*np.ones(50),y,'k--');

plt.legend(('Exact', 'Estimate #1','Homogeneus (base)','Estimate #2'),
           loc='upper left')
plt.title('Displacement magnitude and phase during resonance across stratum')

plt.xlabel('$|u/X_g|$',fontsize=15);
plt.ylabel('$\eta$',fontsize=18);

plt.subplot(1,2,2)
plt.plot(np.angle(Uexact(ALPHA,NN,RR,DE,y)),y,'r',\
         np.angle(Uest(ALPHA,NN,RR,DE,y)),y,'gray',\
         np.angle(np.cos(Pi/2/sqrt(1+1j*DE)*(1-y))/np.cos(Pi/2/sqrt(1+1j*DE))),y,'b',\
         np.angle(Uest2(DE))*np.ones(50),y,'k--');

plt.xlabel('$arg(u/X_g)$',fontsize=15);
```

### Figure 5
New set of values

```python
NN2=0.6
BETA2=0.3
ALPHA2=BETA2**2
#find the corresponding frequency
RR2=findR(ALPHA2,NN2)
```

```python
plt.figure(figsize=(8,8))
plt.tight_layout()
plt.subplot(1,2,1)
plt.title('Displacement magnitude and phase during resonance across stratum')
plt.plot(abs(Uexact(ALPHA2,NN2,RR2,DE,y)),y,'r',\
         abs(Uest(ALPHA2,RayEst(ALPHA2,NN2),RR2,DE,y)),y,'gray',\
         abs(np.cos(Pi/2/sqrt(1+1j*DE)*(1-y))/np.cos(Pi/2/sqrt(1+1j*DE))),y,'b',\
         abs(Uest2(DE))*np.ones(50),y,'k--');

plt.xlabel('$|u/X_g|$',fontsize=15);
plt.ylabel('$\eta$',fontsize=18);

plt.subplot(1,2,2)
plt.plot(np.angle(Uexact(ALPHA2,NN2,RR2,DE,y)),y,'r',\
         np.angle(Uest(ALPHA2,NN2,RayEst(ALPHA2,NN2),DE,y)),y,'gray',\
         np.angle(np.cos(Pi/2/sqrt(1+1j*DE)*(1-y))/np.cos(Pi/2/sqrt(1+1j*DE))),y,'b',\
         np.angle(Uest2(DE))*np.ones(50),y,'k--');

plt.xlabel('$arg(u/U)$',fontsize=15);
plt.legend(('Exact', 'Estimate #1','Homogeneus (base)','Estimate #2'),
           loc='upper right');
```

## Figure 6: Comparison of fundamental frequency of inhomoegenous stratum (generalized parabola)

```python
nn=np.linspace(0.01,0.99)
plotR=[]
for k in range(0,len(nn)):
    plotR.append(2/Pi*findR(0.2**2,nn[k]))
# Plot to compare against Mylonakis
plt.figure(figsize=(8,10))
plt.tight_layout()
plt.subplot(3,1,1)
# beta=0.2
plt.plot(nn,plotR,\
         nn,2/Pi*IntegralEst(0.2**2,nn),\
         nn,2/Pi*RayEst(0.2**2,nn),\
         nn,AsyEst(0.2**2,nn));
plt.axis([0, 1, 0, 1])

plt.legend(('Exact','Estimate #1','Rayleigh Quotient','Asymptotics'),
           loc='lower left')
plt.title('a) $c_{s,top} / c_{s,base}=0.2$')

plt.ylabel('$\omega_s/\omega_{s,base}$',fontsize=18);
# beta=0.5
plt.subplot(3,1,2)
plotR=[]
for k in range(0,len(nn)):
    plotR.append(2/Pi*findR(0.5**2,nn[k]))
plt.plot(nn,plotR,\
         nn,2/Pi*IntegralEst(0.5**2,nn),\
         nn,2/Pi*RayEst(0.5**2,nn),\
         nn,AsyEst(0.5**2,nn));
plt.axis([0, 1, 0, 1])

plt.legend(('Exact','Estimate #1','Rayleigh Quotient','Asymptotics'),
           loc='lower left')
plt.title('b) $c_{s,top} / c_{s,base}=0.5$')

plt.ylabel('$\omega_s/\omega_{s,base}$',fontsize=18);
# beta=0.8
plt.subplot(3,1,3)
plotR=[]
for k in range(0,len(nn)):
    plotR.append(2/Pi*findR(0.8**2,nn[k]))
plt.plot(nn,plotR,\
         nn,2/Pi*IntegralEst(0.8**2,nn),\
         nn,2/Pi*RayEst(0.8**2,nn),\
         nn,AsyEst(0.8**2,nn));
plt.axis([0, 1, 0, 1])

plt.legend(('Exact','Estimate #1','Rayleigh Quotient','Asymptotics'),
           loc='lower left')
plt.title('c) $c_{s,top} / c_{s,base}=0.8$')

plt.ylabel('$\omega_s/\omega_{s,base}$',fontsize=18);
plt.xlabel('$n$',fontsize=18);
```

## Generate Figure 7: amplification comparison assuming gen. parabola

```python
rr=np.linspace(0.1,6*Pi/2,1000)
DE=0.15
NN=0.499
BETA=0.9
ALPHA=BETA**2
# Plot to compare amplitude
plt.figure(figsize=(8,10))
plt.tight_layout()
# FIRST TILE
plt.subplot(3,1,1)
plt.title('a) $c_{s,top} / c_{s,base}=0.9$, damping=15%')
plt.plot(rr*2/Pi,abs(TFR(ALPHA,NN,rr,DE)));
plt.plot(rr*2/Pi,abs(Asy_TR(ALPHA,NN,rr,DE)));
# vertical lines
w_locus=RayEst(ALPHA,NN) # frequency
amplitude=abs(Uest(ALPHA,NN,RayEst(ALPHA,NN),DE,1))
plt.plot([w_locus/(Pi/2),w_locus/(Pi/2)],[0,amplitude],'k--')
plt.axis([0, 6, 0, 15])
w_locus2=IntegralEst(ALPHA,NN) # frequency
amplitude2=4/Pi/DE
plt.plot([w_locus2/(Pi/2),w_locus2/(Pi/2)],[0,amplitude2],'g')
plt.axis([0, 6, 0, 15])
# horizontal lines
plt.plot([0,w_locus/(Pi/2)],[amplitude,amplitude],'k--')
plt.plot([0,w_locus2/(Pi/2)],[amplitude2,amplitude2],'g')

plt.legend(('Exact','Asymptotics','Estimate based on RQ','Estimate based on resonance argument'),
           loc='upper right')

plt.ylabel('$A(\omega)$',fontsize=18);
# SECOND TILE
DE=0.10
BETA=0.5
ALPHA=BETA**2
plt.subplot(3,1,2)
plt.title('b) $c_{s,top} / c_{s,base}=0.5$, damping=10%')
plt.plot(rr*2/Pi,abs(TFR(ALPHA,NN,rr,DE)));
plt.plot(rr*2/Pi,abs(Asy_TR(ALPHA,NN,rr,DE)));
# vertical lines
w_locus=RayEst(ALPHA,NN) # frequency
amplitude=abs(Uest(ALPHA,NN,RayEst(ALPHA,NN),DE,1))
plt.plot([w_locus/(Pi/2),w_locus/(Pi/2)],[0,amplitude],'k--')
plt.axis([0, 6, 0, 15])
w_locus2=IntegralEst(ALPHA,NN) # frequency
amplitude2=4/Pi/DE
plt.plot([w_locus2/(Pi/2),w_locus2/(Pi/2)],[0,amplitude2],'g')
plt.axis([0, 6, 0, 20])
# horizontal lines
plt.plot([0,w_locus/(Pi/2)],[amplitude,amplitude],'k--')
plt.plot([0,w_locus2/(Pi/2)],[amplitude2,amplitude2],'g')

plt.ylabel('$A(\omega)$',fontsize=18);
# THIRD TILE
DE=0.05
BETA=0.1
ALPHA=BETA**2
plt.subplot(3,1,3)
plt.title('c) $c_{s,top} / c_{s,base}=0.1$, damping=5%')
plt.plot(rr*2/Pi,abs(TFR(ALPHA,NN,rr,DE)));
plt.plot(rr*2/Pi,abs(Asy_TR(ALPHA,NN,rr,DE)));
# vertical lines
w_locus=RayEst(ALPHA,NN) # frequency
amplitude=abs(Uest(ALPHA,NN,RayEst(ALPHA,NN),DE,1))
plt.plot([w_locus/(Pi/2),w_locus/(Pi/2)],[0,amplitude],'k--')
plt.axis([0, 6, 0, 15])
w_locus2=IntegralEst(ALPHA,NN) # frequency
amplitude2=4/Pi/DE
plt.plot([w_locus2/(Pi/2),w_locus2/(Pi/2)],[0,amplitude2],'g')
plt.axis([0, 6, 0, 40])
# horizontal lines
plt.plot([0,w_locus/(Pi/2)],[amplitude,amplitude],'k--')
plt.plot([0,w_locus2/(Pi/2)],[amplitude2,amplitude2],'g')

plt.ylabel('$A(\omega)$',fontsize=18);
plt.xlabel('$\omega/\omega_{s,base}$',fontsize=18);
```

## Generate Figure 8: amplification comparison layered site¶

### Function to calculate transfer function for discontinuous layers
This corresponds to an implementation of the classic procedure, which is detailed, e.g., in Kramer's(1996)

```python
def getAmp(alpha_m,H,V_base,Vs,delta,hm):
    Amp=np.zeros(len(om))
    for vv in range(np.size(Amp)):
        L=np.array([[1, 0], [0, 1]])
        for uu in range(len(alpha_m)):
            hmkm=1j*Pi/2*hm[uu]/H*V_base/Vs[uu]*om[vv]*(1-1j*delta/2)
            Laux=np.array([[0.5*(1+alpha_m[uu])*cmath.exp(hmkm),\
            0.5*(1-alpha_m[uu])*cmath.exp(-hmkm)],\
            [0.5*(1-alpha_m[uu])*cmath.exp(hmkm),\
            0.5*(1+alpha_m[uu])*cmath.exp(-hmkm)]])
            L=np.dot(Laux,L) #the new matrix multiply the previous one from the left
        Amp[vv]=2/abs(L[0,0]+L[0,1]+L[1,0]+L[1,1])
    return Amp
```

### Function to calculate the integral estimate of the fundamental frequency for layered site

```python
def getWn(ZZ,Vs,depth):
    # Discretize and normalize Vs into Vs_disc (same #points as ZZ)
    Vs_disc=np.linspace(0,1,1000)
    ii=0
    jj=0
    while ii < len(ZZ):
        # ZZ[ii] in what interval?
        if ZZ[ii]<=depth[jj]: # in the j-th interval
            Vs_disc[ii]=Vs[jj]
        else: # in the next one
            jj+=1
            Vs_disc[ii]=Vs[jj]
        ii+=1
    Vs_disc[-1]=Vs[-1]
    Vs_disc=np.flip(Vs_disc)/Vs[-1]
    # Calculate the weighted harmonic mean of the shear modulus
    C_cor=0
    for ii in range(1000):
        C_cor=C_cor+1/Vs_disc[-(ii+1)]/Vs_disc[-(ii+1)]\
        *ZZ[ii]/H*(ZZ[1]-ZZ[0])/H
    return abs(sqrt(16.0/C_cor/Pi**3))
```

Range of frequencies to be used for plots (expressed as multiples of the fundamental frequency of the homogeneous stratum based on the properties at the base)

```python
om=np.linspace(0,6,1000)
```

### Quick Test (Homogeneous case)

Let us run an example divided in multiple layers with the same properties

```python
DE=0.1
depth=arr.array('f',[20,190,410,518]) # [m]
Vs=arr.array('f',[110,110,110,110]) # Vs distribution [m/s]
#
H=depth[-1] # soft soil depth [m]
ZZ=np.linspace(0,H,1000) # [m]
YY=np.flip(1-ZZ/H) # profile distance from base to top [m]
V_base=Vs[-1] # Vs of the soil in contact with bedrock
# Inter-layer impedance
alpha_m=np.divide(Vs[0:len(Vs)-1],Vs[1:len(Vs)])
alpha_m=alpha_m.tolist()
alpha_m.append(0)
# Layer depth
hm=np.subtract(depth[1:len(depth)],depth[0:len(depth)-1])
hm=np.concatenate(([depth[0]],hm),axis=0)
```

```python
AmpHom=np.zeros(len(om))
for vv in range(np.size(AmpHom)):
    AmpHom[vv]=abs(1/cmath.cos(Pi/2*om[vv]*(1-1j*DE/2)))
```

```python
plt.plot(om,getAmp(alpha_m,H,V_base,Vs,DE,hm));
plt.axis([0, 6, 0, 15]);
plt.plot(om,AmpHom,'--')

plt.legend(('Using getAmp','Homogeneous'),
           loc='upper right')

plt.ylabel('$A(\omega)$',fontsize=18);
plt.xlabel('$\omega/\omega_{s,base}$',fontsize=18);
plt.title('IBRH10');
```

### KikNet Data: IBRH10

```python
depth=arr.array('f',[20,190,410,518]) # [m]
Vs=arr.array('f',[110,380,530,850]) # Vs distribution [m/s]
#
H=depth[-1] # soft soil depth [m]
ZZ=np.linspace(0,H,1000) # [m]
YY=np.flip(1-ZZ/H) # profile distance from base to top [m]
V_base=Vs[-1] # Vs of the soil in contact with bedrock
# Inter-layer impedance
alpha_m=np.divide(Vs[0:len(Vs)-1],Vs[1:len(Vs)])
alpha_m=alpha_m.tolist()
alpha_m.append(0)
# Layer depth
hm=np.subtract(depth[1:len(depth)],depth[0:len(depth)-1])
hm=np.concatenate(([depth[0]],hm),axis=0)
```

```python
DE=0.035
plt.plot(om,getAmp(alpha_m,H,V_base,Vs,DE,hm));
plt.axis([0, 6, 0, 40]);
# vertical lines
w_locus=getWn(ZZ,Vs,depth) # frequency
amplitude=1+Pi*Pi/8/DE
plt.plot([w_locus,w_locus],[0,amplitude],'k--')
# horizontal lines
plt.plot([0,w_locus],[amplitude,amplitude],'k--')

plt.legend(('Exact','Estimate based on resonance argument'),
           loc='upper right')

plt.ylabel('$A(\omega)$',fontsize=18);
plt.xlabel('$\omega/\omega_{s,base}$',fontsize=18);
plt.title('IBRH10');
```

### KikNet Data: TKCH08

```python
depth=arr.array('f',[4,36,78]) # [m]
Vs=arr.array('f',[130,480,590]) # Vs distribution [m/s]
#
H=depth[-1] # soft soil depth [m]
ZZ=np.linspace(0,H,1000) # [m]
YY=np.flip(1-ZZ/H) # profile distance from base to top [m]
V_base=Vs[-1] # Vs of the soil in contact with bedrock
# Inter-layer impedance
alpha_m=np.divide(Vs[0:len(Vs)-1],Vs[1:len(Vs)])
alpha_m=alpha_m.tolist()
alpha_m.append(0)
# Layer depth
hm=np.subtract(depth[1:len(depth)],depth[0:len(depth)-1])
hm=np.concatenate(([depth[0]],hm),axis=0)
```

```python
DE=0.068
plt.plot(om,getAmp(alpha_m,H,V_base,Vs,DE,hm));
plt.axis([0, 6, 0, 25]);
# vertical lines
w_locus=getWn(ZZ,Vs,depth) # frequency
amplitude=1+Pi*Pi/8/DE
plt.plot([w_locus,w_locus],[0,amplitude],'k--')
# horizontal lines
plt.plot([0,w_locus],[amplitude,amplitude],'k--')

plt.legend(('Exact','Estimate based on resonance argument'),
           loc='upper right')

plt.ylabel('$A(\omega)$',fontsize=18);
plt.xlabel('$\omega/\omega_{s,base}$',fontsize=18);
plt.title('TKCH08');
```

### KikNet Data: IBRH17

```python
depth=arr.array('f',[1,10,90,235,300,380,460]) # [m]
Vs=arr.array('f',[90,250,380,470,540,660,820]) # Vs distribution [m/s]
#
H=depth[-1] # soft soil depth [m]
ZZ=np.linspace(0,H,1000) # [m]
YY=np.flip(1-ZZ/H) # profile distance from base to top [m]
V_base=Vs[-1] # Vs of the soil in contact with bedrock
# Inter-layer impedance
alpha_m=np.divide(Vs[0:len(Vs)-1],Vs[1:len(Vs)])
alpha_m=alpha_m.tolist()
alpha_m.append(0)
# Layer depth
hm=np.subtract(depth[1:len(depth)],depth[0:len(depth)-1])
hm=np.concatenate(([depth[0]],hm),axis=0)
```

```python
DE=0.147
plt.plot(om,getAmp(alpha_m,H,V_base,Vs,DE,hm));
plt.axis([0, 6, 0, 10]);
# vertical lines
w_locus=getWn(ZZ,Vs,depth) # frequency
amplitude=1+Pi*Pi/8/DE
plt.plot([w_locus,w_locus],[0,amplitude],'k--')
# horizontal lines
plt.plot([0,w_locus],[amplitude,amplitude],'k--')

plt.legend(('Exact','Estimate based on resonance argument'),
           loc='upper right')

plt.ylabel('$A(\omega)$',fontsize=18);
plt.xlabel('$\omega/\omega_{s,base}$',fontsize=18);
plt.title('IBRH17');
```

Fundamental Resonant Mode Inhomogeneous Soil Deposits.ipynb


# Lecture 34: A Look Ahead; Examples of Regression Example, Sampling from a Finite Population

## Stat 110, Prof. Joe Blitzstein, Harvard University

----

## The Top-10 List

1. Conditioning ... is the soul of statistics!
1. Symmetry ... is powerful but dangerous
1. Random variables and their distributions
1. Stories (proofs, backgrounds of the distributions covered)
1. Linearity
1. Indicator random variables
1. LOTUS
1. Law of Large Numbers
1. Central Limit Theorem
1. Markov Chains

Items 1 through 4 deal with the Big Picture™ questions: _What is randomness? How do we think about uncertainty?_

Items 5 through 7 are for computing expected values (mean, variance & standard deviation).

Items 8 through 10 are important for understanding long-run behavior.

## Where to go from here?

Some topics to study from here on out:

* Statistical inference (we have data, need to estimate parameters or make predictions)
* Regress & linear models
* Finance
* Computational biology
* Stochastic processes

## Advice

* Learn R
* Learn C
* Read Mostly Harmless Econometrics

## Ex. A Simple Linear Regression

You've seen this before:

\begin{align}
  Y &= \beta_0 + \beta_1 \, X + \epsilon
\end{align}

* We want to use $X$ to predict $Y$
* $\beta_j$ are linear coeffiecients, with $\beta_0$ being the value of $Y$ when $x=0$ (default value)
* $\epsilon$ error term (since $X$ is not perfect)
* a common assumption is $\mathbb{E}(\epsilon | X) = 0$ (centered at 0, $\epsilon$'s distribution may or may not be normal)

So how would we solve for $\beta_1$?

We can start by treating $Cov$ as an _operator_!

\begin{align}
  Cov(Y, X) &= Cov\left( (\beta_0 + \beta_1 \, X + \epsilon), X \right) \\
  &= Cov(\beta_0, X) + Cov\left( (\beta_1 \, X), X\right) + Cov(\epsilon, X) \\
  \\
  \text{now } Cov(\beta_0, X) &= 0 &\quad \text{ since } Cov \text{ of constant with anything is } 0 \\
  \\
  \text{and } Cov\left( (\beta_1 \, X), X\right) &= \beta_1 \, Cov(X, X) &\quad \text{by definition of }Var \\
  &= \beta_1 \, Var(X) \\
  \\
  \text{and since } \mathbb{E}(\epsilon) &= \mathbb{E}\left( \mathbb{E}(\epsilon|X) \right) = \mathbb{E}(0) = 0 \\
  \text{and further } \mathbb{E}(\epsilon \, X) &= \mathbb{E}\left( \mathbb{E}(\epsilon \, X | X)  \right) &\quad \text{ by Adam's Law} \\
  &= \mathbb{E}\left( X \mathbb{E}(\epsilon | X) \right) &\quad \text{ since } X \text{ is known, we can pull it out} \\
  &= \mathbb{E}(0) \\
  &= 0 \\
  \text{so }Cov(\epsilon, X) &= \mathbb{E}(\epsilon \, X) - \mathbb{E}(\epsilon) \, \mathbb{E}(X) \\
  &= 0 - 0 = 0 \\\\
  \Rightarrow \beta_1 &= \frac{Cov(X,Y)}{Var(X)} &\quad \text{(population version)}
\end{align}

### Calculate $\beta_1$ with $Cov(X,Y)$ and $Var(X)$

Here we calculate $\beta_1 = \frac{Cov(X,Y)}{Var(X)} \,$ using [numpy.cov](https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.cov.html):

```python
import numpy as np

X = np.array([95, 85, 80, 70, 60])
Y = np.array([85, 95, 70, 65, 70])

# numpy.cov(X, Y) returns the matrix
# [ Cov(X,X), Cov(X,Y)]
# [ Cov(X,Y), Cov(Y,Y)]
covM = np.cov(X,Y)

beta_1 = covM[0,1]/covM[0,0]
beta_1
```

    0.64383561643835618

### Calculate $\beta_1$ via sklearn LinearRegression API

For comparison's sake, we also obtain $\beta_1$ via [`sklearn.linear_model.LinearRegression`](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html)

```python
from sklearn import linear_model

regr = linear_model.LinearRegression()
regr.fit(np.matrix(X).T, Y).coef_[0]
```

    0.64383561643835607

----

## Ex. Sampling from a Finite Population

Here's the set-up:

* We have a finite population , size $N$.
* Let $Y_1, Y_2, ..., Y_N$ be some value of interest (height, weight, opinion.
* Each person in the population can be uniquely identified.
* $Y_j$ are fixed, non-random values, but nevertheless _unknown_.
* Using some sampling scheme to obtain a sample of size $n$.
* Using this sample, we want to infer to the sum of $Y$ (or perhaps the average).
* Assume that the _inclusion probability_ of person $j$ ending up in our sample is $p_j$ (assume that the true value is known).
* Our sample data takes the form $(X_1, Z_1), (X_2, Z_2), ..., (X_n, Z_n)$, where
   * $Z_j$ is the ID of the $j^{th}$ person in our sample
   * $X_j = Y_j$

### The difference between $X_j$ and $Y_j$

It is important to understand the difference between $X_j$ and $Y_j$:

* $Y_j$ is a **_fixed, non-random value_**.
* $X_j$, due to our random sampling (person $j$ was randomly selected from the population with probability $p_j$), is a **_random variable_**.

### How do we get an unbiased estimator for the total?

Let $t_y$ be the true population total $\sum_{1}^{N} Y_i$. How can we use random sampling of this finite population to find $\hat{t_y}$?

The claim is that $\sum_{j=1}^{n} \frac{X_j}{P_{Z_j}}$ is an unbiased estimator for $t_y$ we are looking for.

\begin{align}
  t_y &= \sum_{j=1}^{n} \frac{X_j}{P_{Z_j}} \\
  &= \sum_{j=1}^{N} \frac{I_j \, Y_j}{P_j} &\quad \text{where } I_j = 1 \text{ if person } j \text{ included in sample}\\\\
  \mathbb{E}(t_y) &=  \mathbb{E}\left( \sum_{j=1}^{N} \frac{I_j \, Y_j}{P_j} \right) &\quad \text{ find expected value to get } \hat{t_y} \\
  &= \sum_{j=1}^{N} \frac{P_j \, Y_j}{P_j} &\quad \text{ by linearity} \\
  &= \boxed{ \sum_{j=1}^{N} Y_j }
\end{align}

This is known as the [Horvitz-Thompson Estimator](https://en.wikipedia.org/wiki/Horvitz%E2%80%93Thompson_estimator), or alternately inverse probability weighting.

_But is an unbiased estimator good?_

### Basu's Circus Elephants

Statistics is not easy, and it requires a lot of effort to keep your eyes open and question whether or not a tentative method is really going to yield a proper answer. Here is an anecdote to illustrate an example of when blindly applying an Horvitz-Thompson estimate ends in disaster.

> The circus owner is planning to ship his 50 adult elephants and so he needs a rough estimate of the total
> weight of the elephants. As weighing an elephant is a cumbersome process, the owner wants to estimate
> the total weight by weighing just one elephant. Which elephant should he weigh?
So the owner looks
> back on his records and discovers a list of the elephants' weights taken 3 years ago. He finds that 3 years
> ago Sambo the middle-sized elephant was the average (in weight) elephant in his herd. He checks with
> the elephant trainer who reassures him (the owner) that Sambo may still be considered to be the average
> elephant in the herd. Therefore, the owner plans to weigh Sambo and take 50y (where y is the present
> weight of Sambo) as an estimate of the total weight of the 50 elephants.
But the circus statistician is
> horrified when he learns of the owner's proposed sampling plan. "How can you get an unbiased estimate
> of Y this way?", protests the statistician.
So, together they work out a compromise sampling plan. With
> the help of a table of random numbers they devise a plan that allots a selection probability of 99/100 to
> Sambo and equal selection probabilities of 1/4900 to each of the other 49 elephants. Naturally, Sambo is
> selected and the owner is happy.
"How are you going to estimate Y?", asks the statistician.
"Why? The
> estimate ought to be 50y of course," says the owner.
"Oh! No! That cannot possibly be right," says the
> statistician, "I recently read an article in the Annals of Mathematical Statistics where it is proved that
> the Horvitz-Thompson estimator is the unique hyperadmissible estimator in the class of all generalized
> polynomial unbiased estimators."
"What is the Horvitz-Thompson estimate in this case?" asks the owner,
> duly impressed.
"Since the selection probability for Sambo in our plan was 99/100," says the statistician,
> "the proper estimate of Y is 100y/99 and not 50y."
"And, how would you have estimated Y", inquires
> the incredulous owner, "if our sampling plan made us select, say, the big elephant Jumbo?"
"According to
> what I understand of the Horvitz-Thompson estimation method," says the unhappy statistician, "the proper
> estimate of Y would then have been 4900y, where y is Jumbo's weight".
That is how the statistician lost
> his circus job (and perhaps became a teacher of statistics)

----

View [Lecture 34: A Look Ahead | Statistics 110](http://bit.ly/2Npa7ze) on YouTube.


## Subsurface Data Analytics

## Artificial Neural Networks for Prediction in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

#### Honggeun Jo, Graduate Student, The University of Texas at Austin

##### [LinkedIn](https://www.linkedin.com/in/honggeun-jo/?originalSubdomain=kr) | [Twitter](https://twitter.com/HonggeunJ)

### PGE 383 Exercise: Artificial Neural Networks for Subsurface Modeling in Python

Here's a simple workflow, demonstration of artificial neural networks for subsurface modeling workflows. This should help you get started with building subsurface models that use data analytics and machine learning. Here's some basic details about neural networks.

#### Neural Networks

Machine learning method for supervised learning for classification and regression analysis.  Here are some key aspects of support vector machines.

**Basic Design** *"...a computing system made up of a number of simple, highly interconnected processing elements, which process information by their dynamic state response to external inputs."* Caudill (1989).

**Nature-inspire Computing** based on the neuronal structure in the brain, including many interconnected simple, processing units, known as nodes that are capable of complicated emergent pattern detection due to a large number of nodes and interconnectivity.

**Training and Testing** just like and other predictive model (e.g. linear regression, decision trees and support vector machines) we perform training to fit parameters and testing to tune hyperparameters.  Here we observe the error with training and testing datasets, but do not demonstrate tuning of the hyperparameters.

**Parameters** are the weights applied to each connection and a bias term applied to each node.  For a single node in an artificial neural network, this includes the slope terms, $\beta_i$, and the bias term, $\beta_{0}$.

\begin{equation}
Y = \sum_{i=1}^m \beta_i X + \beta_0
\end{equation}

it can be seen that the number of parameters increases rapidly as we increase the number of nodes and the connectivity between the nodes.

**Layers** the typical artificial neural net is structured with an **input layer**, with one node for each $m$ predictor feature, $X_1,\ldots,X_m$. There is an **ouput layer**, with one node for each $r$ response feature, $Y_1,\ldots,Y_r$.  There may be one or more layers of nodes between the input and output layers, known as **hidden layer(s)**.

**Connections** are the linkages between the nodes in adjacent layers.  For example, in a fully connected artificial neural network, all the input nodes are connected to all of the nodes in the first layer, then all of the nodes in the first layer are connected to the next layer and so forth. Each connection includes a weight parameter as indicated above.

**Nodes** receive the weighted signal from the connected previous layer nodes, sum and then apply this result to the **activation** function in the node. Some example activation functions include:

* **Binary** the node fires or not.  This is represented by a Heaviside step function.

* **Identify** the input is passed to the output $f(x) = x$

* **Linear** the node passes a signal that increases linearly with the weighted input.

* **Logistic** also known as sigmoid or soft step $f(x) = \frac{1}{1+e^{-x}}$

the node output is the nonlinear activiation function applied to the linearly weighted inputs. This is fed to all nodes in the next layer.

**Training Cycles** - the presentation of a batch of data, forward application of the current prediction model to make estimates, calculation of error and then backpropagation of error to correct the artificial neural network parameters to reduce the error over all of the batches.

**Batch** is the set of training data for each training cycle of forward prediction and back propagation of error, drawn to train for each iteration. There is a trade-off, a larger batch results in more computational time per iteration, but a more accurate estimate of the error to adjust the weights.  Smaller batches result in a nosier estimate of the error, but faster epochs, this results in faster learning and even possibly more robust models.

**Epochs** - is a set of training cycles, batches covering all available training data.

**Local Minimums** - if one calculated the error hypersurface over the range of model parameters it would be hyparabolic, there is a global minimium error solution.  But this error hyper surface is rough and it is possible to be stuck in a local minimum. **Learning Rate** and **Mommentum** coefficients are introduced to avoid getting stuck in local minimums.

* **Mommentum** is a hyperparameter to control the use of information from the weight update over the last epoch for consideration in the current epoch.  This can be accomplished with an update vector, $v_i$, a mommentum parameter, $\alpha$, to calculate the current weight update, $v_{i+1}$ given the new update $\theta_{i+1}$.

\begin{equation}
v_{i+1} = \alpha v_i + \theta_{i+1}
\end{equation}

* **Learning Rate** is a hyperparameter that controls the adjustment of the weights in response to the gradient indicated by backpropagation of error

##### Applications to subsurface modeling

We demonstrate the estimation of normal score transformed porosity from depth.  This would be useful for building a vertical trend model.

* modeling the complicated relationship between porosity and depth.

#### Limitations of Neural Network Estimation

Since we demonstrate the use of an artificial neural network to estimate porosity from sparsely sampled data over depth, we should comment on limitations of our artificial neural networks for this estimation problem:

* does not honor the well data

* does not honor the histogram of the data

* does not honor spatial correlation

* does not honor the multivariate relationship

* generally low interpretability models

* requires a large number of data for effective training

* high model complexity with high model variance

#### Workflow Goals

Learn the basics of machine learning in python to predict subsurface features. This includes:

* Loading and visualizing sample data
* Trying out neural nets

#### Objective

In the PGE 383: Subsurface Machine Learning Class, I want to provide hands-on experience with building subsurface modeling workflows. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - 12_sample_data.csv found [here](https://github.com/GeostatsGuy/GeoDataSets/blob/master/12_sample_data.csv).

There are exampled below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

#### Import Required Packages

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
import geostatspy.GSLIB as GSLIB          # GSLIB utilies, visualization and wrapper
import geostatspy.geostats as geostats    # GSLIB methods convert to Python
```

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # for plotting
import seaborn as sns                     # for plotting
import warnings                           # supress warnings from seaborn pairplot
from sklearn.model_selection import train_test_split # train / test DatFrame split
```

We will also need the following packages to train and test our artificial neural nets:

* Tensorflow - open source machine learning, with Keras modulw - high level application programing interface (API) to build and train models

More information is available at [tensorflow install](https://www.tensorflow.org/install).

* This workflow was designed with tensorflow version > 2.0.

To import Tensorflow, set memory growth and check your current version of tensorflow you could run the next block of code.

```python
import tensorflow as tf
physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], True)
tf.__version__                            # check the installed version of tensorflow
```

    '2.4.0'

Let's import all of the tensorflow and keras methods that we will need in our workflow.

```python
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation, Dropout
#from keras.models import Sequential, load_model
#from keras.layers.core import Dense, Dropout, Activation
from tensorflow.keras.optimizers import Adam
from tensorflow.python.keras import backend as k
```

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
os.chdir("c:/PGE383")
```

#### Loading Data
Let's load the provided multivariate, spatial dataset '12_sample_data.csv'.  It is a comma delimited file with:

* Depth ($m$)
* Normal Score Porosity

It is common to transform properties to standard normal for geostatistical workflows.

We load it with the pandas 'read_csv' function into a data frame we called 'df' and then preview it to make sure it loaded correctly.

**Python Tip: using functions from a package** just type the label for the package that we declared at the beginning:

```python
import pandas as pd
```

so we can access the pandas function 'read_csv' with the command:

```python
pd.read_csv()
```

but read csv has required input parameters. The essential one is the name of the file. For our circumstance all the other default parameters are fine. If you want to see all the possible parameters for this function, just go to the docs [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html).

* The docs are always helpful
* There is often a lot of flexibility for Python functions, possible through using various inputs parameters

also, the program has an output, a pandas DataFrame loaded from the data.  So we have to specficy the name / variable representing that new object.

```python
df = pd.read_csv("1D_Porosity.csv")
```

Let's run this command to load the data and then look at the resulting DataFrame to ensure that we loaded it.

```python
df2 = pd.read_csv("1D_Porosity.csv")                    # read a .csv file in as a DataFrame                                     # display first 4 samples in the table as a preview
df2.head()                                                   # we could also use this command for a table preview
```

      Depth
      Nporosity

      0
      0.25
      -1.37

      1
      0.50
      -2.08

      2
      0.75
      -1.67

      3
      1.00
      -1.16

      4
      1.25
      -0.24

#### Data Normalization

We must normalize the features before we apply them in an artificial neural network model. These are the motivations for this normalization:

* remove impact of scale of different types of data (i.e., depth varies between $[0,10]$, but porosity only varies between $[-3,3.0]$).

* activation functions in artificial neural networks are designed to be more sensive to values of nodes closer to 0.0 (i.e., results in higher gradient and improves backpropagation in training)

Let's normalize each feature.

* We apply the min max normalization by-hand to force both the predictor and response features to be bound $[-1,1]$.

* It is easy to backtransform given we keep track of the original min and max values

```python
depth_min = df2['Depth'].values.min(); depth_max = df2['Depth'].values.max()
Npor_min = df2['Nporosity'].values.min(); Npor_max = df2['Nporosity'].values.max()
df2['norm_Depth'] = (df2['Depth'] - depth_min)/(depth_max - depth_min) * 2 - 1
df2['norm_Nporosity'] = (df2['Nporosity'] - Npor_min)/(Npor_max - Npor_min) * 2 - 1
df2.head()
```

      Depth
      Nporosity
      norm_Depth
      norm_Nporosity

      0
      0.25
      -1.37
      -1.000000
      -0.679458

      1
      0.50
      -2.08
      -0.948718
      -1.000000

      2
      0.75
      -1.67
      -0.897436
      -0.814898

      3
      1.00
      -1.16
      -0.846154
      -0.584650

      4
      1.25
      -0.24
      -0.794872
      -0.169300

It is also a good idea to check the summary statistics.

* All normalized features should now range from -1.0 to 1.0

```python
df2.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Depth
      40.0
      5.125000
      2.922613
      0.25
      2.687500
      5.125000e+00
      7.562500
      10.00

      Nporosity
      40.0
      0.022250
      0.992111
      -2.08
      -0.757500
      1.400000e-01
      0.742500
      2.35

      norm_Depth
      40.0
      0.000000
      0.599510
      -1.00
      -0.500000
      -5.551115e-17
      0.500000
      1.00

      norm_Nporosity
      40.0
      -0.050903
      0.447906
      -1.00
      -0.402935
      2.257336e-03
      0.274266
      1.00

#### Separation of Training and Testing Data

We also need to split our data into training / testing datasets so that we:

* can train our artificial neural networks using the training data

* while testing their performance with the withheld testing (validation) data.

```python
X2 = df2.iloc[:,[0,2]]                        # extract the predictor feature - acoustic impedance
y2 = df2.iloc[:,[1,3]]                        # extract the response feature - porosity
X2_train, X2_test, y2_train, y2_test = train_test_split(X2, y2, test_size=0.2, random_state=73073)
```

#### Visualize the Dataset

Let's visualize the train and test data split on a single scatter plot.

* we want to make sure it is fair

* ensure that the test samples are not clustered or too far away from the training data.

We will look at the original data and normalized data, the input to the neural network.

```python
plt.subplot(211)
plt.plot(X2_train['Depth'].values,y2_train['Nporosity'].values, 'o', markerfacecolor='red', markeredgecolor='black', alpha=0.4, label = "Train")
plt.plot(X2_test['Depth'].values,y2_test['Nporosity'].values, 'o', markerfacecolor='blue', markeredgecolor='black', alpha=0.4, label = "Test")
plt.title('Standard Normal Porosity vs. Depth')
plt.xlabel('Z (m)')
plt.ylabel('NPorosity')
plt.legend()
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)

plt.subplot(212)
plt.plot(X2_train['norm_Depth'].values,y2_train['norm_Nporosity'].values, 'o', markerfacecolor='red', markeredgecolor='black', alpha=0.4, label = "Train")
plt.plot(X2_test['norm_Depth'].values,y2_test['norm_Nporosity'].values, 'o', markerfacecolor='blue', markeredgecolor='black', alpha=0.4, label = "Test")
plt.title('Normalized Standard Normal Porosity vs. Normalized Depth')
plt.xlabel('Normalized Z')
plt.ylabel('Normalized NPorosity')
plt.legend()

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.5, wspace=0.2, hspace=0.3)
plt.show()
```

#### Specify the Prediction Locations

Given this training and testing data, let's specify the prediction locations over the range of the observed depths at regularly spaced $nbins$ locations.

```python
# Specify the prediction locations
nbins = 1000
depth_bins = np.linspace(depth_min, depth_max, nbins) # set the bins for prediction
norm_depth_bins = (depth_bins-depth_min)/(depth_max-depth_min)*2-1 # use normalized bins
```

#### Build and Train a Simple Neural Network

For our first model we will build a simple model with:

* 1 predictor feature - depth ($d$)

* 1 responce feature - normal score porosity ($N\{\phi\}$)

we will build a model to predict normal score porosity from depth over all locations in our model $\bf{u} \in AOI$.

\begin{equation}
N\{\phi(\bf{u})\} = \hat{f} (d(\bf{u}))
\end{equation}

and use this model to support the prediction of porosity between the wells.

#### Design, Train and Test a Neural Network

In the following code we use keras / tensorflow to:

1. **Design the Network** - we use a fully connected, feed forward neural network with one node in the input and output layers, to receive the normalized depth and output the normalized (normal score) porosity.  We found by trial and error, given the complexity of the dataset, that we required a significant network width (about 500 nodes) and a network depth of atleast 4 hidden layers.

2. **Select the Optimizer** - we selected the adam optimizer (Kingma and Ba, 2015).  This optimizer is computationally efficient and is well suited to problems with noisy data and sparse gradients. It is an extension of stochastic gradient descent with the addition of adaptive gradient algorithm that calculates per-parameter learning rates to improve learning with sparse gradients, and root mean square propopagation that sets the learning rate based on the recent magnitudes of the gradients for each parameter to improve performance with noisy data.  We include stochastic gradient descent for experimentation.

    * we found a learning rate of 0.01 to 0.001 works well
    * we found the rate of decay parameters of $\beta_1=0.9$ and $\beta_2=0.999$ performed well

3. **Compile the Machine** - specify the optimizer, loss function and the metric for model training.

4. **Train the Network** - fit / train the model parameters over a specified number of batch size within a specified number of epochs.  We specify the train and test normalized predictor and response features.

Then we visualize the model in the original units.

```python
# Design the neural network
model_2 = Sequential([
    Dense(1, activation='linear', input_shape=(1,)),                  # input layer
    Dense(3, activation='relu'),
#     Dense(50, activation='relu'),
#     Dense(50, activation='relu'),
#     Dense(20, activation='relu'),                                    # uncomment these to add hidden layers
#     Dense(20, activation='relu'),
#     Dense(20, activation='relu'),
#     Dense(20, activation='relu'),
    Dense(1, activation='linear'),                                     # output layer
])

# Select the Optimizer
adam = Adam(lr=0.003, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False) # adam optimizer
#sgd = keras.optimizers.SGD(lr=0.001, momentum=0.0, decay = 0.0, nesterov=False)  # stochastic gradient descent

# Compile the Machine
model_2.compile(optimizer=adam,loss='mse',metrics=['accuracy'])

# Train the Network
hist_2 = model_2.fit(X2_train['norm_Depth'], y2_train['norm_Nporosity'],
    batch_size=5, epochs=200,
    validation_data=(X2_test['norm_Depth'], y2_test['norm_Nporosity']),verbose = 0)

# Predict with the Network
pred_norm_Nporsity = model_2.predict(np.array(norm_depth_bins)) # predict with our ANN
pred_Nporosity = ((pred_norm_Nporsity + 1)/2*(Npor_max - Npor_min)+Npor_min)

# Plot the Model Predictions
plt.subplot(1,1,1)
plt.plot(depth_bins,pred_Nporosity,'black',linewidth=2)
plt.plot(X2_train['Depth'].values,y2_train['Nporosity'].values, 'o', markerfacecolor='red', markeredgecolor='black', alpha=1.0, label = "Train")
plt.plot(X2_test['Depth'].values,y2_test['Nporosity'].values, 'o', markerfacecolor='blue', markeredgecolor='black', alpha=1.0, label = "Test")
plt.xlabel('Depth (m)')
plt.ylabel('Porosity (faction)')
plt.legend()
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=0.8, wspace=0.2, hspace=0.2)
plt.show()
```

#### Evaluation of the Model

For my specified artificial neural network design and optimization parameters I have a very flexible model to fit the data.

* artificial neural networks live up to their designation as **Universal Function Approximators**

Let's check the training curve, loss functions for our model over training and testing datasets.

* **square loss** ($L_2$ loss) is the:

\begin{equation}
L_2 = \sum_{\bf{u}_{\alpha} \in AOI} \left(y(\bf{u}_{\alpha}) - \hat{f}(x_1(\bf{u}_{\alpha}),\ldots,x_m(\bf{u}_{\alpha})\right)
\end{equation}

* this is a measure of the inaccuracy over the available data

We can see the progress of the model over epochs in reduction of training and testing error.

* we can observe that the model matches the training data after about 200 epochs, but continues to improve up the 1,000 epochs

```python
# Plot the Loss vs. Training Epoch
plt.subplot(1,1,1)
plt.plot(hist_2.history['loss'])
plt.plot(hist_2.history['val_loss'])
plt.title('Loss function of Artificial Neural Net Example #2')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper right')
plt.ylim(0,0.4)
plt.grid()
plt.tight_layout()
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=0.8, wspace=0.2, hspace=0.2)
plt.show()
```

#### Some Observations

We performed a set of experiments and made the following observations that may help you experiment with the design and training of this artificial neural network. Each machine was trained over 1000 epochs with a batch size of 5.

For a 1 $\times$ 100 $\times$ 100 $\times$ 1 neural network:
* Learning Rate of 0.001 still converging at 1000 epochs, missing some features
* Learning Rate of 0.1 - stuck in a local minum as a step function

For a 1 $\times$ 500 $\times$ 500 $\times$ 1 neural network:
* Learning Rate of 0.01 - 0.001 for a close fit to training data
* Learning Rate of 0.1 - stuck in a local minimum as a line
* Learning Rate of $\le$ 0.0001 still converging at 1000 epochs, missing some features

For a 1 $\times$ 500 $\times$ 500 $\times$ 500 $\times$ 1 neural network:
* Learning Rate of 0.01 - 0.001 for a close fit to training data
* Learning Rate of 0.1 - stuck as a line
* Learning Rate of $\le$ 0.0001 still converging at 1000 epochs, missing some features

#### Visualizing the Neural Net

There are some methods available to interogate the artificial neural net.

* neural net summary

* weights

Here's the summary from our neural net.  It lists by layers the number of nodes and number of parameters.

```python
model_2.summary()                                      # artificial neural network design and number of parameters
```

    Model: "sequential_2"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_7 (Dense)              (None, 1)                 2
    _________________________________________________________________
    dense_8 (Dense)              (None, 3)                 6
    _________________________________________________________________
    dense_9 (Dense)              (None, 1)                 4
    =================================================================
    Total params: 12
    Trainable params: 12
    Non-trainable params: 0
    _________________________________________________________________

We can also see the actual trained weights for each node in each layer.

```python
for layer in model_2.layers:                            # weights for the trained artificial neural network
    g = layer.get_config()
    h = layer.get_weights()
    print(g)
    print(h)
    print('\n')
```

    {'name': 'dense_7', 'trainable': True, 'batch_input_shape': (None, 1), 'dtype': 'float32', 'units': 1, 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'class_name': 'GlorotUniform', 'config': {'seed': None}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}
    [array([[1.3643419]], dtype=float32), array([0.06368703], dtype=float32)]

    {'name': 'dense_8', 'trainable': True, 'dtype': 'float32', 'units': 3, 'activation': 'relu', 'use_bias': True, 'kernel_initializer': {'class_name': 'GlorotUniform', 'config': {'seed': None}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}
    [array([[-0.42711794, -0.64516276,  1.1085054 ]], dtype=float32), array([-0.183792  , -0.28067875, -0.7691435 ], dtype=float32)]

    {'name': 'dense_9', 'trainable': True, 'dtype': 'float32', 'units': 1, 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'class_name': 'GlorotUniform', 'config': {'seed': None}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}
    [array([[-1.2426817 ],
           [-0.5781366 ],
           [ 0.51588565]], dtype=float32), array([0.00843373], dtype=float32)]

We can observe the weights and bias terms associate with our neural network.

#### Comments

This was a basic demonstration of artificial neural networks.

The Texas Center for Geostatistics has many other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available [here](https://github.com/GeostatsGuy/PythonNumericalDemos), along with a package for geostatistics in Python called [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy).

We hope this was helpful,

*Michael* and *Honggeun*

***

#### More on Michael Pyrcz and the Texas Center for Data Analytics and Geostatistics:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```

```python

```

```python

```