# 2.3 高斯分布

高斯分布，又叫正态分布，是连续变量经常使用的一个分布模型，一维的高斯分布如下：

$$
\mathcal{N}\left(x\left|~\mu,\sigma^2\right.\right) = \frac{1}{(2\pi\sigma^2)^{1/2}} \exp\left\{-\frac{1}{2\sigma^2}(x-\mu)^2\right\}
$$

其中 $\mu$ 是均值，$\sigma$ 是方差。

$D$-维的高斯分布如下：

$$
\mathcal{N}\left(\mathbf x\left|~\mathbf{\mu, \Sigma}\right.\right) = \frac{1}{(2\pi)^{D/2}} \frac{1}{|\mathbf\Sigma|^{1/2}} \exp \left\{-\frac{1}{2}(\mathbf x - \mathbf \mu)^\top\mathbf\Sigma^{-1}(\mathbf x - \mathbf \mu)\right\}
$$

其中，$D$ 维向量 $\mathbf \mu$ 是均值，$D\times D$ 矩阵 $\mathbf\Sigma$ 是方差，$|\mathbf\Sigma|$ 是其行列式。

之前我们已经看到，在均值和方差固定时，高斯函数是熵最大的连续分布，因此高斯分布的应用十分广泛。

而中心极限定理告诉我们，对于某个分布一组样本 $x_1, \dots, x_N$，他们的均值 $(x_1+\dots+x_N)/N$ 的分布会随着 $N$ 的增大而越来越接近一个高斯分布。

### 验证高斯分布

下面，我们来验证高斯分布是一个概率分布。

考虑高斯分布的结构中跟 $\mathbf x$ 有关的这个二次型：

$$
\Delta^2 = (\mathbf x - \mathbf \mu)^\top\mathbf\Sigma^{-1}(\mathbf x - \mathbf \mu)
$$

这里的 $\Delta$ 叫做 $\mathbf \mu$ 和 $\mathbf x$ 的马氏距离（Mahalanobis distance），当 $\mathbf \Sigma$ 是单位矩阵的时候为欧氏距离。

在 $\Delta^2$ 相等的地方，高斯分布的概率密度也相等。

不失一般性，我们只需要考虑 $\mathbf\Sigma$ 是对称的情况。

事实上，如果 $\mathbf\Sigma$ 不对称，考虑其逆矩阵 $\mathbf\Lambda$，它总可以写成一个对称矩阵和反对称矩阵的形式：

$$\mathbf\Lambda = \mathbf\Lambda^S + \mathbf\Lambda^A =\frac{\mathbf{\Lambda + \Lambda^\top}}{2} + \frac{\mathbf{\Lambda - \Lambda^\top}}{2}$$

而这个反对称的矩阵的对角线均为 0，从而它的二次型为 $0$ 因此，我们有：

$$
(\mathbf x - \mathbf \mu)^\top\mathbf\Lambda(\mathbf x - \mathbf \mu) = (\mathbf x - \mathbf \mu)^\top\mathbf\Lambda^S(\mathbf x - \mathbf \mu)
$$

所以 $\mathbf\Lambda$ 可以等价于 $\mathbf\Lambda^S$，而对称矩阵的逆矩阵也是对称的，因此 只需要考虑 $\mathbf\Sigma$ 是对称的情况。

对于实对称矩阵，可以找到一组正交的特征向量使得：

$$
\mathbf{\Sigma u}_i = \lambda_i,\lambda_i\geq=0 \mathbf u_i, \mathbf u_i^\top\mathbf u_j = I_{ij}， i,j=1,\dots,D,
$$

其中 $I_{ij}$ 表示单位矩阵的第 $i$ 行 $j$ 列的元素：

$$
I_{ij} = \left\{\begin{align}
1, &~i=j \\ 0, &~i \neq j
\end{align}\right.
$$

从而 $\mathbf\Sigma$ 可以表示为

$$
\mathbf\Sigma = \sum_{i=1}^D \lambda_i \mathbf u_i\mathbf u_i^\top = \mathbf{U\Lambda U}^\top
$$

其中 $\Lambda =\text{diag}(\lambda_1, \dots,\lambda_D)$ 为对角阵，$\mathbf U$ 是 $u_1,\dots,u_D$ 按列拼成的正交矩阵 $\mathbf U^\top\mathbf U = \mathbf U\mathbf U^\top = \mathbf I$。

由正交性：

$$
\mathbf\Sigma^{-1} = (\mathbf{U\Lambda U}^\top)^{-1} = (\mathbf U^\top)^{-1}\mathbf\Lambda^{-1} \mathbf U^{-1} = \mathbf U \mathbf\Lambda^{-1} \mathbf U^\top = \sum_{i=1}^D \frac{1}{\lambda_i} \mathbf u_i\mathbf u_i^\top
$$

令 $y_i=\mathbf u_i^\top (\mathbf{x-\mu})$，则

$$
\Delta^2 = \sum_{i=1}^D \frac{y_i^2}{\lambda_i}
$$

写成向量形式：

$$
\mathbf y = \mathbf{U}^\top\mathbf{(x-\mu)}
$$

因此，马氏距离相当于将 $\mathbf{x}$ 做平移变换后，再在 $\mathbf u_i$ 方向上作了一个大小为 $\lambda_i^{1/2}$ 的尺度变换。

为了满足高斯分布定义的合法性，需要使得所有的 $\lambda_i$ 均大于 0，即 $\mathbf\Sigma$ 是正定的。

考虑新的坐标系下的高斯分布形式，由 $\mathbf U$ 的正交性，我们有

$$
\mathbf x = \mathbf{U} \mathbf{y} + \mu
$$

因此 $\mathbf x$ 对 $\mathbf y$ 的雅克比矩阵为：

$$
J_{ij} = \frac{\partial x_i}{\partial y_j} = U_{ij}
$$

因此其行列式的绝对值为 1：

$$
|\mathbf J|^2 = \left|\mathbf U\right|^2 = \left|\mathbf U\right|\left|\mathbf U^\top\right| = \left|\mathbf U\right|\left|\mathbf U^\top\right| = |\mathbf I| = 1
$$

另一方面，矩阵 $\mathbf\Sigma$ 的行列式可以成特征值的乘积：

$$
|\mathbf\Sigma| = \prod_{j=1}^D \lambda_j
$$

所以在新的坐标系下的分布为：

$$
p(\mathbf y) = p(\mathbf x)|\mathbf J|=\prod_{j=1}^D \frac{1}{\lambda_j^{1/2}}\exp\left(-\frac{y_j^2}{2\lambda_j}\right)
$$

即 $D$ 个独立的单变量高斯分布的乘积。

之前我们证明过单变量高斯分布的积分是 1，因此：

$$
\int p(\mathbf y)\mathbf y = \prod_{j=1}^D \int_{-\infty}^{\infty} \frac{1}{\lambda_j^{1/2}}\exp\left(-\frac{y_j^2}{2\lambda_j}\right) dy_j = 1
$$

所以多维高斯分布的确是归一化的，非负性显然，所以它是一个概率分布。

```python
import numpy as np
import scipy as sp
import matplotlib.pyplot as plt

%matplotlib inline

fig, ax = plt.subplots()

tt = np.linspace(-np.pi, np.pi, 100)

xx = 1 * np.sin(tt)
yy = 2 * np.cos(tt)

ax.plot(- xx * np.sin(np.pi/6) - yy * np.cos(np.pi/6),
        xx * np.cos(np.pi/6) - yy * np.sin(np.pi/6), 'r')

ax.set_xlim([-3, 3])
ax.set_ylim([-2, 2])
ax.set_xlabel(r"$x_1$", fontsize="xx-large")
ax.set_ylabel(r"$x_2$", fontsize="xx-large")

ax.set_xticks([])
ax.set_yticks([])

ax.text(0, 0.3, r"$\mathbf{\mu}$", fontsize="xx-large")

ax.annotate(r"$y_1$",
            fontsize="xx-large",
            xy=(-2.5, -2.5 / np.sqrt(3)), xycoords='data',
            xytext=(2.5, 2.5 / np.sqrt(3)), textcoords='data',
            arrowprops=dict(arrowstyle="<-",
                            connectionstyle="arc3",
                            color="blue"))
ax.annotate(r"$y_2$",
            fontsize="xx-large",
            xy=(1, -np.sqrt(3)), xycoords='data',
            xytext=(-1, np.sqrt(3)), textcoords='data',
            arrowprops=dict(arrowstyle="<-",
                            connectionstyle="arc3",
                            color="blue"))

ax.annotate("",
            xy=(0.55, -0.55 * np.sqrt(3)), xycoords='data',
            xytext=(0.55 + np.sqrt(3), 1-0.55 * np.sqrt(3)), textcoords='data',
            arrowprops=dict(arrowstyle="<->",
                            connectionstyle="arc3"),
            )
ax.annotate("",
            xy=(-1.05*np.sqrt(3), -1.05), xycoords='data',
            xytext=(-1.05*np.sqrt(3)-0.5, -1.05+0.5*np.sqrt(3)), textcoords='data',
            arrowprops=dict(arrowstyle="<->",
                            connectionstyle="arc3"),
            )

ax.text(-2.7, -1, r"$\lambda_2^{1/2}$", fontsize="xx-large")
ax.text(1.4, -0.9, r"$\lambda_1^{1/2}$", fontsize="xx-large")

plt.show()
```

### 均值和协方差

计算一阶矩即均值，令 $\mathbf{z=x-\mu}$：

$$
\begin{aligned}
\mathbb E[\mathbf x] & = \frac{1}{(2\pi)^{D/2}} \frac{1}{|\mathbf\Sigma|^{1/2}} \int \exp \left\{-\frac{1}{2}(\mathbf x - \mathbf \mu)^\top\mathbf\Sigma^{-1}(\mathbf x - \mathbf \mu)\right\} \mathbf x  d\mathbf x\\
& = \frac{1}{(2\pi)^{D/2}} \frac{1}{|\mathbf\Sigma|^{1/2}} \int \exp \left\{-\frac{1}{2}\mathbf z^\top\mathbf\Sigma^{-1}\mathbf z\right\} \mathbf{(z+\mu)}  d\mathbf z
\end{aligned}
$$

含有 $\mathbf{z + \mu}$ 中 $\mathbf z$ 的部分会消失，因为此时 $\mathbf z$ 的部分是一个奇函数，而 $\mathbf \mu$ 的部分相当于它乘上一个多维高斯分布的积分，因此：

$$
\mathbb E[\mathbf x] = \mu
$$

考虑二阶矩，令 $\mathbf{z=x-\mu}$：

$$
\begin{aligned}
\mathbb E[\mathbf x\mathbf x^\top] & = \frac{1}{(2\pi)^{D/2}} \frac{1}{|\mathbf\Sigma|^{1/2}} \int \exp \left\{-\frac{1}{2}(\mathbf x - \mathbf \mu)^\top\mathbf\Sigma^{-1}(\mathbf x - \mathbf \mu)\right\} \mathbf x  \mathbf x^\top d\mathbf x\\
& = \frac{1}{(2\pi)^{D/2}} \frac{1}{|\mathbf\Sigma|^{1/2}} \int \exp \left\{-\frac{1}{2}\mathbf z^\top\mathbf\Sigma^{-1}\mathbf z\right\} \mathbf{(z+\mu)} \mathbf{(z+\mu)}^\top d\mathbf z
\end{aligned}
$$

展开后面的乘积，$\mathbf{\mu z}^\top$ 和 $\mathbf{z\mu}^\top$ 项的积分会因为奇函数的性质消去，$\mathbf{\mu\mu}^\top$ 项是常数乘以分布，因此只剩下 $\mathbf{\mu\mu}^\top$。现在考虑 $\mathbf{zz}^\top$ 项：

从之前的推导中，我们知道：

$$
\mathbf{z=x-\mu=Uy}=\sum_{j=1}^D y_j \mathbf u_j
$$

$$
\mathbf z^\top\mathbf\Sigma^{-1}\mathbf z = \sum_{k=1}^D \frac{y_k^2}{\lambda_k}
$$

其中 $y_j = \mathbf u_j^\top \mathbf z$。

因此我们有：

\begin{aligned}
& \frac{1}{(2\pi)^{D/2}} \frac{1}{|\mathbf\Sigma|^{1/2}} \int \exp \left\{-\frac{1}{2}\mathbf z^\top\mathbf\Sigma^{-1}\mathbf z\right\} \mathbf{z} \mathbf{z}^\top d\mathbf z \\
= & \frac{1}{(2\pi)^{D/2}} \frac{1}{|\mathbf\Sigma|^{1/2}} \sum_{i=1}^D \sum_{j=1}^D \int  \exp \left\{-\sum_{k=1}^D \frac{y_k^2}{2\lambda_k}\right\} y_iy_j \mathbf u_i \mathbf u_j^\top d\mathbf y
\end{aligned}

当 $i \neq j$ 时，奇函数的性质使得积分为 0，所以只有 $i = j$ 的积分项才能保留，因此：

$$
\mathbb E[\mathbf{zz}^\top] = \sum_{i=1}^D \mathbf{u}_i \mathbf u_i^\top \lambda_i = \mathbf \Sigma
$$

$i=j$ 时，非 $i$ 项的 $y_k$ 的积分为 1，$i$ 项的积分相当于对一个均值为 $0$，方差为 $\lambda_i$ 的一维高斯函数求二阶矩，因此为 $0^2 + \lambda_i = \lambda_i$。

所以二阶矩为

$$
\mathbb E[\mathbf{xx}^\top] = \mathbf{\Sigma + \mu\mu}^\top
$$

从而协方差矩阵为

$$
{\rm cov}[{\bf x}] = \mathbb E[(\bf x-\mathbb E[\bf x])(\bf x-\mathbb E[\bf x])^\top] = \mathbb E[\mathbf{xx}^\top] - \mathbb E[\bf x]\mathbb E[\bf x]^\top = \bf \Sigma
$$

### 高斯分布的局限性

尽管高斯分布是一个广泛使用的模型，但是它仍然存在一定的局限性。

一个 $D$ 维的高斯分布的独立参数的数量级是 $O(D^2)$ 的（协方差 $D(D+1)/2$，均值 $D$），因此随着 $D$ 的增大，参数量的增长是平方量级的，此外，求逆操作的时间复杂度也随之增大。

一个解决这个问题的方法是限制协方差矩阵的形式，例如将其限制为对角形式 $\bf \Sigma={\rm diag}(\sigma_i^2)$，或者更进一步，限制为 $\bf \Sigma = \sigma^2 \bf I$，即各向同性（`isotropic`）的高斯分布。

但这样做限制了高斯分布的自由度。

一般高斯分布，对角线形式的高斯分布和各向同性的高斯分布的等高线如下图所示：

```python
from matplotlib.mlab import bivariate_normal

x, y = np.mgrid[-1:1:.01, -1:1:.01]

fig, ax = plt.subplots(1, 3, figsize=(10, 3))

# 斜的椭圆
z = bivariate_normal(x, y, sigmax=2, sigmay=1, sigmaxy=1.5)
ax[0].contour(x, y, z, colors="r",
              levels=np.linspace(0.1*z.min() + 0.9*z.max(), z.max(), 5))

ax[0].set_xticks([])
ax[0].set_yticks([])

# 坐标轴平行的椭圆
z = bivariate_normal(x, y, sigmax=2, sigmay=1, sigmaxy=0)
ax[1].contour(x, y, z, colors="r",
              levels=np.linspace(0.2*z.min() + 0.8*z.max(), z.max(), 5))

ax[1].set_xticks([])
ax[1].set_yticks([])

# 圆
z = bivariate_normal(x, y, sigmax=2, sigmay=2, sigmaxy=0)
ax[2].contour(x, y, z, colors="r",
              levels=np.linspace(0.25*z.min() + 0.75*z.max(), z.max(), 5))

ax[2].set_xticks([])
ax[2].set_yticks([])

plt.show()
```

高斯分布的另一个问题在于他只有一个峰值（`unimodal`），所以不能很好的表示多模态（`multimodal`）的分布。之后我们可以看到，这个问题可以通过引入隐变量来解决。

## 2.3.1 条件高斯分布

高斯分布的一个重要性质是如果两组变量的联合分布是高斯分布，那么其中一组相对于另一组的条件分布也是高斯分布，每组变量的边缘分布也是高斯的。

### 基本设定

考虑 $D$ 维的高斯分布 $p(\mathbf x)= \cal N(\bf x~|~\bf{\mu,\Sigma})$，将其分成两组变量 ${\bf x}_a, {\bf x}_b$。不失一般性，我们假设 ${\bf x}_a$ 是 $\bf x$ 的前 $M$ 个分量，${\bf x}_b$ 是 $\bf x$ 的后 $D-M$ 个分量，即

$$
{\bf x} = \begin{pmatrix}
{\bf x}_a \\
{\bf x}_b
\end{pmatrix}
$$

对应的均值和协方差矩阵可以写为：

$$
\begin{align}
{\bf \mu} & = \begin{pmatrix}
{\bf \mu}_a \\
{\bf \mu}_b
\end{pmatrix} \\
{\bf \Sigma} & = \begin{pmatrix}
{\bf \Sigma}_{aa} & {\bf \Sigma}_{ab} \\
{\bf \Sigma}_{ba} & {\bf \Sigma}_{bb}
\end{pmatrix}
\end{align}
$$

由 ${\bf\Sigma = \Sigma}^{\text T}$，我们可以知道 ${\bf \Sigma}_{aa}, {\bf \Sigma}_{bb}$ 是对称的，且 ${\bf \Sigma}_{ab}={\bf \Sigma}_{ba}^\top$。

为了方便，我们使用协方差的逆矩阵，或者叫精确度矩阵（`precision matrix`）：

$$
{\bf\Lambda \equiv \Sigma}^{-1}
$$

它可以写成：

$$
{\bf \Lambda} = \begin{pmatrix}
{\bf \Lambda}_{aa} & {\bf \Lambda}_{ab} \\
{\bf \Lambda}_{ba} & {\bf \Lambda}_{bb}
\end{pmatrix}
$$

由对称性，我们有 ${\bf \Lambda}_{aa}, {\bf \Lambda}_{bb}$ 是对称的，且 ${\bf \Lambda}_{ab}={\Lambda}_{ba}^\top$。

### 使用精确度矩阵表示

我们不从条件分布的定义出发计算，换一个角度，考虑二次型的部分：

$$
\begin{align}
-\frac{1}{2}(\mathbf x - \mathbf \mu)^\top\mathbf\Sigma^{-1}(\mathbf x - \mathbf \mu) =
&
-\frac{1}{2}(\mathbf x_a - \mathbf \mu_a)^\top\mathbf\Lambda_{aa}(\mathbf x_a - \mathbf \mu_a) -
\frac{1}{2}(\mathbf x_a - \mathbf \mu_a)^\top\mathbf\Lambda_{ab}(\mathbf x_b - \mathbf \mu_b)
\\
& - \frac{1}{2}(\mathbf x_b - \mathbf \mu_b)^\top\mathbf\Lambda_{ba}(\mathbf x_a - \mathbf \mu_a) -
\frac{1}{2}(\mathbf x_b - \mathbf \mu_b)^\top\mathbf\Lambda_{bb}(\mathbf x_b - \mathbf \mu_b)
\end{align}
$$

我们看到，对于 $\mathbf x_a$ 来说，它仍然是一个二次型，因此，条件分布 $p(\mathbf x_a|\mathbf x_b)$ 应该是一个高斯分布。

因为高斯分布由其均值和协方差完全确定，我们的目的变为计算条件分布的均值和方差。

为了计算均值和方法，我们可以使用配方法，对于一个高斯分布来说，我们有：

$$
-\frac{1}{2}(\mathbf x - \mathbf \mu)^\top\mathbf\Sigma^{-1}(\mathbf x - \mathbf \mu) = -\frac{1}{2}\mathbf x^\top \mathbf\Sigma^{-1} \mathbf x + \mathbf x^\top \mathbf\Sigma^{-1} \mathbf \mu + {\rm const}
$$

其中常量表示与 $\mathbf x$ 无关的部分，另外用到了 $\bf \Sigma$ 的对称性。

这告诉我们，$\mathbf\Sigma^{-1}$ 是 $\mathbf x$ 二次项的中间部分，$\mathbf x$ 一次项的中间部分是 $\mathbf\Sigma^{-1} \mathbf \mu$。

回过头来，我们观察到二次型部分中，$\mathbf x_a$ 的二次项为：

$$
-\frac{1}{2}\mathbf x_a^\top \mathbf\Lambda_{aa} \mathbf x_a
$$

我们立刻可以得到条件分布 $p(\mathbf x_a|\mathbf x_b)$ 的协方差矩阵为：
$$
\mathbf \Sigma_{a|b} = \mathbf\Lambda_{aa}^{-1}
$$

在考虑含有 $\mathbf x_a$ 的线性部分（利用 $\mathbf\Lambda_{ab} = \mathbf\Lambda_{ba}^\top$）：

$$
\mathbf x_a^\top\{\mathbf \Lambda_{aa}\mathbf\mu_a - \mathbf \Lambda_{ab}(\mathbf x_b - \mathbf \mu_b)\}
$$

我们可以得到：

$$
\mathbf \Sigma_{a|b}^{-1}\mathbf \mu_{a|b} = \mathbf \Lambda_{aa}\mathbf\mu_a - \mathbf \Lambda_{ab}(\mathbf x_b - \mathbf \mu_b)
$$

从而：

$$
\begin{align}
\mathbf \mu_{a|b} & = \mathbf \Sigma_{a|b}\{\mathbf \Lambda_{aa}\mathbf\mu_a - \mathbf \Lambda_{ab}(\mathbf x_b - \mathbf \mu_b)\} \\
& = \mathbf \mu_a - \mathbf\Lambda_{aa}^{-1}\mathbf \Lambda_{ab}(\mathbf x_b - \mathbf \mu_b)
\end{align}
$$

### 使用协方差矩阵表示

我们也可以使用协方差矩阵的分量来表示条件分布的均值和方差，首先我们使用分块矩阵的 Schur 补（`Schur complement`）来表示分块矩阵的逆：

$$
\begin{pmatrix}
\bf A & \bf B \\
\bf C & \bf D \\
\end{pmatrix}^{-1}
= \begin{pmatrix}
\bf M & {\bf -MBD}^{-1} \\
-{\bf D}^{-1}{\bf CM} & {\bf D}^{-1}+{\bf CMBD}^{-1} \\
\end{pmatrix}
$$

其中 $\mathbf M =(\mathbf{A - BD}^{-1}\mathbf C)^{-1}$，是分块矩阵的 Schur 补。

结合定义式：

$$
\begin{pmatrix}
{\bf \Sigma}_{aa} & {\bf \Sigma}_{ab} \\
{\bf \Sigma}_{ba} & {\bf \Sigma}_{bb}
\end{pmatrix}^{-1} = \begin{pmatrix}
{\bf \Lambda}_{aa} & {\bf \Lambda}_{ab} \\
{\bf \Lambda}_{ba} & {\bf \Lambda}_{bb}
\end{pmatrix}
$$

我们有：

$$
\begin{align}
{\bf \Lambda}_{aa} & = (\mathbf{\Sigma}_{aa} - \mathbf\Sigma_{ab}\mathbf\Sigma_{bb}^{-1}\mathbf \Sigma_{ba})^{-1} \\
{\bf \Lambda}_{ab} & = -(\mathbf{\Sigma}_{aa} - \mathbf\Sigma_{ab}\mathbf\Sigma_{bb}^{-1}\mathbf \Sigma_{ba})^{-1}\mathbf\Sigma_{ab}\mathbf\Sigma_{bb}^{-1}
\end{align}
$$

带入均值和方差的表达式，我们有：

$$
\begin{align}
\mathbf \mu_{a|b} & = \mathbf \mu_a - \mathbf\Lambda_{aa}^{-1}\mathbf \Lambda_{ab}(\mathbf x_b - \mathbf \mu_b) \\
& = \mu_a + \mathbf\Sigma_{ab}\mathbf\Sigma_{bb}^{-1} (\mathbf x_b-\mathbf \mu_b) \\
\mathbf \Sigma_{a|b} & = \mathbf\Lambda_{aa}^{-1} \\
& = \mathbf{\Sigma}_{aa} - \mathbf\Sigma_{ab}\mathbf\Sigma_{bb}^{-1}\mathbf \Sigma_{ba} \\
\end{align}
$$

可以看到，使用精确度矩阵表示形式上更简单一些。

注意到，条件分布 $p(\mathbf x_a|\mathbf x_b)$ 的均值与 $\mathbf x_b$ 呈线性关系，而协方差与 $\mathbf x_a$ 独立，这表示了一个线性高斯模型。

## 2.3.2 边缘高斯分布

现在考虑边缘分布的形式：

$$
p(\mathbf x_a)=\int p(\mathbf x_a, \mathbf x_b) d\mathbf x_b
$$

它也是一个高斯分布。

因为是对 $\mathbf x_b$ 的积分，我们考虑二次型中关于 $\mathbf x_b$ 的部分：

$$
-\frac{1}{2}\mathbf x_b^{\text T}\mathbf\Lambda_{bb}\mathbf x_b + \mathbf x_b^\top\mathbf m =
-\frac{1}{2} (\mathbf x_b-\mathbf\Lambda^{-1}_{bb}\mathbf m)^\top (\mathbf x_b-\mathbf\Lambda^{-1}_{bb}\mathbf m) +
\frac{1}{2} \mathbf m^\top\mathbf \Lambda_{bb}^{-1}\mathbf m
$$

其中

$$
\mathbf m = \mathbf\Lambda_{bb}\mathbf\mu_{b}-\mathbf\Lambda_{ba}(\mathbf x_a - \mathbf\mu_a)
$$

是一个与 $\mathbf x_b$ 无关的项。

配方之后我们发现，关于 $\mathbf x_b$ 的二次型部分可以拆分为两部分：一个高斯分布的二次型再加上一个 跟 $\mathbf x_b$ 无关的部分。

对于高斯分布的二次型部分，积分

$$
\int \exp\left\{-\frac{1}{2} (\mathbf x_b-\mathbf\Lambda^{-1}_{bb}\mathbf m)^\top (\mathbf x_b-\mathbf\Lambda^{-1}_{bb}\mathbf m)\right\} d\mathbf x_b
$$

是一个与 $\mathbf x_a, \mathbf x_b$ 无关的常数项。

考虑第二部分，并加上与 $\mathbf x_a$ 有关的项，我们有：

$$
\begin{align}
& \frac{1}{2} \mathbf m^\top\mathbf \Lambda_{bb}^{-1}\mathbf m
-\frac{1}{2}\mathbf x_a^\top \mathbf\Lambda_{aa} \mathbf x_a + \mathbf x_a^\top\{\mathbf \Lambda_{aa}\mathbf\mu_a + \mathbf \Lambda_{ab}\mathbf \mu_b\} + \text{const} \\
= & -\frac{1}{2} \mathbf x_a^\top (\mathbf\Lambda_{aa} -  \mathbf\Lambda_{ba}\mathbf\Lambda_{bb}^{-1}\mathbf\Lambda_{ba})\mathbf x_a + \mathbf x_a^\top (\mathbf\Lambda_{aa} -  \mathbf\Lambda_{ba}\mathbf\Lambda_{bb}^{-1}\mathbf\Lambda_{ba}) \mathbf \mu_a + \text{const}
\end{align}
$$

从之前的讨论立即可以得到协方差矩阵：

$$
\mathbf\Sigma_a = (\mathbf\Lambda_{aa} -  \mathbf\Lambda_{ba}\mathbf\Lambda_{bb}^{-1}\mathbf\Lambda_{ba})^{-1}
$$

以及均值

$$
\mathbf\Sigma_a (\mathbf\Lambda_{aa} -  \mathbf\Lambda_{ba}\mathbf\Lambda_{bb}^{-1}\mathbf\Lambda_{ba}) \mathbf \mu_a = \mathbf\mu_a
$$

再来利用之前 `Schur` 补的结论：

$$
\begin{pmatrix}
\bf A & \bf B \\
\bf C & \bf D \\
\end{pmatrix}^{-1}
= \begin{pmatrix}
\bf M & {\bf -MBD}^{-1} \\
-{\bf D}^{-1}{\bf CM} & {\bf D}^{-1}+{\bf CMBD}^{-1} \\
\end{pmatrix}
$$

其中 $\mathbf M =(\mathbf{A - BD}^{-1}\mathbf C)^{-1}$，是分块矩阵的 Schur 补。

$$
\begin{pmatrix}
{\bf \Lambda}_{aa} & {\bf \Lambda}_{ab} \\
{\bf \Lambda}_{ba} & {\bf \Lambda}_{bb}
\end{pmatrix}^{-1} =
\begin{pmatrix}
{\bf \Sigma}_{aa} & {\bf \Sigma}_{ab} \\
{\bf \Sigma}_{ba} & {\bf \Sigma}_{bb}
\end{pmatrix}
$$

我们用协方差矩阵来表示这个结果，可以得到：

$$
\mathbf \Sigma_{aa} = (\mathbf\Lambda_{aa} -  \mathbf\Lambda_{ba}\mathbf\Lambda_{bb}^{-1}\mathbf\Lambda_{ba})^{-1}
$$

从而：

$$
\begin{align}
\mathbb E[\mathbf x_a]& = \mathbf\mu_a\\
{\rm cov}[\mathbf x_a]&=\mathbf \Sigma_{aa}
\end{align}
$$

可以看到，边缘分布的均值和协方差就是原来的均值和协方差中相对应的部分。

### 结论

已知 $p(\mathbf x)= \cal N(\bf x~|~\bf{\mu,\Sigma})$ 以及 $\mathbf\Lambda\equiv\mathbf\Sigma^{-1}$ 以及

$$
\begin{align}
{\bf x} = \begin{pmatrix}
{\bf x}_a \\
{\bf x}_b
\end{pmatrix},~& {\bf \mu} = \begin{pmatrix}
{\bf \mu}_a \\
{\bf \mu}_b
\end{pmatrix} \\
{\bf \Sigma} = \begin{pmatrix}
{\bf \Sigma}_{aa} & {\bf \Sigma}_{ab} \\
{\bf \Sigma}_{ba} & {\bf \Sigma}_{bb}
\end{pmatrix},~& {\bf \Lambda} = \begin{pmatrix}
{\bf \Lambda}_{aa} & {\bf \Lambda}_{ab} \\
{\bf \Lambda}_{ba} & {\bf \Lambda}_{bb}
\end{pmatrix}
\end{align}
$$

条件分布为：

$$
\begin{align}
p(\mathbf x_a|\mathbf x_b) & = \mathcal N (\mathbf x|\mathbf \mu_{a|b}, \mathbf \Lambda_{aa}^{-1}) \\
\mathbf \mu_{a|b} & = \mathbf \mu_a - \mathbf \Lambda_{aa}^{-1}\mathbf \Lambda_{ab} (\mathbf x_b-\mathbf \mu_b)
\end{align}
$$

边缘分布为：

$$
p(\mathbf x_a)=\mathcal N (\mathbf x_a|\mathbf \mu_{a}, \mathbf \Sigma_{aa})
$$

```python
from scipy.stats import norm
from matplotlib.mlab import bivariate_normal

x, y = np.mgrid[0:1:.01, 0:1:.01]

fig, ax = plt.subplots(1, 2, figsize=(10, 4.5), dpi=200)

mu_a = 0.5
mu_b = 0.5
sigma_aa = 0.15
sigma_bb = 0.15
sigma_ab = 0.0215

# 联合分布

z = bivariate_normal(x, y,
                     mux=mu_a,
                     muy=mu_b,
                     sigmax=sigma_aa,
                     sigmay=sigma_bb,
                     sigmaxy=sigma_ab)

ax[0].contour(x, y, z, colors="g",
              levels = [0.99 * z.min() + 0.01 * z.max(),
                        0.9 * z.min() + 0.1 * z.max(),
                        0.5 * z.min() + 0.5 * z.max()])

ax[0].plot([0, 1], [0.7, 0.7], 'r')

ax[0].set_xlim(0, 1)
ax[0].set_ylim(0, 1)

ax[0].set_xticks([0, 0.5, 1])
ax[0].set_yticks([0, 0.5, 1])

ax[0].set_xlabel(r"$x_a$", fontsize="xx-large")
ax[0].set_ylabel(r"$x_b$", fontsize="xx-large")

ax[0].text(0.2, 0.8, r"$x_b=0.7$", fontsize="xx-large")
ax[0].text(0.6, 0.3, r"$p(x_a, x_b)$", fontsize="xx-large")

# 边缘分布

xx = np.linspace(0, 1, 100)

pa = norm.pdf(xx, loc=mu_a, scale = sigma_aa)

ax[1].plot(xx, pa, 'b')

pa_b = norm.pdf(xx,
                loc = mu_a + sigma_ab / sigma_bb ** 2 * (0.7 - mu_a),
                scale = np.sqrt(sigma_aa ** 2 - sigma_ab ** 2 / sigma_bb ** 2))

ax[1].set_xlabel(r"$x_a$", fontsize="xx-large")

ax[1].set_xlim(0, 1)
ax[1].set_ylim(0, 10)
ax[1].set_xticks([0, 0.5, 1])
ax[1].set_yticks([0, 5, 10])

ax[1].text(0.2, 7, r"$p(x_a|x_b=0.7)$", fontsize="xx-large")
ax[1].text(0.2, 3, r"$p(x_a)$", fontsize="xx-large")

ax[1].plot(xx, pa_b, 'r')

plt.show()
```

## 2.3.3 高斯变量的贝叶斯理论

之前提到 $p(\mathbf x_a|\mathbf x_b)$ 的均值是 $\mathbf x_b$ 的线性函数。

现在我们假设 $p(\mathbf x)$ 是一个已知的高斯分布，而 $p(\bf y|x)$ 是一个条件高斯分布，且其均值是 $\bf x$ 的线性函数，方差与 $\bf x$ 独立。这就是所谓的高斯线性模型（`linear Gaussian model`）：

$$
\begin{align}
p(\mathbf x)&=\mathcal N(\mathbf x~|~\mathbf \mu, \mathbf \Lambda^{-1}) \\
p(\mathbf y|\mathbf x)&=\mathcal N(\mathbf y~|~\mathbf{Ax+b}, \mathbf L^{-1}) \\
\end{align}
$$

其中 $\mathbf \mu, \mathbf A, \mathbf b$ 是均值的参数，$\mathbf \Lambda, \mathbf L$ 是精确度矩阵。$\mathbf{x,y}$ 的维度分别为 $M, D$，则 $\mathbf A$ 的维度为 $D\times M$。

我们的目的是求 $p(\mathbf y)$ 的分布。

考虑它们的联合分布，令 $\mathbf z=\begin{pmatrix}\mathbf x\\ \mathbf y\end{pmatrix}$，则：

$$
\begin{align}
\ln p(\mathbf z)&=\ln p(\mathbf x) + \ln p(\mathbf{y|x}) \\
&=-\frac{1}{2}(\mathbf{x-\mu})^\top\mathbf\Lambda(\mathbf{x-\mu})
-\frac{1}{2}(\mathbf{y-Ax-b})^\top\mathbf L(\mathbf{y-Ax-b}) + \text{const}
\end{align}
$$

为了得到 $\mathbf z$ 的精确度矩阵，我们考虑二次项：

$$
\begin{align}
& -\frac{1}{2}\mathbf x^\top (\mathbf\Lambda + \mathbf A^{\text T}\mathbf L\mathbf A) \mathbf x -
\frac{1}{2}\mathbf y^\top \mathbf L\mathbf y +
\frac{1}{2}\mathbf y^\top \mathbf L\mathbf A\mathbf+
\frac{1}{2}\mathbf x^\top \mathbf A^\top\mathbf L\mathbf y \\
= & -\frac{1}{2}
\begin{pmatrix}
\mathbf x\\
\mathbf y
\end{pmatrix}^\top
\begin{pmatrix}
\mathbf\Lambda + \mathbf A^\top\mathbf L\mathbf A & -\mathbf A^\top\mathbf L \\
-\mathbf L\mathbf A & \mathbf L
\end{pmatrix}
\begin{pmatrix}
\mathbf x\\
\mathbf y
\end{pmatrix} \\
= & -\frac{1}{2} \mathbf {z^\top Rz}
\end{align}
$$

$\mathbf z$ 的精确度矩阵为：

$$
\mathbf{R} =
\begin{pmatrix}
\mathbf\Lambda + \mathbf A^\top \mathbf L\mathbf A & -\mathbf A^\top \mathbf L \\
-\mathbf L\mathbf A & \mathbf L
\end{pmatrix}
$$

协方差矩阵为（使用 `Schur` 补的结论）：

$$
\text{cov}[\mathbf z]=\mathbf{R}^{-1} = \begin{pmatrix}
\mathbf\Lambda + \mathbf A^\top\mathbf L\mathbf A & -\mathbf A^\top\mathbf L \\
-\mathbf L\mathbf A & \mathbf L
\end{pmatrix}^{-1}
= \begin{pmatrix}
\mathbf\Lambda^{-1} & \mathbf\Lambda^{-1}\mathbf A^\top\\
\mathbf A\mathbf\Lambda^{-1} & \mathbf L^{-1}+\mathbf A\mathbf \Lambda^{-1}\mathbf A^\top
\end{pmatrix}
$$

考虑线性项：

$$
\mathbf x^\text{T}\mathbf{\Lambda\mu}-\mathbf x^\top\mathbf A^\top\mathbf{Lb} + \mathbf y^\top\mathbf{Lb} =
\begin{pmatrix}
\mathbf x\\
\mathbf y
\end{pmatrix}^\top
\begin{pmatrix}
\mathbf{\Lambda\mu-A}^\top\mathbf{Lb}\\
\mathbf{Lb}
\end{pmatrix}
$$

我们有

$$
\mathbb E[\mathbf z]=
\mathbf R^{-1}
\begin{pmatrix}
\mathbf{\Lambda\mu-A^\top Lb}\\
\mathbf{Lb}
\end{pmatrix}
= \begin{pmatrix}
\mathbf{\mu}\\
\mathbf{A\mu+b}
\end{pmatrix}
$$

然后我们由边缘高斯分布的结论马上可以得到：

$$
\begin{align}
\mathbb E[\mathbf y] &= \mathbf{A\mu+b} \\
{\rm cov} [\mathbf y] &= \mathbf L^{-1}+\mathbf A\mathbf \Lambda^{-1}\mathbf A^\top
\end{align}
$$

再来，我们由条件高斯分布的结论得到：

$$
\begin{align}
\mathbb E[\mathbf{x|y}] &= (\mathbf\Lambda + \mathbf A^\top \mathbf L\mathbf A)^{-1}\left\{ \mathbf A^\top \mathbf{L(y-b)} + \mathbf{\Lambda\mu} \right\} \\
{\rm cov} [\mathbf{x|y}] &= (\mathbf\Lambda + \mathbf A^\top\mathbf L\mathbf A)^{-1}
\end{align}
$$

### 结论

已知：

$$
\begin{align}
p(\mathbf x)&=\mathcal N(\mathbf x~|~\mathbf \mu, \mathbf \Lambda^{-1}) \\
p(\mathbf y|\mathbf x)&=\mathcal N(\mathbf y~|~\mathbf{Ax+b}, \mathbf L^{-1}) \\
\end{align}
$$

我们有

$$
\begin{align}
p(\mathbf y)&=\mathcal N(\mathbf y~|~\mathbf{A\mu+b}, \mathbf L^{-1}+\mathbf A\mathbf \Lambda^{-1}\mathbf A^\top) \\
p(\mathbf x|\mathbf y)&=\mathcal N(\mathbf y~|~\mathbf \Sigma\left\{ \mathbf A^\top \mathbf{L(y-b)} + \mathbf{\Lambda\mu} \right\}, \mathbf \Sigma) \\
\end{align}
$$

其中 $\mathbf \Sigma = (\mathbf\Lambda + \mathbf A^\top\mathbf L\mathbf A)^{-1}$。

## 2.3.4 高斯分布最大似然

对于一组独立同分布的高斯观测数据 $\mathbf X=(x_1, \dots,x_N)^\text{T}$，
高斯分布的似然函数为：

$$
\ln p({\bf X|\mu,\Sigma}) = -\frac{ND}{2}\ln (2\pi)-\frac{N}{2}\ln|\mathbf\Sigma|-\frac{1}{2}\sum_{n=1}^N ({\bf x_n-\mu})^\top\mathbf\Sigma^{-1}({\bf x_n-\mu})
$$

其充分统计量为

$$
\sum_{n=1}^N \mathbf x_n, \sum_{n=1}^N \mathbf x_n\mathbf x_n^\top
$$

考虑对 $\mathbf\mu$ 求偏导：

$$
\frac{\partial}{\partial \mathbf \mu} \ln p({\bf X|\mu,\Sigma}) = \sum_{n=1}^N \mathbf\Sigma^{-1} (\bf x_n-\mu)
$$

令其为零（严格来说要验证这是一个最大值点）：

$$
\mathbf \mu_{ML} = \frac{1}{N} \sum_{n=1}^N \mathbf x_n
$$

所以最大似然的均值就是样本均值。

协方差的最大似然是（可以求偏导，对矩阵求偏导的方法参见附录 C）

$$
{\bf\Sigma}_{ML} = \frac{1}{N} \sum_{n=1}^N (\mathbf x_n-\mathbf \mu_{ML})(\mathbf x_n-\mathbf \mu_{ML})^\top
$$

另一方面，在真实的分布下，最大似然估计的均值为

$$
\begin{align}
\mathbb E [\mu_{ML}] & = \mathbf \mu \\
\mathbb E [\Sigma_{ML}] & = \frac{N-1}{N} \bf \Sigma
\end{align}
$$

所以最大似然估计的协方差不是无偏估计，为此，可以使用

$$
\mathbf{\tilde\Sigma}=\frac{1}{N-1} \sum_{n=1}^N (\mathbf x_n-\mathbf \mu_{ML})(\mathbf x_n-\mathbf \mu_{ML})^\top
$$

来纠正。

## 2.3.5 序列估计

序列化方法要求我们每次只处理一个数据，然后扔掉这个数据。
这种方法在对于处理数据量很大（大到不能一次处理所有数据）的问题很重要。

我们考虑从 $N-1$ 个观测数据到 $N$ 个观测数据时，均值的最大似然估计的变化：

$$
\begin{align}
\mu_{ML}^{(N)} & = \frac{1}{N} \sum_{n=1}^N \mathbf x_n \\
& = \frac{1}{N} \mathbf x_N + \frac{1}{N} \sum_{n=1}^N-1 \mathbf x_n\\
& = \frac{1}{N} \mathbf x_N + \frac{N-1}{N} \mu_{ML}^{(N-1)} \\
& = \mu_{ML}^{(N-1)} + \frac{1}{N}(\mathbf x_N - \mu_{ML}^{(N-1)})
\end{align}
$$

这相当于在原来的均值的基础上，增加了一个误差项 $\mathbf x_N - \mu_{ML}^{(N-1)}$ 的变化；同时随着 $N$ 的增大，误差项调整的权重逐渐变小。

### Robbins-Monro 算法

在上面的序列估计算法中，我们得到的是最大似然解的精确结果，而在很多情况下，我们并不一定能得到精确的结果。为此，我们引入 `Robbins-Monro` 算法。

考虑联合分布 $p(z,\theta)$，$z$ 关于 $\theta$ 的条件分布的均值是一个关于 $\theta$ 的函数：

$$
f(\theta)\equiv \mathbb E[z|\theta]=\int zp(z|\theta)dz
$$

我们的目的是找到一个 $\theta^\star$ 使得 $f(\theta^\star)=0$。

对于这个问题，如果我们有很多 $z$ 和 $\theta$ 的数据，我们可以建模来解决这个问题。

不过，现在假设我们每次只观察一个 $z$ 值，然后根据这个值来更新我们对 $\theta^\star$ 的估计，为此，我们先假定 $z$ 的条件方差是有限的：

$$
\mathbb E[(z-f)^2|\theta]<\infty
$$

并且不失一般性的假设 $f(\theta)$ 在 $\theta^\star$ 附近是单调递增的。`Robbins-Monro` 过程给出：

$$
\theta^{(N)} = \theta^{(N-1)} + a_{N-1} z(\theta^{(N-1)})
$$

其中 $z(\theta^{(N)})$ 是 $z$ 在 $\theta$ 取 $\theta^{(N)}$ 时的观测值，参数 $a_N$ 满足：

$$
\begin{align}
\lim_{N\to\infty} a_N & = 0 \\
\sum_{N=1}^\infty a_N & = \infty \\
\sum_{N=1}^\infty a_N^2 & < \infty
\end{align}
$$

第一个条件保证了收敛性；第二个条件保证了不会收敛到最优值附近的点；第三个条件保证误差项最终不会破坏收敛性。

对于最大似然问题，我们要求

$$
\frac{\partial}{\partial \theta} \left\{\left.\frac{1}{N} \sum_{n=1}^N \ln p({x_n|\theta}) \right\}\right|_{\theta_{ML}} = 0
$$

交换求和和偏导，并求极限有：

$$
\lim_{N\to\infty}\frac{1}{N} \sum_{n=1}^N \frac{\partial}{\partial \theta}\ln p({x_n|\theta}) = \mathbb E_x \left[ \frac{\partial}{\partial \theta}\ln p({x|\theta})\right]
$$

我们可以看到，最大化极大似然对应与寻找一个函数的根。`Robbins-Monro` 过程给出：

$$
\theta^{(N)} = \theta^{(N-1)} + a_{N-1} \frac{\partial}{\partial \theta^{(N-1)}}\ln p({x_N|\theta^{(N-1)}})
$$

如果 $\theta$ 是均值，我们有

$$
z = \frac{\partial}{\partial \mu_{ML}}\ln p({x|\mu_{ML}, \sigma^2}) = \frac{1}{\sigma^2} (x-\mu_{ML})
$$

所以 $z$ 的分布是一个均值为 $\mu-\mu_{ML}$ 的高斯分布，我们相应的 $a_N$ 为 $\frac{\sigma^2}{N}$。

## 2.3.6 高斯分布的贝叶斯估计

极大似然估计给出了均值和协方差矩阵的点估计，现在我们通过引入先验分布来考虑它的贝叶斯估计。

### 均值未知，方差已知

考虑一维的高斯变量 $x$，并假设方差 $\sigma^2$ 已知，有 $N$ 个数据点 $\mathbf X=\{x_1,\dots,x_N\}$，其似然函数为

$$
p(\mathbf X|\mu)=\prod_{n=1}^N p(x_n|\mu) =\frac{1}{(2\pi\sigma^2)^\frac{N}{2}}
\exp\left\{-\frac{1}{2\sigma^2} \sum_{n=1}^N (x_n-\mu)^2\right\}
$$

注意似然函数不是一个关于 $\mu$ 的概率分布。

考虑共轭性，我们引入一个高斯先验 $p(\mu)$：

$$
p(\mu)=\mathcal N(\mu|\mu_0, \sigma_0^2)
$$

因此后验分布为：

$$
p(\mu|\mathbf X) \propto p(\mathbf X|\mu)p(\mu)
$$

用上文类似的配方法可以得到：

$$
p(\mu|\mathbf X) = \mathcal N(\mu|\mu_N,\sigma_N^2)
$$

其中

$$
\begin{align}
\mu_N & = \frac{\sigma^2}{N\sigma_0^2 + \sigma^2} \mu_0 + \frac{N\sigma_0^2}{N\sigma_0^2 + \sigma^2} \mu_{ML}\\
\frac{1}{\sigma_N^2} & = \frac{1}{\sigma_0^2} + \frac{N}{\sigma^2}
\end{align}
$$

其中

$$
\mu_{ML} = \frac{1}{N} \sum_{n=1}^N x_n
$$

当 $N = 0$ 时，后验分布就是先验分布，当 $N\to\infty$ 时，后验分布的参数趋于最大似然的结果。

### 均值已知，方差未知

我们使用精确度 $\lambda\equiv 1~/~\sigma^2$ 来进行计算，其似然函数为：

$$
p(\mathbf X|\lambda)=\prod_{n=1}^N \mathcal N(x_n|\mu,\lambda^{-1}) \propto \lambda^{N/2} \exp\left\{-\frac{\lambda}{2} \sum_{n=1}^N(x_n-\mu)^2\right\}
$$

考虑关于 $\lambda$ 的共轭性，我们引入的先验分布为 `Gamma` 分布：

$$
\mathrm{Gam}(\lambda~|~a, b) = \frac{1}{\Gamma(a)} b^a {\lambda^{a-1} \exp(-b\lambda)}, a > 0
$$

不同参数的 `Gam` 分布的图像如下图所示：

```python
from scipy.stats import gamma

x = np.linspace(0.01, 2, 100)

fig, axes = plt.subplots(1, 3, figsize=(10, 2))

A = [0.1, 1, 4]
B = [0.1, 1, 6]

for a, b, ax in zip(A, B, axes):
    y = gamma.pdf(x, a = a, scale=1.0/b)
    ax.plot(x, y, color="r")
    ax.set_ylim(0, 2)
    ax.set_xticks([0, 1, 2])
    ax.set_yticks([0, 1, 2])
    ax.set_xlabel(r"$\lambda$", fontsize="xx-large")
    ax.text(1, 1, "$a={}$\n$b={}$".format(a, b), fontsize="xx-large")

plt.show()
```

其均值和方差分别为：

$$
\begin{align}
\mathbb E[\lambda] & = \frac{a}{b}\\
{\rm var}[\lambda] & = \frac{a}{b^2}
\end{align}
$$

给定先验分布 ${\rm Gam}(\lambda~|~a_0, b_0)$，后验分布为：

$$
p(\lambda|\mathbf X)\propto\lambda^{a_0-1}\lambda^{N/2} \exp\left\{-b_0\lambda_0-\frac{\lambda}{2} \sum_{i=1}^N (x_n-\mu)^2\right\}
$$

从而这也是一个 `Gam` 分布 ${\rm Gam}(\lambda~|~a_N, b_N)$，其中：

$$
\begin{align}
a_N & = a_0 + \frac N 2 \\
b_N & = b_0 + \frac{1}{2} \sum_{i=1}^N (x_n-\mu)^2 = b_0 + \frac{N}{2} \sigma^2_{ML}
\end{align}
$$

因此，我们看到，$N$ 个数据点向参数 $a$ 添加了 $\frac N 2$ 的贡献，因此，我们可以认为初始状态向 $a$ 贡献了 $2a_0$ 个数据点；$N$ 个数据点向参数 $b$ 添加了 $\frac{N}{2} \sigma^2_{ML}$ 的贡献，我们可以认为这 $2a_0$ 个数据点平均每个贡献了 $2b_0/(2a_0)=b_0/a_0$ 的方差。

如果直接考虑方差，而不是精确度，我们使用的先验将是逆 `Gamma` 分布。不过用精确度表示起来更加方便。

### 均值和方差都未知

在这种情况下，似然函数为：

$$
p(\mathbf X|\mu, \lambda)=\left(\frac{\lambda}{2\pi}\right)^\frac{N}{2}
\exp\left\{-\frac{\lambda}{2} \sum_{n=1}^N (x_n-\mu)^2\right\}
\propto \left[\lambda^{1/2} \exp\left(-\frac{\lambda\mu^2}{2}\right)\right]^N
\exp\left\{ \lambda\mu \sum_{n=1}^N x_n - \frac{\lambda}{2} \sum_{n=1}^N x_n^2 \right\}
$$

这是一个指数族函数（`exponential family`）的形式。

由共轭性，考虑这样的先验分布：

$$
p(\mu,\lambda)\propto \left[\lambda^{1/2} \exp\left(-\frac{\lambda\mu^2}{2}\right)\right]^\beta
\exp\left\{ c\lambda\mu  - d\lambda \right\}
= \exp\left\{-\frac{\beta\lambda}{2}(\mu-c/\beta)^2\right\} \lambda^{\beta/2}
\exp\left\{-\left(d-\frac{c^2}{2\beta}\right)\lambda\right\}
$$

由于 $p(\mu,\lambda) = p(\mu|\lambda)p(\lambda)$，通过观察我们发现：

$$
p(\mu,\lambda) = \mathcal N(\mu|\mu_0, (\beta\lambda)^{-1}) \mathrm{Gam}(\lambda|a,b)
$$

其中 $\mu_0 = c/\beta, a=1+\beta/2, b=d-c^2/(2\beta)$。这个分布叫做高斯-伽马分布（`Gaussian-gamma`）或者正态-伽马分布（`normal-gamma`），其图像如图所示：

```python
from scipy.stats import gamma

mu, lam = np.mgrid[-2:2:.01, 0.01:2:.01]

a, b = 5, 6
mu_0 = 0
beta = 2

z = (beta*lam/2/np.pi) ** 1/2 * np.exp(- beta*lam/2 * (mu-mu_0)**2) \
    * gamma.pdf(lam, a=a, scale=1.0/b)

fig, ax = plt.subplots()

ax.contour(mu, lam, z, colors='r')
ax.set_xticks([-2, 0, 2])
ax.set_yticks([0, 1, 2])

ax.set_xlabel(r"$\mu$", fontsize="xx-large")
ax.set_ylabel(r"$\lambda$", fontsize="xx-large")

plt.show()
```

在 $D$ 维高斯分布 $\mathcal N(\mathbf x|\mathbf\mu, \mathbf\Lambda^{-1})$ 的情况下，若均值未知，精确度矩阵已知，先验分布为：

$$\mathcal N(\mathbf \mu|\mathbf \mu_0, \mathbf\Lambda_0^{-1})$$

若均值已知，精确度矩阵未知，先验分布为 `Wishart` 分布：

$$
\mathcal W(\mathbf\Lambda|\mathbf W, \nu)=B|\mathbf\Lambda|^{(\nu-D-1)/2}
\exp\left(-\frac{1}{2}\mathrm{Tr}(\mathbf W^{-1}\mathbf\Lambda)\right)
$$

其中 $\nu$ 叫做分布的自由度（`degrees of freedom`），$\mathbf W$ 是一个 $D\times D$ 矩阵，$\rm Tr$ 是矩阵的迹,归一化参数 $B$ 为

$$
B(\mathbf W, \nu) = |\mathbf W|^{-\nu/2}
\left(2^{\mu D/2} \pi^{D(D-1)/4} \prod_{i=1}^D
\Gamma\left(\frac{\nu+1-i}{2}\right)\right)^{-1}
$$

若均值和精确度矩阵都未知，先验分布为 `normal-Wishart`（或者 `Gaussian-Wishart`）分布：

$$
\mathcal N(\mathbf \mu|\mathbf \mu_0, (\beta\mathbf\Lambda)^{-1})
\mathcal W(\mathbf\Lambda|\mathbf W, \nu)
$$

## 2.3.7 学生 t 分布

之前我们看到精确度的先验分布是一个 Gamma 分布。现在考虑一个单变量高斯分布 $\mathcal N(x|\mu,\tau^{-1})$ 和精确度的一个伽马分布先验 ${\rm Gam}(\tau|a,b)$，我们对精确度进行积分之后，得到 $x$ 的边缘分布为

$$
\begin{align}
p(x|\mu, a,b) & = \int_{0}^{\infty} \mathcal N(x|\mu,\tau^{-1}) {\rm Gam}(\tau|a,b) d\tau \\
& = \int_{0}^{\infty}
\frac{b^a e^{-b\tau} \tau^{a-1}}{\Gamma(a)}
\left(\frac{\tau}{2\pi}\right)^{1/2}
\exp\left\{-\frac{\tau}{2}(x-\mu)^2\right\}
d\tau\\
& =
\frac{b^a}{\Gamma(a)} \left(\frac{1}{2\pi}\right)^{1/2}
\left[b+\frac{(x-\mu)^2}{2}\right]^{-a-1/2} \Gamma(a+1/2)
\end{align}
$$

（关于 $\tau$ 可以凑成一个 ${\rm Gam}(a+1/2, b+\frac{(x-\mu)^2}{2})$ 的核）

令参数 $\nu=2a, \lambda=a/b$，我们有

$$
\text{St}(x|\mu,\lambda,\nu)= \frac{\Gamma(v/2+1/2)}{\Gamma(v/2)} \left(\frac{\lambda}{\pi\nu}\right)^{1/2}
\left[1+\frac{\lambda(x-\mu)^2}{\nu}\right]^{-\nu/2-1/2}
$$

这就是学生 t 分布（`Student’s t-distribution`）。其中，参数 $\lambda$ 叫做精确度，$\mu$ 叫做自由度。

当自由度为 1 时，学生 t 分布退化为柯西分布，当自由度趋于无穷时，学生 t 分布变成均值 $\lambda$ 精度 $\lambda$ 的高斯分布（忽略所有与 $x$ 无关的项，对 $\nu$ 求极限，利用 $\lim_{x\to\infty}(1+\frac{1}{x})^x = e$）

其概率密度函数如图所示：

```python
from scipy.stats import t, norm

xx = np.linspace(-5, 5, 100)

dfs = [0.1, 1]

fig, ax = plt.subplots()

for df in dfs:
    ax.plot(xx, t.pdf(xx, df))

ax.plot(xx, norm.pdf(xx))

ax.set_xlim(-5, 5)
ax.set_xticks([-5, 0, 5])
ax.set_ylim(0, 0.5)

ax.legend([r"$\nu=0.1$", r"$\nu=1.0$", r"$\nu\to\infty$"], fontsize="xx-large")
ax.set_title(r"$\lambda=1,\ \mu=0$", fontsize="xx-large")

plt.show()
```

学生 t 分布可以看成是无穷多的高斯分布的一个混合。

从图中可以看出，它产生的分布的尾巴要比高斯分布的长。事实上，在有异常值的情况下，它比高斯分布更鲁棒一些：

```python
from scipy.stats import t, norm, uniform

tt = norm.rvs(size=30)
xx = np.linspace(-5, 10, 100)

fig, axes = plt.subplots(1, 2, figsize=(10, 4))

axes[0].hist(tt,
             normed=True,
             rwidth=0.5,
             color="yellow",
             bins = np.linspace(-5, 10, 30))

axes[0].set_xlim(-5, 10)
axes[0].set_xticks([-5, 0, 5, 10])
axes[1].set_ylim(0, 0.7)

l, s = norm.fit(tt)

axes[0].plot(xx, norm.pdf(xx, loc=l, scale=s), 'g')

d, l, s = t.fit(tt)
axes[0].plot(xx, t.pdf(xx, df=d, loc=l, scale=s), 'r')

axes[0].legend(["MLE for Gaussian", "MLE for t"])

tt1 = np.hstack([tt, uniform.rvs(size=4, loc=8, scale=2)])

axes[1].hist(tt1,
             normed=True,
             rwidth=0.5,
             color="yellow",
             bins = np.linspace(-5, 10, 30))

axes[1].set_xlim(-5, 10)
axes[1].set_ylim(0, 0.7)
axes[1].set_xticks([-5, 0, 5, 10])

l, s = norm.fit(tt1)

axes[1].plot(xx, norm.pdf(xx, loc=l, scale=s), 'g')

d, l, s = t.fit(tt1)
axes[1].plot(xx, t.pdf(xx, df=d, loc=l, scale=s), 'r')

axes[1].legend(["MLE for Gaussian", "MLE for t"])

plt.show()

```

如果我们取 $\nu=2a, \lambda=a/b, \eta=\tau b /a$，我们有：

$$
\mathrm{St}(x|\mu,\lambda,\nu) = \int_{0}^{\infty} \mathcal N(x~|~\mu,(\eta\lambda)^{-1}) {\rm Gam}(\eta ~|~\nu/2,\nu/2) d\eta
$$

（这也说明了学生 t 分布是一个概率分布：对 $x$ 积分等于 1。）

我们看到，自由度项只在 Gamma 分布出现，因此，我们可以将其推广到高维形式：

$$
\mathrm{St}(x|\mathbf{\mu,\Lambda},\nu) = \int_{0}^{\infty} \mathcal N(x~|~{\mathbf \mu,(\eta\mathbf\Lambda)^{-1}}) {\rm Gam}(\eta ~|~\nu/2,\nu/2) d\eta
$$

与一维的结果类似，我们有：

$$
\mathrm{St}(x|\mathbf{\mu,\Lambda},\nu) = \frac{\Gamma(v/2+D/2)}{\Gamma(v/2)} \frac{|\mathbf\Lambda|^{1/2}}{(\pi\nu)^{D/2}}
\left[1+\frac{\Delta^2}{\nu}\right]^{-\nu/2-D/2}
$$

其中 $\Delta^2 \mathbf{= (x -\mu)^\top\Lambda(x - \mu)}$。

学生 t 分布的均值方差和众数分别为：

$$
\begin{align}
\mathbb E[\mathbf x] & = \mathbf \mu & {\rm if} && \nu > 1\\
{\rm cov}[\mathbf x] & = \frac{\nu}{\nu-2} \mathbf\Lambda^{-1} & {\rm if} && \nu > 2 \\
{\rm mode}[\mathbf x] & = \mathbf \mu \\
\end{align}
$$

## 2.3.8 周期变量和 von Mises 分布

高斯分布并不能很好的解决周期变量的问题。

对于周期变量，一种解决方法是使用极坐标系 $0\leq\theta<2\pi$ 的周期性，使用极坐标的角度 $\theta$ 表示周期变量。

在极坐标系下，我们需要考虑这样的问题，两个角度 $1^\circ, 359^\circ$ 十分接近，如果我们选择 $0^\circ$ 作为原点，均值和标准差分别为 $180^\circ, 179^\circ$；选择 $180^\circ$ 作为原点，均值和标准差分别为 $0^\circ, 1^\circ$。因此我们需要一种解决这种问题的特殊方法。

考虑找到一组周期变量 $\mathcal D=\{\theta_1,\dots, \theta_N\}$ 的均值的问题。之前我们看到，随着坐标系的变化，直接计算的方法的结果也在变化。

为了找到一种不随坐标原点变化的均值衡量方法，我们将这些变量看成是单位圆上的点，即将角度 $\theta$ 映射为点 $\mathbf x$：

```python
xx = np.linspace(0, 2 * np.pi, 100)

_, ax = plt.subplots(figsize=(7, 7))

ax.plot(np.cos(xx), np.sin(xx), 'r')

ax.text(1, -0.2, "$x_1$", fontsize="xx-large")
ax.text(1, 0.2, "$x_2$", fontsize="xx-large")
ax.text(0.2, 1, "$x_3$", fontsize="xx-large")
ax.text(-0.4, 1, "$x_4$", fontsize="xx-large")

ax.axis("equal")
ax.set_xlim(-1.2, 1.2)

ax.spines['right'].set_color('none')
ax.spines['top'].set_color('none')

ax.xaxis.set_ticks_position('bottom')
ax.spines['bottom'].set_position(('data',0))
ax.yaxis.set_ticks_position('left')
ax.spines['left'].set_position(('data',0))

ax.set_xticks([])
ax.set_yticks([])

ax.plot([0, 0.4], [0, 0.6])
ax.text(0.12, 0.3, r'$\bar r$', fontsize="xx-large")
ax.text(0.42, 0.55, r'$\bar x$', fontsize="xx-large")
ax.text(0.35, 0.2, r'$\bar{\theta}$', fontsize="xx-large")

tt = np.linspace(0.2, np.sqrt(0.2 ** 2 + 0.3 ** 2))

ax.fill_between([0, 0.2], [0, 0.3], color="c")
ax.fill_between(tt, np.sqrt(0.2 ** 2 + 0.3 ** 2 - tt ** 2 + 0.001), color="c")

plt.show()
```

我们不直接求 $\theta$ 的均值，而是求这些点 $\bf x$ 的均值：

$$
\mathbf{\bar x} = \frac{1}{N} \sum_{n=1}^N \mathbf x_n
$$

然后得到相应的极坐标的角度 $\bar\theta$。

这样的设定能保证我们的均值 $\bf \bar x$ 不随极坐标的原点设置变化而变化。一般来说，这个均值通常落在单位圆内。

考虑坐标变换，$\mathbf x_n = (\cos\theta_n,\sin\theta_n), \mathbf{\bar x}=(\bar r\cos\bar\theta, \bar r\sin\theta)$，我们有：

$$
\bar r\cos\bar\theta = \frac{1}{N} \sum_{n=1}^N \cos\theta_n, \bar r\sin\theta \frac{1}{N} \sum_{n=1}^N \sin\theta_n
$$

利用正切函数，我们有：

$$
\bar\theta = \tan^{-1} \frac{\sum_n \sin\theta_n}{\sum_n \cos\theta_n}
$$

### von Mises 分布

对于周期函数，我们考虑一个周期为 $2\pi$ 的概率分布 $p(\theta)$，它应当满足这三个条件：

$$
\begin{align}
p(\theta) & \geq 0 \\
\int_{0}^{2\pi} p(\theta) & =1 \\
p(\theta+2\pi) &= p(\theta)
\end{align}
$$

按照单位圆的设定，我们可以假定 $\mathbf x=(x_1,x_2)$ 满足一个高斯分布 $\mathcal N(\mathbf\mu, \sigma^2\mathbf I)$，其中 $\mathbf\mu=\mu_1,\mu_2$，所以：

$$
p(x_1,x_2) = \frac{1}{2\pi\sigma^2} \exp\left\{-\frac{(x_1-\mu_1)^2 + (x_2-\mu_2)^2}{2\sigma^2}\right\}
$$

它的分布等高线是一系列的圆：

```python
xx = np.linspace(0, 2 * np.pi, 100)

_, ax = plt.subplots(figsize=(7, 7))

ax.plot(np.cos(xx), np.sin(xx), 'r')

ax.spines['right'].set_color('none')
ax.spines['top'].set_color('none')

ax.xaxis.set_ticks_position('bottom')
ax.spines['bottom'].set_position(('data',0))
ax.yaxis.set_ticks_position('left')
ax.spines['left'].set_position(('data',0))
ax.set_xticks([])
ax.set_yticks([])

mu1, mu2 = 0.5, 0.8
for r in [0.25, 0.5, 0.75]:
    ax.plot(r * np.cos(xx) + mu1, r * np.sin(xx) + mu2, 'b')

ax.axis('equal')
ax.set_xlim(-1.5, 2)
ax.text(1.35, 1, r"$p(\mathbf{x})$", fontsize="xx-large")
ax.text(-1,-1, r'$r=1$', fontsize="xx-large")
ax.text(1.8, -.2, "$x_1$", fontsize="xx-large")
ax.text(-.2, 1.8, "$x_2$", fontsize="xx-large")

plt.show()
```

考虑极坐标的变换：

$$
\begin{align}
x_1 = r\cos\theta, &&& x_2 = r\sin\theta \\
\mu_1 = r_0\cos\theta_0, &&& \mu_2 = r_0\sin\theta_0
\end{align}
$$

并将 $\mathbf x$ 限制在单位圆上即 $r=1$，我们有：

$$
-\frac{1}{2\sigma^2}\left\{(\cos\theta-r_0\cos\theta_0)^2 + (\sin\theta-r_0\sin\theta_0)^2\right\} =
-\frac{1}{2\sigma^2}\left\{1 + r_0^2 -2r_0\cos\theta\cos\theta_0 -2r_0\sin\theta\sin\theta_0\right\} =
\frac{r_0}{\sigma^2} \cos(\theta-\theta_0) + \mathrm{const}
$$

常数是与 $\theta$ 无关的项。

如果我们定义 $m=\frac{r_0}{\sigma^2}$，我们可以得到这样的一个分布：

$$
p(\theta|\theta_0,m) = \frac{1}{2\pi I_0(m)}\exp\{m\cos(\theta-\theta_0)\}
$$

这样我们就得到了 von Mises 分布，或者叫 `circular normal` 分布，这里 $\theta_0$ 是分布的均值，$m$ 类似于高斯分布中的精确度。

von Mises 分布在直角坐标系和极坐标系的图像如图所示：

```python
from scipy.stats import vonmises

tt = np.linspace(0, 2 * np.pi, 100)

fig, axes = plt.subplots(1, 2, figsize=(10, 4))

axes[0].plot(tt, vonmises.pdf(tt, 5, loc=np.pi/4), 'r')
axes[0].plot(tt, vonmises.pdf(tt, 1, loc=3*np.pi/4), 'b')
axes[0].set_xlim(0, 2*np.pi)
axes[0].legend([r'$m=5,\ \theta_0=\pi/4$', r'$m=1,\ \theta_0=3\pi/4$'], fontsize="x-large", loc=0)

for ax in axes:
    ax.set_xticks([])
    ax.set_yticks([])

ax = fig.add_subplot(122, projection='polar')
ax.set_axis_off()

ax.plot(tt, vonmises.pdf(tt, 5, loc=np.pi/4), 'r')
ax.plot(tt, vonmises.pdf(tt, 1, loc=3*np.pi/4), 'b')
ax.legend([r'$m=5,\ \theta_0=\pi/4$', r'$m=1,\ \theta_0=3\pi/4$'], fontsize="x-large", loc=0)
ax.grid("off")

ax.plot([0, np.pi/4], [0, 1], 'k')
ax.plot([0, 0], [0, 1], 'k')
ax.plot([0, 3*np.pi/4], [0, 1], 'k')
ax.text(0.1, 0.8, r'$0$', fontsize="x-large")
ax.text(-0.2, 0.8, r'$2\pi$', fontsize="x-large")
ax.text(0.3*np.pi, 1, r'$\pi / 4$', fontsize="x-large")
ax.text(0.7*np.pi, 0.8, r'$3\pi / 4$', fontsize="x-large")

plt.show()
```

$I_0(m)$ 是用来归一化概率分布的常数，是修正的 0 阶 Bessel 函数：

$$
I_0(m) =\frac{1}{2\pi} \int_{0}^{2\pi} \exp\{m\cos\theta\} d\theta
$$

```python
from scipy import special

_, ax = plt.subplots()

xx = np.r_[0:10:0.01]

ax.plot(xx, special.i0(xx), 'r')
ax.set_xlabel("$m$", fontsize='x-large')
ax.set_ylabel("$I_0(m)$", fontsize='x-large')
ax.set_xticks([0, 5, 10])
ax.set_yticks(np.r_[0:3001:1000])

plt.show()
```

现在考虑 `von Mises` 分布的最大似然：

$$
\ln p(\mathcal D|\theta_0,m) = -N\ln(2\pi)-N\ln I_0(m) + m\sum_{n=1}^N \cos(\theta_n-\theta_0)
$$

令其对 $\theta_0$ 的偏导为 0，我们有

$$
\sum_{n=1}^N \sin(\theta_n-\theta_0) = \cos\theta_0 \sum_{n=1}^N \sin\theta_n - \sin\theta_0 \sum_{n=1}^N \cos\theta_n = 0
$$

从而

$$
\theta_{ML} = \tan^{-1} \left\{\frac{\sum_{n} \sin \theta_n}{\sum_{n} \cos \theta_n}\right\}
$$

这与我们之前讨论的均值一致。

类似地，考虑对 $m$ 的偏导，利用 $I_0'(m) = I_1(m)$（$I_1$ 是修正的 1 阶 Bessel 函数），我们有：

$$
\frac{I_1(m)}{I_0(m)} = \frac{1}{N} \sum_{n=1}^N \cos(\theta_n-\theta_0)
$$

令

$$
A(m) = \frac{I_1(m)}{I_0(m)}
$$

其图像如下图所示。

因此，我们有

$$
A(m_{ML}) = \frac{1}{N} \cos\theta_0^{ML} \sum_{n=1}^N \cos\theta_n - \frac{1}{N} \sin\theta_0^{ML} \sum_{n=1}^N \sin\theta_n
$$

我们可以通过数值的方法来求 $m_{ML}$ 的值。

```python
_, ax = plt.subplots()

xx = np.r_[0:10:0.01]

ax.plot(xx, special.i1(xx) / special.i0(xx), 'r')

ax.set_xlabel("$m$", fontsize='x-large')
ax.set_ylabel("$A(m)$", fontsize='x-large')
ax.set_xticks(np.r_[0:11:5])
ax.set_yticks(np.r_[0:1.1:0.5])

plt.show()
```

## 2.3.9 高斯混合模型

之前说到，高斯分布是一个单峰模型，对于多峰情况的模拟效果并不好。

```python
from scipy.stats import multivariate_normal

sz = 100

tt1 = multivariate_normal.rvs(cov=np.array([[2,1], [1,2]]),
                              size=sz)
x1, y1 = tt1[:,0], tt1[:,1]

tt2 = multivariate_normal.rvs(mean = np.array([8, 8]),
                              cov=np.array([[1,0.5], [0.5,2]]),
                              size=sz)
x2, y2 = tt2[:,0], tt2[:,1]

fig, axes = plt.subplots(1, 2, figsize=(10, 4))

m1 = np.mean(tt1, axis=0)
c1 = np.dot((tt1 - m1).T, (tt1 - m1)) / sz

m2 = np.mean(tt2, axis=0)
c2 = np.dot((tt2 - m2).T, (tt2 - m2)) / sz

tt = np.vstack([tt1, tt2])

m = np.mean(tt, axis=0)
c = np.dot((tt - m).T, (tt - m)) / sz / 2

x, y = np.mgrid[-5:12:0.1, -5:12:0.1]

z1 = bivariate_normal(x, y,
                      mux=m1[0],
                      muy=m1[1],
                      sigmax=np.sqrt(c1[0][0]),
                      sigmay=np.sqrt(c1[1][1]),
                      sigmaxy = c1[1][0])

z2 = bivariate_normal(x, y,
                      mux=m2[0],
                      muy=m2[1],
                      sigmax=np.sqrt(c2[0][0]),
                      sigmay=np.sqrt(c2[1][1]),
                      sigmaxy = c2[1][0])

z = bivariate_normal(x, y,
                     mux=m[0],
                     muy=m[1],
                     sigmax=np.sqrt(c[0][0]),
                     sigmay=np.sqrt(c[1][1]),
                     sigmaxy = c[1][0])

axes[0].contour(x, y, z, 4, colors='b')
axes[1].contour(x, y, z1, 4, colors='b')
axes[1].contour(x, y, z2, 4, colors='b')

for ax in axes:
    ax.scatter(x1, y1, color='lime')
    ax.scatter(x2, y2, color='lime')

plt.show()
```

但我们可以使用多个高斯分布的混合来模拟多峰模型。

高斯分布的一个简单线性混合就能产生十分复杂的分布，如图所示，红线是三个高斯分布线性组合后的概率密度，蓝线是三个高斯分布按相应比例进行缩放后的概率密度：

```python
xx = np.linspace(-20, 20, 200)

y1 = norm.pdf(xx, loc=0, scale=4)
y2 = norm.pdf(xx, loc=10, scale=1.5)
y3 = norm.pdf(xx, loc=-8, scale=2)

fig, ax = plt.subplots()

ax.plot(xx, y1, 'b')
ax.plot(xx, y2, 'b')
ax.plot(xx, y3, 'b')

ax.plot(xx, y1 + y2 + y3, 'r')

ax.set_xticks([])
ax.set_yticks([])

ax.set_xlabel("$x$", fontsize="xx-large")
ax.set_ylabel("$p(x)$", fontsize="xx-large")

plt.show()
```

考虑 $K$ 个高斯分布的线性组合，即高斯混合分布（`mixture of Gaussians`）：

$$
p(\mathbf x) = \sum_{k=1}^K \pi_k \mathcal N(\mathbf x|\mathbf \mu_k, \mathbf \Sigma_k)
$$

其中 $\mathcal N(\mathbf x|\mathbf \mu_k, \mathbf \Sigma_k)$ 是一个高斯分布，是混合分布的一个组分（`component of the mixture`）。

考虑概率分布的归一性 $\int p(\mathbf x) d\mathbf x = 1$：

$$
\sum_{k=1}^K \pi_k = 1
$$

同时考虑非负性 $p(\mathbf x) \geq 0$：

$$
\forall k, \pi_k \geq 0
$$

于是我们有：

$$
0 \leq \pi_k \leq 1
$$

另一方面，我们有：

$$
p(\mathbf x) = \sum_{k=1}^N p(k) p(\mathbf x|k)
$$

从而我们可以认为 $\pi_k = p(k)$ 是第 $k$ 个组分的先验分布，$p(\mathbf x|k)$ 是给定 $k$ 下的条件分布。

从而我们可以计算 $k$ 的后验分布：

$$
\begin{align}
\gamma_k(\mathbf x) & = p(k|\mathbf x) \\
& = \frac{p(k) p(\mathbf x|k)}{\sum_{k=1}^N p(k) p(\mathbf x|k)} \\
& = \frac{\pi_k \mathcal N(\mathbf x|\mathbf \mu_k, \mathbf \Sigma_k) }{\sum_{k=1}^K \pi_k \mathcal N(\mathbf x|\mathbf \mu_k, \mathbf \Sigma_k) }
\end{align}
$$

其参数为 $\bf \pi, \mu, \Sigma$，其中 $\mathbf \pi = {\pi_1, \dots, \pi_n}, \mathbf \mu = \{\mathbf\mu_1,\dots,\mathbf\mu_n\}, \mathbf\Sigma=\{\mathbf\Sigma_1,\dots,\mathbf\Sigma_n\}$。

对于数据 $\mathbf X=\{\mathbf x_1,\dots,\mathbf x_n\}$，似然函数为

$$
\ln p(\mathbf{X|\pi, \mu, \Sigma}) = \sum_{n=1}^N \ln \left\{ \sum_{k=1}^K \pi_k \mathcal N(\mathbf x_n|\mathbf \mu_k, \mathbf \Sigma_k) \right\}
$$

其最大似然解比较复杂，没有显式的表示形式，可以用数值法求解，或者用 EM 算法求解。


# Modeling spiking neural networks with Brian

## Homework

Some general setup first...

```python
# If Brian2 is not yet installed, the following will install it
# (otherwise it will print a number of "Requirement already satisfied" lines)
%pip install brian2
```

    Requirement already satisfied: brian2 in /home/marcel/anaconda3/envs/datasci_course/lib/python3.8/site-packages (2.4.1)
    Requirement already satisfied: numpy>=1.15 in /home/marcel/anaconda3/envs/datasci_course/lib/python3.8/site-packages (from brian2) (1.19.2)
    Requirement already satisfied: jinja2>=2.7 in /home/marcel/anaconda3/envs/datasci_course/lib/python3.8/site-packages (from brian2) (2.11.2)
    Requirement already satisfied: sympy>=1.2 in /home/marcel/anaconda3/envs/datasci_course/lib/python3.8/site-packages (from brian2) (1.6.2)
    Requirement already satisfied: pyparsing in /home/marcel/anaconda3/envs/datasci_course/lib/python3.8/site-packages (from brian2) (2.4.7)
    Requirement already satisfied: cython>=0.29 in /home/marcel/anaconda3/envs/datasci_course/lib/python3.8/site-packages (from brian2) (0.29.21)
    Requirement already satisfied: setuptools>=24.2 in /home/marcel/anaconda3/envs/datasci_course/lib/python3.8/site-packages (from brian2) (49.6.0.post20201009)
    Requirement already satisfied: MarkupSafe>=0.23 in /home/marcel/anaconda3/envs/datasci_course/lib/python3.8/site-packages (from jinja2>=2.7->brian2) (1.1.1)
    Requirement already satisfied: mpmath>=0.19 in /home/marcel/anaconda3/envs/datasci_course/lib/python3.8/site-packages (from sympy>=1.2->brian2) (1.1.0)
    Note: you may need to restart the kernel to use updated packages.

```python
from brian2 import *
prefs.codegen.target = 'numpy'
```

```python
import matplotlib.pyplot as plt
import numpy as np

%matplotlib inline
```

A plotting function that we will use later

```python
def plot_activity(exc_spike_mon, exc_pop_mon, inh_spike_mon):
    """Plots raster plots, population activity and a histogram of the coefficient of variation (CV) of the
       interspike intervals (ISI) – see Lecture 5 for more details.

       exc_spike_mon : a SpikeMonitor recording the excitatory population
       exc_pop_mon : a PopulationRateMonitor recording the excitatory population
       inh_spike_mon : a SpikeMonitor recording the inhibitory population
       """
    fig, axs = plt.subplots(3, 2, sharex='col', gridspec_kw={'height_ratios': [3, 2, 1],
                                                             'width_ratios': [3, 1]},
                           figsize=(12, 8))
    axs[0, 0].plot(exc_spike_mon.t/ms, exc_spike_mon.i, '|')
    axs[0, 0].set(ylabel='neuron index', title='excitatory neurons')
    # Get the individual spike trains for each neuron
    spike_trains = exc_spike_mon.spike_trains()
    # Calculate the CV of the ISI distribution for each neuron
    CVs = []
    for train in spike_trains.values():
        # We ignore spikes in the first 50ms to focus on the steady-state behaviour
        spikes = train[train>50*ms]
        if len(spikes) < 2:
            CVs.append(np.nan)
        else:
            ISIs = np.diff(spikes/ms)
            CVs.append(np.std(ISIs)/np.mean(ISIs))
    axs[0, 1].hist(CVs, bins=20, range=(0., 1.))
    axs[0, 1].xaxis.set_tick_params(labelbottom=True)
    axs[0, 1].set(xlabel='CV of ISI', yticks=[])
    axs[1, 0].plot(exc_pop_mon.t/ms, exc_pop_mon.smooth_rate(width=2*ms))
    axs[1, 0].hlines(np.mean(exc_pop_mon.rate/Hz), exc_pop_mon.t[0]/ms, exc_pop_mon.t[-1]/ms,
                     color='darkgray')
    axs[1, 0].set_ylabel('pop. firing rate (Hz)')
    axs[1, 0].set_ylim(0, axs[1, 0].get_ylim()[1])
    axs[2, 0].plot(inh_spike_mon.t/ms, inh_spike_mon.i, '|', color='C1')
    axs[2, 0].set(ylabel='neuron index', title='inhibitory neurons')
    axs[2, 0].set_xlabel('time (ms)')
    axs[1, 1].axis('off')
    axs[2, 1].axis('off')
    fig.tight_layout()
```

## Network

The following network is almost identical to the one we used in the tutorial. There are only three differences:
* The initial value of the membrane potential and the external input current are random and not fixed, they are given in Brian syntax referring to a uniformly distributed random number between 0 and 1 as `rand()`. This means that the membrane potential $V$ is initialized randomly between $E_L$ and the threshold, and the external current is randomly chosen between 20mV and 30mV.
* The neuron model has an absolute refractory period of 5ms (a period after each spike where the membrane potential
  is clamped to its value so the neuron cannot spike again)
* We have split up the excitatory and inhibitory synapses a bit more to chose their strengths independently: there are now separate excitatory → excitatory, excitatory → inhibitory, and inhibitory → excitatory synapses (inhibitory → inhibitory synapses have been removed).

```python
start_scope()
N_E = 4000  # excitatory neurons
N_I = 1000  # inhibitory neurons
E_L = -50*mV
V_threshold = -30*mV
V_reset = -55*mV
C = 70*pF
g_L = 10*nS
tau = C/g_L
tau_syn = 2*ms
w_exc_to_exc = 0*mV
w_exc_to_inh = 0*mV
w_inh_to_exc = 0*mV
duration = 1000*ms

neurons = NeuronGroup(N_E + N_I,
                      '''dV/dt = (E_L - V + I_syn + I_ext)/tau : volt (unless refractory)
                         I_ext : volt (constant)
                         dI_syn/dt = -I_syn/tau_syn : volt # decay between spikes''',
                      threshold='V>V_threshold', reset='V=V_reset', method='euler', refractory=5*ms)
# Initialize values
neurons.V = 'E_L + rand()*(V_threshold - E_L)'
exc_neurons = neurons[:N_E]  # Uses "slicing" to get subpopulations of neurons
inh_neurons = neurons[N_E:]

exc_neurons.I_ext = '20*mV + rand()*10*mV'

# Connect neurons to each other
exc_to_exc = Synapses(exc_neurons, exc_neurons, on_pre='I_syn_post += w_exc_to_exc')
exc_to_exc.connect(p=0.02)
exc_to_inh = Synapses(exc_neurons, inh_neurons, on_pre='I_syn_post += w_exc_to_inh')
exc_to_inh.connect(p=0.02)
inh_to_exc = Synapses(inh_neurons, exc_neurons, on_pre='I_syn_post -= w_inh_to_exc')
inh_to_exc.connect(p=0.02)

# Records from 100 excitatory neurons and 25 inhibitory neurons only, to get simpler plots
spike_mon_exc = SpikeMonitor(exc_neurons[:100])
spike_mon_inh = SpikeMonitor(inh_neurons[:25])
pop_mon_exc = PopulationRateMonitor(exc_neurons)  # Records the population rate over time

# run simulation
run(duration, report='text')  # for simulations that take longer to run
```

    Starting simulation at t=0. s for a duration of 1. s
    1. s (100%) simulated in 5s

```python
# Plot the activity using the provided function
plot_activity(spike_mon_exc, pop_mon_exc, spike_mon_inh)
```

## Exercise 1

Run the network once with the parameters above (**"configuration A"**: `w_exc_to_exc = 0*mV`, `w_exc_to_inh = 0*mV`, `w_inh_to_exc = 0*mV`) and once with the parameters `w_exc_to_exc = 0.5*mV`, `w_exc_to_inh = 2*mV`, `w_inh_to_exc = 2*mV` (**"configuration B"**).

For each model, state whether you'd say that individual neurons show *regular* or *irregular activity*, and whether the network is *globally asynchronous* or *globally synchronous*.

### Configuration A

```python
start_scope()
N_E = 4000  # excitatory neurons
N_I = 1000  # inhibitory neurons
E_L = -50*mV
V_threshold = -30*mV
V_reset = -55*mV
C = 70*pF
g_L = 10*nS
tau = C/g_L
tau_syn = 2*ms
w_exc_to_exc = 0*mV
w_exc_to_inh = 0*mV
w_inh_to_exc = 0*mV
duration = 1000*ms

neurons = NeuronGroup(N_E + N_I,
                      '''dV/dt = (E_L - V + I_syn + I_ext)/tau : volt (unless refractory)
                         I_ext : volt (constant)
                         dI_syn/dt = -I_syn/tau_syn : volt # decay between spikes''',
                      threshold='V>V_threshold', reset='V=V_reset', method='euler', refractory=5*ms)
# Initialize values
neurons.V = 'E_L + rand()*(V_threshold - E_L)'
exc_neurons = neurons[:N_E]  # Uses "slicing" to get subpopulations of neurons
inh_neurons = neurons[N_E:]

exc_neurons.I_ext = '20*mV + rand()*10*mV'

# Connect neurons to each other
exc_to_exc = Synapses(exc_neurons, exc_neurons, on_pre='I_syn_post += w_exc_to_exc')
exc_to_exc.connect(p=0.02)
exc_to_inh = Synapses(exc_neurons, inh_neurons, on_pre='I_syn_post += w_exc_to_inh')
exc_to_inh.connect(p=0.02)
inh_to_exc = Synapses(inh_neurons, exc_neurons, on_pre='I_syn_post -= w_inh_to_exc')
inh_to_exc.connect(p=0.02)

# Records from 100 excitatory neurons and 25 inhibitory neurons only, to get simpler plots
spike_mon_exc = SpikeMonitor(exc_neurons[:100])
spike_mon_inh = SpikeMonitor(inh_neurons[:25])
pop_mon_exc = PopulationRateMonitor(exc_neurons)  # Records the population rate over time

# run simulation
run(duration, report='text')  # for simulations that take longer to run
```

    Starting simulation at t=0. s for a duration of 1. s
    1. s (100%) simulated in 5s

```python
plot_activity(spike_mon_exc, pop_mon_exc, spike_mon_inh)
```

Individual neurons fire **regularly** (CV of ISI is 0), the network activity is **globally asynchronous** (global firing rate is constant over time, without global oscillations).

### Configuration B

```python
start_scope()
N_E = 4000  # excitatory neurons
N_I = 1000  # inhibitory neurons
E_L = -50*mV
V_threshold = -30*mV
V_reset = -55*mV
C = 70*pF
g_L = 10*nS
tau = C/g_L
tau_syn = 2*ms
w_exc_to_exc = 0.5*mV
w_exc_to_inh = 2*mV
w_inh_to_exc = 2*mV
duration = 1000*ms

neurons = NeuronGroup(N_E + N_I,
                      '''dV/dt = (E_L - V + I_syn + I_ext)/tau : volt (unless refractory)
                         I_ext : volt (constant)
                         dI_syn/dt = -I_syn/tau_syn : volt # decay between spikes''',
                      threshold='V>V_threshold', reset='V=V_reset', method='euler', refractory=5*ms)
# Initialize values
neurons.V = 'E_L + rand()*(V_threshold - E_L)'
exc_neurons = neurons[:N_E]  # Uses "slicing" to get subpopulations of neurons
inh_neurons = neurons[N_E:]

exc_neurons.I_ext = '20*mV + rand()*10*mV'

# Connect neurons to each other
exc_to_exc = Synapses(exc_neurons, exc_neurons, on_pre='I_syn_post += w_exc_to_exc')
exc_to_exc.connect(p=0.02)
exc_to_inh = Synapses(exc_neurons, inh_neurons, on_pre='I_syn_post += w_exc_to_inh')
exc_to_inh.connect(p=0.02)
inh_to_exc = Synapses(inh_neurons, exc_neurons, on_pre='I_syn_post -= w_inh_to_exc')
inh_to_exc.connect(p=0.02)

# Records from 100 excitatory neurons and 25 inhibitory neurons only, to get simpler plots
spike_mon_exc = SpikeMonitor(exc_neurons[:100])
spike_mon_inh = SpikeMonitor(inh_neurons[:25])
pop_mon_exc = PopulationRateMonitor(exc_neurons)  # Records the population rate over time

# run simulation
run(duration, report='text')  # for simulations that take longer to run
```

    Starting simulation at t=0. s for a duration of 1. s
    1. s (100%) simulated in 5s

```python
plot_activity(spike_mon_exc, pop_mon_exc, spike_mon_inh)
```

Individual neurons fire **regularly** (CV of ISI is mostly close to zero, and in general << 1), the network activity is **globally synchronous** (global firing rate shows strong oscillations).

## Exercise 2

For both parametrizations, plot the spike count of the excitatory neurons as a function of the external input current that they receive. Which of the models has a very clear dependency?

**Hint**: The `SpikeMonitor` `spike_mon_exc` only records from 100 neurons. You can restrict your analysis to these 100 neurons, or change the definition of the `SpikeMonitor` so that it records from all neurons.

### Configuration A

```python
start_scope()
N_E = 4000  # excitatory neurons
N_I = 1000  # inhibitory neurons
E_L = -50*mV
V_threshold = -30*mV
V_reset = -55*mV
C = 70*pF
g_L = 10*nS
tau = C/g_L
tau_syn = 2*ms
w_exc_to_exc = 0*mV
w_exc_to_inh = 0*mV
w_inh_to_exc = 0*mV
duration = 1000*ms

neurons = NeuronGroup(N_E + N_I,
                      '''dV/dt = (E_L - V + I_syn + I_ext)/tau : volt (unless refractory)
                         I_ext : volt (constant)
                         dI_syn/dt = -I_syn/tau_syn : volt # decay between spikes''',
                      threshold='V>V_threshold', reset='V=V_reset', method='euler', refractory=5*ms)
# Initialize values
neurons.V = 'E_L + rand()*(V_threshold - E_L)'
exc_neurons = neurons[:N_E]  # Uses "slicing" to get subpopulations of neurons
inh_neurons = neurons[N_E:]

exc_neurons.I_ext = '20*mV + rand()*10*mV'

# Connect neurons to each other
exc_to_exc = Synapses(exc_neurons, exc_neurons, on_pre='I_syn_post += w_exc_to_exc')
exc_to_exc.connect(p=0.02)
exc_to_inh = Synapses(exc_neurons, inh_neurons, on_pre='I_syn_post += w_exc_to_inh')
exc_to_inh.connect(p=0.02)
inh_to_exc = Synapses(inh_neurons, exc_neurons, on_pre='I_syn_post -= w_inh_to_exc')
inh_to_exc.connect(p=0.02)

# Recording from *all* excitatory neurons
spike_mon_exc = SpikeMonitor(exc_neurons)
spike_mon_inh = SpikeMonitor(inh_neurons[:25])
pop_mon_exc = PopulationRateMonitor(exc_neurons)  # Records the population rate over time

# run simulation
run(duration, report='text')  # for simulations that take longer to run
```

    Starting simulation at t=0. s for a duration of 1. s
    1. s (100%) simulated in 5s

```python
plt.plot(exc_neurons.I_ext/mV, spike_mon_exc.count, '.');
```

### Configuration B

```python
start_scope()
N_E = 4000  # excitatory neurons
N_I = 1000  # inhibitory neurons
E_L = -50*mV
V_threshold = -30*mV
V_reset = -55*mV
C = 70*pF
g_L = 10*nS
tau = C/g_L
tau_syn = 2*ms
w_exc_to_exc = 0.5*mV
w_exc_to_inh = 2*mV
w_inh_to_exc = 2*mV
duration = 1000*ms

neurons = NeuronGroup(N_E + N_I,
                      '''dV/dt = (E_L - V + I_syn + I_ext)/tau : volt (unless refractory)
                         I_ext : volt (constant)
                         dI_syn/dt = -I_syn/tau_syn : volt # decay between spikes''',
                      threshold='V>V_threshold', reset='V=V_reset', method='euler', refractory=5*ms)
# Initialize values
neurons.V = 'E_L + rand()*(V_threshold - E_L)'
exc_neurons = neurons[:N_E]  # Uses "slicing" to get subpopulations of neurons
inh_neurons = neurons[N_E:]

exc_neurons.I_ext = '20*mV + rand()*10*mV'

# Connect neurons to each other
exc_to_exc = Synapses(exc_neurons, exc_neurons, on_pre='I_syn_post += w_exc_to_exc')
exc_to_exc.connect(p=0.02)
exc_to_inh = Synapses(exc_neurons, inh_neurons, on_pre='I_syn_post += w_exc_to_inh')
exc_to_inh.connect(p=0.02)
inh_to_exc = Synapses(inh_neurons, exc_neurons, on_pre='I_syn_post -= w_inh_to_exc')
inh_to_exc.connect(p=0.02)

# Recording from *all* excitatory neurons
spike_mon_exc = SpikeMonitor(exc_neurons)
spike_mon_inh = SpikeMonitor(inh_neurons[:25])
pop_mon_exc = PopulationRateMonitor(exc_neurons)  # Records the population rate over time

# run simulation
run(duration, report='text')  # for simulations that take longer to run
```

    Starting simulation at t=0. s for a duration of 1. s
    1. s (100%) simulated in 5s

```python
plt.plot(exc_neurons.I_ext/mV, spike_mon_exc.count, '.');
```

In **configuration A**, the firing rate very clearly depends on the input they receive (this basically reproduces the f/I curve we plotted in the tutorial). In the other configuration, the dependency is not clear, for intermediate inputs the firing rate does not depend on the external input strength at all.

## Exercise 3

Change the definition of the `exc_to_exc` (excitatory → excitatory) synapses so that each synapse has a **random delay** (the time it takes to propagate a spike to its target) **between 1ms and 3ms**.

You can set delays in the same way as e.g. the input current `I_ext`, by assigning to the `exc_to_exc.delay` attribute, i.e. by using `exc_to_exc.delay = ...`. (Note: You need to add this line *after* `exc_to_exc.connect(...)`).

When you run the new network with random delays with the parameters from *configuration B*: what changes in the network activity compared to the network without delays?

```python
start_scope()
N_E = 4000  # excitatory neurons
N_I = 1000  # inhibitory neurons
E_L = -50*mV
V_threshold = -30*mV
V_reset = -55*mV
C = 70*pF
g_L = 10*nS
tau = C/g_L
tau_syn = 2*ms
w_exc_to_exc = 0.5*mV
w_exc_to_inh = 2*mV
w_inh_to_exc = 2*mV
duration = 1000*ms

neurons = NeuronGroup(N_E + N_I,
                      '''dV/dt = (E_L - V + I_syn + I_ext)/tau : volt (unless refractory)
                         I_ext : volt (constant)
                         dI_syn/dt = -I_syn/tau_syn : volt # decay between spikes''',
                      threshold='V>V_threshold', reset='V=V_reset', method='euler', refractory=5*ms)
# Initialize values
neurons.V = 'E_L + rand()*(V_threshold - E_L)'
exc_neurons = neurons[:N_E]  # Uses "slicing" to get subpopulations of neurons
inh_neurons = neurons[N_E:]

exc_neurons.I_ext = '20*mV + rand()*10*mV'

# Connect neurons to each other
exc_to_exc = Synapses(exc_neurons, exc_neurons, on_pre='I_syn_post += w_exc_to_exc')
exc_to_exc.connect(p=0.02)
exc_to_exc.delay = '1*ms + rand()*2*ms'  # random delay between 1ms and 3ms
exc_to_inh = Synapses(exc_neurons, inh_neurons, on_pre='I_syn_post += w_exc_to_inh')
exc_to_inh.connect(p=0.02)
inh_to_exc = Synapses(inh_neurons, exc_neurons, on_pre='I_syn_post -= w_inh_to_exc')
inh_to_exc.connect(p=0.02)

# Records from 100 excitatory neurons and 25 inhibitory neurons only, to get simpler plots
spike_mon_exc = SpikeMonitor(exc_neurons[:100])
spike_mon_inh = SpikeMonitor(inh_neurons[:25])
pop_mon_exc = PopulationRateMonitor(exc_neurons)  # Records the population rate over time

# run simulation
run(duration, report='text')  # for simulations that take longer to run
```

    Starting simulation at t=0. s for a duration of 1. s
    1. s (100%) simulated in 6s

```python
plot_activity(spike_mon_exc, pop_mon_exc, spike_mon_inh)
```

The global network activity is no longer globally synchronous.


# Neuromatch Academy: Week 1, Day 5, Tutorial 1
# Dimensionality Reduction: Geometric view of data

---

Tutorial objectives
In this notebook we'll explore how multivariate data can be represented in different orthonormal bases. This will help us build intuition that will be helpful in understanding PCA in the following tutorial.

Steps:
 1. Generate correlated multivariate data.
 2. Define an arbitrary orthonormal basis.
 3. Project data onto new basis.

---

```python
#@title Video: Geometric view of data
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="emLW0F-VUag", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video

```

    Video available at https://youtube.com/watch?v=emLW0F-VUag

# Setup
Run these cells to get the tutorial started.

```python
#library imports
import time                        # import time
import numpy as np                 # import numpy
import scipy as sp                 # import scipy
import math                        # import basic math functions
import random                      # import basic random number generator functions

import matplotlib.pyplot as plt    # import matplotlib
from IPython import display

```

```python
#@title Figure Settings
%matplotlib inline
fig_w, fig_h = (8, 8)
plt.rcParams.update({'figure.figsize': (fig_w, fig_h)})
plt.style.use('ggplot')
%config InlineBackend.figure_format = 'retina'
```

```python
#@title Helper functions

def get_data(cov_matrix):
  """
  Returns a matrix of 1000 samples from a bivariate, zero-mean Gaussian

  Note that samples are sorted in ascending order for the first random variable.

  Args:
    cov_matrix (numpy array of floats): desired covariance matrix

  Returns:
    (numpy array of floats) : samples from the bivariate Gaussian, with
                              each column corresponding to a different random variable
  """
  mean = np.array([0,0])
  X = np.random.multivariate_normal(mean,cov_matrix,size = 1000)
  indices_for_sorting = np.argsort(X[:,0])
  X = X[indices_for_sorting,:]
  return X

def plot_data(X):
  """
  Plots bivariate data. Includes a plot of each random variable, and a scatter
  plot of their joint activity. The title indicates the sample correlation
  calculated from the data.

  Args:
    X (numpy array of floats):   Data matrix
                                  each column corresponds to a different random variable

  Returns:
    Nothing.
  """
  fig = plt.figure(figsize=[8,4])
  gs = fig.add_gridspec(2,2)
  ax1 = fig.add_subplot(gs[0,0])
  ax1.plot(X[:,0],color='k')
  plt.ylabel('Neuron 1')
  plt.title('Sample var 1: {:.1f}'.format(np.var(X[:,0])))
  ax1.set_xticklabels([])
  ax2 = fig.add_subplot(gs[1,0])
  ax2.plot(X[:,1],color='k')
  plt.xlabel('Sample Number')
  plt.ylabel('Neuron 2')
  plt.title('Sample var 2: {:.1f}'.format(np.var(X[:,1])))
  ax3 = fig.add_subplot(gs[:, 1])
  ax3.plot(X[:,0],X[:,1],'.',markerfacecolor=[.5,.5,.5], markeredgewidth=0)
  ax3.axis('equal')
  plt.xlabel('Neuron 1 activity')
  plt.ylabel('Neuron 2 activity')
  plt.title('Sample corr: {:.1f}'.format(np.corrcoef(X[:,0],X[:,1])[0,1]))

def plot_basis_vectors(X,W):
  """
  Plots bivariate data as well as new basis vectors.

  Args:
    X (numpy array of floats):   Data matrix
                                  each column corresponds to a different random variable
    W (numpy array of floats):   Square matrix representing new orthonormal basis
                                  each column represents a basis vector

  Returns:
    Nothing.
  """
  plt.figure(figsize=[4,4])
  plt.plot(X[:,0],X[:,1],'.',color=[.5,.5,.5],label='Data')
  plt.axis('equal')
  plt.xlabel('Neuron 1 activity')
  plt.ylabel('Neuron 2 activity')
  plt.plot([0,W[0,0]],[0,W[1,0]],color='r',linewidth=3,label = 'Basis vector 1')
  plt.plot([0,W[0,1]],[0,W[1,1]],color='b',linewidth=3,label = 'Basis vector 2')
  plt.legend()

def plot_data_new_basis(Y):
  """
  Plots bivariate data after transformation to new bases. Similar to plot_data but
  with colors corresponding to projections onto basis 1 (red) and basis 2 (blue).
  The title indicates the sample correlation calculated from the data.

  Note that samples are re-sorted in ascending order for the first random variable.

  Args:
    Y (numpy array of floats):   Data matrix in new basis
                                  each column corresponds to a different random variable

  Returns:
    Nothing.
  """
  fig = plt.figure(figsize=[8,4])
  gs = fig.add_gridspec(2,2)
  ax1 = fig.add_subplot(gs[0,0])
  ax1.plot(Y[:,0],'r')
  plt.xlabel
  plt.ylabel('Projection \n basis vector 1')
  plt.title('Sample var 1: {:.1f}'.format(np.var(Y[:,0])))
  ax1.set_xticklabels([])
  ax2 = fig.add_subplot(gs[1,0])
  ax2.plot(Y[:,1],'b')
  plt.xlabel('Sample number')
  plt.ylabel('Projection \n basis vector 2')
  plt.title('Sample var 2: {:.1f}'.format(np.var(Y[:,1])))
  ax3 = fig.add_subplot(gs[:, 1])
  ax3.plot(Y[:,0],Y[:,1],'.',color=[.5,.5,.5])
  ax3.axis('equal')
  plt.xlabel('Projection basis vector 1')
  plt.ylabel('Projection basis vector 2')
  plt.title('Sample corr: {:.1f}'.format(np.corrcoef(Y[:,0],Y[:,1])[0,1]))

```

# Generate correlated multivariate data

```python
#@title Video: Multivariate data
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="YOan2BQVzTQ", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=YOan2BQVzTQ

To study multivariate data, first we generate it. In this exercise we generate data from a *bivariate normal distribution*. This is an extension of the one-dimensional normal distribution to two dimensions, in which each $x_i$ is marginally normal with mean $\mu_i$ and variance $\sigma_i^2$:

\begin{align}
x_i \sim \mathcal{N}(\mu_i,\sigma_i^2)
\end{align}

Additionally, the joint distribution for $x_1$ and $x_2$ has a specified correlation coefficient $\rho$. Recall that the correlation coefficient is a normalized version of the covariance, and ranges between -1 and +1.

\begin{align}
\rho = \frac{\text{cov}(x_1,x_2)}{\sqrt{\sigma_1^2 \sigma_2^2}}
\end{align}

For simplicity, we will assume that the mean of each variable has already been subtracted, so that $\mu_i=0$. The remaining parameters can be summarized in the covariance matrix:
\begin{equation*}
{\bf \Sigma} =
\begin{pmatrix}
 \text{var}(x_1) & \text{cov}(x_1,x_2) \\
 \text{cov}(x_1,x_2) &\text{var}(x_2)
\end{pmatrix}
\end{equation*}
Note that this is a symmetric matrix with the variances $\text{var}(x_i) = \sigma_i^2$ on the diagonal, and the covariance on the off-diagonal.

### Exercise
We have provided code to draw random samples from a zero-mean bivariate normal distribution. These samples could be used to simulate changes in firing rates for two neurons. Fill in the function below to calculate the covariance matrix given the desired variances and correlation coefficient. The covariance can be found by rearranging the equation above:

\begin{align}
\text{cov}(x_1,x_2) = \rho \sqrt{\sigma_1^2 \sigma_2^2}
\end{align}

Use these functions to generate and plot data while varying the parameters. You should get a feel for how changing the correlation coefficient affects the geometry of the simulated data.

**Suggestions**
* Fill in the function `calculate_cov_matrix` to calculate the covariance.
* Generate and plot the data for $\sigma_1^2 =1$, $\sigma_1^2 =1$, and $\rho = .8$. Try plotting the data for different values of the correlation coefficent: $\rho = -1, -.5, 0, .5, 1$.

```python
help(plot_data)
help(get_data)
```

    Help on function plot_data in module __main__:

    plot_data(X)
        Plots bivariate data. Includes a plot of each random variable, and a scatter
        plot of their joint activity. The title indicates the sample correlation
        calculated from the data.

        Args:
          X (numpy array of floats):   Data matrix
                                        each column corresponds to a different random variable

        Returns:
          Nothing.

    Help on function get_data in module __main__:

    get_data(cov_matrix)
        Returns a matrix of 1000 samples from a bivariate, zero-mean Gaussian

        Note that samples are sorted in ascending order for the first random variable.

        Args:
          cov_matrix (numpy array of floats): desired covariance matrix

        Returns:
          (numpy array of floats) : samples from the bivariate Gaussian, with
                                    each column corresponding to a different random variable

```python

def calculate_cov_matrix(var_1,var_2,corr_coef):
  """
  Calculates the covariance matrix based on the variances and correlation coefficient.

  Args:
    var_1 (scalar):    variance of the first random variable
    var_2 (scalar):    variance of the second random variable
    corr_coef (scalar):      correlation coefficient

  Returns:
    (numpy array of floats) : covariance matrix
  """
  ###################################################################
  ## Insert your code here to:
  ##                calculate the covariance from the variances and correlation
  # cov = ...
  cov_matrix = np.array([[var_1,cov],[cov,var_2]])

  #uncomment once you've filled in the function
  raise NotImplementedError("Student excercise: calculate the covariance matrix!")
  ###################################################################
  return cov

###################################################################
## Insert your code here to:
##                generate and plot bivariate Gaussian data with variances of 1
##                      and a correlation coefficients of: 0.8
##                repeat while varying the correlation coefficient from -1 to 1
###################################################################
variance_1 = 1
variance_2 = 1
corr_coef = 0.8

#uncomment to test your code and plot
#cov_matrix = calculate_cov_matrix(variance_1,variance_2,corr_coef)
#X = get_data(cov_matrix)
#plot_data(X)
```

```python
# to_remove solution

def calculate_cov_matrix(var_1,var_2,corr_coef):
  """
  Calculates the covariance matrix based on the variances and correlation coefficient.

  Args:
    var_1 (scalar):    variance of the first random variable
    var_2 (scalar):    variance of the second random variable
    corr_coef (scalar):      correlation coefficient

  Returns:
    (numpy array of floats) : covariance matrix
  """
  cov = corr_coef * np.sqrt(var_1 * var_2)
  cov_matrix = np.array([[var_1,cov],[cov,var_2]])
  return cov_matrix

cov_matrix = calculate_cov_matrix(variance_1,variance_2,corr_coef)
X = get_data(cov_matrix)

with plt.xkcd():
  plot_data(X)

```

# Define a new orthonormal basis

```python
#@title Video: Orthonormal bases
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="dK526Nbn2Xo", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=dK526Nbn2Xo

Next, we will define a new orthonormal basis of vectors ${\bf u} = [u_1,u_2]$ and ${\bf w} = [w_1,w_2]$. As we learned in the video, two vectors are orthonormal if:

1.   They are orthogonal (i.e., their dot product is zero):
\begin{equation}
{\bf u\cdot w} = u_1 w_1 + u_2 w_2 = 0
\end{equation}
2.   They have unit length:
\begin{equation}
||{\bf u} || = ||{\bf w} || = 1
\end{equation}

In two dimensions, it is easy to make an arbitrary orthonormal basis. All we need is a random vector ${\bf u}$, which we have normalized. If we now define the second basis vector to be ${\bf w} = [-u_2,u_1]$, we can check that both conditions are satisfied:
\begin{equation}
{\bf u\cdot w} = - u_1 u_2 + u_2 u_1 = 0
\end{equation}
and
\begin{equation}
{|| {\bf w} ||} = \sqrt{(-u_2)^2 + u_1^2} = \sqrt{u_1^2 + u_2^2} = 1,
\end{equation}
where we used the fact that ${\bf u}$ is normalized. So, with an arbitrary input vector, we can define an orthonormal basis, which we will write in matrix by stacking the basis vectors horizontally:

\begin{equation}
{{\bf W} } =
\begin{pmatrix}
 u_1 & w_1 \\
 u_2 & w_2
\end{pmatrix}.
\end{equation}

### Exercise

In this exercise you will fill in the function below to define an orthonormal basis, given a single arbitrary 2-dimensional vector as an input.

**Suggestions**
* Modify the function `define_orthonormal_basis` to first normalize the first basis vector $\bf u$.
* Then complete the function by finding a basis vector $\bf w$ that is orthogonal to $\bf u$.
* Test the function using initial basis vector ${\bf u} = [3,1]$. Plot the resulting basis vectors on top of the data scatter plot using the function `plot_basis_vectors`. (For the data, use  $\sigma_1^2 =1$, $\sigma_1^2 =1$, and $\rho = .8$).

```python
help(plot_basis_vectors)
```

    Help on function plot_basis_vectors in module __main__:

    plot_basis_vectors(X, W)
        Plots bivariate data as well as new basis vectors.

        Args:
          X (numpy array of floats):   Data matrix
                                        each column corresponds to a different random variable
          W (numpy array of floats):   Square matrix representing new orthonormal basis
                                        each column represents a basis vector

        Returns:
          Nothing.

```python
def define_orthonormal_basis(u):
  """
  Calculates an orthonormal basis given an arbitrary vector u.

  Args:
    u (numpy array of floats):    arbitrary 2-dimensional vector used for new basis

  Returns:
    (numpy array of floats) : new orthonormal basis
                                columns correspond to basis vectors
  """
  ###################################################################
  ## Insert your code here to:
  ##                normalize vector u
  ##                calculate vector w that is orthogonal to w

  #u = ....
  #w = ...
  #W = np.column_stack((u,w))

  #comment this once you've filled the function
  raise NotImplementedError("Student excercise: implement the orthonormal basis function")
  ###################################################################
  return W

variance_1 = 1
variance_2 = 1
corr_coef = 0.8

cov_matrix = calculate_cov_matrix(variance_1,variance_2,corr_coef)
X = get_data(cov_matrix)
u = np.array([3,1])

#uncomment and run below to plot the basis vectors
##define_orthonormal_basis(u)
#plot_basis_vectors(X,W)
```

```python
# to_remove solution

def define_orthonormal_basis(u):
  """
  Calculates an orthonormal basis given an arbitrary vector u.

  Args:
    u (numpy array of floats):    arbitrary 2-dimensional vector used for new basis

  Returns:
    (numpy array of floats) : new orthonormal basis
                                columns correspond to basis vectors
  """
  u = u / np.sqrt(u[0]**2 + u[1]**2)
  w = np.array([-u[1],u[0]])
  W = np.column_stack((u,w))
  return W

W = define_orthonormal_basis(u)

with plt.xkcd():
  plot_basis_vectors(X,W)
```

# Project data onto new basis

```python
#@title Video: Change of basis
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="5MWSUtpbSt0", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=5MWSUtpbSt0

Finally, we will express our data in the new basis that we have just found. Since $\bf W$ is orthonormal, we can project the data into our new basis using simple matrix multiplication :

\begin{equation}
{\bf Y = X W}.
\end{equation}

We will explore the geometry of the transformed data $\bf Y$ as we vary the choice of basis.

#### Exercise
In this exercise you will fill in the function below to define an orthonormal basis, given a single arbitrary vector as an input.

**Suggestions**
* Complete the function `change_of_basis` to project the data onto the new basis.
* Plot the projected data using the function `plot_data_new_basis`.
* What happens to the correlation coefficient in the new basis? Does it increase or decrease?
* What happens to variance?

```python
def change_of_basis(X,W):
  """
  Projects data onto new basis W.

  Args:
    X (numpy array of floats) : Data matrix
                                each column corresponding to a different random variable
    W (numpy array of floats):  new orthonormal basis
                                columns correspond to basis vectors

  Returns:
    (numpy array of floats) : Data matrix expressed in new basis
  """
  ###################################################################
  ## Insert your code here to:
  ##                project data onto new basis described by W

  #Y = ...
  #comment this once you've filled the function
  raise NotImplementedError("Student excercise: implement change of basis")
  ###################################################################
  return Y

## Unomment below to transform the data by projecting it into the new basis
## Plot the projected data
# Y = change_of_basis(X,W)
# plot_data_new_basis(Y)
# disp(...)
```

```python
# to_remove solution
def change_of_basis(X,W):
  """
  Projects data onto a new basis.

  Args:
    X (numpy array of floats) : Data matrix
                                each column corresponding to a different random variable
    W (numpy array of floats):  new orthonormal basis
                                columns correspond to basis vectors

  Returns:
    (numpy array of floats) : Data matrix expressed in new basis
  """
  Y = np.matmul(X,W)
  return Y

Y = change_of_basis(X,W)

with plt.xkcd():
  plot_data_new_basis(Y)

```

#### Exercise
To see what happens to the correlation as we change the basis vectors, run the cell below. The parameter $\theta$ controls the angle of $\bf u$ in degrees. Use the slider to rotate the basis vectors.

**Questions**
* What happens to the projected data as you rotate the basis?
* How does the correlation coefficient change? How does the variance of the projection onto each basis vector change?
* Are you able to find a basis in which the projected data is uncorrelated?

```python
###### MAKE SURE TO RUN THIS CELL VIA THE PLAY BUTTON TO ENABLE SLIDERS ########

import ipywidgets as widgets

def refresh(theta = 0):
  u = [1,np.tan(theta * np.pi/180.)]
  W = define_orthonormal_basis(u)
  Y = change_of_basis(X,W)
  plot_basis_vectors(X,W)
  plot_data_new_basis(Y)

_ = widgets.interact(refresh,
  theta = (0, 90, 5))
```


# 核方法

核方法往简单地说，就是将一个低维的线性不可分的数据映射到一个高维的空间、并期望映射后的数据在高维空间里是线性可分的。我们以异或数据集为例：在二维空间中、异或数据集是线性不可分的；但是通过将其映射到三维空间、我们可以非常简单地让其在三维空间中变得线性可分。比如定义映射：

$$
\phi(x,y)=\left\{
\begin{aligned}
&(x,y,1),\ \ xy>0 \\
&(x,y,0),\ \ xy\le0
\end{aligned}
\right.
$$

该映射的效果如下图所示：

可以看到，虽然左图的数据集线性不可分、但显然右图的数据集是线性可分的，这就是核方法工作原理的一个不太严谨但仍然合理的解释

应用核方法的思路如下：
+ 将算法表述成样本点内积的组合（这经常能通过算法的对偶形式实现）
+ 设法找到核函数$K(xi,xj)$，它能返回样本点$x_i$、$x_j$被$\phi$作用后的内积
+ 用$K(xi,xj)$替换$xi\cdot xj$、完成低维到高维的映射（同时也完成了从线性算法到非线性算法的转换）

## 核感知机

感知机的原始损失函数为

$$
L(D) = \sum_{i=1}^N\left[ -y_i(w\cdot x_i+b)\right]_+
$$

为让损失函数中只含样本的内积，考虑令

$$
w = \sum_{i=1}^N\alpha_ix_i
$$

则

$$
\begin{align}
L(D) &= \sum_{i=1}^N\left[ -y_i\left[\left(\sum_{j=1}^N\alpha_jx_j\right)\cdot x_i+b\right]\right]_+ \\
&= \sum_{i=1}^N\left[ -y_i\left(\sum_{j=1}^N\alpha_j(x_i\cdot x_j)+b\right)\right]_+
\end{align}
$$

在此之上应用核方法是平凡的：设核函数为$K$，只需把所有的$x_i\cdot x_j$换成$K(x_i,x_j)$即可：

$$
L(D) = \sum_{i=1}^N\left[ -y_i\left(\sum_{j=1}^N\alpha_jK(x_i, x_j)+b\right)\right]_+
$$

于是优化问题变为

$$
\min_{\alpha}\sum_{i=1}^N\left[ -y_i\left(\sum_{j=1}^N\alpha_jK(x_i, x_j)+b\right)\right]_+
$$

易知，我们仍然可以用梯度下降法来求解该问题

```python
import numpy as np

class KP:
    def __init__(self):
        self._x = None
        self._alpha = self._b = self._kernel = None

    # 定义多项式核函数
    @staticmethod
    def _poly(x, y, p=4):
        return (x.dot(y.T) + 1) ** p

    # 定义 rbf 核函数
    @staticmethod
    def _rbf(x, y, gamma):
        return np.exp(-gamma * np.sum((x[..., None, :] - y) ** 2, axis=2))

    def fit(self, x, y, kernel="poly", p=None, gamma=None, lr=0.001, batch_size=128, epoch=10000):
        x, y = np.asarray(x, np.float32), np.asarray(y, np.float32)
        if kernel == "poly":
            p = 4 if p is None else p
            self._kernel = lambda x_, y_: self._poly(x_, y_, p)
        elif kernel == "rbf":
            gamma = 1 / x.shape[1] if gamma is None else gamma
            self._kernel = lambda x_, y_: self._rbf(x_, y_, gamma)
        else:
            raise NotImplementedError("Kernel '{}' has not defined".format(kernel))
        self._alpha = np.zeros(len(x))
        self._b = 0.
        self._x = x
        k_mat = self._kernel(x, x)
        for _ in range(epoch):
            indices = np.random.permutation(len(y))[:batch_size]
            k_mat_batch, y_batch = k_mat[indices], y[indices]
            err = -y_batch * (k_mat_batch.dot(self._alpha) + self._b)
            if np.max(err) < 0:
                continue
            mask = err >= 0
            delta = lr * y_batch[mask]
            self._alpha += np.sum(delta[..., None] * k_mat_batch[mask], axis=0)
            self._b += np.sum(delta)

    def predict(self, x, raw=False):
        x = np.atleast_2d(x).astype(np.float32)
        k_mat = self._kernel(self._x, x)
        y_pred = self._alpha.dot(k_mat) + self._b
        if raw:
            return y_pred
        return np.sign(y_pred).astype(np.float32)
```

### 进行简单的测试

```python
from Util import gen_two_clusters, visualize2d

xc, yc = gen_two_clusters()
kp = KP()
kp.fit(xc, yc, p=1)
print("准确率：{:8.6} %".format((kp.predict(xc) == yc).mean() * 100))
visualize2d(kp, xc, yc, True)
```

### 在非线性数据集上进行测试

```python
from Util import gen_spiral

xs, ys = gen_spiral()
kp = KP()
kp.fit(xs, ys, p=8)
print("准确率：{:8.6} %".format((kp.predict(xs) == ys).mean() * 100))
visualize2d(kp, xs, ys, True)
```

## 核 SVM

用同样的手法不难得出 LinearSVM 的核形式：

$$
L(D)=\frac12\sum_{i=1}^N\sum_{j=1}^N\alpha_i\alpha_jK(x_i,x_j)+C\sum_{i=1}^N\left[ 1-y_i\left(\sum_{j=1}^N\alpha_jK(x_i, x_j)+b\right)\right]_+
$$

可知求导仍然并不困难

```python
class SVM(KP):
    def fit(self, x, y, kernel="rbf", p=None, gamma=None, c=1, lr=0.0001, batch_size=128, epoch=10000):
        x, y = np.asarray(x, np.float32), np.asarray(y, np.float32)
        if kernel == "poly":
            p = 4 if p is None else p
            self._kernel = lambda x_, y_: self._poly(x_, y_, p)
        elif kernel == "rbf":
            gamma = 1 / x.shape[1] if gamma is None else gamma
            self._kernel = lambda x_, y_: self._rbf(x_, y_, gamma)
        else:
            raise NotImplementedError("Kernel '{}' has not defined".format(kernel))
        self._alpha = np.zeros(len(x))
        self._b = 0.
        self._x = x
        k_mat = self._kernel(x, x)
        k_mat_diag = np.diag(k_mat)
        for _ in range(epoch):
            self._alpha -= lr * (np.sum(self._alpha * k_mat, axis=1) + self._alpha * k_mat_diag) * 0.5
            indices = np.random.permutation(len(y))[:batch_size]
            k_mat_batch, y_batch = k_mat[indices], y[indices]
            err = 1 - y_batch * (k_mat_batch.dot(self._alpha) + self._b)
            if np.max(err) <= 0:
                continue
            mask = err > 0
            delta = c * lr * y_batch[mask]
            self._alpha += np.sum(delta[..., None] * k_mat_batch[mask], axis=0)
            self._b += np.sum(delta)
```

### 进行简单的测试

```python
svm = SVM()
svm.fit(xc, yc)
print("准确率：{:8.6} %".format((svm.predict(xc) == yc).mean() * 100))
visualize2d(svm, xc, yc, True)
```

### 在非线性数据集上进行测试

```python
svm = SVM()
svm.fit(xs, ys)
print("准确率：{:8.6} %".format((svm.predict(xs) == ys).mean() * 100))
visualize2d(svm, xs, ys, True)
```

### 和 sklearn 的 SVM 进行比较

```python
from sklearn.svm import SVC

svm = SVC()
svm.fit(xc, yc)
visualize2d(svm, xc, yc, True)
print("准确率：{:8.6} %".format((svm.predict(xc) == yc).mean() * 100))
svm.fit(xs, ys)
visualize2d(svm, xs, ys, True)
print("准确率：{:8.6} %".format((svm.predict(xs) == ys).mean() * 100))
```

孰优孰劣的话有点见仁见智的感觉。当然了，这只是两个非常简单的数据集，在真实数据集上，`sklearn`的`SVM`会比我们实现的`SVM`要好很多