## 朴素贝叶斯 ##

朴素贝叶斯方法是一组基于贝叶斯定理的有监督学习算法，其“朴素”假设是给定类变量值的每一对特征之间存在条件独立性。贝叶斯定理给出了给定类变量$y$与相关特征向量$x_1$到$x_n$的关系:
$$P(y \mid x_1, \dots, x_n) = \frac{P(y) P(x_1, \dots x_n \mid y)}
                                 {P(x_1, \dots, x_n)}$$
使用朴素条件独立假设:
$$P(x_i | y, x_1, \dots, x_{i-1}, x_{i+1}, \dots, x_n) = P(x_i | y)$$
对于所有$i$，这个关系被简化为：
$$
\begin{align}\begin{aligned}P(y \mid x_1, \dots, x_n) \propto P(y) \prod_{i=1}^{n} P(x_i \mid y)\\\Downarrow\\\hat{y} = \arg\max_y P(y) \prod_{i=1}^{n} P(x_i \mid y),\end{aligned}\end{align}
$$
我们可以使用极大后验(MAP)估计来估计$P(y)$和$P(x_i\mid y)$;前者则为训练集中y类的相对频率。不同的朴素贝叶斯分类器的差异主要在于它们对$P(x_i\mid y)$分布的假设不同。

尽管朴素贝叶斯分类器的假设明显过于简化，但它们在许多实际情况下都能很好地工作，比如著名的文档分类和垃圾邮件过滤。它们需要少量的训练数据来估计必要的参数。

事实上,概率模型的训练过程就是参数估计(parameter estimation)过程。对于参数估计,统计学界的两个学派分别提供了不同的解决方案:频率主义学派(Frequentist)认为参数虽然未知,但却是客观存在的固定值,因此,可通过优化似然函数等准则来确定参数值;贝叶斯学派(Bayesian)则认为参数是未观察到的随机变量,其本身也可有分布,因此,可假定参数服从一个先验分布,然后基于观测到的数据来计算参数的后验分布.本节介绍源自频率主义学派的极大似然估计(Maximum Likelihood Estimation,简称MLE),这是根据数据采样来估计概率分布参数的经典方法.

### NB的优点和缺点 ###
与更复杂的方法相比，朴素贝叶斯学习器和分类器可以非常快。类条件特征分布的解耦意味着每个分布都可以独立地估计为一维分布。这反过来又有助于缓解来自维度诅咒的问题。

另一方面，尽管朴素贝叶斯被认为是一个不错的分类器，但它是一个糟糕的估计器，所以来自*predict_proba*的概率输出不需要太认真对待。

### 朴素高斯贝叶斯 ###
假设类条件概率服从高斯分布，其概率密度函数为： $$P(x_i | y_k)=\frac{1}{\sqrt{2\pi\sigma^2_{y_k}}}exp(-\frac{(x_i-\mu_{y_k})^2}{2\sigma^2_{y_k}})$$ 数学期望(mean)：$\mu$，方差：$\sigma^2=\frac{\sum(X-\mu)^2}{N}$, 可以利用MLE对$\mu$和$\sigma^2$进行参数估计。下面展示了该模型的实现代码：

```
#Compute the log of the sum of exponentials of input elements
from scipy.special import  logsumexp
#View inputs as arrays with at least two dimensions.
#np.atleast_2d
from sklearn import datasets
#from sklearn.naive_bayes import GaussianNB
import numpy as np
from abc import ABCMeta, abstractmethod
```

```
class BaseNB(metaclass=ABCMeta):

    @abstractmethod
    def _joint_log_likelihood(self, X):
        """
        计算没有被归一化的后验概率的对数 I.e. ``log P(c) + log P(x|c)``, 右边分子部分
        Returns: [n_sample, n_classes]
        """
        pass
    def predict(self, X):
        """
        X : array-like, shape = [n_samples, n_features]
        """
        jll = self._joint_log_likelihood(X)
        return self.classes_[np.argmax(jll, axis=1)]

    def predict_log_proba(self, X):
        """
        X : array-like, shape = [n_samples, n_features]
        Returns : array-like, shape = [n_samples, n_classes]
        """
        jll = self._joint_log_likelihood(X)
        log_prob_x = logsumexp(jll, axis=1)
        return jll - np.atleast_2d(log_prob_x).T

    def predict_proba(self, X):
        """
        X : array-like, shape = [n_samples, n_features]
        Returns : array-like, shape = [n_samples, n_classes]
        """
        return np.exp(self.predict_log_proba(X))
```

```
class GaussianNB(BaseNB):
    def __init__(self, priors=None, var_smoothing=1e-9):
        self.priors = priors
        self.var_smoothing = var_smoothing
    def _get_mean_variance(self, X, sample_weight=None):
        # Compute (potentially weighted) mean and variance of new datapoints
        if sample_weight is not None:
            n_new = float(sample_weight.sum())
            new_mu = np.average(X, axis=0, weights=sample_weight / n_new)
            new_var = np.average((X - new_mu) ** 2, axis=0,
                                 weights=sample_weight / n_new)
        else:
            n_new = X.shape[0]
            new_var = np.var(X, axis=0)
            new_mu = np.mean(X, axis=0)

        return new_mu, new_var
    def fit(self, X, y, sample_weight=None):
        # If the ratio of data variance between dimensions is too small, it
        # will cause numerical errors. To address this, we artificially
        # boost the variance by epsilon, a small fraction of the standard
        # deviation of the largest dimension.
        self.epsilon_ = self.var_smoothing * np.var(X, axis=0).max()
        self.classes_ = np.unique(y)
        n_features = X.shape[1]
        n_classes = len(self.classes_)
        self.theta_ = np.zeros((n_classes, n_features))
        self.sigma_ = np.zeros((n_classes, n_features))
        self.class_count_ = np.zeros(n_classes, dtype=np.float64)

        for (i, y_i) in enumerate(self.classes_):
            X_i = X[y == y_i, :]
            if sample_weight is not None:
                sw_i = sample_weight[y == y_i]
                N_i = sw_i.sum()
            else:
                sw_i = None
                N_i = X_i.shape[0]
            self.theta_[i, :], self.sigma_[i, :] = self._get_mean_variance(X_i, sw_i)
            self.class_count_[i] += N_i
        self.sigma_[:, :] += self.epsilon_

        # Set priors
        if self.priors:
            self.class_prior_ = np.asarray(self.priors)
        else:
            # Empirical prior, with sample_weight taken into account
            self.class_prior_ = self.class_count_ / self.class_count_.sum()
        return self

    def _joint_log_likelihood(self, X):
        joint_log_likelihood = []
        for i in range(np.size(self.classes_)):
            jointi = np.log(self.class_prior_[i])
            n_ij = - 0.5 * np.sum(np.log(2. * np.pi * self.sigma_[i, :]))
            n_ij -= 0.5 * np.sum(((X - self.theta_[i, :]) ** 2) /
                                 (self.sigma_[i, :]), 1)
            joint_log_likelihood.append(jointi + n_ij)

        joint_log_likelihood = np.array(joint_log_likelihood).T
        return joint_log_likelihood
```

```
iris = datasets.load_iris()
gnb = GaussianNB(priors=[.3, .2, .5])
gnb.fit(iris.data, iris.target)
y_pred = gnb.fit(iris.data, iris.target).predict(iris.data)
#gnb.predict_proba(iris.data)
(y_pred == iris.target).sum() / len(iris.target)
```

    0.95333333333333337

## SVM ##

训练数据集
\begin{align*} \\& T = \left\{ \left( x_{1}, y_{1} \right), \left( x_{2}, y_{2} \right), \cdots, \left( x_{N}, y_{N} \right) \right\} \end{align*}
其中，$x_{i} \in \mathcal{X} = R^{n}, y_{i} \in \mathcal{Y} = \left\{ +1, -1 \right\}, i = 1, 2, \cdots, N$，$x_{i}$为第$i$个特征向量（实例），$y_{i}$为第$x_{i}$的类标记，当$y_{i}=+1$时，称$x_{i}$为正例；当$y_{i}= -1$时，称$x_{i}$为负例，$\left( x_{i}, y_{i} \right)$称为样本点。

### 线性可分支持向量机（硬间隔支持向量机）###
给定线性可分训练数据集，通过间隔最大化或等价地求解相应地凸二次规划问题学习得到分离超平面为
\begin{align*} \\& w^{*} \cdot x + b^{*} = 0 \end{align*}
以及相应的分类决策函数
\begin{align*} \\& f \left( x \right) = sign \left( w^{*} \cdot x + b^{*} \right)  \end{align*}
称为线型可分支持向量机。

超平面$\left( w, b \right)$关于样本点$\left( x_{i}, y_{i} \right)$的函数间隔为
\begin{align*} \\& \hat \gamma_{i} = y_{i} \left( w \cdot x_{i} + b \right) \end{align*}

超平面$\left( w, b \right)$关于训练集$T$的函数间隔
\begin{align*} \\& \hat \gamma = \min_{i = 1, 2, \cdots, N} \hat \gamma_{i} \end{align*}
即超平面$\left( w, b \right)$关于训练集$T$中所有样本点$\left( x_{i}, y_{i} \right)$的函数间隔的最小值。

超平面$\left( w, b \right)$关于样本点$\left( x_{i}, y_{i} \right)$的几何间隔为
 \begin{align*} \\& \gamma_{i} = y_{i} \left( \dfrac{w}{\| w \|} \cdot x_{i} + \dfrac{b}{\| w \|} \right) \end{align*}

超平面$\left( w, b \right)$关于训练集$T$的几何间隔
\begin{align*} \\& \gamma = \min_{i = 1, 2, \cdots, N} \gamma_{i} \end{align*}
即超平面$\left( w, b \right)$关于训练集$T$中所有样本点$\left( x_{i}, y_{i} \right)$的几何间隔的最小值。

函数间隔和几何间隔的关系
\begin{align*} \\& \gamma_{i} = \dfrac{\hat \gamma_{i}}{\| w \|}
\\& \gamma = \dfrac{\hat \gamma}{\| w \|} \end{align*}

最大间隔分离超平面等价为求解
\begin{align*} \\& \max_{w,b} \quad \gamma
\\ & s.t. \quad y_{i} \left( \dfrac{w}{\| w \|} \cdot x_{i} + \dfrac{b}{\| w \|} \right) \geq \gamma, \quad i=1,2, \cdots, N \end{align*}
等价的
\begin{align*}  \\ & \max_{w,b} \quad \dfrac{\hat \gamma}{\| w \|}
\\ & s.t. \quad y_{i} \left( w \cdot x_{i} + b \right) \geq \hat \gamma, \quad i=1,2, \cdots, N \end{align*}
等价的
\begin{align*}  \\ & \min_{w,b} \quad \dfrac{1}{2} \| w \|^{2}
\\ & s.t. \quad y_{i} \left( w \cdot x_{i} + b \right) -1 \geq 0, \quad i=1,2, \cdots, N \end{align*}

线性可分支持向量机学习算法（最大间隔法）：
输入：线性可分训练数据集$T = \left\{ \left( x_{1}, y_{1} \right), \left( x_{2}, y_{2} \right), \cdots, \left( x_{N}, y_{N} \right) \right\}$，其中$x_{i} \in \mathcal{X} = R^{n}, y_{i} \in \mathcal{Y} = \left\{ +1, -1 \right\}, i = 1, 2, \cdots, N$
输出：最大间隔分离超平面和分类决策函数
1. 构建并求解约束最优化问题
\begin{align*}  \\ & \min_{w,b} \quad \dfrac{1}{2} \| w \|^{2}
\\ & s.t. \quad y_{i} \left( w \cdot x_{i} + b \right) -1 \geq 0, \quad i=1,2, \cdots, N \end{align*}
求得最优解$w^{*}, b^{*}$。
2. 得到分离超平面
\begin{align*}  \\ & w^{*} \cdot x + b^{*} = 0 \end{align*}
以及分类决策函数
\begin{align*} \\& f \left( x \right) = sign \left( w^{*} \cdot x + b^{*} \right)  \end{align*}

（硬间隔）支持向量：训练数据集的样本点中与分离超平面距离最近的样本点的实例，即使约束条件等号成立的样本点
\begin{align*} \\ & y_{i} \left( w \cdot x_{i} + b \right) -1 = 0 \end{align*}
对$y_{i} = +1$的正例点，支持向量在超平面
\begin{align*} \\ & H_{1}:w \cdot x + b = 1  \end{align*}
对$y_{i} = -1$的正例点，支持向量在超平面
\begin{align*} \\ & H_{1}:w \cdot x + b = -1  \end{align*}
$H_{1}$和$H_{2}$称为间隔边界。
$H_{1}$和$H_{2}$之间的距离称为间隔，且$|H_{1}H_{2}| = \dfrac{1}{\| w \|} + \dfrac{1}{\| w \|} = \dfrac{2}{\| w \|}$。

最优化问题的求解：
1. 引入拉格朗日乘子$\alpha_{i} \geq 0, i = 1, 2, \cdots, N$构建拉格朗日函数
\begin{align*} \\ & L \left( w, b, \alpha \right) =  \dfrac{1}{2} \| w \|^{2} + \sum_{i=1}^{N} \alpha_{i} \left[- y_{i} \left( w \cdot x_{i} + b \right) + 1  \right]
\\ & = \dfrac{1}{2} \| w \|^{2} - \sum_{i=1}^{N} \alpha_{i}  y_{i} \left( w \cdot x_{i} + b \right) + \sum_{i=1}^{N} \alpha_{i}  \end{align*}
其中，$\alpha = \left( \alpha_{1}, \alpha_{2}, \cdots, \alpha_{N} \right)^{T}$为拉格朗日乘子向量。
2. 求$\min_{w,b}L \left( w, b, \alpha \right)$:
\begin{align*} \\ &  \nabla _{w} L \left( w, b, \alpha \right) = w - \sum_{i=1}^{N} \alpha_{i} y_{i} x_{i} = 0
\\ &  \nabla _{b} L \left( w, b, \alpha \right) = -\sum_{i=1}^{N} \alpha_{i} y_{i} = 0 \end{align*}
得
\begin{align*} \\ &   w ＝ \sum_{i=1}^{N} \alpha_{i} y_{i} x_{i}
\\ &  \sum_{i=1}^{N} \alpha_{i} y_{i} = 0 \end{align*}
代入拉格朗日函数，得
\begin{align*} \\ & L \left( w, b, \alpha \right) =  \dfrac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_{i} \alpha_{j} y_{i} y_{j} \left( x_{i} \cdot x_{j} \right) - \sum_{i=1}^{N} \alpha_{i}  y_{i} \left[ \left( \sum_{j=1}^{N} \alpha_{j} y_{j} x_{j} \right) \cdot x_{i} + b \right] + \sum_{i=1}^{N} \alpha_{i}
\\ & = - \dfrac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_{i} \alpha_{j} y_{i} y_{j} \left( x_{i} \cdot x_{j} \right) - \sum_{i=1}^{N} \alpha_{i} y_{i} b + \sum_{i=1}^{N} \alpha_{i}
\\ & = - \dfrac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_{i} \alpha_{j} y_{i} y_{j} \left( x_{i} \cdot x_{j} \right)  + \sum_{i=1}^{N} \alpha_{i} \end{align*}
即
\begin{align*} \\ & \min_{w,b}L \left( w, b, \alpha \right) = - \dfrac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_{i} \alpha_{j} y_{i} y_{j} \left( x_{i} \cdot x_{j} \right)  + \sum_{i=1}^{N} \alpha_{i}    \end{align*}
3.求$\max_{\alpha} \min_{w,b}L \left( w, b, \alpha \right)$:
\begin{align*} \\ &  \max_{\alpha} - \dfrac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_{i} \alpha_{j} y_{i} y_{j} \left( x_{i} \cdot x_{j} \right)  + \sum_{i=1}^{N} \alpha_{i}
\\ & s.t. \sum_{i=1}^{N} \alpha_{i} y_{i} = 0
\\ & \alpha_{i} \geq 0, \quad i=1,2, \cdots, N  \end{align*}
等价的
\begin{align*} \\ &  \min_{\alpha} \dfrac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_{i} \alpha_{j} y_{i} y_{j} \left( x_{i} \cdot x_{j} \right)  - \sum_{i=1}^{N} \alpha_{i}
\\ & s.t. \sum_{i=1}^{N} \alpha_{i} y_{i} = 0
\\ & \alpha_{i} \geq 0, \quad i=1,2, \cdots, N  \end{align*}

线性可分支持向量机（硬间隔支持向量机）学习算法：
输入：线性可分训练数据集$T = \left\{ \left( x_{1}, y_{1} \right), \left( x_{2}, y_{2} \right), \cdots, \left( x_{N}, y_{N} \right) \right\}$，其中$x_{i} \in \mathcal{X} = R^{n}, y_{i} \in \mathcal{Y} = \left\{ +1, -1 \right\}, i = 1, 2, \cdots, N$
输出：最大间隔分离超平面和分类决策函数
1. 构建并求解约束最优化问题
\begin{align*} \\ &  \min_{\alpha} \dfrac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_{i} \alpha_{j} y_{i} y_{j} \left( x_{i} \cdot x_{j} \right)  - \sum_{i=1}^{N} \alpha_{i}
\\ & s.t. \sum_{i=1}^{N} \alpha_{i} y_{i} = 0
\\ & \alpha_{i} \geq 0, \quad i=1,2, \cdots, N  \end{align*}
求得最优解$\alpha^{*} = \left( \alpha_{1}^{*}, \alpha_{1}^{*}, \cdots, \alpha_{N}^{*} \right) $。
2. 计算
\begin{align*}  \\ & w^{*} = \sum_{i=1}^{N} \alpha_{i}^{*} y_{i} x_{i} \end{align*}
并选择$\alpha^{*}$的一个正分量$\alpha_{j}^{*} > 0$，计算
\begin{align*}  \\ & b^{*} = y_{j} - \sum_{i=1}^{N} \alpha_{i}^{*} y_{i} \left( x_{i} \cdot x_{j} \right) \end{align*}
3. 得到分离超平面
\begin{align*}  \\ & w^{*} \cdot x + b^{*} = 0 \end{align*}
以及分类决策函数
\begin{align*} \\& f \left( x \right) = sign \left( w^{*} \cdot x + b^{*} \right)  \end{align*}

线性支持向量机（软间隔支持向量机）：给定线性不可分训练数据集，通过求解凸二次规划问题
\begin{align*}  \\ & \min_{w,b,\xi} \quad \dfrac{1}{2} \| w \|^{2} + C \sum_{i=1}^{N} \xi_{i}
\\ & s.t. \quad y_{i} \left( w \cdot x_{i} + b \right) \geq 1 - \xi_{i}
\\ & \xi_{i} \geq 0, \quad i=1,2, \cdots, N \end{align*}
学习得到分离超平面为
\begin{align*} \\& w^{*} \cdot x + b^{*} = 0 \end{align*}
以及相应的分类决策函数
\begin{align*} \\& f \left( x \right) = sign \left( w^{*} \cdot x + b^{*} \right)  \end{align*}
称为线型支持向量机。

最优化问题的求解：
1. 引入拉格朗日乘子$\alpha_{i} \geq 0, \mu_{i} \geq 0, i = 1, 2, \cdots, N$构建拉格朗日函数
\begin{align*} \\ & L \left( w, b, \xi, \alpha, \mu \right) =  \dfrac{1}{2} \| w \|^{2} + C \sum_{i=1}^{N} \xi_{i} + \sum_{i=1}^{N} \alpha_{i} \left[- y_{i} \left( w \cdot x_{i} + b \right) + 1 - \xi_{i}  \right] + \sum_{i=1}^{N} \mu_{i} \left( -\xi_{i} \right)
\\ & = \dfrac{1}{2} \| w \|^{2} + C \sum_{i=1}^{N} \xi_{i} - \sum_{i=1}^{N} \alpha_{i} \left[ y_{i} \left( w \cdot x_{i} + b \right) -1 + \xi_{i} \right] - \sum_{i=1}^{N} \mu_{i} \xi_{i}  \end{align*}
其中，$\alpha = \left( \alpha_{1}, \alpha_{2}, \cdots, \alpha_{N} \right)^{T}$以及$\mu = \left( \mu_{1}, \mu_{2}, \cdots, \mu_{N} \right)^{T}$为拉格朗日乘子向量。
2. 求$\min_{w,b}L \left( w, b, \xi, \alpha, \mu \right)$:
\begin{align*} \\ &  \nabla_{w} L \left( w, b, \xi, \alpha, \mu \right) = w - \sum_{i=1}^{N} \alpha_{i} y_{i} x_{i} = 0
\\ &  \nabla_{b} L \left( w, b, \xi, \alpha, \mu \right) = -\sum_{i=1}^{N} \alpha_{i} y_{i} = 0
\\ & \nabla_{\xi_{i}} L \left( w, b, \xi, \alpha, \mu \right) = C - \alpha_{i} - \mu_{i} = 0 \end{align*}
得
\begin{align*} \\ &   w ＝ \sum_{i=1}^{N} \alpha_{i} y_{i} x_{i}
\\ &  \sum_{i=1}^{N} \alpha_{i} y_{i} = 0
\\ & C - \alpha_{i} - \mu_{i} = 0\end{align*}
代入拉格朗日函数，得
\begin{align*} \\ & L \left( w, b, \xi, \alpha, \mu \right) =  \dfrac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_{i} \alpha_{j} y_{i} y_{j} \left( x_{i} \cdot x_{j} \right) + C \sum_{i=1}^{N} \xi_{i} - \sum_{i=1}^{N} \alpha_{i}  y_{i} \left[ \left( \sum_{j=1}^{N} \alpha_{j} y_{j} x_{j} \right) \cdot x_{i} + b \right]
\\ & \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad + \sum_{i=1}^{N} \alpha_{i} - \sum_{i=1}^{N} \alpha_{i} \xi_{i} - \sum_{i}^{N} \mu_{i} \xi_{i}
\\ & = - \dfrac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_{i} \alpha_{j} y_{i} y_{j} \left( x_{i} \cdot x_{j} \right) - \sum_{i=1}^{N} \alpha_{i} y_{i} b + \sum_{i=1}^{N} \alpha_{i} + \sum_{i=1}^{N} \xi_{i} \left( C - \alpha_{i} - \mu_{i}  \right)
\\ & = - \dfrac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_{i} \alpha_{j} y_{i} y_{j} \left( x_{i} \cdot x_{j} \right)  + \sum_{i=1}^{N} \alpha_{i} \end{align*}
即
\begin{align*} \\ & \min_{w,b,\xi}L \left( w, b, \xi, \alpha, \mu \right) = - \dfrac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_{i} \alpha_{j} y_{i} y_{j} \left( x_{i} \cdot x_{j} \right)  + \sum_{i=1}^{N} \alpha_{i}    \end{align*}
3.求$\max_{\alpha} \min_{w,b, \xi}L \left( w, b, \xi, \alpha, \mu \right)$:
\begin{align*} \\ &  \max_{\alpha} - \dfrac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_{i} \alpha_{j} y_{i} y_{j} \left( x_{i} \cdot x_{j} \right)  + \sum_{i=1}^{N} \alpha_{i}
\\ & s.t. \sum_{i=1}^{N} \alpha_{i} y_{i} = 0
\\ & C - \alpha_{i} - \mu_{i} = 0
\\ & \alpha_{i} \geq 0
\\ & \mu_{i} \geq 0, \quad i=1,2, \cdots, N  \end{align*}
等价的
\begin{align*} \\ &  \min_{\alpha} \dfrac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_{i} \alpha_{j} y_{i} y_{j} \left( x_{i} \cdot x_{j} \right)  - \sum_{i=1}^{N} \alpha_{i}
\\ & s.t. \sum_{i=1}^{N} \alpha_{i} y_{i} = 0
\\ & 0 \leq \alpha_{i} \leq C , \quad i=1,2, \cdots, N  \end{align*}

### 线性支持向量机（软间隔支持向量机）学习算法 ###
输入：训练数据集$T = \left\{ \left( x_{1}, y_{1} \right), \left( x_{2}, y_{2} \right), \cdots, \left( x_{N}, y_{N} \right) \right\}$，其中$x_{i} \in \mathcal{X} = R^{n}, y_{i} \in \mathcal{Y} = \left\{ +1, -1 \right\}, i = 1, 2, \cdots, N$
输出：最大间隔分离超平面和分类决策函数
1. 选择惩罚参数$C \geq 0$，构建并求解约束最优化问题
\begin{align*} \\ &  \min_{\alpha} \dfrac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_{i} \alpha_{j} y_{i} y_{j} \left( x_{i} \cdot x_{j} \right)  - \sum_{i=1}^{N} \alpha_{i}
\\ & s.t. \sum_{i=1}^{N} \alpha_{i} y_{i} = 0
\\ & 0 \leq \alpha_{i} \leq C , \quad i=1,2, \cdots, N  \end{align*}
求得最优解$\alpha^{*} = \left( \alpha_{1}^{*}, \alpha_{1}^{*}, \cdots, \alpha_{N}^{*} \right) $。
2. 计算
\begin{align*}  \\ & w^{*} = \sum_{i=1}^{N} \alpha_{i}^{*} y_{i} x_{i} \end{align*}
并选择$\alpha^{*}$的一个分量$0 < \alpha_{j}^{*} < C$，计算
\begin{align*}  \\ & b^{*} = y_{j} - \sum_{i=1}^{N} \alpha_{i}^{*} y_{i} \left( x_{i} \cdot x_{j} \right) \end{align*}
3. 得到分离超平面
\begin{align*}  \\ & w^{*} \cdot x + b^{*} = 0 \end{align*}
以及分类决策函数
\begin{align*} \\& f \left( x \right) = sign \left( w^{*} \cdot x + b^{*} \right)  \end{align*}

（软间隔）支持向量：线性不可分情况下，最优化问题的解$\alpha^{*} = \left( \alpha_{1}^{*}, \alpha_{2}^{*}, \cdots, \alpha_{N}^{*} \right)^{T}$中对应于$\alpha_{i}^{*} > 0$的样本点$\left( x_{i}, y_{i} \right)$的实例$x_{i}$。

实例$x_{i}$的几何间隔
\begin{align*} \\& \gamma_{i} = \dfrac{y_{i} \left( w \cdot x_{i} + b \right)}{ \| w \|} = \dfrac{| 1 - \xi_{i} |}{\| w \|} \end{align*}
且$\dfrac{1}{2} | H_{1}H_{2} | = \dfrac{1}{\| w \|}$
则实例$x_{i}$到间隔边界的距离
\begin{align*} \\& \left| \gamma_{i} - \dfrac{1}{\| w \|} \right| = \left| \dfrac{| 1 - \xi_{i} |}{\| w \|} - \dfrac{1}{\| w \|}  \right|
\\ & = \dfrac{\xi_{i}}{\| w \|}\end{align*}
\begin{align*} \xi_{i} \geq 0 \Leftrightarrow \left\{
\begin{aligned}
\ &  \xi_{i}=0, x_{i}在间隔边界上;
\\ & 0 < \xi_{i} < 1, x_{i}在间隔边界与分离超平面之间;
\\ & \xi_{i}=1, x_{i}在分离超平面上;
\\ & \xi_{i}>1, x_{i}在分离超平面误分类一侧;
\end{aligned}
\right.\end{align*}

线性支持向量机（软间隔）的合页损失函数
\begin{align*} \\& L \left( y \left( w \cdot x + b \right) \right) = \left[ 1 - y \left(w \cdot x + b \right) \right]_{+} \end{align*}
其中，“＋”为取正函数
\begin{align*} \left[ z \right]_{+} = \left\{
\begin{aligned}
\ &  z, z > 0
\\ & 0, z \leq 0
\end{aligned}
\right.\end{align*}

### 核函数 ###
设$\mathcal{X}$是输入空间（欧氏空间$R^{n}$的子集或离散集合），$\mathcal{H}$是特征空间（希尔伯特空间），如果存在一个从$\mathcal{X}$到$\mathcal{H}$的映射
\begin{align*} \\& \phi \left( x \right) : \mathcal{X} \to \mathcal{H}   \end{align*}
使得对所有$x,z \in \mathcal{X}$，函数$K \left(x, z \right)$满足条件
\begin{align*} \\ &  K \left(x, z \right) = \phi \left( x \right) \cdot \phi \left( z \right)  \end{align*}
则称$K \left(x, z \right)$为核函数，$\phi \left( x \right)$为映射函数，式中$\phi \left( x \right) \cdot \phi \left( z \right)$为$\phi \left( x \right)$和$\phi \left( z \right)$的内积。

常用核函数：
1. 多项式核函数
\begin{align*} \\& K \left( x, z \right) = \left( x \cdot z + 1 \right)^{p} \end{align*}
2. 高斯核函数
\begin{align*} \\& K \left( x, z \right) = \exp \left( - \dfrac{\| x - z \|^{2}}{2 \sigma^{2}} \right) \end{align*}

非线性支持向量机：从非线性分类训练集，通过核函数与软间隔最大化，学习得到分类决策函数
\begin{align*} \\& f \left( x \right) = sign \left( \sum_{i=1}^{N} \alpha_{i}^{*} y_{i} K \left(x, x_{i} \right) + b^{*} \right) \end{align*}
称为非线性支持向量机，$K \left( x, z \right)$是正定核函数。

### 非线性支持向量机学习算法 ###
输入：训练数据集$T = \left\{ \left( x_{1}, y_{1} \right), \left( x_{2}, y_{2} \right), \cdots, \left( x_{N}, y_{N} \right) \right\}$，其中$x_{i} \in \mathcal{X} = R^{n}, y_{i} \in \mathcal{Y} = \left\{ +1, -1 \right\}, i = 1, 2, \cdots, N$
输出：分类决策函数
1. 选择适当的核函数$K \left( x, z \right)$和惩罚参数$C \geq 0$，构建并求解约束最优化问题
\begin{align*} \\ &  \min_{\alpha} \dfrac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_{i} \alpha_{j} y_{i} y_{j} K \left( x_{i}, x_{j} \right)  - \sum_{i=1}^{N} \alpha_{i}
\\ & s.t. \sum_{i=1}^{N} \alpha_{i} y_{i} = 0
\\ & 0 \leq \alpha_{i} \leq C , \quad i=1,2, \cdots, N  \end{align*}
求得最优解$\alpha^{*} = \left( \alpha_{1}^{*}, \alpha_{1}^{*}, \cdots, \alpha_{N}^{*} \right) $。
2. 计算
\begin{align*}  \\ & w^{*} = \sum_{i=1}^{N} \alpha_{i}^{*} y_{i} x_{i} \end{align*}
并选择$\alpha^{*}$的一个分量$0 < \alpha_{j}^{*} < C$，计算
\begin{align*}  \\ & b^{*} = y_{j} - \sum_{i=1}^{N} \alpha_{i}^{*} y_{i} K \left( x_{i}, x_{j} \right) \end{align*}
3. 得到分离超平面
\begin{align*}  \\ & w^{*} \cdot x + b^{*} = 0 \end{align*}
以及分类决策函数
\begin{align*} \\& f \left( x \right) = sign \left( \sum_{i=1}^{N} \alpha_{i}^{*} y_{i} K \left( x_{i}, x_{j} \right) + b^{*} \right)  \end{align*}

序列最小最优化（sequential minimal optimization，SMO）算法  要解如下凸二次规划的对偶问题：
\begin{align*} \\ &  \min_{\alpha} \dfrac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_{i} \alpha_{j} y_{i} y_{j} K \left( x_{i}, x_{j} \right)  - \sum_{i=1}^{N} \alpha_{i}
\\ & s.t. \sum_{i=1}^{N} \alpha_{i} y_{i} = 0
\\ & 0 \leq \alpha_{i} \leq C , \quad i=1,2, \cdots, N  \end{align*}

选择$\alpha_{1}, \alpha_{2}$两个变量，其他变量$\alpha_{i} \left( i = 3, 4, \cdots, N \right)$是固定的，SMO的最优化问题的子问题
\begin{align*} \\ &  \min_{\alpha_{1}, \alpha_{2}} W \left( \alpha_{1}, \alpha_{2} \right) = \dfrac{1}{2} K_{11} \alpha_{1}^{2} + \dfrac{1}{2} K_{22} \alpha_{2}^{2} + y_{1} y_{2} K_{12} \alpha_{1} \alpha_{2}
\\ & \quad\quad\quad\quad\quad\quad - \left( \alpha_{1} + \alpha_{2} \right) + y_{1} \alpha_{1} \sum_{i=3}^{N} y_{i} \alpha_{i} K_{i1} + y_{2} \alpha_{2} \sum_{i=3}^{N} y_{i} \alpha_i K_{i2}
\\ & s.t. \quad \alpha_{1} + \alpha_{2} = -\sum_{i=3}^{N} \alpha_{i} y_{i} = \varsigma
\\ & 0 \leq \alpha_{i} \leq C , \quad i=1,2  \end{align*}
其中，$K_{ij} = K \left( x_{i}, x_{j} \right), i,j = 1,2, \cdots, N, \varsigma$是常数，且省略了不含$\alpha_{1}, \alpha_{2}$的常数项。

设凸二次规划的对偶问题的初始可行解为$\alpha_{1}^{old}, \alpha_{2}^{old}$，最优解为$\alpha_{1}^{new}, \alpha_{2}^{new}$，且在沿着约束方向未经剪辑时$\alpha_{2}$的最优解为$ \alpha_{2}^{new,unc}$。

由于$\alpha_{2}^{new}$需要满足$0 \leq \alpha_{i} \leq C$，所以最优解$\alpha_{2}^{new}$的取值范围需满足
\begin{align*} \\ &  L \leq \alpha_{2}^{new} \leq H  \end{align*}
其中，L与H是$\alpha_{2}^{new}$所在的对角线段断点的界。
如果$y_{1} \neq y_{2}$，则
\begin{align*} \\ &  L = \max \left( 0, \alpha_{2}^{old} - \alpha_{1}^{old} \right), H = \min \left( C, C + \alpha_{2}^{old} - \alpha_{1}^{old} \right) \end{align*}
如果$y_{1} = y_{2}$，则
\begin{align*} \\ &  L = \max \left( 0, \alpha_{2}^{old} + \alpha_{1}^{old} - C \right), H = \min \left( C,  \alpha_{2}^{old} + \alpha_{1}^{old} \right) \end{align*}

记
\begin{align*} \\ &  g \left( x \right) = \sum_{i=1}^{N} \alpha_{i} y_{i} K \left( x_{i}, x \right) + b  \end{align*}
令
\begin{align*} \\ &  E_{i} = g \left( x_{i} \right) - y_{i} = \left( \sum_{j=1}^{N} \alpha_{j} y_{j} K \left( x_{j}, x_{i} \right) + b \right) - y_{i}, \quad i=1,2
\\ & v_{i} = \sum_{j=3}^{N} \alpha_{j} y_{j} K \left( x_{i}, x_{j} \right) = g \left( x_{i} \right) - \sum_{j=1}^{2}\alpha_{j} y_{j} K \left( x_{i}, x_{j} \right) - b, \quad i=1,2\end{align*}
则
\begin{align*} \\ &   W \left( \alpha_{1}, \alpha_{2} \right) = \dfrac{1}{2} K_{11} \alpha_{1}^{2} + \dfrac{1}{2} K_{22} \alpha_{2}^{2} + y_{1} y_{2} K_{12} \alpha_{1} \alpha_{2}
\\ & \quad\quad\quad\quad\quad\quad - \left( \alpha_{1} + \alpha_{2} \right) + y_{1} v_{1} \alpha_{1}+ y_{2} v_{2} \alpha_{2}
\end{align*}

由于$\alpha_{1} y_{1} = \varsigma, y_{i}^{2} = 1$，可将$\alpha_{1}$表示为
\begin{align*} \\ &  \alpha_{1} = \left( \varsigma - y_{2} \alpha_{2} \right) y_{1}\end{align*}
代入，得
\begin{align*} \\ &   W \left( \alpha_{2} \right) = \dfrac{1}{2} K_{11} \left[ \left( \varsigma - y_{2} \alpha_{2} \right) y_{1} \right]^{2} + \dfrac{1}{2} K_{22} \alpha_{2}^{2} + y_{1} y_{2} K_{12} \left( \varsigma - y_{2} \alpha_{2} \right) y_{1} \alpha_{2}
\\ & \quad\quad\quad\quad\quad\quad - \left[ \left( \varsigma - y_{2} \alpha_{2} \right) y_{1} + \alpha_{2} \right] + y_{1} v_{1} \left( \varsigma - y_{2} \alpha_{2} \right) y_{1} + y_{2} v_{2} \alpha_{2}
\\ & = \dfrac{1}{2} K_{11} \left( \varsigma - y_{2} \alpha_{2} \right)^{2} + \dfrac{1}{2} K_{22} \alpha_{2}^{2} + y_{2} K_{12} \left( \varsigma - y_{2} \alpha_{2} \right) \alpha_{2}
\\ & \quad\quad\quad\quad\quad\quad - \left( \varsigma - y_{2} \alpha_{2} \right) y_{1} - \alpha_{2} + v_{1} \left( \varsigma - y_{2} \alpha_{2} \right) + y_{2} v_{2} \alpha_{2}
\end{align*}

对$\alpha_{2}$求导
\begin{align*} \\ & \dfrac {\partial W}{\partial \alpha_{2}} = K_{11} \alpha_{2} + K_{22} \alpha_{2} -2 K_{12} \alpha_{2}
\\ & \quad\quad\quad - K_{11} \varsigma y_{2} + K_{12} \varsigma y_{2} + y_{1} y_{2} -1 - v_{1} y_{2} + y_{2} v_{2} \end{align*}
令其为0，得
\begin{align*} \\ & \left( K_{11} + K_{22} - 2 K_{12} \right) \alpha_{2} = y_{2} \left( y_{2} - y_{1} + \varsigma K_{11} - \varsigma K_{12} + v_{1} - v_{2} \right)
\\ & \quad\quad\quad\quad\quad\quad\quad\quad = y_{2} \left[ y_{2} - y_{1} + \varsigma K_{11} - \varsigma K_{12} + \left( g \left( x_{1} \right) - \sum_{j=1}^{2}\alpha_{j} y_{j} K_1j - b \right)
\\ \quad\quad\quad\quad\quad\quad\quad\quad  - \left( g \left( x_{2} \right) - \sum_{j=1}^{2}\alpha_{j} y_{j} K_2j - b \right) \right]\end{align*}

将$\varsigma = \alpha_{1}^{old} y_{1} + \alpha_{2}^{old} y_{2}$代入，得
\begin{align*} \\ & \left( K_{11} + K_{22} - 2 K_{12} \right) \alpha_{2}^{new,unc} = y_{2} \left( \left( K_{11} + K_{22} - 2 K_{12} \right) \alpha_{2}^{old} y_{2} + y_{2} - y_{1} + g \left( x_{1} \right) - g \left( x_{2} \right) \right)
\\ & \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad = \left( K_{11} + K_{22} - 2 K_{12} \right) \alpha_{2}^{old} + y_{2} \left( E_{1} - E_{2} \right) \end{align*}

令$\eta = K_{11} + K_{22} - 2 K_{12}$代入，得
\begin{align*} \\ & \alpha_{2}^{new,unc} = \alpha_{2}^{old} + \dfrac{y_{2} \left( E_{1} - E_{2} \right)}{\eta}\end{align*}

经剪辑后
\begin{align*} \alpha_{2}^{new} = \left\{
\begin{aligned}
\ &  H, \alpha_{2}^{new,unc} > H
\\ & \alpha_{2}^{new,unc}, L \leq \alpha_{2}^{new,unc} \leq H
\\ & L, \alpha_{2}^{new,unc} < L
\end{aligned}
\right.\end{align*}

由于$\varsigma = \alpha_{1}^{old} y_{1} + \alpha_{2}^{old} y_{2}$及$\varsigma = \alpha_{1}^{new} y_{1} + \alpha_{2}^{new} y_{2}$
则
\begin{align*} \\ & \alpha_{1}^{old} y_{1} + \alpha_{2}^{old} y_{2} = \alpha_{1}^{new} y_{1} + \alpha_{2}^{new} y_{2}
\\ & \quad\quad\quad\quad \alpha_{1}^{new} = \alpha_{1}^{old} + y_{1} y_{2} \left( \alpha_{2}^{old} - \alpha_{2}^{new} \right) \end{align*}

由分量$0 < \alpha_{1}^{new} < C$，则
\begin{align*}  \\ & b_1^{new} = y_{1} - \sum_{i=3}^{N} \alpha_{i} y_{i} K_{i1} - \alpha_{1}^{new} y_{1} K_{11} - \alpha_{2}^{new} y_{2} K_{21} \end{align*}
由
\begin{align*} \\ &  E_{1} = g \left( x_{1} \right) - y_{1} = \left( \sum_{j=1}^{N} \alpha_{j} y_{j} K_{ij} + b \right) - y_{1}
\\ & = \sum_{i=3}^{N} \alpha_{i} y_{i} K_{i1} + \alpha_{1}^{old} y_{1} K_{11} + \alpha_{2}^{old} y_{2} K_{21} + b^{old} - y_{1} \end{align*}
则
\begin{align*} \\ & y_{1} - \sum_{i=3}^{N} \alpha_{i} y_{i} K_{i1} = -E_{1} + \alpha_{1}^{old} y_{1} K_{11} + \alpha_{2}^{old} y_{2} K_{21} + b^{old}  \end{align*}
代入，得
\begin{align*} \\ & b_1^{new} =  -E_{1} + y_{1} K_{11} \left( \alpha_{1}^{new} - \alpha_{1}^{old} \right) - y_{2} K_{21} \left( \alpha_{2}^{new} - \alpha_{2}^{old} \right) + b^{old} \end{align*}
同理，得
\begin{align*} \\ & b_2^{new} =  -E_{2} + y_{1} K_{12} \left( \alpha_{1}^{new} - \alpha_{1}^{old} \right) - y_{2} K_{22} \left( \alpha_{2}^{new} - \alpha_{2}^{old} \right) + b^{old} \end{align*}
如果$\alpha_{1}^{new}, \alpha_{2}^{new}$满足$0 < \alpha_{i}^{new} < C, i = 1, 2$，
则
\begin{align*} \\ & b^{new} = b_{1}^{new} = b_{2}^{new}\end{align*}
否则
\begin{align*} \\ & b^{new} = \dfrac{b_{1}^{new} + b_{2}^{new}}{2} \end{align*}

更新$E_{i}$
\begin{align*} \\ & E_{i}^{new} = \sum_{S} y_{j} \alpha_{j} K_{ \left( x_{i}, x_{j} \right)} + b^{new} - y_{i} \end{align*}
其中，$S$是所有支持向量$x_{j}$的集合。

SMO算法：
输入：训练数据集$T = \left\{ \left( x_{1}, y_{1} \right), \left( x_{2}, y_{2} \right), \cdots, \left( x_{N}, y_{N} \right) \right\}$，其中$x_{i} \in \mathcal{X} = R^{n}, y_{i} \in \mathcal{Y} = \left\{ +1, -1 \right\}, i = 1, 2, \cdots, N$，精度$\varepsilon$；
输出：近似解$\hat \alpha$
1. 取初始值$\alpha^{0} = 0$，令$k = 0$；
2. 选取优化变量$\alpha_{1}^{\left( k \right)},\alpha_{2}^{\left( k \right)}$，求解
\begin{align*} \\ &  \min_{\alpha_{1}, \alpha_{2}} W \left( \alpha_{1}, \alpha_{2} \right) = \dfrac{1}{2} K_{11} \alpha_{1}^{2} + \dfrac{1}{2} K_{22} \alpha_{2}^{2} + y_{1} y_{2} K_{12} \alpha_{1} \alpha_{2}
\\ & \quad\quad\quad\quad\quad\quad - \left( \alpha_{1} + \alpha_{2} \right) + y_{1} \alpha_{1} \sum_{i=3}^{N} y_{i} \alpha_{i} K_{i1} + y_{2} \alpha_{2} \sum_{i=3}^{N} y_{i} \alpha_i K_{i2}
\\ & s.t. \quad \alpha_{1} + \alpha_{2} = -\sum_{i=3}^{N} \alpha_{i} y_{i} = \varsigma
\\ & 0 \leq \alpha_{i} \leq C , \quad i=1,2  \end{align*}
求得最优解$\alpha_{1}^{\left( k＋1 \right)},\alpha_{2}^{\left( k+1 \right)}$，更新$\alpha$为$\alpha^{\left( k+1 \right)}$；
3. 若在精度$\varepsilon$范围内满足停机条件
\begin{align*}  \\ & \sum_{i=1}^{N} \alpha_{i} y_{i} = 0
\\ & 0 \leq \alpha_{i} \leq C, i = 1, 2, \cdots, N
\\ & \end{align*}
\begin{align*} y_{i} \cdot g \left( x_{i} \right) = \left\{
\begin{aligned}
\ &  \geq 1, \left\{ x_{i} | \alpha_{i} = 0 \right\}
\\ & = 1, \left\{ x_{i} | 0 < \alpha_{i} < C \right\}
\\ & \leq 1, \left\{ x_{i} | \alpha_{i} = C \right\}
\end{aligned}
\right.\end{align*}
则转4.；否则令$k = k + 1$，转2.；
4.取$\hat \alpha = \alpha^{\left( k + 1 \right)}$。

```
print(__doc__)

import numpy as np
import matplotlib.pyplot as plt
from sklearn import svm, datasets

def make_meshgrid(x, y, h=.02):
    """Create a mesh of points to plot in

    Parameters
    ----------
    x: data to base x-axis meshgrid on
    y: data to base y-axis meshgrid on
    h: stepsize for meshgrid, optional

    Returns
    -------
    xx, yy : ndarray
    """
    x_min, x_max = x.min() - 1, x.max() + 1
    y_min, y_max = y.min() - 1, y.max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                         np.arange(y_min, y_max, h))
    return xx, yy

def plot_contours(ax, clf, xx, yy, **params):
    """Plot the decision boundaries for a classifier.

    Parameters
    ----------
    ax: matplotlib axes object
    clf: a classifier
    xx: meshgrid ndarray
    yy: meshgrid ndarray
    params: dictionary of params to pass to contourf, optional
    """
    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    out = ax.contourf(xx, yy, Z, **params)
    return out

# import some data to play with
iris = datasets.load_iris()
# Take the first two features. We could avoid this by using a two-dim dataset
X = iris.data[:, :2]
y = iris.target

# we create an instance of SVM and fit out data. We do not scale our
# data since we want to plot the support vectors
C = 1.0  # SVM regularization parameter
models = (svm.SVC(kernel='linear', C=C),
          svm.LinearSVC(C=C, max_iter=10000),
          svm.SVC(kernel='rbf', gamma=0.7, C=C),
          svm.SVC(kernel='poly', degree=3, gamma='auto', C=C))
models = (clf.fit(X, y) for clf in models)

# title for the plots
titles = ('SVC with linear kernel',
          'LinearSVC (linear kernel)',
          'SVC with RBF kernel',
          'SVC with polynomial (degree 3) kernel')

# Set-up 2x2 grid for plotting.
fig, sub = plt.subplots(2, 2)
plt.subplots_adjust(wspace=0.4, hspace=0.4)

X0, X1 = X[:, 0], X[:, 1]
xx, yy = make_meshgrid(X0, X1)

for clf, title, ax in zip(models, titles, sub.flatten()):
    plot_contours(ax, clf, xx, yy,
                  cmap=plt.cm.coolwarm, alpha=0.8)
    ax.scatter(X0, X1, c=y, cmap=plt.cm.coolwarm, s=20, edgecolors='k')
    ax.set_xlim(xx.min(), xx.max())
    ax.set_ylim(yy.min(), yy.max())
    ax.set_xlabel('Sepal length')
    ax.set_ylabel('Sepal width')
    ax.set_xticks(())
    ax.set_yticks(())
    ax.set_title(title)

plt.show()
```

```

```


# Worksheet 4

```
%matplotlib inline
```

## Question 1

Convert the ODE

\begin{equation}
  y''' + x y'' + 3 y' + y = e^{−x}
\end{equation}

into a first order system of ODEs.

### Answer Question 1

Step by step we introduce

\begin{align}
  u &= y' \\
  v &= u' \\
    &= y''.
\end{align}

We can therefore write the ODE into a system of ODEs. The first order ODEs for $y$ and $u$ are given by the definitions above. The ODE for $v$ is given from the original equation, substituting in the definition of $u$ where appropriate, to get

\begin{align}
  \begin{pmatrix} y \\ u \\ v \end{pmatrix}' & = \begin{pmatrix} u \\ v \\ e^{-x} - x y'' - 3 y' - y \end{pmatrix} \\
  & = \begin{pmatrix} u \\ v \\ e^{-x} - x v - 3 u - y \end{pmatrix}.
\end{align}

## Question 2

Show by Taylor expansion that the backwards differencing estimate of $f(x)$,

\begin{equation}
  f(x) = \frac{f(x) − f(x − h)}{h}
\end{equation}

is first order accurate.

### Answer Question 2

We have the Taylor series expansion of $f(x − h)$ about $x$ is

\begin{equation}
  f(x − h) = f(x) − h f'(x) + \frac{h^2}{2!} f''(x) + {\mathcal O} (h^3).
\end{equation}

Substituting this in to the backwards difference formula we find

\begin{align}
  \frac{f(x) - f(x - h)}{h} & = \frac{f(x) - f(x) + h f'(x) + \frac{h^2}{2!} f''(x) + {\mathcal O} (h^3)}{h} \\
  & = f'(x) + {\mathcal O} (h)
\end{align}

Therefore the difference between the exact derivative $f'$ and the backwards difference estimate is $\propto h$ and hence the finite difference estimate is first order accurate.

## Question 3

Use Taylor expansion to derive a symmetric or central difference estimate of $f^{(4)}(x)$ on a grid with spacing $h$.

### Answer Question 3

For this we need the Taylor expansions

\begin{align*}
    f(x + h) & = f(x) + h f^{(1)}(x) + \frac{h^2}{2!} f^{(2)}(x) +
    \frac{h^3}{3!} f^{(3)}(x) + \frac{h^4}{4!} f^{(4)}(x) +
    \frac{h^5}{5!} f^{(5)}(x) + \dots \\
    f(x - h) & = f(x) - h f^{(1)}(x) + \frac{h^2}{2!} f^{(2)}(x) -
    \frac{h^3}{3!} f^{(3)}(x) + \frac{h^4}{4!} f^{(4)}(x) -
    \frac{h^5}{5!} f^{(5)}(x) + \dots \\
    f(x + 2 h) & = f(x) + 2 h f^{(1)}(x) + \frac{4 h^2}{2!} f^{(2)}(x) +
    \frac{8 h^3}{3!} f^{(3)}(x) + \frac{16 h^4}{4!} f^{(4)}(x) +
    \frac{32 h^5}{5!} f^{(5)}(x) + \dots \\
    f(x - 2 h) & = f(x) - 2 h f^{(1)}(x) + \frac{4 h^2}{2!} f^{(2)}(x) -
    \frac{8 h^3}{3!} f^{(3)}(x) + \frac{16 h^4}{4!} f^{(4)}(x) -
    \frac{32 h^5}{5!} f^{(5)}(x) + \dots
\end{align*}

By a central or symmetric difference estimate we mean that the coefficient of $f(x \pm n h)$ should have the same magnitude. By comparison with central difference estimates for first and second derivatives we see that for odd order derivatives the coefficients should have opposite signs and for even order the same sign.

So we write our estimate as

\begin{equation*}
    f^{(4)}(x) \simeq A f(x) + B \left( f(x + h) + f(x - h) \right)
    + C \left( f(x + 2 h) +  f(x - 2 h) \right)
\end{equation*}

and we then need to constrain the coefficients $A, B, C$. By looking at terms proportional to $h^s$ we see

\begin{align*}
    h^0: && 0 & = A + 2 B + 2 C \\
    h^1: && 0 & = 0 \\
    h^2: && 0 & = B + 4 C \\
    h^3: && 0 & = 0 \\
    h^4: && \frac{1}{h^4} & = \frac{B}{12} + \frac{16 C}{12}.
\end{align*}

This gives three constraints on our three unknowns so we cannot go to higher order. Solving the equations gives

\begin{equation*}
    A = \frac{6}{h^4}, \qquad B = -\frac{4}{h^4}, \qquad C =
    \frac{1}{h^4}.
\end{equation*}

Writing it out in obvious notation we have

\begin{equation*}
    f_1^{(4)} = \frac{1}{h^4} \left( 6 f_i - 4 (f_{i+1} + f_{i-1}) +
      (f_{i+2} + f_{i-2}) \right).
\end{equation*}

## Question 4

State the convergence rate of Euler's method and the Euler predictor-corrector method.

### Answer Question 4

Euler's method converges as $h$ and the predictor-corrector method as $h^2$.

## Question 5

Explain when multistage methods such as Runge-Kutta methods are useful.

### Answer Question 5

Multistage methods require only one vector of initial data, which must be provided to completely specify the IVP; that is, the method is self-starting. It is also easy to adapt a multistage method to use variable step sizes; that is, to make the algorithm adaptive depending on local error estimates in order to keep the global error within some tolerance. Finally, it is relatively easy to theoretically show convergence. Combining this we see that multistage methods are useful as generic workhorse algorithms and in cases where the function defining the IVP may vary widely in behaviour, so that adaptive algorithms are required.

## Question 6

Explain the power method for finding the largest eigenvalue of a matrix. In particular, explain why it is simpler to find the absolute value, and how to find the phase information.

### Answer Question 6

The idea behind the power method is that most easily seen by writing out a generic vector ${\bf x}$ in terms of the eigenvectors of the matrix $A$ whose eigenvalues we wish to find,

\begin{equation}
  {\bf x} = \sum_{i=1}^N a_i {\bf e}_i,
\end{equation}

where we assume that the eigenvectors are ordered such that the associated eigenvalues have the order $|\lambda_1 | > |\lambda_2 | \ge |\lambda_3 | \ge \dots \ge |\lambda_N |$. Note that we always assume that there is a unique eigenvalue $\lambda_1$ with largest magnitude.

We then note that multiplying this generic vector by the matrix $A$ a number of times gives

\begin{equation}
  A^k {\bf x} = \lambda_1^k \sum_{i=1}^N a_i \left( \frac{\lambda_i}{\lambda_1} \right)^k {\bf e}_i.
\end{equation}

We then note that, for $i = 1$, the ratio of the eigenvalues $(\lambda_i / \lambda_1)^k$ must tend to zero as $k \to \infty$. Therefore in the limit we will "pick out" $\lambda_1$.

Of course, to actually get the eigenvalue itself we have to essentially divide two vectors. That is, we define a sequence $x^{(k)}$ where the initial value $x^{(0)}$ is arbitrary and at each step we multiply by $A$, so that

\begin{equation}
  x^{(k)} = A^k x^{(0)}.
\end{equation}

It follows that we can straightforwardly get $\lambda_1$ by looking at "the ratio of successive iterations". E.g.,

\begin{equation}
  \lim_{k \to \infty} \frac{ \| {\bf x}_{k+1} \| }{ \| {\bf x}_k \| } = | \lambda_1 |.
\end{equation}

This only gives information about the magnitude as we have used the simplest way of getting from a vector to a real number, the absolute value. To retain information about the phase we need to replace the absolute value of the vectors with some linear functional such as the sum of the coefficients.

## Coding Question 1

Apply Euler's method to the ODE

\begin{equation}
  y' + 2y = 2 − e^{−4 x}, \qquad y(0) = 1.
\end{equation}

Find the value of $y(1)$ (analytic answer is $1 − (e^{−2} − e^{−4})/2$) and see how your method converges with resolution.

### Answer Coding Question 1

```
def Euler(f, y0, interval, N = 100):
    """Solve the IVP y' = f(x, y) on the given interval using N+1 points (counting the initial point) with initial data y0."""
    import numpy as np

    h = (interval[1] - interval[0]) / N
    x = np.linspace(interval[0], interval[1], N+1)
    y = np.zeros((len(y0), N+1))
    y[:, 0] = y0

    for i in range(N):
        y[:, i+1] = y[:, i] + h * f(x[i], y[:, i])

    return x, y

def fn_q1(x, y):
    """Function defining the IVP in question 1."""
    import numpy as np

    return 2.0 - np.exp(-4.0*x) - 2.0*y

# Now do the test
import numpy as np

exact_y_end = 1.0 - (np.exp(-2.0) - np.exp(-4.0)) / 2.0

# Test at default resolution
x, y = Euler(fn_q1, np.array([1.0]), [0.0, 1.0])
print "Error at the end point is ", y[:, -1] - exact_y_end

import matplotlib.pyplot as plt
fig = plt.figure(figsize = (12, 8), dpi = 50)
plt.plot(x, y[0, :], 'b-+')
plt.xlabel('$x$', size = 16)
plt.ylabel('$y$', size = 16)

# Now do the convergence test

levels = np.array(range(4, 10))
Npoints = 2**levels
abs_err = np.zeros(len(Npoints))
for i in range(len(Npoints)):
    x, y = Euler(fn_q1, np.array([1.0]), [0.0, 1.0], Npoints[i])
    abs_err[i] = abs(y[0, -1] - exact_y_end)

# Best fit to the errors
h = 1.0 / Npoints
p = np.polyfit(np.log(h), np.log(abs_err), 1)

fig = plt.figure(figsize = (12, 8), dpi = 50)
plt.loglog(h, abs_err, 'kx')
plt.loglog(h, np.exp(p[1]) * h**(p[0]), 'b-')
plt.xlabel('$h$', size = 16)
plt.ylabel('$|$Error$|$', size = 16)
plt.legend(('Euler Errors', "Best fit line slope {:.3}".format(p[0])), loc = "upper left")

plt.show()
```

## Coding Question 2

Apply the standard RK4 method to the above system, again checking that it converges with resolution.

### Answer Coding Question 2

```
def RK4(f, y0, interval, N = 100):
    """Solve the IVP y' = f(x, y) on the given interval using N+1 points (counting the initial point) with initial data y0."""
    import numpy as np

    h = (interval[1] - interval[0]) / N
    x = np.linspace(interval[0], interval[1], N+1)
    y = np.zeros((len(y0), N+1))
    y[:, 0] = y0

    for i in range(N):
        k1 = h * f(x[i]          , y[:, i])
        k2 = h * f(x[i] + h / 2.0, y[:, i] + k1 / 2.0)
        k3 = h * f(x[i] + h / 2.0, y[:, i] + k2 / 2.0)
        k4 = h * f(x[i] + h      , y[:, i] + k3)
        y[:, i+1] = y[:, i] + (k1 + k4 + 2.0 * (k2 + k3)) / 6.0

    return x, y

def fn_q2(x, y):
    """Function defining the IVP in question 2."""
    import numpy as np

    return 2.0 - np.exp(-4.0*x) - 2.0*y

# Now do the test
import numpy as np

exact_y_end = 1.0 - (np.exp(-2.0) - np.exp(-4.0)) / 2.0

# Test at default resolution
x, y = RK4(fn_q1, np.array([1.0]), [0.0, 1.0])
print "Error at the end point is ", y[:, -1] - exact_y_end

import matplotlib.pyplot as plt
fig = plt.figure(figsize = (12, 8), dpi = 50)
plt.plot(x, y[0, :], 'b-+')
plt.xlabel('$x$', size = 16)
plt.ylabel('$y$', size = 16)

# Now do the convergence test

levels = np.array(range(4, 10))
Npoints = 2**levels
abs_err = np.zeros(len(Npoints))
for i in range(len(Npoints)):
    x, y = RK4(fn_q1, np.array([1.0]), [0.0, 1.0], Npoints[i])
    abs_err[i] = abs(y[0, -1] - exact_y_end)

# Best fit to the errors
h = 1.0 / Npoints
p = np.polyfit(np.log(h), np.log(abs_err), 1)

fig = plt.figure(figsize = (12, 8), dpi = 50)
plt.loglog(h, abs_err, 'kx')
plt.loglog(h, np.exp(p[1]) * h**(p[0]), 'b-')
plt.xlabel('$h$', size = 16)
plt.ylabel('$|$Error$|$', size = 16)
plt.legend(('RK4 Errors', "Best fit line slope {0:.3}".format(p[0])), loc = "upper left")

plt.show()
```

## Coding Question 3

Write a code using the power method and inverse power method to compute the largest and smallest eigenvalues of an arbitrary matrix. Apply it to a random $n = 3$ matrix, checking that the correct answer is found. How does the number of iterations required for convergence to a given level vary with the size of the matrix?

### Answer Coding Question 3

```
def PowerMethod(A, tolerance = 1e-10, MaxSteps = 100):
    """Apply the power method to the matrix A to find the largest eigenvalue in magnitude."""
    import numpy as np
    import numpy.linalg as la

    n = np.size(A, 0)
    # Simple initial value
    x = np.ones(n)
    x /= la.norm(x)
    ratio = 1.0
    for k in range(MaxSteps):
        ratio_old = ratio
        x_old = x.copy()
        x = np.dot(A, x)
        ratio = np.sum(x) / np.sum(x_old)
        x /= la.norm(x)
        if (abs(ratio - ratio_old) < tolerance):
            break

    return [ratio, k]

def InversePowerMethod(A, tolerance = 1e-10, MaxSteps = 100):
    """Apply the inverse power method to the matrix A to find the smallest eigenvalue in magnitude."""
    import numpy as np
    import scipy.linalg as la

    n = np.size(A, 0)
    # Simple initial value
    x = np.ones(n)
    x /= la.norm(x)
    ratio = 1.0
    for k in range(MaxSteps):
        ratio_old = ratio
        x_old = x.copy()
        x = la.solve(A, x)
        ratio = np.sum(x) / np.sum(x_old)
        x /= la.norm(x)
        if (abs(ratio - ratio_old) < tolerance):
            break

    return [1.0/ratio, k]

# Test on a random 3x3 matrix
import numpy as np
import scipy.linalg as la

A = np.random.rand(3,3)
max_lambda, iterations_max = PowerMethod(A)
min_lambda, iterations_min = InversePowerMethod(A)
eigenvalues, eigenvectors = la.eig(A)
print "Computed maximum and minimum eigenvalues are", max_lambda, min_lambda
print "True eigenvalues are", eigenvalues

# Now check how the number of iterations depends on the matrix size.
# As we are computing random matrices, do average of 10 attempts
MinMatrixSize = 3
MaxMatrixSize = 50
Attempts = 10
iterations = np.zeros((MaxMatrixSize - MinMatrixSize + 1, Attempts))
for n in range(MinMatrixSize, MaxMatrixSize):
    for a in range(Attempts):
        A = np.random.rand(n, n)
        ratio, iterations[n - MinMatrixSize, a] = PowerMethod(A)

import matplotlib.pyplot as plt
ii = np.mean(iterations, 1)
nn = np.array(range(MinMatrixSize, MaxMatrixSize))
fig = plt.figure(figsize = (12, 8), dpi = 50)
plt.plot(range(MinMatrixSize, MaxMatrixSize+1), np.mean(iterations, 1), 'kx')
plt.xlabel('Matrix Size')
plt.ylabel('Mean number of iterations')

plt.show()
```

We see that the number of iterations is practically unchanged with the size of the matrix.

```
from IPython.core.display import HTML
def css_styling():
    styles = open("../../IPythonNotebookStyles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: Verdana, Arial, Helvetica, sans-serif;
    }
    h2 {
        font-family: Verdana, Arial, Helvetica, sans-serif;
    }
    h3 {
        font-family: Verdana, Arial, Helvetica, sans-serif;
    }
    div.text_cell_render{
        font-family: Gill, Verdana, Arial, Helvetica, sans-serif;
        line-height: 110%;
        font-size: 120%;
        width:700px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
/*    .prompt{
        display: None;
    }*/
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 12pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

> (The cell above executes the style for this notebook. It closely follows the style used in the [12 Steps to Navier Stokes](http://lorenabarba.com/blog/cfd-python-12-steps-to-navier-stokes/) course.)


```python
%matplotlib inline

from matplotlib.pyplot import * # plotting library
# math stuff
from math import *
from numpy import *
# scipy for reference
import scipy.cluster
# makes figures larger
matplotlib.rcParams['figure.figsize'] = (10,5)
```

# Radial Basis Functions, competetive learning and self-organization

In the previous project, we used single perceptron to classify linearly separable data and Multilayer Perceptrons to separate data using hyperplanes.

Radial Basis Functions are a class of symmetric functions where the value is dependent only on the distance from the center, a Radial Basis Function network uses a linear combination of radial basis functions to reconstruct a given function.

Competetive learning is a strategy for unsupervised learning where individual neurons are competing against each other for activation, Radial Basis Functions are used to evaluate which neurons have "won" (details below).

Self-organization is a behaviour resulting from competitive learning, by extension Self Organizing Maps is an algorithm which projects a n-dimensional input into a (often) 2 dimensional plane. It uses competetive learning along with collaboration within a neighbourhood (defined by a radial basis function that decreases over time) to perform self-organization.

We start with Radial Basis Functions.

## Radial Basis Functions

A Radial Basis Function Network is a 3-layer network (inputs, radial basis layer, output layer). It is a two-step learning algorithm:

1. Find representative centers on the dataset using k-means (unsupervised) or random sampling of the inputs
2. Find weight matrix

Let's generate some data to first test our network. We generate two sets of data defined by the following functions:

$$
\begin{align}
y & = \sin(x) \\
z & = \text{sgn}(\sin(x)) \\
  & = \text{square}(x)
\end{align}
$$

where $\text{sgn}(x)$ is the `sign` function.

We generate these for two sets of data, test and training. The test set covers of $ x \in [0, 2\pi] $ and the training set covers $ x \in [0.05, 2\pi] $. Ideally, if the network is trained properly, we should be able to produce identical results when passing the test dataset through the network.

```python
# define x ranges
x_train = arange(0, 2*pi, 0.1)
x_test = arange(0.05, 2*pi, 0.1)

# find training data
y_train = sin(2*x_train)
z_train = sign(y_train)

# find test data
y_test = sin(2*x_test)
z_test = sign(y_test)
```

```python
subplot(1, 2, 1)
plot(x_train, y_train)
grid()
subplot(1, 2, 2)
plot(x_train, z_train)
grid()
suptitle('training dataset')
```

```python
subplot(1, 2, 1)
plot(x_test, y_test)
grid()
subplot(1, 2, 2)
plot(x_test, z_test)
suptitle('test dataset')
grid()
```

### Radial Basis Function Networks

As stated earlier, a Radial Basis Function Network is a 3-layer neural network. It consits of an input layer which are fully connected to the hidden layers and an output layer with a single output node.

or a more compact form:

Note that there are no weights associated between the input and the hidden layer. Let $ N $ denote the number of input and $ n $ the number of nodes in the hidden layer.

The Radial Basis function that we will use is a gaussian distribution and is defined as:

$$
\phi_{i}(x) = e^{\frac{-(x-\mu_i)^2}{2\sigma_{i}^2}}
$$

Where $ i $ is the index of the hidden layer, $ \sigma_{i} $ is the node-specific variance (we assume this to be constant for all nodes in this project), $ \mu_{i} $ is the center of the distribution.

```python
def rbf(x, mu, sig):
    return exp(-power(x - mu, 2) / (2 * power(sig, 2)))

x = arange(-5, 5, 0.1)
y = rbf(x, 0, 1)

plot(x, y)
grid()
```

The idea is that any function can be approximated as a linear combination of $ \phi_{i} $, and we want to optimize the weights $ w_{i} $. For example if we have a set of equidistant distributions with overlapping ends we can scale and combine them to approximate any function (a universal approximator).

```python
centers = arange(-4, 4, 1)
domain = linspace(-4, 4, 200)
combined = zeros(domain.shape)
for i, c in enumerate(centers):
    rang = i * rbf(domain, c, 0.4)
    combined += rang
    plot(domain, rang, color='gray')
plot(domain, combined)
```

The output layer then is a linear combination of the weights and the radial bases:

$$
\hat{f} = \sum_{i}^{n} \phi_{i}(x) w_{i}
$$

We can rewrite it in vector notation:

$$
\mathbf{f} = \mathbf{\Phi w}
$$

Since we want to optimize the weights matrix we need to find a solution to this equation system. There are two ways of doing this, least squares (batch) and Delta Rule (sequential).

#### Least squares

The least squares solution is an analytical solution to the RBF problem. It relies on inverting the $ \mathbf{\Phi} $ matrix. However, since it can be singular (and thus no inverse defined because $ \det{\mathbf{\Phi}} = 0 $ we need to use a pseudo inverse defined as $ \mathbf{\Phi}^\text{T}\mathbf{\Phi} $. Thus the solution is:

$$
\mathbf{\Phi}^\text{T}\mathbf{\Phi w} = \mathbf{\Phi}^\text{T}\mathbf{f}
$$

To implement this, let us first define a function that generates the $ \mathbf{\Phi} $ matrix for any given set of inputs.

The $ \mathbf{\Phi} $ matrix can be derived from:

$$
\hat{f}_k = \sum_{i}^{n} \phi_{i}(x_k) w_{i}
$$

which can be expand it into:

$$
\begin{align}
\phi_{1}(x_1) w_1 + \phi_{2}(x_1) w_2 + \cdots + \phi_{n}(x_1) w_n & = f_1 \\
\phi_{1}(x_2) w_1 + \phi_{2}(x_2) w_2 + \cdots + \phi_{n}(x_2) w_n & = f_2 \\
& \vdots \\
\phi_{1}(x_N) w_1 + \phi_{2}(x_N) w_n + \cdots + \phi_{n}(x_N) w_n & = f_N \\
\end{align}
$$

This can be rewritten in the matrix-vector notation:

$$
\mathbf{\Phi w} = \mathbf{f}
$$

by setting:

$$
\mathbf{\Phi} = \begin{bmatrix}
\phi_1(x_1) & \phi_2(x_1) & \cdots & \phi_n(x_1) \\
\phi_1(x_2) & \phi_2(x_2) & \cdots & \phi_n(x_2) \\
\vdots & \vdots & \ddots & \vdots \\
\phi_1(x_N) & \phi_2(x_N) & \cdots & \phi_n(x_N) \\
\end{bmatrix} \begin{bmatrix}
w_1 \\
w_2 \\
\vdots \\
w_n
\end{bmatrix} = \begin{bmatrix}
f_1 \\
f_2 \\
\vdots \\
f_N
\end{bmatrix}
$$

We start by writing a function to generate the $ \mathbf{\Phi} $ matrix. It will have four parameters:

```
inputs : the set of inputs x_i
sig : a constant value specifying the spread of each RBF
kMeans : whether to use kMeans or random sampling to choose centers. The benefit of kMeans is that it guarantees the centers are maximally separated and thus most representative of the datset.
centers : Preselected centers
```

The final parameter is required due to our functional style of writing the network. Later we will rewrite it in a much more compact way.

```python
def find_phi(inputs, sig=1, kMeans=False, centers=None):
    N = max(inputs.shape) # input
    n = 10 # number of hidden nodes

    if centers is None: # if no centers are supplied, find our own
        if kMeans:
            mu = scipy.cluster.vq.kmeans(inputs, n + 1)[0]
        else: # select n + 1 random samples (with replacement) from the input the + 1 is for the bias node
            mu = random.choice(inputs, (n + 1, 1))
    else: # allow us to reuse centers
        mu = centers

    # initialize phi with zeros
    phi = zeros((N, n + 1))

    for i in range(0, n): # fill the phi matrix column-wise
        phi[:,i] = rbf(inputs - mu[i], 0, sig)

    # set bias node
    phi[:,i+1] = 1

    # return both the phi matrix and the centers (useful for visualisation)
    return phi, mu
```

The training function will simply take a set of inputs and targets, calculate the phi matrix and use the least squares approximation to calculate the weights.

```python

def train_ls(inputs, targets, sig=1, kMeans=False):
    # find phi matrix
    phi, centers = find_phi(inputs, sig, kMeans)

    # calculate weights using least squares
    weights = dot(linalg.pinv(dot(phi.T, phi)), dot(phi.T, targets))

    return weights, centers
```

The predict function will take the centers, weights and test inputs and return the predictions.

```python
def predict_ls(centers, weights, inputs, sig=1):
    phi, centers = find_phi(inputs, sig, False, centers=centers)

    prediction = dot(phi, weights)

    return prediction
```

We can now run these with our test dataset.

```python
sig = 0.5

weights, centers = train_ls(x_train, y_train, sig=sig) # train our network
prediction = predict_ls(centers, weights, x_test, sig=sig) # predict test data
plot(test_x, test_y)
plot(test_x, prediction)
legend(('training', 'prediction'))
scatter(centers, zeros(centers.shape), marker='x', color='red')

x_range = linspace(min(test_x), max(test_x), 1000)
for i, c in enumerate(centers):
    r = weights[i] * rbf(x_range, c, sig)
    plot(x_range, r, ':', alpha=0.4, color='k', linewidth=1)
    vlines(c, 0, weights[i], alpha=0.4, color='k', linewidth=1, linestyle=':')
```

We can see the effect of not using kMeans on the centers. They are randomly distributed across the domain, in this particular case it doesn't affect the performance too much as the data is well represented across the domain. However in cases where most of the dynamics is concentrated unevenly, it's better to use a clustering algorithm such as kMeans. We can see the effect of using kMeans on the same dataset.

```python
sig = 0.5

weights, centers = train_ls(x_train, y_train, sig=sig, kMeans=True) # train our network
prediction = predict_ls(centers, weights, x_test, sig=sig) # predict test data
plot(test_x, test_y)
plot(test_x, prediction)
legend(('training', 'prediction'))
scatter(centers, zeros(centers.shape), marker='x', color='red')

x_range = linspace(min(test_x), max(test_x), 1000)
for i, c in enumerate(centers):
    r = weights[i] * rbf(x_range, c, sig)
    plot(x_range, r, ':', alpha=0.4, color='k', linewidth=1)
    vlines(c, 0, weights[i], alpha=0.4, color='k', linewidth=1, linestyle=':')
```

Clearly, the center are well separated and thus have a higher probability of representing the dataset well. Our implementation currently relies on a lot of data that should be automatically handled, for example we need to pass the `sigma` value to both the training and prediction, and we need to keep track of the centers. Instead, let us package these into a class so that we can perform training and prediction easily.

Ideally instead of:

```python
weights, centers = train_ls(x_train, y_train, sig=sig, kMeans=True) # train our network
prediction = predict_ls(centers, weights, x_test, sig=sig) # predict test data
```

We would like to write:

```python
network = RBF(sig=sig, kMeans=True)
weights, centers = network.train(x_train, y_train)
prediction = network.predict(x_test)
```

which is both visually less dense and concise.

```python
class RBF_LS:
    def __init__(self, sig=1, kMeans=False, nHidden=10):
        self.sig = sig
        self.kMeans = kMeans
        self.nHidden = nHidden

        # these change after every training
        self.centers = None
        self.weights = None

    def rbf(self, x, sig): # from function rbf
        return exp(-power(x, 2) / (2 * power(sig, 2)))

    def calculatePhi(self, inputs, centers, sig): # from function find_phi
        N = max(inputs.shape) # input
        n = self.nHidden # number of hidden nodes

        # initialize phi with zeros
        phi = zeros((N, n + 1))

        for i in range(0, n): # fill the phi matrix column-wise
            phi[:,i] = self.rbf(inputs - centers[i], sig)

        # set bias node
        phi[:,i+1] = 1

        # return both the phi matrix and the centers (useful for visualisation)
        return phi

    def getCenters(self, inputs, n, kMeans=False):
        if kMeans:
            return scipy.cluster.vq.kmeans(inputs, n + 1)[0]
        else: # select n + 1 random samples (with replacement) from the input the + 1 is for the bias node
            return random.choice(inputs, (n + 1, 1))

    def train(self, inputs, targets):
        self.centers = self.getCenters(inputs, self.nHidden, self.kMeans)
        phi = self.calculatePhi(inputs, self.centers, self.sig)

        # calculate weights using least squares
        self.weights = dot(linalg.pinv(dot(phi.T, phi)), dot(phi.T, targets))

        error = linalg.norm(self.predict(inputs) - targets)

        return error

    def predict(self, inputs):
        phi = self.calculatePhi(inputs, self.centers, self.sig)

        prediction = dot(phi, self.weights)

        return prediction
```

We can now use our library to test multiple datasets with fixed parameters very easily.

```python
net = RBF_LS(sig=0.5, kMeans=True, nHidden=10)

net.train(x_train, y_train)
prediction_y = net.predict(x_test)

net.sig=0.2
net.train(x_train, z_train)
prediction_z = net.predict(z_test)

subplot(1, 2, 1)
plot(x_train, y_train)
plot(x_train, prediction_y)
legend(('training', 'prediction'))
subplot(1, 2, 2)
plot(x_train, z_train)
plot(x_train, prediction_z)
legend(('training', 'prediction'))
```

```python
goals = [0.1, 0.01, 0.001]

network = RBF_LS(0.5, kMeans=True)

print("sin(2x) dataset")
print("goal\tnHidden")
for goal in goals:
    error = 1.0
    network.nHidden = 1 # start with 1 hidden layer

    while error > goal:
        error = network.train(x_train, y_train)
        network.nHidden = network.nHidden + 1

    print("%s\t%s" % (goal, network.nHidden))

print("")

print("square(2x) dataset")
print("goal\tnHidden")

network.sig = 0.05
for goal in goals:
    error = 1.0
    network.nHidden = 1 # start with 1 hidden layer

    while error > goal and network.nHidden < x_train.shape[0]:
        network.train(x_train, z_train)
        network.nHidden = network.nHidden + 1

    print("%s\t%s" % (goal, network.nHidden))
```

    sin(2x) dataset
    goal	nHidden
    0.1	17
    0.01	21
    0.001	57

    square(2x) dataset
    goal	nHidden
    0.1	63
    0.01	63
    0.001	63

```python

```


# Importações

```python
import matplotlib.pyplot as plt
import numpy as np

from mpl_toolkits.mplot3d.axes3d import Axes3D
from sympy import symbols, diff

from matplotlib import cm #colormap
from math import log

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

%matplotlib inline
```

## Visualização de gráficos 3d

### Minimizando a função
$$f(x, y) = \frac {1} {3^{-x^2 - y ^ 2} + 1}$$

#### Minimizando
$$f(x,y) = \frac {1}{r+1}$$
Com $r$ igual a $3^{-x^2 - y ^ 2}$

```python
def f(x, y):
    r = 3**(-x**2 - y**2)
    return 1 / (r + 1)
```

```python
#Gerando dados
x = np.linspace(start = -2, stop = 2, num = 200)
y = np.linspace(start = -2, stop = 2, num = 200)

x, y = np.meshgrid(x, y)

```

```python
#Gerando grafico 3d

fig = plt.figure(figsize = [16,12])
ax = fig.gca(projection = "3d")

ax.plot_surface(x, y, f(x, y), cmap = cm.summer, alpha = 0.4)
ax.set_xlabel("X", fontsize = 20)
ax.set_ylabel("Y", fontsize = 20)
ax.set_zlabel("f(x, y)", fontsize = 20)

plt.show()
```

## Derivadas Parciais e Computação Simbolica

### $$ \frac {\partial f}{\partial x} = \frac{2x \ln(3) \cdot 3 ^ {-x^2 - y^2}}{(3^{-x^2 -y^2} + 1)^2}$$

### $$ \frac {\partial f}{\partial y} = \frac{2y \ln(3) \cdot 3 ^ {-x^2 - y^2}}{\left(3^{-x^2 -y^2} + 1\right)^2}$$

```python
a, b = symbols("x, y")

print("Nossa função custo: ", f(a, b))
print("Derivada parcial de f(a, b) em relação a X: ", diff(f(a, b), a))

print("Valor de f(x, y) com x=1.8 e y=1: ", f(a,b).evalf(subs = {a:1.8, b:1}))
```

    Nossa função custo:  1/(3**(-x**2 - y**2) + 1)
    Derivada parcial de f(a, b) em relação a X:  2*3**(-x**2 - y**2)*x*log(3)/(3**(-x**2 - y**2) + 1)**2
    Valor de f(x, y) com x=1.8 e y=1:  0.990604794032582

### Método do gradiente (Gradient descent) com Sympy

```python
#Configuração inicial

multiplicador = 0.1
max_iter = 200
params = np.array([1.8, 1.0])  #INICIAL

for n in range(max_iter):
    gradiente_x = diff(f(a,b), a).evalf(subs = {a:params[0], b:params[1]})
    gradiente_y = diff(f(a,b), b).evalf(subs = {a:params[0], b:params[1]})
    gradientes = np.array([gradiente_x, gradiente_y])

    params = params - multiplicador * gradientes

#Resultados
print("Valores no vetor gradientes: ", gradientes)
print("Menor valor de x: ", params[0])
print("Menor valor de y: ", params[1])
print("Custo: ", f(params[0], params[1]))
```

    Valores no vetor gradientes:  [0.000461440542096373 0.000256355856720208]
    Menor valor de x:  0.000793898510134722
    Menor valor de y:  0.000441054727852623
    Custo:  0.500000226534985

```python
#Derivadas parciais

def fpx(x, y):

    r = 3**(-x**2 - y**2)
    return (2 * x * log(3) * r) / ((r + 1) ** 2)

def fpy(x, y):

    r = 3**(-x**2 - y**2)
    return (2 * y * log(3) * r) / ((r + 1) ** 2)

```

```python
# Configuração inicial

multiplicador = 0.1
max_iter = 200
params = np.array([1.8, 1.0])  #INICIAL

for n in range(max_iter):
    gradiente_x = fpx(params[0], params[1])
    gradiente_y = fpy(params[0], params[1])
    gradientes = np.array([gradiente_x, gradiente_y])

    params = params - multiplicador * gradientes

#Resultados
print("Valores no vetor gradientes: ", gradientes)
print("Menor valor de x: ", params[0])
print("Menor valor de y: ", params[1])
print("Custo: ", f(params[0], params[1]))
```

    Valores no vetor gradientes:  [0.00046144 0.00025636]
    Menor valor de x:  0.0007938985101347202
    Menor valor de y:  0.0004410547278526219
    Custo:  0.5000002265349848

## Plotando método gradiente no grafico 3D & Vetores numpy avançados

```python
# Configuração inicial

multiplicador = 0.5
max_iter = 200
params = np.array([1.8, 1.0])  #INICIAL
valores = params.reshape(1, 2)

for n in range(max_iter):

    gradiente_x = fpx(params[0], params[1])
    gradiente_y = fpy(params[0], params[1])
    gradientes = np.array([gradiente_x, gradiente_y])

    params = params - multiplicador * gradientes
    valores = np.append(valores, params.reshape(1,2), axis = 0)

#Resultados
print("Valores no vetor gradientes: ", gradientes)
print("Menor valor de x: ", params[0])
print("Menor valor de y: ", params[1])
print("Custo: ", f(params[0], params[1]))
```

    Valores no vetor gradientes:  [1.56952449e-26 8.71958048e-27]
    Menor valor de x:  2.0725232663390205e-26
    Menor valor de y:  1.1514018146327904e-26
    Custo:  0.5

```python
#Gerando grafico 3d

fig = plt.figure(figsize = [16,12])
ax = fig.gca(projection = "3d")

ax.set_xlabel("X", fontsize = 20)
ax.set_ylabel("Y", fontsize = 20)
ax.set_zlabel("f(x, y)", fontsize = 20)

ax.plot_surface(x, y, f(x, y), cmap = cm.summer, alpha = 0.4)
ax.scatter(valores[:, 0], valores[:, 1], f(valores[:, 0], valores[:,1]), s = 50)

plt.show()
```

```python
#Praticando numpy arrays

kirk = np.array([["Captain", "MC"]])

hs_band = np.array([["Black Thought", "MC"], ["Questlove", "Drums"]])
print("band>>>>", hs_band.shape)

the_roots = np.append(arr = hs_band, values = kirk, axis = 0)

the_roots = np.append(arr = the_roots, values = [["Malik B", "MC"]], axis = 0)
print(the_roots)

print("Nicknames...")
print(the_roots[:, [0]])
print("Funçoes...")
print(the_roots[:, [1]])
```

    band>>>> (2, 2)
    [['Black Thought' 'MC']
     ['Questlove' 'Drums']
     ['Captain' 'MC']
     ['Malik B' 'MC']]
    Nicknames...
    [['Black Thought']
     ['Questlove']
     ['Captain']
     ['Malik B']]
    Funçoes...
    [['MC']
     ['Drums']
     ['MC']
     ['MC']]

```python

```

```python

```

```python

```

# Trabalhando com Dados e uma função custo real

## MSE: Uma função custo para problemas reais

### $$ RSS = \sum_{i=1} ^ {n} \big(y ^ {(i)} - h_\theta x^{(i)}\big)^2$$
### $$ MSE = \sum_{i=1} ^ {n} \big(y - \hat y \big)^2$$
### $$ MSE = \frac{1}{n} \sum_{i=1} ^ {n} \big(y ^ {(i)} - h_\theta x^{(i)}\big)^2$$
### $$ MSE = \frac{1}{n} \sum_{i=1} ^ {n} \big(y - \hat y \big)^2$$

```python

```

```python

```

```python

```

```python
#Criando valores

x_5 = np.array([[0.1, 1.2, 2.4, 3.2, 4.1, 5.7, 6.5]]).transpose()
y_5 = np.array([1.7, 2.4, 3.5, 3.0, 6.1, 9.4, 8.2]).reshape(7,1)

print("x_5 shape: ", x_5.shape)
print("y_5 shape: ", y_5.shape)
```

    x_5 shape:  (7, 1)
    y_5 shape:  (7, 1)

```python
regr = LinearRegression()
regr.fit(x_5, y_5)

print("Theta 0: ", regr.intercept_[0])
print("Theta 1: ", regr.coef_[0][0])
```

    Theta 0:  0.8475351486029536
    Theta 1:  1.2227264637835915

```python
plt.scatter(x_5, y_5, s= 50)
plt.plot(x_5, regr.predict(x_5), color= "orange", linewidth = 3)

plt.xlabel("Valores de x")
plt.ylabel("Valores de y")

plt.show()
```

```python
#y_hat = theta0 + theta1 * x
print(regr.coef_[0][0])
print(regr.intercept_[0])
y_hat = 0.8475351486029536 + (1.2227264637835915 * x_5)

```

    1.2227264637835915
    0.8475351486029536

```python
def mse(y, y_hat):
    return (1/len(y)) * (sum((y - y_hat) ** 2))
```

```python
print("MSE calculado manualmente: ", mse(y_5, y_hat))
print("MSE usando calculo manual: ", mean_squared_error(y_5, y_hat))
print("MSE calculado automaticamente: ", mean_squared_error(y_5, regr.predict(x_5)))
```

    MSE calculado manualmente:  [0.94796558]
    MSE usando calculo manual:  0.9479655759794577
    MSE calculado automaticamente:  0.9479655759794577

```python
#Criando valores de Theta0 e Theta1
#LEMBRANDO QUE: T0 é a intercessão e T1 é o coeficiente angula
```

```python
n_tetas = 200
t_0 = np.linspace(start= -1, stop = 3, num = n_tetas)
t_1 = np.linspace(start= -1, stop = 3, num = n_tetas)

plot_t0, plot_t1 = np.meshgrid(t_0, t_1)
plot_mse = np.zeros(plot_t0.shape)

```

```python
for i in range(n_tetas):
    for j in range(n_tetas):
        y_hat = plot_t0[i][j] + plot_t1[i][j]*x_5

        plot_mse[i][j] = mse(y_5, y_hat)

```

```python
fig = plt.figure(figsize=[16,10])
ax = fig.gca(projection = "3d")

ax.set_xlabel("teta 0", fontsize = 20)
ax.set_ylabel("teta 1", fontsize = 20)
ax.set_zlabel("MSE", fontsize = 20)

ax.plot_surface(plot_t0, plot_t1, plot_mse, cmap = cm.summer)
```

## Derivadas parciais em respeito a $\theta_0$ e a $\theta_1$:

### $$\frac{\partial MSE}{\partial \theta_0} = -\frac{2}{n} \sum_{i=1}^n \big(y^{(i)} - \theta_0 - \theta_1 x^{(i)} \big) $$
### $$\frac{\partial MSE}{\partial \theta_1} = -\frac{2}{n} \sum_{i=1}^n \big(y^{(i)} - \theta_0 - \theta_1 x^{(i)} \big) \cdot \big(x^{(i)}\big) $$

### MSE e Gradient Descent

```python
#X e y são os dados, e tetas são os dois valores que estamos tentando achar

def grad(x, y, tetas):
    n = y.size

    t0_derivada = (-2/n) * sum(y - tetas[0] - tetas[1] * x)
    t1_derivada = (-2/n) * sum((y - tetas[0] - tetas[1] * x) * x)

    return np.array([t0_derivada[0], t1_derivada[0]])
```

```python
max_iter = 1000
multiplicador = 0.01
tetas = np.array([2.9, 2.9])
teta_valores = tetas.reshape(1,2)
mse_valores = mse(y_5, tetas[0] + tetas[1]*x_5)

for i in range(max_iter):
    tetas = tetas - multiplicador * grad(x_5, y_5, tetas)

    teta_valores = np.append(arr = teta_valores, values = tetas.reshape(1,2), axis = 0)
    mse_valores = np.append(arr = mse_valores, values = mse(y_5, tetas[0] + tetas[1]*x_5), axis = 0)

print("Minimo ocorre em t0:  ", tetas[0])
print("Minimo ocorre em t1 : ", tetas[1])
print("MSE: ", mse(y_5, tetas[0] + tetas[1] * x_5))
```

    Minimo ocorre em t0:   0.8532230461743415
    Minimo ocorre em t1 :  1.2214935332607393
    MSE:  [0.94797511]

```python
fig = plt.figure(figsize=[16,10])
ax = fig.gca(projection = "3d")

ax.plot_surface(plot_t0, plot_t1, plot_mse, cmap = cm.winter, alpha = 0.4)
ax.scatter(teta_valores[:, 0], teta_valores[:, 1], mse_valores, color = "red")

ax.set_xlabel("teta 0", fontsize = 20)
ax.set_ylabel("teta 1", fontsize = 20)
ax.set_zlabel("Cost - MSE", fontsize = 20)
```