# 第四讲：牛顿法、一般线性模型

回到上一讲介绍的逻辑回归：在逻辑回归中，我们用S型函数作为$g(z)$，使用梯度上升的更新规则求参数$\theta$的最大似然估计。下面来介绍另一种方法：

## 7. 牛顿法

我们先回忆一下牛顿法找函数值零点：设函数$f:\ \mathbb{R}\to\mathbb{R}$，我们希望找到使$f(\theta)=0$的$\theta\in\mathbb{R}$值，牛顿法提供下面的更新规则：

$$\theta:=\theta-\frac{f(\theta)}{f'(\theta)}$$

该算法的大概流程为：先做对$\theta$做一次猜测$\theta=\theta_1$，然后过$f(\theta_1)$做$f$的切线，切线交横轴于点$\theta_2$（也就是切线函数值的零点），此时更新$\theta=\theta_2$，再过$f(\theta_2)$做$f$的切线，这样$\theta$就可以逼近使函数值为零的$\theta$取值，下面是牛顿法图解：

在最左的图中为函数$f$的图像，我们希望找到零$f(\theta)=0$的点，从图中看这个点大约取在$1.3$附近。假设我们初始化算法时猜测$\theta^{(0)}=4.5$，牛顿法会计算出函数$f$在$(4.5,\ f(4.5))$点的切线并解出切线的零点（即切线与$y=0$的焦点，如中图所示）。这就是第二次猜测的起点，在图中$\theta^{(1)}$大约在$2.8$附近。而最右侧的图像显示了再一次迭代的结果，这使得$\theta^{(2)}$取到了$1.8$附近。在几轮迭代之后，$\theta$会快速向$1.3$附近收敛。

根据导数（切线斜率）的定义有$f'\left(\theta^{(0)}\right)=\frac{f\left(\theta^{(0)}\right)}{\Delta}$，于是$\Delta=\frac{f\left(\theta^{(0)}\right)}{f'\left(\theta^{(0)}\right)}$，所以$\theta^{(1)}=\theta^{(0)}-\Delta=\theta^{(0)}-\frac{f\left(\theta^{(0)}\right)}{f'\left(\theta^{(0)}\right)}$。则对于一般情况有$\theta^{(t+1)}=\theta^{(t)}-\frac{f\left(\theta^{(t)}\right)}{f'\left(\theta^{(t)}\right)}$。

牛顿法给了我们一个求$f(\theta)=0$的方法，我们也可以利用它来求函数$\mathscr{l}(\theta)$的最大值。函数的最大值点也就是$\mathscr{l}(\theta)$一阶倒数为零的点，所以只需要替换上面方法中的$f(\theta)=\mathscr{l}'(\theta)$即可，我们可以用牛顿方法，按照下面的更新规则求出函数最大值：

$$\theta:=\theta-\frac{\mathscr{l}'(\theta)}{\mathscr{l}''(\theta)}$$

上一讲中，我们在推导逻辑回归的最大值时，使用向量记法的$\theta$，我们将这一标记法应用到牛顿法中，在这种多维条件下一般化的牛顿法（也称作Newton-Raphson方法）为：

$$\theta:=\theta-H^{-1}\nabla_\theta\mathscr{l}(\theta)$$

* 式中的$\nabla_\theta\mathscr{l}(\theta)$仍然表示$\mathscr{l}(\theta)$关于向量$\theta$的每一个分量$\theta_i$的偏导数；
* $H$是一个$n$阶方阵（如果算上截距项应该是$n+1$阶方阵），称为**海森矩阵（Hessian matrix）**，为了便于记忆可以当做是一阶导数（向量）除以二阶导数（乘以方阵的逆），方阵的定义为：

$$H_{ij}=\frac{\partial^2\mathscr{l}(\theta)}{\partial\theta_i\partial\theta_j}$$

相比于（批量）梯度下降法，牛顿法收敛速度通常更快，且逼近最小值时需要的迭代次数更少（牛顿收敛是一个二阶收敛（quadratic convergence））。然而牛顿法单次迭代的代价通常比梯度下降法更大，因为它需要计算整个$n$阶方阵再求逆。通常情况下，在$n$不是很大的条件下，牛顿法总体上更快。当我们使用牛顿法求逻辑回归中对数化似然函数$\mathscr{l}(\theta)$的最大值时，这个算法也被称作**Fisher scoring**。

# 第三部分：一般线性模型（Generalized Linear Models）

到目前为止，我们了解了回归问题、分类问题：在回归问题中我们设$y\mid x;\theta\sim\mathcal{N}\left(\mu,\sigma^2\right)$，而在分类问题中我们设$y\mid x;\theta\sim\mathrm{Bernoulli}(\phi)$，对关于$x$和$\theta$的函数给出合理的$\mu$和$\phi$。我们会在这一节看到，这两种假设都是一族算法模型中的特例，这一族算法模型就是我们曾经提到的一般线性模型（GLMs）。我们还将了解别的GLM家族中的模型如何推导、应用于其他分类、回归问题中。

## 8. 指数族（exponential family）

我们从定义指数分布族开始介绍GLMs，其中属于指数族的分布都可以写为如下形式：

$$p(y;\eta)=b(y)\exp\left(\eta^TT(y)-a(\eta)\right)\tag{1}$$

* 式中的$\eta$称为分布的**自然参数（natural parameter）**或**规范参数（canonical parameter）**；
* $T(y)$称为**充分统计量（sufficient statistic）**（对于我们涉及的分布，通常$T(y)=y$）；
* $a(\eta)$称为**对数配分函数（log partition function）**
* $e^{-a(\eta)}$这个量起到标准化常量的作用，它使得$p(y;\eta)$求和（离散）或对$y$求积分（连续）的结果等于$1$。

如果选定了$T,\ a,\ b$，就能够定义出由$\eta$参数化的一族（或一个集合）分布，之后，我们通过改变$\eta$就能获得这一族分布中不同的分布函数。

我们将展示伯努利分布和高斯分布都是指数分布族中的特例。期望为$\phi$的伯努利分布写作$\mathrm{Bernoulli}(\phi)$，分布$y\in\{0,\ 1\}$上取值，所以$p(y=1;\phi)=p;\ p(y=0;\phi)=1-\phi$，我们改变$\phi$则会得到有不同期望值的伯努利分布。这一类由$\phi$确定的伯努利分布就在指数分布族中，即将$(1)$式中的$T,\ a,\ b$确定下来后，式子就可以化为伯努利分布类型的表达式。

我们将伯努利分布写为：

$$\begin{align}p(y;\phi)&=\phi^y(1-\phi)^{1-y}\\&=\exp(y\log\phi+(1-y)\log(1-\phi))\\&=\exp\left(\left(\log\left(\frac{\phi}{1-\phi}\right)\right)y+log(1-\phi)\right)\end{align}$$

因此，自然参数为$\eta=\log\frac{\phi}{1-\phi}$，有趣的是，如果我们反过来用从$\eta$的表达式中解出$\phi$，则有$\phi=\frac{1}{1+e^{-\eta}}$，这就是逻辑回归中使用的S型函数！当我们用一般线性模型（GLM）的形式表达逻辑回归时，还将再次看到这一现象。为了将伯努利分布完全用指数族分布的形式表达，我们给出其它关于指数分布族的参数：

$$\begin{align}T(y)&=y\\a(\mu)&=-\log(1-\phi)\\&=\log(1+e^{\eta})\\b(y)&=1\end{align}$$

上面的参数表明，伯努利分布在选取适当的参数$T,\ a,\ b$后，就可以写为$(1)$式的形式。

接下来我们考虑高斯分布。回忆由概率模型推导线性回归的过程中，我们发现方差$\sigma^2$的取值并不影响我们最终得到的参数$\theta$和假设函数$h_\theta(x)$，因此我们即使选择任意的$\sigma^2$也不会对分布的表示造成任何影响。为了便于推导，令$\sigma^2=1$。（即使我们将$\sigma^2$以变量的形式留在概率密度函数中，高斯分布也可以表达成指数分布族的形式，此时$\mu\in\mathbb{R}^2$变为一个由$\mu,\ \sigma$表示的向量。为了将带有$\sigma^2$的高斯分布纳入GLM，我们可以给出更一般的指数族定义：

$$p(y;\eta,\tau)=b(a,\tau)\exp\left(\frac{\eta^TT(y)-a(\eta)}{c(\tau)}\right)$$

上式中的$\tau$称为**离散参数（dispersion parameter）**，而高斯分布对应的$c(\tau)=\sigma^2$。不过，对于我们简化的高斯分布，并不需要这种更一般的表达。）

$$\begin{align}p(y;\mu)&=\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{1}{2}(y-\mu)^2\right)\\&=\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{1}{2}y^2\right)\cdot\exp\left(\mu y-\frac{1}{2}\mu^2\right)\end{align}$$

于是，将高斯分布写为指数族形式有：

$$\begin{align}\eta&=\mu\\T(y)&=y\\a(\eta)&=\frac{\mu^2}{2}=\frac{\eta^2}{2}\\b(y)&=\frac{1}{\sqrt{2\pi}}\exp\left(\frac{-y^2}{2}\right)\end{align}$$

指数族当然还有很多别的成员，比如：

* 多项分布（multinomial），我们将在后面介绍，如同伯努利分布对有两个结果的事件建模一样，多项分布对有$K$个结果的事件建模；
* 泊松分布（Poisson），用于计数数据建模，比如样本中放射性元素衰变的书目、某网站访客的数量、商店顾客的数量；
* 伽马分布和指数分布（the gamma and the exponential），用于非负的连续随机变量建模，如处理时间间隔，或是处理在等公交时发生的“下一辆车什么时候到”的问题；
* 贝塔分布和狄利克雷分布（the beta and the Dirichlet），通常用于对分数进行建模，它是对概率分布进行建模的概率分布；
* Wishart分布，这是协方差矩阵的分布。

后面的课程将介绍一种常规方法，来建立对于给定了$x,\theta$的来自上述任意分布的$y$的模型。

## 9. 一般线性模型建模

假如现在需要对我们商店在给定时间内客户数量$y$做出估计，估计的依据来自诸如商店促销活动、近期广告投入、当地天气状况、给定时间是周几等等，将这些作为特征值$x$，我们也知道泊松分布适用于预测来访者数量，那接下来该如何针对问题建立模型呢？幸运的是，泊松分布属于指数分布族，于是我们可以应用一般线性模型（GLM）。在这一节，我们将介绍针对类似问题建立一般线性模型的方法。

一般的，对于想要预测的关于$x$的随机变量$y$的分类或回归问题，我们为了推导出关于问题的一般线性模型，模型需要遵守以下三条关于给定$x$下$y$的条件概率的假设：

1. $y\mid x;\theta\sim\mathrm{ExponentialFamily(\eta)}$，即对于给定的$x,\ \theta$，分布$y$服从某个由$\eta$参数化的指数族分布；
2. 对于给定的$x$，我们的目标是求出对于条件$x$下$T(y)$的期望，即$\mathrm{E}[T(y)\mid x]$。而在大多数情况下$T(y)=y$，所以我们希望学习算法得到的$h(x)=\mathrm{E}[y\mid x]$。（注意，在关于$h_\theta(x)$的选择上，这条假设既适用于逻辑回归又适用于线性回归，比如，$h_\theta(x)=p(y=1\mid x;\theta)=0\cdot p(y=0\mid x;\theta)+1\cdot p(y=1\mid x;\theta)=\mathrm{E}[y\mid x;\theta]$。）
3. 自然参数$\eta$（通常是实数）和输入特征值$x$是线性关系：$\eta=\theta^Tx,\ \eta\in\mathbb{R}$（或者对于向量有$\eta_i=\theta_i^Tx,\ \eta\in\mathbb{R}^*$。）

第三条也许是这些假设中“最欠合理”的了，它看起来更像是构建一般线性模型的”设计决策“，而不是”假设“。这三条假设（设计决策）可以让我们得到一类十分简洁的学习算法，也就是一般线性模型，并且具有良好的特性，比如便于我们学习。此外，所得模型通常对关于$y$的不同类型分布的建模都有良好的效果，比如我们接下来将介绍一般线性模型既能推导出逻辑回归，又能推导出普通最小二乘。

### 9.1 普通最小二乘法

为了证明普通最小二乘是一般线性模型中的特例，需要考虑目标变量$y$（在一般线性模型术语中也称作**响应变量（response variable）**）是连续的，我们使用高斯分布$\mathcal{N}\left(\mu,\ \sigma^2\right)$建立关于给定$x$下$y$的条件概率模型（$\mu$可能与$x$有关）。所以我们令上面的$\mathrm{ExponentialFamily}(\eta)$为高斯分布。在前面推导出的将高斯分布写为指数分布族的公式中，有$\mu=\eta$，所以：

$$\begin{align}h_\theta(x)&=\mathrm{E}[y\mid x;\theta]\\&=\mu\\&=\eta\\&=\theta^Tx\end{align}$$

* 第一个式子来自假设2；
* 第二个式子根据$y\mid x;\theta\sim\mathcal{N}\left(\mu,\sigma^2\right)$，$y$的期望值就是$\mu$；
* 第三个式子来自假设1（结合前推导出的将高斯分布写为指数分布族的公式）；
* 第四个式子来自假设3。

### 9.2 逻辑回归

再来考虑逻辑回归。现在主要看二元分类，也就是$y\in\{0,1\}$，我们很自然的联想到使用伯努利族分布对$y$建模。在前面推导出的将伯努利分布写为指数分布族的公式中，$\phi=\frac{1}{1+e^{-\eta}}$。此外，如果$y\mid x;\theta\sim\mathrm{Bernoulli}(\phi)$，则$\mathrm{E}[y\mid x;\theta]=\phi$。同上面推导普通最小二乘一样：

$$\begin{align}h_\theta(x)&=\mathrm{E}[y\mid x;\theta]\\&=\phi\\&=\frac{1}{1+e^{-\eta}}\\&=\frac{1}{1+e^{-\theta^Tx}}\end{align}$$

于是我们得到了假设函数$h_\theta(x)=\frac{1}{1+e^{-\theta^Tx}}$，这就是上讲中逻辑函数的来源：一旦我们假设对给定$x$的$y$的条件概率服从伯努利分布，就可以推导出伯努利分布的指数分布族形式，进而得到一般线性模型的定义。

再引入一些术语，一个关于自然参数$\eta$的函数$g$给出了分布期望（$g(\eta)=\mathrm{E}[T(y);\eta]$），我们把这个函数称作**正则响应函数（canonical response function）**，其逆$g^{-1}$称作**正则关联函数（canonical link function）**。对于高斯分布，正则响应函数就是$g(\eta)=\eta$；对于伯努利分布就是逻辑函数。（很多其他教材使用$g$正则连接函数，而用$g^{-1}$表示正则响应函数，但我们在这里的表示法沿用以往的机器学习课程，并将在后面的课程中继续使用这种记法。）

### 9.3 Softmax回归

再多看一个一般线性模型的例子：如果在分类问题中，遇到$y$能够取$k$个值（不再限于两个）的情况，即$y\in\{1,2,\cdots,k\}$。继续用垃圾邮件的例子，我们前面将一封邮件通过算法标记为”垃圾邮件“或”正常邮件“，这是一个二元分类问题，现在我们想将其分为三类：”垃圾邮件“、”私人邮件“和”工作邮件“。这个问题中，响应变量仍然是离散值，只不过变成了两个以上的值，我们因此需要使用多项分布来建立模型。

要推导此类问题的一般线性模型，我们首先得将多项分布表达成指数分布族的形式。

为了参数化有$k$个可能取值的多项分布，我们使用$k$个参数$\phi_1,\cdots,\phi_k$，用来表示每个可能得取值发生的概率。然而，这样设置参数可能会有冗余（过度参数化，over-parameterized），也就是参数线性相关（如果知道任意$k-1$个$\phi_i$，都可以使用$\displaystyle\sum_{i=1}^k\phi_i=1$算出最后一个$未知\phi$的值）。所以我们应该设置$k-1$个参数$\phi_1,\cdots,\phi_{k-1}$，则$\phi_i=p(y=i;\phi)$，$p(y=k;\phi)=1-\displaystyle\sum_{i=1}^{k-1}\phi_i$。为了写起来方便，在后面的推导中，我们令$\phi_k=1-\displaystyle\sum_{i=1}^{k-1}\phi_i$，但得清楚这不是一个参数，它是从其他参数中计算出来的。

接下来我们需要定义$T(y)\in\mathbb{R}^{k-1}$（多项分布是少数几个$T(y)\neq y$的概率模型）：

$$T(1)=\begin{bmatrix}1\\0\\0\\\vdots\\0\end{bmatrix},\ T(2)=\begin{bmatrix}0\\1\\0\\\vdots\\0\end{bmatrix},\ T(3)=\begin{bmatrix}0\\0\\1\\\vdots\\0\end{bmatrix},\ \cdots,\ T(k-1)=\begin{bmatrix}0\\0\\0\\\vdots\\1\end{bmatrix},\ T(k)=\begin{bmatrix}0\\0\\0\\\vdots\\0\end{bmatrix}$$

与前面介绍的分布不同的是，这里不再是$T(y)=y$，现在的$T(y)$不再是一个实数，它变成了一个$k-1$维向量，我们使用$(T(y))_i$表示向量$T(y)$的第$i$个分量。

我们再引入一个非常有用的标记：指示函数（indicator function）$1\{\cdot\}$，如果参数为真则返回$1$，反之返回$0$（$1\{True\}=1,\ 1\{False\}=0$）。举个例子$1\{2=3\}=0$而$1\{3=5-2\}=1$。

我们可以利用这个函数将$T(y)$与$y$的关系表示成$(T(y))_i=1\{y=i\}$（则$T(y)=\begin{bmatrix}(T(y))_1\\(T(y))_2\\\vdots\\(T(y))_{k-1}\end{bmatrix}=\begin{bmatrix}1\{y=1\}\\1\{y=2\}\\\vdots\\1\{y=k-1\}\end{bmatrix}$，这样就可以简洁的表示上面的一系列向量了）。更进一步，有$\mathrm{E}[(T(y))_i]=P(y=i)=\phi_i$。

一切准备就绪，可以开始将多项分布化为为指数分布族了：

$$\begin{align}p(y;\phi)&=\phi_1^{1\{y=1\}}\phi_2^{1\{y=2\}}\cdots\phi_{k-1}^{1\{y=k-1\}}\phi_k^{1\{y=k\}}\\&=\phi_1^{1\{y=1\}}\phi_2^{1\{y=2\}}\cdots\phi_{k-1}^{1\{y=k-1\}}\phi_k^{1-\sum_{i=1}^{k-1}1\{y=i\}}\\&=\phi_1^{(T(y))_1}\phi_2^{(T(y))_2}\cdots\phi_{k-1}^{(T(y))_{k-1}}\phi_k^{1-\sum_{i=1}^{k-1}(T(y))_i}\\&=\exp\left((T(y))_1\log(\phi_1)+(T(y))_2\log(\phi_2)+\cdots+(T(y))_{k-1}\log(\phi_{k-1})+\left(1-\sum_{i=1}^{k-1}(T(y))_i\right)\log(\phi_k)\right)\\&=\exp\left((T(y))_1\log\left(\frac{\phi_1}{\phi_k}\right)+(T(y))_2\log\left(\frac{\phi_2}{\phi_k}\right)+\cdots+(T(y))_{k-1}\log\left(\frac{\phi_{k-1}}{\phi_k}\right)+\log(\phi_k)\right)\\&=b(y)\exp\left(\eta^TT(y)-a(\eta)\right)\end{align}$$

对于此式有（注意这里的$\phi_k=1-\displaystyle\sum_{i=1}^{k-1}\phi_i$，它并不是一个参数）：

$$\begin{align}\eta^T&=\begin{bmatrix}\log\frac{\phi_1}{\phi_k}&\log\frac{\phi_2}{\phi_k}&\cdots&\log\frac{\phi_{k-1}}{\phi_k}\end{bmatrix}\\a(\eta)&=-\log(\phi_k)\\b(y)&=1\end{align}$$

于是我们将多项分布写成了指数分布族的形式。

正则连接函数为（参数$\eta$关于期望$\phi$的函数）：

$$\eta_i=\log\frac{\phi_i}{\phi_k}\quad (i=1,\cdots,k-1)$$

为了表示方便，我们令$\eta_k=\log\frac{\phi_k}{\phi_k}=0$，求正则响应函数（即正则连接函数的逆，期望$\phi$关于参数$\eta$的函数）：

$$\begin{align}e^{\eta_i}&=\frac{\phi_i}{\phi_k}\\\phi_ke^{\eta_i}&=\phi_i\\\phi_k\sum_{i=1}^ke^{\eta_i}&=\sum_{i=1}^k\phi_i=1\end{align}$$

第三个式子表明$\phi_k=\frac{1}{\sum_{i=1}^ke^{\eta_i}}$，代回第二个式子得到正则响应函数：

$$\phi_i=\frac{e^{\eta_i}}{\sum_{i=1}^ke^{\eta_i}}\quad (i=1,\cdots,k-1)$$

这个函数是由$\eta$到$\phi$的映射，也叫作**softmax**函数。

接下来需要使用假设3，也就是$\eta_i$是输入变量$x$的线性函数，这次我们要使用括号里关于向量$\eta_i$的定义：$\eta_i=\theta_i^Tx,\ \eta\in\mathbb{R}^{k-1},\ i=1,\cdots,k-1$，这里$\theta$仍旧是特征值$x$的系数，有$\theta_1,\cdots,\theta_{k-1}\in\mathbb{R}^{n+1}$（$n$是训练样本所取特征值的个数）。为了方便公式书写，我们依旧令$\theta_k=0$，则多出来一项$\eta_k=\theta_k^Tx=0$，于是在$x$条件下$y$的概率可以表示为：

$$\begin{align}p(y=i\mid x;\theta)&=\phi_i\\&=\frac{e^{\eta_i}}{\sum_{i=1}^ke^{\eta_j}}\\&=\frac{e^{\theta_i^Tx}}{\sum_{j=1}^ke^{\theta_j^Tx}}\tag{2}\end{align}$$

这就是在$y\in\{1,\cdots,k\}$取值下的分类问题模型，也叫作**softmax回归**，是一般化的逻辑回归。

此时的假设函数为：

$$\begin{align}h_\theta(x)&=\mathrm{E}[T(y)\mid x;\theta]\\&=\mathrm{E}\left[\begin{array}{c|c}1\{y=1\}&\\1\{y=2\}&\\\vdots&x;\theta\\1\{y=k-1\}&\end{array}\right]\\&=\begin{bmatrix}\phi_1\\\phi_2\\\vdots\\\phi_{k-1}\end{bmatrix}\\&=\begin{bmatrix}\frac{\exp\left(\theta_1^Tx\right)}{\sum_{j=1}^k\exp\left(\theta_j^Tx\right)}\\\frac{\exp\left(\theta_2^Tx\right)}{\sum_{j=1}^k\exp\left(\theta_j^Tx\right)}\\\vdots\\\frac{\exp\left(\theta_{k-1}^Tx\right)}{\sum_{j=1}^k\exp\left(\theta_j^Tx\right)}\end{bmatrix}\end{align}$$

可以看出，假设函数会对$p(y=i\mid x;\theta)$的每一个$i=1,\cdots,k$做出概率估计（尽管我们仅把$h_\theta(x)$定义在$k-1$维，但显然第$k$个情况的概率估计$p(y=k\mid x;\theta)$可以从$1-\sum_{i=1}^{k-1}\phi_i$算出）。

最后一步就是求拟合参数了。与普通最小二乘、逻辑回归中的推导方法相似，如果训练集中有$m$个训练样本$\left\{(x^{(i)},y^{(i)});i=1,\cdots,m\right\}$，想通过学习算法得到$\theta_i$，则需要同以前一样，先写出似然函数：

$$\begin{align}L(\theta)&=\prod_{i=1}^mp\left(y^{(i)}\mid x^{(i)};\theta\right)\\&=\prod_{i=1}^m\left(\phi_1^{1\left\{y^{(i)}=1\right\}}\phi_2^{1\left\{y^{(i)}=2\right\}}\cdots\phi_k^{1\left\{y^{(i)}=k\right\}}\right)\\&=\prod_{i=1}^m\left(\prod_{l=1}^k\left(\frac{\exp\left(\theta_l^Tx^{(i)}\right)}{\sum_{j=1}^k\exp\left(\theta_j^Tx^{(j)}\right)}\right)^{1\left\{y^{(i)}=l\right\}}\right)\end{align}$$

再对似然函数取对数：

$$\begin{align}\mathscr{l}(\theta)&=\log L(\theta)\\&=\sum_{i=1}^m\log\prod_{l=1}^k\left(\frac{\exp\left(\theta_l^Tx^{(i)}\right)}{\sum_{j=1}^k\exp\left(\theta_j^Tx^{(j)}\right)}\right)^{1\left\{y^{(i)}=l\right\}}\end{align}$$

在上面的推导中我们使用了$(2)$式给出的$p(y=i\mid x;\theta)$的定义，剩下的就是计算$\mathscr{l}(\theta)$中参数$\theta$的最大似然估计了，计算过程可以使用前面介绍的梯度上升或牛顿法。（如果取$n$个特征值，则向量$x\in\mathbb{R}^{n+1}$（含截距项$x_0=1$），有$\theta_j\in\mathbb{R}^{n+1},\ j=1,\cdots,k-1$（也就是对每一个可能的$y_j$都有一套$\theta_j$拟合，线性无关的共$k-1$个），如果愿意的话可以把$\theta_j,\ i=1,\cdots,k-1$组成一个矩阵，即系数矩阵$\theta\in\mathbb{R}^{(n+1)\times(k-1)}$。）

一般线性模型建模过程可以概括为：

1. 根据训练集$x^{(i)},y^{(i)}$选择概率分布模型，参数为$\phi$；
2. 将该分布写为指数分布族的形式，参数为$\eta$；
3. 可以得到正则响应函数$g(\eta)=\mathrm{E}[T(y);\eta]$；
4. 将$\eta=\theta^Tx$带入正则响应函数得到假设函数$h_\theta(x)=g(\theta^Tx)$；
5. 根据模型的概率解释得到似然函数$L(\theta)=p(y^{(i)}\mid x^{(i)};\theta)$（根据假设函数得到）；
6. 取合适的$\theta$使似然函数最大化。


## Практическая задача №1 по курсу "Биофизика возбудимой клетки"

В этой задаче мы рассматриваем формализм Ходжкина-Хаксли, исследуем влияние концентраций ионов, температуры и потенциалчувствительности токов на характеристики ответа модельного нейрона.

Мы используем средства языка `Python`. Рекомендуется установить его в виде дистрибутива [Anaconda](https://www.continuum.io/downloads).

Для формулирования и решения систем дифференциальных уравнений будем использовать и модуль `PyDSTool`. Его можно установить командой `pip install pydstool`.

```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```python
style.use('ggplot')
rc('font', family='ubuntu') # эта строчка нужна в Ubuntu, чтобы русские буквы отображались в рисунках
```

```python
from __future__ import division
```

```python
import PyDSTool as dst
from PyDSTool import Exp, Pow, Log, Par, Var, Fun
```

## Ионные равновесия

Большинство зависимостей мы будем задавать в символьном виде, т.е. в виде выражений. Для этого нам нужно объявить несколько параметров и переменных.

```python
V = Var('V')

Ci = Par('Ci')
Co = Par('Co')

nai = Par('nai') # концентрация натрия внутри клетки
nao = Par('nao') # концентрация натрия снаружи клетки
ki = Par('ki')   # концентрация калия внутри клетки
ko = Par('ko')   # концентрация калия снаружи клетки

F = 96485 # Кл/моль
R = 8314.4  # Universal gaz constant, мДж/(моль * Кельвин)

celsius = dst.Par('celsius')
T = 273 + celsius

charge = dst.Par('charge')

```

**Электрохимический потенциал**, из которого выводятся остальные выражения:

\begin{equation}
\bar\mu = \mu_0 + RT\ln c + zF\phi
\end{equation}

**Уравнение Нернста**, описывающее равновесный потенциал для отдельного иона ($\Delta \mu = 0$):

\begin{equation}
E =\Delta\phi = \phi_i - \phi_o= \frac{RT}{zF}\ln\frac{c_o}{c_i}
\end{equation}

**Поток вещества через мембрану** (уравнение электродиффузии Нернста-Планка):
\begin{equation}
-J = u c\frac{d\bar\mu}{dx} = uRT\frac{dc}{dx} + uczF\frac{d\phi}{dx}
\end{equation}

**В приближении постоянного поля**, т.е. $\frac{d\phi}{dx} = -\frac{\Delta\phi}{h} = -\frac{V}{h}$, где $h$ -- толщина мембраны, а $V$ -- трансмембранный потенциал, поток ионов описывается уравнением Гольдмана-Ходжкина-Катца:

\begin{equation}
J = P\frac{zFV}{RT}\frac{c_i - c_o e^{-\frac{zFV}{RT}}}{1 - e^{-\frac{zFV}{RT}}},
\end{equation}
где $P$ -- проницаемость мембраны для данного иона. Ток, переносимый потоком ионов: $I = zFJ$.

**Задание:** Выведите уравнение для потока ионов (ГХК) из уравнения Нернста-Планка

```python
# Cимвольные выражения для равновесного потенциала и для потока:

nernst = Fun(R*T*Log(Co/Ci)/F, ['Ci','Co'], 'nernst')

# (для простоты, с единичным зарядом)
GHK = Fun((F**2*V/(R*T))*(Ci - Co*Exp(-F*V/(R*T)))/(1 - Exp(-F*V/(R*T))),
              ['Ci','Co'], 'GHK')
```

Выразим потенциалы реверсии для натрия и калия через их внутри- и экстраклеточные концентрации

```python
ena =  nernst(nai,nao) # потенциал реверсии для Na
ek = nernst(ki, ko)    # потенциал реверсии для K

elk = Par('elk') # потенциал реверсии для утечки

```

## Формализм Ходжкина-Хаксли

В формализме Ходжкина-Хаксли *мгновенный* ток имеет линейную зависимость от потенциала вида $i=g(V-V_r)$, где $g$ -- проводимость, $V_r$ -- потенциал реверсии для данного тока. Проводимость состоит из *максимальной* проводимости, умноженной на долю т.н. "воротных частиц", находящихся в открытом состоянии: $g = \bar{g}x^ay^b$. Воротные частицы -- это формальная абстракция, отражающая кинетику открывания-закрывания ансамбля каналов.

В классической модели Ходжкина-Хаксли натриевый ток имеет две воротные частицы: активационную $x:=m, a=3$ и инакцивационную $y:=h, b=1$. Калиевый ток описывается одной воротной частицей: $x:=n, a=4$, $y:=\_, b=0$. В других нейронах могут быть каливевые токи с инактивацией, натриевые токи без инактивации, а также смешанные токи.

Кинетика воротных частиц описывается линейными дифференциальными уравнениями вида $\dot{x} = \alha(1-x)-\beta x$.
Константы скорости $\alpha$ и $\beta$ нелинейно зависят от потенциала, Ходжкин и Хаксли аппроксимировали их следующими зависимостями:

```python
# "Стандартный" Ходжкин-Хаксли
malpha = 0.1*(V+40.0)/(1 - Exp(-(V+40)/10.))
mbeta = 4*Exp(-(V+65)/18.)

halpha = 0.07*Exp(-(V+65)/20.)
hbeta = 1/(1 + Exp(-(V+35)/10.))

nalpha = 0.01*(V+55)/(1 - Exp(-(V+55)/10.0))
nbeta = 0.125*Exp(-(V+65)/80.)

# Эквивалентная запись:
minf = 1/(1.0 + mbeta/malpha)
hinf = 1/(1.0 + hbeta/halpha)
ninf = 1/(1.0 + nbeta/nalpha)

mtau = 1./(malpha + mbeta)
htau = 1./(halpha + hbeta)
ntau = 1./(nalpha + nbeta)
```

```python
vtest = linspace(-89, 45, 250)
kwx = dict(Exp='np.exp', Pow='np.power')

def eval_at(vstring, obj):
    return eval(str(obj.eval(V=vstring, **kwx)))
```

```python
plot(vtest, eval_at('vtest', minf), label='$m_\infty$')
plot(vtest, eval_at('vtest', hinf), label='$h_\infty$')
plot(vtest, eval_at('vtest', ninf), label='$n_\infty$')
legend(loc='best')
xlabel(u'Трансмембранный потенциал [мВ]')
ylabel(u'равновесное коэффициент ')
```

```python
plot(vtest, eval_at('vtest', mtau), label='$\\tau_m$')
plot(vtest, eval_at('vtest', htau), label='$\\tau_h$')
plot(vtest, eval_at('vtest', ntau), label='$\\tau_n$')

legend(loc='best')
xlabel(u'Трансмембранный потенциал [мВ]')
ylabel(u'константа времени, [мс]')
```

Объявим наши воротные переменные:

```python
m = Var('m')
h = Var('h')
n = Var('n')
```

Дифференциальные уравнения для воротных переменных однотипны:

```python
mdot = (minf-m)/mtau
hdot = (hinf-h)/htau
ndot = (ninf-n)/ntau
```

**Мгновенные токи в формализме Ходжкина-Хаксли линейно зависят от потенциала (в отличие от приближения постоянного поля)**

```python
gnabar = Par('gnabar') # макс. проводимость по Na+
gkbar = Par('gkbar')   # макс. проводимость по K+
glk = Par('glk')
```

```python
ina = gnabar*m**3*h*(V-ena)
ik = gkbar*n**4*(V-ek)

ileak = glk*(V-elk)
```

Сложим все ионные токи, опосредуемые ионными механизмами:

```python
Imech = ina + ik + ileak
```

```python
Istim = Par('Istim')
```

В классической модели Ходжкина-Хаксли:
 - Ena = 50 мВ
 - Ek = -77 мВ
 - Elk = -54.4 мВ
 - темература: 6 $^\circ$C

Подберем соответствующие концентрации Na+ и К+ снаружи и внутри клетки:

```python
print "Ek =", ek.eval(ki=75, ko=3,celsius=6)
print "Ena=", ena.eval(nai=20, nao=155,celsius=6)
```

    Ek = -77.38905428990292
    Ena= 49.23116679430405

Таблица параметров, которую будем использовать:

```python
parameters = dict(nai = 20., nao = 155., gnabar = 120.0,
                  ki = 75.,  ko = 3.,    gkbar = 36.0,
                  elk = -54.5,          glk  = 0.3,
                  celsius = 6., Istim=0)
```

Определим несколько вспомогательных функций, которые помогут нам задать начальные условия таким образом, чтобы при произвольно выбранном $V$ все воротные переменные имели свое стационарное значение при этом трансмембранном потенциале.

```python

steady_gates = dict(m=minf,h=hinf,n=ninf)
def evaluate_at_v(v, obj, parameters):
    return eval(str(obj.eval(V=v, **parameters)))

def make_initials(v_init, parameters):
    initials = {k:evaluate_at_v(v_init, gate, parameters)
            for k,gate in steady_gates.items()}
    initials['V'] = v_init
    return initials
```

Зададим описание (формулировку) модели:

```python
v_init = -65.
#Iramp = Var('Iramp')
hhmodel = dst.args(name = 'HH',
                   pars = parameters,
                   fnspecs={'Iramp':(['tx'], 'min(120,0.1*tx)'),
                           'rpulse':(['tx','width'], '0.5*(1 + tanh(100*(tx)) * tanh(100*(-tx+width)))')},
                   auxvars=['Is'],
                   varspecs={'V':'Istim*rpulse(t-100,400)'-Imech,
                             'm':mdot, 'h':hdot, 'n':ndot,
                             'Is':'Istim*rpulse(t-100,400)'},
                   tdomain=[0,1000],
                   xdomain = {'V':[-100, 50], 'm': [0, 1], 'h': [0, 1], 'n': [0, 1]},
                   ics=make_initials(v_init, parameters))

```

```python
hh_ode = dst.Generator.Vode_ODEsystem(hhmodel)
```

```python
def multi_pulse_protocol(odeset, stims):
    "Стимулируем модель  толчками тока разной амплитуды и строим V(t)"
    f,axs = subplots(len(stims)+1,1, sharex=True, figsize=(14,8));
    max_stim = -1e12
    min_stim = 1e12
    for k,s in enumerate(stims):
        odeset.set(pars=dict(Istim = s))

        traj = odeset.compute('multipulse')
        pts = traj.sample(dt=0.1)
        axs[k].plot(pts['t'], pts['V'], label='I=%1.2f uA/cm^2'%s)
        axs[k].legend()
        axs[k].set_ylabel('V, mV')
        axs[k].set_ylim(-80, 40)
        axs[-1].plot(pts['t'], pts['Is'],'b')
        min_stim = min(min_stim, np.min(pts['Is']))
        max_stim = max(max_stim, np.max(pts['Is']))
    setp(axs[-1],  ylabel='Istim, uA/cm2', xlabel=u'время, мс', xlim=(90, 600), ylim=(min_stim-1, max_stim+1))
```

```python
multi_pulse_protocol(hh_ode, [2,4,8])
```

Обратите внимание на подпороговые колебания потенциала при Istim=4 мкА/см^2

**Задание:** Определите пороговые значения тока стимуляции, при котором возникают: (1) одиночные ПД, (2) ритмический ответ.

**Задание:** Зависит ли частота ритмического ответа от амплитуды возбуждающего тока?

### Спайк "отскока"

В момент убирания длительной гиперполяризации может возникать ПД, так называемый спайк "отскока":

```python
multi_pulse_protocol(hh_ode, [-1,-5])
```

**Задание:** Определите пороговые значения амплитуды и длительности гиперполяризующего тока, при котором возникает спайк отскока.

### Влияние концентрации $[K^+]_o$
Проведем аналогичный опыт при увеличенной концентрации калия снаружи клетки,   $[K^+]_o=30$mM

```python
hh_ode.set(pars={'ko':5})

multi_pulse_protocol(hh_ode, [-2,0,2])
```

**Задание:** определите максимальную $[K^+]_o$, при которой клетка **не** находится в режиме автоколебаний в отсутствие стимула. Как она зависит от $[K^+]_i$?

### Влияние концентрации $[Na^+]_i$
Проведем аналогичный опыт при увеличенной концентрации натрия внутри клетки,   $[Na^+]_i=30$mM

```python
hh_ode.set(pars={'ko':3, 'nai':30})
multi_pulse_protocol(hh_ode, stims=[2,4,8,16])
```

**Задание:** Как изменились порог возбуждения и форма ПД?

```python
# Вернем исходные значения:
hh_ode.set(pars=dict(Istim = 0, ko=3.0, nai=20.0))
```

### Приближение постоянного поля [\*]

Задание повышенной сложности:

Формализм Ходжкина-Хаксли подразумевает *линейную* зависимость тока от сдвига потенциала (закон Ома). Уравнение же для потока ионов Гольдмана-Ходжкина-Катца в приближении постоянного поля диктует нелинейную зависимость тока от трансмембранного потенциала (см. выше).

 Замените выражения для ионных токов с вида $i = \bar{g}x^ay^b(V-V_r)$ на вид $i = \bar{P}x^ay^b\frac{z^2F^2V}{RT}\frac{c_i - c_o e^{-\frac{zFV}{RT}}}{1 - e^{-\frac{zFV}{RT}}}$, где $\bar{P}$ -- это максимальная проницаемость для данного иона.

 Хорошо ли совпадают две кривые ($I_\infty(V)$) для ионов Na+ и К+ при физиологических значениях потенциала и концентраций?

 Как изменится согласие между двумя типами зависимости, если варьировать $[K^+]_o$ и $[K^+]_i$?

 Существенно ли изменится поведение и характеристики возбудимости модели?

## Влияние температуры

Ходжкин и Хаксли проводили опыты и описывали кинетику токов при температуре 6 $^\circ$C. Как изменится поведение системы при изменении температуры? Это важно знать, поскольку во многих случаях параметры токов в разных типах нейронов определяют *in vitro* при комнатных условиях, но потом эти данные нужно перенести на реальную температуру тела животного. Как это сделать?

Очевидно, что температура входит в уравнение Нернста для равновесного потенциала. Однако, это не единственное, что меняется при изменении температуры. Поскольку реакции протекают быстрее при более высоких температурах, это должно отразиться на константах скорости кинетик воротных частиц $\alpha_x$ и $\beta_x$ (или, что эквивалентно, в константе скорости $\tau_x$.

Обычно, для описания этой зависимости вводят т.н. коэффициент $q_{10}$, который описывает, во сколько раз ускоряется кинетика воротных частиц при увеличении температуры на 10К: $\tau(V,T) = \tau(V,T_0)q_{10}^\frac{T_0-T}{10}$. Как правило, берут $q_{10}\approx 3$

Модифицируем дифференциальные уравнения для воротных переменных:

```python
QF = Pow(3, 0.1*(celsius-6.0))
mdott = QF*(minf-m)/mtau
hdott = QF*(hinf-h)/htau
ndott = QF*(ninf-n)/ntau
```

```python
v_init = -65.
#Iramp = Var('Iramp')
hhmodel_t = dst.args(name = 'HH',
                   pars = parameters,
                   fnspecs={'Iramp':(['tx'], 'min(120,0.1*tx)'),
                           'rpulse':(['tx','width'], '0.5*(1 + tanh(100*(tx)) * tanh(100*(-tx+width)))')},
                   auxvars=['Is'],
                   varspecs={'V':'Istim*rpulse(t-100,400)'-Imech,
                             'm':mdott, 'h':hdott, 'n':ndott,
                             'Is':'Istim*rpulse(t-100,400)'},
                   tdomain=[0,1000],
                   xdomain = {'V':[-100, 50], 'm': [0, 1], 'h': [0, 1], 'n': [0, 1]},
                   ics=make_initials(v_init, parameters))
hh_ode_t = dst.Generator.Vode_ODEsystem(hhmodel_t)
```

```python
stim = 15

hh_ode_t.set(pars=dict(Istim = stim, celsius=6.))

traj_6 = hh_ode_t.compute('tempdep')
pts_6 = traj_6.sample(dt=0.01)

hh_ode_t.set(pars=dict(Istim = stim, celsius=23.))
traj_23 = hh_ode_t.compute('tempdep')
pts_23 = traj_23.sample(dt=0.01)

figure(figsize=(14,6))
plot(pts_6['t'], pts_6['V'],label='t=6$^\circ$C')
plot(pts_23['t'], pts_23['V'], label='t=23$^\circ$C')

legend(loc='upper left')
xlim(95,120)
xlabel('time, ms')
ylabel('V, mV')
title(u'Влияние температуры. Istim=%d uA/cm^2'%stim)
```

**Задание:** (1) Объясните наблюдаемые изменения. (2) Изменился ли порог возбуждения?

## Альтернативные варианты параметризации

При моделировании зависимости кинетик воротных переменных Ходжкин и Хаксли исходили из представления о переходе некоторых частиц из закрытого состояния в открытое и обратно с константами $\alpha$ и $\beta$, поэтому они стремились параметризовать именно их зависимости от потенциала $\alpha(V)$, $\beta(V)$. В этом случае кинетика воротной частицы $x$ описывается как $\dot x = \alpha_x(V) (1-x) - \beta_x(V) x$.

При экспериментальных измерениях обычно удобнее описывать кинетику другим (эквивалентным) способом: $\tau_x(V)\dot x = x_\infty(V) - x$, где распределение воротных частиц стремится к равновесному при данном потенциале значению $x_\infty(V)$ с константой времени $\tau_x(V)$. Существует несколько подходов к тому, какими функциями описывать $x_\infty(V)$ и $\tau_\infty(V)$. В наиболее простом варианте зависимость $x_\infty(V)$ описывается функцией Больцмана, а $\tau_x(V)$ -- гауссианом. Такой способ параметризации удобен, потому что позволяет напрямую задавать потенциал, при котором половина "ворот" находится в открытом состоянии, а также крутизну этой зависимости от потенциала.

Функция Больцмана:

\begin{equation}
f(V) = \frac{1}{1 + \exp\frac{V_{1/2}-V}{k}}
\end{equation}

```python
boltzman = Fun(1./(1. + Exp(('Vhalf'- V)/'bk')), ['Vhalf','bk'], 'boltzman')

minf2 = boltzman(-40., 9.)
hinf2 = boltzman(-62., -7.)

ninf2 = boltzman(-53., 15.)
```

```python
gausstau = Fun('Cbase + Camp*Exp(-(Vmax-V)**2/sigma**2)', ['Vmax', 'sigma', 'Camp','Cbase'], 'gausstau')

mtau2 = gausstau(-38., 30., 0.46, 0.04)
htau2 = gausstau(-67., 20., 7.4, 1.2)
ntau2 = gausstau(-79., 50., 4.7, 1.1)
```

```python
plot(vtest, eval_at('vtest', minf2), label='$m_\infty$')
plot(vtest, eval_at('vtest', hinf2), label='$h_\infty$')
plot(vtest, eval_at('vtest', ninf2), label='$n_\infty$')
legend(loc='best')
xlabel(u'Трансмембранный потенциал [мВ]')
ylabel(u'равновесное коэффициент ')
```

**Задание:** Сравните с исходными графиками для кинетики воротных переменных $x_\infty$ и $\tau_x$) в оригинальной модели Х-Х и при альтернативном способе параметризации.

Зададим модель с альтернативной параметризацией и приготовим систему уравнений:

```python
hhmodela = dst.args(name = 'HHa',
                   pars = parameters,
                   fnspecs={'Iramp':(['tx'], 'min(120,0.1*tx)'),
                           'rpulse':(['tx','width'], '0.5*(1 + tanh(100*(tx)) * tanh(100*(-tx+width)))')},
                   auxvars=['Is'],
                   varspecs={'V':'Istim*rpulse(t-100,400)'-Imech,
                             'm':(minf2-m)/mtau2, 'h':(hinf2-h)/htau2, 'n':(ninf2-n)/ntau2,
                             'Is':'Istim*rpulse(t-100,400)'},
                   tdomain=[0,1000],
                   xdomain = {'V':[-100, 50], 'm': [0, 1], 'h': [0, 1], 'n': [0, 1]},
                   ics=make_initials(v_init, parameters))
hh_ode_a = dst.Generator.Vode_ODEsystem(hhmodela)
```

Сравним динамику модели с исходной:

```python
multi_pulse_protocol(hh_ode_a, stims=[2,4,8])
```

**Задание:** Как изменились свойства возбудимости модели?

## Интеграторы и резонаторы

Во многих нейронах К-токи менее чувствительны к потенциалу (активируются при более деполяризованной мембране), чем в оригинальной модели Ходжкина-Хаксли. Сделаем модель такого нейрона и исследуем ее свойства.

```python
Vshift = 30
ninf3 = boltzman(-53+Vshift, 15.)
ntau3 = gausstau(-79+Vshift, 50., 4.7, 1.1)

plot(vtest, eval_at('vtest', ninf2), label='$n_\infty$')
plot(vtest, eval_at('vtest', ninf3), label='$nHT_\infty$')
legend()
xlabel('V, mV')
title(u"Делаем высокопороговый К-ток ($n_\infty$)")
```

```python
plot(vtest, eval_at('vtest', ntau2), label='$\\tau_n$')
plot(vtest, eval_at('vtest', ntau3), label='$\\tau_nHT$')
legend()
xlabel('V, mV')
title(u"Делаем высокопороговый К-ток ($\\tau_n$)")
```

```python
v_init = -65.
#Iramp = Var('Iramp')
hhmodel_ht = dst.args(name = 'HH-ht',
                   pars = parameters,
                   fnspecs={'Iramp':(['tx'], 'min(120,0.1*tx)'),
                           'rpulse':(['tx','width'], '0.5*(1 + tanh(100*(tx)) * tanh(100*(-tx+width)))')},
                   auxvars=['Is'],
                   varspecs={'V':'Istim*rpulse(t-100,400)'-(ina + ik + ileak),
                             'm':(minf2-m)/mtau2, 'h':(hinf2-h)/htau2, 'n':(ninf3-n)/ntau3,
                             'Is':'Istim*rpulse(t-100,400)'},
                   tdomain=[0,1000],
                   xdomain = {'V':[-100, 50], 'm': [0, 1], 'h': [0, 1], 'n': [0, 1]},
                   ics=make_initials(v_init, parameters))
hh_ode_ht = dst.Generator.Vode_ODEsystem(hhmodel_ht)
```

Из-за уменьшения К проводимости около потенциала покоя, такой нейрон находится в автоколебательном режиме.

```python
multi_pulse_protocol(hh_ode_ht, stims=[0])
```

Скомпенсируем уменьшившуюся проводимость по К изменением равновесного потенциала и проводимости утечки так, при отсутствии стимуляции не наблюдались ПД:

```python
hh_ode_ht.set(pars={'glk':0.5, 'elk':-70},)
multi_pulse_protocol(hh_ode_ht, stims=[0])
```

Сдвиг порога К-тока приводит к изменению типа возбудимости нейрона. Теперь варьирование амплитуды стимула приводит к
изменению **задержки** первого ПД и **частоты** ритмического ответа, что не наблюдалось при низкопороговом токе.

```python
multi_pulse_protocol(hh_ode_ht, stims=[0.88, 0.9, 0.92, 1.0, 2])
```

**Задание:** (1) Объясните наблюдаемые изменения. (2) Покажите, что аналогичные изменения в проводимости и потенциале реверсии в случае низкопорогового тока не приведут изменению типа возбудимости. (3) Подумайте, как разные типы возбудимости влияют на потенциальные способы обработки информации нейроном.


## Ladder and DifferentialAddition formula unrolling

```python
from pyecsca.ec.point import Point, InfinityPoint
from pyecsca.ec.mod import Mod, SymbolicMod
from pyecsca.ec.model import ShortWeierstrassModel, EdwardsModel, MontgomeryModel
from pyecsca.ec.coordinates import AffineCoordinateModel
from pyecsca.ec.context import local, DefaultContext
from pyecsca.ec.formula import LadderFormula, DifferentialAdditionFormula
from pyecsca.misc.cfg import getconfig
from sympy import symbols, Mod as SympyMod
import warnings
```

```python
def remove_mod(expression):
    if isinstance(expression, SympyMod):
        return remove_mod(expression.args[0])
    if not expression.args:
        return expression
    args = []
    for arg in expression.args:
        if isinstance(arg, SympyMod):
            arg = remove_mod(arg.args[0])
        else:
            arg = remove_mod(arg)
        args.append(arg)
    return expression.func(*args)
```

## Short Weierstrass

```python
model = ShortWeierstrassModel()
coords = model.coordinates["xz"]
acoords = AffineCoordinateModel(model)
xd, zd, xp, zp, xq, zq = symbols("X1 Z1 X2 Z2 X3 Z3")
a, b = symbols("a b")
p = 53  # This number is unused, but pyecsca needs some prime to be the modulus even for symbolic exec
getconfig().ec.unsatisfied_formula_assumption_action = "warning"
```

```python
PmQ = Point(coords, X=SymbolicMod(xd, p), Z=SymbolicMod(zd, p))
P = Point(coords, X=SymbolicMod(xp, p), Z=SymbolicMod(zp, p))
Q = Point(coords, X=SymbolicMod(xq, p), Z=SymbolicMod(zq, p))
```

```python
for name, formula in coords.formulas.items():
    if not isinstance(formula, (LadderFormula, DifferentialAdditionFormula)):
        continue
    print(f"--- {name} ---")
    with warnings.catch_warnings(record=True) as w:
        with local(DefaultContext()) as ctx:
            res = formula(p, PmQ, P, Q, a=SymbolicMod(a, p), b=SymbolicMod(b, p))[0]
        action, _ = ctx.actions.get_by_index([0])
        for name, values in action.intermediates.items():
            for value in values:
                if name in formula.outputs:  # Remove this test if you want the output of all intermediates.
                    print(f"{name} =", remove_mod(value.value.x))
    if w:
        print(w[0].message)
    print()
```

## Edwards

```python
model = EdwardsModel()
coords = model.coordinates["yz"]
acoords = AffineCoordinateModel(model)
yd, zd, yp, zp, yq, zq = symbols("Y1 Z1 Y2 Z2 Y3 Z3")
c, r, d = symbols("c r d")
p = 53  # This number is unused, but pyecsca needs some prime to be the modulus even for symbolic exec
getconfig().ec.unsatisfied_formula_assumption_action = "warning"
```

```python
PmQ = Point(coords, Y=SymbolicMod(yd, p), Z=SymbolicMod(zd, p))
P = Point(coords, Y=SymbolicMod(yp, p), Z=SymbolicMod(zp, p))
Q = Point(coords, Y=SymbolicMod(yq, p), Z=SymbolicMod(zq, p))
```

```python
for name, formula in coords.formulas.items():
    if not isinstance(formula, (LadderFormula)):
        continue
    print(f"--- {name} ---")
    with warnings.catch_warnings(record=True) as w:
        with local(DefaultContext()) as ctx:
            res = formula(p, PmQ, P, Q, c=SymbolicMod(c, p), r=SymbolicMod(r, p), d=SymbolicMod(d, p))[0]
        action, _ = ctx.actions.get_by_index([0])
        for name, values in action.intermediates.items():
            for value in values:
                if name in formula.outputs:  # Remove this test if you want the output of all intermediates.
                    print(f"{name} =", remove_mod(value.value.x))
    if w:
        print(w[0].message)
    print()
```

## Montgomery

```python
model = MontgomeryModel()
coords = model.coordinates["xz"]
acoords = AffineCoordinateModel(model)
xd, zd, xp, zp, xq, zq = symbols("X1 Z1 X2 Z2 X3 Z3")
a, b = symbols("a b")
p = 53  # This number is unused, but pyecsca needs some prime to be the modulus even for symbolic exec
getconfig().ec.unsatisfied_formula_assumption_action = "warning"
```

```python
PmQ = Point(coords, X=SymbolicMod(xd, p), Z=SymbolicMod(zd, p))
P = Point(coords, X=SymbolicMod(xp, p), Z=SymbolicMod(zp, p))
Q = Point(coords, X=SymbolicMod(xq, p), Z=SymbolicMod(zq, p))
```

```python
for name, formula in coords.formulas.items():
    if not isinstance(formula, (LadderFormula, DifferentialAdditionFormula)):
        continue
    print(f"--- {name} ---")
    with warnings.catch_warnings(record=True) as w:
        with local(DefaultContext()) as ctx:
            res = formula(p, PmQ, P, Q, a=SymbolicMod(a, p), b=SymbolicMod(b, p))[0]
        action, _ = ctx.actions.get_by_index([0])
        for name, values in action.intermediates.items():
            for value in values:
                if name in formula.outputs:  # Remove this test if you want the output of all intermediates.
                    print(f"{name} =", remove_mod(value.value.x))
    if w:
        print(w[0].message)
    print()
```

```python

```


# 4.2 Finding interactions and regions of interactions

### Interaction model

$Y = 0.2X_{1} - 5X_{2} + 10X_2\mathbb{1}_{X_{3} \geq 0} + \varepsilon$,

$\varepsilon \overset{\text{iid}}{\sim} \mathcal{N}(0,1),\quad X_{1},X_{2},X_{3}\overset{\text{iid}}{\sim}U(-1,1) $

```python
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
%matplotlib inline

import xgboost as xgb
from sklearn.metrics import mean_squared_error

from xgboost import XGBRegressor
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import StratifiedKFold
from sklearn.ensemble import GradientBoostingRegressor, GradientBoostingClassifier

import seaborn as sns
from pycebox.ice import ice, ice_plot
from matplotlib.cm import seismic

# import sympy as sym
from sympy import *
```

```python
x1,x2,x3,y,yprime = [],[],[],[],[]
N = 1000

for i in range(0,N):
    X1 = np.random.uniform(-1, 1, size = 1)
    X2 = np.random.uniform(-1, 1, size = 1)
    X3 = np.random.uniform(-1, 1, size = 1)
    eps = np.random.uniform(0, 1, size = 1)
    Y = 0.2*X1 - 5*X2 + 10*X2*(X3>0) + eps

    x1.append(X1)
    x2.append(X2)
    x3.append(X3)

    X3 = Symbol('X3')
    der_y = lambdify(X3, Y)
    y_der = der_y(X3)

    y.append(Y)
    yprime.append(y_der)
```

```python
yprime = np.array(yprime)
data = np.concatenate([x1,x2,x3,y,yprime], axis = 1)
df = pd.DataFrame(data, columns=['X1','X2','X3','Y','Yprime'])
```

```python
df
```

      X1
      X2
      X3
      Y
      Yprime

      0
      -0.338093
      -0.084275
      -0.595002
      0.407253
      0.407253

      1
      0.582738
      0.273395
      -0.019854
      -1.152591
      -1.152591

      2
      -0.458860
      -0.893905
      -0.873435
      4.578279
      4.578279

      3
      0.748917
      -0.221966
      0.620604
      -0.509440
      -0.509440

      4
      0.309883
      -0.105869
      0.811016
      0.516461
      0.516461

      ...
      ...
      ...
      ...
      ...
      ...

      995
      0.990338
      -0.944645
      -0.064512
      5.097646
      5.097646

      996
      -0.580384
      0.775755
      0.897354
      4.314659
      4.314659

      997
      0.222195
      0.517397
      -0.702898
      -2.252765
      -2.252765

      998
      0.879646
      -0.899246
      0.753643
      -3.934264
      -3.934264

      999
      -0.129085
      -0.869703
      -0.652349
      4.964366
      4.964366

1000 rows × 5 columns

```python
X = df[['X1','X2','X3']]
y = df.Y
yprime = df.Yprime
```

```python

```

```python

```

```python
ice_df = ice(X, 'X3', clf.predict)
```

```python
clf = GradientBoostingRegressor(n_estimators=500, learning_rate=0.1,
        max_depth=3, random_state=0).fit(X,y)
```

```python
from sklearn.model_selection import cross_val_score
scores = cross_val_score(clf, X, y, cv=5)
```

```python
scores
```

    array([0.96927958, 0.96162799, 0.96478913, 0.93396571, 0.96786976])

```python
ice_df.head()
```

      data_X3
      -0.997172
      -0.996564
      -0.992282
      -0.992134
      -0.990192
      -0.989683
      -0.988149
      -0.988017
      -0.982979
      -0.982731
      ...
      0.983570
      0.984994
      0.986717
      0.989341
      0.993841
      0.995099
      0.997311
      0.997643
      0.998421
      0.999855

      X1
      0.922825
      -0.777871
      0.303940
      0.366433
      -0.912816
      0.154280
      -0.827577
      -0.860957
      0.590085
      0.422321
      ...
      0.054894
      0.363729
      -0.217623
      -0.461785
      0.293790
      0.382825
      -0.395540
      0.401034
      -0.307885
      0.433182

      X2
      -0.078087
      0.037986
      0.608453
      0.525146
      0.530517
      -0.551559
      -0.279662
      -0.912598
      0.539754
      -0.154074
      ...
      0.604182
      -0.104176
      0.218235
      -0.205297
      -0.590554
      -0.942947
      -0.966663
      -0.590365
      -0.454333
      0.126978

      X3

      -0.997172
      1.169619
      0.371781
      -2.687148
      -1.923469
      -2.665161
      3.429358
      1.862105
      5.048992
      -2.056136
      1.542147
      ...
      -2.535073
      1.260133
      -0.527619
      1.802426
      3.469661
      5.917863
      7.144166
      3.325201
      2.894303
      -0.018702

      -0.996564
      1.169619
      0.371781
      -2.687148
      -1.923469
      -2.665161
      3.429358
      1.862105
      5.048992
      -2.056136
      1.542147
      ...
      -2.535073
      1.260133
      -0.527619
      1.802426
      3.469661
      5.917863
      7.144166
      3.325201
      2.894303
      -0.018702

      -0.992282
      1.176570
      0.376683
      -2.678663
      -1.914984
      -2.668744
      3.436310
      1.865474
      5.043876
      -2.047651
      1.549098
      ...
      -2.526589
      1.267085
      -0.519134
      1.805795
      3.476612
      5.924814
      7.147535
      3.332152
      2.901255
      -0.010217

      -0.992134
      1.258951
      0.474365
      -2.588997
      -1.825318
      -2.597018
      3.518691
      1.961475
      5.108318
      -1.957985
      1.631479
      ...
      -2.436923
      1.349466
      -0.435072
      1.901797
      3.558993
      6.007196
      7.243536
      3.414534
      3.003991
      0.073844

      -0.990192
      1.149151
      0.368012
      -2.708886
      -1.945207
      -2.716907
      3.408891
      1.851675
      5.028019
      -2.077874
      1.521679
      ...
      -2.556812
      1.239666
      -0.541425
      1.791996
      3.449193
      5.926897
      7.163238
      3.304733
      2.894190
      -0.032508

5 rows × 1000 columns

```python
fig, (data_ax, ice_ax) = plt.subplots(ncols=2, sharex=True, sharey=True, figsize=(16, 6))

data_ax.scatter(X.X3, y, c='k', alpha=0.5);

data_ax.set_xlim(-1.05, 1.05);
data_ax.set_xlabel('$X_3$');

data_ax.set_ylabel('$\hat{y}$');

data_ax.set_title('Data');

ice_plot(ice_df, frac_to_plot=0.05, plot_pdp=True,
         plot_points=True,
         c='k', alpha=0.25,
         ax=ice_ax)

ice_ax.set_xlabel('X_3')

ice_ax.set_ylabel('partial yhat')

ice_ax.set_title('ICE (10 curves)')
```

```python

```


Solution to: [Day 6: The Central Limit Theorem I](https://www.hackerrank.com/challenges/s10-the-central-limit-theorem-1/problem)

Table of Contents

- Table of Contents
- Central Limit Theorem
- Solution
    - Imports
    - Input
    - Cumulative Distribution Function
    - Main

```javascript
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')
```

    <IPython.core.display.Javascript object>

# Central Limit Theorem

Given a large enough Sample, N, *the distribution of the sample mean will approach normal distribution*. This holds for a sample of random variables with a finite standard deviation.

Let {X1, X2, X3 ... Xn} be a random data set of size N, that is, a sequence of independent and identically distributed
random variables drawn from distributions of expected values given by µ and finite variances given by σ².

The sample average is:
\begin{equation}
\large
s_{n} =
\frac
{\sum_{i} X_{i}}
{N}
\end{equation}

For large n, the distribution of sample sums $S_{n}$ is close to normal distribution $\mathcal{N}$(µ', σ') where:
- µ' = n * µ
- σ = n ** (1/2) * σ

# Solution

## Imports

```python
from typing import Tuple
import math
```

## Input

```python
def get_input() -> Tuple[int, int, int, int]:
	"""Returns input for Central Limit Theorem I

	Returns:
		Tuple[int, int, int, int]: max_weight, boxes, mean, sd
	"""
	return 9800, 49, 205, 15
```

## Cumulative Distribution Function

```python
def erf(x: int) -> float:
	"""Returns probability that random number in range [-x, x]

	Args:
		x (int): input of error function

	Returns:
		float: error for x

	NOTE:
		- Taken from: https://www.johndcook.com/blog/python_erf/
	"""
	constants = (
		1.061405429,
		-1.453152027,
		1.421413741,
		-0.284496736,
		0.254829592
	)
	p  =  0.3275911
	sign = 1 if x >= 0 else -1
	x = abs(x)
	t = 1 / (1 + p * x)
	y = 0

	for c in constants: y = (y + c) * t
	y = 1 - (y * math.exp(-x*x))

	return sign * y
```

```python
def calc_cum_prob_norm_dist(x: int, mean: int, sd: int) -> float:
	"""Returns cumulative probability for a normal distribution from 0 to X.

	Args:
		x (int): Upper limit of cumulative probability
		mean (int): mean of sample
		sd (int): standard deviation of sample

	Returns:
		float: cumulative probability for [0, X].
	"""
	z = (x - mean) / (sd * 2 ** (1/2))
	return 1/2 * (1 + erf(z))
```

```python
def print_to_scale(num: int) -> None:
	"""Prints number to 4 decimal places

	Args:
		num (int): Number to print
	"""
	print(f"{num :.4f}")
```

## Main

```python
def main():
	"""
	NOTE:
	This is an interesting application of a normal distribution.

	Thus far, I have only used normal distribution assumptions when working with the mean. However, it makes sense
	for other statistics to follow the normal dist as well, e.g., the summation of a sample.

	As such, this solution uses a CDF to calculate the cumulative probability of
	"""
	## Get Input
	max_weight, boxes, avg_weight, sd_weight = get_input()

	## Calculate sum & variance statistics to use
	normal_dist_sum = boxes * avg_weight
	normal_dist_sd = boxes ** (1/2) * sd_weight

	## Find Cumulative distribution
	cum_prob = calc_cum_prob_norm_dist(max_weight, normal_dist_sum, normal_dist_sd)

	print_to_scale(cum_prob)
```

```python
if __name__ == "__main__":
	main()
```

    0.0098


# The Laplace Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Analysis of Passive Electrical Networks

The Laplace transform is a well-established tool for the analysis of differential equations including initial values. [Electrical networks](https://en.wikipedia.org/wiki/Electrical_network) composed of linear passive elements, like resistors, capacitors and inductors can be described mathematically by linear ordinary differential equations (ODEs) with constant coefficients. The Laplace transform provides an elegant way of analyzing such networks. This is illustrated in the following.

### Complex Impedances and Equivalent Networks

The concept of complex impedances is used to analyze passive electrical networks in the Laplace domain. Let's first take a look at the ODEs describing the relation between voltage $u(t)$ and current $i(t)$ for linear passive elements. They are summarized in the second column of the following table

| Element | Temporal Domain | Laplace Domain | Impedance $Z(s)$ |
|:---|:---|:---|:---|
|  | $u(t) = R \cdot i(t)$ | $U(s) = R \cdot I(s)$ | $R$ |
|  | $\begin{matrix} u(t) = L \frac{d}{dt} i(t) \\ i(t) = \frac{1}{L} \int_{0}^{t} u(\tau) d\tau + i(0+) \epsilon(t) \end{matrix}$ | $\begin{matrix} U(s) = s L I(s) - L i(0+) \\ I(s) = \frac{1}{s L} U(s) + \frac{1}{s} i(0+) \end{matrix}$ | $s L$ |
|  | $\begin{matrix} u(t) = \frac{1}{C} \int_{0}^{t} i(\tau) d\tau + u(0+) \epsilon(t) \\ i(t) = C \frac{d}{dt} u(t) \end{matrix}$ | $\begin{matrix} U(s) = \frac{1}{s C} I(s) + \frac{1}{s} u(0+) \\ I(s) = s C U(s) - C u(0+) \end{matrix}$ | $\frac{1}{s C}$

It was assumed that the voltage $u(t)=0$ and current $i(t)=0$ for $t < 0$, hence that both are causal signals. The initial values $u(0+)$ and $i(0+)$ denote their right-sided limit values for $t=0$. For instance $u(0+) = \lim_{\epsilon \to 0} u(0 + \epsilon)$. They initial values represent the energy stored in the capacitors and inductors at time instant $t=0$, respectively. The analysis of a passive electrical network is performed by applying [Kirchhoff's circuit laws](https://en.wikipedia.org/wiki/Kirchhoff's_circuit_laws) resulting in an ODE, describing the relation between input and output voltage for instance. This ODE has to be solved explicitly. See for instance the [previous network analysis example](../systems_time_domain/network_analysis.ipynb).

The time-domain relations can be transformed into the Laplace domain by applying the [differentiation and integration theorem](table_theorems_transforms.ipynb#Properties-and-Theorems) of the Laplace transform. The results are summarized in the third column. The differentiation and integration are now represented by algebraic operations. Kirchhoff's circuit laws can be applied straightforwardly to the transformed quantities using the transformed relations. This is due to the fact that the Laplace transform is a linear operation. The result is an algebraic equation that can be solved straightforward with respect to the desired quantities.

When the initial values $i(0+)$ or $i(0+)$ are zero, the elements can be characterized in the Laplace domain by their [complex impedances](https://en.wikipedia.org/wiki/Electrical_impedance). The complex impedance $Z(s)$ is defined as follows

\begin{equation}
Z(s) = \frac{U(s)}{I(s)}
\end{equation}

Complex impedances can be used to represent a passive electrical network in the Laplace domain. The analysis of an electrical network in the Laplace domain is illustrated by the example given in the next section. Note that similar considerations also apply to mechanical systems and other problems that can be described by ODEs.

### Example: Second-Order Low-Pass Filter

The second-order low-pass filter from the [previous example](../systems_time_domain/network_analysis.ipynb) is analyzed using the Laplace transform. First the step response for zero initial values is computed followed by an analysis including initial values in a second stage.

#### Output signal for zero initial values

It is assumed that no energy is stored in the capacitor and inductor for $t<0$. Consequently, the initial values can be discarded. The equivalent network in the Laplace domain is derived by transforming the input $x(t)$ and output $y(t)$, and introducing the complex impedances from above table for its elements.

Applying [Kirchhoff's circuit laws](https://en.wikipedia.org/wiki/Kirchhoff's_circuit_laws) with the complex impedances of the network elements yields the output signal $Y(s)$ in relation to the input $X(s)$ in the Laplace domain as

\begin{equation}
Y(s) = \frac{1}{LC s^2 + RC s + 1} \cdot X(s)
\end{equation}

This relation is defined in `SymPy` for subsequent evaluation.

```python
import sympy as sym
sym.init_printing()

s = sym.symbols('s', complex=True)
t, R, L, C = sym.symbols('t R L C', positive=True)
X = sym.Function('X')(s)

Y = 1/(L*C*s**2 + R*C*s + 1) * X
Y
```

The response $y(t)$ of the network to a [Heaviside signal](../continuous_signals/standard_signals.ipynb#Heaviside-Signal) at its input, is computed by setting the input to $x(t) = \epsilon(t)$. The Laplace transform $Y(s)$ of the output signal is hence given as

\begin{equation}
Y(s) = \frac{1}{LC s^2 + RC s + 1} \cdot \frac{1}{s}
\end{equation}

The output signal $y(t)$ is computed by inverse Laplace transform of $Y(s)$ for the normalized values $L = .5$, $R = 1$, $C = .4$.

```python
RLC = {R: 1, L: sym.Rational('.5'), C: sym.Rational('.4')}

y = sym.inverse_laplace_transform(Y.subs(RLC).subs(X, 1/s), s, t)
y
```

The result is simplified for sake of readability

```python
y = y.simplify()
y
```

```python
sym.plot(y, (t, 0, 5), xlabel='$t$', ylabel='$y(t)$');
```

The computation of the output signal $y(t)$ did not require the solution of the underlying ODE as in the [previous example](../systems_time_domain/network_analysis.ipynb). Based on the equivalent network in the Laplace domain, only the computation of an inverse Laplace transform was required. Above result is equal to the [solution of the ODE for an Heaviside signal at the input](../systems_time_domain/network_analysis.ipynb#Step-Response).

#### Output signal including initial values

Now the analysis is performed for non-zero initial values. As initial values, the normalized voltage $u_\text{C}(0+) = -1$ at the capacitor and the normalized current $i_\text{L}(0+) = 0$ at the inductor is assumed. The network is analyzed again using Kirchhoff's circuit laws, but now the initial values are not discarded. Using the Laplace domain representation of the network elements from above table, this results in

\begin{align}
Y(s) &= \underbrace{\frac{1}{L C s^2 + R C s + 1} \cdot X(s)}_{Y_\text{ext}(s)} \\
&+ \underbrace{\frac{R C + L C s}{L C s^2 + R C s + 1} \cdot y(0+) + \frac{L}{L C s^2 + R C s + 1} \cdot i_\text{L}(0+)}_{Y_\text{int}(s)}
\end{align}

where the fact has been used that the initial voltage $u_\text{C}(0+)$ at the capacitor is equal to the initial value of the output $y(0+)$. The index for the current $i_\text{L}(t)$ at the inductor is discarded in the remainder for brevity. The terms have been sorted with respect to their dependence on the input $X(s)$, and the initial values $y(0+)$ and $i(0+)$. The part of the output signal which depends only on the input is termed as *external* part $Y_\text{ext}(s)$. The parts of the output signal which depend only on the initial values are termed as *internal* parts $Y_\text{int}(s)$. The output signal is given as superposition of both contributions

\begin{equation}
y(t) = y_\text{ext}(t) + y_\text{int}(t)
\end{equation}

where $y_\text{ext}(t) = \mathcal{L}^{-1} \{ Y_\text{ext}(s) \}$ and $y_\text{int}(t) = \mathcal{L}^{-1} \{ Y_\text{int}(s) \}$.

The external part of the output signal has already been computed in the previous section

```python
yext = y
yext.simplify()
```

The Laplace transform of the internal part $Y_\text{int}(s)$ is defined for evaluation

```python
i0, y0 = sym.symbols('i0 y0', real=True)

Yint = (R*C + L*C*s) / (L*C*s**2 + R*C*s + 1) * \
    y0 + L / (L*C*s**2 + R*C*s + 1) * i0
Yint
```

Now the inverse Laplace transform is computed for the initial values $y(0+)$ and $i(0+)$, and the specific values of $R$, $L$ and $C$ given above

```python
yint = sym.inverse_laplace_transform(
    Yint.subs(RLC).subs(i0, 0).subs(y0, -1), s, t)
yint
```

The output signal $y(t)$ is given as superposition of the external and internal contribution

```python
y = yext + yint
y.simplify()
```

The internal $y_\text{int}(t)$ (green line) and external $y_\text{ext}(t)$ (blue line) part, as well as the output signal $y(t)$ (red line) is plotted for illustration

```python
p1 = sym.plot(yext, (t, 0, 5), line_color='b',
              xlabel='$t$', ylabel='$y(t)$', show=False)
p2 = sym.plot(yint, (t, 0, 5), line_color='g', show=False)
p3 = sym.plot(y, (t, 0, 5), line_color='r', show=False)
p1.extend(p2)
p1.extend(p3)
p1.show()
```

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


# Center of Mass and Moment of Inertia
Renato Naville Watanabe

```python
import numpy as np
import sympy as sym

```

### Center of mass

The center of mass of a set of particles is formally defined as the point where the sum of the vectors linking this point to each particle, weighted by its mass, is zero.

 The sum of the vectors linking the center of mass to each particle is:

\begin{equation}
  \sum\limits_{i = 1}^nm_i{\bf\vec{r_{i/cm}}}= \sum\limits_{i = 1}^nm_i\left({\bf\vec{r_i}} - {\bf\vec{r}_{cm}}\right) = \sum\limits_{i = 1}^nm_i{\bf\vec{r_i}} - \sum\limits_{i = 1}^nm_i{\bf\vec{r}_{cm}} = \sum\limits_{i = 1}^nm_i{\bf\vec{r_i}} - {\bf\vec{r}_{cm}}\sum\limits_{i = 1}^nm_i
\end{equation}
where $n$ is the number of particles.

Now, we equal this sum to zero and isolate ${\bf\vec{r_{cm}}}$:

\begin{equation}
     \sum\limits_{i = 1}^nm_i{\bf\vec{r_i}} - {\bf\vec{r}_{cm}}\sum\limits_{i = 1}^nm_i = 0 \longrightarrow {\bf\vec{r}_{cm}} = \frac{\sum\limits_{i = 1}^nm_i{\bf\vec{r_i}}}{\sum\limits_{i = 1}^nm_i}  = \frac{\sum\limits_{i = 1}^nm_i{\bf\vec{r_i}}}{m_T}
\end{equation}

where $m_T$ is the total mass of the particles.

#### Example

Two masses with mass $m_1$ and $m_2$

\begin{equation}
    \vec{\bf{r_{cm}}} = \frac{m_1\vec{\bf{r_1}}+m_2\vec{\bf{r_2}}}{m_1+m_2} =  \frac{m_1\vec{\bf{r_1}}+m_2\vec{\bf{r_1}}-m_2\vec{\bf{r_1}} +m_2\vec{\bf{r_2}}}{m_1+m_2}=\frac{(m_1+m_2)\vec{\bf{r_1}}}{m_1+m_2} + \frac{m_2(\vec{\bf{r_2}}-\vec{\bf{r_1}})}{m_1+m_2}=\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, = \vec{\bf{r_1}} + \frac{m_2}{m_1+m_2}(\vec{\bf{r_2}}-\vec{\bf{r_1}})
\end{equation}

For a rigid body, the center of mass is defined as the point where the integral of the vectors  linking this point to each differential part of mass, weighted by this differential mass, is zero.

This integral is:

\begin{equation}
    \int\limits_{B} {\bf{\vec{r}_{/cm}}} dm = \int\limits_{B} {\bf{\vec{r}}} - {\bf{\vec{r}_{cm}}}  dm = \int\limits_{B} {\bf{\vec{r}}}\,dm - \int\limits_{B}{\bf{\vec{r}_{cm}}}  dm = \int\limits_{B} {\bf{\vec{r}}}\,dm - {\bf{\vec{r}_{cm}}}\int\limits_{B}\, dm
\end{equation}

Now we equal this integral to zero and isolate ${\bf\vec{r_{cm}}}$:

\begin{equation}
    \int\limits_{B} {\bf{\vec{r}}}\,dm - {\bf{\vec{r}_{cm}}}\int\limits_{B}\, dm  = 0 \longrightarrow  {\bf{\vec{r}_{cm}}} = \frac{ \int\limits_{B} {\bf{\vec{r}}}\,dm}{\int\limits_{B}\, dm}  = \frac{ \int\limits_{B} {\bf{\vec{r}}}\,dm}{m_B}
    \label{eq:cm}
\end{equation}
where $m_B$ is the mass of the body.

#### Example

bar

\begin{equation}
    {\bf{\vec{r}_{cm}}} = \frac{ \int\limits_{B} {\bf{\vec{r}}}\,dm}{m_B}
\end{equation}

The mass of the bar is:

\begin{equation}
    m_b = \rho l
\end{equation}

The integral can be computed as:

\begin{equation}
     \int\limits_{B} {\bf{\vec{r}}}\,dm = \rho\int\limits_0^l x\,dx =\rho\frac{l^2}{2}
\end{equation}

So, the center of mass of the bar is:

\begin{equation}
    {\bf{\vec{r}_{cm}}} = \frac{\rho\frac{l^2}{2}}{\rho l} = \frac{l}{2}
\end{equation}

The center of mass of an homogenous bar is in the middle of the bar.

Now, we will compute the center of mass of the triangle below.

The center of mass is given by Eq.~\eqref{eq:cm}

\begin{equation}
    \vec{\bf{r_{cm}}} = \frac{\int_B \vec{\bf{r}}\,dm}{m}
\end{equation}

The mass of the triangle is equal:

\begin{equation}
    m = \rho A = \frac{\rho bh}{2}
\end{equation}

The integral in the triangle can be computed by separating into two parts: one part we integrate in $x$ until $p$, and the other part from $p$ to $b$. The integration in the $y$ direction will be from 0 to $\frac{xh}{p}$ in the first integral and from 0 to $\frac{(b-x)h}{b-p}$:

\begin{equation}
     \int_B \vec{\bf{r}}\,dm = \rho\int\limits_B \vec{\bf{r}}\,dA = \rho\left(\int\limits_0^p\int\limits_0^\frac{xh}{p} x\hat{\bf{i}} + y\hat{\bf{j}} \,dydx + \int_p^b\int_0^\frac{(b-x)h}{b-p} x\hat{\bf{i}} + y\hat{\bf{j}} \,dydx\right)
\end{equation}

So, the center of mass of the triangle will be:

\begin{equation}
    \vec{\bf{r_{cm}}} = \frac{\rho\left(\int\limits_0^p\int\limits_0^\frac{xh}{p} x\hat{\bf{i}} + y\hat{\bf{j}} \,dydx + \int_p^b\int_0^\frac{(b-x)h}{b-p} x\hat{\bf{i}} + y\hat{\bf{j}} \,dydx\right)}{\frac{\rho bh}{2}} = \frac{2\left(\int\limits_0^p\int\limits_0^\frac{xh}{p} x\hat{\bf{i}} + y\hat{\bf{j}} \,dydx + \int_p^b\int_0^\frac{(b-x)h}{b-p} x\hat{\bf{i}} + y\hat{\bf{j}} \,dydx\right)}{bh}
\end{equation}

The integral above will be solved using the Symbolic library of the Python.

```python
x,y = sym.symbols('x,y')
rho, h, p, b = sym.symbols('rho, h, p, b')
```

```python
xcm = 2*(sym.integrate(sym.integrate(x, (y, 0, x*h/p)), (x, 0, p)) +
      sym.integrate(sym.integrate(x, (y, 0, (b-x)*h/(b-p))), (x, p, b)))/(b*h)
print('x_cm = ',sym.simplify(xcm))
```

    x_cm =  b/3 + p/3

```python
ycm = 2*(sym.integrate(sym.integrate(y, (y, 0, x*h/p)), (x, 0, p)) +
      sym.integrate(sym.integrate(y, (y, 0, (b-x)*h/(b-p))), (x, p, b)))/(b*h)
print('y_cm = ',sym.simplify(ycm))
```

    y_cm =  h/3

So, the center of mass of the triangle is $\vec{\bf{r_{cm}}} = \left(\frac{b+p}{3}\right)\hat{\bf{i}} + \frac{h}{3}\hat{\bf{j}}$

### Center of gravity

Center of gravity of a body is the point where the moment caused by the gravitational force in the whole body relative to this point is zero.

\begin{equation}
    \vec{\bf{M_0}} = \int\limits_{B}  \vec{\bf{r_B}}\, \times\,\vec{\bf{g}}\,dm
\end{equation}

If the acceleration of gravity being applied to the whole body is the same (for all practical purposes, in Biomechanics we can consider it the same in the whole body), the gravity vector can go out of the integral:

\begin{equation}
  \vec{\bf{M_0}}= \int\limits_{B}\vec{\bf{r_B}}\,dm\, \times\,\vec{\bf{g}} =  \int\limits_{B}(\vec{\bf{r}} - \vec{\bf{r_{cm}}})\,dm\, \times\,\vec{\bf{g}} =  \left(\int\limits_{B}\vec{\bf{r}}\,dm -\int\limits_{B}\vec{\bf{r_{G}}}\,dm\,\right) \times\,\vec{\bf{g}}
\end{equation}

Now, we equal this moment to zero and isolate $\vec{\bf{r_{G}}}$:

\begin{equation}
            \left(\int\limits_{B}\vec{\bf{r}}\,dm -\int\limits_{B}\vec{\bf{r_{G}}}\,dm\right) \times\,\vec{\bf{g}} = 0 \longrightarrow \int\limits_{B}\vec{\bf{r}}\,dm -\int\limits_{B}\vec{\bf{r_{G}}}\,dm = 0 \longrightarrow \int\limits_{B}\vec{\bf{r}}\,dm -\vec{\bf{r_{G}}}\int\limits_{B}\,dm = 0\,\,\,\,\,\, \longrightarrow \vec{\bf{r_{G}}} = \frac{ \int\limits_{B}\vec{\bf{r}}\,dm}{\int\limits_{B}\,dm} = \frac{ \int\limits_{B}\vec{\bf{r}}\,dm}{m_B}
\end{equation}
where $m_B$ is the mass of the body.

Note that in this case, that the gravity acceleration is constant, the center of gravity $\vec{\bf{r_{G}}}$ is equal to the center of mass $\vec{\bf{r_{cm}}}$.

### Geometric center

For a rigid body, the geometric center is defined as the point where the integral of the vectors  linking this point to each differential part of mass is zero. It

This integral is:

\begin{equation}
    \int\limits_{B} {\bf{\vec{r}_{/gc}}} dV = \int\limits_{B} {\bf{\vec{r}}} - {\bf{\vec{r}_{gc}}}  dV = \int\limits_{B} {\bf{\vec{r}}}\,dV - \int\limits_{B}{\bf{\vec{r}_{gc}}}  dV = \int\limits_{B} {\bf{\vec{r}}}\,dV - {\bf{\vec{r}_{gc}}}\int\limits_{B}\, dV
\end{equation}

Now we equal this integral to zero and isolate ${\bf\vec{r_{gc}}}$:

\begin{equation}
    \int\limits_{B} {\bf{\vec{r}}}\,dV - {\bf{\vec{r}_{gc}}}\int\limits_{B}\, dV  = 0 \longrightarrow  {\bf{\vec{r}_{gc}}} = \frac{ \int\limits_{B} {\bf{\vec{r}}}\,dV}{\int\limits_{B}\, dV}  = \frac{ \int\limits_{B} {\bf{\vec{r}}}\,dV}{V}
    \label{eq:gc}
\end{equation}
where $V$ is the volume of the body. Note that when the body has a constant density $\rho$, the center of mass is equal to the geometric center:

\begin{equation}
    {\bf{\vec{r}_{cm}}} =  \frac{ \int\limits_{B} {\bf{\vec{r}}}\,dm}{m_B} = \frac{ \int\limits_{B} {\bf{\vec{r}}}\rho\,dV}{\rho V} = \frac{ \rho\int\limits_{B} {\bf{\vec{r}}}\,dV}{\rho V}  =   \frac{ \int\limits_{B} {\bf{\vec{r}}}\,dV}{V} =  {\bf{\vec{r}_{gc}}}
\end{equation}

### Center of mass of a complex system

Now, we will consider a set of $n$ bodies. The center of mass of this set of bodies can be computed by integrating the Eq.~\eqref{eq:cm} over all bodies:

\begin{equation}
     \vec{\bf{r_{G}}} =  \frac{ \int\limits_{B1,B2,...,Bn}\vec{\bf{r}}\,dm}{m_{B1}+m_{B2}+... + m_{Bn}} =\frac{\int\limits_{B1}\vec{\bf{r}}\,dm+\int\limits_{B2}\vec{\bf{r}}\,dm+...+\int\limits_{Bn}\vec{\bf{r}}\,dm}{m_{B1}+m_{B2}+... + m_{Bn}} = \frac{\frac{\int\limits_{B1}\vec{\bf{r}}\,dm}{m_{B1}}m_{B1} + \frac{\int\limits_{B2}\vec{\bf{r}}\,dm}{m_{B2}}m_{B2} +...+ \frac{\int\limits_{Bn}\vec{\bf{r}}\,dm}{m_{Bn}}m_{Bn}}{m_{B1}+m_{B2}+... + m_{Bn}}  = \frac{\vec{\bf{r_{cm_1}}} m_{B1} + \vec{\bf{r_{cm_2}}}m_{B2} +...+ \vec{\bf{r_{cm_n}}}m_{Bn}}{m_{B1}+m_{B2}+... + m_{Bn}}=\frac{\vec{\bf{r_{cm_1}}} m_{B1} + \vec{\bf{r_{cm_2}}}m_{B2} +...+ \vec{\bf{r_{cm_n}}}m_{Bn}}{m_T}
\end{equation}
where $\vec{\bf{r_{cm_i}}}$ is  the center of mass of the body $i$, $m_{Bi}$ is the mass of the body $i$ and $m_T$ is the total mass of the bodies. The expression above shows that we can interpret each body as a particle with its mass and position.

Now we will compute the center of mass of the system shown below.

The system can be seen as a set of three bars, so we can compute it by considering each bar as a particle located at its center of mass.

We have previously computed the center of mass of an homogenous bar as being at its center. So, the center of mass of each bar is:

\begin{equation}
    \vec{\bf{r_{cm_1}}} = -\frac{l_1}{2}\hat{\bf{j}}
\end{equation}

\begin{equation}
    \vec{\bf{r_{cm_2}}} = \frac{l_2}{2}\sin(\theta_1)\hat{\bf{i}}-\frac{l_2}{2}\cos(\theta_1)\hat{\bf{j}}
\end{equation}

\begin{equation}
    \vec{\bf{r_{cm_3}}} = l_2\sin(\theta_1)\hat{\bf{i}}-l_2\cos(\theta_1)\hat{\bf{j}}+\frac{l_3}{2}\sin(\theta_2)\hat{\bf{i}}-\frac{l_3}{2}\cos(\theta_2)\hat{\bf{j}} = \left(l_2\sin(\theta_1)+\frac{l_3}{2}\sin(\theta_2) \right)\hat{\bf{i}} + \left(l_2\cos(\theta_1)-\frac{l_3}{2}\cos(\theta_2) \right)\hat{\bf{j}}
\end{equation}

So, the center of mass of the system is:

\begin{equation}
         \vec{\bf{r_{cm}}} = \frac{m_1 \vec{\bf{r_{cm_1}}}+m_2 \vec{\bf{r_{cm_2}}}+m_3 \vec{\bf{r_{cm_3}}}}{m_1+m_2+m_3} = \frac{-m_1\frac{l_1}{2}\hat{\bf{j}} + m_2(\frac{l_2}{2}\sin(\theta_1)\hat{\bf{i}}-\frac{l_2}{2}\cos(\theta_1)\hat{\bf{j}})+m_3 \left[\left(l_2\sin(\theta_1)+\frac{l_3}{2}\sin(\theta_2) \right)\hat{\bf{i}} + \left(l_2\cos(\theta_1)-\frac{l_3}{2}\cos(\theta_2) \right)\hat{\bf{j}} \right]}{m_1+m_2+m_3} = \frac{m_2\frac{l_2}{2}\sin(\theta_1)\hat{\bf{i}}+m_3 \left(l_2\sin(\theta_1)+\frac{l_3}{2}\sin(\theta_2) \right)\hat{\bf{i}}}{m_1+m_2+m_3}+\frac{-m_1\frac{l_1}{2}\hat{\bf{j}} - m_2\frac{l_2}{2}\cos(\theta_1)\hat{\bf{j}}+m_3 \left(l_2\cos(\theta_1)-\frac{l_3}{2}\cos(\theta_2) \right)\hat{\bf{j}}}{m_1+m_2+m_3} = \frac{m_2\frac{l_2}{2}\sin(\theta_1)+m_3 \left(l_2\sin(\theta_1)+\frac{l_3}{2}\sin(\theta_2)\right)}{m_1+m_2+m_3}\hat{\bf{i}}+\frac{-m_1\frac{l_1}{2} - m_2\frac{l_2}{2}\cos(\theta_1)+m_3 \left(l_2\cos(\theta_1)-\frac{l_3}{2}\cos(\theta_2) \right)}{m_1+m_2+m_3}\hat{\bf{j}} =
\end{equation}

### Moment of inertia

The moment of inertia is a measure of how the mass of the body distributes relative to a given axis passing to a given point O. In other words, the moment of inertia relative to a point O for an axis a is defined how far the mass of the body is from this axis. If we define the versor $\hat{\bf{n_a}}$ as a versor in the direction of the axis, this distance is the croo product between the vector linking the point O to each part of the body and this versor.

Considering that, the moment of inertia relative to the point O for the axis a is defined as:

\begin{equation}
    I_{aa}^{O} = \int\limits_B (\vec{\bf{r_{/O}}} \times \hat{\bf{n_a}}).(\vec{\bf{r_{/O}}} \times \hat{\bf{n_a}})\,dm
    \label{eq:inmomgen}
\end{equation}

For planar movements, we normally compute the moment of inertia relative to the z-axis. So, the Eq.~\eqref{eq:inmomgen} simplifies to:

\begin{equation}
    I_{zz}^O = \int\limits_B (\vec{\bf{r_{/O}}} \times \hat{\bf{k}}).(\vec{\bf{r_{/O}}} \times \hat{\bf{k}})\,dm =  \int\limits_B \left[(x_{/O}\hat{\bf{i}}+y_{/O}\hat{\bf{j}} +z_{/O}\hat{\bf{k}})\times \hat{\bf{k}}\right].\left[(x_{/O}\hat{\bf{i}}+y_{/O}\hat{\bf{j}} +z_{/O}\hat{\bf{k}})\times \hat{\bf{k}}\right]\,dm = \int\limits_B (-x_{/O}\hat{\bf{j}}+y_{/O}\hat{\bf{i}}).(-x_{/O}\hat{\bf{j}}+y_{/O}\hat{\bf{i}})\,dm \,\,\,\,\longrightarrow I_{zz}^O = \int\limits_B x_{/O}^2+y_{/O}^2\,dm
    \label{eq:inmomzzo}
\end{equation}

Frequently, the chosen point to compute the moment of inertia is the center of mass. So if we choose the point O as the center of mass, the Eq.~\eqref{eq:inmomzzo} is written as:

\begin{equation}
    I_{zz}^{cm} = \int\limits_B x_{/cm}^2+y_{/cm}^2\,dm
\end{equation}

Now we will compute the moment of inertia relative to the center of mass of the same bar we computed the center of mass.

The moment of inertia relative to the center of mass is:

\begin{equation}
    I_{zz}^{cm} = \int\limits_B x_{/cm}^2+y_{/cm}^2\,dm = \int\limits_{-l/2}^{l/2} x^2\rho\,dx = \rho\left.\left(\frac{x^3}{3}\right)\right|_{-l/2}^{l/2} = \rho\left(\frac{l^3}{24}+\frac{l^3}{24}\right)=\rho\frac{l^3}{12} = \frac{ml^2}{12}
\end{equation}

#### Radius of gyration

Radius of gyration is defined as the distance that a particle would be from the a point O to have the same moment of inertia that the body have. So, the radius of gyration is defined as:

\begin{equation}
    r_{gyr} = \sqrt{\frac{I_{zz}^{cm}}{m}}
\end{equation}

For example, for the homogenous bar with length $l$, the radius of gyration is:

\begin{equation}
    r_{gyr} = \sqrt{\frac{\frac{ml^2}{12}}{m}}=\sqrt{\frac{l^2}{12}} = \frac{l\sqrt{3}}{6}
\end{equation}

### Parallel axis theorem

If we have computed the moment of inertia relative to the center of mass for an axis, for example the z-axis, and now  want to compute the moment of inertia relative to another point O for an axis parallel to the first, there is an expression to aid the computation of this moment of inertia.

In the figure below, the axis is perpendicular to the figure plane.

The moment of inertia relative to the axis passing through the point O is given by:

\begin{equation}
        I_{zz}^O = \int\limits_B x_{/O}^2+y_{/O}^2\,dm = \int\limits_B (x_{/cm}+x_{cm/O})^2+(y_{/cm}+y_{cm/O})^2\,dm = \int\limits_B x_{/cm}^2+2x_{/cm}x_{cm/O}+x_{cm/O}^2+y_{/cm}^2+2y_{/cm}y_{cm/O}+y_{cm/O}^2\,dm =\underbrace{\int\limits_B x_{/cm}^2+y_{/cm}^2\,dm}_{I_{zz}^{cm}} +\int\limits_B 2x_{/cm}x_{cm/O}\,dm+\int\limits_B x_{cm/O}^2+y_{cm/O}^2\,dm + \int\limits_B 2y_{/cm}y_{cm/O}\,dm = I_{zz}^{cm} +2x_{cm/O}\underbrace{\int\limits_B x_{/cm}\,dm}_{0}+\underbrace{(x_{cm/O}^2+y_{cm/O}^2)}_{d^2}\underbrace{\int\limits_B \,dm}_{m_B} + 2y_{cm/O}\underbrace{\int\limits_B y_{/cm}\,dm}_{0} \longrightarrow I_{zz}^{O} =  I_{zz}^{cm} + m_B.d^2
\end{equation}

The terms $\int\limits_B x_{/cm}\,dm$ and $\int\limits_B y_{/cm}\,dm$ are equal to zero because of the definition of center of mass. The term $d$ is the distance between the two axes. Note that this theorem is valid only for parallel axes.

Now we will compute the moment of inertia the moment of inertia of a bar relative to one of its extremities, by using the parallel axis theorem.

\begin{equation}
    I_{zz}^O = I_{zz}^{cm} + m\left(\frac{l}{2}\right)^2 = \frac{ml^2}{12} + \frac{ml^2}{4} = \frac{ml^2}{3}
\end{equation}

### Moment of inertia of a complex system

To compute the moment of inertia of a set o $n$ bodies relative to a point O for a given axis (for example, the z axis), we must apply the Eq.~\eqref{eq:inmomzzo} for all bodies:

\begin{equation}
    I_{zz}^{O} = \int\limits_{B1,B2,...,Bn} x_{/O}^2+y_{/O}^2\,dm =  \int\limits_{B1} x_{/O}^2+y_{/O}^2\,dm +\int\limits_{B2} x_{/O}^2+y_{/O}^2\,dm +...+\int\limits_{Bn} x_{/O}^2+y_{/O}^2\,dm
\end{equation}

Now, using the parallel axis theorem, we can write the equation above in terms of the moment of inertia relative to the center of mass of each body:

\begin{equation}
    I_{zz}^{O} =  I_{zz_{B1}}^{cm_1} + m_{B1}.||\vec{\bf{r_{cm_1/O}}}||^2 +I_{zz_{B2}}^{cm_2} + m_{B2}.||\vec{\bf{r_{cm_2/O}}}||^2  +...+I_{zz_{Bn}}^{cm_n} + m_{Bn}.||\vec{\bf{r_{cm_n/O}}}||^2
    \label{eq:paral}
\end{equation}
where $I_{zz_{Bi}}^{cm_i}$ is the moment of inertia of the body $i$ relative to its center of mass.

Now we will compute the moment of inertia of the set of eight equal and homogenous bars depicted in the figure below relative to its center of mass.

By symmetry, the center of mass of this system is in the point O. So, to compute the moment of inertia relative to this point, we must use the parallel axis theorem to each bar, and then sum the results, as in Eq.~\eqref{eq:paral}.

\begin{equation}
    I^cm_{zz} = I^{cm1}_{zz}+m\left(\frac{l}{2}\right)^2+I^{cm2}_{zz}+m\left(\frac{l}{2}\right)^2+I^{cm3}_{zz}+m\left(\frac{l}{2}\right)^2+I^{cm4}_{zz}+m\left(\frac{l}{2}\right)^2+I^{cm5}_{zz}+m\left(\frac{l}{2}\right)^2+I^{cm6}_{zz}+m\left(\frac{l}{2}\right)^2+I^{cm7}_{zz}+m\left(\frac{l}{2}\right)^2+I^{cm8}_{zz}+m\left(\frac{l}{2}\right)^2
\end{equation}

The moment of inertia of a bar relative to its center of mass is $\frac{ml^2}{12}$. So the moment of inertia of the system relative to its center of mass is:

\begin{equation}
    I^cm_{zz} = 8.\left[\frac{ml^2}{12}+m\left(\frac{l}{2}\right)^2\right]=\frac{8ml^2}{3}
\end{equation}

### Matrix of Inertia

For 3D movements we must consider the moments of inertia inertia in the three directions besides three other quantities called products of inertia. A product of inertia is defined as in Eq.~\eqref{eq:inmomgen}, but with each versor orthogonal to each other. So, the product of inertia relative to point O for axes a and b is:

\begin{equation}
    I_{ab}^O= \int\limits_B  (\vec{\bf{r_{/O}}} \times \hat{\bf{n_a}}).(\vec{\bf{r_{/O}}} \times \hat{\bf{n_b}})\,dm
    \label{eq:prodinmomgen}
\end{equation}

Note that $I_{ab}^O$ is equal to $I_{ba}^O$.

The matrix of inertia is defined as:

\begin{equation}
    I^O = \left[\begin{array}{ccc}I_{xx}^O&I_{xy}^O&I_{xz}^O\\
                                  I_{yx}^O&I_{yy}^O&I_{yz}^O\\
                                  I_{zx}^O&I_{zy}^O&I_{zz}^O\end{array}\right]
\end{equation}

where each term is defined as follows:

\begin{equation}
    I_{xx}^O = \int\limits_B  (\vec{\bf{r_{/O}}} \times \hat{\bf{i}}).(\vec{\bf{r_{/O}}} \times \hat{\bf{i}})\,dm = \int\limits_B  y_{/O}^2+z_{/O}^2\,dm
\end{equation}

\begin{equation}
    I_{yy}^O = \int\limits_B  (\vec{\bf{r_{/O}}} \times \hat{\bf{j}}).(\vec{\bf{r_{/O}}} \times \hat{\bf{j}})\,dm = \int\limits_B  x_{/O}^2+z_{/O}^2\,dm
\end{equation}

\begin{equation}
    I_{zz}^O = \int\limits_B  (\vec{\bf{r_{/O}}} \times \hat{\bf{k}}).(\vec{\bf{r_{/O}}} \times \hat{\bf{k}})\,dm = \int\limits_B  x_{/O}^2+y_{/O}^2\,dm
\end{equation}

\begin{equation}
    I_{xy}^O = I_{yx}^O =\int\limits_B  (\vec{\bf{r_{/O}}} \times \hat{\bf{i}}).(\vec{\bf{r_{/O}}} \times \hat{\bf{j}})\,dm = \int\limits_B  \left[(x_{/O}\hat{\bf{i}}+y_{/O}\hat{\bf{j}}+z_{/O}\hat{\bf{k}}) \times \hat{\bf{i}}\right].\left[(x_{/O}\hat{\bf{i}}+y_{/O}\hat{\bf{j}}+z_{/O}\hat{\bf{k}}) \times \hat{\bf{j}}\right]\,dm = \int\limits_B  (-y_{/O}\hat{\bf{k}}+z_{/O}\hat{\bf{j}}).(x_{/O}\hat{\bf{k}}-z_{/O}\hat{\bf{i}})\,dm \longrightarrow I_{xy}^O = \int\limits_B  - y_{/O}x_{/O}\,dm
\end{equation}

\begin{equation}
    I_{xz}^O = I_{zx}^O =\int\limits_B  (\vec{\bf{r_{/O}}} \times \hat{\bf{i}}).(\vec{\bf{r_{/O}}} \times \hat{\bf{k}})\,dm = \int\limits_B  \left[(x_{/O}\hat{\bf{i}}+y_{/O}\hat{\bf{j}}+z_{/O}\hat{\bf{k}}) \times \hat{\bf{i}}\right].\left[(x_{/O}\hat{\bf{i}}+y_{/O}\hat{\bf{j}}+z_{/O}\hat{\bf{k}}) \times \hat{\bf{k}}\right]\,dm = \int\limits_B  (-y_{/O}\hat{\bf{k}}+z_{/O}\hat{\bf{j}}).(-x_{/O}\hat{\bf{j}}+y_{/O}\hat{\bf{i}})\,dm \longrightarrow I_{xz}^O = \int\limits_B  - z_{/O}x_{/O}\,dm
\end{equation}

\begin{equation}
    I_{yz}^O = I_{zy}^O =\int\limits_B  (\vec{\bf{r_{/O}}} \times \hat{\bf{j}}).(\vec{\bf{r_{/O}}} \times \hat{\bf{k}})\,dm = \int\limits_B  \left[(x_{/O}\hat{\bf{i}}+y_{/O}\hat{\bf{j}}+z_{/O}\hat{\bf{k}}) \times \hat{\bf{j}}\right].\left[(x_{/O}\hat{\bf{i}}+y_{/O}\hat{\bf{j}}+z_{/O}\hat{\bf{k}}) \times \hat{\bf{k}}\right]\,dm = \int\limits_B  (x_{/O}\hat{\bf{k}}-z_{/O}\hat{\bf{i}}).(-x_{/O}\hat{\bf{j}}+y_{/O}\hat{\bf{i}})\,dm \longrightarrow I_{yz}^O = \int\limits_B  - z_{/O}y_{/O}\,dm
\end{equation}

Now, we will consider a cylinder and compute the moment of inertia for the three axes, relative to the center of mass. The mass of this cylinder is $m = \rho \pi R^2$ and, by symmetry, it is in the middle of the cylinder.

First, for axis $z$:

\begin{equation}
    I_{zz}^{cm} = \int\limits_B  x_{/cm}^2+y_{/cm}^2\,dm = \int\limits_{-h/2}^{h/2}\int\limits_0^R\int\limits_0^{2\pi} r^2 \rho r \,d\theta\,dr\,dz = \rho \int\limits_{-h/2}^{h/2}\int\limits_0^R\int\limits_0^{2\pi} r^3  \,d\theta\,dr\,dz = \rho \int\limits_{-h/2}^{h/2}\int\limits_0^R 2\pi r^3  \,dr\,dz = \rho \int\limits_{-h/2}^{h/2} 2\pi \frac{R^4}{4}  \,dz =  \rho \pi \frac{R^4}{2}h = \frac{mR^2}{2}
\end{equation}

Now, for the $x axis$:

\begin{equation}
    I_{xx}^{cm} = \int\limits_B  z_{/cm}^2+y_{/cm}^2\,dm = \int\limits_{-h/2}^{h/2}\int\limits_0^R\int\limits_0^{2\pi} z^2+r^2\sin^2(\theta) \rho r \,d\theta\,dr\,dz = \int\limits_{-h/2}^{h/2}\int\limits_0^R\int\limits_0^{2\pi} (rz^2+r^3\sin^2(\theta) )\rho  \,d\theta\,dr\,dz
\end{equation}

We will solve this integral using the Sympy lybrary:

```python
h, R, z, theta, r, rho, m1 = sym.symbols('h, R, z, theta, r, rho, m')
m = rho*h*sym.pi*R**2
Ixx = rho*sym.integrate(sym.integrate(sym.integrate(r*z**2+(r**3)*sym.sin(theta)**2, (theta, 0, 2*sym.pi)), (r, 0, R)), (z,-h/2,h/2))/m
sym.simplify(m1*Ixx)
```

So, the moment of inertia of the cylinder for the axis $x$ is:

$I_{xx}^{cm} = \frac{m}{12}(3R^2+h^2)$

By symmetry, the moment of inertia for the axis $y$ is equal to  the moment of inertia for the axis $x$.

$I_{yy}^{cm} = \frac{m}{12}(3R^2+h^2)$

#### Principal axes

It is always possible to choose the direction of the axes in a way that the products of inertia are zero. In this situation, the inertia matrix is:

\begin{equation}
    I^{cm} = \left[\begin{array}{ccc}I_{1}^O&0&0\\
                                  0&I_{2}^O&0\\
                                  0&0&I_{3}^O\end{array}\right]
\end{equation}

where $I_1$, $I_2$ and $I_3$ are the moments of inertia for each of the principal axes.

### Problems
Solve the problems 3.2.5, 3.2.7, 15.3.11, 15.3.15 from Ruina and Pratap book.

## References

- Ruina A, Rudra P (2015) [Introduction to Statics and Dynamics](http://ruina.tam.cornell.edu/Book/index.html). Oxford University Press.

- Kane T, Levinson D (1985) [Dynamics: Theory and Applications](https://ecommons.cornell.edu/handle/1813/638). McGraw-Hill, Inc

```python

```


# Introduction to Deep Learning with PyTorch

In this notebook, you will get an introduction to [PyTorch](http://pytorch.org/), which is a framework for building and training neural networks (NN). ``PyTorch`` in a lot of ways behaves like the arrays you know and love from Numpy. These Numpy arrays, after all, are just *tensors*. PyTorch takes these tensors and makes it simple to move them to GPUs for the faster processing needed when training neural networks. It also provides a module that automatically calculates gradients (for backpropagation!) and another module specifically for building neural networks. All together, PyTorch ends up being more coherent with **Python** and the ``Numpy/Scipy`` stack compared to *TensorFlow* and other frameworks.

## Neural Networks

Deep Learning is based on artificial neural networks which have been around in some form since the late 1950s. The networks are built from individual parts approximating neurons, typically called units or simply "neurons." Each unit has some number of weighted inputs. These weighted inputs are summed together (a linear combination) then passed through an activation function to get the unit's output.

Mathematically this looks like:

$$
\begin{align}
y &= f(w_1 x_1 + w_2 x_2 + b) \\
y &= f\left(\sum_i w_i x_i +b \right)
\end{align}
$$

With vectors this is the dot/inner product of two vectors:

$$
h = \begin{bmatrix}
x_1 \, x_2 \cdots  x_n
\end{bmatrix}
\cdot
\begin{bmatrix}
           w_1 \\
           w_2 \\
           \vdots \\
           w_n
\end{bmatrix}
$$

## Tensors

It turns out neural network computations are just a bunch of linear algebra operations on *tensors*, a generalization of matrices. A vector is a 1-dimensional tensor, a matrix is a 2-dimensional tensor, an array with three indices is a 3-dimensional tensor (RGB color images for example). The fundamental data structure for neural networks are tensors and PyTorch (as well as pretty much every other deep learning framework) is built around tensors.

With the basics covered, it's time to explore how we can use PyTorch to build a simple neural network.

```python
# First, import PyTorch
import torch
```

```python
def activation(x):
    """ Sigmoid activation function

        Arguments
        ---------
        x: torch.Tensor
    """
    return 1/(1+torch.exp(-x))
```

```python
### Generate some data
torch.manual_seed(7) # Set the random seed so things are predictable

# Features are 5 random normal variables
features = torch.randn((1, 5))
# True weights for our data, random normal variables again
weights = torch.randn_like(features)
# and a true bias term
bias = torch.randn((1, 1))
```

Above I generated data we can use to get the output of our simple network. This is all just random for now, going forward we'll start using normal data. Going through each relevant line:

`features = torch.randn((1, 5))` creates a tensor with shape `(1, 5)`, one row and five columns, that contains values randomly distributed according to the normal distribution with a mean of zero and standard deviation of one.

`weights = torch.randn_like(features)` creates another tensor with the same shape as `features`, again containing values from a normal distribution.

Finally, `bias = torch.randn((1, 1))` creates a single value from a normal distribution.

PyTorch tensors can be added, multiplied, subtracted, etc, just like Numpy arrays. In general, you'll use PyTorch tensors pretty much the same way you'd use Numpy arrays. They come with some nice benefits though such as GPU acceleration which we'll get to later. For now, use the generated data to calculate the output of this simple single layer network.
> **Exercise**: Calculate the output of the network with input features `features`, weights `weights`, and bias `bias`. Similar to Numpy, PyTorch has a [`torch.sum()`](https://pytorch.org/docs/stable/torch.html#torch.sum) function, as well as a `.sum()` method on tensors, for taking sums. Use the function `activation` defined above as the activation function.

```python
# MY CODE
activation(torch.sum(features*weights)+bias)
```

    tensor([[0.1595]])

```python
## Calculate the output of this network using the weights and bias tensors
output = activation(torch.sum(features*weights) + bias)
output
```

    tensor([[0.1595]])

You can do the multiplication and sum in the same operation using a matrix multiplication. In general, you'll want to use matrix multiplications since they are more efficient and accelerated using modern libraries and high-performance computing on GPUs.

Here, we want to do a matrix multiplication of the features and the weights. For this we can use [`torch.mm()`](https://pytorch.org/docs/stable/torch.html#torch.mm) or [`torch.matmul()`](https://pytorch.org/docs/stable/torch.html#torch.matmul) which is somewhat more complicated and supports broadcasting. If we try to do it with `features` and `weights` as they are, we'll get an error

```python
>> torch.mm(features, weights)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-13-15d592eb5279> in <module>()
----> 1 torch.mm(features, weights)

RuntimeError: size mismatch, m1: [1 x 5], m2: [1 x 5] at /Users/soumith/minicondabuild3/conda-bld/pytorch_1524590658547/work/aten/src/TH/generic/THTensorMath.c:2033
```

As you're building neural networks in any framework, you'll see this often. Really often. What's happening here is our tensors aren't the correct shapes to perform a matrix multiplication. Remember that for matrix multiplications, the number of columns in the first tensor must equal to the number of rows in the second tensor. Both `features` and `weights` have the same shape, `(1, 5)`. This means we need to change the shape of `weights` to get the matrix multiplication to work.

**Note:** To see the shape of a tensor called `tensor`, use `tensor.shape`. If you're building neural networks, you'll be using this method often.

There are a few options here: [`weights.reshape()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.reshape), [`weights.resize_()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.resize_), [`weights.view()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view) and [`torch.transpose(weights,0,1)`](https://pytorch.org/docs/master/generated/torch.transpose.html).

* `weights.reshape(a, b)` will return a new tensor with the same data as `weights` with size `(a, b)` sometimes, and sometimes a clone, as in it copies the data to another part of memory.
* `weights.resize_(a, b)` returns the same tensor with a different shape. However, if the new shape results in fewer elements than the original tensor, some elements will be removed from the tensor (but not from memory). If the new shape results in more elements than the original tensor, new elements will be uninitialized in memory. Here I should note that the underscore at the end of the method denotes that this method is performed **in-place**. Here is a great forum thread to [read more about in-place operations](https://discuss.pytorch.org/t/what-is-in-place-operation/16244) in PyTorch.
* `weights.view(a, b)` will return a new tensor with the same data as `weights` with size `(a, b)`.
* `torch.transpose(weights,0,1)` will return transposed weights tensor. This returns transposed version of inpjut tensor along dim 0 and dim 1. This is efficient since we do not specify to actual dimesions of weights.

I usually use `.view()`, but any of the three methods will work for this. So, now we can reshape `weights` to have five rows and one column with something like `weights.view(5, 1)`.

One more approach is to use `.t()` to transpose vector of weights, in our case from (1,5) to (5,1) shape.
> **Exercise**: Calculate the output of our little network using matrix multiplication.

```python
# MY CODE
activation(torch.sum(torch.mm(features, weights.t()))+bias)
```

    tensor([[0.1595]])

```python
## Calculate the output of this network using matrix multiplication
output = activation(torch.matmul(features,torch.transpose(weights,0,1)) + bias)
output
```

    tensor([[0.1595]])

### Stack them up!

That's how you can calculate the output for a single neuron. The real power of this algorithm happens when you start stacking these individual units into layers and stacks of layers, into a network of neurons. The output of one layer of neurons becomes the input for the next layer. With multiple input units and output units, we now need to express the weights as a matrix.

The first layer shown on the bottom here are the inputs, understandably called the **input layer**. The middle layer is called the **hidden layer**, and the final layer (on the right) is the **output layer**. We can express this network mathematically with matrices again and use matrix multiplication to get linear combinations for each unit in one operation. For example, the hidden layer ($h_1$ and $h_2$ here) can be calculated

$$
\vec{h} = [h_1 \, h_2] =
\begin{bmatrix}
x_1 \, x_2 \cdots \, x_n
\end{bmatrix}
\cdot
\begin{bmatrix}
           w_{11} & w_{12} \\
           w_{21} &w_{22} \\
           \vdots &\vdots \\
           w_{n1} &w_{n2}
\end{bmatrix}
$$

The output for this small network is found by treating the hidden layer as inputs for the output unit. The network output is expressed simply

$$
y =  f_2 \! \left(\, f_1 \! \left(\vec{x} \, \mathbf{W_1}\right) \mathbf{W_2} \right)
$$

```python
### Generate some data
torch.manual_seed(7) # Set the random seed so things are predictable

# Features are 3 random normal variables
features = torch.randn((1, 3))

# Define the size of each layer in our network
n_input = features.shape[1]     # Number of input units, must match number of input features
n_hidden = 2                    # Number of hidden units
n_output = 1                    # Number of output units

# Weights for inputs to hidden layer
W1 = torch.randn(n_input, n_hidden)
# Weights for hidden layer to output layer
W2 = torch.randn(n_hidden, n_output)

# and bias terms for hidden and output layers
B1 = torch.randn((1, n_hidden))
B2 = torch.randn((1, n_output))
```

> **Exercise:** Calculate the output for this multi-layer network using the weights `W1` & `W2`, and the biases, `B1` & `B2`.

```python
# MY CODE
layer1 = activation(torch.mm(features, W1)+B1)
activation(torch.mm(layer1, W2)+B2)
```

    tensor([[0.3171]])

```python
## Your solution here
h = activation(torch.matmul(features,W1).add_(B1))
output = activation(torch.matmul(h,W2).add_(B2))
output
```

    tensor([[0.3171]])

If you did this correctly, you should see the output `tensor([[ 0.3171]])`.

The number of hidden units are a parameter of the network, often called a **hyperparameter** to differentiate it from the weights and biases parameters. As you'll see later when we discuss training a neural network, the more hidden units a network has, and the more layers, the better able it is to learn from data and make accurate predictions.

## Numpy to Torch and back

Special bonus section! PyTorch has a great feature for converting between Numpy arrays and Torch tensors. To create a tensor from a Numpy array, use `torch.from_numpy()`. To convert a tensor to a Numpy array, use the `.numpy()` method.

```python
import numpy as np
np.set_printoptions(precision=8)
a = np.random.rand(4,3)
a
```

    array([[0.06780818, 0.74142159, 0.47460718],
           [0.4334976 , 0.3686304 , 0.15633714],
           [0.2883236 , 0.23609967, 0.92770486],
           [0.55748966, 0.61987983, 0.22091342]])

```python
torch.set_printoptions(precision=8)
b = torch.from_numpy(a)
b
```

    tensor([[0.06780818, 0.74142159, 0.47460718],
            [0.43349760, 0.36863040, 0.15633714],
            [0.28832360, 0.23609967, 0.92770486],
            [0.55748966, 0.61987983, 0.22091342]], dtype=torch.float64)

```python
b.numpy()
```

    array([[0.06780818, 0.74142159, 0.47460718],
           [0.4334976 , 0.3686304 , 0.15633714],
           [0.2883236 , 0.23609967, 0.92770486],
           [0.55748966, 0.61987983, 0.22091342]])

The memory is shared between the Numpy array and Torch tensor, so if you change the values in-place of one object, the other will change as well.

```python
# Multiply PyTorch Tensor by 2, in place
b.mul_(2)
```

    tensor([[0.13561637, 1.48284319, 0.94921435],
            [0.86699520, 0.73726080, 0.31267429],
            [0.57664719, 0.47219934, 1.85540972],
            [1.11497932, 1.23975967, 0.44182685]], dtype=torch.float64)

```python
# Numpy array matches new values from Tensor
a
```

    array([[0.13561637, 1.48284319, 0.94921435],
           [0.8669952 , 0.7372608 , 0.31267429],
           [0.57664719, 0.47219934, 1.85540972],
           [1.11497932, 1.23975967, 0.44182685]])

### Bonus Exercise

```python

```