# 条件随机场

    序列标注模型条件随机场，这种模型与感知机同属结构化学习大家族，但性能比感知机还要强大。

## 一、机器学习的模型谱系

    生成式模型包含的算法：

> 朴素贝叶斯\
K近邻\
混合高斯模型\
隐马尔科夫模型\
贝叶斯网络\
Sigmoid Belief Networks\
马尔科夫随机场(markov random fields)\
深度信念网络\
LDA文档主题生成模型

    判别式模型包含的算法：

> 线性回归(linear regression)\
逻辑回归(logistic regression)\
线性判别分析\
支持向量机\
CART(classification and regression tree)\
神经网络(NN)\
高斯过程(gaussian process)\
条件随机场(conditional random field)

【假设我们有训练数据(X,Y)，X是属性集合，Y是类别标记。这时来了一个新的样本x，我们想要预测它的类别y。
    我们最终的目的是求得最大的条件概率P(y|x)作为新样本的分类。】

根据建模的究竟是联合概率（同时发生的概率）分布P(x,y)还是条件概率分布P(y|x)。派生出生成式模型与判别式模型，这两个模型最终算的都是
    P(y|x)

### 1、生成式模型

    【一般会对每一个类建立一个模型，有多少个类别，就建立多少个模型。比如说类别标签有｛猫，狗，猪｝，那首先根据猫的特征学习出一个猫的模型，再根据狗的特征学习出狗的模型，之后分别计算新样本 x 跟三个类别的联合概率 P(x,y) ，然后根据贝叶斯公式计算出P(y|x)】

    生成式模型：模拟数据的生成过程，两类随机变量存在因果先后关系，先有因素 y，后有结果 x，这种因果关系由联合分布模拟：

$$P(x,y) = P(y)P(x|y)$$

    通过联合分布 P(x,y)，生成式模型其实间接建模了 P(x)：

$$P(x) = \sum_{y \in Y}P(x,y)$$

    然后依据贝叶斯公式可以得到P(y|x)【最终求的仍然是P(y|x)，只是通过了P(x,y)来求】:

$$P(y|x) = \frac{P(x,y)}{P(x)}$$

    这里有两个缺陷:

    1. P(x) 很难准确估计，因为特征之间并非相互独立，而是存在错综复杂的依赖关系。
    2. P(x) 在分类中也没有直接作用。

    为了克服这两个问题，判别式模型出现。

### 2、判别式模型

    【根据训练数据得到分类函数和分界面，比如说根据SVM模型得到一个分界面，然后直接计算条件概率 P(y|x) ，我们将最大的 P(y|x) 作为新样本的分类。判别式模型是对条件概率建模，学习不同类别之间的最优边界，无法反映训练数据本身的特性，能力有限，其只能告诉我们分类的类别。】

  判别式模型直接跳过了 P(x)，直接对条件概率 P(y|x) 建模。不管 x 内部存在多复杂的关系，也不影响判别式模型对 y 的判断，于是就能够放心大胆的利用各种各样丰富的、有关联的特征。 所以我们会看到感知机分词的准确率高于隐马尔可夫模型。

$$P(y|x) = \frac{exp(score(x,y))}{\sum_{x,y}exp(score(x,y))} \ \ \ \ 说明：exp是指数函数$$

### 2.1、举例说明

    假设要对猫和狗进行区分：

    生成式模型：是根据狗的特征首先学习出一个狗的模型，然后根据猫的特征学习出一个猫的模型，然后从这个动物中提取特征，放到狗模型中看概率是多少，再放到猫模型中看概率是多少，哪个大就是哪个。

    判别式模型：用判别式模型的方法是从历史数据中学习到模型，然后通过提取该动物的特征来预测出是狗的概率，是猫的概率。

### 2.2、数据计算过程样例

    假设现在有一个分类问题，X是特征，Y是类标记。用判别式模型学习一个条件概率分布P(y|x)，用生成式模型学习一个联合概率分布 P(x,y) 。用一个简单的例子来说明这个问题。假设X就是两个特征（1或2），Y有两类（0或1），有如下训练样本（1，0）、（1，0）、（1，1）、（2，1）。

    而学习到的联合概率分布（生成式模型）如下：
| | 0 | 1 |
|-|-|-|
|1|1/2|1/4|
|2|0|1/4|

    则学习到的条件概率分布（判别式模型）如下：

| | 0 | 1 |
|-|-|-|
|1|2/3|1/3|
|2|0|1|

    在实际分类问题中，判别式模型可以直接用来判断特征的类别情况；而生成式模型需要加上贝叶斯公式，然后应用到分类中。但是，生成式模型的概率分布可以有其他应用，就是说生成式模型更一般更普适。不过判别式模型更直接，更简单。两种方法目前交叉较多。由生成式模型可以得到判别式模型，但由判别式模型得不到生成式模型。

### 2.3、用图说明区别

    上图左边为判别式模型而右边为生成式模型，可以很清晰地看到差别，判别式模型是在寻找一个决策边界，通过该边界来将样本划分到对应类别。而生成式模型则不同，它学习了每个类别的边界，它包含了更多信息，可以用来生成样本。

### 2.4、两个模型总结

    不管是生成式模型还是判别式模型，它们最终的判断依据都是条件概率P(y|x)，但是生成式模型先计算了联合概率P(x,y)，再由贝叶斯公式计算得到条件概率。因此，生成式模型可以体现更多数据本身的分布信息，其普适性更广。

####     生成式模型的特点：

    生成式模型学习的是联合概率密度分布 P(X,Y)，可以从统计的角度表示分布的情况，能够反映同类数据本身的相似度，它不关心到底划分不同类的边界在哪里。生成式模型的学习收敛速度更快，当样本容量增加时，学习到的模型可以更快的收敛到真实模型，当存在隐变量时，依旧可以用生成式模型，此时判别式方法就不行了。具体来说，有以下特点：
> 对联合概率建模，学习所有分类数据的分布。\
> 学习到的数据本身信息更多，能反应数据本身特性。\
> 学习成本较高，需要更多的计算资源。\
> 需要的样本数更多，样本较少时学习效果较差。\
> 推断时性能较差。\
> 一定条件下能转换成判别式。

####    判别式模型特点：

    判别式模型直接学习决策函数 Y=f(X) 或者条件概率 P(Y|X) ，不能反映训练数据本身的特性，但它寻找不同类别之间的最优分裂面，反映的是异类数据之间的差异，直接面对预测往往学习准确度更高。具体来说有以下特点：

> 对条件概率建模，学习不同类别之间的最优边界。\
> 捕捉不同类别特征的差异信息，不学习本身分布信息，无法反应数据本身特性。\
> 学习成本较低，需要的计算资源较少。\
> 需要的样本数可以较少，少样本也能很好学习。\
> 预测时拥有较好性能。\
> 无法转换成生成式。

总之，判别式模型和生成式模型都是使后验概率最大化，判别式是直接对后验概率建模，而生成式模型通过贝叶斯定理这一“桥梁”使问题转化为求联合概率。

### 3、有向概率图模型

    概率图模型能够分析错综复杂的随机变量关系。

    有向概率图模型也称为贝叶斯网络。

    贝叶斯网络的一个基本要求是图必须是有向无环图（DAG/directed acyclic graph）。

    概率图模型( Probabilistic Graphical Model, PGM)是用来表示与推断多维随机变量联合分布 P(X,Y) 的强大框架，被广泛用于计算机视觉、知识表达、贝叶斯统计与自然语言处理。它利用节点 V 来表示随机变量，用边 E 连接有关联的随机变量，将多维随机变量分布表示为图 G=(V,E)。这样就带来了一个好处，那就是整个图可以分解为子图再进行分析.子图中的随机变量更少，建模更加简单。具体如何分解，据此派生出有向图模型和无向图模型。

    有向图模型按事件的先后因果顺序将节点连接为有向图。如果事件 A 导致事件 B，则用箭头连接两个事件 A-->B。

    有向图模型都将概率有向图分解为一系列条件概率之积，有向图模型经常用生成式模型来实现。定义 π(v) 表示节点 v 的所有前驱节点，则分布为:

$$P(X,Y)=\prod_{v=V}P(v|π(v))$$

#### 3.1、有向概率图模型有啥用

    很多现实问题都可以使用有向概率图进行建模。

    例如：

    这个图描述了某个学生注册某个大学课程的设定。该图中有 5 个随机变量：
> 课程的难度（Difficulty）：可取两个值，0 表示低难度，1 表示高难度\
    学生的智力水平（Intelligence）：可取两个值，0 表示不聪明，1 表示聪明\
    学生的评级（Grade）：可取三个值，1 表示差，2 表示中，3 表示优\
    学生的 SAT 成绩（SAT）：可取两个值，0 表示低分，1 表示高分\
    在完成该课程后学生从教授那里所得到的推荐信的质量（Letter）：可取两个值，0 表示推荐信不好，1 表示推荐信很好

    该图中的边编码了这些变量之间的依赖关系。
    学生的 Grade 取决于课程的 Difficulty 和学生的 Intelligence；
    而 Grade 又反过来决定了学生能否从教授那里得到一份好的 Letter；
    另外，学生的 Intelligence 除了会影响他们的 Grade，还会影响他们的 SAT 分数。
    注意其中箭头的方向表示了因果关系——Intelligence 会影响 SAT 分数，但 SAT 不会影响 Intelligence。
    我们看到的每个节点关联的表格，它们的正式名称是条件概率分布（CPD/conditional probability distribution）

### 4、无向概率图模型

    无向概率图模型也称为马尔科夫网络。

    相反，无向图模型则不探究每个事件的因果关系，也就是说不涉及条件概率分解。无向图模型的边没有方向，仅仅代表两个事件有关联。

    无向概率图模型的几个概念

#### 4.1、团（clique）

#### 4.2、极大团（maximum clique）

#### 4.3、最大团 （maximul clique）

>团：必须两两相连（任意两点都相连）\
极大团：无法再加入一个节点使其成为团\
最大团：极大团里节点最多的团

#### 4.4、因子节点

    因子节点是两个点间虚拟出来的点，用来辅助计算【暂时这么理解】。

    因子分解：将概率无向图模型的联合概率分布表示为其最大团上的随机变量的函数乘积形式的操作

    依据Hammersley- Clifford定理，无向图模型的联合概率分布表示为其最大团上的随机变量的函数乘积形式

$$P(X,Y) = \frac{1}{Z}\prod_{a}\Psi_{a}(X_{a}, Y_{a})$$

其中，a 是因子节点，$\Psi_{a}$ 则是一个因子节点对应的函数，参数 $X_{a}, Y_{a}$ 是与因子节点相连的所有变量节点。为了将式子约束为概率分布，定义常数 Z 为如下归一化因子:

$$Z = \sum_{x,y}\prod_{a}\Psi_{a}(X_{a}, Y_{a})$$
    在机器学习中，常用指数家族的因子函数:
$$\Psi_{a}(X_{a}, Y_{a}) = exp\left\{\sum_{k}W_{ak}F_{ak}(X_{a}, Y_{a})\right\}$$

其中，k 为特征的编号，$F_{ak}$ 是特征函数，$W_{ak}$ 为相应的特征权重。

判别式模型经常用无向图来表示，只需要在归一化时，对每种 x 都求一个归一化因子:

$$Z(X) = \sum_{y}\prod_{a}\Psi_{a}(X_{a}, Y_{a})$$

然后 P(X,Y) 就转化为判别式模型所需的条件概率分布【之所以能转化是因为在条件随机场模型中，X是给定的，Y是条件随机场】:

$$P(Y|X) = \frac{1}{Z(X)}\prod_{a}\Psi_{a}(X_{a}, Y_{a})$$

到这里，最后一个公式就是条件随机场的一般形式。

### 5、为什么有了有向概率图模型，还需要无向概率图模型

    原因是有些问题使用有向图表示会更加自然，比如上面提到的学生网络，有向图可以轻松描述变量之间的因果关系——学生的智力水平会影响 SAT 分数，但 SAT 分数不会影响智力水平（尽管它也许能反映学生的智力水平）。

    而对于其它一些问题，比如图像，你可能需要将每个像素都表示成一个节点。我们知道相邻的像素互有影响，但像素之间并不存在因果关系；它们之间的相互作用是对称的。所以我们在这样的案例中使用无向图模型。

## 二、条件随机场

条件随机场(Conditional Random Fields, 以下简称CRF)是给定一组输入序列条件下另一组输出序列的条件概率分布模型，在自然语言处理中得到了广泛应用。本系列主要关注于CRF的特殊形式：线性链(Linear chain) CRF。

### 1、什么样的问题需要CRF模型

和HMM类似，在讨论CRF之前，我们来看看什么样的问题需要CRF模型。这里举一个简单的例子：

假设我们有Bob一天从早到晚的一系列照片，Bob想考考我们，要我们猜这一系列的每张照片对应的活动，比如: 工作的照片，吃饭的照片，唱歌的照片等等。一个比较直观的办法就是，我们找到Bob之前的日常生活的一系列照片，然后找Bob问清楚这些照片代表的活动标记，这样我们就可以用监督学习的方法来训练一个分类模型，比如逻辑回归，接着用模型去预测这一天的每张照片最可能的活动标记。

这种办法虽然是可行的，但是却忽略了一个重要的问题，就是这些照片之间的顺序其实是有很大的时间顺序关系的，而用上面的方法则会忽略这种关系。比如我们现在看到了一张Bob闭着嘴的照片，那么这张照片我们怎么标记Bob的活动呢？比较难去打标记。但是如果我们有Bob在这一张照片前一点点时间的照片的话，那么这张照片就好标记了。如果在时间序列上前一张的照片里Bob在吃饭，那么这张闭嘴的照片很有可能是在吃饭咀嚼。而如果在时间序列上前一张的照片里Bob在唱歌，那么这张闭嘴的照片很有可能是在唱歌。

为了让我们的分类器表现的更好，可以在标记数据的时候，可以考虑相邻数据的标记信息。这一点，是普通的分类器难以做到的。而这一块，也是CRF比较擅长的地方。

在实际应用中，自然语言处理中的词性标注(POS Tagging)就是非常适合CRF使用的地方。词性标注的目标是给出一个句子中每个词的词性（名词，动词，形容词等）。而这些词的词性往往和上下文的词的词性有关，因此，使用CRF来处理是很适合的，当然CRF不是唯一的选择，也有很多其他的词性标注方法。

### 2、从随机场到马尔科夫随机场

首先，我们来看看什么是随机场。“随机场”的名字取的很玄乎，其实理解起来不难。随机场是由若干个位置组成的整体，当给每一个位置中按照某种分布随机赋予一个值之后，其全体就叫做随机场。还是举词性标注的例子：假如我们有一个十个词形成的句子需要做词性标注。这十个词每个词的词性可以在我们已知的词性集合（名词，动词...)中去选择。当我们为每个词选择完词性后，这就形成了一个随机场。

了解了随机场，我们再来看看马尔科夫随机场。马尔科夫随机场是随机场的特例，它假设随机场中某一个位置的赋值仅仅与和它相邻的位置的赋值有关，和与其不相邻的位置的赋值无关。继续举十个词的句子词性标注的例子：　如果我们假设所有词的词性只和它相邻的词的词性有关时，这个随机场就特化成一个马尔科夫随机场。比如第三个词的词性除了与自己本身的位置有关外，只与第二个词和第四个词的词性有关。

### 3、从马尔科夫随机场到条件随机场

理解了马尔科夫随机场，再理解CRF就容易了。CRF是马尔科夫随机场的特例，它假设马尔科夫随机场中只有𝑋和𝑌两种变量，𝑋一般是给定的，而𝑌一般是在给定𝑋的条件下我们的输出。这样马尔科夫随机场就特化成了条件随机场。在我们十个词的句子词性标注的例子中，𝑋是词，𝑌是词性。因此，如果我们假设它是一个马尔科夫随机场，那么它也就是一个CRF。

对于CRF，我们给出准确的数学语言描述：

设$𝑋$与$𝑌$是随机变量，$𝑃(𝑌|𝑋)$是给定$𝑋$时$𝑌$的条件概率分布，若随机变量𝑌构成的是一个马尔科夫随机场，则称条件概率分布$𝑃(𝑌|𝑋)$是条件随机场。

### 4、从条件随机场到线性链条件随机场

注意在CRF的定义中，我们并没有要求𝑋和𝑌有相同的结构。而实现中，我们一般都假设𝑋和𝑌有相同的结构，即:

$$X =(x_1,x_2,...x_n),\;\;Y=(y_1,y_2,...y_n)$$

$$【有n个词，对应有n个词性】$$

𝑋和𝑌有相同的结构的CRF就构成了线性链条件随机场(Linear chain Conditional Random Fields,以下简称 linear-CRF)。

在我们的十个词的句子的词性标记中，词有十个，词性也是十个，因此，如果我们假设它是一个马尔科夫随机场，那么它也就是一个linear-CRF。

我们再来看看 linear-CRF的数学定义：

设$X =(x_1,x_2,...x_n)，Y=(y_1,y_2,...y_n)$均为线性链表示的随机变量序列，在给定随机变量序列𝑋的情况下，随机变量𝑌的条件概率分布𝑃(𝑌|𝑋)构成条件随机场，即满足马尔科夫性：

$$P(y_i|X,y_1,y_2,...y_n) = P(y_i|X,y_{i-1},y_{i+1})；i\in[1,n]$$

$$表示在给定一个句子的所有词时，对应某个词的词性的概率，只和这个词的前后词性有关$$

则称𝑃(𝑌|𝑋)为线性链条件随机场。

### 5、线性链条件随机场的参数化形式

对于上一节讲到的linear-CRF，我们如何将其转化为可以学习的机器学习模型呢？这是通过特征函数和其权重系数来定义的。什么是特征函数呢？

在linear-CRF中，特征函数分为两类，第一类【状态特征，对应hmm里的发射矩阵】是定义在$y_i$节点上的节点特征函数，这类特征函数只和当前节点有关，记为：

$$s_l(y_i,X,i),\;\; l =1,2,...L$$

其中𝐿是定义在该节点的节点特征函数的总个数【一个节点会有L个特征函数，符合特征函数的s值为1，不符合的s值为0】，𝑖是当前节点在序列的位置。

第二类【转移特征，对应hmm里的转移矩阵】是定义在$y_i$节点 上下文的局部特征函数，这类特征函数只和当前节点和上一个节点有关，记为：

$$t_k(y_{i-1},y_i,X,i),\;\; k =1,2,...K$$

$$这里为了简化计算，暂时当前词性先只和前一个词的词性有关$$

其中𝐾是定义在该节点的局部特征函数的总个数【一个节点会有K个特征函数，符合特征函数的t值为1，不符合的t值为0】，𝑖是当前节点在序列的位置。之所以只有上下文相关的局部特征函数，没有不相邻节点之间的特征函数，是因为我们的linear-CRF满足马尔科夫性。

无论是节点特征函数还是局部特征函数，它们的取值只能是0或者1。即满足特征条件或者不满足特征条件。同时，我们可以为每个特征函数赋予一个权值，用以表达我们对这个特征函数的信任度。假设$𝑡_𝑘$的权重系数是$𝜆_𝑘,𝑠_𝑙$的权重系数是$𝜇_𝑙$,则linear-CRF由我们所有的$𝑡_𝑘,𝜆_𝑘,𝑠_𝑙,𝜇_𝑙$共同决定。

此时我们得到了linear-CRF的参数化形式如下：

$$P(Y|X) = \frac{1}{Z(X)}exp\Big(\sum\limits_{i,k} \lambda_kt_k(y_{i-1},y_i, X,i) +\sum\limits_{i,l}\mu_ls_l(y_i, X,i)\Big)$$
$$表示在给定X情况下，某个猜测的词性序列的概率。就是给一个句子的所有词，这些词对应的词性组成序列的一种可能占所有可能的概率$$

其中，𝑍(X)为规范化因子【表示给定X情况下，所有可能的词性序列的（特征函数值乘以特征权重）的累加和】：

$$Z(X) =\sum\limits_{y} exp\Big(\sum\limits_{i,k} \lambda_kt_k(y_{i-1},y_i, X,i) +\sum\limits_{i,l}\mu_ls_l(y_i, X,i)\Big)$$

$$y表示一个句子所有词的所有可能词性序列$$

回到特征函数本身，每个特征函数定义了一个linear-CRF的规则，则其系数定义了这个规则的可信度。所有的规则和其可信度一起构成了我们的linear-CRF的最终的条件概率分布。

## 6、线性链条件随机场实例

这里我们给出一个linear-CRF用于词性标注的实例，为了方便，我们简化了词性的种类。假设输入的都是三个词的句子，即$𝑋=(𝑋_1,𝑋_2,𝑋_3)$,输出的词性标记为$𝑌=(𝑌_1,𝑌_2,𝑌_3)$,其中$𝑌\in \{1(名词)，2(动词)\}$

|
特征函数
 |
函数值和对应的条件
 |
特征权重
 |
| :------ | :------ | :------ |
| $t_1$ | =1;($y_1=1,y_2=2,X,2$)或($y_2=1,y_3=2,X,3$)
=0;其他 | $𝜆_1=1$ |
| $t_2$ | =1;($y_1=1,y_2=1,X,2$)
=0;其他 | $𝜆_2=0.5$ |
| $t_3$ | =1;($y_2=2,y_3=1,X,3$)
=0;其他 | $𝜆_3=1$ |
| $t_4$ | =1;($y_1=2,y_2=1,X,2$)
=0;其他 | $𝜆_4=1$ |
| $t_5$ | =1;($y_2=2,y_3=2,X,3$)
=0;其他 | $𝜆_5=0.2$ |
| $s_1$ | =1;($y_1=2,X,1$)
=0;其他 | $\mu_1=1$ |
| $s_2$ | =1;($y_1=2,X,1$)或($y_2=2,X,2$)
=0;其他 | $\mu_1=0.5$ |
| $s_3$ | =1;($y_2=1,X,2$)或($y_3=1,X,3$)
=0;其他 | $\mu_1=0.8$ |
| $s_4$ | =1;($y_3=2,X,3$)
=0;其他 | $\mu_1=0.5$ |

求Y为(1,2,2)词性序列时的非规范化概率。

利用linear-CRF的参数化公式，我们有：

$$exp\Big(\sum\limits_{i,k} \lambda_kt_k(y_{i-1},y_i, X,i) +\sum\limits_{i,l}\mu_ls_l(y_i, X,i)\Big) = exp\Big[\sum\limits_{k=1}^5\lambda_k\sum\limits_{i=2}^3t_k(y_{i-1},y_i, X,i) + \sum\limits_{l=1}^4\mu_l\sum\limits_{i=1}^3s_l(y_i, X,i) \Big]$$

带入(1,2,2)我们有：

$$
\begin{align}
exp\Big[\sum\limits_{k=1}^5\lambda_k\sum\limits_{i=2}^3t_k(y_{i-1},y_i, X,i) + \sum\limits_{l=1}^4\mu_l\sum\limits_{i=1}^3s_l(y_i, X,i) \Big] &= \lambda_1(t_1(y_1=1,y_2=2,X,2)+t_1(y_2=2,y_3=2,X,3)) \\
&+\lambda_2(t_2(y_1=1,y_2=2,X,2)+t_2(y_2=2,y_3=2,X,3)) \\
&+\lambda_3(t_3(y_1=1,y_2=2,X,2)+t_3(y_2=2,y_3=2,X,3)) \\
&+\lambda_4(t_4(y_1=1,y_2=2,X,2)+t_4(y_2=2,y_3=2,X,3)) \\
&+\lambda_5(t_5(y_1=1,y_2=2,X,2)+t_5(y_2=2,y_3=2,X,3)) \\
&+\mu_1(s_1(y_1=1,X,1)+s_2(y_2=2,X,2)+s_3(y_3=2,X,3)) \\
&+\mu_2(s_1(y_1=1,X,1)+s_2(y_2=2,X,2)+s_3(y_3=2,X,3)) \\
&+\mu_3(s_1(y_1=1,X,1)+s_2(y_2=2,X,2)+s_3(y_3=2,X,3)) \\
&+\mu_4(s_1(y_1=1,X,1)+s_2(y_2=2,X,2)+s_3(y_3=2,X,3)) \\
&=(1+0+0+0+0.2)+(1+0.5+0+0.5) \\
&=exp(3.2)
\end{align}
$$

$$
\begin{align}
Z(X)&= y为(y_1=1,y_2=1,y_3=1) \\
&+y为(y_1=1,y_2=1,y_3=2) \\
&+y为(y_1=1,y_2=2,y_3=1) \\
&+y为(y_1=2,y_2=1,y_3=1) \\
&+y为(y_1=2,y_2=1,y_3=2) \\
&+y为(y_1=2,y_2=2,y_3=1) \\
&+y为(y_1=2,y_2=2,y_3=2) \\
&+y为(y_1=1,y_2=2,y_3=2) \\
&=exp(4.2)+exp(5.2)+exp(6.2)+exp(7.2)+exp(8.2)+...+exp(3.2)
\end{align}
$$

$$P(y_1=1,y_2=2,y_3=2|X)=\frac{exp(3.2)}{exp(4.2)+exp(5.2)+exp(6.2)+exp(7.2)+exp(8.2)+...+exp(3.2)}$$

注意：

可以看出一共有2x2x2=8种组合，也就是8种Y可能序列。

## 7、线性链条件随机场的简化形式

在上几节里面，我们用$𝑠_𝑙$表示节点特征函数，用$𝑡_𝑘$表示局部特征函数，同时也用了不同的符号表示权重系数，导致表示起来比较麻烦。其实我们可以对特征函数稍加整理，将其统一起来。

假设我们在某一节点我们有$𝐾_1$个局部特征函数和$𝐾_2$个节点特征函数，总共有$𝐾=𝐾_1+𝐾_2$个特征函数。我们用一个特征函数$𝑓_𝑘(𝑦_{𝑖−1},𝑦_𝑖,X,𝑖)$来统一表示如下:

$$f_k(y_{i-1},y_i,X,i)= \begin{cases} t_k(y_{i-1},y_i, X,i) & {k=1,2,...K_1}\\ s_l(y_i, x,i)& {k=K_1+l,\; l=1,2...,K_2} \end{cases}$$

以为由Y，X参数就可以知道$y_{i-1},y_i,X,i$，所以：

$$f_k(Y,X) = \sum\limits_{i=1}^nf_k(y_{i-1},y_i, X,i)\ \ \ \text{理解为：n代表有n个词}$$

同时我们也统一$𝑓_𝑘(𝑦_{𝑖−1},𝑦_𝑖,X,𝑖)$对应的权重系数$𝑤_𝑘$如下：

$$w_k= \begin{cases} \lambda_k & {k=1,2,...K_1}\\ \mu_l & {k=K_1+l,\; l=1,2...,K_2} \end{cases}$$

这样，我们的linear-CRF的参数化形式简化为：

$$P(Y|X) =  \frac{1}{Z(X)}exp\sum\limits_{k=1}^Kw_kf_k(Y,X)$$

其中，𝑍(X)为规范化因子：

$$Z(X) =\sum\limits_{y}exp\sum\limits_{k=1}^Kw_kf_k(Y,X)$$

如果将上两式中的$𝑤_𝑘$与$𝑓_𝑘$的用向量表示，即:

$$w=(w_1,w_2,...w_K)^T\;\;\; F(Y,X) =(f_1(Y,X),f_2(Y,X),...f_K(Y,X))^T$$

则linear-CRF的参数化形式简化为内积形式如下：

$$P_w(Y|X) = \frac{exp(w \bullet F(Y,X))}{Z_w(X)} = \frac{exp(w \bullet F(Y,X))}{\sum\limits_{y}exp(w \bullet F(Y,X))}$$

## 8、线性链条件随机场的矩阵形式

将上一节统一后的linear-CRF公式加以整理，我们还可以将linear-CRF的参数化形式写成矩阵形式。为此我们定义一个𝑚×𝑚的矩阵𝑀，𝑚为𝑦所有可能的状态的取值个数。𝑀定义如下：

$$M_i(X) = \Big[M_i(y_{i-1},y_i|X)\Big] = \Big[exp(W_i(y_{i-1},y_i|X))\Big] = \Big[  exp(\sum\limits_{k=1}^Kw_kf_k(y_{i-1},y_i,X,i))\Big]$$

我们引入起点和终点标记$𝑦_0=𝑠𝑡𝑎𝑟𝑡,𝑦_{𝑛+1}=𝑠𝑡𝑜𝑝$, 这样，标记序列𝑦的规范化概率可以通过𝑛+1个矩阵元素的乘积得到，即：

$$P_w(Y|X) = \frac{1}{Z_w(X)}\prod_{i=1}^{n+1}M_i(y_{i-1},y_i|X)$$

其中$𝑍_𝑤(X)$为规范化因子。

以上就是linear-CRF的模型基础，后面我们会讨论linear-CRF和HMM类似的三个问题的求解方法。

# 三、前向后向算法评估标记序列概率

## 1、linear-CRF的三个基本问题

在隐马尔科夫模型HMM中，我们讲到了HMM的三个基本问题，而linear-CRF也有三个类似的的基本问题。不过和HMM不同，在linear-CRF中，我们对于给出的观测序列𝑥是一直作为一个整体看待的，也就是不会拆开看(𝑥1,𝑥2,...)，因此linear-CRF的问题模型要比HMM简单一些，如果你很熟悉HMM，那么CRF的这三个问题的求解就不难了。

~~linear-CRF第一个问题是评估，即给定 linear-CRF的条件概率分布𝑃(𝑦|𝑥)【也就是知道了所有特征函数的权重是多少了】, 在给定输入序列𝑥和输出序列𝑦时【计算一个句子中，某个词为某个词性，它的概率是多少；即计算给定的这个y中对应$y_i$取所有的可能值的概率分别是多少】，计算条件概率$𝑃(𝑦_𝑖|𝑥)和𝑃(𝑦_{𝑖−1}，𝑦_𝑖|𝑥)$以及对应的期望。~~

> ~~再来解释下这个问题：\
>给定了$P(y|x)$，你只能拿到所有y对应的序列条件概率分布，却没有在某一序列位置i对应的取值为$y_i$的概率。这个概率需要去计算。\
>举例：3长度序列，2个状态取值，对于给定的x，我们已知$y=(𝑦_0=0,𝑦_1=1,𝑦_2=1)$的条件分布概率。但是此时$y_1=0$或者1的条件分布概率还需要我们去计算。 ~~

linear-CRF第一个问题是评估，即给定linear-CRF的条件概率分布𝑃(𝑦|𝑥)【也就是知道了所有特征函数的权重是多少了】，在给定x的情况下，求条件概率$𝑃(𝑦_𝑖|𝑥)和𝑃(𝑦_{𝑖−1}，𝑦_𝑖|𝑥)$以及对应的期望。

linear-CRF第二个问题是学习，即给定训练数据集𝑋和𝑌，学习linear-CRF的模型参数$𝑤_𝑘$和条件概率$𝑃_𝑤(𝑦|𝑥)$，这个问题的求解比HMM的学习算法简单的多，普通的梯度下降法，拟牛顿法都可以解决。

linear-CRF第三个问题是解码，即给定 linear-CRF的条件概率分布𝑃(𝑦|𝑥),和输入序列𝑥, 计算使条件概率最大的输出序列𝑦。类似于HMM，使用维特比算法可以很方便的解决这个问题。

## 2、第一个问题评估

### 2.1 linear-CRF的前向后向概率概述

给定x输入

例如：给定$x=\{x_1="我",x_2="是",x_3="中国人"\}，y\in \{"动词","名词"\}$

要计算条件概率$𝑃(𝑦_𝑖|𝑥)和𝑃(𝑦_{𝑖−1},𝑦_𝑖|𝑥)$【例如：求$P(y_2="动词"|x)和P(y_1="名词",y_2="动词"|x)$】，我们也可以使用和HMM类似的方法，使用前向后向算法来完成。

我们先看下$𝑃(𝑦_𝑖|𝑥)$等于什么。

>**二维和多维边缘概率** \
>假设有两个随机变量A和B，取值都为{0,1}，我们有:\
>$P(A)=\sum_B P(A,B)=P(A,0)+P(A,1)$ \
>那么\
>$P(A=0)=P(0,0)+P(0,1)$ \
>$P(A=1)=P(1,0)+P(1,1)$ \
>\
>对于三个随机变量A、B、C，取值都为{0,1}，我们依然有: \
>$P(A)=\sum_B \sum_C P(A,B,C)=\sum_B [P(A,B,0)+P(A,B,1)]=[P(A,0,0)+P(A,0,1)]+[P(A,1,0)+P(A,1,1)]$\
>那么\
>$P(A=0)=[P(0,0,0)+P(0,0,1)]+[P(0,1,0)+P(0,1,1)]$ \
>$P(A=1)=[P(1,0,0)+P(1,0,1)]+[P(1,1,0)+P(1,1,1)]$ \
>再来假设\
>$P(A,B,C)=f_1(A)f_2(B)f_3(C)$     $f_1,f_2,f_3$是分别关于变量A，B，C的函数 \
>那么\
>$$
\begin{align}
P(A)=\sum_B \sum_C P(A,B,C)&=\sum_B \sum_C[f_1(A)f_2(B)f_3(C)] \\
&=\sum_B[f_1(A)f_2(B)f_3(0)+f_1(A)f_2(B)f_3(1)]\\
&=[f_1(A)f_2(0)f_3(0)+f_1(A)f_2(0)f_3(1)]+[f_1(A)f_2(1)f_3(0)+f_1(A)f_2(1)f_3(1)]\\
\end{align}
$$
>\
$$
\begin{align}
f_1(A)\big[[\sum_B f_2(B)][\sum_C f_3(C)]\big]&=f_1(A)\big[[f_2(0)+f_2(1)][f_3(0)+f_3(1)]\big]\\
&=f_1(A)\big[f_2(0)f_3(0)+f_2(0)f_3(1)+f_2(1)f_3(0)+f_2(1)f_3(1)\big]\\
&=f_1(A)f_2(0)f_3(0)+f_1(A)f_2(0)f_3(1)+f_1(A)f_2(1)f_3(0)+f_1(A)f_2(1)f_3(1)\\
\end{align}
$$
>所以：\
>$P(A)=\sum_B \sum_C[f_1(A)f_2(B)f_3(C)]=f_1(A)\big[[\sum_B f_2(B)][\sum_C f_3(C)]\big]$这两种形式是等价的

假设一个句子总共有n个词，依据边缘概率有：

$$
\begin{align}
𝑃(𝑦_𝑖|𝑥)
&= \sum_{[y_1,\ y_{i-1}\ ]}\sum_{[y_{i+1}\ ,\ y_{n}\ ]}P(y_1,y_2,...,y_i,...,y_N|x) \\
&= \sum_{[y_1,\ y_{i-1}\ ]}\sum_{[y_{i+1}\ ,\ y_{n}\ ]}P(y|x) \\
&= \sum_{[y_1,\ y_{i-1}\ ]}\sum_{[y_{i+1}\ ,\ y_{n}\ ]}\frac{1}{Z(x)}\prod_{t=1}^{T}exp\left\{\sum_{k=1}^{K}W_{k}F_{k}(y_{t-1}, y_{t}, x)\right\}\\
&我们另f_t(y_{t-1}, y_{t})=exp\left\{\sum_{k=1}^{K}W_{k}F_{k}(y_{t-1}, y_{t}, x)\right\} \\
&那么 \\
&= \sum_{[y_1,\ y_{i-1}\ ]}\sum_{[y_{i+1}\ ,\ y_{n}\ ]}\frac{1}{Z(x)}\prod_{t=1}^{T}f_t(y_{t-1}, y_{t})\\
&假设y_i的取值个数为s，则上式的计算复杂度为O(s^T*T)，指数级复杂度基本就没法暴力计算，所以我们找技巧\\
&=\frac{1}{Z(x)}.\\
&\big[\sum_{y_{i-1}}\big(f_i(y_{i-1}, y_{i})...\sum_{y_2}\big(f_3(y_{2}, y_{3})\sum_{y_1}\big(f_2(y_{1}, y_{2})\sum_{y_0}f_1(y_{0}, y_{1})\big)\big)\big)\big].\\
&\big[\sum_{y_{i+1}}\big(f_{i+1}(y_{i}, y_{i+1})...\sum_{y_{n-1}}\big(f_{n-1}(y_{n-2}, y_{n-1})\sum_{y_{n}}\big(f_n(y_{n-1}, y_{n})\sum_{y_{n+1}}f_{n+1}(y_{n}, y_{n+1})\big)\big)\big)\big]\\
&在这里我们增加了y_0和y_{n+1}是为了方便我们计算 \\
&我们定义\alpha_i(y_i|x)表示序列位置i的标记是y_i时，在位置i之前(包括i)的部分标记序列的非规范化概率，\\
&我们定义\beta_i(y_i|x)表示序列位置i的标记是y_i时，在位置𝑖之后的从i+1到n的部分标记序列的非规范化概率 \\
&在起点和终点处，我们定义： \\
&\alpha_0(y_0|x)= \begin{cases} 1 & {y_0 =start}\\ 0 & {else} \end{cases}\\
&\beta_{n+1}(y_{n+1}|x)= \begin{cases} 1 & {y_{n+1} =stop}\\ 0 & {else} \end{cases}\\
&我们写出递推公式为：\\
&\alpha_i(y_i|x)=\sum_{y_{i-1}}\big(f_i(y_{i-1}, y_{i})\alpha_{i-1}(y_{i-1}|x)\big)\\
&\beta_i(y_i|x)=\sum_{y_{i+1}}\big(f_{i+1}(y_{i}, y_{i+1})\beta_{i+1}(y_{i+1}|x)\big)\\
&所以最终我们的推导结果为：\\
&=\frac{1}{Z(x)}.\alpha_i(y_i|x).\beta_i(y_i|x)
\end{align}
$$

同理可以得到：
$$
\begin{align}
𝑃(𝑦_{𝑖−1},𝑦_𝑖|𝑥)=\frac{1}{Z(x)}.\alpha_{i-1}(y_{i-1}|x)f_i(y_{i-1},y_i).\beta_i(y_i|x)
\end{align}
$$

### 2.2 linear-CRF的期望计算

某函数f(x)的关于某分布P(x)的期望：$\sum_xP(x)f(x)$

有了上一节计算的条件概率，我们也可以很方便的计算联合分布𝑃(𝑥,𝑦)与条件分布𝑃(𝑦|𝑥)的期望。

特征函数$𝑓_𝑘(𝑥,𝑦)$关于条件分布𝑃(𝑦|𝑥)的期望表达式是：

$$\begin{align} E_{P(y|x)}[f_k]  & = E_{P(y|x)}[f_k(y,x)] \\ & = \sum\limits_{i=1}^{n+1} \sum\limits_{y_{i-1}\;\;y_i}P(y_{i-1},y_i|x)f_k(y_{i-1},y_i,x, i) \\ & =  \sum\limits_{i=1}^{n+1} \sum\limits_{y_{i-1}\;\;y_i}f_k(y_{i-1},y_i,x, i)  \frac{\alpha_{i-1}^T(y_{i-1}|x)M_i(y_{i-1},y_i|x)\beta_i(y_i|x)}{Z(x)} \end{align}$$

同样可以计算联合分布𝑃(𝑥,𝑦)的期望：

$$
\begin{align} E_{P(x,y)}[f_k]  & = \sum\limits_{x,y}P(x,y) \sum\limits_{i=1}^{n+1}f_k(y_{i-1},y_i,x, i) \\& =  \sum\limits_{x}\tilde{P}(x) \sum\limits_{y}P(y|x) \sum\limits_{i=1}^{n+1}f_k(y_{i-1},y_i,x, i) \\& =  \sum\limits_{x}\tilde{P}(x)\sum\limits_{i=1}^{n+1} \sum\limits_{y_{i-1}\;\;y_i}f_k(y_{i-1},y_i,x, i)  \frac{\alpha_{i-1}^T(y_{i-1}|x)M_i(y_{i-1},y_i|x)\beta_i(y_i|x)}{Z(x)}
\end{align}
$$

这里假设已知边缘分布 P(X) 的经验分布为 $\tilde{P}(x)$ ，经验分布就是根据训练数据，用频数估计的方式得到 $\tilde{P}(x)=\frac{𝑥}{𝑁}$，可以参考最大熵模型算法。

假设一共有𝐾个特征函数，则𝑘=1,2,...𝐾

## 3、第二个问题模型学习

在linear-CRF模型参数学习问题中，我们给定训练数据集𝑋和对应的标记序列𝑌，𝐾个特征函数$𝑓_𝑘(𝑥,𝑦)$，需要学习linear-CRF的模型参数$𝑤_𝑘$和条件概率$𝑃_𝑤(𝑦|𝑥)$，其中条件概率$𝑃_𝑤(𝑦|𝑥)$和模型参数$𝑤_𝑘$满足一下关系：

$$P_w(y|x) = P(y|x) =  \frac{1}{Z_w(x)}exp\sum\limits_{k=1}^Kw_kf_k(x,y) =  \frac{exp\sum\limits_{k=1}^Kw_kf_k(x,y)}{\sum\limits_{y}exp\sum\limits_{k=1}^Kw_kf_k(x,y)}$$

所以我们的目标就是求出所有的模型参数𝑤𝑘，这样条件概率𝑃𝑤(𝑦|𝑥)可以从上式计算出来。

求解这个问题有很多思路，比如梯度下降法，牛顿法，拟牛顿法。同时，这个模型中$𝑃_𝑤(𝑦|𝑥)$的表达式和最大熵模型原理小结中的模型一样，也可以使用最大熵模型中使用的改进的迭代尺度法(improved iterative scaling, IIS)来求解。

下面我们只简要介绍用梯度下降法的求解思路。

### 3.1 linear-CRF模型参数学习之梯度下降法求解

在使用梯度下降法求解模型参数之前，我们需要定义我们的优化函数，一般极大化条件分布$𝑃_𝑤(𝑦|𝑥)$的对数似然函数如下：

$$L(w)=  log\prod_{x,y}P_w(y|x)^{\tilde{P}(x,y)} = \sum\limits_{x,y}\tilde{P}(x,y)logP_w(y|x)$$

> 这里可以加经验概率也可以不加，加入的好处： \
> 举个简单的例子，中国人口分布，如果你没有先验知识，你会假设每个地方人口的分布都是一样的，均匀的。然后根据你的训练数据来优化你的人口分布认知。\
>如果你有先验知识，知道西部人口少，东部人口多，那么你的先验分布就不是均匀分布了，这也是那个指数经验分布的意义。
>
>这里用这个指数来标示当前样本出现的频度。
>
>这个频度如果没有先验知识，就是从训练样本中得出，可以每个样本取值为1，优化目标也就是你上面提到的直接取对数。否则从先验知识得出。

其中$\tilde{P}(𝑥,𝑦)$为经验分布，可以从先验知识和训练集样本中得到，这点和最大熵模型类似。为了使用梯度下降法，我们现在极小化$𝑓(𝑤)=−𝐿(𝑃_𝑤)$如下：

$$\begin{align}f(w) & = -\sum\limits_{x,y}\tilde{P}(x,y)logP_w(y|x) \\ &=  \sum\limits_{x,y}\tilde{P}(x,y)logZ_w(x) - \sum\limits_{x,y}\tilde{P}(x,y)\sum\limits_{k=1}^Kw_kf_k(x,y) \\& =  \sum\limits_{x}\tilde{P}(x)logZ_w(x) - \sum\limits_{x,y}\tilde{P}(x,y)\sum\limits_{k=1}^Kw_kf_k(x,y) \\& =  \sum\limits_{x}\tilde{P}(x)log\sum\limits_{y}exp\sum\limits_{k=1}^Kw_kf_k(x,y) - \sum\limits_{x,y}\tilde{P}(x,y)\sum\limits_{k=1}^Kw_kf_k(x,y)  \end{align}$$

对𝑤求导可以得到：

$$\frac{\partial f(w)}{\partial w} = \sum\limits_{x,y}\tilde{P}(x)P_w(y|x)f(x,y) -  \sum\limits_{x,y}\tilde{P}(x,y)f(x,y)$$

有了𝑤的导数表达式，就可以用梯度下降法来迭代求解最优的𝑤了。注意在迭代过程中，每次更新𝑤后，需要同步更新$𝑃_𝑤(𝑥,𝑦)$，以用于下一次迭代的梯度计算。

## 4、第三个问题解码

利用维特比算法解码

我们现在已经学习出了$𝑃_𝑤(y|X)$，也就是给定一句话，其中某个词是某个词性的概率，现在我们要解码出，给定一句话，每个词的词性都是什么时，概率最大。

对于我们linear-CRF中的维特比算法，我们的第一个局部状态定义为$𝛿_𝑖(𝑙)$,表示在位置𝑖标记𝑙各个可能取值(1,2...m)对应的非规范化概率的最大值。之所以用非规范化概率是，规范化因子𝑍(𝑥)不影响最大值的比较。根据$𝛿_𝑖(𝑙)$的定义，我们递推在位置𝑖+1标记𝑙的表达式为：

$$\delta_{i+1}(l) = \max_{1 \leq j \leq m}\{\delta_i(j) + \sum\limits_{k=1}^Kw_kf_k(y_{i} =j,y_{i+1} = l,x,i)\}\;, l=1,2,...m$$

和HMM的维特比算法类似，我们需要用另一个局部状态$Ψ_{𝑖+1}(𝑙)$来记录使$𝛿_{𝑖+1}(𝑙)$达到最大的位置𝑖的标记取值,这个值用来最终回溯最优解，$Ψ_{𝑖+1}(𝑙)$的递推表达式为：

$$\Psi_{i+1}(l) = arg\;\max_{1 \leq j \leq m}\{\delta_i(j) + \sum\limits_{k=1}^Kw_kf_k(y_{i} =j,y_{i+1} = l,x,i)\}\; ,l=1,2,...m$$

## 二、条件随机场—分词

### 1、条件随机场介绍

    条件随机场( Conditional Random Field, CRF)是一种给定输入随机变量 x，求解条件概率 p(y|x) 的概率无向图模型。用于序列标注时，特例化为线性链( linear chain )条件随机场。此时，输人输出随机变量为等长的两个序列。

每个 $X_{t}$ 上方有 3 个灰色节点，代表 $X_{t}$ 的 3 个特征，当然还可以是任意数量的特征，体现了特征的丰富性，黑色方块是因子节点，可以理解为一个特征函数 。其中仅仅利用了 $X_{t}$ 和 $X_{t}$ 的特征称作状态特征，利用了 $Y_{t-1}$ 的特征则称作转移特征，与感知机的特征函数相同。

    线性链条件随机场的定义如下:

$$P(Y|X) = \frac{1}{Z(X)}\prod_{t=1}^{T}exp\left\{\sum_{k=1}^{K}W_{k}F_{k}(Y_{t-1}, Y_{t}, X_{t})\right\}$$

   K代表特征的总个数，

   $F_{k}$ 代表第k个特征的特征函数，

   $W_{k}$代表第k个特征函数求出的特征值所对应的权重。

    其中，Z(X)为归一化函数:

$$Z(X) = \sum_{y}\prod_{t=1}^{T}exp\left\{\sum_{k=1}^{K}W_{k}F_{k}(Y_{t-1}, Y_{t}, X_{t})\right\}$$

注意：
    y就是所有可能的序列，计算开销很大。线性链条件随机场公式的分子是一个句子某个类标序列的总得分，分母是一个句子所有可能类标序列的总得分，如果一个句子有10个字，那么所有可能的类标序列就有$4^{10}$种。

    上式定义在所有可能的标注序列上。如果将所有特征函数与权重分别写作向量形式，则线性链条件随机场的定义可简化为:

$$
\begin{aligned}
P(Y|X) &= \frac{1}{Z(X)}\prod_{t=1}^{T}exp\left\{\sum_{k=1}^{K}W_{k}F_{k}(Y_{t-1}, Y_{t}, X_{t})\right\}\\
&= \frac{1}{Z(X)}\prod_{t=1}^{T}exp\left\{W.\phi(Y_{t-1}, Y_{t}, X_{t})\right\}\\
&= \frac{1}{Z(X)}exp\left\{\sum_{t=1}^{T}W.\phi(Y_{t-1}, Y_{t}, X_{t})\right\}
\end{aligned}
$$

    对比结构化感知机的打分函数:

$$score(X,Y) = \sum_{t=1}^{T}W.\phi(Y_{t-1}, Y_{t}, X_{t})$$

    可以发现结构化感知机打分函数与条件随机场的指数部分完全相同，由于给定实例 X，Z(X) 就是一个常数 c，所以有:

$$P(Y|X) = \frac{1}{c}exp\left\{score(X,Y)\right\}$$

    于是，条件随机场就和结构化感知机有以下联系:

> 条件随机场和结构化感知机的特征函数完全一致。\
结构化感知机预测打分越高，条件随机场给予该预测的概率也越大。\
这种相似性使得我们能够复用结构化感知机的预测算法，也就是维特比算法。

### 2、条件随机场的训练

指定训练集$D=\lbrace x^{(i)},y^{(i)}\rbrace _{i=1}^{N}$，N是总样本数（如果是句子，就是所有待训练的句子数）。其中每个$X^{(i)}={X_1^{(i)},X_2^{(i)},...X_T^{(i)}}$为输入序列，$Y^{(i)}={Y_1^{(i)},Y_2^{(i)},...Y_T^{(i)}}$为相应的标签序列。

    根据极大似然估计，我们想要最大化给定模型参数W时，训练集D的似然概率：
$$P(D|W)=\prod_{i=1}^{N}P(Y^{(i)}|X^{(i)})$$

    等价于极大化对数似然函数：
$$L(W)=\sum_{i=1}^{N}lnP(Y^{(i)}|X^{(i)})$$

将$P(Y|X) = \frac{1}{Z(X)}\prod_{t=1}^{T}exp\left\{\sum_{k=1}^{K}W_{k}F_{k}(Y_{t-1}, Y_{t}, X_{t})\right\}$代入上式得到：

$$L(W)=\sum_{i=1}^{N}\sum_{t=1}^{T}\sum_{k=1}^{K}W_{k}F_{k}(y_{t-1}^{(i)},y_{t}^{(i)},x_{t}^{(i)})-\sum_{i=1}^{N}lnZ(X)$$

$$Z(X) = \sum_{y}\prod_{t=1}^{T}exp\left\{\sum_{k=1}^{K}W_{k}F_{k}(Y_{t-1}, Y_{t}, X_{t})\right\}$$

其中N是样本集总数量，现在我们要用随机梯度下降法（SGD）来极大似然函数（准确地说应该是利用梯度上升法，来求最大值），因此每次只用处理一个训练序列，于是有：
$$L(W)=\sum_{t=1}^{T}\sum_{k=1}^{K}W_{k}F_{k}(y_{t-1}^{(i)},y_{t}^{(i)},x_{t}^{(i)})-lnZ(X)$$

    那么对数似然函数的偏导为：

$$
\begin{aligned}
\frac{\partial L}{\partial W_{k}}&=\sum_{t=1}^{T}\sum_{k=1}^{K}F_k(y_{t-1}^{(i)},y_{t}^{(i)},x_{t}^{(i)})-\frac{1}{Z(X)}\frac{\partial Z(X)}{\partial W_{k}}  \\
&=\sum_{t=1}^{T}\sum_{k=1}^{K}F_k(y_{t-1}^{(i)},y_{t}^{(i)},x_{t}^{(i)})-\frac{1}{Z(X)}\sum_{y}\bigg(exp\left\{\sum_{t=1}^{T}\sum_{k=1}^{K}W_kF_{k}(Y_{t-1}, Y_{t}, X_{t})\right\}\sum_{t=1}^{T}\sum_{k=1}^{K}F_{k}(Y_{t-1}, Y_{t}, X_{t})\bigg)  \\
\end{aligned}
$$

因为:

$$P(Y|X)=\frac{1}{Z(X)}\prod_{t=1}^{T}exp\left\{\sum_{k=1}^{K}W_{k}F_{k}(Y_{t-1}, Y_{t}, X_{t})\right\}$$

所以偏导式子化简为：

$$\frac{\partial L}{\partial W_{k}}=\sum_{t=1}^{T}\sum_{k=1}^{K}F_k(y_{t-1}^{(i)},y_{t}^{(i)},x_{t}^{(i)})-\sum_{y}\bigg(P(Y|X)\sum_{t=1}^{T}\sum_{k=1}^{K}F_k(y_{t-1},y_{t},x_{t}^{(i)})\bigg)$$

推导完毕，这个式第一项是真实值，第二项是期望值，当两者相等时，梯度为0，迭代停止。

**这里需要累加所有样本的梯度，因为crf的权重更新是针对所有样本的一次迭代，假设样本数为M：**

$$\frac{\partial L}{\partial W_{k}}=\sum_{m=1}^M\left[\sum_{t=1}^{T}\sum_{k=1}^{K}F_k(y_{t-1}^{(i)},y_{t}^{(i)},x_{t}^{(i)})-\sum_{y}\bigg(P(Y|X)\sum_{t=1}^{T}\sum_{k=1}^{K}F_k(y_{t-1},y_{t},x_{t}^{(i)})\bigg)\right]$$

实际应用中特征函数、观测值的取值数量可能非常大，梯度下降法、牛顿法求解速度太慢，一般采用L-BFGS算法，其原理在《机器学学习与应用》一书中有详细的讲解。

> **第一项真实值**
>
> 就是一句话对应的真实类标序列。
>
> **第二项期望值**
>
> 对一句话的每个字进行类标的排列组合序列计算每个类标序列对应的概率$P(Y|X)$，乘以每个类别序列所对应的$\sum_{t=1}^{T}F_k(y_{t-1},y_{t},x_{t}^{(i)})$累计得分，即为每个类标序列得分在$P(Y|X)$概率分布下的期望值。
>
> **例子：**
>
>
>
> $x^i$ = 小明是中国人；$y^i$=BESSBME
>
> W是每个特征函数的权重，包含了x特征和y特征【这个是依据hanlp的平均结构化感知器的逻辑，可以参考它的权重定义和更新规则】。
>
> 首先利用模型+维特比算法预测出该句子的类标序列$\hat{y}$，然后比较$\hat{y}$和$y^i$，如果不一致，则更新W权重。
>
>关于W偏导式子的第一项就是计算真实类标序列在每个特征上的得分的累加值，如果几个字有相同的特征那么该特征得分就会累加。
>
>第二项就是句子的字数乘以类标数的自由组合的类标序列，求得每个类标序列的得分然后乘以该类标序列的概率。

利用前向-后向算法，特征函数的期望可以用下面的式子进行求解

$$E_{P(Y|X)}(f_k)=\sum_{y}P(y|x){f_k}(y,x)={\sum_{i=1}^{n+1}} \sum_{y_{i-1}\ \ ,y_i} f_k(y_{i-1},y_i,x,i)\bullet{\frac{{\alpha_i}^T(y_{i-1}|x)M_i (y_{i-1},{y_i}|x){\beta_i}(y_i|x)}{Z(x)}}$$

### 3、条件随机场和结构化感知器对比

    结构化感知机和条件随机场的相同点:

>特征函数相同：选取的特征和对应的特征函数是同结构化感知器一样的\
>权重向量相同\
>打分函数相同\
>预测算法相同\
>同属结构化学习

    不同点:

>1.感知机更新参数时，只使用一个训练实例，没有考虑整个数据集【应该是每次只考虑一个字，而没有考虑整个句子；后期已经证实是每个训练实例即每个句子】，难免顾此失彼；而条件随机场对数似然函数及其梯度则使用了整个数据集(整个句子；后期已经证实是整个数据集)【条件随机场应该可以设置batch，即训练多少个句子更新一次权重；后期已经证实他就是整个数据集做一次权重更新】。
>
> 2.条件随机场更新参数更加合理，条件随机场更新参数如下:
$$W \leftarrow W+\phi(X^{i},Y^{i})-E_{w}\left[\phi(X^{i},Y)\right]$$
>
> 对比感知机的更新参数表达式:
>$$W \leftarrow W+\phi(X^{i},Y^{i})-\phi(X^{i},\hat{Y})$$
>
> 两者的差距一目了然，感知机奖励正确答案对应的特征函数 ϕ，但仅惩罚错误最厉害的那个 y，而条件随机场同时惩罚所有答案 y，分摊惩罚总量。

总结：

两者的差距一目了然，感知机奖励正确答案对应的特征函数中$\phi(X^{i},Y^{i})$。但仅惩罚错得最厉害的那一个类标于对应的特征函数$\phi(X^{i},\hat{Y})$，可谓枪打出头鸟。

而条件随机场不但奖励正确答案对应的特征函数，还同时惩罚所有答案y。这样就导致所有错误结果对应的特征函数都受到惩罚，可谓有罪么罚。条件随机场的惩罚总量依然为1。但所有答案分摊。而且每个答案的特征函数受到的惩罚力度精确为模型赋于它的概率，可谓依罪量刑。

正确答案的特征函数反而已经受到一单位的奖励(即$+\phi(X^{i},Y^{i})$)。所以不在平那点小于一单位的您罚(即$\sum_{y}\bigg(P(Y|X)\sum_{t=1}^{T}\sum_{k=1}^{K}F_k(y_{t-1},y_{t},x_{t}^{(i)})\bigg)$，由于$P(Y|X)\leq 1$，所以称作小于单位的惩罚)。

当特征函数的模型期望与经验分布期望一致时(即$P(Y|X) = 1$)，梯度为0，模型参数不再变化。此时，全量梯度上升算法一定收敛到最优解。

注意：

CRF++训练详解

流程概述:

1. 生成特征函数

2. 构建概率图

3. 计算node和path的代价:node代价计算即该node对应的所有特征函数权重;path代价即该path对应的所有特征函数权重之和(node和path是有特定的label的)

4. 前向-后向算法计算alpha和beta, 计算期望

以上都是针对某个训练样本

以下是针对所有样本

5. 计算梯度g(w)(所有训练样本的期望-1),计算目标函数L(W)

6. LGBLF算法一轮迭代后更新W(代码中的alpha)

```python

```


```python
import sympy as sy
import sympy.abc
import matplotlib.pyplot as plt
import numpy as np
import control as pc
```

Part of the big question is whether we are simulating the dynamic response, or the steady-state mathematics iterations. Here below is the steady state reponse mathamtical system design, rather than a dynamic diffferential changes with time. Hence, Python's control toolbox cannot solve a steady-state response of a already stable system. However, attempting to perform a fully stable model means that if we want to perform feedback, there's a factor of recalculating stability from that until differential changes become 0 at different time-steps. So for example, if you were to perform feedback after an initial state, you can get a steady state reponse, but this is not the dynamic response, this is solving for the right answer after a number of iterations, and all the initial steady-state answers are wrong for a feedback system. So this gives a steady state response. If we change an input signal also at another point of time, then there is a new steady state response, but the question is whether you want to account for the time dynamics. Because currently, the system is accounting for the digital computing dynamics rather than the actual transient model, so you can't account for how the system dynamically responds and results in a real-time steady-state. So if an input changes in time, you can actually model the time differential change. If we begin considering delays and inputs into the rest of the system, we need to model time as time and computations as computations - and hence we need a time variable input.

There's also the aspect that optics is fundamentally linear and 3D, so how do we account for differential changes in it? What is its dynamic response? There are only dynamic changes on dynamic inputs. What are time changes in optics? You could argue it is near instantaneous. So there's group delay as the time it takes for light to travel through the waveguide, specially if it has been affected in any way. So there is a need to model transient and steady-state responses, but each of these models should be kept separate.

Now, getting a time domain model of a thermo-optic phase shifter response is non-trivial nor necessarily accurate. So, time is a completely new thing in the optical domain for these type of modelled systems. Yes, there is propagation time effects.

Let's consider an MZI, in one arm there's light going over a certain length which incurrs a certain propagation time. Assuming we're only in steady-state mode, when does change occur? You can differtiate phase shift against length for example, but you would be discretizing space-wise. So imagine Pac-Mans moving in a grid of time, according to their position, their properties change, such as phase and this is also dependent in temperature at a given position, but then there must be a differential in temperature and length to calculate a steady state response. So there's this big aspect of time differentials. Could we work based purely on steady-state time differentials? Actually, that could work. There's also the factor of where to add delays, but if we consider every wavelength arm as a system, then we can simply follow the basic principles of control theory.

Hence, the time discretization must occur at a fraction of the time of propagation of the shortest path in the model.

Let's assume an input to a combiner $E_0(t)$, hence each path signal input becomes $E_1 = E_2 = \frac{E_0}{\sqrt{2}}$.

Beamsplitter Splitter Input at 50/50%
\begin{equation}
\begin{bmatrix} E_{1} \\ E_{2} \end{bmatrix} =
\frac{E_0}{\sqrt{2}}  = \begin{bmatrix} r_{ac}& t_{bc} \\  t_{ad}& r_{bd} \end{bmatrix}\begin{bmatrix} E_0 \\ E_0 \end{bmatrix}  =  \begin{bmatrix} \sqrt{r} &  i\sqrt{1-r} \\  i\sqrt{1-r} &  \sqrt{r}  \end{bmatrix}
\begin{bmatrix} E_0 \\ E_0 \end{bmatrix}
\end{equation}

Beamsplitter Combiner Output for 50/50

\begin{equation}
E_3 = \frac{E_1 + E_2}{\sqrt{2}} = \frac{1}{\sqrt{2}}  \begin{bmatrix} 1 & 1 \end{bmatrix}
\begin{bmatrix} E_1 \\ E_2 \end{bmatrix} = \begin{bmatrix} \sqrt{r} &  i\sqrt{1-r} \\  0 & 0  \end{bmatrix}
\begin{bmatrix} E_1 \\ E_2 \end{bmatrix}
\end{equation}

Assuming a phase shifter on top arm
\begin{equation}
\textbf{ $\Phi$ } = \begin{bmatrix} e^{i\phi} & 0 \\ 0 & 1\end{bmatrix}
\end{equation}

Note we're not yet taking into account Linear and LinearGroup delay, or the quantum probability variable within the design of the device.
Also, for us to be able to have a simulateable model, it must be a finite model - ie steady state must be able to be reached.

The standard model of a 1D light electromagnetic wave is:

$$E_0(t, z) = E_0 e^{j (wt - \beta z)}$$

There's the question as well whether a given model can reach this steady state by ignoring the complex propagation constant of the model? Because if it cannot, then there's no point following the rest of the derivation.

For a 1x1 MZI, we can represent this as a Beamsplitter Splitter - Phase shifter on a length - Beamsplitter Combiner

\begin{equation}
E_3 = \begin{bmatrix} \sqrt{r} &  i\sqrt{1-r} \\  0 & 0  \end{bmatrix}
\begin{bmatrix} e^{i\phi} & 0 \\ 0 & 1\end{bmatrix}
\begin{bmatrix} \sqrt{r} &  i\sqrt{1-r} \\  i\sqrt{1-r} &  \sqrt{r}  \end{bmatrix}
\begin{bmatrix} E_0 \\ E_0 \end{bmatrix}
\end{equation}

```python
t = sy.Symbol("t")
T = sy.Symbol("T")
E_0 = sy.Function("E_0")(t)
n = sy.Function("n")(t,T)

E_0_s = sy.Symbol("E_0")
E_1 = E_2 = E_0 / sy.sqrt(2)
L = sy.Symbol("L")
lambda_0 = sy.Symbol("\lambda_0")
phi = sy.Function("\phi")(t)
r_0, r_1 = sy.symbols("r_0 r_1")

dE_0_s = sy.Symbol("dE_0")
dE_1_s = sy.Symbol("dE_1")
d_phi_d_t_s = sy.Symbol("d\phi")

```

```python
splitter_beamsplitter  = sy.Matrix([[sy.sqrt(r_0), sy.I*sy.sqrt(1 - r_0)],
				[sy.I*sy.sqrt(1 - r_0), sy.sqrt(r_0)]])
combiner_beamsplitter  = sy.Matrix([[sy.sqrt(r_1), sy.I*sy.sqrt(1 - r_1)],
				[0,0]])
phase_changer = sy.Matrix([[sy.exp(sy.I * phi), 0],
				[0, 1]])
E = sy.Matrix([[E_0], [E_0]])

out = combiner_beamsplitter * phase_changer * splitter_beamsplitter * E
out
```

$\displaystyle \left[\begin{matrix}\left(i \sqrt{r_{0}} \sqrt{1 - r_{1}} + i \sqrt{r_{1}} \sqrt{1 - r_{0}} e^{i \phi{\left(t \right)}}\right) \operatorname{E_{0}}{\left(t \right)} + \left(\sqrt{r_{0}} \sqrt{r_{1}} e^{i \phi{\left(t \right)}} - \sqrt{1 - r_{0}} \sqrt{1 - r_{1}}\right) \operatorname{E_{0}}{\left(t \right)}\\0\end{matrix}\right]$

```python
differential_time_response = out.diff(t)
differential_time_response = differential_time_response.replace(E_0.diff(t), dE_0_s)
differential_time_response
```

$\displaystyle \left[\begin{matrix}dE_{0} \left(i \sqrt{r_{0}} \sqrt{1 - r_{1}} + i \sqrt{r_{1}} \sqrt{1 - r_{0}} e^{i \phi{\left(t \right)}}\right) + dE_{0} \left(\sqrt{r_{0}} \sqrt{r_{1}} e^{i \phi{\left(t \right)}} - \sqrt{1 - r_{0}} \sqrt{1 - r_{1}}\right) + i \sqrt{r_{0}} \sqrt{r_{1}} \operatorname{E_{0}}{\left(t \right)} e^{i \phi{\left(t \right)}} \frac{d}{d t} \phi{\left(t \right)} - \sqrt{r_{1}} \sqrt{1 - r_{0}} \operatorname{E_{0}}{\left(t \right)} e^{i \phi{\left(t \right)}} \frac{d}{d t} \phi{\left(t \right)}\\0\end{matrix}\right]$

The problem becomes an equilibrium condition to solve. One could argue that when the output is in equilibrium or in steady state:

\begin{equation}
\begin{bmatrix} \frac{dE_{3}}{dt}  \\ 0 \end{bmatrix}  = 0
\end{equation}

```python
dE_0_d_t = sy.solve(differential_time_response[0], dE_0_s)[0]
dE_0_d_t = dE_0_d_t.replace(E_0, E_0_s)
dE_0_d_t = dE_0_d_t.replace(phi.diff(t), d_phi_d_t_s)
dE_0_d_t
```

$\displaystyle \frac{E_{0} d\phi \sqrt{r_{1}} \left(- i \sqrt{r_{0}} + \sqrt{1 - r_{0}}\right) e^{i \phi{\left(t \right)}}}{\sqrt{r_{0}} \sqrt{r_{1}} e^{i \phi{\left(t \right)}} + i \sqrt{r_{0}} \sqrt{1 - r_{1}} + i \sqrt{r_{1}} \sqrt{1 - r_{0}} e^{i \phi{\left(t \right)}} - \sqrt{1 - r_{0}} \sqrt{1 - r_{1}}}$

```python
c = sy.abc.a + sy.abc.b
d = sy.lambdify([sy.abc.a, sy.abc.b], c)
d(3,3)
```

    6

```python
t = sy.lambdify([E_0_s, r_0, r_1, phi, d_phi_d_t_s], dE_0_d_t)
A = t(1,0.5,0.5,4,5)
np.imag(A)
```

    -2.5000000000000004

```python
s= pc.step_response(pc.ss(A,0.5,0.5,0))
s
```

    /Users/daquintero/opt/anaconda3/lib/python3.8/site-packages/numpy/matrixlib/defmatrix.py:147: ComplexWarning: Casting complex values to real discards the imaginary part
      arr = N.array(data, dtype=dtype, copy=copy)

    (array([0.        , 0.00685368, 0.01370736, 0.02056104, 0.02741472,
            0.0342684 , 0.04112207, 0.04797575, 0.05482943, 0.06168311,
            0.06853679, 0.07539047, 0.08224415, 0.08909783, 0.09595151,
            0.10280519, 0.10965886, 0.11651254, 0.12336622, 0.1302199 ,
            0.13707358, 0.14392726, 0.15078094, 0.15763462, 0.1644883 ,
            0.17134198, 0.17819566, 0.18504933, 0.19190301, 0.19875669,
            0.20561037, 0.21246405, 0.21931773, 0.22617141, 0.23302509,
            0.23987877, 0.24673245, 0.25358612, 0.2604398 , 0.26729348,
            0.27414716, 0.28100084, 0.28785452, 0.2947082 , 0.30156188,
            0.30841556, 0.31526924, 0.32212292, 0.32897659, 0.33583027,
            0.34268395, 0.34953763, 0.35639131, 0.36324499, 0.37009867,
            0.37695235, 0.38380603, 0.39065971, 0.39751339, 0.40436706,
            0.41122074, 0.41807442, 0.4249281 , 0.43178178, 0.43863546,
            0.44548914, 0.45234282, 0.4591965 , 0.46605018, 0.47290385,
            0.47975753, 0.48661121, 0.49346489, 0.50031857, 0.50717225,
            0.51402593, 0.52087961, 0.52773329, 0.53458697, 0.54144065,
            0.54829432, 0.555148  , 0.56200168, 0.56885536, 0.57570904,
            0.58256272, 0.5894164 , 0.59627008, 0.60312376, 0.60997744,
            0.61683111, 0.62368479, 0.63053847, 0.63739215, 0.64424583,
            0.65109951, 0.65795319, 0.66480687, 0.67166055, 0.67851423]),
     array([0.00000000e+00, 1.75348537e-03, 3.58960997e-03, 5.51226845e-03,
            7.52553903e-03, 9.63369213e-03, 1.18411994e-02, 1.41527433e-02,
            1.65732270e-02, 1.91077844e-02, 2.17617920e-02, 2.45408790e-02,
            2.74509404e-02, 3.04981487e-02, 3.36889675e-02, 3.70301650e-02,
            4.05288283e-02, 4.41923785e-02, 4.80285865e-02, 5.20455895e-02,
            5.62519079e-02, 6.06564640e-02, 6.52686005e-02, 7.00981002e-02,
            7.51552073e-02, 8.04506484e-02, 8.59956561e-02, 9.18019918e-02,
            9.78819718e-02, 1.04248492e-01, 1.10915058e-01, 1.17895809e-01,
            1.25205553e-01, 1.32859794e-01, 1.40874768e-01, 1.49267477e-01,
            1.58055722e-01, 1.67258144e-01, 1.76894263e-01, 1.86984518e-01,
            1.97550312e-01, 2.08614057e-01, 2.20199220e-01, 2.32330375e-01,
            2.45033253e-01, 2.58334800e-01, 2.72263229e-01, 2.86848086e-01,
            3.02120305e-01, 3.18112281e-01, 3.34857937e-01, 3.52392790e-01,
            3.70754036e-01, 3.89980621e-01, 4.10113327e-01, 4.31194858e-01,
            4.53269931e-01, 4.76385370e-01, 5.00590206e-01, 5.25935781e-01,
            5.52475856e-01, 5.80266727e-01, 6.09367340e-01, 6.39839424e-01,
            6.71747612e-01, 7.05159587e-01, 7.40146220e-01, 7.76781722e-01,
            8.15143802e-01, 8.55313831e-01, 8.97377016e-01, 9.41422577e-01,
            9.87543941e-01, 1.03583894e+00, 1.08641001e+00, 1.13936442e+00,
            1.19481450e+00, 1.25287786e+00, 1.31367765e+00, 1.37734286e+00,
            1.44400852e+00, 1.51381603e+00, 1.58691346e+00, 1.66345587e+00,
            1.74360562e+00, 1.82753271e+00, 1.91541515e+00, 2.00743938e+00,
            2.10380056e+00, 2.20470312e+00, 2.31036106e+00, 2.42099850e+00,
            2.53685013e+00, 2.65816168e+00, 2.78519047e+00, 2.91820594e+00,
            3.05749023e+00, 3.20333879e+00, 3.35606098e+00, 3.51598075e+00]))

```python
plt.plot(s[0], s[1])
```

```python
s= pc.impulse_response(pc.ss(A,0.5,0.5,0))
s
```

    /Users/daquintero/opt/anaconda3/lib/python3.8/site-packages/numpy/matrixlib/defmatrix.py:147: ComplexWarning: Casting complex values to real discards the imaginary part
      arr = N.array(data, dtype=dtype, copy=copy)

    (array([0.        , 0.00685368, 0.01370736, 0.02056104, 0.02741472,
            0.0342684 , 0.04112207, 0.04797575, 0.05482943, 0.06168311,
            0.06853679, 0.07539047, 0.08224415, 0.08909783, 0.09595151,
            0.10280519, 0.10965886, 0.11651254, 0.12336622, 0.1302199 ,
            0.13707358, 0.14392726, 0.15078094, 0.15763462, 0.1644883 ,
            0.17134198, 0.17819566, 0.18504933, 0.19190301, 0.19875669,
            0.20561037, 0.21246405, 0.21931773, 0.22617141, 0.23302509,
            0.23987877, 0.24673245, 0.25358612, 0.2604398 , 0.26729348,
            0.27414716, 0.28100084, 0.28785452, 0.2947082 , 0.30156188,
            0.30841556, 0.31526924, 0.32212292, 0.32897659, 0.33583027,
            0.34268395, 0.34953763, 0.35639131, 0.36324499, 0.37009867,
            0.37695235, 0.38380603, 0.39065971, 0.39751339, 0.40436706,
            0.41122074, 0.41807442, 0.4249281 , 0.43178178, 0.43863546,
            0.44548914, 0.45234282, 0.4591965 , 0.46605018, 0.47290385,
            0.47975753, 0.48661121, 0.49346489, 0.50031857, 0.50717225,
            0.51402593, 0.52087961, 0.52773329, 0.53458697, 0.54144065,
            0.54829432, 0.555148  , 0.56200168, 0.56885536, 0.57570904,
            0.58256272, 0.5894164 , 0.59627008, 0.60312376, 0.60997744,
            0.61683111, 0.62368479, 0.63053847, 0.63739215, 0.64424583,
            0.65109951, 0.65795319, 0.66480687, 0.67166055, 0.67851423]),
     array([ 0.25      ,  0.26178214,  0.27411955,  0.28703841,  0.30056611,
             0.31473135,  0.32956418,  0.34509607,  0.36135994,  0.37839031,
             0.3962233 ,  0.41489673,  0.43445021,  0.45492521,  0.47636518,
             0.49881558,  0.52232403,  0.54694041,  0.57271691,  0.59970823,
             0.62797161,  0.657567  ,  0.68855718,  0.72100788,  0.75498793,
             0.79056942,  0.8278278 ,  0.86684213,  0.90769514,  0.95047349,
             0.99526793,  1.04217346,  1.09128958,  1.14272047,  1.19657523,
             1.25296808,  1.31201865,  1.37385218,  1.43859984,  1.50639897,
             1.57739336,  1.65173362,  1.72957743,  1.8110899 ,  1.89644394,
             1.98582059,  2.07940943,  2.17740897,  2.2800271 ,  2.38748147,
             2.5       ,  2.61782137,  2.74119549,  2.87038405,  3.00566109,
             3.14731353,  3.29564185,  3.45096066,  3.61359943,  3.78390312,
             3.96223298,  4.14896727,  4.34450207,  4.54925215,  4.76365179,
             4.98815579,  5.22324033,  5.46940406,  5.72716913,  5.9970823 ,
             6.27971608,  6.57566998,  6.88557176,  7.21007876,  7.5498793 ,
             7.90569415,  8.27827804,  8.66842126,  9.07695137,  9.50473491,
             9.95267926, 10.42173459, 10.91289581, 11.42720474, 11.96575231,
            12.52968084, 13.12018651, 13.73852185, 14.38599843, 15.06398965,
            15.77393361, 16.5173362 , 17.29577427, 18.110899  , 18.96443938,
            19.85820587, 20.79409428, 21.77408975, 22.80027098, 23.87481465]))

So now we've got a state-space representation in the form $\dot{\textbf{x}} = \textbf{Ax}$, where our state variable is $E_0$, nonlinear input $\phi$, $\textbf{B} = \textbf{D} = 0 $, and $\textbf{C} = 0 $

Issues with state space modelling with complex numbers https://dsp.stackexchange.com/questions/53554/how-to-model-state-space-for-complex-valued-system-correctly-in-simulink-matlab

```python
plt.plot(s[0], s[1])
```

Part of the big question of this solution is related to the nonlinear relationship of $\frac{d \phi(t)}{dt}$

We know phase change is $ \frac{d\phi}{dT} = \frac{2 \pi L}{\lambda_0} \frac{dn}{dT}$

```python
d_phi_d_T = 2 * sy.pi * L  * n.diff(T)/ lambda_0
phi_T = sy.integrate(d_phi_d_T, T)
phi_T.diff(t)
```

$\displaystyle \frac{2 \pi L \frac{\partial}{\partial t} n{\left(t,T \right)}}{\lambda_0}$

```python

```


```python
"""Header cell, contains modules and functions to make the whole notebook experience better"""
%matplotlib inline
# plots graphs within the notebook

from IPython.display import display,Image, Latex
from sympy.interactive import printing
printing.init_printing(use_latex='mathjax')
from IPython.display import clear_output

import time

from IPython.display import display,Image, Latex

from IPython.display import clear_output

import matplotlib.pyplot as plt
import numpy as np
import math
import scipy.constants as sc

import sympy as sym

font = {'family' : 'serif',
        #'color'  : 'black',
        'weight' : 'normal',
        'size'   : 12,
        }
fontlabel = {'family' : 'serif',
        #'color'  : 'black',
        'weight' : 'normal',
        'size'   : 12,
        }

from matplotlib.ticker import FormatStrFormatter
plt.rc('font', **font)

class PDF(object):
  def __init__(self, pdf, size=(200,200)):
    self.pdf = pdf
    self.size = size

  def _repr_html_(self):
    return ''.format(self.pdf, self.size)

  def _repr_latex_(self):
    return r'\includegraphics[width=1.0\textwidth]{{{0}}}'.format(self.pdf)

class ListTable(list):
    """ Overridden list class which takes a 2-dimensional list of
        the form [[1,2,3],[4,5,6]], and renders an HTML Table in
        IPython Notebook. """

    def _repr_html_(self):
        html = ["<table>"]
        for row in self:
            html.append("<tr>")

            for col in row:
                html.append("<td>{0}</td>".format(col))

            html.append("</tr>")
        html.append("</table>")
        return ''.join(html)

```

```python
import matplotlib.pyplot as plt
import numpy as np
import math
import scipy.constants as sc

import sympy as sym

from math import sin, cos

class Robot(object):
    """Defines basic mobile robot properties"""
    def __init__(self):
        self.pos_x  = 0.0
        self.pos_y  = 0.0
        self.angle  = 0.0
        self.plot   = False
        self._delta = 0.01
        self.step_plot = int(5)
        self.mag_plot = 1.

    # Movement
    def step(self):
        """ updates the x,y and angle """
        self.deltax()
        self.deltay()
        self.deltaa()

    def move(self, seconds):
        """ Moves the robot for an 's' amount of seconds"""
        for i in range(int(seconds/self._delta)):
            self.step()
            if i % self.step_plot == 0 and self.plot: # plot path every 3 steps
                self.plot_xya()

    # Printing-and-plotting:
    def print_xya(self):
        """ prints the x,y position and angle """
        print ("x = " + str(self.pos_x) +" "+ "y = " + str(self.pos_y))
        print ("a = " + str(self.angle))

    def plot_robot(self):
        """ plots a representation of the robot """
        plt.arrow(self.pos_x, self.pos_y, 0.001
                  * cos(self.angle), 0.001 * sin(self.angle),
                  head_width=self.mag_plot*self.length, head_length=self.mag_plot*self.length,
                  fc='k', ec='k')

    def plot_xya(self):
        """ plots a dot in the position of the robot """
        plt.scatter(self.pos_x, self.pos_y, c='r', edgecolors='r')

class DDRobot(Robot):
    """Defines a differential drive robot"""

    def __init__(self):
        Robot.__init__(self)
        self.radius = 0.1
        self.length = 0.4

        self.rt_spd_left = 0.0
        self.rt_spd_right = 0.0

    def deltax(self):
        """ update x depending on l and r angular speeds """
        self.pos_x += self._delta * (self.radius*0.5) \
        * (self.rt_spd_right + self.rt_spd_left)*cos(self.angle)

    def deltay(self):
        """ update y depending on l and r angular speeds """
        self.pos_y += self._delta * (self.radius*0.5) \
        * (self.rt_spd_right + self.rt_spd_left)*sin(self.angle)

    def deltaa(self):
        """ update z depending on l and r angular speeds """
        self.angle += self._delta * (self.radius/self.length) \
        * (self.rt_spd_right - self.rt_spd_left)
# function to convert degrees to radians
def D2R(a):
    return math.pi*a/180
def R2D(a):
    return 180*a/math.pi
# mybot = DDRobot()            # robot called 'enesbot'

# mybot.angle = 3.1416/4        # 45 degrees
# mybot.plot = True             # plot the robot!
# mybot.plot_robot()

# mybot.rt_spd_left = 10
# mybot.rt_spd_right = 10       # straight line
# mybot.move(2)                 # move for 2 seconds

# mybot.rt_spd_left = 12.5664
# mybot.rt_spd_right = 18.8496  # (2m diameter circle)
# mybot.move(1)                 # move for 1 second

# mybot.rt_spd_left = 18.8496
# mybot.rt_spd_right = 12.5664  # (2m diameter circle)
# mybot.move(2.5)               # move for 2.5 second

# mybot.rt_spd_left = 12.5664
# mybot.rt_spd_right = 18.8496  # (2m diameter circle)
# mybot.move(3.5)               # move for 2.5 second

# mybot.plot_robot()

# plt.xlim([-1, 6])               # axis limits
# plt.ylim([-1, 6])

# plt.show()
```

```python
PDF('bot-sketch.pdf',size = (550,400))
```

# Differential Drive Robot - 2: Calibration

Alphabot 2 is a differential drive robot: Two independent motors drive the left and right wheels and the differential speed between the two wheels control the speed and direction of the robot. The following code is copied from http://enesbot.me/kinematic-model-of-a-differential-drive-robot.html

## Objective
The objective is to investigate methods of calibration of the robot.

## Quick Theoretical Background

The robot parameters are:

 Wheel radius: $r$.

 Length between wheels: $L$

 Angular velocity of the left and right wheels: $\omega_L$, $\omega_R$, respectively.

 Angle from horizontal: $\alpha$.

 Position vector of the robot: $(x,y)$.

 Velocity vector of the robot: $V=(\dot{x},\dot{y})$.

Hereafter $\dot{a}$ refers to the time derivative of variable $a$:

$$
\dot{a}=\frac{da}{dt}
$$

The velocities of the wheels are therefore defined as:

$$
V_L=\omega_Lr\text{ and }V_R=\omega_Rr.
$$

The velocity of the robot, taken at the center of the wheels, is simply:

$$
\vec{V}=\frac{V_R+V_L}{2}(\cos(\alpha)\vec{e}_x+\sin(\alpha\vec{e}_y),
$$

yielding the following equations of motions:

$$
\dot{x}=r\frac{\omega_R+\omega_L}{2}\cos{\alpha}
$$

$$
\dot{y}=r\frac{\omega_R+\omega_L}{2}\sin{\alpha}
$$

and

$$
\dot{\alpha}=\frac{r}{L}(\omega_R-\omega_L)
$$

In the previous assignment, an equation was derived to determine the time $T$ needed to turn the robot an angle $\alpha_o$ at given rotation speed $\omega_L$ and $\omega_R$ and robot dimensions $r$ and $L$. This equation started by intergrating the robot angle equation above:

$$
\dot{\alpha}=\frac{d\alpha}{dt}=\frac{r}{L}(\omega_R-\omega_L)
$$

from $0$ to a time $T$ yields

$$
\int_0^T\frac{d\alpha}{dt}dt=\int_0^T\frac{r}{L}(\omega_R-\omega_L)dt
$$

Since the left hand side function is constant with respect to time, the result is simply:

$$
\alpha(T) - \alpha(0) =Tr\frac{\omega_R-\omega_L}{L}
$$

Therefore the time to execute a rotation of angle $\alpha_o=\alpha(T) - \alpha(0)$ is

$$
T = \frac{\alpha_oL}{r(\omega_R-\omega_L)}.\;\;\;\mathrm{Eq. 1}
$$

## Position of the Problem
In a perfect world if your code sets the same rotation speed for both wheels, the robot travels on a straight line. In the present case of independent motors, this outcome is very unlikely. Actually one should not expect the robot to drive straight. Below is a model of drift. Here the drift has two components: a constant drift (it could be a signal offset issue), and complex mechanical friction modeled by a quadratic function of the rotation speed. You will assume that you don't know the equation of the drift.

## Goal
You will derive a calibration curve using polynomial fit and test your calibration on a specific motion.

## Example of drift

Below is a code showing the trajectory of a robot for a series of 5 runs at different rotation speeds $\omega=1, 2.5, 5, 7.5, 10$. The robot starts from the same initial conditions, $x=0,y=0,\alpha=0$, has no drift (perfect robot) and moves for 10 s for all runs.

```python
T = 10
omega = np.array([1, 2.5, 5, 7.5, 10])
end_angle = np.zeros(5)
omega_R = omega
omega_L = omega
mybot = DDRobot()
mybot.mag_plot = 2 #coefficient of magnification of the arrow
mybot.step_plot = 100 # plot location every 100 iterations

for i in range(5):
    mybot.pos_x = 0 #x
    mybot.pos_y = 0 #y
    mybot.angle = D2R(0) #alpha
    mybot.length =0.4 #L
    mybot.radius = 0.1 #r
    mybot.rt_spd_left = omega_L[i]
    mybot.rt_spd_right = omega_R[i]

    mybot.plot = True #True if you want to plot the robot's trajectory
    mybot.plot_robot() #draw an arrow for the location of the robot at t=0

    mybot.move(T) #move the robot for 10s
    mybot.plot_robot() #draw the new location
    end_angle[i] = mybot.angle

plt.xlim(-5,11)
plt.ylim(-5,11)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.show()
```

Now we introduce a constant random drift $\delta$ and see its effect on the different trajectories. The function numpy.random.rand returns a random number between 0 and 1.

```python
delta = np.random.rand(1)
T = 10
omega = np.array([1, 2.5, 5, 7.5, 10])
end_angle = np.zeros(5)
omega_R = omega + delta[0]
omega_L = omega
mybot = DDRobot()
mybot.mag_plot = 2 #coefficient of magnification of the arrow
mybot.step_plot = 100 # plot location every 100 iterations

for i in range(5):
    mybot.pos_x = 0 #x
    mybot.pos_y = 0 #y
    mybot.angle = D2R(0) #alpha
    mybot.length =0.4 #L
    mybot.radius = 0.1 #r
    mybot.rt_spd_left = omega_L[i]
    mybot.rt_spd_right = omega_R[i]

    mybot.plot = True #True if you want to plot the robot's trajectory
    mybot.plot_robot() #draw an arrow for the location of the robot at t=0

    mybot.move(T) #move the robot for 10s
    mybot.plot_robot() #draw the new location
    end_angle[i] = mybot.angle

plt.xlim(-5,11)
plt.ylim(-5,11)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.show()
```

The plot below shows the angle at the end of each run. Based on Eq. 1 above, does this plot make sense?

```python
plt.plot(omega,R2D(end_angle),'o-',label="experiment")
plt.xlabel(r"$\omega$")
plt.ylabel(r"$\alpha\;(^\circ)$ ")
plt.legend(loc=3, bbox_to_anchor=[0, 1.], ncol=2, shadow=False, fancybox=True)
plt.show()
```

The correction is straightforward. For each run, the rotation speed of the right wheel must be corrected by $-\delta$.

```python

T = 10
omega = np.array([1, 2.5, 5, 7.5, 10])
end_angle = np.zeros(5)
mybot = DDRobot()
mybot.mag_plot = 2 #coefficient of magnification of the arrow
mybot.step_plot = 100 # plot location every 100 iterations

for i in range(5):
    mybot.pos_x = 0 #x
    mybot.pos_y = 0 #y
    mybot.angle = D2R(0) #alpha
    mybot.rt_spd_left = omega_L[i]
    mybot.rt_spd_right = omega_R[i] - delta[0]

    mybot.plot = True #True if you want to plot the robot's trajectory
    mybot.plot_robot() #draw an arrow for the location of the robot at t=0

    mybot.move(T) #move the robot for 10s
    mybot.plot_robot() #draw the new location
    end_angle[i] = mybot.angle

plt.xlim(-5,11)
plt.ylim(-5,11)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.show()
```

## Proposed calibration

You will assume that the drift $\delta$ is caused by the right motor and additive:

$$
\omega_L = \omega,\;\omega_R = \omega+\delta(\omega).
$$

If $\delta(\omega)$ is positive, the robot, when programmed to go straight, the robot will actually turn left.

The proposed calibration consists of programming the robot for a series of straight runs for various $\omega$ and constant $T$, $r$ and $L$. Once the robot stops, the deviation from the straight motion is measured by the angle of the robot at time $T$. This angle is a function of $\omega$, $\alpha(\omega)$ if the drift is a function of the rotation speed. Use Eq. 1 above to determine the drift $\delta(\omega)$ as a function of $\alpha(\omega)$, $L$, $T$ and $r$.

```python
""" random quadratic drift equation. Note that every time
you run this cell, the trajectories will change"""
a = np.random.random(2)# random drift amplitude
print(a)
def drift(omega):
    f = a[0] + a[1] * (omega/5)**2
    return f
```

### Uncalibrated robot

```python
T = 10
omega = np.array([1, 2.5, 5, 7.5, 10])
end_angle = np.zeros(5)
omega_R = omega + drift(omega)
omega_L = omega
mybot = DDRobot()
mybot.mag_plot = 2 #coefficient of magnification of the arrow
mybot.step_plot = 100 # plot location every 100 iterations

for i in range(5):
    mybot.pos_x = 0 #x
    mybot.pos_y = 0 #y
    mybot.angle = D2R(0) #alpha
    mybot.length =0.4 #L
    mybot.radius = 0.1 #r
    mybot.rt_spd_left = omega_L[i]
    mybot.rt_spd_right = omega_R[i]

    mybot.plot = True #True if you want to plot the robot's trajectory
    mybot.plot_robot() #draw an arrow for the location of the robot at t=0

    mybot.move(T) #move the robot for 10s
    mybot.plot_robot() #draw the new location
    end_angle[i] = mybot.angle

plt.xlim(-5,11)
plt.ylim(-5,11)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.show()
```

Note in the code above, the angle at the end of the motion is stored in the array end_angle. As shown below, the drift appears to follow a parabola.

```python
plt.plot(omega,R2D(end_angle),'o',label="experiment")
plt.xlabel(r"$\omega$")
plt.ylabel(r"$\alpha$")
plt.legend(loc=3, bbox_to_anchor=[0, 1.], ncol=2, shadow=False, fancybox=True)
plt.show()
```

Let's now fit a quadratic polynomial to these points. The creation of a polynomial in python is shown below using numpy polyfit function. p2_coef are the coefficients $p_i$ of the polynomial $p(x)=p_0x^2+p_1x+p_2$ (see https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.polyfit.html).

```python
p2_coef = np.polyfit(omega,end_angle,2)
print(p2_coef)
```

To generate a function, use the numpy function poly1d:

```python
p2 = np.poly1d(p2_coef)
p2 = np.poly1d(np.polyfit(omega,end_angle,2)) #equivalent compact form

```

You can now generate a fit over a range of your choice, here $0\leq\omega\leq10$

```python
N = 51
om = np.linspace(0,10,N)

plt.plot(omega,R2D(end_angle),'o',label="experiment")
plt.plot(om,R2D(p2(om)),label='fit')
plt.xlabel(r"$\omega$")
plt.ylabel(r"$\alpha$")
plt.legend(loc=3, bbox_to_anchor=[0, 1.], ncol=2, shadow=False, fancybox=True)
plt.show()
```

In Eq. 1, introduce the drift at $\delta$, i.e. $\omega_L=\omega$ and $\omega_R=\omega+\delta$, and derive a correction for the drift at as a function of $\alpha(\omega)$, which here is the array end_angle or the polynomial p2. Using the knowledge of the drift for all 5 rotation speed in the array omega, create a polynomial to compute the drift for $\omega\in[0,10]$ and introduce this polynomial into the code below to correct for the drift.

```python
# your code here
```

```python
omega = np.array([1, 2.5, 5, 7.5, 10])
end_angle = np.zeros(5)
omega_L = omega
omega_R = omega + drift(omega) # DO NOT MODIFY
mybot = DDRobot()
mybot.mag_plot = 2 #coefficient of magnification of the arrow
mybot.step_plot = 100 # plot location every 100 iterations

for i in range(5):
    mybot.pos_x = 0 #x
    mybot.pos_y = 0 #y
    mybot.angle = D2R(0) #alpha
    mybot.length =0.4 #L
    mybot.radius = 0.1 #r
    mybot.rt_spd_left = omega_L[i]
    mybot.rt_spd_right = omega_R[i] #write your correction here

    mybot.plot = True #True if you want to plot the robot's trajectory
    mybot.plot_robot() #draw an arrow for the location of the robot at t=0

    mybot.move(10) #move the robot for 10s
    mybot.plot_robot() #draw the new location
    end_angle[i] = mybot.angle

plt.xlim(-1,11)
plt.ylim(-1,11)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.show()
```

Now apply your correction to the following example. In this example the robot experiences 5 step motions of different durations and speed. The robot is supposed to move in a straight line. Print the angle of the robot at the end of each step. Does your correction work? What would it take the correction to work for positive AND negative rotation speed?

Write your answers here.

```python
omega = np.array([1.5, 2.4, 10, 3 , 7])
T = np.array([2, 3, 4, 2, 3])
end_angle = np.zeros(5)
omega_L = omega
omega_R = omega + drift(omega)
mybot = DDRobot()
mybot.mag_plot = 2 #coefficient of magnification of the arrow
mybot.step_plot = 100 # plot location every 100 iterations
mybot.pos_x = 0 #x
mybot.pos_y = 0 #y
mybot.angle = D2R(0) #alpha
mybot.length =0.4 #L
mybot.radius = 0.1 #r
for i in range(5):

    mybot.rt_spd_left = omega_L[i]
    mybot.rt_spd_right = omega_R[i] # write your correction here

    mybot.plot = True #True if you want to plot the robot's trajectory
    mybot.plot_robot() #draw an arrow for the location of the robot at t=0

    mybot.move(T[i]) #move the robot for 10s
    mybot.plot_robot() #draw the new location
    end_angle[i] = mybot.angle

plt.xlim(-5,11)
plt.ylim(-5,11)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.show()
```

```python
print(R2D(end_angle))
```

Create a complex motion (with rotations and speed variations). Show the result with and without correction. At minimum the motion should include one rotation and two different wheel rotation speed. See example below

```python
Nmoves = 2 # make sure to increase this number if you increase the number of moves
omega = np.array([6, 9])
T = np.array([12,8])
rotation_angle = np.array([90, 90])
rotation_angle = D2R(rotation_angle)
end_angle = np.zeros(Nmoves)
omega_L = omega
omega_R = omega + drift(omega)
t_rotation = rotation_angle*mybot.length/(mybot.radius*(omega_R+omega_L))
mybot = DDRobot()
mybot.mag_plot = 2 #coefficient of magnification of the arrow
mybot.step_plot = 100 # plot location every 100 iterations
mybot.pos_x = 0 #x
mybot.pos_y = 0 #y
mybot.angle = D2R(0) #alpha
mybot.length =0.4 #L
mybot.radius = 0.1 #r
mybot.plot = True #True if you want to plot the robot's trajectory
mybot.plot_robot() #draw an arrow for the location of the robot at t=0
for i in range(Nmoves):

    mybot.rt_spd_left = omega_L[i]
    mybot.rt_spd_right = omega_R[i] # write your correction here

    mybot.move(T[i]) #move the robot
    # rotation
    mybot.rt_spd_left = -omega_L[i]
    mybot.rt_spd_right = omega_R[i] # write your correction here
    mybot.move(t_rotation[i]) #move the robot
    mybot.plot_robot() #draw the new location
    end_angle[i] = mybot.angle

plt.xlim(-5,11)
plt.ylim(-5,11)
plt.xlabel(r"$x$")
plt.ylabel(r"$y$")
plt.show()
```

```python

```


```python
%matplotlib inline

import warnings
warnings.filterwarnings('ignore')
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import scipy.signal as scysig
import scipy.special as scyspe
import scipy.integrate as scyint
import pandas as pds
from IPython.display import Latex
```

```python
from IPython.display import HTML

HTML('''
The raw code for this IPython notebook is by default hidden for easier reading.
To toggle on/off the raw code, click <a href="javascript:code_toggle()">here</a>''')
#<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle the code on/off."></form>''')
```

The raw code for this IPython notebook is by default hidden for easier reading.
To toggle on/off the raw code, click here

# Estimative of Beam Divergence Increase due to Metallic foil

#### An idea of separating the vacuum of the storage ring and the transport line was suggested, in order to facilitate the design for the injection Septum. This can be achieved putting one thin metallic foil in the end of the the transport line and at the beggining of the storage ring vacuum chamber.  In this work we will try to estimate the effect of such windows on the beam parameters. We start by analysing the increase in the angular distribution of the electrons due to multiple scattering by the foils.

We will follow the theory described in the paper:

>Corstens, J.M. et al, An efficient method for calculating plural scattering of relativistic electrons in thin foils and multilayers, NIMB 222 (2004), doi:10.1016/j.nimb.200404.163

According to this reference, we can group the scattering in three types, according to the number of scattering events ($\lambda$) one electron experiences:

\begin{eqnarray}
&\lambda  &= 1     &\Rightarrow \text{single scattering}\\
1 < &\lambda &< 20 &\Rightarrow \text{plural scattering}\\
&\lambda &> 20    &\Rightarrow \text{multiple scattering}
\end{eqnarray}

The authors then describe a method for calculating the angular distribution of the scattered electron, assuming the entrance beam divergence is zero. The differential cross section used is the screened Rutherford cross section:

\begin{eqnarray}
\frac{\text{d}\sigma}{\text{d}\Omega} &=& \left(\frac{Ze^2}{2\pi\epsilon_0 p v}\right)^2 \frac{1}{(\chi_\alpha^2 + \theta^2)^2}\\
\sigma &=& \left(\frac{Ze^2}{2\pi\epsilon_0pv}\right)^2 \frac{\pi}{\chi_\alpha^2} \quad \text{where}
\quad \chi_\alpha = \sqrt{1.13 + 3.76\eta^2}\chi_0, \quad \chi_0 = \frac{\hbar}{0.885 a_0 Z^{-1/3}p}
\quad \text{and} \quad \eta = Z \alpha /\beta \\
\lambda &=& N\sigma t \quad \text{where} \quad N = \text{atomic density} \quad \text{and}\quad t = \text{thickness}
\end{eqnarray}

In our case, the window will be composed of three layers. The first will be a thin foil of Aluminium or Berilium followed by air and later another foil of Aluminium or Berilium. We will model this system by considering as a two layer window, composed of one layer of twice the thickness of each metallic foil followed by one layer of Nitrogen.

First we will obtain the angular distribution of each layer for a parallel incomming beam, then we will convolve both layers distributions and finally we will convolve the two-layers distribution with the beam to get the final angular distribution.

```python
# Constants
e      = 1.6e-19       # electron charge in C
m      = 9.11e-31      # electron mass in Kg
mp     = 1.67262178e-27# proton mass in Kg
hbar   = 1.05e-34      #
alpha  = 1/137         # fine structure constant
mu0    = 4*np.pi*1e-7
c      = 299792458
ep0    = 1/c**2/mu0
a0     = 5.29e-11      # Bohr radius in m
avog   = 6.02e23       # Avogrado number

# beam properties
gamma  = 3e9*e/m/c**2                # gamma = E/E0
beta   = np.math.sqrt(1-1/gamma**2)  # beta = v/c
p      = gamma*m*c*beta              # total momentum
sigx   = 30.5e-6  #31.7              # horizontal beam divergence
sigy   =  2.6e-6  #2.6               # vertical   beam divergence

# Properties of the foil
def foil_prop(key,thick=None):
    if key.startswith('Al'):
        t   = 2*10e-6             # thickness in m
        Z   = 13                  # atomic number
        rho = 2.7 * 1e-3/1e-2**3  # density in kg/m^3
        mf  = 26.9815 * 1e-3      # molar mass in kg/mol
    elif key.startswith('Be'):
        t   = 2*12e-6#100e-6#
        Z   = 4
        rho = 1.86 * 1e-3/1e-2**3
        mf  = 9.012 * 1e-3
    elif key.startswith('N'):
        t   = 5e-2
        Z   = 7 #*2
        rho = 1.251
        mf  = 28.0134 * 1e-3/2

    if thick: t = thick

    N   = rho/mf *avog            # atomic density

    # Interaction properties
    chi0    = hbar / (0.885*a0*Z**(-1/3)*p)
    eta    = Z* alpha /beta
    chi     = np.math.sqrt(1.13 + 3.76*eta**2)*chi0 # Screening angle
    dSdOm  = lambda th:(Z*e**2/(2*np.pi*ep0*p*c*beta))**2 /  ((chi)**2 + th**2)**2 # Differential Cross Section
    S      = ((Z*e**2/(2*np.pi*ep0*p*c*beta))**2 * np.pi/chi**2) # Total Cross Section
    lamb   = N * S * t
    return chi, lamb

# Definition of some important results
chi   = dict()
lamb  = dict()
norma = dict()  #||f||
media = dict()  #expected value
desvio= dict()  #standard deviation
distr = dict()  #distribution
angl  = dict()  #angle

```

## Single Layer Distributions

According to the paper, the angular distribution function in the small angle regime is given by:

\begin{equation}
\int f(\theta,\lambda)\text{d}\Omega \approx \int_0^\infty \!\!\!\!\! \theta f(\theta,\lambda)\text{d}\theta = 1 \quad
\text{where} \quad
f(\theta,\lambda) = \frac{e^{-\lambda}}{2\pi\chi^2_\alpha}\left(\frac{\delta(\xi)}{\xi} +
\sum_{n=1}^\infty \frac{\lambda^n}{n!}g_n(\xi)\right), \quad \text{with} \quad \xi = \theta/\chi_\alpha, \quad
g_n(\chi) = \int_0^\infty \!\!\!\!\! \text{d}u \, u(uK_1(u))^n J_0(u\chi)
\end{equation}

where $g_n(\xi)$ are universal functions because they depend only on the normalized angle. The interpretation of $g_n(\xi)/(2\pi\chi_\alpha^2)$ is the angular distribution after exactly $n$ scattering events and the term $\lambda^n/n!e^{-\lambda}$ is the weight of each partial distribution to the total distribution. The term $\delta(\xi)/\xi$ gives the part of the beam which was not scattered. Bellow are some of these functions:

```python
# Calculation of the g_n functions
def calc_gn(trun):
    norT  = np.linspace(0,120,300)
    fun   = np.zeros((len(norT),trun))
    errn  = np.zeros((len(norT),trun))
    func4 = lambda x, ni: lambda u :u*(u*scyspe.k1(u))**ni * scyspe.j0(u*x)
    fun[:,0] = 2/(1+norT**2)**2                       # given by the paper
    fun[:,1] = scyspe.hyp2f1(2,3,5/2,-norT**2/4)/3*2  # given by reference [7] of the paper
    for j in range(2,trun):
        for i in range(len(norT)):
            fun[i,j],errn[i,j] = scyint.quad(func4(norT[i],j+1),0,np.inf) or 0.0
    return norT, fun, errn

def get_weight(la,i):
    # Strinivasa Ramanujan approximation for log(n!):
    def ramanujan_log_fac(n):
        return n*(np.math.log(n)-1) + np.math.log(n*(1+4*n*(1+2*n)))/6 + np.math.log(np.math.pi)/2
    if i < 120:
        weight = np.math.exp(-la)*la**(i)/np.math.factorial(i)
    else:
        weight = np.math.exp(-la + i*np.math.log(la) - ramanujan_log_fac(i))
    return weight

def distribution(th,la,ch):
    fun = np.zeros(th.shape)
    for i in range(gn.shape[1]):
        fun += get_weight(la,i+1) * np.interp(th,normAngl,gn[:,i], left=0.0, right=0.0)
    return fun /(2*np.pi*ch**2)
```

```python
##### It takes approximately 30s to execute this cell. You just need to run it once ########
n     = 100
normAngl, gn, erro = calc_gn(n)

```

```python
plt.figure(figsize=(18.0,4.0))
plt.subplot(131)
plt.semilogy(normAngl,gn)
plt.grid(axis='both')
plt.xlabel(r'$\xi$',size=16)
plt.ylabel(r'$g_n$',size=16)
plt.subplot(132)
plt.semilogy(normAngl,erro)
plt.xlabel(r'$\xi$',size=16)
plt.ylabel(r'Error in calculation of $g_n$',size=16)
plt.subplot(133)
weight = np.zeros(n)
keys = ('Be','N','Al')
for key in keys:
    chi[key], lamb[key] = foil_prop(key)
    for i in range(n):
        weight[i] = get_weight(lamb[key],i+1)
    plt.semilogy(np.arange(1,n+1),weight,label=key)
plt.legend(loc='best')
plt.xlabel(r'$n$',size=16)
plt.ylabel(r'weight',size=16)
plt.show()
```

With that in mind, knowing the value of our $\lambda$ we can calculated the angular distribution ($f_n(\theta,\lambda)$) truncated in $n$:

```python
plt.figure(figsize=(10.0,4.0))

keys = ('Be','N','Al')
for key in keys:
    angl[key]  = normAngl * chi[key]
    distr[key] = distribution(normAngl, lamb[key], chi[key])
    plt.plot(angl[key]*1e6,distr[key]*1e-12,label=key)

plt.legend(loc='best')
plt.tick_params(axis='both', which='major', labelsize=16)
plt.xlabel(r'$\theta$ [urad]',size=20)
plt.ylabel(r'$f_{{{0:d}}}(\theta) [\mu $rad$^{{-2}}]$'.format(n),size=20)
plt.grid(axis='both')
plt.show()
```

In order to check the the convergence of the truncated distribution, we verify if its normalization: $$
||f|| = \int f(\theta,\lambda) \text{d}\Omega \approx
2\pi\int_0^\infty \theta f(\theta,\lambda) \text{d}\theta
$$ is close enough to unity:

```python
label = r"""\begin{eqnarray}"""

keys = ('Be','N','Al')
for key in keys:
    norma[key]  = 2*np.pi*np.trapz(distr[key]*angl[key],x=angl[key])
    label += r"""||f||_{{{0:2s}}} &=& {{{1:7.5f}}}\\ """.format(key,norma[key])

Latex(label[0:-3]+r"""\end{eqnarray}""")
```

\begin{eqnarray}||f||_{Be} &=& {0.99796}\\ ||f||_{N } &=& {0.99647}\\ ||f||_{Al} &=& {0.99646}\end{eqnarray}

it is possible to calculate the first and second moments of the distribution:
$$\langle\theta\rangle = \int_0^\infty \theta f(\theta,\lambda) \text{d}\Omega,\quad
\langle\theta^2\rangle = \int_0^\infty \theta^2 f(\theta,\lambda) \text{d}\Omega$$

```python
label = r"""\begin{eqnarray}"""
for key in keys:
    media[key]  = 2*np.pi*np.trapz(angl[key]**2*distr[key],x=angl[key])
    desvio[key] = 2*np.pi*np.trapz(angl[key]**3*distr[key],x=angl[key])
    label += r"""\langle\theta\rangle_{{{0:2s}}} &=& {{{1:7.5f}}}\, \mu\text{{rad}}, \quad \langle\theta^2\rangle_{{{0:2s}}} &=& {2:7.5f}\, \mu\text{{rad}}^2\\ """.format(key,media[key]*1e6, np.sqrt(desvio[key]*1e12/2))
Latex(label[0:-3]+r"""\end{eqnarray}""")
```

\begin{eqnarray}\langle\theta\rangle_{Be} &=& {25.90659}\, \mu\text{rad}, \quad \langle\theta^2\rangle_{Be} &=& 24.21808\, \mu\text{rad}^2\\ \langle\theta\rangle_{N } &=& {44.89665}\, \mu\text{rad}, \quad \langle\theta^2\rangle_{N } &=& 40.10869\, \mu\text{rad}^2\\ \langle\theta\rangle_{Al} &=& {55.82446}\, \mu\text{rad}, \quad \langle\theta^2\rangle_{Al} &=& 49.86500\, \mu\text{rad}^2\end{eqnarray}

## Double Layer Scattering

According to the paper, the angular distribution of a beam with initial angular distribution $h_0(\theta)$ after passing througth the window is given by the convolution:

\begin{equation}
h_1(\theta,\lambda) = \int_0^\infty \text{d}\theta'\, \theta'h_0(\theta')\int_0^{2\pi} \text{d}\phi \, f(\theta'',\lambda)
\quad \text{with} \quad \theta'' = \sqrt{\theta'^2 + \theta^2 - 2\theta'\theta\cos(\phi)}
\end{equation}
where $\theta'$ is the incoming azimuthal angle, $\theta''$ is the scattered azimuthal angle, $\phi$ is the difference between the initial and scattered polar angles and $\theta$ is the final angle.
Analogously, to obtain the double layer scattering distribution for a parallel beam, we must convolve:

\begin{equation}
f(\theta) = \int_0^\infty \text{d}\theta'\, \theta'f_1(\theta',\lambda_1)\int_0^{2\pi} \text{d}\phi \, f_2(\theta'',\lambda_2).
\end{equation}

Lets define this convolution:

```python
# Convolution in theta and phi representation
def convolve(x,func1,func2):
    #integral in angle phi
    def fun2(x2,xl2, func):
        phi  = np.linspace(0,2*np.pi,200)
        ret2 = np.zeros(xl2.shape)
        for i in range(len(xl2)):
            ret2[i] = np.trapz(func(np.sqrt(xl2[i]**2 + x2**2 - 2*x2*xl2[i]*np.cos(phi))),x=phi)
        return ret2

    #Integral in theta'
    xl  = x.copy()
    ret = np.zeros(x.shape)
    for i in range(len(x)):
        ret[i] = np.trapz(xl*func1(xl)*fun2(x[i],xl,func2),x=xl)
    return ret
```

Now we can calculate the angular distribution of the convolution:

```python
#### this cell takes 16s to run #####
# Aluminum and air
distN  = lambda x : np.interp(x,angl['N'],distr['N'],left=0.0,right=0.0)
distAl = lambda x : np.interp(x,angl['Al'],distr['Al'],left=0.0,right=0.0)
angl['Al-N']    = angl['N']
distr['Al-N']   = convolve(angl['Al-N'],distAl,distN)

# Berilium and air
distBe = lambda x : np.interp(x,angl['Be'],distr['Be'],left=0.0,right=0.0)
angl['Be-N']    = angl['N']
distr['Be-N']   = convolve(angl['Be-N'],distBe,distN)
```

```python
plt.figure(figsize=(10.0,4.0))
for key in distr.keys():
    plt.semilogy(angl[key]*1e6,distr[key]*1e-12,label=key)
plt.legend(loc='best')
plt.grid(axis='both')
plt.tick_params(labelsize=16)
plt.xlabel(r'$\theta$ [$\mu$rad]',fontsize=20)
plt.ylabel(r'$f(\theta) [\mu $rad$^{-2}]$',fontsize=20)
plt.show()
```

And look at its normalization and first moments:

```python
label = r"""\begin{eqnarray}"""

norma['Al-N'] = 2*np.pi*np.trapz(distr['Al-N']*angl['Al-N'],x=angl['Al-N'])
label += r"""||f||_{{{0:3s}}} &=& {{{1:7.5f}}}\\ """.format('Al-N',norma['Al-N'])

norma['Be-N'] = 2*np.pi*np.trapz(distr['Be-N']*angl['Be-N'],x=angl['Be-N'])
label += r"""||f||_{{{0:3s}}} &=& {{{1:7.5f}}}\\ """.format('Be-N',norma['Be-N'])

Latex(label[0:-3]+r"""\end{eqnarray}""")
```

\begin{eqnarray}||f||_{Al-N} &=& {0.99041}\\ ||f||_{Be-N} &=& {0.99530}\end{eqnarray}

```python
label = r"""\begin{eqnarray}"""

media['Al-N']  = 2*np.pi*np.trapz(angl['Al-N']**2*distr['Al-N'],x=angl['Al-N'])
desvio['Al-N'] = 2*np.pi*np.trapz(angl['Al-N']**3*distr['Al-N'],x=angl['Al-N'])
label += r"""\langle\theta\rangle_{{{0:3s}}} &=& {{{1:7.1f}}}\, \mu\text{{rad}}, \quad \langle\theta^2\rangle_{{{0:3s}}} &=& {2:7.1f}\, \mu\text{{rad}}^2\\ """.format('Al-N',media['Al-N']*1e6, desvio['Al-N']*1e12)

media['Be-N']  = 2*np.pi*np.trapz(angl['Be-N']**2*distr['Be-N'],x=angl['Be-N'])
desvio['Be-N'] = 2*np.pi*np.trapz(angl['Be-N']**3*distr['Be-N'],x=angl['Be-N'])
label += r"""\langle\theta\rangle_{{{0:3s}}} &=& {{{1:7.1f}}}\, \mu\text{{rad}}, \quad \langle\theta^2\rangle_{{{0:3s}}} &=& {2:7.1f}\, \mu\text{{rad}}^2\\ """.format('Be-N',media['Be-N']*1e6, desvio['Be-N']*1e12)

Latex(label[0:-3]+r"""\end{eqnarray}""")
```

\begin{eqnarray}\langle\theta\rangle_{Al-N} &=& {   72.9}\, \mu\text{rad}, \quad \langle\theta^2\rangle_{Al-N} &=&  7726.3\, \mu\text{rad}^2\\ \langle\theta\rangle_{Be-N} &=& {   53.7}\, \mu\text{rad}, \quad \langle\theta^2\rangle_{Be-N} &=&  4424.3\, \mu\text{rad}^2\end{eqnarray}

Notice that:

\begin{eqnarray}
\langle\theta\rangle_{AlN}^2 &\approx \langle\theta\rangle_{Al}^2 + \langle\theta\rangle_{N}^2 \quad\quad \text{and}\quad\quad
\langle\theta^2\rangle_{AlN} &\approx \langle\theta^2\rangle_{Al} + \langle\theta^2\rangle_{N} \\
\langle\theta\rangle_{BeN}^2 &\approx \langle\theta\rangle_{Be}^2 + \langle\theta\rangle_{N}^2 \quad\quad\text{and}\quad\quad
\langle\theta^2\rangle_{BeN} &\approx \langle\theta^2\rangle_{Be} + \langle\theta^2\rangle_{N}
\end{eqnarray}

## Convolution with the Beam:

We know the electron beam posses a gaussian shape in both transverse phase spaces:

\begin{equation}
h(\theta_x,\theta_y) = \frac{1}{2\pi\sigma_{\theta_x}\sigma_{\theta_y}}\exp\left(-\frac{\theta_x^2}{2\sigma_{\theta_x}^2} -\frac{\theta_y^2}{2\sigma_{\theta_y}^2}\right)
\end{equation}

The first thing we need to do is to find a relationship between $(\theta, \phi)$ representation and the $(\theta_x,\theta_y)$ representation.

Looking at the figure below:

we can see that:

\begin{equation}
\left.
\begin{aligned}
\tan(\theta_x) &= \frac{\overline{OX}}{\overline{OZ}} =
\frac{\overline{OC}\cos(\phi)}{\overline{OP}\cos(\theta)} =
\frac{\overline{OP}\sin(\theta)\cos(\phi)}{\overline{OP}\cos(\theta)} = \tan(\theta)\cos(\phi) \\
\tan(\theta_y) &= \frac{\overline{OY}}{\overline{OZ}} =
\frac{\overline{OC}\sin(\phi)}{\overline{OP}\cos(\theta)} =
\frac{\overline{OP}\sin(\theta)\sin(\phi)}{\overline{OP}\cos(\theta)} = \tan(\theta)\sin(\phi)
\end{aligned}
\right\} \Rightarrow
\begin{aligned}
\theta_x &\approx \theta\cos(\phi) \\
\theta_y &\approx \theta\sin(\phi)
\end{aligned}
\end{equation}

This way, we can easily express the distribution in polar coordinates:

\begin{equation}
||h|| = \int_{-\infty}^\infty \!\!\!\!\!\text{d}\theta_y \int_{-\infty}^\infty \!\!\!\!\!\text{d}\theta_x h(\theta_x,\theta_y) =
\int_0^{2\pi} \!\!\!\!\!\text{d}\phi \int_0^\infty \!\!\!\!\!\text{d}\theta\,\, \theta h(\theta,\phi) = 1
\end{equation}

### Round Beam:

Assuming $\sigma_{\theta_x} = \sigma_{\theta_y}$, the beam distribution will not depend on $\phi$, and we can define:

\begin{equation}
h(\theta) = \frac{1}{2\pi\sigma^2}\exp\left(-\frac{\theta^2}{2\sigma^2}\right) \Rightarrow ||h|| = 2\pi\int_0^\infty \!\!\!\!\!\text{d}\theta \,\theta h(\theta) = 1
\end{equation}

The first and second moments of this distribution are given by:

```python
theta = sigx*np.linspace(0,6,1000)
h0 = lambda x,sigma: 1/(2*np.pi*sigma**2)*np.exp(-x**2/(2*sigma**2))

label = r"""\begin{equation}"""
norm  = 2*np.pi*np.trapz(theta   *h0(theta,sigx),x=theta)
medi  = 2*np.pi*np.trapz(theta**2*h0(theta,sigx),x=theta)/sigx/np.sqrt(np.pi/2)
desvi = 2*np.pi*np.trapz(theta**3*h0(theta,sigx),x=theta)/(2*sigx**2)
label += r"""||h|| = {0:7.5f},\quad \frac{{\langle\theta\rangle}}{{\sqrt\frac\pi2\sigma}} = {1:7.5f}\,, \quad \frac{{\langle\theta^2\rangle}}{{2\sigma^2}} = {2:7.5f}""".format(norm,medi, desvi)
Latex(label+r"""\end{equation}""")
```

\begin{equation}||h|| = 1.00000,\quad \frac{\langle\theta\rangle}{\sqrt\frac\pi2\sigma} = 1.00000\,, \quad \frac{\langle\theta^2\rangle}{2\sigma^2} = 1.00000\end{equation}

Doing the convolution of the initial beam distribution with the previous results we obtain:

```python
#### this cell takes 32s to run #####
# convolution with Al-N window
dist   = lambda x : np.interp(x,angl['Al-N'],distr['Al-N'],left=0.0,right=0.0)
h      = lambda x : h0(x,sigx)
angl['RBx-Al-N']   = angl['Al-N']
distr['RBx-Al-N']  = convolve(angl['RBx-Al-N'],h,dist)
h      = lambda x : h0(x,sigy)
angl['RBy-Al-N']   = angl['Al-N']
distr['RBy-Al-N']  = convolve(angl['RBy-Al-N'],h,dist)

# convolution with Be-N window
dist   = lambda x : np.interp(x,angl['Be-N'],distr['Be-N'],left=0.0,right=0.0)
h      = lambda x : h0(x,sigx)
angl['RBx-Be-N']   = angl['Be-N']
distr['RBx-Be-N']  = convolve(angl['RBx-Be-N'],h,dist)
h      = lambda x : h0(x,sigy)
angl['RBy-Be-N']   = angl['Be-N']
distr['RBy-Be-N']  = convolve(angl['RBy-Be-N'],h,dist)
```

```python
plt.figure(figsize=(10.0,4.0))
for key in ('Al-N','RBx-Al-N','RBy-Al-N','Be-N','RBx-Be-N','RBy-Be-N'):
    plt.semilogy(angl[key]*1e6,distr[key]*1e-12,label=key)
plt.xlim(0,angl[key][-1]*1e6)
plt.tick_params(labelsize=16)
plt.xlabel(r'$\theta \,\,[\mu$rad$]$',fontsize=20)
plt.ylabel(r'$h(\theta) \,\,[\mu$rad$^{-2}]$',fontsize=20)
plt.legend(loc='best')
plt.grid(axis='both')
plt.show()
```

```python
label = r"""\begin{eqnarray}"""
for key in ('RBx-Al-N','RBy-Al-N','RBx-Be-N','RBy-Be-N'):
    norma[key] = 2*np.pi*np.trapz(distr[key]*angl[key],x=angl[key])
    label += r"""||h||_{{{0:3s}}} &=& {{{1:7.5f}}}\\ """.format(key,norma[key])

Latex(label[0:-3]+r"""\end{eqnarray}""")
```

\begin{eqnarray}||h||_{RBx-Al-N} &=& {0.99005}\\ ||h||_{RBy-Al-N} &=& {0.97419}\\ ||h||_{RBx-Be-N} &=& {0.99518}\\ ||h||_{RBy-Be-N} &=& {0.97904}\end{eqnarray}

```python
label    = r"""\begin{eqnarray}"""
for key in ('RBx-Al-N','RBy-Al-N','RBx-Be-N','RBy-Be-N'):
    media[key] = 2*np.pi*np.trapz(angl[key]**2*distr[key],x=angl[key])
    desvio[key]= 2*np.pi*np.trapz(angl[key]**3*distr[key],x=angl[key])
    label   += r"""\langle\theta\rangle_{{{0:3s}}} &=& {{{1:7.5f}}}\, \mu\text{{rad}}, \quad \langle\theta^2\rangle_{{{0:3s}}} &=& {2:7.5f}\, \mu\text{{rad}}^2\\ """.format(key,media[key]*1e6, desvio[key]*1e12)

Latex(label[0:-3]+r"""\end{eqnarray}""")
```

\begin{eqnarray}\langle\theta\rangle_{RBx-Al-N} &=& {82.74356}\, \mu\text{rad}, \quad \langle\theta^2\rangle_{RBx-Al-N} &=& 9513.63157\, \mu\text{rad}^2\\ \langle\theta\rangle_{RBy-Al-N} &=& {71.82092}\, \mu\text{rad}, \quad \langle\theta^2\rangle_{RBy-Al-N} &=& 7613.06279\, \mu\text{rad}^2\\ \langle\theta\rangle_{RBx-Be-N} &=& {66.75271}\, \mu\text{rad}, \quad \langle\theta^2\rangle_{RBx-Be-N} &=& 6251.56082\, \mu\text{rad}^2\\ \langle\theta\rangle_{RBy-Be-N} &=& {52.95808}\, \mu\text{rad}, \quad \langle\theta^2\rangle_{RBy-Be-N} &=& 4365.55118\, \mu\text{rad}^2\end{eqnarray}

Notice that:

\begin{eqnarray}
\langle\theta\rangle_{RB-Al-N}^2 &\approx& \langle\theta\rangle_{Al-N}^2 + \langle\theta\rangle_{RB}^2
\quad\quad\text{and}\quad\quad
\langle\theta^2\rangle_{RB-Al-N} &\approx& \langle\theta^2\rangle_{Al-N} + \langle\theta^2\rangle_{RB} \\
\langle\theta\rangle_{RB-Be-N}^2 &\approx& \langle\theta\rangle_{Be-N}^2 + \langle\theta\rangle_{RB}^2
\quad\quad\text{and}\quad\quad
\langle\theta^2\rangle_{RB-Be-N} &\approx& \langle\theta^2\rangle_{Be-N} + \langle\theta^2\rangle_{RB}
\end{eqnarray}

The figures below show a comparison between the total convoluted distribution with a Gaussian Distribution with $\sigma$ given by the rule:

$$ \sigma = \frac{\langle\theta\rangle}{\sqrt\frac\pi2}$$

```python
fig,ax = plt.subplots(1,2, sharey=True,figsize=(16.0,4.0))
ax[0].tick_params(labelsize=16)
sig = media['RBx-Al-N']/np.sqrt(np.pi/2)
ax[0].semilogy(angl['RBx-Al-N']/sig,h0(angl['RBx-Al-N'],sig)*1e-12,label='Gaussian')
ax[0].semilogy(angl['RBx-Al-N']/sig,distr['RBx-Al-N']*1e-12,label=r'$h_{RBx-Al-N}$')
ax[0].set_xlabel(r'$\theta / \sigma$',fontsize=20)
ax[0].set_ylabel(r'$h(\theta) [\mu$rad$^{-2}]$',fontsize=20)
ax[0].legend(loc='best',fontsize=16)
ax[0].grid(axis='both')
ax[0].tick_params(labelsize=16)
sig = media['RBx-Be-N']/np.sqrt(np.pi/2)
ax[1].semilogy(angl['RBx-Be-N']/sig,h0(angl['RBx-Be-N'],sig)*1e-12,label='Gaussian')
ax[1].semilogy(angl['RBx-Be-N']/sig,distr['RBx-Be-N']*1e-12,label=r'$h_{RBx-Be-N}$')
ax[1].set_xlabel(r'$\theta / \sigma$',fontsize=20)
ax[1].legend(loc='best',fontsize=16)
ax[1].grid(axis='both')
ax[1].tick_params(labelsize=16)
fig.show()
```

### Elliptical Beam:

In the case the beam is not round, it is easy to do the convolution in the $(\theta_x,\theta_y)$ representation. In this coordinate system, each initial point $(\theta'_x,\theta'_y)$  can be taken to another point $(\theta_x,\theta_y)$ with a probability $f\left(\sqrt{(\theta_x-\theta'_x)^2 + (\theta_y-\theta'_y)^2}\right)$. This way, the final distribution will be the sum of the contribution of all points $(\theta'_x,\theta'_y)$ to each point $(\theta_x,\theta_y)$:

\begin{equation}
h(\theta_x,\theta_y) =
\int_{-\infty}^\infty \!\!\!\!\! \text{d}\theta'_x
\int_{-\infty}^\infty \!\!\!\!\! \text{d}\theta'_y \,\,
h_0(\theta'_x,\theta'_y)f\left(\sqrt{(\theta_x-\theta'_x)^2 + (\theta_y-\theta'_y)^2}\right)
\end{equation}

```python
def h0xy(x,sigmax,y,sigmay):
    x = np.array(x).ravel()[None,:]
    y = np.array(y).ravel()[:,None]
    return 1/(2*np.pi*sigmax*sigmay)*np.exp(-x**2/(2*sigmax**2)-y**2/(2*sigmay**2))

 # convert distribution in theta to thetax and thetay
def ThetaPhi2XY(x,y, dit):
    x = np.array(x).ravel()[None,:]
    y = np.array(y).ravel()[:,None]
    return np.interp(np.sqrt(x**2+y**2),angl['N'],dit,left=0.0,right=0.0)
```

```python
xx      = np.linspace(-1,1,601)*angl['N'][-1]
yy      = np.linspace(-1,1,601)*angl['N'][-1]
X,Y     = np.meshgrid(xx,yy)
angl['EB']       = (xx,yy)
distr['EB']      = h0xy(xx,sigx,yy,sigy)
diAlN            = ThetaPhi2XY(xx,yy,distr['Al-N'])
angl['EB-Al-N']  = (xx,yy)
distr['EB-Al-N'] = scysig.fftconvolve(distr['EB'],diAlN,mode='same')*(xx[1]-xx[0])*(yy[1]-yy[0])
diBeN            = ThetaPhi2XY(xx,yy,distr['Be-N'])
angl['EB-Al-N']  = (xx,yy)
distr['EB-Be-N'] = scysig.fftconvolve(distr['EB'],diBeN,mode='same')*(xx[1]-xx[0])*(yy[1]-yy[0])
```

```python
fig, ax = plt.subplots(2,3,sharey=True,sharex=True)
fig.set_size_inches(16,8)
ax[0,0].set_xlim([xx.min()*1e6,xx.max()*1e6])
ax[0,0].set_ylim([yy.min()*1e6,yy.max()*1e6])
ax[0,0].set_ylabel(r'$\theta_y [\mu$rad$]$',fontsize=20)
ax[1,0].set_ylabel(r'$\theta_y [\mu$rad$]$',fontsize=20)
Z     = ((distr['EB'],distr['EB']),(diAlN,diBeN),(distr['EB-Al-N'],distr['EB-Be-N']))
title = (('Beam Before',''),('Al-N Window','Be-N Window'),('Beam After',''))
for j in range(3):
    ax[1,j].set_xlabel(r'$\theta_x [\mu$rad$]$',fontsize=20)
    for i in range(2):
        ax[i,j].set_title(title[j][i],fontsize=20)
        ax[i,j].pcolormesh(X*1e6,Y*1e6,Z[j][i]*1e-12)
        ax[i,j].tick_params(labelsize=16)
plt.show()
```

```python
label = r"""\begin{eqnarray}"""

norma['EB-Al-N'] = np.trapz(np.trapz(distr['EB-Al-N'],x=yy,axis=0),x=xx)
label += r"""||h||_{{{0:3s}}} &=& {{{1:7.5f}}}\\ """.format('EB-Be-N',norma['EB-Al-N'])

norma['EB-Be-N'] = np.trapz(np.trapz(distr['EB-Be-N'],x=xx,axis=1),x=yy)
label += r"""||h||_{{{0:3s}}} &=& {{{1:7.5f}}}\\ """.format('EB-Be-N',norma['EB-Be-N'])

Latex(label[0:-3]+r"""\end{eqnarray}""")
```

\begin{eqnarray}||h||_{EB-Be-N} &=& {0.99249}\\ ||h||_{EB-Be-N} &=& {0.99665}\end{eqnarray}

```python
label    = r"""\begin{eqnarray}"""
mex = np.trapz(np.trapz(X*distr['EB-Al-N'],x=yy,axis=0),x=xx)
mey = np.trapz(np.trapz(Y*distr['EB-Al-N'],x=xx,axis=1),x=yy)
media['EB-Al-N']  = (mex,mey)
dex = np.sqrt(np.trapz(np.trapz(X**2*distr['EB-Al-N'],x=yy,axis=0),x=xx))
dey = np.sqrt(np.trapz(np.trapz(Y**2*distr['EB-Al-N'],x=xx,axis=1),x=yy))
desvio['EB-Al-N'] = (dex**2,dey**2)
label   += r"""\sqrt{{\langle\theta_x^2\rangle_{{{0:3s}}}}} &=& {{{1:7.5f}}}\, \mu\text{{rad}}, \quad \sqrt{{\langle\theta_y^2\rangle_{{{0:3s}}}}} &=& {2:7.5f}\, \mu\text{{rad}}^2\\ """.format('EB-Al-N',dex*1e6, dey*1e6)
mex = np.trapz(np.trapz(X*distr['EB-Be-N'],x=yy,axis=0),x=xx)
mey = np.trapz(np.trapz(Y*distr['EB-Be-N'],x=xx,axis=1),x=yy)
media['EB-Be-N']  = (mex,mey)
dex = np.sqrt(np.trapz(np.trapz(X**2*distr['EB-Be-N'],x=yy,axis=0),x=xx))
dey = np.sqrt(np.trapz(np.trapz(Y**2*distr['EB-Be-N'],x=xx,axis=1),x=yy))
desvio['EB-Be-N'] = (dex**2,dey**2)
label   += r"""\sqrt{{\langle\theta_x^2\rangle_{{{0:3s}}}}} &=& {{{1:7.5f}}}\, \mu\text{{rad}}, \quad \sqrt{{\langle\theta_y^2\rangle_{{{0:3s}}}}} &=& {2:7.5f}\, \mu\text{{rad}}^2\\ """.format('EB-Be-N',dex*1e6, dey*1e6)

Latex(label[0:-3]+r"""\end{eqnarray}""")
```

\begin{eqnarray}\sqrt{\langle\theta_x^2\rangle_{EB-Al-N}} &=& {70.08941}\, \mu\text{rad}, \quad \sqrt{\langle\theta_y^2\rangle_{EB-Al-N}} &=& 63.53501\, \mu\text{rad}^2\\ \sqrt{\langle\theta_x^2\rangle_{EB-Be-N}} &=& {56.70426}\, \mu\text{rad}, \quad \sqrt{\langle\theta_y^2\rangle_{EB-Be-N}} &=& 48.14371\, \mu\text{rad}^2\end{eqnarray}

## Numerical Simulations

We have simulated the scattering in Fluka. We hit the window with a non-divergent beam and look at the beam size at a distance $L$ from the source, as shown in the figure bellow:

To get the beam divergence we divide the scales of the detector by the length $L$. To gain precision, we have set the distance $L$ to $3$m and the initial beam is a 2D gaussian with full width half maximum equal to $1\mu\text{m}\times1\mu\text{m}$. We simulated transported $640000$ particles.

The simulations were performed in a Ubuntu virtual machine with the aid of Flair to prepare the inputs for Fluka and process the outputs. The result Flair gives us is a *.txt* file with four coluns, whose meanings are:

$x$ [cm] | $y$ [cm] | $f_n$ [$\frac{1}{\text{cm}^2}$] | Err$(f_n)$ [%]
-|
| | |

where $(x,y)$ represents the transverse grid ($x$ varies slower in the data), $f_n$ is the frequency distribution of the electrons in each point of the grid and Err$(f_n)$ is the error in the calculation of the frequency distribution function.

### Berilium - Air:

```python
#dados = np.loadtxt('FlukaSimulation/Ric_electron_scattering_79_plot.dat')
dados = np.loadtxt('FlukaSimulation/Be12umAir5cm.dat')
#dados = np.loadtxt('FlukaSimulation/Be12um.dat')
L = 299 # cm
nly = np.unique(dados[:,0]).shape[0]
nlx = len(dados)//nly
X = dados[:,0].reshape([nly,nlx]).T/L#*1e-2#
Y = dados[:,1].reshape([nly,nlx]).T/L
angl['Simul-Be-Air']  = (X[0,:].ravel(),Y[:,0].ravel())
distr['Simul-Be-Air'] = dados[:,2].reshape([nly,nlx])*L**2#*1e4#
err  = dados[:,3].reshape([nly,nlx])

#chi['CalcBe'],lamb['CalcBe'] = foil_prop('Al')#,thick=1e-2)
#distr['CalcBe']   = distribution(np.sqrt(X**2+Y**2)/chi['CalcBe'],lamb['CalcBe'],chi['CalcBe'])
#diAlN = dist(X[0,:].ravel(),Y[:,0].ravel(),distr['Al-N'])
diBeN = ThetaPhi2XY(X[0,:].ravel(),Y[:,0].ravel(),distr['Be-N'])
```

```python
key = 'Simul-Be-Air'
fig = plt.figure(figsize=(16.0,4.0))

ax1 = plt.subplot(1,3,1)
ax1.set_title('Simulated', fontsize=16)
ax1.pcolormesh(X*1e6,Y*1e6,distr[key])
ax1.set_xlabel(r'$\theta_x [\mu$rad$]$',fontsize=20)
ax1.set_ylabel(r'$\theta_y [\mu$rad$]$',fontsize=20)
ax1.set_xlim(200*np.array([-1,1]))
ax1.set_ylim(200*np.array([-1,1]))
ax1.tick_params(labelsize=12)

ax2 = plt.subplot(1,3,2)
ax2.set_title('Analytic', fontsize=16)
a = ax2.get_position()
ax2.set_position([a.x0-0.015,a.y0,a.width,a.height])
ax2.pcolormesh(X*1e6,Y*1e6,diBeN)
ax2.set_xlabel(r'$\theta_x [\mu$rad$]$',fontsize=20)
ax2.set_yticklabels('',visible=False)
ax2.set_xlim(200*np.array([-1,1]))
ax2.set_ylim(200*np.array([-1,1]))
ax2.tick_params(labelsize=12)

ax3 = plt.subplot(1,3,3)
ax3.plot(Y[:,nly//2+1] * 1e6,distr[key][:,nly//2+1]*1e-6,label='Simulated')
ax3.plot(Y[:,nly//2+1] * 1e6,diBeN[:,nly//2+1]*1e-6,label='Analytic')
plt.legend(loc='best')
ax3.set_xlim(200*np.array([-1,1]))
ax3.set_xlabel(r'$\theta_y [\mu$rad$]$',fontsize=20)
ax3.set_ylabel(r'$f [$mrad$^{-2}]$',fontsize=20)
ax3.tick_params(labelsize=12)

fig.show()

label = r"""\begin{eqnarray}"""
norma[key]     = np.trapz(np.trapz(distr[key],x=X[0,:].ravel(),axis=1),x=Y[:,0].ravel())
mex      = np.trapz(np.trapz(X*distr[key],x=X[0,:].ravel(),axis=1),x=Y[:,0].ravel())
mey      = np.trapz(np.trapz(Y*distr[key],x=Y[:,0].ravel(),axis=0),x=X[0,:].ravel())
desvio[key]    = [0,0]
desvio[key][0] = np.sqrt(np.trapz(np.trapz(X**2*distr[key],x=X[0,:].ravel(),axis=1),x=Y[:,0].ravel()) -mex**2)
desvio[key][1] = np.sqrt(np.trapz(np.trapz(Y**2*distr[key],x=Y[:,0].ravel(),axis=0),x=X[0,:].ravel()) -mey**2)
label += r"""\text{{Simulated}}: &||f|| = {0:7.5f},\quad \sigma_{{\theta_x}} = {1:7.5f}\,, \quad \sigma_{{\theta_y}} = {2:7.5f}\\""".format(norma[key],desvio[key][0]*1e6, desvio[key][1]*1e6)

no = np.trapz(np.trapz(diBeN,x=X[0,:].ravel(),axis=1),x=Y[:,0].ravel())
mex = np.trapz(np.trapz(X*diBeN,x=X[0,:].ravel(),axis=1),x=Y[:,0].ravel())
mey = np.trapz(np.trapz(Y*diBeN,x=Y[:,0].ravel(),axis=0),x=X[0,:].ravel())
dex = np.sqrt(np.trapz(np.trapz(X**2*diBeN,x=X[0,:].ravel(),axis=1),x=Y[:,0].ravel()) -(mey)**2)
dey = np.sqrt(np.trapz(np.trapz(Y**2*diBeN,x=Y[:,0].ravel(),axis=0),x=X[0,:].ravel()) -(mex)**2)
label += r"""\text{{Analytic}}: &||f|| = {0:7.5f},\quad \sigma_{{\theta_x}} = {1:7.5f}\,, \quad \sigma_{{\theta_y}} = {2:7.5f}""".format(no,dex*1e6, dey*1e6)

norm  = 2*np.pi*np.trapz(theta   *h0(theta,sigx),x=theta)
medi  = 2*np.pi*np.trapz(theta**2*h0(theta,sigx),x=theta)/sigx/np.sqrt(np.pi/2)
desvi = 2*np.pi*np.trapz(theta**3*h0(theta,sigx),x=theta)/(2*sigx**2)

Latex(label+r"""\end{eqnarray}""")
```

### Aluminium - Air:

```python
dados = np.loadtxt('FlukaSimulation/Al10umAir5cm.dat')
L = 299 # cm
nly = np.unique(dados[:,0]).shape[0]
nlx = len(dados)//nly
X = dados[:,0].reshape([nly,nlx]).T/L#*1e-2#
Y = dados[:,1].reshape([nly,nlx]).T/L
angl['Simul-Al-Air']  = (X[0,:].ravel(),Y[:,0].ravel())
distr['Simul-Al-Air'] = dados[:,2].reshape([nly,nlx])*L**2#*1e4#
err  = dados[:,3].reshape([nly,nlx])

diAlN = ThetaPhi2XY(X[0,:].ravel(),Y[:,0].ravel(),distr['Al-N'])
```

```python
key = 'Simul-Al-Air'
fig = plt.figure(figsize=(16.0,4.0))

ax1 = plt.subplot(1,3,1)
ax1.set_title('Simulated', fontsize=16)
ax1.pcolormesh(X*1e6,Y*1e6,distr[key])
ax1.set_xlabel(r'$\theta_x [\mu$rad$]$',fontsize=20)
ax1.set_ylabel(r'$\theta_y [\mu$rad$]$',fontsize=20)
ax1.set_xlim(200*np.array([-1,1]))
ax1.set_ylim(200*np.array([-1,1]))
ax1.tick_params(labelsize=12)

ax2 = plt.subplot(1,3,2)
ax2.set_title('Analytic', fontsize=16)
a = ax2.get_position()
ax2.set_position([a.x0-0.015,a.y0,a.width,a.height])
ax2.pcolormesh(X*1e6,Y*1e6,diBeN)
ax2.set_xlabel(r'$\theta_x [\mu$rad$]$',fontsize=20)
ax2.set_yticklabels('',visible=False)
ax2.set_xlim(200*np.array([-1,1]))
ax2.set_ylim(200*np.array([-1,1]))
ax2.tick_params(labelsize=12)

ax3 = plt.subplot(1,3,3)
ax3.plot(Y[:,nly//2+1] * 1e6,distr[key][:,nly//2+1]*1e-6,label='Simulated')
ax3.plot(Y[:,nly//2+1] * 1e6,diBeN[:,nly//2+1]*1e-6,label='Analytic')
plt.legend(loc='best')
ax3.set_xlim(200*np.array([-1,1]))
ax3.set_xlabel(r'$\theta_y [\mu$rad$]$',fontsize=20)
ax3.set_ylabel(r'$f [$mrad$^{-2}]$',fontsize=20)
ax3.tick_params(labelsize=12)

fig.show()

label = r"""\begin{eqnarray}"""
norma[key]     = np.trapz(np.trapz(distr[key],x=X[0,:].ravel(),axis=1),x=Y[:,0].ravel())
mex      = np.trapz(np.trapz(X*distr[key],x=X[0,:].ravel(),axis=1),x=Y[:,0].ravel())
mey      = np.trapz(np.trapz(Y*distr[key],x=Y[:,0].ravel(),axis=0),x=X[0,:].ravel())
desvio[key]    = [0,0]
desvio[key][0] = np.sqrt(np.trapz(np.trapz(X**2*distr[key],x=X[0,:].ravel(),axis=1),x=Y[:,0].ravel()) -mex**2)
desvio[key][1] = np.sqrt(np.trapz(np.trapz(Y**2*distr[key],x=Y[:,0].ravel(),axis=0),x=X[0,:].ravel()) -mey**2)
label += r"""\text{{Simulated}}: &||f|| = {0:7.5f},\quad \sigma_{{\theta_x}} = {1:7.5f}\,, \quad \sigma_{{\theta_y}} = {2:7.5f}\\""".format(norma[key],desvio[key][0]*1e6, desvio[key][1]*1e6)

no = np.trapz(np.trapz(diAlN,x=X[0,:].ravel(),axis=1),x=Y[:,0].ravel())
mex = np.trapz(np.trapz(X*diAlN,x=X[0,:].ravel(),axis=1),x=Y[:,0].ravel())
mey = np.trapz(np.trapz(Y*diAlN,x=Y[:,0].ravel(),axis=0),x=X[0,:].ravel())
dex = np.sqrt(np.trapz(np.trapz(X**2*diAlN,x=X[0,:].ravel(),axis=1),x=Y[:,0].ravel()) -(mey)**2)
dey = np.sqrt(np.trapz(np.trapz(Y**2*diAlN,x=Y[:,0].ravel(),axis=0),x=X[0,:].ravel()) -(mex)**2)
label += r"""\text{{Analytic}}: &||f|| = {0:7.5f},\quad \sigma_{{\theta_x}} = {1:7.5f}\,, \quad \sigma_{{\theta_y}} = {2:7.5f}""".format(no,dex*1e6, dey*1e6)

norm  = 2*np.pi*np.trapz(theta   *h0(theta,sigx),x=theta)
medi  = 2*np.pi*np.trapz(theta**2*h0(theta,sigx),x=theta)/sigx/np.sqrt(np.pi/2)
desvi = 2*np.pi*np.trapz(theta**3*h0(theta,sigx),x=theta)/(2*sigx**2)

Latex(label+r"""\end{eqnarray}""")
```

We Notice that for both windowns, the amplitude of the distribution function at small angles is smaller in the simulation than in the analytic calculation, which could explain the smaller values for the standard deviation provided by the latter method.

The picture bellow shows the error, in percent, provided by Fluka in the calculation of the distribution function. For the area of interest for us, $(-100\mu\text{rad},100\mu\text{rad})$, the errors are acceptibles.

```python
plt.pcolormesh(X*1e6,Y*1e6, np.abs(err))
plt.xlim(400*np.array([-1,1]))
plt.ylim(400*np.array([-1,1]))
plt.colorbar()
plt.show()
```

```python

```