# Multi_Layer Perceptron

* Author : Harrytsz
* Github : https://github.com/harrytsz/Deep_Learning_Models/

多层感知机：https://blog.csdn.net/xholes/article/details/78461164

            <svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
              <path stroke-linecap="round" d="M5,0 0,2.5 5,5z" id="raphael-marker-block" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0);"></path>
            </svg>

多层感知机：MLP

多层感知机由感知机推广而来，最主要的特点是有多个神经元层，因此也叫深度神经网络(DNN: Deep Neural Networks)。

感知机：PLA

多层感知机是由感知机推广而来，感知机学习算法(PLA: Perceptron Learning Algorithm)用神经元的结构进行描述的话就是一个单独的。

感知机的神经网络表示如下：

<span class="MathJax MathJax_FullWidth" id="MathJax-Element-1-Frame" tabindex="0" style="position: relative;" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mi>u</mi><mo>=</mo><munderover><mo>&#x2211;</mo><mrow class="MJX-TeXAtom-ORD"><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>n</mi></munderover><msub><mi>w</mi><mi>i</mi></msub><msub><mi>x</mi><mi>i</mi></msub><mo>+</mo><mi>b</mi><mspace linebreak="newline" /><mi>y</mi><mo>=</mo><mi>s</mi><mi>i</mi><mi>g</mi><mi>n</mi><mo stretchy="false">(</mo><mi>u</mi><mo stretchy="false">)</mo><mo>=</mo><mrow><mo>{</mo><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><mo>+</mo><mn>1</mn><mo>,</mo><mspace width="1em" /><mi>u</mi><mo>&#x003E;</mo><mn>0</mn></mtd></mtr><mtr><mtd><mo>&#x2212;</mo><mn>1</mn><mo>,</mo><mspace width="1em" /><mi>u</mi><mo>&#x2264;</mo><mn>0</mn></mtd></mtr></mtable><mo fence="true" stretchy="true" symmetric="true"></mo></mrow></math>" role="presentation"><nobr aria-hidden="true">u=∑i=1nwixi+by=sign(u)={+1,u>0−1,u≤0</nobr><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mi>u</mi><mo>=</mo><munderover><mo>∑</mo><mrow class="MJX-TeXAtom-ORD"><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>n</mi></munderover><msub><mi>w</mi><mi>i</mi></msub><msub><mi>x</mi><mi>i</mi></msub><mo>+</mo><mi>b</mi><mspace linebreak="newline"></mspace><mi>y</mi><mo>=</mo><mi>s</mi><mi>i</mi><mi>g</mi><mi>n</mi><mo stretchy="false">(</mo><mi>u</mi><mo stretchy="false">)</mo><mo>=</mo><mrow><mo>{</mo><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><mo>+</mo><mn>1</mn><mo>,</mo><mspace width="1em"></mspace><mi>u</mi><mo>></mo><mn>0</mn></mtd></mtr><mtr><mtd><mo>−</mo><mn>1</mn><mo>,</mo><mspace width="1em"></mspace><mi>u</mi><mo>≤</mo><mn>0</mn></mtd></mtr></mtable><mo fence="true" stretchy="true" symmetric="true"></mo></mrow></math>

从上述内容更可以看出，PLA是一个线性的二分类器，但不能对非线性的数据并不能进行有效的分类。因此便有了对网络层次的加深，理论上，多层网络可以模拟任何复杂的函数。

多层感知机：MLP

多层感知机的一个重要特点就是多层，我们将第一层称之为输入层，最后一层称之有输出层，中间的层称之为隐层。MLP并没有规定隐层的数量，因此可以根据各自的需求选择合适的隐层层数。且对于输出层神经元的个数也没有限制。

MLP神经网络结构模型如下,本文中只涉及了一个隐层，输入只有三个变量<span class="MathJax" id="MathJax-Element-2-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mo stretchy="false">[</mo><msub><mi>x</mi><mn>1</mn></msub><mo>,</mo><msub><mi>x</mi><mn>2</mn></msub><mo>,</mo><msub><mi>x</mi><mn>3</mn></msub><mo stretchy="false">]</mo></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">[x1,x2,x3]</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mo stretchy="false">[</mo><msub><mi>x</mi><mn>1</mn></msub><mo>,</mo><msub><mi>x</mi><mn>2</mn></msub><mo>,</mo><msub><mi>x</mi><mn>3</mn></msub><mo stretchy="false">]</mo></math>和一个偏置量<span class="MathJax" id="MathJax-Element-3-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>b</mi></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">b</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>b</mi></math>，输出层有三个神经元。相比于感知机算法中的神经元模型对其进行了集成。

前向传播

前向传播指的是信息从第一层逐渐地向高层进行传递的过程。以下图为例来进行前向传播的过程的分析。

假设第一层为输入层，输入的信息为<span class="MathJax" id="MathJax-Element-4-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mo stretchy="false">[</mo><msub><mi>x</mi><mn>1</mn></msub><mo>,</mo><msub><mi>x</mi><mn>2</mn></msub><mo>,</mo><msub><mi>x</mi><mn>3</mn></msub><mo stretchy="false">]</mo></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">[x1,x2,x3]</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mo stretchy="false">[</mo><msub><mi>x</mi><mn>1</mn></msub><mo>,</mo><msub><mi>x</mi><mn>2</mn></msub><mo>,</mo><msub><mi>x</mi><mn>3</mn></msub><mo stretchy="false">]</mo></math>。对于层<span class="MathJax" id="MathJax-Element-5-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">l</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi></math>，用<span class="MathJax" id="MathJax-Element-6-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>L</mi><mi>l</mi></msub></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">Ll</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>L</mi><mi>l</mi></msub></math>表示该层的所有神经元，其输出为<span class="MathJax" id="MathJax-Element-7-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mi>l</mi></msub></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">yl</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mi>l</mi></msub></math>,其中第<span class="MathJax" id="MathJax-Element-8-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>j</mi></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">j</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>j</mi></math>个节点的输出为<span class="MathJax" id="MathJax-Element-9-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">y(j)l</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></math>,该节点的输入为<span class="MathJax" id="MathJax-Element-10-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">u(j)l</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></math>，连接第<span class="MathJax" id="MathJax-Element-11-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">l</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi></math>层与<span class="MathJax" id="MathJax-Element-12-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mrow class="MJX-TeXAtom-ORD"><mo>&#x7B2C;</mo></mrow><mo stretchy="false">(</mo><mi>l</mi><mo>&#x2212;</mo><mn>1</mn><mo stretchy="false">)</mo></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">第(l−1)</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow class="MJX-TeXAtom-ORD"><mo>第</mo></mrow><mo stretchy="false">(</mo><mi>l</mi><mo>−</mo><mn>1</mn><mo stretchy="false">)</mo></math>层的权重矩阵为<span class="MathJax" id="MathJax-Element-13-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>W</mi><mi>l</mi></msub></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">Wl</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>W</mi><mi>l</mi></msub></math>，上一层（第<span class="MathJax" id="MathJax-Element-14-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi><mo>&#x2212;</mo><mn>1</mn></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">l−1</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi><mo>−</mo><mn>1</mn></math>层）的第<span class="MathJax" id="MathJax-Element-15-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>i</mi></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">i</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>i</mi></math>个节点到第<span class="MathJax" id="MathJax-Element-16-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">l</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi></math>层第<span class="MathJax" id="MathJax-Element-17-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>j</mi></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">j</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>j</mi></math>个节点的权重为<span class="MathJax" id="MathJax-Element-18-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">w(ji)l</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></math>。

结合之前定义的字母标记，对于第二层的三个神经元的输出则有：

<span class="MathJax" id="MathJax-Element-19-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><msubsup><mi>y</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><msubsup><mi>u</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><munderover><mo>&#x2211;</mo><mrow class="MJX-TeXAtom-ORD"><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>n</mi></munderover><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>1</mn><mi>i</mi></mrow></msubsup><msub><mi>x</mi><mi>i</mi></msub><mo>+</mo><msubsup><mi>b</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>11</mn><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>x</mi><mn>1</mn></msub><mo>+</mo><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>12</mn><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>x</mi><mn>2</mn></msub><mo>+</mo><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>13</mn><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>x</mi><mn>3</mn></msub><mo>+</mo><msubsup><mi>b</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo></mtd></mtr><mtr><mtd><msubsup><mi>y</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>2</mn><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><msubsup><mi>u</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>2</mn><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><munderover><mo>&#x2211;</mo><mrow class="MJX-TeXAtom-ORD"><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>n</mi></munderover><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>2</mn><mi>i</mi></mrow></msubsup><msub><mi>x</mi><mi>i</mi></msub><mo>+</mo><msubsup><mi>b</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>2</mn><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>21</mn><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>x</mi><mn>1</mn></msub><mo>+</mo><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>22</mn><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>x</mi><mn>2</mn></msub><mo>+</mo><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>23</mn><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>x</mi><mn>3</mn></msub><mo>+</mo><msubsup><mi>b</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>2</mn><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo></mtd></mtr><mtr><mtd><msubsup><mi>y</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>3</mn><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><msubsup><mi>u</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>3</mn><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><munderover><mo>&#x2211;</mo><mrow class="MJX-TeXAtom-ORD"><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>n</mi></munderover><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>3</mn><mi>i</mi></mrow></msubsup><msub><mi>x</mi><mi>i</mi></msub><mo>+</mo><msubsup><mi>b</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>3</mn><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>31</mn><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>x</mi><mn>1</mn></msub><mo>+</mo><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>32</mn><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>x</mi><mn>2</mn></msub><mo>+</mo><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>33</mn><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>x</mi><mn>3</mn></msub><mo>+</mo><msubsup><mi>b</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>3</mn><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo></mtd></mtr></mtable></math>" role="presentation" style="text-align: center; position: relative;"><nobr aria-hidden="true">y(1)2=f(u(1)2)=f(∑i=1nw1i2xi+b(1)2)=f(w(11)2x1+w(12)2x2+w(13)2x3+b(1)2)y(2)2=f(u(2)2)=f(∑i=1nw2i2xi+b(2)2)=f(w(21)2x1+w(22)2x2+w(23)2x3+b(2)2)y(3)2=f(u(3)2)=f(∑i=1nw3i2xi+b(3)2)=f(w(31)2x1+w(32)2x2+w(33)2x3+b(3)2)</nobr><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><msubsup><mi>y</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><msubsup><mi>u</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><munderover><mo>∑</mo><mrow class="MJX-TeXAtom-ORD"><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>n</mi></munderover><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>1</mn><mi>i</mi></mrow></msubsup><msub><mi>x</mi><mi>i</mi></msub><mo>+</mo><msubsup><mi>b</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>11</mn><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>x</mi><mn>1</mn></msub><mo>+</mo><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>12</mn><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>x</mi><mn>2</mn></msub><mo>+</mo><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>13</mn><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>x</mi><mn>3</mn></msub><mo>+</mo><msubsup><mi>b</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo></mtd></mtr><mtr><mtd><msubsup><mi>y</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>2</mn><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><msubsup><mi>u</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>2</mn><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><munderover><mo>∑</mo><mrow class="MJX-TeXAtom-ORD"><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>n</mi></munderover><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>2</mn><mi>i</mi></mrow></msubsup><msub><mi>x</mi><mi>i</mi></msub><mo>+</mo><msubsup><mi>b</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>2</mn><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>21</mn><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>x</mi><mn>1</mn></msub><mo>+</mo><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>22</mn><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>x</mi><mn>2</mn></msub><mo>+</mo><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>23</mn><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>x</mi><mn>3</mn></msub><mo>+</mo><msubsup><mi>b</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>2</mn><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo></mtd></mtr><mtr><mtd><msubsup><mi>y</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>3</mn><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><msubsup><mi>u</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>3</mn><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><munderover><mo>∑</mo><mrow class="MJX-TeXAtom-ORD"><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>n</mi></munderover><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>3</mn><mi>i</mi></mrow></msubsup><msub><mi>x</mi><mi>i</mi></msub><mo>+</mo><msubsup><mi>b</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>3</mn><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>31</mn><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>x</mi><mn>1</mn></msub><mo>+</mo><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>32</mn><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>x</mi><mn>2</mn></msub><mo>+</mo><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>33</mn><mo stretchy="false">)</mo></mrow></msubsup><msub><mi>x</mi><mn>3</mn></msub><mo>+</mo><msubsup><mi>b</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>3</mn><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo></mtd></mtr></mtable></math>

将上述的式子转换为矩阵表达式：

<span class="MathJax" id="MathJax-Element-20-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mrow class="MJX-TeXAtom-ORD"><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mn>2</mn></msub></mrow><mo>=</mo><mrow><mo>[</mo><mtable rowspacing="4pt" columnspacing="1em"><mtr><mtd><msubsup><mi>y</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr><mtr><mtd><msubsup><mi>y</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>2</mn><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr><mtr><mtd><msubsup><mi>y</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>3</mn><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr></mtable><mo>]</mo></mrow><mo>=</mo><mi>f</mi><mrow><mo>(</mo><mrow><mrow><mo>[</mo><mtable columnalign="center center center" rowspacing="4pt" columnspacing="1em"><mtr><mtd><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>11</mn></mrow></msubsup></mtd><mtd><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>12</mn></mrow></msubsup></mtd><mtd><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>13</mn></mrow></msubsup></mtd></mtr><mtr><mtd><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>21</mn></mrow></msubsup></mtd><mtd><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>22</mn></mrow></msubsup></mtd><mtd><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>23</mn></mrow></msubsup></mtd></mtr><mtr><mtd><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>31</mn></mrow></msubsup></mtd><mtd><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>32</mn></mrow></msubsup></mtd><mtd><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>33</mn></mrow></msubsup></mtd></mtr></mtable><mo>]</mo></mrow><mrow><mo>[</mo><mtable rowspacing="4pt" columnspacing="1em"><mtr><mtd><msub><mi>x</mi><mn>1</mn></msub></mtd></mtr><mtr><mtd><msub><mi>x</mi><mn>2</mn></msub></mtd></mtr><mtr><mtd><msub><mi>x</mi><mn>3</mn></msub></mtd></mtr></mtable><mo>]</mo></mrow><mo>+</mo><mrow><mo>[</mo><mtable rowspacing="4pt" columnspacing="1em"><mtr><mtd><msubsup><mi>b</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr><mtr><mtd><msubsup><mi>b</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>2</mn><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr><mtr><mtd><msubsup><mi>b</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>3</mn><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr></mtable><mo>]</mo></mrow></mrow><mo>)</mo></mrow><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><mrow class="MJX-TeXAtom-ORD"><mrow class="MJX-TeXAtom-ORD"><msub><mi mathvariant="bold">W</mi><mn mathvariant="bold">2</mn></msub><mi mathvariant="bold">X</mi><mo mathvariant="bold">+</mo><msub><mi mathvariant="bold">b</mi><mn mathvariant="bold">2</mn></msub></mrow></mrow><mo stretchy="false">)</mo></math>" role="presentation" style="text-align: center; position: relative;"><nobr aria-hidden="true">y2=⎡⎣⎢⎢⎢y(1)2y(2)2y(3)2⎤⎦⎥⎥⎥=f⎛⎝⎜⎜⎜⎡⎣⎢⎢w112w212w312w122w222w322w132w232w332⎤⎦⎥⎥⎡⎣⎢x1x2x3⎤⎦⎥+⎡⎣⎢⎢⎢b(1)2b(2)2b(3)2⎤⎦⎥⎥⎥⎞⎠⎟⎟⎟=f(W2X+b2)</nobr><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mrow class="MJX-TeXAtom-ORD"><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mn>2</mn></msub></mrow><mo>=</mo><mrow><mo>[</mo><mtable rowspacing="4pt" columnspacing="1em"><mtr><mtd><msubsup><mi>y</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr><mtr><mtd><msubsup><mi>y</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>2</mn><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr><mtr><mtd><msubsup><mi>y</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>3</mn><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr></mtable><mo>]</mo></mrow><mo>=</mo><mi>f</mi><mrow><mo>(</mo><mrow><mrow><mo>[</mo><mtable columnalign="center center center" rowspacing="4pt" columnspacing="1em"><mtr><mtd><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>11</mn></mrow></msubsup></mtd><mtd><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>12</mn></mrow></msubsup></mtd><mtd><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>13</mn></mrow></msubsup></mtd></mtr><mtr><mtd><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>21</mn></mrow></msubsup></mtd><mtd><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>22</mn></mrow></msubsup></mtd><mtd><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>23</mn></mrow></msubsup></mtd></mtr><mtr><mtd><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>31</mn></mrow></msubsup></mtd><mtd><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>32</mn></mrow></msubsup></mtd><mtd><msubsup><mi>w</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mn>33</mn></mrow></msubsup></mtd></mtr></mtable><mo>]</mo></mrow><mrow><mo>[</mo><mtable rowspacing="4pt" columnspacing="1em"><mtr><mtd><msub><mi>x</mi><mn>1</mn></msub></mtd></mtr><mtr><mtd><msub><mi>x</mi><mn>2</mn></msub></mtd></mtr><mtr><mtd><msub><mi>x</mi><mn>3</mn></msub></mtd></mtr></mtable><mo>]</mo></mrow><mo>+</mo><mrow><mo>[</mo><mtable rowspacing="4pt" columnspacing="1em"><mtr><mtd><msubsup><mi>b</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr><mtr><mtd><msubsup><mi>b</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>2</mn><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr><mtr><mtd><msubsup><mi>b</mi><mn>2</mn><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>3</mn><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr></mtable><mo>]</mo></mrow></mrow><mo>)</mo></mrow><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><mrow class="MJX-TeXAtom-ORD"><mrow class="MJX-TeXAtom-ORD"><msub><mi mathvariant="bold">W</mi><mn mathvariant="bold">2</mn></msub><mi mathvariant="bold">X</mi><mo mathvariant="bold">+</mo><msub><mi mathvariant="bold">b</mi><mn mathvariant="bold">2</mn></msub></mrow></mrow><mo stretchy="false">)</mo></math>

将第二层的前向传播计算过程推广到网络中的任意一层，则:

<span class="MathJax" id="MathJax-Element-21-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mrow><mo>{</mo><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd /><mtd><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><mi>f</mi><mrow><mo>(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>)</mo></mrow></mtd></mtr><mtr><mtd /><mtd><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><munder><mo>&#x2211;</mo><mrow class="MJX-TeXAtom-ORD"><mi>i</mi><mo>&#x2208;</mo><msub><mi>L</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>&#x2212;</mo><mn>1</mn></mrow></msub></mrow></munder><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup><msubsup><mi>y</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>&#x2212;</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>+</mo><msubsup><mi>b</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr><mtr><mtd /><mtd><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mi>l</mi></msub><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">W</mi></mrow><mi>l</mi></msub><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>&#x2212;</mo><mn>1</mn></mrow></msub><mo>+</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">b</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo></mtd></mtr></mtable><mo fence="true" stretchy="true" symmetric="true"></mo></mrow></math>" role="presentation" style="text-align: center; position: relative;"><nobr aria-hidden="true">⎧⎩⎨⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪y(j)l=f(u(j)l)u(j)l=∑i∈Ll−1w(ji)ly(i)l−1+b(j)lyl=f(ul)=f(Wlyl−1+bl)</nobr><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mrow><mo>{</mo><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd></mtd><mtd><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><mi>f</mi><mrow><mo>(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>)</mo></mrow></mtd></mtr><mtr><mtd></mtd><mtd><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><munder><mo>∑</mo><mrow class="MJX-TeXAtom-ORD"><mi>i</mi><mo>∈</mo><msub><mi>L</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>−</mo><mn>1</mn></mrow></msub></mrow></munder><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup><msubsup><mi>y</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>−</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>+</mo><msubsup><mi>b</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr><mtr><mtd></mtd><mtd><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mi>l</mi></msub><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo>=</mo><mi>f</mi><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">W</mi></mrow><mi>l</mi></msub><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>−</mo><mn>1</mn></mrow></msub><mo>+</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">b</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo></mtd></mtr></mtable><mo fence="true" stretchy="true" symmetric="true"></mo></mrow></math>
其中<span class="MathJax" id="MathJax-Element-22-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>f</mi><mo stretchy="false">(</mo><mo>&#x22C5;</mo><mo stretchy="false">)</mo></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">f(⋅)</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>f</mi><mo stretchy="false">(</mo><mo>⋅</mo><mo stretchy="false">)</mo></math>为激活函数，<span class="MathJax" id="MathJax-Element-23-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>b</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">b(j)l</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>b</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></math>为第<span class="MathJax" id="MathJax-Element-24-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">l</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi></math>层第<span class="MathJax" id="MathJax-Element-25-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>j</mi></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">j</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>j</mi></math>个节点的偏置。

反向传播

基本的模型搭建完成后的，训练的时候所做的就是完成模型参数的更新。由于存在多层的网络结构，因此无法直接对中间的隐层利用损失来进行参数更新，但可以利用损失从顶层到底层的反向传播来进行参数的估计。（约定：小写字母—标量，加粗小写字母—向量，大写字母—矩阵）

假设多层感知机用于分类，在输出层有多个神经元，每个神经元对应一个标签。输入样本为<span class="MathJax" id="MathJax-Element-61-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mrow class="MJX-TeXAtom-ORD"><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">x</mi></mrow></mrow><mo>=</mo><mo stretchy="false">[</mo><msub><mi>x</mi><mn>1</mn></msub><mo>,</mo><msub><mi>x</mi><mn>2</mn></msub><mo>,</mo><mo>&#x22EF;</mo><mo>,</mo><msub><mi>x</mi><mi>n</mi></msub><mo stretchy="false">]</mo></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">x=[x1,x2,⋯,xn]</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow class="MJX-TeXAtom-ORD"><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">x</mi></mrow></mrow><mo>=</mo><mo stretchy="false">[</mo><msub><mi>x</mi><mn>1</mn></msub><mo>,</mo><msub><mi>x</mi><mn>2</mn></msub><mo>,</mo><mo>⋯</mo><mo>,</mo><msub><mi>x</mi><mi>n</mi></msub><mo stretchy="false">]</mo></math>,其标签为<span class="MathJax" id="MathJax-Element-62-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mrow class="MJX-TeXAtom-ORD"><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">t</mi></mrow></mrow></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">t</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow class="MJX-TeXAtom-ORD"><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">t</mi></mrow></mrow></math>;

对于层<span class="MathJax" id="MathJax-Element-63-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">l</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi></math>，用<span class="MathJax" id="MathJax-Element-64-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>L</mi><mi>l</mi></msub></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">Ll</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>L</mi><mi>l</mi></msub></math>表示该层的所有神经元，其输出为<span class="MathJax" id="MathJax-Element-65-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mi>l</mi></msub></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">yl</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mi>l</mi></msub></math>,其中第<span class="MathJax" id="MathJax-Element-66-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>j</mi></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">j</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>j</mi></math>个节点的输出为<span class="MathJax" id="MathJax-Element-67-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">y(j)l</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></math>,该节点的输入为<span class="MathJax" id="MathJax-Element-68-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">u(j)l</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></math>，连接第<span class="MathJax" id="MathJax-Element-69-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">l</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi></math>层与<span class="MathJax" id="MathJax-Element-70-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mrow class="MJX-TeXAtom-ORD"><mo>&#x7B2C;</mo></mrow><mo stretchy="false">(</mo><mi>l</mi><mo>&#x2212;</mo><mn>1</mn><mo stretchy="false">)</mo></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">第(l−1)</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow class="MJX-TeXAtom-ORD"><mo>第</mo></mrow><mo stretchy="false">(</mo><mi>l</mi><mo>−</mo><mn>1</mn><mo stretchy="false">)</mo></math>层的权重矩阵为<span class="MathJax" id="MathJax-Element-71-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>W</mi><mi>l</mi></msub></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">Wl</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>W</mi><mi>l</mi></msub></math>，上一层（第<span class="MathJax" id="MathJax-Element-72-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi><mo>&#x2212;</mo><mn>1</mn></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">l−1</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi><mo>−</mo><mn>1</mn></math>层）的第<span class="MathJax" id="MathJax-Element-73-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>i</mi></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">i</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>i</mi></math>个节点到第<span class="MathJax" id="MathJax-Element-74-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">l</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi></math>层第<span class="MathJax" id="MathJax-Element-75-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>j</mi></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">j</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>j</mi></math>个节点的权重为<span class="MathJax" id="MathJax-Element-76-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">w(ji)l</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></math>。

对于网络的最后一层第<span class="MathJax" id="MathJax-Element-77-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>k</mi></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">k</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>k</mi></math>层——输出层，现在定义损失函数：

<span class="MathJax" id="MathJax-Element-78-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><mi>E</mi></mtd><mtd><mi></mi><mo>=</mo><mfrac><mn>1</mn><mn>2</mn></mfrac><munder><mo>&#x2211;</mo><mrow class="MJX-TeXAtom-ORD"><mi>j</mi><mo>&#x2208;</mo><msub><mi>L</mi><mi>k</mi></msub></mrow></munder><mo stretchy="false">(</mo><msup><mi>t</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msup><mo>&#x2212;</mo><msubsup><mi>y</mi><mi>k</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><msup><mo stretchy="false">)</mo><mn>2</mn></msup></mtd></mtr></mtable></math>" role="presentation" style="text-align: center; position: relative;"><nobr aria-hidden="true">E=12∑j∈Lk(t(j)−y(j)k)2</nobr><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><mi>E</mi></mtd><mtd><mi></mi><mo>=</mo><mfrac><mn>1</mn><mn>2</mn></mfrac><munder><mo>∑</mo><mrow class="MJX-TeXAtom-ORD"><mi>j</mi><mo>∈</mo><msub><mi>L</mi><mi>k</mi></msub></mrow></munder><mo stretchy="false">(</mo><msup><mi>t</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msup><mo>−</mo><msubsup><mi>y</mi><mi>k</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><msup><mo stretchy="false">)</mo><mn>2</mn></msup></mtd></mtr></mtable></math>

为了极小化损失函数，通过梯度下降来进行推导：

<span class="MathJax" id="MathJax-Element-79-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mrow><mo>{</mo><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mo>=</mo><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd></mtr><mtr><mtd><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>b</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>b</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mo>=</mo><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>b</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd></mtr></mtable><mo fence="true" stretchy="true" symmetric="true"></mo></mrow></math>" role="presentation" style="text-align: center; position: relative;"><nobr aria-hidden="true">⎧⎩⎨⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪∂E∂w(ji)l∂E∂b(j)l=∂E∂y(j)l∂y(j)l∂w(ji)l=∂E∂y(j)l∂y(j)l∂u(j)l∂u(j)l∂w(ji)l=∂E∂y(j)l∂y(j)l∂b(j)l=∂E∂y(j)l∂y(j)l∂u(j)l∂u(j)l∂b(j)l</nobr><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mrow><mo>{</mo><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mo>=</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd></mtr><mtr><mtd><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>b</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>b</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mo>=</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>b</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd></mtr></mtable><mo fence="true" stretchy="true" symmetric="true"></mo></mrow></math>

在上式子中，根据之前的定义，很容易得到：

<span class="MathJax" id="MathJax-Element-80-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mrow><mo>{</mo><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><msup><mi>f</mi><mo>&#x2032;</mo></msup><mo stretchy="false">(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo></mtd></mtr><mtr><mtd><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><msubsup><mi>y</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>&#x2212;</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr><mtr><mtd><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>b</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><mn>1</mn></mtd></mtr></mtable><mo fence="true" stretchy="true" symmetric="true"></mo></mrow></math>" role="presentation" style="text-align: center; position: relative;"><nobr aria-hidden="true">⎧⎩⎨⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪∂y(j)l∂u(j)l∂u(j)l∂w(ji)l∂u(j)l∂b(j)l=f′(u(j)l)=y(i)l−1=1</nobr><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mrow><mo>{</mo><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><msup><mi>f</mi><mo>′</mo></msup><mo stretchy="false">(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo></mtd></mtr><mtr><mtd><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><msubsup><mi>y</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>−</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr><mtr><mtd><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>b</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><mn>1</mn></mtd></mtr></mtable><mo fence="true" stretchy="true" symmetric="true"></mo></mrow></math>

那么则有：

<span class="MathJax" id="MathJax-Element-81-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mrow><mo>{</mo><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mo>=</mo><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><msup><mi>f</mi><mo>&#x2032;</mo></msup><mo stretchy="false">(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><msubsup><mi>y</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>&#x2212;</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr><mtr><mtd><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>b</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>b</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mo>=</mo><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><msup><mi>f</mi><mo>&#x2032;</mo></msup><mo stretchy="false">(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo></mtd></mtr></mtable><mo fence="true" stretchy="true" symmetric="true"></mo></mrow></math>" role="presentation" style="text-align: center; position: relative;"><nobr aria-hidden="true">⎧⎩⎨⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪∂E∂w(ji)l∂E∂b(j)l=∂E∂y(j)l∂y(j)l∂u(j)l∂u(j)l∂w(ji)l=∂E∂y(j)lf′(u(j)l)y(i)l−1=∂E∂y(j)l∂y(j)l∂u(j)l∂u(j)l∂b(j)l=∂E∂y(j)lf′(u(j)l)</nobr><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mrow><mo>{</mo><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mo>=</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><msup><mi>f</mi><mo>′</mo></msup><mo stretchy="false">(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><msubsup><mi>y</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>−</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr><mtr><mtd><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>b</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>b</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mo>=</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><msup><mi>f</mi><mo>′</mo></msup><mo stretchy="false">(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo></mtd></mtr></mtable><mo fence="true" stretchy="true" symmetric="true"></mo></mrow></math>

另有，下一层所有结点的输入都与前一层的每个结点输出有关，因此损失函数可以认为是下一层的每个神经元结点输入的函数。那么：

<span class="MathJax" id="MathJax-Element-82-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi><mo stretchy="false">(</mo><msubsup><mi>u</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></msubsup><mo>,</mo><msubsup><mi>u</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>2</mn><mo stretchy="false">)</mo></mrow></msubsup><mo>,</mo><mo>.</mo><mo>.</mo><mo>.</mo><mo>,</mo><msubsup><mi>u</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>,</mo><mo>.</mo><mo>.</mo><mo>.</mo><mo>,</mo><msubsup><mi>u</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>K</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd></mtr><mtr><mtd /><mtd><mi></mi><mo>=</mo><munder><mo>&#x2211;</mo><mrow class="MJX-TeXAtom-ORD"><mi>k</mi><mo>&#x2208;</mo><msub><mi>L</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow></msub></mrow></munder><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd></mtr><mtr><mtd /><mtd><mi></mi><mo>=</mo><munder><mo>&#x2211;</mo><mrow class="MJX-TeXAtom-ORD"><mi>k</mi><mo>&#x2208;</mo><msub><mi>L</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow></msub></mrow></munder><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd></mtr><mtr><mtd /><mtd><mi></mi><mo>=</mo><munder><mo>&#x2211;</mo><mrow class="MJX-TeXAtom-ORD"><mi>k</mi><mo>&#x2208;</mo><msub><mi>L</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow></msub></mrow></munder><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><msubsup><mi>w</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr></mtable></math>" role="presentation" style="text-align: center; position: relative;"><nobr aria-hidden="true">∂E∂y(j)l=∂E(u(1)l+1,u(2)l+1,...,u(k)l+1,...,u(K)l+1)∂y(j)l=∑k∈Ll+1∂E∂u(k)l+1∂u(k)l+1∂y(j)l=∑k∈Ll+1∂E∂y(k)l+1∂y(k)l+1∂u(k)l+1∂u(k)l+1∂y(j)l=∑k∈Ll+1∂E∂y(k)l+1∂y(k)l+1∂u(k)l+1w(kj)l+1</nobr><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi><mo stretchy="false">(</mo><msubsup><mi>u</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo></mrow></msubsup><mo>,</mo><msubsup><mi>u</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mn>2</mn><mo stretchy="false">)</mo></mrow></msubsup><mo>,</mo><mo>.</mo><mo>.</mo><mo>.</mo><mo>,</mo><msubsup><mi>u</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>,</mo><mo>.</mo><mo>.</mo><mo>.</mo><mo>,</mo><msubsup><mi>u</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>K</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd></mtr><mtr><mtd></mtd><mtd><mi></mi><mo>=</mo><munder><mo>∑</mo><mrow class="MJX-TeXAtom-ORD"><mi>k</mi><mo>∈</mo><msub><mi>L</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow></msub></mrow></munder><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd></mtr><mtr><mtd></mtd><mtd><mi></mi><mo>=</mo><munder><mo>∑</mo><mrow class="MJX-TeXAtom-ORD"><mi>k</mi><mo>∈</mo><msub><mi>L</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow></msub></mrow></munder><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd></mtr><mtr><mtd></mtd><mtd><mi></mi><mo>=</mo><munder><mo>∑</mo><mrow class="MJX-TeXAtom-ORD"><mi>k</mi><mo>∈</mo><msub><mi>L</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow></msub></mrow></munder><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><msubsup><mi>w</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr></mtable></math>

 此处定义节点的灵敏度为误差对输入的变化率，即：

<span class="MathJax" id="MathJax-Element-83-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mi>&#x03B4;</mi><mo>=</mo><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><mi>u</mi></mrow></mfrac></math>" role="presentation" style="text-align: center; position: relative;"><nobr aria-hidden="true">δ=∂E∂u</nobr><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mi>δ</mi><mo>=</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><mi>u</mi></mrow></mfrac></math>

那么第<span class="MathJax" id="MathJax-Element-84-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">l</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>l</mi></math>层第<span class="MathJax" id="MathJax-Element-85-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>j</mi></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">j</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mi>j</mi></math>个节点的灵敏度为：

<span class="MathJax" id="MathJax-Element-86-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><msubsup><mi>&#x03B4;</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mo>=</mo><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mo>=</mo><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><msup><mi>f</mi><mo>&#x2032;</mo></msup><mo stretchy="false">(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo></math>" role="presentation" style="text-align: center; position: relative;"><nobr aria-hidden="true">δ(j)l=∂E∂u(j)l=∂E∂y(j)l∂y(j)l∂u(j)l=∂E∂y(j)lf′(u(j)l)</nobr><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><msubsup><mi>δ</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mo>=</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mo>=</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><msup><mi>f</mi><mo>′</mo></msup><mo stretchy="false">(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo></math>

结合灵敏度的定义，则有：

<span class="MathJax" id="MathJax-Element-87-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><munder><mo>&#x2211;</mo><mrow class="MJX-TeXAtom-ORD"><mi>k</mi><mo>&#x2208;</mo><msub><mi>L</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow></msub></mrow></munder><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><msubsup><mi>w</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr><mtr><mtd /><mtd><mi></mi><mo>=</mo><munder><mo>&#x2211;</mo><mrow class="MJX-TeXAtom-ORD"><mi>k</mi><mo>&#x2208;</mo><msub><mi>L</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow></msub></mrow></munder><msubsup><mi>&#x03B4;</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mi>k</mi></msubsup><msubsup><mi>w</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr></mtable></math>" role="presentation" style="text-align: center; position: relative;"><nobr aria-hidden="true">∂E∂y(j)l=∑k∈Ll+1∂E∂y(k)l+1∂y(k)l+1∂u(k)l+1w(kj)l+1=∑k∈Ll+1δkl+1w(kj)l+1</nobr><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><munder><mo>∑</mo><mrow class="MJX-TeXAtom-ORD"><mi>k</mi><mo>∈</mo><msub><mi>L</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow></msub></mrow></munder><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><msubsup><mi>w</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr><mtr><mtd></mtd><mtd><mi></mi><mo>=</mo><munder><mo>∑</mo><mrow class="MJX-TeXAtom-ORD"><mi>k</mi><mo>∈</mo><msub><mi>L</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow></msub></mrow></munder><msubsup><mi>δ</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mi>k</mi></msubsup><msubsup><mi>w</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr></mtable></math>

上式两边同时乘上<span class="MathJax" id="MathJax-Element-88-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><msup><mi>f</mi><mo>&#x2032;</mo></msup><mo stretchy="false">(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">f′(u(j)l)</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><msup><mi>f</mi><mo>′</mo></msup><mo stretchy="false">(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo></math>，则有

<span class="MathJax MathJax_FullWidth" id="MathJax-Element-89-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><msubsup><mi>&#x03B4;</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><msup><mi>f</mi><mo>&#x2032;</mo></msup><mo stretchy="false">(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><mo>=</mo><msup><mi>f</mi><mo>&#x2032;</mo></msup><mo stretchy="false">(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><munder><mo>&#x2211;</mo><mrow class="MJX-TeXAtom-ORD"><mi>k</mi><mo>&#x2208;</mo><msub><mi>L</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow></msub></mrow></munder><msubsup><mi>&#x03B4;</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mi>k</mi></msubsup><msubsup><mi>w</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mspace linebreak="newline" /></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">δ(j)l=∂E∂y(j)lf′(u(j)l)=f′(u(j)l)∑k∈Ll+1δkl+1w(kj)l+1</nobr><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><msubsup><mi>δ</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><msup><mi>f</mi><mo>′</mo></msup><mo stretchy="false">(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><mo>=</mo><msup><mi>f</mi><mo>′</mo></msup><mo stretchy="false">(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><munder><mo>∑</mo><mrow class="MJX-TeXAtom-ORD"><mi>k</mi><mo>∈</mo><msub><mi>L</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow></msub></mrow></munder><msubsup><mi>δ</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mi>k</mi></msubsup><msubsup><mi>w</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mspace linebreak="newline"></mspace></math>

注意到上式中表达的是前后两层的灵敏度关系，而对于最后一层，也就是输出层来说，并不存在后续的一层，因此并不满足上式。但输出层的输出是直接和误差联系的，因此可以用损失函数的定义来直接求取偏导数。那么：

<span class="MathJax" id="MathJax-Element-90-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><msubsup><mi>&#x03B4;</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><msup><mi>f</mi><mo>&#x2032;</mo></msup><mo stretchy="false">(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><mo>=</mo><mrow><mo>{</mo><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd /><mtd><msup><mi>f</mi><mo>&#x2032;</mo></msup><mo stretchy="false">(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><munder><mo>&#x2211;</mo><mrow class="MJX-TeXAtom-ORD"><mi>k</mi><mo>&#x2208;</mo><msub><mi>L</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow></msub></mrow></munder><msubsup><mi>&#x03B4;</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mi>k</mi></msubsup><msubsup><mi>w</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mspace width="2em" /><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo>&#x5C42;</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>&#x4E3A;</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>&#x9690;</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>&#x5C42;</mo></mrow></mtd></mtr><mtr><mtd /><mtd><msup><mi>f</mi><mo>&#x2032;</mo></msup><mo stretchy="false">(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><mo stretchy="false">(</mo><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>&#x2212;</mo><msup><mi>t</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msup><mo stretchy="false">)</mo><mspace width="2em" /><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo>&#x5C42;</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>&#x4E3A;</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>&#x8F93;</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>&#x51FA;</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>&#x5C42;</mo></mrow></mtd></mtr></mtable><mo fence="true" stretchy="true" symmetric="true"></mo></mrow></math>" role="presentation" style="text-align: center; position: relative;"><nobr aria-hidden="true">δ(j)l=∂E∂y(j)lf′(u(j)l)=⎧⎩⎨⎪⎪f′(u(j)l)∑k∈Ll+1δkl+1w(kj)l+1l层为隐层f′(u(j)l)(y(j)l−t(j))l层为输出层</nobr><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><msubsup><mi>δ</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>=</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><msup><mi>f</mi><mo>′</mo></msup><mo stretchy="false">(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><mo>=</mo><mrow><mo>{</mo><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd></mtd><mtd><msup><mi>f</mi><mo>′</mo></msup><mo stretchy="false">(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><munder><mo>∑</mo><mrow class="MJX-TeXAtom-ORD"><mi>k</mi><mo>∈</mo><msub><mi>L</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow></msub></mrow></munder><msubsup><mi>δ</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mi>k</mi></msubsup><msubsup><mi>w</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>k</mi><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mspace width="2em"></mspace><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo>层</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>为</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>隐</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>层</mo></mrow></mtd></mtr><mtr><mtd></mtd><mtd><msup><mi>f</mi><mo>′</mo></msup><mo stretchy="false">(</mo><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo stretchy="false">)</mo><mo stretchy="false">(</mo><msubsup><mi>y</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><mo>−</mo><msup><mi>t</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msup><mo stretchy="false">)</mo><mspace width="2em"></mspace><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo>层</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>为</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>输</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>出</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>层</mo></mrow></mtd></mtr></mtable><mo fence="true" stretchy="true" symmetric="true"></mo></mrow></math>

至此，损失函数对各参数的梯度为：

<span class="MathJax" id="MathJax-Element-91-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mrow><mo>{</mo><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mo>=</mo><msubsup><mi>&#x03B4;</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><msubsup><mi>y</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>&#x2212;</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr><mtr><mtd><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>b</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msubsup><mi>b</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mo>=</mo><msubsup><mi>&#x03B4;</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr></mtable><mo fence="true" stretchy="true" symmetric="true"></mo></mrow></math>" role="presentation" style="text-align: center; position: relative;"><nobr aria-hidden="true">⎧⎩⎨⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪⎪∂E∂w(ji)l∂E∂b(j)l=∂E∂u(j)l∂u(j)l∂w(ji)l=δ(j)ly(i)l−1=∂E∂u(j)l∂u(j)l∂b(j)l=δ(j)l</nobr><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mrow><mo>{</mo><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>w</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mo>=</mo><msubsup><mi>δ</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup><msubsup><mi>y</mi><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>−</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>i</mi><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr><mtr><mtd><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>b</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mfrac><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>u</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow><mrow><mi mathvariant="normal">∂</mi><msubsup><mi>b</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mrow></mfrac><mo>=</mo><msubsup><mi>δ</mi><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo stretchy="false">(</mo><mi>j</mi><mo stretchy="false">)</mo></mrow></msubsup></mtd></mtr></mtable><mo fence="true" stretchy="true" symmetric="true"></mo></mrow></math>

上述的推到都是建立在单个节点的基础上，对于各层所有节点，采用矩阵的方式表示，则上述公式可以写成：

<span class="MathJax" id="MathJax-Element-92-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">W</mi></mrow><mi>l</mi></msub></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><msub><mrow class="MJX-TeXAtom-ORD"><mrow class="MJX-TeXAtom-ORD"><mo mathvariant="bold">&#x3B4;</mo></mrow></mrow><mi>l</mi></msub><msubsup><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>&#x2212;</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="normal">T</mi></mrow></msubsup></mtd></mtr><mtr><mtd><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">b</mi></mrow><mi>l</mi></msub></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><msub><mrow class="MJX-TeXAtom-ORD"><mrow class="MJX-TeXAtom-ORD"><mo mathvariant="bold">&#x3B4;</mo></mrow></mrow><mi>l</mi></msub></mtd></mtr><mtr><mtd><msub><mrow class="MJX-TeXAtom-ORD"><mrow class="MJX-TeXAtom-ORD"><mo mathvariant="bold">&#x3B4;</mo></mrow></mrow><mi>l</mi></msub></mtd><mtd><mi></mi><mo>=</mo><mrow><mo>{</mo><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><mo stretchy="false">(</mo></mtd><mtd><msubsup><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">W</mi></mrow><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mi>T</mi></msubsup><msub><mrow class="MJX-TeXAtom-ORD"><mrow class="MJX-TeXAtom-ORD"><mo mathvariant="bold">&#x3B4;</mo></mrow></mrow><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow></msub><mo stretchy="false">)</mo><mo>&#x2218;</mo><msup><mi>f</mi><mo>&#x2032;</mo></msup><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo>,</mo><mspace width="2em" /><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo>&#x5C42;</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>&#x4E3A;</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>&#x9690;</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>&#x5C42;</mo></mrow></mtd></mtr><mtr><mtd /><mtd><mi></mi><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mi>l</mi></msub><mo>&#x2212;</mo><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">t</mi></mrow><mo stretchy="false">)</mo><mo>&#x2218;</mo><msup><mi>f</mi><mo>&#x2032;</mo></msup><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo>,</mo><mspace width="2em" /><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo>&#x5C42;</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>&#x4E3A;</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>&#x8F93;</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>&#x51FA;</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>&#x5C42;</mo></mrow></mtd></mtr></mtable><mo fence="true" stretchy="true" symmetric="true"></mo></mrow></mtd></mtr></mtable></math>" role="presentation" style="text-align: center; position: relative;"><nobr aria-hidden="true">∂E∂Wl∂E∂blδl=δlyTl−1=δl={(WTl+1δl+1)∘f′(ul),l层为隐层(yl−t)∘f′(ul),l层为输出层</nobr><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">W</mi></mrow><mi>l</mi></msub></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><msub><mrow class="MJX-TeXAtom-ORD"><mrow class="MJX-TeXAtom-ORD"><mo mathvariant="bold">δ</mo></mrow></mrow><mi>l</mi></msub><msubsup><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>−</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="normal">T</mi></mrow></msubsup></mtd></mtr><mtr><mtd><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">b</mi></mrow><mi>l</mi></msub></mrow></mfrac></mtd><mtd><mi></mi><mo>=</mo><msub><mrow class="MJX-TeXAtom-ORD"><mrow class="MJX-TeXAtom-ORD"><mo mathvariant="bold">δ</mo></mrow></mrow><mi>l</mi></msub></mtd></mtr><mtr><mtd><msub><mrow class="MJX-TeXAtom-ORD"><mrow class="MJX-TeXAtom-ORD"><mo mathvariant="bold">δ</mo></mrow></mrow><mi>l</mi></msub></mtd><mtd><mi></mi><mo>=</mo><mrow><mo>{</mo><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><mo stretchy="false">(</mo></mtd><mtd><msubsup><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">W</mi></mrow><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow><mi>T</mi></msubsup><msub><mrow class="MJX-TeXAtom-ORD"><mrow class="MJX-TeXAtom-ORD"><mo mathvariant="bold">δ</mo></mrow></mrow><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>+</mo><mn>1</mn></mrow></msub><mo stretchy="false">)</mo><mo>∘</mo><msup><mi>f</mi><mo>′</mo></msup><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo>,</mo><mspace width="2em"></mspace><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo>层</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>为</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>隐</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>层</mo></mrow></mtd></mtr><mtr><mtd></mtd><mtd><mi></mi><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mi>l</mi></msub><mo>−</mo><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">t</mi></mrow><mo stretchy="false">)</mo><mo>∘</mo><msup><mi>f</mi><mo>′</mo></msup><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo>,</mo><mspace width="2em"></mspace><mi>l</mi><mrow class="MJX-TeXAtom-ORD"><mo>层</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>为</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>输</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>出</mo></mrow><mrow class="MJX-TeXAtom-ORD"><mo>层</mo></mrow></mtd></mtr></mtable><mo fence="true" stretchy="true" symmetric="true"></mo></mrow></mtd></mtr></mtable></math>

其中运算符<span class="MathJax" id="MathJax-Element-93-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mo>&#x2218;</mo></math>" role="presentation" style="position: relative;"><nobr aria-hidden="true">∘</nobr><math xmlns="http://www.w3.org/1998/Math/MathML"><mo>∘</mo></math>表示矩阵或者向量中的对应元素相乘。

常见的几个激活函数的导数为：

<span class="MathJax" id="MathJax-Element-94-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><msup><mi>f</mi><mo>&#x2032;</mo></msup><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo></mtd><mtd><mi></mi><mo>=</mo><mi>s</mi><mi>i</mi><mi>g</mi><mi>m</mi><mi>o</mi><mi>i</mi><msup><mi>d</mi><mo>&#x2032;</mo></msup><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo>=</mo><mi>s</mi><mi>i</mi><mi>g</mi><mi>m</mi><mi>o</mi><mi>i</mi><mi>d</mi><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo stretchy="false">(</mo><mn>1</mn><mo>&#x2212;</mo><mi>s</mi><mi>i</mi><mi>g</mi><mi>m</mi><mi>o</mi><mi>i</mi><mi>d</mi><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mo>=</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mi>l</mi></msub><mo stretchy="false">(</mo><mn>1</mn><mo>&#x2212;</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo></mtd></mtr><mtr><mtd><msup><mi>f</mi><mo>&#x2032;</mo></msup><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo></mtd><mtd><mi></mi><mo>=</mo><mi>t</mi><mi>a</mi><mi>n</mi><msup><mi>h</mi><mo>&#x2032;</mo></msup><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo>=</mo><mn>1</mn><mo>&#x2212;</mo><mi>t</mi><mi>a</mi><mi>n</mi><msup><mi>h</mi><mn>2</mn></msup><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo>=</mo><mn>1</mn><mo>&#x2212;</mo><msubsup><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mi>l</mi><mn>2</mn></msubsup></mtd></mtr><mtr><mtd><msup><mi>f</mi><mo>&#x2032;</mo></msup><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo></mtd><mtd><mi></mi><mo>=</mo><mi>s</mi><mi>o</mi><mi>f</mi><mi>t</mi><mi>m</mi><mi>a</mi><msup><mi>x</mi><mo>&#x2032;</mo></msup><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo>=</mo><mi>s</mi><mi>o</mi><mi>f</mi><mi>t</mi><mi>m</mi><mi>a</mi><mi>x</mi><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo>&#x2212;</mo><mi>s</mi><mi>o</mi><mi>f</mi><mi>t</mi><mi>m</mi><mi>a</mi><msup><mi>x</mi><mn>2</mn></msup><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo>=</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mi>l</mi></msub><mo>&#x2212;</mo><msubsup><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mi>l</mi><mn>2</mn></msubsup></mtd></mtr></mtable></math>" role="presentation" style="text-align: center; position: relative;"><nobr aria-hidden="true">f′(ul)f′(ul)f′(ul)=sigmoid′(ul)=sigmoid(ul)(1−sigmoid(ul))=yl(1−yl)=tanh′(ul)=1−tanh2(ul)=1−y2l=softmax′(ul)=softmax(ul)−softmax2(ul)=yl−y2l</nobr><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><msup><mi>f</mi><mo>′</mo></msup><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo></mtd><mtd><mi></mi><mo>=</mo><mi>s</mi><mi>i</mi><mi>g</mi><mi>m</mi><mi>o</mi><mi>i</mi><msup><mi>d</mi><mo>′</mo></msup><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo>=</mo><mi>s</mi><mi>i</mi><mi>g</mi><mi>m</mi><mi>o</mi><mi>i</mi><mi>d</mi><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><mi>s</mi><mi>i</mi><mi>g</mi><mi>m</mi><mi>o</mi><mi>i</mi><mi>d</mi><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mo>=</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mi>l</mi></msub><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo></mtd></mtr><mtr><mtd><msup><mi>f</mi><mo>′</mo></msup><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo></mtd><mtd><mi></mi><mo>=</mo><mi>t</mi><mi>a</mi><mi>n</mi><msup><mi>h</mi><mo>′</mo></msup><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo>=</mo><mn>1</mn><mo>−</mo><mi>t</mi><mi>a</mi><mi>n</mi><msup><mi>h</mi><mn>2</mn></msup><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo>=</mo><mn>1</mn><mo>−</mo><msubsup><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mi>l</mi><mn>2</mn></msubsup></mtd></mtr><mtr><mtd><msup><mi>f</mi><mo>′</mo></msup><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo></mtd><mtd><mi></mi><mo>=</mo><mi>s</mi><mi>o</mi><mi>f</mi><mi>t</mi><mi>m</mi><mi>a</mi><msup><mi>x</mi><mo>′</mo></msup><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo>=</mo><mi>s</mi><mi>o</mi><mi>f</mi><mi>t</mi><mi>m</mi><mi>a</mi><mi>x</mi><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo>−</mo><mi>s</mi><mi>o</mi><mi>f</mi><mi>t</mi><mi>m</mi><mi>a</mi><msup><mi>x</mi><mn>2</mn></msup><mo stretchy="false">(</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">u</mi></mrow><mi>l</mi></msub><mo stretchy="false">)</mo><mo>=</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mi>l</mi></msub><mo>−</mo><msubsup><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mi>l</mi><mn>2</mn></msubsup></mtd></mtr></mtable></math>

根据上述公式，可以得到各层参数的更新公式为：

<span class="MathJax" id="MathJax-Element-95-Frame" tabindex="0" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">W</mi></mrow><mi>l</mi></msub></mtd><mtd><mi></mi><mo>:=</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">W</mi></mrow><mi>l</mi></msub><mo>&#x2212;</mo><mi>&#x03B7;</mi><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">W</mi></mrow><mi>l</mi></msub></mrow></mfrac><mo>=</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">W</mi></mrow><mi>l</mi></msub><mo>&#x2212;</mo><mi>&#x03B7;</mi><msub><mrow class="MJX-TeXAtom-ORD"><mrow class="MJX-TeXAtom-ORD"><mo mathvariant="bold">&#x3B4;</mo></mrow></mrow><mi>l</mi></msub><msubsup><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>&#x2212;</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="normal">T</mi></mrow></msubsup></mtd></mtr><mtr><mtd><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">b</mi></mrow><mi>l</mi></msub></mtd><mtd><mi></mi><mo>:=</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">b</mi></mrow><mi>l</mi></msub><mo>&#x2212;</mo><mi>&#x03B7;</mi><mfrac><mrow><mi mathvariant="normal">&#x2202;</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">&#x2202;</mi><mi>b</mi></mrow></mfrac><mo>=</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">b</mi></mrow><mi>l</mi></msub><mo>&#x2212;</mo><mi>&#x03B7;</mi><msub><mrow class="MJX-TeXAtom-ORD"><mrow class="MJX-TeXAtom-ORD"><mo mathvariant="bold">&#x3B4;</mo></mrow></mrow><mi>l</mi></msub></mtd></mtr></mtable></math>" role="presentation" style="text-align: center; position: relative;"><nobr aria-hidden="true">Wlbl:=Wl−η∂E∂Wl=Wl−ηδlyTl−1:=bl−η∂E∂b=bl−ηδl</nobr><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mtable columnalign="right left right left right left right left right left right left" rowspacing="3pt" columnspacing="0em 2em 0em 2em 0em 2em 0em 2em 0em 2em 0em" displaystyle="true"><mtr><mtd><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">W</mi></mrow><mi>l</mi></msub></mtd><mtd><mi></mi><mo>:=</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">W</mi></mrow><mi>l</mi></msub><mo>−</mo><mi>η</mi><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">W</mi></mrow><mi>l</mi></msub></mrow></mfrac><mo>=</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">W</mi></mrow><mi>l</mi></msub><mo>−</mo><mi>η</mi><msub><mrow class="MJX-TeXAtom-ORD"><mrow class="MJX-TeXAtom-ORD"><mo mathvariant="bold">δ</mo></mrow></mrow><mi>l</mi></msub><msubsup><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">y</mi></mrow><mrow class="MJX-TeXAtom-ORD"><mi>l</mi><mo>−</mo><mn>1</mn></mrow><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="normal">T</mi></mrow></msubsup></mtd></mtr><mtr><mtd><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">b</mi></mrow><mi>l</mi></msub></mtd><mtd><mi></mi><mo>:=</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">b</mi></mrow><mi>l</mi></msub><mo>−</mo><mi>η</mi><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>E</mi></mrow><mrow><mi mathvariant="normal">∂</mi><mi>b</mi></mrow></mfrac><mo>=</mo><msub><mrow class="MJX-TeXAtom-ORD"><mi mathvariant="bold">b</mi></mrow><mi>l</mi></msub><mo>−</mo><mi>η</mi><msub><mrow class="MJX-TeXAtom-ORD"><mrow class="MJX-TeXAtom-ORD"><mo mathvariant="bold">δ</mo></mrow></mrow><mi>l</mi></msub></mtd></mtr></mtable></math>

References:

## 使用 nn 和 optim 来构建多层感知机

```python
import torch
from torch.autograd import Variable
import torch.nn as nn

# Hyper Parameters:
num_samples = 64                    # num_samples 是样本数量
input_size = 1000        # input_size是输入层大小
hidden_size = 100       # hidden_size是隐含层大小，
output_size = 10       # output_size是输出层大小

# 生成随机数当作样本，同时用Variable 来包装这些数据，设置 requires_grad=False 表示在方向传播的时候，
# 我们不需要求这几个 Variable 的导数
x = Variable(torch.randn(num_samples, input_size))
y = Variable(torch.randn(num_samples, output_size))

# 使用 nn 包的 Sequential 来快速构建模型，Sequential可以看成一个组件的容器。
# 它涵盖神经网络中的很多层，并将这些层组合在一起构成一个模型.
# 之后，我们输入的数据会按照这个Sequential的流程进行数据的传输，最后一层就是输出层。
# 默认会帮我们进行参数初始化
model = nn.Sequential(
    nn.Linear(input_size, hidden_size),
    nn.ReLU(),
    nn.Linear(hidden_size, output_size),
)

# 定义损失函数
loss_fn = nn.MSELoss(reduction='sum')

## 设置超参数 ##
learning_rate = 1e-4
num_epochs = 300

# 使用optim包来定义优化算法，可以自动的帮我们对模型的参数进行梯度更新。这里我们使用的是随机梯度下降法。
# 第一个传入的参数是告诉优化器，我们需要进行梯度更新的Variable 是哪些，
# 第二个参数就是学习速率了。
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

## 开始训练 ##
for epoch in range(num_epochs):

    # 向前传播
    y_pred= model(x)

    # 计算损失
    loss = loss_fn(y_pred, y)

    # 显示损失
    if (epoch+1) % 50 == 0:
        print('Epoch [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, loss.data))

    # 在我们进行梯度更新之前，先使用optimier对象提供的清除已经积累的梯度。
    optimizer.zero_grad()

    # 计算梯度
    loss.backward()

    # 更新梯度
    optimizer.step()

```

    Epoch [50/300], Loss: 36.2500
    Epoch [100/300], Loss: 2.0842
    Epoch [150/300], Loss: 0.2775
    Epoch [200/300], Loss: 0.0595
    Epoch [250/300], Loss: 0.0159
    Epoch [300/300], Loss: 0.0047

## 自定制 nn Modules:

很多时候，我们要建立的模型会比PyTorch现有的模型更加复杂，所以，在这个时候，我们需要自己定制自己的 nn.Module 这个时候，我们也需要定义 forward 方法，这个方法的输入是Variable，输出也是Variable。

定制一个自己的 nn Modules，其实就是在 init初始化函数中，将模型需要用到的层给定义好了。

然后重写forward()，在里面把数据在模型中流动的过程给写出来，就完成自己模型的定制了。

还是使用上面的多层感知机的例子。

**展示两种定义模型的写法：**

**1. 第一种**

```python
"""
1st
Author : Harrytsz
"""

import torch
from torch.autograd import Variable

class TwoLayerNet(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        """
        在构造器中，我们会实例化两个线性层，并且注意，下面这句
        super(TwoLayerNet, self).__init__() 千万别忘记了!
        """
        super(TwoLayerNet, self).__init__()
        self.linear1 = torch.nn.Linear(input_size, hidden_size)
        self.relu1 = torch.nn.ReLU()
        self.linear2 = torch.nn.Linear(hidden_size, output_size)

    def forward(self, x):
        Z1 = self.linear1(x)
        A1 = self.relu1(Z1)
        y_pred = self.linear2(A1)
        return y_pred

# Hyper Parameters:
num_samples = 64                    # num_samples 是样本数量
input_size = 1000        # input_size是输入层大小
hidden_size = 100       # hidden_size是隐含层大小，
output_size = 10       # output_size是输出层大小

# 生成随机数当作样本，同时用Variable 来包装这些数据，设置 requires_grad=False
# 表示在方向传播的时候，我们不需要求这几个 Variable 的导数
x = Variable(torch.randn(num_samples, input_size))
y = Variable(torch.randn(num_samples, output_size))

model = TwoLayerNet(input_size, hidden_size, output_size)

# 定义损失函数
loss_fn = nn.MSELoss(reduction='sum')

## 设置超参数 ##
learning_rate = 1e-4
num_epochs = 300

# 使用optim包来定义优化算法，可以自动的帮我们对模型的参数进行梯度更新。这里我们使用的是随机梯度下降法。
# 第一个传入的参数是告诉优化器，我们需要进行梯度更新的Variable 是哪些，
# 第二个参数就是学习速率了。
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

## 开始训练 ##
for epoch in range(num_epochs):

    # 向前传播
    y_pred= model(x)

    # 计算损失
    loss = loss_fn(y_pred, y)

    # 显示损失
    if (epoch+1) % 50 == 0:
        print('Epoch [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, loss.data))

    # 在我们进行梯度更新之前，先使用optimier对象提供的清除已经积累的梯度。
    optimizer.zero_grad()

    # 计算梯度
    loss.backward()

    # 更新梯度
    optimizer.step()

```

    Epoch [50/300], Loss: 46.4772
    Epoch [100/300], Loss: 3.4928
    Epoch [150/300], Loss: 0.4014
    Epoch [200/300], Loss: 0.0627
    Epoch [250/300], Loss: 0.0121
    Epoch [300/300], Loss: 0.0027

**2. 第二种**

```python
"""
2st
Author : Harrytsz
"""

import torch
from torch.autograd import Variable

class TwoLayerNet(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):

        super(TwoLayerNet, self).__init__()
        self.twolayernet = nn.Sequential(
            nn.Linear(input_size, hidden_size),
            nn.ReLU(),
            nn.Linear(hidden_size, output_size),
        )

    def forward(self, x):

        y_pred = self.twolayernet(x)
        return y_pred

# Hyper Parameters:
num_samples = 64                    # num_samples 是样本数量
input_size = 1000        # input_size是输入层大小
hidden_size = 100       # hidden_size是隐含层大小，
output_size = 10       # output_size是输出层大小

# 生成随机数当作样本，同时用Variable 来包装这些数据，设置 requires_grad=False 表示在方向传播的时候，
# 我们不需要求这几个 Variable 的导数
x = Variable(torch.randn(num_samples, input_size))
y = Variable(torch.randn(num_samples, output_size))

model = TwoLayerNet(input_size, hidden_size, output_size)

# 定义损失函数
loss_fn = nn.MSELoss(reduction='sum')

## 设置超参数 ##
learning_rate = 1e-4
num_epochs = 300

# 使用optim包来定义优化算法，可以自动的帮我们对模型的参数进行梯度更新。这里我们使用的是随机梯度下降法。
# 第一个传入的参数是告诉优化器，我们需要进行梯度更新的Variable 是哪些，
# 第二个参数就是学习速率了。
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

## 开始训练 ##
for epoch in range(num_epochs):

    # 向前传播
    y_pred= model(x)

    # 计算损失
    loss = loss_fn(y_pred, y)

    # 显示损失
    if (t+1) % 50 == 0:
        print(loss.data[0])

    # 在我们进行梯度更新之前，先使用optimier对象提供的清除已经积累的梯度。
    optimizer.zero_grad()

    # 计算梯度
    loss.backward()

    # 更新梯度
    optimizer.step()

```