# 误差反向传播法

作者：杨岱川

时间：2019年12月

github：https://github.com/DrDavidS/basic_Machine_Learning

开源协议：[MIT](https://github.com/DrDavidS/basic_Machine_Learning/blob/master/LICENSE)

参考文献：

- 《深度学习入门》，作者：斋藤康毅；
- 《深度学习》，作者：Ian Goodfellow 、Yoshua Bengio、Aaron Courville。
- [Keras overview](https://tensorflow.google.cn/guide/keras/overview)
- [PyTorch Tutorials](https://pytorch.org/tutorials/)
- [CS231n](http://cs231n.stanford.edu/)

## 本节目的

在[3.02 神经网络的训练](https://github.com/DrDavidS/basic_Machine_Learning/blob/master/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%9F%BA%E7%A1%80/3.02%20%E7%A5%9E%E7%BB%8F%E7%BD%91%E7%BB%9C%E7%9A%84%E8%AE%AD%E7%BB%83.ipynb)中，我们简单介绍了神经网络是如何学习的，然后通过数值微分的方法计算了神经网络的损失函数 $L$ 关于权重参数 $W$ 的梯度，也就是 $\cfrac{\partial L}{\partial W}$。数值微分的方式简单，实现方便，但是微分操作在计算效率上很低。所以我们现在会学习一个能够高效计算梯度的方法 —— **反向传播(back propagation)算法**。

为了更精确和清楚地描述反向传播法算法，我们会采用基于**计算图（computational graph）**的形式。计算图描述会更加地直观，然后结合代码加深理解。

## 计算图

计算图将计算过程用图的形式表现出来，这里的图形就是数据结构图，通过多个节点和边表示。为了熟悉计算图，我们先用计算图解一下简单的，然后从这些简单问题出发，逐步深入，最终抵达误差反向传播法。

### 使用计算图求解

首先我们使用一个非常简单的例子作为开始。这个例子即使心算也能解答，这里的目的主要是让大家熟悉计算图。明白计算图的原理后，我们会在后面的复杂计算中看到它所发挥的巨大作用。

>首先给大家讲一讲经济常识，**价外税**的概念。增值税是一种价外税，价外税税款不包含在商品价格内的税，它是按照税收与价格的组成关系对税收进行的分类。实际上，为了方便起见，我们买的绝大多数商品的价格已经包含了增值税，以现行的增值税税率，水果的税率在13%。

>**假定**：
>
>这里我们假定，有这么一家超市，超市里面商品的标价都是**税前价格**。
>
>为了方便计算，我们再假定增值税的税率为10%，不过请记住，现在的大多数商品增值税税率是13%，少部分商品是9%。
>
> **问题1**：我在超市买了 2个 税前原价 100块 一个的苹果，苹果的增值税是 10% ，请计算我实际应该付多少钱？

计算图通过节点和箭头来表示计算过程。节点中是计算的**内容**，比如加法或者乘法之类，计算的**中间结果**写在箭头的上方。

如上图所示，苹果的 100元 流动到了 “$\times2$” 节点，变成 200元，然后传递到下一个节点。然后这个 200元 流向 “$\times1.1$” 节点，变成 220元。因此，从这个计算图可以知道，我应该付 220元。

实际上 “$\times2$”和“$\times1.1$” 等节点作为一个运算整体用圆圈 $○$ 括起来了，不过我们也可以只用 $○$ 来表示乘法运算 $\times$。

如下图所示，我们可以将 “2” 和 “1.1” 分别作为变量 “苹果的个数” 和 “增值税” 标在 $○$ 外面：

再看下一题：

>**问题2**：我在超市买了 2个 税前原价 100块 一个的苹果，以及 3个 税前原件 150块 一个的橘子，增值税是 10% ，请计算我实际应该付多少钱？

同上面的问题，我们还是用计算图来解决问题2，过程如下图：

在这个问题中我们新增了加法节点 “$+$” ，用来计算苹果和橘子的合计金额。构建了计算图后，我们从左到右进行计算，就向电路中的电流流动，计算结果也从左边传递到了右边。

到达最右边的计算结果以后，计算过程就结束了，从图中可以知道，问题2的答案是715元。

综上两个简单例子，我们总结一下计算图解题的流程情况：

>1. 构建计算图
>
>2. 在计算图上，从左向右进行计算。

这里第2步 “从左向右进行计算” 正是我们之前学习过的**前向传播（forward propagation）**。前向传播也叫正向传播，是从计算图出发点到结束点的传播。既然有前向传播，自然我们会想到有反向的情况，从图上就是从右到左。这种传播就是**反向传播（backward propagation）**。反向传播会在导数计算中发挥重要作用。

### 局部计算

计算图的特点就是可以通过传递 **“局部计算”** 获得最终结果。“局部” 的意思就是 “与自己相关的某个小范围”，所以局部计算就是指，无论全局发了什么，都能只根据与自己相关的信息输出接下来的结果。

如上图所示，假设（经过了复杂的计算）我们购买了很多其他的东西，总共花了4000元。这里的重点在于，节点处的计算都是局部的计算，这表明，例如苹果和其他很多东西的求和运算 （$4000+200\to4200$）并不关心 4000 这个数字是怎么计算而来的，只要简单相加就行了。换言之，各个节点处只需要进行与自己有关的计算，不用考虑全局。

综上，计算图可以集中精力于局部计算。无论全局的计算有多么复杂，各个步骤要做的就是对象节点的局部计算。虽然局部计算简单，但是通过传播它的计算结果，可以获得全局的复杂计算的结果，就像汽车厂里面的流水制造线一样。

### 为何使用计算图

前面我们用计算图举了两个很简单的例子，解决了两个问题。那么计算图的优点到底有什么？

一是刚刚提到的局部计算，无论全局是多么复杂，都可以通过局部计算使得各个节点致力于简单的计算，从而简化问题。

二是利用计算图可以保存中间的计算结果，比如刚刚我们计算到苹果和橘子等的分别购买的金额。

三则是最重要的原因，可以**通过反向传播高效计算导数**。

回到最开始的**问题1**，我们一开始计算的是，购买了 2 个苹果加上增值税后最终需要支付的金额。如果我们想要知道，有一天苹果涨价了，那么会在多大程度上影响最终支付的金额呢？

这相当于求 “支付金额关于苹果价格的导数”。假设苹果的价格是 $x$，支付金额为 $L$，则相当于求 $\cfrac{\partial L}{\partial x}$，这个导数的值就表示当苹果价格上涨的时候，支付金额会变多少。

>这里的支付金额函数 $L$有 3 个变量，一个是苹果价格，设为 $x$，另一个是税率，设为 $t$，苹果的数量设为 $n$，故有
>
>$$\large L = n\times x \times t$$
>
>所以，如果想求苹果价格变化导致最终支付结果变化的幅度，求的是 $L$ 关于 $x$ 的偏导数。

如前所述， “支付金额关于苹果价格的导数”的值可以通过计算图的反向传播求出来，至于反向传播具体怎么求，稍后就会介绍。先看看结果：

如上图，反向传播以橙色箭头表示。反向传播传递“局部导数”，将导数的值写在箭头下方。

在这里，反向传播从右到左传递导数的值（$1\to 1.1\to 2.2$）。从这个结果可以知道 “支付金额关于苹果价格的导数”的值为 2.2。换句话说，苹果价格每上涨 1 元，最终支付的金额会多 2.2 元。

>再严格一点，如果苹果的价格增加一个微小值 $\Delta h$，那么最终支付的金额就会增加 $2.2\Delta h$。

这里我们求了 “支付金额关于苹果价格的导数”，同理我们也可以求 “支付金额关于苹果数量的导数”、“支付金额关于增值税的导数”，而且计算中途得到的导数结果（及中间传递的导数）可以被共享，从而可以高效地计算多个导数。

总结一下，计算图的优点就是可以通过正向传播和反向传播高效地计算各个变量的导数。

## 链式法则

前面的计算图主要讲的就是结果的前向传播，和我们平时接触的计算过程相符合，所以感觉上很自然。而反向传播将局部导数向正方向的反方向（从右到左）传递，可能会让人觉得困惑。

其实，传递这个局部导数的原理，是基于**链式法则（chain rule）**的，接下来我们会介绍链式法则，然后阐明其是如何对应计算图上的反向传播。

### 计算图的反向传播

首先看一个例子，假设存在一个 $y=f(x)$ 的计算，其反向传播如图：

如上图所示，反向传播的计算顺序是，将信号 $E$ 乘以节点的局部导数 $\cfrac{\partial y}{\partial x}$，然后将结果传递给下一个节点。这里所说的局部导数是指正向传播中 $y=f(x)$ 的导数，也就是 $y$ 关于 $x$ 的导数 $\cfrac{\partial y}{\partial x}$。

比如，假设 $y=f(x)=x^2$，则局部导数 $\cfrac{\partial y}{\partial x} = 2x$。把这个局部导数乘以上游传过来的值（这里是 $E$ ），然后传递给前面的节点。

这就是反向传播的过程。通过这样的计算，我们可以高效地求出导数的值，这是反向传播的要点。至于具体实现，我们需要从链式法则说起。

### 链式法则简介

首先，我们回顾一下**复合函数**。

复合函数是由多个函数构成的函数，比如 $z={(x+y)}^2$ 是由下面两个式子构成的：

$$
\large
\begin{equation}\begin{split}
z&=t^2\\
t&=x+y
\end{split} \end{equation}
 \tag{5.1}$$

链式法则是关于复合函数导数的性质，定义：

- 如果某个函数由复合函数表示，则该复合函数的导数可以用构成复合函数的各个函数的导数的乘积表示。

实际上这是一个非常简单的性质，以上面的函数为例，$\cfrac{\partial z}{\partial x}$（$z$ 关于 $x$ 的导数）可以用 $\cfrac{\partial z}{\partial t}$ （$z$ 关于 $t$ 的导数）和 $\cfrac{\partial t}{\partial x}$ （$t$ 关于 $x$ 的导数）的乘积来表示。

用数学式表示，就是：

$$\large \frac{\partial z}{\partial x}=\frac{\partial z}{\partial t}\frac{\partial t}{\partial x} \tag{5.2}$$

然后其中两个 $\partial t$ 相互消除，就成了

$$\large \require{cancel} \frac{\partial z}{\partial x}=\frac{\partial z}{\cancel {\partial t}}\frac{\cancel{\partial t}}{\partial x}$$

现在我们使用链式法则来求式 5.2 的 $\cfrac{\partial z}{\partial x}$，首先我们要求偏导：

$$
\large
\begin{equation}\begin{split}
\frac{\partial z}{\partial t} &= 2t\\
\frac{\partial t}{\partial x} &= 1
\end{split} \end{equation}
 \tag{5.3}$$

如式 5.3 所示，$\cfrac{\partial z}{\partial t}$ 等于 $2t$，$\cfrac{\partial t}{\partial x}$ 等于 $1$。这是基于导数公式的解析解。然后最后要计算的$\cfrac{\partial z}{\partial x}$ 可以根据链式法则，计算导数乘积而得：

$$\large \frac{\partial z}{\partial x}=\frac{\partial z}{\partial t}\frac{\partial t}{\partial x}=2t\cdot1=2(x+y) \tag{5.4}$$

### 链式法则和计算图

现在我们尝试将式 5.4 的链式法则的计算用计算图来表示，其中平方运算我们采用 “$**2$” 代替：

如图所示，计算图的反向传播从右到传播信号。反向传播的顺序是，先将节点的输入信号乘以节点的局部导数（偏导数），然后再传递给下一个节点。

比如，反向传播的时候，“$**2$” 节点的输入是 $\cfrac{\partial z}{\partial z}$，将其乘以局部导数 $\cfrac{\partial z}{\partial t}$（因为正向传播时候，输入是 $t$，输出是 $z$，所以这个节点的局部导数是 $\cfrac{\partial z}{\partial t}$ ），然后传递给下一个节点。

另外，上图中反向传播最开始的信号  $\cfrac{\partial z}{\partial z}$ 在前面的数学式中是没有出现的，因为  $\cfrac{\partial z}{\partial z}=1$，所以省略了。

图中还需要注意的是最左边的反向传播的结果。根据链式法则，$\cfrac{\partial z}{\partial z}\cfrac{\partial z}{\partial t}\cfrac{\partial t}{\partial x}=\cfrac{\partial z}{\partial t}\cfrac{\partial t}{\partial x}=\cfrac{\partial z}{\partial x}$ 成立，对应 “$z$ 关于 $x$ 的导数”。也就是说，反向传播是基于链式法则的。

把式 5.3 的结果代入上图中，结果如下图所示， $\cfrac{\partial z}{\partial x}$ 的结果为 $2(x+y)$。

## 反向传播

之前我们介绍了计算图的反向传播是基于链式法则成立的，现在我们将以 “$+$” 和 “$\times$” 等运算为例，介绍反向传播的结构。

### 加法节点的反向传播

首先我们来看看加法节点的反向传播。

以 $z=x+y$ 为对象，观察其反向传播。 $z=x+y$ 的导数可由下式计算：

$$
\large
\begin{equation}\begin{split}
\frac{\partial z}{\partial x}&=1\\[2ex]
\frac{\partial z}{\partial y}&=1
\end{split} \end{equation}
 \tag{5.5}$$

如式 5.5 所示，$\cfrac{\partial z}{\partial x}$ 和 $\cfrac{\partial z}{\partial y}$ 同时都等于1，因此用计算图表示，就如下图所示：

在上图中，反向传播将从上游传过来的导数（这里是 $\cfrac{\partial L}{\partial z}$）乘以 $1$，然后传向下游。也就是说，因为加法节点的反向传播只乘以 $1$，所以输入的值会原封不动地流向下一个节点。

至于本例中，我们把上游传过来的导数的值设为 $\cfrac{\partial L}{\partial z}$，这是因为，我们假定了一个最终输出值为 $L$ 的大型计算图，如下：

$z=x+y$ 的计算位于这个大型计算图的某个地方，从上游会传来 $\cfrac{\partial L}{\partial z}$ 的值，并向下游传递 $\cfrac{\partial L}{\partial x}$ 和 $\cfrac{\partial L}{\partial y}$。

举个例子，假设有“$10+5=15$”这样一个计算，且反向传播时，会从上游传来值 $1.3$（注意这个 $1.3$ 是我们假设的上游值）。用计算图表示，如下图

因为加法结点的反向传播只是将信号输出到下一个结点，所以如上图，反向传播将 $1.3$ 向下一个结点传递。

### 乘法节点的反向传播

加法之后是乘法，我们看看乘法节点的反向传播。我们考虑 $z=xy$。这个式子用导数表示就是：

$$
\large
\begin{equation}\begin{split}
\frac{\partial z}{\partial x}&=y\\[2ex]
\frac{\partial z}{\partial y}&=x
\end{split} \end{equation}
 \tag{5.6}$$

根据式 5.6，我们可以画出计算图。

乘法的反向传播会将上游的值乘以正向传播时的输入信号的“翻转值”后传递给下游。

翻转值表示一种翻转关系，如图所示，正向传播的信号是 $x$ 的话，反向传播时候就是 $y$；正向传播时候信号是 $y$ 的话，反向传播时就是 $x$。

现在来看一个具体例子，比如有 “$10\times 5 = 50$” 这样一个计算，反向传播时候，上游传来一个值 $1.3$。用计算图表示为：

由于乘法的反向传播会乘以输入信号的翻转值，所以各自按照 $1.3\times5=6.5$、$1.3\times10=13$ 计算。另外，加法的反向传播并不需要正向传播的输入信号。但是，乘法的反向传播需要正向传播的输入信号值（这里是 5 和 10），因此，实现乘法节点的反向传播时，需要保存正向传播的输入信号。

## 简单层的实现

回顾一下之前购买苹果的例子，我们将用 Python 来实现这个例子。这里，我们把要实现的计算图乘法节点称为“乘法层（MulLayer）”，加法节点称为“加法层（AddLayer）”。

### 乘法层的实现

每个层的实现都应该有两个共通的方法：**forward()** 和 **backward()**。forward() 对应前向传播，而 backward() 对应反向传播。

现在实现乘法层。乘法层作为 MulLayer 类，实现过程如下：

```python
class MulLayer:
    """乘法层"""
    def __init__(self):
        """初始化"""
        self.x = None
        self.y = None

    def forward(self, x, y):
        """前向传播"""
        self.x = x
        self.y = y
        out = x * y
        return out

    def backward(self, dout):
        """反向传播"""
        dx = dout * self.y  # 翻转x和y
        dy = dout * self.x
        return dx, dy
```

在上述代码中，`__init__()` 会初始化实例变量 $x$ 和 $y$，它们用于保存正向传播时的输入值。`forward()` 接收 $x$ 和 $y$ 两个参数，将它们相乘后输出。`backward()` 将从上游传过来的导数（dout）乘以前向传播的翻转值，然后传给下游。

现在我们可以使用 `MulLayer` 来实现前面购买苹果的例子（两个苹果和增值税）。

使用这个乘法层，上图的正向传播可以如下实现：

```python
apple = 100
apple_num = 2
tax = 1.1
```

```python
# layer
mul_apple_layer = MulLayer()
mul_tax_layer = MulLayer()

# forward
apple_price = mul_apple_layer.forward(apple, apple_num)
price = mul_tax_layer.forward(apple_price, tax)

print("price:", int(price))  # 220
```

    price: 220

此外，关于各个变量的导数可以由 `backward()` 求出：

```python
# backward
dprice = 1
dapple_price, dtax = mul_tax_layer.backward(dprice)
dapple, dapple_num = mul_apple_layer.backward(dapple_price)

print("price:", int(price))
print("dApple:", dapple)
print("dApple_num:", int(dapple_num))
print("dTax:", dtax)
```

    price: 220
    dApple: 2.2
    dApple_num: 110
    dTax: 200

可以看出，运行结果和上面的图中的结果是一样的。

>注意，`backward()` 的参数中需要输入 “关于正向传播时的输出变量的导数”。比如，`mul_apple_layer` 乘法层在正向传播时会输出 apple_price，在反向传播时，则会将 `apple_price` 的导数 `dapple_price` 设为参数。

### 加法层的实现

现在我们来看看加法层是如何实现的，首先看看代码：

```python
class AddLayer:
    """加法层"""
    def __init__(self):
        """
        初始化

        这里的 pass 指的是，加法层不需要保存正向传播的输入信号
        """
        pass

    def forward(self, x, y):
        """前向传播"""
        out = x + y
        return out

    def backward(self, dout):
        """反向传播"""
        dx = dout * 1
        dy = dout * 1
        return dx, dy
```

加法层不需要特意初始化，所以 `__init__()` 中什么也不运行。

加法层的 `forward()` 接收 $x$ 和 $y$ 两个参数，将他们相加后输出。而 `backward()`，我们在前面已经讨论过了，它将上游传来的导数原封不动地传给下游。

有兴趣的同学可以参考《深度学习入门》第 5.4.2 节中的代码实现简单的加法和乘法层的组合，代码参考[这里](https://github.com/oreilly-japan/deep-learning-from-scratch/blob/master/ch05/buy_apple_orange.py)。

## 激活函数层的实现

除了简单的加法和乘法以外，在[3.1 神经网络与前向传播](https://github.com/DrDavidS/basic_Machine_Learning/blob/master/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%9F%BA%E7%A1%80/3.01%20%E7%A5%9E%E7%BB%8F%E7%BD%91%E7%BB%9C%E4%B8%8E%E5%89%8D%E5%90%91%E4%BC%A0%E6%92%AD.ipynb) 中我们还学习过**激活函数**，所以现在我们要实现激活函数的反向传播。

应用计算图的思路，和之前对加法层、乘法层的实现，我们也把神经网络的其他部分实现为一个**类（class）**，这里首先我们来试试实现激活函数的 **ReLU** 层和 **Sigmoid** 层。

### ReLU层

回忆一下激活函数 ReLU(Rectified Linear Unit) 的表示形式：

$$
\large
y =
\begin{cases}
x, & (x \gt 0)  \\
0, & (x \le 0)
\end{cases} \tag{5.7}
$$

通过式 5.7 我们可以求出 $y$ 关于 $x$ 的导数，如式 5.8：

$$
\large
\frac{\partial y}{\partial x} =
\begin{cases}
1, & (x \gt 0)  \\
0, & (x \le 0)
\end{cases} \tag{5.8}
$$

在式 5.8 中，如果前向传播时的输入 $x$ 大于 0 ，则反向传播会将上游的值原封不动地传给下游。反过来，如果前向传播时的 $x$ 小于等于 0，则反向传播中传给下游的信号将停在此处。

用计算图的话，如下：

现在我们来实现 ReLU 层。在神经网络的层的实现中，一般假定 `forward()` 和 `backward()` 的参数是 NumPy 数组。

```python
class Relu:
    """ReLU 激活函数"""
    def __init__(self):
        self.mask = None

    def forward(self, x):
        self.mask = (x <= 0)
        out = x.copy()
        out[self.mask] = 0
        return out

    def backward(self, dout):
        dout[self.mask] = 0
        dx = dout
        return dx
```

其中，`mask` 的作用如下:

```python
x = -1
mask = (x <= 0)
print(mask)
```

    True

```python
import numpy as np

x = np.array([[1, 2, 3], [-1, -2, -3]])
mask = (x <= 0)
print(mask)
```

    [[False False False]
     [ True  True  True]]

`Relu` 类有实例变量 `mask`。这个变量 `mask` 是由 `True` \\ `False` 构成的 NumPy 数组，它会把正向传播时的输入 `x` 的元素中小于等于 0 的地方保存为 `True`，其他地方保存为 `False`

如上面的图所示，如果正向传播时的输入值小于等于 0 ，则反向传播的值为 0。因此，反向传播中会使用正向传播时保存的 mask，将从上游传来的 dout 的 mask 中的元素为 `True` 的地方设为 0。

### Sigmoid层

接下来我们实现 sigmoid 函数。 回顾一下 sigmoid 函数：

$$\large y=\frac{1}{1+e^{-x}}\tag{5.9}$$

用计算图表示式 5.9，如下图：

上图中，除了 “$\times$” 和 “$+$” 节点外，还出现了新的 “$\exp$” 和 “$/$”节点。

“$\exp$” 节点会进行 $y=\exp(x)$ 的运算，即 $y=e^{-x}$，而 “$/$” 节点会进行 $y=\cfrac{1}{x}$ 的计算。

sigmoid的前向传播图由较为复杂的局部计算构成，因此我们依次来看看反向传播的流程。

#### 步骤1

“$/$” 节点表示 $y=\cfrac{1}{x}$ ，它的导数可以解析地表示为：

$$\large \frac{\partial y}{\partial x}=-\frac{1}{x^2}=-y^2 \tag{5.10}$$

根据式 5.10 ，反向传播时，会将上游的值乘以 $-y^2$ （即正向传播的输出的平方乘以 $-1$）后，再传给下游。计算图如下：

#### 步骤2

“$+$” 节点将上游的值原封不动传给下游。如下图所示：

#### 步骤3

“$\exp$” 节点表示 $y=e^{x}$，它的导数表示为：

$$\large \frac{\partial y}{\partial x}=e^x\tag{5.11}$$

在计算图中，上游的值乘以正向传播时的输出（本例中是 $e^{-x}$）后，再传给下游：

> 注意，这里的 $e^x$ 求导之后是它本身，所以反向传播时候就是上游的导数值（比如 $f'(x)$）乘以正向的输出 $e^x$。

#### 步骤4

“$\times$” 节点将正向传播时的值翻转后做乘法运算。因此这里要乘以 $-1$：

根据上述描述，步骤 4 中的计算图就可以进行 Sigmoid 层的反向传播。从图中可以知道，反向传播的输出为 $\cfrac{\partial L}{\partial y}y^2e^{-x}$，这个值会传播给下游的节点。

注意，$\cfrac{\partial L}{\partial y}y^2e^{-x}$ 这个值只根据正向传播时的输入 $x$ 和输出 $y$ 就可以算出来，因此我们可以把 “sigmoid” 节点集约化一下：

上图就是简洁版本的计算图，我们可以省略中间的反向传播过程，计算效率更高。

> 通过对节点的“集约化”，可以不用在意 Sigmoid 层中琐碎的细节，只需要专注它的输入和输出，这一点很重要。

此外，$\cfrac{\partial L}{\partial y}y^2e^{-x}$ 还可以进一步整理：

$$
\large
\begin{equation}\begin{split}
\frac{\partial L}{\partial y}y^2e^{-x}&=\frac{\partial L}{\partial y}\frac{1}{\left(1+e^{-x}\right)^2}e^{-x} \\
&=\frac{\partial L}{\partial y}\frac{1}{1+e^{-x}}\frac{e^{-x}}{1+e^{-x}}\\
&=\frac{\partial L}{\partial y}y(1-y)
\end{split}\end{equation}\tag{5.12}
$$

现在，由式 5.12 可得，Sigmoid 层的反向传播只根据正向传播的输出 $y$ 就能计算出来：

现在我们使用 Python 代码对 Sigmoid 层进行实现：

```python
class Sigmoid:
    """Sigmoid层"""
    def __init__(self):
        """初始化"""
        self.out = None

    def forward(self, x):
        """前向传播"""
        out = 1 / (1 + np.exp(-x))
        self.out = out
        return out

    def backward(self, dout):
        """
        反向传播

        Keyword arguments:
        dout -- 上游的导数
        """
        dx = dout * (1.0 - self.out) * self.out
        return dx
```

## Affine / Softmax 层的实现

### Affine 层

Affine层就是全连接层，又叫仿射层。

神经网络的正向传播中，为了计算加权信号的总和，使用了矩阵的乘积运算（NumPy 中的 `np.dot()` 函数）。

在之前神经网络的前向传播中，我们曾经实现过最简单的部分，回顾一下：

```python
import numpy as np

X = np.random.rand(2)     # 输入
# X = np.asmatrix(X)
W = np.random.rand(2, 3)  # 权重
# W = np.asmatrix(W)
B = np.random.rand(3)     # 偏置
# B = np.asmatrix(B)
```

```python
print(f"X：{X} ,\nX.shape：{X.shape}\n")
print(f"W：{W} ,\nW.shape：{W.shape}\n")
print(f"B：{B} ,\nB.shape：{B.shape}\n")
```

    X：[0.95099499 0.86236951] ,
    X.shape：(2,)

    W：[[0.27628981 0.77139771 0.24660224]
     [0.32572545 0.7945617  0.13803012]] ,
    W.shape：(2, 3)

    B：[0.62884398 0.17308075 0.38642165] ,
    B.shape：(3,)

```python
Y = np.dot(X, W) + B
print(Y)
```

    [1.1724899  1.59188189 0.7399721 ]

这里，$\textbf{X}$、$\textbf{W}$、$\textbf{B}$ 分别是形状为 $(2,)$、$(2,3)$、$(3,)$ 的多维数组。

>如果觉得逗号后面空一格难受，可以使用 `np.asmatrix()` 把多维数组转换为矩阵。

所以神经元的加权和可以用

```python
Y = np.dot(X, W) + B
```

算出来。然后 $\textbf{Y}$ 经过激活函数的转换后，传递给下一层。这就是神经网络正向传播的流程。

现在将这里进行的求矩阵的乘积与偏置的和的运算用计算图表示出来。将乘积运算用 “dot” 节点表示的话，则 `np.dot(X, W) + B` 的运算可以用下图所示计算图表示出来。

上图是比较简单的计算图，要注意的是 $\textbf{X}$、$\textbf{W}$、$\textbf{B}$ 是**矩阵**（多维数组）。而之前我们见到的计算图中流动的都是标量。

现在我们来考虑一下如何实现上图的反向传播。以矩阵为对象的反向传播，按矩阵的各个元素进行计算时，步骤与以标量为对象的计算图一致。这里省略了推导（可以参考后面的计算图的反向传播），直接给出结果如下：

$$
\large
\begin{equation}\begin{split}
\frac{\partial L}{\partial \textbf{X}}&=\frac{\partial L}{\partial \textbf{Y}}\cdot {\textbf{W}}^{\rm T}\\[2ex]
\frac{\partial L}{\partial \textbf{W}}&={\textbf{X}}^{\rm T} \cdot \frac{\partial L}{\partial \textbf{Y}}
\end{split} \end{equation}
 \tag{5.13}$$

其中 ${\textbf{W}}^{\rm T}$ 是 ${\textbf{W}}$ 的转置矩阵。

实际上我们可以试着画出计算图的反向传播过程：

我们看看上面计算图中各个变量的形状。注意，$\textbf{X}$ 和 $\cfrac{\partial L}{\partial \textbf{X}}$ 形状相同，$\textbf{W}$ 和 $\cfrac{\partial L}{\partial \textbf{W}}$ 形状相同。参考如下：

$$
\large
\begin{equation}\begin{split}
\textbf{X} &=(x_0,x_1,\cdots,x_n)\\[2ex]
\frac{\partial L}{\partial \textbf{X}} &=\left( \frac{\partial L}{\partial x_0},\frac{\partial L}{\partial x_1},\cdots,\frac{\partial L}{\partial x_n} \right)
\end{split} \end{equation}
 \tag{5.15}$$

为什么要注意矩阵形状呢？

因为矩阵的乘积运算要求对应元素个数保持一致，通过确认一致性，可以导出式 5.13。 比如 $\cfrac{\partial L}{\partial \textbf{Y}}$ 的形状是 $(3,)$ ，${\textbf{W}}$ 的形状是 $(2,3)$ 时，思考 $\cfrac{\partial L}{\partial \textbf{Y}}$ 和 ${\textbf{W}}^{\rm T}$ 的乘积，使得 $\cfrac{\partial L}{\partial \textbf{X}}$ 的形状为 $(2,)$。

这样一来，自然而然推导出式 5.13。

### 批版本的Affine层

前面介绍的 Affine 层的输入 $X$ 是以单个数据为对象的。实际上我们的数据传入通常是一批一批（batch）传入，现在我们考虑 $N$ 个数据一起进行正向传播的情况，也就是批版本的 Affine 层。

批版本的 Affine 层计算图如下：

和刚刚不同的是，现在 $\textbf{X}$ 的输入形状是 $(N,2)$。之后和前面一样，在计算图上进行矩阵计算。反向传播时，注意矩阵的形状，就能和前面一样推导出 $\cfrac{\partial L}{\partial \textbf{X}}$ 和 $\cfrac{\partial L}{\partial \textbf{W}}$。

加上**偏置**时候要注意！前向传播时，偏置是被加到了 $\textbf{X}\cdot\textbf{W}$ 的每一个数据上（可以回顾一下前向传播的内容）。因此反向传播时，各个数据的反向传播的值需要汇总为偏置的元素。

参考代码:

```python
dY = np.matrix([[1, 2, 3], [4, 5, 6]])
print(dY)
```

    [[1 2 3]
     [4 5 6]]

```python
# 加起来
dB = np.sum(dY, axis=0)
print(dB)
```

    [[5 7 9]]

这个例子中，假定数据有 2 个（$N=2$）。偏置的反向传播会对这 2 个数据的导数按元素进行求和。因此，这里使用了 `np.sum()` 对第 0 轴（以数据为单位的轴，`axis=0`）方向上的元素进行求和。

综上所述，Affine 的实现如下：

```python
class Affine:
    """仿射层"""
    def __init__(self, W, b):
        """初始化"""
        self.W = W
        self.b = b
        self.x = None
        self.dw = None
        self.db = None

    def forward(self, x):
        """前向传播"""
        self.x = x
        out = np.dot(x, self.W) + self.b
        return out

    def backward(self, dout):
        dx = np.dot(dout, self.W.T)
        self.dW = np.dot(self.x.T, dout)
        self.db = np.sum(dout, axis=0)
        return dx
```

### Softmax-with-Loss 层

最后简单介绍一下输出层的 softmax 函数。我们在 [3.02 神经网络的训练](https://github.com/DrDavidS/basic_Machine_Learning/blob/master/03深度学习基础/3.02%20神经网络的训练.ipynb) 中学习了 softmax 函数，知道它会将输入值正规化之后再输出，而每一类输出所加起来的总和为 1。

softmax 层一般包含了作为损失函数的交叉熵误差（cross entropy error），所以称为“Softmax-with-Loss层”。

>Softmax-with-loss 层较为复杂，其推导过程可以参考 《深度学习入门》附录A 或者网上的一些文章，比如[神经网络多分类中softmax+cross-entropy的前向传播和反向传播过程](https://zhuanlan.zhihu.com/p/86184547)。

这里给出一个简化版本的计算图：

如上图所示，softmax 函数记为 Softmax 层，交叉熵误差集为 Cross Entropy Error 层。这里假设进行 3 分类，从前面的层接收 3 个输入。

Softmax 层将输入 $(a_1,a_2,a_3)$ 正规化，输出 $(y_1,y_2,y_3)$。 Cross Entropy Error 层接收 Softmax 的输出 $(y_1,y_2,y_3)$ 和监督标签 $(t_1,t_2,t_3)$，从这些数据中输出损失 $L$。

这里要注意的是反向传播的结果。Softmax 层的反向传播得到了 $(y_1-t_1,y_2-t_2,y_3-t_3)$ 这样的整齐漂亮结果。由于 $(y_1,y_2,y_3)$ 是 Softmax 层的输出，$(t_1,t_2,t_3)$ 是监督数据，所以 $(y_1-t_1,y_2-t_2,y_3-t_3)$ 是 Softmax 层的输出和监督标签的差分。神经网络的反向传播会把这个差分表示的误差传递给前面的层，这是神经网络学习中的重要性质。

神经网络学习的目的就是通过调整权重参数，使神经网络的输出（Softmax的输出）接近监督标签。因此，必须将神经网络的输出与监督标签的误差高效地传递给前面的层。刚刚的 $(y_1-t_1,y_2-t_2,y_3-t_3)$ 正是 Softmax 层与监督标签的差，直接了当地表示了当前神经网络地输出与监督标签的误差。

现在来进行 Softmax-with-Loss 层的实现，过程如下：

```python
def softmax(a):
    """定义 softmax 函数"""
    exp_a = np.exp(a)
    sum_exp_a = np.sum(exp_a)
    y = exp_a / sum_exp_a
    return y

def cross_entropy_error(y, t):
    """定义交叉熵损失函数"""
    delta = 1e-7
    return -np.sum(t * np.log(y + delta))

class SoftmaxWithLoss:
    """SoftmaxWithLoss"""
    def __init__(self, W, b):
        """初始化"""
        self.loss = None  # 损失
        self.y = None     # softmax的输出
        self.t = None     # 监督数据（one hot形式）

    def forward(self, x):
        """前向传播"""
        self.t = t
        self.y = softmax(x)
        self.loss = cross_entropy_error(self.y, self.t)
        return self.loss

    def backward(self, dout):
        batch_size = self.t.shape[0]
        dx = (self.y - self.t) / batch_size
        return dx
```

其中，`softmax()` 和 `cross_entropy_error()` 函数都是 [3.02 神经网络的训练](https://github.com/DrDavidS/basic_Machine_Learning/blob/master/03深度学习基础/3.02%20神经网络的训练.ipynb) 中照搬过来的。所以这里的实现很简单。

## 误差反向传播的实现

### 神经网络学习的全貌图

我们在这里回顾一下神经网络学习的全貌图。神经网络学习的步骤如下所示：

- **前提**

    神经网络中有合适的权重和偏置，调整权重和偏置以拟合训练数据的过程叫做学习。神经网络的学习分为以下四个步骤。

- **步骤一（mini-batch）**

    从训练数据中随机选择一部分数据，大小为 batch-size。

- **步骤二（计算梯度）**

    计算损失函数关于各个可训练参数（权重和偏置）的梯度。

- **步骤三（更新参数）**

    将可训练参数沿着梯度方向进行微小的更新。

- **步骤四（重复）**

    重复步骤一、步骤二、步骤三，直到一定次数，或者损失不再下降为止。

之前介绍过的误差反向传播法会在步骤2中出现。在 [3.02 神经网络的训练](https://github.com/DrDavidS/basic_Machine_Learning/blob/master/03深度学习基础/3.02%20神经网络的训练.ipynb) 中，我们利用数值微分的方法求得这个梯度。

数值微分实现虽然简单，但是计算耗费时间非常多，但是误差反向传播法可以高效快速地计算梯度，因为刚刚我们都是通过计算图求的解析解。

### 误差反向传播法的梯度确认

到目前为止，我们介绍了两种求梯度的方法，一种是基于**数值微分**的方法，另一种是解析性地求解数学式的方法。后一种方法通过使用**误差反向传播法**，即使存在大量的参数，也可以高效地计算梯度。因此在实际编程中，我们通常使用误差反向传播法。

数值微分的计算很耗费时间，而且如果有误差反向传播法的实现的话，就没有必要使用数值微分实现了。那么数值微分有什么用呢？

实际上，在确认误差反向传播法的实现是否正确时，需要使用数值微分。

数值微分的优点是实现简单，因此，在一般情况下不太容易出错。而误差反向传播法的实现很复杂，在编程实现时相对更容易出错，所以我们经常会比较数值微分和误差反向传播法的结果，以确认误差反向传播法的实现是否正确。

确认数值微分所求出的梯度的结果和误差反向传播法求出的的结果是否一致（非常非常接近，不一定完全一致）的操作叫做 **梯度确认（gradient check）**，这里只要求明白其涵义，而具体实现可以参考[gradient_check.py](https://github.com/oreilly-japan/deep-learning-from-scratch/blob/master/ch05/gradient_check.py)。

## 小结

这次学习中我们介绍了将计算过程可视化的**计算图**，并且使用计算图，介绍了神经网络中的**误差反向传播法**，然后以层（layer）为单位实现了神经网络中的处理。

我们学过的层有：

- ReLU
- Softmax-with-Loss
- Affine
- Softmax

等，并且在其中通过计算图解释了其**前向传播**和**反向传播**的原理，实现了 `forward()` 和 `backward()` 方法。

通过数据的前向传播和反向传播，我们可以高效地计算权重参数地梯度。然后通过对层的模块化实现，我们可以自由地组装层，轻松构建出自己想要的网络结构。

```python

```

03深度学习基础/3.03 误差反向传播法.ipynb

03深度学习基础/3.03 误差反向传播法.ipynb

03深度学习基础/3.03 误差反向传播法.ipynb


```python
import numpy as np
from sklearn.metrics import confusion_matrix
import pickle
import random
from copy import deepcopy
import itertools
import datetime
import time
import json
import torch
import torch.nn as nn
import torch.optim as optim
from torch.autograd import Variable
from torch.optim.lr_scheduler import LambdaLR
from sympy import symbols
import pprint as pp

import sys, os
sys.path.append(os.path.join(os.path.dirname("__file__"), '..', '..'))
from AI_physicist.theory_learning.models import Loss_Fun_Cumu
from AI_physicist.theory_learning.theory_model import Theory_Training, get_loss, get_best_model_idx, get_preds_valid, get_valid_prob
from AI_physicist.theory_learning.theory_hub import Theory_Hub, load_model_dict_at_theory_hub
from AI_physicist.theory_learning.util_theory import plot_theories, plot3D, process_loss, plot_loss_record, to_one_hot, plot_indi_domain, get_mystery, to_pixels, standardize_symbolic_expression, check_expression_matching
from AI_physicist.theory_learning.util_theory import get_epochs, export_csv_with_domain_prediction, extract_target_expression, get_coeff_learned_list, load_theory_training, get_dataset, get_piecewise_dataset
from AI_physicist.settings.filepath import theory_PATH
from AI_physicist.settings.global_param import COLOR_LIST
from AI_physicist.pytorch_net.util import Loss_Fun, make_dir, Loss_with_uncertainty, Early_Stopping, record_data, plot_matrices, get_args, serialize, to_string, filter_filename, to_np_array, to_Variable
from AI_physicist.pytorch_net.net import Conv_Autoencoder, load_model_dict, train_simple

try:
    %matplotlib inline
    isplot = True
except:
    import matplotlib
    matplotlib.use('Agg')
    isplot = False
is_cuda = torch.cuda.is_available()
standardize = standardize_symbolic_expression
```

## Dataset loading and important settings:

```python
########################################
# Setting up path to dataset files. The data files are csv_dirname + env_name + ".csv",
# where env_name are elements in the csv_filename_list. For each csv file, the first
# (num_output_dims * num_input_steps) columns are the past (E.g., if num_output_dims=2, then
# they are arranged as (x_{t-num_input_steps}, y_{t-num_input_steps}, ... x_{t-1}, y_{t-1}) ).
# The next num_output_dims columns are the target prediction for the future.
# If is_classified = True, the last column in the csv should provide the true_domain id for evaluating
# whether the domain prediction is correct (not used for training)
########################################
csv_dirname = "../datasets/GROUND_TRUTH/"  # Path for the dataset file
is_classified = True             # If True, the last column in the csv file should provide the true_domain id for evaluation.
csv_filename_list = get_mystery(50000, range(4,7), range(1, 6), is_classified) + get_mystery(50000, [20], range(1, 6), is_classified) + get_mystery(50000, range(7, 11), range(1, 6), is_classified)
print("Environments:\n", csv_filename_list)
num_output_dims = 2               # It sets the dimension of output
num_input_steps = 2               # It sets the number of steps for the input

########################################
# Other important settings:
########################################
exp_mode = "continuous"           # Choose from "continuous" (full AI Physicist), "newb" (newborn) and "base" (baseline)
forward_steps = 1                 # Number of forward steps to predict
data_format = "states"            # Choose from "states" or "images"
pred_nets_activation = "linear"   # Activation for the prediction function f. Choose from "linear", "leakyRelu"
num_layers = 3                    # Number of layers for the prediction function f.

num_theories_init = 4             # Number of theories to start with.
add_theory_loss_threshold = 2e-6  # MSE threshold for individual data points above which to add a new theory to fit.
add_theory_criteria = ("loss_with_domain", 0)  # Criteria and threshold of loss increase to determine whether to accept adding the theory.
add_theory_quota = 1              # maximum number of theories to add at each phase.
add_theory_limit = None           # maximum allowed number of theories. If None, do not set limit.

is_Lagrangian = False             # If True, learn the Lagrangian. If False, learn Equation of Motion (EOM).
load_previous = True              # Whether to load previously trained instances on
```

## Settings up:

```python
# Other settings:
exp_id = "exp1.0"
env_source = "file"
pred_nets_neurons = 8
domain_net_neurons = 8
domain_pred_mode = "onehot"
mse_amp = 1e-7
scheduler_settings = ("ReduceLROnPlateau", 40, 0.1)   # Settings for the learning rate scheduler
# scheduler_settings = ("LambdaLR", "exp", 2, False)    # Settings for the learning rate scheduler
simplify_criteria = ("DLs", 0, 3, "relative") # The (criteria type, threshold, patience, compare_mode) upon which not satisfied, we break the current simplification and continue to the next layer/model
optim_type = ("adam", 5e-3)
optim_domain_type = ("adam", 1e-3)
optim_autoencoder_type = ("adam", 1e-5, 1e-1) # optim_type, lr, loss_scale
reg_mode = "L1"
reg_amp = 1e-8
reg_smooth = None
reg_domain_mode = "L1"
reg_domain_amp = 1e-5
batch_size = 10000
loss_core = "DLs"
loss_order = -1
loss_decay_scale = None
is_mse_decay = False
num_examples = 20000
epochs = 10000
iter_to_saturation = int(epochs / 2)
MDL_mode = "both"
date_time = "{0}-{1}".format(datetime.datetime.now().month, datetime.datetime.now().day)
seed = 0
array_id = "0"

loss_balance_model_influence = False
loss_success_threshold = 1e-4   # MSE level you regard as success
theory_add_mse_threshold = 0.05 # MSE level below which you will add to the theory hub
theory_remove_fraction_threshold = 0.005  # Fraction threshold below which you will remove a theory after each stage of training.
matching_numerical_tolerance = 2e-4 # The tolerance below which you regard the numerical coefficient matches.
matching_snapped_tolerance = 1e-9   # The tolerance below which you regard the snapped coefficient matches.
max_trial_times = 1         # Maximum number of trial times before going on to next target (DEFAULT=1)
is_simplify_model = True    # Whether to perform simplification of theory models
is_simplify_domain = False  # Whether to perform simplification of theory domains
record_mode = 2             # Record data mode. Choose from 0 (minimal recording), 1, 2 (record everything)
show_3D_plot = False
show_vs = False

big_domain_dict = [(key, [1, 2]) for key in get_mystery([20000, 30000, 40000, 50000], range(4, 7), range(11))] + \
                  [(key, [1, 2]) for key in get_mystery([40000, 50000], [20], range(11))] + \
                  [(key, [1, 2, 3]) for key in get_mystery([20000, 30000, 40000, 50000], range(7, 10), range(11))] + \
                  [(key, [1, 2, 3, 4]) for key in get_mystery([20000, 30000, 40000, 50000], [10], range(11))]
big_domain_dict = {key: item for key, item in big_domain_dict}
########################################
# Settings for double pendulum:
is_pendulum = False
is_pendulum = get_args(is_pendulum, 27, "bool")
if is_pendulum:
    num_layers = 5
    pred_nets_neurons = 160
    domain_net_neurons = 16
    reg_domain_amp = 1e-7
    show_3D_plot = False
    pred_nets_activation = "tanh"
    csv_filename_list = ['mysteryt_px_10', 'mysteryt_px_11', 'mysteryt_px_12', 'mysteryt_px_13', 'mysteryt_px_20', 'mysteryt_px_21', 'mysteryt_px_23']
    is_simplify_model = False
    is_simplify_domain = False
    add_theory_loss_threshold = 1e-4
    num_theories_init = 2
    add_theory_limit = 2
    is_xv = True
    record_mode = 1
#     reg_smooth = (0.1, 2, 10, 1e-2, 1)
    if is_xv:
        csv_filename_list = ['mysteryt_pxv_10', 'mysteryt_pxv_11', 'mysteryt_pxv_12', 'mysteryt_pxv_13', 'mysteryt_pxv_20', 'mysteryt_pxv_21', 'mysteryt_pxv_23']
        num_output_dims = 4
        num_input_steps = 1
        add_theory_loss_threshold = 3e-4
    for key in csv_filename_list:
        big_domain_dict[key] = [0, 1]
########################################

########################################
# Settings for data_format = "images":
if data_format == "images":
    batch_size = 100
    epochs = 10000
    loss_core = "mse"
    add_theory_quota = 0
    is_simplify_model = False
    is_simplify_domain = False
########################################

########################################
# Settings for Lagrangian:
if is_Lagrangian:
    num_input_steps = 3
    is_simplify_model = False
########################################

exp_id = get_args(exp_id, 1)
env_source = get_args(env_source, 2)
exp_mode = get_args(exp_mode, 3)
num_theories_init = get_args(num_theories_init, 4, "int")
pred_nets_neurons = get_args(pred_nets_neurons, 5, "int")
pred_nets_activation = get_args(pred_nets_activation, 6)
domain_net_neurons = get_args(domain_net_neurons, 7, "int")
domain_pred_mode = get_args(domain_pred_mode, 8)
mse_amp = get_args(mse_amp, 9, "float")
simplify_criteria = get_args(simplify_criteria, 10, "tuple")
scheduler_settings = get_args(scheduler_settings, 11, "tuple")
optim_type = get_args(optim_type, 12, "tuple")
optim_domain_type = get_args(optim_domain_type, 13, "tuple")
reg_amp = get_args(reg_amp, 14, "float")
reg_domain_amp = get_args(reg_domain_amp, 15, "float")
batch_size = get_args(batch_size, 16, "int")
loss_core = get_args(loss_core, 17)
loss_order = get_args(loss_order, 18)
loss_decay_scale = get_args(loss_decay_scale, 19, "eval")
is_mse_decay = get_args(is_mse_decay, 20, "bool")
loss_balance_model_influence = get_args(loss_balance_model_influence, 21, "bool")
num_examples = get_args(num_examples, 22, "int")
iter_to_saturation = get_args(iter_to_saturation, 23, "eval")
MDL_mode = get_args(MDL_mode, 24)
num_output_dims = get_args(num_output_dims, 25, "int")
num_layers = get_args(num_layers, 26, "int")
date_time = get_args(date_time, 28)
seed = get_args(seed, 29, "int")
array_id = get_args(array_id, 30, "int")
is_batch = False
array_id_core = array_id
if isinstance(env_source, str) and "mystery" in env_source:
    csv_filename_list = [env_source]
    env_source = "file"
    array_id = seed
    load_previous = False
    is_batch = True

if exp_mode == "continuous":
    is_propose_models = True
elif exp_mode == "newb":
    is_propose_models = False
elif exp_mode == "base":
    is_propose_models = False
    num_theories_init = 1
    pred_nets_activation = "leakyRelu" if pred_nets_activation == "linear" else pred_nets_activation
    pred_nets_neurons *= 2
    epochs *= 5
    add_theory_quota = 0
    is_simplify_model = False
    is_simplify_domain = False
    loss_core = "mse"
    MDL_mode = "None"
else:
    raise
batch_size = min(batch_size, num_examples)
save_image = True
render = False
loss_floor = 1e-12
simplify_lr = 1e-6
simplify_epochs = 40000
simplify_patience = 200
target_symbolic_expressions = {}
view_init = (10, 190)           # Angle you want to view the 3D plots

if iter_to_saturation is not None:
    iter_to_saturation = int(iter_to_saturation)
    reg_multiplier = np.linspace(0, 1, iter_to_saturation) ** 2
else:
    reg_multiplier = None

if epochs <= 3000:
    change_interval = 2
    record_interval = 5
else:
    change_interval = 10
    record_interval = 5

np.random.seed(seed)
random.seed(seed)
torch.manual_seed(seed)

dirname = theory_PATH + "/{0}_{1}/".format(exp_id, date_time)
filename = dirname + "{0}_{1}_num_{2}_pred_{3}_{4}_dom_{5}_{6}_mse_{7}_sim_{8}_optim_{10}_{11}_reg_{12}_{13}_batch_{14}_core_{15}_order_{16}_lossd_{17}_{18}_infl_{19}_#_{20}_mul_{21}_MDL_{22}_{23}D_{24}L_id_{25}_{26}.p".format(
            env_source, exp_mode, num_theories_init, pred_nets_neurons, pred_nets_activation, domain_net_neurons, domain_pred_mode, mse_amp,
            to_string(simplify_criteria), to_string(scheduler_settings, num_strings = 5), to_string(optim_type), to_string(optim_domain_type), reg_amp, reg_domain_amp, batch_size, loss_core,
            loss_order, loss_decay_scale, is_mse_decay, loss_balance_model_influence, num_examples, iter_to_saturation, MDL_mode, num_output_dims, num_layers, seed, array_id,
)
if is_batch:
    filename_batch = filename[:-2] + "_model/" + "{0}_{1}_num_{2}_pred_{3}_{4}_dom_{5}_{6}_mse_{7}_sim_{8}_optim_{10}_{11}_reg_{12}_{13}_batch_{14}_core_{15}_order_{16}_lossd_{17}_{18}_infl_{19}_#_{20}_mul_{21}_MDL_{22}_{23}D_{24}L_id_{25}_{26}.p".format(
            csv_filename_list[0], exp_mode, num_theories_init, pred_nets_neurons, pred_nets_activation, domain_net_neurons, domain_pred_mode, mse_amp,
            to_string(simplify_criteria), to_string(scheduler_settings, num_strings = 5), to_string(optim_type), to_string(optim_domain_type), reg_amp, reg_domain_amp, batch_size, loss_core,
            loss_order, loss_decay_scale, is_mse_decay, loss_balance_model_influence, num_examples, iter_to_saturation, MDL_mode, num_output_dims, num_layers, seed, array_id_core,
    )
    make_dir(filename_batch)
make_dir(filename)
make_dir(filename[:-2] + "_mys/file")
if save_image:
    make_dir(filename[:-2] + "/images")

# Load theory_hub:
filename_hub = filename[:-2] + "_hub.p"
print(filename)

# Load or initialize theory hub:
load_augmenting_hub = False
if load_augmenting_hub:
    is_propose_models = True
    dirname_augmenting_hub = theory_PATH + "/{0}_{1}/".format("transfer-hit5.5", "10-9")
    hub_candi1 = "{0}_{1}_num_{2}_pred_{3}_{4}_dom_{5}_{6}_mse_{7}_sim_{8}_optim_{10}_{11}_reg_{12}_{13}".format(
                env_source, "continuous", num_theories_init, pred_nets_neurons, pred_nets_activation, domain_net_neurons, domain_pred_mode, mse_amp,
                to_string(simplify_criteria), to_string(scheduler_settings, num_strings = 5), to_string(optim_type), to_string(optim_domain_type), reg_amp, reg_domain_amp,

    )
    hub_candi2 = "core_{0}_order_{1}_lossd_{2}_{3}_infl_{4}".format(loss_core, loss_order, loss_decay_scale, is_mse_decay, loss_balance_model_influence)
    hub_candi3 = "mul_{0}_MDL_{1}_{2}D_{3}L_id_{4}".format(iter_to_saturation, MDL_mode, num_output_dims, num_layers, seed, array_id)
    filename_augmenting_hub = dirname_augmenting_hub + filter_filename(dirname_augmenting_hub, include = [hub_candi1, hub_candi2, hub_candi3, "hub.p"])[0]
    theory_hub = load_model_dict_at_theory_hub(pickle.load(open(filename_augmenting_hub, "rb")), is_cuda = is_cuda)
    pickle.dump(theory_hub.model_dict, open(filename_hub, "wb"))
    print("load augmenting hub succeed: {0}".format(filename_augmenting_hub))
else:
    if load_previous:
        try:
            theory_hub = load_model_dict_at_theory_hub(pickle.load(open(filename_hub, "rb")), is_cuda = is_cuda)
        except:
            theory_hub = Theory_Hub()
            pickle.dump(theory_hub.model_dict, open(filename_hub, "wb"))
    else:
        theory_hub = Theory_Hub()
        pickle.dump(theory_hub.model_dict, open(filename_hub, "wb"))

def save_to_hub(pred_nets, domain_net, theory_hub, theory_type, theory_add_threshold, is_Lagrangian):
    """Save qualified theories to theory_hub"""
    if load_previous:
        theory_hub = load_model_dict_at_theory_hub(pickle.load(open(filename_hub, "rb")))
    added_theory_info = theory_hub.add_theories(name = hub_theory_name if theory_type == "neural" else hub_theory_name + "_simplified",
                                                pred_nets = pred_nets,
                                                domain_net = domain_net,
                                                dataset = dataset,
                                                threshold = theory_add_threshold,
                                                is_Lagrangian = is_Lagrangian,
                                               )
    if theory_type == "neural":
        name = "theory"
    elif theory_type == "simplified":
        name = "simplified_theory"
    else:
        raise
    info_dict[env_name]["added_{0}_info".format(name)] = added_theory_info
    pickle.dump(theory_hub.model_dict, open(filename_hub, "wb"))

def load_info_dict(info_dict, filename, trial_times = 1):
    load_succeed = False
    for i in range(trial_times):
        try:
            info_dict_updated = pickle.load(open(filename, "rb"))
            print("Succesfully loaded {0}".format(filename))
            info_dict_updated.update(info_dict)
            info_dict = info_dict_updated
            load_succeed = True
            break
        except Exception as e:
            print("Load trial {0}: ".format(i), e)
            time.sleep(2 ** i * (0.8 + 0.4 * np.random.rand()))
    return info_dict, load_succeed
```

    ../data/theory/exp1.0_11-3/file_continuous_num_4_pred_8_linear_dom_8_onehot_mse_1e-07_sim_DLs-0-3-relative_optim_adam-0.005_adam-0.001_reg_1e-08_1e-05_batch_10000_core_DLs_order_-1_lossd_None_False_infl_False_#_20000_mul_5000_MDL_both_2D_3L_id_0_0.p

## Theory learning and simplification:

```python
info_dict = {}
info_dict["array_id"] = array_id
info_dict["reg_smooth"] = reg_smooth
if load_previous:
    info_dict, load_succeed = load_info_dict(info_dict, filename)
    if not load_succeed:
        pickle.dump(info_dict, open(filename, "wb"))

###########################################################################################################################
# Level I. Iterate over environments:
###########################################################################################################################
for env_name in csv_filename_list:
    big_domain_ids = big_domain_dict[env_name] if env_name in big_domain_dict else None
    # Dealing with specific environments:
    print("\n" + "=" * 150 + "\nnew environment\n" + "{0}\n".format(env_name) + "=" * 150 + "\n\n")

    # Check if the environment has been run. If so, skip:
    if is_simplify_domain is True:
        assert is_simplify_model is True
        key_to_check = "domain_net_simplified"
    elif is_simplify_model is True:
        key_to_check = "pred_nets_simplified"
    else:
        key_to_check = "pred_nets"
    if env_name not in info_dict:
        info_dict[env_name] = {}
    if key_to_check in info_dict[env_name]:
        print("{0} already ran. Go to the next environment".format(env_name))
        continue
    hub_theory_name = "{0}_id_{1}".format(env_name, array_id)

    # Load or generate dataset:
    np.random.seed(seed)
    random.seed(seed)
    torch.manual_seed(seed)

    # Get dataset from file:
    dataset = get_dataset(filename=csv_dirname + env_name + ".csv",
                          num_output_dims=num_output_dims,
                          is_classified=is_classified,
                          num_input_steps=num_input_steps,
                          forward_steps=forward_steps,
                          num_examples=num_examples,
                          is_Lagrangian=is_Lagrangian,
                          is_cuda=is_cuda,
                         )
    if is_Lagrangian:
        num_output_dims = 4
        num_input_steps = 2

    ((X_train, y_train), (X_test, y_test), _), info = dataset
    if num_output_dims <= 2 and "true_domain_train" in info:
        plot_indi_domain(X_train, domain = info["true_domain_train"], is_show = isplot, filename = filename[:-2] + "/{0}_indi-domains.png".format(env_name) if save_image else None)

    # Setting up network and training hyperparameters:
    # Structure for theory model:
    if is_Lagrangian:
        struct_param_pred = [[1, "Symbolic_Layer", {"symbolic_expression": "g0 * x0 + g1 * x2"}]]
    else:
        struct_param_pred = [[pred_nets_neurons, "Simple_Layer", {}]] * (num_layers - 1) + [[y_train.size(1), "Simple_Layer", {"activation": "linear"}]]
    settings_pred = {"activation": pred_nets_activation}   # Default activation if not stipulated

    if data_format == "images":
        X_train, y_train, X_test, y_test = to_pixels(X_train, y_train, X_test, y_test, size = (39, 39), xlim = (-2, 2), ylim = (-2, 2), radius = 3, shape = "circle", is_cuda = is_cuda)

    # Structure for theory domain:
    struct_param_domain = [
        [domain_net_neurons, "Simple_Layer", {}],
        [domain_net_neurons, "Simple_Layer", {}],
        [num_theories_init, "Simple_Layer", {"activation": "linear"}],
    ]
    settings_domain = {"activation": "leakyRelu"} # Default activation if not stipulated
    loss_types = {
                    "pred-based_generalized-mean_{0}".format(loss_order): {"amp": 1., "decay_on": True},
                    "pred-based_generalized-mean_1": {"amp": mse_amp, "decay_on": is_mse_decay},
    }

    # Setting up regularization:
    reg_dict = {"pred_nets": {"weight": reg_amp, "bias": reg_amp}}
    reg_domain_dict = {"domain_net": {"weight": reg_domain_amp, "bias": reg_domain_amp}}

    # Settings for the simultaneous fitting of domain:
    domain_fit_setting = {"optim_domain_type": optim_domain_type, "reg_domain_dict": reg_domain_dict, "reg_domain_mode": reg_domain_mode}

    if data_format == "images":
        enc_fully_size = 512
        latent_size = 2
        num_channels = 32
        struct_param_encoder = [
            [num_channels, "Conv2d", {"kernel_size": 3, "stride": 2}],
            [num_channels, "Conv2d", {"kernel_size": 3, "stride": 2}],
            [num_channels, "Conv2d", {"kernel_size": 3, "stride": 2}],
            [latent_size, "Simple_Layer", {"layer_input_size": enc_fully_size}],
        ]
        struct_param_decoder = [
            [enc_fully_size, "Simple_Layer", {"layer_input_size": latent_size}],
            [num_channels, "ConvTranspose2d", {"kernel_size": 3, "stride": 2, "layer_input_size": (num_channels, 4, 4)}],
            [num_channels, "ConvTranspose2d", {"kernel_size": 3, "stride": 2}],
            [1, "ConvTranspose2d", {"kernel_size": 3, "stride": 2}],
        ]
        autoencoder = Conv_Autoencoder(
            input_channels_encoder = 1,
            input_channels_decoder = num_channels,
            struct_param_encoder = struct_param_encoder,
            struct_param_decoder = struct_param_decoder,
            share_model_among_steps = True,
            settings = {"activation": "leakyReluFlat"},
            is_cuda = is_cuda,
        )
        print("Pre-training autoencoder:")
        train_simple(autoencoder, X_train, X_train, loss_type = "mse")
        print("Pre-training autoencoder completes.")
        if isplot:
            X_train_recons = autoencoder(X_train)
            for i in np.random.randint(len(X_train), size = 4):
                plot_matrices(torch.cat([X_train[i], X_train_recons[i]], 0), images_per_row = 4)

    else:
        autoencoder = None

    #########################################################################################################
    # Level II.a. Iterate over number of trials:
    #########################################################################################################
    event_name = "hit_until_loss_threshold"
    for trial_times in range(max_trial_times):
        print("=" * 120 + "\nTrial {0}\n".format(trial_times) + "=" * 120 + "\n\n")
        # Each trial start with a different seed:
        torch.manual_seed(seed + trial_times)
        info_dict_single = {}

        # Propose theory from theory_hub:
        if load_previous:
            theory_hub = load_model_dict_at_theory_hub(pickle.load(open(filename_hub, "rb")), is_cuda = is_cuda)
        if is_propose_models:
            print("From theory_hub:")
            proposed_theory_models, propose_evaluation = theory_hub.propose_theory_models(X_train, y_train, max_num_models = max(2, int(num_theories_init * 0.7)), types = ["neural"], is_Lagrangian = is_Lagrangian)
        else:
            proposed_theory_models = {}
            propose_evaluation = {}
        info_dict_proposed_models = {}
        for name, theory_info in proposed_theory_models.items():
            info_dict_proposed_models[name] = {}
            for key, item in theory_info.items():
                if key == "theory_model":
                    info_dict_proposed_models[name][key] = item.model_dict
                else:
                    info_dict_proposed_models[name][key] = item
        info_dict_single["proposed_theory_models"] = info_dict_proposed_models
        info_dict_single["proposed_theory_models_evaluation"] = propose_evaluation

        # Initialize Theory_Training:
        T = Theory_Training(
            num_theories = num_theories_init,
            proposed_theory_models = proposed_theory_models,
            input_size = info["input_size"],
            struct_param_pred = struct_param_pred,
            struct_param_domain = struct_param_domain,
            settings_pred = settings_pred,
            settings_domain = settings_domain,
            autoencoder = autoencoder,
            loss_types = loss_types,
            loss_core = loss_core,
            loss_order = loss_order,
            loss_balance_model_influence = loss_balance_model_influence,
            is_Lagrangian = is_Lagrangian,
            reg_multiplier = reg_multiplier,
            is_cuda = is_cuda,
        )

        #################################################################################################
        # Level III.a Iterative training of models and domains
        #################################################################################################
        print("\n" + "=" * 80)
        print("{0}, trial times {1}, training model:".format(env_name, trial_times))
        print("=" * 80 + "\n")
        loss_precision_floor_init = 10
        T.set_loss_core(loss_core, loss_precision_floor_init)
        data_record = T.iterative_train(
            X_train,
            y_train,
            validation_data = (X_test, y_test),
            optim_type = optim_type,
            optim_autoencoder_type = optim_autoencoder_type,
            reg_dict = reg_dict,
            reg_mode = reg_mode,
            reg_smooth = reg_smooth,
            domain_fit_setting = domain_fit_setting,
            forward_steps = forward_steps,
            domain_pred_mode = domain_pred_mode,
            scheduler_settings = scheduler_settings,
            loss_order_decay = (lambda epoch: - epoch / float(loss_decay_scale)) if loss_decay_scale is not None else None,
            view_init = view_init,
            epochs = epochs,
            patience = int(epochs / 50) if exp_mode != "base" else None,
            batch_size = batch_size,
            inspect_interval = int(epochs / 100) if is_pendulum else int(epochs / 10),
            change_interval = change_interval,
            record_interval = record_interval,
            record_mode = record_mode,
            isplot = isplot,
            filename = filename[:-2] + "/{0}_training-model".format(env_name) if save_image else None,
            raise_nan = False,
            add_theory_quota = add_theory_quota,
            add_theory_limit = add_theory_limit,
            add_theory_criteria = add_theory_criteria,
            add_theory_loss_threshold = add_theory_loss_threshold,
            loss_floor = loss_floor,
            true_domain_test = info["true_domain_test"] if "true_domain_test" in info else None,
            num_output_dims = num_output_dims,
            prefix = "{0}_train_model:".format(env_name),
            show_3D_plot = show_3D_plot,
            show_vs = show_vs,
            big_domain_ids = big_domain_ids,
            fix_adaptive_precision_floor = True,
        )
        data_record["loss_precision_floor"] = loss_precision_floor_init
        info_dict_single["data_record"] = deepcopy(data_record)
        info_dict_single["pred_nets"] = deepcopy(T.pred_nets.model_dict)
        info_dict_single["removed_theories_0"] = T.remove_theories_based_on_data(X_test, y_test, threshold = theory_remove_fraction_threshold)

        # Perform MDL training after MSE training if stipulated:
        if MDL_mode == "both":
            if loss_core == "mse":
                T.set_loss_core("DLs")
            if hasattr(T, "optimizer"):
                delattr(T, "optimizer")
            data_record_MDL1 = T.iterative_train_schedule(
                X_train,
                y_train,
                validation_data = (X_test, y_test),
                optim_type = ("adam", 1e-4),
                reg_dict = reg_dict,
                reg_mode = reg_mode,
                reg_smooth = reg_smooth,
                domain_fit_setting = domain_fit_setting,
                forward_steps = forward_steps,
                domain_pred_mode = domain_pred_mode,
                scheduler_settings = scheduler_settings,
                loss_order_decay = (lambda epoch: - epoch / float(loss_decay_scale)) if loss_decay_scale is not None else None,
                view_init = view_init,
                epochs = int(epochs / 2),
                patience = int(epochs / 50) if exp_mode != "base" else None,
                batch_size = batch_size,
                inspect_interval = int(epochs / 100) if is_pendulum else int(epochs / 10),
                change_interval = change_interval,
                record_interval = record_interval,
                record_mode = record_mode,
                isplot = isplot,
                filename = filename[:-2] + "/{0}_training-modelDL1".format(env_name) if save_image else None,
                raise_nan = False,
                add_theory_quota = add_theory_quota,
                add_theory_limit = add_theory_limit,
                add_theory_criteria = add_theory_criteria,
                add_theory_loss_threshold = add_theory_loss_threshold,
                theory_remove_fraction_threshold = theory_remove_fraction_threshold,
                loss_floor = loss_floor,
                true_domain_test = info["true_domain_test"] if "true_domain_test" in info else None,
                num_output_dims = num_output_dims,
                prefix = "{0}_train_modelDL1:".format(env_name),
                show_3D_plot = show_3D_plot,
                show_vs = show_vs,
                big_domain_ids = big_domain_ids,
                num_phases = 4,
            )
            info_dict_single["data_record_MDL1"] = deepcopy(data_record_MDL1)

            T.domain_net_on = True
            print("\ndomain_net turned on, and each theory only optimize in its domain:\n")
            data_record_MDL2 = T.iterative_train_schedule(
                X_train,
                y_train,
                validation_data = (X_test, y_test),
                optim_type = ("adam", 1e-4),
                reg_dict = reg_dict,
                reg_mode = reg_mode,
                reg_smooth = reg_smooth,
                domain_fit_setting = domain_fit_setting,
                forward_steps = forward_steps,
                domain_pred_mode = domain_pred_mode,
                scheduler_settings = scheduler_settings,
                loss_order_decay = (lambda epoch: - epoch / float(loss_decay_scale)) if loss_decay_scale is not None else None,
                view_init = view_init,
                epochs = int(epochs / 2),
                patience = int(epochs / 50) if exp_mode != "base" else None,
                batch_size = batch_size,
                inspect_interval = int(epochs / 100) if is_pendulum else int(epochs / 10),
                change_interval = change_interval,
                record_interval = record_interval,
                record_mode = record_mode,
                isplot = isplot,
                filename = filename[:-2] + "/{0}_training-modelDL2".format(env_name) if save_image else None,
                raise_nan = False,
                add_theory_quota = add_theory_quota,
                add_theory_limit = add_theory_limit,
                add_theory_criteria = add_theory_criteria,
                add_theory_loss_threshold = add_theory_loss_threshold,
                theory_remove_fraction_threshold = theory_remove_fraction_threshold,
                loss_floor = loss_floor,
                true_domain_test = info["true_domain_test"] if "true_domain_test" in info else None,
                num_output_dims = num_output_dims,
                prefix = "{0}_train_modelDL2:".format(env_name),
                show_3D_plot = show_3D_plot,
                show_vs = show_vs,
                big_domain_ids = big_domain_ids,
                num_phases = 2,
            )
            info_dict_single["data_record_MDL2"] = deepcopy(data_record_MDL2)
            if hasattr(T, "optimizer"):
                delattr(T, "optimizer")

        # Remove theories whose fraction is below the remove_theshold:
        fraction_list = T.get_fraction_list(X_test, y_test)
        removed_theories = T.remove_theories_based_on_data(X_test, y_test, threshold = theory_remove_fraction_threshold)
        info_dict_single["fraction_list"] = fraction_list
        info_dict_single["removed_theories"] = removed_theories
        info_dict_single["domain_net"] = deepcopy(T.domain_net.model_dict)

        # Record pred_nets and domain_nets before simplification
        pred_nets = deepcopy(T.pred_nets)
        domain_net = deepcopy(T.domain_net)

        #########################################################################################################
        # Level III.b. Simplify theory models and/or domains:
        #########################################################################################################
        kwargs = {"true_domain_test": info["true_domain_test"] if "true_domain_test" in info else None,
                  "big_domain_ids": big_domain_ids,
                  "is_Lagrangian": is_Lagrangian,
                 }
        if is_simplify_model:
            print("\n" + "=" * 150)
            print("{0}, {1}".format(env_name, "simplifying model:"))
            print("=" * 150 + "\n")
            info_dict_single["data_record_simplification-model"] = {}
            record_data(info_dict_single["data_record_simplification-model"], [T.get_losses(X_test, y_test, **kwargs), T.pred_nets.model_dict, "before_simplification"], ["all_losses_dict", "pred_nets_model_dict", "event"])

            # Get prediction by theory domains:
            valid_onehot = to_one_hot(T.domain_net_forward(X_train).max(1)[1], T.num_theories)
            valid_onehot_test = to_one_hot(T.domain_net_forward(X_test).max(1)[1], T.num_theories)
            validation_pred_nets = (X_test, y_test, valid_onehot_test)

            # Collapse Layers:
            loss_record = T.pred_nets.simplify(X_train, y_train, valid_onehot, "collapse_layers", simplify_criteria = simplify_criteria, validation_data = validation_pred_nets, is_Lagrangian = is_Lagrangian)
            T.pred_nets.get_weights_bias(W_source = "core", b_source = "core", verbose = True)
            record_data(info_dict_single["data_record_simplification-model"], [T.get_losses(X_test, y_test, **kwargs), loss_record, T.pred_nets.model_dict, "after_collapsing_layer"], ["all_losses_dict", "loss_record", "pred_nets_model_dict", "event"])

            # Local snapping:
            loss_record = T.pred_nets.simplify(X_train, y_train, valid_onehot, "local", loss_type = "DLs", loss_precision_floor = T.loss_precision_floor, simplify_criteria = simplify_criteria, validation_data = validation_pred_nets, patience = simplify_patience, lr = simplify_lr, epochs = simplify_epochs, is_Lagrangian = is_Lagrangian, verbose = 2)
            record_data(info_dict_single["data_record_simplification-model"], [T.get_losses(X_test, y_test, **kwargs), loss_record, T.pred_nets.model_dict, "after_local_snapping"], ["all_losses_dict", "loss_record", "pred_nets_model_dict", "event"])

            # Integer snapping:
            loss_record = T.pred_nets.simplify(X_train, y_train, valid_onehot, "snap", snap_mode = "integer", loss_type = "DLs", loss_precision_floor = T.loss_precision_floor, simplify_criteria = simplify_criteria, validation_data = validation_pred_nets, patience = simplify_patience, lr = simplify_lr, epochs = simplify_epochs, is_Lagrangian = is_Lagrangian, verbose = 2)
            record_data(info_dict_single["data_record_simplification-model"], [T.get_losses(X_test, y_test, **kwargs), loss_record, T.pred_nets.model_dict, "after_integer_snapping"], ["all_losses_dict", "loss_record", "pred_nets_model_dict", "event"])

            # Rational snapping:
            loss_record = T.pred_nets.simplify(X_train, y_train, valid_onehot, "snap", snap_mode = "rational", loss_type = "DLs", loss_precision_floor = T.loss_precision_floor, simplify_criteria = simplify_criteria, validation_data = validation_pred_nets, patience = simplify_patience, lr = simplify_lr, epochs = simplify_epochs, is_Lagrangian = is_Lagrangian, verbose = 2)
            record_data(info_dict_single["data_record_simplification-model"], [T.get_losses(X_test, y_test, **kwargs), loss_record, T.pred_nets.model_dict, "after_rational_snapping"], ["all_losses_dict", "loss_record", "pred_nets_model_dict", "event"])

            # To symbolic:
            loss_record = T.pred_nets.simplify(X_train, y_train, valid_onehot, "to_symbolic", simplify_criteria = simplify_criteria, validation_data = validation_pred_nets, is_Lagrangian = is_Lagrangian)
            T.pred_nets.get_sympy_expression()
            record_data(info_dict_single["data_record_simplification-model"], [T.get_losses(X_test, y_test, **kwargs), loss_record, T.pred_nets.model_dict, "after_to_symbolic_and_snapping"], ["all_losses_dict", "loss_record", "pred_nets_model_dict", "event"])

            # Saving and show expression:
            info_dict_single["pred_nets_simplified"] = deepcopy(T.pred_nets.model_dict)
            print()
            print("MSE curve for simplification of the theory_model:")
            if isplot:
                try:
                    plot_loss_record(info_dict_single['data_record_simplification-model'], T.pred_nets.num_models)
                except:
                    pass
            if save_image:
                T.plot(X_test, y_test, true_domain = info["true_domain_test"] if "true_domain_test" in info else None,
                       view_init = view_init, figsize = (10, 8), filename = filename[:-2] + "/{0}_model-simplified".format(env_name) if save_image else None, is_show = isplot)

            # Simplifying domain:
            if is_simplify_domain:
                print("\n" + "=" * 150)
                print("{0}, {1}".format(env_name, "simplifying domain:"))
                print("=" * 150 + "\n")
                info_dict_single["data_record_simplification-domain-model"] = {}
                record_data(info_dict_single["data_record_simplification-domain-model"], [T.get_losses(X_test, y_test, **kwargs), T.domain_net.model_dict, "before_domain_simplification"], ["all_losses_dict", "domain_net_model_dict", "event"])

                best_theory_idx = get_best_model_idx(T.net_dict, X_train, y_train, loss_fun_cumu = T.loss_fun_cumu)
                loss_record = T.domain_net.simplify(X_train, best_theory_idx, mode = "to_symbolic", loss_type = "cross-entropy", simplify_criteria = simplify_criteria)
                record_data(info_dict_single["data_record_simplification-domain-model"], [T.get_losses(X_test, y_test, **kwargs), loss_record, T.domain_net.model_dict, "after_to_symbolic"], ["all_losses_dict", "loss_record", "domain_net_model_dict", "event"])

                loss_record = T.domain_net.simplify(X_train, best_theory_idx, mode = "pair_snap", loss_type = "cross-entropy", snap_mode = "integer", simplify_criteria = simplify_criteria)
                record_data(info_dict_single["data_record_simplification-domain-model"], [T.get_losses(X_test, y_test, **kwargs), loss_record, T.domain_net.model_dict, "after_pair_snap"], ["all_losses_dict", "loss_record", "domain_net_model_dict", "event"])
                info_dict_single["domain_net_simplified_pair_snap"] = deepcopy(T.domain_net.model_dict)

                loss_record = T.domain_net.simplify(X_train, best_theory_idx, mode = "activation_snap", loss_type = "cross-entropy", activation_source = "sigmoid",
                                                    activation_target = "heaviside", epochs = 10000, simplify_criteria = simplify_criteria)
                record_data(info_dict_single["data_record_simplification-domain-model"], [T.get_losses(X_test, y_test, **kwargs), loss_record, T.domain_net.model_dict, "after_activation_snap"], ["all_losses_dict", "loss_record", "domain_net_model_dict", "event"])
                info_dict_single["domain_net_simplified_activation_snap"] = deepcopy(T.domain_net.model_dict)

                loss_record = T.domain_net.simplify(X_train, best_theory_idx, mode = "pair_snap", loss_type = "cross-entropy", snap_mode = "integer", simplify_criteria = simplify_criteria)
                record_data(info_dict_single["data_record_simplification-domain-model"], [T.get_losses(X_test, y_test, **kwargs), loss_record, T.domain_net.model_dict, "after_second_pair_snap"], ["all_losses_dict", "loss_record", "domain_net_model_dict", "event"])
                info_dict_single["domain_net_simplified_final"] = deepcopy(T.domain_net.model_dict)
                print("\nSimplified domain expressions:")
                T.domain_net.show_expression()
                print()
                if save_image:
                    T.plot(X_test, y_test, true_domain = info["true_domain_test"] if "true_domain_test" in info else None,
                           view_init = view_init, figsize = (10, 8), filename = filename[:-2] + "/{0}_domain-simplified".format(env_name) if save_image else None, is_show = isplot)

            print("\n\n")

        if isplot:
            try:
                process_loss(info_dict_single, loss_core = loss_core)
            except:
                pass

        # Record and determine whether to run another trial:
        loss_dict = T.get_losses(X_test, y_test)
        loss_evaluation = loss_dict["mse_with_domain"] if exp_mode != "base" else loss_dict["mse_without_domain"]
        record_data(info_dict[env_name], [loss_evaluation, info_dict_single, event_name], ["loss_evaluation", "info_dict_single", "event"])
#         info_dict, _ = load_info_dict(info_dict, filename)
        print("#" * 70 + "\nEvaluation for current trial for {0}:\n".format(env_name) + "#" * 70)
        if loss_evaluation < loss_success_threshold:
            if exp_mode in ["continuous", "newb"]:
                event_name = "hit_until_symbolic_matching"
                if env_name not in target_symbolic_expressions or target_symbolic_expressions[env_name] is None or is_simplify_model is False:
                    print("When the target_symbolic_expression for the corresponding env_name is None or 'is_simplify_model' is False, cannot compare symbolic matching.")
                    print("\nThe loss is below the success threshold {0}. Break.\n".format(loss_success_threshold))
                    break
                else:
                    is_symbolic_matching, num_matches = check_expression_matching(T.pred_nets.get_sympy_expression(),
                                                                                target_symbolic_expressions[env_name] if env_name in target_symbolic_expressions else [standardize("0")],
                                                                                tolerance = matching_numerical_tolerance,
                                                                                snapped_tolerance = matching_snapped_tolerance)
                    if is_symbolic_matching:
                        print("\nThe symbolic expressions fully match the target expressions. Break.\n")
                        break
                    else:
                        print("\n{0} out of {1} expressions match the target expression {2}. Do another trial.\n".format(num_matches, len(T.pred_nets.get_sympy_expression()), target_symbolic_expressions[env_name]))
            elif exp_mode == "base":
                print("\nThe loss is below the success threshold {0}. Break.\n".format(loss_success_threshold))
                break
            else:
                raise Exception("exp_mode {0} not recognized!".format(exp_mode))
        else:
            print("\nThe loss does not get down to {0}, Do another trial.".format(loss_success_threshold))
        print("#" * 70 + "\n")

    # Record the last trial (either succeed or at the max_trial):
    export_csv_with_domain_prediction(env_name=env_name,
                                      domain_net=T.domain_net,
                                      num_output_dims=num_output_dims,
                                      num_input_steps=num_input_steps,
                                      csv_dirname=csv_dirname,
                                      write_dirname=filename[:-2] + "_mys/",
                                      is_Lagrangian=is_Lagrangian,
                                      is_cuda=is_cuda,
                                     )
    info_dict[env_name]["pred_nets"] = pred_nets.model_dict
    info_dict[env_name]["domain_net"] = domain_net.model_dict
    if hasattr(T, "autoencoder"):
        info_dict[env_name]["autoencoder"] = T.autoencoder.model_dict
    if is_simplify_model:
        info_dict[env_name]["pred_nets_simplified"] = T.pred_nets.model_dict
    if is_simplify_domain:
        info_dict[env_name]["domain_net_simplified"] = T.domain_net.model_dict
    # Add theory to the theory hub:
    save_to_hub(pred_nets, domain_net, theory_hub, theory_type = "neural", theory_add_threshold = theory_add_mse_threshold, is_Lagrangian = is_Lagrangian)
    save_to_hub(T.pred_nets, T.domain_net, theory_hub, theory_type = "simplified", theory_add_threshold = theory_add_mse_threshold, is_Lagrangian = is_Lagrangian)

    # Save files:
    if not is_batch:
        pickle.dump(info_dict, open(filename, "wb"))
    else:
        pickle.dump(info_dict, open(filename_batch, "wb"))

    # Write the learned expression to csv_file:
    coeff_learned_list, is_snapped_list = get_coeff_learned_list(T.pred_nets)
    open(filename[:-2] + "_mys/" + env_name + "_learned.csv", 'w').close()
    open(filename[:-2] + "_mys/" + env_name + "_snapped.csv", 'w').close()
    with open(filename[:-2] + "_mys/" + env_name + "_learned.csv", "a") as f:
        for coeff in coeff_learned_list:
            if len(coeff) > 0:
                line = ",".join([str(element) for element in coeff]) + '\n'
                f.write(line)
    with open(filename[:-2] + "_mys/" + env_name + "_snapped.csv", "a") as f:
        for is_snapped in is_snapped_list:
            if len(is_snapped) > 0:
                line = ",".join([str(element) for element in is_snapped]) + '\n'
                f.write(line)

    if isplot:
        try:
            process_loss(info_dict[env_name]["info_dict_single"][-1], loss_core = loss_core)
        except Exception as e:
            print(e)
```