# 下準備

$$
\newcommand{\vect}[1]{\mathbf{#1}}
\newcommand{\mat}[1]{\mathbf{#1}}
$$

## 数式の表記

変数: 小文字イタリック $x$

定数: 大文字イタリック $X$

ベクトル: 小文字ローマン体太字 $\vect{x}$

行列: 大文字ローマン体太字 $\mat{X}$

## 公式実装のclone

```
!git clone --depth=1 https://github.com/oreilly-japan/deep-learning-from-scratch-2.git
import sys
sys.path.append('deep-learning-from-scratch-2')
```

    Cloning into 'deep-learning-from-scratch-2'...
    remote: Enumerating objects: 73, done.
    remote: Counting objects: 100% (73/73), done.
    remote: Compressing objects: 100% (71/71), done.
    remote: Total 73 (delta 13), reused 14 (delta 0), pack-reused 0
    Unpacking objects: 100% (73/73), done.

# 6章 ゲート付きRNN

## 6.1 RNNの問題点（勾配消失・勾配爆発）

5章のシンプルなRNNは時系列が長くなると，逆伝播の際にtanhの微分値

$$
\frac{\delta \mathrm{tanh}(\vect{x})}{\delta \vect{x}} = 1 - \mathrm{tanh}(\vect{x})^2, (-1 \leq \mathrm{tanh}(\vect{x}) \leq 1)
$$

をtごとに損失値の勾配にかけ算することで勾配消失につながりやすくなる．

また，同じ重み $\mat{W_h}$ を損失値の勾配に何度もかけ算することで，勾配爆発につながりやすくなる．

この勾配消失と勾配爆発の二つの問題のうち，勾配爆発は**勾配クリッピング**という比較的簡単な手法で抑えることができる．

勾配クリッピングは以下のように計算する．あるイテレーションの結果，ニューラルネットワークのすべてのパラメータの勾配（いままでの実装で言うところの`self.grads`）を$\hat{\vect{g}}$としたとき，そのL2ノルム$||\hat{\vect{g}}||$があらかじめ定められた閾値$\mathit{threshold}$を超えていたら，

$$
\hat{\vect{g}} = \frac{\mathit{threshold}}{||\hat{\vect{g}}||} \hat{\vect{g}}
$$

でパラメータの勾配を修正する．（勾配は必ず1倍以下の値になる）

## 6.2 勾配消失とLSTM

シンプルなRNNのもう一つの問題である勾配消失については以下のように解決する（これがいわゆるLSTM）．

まず，$\vect{h_{t-1}} \mat{W_h} + \vect{x_t} \mat{W_x} + \vect{b}$を必ずtanhに放り込むのではなく，別途直接次の時刻$\mathrm{t+1}$に渡すルートを作る．（勾配消失の原因になるtanhをスキップ）
このルートに相当する変数を**セル**（または記憶セル）と呼び，$\vect{c_t}$で表現する．

そして，以下の3種類のゲートを持たせる．

- Outputゲート（$\vect{o}$）: 出力する隠れベクトル$\vect{h_t}$にどのくらい$\tanh(\vect{c_t})$を反映させるか決める
- Forgetゲート（$\vect{f}$）: $\vect{c_{t-1}}$ をどのくらい $\vect{c_t}$ に反映させるか決める
- Inputゲート（$\vect{i}$）: 従来のシンプルなRNNの出力$\vect{g} = \tanh(\vect{h_{t-1}} \mat{W_h} + \vect{x_t} \mat{W_x} + \vect{b})$をどのくらい $\vect{c_t}$ に反映させるか決める．

これらのゲートの出力は0～1の係数になる．その値は，シンプルなRNNの出力と同様に，$\vect{h_{t-1}}$と$\vect{x_t}$から求める．具体的には以下のように，sigmoid関数（$\sigma$）を用いて求める．（$\otimes$はアダマール積を表し，ベクトルを要素ごとに積算する）

$$
\begin{align}
\vect{f} &= \sigma(\vect{h_{t-1}} \mat{W_h^{(\vect{f})}} + \vect{x_t} \mat{W_x^{(\vect{f})}} + \vect{b}) \\
\vect{g} &= \tanh(\vect{h_{t-1}} \mat{W_h^{(\vect{g})}} + \vect{x_t} \mat{W_x^{(\vect{g})}} + \vect{b}) \\
\vect{i} &= \sigma(\vect{h_{t-1}} \mat{W_h^{(\vect{i})}} + \vect{x_t} \mat{W_x^{(\vect{i})}} + \vect{b}) \\
\vect{o} &= \sigma(\vect{h_{t-1}} \mat{W_h^{(\vect{o})}} + \vect{x_t} \mat{W_x^{(\vect{o})}} + \vect{b}) \\ \\
\vect{c_t} &= \vect{f} \otimes \vect{c_{t-1}} + \vect{i} \otimes \vect{g} \\
\vect{h_t} &= \vect{o} \otimes \tanh(\vect{c_t})
\end{align}
$$

## 6.3 LSTMの実装（`LSTM`，`TimeLSTM`クラス）

同じような行列積を4回計算することになるので，実装上は全ての$\mat{W_x}, \mat{W_h}, \vect{b}$を列方向にコンカチして，一回の行列積で4つの式の行列積を計算する．

```
class LSTM:
  def __init__(self, Wx, Wh, b):  # Wx, Wh, bはすでに4つ分コンカチ済みの想定
    self.params = [Wx, Wh, b]
    self.grads = [np.zeros_like(Wx), np.zeros_like(Wh), np.zeros_like(b)]
    self.cache = None

  def forward(self, x, h_prev, c_prev):
    Wx, Wh, b = self.params
    N, H = h_prev.shape

    A = np.dot(x, Wx) + np.dot(h_prev, Wh) + b  # まとめて計算する(AはALLのA）

    # まとめて計算した結果をsliceで取得
    f = A[:, :H]
    g = A[:, H:2*H]
    i = A[:, 2*H:3*H]
    o = A[:, 3*H:]

    f = sigmoid(f)
    g = np.tanh(g)
    i = sigmoid(i)
    o = sigmoid(o)

    c_next = f * c_prev + g * i
    h_next = o * np.tanh(c_next)

    self.cache = (x, h_prev, c_prev, i, f, g, o, c_next)
    return h_next, c_next
```

TimeRNNと同様にTimeLSTMレイヤを実装する．内容はTimeRNNとほぼ同じなので省略．違いはRNNクラスの代わりにLSTMクラスを用いるのと，逆伝播時にdhとともにdcを渡してあげることだけ．

## 6.4 LSTMを使った言語モデル（`Rnnlm`クラス）

5章の`SimpleRnnlm`クラスのTimeRNN部分をTimeLSTMに変更するなどして，`Rnnlm`クラスを実装する．

これを`RnnlmTrainer`のモデルとして使って学習．（勾配クリッピングも利用）

```
%set_env PYTHONPATH=/content/deep-learning-from-scratch-2:/env/python
!echo $PYTHONPATH
!python /content/deep-learning-from-scratch-2/ch06/train_rnnlm.py
```

    env: PYTHONPATH=/content/deep-learning-from-scratch-2:/env/python
    /content/deep-learning-from-scratch-2:/env/python
    Downloading ptb.train.txt ...
    Done
    Downloading ptb.test.txt ...
    Done
    | epoch 1 |  iter 1 / 1327 | time 0[s] | perplexity 10001.31
    | epoch 1 |  iter 21 / 1327 | time 8[s] | perplexity 2666.52
    | epoch 1 |  iter 41 / 1327 | time 16[s] | perplexity 1245.54
    | epoch 1 |  iter 61 / 1327 | time 23[s] | perplexity 992.81
    | epoch 1 |  iter 81 / 1327 | time 31[s] | perplexity 826.11
    | epoch 1 |  iter 101 / 1327 | time 39[s] | perplexity 644.00
    | epoch 1 |  iter 121 / 1327 | time 46[s] | perplexity 646.03
    | epoch 1 |  iter 141 / 1327 | time 54[s] | perplexity 605.71
    | epoch 1 |  iter 161 / 1327 | time 62[s] | perplexity 575.70
    | epoch 1 |  iter 181 / 1327 | time 69[s] | perplexity 593.42
    | epoch 1 |  iter 201 / 1327 | time 77[s] | perplexity 503.02
    | epoch 1 |  iter 221 / 1327 | time 85[s] | perplexity 482.71
    | epoch 1 |  iter 241 / 1327 | time 92[s] | perplexity 442.35
    | epoch 1 |  iter 261 / 1327 | time 100[s] | perplexity 464.36
    | epoch 1 |  iter 281 / 1327 | time 107[s] | perplexity 452.39
    | epoch 1 |  iter 301 / 1327 | time 115[s] | perplexity 387.56
    | epoch 1 |  iter 321 / 1327 | time 122[s] | perplexity 348.65
    | epoch 1 |  iter 341 / 1327 | time 130[s] | perplexity 406.42
    | epoch 1 |  iter 361 / 1327 | time 137[s] | perplexity 408.71
    | epoch 1 |  iter 381 / 1327 | time 145[s] | perplexity 332.90
    | epoch 1 |  iter 401 / 1327 | time 153[s] | perplexity 351.18
    | epoch 1 |  iter 421 / 1327 | time 160[s] | perplexity 349.44
    | epoch 1 |  iter 441 / 1327 | time 168[s] | perplexity 332.47
    | epoch 1 |  iter 461 / 1327 | time 175[s] | perplexity 331.00
    | epoch 1 |  iter 481 / 1327 | time 183[s] | perplexity 306.19
    | epoch 1 |  iter 501 / 1327 | time 190[s] | perplexity 319.53
    | epoch 1 |  iter 521 / 1327 | time 198[s] | perplexity 300.53
    | epoch 1 |  iter 541 / 1327 | time 205[s] | perplexity 325.06
    | epoch 1 |  iter 561 / 1327 | time 213[s] | perplexity 288.15
    | epoch 1 |  iter 581 / 1327 | time 220[s] | perplexity 259.15
    | epoch 1 |  iter 601 / 1327 | time 227[s] | perplexity 341.51
    | epoch 1 |  iter 621 / 1327 | time 235[s] | perplexity 310.58
    | epoch 1 |  iter 641 / 1327 | time 242[s] | perplexity 283.05
    | epoch 1 |  iter 661 / 1327 | time 250[s] | perplexity 272.32
    | epoch 1 |  iter 681 / 1327 | time 257[s] | perplexity 229.68
    | epoch 1 |  iter 701 / 1327 | time 265[s] | perplexity 248.96
    | epoch 1 |  iter 721 / 1327 | time 272[s] | perplexity 262.79
    | epoch 1 |  iter 741 / 1327 | time 279[s] | perplexity 221.41
    | epoch 1 |  iter 761 / 1327 | time 287[s] | perplexity 229.69
    | epoch 1 |  iter 781 / 1327 | time 294[s] | perplexity 218.27
    | epoch 1 |  iter 801 / 1327 | time 302[s] | perplexity 240.95
    | epoch 1 |  iter 821 / 1327 | time 309[s] | perplexity 224.15
    | epoch 1 |  iter 841 / 1327 | time 317[s] | perplexity 230.79
    | epoch 1 |  iter 861 / 1327 | time 324[s] | perplexity 223.46
    | epoch 1 |  iter 881 / 1327 | time 331[s] | perplexity 204.54
    | epoch 1 |  iter 901 / 1327 | time 339[s] | perplexity 254.58
    | epoch 1 |  iter 921 / 1327 | time 346[s] | perplexity 227.05
    | epoch 1 |  iter 941 / 1327 | time 354[s] | perplexity 228.84
    | epoch 1 |  iter 961 / 1327 | time 361[s] | perplexity 243.32
    | epoch 1 |  iter 981 / 1327 | time 368[s] | perplexity 229.47
    | epoch 1 |  iter 1001 / 1327 | time 376[s] | perplexity 192.09
    | epoch 1 |  iter 1021 / 1327 | time 383[s] | perplexity 226.10
    | epoch 1 |  iter 1041 / 1327 | time 390[s] | perplexity 208.09
    | epoch 1 |  iter 1061 / 1327 | time 398[s] | perplexity 196.13
    | epoch 1 |  iter 1081 / 1327 | time 405[s] | perplexity 167.22
    | epoch 1 |  iter 1101 / 1327 | time 413[s] | perplexity 188.26
    | epoch 1 |  iter 1121 / 1327 | time 420[s] | perplexity 228.12
    | epoch 1 |  iter 1141 / 1327 | time 427[s] | perplexity 206.77
    | epoch 1 |  iter 1161 / 1327 | time 435[s] | perplexity 197.89
    | epoch 1 |  iter 1181 / 1327 | time 442[s] | perplexity 192.64
    | epoch 1 |  iter 1201 / 1327 | time 450[s] | perplexity 161.32
    | epoch 1 |  iter 1221 / 1327 | time 457[s] | perplexity 159.58
    | epoch 1 |  iter 1241 / 1327 | time 464[s] | perplexity 188.27
    | epoch 1 |  iter 1261 / 1327 | time 472[s] | perplexity 172.36
    | epoch 1 |  iter 1281 / 1327 | time 479[s] | perplexity 178.84
    | epoch 1 |  iter 1301 / 1327 | time 486[s] | perplexity 221.23
    | epoch 1 |  iter 1321 / 1327 | time 494[s] | perplexity 209.67
    | epoch 2 |  iter 1 / 1327 | time 496[s] | perplexity 227.78
    | epoch 2 |  iter 21 / 1327 | time 504[s] | perplexity 203.99
    | epoch 2 |  iter 41 / 1327 | time 511[s] | perplexity 190.24
    | epoch 2 |  iter 61 / 1327 | time 518[s] | perplexity 176.31
    | epoch 2 |  iter 81 / 1327 | time 526[s] | perplexity 158.73
    | epoch 2 |  iter 101 / 1327 | time 533[s] | perplexity 150.59
    | epoch 2 |  iter 121 / 1327 | time 540[s] | perplexity 160.85
    | epoch 2 |  iter 141 / 1327 | time 548[s] | perplexity 177.05
    | epoch 2 |  iter 161 / 1327 | time 555[s] | perplexity 192.92
    | epoch 2 |  iter 181 / 1327 | time 563[s] | perplexity 199.65
    | epoch 2 |  iter 201 / 1327 | time 570[s] | perplexity 185.04
    | epoch 2 |  iter 221 / 1327 | time 577[s] | perplexity 185.39
    | epoch 2 |  iter 241 / 1327 | time 584[s] | perplexity 176.38
    | epoch 2 |  iter 261 / 1327 | time 592[s] | perplexity 186.87
    | epoch 2 |  iter 281 / 1327 | time 599[s] | perplexity 186.78
    | epoch 2 |  iter 301 / 1327 | time 606[s] | perplexity 166.64
    | epoch 2 |  iter 321 / 1327 | time 614[s] | perplexity 138.41
    | epoch 2 |  iter 341 / 1327 | time 621[s] | perplexity 172.54
    | epoch 2 |  iter 361 / 1327 | time 628[s] | perplexity 198.02
    | epoch 2 |  iter 381 / 1327 | time 636[s] | perplexity 154.52
    | epoch 2 |  iter 401 / 1327 | time 643[s] | perplexity 169.88
    | epoch 2 |  iter 421 / 1327 | time 650[s] | perplexity 155.02
    | epoch 2 |  iter 441 / 1327 | time 658[s] | perplexity 162.10
    | epoch 2 |  iter 461 / 1327 | time 665[s] | perplexity 157.34
    | epoch 2 |  iter 481 / 1327 | time 672[s] | perplexity 156.22
    | epoch 2 |  iter 501 / 1327 | time 680[s] | perplexity 170.39
    | epoch 2 |  iter 521 / 1327 | time 687[s] | perplexity 173.50
    | epoch 2 |  iter 541 / 1327 | time 694[s] | perplexity 175.80
    | epoch 2 |  iter 561 / 1327 | time 701[s] | perplexity 154.58
    | epoch 2 |  iter 581 / 1327 | time 709[s] | perplexity 138.31
    | epoch 2 |  iter 601 / 1327 | time 716[s] | perplexity 190.61
    | epoch 2 |  iter 621 / 1327 | time 723[s] | perplexity 183.12
    | epoch 2 |  iter 641 / 1327 | time 731[s] | perplexity 164.71
    | epoch 2 |  iter 661 / 1327 | time 738[s] | perplexity 155.80
    | epoch 2 |  iter 681 / 1327 | time 745[s] | perplexity 130.31
    | epoch 2 |  iter 701 / 1327 | time 753[s] | perplexity 151.84
    | epoch 2 |  iter 721 / 1327 | time 760[s] | perplexity 159.91
    | epoch 2 |  iter 741 / 1327 | time 767[s] | perplexity 134.13
    | epoch 2 |  iter 761 / 1327 | time 775[s] | perplexity 131.71
    | epoch 2 |  iter 781 / 1327 | time 782[s] | perplexity 135.15
    | epoch 2 |  iter 801 / 1327 | time 789[s] | perplexity 147.54
    | epoch 2 |  iter 821 / 1327 | time 797[s] | perplexity 144.09
    | epoch 2 |  iter 841 / 1327 | time 804[s] | perplexity 144.93
    | epoch 2 |  iter 861 / 1327 | time 811[s] | perplexity 147.59
    | epoch 2 |  iter 881 / 1327 | time 818[s] | perplexity 130.68
    | epoch 2 |  iter 901 / 1327 | time 826[s] | perplexity 167.14
    | epoch 2 |  iter 921 / 1327 | time 833[s] | perplexity 147.59
    | epoch 2 |  iter 941 / 1327 | time 840[s] | perplexity 153.93
    | epoch 2 |  iter 961 / 1327 | time 848[s] | perplexity 165.14
    | epoch 2 |  iter 981 / 1327 | time 855[s] | perplexity 154.60
    | epoch 2 |  iter 1001 / 1327 | time 862[s] | perplexity 131.94
    | epoch 2 |  iter 1021 / 1327 | time 870[s] | perplexity 157.19
    | epoch 2 |  iter 1041 / 1327 | time 877[s] | perplexity 145.11
    | epoch 2 |  iter 1061 / 1327 | time 884[s] | perplexity 129.22
    | epoch 2 |  iter 1081 / 1327 | time 892[s] | perplexity 110.60
    | epoch 2 |  iter 1101 / 1327 | time 899[s] | perplexity 119.53
    | epoch 2 |  iter 1121 / 1327 | time 906[s] | perplexity 153.50
    | epoch 2 |  iter 1141 / 1327 | time 914[s] | perplexity 142.61
    | epoch 2 |  iter 1161 / 1327 | time 921[s] | perplexity 132.73
    | epoch 2 |  iter 1181 / 1327 | time 928[s] | perplexity 135.61
    | epoch 2 |  iter 1201 / 1327 | time 936[s] | perplexity 113.32
    | epoch 2 |  iter 1221 / 1327 | time 943[s] | perplexity 109.51
    | epoch 2 |  iter 1241 / 1327 | time 950[s] | perplexity 131.22
    | epoch 2 |  iter 1261 / 1327 | time 957[s] | perplexity 123.90
    | epoch 2 |  iter 1281 / 1327 | time 965[s] | perplexity 123.14
    | epoch 2 |  iter 1301 / 1327 | time 972[s] | perplexity 158.54
    | epoch 2 |  iter 1321 / 1327 | time 979[s] | perplexity 153.01
    | epoch 3 |  iter 1 / 1327 | time 982[s] | perplexity 161.89
    | epoch 3 |  iter 21 / 1327 | time 989[s] | perplexity 144.32
    | epoch 3 |  iter 41 / 1327 | time 996[s] | perplexity 137.17
    | epoch 3 |  iter 61 / 1327 | time 1004[s] | perplexity 127.28
    | epoch 3 |  iter 81 / 1327 | time 1011[s] | perplexity 117.14
    | epoch 3 |  iter 101 / 1327 | time 1018[s] | perplexity 106.76
    | epoch 3 |  iter 121 / 1327 | time 1025[s] | perplexity 117.05
    | epoch 3 |  iter 141 / 1327 | time 1033[s] | perplexity 127.14
    | epoch 3 |  iter 161 / 1327 | time 1040[s] | perplexity 141.83
    | epoch 3 |  iter 181 / 1327 | time 1047[s] | perplexity 150.83
    | epoch 3 |  iter 201 / 1327 | time 1055[s] | perplexity 141.23
    | epoch 3 |  iter 221 / 1327 | time 1062[s] | perplexity 141.81
    | epoch 3 |  iter 241 / 1327 | time 1069[s] | perplexity 136.35
    | epoch 3 |  iter 261 / 1327 | time 1076[s] | perplexity 141.71
    | epoch 3 |  iter 281 / 1327 | time 1084[s] | perplexity 144.82
    | epoch 3 |  iter 301 / 1327 | time 1091[s] | perplexity 124.71
    | epoch 3 |  iter 321 / 1327 | time 1098[s] | perplexity 103.06
    | epoch 3 |  iter 341 / 1327 | time 1106[s] | perplexity 125.25
    | epoch 3 |  iter 361 / 1327 | time 1113[s] | perplexity 154.42
    | epoch 3 |  iter 381 / 1327 | time 1120[s] | perplexity 116.08
    | epoch 3 |  iter 401 / 1327 | time 1128[s] | perplexity 130.41
    | epoch 3 |  iter 421 / 1327 | time 1135[s] | perplexity 113.65
    | epoch 3 |  iter 441 / 1327 | time 1142[s] | perplexity 124.59
    | epoch 3 |  iter 461 / 1327 | time 1149[s] | perplexity 118.74
    | epoch 3 |  iter 481 / 1327 | time 1157[s] | perplexity 120.65
    | epoch 3 |  iter 501 / 1327 | time 1164[s] | perplexity 130.10
    | epoch 3 |  iter 521 / 1327 | time 1171[s] | perplexity 137.52
    | epoch 3 |  iter 541 / 1327 | time 1179[s] | perplexity 136.36
    | epoch 3 |  iter 561 / 1327 | time 1186[s] | perplexity 119.23
    | epoch 3 |  iter 581 / 1327 | time 1193[s] | perplexity 106.51
    | epoch 3 |  iter 601 / 1327 | time 1200[s] | perplexity 151.11
    | epoch 3 |  iter 621 / 1327 | time 1208[s] | perplexity 144.14
    | epoch 3 |  iter 641 / 1327 | time 1215[s] | perplexity 130.57
    | epoch 3 |  iter 661 / 1327 | time 1222[s] | perplexity 121.14
    | epoch 3 |  iter 681 / 1327 | time 1230[s] | perplexity 100.65
    | epoch 3 |  iter 701 / 1327 | time 1237[s] | perplexity 119.20
    | epoch 3 |  iter 721 / 1327 | time 1244[s] | perplexity 127.06
    | epoch 3 |  iter 741 / 1327 | time 1251[s] | perplexity 109.01
    | epoch 3 |  iter 761 / 1327 | time 1259[s] | perplexity 103.03
    | epoch 3 |  iter 781 / 1327 | time 1266[s] | perplexity 106.17
    | epoch 3 |  iter 801 / 1327 | time 1273[s] | perplexity 115.48
    | epoch 3 |  iter 821 / 1327 | time 1281[s] | perplexity 117.05
    | epoch 3 |  iter 841 / 1327 | time 1288[s] | perplexity 115.62
    | epoch 3 |  iter 861 / 1327 | time 1296[s] | perplexity 121.45
    | epoch 3 |  iter 881 / 1327 | time 1303[s] | perplexity 106.89
    | epoch 3 |  iter 901 / 1327 | time 1310[s] | perplexity 132.69
    | epoch 3 |  iter 921 / 1327 | time 1317[s] | perplexity 118.68
    | epoch 3 |  iter 941 / 1327 | time 1325[s] | perplexity 127.36
    | epoch 3 |  iter 961 / 1327 | time 1332[s] | perplexity 132.83
    | epoch 3 |  iter 981 / 1327 | time 1339[s] | perplexity 125.01
    | epoch 3 |  iter 1001 / 1327 | time 1347[s] | perplexity 110.05
    | epoch 3 |  iter 1021 / 1327 | time 1354[s] | perplexity 129.19
    | epoch 3 |  iter 1041 / 1327 | time 1361[s] | perplexity 120.77
    | epoch 3 |  iter 1061 / 1327 | time 1368[s] | perplexity 102.86
    | epoch 3 |  iter 1081 / 1327 | time 1376[s] | perplexity 87.73
    | epoch 3 |  iter 1101 / 1327 | time 1383[s] | perplexity 94.89
    | epoch 3 |  iter 1121 / 1327 | time 1391[s] | perplexity 122.03
    | epoch 3 |  iter 1141 / 1327 | time 1398[s] | perplexity 114.87
    | epoch 3 |  iter 1161 / 1327 | time 1405[s] | perplexity 106.59
    | epoch 3 |  iter 1181 / 1327 | time 1413[s] | perplexity 112.49
    | epoch 3 |  iter 1201 / 1327 | time 1420[s] | perplexity 94.41
    | epoch 3 |  iter 1221 / 1327 | time 1427[s] | perplexity 89.25
    | epoch 3 |  iter 1241 / 1327 | time 1434[s] | perplexity 105.65
    | epoch 3 |  iter 1261 / 1327 | time 1442[s] | perplexity 105.26
    | epoch 3 |  iter 1281 / 1327 | time 1449[s] | perplexity 101.07
    | epoch 3 |  iter 1301 / 1327 | time 1456[s] | perplexity 130.71
    | epoch 3 |  iter 1321 / 1327 | time 1464[s] | perplexity 127.46
    | epoch 4 |  iter 1 / 1327 | time 1466[s] | perplexity 134.02
    | epoch 4 |  iter 21 / 1327 | time 1473[s] | perplexity 122.46
    | epoch 4 |  iter 41 / 1327 | time 1481[s] | perplexity 108.67
    | epoch 4 |  iter 61 / 1327 | time 1488[s] | perplexity 108.33
    | epoch 4 |  iter 81 / 1327 | time 1495[s] | perplexity 96.33
    | epoch 4 |  iter 101 / 1327 | time 1503[s] | perplexity 86.31
    | epoch 4 |  iter 121 / 1327 | time 1510[s] | perplexity 96.04
    | epoch 4 |  iter 141 / 1327 | time 1517[s] | perplexity 105.31
    | epoch 4 |  iter 161 / 1327 | time 1524[s] | perplexity 118.14
    | epoch 4 |  iter 181 / 1327 | time 1532[s] | perplexity 128.67
    | epoch 4 |  iter 201 / 1327 | time 1539[s] | perplexity 119.22
    | epoch 4 |  iter 221 / 1327 | time 1546[s] | perplexity 122.76
    | epoch 4 |  iter 241 / 1327 | time 1553[s] | perplexity 116.67
    | epoch 4 |  iter 261 / 1327 | time 1561[s] | perplexity 116.85
    | epoch 4 |  iter 281 / 1327 | time 1568[s] | perplexity 122.86
    | epoch 4 |  iter 301 / 1327 | time 1575[s] | perplexity 104.81
    | epoch 4 |  iter 321 / 1327 | time 1582[s] | perplexity 85.26
    | epoch 4 |  iter 341 / 1327 | time 1590[s] | perplexity 100.67
    | epoch 4 |  iter 361 / 1327 | time 1597[s] | perplexity 129.95
    | epoch 4 |  iter 381 / 1327 | time 1604[s] | perplexity 98.65
    | epoch 4 |  iter 401 / 1327 | time 1611[s] | perplexity 110.70
    | epoch 4 |  iter 421 / 1327 | time 1618[s] | perplexity 93.56
    | epoch 4 |  iter 441 / 1327 | time 1626[s] | perplexity 104.21
    | epoch 4 |  iter 461 / 1327 | time 1633[s] | perplexity 99.45
    | epoch 4 |  iter 481 / 1327 | time 1640[s] | perplexity 103.12
    | epoch 4 |  iter 501 / 1327 | time 1648[s] | perplexity 109.42
    | epoch 4 |  iter 521 / 1327 | time 1655[s] | perplexity 117.14
    | epoch 4 |  iter 541 / 1327 | time 1662[s] | perplexity 113.92
    | epoch 4 |  iter 561 / 1327 | time 1669[s] | perplexity 103.87
    | epoch 4 |  iter 581 / 1327 | time 1676[s] | perplexity 89.19
    | epoch 4 |  iter 601 / 1327 | time 1684[s] | perplexity 128.77
    | epoch 4 |  iter 621 / 1327 | time 1691[s] | perplexity 122.62
    | epoch 4 |  iter 641 / 1327 | time 1698[s] | perplexity 112.01
    | epoch 4 |  iter 661 / 1327 | time 1705[s] | perplexity 103.55
    | epoch 4 |  iter 681 / 1327 | time 1713[s] | perplexity 84.78
    | epoch 4 |  iter 701 / 1327 | time 1720[s] | perplexity 102.05
    | epoch 4 |  iter 721 / 1327 | time 1727[s] | perplexity 109.23
    | epoch 4 |  iter 741 / 1327 | time 1734[s] | perplexity 95.89
    | epoch 4 |  iter 761 / 1327 | time 1742[s] | perplexity 88.65
    | epoch 4 |  iter 781 / 1327 | time 1749[s] | perplexity 88.24
    | epoch 4 |  iter 801 / 1327 | time 1756[s] | perplexity 98.55
    | epoch 4 |  iter 821 / 1327 | time 1763[s] | perplexity 102.79
    | epoch 4 |  iter 841 / 1327 | time 1771[s] | perplexity 100.48
    | epoch 4 |  iter 861 / 1327 | time 1778[s] | perplexity 106.37
    | epoch 4 |  iter 881 / 1327 | time 1785[s] | perplexity 91.92
    | epoch 4 |  iter 901 / 1327 | time 1793[s] | perplexity 116.16
    | epoch 4 |  iter 921 / 1327 | time 1800[s] | perplexity 102.63
    | epoch 4 |  iter 941 / 1327 | time 1807[s] | perplexity 112.53
    | epoch 4 |  iter 961 / 1327 | time 1814[s] | perplexity 113.79
    | epoch 4 |  iter 981 / 1327 | time 1822[s] | perplexity 107.30
    | epoch 4 |  iter 1001 / 1327 | time 1829[s] | perplexity 98.24
    | epoch 4 |  iter 1021 / 1327 | time 1836[s] | perplexity 113.41
    | epoch 4 |  iter 1041 / 1327 | time 1843[s] | perplexity 106.03
    | epoch 4 |  iter 1061 / 1327 | time 1851[s] | perplexity 88.96
    | epoch 4 |  iter 1081 / 1327 | time 1858[s] | perplexity 77.85
    | epoch 4 |  iter 1101 / 1327 | time 1865[s] | perplexity 79.29
    | epoch 4 |  iter 1121 / 1327 | time 1872[s] | perplexity 103.49
    | epoch 4 |  iter 1141 / 1327 | time 1880[s] | perplexity 101.25
    | epoch 4 |  iter 1161 / 1327 | time 1887[s] | perplexity 92.13
    | epoch 4 |  iter 1181 / 1327 | time 1894[s] | perplexity 97.43
    | epoch 4 |  iter 1201 / 1327 | time 1901[s] | perplexity 84.31
    | epoch 4 |  iter 1221 / 1327 | time 1909[s] | perplexity 76.26
    | epoch 4 |  iter 1241 / 1327 | time 1916[s] | perplexity 91.15
    | epoch 4 |  iter 1261 / 1327 | time 1923[s] | perplexity 93.75
    | epoch 4 |  iter 1281 / 1327 | time 1931[s] | perplexity 89.15
    | epoch 4 |  iter 1301 / 1327 | time 1938[s] | perplexity 112.28
    | epoch 4 |  iter 1321 / 1327 | time 1945[s] | perplexity 111.18

    evaluating perplexity ...
    234 / 235
    test perplexity:  135.38238800005408

```

```


```python
import sys
sys.path.append('..')
import os
import torch
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import LabelEncoder, MinMaxScaler, KBinsDiscretizer
from sklearn.impute import SimpleImputer
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold
from torch.utils.data import TensorDataset, DataLoader
from torchvision.models import resnet18
import torch.nn.functional as F
import time
from sklearn.metrics import accuracy_score
from sympy import simplify_logic
from sklearn.model_selection import train_test_split

import lens
from lens.utils.base import validate_network, set_seed, tree_to_formula
from lens.utils.layer import prune_logic_layers
from lens import logic

from dSprites.dSprites_loader import load_dsprites, concept_filters, get_shape_scale
from dSprites.dSprites_style_I2C import i2c_style

results_dir = './results_ll/dsprites'
if not os.path.isdir(results_dir):
    os.makedirs(results_dir)

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
n_rep = 10
tot_epochs = 10001
prune_epochs = 60001
```

```python
def get_data(path):
    # Load dataset specified by schema
    dataset_schema = 'small_skip'
    # dataset_schema = 'big_skip'

    # Get filtered concept values
    shape_range, scale_range, rot_range, x_pos_range, y_pos_range = concept_filters(dataset_schema)

    # Define function for filtering out specified concept values only
    def c_filter_fn(concepts):
        in_shape_range = (concepts[1] in shape_range)
        in_scale_range = (concepts[2] in scale_range)
        in_rot_range = (concepts[3] in rot_range)
        in_x_range = (concepts[4] in x_pos_range)
        in_y_range = (concepts[5] in y_pos_range)

        return (in_shape_range and in_scale_range and in_rot_range and in_x_range and in_y_range)

    label_fn = get_shape_scale(shape_range, scale_range)

    # Load dataset
    x_train, y_train, x_val, y_val, x_test, y_test, c_train, c_val, c_test, c_names = load_dsprites(path,
                                                                               c_filter_fn=c_filter_fn,
                                                                               label_fn=label_fn,
                                                                               train_test_split_flag=True)

    return x_train, y_train, x_val, y_val, x_test, y_test, c_train, c_val, c_test, c_names

# Load the model, as well as input, label, and concept data
data = get_data(path='./dSprites/data/dsprites.npz')
x_train, y_train, x_val, y_val, x_test, y_test, c_train, c_val, c_test, c_names = data
x_train = torch.FloatTensor(x_train)
y_train = torch.FloatTensor(y_train)
x_val = torch.FloatTensor(x_val)
y_val = torch.FloatTensor(y_val)
x_test = torch.FloatTensor(x_test)
y_test = torch.FloatTensor(y_test)
c_train = torch.FloatTensor(c_train)
c_val = torch.FloatTensor(c_val)
c_test = torch.FloatTensor(c_test)
print("Data loaded successfully...")
```

    x_train shape: (20993, 3, 64, 64)
    c_train shape: (20993, 50)
    y_train shape: (20993, 18)
    Number of images in x_train 20993
    Number of images in x_val 10341
    Number of images in x_test 5530
    Data loaded successfully...

```python
args = {
    'models_dir': './models_ll/dSprites/',
    'model_style': 'CBM',
    'seed': 0,
    'batch_size': 256,
}

if not os.path.isdir(args['models_dir']):
    os.makedirs(args['models_dir'])

train_dataset = TensorDataset(x_train, c_train, y_train)
train_loader = DataLoader(train_dataset, batch_size=args['batch_size'], shuffle=False)
val_dataset = TensorDataset(x_val, c_val, y_val)
valid_loader = DataLoader(val_dataset, batch_size=args['batch_size'], shuffle=False)
test_dataset = TensorDataset(x_test, c_test, y_test)
test_loader = DataLoader(test_dataset, batch_size=args['batch_size'], shuffle=False)
```

```python
n_concepts = c_train.size(1)
n_outputs = y_train.size(1)
```

```python
colors = ['white']
shapes = ['square', 'ellipse', 'heart']
scale = ['very small', 'small', 's-medium', 'b-medium', 'big', 'very big']
rotation = ['0°', '5°', '10°', '15°', '20°', '25°', '30°', '35°']
x_pos = ['x0', 'x2', 'x4', 'x6', 'x8', 'x10', 'x12', 'x14', 'x16', 'x18', 'x20', 'x22', 'x24', 'x26', 'x28', 'x30']
y_pos = ['y0', 'y2', 'y4', 'y6', 'y8', 'y10', 'y12', 'y14', 'y16', 'y18', 'y20', 'y22', 'y24', 'y26', 'y28', 'y30']
concepts = colors + shapes + scale + rotation + x_pos + y_pos
len(concepts)
```

    50

```python
def l1_loss(model):
    loss = 0
    for module in model.children():
        if isinstance(module, lens.nn.XLogic):
            loss += torch.norm(module.weight, 1) + torch.norm(module.bias, 1)
            break
    return loss
```

```python
def cbm_learning(net, dataloader, device, optimizer, criterion, epoch, n_epochs, train=False, base=False):
    running_loss = 0.0
    y_correct = 0
    c_correct = 0
    c_total = 0
    y_total = 0
    need_pruning = True
    prune_epochs = 30
    total_step = len(dataloader)
    for batch_idx, (data_, concepts_, target_) in enumerate(dataloader):
        data_, concepts_, target_ = data_.to(device), concepts_.to(device), target_.to(device)

        if train:
            optimizer.zero_grad()

        if base:
            c_preds = net[0](data_)
            loss = criterion(c_preds, concepts_)
        else:
            c_preds = net[0](data_)
            y_preds = net[1](c_preds)
            loss = criterion(y_preds, target_) + 0.00001 * l1_loss(net_top)

        if train:
            loss.backward()
            optimizer.step()

            if epoch > prune_epochs and need_pruning and not base:
                lens.utils.layer.prune_logic_layers(net[1].to(device), fan_in=9, device=device)
                need_pruning = False

        running_loss += loss.item()

        c_correct += torch.sum((c_preds > 0.5).eq(concepts_)).item()
        c_total += concepts_.size(0) * concepts_.size(1)
        if not base:
            y_correct += torch.sum((y_preds.argmax(dim=1)).eq(target_.argmax(dim=1))).item()
            y_total += target_.size(0)

        if (batch_idx) % 20 == 0 and train:
            print(f'Epoch [{epoch}/{n_epochs}], Step [{batch_idx}/{total_step}], Loss: {loss.item():.4f}')

    c_accuracy = c_correct / c_total
    if base:
        y_accuracy = 0
    else:
        y_accuracy = y_correct / y_total
    loss = running_loss / total_step

    return net, c_accuracy, y_accuracy, loss
```

```python
net_base = resnet18(pretrained=False)
num_ftrs = net_base.fc.in_features
net_base.fc = torch.nn.Linear(num_ftrs, n_concepts)
net_top = torch.nn.Sequential(*[
    lens.nn.XLogic(n_concepts, 32, first=True),
    torch.nn.LeakyReLU(),
    torch.nn.Linear(32, 32),
    torch.nn.LeakyReLU(),
    torch.nn.Linear(32, 32),
    torch.nn.LeakyReLU(),
    torch.nn.Linear(32, n_outputs),
    lens.nn.XLogic(n_outputs, n_outputs, top=True),
])
net = torch.nn.Sequential(*[net_base, net_top]).to(device)

file_name = './models_ll/dsprites/trained_model_l1'
```

```python
for base in [True, False]:
    print(base)
    net_suffix = 'base' if base else 'top'
    if os.path.isfile(f'{file_name}_{net_suffix}.pt'):
        # load trained model
        net.load_state_dict(torch.load(f'{file_name}_{net_suffix}.pt'))

    else:
        set_seed(0)
        need_pruning = True

        criterion = torch.nn.MSELoss()

        if base:
            n_epochs = 20
            optimizer = torch.optim.AdamW(net.parameters(), lr=0.0001)
        else:
            n_epochs = 50
            optimizer = torch.optim.AdamW(net.parameters(), lr=0.001)
            for param in net[0].parameters():
                param.requires_grad = False

        valid_loss_min = np.Inf
        c_train_accuracy = 0
        val_loss = []
        val_acc = []
        val_c_acc = []
        test_loss = []
        test_acc = []
        test_c_acc = []
        train_loss = []
        train_acc = []
        train_c_acc = []
        epoch = 1
        for epoch in range(1, n_epochs + 1):
            print(f'Epoch {epoch}\n')

            net, c_accuracy, y_accuracy, loss = cbm_learning(net, train_loader,
                                                             device, optimizer, criterion,
                                                             epoch, n_epochs, train=True,
                                                             base=base)
            c_train_accuracy = c_accuracy
            train_c_acc.append(c_accuracy)
            train_acc.append(y_accuracy)
            train_loss.append(loss)
            print(f'\ntrain-loss: {loss:.4f}, train-c-acc: {c_accuracy:.4f}, train-cy-acc: {y_accuracy:.4f}')

            with torch.no_grad():
                net.eval()

                net, c_accuracy, y_accuracy, loss = cbm_learning(net, valid_loader,
                                                                 device, optimizer, criterion,
                                                                 epoch, n_epochs, train=False,
                                                                 base=base)
                val_c_acc.append(c_accuracy)
                val_acc.append(y_accuracy)
                val_loss.append(loss)
                print(f'validation loss: {loss:.4f}, validation-c-acc: {c_accuracy:.4f}, validation-y-acc: {y_accuracy:.4f}')

                if loss < valid_loss_min:
                    valid_loss_min = loss
                    torch.save(net.state_dict(), f'{file_name}_{net_suffix}.pt')
                    print('Improvement-Detected, save-model')

                net, c_accuracy, y_accuracy, loss = cbm_learning(net, test_loader,
                                                                 device, optimizer, criterion,
                                                                 epoch, n_epochs, train=False,
                                                                 base=base)
                test_c_acc.append(c_accuracy)
                test_acc.append(y_accuracy)
                test_loss.append(loss)
                print(f'test loss: {loss:.4f}, test-c-acc: {c_accuracy:.4f}, test-y-acc: {y_accuracy:.4f}\n')

            net.train()

        results = pd.DataFrame({
            'test_acc': test_acc,
            'test_c_acc': test_c_acc,
            'test_loss': test_loss,
            'val_acc': val_acc,
            'val_c_acc': val_c_acc,
            'val_loss': val_loss,
            'train_acc': train_acc,
            'train_c_acc': train_c_acc,
            'train_loss': train_loss,
        })
```

    True
    False
    Epoch 1

    Epoch [1/50], Step [0/83], Loss: 0.2497
    Epoch [1/50], Step [20/83], Loss: 0.2346
    Epoch [1/50], Step [40/83], Loss: 0.2015
    Epoch [1/50], Step [60/83], Loss: 0.0924
    Epoch [1/50], Step [80/83], Loss: 0.0545

    train-loss: 0.1670, train-c-acc: 0.9944, train-cy-acc: 0.0499
    validation loss: 0.0547, validation-c-acc: 0.9786, validation-y-acc: 0.0586
    Improvement-Detected, save-model
    test loss: 0.0547, test-c-acc: 0.9791, test-y-acc: 0.0530

    Epoch 2

    Epoch [2/50], Step [0/83], Loss: 0.0548
    Epoch [2/50], Step [20/83], Loss: 0.0547
    Epoch [2/50], Step [40/83], Loss: 0.0546
    Epoch [2/50], Step [60/83], Loss: 0.0541
    Epoch [2/50], Step [80/83], Loss: 0.0537

    train-loss: 0.0544, train-c-acc: 0.9944, train-cy-acc: 0.0547
    validation loss: 0.0538, validation-c-acc: 0.9786, validation-y-acc: 0.0586
    Improvement-Detected, save-model
    test loss: 0.0539, test-c-acc: 0.9791, test-y-acc: 0.0530

    Epoch 3

    Epoch [3/50], Step [0/83], Loss: 0.0539
    Epoch [3/50], Step [20/83], Loss: 0.0537
    Epoch [3/50], Step [40/83], Loss: 0.0538
    Epoch [3/50], Step [60/83], Loss: 0.0536
    Epoch [3/50], Step [80/83], Loss: 0.0534

    train-loss: 0.0537, train-c-acc: 0.9944, train-cy-acc: 0.0547
    validation loss: 0.0535, validation-c-acc: 0.9786, validation-y-acc: 0.0586
    Improvement-Detected, save-model
    test loss: 0.0535, test-c-acc: 0.9791, test-y-acc: 0.0530

    Epoch 4

    Epoch [4/50], Step [0/83], Loss: 0.0535
    Epoch [4/50], Step [20/83], Loss: 0.0535
    Epoch [4/50], Step [40/83], Loss: 0.0534
    Epoch [4/50], Step [60/83], Loss: 0.0534
    Epoch [4/50], Step [80/83], Loss: 0.0533

    train-loss: 0.0534, train-c-acc: 0.9944, train-cy-acc: 0.0596
    validation loss: 0.0534, validation-c-acc: 0.9786, validation-y-acc: 0.0743
    Improvement-Detected, save-model
    test loss: 0.0534, test-c-acc: 0.9791, test-y-acc: 0.0660

    Epoch 5

    Epoch [5/50], Step [0/83], Loss: 0.0534
    Epoch [5/50], Step [20/83], Loss: 0.0533
    Epoch [5/50], Step [40/83], Loss: 0.0533
    Epoch [5/50], Step [60/83], Loss: 0.0533
    Epoch [5/50], Step [80/83], Loss: 0.0532

    train-loss: 0.0533, train-c-acc: 0.9944, train-cy-acc: 0.0686
    validation loss: 0.0533, validation-c-acc: 0.9786, validation-y-acc: 0.1056
    Improvement-Detected, save-model
    test loss: 0.0533, test-c-acc: 0.9791, test-y-acc: 0.1014

    Epoch 6

    Epoch [6/50], Step [0/83], Loss: 0.0533
    Epoch [6/50], Step [20/83], Loss: 0.0532
    Epoch [6/50], Step [40/83], Loss: 0.0532
    Epoch [6/50], Step [60/83], Loss: 0.0532
    Epoch [6/50], Step [80/83], Loss: 0.0531

    train-loss: 0.0532, train-c-acc: 0.9944, train-cy-acc: 0.0764
    validation loss: 0.0532, validation-c-acc: 0.9786, validation-y-acc: 0.1160
    Improvement-Detected, save-model
    test loss: 0.0532, test-c-acc: 0.9791, test-y-acc: 0.1103

    Epoch 7

    Epoch [7/50], Step [0/83], Loss: 0.0531
    Epoch [7/50], Step [20/83], Loss: 0.0531
    Epoch [7/50], Step [40/83], Loss: 0.0531
    Epoch [7/50], Step [60/83], Loss: 0.0531
    Epoch [7/50], Step [80/83], Loss: 0.0530

    train-loss: 0.0531, train-c-acc: 0.9944, train-cy-acc: 0.0855
    validation loss: 0.0531, validation-c-acc: 0.9786, validation-y-acc: 0.1250
    Improvement-Detected, save-model
    test loss: 0.0531, test-c-acc: 0.9791, test-y-acc: 0.1193

    Epoch 8

    Epoch [8/50], Step [0/83], Loss: 0.0530
    Epoch [8/50], Step [20/83], Loss: 0.0530
    Epoch [8/50], Step [40/83], Loss: 0.0529
    Epoch [8/50], Step [60/83], Loss: 0.0529
    Epoch [8/50], Step [80/83], Loss: 0.0529

    train-loss: 0.0530, train-c-acc: 0.9944, train-cy-acc: 0.0990
    validation loss: 0.0529, validation-c-acc: 0.9786, validation-y-acc: 0.1569
    Improvement-Detected, save-model
    test loss: 0.0529, test-c-acc: 0.9791, test-y-acc: 0.1485

    Epoch 9

    Epoch [9/50], Step [0/83], Loss: 0.0529
    Epoch [9/50], Step [20/83], Loss: 0.0528
    Epoch [9/50], Step [40/83], Loss: 0.0528
    Epoch [9/50], Step [60/83], Loss: 0.0527
    Epoch [9/50], Step [80/83], Loss: 0.0525

    train-loss: 0.0528, train-c-acc: 0.9944, train-cy-acc: 0.1344
    validation loss: 0.0527, validation-c-acc: 0.9786, validation-y-acc: 0.1556
    Improvement-Detected, save-model
    test loss: 0.0527, test-c-acc: 0.9791, test-y-acc: 0.1519

    Epoch 10

    Epoch [10/50], Step [0/83], Loss: 0.0525
    Epoch [10/50], Step [20/83], Loss: 0.0525
    Epoch [10/50], Step [40/83], Loss: 0.0522
    Epoch [10/50], Step [60/83], Loss: 0.0516
    Epoch [10/50], Step [80/83], Loss: 0.0505

    train-loss: 0.0520, train-c-acc: 0.9944, train-cy-acc: 0.1427
    validation loss: 0.0511, validation-c-acc: 0.9786, validation-y-acc: 0.1240
    Improvement-Detected, save-model
    test loss: 0.0511, test-c-acc: 0.9791, test-y-acc: 0.1239

    Epoch 11

    Epoch [11/50], Step [0/83], Loss: 0.0503
    Epoch [11/50], Step [20/83], Loss: 0.0498
    Epoch [11/50], Step [40/83], Loss: 0.0489
    Epoch [11/50], Step [60/83], Loss: 0.0475
    Epoch [11/50], Step [80/83], Loss: 0.0466

    train-loss: 0.0487, train-c-acc: 0.9944, train-cy-acc: 0.1555
    validation loss: 0.0482, validation-c-acc: 0.9786, validation-y-acc: 0.2546
    Improvement-Detected, save-model
    test loss: 0.0482, test-c-acc: 0.9791, test-y-acc: 0.2488

    Epoch 12

    Epoch [12/50], Step [0/83], Loss: 0.0463
    Epoch [12/50], Step [20/83], Loss: 0.0467
    Epoch [12/50], Step [40/83], Loss: 0.0462
    Epoch [12/50], Step [60/83], Loss: 0.0441
    Epoch [12/50], Step [80/83], Loss: 0.0423

    train-loss: 0.0455, train-c-acc: 0.9944, train-cy-acc: 0.3127
    validation loss: 0.0449, validation-c-acc: 0.9786, validation-y-acc: 0.3403
    Improvement-Detected, save-model
    test loss: 0.0449, test-c-acc: 0.9791, test-y-acc: 0.3392

    Epoch 13

    Epoch [13/50], Step [0/83], Loss: 0.0416
    Epoch [13/50], Step [20/83], Loss: 0.0407
    Epoch [13/50], Step [40/83], Loss: 0.0367
    Epoch [13/50], Step [60/83], Loss: 0.0312
    Epoch [13/50], Step [80/83], Loss: 0.0288

    train-loss: 0.0362, train-c-acc: 0.9944, train-cy-acc: 0.5124
    validation loss: 0.0339, validation-c-acc: 0.9786, validation-y-acc: 0.6283
    Improvement-Detected, save-model
    test loss: 0.0336, test-c-acc: 0.9791, test-y-acc: 0.6259

    Epoch 14

    Epoch [14/50], Step [0/83], Loss: 0.0272
    Epoch [14/50], Step [20/83], Loss: 0.0257
    Epoch [14/50], Step [40/83], Loss: 0.0231
    Epoch [14/50], Step [60/83], Loss: 0.0205
    Epoch [14/50], Step [80/83], Loss: 0.0183

    train-loss: 0.0234, train-c-acc: 0.9944, train-cy-acc: 0.8081
    validation loss: 0.0269, validation-c-acc: 0.9786, validation-y-acc: 0.7741
    Improvement-Detected, save-model
    test loss: 0.0266, test-c-acc: 0.9791, test-y-acc: 0.7814

    Epoch 15

    Epoch [15/50], Step [0/83], Loss: 0.0173
    Epoch [15/50], Step [20/83], Loss: 0.0163
    Epoch [15/50], Step [40/83], Loss: 0.0149
    Epoch [15/50], Step [60/83], Loss: 0.0128
    Epoch [15/50], Step [80/83], Loss: 0.0105

    train-loss: 0.0148, train-c-acc: 0.9944, train-cy-acc: 0.9241
    validation loss: 0.0231, validation-c-acc: 0.9786, validation-y-acc: 0.7913
    Improvement-Detected, save-model
    test loss: 0.0229, test-c-acc: 0.9791, test-y-acc: 0.7980

    Epoch 16

    Epoch [16/50], Step [0/83], Loss: 0.0103
    Epoch [16/50], Step [20/83], Loss: 0.0096
    Epoch [16/50], Step [40/83], Loss: 0.0084
    Epoch [16/50], Step [60/83], Loss: 0.0080
    Epoch [16/50], Step [80/83], Loss: 0.0064

    train-loss: 0.0090, train-c-acc: 0.9944, train-cy-acc: 0.9623
    validation loss: 0.0218, validation-c-acc: 0.9786, validation-y-acc: 0.8073
    Improvement-Detected, save-model
    test loss: 0.0214, test-c-acc: 0.9791, test-y-acc: 0.8085

    Epoch 17

    Epoch [17/50], Step [0/83], Loss: 0.0061
    Epoch [17/50], Step [20/83], Loss: 0.0059
    Epoch [17/50], Step [40/83], Loss: 0.0052
    Epoch [17/50], Step [60/83], Loss: 0.0042
    Epoch [17/50], Step [80/83], Loss: 0.0031

    train-loss: 0.0053, train-c-acc: 0.9944, train-cy-acc: 0.9945
    validation loss: 0.0203, validation-c-acc: 0.9786, validation-y-acc: 0.8062
    Improvement-Detected, save-model
    test loss: 0.0199, test-c-acc: 0.9791, test-y-acc: 0.8107

    Epoch 18

    Epoch [18/50], Step [0/83], Loss: 0.0029
    Epoch [18/50], Step [20/83], Loss: 0.0032
    Epoch [18/50], Step [40/83], Loss: 0.0030
    Epoch [18/50], Step [60/83], Loss: 0.0025
    Epoch [18/50], Step [80/83], Loss: 0.0020

    train-loss: 0.0031, train-c-acc: 0.9944, train-cy-acc: 0.9958
    validation loss: 0.0192, validation-c-acc: 0.9786, validation-y-acc: 0.8138
    Improvement-Detected, save-model
    test loss: 0.0187, test-c-acc: 0.9791, test-y-acc: 0.8195

    Epoch 19

    Epoch [19/50], Step [0/83], Loss: 0.0020
    Epoch [19/50], Step [20/83], Loss: 0.0025
    Epoch [19/50], Step [40/83], Loss: 0.0024
    Epoch [19/50], Step [60/83], Loss: 0.0021
    Epoch [19/50], Step [80/83], Loss: 0.0017

    train-loss: 0.0025, train-c-acc: 0.9944, train-cy-acc: 0.9962
    validation loss: 0.0193, validation-c-acc: 0.9786, validation-y-acc: 0.8070
    test loss: 0.0187, test-c-acc: 0.9791, test-y-acc: 0.8166

    Epoch 20

    Epoch [20/50], Step [0/83], Loss: 0.0017
    Epoch [20/50], Step [20/83], Loss: 0.0022
    Epoch [20/50], Step [40/83], Loss: 0.0022
    Epoch [20/50], Step [60/83], Loss: 0.0018
    Epoch [20/50], Step [80/83], Loss: 0.0016

    train-loss: 0.0022, train-c-acc: 0.9944, train-cy-acc: 0.9966
    validation loss: 0.0194, validation-c-acc: 0.9786, validation-y-acc: 0.8029
    test loss: 0.0188, test-c-acc: 0.9791, test-y-acc: 0.8139

    Epoch 21

    Epoch [21/50], Step [0/83], Loss: 0.0016
    Epoch [21/50], Step [20/83], Loss: 0.0020
    Epoch [21/50], Step [40/83], Loss: 0.0020
    Epoch [21/50], Step [60/83], Loss: 0.0017
    Epoch [21/50], Step [80/83], Loss: 0.0015

    train-loss: 0.0021, train-c-acc: 0.9944, train-cy-acc: 0.9969
    validation loss: 0.0193, validation-c-acc: 0.9786, validation-y-acc: 0.8035
    test loss: 0.0187, test-c-acc: 0.9791, test-y-acc: 0.8145

    Epoch 22

    Epoch [22/50], Step [0/83], Loss: 0.0015
    Epoch [22/50], Step [20/83], Loss: 0.0019
    Epoch [22/50], Step [40/83], Loss: 0.0019
    Epoch [22/50], Step [60/83], Loss: 0.0016
    Epoch [22/50], Step [80/83], Loss: 0.0014

    train-loss: 0.0019, train-c-acc: 0.9944, train-cy-acc: 0.9968
    validation loss: 0.0190, validation-c-acc: 0.9786, validation-y-acc: 0.8040
    Improvement-Detected, save-model
    test loss: 0.0185, test-c-acc: 0.9791, test-y-acc: 0.8152

    Epoch 23

    Epoch [23/50], Step [0/83], Loss: 0.0014
    Epoch [23/50], Step [20/83], Loss: 0.0018
    Epoch [23/50], Step [40/83], Loss: 0.0018
    Epoch [23/50], Step [60/83], Loss: 0.0015
    Epoch [23/50], Step [80/83], Loss: 0.0013

    train-loss: 0.0018, train-c-acc: 0.9944, train-cy-acc: 0.9970
    validation loss: 0.0187, validation-c-acc: 0.9786, validation-y-acc: 0.8059
    Improvement-Detected, save-model
    test loss: 0.0182, test-c-acc: 0.9791, test-y-acc: 0.8168

    Epoch 24

    Epoch [24/50], Step [0/83], Loss: 0.0014
    Epoch [24/50], Step [20/83], Loss: 0.0017
    Epoch [24/50], Step [40/83], Loss: 0.0017
    Epoch [24/50], Step [60/83], Loss: 0.0014
    Epoch [24/50], Step [80/83], Loss: 0.0013

    train-loss: 0.0018, train-c-acc: 0.9944, train-cy-acc: 0.9971
    validation loss: 0.0185, validation-c-acc: 0.9786, validation-y-acc: 0.8079
    Improvement-Detected, save-model
    test loss: 0.0179, test-c-acc: 0.9791, test-y-acc: 0.8175

    Epoch 25

    Epoch [25/50], Step [0/83], Loss: 0.0013
    Epoch [25/50], Step [20/83], Loss: 0.0016
    Epoch [25/50], Step [40/83], Loss: 0.0016
    Epoch [25/50], Step [60/83], Loss: 0.0014
    Epoch [25/50], Step [80/83], Loss: 0.0012

    train-loss: 0.0017, train-c-acc: 0.9944, train-cy-acc: 0.9971
    validation loss: 0.0183, validation-c-acc: 0.9786, validation-y-acc: 0.8091
    Improvement-Detected, save-model
    test loss: 0.0178, test-c-acc: 0.9791, test-y-acc: 0.8179

    Epoch 26

    Epoch [26/50], Step [0/83], Loss: 0.0013
    Epoch [26/50], Step [20/83], Loss: 0.0016
    Epoch [26/50], Step [40/83], Loss: 0.0016
    Epoch [26/50], Step [60/83], Loss: 0.0013
    Epoch [26/50], Step [80/83], Loss: 0.0012

    train-loss: 0.0016, train-c-acc: 0.9944, train-cy-acc: 0.9973
    validation loss: 0.0183, validation-c-acc: 0.9786, validation-y-acc: 0.8086
    Improvement-Detected, save-model
    test loss: 0.0177, test-c-acc: 0.9791, test-y-acc: 0.8186

    Epoch 27

    Epoch [27/50], Step [0/83], Loss: 0.0012
    Epoch [27/50], Step [20/83], Loss: 0.0015
    Epoch [27/50], Step [40/83], Loss: 0.0015
    Epoch [27/50], Step [60/83], Loss: 0.0013
    Epoch [27/50], Step [80/83], Loss: 0.0011

    train-loss: 0.0016, train-c-acc: 0.9944, train-cy-acc: 0.9973
    validation loss: 0.0182, validation-c-acc: 0.9786, validation-y-acc: 0.8092
    Improvement-Detected, save-model
    test loss: 0.0176, test-c-acc: 0.9791, test-y-acc: 0.8174

    Epoch 28

    Epoch [28/50], Step [0/83], Loss: 0.0012
    Epoch [28/50], Step [20/83], Loss: 0.0015
    Epoch [28/50], Step [40/83], Loss: 0.0015
    Epoch [28/50], Step [60/83], Loss: 0.0013
    Epoch [28/50], Step [80/83], Loss: 0.0011

    train-loss: 0.0015, train-c-acc: 0.9944, train-cy-acc: 0.9974
    validation loss: 0.0181, validation-c-acc: 0.9786, validation-y-acc: 0.8095
    Improvement-Detected, save-model
    test loss: 0.0175, test-c-acc: 0.9791, test-y-acc: 0.8181

    Epoch 29

    Epoch [29/50], Step [0/83], Loss: 0.0012
    Epoch [29/50], Step [20/83], Loss: 0.0014
    Epoch [29/50], Step [40/83], Loss: 0.0014
    Epoch [29/50], Step [60/83], Loss: 0.0012
    Epoch [29/50], Step [80/83], Loss: 0.0011

    train-loss: 0.0015, train-c-acc: 0.9944, train-cy-acc: 0.9974
    validation loss: 0.0181, validation-c-acc: 0.9786, validation-y-acc: 0.8104
    Improvement-Detected, save-model
    test loss: 0.0174, test-c-acc: 0.9791, test-y-acc: 0.8190

    Epoch 30

    Epoch [30/50], Step [0/83], Loss: 0.0011
    Epoch [30/50], Step [20/83], Loss: 0.0014
    Epoch [30/50], Step [40/83], Loss: 0.0014
    Epoch [30/50], Step [60/83], Loss: 0.0012
    Epoch [30/50], Step [80/83], Loss: 0.0011

    train-loss: 0.0015, train-c-acc: 0.9944, train-cy-acc: 0.9975
    validation loss: 0.0180, validation-c-acc: 0.9786, validation-y-acc: 0.8096
    Improvement-Detected, save-model
    test loss: 0.0174, test-c-acc: 0.9791, test-y-acc: 0.8199

    Epoch 31

    Epoch [31/50], Step [0/83], Loss: 0.0011
    Epoch [31/50], Step [20/83], Loss: 0.0018
    Epoch [31/50], Step [40/83], Loss: 0.0018
    Epoch [31/50], Step [60/83], Loss: 0.0013
    Epoch [31/50], Step [80/83], Loss: 0.0011

    train-loss: 0.0018, train-c-acc: 0.9944, train-cy-acc: 0.9949
    validation loss: 0.0170, validation-c-acc: 0.9786, validation-y-acc: 0.8215
    Improvement-Detected, save-model
    test loss: 0.0162, test-c-acc: 0.9791, test-y-acc: 0.8300

    Epoch 32

    Epoch [32/50], Step [0/83], Loss: 0.0011
    Epoch [32/50], Step [20/83], Loss: 0.0014
    Epoch [32/50], Step [40/83], Loss: 0.0016
    Epoch [32/50], Step [60/83], Loss: 0.0012
    Epoch [32/50], Step [80/83], Loss: 0.0010

    train-loss: 0.0015, train-c-acc: 0.9944, train-cy-acc: 0.9959
    validation loss: 0.0168, validation-c-acc: 0.9786, validation-y-acc: 0.8257
    Improvement-Detected, save-model
    test loss: 0.0160, test-c-acc: 0.9791, test-y-acc: 0.8329

    Epoch 33

    Epoch [33/50], Step [0/83], Loss: 0.0010
    Epoch [33/50], Step [20/83], Loss: 0.0013
    Epoch [33/50], Step [40/83], Loss: 0.0015
    Epoch [33/50], Step [60/83], Loss: 0.0011
    Epoch [33/50], Step [80/83], Loss: 0.0009

    train-loss: 0.0015, train-c-acc: 0.9944, train-cy-acc: 0.9960
    validation loss: 0.0167, validation-c-acc: 0.9786, validation-y-acc: 0.8259
    Improvement-Detected, save-model
    test loss: 0.0159, test-c-acc: 0.9791, test-y-acc: 0.8336

    Epoch 34

    Epoch [34/50], Step [0/83], Loss: 0.0010
    Epoch [34/50], Step [20/83], Loss: 0.0012
    Epoch [34/50], Step [40/83], Loss: 0.0015
    Epoch [34/50], Step [60/83], Loss: 0.0011
    Epoch [34/50], Step [80/83], Loss: 0.0009

    train-loss: 0.0014, train-c-acc: 0.9944, train-cy-acc: 0.9960
    validation loss: 0.0166, validation-c-acc: 0.9786, validation-y-acc: 0.8267
    Improvement-Detected, save-model
    test loss: 0.0158, test-c-acc: 0.9791, test-y-acc: 0.8338

    Epoch 35

    Epoch [35/50], Step [0/83], Loss: 0.0010
    Epoch [35/50], Step [20/83], Loss: 0.0012
    Epoch [35/50], Step [40/83], Loss: 0.0014
    Epoch [35/50], Step [60/83], Loss: 0.0010
    Epoch [35/50], Step [80/83], Loss: 0.0009

    train-loss: 0.0014, train-c-acc: 0.9944, train-cy-acc: 0.9961
    validation loss: 0.0166, validation-c-acc: 0.9786, validation-y-acc: 0.8263
    test loss: 0.0159, test-c-acc: 0.9791, test-y-acc: 0.8335

    Epoch 36

    Epoch [36/50], Step [0/83], Loss: 0.0009
    Epoch [36/50], Step [20/83], Loss: 0.0011
    Epoch [36/50], Step [40/83], Loss: 0.0013
    Epoch [36/50], Step [60/83], Loss: 0.0010
    Epoch [36/50], Step [80/83], Loss: 0.0009

    train-loss: 0.0013, train-c-acc: 0.9944, train-cy-acc: 0.9961
    validation loss: 0.0166, validation-c-acc: 0.9786, validation-y-acc: 0.8266
    Improvement-Detected, save-model
    test loss: 0.0158, test-c-acc: 0.9791, test-y-acc: 0.8335

    Epoch 37

    Epoch [37/50], Step [0/83], Loss: 0.0009
    Epoch [37/50], Step [20/83], Loss: 0.0011
    Epoch [37/50], Step [40/83], Loss: 0.0013
    Epoch [37/50], Step [60/83], Loss: 0.0010
    Epoch [37/50], Step [80/83], Loss: 0.0008

    train-loss: 0.0013, train-c-acc: 0.9944, train-cy-acc: 0.9963
    validation loss: 0.0166, validation-c-acc: 0.9786, validation-y-acc: 0.8256
    Improvement-Detected, save-model
    test loss: 0.0158, test-c-acc: 0.9791, test-y-acc: 0.8325

    Epoch 38

    Epoch [38/50], Step [0/83], Loss: 0.0009
    Epoch [38/50], Step [20/83], Loss: 0.0011
    Epoch [38/50], Step [40/83], Loss: 0.0013
    Epoch [38/50], Step [60/83], Loss: 0.0010
    Epoch [38/50], Step [80/83], Loss: 0.0008

    train-loss: 0.0013, train-c-acc: 0.9944, train-cy-acc: 0.9964
    validation loss: 0.0166, validation-c-acc: 0.9786, validation-y-acc: 0.8254
    Improvement-Detected, save-model
    test loss: 0.0158, test-c-acc: 0.9791, test-y-acc: 0.8331

    Epoch 39

    Epoch [39/50], Step [0/83], Loss: 0.0009
    Epoch [39/50], Step [20/83], Loss: 0.0011
    Epoch [39/50], Step [40/83], Loss: 0.0012
    Epoch [39/50], Step [60/83], Loss: 0.0010
    Epoch [39/50], Step [80/83], Loss: 0.0008

    train-loss: 0.0012, train-c-acc: 0.9944, train-cy-acc: 0.9965
    validation loss: 0.0166, validation-c-acc: 0.9786, validation-y-acc: 0.8249
    Improvement-Detected, save-model
    test loss: 0.0158, test-c-acc: 0.9791, test-y-acc: 0.8329

    Epoch 40

    Epoch [40/50], Step [0/83], Loss: 0.0009
    Epoch [40/50], Step [20/83], Loss: 0.0010
    Epoch [40/50], Step [40/83], Loss: 0.0012
    Epoch [40/50], Step [60/83], Loss: 0.0009
    Epoch [40/50], Step [80/83], Loss: 0.0008

    train-loss: 0.0012, train-c-acc: 0.9944, train-cy-acc: 0.9965
    validation loss: 0.0165, validation-c-acc: 0.9786, validation-y-acc: 0.8252
    Improvement-Detected, save-model
    test loss: 0.0158, test-c-acc: 0.9791, test-y-acc: 0.8340

    Epoch 41

    Epoch [41/50], Step [0/83], Loss: 0.0008
    Epoch [41/50], Step [20/83], Loss: 0.0010
    Epoch [41/50], Step [40/83], Loss: 0.0012
    Epoch [41/50], Step [60/83], Loss: 0.0009
    Epoch [41/50], Step [80/83], Loss: 0.0008

    train-loss: 0.0012, train-c-acc: 0.9944, train-cy-acc: 0.9966
    validation loss: 0.0165, validation-c-acc: 0.9786, validation-y-acc: 0.8255
    Improvement-Detected, save-model
    test loss: 0.0158, test-c-acc: 0.9791, test-y-acc: 0.8342

    Epoch 42

    Epoch [42/50], Step [0/83], Loss: 0.0008
    Epoch [42/50], Step [20/83], Loss: 0.0010
    Epoch [42/50], Step [40/83], Loss: 0.0012
    Epoch [42/50], Step [60/83], Loss: 0.0009
    Epoch [42/50], Step [80/83], Loss: 0.0008

    train-loss: 0.0012, train-c-acc: 0.9944, train-cy-acc: 0.9967
    validation loss: 0.0164, validation-c-acc: 0.9786, validation-y-acc: 0.8261
    Improvement-Detected, save-model
    test loss: 0.0157, test-c-acc: 0.9791, test-y-acc: 0.8344

    Epoch 43

    Epoch [43/50], Step [0/83], Loss: 0.0008
    Epoch [43/50], Step [20/83], Loss: 0.0010
    Epoch [43/50], Step [40/83], Loss: 0.0011
    Epoch [43/50], Step [60/83], Loss: 0.0009
    Epoch [43/50], Step [80/83], Loss: 0.0008

    train-loss: 0.0011, train-c-acc: 0.9944, train-cy-acc: 0.9968
    validation loss: 0.0164, validation-c-acc: 0.9786, validation-y-acc: 0.8269
    Improvement-Detected, save-model
    test loss: 0.0156, test-c-acc: 0.9791, test-y-acc: 0.8353

    Epoch 44

    Epoch [44/50], Step [0/83], Loss: 0.0008
    Epoch [44/50], Step [20/83], Loss: 0.0009
    Epoch [44/50], Step [40/83], Loss: 0.0011
    Epoch [44/50], Step [60/83], Loss: 0.0009
    Epoch [44/50], Step [80/83], Loss: 0.0007

    train-loss: 0.0011, train-c-acc: 0.9944, train-cy-acc: 0.9968
    validation loss: 0.0163, validation-c-acc: 0.9786, validation-y-acc: 0.8269
    Improvement-Detected, save-model
    test loss: 0.0156, test-c-acc: 0.9791, test-y-acc: 0.8353

    Epoch 45

    Epoch [45/50], Step [0/83], Loss: 0.0008
    Epoch [45/50], Step [20/83], Loss: 0.0009
    Epoch [45/50], Step [40/83], Loss: 0.0011
    Epoch [45/50], Step [60/83], Loss: 0.0009
    Epoch [45/50], Step [80/83], Loss: 0.0007

    train-loss: 0.0011, train-c-acc: 0.9944, train-cy-acc: 0.9968
    validation loss: 0.0163, validation-c-acc: 0.9786, validation-y-acc: 0.8270
    Improvement-Detected, save-model
    test loss: 0.0156, test-c-acc: 0.9791, test-y-acc: 0.8363

    Epoch 46

    Epoch [46/50], Step [0/83], Loss: 0.0008
    Epoch [46/50], Step [20/83], Loss: 0.0009
    Epoch [46/50], Step [40/83], Loss: 0.0011
    Epoch [46/50], Step [60/83], Loss: 0.0008
    Epoch [46/50], Step [80/83], Loss: 0.0007

    train-loss: 0.0011, train-c-acc: 0.9944, train-cy-acc: 0.9969
    validation loss: 0.0163, validation-c-acc: 0.9786, validation-y-acc: 0.8271
    Improvement-Detected, save-model
    test loss: 0.0155, test-c-acc: 0.9791, test-y-acc: 0.8362

    Epoch 47

    Epoch [47/50], Step [0/83], Loss: 0.0007
    Epoch [47/50], Step [20/83], Loss: 0.0009
    Epoch [47/50], Step [40/83], Loss: 0.0010
    Epoch [47/50], Step [60/83], Loss: 0.0008
    Epoch [47/50], Step [80/83], Loss: 0.0007

    train-loss: 0.0011, train-c-acc: 0.9944, train-cy-acc: 0.9969
    validation loss: 0.0162, validation-c-acc: 0.9786, validation-y-acc: 0.8273
    Improvement-Detected, save-model
    test loss: 0.0155, test-c-acc: 0.9791, test-y-acc: 0.8360

    Epoch 48

    Epoch [48/50], Step [0/83], Loss: 0.0007
    Epoch [48/50], Step [20/83], Loss: 0.0009
    Epoch [48/50], Step [40/83], Loss: 0.0010
    Epoch [48/50], Step [60/83], Loss: 0.0008
    Epoch [48/50], Step [80/83], Loss: 0.0007

    train-loss: 0.0011, train-c-acc: 0.9944, train-cy-acc: 0.9970
    validation loss: 0.0162, validation-c-acc: 0.9786, validation-y-acc: 0.8267
    Improvement-Detected, save-model
    test loss: 0.0155, test-c-acc: 0.9791, test-y-acc: 0.8362

    Epoch 49

    Epoch [49/50], Step [0/83], Loss: 0.0007
    Epoch [49/50], Step [20/83], Loss: 0.0008
    Epoch [49/50], Step [40/83], Loss: 0.0010
    Epoch [49/50], Step [60/83], Loss: 0.0008
    Epoch [49/50], Step [80/83], Loss: 0.0007

    train-loss: 0.0010, train-c-acc: 0.9944, train-cy-acc: 0.9970
    validation loss: 0.0162, validation-c-acc: 0.9786, validation-y-acc: 0.8269
    Improvement-Detected, save-model
    test loss: 0.0154, test-c-acc: 0.9791, test-y-acc: 0.8369

    Epoch 50

    Epoch [50/50], Step [0/83], Loss: 0.0007
    Epoch [50/50], Step [20/83], Loss: 0.0008
    Epoch [50/50], Step [40/83], Loss: 0.0010
    Epoch [50/50], Step [60/83], Loss: 0.0008
    Epoch [50/50], Step [80/83], Loss: 0.0007

    train-loss: 0.0010, train-c-acc: 0.9944, train-cy-acc: 0.9971
    validation loss: 0.0161, validation-c-acc: 0.9786, validation-y-acc: 0.8271
    Improvement-Detected, save-model
    test loss: 0.0154, test-c-acc: 0.9791, test-y-acc: 0.8369

```python
n = 1000
c_preds = net[0].cpu()(x_test[:n].cpu()).detach()
y_preds = net[1].cpu()(c_preds.cpu()).detach()
model_accuracy = accuracy_score(y_test[:n].cpu().detach().argmax(axis=1).numpy(), y_preds.argmax(axis=1).numpy())
print(f'Model\'s accuracy: {model_accuracy:.4f}')
```

    Model's accuracy: 0.9500

```python
# positive class
start = time.time()
class_explanation, class_explanations = lens.logic.explain_class(net[1].cpu(),
                                                               c_preds.cpu(),
                                                               y_val[:n].cpu(),
                                                               binary=False, target_class=0,
                                                               topk_explanations=1)
elapsed_time = time.time() - start
explanation = logic.base.replace_names(class_explanation, concepts)
print(f'Class 1 - Global explanation: "{explanation}"')
```

    Class 1 - Global explanation: "square & very small & ~ellipse & ~heart & ~small & ~s-medium & ~b-medium & ~big & ~very big"

```python
if class_explanation:
    explanation = logic.base.replace_names(class_explanation, concepts)
    explanation_accuracy, y_formula = logic.base.test_explanation(class_explanation,
                                                                  target_class=0,
                                                                  x=c_test[:n], y=y_test[:n],
                                                                  metric=accuracy_score)

    c_preds = net[0].cpu()(x_test[:n].cpu()).detach()
    y_preds = net[1].cpu()(c_preds.cpu()).detach()
    explanation_fidelity = lens.logic.fidelity(y_formula, y_preds.argmax(axis=1).eq(0).numpy())
    explanation_complexity = lens.logic.complexity(class_explanation)

    print(f'Class 1 - Global explanation: "{explanation}" - Accuracy: {explanation_accuracy:.4f}')
    print(f'Fidelity: "{explanation_fidelity:.4f}" - Complexity: "{explanation_complexity}"')
    print(f'Elapsed time {elapsed_time}')
```

    Class 1 - Global explanation: "square & very small & ~ellipse & ~heart & ~small & ~s-medium & ~b-medium & ~big & ~very big" - Accuracy: 1.0000
    Fidelity: "0.9980" - Complexity: "9"
    Elapsed time 0.2170257568359375