# Multi Layer Perceptron (MLP) の実装

# 目次

1. [全体像](#pipeline)
1. [目標](#goal)
- [下準備](#prepare)
- [活性化関数](#activate)
- [線形層](#linear)
- [多層パーセプトロン](#mlp)
- [学習](#train)

# 1. 全体像
- 変数のあとの括弧は，行列の形状 (numpy.ndarrayのshape) を表す．
- Nはバッチサイズ

※ バッチ処理とは……画像を1枚ずつ処理するのではなく，複数枚まとめて処理すること．画像1枚は，28×28のサイズから(784, )の形状をもつ1次元配列になるようにリサイズされているが，これをN枚まとめて(N, 784)の2次元配列として扱う．N枚まとめて行列演算を実行することで，1枚ずつ処理するより高速に処理できるようになる．

## 2. 目標
- 多層 (例では3層) のニューラルネットワークを構築する．
- まず**活性化関数**をそれぞれ実装し，次に**線形層**の実装を行い，最後にそれらをまとめて**多層パーセプトロン**の実装を行う．
- 最終的には，下のように各レイヤーの入出力のユニット数と活性化関数を指定するだけでモデルが構築できるようにする．
- 例：3層・隠れ層のユニット数が1000．活性化関数はReLUを用いる場合
```python
model = MLP([Linear(784, 1000, ReLU),
                        Linear(1000, 1000, ReLU),
                        Linear(1000, 10, Softmax)])
```
- こうすると，4層以上への拡張や，活性化関数の変更などがしやすい

## 3. 下準備

### ライブラリのインポート

- matplotlib: 図やグラフの描画など．
- numpy: 行列演算など
- sklearn: scikit-learn．様々な機械学習のモデルが利用できるが，今回はMNISTのデータをダウンロードするのに用いる．

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split
#from __future__ import unicode_literals
#from __future__ import print_function
#from __future__ import division
#from __future__ import absolute_import
```

```python
#from google.colab import drive # driveを接続
#drive.mount('/content/gdrive')
```

```python
# drive中の課題ファイルのあるディレクトリに移動
#%cd /content/gdrive/My Drive/sentanjinnkouchinou/先端人工知能論I_200512

#from test_mlp import *  # テスト用
```

### MNISTデータの読み込み

- データをダウンロードする．一度ダウンロードすると，その後はデータを参照して読み込んでくれるので，毎回ダウンロードしなくても良くなる．
- Xが画像データ，Yが正解データ
- mnistのデータは，0~255のint型で表されているが，これを**255で割って正規化**する．

```python
X, Y = fetch_openml('mnist_784', version=1, data_home="./data/", return_X_y=True)
```

```python
#X = X / 255.
X_average = X.mean(axis = 1)
X_stddev = X.std(axis = 1)
X_numofsample = X.shape[0]
X = (X - X_average.reshape(X_numofsample, 1))/X_stddev.reshape(X_numofsample, 1)
Y = Y.astype("int")
```

### データセットの可視化

```python
for i in range(10):
    plt.subplot(2, 5, i + 1)
    plt.imshow(X[i * 6500].reshape(28, 28), cmap='gray_r')
    plt.axis("off")
```

### 訓練・テストデータに分割

- 訓練データで学習し，同じ訓練データで性能の評価を行うと，訓練データでは良い性能を示すが，データを少しでも変えると性能が低下してしまうことがある（**過学習**）．
- よって，学習する訓練データとは異なるテストデータで性能評価を行う．

```python
train_x, test_x, train_y, test_y = train_test_split(X, Y, test_size=0.2, random_state=2)
train_y = np.eye(10)[train_y].astype(np.int32)
test_y = np.eye(10)[test_y].astype(np.int32)
train_n = train_x.shape[0]
test_n = test_x.shape[0]
```

## 4. 活性化関数の実装

- ここでは，活性化関数として広く知られる Sigmoid関数と，ReLU関数，及び出力層の活性化関数であるSoftmax関数の実装を行う．
- 各関数の詳細については，講義スライドを参照

### 課題：Sigmoid関数の実装

- 順伝播計算

    - $h(x) = \sigma(x) = \dfrac{1}{1+e^{-x}}$

- 逆伝播計算

    - $h'(x) = \sigma'(x) = \sigma(x)\;(1-\sigma(x))$

※ 順伝播の $\sigma(x)$ と，逆伝播の $\sigma(x)$ は同じなので，2回計算しなくてもOK

    ヒント

        - np.exp(x) を用いて$\exp(x)$を計算できる

        - 順伝播の計算結果は， self.y に保存されているので，逆伝播計算ではそれを使おう

```python
class Sigmoid:
    def __init__(self):
        self.y = None

    def __call__(self, x):
        y = 1/(1+np.exp(-x))
        self.y = y
        return y

    def backward(self):
        ry = self.y*(1 - self.y)
        return ry  # 逆伝播計算
```

```python
#test_sigmoid(Sigmoid)
```

### ReLU関数

- 順伝播
    - $h(x) = \max(0, x)$
- 逆伝播
    - $h'(x) =
        \left\{ \begin{array}{ll}
            1 & (x > 0) \\
            0 & (x \leq 0) \\
        \end{array} \right.$

    ヒント

        - ndarrayから，0より大きい要素をTrue, 0以下の要素をFalseとなるようなマスクを作成する

        >>> a = np.array([[-1, 1], [0, 1]])
        >>> print(a)
        [[-1  1]
         [ 0  1]]
        >>> print(a > 0)
        [[False  True]
         [False  True]]
        >>> print(a * (a>0))
        [[0 1]
         [0 1]]

```python
class ReLU:
    def __init__(self):
        self.x = None

    def __call__(self, x):
        self.x = x
        return x * (x>0)  # 順伝播計算

    def backward(self):
        return 1 * (self.x>0)  # 逆伝播計算
```

```python
#test_relu(ReLU)
```

### Softmax関数

- ロジスティック回帰と同様に実装する

- 入力：$\boldsymbol{X}=(\boldsymbol{x_1},\boldsymbol{x_2},\cdots,\boldsymbol{x_N})^T\in\mathbb{R}^{N\times K}$（データ行列）

- 出力：$\boldsymbol{Y}=(\boldsymbol{y_1},\boldsymbol{y_2},\cdots,\boldsymbol{y_N})^T\in\mathbb{R}^{N\times K},\,\,\,y_{nk} = softmax(\boldsymbol{x_n})_k$

- オーバーフローを防ぐために$\boldsymbol{x}_n$の最大値を$\boldsymbol{x}_n$自身から引く

$$
\begin{align}
softmax(\boldsymbol{x})_k&= \frac{\exp (x_{k})} {\Sigma_{i=1}^{K}{\exp (x_{i})}}=\frac{\exp (-x_{max})\exp (x_{k})}{\exp (-x_{max})\Sigma_{i=1}^{K}{\exp (x_{i})}}=\frac{\exp (x_{k}-x_{max})} {\Sigma_{i=1}^{K}{\exp (x_{i}-x_{max})}}
\end{align}
$$

    ヒント: 最大値の取得

        - ndarrayから，最大値を取得したい

        >>> A = np.array([[1, 2], [3, 4], [5, 6]])
        >>> print(A.shape)
        (3, 2)
        >>> A.max()
        6

        - ここでは，N×Kの配列について，1次元目の要素ごとに計N個の最大値を取得したい

        >>> A.max(axis=1)
        array([2, 4, 6])

        - さらに，配列の形状はN×1にしておきたい

        >>> A.max(axis=1, keepdims=True)
        array([[2],
       [4],
       [6]])

```python
class Softmax:
    def __init__(self):
        self.y = None

    def __call__(self, x):
      exp_x = np.exp(x - x.max(axis=1, keepdims = True))
      y = exp_x / np.sum(exp_x, axis=1, keepdims = True)
      self.y = y
      return y
```

```python
#test_softmax(Softmax)
```

### Swish関数（実装途中）

- 入力：$\boldsymbol{X}=(\boldsymbol{x_1},\boldsymbol{x_2},\cdots,\boldsymbol{x_N})^T\in\mathbb{R}^{N\times K}$（データ行列）

- 出力：$\boldsymbol{Y}=(\boldsymbol{y_1},\boldsymbol{y_2},\cdots,\boldsymbol{y_N})^T\in\mathbb{R}^{N\times K},\,\,\,y_{nk} = swish(\boldsymbol{x_n})_k$

$$
\begin{align}
swish(\boldsymbol{x})_k&=\sigma(\boldsymbol{x})x
\end{align}
$$

```python
class swish:
    def __init__(self):
        self.y = None

    def __call__(self, x, beta):
      y = x*(1/(1+np.exp(-beta*x)))
      return y

    def backward(self):
      return   # 逆伝播計算
```

### Mish関数（導関数は手計算）
https://arxiv.org/abs/1908.08681

Mish: A Self Regularized Non-Monotonic Neural Activation Function

- 入力：$\boldsymbol{X}=(\boldsymbol{x_1},\boldsymbol{x_2},\cdots,\boldsymbol{x_N})^T\in\mathbb{R}^{N\times K}$（データ行列）

- 出力：$\boldsymbol{Y}=(\boldsymbol{y_1},\boldsymbol{y_2},\cdots,\boldsymbol{y_N})^T\in\mathbb{R}^{N\times K},\,\,\,y_{nk} = mish(\boldsymbol{x_n})_k$

$$
\begin{align}
mish(x)&=x\cdot \tanh(softplus(\boldsymbol{x}))
\end{align}
$$
$$
\begin{align}
mish'(x)&=\dfrac{e^{x}(4(x+1)+4e^{2x}+e^{3x}+e^{4x}(4x+6))}{(2e^{x}+e^{2x}+2)^2}
\end{align}
$$

```python
def tanh(x):
  return (np.exp(x) - np.exp(-x)) / (np.exp(x) + np.exp(-x))
def softplus(x):
  return np.log(1.0 + np.exp(x))
def omega(x):
  return 4*(x+1) + 4*np.exp(2*x) + np.exp(3*x) + np.exp(x)*(4*x+6)
def delta(x):
  return 2*np.exp(x) + np.exp(2*x) + 2
```

```python
class mish:
    def __init__(self):
        self.x = None

    def __call__(self, x):
        self.x = x
        y = x * tanh(softplus(x))
        return y  # 順伝播計算

    def backward(self):
        y = np.exp(self.x) * omega(self.x) / delta(self.x)**2
        return y  # 逆伝播計算
```

## 5. 線形層
- 引数
    - in_dim : 入力のユニット数
    - out_dim : 出力のユニット数
    - activation : 活性化関数
- 計算
    - 順伝播

        - ヒント : np.dot(A, B)
    - 逆伝播
        - 入力 (dout)
            - 一つ上の層 (l+1層) からの出力 $(\boldsymbol{o}^{(l+1)})$
        - 誤差

            - $\odot$ は要素積
        - 勾配計算

    ヒント: 行列積の計算について

        $
A = \left(
    \begin{array}{ccc}
      0 & 1 & 2 \\
      1 & 2 & 3
    \end{array}
  \right)
$ ,
$
B = \left(
\begin{array}{cc}
  0 & 1\\
  1 & 2 \\
  2 & 3
\end{array}
\right)
$ としたとき， $C = AB$ を計算する例

        >>> A = np.array([[0, 1, 2], [1, 2, 3]])
        >>> B = np.array([[0, 1], [1, 2], [2, 3]])
        >>> print(A.shape, B.shape)
        (2, 3) (3, 2)
        >>> print(a > 0)
        [[False  True]
         [False  True]]
        >>> C = np.dot(A, B)
        >>> print(C)
        [[ 5  8]
         [ 8 14]]

    ヒント: 行列の要素積

        >>> A = np.array([[0, 1], [2, 3]])
        >>> B = np.array([[1, 2], [3, 4]])
        >>> print(A*B)
        [[ 0  2]
         [ 6 12]]

```python
class Linear:
    def __init__(self, in_dim, out_dim, activation):
        self.W = np.random.uniform(low=-0.08, high=0.08, size=(in_dim, out_dim)) #ただの一様分布
        #self.W = np.random.uniform() Xavierの一様分布
        #self.W = np.random.normal(0, np.sqrt(2/in_dim), size=(in_dim, out_dim)) # Heの正規分布，　ReLU用
        #self.W = np.random.uniform(low = -np.sqrt(in_dim), high = np.sqrt(in_dim), size=(in_dim, out_dim)) # Heの一様分布，　ReLU用
        self.b = np.zeros(out_dim)
        self.activation = activation()
        self.delta = None
        self.x = None
        self.dW = None
        self.db = None

    def __call__(self, x, train_flg):
        # 順伝播計算
        self.x = x
        u = np.dot(x, self.W) + self.b  # self.W, self.b, x を用いて u を計算しよう
        self.z = self.activation(u)
        return self.z

    def backward(self, dout):
        # 誤差計算
        self.delta = dout*self.activation.backward() # dout と活性化関数の逆伝播 (self.activation.backward()) を用いて delta を計算しよう
        dout = np.dot(self.delta, self.W.T) # self.delta, self.W を用いて 出力 o を計算しよう

        # 勾配計算
        self.dW = np.dot(self.x.T,self.delta)  # dW を計算しよう
        self.db = np.dot(np.ones(len(self.x)),self.delta)# db を計算しよう

        return dout
```

```python
class Dropout:
    def __init__(self, in_dim, out_dim, dropout_ratio=0.5):
        self.dropout_ratio = dropout_ratio
        self.mask = None
        self.W = np.random.uniform(low=-0.08, high=0.08, size=(in_dim, out_dim))
        self.b = np.zeros(out_dim)
        self.delta = 0
        self.x = 0
        self.dW = 0
        self.db = 0

    def __call__(self, x, train_flg):
      if train_flg:
        self.mask = np.random.rand(*x.shape) > self.dropout_ratio
        return x * self.mask
      else:
        return x * (1.0 - self.dropout_ratio)
    def backward(self, dout):
        return dout * self.mask
# dropoutよくわからん
```

```python
class BatchNormalization:
    def __init__(self, gamma, beta, momentum=0.9, running_mean=None, running_var=None):
        self.gamma = gamma
        self.beta = beta
        self.momentum = momentum
        self.input_shape = None # Conv層の場合は4次元、全結合層の場合は2次元
        self.running_mean = running_mean
        self.running_var = running_var

        # backward時に使用する中間データ
        self.batch_size = None
        self.xc = None
        self.std = None
        self.dgamma = None
        self.dbeta = None

    def forward(self, x, train_flg=True):
        self.input_shape = x.shape
        if x.ndim != 2:
            N, C, H, W = x.shape
            x = x.reshape(N, -1)

        out = self.__forward(x, train_flg)

        return out.reshape(*self.input_shape)

    def __forward(self, x, train_flg):
        if self.running_mean is None:
            N, D = x.shape
            self.running_mean = np.zeros(D)
            self.running_var = np.zeros(D)

        if train_flg:
            mu = x.mean(axis=0)
            xc = x - mu
            var = np.mean(xc**2, axis=0)
            std = np.sqrt(var + 10e-7)
            xn = xc / std

            self.batch_size = x.shape[0]
            self.xc = xc
            self.xn = xn
            self.std = std
            self.running_mean = self.momentum * self.running_mean + (1-self.momentum) * mu
            self.running_var = self.momentum * self.running_var + (1-self.momentum) * var
        else:
            xc = x - self.running_mean
            xn = xc / ((np.sqrt(self.running_var + 10e-7)))

        out = self.gamma * xn + self.beta
        return out

    def backward(self, dout):
        if dout.ndim != 2:
            N, C, H, W = dout.shape
            dout = dout.reshape(N, -1)

        dx = self.__backward(dout)

        dx = dx.reshape(*self.input_shape)
        return dx

```

## 6. 多層パーセプトロンの実装
- 今までに実装してきた活性化関数，Linear層を組み合わせて，多層のパーセプトロンを実装する．
- ここでは，先に実装したLinear層を組み合わせ，全体を通した**順伝播計算**と，**損失の計算**，**誤差逆伝播計算**，及び**重み・バイアスの更新**を実装する．
- 例：3層・隠れ層のユニット数が1000．活性化関数はReLUを用いる場合
```python
model = MLP([Linear(784, 1000, ReLU),
                        Linear(1000, 1000, ReLU),
                        Linear(1000, 10, Softmax)])
```
- **lr** : 学習率 (learning rate)．学習率とは，重み・バイアスの更新量を決定するハイパーパラメータ．つまり，パラメータ更新量 = 学習率 × 現在の勾配

### 全体像の再掲

### 課題 : 多層パーセプトロン

```python
class MLP():
    def __init__(self, layers):
        self.layers = layers

    def train(self, x, t, lr, train_flg=True):
        # 1. 順伝播
        self.y = x
        for layer in self.layers:
            self.y = layer(self.y, train_flg)  # 順伝播計算を順番に行い， 出力 y を計算しよう

        # 2. 損失関数の計算
        self.loss = np.sum(-t*np.log(self.y + 1e-7)) / len(x)

        # 3. 誤差逆伝播
        # 3.1. 最終層
        # 3.1.1. 最終層の誤差・勾配計算
        batchsize = len(self.layers[-1].x)
        delta = (self.y - t) / batchsize
        self.layers[-1].delta = delta
        self.layers[-1].dW = np.dot(self.layers[-1].x.T, self.layers[-1].delta)
        self.layers[-1].db = np.dot(np.ones(batchsize), self.layers[-1].delta)
        dout = np.dot(self.layers[-1].delta, self.layers[-1].W.T)

        # 3.1.2. 最終層のパラメータ更新
        self.layers[-1].W -= lr * self.layers[-1].dW # self.layers[-1].dW を用いて最終層の重みを更新しよう
        self.layers[-1].b -= lr * self.layers[-1].db  # self.layers[-1].db を用いて最終層のバイアスを更新しよう

        # 3.2. 中間層
        for layer in self.layers[-2::-1]:
            # 3.2.1. 中間層の誤差・勾配計算
            dout = layer.backward(dout)  # 逆伝播計算を順番に実行しよう

            # 3.2.2. パラメータの更新
            layer.W -= lr * layer.dW  # 各層の重みを更新
            layer.b -= lr *layer.db # 各層のバイアスを更新

        return self.loss

    def test(self, x, t, train_flg=False):
        # 性能をテストデータで調べるために用いる
        # よって，誤差逆伝播は不要
        # 順伝播 (train関数と同様)
        self.y = x
        for layer in self.layers:
            self.y = layer(self.y, train_flg)
        self.loss = np.sum(-t*np.log(self.y + 1e-7)) / len(x)
        return self.loss
```

## モデルの構築
- ここでは，図に示してきたような3層のニューラルネットワークを構築する
- 活性化関数はSigmoid関数とし， 隠れ層のニューロン数はいずれも1000とする．

# 7. 学習

- n_epoch : エポック数．1エポックとは，学習時に訓練データをすべて学習した回数を表す．
- batchsize: バッチサイズ．
- lr: 学習率 (learning rate)．

```python
model = MLP([
Linear(784, 1000, ReLU),
Dropout(1000, 1000, 0.2),
Linear(1000, 1000, ReLU),
Dropout(1000, 1000, 0.5),
Linear(1000, 1000, ReLU),
Dropout(1000, 1000, 0.5),
Linear(1000, 1000, ReLU),
Dropout(1000, 1000, 0.5),
Linear(1000, 1000, ReLU),
Dropout(1000, 1000, 0.5),
Linear(1000, 10, Softmax)])
batchsize = 100
for lr, n_epoch in [[0.01, 300]]:
  print('n_epoch={}, batchsize={}, lr={}'.format(n_epoch,batchsize,lr))
  for epoch in range(n_epoch):
      print('epoch %d | ' % epoch, end="")

      # 訓練
      sum_loss = 0
      pred_y = []
      perm = np.random.permutation(train_n)

      for i in range(0, train_n, batchsize):
          x = train_x[perm[i: i+batchsize]]
          t = train_y[perm[i: i+batchsize]]
          sum_loss += model.train(x, t, lr) * len(x)
          # model.y には， (N, 10)の形で，画像が0~9の各数字のどれに分類されるかの事後確率が入っている
          # そこで，最も大きい値をもつインデックスを取得することで，識別結果を得ることができる
          pred_y.extend(np.argmax(model.y, axis=1))

      loss = sum_loss / train_n

      # accuracy : 予測結果を1-hot表現に変換し，正解との要素積の和を取ることで，正解数を計算できる．
      accuracy = np.sum(np.eye(10)[pred_y] * train_y[perm]) / train_n
      print('Train loss %.3f, accuracy %.4f | ' %(loss, accuracy), end="")

      # テスト
      sum_loss = 0
      pred_y = []

      for i in range(0, test_n, batchsize):
          x = test_x[i: i+batchsize]
          t = test_y[i: i+batchsize]

          sum_loss += model.test(x, t) * len(x)
          pred_y.extend(np.argmax(model.y, axis=1))

      loss = sum_loss / test_n
      accuracy = np.sum(np.eye(10)[pred_y] * test_y) / test_n
      print('Test loss %.3f, accuracy %.4f' %(loss, accuracy))

```

    n_epoch=300, batchsize=100, lr=0.01
    epoch 0 | Train loss 2.272, accuracy 0.1531 | Test loss 2.089, accuracy 0.5879
    epoch 1 | Train loss 1.924, accuracy 0.3443 | Test loss 1.224, accuracy 0.6434
    epoch 2 | Train loss 1.197, accuracy 0.5757 | Test loss 0.696, accuracy 0.7644
    epoch 3 | Train loss 0.852, accuracy 0.7055 | Test loss 0.521, accuracy 0.8329
    epoch 4 | Train loss 0.676, accuracy 0.7773 | Test loss 0.423, accuracy 0.8677
    epoch 5 | Train loss 0.567, accuracy 0.8170 | Test loss 0.359, accuracy 0.8896
    epoch 6 | Train loss 0.498, accuracy 0.8426 | Test loss 0.319, accuracy 0.9043
    epoch 7 | Train loss 0.444, accuracy 0.8633 | Test loss 0.292, accuracy 0.9129
    epoch 8 | Train loss 0.408, accuracy 0.8762 | Test loss 0.271, accuracy 0.9181
    epoch 9 | Train loss 0.376, accuracy 0.8851 | Test loss 0.253, accuracy 0.9243
    epoch 10 | Train loss 0.350, accuracy 0.8938 | Test loss 0.234, accuracy 0.9301
    epoch 11 | Train loss 0.327, accuracy 0.9026 | Test loss 0.218, accuracy 0.9345
    epoch 12 | Train loss 0.311, accuracy 0.9063 | Test loss 0.209, accuracy 0.9371
    epoch 13 | Train loss 0.292, accuracy 0.9116 | Test loss 0.196, accuracy 0.9417
    epoch 14 | Train loss 0.279, accuracy 0.9161 | Test loss 0.186, accuracy 0.9443
    epoch 15 | Train loss 0.265, accuracy 0.9214 | Test loss 0.177, accuracy 0.9471
    epoch 16 | Train loss 0.255, accuracy 0.9239 | Test loss 0.171, accuracy 0.9489
    epoch 17 | Train loss 0.241, accuracy 0.9282 | Test loss 0.166, accuracy 0.9506
    epoch 18 | Train loss 0.232, accuracy 0.9301 | Test loss 0.159, accuracy 0.9528
    epoch 19 | Train loss 0.226, accuracy 0.9331 | Test loss 0.153, accuracy 0.9549
    epoch 20 | Train loss 0.216, accuracy 0.9360 | Test loss 0.146, accuracy 0.9564
    epoch 21 | Train loss 0.206, accuracy 0.9386 | Test loss 0.142, accuracy 0.9579
    epoch 22 | Train loss 0.201, accuracy 0.9414 | Test loss 0.138, accuracy 0.9589
    epoch 23 | Train loss 0.194, accuracy 0.9416 | Test loss 0.134, accuracy 0.9601
    epoch 24 | Train loss 0.186, accuracy 0.9450 | Test loss 0.131, accuracy 0.9609
    epoch 25 | Train loss 0.180, accuracy 0.9471 | Test loss 0.128, accuracy 0.9614
    epoch 26 | Train loss 0.173, accuracy 0.9474 | Test loss 0.125, accuracy 0.9626
    epoch 27 | Train loss 0.169, accuracy 0.9487 | Test loss 0.122, accuracy 0.9636
    epoch 28 | Train loss 0.164, accuracy 0.9514 | Test loss 0.118, accuracy 0.9645
    epoch 29 | Train loss 0.159, accuracy 0.9529 | Test loss 0.116, accuracy 0.9651
    epoch 30 | Train loss 0.157, accuracy 0.9537 | Test loss 0.114, accuracy 0.9657
    epoch 31 | Train loss 0.150, accuracy 0.9553 | Test loss 0.112, accuracy 0.9664
    epoch 32 | Train loss 0.148, accuracy 0.9563 | Test loss 0.109, accuracy 0.9674
    epoch 33 | Train loss 0.146, accuracy 0.9573 | Test loss 0.107, accuracy 0.9676
    epoch 34 | Train loss 0.143, accuracy 0.9577 | Test loss 0.104, accuracy 0.9687
    epoch 35 | Train loss 0.138, accuracy 0.9593 | Test loss 0.102, accuracy 0.9691
    epoch 36 | Train loss 0.134, accuracy 0.9590 | Test loss 0.101, accuracy 0.9695
    epoch 37 | Train loss 0.131, accuracy 0.9605 | Test loss 0.100, accuracy 0.9703
    epoch 38 | Train loss 0.128, accuracy 0.9618 | Test loss 0.098, accuracy 0.9706
    epoch 39 | Train loss 0.124, accuracy 0.9628 | Test loss 0.097, accuracy 0.9710
    epoch 40 | Train loss 0.123, accuracy 0.9633 | Test loss 0.095, accuracy 0.9717
    epoch 41 | Train loss 0.117, accuracy 0.9652 | Test loss 0.093, accuracy 0.9724
    epoch 42 | Train loss 0.116, accuracy 0.9659 | Test loss 0.091, accuracy 0.9725
    epoch 43 | Train loss 0.111, accuracy 0.9666 | Test loss 0.090, accuracy 0.9725
    epoch 44 | Train loss 0.112, accuracy 0.9669 | Test loss 0.089, accuracy 0.9733
    epoch 45 | Train loss 0.110, accuracy 0.9676 | Test loss 0.089, accuracy 0.9733
    epoch 46 | Train loss 0.106, accuracy 0.9681 | Test loss 0.088, accuracy 0.9742
    epoch 47 | Train loss 0.101, accuracy 0.9695 | Test loss 0.086, accuracy 0.9745
    epoch 48 | Train loss 0.103, accuracy 0.9695 | Test loss 0.085, accuracy 0.9746
    epoch 49 | Train loss 0.098, accuracy 0.9709 | Test loss 0.085, accuracy 0.9746
    epoch 50 | Train loss 0.097, accuracy 0.9706 | Test loss 0.084, accuracy 0.9749
    epoch 51 | Train loss 0.096, accuracy 0.9714 | Test loss 0.083, accuracy 0.9754
    epoch 52 | Train loss 0.092, accuracy 0.9727 | Test loss 0.082, accuracy 0.9754
    epoch 53 | Train loss 0.093, accuracy 0.9714 | Test loss 0.082, accuracy 0.9751
    epoch 54 | Train loss 0.090, accuracy 0.9733 | Test loss 0.080, accuracy 0.9756
    epoch 55 | Train loss 0.088, accuracy 0.9729 | Test loss 0.080, accuracy 0.9756
    epoch 56 | Train loss 0.086, accuracy 0.9739 | Test loss 0.080, accuracy 0.9758
    epoch 57 | Train loss 0.085, accuracy 0.9739 | Test loss 0.079, accuracy 0.9756
    epoch 58 | Train loss 0.082, accuracy 0.9754 | Test loss 0.078, accuracy 0.9757
    epoch 59 | Train loss 0.081, accuracy 0.9751 | Test loss 0.077, accuracy 0.9766
    epoch 60 | Train loss 0.081, accuracy 0.9754 | Test loss 0.077, accuracy 0.9766
    epoch 61 | Train loss 0.079, accuracy 0.9755 | Test loss 0.076, accuracy 0.9771
    epoch 62 | Train loss 0.075, accuracy 0.9768 | Test loss 0.075, accuracy 0.9768
    epoch 63 | Train loss 0.075, accuracy 0.9768 | Test loss 0.075, accuracy 0.9771
    epoch 64 | Train loss 0.072, accuracy 0.9774 | Test loss 0.075, accuracy 0.9778
    epoch 65 | Train loss 0.073, accuracy 0.9778 | Test loss 0.074, accuracy 0.9771
    epoch 66 | Train loss 0.072, accuracy 0.9778 | Test loss 0.074, accuracy 0.9779
    epoch 67 | Train loss 0.068, accuracy 0.9785 | Test loss 0.073, accuracy 0.9781
    epoch 68 | Train loss 0.067, accuracy 0.9794 | Test loss 0.073, accuracy 0.9779
    epoch 69 | Train loss 0.069, accuracy 0.9785 | Test loss 0.072, accuracy 0.9782
    epoch 70 | Train loss 0.065, accuracy 0.9803 | Test loss 0.072, accuracy 0.9781
    epoch 71 | Train loss 0.065, accuracy 0.9800 | Test loss 0.072, accuracy 0.9782
    epoch 72 | Train loss 0.062, accuracy 0.9808 | Test loss 0.072, accuracy 0.9789
    epoch 73 | Train loss 0.064, accuracy 0.9805 | Test loss 0.071, accuracy 0.9784
    epoch 74 | Train loss 0.061, accuracy 0.9811 | Test loss 0.071, accuracy 0.9786
    epoch 75 | Train loss 0.059, accuracy 0.9814 | Test loss 0.070, accuracy 0.9789
    epoch 76 | Train loss 0.056, accuracy 0.9827 | Test loss 0.071, accuracy 0.9790
    epoch 77 | Train loss 0.057, accuracy 0.9824 | Test loss 0.071, accuracy 0.9796
    epoch 78 | Train loss 0.056, accuracy 0.9827 | Test loss 0.070, accuracy 0.9791
    epoch 79 | Train loss 0.056, accuracy 0.9829 | Test loss 0.071, accuracy 0.9791
    epoch 80 | Train loss 0.054, accuracy 0.9831 | Test loss 0.069, accuracy 0.9793
    epoch 81 | Train loss 0.053, accuracy 0.9834 | Test loss 0.069, accuracy 0.9801
    epoch 82 | Train loss 0.052, accuracy 0.9845 | Test loss 0.069, accuracy 0.9799
    epoch 83 | Train loss 0.052, accuracy 0.9837 | Test loss 0.069, accuracy 0.9804
    epoch 84 | Train loss 0.052, accuracy 0.9839 | Test loss 0.070, accuracy 0.9794
    epoch 85 | Train loss 0.050, accuracy 0.9839 | Test loss 0.069, accuracy 0.9795
    epoch 86 | Train loss 0.048, accuracy 0.9853 | Test loss 0.069, accuracy 0.9794
    epoch 87 | Train loss 0.049, accuracy 0.9847 | Test loss 0.069, accuracy 0.9802
    epoch 88 | Train loss 0.047, accuracy 0.9854 | Test loss 0.068, accuracy 0.9808
    epoch 89 | Train loss 0.046, accuracy 0.9857 | Test loss 0.069, accuracy 0.9802
    epoch 90 | Train loss 0.047, accuracy 0.9851 | Test loss 0.068, accuracy 0.9804
    epoch 91 | Train loss 0.046, accuracy 0.9854 | Test loss 0.068, accuracy 0.9803
    epoch 92 | Train loss 0.043, accuracy 0.9866 | Test loss 0.068, accuracy 0.9809
    epoch 93 | Train loss 0.045, accuracy 0.9862 | Test loss 0.068, accuracy 0.9809
    epoch 94 | Train loss 0.041, accuracy 0.9872 | Test loss 0.068, accuracy 0.9807
    epoch 95 | Train loss 0.042, accuracy 0.9865 | Test loss 0.068, accuracy 0.9809
    epoch 96 | Train loss 0.041, accuracy 0.9867 | Test loss 0.067, accuracy 0.9809
    epoch 97 | Train loss 0.041, accuracy 0.9868 | Test loss 0.068, accuracy 0.9809
    epoch 98 | Train loss 0.039, accuracy 0.9879 | Test loss 0.068, accuracy 0.9809
    epoch 99 | Train loss 0.040, accuracy 0.9875 | Test loss 0.068, accuracy 0.9808
    epoch 100 | Train loss 0.039, accuracy 0.9877 | Test loss 0.068, accuracy 0.9807
    epoch 101 | Train loss 0.039, accuracy 0.9883 | Test loss 0.068, accuracy 0.9811
    epoch 102 | Train loss 0.037, accuracy 0.9883 | Test loss 0.068, accuracy 0.9814
    epoch 103 | Train loss 0.036, accuracy 0.9887 | Test loss 0.068, accuracy 0.9814
    epoch 104 | Train loss 0.035, accuracy 0.9887 | Test loss 0.067, accuracy 0.9817
    epoch 105 | Train loss 0.036, accuracy 0.9892 | Test loss 0.067, accuracy 0.9813
    epoch 106 | Train loss 0.033, accuracy 0.9889 | Test loss 0.068, accuracy 0.9812
    epoch 107 | Train loss 0.034, accuracy 0.9892 | Test loss 0.068, accuracy 0.9816
    epoch 108 | Train loss 0.035, accuracy 0.9891 | Test loss 0.067, accuracy 0.9821
    epoch 109 | Train loss 0.033, accuracy 0.9892 | Test loss 0.067, accuracy 0.9821
    epoch 110 | Train loss 0.032, accuracy 0.9897 | Test loss 0.067, accuracy 0.9821
    epoch 111 | Train loss 0.032, accuracy 0.9894 | Test loss 0.067, accuracy 0.9821
    epoch 112 | Train loss 0.031, accuracy 0.9903 | Test loss 0.067, accuracy 0.9819
    epoch 113 | Train loss 0.030, accuracy 0.9909 | Test loss 0.068, accuracy 0.9816
    epoch 114 | Train loss 0.031, accuracy 0.9900 | Test loss 0.067, accuracy 0.9820
    epoch 115 | Train loss 0.029, accuracy 0.9908 | Test loss 0.068, accuracy 0.9819
    epoch 116 | Train loss 0.031, accuracy 0.9904 | Test loss 0.067, accuracy 0.9820
    epoch 117 | Train loss 0.029, accuracy 0.9912 | Test loss 0.067, accuracy 0.9819
    epoch 118 | Train loss 0.028, accuracy 0.9912 | Test loss 0.068, accuracy 0.9819
    epoch 119 | Train loss 0.029, accuracy 0.9904 | Test loss 0.068, accuracy 0.9818
    epoch 120 | Train loss 0.028, accuracy 0.9908 | Test loss 0.069, accuracy 0.9824
    epoch 121 | Train loss 0.028, accuracy 0.9913 | Test loss 0.068, accuracy 0.9822
    epoch 122 | Train loss 0.026, accuracy 0.9917 | Test loss 0.069, accuracy 0.9819
    epoch 123 | Train loss 0.028, accuracy 0.9915 | Test loss 0.068, accuracy 0.9824
    epoch 124 | Train loss 0.026, accuracy 0.9915 | Test loss 0.068, accuracy 0.9819
    epoch 125 | Train loss 0.025, accuracy 0.9921 | Test loss 0.069, accuracy 0.9819
    epoch 126 | Train loss 0.026, accuracy 0.9917 | Test loss 0.069, accuracy 0.9816
    epoch 127 | Train loss 0.025, accuracy 0.9922 | Test loss 0.069, accuracy 0.9821
    epoch 128 | Train loss 0.025, accuracy 0.9925 | Test loss 0.070, accuracy 0.9822
    epoch 129 | Train loss 0.025, accuracy 0.9921 | Test loss 0.070, accuracy 0.9818
    epoch 130 | Train loss 0.023, accuracy 0.9929 | Test loss 0.069, accuracy 0.9821
    epoch 131 | Train loss 0.024, accuracy 0.9925 | Test loss 0.069, accuracy 0.9817
    epoch 132 | Train loss 0.023, accuracy 0.9928 | Test loss 0.070, accuracy 0.9821
    epoch 133 | Train loss 0.022, accuracy 0.9931 | Test loss 0.070, accuracy 0.9825
    epoch 134 | Train loss 0.023, accuracy 0.9925 | Test loss 0.069, accuracy 0.9824
    epoch 135 | Train loss 0.022, accuracy 0.9931 | Test loss 0.068, accuracy 0.9828
    epoch 136 | Train loss 0.022, accuracy 0.9932 | Test loss 0.069, accuracy 0.9826
    epoch 137 | Train loss 0.022, accuracy 0.9931 | Test loss 0.069, accuracy 0.9828
    epoch 138 | Train loss 0.021, accuracy 0.9935 | Test loss 0.069, accuracy 0.9827
    epoch 139 | Train loss 0.022, accuracy 0.9931 | Test loss 0.070, accuracy 0.9821
    epoch 140 | Train loss 0.022, accuracy 0.9930 | Test loss 0.069, accuracy 0.9827
    epoch 141 | Train loss 0.020, accuracy 0.9938 | Test loss 0.070, accuracy 0.9823
    epoch 142 | Train loss 0.020, accuracy 0.9940 | Test loss 0.071, accuracy 0.9825
    epoch 143 | Train loss 0.021, accuracy 0.9936 | Test loss 0.070, accuracy 0.9824
    epoch 144 | Train loss 0.019, accuracy 0.9941 | Test loss 0.070, accuracy 0.9826
    epoch 145 | Train loss 0.019, accuracy 0.9941 | Test loss 0.070, accuracy 0.9826
    epoch 146 | Train loss 0.020, accuracy 0.9939 | Test loss 0.070, accuracy 0.9823
    epoch 147 | Train loss 0.020, accuracy 0.9936 | Test loss 0.070, accuracy 0.9825
    epoch 148 | Train loss 0.019, accuracy 0.9937 | Test loss 0.069, accuracy 0.9829
    epoch 149 | Train loss 0.020, accuracy 0.9937 | Test loss 0.069, accuracy 0.9829
    epoch 150 | Train loss 0.018, accuracy 0.9947 | Test loss 0.070, accuracy 0.9827
    epoch 151 | Train loss 0.018, accuracy 0.9944 | Test loss 0.071, accuracy 0.9823
    epoch 152 | Train loss 0.018, accuracy 0.9944 | Test loss 0.071, accuracy 0.9829
    epoch 153 | Train loss 0.017, accuracy 0.9942 | Test loss 0.070, accuracy 0.9828
    epoch 154 | Train loss 0.017, accuracy 0.9946 | Test loss 0.070, accuracy 0.9829
    epoch 155 | Train loss 0.017, accuracy 0.9946 | Test loss 0.070, accuracy 0.9831
    epoch 156 | Train loss 0.017, accuracy 0.9946 | Test loss 0.071, accuracy 0.9829
    epoch 157 | Train loss 0.016, accuracy 0.9952 | Test loss 0.071, accuracy 0.9831
    epoch 158 | Train loss 0.017, accuracy 0.9948 | Test loss 0.071, accuracy 0.9829
    epoch 159 | Train loss 0.016, accuracy 0.9948 | Test loss 0.071, accuracy 0.9828
    epoch 160 | Train loss 0.017, accuracy 0.9945 | Test loss 0.071, accuracy 0.9829
    epoch 161 | Train loss 0.016, accuracy 0.9950 | Test loss 0.070, accuracy 0.9831
    epoch 162 | Train loss 0.017, accuracy 0.9945 | Test loss 0.070, accuracy 0.9830
    epoch 163 | Train loss 0.016, accuracy 0.9952 | Test loss 0.072, accuracy 0.9824
    epoch 164 | Train loss 0.017, accuracy 0.9948 | Test loss 0.071, accuracy 0.9829
    epoch 165 | Train loss 0.015, accuracy 0.9950 | Test loss 0.070, accuracy 0.9831
    epoch 166 | Train loss 0.016, accuracy 0.9949 | Test loss 0.072, accuracy 0.9825
    epoch 167 | Train loss 0.016, accuracy 0.9950 | Test loss 0.072, accuracy 0.9826
    epoch 168 | Train loss 0.016, accuracy 0.9947 | Test loss 0.072, accuracy 0.9828
    epoch 169 | Train loss 0.014, accuracy 0.9951 | Test loss 0.070, accuracy 0.9829
    epoch 170 | Train loss 0.015, accuracy 0.9956 | Test loss 0.072, accuracy 0.9829
    epoch 171 | Train loss 0.014, accuracy 0.9957 | Test loss 0.072, accuracy 0.9825
    epoch 172 | Train loss 0.014, accuracy 0.9957 | Test loss 0.072, accuracy 0.9836
    epoch 173 | Train loss 0.015, accuracy 0.9955 | Test loss 0.071, accuracy 0.9831
    epoch 174 | Train loss 0.014, accuracy 0.9956 | Test loss 0.072, accuracy 0.9830
    epoch 175 | Train loss 0.015, accuracy 0.9953 | Test loss 0.071, accuracy 0.9834
    epoch 176 | Train loss 0.013, accuracy 0.9957 | Test loss 0.072, accuracy 0.9832
    epoch 177 | Train loss 0.013, accuracy 0.9957 | Test loss 0.073, accuracy 0.9829
    epoch 178 | Train loss 0.014, accuracy 0.9955 | Test loss 0.072, accuracy 0.9833
    epoch 179 | Train loss 0.013, accuracy 0.9959 | Test loss 0.073, accuracy 0.9834
    epoch 180 | Train loss 0.013, accuracy 0.9955 | Test loss 0.073, accuracy 0.9831
    epoch 181 | Train loss 0.014, accuracy 0.9956 | Test loss 0.072, accuracy 0.9836
    epoch 182 | Train loss 0.014, accuracy 0.9956 | Test loss 0.073, accuracy 0.9835
    epoch 183 | Train loss 0.012, accuracy 0.9962 | Test loss 0.074, accuracy 0.9831
    epoch 184 | Train loss 0.012, accuracy 0.9958 | Test loss 0.073, accuracy 0.9834
    epoch 185 | Train loss 0.012, accuracy 0.9964 | Test loss 0.073, accuracy 0.9836
    epoch 186 | Train loss 0.011, accuracy 0.9966 | Test loss 0.073, accuracy 0.9836
    epoch 187 | Train loss 0.012, accuracy 0.9961 | Test loss 0.073, accuracy 0.9836
    epoch 188 | Train loss 0.012, accuracy 0.9961 | Test loss 0.073, accuracy 0.9834
    epoch 189 | Train loss 0.012, accuracy 0.9962 | Test loss 0.074, accuracy 0.9834
    epoch 190 | Train loss 0.012, accuracy 0.9961 | Test loss 0.072, accuracy 0.9832
    epoch 191 | Train loss 0.012, accuracy 0.9962 | Test loss 0.073, accuracy 0.9830
    epoch 192 | Train loss 0.011, accuracy 0.9964 | Test loss 0.073, accuracy 0.9835
    epoch 193 | Train loss 0.011, accuracy 0.9966 | Test loss 0.074, accuracy 0.9836
    epoch 194 | Train loss 0.011, accuracy 0.9964 | Test loss 0.073, accuracy 0.9831
    epoch 195 | Train loss 0.011, accuracy 0.9967 | Test loss 0.073, accuracy 0.9834
    epoch 196 | Train loss 0.011, accuracy 0.9962 | Test loss 0.072, accuracy 0.9837
    epoch 197 | Train loss 0.011, accuracy 0.9964 | Test loss 0.073, accuracy 0.9833
    epoch 198 | Train loss 0.011, accuracy 0.9964 | Test loss 0.073, accuracy 0.9834
    epoch 199 | Train loss 0.011, accuracy 0.9965 | Test loss 0.073, accuracy 0.9832
    epoch 200 | Train loss 0.010, accuracy 0.9965 | Test loss 0.074, accuracy 0.9834
    epoch 201 | Train loss 0.011, accuracy 0.9965 | Test loss 0.074, accuracy 0.9831
    epoch 202 | Train loss 0.010, accuracy 0.9968 | Test loss 0.075, accuracy 0.9835
    epoch 203 | Train loss 0.010, accuracy 0.9972 | Test loss 0.074, accuracy 0.9838
    epoch 204 | Train loss 0.010, accuracy 0.9965 | Test loss 0.077, accuracy 0.9834
    epoch 205 | Train loss 0.011, accuracy 0.9969 | Test loss 0.075, accuracy 0.9832
    epoch 206 | Train loss 0.010, accuracy 0.9966 | Test loss 0.074, accuracy 0.9834
    epoch 207 | Train loss 0.009, accuracy 0.9969 | Test loss 0.075, accuracy 0.9837
    epoch 208 | Train loss 0.010, accuracy 0.9972 | Test loss 0.075, accuracy 0.9834
    epoch 209 | Train loss 0.010, accuracy 0.9969 | Test loss 0.075, accuracy 0.9832
    epoch 210 | Train loss 0.009, accuracy 0.9974 | Test loss 0.075, accuracy 0.9835
    epoch 211 | Train loss 0.010, accuracy 0.9968 | Test loss 0.075, accuracy 0.9835
    epoch 212 | Train loss 0.009, accuracy 0.9975 | Test loss 0.077, accuracy 0.9834
    epoch 213 | Train loss 0.010, accuracy 0.9972 | Test loss 0.076, accuracy 0.9834
    epoch 214 | Train loss 0.008, accuracy 0.9975 | Test loss 0.077, accuracy 0.9832
    epoch 215 | Train loss 0.009, accuracy 0.9972 | Test loss 0.076, accuracy 0.9840
    epoch 216 | Train loss 0.010, accuracy 0.9970 | Test loss 0.075, accuracy 0.9833
    epoch 217 | Train loss 0.009, accuracy 0.9973 | Test loss 0.077, accuracy 0.9834
    epoch 218 | Train loss 0.009, accuracy 0.9973 | Test loss 0.076, accuracy 0.9829
    epoch 219 | Train loss 0.009, accuracy 0.9972 | Test loss 0.076, accuracy 0.9834
    epoch 220 | Train loss 0.009, accuracy 0.9972 | Test loss 0.075, accuracy 0.9835
    epoch 221 | Train loss 0.009, accuracy 0.9973 | Test loss 0.075, accuracy 0.9836
    epoch 222 | Train loss 0.008, accuracy 0.9975 | Test loss 0.076, accuracy 0.9829
    epoch 223 | Train loss 0.008, accuracy 0.9975 | Test loss 0.076, accuracy 0.9831
    epoch 224 | Train loss 0.009, accuracy 0.9974 | Test loss 0.077, accuracy 0.9831
    epoch 225 | Train loss 0.008, accuracy 0.9974 | Test loss 0.076, accuracy 0.9838
    epoch 226 | Train loss 0.008, accuracy 0.9976 | Test loss 0.077, accuracy 0.9834
    epoch 227 | Train loss 0.008, accuracy 0.9975 | Test loss 0.076, accuracy 0.9836
    epoch 228 | Train loss 0.009, accuracy 0.9973 | Test loss 0.075, accuracy 0.9837
    epoch 229 | Train loss 0.008, accuracy 0.9974 | Test loss 0.075, accuracy 0.9840
    epoch 230 | Train loss 0.008, accuracy 0.9974 | Test loss 0.077, accuracy 0.9837
    epoch 231 | Train loss 0.008, accuracy 0.9975 | Test loss 0.077, accuracy 0.9840
    epoch 232 | Train loss 0.008, accuracy 0.9974 | Test loss 0.077, accuracy 0.9841
    epoch 233 | Train loss 0.007, accuracy 0.9976 | Test loss 0.077, accuracy 0.9837
    epoch 234 | Train loss 0.008, accuracy 0.9972 | Test loss 0.077, accuracy 0.9834
    epoch 235 | Train loss 0.008, accuracy 0.9974 | Test loss 0.077, accuracy 0.9838
    epoch 236 | Train loss 0.008, accuracy 0.9976 | Test loss 0.075, accuracy 0.9838
    epoch 237 | Train loss 0.007, accuracy 0.9978 | Test loss 0.077, accuracy 0.9835
    epoch 238 | Train loss 0.007, accuracy 0.9976 | Test loss 0.076, accuracy 0.9838
    epoch 239 | Train loss 0.008, accuracy 0.9973 | Test loss 0.076, accuracy 0.9839
    epoch 240 | Train loss 0.008, accuracy 0.9975 | Test loss 0.078, accuracy 0.9841
    epoch 241 | Train loss 0.008, accuracy 0.9978 | Test loss 0.077, accuracy 0.9836
    epoch 242 | Train loss 0.008, accuracy 0.9976 | Test loss 0.078, accuracy 0.9839
    epoch 243 | Train loss 0.007, accuracy 0.9976 | Test loss 0.079, accuracy 0.9838
    epoch 244 | Train loss 0.007, accuracy 0.9978 | Test loss 0.077, accuracy 0.9841
    epoch 245 | Train loss 0.007, accuracy 0.9978 | Test loss 0.077, accuracy 0.9839
    epoch 246 | Train loss 0.008, accuracy 0.9975 | Test loss 0.077, accuracy 0.9835
    epoch 247 | Train loss 0.007, accuracy 0.9979 | Test loss 0.077, accuracy 0.9839
    epoch 248 | Train loss 0.007, accuracy 0.9976 | Test loss 0.079, accuracy 0.9835
    epoch 249 | Train loss 0.007, accuracy 0.9981 | Test loss 0.080, accuracy 0.9839
    epoch 250 | Train loss 0.007, accuracy 0.9979 | Test loss 0.079, accuracy 0.9836
    epoch 251 | Train loss 0.007, accuracy 0.9977 | Test loss 0.079, accuracy 0.9834
    epoch 252 | Train loss 0.007, accuracy 0.9979 | Test loss 0.079, accuracy 0.9834
    epoch 253 | Train loss 0.007, accuracy 0.9979 | Test loss 0.080, accuracy 0.9837
    epoch 254 | Train loss 0.007, accuracy 0.9978 | Test loss 0.080, accuracy 0.9835
    epoch 255 | Train loss 0.007, accuracy 0.9979 | Test loss 0.079, accuracy 0.9845
    epoch 256 | Train loss 0.006, accuracy 0.9980 | Test loss 0.078, accuracy 0.9841
    epoch 257 | Train loss 0.007, accuracy 0.9979 | Test loss 0.077, accuracy 0.9842
    epoch 258 | Train loss 0.007, accuracy 0.9979 | Test loss 0.077, accuracy 0.9839
    epoch 259 | Train loss 0.006, accuracy 0.9980 | Test loss 0.078, accuracy 0.9843
    epoch 260 | Train loss 0.006, accuracy 0.9981 | Test loss 0.079, accuracy 0.9836
    epoch 261 | Train loss 0.007, accuracy 0.9980 | Test loss 0.079, accuracy 0.9839
    epoch 262 | Train loss 0.006, accuracy 0.9977 | Test loss 0.078, accuracy 0.9835
    epoch 263 | Train loss 0.006, accuracy 0.9981 | Test loss 0.079, accuracy 0.9836
    epoch 264 | Train loss 0.006, accuracy 0.9981 | Test loss 0.079, accuracy 0.9838
    epoch 265 | Train loss 0.006, accuracy 0.9981 | Test loss 0.078, accuracy 0.9841
    epoch 266 | Train loss 0.005, accuracy 0.9985 | Test loss 0.080, accuracy 0.9839
    epoch 267 | Train loss 0.006, accuracy 0.9982 | Test loss 0.079, accuracy 0.9840
    epoch 268 | Train loss 0.006, accuracy 0.9982 | Test loss 0.080, accuracy 0.9837
    epoch 269 | Train loss 0.006, accuracy 0.9982 | Test loss 0.081, accuracy 0.9834
    epoch 270 | Train loss 0.006, accuracy 0.9982 | Test loss 0.081, accuracy 0.9844
    epoch 271 | Train loss 0.005, accuracy 0.9985 | Test loss 0.080, accuracy 0.9841
    epoch 272 | Train loss 0.006, accuracy 0.9983 | Test loss 0.081, accuracy 0.9844
    epoch 273 | Train loss 0.006, accuracy 0.9980 | Test loss 0.079, accuracy 0.9839
    epoch 274 | Train loss 0.007, accuracy 0.9977 | Test loss 0.080, accuracy 0.9842
    epoch 275 | Train loss 0.006, accuracy 0.9980 | Test loss 0.080, accuracy 0.9841
    epoch 276 | Train loss 0.006, accuracy 0.9981 | Test loss 0.080, accuracy 0.9839
    epoch 277 | Train loss 0.007, accuracy 0.9980 | Test loss 0.080, accuracy 0.9841
    epoch 278 | Train loss 0.006, accuracy 0.9982 | Test loss 0.080, accuracy 0.9845
    epoch 279 | Train loss 0.005, accuracy 0.9983 | Test loss 0.079, accuracy 0.9846
    epoch 280 | Train loss 0.006, accuracy 0.9982 | Test loss 0.080, accuracy 0.9841
    epoch 281 | Train loss 0.006, accuracy 0.9980 | Test loss 0.081, accuracy 0.9844
    epoch 282 | Train loss 0.006, accuracy 0.9982 | Test loss 0.080, accuracy 0.9839
    epoch 283 | Train loss 0.006, accuracy 0.9983 | Test loss 0.080, accuracy 0.9840
    epoch 284 | Train loss 0.004, accuracy 0.9988 | Test loss 0.079, accuracy 0.9846
    epoch 285 | Train loss 0.006, accuracy 0.9983 | Test loss 0.080, accuracy 0.9845
    epoch 286 | Train loss 0.006, accuracy 0.9982 | Test loss 0.081, accuracy 0.9835
    epoch 287 | Train loss 0.005, accuracy 0.9983 | Test loss 0.079, accuracy 0.9838
    epoch 288 | Train loss 0.005, accuracy 0.9981 | Test loss 0.082, accuracy 0.9839
    epoch 289 | Train loss 0.005, accuracy 0.9983 | Test loss 0.081, accuracy 0.9841
    epoch 290 | Train loss 0.006, accuracy 0.9986 | Test loss 0.080, accuracy 0.9839
    epoch 291 | Train loss 0.006, accuracy 0.9984 | Test loss 0.082, accuracy 0.9836
    epoch 292 | Train loss 0.005, accuracy 0.9985 | Test loss 0.082, accuracy 0.9835
    epoch 293 | Train loss 0.005, accuracy 0.9982 | Test loss 0.081, accuracy 0.9839
    epoch 294 | Train loss 0.005, accuracy 0.9984 | Test loss 0.081, accuracy 0.9841
    epoch 295 | Train loss 0.005, accuracy 0.9986 | Test loss 0.080, accuracy 0.9842
    epoch 296 | Train loss 0.005, accuracy 0.9984 | Test loss 0.080, accuracy 0.9844
    epoch 297 | Train loss 0.005, accuracy 0.9988 | Test loss 0.080, accuracy 0.9844
    epoch 298 | Train loss 0.005, accuracy 0.9984 | Test loss 0.081, accuracy 0.9842
    epoch 299 | Train loss 0.006, accuracy 0.9982 | Test loss 0.083, accuracy 0.9845

初期設定では，98%前後のaccuracyになったのではないかと思います．

ニューラルネットワークの学習には，様々なハイパーパラメータ（学習率など）を上手く設定する必要があります．

また，活性化関数や，重みの初期値の工夫，Dropoutなどのテクニックを用いることで，さらに性能が向上する可能性があります．

モデルを工夫して，よりよい性能を発揮するニューラルネットワークを構築してみましょう．

実行結果メモ
(1)
```
model = MLP(
[Linear(784, 1000, Sigmoid),
Linear(1000, 1000, Sigmoid),
Linear(1000, 10, Softmax)])
n_epoch = 20*4
batchsize = 100
lr = 1,0.5,0.1,0.05,0.01
```
test accuracy: 0.9811

(2)
```
model = MLP(
[Linear(784, 1000, ReLU),
Linear(1000, 1000, ReLU),
Linear(1000, 10, Softmax)])
n_epoch = 20
batchsize = 100
lr = 1,0.5,0.1,0.05,0.01
```
test accuracy: 0.9845

DropoutとMish，せっかく実装したのにあんまり効果がなくて悲しい．
もう少し層の深さが必要？

(3)
```
model = MLP([
Linear(784, 1000, ReLU),
Linear(1000, 1000, ReLU),
Linear(1000, 1000, ReLU),
Dropout(1000, 1000, 0.5),
Linear(1000, 10, Softmax)])
n_epoch = 20
batchsize = 100
lr = 1,0.5,0.1,0.05,0.01
epoch = 10,20,20,20,20
初期値 = 一様分布
```
test accuracy: 0.9859

過学習してますねぇ…

```
model = MLP([
Linear(784, 1000, ReLU),
Dropout(1000, 1000, 0.2),
Linear(1000, 1000, ReLU),
Dropout(1000, 1000, 0.5),
Linear(1000, 1000, ReLU),
Dropout(1000, 1000, 0.5),
Linear(1000, 1000, ReLU),
Dropout(1000, 1000, 0.5),
Linear(1000, 1000, ReLU),
Dropout(1000, 1000, 0.5),
Linear(1000, 10, Softmax)])
batchsize = 100
lr, n_epoch in [ [0.1, 20], [0.05, 20], [0.01, 20], [0.005, 20]]
```
test accuracy: 0.9861

```python

```


# 数字認識タスク
Notebook の利用が初めての場合 - ようこそ！Notebook はコードの近くに結果が表示されるのを見ながら実行できる素晴らしいツールです。
1. セルを実行するには、セルをクリックして SHIFT-Enter を押します。
2. 何かが実行されると、変数はメモリに保存されます - それらを調べてください。

# 訳注 Jupyter Notebook の使い方:
このワークショップでは、Azure Notebooks Service によってホストされている Jupyter ノートブックを使用します。あなたが Jupyter に精通しているならば、このセルを飛ばしてください。そうでない場合は、ここでいくつかのヒントがあります。

* Python 3 のカーネルを選択してください。
* Jupyter には「セル」があります。ノートブックのどこかをクリックすると、ポインタを含むセルが選択されます。
* セル内のコードを実行または実行するには、トップメニューで [セル]-[セルの実行] を選択します。または、Shift + Enterキーを使用することもできます。
* セルを編集するには - セルをダブルクリックします。
* セルを実行すると、セルの左側の外側にあるポインタが [ ] から [ \* ] に変わります。セルの実行が終了すると、[ \* ] が数字に変わります。
* ノートブックまたはセルがハング(フリーズ)した場合は、トップメニューからノートブック全体の [Kernel]-[Restart Kernel and Clear All Outputs] を選択するか、そのセルだけを選択して [Kernel]-[Interrupt kernel] を選択します。再起動には数秒かかり、その後ですべてのセルを再実行する必要があります。

Jupyterでは、順番に実行する必要はありません。すでに実行された前のセルに戻って再度実行することができます。特定のセルのコードが、前後のセルと依存関係を持たない限り、問題ありません。

**訳注** このノートブック (start.ipynb)は **Python 3 カーネル** で実行してください。デフォルトでは Python 3 になっているはずです（ノートブックの右上の表示で確認できます）。
Python 3.6 で実行するとライブラリのバージョン違いに起因するエラーが発生するかもしれません。
Python 3 以外になっている場合は、ノートブックのメニューで [Kernel] -[Change Kernel] - [Python 3] を選択してください。

## 始めましょう
最初のセルは必要なライブラリをインポートします。ここから始めましょう。

```python
import torch
import numpy as np
from PIL import Image
import torch.nn as nn
import torch.onnx as onnx
import torch.nn.functional as F
import matplotlib.pyplot as plt
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
```

## データ!
データがなければ、機械学習では何もできません。今回は「**28x28のベクトルで数字を予測できるか？**」という課題を与えられています。

用意できたら、データを見てみる必要があります。次のセルは数字を視覚化するヘルパー関数です（健全性チェック）。

```python
def draw_digits(digits):
    fig, axes = plt.subplots(6, 20, figsize=(18, 7),
                            subplot_kw={'xticks':[], 'yticks':[]},
                            gridspec_kw=dict(hspace=0.1, wspace=0.1))
    for i, ax in enumerate(axes.flat):
        X, y = digits[i]
        ax.imshow(255 - X.reshape(28,28) * 255, cmap='gray')
        ax.set_title('{:.0f}'.format(torch.argmax(y).item()))
```

次のセルは、標準的な数値データセット（MNISTと呼ばれる）をダウンロードします。この呼び出しの transform 部分と target_transform 部分は、これから行うモデルにより適したデータにするための変換手順がいくつか入っています。

```python
digits = datasets.MNIST('data', train=True, download=True,
                        transform=transforms.Compose([
                            transforms.ToTensor(),
                            transforms.Lambda(lambda x: x.reshape(28*28))
                        ]),
                        target_transform=transforms.Compose([
                            transforms.Lambda(lambda y: torch.zeros(10, dtype=torch.float).scatter_(0, y, 1))
                        ])
                     )
```

ここでデータの健全性をチェックします。

```python
draw_digits(digits)
```

xやyを表示して、ざっとデータを見てください。これらは本当に単なる数字です！それらを見てみるといくつか奇妙に思えるかもしれません：

1.「画像は 0～255 の 784個のベクトルではないのですか？」と思いましたか。データは見た通りです！新しい範囲が 0～1 になるように255で割ってベクトルを正規化しただけです（数値がより正確になるようにしました）
2.「y が数値の答えだったのではないですか？代わりに 10次元のベクトルになっています」と思いましたか。その通りです！これは答えの One-Hot 表現と呼ばれます。正解のインデクスの位置に "1" が入っています。繰り返しになりますが、これにより、これから作成するモデルで数値演算がよりよく機能するようになります。

```python
x, y = digits[0]
print(x)
print(y)
```

# モデルを選択する
いくつかのデータが得られたので、今度はうまくいくと思われるモデルの選択を開始します。これがデータサイエンスの科学的な部分が出てくるところです：私たちは推測し、それから仮定が正しいかどうかチェックします。 784個の蛇口から10本の異なるホースに水を分配する必要がある水道管のようなモデルを想像してみてください。これらの784個の蛇口は数字の個々のピクセルを表し、最後の10個のホースは実際の数（または少なくとも水が最も多く出ているもののインデックス）を表します。私たちの仕事は今その間の配管を選ぶことです。

次の3つのセルは、順に難しくなる 3つの別の構造を表しています。

1. 最初のモデルは単純な線形モデルです
2. 2番目のモデルは3層ニューラルネットワークです
3. 最後のモデルは全てが畳み込み層で構成されるニューラルネットワークです

それらがどのように働くかを完全に説明することはこのチュートリアルの範囲外ですが、正しいインデックスから最も多くの水を押し出せるように、最後に正しい水圧を生み出すように調整されなければならない内部ノブで配管していると想像してください 。下のセルほど、配管とそれに対応する内部ノブがより複雑になります。

```python
class SimpleLinear(nn.Module):
    def __init__(self):
        super(SimpleLinear, self).__init__()
        self.layer1 = nn.Linear(28*28, 10)

    def forward(self, x):
        x = self.layer1(x)
        return F.softmax(x, dim=1)
```

```python
class NeuralNework(nn.Module):
    def __init__(self):
        super(NeuralNework, self).__init__()
        self.layer1 = nn.Linear(28*28, 512)
        self.layer2 = nn.Linear(512, 512)
        self.output = nn.Linear(512, 10)

    def forward(self, x):
        x = F.relu(self.layer1(x))
        x = F.relu(self.layer2(x))
        x = self.output(x)
        return F.softmax(x, dim=1)
```

```python
class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = x.view(-1, 1, 28, 28)
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.softmax(x, dim=1)
```

# モデルパラメータの最適化
いくつかのモデルができたので、数字を認識するのに良い仕事ができるかどうか確認するために内部パラメータを最適化する時が来ました！訓練する方法を調整するために最適化アルゴリズムを与える際に使用する、いくつかのパラメータです - これらはハイパーパラメータと呼ばれます。次に示す2つの変数です。

```python
learning_rate = 1e-3
batch_size = 64
epochs = 5
```

'learning_rate' は基本的に、アルゴリズムがモデルパラメータを学習する速度を指定します。あなたが「それを5000万に設定するのはどうかな？」と考えたとします。それが悪い考えである理由の最も良い例えはゴルフです。私はゴルフがうまくないので、本当によくわかりませんが、ショットを沈めようとしているのに毎回同じ距離ボールを打つようなものです。簡単ですね？今いる場所から穴までの正確な距離を打てばいいのです！それで完了です！さて、あなたは穴がどこにあるのかわからないが、おおざっぱな方向は知っているとします。あなたが選ぶ距離は実際に重要です。距離が長すぎると穴を逃すことになり、それを叩くと再び行きすぎます。距離が短すぎると、穴にたどり着くまでに時間がかかりますが、最終的には穴に到達するはずです。基本的には、1ショットあたりの正しい距離を推測してから試してみる必要があります。それが、穴に入れるための正しいパラメーターを見つけられるように、基本的に学習率がすることです。（さて、ゴルフの話は終わりです）

以下は、完全に動作するのに必要な 3つのものです:
1. **モデル** - 私たちが作ろうとしている関数であり、正解の数値を返すべきものです
2. **コスト関数** (損失関数と呼ばれることもあります) - ゴルフの話は終わったと約束したことを覚えていますが、嘘をつきました。おかしなゴルフゲームであなたが穴の大体の方向を知っていると言ったのを覚えていますか？ コスト関数は穴までの距離を教えてくれます。ゼロになると穴に入ったことになるのです！実際の科学的な用語では、コスト関数は、モデルが正しい答えを得るのにどれほど悪いかを示すものです。ゴルフショットと同じように、コスト関数が減少するのも見るべきです。減少しなければ、何か間違っています。そうならば、ショットの距離（または学習率）をもっと小さい値に変更して、もう一度試してみます。それでもうまくいかない場合は、モデルを変更してください。
3. **オプティマイザー** - この部分は、実際にモデルパラメーターを変更する部分です。それは我々がボールを打つべき方向の意味を持ち、正しい数字を予測する最良の内部の蛇口を見つけるために、モデル内部のすべての値を更新します。ここでは、バイナリークロスエントロピーコスト関数を使用します。これがうまく機能することがわかっているからです。さまざまなシナリオに合わせて、さまざまなコスト関数を選択できます。

```python
# 実行する場所
device = 'cuda' if torch.cuda.is_available() else 'cpu'
print('Using {} device'.format(device))

# モデルの選択
model = SimpleLinear().to(device)
#model = NeuralNework().to(device)
#model = CNN().to(device)
print(model)

# 最良のパラメーターを決定するのに使用するコスト関数
cost = torch.nn.BCELoss()

# 最適なパラメーターを作成するために使用
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
```

'CUDA' について言及するのをすっかり忘れていました。基本的に for ループの中はすべての大きな行列の乗算問題であるため、GPUはこのプロセス全体をスピードアップします。PyTorchは基本的にどこで実行するか（CPU または CUDA - CUDA は計算を GPU に移行するためのプラットフォームです）をモデルに伝えるだけでいいのです。素晴らしい。

それでは、学習に進みます！データローダーの仕事は、データセット全体（ここでは 60,000個の例とそれに対応するラベル）を反復処理することですが、処理するデータを batch_sizeの サイズごとに取得します。batch_size は選択する必要があるもう1つのハイパーパラメータです。'epoch' は、データセット全体をループする回数です（これもまた、実験の進行状況に基づいて選択したものです）。

次のセルの実行には時間がかかります。

```python
# loader
dataloader = DataLoader(digits, batch_size=batch_size, num_workers=0, pin_memory=True)

# golfing time!
for t in range(epochs):
    print('Epoch {}'.format(t))
    print('-------------------------------')
    # go through the entire dataset once
    for batch, (X, Y) in enumerate(dataloader):
        X, Y = X.to(device), Y.to(device)
        # zero out gradient
        optimizer.zero_grad()
        # make a prediction on this batch!
        pred = model(X)
        # how bad is it?
        loss = cost(pred, Y)
        # compute gradients
        loss.backward()
        # update parameters
        optimizer.step()

        if batch % 100 == 0:
            print('loss: {:>10f}  [{:>5d}/{:>5d}]'.format(loss.item(), batch * len(X), len(dataloader.dataset)))

    print('loss: {:>10f}  [{:>5d}/{:>5d}]\n'.format(loss.item(), len(dataloader.dataset), len(dataloader.dataset)))

print('Done!')
```

さて、ここでは実際には何をしたのでしょうか。素晴らしい質問です。最初の実行した SimpleLinear モデルから説明します。基本的に1つの行列と1つのベクトルを学習しています。それだけです。実際にはこういうことです:

\begin{align}
prediction = W \cdot x + b
\end{align}

\begin{align}
\begin{bmatrix}
\hat{y}_1 \\ \vdots \\ \hat{y}_{10}
\end{bmatrix}_{10 \times 1} =
\begin{bmatrix}
w_{1,1} &  \ldots &  w_{1,784} \\
\vdots    & \ddots &  \\
w_{10,1}  & \ldots & w_{10, 784} \\
\end{bmatrix}_{10 \times 784}
\cdot
\begin{bmatrix}x_1 \\ x_2 \\ \vdots \\ x_{783} \\ x_{784}\end{bmatrix}_{784 \times 1}
 +
\begin{bmatrix}
b_1 \\ \vdots \\ b_{10}
\end{bmatrix}_{10 \times 1}
\end{align}

行列 W とベクトル b は、SimpleLinear モデルが学習するものです。出力は 10×1 行列で、最大値を持っているものが予測したい数値のインデックスです。アルゴリズムが 2つの変数（およびサイズ）について学習した内容を見てみましょう:

```python
for p in model.parameters():
    print(p.shape)
    print(p)
```

# 動作しているか？
正常に機能しているかどうかを判断する最良の方法は、学習プロセスで使用されていないデータでモデルをテストすることです。幸いなことに、そのようなデータセットがあります（これは既に使用したデータとは分けておいたものです）。以前と同じ方法でそれをすべてロードし、前のデータとは異なることを示すために表示してみます。

```python
test_digits = datasets.MNIST('data', train=False, download=True,
                        transform=transforms.Compose([
                            transforms.ToTensor(),
                            transforms.Lambda(lambda x: x.reshape(28*28))
                        ]),
                        target_transform=transforms.Compose([
                            transforms.Lambda(lambda y: torch.zeros(10, dtype=torch.float).scatter_(0, y, 1))
                        ])
                     )
draw_digits(test_digits)
```

最初の数値でテストしましょう（7 です）。

```python
x, y = test_digits[0]
x = x.to(device).view(1, 28*28)
```

評価に使用するモデル（推論とも呼ばれます）を指定して、これまでに使ったことのない数値を渡します。

```python
model.eval()
with torch.no_grad():
    pred = model(x)
    pred = pred.to('cpu').detach()[0]

print(pred)
```

予測された数字が実際の数字と一致するかどうか見てみましょう:

```python
pred.argmax(0), y.argmax(0)
```

**すべて** のテストデータに対してどれだけうまくいっているか見てみましょう！

```python
# 推論用の数値のデータローダー
test_dataloader = DataLoader(test_digits, batch_size=64, num_workers=0, pin_memory=True)

# モデルを評価モードに設定
model.eval()
test_loss = 0
correct = 0

# ループ
with torch.no_grad():
    for batch, (X, Y) in enumerate(dataloader):
        X, Y = X.to(device), Y.to(device)
        pred = model(X)

        test_loss += cost(pred, Y).item()
        correct += (pred.argmax(1) == Y.argmax(1)).type(torch.float).sum().item()

test_loss /= len(dataloader.dataset)
correct /= len(dataloader.dataset)
print('\nTest Error:')
print('acc: {:>0.1f}%, avg loss: {:>8f}'.format(100*correct, test_loss))
```

# モデルの保存
今回は PyTorch を使っていますが、すべてのフレームワークは異なります。モデルを (PyTorch の）内部フォーマットと ONNXフォーマットの両方で保存します（これらは大きな行列Wとベクトルbであることを覚えています）。モデルは数字を認識する必要があるたびに、実行されるプログラムにアセットとしてロードすることができます。

```python
# グラフ全体を通すためのダミーの変数を作成
x = torch.randint(255, (1, 28*28), dtype=torch.float).to(device) / 255
onnx.export(model, x, 'model.onnx')
print('Saved onnx model to model.onnx')

# PyTorch モデルを保存
torch.save(model.state_dict(), 'model.pth')
print('Saved PyTorch Model to model.pth')
```

# 試してみましょう！
これでプロセス全体が完了したので、上に戻って試してみてください。次のように変更してみましょう:
1. 他のモデル（SimpleLinear を NeuralNetwork, CNN）を選択。他のモデルは、画像が最終的な答えを得るために通過する追加の行列（Wとb）を学習することを除けば、呼び出し方はほとんど同じです。
2. learning_rate、batch_size、epochなどのハイパーパラメーター。 変更で結果が良くなったり悪くなったりしますか？ 92％は問題ないですが、他のモデルとハイパーパラメータの組み合わせのほうがいいでしょうか。

## 最後に
フィードバックを歓迎します！これは役に立ちましたか？ 分かりにくい部分はありますか？簡単にでもいいので教えてください。


## Histograms of Oriented Gradients (HOG)

As we saw with the ORB algorithm, we can use keypoints in images to do keypoint-based matching to detect objects in images. These type of algorithms work great when you want to detect objects that have a lot of consistent internal features that are not affected by the background. For example, these algorithms work well for facial detection because faces have a lot of consistent internal features that don’t get affected by the image background, such as the eyes, nose, and mouth. However, these type of algorithms don’t work so well when attempting to do more general object recognition, say for example, pedestrian detection in images. The reason is that people don’t have consistent internal features, like faces do, because the body shape and style of every person is different (see Fig. 1). This means that every person is going to have a different set of internal features, and so we need something that can more generally describe a person.

Fig. 1. - Pedestrians.

One option is to try to detect pedestrians by their contours instead.  Detecting objects in images by their contours (boundaries) is very challenging because we have to deal with the difficulties brought about by the contrast between the background and the foreground. For example, suppose you wanted to detect a pedestrian in an image that is walking in front of a white building and she is wearing a white coat and black pants (see Fig. 2). We can see in Fig. 2, that since the background of the image is mostly white, the black pants are going to have a very high contrast, but the coat, since it is white as well, is going to have very low contrast. In this case, detecting the edges of pants is going to be easy but detecting the edges of the coat is going to be very difficult. This is where **HOG** comes in. HOG stands for **Histograms of Oriented Gradients** and it was first introduced by Navneet Dalal and Bill Triggs in 2005.

Fig. 2. - High and Low Contrast.

The HOG algorithm works by creating histograms of the distribution of gradient orientations in an image and then normalizing them in a very special way. This special normalization is what makes HOG so effective at detecting the edges of objects even in cases where the contrast is very low. These normalized histograms are put together into a feature vector, known as the HOG descriptor, that can be used to train a machine learning algorithm, such as a Support Vector Machine (SVM), to detect objects in images based on their boundaries (edges). Due to its great success and reliability, HOG has become one of the most widely used algorithms in computer vison for object detection.

In this notebook, you will learn:

* How the HOG algorithm works
* How to use OpenCV to create a HOG descriptor
* How to visualize the HOG descriptor.

# The HOG Algorithm

As its name suggests, the HOG algorithm, is based on creating histograms from the orientation of image gradients. The HOG algorithm is implemented in a series of steps:

1. Given the image of particular object, set a detection window (region of interest) that covers the entire object in the image (see Fig. 3).

2. Calculate the magnitude and direction of the gradient for each individual pixel in the detection window.

3. Divide the detection window into connected *cells* of pixels, with all cells being of the same size (see Fig. 3). The size of the cells is a free parameter and it is usually chosen so as to match the scale of the features that want to be detected. For example, in a 64 x 128 pixel detection window, square cells 6 to 8 pixels wide are suitable for detecting human limbs.

4. Create a Histogram for each cell, by first grouping the gradient directions of all pixels in each cell into a particular number of orientation (angular) bins; and then adding up the gradient magnitudes of the gradients in each angular bin (see Fig. 3). The number of bins in the histogram is a free parameter and it is usually set to 9 angular bins.

5. Group adjacent cells into *blocks* (see Fig. 3). The number of cells in each block is a free parameter and all blocks must be of the same size. The distance between each block (known as the stride) is a free parameter but it is usually set to half the block size, in which case you will get overlapping blocks (*see video below*).  The HOG algorithm has been shown empirically to work better with overlapping blocks.

6. Use the cells contained within each block to normalize the cell histograms in that block (see Fig. 3). If you have overlapping blocks this means that most cells will be normalized with respect to different blocks (*see video below*). Therefore, the same cell may have several different normalizations.

7. Collect all the normalized histograms from all the blocks into a single feature vector called the HOG descriptor.

8. Use the resulting HOG descriptors from many images of the same type of object to train a machine learning algorithm, such as an SVM, to detect those type of objects in images. For example, you could use the HOG descriptors from many images of pedestrians to train an SVM to detect pedestrians in images. The training is done with both positive a negative examples of the object you want detect in the image.

9. Once the SVM has been trained, a sliding window approach is used to try to detect and locate objects in images.  Detecting an object in the image entails finding the part of the image that looks similar to the HOG pattern learned by the SVM.

Fig. 3. - HOG Diagram.

Vid. 1. - HOG Animation.

# Why The HOG Algorithm Works

As we learned above, HOG creates histograms by adding the magnitude of the gradients in particular orientations in localized portions of the image called *cells*.  By doing this we guarantee that stronger gradients will contribute more to the magnitude of their respective angular bin, while the effects of weak and randomly oriented gradients resulting from noise are minimized. In this manner the histograms tell us the dominant gradient orientation of each cell.

### Dealing with contrast

Now, the magnitude of the dominant orientation can vary widely due to variations in local illumination and the contrast between the background and the foreground.

To account for the background-foreground contrast differences, the HOG algorithm tries to detect edges locally.  In order to do this, it defines groups of cells, called **blocks**, and normalizes the histograms using this local group of cells. By normalizing locally, the HOG algorithm can detect the edges in each block very reliably; this is called **block normalization**.

In addition to using block normalization, the HOG algorithm also uses overlapping blocks to increase its performance. By using overlapping blocks, each cell contributes several independent components to the final HOG descriptor, where each component corresponds to a cell being normalized with respect to a different block. This may seem redundant but, it has been shown empirically that by normalizing each cell several times with respect to different local blocks, the performance of the HOG algorithm increases dramatically.

### Loading Images and Importing Resources

The first step in building our HOG descriptor is to load the required packages into Python and to load our image.

We start by using OpenCV to load an image of a triangle tile. Since, the `cv2.imread()` function loads images as BGR we will convert our image to RGB so we can display it with the correct colors. As usual we will convert our BGR image to Gray Scale for analysis.

```python
import cv2
import numpy as np
import matplotlib.pyplot as plt

# Set the default figure size
plt.rcParams['figure.figsize'] = [17.0, 7.0]

# Load the image
image = cv2.imread('./images/triangle_tile.jpeg')

# Convert the original image to RGB
original_image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# Convert the original image to gray scale
gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

# Print the shape of the original and gray scale images
print('The original image has shape: ', original_image.shape)
print('The gray scale image has shape: ', gray_image.shape)

# Display the images
plt.subplot(121)
plt.imshow(original_image)
plt.title('Original Image')
plt.subplot(122)
plt.imshow(gray_image, cmap='gray')
plt.title('Gray Scale Image')
plt.show()
```

# Creating The HOG Descriptor

We will be using OpenCV’s `HOGDescriptor` class to create the HOG descriptor. The parameters of the HOG descriptor are setup using the `HOGDescriptor()` function. The parameters of the `HOGDescriptor()` function and their default values are given below:

`cv2.HOGDescriptor(win_size = (64, 128),
                  block_size = (16, 16),
                  block_stride = (8, 8),
                  cell_size = (8, 8),
                  nbins = 9,
                  win_sigma = DEFAULT_WIN_SIGMA,
                  threshold_L2hys = 0.2,
                  gamma_correction = true,
                  nlevels = DEFAULT_NLEVELS)`

Parameters:

* **win_size** – *Size*
Size of detection window in pixels (*width, height*). Defines the region of interest. Must be an integer multiple of cell size.

* **block_size** – *Size*
Block size in pixels (*width, height*). Defines how many cells are in each block. Must be an integer multiple of cell size and it must be smaller than the detection window. The smaller the block the finer detail you will get.

* **block_stride** – *Size*
Block stride in pixels (*horizontal, vertical*). It must be an integer multiple of cell size. The `block_stride` defines the distance between adjecent blocks, for example, 8 pixels horizontally and 8 pixels vertically. Longer `block_strides` makes the algorithm run faster (because less blocks are evaluated) but the algorithm may not perform as well.

* **cell_size** – *Size*
Cell size in pixels (*width, height*). Determines the size fo your cell. The smaller the cell the finer detail you will get.

* **nbins** – *int*
Number of bins for the histograms. Determines the number of angular bins used to make the histograms. With more bins you capture more gradient directions. HOG uses unsigned gradients, so the angular bins will have values between 0 and 180 degrees.

* **win_sigma** – *double*
Gaussian smoothing window parameter. The performance of the HOG algorithm can be improved by smoothing the pixels near the edges of the blocks by applying a Gaussian spatial window to each pixel before computing the histograms.

* **threshold_L2hys** – *double*
L2-Hys (Lowe-style clipped L2 norm) normalization method shrinkage. The L2-Hys method is used to normalize the blocks and it consists of an L2-norm followed by clipping and a renormalization. The clipping limits the maximum value of the descriptor vector for each block to have the value of the given threshold (0.2 by default). After the clipping the descriptor vector is renormalized as described in *IJCV*, 60(2):91-110, 2004.

* **gamma_correction** – *bool*
Flag to specify whether the gamma correction preprocessing is required or not. Performing gamma correction slightly increases the performance of the HOG algorithm.

* **nlevels** – *int*
Maximum number of detection window increases.

As we can see, the  `cv2.HOGDescriptor()`function supports a wide range of parameters. The first few arguments (`block_size, block_stride, cell_size`, and `nbins`) are probably the ones you are most likely to change. The other parameters can be safely left at their default values and you will get good results.

In the code below, we will use the `cv2.HOGDescriptor()`function to set the cell size, block size, block stride, and the number of bins for the histograms of the HOG descriptor. We will then use `.compute(image)`method to compute the HOG descriptor (feature vector) for the given `image`.

```python
# Specify the parameters for our HOG descriptor

# Cell Size in pixels (width, height). Must be smaller than the size of the detection window
# and must be chosen so that the resulting Block Size is smaller than the detection window.
cell_size = (6, 6)

# Number of cells per block in each direction (x, y). Must be chosen so that the resulting
# Block Size is smaller than the detection window
num_cells_per_block = (2, 2)

# Block Size in pixels (width, height). Must be an integer multiple of Cell Size.
# The Block Size must be smaller than the detection window
block_size = (num_cells_per_block[0] * cell_size[0],
              num_cells_per_block[1] * cell_size[1])

# Calculate the number of cells that fit in our image in the x and y directions
x_cells = gray_image.shape[1] // cell_size[0]
y_cells = gray_image.shape[0] // cell_size[1]

# Horizontal distance between blocks in units of Cell Size. Must be an integer and it must
# be set such that (x_cells - num_cells_per_block[0]) / h_stride = integer.
h_stride = 1

# Vertical distance between blocks in units of Cell Size. Must be an integer and it must
# be set such that (y_cells - num_cells_per_block[1]) / v_stride = integer.
v_stride = 1

# Block Stride in pixels (horizantal, vertical). Must be an integer multiple of Cell Size
block_stride = (cell_size[0] * h_stride, cell_size[1] * v_stride)

# Number of gradient orientation bins
num_bins = 9

# Specify the size of the detection window (Region of Interest) in pixels (width, height).
# It must be an integer multiple of Cell Size and it must cover the entire image. Because
# the detection window must be an integer multiple of cell size, depending on the size of
# your cells, the resulting detection window might be slightly smaller than the image.
# This is perfectly ok.
win_size = (x_cells * cell_size[0] , y_cells * cell_size[1])

# Print the shape of the gray scale image for reference
print('\nThe gray scale image has shape: ', gray_image.shape)
print()

# Print the parameters of our HOG descriptor
print('HOG Descriptor Parameters:\n')
print('Window Size:', win_size)
print('Cell Size:', cell_size)
print('Block Size:', block_size)
print('Block Stride:', block_stride)
print('Number of Bins:', num_bins)
print()

# Set the parameters of the HOG descriptor using the variables defined above
hog = cv2.HOGDescriptor(win_size, block_size, block_stride, cell_size, num_bins)

# Compute the HOG Descriptor for the gray scale image
hog_descriptor = hog.compute(gray_image)
```

    The gray scale image has shape:  (250, 250)

    HOG Descriptor Parameters:

    Window Size: (246, 246)
    Cell Size: (6, 6)
    Block Size: (12, 12)
    Block Stride: (6, 6)
    Number of Bins: 9

# Number of Elements In The HOG Descriptor

The resulting HOG Descriptor (feature vector), contains the normalized histograms from all cells from all blocks in the detection window concatenated in one long vector. Therefore, the size of the HOG feature vector will be given by the total number of blocks in the detection window, multiplied by the number of cells per block, times the number of orientation bins:

\begin{equation}
\mbox{total_elements} = (\mbox{total_number_of_blocks})\mbox{ } \times \mbox{ } (\mbox{number_cells_per_block})\mbox{ } \times \mbox{ } (\mbox{number_of_bins})
\end{equation}

If we don’t have overlapping blocks (*i.e.* the `block_stride`equals the `block_size`), the total number of blocks can be easily calculated by dividing the size of the detection window by the block size. However, in the general case we have to take into account the fact that we have overlapping blocks. To find the total number of blocks in the general case (*i.e.* for any `block_stride` and `block_size`), we can use the formula given below:

\begin{equation}
\mbox{Total}_i = \left( \frac{\mbox{block_size}_i}{\mbox{block_stride}_i} \right)\left( \frac{\mbox{window_size}_i}{\mbox{block_size}_i} \right) - \left [\left( \frac{\mbox{block_size}_i}{\mbox{block_stride}_i} \right) - 1 \right]; \mbox{  for  } i = x,y
\end{equation}

Where Total$_x$, is the total number of blocks along the width of the detection window, and Total$_y$, is the total number of blocks along the height of the detection window. This formula for Total$_x$ and Total$_y$, takes into account the extra blocks that result from overlapping. After calculating Total$_x$ and Total$_y$, we can get the total number of blocks in the detection window by multiplying Total$_x$  $\times$ Total$_y$. The above formula can be simplified considerably because the `block_size`, `block_stride`, and `window_size`are all defined in terms of the `cell_size`. By making all the appropriate substitutions and cancelations the above formula reduces to:

\begin{equation}
\mbox{Total}_i = \left(\frac{\mbox{cells}_i - \mbox{num_cells_per_block}_i}{N_i}\right) + 1\mbox{  }; \mbox{  for  } i = x,y
\end{equation}

Where cells$_x$ is the total number of cells along the width of the detection window, and cells$_y$, is the total number of cells along the height of the detection window. And $N_x$ is the horizontal block stride in units of `cell_size` and $N_y$ is the vertical block stride in units of `cell_size`.

Let's calculate what the number of elements for the HOG feature vector should be and check that it matches the shape of the HOG Descriptor calculated above.

```python
# Calculate the total number of blocks along the width of the detection window
tot_bx = np.uint32(((x_cells - num_cells_per_block[0]) / h_stride) + 1)

# Calculate the total number of blocks along the height of the detection window
tot_by = np.uint32(((y_cells - num_cells_per_block[1]) / v_stride) + 1)

# Calculate the total number of elements in the feature vector
tot_els = (tot_bx) * (tot_by) * num_cells_per_block[0] * num_cells_per_block[1] * num_bins

# Print the total number of elements the HOG feature vector should have
print('\nThe total number of elements in the HOG Feature Vector should be: ',
      tot_bx, 'x',
      tot_by, 'x',
      num_cells_per_block[0], 'x',
      num_cells_per_block[1], 'x',
      num_bins, '=',
      tot_els)

# Print the shape of the HOG Descriptor to see that it matches the above
print('\nThe HOG Descriptor has shape:', hog_descriptor.shape)
print()
```

    The total number of elements in the HOG Feature Vector should be:  40 x 40 x 2 x 2 x 9 = 57600

    The HOG Descriptor has shape: (57600, 1)

# Visualizing The HOG Descriptor

We can visualize the HOG Descriptor by plotting the histogram associated with each cell as a collection of vectors. To do this, we will plot each bin in the histogram as a single vector whose magnitude is given by the height of the bin and its orientation is given by the angular bin that its associated with. Since any given cell might have multiple histograms associated with it, due to the overlapping blocks, we will choose to average all the histograms for each cell to produce a single histogram for each cell.

OpenCV has no easy way to visualize the HOG Descriptor, so we have to do some manipulation first in order to visualize it. We will start by reshaping the HOG Descriptor in order to make our calculations easier.  We will then compute the average histogram of each cell and finally we will convert the histogram bins into vectors. Once we have the vectors, we plot the corresponding vectors for each cell in an image.

The code below produces an interactive plot so that you can interact with the figure. The figure contains:
* the grayscale image,
* the HOG Descriptor (feature vector),
* a zoomed-in portion of the HOG Descriptor, and
* the histogram of the selected cell.

**You can click anywhere on the gray scale image or the HOG Descriptor image to select a particular cell**. Once you click on either image a *magenta* rectangle will appear showing the cell you selected. The Zoom Window will show you a zoomed in version of the HOG descriptor around the selected cell; and the histogram plot will show you the corresponding histogram for the selected cell. The interactive window also has buttons at the bottom that allow for other functionality, such as panning, and giving you the option to save the figure if desired. The home button returns the figure to its default value.

**NOTE**: If you are running this notebook in the Udacity workspace, there is around a 2 second lag in the interactive plot. This means that if you click in the image to zoom in, it will take about 2 seconds for the plot to refresh.

```python
%matplotlib notebook

import copy
import matplotlib.patches as patches

# Set the default figure size
plt.rcParams['figure.figsize'] = [9.8, 9]

# Reshape the feature vector to [blocks_y, blocks_x, num_cells_per_block_x, num_cells_per_block_y, num_bins].
# The blocks_x and blocks_y will be transposed so that the first index (blocks_y) referes to the row number
# and the second index to the column number. This will be useful later when we plot the feature vector, so
# that the feature vector indexing matches the image indexing.
hog_descriptor_reshaped = hog_descriptor.reshape(tot_bx,
                                                 tot_by,
                                                 num_cells_per_block[0],
                                                 num_cells_per_block[1],
                                                 num_bins).transpose((1, 0, 2, 3, 4))

# Print the shape of the feature vector for reference
print('The feature vector has shape:', hog_descriptor.shape)

# Print the reshaped feature vector
print('The reshaped feature vector has shape:', hog_descriptor_reshaped.shape)

# Create an array that will hold the average gradients for each cell
ave_grad = np.zeros((y_cells, x_cells, num_bins))

# Print the shape of the ave_grad array for reference
print('The average gradient array has shape: ', ave_grad.shape)

# Create an array that will count the number of histograms per cell
hist_counter = np.zeros((y_cells, x_cells, 1))

# Add up all the histograms for each cell and count the number of histograms per cell
for i in range (num_cells_per_block[0]):
    for j in range(num_cells_per_block[1]):
        ave_grad[i:tot_by + i,
                 j:tot_bx + j] += hog_descriptor_reshaped[:, :, i, j, :]

        hist_counter[i:tot_by + i,
                     j:tot_bx + j] += 1

# Calculate the average gradient for each cell
ave_grad /= hist_counter

# Calculate the total number of vectors we have in all the cells.
len_vecs = ave_grad.shape[0] * ave_grad.shape[1] * ave_grad.shape[2]

# Create an array that has num_bins equally spaced between 0 and 180 degress in radians.
deg = np.linspace(0, np.pi, num_bins, endpoint = False)

# Each cell will have a histogram with num_bins. For each cell, plot each bin as a vector (with its magnitude
# equal to the height of the bin in the histogram, and its angle corresponding to the bin in the histogram).
# To do this, create rank 1 arrays that will hold the (x,y)-coordinate of all the vectors in all the cells in the
# image. Also, create the rank 1 arrays that will hold all the (U,V)-components of all the vectors in all the
# cells in the image. Create the arrays that will hold all the vector positons and components.
U = np.zeros((len_vecs))
V = np.zeros((len_vecs))
X = np.zeros((len_vecs))
Y = np.zeros((len_vecs))

# Set the counter to zero
counter = 0

# Use the cosine and sine functions to calculate the vector components (U,V) from their maginitudes. Remember the
# cosine and sine functions take angles in radians. Calculate the vector positions and magnitudes from the
# average gradient array
for i in range(ave_grad.shape[0]):
    for j in range(ave_grad.shape[1]):
        for k in range(ave_grad.shape[2]):
            U[counter] = ave_grad[i,j,k] * np.cos(deg[k])
            V[counter] = ave_grad[i,j,k] * np.sin(deg[k])

            X[counter] = (cell_size[0] / 2) + (cell_size[0] * i)
            Y[counter] = (cell_size[1] / 2) + (cell_size[1] * j)

            counter = counter + 1

# Create the bins in degress to plot our histogram.
angle_axis = np.linspace(0, 180, num_bins, endpoint = False)
angle_axis += ((angle_axis[1] - angle_axis[0]) / 2)

# Create a figure with 4 subplots arranged in 2 x 2
fig, ((a,b),(c,d)) = plt.subplots(2,2)

# Set the title of each subplot
a.set(title = 'Gray Scale Image\n(Click to Zoom)')
b.set(title = 'HOG Descriptor\n(Click to Zoom)')
c.set(title = 'Zoom Window', xlim = (0, 18), ylim = (0, 18), autoscale_on = False)
d.set(title = 'Histogram of Gradients')

# Plot the gray scale image
a.imshow(gray_image, cmap = 'gray')
a.set_aspect(aspect = 1)

# Plot the feature vector (HOG Descriptor)
b.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 5)
b.invert_yaxis()
b.set_aspect(aspect = 1)
b.set_facecolor('black')

# Define function for interactive zoom
def onpress(event):

    #Unless the left mouse button is pressed do nothing
    if event.button != 1:
        return

    # Only accept clicks for subplots a and b
    if event.inaxes in [a, b]:

        # Get mouse click coordinates
        x, y = event.xdata, event.ydata

        # Select the cell closest to the mouse click coordinates
        cell_num_x = np.uint32(x / cell_size[0])
        cell_num_y = np.uint32(y / cell_size[1])

        # Set the edge coordinates of the rectangle patch
        edgex = x - (x % cell_size[0])
        edgey = y - (y % cell_size[1])

        # Create a rectangle patch that matches the the cell selected above
        rect = patches.Rectangle((edgex, edgey),
                                  cell_size[0], cell_size[1],
                                  linewidth = 1,
                                  edgecolor = 'magenta',
                                  facecolor='none')

        # A single patch can only be used in a single plot. Create copies
        # of the patch to use in the other subplots
        rect2 = copy.copy(rect)
        rect3 = copy.copy(rect)

        # Update all subplots
        a.clear()
        a.set(title = 'Gray Scale Image\n(Click to Zoom)')
        a.imshow(gray_image, cmap = 'gray')
        a.set_aspect(aspect = 1)
        a.add_patch(rect)

        b.clear()
        b.set(title = 'HOG Descriptor\n(Click to Zoom)')
        b.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 5)
        b.invert_yaxis()
        b.set_aspect(aspect = 1)
        b.set_facecolor('black')
        b.add_patch(rect2)

        c.clear()
        c.set(title = 'Zoom Window')
        c.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 1)
        c.set_xlim(edgex - cell_size[0], edgex + (2 * cell_size[0]))
        c.set_ylim(edgey - cell_size[1], edgey + (2 * cell_size[1]))
        c.invert_yaxis()
        c.set_aspect(aspect = 1)
        c.set_facecolor('black')
        c.add_patch(rect3)

        d.clear()
        d.set(title = 'Histogram of Gradients')
        d.grid()
        d.set_xlim(0, 180)
        d.set_xticks(angle_axis)
        d.set_xlabel('Angle')
        d.bar(angle_axis,
              ave_grad[cell_num_y, cell_num_x, :],
              180 // num_bins,
              align = 'center',
              alpha = 0.5,
              linewidth = 1.2,
              edgecolor = 'k')

        fig.canvas.draw()

# Create a connection between the figure and the mouse click
fig.canvas.mpl_connect('button_press_event', onpress)
plt.show()
```

    The feature vector has shape: (57600, 1)
    The reshaped feature vector has shape: (40, 40, 2, 2, 9)
    The average gradient array has shape:  (41, 41, 9)

    <IPython.core.display.Javascript object>

# Understanding The Histograms

Let's take a look at a couple of snapshots of the above figure to see if the histograms for the selected cell make sense. Let's start looking at a cell that is inside a triangle and not near an edge:

Fig. 4. - Histograms Inside a Triangle.

In this case, since the triangle is nearly all of the same color there shouldn't be any dominant gradient in the selected cell. As we can clearly see in the Zoom Window and the histogram, this is indeed the case. We have many gradients but none of them clearly dominates over the other.

Now let’s take a look at a cell that is near a horizontal edge:

Fig. 5. - Histograms Near a Horizontal Edge.

Remember that edges are areas of an image where the intensity changes abruptly. In these cases, we will have a high intensity gradient in some particular direction. This is exactly what we see in the corresponding histogram and Zoom Window for the selected cell.  In the Zoom Window, we can see that the dominant gradient is pointing up, almost at 90 degrees, since that’s the direction in which there is a sharp change in intensity. Therefore, we should expect to see the 90-degree bin in the histogram to dominate strongly over the others. This is in fact what we see.

Now let’s take a look at a cell that is near a vertical edge:

Fig. 6. - Histograms Near a Vertical Edge.

In this case we expect the dominant gradient in the cell to be horizontal, close to 180 degrees, since that’s the direction in which there is a sharp change in intensity. Therefore, we should expect to see the 170-degree bin in the histogram to dominate strongly over the others. This is what we see in the histogram but we also see that there is another dominant gradient in the cell, namely the one in the 10-degree bin. The reason for this, is because the HOG algorithm is using unsigned gradients, which means 0 degrees and 180 degrees are considered the same. Therefore, when the histograms are being created, angles between 160 and 180 degrees, contribute proportionally to both the 10-degree bin and the 170-degree bin. This results in there being two dominant gradients in the cell near the vertical edge instead of just one.

To conclude let’s take a look at a cell that is near a diagonal edge.

Fig. 7. - Histograms Near a Diagonal Edge.

To understand what we are seeing, let’s first remember that gradients have an *x*-component, and a *y*-component, just like vectors. Therefore, the resulting orientation of a gradient is going to be given by the vector sum of its components. For this reason, on vertical edges the gradients are horizontal, because they only have an x-component, as we saw in Figure 4. While on horizontal edges the gradients are vertical, because they only have a y-component, as we saw in Figure 3. Consequently, on diagonal edges, the gradients are also going to be diagonal because both the *x* and *y* components are non-zero.  Since the diagonal edges in the image are close to 45 degrees, we should expect to see a dominant gradient orientation in the 50-degree bin. This is in fact what we see in the histogram but, just like in Figure 4., we see there are two dominant gradients instead of just one. The reason for this is that when the histograms are being created, angles that are near the boundaries of bins, contribute proportionally to the adjacent bins.  For example, a gradient with an angle of 40 degrees, is right in the middle of the 30-degree and 50-degree bin. Therefore, the magnitude of the gradient is split evenly into the 30-degree and 50-degree bin. This results in there being two dominant gradients in the cell near the diagonal edge instead of just one.

Now that you know how HOG is implemented, in the workspace you will find a notebook named *Examples*. In there, you will be able set your own paramters for the HOG descriptor for various images. Have fun!

```python

```