## 특잇값 분해
- 정방행렬은 고유분해로 고윳값과 고유벡터를 찾을 수 있었다
- 정방행렬이 아닌 행렬은 고유분해가 불가능
- 하지만 고유분해와 비슷한 특잇값 분해를 할 수 있다.

$N \times M$ 크기의 행렬 $A$에 대해
$$
\begin{align}
A = U\Sigma V^T
\tag{3.4.1}
\end{align}
$$

여기에서 $U, \Sigma, V$는 다음 조건을 만족해야 한다.

* 대각성분이 양수인 대각행렬이어야 한다. 큰 수부터 작은 수 순서로 배열한다.

$$
\begin{align}
\Sigma \in \mathbf{R}^{N \times M}
\tag{3.4.2}
\end{align}
$$

* $U$는 $N$차원 정방행렬로 모든 열벡터가 단위벡터이고 서로 직교해야 한다.

$$
\begin{align}
U \in \mathbf{R}^{N \times N}
\tag{3.4.3}
\end{align}
$$

* $V$는 $M$차원 정방행렬로 모든 열벡터가 단위벡터이고 서로 직교해야 한다.

$$
\begin{align}
V \in \mathbf{R}^{M \times M}
\tag{3.4.4}
\end{align}
$$

위 조건을 만족하는 행렬 $\Sigma$의 대각성분들을 **특잇값(singular value)**, 행렬 $U$의 열벡터들을 **왼쪽 특이벡터**(left singular vector), 행렬 $V$의 행벡터들을 **오른쪽 특이벡터**(right singular vector)라고 부른다.

> **[정리] 특이분해는 모든 행렬에 대해 가능하다. 즉 어떤 행렬이 주어지더라도 위와 같이 특이분해할 수 있다.**

### 1차원 근사
2차원 평면 위에 3개의 2차원 벡터 $a_1, a_2, a_3$가 있다. 원점을 지나면서 모든 점들과 가능한 한 가까이 있는 직선을 만들고 싶다면 직선의 방향을 어떻게 해야 할까? 직선의 방향을 나타내는 단위 벡터를 $w$라고 하자.

벡터 $w$와 점 $a_i$의 거리의 제곱은 다음처럼 계산할 수 있다.(연습 문제 3.1.9)

$$
\begin{align}
\Vert a_i^{\perp w}\Vert^2 = \Vert a_i\Vert^2 - \Vert a_i^{\Vert w}\Vert^2 = \Vert a_i\Vert^2 - (a_i^Tw)^2
\tag{3.4.34}
\end{align}
$$

벡터 $a_1, a_2, a_3$를 행벡터로 가지는 행렬 $A$를 가정하면

$$
\begin{align}
A = \begin{bmatrix} a_1^T \\ a_2^T \\ a_3^T \end{bmatrix}
\tag{3.4.35}
\end{align}
$$

행벡터의 놈의 제곱의 합은 행렬의 놈이므로 모든 점들과의 거리의 제곱의 합은 행렬의 놈으로 계산된다. (연습 문제 2.3.2)

$$
\begin{align}
\begin{aligned}
\sum_{i=1}^3 \Vert a_i^{\perp w}\Vert^2
&= \sum_{i=1}^3 \Vert a_i\Vert^2 - \sum_{i=1}^3  (a_i^Tw)^2 \\
&= \Vert A \Vert^2 - \Vert Aw\Vert^2 \\
\end{aligned}
\tag{3.4.36}
\end{align}
$$

점 $a_i$의 위치가 고정되어 있으므로 행렬 $A$의 놈 값은 고정되어 있다. 따라서 이 값이 가장 작아지려면 $\Vert Aw\Vert^2$의 값이 가장 크게 만드는 $w$를 찾아야 한다.이 문제는 다음처럼 수식으로 쓸 수 있다.

$$
\begin{align}
\arg\max_w \Vert Aw \Vert^2
\tag{3.4.37}
\end{align}
$$

#### 연습 문제 3.1.9

만약 $v$가 원점을 지나는 직선의 방향을 나타내는 단위벡터라고 하자. 이때 그 직선 위에 있지 않는 어떤 점 $x$와 그 직선과의 거리의 제곱이 다음과 같음을 증명하라.

$$
\begin{align}
\| x \|^2 - (x^Tv)^2
\tag{3.1.44}
\end{align}
$$

3.1.9 해답

따라서
이다.
v-직선 위에 있지 않는 어떤 점 x와 v-직선 사이의 거리는 $∥x^⊥v∥$이다

이때 직교성분의 길이를 구하면 v가 단위벡터$(||v|| = 1)$라는 점을 이용하여
$$∥x|v∥^2 = x^Tv /∥v∥ = x^Tv$$

따라서
$$∥x^⊥v∥2 = ∥x∥^2 −(x^Tv)^2$$

## PCA

```python
from sklearn.datasets import load_iris
iris = load_iris()
N = 10  # 앞의 10송이만 선택
```

```python
N
```

    10

```python
X = iris.data[:N, :2]
```

```python
X
```

    array([[5.1, 3.5],
           [4.9, 3. ],
           [4.7, 3.2],
           [4.6, 3.1],
           [5. , 3.6],
           [5.4, 3.9],
           [4.6, 3.4],
           [5. , 3.4],
           [4.4, 2.9],
           [4.9, 3.1]])

```python
from sklearn.decomposition import PCA

pca1 = PCA(n_components=1)
```

```python
X_low = pca1.fit_transform(X)
X_low
#잠재변수. 크기를 나타냄. 2차원을 1차원으로 축소
#원점을 데이터가 모여있는 가운데로 옮겨서 마이너스 값도 나옴
```

    array([[ 0.30270263],
           [-0.1990931 ],
           [-0.18962889],
           [-0.33097106],
           [ 0.30743473],
           [ 0.79976625],
           [-0.11185966],
           [ 0.16136046],
           [-0.61365539],
           [-0.12605597]])

```python
X2 = pca1.inverse_transform(X_low)
X2
#저차원으로 투영시킨 것을 다시 원래 좌표로 표시되도록 변환
```

    array([[5.06676112, 3.53108532],
           [4.7240094 , 3.1645881 ],
           [4.73047393, 3.17150049],
           [4.63393012, 3.06826822],
           [5.06999338, 3.53454152],
           [5.40628057, 3.89412635],
           [4.78359423, 3.22830091],
           [4.97021731, 3.42785306],
           [4.44084251, 2.86180369],
           [4.77389743, 3.21793233]])

```python
pca1.mean_
#데이터가 뭉쳐있는 중심위치, 즉 원점
```

    array([4.86, 3.31])

```python
pca1.components_
#기저벡터의 방향을 나타내는 단위벡터
#오른쪽 특이벡터중에서 가장 큰 값
#v1
#1차원으로 내렸기 때문에 v1만 나옴
#2차원으로 내리면 v1, v2 가 나옴
```

    array([[ 0.36138659, -0.08452251,  0.85667061,  0.3582892 ]])

```python
from sklearn.datasets import load_iris
iris = load_iris()
N = 50
```

```python
X = iris.data
X
```

    array([[5.1, 3.5, 1.4, 0.2],
           [4.9, 3. , 1.4, 0.2],
           [4.7, 3.2, 1.3, 0.2],
           [4.6, 3.1, 1.5, 0.2],
           [5. , 3.6, 1.4, 0.2],
           [5.4, 3.9, 1.7, 0.4],
           [4.6, 3.4, 1.4, 0.3],
           [5. , 3.4, 1.5, 0.2],
           [4.4, 2.9, 1.4, 0.2],
           [4.9, 3.1, 1.5, 0.1],
           [5.4, 3.7, 1.5, 0.2],
           [4.8, 3.4, 1.6, 0.2],
           [4.8, 3. , 1.4, 0.1],
           [4.3, 3. , 1.1, 0.1],
           [5.8, 4. , 1.2, 0.2],
           [5.7, 4.4, 1.5, 0.4],
           [5.4, 3.9, 1.3, 0.4],
           [5.1, 3.5, 1.4, 0.3],
           [5.7, 3.8, 1.7, 0.3],
           [5.1, 3.8, 1.5, 0.3],
           [5.4, 3.4, 1.7, 0.2],
           [5.1, 3.7, 1.5, 0.4],
           [4.6, 3.6, 1. , 0.2],
           [5.1, 3.3, 1.7, 0.5],
           [4.8, 3.4, 1.9, 0.2],
           [5. , 3. , 1.6, 0.2],
           [5. , 3.4, 1.6, 0.4],
           [5.2, 3.5, 1.5, 0.2],
           [5.2, 3.4, 1.4, 0.2],
           [4.7, 3.2, 1.6, 0.2],
           [4.8, 3.1, 1.6, 0.2],
           [5.4, 3.4, 1.5, 0.4],
           [5.2, 4.1, 1.5, 0.1],
           [5.5, 4.2, 1.4, 0.2],
           [4.9, 3.1, 1.5, 0.2],
           [5. , 3.2, 1.2, 0.2],
           [5.5, 3.5, 1.3, 0.2],
           [4.9, 3.6, 1.4, 0.1],
           [4.4, 3. , 1.3, 0.2],
           [5.1, 3.4, 1.5, 0.2],
           [5. , 3.5, 1.3, 0.3],
           [4.5, 2.3, 1.3, 0.3],
           [4.4, 3.2, 1.3, 0.2],
           [5. , 3.5, 1.6, 0.6],
           [5.1, 3.8, 1.9, 0.4],
           [4.8, 3. , 1.4, 0.3],
           [5.1, 3.8, 1.6, 0.2],
           [4.6, 3.2, 1.4, 0.2],
           [5.3, 3.7, 1.5, 0.2],
           [5. , 3.3, 1.4, 0.2],
           [7. , 3.2, 4.7, 1.4],
           [6.4, 3.2, 4.5, 1.5],
           [6.9, 3.1, 4.9, 1.5],
           [5.5, 2.3, 4. , 1.3],
           [6.5, 2.8, 4.6, 1.5],
           [5.7, 2.8, 4.5, 1.3],
           [6.3, 3.3, 4.7, 1.6],
           [4.9, 2.4, 3.3, 1. ],
           [6.6, 2.9, 4.6, 1.3],
           [5.2, 2.7, 3.9, 1.4],
           [5. , 2. , 3.5, 1. ],
           [5.9, 3. , 4.2, 1.5],
           [6. , 2.2, 4. , 1. ],
           [6.1, 2.9, 4.7, 1.4],
           [5.6, 2.9, 3.6, 1.3],
           [6.7, 3.1, 4.4, 1.4],
           [5.6, 3. , 4.5, 1.5],
           [5.8, 2.7, 4.1, 1. ],
           [6.2, 2.2, 4.5, 1.5],
           [5.6, 2.5, 3.9, 1.1],
           [5.9, 3.2, 4.8, 1.8],
           [6.1, 2.8, 4. , 1.3],
           [6.3, 2.5, 4.9, 1.5],
           [6.1, 2.8, 4.7, 1.2],
           [6.4, 2.9, 4.3, 1.3],
           [6.6, 3. , 4.4, 1.4],
           [6.8, 2.8, 4.8, 1.4],
           [6.7, 3. , 5. , 1.7],
           [6. , 2.9, 4.5, 1.5],
           [5.7, 2.6, 3.5, 1. ],
           [5.5, 2.4, 3.8, 1.1],
           [5.5, 2.4, 3.7, 1. ],
           [5.8, 2.7, 3.9, 1.2],
           [6. , 2.7, 5.1, 1.6],
           [5.4, 3. , 4.5, 1.5],
           [6. , 3.4, 4.5, 1.6],
           [6.7, 3.1, 4.7, 1.5],
           [6.3, 2.3, 4.4, 1.3],
           [5.6, 3. , 4.1, 1.3],
           [5.5, 2.5, 4. , 1.3],
           [5.5, 2.6, 4.4, 1.2],
           [6.1, 3. , 4.6, 1.4],
           [5.8, 2.6, 4. , 1.2],
           [5. , 2.3, 3.3, 1. ],
           [5.6, 2.7, 4.2, 1.3],
           [5.7, 3. , 4.2, 1.2],
           [5.7, 2.9, 4.2, 1.3],
           [6.2, 2.9, 4.3, 1.3],
           [5.1, 2.5, 3. , 1.1],
           [5.7, 2.8, 4.1, 1.3],
           [6.3, 3.3, 6. , 2.5],
           [5.8, 2.7, 5.1, 1.9],
           [7.1, 3. , 5.9, 2.1],
           [6.3, 2.9, 5.6, 1.8],
           [6.5, 3. , 5.8, 2.2],
           [7.6, 3. , 6.6, 2.1],
           [4.9, 2.5, 4.5, 1.7],
           [7.3, 2.9, 6.3, 1.8],
           [6.7, 2.5, 5.8, 1.8],
           [7.2, 3.6, 6.1, 2.5],
           [6.5, 3.2, 5.1, 2. ],
           [6.4, 2.7, 5.3, 1.9],
           [6.8, 3. , 5.5, 2.1],
           [5.7, 2.5, 5. , 2. ],
           [5.8, 2.8, 5.1, 2.4],
           [6.4, 3.2, 5.3, 2.3],
           [6.5, 3. , 5.5, 1.8],
           [7.7, 3.8, 6.7, 2.2],
           [7.7, 2.6, 6.9, 2.3],
           [6. , 2.2, 5. , 1.5],
           [6.9, 3.2, 5.7, 2.3],
           [5.6, 2.8, 4.9, 2. ],
           [7.7, 2.8, 6.7, 2. ],
           [6.3, 2.7, 4.9, 1.8],
           [6.7, 3.3, 5.7, 2.1],
           [7.2, 3.2, 6. , 1.8],
           [6.2, 2.8, 4.8, 1.8],
           [6.1, 3. , 4.9, 1.8],
           [6.4, 2.8, 5.6, 2.1],
           [7.2, 3. , 5.8, 1.6],
           [7.4, 2.8, 6.1, 1.9],
           [7.9, 3.8, 6.4, 2. ],
           [6.4, 2.8, 5.6, 2.2],
           [6.3, 2.8, 5.1, 1.5],
           [6.1, 2.6, 5.6, 1.4],
           [7.7, 3. , 6.1, 2.3],
           [6.3, 3.4, 5.6, 2.4],
           [6.4, 3.1, 5.5, 1.8],
           [6. , 3. , 4.8, 1.8],
           [6.9, 3.1, 5.4, 2.1],
           [6.7, 3.1, 5.6, 2.4],
           [6.9, 3.1, 5.1, 2.3],
           [5.8, 2.7, 5.1, 1.9],
           [6.8, 3.2, 5.9, 2.3],
           [6.7, 3.3, 5.7, 2.5],
           [6.7, 3. , 5.2, 2.3],
           [6.3, 2.5, 5. , 1.9],
           [6.5, 3. , 5.2, 2. ],
           [6.2, 3.4, 5.4, 2.3],
           [5.9, 3. , 5.1, 1.8]])

```python
from sklearn.decomposition import PCA

pca1 = PCA(n_components=1)
```

```python
X_low = pca1.fit_transform(X)
X_low
```

    array([[-2.68412563],
           [-2.71414169],
           [-2.88899057],
           [-2.74534286],
           [-2.72871654],
           [-2.28085963],
           [-2.82053775],
           [-2.62614497],
           [-2.88638273],
           [-2.6727558 ],
           [-2.50694709],
           [-2.61275523],
           [-2.78610927],
           [-3.22380374],
           [-2.64475039],
           [-2.38603903],
           [-2.62352788],
           [-2.64829671],
           [-2.19982032],
           [-2.5879864 ],
           [-2.31025622],
           [-2.54370523],
           [-3.21593942],
           [-2.30273318],
           [-2.35575405],
           [-2.50666891],
           [-2.46882007],
           [-2.56231991],
           [-2.63953472],
           [-2.63198939],
           [-2.58739848],
           [-2.4099325 ],
           [-2.64886233],
           [-2.59873675],
           [-2.63692688],
           [-2.86624165],
           [-2.62523805],
           [-2.80068412],
           [-2.98050204],
           [-2.59000631],
           [-2.77010243],
           [-2.84936871],
           [-2.99740655],
           [-2.40561449],
           [-2.20948924],
           [-2.71445143],
           [-2.53814826],
           [-2.83946217],
           [-2.54308575],
           [-2.70335978],
           [ 1.28482569],
           [ 0.93248853],
           [ 1.46430232],
           [ 0.18331772],
           [ 1.08810326],
           [ 0.64166908],
           [ 1.09506066],
           [-0.74912267],
           [ 1.04413183],
           [-0.0087454 ],
           [-0.50784088],
           [ 0.51169856],
           [ 0.26497651],
           [ 0.98493451],
           [-0.17392537],
           [ 0.92786078],
           [ 0.66028376],
           [ 0.23610499],
           [ 0.94473373],
           [ 0.04522698],
           [ 1.11628318],
           [ 0.35788842],
           [ 1.29818388],
           [ 0.92172892],
           [ 0.71485333],
           [ 0.90017437],
           [ 1.33202444],
           [ 1.55780216],
           [ 0.81329065],
           [-0.30558378],
           [-0.06812649],
           [-0.18962247],
           [ 0.13642871],
           [ 1.38002644],
           [ 0.58800644],
           [ 0.80685831],
           [ 1.22069088],
           [ 0.81509524],
           [ 0.24595768],
           [ 0.16641322],
           [ 0.46480029],
           [ 0.8908152 ],
           [ 0.23054802],
           [-0.70453176],
           [ 0.35698149],
           [ 0.33193448],
           [ 0.37621565],
           [ 0.64257601],
           [-0.90646986],
           [ 0.29900084],
           [ 2.53119273],
           [ 1.41523588],
           [ 2.61667602],
           [ 1.97153105],
           [ 2.35000592],
           [ 3.39703874],
           [ 0.52123224],
           [ 2.93258707],
           [ 2.32122882],
           [ 2.91675097],
           [ 1.66177415],
           [ 1.80340195],
           [ 2.1655918 ],
           [ 1.34616358],
           [ 1.58592822],
           [ 1.90445637],
           [ 1.94968906],
           [ 3.48705536],
           [ 3.79564542],
           [ 1.30079171],
           [ 2.42781791],
           [ 1.19900111],
           [ 3.49992004],
           [ 1.38876613],
           [ 2.2754305 ],
           [ 2.61409047],
           [ 1.25850816],
           [ 1.29113206],
           [ 2.12360872],
           [ 2.38800302],
           [ 2.84167278],
           [ 3.23067366],
           [ 2.15943764],
           [ 1.44416124],
           [ 1.78129481],
           [ 3.07649993],
           [ 2.14424331],
           [ 1.90509815],
           [ 1.16932634],
           [ 2.10761114],
           [ 2.31415471],
           [ 1.9222678 ],
           [ 1.41523588],
           [ 2.56301338],
           [ 2.41874618],
           [ 1.94410979],
           [ 1.52716661],
           [ 1.76434572],
           [ 1.90094161],
           [ 1.39018886]])

```python
X2 = pca1.inverse_transform(X_low)
X2
#저차원으로 투영시킨 것을 다시 원래 좌표로 표시되도록 변환
```

    array([[4.87332632, 3.28420238, 1.45858847, 0.23764012],
           [4.86247892, 3.28673941, 1.4328746 , 0.22688569],
           [4.79929088, 3.30151808, 1.2830867 , 0.16423922],
           [4.85120324, 3.28937661, 1.40614547, 0.21570665],
           [4.85721176, 3.28797132, 1.42038875, 0.22166368],
           [5.01906124, 3.25011732, 1.8040546 , 0.38212597],
           [4.82402881, 3.29573228, 1.34172822, 0.18876513],
           [4.89427975, 3.27930171, 1.50825879, 0.25841396],
           [4.80023332, 3.30129766, 1.28532076, 0.16517358],
           [4.87743522, 3.28324137, 1.46832867, 0.2417138 ],
           [4.93735627, 3.2692268 , 1.61037212, 0.30112127],
           [4.89911863, 3.27816997, 1.51972939, 0.26321136],
           [4.8364708 , 3.29282229, 1.37122209, 0.20110048],
           [4.67829389, 3.32981733, 0.99626209, 0.04427928],
           [4.887556  , 3.28087429, 1.49232008, 0.25174784],
           [4.98105082, 3.25900735, 1.7139505 , 0.34444132],
           [4.89522554, 3.27908051, 1.51050079, 0.25935164],
           [4.88627441, 3.28117403, 1.48928206, 0.25047723],
           [5.04834776, 3.24326768, 1.87347859, 0.41116148],
           [4.90806975, 3.27607645, 1.54094812, 0.27208576],
           [5.00843771, 3.252602  , 1.77887141, 0.37159349],
           [4.92407237, 3.27233369, 1.5788825 , 0.28795123],
           [4.68113595, 3.32915262, 1.00299923, 0.04709698],
           [5.01115644, 3.25196613, 1.78531617, 0.37428891],
           [4.99199541, 3.25644759, 1.73989475, 0.35529211],
           [4.9374568 , 3.26920329, 1.61061043, 0.30122094],
           [4.95113486, 3.26600421, 1.64303441, 0.31478177],
           [4.91734528, 3.27390705, 1.56293585, 0.28128179],
           [4.88944088, 3.28043344, 1.4967882 , 0.25361656],
           [4.89216766, 3.27979569, 1.50325206, 0.25631997],
           [4.90828222, 3.27602676, 1.54145178, 0.27229641],
           [4.97241604, 3.26102689, 1.69348167, 0.33588055],
           [4.88607   , 3.28122184, 1.4887975 , 0.25027457],
           [4.90418472, 3.2769851 , 1.53173861, 0.26823403],
           [4.89038332, 3.28021302, 1.49902225, 0.25455092],
           [4.80751203, 3.29959528, 1.30257503, 0.17238991],
           [4.8946075 , 3.27922505, 1.50903573, 0.2587389 ],
           [4.83120365, 3.2940542 , 1.35873624, 0.19587847],
           [4.76621986, 3.30925286, 1.20469151, 0.13145165],
           [4.90733978, 3.27624718, 1.53921772, 0.27136205],
           [4.84225546, 3.29146935, 1.38493468, 0.20683556],
           [4.81360969, 3.29816914, 1.31702958, 0.17843531],
           [4.7601108 , 3.31068167, 1.19020992, 0.12539495],
           [4.97397651, 3.26066192, 1.69718078, 0.33742765],
           [5.04485355, 3.24408492, 1.86519552, 0.40769721],
           [4.86236698, 3.28676559, 1.43260925, 0.22677471],
           [4.92608058, 3.27186401, 1.58364299, 0.28994223],
           [4.81718978, 3.29733181, 1.32551622, 0.18198471],
           [4.92429624, 3.27228133, 1.57941319, 0.28817318],
           [4.86637536, 3.2858281 , 1.44211114, 0.23074873],
           [6.30765211, 2.94873664, 4.8586724 , 1.6596725 ],
           [6.18032219, 2.97851706, 4.55683552, 1.5334339 ],
           [6.37251256, 2.93356682, 5.01242476, 1.72397704],
           [5.9095819 , 3.04183886, 3.9150429 , 1.26501409],
           [6.23655926, 2.96536411, 4.69014608, 1.58918898],
           [6.07522394, 3.00309785, 4.30769904, 1.42923643],
           [6.23907357, 2.96477605, 4.69610628, 1.59168174],
           [5.57261044, 3.12065106, 3.11624863, 0.93093077],
           [6.22066858, 2.96908069, 4.65247704, 1.57343449],
           [5.84017286, 3.05807252, 3.75050807, 1.19619995],
           [5.65980645, 3.10025732, 3.32294764, 1.01737943],
           [6.02825433, 3.01408328, 4.19635711, 1.3826694 ],
           [5.93909229, 3.03493685, 3.98499759, 1.29427155],
           [6.19927546, 2.97408419, 4.60176444, 1.55222473],
           [5.78047904, 3.07203394, 3.60900325, 1.13701775],
           [6.17864978, 2.97890821, 4.55287106, 1.53177583],
           [6.08195103, 3.00152449, 4.32364569, 1.43590587],
           [5.92865851, 3.03737715, 3.96026421, 1.2839272 ],
           [6.18474744, 2.97748206, 4.56732562, 1.53782122],
           [5.85967776, 3.05351064, 3.79674462, 1.21553767],
           [6.24674311, 2.96298227, 4.71428699, 1.59928554],
           [5.97266941, 3.0270837 , 4.06459249, 1.32756089],
           [6.31247958, 2.94760757, 4.87011597, 1.66445859],
           [6.17643381, 2.97942649, 4.54761807, 1.52957885],
           [6.10167174, 2.99691213, 4.37039383, 1.45545756],
           [6.16864428, 2.98124833, 4.52915293, 1.52185609],
           [6.3247091 , 2.94474728, 4.89910618, 1.6765833 ],
           [6.40630214, 2.92566398, 5.09252332, 1.75747702],
           [6.13724567, 2.98859196, 4.45472219, 1.49072659],
           [5.73289945, 3.08316204, 3.49621536, 1.08984597],
           [5.81871333, 3.06309156, 3.69963804, 1.17492435],
           [5.77480631, 3.0733607 , 3.595556  , 1.13139365],
           [5.89263684, 3.04580204, 3.87487447, 1.24821427],
           [6.34205638, 2.94069003, 4.94022808, 1.6937819 ],
           [6.05583098, 3.00763355, 4.26172784, 1.41000969],
           [6.13492111, 2.98913564, 4.4492118 , 1.48842195],
           [6.28447465, 2.95415747, 4.80373   , 1.63669369],
           [6.13789782, 2.98843943, 4.45626813, 1.49137315],
           [5.93221914, 3.03654437, 3.96870471, 1.28745731],
           [5.90347284, 3.04326767, 3.90056131, 1.25895739],
           [6.01130593, 3.01804724, 4.15618074, 1.36586626],
           [6.165262  , 2.98203939, 4.5211352 , 1.5185028 ],
           [5.9266503 , 3.03784683, 3.95550372, 1.2819362 ],
           [5.588725  , 3.11688213, 3.15444835, 0.94690721],
           [5.97234166, 3.02716036, 4.06381555, 1.32723595],
           [5.96329   , 3.0292774 , 4.04235851, 1.31826187],
           [5.97929263, 3.02553464, 4.08029289, 1.33412734],
           [6.07555169, 3.00302119, 4.30847598, 1.42956138],
           [5.51574728, 3.13395045, 2.98145391, 0.87455497],
           [5.95138823, 3.03206103, 4.01414523, 1.3064621 ],
           [6.75807245, 2.84339056, 5.92639841, 2.10623234],
           [6.3547806 , 2.93771404, 4.97039098, 1.70639706],
           [6.78896496, 2.8361653 , 5.99962943, 2.13686008],
           [6.55581822, 2.89069457, 5.4469527 , 1.90571161],
           [6.69259396, 2.85870492, 5.771181  , 2.04131507],
           [7.07097758, 2.77020708, 6.66814323, 2.41645561],
           [6.03169968, 3.01327747, 4.20452434, 1.38608522],
           [6.90313098, 2.8094637 , 6.27026114, 2.2500476 ],
           [6.6821943 , 2.86113724, 5.7465285 , 2.03100454],
           [6.89740802, 2.81080221, 6.25669482, 2.2443737 ],
           [6.44387623, 2.916876  , 5.18159307, 1.79472906],
           [6.49505862, 2.90490527, 5.30292144, 1.84547277],
           [6.62594917, 2.87429207, 5.61319884, 1.97524148],
           [6.3298188 , 2.9435522 , 4.91121877, 1.6816492 ],
           [6.41646653, 2.92328669, 5.11661809, 1.76755428],
           [6.53157833, 2.89636389, 5.3894918 , 1.88167948],
           [6.54792482, 2.89254071, 5.42824131, 1.89788586],
           [7.10350839, 2.76259865, 6.74525783, 2.4487076 ],
           [7.2150287 , 2.73651584, 7.00961786, 2.55927208],
           [6.31342202, 2.94738715, 4.87235002, 1.66539295],
           [6.72071417, 2.85212806, 5.83784024, 2.06919426],
           [6.27663626, 2.95599075, 4.785149  , 1.62892248],
           [7.10815751, 2.76151129, 6.75627862, 2.45331687],
           [6.34521479, 2.93995133, 4.94771512, 1.69691324],
           [6.66564341, 2.86500823, 5.70729443, 2.0145955 ],
           [6.78803058, 2.83638383, 5.99741447, 2.13593371],
           [6.29814131, 2.95096106, 4.83612695, 1.65024321],
           [6.30993115, 2.94820361, 4.86407488, 1.661932  ],
           [6.61077705, 2.87784059, 5.57723317, 1.9601994 ],
           [6.7063256 , 2.85549331, 5.80373199, 2.05492902],
           [6.87027577, 2.81714801, 6.19237754, 2.21747399],
           [7.01085548, 2.78426867, 6.52562316, 2.35684881],
           [6.62372514, 2.87481223, 5.60792675, 1.97303651],
           [6.36523384, 2.93526919, 4.99517049, 1.71676071],
           [6.48706939, 2.90677382, 5.2839829 , 1.83755202],
           [6.95513916, 2.79729982, 6.39354706, 2.30161002],
           [6.61823412, 2.8760965 , 5.59491022, 1.96759255],
           [6.53181026, 2.89630965, 5.39004159, 1.88190942],
           [6.26591219, 2.95849893, 4.7597275 , 1.61829033],
           [6.60499574, 2.87919274, 5.56352852, 1.95446764],
           [6.67963782, 2.86173516, 5.74046831, 2.02846996],
           [6.53801514, 2.89485843, 5.40475032, 1.88806112],
           [6.3547806 , 2.93771404, 4.97039098, 1.70639706],
           [6.769572  , 2.840701  , 5.95365822, 2.11763334],
           [6.71743577, 2.85289483, 5.83006876, 2.06594396],
           [6.54590855, 2.89301229, 5.42346172, 1.89588687],
           [6.39523087, 2.92825337, 5.06627875, 1.74650063],
           [6.48094422, 2.9082064 , 5.26946311, 1.83147934],
           [6.53030814, 2.89666097, 5.3864808 , 1.88042018],
           [6.34572895, 2.93983108, 4.94893393, 1.69742298]])

```python
# #경고 무시
# import warnings
# warnings.simplefilter('ignore')

# 자주 사용하는 패키지를 임포트
import matplotlib as mpl
import matplotlib.pylab as plt
from mpl_toolkits.mplot3d import Axes3D
import seaborn as sns
import numpy as np
import scipy as sp
import pandas as pd
import statsmodels.api as sm
import sklearn as sk

# matplotlib 설정
mpl.use('Agg')

# seaborn 설정
sns.set()
sns.set_style("whitegrid")
sns.set_color_codes()
```

```python
%matplotlib inline
```

```python
from sklearn.datasets import fetch_olivetti_faces

faces_all = fetch_olivetti_faces()
K = 10  # 10번 인물의 사진만 선택
faces = faces_all.images[faces_all.target == K]

N = 2
M = 5
fig = plt.figure(figsize=(10, 5))
plt.subplots_adjust(top=1, bottom=0, hspace=0, wspace=0.05)
for i in range(N):
    for j in range(M):
        k = i * M + j
        ax = fig.add_subplot(N, M, k+1)
        ax.imshow(faces[k], cmap=plt.cm.bone)
        ax.grid(False)
        ax.xaxis.set_ticks([])
        ax.yaxis.set_ticks([])
plt.suptitle("Face")
plt.tight_layout()
plt.show()
```

```python
from sklearn.decomposition import PCA
pca3 = PCA(n_components=2)
X3 = faces_all.data[faces_all.target == K]
W3 = pca3.fit_transform(X3)
X32 = pca3.inverse_transform(W3)
```

```python
N = 2
M = 5
fig = plt.figure(figsize=(10, 5))
plt.subplots_adjust(top=1, bottom=0, hspace=0, wspace=0.05)
for i in range(N):
    for j in range(M):
        k = i * M + j
        ax = fig.add_subplot(N, M, k+1)
        ax.imshow(X32[k].reshape(64, 64), cmap=plt.cm.bone)
        ax.grid(False)
        ax.xaxis.set_ticks([])
        ax.yaxis.set_ticks([])
plt.suptitle("Face2")
plt.tight_layout()
plt.show()
```

```python
face_mean = pca3.mean_.reshape(64, 64)
face_p1 = pca3.components_[0].reshape(64, 64)
face_p2 = pca3.components_[1].reshape(64, 64)

plt.subplot(131)
plt.imshow(face_mean, cmap=plt.cm.bone)
plt.grid(False)
plt.xticks([])
plt.yticks([])
plt.title("평균 얼굴")
plt.subplot(132)
plt.imshow(face_p1, cmap=plt.cm.bone)
plt.grid(False)
plt.xticks([])
plt.yticks([])
plt.title("주성분 1")
plt.subplot(133)
plt.imshow(face_p2, cmap=plt.cm.bone)
plt.grid(False)
plt.xticks([])
plt.yticks([])
plt.title("주성분 2")
plt.show()
```

```python
N = 2
M = 5
fig = plt.figure(figsize=(10, 5))
plt.subplots_adjust(top=1, bottom=0, hspace=0, wspace=0.05)
for i in range(N):
    for j in range(M):
        k = i * M + j
        ax = fig.add_subplot(N, M, k+1)
        w = 1.5 * (k - 5) if k < 5 else 1.5 * (k - 4)
        ax.imshow(face_mean + w * face_p1, cmap=plt.cm.bone)
        ax.grid(False)
        ax.xaxis.set_ticks([])
        ax.yaxis.set_ticks([])
        plt.title("주성분1의 비중={}".format(w))
plt.suptitle("평균 얼굴에 주성분1을 더한 사진")
plt.tight_layout()
plt.show()
```

```python
N = 2
M = 5
fig = plt.figure(figsize=(10, 5))
plt.subplots_adjust(top=1, bottom=0, hspace=0, wspace=0.05)
for i in range(N):
    for j in range(M):
        k = i * M + j
        ax = fig.add_subplot(N, M, k+1)
        w = 1.5 * (k - 5) if k < 5 else 1.5 * (k - 4)
        ax.imshow(face_mean + w * face_p2, cmap=plt.cm.bone)
        ax.grid(False)
        ax.xaxis.set_ticks([])
        ax.yaxis.set_ticks([])
        plt.title("주성분2의 비중={:.1f}".format(w))
plt.suptitle("평균 얼굴에 주성분2를 더한 사진")
plt.tight_layout()
plt.show()
```

```python

```


# DL Indaba Practical 3
# Convolutional Neural Networks
*Developed by Stephan Gouws, Avishkar Bhoopchand & Ulrich Paquet.*

**Introduction**

In this practical we will cover the basics of convolutional neural networks, or "ConvNets". ConvNets were invented in the late 1980s/early 1990s, and have had tremendous success especially with vision (although they have also been used to great success in speech processing pipelines, and more recently, for machine translation).

We will work to build our mathematical and algorithmic intuition around the "convolution" operation. Then we will construct a deep feedforward convolutional model with which we can classify MNIST digits with over 99% accuracy (our best model yet!).

**Learning objectives**

Understand:
* what a convolutional layer is & how it's different from a fully-connected layer (including the assumptions and trade-offs that are being made),
* how and when to use convolutional layers (relate it to the assumptions the model makes),
* how backpropagation works through convolutional layers.

**What is expected of you:**

* Read through the explanations and make sure you understand how to implement the convolutional forwards pass.
* Do the same for the backwards phase.
* Train a small model on MNIST.
* At this point, flag a tutor and they will give you access to a GPU instance. Now use the hyperparameters provided to train a state-of-the-art ConvNet model on MNIST.

```python
# Import TensorFlow and some other libraries we'll be using.
import datetime
import numpy as np
import tensorflow as tf

from tensorflow.examples.tutorials.mnist import input_data

# Import Matplotlib and set some defaults
from matplotlib import pyplot as plt
plt.ioff()
%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# Download the MNIST dataset onto the local machine.
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
```

    Extracting MNIST_data/train-images-idx3-ubyte.gz
    Extracting MNIST_data/train-labels-idx1-ubyte.gz
    Extracting MNIST_data/t10k-images-idx3-ubyte.gz
    Extracting MNIST_data/t10k-labels-idx1-ubyte.gz

# ConvNet Architectures
When modelling an image using a regular feed-forward network, we quickly find that the number of model parameters grows exponentially. For example, our 2 layer MNIST feed-forward model from the previous practical already had over 600 000 parameters!

**QUESTION**: How many parameters would a feed-forward network require if it had 2 hidden layers with 512 and 256 neurons respectively, an output size of 10 and an input image of shape [32, 32, 3]?

ConvNets address this model parameter issue by exploiting structure in the inputs to the network (in particular, by making the assumption that the input is a 3D volume, which applies to images for example). The two key differences between a ConvNet and a Feed-forward network are:
* ConvNets have neurons that are arranged in 3 dimensions: width, height, depth (depth here means the depth of an activation volume, not the depth of a deep neural network!)
* The neurons in each layer are only connected to a small region of the layer before it.

**QUESTION**: Unfortunately there is no such thing as a free lunch. What do you think is the trade-off a ConvNet makes for the reduction in memory required by fewer parameters?

Generally a ConvNet architecture is made up of different types of layers, the most common being **convolutional layers**, **pooling layers** and **fully connected layers** that we encountered in the last practical.

ConvNet architectures were key to the tremendous success of deep learning in machine vision. In particular, the first deep learning model to win the ImageNet competition in 2012 was called AlexNet (after Alex Krizhevsky, one of its inventors). It had 5 convolutional layers followed by 3 fully connected layers. Later winners included GoogLeNet and ResNet which also used batch normalisation, a technique we will see in this practical. If you're curious, have a look at [this](https://medium.com/towards-data-science/neural-network-architectures-156e5bad51ba) link for a great summary of different ConvNet archiectures.

We will start by implementing the forward and backward passes of these layers in Numpy to get a good sense for how they work. Afterwards, we will implement a full ConvNet classifier in TensorFlow that we will apply to the MNIST dataset. This model should give us the best test accuracy we've seen so far!

## Convolutional Layers
A convolutional layer maps an **input volume**\* to an **output volume** through a set of **learnable filters**, which make up the parameters of the layer. Every filter is small spatially (along width and height), but extends through the full depth of the input volume. (Eg: A filter in the first layer of a ConvNet might have size [5, 5, 3]). During the forward pass, we convolve ("slide") each filter across the width and height of the input volume and compute dot products between the entries of the filter and the input at any position. As we slide the filter over the width and height of the input volume we will produce a 2-dimensional activation map that gives the responses of that filter at every spatial position. Each convolutional layer will have a set of filters, and each of them will produce a separate 2-dimensional activation map. We will stack these activation maps along the depth dimension to produce the output volume.

The following diagram and animation illustrates these ideas, make sure you understand them!

\* An input volume refers to a 3 dimensional input. For example, a colour image is often represented as a 3 dimensional tensor of shape `[width, height, channels]` where `channels` refers to the colour values. A common colour encoding is RGB which has a value between 0 and 256 for each of the red, green and blue channels.

### What size is the output volume?
The size of the output volume is controlled by the hyperparameters of the convolutional layer:
* **Filter Size** (F) defines the width and height of the filters in the layer. Note that filters always have the same depth as the inputs to the layer.
* **Depth** (D) of the layer defines the number of filters in the layer.
* **Stride** (S) defines the number of pixels by which we move the filter when "sliding" it along the input volume. Typically this value would be 1, but values of 2 and 3 are also sometimes used.
* **Padding** (P) refers to the number of 0 pixels we add to the input volume along the width and height dimensions. This parameter is useful in that it gives us more control over the desired size of the output volume and in fact is often used to ensure that the output volume has the same width and height as the input volume.

If the width of the input volume is $w$, the width of the output volume will be $(w−F+2P)/S+1$. (**QUESTION:** Why?). Similarly for the height ($h$).

**QUESTION**: What is the final 3D shape of the output volume?

### Implementing the forward pass
The parameters of a convolutional layer, with padded input $X^{pad}$, are stored in a weight tensor, $W$ of shape $[F, F, I, D]$ and bias vector $b$ of shape $[D]$ where I is the depth of $X$.
For each filter $d \in [0,D)$ in our convolutional layer, the value of the output volume ($O$) at position $(i, j, d)$ is given by:

\begin{align}
O_{ij}^d = b_{d} + \sum_{a=0}^{F-1} \sum_{b=0}^{F-1} \sum_{c=0}^{I-1} W_{a, b, c, d} X^{pad}_{i+a, j+b, c}    && (1)
\end{align}

Don't be put off by all the notation, it's actually quite simple, see if you can tie this formula to the explanation of the convolutional layer and diagrams you saw earlier.

**QUESTION**: The formula above assumed a stride size of 1 for simplicity. Can you modify the formula to work with an arbitrary stride?

Now let's implement the forward pass of a convolutional layer in Numpy:

```python
## IMPLEMENT-ME: ...

# Conv layer forward pass
def convolutional_forward(X, W, b, filter_size, depth, stride, padding):
    # X has size [batch_size, input_width, input_height, input_depth]
    # W has shape [filter_size, filter_size, input_depth, depth]
    # b has shape [depth]
    batch_size, input_width, input_height, input_depth = X.shape

    # Check that the weights are of the expected shape
    assert W.shape == (filter_size, filter_size, input_depth, depth)

    # QUESTION: Calculate the width and height of the output
    # output_width = ...
    # output_height = ...
    #
    # ANSWER:
    output_width = (input_width - filter_size + 2*padding) / stride + 1
    output_height = (input_height - filter_size + 2*padding) / stride + 1
    ####

    # Apply padding to the width and height dimensions of the input
    X_padded = np.pad(X, ((0,0), (padding, padding), (padding, padding), (0,0)), 'constant')

    # Allocate the output Tensor
    out = np.zeros((batch_size, output_width, output_height, depth))

    # NOTE: There is a more efficient way of doing a convolution, but this most
    # clearly illustrates the idea.

    for i in range(output_width):    # Loop over the output width dimension
        for j in range(output_height):    # Loop over the output height dimension

            # Select the current block in the input that the filter will be applied to
            block_width_start = i * stride
            block_width_end = block_width_start + filter_size

            block_height_start = j * stride
            block_height_end = block_height_start + filter_size

            block = X_padded[:, block_width_start:block_width_end, block_height_start:block_height_end, :]

            for d in range(depth):    # Loop over the filters in the layer (output depth dimension)

                filter_weights = W[:, :, :, d]
                # QUESTION: Apply the filter to the block over all inputs in the batch
                # out[:, w, h, f] = ...
                # HINT: Have a look at numpy's sum function and pay attention to the axis parameter
                # ANSWER:
                out[:, i, j, d] = np.sum(block * filter_weights, axis=(1,2,3)) + b[d]
                ###

    return out

```

Let's test our layer on some dummy data:

```python
### Hyperparameters
batch_size = 2
input_width = 4
input_height = 4
input_depth = 3
filter_size = 4
output_depth = 3
stride = 2
padding = 1
###

# Create a helper function that calculates the relative error between two arrays
def relative_error(x, y):
    """ returns relative error """
    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))

# Define the shapes of the input and weights
input_shape = (batch_size, input_width, input_height, input_depth)
w_shape = (filter_size, filter_size, input_depth, output_depth)

# Create the dummy input
X = np.linspace(-0.1, 0.5, num=np.prod(input_shape)).reshape(input_shape)

# Create the weights and biases
W = np.linspace(-0.2, 0.3, num=np.prod(w_shape)).reshape(w_shape)
b = np.linspace(-0.1, 0.2, num=output_depth)

# Get the output of the convolutional layer
out = convolutional_forward(X, W, b, filter_size, output_depth, stride, padding)

correct_out = np.array(
        [[[[8.72013250e-02, 2.37300699e-01, 3.87400074e-01],
             [1.34245123e-01, 2.86133235e-01, 4.38021347e-01]],
            [[8.21928598e-02, 2.39447184e-01, 3.96701509e-01],
             [4.47552448e-04, 1.59490615e-01, 3.18533677e-01]]],
         [[[1.11179021e+00, 1.29050939e+00, 1.46922856e+00],
             [9.01255797e-01, 1.08176371e+00, 1.26227162e+00]],
            [[7.64688995e-02, 2.62343025e-01, 4.48217151e-01],
             [-2.62854619e-01, -7.51917556e-02, 1.12471108e-01]]]])

# Compare your output to the "correct" ones
# The difference should be around 2e-8 (or lower)

print 'Testing convolutional_forward'
diff = relative_error(out, correct_out)
if diff <= 2e-8:
    print 'PASSED'
else:
    print 'The difference of %s is too high, try again' % diff
```

    Testing convolutional_forward
    PASSED

### The derivative of a convolutional layer
Assume we have some final loss function L and by following the steps of backpropagation, have computed the derivative of this loss up to the output of our convolutional layer ($\frac{\partial L}{\partial O}$ or `dout` in the code below). In order to update the parameters of our layer, we require the derivative of L with respect to the weights and biases of the convolutional layer ($\frac{\partial L}{\partial W}$ and $\frac{\partial L}{\partial b}$). We also require the derivative with respect to the inputs of the layer ($\frac{\partial L}{\partial X}$) in order to propagate the error back to the preceding layers. Unfortunately calculating these derivatives can be a little fiddly due to having to keep track of multiple indices. The calculus is very basic though!

We start with the easiest one, $\frac{\partial L}{\partial b}$:

\begin{align}
\frac{\partial L}{\partial b} &= \frac{\partial L}{\partial O} \frac{\partial O}{\partial b} && \vartriangleright \text{(Chain Rule)} \\
 &= \frac{\partial L}{\partial O} \mathbf{1} && \vartriangleright (\frac{\partial O}{\partial b} = 1 \text{ from equation } (1))
\end{align}

Now we tackle $\frac{\partial L}{\partial W}$:

\begin{align}
\frac{\partial L}{\partial W} &= \frac{\partial L}{\partial O} \frac{\partial O}{\partial W} && \vartriangleright \text{(Chain Rule)}
\end{align}

Let's calculate this derivative with respect to a single point $W_{abcd}$ in our weight tensor ($O_w$ and $O_h$ are the output width and height respectively):

\begin{align}
\frac{\partial L}{\partial W_{abcd}} &= \sum_{i=0}^{O_w-1} \sum_{j=0}^{O_h-1} \frac{\partial L}{\partial O_{ij}^d} \frac{\partial O_{ij}^d}{\partial W_{abcd}}
\end{align}

**QUESTION**: Why do we sum over the outputs here? **HINT**: Think about how many times a particular weight gets used.

Now, looking at equation $(1)$, we can easily calculate $\frac{\partial O_{ij}^d}{\partial W_{abcd}}$ as:

\begin{align}
\frac{\partial O_{ij}^d}{\partial W_{abcd}} &= X^{pad}_{i+a, j+b, c}
\end{align}

Which gives a final result of:

\begin{align}
\frac{\partial L}{\partial W_{abcd}} &= \sum_{i=0}^{O_w-1} \sum_{j=0}^{O_h-1} \frac{\partial L}{\partial O_{ij}^d} X^{pad}_{i+a, j+b, c}
\end{align}

Finally, we need $\frac{\partial L}{\partial X}$, the derivative of the loss with respect to the **input** of the layer. This is sometimes also called a "delta". Remember, that before doing the convolution, we applied padding to the input $X$ to get $X^{pad}$. It's easier to calculate the derivative with respect to $X^{pad}$, which appears in our convolution equation, and then remove the padding later on to arrive at the delta. Unfortunately we need to introduce some *more* indexing for the individual components of $X^{pad}$:

\begin{align}
\frac{\partial L}{\partial X^{pad}_{mnc}} &= \sum_{i=0}^{O_w-1} \sum_{j=0}^{O_h-1} \sum_{d=0}^{D-1} W_{m-i, n-j, c, d} \frac{\partial L}{\partial O_{ij}^d}
\end{align}

Where do the indices $m-i$ and $n-j$ come from? Notice in equation $(1)$ that the padded input $X^{pad}_{i+a, j+b, c}$ is multiplied by the weight $W_{abcd}$. Now, when we index $X^{pad}$ with $m$ and $n$, setting $m=i+a$ and $n=j+b$ gives us $a=m-i$ and $b=n-j$ for the indices of $W$!

Phew! Spend a few minutes to understand these equations, particularly where the indices come from. Ask a tutor if you get stuck!

Note: Did you notice that the delta, $\frac{\partial L}{\partial X^{pad}_{mnc}}$ looks suspiciously like the convolutional forward equation with the inputs $X^{pad}$ replaced by $\frac{\partial L}{\partial O_{ij}^d}$ and different indexing into the weights? In fact the delta is exactly that, the forward convolution applied to the incoming derivative, with the filters flipped along the width and height axes.

Now let's implement this in Numpy:

```python
## IMPLEMENT-ME: ...

def convolutional_backward(dout, X, W, b, filter_size, depth, stride, padding):
    batch_size, input_width, input_height, input_depth = X.shape

    # Apply padding to the width and height dimensions of the input
    X_padded = np.pad(X, ((0,0), (padding, padding), (padding, padding), (0,0)), 'constant')

    # Calculate the width and height of the forward pass output
    output_width = (input_width - filter_size + 2*padding) / stride + 1
    output_height = (input_height - filter_size + 2*padding) / stride + 1

    # Allocate output arrays
    # QUESTION: What is the shape of dx? dw? db?
    # ANSWER: ...
    dx_padded = np.zeros_like(X_padded)
    dw = np.zeros_like(W)
    db = np.zeros_like(b)

    # QUESTION: Calculate db, the derivative of the final loss with respect to the bias term
    # HINT: Have a look at the axis parameter of the np.sum function.
    db = np.sum(dout, axis = (0, 1, 2))

    for i in range(output_width):
        for j in range(output_height):

            # Select the current block in the input that the filter will be applied to
            block_width_start = i*stride
            block_width_end = block_width_start+filter_size

            block_height_start = j*stride
            block_height_end = block_height_start + filter_size

            block = X_padded[:, block_width_start:block_width_end, block_height_start:block_height_end, :]

            for d in range(depth):
                # QUESTION: Calculate dw[:,:,:,f], the derivative of the loss with respect to the weight parameters of the f'th filter.
                # HINT: You can do this in a loop if you prefer, or use np.sum and "None" indexing to get your result to the correct
                # shape to assign to dw[:,:,:,f], see (https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#numpy.newaxis)
                dw[:,:,:,d] += np.sum(block*(dout[:,i,j,d])[:,None,None,None], axis=0)

            dx_padded[:,block_width_start:block_width_end, block_height_start:block_height_end, :] += np.einsum('ij,klmj->iklm', dout[:,i,j,:], W)

        # Now we remove the padding to arrive at dx
        dx = dx_padded[:,padding:-padding, padding:-padding, :]

    return dx, dw, db

```

Finally, we test the backward pass using numerical gradient checking. This compares the gradients generated by our backward function, with a numerical approximation obtained by treating our forward function as a "black box". This gradient checking is a very important testing tool when building your own neural network components or back-propagation system!

```python
def eval_numerical_gradient_array(f, x, df, h=1e-5):
        """
        Evaluate a numeric gradient for a function that accepts a numpy
        array and returns a numpy array.
        """

        # QUESTION: Can you describe intuitively what this function is doing?

        grad = np.zeros_like(x)
        it = np.nditer(x, flags=['multi_index'], op_flags=['readwrite'])
        while not it.finished:
            ix = it.multi_index

            oldval = x[ix]
            x[ix] = oldval + h
            pos = f(x).copy()
            x[ix] = oldval - h
            neg = f(x).copy()
            x[ix] = oldval

            grad[ix] = np.sum((pos - neg) * df) / (2 * h)
            it.iternext()
        return grad

np.random.seed(231)

# Normally, backpropagation will have calculated a derivative of the final loss with respect to
# the output of our layer. Since we're testing our layer in isolation here, we'll just pretend
# and use a random value
dout = np.random.randn(2, 2, 2, 3)

dx_num = eval_numerical_gradient_array(lambda x: convolutional_forward(X, W, b, filter_size, output_depth, stride, padding), X, dout)
dw_num = eval_numerical_gradient_array(lambda w: convolutional_forward(X, W, b, filter_size, output_depth, stride, padding), W, dout)
db_num = eval_numerical_gradient_array(lambda b: convolutional_forward(X, W, b, filter_size, output_depth, stride, padding), b, dout)

out = convolutional_forward(X, W, b, filter_size, output_depth, stride, padding)
dx, dw, db = convolutional_backward(dout, X, W, b, filter_size, output_depth, stride, padding)

# Your errors should be around 1e-8'
print('Testing conv_backward_naive function')

dx_diff = relative_error(dx, dx_num)
if dx_diff < 1e-8:
    print 'dx check: PASSED'
else:
    print 'The difference of %s on dx is too high, try again!' % dx_diff

dw_diff = relative_error(dw, dw_num)
if dw_diff < 1e-8:
    print 'dw check: PASSED'
else:
    print 'The difference of %s on dw is too high, try again!' % dw_diff

db_diff = relative_error(db, db_num)
if db_diff < 1e-8:
    print 'db check: PASSED'
else:
    print 'The difference of %s on db is too high, try again!' % db_diff

```

    Testing conv_backward_naive function
    dx check: PASSED
    dw check: PASSED
    db check: PASSED

## (Max) Pooling Layers
The purpose of a pooling layer is to is to reduce the spatial size of the representation and therefore control the number of parameters in the network. A pooling layer has no trainable parameters itself. It applies some 2D aggegation operation (usually a MAX, but others like average may also be used) to regions of the input volume. This is done independently for each depth dimension of the input. For example, a 2x2 max pooling operation with a stride of 2, downsamples every depth slice of the input by 2 along both the width and height.

The output volume of a pooling layer alwyas has the same depth as the input volume. The width and height are calcualted as follows:
$(W−F)/S+1$ where W is the width/height of the

### Implementing the forward pass
We again implement this in Numpy:

```python
def max_pool_forward(X, pool_size, stride):
    batch_size, input_width, input_height, input_depth = X.shape

    # Calculate the output dimensions
    output_width = (input_width - pool_size)/stride + 1
    output_height = (input_height - pool_size)/stride + 1

    # Allocate the output array
    out = np.zeros((batch_size, output_width, output_height, input_depth))

    # Select the current block in the input that the filter will be applied to
    for w in range(output_width):
        for h in range(output_height):
            block_width_start = w*stride
            block_width_end = block_width_start+pool_size

            block_height_start = h*stride
            block_height_end = block_height_start + pool_size

            block = X[:, block_width_start:block_width_end, block_height_start:block_height_end, :]
            ## IMPLEMENT-ME CANDIDATE
            out[:,w,h,:] = np.max(block, axis=(1,2))

    return out

```

Now we can test the `max_pool_forward` function.

```python
### Hyperparameters
batch_size = 2
input_width = 4
input_height = 4
input_depth = 3
pool_size = 2
stride = 2
###

input_shape = (batch_size, input_width, input_height, input_depth)
X = np.linspace(-0.3, 0.4, num=np.prod(input_shape)).reshape(input_shape)

out = max_pool_forward(X, pool_size, stride)

correct_out = np.array([
        [[[-0.18947368, -0.18210526, -0.17473684],
            [-0.14526316, -0.13789474, -0.13052632]],
         [[-0.01263158, -0.00526316, 0.00210526],
            [0.03157895, 0.03894737, 0.04631579]]],
        [[[0.16421053, 0.17157895, 0.17894737],
            [0.20842105, 0.21578947, 0.22315789]],
         [[0.34105263, 0.34842105, 0.35578947],
            [0.38526316, 0.39263158, 0.4]]]])

# Compare the output. The difference should be less than 1e-6.
print('Testing max_pool_forward function:')
diff = relative_error(out, correct_out)
if diff < 1e-6:
    print 'PASSED'
else:
    print 'The difference of %s is too high, try again!' % diff
```

    Testing max_pool_forward function:
    PASSED

### The derivative of a max-pool layer

The max-pooling layer has no learnable parameters of its own, so the only derivative of concern is that of the output of the layer with respect to the input for the purpose of backpropagating the error through the layer. This is easy to calculate as it only requires that we recalculate (or remember) which value in each block was the maximum. Since each output depends only on one value in some FxF block of the input, the gradients of the max-pool layer will be sparse.

Let's implement the backward pass in Numpy:

```python
def max_pool_backward(dout, X, max_pool_output, pool_size, stride):
    batch_size, input_width, input_height, input_depth = X.shape

    # Calculate the output dimensions
    output_width = (input_width - pool_size)/stride + 1
    output_height = (input_height - pool_size)/stride + 1

    # QUESTION: What is the size of dx, the derivative with respect to x?
    # Allocate an array to hold the derivative
    dx = np.zeros_like(X)

    for w in range(output_width):
        for h in range(output_height):
            # Which block in the input did the value at the forward pass output come from?
            block_width_start = w*stride
            block_width_end = block_width_start+pool_size

            block_height_start = h*stride
            block_height_end = block_height_start + pool_size

            block = X[:, block_width_start:block_width_end, block_height_start:block_height_end, :]

            # What was the maximum value
            max_val = max_pool_output[:, w, h, :]

            # Which values in the input block resulted in the output?
            responsible_values = block == max_val[:, None, None, :]

            # Add the contribution of the current block to the gradient
            dx[:,block_width_start:block_width_end,block_height_start:block_height_end, :] += responsible_values * (dout[:,w,h,:])[:,None,None,:]

    return dx
```

And we again use numerical gradient checking to ensure that the backward function is correct:

```python
# Define a hypothetical derivative of the loss function with respect to the output of the max-pooling layer.
dout = np.random.randn(batch_size, pool_size, pool_size, input_depth)

dx_num = eval_numerical_gradient_array(lambda x: max_pool_forward(x, pool_size, stride), X, dout)
out = max_pool_forward(X, pool_size, stride)
dx = max_pool_backward(dout, X, out, pool_size, stride)

# Your error should be less than 1e-12
print('Testing max_pool_backward function:')
diff = relative_error(dx, dx_num)
if diff < 1e-12:
    print 'PASSED'
else:
    print 'The diff of %s is too large, try again!' % diff
```

    Testing max_pool_backward function:
    PASSED

## Optimisation - an exercise for later
Our implementations of convolutional and max-pool layers were based on loops, which are easy to understand, but are slow and inefficient compared to a vectorised implementation exploiting matrix multiplications. The vectorised form is how these layers are actually implemented in practice and are also required to make efficient use of GPUs in frameworks that support it, like TensorFlow. As an exercise, once you fully understand how the layers work, try to rewrite the code such that the convolution and max-pool operations are each implemented in a **single matrix multiplication**.

(HINT: Matlab has a function called "im2col" that rearranges blocks of an image into columns, you will need to achieve something similar using Numpy!)

# Building a 2-layer ConvNet in TensorFlow

Now that we understand the convolutional and max pool layers, let's switch back to TensorFlow and build a 2-layer ConvNet classifier that we can apply to MNIST. We reuse essentially the same classifier framework we used in Practical 2 as well as the training and plotting functions, but we have added support for 2 new forms of regularisation, **dropout** and **batch normalisation**. These are explained in more detail later.

```python
class BaseSoftmaxClassifier(object):
    def __init__(self, input_size, output_size):
        # Define the input placeholders. The "None" dimension means that the
        # placeholder can take any number of images as the batch size.
        self.x = tf.placeholder(tf.float32, [None, input_size])
        self.y = tf.placeholder(tf.float32, [None, output_size])

        # We add an additional input placeholder for Dropout regularisation
        self.keep_prob = tf.placeholder(tf.float32, name="keep_prob")

        # And one for bath norm regularisation
        self.is_training = tf.placeholder(tf.bool, name="is_training")

        self.input_size = input_size
        self.output_size = output_size

        # You should override these in your build_model() function.
        self.logits = None
        self.predictions = None
        self.loss = None

        self.build_model()

    def get_logits(self):
        return self.logits

    def build_model(self):
        # OVERRIDE THIS FOR YOUR PARTICULAR MODEL.
        raise NotImplementedError("Subclasses should implement this function!")

    def compute_loss(self):
        """All models share the same softmax cross-entropy loss."""
        assert self.logits is not None    # Ensure that logits has been created!
        data_loss = tf.reduce_mean(
                tf.nn.softmax_cross_entropy_with_logits(logits=self.logits, labels=self.y))

        return data_loss

    def accuracy(self):
        # Calculate accuracy.
        assert self.predictions is not None    # Ensure that pred has been created!
        correct_prediction = tf.equal(tf.argmax(self.predictions, 1), tf.argmax(self.y, 1))
        accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
        return accuracy
```

Lets also bring in the training and plotting routines we developed in Prac 2:

```python
def train_tf_model(tf_model,
                   session,    # The active session.
                   num_epochs,    # Max epochs/iterations to train for.
                   batch_size=100,    # Number of examples per batch.
                   keep_prob=1.0,    # (1. - dropout) probability, none by default.
                   optimizer_fn=None,    # TODO(sgouws): more correct to call this optimizer_obj
                   report_every=1, # Report training results every nr of epochs.
                   eval_every=1,    # Evaluate on validation data every nr of epochs.
                   stop_early=True,    # Use early stopping or not.
                   verbose=True):

    # Get the (symbolic) model input, output, loss and accuracy.
    x, y = tf_model.x, tf_model.y
    loss = tf_model.loss
    accuracy = tf_model.accuracy()

    # Compute the gradient of the loss with respect to the model parameters
    # and create an op that will perform one parameter update using the specific
    # optimizer's update rule in the direction of the gradients.
    if optimizer_fn is None:
        optimizer_fn = tf.train.AdamOptimizer(1e-4)

    # For batch normalisation: Ensure that the mean and variance tracking
    # variables get updated at each training step
    update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
    with tf.control_dependencies(update_ops):
        optimizer_step = optimizer_fn.minimize(loss)

    # Get the op which, when executed, will initialize the variables.
    init = tf.global_variables_initializer()
    # Actually initialize the variables (run the op).
    session.run(init)

    # Save the training loss and accuracies on training and validation data.
    train_costs = []
    train_accs = []
    val_costs = []
    val_accs = []

    mnist_train_data = mnist.train

    prev_c_eval = 1000000

    # Main training cycle.
    for epoch in range(num_epochs):

        avg_cost = 0.
        avg_acc = 0.
        total_batch = int(mnist.train.num_examples / batch_size)

        # Loop over all batches.
        for i in range(total_batch):
            batch_x, batch_y = mnist_train_data.next_batch(batch_size)

            # Run optimization op (backprop) and cost op (to get loss value),
            # and compute the accuracy of the model.
            feed_dict = {x: batch_x, y: batch_y, tf_model.keep_prob: keep_prob,
                                    tf_model.is_training: True}
            _, c, a = session.run(
                    [optimizer_step, loss, accuracy], feed_dict=feed_dict)

            # Compute average loss/accuracy
            avg_cost += c / total_batch
            avg_acc += a / total_batch

        train_costs.append((epoch, avg_cost))
        train_accs.append((epoch, avg_acc))

        # Display logs per epoch step
        if epoch % report_every == 0 and verbose:
            print "Epoch:", '%04d' % (epoch+1), "Training cost=", \
                        "{:.9f}".format(avg_cost)

        if epoch % eval_every == 0:
            val_x, val_y = mnist.validation.images, mnist.validation.labels

            feed_dict = {x : val_x, y : val_y, tf_model.keep_prob: 1.0,
                                    tf_model.is_training: False}
            c_eval, a_eval = session.run([loss, accuracy], feed_dict=feed_dict)

            if verbose:
                print "Epoch:", '%04d' % (epoch+1), "Validation acc=", \
                            "{:.9f}".format(a_eval)

            if c_eval >= prev_c_eval and stop_early:
                print "Validation loss stopped improving, stopping training early after %d epochs!" % (epoch + 1)
                break

            prev_c_eval = c_eval

            val_costs.append((epoch, c_eval))
            val_accs.append((epoch, a_eval))

    print "Optimization Finished!"
    return train_costs, train_accs, val_costs, val_accs
```

```python
# Helper functions to plot training progress.

def my_plot(list_of_tuples):
    """Take a list of (epoch, value) and split these into lists of
    epoch-only and value-only. Pass these to plot to make sure we
    line up the values at the correct time-steps.
    """
    plt.plot(*zip(*list_of_tuples))

def plot_multi(values_lst, labels_lst, y_label, x_label='epoch'):
    # Plot multiple curves.
    assert len(values_lst) == len(labels_lst)
    plt.subplot(2, 1, 2)

    for v in values_lst:
        my_plot(v)
    plt.legend(labels_lst, loc='upper left')

    plt.xlabel(x_label)
    plt.ylabel(y_label)
    plt.show()
```

Now define some helper functions to build a convolutional layer and a linear layer (this is mostly the same as the previous practical, but we use slightly different weight and bias initializations which seem to work better with ConvNets on MNIST). In terms of regularisation, we use **dropout** rather than the L2 regularisation from the previous practical.

## Dropout
Dropout is a neural-network regularisation technique that is applied during model training. At each training step, a proportion `(1-keep_prob)` of neurons the network are "dropped out" (their inputs and outputs are set to 0, effectively ignoring their contribution) while the remaining `keep_prob` fraction are "let through" (**Nit**: they're actually rescaled by `1/keep_prob` to ensure that the variance of the pre-activations at the next layer remains unchanged). This can be interpreted as there being actually $2^n$ different network architectures (where n is the number of neurons) while only one is being trained at each training step. At test time, we use the full network, where each neuron's contribution is weighted by `keep_prob`. This is effectively the average of all the network possibilities and therefore dropout can also be thought of as an ensemble technique.

In our ConvNet architecture, the majority of neurons occur in the fully connected layer between the convolutional layers and the output. It is therefore this fully connected layer that we are most concerned about overfitting and this is where we apply dropout.

```python
def _convolutional_layer(inputs, filter_size, output_depth):
    """Build a convolutional layer with `output_depth` square
    filters, each of size `filter_size` x `filter_size`."""

    input_features = inputs.shape[3]

    weights = tf.get_variable(
            "conv_weights",
            [filter_size, filter_size, input_features, output_depth],
            dtype=tf.float32,
            initializer=tf.truncated_normal_initializer(stddev=0.1))

    ## IMPLEMENT-ME CANDIDATE
    conv = tf.nn.conv2d(inputs, weights, strides=[1, 1, 1, 1], padding='SAME')
    return conv

```

```python
def _dense_linear_layer(inputs, layer_name, input_size, output_size, weights_initializer):
    """
    Builds a layer that takes a batch of inputs of size `input_size` and returns
    a batch of outputs of size `output_size`.

    Args:
        inputs: A `Tensor` of shape [batch_size, input_size].
        layer_name: A string representing the name of the layer.
        input_size: The size of the inputs
        output_size: The size of the outputs

    Returns:
        out, weights: tuple of layer outputs and weights.

    """
    # Name scopes allow us to logically group together related variables.
    # Setting reuse=False avoids accidental reuse of variables between different runs.
    with tf.variable_scope(layer_name, reuse=False):
        # Create the weights for the layer
        layer_weights = tf.get_variable("weights",
                                        shape=[input_size, output_size],
                                        dtype=tf.float32,
                                        initializer=weights_initializer)
        # Create the biases for the layer
        layer_bias = tf.get_variable("biases",
                                     shape=[output_size],
                                     dtype=tf.float32,
                                     initializer=tf.constant_initializer(0.1))

        outputs = tf.matmul(inputs, layer_weights) + layer_bias

    return outputs
```

Now build the ConvNetClassifier, we make the number of convolutional layers and the filter sizes parameters so that you can easily experiment with different variations.

```python
class ConvNetClassifier(BaseSoftmaxClassifier):
    def __init__(self,
                 input_size,     # The size of the input
                 output_size,    # The size of the output
                 filter_sizes=[], # The number of filters to use per convolutional layer
                 output_depths=[], # The number of features to output per convolutional layer
                 hidden_linear_size=512,    # The size of the hidden linear layer
                 use_batch_norm=False,     # Flag indicating whether or not to use batch normalisation
                 linear_weights_initializer=tf.truncated_normal_initializer(stddev=0.1)):

        assert len(filter_sizes) == len(output_depths)

        self.filter_sizes = filter_sizes
        self.output_depths = output_depths
        self.linear_weights_initializer = linear_weights_initializer
        self.use_batch_norm = use_batch_norm
        self.hidden_linear_size = hidden_linear_size

        super(ConvNetClassifier, self).__init__(input_size, output_size)

    def build_model(self):
        # Architecture: INPUT - {CONV - RELU - POOL}*N - FC

        # Reshape the input to [batch_size, width, height, input_depth]
        conv_input = tf.reshape(self.x, [-1, 28, 28, 1])

        prev_inputs = conv_input

        # Create the CONV-RELU-POOL layers:
        for layer_number, (layer_filter_size, layer_features) in enumerate(
                zip(self.filter_sizes, self.output_depths)):
            with tf.variable_scope("layer_{}".format(layer_number), reuse=False):
                # Create the convolution:
                conv = _convolutional_layer(prev_inputs, layer_filter_size, layer_features)

                # Apply batch normalisation, if required
                if self.use_batch_norm:
                    conv = tf.contrib.layers.batch_norm(conv, center=True, scale=True,
                                                        is_training=self.is_training)

                # Apply the RELU activation with a bias
                bias = tf.get_variable("bias", [layer_features], dtype=tf.float32, initializer=tf.constant_initializer(0.1))
                relu = tf.nn.relu(conv + bias)

                # Apply max-pooling using patch-sizes of 2x2
                pool = tf.nn.max_pool(relu, ksize=[1, 2, 2, 1],
                                                            strides=[1, 2, 2, 1],
                                                            padding='SAME')

                # QUESTION: What is the shape of the pool tensor?
                # ANSWER: ...

                prev_inputs = pool

        # QUESTION: What is the shape of prev_inputs now?

        # We need to flatten the last (non-batch) dimensions of the convolutional
        # output in order to pass it to a fully-connected layer:
        flattened = tf.contrib.layers.flatten(prev_inputs)

        # Create the fully-connected (linear) layer that maps the flattened inputs
        # to `hidden_linear_size` hidden outputs
        flat_size = flattened.shape[1]
        fully_connected = _dense_linear_layer(
                flattened, "fully_connected", flat_size, self.hidden_linear_size, self.linear_weights_initializer)

        # Apply batch normalisation, if required
        if self.use_batch_norm:
            fully_connected = tf.contrib.layers.batch_norm(
                    fully_connected, center=True, scale=True, is_training=self.is_training)

        fc_relu = tf.nn.relu(fully_connected)

        fc_drop = tf.nn.dropout(fc_relu, self.keep_prob)

        # Now we map the `hidden_linear_size` outputs to the `output_size` logits, one for each possible digit class
        logits = _dense_linear_layer(
                fc_drop, "logits", self.hidden_linear_size, self.output_size, self.linear_weights_initializer)

        self.logits = logits
        self.predictions = tf.nn.softmax(self.logits)
        self.loss = self.compute_loss()

```

Finally a function that wraps up the training and evaluation of the model (the same as prac 2)

```python
def build_train_eval_and_plot(build_params, train_params, verbose=True):
    tf.reset_default_graph()
    m = ConvNetClassifier(**build_params)

    with tf.Session() as sess:
        # Train model on the MNIST dataset.

        train_losses, train_accs, val_losses, val_accs = train_tf_model(
                m,
                sess,
                verbose=verbose,
                **train_params)

        # Now evaluate it on the test set:

        accuracy_op = m.accuracy()    # Get the symbolic accuracy operation
        # Calculate the accuracy using the test images and labels.
        accuracy = accuracy_op.eval({m.x: mnist.test.images,
                                     m.y: mnist.test.labels,
                                     m.keep_prob: 1.0,
                                     m.is_training: False})

        if verbose:
            print "Accuracy on test set:", accuracy
            # Plot losses and accuracies.
            plot_multi([train_losses, val_losses], ['train', 'val'], 'loss', 'epoch')
            plot_multi([train_accs, val_accs], ['train', 'val'], 'accuracy', 'epoch')

        ret = {'train_losses': train_losses, 'train_accs' : train_accs,
               'val_losses' : val_losses, 'val_accs' : val_accs,
               'test_acc' : accuracy}

        # Evaluate the final convolutional weights
        conv_variables = [v for v in tf.trainable_variables() if "conv_weights" in v.name]
        conv_weights = sess.run(conv_variables)

        return m, ret, conv_weights
```

Now train and evaluate the ConvNet model on MNIST.

**NOTE**: Hopefully you answered the question in the first section about the tradeoffs a ConvNet makes with "extra computation"! Unfortuntaly the VMs we're using are pretty low-powered, so we will train a very small ConvNet just to check that it works. Once you've got this small ConvNet working, chat to a tutor to get access to a machine with a GPU and run with the following configuration to get to the promised 99%+ accuracy on MNIST!

```
model_params = {
        'input_size': 784,
        'output_size': 10,
        'filter_sizes': [5, 5],
        'output_depths': [32, 64],
        'hidden_linear_size': 1024,
        'use_batch_norm': False
}

training_params = {
        'keep_prob': 0.5,
        'num_epochs': 20,
        'batch_size': 50,
        'stop_early': False,
}
```

```python
%%time

# Create a ConvNet classifier with 2 CONV-RELU-POOL layers, with 7x7 filters,
# 32 and 64 output features and a hidden linear layer size of 512.
model_params = {
        'input_size': 784,
        'output_size': 10,
        'filter_sizes': [5],
        'output_depths': [4],
        'hidden_linear_size': 128,
        'use_batch_norm': False
}

training_params = {
        'keep_prob': 0.5,
        'num_epochs': 5,
        'batch_size': 50,
        'stop_early': False,
}

trained_model, training_results, conv_weights = build_train_eval_and_plot(
        model_params,
        training_params,
        verbose=True
)
```

The ConvNet classifier takes quite a long time to train, but gives a very respectable test accuracy of over **99%**!

## What has the network learned?
Remember that a filter in a convolutional layer is used to multiply blocks in the input volume. Let's plot the weights of the first layer of the trained model. Darker pixels indicate that the particular filter reacts more strongly to those regions of the input blocks. Notice how each filter has learned to react differently to different patterns in the input. It's tricky to see in our tiny filters, but those in lower layers of ConvNets, particularly when applied to natural images, often function as simple Gabor filters or edge detectors, while filters in higher layers often react to more abstract shapes and concepts.

```python
weights = conv_weights[0]
_, _, _, out_depth = weights.shape
grid_size = int(out_depth**0.5)

fig = plt.figure()

i = 1
for r in range(grid_size):
    for c in range(grid_size):
        ax = fig.add_subplot(grid_size, grid_size, i)
        ax.imshow(weights[:, :, 0, r*grid_size+c], cmap="Greys")
        i += 1
plt.show()
```

# Aside: The Effect of Random Initialization - RUN THIS ON A GPU INSTANCE ONLY!
Initialization of model parameters matters! Here is a ConvNet with different, but seemingly sensible, initialization of the weights in the linear layer. Running this gives significantly worse results. Judging by the accuracy plot, it's possible that training this model long enough will get it to a simliar level as before, but it will take much longer. This shows that initialization of model parameters is a an important consideration, especially as models become more complex. In practice, there are a number of different initialization schemes to consider. In particular, [Xavier](http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization) tends to work well with ConvNets and is worth considering. We won't go into any details in this practical though.

```python
%%time

# Create a ConvNet classifier with 2 CONV-RELU-POOL layers, with filter sizes of
# 5 and 5 and 32 and 64 output features.
model_params = {
        'input_size': 784,
        'output_size': 10,
        'filter_sizes': [5, 5],
        'output_depths': [32, 64],
        'hidden_linear_size': 1024,
        'use_batch_norm': False
        'linear_weights_initializer': tf.random_normal_initializer()
}

training_params = {
        'keep_prob': 0.5,
        'num_epochs': 5,
        'batch_size': 50,
        'stop_early': False,
}

trained_model, training_results = build_train_eval_and_plot(
        model_params,
        training_params,
        verbose=True
)
```

## Batch Normalisation
Batch normalisation (batch norm) is a more recent (2015) and arguably more powerful normalisation technique than dropout. It is based on the observation that machine learning models often perform better and train faster when their inputs are normalised to have 0 mean and unit variance. In multi-layered deep neural networks, the output of one layer becomes the input to the next. The insight behind batch norm is that each of these layer inputs can also be normalised. Batch norm has been shown to have numerous benefits including:
* Networks tend to train faster
* Allows higher learning rates to be used (further improving training speed).
* Reduced sensitivity to weight initialisation.
* Makes certain activation functions feasible in deep networks (When inputs have very large (absolute) expected values, certain activation functions become saturated (For example, the output of sigmoid is always to 1 for large inputs). Relu activations can also "die out" when the expected value of the input is a large negative value (why?). This results in wasted computation as these neurons become uninformative. Normalising the inputs to have 0 mean keeps these activation functions in the "sensible" parts of their domains.)

### How does it work?
To normalise some inputs X, ideally we would like to set
$\hat X = \frac{X - E[X]}{\sqrt{VAR[X]}}$
but this requires knowledge of the population mean and variance statistics, which we don't know, at least during training. We therefore use the **sample mean** and **sample variance** of each batch encountered during training as unbiased estimates of these statistics. During testing, we use statistics gathered throughout training as better estimates of the population statistics. In addition to this, we would like the model to have some flexibility over the extent to which batch norm is applied, and this flexibility should be learned! In order to do this, we introduce two new trainable parameters, $\gamma$ and $\beta$ for each layer that batch norm is applied to. Suppose we have a batch of inputs to a layer, $B={x_1,...,x_m}$, we normalise these as follows:

$\mu_B = \frac{1}{m} \sum_{i=1}^{m} x_i$     (Batch mean)

${\sigma_B}^2 = \frac{1}{m} \sum_{i=1}^{m} (x_i - \mu_B)^2$     (Batch variance)

$\hat x_i= \frac{x_i - \mu_B}{\sqrt{{\sigma_B}^2}}$     (Normalised)

$y_i = \gamma \hat x_i + \beta$     (Scale and shift)

At test time, we normalise using the mean and variance computed over the entire training set:

$E[x] = E_B[\mu_B]$

$VAR[x] = \frac{m}{m-1}E_B[{\sigma_B}^2]$

$\hat x = \frac{x - E[x]}{\sqrt{VAR[x]}}$

$y = \gamma \hat x + \beta$

### Implementation Details
Tracking the mean and variance over the training set can become a little fiddly. Many implementations also use a *moving average* of the batch mean and variance as estimates of the population mean and variance for use during testing. Luckily, TensorFlow provides batch norm out of the box in the form of the `tf.contrib.layers.batch_norm` function.

Since the behaviour of batch norm changes during training and testing, we need to pass a placeholder input to the function that indicates which phase we are in. Furthermore, the batch norm function uses variable updates to track the moving average mean and variance. These values are not used during training and so TensorFlow's graph execution logic will not naturally run these updates when you run a training step. In order to get around this, the `batch_norm` function adds these update ops to a *graph collection* that we can access in our training function. The following code, which you will see in the `train_tf_model` function retrieves these ops and then adds a *control dependency* to the optimiser step. This effectively tells TensorFlow that the update_ops must be run before the optimizer_step can be run, ensuring that the estimates are updated whenever we do a training step.

```
update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(update_ops):
    optimizer_step = optimizer_fn.minimize(loss)
```

Further choices to consider when using batch norm are where to apply it (some apply it immediately before each activation function, some *after* the activation function), whether to apply it to all layers and whether or not to *share* the gamma and beta parameters over all layers or have separate values for each layer. .

Have a look at the ConvNetClassifer class above to see what choices were made, try changing these and see what results you get! (See the [TensorFlow documentation](https://www.tensorflow.org/api_docs/python/tf/contrib/layers/batch_norm) for a list of even more parameters you can experiment with)

Now, finally, let's switch batch norm on and see how our ConvNetClassifier performs. (Note: we shouldn't expect it to necessarily perform better than dropout as we are already close to the limits of how well we can classify MNIST with our relatively small ConvNet!)

```python
%%time

## UNFORTUNATELY THIS WILL ALSO NOT WORK ON THE VM's, YOU'LL NEED TO GET A GPU INSTANCE TO RUN THIS!

# Create a ConvNet classifier with 2 CONV-RELU-POOL layers, with filter sizes of
# 5 and 5 and 32 and 64 output features.
model_params = {
        'input_size': 784,
        'output_size': 10,
        'filter_sizes': [5, 5],
        'output_depths': [32, 64],
        'use_batch_norm': True,
}

training_params = {
        'keep_prob': 1.0,    # Switch off dropout
        'num_epochs': 15,
        'batch_size': 50,
        'stop_early': False,
}

trained_model, training_results = build_train_eval_and_plot(
        model_params,
        training_params,
        verbose=True
)

# QUESTION: Try experimenting with different archictures and hyperparameters and
# see how well you can classify MNIST digits!
```

# NB: Before you go (5min)

Pair up with someone else and go through the questions in "Learning Objectives" at the top. Take turns explaining each of these to each other, and be sure to ask the tutors if you're both unsure!

# Additional resources

* Chip Huyen's wonderful script: https://github.com/chiphuyen/stanford-tensorflow-tutorials/blob/master/examples/07_convnet_mnist.py
* NVidia's ConvNet Tutorial: https://github.com/alrojo/tensorflow-tutorial/blob/master/lab2_CNN/lab2_CNN.ipynb
* https://github.com/DeepLearningDTU/nvidia_deep_learning_summercamp_2016/blob/master/lab2/lab2_CNN.ipynb

# Feedback

Please send any bugs and comments to dli-practicals@googlegroups.com.


## Linear Discriminant Functions and Support Vector Machines
### Siddheswar Chandrasekhar

### Part 1
##### We train a Support Vector classifier using the MNIST training dataset and report performance using MNIST testing dataset

```python
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# @Author: Siddheswar C
# @Email: innocentdevil.sid007@gmail.com

import tensorflow as tf
import numpy as np
from sklearn import svm, metrics
from sklearn.metrics import confusion_matrix
from matplotlib import pyplot as plt
```

```python
class ConfusionMatrix:

    # Calculate Confusion Matrix
    @staticmethod
    def calculate(y, yhat):
        return confusion_matrix(y, yhat)

    # Plot Confusion Matrix
    @staticmethod
    def plot(title, confusion, matrix_dimension):
        plt.clf()
        plt.xlabel('Actual Target (y)')
        plt.ylabel('Predicted Target (yhat)')
        plt.grid(False)
        plt.xticks(np.arange(matrix_dimension))
        plt.yticks(np.arange(matrix_dimension))
        plt.title(title)
        plt.imshow(confusion, cmap=plt.cm.jet, interpolation='nearest')

        for i, cas in enumerate(confusion):
            for j, count in enumerate(cas):
                if count > 0:
                    xoff = .07 * len(str(count))
                    plt.text(j - xoff, i + .2, int(count), fontsize=9, color='white')
```

```python
def support_vector_machine(training_data, training_label, test_data, test_label):
    confusion_mat = ConfusionMatrix()

    print('Performing SVM')

    # Classify as SVM
    classifier = svm.SVC(C=1.0, kernel='linear', gamma='scale')

    # Perform Training
    classifier.fit(training_data, training_label)

    # Predict
    yhat = classifier.predict(test_data)

    accuracy = metrics.accuracy_score(test_label, yhat)
    confusion = confusion_mat.calculate(test_label, yhat)
    confusion_mat.plot('SVM on MNIST dataset', confusion, confusion.shape[0])

    return accuracy, confusion
```

```python
# Read and process the dataset
print('Loading MNIST data')
mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

IMAGE_SIZE = x_train.shape[-1]

x_train = x_train.reshape((-1, IMAGE_SIZE*IMAGE_SIZE))
x_test = x_test.reshape((-1, IMAGE_SIZE*IMAGE_SIZE))

accuracy, confusion = support_vector_machine(x_train, y_train, x_test, y_test)
print('Accuracy: ' + str(accuracy*100) + '%')
```

### Part 2
##### We identify the Lagrange dual problem given a primal problem

\begin{align*}
\text{minimize  } \frac{1}{2}w^T.w+C \sum_{n=1}^{N} \xi_{i}
\\
\text{subject to } y_i.(w^T.x_i) \geq {1-\xi_{i}} \text{ and } \xi_i \geq 0 \text{ for i }=1,...,N
\end{align*}
The margin is :
 $\gamma = \frac{1}{\sqrt{w^T.w +C \sum_{i=1}^{N} \xi_i }}$

#### Lagrange function :

\begin{align}
{\mathcal{L}} = \frac{1}{2} {\overrightarrow{w}}^T\overrightarrow{w} + C \sum_{i=1}^N\xi_i + \Sigma_i\alpha_i(1-y_i.w^Tx_i -\xi_i) - \sum_{i=1}^N \beta_i \xi_i
\end{align}

Lagrange multipliers $\alpha \geq 0$, $ \beta \geq 0$,

Inequality constraints $1 - y_i(w^Tx_i) - \xi_i \leq 0$ and $\xi_i \geq 0$ for $i=1,...,N$

#### Claim:

$\underbrace{\underset{\alpha,\beta}{max} \underset{w,\xi}{min} \mathcal{L}}_\text{dual solution} \leq \underbrace{\underset{w,\xi}{min} \underset{\alpha,\beta}{max} \mathcal{L}}_\text{primal solution}$

\begin{align}
{\mathcal{L}} = \frac{1}{2} {\overrightarrow{w}}^T\overrightarrow{w} + C \sum_{i=1}^N\xi_i + \Sigma_i\alpha_i(1-y_i.w^Tx_i -\xi_i) - \sum_{i=1}^N \beta_i \xi_i
\end{align}

\begin{align}
\\
  \frac{\partial_{\mathcal{L}}}{\partial_w} = w - \sum_i \alpha_i y_i x_i = 0 \implies w = \sum_i \alpha_i y_i \overrightarrow{x_i}
  \\
  \frac{\partial_{\mathcal{L}}}{\partial_\xi} = 0 \implies C - \alpha_i - \beta_i = 0
  \\
  \implies 0 \leq \alpha_i \leq C \text{ and } \beta_i \geq 0
\end{align}

\begin{align}
\text {Substituting } w = \sum_i \alpha_i y_i \overrightarrow{x_i}, C=\alpha_i + \beta_i
\text { into Lagrange function, we get the dual problem of maximizing: }
\end{align}

\begin{align}
{\mathcal{L}} = \frac{1}{2} w^T \sum_i \alpha_i y_i \overrightarrow{x_i} + (\alpha_i + \beta_i) \sum_{i=1}^N\xi_i + \Sigma_i\alpha_i(1-y_i.w^Tx_i -\xi_i) - \sum_{i=1}^N \beta_i \xi_i
\\= \frac{1}{2} w^T \sum_i \alpha_i y_i \overrightarrow{x_i} + \alpha_i\sum_{i=1}^N\xi_i + \beta_i \sum_{i=1}^N\xi_i +\sum_i \alpha_i - w^T \sum_i \alpha_i y_i \overrightarrow{x_i} - \sum_i \alpha_i \xi_i - \sum_{i=1}^N \beta_i \xi_i
\end{align}

\begin{align}
\\=\sum_i \alpha_i- \frac{1}{2} \sum_{i,j} \alpha_i \alpha_j y_i y_j (\overrightarrow{x_i}^T x_j )
\end{align}

The primal margin is :
 $\gamma = \frac{1}{\sqrt{w^T.w +C \sum_{i=1}^{N} \xi_i }}$

The dual margin is :
$\gamma = \frac{1}{\sqrt{\alpha_i \alpha_j y_i y_j (x_i^T x_j )}}$

#### Benefits of maximizing the margin

Maximizing the margin helps avoid overfitting. We may have found a hyperplane that completely seperates the classes in our training set, but a situation may arise that when a new data comes in, it is classifies on the wrong side of the hyperplane. This would happen when the hyperplane is oriented too close to the training data points.
A large margin would help generalize the model and make better predictions on new datasets.

#### Characterize the support vectors

* Lying on the margin boundaries w^T x + b = −1 and w^T x + b = +1 (ξn = 0)

* Lying within the margin region (0 < ξn < 1) but still on the correct side

* Lying on the wrong side of the hyperplane (ξn ≥ 1)

#### Benefits of solving the dual problem instead of the primal problem

* It's easier to optimize in dual than prime when the number of data points is less than the number of dimensions.

* Irrespective of how many dimensions there are, dual representation only has as many parameters as the number of data points

* When the number of data points grow, this advantage turns into a liability.

* Regularizing the "sparse support vector" associated with the dual hypothesis is sometimes more intuitive than regularizing the vector of regression coefficients

### Part 3

\begin{equation}
\underset{w_{m}\in H,\xi \in R^l}{min} \frac{1}{2} \sum_{m=1}^{k} w_{m}^{T} w_{m} + C \sum_{i=1}^{l}  \xi_{i}
\end{equation}

\begin{align}
{\text{subject to,   }}  w_{y_i}^T \varphi(x_i) - w_{t}^{T}\varphi(x_i) \geq 1 - \delta_{y_i,t} - \xi_i
\end{align}
\begin{align}
i = 1,...,l, t \in {1,...,k},
\end{align}

The resulting Decision function is:
\begin{align}
argmax_m f_m(x) = argmax_m w^{T}_{m} \varphi(x)
\end{align}