# MuSLIT : Multi-band Sparse Linear Inversion Tool

Preliminary tests for an adaptation of principle of SLIT in the formulation of non-negative matrix factorization (NMF).

We first make use of the following libraries, at least for these tests :
- **PySAP** for efficient wavelet transforms
- **proxmin** for NMF optimizer (Block-SDMM)

_some $\LaTeX$ definitions_
$\newcommand{\mat}[1]{\mathsf{#1}}$
$\newcommand{\matind}[2]{\mathsf{#1}_{#2}}$
$\newcommand{\RoneD}[1]{\mathbb{R}^{#1}}$
$\newcommand{\RtwoD}[2]{\mathbb{R}^{#1\times#2}}$
$\newcommand{\RthreeD}[3]{\mathbb{R}^{#1\times#2\time#3}}$
$\newcommand{\Nps}{N_\mathrm{p,\,s}}$
$\newcommand{\Npi}{N_\mathrm{p,\,i}}$
$\newcommand{\Np}{N_\mathrm{p}}$
$\newcommand{\Nb}{N_\mathrm{b}}$
$\newcommand{\Ns}{N_\mathrm{s}}$
$\newcommand{\ie}{\emph{i.e.}}$
$\newcommand{\atiter}[1]{^{(#1)}}$
$\newcommand{\Niter}{N_{\mathrm{iter}}}$
$\newcommand{\Nsubiter}{N_{\mathrm{subiter}}}$
$\newcommand{\FISTA}[1]{\mathrm{FISTA}(#1)}$
$\newcommand{\normzero}[1]{\left\lVert #1 \right\rVert_0}$
$\newcommand{\normone}[1]{\left\lVert #1 \right\rVert_1}$
$\newcommand{\normtwo}[1]{\left\lVert #1 \right\rVert_2}$
$\newcommand{\fidelity}{\mathcal{D} \left( \mat{Y}\, |\, \mat{A_S},\, \mat{A_G},\, \mat{S},\, \mat{G} \right)}$

$\renewcommand{\algorithmicrequire}{\textbf{Input:}}$
$\renewcommand{\algorithmicensure}{\textbf{Output:}}$

$\DeclareMathOperator*{\argmin}{arg\,min}$

## Problem statement (ignoring PSF for now)

The problem we want to solve can be written as the following minimizatiton problem, with forward model $\mat{Y}=\mat{A\,S}$.

- Synthesis formulation :
\begin{align}
        \argmin_{\mat{A},\,\alpha_\mat{G},\,\alpha_\mat{S}}&\ \normtwo{\mat{Y}-\mat{A}
        \begin{bmatrix}
            \left(\mat{F_\kappa}\Phi^\top\alpha_\mat{S}\right)^\top \\
            \left(\Phi^\top\alpha_\mat{G}\right)^\top
        \end{bmatrix}
        }^2 + i_{\cdot\geq0}\!\left(\mat{A}\right) + i_{\cdot\geq0}\!\left(\Phi^\top\alpha_\mat{S}\right) + i_{\cdot\geq0}\!\left(\Phi^\top\alpha_\mat{G}\right) + g_1\!\left(\mat{A}\right) + \lambda_\mat{S}\normzero{\alpha_\mat{S}} + \lambda_\mat{G}\normzero{\alpha_\mat{G}}
\end{align}

- Analysis formulation (preferred, but no prox for $\normzero{\Phi\mat{X}}$ !!) :
\begin{align}
        \label{eq:final_eq_analysis}
        \argmin_{\mat{A},\,\mat{G},\,\mat{S}}&\ \normtwo{\mat{Y}-\mat{A}
        \begin{bmatrix}
            \left(\mat{F_\kappa}\mat{S}\right)^\top \\
            \mat{G}^\top
        \end{bmatrix}
        }^2 + i_{\cdot\geq0}\!\left(\mat{A}\right) + i_{\cdot\geq0}\!\left(\mat{S}\right) + i_{\cdot\geq0}\!\left(\mat{G}\right) + g_1\!\left(\mat{A}\right) + \lambda_\mat{S}\normzero{\alpha_\mat{S}} + \lambda_\mat{G}\normzero{\alpha_\mat{G}}
\end{align}

For implementation practicity, the constraint $i_{\cdot\geq0}\!\left(\Phi^\top\alpha_\mat{S}\right)$ may be applied as $i_{\cdot\geq0}\!\left(\mat{F_\kappa}\Phi^\top\alpha_\mat{S}\right)$. Both formulations are equivalent, since $\mat{F_\kappa}$ is non-negative.

#### Adding the PSF convolution

The forward model becomes $\mat{Y}=\mat{H}*\mat{A\,X}$, with $\mat{H}*\cdot$ being the PSF convolution in each band :
\begin{align}
        \mat{H} =
        \begin{bmatrix}
            \mat{H_1} & & \\
             & \ddots & \\
             & & \mat{H}_{\Nb}
        \end{bmatrix}
\end{align}

```python
import os
import sys
import PIL
import copy
import numpy as np
import pickle as pkl
import matplotlib.pyplot as plt
import astropy.io.fits as pyfits

from pylensmodels.mass.spemd import SPEMD_glee
from pylensmodels.light.gaussian import Gaussian, GaussianElliptical
from pylensmodels.utils import coordinates

import MuSLIT.utils.plot as plot_utils
from MuSLIT.lensing import planes
from MuSLIT.simulation import quick_sim

lin = lambda x: x
log = lambda x: np.log10(x)
```

# 1. Generate a mock lens
Lens mass model : SPEMD (using GLEE parameter conventions)
Lens light model : Spherical gaussian
Source light model : Elliptical gaussian

```python
sim_lens_multiband = quick_sim.simple_multiband_lens(sigma_noise=0.05)
```

    Deflection angles have been provided

```python
plot_utils.plot_rgb_bands(sim_lens_multiband)
```

# 2. Reconstruct the source using _MuSLIT_

```python

```


# Inference example for trained 2D U-Net model on BraTS.
Takes a trained model and performs inference on a few validation examples.

```python
!which python
!conda list | grep tensorflow
```

    /home/ubuntu/anaconda3/bin/python
    tensorflow                1.12.0                    <pip>

```python
%matplotlib inline
import os
import psutil
import numpy as np
import tensorflow as tf
import keras as K
import h5py
import time
import matplotlib.pyplot as plt # :metal:

print ("We are using Tensorflow version", tf.__version__,\
       "with Intel(R) MKL", "enabled" if tf.pywrap_tensorflow.IsMklEnabled() else "disabled",)
```

    We are using Tensorflow version 1.12.0 with Intel(R) MKL disabled

    Using TensorFlow backend.

Setup settings

```python
data_path = os.path.join("../data/144x144/")
data_filename = "Task01_BrainTumour.h5"

output_path = os.path.join("./output/")
inference_filename = "unet_model_for_decathlon.hdf5"

data_fn = os.path.join(data_path, data_filename)
model_fn = os.path.join(output_path, inference_filename)
```

The Sørensen–Dice coefficient is a statistic used for comparing the similarity of two samples. Given two sets, X and Y, it is defined as

\begin{equation}
DSC = \frac{2|X\cap Y|}{|X|+|Y|}
\end{equation}

```python
def calc_dice(y_true, y_pred, smooth=1.):
    numerator = 2.0 * np.sum(y_true * y_pred) + smooth
    denominator = np.sum(y_true) + np.sum(y_pred) + smooth
    coef = numerator / denominator
    return coef

def dice_coef(y_true, y_pred, axis=(1, 2), smooth=1.):
    """
    Sorenson (Soft) Dice
    \frac{  2 \times \left | T \right | \cap \left | P \right |}{ \left | T \right | +  \left | P \right |  }
    where T is ground truth mask and P is the prediction mask
    """
    intersection = tf.reduce_sum(y_true * y_pred, axis=axis)
    union = tf.reduce_sum(y_true + y_pred, axis=axis)
    numerator = tf.constant(2.) * intersection + smooth
    denominator = union + smooth
    coef = numerator / denominator

    return tf.reduce_mean(coef)

def dice_coef_loss(target, prediction, axis=(1, 2), smooth=1.):
    """
    Sorenson (Soft) Dice loss
    Using -log(Dice) as the loss since it is better behaved.
    Also, the log allows avoidance of the division which
    can help prevent underflow when the numbers are very small.
    """
    intersection = tf.reduce_sum(prediction * target, axis=axis)
    p = tf.reduce_sum(prediction, axis=axis)
    t = tf.reduce_sum(target, axis=axis)
    numerator = tf.reduce_mean(intersection + smooth)
    denominator = tf.reduce_mean(t + p + smooth)
    dice_loss = -tf.log(2.*numerator) + tf.log(denominator)

    return dice_loss

def combined_dice_ce_loss(y_true, y_pred, axis=(1, 2), smooth=1.,
                          weight=0.9):
    """
    Combined Dice and Binary Cross Entropy Loss
    """
    return weight*dice_coef_loss(y_true, y_pred, axis, smooth) + \
        (1-weight)*K.losses.binary_crossentropy(y_true, y_pred)
```

Calculate the Dice and plot the predicted masks for an input image (`img_no`).

```python
def plot_results(model, imgs_validation, msks_validation, img_no, png_directory):
    img = imgs_validation[[img_no], ]
    msk = msks_validation[[img_no], ]

    start_time = time.time()
    pred_mask = model.predict(img, verbose=1, steps=None)
    print ("Time for prediction: ", '%.0f'%((time.time()-start_time)*1000),"ms")

    dice_score = calc_dice(pred_mask, msk)

    print("Dice score for Image #{} = {:.4f}".format(img_no,
                                                     dice_score))

    plt.figure(figsize=(15, 15))
    plt.subplot(1, 3, 1)
    plt.imshow(img[0, :, :, 0], cmap="bone", origin="lower")
    plt.axis("off")
    plt.title("MRI Input", fontsize=20)
    plt.subplot(1, 3, 2)
    plt.imshow(msk[0, :, :, 0], origin="lower")
    plt.axis("off")
    plt.title("Ground truth", fontsize=20)
    plt.subplot(1, 3, 3)
    plt.imshow(pred_mask[0, :, :, 0], origin="lower")
    plt.axis("off")
    plt.title("Prediction\nDice = {:.4f}".format(dice_score), fontsize=20)

    plt.tight_layout()

    png_name = os.path.join(png_directory, "pred{}.png".format(img_no))
    plt.savefig(png_name, bbox_inches="tight", pad_inches=0)
    print("Saved png file to {}".format(png_name))
```

Load the data, the model and create an output directory for the images

```python
# Load data
df = h5py.File(data_fn, "r")
imgs_validation = df["imgs_validation"]
msks_validation = df["msks_validation"]

# Load model
model = K.models.load_model(model_fn, custom_objects={
"combined_dice_ce_loss": combined_dice_ce_loss,
"dice_coef_loss": dice_coef_loss,
"dice_coef": dice_coef})

# Create output directory for images
png_directory = "inference_examples"
if not os.path.exists(png_directory):
    os.makedirs(png_directory)

# Plot some results
# The plots will be saved to the png_directory
# Just picking some random samples.

```

```python
indicies_validation = [40,61,400,1100,4385,5566,5673,6433,7864,8899,9003,9722,10591]
indicies_validation = [40]

for idx in indicies_validation:
    plot_results(model, imgs_validation, msks_validation, idx, png_directory)
```

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
   http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

SPDX-License-Identifier: EPL-2.0


# GIAO 的 RHF 核磁 (屏蔽) 共振常数 (NMR) 数值导数

> 创建时间：2020-08-31

在这篇文档中，我们会讨论使用 PySCF 以及其作为 libcint 的接口，计算 GIAO 的 RHF 数值核磁 (屏蔽) 共振常数 (Nuclear Magnetic Resonance constant, NMR) 的程序。该文档大量参考 PySCF 的代码 [nmr/rhf.py](https://github.com/pyscf/pyscf/blob/master/pyscf/prop/nmr/rhf.py)。

与上一篇文档一样，我们的讨论中所使用到的分子体系 `mol` 会是非对称的氨分子，并且取用最小基组。其 RHF 计算放在实例 `mf`，而 NMR 计算实例会放在 `mf_nmr`。

```python
from pyscf import gto, scf, dft
from pyscf.prop import nmr
from pyscf.data import nist
from scipy import constants
import numpy as np

np.set_printoptions(precision=5, linewidth=150, suppress=True)
```

```python
mol = gto.Mole()
mol.atom = """
N  0.  0.  0.
H  0.  1.  0.2
H  0.1 0.3 1.5
H  0.9 0.4 -.2
"""
mol.basis = "STO-3G"
mol.verbose = 0
mol.build()
coord_orig = np.zeros(3)
```

```python
nocc, nao, nmo, natm = mol.nelec[0], mol.nao, mol.nao, mol.natm
```

其自洽场能量为

```python
mf = scf.RHF(mol).run()
mf.e_tot
```

    -55.25354051468657

核磁屏蔽张量 (Shielding Constant) $\sigma_{ts}^A$ 可以表示如下 (维度 $(A, t, s)$)：

$$
\sigma_{ts}^A = \frac{\partial^2 E_\mathrm{tot}}{\partial \mathscr{B}_t \partial \mu_{A_s}}
$$

```python
mf_nmr = nmr.RHF(mf)
mf_nmr.kernel()
```

    array([[[305.25641,  22.67382,  12.29714],
            [-22.38193, 180.11491,  -2.84064],
            [ 13.65836, -28.07439, 383.02872]],

           [[ 25.68677,  -0.27932,  -0.31477],
            [ -0.84315,  35.70697,  -4.26016],
            [ -1.06334,   2.07184,  28.17535]],

           [[ 24.06192,  -0.12066,   0.81476],
            [  0.99821,  24.76595,  -0.86457],
            [ -0.74954,   0.2574 ,  31.65924]],

           [[ 39.48533,   4.28363,  -7.95068],
            [  0.89957,  26.69818,  -0.9659 ],
            [ -4.40617,  -0.48147,  28.90276]]])

上述结果是以 ppm 为单位的表达。

## 微扰原子矩阵的表达

### 外磁场微扰的一阶 Core Hamiltonian

关于该微扰矩阵，我们已经在前两篇文档中有比较多的说明了。

$$
\begin{split}\begin{align}
h_{\mu \nu}^{\mathscr{B}_t}
&=
\frac{1}{2} \langle \mu | \hat l_t | \nu \rangle_{\mathrm{Gauge} \rightarrow \boldsymbol{R}_\nu}
+ \langle U_\mathrm{g}^t \mu | \hat t | \nu \rangle
+ \langle U_\mathrm{g}^t \mu | \hat v_\mathrm{nuc} | \nu \rangle
\\ & \quad
+ \sum_{\kappa \lambda} ( U_\mathrm{g}^t \mu \nu | \kappa \lambda ) D_{\kappa \lambda}^{(0)}
- \frac{1}{2} \sum_{\kappa \lambda} ( U_\mathrm{g}^t \mu \lambda | \kappa \nu ) D_{\kappa \lambda}^{(0)}
- \frac{1}{2} \sum_{\kappa \lambda} ( U_\mathrm{g}^t \kappa \nu | \mu \lambda ) D_{\kappa \lambda}^{(0)}
\end{align}\end{split}
$$

由于在 PySCF 中，函数 `nmr.rhf.make_h10` 就是专门用于生成该矩阵的，因此我们就用下述代码生成 `hcore_1_B` $h_{\mu \nu}^{\mathscr{B}_t}$ 表示该矩阵。

```python
dm_guess = mf.make_rdm1()
hcore_1_B = 1j * nmr.rhf.make_h10(mol, dm_guess)
```

### 外磁场微扰的一阶重叠矩阵

该微扰矩阵也已经在前两篇文档中有比较多的说明了。

$$
S_{\mu \nu}^{\mathscr{B}_t} = \langle U_g^t \mu | \nu \rangle
$$

我们用下述代码生成 `ovlp_1_B` $S_{\mu \nu}^{\mathscr{B}_t}$：

```python
ovlp_1_B = - 1j * mol.intor("int1e_igovlp")
```

### 核磁偶极的一阶 Core Hamiltonian

核磁偶极 $\mu_{A_t}$ 所产生的一阶算符贡献可以表达为

$$
\hat h {}^{(1)} (\boldsymbol{\mu}_A)
= - i \alpha^2 \boldsymbol{\mu}_A \cdot \left( \boldsymbol{\nabla} \frac{1}{\boldsymbol{r - \boldsymbol{R}_A}} \times \boldsymbol{\nabla} \right)
= - i \alpha^2 \boldsymbol{\mu}_A \cdot \left( \frac{\boldsymbol{r} - \boldsymbol{R}_A}{|\boldsymbol{r} - \boldsymbol{R}_A|^3} \times \boldsymbol{\nabla} \right)
$$

其中，$\boldsymbol{R}_A$ 表示原子 $A$ 的核坐标，$\alpha$ 表示精细结构常数，$1/\alpha \simeq 137$。该常数可以从 PySCF 中获得，也可以从 SciPy 中获得。注意等式左边的 $\boldsymbol{\mu_A}$ 看作是外加的核磁偶极大小，而等号右边的 $\mu$ 表示原子轨道，两者意义不同；等号左边角标 $\boldsymbol{A}$ 表示原子核坐标向量，而之前两篇文档中的 $A$ 在很多文章或教材中表示 $\frac{1}{2} \boldsymbol{B} \times \boldsymbol{r}$。

```python
1 / constants.alpha
```

    137.0359990836958

但为了方便，在最后核算结果之前，我们会暂且将 $\alpha$ 当作 1 来处理。

在 PySCF 中，实现上述过程的积分字符是 `int1e_ia01p`；但其使用需要告知 `gto.mole.intor` 函数以其 $1 / \boldsymbol{r}$ 的规范原点位置具体处在哪个原子核中心。

$$
h_{\mu \nu}^{\mu_{A_t}} = - i \langle \mu | \boldsymbol{\nabla} \frac{1}{\boldsymbol{r}} \times \boldsymbol{\nabla} | \nu \rangle_{\text{Gauge of } \boldsymbol{r} \rightarrow \boldsymbol{R}_A}
$$

我们用下述代码生成 `hcore_1_m` $h_{\mu \nu}^{\mu_{A_t}}$：(维度为 $(A, t, \mu, \nu)$)

```python
hcore_1_m = np.zeros((natm, 3, nao, nao), dtype=np.complex128)
for atom_idx in range(natm):
    with mol.with_rinv_orig(mol.atom_coord(atom_idx)):
        hcore_1_m[atom_idx] = - 1j * mol.intor("int1e_ia01p")
```

### 磁场与核磁偶极的二阶 Core Hamiltonian

磁场与核磁偶极之间的算符乘积会产生二阶算符贡献项：

$$
\begin{align}
\hat h {}^{(2)} (\boldsymbol{\mathscr{B}}, \boldsymbol{\mu}_A) | \nu \rangle
&= \frac{\alpha^2}{2} \boldsymbol{\mathscr{B}}^\mathrm{T} \left( (\boldsymbol{r} - \boldsymbol{R}_\nu) \cdot \boldsymbol{\nabla} \frac{1}{\boldsymbol{r} - \boldsymbol{R}_A} - (\boldsymbol{r} - \boldsymbol{R}_\nu) \boldsymbol{\nabla} \frac{1}{(\boldsymbol{r} - \boldsymbol{R}_A)^\mathrm{T}} \right) \boldsymbol{\mu}_A | \nu \rangle \\
&\quad + \alpha^2 \boldsymbol{\mathscr{B}}^\mathrm{T} \boldsymbol{U}_\mathrm{g} \left( \boldsymbol{\nabla} \frac{1}{\boldsymbol{r} - \boldsymbol{R}_A} \times \boldsymbol{\hat p} \right)^\mathrm{T} \boldsymbol{\mu}_A | \nu \rangle
\end{align}
$$

去除精细结构常数 $\alpha$ 的贡献后，其矩阵的表达形式则是

$$
\begin{align}
h_{\mu \nu}^{A_s t} \mathscr{B}_t \mu_{A_s}
&= \langle \mu | - \frac{1}{2} \frac{(t - t_\nu) (s - s_A)}{|\boldsymbol{r} - \boldsymbol{R}_A|^3} | \nu \rangle
- \delta_{ts} \langle \mu | - \frac{1}{2} \sum_{w} \frac{(w - w_\nu) (w - w_A)}{|\boldsymbol{r} - \boldsymbol{R}_A|^3} | \nu \rangle \\
&\quad + \langle U_\mathrm{g}^t \mu | \left( \boldsymbol{\nabla} \frac{1}{\boldsymbol{r} - \boldsymbol{R}_A} \times \boldsymbol{\hat p} \right)_s | \nu \rangle
\end{align}
$$

我们用下述代码生成 `hcore_2` $h_{\mu \nu}^{A_s t}$：(维度为 $(A, t, s, \mu, \nu)$，注意维度 $t$ 对应外磁场，而 $A, s$ 对应核磁偶极)

```python
hcore_2 = np.zeros((natm, 3, 3, nao, nao))
for atom_idx in range(natm):
    with mol.with_rinv_origin(mol.atom_coord(atom_idx)):
        hcore_2[atom_idx] += mol.intor("int1e_giao_a11part").reshape((3, 3, nao, nao))
        hcore_2[atom_idx] -= np.einsum("ts, uv -> tsuv", np.eye(3), mol.intor("int1e_giao_a11part").reshape((3, 3, nao, nao)).trace(axis1=0, axis2=1))
        hcore_2[atom_idx] += mol.intor("int1e_a01gp").reshape((3, 3, nao, nao))
```

## 数值导数求 NMR 核磁屏蔽张量

随后我们就可以通过数值梯度求核磁屏蔽张量 $\sigma_{ts}^A$ (维度 $(A, t, s)$)：

$$
\sigma_{ts}^A = \frac{\partial^2 E_\mathrm{tot}}{\partial \mathscr{B}_t \partial \mu_{A_s}}
$$

在此之前，我们仍然需要构造一个通过更改 `get_hcore` Core Hamiltonian 与 `get_ovlp` 重叠矩阵的 PySCF 自洽场实例，以施加外场获得能量的函数 `eng_nmr_field`。其输入的参数 `dev_xyz_B` 是三维外加磁场大小 (对应维度 $t$)，`dev_xyz_m` 是三维外加核磁偶极大小 (对应维度 $s$)，`atom_idx` 是原子序号 (对应维度 $A$)。

$$
\begin{align}
h_{\mu \nu} &= h_{\mu \nu}^{(0)} + \mathscr{B}_t h_{\mu \nu}^{\mathscr{B}_t} + \mu_{A_s} h_{\mu \nu}^{\mu_{A_s}} + \mathscr{B}_t \mu_{A_s} h_{\mu \nu}^{\mathscr{B}_t \mu_{A_s}} \\
S_{\mu \nu} &= S_{\mu \nu}^{(0)} + \mathscr{B}_t S_{\mu \nu}^{\mathscr{B}_t}
\end{align}
$$

```python
dm_guess = mf.make_rdm1()

def eng_nmr_field(dev_xyz_B, dev_xyz_m, atom_idx):
    mf = scf.RHF(mol)
    def get_hcore(mol_=mol):
        hcore_total  = np.asarray(scf.rhf.get_hcore(mol_), dtype=np.complex128)
        hcore_total += np.einsum("tuv, t -> uv", hcore_1_B, dev_xyz_B)
        hcore_total += np.einsum("tuv, t -> uv", hcore_1_m[atom_idx], dev_xyz_m)
        hcore_total += np.einsum("tsuv, t, s -> uv", hcore_2[atom_idx], dev_xyz_B, dev_xyz_m)
        return hcore_total
    def get_ovlp(mol_):
        ovlp_total  = np.asarray(scf.rhf.get_ovlp(mol_), dtype=np.complex128)
        ovlp_total += np.einsum("tuv, t -> uv", ovlp_1_B, dev_xyz_B)
        return ovlp_total
    mf.get_hcore = get_hcore
    mf.get_ovlp  = get_ovlp
    return mf.kernel(dm=dm_guess)
```

随后使用与前文类似的数值梯度方式就能得到核磁屏蔽常数了；所使用的数值差分大小对于外磁场与核磁偶极均为 $10^{-4}$ a.u.。

```python
interval = 1e-4
num_nmr = np.zeros((natm, 3, 3))
for atom_idx in range(natm):
    for t in range(3):
        for s in range(3):
            dev_xyzs_B, dev_xyzs_m = np.zeros((2, 3)), np.zeros((2, 3))
            dev_xyzs_B[0, t] = dev_xyzs_m[0, s] = -interval
            dev_xyzs_B[1, t] = dev_xyzs_m[1, s] =  interval
            num_nmr[atom_idx, t, s] = (
                + eng_nmr_field(dev_xyzs_B[0], dev_xyzs_m[0], atom_idx)
                - eng_nmr_field(dev_xyzs_B[1], dev_xyzs_m[0], atom_idx)
                - eng_nmr_field(dev_xyzs_B[0], dev_xyzs_m[1], atom_idx)
                + eng_nmr_field(dev_xyzs_B[1], dev_xyzs_m[1], atom_idx)
            ) / (4 * interval**2)
```

```python
num_nmr
```

    array([[[ 5.73238,  0.42579,  0.23092],
            [-0.4203 ,  3.38236, -0.05334],
            [ 0.25651, -0.52722,  7.19284]],

           [[ 0.48237, -0.00525, -0.00591],
            [-0.01584,  0.67054, -0.08   ],
            [-0.01997,  0.03891,  0.5291 ]],

           [[ 0.45185, -0.00226,  0.01529],
            [ 0.01875,  0.46508, -0.01624],
            [-0.01408,  0.00484,  0.59453]],

           [[ 0.74149,  0.08044, -0.14931],
            [ 0.01689,  0.50136, -0.01814],
            [-0.08274, -0.00904,  0.54276]]])

留意到我们之前一直都使用去除结构精细常数的结果，因此我们需要乘以 $\alpha^2$。同时由于单位是 ppm，因此最终我们需要乘以 $10^6 \alpha^2$：

```python
num_nmr * constants.alpha**2 * 10**6
```

    array([[[305.25688,  22.67379,  12.2969 ],
            [-22.38171, 180.11542,  -2.84064],
            [ 13.65948, -28.0751 , 383.02872]],

           [[ 25.68659,  -0.27935,  -0.3146 ],
            [ -0.84328,  35.70701,  -4.26026],
            [ -1.06338,   2.07184,  28.17533]],

           [[ 24.06171,  -0.12057,   0.81443],
            [  0.99832,  24.76594,  -0.86458],
            [ -0.74969,   0.25765,  31.65949]],

           [[ 39.48546,   4.28353,  -7.95088],
            [  0.89964,  26.69806,  -0.9661 ],
            [ -4.40616,  -0.48158,  28.9029 ]]])


# Logbook

```python
# %load ../imports.py
"""
These is the standard setup for the notebooks.
"""

%matplotlib inline
%load_ext autoreload
%autoreload 2

#from jupyterthemes import jtplot
#jtplot.style(theme='onedork', context='notebook', ticks=True, grid=False)

import pandas as pd
pd.options.display.max_rows = 999
pd.options.display.max_columns = 999
pd.set_option("display.max_columns", None)
import numpy as np
import os
import matplotlib.pyplot as plt
#plt.style.use('paper')

#import data
import copy
from rolldecay.bis_system import BisSystem
from rolldecay import database
from mdldb.tables import Run

from sklearn.pipeline import Pipeline
from rolldecayestimators.transformers import CutTransformer, LowpassFilterDerivatorTransformer, ScaleFactorTransformer, OffsetTransformer
from rolldecayestimators.direct_estimator_cubic import EstimatorQuadraticB, EstimatorCubic
from rolldecayestimators.ikeda_estimator import IkedaQuadraticEstimator
import rolldecayestimators.equations as equations
import rolldecayestimators.lambdas as lambdas
from rolldecayestimators.substitute_dynamic_symbols import lambdify
import rolldecayestimators.symbols as symbols
import sympy as sp

from sklearn.metrics import r2_score

```

    c:\dev\rolldecay\venv\lib\site-packages\matplotlib\style\core.py:167: UserWarning: In C:\Users\maa\.matplotlib\stylelib\paper.mplstyle: Duplicate key in file "C:\Users\maa\.matplotlib\stylelib\paper.mplstyle", line #461
      styles = read_style_directory(stylelib_path)
    c:\dev\rolldecay\venv\lib\site-packages\matplotlib\style\core.py:167: UserWarning: In C:\Users\maa\.matplotlib\stylelib\paper.mplstyle: Duplicate key in file "C:\Users\maa\.matplotlib\stylelib\paper.mplstyle", line #462
      styles = read_style_directory(stylelib_path)
    c:\dev\rolldecay\venv\lib\site-packages\h5py\__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      from ._conv import register_converters as _register_converters

## Nomenclature
| Variable | Explain |
|---|---|
|$\pi$| example |

Here is a cell link: [Logbook](#logbook)

## 2020-09-22
What have I done today?

## References

```python

```


_Lambda School Data Science, Unit 2_

# Sprint Challenge: Predict Steph Curry's shots 🏀

For your Sprint Challenge, you'll use a dataset with all Steph Curry's NBA field goal attempts. (Regular season and playoff games, from October 28, 2009, through June 5, 2019.)

You'll predict whether each shot was made, using information about the shot and the game. This is hard to predict! Try to get above 60% accuracy. The dataset was collected with the [nba_api](https://github.com/swar/nba_api) Python library.

```python
%%capture
import sys

if 'google.colab' in sys.modules:
    # Install packages in Colab
    !pip install category_encoders==2.*
    !pip install pandas-profiling==2.*
```

```python
# Read data
import pandas as pd
url = 'https://drive.google.com/uc?export=download&id=1fL7KPyxgGYfQDsuJoBWHIWwCAf-HTFpX'
df = pd.read_csv(url)

# Check data shape
assert df.shape == (13958, 20)
```

To demonstrate mastery on your Sprint Challenge, do all the required, numbered instructions in this notebook.

To earn a score of "3", also do all the stretch goals.

You are permitted and encouraged to do as much data exploration as you want.

**1. Begin with baselines for classification.** Your target to predict is `shot_made_flag`. What is your baseline accuracy, if you guessed the majority class for every prediction?

**2. Hold out your test set.** Use the 2018-19 season to test. NBA seasons begin in October and end in June. You'll know you've split the data correctly when your test set has 1,709 observations.

**3. Engineer new feature.** Engineer at least **1** new feature, from this list, or your own idea.
- **Homecourt Advantage**: Is the home team (`htm`) the Golden State Warriors (`GSW`) ?
- **Opponent**: Who is the other team playing the Golden State Warriors?
- **Seconds remaining in the period**: Combine minutes remaining with seconds remaining, to get the total number of seconds remaining in the period.
- **Seconds remaining in the game**: Combine period, and seconds remaining in the period, to get the total number of seconds remaining in the game. A basketball game has 4 periods, each 12 minutes long.
- **Made previous shot**: Was Steph Curry's previous shot successful?

**4. Decide how to validate** your model. Choose one of the following options. Any of these options are good. You are not graded on which you choose.
- **Train/validate/test split: train on the 2009-10 season through 2016-17 season, validate with the 2017-18 season.** You'll know you've split the data correctly when your train set has 11,081 observations, and your validation set has 1,168 observations.
- **Train/validate/test split: random 80/20%** train/validate split.
- **Cross-validation** with independent test set. You may use any scikit-learn cross-validation method.

**5.** Use a scikit-learn **pipeline** to **encode categoricals** and fit a **Decision Tree** or **Random Forest** model.

**6.** Get your model's **validation accuracy.** (Multiple times if you try multiple iterations.)

**7.** Get your model's **test accuracy.** (One time, at the end.)

**8.** Given a **confusion matrix** for a hypothetical binary classification model, **calculate accuracy, precision, and recall.**

### Stretch Goals
- Engineer 4+ new features total, either from the list above, or your own ideas.
- Make 2+ visualizations to explore relationships between features and target.
- Optimize 3+ hyperparameters by trying 10+ "candidates" (possible combinations of hyperparameters). You can use `RandomizedSearchCV` or do it manually.
- Get and plot your model's feature importances.

## 1. Begin with baselines for classification.

>Your target to predict is `shot_made_flag`. What would your baseline accuracy be, if you guessed the majority class for every prediction?

```python
df.head()
```

      game_id
      game_event_id
      player_name
      period
      minutes_remaining
      seconds_remaining
      action_type
      shot_type
      shot_zone_basic
      shot_zone_area
      shot_zone_range
      shot_distance
      loc_x
      loc_y
      shot_made_flag
      game_date
      htm
      vtm
      season_type
      scoremargin_before_shot

      0
      20900015
      4
      Stephen Curry
      1
      11
      25
      Jump Shot
      3PT Field Goal
      Above the Break 3
      Right Side Center(RC)
      24+ ft.
      26
      99
      249
      0
      2009-10-28
      GSW
      HOU
      Regular Season
      2.0

      1
      20900015
      17
      Stephen Curry
      1
      9
      31
      Step Back Jump shot
      2PT Field Goal
      Mid-Range
      Left Side Center(LC)
      16-24 ft.
      18
      -122
      145
      1
      2009-10-28
      GSW
      HOU
      Regular Season
      0.0

      2
      20900015
      53
      Stephen Curry
      1
      6
      2
      Jump Shot
      2PT Field Goal
      In The Paint (Non-RA)
      Center(C)
      8-16 ft.
      14
      -60
      129
      0
      2009-10-28
      GSW
      HOU
      Regular Season
      -4.0

      3
      20900015
      141
      Stephen Curry
      2
      9
      49
      Jump Shot
      2PT Field Goal
      Mid-Range
      Left Side(L)
      16-24 ft.
      19
      -172
      82
      0
      2009-10-28
      GSW
      HOU
      Regular Season
      -4.0

      4
      20900015
      249
      Stephen Curry
      2
      2
      19
      Jump Shot
      2PT Field Goal
      Mid-Range
      Left Side Center(LC)
      16-24 ft.
      16
      -68
      148
      0
      2009-10-28
      GSW
      HOU
      Regular Season
      0.0

```python
target = 'shot_made_flag'
```

```python
baseline = df[target].value_counts(normalize=True)
```

```python
baseline[1] #< - - - This is our baseline
```

    0.4729187562688064

```python
### We have a 47% baseline accuracy, meaning if we guessed that he got it in the net
### every time, we would be right a little over 47% of the time.
```

## 2. Hold out your test set.

>Use the 2018-19 season to test. NBA seasons begin in October and end in June. You'll know you've split the data correctly when your test set has 1,709 observations.

```python
test = df[(df['game_date'] >= '2018-10') & (df['game_date'] <= '2019-07')]
```

```python
test.shape
```

    (1709, 20)

## 3. Engineer new feature.

>Engineer at least **1** new feature, from this list, or your own idea.
>
>- **Homecourt Advantage**: Is the home team (`htm`) the Golden State Warriors (`GSW`) ?
>- **Opponent**: Who is the other team playing the Golden State Warriors?
>- **Seconds remaining in the period**: Combine minutes remaining with seconds remaining, to get the total number of seconds remaining in the period.
>- **Seconds remaining in the game**: Combine period, and seconds remaining in the period, to get the total number of seconds remaining in the game. A basketball game has 4 periods, each 12 minutes long.
>- **Made previous shot**: Was Steph Curry's previous shot successful?

```python
df.head()
```

      game_id
      game_event_id
      player_name
      period
      minutes_remaining
      seconds_remaining
      action_type
      shot_type
      shot_zone_basic
      shot_zone_area
      shot_zone_range
      shot_distance
      loc_x
      loc_y
      shot_made_flag
      game_date
      htm
      vtm
      season_type
      scoremargin_before_shot

      0
      20900015
      4
      Stephen Curry
      1
      11
      25
      Jump Shot
      3PT Field Goal
      Above the Break 3
      Right Side Center(RC)
      24+ ft.
      26
      99
      249
      0
      2009-10-28
      GSW
      HOU
      Regular Season
      2.0

      1
      20900015
      17
      Stephen Curry
      1
      9
      31
      Step Back Jump shot
      2PT Field Goal
      Mid-Range
      Left Side Center(LC)
      16-24 ft.
      18
      -122
      145
      1
      2009-10-28
      GSW
      HOU
      Regular Season
      0.0

      2
      20900015
      53
      Stephen Curry
      1
      6
      2
      Jump Shot
      2PT Field Goal
      In The Paint (Non-RA)
      Center(C)
      8-16 ft.
      14
      -60
      129
      0
      2009-10-28
      GSW
      HOU
      Regular Season
      -4.0

      3
      20900015
      141
      Stephen Curry
      2
      9
      49
      Jump Shot
      2PT Field Goal
      Mid-Range
      Left Side(L)
      16-24 ft.
      19
      -172
      82
      0
      2009-10-28
      GSW
      HOU
      Regular Season
      -4.0

      4
      20900015
      249
      Stephen Curry
      2
      2
      19
      Jump Shot
      2PT Field Goal
      Mid-Range
      Left Side Center(LC)
      16-24 ft.
      16
      -68
      148
      0
      2009-10-28
      GSW
      HOU
      Regular Season
      0.0

```python
def home_adv(df):
  export = df
  for cell in export['htm']:
    export['home_adv'] = df['htm'] == 'GSW'
  return export
```

```python
df = home_adv(df)
```

```python
df[df['home_adv'] == False]
```

      game_id
      game_event_id
      player_name
      period
      minutes_remaining
      seconds_remaining
      action_type
      shot_type
      shot_zone_basic
      shot_zone_area
      shot_zone_range
      shot_distance
      loc_x
      loc_y
      shot_made_flag
      game_date
      htm
      vtm
      season_type
      scoremargin_before_shot
      home_adv

      12
      20900030
      9
      Stephen Curry
      1
      10
      39
      Jump Shot
      3PT Field Goal
      Above the Break 3
      Right Side Center(RC)
      24+ ft.
      24
      152
      195
      0
      2009-10-30
      PHX
      GSW
      Regular Season
      2.0
      False

      13
      20900030
      18
      Stephen Curry
      1
      9
      27
      Jump Shot
      2PT Field Goal
      Mid-Range
      Left Side Center(LC)
      16-24 ft.
      22
      -81
      205
      1
      2009-10-30
      PHX
      GSW
      Regular Season
      -0.0
      False

      14
      20900030
      258
      Stephen Curry
      2
      1
      28
      Jump Shot
      2PT Field Goal
      Mid-Range
      Right Side Center(RC)
      16-24 ft.
      20
      125
      162
      1
      2009-10-30
      PHX
      GSW
      Regular Season
      -9.0
      False

      15
      20900030
      264
      Stephen Curry
      2
      1
      0
      Jump Shot
      3PT Field Goal
      Above the Break 3
      Center(C)
      24+ ft.
      26
      -76
      250
      1
      2009-10-30
      PHX
      GSW
      Regular Season
      -10.0
      False

      16
      20900030
      313
      Stephen Curry
      3
      8
      35
      Jump Shot
      2PT Field Goal
      Mid-Range
      Left Side(L)
      8-16 ft.
      15
      -158
      -6
      0
      2009-10-30
      PHX
      GSW
      Regular Season
      -10.0
      False

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      13922
      41800402
      399
      Stephen Curry
      3
      9
      17
      Driving Layup Shot
      2PT Field Goal
      Restricted Area
      Center(C)
      Less Than 8 ft.
      1
      -1
      17
      0
      2019-06-02
      TOR
      GSW
      Playoffs
      4.0
      False

      13923
      41800402
      489
      Stephen Curry
      3
      3
      29
      Pullup Jump shot
      3PT Field Goal
      Above the Break 3
      Left Side Center(LC)
      24+ ft.
      25
      -221
      132
      0
      2019-06-02
      TOR
      GSW
      Playoffs
      11.0
      False

      13924
      41800402
      501
      Stephen Curry
      3
      2
      56
      Driving Floating Bank Jump Shot
      2PT Field Goal
      Mid-Range
      Right Side(R)
      8-16 ft.
      11
      93
      68
      0
      2019-06-02
      TOR
      GSW
      Playoffs
      9.0
      False

      13925
      41800402
      527
      Stephen Curry
      3
      0
      40
      Jump Shot
      3PT Field Goal
      Above the Break 3
      Right Side Center(RC)
      24+ ft.
      28
      109
      258
      1
      2019-06-02
      TOR
      GSW
      Playoffs
      7.0
      False

      13926
      41800402
      530
      Stephen Curry
      3
      0
      10
      Pullup Jump shot
      3PT Field Goal
      Above the Break 3
      Left Side Center(LC)
      24+ ft.
      27
      -105
      251
      0
      2019-06-02
      TOR
      GSW
      Playoffs
      8.0
      False

7087 rows × 21 columns

## **4. Decide how to validate** your model.

>Choose one of the following options. Any of these options are good. You are not graded on which you choose.
>
>- **Train/validate/test split: train on the 2009-10 season through 2016-17 season, validate with the 2017-18 season.** You'll know you've split the data correctly when your train set has 11,081 observations, and your validation set has 1,168 observations.
>- **Train/validate/test split: random 80/20%** train/validate split.
>- **Cross-validation** with independent test set. You may use any scikit-learn cross-validation method.

```python
### Split using 09/10 through 16/17 for train, 17/18 for val and 18/19 for test
train = df[(df['game_date'] >= '2009-10') & (df['game_date'] <= '2017-07')]
val  = df[(df['game_date'] >= '2017-10') & (df['game_date'] <= '2018-07')]
test = df[(df['game_date'] >= '2018-10') & (df['game_date'] <= '2019-07')]
```

```python
print(train.shape)
print(val.shape)
print(test.shape)
```

    (11081, 21)
    (1168, 21)
    (1709, 21)

## 5. Use a scikit-learn pipeline to encode categoricals and fit a Decision Tree or Random Forest model.

```python
from scipy.stats import randint, uniform
import category_encoders as ce
from sklearn.impute import SimpleImputer
from sklearn.feature_selection import VarianceThreshold
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import RandomizedSearchCV

```

    /usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
      import pandas.util.testing as tm

```python
### Set the target to the status group column
target = 'shot_made_flag'

### Create a dataframe with the other features
df_train = train.drop(columns=[target,'player_name'])
```

```python
### Get a list of the numeric features
numeric_features = df_train.select_dtypes(include='number').columns.tolist()

### Get a series with the cardinality of the nonnumeric features
cardinality = df_train.select_dtypes(exclude='number').nunique()

### Get a list of all categorical features with cardinality <=
categorical_features = cardinality[cardinality <= 20].index.tolist()

### Combine the lists
features = numeric_features + categorical_features

X_train = train[features]
y_train = train[target]
X_test = test[features]
y_test = test[target]
X_val = val[features]
y_val = val[target]
```

```python
X_train
```

```python
pipeline = make_pipeline(
    ce.OrdinalEncoder(),
    RandomForestClassifier(random_state=56)
)

param_distributions = {
    'randomforestclassifier__n_estimators': randint(50, 500),
    'randomforestclassifier__max_depth': [5, 10, 15, 20, None],

}

search = RandomizedSearchCV(
    pipeline,
    param_distributions=param_distributions,
    n_iter=25,
    cv=2,
    scoring='accuracy',
    verbose=10,
    return_train_score=True,
    n_jobs=2
)

search.fit(X_train, y_train);
```

```python
print('Best hyperparameters', search.best_params_)
print('Cross-validation Accuracy', -search.best_score_)
```

    Best hyperparameters {'randomforestclassifier__max_depth': 5, 'randomforestclassifier__n_estimators': 107}
    Cross-validation Accuracy -0.5569894784986484

```python
### This shows parameters tested and the outcomes for each iteration run
pd.DataFrame(search.cv_results_).sort_values(by='rank_test_score').T
```

      16
      22
      3
      11
      4
      9
      14
      5
      6
      1
      21
      8
      20
      10
      12
      19
      7
      24
      23
      17
      0
      13
      15
      18
      2

      mean_fit_time
      1.86566
      2.25997
      1.07133
      0.800716
      0.384891
      0.872998
      4.42172
      2.04271
      5.37547
      2.11378
      5.09159
      1.89116
      3.45839
      2.72211
      2.1427
      2.40641
      6.624
      4.98437
      2.6118
      5.02159
      6.04058
      3.81592
      1.12724
      3.51611
      2.7356

      std_fit_time
      0.00532889
      0.00178087
      0.00613308
      0.0021522
      0.000128031
      0.012194
      0.00750661
      0.0124761
      0.164587
      0.0731187
      0.0178747
      0.0335208
      0.086913
      0.0540954
      0.0387574
      0.0316125
      0.21495
      0.100278
      0.00790238
      0.0692366
      0.103255
      0.0700758
      0.0270191
      0.0263519
      0.0240211

      mean_score_time
      0.208056
      0.252902
      0.126626
      0.104293
      0.0565867
      0.107379
      0.396378
      0.194318
      0.438691
      0.182856
      0.428925
      0.170293
      0.277424
      0.226058
      0.190417
      0.208027
      0.520404
      0.373326
      0.23314
      0.407741
      0.498049
      0.314803
      0.105514
      0.309486
      0.251082

      std_score_time
      0.0001477
      0.00571251
      0.000138879
      0.00543916
      0.000460863
      0.000889421
      0.0116577
      0.0049299
      0.0435863
      0.0117835
      0.0114348
      0.00283217
      0.0169436
      0.00696039
      0.0101023
      0.0031482
      0.0247679
      0.0533562
      0.0123109
      0.0287839
      0.0301688
      0.0114181
      0.00226581
      0.00791848
      0.0148786

      param_randomforestclassifier__max_depth
      5
      5
      5
      5
      5
      5
      10
      10
      None
      None
      15
      20
      None
      20
      20
      20
      None
      15
      15
      20
      20
      20
      None
      15
      15

      param_randomforestclassifier__n_estimators
      292
      355
      161
      116
      50
      130
      456
      208
      353
      136
      401
      128
      225
      185
      145
      160
      436
      406
      203
      350
      417
      265
      70
      271
      213

      params
      {'randomforestclassifier__max_depth': 5, 'rand...
      {'randomforestclassifier__max_depth': 5, 'rand...
      {'randomforestclassifier__max_depth': 5, 'rand...
      {'randomforestclassifier__max_depth': 5, 'rand...
      {'randomforestclassifier__max_depth': 5, 'rand...
      {'randomforestclassifier__max_depth': 5, 'rand...
      {'randomforestclassifier__max_depth': 10, 'ran...
      {'randomforestclassifier__max_depth': 10, 'ran...
      {'randomforestclassifier__max_depth': None, 'r...
      {'randomforestclassifier__max_depth': None, 'r...
      {'randomforestclassifier__max_depth': 15, 'ran...
      {'randomforestclassifier__max_depth': 20, 'ran...
      {'randomforestclassifier__max_depth': None, 'r...
      {'randomforestclassifier__max_depth': 20, 'ran...
      {'randomforestclassifier__max_depth': 20, 'ran...
      {'randomforestclassifier__max_depth': 20, 'ran...
      {'randomforestclassifier__max_depth': None, 'r...
      {'randomforestclassifier__max_depth': 15, 'ran...
      {'randomforestclassifier__max_depth': 15, 'ran...
      {'randomforestclassifier__max_depth': 20, 'ran...
      {'randomforestclassifier__max_depth': 20, 'ran...
      {'randomforestclassifier__max_depth': 20, 'ran...
      {'randomforestclassifier__max_depth': None, 'r...
      {'randomforestclassifier__max_depth': 15, 'ran...
      {'randomforestclassifier__max_depth': 15, 'ran...

      split0_test_score
      0.555676
      0.555856
      0.555495
      0.555495
      0.556037
      0.555134
      0.55351
      0.553149
      0.550442
      0.551525
      0.552066
      0.55333
      0.552247
      0.550803
      0.552608
      0.551886
      0.550623
      0.551345
      0.552066
      0.550623
      0.550623
      0.550803
      0.551345
      0.550803
      0.552247

      split1_test_score
      0.559567
      0.558845
      0.557581
      0.556679
      0.556137
      0.556498
      0.541155
      0.536101
      0.53574
      0.534477
      0.533755
      0.53213
      0.533032
      0.534296
      0.53213
      0.531949
      0.532852
      0.531769
      0.530505
      0.531769
      0.531408
      0.530144
      0.527617
      0.526895
      0.525451

      mean_test_score
      0.557621
      0.557351
      0.556538
      0.556087
      0.556087
      0.555816
      0.547333
      0.544625
      0.543091
      0.543001
      0.54291
      0.54273
      0.54264
      0.54255
      0.542369
      0.541918
      0.541737
      0.541557
      0.541286
      0.541196
      0.541015
      0.540474
      0.539481
      0.538849
      0.538849

      std_test_score
      0.00194546
      0.00149421
      0.00104291
      0.000591651
      5.01838e-05
      0.000681871
      0.00617748
      0.00852408
      0.00735104
      0.00852423
      0.00915595
      0.0105999
      0.0096072
      0.00825354
      0.0102389
      0.00996824
      0.00888532
      0.00978778
      0.0107805
      0.00942684
      0.00960734
      0.0103293
      0.0118636
      0.0119539
      0.0133978

      rank_test_score
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25

      split0_train_score
      0.606859
      0.606318
      0.60704
      0.605235
      0.605054
      0.605415
      0.788087
      0.786643
      1
      1
      0.985379
      1
      1
      1
      1
      1
      1
      0.984657
      0.984296
      1
      1
      1
      1
      0.985921
      0.984657

      split1_train_score
      0.585093
      0.584552
      0.585273
      0.586356
      0.586898
      0.586176
      0.762859
      0.761956
      1
      1
      0.978163
      1
      1
      1
      1
      1
      1
      0.978524
      0.97726
      1
      1
      1
      1
      0.976358
      0.978704

      mean_train_score
      0.595976
      0.595435
      0.596157
      0.595795
      0.595976
      0.595795
      0.775473
      0.774299
      1
      1
      0.981771
      1
      1
      1
      1
      1
      1
      0.98159
      0.980778
      1
      1
      1
      1
      0.981139
      0.981681

      std_train_score
      0.0108831
      0.0108831
      0.0108831
      0.0094392
      0.00907824
      0.00961969
      0.012614
      0.0123431
      0
      0
      0.00360814
      0
      0
      0
      0
      0
      0
      0.00306665
      0.0035178
      0
      0
      0
      0
      0.00478126
      0.00297642

## 6.Get your model's validation accuracy

> (Multiple times if you try multiple iterations.)

```python
val_accuracy = search.score(X_val,y_val)
```

    0.5779109589041096

```python
print('Validation Accuracy is:',val_accuracy)
```

    Validation Accuracy is: 0.5779109589041096

```python

```

## 7. Get your model's test accuracy

> (One time, at the end.)

```python
test_accuracy = search.score(X_test,y_test)
```

```python
print('Test Accuracy is:',test_accuracy)
```

    Test Accuracy is: 0.5763604447045055

## 8. Given a confusion matrix, calculate accuracy, precision, and recall.

Imagine this is the confusion matrix for a binary classification model. Use the confusion matrix to calculate the model's accuracy, precision, and recall.

    Predicted

    Negative
    Positive

    Actual
    Negative
    85
    58

    Positive
    8
     36

### Calculate accuracy

  \begin{align}
    AC = \frac {TN + TP}{TN + FP + FN + TP}
  \end{align}

  \begin{align}
    AC = \frac {85 + 36}{85 + 58 + 8 + 36}
  \end{align}

  \begin{align}
    AC = \frac {121}{187}
  \end{align}

  \begin{align}
    AC = 0.64705882
  \end{align}

### Calculate precision

  \begin{align}
    precision: \frac {TP}{FP + TP}
  \end{align}

  \begin{align}
    precision: \frac {36}{58 + 36}
  \end{align}

\begin{align}
    precision: \frac {36}{94}
  \end{align}

\begin{align}
    precision: 0.382978723404255
  \end{align}

### Calculate recall

  \begin{align}
    recall = \frac {TP}{FN + TP}
  \end{align}

\begin{align}
    recall = \frac {36}{8 + 36}
  \end{align}

\begin{align}
    recall = \frac {36}{44}
  \end{align}

\begin{align}
    recall = 0.818181818181818
  \end{align}


```python
import pandas
import numpy as np
import tensorflow.keras as keras
from IPython.display import clear_output
from keras.models import Model
from keras.layers import Dense, Input
from IPython.display import display
import sympy as sp
sp.init_printing(use_latex = True)
import math

import matplotlib.pyplot as plt
%matplotlib inline

# Some Constants
WHITE = 1
BLACK = -1
EMPTY = 0
# CHANGING BOARD SIZE APPARENTLY DOESNT WORK. Let me know if you want this fixed.
WIDTH = 9
BOARD_SIZE = WIDTH*WIDTH

def index_to_coordinate(index): #Returns row, column
    return int(index / WIDTH), index % WIDTH

def coordinate_to_index(x, y): #/Returns index
    return y*WIDTH+x

```

```python
#defining input shape (board state)
#input_size = X.shape[1]

input_shape = (82, )

# The model needs to be created here. Right now this is a smaller convolutional network I made to try to reduce training times
model = keras.models.Sequential()
model.add(keras.layers.Dense(82, input_shape = input_shape, kernel_initializer='random_normal'))
model.add(keras.layers.Dense(82, activation = 'relu'))
model.add(keras.layers.Dense(82, activation = 'relu'))
model.add(keras.layers.Dense(82, activation = 'relu'))
model.add(keras.layers.Dense(82, activation = 'relu'))
model.add(keras.layers.Dense(82, activation = 'softmax'))

model.compile(loss=keras.losses.CategoricalCrossentropy(), optimizer = keras.optimizers.Adam(), metrics = [keras.metrics.CategoricalAccuracy()])

model.load_weights('mini_weights.h5') # I had to resist the urge to call this frosted mini weights
```

```python
def encode(board, color): # Converts a board position into a string, to reduce the memory needed to store past position states
    space = 0
    out = str(color)
    for i in range(WIDTH*WIDTH): # Loop through each tile on the board and encode it
        if board[i] != 0: # Non-Empty Tile
            if board[i] == BLACK: # Black piece
                out += "b"
            if board[i] == WHITE: # White piece
                out += "w"
            if space > 0: # Show that the previous streak of empty tiles has ended.
                out += "." + str(space) + "."
                space = 0
        else: # Space tells how many empty squares there were in a row before a piece was found. An empty board would be encoded as "" (best case), a full board as wbwbwbwb. . . (worst case)
            space += 1
    return out

def decode(notation): # Decodes a string created by the encode function and converts it to an array of length 81.
    index = 1
    strIndex = 0
    skip = 0
    board = np.zeros((WIDTH*WIDTH))
    color = notation[0]
    board[:] = -1
    while index < WIDTH*WIDTH and strIndex < notation.size():
        if notation[strIndex] == 'b' or notation[strIndex] == 'w':
            if notation[strIndex] == 'b': # Black Piece on this tile
                board[index] = BLACK
            else:                         # White piece on this tile
                board[index] = WHITE
            index += 1
        elif notation[strIndex] == '.':   # Empty tiles
            skip = 0
            strIndex += 1
            while notation[strIndex] != '.': # Streaks of empty tiles are coded as numbers followed by a period. This reads how many empty tiles there were, stopping once it sees a period.
                skip *= 10
                skip += notation[strIndex] - '0'
                strIndex += 1
            index += skip
    return board, color
```

```python
def Flood(board, index, color): # Performs a flood fill to check if tiles are completely surrounded. If they are, then it returns a 1 and a list of the pieces to be captured.
    closed = [] # Closed List
    open_list = [index] # Open List

    x = index % WIDTH # X coordinate
    y = int(index / WIDTH) # Y coordinate
    if board[index] != color:
        return -1, []
    while len(open_list) > 0:
        x = open_list[-1] % WIDTH
        y = int(open_list[-1] / WIDTH)
        closed.append(open_list.pop())

        for a in range(-1, 2): # This nested loop checks in the 4 cardinal directions adjacent to a tile
            for b in range(-1, 2):
                if (a == 0 or b == 0) and a != b: # <- make sure not to check diagonals
                    if a + y >= 0 and b + x >= 0 and a + y < WIDTH and b + x < WIDTH: # Make sure coords aren't out of bounds
                        idx = (x+b)+(y+a)*WIDTH # index, for simplification
                        if board[idx] == EMPTY: # If it's empty, stop searching, there will be no capture here.
                            return -1, []
                        if board[idx] == color: # If it's another allied stone, check to see if its liberties are taken
                            if not idx in closed and not idx in open_list: # Make sure that stone hasn't already been checked/will be checked to avoid infinite loops
                                open_list.append(idx)
    return 1, closed # No open tiles were ever found to stop the loop, so all liberties were taken.

def Move(bd, index, color): # Make a move. Return 1, board if successful, -1, parameter board if unsuccessful (ie suicide move)
    # Make a move at the index. color 1 = white, color -1 = black
    ENEMY = color * -1 # Opposite color of the player's
    board = np.array(bd) # This is done to make a shallow copy of the parameter
    board[index] = color # This is where the move is made.
    # Check for captured pieces:
    x = index % WIDTH
    y = int(index / WIDTH)

    captured = []
    if x > 0: # Check for capture 1 sq to the left
        captured += Flood(board, (x-1)+y*WIDTH, ENEMY)[1]

    if x < WIDTH-1: # Check for capture 1 sq to the right
        captured += Flood(board, (x+1)+y*WIDTH, ENEMY)[1]

    if y > 0: # Check for capture 1 sq up
        captured += Flood(board, x+(y-1)*WIDTH, ENEMY)[1]

    if y < WIDTH-1: # Check for capture 1 sq down
        captured += Flood(board, x+(y+1)*WIDTH, ENEMY)[1]

    if len(captured) > 0:
        for i in captured:
            board[i] = EMPTY

    if Flood(board, index, color)[0] != -1: # Check for Suicides
        return -1, bd

    return 1, board

def createMask(board, positions, color):
    mask = np.zeros(WIDTH*WIDTH)
    for a in range(WIDTH*WIDTH):
        mask[a] = 0
        if board[a] == EMPTY:
            mask[a] = 1
        current = ""
        for a in range(WIDTH*WIDTH):
            if mask[a]:
                variation = Move(board[:], a, color)
                if variation[0] == -1:
                    mask[a] = -1
                else:
                    if len(positions) > 0:
                        position = encode(variation[1], color*-1)
                        if position in positions:
                            mask[a] = -1
    return mask
```

```python
def printBoard(board, turn):
    string = ""
    for y in range(-1, WIDTH):
        for x in range(-1, WIDTH):
            if x < 0:
                if y < 0:
                    string += '# '
                else:
                    string += str(y+1)+" "
            elif y < 0 and x > -1:
                string += ' ' + chr(65+x)
            else:
                if board[x+y*WIDTH] == BLACK:
                    string += ' @'
                elif board[x+y*WIDTH] == WHITE:
                    string += ' O'
                elif board[x+y*WIDTH] == EMPTY:
                    string += ' .'
        string += '\n'
    print(string)

def stringToIndex(mv):
    # Output 81 for pass,
    # 0-80 for a1-i8
    #-1 if invalid
    move = mv.lower()
    xC = 0
    yC = 0
    index = 0

    if move == "pass":
        return 81
    else:
        xC = ord(move[0]) - ord('a')
        yC = ord(move[1]) - ord('1')
        if xC < 0 or xC > WIDTH or yC < 0 or yC > WIDTH:
            return -1
        return xC+yC*WIDTH

def getEngineMove(bd, msk, color): # Get the index which the engine wants to move to.
    board = np.append(bd, np.array([color]))
    mask = np.append(np.array([0]), msk) # Size 82 np array, 0 = pass, 1-81 = board indexes
    net_moveList = (mask * model(np.array([board]), training=False)[0])
    net_move = 0
    for x in range(len(net_moveList)):
        if net_moveList[x] > net_moveList[net_move]:
            net_move = x
    return net_move-1

def main():
    board = np.zeros(WIDTH*WIDTH) # Board here is an 81 size array, but net has to have 82 size input
    positions = []
    color = -1
    pass_count = 0
    val = ""
    xC = 0
    yC = 0

    while val != "quit" and pass_count < 2:
        positions.append(encode(board, color))
        mask = createMask(board, positions, color)
        #clear_output(wait=True) # IF THE CODE IS FREEZING UP DURING GAMEPLAY, DELETE/COMMENT THIS LINE. JUPYTER SOMETIMES FREEZED TRYING TO CLEAR OUTPUT
        printBoard(board, 0)
        if False: # The net plays black's moves here. if TRUE -> net plays both sides. if FALSE -> player plays both sides. "if color == BLACK:" net plays black, player plays white
            if color == BLACK:
                string = "Black's Move: "
            else: string = "White's Move: "
            net_move = getEngineMove(board, mask, color)
            if net_move > 0:
                board = Move(board, net_move, color)[1]
                pass_count = 0
            else:
                pass_count += 1
            color *= -1
            string += ": " + str(net_move) # Print what index the net choses to play at.
            print(string)
        else:
            if color == BLACK:
                string = "Black's Move: "
            else: string = "White's Move: "
            val = input(string)
            if val == "pass":
                color *= -1
                pass_count += 1
            elif val != "quit":
                pass_count = 0
                val = stringToIndex(val)
                if val >= 0 and val < 81 and mask[val] == 1: # Check if the move is legal
                    board = Move(board, val, color)[1] # Make the move
                    color *= -1
                else:
                    print(mask[val])
                    print("Illegal move!", end = "\n")
    if pass_count >= 2:
        print("Players agreed to end the game.", end = "\n")
main()
```

    #  A B C D E F G H I
    1  . . . . . . . . .
    2  . . . . . . . . .
    3  . . . . . . . . .
    4  . . . . . . . . .
    5  . . . . . . . . .
    6  . . . . . . . . .
    7  . . . . . . . . .
    8  . . . . . . . . .
    9  . . . . . . . . .

    Black's Move:  i8

    #  A B C D E F G H I
    1  . . . . . . . . .
    2  . . . . . . . . .
    3  . . . . . . . . .
    4  . . . . . . . . .
    5  . . . . . . . . .
    6  . . . . . . . . .
    7  . . . . . . . . .
    8  . . . . . . . . @
    9  . . . . . . . . .

    White's Move:  a1

    #  A B C D E F G H I
    1  O . . . . . . . .
    2  . . . . . . . . .
    3  . . . . . . . . .
    4  . . . . . . . . .
    5  . . . . . . . . .
    6  . . . . . . . . .
    7  . . . . . . . . .
    8  . . . . . . . . @
    9  . . . . . . . . .

    Black's Move:  h9

    #  A B C D E F G H I
    1  O . . . . . . . .
    2  . . . . . . . . .
    3  . . . . . . . . .
    4  . . . . . . . . .
    5  . . . . . . . . .
    6  . . . . . . . . .
    7  . . . . . . . . .
    8  . . . . . . . . @
    9  . . . . . . . @ .

    White's Move:  h8

    #  A B C D E F G H I
    1  O . . . . . . . .
    2  . . . . . . . . .
    3  . . . . . . . . .
    4  . . . . . . . . .
    5  . . . . . . . . .
    6  . . . . . . . . .
    7  . . . . . . . . .
    8  . . . . . . . O @
    9  . . . . . . . @ .

    Black's Move:  a2

    #  A B C D E F G H I
    1  O . . . . . . . .
    2  @ . . . . . . . .
    3  . . . . . . . . .
    4  . . . . . . . . .
    5  . . . . . . . . .
    6  . . . . . . . . .
    7  . . . . . . . . .
    8  . . . . . . . O @
    9  . . . . . . . @ .

    White's Move:  i7

    #  A B C D E F G H I
    1  O . . . . . . . .
    2  @ . . . . . . . .
    3  . . . . . . . . .
    4  . . . . . . . . .
    5  . . . . . . . . .
    6  . . . . . . . . .
    7  . . . . . . . . O
    8  . . . . . . . O @
    9  . . . . . . . @ .

    Black's Move:  b1

    #  A B C D E F G H I
    1  . @ . . . . . . .
    2  @ . . . . . . . .
    3  . . . . . . . . .
    4  . . . . . . . . .
    5  . . . . . . . . .
    6  . . . . . . . . .
    7  . . . . . . . . O
    8  . . . . . . . O @
    9  . . . . . . . @ .

    White's Move:  i9

    #  A B C D E F G H I
    1  . @ . . . . . . .
    2  @ . . . . . . . .
    3  . . . . . . . . .
    4  . . . . . . . . .
    5  . . . . . . . . .
    6  . . . . . . . . .
    7  . . . . . . . . O
    8  . . . . . . . O .
    9  . . . . . . . @ O

```python

```


##**Chapter 10 – Introduction to Neural Networks with Keras**

 Photo Credits: Galaxy's Edge by Rod Long licensed under the Unsplash License

> *The defnition of AI is a highly contested concept. It often refers to technologies that demonstrate levels of independent intelligence from humans. By its very
defnition, it is an intelligence that is differentiated from natural intelligence; it is
a constructed, artificial, or machine intelligence.*

$\quad$Ryan, M. (2020). In AI we trust: ethics, artificial intelligence, and reliability. Science and Engineering Ethics, 26(5), 2749-2767.

This notebook will be used in the lab session for week 5 of the course, covers Chapters 10 of Géron, and builds on the [notebooks made available on _Github_](https://github.com/ageron/handson-ml2).

Need a reminder of last week's labs? Click [_here_](https://colab.research.google.com/github/tbeucler/2022_ML_Earth_Env_Sci/blob/main/Lab_Notebooks/Week_4_Dimensionality_Reduction_Clustering.ipynb) to go to notebook for week 4 of the course.

## Notebook Setup

First, let's import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures. We also check that Python 3.5 or later is installed (although Python 2.x may work, it is deprecated so we strongly recommend you use Python 3 instead), as well as Scikit-Learn ≥0.20 and TensorFlow ≥2.0.

```python
# Python ≥3.5 is required
import sys
assert sys.version_info >= (3, 5)

# Scikit-Learn ≥0.20 is required
import sklearn
assert sklearn.__version__ >= "0.20"

try:
    # %tensorflow_version only exists in Colab.
    %tensorflow_version 2.x
except Exception:
    pass

# TensorFlow ≥2.0 is required
import tensorflow as tf
assert tf.__version__ >= "2.0"

# Common imports
import numpy as np
import os

# to make this notebook's output stable across runs
rnd_seed = 42
rnd_gen = np.random.default_rng(rnd_seed)

# To plot pretty figures
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
mpl.rc('axes', labelsize=14)
mpl.rc('xtick', labelsize=12)
mpl.rc('ytick', labelsize=12)

# Where to save the figures
PROJECT_ROOT_DIR = "."
CHAPTER_ID = "ann"
IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, "images", CHAPTER_ID)
os.makedirs(IMAGES_PATH, exist_ok=True)

def save_fig(fig_id, tight_layout=True, fig_extension="png", resolution=300):
    path = os.path.join(IMAGES_PATH, fig_id + "." + fig_extension)
    print("Saving figure", fig_id)
    if tight_layout:
        plt.tight_layout()
    plt.savefig(path, format=fig_extension, dpi=resolution)

# Initialize the run_index
run_index = None

# Loading Tensorboard
%load_ext tensorboard
```

    The tensorboard extension is already loaded. To reload it, use:
      %reload_ext tensorboard

## Data Setup

Today, we'll once again be working on the MNIST handwritten digit database - we're becoming experts in typography! ✍

Let's begin by importing the dataset from the keras dataset library.

### **Q1) Load the MNIST dataset from Keras. Divide it into a training, validation, and test dataset**

*Hint 1: To access the Keras library, you can either reimport keras (e.g., `import tensorflow.keras as keras`), or you can access it from the instance of tensorflow we imported during setup (i.e., using `tf.keras`)*

*Hint 2: [Here is the documentation](https://keras.io/api/datasets/mnist/) for the Keras implementation of the MNIST dataset*

*Hint 3: If you use the `mnist.load_data()` method, what will be returned will be a set of tuples: (training_data, testing_data), where training_data and testing_data are tuples of inputs and labels (X, y)*

*Hint 4: You can break down the training dataset from the `.load()` method into a training and validation dataset. Since the full training dataset includes 60 000 samples, try using 50 000 samples as training data and 10 000 samples as validation data.*

```python
# Load the keras dataset data
((X_train_full,y_train_full),(X_test,y_test)) = tf.keras.datasets.mnist.load_data()

```

```python
from sklearn.model_selection import train_test_split

X_train, X_valid, y_train, y_valid = train_test_split(X_train_full, y_train_full,
                                                      test_size=10000,
                                                      train_size=50000)
```

What does our data look like? Let's get an idea of the values and figure out what kind of preprocessing we should do before training our neural network.

### **Q2) Print the shape of the training, validation, and test sets. Then, print the maximum and minimum input values.**

*Hint 1: You loaded the data as numpy arrays. Thus, you can rely on the built-in methods for finding the shape and min/max values.*

*Hint 2: Click for the documentation on [`ndarray.max()`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.max.html), [`ndarray.min()`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.min.html), and [`ndarray.shape`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.shape.html)*

```python
print('X_train shape :',X_train.shape,'\ny_train shape :',y_train.shape,
      '\nX_test shape :',X_test.shape,'\ny_test shape :',y_test.shape,
      '\nX_valid shape :',X_valid.shape,'\ny_valid shape :',y_valid.shape)
print('\n')
print('X_train min :',X_train.min(),'  X_train max :',X_train.max(),
      '\ny_train min :',y_train.min(),'  y_train max :',y_train.max(),
      '\nX_test min :',X_test.min(),'  X_test max :',X_train.max(),
      '\ny_test min :',y_test.min(),'  y_test max :',y_train.max(),
      '\nX_valid min :',X_valid.min(),'  X_valid max :',X_train.max(),
      '\ny_valid min :',y_valid.min(),'  y_valid max :',y_train.max())
```

    X_train shape : (50000, 28, 28)
    y_train shape : (50000,)
    X_test shape : (10000, 28, 28)
    y_test shape : (10000,)
    X_valid shape : (10000, 28, 28)
    y_valid shape : (10000,)

    X_train min : 0   X_train max : 255
    y_train min : 0   y_train max : 9
    X_test min : 0   X_test max : 255
    y_test min : 0   y_test max : 9
    X_valid min : 0   X_valid max : 255
    y_valid min : 0   y_valid max : 9

If you used the same train/validation split as we did, you should have 50k samples in the training set, 10k in the validation set, and 10k in the test set.

Since the data represents grayscale image values, data values should vary between 0 and 255; Normalize the data by dividing it by 255.
### **Q3) Normalize the input data for the training, validation, and testing sets**

*Hint 1: The datasets are stored as simple numpy arrays, so you can perform arithmetic operations on them!*

```python
X_train = X_train/255
X_test = X_test/255
X_valid = X_valid/255
```

We now have the normalized training, validation, and testing data that we'll use to train our neural network. Before moving on, it might be worth it to make a small visualiation of samples in our data to ensure that everything worked out correctly.

### **Q4) Write a function that:

1) Takes in an input dataset and its labels, a number of rows, and a number of columns
 2) Prints out a random n_rows by n_columns sample of images with their labels
**

*Hint 1: You can use the `rnd_seed.integers()` generator to generate a set of integers between 0 and the number of samples, with a size of (rows,columns). [Here is some documentation that can help](https://numpy.org/doc/stable/reference/random/generator.html#simple-random-data). It's best practice to take in the random generator as an argument for your function.*

*Hint 2: You can use matplotlib's `fig, axes = plt.subplots()` to make a grid of axes and call the `imshow()` method on each ax in order to plot the digit. It is recommended that you use the `cmap='binary'` argument in imshow to print the digits in black and white*. Click on the links for the documentation to [`plt.sublopts()`](https://matplotlib.org/3.5.0/api/_as_gen/matplotlib.pyplot.subplots.html), [`plt.imshow()`](https://matplotlib.org/3.5.0/api/_as_gen/matplotlib.pyplot.imshow.html), and [the colormaps (i.e., cmap values)](https://matplotlib.org/stable/gallery/color/colormap_reference.html) available in matplotlib.

*Hint 3: You can iterate using numpy `ndenumerate()` method, which will return the n-dimensional index of the array and the element located there. This will be useful when iterating through the indices you generated and plotting the corresponding digit and label*

*Hint 4: Feeling stuck? [Here is sample code for a way this function can be implemented.](https://unils-my.sharepoint.com/:t:/g/personal/tom_beucler_unil_ch/ERT6Sl_NHp5Nt2YjzkDPzLwB0RQ7_rVAl3RDx4BfHI047g?download=1)*

```python
def sample_plotter(ds, lab, rows, cols):

    # Create a set of indices to access the sample images/labels
    ind = rnd_gen.integers(0,ds.shape[0], size=(rows, cols))
    # Create a figure with n_rows and n_columns
    fig, ax = plt.subplots(rows, cols)
    # Plot each selected digit
    for idx, pos in np.ndenumerate(ind):
      ax[idx].imshow(ds[pos], cmap='binary')
      ax[idx].axis('off')
      ax[idx].title.set_text(lab[pos])
    return
```

Now that our function is defined, let's go ahead and print out a 4 row by 8 column sample from each dataset.

### **Q5) Grab a 4x8 sample of digits from each dataset and print out the image and labels**

```python
sample_plotter(X_train, y_train, 4, 8)

```

We're now ready to start developing our neural network. The first thing that we want to do is figure out an appropriate learning rate for our model - after all, we want to choose one that converges to a solution *and* is the least computationally expensive possible.

Let's start by setting up a keras *callback* [(click here for the documentation)](https://keras.io/api/callbacks/), a type of object that will allow us to change the learning rate after every iteration (i.e., after every batch of data). We will set up what is called an exponential learning rate (that is, the learning will increase by a factor of $k$ after each iteration). Expressed mathematically,
\begin{align}
\eta_{\scriptsize{t}} = \eta_{\scriptsize{0}} \, \cdot \, k^{\scriptsize{t}}
\end{align}
where $t$ is the current iteration.

As a reminder, an epoch is an iteration through the entire training dataset, while a batch is an iteration through a predefined subset of . It's important to make this distinction, as ML algorithms are often trained in batches when dealing with large datasets, and we *normally* do not want to change the learning rate in between batches during model training. However, we will do so during this evaluation phase in order to determine an adequate learning rate.

We will therefore set a callback that will do two things after the end of each batch:

> 1) Keep a track of the losses
 2) Adjust the learning rate by multiplying it by a predefined factor

### **Q5) Set up an *Exponential_Learning_Rate* callback that, after each batch, logs the value of the loss function and learning rate, and then multiplies the learning rate by a factor of $k$**

*Hint 1: Multiple backend options are available with Keras. We will be using tensorflow, but the code is thought to be written in such a way that a different backend **could** be used. `tf.keras.backend` has a `.backend()` method that allows you to check what backend is being used.*

*Hint 2: You should extend the `tf.keras.callbacks.Callback` class. (Confused about extending classes? [Here is a question on stack overflow](https://stackoverflow.com/questions/15526858/how-to-extend-a-class-in-python) that could provide some context) *

*Hint 3: The ExponentialLearningRate callback we will implement will need to take in the $k$ factor durint its initialization ([here's a quick overview](https://stackoverflow.com/questions/625083/what-do-init-and-self-do-in-python) on the __init__ contructor method and **self** arguments in classes, with a focus on python.). You will also need to save an empty list as an attribute for both the losses and the learning rates*

*Hint 4: Keras model optimizers have an attribute where the learning rate is stored: `model.optimizer.learning_rate`. In order to read the value, you will have to use the keras backend's `.get_value()` method with the model's learning rate as an argument*

*Hint 5: the on_train_batch_end method pass the `logs` argument into the function. You can access the loss function by using `logs['loss']`*

*Hint 6: In order to set the learning rate to a different value, you will have to depend on the keras backend's `.set_value()` method. This method takes in two arguments: the first is the value that will be set (e.g., the learning rate in the model's optimizer) and the value that it will be set to (e.g., the learning rate multiplied by the k factor).*

*Hint 7: Unlike in other documentations we've seen, `backend.get_value()` and `backend.set_value()` don't yet have their own page. However, [here is the link](https://www.tensorflow.org/guide/keras/custom_callback#learning_rate_scheduling) to an example where both methods are used in a learning rate scheduler.*

```python
# We'll start by making it easier to access the keras backend. See hint #1 for
# more details
K = tf.keras.backend

# Use the .backend() method to determine what backend we're running
K.backend()
```

    'tensorflow'

```python
# the code cell above!

class ExponentialLearningRate(tf.keras.callbacks.Callback): #define the ExponentialLearningRate class
    # Start
    def __init__(self, factor):
        self.factor = factor # set the factor
        self.losses = [] # initialize the losses list
        self.LR = [] # initialize the learning rates list

    def on_batch_end(self, batch, logs):
        # Add the value of the learning rate to the list
        self.LR.append(K.get_value(self.model.optimizer.learning_rate))

        # Add the value of the loss
        self.losses.append(logs['loss'])

        # Set the value of the
        K.set_value(self.model.optimizer.learning_rate, self.model.optimizer.learning_rate * self.factor)
```

Now that we've defined out callback, we can go ahead and start thinking about our neural network. For consistency's sake, let's start by clearing the Keras backend and setting our random state.

```python
# Run this cell
K.clear_session()
np.random.seed(rnd_seed)
tf.random.set_seed(rnd_seed)
```

Let's make a simple neural network model using Keras. For this, we will rely on a [*Sequential model*](https://keras.io/guides/sequential_model/), since we will want all of the inputs of one layer to be fed into the next layer. We recommend using the architecture described in the diagram below, but feel free to define your own architecture!

### **Q6) Write a sequential Keras model that will predict the digit class.**

*Hint 1: You can add the layers in the sequential model when initializing the model. It expects the layers in a list. Alternatively, you can add them one by one using the model's `.add()` method. [Check out the documentation here](https://keras.io/guides/sequential_model/#creating-a-sequential-model).*

*Hint 2: The input images should be flattened before feeding them into any densely connected layers. [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Flatten) for the flatten layer.*

*Hint 3: You want to use simple, densely connected layers for this exercise. [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense) for the dense layer.*

*Hint 4: Using a dense layer with the number of units set to the number of classes (e.g., the number of different digits in the MNIST dataset: 10) using a softmax activation unit can be interpreted as a probability of the input belonging to a given class. [Here is the documentation](https://keras.io/api/layers/activations/#softmax-function) for the softmax activation function in Keras*

```python
from keras.models import Sequential

model = tf.keras.models.Sequential([ # call the keras sequential model class
                            tf.keras.layers.Flatten(),  # 1st Layer
                            tf.keras.layers.Dense(300,activation='relu'),  # 2nd Layer
                            tf.keras.layers.Dense(100,activation='relu'),  # 3rd Layer
                            tf.keras.layers.Dense(10,activation='softmax')]) # 4th Layer
```

Now that we have a model defined, we need to run its `.compile()' method, in which we will give the model the following hyper-parameters:
> 1) Loss function will be set to sparse categorical cross entropy
 2) The optimizer will be set to Stochastic Gradient Descent with a learning rate of 1e-3
 3) The model metrics will include the accuracy score

### **Q7) Compile the model with the given hyperparameters (i.e., loss function, optimizer, and metrics) and instantiate the callback we defined previously using a $k$ factor of 1.005 (i.e., a 0.5% increase in learning rate per batch)**

*Hint 1: [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/sparse_categorical_crossentropy) for the sparse categorical cross entropy loss function in keras. You can simply reference the function using `loss='sparse_categorical_crossentropy'` when compiling.*

*Hint 2: [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/SGD) for the Stochastic Gradient Descent optimizer in keras*

*Hint 3: [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/Accuracy) for the accuracy score implementation in keras. Like with the sparse_categorical_cross_entropy loss, you can reference the accuracy score in the metrics list, e.g. by setting `metrics=['accuracy']` when compiling.*

```python
model.compile(loss='sparse_categorical_crossentropy', # Set the loss function
              optimizer=tf.keras.optimizers.SGD(learning_rate=0.01), # Set the optimizer and learning rate
              metrics=['accuracy']) # Set the metrics
```

```python
exponential_lr_callback = ExponentialLearningRate(factor=1.05)

```

Let's go ahead and train the compiled model for a single epoch.

### **Q8) Fit the model for a single epoch, using the exponential learning rate callback we defined in the previous code cell. Then, plot the Loss vs Learning rate.**

*Hint 1: Just like in scikit-learn, the keras model includes a `.fit()` method to train the algorithm! [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit).*

*Hint 2: After training, you can access the recorded losses and corresponding learning rates using the attributes we defined when we defined the class in Q5!*

```python
history = model.fit(X_train, # set the training inputs
                    y_train, # set the training labels
                    epochs=1, #set the number of epochs
                    validation_data=(X_valid, y_valid), # set validation input/labels
                    callbacks=[exponential_lr_callback]) # Set the callback
```

    1563/1563 [==============================] - 10s 6ms/step - loss: nan - accuracy: 0.1208 - val_loss: nan - val_accuracy: 0.0984

```python
# Plotting
fig, ax = plt.subplots()

ax.plot(exponential_lr_callback.LR, # learning rates
        exponential_lr_callback.losses) # losses

# Define a tuple with (min_learning_rate, max_learn_rate)
x_limits = ( min(exponential_lr_callback.LR), 10 )

# Set the xscale to logarithmic
ax.set_xscale('log')

# Draw a horizontal line at the minimum loss value
ax.hlines(min(exponential_lr_callback.losses), #Find the minimum loss value to draw a horizontal line
          *x_limits, # the star unpacks x_limits to the expected num of args
          'g')

# Set the limits for drawing the curves
ax.set_xlim(x_limits)
ax.set_ylim(0,exponential_lr_callback.losses[0]+0.1) # use the initial loss as the top y boundary

# Display gridlines to see better
ax.grid(which='both')

ax.set_xlabel("Learning rate")
ax.set_ylabel("Loss")

```

If you used the architecture we defined above with the learning rate we defined above, you should produce a graph that looks like this:

In this graph, you can see that the loss reaches a minimum at around 6e-1 and then begins to shoot up violently. Let's avoid that by using half that value (e.g., 3e-1).

If you have a different curve, try setting your learning rate to half of the learning rate with the minimum loss! 😃

Now that we have an idea of what the learning rate should be, let's go ahead and start from scratch once more.

```python
# Run this cell - let's go back to a clean slate!
K.clear_session()
np.random.seed(rnd_seed)
tf.random.set_seed(rnd_seed)
```

We also want to instantiate the model again - the weights in our current model are quite bad and if we use it as is it won't be able to learn since the weights are too far away from the solution. There are other ways to do this, but since our model is quite simple it's worth it to just redefine and recompile it.

### **Q9) Redefine and re-compile the model with the learning rate you found in Q8.**

```python

# redefine the model
model = tf.keras.models.Sequential([ # call the sequential model class
    tf.keras.layers.Flatten(), # flatten the data
    tf.keras.layers.Dense(300,activation='relu'), # densely connected ReLU layer, 300 units
    tf.keras.layers.Dense(100,activation='relu'), # densely connected ReLU layer, 100 units
    tf.keras.layers.Dense(10,activation='softmax')]) # densely connected Softmax layer, 10 units

```

```python
model.compile(loss='sparse_categorical_crossentropy', # Set the loss function
              optimizer=tf.keras.optimizers.SGD(learning_rate=1e-3), # Set the optimizer and learning rate
              metrics=['accuracy']) # Set the metrics
```

We're now going to set up a saving directory in case you want to try running the model with different learning rates or other hyper-parameters!

```python
#Change this number and rerun this cell whenever you want to change runs
run_index = 1

run_logdir = os.path.join(os.curdir, "my_mnist_logs", "run_{:03d}".format(run_index))

print(run_logdir)
```

    ./my_mnist_logs/run_001

We'll also set up some additional callbacks.
> 1) An early stopping callback ([documentation here](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping)). This callback will stop the training if no improvement is found after a `patience` number of epochs.
 2) A model checkpoint callback ([documentation here](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint)). This callback will ensure that only the best version of the model is kept (in case your model's performance reaches a maximum and then deteriorates after a certain number of epochs)
 3) A tensorboard callback ([documentation here](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/TensorBoard)). This callback will enable using Tensorboard to visualize learning curves, metrics, etc. Handy 🙌!

```python
early_stopping_cb = tf.keras.callbacks.EarlyStopping(patience=20)
checkpoint_cb = tf.keras.callbacks.ModelCheckpoint("my_mnist_model.h5", save_best_only=True)
tensorboard_cb = tf.keras.callbacks.TensorBoard(run_logdir)
```

Let's go ahead and fit the model again!

### **Q10) Fit the updated model for 100 epochs**

```python
history = model.fit(X_train, # inputs
                    y_train, # labels
                    epochs=100, #epochs
                    validation_data=(X_valid, y_valid),
                    callbacks=[checkpoint_cb, early_stopping_cb, tensorboard_cb])
```

Finally, we need to evaluate the performance of our model. Go ahead and try it out on the test set!

### **Q11) Evaluate the model on the test set.**

*Hint 1: Keras models include an `evaluate()` method that takes in the test set inputs/labels. [Here is the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/Model#evaluate).*

```python
# Rollback to best model, which was saved by the callback
model = tf.keras.models.load_model("my_mnist_model.h5") # rollback to best model

# Evaluate the model
model.evaluate(X_test, y_test)
```

Finally, we can use tensorboard to check out our model's performance! Note that the tensorboard extension was loaded in the notebook setup cell.

```python
%tensorboard --logdir=./my_mnist_logs --port=6006
```

An enthusiastic (albeit somewhat sick 😷) TA noted that during the development of the notebook the accuracy reached on the test dataset was 97.84%. Additionally, the tensorboard curves from the test run is given below: