# Fishbone-Moncrief Initial Data

## Author: Zach Etienne
### Formatting improvements courtesy Brandon Clark

[comment]: <> (Abstract: TODO)

[comment]: <> (Notebook Status and Validation Notes: TODO)

### NRPy+ Source Code for this module: [FishboneMoncriefID/FishboneMoncriefID.py](../edit/FishboneMoncriefID/FishboneMoncriefID.py)

## Introduction:
This goal of this module will be to construct Fishbone-Moncrief initial data for GRMHD simulations in a format suitable for the Einstein Toolkit (ETK). We will be using the equations as derived in [the original paper](http://articles.adsabs.harvard.edu/cgi-bin/nph-iarticle_query?1976ApJ...207..962F&data_type=PDF_HIGH&whole_paper=YES&type=PRINTER&filetype=.pdf), which will hereafter be called "***the FM paper***". Since we want to use this with the ETK, our final result will be in Cartesian coordinates. The natural coordinate system for these data is spherical, however, so we will use [reference_metric.py](../edit/reference_metric.py) ([**Tutorial**](Tutorial-Reference_Metric.ipynb)) to help with the coordinate transformation.

This notebook documents the equations in the NRPy+ module [FishboneMoncrief.py](../edit/FishboneMoncriefID/FishboneMoncriefID.py).  Then, we will build an Einstein Toolkit [thorn](Tutorial-ETK_thorn-FishboneMoncriefID.ipynb) to set this initial data.

# Table of Contents
$$\label{toc}$$

This notebook is organized as follows

1. [Step 1](#initializenrpy): Initialize core Python/NRPy+ modules
1. [Step 2](#fishbonemoncrief): Implementing Fishbone-Moncrief initial data within NRPy+
    1. [Step 2.a](#registergridfunctions): Register within NRPy+ needed gridfunctions and initial parameters
    1. [Step 2.b](#l_of_r): Specific angular momentum $l(r)$
    1. [Step 2.c](#enthalpy): Specific enthalpy $h$
    1. [Step 2.d](#pressure_density): Pressure and density, from the specific enthalpy
    1. [Step 2.e](#covariant_velocity): Nonzero covariant velocity components $u_\mu$
    1. [Step 2.f](#inverse_bl_metric): Inverse metric $g^{\mu\nu}$ for the black hole in Boyer-Lindquist coordinates
    1. [Step 2.g](#xform_to_ks): Transform components of four-veloicty $u^\mu$ to Kerr-Schild
    1. [Step 2.h](#ks_metric): Define Kerr-Schild metric $g_{\mu\nu}$ and extrinsic curvature $K_{ij}$
    1. [Step 2.i](#magnetic_field): Seed poloidal magnetic field $B^i$
    1. [Step 2.j](#adm_metric): Set the ADM quantities $\alpha$, $\beta^i$, and $\gamma_{ij}$ from the spacetime metric $g_{\mu\nu}$
    1. [Step 2.k](#magnetic_field_comoving_frame): Set the magnetic field components in the comoving frame $b^\mu$, and $b^2$, which is twice the magnetic pressure
    1. [Step 2.l](#lorentz_fac_valencia): Lorentz factor $\Gamma = \alpha u^0$ and Valencia 3-velocity $v^i_{(n)}$
1. [Step 3](#output_to_c): Output SymPy expressions to C code, using NRPy+

1. [Step 4](#code_validation): Code Validation against Code Validation against `FishboneMoncriefID.FishboneMoncriefID` NRPy+ module NRPy+ module
1. [Step 5](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Step 1: Initialize core Python/NRPy+ modules \[Back to [top](#toc)\]
$$\label{initializenrpy}$$

We begin by importing the packages and NRPy+ modules that we will need. We will also set some of the most commonly used parameters.

```python
# Step 1a: Import needed NRPy+ core modules:
import NRPy_param_funcs as par
import indexedexp as ixp
import grid as gri
import finite_difference as fin
from outputC import *
import loop

import reference_metric as rfm
par.set_parval_from_str("reference_metric::CoordSystem","Cartesian")
rfm.reference_metric()
#Set the spatial dimension parameter to 3.
par.set_parval_from_str("grid::DIM", 3)
DIM = par.parval_from_str("grid::DIM")

thismodule = "FishboneMoncriefID"
```

# Step 2: The Fishbone-Moncrief Initial Data Prescription \[Back to [top](#toc)\]
$$\label{fishbonemoncrief}$$

With NRPy's most important functions now available to us, we can start to set up the rest of the tools we will need to build the initial data.

## Step 2.a: Register within NRPy+ needed gridfunctions and initial parameters \[Back to [top](#toc)\]
$$\label{registergridfunctions}$$

We will now register the gridfunctions we expect to use. Critically, we register the physical metric and extrinsic curvature tensors.

```python
gPhys4UU = ixp.register_gridfunctions_for_single_rank2("AUX","gPhys4UU", "sym01", DIM=4)
KDD = ixp.register_gridfunctions_for_single_rank2("EVOL","KDD", "sym01")

# Variables needed for initial data given in spherical basis
r, th, ph = gri.register_gridfunctions("AUX",["r","th","ph"])

r_in,r_at_max_density,a,M = par.Cparameters("REAL",thismodule,
                                            ["r_in","r_at_max_density",    "a","M"],
                                            [   6.0,              12.0, 0.9375,1.0])

kappa,gamma = par.Cparameters("REAL",thismodule,["kappa","gamma"], [1.0e-3, 4.0/3.0])

LorentzFactor = gri.register_gridfunctions("AUX","LorentzFactor")
```

## Step 2.b: Specific angular momentum $l(r)$ \[Back to [top](#toc)\]
$$\label{l_of_r}$$

Now, we can begin actually building the ID equations. We will start with the value of the angular momentum $l$ at the position $r \equiv$`r_at_max_density` where the density is at a maximum, as in equation 3.8 of the FM paper:
\begin{align}
l(r) &= \pm \left( \frac{M}{r^3} \right) ^{1/2}
    \left[ \frac{r^4+r^2a^2-2Mra^2 \mp a(Mr)^{1/2}(r^2-a^2)}
    {r^2 -3Mr \pm 2a(Mr)^{1/2}} \right].
\end{align}

```python
def calculate_l_at_r(r):
    l  = sp.sqrt(M/r**3) * (r**4 + r**2*a**2 - 2*M*r*a**2 - a*sp.sqrt(M*r)*(r**2-a**2))
    l /= r**2 - 3*M*r + 2*a*sp.sqrt(M*r)
    return l

# First compute angular momentum at r_at_max_density, TAKING POSITIVE ROOT. This way disk is co-rotating with black hole
# Eq 3.8:
l = calculate_l_at_r(r_at_max_density)
```

## Step 2.c: Specific enthalpy $h$ \[Back to [top](#toc)\]
$$\label{enthalpy}$$

Next, we will follow equation 3.6 of the FM paper to compute the enthalpy $h$ by first finding its logarithm $\ln h$. Fortunately, we can make this process quite a bit simpler by first identifying the common subexpressions. Let
\begin{align}
\Delta &= r^2 - 2Mr + a^2 \\
\Sigma &= r^2 + a^2 \cos^2 (\theta) \\
A &= (r^2+a^2)^2 - \Delta a^2 \sin^2(\theta);
\end{align}
furthermore, let
\begin{align}
\text{tmp3} &= \sqrt{\frac{1 + 4 l^2 \Sigma^2 \Delta}{A \sin^2 (\theta)}}. \\
\end{align}
(These terms reflect the radially-independent part of the log of the enthalpy, `ln_h_const`.)
So,
$$
{\rm ln\_h\_const} = \frac{1}{2} * \log \left( \frac{1+\text{tmp3}}{\Sigma \Delta/A} \right) - \frac{1}{2} \text{tmp3} - \frac{2aMrl}{A}
$$

```python
# Eq 3.6:
# First compute the radially-independent part of the log of the enthalpy, ln_h_const
Delta = r**2 - 2*M*r + a**2
Sigma = r**2 + a**2*sp.cos(th)**2
A = (r**2 + a**2)**2 - Delta*a**2*sp.sin(th)**2

# Next compute the radially-dependent part of log(enthalpy), ln_h
tmp3 = sp.sqrt(1 + 4*l**2*Sigma**2*Delta/(A*sp.sin(th))**2)
# Term 1 of Eq 3.6
ln_h  = sp.Rational(1,2)*sp.log( ( 1 + tmp3) / (Sigma*Delta/A))
# Term 2 of Eq 3.6
ln_h -= sp.Rational(1,2)*tmp3
# Term 3 of Eq 3.6
ln_h -= 2*a*M*r*l/A

```

Additionally, let
\begin{align}
\Delta_{\rm in} &= r_{\rm in}^2 - 2Mr_{\rm in} + a^2 \\
\Sigma_{\rm in} &= r_{\rm in}^2 + a^2 \cos^2 (\pi/2) \\
A_{\rm in} &= (r_{\rm in}^2+a^2)^2 - \Delta_{\rm in} a^2 \sin^2(\pi/2)
\end{align}
and
\begin{align}
\text{tmp3in} &= \sqrt{\frac{1 + 4 l^2 \Sigma_{\rm in}^2 \Delta_{\rm in}}{A_{\rm in} \sin^2 (\theta)}}, \\
\end{align}
corresponding to the radially Independent part of log(enthalpy), $\ln h$:
\begin{align}
{\rm mln\_h\_in} = -\frac{1}{2} * \log \left( \frac{1+\text{tmp3in}}{\Sigma_{\rm in} \Delta_{\rm in}/A_{\rm in}} \right) + \frac{1}{2} \text{tmp3in} + \frac{2aMr_{\rm in}l}{A_{\rm in}}. \\
\end{align}
(Note that there is some typo in the expression for these terms given in Eq 3.6, so we opt to just evaluate negative of the first three terms at r=`r_in` and th=pi/2 (the integration constant), as described in the text below Eq. 3.6.)

So, then, we exponentiate:
\begin{align}
\text{hm1} \equiv h-1 &= e^{{\rm ln\_h}+{\rm mln\_h\_in}}-1. \\
\end{align}

```python
# Next compute the radially-INdependent part of log(enthalpy), ln_h
# Note that there is some typo in the expression for these terms given in Eq 3.6, so we opt to just evaluate
#   negative of the first three terms at r=r_in and th=pi/2 (the integration constant), as described in
#   the text below Eq. 3.6, basically just copying the above lines of code.
# Delin = Delta_in ; Sigin = Sigma_in ; Ain = A_in .
Delin = r_in**2 - 2*M*r_in + a**2
Sigin = r_in**2 + a**2*sp.cos(sp.pi/2)**2
Ain   = (r_in**2 + a**2)**2 - Delin*a**2*sp.sin(sp.pi/2)**2

tmp3in = sp.sqrt(1 + 4*l**2*Sigin**2*Delin/(Ain*sp.sin(sp.pi/2))**2)
# Term 4 of Eq 3.6
mln_h_in  = -sp.Rational(1,2)*sp.log( ( 1 + tmp3in) / (Sigin*Delin/Ain))
# Term 5 of Eq 3.6
mln_h_in += sp.Rational(1,2)*tmp3in
# Term 6 of Eq 3.6
mln_h_in += 2*a*M*r_in*l/Ain

hm1 = sp.exp(ln_h + mln_h_in) - 1
```

## Step 2.d: Pressure and density, from the specific enthalpy \[Back to [top](#toc)\]
$$\label{pressure_density}$$

Python 3.4 + SymPy 1.0.0 has a serious problem taking the power here; it hangs forever, so instead we use the identity $x^{1/y} = \exp(\frac{1}{y} * \log(x))$. Thus, our expression for density becomes (in Python 2.7 + SymPy 0.7.4.1):

\begin{align}
\rho_0 &= \left( \frac{(h-1)(\gamma-1)}{\kappa \gamma} \right)^{1/(\gamma-1)} \\
&= \exp \left[ {\frac{1}{\gamma-1} \log \left( \frac{(h-1)(\gamma-1)}{\kappa \gamma}\right)} \right]
\end{align}

Additionally, the pressure $P_0 = \kappa \rho_0^\gamma$

```python
rho_initial,Pressure_initial = gri.register_gridfunctions("AUX",["rho_initial","Pressure_initial"])

# Python 3.4 + sympy 1.0.0 has a serious problem taking the power here, hangs forever.
# so instead we use the identity x^{1/y} = exp( [1/y] * log(x) )
# Original expression (works with Python 2.7 + sympy 0.7.4.1):
# rho_initial = ( hm1*(gamma-1)/(kappa*gamma) )**(1/(gamma - 1))
# New expression (workaround):
rho_initial = sp.exp( (1/(gamma-1)) * sp.log( hm1*(gamma-1)/(kappa*gamma) ))
Pressure_initial = kappa * rho_initial**gamma
```

## Step 2.e: Nonzero covariant velocity components $u_\mu$ \[Back to [top](#toc)\]
$$\label{covariant_velocity}$$

We now want to compute eq 3.3; we will start by finding $e^{-2 \chi}$ in Boyer-Lindquist (BL) coordinates. By eq 2.16, $\chi = \psi - \nu$, so, by eqs. 3.5,
\begin{align}
e^{2 \nu} &= \frac{\Sigma \Delta}{A} \\
e^{2 \psi} &= \frac{A \sin^2 \theta}{\Sigma} \\
e^{-2 \chi} &= e^{2 \nu} / e^{2 \psi} = e^{2(\nu - \psi)}.
\end{align}

Next, we will calculate the 4-velocity $u_i$ of the fluid disk in BL coordinates. We start with eqs. 3.3 and 2.13, finding
\begin{align}
u_{(r)} = u_{(\theta)} &= 0 \\
u_{(\phi)} &= \sqrt{-1+ \frac{1}{2}\sqrt{1 + 4l^2e^{-2 \chi}}} \\
u_{(t)} &= - \sqrt{1 + u_{(\phi)}^2}.
\end{align}

Given that $\omega = 2aMr/A$, we then find that, in BL coordinates,
\begin{align}
u_r = u_{\theta} &= 0 \\
u_{\phi} &= u_{(\phi)} \sqrt{e^{2 \psi}} \\
u_t &= u_{(t)} \sqrt{e^{2 \nu}} - \omega u_{\phi},
\end{align}
using eq. 2.13 to get the last relation.

```python
# Eq 3.3: First compute exp(-2 chi), assuming Boyer-Lindquist coordinates
#    Eq 2.16: chi = psi - nu, so
#    Eq 3.5 -> exp(-2 chi) = exp(-2 (psi - nu)) = exp(2 nu)/exp(2 psi)
exp2nu  = Sigma*Delta / A
exp2psi = A*sp.sin(th)**2 / Sigma
expm2chi = exp2nu / exp2psi

# Eq 3.3: Next compute u_(phi).
u_pphip = sp.sqrt((-1 + sp.sqrt(1 + 4*l**2*expm2chi))/2)
# Eq 2.13: Compute u_(t)
u_ptp   = -sp.sqrt(1 + u_pphip**2)

# Next compute spatial components of 4-velocity in Boyer-Lindquist coordinates:
uBL4D = ixp.zerorank1(DIM=4) # Components 1 and 2: u_r = u_theta = 0
# Eq 2.12 (typo): u_(phi) = e^(-psi) u_phi -> u_phi = e^(psi) u_(phi)
uBL4D[3] = sp.sqrt(exp2psi)*u_pphip

# Assumes Boyer-Lindquist coordinates:
omega = 2*a*M*r/A
# Eq 2.13: u_(t) = 1/sqrt(exp2nu) * ( u_t + omega*u_phi )
#     -->  u_t = u_(t) * sqrt(exp2nu) - omega*u_phi
#     -->  u_t = u_ptp * sqrt(exp2nu) - omega*uBL4D[3]
uBL4D[0] = u_ptp*sp.sqrt(exp2nu) - omega*uBL4D[3]
```

## Step 2.f: Inverse metric $g^{\mu\nu}$ for the black hole in Boyer-Lindquist coordinates \[Back to [top](#toc)\]
$$\label{inverse_bl_metric}$$

Next, we will use eq. 2.1 to find the inverse physical (as opposed to conformal) metric in BL coordinates, using the shorthands defined in eq. 3.5:
\begin{align}
g_{tt}                  &= - \frac{\Sigma \Delta}{A} + \omega^2 \sin^2 \theta \frac{A}{\Sigma} \\
g_{t \phi} = g_{\phi t} &= - \omega \sin^2 \theta \frac{A}{\Sigma} \\
g_{\phi \phi}           &= \sin^2 \theta \frac{A}{\Sigma},
\end{align}
which can be inverted to show that
\begin{align}
g^{tt}                  &= - \frac{A}{\Delta \Sigma} \\
g^{t \phi} = g^{\phi t} &= \frac{2aMr}{\Delta \Sigma} \\
g^{\phi \phi}           &= - \frac{4a^2M^2r^2}{\Delta A \Sigma} + \frac{\Sigma^2}{A \Sigma \sin^2 \theta}.
\end{align}

With this, we will now be able to raise the index on the BL $u_i$: $u^i = g^{ij} u_j$

```python
# Eq. 3.5:
# w = 2*a*M*r/A;
# Eqs. 3.5 & 2.1:
# gtt = -Sig*Del/A + w^2*Sin[th]^2*A/Sig;
# gtp = w*Sin[th]^2*A/Sig;
# gpp = Sin[th]^2*A/Sig;
# FullSimplify[Inverse[{{gtt,gtp},{gtp,gpp}}]]
gPhys4BLUU = ixp.zerorank2(DIM=4)
gPhys4BLUU[0][0] = -A/(Delta*Sigma)
# DO NOT NEED TO SET gPhys4BLUU[1][1] or gPhys4BLUU[2][2]!
gPhys4BLUU[0][3] = gPhys4BLUU[3][0] = -2*a*M*r/(Delta*Sigma)
gPhys4BLUU[3][3] = -4*a**2*M**2*r**2/(Delta*A*Sigma) + Sigma**2/(A*Sigma*sp.sin(th)**2)

uBL4U = ixp.zerorank1(DIM=4)
for i in range(4):
    for j in range(4):
        uBL4U[i] += gPhys4BLUU[i][j]*uBL4D[j]
```

## Step 2.g: Transform components of four-velocity $u^\mu$ to Kerr-Schild \[Back to [top](#toc)\]
$$\label{xform_to_ks}$$

Now, we will transform the 4-velocity from the Boyer-Lindquist to the Kerr-Schild basis. This algorithm is adapted from [HARM](https://github.com/atchekho/harmpi/blob/master/init.c). This definees the tensor `transformBLtoKS`, where the diagonal elements are $1$, and the non-zero off-diagonal elements are
\begin{align}
\text{transformBLtoKS}_{tr} &= \frac{2r}{r^2-2r+a^2} \\
\text{transformBLtoKS}_{\phi r} &=  \frac{a}{r^2-2r+a^2} \\
\end{align}

```python
# https://github.com/atchekho/harmpi/blob/master/init.c
# Next transform Boyer-Lindquist velocity to Kerr-Schild basis:
transformBLtoKS = ixp.zerorank2(DIM=4)
for i in range(4):
    transformBLtoKS[i][i] = 1
transformBLtoKS[0][1] = 2*r/(r**2 - 2*r + a*a)
transformBLtoKS[3][1] =   a/(r**2 - 2*r + a*a)
#uBL4U = ixp.declarerank1("UBL4U",DIM=4)
# After the xform below, print(uKS4U) outputs:
# [UBL4U0 + 2*UBL4U1*r/(a**2 + r**2 - 2*r), UBL4U1, UBL4U2, UBL4U1*a/(a**2 + r**2 - 2*r) + UBL4U3]
uKS4U = ixp.zerorank1(DIM=4)
for i in range(4):
    for j in range(4):
        uKS4U[i] += transformBLtoKS[i][j]*uBL4U[j]
```

## Step 2.h: Define Kerr-Schild metric $g_{\mu\nu}$ and extrinsic curvature $K_{ij}$ \[Back to [top](#toc)\]
$$\label{ks_metric}$$

We will also adopt the Kerr-Schild metric for Fishbone-Moncrief disks. Further details can be found in [Cook's Living Review](http://gravity.psu.edu/numrel/jclub/jc/Cook___LivRev_2000-5.pdf) article on initial data, or in the appendix of [this](https://arxiv.org/pdf/1704.00599.pdf) article. So, in KS coordinates,
\begin{align}
\rho^2 &= r^2 + a^2 \cos^2 \theta \\
\Delta &= r^2 - 2Mr + a^2 \\
\alpha &= \left(1 + \frac{2Mr}{\rho^2}\right)^{-2} \\
\beta^0 &= \frac{2 \alpha^2 Mr}{\rho^2} \\
\gamma_{00} &= 1 + \frac{2Mr}{\rho^2} \\
\gamma_{02} = \gamma_{20} &= -\left(1+\frac{2Mr}{\rho^2}\right) a \sin^2 \theta \\
\gamma_{11} &= \rho^2 \\
\gamma_{22} &= \left(r^2+a^2+\frac{2Mr}{\rho^2} a^2 \sin^2 \theta\right) \sin^2 \theta.
\end{align}
(Note that only the non-zero components of $\beta^i$ and $\gamma_{ij}$ are defined here.)

```python
# Adopt the Kerr-Schild metric for Fishbone-Moncrief disks
# http://gravity.psu.edu/numrel/jclub/jc/Cook___LivRev_2000-5.pdf
# Alternatively, Appendix of https://arxiv.org/pdf/1704.00599.pdf
rhoKS2  = r**2 + a**2*sp.cos(th)**2 # Eq 79 of Cook's Living Review article
DeltaKS = r**2 - 2*M*r + a**2    # Eq 79 of Cook's Living Review article
alphaKS = 1/sp.sqrt(1 + 2*M*r/rhoKS2)
betaKSU = ixp.zerorank1()
betaKSU[0] = alphaKS**2*2*M*r/rhoKS2
gammaKSDD = ixp.zerorank2()
gammaKSDD[0][0] = 1 + 2*M*r/rhoKS2
gammaKSDD[0][2] = gammaKSDD[2][0] = -(1 + 2*M*r/rhoKS2)*a*sp.sin(th)**2
gammaKSDD[1][1] = rhoKS2
gammaKSDD[2][2] = (r**2 + a**2 + 2*M*r/rhoKS2 * a**2*sp.sin(th)**2) * sp.sin(th)**2
```

We can also define the following useful quantities, continuing in KS coordinates:
\begin{align}
A &= a^2 \cos (2 \theta) + a^2 +2r^2 \\
B &= A + 4Mr \\
D &= \sqrt{\frac{2Mr}{a^2 \cos^2 \theta +r^2}+1};
\end{align}
we will also define the extrinsic curvature:
\begin{align}
K_{00}          &= D\frac{A+2Mr}{A^2 B} (4M(a^2 \cos(2 \theta)+a^s-2r^2)) \\
K_{01} = K_{10} &= \frac{D}{AB} (8a^2Mr\sin \theta \cos \theta) \\
K_{02} = K_{20} &= \frac{D}{A^2} (-2aM \sin^2 \theta (a^2\cos(2 \theta)+a^2-2r^2)) \\
K_{11}          &= \frac{D}{B} (4Mr^2) \\
K_{12} = K_{21} &= \frac{D}{AB} (-8a^3Mr \sin^3 \theta \cos \theta) \\
K_{22}          &= \frac{D}{A^2 B} (2Mr \sin^2 \theta (a^4(r-M) \cos(4 \theta) + a^4 (M+3r) + 4a^2 r^2 (2r-M) + 4a^2 r \cos(2 \theta) (a^2 + r(M+2r)) + 8r^5)). \\
\end{align}
Note that the indexing for extrinsic curvature only runs from 0 to 2, since there are no time components to the tensor.

```python
AA = a**2 * sp.cos(2*th) + a**2 + 2*r**2
BB = AA + 4*M*r
DD = sp.sqrt(2*M*r / (a**2 * sp.cos(th)**2 + r**2) + 1)
KDD[0][0] = DD*(AA + 2*M*r)/(AA**2*BB) * (4*M*(a**2 * sp.cos(2*th) + a**2 - 2*r**2))
KDD[0][1] = KDD[1][0] = DD/(AA*BB) * 8*a**2*M*r*sp.sin(th)*sp.cos(th)
KDD[0][2] = KDD[2][0] = DD/AA**2 * (-2*a*M*sp.sin(th)**2 * (a**2 * sp.cos(2*th) + a**2 - 2*r**2))
KDD[1][1] = DD/BB * 4*M*r**2
KDD[1][2] = KDD[2][1] = DD/(AA*BB) * (-8*a**3*M*r*sp.sin(th)**3*sp.cos(th))
KDD[2][2] = DD/(AA**2*BB) * \
            (2*M*r*sp.sin(th)**2 * (a**4*(r-M)*sp.cos(4*th) + a**4*(M+3*r) +
             4*a**2*r**2*(2*r-M) + 4*a**2*r*sp.cos(2*th)*(a**2 + r*(M+2*r)) + 8*r**5))
```

We must also compute the inverse and determinant of the KS metric. We can use the NRPy+ [indexedexp.py](../edit/indexedexp.py) function to do this easily for the inverse physical 3-metric $\gamma^{ij}$, and then use the lapse $\alpha$ and the shift $\beta^i$ to find the full, inverse 4-dimensional metric, $g^{ij}$. We use the general form relating the 3- and 4- metric from (B&S 2.122)
\begin{equation}
g_{\mu\nu} = \begin{pmatrix}
-\alpha^2 + \beta\cdot\beta & \beta_i \\
\beta_j & \gamma_{ij}
\end{pmatrix},
\end{equation}
and invert it. That is,
\begin{align}
g^{00}          &= -\frac{1}{\alpha^2} \\
g^{0i} = g^{i0} &= \frac{\beta^{i-1}}{\alpha^2} \\
g^{ij} = g^{ji} &= \gamma^{(i-1) (j-1)} - \frac{\beta^{i-1} \beta^{j-1}}{\alpha^2},
\end{align}
keeping careful track of the differences in the indexing conventions for 3-dimensional quantities and 4-dimensional quantities (Python always indexes lists from 0, but in four dimensions, the 0 direction corresponds to time, while in 3+1, the connection to time is handled by other variables).

```python
# For compatibility, we must compute gPhys4UU
gammaKSUU,gammaKSDET = ixp.symm_matrix_inverter3x3(gammaKSDD)
# See, e.g., Eq. 4.49 of https://arxiv.org/pdf/gr-qc/0703035.pdf , where N = alpha
gPhys4UU[0][0] = -1 / alphaKS**2
for i in range(1,4):
    if i>0:
        # if the quantity does not have a "4", then it is assumed to be a 3D quantity.
        #  E.g., betaKSU[] is a spatial vector, with indices ranging from 0 to 2:
        gPhys4UU[0][i] = gPhys4UU[i][0] = betaKSU[i-1]/alphaKS**2
for i in range(1,4):
    for j in range(1,4):
        # if the quantity does not have a "4", then it is assumed to be a 3D quantity.
        #  E.g., betaKSU[] is a spatial vector, with indices ranging from 0 to 2,
        #    and gammaKSUU[][] is a spatial tensor, with indices again ranging from 0 to 2.
        gPhys4UU[i][j] = gPhys4UU[j][i] = gammaKSUU[i-1][j-1] - betaKSU[i-1]*betaKSU[j-1]/alphaKS**2
```

## Step 2.i: Seed poloidal magnetic field $B^i$ \[Back to [top](#toc)\]
$$\label{magnetic_field}$$

The original Fishbone-Moncrief initial data prescription describes a non-self-gravitating accretion disk in hydrodynamical equilibrium about a black hole. The following assumes that a very weak magnetic field seeded into this disk will not significantly disturb this equilibrium, at least on a dynamical (free-fall) timescale.

Now, we will set up the magnetic field that, when simulated with a GRMHD code, will give us insight into the electromagnetic emission from the disk. We define the vector potential $A_i$ to be proportional to $\rho_0$, and, as usual, let the magnetic field $B^i$ be the curl of the vector potential.

```python
A_b = par.Cparameters("REAL",thismodule,"A_b",1.0)

A_3vecpotentialD = ixp.zerorank1()
# Set A_phi = A_b*rho_initial FIXME: why is there a sign error?
A_3vecpotentialD[2] = -A_b * rho_initial

BtildeU = ixp.register_gridfunctions_for_single_rank1("EVOL","BtildeU")
# Eq 15 of https://arxiv.org/pdf/1501.07276.pdf:
# B = curl A -> B^r = d_th A_ph - d_ph A_th
BtildeU[0] = sp.diff(A_3vecpotentialD[2],th) - sp.diff(A_3vecpotentialD[1],ph)
# B = curl A -> B^th = d_ph A_r - d_r A_ph
BtildeU[1] = sp.diff(A_3vecpotentialD[0],ph) - sp.diff(A_3vecpotentialD[2],r)
# B = curl A -> B^ph = d_r A_th - d_th A_r
BtildeU[2] = sp.diff(A_3vecpotentialD[1],r)  - sp.diff(A_3vecpotentialD[0],th)
```

## Step 2.j: Set the ADM quantities $\alpha$, $\beta^i$, and $\gamma_{ij}$ from the spacetime metric $g_{\mu\nu}$ \[Back to [top](#toc)\]
$$\label{adm_metric}$$

Now, we wish to build the 3+1-dimensional variables in terms of the inverse 4-dimensional spacetime metric $g^{ij},$ as demonstrated in eq. 4.49 of [Gourgoulhon's lecture notes on 3+1 formalisms](https://arxiv.org/pdf/gr-qc/0703035.pdf) (letting $N=\alpha$). So,
\begin{align}
\alpha &= \sqrt{-\frac{1}{g^{00}}} \\
\beta^i &= \alpha^2 g^{0 (i+1)} \\
\gamma^{ij} &= g^{(i+1) (j+1)} + \frac{\beta^i \beta_j}{\alpha^2},
\end{align}
again keeping careful track of the differences in the indexing conventions for 3-dimensional quantities and 4-dimensional quantities. We will also take the inverse of $\gamma^{ij}$, obtaining (naturally) $\gamma_{ij}$ and its determinant $|\gamma|$. (Note that the function we use gives the determinant of $\gamma^{ij}$, which is the reciprocal of $|\gamma|$.)

```python
# Construct spacetime metric in 3+1 form:
# See, e.g., Eq. 4.49 of https://arxiv.org/pdf/gr-qc/0703035.pdf , where N = alpha
alpha = gri.register_gridfunctions("EVOL",["alpha"])
betaU = ixp.register_gridfunctions_for_single_rank1("EVOL","betaU")

alpha = sp.sqrt(1/(-gPhys4UU[0][0]))
betaU = ixp.zerorank1()
for i in range(3):
    betaU[i] = alpha**2 * gPhys4UU[0][i+1]
gammaUU = ixp.zerorank2()
for i in range(3):
    for j in range(3):
        gammaUU[i][j] = gPhys4UU[i+1][j+1] + betaU[i]*betaU[j]/alpha**2

gammaDD = ixp.register_gridfunctions_for_single_rank2("EVOL","gammaDD","sym01")
gammaDD,igammaDET = ixp.symm_matrix_inverter3x3(gammaUU)
gammaDET = 1/igammaDET
```

Now, we will lower the index on the shift vector $\beta_j = \gamma_{ij} \beta^i$ and use that to calculate the 4-dimensional metric tensor, $g_{ij}$. So, we have
\begin{align}
g_{00}                    &= -\alpha^2 + \beta^2 \\
g_{0 (i+1)} = g_{(i+1) 0} &= \beta_i \\
g_{(i+1) (j+1)}           &= \gamma_{ij},
\end{align}
where $\beta^2 \equiv \beta^i \beta_i$.

```python
###############
# Next compute g_{\alpha \beta} from lower 3-metric, using
# Eq 4.47 of https://arxiv.org/pdf/gr-qc/0703035.pdf
betaD = ixp.zerorank1()
for i in range(3):
    for j in range(3):
        betaD[i] += gammaDD[i][j]*betaU[j]

beta2 = sp.sympify(0)
for i in range(3):
    beta2 += betaU[i]*betaD[i]

gPhys4DD = ixp.zerorank2(DIM=4)
gPhys4DD[0][0] = -alpha**2 + beta2
for i in range(3):
    gPhys4DD[0][i+1] = gPhys4DD[i+1][0] = betaD[i]
    for j in range(3):
        gPhys4DD[i+1][j+1] = gammaDD[i][j]
```

## Step 2.k: Set the magnetic field components in the comoving frame $b^\mu$, and $b^2$, which is twice the magnetic pressure \[Back to [top](#toc)\]
$$\label{magnetic_field_comoving_frame}$$

Next compute $b^{\mu}$ using Eqs 23, 24, 27 and 31 of [this paper](https://arxiv.org/pdf/astro-ph/0503420.pdf):
\begin{align}
B^i &= \frac{\tilde{B}}{\sqrt{|\gamma|}} \\
B^0_{(u)} &= \frac{u_{i+1} B^i}{\alpha} \\
b^0 &= \frac{B^0_{(u)}}{\sqrt{4 \pi}} \\
b^{i+1} &= \frac{\frac{B^i}{\alpha} + B^0_{(u)} u^{i+1}}{u^0 \sqrt{4 \pi}}
\end{align}

```python
###############
# Next compute b^{\mu} using Eqs 23 and 31 of https://arxiv.org/pdf/astro-ph/0503420.pdf
uKS4D = ixp.zerorank1(DIM=4)
for i in range(4):
    for j in range(4):
        uKS4D[i] += gPhys4DD[i][j] * uKS4U[j]

# Eq 27 of https://arxiv.org/pdf/astro-ph/0503420.pdf
BU = ixp.zerorank1()
for i in range(3):
    BU[i] = BtildeU[i]/sp.sqrt(gammaDET)

# Eq 23 of https://arxiv.org/pdf/astro-ph/0503420.pdf
BU0_u = sp.sympify(0)
for i in range(3):
    BU0_u += uKS4D[i+1]*BU[i]/alpha

smallbU = ixp.zerorank1(DIM=4)
smallbU[0] = BU0_u   / sp.sqrt(4 * sp.pi)
# Eqs 24 and 31 of https://arxiv.org/pdf/astro-ph/0503420.pdf
for i in range(3):
    smallbU[i+1] = (BU[i]/alpha + BU0_u*uKS4U[i+1])/(sp.sqrt(4*sp.pi)*uKS4U[0])

smallbD = ixp.zerorank1(DIM=4)
for i in range(4):
    for j in range(4):
        smallbD[i] += gPhys4DD[i][j]*smallbU[j]

smallb2 = sp.sympify(0)
for i in range(4):
    smallb2 += smallbU[i]*smallbD[i]
```

## Step 2.l: Lorentz factor $\Gamma = \alpha u^0$ and Valencia 3-velocity $v^i_{(n)}$ \[Back to [top](#toc)\]
$$\label{lorentz_fac_valencia}$$

Now, we will define the Lorentz factor ($= \alpha u^0$) and the Valencia 3-velocity $v^i_{(n)}$, which sets the 3-velocity as measured by normal observers to the spatial slice:
\begin{align}
v^i_{(n)} &= \frac{u^i}{u^0 \alpha} + \frac{\beta^i}{\alpha}, \\
\end{align}
as shown in eq 11 of [this](https://arxiv.org/pdf/1501.07276.pdf) paper. We will also compute the product of the square root of the determinant of the 3-metric with the lapse.

```python
###############
LorentzFactor = alpha * uKS4U[0]
# Define Valencia 3-velocity v^i_(n), which sets the 3-velocity as measured by normal observers to the spatial slice:
#  v^i_(n) = u^i/(u^0*alpha) + beta^i/alpha. See eq 11 of https://arxiv.org/pdf/1501.07276.pdf
Valencia3velocityU = ixp.zerorank1()
for i in range(3):
    Valencia3velocityU[i] = uKS4U[i + 1] / (alpha * uKS4U[0]) + betaU[i] / alpha

sqrtgamma4DET = sp.symbols("sqrtgamma4DET")
sqrtgamma4DET = sp.sqrt(gammaDET)*alpha
```

## Step 3: Output above-generated expressions to C code, using NRPy+ \[Back to [top](#toc)\]
$$\label{output_to_c}$$

Finally, we have contructed the underlying expressions necessary for the Fishbone-Moncrief initial data. By means of demonstration, we will use NRPy+'s `FD_outputC()` to print the expressions. (The actual output statements are commented out right now, to save time in testing.)

```python
KerrSchild_CKernel = [\
                     lhrh(lhs=gri.gfaccess("out_gfs","alpha"),rhs=alpha),\
                     lhrh(lhs=gri.gfaccess("out_gfs","betaU0"),rhs=betaU[0]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","betaU1"),rhs=betaU[1]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","betaU2"),rhs=betaU[2]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","gammaDD00"),rhs=gammaDD[0][0]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","gammaDD01"),rhs=gammaDD[0][1]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","gammaDD02"),rhs=gammaDD[0][2]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","gammaDD11"),rhs=gammaDD[1][1]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","gammaDD12"),rhs=gammaDD[1][2]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","gammaDD22"),rhs=gammaDD[2][2]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","KDD00"),rhs=KDD[0][0]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","KDD01"),rhs=KDD[0][1]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","KDD02"),rhs=KDD[0][2]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","KDD11"),rhs=KDD[1][1]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","KDD12"),rhs=KDD[1][2]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","KDD22"),rhs=KDD[2][2]),\
                     ]

#fin.FD_outputC("stdout",KerrSchild_CKernel)

FMdisk_Lorentz_uUs_CKernel = [\
                             lhrh(lhs=gri.gfaccess("out_gfs","LorentzFactor"),rhs=LorentzFactor),\
#                              lhrh(lhs=gri.gfaccess("out_gfs","uKS4U1"),rhs=uKS4U[1]),\
#                              lhrh(lhs=gri.gfaccess("out_gfs","uKS4U2"),rhs=uKS4U[2]),\
#                              lhrh(lhs=gri.gfaccess("out_gfs","uKS4U3"),rhs=uKS4U[3]),\
                             ]

#fin.FD_outputC("stdout",FMdisk_Lorentz_uUs_CKernel)

FMdisk_hm1_rho_P_CKernel = [\
#                           lhrh(lhs=gri.gfaccess("out_gfs","hm1"),rhs=hm1),\
                           lhrh(lhs=gri.gfaccess("out_gfs","rho_initial"),rhs=rho_initial),\
                           lhrh(lhs=gri.gfaccess("out_gfs","Pressure_initial"),rhs=Pressure_initial),\
                           ]

#fin.FD_outputC("stdout",FMdisk_hm1_rho_P_CKernel)

udotu = sp.sympify(0)
for i in range(4):
    udotu += uKS4U[i]*uKS4D[i]
#NRPy_file_output(OUTDIR+"/standalone-spherical_coords/NRPy_codegen/FMdisk_Btildes.h", [],[],[],
#                 ID_protected_variables + ["r","th","ph"],
#                 [],[uKS4U[0], "uKS4Ut", uKS4U[1],"uKS4Ur", uKS4U[2],"uKS4Uth", uKS4U[3],"uKS4Uph",
#                     uKS4D[0], "uKS4Dt", uKS4D[1],"uKS4Dr", uKS4D[2],"uKS4Dth", uKS4D[3],"uKS4Dph",
#                     uKS4D[1] * BU[0] / alpha, "Bur", uKS4D[2] * BU[1] / alpha, "Buth", uKS4D[3] * BU[2] / alpha, "Buph",
#                     gPhys4DD[0][0], "g4DD00", gPhys4DD[0][1], "g4DD01",gPhys4DD[0][2], "g4DD02",gPhys4DD[0][3], "g4DD03",
#                     BtildeU[0], "BtildeUr", BtildeU[1], "BtildeUth",BtildeU[2], "BtildeUph",
#                     smallbU[0], "smallbUt", smallbU[1], "smallbUr", smallbU[2], "smallbUth",smallbU[3], "smallbUph",
#                     smallb2,"smallb2",udotu,"udotu"])

FMdisk_Btildes_CKernel = [\
                         lhrh(lhs=gri.gfaccess("out_gfs","BtildeU0"),rhs=BtildeU[0]),\
                         lhrh(lhs=gri.gfaccess("out_gfs","BtildeU1"),rhs=BtildeU[1]),\
                         lhrh(lhs=gri.gfaccess("out_gfs","BtildeU2"),rhs=BtildeU[2]),\
                         ]

#fin.FD_outputC("stdout",FMdisk_Btildes_CKernel)
```

We will now use the relationships between coordinate systems provided by [reference_metric.py](../edit/reference_metric.py) to convert our expressions to Cartesian coordinates. See [Tutorial-Reference_Metric](Tutorial-Reference_Metric.ipynb) for more detail.

```python
# Now that all derivatives of ghat and gbar have been computed,
# we may now substitute the definitions r = rfm.xxSph[0], th=rfm.xxSph[1],...
# WARNING: Substitution only works when the variable is not an integer. Hence the if not isinstance(...,...) stuff.
# If the variable isn't an integer, we revert transcendental functions inside to normal variables. E.g., sin(x2) -> sinx2
#  Reverting to normal variables in this way makes expressions simpler in NRPy, and enables transcendental functions
#  to be pre-computed in SENR.

alpha = alpha.subs(r,rfm.xxSph[0]).subs(th,rfm.xxSph[1]).subs(ph,rfm.xxSph[2])
for i in range(DIM):
    betaU[i] = betaU[i].subs(r,rfm.xxSph[0]).subs(th,rfm.xxSph[1]).subs(ph,rfm.xxSph[2])
    for j in range(DIM):
        gammaDD[i][j] = gammaDD[i][j].subs(r,rfm.xxSph[0]).subs(th,rfm.xxSph[1]).subs(ph,rfm.xxSph[2])
        KDD[i][j]     = KDD[i][j].subs(r,rfm.xxSph[0]).subs(th,rfm.xxSph[1]).subs(ph,rfm.xxSph[2])

# GRMHD variables:
# Density and pressure:
hm1           = hm1.subs(r,rfm.xxSph[0]).subs(th,rfm.xxSph[1]).subs(ph,rfm.xxSph[2])
rho_initial          = rho_initial.subs(r,rfm.xxSph[0]).subs(th,rfm.xxSph[1]).subs(ph,rfm.xxSph[2])
Pressure_initial     = Pressure_initial.subs(r,rfm.xxSph[0]).subs(th,rfm.xxSph[1]).subs(ph,rfm.xxSph[2])
LorentzFactor = LorentzFactor.subs(r,rfm.xxSph[0]).subs(th,rfm.xxSph[1]).subs(ph,rfm.xxSph[2])

# "Valencia" three-velocity
for i in range(DIM):
    BtildeU[i] = BtildeU[i].subs(r,rfm.xxSph[0]).subs(th,rfm.xxSph[1]).subs(ph,rfm.xxSph[2])
    uKS4U[i+1] = uKS4U[i+1].subs(r,rfm.xxSph[0]).subs(th,rfm.xxSph[1]).subs(ph,rfm.xxSph[2])
    uBL4U[i+1] = uBL4U[i+1].subs(r,rfm.xxSph[0]).subs(th,rfm.xxSph[1]).subs(ph,rfm.xxSph[2])
    Valencia3velocityU[i] = Valencia3velocityU[i].subs(r,rfm.xxSph[0]).subs(th,rfm.xxSph[1]).subs(ph,rfm.xxSph[2])

```

At last, we will use our reference metric formalism and the Jacobian associated with the two coordinate systems to convert the spherical initial data to Cartesian coordinates. The module reference_metric.py provides us with the definition of $r, \theta, \phi$ in Cartesian coordinates. To find dthe Jacobian to then transform from spherical to Cartesian, we must find the tensor \begin{equation} \frac{\partial x_i}{\partial y_j}, \end{equation} where $x_i \in \{r,\theta,\phi\}$ and $y_i \in \{x,y,z\}$. We will also compute its inverse.

```python

# uUphi = uKS4U[3]
# uUphi = sympify_integers__replace_rthph(uUphi,r,th,ph,rfm.xxSph[0],rfm.xxSph[1],rfm.xxSph[2])
# uUt = uKS4U[0]
# uUt = sympify_integers__replace_rthph(uUt,r,th,ph,rfm.xxSph[0],rfm.xxSph[1],rfm.xxSph[2])

# Transform initial data to our coordinate system:
# First compute Jacobian and its inverse
drrefmetric__dx_0UDmatrix = sp.Matrix([[sp.diff(rfm.xxSph[0],rfm.xx[0]), sp.diff( rfm.xxSph[0],rfm.xx[1]), sp.diff( rfm.xxSph[0],rfm.xx[2])],
                                       [sp.diff(rfm.xxSph[1],rfm.xx[0]), sp.diff(rfm.xxSph[1],rfm.xx[1]), sp.diff(rfm.xxSph[1],rfm.xx[2])],
                                       [sp.diff(rfm.xxSph[2],rfm.xx[0]), sp.diff(rfm.xxSph[2],rfm.xx[1]), sp.diff(rfm.xxSph[2],rfm.xx[2])]])
dx__drrefmetric_0UDmatrix = drrefmetric__dx_0UDmatrix.inv()

# Declare as gridfunctions the final quantities we will output for the initial data
IDalpha = gri.register_gridfunctions("EVOL","IDalpha")
IDgammaDD = ixp.register_gridfunctions_for_single_rank2("EVOL","IDgammaDD","sym01")
IDKDD = ixp.register_gridfunctions_for_single_rank2("EVOL","IDKDD","sym01")
IDbetaU   = ixp.register_gridfunctions_for_single_rank1("EVOL","IDbetaU")
IDValencia3velocityU = ixp.register_gridfunctions_for_single_rank1("EVOL","IDValencia3velocityU")

IDalpha = alpha
for i in range(3):
    IDbetaU[i] = 0
    IDValencia3velocityU[i] = 0
    for j in range(3):
        # Matrices are stored in row, column format, so (i,j) <-> (row,column)
        IDbetaU[i]   += dx__drrefmetric_0UDmatrix[(i,j)]*betaU[j]
        IDValencia3velocityU[i]   += dx__drrefmetric_0UDmatrix[(i,j)]*Valencia3velocityU[j]
        IDgammaDD[i][j] = 0
        IDKDD[i][j] = 0
        for k in range(3):
            for l in range(3):
                IDgammaDD[i][j] += drrefmetric__dx_0UDmatrix[(k,i)]*drrefmetric__dx_0UDmatrix[(l,j)]*gammaDD[k][l]
                IDKDD[i][j]     += drrefmetric__dx_0UDmatrix[(k,i)]*drrefmetric__dx_0UDmatrix[(l,j)]*    KDD[k][l]

# -={ Spacetime quantities: Generate C code from expressions and output to file }=-
KerrSchild_to_print = [\
                     lhrh(lhs=gri.gfaccess("out_gfs","IDalpha"),rhs=IDalpha),\
                     lhrh(lhs=gri.gfaccess("out_gfs","IDbetaU0"),rhs=IDbetaU[0]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","IDbetaU1"),rhs=IDbetaU[1]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","IDbetaU2"),rhs=IDbetaU[2]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","IDgammaDD00"),rhs=IDgammaDD[0][0]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","IDgammaDD01"),rhs=IDgammaDD[0][1]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","IDgammaDD02"),rhs=IDgammaDD[0][2]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","IDgammaDD11"),rhs=IDgammaDD[1][1]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","IDgammaDD12"),rhs=IDgammaDD[1][2]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","IDgammaDD22"),rhs=IDgammaDD[2][2]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","IDKDD00"),rhs=IDKDD[0][0]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","IDKDD01"),rhs=IDKDD[0][1]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","IDKDD02"),rhs=IDKDD[0][2]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","IDKDD11"),rhs=IDKDD[1][1]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","IDKDD12"),rhs=IDKDD[1][2]),\
                     lhrh(lhs=gri.gfaccess("out_gfs","IDKDD22"),rhs=IDKDD[2][2]),\
                     ]

# -={ GRMHD quantities: Generate C code from expressions and output to file }=-
FMdisk_GRHD_hm1_to_print = [lhrh(lhs=gri.gfaccess("out_gfs","rho_initial"),rhs=rho_initial)]

FMdisk_GRHD_velocities_to_print = [\
                                 lhrh(lhs=gri.gfaccess("out_gfs","IDValencia3velocityU0"),rhs=IDValencia3velocityU[0]),\
                                 lhrh(lhs=gri.gfaccess("out_gfs","IDValencia3velocityU1"),rhs=IDValencia3velocityU[1]),\
                                 lhrh(lhs=gri.gfaccess("out_gfs","IDValencia3velocityU2"),rhs=IDValencia3velocityU[2]),\
                                 ]
```

To verify this against the old version of FishboneMoncriefID from the old version of NRPy, we use the `mathematica_code()` output function.

```python
# Comment out debug code for now, to reduce this file's size.

#from mathematica_output import *

# print("ID1alpha = " + sp.mathematica_code(IDalpha) + ";")
# print("ID1beta0 = " + sp.mathematica_code(IDbetaU[0]) + ";")
# print("ID1beta1 = " + sp.mathematica_code(IDbetaU[1]) + ";")
# print("ID1beta2 = " + sp.mathematica_code(IDbetaU[2]) + ";")
# print("ID1gamma00 = " + sp.mathematica_code(IDgammaDD[0][0]) + ";")
# print("ID1gamma01 = " + sp.mathematica_code(IDgammaDD[0][1]) + ";")
# print("ID1gamma02 = " + sp.mathematica_code(IDgammaDD[0][2]) + ";")
# print("ID1gamma11 = " + sp.mathematica_code(IDgammaDD[1][1]) + ";")
# print("ID1gamma12 = " + sp.mathematica_code(IDgammaDD[1][2]) + ";")
# print("ID1gamma22 = " + sp.mathematica_code(IDgammaDD[2][2]) + ";")
# print("ID1K00 = " + sp.mathematica_code(IDKDD[0][0]) + ";")
# print("ID1K01 = " + sp.mathematica_code(IDKDD[0][1]) + ";")
# print("ID1K02 = " + sp.mathematica_code(IDKDD[0][2]) + ";")
# print("ID1K11 = " + sp.mathematica_code(IDKDD[1][1]) + ";")
# print("ID1K12 = " + sp.mathematica_code(IDKDD[1][2]) + ";")
# print("ID1K22 = " + sp.mathematica_code(IDKDD[2][2]) + ";")

# print("hm11 = " + sp.mathematica_code(hm1) + ";")

# print("ID1Valencia3velocityU0 = " + sp.mathematica_code(IDValencia3velocityU[0]) + ";")
# print("ID1Valencia3velocityU1 = " + sp.mathematica_code(IDValencia3velocityU[1]) + ";")
# print("ID1Valencia3velocityU2 = " + sp.mathematica_code(IDValencia3velocityU[2]) + ";")

```

# Step 4: Code Validation against `FishboneMoncriefID.FishboneMoncriefID` NRPy+ module \[Back to [top](#toc)\]
$$\label{code_validation}$$

Here, as a code validation check, we verify agreement in the SymPy expressions for these Fishbone-Moncrief initial data between

1. this tutorial and
2. the NRPy+ [FishboneMoncriefID.FishboneMoncriefID](../edit/FishboneMoncriefID/FishboneMoncriefID.py) module.

```python
gri.glb_gridfcs_list = []

import FishboneMoncriefID.FishboneMoncriefID as fmid
fmid.FishboneMoncriefID()

print("IDalpha - fmid.IDalpha = " + str(IDalpha - fmid.IDalpha))
print("rho_initial - fmid.rho_initial = " + str(rho_initial - fmid.rho_initial))
print("hm1 - fmid.hm1 = " + str(hm1 - fmid.hm1))

for i in range(DIM):
    print("IDbetaU["+str(i)+"] - fmid.IDbetaU["+str(i)+"] = " + str(IDbetaU[i] - fmid.IDbetaU[i]))
    print("IDValencia3velocityU["+str(i)+"] - fmid.IDValencia3velocityU["+str(i)+"] = "\
          + str(IDValencia3velocityU[i] - fmid.IDValencia3velocityU[i]))
    for j in range(DIM):
        print("IDgammaDD["+str(i)+"]["+str(j)+"] - fmid.IDgammaDD["+str(i)+"]["+str(j)+"] = "
              + str(IDgammaDD[i][j] - fmid.IDgammaDD[i][j]))
        print("IDKDD["+str(i)+"]["+str(j)+"] - fmid.IDKDD["+str(i)+"]["+str(j)+"] = "
              + str(IDKDD[i][j] - fmid.IDKDD[i][j]))
```

    IDalpha - fmid.IDalpha = 0
    rho_initial - fmid.rho_initial = 0
    hm1 - fmid.hm1 = 0
    IDbetaU[0] - fmid.IDbetaU[0] = 0
    IDValencia3velocityU[0] - fmid.IDValencia3velocityU[0] = 0
    IDgammaDD[0][0] - fmid.IDgammaDD[0][0] = 0
    IDKDD[0][0] - fmid.IDKDD[0][0] = 0
    IDgammaDD[0][1] - fmid.IDgammaDD[0][1] = 0
    IDKDD[0][1] - fmid.IDKDD[0][1] = 0
    IDgammaDD[0][2] - fmid.IDgammaDD[0][2] = 0
    IDKDD[0][2] - fmid.IDKDD[0][2] = 0
    IDbetaU[1] - fmid.IDbetaU[1] = 0
    IDValencia3velocityU[1] - fmid.IDValencia3velocityU[1] = 0
    IDgammaDD[1][0] - fmid.IDgammaDD[1][0] = 0
    IDKDD[1][0] - fmid.IDKDD[1][0] = 0
    IDgammaDD[1][1] - fmid.IDgammaDD[1][1] = 0
    IDKDD[1][1] - fmid.IDKDD[1][1] = 0
    IDgammaDD[1][2] - fmid.IDgammaDD[1][2] = 0
    IDKDD[1][2] - fmid.IDKDD[1][2] = 0
    IDbetaU[2] - fmid.IDbetaU[2] = 0
    IDValencia3velocityU[2] - fmid.IDValencia3velocityU[2] = 0
    IDgammaDD[2][0] - fmid.IDgammaDD[2][0] = 0
    IDKDD[2][0] - fmid.IDKDD[2][0] = 0
    IDgammaDD[2][1] - fmid.IDgammaDD[2][1] = 0
    IDKDD[2][1] - fmid.IDKDD[2][1] = 0
    IDgammaDD[2][2] - fmid.IDgammaDD[2][2] = 0
    IDKDD[2][2] - fmid.IDKDD[2][2] = 0

# Step 5: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename [Tutorial-FishboneMoncriefID.pdf](Tutorial-FishboneMoncriefID.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
!jupyter nbconvert --to latex --template latex_nrpy_style.tplx --log-level='WARN' Tutorial-FishboneMoncriefID.ipynb
!pdflatex -interaction=batchmode Tutorial-FishboneMoncriefID.tex
!pdflatex -interaction=batchmode Tutorial-FishboneMoncriefID.tex
!pdflatex -interaction=batchmode Tutorial-FishboneMoncriefID.tex
!rm -f Tut*.out Tut*.aux Tut*.log
```

    [NbConvertApp] Converting notebook Tutorial-FishboneMoncriefID.ipynb to latex
    [pandoc warning] Duplicate link reference `[comment]' "source" (line 17, column 1)
    [NbConvertApp] Writing 129778 bytes to Tutorial-FishboneMoncriefID.tex
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode


Copyright (c) 2015, 2016 [Sebastian Raschka](sebastianraschka.com)

2016 [Li-Yi Wei](http://www.liyiwei.org/)

https://github.com/1iyiwei/pyml

[MIT License](https://github.com/1iyiwei/pyml/blob/master/LICENSE.txt)

# Python Machine Learning - Code Examples

# Chapter 12 - Training Artificial Neural Networks for Image Recognition

Deep learning
* hot topic
* neural networks with many layers (i.e. in the range of hundreds or thousands)

Basic understanding of multi-layer neural networks
* predict via forward propagation
* training via backp ropagation
* application in handwritten digit classification
* debugging

Other types of neural networks
* CNN - convolutional neural networks
* RNN - recurrent neural networks

Note that the optional watermark extension is a small IPython notebook plugin that I developed to make the code reproducible. You can just skip the following line(s).

```python
%load_ext watermark
%watermark -a '' -u -d -v -p numpy,scipy,matplotlib
```

    last updated: 2016-11-07

    CPython 3.5.2
    IPython 4.2.0

    numpy 1.11.1
    scipy 0.17.1
    matplotlib 1.5.1

*The use of `watermark` is optional. You can install this IPython extension via "`pip install watermark`". For more information, please see: https://github.com/rasbt/watermark.*

### Overview

- [Modeling complex functions with artificial neural networks](#Modeling-complex-functions-with-artificial-neural-networks)
  - [Single-layer neural network recap](#Single-layer-neural-network-recap)
  - [Introducing the multi-layer neural network architecture](#Introducing-the-multi-layer-neural-network-architecture)
  - [Activating a neural network via forward propagation](#Activating-a-neural-network-via-forward-propagation)
- [Classifying handwritten digits](#Classifying-handwritten-digits)
  - [Obtaining the MNIST dataset](#Obtaining-the-MNIST-dataset)
  - [Implementing a multi-layer perceptron](#Implementing-a-multi-layer-perceptron)
- [Training an artificial neural network](#Training-an-artificial-neural-network)
  - [Computing the logistic cost function](#Computing-the-logistic-cost-function)
  - [Training neural networks via backpropagation](#Training-neural-networks-via-backpropagation)
- [Developing your intuition for backpropagation](#Developing-your-intuition-for-backpropagation)
- [Debugging neural networks with gradient checking](#Debugging-neural-networks-with-gradient-checking)
- [Convergence in neural networks](#Convergence-in-neural-networks)
- [Other neural network architectures](#Other-neural-network-architectures)
  - [Convolutional Neural Networks](#Convolutional-Neural-Networks)
  - [Recurrent Neural Networks](#Recurrent-Neural-Networks)
- [A few last words about neural network implementation](#A-few-last-words-about-neural-network-implementation)
- [Summary](#Summary)

```python
from IPython.display import Image
%matplotlib inline
```

# Modeling complex functions with artificial neural networks

We introduced single-layer neural networks for linear classification.

Mutli-layer neural network is an extension of single-layer neural networks.

A bit history of multi-layer neural networks:
* Started in the 1950s, but has been limited by available data and computation power.
* Regained popularity in recent years due to data explosion and Moore's law.

Simple algorithms can work well with enough data and computation
* [bid your time](http://www.macleans.ca/society/science/the-meaning-of-alphago-the-ai-program-that-beat-a-go-champ/)

Many cool applications:
* largely pursued by companies such as Google, Facebook, Amazon, Microsoft, Apple, Baidu, etc.

## Single-layer neural network recap

$\mathbf{x}$: input sample,
$y$: corresponding label

$\mathbf{w}$: weight parameters

$\mathbf{z}$: linear input
$$
z = \mathbf{w}^T \mathbf{x}
$$

$a$: activation,
$\phi$: activation function:
$$
a = \phi(z)
$$

* Direct thresholding - perceptron

* Linear activation function - Adaline
$$
\phi(z) = z
$$

* Sigmoid activation function - logistic regression
$$
\phi(z) = \frac{1}{1+e^{-z}}
$$

$\hat{y}$: predicted output
$$
\begin{align}
\hat{y} &=
\begin{cases}
1 & \phi(z) \geq 0
\\
-1 & \phi(z) < 0
\end{cases}
\end{align}
$$

### Training single-layer network

#### Objective function

Sum-of-squared error for linear activation:
$$
\begin{align}
J\left(\mathbf{w}\right) = \frac{1}{2} \sum_i \left( \phi\left(z^{(i)}\right) - y^{(i)} \right)^2
\end{align}
$$

Logistic regression for Sigmoid:
$$
\begin{align}
J\left(\mathbf{w} \right) = \sum_i -y^{(i)}\log\left(\phi\left(z^{(i)}\right)\right) - \left(1 - y^{(i)} \right) \log\left(1 - \phi\left( z^{(i)}\right) \right)
\end{align}
$$

#### Derivative

With proper (clever?) combination of $J$ and $\phi$ we can reach a simple and elegant derivative:
$$
\begin{align}
\frac{\partial J}{\partial \mathbf{w}} &=
\sum_i \mathbf{x}^{(i)} \left( \phi\left(z^{(i)}\right) - y^{(i)} \right)
\end{align}
$$
The above formula works for perceptron, adaptive linear neuron, and logistic regression.

#### Gradient descent:
$$
\begin{align}
\delta \mathbf{w} &= -\eta \frac{\partial J}{\partial \mathbf{w}}
\\
\mathbf{w} &\leftarrow \mathbf{w} + \delta \mathbf{w}
\end{align}
$$

Stochastic gradient descent
* Just take a subset instead of all samples in computing $\frac{\partial J}{\partial \mathbf{w}}$

## Vector version

We can assume all functions and operators take on vector inputs similar to scalar inputs, just operate component-wise.
For example:
$$
\begin{align}
\mathbf{v} &= \left(v_1, v_2, \cdots, v_n \right)
\\
f(\mathbf{v}) &=  \left(f\left(v_1\right), f\left(v_2\right), \cdots, f\left(v_n\right) \right)
\end{align}
$$
And
$$
\begin{align}
\mathbf{x} &= \left(x_1, x_2, \cdots, x_n \right)
\\
\mathbf{y} &= \left(y_1, y_2, \cdots, y_n \right)
\\
\mathbf{x} + \mathbf{y} &= \left(x_1 + y_1, x_2 + y_2, \cdots, x_n + y_n \right)
\\
\mathbf{x} * \mathbf{y} &= \left(x_1 * y_1, x_2 * y_2, \cdots, x_n * y_n \right)
\end{align}
$$

We can also keep the inner product:
$$
\begin{align}
\mathbf{x} \mathbf{y} &= \sum_{i=1}^n x_i y_i
\end{align}
$$

Then we can directly use any output cost function, such as in logistic regression:
$$
\begin{align}
L\left(\mathbf{W}, \mathbf{X}, \mathbf{Y} \right) &= \sum_i -\mathbf{y}^{(i)}\log\left(\mathbf{a}^{(i)}\right) - \left(1 - \mathbf{y}^{(i)} \right) \log\left(1 - \mathbf{a}^{(i)} \right)
\end{align}
$$
* $i$ runs through all training samples
* $\mathbf{y}$ one-hot vector for classification
$$
\begin{align}
\mathbf{a^{(i)}} =
\begin{bmatrix}
0.1
\\
0.9
\\
\vdots
\\
0.3
\end{bmatrix}
,
\mathbf{y^{(i)}} =
\begin{bmatrix}
0
\\
1
\\
\vdots
\\
0
\end{bmatrix}
\end{align}
$$
* $\mathbf{W}$ is the set of all weights in the network (across all layers)

## Introducing the multi-layer neural network architecture

We can extend single layer networks to multiple layers

Often called:
* multi-layer feedforward neural network
* multi-layer perceptron (MLP) even if other types of activations are used

### MLP example

One hidden layer

Two weight layers
* input layer fully connected to hidden layer
* hidden layer fully connected to output layer

Connection types
* full: every node in one layer is connected to every node in another layer
* other types possible, such as convolutional (to describe later)

Hyper-parameters
* network architecture
* number of layers
* number of nodes in each layer
* bias: $x_0 = a_0 = 1$

### Why multiple layers?

The latter layers collect information from earlier layers, and can process higher level information.

Different layers focus on different levels of information.

Facial recognition example:
* earlier layers: identify eyes, nose, mouth, etc.
* later layers: identify the relative positions of eyes, noise, mouth, etc.

### MLP representation

$\mathbf{a}_j^{(i)}$: $j$th node in the $i$th activation layer

$\mathbf{a}^{(1)} = \mathbf{x}$: the input can be considered as the first activation

$\mathbf{W}_{j,k}^{(\ell)}$: weight connecting $k$th node in layer $\ell$ to $j$th node in layer $\ell+1$
* like for matrix multiplication: $\mathbf{z}^{\ell+1} = \mathbf{W}^{(\ell)} \mathbf{a}^{(\ell)}$

Can have multiple outputs for multi-class classification
* e.g. one hot: (1, 0, 0) for class 1, (0, 1, 0) for class 2, (0, 0, 1) for class 3

### Number of layers

People use different conventions to count the number of layers for a network.

Number of layers with weights
* e.g. adding a threshold after activation does not count as another layer
* avoid ambiguity
* reflect computational power and complexity

Number of hidden layers (with weights) for multi-layer networks

Number of input + output + hidden layers
* usually 1 + number of weight layers

Deep neural network
* conventional definition: $>1$ hidden layer
* modern definition: a lot more than that (hundreds or thousands)

# MLP classifying and training

Both are extensions from the single layer networks.

Called forward and backward propagation for MLP.

## Overview

1.
Starting at the input layer, we forward propagate the activations from one layer to the next until we reach the last layer to produce the output.

2.
Based on the output, we calculate the error that we want to minimize with a cost function.

3.
We backpropagate the error, find its derivative with respect to each weight in the network, and update the model weights.

Classification/forward-propagation takes step 1.

Training/back-propagation repeats steps 1 to 3 (through multiple batches/epochs).

## Activating a neural network via forward propagation

Pass the input sample $\mathbf{x}$ through each layer of the network.

Start from the input layer (initialization):
$$
\begin{align}
\mathbf{a}^{1} = \mathbf{x}
\end{align}
$$

For each layer $\ell \in [1, 2, \cdots, L-1]$ where the last layer is $L$:
$$
\begin{align}
\mathbf{z}^{(\ell + 1)} &= \mathbf{W}^{(\ell)} \mathbf{a}^{(\ell)}
\\
\mathbf{a}^{(\ell + 1)} &= \phi\left( \mathbf{z}^{(\ell+1)}\right)
\end{align}
$$

Compute the output via certain output function $o$ (thresholding, one-hot, softmax, etc.):
$$
\hat{\mathbf{y}} = o\left(\mathbf{a}^{(L)}\right)
$$

Start from the input layer (initialization):
$$
\begin{align}
\mathbf{a}^{1} = \mathbf{x}
\end{align}
$$

For each layer $\ell \in [1, 2, \cdots, L-1]$ where the last layer is $L=3$:

$$
\begin{align}
\mathbf{W}^{(1)} &: 3 \times 4 \; \mbox{matrix}
\\
\mathbf{z}^{(2)} &= \mathbf{W}^{(1)} \mathbf{a}^{(1)}
\\
\mathbf{a}^{(2)} &= \phi\left( \mathbf{z}^{(2)}\right)
\end{align}
$$

$$
\begin{align}
\mathbf{W}^{(2)} &: 3 \times 4 \; \mbox{matrix}
\\
\mathbf{z}^{(3)} &= \mathbf{W}^{(2)} \mathbf{a}^{(2)}
\\
\mathbf{a}^{(3)} &= \phi\left( \mathbf{z}^{(3)}\right)
\end{align}
$$

Compute the output via certain output function $o$ (thresholding, one-hot, softmax, etc.):
$$
\hat{\mathbf{y}} = o\left(\mathbf{a}^{(3)}\right)
$$

## Generalization to multiple inputs

We can generalize the above to handle multiple inputs $\mathbf{X}$ instead of just one input sample $\mathbf{x} \in \mathbf{X}$.
* Recall the rows/cols of matrix $\mathbf{X}$ correspond to the samples/features.

Matrix multiplication and vector/matrix function evaluation allow us to simply capitalize all lower case symbols into upper case ones.

Start from the input layer (initialization):
$$
\begin{align}
\mathbf{A}^{1} = \mathbf{X}^T
\end{align}
$$
* the transpose is to convert samples from rows into columns for subsequent matrix multiplication

For each layer $\ell \in [1, 2, \cdots, L-1]$ where the last layer is $L$:
$$
\begin{align}
\mathbf{Z}^{(\ell + 1)} &= \mathbf{W}^{(\ell)} \mathbf{A}^{(\ell)}
\\
\mathbf{A}^{(\ell + 1)} &= \phi\left( \mathbf{Z}^{(\ell+1)}\right)
\end{align}
$$

Compute the output via certain output function $o$ (thresholding, one-hot, softmax, etc.):
$$
\hat{\mathbf{Y}} = o\left(\mathbf{A}^{(L)}\right)
$$

## Activation function

Choose non-linear functions for internal activations to solve general problems.

If all internal nodes have linear activation, we can show that a multi-layer network can be reduce to a single layer network.

Sigmoid $\phi(z) = \frac{1}{1 + e^{-z}}$ and tanh $\phi(z) = \frac{1-e^{-z}}{1+e^{-z}}$ are common choices for activation $\phi$ in hidden layers.

ReLu is chosen for deep networks for faster convergence.
$$
\phi(z) = \max\left(0, z\right)
$$

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

def plot_func(x, y, title):
    plt.plot(x, y)
    plt.axvline(0.0, color='k')
    plt.ylim(-1.1, 1.1)
    plt.xlabel('z')
    plt.ylabel('$\phi (z)$')

    # y axis ticks and gridline
    plt.yticks([0.0, 0.5, 1.0])
    ax = plt.gca()
    ax.yaxis.grid(True)

    plt.title(title)
    plt.tight_layout()
    plt.show()

def linear(z):
    return z

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

def tanh(z):
    return (1.0 - np.exp(-z))/(1.0 + np.exp(-z))

def relu(z):
    return np.maximum(0, z)

z = np.arange(-7, 7, 0.1)

activations = ['sigmoid', 'tanh', 'relu']

for i, activation in enumerate(activations):
    if(activation == 'sigmoid'):
        phi_z = sigmoid(z)
    elif (activation == 'tanh'):
        phi_z = tanh(z)
    elif (activation == 'relu'):
        phi_z = relu(z)
    else:
        phi_z = linear(z)

    plot_func(z, phi_z, activation)
```

# Classifying handwritten digits

Let's apply MLP for a practical application before talking more about math and algorithms.

## The MNIST dataset

A popular dataset for testing machine learning algorithms.

Mixed National Institute of Standards and Technology database, constructed by Yann LeCun.
* the origin of ATM taking checks

Handwritten digits from 250 people
* 50% high school students
* 50% Census Bureau employees

Properties
* $\mathbf{X}$: $28 \times 28$-pixel images
* $\mathbf{y}$: labels in $[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]$
* 60,000 training samples
* 10,000 test samples

## Obtaining the MNIST dataset

The MNIST dataset is publicly available at http://yann.lecun.com/exdb/mnist/ and consists of the following four parts:

- Training set images: train-images-idx3-ubyte.gz (9.9 MB, 47 MB unzipped, 60,000 samples)
- Training set labels: train-labels-idx1-ubyte.gz (29 KB, 60 KB unzipped, 60,000 labels)
- Test set images: t10k-images-idx3-ubyte.gz (1.6 MB, 7.8 MB, 10,000 samples)
- Test set labels: t10k-labels-idx1-ubyte.gz (5 KB, 10 KB unzipped, 10,000 labels)

In this section, we will only be working with a subset of MNIST, thus, we only need to download the training set images and training set labels. After downloading the files, I recommend unzipping the files using the Unix/Linux gzip tool from the terminal for efficiency, e.g., using the command

    gzip *ubyte.gz -d

in your local MNIST download directory, or, using your favorite unzipping tool if you are working with a machine running on Microsoft Windows. The images are stored in byte form, and using the following function, we will read them into NumPy arrays that we will use to train our MLP.

Li-Yi: I enhanced the functions below so that we can do everything (including downloading and decompression) inside ipynb.

```python
import os
import os.path
import struct
import gzip
import numpy as np

def open_mnist(full_path):
    if full_path.find('.gz') >= 0:
        return gzip.open(full_path, 'rb')
    else:
        return open(full_path, 'rb')

def pick_mnist(path, name, exts):
    for ext in exts:
        full_path = os.path.join(path, name + ext)
        if os.path.isfile(full_path):
            return full_path
    # none of the exts options works
    return None

def load_mnist(path, kind='train', exts=['', '.gz']):
    """Load MNIST data from `path`"""
    labels_path = pick_mnist(path, kind + '-labels-idx1-ubyte', exts)
    images_path = pick_mnist(path, kind + '-images-idx3-ubyte', exts)

    with open_mnist(labels_path) as lbpath:
        magic, n = struct.unpack('>II', lbpath.read(8))
        if(magic != 2049):
            raise IOError(str(magic) + ' != ' + str(2049))

        # np.fromfile does not work with gzip open
        # http://stackoverflow.com/questions/15966335/efficient-numpy-fromfile-on-zipped-files
        # labels = np.fromfile(lbpath, dtype=np.uint8)
        content = lbpath.read()
        labels = np.frombuffer(content, dtype=np.uint8)
        if(len(labels) != n):
            raise IOError(str(len(labels)) + ' != ' + str(n))

    with open_mnist(images_path) as imgpath:
        magic, num, rows, cols = struct.unpack(">IIII", imgpath.read(16))
        if(magic != 2051):
            raise IOError(str(magic) + ' != ' + str(2051))

        # images = np.fromfile(imgpath, dtype=np.uint8).reshape(num, rows*cols)
        content = imgpath.read()
        images = np.frombuffer(content, dtype=np.uint8).reshape(num, rows*cols)
        if(num != len(labels)):
            raise IOError(str(num) + ' != ' + str(len(labels)))

    return images, labels
```

```python
mnist_data_folder = os.path.join('..', 'datasets', 'mnist')
exts = ['', '.gz'] # for already gunzipped files and not yet gzipped files

X_train, y_train = load_mnist(mnist_data_folder, kind='train', exts=exts)
print('Rows: %d, columns: %d' % (X_train.shape[0], X_train.shape[1]))
X_test, y_test = load_mnist(mnist_data_folder, kind='t10k', exts=exts)
print('Rows: %d, columns: %d' % (X_test.shape[0], X_test.shape[1]))
```

    Rows: 60000, columns: 784
    Rows: 10000, columns: 784

Note we flatten each $28\times28$ image into a $784 = 28^2$ vector as input to the neural network.

Visualize the first digit of each class:

```python
import matplotlib.pyplot as plt

fig, ax = plt.subplots(nrows=2, ncols=5, sharex=True, sharey=True)
ax = ax.flatten()
for i in range(10):
    img = X_train[y_train == i][0].reshape(28, 28)
    ax[i].imshow(img, cmap='Greys', interpolation='nearest')

ax[0].set_xticks([])
ax[0].set_yticks([])
plt.tight_layout()
# plt.savefig('./figures/mnist_all.png', dpi=300)
plt.show()
```

Visualize 25 different versions of "7":

```python
fig, ax = plt.subplots(nrows=5, ncols=5, sharex=True, sharey=True,)
ax = ax.flatten()
for i in range(25):
    img = X_train[y_train == 7][i].reshape(28, 28)
    ax[i].imshow(img, cmap='Greys', interpolation='nearest')

ax[0].set_xticks([])
ax[0].set_yticks([])
plt.tight_layout()
# plt.savefig('./figures/mnist_7.png', dpi=300)
plt.show()
```

Uncomment the following lines to optionally save the data in CSV format.
However, note that those CSV files will take up a substantial amount of storage space:

- train_img.csv 1.1 GB (gigabytes)
- train_labels.csv 1.4 MB (megabytes)
- test_img.csv 187.0 MB
- test_labels 144 KB (kilobytes)

```python
# np.savetxt('train_img.csv', X_train, fmt='%i', delimiter=',')
# np.savetxt('train_labels.csv', y_train, fmt='%i', delimiter=',')
# X_train = np.genfromtxt('train_img.csv', dtype=int, delimiter=',')
# y_train = np.genfromtxt('train_labels.csv', dtype=int, delimiter=',')

# np.savetxt('test_img.csv', X_test, fmt='%i', delimiter=',')
# np.savetxt('test_labels.csv', y_test, fmt='%i', delimiter=',')
# X_test = np.genfromtxt('test_img.csv', dtype=int, delimiter=',')
# y_test = np.genfromtxt('test_labels.csv', dtype=int, delimiter=',')

```

## Implementing a multi-layer perceptron

This is an implementation of a single-hidden neural network.

scikit-learn does not contain neural network stuff, but there are other options that we can use.
* e.g. Keras, Torch7, Theano, Caffe, TensorFlow, etc.

We have not talked about backpropagation yet, but having this code as reference can help understanding.

```python
import numpy as np
from scipy.special import expit
import sys

class NeuralNetMLP(object):
    """ Feedforward neural network / Multi-layer perceptron classifier.

    Parameters
    ------------
    n_output : int
        Number of output units, should be equal to the
        number of unique class labels.
    n_features : int
        Number of features (dimensions) in the target dataset.
        Should be equal to the number of columns in the X array.
    n_hidden : int (default: 30)
        Number of hidden units.
    l1 : float (default: 0.0)
        Lambda value for L1-regularization.
        No regularization if l1=0.0 (default)
    l2 : float (default: 0.0)
        Lambda value for L2-regularization.
        No regularization if l2=0.0 (default)
    epochs : int (default: 500)
        Number of passes over the training set.
    eta : float (default: 0.001)
        Learning rate.
    alpha : float (default: 0.0)
        Momentum constant. Factor multiplied with the
        gradient of the previous epoch t-1 to improve
        learning speed
        w(t) := w(t) - (grad(t) + alpha*grad(t-1))
    decrease_const : float (default: 0.0)
        Decrease constant. Shrinks the learning rate
        after each epoch via eta / (1 + epoch*decrease_const)
    shuffle : bool (default: True)
        Shuffles training data every epoch if True to prevent circles.
    minibatches : int (default: 1)
        Divides training data into k minibatches for efficiency.
        Normal gradient descent learning if k=1 (default).
    random_state : int (default: None)
        Set random state for shuffling and initializing the weights.

    Attributes
    -----------
    cost_ : list
      Sum of squared errors after each epoch.

    """
    def __init__(self, n_output, n_features, n_hidden=30,
                 l1=0.0, l2=0.0, epochs=500, eta=0.001,
                 alpha=0.0, decrease_const=0.0, shuffle=True,
                 minibatches=1, random_state=None):

        np.random.seed(random_state)
        self.n_output = n_output
        self.n_features = n_features
        self.n_hidden = n_hidden
        self.w1, self.w2 = self._initialize_weights()
        self.l1 = l1
        self.l2 = l2
        self.epochs = epochs
        self.eta = eta
        self.alpha = alpha
        self.decrease_const = decrease_const
        self.shuffle = shuffle
        self.minibatches = minibatches

    def _encode_labels(self, y, k):
        """Encode labels into one-hot representation

        Parameters
        ------------
        y : array, shape = [n_samples]
            Target values.

        Returns
        -----------
        onehot : array, shape = (n_labels, n_samples)

        """
        onehot = np.zeros((k, y.shape[0]))
        for idx, val in enumerate(y):
            onehot[val, idx] = 1.0
        return onehot

    def _initialize_weights(self):
        """Initialize weights with small random numbers."""
        w1 = np.random.uniform(-1.0, 1.0,
                               size=self.n_hidden*(self.n_features + 1))
        w1 = w1.reshape(self.n_hidden, self.n_features + 1)
        w2 = np.random.uniform(-1.0, 1.0,
                               size=self.n_output*(self.n_hidden + 1))
        w2 = w2.reshape(self.n_output, self.n_hidden + 1)
        return w1, w2

    def _sigmoid(self, z):
        """Compute logistic function (sigmoid)

        Uses scipy.special.expit to avoid overflow
        error for very small input values z.

        """
        # return 1.0 / (1.0 + np.exp(-z))
        return expit(z)

    def _sigmoid_gradient(self, z):
        """Compute gradient of the logistic function"""
        sg = self._sigmoid(z)
        return sg * (1.0 - sg)

    def _add_bias_unit(self, X, how='column'):
        """Add bias unit (column or row of 1s) to array at index 0"""
        if how == 'column':
            X_new = np.ones((X.shape[0], X.shape[1] + 1))
            X_new[:, 1:] = X
        elif how == 'row':
            X_new = np.ones((X.shape[0] + 1, X.shape[1]))
            X_new[1:, :] = X
        else:
            raise AttributeError('`how` must be `column` or `row`')
        return X_new

    def _feedforward(self, X, w1, w2):
        """Compute feedforward step

        Parameters
        -----------
        X : array, shape = [n_samples, n_features]
            Input layer with original features.
        w1 : array, shape = [n_hidden_units, n_features]
            Weight matrix for input layer -> hidden layer.
        w2 : array, shape = [n_output_units, n_hidden_units]
            Weight matrix for hidden layer -> output layer.

        Returns
        ----------
        a1 : array, shape = [n_samples, n_features+1]
            Input values with bias unit.
        z2 : array, shape = [n_hidden, n_samples]
            Net input of hidden layer.
        a2 : array, shape = [n_hidden+1, n_samples]
            Activation of hidden layer.
        z3 : array, shape = [n_output_units, n_samples]
            Net input of output layer.
        a3 : array, shape = [n_output_units, n_samples]
            Activation of output layer.

        """
        a1 = self._add_bias_unit(X, how='column')
        z2 = w1.dot(a1.T)
        a2 = self._sigmoid(z2)
        a2 = self._add_bias_unit(a2, how='row')
        z3 = w2.dot(a2)
        a3 = self._sigmoid(z3)
        return a1, z2, a2, z3, a3

    def _L2_reg(self, lambda_, w1, w2):
        """Compute L2-regularization cost"""
        return (lambda_/2.0) * (np.sum(w1[:, 1:] ** 2) +
                                np.sum(w2[:, 1:] ** 2))

    def _L1_reg(self, lambda_, w1, w2):
        """Compute L1-regularization cost"""
        # Li-Yi: there shouldn't be that 2.0 term for L1
        scale = 2.0 # original code
        scale = 1.0 # correct code (I think)
        return (lambda_/scale) * (np.abs(w1[:, 1:]).sum() +
                                  np.abs(w2[:, 1:]).sum())

    def _get_cost(self, y_enc, output, w1, w2):
        """Compute cost function.

        Parameters
        ----------
        y_enc : array, shape = (n_labels, n_samples)
            one-hot encoded class labels.
        output : array, shape = [n_output_units, n_samples]
            Activation of the output layer (feedforward)
        w1 : array, shape = [n_hidden_units, n_features]
            Weight matrix for input layer -> hidden layer.
        w2 : array, shape = [n_output_units, n_hidden_units]
            Weight matrix for hidden layer -> output layer.

        Returns
        ---------
        cost : float
            Regularized cost.

        """
        term1 = -y_enc * (np.log(output))
        term2 = (1.0 - y_enc) * np.log(1.0 - output)
        cost = np.sum(term1 - term2)
        L1_term = self._L1_reg(self.l1, w1, w2)
        L2_term = self._L2_reg(self.l2, w1, w2)
        cost = cost + L1_term + L2_term
        return cost

    def _get_gradient(self, a1, a2, a3, z2, y_enc, w1, w2):
        """ Compute gradient step using backpropagation.

        Parameters
        ------------
        a1 : array, shape = [n_samples, n_features+1]
            Input values with bias unit.
        a2 : array, shape = [n_hidden+1, n_samples]
            Activation of hidden layer.
        a3 : array, shape = [n_output_units, n_samples]
            Activation of output layer.
        z2 : array, shape = [n_hidden, n_samples]
            Net input of hidden layer.
        y_enc : array, shape = (n_labels, n_samples)
            one-hot encoded class labels.
        w1 : array, shape = [n_hidden_units, n_features]
            Weight matrix for input layer -> hidden layer.
        w2 : array, shape = [n_output_units, n_hidden_units]
            Weight matrix for hidden layer -> output layer.

        Returns
        ---------
        grad1 : array, shape = [n_hidden_units, n_features]
            Gradient of the weight matrix w1.
        grad2 : array, shape = [n_output_units, n_hidden_units]
            Gradient of the weight matrix w2.
        both include the bias terms to match w1 and w2
        """
        # backpropagation
        sigma3 = a3 - y_enc # [num_output_units, num_samples]
        z2 = self._add_bias_unit(z2, how='row')
        # Li-Yi: we don't really need z2 as sigmoid_gradient(z2) = a2*(1-a2)
        sigma2 = w2.T.dot(sigma3) * self._sigmoid_gradient(z2) # [n_hidden_units, num_samples]
        sigma2 = sigma2[1:, :] # remove the bias term
        grad1 = sigma2.dot(a1)
        grad2 = sigma3.dot(a2.T)

        # regularize
        # Li-Yi: the L1 part doesn't seem correct to me
        # original code
        # grad1[:, 1:] += (w1[:, 1:] * (self.l1 + self.l2))
        # grad2[:, 1:] += (w2[:, 1:] * (self.l1 + self.l2))
        # corrected code
        # grad1[:, 1:] += (np.sign(w1[:, 1:]) * self.l1 + w1[:, 1:] * self.l2)
        # grad2[:, 1:] += (np.sign(w2[:, 1:]) * self.l1 + w2[:, 1:] * self.l2)

        grad1[:, 1:] += self.l2 * w1[:, 1:]
        grad1[:, 1:] += self.l1 * np.sign(w1[:, 1:])
        grad2[:, 1:] += self.l2 * w2[:, 1:]
        grad2[:, 1:] += self.l1 * np.sign(w2[:, 1:])

        return grad1, grad2

    def predict(self, X):
        """Predict class labels

        Parameters
        -----------
        X : array, shape = [n_samples, n_features]
            Input layer with original features.

        Returns:
        ----------
        y_pred : array, shape = [n_samples]
            Predicted class labels.

        """
        if len(X.shape) != 2:
            raise AttributeError('X must be a [n_samples, n_features] array.\n'
                                 'Use X[:,None] for 1-feature classification,'
                                 '\nor X[[i]] for 1-sample classification')

        a1, z2, a2, z3, a3 = self._feedforward(X, self.w1, self.w2)
        y_pred = np.argmax(a3, axis=0) #Li-Yi: original code says z3 which doesn't seem right
        return y_pred

    def fit(self, X, y, print_progress=False):
        """ Learn weights from training data.

        Parameters
        -----------
        X : array, shape = [n_samples, n_features]
            Input layer with original features.
        y : array, shape = [n_samples]
            Target class labels.
        print_progress : bool (default: False)
            Prints progress as the number of epochs
            to stderr.

        Returns:
        ----------
        self

        """
        self.cost_ = []
        X_data, y_data = X.copy(), y.copy()
        y_enc = self._encode_labels(y, self.n_output)

        delta_w1_prev = np.zeros(self.w1.shape)
        delta_w2_prev = np.zeros(self.w2.shape)

        for i in range(self.epochs):

            # adaptive learning rate
            self.eta /= (1 + self.decrease_const*i)

            if print_progress:
                sys.stderr.write('\rEpoch: %d/%d' % (i+1, self.epochs))
                sys.stderr.flush()

            if self.shuffle:
                idx = np.random.permutation(y_data.shape[0])
                X_data, y_enc = X_data[idx], y_enc[:, idx]

            mini = np.array_split(range(y_data.shape[0]), self.minibatches)
            for idx in mini:

                # feedforward
                a1, z2, a2, z3, a3 = self._feedforward(X_data[idx],
                                                       self.w1,
                                                       self.w2)
                cost = self._get_cost(y_enc=y_enc[:, idx],
                                      output=a3,
                                      w1=self.w1,
                                      w2=self.w2)
                self.cost_.append(cost)

                # compute gradient via backpropagation
                grad1, grad2 = self._get_gradient(a1=a1, a2=a2,
                                                  a3=a3, z2=z2,
                                                  y_enc=y_enc[:, idx],
                                                  w1=self.w1,
                                                  w2=self.w2)

                delta_w1, delta_w2 = self.eta * grad1, self.eta * grad2
                self.w1 -= (delta_w1 + (self.alpha * delta_w1_prev))
                self.w2 -= (delta_w2 + (self.alpha * delta_w2_prev))
                delta_w1_prev, delta_w2_prev = delta_w1, delta_w2

        return self
```

## Experiment of MLP with MNIST dataset

```python
num_epochs = 1000
num_batches = 50

nn = NeuralNetMLP(n_output=10,
                  n_features=X_train.shape[1],
                  n_hidden=50,
                  l2=0.1,
                  l1=0.0,
                  epochs=num_epochs,
                  eta=0.001,
                  alpha=0.001,
                  decrease_const=0.00001,
                  minibatches=num_batches,
                  shuffle=True,
                  random_state=1)
```

```python
_ = nn.fit(X_train, y_train, print_progress=True)
```

    Epoch: 1000/1000

```python
import matplotlib.pyplot as plt

# plot the cost for each mini-batch
plt.plot(range(len(nn.cost_)), nn.cost_)
plt.ylim([0, 2000])
plt.ylabel('Cost')
# plt.xlabel('Epochs * ' + str(num_batches))
plt.xlabel('Mini-batches')
plt.tight_layout()
# plt.savefig('./figures/cost.png', dpi=300)
plt.show()
```

The cost curve is noisy due to the use of stochastic gradient descent.

We can smooth it out by averaging the cost for each epoch.

```python
# plot the average cost per epoch
batches = np.array_split(range(len(nn.cost_)), num_epochs)
cost_ary = np.array(nn.cost_)
cost_avgs = [np.mean(cost_ary[i]) for i in batches]

plt.plot(range(len(cost_avgs)), cost_avgs, color='red')
plt.ylim([0, 2000])
plt.ylabel('Cost')
plt.xlabel('Epochs')
plt.tight_layout()
#plt.savefig('./figures/cost2.png', dpi=300)
plt.show()
```

The process converged around 800 epochs.

```python
y_train_pred = nn.predict(X_train)

if sys.version_info < (3, 0):
    acc = ((np.sum(y_train == y_train_pred, axis=0)).astype('float') /
           X_train.shape[0])
else:
    acc = np.sum(y_train == y_train_pred, axis=0) / X_train.shape[0]

print('Training accuracy: %.2f%%' % (acc * 100))
```

    Training accuracy: 97.61%

```python
y_test_pred = nn.predict(X_test)

if sys.version_info < (3, 0):
    acc = ((np.sum(y_test == y_test_pred, axis=0)).astype('float') /
           X_test.shape[0])
else:
    acc = np.sum(y_test == y_test_pred, axis=0) / X_test.shape[0]

print('Test accuracy: %.2f%%' % (acc * 100))
```

    Test accuracy: 95.69%

Good accuracy (low bias)
* both the training and test accuracy are pretty good

Slight overfitting (low variance)
* test accuracy is slightly worse than training accuracy

We can tune the hyper-parameters via cross-validation and grid search, etc.
* exercise

Visualize the mis-classified digits:

```python
miscl_img = X_test[y_test != y_test_pred][:25]
correct_lab = y_test[y_test != y_test_pred][:25]
miscl_lab = y_test_pred[y_test != y_test_pred][:25]

fig, ax = plt.subplots(nrows=5, ncols=5, sharex=True, sharey=True)
ax = ax.flatten()
for i in range(25):
    img = miscl_img[i].reshape(28, 28)
    ax[i].imshow(img, cmap='Greys', interpolation='nearest')
    ax[i].set_title('%d) t: %d p: %d' % (i+1, correct_lab[i], miscl_lab[i]))

ax[0].set_xticks([])
ax[0].set_yticks([])
plt.tight_layout()
# plt.savefig('./figures/mnist_miscl.png', dpi=300)
plt.show()
```

Some of these are even difficult for humans to classify correctly.

# Training an artificial neural network

Now we see a neural network implementation and experimentation, we can talk about the training part.

## Computing the logistic cost function

Similar to the single-layer case, we can apply logistic cost function to the output layer:

$$
\begin{align}
\mathbf{a} &= \phi\left(\mathbf{z}\right)
\end{align}
$$

Specifically, the cost function will be:
$$
\begin{align}
L\left(\mathbf{W}, \mathbf{X}, \mathbf{Y} \right) &= \sum_i -\mathbf{y}^{(i)}\log\left(\mathbf{a}^{(i)}\right) - \left(1 - \mathbf{y}^{(i)} \right) \log\left(1 - \mathbf{a}^{(i)} \right)
\end{align}
$$
* $i$ runs through all training samples
* $\mathbf{y}$ one-hot vector for classification
$$
\begin{align}
\mathbf{a^{(i)}} =
\begin{bmatrix}
0.1
\\
0.9
\\
\vdots
\\
0.3
\end{bmatrix}
,
\mathbf{y^{(i)}} =
\begin{bmatrix}
0
\\
1
\\
\vdots
\\
0
\end{bmatrix}
\end{align}
$$
* $\mathbf{W}$ is the set of all weights in the network (across all layers)

## Regularization

We can apply regularization with strength $\lambda$ similar to other ML models:

$L_2$:
$$
\begin{align}
P(\mathbf{W}) &= \frac{1}{2} \| \mathbf{W} \|_2^2
\\
&= \frac{1}{2} \sum_{\ell} \sum_k \sum_j \left(\mathbf{W}_{jk}^{(\ell)}\right)^2
\end{align}
$$

$L_1$:
$$
\begin{align}
P(\mathbf{W}) &= \| \mathbf{W} \|_1^1
\\
&= \sum_{\ell} \sum_k \sum_j \left|\mathbf{W}_{jk}^{(\ell)}\right|
\end{align}
$$

$$
\begin{align}
J\left(\mathbf{W}\right) = L\left(\mathbf{W}, \mathbf{X}, \mathbf{Y} \right) + \lambda P\left(\mathbf{W}\right)
\end{align}
$$

## Gradient descent

Need to compute partial derivative for each weight parameter
* for all weights connecting node $k$ in layer $\ell$ to node $j$ in layer $\ell +1$
$$
\frac{\partial J}{\partial \mathbf{W}_{jk}^{(\ell)}}
$$

Then
$$
\begin{align}
\delta \mathbf{W} &= -\eta \frac{\partial J}{\partial \mathbf{W}}
\\
\mathbf{W} &\leftarrow \mathbf{W} + \delta \mathbf{W}
\end{align}
$$

Learn by small perturbation of the parameters $\mathbf{W}$

In a single-hidden layer network, we have two layers of weights.
* $\mathbf{W}^{(1)}$ connects input layer to hidden layer
* $\mathbf{W}^{(2)}$ connects hidden layer to output layer
* each $\mathbf{W}^{(\ell)}$ has size $N_{\ell+1} \times N_{\ell}$, where $N_{\ell}$ is the number of nodes at layer $\ell$

## Training neural networks via backpropagation

### Forward propagation

Start from the input layer (initialization):
$$
\begin{align}
\mathbf{A}^{1} = \mathbf{X}^T
\end{align}
$$

For each layer $\ell \in [1, 2, \cdots, L-1]$ where the last layer is $L$:
$$
\begin{align}
\mathbf{Z}^{(\ell + 1)} &= \mathbf{W}^{(\ell)} \mathbf{A}^{(\ell)}
\\
\mathbf{A}^{(\ell + 1)} &= \phi\left( \mathbf{Z}^{(\ell+1)}\right)
\end{align}
$$

Compute the output via certain output function $o$ (thresholding, one-hot, softmax, etc.):
$$
\hat{\mathbf{Y}} = o\left(\mathbf{A}^{(L)}\right)
$$

### Backward propagation

Let's start with one sample at a time.
* i.e. mini-batch with size 1

We first compute the $\delta$ terms for all the layers.
$$
\delta^{(\ell)} = \frac{\partial J}{\partial \mathbf{z}^{(\ell)}}
$$

$*$ means component-wise multiplication.

Start from the output layer:
$$
\begin{align}
\mathbf{\delta}^{(L)} =
\frac{\partial J\left(\mathbf{a}^{(L)}\right)}{\partial \mathbf{a}}
*
\frac{\phi\left(\mathbf{z}^{(L)}\right)}{\partial \mathbf{z}}
\end{align}
$$

* can be simplified to the following for our loss function above:
$$
\mathbf{\delta}^{(L)} = \mathbf{a^{(L)}} - \mathbf{y}
$$

For each internal layer $\ell \in [L-1, \cdots, 2]$:
$$
\begin{align}
\mathbf{\delta}^{(\ell)} =
\left( \mathbf{W}^{(\ell)} \right)^T
\mathbf{\delta}^{(\ell+1)}
*
\frac{\partial \phi\left( \mathbf{z}^{(\ell)} \right)}{\partial \mathbf{z}}
\end{align}
$$
* for sigmoid $\phi$ we have
$$
\frac{\partial \phi\left(\mathbf{z}\right)}{\partial \mathbf{z}} =
\phi\left(\mathbf{z}\right) * \left(1 - \phi\left(\mathbf{z}\right)\right)
$$

After having all the $\mathbf{\delta}$ terms we can compute the individual derivate as:
$$
\begin{align}
\frac{\partial J}{\partial \mathbf{W}_{i,j}^{(\ell)}}
&=
\mathbf{\delta}_i^{(\ell+1)}
\mathbf{a}_j^{(\ell)}
\end{align}
$$

In matrix form:
$$
\begin{align}
\frac{\partial J}{\partial \mathbf{W}^{(\ell)}}
&=
\mathbf{\delta}^{(\ell+1)}
\left[ \mathbf{a}^{(\ell)} \right]^T
\end{align}
$$

If you want to optimize the input as well (e.g. deep dream) you can propagate further:
$$
\begin{align}
\mathbf{\delta}^{(1)} =
\frac{\partial J}{\partial \mathbf{x}} =
\left(\mathbf{W}^{(1)}\right)^T \mathbf{\delta}^{(2)}
\end{align}
$$

#### Example

Start from the output layer $L=3$:
$$
\begin{align}
\mathbf{\delta}^{(3)} &=
\frac{\partial J\left(\mathbf{a}^{(3)}\right)}{\partial \mathbf{a}}
*
\frac{\phi\left(\mathbf{z}^{(3)}\right)}{\partial \mathbf{z}}
\\
&= \mathbf{a^{(3)}} - \mathbf{y}
\end{align}
$$

For each internal layer $\ell \in [L-1, \cdots, 2]$:
$$
\begin{align}
\mathbf{\delta}^{(2)} =
\left( \mathbf{W}^{(2)} \right)^T
\mathbf{\delta}^{(3)}
*
\frac{\partial \phi\left( \mathbf{z}^{(2)} \right)}{\partial \mathbf{z}}
\end{align}
$$

* for sigmoid $\phi$ we have
$$
\frac{\partial \phi\left(\mathbf{z}\right)}{\partial \mathbf{z}} =
\phi\left(\mathbf{z}\right) * \left(1 - \phi\left(\mathbf{z}\right)\right)
$$

After having all the $\mathbf{\delta}$ terms we can compute the individual derivate as:
$$
\begin{align}
\frac{\partial J}{\partial \mathbf{W}^{(\ell)}}
&=
\mathbf{\delta}^{(\ell+1)}
\left[ \mathbf{a}^{(\ell)} \right]^T
\end{align}
$$

If you want to optimize the input as well (e.g. deep dream) you can propagate further:
$$
\begin{align}
\mathbf{\delta}^{(1)} =
\frac{\partial J}{\partial \mathbf{x}} =
\left(\mathbf{W}^{(1)}\right)^T \mathbf{\delta}^{(2)}
\end{align}
$$

### Backpropagation with batches

Use matrix versions for vectors above by stacking column vectors.

Start from the output layer:
$$
\begin{align}
\mathbf{\delta}^{(L)} =
\frac{\partial J\left(\mathbf{A}^{(L)}\right)}{\partial \mathbf{A}}
*
\frac{\phi\left(\mathbf{Z}^{(L)}\right)}{\partial \mathbf{Z}}
\end{align}
$$
* shape: num-outputs $\times$ mini-batch-size
* can be simplified to the following for our loss function above:
$$
\mathbf{\delta}^{(L)} = \mathbf{A^{(L)}} - \mathbf{Y}
$$

For each internal layer $\ell \in [L-1, \cdots, 2]$:
$$
\begin{align}
\mathbf{\delta}^{(\ell)} =
\left( \mathbf{W}^{(\ell)} \right)^T
\mathbf{\delta}^{(\ell+1)}
*
\frac{\partial \phi\left( \mathbf{Z}^{(\ell)} \right)}{\partial \mathbf{Z}}
\end{align}
$$
* shape: num-nodes-layer-$\ell$ (the input size for $\mathbf{W}^{\ell}$) $\times$ mini-batch-size

After having all the $\mathbf{\delta}$ terms we can compute the individual derivate as:
$$
\begin{align}
\frac{\partial J}{\partial \mathbf{W}^{(\ell)}}
&=
\mathbf{\delta}^{(\ell+1)}
\left[\mathbf{A}^{(\ell)}\right]^T
\end{align}
$$

### Backprogagation with regularization

For $L_2$:
$$
\begin{align}
P\left(\mathbf{W}\right) &= \frac{1}{2} \| \mathbf{W} \|_2^2
\\
\frac{\partial P}{\partial \mathbf{W}} &= \mathbf{W}
\end{align}
$$

For $L_1$:
$$
\begin{align}
P\left(\mathbf{W}\right) &= \| \mathbf{W} \|_1^1
\\
\frac{\partial P}{\partial \mathbf{W}} &= sign(W)
\end{align}
$$

## Gradient descent

For all layers $\ell \in [1, 2, \cdots, L-1]$:
$$
\begin{align}
\mathbf{W}^{(\ell)}
\leftarrow
\mathbf{W}^{(\ell)} - \eta \left( \frac{\partial J}{\partial \mathbf{W}^{(\ell)}} + \lambda \frac{\partial P}{\partial \mathbf{W}^{(\ell)}} \right)
\end{align}
$$

# Developing your intuition for backpropagation

| propagatioin |  forward  | backward |
|--|-----------|----------|
|key equation| $ \mathbf{A}^{(\ell + 1)} = \phi\left( \mathbf{W}^{(\ell)} \mathbf{A}^{\ell} \right) $ | $ \mathbf{\delta}^{(\ell)} = \left( \mathbf{W}^{(\ell)} \right)^T \mathbf{\delta}^{(\ell+1)} * \phi^{\prime}\left( \mathbf{Z}^{(\ell)} \right)$ |
|direction| input $\rightarrow$ output | input $\leftarrow$ output |
|key quantity | $\mathbf{A}$ | $\mathbf{\delta}$ |
|$\mathbf{W}$ orientation | $\mathbf{W}^{(\ell)}$ | $\left[\mathbf{W}^{(\ell)}\right]^T$|
|$\phi$ | original | derivative  |

Back-propagation has similar time complexity as forward-propagation.
* very important for efficiency
* prior to the discovery of back-propagation, training is very slow

## Derivation

Essentially an application of chain-rule
$$
\begin{align}
y &= f\left(g\left(x\right)\right)
\\
\frac{\partial y}{\partial x} &= \frac{\partial f}{\partial g} \frac{\partial g}{\partial x}
\end{align}
$$

We will use the single-sample (mini-batch size 1) version below for clarity.
The matrix (general mini-batch size) version is similar.

Output layer $L$:
$$
\begin{align}
\mathbf{\delta}^{(L)} &= \frac{\partial J\left(\mathbf{a}^{(L)}\right)}{\partial \mathbf{z}}
\\
&= \frac{\partial J\left(\mathbf{a}^{(L)}\right)}{\partial \mathbf{a}}
*
\frac{\partial \mathbf{a}\left(\mathbf{z}^{(L)}\right)}{\partial \mathbf{z}}
\\
&=
\frac{\partial J\left(\mathbf{a}^{(L)}\right)}{\partial \mathbf{a}}
*
\frac{\partial \phi\left(\mathbf{z}^{(L)}\right)}{\partial \mathbf{z}}
\end{align}
$$

Internal layer $\ell \in [L-1, \cdots, 2]$:
$$
\begin{align}
\mathbf{\delta}^{(\ell)} &=
\frac{\partial J\left(\mathbf{a}^{(L)}\right)}{\partial \mathbf{z}^{(\ell)}}
\\
&=
\frac{\partial J\left(\mathbf{a}^{(L)}\right)}{\partial \mathbf{a}^{(\ell)}}
*
\frac{\partial \mathbf{a}^{(\ell)}}{\partial \mathbf{z}^{(\ell)}}
\\
&=
\frac{\partial J\left(\mathbf{a}^{(L)}\right)}{\partial \mathbf{z}^{(\ell+1)}}
*
\frac{\partial \mathbf{z}^{(\ell+1)}}{\partial \mathbf{a}^{(\ell)}}
*
\frac{\partial \mathbf{a}^{(\ell)}}{\partial \mathbf{z}^{(\ell)}}
\\
&=
\frac{\partial \mathbf{z}^{(\ell+1)}}{\partial \mathbf{a}^{(\ell)}}
*
\frac{\partial J\left(\mathbf{a}^{(L)}\right)}{\partial \mathbf{z}^{(\ell+1)}}
*
\frac{\partial \mathbf{a}^{(\ell)}}{\partial \mathbf{z}^{(\ell)}}
\\
&=
\frac{\partial \mathbf{z}^{(\ell+1)}}{\partial \mathbf{a}^{(\ell)}}
*
\mathbf{\delta}^{(\ell+1)}
*
\frac{\partial \mathbf{a}^{(\ell)}}{\partial \mathbf{z}^{(\ell)}}
\\
&=
\left( \mathbf{W}^{(\ell)} \right)^T
\mathbf{\delta}^{(\ell+1)}
*
\frac{\partial \phi\left( \mathbf{z}^{(\ell)} \right)}{\partial \mathbf{z}}
\end{align}
$$

Input layer $1$:
$$
\begin{align}
\mathbf{\delta}^{(1)} &=
\frac{\partial J\left( \mathbf{a}^{(L)} \right)}{\partial \mathbf{x}}
\\
&=
\frac{\partial J\left( \mathbf{a}^{(L)} \right)}{\partial \mathbf{z}^{(2)}}
*
\frac{\partial \mathbf{z}^{(2)}}{\partial \mathbf{x}}
\\
&=
\frac{\partial \mathbf{z}^{(2)}}{\partial \mathbf{a}^{(1)}}
*
\mathbf{\delta}^{(2)}
\\
&=
\left(\mathbf{W}^{(1)}\right)^T \mathbf{\delta}^{(2)}
\end{align}
$$

You can derive the inidividual scalar variables $\mathbf{\delta}_k^{(\ell)}$ if the vector/matrix version above is still not clear.

# Debugging neural networks with gradient checking

We can compare analytical gradients and numerical gradients for debugging.

Numerical gradient:
$$
\begin{align}
\frac{\partial J\left(\mathbf{W} \right)}{{\partial \mathbf{W}_{i,j}^{(\ell)}}}
\simeq
\frac{J\left(\mathbf{W}_{i,j}^{(\ell)} + \epsilon \right) - J \left( \mathbf{W}_{i,j}^{(\ell)}\right)}{\epsilon}
\end{align}
$$
for sufficiently small $\epsilon$.

Symmetric version is better:
$$
\begin{align}
\frac{\partial J\left(\mathbf{W} \right)}{{\partial \mathbf{W}_{i,j}^{(\ell)}}}
\simeq
\frac{J\left(\mathbf{W}_{i,j}^{(\ell)} + \epsilon \right) - J \left( \mathbf{W}_{i,j}^{(\ell)}- \epsilon \right)}{2\epsilon}
\end{align}
$$

Sample code below

Make it simple to minimize errors in the check code
* very slow, compared to back-propagation
* brute-force OK for testing sufficiently small cases (e.g. a small number of inputs)

```python
from scipy import __version__ as scipy_ver
from numpy import __version__ as numpy_ver

print('The following code requires NumPy >= 1.9.1. Your NumPy version is %s.' % (numpy_ver))
print('The following code requires SciPy >= 0.14.0. Your SciPy version is %s. ' % (scipy_ver))
```

    The following code requires NumPy >= 1.9.1. Your NumPy version is 1.11.1.
    The following code requires SciPy >= 0.14.0. Your SciPy version is 0.17.1.

```python
import numpy as np
from scipy.special import expit
import sys

# Li-Yi: I removed duplicated code below via inheritance
# We can and should use another class for gradient check
# but this is machine learning not software engineering
# so I try not to go overboard
class MLPGradientCheck(NeuralNetMLP):

    def _gradient_checking(self, X, y_enc, w1, w2, epsilon, grad1, grad2):
        """ Apply gradient checking (for debugging only)

        Returns
        ---------
        relative_error : float
          Relative error between the numerically
          approximated gradients and the backpropagated gradients.

        """
        num_grad1 = np.zeros(np.shape(w1))
        epsilon_ary1 = np.zeros(np.shape(w1))
        for i in range(w1.shape[0]):
            for j in range(w1.shape[1]):
                epsilon_ary1[i, j] = epsilon
                a1, z2, a2, z3, a3 = self._feedforward(X,
                                                       w1 - epsilon_ary1, w2)
                cost1 = self._get_cost(y_enc, a3, w1-epsilon_ary1, w2)
                a1, z2, a2, z3, a3 = self._feedforward(X,
                                                       w1 + epsilon_ary1, w2)
                cost2 = self._get_cost(y_enc, a3, w1 + epsilon_ary1, w2)
                num_grad1[i, j] = (cost2 - cost1) / (2.0 * epsilon)
                epsilon_ary1[i, j] = 0

        num_grad2 = np.zeros(np.shape(w2))
        epsilon_ary2 = np.zeros(np.shape(w2))
        for i in range(w2.shape[0]):
            for j in range(w2.shape[1]):
                epsilon_ary2[i, j] = epsilon
                a1, z2, a2, z3, a3 = self._feedforward(X, w1,
                                                       w2 - epsilon_ary2)
                cost1 = self._get_cost(y_enc, a3, w1, w2 - epsilon_ary2)
                a1, z2, a2, z3, a3 = self._feedforward(X, w1,
                                                       w2 + epsilon_ary2)
                cost2 = self._get_cost(y_enc, a3, w1, w2 + epsilon_ary2)
                num_grad2[i, j] = (cost2 - cost1) / (2.0 * epsilon)
                epsilon_ary2[i, j] = 0

        num_grad = np.hstack((num_grad1.flatten(), num_grad2.flatten()))
        grad = np.hstack((grad1.flatten(), grad2.flatten()))
        norm1 = np.linalg.norm(num_grad - grad)
        norm2 = np.linalg.norm(num_grad)
        norm3 = np.linalg.norm(grad)
        relative_error = norm1 / (norm2 + norm3)
        return relative_error

    def fit(self, X, y, print_progress=False):
        """ Learn weights from training data.

        Parameters
        -----------
        X : array, shape = [n_samples, n_features]
            Input layer with original features.
        y : array, shape = [n_samples]
            Target class labels.
        print_progress : bool (default: False)
            Prints progress as the number of epochs
            to stderr.

        Returns:
        ----------
        self

        """
        self.cost_ = []
        X_data, y_data = X.copy(), y.copy()
        y_enc = self._encode_labels(y, self.n_output)

        delta_w1_prev = np.zeros(self.w1.shape)
        delta_w2_prev = np.zeros(self.w2.shape)

        for i in range(self.epochs):

            # adaptive learning rate
            self.eta /= (1 + self.decrease_const*i)

            if print_progress:
                sys.stderr.write('\rEpoch: %d/%d' % (i+1, self.epochs))
                sys.stderr.flush()

            if self.shuffle:
                idx = np.random.permutation(y_data.shape[0])
                X_data, y_enc = X_data[idx], y_enc[idx]

            mini = np.array_split(range(y_data.shape[0]), self.minibatches)
            for idx in mini:

                # feedforward
                a1, z2, a2, z3, a3 = self._feedforward(X[idx],
                                                       self.w1,
                                                       self.w2)
                cost = self._get_cost(y_enc=y_enc[:, idx],
                                      output=a3,
                                      w1=self.w1,
                                      w2=self.w2)
                self.cost_.append(cost)

                # compute gradient via backpropagation
                grad1, grad2 = self._get_gradient(a1=a1, a2=a2,
                                                  a3=a3, z2=z2,
                                                  y_enc=y_enc[:, idx],
                                                  w1=self.w1,
                                                  w2=self.w2)

                # start gradient checking
                grad_diff = self._gradient_checking(X=X_data[idx],
                                                    y_enc=y_enc[:, idx],
                                                    w1=self.w1,
                                                    w2=self.w2,
                                                    epsilon=1e-5,
                                                    grad1=grad1,
                                                    grad2=grad2)

                if grad_diff <= 1e-7:
                    print('Ok: %s' % grad_diff)
                elif grad_diff <= 1e-4:
                    print('Warning: %s' % grad_diff)
                else:
                    print('PROBLEM: %s' % grad_diff)

                # update weights; [alpha * delta_w_prev] for momentum learning
                delta_w1, delta_w2 = self.eta * grad1, self.eta * grad2
                self.w1 -= (delta_w1 + (self.alpha * delta_w1_prev))
                self.w2 -= (delta_w2 + (self.alpha * delta_w2_prev))
                delta_w1_prev, delta_w2_prev = delta_w1, delta_w2

        return self
```

```python
nn_check = MLPGradientCheck(n_output=10,
                            n_features=X_train.shape[1],
                            n_hidden=10,
                            l2=0.0,
                            l1=0.0,
                            epochs=10,
                            eta=0.001,
                            alpha=0.0,
                            decrease_const=0.0,
                            minibatches=1,
                            shuffle=False,
                            random_state=1)
```

```python
_ = nn_check.fit(X_train[:5], y_train[:5], print_progress=False)
```

    Ok: 2.59683296507e-10
    Ok: 2.945509836e-10
    Ok: 2.42566311219e-10
    Ok: 2.2457046552e-10
    Ok: 3.37843531501e-10
    Ok: 3.6260415027e-10
    Ok: 2.22383718134e-10
    Ok: 2.33082209854e-10
    Ok: 3.33225341888e-10
    Ok: 2.13281783941e-10

# Convergence in neural networks

Compared to gradient descent, stochastic gradient descent can
* compute faster
* adapt better (online learning)
* better avoid local minimum

This is especially so for complex optimization problems such as neural networks with many parameters.

There are many other techniques for better train deep networks
* dropout, etc.

# Other neural network architectures

Multi-layer perceptron (MLP) is a basic and common neural network architecture.
* fully connected

There are other more recent, popular network architectures:
* convolutional neural networks (CNN)
* recurrent neural networks (RNN)
* etc.

This is a very active research field.

## Convolutional Neural Networks

A multi-layer neural network is like a multi-resolution image pyramid.

Use weight connection to emulate convolution kernels in image processing.
* receptive field

Each node in layer $\ell+1$ (feature map) is only connects to a small set of nodes (receptive field) in layer $\ell$.
* kernel weights are fixed and shared among all nodes in the same layer

CNN is sparser and implicitly regularized than MLP, and thus can be deeper/wider.

Very effective for image classification and related applications.

A CNN can contain a variety of layers
* convolution
* pooling (max, min, average followed by sub-sampling)
* fully connected

## Recurrent Neural Networks

MLP is feed forward only.

RNN allows feed backward through time (steps).
* essentially MLP + time variable
* or multiple MPL stacked together through time steps

Harder to train than MLP
* long short term memory (LSTM) can help

Very effective for sequential/temporal applications
* speech analysis and synthesis
* languate translation
* hand-writing synthesis
* image/video captioning

# Generality of neural networks

A MLP can emulate any functions

A RNN can emulate any programs

# A few last words about neural network implementation

We use hand-written MLP code for didactic purpose here.

Deep learning libraries exist that we can directly use.
* e.g. Keras, Torch7, Theano, Caffe, TensorFlow, etc.

https://www.quora.com/Is-TensorFlow-better-than-other-leading-libraries-such-as-Torch-Theano

We will introduce Keras and Theano next.

# Summary

We learned the basic concepts and architecture of multi-layer neural networks.
* forward-propagation for prediction
* backward-propagation for training

Other neural network architectures
* convolutional neural networks (CNN)
* recurrent neural networks (RNN)

Neural network libraries

# Reading

* PML Chapter 12
* IML Chapter 11
* [Neural networks and deep learning](http://neuralnetworksanddeeplearning.com/)