# 01 - Introduction to seismic modelling

This notebook is the first in a series of tutorials highlighting various aspects of seismic inversion based on Devito operators. In this first example we aim to highlight the core ideas behind seismic modelling, where we create a numerical model that captures the processes involved in a seismic survey. This forward model will then form the basis for further tutorials on the implementation of inversion processes using Devito operators.

## Modelling workflow

The core process we are aiming to model is a seismic survey, which consists of two main components:

- **Source** - A source is positioned at a single or a few physical locations where artificial pressure is injected into the domain we want to model. In the case of land survey, it is usually dynamite blowing up at a given location, or a vibroseis (a vibrating engine generating continuous sound waves). For a marine survey, the source is an air gun sending a bubble of compressed air into the water that will expand and generate a seismic wave.
- **Receiver** - A set of microphones or hydrophones are used to measure the resulting wave and create a set of measurements called a *Shot Record*. These measurements are recorded at multiple locations, and usually at the surface of the domain or at the bottom of the ocean in some marine cases.

In order to create a numerical model of a seismic survey, we need to solve the wave equation and implement source and receiver interpolation to inject the source and record the seismic wave at sparse point locations in the grid.

## The acoustic seismic wave equation
The acoustic wave equation for the square slowness $m$, defined as $m=\frac{1}{c^2}$, where $c$ is the speed of sound in the given physical media, and a source $q$ is given by:

\begin{cases}
 &m \frac{d^2 u(x,t)}{dt^2} - \nabla^2 u(x,t) = q \ \text{in } \Omega \\
 &u(.,t=0) = 0 \\
 &\frac{d u(x,t)}{dt}|_{t=0} = 0
\end{cases}

with the zero initial conditions to guarantee unicity of the solution.
The boundary conditions are Dirichlet conditions:
\begin{equation}
 u(x,t)|_\delta\Omega = 0
\end{equation}

where $\delta\Omega$ is the surface of the boundary of the model $\Omega$.

# Finite domains

The last piece of the puzzle is the computational limitation. In the field, the seismic wave propagates in every direction to an "infinite" distance. However, solving the wave equation in a mathematically/discrete infinite domain is not feasible. In order to compensate, Absorbing Boundary Conditions (ABC) or Perfectly Matched Layers (PML) are required to mimic an infinite domain. These two methods allow to approximate an infinite media by damping and absorbing the waves at the limit of the domain to avoid reflections.

The simplest of these methods is the absorbing damping mask. The core idea is to extend the physical domain and to add a Sponge mask in this extension that will absorb the incident waves. The acoustic wave equation with this damping mask can be rewritten as:

\begin{cases}
 &m \frac{d^2 u(x,t)}{dt^2} - \nabla^2 u(x,t) + \eta \frac{d u(x,t)}{dt}=q  \ \text{in } \Omega \\
 &u(.,0) = 0 \\
 &\frac{d u(x,t)}{dt}|_{t=0} = 0
\end{cases}

where $\eta$ is the damping mask equal to $0$ inside the physical domain and increasing inside the sponge layer. Multiple choice of profile can be chosen for $\eta$ from linear to exponential.

# Seismic modelling with devito

We describe here a step by step setup of seismic modelling with Devito in a simple 2D case. We will create a physical model of our domain and define a single source and an according set of receivers to model for the forward model. But first, we initialize some basic utilities.

```python
import numpy as np
%matplotlib inline
```

## Define the physical problem

The first step is to define the physical model:

- What are the physical dimensions of interest
- What is the velocity profile of this physical domain

We will create a simple velocity model here by hand for demonstration purposes. This model essentially consists of two layers, each with a different velocity: $1.5km/s$ in the top layer and $2.5km/s$ in the bottom layer. We will use this simple model a lot in the following tutorials, so we will rely on a utility function to create it again later.

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import Model, plot_velocity

# Define a physical size
shape = (101, 101)  # Number of grid point (nx, nz)
spacing = (10., 10.)  # Grid spacing in m. The domain size is now 1km by 1km
origin = (0., 0.)  # What is the location of the top left corner. This is necessary to define
# the absolute location of the source and receivers

# Define a velocity profile. The velocity is in km/s
v = np.empty(shape, dtype=np.float32)
v[:, :51] = 1.5
v[:, 51:] = 2.5

# With the velocity and model size defined, we can create the seismic model that
# encapsulates this properties. We also define the size of the absorbing layer as 10 grid points
model = Model(vp=v, origin=origin, shape=shape, spacing=spacing,
              space_order=2, nbpml=10)

plot_velocity(model)
```

# Acquisition geometry

To fully define our problem setup we also need to define the source that injects the wave to model and the set of receiver locations at which to sample the wavefield. The source time signature will be modelled using a Ricker wavelet defined as

\begin{equation}
  q(t) = (1-2\pi^2 f_0^2 (t - \frac{1}{f_0})^2 )e^{- \pi^2 f_0^2 (t - \frac{1}{f_0})}
\end{equation}

To fully define the source signature we first need to define the time duration for our model and the timestep size, which is dictated by the CFL condition and our grid spacing. Luckily, our `Model` utility provides us with the critical timestep size, so we can fully discretize our model time axis as an array:

```python
from examples.seismic import TimeAxis

t0 = 0.  # Simulation starts a t=0
tn = 1000.  # Simulation last 1 second (1000 ms)
dt = model.critical_dt  # Time step from model grid spacing

time_range = TimeAxis(start=t0, stop=tn, step=dt)
```

The source is positioned at a $20m$ depth and at the middle of the $x$ axis ($x_{src}=500m$), with a peak wavelet frequency of $10Hz$.

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import RickerSource

f0 = 0.010  # Source peak frequency is 10Hz (0.010 kHz)
src = RickerSource(name='src', grid=model.grid, f0=f0,
                   npoint=1, time_range=time_range)

# First, position source centrally in all dimensions, then set depth
src.coordinates.data[0, :] = np.array(model.domain_size) * .5
src.coordinates.data[0, -1] = 20.  # Depth is 20m

# We can plot the time signature to see the wavelet
src.show()
```

Similarly to our source object, we can now define our receiver geometry as a symbol of type `Receiver`. It is worth noting here that both utility classes, `RickerSource` and `Receiver` are thin wrappers around the Devito's `SparseTimeFunction` type, which encapsulates sparse point data and allows us to inject and interpolate values into and out of the computational grid. As we have already seen, both types provide a `.coordinates` property to define the position within the domain of all points encapsulated by that symbol.

In this example we will position receivers at the same depth as the source, every $10m$ along the x axis. The `rec.data` property will be initialized, but left empty, as we will compute the receiver readings during the simulation.

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import Receiver

# Create symbol for 101 receivers
rec = Receiver(name='rec', grid=model.grid, npoint=101, time_range=time_range)

# Prescribe even spacing for receivers along the x-axis
rec.coordinates.data[:, 0] = np.linspace(0, model.domain_size[0], num=101)
rec.coordinates.data[:, 1] = 20.  # Depth is 20m

# We can now show the source and receivers within our domain:
# Red dot: Source location
# Green dots: Receiver locations (every 4th point)
plot_velocity(model, source=src.coordinates.data,
              receiver=rec.coordinates.data[::4, :])
```

# Finite-difference discretization

Devito is a finite-difference DSL that solves the discretized wave-equation on a Cartesian grid. The finite-difference approximation is derived from Taylor expansions of the continuous field after removing the error term.

## Time discretization

We only consider the second order time discretization for now. From the Taylor expansion, the second order discrete approximation of the second order time derivative is:
\begin{equation}
\begin{aligned}
 \frac{d^2 u(x,t)}{dt^2} = \frac{\mathbf{u}(\mathbf{x},\mathbf{t+\Delta t}) - 2 \mathbf{u}(\mathbf{x},\mathbf{t}) + \mathbf{u}(\mathbf{x},\mathbf{t-\Delta t})}{\mathbf{\Delta t}^2} + O(\mathbf{\Delta t}^2).
\end{aligned}
\end{equation}

where $\mathbf{u}$ is the discrete wavefield, $\mathbf{\Delta t}$ is the discrete
time-step (distance between two consecutive discrete time points) and $O(\mathbf{\Delta
  t}^2)$ is the discretization error term. The discretized approximation of the
second order time derivative is then given by dropping the error term. This derivative is represented in Devito by `u.dt2` where u is a `TimeFunction` object.

## Spatial discretization

We define the discrete Laplacian as the sum of the second order spatial
derivatives in the three dimensions:
\begin{equation}
\begin{aligned}
\Delta \mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t})= \sum_{j=1}^{j=\frac{k}{2}} \Bigg[\alpha_j \Bigg(&
\mathbf{u}(\mathbf{x+jdx},\mathbf{y},\mathbf{z},\mathbf{t})+\mathbf{u}(\mathbf{x-jdx},\mathbf{y},\mathbf{z},\mathbf{t}) + \\
&\mathbf{u}(\mathbf{x},\mathbf{y+jdy},\mathbf{z},\mathbf{t})+\mathbf{u}(\mathbf{x},\mathbf{y-jdy},\mathbf{z}\mathbf{t}) + \\
&\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z+jdz},\mathbf{t})+\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z-jdz},\mathbf{t})\Bigg) \Bigg] + \\
&3\alpha_0 \mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}).
\end{aligned}
\end{equation}

This derivative is represented in Devito by `u.laplace` where u is a `TimeFunction` object.

## Wave equation

With the space and time discretization defined, we can fully discretize the wave-equation with the combination of time and space discretizations and obtain the following second order in time and $k^{th}$ order in space discrete stencil to update one grid point at position $\mathbf{x}, \mathbf{y},\mathbf{z}$ at time $\mathbf{t}$, i.e.
\begin{equation}
\begin{aligned}
\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t+\Delta t}) = &2\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) - \mathbf{u}(\mathbf{x},\mathbf{y}, \mathbf{z},\mathbf{t-\Delta t}) +\\
& \frac{\mathbf{\Delta t}^2}{\mathbf{m(\mathbf{x},\mathbf{y},\mathbf{z})}} \Big(\Delta \mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) + \mathbf{q}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) \Big).
\end{aligned}
\end{equation}

```python
# In order to represent the wavefield u and the square slowness we need symbolic objects
# corresponding to time-space-varying field (u, TimeFunction) and
# space-varying field (m, Function)
from devito import TimeFunction

# Define the wavefield with the size of the model and the time dimension
u = TimeFunction(name="u", grid=model.grid, time_order=2, space_order=2)

# We can now write the PDE
pde = model.m * u.dt2 - u.laplace + model.damp * u.dt

# The PDE representation is as on paper
pde
```

$\displaystyle \operatorname{damp}{\left(x,y \right)} \frac{\partial}{\partial t} u{\left(t,x,y \right)} - \frac{\partial^{2}}{\partial x^{2}} u{\left(t,x,y \right)} - \frac{\partial^{2}}{\partial y^{2}} u{\left(t,x,y \right)} + \frac{\frac{\partial^{2}}{\partial t^{2}} u{\left(t,x,y \right)}}{\operatorname{vp}^{2}{\left(x,y \right)}}$

```python
# This discrete PDE can be solved in a time-marching way updating u(t+dt) from the previous time step
# Devito as a shortcut for u(t+dt) which is u.forward. We can then rewrite the PDE as
# a time marching updating equation known as a stencil using customized SymPy functions
from devito import Eq, solve

stencil = Eq(u.forward, solve(pde, u.forward))
```

# Source injection and receiver interpolation

With a numerical scheme to solve the homogenous wave equation, we need to add the source to introduce seismic waves and to implement the measurement operator, and interpolation operator. This operation is linked to the discrete scheme and needs to be done at the proper time step. The semi-discretized in time wave equation with a source reads:

\begin{equation}
\begin{aligned}
\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t+\Delta t}) = &2\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) - \mathbf{u}(\mathbf{x},\mathbf{y}, \mathbf{z},\mathbf{t-\Delta t}) +\\
& \frac{\mathbf{\Delta t}^2}{\mathbf{m(\mathbf{x},\mathbf{y},\mathbf{z})}} \Big(\Delta \mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) + \mathbf{q}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) \Big).
\end{aligned}
\end{equation}

It shows that in order to update $\mathbf{u}$ at time $\mathbf{t+\Delta t}$ we have to inject the value of the source term $\mathbf{q}$ of time $\mathbf{t}$. In Devito, it corresponds the update of $u$ at index $t+1$ (t = time implicitly) with the source of time $t$.
On the receiver side, the problem is either as it only requires to record the data at the given time step $t$ for the receiver at time $time=t$.

```python
# Finally we define the source injection and receiver read function to generate the corresponding code
src_term = src.inject(field=u.forward, expr=src * dt**2 / model.m)

# Create interpolation expression for receivers
rec_term = rec.interpolate(expr=u.forward)
```

# Devito operator and solve
After constructing all the necessary expressions for updating the wavefield, injecting the source term and interpolating onto the receiver points, we can now create the Devito operator that will generate the C code at runtime. When creating the operator, Devito's two optimization engines will log which performance optimizations have been performed:
* **DSE:** The Devito Symbolics Engine will attempt to reduce the number of operations required by the kernel.
* **DLE:** The Devito Loop Engine will perform various loop-level optimizations to improve runtime performance.

**Note**: The argument `subs=model.spacing_map` causes the operator to substitute values for our current grid spacing into the expressions before code generation. This reduces the number of floating point operations executed by the kernel by pre-evaluating certain coefficients.

```python
#NBVAL_IGNORE_OUTPUT
from devito import Operator

op = Operator([stencil] + src_term + rec_term, subs=model.spacing_map)
```

Now we can execute the create operator for a number of timesteps. We specify the number of timesteps to compute with the keyword `time` and the timestep size with `dt`.

```python
#NBVAL_IGNORE_OUTPUT
op(time=time_range.num-1, dt=model.critical_dt)
```

    Operator `Kernel` run in 0.01 s

After running our operator kernel, the data associated with the receiver symbol `rec.data` has now been populated due to the interpolation expression we inserted into the operator. This allows us the visualize the shot record:

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import plot_shotrecord

plot_shotrecord(rec.data, model, t0, tn)
```

```python
assert np.isclose(np.linalg.norm(rec.data), 450, rtol=10)
```


# Probabilistic Programming

A Probabilistic Programming Language (PPL) is a computer language providing statistical modelling and inference functionalities, in order to reason about random variables, probability distributions and conditioning problems. The most popular PPLs are `Stan`, `PyMC`, `Pyro` and `Edward`.

A probabilistic program is a mix of **deterministic computation** and **sampling**, which allows to *draw random values* from distributions, to *condition* variables on observations and to perform *inference*.

### Pyro
Pyro is a universal probabilistic programming language based on Python.

It can represent any probabilistic model, while providing automatic optimization-based inference that is flexible and scalable to large data sets.
Pyro builds on PyTorch library, which supports GPU-accelerated tensor math and includes automatic differentiation, a technique for efficiently computing gradients.

### Models
The basic unit of probabilistic programs is the stochastic function (or model). A statistical model is a mathematical description of how the values of some knowns and unknowns could generate the observed data.

S. Wood, "Core Statistics"

A stochastic function in Python is an arbitrary function that combines two ingredients:

- deterministic Python code
- primitive stochastic functions that call a random number generator

**Using `sample()` primitive**

Drawing a sample from the unit normal distribution $\mathcal{N}(0,1)$.

```python
import torch
import pyro
pyro.set_rng_seed(1) # for reproducibility

loc = 0. # mean
scale = 1. # standard deviation

# using pytorch
normal = torch.distributions.Normal(loc, scale) # create a normal distribution object
x = normal.rsample() # draw a sample from N(0,1)
print("pytorch sample:\t", x)

# using pyro
x = pyro.sample("sample_name", pyro.distributions.Normal(loc, scale))

print("pyro sample:\t", x)
```

    pytorch sample:	 tensor(0.6614)
    pyro sample:	 tensor(0.2669)

Pyro samples are named: Pytorch backend uses these names to uniquely identify sample statements and change their behavior at runtime depending on how the enclosing stochastic function is being used.

**Drawing multiple samples**

Now we draw multiple samples from $\mathcal{N}(2,2)$ and $\text{Exp}(0.3)$ distributions and plot the corresponding histograms.

```python
import seaborn as sns
import matplotlib.pyplot as plt
import pyro.distributions as dist

# distributions
normal =  dist.Normal(2, 2)
exp = dist.Exponential(0.3)

# multiple samples
normal_samples = [pyro.sample("n",normal) for i in range(200)]
exp_samples = [pyro.sample("n",exp) for i in range(200)]

#plot
fig, axes = plt.subplots(1, 2, figsize=(12,4))
sns.distplot(normal_samples, ax=axes[0])
sns.distplot(exp_samples, ax=axes[1])
axes[0].set_title('Normal')
axes[1].set_title('Exponential')
plt.show()
```

Seaborn `distplot()` automatically estimates the PDFs over histogram bins.

**Simple stochastic model**

Suppose we want to reason about how temperature interacts with sunny and cloudy weather. We can define a simple stochastic function `weather()` describing the interaction

$$ \mathcal{N}(12.0,5.0^2) \; \text{for cloudy weather}$$

$$\mathcal{N}(23.0,6.0^2) \; \text{for sunny weather} \; $$

```python
def weather():
    # generate a binary sample
    is_cloudy = pyro.sample('cloudy', dist.Bernoulli(0.3))

    # convert binary sample into categorical
    is_cloudy = 'cloudy' if is_cloudy.item() == 1.0 else 'sunny'

    loc_temp = {'cloudy': 12.0, 'sunny': 23.0}[is_cloudy]
    scale_temp = {'cloudy': 5.0, 'sunny': 6.0}[is_cloudy]
    temp = pyro.sample('temp', dist.Normal(loc_temp, scale_temp))

    return {"weather":is_cloudy, "temp":temp.item()}

[weather() for _ in range(5)]
```

    [{'weather': 'cloudy', 'temp': 8.512848854064941},
     {'weather': 'cloudy', 'temp': 12.010722160339355},
     {'weather': 'sunny', 'temp': 19.3258056640625},
     {'weather': 'sunny', 'temp': 14.052600860595703},
     {'weather': 'sunny', 'temp': 32.76374816894531}]

We could use this stochastic function to model the sales of ice cream based on the weather.

```python
def ice_cream_sales():
    is_cloudy, temp = weather()
    expected_sales = 200. if is_cloudy == 1 and temp > 35.0 else 20.
    sales = pyro.sample('ice_cream', pyro.distributions.Normal(expected_sales, 10.0))
    return sales

[ice_cream_sales() for _ in range(5)]
```

    [tensor(31.2418),
     tensor(36.3404),
     tensor(18.3335),
     tensor(23.0716),
     tensor(7.7725)]

## Inference

The purpose of statistical inference is that of using a statistical model to infer the values of the unknowns that are consistent with the observed data.

|Frequentist interpretation|Bayesian interpretation|
|:-:|:-:|
|Probability measures a proportion of outcomes. | Probability measures the believability in an event. |
|There is randomness in our estimation of the parameters, but not in the parameters themselves, which are considered as fixed.| Parameters are treated as random variables and our belief about these parameters is updated in the light of data.|

### Bayesian inference

**Bayes theorem**

Let $A$ and $B$ be two events, such that $P(B)\neq0$, then $
P(A|B) = \frac{P(A,B)}{P(B)}=\frac{P(B|A)P(A)}{P(B)}$.

https://medium.com/informatics-lab/probabilistic-programming-1535d7882dbe

**Bayes theorem example**

There are two boxes $b_1$ and $b_2$. Box 1 contains three red and five white balls and box 2 contains two red and five white balls. A box $B\in\{b1,b2\}$ is chosen at random with $P(B=b_1)=P(B=b_2)=0.5$ and a ball chosen at random from this box turns out to be red.
What is the posterior probability that the red ball came from box 1?

$R\in\{0,1\}$ indicates whether the chosen ball is red or not.

From Bayes theorem we get
$$
P(B=b_1|R=1)=\frac{P(B=b_1,R=1)}{P(R=1)}.
$$

and $P(B=b_1,R=1) = P(R=1|B=b_1)P(B=b_1)=\frac{3}{8}\cdot \frac{1}{2}$.

From the law of total probability $P(R=1)=\sum_{i\in\{1,2\}}P(R=1|B=b_i)P(B=b_i)=\frac{3}{8}\cdot \frac{1}{2}+\frac{2}{7}\cdot\frac{1}{2}=\frac{37}{112}$.

Consequently, $$P(B=b_1|R=1)=\frac{3}{16}\cdot\frac{112}{37}=\frac{21}{37}\approx 0.56$$

**Posterior probability**

Under the Bayesian paradigm we do not estimate parameters, we
compute their distribution based on the given data.

The posterior probability is derived according to Bayes' theorem

$$
p(\theta|x) = \frac{p(x|\theta)p(\theta)}{p(x)}
$$

and the idea of uncertainty is preserved by the specific interpretation attributed to the involved terms:

- **prior probability** $p(\theta)$ = degree of belief of event occurring before observing any evidence
- **evidence** $p(x)$ = observed data
- **likelihood** $p(x|\theta)$ = compatibility of the evidence with the given hypothesis
- **posterior probability** $p(\theta|x)$ = updated belief given the evidence

https://www.researchgate.net/figure/Bayesian-updating-of-the-prior-distribution-to-posterior-distribution-The-Posterior_fig1_320507985

### Conjugate priors

If the posterior distribution $p(\theta|x)$ belongs to the same family as the prior distribution $p(\theta)$, then the prior is said to be a **conjugate prior** for the likelihood function $p(x|\theta)$.
This is a particularly convenient case in which the posterior distribution has a closed-form expression.

These are just a few examples of conjugate priors:

|Conjugate prior distribution| Likelihood | Prior hyperparameters | Posterior hyperparameters|
|:------:|:-----:|:-----:|:------:|
|Normal|Normal (known var.)|$$\mu_0,\sigma_0^2$$|$${\frac{1}{\frac{1}{\sigma_0^2}+\frac{n}{\sigma^2}}\Bigg(\frac{\mu_0}{\sigma_0^2}+\frac{\sum_{i=1}^n x_i}{\sigma^2}\Bigg),\Bigg(\frac{1}{\sigma_0^2}+\frac{n}{\sigma^2}\Bigg)^{-1}}$$|
|Inverse Gamma|Normal (known mean)|$$\alpha,\beta$$|$$\alpha+\frac{n}{2},\beta+\frac{\sum_{i=1}^n (x_i-\mu)^2}{2}$$|
|Beta|Binomial|$$\alpha,\beta$$|$$\alpha+\sum_{i=1}^n x_i, n-\sum_{i=1}^n x_i+\beta$$|
|Gamma|Poisson|$$k,\theta$$|$$k+\sum_{i=1}^n x_i,\frac{\theta}{n\theta+1}$$|
|Gamma|Exponential|$$\alpha,\beta$$|$$\alpha+n,\beta+\sum_{i=1}^n x_i$$|

**Beta-Binomial case**

$Beta(\alpha,\beta)$ prior and $x\sim Bin(n,\pi)$ likelihood result in the posterior

\begin{align}
p(\pi|x,\alpha,\beta)&\propto \pi^x (1-\pi)^{n-x} \pi^{\alpha-1}(1-\pi)^{\beta-1}\\
                     &\propto \pi^{x+\alpha-1}(1-\pi)^{n-x+\beta-1}
\end{align}

which is a $Beta(x+\alpha, n-x+\beta)$.

### Approximate inference

Prior $p(\theta)$ and likelihood $p(x|\theta)$ functions are usually known as part of the model, while the computation of the normalization factor

$$
p(x) = \int_\theta p(x|\theta) p (\theta) d \theta
$$

can easily become intractable in the high-dimensional cases.

**Example of intractable posterior**

Suppose we are trying to figure out how much something weighs, but the scale we’re using is unreliable and gives slightly different answers every time we weigh the same object. We could try to compensate for this variability by integrating the noisy measurement information with a guess based on some *prior knowledge* about the object.

$$weight \, | \, guess \sim \mathcal{N}(guess, 1)$$

$$ measurement \, | \, guess, weight \sim \mathcal{N}(weight, 0.75^2) $$

```python
def scale(guess):
    weight = pyro.sample("weight", dist.Normal(guess, 1.0))
    measurement = pyro.sample("measurement", dist.Normal(weight, 0.75))
    return measurement
```

The model is quite simple, so we are able to determine our posterior distribution of interest analytically. But in general the exact computation of the posterior of an arbitrary stochastic function is intractable.

Even the `scale` model with a non-linear function may become intractable.

```python
def intractable_scale(guess):
    weight = pyro.sample("weight", dist.Normal(guess, 1.0))
    measurement = pyro.sample("measurement",
                              dist.Normal(some_nonlinear_function(weight), 0.75))
    return measurement
```

Approximate inference addresses the need of applying Bayesian learning to more complex problems and to the high-dimensional datasets that we are dealing with in machine learning.

Examples of approximate inference include Variational Bayesian methods, Markov chain Monte Carlo, Markov Random Fields and Bayesian Networks.
We can identify two main categories for approximate inference:
- **Stochastic methods** turn the problem of inference into a problem of sampling from the posterior distribution of interest;
- **Deterministic methods** substitute inference with optimization problems.

## References
- S. Wood, "Core Statistics"
- [Pyro library](https://pyro.ai/)
- [Pyro documentation](https://docs.pyro.ai/en/1.1.0/index.html)
- [Probabilistic Programming & Bayesian Methods for Hackers](https://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/)


# Application: Cavity Flow

One of the most common validation cases in CFD is the lid-driven cavity flow.  We take a square cavity filled with a fluid and set the velocity of the lid to some constant value.  The flow within the cavity is driven by the lid, a spiral flow pattern develops and two distinctive pressure zones are visible in the upper corners against the lid.

```python
import numpy
```

The Poisson equation is an elliptic PDE which almost always means using an iterative solver.  We're going to use the Jacobi method.  There are better ways, but that's beside the point.

Here's the pressure Poisson equation:

\begin{align}
p_{i,j}^{n+1} &= \frac{1}{4}\left(p_{i+1,j}^{n}+p_{i-1,j}^{n}+p_{i,j+1}^{n}+p_{i,j-1}^{n}\right) \\
&-\frac{\rho \Delta x}{16} \left( \frac{2}{\Delta t} \left(u_{i+1,j} - u_{i-1,j} + v_{i,j+1} - v_{i,j-1}\right) \right . \\
&-\frac{2}{\Delta x}\left(u_{i,j+1} - u_{i,j-1} \right) \left(v_{i+1,j} - v_{i-1,j} \right) \\
&- \left . \frac{\left(u_{i+1,j} - u_{i-1,j} \right)^2}{\Delta x}
- \frac{ \left(v_{i,j+1} - v_{i,j-1} \right)^2 }{\Delta x} \right) \\
\end{align}

That looks a little nasty, but we only care about the top line when we iterate, since the bottom three lines depend only on values that don't change when we're correcting the pressure field.  Because it doesn't change, we break it out into a separate function.

```python
def velocity_term(b, rho, dt, u, v, dx):
    b[1:-1, 1:-1] = (
        rho * dx / 16 *
        (2 / dt * (u[1:-1, 2:] -
                    u[1:-1, :-2] +
                    v[2:, 1:-1] -
                    v[:-2, 1:-1]) -
        2 / dx * (u[2:, 1:-1] - u[:-2, 1:-1]) *
                 (v[1:-1, 2:] - v[1:-1, :-2]) -
        (u[1:-1, 2:] - u[1:-1, :-2])**2 / dx -
        (v[2:, 1:-1] - v[:-2, 1:-1])**2 / dx)
                     )

    return b
```

Now, to calculate the pressure field, we pass in the original pressure field, the value `b` (which is the result of the `velocity_term` function above) and a target value for difference between two iterates.  We repeatedly update the pressure field until the difference of the L2 norm between two successive iterations is less than that target value.

```python
def pressure_poisson(p, b, l2_target):
    iter_diff = l2_target + 1
    n = 0
    while iter_diff > l2_target and n <= 500:

        pn = p.copy()
        p[1:-1,1:-1] = (.25 * (pn[1:-1, 2:] +
                               pn[1:-1, :-2] +
                               pn[2:, 1:-1] +
                               pn[:-2, 1:-1]) -
                               b[1:-1, 1:-1])

        p[:, 0] = p[:, 1]   #dp/dx = 0 at x = 0
        p[:, -1] = p[:, -2] #dp/dx = 0 at x = 2
        p[0, :] = p[1, :]   #dp/dy = 0 at y = 0
        p[-1, :] = 0        #p = 0 at y = 2

        if n % 10 == 0:
            iter_diff = numpy.sqrt(numpy.sum((p - pn)**2)/numpy.sum(pn**2))

        n += 1

    return p
```

In the interests of brevity, we're only going to worry about the pressure Poisson solver.  The rest of the 2D Navier-Stokes solution is encapsulated in the function `cavity_flow`, which we've prepared ahead of time and saved in a helper file. If you want to dig deeper into how the `cavity_flow` function works, check out "The 12 Steps to Navier-Stokes" at [CFD Python](https://github.com/barbagroup/CFDPython).

For now, though, we just need to import the function:

```python
from snippets.ns_helper import cavity_flow
```

We'll also load up [pickled](https://docs.python.org/3/library/pickle.html) initial conditions, so we can reliably compare final solutions.

```python
import pickle
```

```python
def run_cavity():
    nx = 41
    ny = 41
    with open('IC.pickle', 'rb') as f:
        u, v, p, b = pickle.load(f)

    dx = 2 / (nx - 1)
    dt = .005
    nt = 1000

    u, v, p = cavity_flow(u, v, p, nt, dt, dx,
                         velocity_term,
                         pressure_poisson,
                         rtol=1e-4)

    return u, v, p
```

So what does this all do?  Let's check it out.

```python
u, v, p = run_cavity()
```

```python
%matplotlib inline
from snippets.ns_helper import quiver_plot
```

```python
quiver_plot(u, v, p)
```

#### Save NumPy answers for comparison

This will create a binary file, saved to disk as `numpy_ans.pickle` that contains the values of `u`, `v`, and `p`.

```python
with open('numpy_ans.pickle', 'wb') as f:
    pickle.dump((u, v, p), f)
```

Let's profile the `cavity_flow` function and see if there's a specific place that's really hurting our performance.

```python
%timeit run_cavity()
```

Remember that we need to load the `line_profiler` extension before we can use the `lprun` magic.

```python
%load_ext line_profiler
```

```python
%lprun -f cavity_flow run_cavity()
```

## Where is the bottleneck?

Clearly the PPE is the problem here, so let's use `numba` to rewrite it.

## [Exercise: Speed up the PPE](./exercises/05.Cavity.Flow.Exercises.ipynb#Exercise-1)

```python
from numba import jit
```

```python
%load snippets/ppe_numba.py
```

Since we have redefined the `pressure_poisson` function, we can run `run_cavity` again and we'll be using the new and improved `jit`ted PPE.

We don't want to overwrite our results, so we can choose different variable names to store the final velocity and pressure fields.

```python
u_numba, v_numba, p_numba = run_cavity()
```

We use `numpy.allclose` to check that each value of the Python and Numba fields match to within a specified tolerance (the default tolerance is 1$\tt{E}^-$08).

```python
assert numpy.allclose(p, p_numba)
assert numpy.allclose(u, u_numba)
assert numpy.allclose(v, v_numba)
```

If there were no errors raised by the previous cell, then the two answers match.  Hooray!
If the answers match, we should see the same plot as above.  If they don't match, we can also check the plot to see if we're just a little bit off or completely wrong (although that can be hard to judge sometimes).

```python
quiver_plot(u_numba, v_numba, p_numba)
```

And now we can check to see how much the performance has been improved:

```python
%timeit run_cavity()
```

Not as spectacular as some of the other examples, but remember, we started off using NumPy arrays, not regular nested Python loops, so this is still a very respectable speedup.

Now if we re-profile the code, we can see how the runtime percentages have shifted around the improved PPE.  It's almost certainly going to remain the most expensive part of the solve, but it should be a smaller percentage overall.

```python
%lprun -f cavity_flow run_cavity()
```

## One more bit of optimization?

As we improve the performance of the Pressure Poisson function, the profiling results will change and indicate other possible hotspots that we want to improve. It can be hard to know when to _stop_ improving, to recognize that the gains are not worth what you are investing in rewriting code.

In the profile above (after rewriting the PPE) you'll notice that the `velocity_term` now occupies a larger percentage of the total run time. Here's a rewritten version of `velocity_term` that makes use of Numba so you can compare how much more performance we might be able to eke out of this simple cavity flow solver.

```python
@jit(nopython=True)
def velocity_term(b, rho, dt, u, v, dx):
    J, I = b.shape

    for i in range(1, I):
        for j in range(1, J):
            b[j, i] = (
            rho * dx / 16 *
            (2 / dt * (u[j, i + 1] -
                      u[j, i - 1] +
                      v[j + 1, i] -
                      v[j - 1, i]) -
            2 / dx * (u[j + 1, i] - u[j - 1, i]) *
                     (v[j, i + 1] - v[j, i - 1]) -
            (u[j, i + 1] - u[j, i - 1])**2 / dx -
            (v[j + 1, i] - v[j - 1, i])**2 / dx)
            )
    return b
```

```python
%timeit run_cavity()
```


## The network SIR model

In order to introduce mean-field approach of network model, we choose SIR model since this is a simplest model in our implementation.
The dynamical equations are given as

\begin{align}
\dot{S_{in}}	    &=-\lambda_{in}(t)S_{in}+\sigma_{in},\\
\dot{I}_{in}^{a}	&=\alpha\lambda_{in}(t)S_{in}-\gamma_{I^{a}}I_{in}^{a},\\
\dot{I}_{in}^{s}	&=\bar{\alpha}\lambda_{in}(t)S_{in}-\gamma_{I^{s}}I_{in}^{s},\\
\dot{R}_{in}^{s}	&=\gamma_{I^{a}}I_{in}^{a} + \gamma_{I^{s}}I_{in}^{s}.\\
\end{align}

The key idea is to note that the rate of infection of a susceptible individual in age group `i` at the node `n` is given as

\begin{align}
\lambda_{in}(t)=\lambda^0_{in}(t)=\beta\sum_{j=1}^{M}\left(C_{ij}\frac{I_{jn}^{a}}{N^{H}_{jn}}+f_{sa}C_{ij}\frac{I_{jn}^{s}}{N^{H}_{jn}}\right),
\end{align}

Commute can then be modeled by changing this to

\begin{align}
\lambda_{in}^{T}(t)=\beta r^{T}\sum_{j=1}^{M}\sum_{m=1}^{N_{d}} \sum_{<k,l>} C_{ij} \frac{D_{j,kl}}{N^{H}_{j,l}} \frac{I_{jl}^{a} + f_{sa} I_{jl}^{s}}{N^{T}_{j,mn}} \frac{D_{i,mn}}{N^{H}_{i,n}},
\end{align}

while workplace interaction can be modeled as

\begin{align}
\lambda_{in}^{W}(t)=\beta r^{W}\sum_{j=1}^{M}\sum_{m,k=1}^{N_{d}} C_{ij} \frac{D_{j,mk}}{N^{H}_{jk}} \frac{I_{jk}^{a} + f_{sa} I_{jk}^{s}}{N^{W}_{jm}}  \frac{D_{i,mn}}{N^{H}_{in}}.
\end{align}

The derivation of all equations is wrriten in [the documentation](https://github.com/lukastk/PyRossGeo/tree/master/docs/mft-model.pdf).
The simulation then runs by choosing appropriate $\lambda$ as a function of time. The constants $r_T$ and $r_W$ control the strength of interactions. In this model, we assume that each route between the nodes connects neighbour nodes only.

The definition of the variables are in following.
$D_{i,nm}$ is the number of people working at node `n` and livingin `m` in the age group `i`, $N^{H}_{in}=\sum_{m} D_{i,mn}$ is the total number of residence in node `n` and $N^{W}_{in}=\sum_{m} D_{i,nm}$ is the total number of people working at node `n`.
$N_{i,nm}^{T}$ is the effective number of the commuter in the age group `i` using the route between `n` and `m`:

\begin{align}
N^{T}_{i,nm}=\sum_{<kl>}R_{i,kl}\frac{d_{kl}}{d_{nm}},
\end{align}

where $<kl>$ indicate summation of all pair of neighbour node $k$ and $l$ included in the shortest path between node
`n` and `m` and $d_{nm}$ is the distance between node `n` and `m` along the shortest path. $R_{i,nm}=\sum_{<kl>}D_{i,kl}$ is the number of the commuter in the age group `i` using the route between the neighbor node `n` and `m`. The node `n` and `m` are included in the shortest path between node pair
$(k,l)$ took up all combinations satisfying the conditions.
Hence, we only need $D_{i,nm}$ and $d_{nm}$ in order to construct this network model. $D_{i,nm}$ can be obtained census datasets and $d_{nm}$ can be determined by the geographical locations.

This tutorial provide how to determine $D_{i,nm}$ and $d_{nm}$ in the section 1 to 7, and how to perform the network SIR model in the section 8 and following.

To run the section 1 to 7, that is a part of other notebook coded by Jakub originally, the data and shape files need to be downloaded (~1GB) and put in the corresponding folders. They can be found here:

* 2011 Census: Table wu03ew (MSOA, method of transport), wu02ew (MSOA, age), wf01bew (OA) which can be downloaded [here](https://www.nomisweb.co.uk/census/2011/bulk/rOD1)
* Shape files: [MSOA](https://geoportal.statistics.gov.uk/datasets/middle-layer-super-output-areas-december-2011-boundaries-ew-bgc), [LAD](https://geoportal.statistics.gov.uk/datasets/local-authority-districts-december-2019-boundaries-uk-bfc)
* Resident data: [for both 2011 and 2018](https://www.ons.gov.uk/peoplepopulationandcommunity/populationandmigration/populationestimates/datasets/middlesuperoutputareamidyearpopulationestimates) in MSOA
* MSOA to LAD conversion: [here](https://geoportal.statistics.gov.uk/datasets/postcode-to-output-area-to-lower-layer-super-output-area-to-middle-layer-super-output-area-to-local-authority-district-february-2020-lookup-in-the-uk)
* Scottish Data workplace data: [here under the origin destination-travel heading](https://www.scotlandscensus.gov.uk/ods-web/data-warehouse.html#additionaltab)

**Practically, required data is already there and you can start from the section 8, if you prefer.**

**1**. Start to reading the census data. If there are census.txt and distances.txt, you can start at the section **8** which is the calculation section.

```python
%matplotlib inline
import numpy as np
import pyross
import matplotlib.pyplot as plt
from scipy.spatial import Delaunay, delaunay_plot_2d, Voronoi, voronoi_plot_2d
from scipy.sparse.csgraph import shortest_path, floyd_warshall, dijkstra, bellman_ford, johnson
from scipy.sparse import csr_matrix
import math
import pandas as pd
import geopandas as gpd

#First load everything you need
totalList= ["City of London", "Camden", "Greenwich", "Hackney", "Hammersmith and Fulham", "Islington", "Kensington and Chelsea",
            "Lambeth", "Lewisham", "Southwark", "Tower Hamlets", "Wandsworth", "Westminster", "Barking and Dagenham", "Barnet", "Bexley", "Brent",
            "Bromley", "Croydon", "Ealing", "Enfield", "Haringey", "Harrow", "Havering", "Hillingdon", "Hounslow", "Kingston upon Thames",
            "Merton", "Newham", "Redbridge", "Richmond upon Thames", "Sutton", "Waltham Forest"]
df = pd.DataFrame(totalList)
df=df.rename(columns={0:'lad19nm'})
geometry_msoa = gpd.read_file("shape/Middle_Layer_Super_Output_Areas_December_2011_Boundaries_EW_BGC.shp")
geometry_msoa = geometry_msoa.rename(columns={'msoa11cd':'msoa'})
geometry_lad = gpd.read_file("shape/Local_Authority_Districts_December_2019_Boundaries_UK_BFC.shp").rename(columns={'lad19cd':'lad'})
msoa_to_lad = pd.read_csv('data/PCD_OA_LSOA_MSOA_LAD_FEB20_UK_LU.csv', encoding="latin_1")
```

    /home/hidekb/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py:3063: DtypeWarning: Columns (13) have mixed types.Specify dtype option on import or set low_memory=False.
      interactivity=interactivity, compiler=compiler, result=result)

**2**. Here I make London but in principle this can be done for anything, files are all the standard files Pauls assumes in workplace data

```python
geometry_lad=geometry_lad.merge(df,on='lad19nm',how='right')
geometry_lad["ID"] = np.linspace(0,len(geometry_lad)-1,len(geometry_lad))
msoas_london = msoa_to_lad[msoa_to_lad.ladnm.isin(totalList).fillna(False)][['msoa11cd']].drop_duplicates().rename(columns={'msoa11cd':'msoa'})
geometry_msoa=geometry_msoa.merge(msoas_london, on='msoa')
geometry_msoa.plot()
geometry_msoa["ID"] = np.linspace(0,len(geometry_msoa)-1,len(geometry_msoa))
geometry_msoa.head()
```

**3**. Next you must make a list of neighbours using the "ID" column which can be changed

```python
geometry_msoa["NEIGHBORS"] = None  # add NEIGHBORS column
a=np.zeros(len(geometry_msoa))
AllNeighbors=np.array(a,dtype=object)
for index, ladSpec in geometry_msoa.iterrows():
    # get 'not disjoint' countries
    neighbors = geometry_msoa[~geometry_msoa.geometry.disjoint(ladSpec.geometry)].ID.tolist()
    # remove own name from the list
    neighbors = [ name for name in neighbors if ladSpec.ID != name ]
    #Put it all together
    AllNeighbors[index]=neighbors
```

**4**. Next you add representative points to each msoa/lad/whatever

```python
geometry_msoa["Center"]=geometry_msoa.representative_point()
geometry_msoa.head()
## Here, Nnode value is obtained from msoa file
Nnode=len(geometry_msoa)
```

```python
AllNeighbors[0]
```

    [186.0, 187.0, 356.0, 554.0, 555.0, 778.0, 846.0, 852.0, 857.0, 858.0, 944.0]

**5**. You extract these into a numpy format

```python
listarray = []
for pp in geometry_msoa["Center"]:
    listarray.append([pp.x, pp.y])
# list_arrays = [ np.array((geom["Center"][0][0], geom["Center"][1][0])) for geom in geometry_lad["Center"] ]
print(listarray[0])
```

    [532384.2774694173, 181344.97649999987]

**6**. Then you make an array of distance between nearest neighbours

```python
distances=np.zeros((len(geometry_msoa),len(geometry_msoa)))
Position = []
#Transforms it from a shapeobject into numpy
for pp in geometry_msoa["Center"]:
    Position.append([pp.x, pp.y])
#gets the distances
for i in range(0,len(AllNeighbors)):
    for j in range(0,len(AllNeighbors[i])):
        x=Position[i][0]-Position[int(AllNeighbors[i][j])][0]
        y=Position[i][1]-Position[int(AllNeighbors[i][j])][1]
        distances[i][int(AllNeighbors[i][j])]=math.sqrt(x*x+y*y)
```

Then you finish making a distance matrix between all the nodes and a matrix predecessors that tells you how to get from one node to the next

**7**. Next you load the commuting data

```python
age_commute_table = pd.read_csv("data/wu02ew_msoa.csv")
work_in_GLMSOA = age_commute_table[age_commute_table['Area of workplace'].isin(geometry_msoa.msoa) ]
work_in_GLMSOA =work_in_GLMSOA[work_in_GLMSOA['Area of residence'].isin(geometry_msoa.msoa)]
#work_in_GLMSOA =work_in_GLMSOA[work_in_GLMSOA['Area of residence']!=work_in_GLMSOA['Area of workplace']]
work_in_GLMSOA.index = pd.RangeIndex(len(work_in_GLMSOA.index))
work_in_GLMSOA.head()
```

      Area of residence
      Area of workplace
      All categories: Age 16 and over
      16-24
      25-34
      35-49
      50-64
      65-74
      75+

      0
      E02000001
      E02000001
      1506
      128
      625
      469
      251
      29
      4

      1
      E02000001
      E02000014
      2
      0
      0
      2
      0
      0
      0

      2
      E02000001
      E02000016
      3
      0
      1
      1
      1
      0
      0

      3
      E02000001
      E02000025
      1
      0
      0
      1
      0
      0
      0

      4
      E02000001
      E02000028
      1
      0
      0
      1
      0
      0
      0

Here, you load the commuting data of older than 16

```python
M = 7 #Aged Group
census_matrix = np.zeros((M + 1,Nnode,Nnode))
#census_matrix is constructed from census_file
#census_matrix[i][j] - i:place of work, j: usual residence
for i in range(0, len(work_in_GLMSOA)):
    indexhome=int(geometry_msoa.loc[geometry_msoa['msoa'] == work_in_GLMSOA["Area of residence"][i]]["ID"].values)
    indexwork=int(geometry_msoa.loc[geometry_msoa['msoa'] == work_in_GLMSOA["Area of workplace"][i]]["ID"].values)
    #Cij=int(work_in_GLMSOA["All categories: Age 16 and over"][i])
    census_matrix[1][indexwork][indexhome] = int(work_in_GLMSOA['16-24'][i])
    census_matrix[2][indexwork][indexhome] = int(work_in_GLMSOA['25-34'][i])
    census_matrix[3][indexwork][indexhome] = int(work_in_GLMSOA['35-49'][i])
    census_matrix[4][indexwork][indexhome] = int(work_in_GLMSOA['50-64'][i])
    census_matrix[5][indexwork][indexhome] = int(work_in_GLMSOA['65-74'][i])
    census_matrix[6][indexwork][indexhome] = int(work_in_GLMSOA['75+'][i])

f_distances = open('distances.txt', mode='w')
for i in range(Nnode):
    for j in range(Nnode - 1):
        f_distances.write(str(distances[i][j]) + ' ')
    f_distances.write(str(distances[i][Nnode - 1]) + '\n')
f_distances.close()
```

Next, you load the data of younger than 16 from population dataset

```python
msoas_pop = pd.read_csv("data/SAPE21DT3a-mid-2018-msoa-on-2019-LA-syoa-estimates-formatted.csv",sep=',')
msoas_pop = msoas_pop[msoas_pop['Contents'].isin(geometry_msoa.msoa) ]
msoas_pop=msoas_pop.drop(['Unnamed: 1'], axis=1)
msoas_pop.index = pd.RangeIndex(len(msoas_pop.index))
msoas_pop.head()
```

      Contents
      Unnamed: 2
      Unnamed: 3
      Unnamed: 4
      Unnamed: 5
      Unnamed: 6
      Unnamed: 7
      Unnamed: 8
      Unnamed: 9
      Unnamed: 10
      ...
      Unnamed: 85
      Unnamed: 86
      Unnamed: 87
      Unnamed: 88
      Unnamed: 89
      Unnamed: 90
      Unnamed: 91
      Unnamed: 92
      Unnamed: 93
      Unnamed: 94

      0
      E02000166
      Camden 001
      9,092
      84
      90
      67
      91
      94
      111
      111
      ...
      55
      32
      42
      38
      27
      35
      24
      24
      16
      64

      1
      E02000167
      Camden 002
      9,437
      94
      98
      111
      103
      119
      142
      172
      ...
      55
      53
      53
      41
      32
      37
      26
      26
      28
      127

      2
      E02000168
      Camden 003
      9,501
      86
      116
      97
      75
      116
      103
      107
      ...
      25
      32
      31
      31
      17
      15
      17
      23
      18
      49

      3
      E02000169
      Camden 004
      9,199
      107
      111
      136
      138
      121
      137
      147
      ...
      59
      39
      38
      33
      40
      23
      22
      24
      27
      115

      4
      E02000170
      Camden 005
      9,690
      97
      100
      109
      110
      117
      113
      121
      ...
      31
      34
      21
      22
      25
      15
      9
      24
      11
      54

5 rows × 94 columns

Here, census_matix store the population of younger then 16

```python
PopulationAgeStructuredDivided=np.zeros((msoas_pop.shape[0],8))
for j in range(0,msoas_pop.shape[0]):
    area_id = int(geometry_msoa.loc[geometry_msoa['msoa'] == msoas_pop['Contents'][j]]["ID"].values)
    PopulationAgeStructuredDivided[j][0] = area_id
    for i in range(0,5):
        PopulationAgeStructuredDivided[j][1] += float(msoas_pop.iloc[j,i+3].replace(',',''))
        census_matrix[0][area_id][area_id] += int(msoas_pop.iloc[j,i+3].replace(',',''))
    for i in range(5,16):
        PopulationAgeStructuredDivided[j][2] += float(msoas_pop.iloc[j,i+3].replace(',',''))
        census_matrix[0][area_id][area_id] += int(msoas_pop.iloc[j,i+3].replace(',',''))
    for i in range(16,25):
        PopulationAgeStructuredDivided[j][3] += float(msoas_pop.iloc[j,i+3].replace(',',''))
    for i in range(25,35):
        PopulationAgeStructuredDivided[j][4] += float(msoas_pop.iloc[j,i+3].replace(',',''))
    for i in range(35,50):
        PopulationAgeStructuredDivided[j][5] += float(msoas_pop.iloc[j,i+3].replace(',',''))
    for i in range(65,75):
        PopulationAgeStructuredDivided[j][6] += float(msoas_pop.iloc[j,i+3].replace(',',''))
    for i in range(75,91):
        PopulationAgeStructuredDivided[j][7] += float(msoas_pop.iloc[j,i+3].replace(',',''))

msoas_popAS=pd.DataFrame(PopulationAgeStructuredDivided)
msoas_popAS.head()
```

      0
      1
      2
      3
      4
      5
      6
      7

      0
      160.0
      426.0
      1306.0
      736.0
      1287.0
      1796.0
      964.0
      750.0

      1
      161.0
      525.0
      1385.0
      558.0
      1321.0
      2275.0
      877.0
      891.0

      2
      162.0
      490.0
      1134.0
      738.0
      2071.0
      2181.0
      758.0
      509.0

      3
      163.0
      613.0
      1537.0
      944.0
      1120.0
      2139.0
      718.0
      737.0

      4
      164.0
      533.0
      1230.0
      805.0
      2319.0
      2307.0
      627.0
      521.0

The commuting data write down into census.txt.

```python
msoas_popAS=msoas_popAS.rename(columns={0:"ID", 1:"0-5", 2:"5-15", 3:"16-24", 4:"25-34", 5:"35-49", 6:"50-64", 7:"65-74", 8:"75+"})
msoas_popAS.head()
msoas_popAS.to_csv('statPop.csv',index=False)

#Make Total
for i in range(Nnode):
    for j in range(Nnode):
        for alp in range(M):
            census_matrix[M,i,j] += census_matrix[alp,i,j]

f_census = open('census.txt', mode='w')
for k in range(M):
    for i in range(Nnode):
        for j in range(Nnode - 1):
            f_census.write(str(census_matrix[k][i][j]) + ' ')
        f_census.write(str(census_matrix[k][i][Nnode - 1]) + '\n')
f_census.close()
```

**8**. The following section is the calculation section. If there are census.txt and distances.txt, you can start at here.

**9**. $D_{i,nm}$ and $d_{nm}$ is constructed form census.txt and distances.txt. This dataset is the London including 983 nodes.

```python
import numpy as np
import pyrossgeo
import matplotlib.pyplot as plt
import math
import os
import pandas as pd

model_def = {
    "settings" : {
        "classes" : ["S", "I"],
        "stochastic_threshold_from_below" : [1000, 1000],
        "stochastic_threshold_from_above" : [500, 500],
        "contact_scaling" : "powerlaw",
        "contact_scaling_parameters" : [0, 1.0, 0.5] # a + b * rho^c
    },

    "S" : {
        "linear"    : [],
        "infection" : [ ["I", "-beta"] ]
    },

    "I" : {
        "linear"    : [ ["I", "-gI"] ],
        "infection" : [ ["I", "beta"] ]
    }
}

M = 7 #Aged Group

cutoff = 0
Restricted = [] #the restirected area
if os.path.exists('census.txt') and os.path.exists('distances.txt'):
    f_census = open('census.txt')
    f_distances = open('distances.txt')
    c_lines = f_census.readlines()
    d_lines = f_distances.readlines()
    if len(c_lines) != len(d_lines)*M:
        print('Error. Wrong file Status\n')
        exit()

    Nd=int(len(c_lines)/M)
    census_matrix = np.zeros((M, Nd, Nd), dtype=np.uint16)
    distances=np.zeros((Nd, Nd))

    count = 0
    aged_id = 0
    for line in c_lines:
        parameters = line.split()
        for i in range(len(parameters)):
            census_matrix[aged_id,count,i] = float(parameters[i])
        count += 1
        if count == Nd:
            count = 0
            aged_id += 1

    #Restricted Area
    for alp in range(M):
        for i in Restricted:
            for j in range(Nd):
                cij = census_matrix[alp,i,j]
                census_matrix[alp,i,j] = 0.0
                census_matrix[alp,j,j] += cij

                cij = census_matrix[alp,j,i]
                census_matrix[alp,j,i] = 0.0
                census_matrix[alp,i,i] += cij

    #Cutoff
    nonzero_element = 0
    cutoff_total = 0
    c_matrix = census_matrix.copy()
    for alp in range(M):
        for i in range(Nd):
            for j in range(Nd):
                cij = c_matrix[alp,i,j]
                if cij > cutoff:
                    nonzero_element += 1
                    cutoff_total += c_matrix[alp,i,j]
                else:
                    census_matrix[alp,i,j] = 0.0
                    #max_wid = np.argsort(census_matrix[:,j])[Nd- 1]
                    census_matrix[alp,j,j] += cij
    print("Nonzero element " + str(nonzero_element) + '/' + str(M*Nd**2) + ' ' + str(cutoff_total))

    Nh = np.zeros((M + 1,Nd)) # The population of ususal residence
    Nw = np.zeros((M + 1,Nd)) # The number of people working in specific node.
    for alp in range(M):
        for i in range(Nd):
            Nh[alp,i] = 0.
            Nw[alp,i] = 0.
            for j in range(Nd):
                Nh[alp,i] += census_matrix[alp,j,i]
                Nw[alp,i] += census_matrix[alp,i,j]
    for alp in range(M):
        for i in range(Nd):
            Nh[M,i] += Nh[alp,i] ## N^{H}_{i} residence in Node i
            Nw[M,i] += Nw[alp,i] ## N^{W}_i ##working in Node i

    count = 0
    for line in d_lines:
        parameters = line.split()
        for i in range(len(parameters)):
            distances[count,i] = float(parameters[i])
        count += 1

# set the matarix D_{nm}
Dnm = census_matrix.copy()

# set the matarix d_{nm}
dnm = distances.copy()

df = pd.read_csv('area.csv')
Area = df.values[:,1]/10**6

M1 = M*Nd

max_wid = np.argsort(Nw[M])[Nd- 1]
print("Maxmimum City at Working", max_wid, Nw[M,max_wid])
max_rid = np.argsort(Nh[M])[Nd- 1]
print("Maxmimum City of Residence", max_rid, Nh[M,max_rid])
```

    Nonzero element 749693/6764023 4760944
    Maxmimum City at Working 0 245633.0
    Maxmimum City of Residence 859 9752.0

**10**. To start network model calculation, we must set the  contact matrix $C_{ij}$, model parameters and the initial state. $C_{ij}$ and model parameters can be inferred if we obtain the reasonable initial state. In this tutorial, we set the preliminary values.

```python
# set the contact structure
C = np.identity(M)
for i in range(M - 1):
    C[i + 1, i] = 1.0
    C[i, i + 1] = 1.0
C *= 1.0
CH = C # in HOME
CT = C # in TRANS
CW = C # in WORK

# parameter
beta  = 0.2/24          # infection rate
gI    = 0.1/24          # recovery rate of infectives
rW    = 1.0             # the interaction control in WORK
rT    = 1.0             # the interaction control in TRANS
trv_restriction = 0.0   # travel restriction: 1.0 means everyone stay home

# Initial state
max_wid = np.argsort(Nw[M])[Nnode- 1]
#print("maxmimum population at Working", max_wid)
max_rid = np.argsort(Nh[M])[Nnode- 1]
#print("maxmimum population of Residence", max_rid)
min_wid = np.argsort(Nw[M])[0]
#print("minimum population at Working", min_wid)
min_rid = np.argsort(Nh[M])[0]
#print("minimum population of Residence", min_rid)

I_0 = np.zeros((Nnode*M))
I_0[1*Nnode + 859] += 10
R_0 = np.zeros((Nnode*M))
N_0 = np.zeros((Nnode*M))
for alp in range(M):
    for i in range(Nnode):
        N_0[Nnode*alp + i] = Nh[alp,i]
S_0  = N_0 - (I_0 + R_0)

# option
cutoff    = 1.0         # We ignore the D_{i,nm}, when D_{i,nm} is less than this number.
trv_restriction = 0.0   # travel restriction: 1.0 means everyone stay home

# the contact matrix
def contactMatrix(t):
    t_p_24 = t%24
    if t_p_24 < 8.0 or t_p_24 > 18.0: # in HOME
        return CH
    elif t_p_24 > 9.0 and t_p_24 < 17.0: # in WORK
        return CW
    else: # in TRANS
        return CT

# set the matarix D_{nm}
Dnm = census_matrix.copy()

# set the matarix d_{nm}
dnm = distances.copy()

# instantiate model
parameters = {'beta':beta, 'gI':gI, 'rW':rW, 'rT':rT}
geographical_model = 'SpacialCompartment'
model = pyrossgeo.mft.MeanFieldTheory(geographical_model, model_def, parameters, M, Nd, Dnm, dnm, Area, trv_restriction, cutoff)
```

    {'S': 0, 'I': 1}
    #Start to calculate fixed variables
    Nonzero element 371423/47348161 4565343.0
    Max index J 983
    Max index I 219
    #Finish calculating fixed variables

 After setting the parameters, the calculation of network model will be started. **The modern laptop can finish this calculation about 3 seconds.** We can improve this computational rate.

```python
rT=1.0
# intialize the model
parameters = {'beta':beta, 'gI':gI, 'rW':rW, 'rT':rT}
model.initialize(model_def, parameters, trv_restriction)

# duration of simulation
Tf=24*100; Nf=(1.0/24.0)*Tf # Maximum time step equals as Tf/Nf
# simulate model
data=model.simulate(np.concatenate((S_0, I_0)), contactMatrix, Tf, Nf, Ti=0) # Rapid
```

    {'S': 0, 'I': 1}
    #Start to calculate fixed variables
    Nonzero element 371423/47348161 4760936.0
    Max index J 983
    Max index I 219
    #Finish calculating fixed variables
    SpacialCompartment
    travel restriction 0.0
    cutoff 1.0
    highspeed 0
    #Calculation Start
    Time 100.1467540220921
    Time 201.20202066275976
    Time 302.7151989935382
    Time 402.716488245659
    Time 509.55759119880736
    Time 605.5575911988074
    Time 701.5575911988074
    Time 803.5575911988074
    Time 905.5575911988074
    Time 1001.5575911988074
    Time 1109.5575911988074
    Time 1205.5575911988074
    Time 1301.5575911988074
    Time 1403.5575911988074
    Time 1505.5575911988074
    Time 1601.5575911988074
    Time 1709.5575911988074
    Time 1805.5575911988074
    Time 1901.5575911988074
    Time 2003.5575911988074
    Time 2105.557591198807
    Time 2201.557591198807
    Time 2309.557591198807
    Time 2400.0

**11**. Loading the simulated data

```python
S  = []
I = []
for i in range(Nnode*M):
    S.append(data.get('X')[i].flatten())
    I.append(data.get('X')[i + M1].flatten())
t = data['t']

# In 1 dimension, the data index in age group i and node j can be expressed as i*N_{d} + j.
#| Age group 0 | Age group 1 | Age group 2 | Age group 3 | Age group 4 | Age group 5 | Age group 6 |
#|1,2,...,N_{d}|1,2,...,N_{d}|1,2,...,N_{d}|1,2,...,N_{d}|1,2,...,N_{d}|1,2,...,N_{d}|1,2,...,N_{d}|
```

**12**. Plotting the results

```python
fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

total_w = 0.0
total_r = 0.0
for i in range(Nnode):
    total_w += Nw[M][i]
    total_r += Nh[M][i]
#print('Toatl', total_w, total_r)

S_total = []
I_total = []
for i in range(len(I[0])):
    S_total.append(0.0)
    I_total.append(0.0)
for i in range(Nnode*M):
    S_total += S[i]
    I_total += I[i]
plt.plot(t/24, S_total, '-', label='$S$', lw=4)
plt.plot(t/24, I_total, '-', label='$I$', lw=4)
plt.plot(t/24, total_r - S_total - I_total, '-', label='$R$', lw=4)

plt.legend(fontsize=26); plt.grid()
#plt.autoscale(enable=True, axis='x', tight=True)

ax=plt.subplot()
#ax.grid(which="both")
#ax.set_xlabel("x")
#ax.set_ylabel("y")

ax.set_xlim([0,100])
#ax.set_ylim([0,2.0*10**1])

f_data = open('network-SIR.txt', mode='w')
for i in range(len(t)):
    f_data.write(str(t[i]/24) + ' ' + str(S_total[i]) + ' ' + str(I_total[i]) + '\n')
f_data.close()
```

The results of each node

```python
fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

In_total = []
for i in range(Nnode):
    In_total.append(0.0)

for i in range(Nnode):
    for j in range(M):
        In_total[i] += I[j*M + i]
for i in range(0,Nnode,100):
    plt.plot(t/24, In_total[i], '-', label='Node ' + str(i), lw=4)

plt.legend(fontsize=26); plt.grid()
#plt.autoscale(enable=True, axis='x', tight=True)

ax=plt.subplot()
#ax.grid(which="both")
#ax.set_xlabel("x")
#ax.set_ylabel("y")

ax.set_xlim([0,100])
#ax.set_ylim([0,2.0*10**1])
```


# Linear models with CNN features

```python
# Rather than importing everything manually, we'll make things easy
#   and load them all in utils.py, and just import them from there.
%matplotlib inline
import utils; reload(utils)
from utils import *
```

## Introduction

We need to find a way to convert the imagenet predictions to a probability of being a cat or a dog, since that is what the Kaggle competition requires us to submit. We could use the imagenet hierarchy to download a list of all the imagenet categories in each of the dog and cat groups, and could then solve our problem in various ways, such as:

- Finding the largest probability that's either a cat or a dog, and using that label
- Averaging the probability of all the cat categories and comparing it to the average of all the dog categories.

But these approaches have some downsides:

- They require manual coding for something that we should be able to learn from the data
- They ignore information available in the predictions; for instance, if the models predicts that there is a bone in the image, it's more likely to be a dog than a cat.

A very simple solution to both of these problems is to learn a linear model that is trained using the 1,000 predictions from the imagenet model for each image as input, and the dog/cat label as target.

```python
%matplotlib inline
from __future__ import division,print_function
import os, json
from glob import glob
import numpy as np
import scipy
from sklearn.preprocessing import OneHotEncoder
from sklearn.metrics import confusion_matrix
np.set_printoptions(precision=4, linewidth=100)
from matplotlib import pyplot as plt
import utils; reload(utils)
from utils import plots, get_batches, plot_confusion_matrix, get_data
```

```python
from numpy.random import random, permutation
from scipy import misc, ndimage
from scipy.ndimage.interpolation import zoom

import keras
from keras import backend as K
from keras.utils.data_utils import get_file
from keras.models import Sequential
from keras.layers import Input
from keras.layers.core import Flatten, Dense, Dropout, Lambda
from keras.layers.convolutional import Convolution2D, MaxPooling2D, ZeroPadding2D
from keras.optimizers import SGD, RMSprop
from keras.preprocessing import image
```

## Linear models in keras

It turns out that each of the Dense() layers is just a *linear model*, followed by a simple *activation function*. We'll learn about the activation function later - first, let's review how linear models work.

A linear model is (as I'm sure you know) simply a model where each row is calculated as *sum(row * weights)*, where *weights* needs to be learnt from the data, and will be the same for every row. For example, let's create some data that we know is linearly related:

```python
x = random((30,2))
y = np.dot(x, [2., 3.]) + 1.
```

```python
x[:5]
```

```python
y[:5]
```

We can use keras to create a simple linear model (*Dense()* - with no activation - in Keras) and optimize it using SGD to minimize mean squared error (*mse*):

```python
lm = Sequential([ Dense(1, input_shape=(2,)) ])
lm.compile(optimizer=SGD(lr=0.1), loss='mse')
```

(See the *Optim Tutorial* notebook and associated Excel spreadsheet to learn all about SGD and related optimization algorithms.)

This has now learnt internal weights inside the lm model, which we can use to evaluate the loss function (MSE).

```python
lm.evaluate(x, y, verbose=0)
```

```python
lm.fit(x, y, nb_epoch=5, batch_size=1)
```

```python
lm.evaluate(x, y, verbose=0)
```

And, of course, we can also take a look at the weights - after fitting, we should see that they are close to the weights we used to calculate y (2.0, 3.0, and 1.0).

```python
lm.get_weights()
```

## Train linear model on predictions

Using a Dense() layer in this way, we can easily convert the 1,000 predictions given by our model into a probability of dog vs cat--simply train a linear model to take the 1,000 predictions as input, and return dog or cat as output, learning from the Kaggle data. This should be easier and more accurate than manually creating a map from imagenet categories to one dog/cat category.

### Training the model

We start with some basic config steps. We copy a small amount of our data into a 'sample' directory, with the exact same structure as our 'train' directory--this is *always* a good idea in *all* machine learning, since we should do all of our initial testing using a dataset small enough that we never have to wait for it.

```python
#path = "data/dogscats/sample/"
path = "data/dogscats/"
model_path = path + 'models/'
if not os.path.exists(model_path): os.mkdir(model_path)
```

We will process as many images at a time as our graphics card allows. This is a case of trial and error to find the max batch size - the largest size that doesn't give an out of memory error.

```python
batch_size=100
#batch_size=4
```

We need to start with our VGG 16 model, since we'll be using its predictions and features.

```python
from vgg16 import Vgg16
vgg = Vgg16()
model = vgg.model
```

Our overall approach here will be:

1. Get the true labels for every image
2. Get the 1,000 imagenet category predictions for every image
3. Feed these predictions as input to a simple linear model.

Let's start by grabbing training and validation batches.

```python
# Use batch size of 1 since we're just doing preprocessing on the CPU
val_batches = get_batches(path+'valid', shuffle=False, batch_size=1)
batches = get_batches(path+'train', shuffle=False, batch_size=1)
```

Loading and resizing the images every time we want to use them isn't necessary - instead we should save the processed arrays. By far the fastest way to save and load numpy arrays is using bcolz. This also compresses the arrays, so we save disk space. Here are the functions we'll use to save and load using bcolz.

```python
import bcolz
def save_array(fname, arr): c=bcolz.carray(arr, rootdir=fname, mode='w'); c.flush()
def load_array(fname): return bcolz.open(fname)[:]
```

We have provided a simple function that joins the arrays from all the batches - let's use this to grab the training and validation data:

```python
val_data = get_data(path+'valid')
```

```python
trn_data = get_data(path+'train')
```

```python
trn_data.shape
```

```python
save_array(model_path+ 'train_data.bc', trn_data)
save_array(model_path + 'valid_data.bc', val_data)
```

We can load our training and validation data later without recalculating them:

```python
trn_data = load_array(model_path+'train_data.bc')
val_data = load_array(model_path+'valid_data.bc')
```

```python
val_data.shape
```

Keras returns *classes* as a single column, so we convert to one hot encoding

```python
def onehot(x): return np.array(OneHotEncoder().fit_transform(x.reshape(-1,1)).todense())
```

```python
val_classes = val_batches.classes
trn_classes = batches.classes
val_labels = onehot(val_classes)
trn_labels = onehot(trn_classes)
```

```python
trn_labels.shape
```

```python
trn_classes[:4]
```

```python
trn_labels[:4]
```

...and their 1,000 imagenet probabilties from VGG16--these will be the *features* for our linear model:

```python
trn_features = model.predict(trn_data, batch_size=batch_size)
val_features = model.predict(val_data, batch_size=batch_size)
```

```python
trn_features.shape
```

```python
save_array(model_path+ 'train_lastlayer_features.bc', trn_features)
save_array(model_path + 'valid_lastlayer_features.bc', val_features)
```

We can load our training and validation features later without recalculating them:

```python
trn_features = load_array(model_path+'train_lastlayer_features.bc')
val_features = load_array(model_path+'valid_lastlayer_features.bc')
```

Now we can define our linear model, just like we did earlier:

```python
# 1000 inputs, since that's the saved features, and 2 outputs, for dog and cat
lm = Sequential([ Dense(2, activation='softmax', input_shape=(1000,)) ])
lm.compile(optimizer=RMSprop(lr=0.1), loss='categorical_crossentropy', metrics=['accuracy'])
```

We're ready to fit the model!

```python
batch_size=64
```

```python
batch_size=4
```

```python
lm.fit(trn_features, trn_labels, nb_epoch=3, batch_size=batch_size,
       validation_data=(val_features, val_labels))
```

```python
lm.summary()
```

### Viewing model prediction examples

Keras' *fit()* function conveniently shows us the value of the loss function, and the accuracy, after every epoch ("*epoch*" refers to one full run through all training examples). The most important metrics for us to look at are for the validation set, since we want to check for over-fitting.

- **Tip**: with our first model we should try to overfit before we start worrying about how to handle that - there's no point even thinking about regularization, data augmentation, etc if you're still under-fitting! (We'll be looking at these techniques shortly).

As well as looking at the overall metrics, it's also a good idea to look at examples of each of:
1. A few correct labels at random
2. A few incorrect labels at random
3. The most correct labels of each class (ie those with highest probability that are correct)
4. The most incorrect labels of each class (ie those with highest probability that are incorrect)
5. The most uncertain labels (ie those with probability closest to 0.5).

Let's see what we, if anything, we can from these (in general, these are particularly useful for debugging problems in the model; since this model is so simple, there may not be too much to learn at this stage.)

Calculate predictions on validation set, so we can find correct and incorrect examples:

```python
# We want both the classes...
preds = lm.predict_classes(val_features, batch_size=batch_size)
# ...and the probabilities of being a cat
probs = lm.predict_proba(val_features, batch_size=batch_size)[:,0]
probs[:8]
```

```python
preds[:8]
```

Get the filenames for the validation set, so we can view images:

```python
filenames = val_batches.filenames
```

```python
# Number of images to view for each visualization task
n_view = 4
```

Helper function to plot images by index in the validation set:

```python
def plots_idx(idx, titles=None):
    plots([image.load_img(path + 'valid/' + filenames[i]) for i in idx], titles=titles)
```

```python
#1. A few correct labels at random
correct = np.where(preds==val_labels[:,1])[0]
idx = permutation(correct)[:n_view]
plots_idx(idx, probs[idx])
```

```python
#2. A few incorrect labels at random
incorrect = np.where(preds!=val_labels[:,1])[0]
idx = permutation(incorrect)[:n_view]
plots_idx(idx, probs[idx])
```

```python
#3. The images we most confident were cats, and are actually cats
correct_cats = np.where((preds==0) & (preds==val_labels[:,1]))[0]
most_correct_cats = np.argsort(probs[correct_cats])[::-1][:n_view]
plots_idx(correct_cats[most_correct_cats], probs[correct_cats][most_correct_cats])
```

```python
# as above, but dogs
correct_dogs = np.where((preds==1) & (preds==val_labels[:,1]))[0]
most_correct_dogs = np.argsort(probs[correct_dogs])[:n_view]
plots_idx(correct_dogs[most_correct_dogs], 1-probs[correct_dogs][most_correct_dogs])
```

```python
#3. The images we were most confident were cats, but are actually dogs
incorrect_cats = np.where((preds==0) & (preds!=val_labels[:,1]))[0]
most_incorrect_cats = np.argsort(probs[incorrect_cats])[::-1][:n_view]
plots_idx(incorrect_cats[most_incorrect_cats], probs[incorrect_cats][most_incorrect_cats])
```

```python
#3. The images we were most confident were dogs, but are actually cats
incorrect_dogs = np.where((preds==1) & (preds!=val_labels[:,1]))[0]
most_incorrect_dogs = np.argsort(probs[incorrect_dogs])[:n_view]
plots_idx(incorrect_dogs[most_incorrect_dogs], 1-probs[incorrect_dogs][most_incorrect_dogs])
```

```python
#5. The most uncertain labels (ie those with probability closest to 0.5).
most_uncertain = np.argsort(np.abs(probs-0.5))
plots_idx(most_uncertain[:n_view], probs[most_uncertain])
```

Perhaps the most common way to analyze the result of a classification model is to use a [confusion matrix](http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/). Scikit-learn has a convenient function we can use for this purpose:

```python
cm = confusion_matrix(val_classes, preds)
```

We can just print out the confusion matrix, or we can show a graphical view (which is mainly useful for dependents with a larger number of categories).

```python
plot_confusion_matrix(cm, val_batches.class_indices)
```

### About activation functions

Do you remember how we defined our linear model? Here it is again for reference:

```python
lm = Sequential([ Dense(2, activation='softmax', input_shape=(1000,)) ])
```

And do you remember the definition of a fully connected layer in the original VGG?:

```python
model.add(Dense(4096, activation='relu'))
```

You might we wondering, what's going on with that *activation* parameter? Adding an 'activation' parameter to a layer in Keras causes an additional function to be called after the layer is calculated. You'll recall that we had no such parameter in our most basic linear model at the start of this lesson - that's because a simple linear model has no *activation function*. But nearly all deep model layers have an activation function - specifically, a *non-linear* activation function, such as tanh, sigmoid (```1/(1+exp(x))```), or relu (```max(0,x)```, called the *rectified linear* function). Why?

The reason for this is that if you stack purely linear layers on top of each other, then you just end up with a linear layer! For instance, if your first layer was ```2*x```, and your second was ```-2*x```, then the combination is: ```-2*(2*x) = -4*x```. If that's all we were able to do with deep learning, it wouldn't be very deep! But what if we added a relu activation after our first layer? Then the combination would be: ```-2 * max(0, 2*x)```. As you can see, that does not simplify to just a linear function like the previous example--and indeed we can stack as many of these on top of each other as we wish, to create arbitrarily complex functions.

And why would we want to do that? Because it turns out that such a stack of linear functions and non-linear activations can approximate any other function just as close as we want. So we can **use it to model anything**! This extraordinary insight is known as the *universal approximation theorem*. For a visual understanding of how and why this works, I strongly recommend you read Michael Nielsen's [excellent interactive visual tutorial](http://neuralnetworksanddeeplearning.com/chap4.html).

The last layer generally needs a different activation function to the other layers--because we want to encourage the last layer's output to be of an appropriate form for our particular problem. For instance, if our output is a one hot encoded categorical variable, we want our final layer's activations to add to one (so they can be treated as probabilities) and to have generally a single activation much higher than the rest (since with one hot encoding we have just a single 'one', and all other target outputs are zero). Our classication problems will always have this form, so we will introduce the activation function that has these properties: the *softmax* function. Softmax is defined as (for the i'th output activation): ```exp(x[i]) / sum(exp(x))```.

I suggest you try playing with that function in a spreadsheet to get a sense of how it behaves.

We will see other activation functions later in this course - but relu (and minor variations) for intermediate layers and softmax for output layers will be by far the most common.

# Modifying the model

## Retrain last layer's linear model

Since the original VGG16 network's last layer is Dense (i.e. a linear model) it seems a little odd that we are adding an additional linear model on top of it. This is especially true since the last layer had a softmax activation, which is an odd choice for an intermediate layer--and by adding an extra layer on top of it, we have made it an intermediate layer. What if we just removed the original final layer and replaced it with one that we train for the purpose of distinguishing cats and dogs? It turns out that this is a good idea - as we'll see!

We start by removing the last layer, and telling Keras that we want to fix the weights in all the other layers (since we aren't looking to learn new parameters for those other layers).

```python
vgg.model.summary()
```

```python
model.pop()
for layer in model.layers: layer.trainable=False
```

**Careful!** Now that we've modified the definition of *model*, be careful not to rerun any code in the previous sections, without first recreating the model from scratch! (Yes, I made that mistake myself, which is why I'm warning you about it now...)

Now we're ready to add our new final layer...

```python
model.add(Dense(2, activation='softmax'))
```

```python
??vgg.finetune
```

...and compile our updated model, and set up our batches to use the preprocessed images (note that now we will also *shuffle* the training batches, to add more randomness when using multiple epochs):

```python
gen=image.ImageDataGenerator()
batches = gen.flow(trn_data, trn_labels, batch_size=batch_size, shuffle=True)
val_batches = gen.flow(val_data, val_labels, batch_size=batch_size, shuffle=False)
```

We'll define a simple function for fitting models, just to save a little typing...

```python
def fit_model(model, batches, val_batches, nb_epoch=1):
    model.fit_generator(batches, samples_per_epoch=batches.N, nb_epoch=nb_epoch,
                        validation_data=val_batches, nb_val_samples=val_batches.N)
```

...and now we can use it to train the last layer of our model!

(It runs quite slowly, since it still has to calculate all the previous layers in order to know what input to pass to the new final layer. We could precalculate the output of the penultimate layer, like we did for the final layer earlier - but since we're only likely to want one or two iterations, it's easier to follow this alternative approach.)

```python
opt = RMSprop(lr=0.1)
model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])
```

```python
fit_model(model, batches, val_batches, nb_epoch=2)
```

Before moving on, go back and look at how little code we had to write in this section to finetune the model. Because this is such an important and common operation, keras is set up to make it as easy as possible. We didn't even have to use any external helper functions in this section.

It's a good idea to save weights of all your models, so you can re-use them later. Be sure to note the git log number of your model when keeping a research journal of your results.

```python
model.save_weights(model_path+'finetune1.h5')
```

```python
model.load_weights(model_path+'finetune1.h5')
```

```python
model.evaluate(val_data, val_labels)
```

We can look at the earlier prediction examples visualizations by redefining *probs* and *preds* and re-using our earlier code.

```python
preds = model.predict_classes(val_data, batch_size=batch_size)
probs = model.predict_proba(val_data, batch_size=batch_size)[:,0]
probs[:8]
```

```python
cm = confusion_matrix(val_classes, preds)
```

```python
plot_confusion_matrix(cm, {'cat':0, 'dog':1})
```

## Retraining more layers

Now that we've fine-tuned the new final layer, can we, and should we, fine-tune *all* the dense layers? The answer to both questions, it turns out, is: yes! Let's start with the "can we" question...

### An introduction to back-propagation

The key to training multiple layers of a model, rather than just one, lies in a technique called "back-propagation" (or *backprop* to its friends). Backprop is one of the many words in deep learning parlance that is creating a new word for something that already exists - in this case, backprop simply refers to calculating gradients using the *chain rule*. (But we will still introduce the deep learning terms during this course, since it's important to know them when reading about or discussing deep learning.)

As you (hopefully!) remember from high school, the chain rule is how you calculate the gradient of a "function of a function"--something of the form *f(u), where u=g(x)*. For instance, let's say your function is ```pow((2*x), 2)```. Then u is ```2*x```, and f(u) is ```power(u, 2)```. The chain rule tells us that the derivative of this is simply the product of the derivatives of f() and g(). Using *f'(x)* to refer to the derivative, we can say that: ```f'(x) = f'(u) * g'(x) = 2*u * 2 = 2*(2*x) * 2 = 8*x```.

Let's check our calculation:

```python
# sympy let's us do symbolic differentiation (and much more!) in python
import sympy as sp
# we have to define our variables
x = sp.var('x')
# then we can request the derivative or any expression of that variable
pow(2*x,2).diff()
```

The key insight is that the stacking of linear functions and non-linear activations we learnt about in the last section is simply defining a function of functions (of functions, of functions...). Each layer is taking the output of the previous layer's function, and using it as input into its function. Therefore, we can calculate the derivative at any layer by simply multiplying the gradients of that layer and all of its following layers together! This use of the chain rule to allow us to rapidly calculate the derivatives of our model at any layer is referred to as *back propagation*.

The good news is that you'll never have to worry about the details of this yourself, since libraries like Theano and Tensorflow (and therefore wrappers like Keras) provide *automatic differentiation* (or *AD*). ***TODO***

### Training multiple layers in Keras

The code below will work on any model that contains dense layers; it's not just for this VGG model.

NB: Don't skip the step of fine-tuning just the final layer first, since otherwise you'll have one layer with random weights, which will cause the other layers to quickly move a long way from their optimized imagenet weights.

```python
layers = model.layers
# Get the index of the first dense layer...
first_dense_idx = [index for index,layer in enumerate(layers) if type(layer) is Dense][0]
# ...and set this and all subsequent layers to trainable
for layer in layers[first_dense_idx:]: layer.trainable=True
```

Since we haven't changed our architecture, there's no need to re-compile the model - instead, we just set the learning rate. Since we're training more layers, and since we've already optimized the last layer, we should use a lower learning rate than previously.

```python
K.set_value(opt.lr, 0.01)
fit_model(model, batches, val_batches, 3)
```

This is an extraordinarily powerful 5 lines of code. We have fine-tuned all of our dense layers to be optimized for our specific data set. This kind of technique has only become accessible in the last year or two - and we can already do it in just 5 lines of python!

```python
model.save_weights(model_path+'finetune2.h5')
```

There's generally little room for improvement in training the convolutional layers, if you're using the model on natural images (as we are). However, there's no harm trying a few of the later conv layers, since it may give a slight improvement, and can't hurt (and we can always load the previous weights if the accuracy decreases).

```python
for layer in layers[12:]: layer.trainable=True
K.set_value(opt.lr, 0.001)
```

```python
fit_model(model, batches, val_batches, 4)
```

```python
model.save_weights(model_path+'finetune3.h5')
```

You can always load the weights later and use the model to do whatever you need:

```python
model.load_weights(model_path+'finetune2.h5')
model.evaluate_generator(get_batches('valid', gen, False, batch_size*2), val_batches.N)
```

```python

```