 $\quad \quad \quad $  *Credit: [Japan Times](https://www.japantimes.co.jp/news/2017/11/10/national/japans-casinos-work-keep-yakuza-deal-problem-drinking-experts/#.W-RJPJMzaUk).*

# From Scratch: Bayesian Inference, Markov Chain Monte Carlo and Metropolis Hastings, in python

In one of the courses during my data science degree, I came across a technique called Markov Chain Monte Carlo, or as it's more commonly referred to, MCMC. The description for this method stated something along the lines of: MCMC is a class of techniques for sampling from a probability distribution and can be used to estimate the distribution of parameters given a set of observations.

Back then, I did not think much of it. I thought, "oh it's just another sampling technique", and I decided I'd read on it when I'd practically need it. This need never emerged, or perhaps it did and I wrongly used something else.

## So why the interest now?

Recently, I have seen a few discussions about MCMC and some of its implementations, specifically the Metropolis-Hastings algorithm and the PyMC3 library. $Markov \: Chain\: Monte\: Carlo\: in\: Python\: - \:A \:Complete\: Real$-$World \:Implementation$ was the article that caught my attention the most. In it, William Koehrsen explains how he was able to learn the approach by applying it to a real world problem: to estimate the parameters of a logistic function that represents his sleeping patterns.

\begin{equation} P(sleep/t,\alpha,\beta)=\dfrac{1}{1+e^{\beta t + \alpha}}  \end{equation}

Mr. Koehrsen uses the PyMC3 implementation of the Metropolis-Hastings algorithm to estimate $\beta$ and $\alpha$, thus computing the entire parameter space, and deriving the most likely logistic model.

## So why am I talking about all that?

I this article, I propose to implement from scratch, my own version of the Metropolis-Hastings algorithm to find parameter distributions for a dummy data example and then of a real world problem.

I figured that if I get my hands dirty, I might finally be able to understand it. I will only use numpy to implement the algorithm, and matplotlib to draw pretty things. Alternatively, scipy can be used to compute the density functions, but I will also show how to implement them using numpy.

## Flow of the article:

    * At first, I will introduce Bayesian inference, MCMC-MH and their mathematical components.
    * Second, I will explain the algorithm using dummy data.
    * Third, I will apply it to a real world problem.

# Part 1: Bayesian inference, Markov Chain Monte Carlo, and Metropolis-Hastings

## A bird's eye view on the philosophy of probabilities

In order to talk about Bayesian inference and what follows it, I shall first explain what the Bayesian view of probability is, and situate it within its historical context

### Frequentist vs Bayesian thinking

There are two major interpretations to probabilities: Bayesian and Frequentist.

From a **Frequentist's** perspective, probabilities represent long term frequencies with which events occur. A frequentist can say that the probability of having tails from a coin toss is equal to 0.5 *on the long run*. Each new experiment, can be considered as one of an infinite sequence of possible repetitions of the same experiment. The idea is that there is *no* belief in a frequentist's view of probability. The probability of event $x$ happening out of n trials is equal to the following frequency: $P(x)=\dfrac{n_x}{n}$, and the true probability is reached when $n->\infty$. Frequentists will never say "I am 45% (0.45) sure that there is lasagna for lunch today", since this does not happen on the long run. Commonly, a frequentist approach is referred to as the *objective* approach since there is no expression of belief and/or prior events in it.

On the other hand, in **Bayesian** thinking, probabilities are treated as an expression of belief. Therefore it is perfectly reasonable for a Bayesian to say "I am 50% (0.5) sure that there is lasagna for lunch today". By combining *prior* beliefs and current events (the *evidence*) one can compute the *posterior*, the belief that there is lasagna today. The idea behind Bayesian thinking is to keep updating the beliefs as more evidence is provided. Since this approach deals with belief, it is usually referred to as the *subjective* view on probability.

### Bayesian inference

 $\quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad $  *A supposed portrait of Thomas Bayes, an English statistician, philosopher, and theologian. Image Credit: [Farnam Street](https://fs.blog/2018/09/bayes-theorem/).*

In the philosophy of decision making, Bayesian inference is closely related to Bayesian probability, in the sense that it manipulates priors, evidence, and likelihood to compute the posterior. Given some event B, what is the probability that event A occurs?. This is answered by Bayes' famous formula: $P(A/B)=\dfrac{P(B/A)P(A)}{P(B)}$

* $P(A/B)$ is the **posterior**. What we wish to compute.

* $P(B/A)$ is the **likelihood**. Assuming A occured, how likely is B.

* $P(A)$ is the **prior**. How likely the event $A$ is regardless of evidence.

* $P(B)$ is the **evidence**. How likely the evidence $B$ is regardless of the event.

In our case, we are mostly interested in the specific formulation of Bayes' formula:

$P(\theta/D)=\dfrac{P(D/\theta)P(\theta)}{P(D)}$ where, $P(\theta/D)$ is the **posterior**, $P(D/\theta)$ is the **likelihood**, $P(\theta)$ is the **prior** and $P(D)$ is the **evidence**.

We would like to find the most likely distribution of $\theta$, the parameters of the model explaining the data, D.

Computing some of these probabilities can be tedious, especially the evidence $P(D)$. Also, other problems can arise such as those of ensuring conjugacy, which I will not dive into in this article. Luckily, some techniques, namely MCMC, allow us to sample from the posterior, and a draw distributions over our parameters without having to worry about computing the evidence, nor about conjugacy.

### Markov Chain Monte Carlo

MCMC allows us to draw samples from any distribution that we can't sample from directly. It can be used to sample from the posterior distribution over parameters.
It has seen much success in many applications, such as computing the distribution of parameters, given a set of observations and some prior belief, and also computing high dimensional integrals in physics and in digital communications.

Bottom line: **It can be used to compute the distribution over the parameters, given a set of observations and a prior belief.**

### Metropolis-Hastings

MCMC is a class of methods. Metropolis-Hastings is a specific implementation of MCMC. It works well in high dimensional spaces as opposed to Gibbs sampling and rejection sampling.

This technique requires a simple distribution called the **proposal distribution** (Which I like to call **transition model**) $Q(\theta^\prime/\theta)$ to help draw samples from an intractable posterior distribution $P(\Theta=\theta/D)$.

Metropolis-Hastings uses $Q$ to randomly walk in the distribution space, accepting or rejecting jumps to new positions based on how likely the sample is. This "memoriless" random walk is the "Markov Chain" part of MCMC.

The "likelihood" of each new sample is decided by a function $f$ . That's why $f$ must be proportional to the posterior we want to sample from. f is commonly chosen to be a probability density function that expresses this proportionality.

To get a new position of the parameter, just take our current one $\theta$, and propose a new one $\theta^\prime$, that is a random sample drawn from $Q(\theta^\prime/\theta)$. Often this is a symmetric distribution. For instance, a normal distribution with mean $\theta$ and some standard deviation $\sigma$: $Q(\theta^\prime/\theta) = \mathcal{N}(\theta, \sigma)$

To decide if $\theta^\prime$ is to be accepted or rejected, the following ratio must be computed for each new value of $\theta^\prime$: $\dfrac{P(\theta^\prime/D)}{P(\theta/D)}$. Using Bayes' formula this can be easily re-formulated as: $\dfrac{P(D/\theta^\prime)P(\theta^\prime)}{P(D/\theta)P(\theta)}$ (The evidence $P(D)$ is simply crossed out during the division).  $\dfrac{P(D/\theta^\prime)P(\theta^\prime)}{P(D/\theta)P(\theta)}$ is also equivalent to $\dfrac{\prod_i^nf(d_i/\Theta=\theta^\prime)P(\theta^\prime)}{\prod_i^nf(d_i/\Theta=\theta)P(\theta)}$

$$\begin{equation} P(\text{accept}) = \begin{cases}\dfrac{\prod_i^nf(d_i/\Theta=\theta^\prime)P(\theta^\prime)}{\prod_i^nf(d_i/\Theta=\theta)P(\theta)}, & \prod_i^nf(d_i/\Theta=\theta)P(\theta)>\prod_i^nf(d_i/\Theta=\theta^\prime)P(\theta^\prime) \\  1, & \prod_i^nf(d_i/\Theta=\theta)P(\theta)\leq \prod_i^nf(d_i/\Theta=\theta^\prime)P(\theta^\prime) \end{cases} \end{equation}$$

This means that if a θ' is more likely than the current θ, then we always accept θ'. If it is less likely than the current θ, then we might accept it or reject it randomly with decreasing probability, the less likely it is.

*Note: The prior components are often crossed if there is no preference or restrictions on the parameters.*

#### Metropolis-Hastings Algorithm:
* given:
    * $f$, the PDF of the distribution to sample from
    * $Q$, the transition model
    * $\theta_0$, a first guess for $\theta$
    * $\theta = \theta_0$

* for $n$ iterations
    * $p =  f(D/\Theta=\theta)P(\theta)$
    *  $\theta^\prime = Q(\theta_i)$
    *  $p^\prime = f(D/\Theta=\theta^\prime)P(\theta^\prime)$
    *  $ratio = \dfrac{p^\prime}{p}$
    * generate a uniform random number $r$ in [0,1]
    *  if $r<ratio$:
        * set $\theta_i = \theta^\prime$

```python
import numpy as np
import scipy
import scipy.stats
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
```

# Part 2: Dummy data example
## Step 1: Data generation

We generate 30,000 samples from a normal distribution with $\mu$ = 10, and $\sigma$= 3, but we can only observe 1000 of them.

```python
mod1=lambda t:np.random.normal(10,3,t)

#Form a population of 30,000 individual, with average=10 and scale=3
population = mod1(30000)
#Assume we are only able to observe 1,000 of these individuals.
observation = population[np.random.randint(0, 30000, 1000)]

fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(1,1,1)
ax.hist( observation,bins=35 ,)
ax.set_xlabel("Value")
ax.set_ylabel("Frequency")
ax.set_title("Figure 1: Distribution of 1000 observations sampled from a population of 30,000 with $\mu$=10, $\sigma$=3")
mu_obs=observation.mean()
mu_obs
```

## Step 2: What do we want?

We would like to find a distribution for $\sigma_{obs}$ using the 1000 observed samples. Those of you who are adept at mathematics will say that there is a formula for computing $\sigma$ ( $\sigma=\sqrt{\dfrac{1}{n}\sum_i^n(d_i-\mu)^2}$)! Why do we want to sample and whatnot?? Well, this is just a dummy data example, the real problem is in part 3, where the parameters cannot be computed directly. Plus here, we are not trying to find *a* value for $\sigma$, but rather, we are trying to compute a distribution of the possible values of $\sigma$.

## Step 3: Define the PDF and the transition model.

From Figure 1, we can see that the data is normally distributed. The mean can be easily computed by taking the average of the values of the 1000 samples. By doing that, we get for example $\mu_{obs}=9.8$.

### For the transition model/ proposal distribution:
I have no specific distribution in mind, so I will choose a simple one: the Normal distribution!

\begin{equation} Q(\sigma_{new} / \sigma_{current}) = N(\mu=\sigma_{current},\sigma'=1) \end{equation}

Note that $\sigma'$ is unrelated to $\sigma_{new}$ and $\sigma_{current}$. It simply specifies the standard deviation of the parameter space. It can be any value desired. It only affects the convergence time of the algorithm.

### For the PDF:
Since f should be proportional to the posterior, we choose f to be the following Probability Density Function (PDF), for each data point $d_i$ in the data D:

\begin{equation} f(d_i/ \mu,\sigma^2) = \dfrac{1}{\sqrt{2\pi\sigma^2}}e^{-\dfrac{(d_i-\mu)^2}{2\sigma^2}} \end{equation}

In our case, $\theta$ is made up of two values: $[\mu,\sigma]$, and that $\mu$ is a constant, $\mu = \mu_{obs}$.

## Step 4: Define when we accept or reject $\sigma_{new}$:
We accept $\sigma_{new}$ if:

$\dfrac{Likelihood(D/\mu_{obs},\sigma_{new})*prior(\mu_{obs},\sigma_{new})}{Likelihood(D/\mu_{obs},\sigma_{current})*prior(\mu_{obs},\sigma_{current})}>1     \quad \quad \quad \quad \quad      (1)$

If this ratio is smaller or equal to 1, then we compare it to a uniformly generated random number in the closed set [0,1]. If the ratio is larger than the random number, we accept $\sigma_{new}$, otherwise we reject it.

*Note: Since we will be computing this ratio to decide which parameters should be accepted, it is imperative to make sure that the adopted function $f$  is proportional to the posterior itself, $P(\sigma/ D,\mu)$, which in that case is verified. ($f$ is the PDF of P)*

## Step 5: Define the prior and the likelihood:
### For the Prior $P(\theta)$ which we can alternatively note $P(\sigma)$ since $\mu$ is constant:
We don't have any preferences for the values that $\sigma_{new}$ and $\sigma_{current}$ can take. The only thing worth noting is that they should be positive. Why? Intuitively, the standard deviation measures dispersion. Dispersion is a distance, and distances cannot be negative. Mathematically, $\sigma=\sqrt{\dfrac{1}{n}\sum_i^n(d_i-\mu)^2}$, and the square root of a number cannot be negative. We strictly enforce this in the prior.

### For the likelihood :
The total likelihood for a set of observation $D$ is: $Likelihood(D/\mu_{obs},\sigma_{a}) = \prod_i^n f(d_i/\mu_{obs},\sigma_{a}) $, where $a=new \: or \: current$.

In our case, we will log both the prior and the likelihood function. Why log? Simply because it helps with numerical stability, i.e. multiplying thousands of small values (probabilities, likelihoods, etc..) can cause an underflow in the system's memory, and the log is a perfect solution because it transforms multiplications to additions and small positive numbers into non-small negative numbers.

Therefore our acceptance condition from equation $(1)$ becomes:

Accept $\sigma_{new}$ if:

$\quad \quad \quad \quad \quad Log(Likelihood(D/\mu_{obs},\sigma_{new})) + Log(prior(\mu_{obs},\sigma_{new})) - (Log(Likelihood(D/\mu_{obs},\sigma_{current}))+$

$Log(prior(\mu_{obs},\sigma_{current})))>0$

 $\quad$

 Equivalent to:

 $\sum_i^nLog(f(d_i/\mu_{obs},\sigma_{new})) + Log(prior(\mu_{obs},\sigma_{new})) - \sum_i^nLog(f(d_i/\mu_{obs},\sigma_{current}))-Log(prior(\mu_{obs},\sigma_{current}))>0$

 $\quad$

  Equivalent to:

  $\sum_i^nLog(f(d_i/\mu_{obs},\sigma_{new})) + Log(prior(\mu_{obs},\sigma_{new})) > \sum_i^nLog(f(d_i/\mu_{obs},\sigma_{current}))+Log(prior(\mu_{obs},\sigma_{current}))$

   $\quad$

  Equivalent to:

  $\sum_i^n (-Log(\sigma_{new}\sqrt{2\pi})-\dfrac{(d_i-\mu_{obs})^2}{2\sigma_{new}^2}) + Log(prior(\mu_{obs},\sigma_{new})) \quad > $

 $ \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad(2)$
 $ \quad \quad \quad \quad \quad \sum_i^n (-Log(\sigma_{current}\sqrt{2\pi})-\dfrac{(d_i-\mu_{obs})^2}{2\sigma_{current}^2})+Log(prior(\mu_{obs},\sigma_{current}))  $

  This form can be reduced even more by taking the square root and the multiplication out of the log.

```python
#The tranistion model defines how to move from sigma_current to sigma_new
transition_model = lambda x: [x[0],np.random.normal(x[1],0.5,(1,))[0]]

def prior(x):
    #x[0] = mu, x[1]=sigma (new or current)
    #returns 1 for all valid values of sigma. Log(1) =0, so it does not affect the summation.
    #returns 0 for all invalid values of sigma (<=0). Log(0)=-infinity, and Log(negative number) is undefined.
    #It makes the new sigma infinitely unlikely.
    if(x[1] <=0):
        return 0
    return 1

#Computes the likelihood of the data given a sigma (new or current) according to equation (2)
def manual_log_like_normal(x,data):
    #x[0]=mu, x[1]=sigma (new or current)
    #data = the observation
    return np.sum(-np.log(x[1] * np.sqrt(2* np.pi) )-((data-x[0])**2) / (2*x[1]**2))

#Same as manual_log_like_normal(x,data), but using scipy implementation. It's pretty slow.
def log_lik_normal(x,data):
    #x[0]=mu, x[1]=sigma (new or current)
    #data = the observation
    return np.sum(np.log(scipy.stats.norm(x[0],x[1]).pdf(data)))

#Defines whether to accept or reject the new sample
def acceptance(x, x_new):
    if x_new>x:
        return True
    else:
        accept=np.random.uniform(0,1)
        # Since we did a log likelihood, we need to exponentiate in order to compare to the random number
        # less likely x_new are less likely to be accepted
        return (accept < (np.exp(x_new-x)))

def metropolis_hastings(likelihood_computer,prior, transition_model, param_init,iterations,data,acceptance_rule):
    # likelihood_computer(x,data): returns the likelihood that these parameters generated the data
    # transition_model(x): a function that draws a sample from a symmetric distribution and returns it
    # param_init: a starting sample
    # iterations: number of accepted to generated
    # data: the data that we wish to model
    # acceptance_rule(x,x_new): decides whether to accept or reject the new sample
    x = param_init
    accepted = []
    rejected = []
    for i in range(iterations):
        x_new =  transition_model(x)
        x_lik = likelihood_computer(x,data)
        x_new_lik = likelihood_computer(x_new,data)
        if (acceptance_rule(x_lik + np.log(prior(x)),x_new_lik+np.log(prior(x_new)))):
            x = x_new
            accepted.append(x_new)
        else:
            rejected.append(x_new)

    return np.array(accepted), np.array(rejected)
```

## Step 6: Run the algorithm with initial parameters and collect accepted and rejected samples

```python
accepted, rejected = metropolis_hastings(manual_log_like_normal,prior,transition_model,[mu_obs,0.1], 50000,observation,acceptance)
```

    D:\Anaconda3\lib\site-packages\ipykernel_launcher.py:17: RuntimeWarning: invalid value encountered in log
    D:\Anaconda3\lib\site-packages\ipykernel_launcher.py:51: RuntimeWarning: divide by zero encountered in log

### The algorithm accepted 8803 samples (which might be different on each new run). The last 10 samples contain the following  values for $\sigma$:

```python
accepted[-10:,1]
```

    array([3.12775158, 3.15044065, 3.17207839, 3.1021715 , 3.1767237 ,
           3.02948795, 3.10115514, 3.03894704, 3.01642534, 3.0320166 ])

```python
print(accepted.shape)
```

    (8693, 2)

```python
fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(2,1,1)

ax.plot( rejected[0:50,1], 'rx', label='Rejected',alpha=0.5)
ax.plot( accepted[0:50,1], 'b.', label='Accepted',alpha=0.5)
ax.set_xlabel("Iteration")
ax.set_ylabel("$\sigma$")
ax.set_title("Figure 2: MCMC sampling for $\sigma$ with Metropolis-Hastings. First 50 samples are shown.")
ax.grid()
ax.legend()

ax2 = fig.add_subplot(2,1,2)
to_show=-accepted.shape[0]
ax2.plot( rejected[to_show:,1], 'rx', label='Rejected',alpha=0.5)
ax2.plot( accepted[to_show:,1], 'b.', label='Accepted',alpha=0.5)
ax2.set_xlabel("Iteration")
ax2.set_ylabel("$\sigma$")
ax2.set_title("Figure 3: MCMC sampling for $\sigma$ with Metropolis-Hastings. All samples are shown.")
ax2.grid()
ax2.legend()

fig.tight_layout()
accepted.shape
```

So, starting from an initial σ of 0.1, the algorithm converged pretty quickly to the expected value of 3. That said, it's only sampling in a 1D space…. so it's not very surprising.

### We consider the initial 25% of the values of $\sigma$ to be "burn-in", so we drop them.
### Let's visualize the trace of  $\sigma$ and the histogram of the trace.

```python

show=int(-0.75*accepted.shape[0])
hist_show=int(-0.75*accepted.shape[0])

fig = plt.figure(figsize=(20,10))
ax = fig.add_subplot(1,2,1)
ax.plot(accepted[show:,1])
ax.set_title("Figure 4: Trace for $\sigma$")
ax.set_ylabel("$\sigma$")
ax.set_xlabel("Iteration")
ax = fig.add_subplot(1,2,2)
ax.hist(accepted[hist_show:,1], bins=20,density=True)
ax.set_ylabel("Frequency (normed)")
ax.set_xlabel("$\sigma$")
ax.set_title("Figure 5: Histogram of $\sigma$")
fig.tight_layout()

ax.grid("off")

```

### The most likely value for $\sigma$ is around 3.1. This is a bit more than the original value of 3.0. The difference is due to us observing only 3.33% of the original population (1,000 out of 30,000)

## Predictions:
First, we average the last 75% of accepted samples of σ, and we generate 30,000 random individuals from a normal distribution with μ=9.8 and σ=3.05 (the average of the last 75% of accepted samples) which is actually better than the most likely value of 3.1.

```python
mu=accepted[show:,0].mean()
sigma=accepted[show:,1].mean()
print(mu, sigma)
model = lambda t,mu,sigma:np.random.normal(mu,sigma,t)
observation_gen=model(population.shape[0],mu,sigma)
fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(1,1,1)
ax.hist( observation_gen,bins=70 ,label="Predicted distribution of 30,000 individuals")
ax.hist( population,bins=70 ,alpha=0.5, label="Original values of the 30,000 individuals")
ax.set_xlabel("Mean")
ax.set_ylabel("Frequency")
ax.set_title("Figure 6: Posterior distribution of predicitons")
ax.legend()
```

# Part 3: A real world example

*Credit: Amateur astronomer Alan Friedman on July 10, 2012.*

A sunspot is a region on the Sun's surface (photosphere) that is marked by a lower temperature than its environment. These reduced temperatures are caused by concentrations of magnetic field flux that inhibit convection by an effect similar to [eddy current brakes](https://en.wikipedia.org/wiki/Eddy_current). Sunspots usually appear in pairs of opposite magnetic polarity. Their number varies according to the approximately 11-year solar cycle.

The data we will be working on is the "Monthly mean total sunspot number", for each month from January 1749 to November 2018. This is data collected, curated and made publicly available by the [World Data Center for the production, preservation and dissemination of the international sunspot number](http://www.sidc.be/silso/home).

## Let's plot the data over the years to see what the distribution might be like.

```python
sunspots = np.loadtxt("data/SN_m_tot_V2.0.csv", delimiter=';')

#years is the third column
years = sunspots[:,2]
#activity is the 4th column
activity = sunspots[:,3] + 0.1
print(years.shape,activity.shape)
fig = plt.figure(figsize=(20,20))
ax = fig.add_subplot(2,1,1)

ax.plot(years, activity, label="Sunspot count")
ax.set_yticks(np.arange(0,500,100))
ax.set_xlabel("Years")
ax.set_ylabel("Monthly mean count of sunspots")
ax.set_title("Figure 7: Sunspots, 1749-2018")
ax.legend()

ax = fig.add_subplot(2,1,2)
print(years[432])
ax.plot(years[:432], activity[:432], label="Sunspot count")
ax.set_yticks(np.arange(0,500,100))
ax.set_xlabel("Years")
ax.set_ylabel("Monthly mean count of sunspots")
ax.set_title("Figure 8: Sunspots, 1749-1785")
ax.legend()
```

## It seems like we could model this phenomenon with a gamma distribution, with a new cycle resetting every 12 years.

A gamma distribution Γ is a two-parameter family of continuous probability distributions, the parameters are the shape a and the scale b. A random variable X that is gamma-distributed is noted X~Γ(a, b), and in our case X is the count of sunspots. The two parameters a and b are the unknowns that we would like to calculate distributions for.
*Credit: Wikipedia Commons.*

For example, in the first cycle, the sunspot counts start from their highest at about 300 at the end of 1749, and fall to their lowest 6 years after, during 1755. Then the number rises up again to it's maximum during 1761 and 1762 before falling again during 1766 and so on...

## Let's make sure by plotting a histogram of sunspot counts:

```python
fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(1,1,1)
ax.hist(activity, bins=40, density=True)
ax.set_xlabel("Sunspot count")
ax.set_ylabel("Frequency")
ax.set_title("Figure 9: Histogram showing the frequency of sunspot counts over 270 years (1749-2018)")
```

## Indeed, it does seem like the frequency of counts follows a gamma distribution

The gamma distribution, has for PDF, $f/f(x;a,b) =\dfrac{b^a x^{a-1}e^{-b x}}{\Gamma{(a)}}$ where $\Gamma$ is the gamma function (not to be confused with the gamma distribution which requires 2 parameters): $\Gamma{(a)}=(a-1)!$

Following the same procedure as in the dummy data example, we can write down the log likelihood from this pdf (see code below). Alternatively, one could use the $scipy.stats.gamma(a,b).pdf(x)$ function to compute it.

Since a and b must be positive, we enforce this in the prior.

```python
transition_model = lambda x: np.random.normal(x,[0.05,5],(2,))
import math
def prior(w):
    if(w[0]<=0 or w[1] <=0):
        return 0
    else:
        return 1

def manual_log_lik_gamma(x,data):
    return np.sum((x[0]-1)*np.log(data) - (1/x[1])*data - x[0]*np.log(x[1]) - np.log(math.gamma(x[0])))

def log_lik_gamma(x,data):
    return np.sum(np.log(scipy.stats.gamma(a=x[0],scale=x[1],loc=0).pdf(data)))

```

```python
accepted, rejected = metropolis_hastings(manual_log_lik_gamma,prior,transition_model,[4, 10], 50000,activity,acceptance)
```

```python
print(accepted.shape)
accepted[-10:]
```

    (8537, 2)

    array([[ 0.97009904, 85.34205726],
           [ 0.97018033, 88.24737974],
           [ 0.97439353, 87.74024406],
           [ 0.97891065, 85.53414066],
           [ 0.96527862, 86.48928538],
           [ 0.98343021, 84.39404222],
           [ 1.00547929, 82.69265895],
           [ 0.95034478, 86.55690013],
           [ 0.98935646, 85.16989734],
           [ 1.00038475, 82.32425891]])

Starting from a=4, and b =10, the algorithm accepted 8561 pairs of samples, the last value for a is 1.01307402 and the last value for b is 83.40995308, which are pretty far off the initial values.

## As with the dummy data example, let's see how the algorithm worked its way to these values:

```python

fig = plt.figure(figsize=(10,20))
ax = fig.add_subplot(3,1,1)
ax.plot(accepted[:50,0], accepted[:50,1], label="Path")
ax.plot(accepted[:50,0], accepted[:50,1], 'b.', label='Accepted')
ax.plot(rejected[:50,0], rejected[:50,1], 'rx', label='Rejected')
ax.set_xlabel("a")
ax.set_ylabel("b")
ax.legend()
ax.set_title("Figure 10: MCMC sampling for $a$ and $b$ with Metropolis-Hastings. First 50 samples are shown.")

ax = fig.add_subplot(3,1,2)
ax.plot(accepted[:,0], accepted[:,1], label="Path")
ax.plot(accepted[:,0], accepted[:,1], 'b.', label='Accepted',alpha=0.3)
ax.plot(rejected[:,0], rejected[:,1], 'rx', label='Rejected',alpha=0.3)
ax.set_xlabel("a")
ax.set_ylabel("b")
ax.legend()
ax.set_title("Figure 11: MCMC sampling for $a$ and $b$ with Metropolis-Hastings. All samples are shown.")

to_show=50
ax = fig.add_subplot(3,1,3)
ax.plot(accepted[-to_show:,0], accepted[-to_show:,1], label="Path")
ax.plot(accepted[-to_show:,0], accepted[-to_show:,1], 'b.', label='Accepted',alpha=0.5)
ax.plot(rejected[-to_show:,0], rejected[-to_show:,1], 'rx', label='Rejected',alpha=0.5)
ax.set_xlabel("a")
ax.set_ylabel("b")
ax.legend()
ax.set_title("Figure 12: MCMC sampling for $a$ and $b$ with Metropolis-Hastings. Last 50 samples are shown.")
```

As we can see from figures 10, 11, and 12, the algorithm converges quickly to the [a=1,b=85] zone.

Tip: when the algorithm starts to heavily reject samples, that means that we have reached a zone of saturation of the likelihood. Commonly, this can be interpreted as having reached the optimal parameter space from which we can sample, i.e. there is very little reason for the algorithm to accept new values. This is marked in figures 11, and 12 where the algorithm no longer accepts any values outside of a small range.

### We consider the initial 50% of the values of a and b to be "burn-in", so we drop them.
### Let's visualize the traces of  and b and the histogram of the traces.

```python
show=int(-0.5*accepted.shape[0])
hist_show=int(-0.50*accepted.shape[0])

fig = plt.figure(figsize=(15,7))
ax = fig.add_subplot(1,2,1)
ax.plot(accepted[show:,0])
ax.set_title("Figure 13: Trace for $a$")
ax.set_xlabel("Iteration")
ax.set_ylabel("a")
ax = fig.add_subplot(1,2,2)
ax.hist(accepted[hist_show:,0], bins=20, density=True)
ax.set_ylabel("Frequency (normed)")
ax.set_xlabel("a")
ax.set_title("Figure 14: Histogram of $a$")
fig.tight_layout()

fig = plt.figure(figsize=(15,7))
ax = fig.add_subplot(1,2,1)
ax.plot(accepted[show:,1])
ax.set_title("Figure 15: Trace for $b$")
ax.set_xlabel("Iteration")
ax.set_ylabel("b")
ax = fig.add_subplot(1,2,2)
ax.hist(accepted[hist_show:,1], bins=20, density=True)
ax.set_ylabel("Frequency (normed)")
ax.set_xlabel("b")
ax.set_title("Figure 16: Histogram of $b$")
fig.tight_layout()

fig = plt.figure(figsize=(10,7))
ax = fig.add_subplot(1,1,1)
xbins, ybins = np.linspace(0.8,1.2,30), np.linspace(75,90,30)
counts, xedges, yedges, im = ax.hist2d(accepted[hist_show:,0], accepted[hist_show:,1], density=True, bins=[xbins, ybins])
ax.set_xlabel("a")
ax.set_ylabel("b")
fig.colorbar(im, ax=ax)
ax.set_title("2D histogram showing the joint distribution of $a$ and $b$")
```

## Prediction time

First, we average the last 50% of accepted samples of a and b, and we generate random individuals from a Γ distribution. $a_{average}$=0.9866200759935773 and $b_{average}$=83.70749712447888.

```python
show=-int(0.5*accepted.shape[0])

mu=accepted[show:,0].mean()
sigma=accepted[show:,1].mean()
print(mu, sigma)
model = lambda t,mu,sigma:np.random.gamma(mu,sigma,t)
t=np.arange(activity.shape[0])
observation_gen=model(t.shape[0],mu,sigma)
fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(1,1,1)

ax.hist( observation_gen,bins=np.linspace(0,500,50) ,density=True,label="Predicted values")
ax.hist( activity,bins=np.linspace(0,500,50) ,alpha=0.5,density=True, label="Original values")
ax.set_xlabel("Count")
ax.set_ylabel("Frequency")
ax.set_title("Figure 17: Posterior distribution of predicitons")
ax.legend()
```

## Evaluation
### Evaluation of the proposal distribution
How do we specify the parameters for the distribution Q? Should we move far from the current sample θ, or stay relatively close? These questions can be answered by measuring the auto-correlation between accepted samples: we do not want the accepted samples to be too correlated between one another. We don't want distant samples to be too correlated as we are trying to implement a markov chain, i.e. a sample should only depend on its previous sample, and the auto-correlation plot should show a quick, exponential decrease between the correlation of sample i and i-1,i-2,...i-n

The auto-correlation is defined by computing the following function for each lag $k$:
$r_k=\dfrac{\sum_{i=1}^{N-k}(Y_i-Y_{avg})(Y_{i+k}-Y_{avg})}{\sum_{i=1}^{N}(Y_i-Y_{avg})^2}$

The lag $k$, is basically the *range* around a sample $Y_i$ in which we would like to measure the correlation.

The plots below show the auto-correlation for a, b for k going from 1 to 100. A lag of k=1 means that we are measuring the correlation of a sample with itself, so we expect it to be equal to 1. The higher k goes, the lower that correlation ought to be.

```python
mean_acc_0=accepted[show:,0].mean()
mean_acc_1=accepted[show:,1].mean()
print(mean_acc_0,mean_acc_1)

lag=np.arange(1,100)
def autocorr(accepted,lag):
    num_0=0
    denom_0=0
    num_1=0
    denom_1=0
    for i in range(accepted.shape[0]-lag):
        num_0+=(accepted[i,0]-mean_acc_0)*(accepted[i+lag,0]-mean_acc_0)
        num_1+=(accepted[i,1]-mean_acc_1)*(accepted[i+lag,1]-mean_acc_1)
        denom_0+=(mean_acc_0-accepted[i,0])**2
        denom_1+=(mean_acc_1-accepted[i,1])**2
    rk_0=num_0/denom_0
    rk_1=num_1/denom_1
    return rk_0, rk_1

accepted_reversed=accepted[show:,:]
result=np.zeros((2,lag.shape[0]))
#print(lag)
for l in lag:
    result[:,l-1]=autocorr(accepted_reversed,l)

###Instead of writing an autocorrelation function, one could simply use thee autocorr function provided in pymc3
#from pymc3.stats import autocorr

fig, ax = plt.subplots()
#ax.plot(lag, [autocorr(accepted[show:,1], l) for l in lags], label='auto b')
#ax.plot(lag, [autocorr(accepted[show:,0], l) for l in lags], label='auto a')
ax.plot(lag, result[1,:], label='Auto correlation for b')
ax.plot(lag, result[0,:], label='Auto correlation for a')
ax.legend(loc=0)
ax.set_title("Figure 18: Auto-correlation for a and b, for k=1..100")
ax.set(xlabel='lag', ylabel='autocorrelation', ylim=(-.1, 1))
```

In our case, we are lucky to have a low enough correlation. In general, we might want to setup the parameters of the proposal distribution, Q, automatically, one common method is to keep adjusting the proposal parameters so that more than 50% proposals are rejected. Alternatively, one could use an enhanced version of MCMC called Hamiltonian Monte Carlo, which reduces the correlation between successive sampled states and reaches the stationary distribution quicker.

## Conclusion
While the abstraction behind this algorithm may seem out of grasp at first, the implementation is actually pretty simple, and gives awesome results. In fact, the great thing about probabilistic programming, notably MCMC is that you only need to write down the model and then run it. There is no need to compute evidence, or ensure some constraining mathematical properties.

## Resources
Peter Driscoll, "A comparison of least-squares and Bayesian fitting techniques to radial velocity data sets"

Carson Chow, "MCMC and fitting models to data"

John H. Williamson, "Data Fundamentals - Probabilities"

Simon Rogers, "A first course in machine learning"


```python
%matplotlib inline
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```python
from parsing import parser, digit
from plotting import plotter, voronoi
from analysis import training, sampling, testing, classify
from config import settings

import warnings
warnings.filterwarnings('ignore')
```

```python
n_observation_classes = 256
n_hidden_states = 30
n_iter = 10000
tol = 0.1
```

```python
parse = parser.Parser();
train_digits = parse.parse_file('data/pendigits-train');
test_digits = parse.parse_file('data/pendigits-test')
```

```python
pylab.rcParams['figure.figsize'] = (5, 5);
plotter.plot_digit(train_digits[6], True)
```

```python
pylab.rcParams['figure.figsize'] = (15, 6);
plotter.plot_digits_heatmap(train_digits, True);
```

```python
centroids = training.get_digit_kmeans_centroids(train_digits, n_observation_classes - 3)
```

```python
pylab.rcParams['figure.figsize'] = (10, 10);
voronoi.plot_centroids(centroids);
```

```python
training.set_digit_observations(train_digits, centroids, n_observation_classes)
```

```python
pylab.rcParams['figure.figsize'] = (5, 5);
plotter.plot_digit_observations(train_digits[11], centroids, n_observation_classes, True)
```

```python
hidden_markov_models = training.train_hmm(train_digits, n_observation_classes, n_hidden_states, n_iter, tol)
```

```python
samplings = sampling.get_samplings(hidden_markov_models, n_observation_classes, centroids, 100)
```

```python
pylab.rcParams['figure.figsize'] = (5, 5);
plotter.plot_digit(samplings[2][8], True)
```

```python
pylab.rcParams['figure.figsize'] = (15, 6);
plotter.plot_digit_samples_heatmap(samplings)
```

```python
test_labels_probabilities = classify.get_labels_probabilities(test_digits, hidden_markov_models, centroids, n_observation_classes, n_hidden_states,
                                                               n_iter, tol, True, True, "test_labels_probabilities.dat")
```

```python
test_classifier = classify.WeightedClassifier(test_digits, test_labels_probabilities)
prediction_matrix = test_classifier.get_prediction_matrix()
for row in prediction_matrix:
    print(row)
```

    [328, 0, 0, 0, 1, 0, 1, 0, 33, 0]
    [1, 282, 49, 1, 0, 1, 0, 5, 14, 11]
    [0, 22, 326, 0, 3, 0, 0, 9, 3, 1]
    [0, 5, 0, 312, 0, 4, 0, 2, 2, 11]
    [0, 5, 12, 0, 160, 0, 2, 0, 160, 24]
    [0, 0, 2, 0, 0, 73, 5, 0, 193, 62]
    [3, 3, 0, 0, 0, 0, 310, 0, 18, 2]
    [0, 55, 20, 23, 0, 0, 0, 24, 51, 191]
    [0, 1, 0, 0, 0, 9, 0, 0, 307, 19]
    [0, 10, 0, 0, 1, 6, 0, 1, 6, 312]

```python
test_classifier.print_classification_performance()
```

    total classification accuracy : 69.60251644266515
    label 1 : 90.35812672176309
    label 2 : 77.47252747252747
    label 3 : 89.56043956043956
    label 4 : 92.85714285714286
    label 5 : 44.07713498622589
    label 6 : 21.791044776119403
    label 7 : 92.26190476190476
    label 8 : 6.593406593406593
    label 9 : 91.36904761904762
    label 0 : 92.85714285714286

```python
gaussian_hidden_markov_models = training.train_gaussian_hmm(train_digits, n_hidden_states, n_iter, tol)
```

```python
for i in range(0, 10):
    print(classify.get_gaussian_hmm_probability(test_digits[0], gaussian_hidden_markov_models[i]))
```

    -85.3174667263
    -304.582351302
    -212.766422804
    -82.5324956152
    -28.4598359226
    -293.374894692
    -172.812791103
    9.64127185866
    -16.3436783895
    -142.691638808

```python
gaussian_test_labels_probabilities = classify.get_gaussian_labels_probabilities(test_digits, gaussian_hidden_markov_models, n_observation_classes, n_hidden_states,
                                                               n_iter, tol, True, True, "gaussian_test_labels_probabilities.dat")
```

```python
gaussian_test_classifier = classify.GaussianClassifier(test_digits, gaussian_test_labels_probabilities)
prediction_matrix = gaussian_test_classifier.get_prediction_matrix()
for row in prediction_matrix:
    print(row)
```

    [332, 0, 0, 0, 4, 0, 0, 0, 27, 0]
    [1, 306, 45, 0, 0, 0, 1, 5, 2, 4]
    [0, 12, 350, 0, 1, 0, 0, 1, 0, 0]
    [0, 3, 0, 324, 0, 0, 0, 3, 0, 6]
    [0, 2, 0, 0, 328, 2, 0, 0, 15, 16]
    [0, 2, 0, 0, 0, 315, 0, 0, 14, 4]
    [6, 0, 0, 0, 0, 0, 329, 0, 1, 0]
    [0, 38, 0, 0, 1, 0, 0, 323, 0, 2]
    [0, 0, 0, 0, 0, 2, 0, 0, 327, 7]
    [0, 8, 0, 0, 1, 0, 0, 2, 2, 323]

```python
gaussian_test_classifier.print_classification_performance()
```

    total classification accuracy : 93.13697454961395
    label 1 : 91.46005509641873
    label 2 : 84.06593406593407
    label 3 : 96.15384615384616
    label 4 : 96.42857142857143
    label 5 : 90.35812672176309
    label 6 : 94.02985074626865
    label 7 : 97.91666666666667
    label 8 : 88.73626373626374
    label 9 : 97.32142857142857
    label 0 : 96.13095238095238

# Dynamic Time Warping

\begin{equation}
y = \mathop{\arg\,\max}\limits{l \in L} \frac{1}{n_l} \sum_{i = 0}^{n_l} \text{fastdtw\_distance}(x_i^l, x_{\text{test}})
\end{equation}

```python
import dtw

print(dtw.score())
```

    0.898484415213


Last updated by: Jonathan Liu, 10/15/2020

In this notebook we will investigate the behavior of onset time distributions for the simple case of a Markov chain with irreversible transitions and identical transition rates. We will discover that a transiently increasing rate results in lower noise in the ensuing onset time than a steady-state rate.

First, consider a Markov chain with $k+1$ states and $k$ irreversible transitions rates, each with rate $\beta$. Labeling the first state with index $0$, the next with $1$, and so on, we have the reaction network:

\begin{equation}
0 \xrightarrow{\beta} 1 \xrightarrow{\beta} ... \xrightarrow{\beta} k
\end{equation}

We will be interested in the mean and variance of the distribution of times $P_k(t)$ to start at state $0$ and reach the final state $k$.

We will first consider the simple case where the transition rate $\beta$ is constant in time. In this case, the distribution $P_k(t)$ is simply given by a Gamma distribution with shape parameter $k$ and rate parameter $\beta$. $P_k(t)$ then has the form

\begin{equation}
P_k(t) = \frac{\beta^k}{\Gamma(k)}t^{k-1}e^{-\beta t}
\end{equation}

where $\Gamma$ is the Gamma function.

The mean $\mu_k$ and variance $\sigma^2_k$ have simple analytical expressions and are given by

\begin{equation}
\mu_k = \frac{k}{\beta} \\
\sigma^2_k = \frac{k}{\beta^2}
\end{equation}

Next, we can examine this system from the perspective of developmental biology. For example, let's consider the system to model the transition of chromatin from an inaccessible, silent state to an accessible, transcriptionally competent state. This could correspond to the time immediately following a nuclear division in the early fruit fly embryo. From a developmental perspective, it's important that $\mu_k$ and $\sigma^2_k$ can be tuned for optimal performance. Namely, given a target mean onset time $\mu$, we would like to minimize the resulting noise $\sigma^2$ in the distribution of onset times around that mean.

For ease of visualization, we will work with the squared CV $CV^2 = \frac{\sigma^2}{\mu^2}$, instead of the variance.

```python
#Import necessary packages
#matplotlib inline
import numpy as np
from scipy.spatial import ConvexHull
import matplotlib.pyplot as plt
import scipy.special as sps
```

```python
#Simulation for calculating onset times

def CalculatetOn_NEqualTransitions(time,dt,w,N_trans,N_cells):
#Calculates the onset time for a model with N irreversible transitions of
#equal magnitude. The transition rate can be time-varying, but is the same
#global rate for each transition. The model assumes N+1 states, beginning
#in the 0th state. Using finite timesteps and a Markov chain formalism, it
#simulates N realizations of the overall time it takes to reach the
#(N+1)th state. This is vectorized so it calculates it for all AP
#positions.

#Last updated by Jonathan Liu, 10/15/2020

# Inputs:
#   time: simulation time vector
#   dt: simulation timestep
#   w: transition probability vector at each timepoint
#   N_trans: number of irreversible transitions
#   N_cells: number of cells to simulate

# Outputs:
#   t_on: time to reach the final state for each cell (length = N_cells)

## Setup variables
    t_on = np.empty(N_cells) #Time to transition to final ON state for each cell
    t_on[:] = np.nan
    state = np.zeros(N_cells) #State vector describing current state of each cell
    finished_states = np.zeros(N_cells) #Vector storing finished statuses of each cell

    ## Run simulation
    #Loop over time
    #q = waitbar(0,'Running simulation...')
    for i in range(len(time)):
        if np.sum(finished_states) == N_cells: #If all cells have turned on, stop the simulation
            #print('Halting simulation since all cells have turned on.')
            break
        #Simulate binomial random variable to see if each cell has transitioned
        #If the input transition rate is a nan, this will manifest as never
        #transitioning.
        p = w[i] * dt #Probability of transition at this timestep
        transitioned = np.random.binomial(1,p,N_cells) #Binary transition decision for each cell

        #Advance the cells that did transition to the next state
        states_to_advance = transitioned == 1
        state[transitioned == 1] = state[transitioned == 1] + 1

        #See if any states have reached the ON state
        t_on[state == N_trans] = time[i]
        finished_states[state == N_trans] = 1
        state[state == N_trans] = np.nan #Move finished states out of consideration

        #waitbar(i/length(time),q,'Running simulation...')
    return t_on
```

The ensuing code calculates the mean and squared CV for varying values of the step number $k$ and the transition rate $\beta$. The final results can be easily visualized in a 2D parameter space, with the mean $mu_k$ as the x axis and the squared CV $\CV^2_k$ as the y axis. We see that the squared CV is constant with regards to the mean. Of particular interest is the boundary at the bottom of the parameter space. For a given mean and upper limit to $k$ and $\beta$, there is a minimum squared CV below which is unachievable.

```python
#Plot the mean and variance of the Gamma distribution in 2D parameter space, for a given set of parameters
#Function returning the mean and variance of a Gamma distribution
def MeanVarGamDist(shape,rate):
    return shape/rate, shape/rate**2

#Let's create a grid of shape and rate parameters
n_steps = np.arange(1,5)
rate = np.arange(0.5,5,0.1)
means_const = np.zeros((len(n_steps),len(rate)))
variances_const = np.zeros((len(n_steps),len(rate)))

for i in range(len(n_steps)):
    for j in range(len(rate)):
        means_const[i,j], variances_const[i,j] = MeanVarGamDist(n_steps[i],rate[j])

CV2_const = variances_const / means_const**2

plt.figure()
plt.plot(means_const,CV2_const, 'b.')
plt.xlim(0,4)
plt.xlabel('mean')
plt.ylabel('CV^2')

```

Next, we will investigate the changes to this parameter space by using a transient rate $\beta(t)$. This is of biological interest because many developmental processes occur out of steady state. For example, several models of chromatin accessibility hypothesize that the rate of chromatin state transitioning is coupled to the activity of pioneer factors like Zelda. During each rapid cell cycle division event in the early fly embryo, the nuclear membrane breaks down and reforms again, and transcription factors are expelled and re-introduced back into the nucleus. Thus, after each division event, there is a transient period during which the concentration of pioneer factors at a given gene locus is out of steady state.

For now, we will assume a reasonable form for the transition rate. Considering $\beta$ to be a proxy for Zelda concentration, for example, we will write down this transient $\beta(t)$ as the result of a simple diffusive process with form

\begin{equation}
\beta(t) = \beta (1 - e^{-t / \tau} )
\end{equation}

Here, $\beta$ is the asymptotic, saturating value of $\beta(t)$, and $\tau$ is the time constant governing the time-varying nature of the transition rate. For a diffusive process, $\tau$ would be highly dependent on the diffusion constant, for example.

For comparison, the time plots of the constant and transient input are shown below, for $\tau = 3$ and $\beta = 1$.

```python
time = np.arange(0,10,0.1)
dt = 0.1
w_base = 1
w_const = w_base * np.ones(time.shape)
N_trans = 2
N_cells = 1000

#Now with transient exponential rate
tau = 3
w_trans = w_base * (1 - np.exp(-time / tau))

#Plot the inputs
plt.figure()
plt.plot(time,w_const,label='constant')
plt.plot(time,w_trans,label='transient')
plt.xlabel('time')
plt.ylabel('rate')
plt.legend()
```

Because of the time-varying nature of $\beta(t)$, the resulting distribution $P_k(t)$ no longer obeys a simple Gamma distribution, and an analytical solution is difficult (or even impossible). Nevertheless, we can easily simulate the distribution and calculate $\mu_k$ and $\sigma^2_k$ numerically. The results are shown below in red, compared to the steady state case in blue.

```python
#Now using the transient simulation
time = np.arange(0,10,0.1)
dt = 0.1
N_trans = 2
N_cells = 1000
tau = 3

means_trans = np.zeros((len(n_steps),len(rate)))
CV2_trans = np.zeros((len(n_steps),len(rate)))
for i in range(len(n_steps)):
    for j in range(len(rate)):
        w_trans = rate[j] * (1 - np.exp(-time / tau))
        t_on_trans = CalculatetOn_NEqualTransitions(time,dt,w_trans,n_steps[i],N_cells)
        means_trans[i,j] = np.mean(t_on_trans)
        CV2_trans[i,j] = np.var(t_on_trans)/np.mean(t_on_trans)**2

plt.figure()
plt.plot(means_const,CV2_const, 'b.', label='steady state')
plt.plot(means_trans,CV2_trans, 'r.', label='transient')

plt.xlim(0,4)
plt.xlabel('mean')
plt.ylabel('CV^2')

handles, labels = plt.gca().get_legend_handles_labels()
by_label = dict(zip(labels, handles))
plt.legend(by_label.values(), by_label.keys())
```

We see that the consequence of a slowly increasing transient transition rate is to move the parameter space to the right and down. That is, the transient transition rate serves to increase the mean and decrease the squared CV. This is interesting if we consider, for example, a vertical slice of constant mean. For example, consider the line $\mu = 2$. Here, for a given upper bound on the number of state $k=4$, the steady-state model can only achieve a minimum squared CV of about $0.25$. In contrast, the transient model with a time constant of $\tau = 3$ can achieve a lower minimum squared CV of around $0.15$!

This implies that having a transient transition rate can actually produce a benefit of lowering the noise in onset times for a given mean onset time. While the transient nature of pioneer factor concentrations is an inevitable result of rapid cell divisions, this suggests that the transience can actually be harnessed to decrease overall noise in timing.

To see this more explicitly, let's simulate results for fixed number of steps $k = 3$, for varying values of the transition rate $\beta$. We're also going to change the time constant $\tau$ of the transient input.

```python
#Model parameters
n_steps = 3
rate = np.arange(0.5,5,0.1)
tau = np.array([0.0001,0.5,1,2,3,4])
means = np.zeros((len(tau),len(rate)))
variances = np.zeros((len(tau),len(rate)))

#Simulation parameters
time = np.arange(0,10,0.1)
dt = 0.1
N_trans = 2
N_cells = 1000

for i in range(len(tau)):
    for j in range(len(rate)):
        w_trans = rate[j] * (1 - np.exp(-time / tau[i]))
        t_on_trans = CalculatetOn_NEqualTransitions(time,dt,w_trans,n_steps,N_cells)
        means[i,j] = np.mean(t_on_trans)
        variances[i,j] = np.var(t_on_trans)

CV2 = variances / means**2

#Plot results
plt.figure()
for i in range(len(tau)):
    plt.plot(means[i,:],CV2[i,:],label='tau = ' + str(tau[i]))

plt.legend()
plt.xlabel('mean')
plt.ylabel('CV^2')
plt.title('n = ' + str(n_steps) + ' steps, varying rate from 0.5 to 5')
```

The net effect of increasing the time constant, i.e. making the transient input even slower to ramp up, appears to push the curve of squared CV2/mean further to the right and down. However, it appears to asymptote for a slow enough input. Here, the fastest rate used was $\beta = 5$, giving a time scale of $0.2$. It looks like around $\tau = 2$ the curves asymptote, so the rough rule of thumb here is that when the time constant $\tau$ is an order of magnitude slower than the time scale of the transition rates $\beta$, any further slowdown is negligible.

Nevertheless, if we consider a vertical slice of constant mean, we see that transience can decrease the variance by up to a factor of about four!

```python

```


This demo is outdated, please use `~/src/tests/test_workflow_COVID19_SEIRD.py` as an example use of the COVID19_SEIQRD model.

# Covid-19: From model prediction to model predictive control

## A demo of the deterministic, national-level, age-stratified modeling framework

*Code by T.W. Alleman and Jenna Vergynst in consultation with the BIOMATH research unit headed by prof. Ingmar Nopens and prof. Jan baetens of the KERMIT research unit.*

Copyright (c) 2021 by T.W. Alleman, BIOMATH, Ghent University. All Rights Reserved.

In `src/covid19model/models.py`, two national-level extended SEIRD compartmental models are coded,
1. COVID19_SEIRD: a deterministic model, using ordinary differential equations (ODEs). The equations are integrated using Scipy's `odeint` with a variable timestep, it is a 'continuous-time' model.
2. COVID19_SEIRD_sto: a stochastic model, using stochastic difference equations (SDEs). The equations are iterated using a fixed timestep of one day and is said to be a 'discrete-time' model. The model is not used anymore because it is slower than the deterministic model variant a

A third model, a spatially-explicit variant of COVID19_SEIRD is currently being implemented by Michiel Rollier, a demo will also be provided.

The deterministic model was used for all results detailed in our first COVID-19 related preprint: *A deterministic, age-stratified, extended SEIRD model for assessing the effect of non-pharmaceutical interventions on SARS-CoV-2 spread in Belgium*, which can be found here: https://www.medrxiv.org/content/10.1101/2020.07.17.20156034v2

The documentation given in this tutorial has been redrafted from the above preprint. References have been omitted for the sake of simplicity.

```python
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import pandas as pd
import datetime
import scipy
import json
import random
```

Load the covid 19 custom development code

```python
from covid19model.optimization import objective_fcns
from covid19model.models import models
from covid19model.models.utils import draw_sample_COVID19_SEIRD
from covid19model.data import mobility, sciensano, model_parameters
from covid19model.visualization.output import population_status, infected
from covid19model.visualization.optimization import plot_fit, traceplot
```

```python
# OPTIONAL: Load the "autoreload" extension so that package code can change
%load_ext autoreload
# OPTIONAL: always reload modules so that as you change code in src, it gets loaded
%autoreload 2
```

## Introduction

### Model dynamics

The SEIR(D) model was first proposed in 1929 by two Scottish scientists. It is a compartmental model that subdivides the human population into four groups: 1) susceptible individuals, 2) exposed individuals in the latent phase 1, 3) infectious individuals capable of transmitting the disease and 4) individuals removed from the population either through immunization or death. Despite being a simple and idealized reality, the SEIR(D) dynamics are used extensively to predict the outbreak of infectious diseases and this was no different during the SARS-CoV-2 outbreak earlier this year.

In this work, we extend the SEIRD model to incorporate more expert knowledge on SARS-CoV-2. For that purpose, the infectious compartment is split into four parts. The first is a period of presymptomatic infectiousness because several studies have shown that pre-symptomatic transmission is a dominant transmission mechanism of SARS-CoV-2. After the period of pre-symptomatic transmission, three possible infectious outcomes are modeled: (1) Asymptomatic outcome, for individuals who show no symptoms at all, (2) Mild outcome, for individuals with mild symptoms who recover at home, and (3) Hospitalization, when mild symptoms worsen. Children and young adults have a high propensity to experience an asymptomatic or mild outcome, while older individual have a high propensity to be hospitalized.

In general, Belgian hospitals have two wards for COVID-19 patients: 1) Cohort, where patients are not monitored permanently and 2) Intensive care, for patients with the most severe symptoms. Intensive care includes permanent monitoring, the use of ventilators or the use of extracorporeal membrane oxygenation (ECMO). Patients generally spend limited time in the emergency room and/or in a buffer ward before going to Cohort. After spending limited time in Cohort, some patients are transferred to ICU. Patients can perish in both wards, but mortalities are generally lower in Cohort. After a stay in an ICU, patients return to Cohort for recovery in the hospital. During the recovery stay, mortality is limited. The above is a short summary of hospital dynamics based on interviewing Ghent University hospital staff and examining the hospital data.

### Model framework and equations

#### Age-stratification

We introduced heterogeneity in the deterministic implementation by means of age-stratification. Every population compartment is split into a number of age classes, the age-groups have different contact rates with other age-groups and the disease progresses differently for each age-group, making the model behaviour more realistic. Our age-stratified model consists of 9 age classes, i.e., [0-10[, [10-20[, [20-30[, [30-40[, [40-50[, [50-60[, [60-70[, [70-80[, [80- $\infty$[. The age-stratified implementation provides a good balance between added complexity and computational resources.

#### Deterministic framework

Our extended SEIRD model is implemented using two frameworks: a deterministic and a stochastic framework. The deterministic equations are obtained by writing down the following equation,

`rate of change = in - out`,

for every of the 11 population compartments. This results in the following system of coupled ordinary differential equations,
\begin{eqnarray}
\dot{S_i} &=& - \beta s_i S_i \sum_{j=1}^{N} N_{c,ij} \Big( \frac{I_j+A_j}{T_j} \Big) + \zeta R_i, \label{eq:system1} \\
\dot{E_i} &=& \beta s_i S_i \sum_{j=1}^{N} N_{c,ij} \Big( \frac{I_j+A_j}{T_j} \Big) - (1/\sigma) \cdot E_i, \label{eq:system2} \\
\dot{I_i} &=& (1/\sigma) E_i - (1/\omega) I_i, \label{eq:system3}\\
\dot{A_i} &=& (\text{a}_i/\omega) I_i - (1/d_{\text{a}}) A_i, \label{eq:system4}\\
\dot{M_i} &=&  ((1-\text{a}_i) / \omega ) I_i - ( (1-h_i)/d_m + h_i/d_{\text{hospital}} ) M_i,\label{eq:system5} \\
\dot{ER_i} &=& (h_i/d_{\text{hospital}}) M_i - (1/d_{\text{ER}}) ER_i, \label{eq:system6} \\
\dot{C_i} &=& c_i (1/d_{\text{ER}}) ER_i  - (m_{C, i}/d_{c,D}) C_i - ((1 - m_{C, i})/d_{c,R}) C_i, \label{eq:system7} \\
\dot{ICU_i} &=& (1-c_i) (1/d_{\text{ER}}) ER_i - (m_{ICU,i}/d_{\text{ICU},D}) ICU_i  \label{eq:system8} \\
&& - ((1-m_{ICU,i})/d_{\text{ICU},R}) ICU_i, \label{eq:system9}\\
\dot{C}_{\text{ICU,rec,i}} &=& ((1-m_{ICU,i})/d_{\text{ICU},R}) ICU_i - (1/d_{\text{ICU,rec}}) C_{\text{ICU,rec,i}}, \label{eq:system10}\\
\dot{D_i} &=&  (m_{ICU,i}/d_{\text{ICU},D}) ICU_i +  (m_{C,i}/d_{\text{c},D}) C_i , \label{eq:system11}\\
\dot{R_i} &=&  (1/d_a) A_i + ((1-h_i)/d_m) M_i + ((1-m_{C,i})/d_{c,R}) C_i \\
&& + (1/d_{\text{ICU,rec}}) C_{\text{ICU,rec,i}} - \zeta R_i \label{eq:system13},
\end{eqnarray}

for $i = 1,2,...,9$. Here, $T_i$ stands for total population, $S_i$ stands for susceptible, $E_i$ for exposed, $I_i$ for pre-symptomatic and infectious, $A_i$ for asymptomatic and infectious, $M_i$ for mildly symptomatic and infectious, $ER_i$ for emergency room and/or buffer ward, $C_i$ for cohort, $C_{\text{ICU,rec,i}}$ for a recovery stay in Cohort coming from Intensive Care, $ICU_i$ for Intensive Care Unit, $D_i$ for dead and $R_i$ for recovered. Using the above notation, all model states are 9-dimensional vectors,
$$ \mathbf{S} = [S_1(t)\ S_2(t)\ ...\ S_i(t)]^T, $$
where $S_i(t)$ denotes the number of susceptibles in age-class i at time t after the introduction of the virus in the population.

These equations are implemented in the function `COVID19_SEIRD` located in `src/covid19model/models.py`. The integration is performed in `_sim_single` located in `src/covid19model/base.py` by using Scipy's `solve_ivp`. The integrator timestep depends on the rate of change of the system and the solver method is thus referred to as a 'continuous-time' solver. The implementation uses non-integer individuals.

#### Stochastic framework

By defining the probabilities of transitioning (propensities) from one state to another, a system of coupled stochastic difference equations (SDEs) can be obtained. The probability to transition from one state to another is assumed to be exponentially distributed. As an example, consider the average time a patient spends in an ICU when recovering, which is $d_{\text{ICU,R}} = 9.9$ days. The chances of survival in ICU are $(1-m_{\text{ICU,i}})$, where $m_{\text{ICU,i}}$ is the mortality in ICU for an individual in age group $i$. If the probability of transitioning from state ICU to state $C_{\text{ICU,rec}}$ on any given day is assumed to be exponentially distributed, then for an individual in age group $i$,

\begin{equation}
P(ICU_i \rightarrow C_{\text{ICU,rec,i}}) = 1 - \text{exp} \Bigg[ - \frac{1-m_{\text{ICU},i}}{d_{\text{ICU,R}}}\Bigg].
\end{equation}

If a transitioning between states is defined as "succes", we can regard the number of individuals transitioning from ICU to a Cohort recovery ward as a binomial experiment. On a given day, the number of individuals transitioning is,

\begin{equation}
(\text{ICU}_i \rightarrow C_{\text{ICU,rec,i}})(k) \sim \text{Binomial}\Bigg(\text{ICU}_i(k),\ 1 - \text{exp}\Bigg[- \frac{1-m_{\text{ICU,i}}}{d_{\text{ICU,R}}}\Bigg]\Bigg).
\end{equation}

For a discrete stepsize $l$, there are 15 possible transitions,
\begin{eqnarray}
(S_i \rightarrow E_i) (k) &\sim& \text{Binomial}\Bigg(S_i(k), 1 - \text{exp}\Bigg[- l \beta s_i \sum_{j=1}^{N} N_{c,ij} \Big( \frac{I_j+A_j}{T_j} \Big) \Bigg]\Bigg)\\
(E_i \rightarrow I_i) (k) &\sim& \text{Binomial}\Bigg(E_i(k), 1 - \text{exp}\Bigg[- l\ \frac{1}{\sigma}\Bigg]\Bigg)\\
(I_i \rightarrow A_i) (k) &\sim& \text{Binomial}\Bigg(I_i(k), 1 - \text{exp}\Bigg[- l\ \frac{a_i}{\omega}\Bigg]\Bigg)\\
(I_i \rightarrow M_i) (k) &\sim& \text{Binomial}\Bigg(I_i(k), 1 - \text{exp}\Bigg[- l\ \frac{1-a_i}{\omega}\Bigg]\Bigg)\\
(A_i \rightarrow R_i) (k) &\sim& \text{Binomial}\Bigg(A_i(k), 1 - \text{exp}\Bigg[- l\ \frac{1}{d_a}\Bigg]\Bigg)\\
(M_i \rightarrow R_i) (k) &\sim& \text{Binomial}\Bigg(M_i(k), 1 - \text{exp}\Bigg[- l\ \frac{1-h_i}{d_m}\Bigg]\Bigg)\\
(M_i \rightarrow ER_i) (k) &\sim& \text{Binomial}\Bigg(M_i(k), 1 - \text{exp}\Bigg[- l\ \frac{h_i}{d_{\text{hospital}}}\Bigg]\Bigg)\\
(ER_i \rightarrow C_i) (k) &\sim& \text{Binomial}\Bigg(ER_i(k), 1 - \text{exp}\Bigg[- l\ \frac{c_i}{d_{\text{ER}}}\Bigg]\Bigg)\\
(ER_i \rightarrow ICU_i) (k) &\sim& \text{Binomial}\Bigg(ER_i(k), 1 - \text{exp}\Bigg[- l\ \frac{1-c_i}{d_{\text{ER}}}\Bigg]\Bigg)\\
(C_i \rightarrow R_i) (k) &\sim& \text{Binomial}\Bigg(C_i(k), 1 - \text{exp}\Bigg[- l\ \frac{1-m_{C,i}}{d_{C,R}}\Bigg]\Bigg)\\
(ICU_i \rightarrow C_{\text{ICU,rec,i}}) (k) &\sim& \text{Binomial}\Bigg(ICU_i(k), 1 - \text{exp}\Bigg[- l\ \frac{1-m_{\text{ICU,i}}}{d_{ICU,R}}\Bigg]\Bigg)\\
(C_{\text{ICU,rec,i}} \rightarrow R_i) (k) &\sim& \text{Binomial}\Bigg(C_{\text{ICU,rec,i}}(k), 1 - \text{exp}\Bigg[- l\ \frac{1}{d_{\text{ICU,rec}}}\Bigg]\Bigg)\\
(C_i \rightarrow D_i) (k) &\sim& \text{Binomial}\Bigg(C_i(k), 1 - \text{exp}\Bigg[- l\ \frac{m_{C,i}}{d_{C,D}}\Bigg]\Bigg)\\
(ICU_i \rightarrow D_i) (k) &\sim& \text{Binomial}\Bigg(ICU_i(k), 1 - \text{exp}\Bigg[- l\ \frac{m_{\text{ICU,i}}}{d_{\text{ICU,D}}}\Bigg]\Bigg)\\
(R_i \rightarrow S_i) (k) &\sim& \text{Binomial}\Bigg(R_i(k), 1 - \text{exp}\Bigg[- l\ \zeta \Bigg]\Bigg)\\
\end{eqnarray}

And the system of equations becomes,

\begin{eqnarray}
S_i(k+1) &=& S_i(k) + (R_i \rightarrow S_i) (k) - (S_i \rightarrow E_i) (k) \\
E_i(k+1) &=& E_i(k) + (S_i \rightarrow E_i) (k) - (E_i \rightarrow I_i) (k) \\
I_i(k+1) &=& I_i(k) + (E_i \rightarrow I_i) (k) - (I_i \rightarrow A_i) - (I_i \rightarrow M_i) (k) \\
A_i(k+1) &=& A_i(k) + (I_i \rightarrow A_i) (k) - (A_i \rightarrow R_i) (k) \\
M_i(k+1) &=& M_i(k) + (I_i \rightarrow M_i) (k) - (M_i \rightarrow R_i) (k) - (M_i \rightarrow ER_i) (k) \\
ER_i(k+1) &=& ER_i(k) + (M_i \rightarrow ER_i) (k) - (ER_i \rightarrow C_i) (k) - (ER_i \rightarrow ICU_i) (k) \\
C_i(k+1) &=& C_i(k) + (ER_i \rightarrow C_i) (k) - (C_i \rightarrow R_i) (k) - (C_i \rightarrow D_i) (k) \\
C_{\text{ICU,rec,i}}(k+1) &=& C_{\text{ICU,rec,i}}(k)  + (ICU_i \rightarrow C_{\text{ICU,rec,i}}) (k) - (C_{\text{ICU,rec,i}} \rightarrow R_i) (k) \\
R_i(k+1) &=& R_i(k) + (A_i \rightarrow R_i) (k)  + (M_i \rightarrow R_i) (k) + (C_i \rightarrow R_i) (k) + (C_{\text{ICU,rec,i}} \rightarrow R_i) (k)  - (R_i \rightarrow S_i) (k) \\
D_i(k+1) &=& D_i(k) + (ICU_i \rightarrow D_i) (k) + (C_i \rightarrow D_i) (k) \\
\end{eqnarray}

These equations are implemented in the function `COVID19_SEIRD_sto` located in `src/covid19model/models.py`. The computation itself is performed in the function `solve_discrete` located in `src/covid19model/base.py`. Please note that the deterministic model uses **differentials** in the model defenition and must be integrated, while the stochastic model uses **differences** and must be iterated. The discrete timestep is fixed at one day (for now). The stochastic implementation only uses integer individuals, which is considered an advantage over the deterministic implementation.

### Transmission rate and social contact data

In our model, the transmission rate of the disease depends on the product of four contributions. The first contribution, $(I+A)/T$, is the fraction of contagious individuals in the population. The second contribution, $\mathbf{N}_c$, is the average number of human-to-human interactions per day. The third contribution, $s_i$, is the relative susceptiblity to SARS-CoV-2 infection in age group $i$, and the fourth contribution, $\beta$, is the probability of contracting COVID-19 when encountering a contagious individual under the assumption of 100 \% susceptibility to SARS-CoV-2 infection. We assume that the per contact transmission probability $\beta$ is independent of age and we will infer its distribution by calibrating the model to national Belgian hospitalization data. The number of human-human interactions, $\mathbf{N}_c$, are both place and age-dependent. These matrices assume the form of a 9x9 *interaction matrix* where an entry X, Y denotes the number of social contacts age group X has with age group Y per day. These matrices are available for homes, schools, workplaces, in public transport, and leisure activities, from a survey study by Lander Willem (2012). The total number of social interactions is given by the sum of the contributions in different places,

\begin{equation}\label{eq:interaction_matrices}
\mathbf{N_{\text{c}}} = \mathbf{N_{\text{c, home}}} + \mathbf{N_{\text{c, schools}}} + \mathbf{N_{\text{c, work}}} + \mathbf{N_{\text{c, transport}}} + \mathbf{N_{\text{c, leisure}}} + \mathbf{N_{\text{c, others}}}
\end{equation}

The function `get_interaction_matrices` located in `src/covid19model/data/model_parameters.py` can be used to load the interaction matrices and the demographic data for Belgium. The function can be used to extract the pre-pandemic interaction matrices, which are available for homes, workplaces, schools, public transport, leisure activities. The function can also be used to extract contact matrices under lockdown measures (CoMiX study). The function has an optional argument `intensity` to alter the nature/duration of the contacts included in the interaction matrix. This is necessary because a contact was defined by Willem as any two-way conversation longer than 3 sentences. However, an infectious disease may only spread upon more 'intense' contact, hence the need to exclude or include contacts based on duration and intensity. Valid intensities include 'all' (default), 'physical_only', 'less_5_min', less_15_min', 'more_one_hour', 'more_four_hours'. For further details, I refer to the docstring documentation of the function.

https://lwillem.shinyapps.io/socrates_rshiny/

#### CoMiX

```python
initN, Nc_comix,dates = model_parameters.get_interaction_matrices(dataset='comix', intensity = 'physical_only', wave=4)
```

#### Willem 2012

```python
# Load the interaction matrices (size: 9x9)
initN, Nc_home, Nc_work, Nc_schools, Nc_transport, Nc_leisure, Nc_others, Nc_total = model_parameters.get_interaction_matrices(dataset='willem_2012')
# Define the number of age categories
levels = initN.size
```

The coefficients of the contact matrices in different places can be altered to model a goverment policy. For instance, to model the Belgian lockdown, the mobility reductions deduced from the Google community mobility reports were used as coefficients for the different interaction matrices. We assumed workplace interactions were down to only 40 % of their prepandemic values before the lockdown.

https://www.google.com/covid19/mobility/

Our code has built-in functionality to download an visualize these mobility reports. To do so, use the following code:

```python
df_google = mobility.get_google_mobility_data(update=False, plot=True)
```

Alternatively, data from Apple can be used using the following command,

```python
df_apple = mobility.get_apple_mobility_data(update=False)
```

```python
fig,ax = plt.subplots(figsize=(10,3))
ax.plot(df_apple.loc[('Belgium','transit')])
ax.plot(df_google['transport'])
ax.legend(['Apple','Google'])
ax.set_title('Public transport mobility')
```

### Social interatia and time-dependent parameters

#### Social inertia

The model takes into account the effect of *social inertia* when measures are taken. In reality, social restrictions or relaxations represent a change in behaviour which is gradual and cannot be modeled using a step-wise change of the social interaction matrix $\mathbf{N_c}$. This can be seen when closely inspecting the *Google community mobility report* above. Multiple functions can be used to model the effects of social compliance, e.g. a delayed or non-delayed ramp, or a logistic function. In our model, we use a delayed ramp to model compliance,

\begin{equation}
\mathbf{N_{c}}^{k} = \mathbf{N_{\text{c, old}}} + f^{k} (\mathbf{N_{\text{c, new}}} - \mathbf{N_{\text{c, old}}})
\end{equation}

where,
\begin{equation}
    f^k=
\begin{cases}
	0.0,& \text{if } k\leq \tau\\
    \frac{k}{l} - \frac{\tau}{l},& \text{if } \tau < k\leq \tau + l\\
    1.0,              & \text{otherwise}
\end{cases}
\end{equation}

where $\tau$ is the number of days before measures start having an effect and $l$ is the number of additional days after the time delay until full compliance is reached. Both parameters were calibrated to the daily number of hospitalizations in Belgium (notebooks `notebooks/0.1-twallema-calibration-deterministic.ipynb` and `notebooks/0.1-twallema-calibration-stochastic.ipynb`). $k$ denotes the number of days since a change in social policy.

#### The use of time-dependent parameters

Any model parameter can be altered during the course of a simulation by defining a time dependent parameter function. The function below is applied to the interaction matrix $\mathbf{N}_C$. On the time `policy_time`, the interaction matrix is altered from a value `policy1` to `policy2` according to the delayed ramp compliance model with parameters `l` and `tau`. The prevention parameter ($\Omega$) can be used to 'downgrade' the weight of the social contacts under lockdown measures.

```python
from covid19model.models.time_dependant_parameter_fncs import lockdown_func
from covid19model.models.utils import draw_sample_COVID19_SEIRD
```

### Basic reproduction number

The basic reproduction number $R_0$, defined as the expected number of secondary cases directly generated by one case in a population where all individuals are susceptible to infection, is computed using the next generation matrix (NGM) approach introducted by Diekmann. For our model, the basic reproduction number of age group $i$ is,
\begin{equation}\label{eq:reproduction_number}
R_{0,i} = (a_i d_a + \omega) \beta s_i \sum_{j=1}^{N} N_{c,ij}
\end{equation}
and the population basic reproduction number is calculated as the weighted average over all age groups using the demographics of Belgium. The detailed algebra underlying the computation equation of the basic reproduction number is presented in the supplementary materials of our manuscript (https://www.medrxiv.org/content/10.1101/2020.07.17.20156034v2).

### Model parameters

## Simulations with the deterministic model

### Model initialization

```python
# Load the parameters using `get_COVID19_SEIRD_parameters()`.
params = model_parameters.get_COVID19_SEIRD_parameters()
params.update({'policy0': Nc_total,
              'policy1': 1.0*Nc_home + (1-0.60)*Nc_work + (1-0.70)*Nc_transport + (1-0.30)*Nc_others + (1-0.80)*Nc_leisure,
               'l': 1,
              'tau': 5,
              'prevention': 0.5,
              'start_date': pd.to_datetime('2020-03-15')})
# Define the initial condition: one exposed inidividual in every age category
initial_states = {'S': initN, 'E': np.ones(levels)}
model = models.COVID19_SEIRD(initial_states, params, time_dependent_parameters={'Nc': lockdown_func})
```

### Load posterior parameter distributions of calibrated parameters

Both the deterministic and stochastic model were calibrated to the national hospitalization data provided by the Scientific Institute of Public Health (Sciensano).

The parameters $\beta$, $l$, $\tau$ and $\Omega$ were calibrated.

The aim of such a calibration is to obtain a parameter set which leads to a good agreement between the model predictions and the experimental data. Five timeseries starting on Sunday, March 15th, 2020 have been made publically available by the Belgian Scientific Institute of Public Health (Sciensano),

- the total number of patients in the hospitals ($H_{\text{tot}}$)
- the total number of patients in ICU ($ICU_{\text{tot}}$)
- the daily hospitalizations ($H_{\text{in}}$)
- the daily hospital discharges ($H_{\text{out}}$)
- the daily number of deaths ($D_{\text{new}}$)

We assume the timeseries are independent and identically distributed (i.d.d.) sequences of normal random variables having a mean $\hat{y}(k)$ and an unknown variance $\hat{\sigma}^2$. The resulting log-likelihood function is,

\begin{equation}
\ln \big( P( y \mid \mathbf{\hat{\theta}}) \big) = - \frac{1}{2} \sum_{k=1}^{N} \Bigg[ \frac{\big(y(k) - \hat{y}(k, \mathbf{\hat{\theta}})\big)^2}{\hat{\sigma}^2} + \ln\big(2 \pi \hat{\sigma}^2\big) \Bigg]\ ,
\end{equation}

where $\mathbf{\hat{\theta}}$ and $\hat{\sigma}^2$ are computed to maximize the log-likelihood function. The fitting procedure is performed in two steps. First, the transmission parameter $\beta$ is calibrated using the daily hospitalizations from March 15th, 2020 until March 22nd, 2020. These data are used because they do not depend on the durations of the hospitalizations, $d_c$ and $d_{\text{ICU}}$, which are computed from the limited and imperfect hospital dataset. In addition to the calibration of the transmission parameter $\beta$, the number of days between the first infectious individuals entering the country and the onset of data collection (March 15th, 2020) must be estimated. The latter is denoted as excess time, $t_e$. Second, the daily hospitalizations ($H_{\text{in}}$) from March 15th, 2020 until April 20th, 2020 were used to infer the slope of the compliance ramp ($l$), as well as the effect of prevention measures and *network* effects in the workplace and on public transport ($\Omega$).

We performed the above optimizations in two steps. First, we used a particle swarm optimization (PSO) algorithm to scan the parameter space and find the global maximum likelihood. We then used the obtained maximum-likelihood estimates as initial values for the ensemble sampler for Markov Chain Monte Carlo (MCMC) proposed by Goodman 2010. For all parameters, uniform prior distributions were used. The sampler was always run on $2n$ parallel chains, were $n$ was the number of parameters to be sampled from the posterior distributions. For each chain, we drew at least 50-fold more samples than the autocorrelation time. Next, we visualised the samples using the scatterplot matrix proposed by Foreman (2016), further referred to as *cornerplot*. In these visualizations, each one- and two-dimensional projection of the sample is plotted to reveal correlations. During each step of the optimization procedure, previously sampled parameters were propagated.

The calibration can be found in the notebook `notebooks/0.1-twallema-calibration-deterministic.ipynb`. The posterior samples of the calibrated parameters can be found in `data/interim/model_parameters/deterministic_22072020.json`.

```python
# Scrape the publically available data from the Scientific institute of Public Health (Sciensano)
df_sciensano = sciensano.get_sciensano_COVID19_data(update=False)
df_sciensano.tail()
```

```python
# Load previous samples
with open('../../data/interim/model_parameters/COVID19_SEIRD/calibrations/national/classical/BE_2020-11-05.json', 'r') as fp:
    samples_dict = json.load(fp)
```

### Simulating

A simulation is run by using the attribute function *sim*, which uses one argument, the simulation time, as its input.

```
out = model.sim(100)
```

The snippet above would run a 100 day simulation with the initialized model. However, to make predictions on how COVID-19 will spread it is more convenient to have the simulation time as dates, rather than integer numbers. The snippet above will NOT work in conjuction with the time-varying policy implemented in `lockdown_func`. To perform a simulation using datetimes, the end date of the simulation is given as the first argument of the `sim` function. Next, the start date of data collection must be defined, for Belgium this is March 15th, 2020.

As an initial condition, we use one exposed individual in every age group. This implies the simulation must be run for some time between its start, and the start of data collection on March 15th, 2020. In our code, this concept is named `warmup` and was calibrated to high-level Belgian hospitalization data.

```python
out=model.sim('2020-09-01',start_date='2020-03-15',warmup=samples_dict['warmup'])
```

#### Extracting model output

The output format of the deterministic and stochastic model are an `xarray` dataset. This flexible python package introduces labels in the form of dimensions, coordinates and attributes on top of raw NumPy-like arrays. Relevant to our work are the coordinates,
1. 'stratification': the nine different age groups used in the model.
2. 'time': simulation time

and the data variables, which are the 11 model states (S,E,I,...,R,D). Future work will include a spatial extension, which will simply use a new coordinate 'NIS'.

```python
out
```

As an example of output handling, consider the following simple code snippet. Here, the total number of patients in Belgian ICUs over time is given. Summing over all age groups is simple.

```python
plt.plot(out['time'],out['ICU'].sum(dim='Nc'))
```

#### Visualizing the results using builtin methods

Two custom visualization routines are included in `src/covid19model/visualization/output.py` for rapid prototyping, `infected` and `population_status`. For more details, we refer to the docstring in the aforementioned python script.

```python
infected(out)
population_status(out)
```

#### Using posterior samples from previously calibrated parameters to quantify uncertainty

```python
out = model.sim('2020-09-01',start_date='2020-03-15',excess_time=samples_dict['warmup'],N=20,draw_fcn=draw_sample_COVID19_SEIRD,samples=samples_dict)
```

```python
out
```

```python
plt.fill_between(pd.to_datetime(out["time"].values), out["ICU"].quantile(0.05,dim="draws").sum(dim="Nc"), out["ICU"].quantile(0.95,dim="draws").sum(dim="Nc"),alpha=0.30)
plt.plot(out["time"],out["ICU"].mean(dim="draws").sum(dim="Nc"),'--')
```

## Simulations with the stochastic model

### Model initialization

When initializing the stochastic model, a flag `discrete` must be used to switch from the integrator to the iterator.

```python
# Load the parameters using `get_COVID19_SEIRD_parameters()`.
params = model_parameters.get_COVID19_SEIRD_parameters()
# Add the delayed ramp parameters and policy changes during lockdown to model parameters dictionary
params.update({'policy0': Nc_total,
              'policy1': 1.0*Nc_home + (1-0.60)*Nc_work + (1-0.70)*Nc_transport + (1-0.30)*Nc_others + (1-0.80)*Nc_leisure,
               'l': 1,
              'tau': 5,
              'prevention': 0.5,
              'start_date': pd.to_datetime('2020-03-15')})
# Define the initial condition: ten exposed inidividual in every age category
# Why 10? When only using one exposed, there is a big chance the chain will go extinct!
initial_states = {'S': initN, 'E': 3*np.ones(levels)}
# Initialize the model, the only difference with the stochastic model is the `discrete` flag
model = models.COVID19_SEIRD_sto(initial_states, params, discrete=True, time_dependent_parameters={'Nc': lockdown_func})
```

### Load posterior parameter distributions of calibrated parameters

The calibration procedure was exactly the same as for the deterministic model. For more details we refer to the calibration notebook, located in `notebooks/0.1-twallema-calibration-stochastic` . The posterior distributions are located in `data/interim/model_parameters/stochastic_15-08-2020.json`.

#### We'll use the deterministic samples, as no calibration of the stochastic variant is currently available.

### Simulating

After the model initialization, the code works in the exact same way as the deterministic model.

```python
out = model.sim('2020-09-01',start_date='2020-03-15',excess_time=samples_dict['warmup'],N=100,draw_fcn=draw_sample_COVID19_SEIRD,samples=samples_dict)
```

```python
plt.fill_between(pd.to_datetime(out["time"].values), out["ICU"].quantile(1-0.95,dim="draws").sum(dim="Nc"), out["ICU"].quantile(0.95,dim="draws").sum(dim="Nc"),alpha=0.20,color='blue')
plt.fill_between(pd.to_datetime(out["time"].values), out["ICU"].quantile(1-0.68,dim="draws").sum(dim="Nc"), out["ICU"].quantile(0.68,dim="draws").sum(dim="Nc"),alpha=0.80,color='blue')
#plt.plot(pd.to_datetime(out["time"].values),out["ICU"].mean(dim="draws").sum(dim="Nc"),'--',color='blue')
```

```python

```


# 第六讲：事件模型、函数间隔与几何间隔

接着上一讲，进一步讨论朴素贝叶斯算法及事件模型。

## 2.2 文本分类的事件模型

在结束生成学习算法前，我们再来介绍一种针对文本分类的模型。虽然前面介绍的朴素贝叶斯算法对于很多分类问题都要良好的效果，不过对于文本分类，我们有一种更好的模型。

在特定语境的文本分类中，朴素贝叶斯算法使用了**多元伯努利事件模型（multi-variate Bernoulli event model）**（特征值按照伯努利试验取$x_i\in\{0,1\}$，特征值向量长度为字典长度）。回忆一下上一讲的知识，在这个模型中，我们首先假设下一封邮件的发送方已经确定，一个随机的邮件发送人（可能是垃圾邮件制造者，也可能只是普通联系人，是由我们确定的条件概率中的条件）；接着我们假设，发送者会遍历词典，独立的决定是否将单词$x_i$写入邮件（该词语相对于其他词语独立），而且该词语服从概率$p(x_i=1\mid y)=\phi_{i\mid y}$（如果是垃圾邮件制造者则可能更倾向于发送带有“buy”、“sales”、“discount”等单词的 邮件，而普通联系人则会选择更正常的词语分布发送正常邮件）。于是我们得到，$p(x\mid y)=\prod_{i=1}^np(x_i\mid y)$和$p(y)$，再根据$\mathrm{arg}\displaystyle\operatorname*{max}_yp(x\mid y)=\mathrm{arg}\displaystyle\operatorname*{max}_yp(x\mid y)p(y)$最终得到该邮件是垃圾邮件的概率：$p(y)\prod_{i=1}^np(x_i\mid y)$。这个模型存在一些小问题，比如，它不能讲词语的出现的次数反应在输入特征值向量中，如果一封邮件多次出现“buy”这个词，那么此邮件确实更有可能是垃圾邮件。

下面介绍的模型是朴素贝叶斯算法算法的简单变形，称为**多项事件模型（multinomial event model）**。接下来，我们将使用不同的记法和不同的特征值描述邮件。使用$x_i$表示邮件中的第$i$个词语，$x_i$在是一个在$\{1,\cdots,\lvert V\rvert\}$取值的整数，而$\lvert V\rvert$就是词汇表（词典）的大小。于是，一封邮件被我们表示为$(x_1,x_2,\cdots,x_n)$向量，注意，对于不同的邮件$n$是不同的。举个例子，对于一封以“A NIPS”开头的邮件，有$x_1=1$（“a”是词典的第一个单词）、$x_2=35000$（假设“nips”是词典的第35000个单词）。

在多项事件模型中，我们像以前一样假设邮件发送方已经由一个随机过程确定（根据$p(y)$），可能是垃圾邮件制造者，也可能是普通联系人；接着，发件人从某多项分布（$p(x_i\mid y)$）中选出一个词$x_1$；然后根据相同的多项分布选出第二个词$x_2$（独立于$x_1$），继续重复这个动作选出$x_3,x_4,\cdot,x_n$；最后，邮件编写结束。因此，该邮件是垃圾邮件的概率为$p(y)\prod_{i=1}^np(x_i\mid y)$，这个式子和前面的完全一样，但是请注意，这里的特征值$x_i$是邮件第$i$个单词，其取值不再是多元伯努利事件模型中的$\{0,1\}$了，它现在是一个多项分布。

多项事件模型的参数为：$\phi_y=p(y)$（同以前一样），$\phi_{k\mid y=1}=p(x_j=k\mid y=1),\ \phi_{k\mid y=0}=p(x_j=k\mid y=0)$（对任意$j$）。注意到这里我们假设了对于任意$j$都有其$p(x_j\mid y)$相等，即在$y$条件下关于某词语的概率分布与该词语出现在邮件里的位置（$j$）无关。

对于训练集$\left\{\left(x^{(i)},y^{(i)}\right);i=1,\cdots,m\right\}$（其中$x^{(i)}=\left(x_1^{(i)},x_2^{(i)},\cdots,x_{n_i}^{(i)}\right)$代表训练集第$i$封样本邮件特征值向量，邮件共有$n_i$个单词），参数的似然函数为：

$$\begin{align}\mathcal{L}\left(\phi_y,\phi_{k\mid y=0},\phi_{k\mid y=1}\right)&=\prod_{i=1}^mp\left(x^{(i)},y^{(i)}\right)\\&=\prod_{i=1}^m\left(\left(\prod_{j=1}^{n_i}p\left(x_j^{(i)}\mid y;\phi_{k\mid y=1},\phi_{k\mid y=0}\right)\right)p\left(y^{(i)};\phi_y\right)\right)\end{align}$$

最大化似然函数将得到各参数的最大似然估计：

$$\begin{align}\phi_{k\mid y=1}&=p(x_j=k\mid y=1)&=\frac{\sum_{i=1}^m\sum_{j=1}^{n_i}1\left\{x_j^{(i)}=k\land y^{(i)}=1\right\}}{\sum_{i=1}^m1\left\{y^{(i)}=1\right\}n_i}\\\phi_{k\mid y=0}&=p(x_j=k\mid y=0)&=\frac{\sum_{i=1}^m\sum_{j=1}^{n_i}1\left\{x_j^{(i)}=k\land y^{(i)}=0\right\}}{\sum_{i=1}^m1\left\{y^{(i)}=0\right\}n_i}\\\phi_y&=p(y=1)&=\frac{\sum_{i=1}^m1\left\{y^{(i)}=1\right\}}{m}\end{align}$$

我们对$\phi_{k\mid y=0},\phi_{k\mid y=1}$应用拉普拉斯平滑（在实际问题中应用拉普拉斯平滑通常可以得到更好的模型），即在分子上加一，分母上加$\lvert V\rvert$，得到：

$$\begin{align}\phi_{k\mid y=1}&=\frac{\left(\sum_{i=1}^m\sum_{j=1}^{n_i}1\left\{x_j^{(i)}=k\land y^{(i)}=1\right\}\right)+1}{\left(\sum_{i=1}^m1\left\{y^{(i)}=1\right\}n_i\right)+\lvert V\rvert}\\\phi_{k\mid y=0}&=\frac{\left(\sum_{i=1}^m\sum_{j=1}^{n_i}1\left\{x_j^{(i)}=k\land y^{(i)}=0\right\}\right)+1}{\left(\sum_{i=1}^m1\left\{y^{(i)}=0\right\}n_i\right)+\lvert V\rvert}\end{align}$$

在处理文本分类问题时，多项事件模型通常比原始的朴素贝叶斯算法效果更好，一个可能的原因是因为它考虑了每个单词出现的次数。

尽管朴素贝叶斯分类器不是最好的分类算法，但它的效果一般都非常好，再加上它简单且易于实现的特性，我们通常用它作为“首选试验算法”。（使用朴素贝叶斯算法最终会得到一个逻辑函数形式的后验分布，也就是说，朴素贝叶斯算法也属于指数分布族，它仍然是一个线性分类器。）