```python
import numpy as np
from matplotlib import pyplot as plt
from scipy.io import loadmat
import sys, os
from utils import *
import math
%matplotlib inline
```

## Building Models of Spike Train Data

### _Overview_
#### In this tutorial, we will be analyzing the firing activity of a retinal ganglion cell in cuture, exposed to low-light and high-light  environments. The aims of the tutorial follow:

* To visualize spike-train data
* To compute and interpret summary statistics
* To build models of the inter-spike-interval distribution as a function of the light level
* To use these models to interpret new data

### Background

#### The scientific question at hand

Neurons in the retina respond to patterns of light. When retinal ganglion cells are grown in culture and held under constant environmental conditions, these neurons exhibit spontaneous firing activity. A researcher extracted the retina from goldfish and sustained them in culture and performed electrophysiological recordings under two conditions: low ambient light levels and high ambient light levels. The aim of the study was to characterize the variability of the goldfish ganglion cell firing due to the common noise source, the background light level. [1]

#### Neural spike trains as point processes

Point processes can be considered, approximately, to be a binary time series. They can be described in terms of the event times,  inter-event intervals or the number of events that occur. When describing a set of spikes according to the inter-event intervals, we define the interspike intervals as the length of time between one event and the next. These processes can exhibit a variety of history dependent behaviors. The simplest type of point process is a poisson process.

***Poisson processes*** are point processes that are memoryless: the probability of an event happening at a particular time is independent on the occurrence or timing of past events. Homogeneous Poisson processes describe irregular sequences of events that lack a temporal structure. There is a theorem that states that a point process is a homogeneous Poisson process with rate $\lambda$ if and only if the inter-event waiting times are independent, identically distributed exponential variables with rate $\lambda$. We will begin this tutorial by considering the exponential distribution as a plausible model for the interspike intervals.

We will see, however, that the temporal structure of inter-events is relevant because we are often interested in measuring how spiking activity changes with respect to a stimulus. For this reason, the history or previous neural activity influences the spiking activity over time. We will explore this by examining how well the inverse gaussian distribution describes the data.

For a more detailed description of point processes and their use for analyzing neural data, check out Dr. Uri Eden's
point process notes
.

## Application

In order to assess whether there is a difference between the interspike intervals for the retinal ganglion cells between these two conditions, we will model the distribution of the interspike intervals.

### Loading the data

The file 'Ch8-spikes-1.mat' in the directory 'Case-Studies-Kramer-Eden/Chapter8' contains a set of spike-times for two retinal ganglion cells, one exposed to low-level ambient light and another to high-level ambient light.

For this tutorial, we have defined a library called utils with general purpose functions to help assist the learning process. In order to load the data, run the following line of code. You will see that ``retinal_ganglion_data_dict`` is a python dictionary with keys ``SpikesLow`` and ``SpikesHigh``. Both of these lists have units in seconds.

```python
retinal_ganglion_data_path = '../Case-Studies-Kramer-Eden/Chapter8/Ch8-spikes-1.mat'
retinal_ganglion_data_dict = load_retinal_ganglion_data(retinal_ganglion_data_path)
print(retinal_ganglion_data_dict.keys())
spike_train_low = retinal_ganglion_data_dict['SpikesLow']
spike_train_high= retinal_ganglion_data_dict['SpikesHigh']
```

    dict_keys(['SpikesLow', 'SpikesHigh'])

### Exploratory Data Analysis

#### Comparing firing rates

One of the fundamental tenants of neuroscience is that neurons respond to a stimulus by increasing their 'firing rate', the number of spikes that occur per unit time:

\begin{align*}FR = \frac{\text{number of spikes}}{\Delta t}\end{align*}

Given that we have a dataset for which the neuron was exposed to a higher level of light, we can start by exploring whether the firing rate of this neuron is higher under the high light condition than under the low light condition.

#### Computing the average firing rate.

In order to compute the average firing rate, we need to find the number of events that occurred over the duration of the experiment. The lists ``spike_train_low``  and ``spike_train_high`` contain the times where these events occurred. Write a function to obtain the number of spikes that occurred for the low and high conditions. Use this function to compute the average firing rate for the duration of the experiment (30 seconds).

```python
def num_spikes(spike_train):
    pass

def mean_FR(spike_train):
    pass
```

```python
mean_FR_high= mean_FR(spike_train_high)
mean_FR_low = mean_FR(spike_train_low)

#print('Average Firing Rate for the low light condition: ', mean_FR_low  , 'spikes per second')
#print('Average Firing Rate for the high light condition:', mean_FR_high , 'spikes per second')

```

#### Visualizing spike trains

The firing rate has a downfall: its value depends on the interval for which it is computed and on the time interval used for its calculation. This can be seen by looking at the raster plots for the neural activity. A raster plot shows the occurrence of a spike for a sequence of times, the temporal duration of the experiment, for instance. This sequence is called a ***spike train***. A neural spike train represents the time series of spikes that occurred during a recording session.

The following sections show raster plots for the spiking activity for the two conditions. Each row has as x axis the time in seconds, and each vertical blue line represents a spike.

Take a look at the spike trains below. What do these spike trains suggests as possible hypotheses worth exploring in the data?

What information is lost in the average firing rate that is suggested by the spike train?

```python
make_raster_plot(spike_train_low, 'Raster plot for the low background light condition')
```

```python
make_raster_plot(spike_train_high, 'Raster plot for the high background light condition')
```

### The interspike intervals

One possible hypothesis worth exploring is whether there is a difference in the inter-event times. That is, the spike trains suggest that, for the high light condition, there are some segments where the neuron seems to have a higher firing rate, and other segments where the neuron has a lower firing rate. We can explore this by computing the inter-event times, which for neural data are called the interspike intervals.

In order to compare the inter-event times between the low ambient light condition and the high ambient light condition, we first need to compute the inter-spike intervals. The inter-spike intervals are the waiting times between the spike times.

**Write a function that takes as input the list of spike times and returns as output the amount of time between each of the spike times**

```python
def compute_inter_spike_intervals(spike_times):
    pass
```

Use the function ``plot_ISIs`` to visualize the interspike intervals for the low and high conditions. This function takes as input ``ISIs``, ``n_bins``, and ``plot_title``, which are respectively the interspike intervals, the number of bins (an integer), and a title (a string).

Uncomment the following code segment to visualize the empirical distribution of interspike times for each of the light conditions. What do these histograms suggest?

```python
spike_trains_high = retinal_ganglion_data_dict['SpikesHigh']
spike_trains_low  = retinal_ganglion_data_dict['SpikesLow']
#n_bins = 50
#plot_title1 = 'Inter-Spike-Intervals for the High Light Condition'
#plot_title2 = 'Inter-Spike-Intervals for the Low Light Condition'

#low_ISIs    = compute_inter_spike_intervals(spike_trains_low)
#high_ISIs   = compute_inter_spike_intervals(spike_trains_high)

#plot_ISIs(low_ISIs,  n_bins, plot_title2)
#plot_ISIs(high_ISIs, n_bins, plot_title1)
```

### Building Statistical Models of the Inter-Spike-Intervals

In order to investigate whether there is a difference in the inter-spike-intervals for the two conditions, we need a statistical model. We will investigate which of the following distributions best describes the data. We will explore the **exponential** and **inverse Gaussian** distributions.

In order to do so, we will compare the model probability density estimates to the normalized histograms of the Inter-Spike Intervals. Let's start with the exponential distribution.

### The exponential distribution

An **exponential random variable** is a continuous random variable on the interval $[0, \infty)$ and has probability density function given by

\begin{align}
f_x(x) = P(0\leq X \leq \infty) & = \int_0^\infty \! \lambda e^{-\lambda x} \, \mathrm{d}x.
\end{align}

Its cumulative density function is given by

\begin{align}
F_x(x) = P(X \leq x) & = \int_0^x \! 1 - \lambda e^{-\lambda x} \, \mathrm{d}x.
\end{align}

It is a useful distribution for describing waiting times that are **memoryless**, where the next waiting time can be thought of as being independent of the previous waiting times. Furthermore, it has a **rate parameter** $\lambda > 0$.

The following functions will allow you to visualize the exponential PDF and CDF for different $\lambda$ values. Take a look at how the shape of the distribution changes for different parameters $\lambda$. This parameter is the ***rate*** parameter and must be greater than 0.

**Create a vector of different $\lambda$ parameters and an array of x values. Then, use tue function ``exponential_pdf`` to visualize the PDF and the function ``exponential_cdf`` to visualize the CDF for the corresponding $\lambda$ parameters. Use the function ``plot_exponential_dist`` to visualize the CDF and the PDF by uncommenting the last two lines of code in the next block.**

***Hint: the function np.arange is useful for creating arrays within a particular range and at specific increments***

```python

exponential_pdf = lambda x, lam: lam * np.exp(-x*lam)
exponential_cdf = lambda x, lam: 1 - np.exp(-x*lam)

#plot_exponential_dist( exponential_pdf, x_vals, lambdas, 1)
#plot_exponential_dist( exponential_cdf, x_vals, lambdas, 0)
```

Our goal now is to find an appropriate rate parameter $\lambda$ for which to model the interspike intervals for the two light conditions. In order to explore which $\lambda$ parameters might be reasonable, we can overlay the exponential PDF over the **empirical probability distribution**. To compute the empirical probability distribution, we will define small time bins (1ms) and will count the number of times that a given interspike interval falls within each of these time bins.

**Compute the empirical probability distribution for 1000ms time bins and overlay the probability density function for the exponential distribution for different $\lambda$s. Which $\lambda$ best fits the observed ISI distribution?**

***Hint: create a vector that goes from 0 to 0.5 in 0.001 increments. Look at the documentation for np.histogram in order to compute the number of times that the interspike intervals fall within each of these bins***

```python
#Your code here
```

### The method of maximum likelihood maximizes the likelihood function

If we make the assumption that the interspike intervals are independent, and that they are exponentially distributed, then the joint probability distribution of the observed data is given by:

\begin{align}
f(x_1,x_2,\cdots,x_n)
& = f(x_1)f(x_2) \cdots f(x_n) \\
& = \lambda e^{-\lambda x_1 } \lambda e^{-\lambda x_2} \cdots \lambda e^{-\lambda x_n }  \\
& = \lambda^n e^{ -\lambda \sum_{i=1}^{n}{x_i} }
\end{align}

Here, we are considering the PDF as a function of the parameter $\lambda$, rather than as a function of the random variable $x$. In this scenario, $f(x|\lambda)$ is considered the **likelihood function** and we will denote it by $L(\lambda)$. The value of $\lambda$ that maximizes the probability of the data is termed the **maximum likelihood estimator** of $\lambda$. Notationally, we write it as $\hat{\lambda}$.

We can determine the value of $\lambda$ that maximizes the probability of the data through **Maximum Likelihood** methods. That is, we take the derivative of the likelihood function, set it to zero, and solve for $\lambda$. Since often times, the values for $L(\lambda)$ are quite large, it is convenient to take the logarithm and maximize the log-likelihood function $log(L(\lambda)) = \ell(\lambda)$.

Let's compute the value of $\hat{\lambda}$ analytically:

\begin{align}
\ell (\lambda)
& = log( L(\lambda) ) \\
& = log( \lambda^n e^{ -\lambda \sum_{i=1}^{n}{x_i} } ) \\
& = n log(\lambda) - \lambda \sum_{i=1}^{n}{x_i} \\
\frac{\partial \ell (\lambda) }{\partial \lambda}
& = \frac{n}{\lambda} - \sum_{i=1}^{n}{x_i} = 0 \\
\Rightarrow
\hat{\lambda} &= \frac{n}{\sum_{i=1}^{n}{x_i} } \\
\end{align}

**Create a function called ``log_lik_fun`` that takes as input $\lambda$, a numpy array of lambda values, and $x$, the data, and computes the log-likelihood for the exponential distribution. Then, plot the log-likelihood function $\ell (\lambda)$ for different values of $\lambda$ and compute the maximum likelihood estimate $\hat{\lambda}_{ML}$ for the low-light and the high-light conditions. Compare the analytical estimate to the maximum value of the log-likelihood plot.**

***Hint: for getting the maximum value of the line below, use the function ``np.argmax(log_lik)``, which will return the index corresponding to the maximum value of ``lambda_vals``, where ``lambda_vals`` is a vector of the lambdas for which you are plotting the log-likelihood function***

```python
def log_lik_fun(lambdas, ISIs):
    pass
```

**Now, overlay the probability density function for the exponential distribution with the $\hat{\lambda}_{ML}$ estimates and the binned counts from the previous section. How do these distributions differ?**

```python
#Your code here
```

```python
#Your code here
```

### Scientific inference requires having a notion of precision over the estimates.

Confidence intervals allow for quantifying our certainty about our estimate.

 "A 95% confidence interval (CI) for a parameter $\theta$ is an interval of the form $(L, U)$, (L for lower, U for upper), where $L = L(X_1, \cdots , X_n)$ and $U = U(X_1, \cdots, X_n)$ are random variables derived from the data and $P(L \le \theta \le U) = 0.95\%$" -Analysis of Neural Data, Brown <it> et. al</it>.

This statement implies that if $\theta$ were the true value, then the probability that the interval $(L, U)$ contains the $\theta$ is approximately 95%. This does **not** imply that, based on the data, the probability that $(L, U)$ contains $\theta$ is 95%. Think about this statement. Can you see why the two are different?

Another way of interpreting confidence intervals is to draw ***N*** random samples independently and to compute the CIs for each sample. If we do this many times, at the limit, approximately 95% of those CIs would contain $\theta$, if $\theta$ is the true value of the estimate.

Now that we have computed $\hat{\theta}_{ML}$, we want to determine our level of uncertainty over this estimate. We can do so through the **parametric bootstrap**. The parametric bootstrap involves generating pseudo-data from an estimated parametric distribution.

The algorithm follows.

Let $X = x_1, \dots, x_n$ be a set of data with $N$ datapoints.

Choose $G$ to be the number of samples of pseudo-data to generate. We want G to be large (i.e. pick 10,000).

**do**
For $g = 1:G$

   Draw a random sample $U^{(g)} = u_1, \cdots, U_n$ from the distribution with pdf $f(x_i|\hat{\theta})$.

   Compute $\hat{\theta}^{(g)}_{ML}$ based on the bootstrap sample $U^{(g)}$.

   Set $W^{(g)} = \hat{\lambda}^{(g)}$

**end**

Compute $\bar{W} = \frac{1}{G}\sum_{i=1}^{G}{W^{(g)}}$

Sort $W^{(g)}$ in ascending order

Compute the 95% Confidence intervals as the 25th and 95th element of the sorted vector of $W^{(g)}$

**Perform a parametric boostrap procedure to compute $\hat{\theta}_{ML}$ for the exponential distribution, along with its 95% confidence intervals by implementing the algorithm defined above. Then, plot a histogram of the bootstrap estimates for $\hat{\lambda}$ along with the 95% confidence intervals. Can you think of a way to test whether the model parameters are different for the low light condition versus the high light condition?**

```python
alpha   = 0.05
N_boots = 10000
#X_low   = len(low_ISIs)
#X_high  = len(high_ISIs)

#uci_idx = int(((1-alpha)*N_boots))
#lci_idx = int((alpha * N_boots))

#Your code here
```

### Is the difference between the $\hat{\lambda}_{ML}$ estimates for the low-light and high-light conditions statistically significant?

Is the difference between the $\hat{\lambda}_{ML}$ estimates for the low-light and high-light conditions statistically significant? In order to answer this question, we can perform a **parametric bootstrap procedure**. That is, we will estimate $\lambda$ by drawing N random samples with replacement from both estimated distributions. Each random draw will be a vector of the size of the data. We will compute the ML estimate for each random draw. This will give us a distribution of the ML estimates over the data. We can then compare the two distributions for the ISIs under high light condition and under low light condition. To do so, we can use the numpy function  np.random.choice , which takes as input a vector, the number of samples, and a boolean corresponding to whether to sample with replacement or not. We want to sample with replacement.

Now that we have a distribution for the ML estimate for each of the conditions, we can compare these distributions to determine whether the difference between the two parameters is statistically significant.

**Compute the difference between these samples. What are the confidence intervals for the difference between the two distributions? Do the confidence intervals for the difference include 0?**

```python
#Your code here
```

### Goodness of fit: how well does the model describe the data?

A way of checking whether a set of data comes from, approximately, a particular probability model, is to generate **Quantile-Quantile Plots**. Quantile quantile plots allow for comparing the observed quantiles of the data to the theoretical quantiles of the probability distribution. To do so, we sort

Let $X$ be an array containing the data. To compute the observed quantiles, we sort the data in ascending order, $x_{(1)}, \dots, x_{(r)}$ and then say that $x_{(r)}$  is the quantile of order $\frac{r}{(n+1}$, also called the $\frac{100r}{(n+1)}$ percentile. We want to see whether the data are a sample from a distribution having a CDF $F(x)$ then we may define the set of quantile $y_(r) = F^{-1}\frac{r}{(n+1)}$, for $r = 1, \dots , n$, where $n$ is the number of elements in the data set.

**Write down the formula for the inverse CDF for the exponential distribution and write a function that takes as input a list and returns the theoretical quantiles for the exponential distribution. Then, plot the sorted items of the data against the theoretical quantiles. Check your understanding of the process by comparing your plot to the line of code below. Is the exponential distribution a good model for the data? Why, or why not?**

```python
#Your code here

#Complete the following code below to compare it to numpy's probplot function.
from scipy.stats import probplot
from scipy import stats
#lam_ml_low_isis is the ML estimate for the low interspike intervals.
#fig = plt.figure()
#ax  = fig.add_subplot(1,1,1)
#res = stats.probplot(low_ISIs, dist=stats.expon, sparams=(lam_ml_low_isis,), plot=ax)
```

### The inverse Gaussian distribution

We know from physiology that neurons have an absolute refractory period. That is, once a neuron spikes, it cannot do so again for a certain amount of time. This means that the spikes are **not** independent events. A distribution that can be used for modelling this is the inverse Gaussian distribution.

The inverse gaussian distribution has PDF

Am **inverse gaussian random variable** is a continuous random variable on the interval $[0, \infty)$ and has probability density function given by

\begin{align}
f_x(x) = P(0\leq X \leq \infty) & = \int_0^\infty \! \frac{\lambda}{2 \pi x ^3}x^{k-1}e^{\frac{-\lambda(x-\mu)^2}{2 \mu^2 x}}\, \mathrm{d}x.
\end{align}

Its cumulative density function is given by

\begin{align}
F_x(x) = P(X \leq x) & = \int_0^x \! \Phi\Bigg(\sqrt{\frac{\lambda}{x}} \frac{x}{\mu}-1\Bigg) + e^{\frac{2\lambda}{\mu}} \Phi \Bigg(-\sqrt{ \frac{\lambda}{x} } \Bigg), \mathrm{d}x.
\end{align}

where $\Phi$ is the standard normal gaussian CDF , $\mu > 0$ , $\lambda > 0$ and $x \in (0, \infty)$.

**Generate different $\lambda$ and $\mu$ parameters and visualize the shape of the PDF for the Inverse Gaussian Distribution and for its CDF. How does the shape compare to that of the exponential distribution? Overlay the Probability Density Functions for the two distributions. Which one do you think will describe the data best?**

```python
inv_gauss_pdf = lambda x, lam, mu: (lam/ (2*math.pi*(x**3))) ** 0.5 * np.exp( np.divide( (-lam * ((x - mu)**2)), (2*x*(mu**2))))
inv_gauss_cdf = lambda x, lam, mu: norm.cdf(np.sqrt(np.divide(lam,x)) * (np.divide(x,mu)-1)) + (np.exp(np.divide(2*lam,mu)) * norm.cdf(-np.sqrt(np.divide(lam,x))*(np.divide(x,mu)+1)))
lambdas = [1,0.2,3,1,0.2]
mu_vals = [1,1,1,3,3]
x_vals  = np.linspace(0.000001,3,100)
plot_igauss_dist(inv_gauss_pdf, x_vals, lambdas, mu_vals, 1, [0, 3])
plot_igauss_dist(inv_gauss_cdf, x_vals, lambdas, mu_vals, 0, [0, 3])
```

We can see that the inverse gaussian distribution has two parameters that determine its shape, $\lambda$ and $\mu$. As with the exponential distribution, we can compute the maximum likelihood solution for the model, which will be the pair of parameter estimates $\hat{\mu}$ and $\hat{\lambda}$ that maximize the likelihood of the data.

If we assume that the interspike intervals are independent of eachother, the likelihood function is given by:

$L(\mu, \lambda) = \Pi_{i=1}^{N}\sqrt{\frac{\lambda}{2 \pi x_{i}^3}}\exp{\frac{-\lambda(x_i-\mu)^2}{2x_{i}\mu^2}}$

The log-likelihood function is then given by

$\ell (\mu, \lambda) = \frac{N}{2} log \frac{\lambda}{2 \pi} = \frac{3}{2} \sum_{i=1}^{N}log(x_i) - \sum_{i=1}^{N} \frac{\lambda (x_i - \mu)^2}{2x_i \mu^2}$

We can now repeat the steps performed above to assess whether the inverse gaussian distribution fits the data well.

1. Derive the maximum likelihood estimators for $\hat{\mu}$ and $\hat{\lambda}$ by taking the partial derivatives of the log-likelihood function with respect of each of these parameters.

2. Use the function above ``inv_gauss_pdf`` to create a model for the inverse gaussian distribution, using the analytical solution to the maximum-likelihood estimates.

3. Assess whether the model fits the data well by ploting the theoretical quantiles for the inverse CDF against the sorted values for both datasets.

4. Which of the two models is a better fit to the data?

### Inference

Based on the model that best describes the data, perform the following steps to compare the interspike intervals between the two conditions:

1. Compute the 95% confidence intervals for the maximum likelihood estimates through the parametric bootstrap procedure.

2. Compare the ML estimates for the high and low conditions using the bootstrap procedure implemented for the exponential distribution.

### Decision

Based on the results from the model that best fits the data, what can we conclude about the interspike intervals for the high-light condition? Is their mean firing rate higher or lower than that for the low-light condition? What about the shape of the distribution? Does it suggest that the cell is likelier to fire in bursts with short ISIs in the high-light condition, or is it the opposite?

### Acknowledgements

I am grateful to Dr. Brown for allowing me to base this tutorial on the course notes for
 Statistics for Brain and Cognitive Sciences.

I am grateful to Dr. Kramer for allowing me to use the data from his book Case Studies in neural data analysis.

I am also grateful to Dr. Sassanfar and Dr. Ordoñez for the oportunity to teach this tutorial.

### References

[1] Ginsburg, K. S., Johnsen, J. A., & Levine, M. W. (1984). Common noise in the firing of neighbouring ganglion cells in goldfish retina. The Journal of physiology, 351(1), 433-450.

[2] Iyengar, S., & Liao, Q. (1997). Modeling neural activity using the generalized inverse Gaussian distribution. Biological Cybernetics, 77(4), 289-295.

[3] Levine, M. W. (1991). The distribution of the intervals between neural impulses in the maintained discharges of retinal ganglion cells. Biological cybernetics, 65(6), 459-467.

[4] Eden, Uri T., Kramer, Mark A. (2016). Case studies in neural data analysis. MIT Press.

[5] Kass, R. E., Eden, U. T., & Brown, E. N. (2014). Analysis of neural data (Vol. 491). New York: Springer.


```python
from __future__ import division
from numpy import array,linspace,random,warnings
from scipy.stats import bernoulli,norm
from matplotlib import cm
from matplotlib.pyplot import figure,subplots
```

```python
ma_true = 4
mb_true = 7
fa = norm(ma_true,1)
fb = norm(mb_true,1)
fz = bernoulli(0.25)

def sample(n=10):
    tmp = fz.rvs(n)
    return tmp*(fb.rvs(n))+(1-tmp)*fa.rvs(n)
xs = sample(1000)
```

```python
f,ax = subplots()
x = linspace(ma_true-2,mb_true+2,100)
ax.plot(x,fa.pdf(x),label = 'Group A')
ax.plot(x,fb.pdf(x),label = 'Group B')
ax.hist(xs,bins = 50,density =1,label = 'Samples')
ax.legend(loc = 0)
```

```python
#Minimizing the expectation
import sympy
from sympy.abc import x,z
from sympy import stats

mu_a,mu_b = sympy.symbols('mu_a,mu_b')
na = stats.Normal('x',mu_a,1)
nb = stats.Normal('x',mu_b,1)

L = (stats.density(na)(x) + stats.density(nb)(x))/2
```

```python
def ez(x,mu_a,mu_b):
    return norm(mu_a).pdf(x) /(norm(mu_a).pdf(x) + norm(mu_b).pdf(x))
```

```python
Lf = sympy.lambdify((x,mu_a,mu_b),sympy.log(abs(L)),'numpy')

def run():
    out,lout = [],[]
    mu_a_n = random.random()*10
    mu_b_n = random.random()*10
    for i in range(20):
        tau = ez(xs,mu_a_n,mu_b_n)
        lout.append(sum(Lf(xs,mu_a_n,mu_b_n)))
        out.append((mu_a_n,mu_b_n))
        mu_a_n = (sum(tau*xs)/sum(tau))
        mu_b_n = (sum((1-tau)*xs)/sum(1-tau))
    return out,lout

out,lout = run()
fig = figure()
fig.set_figwidth(12)
ax = fig.add_subplot(121)
ax.plot(array(out),'o-')
ax.legend(('mu_a','mu_b'),loc = 0)
ax.hlines([ma_true,mb_true],0,len(out),['r','g'])
ax.set_xlabel('iteration',fontsize = 18)
ax.set_ylabel('$\mu_a,\mu_b$ values',fontsize = 24)
ax = fig.add_subplot(122)
ax.plot(array(lout),'o-')
ax.set_xlabel('iteration',fontsize=18)
ax.set_title('Incomplete likelihood',fontsize = 16)
```

```python
out,lout = run()
mua_step = linspace(0,10,30)
mub_step = linspace(0,10,20)
z = Lf(xs,mua_step[:,None],mub_step[:,None,None]).sum(axis = 2)
fig = figure(figsize=(8,5))
ax = fig.add_subplot(111)
p = ax.contourf(mua_step,mub_step,z,30,cmap=cm.gray)
xa,xb = zip(*out)
ax.plot(xa,xb,'ro')
ax.plot(ma_true,mb_true,'bs')
ax.plot(xa[0],xb[0],'gx',ms=15,mew=2.0)
ax.text(xa[0],xb[0],'start',color = 'g',fontsize=11.0)
ax.set_xlabel('$\mu_a$',fontsize=24)
ax.set_ylabel('$\mu_b$',fontsize=24)
ax.set_title('Incomplete Likelihood',fontsize = 18)
fig.colorbar(p)
```

```python

```

Lesson 2/Force Alignment Expectation/Force Alignment Expectation/Force Alignment Expectation.ipynb


# Dispersion relations in a micropolar medium

We are interested in computing the dispersion relations in a
homogeneous micropolar solid.

## Wave propagation in micropolar solids

The equations of motion for a micropolar solid are given by [[1, 2]](#References)

\begin{align}
&c_1^2
\nabla\nabla\cdot\mathbf{u}- c_2^2\nabla\times\nabla\times\mathbf{u} + K^2\nabla\times\boldsymbol{\theta} = -\omega^2 \mathbf{u} \, ,\\
&c_3^2 \nabla\nabla\cdot\boldsymbol{\theta} - c_4^2\nabla\times\nabla\times\boldsymbol{\theta} + Q^2\nabla\times\mathbf{u} - 2Q^2\boldsymbol{\theta} = -\omega^2 \boldsymbol{\theta} \,
\end{align}

where $\mathbf{u}$ is the displacement vector and $\boldsymbol{\theta}$ is the microrrotations vector,
and where: $c_1$ represents the phase/group speed for the longitudinal wave
($P$) that is non-dispersive as in the classical case, $c_2$ represents the
high-frequency limit phase/group speed for a transverse wave ($S$) that is dispersive unlike the classical counterpart, $c_3$ represents the high-frequency limit phase/group speed
for a longitudinal-rotational wave ($LR$) with a corkscrew-like motion that is
dispersive and does not have a classical counterpart, $c_4$ represents
the high-frequency limit phase/group speed for a transverse-rotational wave ($TR$) that is dispersive and does not have a classical counterpart, $Q$ represents the cut-off frequency for rotational
waves appearance, and $K$ quantifies the difference between the low-frequency
and high-frequency phase/group speed for the S-wave. These parameters are defined by:

\begin{align}
c_1^2 = \frac{\lambda +2\mu}{\rho},\quad &c_3^2 =\frac{\beta + 2\eta}{J},\\
c_2^2 = \frac{\mu +\alpha}{\rho},\quad &c_4^2 =\frac{\eta + \varepsilon}{J},\\
Q^2= \frac{2\alpha}{J},\quad &K^2 =\frac{2\alpha}{\rho} \, ,
\end{align}

## Dispersion relations

To identify types of propagating waves that can arise in the micropolar medium
it is convenient to expand the displacement and rotation vectors in terms of
scalar and vector potentials

\begin{align}
\mathbf{u} &= \nabla \phi + \nabla\times\boldsymbol{\Gamma}\, ,\\
\boldsymbol{\theta} &= \nabla \tau + \nabla\times\mathbf{E}\, ,
\end{align}

subject to the conditions:

\begin{align}
&\nabla\cdot\boldsymbol{\Gamma} = 0\\
&\nabla\cdot\mathbf{E} = 0\, .
\end{align}

Using the above in the displacements equations of motion yields the following
equations, after some manipulations

\begin{align}
c_1^2 \nabla^2 \phi &= \frac{\partial^2 \phi}{\partial t^2}\, ,\\
c_3^2 \nabla^2 \tau - 2Q^2\tau &= \frac{\partial^2 \tau}{\partial t^2}\, ,\\
\begin{bmatrix}
c_2^2 \nabla^2 &K^2\nabla\times\, ,\\
Q^2\nabla\times &c_4^2\nabla^2 - 2Q^2
\end{bmatrix}
\begin{Bmatrix} \boldsymbol{\Gamma}\\ \mathbf{E}\end{Bmatrix} &=
\frac{\partial^2}{\partial t^2} \begin{Bmatrix} \boldsymbol{\Gamma}\\ \mathbf{E}\end{Bmatrix} \, ,
\end{align}

where we can see that the equations for the scalar potentials are uncoupled,
while the ones for the vector potentials are coupled.

Writing the vector potentials as plane waves of amplitude $ \mathbf{A}$ and $
\mathbf{B}$, wave number $\kappa$ and circular frequency $\omega$ that propagate
along the \(x\) axis,

\begin{align}
\boldsymbol{\Gamma} &= \mathbf{A}\exp(i\kappa x - i\omega t)\\
\mathbf{E} &= \mathbf{B}\exp(i\kappa x - i\omega t)\, .
\end{align}

We can do these calculations using some the functions available functions in the package.

```python
from sympy import Matrix, diff, symbols, exp, I, sqrt
from sympy import simplify, expand, solve, limit
from sympy import init_printing, pprint, factor
from continuum_mechanics.vector import lap_vec, curl, div
```

```python
init_printing()
```

```python
A1, A2, A3, B1, B2, B3 = symbols("A1 A2 A3 B1 B2 B3")
kappa, omega, t, x = symbols("kappa omega t x")
c1, c2, c3, c4, K, Q = symbols("c1 c2 c3 c4 K Q", positive=True)
```

We define the vector potentials $\boldsymbol{\Gamma}$ and $\mathbf{E}$.

```python
Gamma = Matrix([A1, A2, A3]) * exp(I*kappa*x - I*omega*t)
E = Matrix([B1, B2, B3]) * exp(I*kappa*x - I*omega*t)
```

And compute the equations using the vector operators. Namely,
the Laplace ([`vector.lap_vec()`](https://continuum-mechanics.readthedocs.io/en/latest/modules.html#vector.lap_vec)  and the curl
([`vector.curl()`](https://continuum-mechanics.readthedocs.io/en/latest/modules.html#vector.curl))
operators.

```python
eq1 = c2**2 * lap_vec(Gamma) + K**2*curl(E) - Gamma.diff(t, 2)
eq2 = Q**2 * curl(Gamma) + c4**2*lap_vec(E) - 2*Q**2*E - E.diff(t, 2)
eq1 = simplify(eq1/exp(I*kappa*x - I*omega*t))
eq2 = simplify(eq2/exp(I*kappa*x  - I*omega*t))
eq = eq1.col_join(eq2)
```

We can compute the matrix for the system using [`.jacobian()`](https://docs.sympy.org/1.5.1/modules/matrices/matrices.html#sympy.matrices.matrices.MatrixCalculus.jacobian)

```python
M = eq.jacobian([A1, A2, A3, B1, B2, B3])
M
```

And, we are interested in the determinant of the matrix $M$.

```python
factor(M.det())
```

The roots for this polynomial (in $\omega^2$) represent the dispersion
relations.

```python
disps = solve(M.det(), omega**2)
for disp in disps:
    display(disp)
```

## References

1. Nowacki, W. (1986). Theory of asymmetric elasticity. Pergamon Press, Headington Hill Hall, Oxford OX 3 0 BW, UK, 1986.

2. Guarín-Zapata, N., Gomez, J., Valencia, C., Dargush, G. F., & Hadjesfandiari, A. R. (2020). Finite element modeling of micropolar-based phononic crystals. Wave Motion, 92, 102406.


# Pocket Coil
-------------------------------------------------------------
A simple pocket sized QRP load coil design for a telescoping antenna

By: Ara Kourchians - N6ARA

## Approach
1) Use existing vertical antenna load coil models to calculate required inductance for a given frequency and length antenna

2) Calculate coil turns for a given winding diameter and wire diameter

3) Calculate counterpoise length

## 1 - Antenna Load Coil Model

### 1.1) Source: J. Hall, "Off-center loaded dipole antennas", QST Sept 1974, 28-34

\begin{equation}
L_{\mu H} = \frac{10^6}{68 \pi^2 f^2} \left\{ \frac{ \left[ ln \frac{24 \left(\frac{234}{f} - B\right)}{D} - 1 \right] \left[ \left(1 - \frac{fB}{234} \right)^2 - 1 \right] }{ \frac{234}{f} - B } - \frac{ \left[ ln \frac{24 \left(\frac{A}{2} - B\right)}{D} - 1 \right] \left[ \left(\frac{ \frac{fA}{2} - fB}{234} \right)^2 - 1 \right] }{ \frac{A}{2} - B } \right\}
\end{equation}

   Where:

   f = Frequency (MHz)

   A = Length of Dipole (ft)

   B = Distance from Feedpoint to Load Coil (ft)

   D = Diameter of radiator (in)

### 1.2) Simplify the equation by multiplying A by 2 so the input is the Vertical Antenna Length

\begin{equation}
L_{\mu H} = \frac{10^6}{68 \pi^2 f^2} \left\{ \frac{ \left[ ln \frac{24 \left(\frac{234}{f} - B\right)}{D} - 1 \right] \left[ \left(1 - \frac{fB}{234} \right)^2 - 1 \right] }{ \frac{234}{f} - B } - \frac{ \left[ ln \frac{24 \left(A - B\right)}{D} - 1 \right] \left[ \left(\frac{fA - fB}{234} \right)^2 - 1 \right] }{ A - B } \right\}
\end{equation}

   Where:

   f = Frequency (MHz)

   A = Vertical Antenna Length (ft)

   B = Distance from Feedpoint to Load Coil (ft)

   D = Diameter of radiator (in)

### 1.3) Now we convert this equation into sensible units:

   1 m = 3.28084 ft

   1 mm = 0.0393701 in

\begin{equation}
L_{\mu H} = \frac{10^6}{68 \pi^2 f^2} \left\{ \frac{ \left[ ln \frac{24 \left(\frac{234}{f} - (B*3.28084)\right)}{D*0.0393701} - 1 \right] \left[ \left(1 - \frac{f*(B*3.28084)}{234} \right)^2 - 1 \right] }{ \frac{234}{f} - (B*3.28084) } - \frac{ \left[ ln \frac{24 \left((A*3.28084) - (B*3.28084)\right)}{D*0.0393701} - 1 \right] \left[ \left(\frac{f(A*3.28084) - f*(B*3.28084)}{234} \right)^2 - 1 \right] }{ (A*3.28084) - (B*3.28084) } \right\}
\end{equation}

   Where:

   f = Frequency (MHz)

   A = Vertical Antenna Length (m)

   B = Distance from Feedpoint to Load Coil (m)

   D = Diameter of radiator (mm)

```python
import numpy as np

f = 14.000  # (MHz) Desired Operating Frequency
A = 0.9906  # (m)   Telescoping Antenna Length
B = 0       # (m)   Distance from feedpoint to coil
D = 4       # (mm)  Antenna Diameter
```

```python
LuH = (10**6)/(68*(np.pi**2)*(f**2)) * \
(

(
    (
    (np.log((24*((234/f)-(B*3.28084)))/(D*0.0393701)) - 1) * (((1-((f*(B*3.28084))/234))**2) - 1)
    )
    /( (234/f)-(B*3.28084) )
)

-

(
    (
    (np.log((24*((A*3.28084)-(B*3.28084)))/(D*0.0393701)) - 1) * (((((f*(A*3.28084))-(f*(B*3.28084)))/234)**2) - 1)
    )
    /( (A*3.28084)-(B*3.28084) )
)

)

print("\nL_uH =", LuH, "uH\n")
```

    L_uH = 11.71514562668202 uH

## 2 - Calculate Coil Turns

### 2.1) Source: Inductor Coil Model - Fundamentals of Electromagnetism, Wentworth 1st Ed, p.466

\begin{equation}
L_{coil} = \frac{\mu{N^2}\pi{r^2}}{h}
\end{equation}

   Where:

   L_coil = Inductance (H)

   N = Number of turns (#)

   mu = Relative Permeability

   r = Loop Radius (m)

   h = Coil Length (m)

### 2.2)  Change loop area to loop diameter

\begin{equation}
L_{coil} = \frac{\mu{N^2}\pi \left(\frac{D}{2}\right)^2}{h}
\end{equation}

   Where:

   L_coil = Inductance (H)

   N = Number of turns (#)

   mu = Relative Permeability

   D = Loop Diameter (m)

   h = Coil Length (m)

### 2.3) Rearrange equation to solve for coil turns

\begin{equation}
N = \sqrt{\frac{L_{coil} * h}{\mu\pi \left( \frac{D}{2} \right)^2 }}
\end{equation}

   Where:

   L_coil = Inductance (H)

   N = Number of turns (#)

   mu = Relative Permeability

   D = Loop Diameter (m)

   h = Coil Length (m)

```python
L_coil = LuH          # uH
mu = 1               # (const) Air
D_coil = 0.016       # (m) Selected loop diameter
H_coil = 0.055       # (m) Selected loop height
```

```python
N = np.sqrt( (L_coil*H_coil) / (mu*np.pi*((D_coil/2)**2)) )

print("\nN =", N, " turns\n")
```

    N = 56.60962358944353  turns

```python
print("\nPocket Coil Design Summary:\n")
print("Frequency =" , format(f, '.3f'), "MHz\n")
print("Coil Diameter =" , format(D_coil, '.2f'), "m")
print("Coil Height =" , format(H_coil, '.3f'), "m")
print("Coil Inductance =" , format(L_coil, '.3f'), "uH\n")
print("N =", format(N, '.0f'), "turns\n")
print("Counterpoise (ideal) =", format(71.3/f, '.2f'), "m")      # L(m) = 71.3 / F
print("Counterpoise (drag/short) =", format(56/f, '.2f'), "m")   # L(m) = 56/f
```

    Pocket Coil Design Summary:

    Frequency = 14.000 MHz

    Coil Diameter = 0.02 m
    Coil Height = 0.055 m
    Coil Inductance = 11.715 uH

    N = 57 turns

    Counterpoise (ideal) = 5.09 m
    Counterpoise (drag/short) = 4.00 m

```python
# TODO
# OPTIONAL: Smith Chart plot of load coil

# Install scikit-rf to plot the load coil inductance on a smith chart
# %pip install scikit-rf
#from pylab import *

#import skrf as rf
#from skrf.data import ring_slot
#ring_slot
#ring_slot.plot_s_smith()
```

```python

```


These exercises teach common workflows which involve SciML's tools like
DifferentialEquations.jl, DiffEqFlux.jl, and the connections to parts like
stochastic differential equations and Bayesian estimation.
The designation (B) is for "Beginner", meaning that a user new to the package
should feel comfortable trying this exercise. An exercise designated (I) is
for "Intermediate", meaning the user may want to have some previous background
in DifferentialEquations.jl or try some (B) exercises first. The additional
(E) designation is for "Experienced", which are portions of exercises which may
take some work.

The exercises are described as follows:

- Exercise 1 takes the user through solving a stiff ordinary differential equation
  and using the ModelingToolkit.jl to automatically convert the function to a
  symbolic form to derive the analytical Jacobian to speed up the solver. The
  same biological system is then solved with stochasticity, utilizing
  EnsembleProblems to understand 95% bounds on the solution. Finally,
  probabilistic programming is employed to perform Bayesian parameter estimation
  of the parameters against data.
- Exercise 2 takes the user through defining hybrid delay differential equation,
  that is a differential equation with events, and using differentiable programming
  techniques (automatic differentiation) to to perform gradient-based parameter
  estimation.
- Exercise 3 takes the user through differential-algebraic equation (DAE)
  modeling, the concept of index, and using both mass-matrix and implicit
  ODE representations. This will require doing a bit of math, but the student
  will understand how to change their equations to make their DAE numerically
  easier for the integrators.
- Exercise 4 takes the user through optimizing a PDE solver, utilizing
  automatic sparsity pattern recognition, automatic conversion of numerical
  codes to symbolic codes for analytical construction of the Jacobian,
  preconditioned GMRES, and setting up a solver for IMEX and GPUs, and compute
  adjoints of PDEs.
- Exercise 5 focuses on a chaotic orbit, utilizing parallel ensembles across
  supercomputers and GPUs to quickly describe phase space.
- Exercise 6 takes the user through training a neural stochastic differential
  equation, using GPU-accleration and adjoints through Flux.jl's neural
  network framework to build efficient training codes.

This exercise worksheet is meant to be a living document leading new users through
a deep dive of the DifferentialEquations.jl feature set. If you further suggestions
or want to contribute new problems, please open an issue or PR at the
SciMLTutorials.jl repository.

# Problem 1: Investigating Sources of Randomness and Uncertainty in a Stiff Biological System (B)

In this problem we will walk through the basics of simulating models with
DifferentialEquations.jl. Let's take the
[Oregonator model of the Belousov-Zhabotinskii chemical reaction system](https://www.radford.edu/~thompson/vodef90web/problems/demosnodislin/Demos_Pitagora/DemoOrego/demoorego.pdf).
This system describes a classical example in non-equilibrium thermodynmics
and is a well-known natural chemical oscillator.

## Part 1: Simulating the Oregonator ODE model

When modeling, usually one starts off by investigating the deterministic model.
The deterministic ODE formulation of the Oregonator is
given by the equations

$$\begin{align}
\frac{dx}{dt} &= s(y-xy + x - qx^2)\\
\frac{dy}{dt} &= (-y - xy + z)/s\\
\frac{dz}{dt} &= w(x - z)\end{align}$$

with parameter values $s=77.27$, $w=0.161$, and $q=8.375 \times 10^{-6}$, and
initial conditions $x(0)=1$, $y(0)=2$, and $z(0)=3$. Use
[the tutorial on solving ODEs](https://docs.juliadiffeq.org/dev/tutorials/ode_example)
to solve this differential equation on the
timespan of $t\in[0,360]$ with the default ODE solver. To investigate the result,
plot the solution of all components over time, and plot the phase space plot of
the solution (hint: use `vars=(1,2,3)`). What shape is being drawn in phase space?

## Part 2: Investigating Stiffness

Because the reaction rates of `q` vs `s` is very large, this model has a "fast"
system and a "slow" system. This is typical of ODEs which exhibit a property
known as stiffness. Stiffness changes the ODE solvers which can handle the
equation well. [Take a look at the ODE solver page](https://docs.juliadiffeq.org/dev/solvers/ode_solve)
and investigate solving the equation using methods for non-stiff equations
(ex: `Tsit5`) and stiff equations (ex: `Rodas5`).

Benchmark using $t\in[0,50]$ using `@btime` from BenchmarkTools.jl. What
happens when you increase the timespan?

## (Optional) Part 3: Specifying Analytical Jacobians (I)

Stiff ODE solvers internally utilize the Jacobian of the ODE system in order
to improve the stepsizes in the solution. However, computing and factorizing
the Jacobian is costly, and thus it can be beneficial to provide the analytical
solution.

Use the
[ODEFunction definition page](https://docs.juliadiffeq.org/dev/features/performance_overloads)
to define an `ODEFunction` which holds both the OREGO ODE and its Jacobian, and solve using `Rodas5`.

## (Optional) Part 4: Automatic Symbolicification and Analytical Jacobian Calculations

Deriving Jacobians by hand is tedious. Thankfully symbolic mathematical systems
can do the work for you. And thankfully, DifferentialEquations.jl has tools
to automatically convert numerical problems into symbolic problems to perform
the analysis on!

follow the [ModelingToolkit.jl README](https://github.com/JuliaDiffEq/ModelingToolkit.jl)
to automatically convert your ODE definition
to its symbolic form using `modelingtoolkitize` and calculate the analytical
Jacobian. Use the compilation functions to build the `ODEFunction` with the
embedded analytical solution.

## Part 5: Adding stochasticity with stochastic differential equations

How does this system react in the presense of stochasticity? We can investigate
this question by using stochastic differential equations. A stochastic
differential equation formulation of this model is known as the multiplicative
noise model, is created with:

$$\begin{align}
dx &= s(y-xy + x - qx^2)dt + \sigma_1 x dW_1\\
dy &= \frac{-y - xy + z}{s}dt + \sigma_2 y dW_2\\
dz &= w(x - z)dt + \sigma_3 z dW_3\end{align}$$

with $\sigma_i = 0.1$ where the `dW` terms describe a Brownian motion, a
continuous random process with normally distributed increments. Use the
[tutorial on solving SDEs](https://docs.juliadiffeq.org/dev/tutorials/sde_example)
to solve simulate this model. Then,
[use the `EnsembleProblem`](https://docs.juliadiffeq.org/dev/features/ensemble)
to generate and plot 100 trajectories of the stochastic model, and use
`EnsembleSummary` to plot the mean and 5%-95% region over time.

Try solving with the `ImplicitRKMil` and `SOSRI` methods. Notice that it isn't
stiff every single time!

(For fun, see if you can make the Euler-Maruyama `EM()` method solve this equation.
This requires a choice of `dt` small enough to be stable. This is the "standard"
method!)

## Part 6: Gillespie jump models of discrete stochasticity

When biological models have very few particles, continuous models no longer
make sense, and instead using the full discrete formulation can be required
to accuracy describe the dynamics. A discrete differential equation, or
Gillespie model, is a continuous-time Markov chain with Poisson-distributed
jumps. A discrete description of the Oregonator model is given by a chemical
reaction systems:

```julia
A+Y -> X+P
X+Y -> 2P
A+X -> 2X + 2Z
2X  -> A + P (note: this has rate kX^2!)
B + Z -> Y
```

where reactions take place at a rate which is propoertional to its components,
i.e. the first reaction has a rate `k*A*Y` for some `k`.
Use the [tutorial on Gillespie SSA models](https://docs.juliadiffeq.org/dev/tutorials/discrete_stochastic_example)
to implement the `JumpProblem` for this model, and use the `EnsembleProblem`
and `EnsembleSummary` to characterize the stochastic trajectories.

For what rate constants does the model give the oscillatory dynamics for the
ODE approximation? For information on the true reaction rates, consult
[the original paper](https://pubs.acs.org/doi/abs/10.1021/ja00780a001).

## Part 7: Probabilistic Programming / Bayesian Parameter Estimation with DiffEqBayes.jl + Turing.jl (I)

In many casees, one comes to understand the proper values for their model's
parameters by utilizing data fitting techniques. In this case, we will use
the DiffEqBayes.jl library to perform a Bayesian estimation of the parameters.
For our data we will the following potential output:

```julia
t = 0.0:1.0:30.0
data = [1.0 2.05224 2.11422 2.1857 2.26827 2.3641 2.47618 2.60869 2.7677 2.96232 3.20711 3.52709 3.97005 4.64319 5.86202 9.29322 536.068 82388.9 57868.4 1.00399 1.00169 1.00117 1.00094 1.00082 1.00075 1.0007 1.00068 1.00066 1.00065 1.00065 1.00065
        2.0 1.9494 1.89645 1.84227 1.78727 1.73178 1.67601 1.62008 1.56402 1.50772 1.45094 1.39322 1.33366 1.2705 1.19958 1.10651 0.57194 0.180316 0.431409 251.774 591.754 857.464 1062.78 1219.05 1335.56 1419.88 1478.22 1515.63 1536.25 1543.45 1539.98
        3.0 2.82065 2.68703 2.58974 2.52405 2.48644 2.47449 2.48686 2.52337 2.58526 2.67563 2.80053 2.9713 3.21051 3.5712 4.23706 12.0266 14868.8 24987.8 23453.4 19202.2 15721.6 12872.0 10538.8 8628.66 7064.73 5784.29 4735.96 3877.66 3174.94 2599.6]
```

[Follow the exmaples on the parameter estimation page](https://docs.juliadiffeq.org/latest/analysis/parameter_estimation/#Bayesian-Inference-Examples-1)
to perform a Bayesian parameter estimation. What are the most likely parameters
for the model given the posterior parameter distributions?

Use the `ODEProblem` to perform the fit. If you have time, use the `EnsembleProblem`
of `SDEProblem`s to perform a fit over averages of the SDE solutions. Note that
the SDE fit will take significantly more computational resources! See the GPU
parallelism section for details on how to accelerate this.

## (Optional) Part 8: Using DiffEqBiological's Reaction Network DSL

DiffEqBiological.jl is a helper library for the DifferentialEquations.jl
ecosystem for defining chemical reaction systems at a high leevel for easy
simulation in these various forms. Use the descrption
[from the Chemical Reaction Networks documentation page](https://docs.juliadiffeq.org/dev/models/biological)
to build a reaction network and generate the ODE/SDE/jump equations, and
compare the result to your handcoded versions.

# Problem 2: Fitting Hybrid Delay Pharmacokinetic Models with Automated Responses (B)

Hybrid differential equations are differential equations with events, where
events are some interaction that occurs according to a prespecified condition.
For example, the bouncing ball is a classic hybrid differential equation given
by an ODE (Newton's Law of Gravity) mixed with the fact that, whenever the
ball hits the floor (`x=0`), then the velocity of the ball flips (`v=-v`).

In addition, many models incorporate delays, that is the driving force of the
equation is dependent not on the current values, but values from the past.
These delay differential equations model how individuals in the economy act
on old information, or that biological processes take time to adapt to a new
environment.

In this equation we will build a hybrid delayed pharmacokinetic model and
use the parameter estimation techniques to fit this it to a data.

## Part 1: Defining an ODE with Predetermined Doses

First, let's define the simplest hybrid ordinary differential equation: an ODE
where the events take place at fixed times. The ODE we will use is known as
the one-compartment model:

$$\begin{align}
\frac{d[Depot]}{dt} &= -K_a [Depot] + R\\
\frac{d[Central]}{dt} &= K_a [Depot] - K_e [Central]\end{align}$$

with $t \in [0,90]$, $u_0 = [100.0,0]$, and $p=[K_a,K_e]=[2.268,0.07398]$.

With this model, use [the event handling documentation page](https://docs.juliadiffeq.org/dev/features/callback_functions)
to define a `DiscreteCallback` which fires at `t ∈ [24,48,72]` and adds a
dose of 100 into `[Depot]`. (Hint: you'll want to set `tstops=[24,48,72]` to
force the ODE solver to step at these times).

## Part 2: Adding Delays

Now let's assume that instead of there being one compartment, there are many
transit compartment that the drug must move through in order to reach the
central compartment. This effectively delays the effect of the transition from
`[Depot]` to `[Central]`. To model this effect, we will use the delay
differential equation which utilizes a fixed time delay $\tau$:

$$\begin{align}
\frac{d[Depot]}{dt} &= -K_a [Depot](t)\\
\frac{d[Central]}{dt} &= K_a [Depot](t-\tau) - K_e [Central]\end{align}$$

where the parameter $τ = 6.0$.
[Use the DDE tutorial](https://docs.juliadiffeq.org/dev/tutorials/dde_example)
to define and solve this delayed version of the hybrid model.

## Part 3: Automatic Differentiation (AD) for Optimization (I)

In order to fit parameters $(K_a,K_e,\tau)$ we will want to be able to calculate
the gradient of the solution with respect to the initial conditions. One way to
do this is via Automatic Differentition (AD). For small numbers of parameters
(<100), it is fastest to use Forward-Mode Automatic Differentition
(even faster than using adjoint sensitivity analysis!). Thus for this problem
we will make use of ForwardDiff.jl to use Dual number arithmetic to retrive
both the solution and its derivative w.r.t. parameters in a single solve.

[Use the information from the page on local sensitvity analysis](https://docs.juliadiffeq.org/dev/analysis/sensitivity)
to define the input dual numbers, solve the equation, and plot both the solution
over time and the derivative of the solution w.r.t. the parameters.

## Part 4: Fitting Known Quantities with DiffEqParamEstim.jl + Optim.jl

Now let's fit the delayed model to a dataset. For the data, use the array

```julia
t = 0.0:12.0:90.0
data = [100.0 0.246196 0.000597933 0.24547 0.000596251 0.245275 0.000595453 0.245511
        0.0 53.7939 16.8784 58.7789 18.3777 59.1879 18.5003 59.2611]
```

Use [the parameter estimation page](https://docs.juliadiffeq.org/dev/analysis/parameter_estimation)
to define a loss function with `build_loss_objective` and optimize the parameters
against the data. What parameters were used to generate the data?

## Part 5: Implementing Control-Based Logic with ContinuousCallbacks (I)

Now that we have fit our delay differential equation model to the dataset, we
want to start testing out automated treatment strategies. Let's assume that
instead of giving doses at fixed time points, we invent a wearable which
monitors the patient and administers a dose whenever the internal drug
concentration falls below 25. To model this effect, we will need to use
`ContinuousCallbacks` to define a callback that triggers when `[Central]` falls
below the threshold value.

[Use the documentation on the event handling page](https://docs.juliadiffeq.org/dev/features/callback_functions) to define such a callback,
and plot the solution over time. How many times does the auto-doser administer
a dose? How much does this change as you change the delay time $\tau$?

## Part 6: Global Sensitivity Analysis with the Morris and Sobol Methods

To understand how the parameters effect the solution in a global sense, one
wants to use Global Sensitivity Analysis. Use the
[GSA documentation page](https://docs.juliadiffeq.org/dev/analysis/global_sensitivity)
perform global sensitivity analysis and quantify the effect of the various
parameters on the solution.

# Problem 3: Differential-Algebraic Equation Modeling of a Double Pendulum (B)

Differential-Algebraic Equaton (DAE) systems are like ODEs but allow for adding
constraints into the models. This problem will look at solving the double
penulum problem with enforcement of the rigid body constraints, requiring that
the total distance `L` is constant throughout the simulation. While these
equations can be rewritten in an ODE form, in many cases it can be simpler
to solve the equation directly with the constraints. This tutorial will
cover both the idea of index, how to manually perform index reduction,
and how to make use of mass matrix and implicit ODE solvers to handle these
problems.

## Part 1: Simple Introduction to DAEs: Mass-Matrix Robertson Equations

A mass-matrix ordinary differential equation (ODE) is an ODE where the
left-hand side, the derivative side, is multiplied by a matrix known as the
mass matrix. This is described as:

$$Mu' = f(u,p,t)$$

where $M$ is the mass matrix. When $M$ is invertible, there is an ODE which is
equivalent to this formulation. When $M$ is not invertible, this can have a
distinctly different behavior and is as Differential-Algebraic Equation (DAE).

Solve the Robertson DAE:

$$\begin{align}
\frac{dy_1}{dt} &= -0.04y_1 + 10^4 y_2y_3\\
\frac{dy_2}{dt} &=  0.04y_1 - 10^4 y_2y_3 - 3\times 10^7 y_2^2\\
1 &= y_1 + y_2 + y_3\end{align}$$

with $y(0) = [1,0,0]$ and $dy(0) = [-0.04,0.04,0.0]$ using the mass-matrix
formulation and `Rodas5()`. Use the
[ODEProblem page](https://docs.juliadiffeq.org/dev/types/ode_types)
to find out how to declare a mass matrix.

(Hint: what if the last row has all zeros?)

## Part 2: Solving the Implicit Robertson Equations with IDA

Use the [DAE Tutorial](https://docs.juliadiffeq.org/dev/tutorials/dae_example)
to define a DAE in its implicit form and solve the Robertson equation with IDA.
Why is `differential_vars = [true,true,false]`?

## Part 3: Manual Index Reduction of the Single Pendulum

The index of a DAE is a notion used to measure distance from
its related ODE. There are many different definitions of index,
but we're going to stick to the idea of differential index:
the number of differentiations required to convert a system
of DAEs into explicit ODE form. DAEs of high index are
usually transformed via a procedure called index reduction.
The following example will demonstrate this.

Consider the index 3 DAE system of the cartesian pendulum.
After writing down the force equations in both directions,
we arrive at the following DAE:

$$
\begin{align}
m\ddot{x} &= \frac{x}{L}T \\
m\ddot{y} &= \frac{y}{L}T - mg \\
x^2 + y^2 &= L
\end{align}
$$

Notice that we don't have an equation describing the
behaviour of `T`. Let us now perform index reduction to
extract an equation for `T`

Differentiate this third equation twice with respect to time
to reduce it from index 3 to index 1.

## Part 4: Single Pendulum Solution with IDA
Write these equations in implicit form and solve the system using
IDA.

## Part 5: Solving the Double Penulum DAE System

The following equations describe a double
pendulum system:
$$
\begin{align}
m_2\ddot{x_2} &= \frac{x_2}{L_2}T_2 \\
m_2\ddot{y_2} &= \frac{y_2}{L_2}T_2 - m_2g \\
{x_2}^2 + {y_2}^2 &= L_2 \\
m_1\ddot{x_1} &= \frac{x_1}{L_1}T_1 - \frac{x_2}{L_2}T_2 \\
m_2\ddot{y_1} &= \frac{y_1}{L_1}T_2 - m_1g - \frac{y_2}{L_2}T_2 \\
{x_1}^2 + {y_1}^2 &= L_1 \\
\end{align}
$$

Perform index reduction and solve it like in the previous example.

# Problem 4: Performance Optimizing and Parallelizing Semilinear PDE Solvers (I)

This problem will focus on implementing and optimizing the solution of the
2-dimensional Brusselator equations. The BRUSS equations are a well-known
highly stiff oscillatory system of partial differential equations which are
used in stiff ODE solver benchmarks. In this tutorial we will walk first
through a simple implementation, then do allocation-free implementations and
looking deep into solver options and benchmarking.

## Part 1: Implementing the BRUSS PDE System as ODEs

The Brusselator PDE is defined as follows:

$$\begin{align}
\frac{\partial u}{\partial t} &= 1 + u^2v - 4.4u + \alpha(\frac{\partial^2 u}{\partial x^2} + \frac{\partial^2 u}{\partial y^2}) + f(x, y, t)\\
\frac{\partial v}{\partial t} &= 3.4u - u^2v + \alpha(\frac{\partial^2 u}{\partial x^2} + \frac{\partial^2 u}{\partial y^2})\end{align}$$

where

$$f(x, y, t) = \begin{cases}
5 & \quad \text{if } (x-0.3)^2+(y-0.6)^2 ≤ 0.1^2 \text{ and } t ≥ 1.1 \\
0 & \quad \text{else}\end{cases}$$

and the initial conditions are

$$\begin{align}
u(x, y, 0) &= 22\cdot y(1-y)^{3/2} \\
v(x, y, 0) &= 27\cdot x(1-x)^{3/2}\end{align}$$

with the periodic boundary condition

$$\begin{align}
u(x+1,y,t) &= u(x,y,t) \\
u(x,y+1,t) &= u(x,y,t)\end{align}$$

on a timespan of $t \in [0,22]$.

To solve this PDE, we will discretize it into a system of ODEs with the finite
difference method. We discretize `u` and `v` into arrays of the values at each
time point: `u[i,j] = u(i*dx,j*dy)` for some choice of `dx`/`dy`, and same for
`v`. Then our ODE is defined with `U[i,j,k] = [u v]`. The second derivative
operator, the Laplacian, discretizes to become a tridiagonal matrix with
`[1 -2 1]` and a `1` in the top right and bottom left corners. The nonlinear functions
are then applied at each point in space (they are broadcast). Use `dx=dy=1/32`.

You will know when you have the correct solution when you plot the solution
at `x=y=0` and see a periodic orbit, e.g., `ts=0:0.05:22; plot(ts, sol1.(ts,
idxs=1))`.

If you are not familiar with this process, see
[the Gierer-Meinhardt example from the SciMLTutorials.](http://tutorials.juliadiffeq.org/html/introduction/03-optimizing_diffeq_code.html)

Note: Start by doing the simplest implementation!

## Part 2: Optimizing the BRUSS Code

PDEs are expensive to solve, and so we will go nowhere without some code
optimizing! Follow the steps described in the
[the Gierer-Meinhardt example from the SciMLTutorials](http://tutorials.juliadiffeq.org/html/introduction/03-optimizing_diffeq_code.html)
to optimize your Brusselator code. Try other formulations and see what ends
up the fastest! Find a trade-off between performance and simplicity that suits
your needs.

## Part 3: Exploiting Jacobian Sparsity with Color Differentiation

Use the `sparsity!` function from [SparseDiffTools](https://github.com/JuliaDiffEq/SparseDiffTools.jl)
to generate the sparsity pattern for the Jacobian of this problem. Follow
the documentations [on the DiffEqFunction page](https://docs.juliadiffeq.org/dev/features/performance_overloads)
to specify the sparsity pattern of the Jacobian. Generate an add the color
vector to speed up the computation of the Jacobian.

## (Optional) Part 4: Structured Jacobians

Specify the sparsity pattern using a BlockBandedMatrix from
[BlockBandedMatrices.jl](https://github.com/JuliaMatrices/BlockBandedMatrices.jl)
to accelerate the previous sparsity handling tricks.

## (Optional) Part 5: Automatic Symbolicification and Analytical Jacobian

Use the `modelingtoolkitize` function from ModelingToolkit.jl to convert your
numerical ODE function into a symbolic ODE function and use that to compute and
solve with an analytical sparse Jacobian.

## Part 6: Utilizing Preconditioned-GMRES Linear Solvers

Use the [linear solver specification page](https://docs.juliadiffeq.org/dev/features/linear_nonlinear)
to solve the equation with `TRBDF2` with GMRES. Use the Sundials documentation
to solve the equation with `CVODE_BDF` with Sundials' special internal GMRES.
To both of these, use the [AlgebraicMultigrid.jl](https://github.com/JuliaLinearAlgebra/AlgebraicMultigrid.jl)
to add a preconditioner to the GMRES solver.

## Part 7: Exploring IMEX and Exponential Integrator Techniques (E)

Instead of using the standard `ODEProblem`, define a [`SplitODEProblem`](https://docs.juliadiffeq.org/dev/types/split_ode_types)
to move some of the equation to the "non-stiff part". Try different splits
and solve with `KenCarp4` to see if the solution can be accelerated.

Next, use `MatrixFreeOperator` and `DiffEqArrayOperator` to define part of the equation as linear, and
use the `ETDRK4` exponential integrator to solve the equation. Note that this
technique is not appropriate for this equation since it relies on the
nonlinear term being non-stiff for best results.

## Part 8: Work-Precision Diagrams for Benchmarking Solver Choices

Use the `WorkPrecisionSet` method from
[DiffEqDevTools.jl](https://github.com/JuliaDiffEq/DiffEqDevTools.jl) to
benchmark multiple different solver methods and find out what combination is
most efficient.
[Take a look at DiffEqBenchmarks.jl](https://github.com/JuliaDiffEq/DiffEqBenchmarks.jl)
for usage examples.

## Part 9: GPU-Parallelism for PDEs (E)

Fully vectorize your implementation of the ODE and use a `CuArray` from
[CuArrays.jl](https://github.com/JuliaGPU/CuArrays.jl) as the initial condition
to cause the whole solution to be GPU accelerated.

## Part 10: Adjoint Sensitivity Analysis for Gradients of PDEs

In order to optimize the parameters of a PDE, you need to be able to compute
the gradient of the solution with respect to the parameters. This is done
through sensitivity analysis. For PDEs, generally the system is at a scale
where forward sensitivity analysis (forward-mode automatic differentiation)
is no longer suitable, and for these cases one uses adjoint sensitivity analysis.

Rewrite the PDE so the constant terms are parameters, and use the
[adjoint sensitivity analysis](https://docs.juliadiffeq.org/latest/analysis/sensitivity/#Adjoint-Sensitivity-Analysis-1)
documentation to solve for the solution gradient with a cost function being the
L2 distance of the solution from the value 1. Solve with interpolated and
checkpointed adjoints. Play with using reverse-mode automatic differentiation
vs direct computation of vector-Jacobian products using the `autojacvec` option
of the `SensitivityAlg`. Find the set of options most suitable for this PDE.

If you have compute time, use this adjoint to optimize the parameters of the
PDE with respect to this cost function.

# Problem 5: Global Parameter Sensitivity and Optimality with GPU and Distributed Ensembles (B)

In this example we will investigate how the parameters "generally" effect the
solution in the chaotic Henon-Heiles system. By "generally" we will use global
sensitivity analysis methods to get an average global characterization of the
parameters on the solution. In addition to a global sensitivity approach, we
will generate large ensembles of solutions with different parameters using
a GPU-based parallelism approach.

## Part 1: Implementing the Henon-Heiles System (B)

The Henon-Heiles Hamiltonian system is described by the ODEs:

$$\begin{align}
\frac{dp_1}{dt} &= -q_1 (1 + 2q_2)\\
\frac{dp_2}{dt} &= -q_2 - (q_1^2 - q_2^2)\\
\frac{dq_1}{dt} &= p_1\\
\frac{dq_2}{dt} &= p_2\end{align}$$

with initial conditions $u_0 = [0.1,0.0,0.0,0.5]$.
Solve this system over the timespan $t\in[0,1000]$

## (Optional) Part 2: Alternative Dynamical Implmentations of Henon-Heiles (B)

The Henon-Heiles defines a Hamiltonian system with certain structures which
can be utilized for a more efficient solution. Use [the Dynamical problems page](https://docs.juliadiffeq.org/dev/types/dynamical_types)
to define a `SecondOrderODEProblem` corresponding to the acceleration terms:

$$\begin{align}
\frac{d^2q_1}{dt^2} &= -q_1 (1 + 2q_2)\\
\frac{d^2q_2}{dt^2} &= -q_2 - (q_1^2 - q_2^2)\end{align}$$

Solve this with a method that is specific to dynamical problems, like `DPRKN6`.

The Hamiltonian can also be directly described:

$$H(p,q) = \frac{1}{2}(p_1^2 + p_2^2) + \frac{1}{2}(q_1^2+q_2^2+2q_1^2 q_2 - \frac{2}{3}q_2^3)$$

Solve this problem using the `HamiltonianProblem` constructor from DiffEqPhysics.jl.

## Part 3: Parallelized Ensemble Solving

To understand the orbits of the Henon-Heiles system, it can be useful to solve
the system with many different initial conditions. Use the
[ensemble interface](https://docs.juliadiffeq.org/dev/features/ensemble)
to solve with randomized initial conditions in parallel using threads with
`EnsembleThreads()`. Then, use `addprocs()` to add more cores and solve using
`EnsembleDistributed()`. The former will solve using all of the cores on a
single computer, while the latter will use all of the cores on which there
are processors, which can include thousands across a supercomputer! See
[Julia's parallel computing setup page](https://docs.julialang.org/en/v1/manual/parallel-computing/index.html)
for more details on the setup.

## Part 4: Parallelized GPU Ensemble Solving

Setup the CUDAnative.jl library and use the `EnsembleGPUArray()` method to
parallelize the solution across the thousands of cores of a GPU. Note that
this will efficiency solve for hundreds of thousands of trajectores.

# Problem 6: Training Neural Stochastic Differential Equations with GPU acceleration (I)

In the previous models we had to define a model. Now let's shift the burden of
model-proofing onto data by utilizing neural differential equations. A neural
differential equation is a differential equation where the model equations
are replaced, either in full or in part, by a neural network. For example, a
neural ordinary differential equation is an equation $u^\prime = f(u,p,t)$
where $f$ is a neural network. We can learn this neural network from data using
various methods, the easiest of which is known as the single shooting method,
where one chooses neural network parameters, solves the equation, and checks
the ODE's solution against data as a loss.

In this example we will define and train various forms of neural differential
equations. Note that all of the differential equation types are compatible with
neural differential equations, so this is only going to scratch the surface of
the possibilites!

## Part 1: Constructing and Training a Basic Neural ODE

Use the [DiffEqFlux.jl README](https://github.com/JuliaDiffEq/DiffEqFlux.jl) to
construct a neural ODE to train against the training data:

```julia
u0 = Float32[2.; 0.]
datasize = 30
tspan = (0.0f0,1.5f0)

function trueODEfunc(du,u,p,t)
    true_A = [-0.1 2.0; -2.0 -0.1]
    du .= ((u.^3)'true_A)'
end
t = range(tspan[1],tspan[2],length=datasize)
prob = ODEProblem(trueODEfunc,u0,tspan)
ode_data = Array(solve(prob,Tsit5(),saveat=t))
```

## Part 2: GPU-accelerating the Neural ODE Process

Use the `gpu` function from Flux.jl to transform all of the calculations onto
the GPU and train the neural ODE using GPU-accelerated `Tsit5` with adjoints.

## Part 3: Defining and Training a Mixed Neural ODE

Gather data from the Lotka-Volterra equation:

```julia
function lotka_volterra(du,u,p,t)
  x, y = u
  α, β, δ, γ = p
  du[1] = dx = α*x - β*x*y
  du[2] = dy = -δ*y + γ*x*y
end
u0 = [1.0,1.0]
tspan = (0.0,10.0)
p = [1.5,1.0,3.0,1.0]
prob = ODEProblem(lotka_volterra,u0,tspan,p)
sol = Array(solve(prob,Tsit5())(0.0:1.0:10.0))
```

Now use the
[mixed neural section of the documentation](https://github.com/JuliaDiffEq/DiffEqFlux.jl#mixed-neural-des)
to define the mixed neural ODE where the functional form of $\frac{dx}{dt}$ is
known, and try to derive a neural formulation for $\frac{dy}{dt}$ directly from
the data.

## Part 4: Constructing a Basic Neural SDE

Generate data from the Lotka-Volterra equation with multiplicative noise

```julia
function lotka_volterra(du,u,p,t)
  x, y = u
  α, β, δ, γ = p
  du[1] = dx = α*x - β*x*y
  du[2] = dy = -δ*y + γ*x*y
end
function lv_noise(du,u,p,t)
  du[1] = p[5]*u[1]
  du[2] = p[6]*u[2]
end
u0 = [1.0,1.0]
tspan = (0.0,10.0)
p = [1.5,1.0,3.0,1.0,0.1,0.1]
prob = SDEProblem(lotka_volterra,lv_noise,u0,tspan,p)
sol = [Array(solve(prob,SOSRI())(0.0:1.0:10.0)) for i in 1:20] # 20 solution samples
```

Train a neural stochastic differential equation $dX = f(X)dt + g(X)dW_t$ where
both the drift ($f$) and the diffusion ($g$) functions are neural networks.
See if constraining $g$ can make the problem easier to fit.

## Part 5: Optimizing the training behavior with minibatching (E)

Use minibatching on the data to improve the training procedure. An example
[can be found at this PR](https://github.com/FluxML/model-zoo/pull/88).


# Classification

The goal of classification is to find a function that separates the data into positive/negative labels. In the case of a linear classifier, this reduces to finding a set of parameters $w^\star$ such that, $$ \begin{align} w^\star &= \arg \min_w \sum_{i=1}^{N} \left[y_i\neq \text{sign} (w^\top x_i) \right] \\ &= \arg \min_w \sum_{i=1}^{N} l_{0/1} (w; x_i, y_i) \end{align}.$$

The problem with the $l_{0/1}$ loss, is that it is non-convex (and non-differentiable), hence other surrogate losses must be used to optimize the number of points.

```python
# Code source: Sebastian Curi and Andreas Krause, based on Jaques Grobler (sklearn demos).
# License: BSD 3 clause

# We start importing some modules and running some magic commands
% matplotlib inline
% reload_ext autoreload
% load_ext autoreload
% autoreload 2

# General math and plotting modules.
import numpy as np
import matplotlib.pyplot as plt

# Project files.
from util import gradient_descent, generate_linear_separable_data, generate_circular_separable_data
import plot_helpers
from classifiers import Perceptron, SVM, Logistic
from regularizers import L1Regularizer, L2Regularizer

# Widget and formatting modules
import ipywidgets
from ipywidgets import interact, interactive, interact_manual
import pylab
# If in your browser the figures are not nicely vizualized, change the following line.
pylab.rcParams['figure.figsize'] = (10, 5)

# Machine Learning library.
import sklearn
from sklearn import svm
from sklearn.linear_model import SGDClassifier

import warnings
warnings.filterwarnings('ignore')
# from sklearn import datasets, linear_model

```

```python
num_points = 100  # Number of points per class
noise = 0.5  # Noise Level (needed for data generation).
np.random.seed(42)
X, Y = generate_linear_separable_data(num_points, noise=noise, dim=2)
# X = X - np.mean(X)

fig = plt.subplot(111)
opt = {'marker': 'ro', 'label': '+', 'size': 8}
plot_helpers.plot_data(X[np.where(Y == 1)[0], 0], X[np.where(Y == 1)[0], 1], fig=fig, options=opt)
opt = {'marker': 'bs', 'label': '-', 'x_label': '$x$', 'y_label': '$y$', 'size': 8, 'legend': True}
plot_helpers.plot_data(X[np.where(Y == -1)[0], 0], X[np.where(Y == -1)[0], 1], fig=fig, options=opt)
```

```python
# Separate into train and test sets!
np.random.seed(42)
indexes = np.arange(0, 2*num_points, 1)
np.random.shuffle(indexes)
num_train = int(np.ceil(2*.05*num_points))

X_train = X[indexes[:num_train]]
Y_train = Y[indexes[:num_train]]

X_test = X[indexes[num_train:]]
Y_test = Y[indexes[num_train:]]

fig = plt.subplot(111)

opt = {'marker': 'ro', 'fillstyle': 'full', 'label': '+ Train', 'size': 8}
plot_helpers.plot_data(X_train[np.where(Y_train == 1)[0], 0], X_train[np.where(Y_train == 1)[0], 1], fig=fig, options=opt)
opt = {'marker': 'bs', 'fillstyle': 'full', 'label': '- Train', 'size': 8}
plot_helpers.plot_data(X_train[np.where(Y_train == -1)[0], 0], X_train[np.where(Y_train == -1)[0], 1], fig=fig, options=opt)

opt = {'marker': 'ro', 'fillstyle': 'none', 'label': '+ Test', 'size': 8}
plot_helpers.plot_data(X_test[np.where(Y_test == 1)[0], 0], X_test[np.where(Y_test == 1)[0], 1], fig=fig, options=opt)
opt = {'marker': 'bs', 'fillstyle': 'none', 'label': '- Test', 'size': 8,
       'x_label': '$x$', 'y_label': '$y$', 'legend': True}
plot_helpers.plot_data(X_test[np.where(Y_test == -1)[0], 0], X_test[np.where(Y_test == -1)[0], 1], fig=fig, options=opt)

```

## The Perceptron Algorithm

The perceptron loss is defined as: $$L(w; X, Y) =  \sum_{i=1}^{N} L_p(w; x_i, y_i) = \sum_{i=1}^{N} \max \{ 0, -y_i w^\top x_i \}.$$

The loss function is continuous, but not differentialbe at $y_i w^\top x_i=0$. The subgradient, however, exists and hence (stochastic) gradient descent converges. The subgradient is:

$$ \partial L_p(w; x_i,y_i) = \left\{\begin{array}{cc} 0 & \text{if } -y_i w^\top x_i < 0 \\ -y_i x_i & \text{if } -y_i w^\top x_i > 0 \\ \left[0, -y_i x_i \right] & \text{if } -y_i w^\top x_i = 0 \end{array}  \right. $$

```python
def change_learning_params(reg, eta0, n_iter, batch_size):
    classifier = Perceptron(X_train, Y_train)
    classifier.load_test_data(X_test, Y_test)

    regularizer = L2Regularizer(np.power(10., reg))
    np.random.seed(42)
    w0 = np.random.randn(3, )

    opts = {'eta0': eta0,
            'n_iter': n_iter,
            'batch_size': batch_size,
            'n_samples': X_train.shape[0],
            'algorithm': 'SGD',
            'learning_rate_scheduling': None,
            }
    trajectory, indexes = gradient_descent(w0, classifier, regularizer, opts)

    contour_plot = plt.subplot(121)
    error_plot = plt.subplot(122)

    opt = {'marker': 'ro', 'fillstyle': 'full', 'label': '+ Train', 'size': 8}
    plot_helpers.plot_data(X_train[np.where(Y_train == 1)[0], 0], X_train[np.where(Y_train == 1)[0], 1], fig=contour_plot, options=opt)
    opt = {'marker': 'bs', 'fillstyle': 'full', 'label': '- Train', 'size': 8}
    plot_helpers.plot_data(X_train[np.where(Y_train == -1)[0], 0], X_train[np.where(Y_train == -1)[0], 1], fig=contour_plot, options=opt)

    opt = {'marker': 'ro', 'fillstyle': 'none', 'label': '+ Test', 'size': 8}
    plot_helpers.plot_data(X_test[np.where(Y_test == 1)[0], 0], X_test[np.where(Y_test == 1)[0], 1], fig=contour_plot, options=opt)
    opt = {'marker': 'bs', 'fillstyle': 'none', 'label': '- Test', 'size': 8}
    plot_helpers.plot_data(X_test[np.where(Y_test == -1)[0], 0], X_test[np.where(Y_test == -1)[0], 1], fig=contour_plot, options=opt)

    contour_opts = {'n_points': 50, 'x_label': '$x$', 'y_label': '$y$', 'sgd_point': True, 'n_classes': 4}
    error_opts = {'epoch': 5, 'x_label': '$t$', 'y_label': 'error'}

    opts = {'contour_opts': contour_opts, 'error_opts': error_opts}
    plot_helpers.classification_progression(X, Y, trajectory, indexes, classifier,
                                            contour_plot=contour_plot, error_plot=error_plot,
                                            options=opts)

interact_manual(change_learning_params,
                reg=ipywidgets.FloatSlider(value=-3,
                                    min=-3,
                                    max=3,
                                    step=0.5,
                                    readout_format='.1f',
                                    description='Regularization 10^:',
                                    style={'description_width': 'initial'},
                                    continuous_update=False),
                eta0=ipywidgets.FloatSlider(value=1,
                                            min=1e-1,
                                            max=2,
                                            step=1 * 1e-1,
                                            readout_format='.1f',
                                            description='Learning rate:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False),
                n_iter=ipywidgets.IntSlider(value=30,
                                            min=5,
                                            max=100,
                                            step=1,
                                            description='Number of iterations:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False),
                batch_size=ipywidgets.IntSlider(value=1,
                                            min=1,
                                            max=X_train.shape[0],
                                            step=1,
                                            description='Batch Size:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False)
               );
```

## The SVM Algorithm

The svm loss is defined as: $$L(w; X, Y) =  \sum_{i=1}^{N} L_{\text{svm}} (w; x_i, y_i) =  \sum_{i=1}^{N} \max \{ 0, 1-y_i w^\top x_i \}.$$

The loss function is continuous, but not differentialbe at $y_i w^\top x_i=0$. The subgradient, however, exists and hence (stochastic) gradient descent converges. The subgradient is:

$$ \partial L_{\text{svm}}(w;x_i,y_i) = \left\{\begin{array}{cc} 0 & \text{if } 1-y_i w^\top x_i < 0 \\ -y_i x_i & \text{if } 1-y_i w^\top x_i > 0 \\ \left[0, -y_i x_i \right] & \text{if } 1-y_i w^\top x_i = 0 \end{array}  \right. $$

The difference with the perceptron loss is that the SVM loss includes a loss margin.

```python
def change_learning_params(reg, eta0, n_iter, batch_size):
    classifier = SVM(X_train, Y_train)
    classifier.load_test_data(X_test, Y_test)

    regularizer = L2Regularizer(np.power(10., reg))
    np.random.seed(42)
    w0 = np.random.randn(3, )

    opts = {'eta0': eta0,
            'n_iter': n_iter,
            'batch_size': batch_size,
            'n_samples': X_train.shape[0],
            'algorithm': 'SGD',
            'learning_rate_scheduling': None
            }

    trajectory, indexes = gradient_descent(w0, classifier, regularizer, opts)

    contour_plot = plt.subplot(121)
    error_plot = plt.subplot(122)

    opt = {'marker': 'ro', 'fillstyle': 'full', 'label': '+ Train', 'size': 8}
    plot_helpers.plot_data(X_train[np.where(Y_train == 1)[0], 0], X_train[np.where(Y_train == 1)[0], 1], fig=contour_plot, options=opt)
    opt = {'marker': 'bs', 'fillstyle': 'full', 'label': '- Train', 'size': 8}
    plot_helpers.plot_data(X_train[np.where(Y_train == -1)[0], 0], X_train[np.where(Y_train == -1)[0], 1], fig=contour_plot, options=opt)

    opt = {'marker': 'ro', 'fillstyle': 'none', 'label': '+ Test', 'size': 8}
    plot_helpers.plot_data(X_test[np.where(Y_test == 1)[0], 0], X_test[np.where(Y_test == 1)[0], 1], fig=contour_plot, options=opt)
    opt = {'marker': 'bs', 'fillstyle': 'none', 'label': '- Test', 'size': 8}
    plot_helpers.plot_data(X_test[np.where(Y_test == -1)[0], 0], X_test[np.where(Y_test == -1)[0], 1], fig=contour_plot, options=opt)

    contour_opts = {'n_points': 100, 'x_label': '$x$', 'y_label': '$y$', 'sgd_point': True, 'n_classes': 4}
    error_opts = {'epoch': 5, 'x_label': '$t$', 'y_label': 'error'}

    opts = {'contour_opts': contour_opts, 'error_opts': error_opts}
    plot_helpers.classification_progression(X, Y, trajectory, indexes, classifier,
                                            contour_plot=contour_plot, error_plot=error_plot,
                                            options=opts)

interact_manual(change_learning_params,
                reg=ipywidgets.FloatSlider(value=-3,
                                    min=-3,
                                    max=3,
                                    step=0.5,
                                    readout_format='.1f',
                                    description='Regularization 10^:',
                                    style={'description_width': 'initial'},
                                    continuous_update=False),
                eta0=ipywidgets.FloatSlider(value=1,
                                            min=1e-1,
                                            max=2,
                                            step=1 * 1e-1,
                                            readout_format='.1f',
                                            description='Learning rate:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False),
                n_iter=ipywidgets.IntSlider(value=50,
                                            min=5,
                                            max=100,
                                            step=1,
                                            description='Number of iterations:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False),
                batch_size=ipywidgets.IntSlider(value=1,
                                            min=1,
                                            max=X_train.shape[0],
                                            step=1,
                                            description='Batch Size:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False)
               );
```

## The Logistic Regression Algorithm

The logistic loss is defined as: $$L(w; X, Y) = \sum_{i=1}^{N} L_{\sigma}(w; x_i, y_i) = \sum_{i=1}^{N} \log \left( 1 + \exp \left(-y_i w^\top x_i \right) \right) .$$

The loss function is differentiable, hence the gradient is:

$$ \frac{\partial L_{\sigma}(w; x_i, y_i)}{\partial w} = -\frac{\exp \left(-y_i w^\top x_i \right)}{1+\exp \left(-y_i w^\top x_i \right)} y x_i$$

```python
def change_learning_params(reg, eta0, n_iter, batch_size):
    classifier = Logistic(X_train, Y_train)
    classifier.load_test_data(X_test, Y_test)

    regularizer = L2Regularizer(np.power(10., reg))
    np.random.seed(42)
    w0 = np.random.randn(3, )

    opts = {'eta0': eta0,
            'n_iter': n_iter,
            'batch_size': batch_size,
            'n_samples': X_train.shape[0],
            'algorithm': 'SGD',
            'learning_rate_scheduling': 'AdaGrad'
            }

    trajectory, indexes = gradient_descent(w0, classifier, regularizer, opts)

    contour_plot = plt.subplot(121)
    error_plot = plt.subplot(122)

    opt = {'marker': 'ro', 'fillstyle': 'full', 'label': '+ Train', 'size': 8}
    plot_helpers.plot_data(X_train[np.where(Y_train == 1)[0], 0], X_train[np.where(Y_train == 1)[0], 1], fig=contour_plot, options=opt)
    opt = {'marker': 'bs', 'fillstyle': 'full', 'label': '- Train', 'size': 8}
    plot_helpers.plot_data(X_train[np.where(Y_train == -1)[0], 0], X_train[np.where(Y_train == -1)[0], 1], fig=contour_plot, options=opt)

    opt = {'marker': 'ro', 'fillstyle': 'none', 'label': '+ Test', 'size': 8}
    plot_helpers.plot_data(X_test[np.where(Y_test == 1)[0], 0], X_test[np.where(Y_test == 1)[0], 1], fig=contour_plot, options=opt)
    opt = {'marker': 'bs', 'fillstyle': 'none', 'label': '- Test', 'size': 8}
    plot_helpers.plot_data(X_test[np.where(Y_test == -1)[0], 0], X_test[np.where(Y_test == -1)[0], 1], fig=contour_plot, options=opt)

    contour_opts = {'n_points': 100, 'x_label': '$x$', 'y_label': '$y$', 'sgd_point': True, 'n_classes': 4}
    error_opts = {'epoch': 5, 'x_label': '$t$', 'y_label': 'error'}

    opts = {'contour_opts': contour_opts, 'error_opts': error_opts}
    plot_helpers.classification_progression(X, Y, trajectory, indexes, classifier,
                                            contour_plot=contour_plot, error_plot=error_plot,
                                            options=opts)

interact_manual(change_learning_params,
                reg=ipywidgets.FloatSlider(value=-3,
                                    min=-3,
                                    max=3,
                                    step=0.5,
                                    readout_format='.1f',
                                    description='Regularization 10^:',
                                    style={'description_width': 'initial'},
                                    continuous_update=False),
                eta0=ipywidgets.FloatSlider(value=0.5,
                                            min=1e-1,
                                            max=2,
                                            step=1 * 1e-1,
                                            readout_format='.1f',
                                            description='Learning rate:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False),
                n_iter=ipywidgets.IntSlider(value=100,
                                            min=5,
                                            max=50,
                                            step=1,
                                            description='Number of iterations:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False),
                batch_size=ipywidgets.IntSlider(value=1,
                                            min=1,
                                            max=X_train.shape[0],
                                            step=1,
                                            description='Batch Size:',
                                            style={'description_width': 'initial'},
                                            continuous_update=False)
               );
```

# L1 Regularization

The L-1 regularization method uses a regularizer of the form $R(w) = ||w||_1$ which is non-differentiable. However, the subgradient exits and is:

$$\partial(||w|||_1) = \left\{\begin{array}{cc} \text{sign} (w)& \text{if } w \neq 0 \\ [-1, 1]  & \text{if } w = 0 \end{array} \right. $$

This regularization method penalizes weights and induces sparsity in the solutions. That is, most of the entries of the solution $w^\star$ will be zero.

```python
num_positive = 50  # Number of points per class
num_negative = 50  # Number of points per class

noise = 0.3  # Noise Level (needed for data generation).

X, Y = generate_linear_separable_data(num_positive, num_negative, offset=np.array([1, .2]), noise=noise, dim=2)
X = X - np.mean(X, axis=0)

def regularization(regularizer, reg):
    np.random.seed(42)
    classifier = SGDClassifier(loss='hinge', penalty=regularizer, alpha = np.power(10., reg), random_state=1)
    classifier.fit(X[:,:2], Y)

    X0, X1 = X[:, 0], X[:, 1]
    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    h = .02
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

    fig = plt.subplot(111)
    contour = plot_helpers.plot_contours(fig, classifier, xx, yy, cmap=plt.cm.jet, alpha=0.3)
    plt.colorbar(contour)
    opt = {'marker': 'r*', 'label': '+'}
    plot_helpers.plot_data(X[np.where(Y == 1)[0], 0], X[np.where(Y == 1)[0], 1], fig=fig, options=opt)
    opt = {'marker': 'bs', 'label': '-', 'legend': True}
    plot_helpers.plot_data(X[np.where(Y == -1)[0], 0], X[np.where(Y == -1)[0], 1], fig=fig, options=opt)

interact(regularization,
         regularizer=ipywidgets.RadioButtons(
             options=['l1', 'l2'],
             value='l1',
             description='Algorithm:',
             style={'description_width': 'initial'}),
         reg=ipywidgets.FloatSlider(
             value=-3,
             min=-3,
             max=0,
             step=0.5,
             description='Regularizer 10^:',
             style={'description_width': 'initial'},
             continuous_update=False)
         );
```

# Cost Sensitivity

When there are more positive than negative examples the then, to overcome this uneven distribution of data, the cost of mislabeling can be changed between classes.

The cost can be expressed as:  $$ w^\star = \arg \min_w \sum_{i=1}^{N} c(y_i) \left[y_i\neq \text{sign} (w^\top x_i) \right], $$
where $c(y_i)$ is a cost that depends on the class.

```python
num_positive = 20  # Number of points per class
num_negative = 100  # Number of points per class
noise = 0.3  # Noise Level (needed for data generation).

X, Y = generate_linear_separable_data(num_positive, num_negative, noise=noise, dim=2)
from sklearn.linear_model import SGDClassifier
from sklearn.svm import LinearSVC

def cost_sensitivity(algorithm, ratio):
    class_weight = {-1: 1, 1:np.power(10., ratio)}
    if algorithm == 'SVM':
        classifier = LinearSVC(class_weight=class_weight)
    elif algorithm == 'Perceptron':
        classifier = SGDClassifier(loss='perceptron', random_state=1, class_weight=class_weight)

    classifier.fit(X[:,:2], Y)

    X0, X1 = X[:, 0], X[:, 1]
    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    h = .02
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

    fig = plt.subplot(111)
    plot_helpers.plot_contours(fig, classifier, xx, yy, cmap=plt.cm.jet, alpha=0.3)
    opt = {'marker': 'r*', 'label': '+'}
    plot_helpers.plot_data(X[np.where(Y == 1)[0], 0], X[np.where(Y == 1)[0], 1], fig=fig, options=opt)
    opt = {'marker': 'bs', 'label': '-'}
    plot_helpers.plot_data(X[np.where(Y == -1)[0], 0], X[np.where(Y == -1)[0], 1], fig=fig, options=opt)

interact(cost_sensitivity,
         algorithm=ipywidgets.RadioButtons(
             options=['SVM', 'Perceptron'],
             value='SVM',
             description='Algorithm:',
             style={'description_width': 'initial'}),
         ratio=ipywidgets.FloatSlider(
             value=0,
             min=-1,
             max=3,
             step=0.5,
             description='Cost Ratio 10^:',
             style={'description_width': 'initial'},
             continuous_update=False)
         );
```

# Multiclass Classification

In multiclass classification there are two strategies: OvO (One vs One) or OvR (One vs The Rest).

```python
from sklearn.multiclass import OneVsOneClassifier, OneVsRestClassifier
from sklearn import datasets
from sklearn.svm import LinearSVC
from sklearn.linear_model import SGDClassifier

np.random.seed(1)
iris = datasets.load_iris()
X, y = iris.data[:,:2], iris.target
mean = X.mean(axis=0)
std = X.std(axis=0)
X = (X - mean) / std

def multiclass(strategy):
    if strategy == 'OvO':
        classifier =  OneVsOneClassifier(SGDClassifier(loss='perceptron', alpha=0.001, n_iter=100, random_state=0))
#         classifier =  OneVsOneClassifier(LinearSVC(random_state=0))
        colors = ['g', 'r', 'b']
    elif strategy == 'OvR' or strategy == 'OvA':
        classifier =  OneVsRestClassifier(SGDClassifier(loss='perceptron', alpha=0.001, n_iter=100, random_state=0))
#         classifier =  OneVsRestClassifier(LinearSVC(random_state=0))
        colors = ['b', 'g', 'r']

    classifier.fit(X, y)
    X0, X1 = X[:, 0], X[:, 1]
    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    h = .02
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

    fig = plt.subplot(111)
    plot_helpers.plot_contours(fig, classifier, xx, yy, cmap=plt.cm.jet, alpha=0.3)

    opt = {'marker': 'bs', 'label': '-', 'x_label': '$x$', 'y_label': '$y$', 'size': 8}
    plot_helpers.plot_data(X[np.where(y == 0)[0], 0], X[np.where(y == 0)[0], 1], fig=fig, options=opt)
    opt = {'marker': 'g*', 'label': '+', 'size': 8}
    plot_helpers.plot_data(X[np.where(y == 1)[0], 0], X[np.where(y == 1)[0], 1], fig=fig, options=opt)
    opt = {'marker': 'ro', 'label': '-', 'x_label': '$x$', 'y_label': '$y$', 'size': 8, 'legend': True}
    plot_helpers.plot_data(X[np.where(y == 2)[0], 0], X[np.where(y == 2)[0], 1], fig=fig, options=opt)

    def plot_hyperplane(clf, color):
        coef = clf.coef_
        intercept = clf.intercept_
        def line(x0):
            return (-(x0 * coef[0, 0]) - intercept[0]) / coef[0, 1]

        plt.plot([x_min, x_max], [line(x_min), line(x_max)],
                 ls="--", color=color)

    for estimator, color in zip(classifier.estimators_, colors):
        plot_hyperplane(estimator, color)

    fig.set_xlim([x_min, x_max])
    fig.set_ylim([y_min, y_max]);

interact(multiclass, strategy=['OvO', 'OvA']);
```

```python

```


# SIRD: A Epidemic Model with Social Distancing
**Prof. Tony Saad (www.tsaad.net)
Department of Chemical Engineering
University of Utah**

```python
#HIDDEN
from routines import plot_sird_model
import ipywidgets as widgets
from ipywidgets import interact, interact_manual
%matplotlib inline
%config InlineBackend.figure_formats = ['svg']

continuousUpdate = False

style = {'description_width': 'initial'}
beta = widgets.BoundedFloatText(value=0.5,min=0,max=1,step=0.01,description='Infection rate ($\\beta$):',continuous_update=continuousUpdate, style=style)
days = widgets.BoundedFloatText(value=14,min=1,max=20,step=1, description='Recovery (days):' ,style=style,continuous_update=continuousUpdate)
δ1 = widgets.BoundedFloatText(value=0.1,min=0,max=20.0,step=0.01,description='D to S ($\delta_1$):',readout_format='.3f',style=style,continuous_update=continuousUpdate)
δ2 = widgets.BoundedFloatText(value=0.15,min=0,max=20.0,step=0.01,description='S to D ($\delta_2$):',readout_format='.3f',style=style,continuous_update=continuousUpdate)
months = widgets.BoundedFloatText(value=12,min=1,max=60,step=1, description='Simulation (months):',style=style,continuous_update=continuousUpdate)
vaccinateAfter = widgets.BoundedFloatText(value=8,min=1,description='Vaccine found (months):',max=64,style=style,continuous_update=continuousUpdate)

minI = widgets.BoundedFloatText(value=1.0,min=0.0,max=50.0,step=0.001,description='Min Infected (%):',readout_format='.3f',style=style,continuous_update=continuousUpdate, disabled=True)
maxI = widgets.BoundedFloatText(value=10.0,min=0,max=50.0,step=0.001,description='Max Infected (%):',readout_format='.3f',style=style,continuous_update=continuousUpdate, disabled=True)

semilogy = widgets.Checkbox(value=False,description='semilogy plot:',style=style)

# widgets.link((maxI, 'value'), (minI, 'max'))
def on_value_change(change):
    minI.max = maxI.value*0.99
maxI.observe(on_value_change)

distanceModel = widgets.Dropdown(options=[('Constant'), ('Reactive')],description='Social Distancing Model:',style=style)
def show_max_min(change):
    if distanceModel.value == 'Constant':
        maxI.disabled=True
        minI.disabled=True
    else:
        maxI.disabled=False
        minI.disabled=False
distanceModel.observe(show_max_min)

ui1 = widgets.HBox([beta,months])
ui2 = widgets.HBox([days,vaccinateAfter])
ui3 = widgets.HBox([δ1,δ2])
ui4 = widgets.HBox([minI,maxI])
ui5 = widgets.HBox([distanceModel, semilogy])

out = widgets.interactive_output(plot_sird_model, {'infection_rate': beta, 'incubation_period': days, 'D_to_S': δ1, 'S_to_D': δ2, 'tend_months':months, 'vaccinateAfter': vaccinateAfter,'minIpercent':minI, 'maxIpercent':maxI,'distanceModel':distanceModel,'semilogy':semilogy})

display(ui1,ui2,ui3,ui4, ui5, out)
```

    HBox(children=(BoundedFloatText(value=0.5, description='Infection rate ($\\beta$):', max=1.0, step=0.01, style…

    HBox(children=(BoundedFloatText(value=14.0, description='Recovery (days):', max=20.0, min=1.0, step=1.0, style…

    HBox(children=(BoundedFloatText(value=0.1, description='D to S ($\\delta_1$):', max=20.0, step=0.01, style=Des…

    HBox(children=(BoundedFloatText(value=1.0, description='Min Infected (%):', disabled=True, max=50.0, step=0.00…

    HBox(children=(Dropdown(description='Social Distancing Model:', options=('Constant', 'Reactive'), style=Descri…

    Output()

# Description
SIRD is an epidemic model for infectious disease and is a variation of the SIR model with the addition of social distancing. A compartmental diagram is shown below:

The governing equations are:
\begin{equation}
\frac{\text{d}S}{\text{d}t} = -\beta SI - \delta_1 S + \delta_2 D
\end{equation}
\begin{equation}
\frac{\text{d}I}{\text{d}t} = \beta SI - \gamma I
\end{equation}
\begin{equation}
\frac{\text{d}R}{\text{d}t} =\gamma I
\end{equation}
\begin{equation}
\frac{\text{d}D}{\text{d}t} =\delta_1 S - \delta_2 D
\end{equation}
where:
* $\beta$ is the average infection rate
* $\gamma$ is the fraction of people recovering, and is equal to $1/d$ where $d$ is the average number of days for recovery
* $\delta_1$ is the fraction at which people move from being socially distanced back to the general population
* $\delta_2$ is the average fraction at which people become socially distant

The goal of this model is to better understanding the effectiveness of social distancing on "flattening" the curve of an infectious disease. For example, is full lock-down better than a step-by-step implementation of social distancing?

## Social Distancing Models
Two social distancing models are supported on this page, Constant and Reactive.

### Constant Social Distancing
Assumes that both $\delta_1$ and $\delta_2$ are constant

### Reactive Social Distancing
This Model triggers social distancing when the # of infected reaches a certain percentage of the population (designated as Max Infected in the GUI, $I_\text{max}$) and relaxes social distancing when it reaches a minimum # of infected (designated as Min Infected in the GUI, $I_\text{min}$). This produces very interesting dynamics especially as $I_\text{max} \to I_\text{min}$

## Slides
You can see a more detailed description in the following slides.