###### Copyright (c)2015 DiGangi, C.

# Managing Epidemics Through Mathematical Modeling

This lesson will examine the spread of an epidemic over time using Euler's method. The model is a system of non-linear ODEs which is based on the classic Susceptible, Infected, Recovered (SIR) model. This model introduces a new parameter  to include vaccinations. We will examine the various paremeters of the model and define conditions necessary to erradicate the epidemic.

In this module we will also introduce ipywigets, an IPython library that allows you to add widgets to your notebooks and make them interactive! We will be using widgets to vary our parameters and see how changing different parameters affects the results of the model. This is a great technique for making quick and easy comparisons because you don't have to re-run your cell for the widget to make changes to the graph.

## Introducing Model Parameters

The most important part of understanding any model is understanding the nomenclature that is associated with it. Please review the below terms carefully and make sure you understand what each parameter represents.

$S$: Susceptible Individuals

$V$: Vaccinated Individuals

$I$: Infected Individuals

$R$: Recovered Individuals with Immunity (Cannot get infected again)

$p$: Fraction of individuals who are vaccinated at birth

$e$: Fraction of the vaccinated individuals that are successfully vaccinated

$\mu$: Average Death Rate

$\beta$: Contact Rate (Rate at which Susceptibles come into contact with Infected)

$\gamma$: Recovery Rate

$R_0$: Basic Reporoduction Number

$N$: Total Population ($S + V + I + R$)

## Basic SVIR Model

## Model Assumptions
The model will make the following assumptions:

- The population N is held constant
- The birth rate and death rate are equal
- The death rate is the same across all individuals (Infected do not have higher death rate)
- A susceptible individual that comes in contact with an infected automatically becomes infected
- Once an individual has recovered they are forever immune and not reintroduced into the susceptible population
- Vaccination does not wear off (vaccinated cannot become infected)

### Susceptible Equation
Let's examine the model by component. First we will breakdown the equation for susceptible individuals. In order to find the rate of change of susceptible individuals we must calculate the number of newborns that are not vaccinated:

$$(1-ep) \mu N$$

The number of Susceptible Individuals that become infected:

$$ \beta IS_{infections}$$

and finally the number of Susceptibles that die:

$$ \mu S_{deaths}$$

Therefore the change in Susceptible Indivduals becomes:

$$\frac{dS}{dt} = (1-ep) \mu N - \beta IS - \mu S$$

### Vaccinated Equation

Now examining the vaccinated individuals we start with the newborns that are vaccinated:

$$ep \mu N$$

And the number of vaccinated individuals that die:

$$\mu V$$

The change in vaccinated individuals becomes:

$$\frac{dV}{dt} = ep \mu N - \mu V$$

### Infected Equation

For the infected individuals we start with the number of Susceptible individuals that are exposed and become infected:

$$\beta IS_{infections}$$

Next we need the number of Infected individuals that recovered:

$$\gamma I_{recoveries}$$

Finally we examine the infected who die:

$$\mu I_{deaths}$$

Putting this all together we get the following equation:

$$\frac{dI}{dt} = \beta IS - \gamma I - \mu I$$

### Recovered Equation

The number of recovered individuals first relies on the infected who recover:

$$\gamma I$$

Next it depeds on the recovered individuals who die:

$$\mu R$$

Putting this together yeilds the equation:

$$\frac{dR}{dt} = \gamma I - \mu R$$

## Model Summary

The complete model is as follows:

$$\frac{dS}{dt} = (1-ep) \mu N - \beta IS - \mu S$$

$$\frac{dV}{dt} = ep \mu N - \mu V$$

$$\frac{dI}{dt} = \beta IS - \gamma I - \mu I$$

$$\frac{dR}{dt} = \gamma I - \mu R$$

This is a very simplified model because of the complexities of infectious diseases.

## Implementing Numerical Solution with Euler!

For the numerical solution we will be using Euler's method since we are only dealing with time derivatives. Just to review, for Euler's method we replace the time derivative by the following:

$$\frac{dS}{dt} = \frac{S^{n+1} - S^n}{\Delta t}$$

where n represents the discretized time.

Therefore after we discretize our model we have:

$$\frac{S^{n+1} - S^n}{\Delta t} = (1-ep) \mu N - \beta IS^n - \mu S^n$$

$$\frac{V^{n+1} - V^n}{\Delta t} = ep \mu N - \mu V^n$$

$$\frac{I^{n+1} - I^n}{\Delta t} = \beta I^nS^n - \gamma I^n - \mu I^n$$

$$\frac{R^{n+1} - R^n}{\Delta t} = \gamma I^n - \mu R^n$$

And now solving for the value at the next time step yeilds:

$$S^{n+1}  = S^n + \Delta t \left((1-ep) \mu N - \beta IS^n - \mu S^n \right)$$

$$V^{n+1} = V^n +  \Delta t ( ep \mu N - \mu V^n)$$

$$I^{n+1} = I^n + \Delta t (\beta I^nS^n - \gamma I^n - \mu I^n)$$

$$R^{n+1}  = R^n + \Delta t ( \gamma I^n - \mu R^n)$$

If we want to implement this into our code we can build arrays to hold our system of equations. Assuming u is our solution matrix and f(u) is our right hand side:

\begin{align}
u & = \begin{pmatrix} S \\ V \\ I \\ R \end{pmatrix} & f(u) & = \begin{pmatrix} S^n + \Delta t \left((1-ep) \mu N - \beta IS^n - \mu S^n \right) \\ V^n +  \Delta t ( ep \mu N - \mu V^n) \\ I^n + \Delta t (\beta I^nS^n - \gamma I^n - \mu I^n) \\ R^n + \Delta t ( \gamma I^n - \mu R^n) \end{pmatrix}.
\end{align}

## Solve!

Now we will implement this solution below. First we will import the necessary python libraries

```python
%matplotlib inline
import numpy
from matplotlib import pyplot
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
```

Let us first define our function $f(u)$ that will calculate the right hand side of our model. We will pass in the array $u$ which contains our different populations and set them individually in the function:

```python
def f(u):
    """Returns the right-hand side of the epidemic model equations.

    Parameters
    ----------
    u : array of float
        array containing the solution at time n.
        u is passed in and distributed to the different components by calling the individual value in u[i]

    Returns
    -------
    du/dt : array of float
        array containing the RHS given u.
    """

    S = u[0]
    V = u[1]
    I = u[2]
    R = u[3]

    return numpy.array([(1-e*p)*mu*N - beta*I*S - mu*S,
                       e*p*mu*N - mu*V,
                       beta*I*S - gamma*I - mu*I,
                       gamma*I - mu*R])
```

Next we will define the euler solution as a function so that we can call it as we iterate through time.

```python
def euler_step(u, f, dt):

    """Returns the solution at the next time-step using Euler's method.

    Parameters
    ----------
    u : array of float
        solution at the previous time-step.
    f : function
        function to compute the right hand-side of the system of equation.
    dt : float
        time-increment.

    Returns
    -------
        approximate solution at the next time step.
    """

    return u + dt * f(u)
```

Now we are ready to set up our initial conditions and solve! We will use a simplified population to start with.

```python
e = .1 #vaccination success rate
p = .75 # newborn vaccination rate
mu = .02 # death rate
beta = .002 # contact rate
gamma = .5 # Recovery rate

S0 = 100 # Initial Susceptibles
V0 = 50 # Initial Vaccinated
I0 = 75 # Initial Infected
R0 = 10 # Initial Recovered

N = S0 + I0 + R0 + V0 #Total population (remains constant)
```

Now we will implement our discretization using a for loop to iterate over time. We create a numpy array $u$ that will hold all of our values at each time step for each component (SVIR). We will use dt of 1 to represent 1 day and iterate over 365 days.

```python
T = 365 # Iterate over 1 year
dt = 1 # 1 day
N = int(T/dt)+1 # Total number of iterations
t = numpy.linspace(0, T, N) # Time discretization

u = numpy.zeros((N,4)) # Initialize the solution array with zero values
u[0] = [S0, V0, I0, R0] # Set the initial conditions in the solution array

for n in range(N-1): # Loop through time steps
    u[n+1] = euler_step(u[n], f, dt) # Get the value for the next time step using our euler_step function

```

Now we use python's pyplot library to plot all of our results on the same graph:

```python
pyplot.figure(figsize=(15,5))
pyplot.grid(True)
pyplot.xlabel(r'time', fontsize=18)
pyplot.ylabel(r'population', fontsize=18)
pyplot.xlim(0, 500)
pyplot.title('Population of SVIR model over time', fontsize=18)
pyplot.plot(t,u[:,0], color= 'red', lw=2, label = 'Susceptible');
pyplot.plot(t,u[:,1], color='green', lw=2, label = 'Vaccinated');
pyplot.plot(t,u[:,2], color='black', lw=2, label = 'Infected');
pyplot.plot(t,u[:,3], color='blue', lw=2, label = 'Recovered');
pyplot.legend();
```

The graph is interesting because it exhibits some oscillating behavior. You can see that under the given parameters, the number of infected people drops within the first few days. Notice that the susceptible individuals grow until about 180 days. The return of infection is a result of too many susceptible people in the population. The number of infected looks like it goes to zero but it never quite reaches zero. Therfore, when we have $\beta IS$, when $S$ gets large enough the infection will start to be reintroduced into the population.

If we want to examine how the population changes under new conditions, we could re-run the below cell with new parameters:

```python
#Changing the following parameters
e = .5 #vaccination success rate
gamma = .1 # Recovery rate

S0 = 100 # Initial Susceptibles
V0 = 50 # Initial Vaccinated
I0 = 75 # Initial Infected
R0 = 10 # Initial Recovered

N = S0 + I0 + R0 + V0 #Total population (remains constant)

T = 365 # Iterate over 1 year
dt = 1 # 1 day
N = int(T/dt)+1 # Total number of iterations
t = numpy.linspace(0, T, N) # Time discretization

u = numpy.zeros((N,4)) # Initialize the solution array with zero values
u[0] = [S0, V0, I0, R0] # Set the initial conditions in the solution array

for n in range(N-1): # Loop through time steps
    u[n+1] = euler_step(u[n], f, dt) # Get the value for the next time step using our euler_step function

pyplot.figure(figsize=(15,5))
pyplot.grid(True)
pyplot.xlabel(r'time', fontsize=18)
pyplot.ylabel(r'population', fontsize=18)
pyplot.xlim(0, 500)
pyplot.title('Population of SVIR model over time', fontsize=18)
pyplot.plot(t,u[:,0], color= 'red', lw=2, label = 'Susceptible');
pyplot.plot(t,u[:,1], color='green', lw=2, label = 'Vaccinated');
pyplot.plot(t,u[:,2], color='black', lw=2, label = 'Infected');
pyplot.plot(t,u[:,3], color='blue', lw=2, label = 'Recovered');
pyplot.legend();
```

However, every time we want to examine new parameters we have to go back and change the values within the cell and re run our code. This is very cumbersome if we want to examine how different parameters affect our outcome. If only there were some solution we could implement that would allow us to change parameters on the fly without having to re-run our code...

## ipywidgets!

Well there is a solution we can implement! Using a python library called ipywidgets we can build interactive widgets into our notebook that allow for user interaction. If you do not have ipywidets installed, you can install it using conda by simply going to the terminal and typing:

conda install ipywidgets

Now we will import our desired libraries

```python
from ipywidgets import interact, HTML, FloatSlider
from IPython.display import clear_output, display
```

The below cell is a quick view of a few different interactive widgets that are available. Notice that we must define a function (in this case $z$) where we call the function $z$ and parameter $x$, where $x$ is passed into the function $z$.

```python
def z(x):
    print(x)
interact(z, x=True) # Checkbox
interact(z, x=10) # Slider
interact(z, x='text') # Text entry

```

    text

    <function __main__.z>

# Redefining the Model to Accept Parameters

In order to use ipywidgets and pass parameters in our functions we have to slightly redefine our functions to accept these changing parameters. This will ensure that we don't have to re-run any code and our graph will update as we change parameters!

We will start with our function $f$. This function uses our initial parameters $p$, $e$, $\mu$, $\beta$, and $\gamma$. Previously, we used the global definition of these variables so we didn't include them inside the function. Now we will be passing in both our array $u$ (which holds the different populations) and a new array called $init$ (which holds our initial parameters).

```python
def f(u, init):

    """Returns the right-hand side of the epidemic model equations.

    Parameters
    ----------
    u : array of float
        array containing the solution at time n.
        u is passed in and distributed to the different components by calling the individual value in u[i]
    init : array of float
        array containing the parameters for the model

    Returns
    -------
    du/dt : array of float
        array containing the RHS given u.
    """

    S = u[0]
    V = u[1]
    I = u[2]
    R = u[3]

    p = init[0]
    e = init[1]
    mu = init[2]
    beta = init[3]
    gamma = init[4]

    return numpy.array([(1-e*p)*mu*N - beta*I*S - mu*S,
                       e*p*mu*N - mu*V,
                       beta*I*S - gamma*I - mu*I,
                       gamma*I - mu*R])
```

Now we will change our $euler step$ function which calls our function $f$ to include the new $init$ array that we are passing.

```python
def euler_step(u, f, dt, init):
    return u + dt * f(u, init)
```

In order to make changes to our parameters, we will use slider widgets. Now that we have our functions set up, we will build another function which we will use to update the graph as we move our slider parameters. First we must build the sliders for each parameter. Using the FloatSlider method from ipywidgets, we can specify the min and max for our sliders and a step to increment.

Next we build the update function which will take in the values of the sliders as they change and re-plot the graph. The function follows the same logic as before with the only difference being the changing parameters.

Finally we specify the behavior of the sliders as they change values and call our update function.

```python
#Build slider for each parameter desired

pSlider = FloatSlider(description='p', min=0, max=1, step=0.1)
eSlider = FloatSlider(description='e', min=0, max=1, step=0.1)
muSlider = FloatSlider(description='mu', min=0, max=1, step=0.005)
betaSlider = FloatSlider(description='beta', min=0, max=.01, step=0.0005)
gammaSlider = FloatSlider(description='gamma', min=0, max=1, step=0.05)

#Update function will update the plotted graph every time a slider is changed

def update():

    """Returns a graph of the new results for a given slider parameter change.

    Parameters
    ----------
    p : float value of slider widget
    e : float value of slider widget
    mu : float value of slider widget
    beta : float value of slider widget
    gamma : float value of slider widget

    Returns
    -------
    Graph representing new populations
    """

    #the following parameters use slider.value to get the value of the given slider
    p = pSlider.value
    e = eSlider.value
    mu = muSlider.value
    beta = betaSlider.value
    gamma = gammaSlider.value

    #inital population
    S0 = 100
    V0 = 50
    I0 = 75
    R0 = 10

    N = S0 + I0 + R0 + V0

    #Iteration parameters
    T = 365
    dt = 1
    N = int(T/dt)+1
    t = numpy.linspace(0, T, N)

    u = numpy.zeros((N,4))
    u[0] = [S0, V0, I0, R0]

    #Array of parameters
    init = numpy.array([p,e,mu,beta,gamma])

    for n in range(N-1):
        u[n+1] = euler_step(u[n], f, dt, init)

    #Plot of population with gicen slider parameters
    pyplot.figure(figsize=(15,5))
    pyplot.grid(True)
    pyplot.xlabel(r'time', fontsize=18)
    pyplot.ylabel(r'population', fontsize=18)
    pyplot.xlim(0, 500)
    pyplot.title('Population of SVIR model over time', fontsize=18)
    pyplot.plot(t,u[:,0], color= 'red', lw=2, label = 'Susceptible');
    pyplot.plot(t,u[:,1], color='green', lw=2, label = 'Vaccinated');
    pyplot.plot(t,u[:,2], color='black', lw=2, label = 'Infected');
    pyplot.plot(t,u[:,3], color='blue', lw=2, label = 'Recovered');
    pyplot.legend();

    #Clear the output otherwise it will create a new graph every time so you will end up with multiple graphs
    clear_output(True) #This ensures it recreates the data on the initial graph

#Run the update function on slider values change
pSlider.on_trait_change(update, 'value')
eSlider.on_trait_change(update, 'value')
muSlider.on_trait_change(update, 'value')
betaSlider.on_trait_change(update, 'value')
gammaSlider.on_trait_change(update, 'value')

display(pSlider, eSlider, muSlider, betaSlider, gammaSlider) #Display sliders
update() # Run initial function
```

Notice that the graph starts with all parameters equal to zero. Unfortunately we cannot set the initial value of the slider. We can work around this using conditional statements to see if the slider values are equal to zero, then use different parameters.

Notice that as you change the parameters the graph starts to come alive! This allows you to quickly compare how different parameters affect the results of our model!

## Dig deeper?

Using the ipywidget library, create a new function that allows for user input. Using the python array of objects below, which contains various diseases and their initial parameters, have the user type in one of the disease names and return the graph corresponding to that disease! You can use the ipywidget text box to take in the value from the user and then pass that value to a function that will call out that disease from the object below!

```python
Disease = [{'name': "Ebola", 'p': 0, 'e': 0, 'mu': .04, 'beta': .005, 'gamma': 0}, \
           {'name': "Measles", 'p': .9, 'e': .9, 'mu': .02, 'beta': .002, 'gamma': .9}, \
           {'name': "Tuberculosis", 'p': .5, 'e': .2, 'mu': .06, 'beta': .001, 'gamma': .3}]

#Example

def z(x):
    print(x)
interact(z, x = 'Text')
```

    Text

    <function __main__.z>

## References

* Scherer, A. and McLean, A. "Mathematical Models of Vaccination", British Medical Bulletin Volume 62 Issue 1, 2015 Oxford University Press. [Online](http://bmb.oxfordjournals.org/content/62/1/187.long#F2)

* Barba, L., "Practical Numerical Methods with Python" George Washington University

* For a good explanation of some of the simpler models and overview of parameters, visit this [Wiki Page](https://en.wikipedia.org/wiki/Epidemic_model)

* Slider tutorial posted on [github](https://github.com/ipython/ipywidgets/issues/145)

```python
from IPython.core.display import HTML
css_file = 'numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<link href='https://fonts.googleapis.com/css?family=Source+Code+Pro' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
    margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
    font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
    font-family: "Source Code Pro";
    font-size: 90%;
}

.alert-box {
    padding:10px 10px 10px 36px;
    margin:5px;
}

.success {
    color:#666600;
    background:rgb(240,242,229);
}
</style>

```python

```


# HW 1

## Imports

```python
import sympy
import numpy
import plotly.graph_objects as go
```

## Problem 1

```python
zeta = sympy.symbols('zeta')
```

```python
lam = sympy.Matrix([
    [sympy.cosh(zeta), -sympy.sinh(zeta), 0, 0],
    [-sympy.sinh(zeta), sympy.cosh(zeta), 0, 0],
    [0, 0, 1, 0],
    [0, 0, 0, 1],
])

lam
```

$\displaystyle \left[\begin{matrix}\cosh{\left(\zeta \right)} & - \sinh{\left(\zeta \right)} & 0 & 0\\- \sinh{\left(\zeta \right)} & \cosh{\left(\zeta \right)} & 0 & 0\\0 & 0 & 1 & 0\\0 & 0 & 0 & 1\end{matrix}\right]$

```python
eta = sympy.Matrix([
    [-1, 0, 0, 0],
    [0, 1, 0, 0],
    [0, 0, 1, 0],
    [0, 0, 0, 1],
])

eta
```

$\displaystyle \left[\begin{matrix}-1 & 0 & 0 & 0\\0 & 1 & 0 & 0\\0 & 0 & 1 & 0\\0 & 0 & 0 & 1\end{matrix}\right]$

```python
sympy.simplify(lam.transpose() * eta * lam)
```

$\displaystyle \left[\begin{matrix}-1 & 0 & 0 & 0\\0 & 1 & 0 & 0\\0 & 0 & 1 & 0\\0 & 0 & 0 & 1\end{matrix}\right]$

```python
sympy.simplify(eta * lam)
```

$\displaystyle \left[\begin{matrix}- \cosh{\left(\zeta \right)} & \sinh{\left(\zeta \right)} & 0 & 0\\- \sinh{\left(\zeta \right)} & \cosh{\left(\zeta \right)} & 0 & 0\\0 & 0 & 1 & 0\\0 & 0 & 0 & 1\end{matrix}\right]$

```python
print(lam.transpose()._repr_latex_())
```

    $\displaystyle \left[\begin{matrix}\cosh{\left(\zeta \right)} & - \sinh{\left(\zeta \right)} & 0 & 0\\- \sinh{\left(\zeta \right)} & \cosh{\left(\zeta \right)} & 0 & 0\\0 & 0 & 1 & 0\\0 & 0 & 0 & 1\end{matrix}\right]$

## Problem 2

```python
def star_rad(conf_time: float, rad_0: float) -> float:
    """Surface of star radial coordinate in conformal time"""
    return 0.5 * rad_0 * (1 + numpy.cos(conf_time))

def star_time(conf_time: float, rad_0: float, mass: float = 1) -> float:
    """Surface of star time coordinate in conformal time"""
    return numpy.sqrt((rad_0 ** 3) / (8 * mass)) * (conf_time + numpy.sin(conf_time))

def a_m(rad_0: float, mass: float = 1) -> float:
    return numpy.sqrt(rad_0 ** 3 / (2 * mass))

def co_rad(conf_time: float, rad_0: float, mass: float = 1) -> float:
    return 0.5 * a_m(rad_0, mass=mass) * (1 + numpy.cos(conf_time))

def co_time(conf_time: float, rad_0: float, mass: float = 1) -> float:
    return 0.5 * a_m(rad_0, mass=mass) * (conf_time + numpy.sin(conf_time))
```

```python
conf_times = numpy.arange(0, numpy.pi, 0.01)
mass_fracs = [0.1, 0.25, 0.5, 1.0]
r_full = 5.0
r_0s = numpy.array(mass_fracs) ** (1/3) * r_full

rads = []
times = []

for r_0, mass_frac in zip(r_0s, mass_fracs):
    print(r_0)
    rads.append(star_rad(conf_times, r_0))
    times.append(star_time(conf_times, r_0, mass=mass_frac))
```

    2.3207944168063896
    3.149802624737183
    3.968502629920499
    5.0

```python
fig = go.Figure()

for r_0, r, t in zip(r_0s, rads, times):
    fig.add_trace(go.Scatter(x=r, y=t, mode='lines', name='{:.2f}'.format(r_0)))

fig.update_layout(
                  yaxis_range=[0,25],
                  xaxis_range=[0,6],
                  width=600,
                  height=600,
                  showlegend=True,
                  title_text=r'$\text{Oppenheimer Snyder }(\tau, R)$',
                  title_x=0.5,
                  xaxis_title=r'$R / M$',
                  yaxis_title=r'$\tau/M$')
fig.show()
```

```python
fig.write_image('hw1-prob2.pdf')
```

```python

```

```python

```

```python

```


### Importing packages

#### Others

```python
# Used to read the data file
import pandas as pd

# Used in the definition of the model and its initial condition
import numpy as np

# Used for plots other than those generated by the parameter estimation module
import matplotlib.pyplot as plt
%matplotlib inline
```

#### The module itself

```python
import PDEparams as pde
```

### Defining the model

We use the Fisher-KPP equation defined by
$$\begin{align}
\frac{\partial u}{\partial t} &= D \frac{\partial^2 u}{\partial x^2} + r\,u(1-u)
\end{align}$$

with parameters $D$ and $r$.

#### Boundary conditions

We use no-flux boundary conditions in the domain defined by the data, *i.e.*,

$$\frac{\partial u}{\partial x} = 0$$

at the boundaries.

#### Second-order derivatives of $u$

In practice, if we consider $u$ to be a discrete 1D vector $(u_0, u_1, ...)$, then we can obtain the central differences

$$\begin{align}
h\, u'_i &= u_{i+1}-u_{i-1}\\
h^2\, u''_i &= u_{i+1}+u_{i-1}-2u_i
\end{align}$$

where $h=dx$, $u' = \partial u/\partial x$, and $u'' = \partial^2 u/\partial x^2$. If we have a boundary condition $u'_0 = 0$, and we take an imaginary point $u_{-1}$ to the left of our domain so that

$$h^2\, u''_0 = u_{1}+u_{-1}-2u_0$$

at the boundary, we obtain from the first-order difference $u_{-1} = u_1$. Therefore, in our case

$$\begin{align}h^2u''_0 &= 2u_1 - 2u_0
\end{align}$$

```python
def Fisher(u, t, x, D, r):
    '''The input u corresponds to the current state of the system, and it's a vector of values u(x). Since
    the input is in 1D, no pre-processing is needed.

    t is the current time.

    x is the spatial grid of the model.

    D and r correspond to the unknown parameters.
    '''

    # We compute step in x
    dx = x[1]-x[0]

    # We initialise the spatial derivatives we need as empty arrays of the same shape of our variable
    dudxx = np.empty_like(u)

    # Second-order derivative

    # Left boundary. In the grid array, first element: [0]
    dudxx[0] = (2.0*u[1] - 2.0*u[0])/dx**2

    # Away from the x-boundary. In the grid array, second to penultimate elements: [1:-1]
    # np.diff(array, order) is used to compute 2nd-order differences
    dudxx[1:-1] = np.diff(u,2)/dx**2

    # Right boundary. In the grid array, last element: [-1]
    dudxx[-1] = (2.0*u[-2] - 2.0*u[-1])/dx**2

    # Adding the second term
    # We use np.maximum() to force the values of ru(1-u) to be non-negative and avoid small numerical errors

    dudt = D*dudxx + np.maximum(r*u*(1-u),0)

    return dudt
```

Here we specify the function that defines the initial condition for $u$ which, when generating the data, has been set to $u=1$ for $x<-0.3$ and $u=0$ otherwise.

```python
def initfunc(x):
    return 1.*int(x<-0.3)
```

### Using `PDEparams` to estimate parameters

First, we load the data from the `.csv` file.

The data consist of 3 replicates, and have been generated using parameter values $D=0.1$, $r=5$.

The columns are, in order: $t$, $x$, $u$.

```python
df = pd.read_csv('FisherData.csv')

df.head()
```

      0
      1
      2

      0
      0.075758
      -0.479798
      0.911668

      1
      0.075758
      -0.479798
      0.914364

      2
      0.075758
      -0.479798
      0.911544

      3
      0.075758
      -0.429293
      0.865805

      4
      0.075758
      -0.429293
      0.866684

#### Constructing the `PDEmodel` object.

The inputs are

**Required:**
1. The data table `df`.
2. The model `Fisher`.
3. The list of initial condition functions. In this case, a list of one element.
4. The bounds for the parameter values.

**Optional:**
1. The parameter names.
2. The number of variables: 1.
3. The number of spatial dimensions: 1.
4. The number of replicates in the data: 3. **(Default is 1, this needs to be provided in this case)**
5. The indices of the measured variables. In this case, the default `None`, since we have only one variable.
6. The function to apply to the output. In this case, the default `None`, since our data is directly $u$.

```python
my_model = pde.PDEmodel(df, Fisher, [initfunc], bounds=[(0.01, 0.2), (4,6)],
                        param_names=[r'$D$', r'$r$'], nvars=1, ndims=1, nreplicates=3, obsidx=None, outfunc=None)
```

Let us plot the initial condition.

```python
plt.plot(my_model.space, my_model.initial_condition)
plt.xlabel(r'$x$')
plt.ylabel(r'$u(x,t=0)$')
```

#### Finding the best fit parameters.

```python
%%time
my_model.fit()
```

            $D$     $r$
    0  0.094684  5.1056
    CPU times: user 19.8 s, sys: 12.1 s, total: 32 s
    Wall time: 21.5 s

```python
my_model.best_params
```

      $D$
      $r$

      0
      0.094684
      5.1056

```python
my_model.best_error
```

    3.515106675704532e-05

#### Likelihood profiles

We use the default grid of 100 points per parameter.

**Note:** if you see a "widget not found" message, just ignore it; a progress bar will appear when you run the cell below.

```python
%%time
my_model.likelihood_profiles()
```

    HBox(children=(IntProgress(value=0, description='parameters', max=2, style=ProgressStyle(description_width='in…

    HBox(children=(IntProgress(value=0, description='values within parameters', style=ProgressStyle(description_wi…

    HBox(children=(IntProgress(value=0, description='values within parameters', style=ProgressStyle(description_wi…

    CPU times: user 30min 54s, sys: 27min 55s, total: 58min 50s
    Wall time: 31min 27s

The raw results

```python
my_model.result_profiles
```

      parameter
      value
      error

      0
      $D$
      0.010000
      0.164802

      1
      $D$
      0.011919
      0.144492

      2
      $D$
      0.013838
      0.125980

      3
      $D$
      0.015758
      0.109224

      4
      $D$
      0.017677
      0.094326

      5
      $D$
      0.019596
      0.081308

      6
      $D$
      0.021515
      0.070059

      7
      $D$
      0.023434
      0.060386

      8
      $D$
      0.025354
      0.052074

      9
      $D$
      0.027273
      0.044923

      10
      $D$
      0.029192
      0.038758

      11
      $D$
      0.031111
      0.033430

      12
      $D$
      0.033030
      0.028814

      13
      $D$
      0.034949
      0.024809

      14
      $D$
      0.036869
      0.021328

      15
      $D$
      0.038788
      0.018298

      16
      $D$
      0.040707
      0.015661

      17
      $D$
      0.042626
      0.013365

      18
      $D$
      0.044545
      0.011367

      19
      $D$
      0.046465
      0.009631

      20
      $D$
      0.048384
      0.008124

      21
      $D$
      0.050303
      0.006821

      22
      $D$
      0.052222
      0.005696

      23
      $D$
      0.054141
      0.004731

      24
      $D$
      0.056061
      0.003907

      25
      $D$
      0.057980
      0.003210

      26
      $D$
      0.059899
      0.002624

      27
      $D$
      0.061818
      0.002140

      28
      $D$
      0.063737
      0.001745

      29
      $D$
      0.065657
      0.001431

      ...
      ...
      ...
      ...

      170
      $r$
      5.414141
      0.000149

      171
      $r$
      5.434343
      0.000164

      172
      $r$
      5.454545
      0.000179

      173
      $r$
      5.474747
      0.000195

      174
      $r$
      5.494949
      0.000211

      175
      $r$
      5.515152
      0.000229

      176
      $r$
      5.535354
      0.000247

      177
      $r$
      5.555556
      0.000265

      178
      $r$
      5.575758
      0.000285

      179
      $r$
      5.595960
      0.000304

      180
      $r$
      5.616162
      0.000325

      181
      $r$
      5.636364
      0.000346

      182
      $r$
      5.656566
      0.000368

      183
      $r$
      5.676768
      0.000390

      184
      $r$
      5.696970
      0.000413

      185
      $r$
      5.717172
      0.000436

      186
      $r$
      5.737374
      0.000460

      187
      $r$
      5.757576
      0.000485

      188
      $r$
      5.777778
      0.000510

      189
      $r$
      5.797980
      0.000535

      190
      $r$
      5.818182
      0.000561

      191
      $r$
      5.838384
      0.000587

      192
      $r$
      5.858586
      0.000614

      193
      $r$
      5.878788
      0.000642

      194
      $r$
      5.898990
      0.000669

      195
      $r$
      5.919192
      0.000698

      196
      $r$
      5.939394
      0.000726

      197
      $r$
      5.959596
      0.000755

      198
      $r$
      5.979798
      0.000785

      199
      $r$
      6.000000
      0.000815

200 rows × 3 columns

Visualisation

```python
my_model.plot_profiles()
```

#### Bootstrapping

We use the default 100 repetitions.

**Note:** if you see a "widget not found" message, just ignore it; a progress bar will appear when you run the cell below.

```python
%%time
my_model.bootstrap()
```

    HBox(children=(IntProgress(value=0, description='runs', style=ProgressStyle(description_width='initial')), HTM…

                  $D$         $r$
    count  100.000000  100.000000
    mean     0.094740    5.103248
    std      0.000064    0.002262
    min      0.094536    5.096778
    25%      0.094699    5.101904
    50%      0.094747    5.103201
    75%      0.094784    5.104362
    max      0.094897    5.109668
    CPU times: user 28min, sys: 28min 45s, total: 56min 45s
    Wall time: 27min 28s

The summary

```python
my_model.bootstrap_summary
```

      $D$
      $r$

      count
      100.000000
      100.000000

      mean
      0.094740
      5.103248

      std
      0.000064
      0.002262

      min
      0.094536
      5.096778

      25%
      0.094699
      5.101904

      50%
      0.094747
      5.103201

      75%
      0.094784
      5.104362

      max
      0.094897
      5.109668

The raw results

```python
my_model.bootstrap_raw
```

      $D$
      $r$

      0
      0.094666
      5.105857

      1
      0.094802
      5.101306

      2
      0.094698
      5.104337

      3
      0.094710
      5.104589

      4
      0.094795
      5.101503

      5
      0.094731
      5.103505

      6
      0.094798
      5.102691

      7
      0.094744
      5.103123

      8
      0.094751
      5.102683

      9
      0.094750
      5.103047

      10
      0.094671
      5.106643

      11
      0.094897
      5.096778

      12
      0.094746
      5.102953

      13
      0.094685
      5.104622

      14
      0.094816
      5.101104

      15
      0.094709
      5.104438

      16
      0.094745
      5.102176

      17
      0.094782
      5.103249

      18
      0.094702
      5.104978

      19
      0.094704
      5.104017

      20
      0.094798
      5.101398

      21
      0.094695
      5.104247

      22
      0.094814
      5.100049

      23
      0.094609
      5.105373

      24
      0.094806
      5.102363

      25
      0.094733
      5.103153

      26
      0.094748
      5.103949

      27
      0.094725
      5.101927

      28
      0.094800
      5.101076

      29
      0.094776
      5.103519

      ...
      ...
      ...

      70
      0.094730
      5.103557

      71
      0.094668
      5.106973

      72
      0.094536
      5.109668

      73
      0.094853
      5.099582

      74
      0.094770
      5.101361

      75
      0.094680
      5.104199

      76
      0.094795
      5.101674

      77
      0.094789
      5.102594

      78
      0.094738
      5.102674

      79
      0.094788
      5.100907

      80
      0.094756
      5.100542

      81
      0.094619
      5.106410

      82
      0.094732
      5.103821

      83
      0.094698
      5.104928

      84
      0.094759
      5.103685

      85
      0.094811
      5.101733

      86
      0.094779
      5.104111

      87
      0.094763
      5.102247

      88
      0.094774
      5.102384

      89
      0.094596
      5.107622

      90
      0.094734
      5.102828

      91
      0.094762
      5.103256

      92
      0.094718
      5.104975

      93
      0.094714
      5.103760

      94
      0.094762
      5.101073

      95
      0.094699
      5.105013

      96
      0.094811
      5.101284

      97
      0.094697
      5.104681

      98
      0.094658
      5.104336

      99
      0.094690
      5.103858

100 rows × 2 columns

Visualisation

```python
my_model.plot_bootstrap()
```

### Only a function of the variables is observed

Now we consider the case when it's not the raw outputs of the system that are observed, but rather a function of them. In this case, let us assume that the observed quantity is

$$\mathcal{F}(u) = u(1-u)$$

```python
df2 = df[df.columns[[0,1]]] # we take only the space-time coordinates

# We add the new function
df2[r'$\mathcal{F}$'] = df['2']-df['2']*df['2'] # u is column 2 in the data table

df2.head()
```

      0
      1
      $\mathcal{F}$

      0
      0.075758
      -0.479798
      0.080530

      1
      0.075758
      -0.479798
      0.078302

      2
      0.075758
      -0.479798
      0.080632

      3
      0.075758
      -0.429293
      0.116187

      4
      0.075758
      -0.429293
      0.115543

#### Constructing the `PDEmodel` object.

The inputs are the same as before, except for the data—now `df2`—and `outfunc`, which specifies the function to be applied to the outputs before computing the error.

```python
def F(z):
    return z*(1-z)
```

```python
my_model2 = pde.PDEmodel(df2, Fisher, [initfunc], bounds=[(0.01, 0.2), (4,6)],
                         param_names=[r'$D$', r'$r$'], nvars=1, ndims=1, nreplicates=3, obsidx=None, outfunc=F)
```

#### Finding the best fit parameters.

```python
%%time
my_model2.fit()
```

            $D$       $r$
    0  0.094969  5.083631
    CPU times: user 16.8 s, sys: 18.5 s, total: 35.3 s
    Wall time: 16.1 s

```python
my_model2.best_params
```

      $D$
      $r$

      0
      0.094969
      5.083631

```python
my_model2.best_error
```

    1.3675800592789573e-05

#### Likelihood profiles

**Note:** if you see a "widget not found" message, just ignore it; a progress bar will appear when you run the cell below.

```python
%%time
my_model2.likelihood_profiles()
```

    HBox(children=(IntProgress(value=0, description='parameters', max=2, style=ProgressStyle(description_width='in…

    HBox(children=(IntProgress(value=0, description='values within parameters', style=ProgressStyle(description_wi…

    HBox(children=(IntProgress(value=0, description='values within parameters', style=ProgressStyle(description_wi…

    CPU times: user 26min 57s, sys: 24min 53s, total: 51min 50s
    Wall time: 24min 40s

The raw results

```python
my_model2.result_profiles
```

      parameter
      value
      error

      0
      $D$
      0.010000
      0.014846

      1
      $D$
      0.011919
      0.014109

      2
      $D$
      0.013838
      0.013432

      3
      $D$
      0.015758
      0.012794

      4
      $D$
      0.017677
      0.012112

      5
      $D$
      0.019596
      0.011336

      6
      $D$
      0.021515
      0.010480

      7
      $D$
      0.023434
      0.009585

      8
      $D$
      0.025354
      0.008694

      9
      $D$
      0.027273
      0.007834

      10
      $D$
      0.029192
      0.007023

      11
      $D$
      0.031111
      0.006268

      12
      $D$
      0.033030
      0.005573

      13
      $D$
      0.034949
      0.004938

      14
      $D$
      0.036869
      0.004361

      15
      $D$
      0.038788
      0.003839

      16
      $D$
      0.040707
      0.003369

      17
      $D$
      0.042626
      0.002948

      18
      $D$
      0.044545
      0.002572

      19
      $D$
      0.046465
      0.002237

      20
      $D$
      0.048384
      0.001941

      21
      $D$
      0.050303
      0.001680

      22
      $D$
      0.052222
      0.001451

      23
      $D$
      0.054141
      0.001252

      24
      $D$
      0.056061
      0.001079

      25
      $D$
      0.057980
      0.000931

      26
      $D$
      0.059899
      0.000806

      27
      $D$
      0.061818
      0.000700

      28
      $D$
      0.063737
      0.000609

      29
      $D$
      0.065657
      0.000527

      ...
      ...
      ...
      ...

      170
      $r$
      5.414141
      0.000074

      171
      $r$
      5.434343
      0.000081

      172
      $r$
      5.454545
      0.000088

      173
      $r$
      5.474747
      0.000096

      174
      $r$
      5.494949
      0.000104

      175
      $r$
      5.515152
      0.000112

      176
      $r$
      5.535354
      0.000121

      177
      $r$
      5.555556
      0.000130

      178
      $r$
      5.575758
      0.000140

      179
      $r$
      5.595960
      0.000149

      180
      $r$
      5.616162
      0.000159

      181
      $r$
      5.636364
      0.000169

      182
      $r$
      5.656566
      0.000180

      183
      $r$
      5.676768
      0.000191

      184
      $r$
      5.696970
      0.000202

      185
      $r$
      5.717172
      0.000213

      186
      $r$
      5.737374
      0.000225

      187
      $r$
      5.757576
      0.000236

      188
      $r$
      5.777778
      0.000248

      189
      $r$
      5.797980
      0.000261

      190
      $r$
      5.818182
      0.000273

      191
      $r$
      5.838384
      0.000286

      192
      $r$
      5.858586
      0.000299

      193
      $r$
      5.878788
      0.000312

      194
      $r$
      5.898990
      0.000326

      195
      $r$
      5.919192
      0.000340

      196
      $r$
      5.939394
      0.000353

      197
      $r$
      5.959596
      0.000368

      198
      $r$
      5.979798
      0.000382

      199
      $r$
      6.000000
      0.000396

200 rows × 3 columns

Visualisation

```python
my_model2.plot_profiles()
```

#### Bootstrapping

**Note:** if you see a "widget not found" message, just ignore it; a progress bar will appear when you run the cell below.

```python
%%time
my_model2.bootstrap()
```

    HBox(children=(IntProgress(value=0, description='runs', style=ProgressStyle(description_width='initial')), HTM…

                  $D$         $r$
    count  100.000000  100.000000
    mean     0.095067    5.080556
    std      0.000071    0.002290
    min      0.094886    5.074651
    25%      0.095025    5.079063
    50%      0.095074    5.080528
    75%      0.095113    5.082047
    max      0.095243    5.087920
    CPU times: user 27min 1s, sys: 29min 4s, total: 56min 5s
    Wall time: 24min 23s

The summary

```python
my_model2.bootstrap_summary
```

      $D$
      $r$

      count
      100.000000
      100.000000

      mean
      0.095067
      5.080556

      std
      0.000071
      0.002290

      min
      0.094886
      5.074651

      25%
      0.095025
      5.079063

      50%
      0.095074
      5.080528

      75%
      0.095113
      5.082047

      max
      0.095243
      5.087920

The raw results

```python
my_model2.bootstrap_raw
```

      $D$
      $r$

      0
      0.095120
      5.077602

      1
      0.095122
      5.079371

      2
      0.095101
      5.080106

      3
      0.095056
      5.078753

      4
      0.094886
      5.087920

      5
      0.095074
      5.079421

      6
      0.095087
      5.080536

      7
      0.095014
      5.083736

      8
      0.095091
      5.079607

      9
      0.095077
      5.079072

      10
      0.095135
      5.079777

      11
      0.095060
      5.080693

      12
      0.095051
      5.080331

      13
      0.095084
      5.079636

      14
      0.094975
      5.082261

      15
      0.095083
      5.080486

      16
      0.095135
      5.080915

      17
      0.095015
      5.080661

      18
      0.095103
      5.081113

      19
      0.095091
      5.080658

      20
      0.095028
      5.082916

      21
      0.095185
      5.078336

      22
      0.095059
      5.077571

      23
      0.095100
      5.076753

      24
      0.095030
      5.079962

      25
      0.095078
      5.082028

      26
      0.095074
      5.080825

      27
      0.095008
      5.081758

      28
      0.095126
      5.078971

      29
      0.095120
      5.077627

      ...
      ...
      ...

      70
      0.095108
      5.081082

      71
      0.095061
      5.081164

      72
      0.095105
      5.079508

      73
      0.095085
      5.080144

      74
      0.094995
      5.082997

      75
      0.095050
      5.078992

      76
      0.095009
      5.082134

      77
      0.095075
      5.081099

      78
      0.095074
      5.079624

      79
      0.095112
      5.077622

      80
      0.095113
      5.082174

      81
      0.095076
      5.080310

      82
      0.095027
      5.082834

      83
      0.095070
      5.080591

      84
      0.094988
      5.082555

      85
      0.095041
      5.082848

      86
      0.095013
      5.083177

      87
      0.095153
      5.079036

      88
      0.095030
      5.083744

      89
      0.095110
      5.077943

      90
      0.094904
      5.085185

      91
      0.095133
      5.076794

      92
      0.094934
      5.083864

      93
      0.095031
      5.079779

      94
      0.095094
      5.081389

      95
      0.095021
      5.081919

      96
      0.094968
      5.081443

      97
      0.095046
      5.080520

      98
      0.095098
      5.078301

      99
      0.095182
      5.076560

100 rows × 2 columns

Visualisation

```python
my_model2.plot_bootstrap()
```


```python
import sympy as sp
from sympy.interactive import printing
printing.init_printing()

import symbtools as st

```

```python
%load_ext ipydex.displaytools
```

```python
xx = st.symb_vector("x1, x2") ##:

xxd = st.time_deriv(xx, xx) ##:
xxdd = st.time_deriv(xx, xx, order=2) ##:

st.make_global(xx, xxd, xxdd)
```

```python
sp.integrate(xdot1, st.t) ##:

st.smart_integrate(xdot1, st.t) ##:
```

```python
# nonrivial case
res = st.smart_integrate(x2 + 3*xdot1 + 10*xddot2, st.t) ##:
```


```python
import math
import bisect

import pint
import matplotlib.pyplot as plt
import numpy as np
import sympy

ureg = pint.UnitRegistry()
ureg.default_format = '.2f'

## Solves an expression for any variable within it
## Evaluates the expression (in the current context) with a unit coversion
def solve(exp, variable, unit):
    # Convert from string expression to symbol one
    # Solve the equaltion for the desired variable and convert to string
    # Evaluate the expression string with the desired unit conversion
    exp = sympy.sympify(exp)
    exp = sympy.sstr(sympy.solve(exp, variable)[0])
    return eval("(" + exp + ").to(ureg." + unit + ")")

## Values for each decade of 1% resistors
RESISTOR_DECADE = [
    10.0, 10.2, 10.5, 10.7, 11.0, 11.3, 11.5, 11.8, 12.1, 12.4, 12.7, 13.0, 13.3, 13.7, 14.0, 14.3,
    14.7, 15.0, 15.4, 15.8, 16.2, 16.5, 16.9, 17.4, 17.8, 18.2, 18.7, 19.1, 19.6, 20.0, 20.5, 21.0,
    21.5, 22.1, 22.6, 23.2, 23.7, 24.3, 24.9, 25.5, 26.1, 26.7, 27.4, 28.0, 28.7, 29.4, 30.1, 30.9,
    31.6, 32.4, 33.2, 34.0, 34.8, 35.7, 36.5, 37.4, 38.3, 39.2, 40.2, 41.2, 42.2, 43.2, 44.2, 45.2,
    46.4, 47.5, 48.7, 49.9, 51.1, 52.3, 53.6, 54.9, 56.2, 57.6, 59.0, 60.4, 61.9, 63.4, 64.9, 66.5,
    68.1, 69.8, 71.5, 73.2, 75.0, 76.8, 78.7, 80.6, 82.5, 84.5, 86.6, 88.7, 90.9, 93.1, 95.3, 97.6
]

## Generate a list of all 1% resistor values
RESISTOR_DECADES = [1, 10, 100, 1000, 10000, 100000, 1000000]
RESISTOR_VALUES  = sum([[val*idx for val in RESISTOR_DECADE] for idx in RESISTOR_DECADES], [])

## Binary search through resistor values for the nearest value
def select_resistor(value):
    idx = bisect.bisect_left(RESISTOR_VALUES, value.to(ureg.Ω).magnitude)
    return RESISTOR_VALUES[idx]
```

## USB Hub System Definitions

```python
Vinmax = 24 * ureg.V
Vinmin = 12 * ureg.V
Vout = 5 * ureg.V
Ilim = 5.95 * ureg.A

## Resistor divider determines operating frequency
## Rup is between FREQ and VIN, Rdn is to GND
## If FREQ connected to VIN (without a pull down resistor) then fsw will be 600 kHz

# Rup = 100 * ureg.kΩ # Recommended value from datasheet
# Rdn = 1000 * ureg.kΩ
Rdn = False
```

## MIC45208 Device Properties

#### Switching Frequency

```python
## Calculate switching frequency from Rup & Rdn
if Rdn == False:
    fsw = 600 * ureg.kHz
else:
    fsw = 600 * ureg.kHz * Rdn / (Rup + Rdn)

## Device Properties (from datasheet)
L     = 1.0 * ureg.uH    # Inductor is integrated into the package
Rdson = 16 * ureg.mΩ     # On-resistance of low-side power MOSFET
Icl   = 70 * ureg.uA     # Current-limit source current
Vcloffset = 14 * ureg.mV # Current-limit threshold
Toffmin = 200 * ureg.ns  # Minimum off time
```

## Calculations of duty cycle @ different operating voltages

```python
Ton = lambda Vop: (Vout / (Vop*fsw)).to(ureg.ns)
Tonmin = Ton(Vinmax)
Tonmax = Ton(Vinmin)
Ts     = (1/fsw).to(ureg.ns)
Dmax   = (Ts - Toffmin) / Ts

print("FSW       :", fsw.to(ureg.kHz))
print("Duty(max) :", Dmax)
print("Toff(min) :", Toffmin.to(ureg.ns))

for Vop in [Vinmin/2, Vinmin, Vinmax]:
    Toff = Ts - Ton(Vop)
    D = Ton(Vop) / Ts
    print()
    print("-- VIN", Vop, "--")
    print("   Ton  :", Ton(Vop))
    print("   Toff :", Toff)
    print("   Duty :", D)

    if Toff < Toffmin*1.25:
        print("   WARN : Toff close to or lower than Toffmin")

    if D > min(0.85, Dmax):
        print("   WARN : Tudy cycle exceeds specification max")
```

    FSW       : 600.00 kilohertz
    Duty(max) : 0.88 dimensionless
    Toff(min) : 200.00 nanosecond

    -- VIN 6.00 volt --
       Ton  : 1388.89 nanosecond
       Toff : 277.78 nanosecond
       Duty : 0.83 dimensionless

    -- VIN 12.00 volt --
       Ton  : 694.44 nanosecond
       Toff : 972.22 nanosecond
       Duty : 0.42 dimensionless

    -- VIN 24.00 volt --
       Ton  : 347.22 nanosecond
       Toff : 1319.44 nanosecond
       Duty : 0.21 dimensionless

## Component Selection
### Current Limit Resistor

Based on desired current limit and other operating points (inductor peak to peak current, MOSFET resistance, etc) the current limiting resistor between SW and ILIM is determined.

The MOSFET RDS(ON) varies 30% to 40% with temperature; therefore, it is recommended to add a 50% margin to ILIM to avoid false current limiting due to increased MOSFET junction temperature rise.  After finding the ideal Rlimit values, we contrain it to actual resistor values and calculate the resutling current limit.

```python
ILpp = Vout * (Vinmax - Vout) / (Vinmax * fsw * L)
print("Inductor peak to peak :", ILpp.to(ureg.A))

exp = "((Ilim*1.5 - ILpp/2.0)*Rdson + Vcloffset)/Icl - Rlimit"

Rlimitideal = solve(exp, variable='Rlimit', unit='Ω')
Rlimit = (select_resistor(Rlimitideal) * ureg.Ω).to(ureg.kΩ)

if Rlimit != Rlimitideal:
    print("Rlimit : changing from ", Rlimitideal.to(ureg.Ω), "to", Rlimit.to(ureg.Ω))
else:
    print("Rlimit :", Rlimit)

Ilimactual = solve(exp, variable='Ilim', unit='A')
print("Ilimit :", Ilimactual)

# Climit value copied from EVM design guide
Climit = 15 * ureg.pF
tc = Rlimit * Climit
print("Ilimit TC :", tc.to(ureg.ns))

if tc > Toffmin * 0.2:
    print("WARN : Current limit time constant too close to minimum off time")
```

    Inductor peak to peak : 6.60 ampere
    Rlimit : changing from  1486.03 ohm to 1500.00 ohm
    Ilimit : 5.99 ampere
    Ilimit TC : 22.50 nanosecond

### Input Capacitor Selection

```python
## Build dict of [Value, ESR, Impedance] of caps at 12v
CAPS = {}
CAPS["0805_1u0"] = [ 1.0*ureg.uF, 18.6, 288.0] ## C0805C105K5RAC
CAPS["1206_4u7"] = [ 4.7*ureg.uF, 13.1,  62.3] ## C1206C475K5PAC
CAPS["2917_33u"] = [33.0*ureg.uF, 29.3,  29.8] ## T521D336M035ATE065

Cins = ["2917_33u"]
# Cins = ["1206_4u7", "1206_4u7", "0805_1u0"] + ["2917_33u", "2917_33u"]

f = fsw

Zsum = 0
Ceff = 0

for key in Cins:
    cap = CAPS[key]

    Ceff += cap[0]
    esr   = cap[1]*ureg.mΩ
    imp   = cap[2]*ureg.mΩ * -1j

    ## Impedance for an idea cap. This is not accurate as bias voltage increases.
    ## Therefore, we use ESR from the SKU datasheets at 12v (listed above).
    # imp = (-1j / (2*math.pi*f*cap[0])).to(ureg.mΩ)

    Zsum += 1/(esr + imp)

ESRCin = (1 / Zsum).real
Cinsum = Ceff.to(ureg.uF)

print("Cinsum:", Cinsum)

## Current increases with duty cycle, so calculate at VIN = 12
## Load on cap increases as duty cycle drops, so calculate at VIN = 24
for vin in [Vinmin, Vinmax]:
    D = Ton(vin) / Ts
    Icinrms = Ilim * math.sqrt(D*(1-D))

    ## Inductor current changes as vin does, so we need to recalculate it
    ILpp = Vout * (vin - Vout) / (vin * fsw * L)

    ## These require knowing the ESR of the input caps
    dVin = ILpp * ESRCin
    Cinreq = Ilim * (1-D) / (fsw * dVin)
    Pcinrms = Icinrms**2 * ESRCin

    print()
    print("Vin @ {}".format(vin))
    print("Cin ")
    print("  dV    :", dVin.to(ureg.mV))
    print("  req   :", Cinreq.to(ureg.uF))
    print("  I rms :", Icinrms)
    print("  P rms :", Pcinrms.to(ureg.W))

```

    Cinsum: 33.00 microfarad

    Vin @ 12.00 volt
    Cin
      dV    : 142.43 millivolt
      req   : 40.61 microfarad
      I rms : 2.93 ampere
      P rms : 0.25 watt

    Vin @ 24.00 volt
    Cin
      dV    : 193.30 millivolt
      req   : 40.61 microfarad
      I rms : 2.42 ampere
      P rms : 0.17 watt

### Output Voltage Setting Components

A typical value of RFB1 used on the standard evaluation board is 10 kΩ. If R1 is too large, it may allow noise to be introduced into the voltage feedback loop. If RFB1 is too small in value, it will decrease the efficiency of the power supply, especially at light loads. Once RFB1 is selected, RFB2 can be calculated.

```python
Vfb  = 0.8 * ureg.V
RFB1 = 10 * ureg.kΩ

# Boost 5v output a bit to compenstate for loss in downstream cabling
Vout = Vout * 1.02

RFB2ideal = Vfb * RFB1 / (Vout - Vfb)
RFB2 = (select_resistor(RFB2ideal) * ureg.Ω).to(ureg.kΩ)

if RFB2ideal != RFB2:
    print("RFB2 : changing from ", RFB2ideal.to(ureg.Ω), "to", RFB2.to(ureg.Ω))
else:
    print("RFB2 :", RFB2)

Voutactual = Vfb * (1+ RFB1/RFB2)
print("Vout : changing from ", Vout, "to", Voutactual)
```

    RFB2 : changing from  1860.47 ohm to 1870.00 ohm
    Vout : changing from  5.10 volt to 5.08 volt


Learning poisson parameters from aggregated data
================================================

In this note, we will study the problem of learning parameters of poisson distributions from *aggregated* data. In other words, the data does not directly reflect realizations of these distributions, but rather contains aggregates (i.e. sums) of such values.

Consider the following scenario as a concrete example: A city consists of five areas: **A, B, C, D, E**. We are interested in the distribution of morning trips. We know that there exist three types of travelers:

1. *Workers*: going from home to work
2. *Shoppers*: going from home to a shopping area
3. *Kids*: going from home to school

Assume we only have the data for trips that originate from area **A**. Our data only contains the number of trips originating from **A** in the morning. We can not say anything about the purpose of trips by merely relying on the data.

Know assume that we have the following pieces of background knowledge:

1. The trips of each group follows a poisson distribution
2. For each travel type, the rates of poisson distributions for different destinations are equal.
3. We know the types of trips for each destination (They are presented in the image below).

**_The second assumption requires us to adjust for the population of areas when applying the following approach. More on this in another note._**

```
from IPython.display import SVG
SVG(filename='areas.svg')
```

Our problem consists of finding three rates: $\lambda_w$ for working trips, $\lambda_s$ for shopping trips, and $\lambda_k$ for school trips.

## Reminder: Poisson density function
Remember that the poisson distribution has the following density function:
\begin{align}
Pr(X=k) = \frac{\lambda^k e^{-\lambda}}{k!}
\end{align}

#### example
This equation implies that under the assumption $\lambda_w = 10$, the probability of having 7 trips from **A** to **E** in the morning is $$\frac{{10}^{7} * e^{-10}}{7!} = 0.09$$

The value 0.09 is the *likelihood* of observation $X_{A \to E}=7$.

In the next section we'll see an interesting property that will help us in our analysis.

## Sum of independent poisson variables
*For independent random variables $X_1,\ldots,X_n$ which follow poisson distributions with parameters $\lambda_1,\ldots,\lambda_n$, The variable $Y=\sum_1^n X_i$ follows a poisson distribution with parameter $\lambda = \sum_1^n \lambda_i$*

#### example
According to this property, under the assumption $\lambda_w = 10, \lambda_s = 3$, the trips from **A** to **B** have a poisson distribution with parameter $\lambda = 10 + 3 = 13$, and hence the probability of observing 15 trips is:

\begin{equation}
\frac{{13}^{15}*e^{-13}}{15!} = 0.088
\end{equation}

The value 0.088 is the *likelihood* of the observation $X_{A \to B} = 15$.

# Learning parameters from data

Now assume that we have the aggregated data for the morning trips originating from area **A** for a number of days:

|Destination | Day1 | Day2 | Day3 |
|---|:---:|:---:|:---:|
|*B*|99|92|78|
|*C*|7|6|10|
|*D*|119|94|98|
|*E*|89|85|81|

We have three unknowns: $\lambda_w, \lambda_s, \lambda_k$. Each of them can take a value in the range $[0, \infty)$. We want to pick values that are most consistent with the data. So we select the parameters that maximize the likelihood of data.

To compute the likelihood, we first compute the probability of each observation (for each destination-day). Assuming that *trips to different destination* and *trips of different days* are independent, the likelihood will be the product of these probabilities.

So the probability of the observation for destination **B** in day 1 is:
\begin{equation}
\frac{e^{-(\lambda_w + \lambda_s)} * {(\lambda_w + \lambda_s)}^{99}}{99!}
\end{equation}

Remember that the trips from **A** to **B** are taken by *shoppers* and *workers*:

```
from IPython.display import SVG
SVG(filename='only_b.svg')
```

As another example, consider the probability of observation for destination **C** in day 2:
\begin{equation}
\frac{e^{-\lambda_s} * {\lambda_s}^{6}}{6!}
\end{equation}

Remember that the trips from **A** to **C** are taken only by *shoppers*.

```
from IPython.display import SVG
SVG(filename='only_c.svg')
```

By the same principle, the likelihood for the whole data ($\mathcal{D}$) is computed as:

\begin{align}
L(\mathcal{D}|\lambda_w, \lambda_s, \lambda_k) =
&\frac{e^{-3*(\lambda_w + \lambda_s)} * {(\lambda_w + \lambda_s)}^{99+92+87}}{99!*92!*87!} + \\
&\frac{e^{-3*\lambda_s} * {\lambda_s}^{7+6+10}}{7!*6!*10!} + \\
&\frac{e^{-3*(\lambda_w + \lambda_k)} * {(\lambda_w + \lambda_k)}^{119+94+98}}{119!*94!*98!} + \\
&\frac{e^{-3*\lambda_w} * {\lambda_w}^{89+85+81}}{89!*85!*81!}
\end{align}

Our problem is to find the values $\lambda_w, \lambda_s, \lambda_k$ such that the likelihood is maximized:
\begin{equation}
\lambda^* = \underset{\lambda}{\operatorname{argmax}} L(\mathcal{D}|\lambda)
\end{equation}

To facilitate the computations, we can formulate the problem as that of maximizing the *logarithm of likelihood*.

\begin{align}
LL(\mathcal{D}|\lambda_w, \lambda_s, \lambda_k) \equiv &\log(L(\mathcal{D}|\lambda_w, \lambda_s, \lambda_k)) = \\
&-9 \lambda_w - 6 \lambda_s - 3 \lambda_k + \\
&278 \log (\lambda_w + \lambda_s) +
23 \log (\lambda_s) + \\
&311 \log (\lambda_w + \lambda_k) +
255 \log (\lambda_w) + \mathcal{C}
\end{align}

Since the constant $\mathcal{C}$ (emerged from the factorial terms in denominators) does not affect the optimizaiton problem, we safely drop it.

## Parameter learning as an optimization problem

We formulated our problem as an instance of continuous optimization problem:

\begin{align}
\text{maximize} &LL(\mathcal{D}|\lambda) \\
&\lambda \geq 0
\end{align}

For our example dataset, the actual optimization problem will be:

\begin{align}
\text{maximize} & -9 \lambda_w - 6 \lambda_s - 3 \lambda_k + \\
&278 \log (\lambda_w + \lambda_s) +
23 \log (\lambda_s) + \\
&311 \log (\lambda_w + \lambda_k) +
255 \log (\lambda_w) \\
&\lambda_w, \lambda_s, \lambda_k \geq 0
\end{align}

Note that increasing the number of days only changes the constant coefficients in the objective function of this optimization problem. For example, assume that the number of trips for 10 days are reported in the following table:

|Destination | Day1 | Day2 | Day3 |Day4 |Day5 |Day6 |Day7 |Day8 |Day9 |Day10 |
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|*B*|99|92|78|87 | 78 | 76 | 74 | 107 | 75 | 97 |
|*C*|7|6|10|13 | 11 | 11 | 13 | 8 | 17 | 10 |
|*D*|119|94|98|126 | 118 | 117 | 125 | 127 | 108 | 120 |
|*E*|89|85|81|74 | 79 | 68 | 78 | 75 | 80 | 68 |

Then the new optimization problem will be:

\begin{align}
\text{maximize} & -30 \lambda_w - 20 \lambda_s - 10 \lambda_k + \\
&846 \log (\lambda_w + \lambda_s) +
100 \log (\lambda_s) + \\
&1094 \log (\lambda_w + \lambda_k) +
831 \log (\lambda_w) \\
&\lambda_w, \lambda_s, \lambda_k \geq 0
\end{align}

Generalizing to the data of $n$ days is straightforward: Denote the total number of trips to **B**, **C**, **D**, and **E** by $N_b$, $N_c$, $N_d$, and $N_e$. Then the optimization problem will be:

\begin{align}
\text{maximize} & -3n \lambda_w - 2n \lambda_s - n \lambda_k + \\
&N_b \log (\lambda_w + \lambda_s) +
N_c \log (\lambda_s) + \\
&N_d \log (\lambda_w + \lambda_k) +
N_e \log (\lambda_w) \\
&\lambda_w, \lambda_s, \lambda_k \geq 0
\end{align}

Now it is time to solve this problem.

# Solving the optimization problem

In this section we try to solve the optimization problem using the 10-day data. The actual values used for sampling the data were $\lambda_w = 80$, $\lambda_s = 10$, and $\lambda_k = 30$.

```
import numpy as np
def ll_data(n, N):
    def loglikelihood(lambdas):
        lw = lambdas[0]
        ls = lambdas[1]
        lk = lambdas[2]
        res = -3*n*lw - 2*n*ls - n*lk
        res += N[0] * np.log(lw+ls) + N[1] * np.log(ls)
        res += N[2] * np.log(lw+lk) + N[3] * np.log(lw)
        return -res #reversing the sign to maximize
    return loglikelihood
```

```
from scipy.optimize import minimize
def max_loglikelihood(N, n):
    first_guess = [1, 1, 1]
    bounds = [(0, None)]*3 #lambdas should be non-negative
    res = minimize(ll_data(n, N), x0, bounds=bounds, method='L-BFGS-B')
    print 'lw:%.2f ls:%.2f lk:%.2f  --> %.2f'%(res.x[0], res.x[1], res.x[2],-res.fun)

n = 10
N = [846, 100, 1094, 831]
max_loglikelihood(N, n)
```

    lw:79.31 ls:9.54 lk:30.09  --> 9921.12

Solving the optimization problem gives us the values $\lambda_w = 79.31$, $\lambda_s = 9.54$, and $\lambda_k = 30.09$.

We can also solve the problem with the data for 100 days:

```
n = 100
N = [8979, 992, 11185, 7926]
max_loglikelihood(N, n)
```

    lw:79.53 ls:9.95 lk:32.32  --> 100996.61

The value for $\lambda_k$ (32.32) has diverged from the actual value (30). However, this is not due to the quality of solutions to the optimization problem. Replacing the actual values in the log-likelihood function evaluates to a lower result:

```
print '%.3f' % -ll_data(n, N)([80, 10, 30])
```

    100994.771

We can conclude that the information available in the 100-day data does not allow for a perfect prediction of $\lambda_k$.

The partitioning issue
========================

Let us consider another scenario. This scenario is equivalent to the previous one in all aspects, except for the composition of trips: The trips to **B** and **D** are taken by *workers* and *shoppers*, and the trips to **C** and **E** are taken by *kids*. The figure below shows the composition of trips to each area.

```
from IPython.display import SVG
SVG(filename='areas2.svg')
```

Denoting the total number of trips to **B**, **C**, **D**, and **E** by $N_b$, $N_c$, $N_d$, and $N_e$, the optimization problem for this scenario will be:

#### (version 1)
\begin{align}
\text{maximize} & -2n (\lambda_w + \lambda_s) - 2n \lambda_k + \\
&N_b \log (\lambda_w + \lambda_s) +
N_c \log (\lambda_k) + \\
&N_d \log (\lambda_w + \lambda_s) +
N_e \log (\lambda_k) \\
&\lambda_w, \lambda_s, \lambda_k \geq 0
\end{align}

As you see, the variables $\lambda_w$ and $\lambda_s$ always appear in the term $\lambda_w + \lambda_s$. One can substitute $\lambda_{ws}$ for $\lambda_w + \lambda_s$ and re-write the problem as:

#### (version 2)
\begin{align}
\text{maximize} & -2n (\lambda_{ws}) - 2n \lambda_k + \\
&N_b \log (\lambda_{ws}) +
N_c \log (\lambda_k) + \\
&N_d \log (\lambda_{ws}) +
N_e \log (\lambda_k) \\
&\lambda_{ws}, \lambda_k \geq 0
\end{align}

The problem is that, once $\lambda_{ws}$ is obtained, each partitioning of $\lambda_{ws}$ into $\lambda_w$ and $\lambda_s = 1 - \lambda_w$ gives the same likelihood. This means that the values $\lambda_w$ and $\lambda_s$ obtained by solving the first version of the optimization problem are arbitrary conditioned on their sum.

Let us check this by solving the first version:

```
def ll_data(n, N):
    def loglikelihood(lambdas):
        lw = lambdas[0]
        ls = lambdas[1]
        lk = lambdas[2]
        res = -2*n*lw - 2*n*ls - 2*n*lk
        res += N[0] * np.log(lw+ls) + N[1] * np.log(lk)
        res += N[2] * np.log(lw+ls) + N[3] * np.log(lk)
        return -res #reversing the sign to maximize
    return loglikelihood
```

```
n = 100
N = [8929, 2904, 9031, 2973]
max_loglikelihood(N, n)
```

    lw:44.90 ls:44.90 lk:29.39  --> 76806.73

Solving the optimization problem gives $\lambda_w = 44.9$, $\lambda_s = 44.9$, and $\lambda_k = 29.39$. However, all we know from the data about $\lambda_w$ and $\lambda_s$ is that $\lambda_w + \lambda_s = 89.8$. Any partitioning of this sum into non-negative pairs of $\lambda_w$ and $\lambda_s$ will give the same log-likelihood:

```
print '%.3f' % -ll_data(n, N)([44.9, 44.9, 29.39])
print '%.3f' % -ll_data(n, N)([89.8, 0, 29.39])
print '%.3f' % -ll_data(n, N)([0, 89.8, 29.39])
```

    76806.733
    76806.733
    76806.733

## Is this really a problem?
Tias made this important observation:
> The partionting problem does not matter in our application, as long as we are going to sample according to the same trip decomposition.

It is true! To sample from the distribution specified in the current scenario, we only need to know $\lambda_k$ and $\lambda_s + \lambda_w$. Hence the partitioning issue does not pose a practical limitation.


```python
# Standard IPython notebook imports
%matplotlib inline

import os

from io import StringIO
import itertools
import httpimport
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from tqdm import tqdm_notebook

import scipy.stats as sstats
import scipy.optimize as sopt

import seaborn as sns

import sklearn.datasets
import sklearn.ensemble
import sklearn.svm
import sklearn.tree
from sklearn.utils import resample
from sklearn.model_selection import train_test_split

import cvxopt
sns.set_style('whitegrid')
```

# SVM Theory
A linear SVM assigns points $x^{(i)}\in\mathbb{R}^n$ to one of two
classes,  $y^{(i)}\in\{-1,1\}$ using the decision rule:

\begin{equation}
y = \text{signum}(w^T x + b).
\end{equation}

SVM training consists of finding weights $w\in\mathbb{R}^n$
and bias $b\in\mathbb{R}$ that maximize the separation margin. This
corresponds to solving the following quadratic optimization problem:

\begin{equation}
\begin{split}
  \min_{w,b,\xi}  &\frac{1}{2}w^Tw  + C\sum_{i=1}^m \xi_i  \\
  \text{s.t. } & y^{(i)}(w^T x^{(i)} + b) \geq 1- \xi_i\;\; \forall_i \\
  & \xi_i \geq 0 \;\; \forall_i.
\end{split}
\end{equation}

# 1. SVM on IRIS

```python
# load iris, extract petal_length and petal_width of versicolors and virginicas

iris = sklearn.datasets.load_iris()
print('Features: ', iris.feature_names)
print('Targets: ', iris.target_names)
petal_length = iris.data[:,iris.feature_names.index('petal length (cm)')]
petal_width = iris.data[:, iris.feature_names.index('petal width (cm)')]

IrisX = np.array(iris.data.T)
IrisX = IrisX[:, iris.target!=0]

IrisX2F = np.vstack([petal_length, petal_width])
IrisX2F = IrisX2F[:, iris.target!=0]

# Set versicolor=0 and virginia=1
IrisY = (iris.target[iris.target!=0]-1).reshape(1,-1).astype(np.float64)

plt.scatter(IrisX2F[0,:], IrisX2F[1,:], c=IrisY.ravel(),
            cmap='spring', edgecolors='k')
plt.xlabel('petal_length')
plt.ylabel('petal_width')
```

## 1.1 Linear SVM

```python
svm_model = sklearn.svm.SVC(gamma='auto', kernel='linear', C=60)
svm_model.fit(IrisX2F.T, IrisY.T.ravel())

print("libsvm error rate: %f" % (svm_model.predict(IrisX2F.T)!=IrisY).mean())
```

    libsvm error rate: 0.060000

## 1.2 Decision boundary

```python
petal_lengths, petal_widths = np.meshgrid(np.linspace(IrisX2F[0,:].min(), IrisX2F[0,:].max(), 100),
                                          np.linspace(IrisX2F[1,:].min(), IrisX2F[1,:].max(), 100))

IrisXGrid = np.vstack([petal_lengths.ravel(), petal_widths.ravel()])
predictions_Grid = svm_model.predict(IrisXGrid.T)

plt.contourf(petal_lengths, petal_widths, predictions_Grid.reshape(petal_lengths.shape), cmap='spring')

plt.scatter(IrisX2F[0,:], IrisX2F[1,:], c=IrisY.ravel(),
            cmap='spring', edgecolors='k')

plt.xlabel('petal_length')
plt.ylabel('petal_width')
plt.title('Decision boundary found by libsvm')
```

## 1.3 Plot only support vectors with size equal to their coefficients

```python
support_vector_indices = svm_model.support_
support_vector_coefficients = svm_model.dual_coef_
print(support_vector_coefficients)

plt.contourf(petal_lengths, petal_widths, predictions_Grid.reshape(petal_lengths.shape), cmap='spring')
plt.scatter(IrisX2F[0,support_vector_indices], IrisX2F[1,support_vector_indices],
            c=IrisY.ravel()[support_vector_indices],
            s=abs(support_vector_coefficients),
            cmap='spring',
            edgecolors='k')

plt.xlabel('petal_length')
plt.ylabel('petal_width')
plt.title('Decision boundary found by libsvm')
```

## 1.4 Nonlinear SVM with Gaussian kernel

```python
svm_gauss_model = sklearn.svm.SVC(gamma=100, kernel='rbf', C=200)
svm_gauss_model.fit(IrisX2F.T, IrisY.T.ravel())

print("libsvm error rate: %f" % (svm_gauss_model.predict(IrisX2F.T)!=IrisY).mean())

predictions_Grid = svm_gauss_model.predict(IrisXGrid.T)

support_vector_indices = svm_gauss_model.support_
support_vector_coefficients = svm_gauss_model.dual_coef_

# TODO: plot the decision boundary with support vectors
plt.contourf(petal_lengths, petal_widths, predictions_Grid.reshape(petal_lengths.shape), cmap='spring')
plt.scatter(IrisX2F[0,support_vector_indices], IrisX2F[1,support_vector_indices],
            c=IrisY.ravel()[support_vector_indices],
            s=abs(support_vector_coefficients),
            cmap='spring',
            edgecolors='k')

plt.xlabel('petal_length')
plt.ylabel('petal_width')
plt.title('Decision boundary found by libsvm')
```

```python
#
# Now solve the SVM using the QP solver
#

n, m = IrisX2F.shape
C=10.0

P = np.zeros((n+1+m, n+1+m)) #w, bias, xi
q = np.zeros((n+1+m,1))
G = np.zeros((2*m, n+1+m)) # we have two constrains for each data point:
                           # that the margin is equal to 1-xi
                           # and that xi is nonnegative
h = np.zeros((2*m,1))
#
# TODO: fill in P, q, G, h
#

TODO

#
# Now run the solver
#
ret  = cvxopt.solvers.qp(cvxopt.matrix(P), cvxopt.matrix(q),
                         cvxopt.matrix(G), cvxopt.matrix(h), )

ret = np.array(ret['x'])

#
# extract the weights and biases
#
W = ret[:n].reshape(-1,1)
b = ret[n]

#
# Extract the weight and bias from libsvm for comparison
#
Wlibsvm = TODO
blibsvm = TODO

print('W', W.T, 'Wlibsvm', Wlibsvm)
print('b', b, 'blibsvm', blibsvm)
```

         pcost       dcost       gap    pres   dres
     0: -8.2045e+03  4.6654e+03  2e+04  2e+01  3e+00
     1:  5.3426e+01 -5.8951e+02  3e+03  2e+00  4e-01
     2:  1.7058e+02 -1.9137e+01  4e+02  3e-01  5e-02
     3:  1.4063e+02  7.5166e+01  1e+02  7e-02  1e-02
     4:  1.3820e+02  9.8722e+01  6e+01  3e-02  5e-03
     5:  1.4016e+02  1.1207e+02  3e+01  4e-03  8e-04
     6:  1.3507e+02  1.1797e+02  2e+01  1e-03  2e-04
     7:  1.2551e+02  1.2356e+02  2e+00  2e-04  3e-05
     8:  1.2445e+02  1.2440e+02  5e-02  4e-06  6e-07
     9:  1.2442e+02  1.2442e+02  1e-03  9e-08  2e-08
    10:  1.2442e+02  1.2442e+02  2e-05  1e-09  1e-10
    Optimal solution found.
    W [[2.75844069 4.827271  ]] Wlibsvm [[2.75873306 4.82861713]]
    b [-21.2054472] blibsvm [-21.20914737]

# Problem 3 [2p]

Repeat 100 bootstrap experiments to establish the effect of constant $C$ on SVM.

For each experiment do the following:

1. Sample (with replacement) a bootstrap dataset equal in size to the
    training dataset. This will be this experiment's training dataset.
2. Prepare the experiment's testing dataset by using samples not
    inluded in the bootstrap dataset.
3. For all $C$ from the set $\{10^{-4}, 10^{-3.5}, 10^{-3.}, \ldots, 10^{6}\}$
    fit a nonlinear SVM (Gaussian kernel, called \texttt{rbf} in
    LIBSVM using the default $\gamma$) and record the training and
    testing errors.

Analyze a box plot of errors as a function of $C$. Can you see its
influence on the training and testing error, as well as on the
testing error variability?

**Indicate regions of overfitting and underfitting.**

```python
res = []
iters = 100
for rep in range(iters):
    bootstrap_sel = resample(np.arange(len(IrisX.T)), n_samples = len(IrisX.T))
    test_sel = np.array([i for i in np.arange(len(IrisX.T)) if not i in bootstrap_sel])

    bootstrap_IrisX = IrisX2F[:,bootstrap_sel].T
    bootstrap_IrisY = IrisY[:,bootstrap_sel].T

    test_IrisX = IrisX2F[:,test_sel].T
    test_IrisY = IrisY[:,test_sel].T

    #
    # TODO: Loop over a list of exponents.
    #
    for Cexponent in np.arange(-4, 6.5, 0.5):
        C = 10.0**Cexponent
        svm_model = sklearn.svm.SVC(gamma='auto', kernel='rbf', C=C)
        svm_model.fit(bootstrap_IrisX, bootstrap_IrisY.ravel())
        train_acc = (svm_model.predict(bootstrap_IrisX) == bootstrap_IrisY.ravel()).mean()
        test_acc = (svm_model.predict(test_IrisX) == test_IrisY.ravel()).mean()

        # VISUALIZATION
        # predictions_Grid = svm_model.predict(IrisXGrid.T)
        # plt.contourf(petal_lengths, petal_widths, predictions_Grid.reshape(petal_lengths.shape), cmap='spring')
        # plt.scatter(bootstrap_IrisX[:, 0], bootstrap_IrisX[:, 1],
        #     c=svm_model.predict(bootstrap_IrisX),
        #     cmap='spring',
        #     edgecolors='black')
        # plt.show()
        # print(train_acc, test_acc)

        res.append(dict(Cexponent=Cexponent, err=1-test_acc, subset='test'))
        res.append(dict(Cexponent=Cexponent, err=1-train_acc, subset='train'))

res = pd.DataFrame(res)
chart = sns.catplot(kind='box', x='Cexponent', y='err', col='subset',
            color='blue', data=res)
chart.set_xticklabels(rotation=45)
```

# Problem 4 [3p bonus]

Implement a nonlinear SVM by solving the dual problem using the Quadratic Programming solver. Compare results with LIBSVM.

Please see [page 20 if CS229 lecture notes](http://cs229.stanford.edu/notes/cs229-notes3.pdf) for problem formulation.

# Problem 5 [2p bonus]
Compare two ways to implement a multi-class
SVM: by training ``1-vs-1'' classifier for each class combination,
and by training a ``1-vs-rest'' classifier for each clas. See
http://www.csie.ntu.edu.tw/\~cjlin/papers/multisvm.pdf for
details.

# Problem 6 [4p bonus]

Implement a Kernelized linear regression. Train it on Iris using a Gaussian kernel. Compare to the non-linear SVM.

```python

```


# Spectral Estimation of Random Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## The Periodogram

The [periodogram](https://en.wikipedia.org/wiki/Spectral_density_estimation#Periodogram) is an estimator for the power spectral density (PSD) $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of a random signal $x[k]$. For the following it is assumed that $x[k]$ is drawn from a wide-sense ergodic real-valued random process.

### Definition

The PSD is defined as the [discrete-time Fourier transformation (DTFT) of the auto-correlation function (ACF)](../random_signals/power_spectral_densities.ipynb#Definition)

\begin{equation}
\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \mathcal{F}_* \{ \varphi_{xx}[\kappa] \}
\end{equation}

Hence, the PSD can be computed from an estimate of the ACF. Let's assume that we want to estimate the PSD from $N$ samples of the random signal $x[k]$ by way of the ACF. The signal $x[k]$ is truncated to $N$ samples by multiplication (windowing) with the rectangular signal $\text{rect}_N[k]$ of length $N$

\begin{equation}
x_N[k] = x[k] \cdot \text{rect}_N[k]
\end{equation}

where $x_N[k]$ denotes the truncated signal.
The ACF is estimated by applying its definition in a straightforward manner. For a random signal $x_N[k]$ of finite length, the estimated ACF $\hat{\varphi}_{xx}[\kappa]$ can be expressed [in terms of a convolution](../random_signals/correlation_functions.ipynb#Definition)

\begin{equation}
\hat{\varphi}_{xx}[\kappa] = \frac{1}{N} \cdot x_N[k] * x_N[-k]
\end{equation}

Applying the DTFT to both sides and rearranging the terms yields

\begin{equation}
\hat{\Phi}_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{1}{N} \, X_N(\mathrm{e}^{\,\mathrm{j}\,\Omega})\, X_N(\mathrm{e}^{\,-\mathrm{j}\,\Omega}) =
\frac{1}{N} \, | X_N(\mathrm{e}^{\,\mathrm{j}\,\Omega}) |^2
\end{equation}

where the intermediate and last equalities result from the symmetry relations of the DTFT. This estimate of the PSD is known as the periodogram. It can be computed directly from the DTFT

\begin{equation}
X_N(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sum_{k=0}^{N-1} x_N[k] \, \mathrm{e}^{\,-\mathrm{j}\,\Omega\,k}
\end{equation}

of the truncated random signal.

### Example - Periodogram

The following example estimates the PSD of a random process which draws samples from normal distributed white noise with zero mean and unit variance. The true PSD is hence given as $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = 1$. In order to compute the periodogram by the discrete Fourier transformation (DFT), the signal $x[k]$ has to be zero-padded to ensure that squaring (multiplying) the spectra does not result in a circular convolution.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

N = 128  # number of samples

# generate random signal
np.random.seed(5)
x = np.random.normal(size=N)

# compute magnitude of the periodogram
x = np.concatenate((x, np.zeros_like(x)))
X = np.fft.rfft(x)
Om = np.linspace(0, np.pi, len(X))
Sxx = 1/N * abs(X)**2

# plot results
plt.figure(figsize=(10,4))
plt.stem(Om, Sxx, 'b', label=r'$|\hat{\Phi}_{xx}(e^{j \Omega})|$')
plt.plot(Om, np.ones_like(Sxx), 'r', label=r'$\Phi_{xx}(e^{j \Omega})$')
plt.title('Estimated and true PSD')
plt.xlabel(r'$\Omega$')
plt.axis([0, np.pi, 0, 4])
plt.legend()

# compute mean value of the periodogram
print('Mean value of the periodogram: {0:1.3f}'.format(np.mean(np.abs(Sxx))))
print('Variance of the periodogram: {0:1.3f}'.format(np.var(np.abs(Sxx))))
```

**Exercise**

* What do you have to change to evaluate experimentally if the periodogram is a consistent estimator?
* Based on the results, is the periodogram a consistent estimator?

Solution: The conditions for consistency have to be checked for the limiting case of a infinitely long signal ($N \to \infty$). Increasing the length `N` of the random signal in above example reveals that the periodogram can be assumed to be bias free, $b_{\hat{\Phi}_{xx}} = 0$. However, its variance $\sigma^2_{\hat{\Phi}_{xx}}$ does not tend to zero. The reason for this is that by increasing the length $N$ of the random signal also the number of spectral coefficients is increased by the same amount.

### Evaluation

From above numerical example it should have become clear that the periodogram is no consistent estimator for the PSD $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. It can be shown that the estimator is asymptotically bias free for $N \to \infty$, hence

\begin{equation}
\lim_{N \to \infty} E\{ \hat{\Phi}_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})  \} = \Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})
\end{equation}

This is due to the [leakage effect](../spectral_analysis_deterministic_signals/leakage_effect.ipynb) which limits the spectral resolution for signals of finite length.

The variance of the estimator does not converge towards zero

\begin{equation}
\lim_{N \to \infty} \sigma^2_{\hat{\Phi}_{xx}} \neq 0
\end{equation}

This is due to the fact that by increasing $N$ also the number of independent frequencies $\Omega = \frac{2 \pi}{N} \mu$ for $\mu = 0,1,\dots,N-1$ increases.

The periodogram is the basis for a variety of advanced estimation techniques for the PSD. These techniques rely on averaging or smoothing of (overlapping) periodograms.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2018*.


Introducción a la computación numérica y simbólica con Python

# 7. Computación simbólica

El lenguaje de las matemáticas no es numérico, sino simbólico. Los humanos somos capaces de manejar información abstracta que nos permite realizar transformaciones del siguiente tipo:

$f(x) = x^2-x-2 = (x+1)(x-2)$

${d^2 sen(3x)\over dx^2} = -9\ cos(3x)$

Existen programas como $\it{Maple}$ o $\it{Mathematica}$ que trabajan con símbolos. Son muy potentes pero tienen un precio de licencia elevado. Existe una alternativa open source basada en Python, el paquete `SymPy`.

`SymPy` puede ejecutarse online en la URL https://live.sympy.org

NOTA: En estos notebooks usamos LaTeX para construir las fórmulas en la parte descriptiva.

## Definición de símbolos

Cuando trabajamos con Python o cualquier otro lenguaje común de programación, estamos acostumbrados a usar variables
con las que operamos.

```python
import sympy as sp
import numpy as np
import matplotlib.pyplot as plt
```

```python
a = 7
b = 8
print("El resultado de la operación a + b es ",a+b)
```

    El resultado de la operación a + b es  15

Con `SymPy` podemos emplear símbolos.

```python
x = sp.symbols('x')
y = sp.symbols('y')
print('El resultado de la operación x + y en SympPy es',x+y)
```

    El resultado de la operación x + y en SympPy es x + y

Se han definido dos símbolos $x$,$y$ y el resultado es un nuevo símbolo $x+y$, no un valor numérico. Con SymPy se pueden realizar operaciones simbólicas de mayor complejidad.

```python
factor1 = x+1
factor2 = x-2
pfact = factor1*factor2
print("El producto de los factores (",factor1,")(",factor2, ") es",pfact)
```

    El producto de los factores ( x + 1 )( x - 2 ) es (x - 2)*(x + 1)

Aparentemente no ha ocurrido nada, salvo que la expresión aparece con la sintaxis propia de Python. Podemos hacer que el polinomio aparezca desarrollado.

```python
print("El producto de los factores (",factor1,")(",factor2, ") es",sp.expand(pfact))
```

    El producto de los factores ( x + 1 )( x - 2 ) es x**2 - x - 2

La función `sympyfy` convierte un texto en objetos SymPy siempre que la sintaxis sea correcta. Para que funcione correctamente los símbolos tienen que haberse definido.

```python
frac = sp.symbols('frac')
frac = sp.sympify('x**2/y')
print(frac)
```

    x**2/y

## Tipos

Aparte de los símbolos, `SymPy` puede manejar tipos de datos que no son los habituales en Python pero sí en el razonamiento matemático humano. Por ejemplo, podemos usar fracciones.

```python
# Fracciones
var1 = sp.sympify('1/6')
print("var1 =",var1)
var2 = sp.sympify('(5/8)')*var1
print("var2 =",var2)
print("Tipo de var2",type(var2))
print("Valor de var2 en punto flotante con 10 decimales de precisión",var2.evalf(10))

```

    var1 = 1/6
    var2 = 5/48
    Tipo de var2 <class 'sympy.core.numbers.Rational'>
    Valor de var2 en punto flotante con 10 decimales de precisión 0.1041666667
    var3 sqrt(2)
    Tipo de var3 <class 'sympy.core.power.Pow'>
    var2*var3 = 5*sqrt(2)/48
    Tipo de var2*var3  <class 'sympy.core.mul.Mul'> valor en punto flotante = 0.1473139127

También podemos usar potencias.

```python
# Potencias
var3= sp.sqrt(2)
print("var3 =",var3)
print("Tipo de var3",type(var3))
print("var2*var3 =",sp.Mul(var2,var3))
print("Tipo de var2*var3 ",type(var2*var3),"valor en punto flotante =",(var2*var3).evalf(10))

```

    var3 = sqrt(2)
    Tipo de var3 <class 'sympy.core.power.Pow'>
    var2*var3 = 5*sqrt(2)/48
    Tipo de var2*var3  <class 'sympy.core.mul.Mul'> valor en punto flotante = 0.1473139127

## Evaluación

Con frecuencia, necesitamos evaluar una expresión simbólica para determinado valor. Si queremos conocer el valor de $f(x) = x^3 -x -2*cos(\pi x)$ en el punto $x=0.3$, tenemos que usar el método `subs` para cambiar la incógnita por el valor y el método `evalf` para evaluar la expresión.

```python
x,f = sp.symbols('x f')
f = x**3 - x - 2*sp.cos(sp.pi * x)
print("La función es f(x)=",f)
print("Su valor para x = 3 es f(0.3) :",(f.subs(x,0.3)).evalf(8))      # El parámetro de evalf indica el número de cifras decimales
```

    La función es f(x)= x**3 - x - 2*cos(pi*x)
    Su valor para x = 3 es f(0.3) : -1.4485705

Cuando usamos `SymPy` tenemos que tener mucho cuidado con las comparaciones. El operador `==` compara si dos expresiones son iguales.

```python
x,y,z = sp.symbols('x y z')
y = sp.sin(x)
z = sp.sin(x)
# los símbolos y,z son estructuralmente idénticos
print(y,'==',z,y == z)
```

    sin(x) == sin(x) True

Sin embargo `SymPy` no es capaz de detectar por defecto que $cos(2x) == cos^2(x)-sin^2(x)$ con el operador `==`, para eso hay que usar el método `equals`

```python
a,b = sp.symbols('a b')
a = sp.cos(x)**2 - sp.sin(x)**2
b = sp.cos(2*x)
print(b,'==',a,b==a)

# Pero sí es capaz de hacerlo con el método equals
print(b,"equals",a,a.equals(b))

```

    cos(2*x) == -sin(x)**2 + cos(x)**2 False
    cos(2*x) equals -sin(x)**2 + cos(x)**2 True

Podemos dibujar la gráfica de la función usando este mismo procedimiento.

## SymPy para gráficas

Podemos usar el método `evalf` para calcular punto a punto el valor de una expresión que queremos representar gráficamente.

```python
f = x**3 - x - 2*sp.cos(sp.pi * x)
print("La función es f(x)=",f)
# Vector de valores x
xpuntos = np.linspace(0,2,100)
ypuntos = []
for i in xpuntos:
    ypuntos.append((f.subs(x,i)).evalf(8)) # Evaluación de f(x) en cada punto
plt.figure(figsize=(8,5)) # tamaño de la gráfica en pulgadas
plt.plot(xpuntos,ypuntos)
plt.title('$x^3-x-2cos(\pi x)$')
plt.xlabel('$x$') # título del eje horizontal
plt.ylabel('$y$') # título del eje  vertical
plt.show()
```

Hay una manera más eficiente de evaluar en múltiples puntos, la función `lambdify`.

```python
expresionmat = sp.symbols('expresionmat')
a = np.linspace(0,2*np.pi,1000)
expresionmat = sp.sin(x)**(3)-2*sp.cos(3*x)+sp.log(1+x)*0.1
f = sp.lambdify(x, expresionmat, "numpy")  # Convierte la expresión simbólica a la funciones equivalentes de de NumPy
plt.plot(a,f(a))
plt.title(expresionmat)
plt.show()
```

## Presentación

Una de las características más interesantes de `SymPy` es su capacidad de formatear la salida como en un texto matemático convencional.

```python
# La función f del ejemplo anterior está escrita con sintaxis Python que es la que entiende el lenguaje
print(expresionmat)
```

    0.1*log(x + 1) + sin(x)**3 - 2*cos(3*x)

```python
# SymPy permite presentar la misma expresión

sp.init_printing()
sp.pprint(expresionmat)
expresionmat
```

```python
# Si el equipo tiene instalado LaTeX, la salida será aun más legible

from sympy import init_session
init_session(use_latex=True)
expresionmat

```

```python
x = sp.sqrt(3)
pprint(Mul(x,  x, evaluate=False))
print('es igual a',x*x)
```

    √3⋅√3
    es igual a 3

```python
a11,a12,a21,a22 = symbols('a11 a12 a21 a22')
A = sp.Matrix([
            [a11, a12],
            [a21, a22]
    ])
A
```

```python
b11,b21 = symbols('b11 b21')
B = sp.Matrix([[b11, b21]])
B.T
```

```python
C=A*B.T
C
```

---

    (c) 2020 Javier García Algarra. www.u-tad.com

Licensed under a Creative Commons Reconocimiento 4.0 Internacional License


```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import statsmodels.api as sm

pd.plotting.register_matplotlib_converters()
```

# Filtering

*Filtering* time series data is a process of extracting certain components or signals from the data. The idea is that time series data are the result of a collection of processes — both stochastic and deterministic — that have different *frequencies*. Many filters exist for extracting cyclical and/or trend components from business cycle data.

In this Notebook, I will show you how to use a couple of popular filters to extract business cycle components from real GDP per capita data.

## Data

Real GDP per capita data are obtained from FRED (Series ID: A939RX0Q048SBEA).

```python
# Download real GDP per capita data from FRED
gdp = pd.read_csv("https://fred.stlouisfed.org/data/A939RX0Q048SBEA.txt",skiprows=14,sep='\s+',parse_dates=True,index_col=0)

# Convert units to thousands of dollars
gdp = gdp/1000

# Plot
ax = gdp.plot(legend=False,title='Real GDP per Capita')
ax.set_xlabel('Date')
ax.set_ylabel('Dollars')
```

In specting the data, it's apparent that there is a slow-moving (low-frequency) trend component and a cyclical (high-frequency) component. Most filters compute the cyclical component as a deviation (or diffrence) from trend. So we take the log of GDP before applying any filters. Here's why. Let $Y_t$ denote real GDP per capita and $y_t = \log Y_t$. Then if we filter $y_t$ into a trend and cyclical component,

\begin{align}
y_t & = y_t^{\text{trend}} + y_t^{\text{cycle}}
\end{align}

Since $y_t$ is the log of $Y_t$:

\begin{align}
y_t^{\text{cycle}} & = y_t - y_t^{\text{trend}}\\
& = \log Y_t - \log Y_t^{\text{trend}} \\
& \approx \frac{Y_t - Y_t^{\text{trend}}}{Y_t^{\text{trend}}}
\end{align}

Then the cyclical component of log real GDP per capita will be approximately the percentage deviation of GDP from trend (divided by 100).

```python
# Take (natural) log of GDP
gdp = np.log(gdp)

# Plot
ax = gdp.plot(legend=False,title='Real GDP per Capita')
ax.set_xlabel('Date')
ax.set_ylabel('log Dollars')
```

## Linear filter

The linear filter assumes that the trend component is a linear function of time:

\begin{align}
y_t^{\text{trend}} & = \alpha_0 + \alpha_1 t
\end{align}

where $t = 0,1,2,\ldots$. To find $\alpha_0$ and $\alpha_1$, regress $y_t$ on $t$ and a constant. The predicted values of the regression will equal the estimated trend component and the residuals of the regression will equal the estimated cyclical component.

```python
# Create time values
time = np.arange(0,len(gdp))

# Create X and Y variables for sm.OLS
X = sm.add_constant(time)
Y = gdp

# Create OLS model and fit
model =  sm.OLS(Y,X)
results = model.fit()

# Create variable with linear trend. Note that the results.predict() returns a
# Numpy array so I put it in a Pandas Series with the same dates as GDP.
y_trend_linear = pd.Series(results.predict(),index = gdp.index)

# Cyclical component
y_cycle_linear = results.resid
```

Plot trend log GDP and log GDP implied by the linear filter.

```python
plt.plot(gdp,label='actual')
plt.plot(y_trend_linear,label='trend')
plt.title('Real GDP per Capita')
plt.ylabel('log Dollars')
plt.legend()
```

Plot the cyclical component of log GDP implied by the linear filter.

```python
plt.plot(y_cycle_linear*100)
plt.title('Percent Deviation of Real GDP per Capita from Trend')
plt.ylabel('Percent')
```

This filter is only good if the trend is well-approximated by a linear function of time. But as you can see, according to the linear trend, there are long periods (decades) where GDP is computed to be above or below trend.

## Hodrick-Prescott Filter

The Hodrick-Prescott or HP filter is one of the most commonly used filters in macroeconomics. The HP filter doesn't assume a constant trend and instead computes the trend to minimize a complicated objective function. See: https://en.wikipedia.org/wiki/Hodrick%E2%80%93Prescott_filter.

Implement the HP filter using `statsmodels.api.tsa.filters.hpfilter()`. This function returns a trend and cyclical series.

```python
# Compute the HP filter
y_cycle_hp, y_trend_hp = sm.tsa.filters.hpfilter(gdp)
```

Plot trend log GDP and log GDP implied by the HP filter.

```python
plt.plot(gdp,label='actual')
plt.plot(y_trend_hp,label='trend')
plt.title('Real GDP per Capita')
plt.ylabel('log Dollars')
plt.legend()
```

Notice how the trend computed by the HP filter has gradual movements that are not present in the trend computed with the linear filter.

Now, plot the cyclical component of log GDP implied by the HP filter.

```python
plt.plot(y_cycle_hp*100)
plt.title('Percent Deviation of Real GDP per Capita from Trend')
plt.ylabel('Percent')
```

Cyclical component of GDP computed by the HP-filter appear to return to zero more frequently than what was computed by the linear filter.

## Baxter-King or Band-Pass Filter

The Baxter-King or band-pass or BP filter is another commonly used filter in macroeconomics. The BP filter only returns a cyclical series and some data are lost at the beginning and end of the sample. See: https://en.wikipedia.org/wiki/Band-pass_filter

Implement the BP filter using `statsmodels.api.tsa.filters.bkfilter()`.

```python
# Compute the BP filter
y_cycle_bp = sm.tsa.filters.bkfilter(gdp)
```

Plot the cyclical component of log GDP implied by the BP filter.

```python
plt.plot(y_cycle_bp*100)
plt.title('Percent Deviation of Real GDP per Capita from Trend')
plt.ylabel('Percent')
```

## Comparison of the HP and BP Filters

Since the linear filter isn't very good, let's focus on comparing cyclical components of GDP computed by the HP and BP filters.

```python
fig = plt.figure(figsize=(12,4))
plt.plot(y_cycle_hp*100,label='HP filter')
plt.plot(y_cycle_bp*100,label='BP filter')
plt.title('Percent Deviation of Real GDP per Capita from Trend')
plt.ylabel('Percent')
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
```

Apparently the BP filter generates a smoother series, but the results are generally comparable. It's clear that even though the functions are very different, they are computing something similar.

## Additional Notes

The default behavior for the HP and BP filter functions in statsmodels is to assume that data input are quarterly. If the data are not quarterly, then you must specify the arguments of the respective functions. Also, for data that are in terms of rates (like inflation or interest), do not take logs of the data before filtering.

In the next example, I show how to filter the business cycle component out of the monthly unemployment rate data.

```python
# Download real GDP per capita data from FRED
unemp = pd.read_csv("https://fred.stlouisfed.org/data/UNRATE.txt",skiprows=23,sep='\s+',parse_dates=True,index_col=0)

# Plot
plt.plot(unemp)
plt.title('Unemployment Rate')
plt.xlabel('Date')
plt.ylabel('Percent')
```

```python
# Compute the HP filter
u_cycle_hp, u_trend_hp = sm.tsa.filters.hpfilter(unemp,lamb=129600)

# Compute the BP filter
u_cycle_bp = sm.tsa.filters.bkfilter(unemp,low=24,high=84,K=84)

# Plot computed cyclical components
fig = plt.figure(figsize=(12,4))
plt.plot(u_cycle_hp*1,label='HP filter')
plt.plot(u_cycle_bp*1,label='BP filter')
plt.title('Deviation of the Unemploymebnt Rate from Trend')
plt.ylabel('Percent')
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
```

An alternative to what was just done would be to resample the monthly data as quarterly data. This is often necessary anyway since most macroeconomic data are quarterly and generally all data used in statistical analysis should have the same frequency.

```python
# Compute the average unemployment rate over three month windows
unempQ = unemp.resample('QS').mean()

# Verify that the dates are incremented every three months
unempQ.head()
```

      VALUE

      DATE

      1948-01-01
      3.733333

      1948-04-01
      3.666667

      1948-07-01
      3.766667

      1948-10-01
      3.833333

      1949-01-01
      4.666667

```python
# Plot the quarterly and monthly series
plt.plot(unemp,label='Monthly')
plt.plot(unempQ,label='Quarterly')
plt.title('Unemployment Rate')
plt.xlabel('Date')
plt.ylabel('Percent')
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
```

```python

```