(c) Juan Gomez 2019. Thanks to Universidad EAFIT for support. This material is part of the course Introduction to Finite Element Analysis

# SOLUTION: Introduction to One-Dimensional Lagrange Interpolation

## By Juan Gomez

```python
%matplotlib notebook
import matplotlib.pyplot as plt
import numpy as np
from scipy import interpolate
import sympy as sym
```

### Class activity

**Follow the steps indicated next to implement, in an independent notebook, an interpolation scheme typical of the ones used in FEM codes. The resulting scheme would correspond to the interpolation within a single element**.

## Shape functions

* Assuming that a constant interpolation domain is defined as $x \in \left[ {{-1.0},{+1.0}} \right]$ with nodal points corresponding to $x^0 = -1.0$, $x^1 = +1.0$, $x^2 = 0.0$, $x^3 = -0.25$ and $x^4 = +0.25$ use the function `LagrangPoly()` to generate the interpolation functions associated to these 5 nodal points. In a single graph plot the resulting polynomials.

```python
def LagrangPoly(x, order, i, xi=None):
    if xi == None:
        xi = sym.symbols('x:%d' % (order+1))
    index = list(range(order+1))
    index.pop(i)
    return sym.prod([(x-xi[j])/(xi[i]-xi[j]) for j in index])
```

```python
npts = 200
yy = np.zeros((npts))
xx = np.linspace(-1, 1, npts)
```

```python
x = sym.symbols('x')
pol = []
pol.append(sym.simplify(LagrangPoly(x, 4, 0, [-1, 1, 0 , -0.25 , 0.25])))
pol.append(sym.simplify(LagrangPoly(x, 4, 1, [-1, 1, 0 , -0.25 , 0.25])))
pol.append(sym.simplify(LagrangPoly(x, 4, 2, [-1, 1, 0 , -0.25 , 0.25])))
pol.append(sym.simplify(LagrangPoly(x, 4, 3, [-1, 1, 0 , -0.25 , 0.25])))
pol.append(sym.simplify(LagrangPoly(x, 4, 4, [-1, 1, 0 , -0.25 , 0.25])))
#print(pol)
```

```python
plt.figure(0)
plt.grid()
for k in range(5):
    for i in range(npts):
        yy[i] = pol[k].subs([(x, xx[i])])
    plt.plot(xx, yy)
```

    <IPython.core.display.Javascript object>

## Interpolating polynomial

* Use the interpolation functions found in the previous step, to define a matrix $\left[N(x)\right]$, which will be called in what follows the `function interpolation matrix`, in such a way that the interpolation operation can be conducted using matrix operations like:

$$\hat f(x) = \left[N(x)\right]\left\{F\right\}$$

and where $F$ is a vector storing the nodal values of the function. In your Notebook print the symbolic version of the interpolation matrix.

* In your Notebook plot a function $f(x)$ (representing temperature in a bar) and its interpolated version $\hat f(x)$ using the matrix scheme developed in the previous step. The code should plot also the first order derivative of the function.

```python
fx = lambda x: x**4 + 4.0*x**3 + 3.0*x**2 - 10.0
```

```python
npts = 200
yy = np.zeros((npts))
xx = np.linspace(-1, 1, npts)
```

```python
fd = np.array([fx(-1.0), fx(1.0), fx(0.0) , fx(-0.25) , fx(+0.25)])
print(fd)
```

    [-10.          -2.         -10.          -9.87109375  -9.74609375]

```python
print(pol[0]*fd[0] + pol[1]*fd[1] + pol[2]*fd[2] + pol[3]*fd[3] + pol[4]*fd[4])
```

    -160.0*x**4 + 170.0*x**2 - 5.0*x*(0.8*x - 0.2)*(x - 1)*(1.33333333333333*x + 0.333333333333333) - 1.0*x*(0.8*x + 0.2)*(x + 1)*(1.33333333333333*x - 0.333333333333333) + 42.1166666666667*x*(x - 1)*(x + 1)*(2.0*x - 0.5) + 41.5833333333333*x*(x - 1)*(x + 1)*(2.0*x + 0.5) - 10.0

```python
plt.figure(2)
plt.grid()
for i in range(npts):
    yy[i] = fd[0]*pol[0].subs([(x, xx[i])]) + fd[1]*pol[1].subs([(x, xx[i])]) \
        + fd[2]*pol[2].subs([(x, xx[i])]) + fd[3]*pol[3].subs([(x, xx[i])]) \
        + fd[4]*pol[4].subs([(x, xx[i])])

zz = fx(xx)
plt.plot([-1, 1, 0 , -0.25 , 0.25], fd, 'ko')
plt.plot(xx, yy, 'r--')
plt.plot(xx, zz)
```

    <IPython.core.display.Javascript object>

    [<matplotlib.lines.Line2D at 0xb1561e5f8>]

## Derivative

```python
fdx = lambda x: 4*x**3 + 12.0*x**2 + 6.0*x

fc = np.array([fdx(-1.0), fdx(1.0), fdx(0.0) , fdx(-0.25) , fdx(0.25)])
```

The interpolatted derivatives are then obtained according to:

\begin{equation}
\hat f'(x) = \frac{dL^0(x)}{dx} f^0 + \frac{dL^1(x)}{dx}(x) f^1 + \frac{dL^2(x)}{dx} f^2 + + \frac{dL^3(x)}{dx} f^3 + + \frac{dL^4(x)}{dx} f^4
\end{equation}

```python
dpol = []
dpol.append(sym.diff(pol[0], x))
dpol.append(sym.diff(pol[1], x))
dpol.append(sym.diff(pol[2], x))
dpol.append(sym.diff(pol[3], x))
dpol.append(sym.diff(pol[4], x))
#print(dpol)
#print(dpol[0]*fd[0] + dpol[1]*fd[1] + dpol[2]*fd[2] + dpol[3]*fd[3] + dpol[4]*fd[4])
#
plt.figure(3)
yy = fdx(xx)
plt.plot(xx, yy, 'r--')
plt.plot([-1, 1, 0 , -0.25 , 0.25], fc, 'ko')
#
for i in range(npts):
    yy[i] = fd[0]*dpol[0].subs([(x, xx[i])]) + fd[1] * \
        dpol[1].subs([(x, xx[i])]) + fd[2]*dpol[2].subs([(x, xx[i])]) \
    + fd[3]*dpol[3].subs([(x, xx[i])]) + fd[4]*dpol[4].subs([(x, xx[i])])
plt.plot(xx, yy)
```

    <IPython.core.display.Javascript object>

    [<matplotlib.lines.Line2D at 0xb1590ae48>]

## Matrix scheme like in the FEM

* To conduct the interpolation in the Notebook create a Python function or subroutine that takes as input parameters the coordinate $x$ where the function is to be approximated and the vector of nodal values of the temperature function and returns upon execution the interpolated value of the function.

```python
def shape1d(x , un):
    N = sym.Matrix([[x*(0.8*x - 0.2)*(x - 1)*(1.33333333333333*x + 0.333333333333333)/2,
         x*(0.8*x + 0.2)*(x + 1)*(1.33333333333333*x - 0.333333333333333)/2,
         16.0*x**4 - 17.0*x**2 + 1.0 , -4.26666666666667*x*(x - 1)*(x + 1)*(2.0*x - 0.5) ,
         -4.26666666666667*x*(x - 1)*(x + 1)*(2.0*x + 0.5) ]])
    u_int = N*un
#
    return u_int
```

```python
uu = sym.Matrix([-10.0, -2.0 , -10.0 , -9.87 , -9.74])

xx = 0.50

u_int = shape1d(xx , uu)
print(u_int)
```

    Matrix([[-8.67200000000003]])

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('./nb_style.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>

<style>

/*
Template for Notebooks for Modelación computacional.

Based on Lorena Barba template available at:

    https://github.com/barbagroup/AeroPython/blob/master/styles/custom.css
*/

/* Fonts */
@font-face {
font-family: "Computer Modern";
src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

/* Text */
div.cell{
width:800px;
margin-left:16% !important;
margin-right:auto;
}
h1 {
font-family: 'Alegreya Sans', sans-serif;
}
h2 {
font-family: 'Fenix', serif;
}
h3{
font-family: 'Fenix', serif;
margin-top:12px;
margin-bottom: 3px;
}
h4{
font-family: 'Fenix', serif;
}
h5 {
font-family: 'Alegreya Sans', sans-serif;
}
div.text_cell_render{
font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
line-height: 135%;
font-size: 120%;
width:600px;
margin-left:auto;
margin-right:auto;
}
.CodeMirror{
font-family: "Source Code Pro";
font-size: 90%;
}
/* .prompt{
display: None;
}*/
.text_cell_render h1 {
font-weight: 200;
font-size: 50pt;
line-height: 100%;
color:#CD2305;
margin-bottom: 0.5em;
margin-top: 0.5em;
display: block;
}
.text_cell_render h5 {
font-weight: 300;
font-size: 16pt;
color: #CD2305;
font-style: italic;
margin-bottom: .5em;
margin-top: 0.5em;
display: block;
}
.warning{
color: rgb( 240, 20, 20 )
}
</style>

```python

```


<p align="center">

</p>

## Interactive Hypothesis Testing Demonstration

### Boostrap and Analytical Methods for Hypothesis Testing, Difference in Means

* we calculate the hypothesis test for different in means with boostrap and compare to the analytical expression

* **Welch's t-test**: we assume the features are Gaussian distributed and the variance are unequal

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

#### Hypothesis Testing

Powerful methodology for spatial data analytics:

1. extracted sample set 1 and 2, the means look different, but are they?
2. should we suspect that the samples are in fact from 2 different populations?

Now, let's try the t-test, hypothesis test for difference in means.  This test assumes that the variances are similar along with the data being Gaussian distributed (see the course notes for more on this).  This is our test:

\begin{equation}
H_0: \mu_{X1} = \mu_{X2}
\end{equation}

\begin{equation}
H_1: \mu_{X1} \ne \mu_{X2}
\end{equation}

To test this we will calculate the t statistic with the bootstrap and analytical approaches.

#### The Welch's t-test for Difference in Means by Analytical and Empirical Methods

We work with the following test statistic, *t statistic*, from the two sample sets.

\begin{equation}
\hat{t} = \frac{\overline{x}_1 - \overline{x}_2}{\sqrt{\frac{s^2_1}{n_1} + \frac{s^2_2}{n_2}}}
\end{equation}

where $\overline{x}_1$ and $\overline{x}_2$ are the sample means, $s^2_1$ and $s^2_2$ are the sample variances and $n_1$ and $n_2$ are the numer of samples from the two datasets.

The critical value, $t_{critical}$ is calculated by the analytical expression by:

\begin{equation}
t_{critical} = \left|t(\frac{\alpha}{2},\nu)\right|
\end{equation}

The degrees of freedom, $\nu$, is calculated as follows:

\begin{equation}
\nu = \frac{\left(\frac{1}{n_1} + \frac{\mu}{n_2}\right)^2}{\frac{1}{n_1^2(n_1-1)} + \frac{\mu^2}{n_2^2(n_2-1)}}
\end{equation}

Alternatively, the sampling distribution of the t_{statistic} and t_{critical} may be calculated empirically with bootstrap.

The workflow proceeds as:

* shift both sample sets to have the mean of the combined data, $x_1$ → $x^*_1$, $x_2$ → $x^*_2$

* for each bootstrap realization, $\ell=1\ldots,L$

    * perform $n_1$ Monte Carlo simulations, draws with replacement, from sample set $x^*_1$

    * perform $n_2$ Monte Carlo simulations, draws with replacement, from sample set $x^*_2$

    * calculate the t_{statistic} realization, $\hat{t}^{\ell}$ given the resulting sample means $\overline{x}^{*,\ell}_1$ and $\overline{x}^{*,\ell}_2$ and the sample variances $s^{*,2,\ell}_1$ and $s^{*,2,\ell}_2$

* pool the results to assemble the $t_{statistic}$ sampling distribution

* calculate the cumulative probability of the observed t_{statistic}m, $\hat{t}$, from the boostrap distribution based on $\hat{t}^{\ell}$, $\ell = 1,\ldots,L$.

Here's some prerequisite information on the boostrap.

#### Bootstrap

Uncertainty in the sample statistics
* one source of uncertainty is the paucity of data.
* do 200 or even less wells provide a precise (and accurate estimate) of the mean? standard deviation? skew? P13?

Would it be useful to know the uncertainty in these statistics due to limited sampling?
* what is the impact of uncertainty in the mean porosity e.g. 20%+/-2%?

**Bootstrap** is a method to assess the uncertainty in a sample statistic by repeated random sampling with replacement.

Assumptions
* sufficient, representative sampling, identical, idependent samples

Limitations
1. assumes the samples are representative
2. assumes stationarity
3. only accounts for uncertainty due to too few samples, e.g. no uncertainty due to changes away from data
4. does not account for boundary of area of interest
5. assumes the samples are independent
6. does not account for other local information sources

The Bootstrap Approach (Efron, 1982)

Statistical resampling procedure to calculate uncertainty in a calculated statistic from the data itself.
* Does this work?  Prove it to yourself, for uncertainty in the mean solution is standard error:

\begin{equation}
\sigma^2_\overline{x} = \frac{\sigma^2_s}{n}
\end{equation}

Extremely powerful - could calculate uncertainty in any statistic!  e.g. P13, skew etc.
* Would not be possible access general uncertainty in any statistic without bootstrap.
* Advanced forms account for spatial information and sampling strategy (game theory and Journel’s spatial bootstrap (1993).

Steps:

1. assemble a sample set, must be representative, reasonable to assume independence between samples

2. optional: build a cumulative distribution function (CDF)
    * may account for declustering weights, tail extrapolation
    * could use analogous data to support

3. For $\ell = 1, \ldots, L$ realizations, do the following:

    * For $i = \alpha, \ldots, n$ data, do the following:

        * Draw a random sample with replacement from the sample set or Monte Carlo simulate from the CDF (if available).

6. Calculate a realization of the sammary statistic of interest from the $n$ samples, e.g. $m^\ell$, $\sigma^2_{\ell}$. Return to 3 for another realization.

7. Compile and summarize the $L$ realizations of the statistic of interest.

This is a very powerful method.  Let's try it out and compare the result to the analytical form of the confidence interval for the sample mean.

#### Objective

Provide an example and demonstration for:

1. interactive plotting in Jupyter Notebooks with Python packages matplotlib and ipywidgets
2. provide an intuitive hands-on example of confidence intervals and compare to statistical boostrap

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

#### Load the Required Libraries

The following code loads the required libraries.

```python
%matplotlib inline
from ipywidgets import interactive                        # widgets and interactivity
from ipywidgets import widgets
from ipywidgets import Layout
from ipywidgets import Label
from ipywidgets import VBox, HBox
import matplotlib.pyplot as plt                           # plotting
import numpy as np                                        # working with arrays
import pandas as pd                                       # working with DataFrames
from scipy import stats                                   # statistical calculations
import random                                             # random drawing / bootstrap realizations of the data
```

#### Make a Synthetic Dataset

This is an interactive method to:

* select a parametric distribution

* select the distribution parameters

* select the number of samples and visualize the synthetic dataset distribution

```python

# interactive calculation of the sample set (control of source parametric distribution and number of samples)
l = widgets.Text(value=' Interactive Hypothesis Testing, Difference in Means, Analytical & Bootstrap Methods, Michael Pyrcz, Associate Professor, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))

n1 = widgets.IntSlider(min=0, max = 100, value = 10, step = 1, description = '$n_{1}$',orientation='horizontal',layout=Layout(width='300px', height='30px'),continuous_update=False)
n1.style.handle_color = 'red'

m1 = widgets.FloatSlider(min=0, max = 50, value = 3, step = 1.0, description = '$\overline{x}_{1}$',orientation='horizontal',layout=Layout(width='300px', height='30px'),continuous_update=False)
m1.style.handle_color = 'red'

s1 = widgets.FloatSlider(min=0, max = 10, value = 3, step = 0.25, description = '$s_1$',orientation='horizontal',layout=Layout(width='300px', height='30px'),continuous_update=False)
s1.style.handle_color = 'red'

ui1 = widgets.VBox([n1,m1,s1],)                               # basic widget formatting

n2 = widgets.IntSlider(min=0, max = 100, value = 10, step = 1, description = '$n_{2}$',orientation='horizontal',layout=Layout(width='300px', height='30px'),continuous_update=False)
n2.style.handle_color = 'yellow'

m2 = widgets.FloatSlider(min=0, max = 50, value = 3, step = 1.0, description = '$\overline{x}_{2}$',orientation='horizontal',layout=Layout(width='300px', height='30px'),continuous_update=False)
m2.style.handle_color = 'yellow'

s2 = widgets.FloatSlider(min=0, max = 10, value = 3, step = 0.25, description = '$s_2$',orientation='horizontal',layout=Layout(width='300px', height='30px'),continuous_update=False)
s2.style.handle_color = 'yellow'

ui2 = widgets.VBox([n2,m2,s2],)                               # basic widget formatting

L = widgets.IntSlider(min=10, max = 1000, value = 100, step = 1, description = '$L$',orientation='horizontal',layout=Layout(width='300px', height='30px'),continuous_update=False)
L.style.handle_color = 'gray'

alpha = widgets.FloatSlider(min=0, max = 50, value = 3, step = 1.0, description = '$α$',orientation='horizontal',layout=Layout(width='300px', height='30px'),continuous_update=False)
alpha.style.handle_color = 'gray'

ui3 = widgets.VBox([L,alpha],)                                # basic widget formatting

ui4 = widgets.HBox([ui1,ui2,ui3],)                               # basic widget formatting

ui2 = widgets.VBox([l,ui4],)

def f_make(n1, m1, s1, n2, m2, s2, L, alpha):                                      # function to take parameters, make sample and plot

    np.random.seed(73073)
    x1 = np.random.normal(loc=m1,scale=s1,size=n1)
    np.random.seed(73074)
    x2 = np.random.normal(loc=m2,scale=s2,size=n2)

    mu = (s2*s2)/(s1*s1)
    nu = ((1/n1 + mu/n2)*(1/n1 + mu/n2))/(1/(n1*n1*(n1-1)) + ((mu*mu)/(n2*n2*(n2-1))))

    prop_values = np.linspace(-8.0,8.0,100)
    analytical_distribution = stats.t.pdf(prop_values,df = nu)
    analytical_tcrit = stats.t.ppf(1.0-alpha*0.005,df = nu)

    # Analytical Method with SciPy
    t_stat_observed, p_value_analytical = stats.ttest_ind(x1,x2,equal_var=False)

    # Bootstrap Method
    global_average = np.average(np.concatenate([x1,x2]))   # shift the means to be equal to the globla mean
    x1s = x1 - np.average(x1) + global_average
    x2s = x2 - np.average(x2) + global_average

    t_stat = np.zeros(L); p_value = np.zeros(L)

    random.seed(73075)
    for l in range(0, L):                      # loop over realizations
        samples1 = random.choices(x1s, weights=None, cum_weights=None, k=len(x1s))
        #print(samples1)
        samples2 = random.choices(x2s, weights=None, cum_weights=None, k=len(x2s))
        #print(samples2)
        t_stat[l], p_value[l] = stats.ttest_ind(samples1,samples2,equal_var=False)

    bootstrap_lower = np.percentile(t_stat,alpha * 0.5)
    bootstrap_upper = np.percentile(t_stat,100.0 - alpha * 0.5)

    plt.subplot(121)
    #print(t_stat)

    plt.hist(x1,cumulative = False, density = True, alpha=0.4,color="red",edgecolor="black", bins = np.linspace(0,50,50), label = '$x_1$')
    plt.hist(x2,cumulative = False, density = True, alpha=0.4,color="yellow",edgecolor="black", bins = np.linspace(0,50,50), label = '$x_2$')
    plt.ylim([0,0.4]); plt.xlim([0.0,30.0])
    plt.title('Sample Distributions'); plt.xlabel('Value'); plt.ylabel('Density')
    plt.legend()

    #plt.hist(x2)

    plt.subplot(122)
    plt.ylim([0,0.6]); plt.xlim([-8.0,8.0])
    plt.title('Bootstrap and Analytical $t_{statistic}$ Sampling Distributions'); plt.xlabel('$t_{statistic}$'); plt.ylabel('Density')
    plt.plot([t_stat_observed,t_stat_observed],[0.0,0.6],color = 'black',label='observed $t_{statistic}$')
    plt.plot([bootstrap_lower,bootstrap_lower],[0.0,0.6],color = 'blue',linestyle='dashed',label = 'bootstrap interval')
    plt.plot([bootstrap_upper,bootstrap_upper],[0.0,0.6],color = 'blue',linestyle='dashed')
    plt.plot(prop_values,analytical_distribution, color = 'red',label='analytical $t_{statistic}$')
    plt.hist(t_stat,cumulative = False, density = True, alpha=0.2,color="blue",edgecolor="black", bins = np.linspace(-8.0,8.0,50), label = 'bootstrap $t_{statistic}$')

    plt.fill_between(prop_values, 0, analytical_distribution, where = prop_values <= -1*analytical_tcrit, facecolor='red', interpolate=True, alpha = 0.2)
    plt.fill_between(prop_values, 0, analytical_distribution, where = prop_values >= analytical_tcrit, facecolor='red', interpolate=True, alpha = 0.2)
    ax = plt.gca()
    handles,labels = ax.get_legend_handles_labels()
    handles = [handles[0], handles[2], handles[3], handles[1]]
    labels = [labels[0], labels[2], labels[3], labels[1]]

    plt.legend(handles,labels,loc=1)

    plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
    plt.show()

# connect the function to make the samples and plot to the widgets
interactive_plot = widgets.interactive_output(f_make, {'n1': n1, 'm1': m1, 's1': s1, 'n2': n2, 'm2': m2, 's2': s2, 'L': L, 'alpha': alpha})
interactive_plot.clear_output(wait = True)                # reduce flickering by delaying plot updating
```

### Boostrap and Analytical Methods for Hypothesis Testing, Difference in Means

* including the analytical and bootstrap methods for testing the difference in means
* interactive plot demonstration with ipywidget, matplotlib packages

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

### The Problem

Let's simulate bootstrap, resampling with replacement from a hat with $n_{red}$ and $n_{green}$ balls

* **$n_1$**, **$n_2$** number of samples, **$\overline{x}_1$**, **$\overline{x}_2$** means and **$s_1$**, **$s_2$** standard deviation of the 2 sample sets
* **$L$**: number of bootstrap realizations
* **$\alpha$**: alpha level

```python
display(ui2, interactive_plot)                            # display the interactive plot
```

    VBox(children=(Text(value=' Interactive Hypothesis Testing, Difference in Means, Analytical & Bootstrap Method…

    Output()

#### Observations

Some observations:

* lower dispersion and higher difference in means increases the absolute magnitude of the observed $t_{statistic}$

* the bootstrap distribution closely matches the analytical distribution if $L$ is large enough

* it is possible to use bootstrap to calculate the sampling distribution instead of relying on the theoretical express distribution, in this case the Student's t distribution.

#### Comments

This was a demonstration of interactive hypothesis testing for the significance in difference in means aboserved between 2 sample sets in Jupyter Notebook Python with the ipywidgets and matplotlib packages.

I have many other demonstrations on data analytics and machine learning, e.g. on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


# Solwing the Solow model with R&D

## Intro

In this project we will try to solve the solow model with R&D and furthermore extend the model by including human capital.

Our goal will be to dig into the following aspects:

* Finding the steady state rates by solveing for several transition equation
* Finding numeric values for the steady state rates
* Visualizing the transition towards steady state

The realistic parameters, which will be used to find a numeric value for the steady state rates can be found in the textbook "*Introducing Advanced Macroeconomics*" by Peter Birch Sørensen and Hans Jørgen Whitta-Jacobsen.

# Post peer feedback changes

* Improved both transition plots - More interactive features, made arrow follow the equilibrium etc.
* Some people have troubles running the plots - Theses errors should be fixed now
* Fixed typos and mislabels

### More Feedback ideas, which was taken into considerations but ultimately was not implemented

* Direction fields
* Calculating speed of convergence

```python
import numpy as np
import matplotlib.pyplot as plt
import ipywidgets as widgets
import sympy as sm

sm.init_printing(use_unicode=True)
```

We consider the following model for a closed economy, where the following equations are given:

1. \\[Y_{t}=K_{t}^{\alpha}\left(A_{t}L_{Y_{t}}\right)^{1-\alpha}, 0<\alpha<1\\]
2. \\[K_{t+1}-K_{t}=s_{K}Y_{t}-\delta K_{t},0<\delta<1,0<s_{k}<1,K_{0}>0  \text{ provided} \\]
3. \\[A_{t+1}-A_{t}=\rho A_{t}^{\phi}L_{A_{t}}^{\lambda},\rho>0,\phi>0,0<\lambda<1 \\]
4. \\[L_{A_{t}}=s_{R}L_{t},0<s_{R}<1 \\]
5. \\[L_{t}=L_{A_{t}}+L_{Y_{t}} \\]
6. \\[L_{t+1}=\left(1+n\right)L_{t},n>0 \\]

Equation (1) is a Cobb-Douglas production function, which describes the aggregated production, $Y_{t}$, as a function of financial capital, $K_{t}$, production workers, $L_{Y_{t}}$ and the knowledge level, $A_{t}$, which determines the productivity of the workers.

Equation (2) describes how financial capital develops over time, where $s_{K}$ is the saving rate.

Equation (3) shows the development in the knowledge level, where $A_{t}$ is an expression for the knowledge at the time $t$ and $L_{A_{t}}$ is the amount of researchers. Equation (4) shows that the latter is the product of the population, $L_{t}$, and the share of researchers, $s_{R}$.

The following definitions will also be used throughout this project:

7. \\[ y_{t}=\frac{Y_{t}}{L_{Y_{t}}};k_{t}=\frac{K_{t}}{L_{Y_{t}}} \\]
8. \\[\tilde{y}=\frac{y_{t}}{A_{t}};\tilde{k}=\frac{k_{t}}{A_{t}} \\]

Where **7.** is the definition for capital-labo ratio and **8.** is the definition for the technology adjusted capital-labor ratio.

# The growth in technology
Our first goal in this model is to find the steady state rate for the exact growth rate of technology.

The following transitions equation is given for the growth in technology:

9. \\[g_{t+1}=\left(1+n\right)^{\lambda}g_{t}\left(1+g_{t}\right)^{\phi-1}\\]

We then wish to find the steady state for this equation. That is to solve for $g^*$ the following equation:
10. \\[g^*=\left(1+n\right)^{\lambda}g^*\left(1+g^*\right)^{\phi-1}\\]

```python
# Define parameters as symbols
g = sm.symbols('g')
n = sm.symbols('n')
l = sm.symbols('lambda')
phi = sm.symbols('phi')
```

```python
# Define the transition equation
trans_g = sm.Eq(1,(1+n)**(l)*1*(1+g)**(phi-1))
```

```python
# Solve for g
ss_g = sm.solve(trans_g,g)[0]
ss_g
```

Next we will estimate the numeric value of the steady state growth rate of technology by using realistic parameters.
$n = 0.01$
$\lambda = 1$
$\phi = 0.50$

```python
# Define function and print with given parameters
sol_func_g = sm.lambdify((n,l,phi),ss_g)
sol_num_g = sol_func_g(0.01,1,0.5)
f"The steady state growth rate of technology is {sol_num_g:.2f}"
```

    'The steady state growth rate of technology is 0.02'

By using these given parameters, the steady state rate of growth in technology is estimated to be 2%.

# Solving the "simple" model with R&D

Our next step is to solve the simple model with R&D.
The transition equation for capital accumulation can then be expressed as:

11. \\[ {\tilde{k}_{t+1}=\frac{1}{\left(1+n\right)\left(1+g_{t}\right)}\left(s_{k}\tilde{k}_{t}^{\alpha}+\left(1-\delta\right)\tilde{k}_{t}\right)} \\]

To find the solution for this model, we will have to find the steady state rate. That is to solve for $\tilde{k}$ in the following equation:

12. \\[{\tilde{k}^{*}=\frac{1}{\left(1+n\right)\left(1+g_{t}\right)}\left(s_{k}\tilde{k}^{\alpha*}+\left(1-\delta\right)\tilde{k}^{*}\right)} \\]

But first, we will have to define the following parameters and variables as symbols

```python
# Define variables and parameters as symbols
k = sm.symbols('k')
alpha = sm.symbols('alpha')
delta = sm.symbols('delta')
s_k = sm.symbols('s_K')
```

```python
# Define equation and solve for k
trans_k = sm.Eq(k,(s_k*k**alpha+(1-delta)*k)/((1+n)*(1+g)))
ss_k = sm.solve(trans_k,k)[0]
ss_k
```

Let's try to find the numeric value for the steady state growth rate of capital per effecient worker by using realistic paramters:
$$s_K = 0.2$$
$$g = 0.02$$
$$n = 0.01$$
$$\delta = 0.06$$
$$\alpha = 1/3 $$

```python
# Define function and print with parameters
sol_func_k = sm.lambdify((s_k,g,n,delta,alpha),ss_k)
sol_num_k = sol_func_k(0.2,0.02,0.01,0.06,1/3)
f"The Adjusted capital-labor ratio SS is {sol_num_k:.2f}"
```

    'The Adjusted capital-labor ratio SS is 3.30'

We could also estimate the rate by changing some parameters and holding the others fixed. Hence, we create a slider

```python
# Define function for the solution
def sol_num_k_func(s_k,g,n,delta,alpha):
    return f"The steady state growth rate of capital per effecient worker is {((delta + g * n + g + n)/s_k)**(1/(alpha-1)):.3f}"
```

```python
# Create slider
slider_g_r = widgets.interact(sol_num_k_func,
    s_k=widgets.FloatSlider(description="$s_k$", min=0.0, max=0.5, step=0.05, value=0.2),
    g=widgets.FloatSlider(description="$g$", min=0.0, max=0.1, step=0.01, value=0.02),
    n=widgets.FloatSlider(description="$n$", min=0.0, max=0.1, step=0.01, value=0.01),
    delta=widgets.FloatSlider(description="$\delta$", min=0.0, max=0.2, step=0.02, value=0.06),
    alpha=widgets.FloatSlider(description="$a$", min=0.1, max=0.67, step=0.03, value=1/3)                      )

```

    interactive(children=(FloatSlider(value=0.2, description='$s_k$', max=0.5, step=0.05), FloatSlider(value=0.02,…

## Visualization of the path towards steady state - The transition diagram
Our next goal is to create a plot for the transition diagram.

```python
# Define functions of the transition equation and the numeric SS solutions.
def trans_k_func(k, s_k, alpha, delta, n, g):
    return (s_k*k**alpha+(1-delta)*k)/((1+n)*(1+g))
def sol_val_k_func(s_k,g,n,delta,alpha):
    return  ((delta + g * n + g + n)/s_k)**(1/(alpha-1))
```

## Changing the value of the savings ratio
What will happen to our plot, if the savaings ratio was to be changed. We will explore this by creating a simple interactive slider.

**Please run the below cell twice to resize the plot propely**

```python
from ipywidgets import interactive

def slider(s_k):
    plt.figure(dpi = 100)
    plt.rc('text', usetex=True)
    plt.rc('font', family='sans-serif')

    #range for k
    k = np.arange(0.0, 15, 0.005)

    # paramters:
    s_K = 0.2
    g = 0.02
    n = 0.01
    delta = 0.06
    alpha = 1/3

    # label and title
    plt.xlabel(r'$k_t$',fontsize=20 )
    plt.ylabel(r'$k_{t+1}$', fontsize=20)
    plt.title(r'The transition diagram', fontsize=20)

    # plot the transition equation and 45-degree equation
    plt.plot(k, trans_k_func(k, s_k, alpha, delta, n, g))
    plt.plot(k, k)

    sol_val_k = sol_val_k_func(
                                delta = delta, g = g, n = n, s_k = s_k, alpha = alpha)

    # Legend
    plt.legend((r'$\frac{1}{\left(1+n\right)\left(1+g\right)}\left(s\tilde{k}_{t}^{\alpha}+\left(1-\delta\right)\tilde{k}_{t}\right)$',
            r'$\tilde{k}_{t+1}=\tilde{k}_{t}$'),
           shadow=True, handlelength=1.4, fontsize=12)

    # Create arrow, which points at SS
    plt.annotate( f'Adjusted capital-labor ratio SS {sol_val_k:.3f}',
                 xy=(sol_val_k, sol_val_k),
                 xytext=(sol_val_k-1, sol_val_k-3),
                arrowprops=dict(facecolor='black', shrink=0.01),
            )

# plot the figure and interactive slider
interactive_plot = interactive(slider,
                               s_k=widgets.FloatSlider(description="$s_k$",
                                                       min=0.0,
                                                       max=0.5,
                                                       step=0.04,
                                                       value=0.20,
                                                       continuous_update=True)
                              )

interactive_plot
```

    interactive(children=(FloatSlider(value=0.2, description='$s_k$', max=0.5, step=0.04), Output()), _dom_classes…

We see as the savings ratio increases, so does the steady state rate.

# Extending the model with human capital

We now want to extend the model and therefore adds *Human capital*.

Equation 1. will be swapped by:

1'. \\[Y_{t}=K_{t}^{\alpha}H_{t}^{\varphi}\left(A_{t}L_{Y_{t}}\right)^{1-\alpha-\varphi} \\]

And a new human capital accumulation equation is given as:

13. \\[H_{t+1}-H_{t}=s_{H}Y_{t}-\delta H_{t},0<\delta<1,0<s_{H}<1,H_{0}>0  \text{ provided} \\]

which will change equation (1) and (3). Additionally the equation below will be added to the model, which describes how human capital develops over time. Here $s_{H}$ is the saving rate in human capital. $\delta$ is the depreciation rate.
The following is the definition of per production worker variable and per effective production worker variable: $h_{t}=\frac{H_{t}}{L_{Y_{y}}}$ &  $\tilde{h}=\frac{h_{t}}{A_{t}}$

With human capital included in the model, a new transition equation is given for human capital per effecient worker and capital per effecient worker.

14. \\[ \tilde{h}_{t+1}=\frac{1}{\left(1+n\right)\left(1+g_{t}\right)}\left(s_{K}\tilde{k}_{t}^{\alpha}\tilde{h}_{t}^{\varphi}+\left(1-\delta\right)\tilde{h}_{t}\right) \\]

15. \\[ \tilde{k}_{t+1}=\frac{1}{\left(1+n\right)\left(1+g_{t}\right)}\left(s_{H}\tilde{k}_{t}^{\alpha}\tilde{h}_{t}^{\varphi}+\left(1-\delta\right)\tilde{k}_{t}\right)\\]

Similar to the previous simple model, we will have to solve the following two equations to find the steady state values.

  \\[ \tilde{k}^{*}=\frac{1}{\left(1+n\right)\left(1+g_{t}\right)}\left(s_{K}\tilde{k}^{*{\alpha}}\tilde{h}^{*{\varphi}}+\left(1-\delta\right)\tilde{k}^*\right)\\]

  \\[ \tilde{h}^{*}=\frac{1}{\left(1+n\right)\left(1+g_{t}\right)}\left(s_{H}\tilde{k}^{*{\alpha}}\tilde{h}^{*{\varphi}}+\left(1-\delta\right)\tilde{h}^*\right)\\]

```python
# Define parameters as symbols
h = sm.symbols('h')
k = sm.symbols('k')
s_h =sm.symbols('s_H')
s_k = sm.symbols('s_K')
g = sm.symbols('g')
n = sm.symbols('n')
delta = sm.symbols('delta')
vphi = sm.symbols('varphi')
alpha = sm.symbols('alpha')
vphi = sm.symbols('varphi')
```

```python
# Define the two new transition equations
trans_k2 = sm.Eq(k,(s_k*k**alpha*h**vphi+(1-delta)*k)/((1+n)*(1+g)))
trans_h = sm.Eq(h,(s_h*k**alpha*h**vphi+(1-delta)*h)/((1+n)*(1+g)))
```

```python
# Solve a system of two equations
ss_kh = sm.solve([trans_k2,trans_h], (k,h))[0]
ss_kh
```

While this might not be the most aesthetically pleasing solution, it is the correct.

Next we are able to find the steady state rate for GDP per effecient worker, $\tilde{y}^* = (\tilde{k})^\alpha*(\tilde{h})^\varphi $.

```python
# Find the steady state rate for GDP per effecient worker
ss_gdp_per_worker = ss_kh[0]**vphi*ss_kh[1]**alpha
ss_gdp_per_worker
```

Next we will estimate the numeric value of the steady state rate of:

* Capital per effecient worker, $k^*$
* Human Capital per effecient worker, $h^*$
* GDP per effecient worker, $y^*$

by using the following paramters:
$$s_K = 0.2$$
$$s_H = 0.2$$
$$g = 0.02$$
$$n = 0.01$$
$$\delta = 0.06$$
$$\alpha = \varphi = 1/3 $$

```python
# Define and insert parameters for capital
sol_func_k2 = sm.lambdify((s_k,s_h,g,n,delta,alpha,vphi),ss_kh[0])
sol_num_k2 = sol_func_k2(0.2,0.2,0.02,0.01,0.06,1/3,1/3)
f" The Adjusted capital-labor ratio SS is {sol_num_k2:.3f}"
```

    ' The Adjusted capital-labor ratio SS is 10.901'

```python
# Define and insert parameters for human capital
sol_func_h = sm.lambdify((s_k,s_h,g,n,delta,alpha,vphi),ss_kh[1])
sol_num_h = sol_func_h(0.2,0.2,0.02,0.01,0.06,1/3,1/3)
f"The Adjusted human capital-labor ratio SS is {sol_num_h:.3f}"
```

    'The Adjusted human capital-labor ratio SS is 10.901'

```python
# Define and insert parameters for GDP
sol_func_y = sm.lambdify((s_k,s_h,g,n,delta,alpha,vphi),ss_gdp_per_worker)
sol_num_y = sol_func_y(0.2,0.2,0.02,0.01,0.06,1/3,1/3)
f"The steady state rate for GDP pr. worker is {sol_num_y:.3f}"
```

    'The steady state rate for GDP pr. worker is 4.916'

## Visualization of the path towards steady state - The transition diagram
Our next goal is to create a plot for the transition diagram for the model including human capital.
First we have to find the "nullclines". In order for us to this, we first have to find the solow equations. This is done by substracting $\tilde{k}_{t}$ or $\tilde{h}_{t}$ from the respective transition equation.

16. \\[ \tilde{h}_{t+1}-\tilde{h}_{t}=\frac{1}{\left(1+n\right)\left(1+g_{t}\right)}\left(s_{K}\tilde{k}_{t}^{\alpha}\tilde{h}_{t}^{\varphi}+\left(1-\delta\right)\tilde{h}_{t}\right)-\tilde{h}_{t} \\]

17. \\[ \tilde{k}_{t+1}-\tilde{k}_{t}=\frac{1}{\left(1+n\right)\left(1+g_{t}\right)}\left(s_{H}\tilde{k}_{t}^{\alpha}\tilde{h}_{t}^{\varphi}+\left(1-\delta\right)\tilde{k}_{t}\right)-\tilde{k}_{t} \\]

To find the nullclines, we will have to solve for $\tilde{h}$ in the two solow equations.

```python
# Define the two solow equations
solow_k = sm.Eq(k-k,((s_k*k**alpha*h**vphi+(1-delta)*k)/((1+n)*(1+g))-k))
solow_h = sm.Eq(h-h,((s_h*k**alpha*h**vphi+(1-delta)*h)/((1+n)*(1+g))-h))
solow_k, solow_h
```

```python
# Now solve for each of the equation with respect to h
solow_sol_k = sm.solve(solow_k, h)[0]
solow_sol_h = sm.solve(solow_h, h)[0]
solow_sol_k, solow_sol_h
```

The first nullcine is labed as ($\Delta\tilde{k}_{t} = 0)$ and the second nullcline as ($\Delta\tilde{h}_{t} = 0)$.

```python
# Parameters
g = 0.02
n = 0.01
delta = 0.06
alpha = 1/3
vphi = 1/3

# Define the two null clines as functions
def nullcline_1(k, s_k, s_h, alpha, delta, n, g):
    return (((k**(-alpha+1)*(delta + g*n + g + n)) / s_k)**(1/vphi))
def nullcline_2(k, s_k, s_h, alpha, delta, n, g):
    return (((k**(-alpha)*(delta + g*n + g + n)) / s_h)**(1/(vphi-1)))
```

```python
def slider2(s_k, s_h):
    # Range for k
    k = np.arange(0.0,80.0,0.1)

    plt.figure(dpi=100)

    # labels and title
    plt.xlabel(r'$\tilde{k_t}$',fontsize=20, color = "black" )
    plt.ylabel(r'$\tilde{h_t}$', fontsize=20, color = "black")
    plt.title(r'The transition diagram with human capital', fontsize=20)

    # Plot the nullclines
    plt.plot(k, nullcline_1(k, s_k, s_h, alpha, delta, n, g))
    plt.plot(k, nullcline_2(k, s_k, s_h, alpha, delta, n, g))

    # Calculate the numeric SS values
    sol_val_h = sol_func_h(s_k,s_h,g,n,delta,alpha,vphi)
    sol_val_k2 = sol_func_k2(s_k,s_h,g,n,delta,alpha,vphi)

    # Dashed lines for SS values
    plt.axhline(y=sol_val_h, xmin=0., xmax=1, linestyle = '--', color = "black")
    plt.axvline(x=sol_val_k2, ymin=0., ymax=1, linestyle = '--', color = "black")

    # Legend
    plt.legend((r'$\left[\Delta\tilde{k}_{t}=0\right]$',
                    r'$\left[\Delta\tilde{h}_{t}=0\right]$'),
                   shadow=True, handlelength=1.4, fontsize=16)

    # labels for dashed lines
    plt.annotate(r'$\tilde{k}^{*}$', xy=(0, 0), xytext=(sol_val_k2, -5))
    plt.annotate(r'$\tilde{h}^{*}$', xy=(0, 0), xytext=(-5, sol_val_h))
    plt.axis([0,80,0,80])
    plt.show()

# plot the figure and interactive slider for s_k and s_h
interactive_plot = interactive(slider2,
                               s_k=widgets.FloatSlider(description="$s_k$",
                                                       min=0.0,
                                                       max=0.4,
                                                       step=0.04,
                                                       value=0.20,
                                                       continuous_update=True),
                               s_h=widgets.FloatSlider(description="$s_h$",
                                                       min=0.0,
                                                       max=0.4,
                                                       step=0.04,
                                                       value=0.20,
                                                       continuous_update=True)
                              )

interactive_plot
```

    interactive(children=(FloatSlider(value=0.2, description='$s_k$', max=0.4, step=0.04), FloatSlider(value=0.2, …

# Finding the steady state growth path for GDP per capita

Under the assumption that $\phi < 1 $, we'll try to derive the steady-state growth path for GDP per capita. \\[\hat{y}_{t}^{*}=\tilde{y}^{*}\left(1-s_{R}\right)\left(\frac{\rho}{g_{se}}\right)^{\frac{1}{1-\phi}}s_{R}^{\frac{\lambda}{1-\phi}}\left(1+g_{se}\right)^{t}L_{0}^{\frac{\lambda}{1-\phi}}\\]

```python
# Define the following parameters as symbols
y_hat = sm.symbols('\hat{y}_{t}^{*}')
y_tilde = sm.symbols(r'\tilde{y}^{*}')
l = sm.symbols('lambda')
phi = sm.symbols('phi')
s_r = sm.symbols('s_{R}')
rho = sm.symbols('rho')
g_se = sm.symbols('g_{se}')
t = sm.symbols('t')
L_0 = sm.symbols('L_{0}')
```

```python
# Defining the equation
ss_gp = y_tilde*(1-s_r)*(rho/g_se)**(1/(1-phi))*s_r**(l/(1-phi))*(1+g_se)**t*L_0**(l/(1-phi))
```

Next we will find the "Golden rule" for $s_R$, which is the optimal share of the population being researchers. This is done by differentiation with respect to $s_R$,$\frac{\partial\hat{y}_{t}^{*}}{\partial s_{R}}$,  and then solving for $s_R$.

```python
# Differentiate the equation with respect to s_R
diff_ss_gp = sm.diff(ss_gp,s_r)
diff_ss_gp
```

```python
#Solve for s_R
golden_rule = sm.solve(diff_ss_gp,s_r)[0]
golden_rule
```

As presented above, the optimal share of researchers is $\left[\frac{\lambda}{\lambda-\phi+1}\right]$. Next let's try to find the numerical value by using the following realistic parameters. $\lambda = 1, \phi = 0.5$

```python
# parametrize the golden rule
golden_rule_num = sm.lambdify((l,phi),golden_rule)
g_r = golden_rule_num(1,0.5)
f"The optimal share of researchers are {g_r:.3f}"
```

    'The optimal share of researchers are 0.667'

Last, let's create a simple slider for changing values of $\phi$ while holding $\lambda$ fixed.

```python
# Define function of the golden rule
def g_rule(l,phi):
    return f"s_r = {l/(1+l-phi):.3f}"
```

```python
# Create slider
slider_g_r = widgets.interact(g_rule,
    phi=widgets.FloatSlider(description="$\phi$", min=0.0, max=1.0, step=0.05, value=0.5),
    l=widgets.fixed(1)
                       )
```

    interactive(children=(FloatSlider(value=0.5, description='$\\phi$', max=1.0, step=0.05), Output()), _dom_class…

We see as $\phi$, which is a messaure for how capital affects "knowledge",  continues to grow the optimal share of researchers also increases.

# Overall conclusion and further perspectives

In this project, we have succeded in solving the solow model with R&D and extended it with human capital. The dynamic paths towards steady state have been visualized by using realistic parameters. We have also seen how the steady state rate develops, when some of the parameters are changed.

To extend this project more simulations and visualization are posible options.


# Big Assignment 3

## Background

The data for this exercise were used in Ebonya Washington's paper: "Female Socialization: How Daughters Affect Their Legislator Fathers' Voting on Women's Issues."  published in the American Economic Review in 2008. The paper asks whether having daughters influences the voting behavior of members of the US Congress. The hypothesis is that having (more) daughters makes legislators more likely to vote liberally on issues concerning women.

For this exercise, we will focus on votes that took place in the 108th Congress, which held session in 2003/04. As a measure of a liberal voting record, we use scores assigned by the American Association of University Women (AAUW), a liberal group that concerns itself with issues of interest to women. For the 108th Congress, the AAUW selected 9 pieces of legislation in the areas of education, equality and reproductive rights. The AAUW then assigned a score to each member of Congress. The scores range from 0 to 100 and measure the percentage of times the legislator voted in favor of the position held by the AAUW.

 The dataset `legislators.dta` contains the following characteristics for 320 members of the 108th Congress:

 * $ngirls$ number of daughters
 * $totchi$ number of children
 * $age$ Age
 * $female$ indicator for being female
 * $repub$ indicator for being a Republican
 *  $moredef$ proportion of people in the legislator's district who are in favor of "more spending on
defense"
 *  $aauw$ AAUW score

### (a) Estimate and report results for the following regression models:

Load in the data set `legislators.dta`. Remember, you will first need to call the `haven` package to do so.

Generate a variable `ngirls2` $=\text{ngirls}^2$

Generate an interaction variable `repubngirls` $= \text{repub}*\text{ngirls}$

Generate an interaction variable `repubngirls2` $=\text{repub}*\text{ngirls2}$

Estimate the following three regression models:

\begin{align}
aauw&=\beta_0 +\beta_1female+\beta_2repub+\beta_3ngirls+u  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ (1) \\
aauw&=\beta_0 +\beta_1female+\beta_2repub+\beta_3ngirls+\beta_4 ngirls2+u ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~(2)  \\
aauw&=\beta_0 + \beta_1female+\beta_2repub+\beta_3ngirls+\beta_4 ngirls2+\beta_5 repubngirls+\beta_6 repubngirls2+\beta_7 totchi + \beta_8 moredef + u\ \ \ (3)
\end{align}

```R
# Add Code for part (a) here.
```

### (b) Suggest which model is the best fit to the data

Write your answer here.

### (c) Interpret the effect of `ngirls` on `aauw` in each model

```R
# Add code for part (c) here if needed
```

Write your answer here.

### (d) Test whether there is an effect of the number of daughters on aauw scores using the second model. Be sure to describe carefully the null and alternative hypothesis.

Step 1: Estimate the restricted model.

Step 2: Calculate the sum of squared residuals from the restricted and unrestricted models.

Step 3: Apply the F-stat formula.

Alternative (easier):

Step 2 (alternative): Find the $R^2$ in the summary of the restricted and unrestricted models.

Step 3 (alternative): Apply the $R^2$ version of the F-stat formula.

Using model 2, the null hypothesis is that $\beta_3 = \beta_4 = 0$. The alternative hypothesis is that either $\beta_3$ or $\beta_4$ (or both) are not zero.

```R
#Write code for part (d) here

```

Write your answer here.

### (e) Using the third model, predict the aauw score for a male democrat who has 1 daughter and 1 son, and who has 20% of constituents who want more spending on defense. Suggest a $95\%$ CI for that predicted value.

Hint: See part 2-A of section notes 7.

```R
#Write code for part (e) here.
```

Add your written answer for part (e) here.

### (f) Suppose a male Democrat with 1 daughter and 1 son defeats the incumbent Senator in a state where 20% of constituents want more spending on defense. Generate a $95\%$ CI for his particular AAUW score, continuing to use the third model.

Hint: See part 2-B of Section Notes 7.

```R
#Write any code you need for this part here
```

Write your answer here.

### (g) Interpret the coeffcient $\beta_2$ from model (3).

Write your answer here

### (h) Suppose you think Republicans and non-Republicans may have different gender patterns in voting. That is, republican men may vote differently than Republican women, who may vote differently than Democratic women who may vote differently than Democratic men. Write down an estimation equaiton to test whether republican women, Democratic women, and Democratic men each vote differently than Republican men.

Write your answer here.

### (i) Implement your test. Interpret each coeffcient.

```R
#Write your code here
```

Write your answer here.

### (j) Adapt your regression to test whether democratic women vote differently than republican women. Please report your results (write out the estimating equations).

```R
#Add your code for part j here
```

Write your answer here.

## Downloading your Notebook

Download a PDF copy of your notebook by using __File > Print Preview.

Once you have generated the print preview, use Print > Save as PDF to create the PDF.


# Variational Bayesian Inference

In this notebook, we will review the variational Bayes process, beginning with a technical introduction to the formalism and derivation, followed by a python implementation.
The material covered here references Blei et al., 2018, Varitional Inference: A Review for Statisticians and Chappel et al., 2016. The FMRIB Variational Bayes Tutorial.

## Technical Overview

### Problem Statement

Similarly to problems addressed by sampling methods, the goal of variational inference (VI) is to approximate parameter distributions from data, specifically in cases where an analytical treatment is intractable.

Consider the following example (from Blei et al., 2018):

For latent variabels $\mathbf{z} = z_{1:m}$ and observations $\mathbf{x} = x_{1:n}$, the posterior conditional density is given by :

$$ p(\mathbf{z} \mid \mathbf{x}) = \frac{p(\mathbf{x} \mid \mathbf{z}) * p(\mathbf{z})}{p(\mathbf{x})} $$

The denominator, ${p(\mathbf{x})}$, whose value is needed to compute the posterior, is calculated by:

$$ {p(\mathbf{x})} = \int p(\mathbf{z} , \mathbf{x}) dz $$

This integral is often intractable or too computationally expensive to be feasible. For similar reasons (notably the number of latent variables)sampling methods are slow to converge.

VI aims to circumvent the large time complexity by approaching the problem through optimisation.

The process begins by positing a contrived _approximate_ density, $\mathfrak{D}$ of latent variables $\mathbf{z}$. Then, using this density can find a set of valuers for $q(\mathbf{z}) \in \mathfrak{D}$ whose values maximise the Kullback-Liebler divergence between the approximate density and the true posterior.

$$ q^{*}(\mathbf{z}) = \underset{q(\mathbf{z}) \in \mathfrak{D}}{argmin} \mathrm{KL}(q(\mathbf{z}) \mid \mid p(\mathbf{z} \mid \mathbf{x})) $$

### VI Computation

To compute the optimsation problem stated above requires we do manipulation of the terms. This is because the problem is still dependent on the _evidence_ term ($p(\mathbf{x})$; recall, the intragral of this is often difficult/intractable). To see this clearly, we can rewrite out expression using Bayes rule:

$$ q^{*}(\mathbf{z}) = \underset{q(\mathbf{z}) \in \mathfrak{D}}{argmin} \mathrm{KL}\bigg(q(\mathbf{z}) \mid \mid \frac{p(\mathbf{z}, \mathbf{x})}{p(\mathbf{x})}\bigg) $$

We can avoid evidence term using the following manipulations. First, let's break down the KL divergence, recalling that the KL divergence can otherwise be expressed as the expectated value of the first argument minus the expected value of the second argument:

$$ KL(q(\mathbf{z}) \mid \mid p(\mathbf{z} \mid \mathbf{x})) = \mathbb{E}[\log q(\mathbf{z})] - \mathbb{E}[\log p(\mathbf{z} , \mathbf{x})] + \log p(\mathbf{x}) $$

We formulate the Evidence Lower Bound (ELBO) by multiplying both sides by -1:

\begin{align}
 ELBO(q) &= - KL(q(\mathbf{z}) \mid \mid p(\mathbf{z} \mid \mathbf{x})) + \log p(\mathbf{x}) \\

         &= \mathbb{E}[\log p(\mathbf{z}, \mathbf{x})] - \mathbb{E}[\log q(\mathbf{z})]
\end{align}

Note that, maximsing the ELBO is the same as minimising the KL divergence, since the evidence is constant and additive.

By exanding the joint probaibility, we can rewrite the ELBO in terms of the log likelihood and KL divergence between our prior $p(\mathbf{z})$ and our variational density $q(\mathbf{z})$:

\begin{align}
ELBO(q) &= \mathbb{E}[\log p(\mathbf{x} \mid \mathbf{z})] + \mathbb{E}[\log p(\mathbf{z})] - \mathbb{E}[\log q(\mathbf{z})] \\

&= \mathbb{E}[\log p(\mathbf{x} \mid \mathbf{z})] - KL(q(\mathbf{z}) \mid \mid p(\mathbf{z}))
\end{align}

By maximising the ELBO or minimising the KL divergence, we can solve the optimisation problem. Now, we need a method to do this.

### Optimisation

There are many method available for solving the optimsation problem. Here we appeal to variational calculus (hence, variational inference).

Full derivation to follow...

Optimisation equates to updating parameters w.r.t the expectation of all other parameters given the current posterior:

$$ \log q(z_i) \approx \int q(z_{-i}) \log p(x \mid z) p(w) dw_{-i}$$

## Example

We will work with an exponential decay model to explore the variational Bayes framework and derive further equations needed.

### Model Definition

$$ y = ae^{-\lambda t} + \epsilon $$
$$ \epsilon = \mathcal{N}(0,\phi^{1}) $$

We can define this using a simple lambda function and some parameter values.

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
def f(theta, t):
    """ Sipmle exponential decay model

    args:
    theta : array, float
            parameters for function
        t : array, float
            timesteps to evaluate at
    """
    a, l = theta
    f1 = a * np.exp(-l * t)

    return f1
```

```python
a = 42
l = 1.5
theta = [a, l]
t = np.linspace(0,20,100)

phi = 100
```

```python
model = f(theta, t)
data = f(theta, t) + np.random.normal(0, np.sqrt(1/10), t.shape[0])
```

```python
plt.plot(t, model, linewidth = 3, alpha = 0.6)
plt.scatter(t, data, c = 'orange')
plt.show()
```

#### Forward Model Specification

We can set up a forard model in the following way:

$$ \mathbf{y} = \mathbf{g} + \epsilon $$

$$ \mathbf{g} = \{g(\theta_{1}), . . ., g(\theta_{k})\} $$

$$ \epsilon \approx \mathcal{N}(0, \Phi^{-1}) $$

$$ \Phi = \{\phi_1, . . ., \phi_n\} $$

Where $\mathbf{y}$ is a vector of N observation assumed to be generated by a nonlinear forward function $g(\theta) + \epsilon $ with $k$ parameters, \theta, and $k$ precision parameters, $\phi$.

#### Inference Specification

Using this forward model, we can parameterise the log-likelihood as multivariate normal distribution, with the data conditioned on parameters:

$$ p(\mathbf{y} \mid \mathbf{\Theta}) = \frac{N}{2} \log(\Phi) - \frac{1}{2}(\mathbf{y} - \mathbf{g})^{T} \Phi (\mathbf{y} - \mathbf{g}) $$

Where $\Theta$ is the set of independent parameters. We factorise the approximate posterior accordingly:

$$ q(\Theta \mid \mathbf{y}) = q(\theta_n)q(\phi_n) $$

and choose conjucate distributions for their form:

$$ q(\theta \mid \mathbf{y}) \approx MVN(\theta; \mathbf{m}, \Lambda^{-1}) $$

$$ q(\phi \mid \mathbf{y}) \approx Ga(\phi; s, c) $$

Using these, we can construct the log posterior and thus update equations.

\begin{align}
L &= \log p(\mathbf{y} \mid \mathbf{\Theta}) + \log p(\theta) + \log p(\phi) \\
&= \underbrace{\frac{N}{2} \log(\Phi) - \frac{1}{2}(\mathbf{y} - \mathbf{g})^{T} \Phi (\mathbf{y} - \mathbf{g})}_{\log p(\mathbf{y} \mid \mathbf{\Theta})}
- \underbrace{\frac{1}{2}(\theta - \mathbf{m_0})^{T} \Lambda_0 (\theta - \mathbf{m_0})}_{\log p(\theta)}
+ \underbrace{(c_0 - 1)\log\phi - \frac{1}{s_0} \phi}_{\log p(\phi)}
+ \underbrace{const\{\theta, \phi\}}_{\text{constant terms}}
\end{align}

##### Compuational side note:

We will need to make an approximation to the function $g(\theta)$ for mathematical and computational convenience. Simply, we will take a first order Taylor expansion around the mean of the posterior distribution.

$$ g(\theta) \approx g(\mathbf{m}) + \mathbf{J}(\theta - \mathbf{m}) $$

Where $\mathbf{m}$ is the mean value and $\mathbf{J}$ is the Jacobian matrix.

In practice, we will compute this approximation using a simply central difference scheme. Let's define some functions to do this for us.

```python
def time_step(theta):
    """Control time stepping interval proportional to the parameter magnitiude.

    args:
    theta:: float
    """

    delta = theta * 1e-5

    if delta < 0:
            delta = -delta
    if delta < 1e-10:
            delta = 1e-10

    return delta

def central_difference(i, theta, delta, t):
    """ Calculate the derivate using a first order central difference approximation

    args:
        i : int
            parameter index
    theta : np.array float
            array of paramters for function, f
    delta : float
            time-step for central difference
        t : np.array float
            time steps to evaluate at
    returns:
        df : np.array, float
             first order approximation, df, for time steps, t
    """
    dtheta = np.array(theta), np.array(theta)

    dtheta[0][i] += delta
    dtheta[1][i] -= delta
    df = (f(dtheta[0], t) - f(dtheta[1], t)) / (2 * delta)
    return df

def Jacobian(theta, t):
    """Compute the Jacobian for globally defined function, f, with parameter set Theta

    args:
    theta : np.array, float
            parameters of global function, f
        t : np.array, float
            time steps to evaluate at

    returns:
        J : np.array, float
            Jacobian vector/matrix evaluated at theta, t.
    """
    J = None
    #f_n = len(f)
    p_n = len(theta)

    for i in range(p_n):
        delta = time_step(theta[i])
        if J is None:
                J = np.zeros([len(t), len(theta)], dtype=np.float32)
        J[:,i] = central_difference(i, theta, delta, t)

    return J
```

### Deriving updates

For our model, we will need to derive updates on our model parameters $\theta$ and noise parameters $phi$.

Recall the the derivation for our updates w.r.t. the free energy functional:

$$ p(\theta_{i}) = \int p(\Theta \mid y) p(\theta_{-i}) d\theta_{-i} $$

Where $\theta_{-i}$ are all parameters that are not $\theta$. In our case, we have two parameters $\theta$ and $\phi$. Which will be given by:

$$ p(\theta) = \int p(\Theta \mid y) p(\phi) d\phi $$

and

$$ p(\phi) = \int p(\Theta \mid y) p(\theta) d\theta $$

#### Update on theta

We will first derive our update on model parameters $\theta$.

Recall the forms of the log posterior and parameters. With these, we can write down the full form of the update equation:

$$ q(\theta) =  \int \bigg( \frac{N}{2} \log(\phi) - \frac{1}{2}(\mathbf{y} - \mathbf{g})^{T} \Phi (\mathbf{y} - \mathbf{g}) - \frac{1}{2}(\theta - \mathbf{m_0})^{T} \Lambda_0 (\theta - \mathbf{m_0}) + (c_0 - 1)\log\phi - \frac{1}{s_0} \phi + const\{\theta, \phi\} \bigg) Ga(\phi; s, c) d\phi $$

Where $\mathbf{g}$ is an abbreviation for the function $g(\theta)$. We can start simplifying this by first removing terms from that are constant w.r.t. $\theta$ (remember, we are trying to optimise $\theta$) and factoring out terms in the intregral constant w.r.t. $\phi$.

In so doing, we arrive at the following:
$$ q(\theta) =  - \frac{1}{2}(\theta - \mathbf{m_0})^{T} \Lambda_0 (\theta - \mathbf{m_0}) - \frac{1}{2}(\mathbf{y} - \mathbf{g})^{T}(\mathbf{y} - \mathbf{g}) \int \phi Ga(\phi; s, c) d\phi + const{\theta} $$

The integral term corresponds to the mode of the Gamme distribution, which, since we are using precision (the inverse of variance), is equal to $sc$. Thus, we have:

$$ q(\theta) =  - \frac{1}{2}(\theta - \mathbf{m_0})^{T} \Lambda_0 (\theta - \mathbf{m_0}) - \frac{1}{2}sc(\mathbf{y} - \mathbf{g})^{T}(\mathbf{y} - \mathbf{g}) + const{\theta} $$

To complete the derivation, we substitute the function $\mathbf{g}$ for it's Taylor approximation, given above and restated here.

$$ g(\theta) \approx g(\mathbf{m}) + \mathbf{J}(\theta - \mathbf{m}) $$
$$ \mathbf{y} - \mathbf{g} = \mathbf{k} + \mathbf{J}(\theta - \mathbf{m})$$

Substituting this in and droppping the constant term, we obtain:
$$ q(\theta) = - \frac{1}{2}(\theta - \mathbf{m_0})^{T} \Lambda_0 (\theta - \mathbf{m_0}) - \frac{1}{2}sc(\mathbf{k} + \mathbf{J}(\theta - \mathbf{m}))^{T}(\mathbf{k} + \mathbf{J}(\theta - \mathbf{m})$$

Expanding this out:
$$q(\theta) = -\frac{1}{2} \bigg( \theta^{T}\Lambda_0\theta + \theta^{T}\Lambda_0 \mathbf{m_0} + m_0^{T}\Lambda_0\theta + \mathbf{m_0}\Lambda_0 \mathbf{m_0} sc (\theta^{T}\mathbf{J}^{T}\mathbf{J}\theta + \theta^{T}\mathbf{J}^{T}\mathbf{k} - \theta^{T}\mathbf{J}^{T}\mathbf{Jm} + \mathbf{kJ}\theta + \mathbf{m}^{T}\mathbf{J}^{T}\mathbf{J}\theta) \bigg) $$

We can then exlude terms that do not depend on $\theta$ and group like terms together

$$ = -\frac{1}{2} \bigg(\theta^{T}\Lambda_0\theta + sc \theta^{T}\mathbf{J}^{T}\mathbf{J}\theta  + \theta^{T}\Lambda_0 \mathbf{m_0} + \theta^{T}\mathbf{J}^{T}k - \theta^{T}\mathbf{J}^{T}\mathbf{Jm} + \mathbf{m_0}^{T}\Lambda_0\theta + \mathbf{kJ}\theta + \mathbf{m}^{T}\mathbf{J}^{T}\mathbf{J}\theta \bigg) $$

Which can be factorised as:

$$ = -\frac{1}{2} \bigg(\theta^T(\Lambda_0 + dc \mathbf{J}^T \mathbf{J})\theta - \theta^T(\Lambda_0 \mathbf{m_0} + sc\mathbf{J}(\mathbf{k + Jm})) -(\Lambda_0 \mathbf{m_0} + sc\mathbf{J}(\mathbf{k + Jm})^T \theta \bigg) $$

Remember we specify the log posterior as a multivariate normal distribution. Thus, analysing the corresponding coefficients given by the multivariate normal form, we arrive at the update equations:

$$
\Lambda = sc \mathbf{J}^T \mathbf{J} + \Lambda_0
\\
\\
\text{and}
\\
\\
\mathbf{m} = \Lambda^{-1} ( sc \mathbf{J}^T (\mathbf{k + Jm}) + \Lambda_0 \mathbf{m_0})
$$

We can write a simple function to compute this update:

```python
def parameter_update(error, params, priors, J):
    """ Update forward model function parameters theta in accordance with the update equations above

    args:
        error : array, float
                vector of the difference between observations and model prediction
       params : tuple
                parameters values
       priors : tuple
                priors values
            J : array, float
                array of Jacbian values from the model given parameter updates (calculated above)
    returns:
       params : tuple
                updated parameters values
    """
    m, p, s, c = params
    m0, p0, _, _ = priors

    p_new = s*c*np.dot(J.transpose(), J) + p0
    c_new = np.linalg.inv(p_new)
    m_new = np.dot(c_new, (s * c * np.dot(J.transpose(), (error +    np.dot(J, m))) + np.dot(p0, m0)))

    params[0][:], params[1][:] = m_new, p_new

    return params
```

#### Update on noise

We now move on to deriving update equations for noise. Expanding on the form above, the full form of the update equation will be:

$$ q(\phi) =  \int \bigg( \frac{N}{2} \log(\phi) - \frac{1}{2}(\mathbf{y} - \mathbf{g})^{T} \Phi (\mathbf{y} - \mathbf{g}) - \frac{1}{2}(\theta - \mathbf{m_0})^{T} \Lambda_0 (\theta - \mathbf{m_0}) + (c_0 - 1)\log\phi - \frac{1}{s_0} \phi + const\{\theta, \phi\} \bigg) MVN(\theta; m, \Lambda) d\theta $$

As before, we can remove terms from the integral that are constant w.r.t. $\theta$ and drop terms that are constant w.r.t. $\phi$. Thus we have:

$$ = \bigg( \frac{N}{2} + c_0 -1 \bigg)\log\phi - \frac{1}{s_0} \phi - \frac{1}{2} \phi \int (\mathbf{y} - \mathbf{g})^{T}(\mathbf{y} - \mathbf{g}) MVN(\theta; m, \Lambda) d\theta $$

Swapping in our linearisation for $g(\theta)$ and expanding out, we obtain:

$$ = \bigg( \frac{N}{2} + c_0 -1 \bigg)\log\phi - \frac{1}{s_0} \phi - \frac{1}{2} \phi \int \bigg( \mathbf{k}^T\mathbf{k} - (\theta - \mathbf{m})^T\mathbf{J}^T\mathbf{k} - \mathbf{k}^T\mathbf{J}(\theta - \mathbf{m}) + (\theta - \mathbf{m})^T\mathbf{J}^T\mathbf{J}(\theta - \mathbf{m})\bigg)MVN(\theta; m, \Lambda) d\theta $$

When expanded out with the multivariate normal on $\theta$, the integrals of the secona and third terms cancel out, since it generates an expectation on the MVN (equalling the mean), m is constant with $d\theta$ and the integral of the MVN is equal 1; thus, we have $m - m = 0$. We also make use of the general rule result:

$$ \int (\theta - \mathbf{m})^T\mathbf{U}(\theta - \mathbf{m})MVN(\theta; m, \Lambda) d\theta = Tr(\Lambda^{-1} U)  $$

Therefore, the result of intial integral is equal to:
$$ q(\phi) = \bigg( \frac{N}{2} + c_0 -1 \bigg)\log\phi - \frac{1}{s_0} \phi - \frac{1}{2} \phi \bigg(\mathbf{k}^T\mathbf{k} + Tr(\Lambda^{-1} \mathbf{J}^T \mathbf{J})\bigg) $$

Comparing this result with the form of a gamma distribution, we can compare coefficients for its moments and see that the update equations will be:

$$ c = \frac{N}{2} + c_0 $$
$$ \frac{1}{s} = \frac{1}{s_0} + \frac{1}{2}\mathbf{k}^T\mathbf{k} + Tr(\Lambda^{-1} \mathbf{J}^T \mathbf{J}) $$

Once again, we can compte a simple function to compute this.

```python
def noise_update(error, params, priors, J):
    """ Update forward model function parameters phi in accordance with the update equations above

    args:
        error : array, float
                vector of the difference between observations and model prediction
       params : tuple
                parameters values
       priors : tuple
                priors values
            J : array, float
                array of Jacbian values from the model given parameter updates (calculated above)
    returns:
       params : tuple
                updated parameters values
    """
    _, p, _, _ = params
    _, _, s0, c0 = priors

    N = len(data)

    c = np.linalg.inv(p)
    c_new = N/2 + c0
    s_new = 1/(1/s0 + 1/2 * np.dot(error.transpose(), error) + 1/2 * np.trace(np.dot(c, np.dot(J.transpose(), J))))

    params[2][:], params[3][:] = c_new, s_new

    return params
```

```python
def error_update(y, theta, t):
    """Calculate difference between data and model with updated parameters

    args:
        y : array, float
            vector of noisy data (observations)
    theta : array, float
            vector of parameter values for model, f
        t : array, float
            vector of time steps at which to evaluate model

    returns:
    error : array, float
            vector of difference between noisy data and updated model
    """

    error = y - f(theta[:2], t)

    return error
```

```python
def fit(data, params, priors, t, n):

    theta = np.zeros((n, len(params[0])))
    for i in range(n):
        theta[i,:] = params[0]
        error = error_update(data, theta[i,:], t)

        J = Jacobian(theta[i,:], t)
        params = parameter_update(error, params, priors, J)
        params = noise_update(error, params, priors, J)
        print(params)

    return params, theta

```

```python
m = np.array([1.0, 1.0])
C = np.array([[1.0, 0], [0.0, 1.0]])
p = np.linalg.inv(C)
c = np.array([1e-8])
s = np.array([50.0])

init_params = m, p, c, s
```

```python
m0 = np.array([1.0, 1.0])
a_var0, lam_var0 = np.array([1000, 1.0])
C0 = np.array([[a_var0, 0], [0, lam_var0]])
p0 = np.linalg.inv(C0)

beta_mean0 = 1
beta_var0 = 1000
# c=scale, s=shape parameters for Gamma distribution
c0 = beta_var0 / beta_mean0
s0 = beta_mean0**2 / beta_var0

priors = m0, p0, c0, s0
```

```python
params, theta_n = fit(data, init_params, priors, t, n=100)
```

    (array([1.05144648, 0.99998455]), array([[ 1.00150429e-03, -6.10400558e-07],
           [-6.10400558e-07,  1.00000062e+00]]), array([50.001]), array([0.00029803]))
    (array([34.56800553,  1.15395915]), array([[ 0.04583344, -0.0191285 ],
           [-0.0191285 ,  1.02038578]]), array([50.001]), array([0.00053172]))
    (array([40.64869248,  1.44492058]), array([[ 0.07234559, -0.83878455],
           [-0.83878455, 26.58017469]]), array([50.001]), array([0.01089733]))
    (array([42.28314592,  1.52020275]), array([[  1.23311602, -12.76151673],
           [-12.76151673, 370.1479491 ]]), array([50.001]), array([0.1073437]))
    (array([42.36458576,  1.52407141]), array([[  11.69599848, -117.77508799],
           [-117.77508799, 3379.16226228]]), array([50.001]), array([0.18443964]))
    (array([42.36780555,  1.52424174]), array([[  20.05859863, -201.69264506],
           [-201.69264506, 5783.49832491]]), array([50.001]), array([0.1873373]))
    (array([42.36789104,  1.52424657]), array([[  20.37206743, -204.82975272],
           [-204.82975272, 5873.26545289]]), array([50.001]), array([0.1873973]))
    (array([42.36787601,  1.52424564]), array([[  20.37854732, -204.89440961],
           [-204.89440961, 5875.11537456]]), array([50.001]), array([0.18739853]))
    (array([42.36789893,  1.52424693]), array([[  20.37868699, -204.89597897],
           [-204.89597897, 5875.15942469]]), array([50.001]), array([0.18739855]))
    (array([42.36789078,  1.52424619]), array([[  20.3786758 , -204.89571521],
           [-204.89571521, 5875.1520541 ]]), array([50.001]), array([0.18739855]))
    (array([42.36788439,  1.52424595]), array([[  20.37868544, -204.89590102],
           [-204.89590102, 5875.15912025]]), array([50.001]), array([0.18739855]))
    (array([42.36788341,  1.52424597]), array([[  20.37868739, -204.89591606],
           [-204.89591606, 5875.1596111 ]]), array([50.001]), array([0.18739855]))
    (array([42.36788275,  1.52424588]), array([[  20.37868723, -204.89591442],
           [-204.89591442, 5875.15956415]]), array([50.001]), array([0.18739855]))
    (array([42.36788189,  1.52424574]), array([[  20.37868721, -204.89591426],
           [-204.89591426, 5875.1601314 ]]), array([50.001]), array([0.18739855]))
    (array([42.36789461,  1.52424654]), array([[  20.37868718, -204.89598545],
           [-204.89598545, 5875.16126659]]), array([50.001]), array([0.18739855]))
    (array([42.36788603,  1.52424637]), array([[  20.37868267, -204.89582414],
           [-204.89582414, 5875.15496248]]), array([50.001]), array([0.18739855]))
    (array([42.36788979,  1.52424638]), array([[  20.37868305, -204.895828  ],
           [-204.895828  , 5875.15507304]]), array([50.001]), array([0.18739855]))
    (array([42.36789127,  1.52424648]), array([[  20.37868288, -204.89584412],
           [-204.89584412, 5875.1555949 ]]), array([50.001]), array([0.18739855]))
    (array([42.36789314,  1.52424666]), array([[  20.37868294, -204.89584475],
           [-204.89584475, 5875.15504102]]), array([50.001]), array([0.18739855]))
    (array([42.36788   ,  1.52424599]), array([[  20.37868299, -204.89577375],
           [-204.89577375, 5875.15333914]]), array([50.001]), array([0.18739855]))
    (array([42.36788298,  1.52424591]), array([[  20.37868753, -204.89589957],
           [-204.89589957, 5875.15850675]]), array([50.001]), array([0.18739855]))
    (array([42.36788158,  1.52424569]), array([[  20.37868716, -204.89591378],
           [-204.89591378, 5875.16011772]]), array([50.001]), array([0.18739855]))
    (array([42.36789744,  1.52424691]), array([[  20.37868942, -204.89602129],
           [-204.89602129, 5875.16126927]]), array([50.001]), array([0.18739855]))
    (array([42.3678873 ,  1.52424594]), array([[  20.37867595, -204.89569888],
           [-204.89569888, 5875.15209841]]), array([50.001]), array([0.18739855]))
    (array([42.36788342,  1.52424588]), array([[  20.37868764, -204.89593648],
           [-204.89593648, 5875.16082799]]), array([50.001]), array([0.18739855]))
    (array([42.3678822 ,  1.52424578]), array([[  20.37868728, -204.89591494],
           [-204.89591494, 5875.16015096]]), array([50.001]), array([0.18739855]))
    (array([42.36789267,  1.52424644]), array([[  20.37868719, -204.8959677 ],
           [-204.8959677 , 5875.16069824]]), array([50.001]), array([0.18739855]))
    (array([42.36789481,  1.52424675]), array([[  20.37868268, -204.89586002],
           [-204.89586002, 5875.15553835]]), array([50.001]), array([0.18739855]))
    (array([42.36787806,  1.52424563]), array([[  20.37868076, -204.8957381 ],
           [-204.8957381 , 5875.15334198]]), array([50.001]), array([0.18739855]))
    (array([42.36790125,  1.5242471 ]), array([[  20.37868981, -204.89602515],
           [-204.89602515, 5875.16080803]]), array([50.001]), array([0.18739855]))
    (array([42.3678952 ,  1.52424669]), array([[  20.37867365, -204.89568037],
           [-204.89568037, 5875.14979245]]), array([50.001]), array([0.18739855]))
    (array([42.36787708,  1.5242458 ]), array([[  20.37868328, -204.89577668],
           [-204.89577668, 5875.15399516]]), array([50.001]), array([0.18739855]))
    (array([42.3678885 ,  1.52424611]), array([[  20.37868757, -204.89593577],
           [-204.89593577, 5875.15966363]]), array([50.001]), array([0.18739855]))
    (array([42.36787837,  1.52424565]), array([[  20.37868708, -204.89589503],
           [-204.89589503, 5875.15952039]]), array([50.001]), array([0.18739855]))
    (array([42.36790124,  1.5242471 ]), array([[  20.37868956, -204.89602263],
           [-204.89602263, 5875.16073582]]), array([50.001]), array([0.18739855]))
    (array([42.3678952 ,  1.52424669]), array([[  20.37867366, -204.8956804 ],
           [-204.8956804 , 5875.14979346]]), array([50.001]), array([0.18739855]))
    (array([42.36787714,  1.52424581]), array([[  20.37868328, -204.89577668],
           [-204.89577668, 5875.15399505]]), array([50.001]), array([0.18739855]))
    (array([42.36788829,  1.52424609]), array([[  20.37868757, -204.89593577],
           [-204.89593577, 5875.15966367]]), array([50.001]), array([0.18739855]))
    (array([42.36787801,  1.52424563]), array([[  20.37868708, -204.89589505],
           [-204.89589505, 5875.15952098]]), array([50.001]), array([0.18739855]))
    (array([42.36790126,  1.5242471 ]), array([[  20.37868955, -204.89602254],
           [-204.89602254, 5875.16073317]]), array([50.001]), array([0.18739855]))
    (array([42.3678952 ,  1.52424669]), array([[  20.37867364, -204.8956803 ],
           [-204.8956803 , 5875.14979046]]), array([50.001]), array([0.18739855]))
    (array([42.36787708,  1.5242458 ]), array([[  20.37868328, -204.89577668],
           [-204.89577668, 5875.15399522]]), array([50.001]), array([0.18739855]))
    (array([42.3678885 ,  1.52424611]), array([[  20.37868757, -204.89593577],
           [-204.89593577, 5875.15966363]]), array([50.001]), array([0.18739855]))
    (array([42.36787837,  1.52424565]), array([[  20.37868708, -204.89589503],
           [-204.89589503, 5875.15952039]]), array([50.001]), array([0.18739855]))
    (array([42.36790124,  1.5242471 ]), array([[  20.37868956, -204.89602263],
           [-204.89602263, 5875.16073582]]), array([50.001]), array([0.18739855]))
    (array([42.3678952 ,  1.52424669]), array([[  20.37867366, -204.8956804 ],
           [-204.8956804 , 5875.14979346]]), array([50.001]), array([0.18739855]))
    (array([42.36787714,  1.52424581]), array([[  20.37868328, -204.89577668],
           [-204.89577668, 5875.15399505]]), array([50.001]), array([0.18739855]))
    (array([42.36788829,  1.52424609]), array([[  20.37868757, -204.89593577],
           [-204.89593577, 5875.15966367]]), array([50.001]), array([0.18739855]))
    (array([42.36787801,  1.52424563]), array([[  20.37868708, -204.89589505],
           [-204.89589505, 5875.15952098]]), array([50.001]), array([0.18739855]))
    (array([42.36790126,  1.5242471 ]), array([[  20.37868955, -204.89602254],
           [-204.89602254, 5875.16073317]]), array([50.001]), array([0.18739855]))
    (array([42.3678952 ,  1.52424669]), array([[  20.37867364, -204.8956803 ],
           [-204.8956803 , 5875.14979046]]), array([50.001]), array([0.18739855]))
    (array([42.36787708,  1.5242458 ]), array([[  20.37868328, -204.89577668],
           [-204.89577668, 5875.15399522]]), array([50.001]), array([0.18739855]))
    (array([42.3678885 ,  1.52424611]), array([[  20.37868757, -204.89593577],
           [-204.89593577, 5875.15966363]]), array([50.001]), array([0.18739855]))
    (array([42.36787837,  1.52424565]), array([[  20.37868708, -204.89589503],
           [-204.89589503, 5875.15952039]]), array([50.001]), array([0.18739855]))
    (array([42.36790124,  1.5242471 ]), array([[  20.37868956, -204.89602263],
           [-204.89602263, 5875.16073582]]), array([50.001]), array([0.18739855]))
    (array([42.3678952 ,  1.52424669]), array([[  20.37867366, -204.8956804 ],
           [-204.8956804 , 5875.14979346]]), array([50.001]), array([0.18739855]))
    (array([42.36787714,  1.52424581]), array([[  20.37868328, -204.89577668],
           [-204.89577668, 5875.15399505]]), array([50.001]), array([0.18739855]))
    (array([42.36788829,  1.52424609]), array([[  20.37868757, -204.89593577],
           [-204.89593577, 5875.15966367]]), array([50.001]), array([0.18739855]))
    (array([42.36787801,  1.52424563]), array([[  20.37868708, -204.89589505],
           [-204.89589505, 5875.15952098]]), array([50.001]), array([0.18739855]))
    (array([42.36790126,  1.5242471 ]), array([[  20.37868955, -204.89602254],
           [-204.89602254, 5875.16073317]]), array([50.001]), array([0.18739855]))
    (array([42.3678952 ,  1.52424669]), array([[  20.37867364, -204.8956803 ],
           [-204.8956803 , 5875.14979046]]), array([50.001]), array([0.18739855]))
    (array([42.36787708,  1.5242458 ]), array([[  20.37868328, -204.89577668],
           [-204.89577668, 5875.15399522]]), array([50.001]), array([0.18739855]))
    (array([42.3678885 ,  1.52424611]), array([[  20.37868757, -204.89593577],
           [-204.89593577, 5875.15966363]]), array([50.001]), array([0.18739855]))
    (array([42.36787837,  1.52424565]), array([[  20.37868708, -204.89589503],
           [-204.89589503, 5875.15952039]]), array([50.001]), array([0.18739855]))
    (array([42.36790124,  1.5242471 ]), array([[  20.37868956, -204.89602263],
           [-204.89602263, 5875.16073582]]), array([50.001]), array([0.18739855]))
    (array([42.3678952 ,  1.52424669]), array([[  20.37867366, -204.8956804 ],
           [-204.8956804 , 5875.14979346]]), array([50.001]), array([0.18739855]))
    (array([42.36787714,  1.52424581]), array([[  20.37868328, -204.89577668],
           [-204.89577668, 5875.15399505]]), array([50.001]), array([0.18739855]))
    (array([42.36788829,  1.52424609]), array([[  20.37868757, -204.89593577],
           [-204.89593577, 5875.15966367]]), array([50.001]), array([0.18739855]))
    (array([42.36787801,  1.52424563]), array([[  20.37868708, -204.89589505],
           [-204.89589505, 5875.15952098]]), array([50.001]), array([0.18739855]))
    (array([42.36790126,  1.5242471 ]), array([[  20.37868955, -204.89602254],
           [-204.89602254, 5875.16073317]]), array([50.001]), array([0.18739855]))
    (array([42.3678952 ,  1.52424669]), array([[  20.37867364, -204.8956803 ],
           [-204.8956803 , 5875.14979046]]), array([50.001]), array([0.18739855]))
    (array([42.36787708,  1.5242458 ]), array([[  20.37868328, -204.89577668],
           [-204.89577668, 5875.15399522]]), array([50.001]), array([0.18739855]))
    (array([42.3678885 ,  1.52424611]), array([[  20.37868757, -204.89593577],
           [-204.89593577, 5875.15966363]]), array([50.001]), array([0.18739855]))
    (array([42.36787837,  1.52424565]), array([[  20.37868708, -204.89589503],
           [-204.89589503, 5875.15952039]]), array([50.001]), array([0.18739855]))
    (array([42.36790124,  1.5242471 ]), array([[  20.37868956, -204.89602263],
           [-204.89602263, 5875.16073582]]), array([50.001]), array([0.18739855]))
    (array([42.3678952 ,  1.52424669]), array([[  20.37867366, -204.8956804 ],
           [-204.8956804 , 5875.14979346]]), array([50.001]), array([0.18739855]))
    (array([42.36787714,  1.52424581]), array([[  20.37868328, -204.89577668],
           [-204.89577668, 5875.15399505]]), array([50.001]), array([0.18739855]))
    (array([42.36788829,  1.52424609]), array([[  20.37868757, -204.89593577],
           [-204.89593577, 5875.15966367]]), array([50.001]), array([0.18739855]))
    (array([42.36787801,  1.52424563]), array([[  20.37868708, -204.89589505],
           [-204.89589505, 5875.15952098]]), array([50.001]), array([0.18739855]))
    (array([42.36790126,  1.5242471 ]), array([[  20.37868955, -204.89602254],
           [-204.89602254, 5875.16073317]]), array([50.001]), array([0.18739855]))
    (array([42.3678952 ,  1.52424669]), array([[  20.37867364, -204.8956803 ],
           [-204.8956803 , 5875.14979046]]), array([50.001]), array([0.18739855]))
    (array([42.36787708,  1.5242458 ]), array([[  20.37868328, -204.89577668],
           [-204.89577668, 5875.15399522]]), array([50.001]), array([0.18739855]))
    (array([42.3678885 ,  1.52424611]), array([[  20.37868757, -204.89593577],
           [-204.89593577, 5875.15966363]]), array([50.001]), array([0.18739855]))
    (array([42.36787837,  1.52424565]), array([[  20.37868708, -204.89589503],
           [-204.89589503, 5875.15952039]]), array([50.001]), array([0.18739855]))
    (array([42.36790124,  1.5242471 ]), array([[  20.37868956, -204.89602263],
           [-204.89602263, 5875.16073582]]), array([50.001]), array([0.18739855]))
    (array([42.3678952 ,  1.52424669]), array([[  20.37867366, -204.8956804 ],
           [-204.8956804 , 5875.14979346]]), array([50.001]), array([0.18739855]))
    (array([42.36787714,  1.52424581]), array([[  20.37868328, -204.89577668],
           [-204.89577668, 5875.15399505]]), array([50.001]), array([0.18739855]))
    (array([42.36788829,  1.52424609]), array([[  20.37868757, -204.89593577],
           [-204.89593577, 5875.15966367]]), array([50.001]), array([0.18739855]))
    (array([42.36787801,  1.52424563]), array([[  20.37868708, -204.89589505],
           [-204.89589505, 5875.15952098]]), array([50.001]), array([0.18739855]))
    (array([42.36790126,  1.5242471 ]), array([[  20.37868955, -204.89602254],
           [-204.89602254, 5875.16073317]]), array([50.001]), array([0.18739855]))
    (array([42.3678952 ,  1.52424669]), array([[  20.37867364, -204.8956803 ],
           [-204.8956803 , 5875.14979046]]), array([50.001]), array([0.18739855]))
    (array([42.36787708,  1.5242458 ]), array([[  20.37868328, -204.89577668],
           [-204.89577668, 5875.15399522]]), array([50.001]), array([0.18739855]))
    (array([42.3678885 ,  1.52424611]), array([[  20.37868757, -204.89593577],
           [-204.89593577, 5875.15966363]]), array([50.001]), array([0.18739855]))
    (array([42.36787837,  1.52424565]), array([[  20.37868708, -204.89589503],
           [-204.89589503, 5875.15952039]]), array([50.001]), array([0.18739855]))
    (array([42.36790124,  1.5242471 ]), array([[  20.37868956, -204.89602263],
           [-204.89602263, 5875.16073582]]), array([50.001]), array([0.18739855]))
    (array([42.3678952 ,  1.52424669]), array([[  20.37867366, -204.8956804 ],
           [-204.8956804 , 5875.14979346]]), array([50.001]), array([0.18739855]))
    (array([42.36787714,  1.52424581]), array([[  20.37868328, -204.89577668],
           [-204.89577668, 5875.15399505]]), array([50.001]), array([0.18739855]))
    (array([42.36788829,  1.52424609]), array([[  20.37868757, -204.89593577],
           [-204.89593577, 5875.15966367]]), array([50.001]), array([0.18739855]))
    (array([42.36787801,  1.52424563]), array([[  20.37868708, -204.89589505],
           [-204.89589505, 5875.15952098]]), array([50.001]), array([0.18739855]))
    (array([42.36790126,  1.5242471 ]), array([[  20.37868955, -204.89602254],
           [-204.89602254, 5875.16073317]]), array([50.001]), array([0.18739855]))

```python
estimate = np.zeros((100,5))
for i in range(5):
    estimate[:,i] = f(theta_n[i*20], t)
```

```python
plt.plot(t, model, linewidth = 3, alpha = 0.6)
plt.scatter(t, data, c = 'orange')
plt.show()

plt.plot(t, estimate, linewidth = 1, alpha = 0.6)
plt.scatter(t, data, c = 'orange')
plt.show()
```