# Local Linear Regression
Spring 2019 AME-70790 Final Project

Nicholas Geneva (ngeneva@nd.edu)

Reference: Wand, M. P., & Jones, M. C. (1994). Kernel smoothing. Chapman and Hall/CRC.
___
Another application of kernel smoothers is *kernel regression* which is a non-parametric regression method that shares many of the same ideas as kernel density estimation.
One of the main advantages of kernel regression methods is their underlying simplicity compared to other non-parametric methods such as spline functions or wavelets.
We will specifically focus on a subset of kernel regression methods called *local polynomial estimators*.

#### Local Polynomial Estimators
Let us consider an arbitrary set of $n$ training points $\left\{x_{i},y_{i}\right\}_{i=1}^{n}$.
To formulate the local polynomial estimator we start by first looking at a standard least squares regression model with a polynomial basis function:
    $$\begin{gather}
    \mathcal{L} = \sum_{i=1}^{n}\left(y_{i}-\textbf{w}^{T}\textbf{X}\right)^{2},\\
    \textbf{w} = \left[ w_{1}, w_{2}, ... , w_{p}\right]^{T}, \quad \textbf{X}=\left[\begin{matrix} 1 & x_{1} & x_{1}^{2} &... & x_{1}^{p} \\ 1 & x_{2} & x_{2}^{2} &... & x_{2}^{p} \\ ... & & & & ... \\ 1 & x_{n} & x_{n}^{2} &... & x_{n}^{p} \\\end{matrix}\right].
    \end{gather}$$
In the standard least square approach we would take the derivative of the loss with respect to the weights and set it equal to zero.
However, we will take a slightly different approach, consider the parameterization $m_{w}(x)=\textbf{w}^{T}\textbf{X}$ where the function $m_{w}$ now represents the polynomial that we with to fit.
Now rather than considering the polynomial directly, we approximate it using Taylor expansion to the $p$-th order:
    $$\begin{gather}
    \mathcal{L} = \sum_{i=1}^{n}\left(y_{i}-m_{w}(x_{i})\right)^{2},\\
    m_{w}(x_{i})\approx m(x)+m'(x)(x_{i}-x)+\frac{m''(x)}{2}(x_{i}-x)^{2}+...+\frac{m^{(p)}(x)}{p!}(x_{i}-x)^{p},\\
    \mathcal{L} = \sum_{i=1}^{n}\left(y_{i}-\sum_{j=0}^{p}\frac{m^{(j)}(x)}{j!}(x_{i}-x)^{j}\right)^{2}= \sum_{i=1}^{n}\left(y_{i}-\sum_{j=0}^{p}\beta_{j}(x_{i}-x)^{j}\right)^{2},
    \end{gather}$$
where we have replaced the coefficients of the Taylor expansion with $\beta_{j}$.
The last addition is the addition of a kernel operator such the contributions of each datum $(x_{i},y_{i})$ are based on its spacial location to the point of interest:
    $$\mathcal{L} = \sum_{i=1}^{n}\left(y_{i}-\sum_{j=0}^{p}\beta_{j}(x_{i}-x)^{j}\right)^{2}K_{h}(x_{i}-x).$$
We can reformulate the loss in matrix form and solve similar to standard least squares regression:
    $$\begin{align}
        \mathcal{L}&=(\textbf{Y}-\boldsymbol{\beta}^{T}\textbf{X}_{x})^{2}\textbf{W}_{x}\\
        \frac{\partial \mathcal{L}}{\partial\boldsymbol{\beta}}&=2\textbf{X}_{x}^{T}\textbf{W}_{x}\textbf{X}_{x}\boldsymbol{\beta} - 2\textbf{X}_{x}^{T}\textbf{W}_{x}\textbf{Y}\\
        \hat{\boldsymbol{\beta}}(x) &= \left(\textbf{X}_{x}^{T}\textbf{W}_{x}\textbf{X}_{x}\right)^{-1}\textbf{X}_{x}^{T}\textbf{W}_{x}\textbf{Y},
    \end{align}$$
assuming $\textbf{X}^{T}\textbf{W}\textbf{X}$ is invertible. Each matrix is defined as follows:
    $$\textbf{Y}=\left[y_{1},y_{2},...,y_{n}\right]^{T}, \quad
    \textbf{X}_{x}=\left[\begin{matrix} 1 & x_{1}-x &... & x_{1}^{p}-x \\ 1 & x_{2}-x & ... & x_{2}^{p}-x \\ ... & & & ... \\ 1 & x_{n}-x &... & x_{n}^{p}-x \\\end{matrix}\right], \quad
    \textbf{W}_{x}=\textrm{diag}\left\{K_{h}(x_{1}-x), ..., K_{h}(x_{p}-x)\right\}.$$
Since we have defined the Taylor expansion centered around point $x$, the prediction at point $x$ is then simply the intercept coefficient:
    $$\hat{\beta}_{0} = \hat{m}(x) = \textbf{e}^{T}\left(\textbf{X}_{x}^{T}\textbf{W}_{x}\textbf{X}_{x}\right)^{-1}\textbf{X}_{x}^{T}\textbf{W}_{x}\textbf{Y},$$
where $\textbf{e}^{T}$ is a $[p+1]$ vector with the first element equal to 1 and the rest zero.
Note that unlike linear regression, this expression must be evaluated at every predictive point.
This makes the computational cost of local polynomial estimators much more expensive.

#### Local Linear Regression
With the general formulation of the local polynomial estimator defined with its solution, let us start by taking a look at *local linear regression* which can be found from the equations above by setting $p=1$:
    $$\mathcal{L} = \sum_{i=1}^{n}\left(y_{i}-\beta_{0} + \beta_{1}(x_{i}-x)\right)^{2}K_{h}(x_{i}-x).$$
For the kernel function, we will continue the normal distribution $K_{h}=\mathcal{N}(0,h^{2})$. Now for our regression problem we will consider the following function:
    $$m(x)=2\exp\left\{-x^{2}/(0.3)^{2}\right\}+3\exp\left\{-(x-1)^{2}/(0.7)^{2}\right\}, \quad x\in[0,1].$$
When generating our training data, we evenly sample points between [0,1] and add additional output-wise noise to each observation:
    $$y_{i}=m(x_{i})+0.075\epsilon_{i},\quad \epsilon_{i}\sim\mathcal{N}(0,1).$$

```python
import numpy as np
from scipy.stats import norm
import matplotlib.pyplot as plt
```

```python
plt.close("all")
np.random.seed(123)
ntrain = 100 # Number of training points
h = 0.05 # Kernel Bandwidth

# Generate training data
x_train = np.linspace(0,1,ntrain)
Y = 2*np.exp(-x_train**2/(0.3**2)) + 3*np.exp(-(x_train-1)**2/(0.7**2)) + 0.075*np.random.randn(ntrain)

# Predict
n_pred = 50
x_pred = np.linspace(0,1,n_pred)
betas = np.zeros((n_pred, 2))
for i, x0 in enumerate(x_pred):
    # Solve for betas
    X = np.stack([np.ones(ntrain), x_train-x0], axis=1)
    W = np.diag(norm.pdf(x_train-x0, loc=0, scale=h))
    betas[i] = np.linalg.inv(X.T.dot(W).dot(X)).dot(X.T).dot(W).dot(Y)
```

With the beta coefficients calculated we have the predictions of the model.
As one can imagine, when the number of points one needs to predict at increases significantly this local linear regression model will be unusable.

```python
fig = plt.figure(figsize=(10,5))
ax = []
ax.append(plt.subplot2grid((1, 2), (0, 0)))
ax.append(plt.subplot2grid((1, 2), (0, 1)))

# Plot target
x_target = np.linspace(0,1,200)
y_target = 2*np.exp(-x_target**2/(0.3**2)) + 3*np.exp(-(x_target-1)**2/(0.7**2))
ax[0].plot(x_target, y_target, '-', color='k', label='Target')

# Plot training data
ax[0].scatter(x_train, Y, c='k', s=15.0, marker='x', linewidth=1.0, label='Training Data', alpha=0.5)

# Plot local linear kernel estimate
# Recall that the prediction at each point is simply the intercept value beta0
ax[0].plot(x_pred, betas[:,0], '--', color='r', label='Prediction')
ax[0].set_xlabel('x')
ax[0].set_ylabel('y')
ax[0].legend()

# Plot several local linear fits
p1, = ax[1].plot(x_target, y_target, '-', color='k', label='Target')
idxs = [int(n_pred/5), int(n_pred/2), int(4*n_pred/5)]
for i, idx in enumerate(idxs):
    x_cen = x_pred[idx]
    x_loc = x_pred[idx-int(n_pred/10):idx+int(n_pred/10)]
    p2, = ax[1].plot(x_loc, betas[idx,0]+betas[idx,1]*(x_loc - x_cen), '-.', color='b')
    p3, = ax[1].plot(x_target, 0.025*norm.pdf(x_target-x_cen, loc=0, scale=h)+np.min(Y), '--', color='k', linewidth=0.5)
    ax[1].plot([x_cen,x_cen], [np.min(Y), betas[idx,0]], '--', color='k', linewidth=0.5) # center line

ax[1].set_xlabel('x')
ax[1].set_ylabel('y')
ax[1].legend((p1,p2,p3),('Target', 'Linear Fits', 'Kernel Weight'))

# Save and show figure
plt.savefig('figs/05_local_linear_regression.pdf')
plt.savefig('figs/05_local_linear_regression.png')
plt.show()
```

On the left we can see the prediction of the local linear regression which does a very nice job at estimating the function. On the right three local linear fits which illustrates how the local linear regression predicts by fitting a linear model centered at each prediction point.
Below are the kernel weights which illustrate the range of training points that have influence on the prediction.

#### Bandwidth Sensitivity
Similar to kernel density estimation, the bandwidth of the kernel can be very influential over the prediction quality.

```python
band_widths = [0.005, 0.05, 0.2]
# Predict
n_pred = 50
x_pred = np.linspace(0,1,n_pred)
betas = np.zeros((len(band_widths), n_pred, 2))
for i, h in enumerate(band_widths):
    for j, x0 in enumerate(x_pred):
        # Solve for betas
        X = np.stack([np.ones(ntrain), x_train-x0], axis=1)
        W = np.diag(norm.pdf(x_train-x0, loc=0, scale=h))
        betas[i,j] = np.linalg.inv(X.T.dot(W).dot(X)).dot(X.T).dot(W).dot(Y)
```

```python
fig = plt.figure(figsize=(15,4))
ax = []
ax.append(plt.subplot2grid((1, 3), (0, 0)))
ax.append(plt.subplot2grid((1, 3), (0, 1)))
ax.append(plt.subplot2grid((1, 3), (0, 2)))

for i, h in enumerate(band_widths):
    # Kernel prediction
    ax[i].plot(x_pred, betas[i,:,0], '--', c='r', label='KDE')
    # Target
    ax[i].plot(x_target, y_target, '--', color='k', label='Target')
    ax[i].set_title('Bandwidth: {:.02f}'.format(h))
    ax[i].set_xlabel('x')
    ax[i].set_ylabel('y')

# Save and show figure
plt.savefig('figs/05_local_linear_bandwidth.pdf')
plt.savefig('figs/05_local_linear_bandwidth.png')
plt.show()
```

Here we see precisely the trends seen for kernel density estimation.
If the bandwidth is too low the prediction is noisy and if it is too high the prediction is over smoothed.


# Solve Cournot oligopoly model via collocation

**Randall Romero Aguilar, PhD**

This demo is based on the original Matlab demo accompanying the  Computational Economics and Finance 2001 textbook by Mario Miranda and Paul Fackler.

Original (Matlab) CompEcon file: **demapp07.m**

Running this file requires the Python version of CompEcon. This can be installed with pip by running

    !pip install compecon --upgrade

Last updated: 2021-Oct-01

## About

To illustrate implementation of the collocation method for implicit function problems, consider the example of Cournot oligopoly. In the standard microeconomic model of the firm, the firm maximizes profit by equating marginal revenue to marginal cost (MC). An oligopolistic firm, recognizing that its actions affect price, takes the marginal revenue to be $p + q\frac{dp}{dq}$, where $p$ is price, $q$ is quantity produced, and $\frac{dp}{dq}$ is the marginal impact of output on market price. The Cournot assumption is that the firm acts as if any change in its output will be unmatched by its competitors. This implies that

\begin{equation}
    \frac{dp}{dq} = \frac{1}{D'(p)}
\end{equation}

where $D(p)$ is the market demand curve.

Suppose we wish to derive the effective supply function for the firm, which specifies
the quantity $q = S(p)$ it will supply at any price. The firm's effective supply function is
characterized by the functional equation

\begin{equation}
    p + \frac{S(p)}{D'(p)} - MC(S(p)) = 0
\end{equation}

for all positive prices $p$. In simple cases, this function can be found explicitly.  However,
in more complicated cases, no explicit solution exists.

## Initial tasks

```python
import numpy as np
import matplotlib.pyplot as plt
from compecon import BasisChebyshev, NLP, demo
```

## The model
### Parameters

Here, the demand elasticity and the marginal cost function parameter are

```python
alpha, eta = 1.0, 3.5
```

```python
D = lambda p: p**(-eta)
```

### Approximation structure

A degree-25 Chebychev basis on the interval [0.5, 3.0] is selected; also, the associated collocation nodes `p` are computed.

```python
n, a, b =  25, 0.5, 2.0
S = BasisChebyshev(n, a, b, labels=['price'], y=np.ones(n))
p = S.nodes
```

```python
S2 = BasisChebyshev(n, a, b, labels=['price'], l=['supply'])
S2.y = np.ones_like(p)
```

### Residual function

Suppose, for example, that

\begin{equation}
    D(p) = p^{-\eta} \quad\text{and}\quad MC(q) = \alpha\sqrt{q} + q^2
\end{equation}

Then the functional equation to be solved for S(p),

\begin{equation}
 \left[p - \frac{S(p)p^{\eta+1}}{\eta}\right] -\left[\alpha\sqrt{S(p)} + S(p)^2\right] = 0
\end{equation}

has no known closed-form solution.

```python
def resid(c):
    S.c = c  # update interpolation coefficients
    q = S(p) # compute quantity supplied at price nodes
    marginal_income = p - q * (p ** (eta+1) / eta)
    marginal_cost = alpha * np.sqrt(q) + q ** 2
    return  marginal_income - marginal_cost
```

Notice that `resid` only takes one argument. The other parameters (`Q`, `p`, `eta`, `alpha`) should be declared as such in the main script, were Python's scoping rules will find them.

### Solve for effective supply function

Class `NLP` defines nonlinear problems. It can be used to solve `resid` by Broyden's method.

```python
cournot = NLP(resid)
S.c = cournot.broyden(S.c, tol=1e-12)
```

### Plot demand and effective supply for m=5 firms

```python
prices = np.linspace(a, b, 501)

fig1, ax = plt.subplots()
ax.plot(5 * S(prices), prices, label='Supply')
ax.plot(D(prices), prices, label='Demand')

ax.set(title='Cournot Effective Firm Supply Function',
       xlabel='Quantity',
       ylabel='Price',
       xlim=[0, 4],
       ylim=[0.5, 2])

ax.legend();
```

### Plot residual

Notice that `resid` does not take explicit parameters, so to evaluate it when prices are `prices` we need to assign `p = prices`.

In order to assess the quality of the approximation, one plots the residual function over the approximation domain. Here, the residual function is plotted by computing the residual at a refined grid of 501 equally spaced points.

```python
p = prices
fig2, ax = plt.subplots()

ax.hlines(0, a, b, 'k', '--', lw=2)
ax.plot(prices, resid(S.c))
ax.set(title='Residual Function for Cournot Problem',
       xlabel='Quantity',
       ylabel='Residual');
```

### Plot demand and effective supply for m=1, 3, 5, 10, 15, 20 firms

```python
fig3, ax = plt.subplots(figsize=[9,4])

ax.set(title='Industry Supply and Demand Functions',
       xlabel='Quantity',
       ylabel='Price',
       xlim=[0, 12])

lcolor = [z['color']  for z in plt.rcParams['axes.prop_cycle']]

for i, m in enumerate([1, 3, 5, 10, 15, 20]):
    ax.plot(m*S(prices), prices) # supply
    ax.annotate(f'm={m:d}', [m*S(1.2)+.025, 1.4-i/12], color=lcolor[i], fontsize=12)

ax.plot(D(prices), prices, linewidth=4, color='grey') # demand
ax.annotate('demand', [10, 0.6], color='grey', fontsize=12);
```

### Plot equilibrium price as a function of number of firms m

```python
m = np.arange(1,26)
x0 = np.full_like(m, 0.7, dtype=float) #initial guess
eqprices = NLP(lambda p: m*S(p) - D(p)).broyden(x0)

fig4, ax = plt.subplots()
ax.set(title='Cournot Equilibrium Price as Function of Industry Size',
       xlabel='Number of Firms',
       ylabel='Price')
ax.plot(m, eqprices);
```

### Save all figures to disc

```python
#demo.savefig([fig1,fig2,fig3,fig4], name='demapp07')
```

_build/jupyter_execute/notebooks/app/07 Solve Cournot oligopoly model via collocation.ipynb


# 1 Step Adams Moulton
#### John S Butler
john.s.butler@tudublin.ie
[Course Notes](https://johnsbutler.netlify.com/files/Teaching/Numerical_Analysis_for_Differential_Equations.pdf)    [Github](https://github.com/john-s-butler-dit/Numerical-Analysis-Python)

This notebook implements the 1 step Adams Moulton method for three different population intial value problems.

# Formula
The general 1 step Adams-Moulton method for the first order differential equation
\begin{equation} y^{'} = f(t,y) \end{equation}
numerical approximates $y$ the at time point $t_i$ as $w_i$
with the  formula:
\begin{equation} w_{i+1}=w_i+\frac{h}{2}\big[f(t_{i+1},w_{i+1})+f(t_{i},w_{i})\big],\end{equation}
for $i=0,...,N-1$, where

and $h$ is the stepsize.

To illustrate the method we will apply it to three intial value problems:
## 1. Linear
Consider the linear population Differential Equation
\begin{equation} y^{'}=0.1y, \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}y(2000)=6.\end{equation}

## 2. Non-Linear Population Equation
Consider the non-linear population Differential Equation
\begin{equation} y^{'}=0.2y-0.01y^2, \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}y(2000)=6.\end{equation}

## 3. Non-Linear Population Equation with an oscillation
Consider the non-linear population Differential Equation with an oscillation
\begin{equation} y^{'}=0.2y-0.01y^2+\sin(2\pi t), \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}y(2000)=6.\end{equation}

#### Setting up Libraries

```python
## Library
import numpy as np
import math
import pandas as pd
%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings

warnings.filterwarnings("ignore")
```

## Discrete Interval
The continuous time $a\leq t \leq b $ is discretised into $N$ points seperated by a constant stepsize
\begin{equation} h=\frac{b-a}{N}.\end{equation}
Here the interval is $2000\leq t \leq 2020,$
\begin{equation} h=\frac{2020-2000}{200}=0.1.\end{equation}
This gives the 201 discrete points:
\begin{equation} t_0=2000, \ t_1=2000.1, \ ... t_{200}=2020. \end{equation}
This is generalised to
\begin{equation} t_i=2000+i0.1, \ \ \ i=0,1,...,200.\end{equation}
The plot below shows the discrete time steps:

```python
### DISCRETE TIME
N=20
t_end=2020.0
t_start=2000.0
h=((t_end-t_start)/N)
t=np.arange(t_start,t_end+h/2,h)

## PLOTS TIME
fig = plt.figure(figsize=(10,4))
plt.plot(t,0*t,'o:',color='red')
plt.title('Illustration of discrete time points for h=%s'%(h))
plt.show()
len(t)
```

# 1. Linear Population Equation
## Exact Solution
The linear population equation
\begin{equation} y^{'}=0.1y, \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}y(2000)=6.\end{equation}
has a known exact (analytic) solution
\begin{equation} y(t)=6e^{0.1(t-2000)}. \end{equation}

## Specific 1 step Adams Moulton
The specific 1 step Adams Moulton for the linear population equation is:

\begin{equation}w_{i+1}=w_{i}+\frac{h}{2}\big[0.1w_{i+1}+0.1w_i\big] \end{equation}
where
\begin{equation}f(t,y)=0.1y,\end{equation}

```python
## THIS IS THE RIGHT HANDSIDE OF THE LINEAR POPULATION DIFFERENTIAL
## EQUATION
def linfun(t,w):
    ftw=0.1*w
    return ftw
```

re-arranging,
\begin{equation} \big(1-\frac{h}{2}(0.1)\big)w_{i+1}=w_i+\frac{h}{2}\big[0.1w_i\big], \end{equation}
\begin{equation} w_{i+1}=\frac{w_i+\frac{h}{2}\big[0.1w_i\big]}{\big(1-\frac{h}{2}(0.1)\big)}. \end{equation}

```python
### INSERT METHOD HERE
w=np.zeros(N+1) # a list of 2000+1 zeros
w[0]=6 # INITIAL CONDITION
for i in range(0,N):
    w[i+1]=(w[i]+h/2*(linfun(t[i],w[i])))/(1-0.1*h/2)

```

## Plotting Results

```python
## PLOTTING METHOD
y=6*np.exp(0.1*(t-2000)) # EXACT SOLUTION
fig = plt.figure(figsize=(8,4))
plt.plot(t,w,'o:',color='purple',label='Taylor')
plt.plot(t,y,'s:',color='black',label='Exact')
plt.legend(loc='best')
plt.show()
```

## Table
The table below shows the time, the numerical approximation, $w$,  the exact solution, $y$, and the exact error $|y(t_i)-w_i|$ for the linear population equation:

```python
d = {'time t_i': t, 'Adams Approx w': w}
df = pd.DataFrame(data=d)
df
```

      time t_i
      Adams Approx w

      0
      2000.0
      6.000000

      1
      2001.0
      6.631579

      2
      2002.0
      7.329640

      3
      2003.0
      8.101181

      4
      2004.0
      8.953937

      5
      2005.0
      9.896456

      6
      2006.0
      10.938189

      7
      2007.0
      12.089577

      8
      2008.0
      13.362164

      9
      2009.0
      14.768708

      10
      2010.0
      16.323308

      11
      2011.0
      18.041551

      12
      2012.0
      19.940662

      13
      2013.0
      22.039679

      14
      2014.0
      24.359645

      15
      2015.0
      26.923819

      16
      2016.0
      29.757905

      17
      2017.0
      32.890316

      18
      2018.0
      36.352454

      19
      2019.0
      40.179029

      20
      2020.0
      44.408400

## 2. Non-Linear Population Equation
\begin{equation} y^{'}=0.2y-0.01y^2, \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}y(2000)=6.\end{equation}
## Specific 1 step Adams-Moutlon method for the Non-Linear Population Equation
The specific Adams-Moulton difference equation for the non-linear population equations is:

\begin{equation}w_{i+1}=w_{i}+\frac{h}{2}\big[ 0.2 w_{i+1}-0.01 w_{i+1}^2
                                + 0.2 w_i-0.01 w_i^2\big] \end{equation}
re-arranging
\begin{equation}w_{i+1}-\frac{h}{2}\big( 0.2 w_{i+1}-0.01 w_{i+1}^2 \big)=w_{i}+\frac{h}{2}
                                \big[ 0.2 w_i-0.01 w_i^2\big] \end{equation}

for $i=0,...,199$, where $w_i$ is the numerical approximation of $y$ at time $t_i$, with step size $h$ and the initial condition
\begin{equation}w_0=6.\end{equation}

PROBLEM WE CANNOT MOVE THE SQUARED (NON-LINEAR TERM) TO THE RIGHT HAND SIDE SO WE CAN SOLVE FOR w[i+1].
For this reason we will use a predictor-corrector method,
The predictor will be the 2-step Adams Bashforth
\begin{equation} w_{i+1}^{p}=w_{i}+\frac{h}{2}\big[ 3(0.2 w_{i}-0.01 w_{i}^2)
                                - (0.2 w_{i-1}-0.01 w_{i-1}^2) \big], \end{equation}
with the corrector being the 1-step Adams Moulton,
\begin{equation}w_{i+1}=w_{i}+\frac{h}{2}\big[ 0.2 w_{i+1}^p-0.01 (w^p_{i+1})^2
                                + 0.2 w_i-0.01 w_i^2\big], \end{equation}
for i=1,...200.

```python
def nonlinfun(t,w):
    ftw=0.2*w-0.01*w*w
    return ftw
```

```python
### INSERT METHOD HERE
w=np.zeros(N+1)
w_p=np.zeros(N+1)
w[0]=6
w[1]=6.084 # FROM THE THE TAYLOR METHOD
w_p[0]=6
w_p[1]=6.084 # FROM THE THE TAYLOR METHOD
for n in range(1,N):
    ## Predictor
    w_p[n+1]=w[n]+h/2*(3*nonlinfun(t[n],w[n])-
                       nonlinfun(t[n-1],w[n-1]))
    ## Corrector
    w[n+1]=w[n]+h/2*(nonlinfun(t[n+1],w_p[n+1])+
                       nonlinfun(t[n],w[n]))
```

## Results
The plot below shows the numerical approximation, $w$ (circles) for the non-linear population equation:

```python
fig = plt.figure(figsize=(8,4))
plt.plot(t,w,'-.',color='purple',label='Correcter ')
plt.plot(t,w_p,':',color='red',label='Predictor ')

plt.title('Non Linear Population Equation')
plt.legend(loc='best')
plt.xlabel('time (yrs)')
plt.ylabel('Population in billions')
plt.show()

fig = plt.figure(figsize=(8,4))
plt.plot(t,np.abs(w-w_p),':',color='purple',label='|w-w_p| ')
plt.title('Comparison of Predictor and Corrector')
plt.legend(loc='best')
plt.xlabel('time (yrs)')
plt.ylabel('Population in billions')
plt.show()

```

## Table
The table below shows the time and the numerical approximation, $w$,  for the non-linear population equation:

```python
d = {'time t_i': t, 'Adams Approx w': w}
df = pd.DataFrame(data=d)
df
```

      time t_i
      Adams Approx w

      0
      2000.0
      6.000000

      1
      2001.0
      6.084000

      2
      2002.0
      6.960322

      3
      2003.0
      7.892040

      4
      2004.0
      8.863456

      5
      2005.0
      9.856908

      6
      2006.0
      10.853082

      7
      2007.0
      11.832473

      8
      2008.0
      12.776881

      9
      2009.0
      13.670688

      10
      2010.0
      14.501747

      11
      2011.0
      15.261810

      12
      2012.0
      15.946493

      13
      2013.0
      16.554881

      14
      2014.0
      17.088907

      15
      2015.0
      17.552635

      16
      2016.0
      17.951549

      17
      2017.0
      18.291930

      18
      2018.0
      18.580348

      19
      2019.0
      18.823293

      20
      2020.0
      19.026909

## 3. Non-Linear Population Equation with an oscilation
\begin{equation} y^{'}=0.2y-0.01y^2+\sin(2\pi t), \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}y(2000)=6.\end{equation}

## Specific 2 Step Adams Moulton for the Non-Linear Population Equation with an oscilation

The specific Adams-Moulton difference equation for the non-linear population equations is:

\begin{equation}w_{i+1}=w_{i}+\frac{h}{2}\big[ 0.2 w_{i+1}-0.01 w_{i+1}^2 +\sin(2\pi t_{i+1})
                                + 0.2 w_i-0.01 w_i^2+\sin(2\pi t_{i})\big] \end{equation}

for $i=1,...,199$, where $w_i$ is the numerical approximation of $y$ at time $t_i$, with step size $h$ and the initial condition
\begin{equation}w_0=6.\end{equation}

 As $w_1$ is required for the method but unknown we will use the numerical solution of a one step method to approximate the value. Here, we use the 2nd order Runge Kutta approximation (see [Runge Kutta notebook](https://github.com/john-s-butler-dit/Numerical-Analysis-Python/blob/master/Chapter%2003%20-%20Runge%20Kutta/01_2nd%20Order%20Runge%20Kutta%20Population%20Equations.ipynb) )
\begin{equation}w_1=6.11.\end{equation}

PROBLEM WE CANNOT MOVE THE SQUARED (NON-LINEAR TERM) TO THE RIGHT HAND SIDE SO WE CAN SOLVE FOR w[i+1].
For this reason we will use a predictor-corrector method,
The predictor will be the 2-step Adams Bashforth
\begin{equation} w_{i+1}^{p}=w_{i}+\frac{h}{2}\big[ 3(0.2 w_{i}-0.01 w_{i}^2+\sin(2\pi t_{i})
                                - (0.2 w_{i-1}-0.01 w_{i-1}^2+\sin(2\pi t_{i-1}) \big] \end{equation}
with the corrector being the 1-step Adams Moulton,
\begin{equation}w_{i+1}=w_{i}+\frac{h}{2}\big[ 0.2 w_{i+1}^p-0.01 (w^p_{i+1})^2 +\sin(2\pi t_{i+1}
                                + 0.2 w_i-0.01 w_i^2+\sin(2\pi t_{i}\big], \end{equation}

```python
def nonlin_oscfun(t,w):
    ftw=0.2*w-0.01*w*w+np.sin(2*np.math.pi*t)
    return ftw
```

```python
## INSERT METHOD HERE
w=np.zeros(N+1)
w[0]=6
w[1]=6.11
for n in range(1,N):
    w[n+1]=w[n]+h/2
```

## Results
The plot below shows the  numerical approximation, $w$ (circles) for the non-linear population equation:

```python
fig = plt.figure(figsize=(8,4))
plt.plot(t,w,'o:',color='purple',label='Adams-Moulton')
plt.title('Population Equation with seasonal oscilation')
plt.xlabel('time (yrs)')
plt.ylabel('Population in Billions')
plt.legend(loc='best')
plt.show()
```

## Table
The table below shows the time and the numerical approximation, $w$,  for the non-linear population equation:

```python
d = {'time t_i': t, 'Adams Approx w': w}
df = pd.DataFrame(data=d)
df

```

      time t_i
      Adams Approx w

      0
      2000.0
      6.00

      1
      2001.0
      6.11

      2
      2002.0
      6.61

      3
      2003.0
      7.11

      4
      2004.0
      7.61

      5
      2005.0
      8.11

      6
      2006.0
      8.61

      7
      2007.0
      9.11

      8
      2008.0
      9.61

      9
      2009.0
      10.11

      10
      2010.0
      10.61

      11
      2011.0
      11.11

      12
      2012.0
      11.61

      13
      2013.0
      12.11

      14
      2014.0
      12.61

      15
      2015.0
      13.11

      16
      2016.0
      13.61

      17
      2017.0
      14.11

      18
      2018.0
      14.61

      19
      2019.0
      15.11

      20
      2020.0
      15.61

```python

```

```python

```

```python

```

Chapter 04 - Multistep Methods/404_Adams Moulton Population Equations.ipynb


# Monodromy Matrices and Scaled Potentials

Given a real-valued $K$-periodic potential $(v(0),\dots,v(K-1))$, the corresponding monodromy matrix reads

\begin{equation}
    M = \begin{pmatrix} -v(K-1)&-1\\1&0\end{pmatrix}
\begin{pmatrix} -v(K-2)&-1\\1&0\end{pmatrix}
\cdots
\begin{pmatrix} -v(0)&-1\\1&0\end{pmatrix}
\end{equation}

If $v$ is a potential over the alphabet $\Sigma_\lambda = \{0,\lambda\}$ with $\lambda \in \mathbb{R}$, then the corresponding matrix entries of $M$ are polynomials in $\lambda$.

The location of the zeros of the entry $M_{2,1}$ is linked to the invertibility of the corresponding one-sided periodic Schrödinger operator $H_+$. More precisely, if $|\mathrm{tr}(M) | > 2$ and $M_{2,1} \neq 0$, then $H_+$ is invertible.

This notepad will systematically determine all potentials up to length $9$ over the alphabet $\Sigma_0$ and calculate the corresponding monodromy matrices, the zeros of the entry $M_{2,1}$ and the trace of $M$. See the section "Systematic Studies of $\{0, \lambda\}$-Valued Potentials" for details. In particular, all polynomial roots are given by closed-form algebraic expressions.

# Data Generation

Run the cell below with `Kmin = 1` and `Kmax = 9` in order to generate all data for the proofs in the section "Systematic Studies of $\{0, \lambda\}$-Valued Potentials".

The cell will produce a dictionary `pots`, where the keys are given by stringified potentials and the corresponding values consist of the symbolic monodromy matrix (`key='mon'`), the zeros of the matrix entry (`key='sol'`) and the value of the trace (`key='trace'`).

# Rendering of Output

After all results have been calculated. Run the cell that renders "Human Readable Output".

# Interactive Data Analysis

In order to access and further analyse the values in the dictionary for the potential `v = [0,1,1]` use the syntax
```python
v = [0,1,1]
pots[str(v)]
```
More examples follow below in the analysis section.

# JSON Export

It is possible to export the files in JSON, see the corresponding section at the end of this notebook.

```python
# Run this cell once at startup
###############################

l = var('l')

def mon(v):
##########################
# Calculate the general monodromy matrix with scaling l
#
# Example:
# v = [1,1,0]; mon(v).expand()
# > [      l       1]
# > [l^2 - 1       l]
#
##########################
    M = identity_matrix(2)
    for vv in v:
        M = matrix(2,2,[ - l*vv, -1, 1, 0])*M
    return M

def analyse_v(v):
##########################
# calculate monodromy, zeros and check trace.
# return a dictionary containing symbolic elements
#
# Example:
# v = [1,1,0]; analyse_v(v)
##########################
    data = {}
#monodromy
    M = mon(v).expand()
    data['mon'] = M
#zeros
    sols = solve(M[1][0] == 0, l)
    sol = list(map(lambda s: s.rhs(), sols))
    sol.sort()
    data['sol'] = sol
#trace
    data['trace'] = list(map(lambda s: M.trace().subs(l=s), sol))
    return data

#######Data Creation######
# input
Kmin = 1
Kmax = 8 #maximal period length, not larger than 9 to guarantee algebraic roots
sigma = [0,1] # alphabet

pots = {} #empty dictionary
for K in range(Kmin,Kmax + 1):
    v_list = Tuples(sigma,K).list()
    for v in v_list:
        pots[str(v)] = deepcopy(analyse_v(v))
```

# Human Readable Output

The cell below outputs all potentials, their corresponding monodromy matrices, the zeros of the entry M_{1,2}, and the corresponding value of the trace of M. For an interactive data analysis and further documentation, see the cells below.

```python
for k in range(1,9):
    show("Period Length K = ", k)
    v_list = Tuples(sigma,k).list()
    for v in v_list:
        # skip the following potentials for readability
        # they are treated separately at the the Section
        # "Skipped potentials" of this notebook
        if (v == [1,1,1,1,1,1,1] or v == [1,1,1,1,1,1,0]):
            continue
        show((str(v), pots[str(v)]))
```

## Skipped Potentials

The following cell analyses the previously skipped potentials

`v = [1,1,1,1,1,1,0]`

and

`v = [1,1,1,1,1,1,1]`

Both potentials lead to the monodromy matrix entry $M_{1,2}$.

$$
M_{1,2} = l^6 - 5l^4 + 6l^2 - 1
$$

The algebraic roots produced with SageMath don't fully simplify with some remaining expressions that appear to have a non-vanishing imaginary part. The following calculations show that the imaginary parts add up, so that  roots are only taken of positive real numbers.

Note that
$$
9 \cdot \left(\frac{7}{18} i \, \sqrt{3} + \frac{7}{54}\right)^{\frac{1}{3}} =
3 \, \sqrt{7} {\left(\cos\left(\frac{1}{3} \, \arctan\left(3 \, \sqrt{3}\right)\right) + i \, \sin\left(\frac{1}{3} \, \arctan\left(3 \, \sqrt{3}\right)\right)\right)}
$$

and

$$
\frac{7}{{\left(\frac{7}{18} i \, \sqrt{3} + \frac{7}{54}\right)}^{\frac{1}{3}}} =
\left(\cos\left(\frac{1}{3} \, \arctan\left(3 \, \sqrt{3}\right)\right) - i \, \sin\left(\frac{1}{3} \, \arctan\left(3 \, \sqrt{3}\right)\right)\right)
$$

This shows that
$$
\mathrm{Im} \left( 9 \cdot \left(\frac{7}{18} i \, \sqrt{3} + \frac{7}{54}\right)^{\frac{1}{3}} +
\frac{7}{{\left(\frac{7}{18} i \, \sqrt{3} + \frac{7}{54}\right)}^{\frac{1}{3}}} \right) = 0
$$

and

$$
\mathrm{Re} \left( 9 \cdot \left(\frac{7}{18} i \, \sqrt{3} + \frac{7}{54}\right)^{\frac{1}{3}} +
\frac{7}{{\left(\frac{7}{18} i \, \sqrt{3} + \frac{7}{54}\right)}^{\frac{1}{3}}} \right) > 0
$$

```python
# Run this cell in order to analyse the roots of
# the monodromy matrices for v = [1,1,1,1,1,1,0]
# and v = [1,1,1,1,1,1,1]
v = [1,1,1,1,1,1,0]
print("Monodromy Matrix M for v = [1,1,1,1,1,1,0]")
show(pots[str(v)]['mon'])
v = [1,1,1,1,1,1,1]
print("Monodromy Matrix M for v = [1,1,1,1,1,1,1]")
show(pots[str(v)]['mon'])
print("Zeros of M_{2,1}")
for s in pots[str(v)]['sol']:
    show(s)
```

# Result Filtering

The dictionary `pots` contains all possible periodic potentials. In order to filter the results we need to do 3 things

1. create a suitable list of potentials
2. decide whether we want to filter out candidates that don't fulfill the trace condition
3. decide whether we want to allow irrational scaling factors

## Interpretation of Results

### Example 1
```python
v_list = Tuples(sigma,2).list()
for v in v_list:
    print([v, [s for s in pots[str(v)]['sol']],  [s if s in QQ else s.n() for s in pots[str(v)]['trace']]])
```
gives as first line of output
```
[[0, 0], [r1], [-2]]
```
This means that the entry $M_{2,1}$ of monodromy matrix for the potential $v = (0,0)$ is zero for all scalings (`r1` corresponds to $\mathbb{R}$) and that the value of the trace of $M$ is always $-2$.
The line
```
[[1, 0], [0], [-2]]
```
means that the entry $M_{2,1}$ of monodromy matrix for the potential $v = (1,0)$ is zero for the scaling `l = 1` and that the value of the trace of $M$ is, in this case, $-2$.

### Example 2

Similarly to Example 1, we can look at potentials with period $K =  5$. A line like
```
[[1, 0, 1, 1, 0], [-1/2*sqrt(2), 1/2*sqrt(2)], [2.12132034355964, -2.12132034355964]]
```
means that the entry $M_{2,1}$ of monodromy matrix for the potential $v = (1, 0, 1, 1, 0)$ is zero for the tuple of $\left( -\frac{\sqrt{2}}{2}, \frac{\sqrt{2}}{2}\right)$ and that the corresponding values of the trace of $M$ are $(2.12\dots, -2.12\dots)$.

### Example 3

Only list potentials such that $M_{2,1}$ has at least one rational zero.

```python
v_list = Tuples(sigma,9).list()
for v in v_list:
    isRational = [s in QQ for s in pots[str(v)]['sol']]
    if True in isRational:
        print([v, [s for s in pots[str(v)]['sol']],  [s if s in QQ else s.n() for s in pots[str(v)]['trace']]])
```

### Example 4

Only list potentials such that $M_{2,1}$ has at least one rational zero and have absolute trace larger than 2.

```python
v_list = Tuples(sigma,9).list()
for v in v_list:
    isRational = [s in QQ for s in pots[str(v)]['sol']]
    if True in isRational:
        isTraceGt2 = [abs(s.n()) > 2 for s in pots[str(v)]['trace']]
        if True in isTraceGt2:
            print([v, [s for s in pots[str(v)]['sol']],  [s if s in QQ else s.n() for s in pots[str(v)]['trace']]])
```

```python
# example code, modify as indicated above

v_list = Tuples(sigma,5).list()
for v in v_list:
    print([v, [s.full_simplify() for s in pots[str(v)]['sol']],  [s if s in QQ else s.n() for s in pots[str(v)]['trace']]])
```

# JSON Output

Export the `pots` dictionary in JSON file format.

You can read the input using a simple Python program
```python
#!/bin/python3

import json

with open('pots_export.json') as json_file:
    data = json.load(json_file)
    for key in data:
        print(key)
```

or an online JSON reader like https://jsonformatter.org/json-reader .

```python
# run this cell to in order to save pots into json file

import json
def jsonify_dict(input_dict):
##########################
# create json object from dictionary,
# stringify everything for serialization
##########################
    json_object ={}
    for key in input_dict:
        dictionary = {}
        dictionary['mon'] = str(input_dict[key]['mon'])
        dictionary['sol'] = str(input_dict[key]['sol'])
        dictionary['trace'] = str(input_dict[key]['trace'])
        json_object[str(key)] = deepcopy(dictionary)
    return json_object

json_dict = jsonify_dict(pots)

with open('pots_export.json', 'w') as outfile:
    json.dump(json_dict,outfile)
```


Daniel Barker - Cameron SLB, 2017

# Introduction

Over the last few years, I've spent the majority of my professional career developing models of complex dynamic control systems. Historically, I've focused on a physics based approach to the development of such models. Lately, however, I've been working on a new project with a heavy data science focus. One recurring theme that I've noticed in both approaches is the centrality of information.

This has led to a broader interest in data, information, knowledge, and wisdom as well. The objective of this notebook is to learn the fundamentals of information theory in order to gain insight into the fundamentals of data science and modeling/simulation of complex systems. I have found I learn best by teaching, so this notebook will serve as a tutorial for myself (and perhaps others) in the basics of information theory. As this is first and foremost a personal learning objective, my examples will tend to draw from things that are relevant and meaningful to me, but perhaps not others. Readers can expect to see a concrete examples focused on wine, books, and the energy industry, likely among other topics.

I will draw from whatever sources are relevant to learn the material, but the primary test I will be following is <link>"Information Theory: A Tutorial Introduction"</link> by Dr. James V. Stone.

# 1. Information Definition

## 1.1 Definition of a bit

We all a great deal of gratitude to Claude Shannon for his 1933 paper, "A Mathematical Theory of Communication". Without Shannon's seminal work developing the formal study of information theory, all of our modern digital conveniences, from smartphones to the internet, would not be possible.

One of Shannon's central concepts was a formal definition of information, to which he ascribed the unit "bits". Being in posession of one "bit" of information allows you to choose between two equally probably outcomes (e.g. turning left or right on a street).

I like wine, so I'll expound upon this definition with a concrete example. Imagine you are in a restaurant. The sommelier offers you a red or white bottle of wine. Of the red bottles, I then have an option of a lighter or heavier bodied wine, say a Pinot Noir and a Cabernet Sauvignon. Likewise for the whites, I have the option of choosing a Champagne or a Chardonnay. In total, that gives us four possibilities of equal probability (in theory, we all know I would choose the Champagne or Chardonnay). How does the sommelier figure out what wine I want?

Whether he realizes it or not, he directly uses information theory. Likely, his first question would be "red or white?". My response "red" to that question contains one bit of information. Next, he might ask if I want a lighter or heavier bodied red - my answer "lighter bodied" to this question would also contain 1 bit of information, leading the sommelier to recommend the Pinot Noir.

This leads to a generalizable property of binary decision making that is at the heart of information theory. Given 4 possible equally probably outcomes (i.e. the four different bottles of wine), I need 2 bits of information in order to discriminate between the various choices and arrive at a decision. More generally, if I let 'n' equal the number of forks or decisions required to be made (i.e. the number of bits of information required) and 'm' the number of equally probable outcomes, I find that there is a logarithmic relationship between the number of outcomes and the bits of information required.

```python
n = 2
print('The number of bits of information we have = ' + str(n))
# The following equation then gives us the number of possible alternatives
m = 2**n
print('The number of equally probable alternatives given that information = ' + str(m))
```

    The number of bits of information we have = 2
    The number of equally probable alternatives given that information = 4

Conversely, let's assume there are 4096 different varietals of wine in the world - how many yes/no questions (i.e. bits of information) would we have to ask to arrive at an individual varietal? This is where the logarithmic definition developed by Shannon is particularly useful:

```python
import math

m = 4096
print('The number of varietals is = ' + str(m))
n = math.log(m, 2)
print('The number information bits required to choose from that number of varietals is = ' + str(int(n)))
```

    The number of varietals is = 4096
    The number information bits required to choose from that number of varietals is = 12

This means with *only* 12 questions, we could in theory discriminate between 4096 varietals and arrive at a single choice! It's important to note that this is predicated on asking "good" questions. What is a good question - well for our purposes here, mathematically speaking a good question is one that eliminates half the possibilities.

This brings up on important distinction between bits and binary digits. If I ask a question about the grape varietals like, "do any of them grow underground?", the answer to that question is an obvious 'no' since we're not talking about potatoes. That answer 'no' could be encoded as a binary digit (i.e. 0), but it's very important to note that in this case, this binary digit doesn't actually contain any information. Why? Because the question posed didn't allow me to narrow down the list from the set of equally probably varietals. This means that the binary digit, even though it contained a variable (= 0) it didn't actually contain any information. The key point being that bits (i.e. information) and binary digits are NOT necessarily equivalent, and to equate them is actually a category error.

# 2. Entropy of Discrete Variables

This chapter opens with a discussion of the following important ground rules:

Probability

Discrete Variables

Random Variables

These are all important details for more complicated problems, but for sake of brevity I won't discuss them here. Revisit this section if any of these topics seem murky.

What is a message? A message (:= 's') is generated by a source, which generated a sequence of ordered symbols from 1 to k. How do we transmit such a message?

That was one of the things Shannon defined in his original paper. One of Shannon's brilliant insights, and one that I still struggle with, is realizing that the meaning of the message was of no consequence to the engineering problem of sending and receiving a message across a noisy channel. As long as the message is a sequence within the set of all possible sequences, the problem of meaning is irrelevant.

Shannon helped explain this with a diagram very similar to the one shown in the image below (see earlier reference to text). The source generates a message 's' (the sequence of ordered symbols). The message is encoded with some function 'g(s)', which results in an encoded message 'x'. The encoded message is a sequence of codewords. The number of symbols in 's' and the number of codewords in 'x' need not necessarily be equal; the message could be compressed. With lossless compression, the original message s can theoretically be retrieved exactly, while lossy compression irretrievably loses some of the information in the transormation from 's' to 'x'.

This encoded message is then passed as an input into the communication channel. However, a communication channel has the problem of noise. This means that at the output of our channel, our message x is transormed into output y, which consists of the original encoded message 'x' plus the noise. The decoder then attempts to recover the original encoded message 'x', which is interpreted as the original message 's'.

## 2.3 Shannon's Desiderata

Shannon defined the following four requirements for his theory of information to be mathematically useful:

1. Continuity - Changes in the amount of information associated with an outcome (e.g. a coin flip) must be continuous as the probability of that outcome changes. That is, there cannot be discrete jumps in the amount of information when we change the probability of a coin flip coming up heads vs. tails.

2. Symmetry - The amount of information contained in a sequence of outcomes is not dependent on the order of those outcomes.

3. Maximal Value - If a set of outcomes are all equally probable, the amount of information cannot be increased. This will make sense when we talk in more detail about entropy later, but as an example, if I have a jar of fair coins, I can't distinguish one coin from another simply by performing trials of coin flips, because the probability distribution for all will be 0.5. However, if one of the coins is unfair (i.e. not equally probable), this adds information which allows me to distinguish the unfair coin from the set of all other coins by performing a set of coin flip trials.

4. Additive - The information associated with a set of outcomes is obtained by adding the information of the individual outcomes (this one isn't immediately clear to me).

## 2.4 Information, Surprise, and Entropy

The intuition described above in point 3 on maximal value foreshadows a more rigorous definition of surprise, which will eventually lead us to a definition of entropy. Intuitively, something is surprising to us if some observation deviates from an expected value. For example, if I walk into a restaurant on any given Friday night, and suddenly 10 of my friends jump out and yell "surprise" for my birthday, this is a presumably unexpected outcome (although since we would be aware that that particular day is our birthday, we might suspect a surprise could happen, which is additional information, but Bayes' Law is for another day...).

More concretely, we could say surprise is equal to 1/p(x). Since the maximal probability value of an event is equal to 1 (i.e. we're certain that outcome will happen), then something that is has no surprise to us at all would be equal to 1/1 = 1. If we then take the limit as the probability of an event goes to zero, my surprise grows to infinity.

An infinite amount of surprise isn't a particularly useful concept (although it might make for an interesting philosophical question: what is an infinite amount of surprise?), so Shannon decided a better way to describe surprise would be the logarithm of 1/p(x), which regulates an infinite amount of surprise to an asymptotic value. This definition is known as the Shannon information of x. At its core Shannon information is a measure of surprise.

Using the standard notation 'h' used by Shannon, for binary outcomes we use the base 2 log (which gives us information in units of 'bits')

\begin{equation}h(x)=log_{2}(\frac{1}{p(x)}) \quad bits\end{equation}

Which is usually simplified to:

\begin{equation}h(x)=-log_{2}p(x) \quad bits\end{equation}

This leads to a key point: In order to calculate the Shannon surprise of an outcome, we need the probability distribution 'p(X)' in order to calculate the surprise of a particular outcome.

Shannon Information == Entropy:
Calculating the Shannon suprisal of my hypothetical surprise birthday party might be useful as a thought experiment, but practically speaking, we're likely more interested in the how much suprise, on average, is associated with an entire set of possible values. This leads into Shannon's discussion on information entropy, which is equivalent to the idea of Shannon surprise.

```python

```

```python

```


# Lambda School Data Science - A First Look at Data

## Lecture - let's explore Python DS libraries and examples!

The Python Data Science ecosystem is huge. You've seen some of the big pieces - pandas, scikit-learn, matplotlib. What parts do you want to see more of?

```
# TODO - we'll be doing this live, taking requests
# and reproducing what it is to look up and learn things
```

## Assignment - now it's your turn

Pick at least one Python DS library, and using documentation/examples reproduce in this notebook something cool. It's OK if you don't fully understand it or get it 100% working, but do put in effort and look things up.

```
import sympy as sym #Using Sympy library to simplify/represent expressions

x=sym.symbols('x')

x-x

```

    0

```
m, n= sym.symbols('m, n')

sqrs=(m-n)**2
sqrs
```

    (m - n)**2

```
sym.init_printing() #puts the expression into more pleasant form
sqrs
```

```
sqrs.expand() #expands expressions
```

```

sym.limit(1/x,x,0,dir="-") #Computes limits
```

```
#Can differentiate and integrate symbols
sym.diff(sym.sin(x**2), x)

```

```
sym.integrate(x ** 2 - sym.cos(x), x)
```

```
#In tandem with matplotlib we can plot general equations

%matplotlib inline

cubic=x**3
p=sym.plot(cubic);
```

### Assignment questions

After you've worked on some code, answer the following questions in this text block:

1.  Describe in a paragraph of text what you did and why, as if you were writing an email to somebody interested but nontechnical.

2.  What was the most challenging part of what you did?

3.  What was the most interesting thing you learned?

4.  What area would you like to explore with more time?

1. Perusing the list that Ryan Allred provided, I went deep into the DS notebook world for mathematics and found a notebook sampling a library I hadnt used yet. I picked sympy because I found the ease with which it computed symbolic math astounding. I used sympy to compute expressions with variables and to do some light calculus, as well as to graph.

2. The most challenging part was perusing Github for notebooks. Its like a vast sea of libraries and applications of thought-- which was a tad daunting.

3. How to use Github and that symbolic math can be easily computed.
4. I feel like a lot of this was teaching me how to self-teach with the tools available to me; Github, google, etc. I'd like to learn more about these tools so I can easily create things of my own.

## Stretch goals and resources

Following are *optional* things for you to take a look at. Focus on the above assignment first, and make sure to commit and push your changes to GitHub (and since this is the first assignment of the sprint, open a PR as well).

- [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/)
- [scikit-learn documentation](http://scikit-learn.org/stable/documentation.html)
- [matplotlib documentation](https://matplotlib.org/contents.html)
- [Awesome Data Science](https://github.com/bulutyazilim/awesome-datascience) - a list of many types of DS resources

Stretch goals:

- Find and read blogs, walkthroughs, and other examples of people working through cool things with data science - and share with your classmates!
- Write a blog post (Medium is a popular place to publish) introducing yourself as somebody learning data science, and talking about what you've learned already and what you're excited to learn more about.


```python
import sympy as sm
import sympy.physics.mechanics as me
me.init_vprinting()
```

```python
N, A, B = sm.symbols("N, A, B", cls=me.ReferenceFrame)
```

```python
alpha, beta, d = me.dynamicsymbols('alpha, beta, d')
```

```python
A.orient_axis(N, alpha, N.z)
B.orient_axis(A, beta, A.x)
```

```python
A.ang_vel_in(N)
```

```python
A.ang_vel_in(N).express(A)
```

```python
B.ang_vel_in(A)
```

```python
A.ang_vel_in(N) + B.ang_vel_in(A)
```

```python
B.ang_vel_in(N)
```

```python
A.ang_acc_in(N)
```

```python
B.ang_acc_in(A)
```

```python
B.ang_acc_in(N)
```

```python
A.ang_acc_in(N) + B.ang_acc_in(A)
```

```python
B.ang_acc_in(N).express(B)
```

```python

```


## Simple Linear Regression

In this example, we illustrate how to solve a linear regression problem.
Suppose we have training data, can we fit a neural network on it? The trained neural network (an MLP) is then validated on the test split.

Let's load the python packages first.

```python
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

# numpy package
import numpy as np
# for plotting
import matplotlib.pyplot as plt

# keras modules
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.utils import plot_model
```

### Generate Artificial Data

Assuming we have 1D input and 1D output, let us generate the train data assuminng the underlying function is a polynomial:
\begin{align}
y= f(x) = x^2 - 2x + 1
\end{align}

```python
# generate train x data
x_train = np.random.uniform(low=-1.0, high=2.0, size=[200,1])
# generate train y data
y_train = x_train*x_train - 2*x_train + 1

plt.figure()
plt.plot(x_train, y_train, 'bo')
```

Then, let us generate the test data and plot the train and test data together. Note that the test data is beyond the [-1,2] range of `x_train`. This will enable us to verify if our model can generalize (ie make prediction from never seen before data)

```python
# generate test x data
x_test = np.random.uniform(low=-2.0, high=3.0, size=[20,1])
# generate test y data
y_test = x_test*x_test - 2*x_test + 1

plt.plot(x_train, y_train, 'bo', x_test, y_test, 'r*')
plt.show()
```

### 3-layer MLP Model

Then, let us build a 3-layer 64-128-1 MLP.

```python
# build 2-layer MLP network
model = Sequential(name='linear_regressor')
# 1st layer (input layer) has 64 units (perceptron), input is 1-dim
model.add(Dense(units=64, input_dim=1, activation='relu', name='input_layer'))
# 2nd layer has 128 unit, output is 1-dim
model.add(Dense(units=128, activation='relu', name='hidden_layer'))
# 3rd layer (output layer) has 1 unit, output is 1-dim
model.add(Dense(units=1, name='output_layer'))
# print summary to double check the network
model.summary()
# indicate the loss function and use stochastic gradient descent
# (sgd) as optimizer
model.compile(loss='mse', optimizer='sgd')
# feed the network with complete dataset (1 epoch) 100 times
# batch size of sgd is 4
history = model.fit(x_train, y_train, epochs=100, batch_size=4, validation_data=(x_test, y_test))
# simple validation by predicting the output based on x
y_pred = model.predict(x_test, verbose=0)
```

    Model: "linear_regressor"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    input_layer (Dense)          (None, 64)                128
    _________________________________________________________________
    hidden_layer (Dense)         (None, 128)               8320
    _________________________________________________________________
    output_layer (Dense)         (None, 1)                 129
    =================================================================
    Total params: 8,577
    Trainable params: 8,577
    Non-trainable params: 0
    _________________________________________________________________
    Epoch 1/100
    50/50 [==============================] - 0s 3ms/step - loss: 1.0276 - val_loss: 4.9706
    Epoch 2/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.1931 - val_loss: 2.7254
    Epoch 3/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0804 - val_loss: 2.2440
    Epoch 4/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0575 - val_loss: 1.8591
    Epoch 5/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0475 - val_loss: 1.6330
    Epoch 6/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0409 - val_loss: 1.5586
    Epoch 7/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0375 - val_loss: 1.4400
    Epoch 8/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0333 - val_loss: 1.4230
    Epoch 9/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0307 - val_loss: 1.3709
    Epoch 10/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0289 - val_loss: 1.2765
    Epoch 11/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0266 - val_loss: 1.3010
    Epoch 12/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0244 - val_loss: 1.2236
    Epoch 13/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0235 - val_loss: 1.2113
    Epoch 14/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0214 - val_loss: 1.2156
    Epoch 15/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0196 - val_loss: 1.1431
    Epoch 16/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0186 - val_loss: 1.1463
    Epoch 17/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0171 - val_loss: 1.1123
    Epoch 18/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0156 - val_loss: 1.0624
    Epoch 19/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0146 - val_loss: 1.0403
    Epoch 20/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0138 - val_loss: 1.0194
    Epoch 21/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0130 - val_loss: 0.9956
    Epoch 22/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0110 - val_loss: 0.9711
    Epoch 23/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0113 - val_loss: 0.9489
    Epoch 24/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0103 - val_loss: 0.9598
    Epoch 25/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0097 - val_loss: 0.9324
    Epoch 26/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0089 - val_loss: 0.9162
    Epoch 27/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0084 - val_loss: 0.9133
    Epoch 28/100
    50/50 [==============================] - 0s 2ms/step - loss: 0.0079 - val_loss: 0.9003
    Epoch 29/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0072 - val_loss: 0.9243
    Epoch 30/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0070 - val_loss: 0.8808
    Epoch 31/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0064 - val_loss: 0.8721
    Epoch 32/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0061 - val_loss: 0.8552
    Epoch 33/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0057 - val_loss: 0.8433
    Epoch 34/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0053 - val_loss: 0.8256
    Epoch 35/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0051 - val_loss: 0.8175
    Epoch 36/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0047 - val_loss: 0.7790
    Epoch 37/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0046 - val_loss: 0.7914
    Epoch 38/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0044 - val_loss: 0.7798
    Epoch 39/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0041 - val_loss: 0.7754
    Epoch 40/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0040 - val_loss: 0.7534
    Epoch 41/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0038 - val_loss: 0.7344
    Epoch 42/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0035 - val_loss: 0.7535
    Epoch 43/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0033 - val_loss: 0.7402
    Epoch 44/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0032 - val_loss: 0.7357
    Epoch 45/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0032 - val_loss: 0.7198
    Epoch 46/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0031 - val_loss: 0.7145
    Epoch 47/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0029 - val_loss: 0.7006
    Epoch 48/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0028 - val_loss: 0.6839
    Epoch 49/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0026 - val_loss: 0.7008
    Epoch 50/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0026 - val_loss: 0.6864
    Epoch 51/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0024 - val_loss: 0.6754
    Epoch 52/100
    50/50 [==============================] - 0s 946us/step - loss: 0.0024 - val_loss: 0.6899
    Epoch 53/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0024 - val_loss: 0.6637
    Epoch 54/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0022 - val_loss: 0.6664
    Epoch 55/100
    50/50 [==============================] - 0s 862us/step - loss: 0.0023 - val_loss: 0.6661
    Epoch 56/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0021 - val_loss: 0.6515
    Epoch 57/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0020 - val_loss: 0.6346
    Epoch 58/100
    50/50 [==============================] - 0s 913us/step - loss: 0.0020 - val_loss: 0.6373
    Epoch 59/100
    50/50 [==============================] - 0s 937us/step - loss: 0.0019 - val_loss: 0.6442
    Epoch 60/100
    50/50 [==============================] - 0s 931us/step - loss: 0.0018 - val_loss: 0.6325
    Epoch 61/100
    50/50 [==============================] - 0s 957us/step - loss: 0.0018 - val_loss: 0.6241
    Epoch 62/100
    50/50 [==============================] - 0s 968us/step - loss: 0.0018 - val_loss: 0.6105
    Epoch 63/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0017 - val_loss: 0.6168
    Epoch 64/100
    50/50 [==============================] - 0s 944us/step - loss: 0.0017 - val_loss: 0.6222
    Epoch 65/100
    50/50 [==============================] - 0s 934us/step - loss: 0.0016 - val_loss: 0.6176
    Epoch 66/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0016 - val_loss: 0.6129
    Epoch 67/100
    50/50 [==============================] - 0s 930us/step - loss: 0.0015 - val_loss: 0.5915
    Epoch 68/100
    50/50 [==============================] - 0s 917us/step - loss: 0.0015 - val_loss: 0.6006
    Epoch 69/100
    50/50 [==============================] - 0s 908us/step - loss: 0.0014 - val_loss: 0.5912
    Epoch 70/100
    50/50 [==============================] - 0s 990us/step - loss: 0.0014 - val_loss: 0.5981
    Epoch 71/100
    50/50 [==============================] - 0s 980us/step - loss: 0.0014 - val_loss: 0.5944
    Epoch 72/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0014 - val_loss: 0.5867
    Epoch 73/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0013 - val_loss: 0.5740
    Epoch 74/100
    50/50 [==============================] - 0s 962us/step - loss: 0.0013 - val_loss: 0.5755
    Epoch 75/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0013 - val_loss: 0.5755
    Epoch 76/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0012 - val_loss: 0.5674
    Epoch 77/100
    50/50 [==============================] - 0s 846us/step - loss: 0.0012 - val_loss: 0.5701
    Epoch 78/100
    50/50 [==============================] - 0s 900us/step - loss: 0.0012 - val_loss: 0.5657
    Epoch 79/100
    50/50 [==============================] - 0s 930us/step - loss: 0.0011 - val_loss: 0.5661
    Epoch 80/100
    50/50 [==============================] - 0s 906us/step - loss: 0.0011 - val_loss: 0.5600
    Epoch 81/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0011 - val_loss: 0.5582
    Epoch 82/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0011 - val_loss: 0.5476
    Epoch 83/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0011 - val_loss: 0.5522
    Epoch 84/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0010 - val_loss: 0.5569
    Epoch 85/100
    50/50 [==============================] - 0s 1ms/step - loss: 0.0010 - val_loss: 0.5534
    Epoch 86/100
    50/50 [==============================] - 0s 1ms/step - loss: 9.9959e-04 - val_loss: 0.5383
    Epoch 87/100
    50/50 [==============================] - 0s 1ms/step - loss: 9.7837e-04 - val_loss: 0.5421
    Epoch 88/100
    50/50 [==============================] - 0s 1ms/step - loss: 9.6033e-04 - val_loss: 0.5373
    Epoch 89/100
    50/50 [==============================] - 0s 1ms/step - loss: 9.5100e-04 - val_loss: 0.5317
    Epoch 90/100
    50/50 [==============================] - 0s 1ms/step - loss: 8.7060e-04 - val_loss: 0.5299
    Epoch 91/100
    50/50 [==============================] - 0s 1ms/step - loss: 9.2609e-04 - val_loss: 0.5346
    Epoch 92/100
    50/50 [==============================] - 0s 1ms/step - loss: 8.8382e-04 - val_loss: 0.5254
    Epoch 93/100
    50/50 [==============================] - 0s 1ms/step - loss: 8.7679e-04 - val_loss: 0.5178
    Epoch 94/100
    50/50 [==============================] - 0s 1ms/step - loss: 8.3923e-04 - val_loss: 0.5225
    Epoch 95/100
    50/50 [==============================] - 0s 1ms/step - loss: 8.1455e-04 - val_loss: 0.5237
    Epoch 96/100
    50/50 [==============================] - 0s 1ms/step - loss: 8.0873e-04 - val_loss: 0.5184
    Epoch 97/100
    50/50 [==============================] - 0s 1ms/step - loss: 8.0517e-04 - val_loss: 0.5233
    Epoch 98/100
    50/50 [==============================] - 0s 993us/step - loss: 7.8329e-04 - val_loss: 0.5172
    Epoch 99/100
    50/50 [==============================] - 0s 1ms/step - loss: 7.5604e-04 - val_loss: 0.5214
    Epoch 100/100
    50/50 [==============================] - 0s 974us/step - loss: 7.6083e-04 - val_loss: 0.5108
    WARNING:tensorflow:11 out of the last 11 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f9590811440> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.

### Qualitative Evaluation
Let us plot the prediction (green dots) vs test dataset (red stars).
We can see that the test data deviates largely when the input is beyond the range of the train data.

```python
plt.plot(x_test, y_test, 'r*', x_test, y_pred, 'go', )
plt.show()
```

### Examine the Train History

The `history` variable stores value informationn during training such as value of loss function.

```python
print(history.history.keys())
```

    dict_keys(['loss', 'val_loss'])

```python
# Plot history: MSE
plt.plot(history.history['loss'], label='Train loss')
plt.plot(history.history['val_loss'], label='Test loss')
plt.title('MSE')
plt.ylabel('MSE value')
plt.xlabel('Epochs')
plt.legend(loc="upper right")
plt.show()
```

```python

```


```python
%pylab inline
%config InlineBackend.figure_format = 'retina'
from ipywidgets import interact
import scipy.stats as stats
import pandas as pd
# import pymc3 as pm
import arviz as az
import seaborn as sns

import tensorflow as tf
import tensorflow_probability as tfp
tfd = tfp.distributions
tfb = tfp.bijectors

import warnings # the warning spam is pointless and annoying
warnings.simplefilter(action="ignore", category=FutureWarning)
```

    Populating the interactive namespace from numpy and matplotlib

# Example
Suppose we are modeling animal movement, specifically polar bears. We are planning to utilize data that comes from attaching a tracking collar to our polar bear. But first we are exploring possible models.

Let's start by modeling the movement of a bear as a sequence of random steps.

## Suppose for model 1 (m1) we pick normal random steps $x,y \sim \text{Normal}(0, 1)$

```python
Nsteps = 10000

## normally distributed steps
X_m1 = normal(0, 1, [Nsteps, 2])
```

```python
## random path of a hypothetical polar bear
Z_m1 = X_m1.cumsum(axis=0) ## this is the path that results from adding up steps
plot(Z_m1[:, 0], Z_m1[:, 1])
```

```python
H, xb, yb = histogram2d(X_m1[:, 0], X_m1[:, 1])
pcolor(xb, yb, H);
# sns.kdeplot(X[:, 0], X[:, 1])
```

```python
sns.distplot(X_m1[:, 1])
```

## Suppose for model 2 (m2) we pick a random step length $r\sim\text{Normal}(0, 1)$ and a random direction $\theta \sim \text{Uniform}(0, 2\pi)$

```python
## Random step length and a random direction
R_m2 = normal(0, 1, Nsteps) ## normal step length
theta_m2 = 2*pi*rand(Nsteps) ## uniform direction
```

```python
X_m2 = stack([R_m2*cos(theta_m2), R_m2*sin(theta_m2)], 1)
Z_m2 = X_m2.cumsum(axis=0) ## this is the path that results from adding up steps
plot(Z_m2[:, 0], Z_m2[:, 1]);
```

```python
H, xb, yb = histogram2d(X_m2[:, 0], X_m2[:, 1])
pcolor(xb, yb, H);
# sns.kdeplot(X2[:, 0], X2[:, 1])
```

```python
sns.distplot(R_m2);
```

## Are the two models equivalent?

```python
sns.distplot(X_m1[:, 0], label='model 1')
sns.distplot(X_m2[:, 0], label='model 2')# a slice of our 2D density does not look Normal
legend()
```

## what happens if we look at step lengths from the first model where $x,y\sim\text{Normal}(0, 1)$?

```python
R_m1 = sqrt(X_m1[:, 0]**2 + X_m1[:, 1]**2) ## length of the step
sns.distplot(R_m1, label='model 1')
sns.distplot(absolute(R_m2), label='model 2')
xlabel(r'$r$', fontsize=24)
legend();
```

## Conclusion: these are different models and not equivalent
The fundamental reason for the confusion in our intuition is that we failed to account for how the *area* (volume in general) of an infinitesimal region changes when we change variables.

Consider the region $S_{dr}$ to be the small circle of radius $dr$ around $x,y=0,0$. The area of $S_{dr}$ is $A=\pi dr^2$. In the first model, the center of $S_{dr}$ is the most likely spot, yet it has very small area, so samples will be very unlikely to be found in $S_{dr}$. This is kind of like the bulls eye of a dart board. The probability of hitting the bulls eye is $P(x \in S_{dr}) \approx \text{Area}(S_{dr}) p(0)$. **We have the competition of two factors: the small area $\text{Area}(S_{dr})$ and the large density $p(0)$.** The combination yields a vanishing density in the variable $r$ as $r\to 0^+$.

In the second model we pick normal random step lengths (in model 2) which will result in the density for $r$ being Gaussian and therefore maximal at $r=0$.

### We need a conversion factor that tells us how $\text{Area}(x, y; dx, dy)$ is related to $\text{Area}(r, \theta; dr, d\theta)$

We have
$$ x = r\cos(\theta), \quad y = r\sin(\theta) $$
The Jacobian
$$ J = \begin{bmatrix}
\cos(\theta) & -r\sin(\theta) \\
\sin(\theta) & r\cos(\theta)
\end{bmatrix}$$
tells us
$$
\begin{bmatrix}
dx \\ dy
\end{bmatrix}
=
\begin{bmatrix}
\cos(\theta) & -r\sin(\theta) \\
\sin(\theta) & r\cos(\theta)
\end{bmatrix}
\begin{bmatrix}
dr \\ d\theta
\end{bmatrix}
$$
The area of an infinitesimal region satisfies
$$ dxdy = \det(J)dr d\theta.$$
The determinant of the Jacobian is
$$\det(J) = r.$$

---------------------------------------------

# Change of variables for a random variable
Suppose that we have a RV $X\in \mathbb{R}$ with density function $p(x)$. Suppose that $f(\cdot)$ is a bijection so that
$$ y = f(x) \Leftarrow\Rightarrow f^{-1}(y) = x.$$

Consider the new RV $$Y = f(X).$$ What is the density function for $Y$? Is it $$ g(y) = p(f^{-1}(y))?$$

Consider the integral
$$\int_{-\infty}^{\infty}p(x)dx = 1$$
We must also have
$$\int_{-\infty}^{\infty}g(y)dy = 1$$
So then
$$\int_{-\infty}^{\infty}p(f^{-1}(y))dy = ?$$

## Recall Calculus
How do we do a change of variables in an integral?
$$ y = f(x) \Rightarrow dy = f'(x)dx \Rightarrow dx = \frac{dy}{f'(x)}$$

$$ 1 = \int_{-\infty}^{\infty}p(x)dx
= \int_{-\infty}^{\infty}p(f^{-1}(y)) dx
= \int_{-\infty}^{\infty}p(f^{-1}(y)) \frac{dy}{f'(x)}
= \int_{-\infty}^{\infty}\frac{p(f^{-1}(y))}{f'(f^{-1}(y))} dy$$

Alternatively,
$$ x = f^{-1}(y) \Rightarrow dx = (f^{-1})'(y)dy,$$
$$ 1 = \int_{-\infty}^{\infty} (f^{-1})'(y) p(f^{-1}(y)) dy$$

## Conclusion
The probability density for $Y = f(X)$ is
$$g(y) = (f^{-1})'(y) p(f^{-1}(y)) $$

## In higher dimensions
If $X\in \mathbb{R}^n$, $Y \in\mathbb{R}^n$, and $Y = f(X)$ then the density function for $Y$ is
$$ g(y) = \det(D_y [f^{-1}](y)) p(f^{-1}(y)), $$
where $D_y$ denotes the operator that yields the Jacobian matrix of partial derivatives.

## The volume conversion factor we were looking for is given by
$$\det(D_y [f^{-1}](y))$$

## Measure theory generalization
The volume conversion factor is called the [Radon-Nikodym derivative](https://en.wikipedia.org/wiki/Radon%E2%80%93Nikodym_theorem).

----------------------

# Bijectors
A [bijector](https://en.wikipedia.org/wiki/Bijection) is a function that can be inverted. That is, if $y = f(x)$ then there exists an inverse function such that $x = f^{-1}(y)$.

## Experiment
The following is an experiment with the kernel density estimator (KDE). The standard KDE is meant to approximate the density function for an unbounded RV $x\in\mathbb{R}$ (or higher dimensions). The idea in this experiment is to impose constraints on the RV so that, for example, the density is restricted to $x\geq 0$ or $x \in [a, b]$. This is a reasonable idea, but **fair warning, I do not know how well behaved the resulting approximation will be.**

```python
Nsamples = 5000
u = rand(Nsamples)
lam = 5.
tau = -log(u)/lam
f = tfb.Log()
y = f.forward(tau)
py = stats.kde.gaussian_kde(y)
# plot(py(linspace(-5, 3)))
def p_tau(tau):
    y = array(f.forward(tau))
    logJy = array(f.forward_log_det_jacobian(tau, 0))
    return exp(logJy + py.logpdf(y))
tau_plot = linspace(1e-2, 2, 100)
density_plot = p_tau(tau_plot)
plot(tau_plot, density_plot, 'r')
plot(tau_plot, lam*exp(-lam*tau_plot), 'k')
trapz(density_plot, tau_plot)
sns.distplot(tau)
```

```python
Nsamples = 5000
u = -1 + 2*rand(Nsamples)
# f = tfb.Invert(tfb.Tanh())
f = tfb.Invert(tfb.SoftClip(low=-1-1e-7, high=1+1e-7, hinge_softness=4))
y = f.forward(u)
py = stats.kde.gaussian_kde(y)
# plot(py(linspace(-5, 3)))
def p_u(u):
    y = array(f.forward(u))
    logJy = array(f.forward_log_det_jacobian(u, 0))
    return exp(logJy + py.logpdf(y))
u_plot = linspace(-1, 1, 100)
density_plot = p_u(u_plot)
plot(u_plot, density_plot, 'r')
ylim(0, density_plot.max()*1.2)
trapz(density_plot, u_plot)
sns.distplot(u)
```

# Bijectors with MCMC
We will recreate our globe tossing model. Recall the probability parameter in a Binomial is $0\leq p \leq 1$. How do we sample the posterior $\pi(p)$ when Hamiltonian Monte Carlo is designed to sample distributions in $\mathbb{R}^n$?

```python
w = 6 ## from Chapter 3 and Week 3 lecture
n = 9

# data = 1*(rand(100) < 0.7)
# w = data.sum()
# n = data.size
```

```python
Nsteps = 2000
N_burn_in = Nsteps

Nchains = 4

def log_target_density(p):
    return tfd.Binomial(n, probs=p).unnormalized_log_prob(w)

@tf.function()
def sampler():
    def trace_everything(states, previous_kernel_results):
        return previous_kernel_results
    Xinit = rand(Nchains)
    kernel = tfp.mcmc.NoUTurnSampler(
        target_log_prob_fn=log_target_density,
        max_tree_depth=8,
        max_energy_diff=1000,
        step_size=tf.cast(0.1, tf.float64))

    # kernel = tfp.mcmc.HamiltonianMonteCarlo(
    #     target_log_prob_fn=log_target_density,
    #     num_leapfrog_steps=10,
    #     step_size=tf.cast(0.1, tf.float64))

    bijectors = [tfb.NormalCDF()]

    transformed_kernel = tfp.mcmc.TransformedTransitionKernel(
        inner_kernel=kernel,
        bijector=bijectors)

    adaptive_kernel = tfp.mcmc.DualAveragingStepSizeAdaptation(
        inner_kernel=transformed_kernel,
        num_adaptation_steps=int(0.8 * N_burn_in),
        target_accept_prob=tf.cast(0.7, tf.float64))

    Xchain, trace = tfp.mcmc.sample_chain(
        num_results=Nsteps,
        num_burnin_steps=N_burn_in,
        current_state=Xinit,
        trace_fn=trace_everything,
        kernel=adaptive_kernel,
        parallel_iterations=4)
    return Xchain, trace
Xchain, trace = sampler()
Xchain = array(Xchain)
```

    WARNING:tensorflow:@custom_gradient grad_fn has 'variables' in signature, but no ResourceVariables were used on the forward pass.

```python
tfp.mcmc.effective_sample_size(Xchain)
```

    <tf.Tensor: shape=(4,), dtype=float64, numpy=array([1339.05453023, 1264.62717084, 1414.23880154, 1536.30757449])>

```python
sns.distplot(Xchain.flatten())
x_to_plot = linspace(0, 1, 100)
plot(x_to_plot, stats.beta.pdf(x_to_plot, w + 1, n - w + 1), '--k', label="True posterior")
legend()
xlabel(r'$p$', fontsize=24)
ylabel('probability density', fontsize=20);
```

# A full working Tensorflow + Bijector version of the random time HMC from Week 11

```python
w = 6 ## from Chapter 3 and Week 3 lecture
n = 9
```

```python
Nsteps = 2000
N_burn_in = Nsteps
Nchains = 8

lam = 0.7
dt = 0.1

bijector = tfb.NormalCDF() ## our bijector

x0 = normal(0, 1, [Nchains, 1])

@tf.function()
def sample(x0):
    def L(x):
        log_likelihood = tfd.Binomial(n, probs=bijector.forward(x)).unnormalized_log_prob(w)
        log_prior = 0. ## we could change this to an informative prior
        log_J = bijector.forward_log_det_jacobian(x, 0)
        return log_likelihood + log_prior + log_J
    def gradL(x):
        with tf.GradientTape() as tape:
            tape.watch(x)
            L_at_x = L(x)
        return tape.gradient(L_at_x, x)
    def H(x, p):
        return -L(x) + 0.5*p**2

    x = tf.identity(x0) ## make a copy of the values in the Variable x0

    ## Burn in steps
    for j in tf.range(N_burn_in): ## notice that this is not `arange` or `range`
        p0 = tf.random.normal(mean=0, stddev=1, shape=[Nchains, 1], dtype=tf.float64)
        p = tf.identity(p0) ## make a copy of p0
        tau = -tf.math.log(tf.random.uniform(shape=[1]))/lam ## generate an exponential RV for time
        K = tf.cast(tau/dt, tf.int64) ## cast (convert) a float to an integer
        for k in tf.range(K[0]):
            _p = p + 0.5*dt*gradL(x)
            x = x + dt*_p
            p = _p + 0.5*dt*gradL(x)

    Xchain = tf.TensorArray( ## We have to use this to store values in a tensorflow loop
        tf.float64,
        size=Nsteps,
        element_shape=[Nchains, 1])
    Xchain = Xchain.write(0, x) ## The write method saves a time step (the first in this case)

    for j in tf.range(Nsteps-1):
        p0 = tf.random.normal(mean=0, stddev=1, shape=[Nchains, 1], dtype=tf.float64)
        x0 = tf.identity(x) ## make a copy of the values in the Variable x
        p = tf.identity(p0)
        tau = -tf.math.log(tf.random.uniform(shape=[1]))/lam ## generate an exponential RV for time
        K = tf.cast(tau/dt, tf.int64)
        for k in tf.range(K[0]):
            _p = p + 0.5*dt*gradL(x)
            x = x + dt*_p
            p = _p + 0.5*dt*gradL(x)
        log_p_MH = H(x0, p0) - H(x, p) ## if our numerical integrator is perfect, this should always be one
        u = tf.random.uniform(shape=[Nchains, 1], dtype=tf.float64)
        chi = tf.cast(tf.math.log(u) < log_p_MH, tf.float64) ## convert a boolean (True/False) to a float
        x = x0*(1 - chi) + x*chi
        Xchain = Xchain.write(j+1, x)
    return bijector.forward(Xchain.stack()) ## note the final application of the bijector
Xchain = sample(x0)
Xchain = array(Xchain)
```

```python
tfp.mcmc.effective_sample_size(Xchain)
```

    <tf.Tensor: shape=(8, 1), dtype=float64, numpy=
    array([[1597.48126007],
           [1221.27567889],
           [1412.30453425],
           [1440.66529007],
           [1630.61013096],
           [1382.13776459],
           [1635.34584095],
           [1465.03662131]])>

```python
sns.distplot(Xchain.flatten())
x_to_plot = linspace(0, 1, 100)
plot(x_to_plot, stats.beta.pdf(x_to_plot, w + 1, n - w + 1), '--k', label="True posterior")
legend()
xlabel(r'$p$', fontsize=24)
ylabel('probability density', fontsize=20);
```

```python

```

# State space models
Let us continue with our example about tracking polar bears. Suppose we now have data from a tracking collar of one of our bears. The GPS measurements that come from the collar are samples at equal time intervals of 1 hour. Our collar measurements will be given by
$$ z_{k} \sim \text{Normal}(x_{k}, \sigma_{\rm obs}),$$
where $x_{k}$ is the actual position of the bear at times $t_k = k$. We will model the actual position of the bear as a simple random walk
$$ x_k \sim \text{Normal}(x_{k-1}, \sigma)$$

The full generative model is then
\begin{align}
  z_{k} &\sim \text{Normal}(x_{k}, \sigma_{\rm obs}) \\
   x_k &\sim \text{Normal}(x_{k-1}, \sigma) \\
   x_0 & \sim  \text{Normal}(0, 25)\\
   \sigma_{\rm obs} &\sim ??? \\
   \sigma &\sim ???
\end{align}

## The variables $x_i$ are unobserved
Things we might be interested in knowing:
  1. The full joint posterior over the unobserved positions
    $$\pi(x_0, \ldots, x_T | z_1, \ldots, z_T; \theta) = \frac{\prod_{k=1}^{T}g(z_k| x_k)f(x_k | x_{k-1})p_0(x_0)}{\int_{R}\cdots \int_{R}\prod_{k=1}^{T}g(z_k| x_k)f(x_k | x_{k-1})p_0(x_0) \,dx_0\cdots dx_T}$$
  2. The marginals
    $$\pi(x_k | z_1, \ldots, z_T)$$

# Expectation Maximization Algorithm
  1. E-step: given fixed parameters $\theta_j$ compute
  $$ X_{1:T} | z_{1:T}, \theta_j$$
  \begin{align}
  Q(\theta | \theta_j) &= E[\log(\prod_{k=1}^{T}g(z_k| X_k, \theta)f(X_k | X_{k-1}, \theta)p_0(X_0|\theta)) | \theta_j]\\
  &= E[\sum_{k=1}^{T}\log(g(z_k| X_k, \theta)) + \sum_{k=1}^{T}\log(f(X_k | X_{k-1}, \theta)) + \log(p_0(X_0|\theta)) | \theta_j]
  \end{align}
  2. M-step: compute $$\theta_{j+1} = \text{arg max}_{\theta} Q(\theta | \theta_j)  $$

Repeat these steps until convergence is reached.

## Simulate collar tracking GPS data

```python
Nt = 10
sigma_obs = 0.2
sigma = 2.

X0true = -25. + 50.*rand(1, 2)
Xtrue = X0true + normal(0, 2., [Nt, 2]).cumsum(axis=0)
Z = Xtrue + normal(0, sigma_obs, [Nt, 2])
plot(Z[:, 0], Z[:, 1]);
```

## State space model in Tensorflow
First, we will sample the posterior
$$\pi(x_0, x_1, \ldots, x_T | z_1, \ldots, z_T; \theta) \propto \prod_{k=1}^{T}g(z_k| x_k)f(x_k | x_{k-1})p_0(x_0)$$
The generative model is
$$p(x_0, x_1, \ldots, x_T, z_1, \ldots, z_T | \theta) = \prod_{k=1}^{T}g(z_k| x_k)f(x_k | x_{k-1})p_0(x_0)$$

```python
Nsteps = 50
N_burn_in = Nsteps
Nchains = 2

Sigma_obs = sigma_obs*tf.ones([1, 1, 2], dtype=tf.float64)
Sigma = sigma*tf.ones([1, 1, 2], dtype=tf.float64)
Sigma_x0 = 100*tf.ones([1, 2], dtype=tf.float64)
def log_target_density(Xfull):
    ## X : [parallel chains, time, space]
    X0 = Xfull[:, 0, :]
    X = Xfull[:, 1:, :] ## same shape as `Z`
    ## observation likelihood: density for Z given X
    G = tfd.MultivariateNormalDiag(
        loc=X,
        scale_diag=Sigma_obs)
    logG = tf.reduce_sum(
        G.unnormalized_log_prob(Z),
        axis=1)
    ## motion model: density for X
    F = tfd.MultivariateNormalDiag(
        loc=Xfull[:, :-1, :],
        scale_diag=Sigma)
    logF = tf.reduce_sum(
        F.unnormalized_log_prob(X),
        axis=1)
    ## initial position prior: density for X0
    F0 = tfd.MultivariateNormalDiag(
        loc=tf.zeros([1, 2], dtype=tf.float64),
        scale_diag=Sigma_x0)
    logF0 = F0.unnormalized_log_prob(X0)
    return logG + logF + logF0

@tf.function()
def track_sampler(Xinit):
    kernel = tfp.mcmc.NoUTurnSampler(
        target_log_prob_fn=log_target_density,
        max_tree_depth=8,
        max_energy_diff=1000,
        step_size=tf.cast(0.1, tf.float64))

    # kernel = tfp.mcmc.HamiltonianMonteCarlo(
    #     target_log_prob_fn=log_target_density,
    #     num_leapfrog_steps=10,
    #     step_size=tf.cast(0.1, tf.float64))

    adaptive_kernel = tfp.mcmc.DualAveragingStepSizeAdaptation(
        inner_kernel=kernel,
        num_adaptation_steps=int(0.8 * N_burn_in),
        target_accept_prob=tf.cast(0.7, tf.float64))

    Xchain = tfp.mcmc.sample_chain(
        num_results=Nsteps,
        num_burnin_steps=N_burn_in,
        current_state=Xinit,
        #trace_fn=trace_everything,
        trace_fn=None,
        kernel=adaptive_kernel,
        parallel_iterations=4)
    return Xchain

Xinit = zeros([Nchains, Nt+1, 2])
Xinit[:, 0, :] = normal(0, 25., [Nchains, 2])
Xinit[:, 1:, :] = Z + normal(0, 5., [Nchains, Nt, 2])
Xchain = track_sampler(Xinit)
Xchain = array(Xchain)
```

```python
mean(tfp.mcmc.effective_sample_size(Xchain))
```

    44.1706914348532

```python
## plot results
```

```python
figure(1, [7, 7])
## Xchain: [mcmc steps, parallel chains, time, space]
plot(Xchain[..., 0].reshape(-1, Nt+1), Xchain[..., 1].reshape(-1, Nt+1), '.')
plot(Z[:, 0], Z[:, 1], 'r')
xlabel('x', fontsize=20)
ylabel('y', fontsize=20);
```

# Suppose we want to sample $X$ at times that are in between the observation times

```python
Nt = 100
sigma_subsample = sigma/sqrt(10)

Zpadded = zeros([Nt, 2])
Zpadded[::10] = Z
### where Z is zero, we will modify our model so that these values of Z have no effect

## this is an indicator variable for when we have measured positions
CHI_observed = zeros([Nt])
CHI_observed[::10] = 1.
```

```python
Nsteps = 50
N_burn_in = Nsteps
Nchains = 5

Sigma_obs = sigma_obs*tf.ones([1, 1, 2], dtype=tf.float64)
Sigma = sigma_subsample*tf.ones([1, 1, 2], dtype=tf.float64)
Sigma_x0 = 100*tf.ones([1, 2], dtype=tf.float64)
def log_target_density(Xfull):
    ## X : [parallel chains, time, space]
    X0 = Xfull[:, 0, :]
    X = Xfull[:, 1:, :] ## same shape as `Z`
    ## observation likelihood: density for Z given X
    G = tfd.MultivariateNormalDiag(
        loc=X,
        scale_diag=Sigma_obs)
    logG = tf.reduce_sum(
        CHI_observed[None, :]*G.unnormalized_log_prob(Zpadded),
        axis=1)
    ## motion model: density for X
    F = tfd.MultivariateNormalDiag(
        loc=Xfull[:, :-1, :],
        scale_diag=Sigma)
    logF = tf.reduce_sum(
        F.unnormalized_log_prob(X),
        axis=1)
    ## initial position prior: density for X0
    F0 = tfd.MultivariateNormalDiag(
        loc=tf.zeros([1, 2], dtype=tf.float64),
        scale_diag=Sigma_x0)
    logF0 = F0.unnormalized_log_prob(X0)
    return logG + logF + logF0

@tf.function()
def track_sampler(Xinit):
    ## We are sampling a distribution over 200 dimensions

    kernel = tfp.mcmc.NoUTurnSampler(
        target_log_prob_fn=log_target_density,
        max_tree_depth=8,
        max_energy_diff=1000,
        step_size=tf.cast(0.1, tf.float64))

    # kernel = tfp.mcmc.HamiltonianMonteCarlo(
    #     target_log_prob_fn=log_target_density,
    #     num_leapfrog_steps=10,
    #     step_size=tf.cast(0.1, tf.float64))

    adaptive_kernel = tfp.mcmc.DualAveragingStepSizeAdaptation(
        inner_kernel=kernel,
        num_adaptation_steps=int(0.8 * N_burn_in),
        target_accept_prob=tf.cast(0.7, tf.float64))

    Xchain = tfp.mcmc.sample_chain(
        num_results=Nsteps,
        num_burnin_steps=N_burn_in,
        current_state=Xinit,
        #trace_fn=trace_everything,
        trace_fn=None,
        kernel=adaptive_kernel,
        parallel_iterations=4)
    return Xchain

Xinit = zeros([Nchains, Nt+1, 2])
Xinit[:, 0, :] = normal(0, 25., [Nchains, 2])
Xinit[:, 1:, :] = Zpadded + normal(0, 5., [Nchains, Nt, 2])
Xchain = track_sampler(Xinit)
Xchain = array(Xchain)
```

```python
mean(tfp.mcmc.effective_sample_size(Xchain))
```

    46.864604441106906

## This plot shows the MCMC result as a time series of marginal distributions for the position at time $t$

```python
xmin = Xchain.min()
xmax = Xchain.max()
@interact(t=(0, Nt, 1))
def pfun(t=0):
    fig = figure(1, [7, 7])
    plot(Xchain[:, :, t, 0], Xchain[:, :, t, 1], 'r.', ms=2)
    if t > 0 and CHI_observed[t-1] == 1:
        plot(Zpadded[t-1, 0], Zpadded[t-1, 1], 'ko', mfc='none', ms=25)
    xlim(xmin, xmax)
    ylim(xmin, xmax)
    xlabel(r'$x$', fontsize=24)
    ylabel(r'$y$', fontsize=24)
```

    interactive(children=(IntSlider(value=0, description='t'), Output()), _dom_classes=('widget-interact',))

## This shows the MCMC result as a collection of sample trajectories
Each parallel chain (at a given step) represents an independent path $X_{0:T}$. Here each is shown as a different color.

```python
xmin = Xchain.min()
xmax = Xchain.max()
@interact(t=(0, Nt, 1))
def pfun(t=0):
    fig = figure(1, [7, 7])
    for i in arange(5):
        plot(Xchain[-1, i, t, 0], Xchain[-1, i, t, 1], '.', ms=10)
    if t > 0 and CHI_observed[t-1] == 1:
        plot(Zpadded[t-1, 0], Zpadded[t-1, 1], 'ko', mfc='none', ms=25)
    xlim(xmin, xmax)
    ylim(xmin, xmax)
    xlabel(r'$x$', fontsize=24)
    ylabel(r'$y$', fontsize=24)
```

    interactive(children=(IntSlider(value=0, description='t'), Output()), _dom_classes=('widget-interact',))

```python

```


Make sure you remove `raise NotImplementedError()` and fill in any place that says `# YOUR CODE HERE`, as well as your `NAME`, `ID`, and `LAB_SECTION` below:

```python
NAME = "FAHMID BIN KIBRIA"
ID = "19201063"
LAB_SECTION = "02"
```

---

# Fixed Point Iteration

### Fixed point:
A number $\xi$ is called a **fixed point** to function $g(x)$ if $g(\xi) = \xi$. Using fixed points are a nice strategy to find roots of an equation. In this method if we are trying to find a root of $f(x) = 0$, we try to write the function in the form, $x = g(x)$. That is,
$$
f(x) = x - g(x) = 0
$$

So, if $\xi$ is a fixed point of $g(x)$ it would also be a root of $f(x)=0$, because,
$$
f(\xi) = \xi - g(\xi) = \xi - \xi = 0
$$

We can find a suitable $g(x)$ in any number of ways. Not all of them would converge, whereas, Some would converge very fast. For example, consider $Eq. 6.1$.

\begin{align}
&         &    f(x) &=x^5 + 2.5x^4 - 2x^3 -6x^2 + x + 2 \\
&\implies &x - g(x) &=x^5 + 2.5x^4 - 2x^3 -6x^2 + x + 2 \\
&\implies &    g(x) &=-x^5 - 2.5x^4 + 2x^3 + 6x^2 - 2 \tag{6.2}\\
\end{align}

again,
$$
f(x) = x^5 + 2.5x^4 - 2x^3 -6x^2 + x + 2 = 0\\
$$
\begin{align}
&\implies &6x^2 &= x^5 + 2.5x^4 - 2x^3 + x + 2 \\
&\implies &x^2  &= \frac{1}{6}(x^5 + 2.5x^4 - 2x^3 + x + 2)\\
&\implies &x    &= \sqrt{\frac{1}{6}(x^5 + 2.5x^4 - 2x^3 + x + 2)}\\
&\implies &g(x) &= \sqrt{\frac{1}{6}(x^5 + 2.5x^4 - 2x^3 + x + 2)}\tag{6.3}\\
\end{align}

Similarly,
\begin{align}
&         &2.5x^4 &= -x^5 + 2x^3 + 6x^2 - x - 2 \\
&\implies &x^4  &= \frac{1}{2.5}(-x^5 + 2x^3 + 6x^2 - x - 2)\\
&\implies &x    &= \sqrt[\leftroot{-1}\uproot{2}\scriptstyle 4]{\frac{1}{2.5}(-x^5 + 2x^3 + 6x^2 - x - 2)}\\
&\implies &g(x) &= \sqrt[\leftroot{-1}\uproot{2}\scriptstyle 4]{\frac{1}{2.5}(-x^5 + 2x^3 + 6x^2 - x - 2)}\tag{6.4}\\
\end{align}

### B. Complete the code below
For this example we will use a couple of $g(x)$ function to find out which one converges faster.

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from numpy.polynomial import Polynomial

f = Polynomial([2.0, 1.0, -6.0, -2.0, 2.5, 1.0])
g1 = Polynomial([-2.0, 0.0, 6.0, 2.0, -2.5, -1.0])

def g2(x):
    p = Polynomial([2.0, 1.0, 0.0, -2.0, 2.5, 1.0])
    return np.sqrt(p(x)/6)

def g3(x):
    p = Polynomial([-2.0, -1.0, 6.0, 2.0, 0.0, -1.0])
    return np.power(p(x)/2.5, 1.0/4.0)

a1 = 0.8
g1_a = []

a2 = 0.8
g2_a = []

a3 = 0.8
g3_a = []

# YOUR CODE HERE
val = True
error = 0.1E-6
while val:
    a1 = g1(a1)
    g1_a.append(a1)
    a2 = g2(a2)
    g2_a.append(a2)
    a3 = g3(a3)
    g3_a.append(a3)
    val = abs(f(a1)) > error or abs(f(a2)) > error or abs(f(a3)) > error
```

```python
xs = np.linspace(-2.5, 1.6, 100)
ys = f(xs)
dictionary = {
    'x': xs,
    'y': ys
}
plt.axhline(y=0, color='k')
plt.plot(xs, f(xs), label='f(x)')
plt.plot(xs, g1(xs), label='g1(x)')
plt.plot(xs, g2(xs), label='g2(x)')
plt.plot(xs, g3(xs), label='g3(x)')
plt.legend()

if len(g1_a) > 0:
    root = np.array([g1_a[len(g1_a)-1], g2_a[len(g2_a)-1], g3_a[len(g3_a)-1]])
    plt.plot(root, f(root), 'ro')

print(pd.DataFrame({'g1(x)':g1_a, 'g2(x)':g2_a, 'g3(x))':g3_a,}))

# Test case:
np.testing.assert_array_almost_equal(root, [-2., 0.67242436, 1.33033625])
```

```python

```