# Pyomo - Stock problem v2

Pyomo installation: see http://www.pyomo.org/installation

```
pip install pyomo
```

```python
%matplotlib inline

import matplotlib.pyplot as plt
```

```python
from pyomo.environ import *
```

* $s_t$ = the battery level at time $t$

$$
\begin{align}
    \min_{x_{inj}, x_{ext}, x_{grid}} & \quad \sum_t p_t x_{inj,t} + \sum_t p_t x_{grid,t} \quad\quad\quad\quad \text{(P1)} \\
    \text{s.t.}        & \quad s_0 = 0 \\
                       & \quad s_t = s_{t-1} + x_{inj,t} - x_{ext,t} \\
                       & \quad x_{inj,t} \leq s_{\max} - s_{t-1}   \quad\quad\quad \quad ~~ (L4) \\
                       & \quad x_{inj,t} \geq 0 \\
                       & \quad x_{ext,t} \leq s_{t-1}               \quad\quad \quad \quad \quad \quad \quad (L6) \\
                       & \quad x_{ext,t} \geq 0 \\
                       & \quad x_{ext,t} + x_{grid,t} = \text{needs}_{t}               \quad \quad \quad ~ (L8)
\end{align}
$$

- L4: can't inject more than the "free space" of the battery
- L6: can't extract more than what have been stored in the battery
- L8: fulfill the needs at time t

```python
# Cost of energy on the market

price = [10, 20, 10, 20]
needs = [ 0, 30, 20, 50]

stock_max = 100           # battery capacity
```

```python
T = list(range(len(price) + 1))       # num decision variables
```

```python
plt.plot(T[:-1], price, "o-")
plt.xlabel("Unit of time (t)")
plt.ylabel("Price of one unit of energy (p)")
plt.title("Price of energy on the market")
plt.show();
```

```python
plt.plot(T[:-1], needs, "o-")
plt.xlabel("Unit of time (t)")
plt.ylabel("Needs")
plt.title("Needs")
plt.show();
```

```python
model = ConcreteModel(name="Stock problem v1")

model.x_inj  = Var(T, within=NonNegativeReals)
model.x_ext  = Var(T, within=NonNegativeReals)
model.x_grid = Var(T, within=NonNegativeReals)
model.s = Var(T, within=NonNegativeReals)

def objective_fn(model):
    return sum(price[t-1] * model.x_inj[t] + price[t-1] * model.x_grid[t] for t in T if t != 0)

model.obj = Objective(rule=objective_fn, sense=minimize)

########

model.constraint_s0 = Constraint(expr = model.s[0] == 0)

def constraint_stock_level(model, t):
    if t > 0:
        return model.s[t] == model.s[t-1] + model.x_inj[t] - model.x_ext[t]
    else:
        return Constraint.Skip

model.constraint_st = Constraint(T, rule=constraint_stock_level)

def constraint_x_inj_sup(model, t):
    if t > 0:
        return model.x_inj[t] <= stock_max - model.s[t-1]
    else:
        return Constraint.Skip

model.constraint_x_inj_sup = Constraint(T, rule=constraint_x_inj_sup)

def constraint_x_ext_sup(model, t):
    if t > 0:
        return model.x_ext[t] <= model.s[t-1]
    else:
        return Constraint.Skip

model.constraint_x_ext_sup = Constraint(T, rule=constraint_x_ext_sup)

def constraint_needs(model, t):
    if t > 0:
        return model.x_ext[t] + model.x_grid[t] == needs[t-1]
    else:
        return Constraint.Skip

model.constraint_needs = Constraint(T, rule=constraint_needs)

########

model.pprint()

# @tail:
print()
print("-" * 60)
print()

opt = SolverFactory('glpk')

results = opt.solve(model)    # solves and updates instance

model.display()

print()
print("Optimal solution (x_inj):  ", [value(model.x_inj[t]) for t in T if t != 0])
print("Optimal solution (x_ext):  ", [value(model.x_ext[t]) for t in T if t != 0])
print("Optimal solution (x_grid): ", [value(model.x_grid[t]) for t in T if t != 0])
print("Gain of the optimal solution: ", value(model.obj))
# @:tail
```


# Systems of Equations
Imagine you are at a casino, and you have a mixture of £10 and £25 chips. You know that you have a total of 16 chips, and you also know that the total value of chips you have is £250. Is this enough information to determine how many of each denomination of chip you have?

Well, we can express each of the facts that we have as an equation. The first equation deals with the total number of chips - we know that this is 16, and that it is the number of £10 chips (which we'll call ***x*** ) added to the number of £25 chips (***y***).

The second equation deals with the total value of the chips (£250), and we know that this is made up of ***x*** chips worth £10 and ***y*** chips worth £25.

Here are the equations

\begin{equation}x + y = 16 \end{equation}
\begin{equation}10x + 25y = 250 \end{equation}

Taken together, these equations form a *system of equations* that will enable us to determine how many of each chip denomination we have.

## Graphing Lines to Find the Intersection Point
One approach is to determine all possible values for x and y in each equation and plot them.

A collection of 16 chips could be made up of 16 £10 chips and no £25 chips, no £10 chips and 16 £25 chips, or any combination between these.

Similarly, a total of £250 could be made up of 25 £10 chips and no £25 chips, no £10 chips and 10 £25 chips, or a combination in between.

Let's plot each of these ranges of values as lines on a graph:

```python
%matplotlib inline
from matplotlib import pyplot as plt

# Get the extremes for number of chips
chipsAll10s = [16, 0]
chipsAll25s = [0, 16]

# Get the extremes for values
valueAll10s = [25,0]
valueAll25s = [0,10]

# Plot the lines
plt.plot(chipsAll10s,chipsAll25s, color='blue')
plt.plot(valueAll10s, valueAll25s, color="orange")
plt.xlabel('x (£10 chips)')
plt.ylabel('y (£25 chips)')
plt.grid()

plt.show()
```

Looking at the graph, you can see that there is only a single combination of £10 and £25 chips that is on both the line for all possible combinations of 16 chips and the line for all possible combinations of £250. The point where the line intersects is (10, 6); or put another way, there are ten £10 chips and six £25 chips.

### Solving a System of Equations with Elimination
You can also solve a system of equations mathematically. Let's take a look at our two equations:

\begin{equation}x + y = 16 \end{equation}
\begin{equation}10x + 25y = 250 \end{equation}

We can combine these equations to eliminate one of the variable terms and solve the resulting equation to find the value of one of the variables. Let's start by combining the equations and eliminating the x term.

We can combine the equations by adding them together, but first, we need to manipulate one of the equations so that adding them will eliminate the x term. The first equation includes the term ***x***, and the second includes the term ***10x***, so if we multiply the first equation by -10, the two x terms will cancel each other out. So here are the equations with the first one multiplied by -10:

\begin{equation}-10(x + y) = -10(16) \end{equation}
\begin{equation}10x + 25y = 250 \end{equation}

After we apply the multiplication to all of the terms in the first equation, the system of equations look like this:

\begin{equation}-10x + -10y = -160 \end{equation}
\begin{equation}10x + 25y = 250 \end{equation}

Now we can combine the equations by adding them. The ***-10x*** and ***10x*** cancel one another, leaving us with a single equation like this:

\begin{equation}15y = 90 \end{equation}

We can isolate ***y*** by dividing both sides by 15:

\begin{equation}y = \frac{90}{15} \end{equation}

So now we have a value for ***y***:

\begin{equation}y = 6 \end{equation}

So how does that help us? Well, now we have a value for ***y*** that satisfies both equations. We can simply use it in either of the equations to determine the value of ***x***. Let's use the first one:

\begin{equation}x + 6 = 16 \end{equation}

When we work through this equation, we get a value for ***x***:

\begin{equation}x = 10 \end{equation}

So now we've calculated values for ***x*** and ***y***, and we find, just as we did with the graphical intersection method, that there are ten £10 chips and six £25 chips.

You can run the following Python code to verify that the equations are both true with an ***x*** value of 10 and a ***y*** value of 6.

```python
x = 10
y = 6
print ((x + y == 16) & ((10*x) + (25*y) == 250))
```

Basics Of Algebra by Hiren/01-03-Systems of Equations.ipynb


# Introduction to probabilistic programming with PyMC3

```python
import os,sys
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import pymc3 as pm
import scipy.stats as st

%matplotlib inline
%precision 4
plt.style.use('bmh')
```

## Coin toss example

We are looking to determine if our coin is biased.  We can investigate the fairness of a coin in several ways.  One useful distribution is the Binomial.  It is parameterized by the number of trials ($n$) and the success probability in each trial ($p$).

```python
from IPython.display import Image
Image(filename='../binomial.png')
```

```python
## lets start with some data
n = 100
pcoin = 0.62 # actual value of p for coin
results = st.bernoulli(pcoin).rvs(n)
h = sum(results)
print("We observed %s heads out of %s"%(h,n))

## the expected distribution
p = 0.5
rv = st.binom(n,p)
mu = rv.mean()
sd = rv.std()
print("The expected distribution for a fair coin is mu=%s, sd=%s"%(mu,sd))

## p-value by simulation
nsamples = 100000
xs = np.random.binomial(n, p, nsamples)
print("Simulation p-value - %s"%(2*np.sum(xs >= h)/(xs.size + 0.0)))

## p-value by binomial test
print("Binomial test - %s"%st.binom_test(h, n, p))

## MLE
print("Maximum likelihood %s"%(np.sum(results)/float(len(results))))

## bootstrap
bs_samples = np.random.choice(results, (nsamples, len(results)), replace=True)
bs_ps = np.mean(bs_samples, axis=1)
bs_ps.sort()
print("Bootstrap CI: (%.4f, %.4f)" % (bs_ps[int(0.025*nsamples)], bs_ps[int(0.975*nsamples)]))
```

    We observed 59 heads out of 100
    The expected distribution for a fair coin is mu=50.0, sd=5.0
    Simulation p-value - 0.08668
    Binomial test - 0.0886260801141
    Maximum likelihood 0.59
    Bootstrap CI: (0.4900, 0.6900)

### Lets perform inference with PyMC3

```python
print("n = %s"%n)
print("h = %s"%h)
alpha = 2
beta = 2

niter = 1000
with pm.Model() as model: # context management
    # define priors
    p = pm.Beta('p', alpha=alpha, beta=beta)

    # define likelihood
    y = pm.Binomial('y', n=n, p=p, observed=h)

    # inference
    start = pm.find_MAP() # Use MAP estimate (optimization) as the initial state for MCMC
    step = pm.Metropolis() # Have a choice of samplers
    trace = pm.sample(niter, step, start, random_seed=123, progressbar=True)
```

    n = 100
    h = 59

    Applied logodds-transform to p and added transformed p_logodds_ to model.
    100%|██████████| 1000/1000 [00:00<00:00, 8636.83it/s]

```python
fig = plt.figure(figsize=(8,6))
ax = fig.add_subplot(111)

ax.hist(trace['p'], 15, histtype='step', normed=True, label='post');
x = np.linspace(0, 1, 100)
ax.plot(x, st.beta.pdf(x, alpha, beta), label='prior');
ax.legend(loc='best');
```

## Estimating the mean and standard deviation of a normal

One of the amazing things about probabilistic programming in general is that to evaluate a complex model it is just an extension of what we would do with simple models---we go about it using the same process.

$$
X \sim \mathcal{N}(\mu,\sigma^{2})
$$

```python
# generate observed data
N = 100
_mu = np.array([10])
_sigma = np.array([2])
y = np.random.normal(_mu, _sigma, N)

niter = 1000
with pm.Model() as model:
    # define priors
    mu = pm.Uniform('mu', lower=0, upper=100, shape=_mu.shape)
    sigma = pm.Uniform('sigma', lower=0, upper=10, shape=_sigma.shape)

    # define likelihood
    y_obs = pm.Normal('Y_obs', mu=mu, sd=sigma, observed=y)

    # inference
    start = pm.find_MAP()
    step = pm.Slice()
    trace = pm.sample(niter, step, start, random_seed=123, progressbar=True)
```

    Applied interval-transform to mu and added transformed mu_interval_ to model.
    Applied interval-transform to sigma and added transformed sigma_interval_ to model.
    100%|██████████| 1000/1000 [00:01<00:00, 918.42it/s]

```python
fig = plt.figure(figsize=(10,6))
ax1 = fig.add_subplot(1,2,1);
ax2 = fig.add_subplot(1,2,2);

ax1.hist(trace['mu'][-niter/2:,0], 25, histtype='step');
ax2.hist(trace['sigma'][-niter/2:,0], 25, histtype='step');
```

## Switchpoint analysis

This example comes from [Cam Davidson-Pilon's book](https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers)

We will be looking at count data---specifically at the frequency of text messages recieved over a period of time.  We will use the Poisson distribution to help use investigate this series of events.

```python
from IPython.display import Image
Image(filename='../poisson.png')
```

```python
fig = plt.figure(figsize=(12.5, 3.5))
ax = fig.add_subplot(111)
count_data = np.loadtxt("data/txtdata.csv")
n_count_data = len(count_data)
ax.bar(np.arange(n_count_data), count_data, color="#348ABD")
ax.set_xlabel("Time (days)")
ax.set_ylabel("count of text-msgs received")
ax.set_title("Did the user's texting habits change over time?")
ax.set_xlim(0, n_count_data);
```

These are time-course data.  Can we infer when a behavioral change occurred?

Lets denote the day with $i$ and text-message count by $C_i$,

$$ C_i \sim \text{Poisson}(\lambda) $$

   * Does the rate $\lambda$ change?
   * Is there a day (call it $\tau$) where the parameter $\lambda$ suddenly jumps to a higher value?
   * We are looking for a _switchpoint_ s.t.

$$
\lambda =
\begin{cases}
\lambda_1  & \text{if } t \lt \tau \cr
\lambda_2 & \text{if } t \ge \tau
\end{cases}
$$

If no sudden change occurred and indeed $\lambda_1 = \lambda_2$, then the $\lambda$s posterior distributions should look about equal.

We are interested in inferring the unknown $\lambda$s. We are using Bayesian inference so we need priors. The *exponential* distribution provides a continuous density function for positive numbers.

\begin{align}
&\lambda_1 \sim \text{Exp}( \alpha ) \\\
&\lambda_2 \sim \text{Exp}( \alpha )
\end{align}

$\alpha$ is a *hyper-parameter*. In literal terms, it is a parameter that influences other parameters.

$$\frac{1}{N}\sum_{i=0}^N \;C_i \approx E[\; \lambda \; |\; \alpha ] = \frac{1}{\alpha}$$

What about $\tau$? Because of the noisiness of the data, it's difficult to pick out a priori when $\tau$ might have occurred. Instead, we can assign a *uniform prior belief* to every possible day. This is equivalent to saying

\begin{align}
& \tau \sim \text{DiscreteUniform(1,70) }\\\\
& \Rightarrow P( \tau = k ) = \frac{1}{70}
\end{align}

So what does this look like in PyMC3?

```python
import pymc3 as pm
import theano.tensor as tt

## assign lambdas and tau to stochastic variables
with pm.Model() as model:
    alpha = 1.0/count_data.mean()  # Recall count_data is the
                                   # variable that holds our txt counts
    lambda_1 = pm.Exponential("lambda_1", alpha)
    lambda_2 = pm.Exponential("lambda_2", alpha)
    tau = pm.DiscreteUniform("tau", lower=0, upper=n_count_data)

## create a combined function for lambda (it is still a RV)
with model:
    idx = np.arange(n_count_data) # Index
    lambda_ = pm.math.switch(tau >= idx, lambda_1, lambda_2)

## combine the data with our proposed data generation scheme
with model:
    observation = pm.Poisson("obs", lambda_, observed=count_data)

## inference
with model:
    step = pm.Metropolis()
    trace = pm.sample(10000, tune=5000,step=step)
```

    Applied log-transform to lambda_1 and added transformed lambda_1_log_ to model.
    Applied log-transform to lambda_2 and added transformed lambda_2_log_ to model.
    100%|██████████| 10000/10000 [00:03<00:00, 3171.19it/s]

```python
## get the variables we want to plot from our trace
lambda_1_samples = trace['lambda_1']
lambda_2_samples = trace['lambda_2']
tau_samples = trace['tau']

# draw histogram of the samples:
fig = plt.figure(figsize=(12.5,10))
ax1 = fig.add_subplot(311)
ax2 = fig.add_subplot(312)
ax3 = fig.add_subplot(313)

for ax in [ax1,ax2]:
    ax.set_autoscaley_on(False)

## axis 1
ax1.hist(lambda_1_samples, histtype='stepfilled', bins=30, alpha=0.85,
         label="posterior of $\lambda_1$", color="#A60628", normed=True)
ax1.legend(loc="upper left")
ax1.set_title(r"""Posterior distributions of the variables
    $\lambda_1,\;\lambda_2,\;\tau$""")
ax1.set_xlim([15, 30])
ax1.set_xlabel("$\lambda_1$ value")

## axis 2
ax2.hist(lambda_2_samples, histtype='stepfilled', bins=30, alpha=0.85,
         label="posterior of $\lambda_2$", color="#7A68A6", normed=True)
ax2.legend(loc="upper left")
ax2.set_xlim([15, 30])
ax2.set_xlabel("$\lambda_2$ value")

## axis 3
w = 1.0 / tau_samples.shape[0] * np.ones_like(tau_samples)
ax3.hist(tau_samples, bins=n_count_data, alpha=1,
         label=r"posterior of $\tau$",
         color="#467821", weights=w, rwidth=2.)
ax3.set_xticks(np.arange(n_count_data))

ax3.legend(loc="upper left")
ax3.set_ylim([0, .75])
ax3.set_xlim([35, len(count_data)-20])
ax3.set_xlabel(r"$\tau$ (in days)")
ax3.set_ylabel("probability");
plt.subplots_adjust(hspace=0.4)
```

### How do we plot the expection distribution of this model?

```python
fig = plt.figure(figsize=(12.5,5))
ax = fig.add_subplot(111)

N = tau_samples.shape[0]
expected_texts_per_day = np.zeros(n_count_data)
for day in range(0, n_count_data):
    ix = day < tau_samples
    expected_texts_per_day[day] = (lambda_1_samples[ix].sum()
                                   + lambda_2_samples[~ix].sum()) / N

ax.plot(range(n_count_data), expected_texts_per_day, lw=4, color="#E24A33",
         label="expected number of text-messages received")
ax.set_xlim(0, n_count_data)
ax.set_xlabel("Day")
ax.set_ylabel("Expected # text-messages")
ax.set_title("Expected number of text-messages received")
ax.set_ylim(0, 60)
ax.bar(np.arange(len(count_data)), count_data, color="#348ABD", alpha=0.65,label="observed texts per day")

ax.legend(loc="upper left");
```

A distribution allows us to see the uncertainty in our estimates.  The two distribution for $\lambda$ are distinct.  Near day 45, there was a 50% chance that the user's behavior changed.

> In fact, the 45th day corresponds to Christmas, and I moved away to Toronto the next month, leaving a girlfriend behind

      -Cam Davidson Pilon

```python

```


## How do distributions transform under a change of variables ?

Kyle Cranmer, March 2016

```python
%pylab inline --no-import-all
```

    Populating the interactive namespace from numpy and matplotlib

We are interested in understanding how distributions transofrm under a change of variables.
Let's start with a simple example. Think of a spinner like on a game of twister.

We flick the spinner and it stops. Let's call the angle of the pointer $x$. It seems a safe assumption that the distribution of $x$ is uniform between $[0,2\pi)$... so $p_x(x) = 1/\sqrt{2\pi}$

Now let's say that we change variables to $y=\cos(x)$ (sorry if the names are confusing here, don't think about x- and y-coordinates, these are just names for generic variables).  The question is this:
** what is the distribution of y?**  Let's call it $p_y(y)$

Well it's easy to do with a simulation, let's try it out

```python
# generate samples for x, evaluate y=cos(x)
n_samples = 100000
x = np.random.uniform(0,2*np.pi,n_samples)
y = np.cos(x)
```

```python
# make a histogram of x
n_bins = 50
counts, bins, patches = plt.hist(x, bins=50, normed=True, alpha=0.3)
plt.plot([0,2*np.pi], (1./2/np.pi, 1./2/np.pi), lw=2, c='r')
plt.xlim(0,2*np.pi)
plt.xlabel('x')
plt.ylabel('$p_x(x)$')
```

Ok, now let's make a histogram for $y=\cos(x)$

```python
counts, y_bins, patches = plt.hist(y, bins=50, normed=True, alpha=0.3)
plt.xlabel('y')
plt.ylabel('$p_y(y)$')
```

It's not uniform! Why is that?  Let's look at the $x-y$ relationship

```python
# make a scatter of x,y
plt.scatter(x[:300],y[:300]) #just the first 300 points

xtest = .2
plt.plot((-1,xtest),(np.cos(xtest),np.cos(xtest)), c='r')
plt.plot((xtest,xtest),(-1.5,np.cos(xtest)), c='r')
xtest = xtest+.1
plt.plot((-1,xtest),(np.cos(xtest),np.cos(xtest)), c='r')
plt.plot((xtest,xtest),(-1.5,np.cos(xtest)), c='r')

xtest = 2*np.pi-xtest
plt.plot((-1,xtest),(np.cos(xtest),np.cos(xtest)), c='g')
plt.plot((xtest,xtest),(-1.5,np.cos(xtest)), c='g')
xtest = xtest+.1
plt.plot((-1,xtest),(np.cos(xtest),np.cos(xtest)), c='g')
plt.plot((xtest,xtest),(-1.5,np.cos(xtest)), c='g')

xtest = np.pi/2
plt.plot((-1,xtest),(np.cos(xtest),np.cos(xtest)), c='r')
plt.plot((xtest,xtest),(-1.5,np.cos(xtest)), c='r')
xtest = xtest+.1
plt.plot((-1,xtest),(np.cos(xtest),np.cos(xtest)), c='r')
plt.plot((xtest,xtest),(-1.5,np.cos(xtest)), c='r')

xtest = 2*np.pi-xtest
plt.plot((-1,xtest),(np.cos(xtest),np.cos(xtest)), c='g')
plt.plot((xtest,xtest),(-1.5,np.cos(xtest)), c='g')
xtest = xtest+.1
plt.plot((-1,xtest),(np.cos(xtest),np.cos(xtest)), c='g')
plt.plot((xtest,xtest),(-1.5,np.cos(xtest)), c='g')

plt.ylim(-1.5,1.5)
plt.xlim(-1,7)
```

The two sets of vertical lines are both separated by $0.1$. The probability  $P(a < x < b)$ must equal the probability of $P( cos(b) < y < cos(a) )$. In this example there are two different values of $x$ that give the same $y$ (see green and red lines), so we need to take that into account. For now, let's just focus on the first part of the curve with $x<\pi$.

So we can write (this is the important equation):

\begin{equation}
\int_a^b p_x(x) dx = \int_{y_b}^{y_a} p_y(y) dy
\end{equation}
where $y_a = \cos(a)$ and $y_b = \cos(b)$.

and we can re-write the integral on the right by using a change of variables (pure calculus)

\begin{equation}
\int_a^b p_x(x) dx = \int_{y_b}^{y_a} p_y(y) dy = \int_a^b p_y(y(x))  \left| \frac{dy}{dx}\right| dx
\end{equation}

notice that the limits of integration and integration variable are the same for the left and right sides of the equation, so the integrands must be the same too. Therefore:

\begin{equation}
p_x(x) = p_y(y)  \left| \frac{dy}{dx}\right|
\end{equation}
and equivalently
\begin{equation}
p_y(y) = p_x(x) \,/ \,\left| \, {dy}/{dx}\, \right |
\end{equation}

The factor $\left|\frac{dy}{dx} \right|$ is called a Jacobian. When it is large it is stretching the probability in $x$ over a large range of $y$, so it makes sense that it is in the denominator.

```python
plt.plot((0.,1), (0,.3))
plt.plot((0.,1), (0,0), lw=2)
plt.plot((1.,1), (0,.3))
plt.ylim(-.1,.4)
plt.xlim(-.1,1.6)
plt.text(0.5,0.2, '1', color='b')
plt.text(0.2,0.03, 'x', color='black')
plt.text(0.5,-0.05, 'y=cos(x)', color='g')
plt.text(1.02,0.1, '$\sin(x)=\sqrt{1-y^2}$', color='r')
```

In our case:
\begin{equation}
\left|\frac{dy}{dx} \right| =  \sin(x)
\end{equation}

Looking at the right-triangle above you can see $\sin(x)=\sqrt{1-y^2}$ and finally there will be an extra factor of 2 for $p_y(y)$ to take into account $x>\pi$. So we arrive at
\begin{equation}
p_y(y) = 2 \times \frac{1}{2 \pi} \frac{1}{\sin(x)} = \frac{1}{\pi} \frac{1}{\sin(\arccos(y))} = \frac{1}{\pi} \frac{1}{\sqrt{1-y^2}}
\end{equation}

  Notice that when $y=\pm 1$ the pdf is diverging. This is called a [caustic](http://www.phikwadraat.nl/huygens_cusp_of_tea/) and you see them in your coffee and rainbows!

|   |   |
|---|---|
|   |   |

**Let's check our prediction**

```python
counts, y_bins, patches = plt.hist(y, bins=50, normed=True, alpha=0.3)
pdf_y = (1./np.pi)/np.sqrt(1.-y_bins**2)
plt.plot(y_bins, pdf_y, c='r', lw=2)
plt.ylim(0,5)
plt.xlabel('y')
plt.ylabel('$p_y(y)$')
```

Perfect!

## A trick using the cumulative distribution function (cdf) to generate random numbers

Let's consider a different variable transformation now -- it is a special one that we can use to our advantage.
\begin{equation}
y(x) = \textrm{cdf}(x) = \int_{-\infty}^x p_x(x') dx'
\end{equation}

Here's a plot of a distribution and cdf for a Gaussian.

(NOte: the axes are different for the pdf and the cdf http://matplotlib.org/examples/api/two_scales.html

```python
from scipy.stats import norm
```

```python
x_for_plot = np.linspace(-3,3, 30)
fig, ax1 = plt.subplots()

ax1.plot(x_for_plot, norm.pdf(x_for_plot), c='b')
ax1.set_ylabel('p(x)', color='b')
for tl in ax1.get_yticklabels():
    tl.set_color('b')

ax2 = ax1.twinx()
ax2.plot(x_for_plot, norm.cdf(x_for_plot), c='r')
ax2.set_ylabel('cdf(x)', color='r')
for tl in ax2.get_yticklabels():
    tl.set_color('r')
```

Ok, so let's use our result about how distributions transform under a change of variables to predict the distribution of $y=cdf(x)$. We need to calculate

\begin{equation}
\frac{dy}{dx} = \frac{d}{dx} \int_{-\infty}^x p_x(x') dx'
\end{equation}

Just like particles and anti-particles, when derivatives meet anti-derivatives they annihilate. So $\frac{dy}{dx} = p_x(x)$, which shouldn't be a surprise.. the slope of the cdf is the pdf.

So putting these together we find the distribution for $y$ is:

\begin{equation}
p_y(y) = p_x(x) \, / \, \frac{dy}{dx} = p_x(x) /p_x(x) = 1
\end{equation}

So it's just a uniform distribution from $[0,1]$, which is perfect for random numbers.

We can turn this around and generate a uniformly random number between $[0,1]$, take the inverse of the cdf and we should have the distribution we want for $x$.

Let's try it for a Gaussian. The inverse of the cdf for a Gaussian is called [ppf](http://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.stats.norm.html)

```python
norm.ppf.__doc__
```

    '\n        Percent point function (inverse of `cdf`) at q of the given RV.\n\n        Parameters\n        ----------\n        q : array_like\n            lower tail probability\n        arg1, arg2, arg3,... : array_like\n            The shape parameter(s) for the distribution (see docstring of the\n            instance object for more information)\n        loc : array_like, optional\n            location parameter (default=0)\n        scale : array_like, optional\n            scale parameter (default=1)\n\n        Returns\n        -------\n        x : array_like\n            quantile corresponding to the lower tail probability q.\n\n        '

```python
#check it out
norm.cdf(0), norm.ppf(0.5)
```

    (0.5, 0.0)

Ok, let's use CDF trick to generate Normally-distributed (aka Gaussian-distributed) random numbers

```python
rand_cdf = np.random.uniform(0,1,10000)
rand_norm = norm.ppf(rand_cdf)
```

```python
_ = plt.hist(rand_norm, bins=30, normed=True, alpha=0.3)
plt.xlabel('x')
```

**Pros**: The great thing about this technique is it is very efficient. You only generate one random number per random $x$.

**Cons**: the downside is you need to know how to compute the inverse cdf for $p_x(x)$ and that can be difficult. It works for a distribution like a Gaussian, but for some random distribution this might be even more computationally expensive than the accept/reject approach. This approach also doesn't really work if your distribution is for more than one variable.

## Going full circle

Ok, let's try it for our distribution of $y=\cos(x)$ above.  We found

\begin{equation}
p_y(y) = \frac{1}{\pi} \frac{1}{\sqrt{1-y^2}}
\end{equation}

So the CDF is  (see Wolfram alpha for [integral](http://www.wolframalpha.com/input/?i=integrate%5B1%2Fsqrt%5B1-x%5E2%5D%2FPi%5D) )
\begin{equation}
cdf(y') = \int_{-1}^{y'} \frac{1}{\pi} \frac{1}{\sqrt{1-y^2}} = \frac{1}{\pi}\arcsin(y') + C
\end{equation}
and we know that for $y=-1$ the CDF must be 0, so the constant is $1/2$ and by looking at the plot or remembering some trig you know that it's also $cdf(y') = (1/\pi) \arccos(y')$.

So to apply the trick, we need to generate uniformly random variables $z$ between 0 and 1, and then take the inverse of the cdf to get $y$. Ok, so what would that be:
\begin{equation}
y = \textrm{cdf}^{-1}(z) = \cos(\pi z)
\end{equation}

**Of course!** that's how we started in the first place, we started with a uniform $x$ in $[0,2\pi]$ and then defined $y=\cos(x)$. So we just worked backwards to get where we started.  The only difference here is that we only evaluate the first half: $\cos(x < \pi)$

```python

```


# Normal and tangential coordinates

In the normal and tangential coordinate system, the (vector) equation of motion

$$
\sum \mathbf{F} = m \mathbf{a}
$$

decomposes into the three scalar equations for the tangential ($t$),
normal ($n$), and binormal ($b$) directons:

$$
\begin{align}
\sum F_t &= m a_t = m \dot{v} = m \ddot{s} \\
\sum F_n &= m a_n = m \frac{v^2}{\rho} \\
\sum F_b &= m a_b = 0 \;,
\end{align}
$$

where $F_i$ is a force in the $i$ direction, $m$ is the particle mass,
$v$ is the velocity, and $\rho$ is the radius of curvature.

The tangential acceleration $a_t$ is positive or negative in the direction of motion,
the normal acceleration $a_n$ is **always** positive in the normal direction,
and the binormal acceleration $a_b$ is **always** zero, because motion lies in the plane
formed by the normal and tangential directions.

## Example: race car at banking angle (no friction)

A Formula 1 race car of mass $m$ = 740 kg is traveling on a track at
constant velocity $v$ = 60 m/s, where the radius of curvature is $\rho$ = 400 m.
What is the banking angle $\theta$ necessary for the car to avoid sliding as it
goes around this curve?

First, let's draw a free-body diagram for the car:

Now, write the three scalar equations of motion for the car.
The equation in the tangential direction does not really tell us much,
since the car is moving at a constant speed in its direction of motion:

$$
\sum F_t = m a_t = m \dot{v} = 0 \;.
$$

In the normal direction, the only force is the normal component of the resultant force:

$$
\begin{align}
\sum F_n &= m a_n \\
N_C \sin \theta &= m \frac{v^2}{\rho}
\end{align}
$$

and in the binormal direction, we have both the component of the resultant force and also the car's weight, but the binormal acceleration is zero:

$$
\begin{align}
\sum F_b &= m a_b = 0 \\
N_C \cos \theta - m g &= 0 \\
\rightarrow N_C \cos \theta &= m g
\end{align}
$$

If we divide the latter equation from the former, we can solve for the banking angle:

$$
\begin{align}
\frac{N_C \sin \theta}{N_C \cos \theta} &= \frac{m v^2 / \rho}{mg} \\
\tan \theta &= \frac{v^2}{\rho g} \\
\therefore \theta &= \tan^{-1} \left(\frac{v^2}{\rho g} \right)
\end{align}
$$

For the parameters given:

```python
import numpy as np

mass = 740 # kg
velocity = 60 # m/s
rho = 400 # m
g = 9.81 # m/s^2

theta = np.arctan(velocity**2 / (rho * g))
print(f'theta ={theta * 180/np.pi: .2f}°')
```

    theta = 42.53°

## Example: race car at banking angle (with friction)

Now, consider the same situation, but account for the effect of friction,
which will counter the car's motion in the outward direction.
What is the new banking angle needed to avoid the car sliding in this case?
Assume the coefficient of static friction is $\mu_s = 0.2$.

```python
mu = 0.2
```

In this case, we now have to account for components of the friction force in the normal and binormal directions, where the friction force is

$$
f = \mu_s N_C \;.
$$

In the normal direction, we have

$$
\begin{align}
\sum F_n &= m a_n \\
N_C \sin \theta + f \cos \theta &= m \frac{v^2}{\rho}
\end{align}
$$

and in the binormal direction

$$
\begin{align}
\sum F_b &= m a_b = 0 \\
N_C \cos \theta - f \sin \theta - m g &= 0 \\
\rightarrow N_C \cos \theta - f \sin \theta &= m g
\end{align}
$$

Combining the two equations (again, by dividing the first by the second) and recalling that $f = \mu_s N_c$:

$$
\begin{align}
\frac{N_C \sin \theta + f \cos \theta}{N_C \cos \theta - f \sin \theta} &= \frac{m v^2 / \rho}{m g} \\
\frac{\sin \theta + \mu_s \cos \theta}{\cos \theta - \mu_s \sin \theta} &= \frac{v^2}{\rho g} \;.
\end{align}
$$

This is our equation to find the banking angle, but unfortunately it has no closed-form solution. So, how do we find $\theta$? Using a numerical method!

### Method 1: manual iteration

We could first attack this problem by manually guessing and checking different values of $\theta$, until the left-hand side of the equation equals the right-hand side.
For example, trying different values from 20° to 40°:

```python
# need to convert to radians
vals = np.arange(20, 41, 2) * np.pi / 180

print('Theta   LHS    RHS')
for theta in vals:
    lhs = (np.sin(theta) + mu*np.cos(theta)) / (np.cos(theta) - mu*np.sin(theta))
    rhs = velocity**2 / (rho*g)
    print(f'{theta*180/np.pi: 4.1f}° {lhs: 5.3f} {rhs: 5.3f}')
```

    Theta   LHS    RHS
     20.0°  0.608  0.917
     22.0°  0.657  0.917
     24.0°  0.708  0.917
     26.0°  0.762  0.917
     28.0°  0.819  0.917
     30.0°  0.879  0.917
     32.0°  0.943  0.917
     34.0°  1.011  0.917
     36.0°  1.084  0.917
     38.0°  1.163  0.917
     40.0°  1.249  0.917

So, clearly the correct value is between 30° and 32°. A bit more manual iteration shows that the correct angle is just about 31.2°:

```python
theta = 31.2 * np.pi / 180

lhs = (np.sin(theta) + mu*np.cos(theta)) / (np.cos(theta) - mu*np.sin(theta))
rhs = velocity**2 / (rho*g)
print(lhs, rhs)
```

    0.9166501382856722 0.9174311926605505

### Method 2: `root_scalar`

Manually solving like this would be quite tedious; fortunately, there are numerical methods for solving scalar equations like this. We refer to this at **root finding**, since to solve we
formulate the equation like $F(x) = 0$, and find the value of the unknown variable that makes the function zero (i.e., the root).

In this case, we make the equation into the form

$$
F(\theta) = \frac{\sin \theta + \mu_s \cos \theta}{\cos \theta - \mu_s \sin \theta} - \frac{v^2}{\rho g} = 0 \;,
$$

Then, to solve, we can use the `root_scalar` function provided in the `scipy.optimize` module,
which needs us to provide it with a function that returns $F(\theta)$ for candidate values of $\theta$ (with the goal of finding the one that makes $F(\theta)=0$), along with a few guess values:

```python
from scipy.optimize import root_scalar

def f(theta):
    '''This function evaluates the equation for finding theta.
    '''
    return (
        (np.sin(theta) + mu*np.cos(theta)) / (np.cos(theta) - mu*np.sin(theta)) -
        velocity**2 / (rho*g)
        )

sol = root_scalar(f, x0=(20*np.pi/180), x1=(40*np.pi/180))
print(f'theta ={sol.root * 180/np.pi: .2f}°')
```

    theta = 31.22°

```python

```


# Technical Details

As with all packages, numerous technical details that are abstracted away from the user. Now to ensure a clean interface, this abstraction is entirely necessary. However, it can sometimes be confusing when navigating a package's source code to pin down what's going on when there's so many _under the hood_ operations taking place. In this notebook, I'll aim to shed some light on all of the tricks that we do in GPJax to help elucidate the code to anyone wishing to extend GPJax for their own uses.

## Parameter Transformations

### Motivations

Many parameters in a Gaussian process are what we call a _constrained parameter_. By this, we mean that the parameter's value is only defined on a subset of $\mathbb{R}$. One example of this is the lengthscale parameter in any of the stationary kernels. It would not make sense to have a negative lengthscale, and as such the parameter's value is constrained to exist only on the positive real line.

Whilst mathematically correct, constrained parameters can become a pain when optimising as many optimisers are designed to operate on an unconstrained space. Further, it can often be computationally inefficient to restrict the search space of an optimiser. For these reasons, we instead transform the constrained parameter to exist in an unconstrained space. The optimisation is then done on this unconstrained parameter before we transform it back when we need to evaluate its value.

Only bijective transformations are valid as we cannot afford to lose our original parameter value when transforming. As such, we have to be careful about which transformations we use. Some common choices include the log-exponential bijection and the softplus transform. We, by default, opt for the softplus transformation in GPJax as it is less prone to numerical overflow in comparison to log-exp transformations.

### Implementation

When it comes to implementations, we attach the transformation directly to the `Parameter` class. It is an optional argument that one can specify when instantiating their parameter. To see this, consider the following example

```python
from gpjax.parameters import Parameter
from gpjax.transforms import Softplus
import jax.numpy as jnp

x = Parameter(jnp.array(1.0), transform=Softplus())
```

    WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)

Now we know that the softplus transformation operation on an input $x \in \mathbb{R}_{>0}$ can be written as
$$\alpha(x) = \log(\exp(x)-1)$$
where $\alpha(x) \in \mathbb{R}$. In this instance, it can be seen that $\alpha(1)=0.54$. Now this unconstrained value is stored within the parameter's `value` property

```python
print(x.value)
```

    0.541324854612918

whilst the original constrained value can be computed by accesing the parameter's `untransform` property

```python
print(x.untransform)
```

    1.0

### Custom transformation

Should you wish to define your own custom transformation, then this can easily be done by extending the `Transform` class within `gpjax.transforms` and defining a forward transformation and a backward transformation.

```python
class Transform:
    def __init__(self, name="Transformation"):
        self.name = name

    @staticmethod
    def forward(x: jnp.ndarray) -> jnp.ndarray:
        raise NotImplementedError

    @staticmethod
    def backward(x: jnp.ndarray) -> jnp.ndarray:
        raise NotImplementedError
```

The `forward` method is the transformation that maps from a constrained space to an unconstrained space, whilst the `backward` method is the transformation that reverses this. A nice example of this can be seen for the earlier used softplus transformation

```python
from jax.nn import softplus

class Softplus(Transform):
    def __init__(self):
        super().__init__(name='Softplus')

    @staticmethod
    def forward(x: jnp.ndarray) -> jnp.ndarray:
        return jnp.log(jnp.exp(x) - 1.)

    @staticmethod
    def backward(x: jnp.ndarray) -> jnp.ndarray:
        return softplus(x)
```

## Prior distributions

### Motivations

Often when we use Gaussian processes, we do so as they facilitate incorporation of prior information into the model. Implicitly, by the very use of a Gaussian process, we are incorporating our prior information around the functional behaviour of the latent function that we are seeking to recover. However, we can take this one step further by placing priors on the hyperparameters of the Gaussian process. Going into the details of which priors are recommended and how to go about selecting them goes beyond the scope of this article, but it suffices to say that doing so can greatly enhance the utility of a Gaussian process.

At least in my own experience, when priors are placed on the hyperparameters of a Gaussian process they are specified with respect to the constrained parameter value. As an example of this, consider the lengthscale parameter $\ell \in \mathbb{R}_{>0}$. When specifying a prior distribution $p_{0}(\ell)$, I would typically select a distribution that has support on the positive real line, such as the Gamma distribution. An opposing approach would be to transform the parameter so that it is defined on the entire real line (as discussed in §1) and then specify a prior distribution such as a Gaussian that has unconstrained support. Deciding which of these two approaches to adopt in GPJax is somewhat a moot point to me, so I've opted for priors to be defined on the constrained parameter. That being said, I'd be more than open to altering this opinion is people felt strongly that priors should be defined on the unconstrained parameter value.

### Implementation

Regarding the implementational details of enabling prior specification, this is hopefully a more lucid concept upon code inspection. As with the earlier discussed parameter transformations, the notion of a prior distribution is acknowledged in the definition of a parameter. To exactly specify a prior distribution, one should simply call in the relevant distribution from TensorFlow probability's distributions module. For an example of this, consider the parameter `x` that was earlier defined.

```python
from tensorflow_probability.substrates.jax import distributions as tfd

x.prior = tfd.Gamma(concentration = 3., rate = 2.)
```

If we momentarily pause to consider the state of this parameter now, then we have a constrained parameter value with a corresponding prior distribution. When it comes to deriving our posterior distribution, then we know that it is proportional to the product of the likelihood and the prior density function. As addition is less prone to numerical overflow than multiplication, we take the log of this produce. The log of a product is just a sum of logs, meaning that our log-posterior is then proportional to the sum of our log-likelihood and the log-prior density. Therefore, to connect the value of our parameter and its respective prior distribution, the only implementational point left to cover is how to evaluate the parameters log-prior density. This can be done through the following `@property`

```python
print(x.log_density)
```

    -0.613706111907959

Naturally, should one wish to evaluate the prior density of the parameter, then the exponent can be taken

```python
print(jnp.exp(x.log_density))
```

    0.5413408768770793

## Cholesky decomposition

### Motivation

#### Single-step of the Cholesky decomposition

Before we examine the use of Cholesky decompositions in GPJax, I think it's worthwhile to first see why we even need the Cholesky decomposition. In mathematics and statistics, we are often presented with the task of solving a linear system of equations $A\mathbf{x}=\mathbf{y}$ for $A \in \mathbb{R}^{m \times n}$ and $\mathbf{x} \in \mathbb{R}^{n}$, $\mathbf{y}\in\mathbb{R}^{m}$. In this scenario, our goal is to identify the values of $\mathbf{x}$ and for people who studied numerical methods, this probably evokes flashbacks to manually calculating $\mathbf{x}$ using Gaussian elimination. Whilst manageable for a handful of unknown variables, we certainly would not want to do Gaussian elimination for systems containing thousands of variables. Fortunately, computers are quite good at solving these types of problem.

Let our matrix $A$ now be symmetric in $\mathbb{R}^{n \times n}$ (analogously termed Hermitian in $\mathbb{C}^{n \times n}$) and positive definite (SPD) i.e.
$$\begin{align*} A & = A^{\top} \quad & \text{(symmetry)} \\
\lambda^{\top}A\lambda & > 0 \text{ , for all } \lambda \in \mathbb{R}^{n}\quad & \text{(positive-definite)}\end{align*}.$$
If we were to now begin to apply Gaussian elimination to $A$ with a 1 element in the upper-left, then the first step would yield
$$A = \begin{bmatrix}1 & \mathbf{v}^{\top}\\
\mathbf{v} & K \end{bmatrix} = \begin{bmatrix}1 & \mathbf{0} \\
\mathbf{v} & I \end{bmatrix}\begin{bmatrix}1 & \mathbf{v}^{\top} \\
\mathbf{0} & K - \mathbf{v} \mathbf{v}^{\top} \end{bmatrix}. \tag{1}$$
Under the process of regular Gaussian elimination, we would now move onto the second column and introduce $(n-2)$-length $\mathbf{0}$ vector here too. However, doing so would invalidate the matrix's symmetry. Therefore, before proceeding to the second column, a Cholesky decomposition introduce an of $(n-1)$-length $\mathbf{0}$ vector into the fist row too. To achieve this upper-right triangulation of our reduced matrix system in (1) we can write
$$\begin{bmatrix}1 & \mathbf{v}^{\top} \\
\mathbf{0} & K - \mathbf{vv}^{\top} \end{bmatrix} = \begin{bmatrix}1 & \mathbf{0} \\
\mathbf{0} & K - \mathbf{vv}^{\top} \end{bmatrix}\begin{bmatrix}1 & \mathbf{v}^{\top} \\ \mathbf{0} & I \end{bmatrix}. \tag{2}$$
Bringing the expressions in (1) and (2) together now we get the following factorisation of $A$
$$\begin{align}A & = \begin{bmatrix}1 & \mathbf{v}^{\top}\\
\mathbf{v} & K \end{bmatrix} \nonumber \\ & = \begin{bmatrix}1 & \mathbf{0} \\
\mathbf{v} & I \end{bmatrix} \begin{bmatrix}1 & \mathbf{0} \\
\mathbf{0} & K - \mathbf{vv}^{\top} \end{bmatrix}\begin{bmatrix}1 & \mathbf{v}^{\top} \\ \mathbf{0} & I \end{bmatrix}. \tag{3}\end{align}$$

A Cholesky factorisation will apply the operation described in (3) iteratively for all remaining row-column pairs.

#### Generalisation

At this point, you may well ask the question of how this factorisation can ever be generalisable as it is by no means guaranteed that $A_{1, 1} = 1$ is true. Simply put, we can introduce a constant $\alpha$ such that $\alpha = \sqrt{A_{1,1}}$. Working through (3) with an upper-left entry of $\alpha^2$ now instead of 1 we get the general form of a Cholesky decomposition
$$
\begin{align}A & = \begin{bmatrix}\alpha^2 & \mathbf{v}^{\top}\\
\mathbf{v} & K \end{bmatrix} \nonumber \\
& = \begin{bmatrix}\alpha & \mathbf{0} \\
\frac{\mathbf{v}}{\alpha} & I \end{bmatrix} \begin{bmatrix}1 & \mathbf{0} \\
\mathbf{0} & K - \frac{\mathbf{vv}^{\top}}{\alpha^2} \end{bmatrix}\begin{bmatrix}1 & \frac{\mathbf{v}^{\top}}{\alpha} \\ \mathbf{0} & I \end{bmatrix} \\
& = L_{1}^{\top}A_{1}L_{1}\end{align}$$
When this process is iteratively applied from the upper-left element of $A$ down to the lower-right element, we get the complete factorisation which we know as the Cholesky decomposition
$$\begin{align}A & = L_1^{\top}L_2^{\top}\ldots L_n^{\top} L_n \ldots L_2 L_1 \\
& = L^{\top}L\end{align}$$
for an upper-triangular $L$.

#### Uniqueness

For a Cholesky decomposition to work, the upper-left element of $K-\frac{\mathbf{vv}^{\top}}{\alpha^2}$ must be positive, and it's not trivial as to why this is true. However, when considering the first step of the Cholesky decomposition, we established at that $A$ and $L_1^{-\top}A L_1^{-1}$ are both SPD. Further, $K-\frac{\mathbf{vv}^{\top}}{\alpha^2}$ is clearly the principal submatrix of $L_1^{-\top}A L_1^{-1}$ and is therefore positive definite. Right at the start when we defined a positive-definite matrix we saw that for a PD matrix the diagonals are all positive, and therefore the upper-left element of $K-\frac{\mathbf{vv}^{\top}}{\alpha^2}$ is guaranteed to be positive. Having proved this now for $n=1$, we can apply proof by induction to show that this positiveness is true for all steps of a Cholesky factorisation.

A generalisation of this result is that every SPD matrix has a unique Cholesky factorisation. This can be proved by considering the fact that each $\alpha$ is unique as it is determined by the form of $L^{\top}L$. Once $\alpha$ is determined, the first row of $L^{\top}$ is also available. This is true for each step of the decomposition, and therefore the factorisation is unique.

#### Computational details

In the words of [Linus Torvalds](https://en.wikipedia.org/wiki/Linus_Torvalds), _“Talk is cheap. Show me the code.”_, it helps at this point to see the Cholesky decomposition through an algorithmic lens.
```python
L = A
for i in range(n):
    for j in range(i+1, n):
        L[j, j:n] -= L[i, j:n] * R[i, j]/R[i, i]
    R[i, i:n] = R[i, i:n]/R[i, i]**0.5
```

The triangular nature of L is unveiled here the inner for loop's decreasing span. It is also the multiplication and subtraction operations of this inner loop that dominate the computational cost of a Cholesky decomposition. At the $i^{\text{th}}$ step of the loop, there are $i-j+1$ subtractions and $i-j+1$ multiplications required. By then summing over $i \in 1:n$ and $j \in (i+1):n$ we get a scaling of $\frac{1}{3}n^3$ floating-point operations (flops) with $n$. If we wanted to achieve the same factorisation using Gaussian elimination, then our computational complexity would scale $\frac{2}{3}n^3$ flops; twice as slow in comparison to a Cholesky factorisation.

#### More stable computations

Now that we've established what a Cholesky factorisation is and how one might go about computing the factors, a natural question one might ask is why we would go to this effort? The answer lies in the increased stability of a Cholesky factor in comparison to Gaussian elimination. The stabilisation comes from the fact that the values of $L$ are upper-bounded by the values of $A$. When working with the $2$-norm operator, we can actually go one further and state that $\lVert L^{\top} \rVert := \lVert L \rVert := \lVert A \rVert^{\frac{1}{2}}$. More generally, for any finite norm, the values $\lVert L\rVert$ and $\lVert A \rVert^{\frac{1}{2}}$ cannot differ by more than a factor of $\sqrt{n}$.

#### Matrix inversion

Drawing upon the concept of Cholesky factorisation, we can recast our original problem of solving a system of linear equations using the lower-triangular matrices that are given in a Cholesky factorisation
$$\begin{align} A\mathbf{x} & = \mathbf{b} \nonumber \\
\implies  LL^{\top}\mathbf{x} & = \mathbf{b} \nonumber \\
\implies  LL^{\top}\mathbf{x} & = L\mathbf{y} \ \text{ for some } \mathbf{y} \nonumber \\
\implies  L^{\top}\mathbf{x} & = \mathbf{y}
\end{align}$$
For conciseness, we'll now adopt the backslash notation $A\backslash \mathbf{b}$ to indicate that the vector $\mathbf{x}$ solves the linear system of equations $A \mathbf{x} = \mathbf{b}$. By this, we can equate $\mathbf{x}$ such that
$$\mathbf{x} = L^{\top} \backslash (L \backslash \mathbf{b}).$$
As $L$ is a lower-triangular matrix, we can use forward-substitution to determine the values of $\mathbf{b} = L\mathbf{y}$. Similarily, $\mathbf{x} = L^{\top}\mathbf{y}$ can be solved using backward-substitution due to the upper-triangular structure in $L^{\top}$.

#### Determinants

A final note on the Cholesky factorisation that is relavent within the context of Gaussian processes is that for a $n \times n$ SPD matrix $A$, the matrix's determinant and log-determinant can be computed by
$$\lvert A \rvert = \prod_{i=1}^n L^2_{ii} \quad \text{and} \quad \log\lvert A \rvert = 2\log\sum_{i=1}^n L_{i,i}$$
respectively.

### Implementation

Due to the enhanced stability that is experience when working with Cholesky factors in comparison to regular SPD matrices, all matrix inversions in GPJax are done using the Cholesky factors. To see this, consider the evaluation of the conjugate Gaussian process' marginal log-likelihood term. Mathematically, this quantity is written as
$$\log p(\mathbf{y} | X, \theta) = -0.5 \left(\operatorname{logdet}(K_{xx}+\sigma_n^2 I) + \mathbf{y}^{\top}(K_{xx} + \sigma_n^2 I)^{-1}\mathbf{y} + N \log (2 \pi) \right).$$
However, by using the Cholesky factorisation $L = \operatorname{cholesky}(K_{xx}+\sigma_n^2 I)$ to compute the matrix inverses and determinants, we instead write
$$\log p(\mathbf{y}| X, \theta) = -0.5 \left(\mathbf{y}^{\top}\boldsymbol{\alpha} + \sum\limits_{i=1}^{n}\log L_{ii} + N \log (2 \pi) \right)$$
where
$$\boldsymbol{\alpha} = L^{\top}\backslash (L \backslash \mathbf{y}).$$

It is this second form of the marginal log-likelihood that is used within GPJax due to its enhanced numerical stability; a crucial attribute when optimising the model's parameters with respect to it.

## Floating point precision

By default, Jax uses 32-bit floats. From the excellent Jax documentation, the rationale behind this is _" to mitigate the Numpy API’s tendency to aggressively promote operands to double"_. For many machine learning algorithms, 32-bit precision is probably sufficient. However, for Gaussian processes, we have to deal with challenging matrix inversion and we want to mitigate the effects of numerical rounding as much as possible.

To see this, consider a SPD matrix $A \in \mathbb{R}^{n \times n}$ and the vectors $\mathbf{x}, \mathbf{y} \in \mathbb{R}^{n}$ which, together, form a linear system of equations
$$ \mathbf{y} = A\mathbf{x}.$$
As the matrix is PD, we know that it is then invertible so we can then write
$$\begin{align*}\mathbf{y}^{\top}A^{-1}\mathbf{y} & = \mathbf{x}K^{\top}K^{-1}K\mathbf{x}\\
& = \mathbf{x}^{\top}K\mathbf{x}>0\end{align*}$$
i.e., for a PD matrix $A$ that is invertible, its inverse $A^{-1}$ is also invertible.

Whilst we might be happy with this fact theoretically, the result breaks down computationally due to the floating-point rounding that occurs in our machines. To convince ourself of this, we can define a function that checks if a matrix is PD.

```python
from typing import Callable
import jax

def is_positive_definte(A: jnp.ndarray):
    return (jnp.linalg.eigvals(A)>0).all()

def sqexp_kernel(x: jnp.ndarray, y: jnp.ndarray, variance: float = 1.0, lengthscale: float = 1.0):
    tau = jnp.square(x-y)
    return jnp.square(variance)*jnp.exp(-0.5*tau/jnp.square(lengthscale))

def kernel(func: Callable, x: jnp.ndarray, y: jnp.ndarray):
    return jax.vmap(lambda x: jax.vmap(lambda y: func(x, y))(y))(x)
```

In the above chunk of code, we have defined the squared exponential kernel for which we know the resultant Gram matrix is PD. However, as we'll see below, even for a small number of points, the rounding introduced by 32-bit precision yields a non-SPD matrix.

```python
x = jnp.linspace(-3., 3., 20, dtype=jnp.float32)
K = kernel(sqexp_kernel, x, x)
Kinv = jnp.linalg.inv(K)

print(is_positive_definte(K))
print(is_positive_definte(Kinv))
```

    False
    False

If we now redefine our original data vector `x` as an array with 64bit precision and force Jax to use 64-bit precision, then this same matrix is now PD.

```python
from jax.config import config; config.update("jax_enable_x64", True)

x = jnp.linspace(-3., 3., 20, dtype=jnp.float64)
K = kernel(sqexp_kernel, x, x)
Kinv = jnp.linalg.inv(K)

print(is_positive_definte(K))
print(is_positive_definte(Kinv))
```

    True
    True

For this reason, we enforce 64bit precision in GPJax by loading the configuration import above when one imports GPJax. To further enforce numerical stability in the kernel matrix, we add some _jitter_ to the matrix's diagonal. In doing this, we do introduce some numerical error, but the amount is often tolerable.

Again, we can see the effect of this jitter in the above example by reducing the distance between our points in `x`. This will reduce the size of the kernel matrix's eigenvalues and increase the chance of numerical rounding being a problem, even with 64-bit precision.

```python
x = jnp.linspace(-3., 3., 200, dtype=jnp.float64)
K = kernel(sqexp_kernel, x, x)
Kinv = jnp.linalg.inv(K)

print(is_positive_definte(K))
print(is_positive_definte(Kinv))
```

    False
    False

If we now add a tiny amount ($10^{-12}$) of jitter to the diagonal of K, then our inversion should restabilise.

```python
jitter = jnp.eye(K.shape[0])*1e-12
K += jitter
Kinv = jnp.linalg.inv(K)

print(is_positive_definte(K))
print(is_positive_definte(Kinv))
```

    True
    True

As can be seen, our matrix inversions are now much more stable.

In GPJax, we set the default amount of jitter as $10^{6}$. Often you'll be able to use a far smaller amount of jitter, particularly if you have a small amount of well-spaced input data. Conversely, you may need more jitter for some of your problems. Fortunately, this value can be altered upon instantiation of the GP prior through the option `jitter` argument. For example, this is how you could manually set the jitter to $10^{-10}$

```python
from gpjax.gps import Prior
from gpjax import RBF

f = Prior(RBF, jitter=1e-10)
```


# Section 5.6 $\quad$ Least Squares

**Recall** An $m\times n$ linear system $A\mathbf{x}=\mathbf{b}$ is consistent if and only if

**Question** What can we do if the system $A\mathbf{x}=\mathbf{b}$ is inconsistent?

>The **least square solution** to the linear system $A\mathbf{x}=\mathbf{b}$ is the solution to the system

**Remark** If $A$ is an $m\times n$ matrix,

### Example 1

Determine the least square solution to $A\mathbf{x}=\mathbf{b}$, where
\begin{equation*}
  A =
  \left[
    \begin{array}{cc}
      2 & 1 \\
      1 & 0 \\
      0 & -1 \\
      -1 & 1 \\
    \end{array}
  \right],~~~~~~
  \mathbf{b} =
  \left[
    \begin{array}{c}
      3 \\
      1 \\
      2 \\
      -1\\
    \end{array}
  \right].
\end{equation*}

```python
from sympy import *

A = Matrix([[2, 1], [1, 0], [0, -1], [-1, 1]]);
b = Matrix([3, 1, 2, -1]);

A.LDLsolve(b)
```

    Matrix([
    [24/17],
    [-8/17]])

Least square problems often arise in constructing a mathematical model from discrete data.

### Example 2

The following data shows U.S. per capita health care expenditures

Year | Per Capita Expenditures (in \$)
-----|------
1960    | $\qquad\qquad$ 143
1970    | $\qquad\qquad$ 348
1980    | $\qquad\qquad$ 1,067
1990    | $\qquad\qquad$ 2,738
1995    | $\qquad\qquad$ 3,698
2000    | $\qquad\qquad$ 4,560

- Determine the line of best fit to the given data.
- Predict the per capita expenditure for the year 2005, 2010, and 2015.

```python
from sympy import *
import numpy as np
import matplotlib.pyplot as plt

A = Matrix([[1960, 1], [1970, 1], [1980, 1], [1990, 1], [1995, 1], [2000, 1]]);
b = Matrix([143, 348, 1067, 2738, 3698, 4560]);
linePara = A.LDLsolve(b);

plt.xlabel('Year');
plt.ylabel('Per Capita Expenditures (in $)');
plt.title('U.S. per capita health care expenditures');
plt.plot(A[:,0], b, 'o', label = 'Data');
x = np.linspace(1950, 2030, 1000);
y = x * linePara[0] + linePara[1];
plt.plot(x, y, label = 'Prediction');
plt.legend();
plt.show()

2005*linePara[0] + linePara[1], 2010*linePara[0] + linePara[1], 2015*linePara[0] + linePara[1]
```

    (14026/3, 15748/3, 17470/3)


$\newcommand{\pr}{\textrm{Pr}}$
$\newcommand{\l}{\left}$
$\newcommand{\r}{\right}$
$\newcommand\given[1][]{\:#1\vert\:}$
$\newcommand{\var}{\textrm{Var}}$
$\newcommand{\mc}{\mathcal}$
$\newcommand{\lp}{\left(}$
$\newcommand{\rp}{\right)}$
$\newcommand{\lb}{\left\{}$
$\newcommand{\rb}{\right\}}$
$\newcommand{\iid}{\textrm{i.i.d. }}$
$\newcommand{\ev}{\textrm{E}}$

# 3.1 The binomial model

For exchangeable binary data, our joint beliefs about the i.i.d. responses $Y_1, \cdots, Y_n$
are well approximated by

* our beliefs about $\theta = \sum_{i=1}^N Y_i / N$
* the model that, conditional on $\theta$, the $Y_i$'s are i.i.d. binary random variables
with expectation $\theta$

If $Y_1, \cdots, Y_n \given \theta$ are i.i.d. binary$\lp\theta\rp$,

$$p\lp\theta \given y_1, \cdots, y_n \rp = \theta^{\sum y_i} \lp 1-\theta \rp^{n-\sum y_i}
\times p\lp \theta \rp / p\lp y_1, \cdots, y_n \rp$$

Given observations $y_1, \cdots, y_n$, we can use Bayes' rule to calculate
$p\lp \theta | \given y_1, \cdots, y_n \rp$. If we have a uniform prior, the denominator $p\lp y_1, \cdots, y_n \rp$
is known from calculus to be the beta distribution. An uncertain quantity $\theta$, known
to be between 0 and 1, has a $\textrm{beta}\lp a,b\rp$ distribution if
$$p\lp\theta\rp = \textrm{dbeta}\lp\theta, a, b\rp =
\frac{\Gamma\lp a+b \rp}{\Gamma\lp a \rp \Gamma\lp b \rp}\theta^{a-1} \lp 1-\theta \rp^{b-1}$$

for $0 \leq \theta \leq 1$. In the case of our binary data, $a$ is the number of success (e.g. $Y_i = 1$) and
$b$ is the number of failures (e.g. $Y_i = 0$). For a random variable with a beta distribution,

* $\textrm{mode}\l[\theta\r] = \frac{ a-1}{\lp a-1 \rp + \lp b-1 \rp}$ if $a>1$, $b>1$
* $\ev\l[\theta\r] = \frac{a}{a + b}$
* $\var\l[\theta\r] = \frac{ab}{\lp a+b+1 \rp\lp a+b\rp^2}
= \frac{\ev\l[\theta\r] \ev\l[1 - \theta\r]}{a + b + 1}$

## 3.1.1 Inference for exchangeable binary data

If we compare the relative probabilities of any two $\theta$ values $\theta_a$ and $\theta_b$
by calculating $\frac{p\lp\theta_a \given y_1, \cdots, y_n\rp}{p\lp\theta_b \given y_1, \cdots, y_n\rp}$,
we will see that the resulting expression is only a function of $\sum_{i=1}^n y_i$. This means that
$\sum_{i=1}^n Y_i$ contains all the information about $\theta$ available from the data, so we say
that $\sum_{i=1}^n Y_i$ is a **sufficient statistic** for $\theta$ and $p\lp y_1, \cdots, y_n \given \theta\rp$.
Basically, the individual binary responses don't give any more info than knowing in total how many
responses were 1. It is sufficient to know $\sum_{i=1}^n Y_i$ in order to make inferences about $\theta$.
In this case where $Y_1, \cdots, Y_n \given \theta$ are i.i.d. $\textrm{binary}\lp\theta\rp$ random variables,
the sufficient statistic $\sum_{i=1}^n Y_i$ has a binomial distribution with parameters $\lp n, \theta\rp$.

A random variable $Y\in \lb 0,1,\cdots,n\rb$ has a $\textrm{binomial}\lp n, \theta\rp$ distribution if
$$\pr\lp Y=y\given\theta\rp = \textrm{dbinom}\lp y,n,\theta\rp = {{n}\choose{y}} \theta^y\lp 1 - \theta\rp^{n-y},
y\in\lb 0,1,\cdots,n\rb$$
* $\ev\l[Y \given\theta\r] = n\theta$
* $\var\l[Y\given\theta\r] = n\theta\lp 1 - \theta\rp$

### Posterior inference under a uniform prior distribution

Having observed $Y=y$, our task is to obtain the posterior distribution of $\theta$:
$$p\lp \theta \given y\rp = \frac{p\lp y \given \theta\rp p\lp \theta \rp}{p\lp y \rp}$$
Using the binomial pdf and the fact that the denominator is the beta distribution, we
find that
$$p\lp\theta\given y\rp = \textrm{beta}\lp y + 1, n-y+1 \rp$$

### Posterior distributions under beta prior distributions

The uniform prior distribution has $p\lp\theta\rp = 1$ for all $\theta \in \l[0,1\r]$.
This is equivalent to a beta distribution with $a=1,b=1$. Note that $\Gamma\lp 1 \rp = 1$
by convention. We saw above that if $\theta \sim \textrm{beta}\lp 1,1\rp \textrm{(uniform)}$
and $Y \sim \textrm{binomial}\lp n, \theta \rp$, then
$\lb \theta \given Y=y \rp \sim \textrm{beta}\lp 1+y, 1+n-y\rp$. If we use a beta prior
$\theta \sim \textrm{beta}\lp a,b \rp$, then
$p\lp\theta \given y\rp = \textrm{dbeta}\lp\theta, a + y, b + n - y\rp$. So the ones are
replaced with $a$ and $b$. We can see this from
\begin{align}
p\lp \theta \given y\rp &= \frac{p\lp \theta \rp p\lp y \given \theta\rp}{p\lp y\rp} \\
&= \frac{1}{p\lp y \rp} \times \frac{\Gamma\lp a+b \rp}{\Gamma\lp a \rp \Gamma\lp b \rp}\theta^{a-1} \lp 1-\theta \rp^{b-1} \times {{n}\choose{y}} \theta^y\lp 1 - \theta\rp^{n-y}
\end{align}

The first part of this expression $\frac{1}{p\lp y \rp}$ is the normalizing factor. The second part
comes from the fact that the prior $p\lp \theta \rp$ is a beta distribution. The third part
is the binomial pdf. We can then write
\begin{align}
p\lp \theta \given y\rp &= c\lp n, y, a, b\rp \times\theta^{a+y-1}\lp 1 - \theta\rp^{b+n-y-1} \\
&= \textrm{dbeta}\lp \theta, a + y, b + n -y\rp
\end{align}

The first line says that $p\lp \theta \given y\rp$ is proportional to
$\theta^{a+y-1}\lp 1 - \theta\rp^{b+n-y-1}$ which has the shape of a beta distribution.
Since $p\lp \theta \given y\rp$ and the beta density must both integrate to 1, this means
that $p\lp \theta \given y\rp$ and the beta density are in fact the same. This **trick**
will be used throughout the book:

* We will recognize the posterior distribution is proportional to a known probability density,
and therefore must equal that density.

### Conjugacy

We have shown that a beta prior distribution and a binomial sampling model lead to a beta
posterior distribution. We say that the class of beta priors is **conjugate** for the
binomial sampling model.

**Definition 4 (Conjugate)**: A class $\mathcal{P}$ of prior distributions for $\theta$
is called conjugate for a sampling model $p\lp y\given\theta\rp$ if
$$p\lp \theta \rp \in \mathcal{P} \rightarrow p\lp\theta\given y\rp \in \mathcal{P}$$
In other words, the distribution is conjugate when the prior and posterior are in the same family.

While conjugate priors are easy to calculate, they may not always be the best model.
Mixtures of conjugate priors are very flexible and also computationally tractable.

### Combining information

If $\theta | Y=y \sim \textrm{beta}\lp a+y, b+n-y\rp$, then $E\l[ \theta \given y\r] = \frac{a+y}{a+b+n}$.
This **posterior expectation** is a combination of prior and data information:
\begin{align}
E\l[ \theta \given y\r] &= \frac{a+y}{a+b+n} \\
&= \frac{a+b}{a+b+n}\frac{a}{a+b} + \frac{n}{a+b+n}\frac{y}{n} \\
&=\frac{a+b}{a+b+n} \times \textrm{ prior expectation } + \frac{n}{a+b+n} \times \textrm{ data average}
\end{align}
For this model, the posterior expectation (or posterior mean) is a weighted average of the prior expectation
and the sample average, with weights proportional to $a+b$ and $n$ respectively. This leads to an
interpretation of $a$ and $b$ as "prior data":

* $a \approx$ "prior number of 1s"
* $b \approx$ "prior number of 0s"
* $a + b \approx$ "prior sample size"

If our sample size $n$ is much larger than $a+b$, then we would probably want a majority of our
information about $\theta$ to come from the data as opposed to the prior. We can see that is the
case if we look back at
$$ E\l[ \theta \given y\r]
=\frac{a+b}{a+b+n} \times \textrm{ prior expectation } + \frac{n}{a+b+n} \times \textrm{ data average}$$
When $n >> a+b$, $E\l[ \theta \given y\r]  \approx \frac{y}{n}$, the data average.

We can also reparameterize $a$ and $b$ in terms of a "prior sample size" $n' = a + b$ and "prior frequency"
$\theta' = a / n$. This may be easier to use when we are thinking of defining our prior because we can
define what we think the frequency might be ($\theta'$) and how many samples worth of data we are sure
($n'$). In particular, we can choose $n'$ based on how many samples we know we will get so we can sort
of specify how much weight we are putting on our prior.

### Prediction

We can create a predictive distribution for data we have yet to observe.
Let's consider binary data with $y_1, \cdots, y_n$ as the observed outcomes
of $n$ binary random variables. Let $\tilde{Y} \in \lb0, 1\rb$ be an additional
outcome yet to be observed. The **predictive distributon** of $\tilde{Y}$ is
the conditional distribution of $\tilde{Y}$ given $\lb Y_1=y_1, \cdots, Y_n=y_n\rb$.
For conditionally i.i.d. binary variables this distribution can be derived from
the distribution of $\tilde{Y}$ given $\theta$ and the posterior distribution of
$\theta$:
\begin{align}
\pr\lp\tilde{Y} = 1 \given y_1, \cdots, y_n\rp &= \ev\l[\theta \given y_1, \cdots, y_n\r] \\
&= \frac{a + \sum_{i=1}^n y_i}{a+b+n} \\
\pr\lp\tilde{Y} = 0 \given y_1, \cdots, y_n\rp &= \frac{a + \sum_{i=1}^n \lp 1-y_i \rp}{a+b+n}
\end{align}

You can see that the predictive distribution does not rely on unknown quantities but does
rely on the observed data which makes sense.

## 3.1.2 Confidence regions

**Definition 5 (Bayesian coverage)**: An interval $\l[l\lp y\rp, u\lp y\rp\r]$, based on the
observed data $Y=y$, has 95% Bayesian coverage for $\theta$ if
$$\pr \lp l\lp y\rp \leq \theta \leq u\lp y\rp \given Y=y\rp =0.95$$

**Definition 6 (frequentist coverage)**: A random interval $\l[l\lp Y\rp, u\lp Y\rp\r]$
has 95% frequentist coverage for $\theta$ if
$$\pr \lp l\lp Y\rp \leq \theta \leq u\lp Y\rp \given \theta\rp =0.95$$

In a sense, the frequentist and Bayesian notions of coverage describe pre- and post-experimental
coverage respectively.

My interpretation of this is that when you calculate a frequentist coverage region (same
as confidence interval as far as I know), the interpretation is based on *future* experiments.
For instance, typically we think of the frequentist 95% confidence interval as having a 95%
chance of encompassing the true parameter value. However, the confidence interval actually means
that there is a 95% chance that the confidence interval from a *future* experiment will contain
the true parameter value. When we perform the experiment, the formula for the confidence interval
is specified before the experiment, so the experiment itself is the "future" experiment. The
calculation of the confidence interval is the realization of the idea that the confidence interval
calculated from this "future" experiment has a 95% chance of containing the true parameter value.

In other words, in the frequentist case, you believe that there is a "true" value of $\theta$,
so before you even do your experiment,
you expect that the resulting confidence interval will contain the "true" value of $\theta$
with 95% probability.

The types of intervals constructed in this book are equal to the frequentist confidence intervals
asymptotically. Bayesian confidence intervals are also known as **credible regions** or credible intervals.

### Quantile-based interval

To make a $100 \times \lp 1- \alpha\rp \%$ quantile-based confidence interval,
find numbers $\theta_{\alpha / 2} < \theta_{1 - \alpha / 2}$ such that

1. $\pr\lp \theta \leq \theta_{\alpha / 2} \given Y=y\rp = \alpha / 2$
2. $\pr\lp \theta \geq \theta_{1 - \alpha / 2} \given Y=y\rp = \alpha / 2$

The numbers $\theta_{\alpha / 2}$, $\theta_{1 - \alpha / 2}$ are the $\alpha / 2$
and $1 - \alpha / 2$ posterior quantiles of $\theta$, so
$$\pr\lp \theta \in \l[\theta_{\alpha / 2}, \theta_{1 - \alpha / 2}\r] \given Y=y \rp = 1 - \alpha$$

### Highest posterior density (HPD) region

**Definition 7 (HPD region)**: A $100 \times \lp 1-\alpha\rp\%$ HPD region consists of a
subset of the parameter space, $s\lp y\rp \subset \Theta$ such that

1. $\pr\lp\theta \in s\lp y\rp \given Y=y\rp = 1 - \alpha$
2. If $\theta_a \in s\lp y \rp$, and $\theta_b \notin s\lp y \rp$, then
$p\lp \theta_a \given Y=y\rp > p\lp\theta_b \given Y=y\rp$

The first point says that the subset $s\lp y\rp$ contains $1-\alpha$ of the posterior probability.
This is the same as the $1-\alpha$ confidence interval. The second point says that every point in
$s\lp y \rp$ has a higher posterior probability than every point outside of $s\lp y \rp$. This
is not necessarily true for the $1-\alpha$ confidence interval. If the posterior is bimodal, the HPD
may not be a continuous region. You can imagine calculate the HPD by starting a horizontal line
at the top of a plot of the posterior distribution and moving it down until there is $1-\alpha$ of the
posterior above the line. This would be the HPD region.

# 3.2 The Poisson model

When our sample space is $\mathcal{Y} = \lb 0, 1, 2, \cdots \rb$, perhaps the simplest model
is the Poisson model. A random variable $Y$ has a Poisson distribution with mean $\theta$ if
$$p\lp Y=y \given \theta \rp = \textrm{dpois}\lp y,\theta\rp = \theta^ye^{-\theta}/y!$$
for $y \in \lb 0, 1, 2, \cdots \rb$. A Poisson random variable has

* $\ev \l[Y\given\theta\r] = \theta$
* $\textrm{Var}\l[Y\given\theta\r] = \theta$

The Poisson family of distributions is said to have a mean-variance relationship because
if one Poisson has a larger mean than another, it will also have a larger variance.

## 3.2.1 Posterior inference

If we model $Y_1, \cdots, Y_n$ as i.i.d. Poisson with mean $\theta$, then the joint pdf
of our sample data is
\begin{align}
\pr\lp Y_1=y_1, \cdots, Y_n=y_n \given\theta\rp &= \prod_{i=1}^n p\lp y_i\given\theta\rp \\
&=\prod_{i=1}^n\frac{1}{y_i!}\theta^{y_i}e^{-\theta} \\
&=c\lp y_1, \cdots, y_n \rp\theta^{\sum y_i}e^{-n\theta}
\end{align}
We can compare two values of $\theta$ *a posteriori* and see
\begin{align}
\frac{p\lp \theta_a \given y \rp}{p \lp \theta_b \given y \rp} &=
\frac{e^{-n\theta_a} \theta_a^{\sum y_i} p\lp\theta_a\rp}{e^{-n\theta_b} \theta_b^{\sum y_i} p\lp\theta_b\rp}
\end{align}
As with the binary model, $\sum_{i=1}^n Y_i$ contains all the information about $\theta$ that is
available in the data, so $\sum_{i=1}^n Y_i$ is a sufficient statistic. Furthermore,
$\lb\sum_{i=1}^n Y_i\rb \sim \textrm{Poisson}\lp n\theta\rp$.

### Conjugate prior

A class of prior densities is conjugate for the sampling model $p\lp y_1, \cdots, y_n\rp$ if the
posterior distribution is also in the class. For the Poisson sampling model, our posterior for
$\theta$ has the following form
\begin{align}
p\lp y_1, \cdots, y_n\rp &\propto p\lp \theta\rp \times p\lp y_1, \cdots, y_n \given \theta\rp
&\propto p\lp \theta\rp \times \theta^{\sum y_i} e^{-n\theta}
\end{align}
We need to find a conjugate class of densities. The simplest class of densities of this form
is known as the family of gamma distributions.

### Gamma distribution

An uncertain positive quantity $\theta$ has a $\textrm{gamma}\lp a,b \rp$ distribution if
$$p\lp \theta \rp = \textrm{dgamma}\lp\theta,a,b\rp = \frac{b^a}{\Gamma\lp a\rp} \theta^{a-1}e^{-b\theta}$$
for $\theta,a,b > 0$. For such a random variable

* $\ev\l[\theta\r] = a/b$
* $\var\l[\theta\r] = a/b^2$
* $\textrm{mode}\l[\theta\r] = \begin{cases} \lp a-1\rp/b & \mbox{if }a>1 \\
0 & \mbox{if }a\leq 1\end{cases}$

### Posterior distribution of $\theta$

The gamma family is conjugate for the Poisson sampling model:
$$
\begin{array}{c}
\theta \sim \textrm{gamma}\lp a,b \rp \\
Y_1, \cdots, Y_n \given \theta \sim \textrm{Poisson}\lp\theta\rp
\end{array}
\Rightarrow
\lb \theta \given Y_1, \cdots, Y_n\rb \sim \textrm{gamma}\lp a + \sum_{i=1}^n Y_i, b+n\rp
$$
The posterior expectation of the $\theta$ is a convex combination of the prior expectation
and the sample average
$$\ev\l[\theta\given y_1, \cdots, y_n\r] = \frac{b}{b+n}\frac{a}{b} + \frac{n}{b+n}\frac{\sum y_i}{n}$$

* $b$ is interpreted as the number of prior observations
* $a$ is interpreted as the sum of counts from $b$ prior observations

As for the binary model, the information from the data dominates for large $n$.

We can derive the posterior predictive distribution as
$$p\lp\tilde{y} | y_1, \cdots, y_n\rp =
\frac{\Gamma\lp a + \sum y_i + \tilde{y}\rp}{\Gamma\lp\tilde{y} + 1\rp\Gamma\lp a + \sum y_i\rp}
\lp\frac{b + n}{b + n + 1}\rp^{a + \sum y_i}\lp\frac{1}{b+n_1}\rp^\tilde{y}$$
for $y\in\lb 0,1,2,\cdots\rb$. This is the negative binomial distribution with parameters
$\lp a+\sum y_i, b+n\rp$ for which

* $\ev\l[\tilde{Y} \given y_1, \cdots, y_n\r] = \frac{a + \sum y_i}{b + n} = \ev\l[\theta \given y_1, \cdots, y_n\r]$
* $\var\l[\tilde{Y} \given y_1, \cdots, y_n\r] = \frac{a + \sum y_i}{b + n}\frac{b+n+1}{b+n} =
\var\l[\theta \given y_1, \cdots, y_n\r] \times \lp b+n+1\rp = \ev\l[\tilde{Y} \given y_1, \cdots, y_n\r] \times
\frac{b+n+1}{b+n}$

The predictive variance is to some extent a measure of our posterior uncertainty about a new sample
$\tilde{Y}$ from the population. Uncertainty about $\tilde{Y}$ stems from uncertainty about the population
(uncertainty in our estimate of $\theta$) and variability in sampling. For large $n$, the uncertainty about
$\theta$ is small ($\lp b+n+1\rp/\lp b+n\rp \approx 1$) and so the predictive variability
becomes $\theta$, the sampling variability from the Poisson. For small $n$, the uncertainty for $\tilde{Y}$
includes additional uncertainty about $\theta$.

# 3.3 Exponential families and conjugate priors

The binomial and Poisson models are both instances of one-parameter **exponential family models**.
A one-parameter exponential family model is any model whose densities can be expressed as
$$p\lp y\given \phi\rp = h\lp y\rp c\lp\phi\rp e^{\phi t\lp y\rp}$$ where $\phi$ is the unknown
parameter and $t\lp y\rp$ is the sufficient statistic.

Combining priors of the form
$$p\lp \phi \given n_0, t_0\rp = \kappa\lp n_0, t_0\rp c\lp \phi \rp^{n_0} e^{n_0 t_0 \phi}$$
with information from $Y_1, \cdots, Y_n \sim \iid p\lp y\given \phi\rp$ yields the following
posterior distribution
\begin{align}
p\lp \phi \given y_1, \cdots, y_n\rp &\propto p\lp\phi\rp p\lp y_1, \cdots, y_n \given \phi\rp \\
&\propto c\lp \phi\rp^{n_0 + n} \exp\lb\phi\times\l[ n_0t_0 + \sum_{i=1}^n t\lp y_i \rp\r]\rb \\
&\propto p\lp \phi \given n_0 + n, n_0t_0 + n\bar{t}\lp \mathbf{y} \rp\rp
\end{align}
where $t\lp \mathbf{y} \rp = \sum t\lp y_i \rp/n$. The similarity between the posterior and prior
distributions suggests that $n_0$ can be interpreted as a "prior sample size" and $t_0$ as a
"prior guess" of $t\lp Y\rp$. In fact
$$\ev\l[t\lp Y\rp\r] = \ev\l[\ev\l[t\lp Y\rp \given \phi\r]\r] =
\ev\l[-c'\lp \phi\rp/c\lp\phi\rp\r] = t_0$$
so $t_0$ represents the expected value of $t\lp Y\rp$. The parameter $n_0$ is a measure of how
informative the prior is. If you want to think of the relative strength of the prior versus the
data, you can think of it that the prior distribution $p\lp \phi \given n_0,t_0\rp$
contains the same information as $n_0$ independent samples from the population.

You can take the binomial and Poisson pdfs and rewrite them in the exponential family form.

```python

```

readings/A First Course in Bayesian Statistical Methods/3. One-parameter models.ipynb


## Setting up python environment

```python
# reset all previously defined varibles
%reset -f

# import everything from sympy moduleb
from sympy import *

# pretty math formatting
init_printing()   # latex
```

### Symbolic variables must be declared

```python
x,y,z = symbols('x y z')
t     = symbols('t')
```

## Example

Evaluate the line integral $\int_C y \, ds$ along the parabola $y = 2\sqrt{x}$ from $A(3,2\sqrt{3})$ to $B (24,  4\sqrt{6})$.

```python
x = t
y = 2*sqrt(t)

f = y

s = sqrt( diff(x,t)**2 + diff(y,t)**2 )

# integration
integrate(f*s,[t,3,24])

```

---------------------------------
## Example

Find the work done by the vector field $\mathbf{F} = r^2 \mathbf{i}$ in moving a particle along the helix $x = \cos{t}, y = \sin{t}, z = t$ from the point $(1,0,0)$ to $(1,0,4 \pi)$.

Note: $r^2 = x^2 + y^2 + z^2$

```python
x = cos(t)
y = sin(t)
z = t

F = [x**2 + y**2 + z**2  , 0   , 0]

integrate(
          F[0]*diff(x,t) + F[1]*diff(y,t) + F[2]*diff(z,t),
          [t,0,4*pi]
         )

```

---------------------------
## Example

Evaluate $\displaystyle \int_A^B 2xy\,dx + (x^2 - y^2)\, dy$ along the arc of the circle $x^2 + y^2 = 1$ in the first quadrant from $A(1,0)$ to $B(1,0)$.

```python
x = cos(t)
y = sin(t)

F = [2*x*y, x**2 - y**2]

integrate(
          F[0]*diff(x,t) + F[1]*diff(y,t),
          [t,0,pi/2]
         )
```

-------------------
## Example

Prove that $\displaystyle \mathbf{F} = (y^2 \cos{x} + z^3)\mathbf{i} + (2y \sin{x} - 4)\mathbf{j} +(3xz^2 + z) \mathbf{k}$ is a conservative force field. Hence find the work done in moving an object in this field from point $(0,1,-1)$ to $(\pi/2, -1, 2)$.

### Note:

If a vector field $\mathbf{F} = F_1 \mathbf{i} + F_2 \mathbf{j} +  F_3 \mathbf{k
} $ is conservative then

$$
\mathbf{curl} (\mathbf{F}) = \left|
\begin{array}{ccc}
\mathbf{i} & \mathbf{j} & \mathbf{k} \\
\frac{\partial}{\partial x} & \frac{\partial}{\partial y} & \frac{\partial}{\partial z} \\
F_1 & F_2 & F_3
\end{array}
\right| = \vec{0}
$$

```python
# reset variables from previous examples
x,y,z = symbols('x y z')

def curl(F):
    c1 = diff(F[2],y) - diff(F[1],z)
    c2 = diff(F[0],z) - diff(F[2],x)
    c3 = diff(F[1],x) - diff(F[0],y)
    return [c1,c2,c3]
```

```python
F = [(y**2 *cos(x) + z**3), 2*y* sin(x) - 4. , 3*x*z**2 + z ]
```

```python
curl(F)

```

Zero curl implies conservative vector field.

-----------------------------------------

## Example

Evaluate
$$
\underset{R}{
\int\!\!\!\!\int} (x^2 + y^2) \, dA
$$
over the triangle with vertices $(0,0)$, $(2,0)$, and $(1,1)$.

```python
x,y = symbols('x y')

integrate(
    integrate(x**2 + y**2, [x,y,2-y]),
    [y,0,1])

```

--------------------

## Example

Evaluate

$$
\underset{R}{
\int \!\!\! \int} (x + 2y )^{-1/2} \, dA
$$
over the region $x - 2y \le 1$ and $x \ge y^2 +1$.

```python
x,y = symbols('x y')

# does not work in one shot
integrate((x + 2*y)**(-Rational(1,2)), [x, y**2+1, 1+2*y])

```

```python
#manually simlipy one radical

integrate(2*sqrt(4*y + 1) - 2*(y+1),[y,0,2])
```

```python
integrate(
integrate(x**2 + y**2, [y,0,2 - x]),
    [x,1,2])
```

```python

```