```python
from IPython.display import display, Math, Latex
import pystan
import matplotlib.pyplot as plt
import numpy as np
```

<http://www.DataJavelin.com>
## Dr Peter Hurley

 What is Probability?

 Idea: Probability is the Long Run Expected Number of Occurances of an Event

 When you roll a fair dice once it will come up with only one answer, but after doing this many times each outcome will occur 1/6 of the time. So the probability of an outcome is 1/6.

 Another way of saying this: The probability of an event A is equal to the number of ways in which A can occur divided by the number of ways in which any event can happen.

 This then helps us answer questions such as, what is the probability of drawing three aces in a row from a shuffled deck of cards?

 There are four ways of picking the first ace, out of 52 possible outcomes, three ways of picking the first ace, out of 51 possible outcomes, and two ways of picking the first ace, out of 50 possible outcomes. Therefore:

\begin{equation}
P(3\ Aces) = \frac{4}{52} * \frac{3}{51} * \frac{2}{50}
\end{equation}

## ***QUESTION***

 We can also ask more nuanced questions, such as what is the probability of event A given event B. For example, what is the probability of a card is a King given it is a face card?

\begin{equation}
P(A\ |\ B) =  \frac{P(A \cap	B)}{P(B)}
\end{equation}

```python

```

## ***QUESTION***

 Suppose a family has two children and suppose one of the children is a boy. What is the probability that both children are boys?

```python

```

## ***QUESTION***

The Monte Hall Problem:

Suppose you're on a game show, and you're given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what's behind the doors, opens another door, say No. 3, which has a goat. He then says to you, "Do you want to pick door No. 2?" Is it to your advantage to switch your choice?

```python

```

 We can manipulate these equations as in other areas of maths as we did above

\begin{equation}
P(A\ |\ B) =  \frac{P(A \cap	B)}{P(B)}
\end{equation}

\begin{equation}
P(A \cap	B) = P(A) * P(B | A)
\end{equation}

\begin{equation}
P(A \cap	B) = P(B) * P(A | B)
\end{equation}

\begin{equation}
P(A) * P(B | A) = P(B) * P(A | B)
\end{equation}

\begin{equation}
P(B | A) = \frac{P(B) * P(A | B)}{P(A)}
\end{equation}

 This last equation is Bayes Theorem, a source of heated debate among statisticians throughout the 20th century.

 You may wonder what the fuss is as surely everyone accepts the above?

 The issue comes in when we consider how far we can apply this equation.

## Probabilistic models
* A model describes data that one could observe from a system
* Uses probability theory to express all forms of uncertainty and noise associated with our model
* Inverse probability (i.e. Bayes rule) to infer parameters, adapt models, make predictions and learn from data.

A good example of how Bayesian probability is more intuitive is to think of a situation such as the probability of rain, given the sky is cloudy, or that there is a sprinkler nearby. We use prior information to decide if it has rained, rather than just looking at the number of times it has rained in past.

Alternatively there can also be cases where a frequentist model breaks down. Suppose you are an astronomer looking at Saturn with an old telescope. Combining many measurements of the shape of the planet will not improve your result on resolving the shape. However given a model of how a planet with a ring or without you can test how well each model describes Saturn and infer the correct shape.

Other key cases are those in which you can never repeat the experiment, e.g. modelling how the universe grew since the big bang or the diversification of songbirds in the Andes.

\begin{equation}
P(B | A) = \frac{P(B) * P(A | B)}{P(A)}
\end{equation}

$P(B | A)$ = Posterior

$P(B)$ = Prior

$P(A | B)$ = Likelihood

$P(A)$ = Marginal Likelihood

 Transparent way of including model prior information

Gives full probability distribution

 As the area under a posterior must equal one the marginal likelihood can be thought of as a normalistion value, and so the equation is often written as:

\begin{equation}
P(B | A) \propto P(B) * P(A | B)
\end{equation}

 In that case your posterior is your prior state of belief being updated by the new data.

P(A) is actually an integral which is often difficult to solve, and must be done numerically,  motivates the need for MCMC type sampling. MCMC-like algorithms will generate points proportionally to the posterior values in each region. A simplistic way of thinking about this is a sampler generates some initial starting parameter values, calculates how well a model with these parameters explains the data and then generates a new set of parameter values nearby. If these new values are an improvement we always accept the new point and move there, otherwise we have a small chance we move there. The exact way in which this works varies by method.

 So far so good, but there is a lot of debate and discomfort with how you choose an appropriate prior. However disagreements over what prior is used can at least be transparent, and the correct description of the current state of knowledge before new data is an important and necessary part of debate.

### Why bother with all this Bayesian probabilistic stuff?
* Prior information help us extract more information from data

### Prob. model of straight line
$\mathbf{y}=m\mathbf{x}+c + \epsilon$

likelihood = $p(\mathbf{y}|m,c,\mathbf{x}) = \mathcal{N}(m\mathbf{x}+c,\sigma^2)$

prior = $p(m,c) = \mathcal{N}(0,\Sigma_p)$

posterior (after a bit of maths)

$p(m,c|\mathbf{y},\mathbf{x}) \propto  \mathcal{N}(\sigma^{-2}(\sigma^{-2}\mathbf{x}\mathbf{x}^T + \Sigma_{p}^{-1})^{-1}\mathbf{x}\mathbf{y},$
$\sigma^{-2}\mathbf{x}\mathbf{x}^T + \Sigma_{p}^{-1})$

Simple example straight line, error only in y, simple gaussian prior

What happens if intercept is positive? gets complicated fast

Have prior information on intercept

$c \sim \mathcal{N}(0.3,0.1)$

* The uncertainty in our inferred model parameters contains information.

* We do not want to just know the best answer, we
want to know how uncertain we are.

### The mode is an inappropriate measure of Probability mass for High D!!!

Think about mass $m$ and density $\rho$

$$
m=\int \rho dV
$$

Probability works in a similar way:

$$
\mathbb{E}_{\pi}[f]=\int_{Q}\pi(q)f(q)dq
$$

To compute expectations efficiently with MCMC, we want to spend time in areas of the posterior that will contribute most to the expectation.

There is a balance between density and volume.. but how does volume change with number of dimensions??

### 1D

```python
plt.hist(np.random.normal(0,1,50000),bins=np.arange(-4.25,4.25,0.5),normed=True,alpha=0.5);
plt.axvline(x=-0.25,c='r')
plt.axvline(x=0.25,c='r')

plt.axvline(x=1.25,c='g')
plt.axvline(x=1.75,c='g')
```

Where is most of the mass?

Centre (red): $0.4 \times 0.5 = 0.2$

Edge (green): $0.13 \times 0.5 = 0.065$

```python
points=np.random.multivariate_normal([0,0],[[1,0],[0,1]],50000)
plt.hist2d(points[:,0],points[:,1],bins=np.arange(-4.25,4.25,0.5),normed=True);
plt.colorbar()
```

Where is most of the mass?

Centre (red):  $0.15×0.5^2=0.0375$

Edge (green):  $\approx 0.08 \times 0.5^2 \times 16=0.32$

To identify the regions of parameter space that dominate expectations we should instead consider the product of density and volume. In high dimensions, the volume behaves very differently from the density, resulting in a tension that pushes the important regions of parameter space away from the mode further out towards the tails of the distribution into a region known as the **typical set**. The move to a thin shell with higher dimensions is called the **concentration of measure**.

For more info, see this [blog post](https://mc-stan.org/users/documentation/case-studies/curse-dims.html)

### Using Uncertianty in decision making means making better decisions

* Business consider risk when making decisions
* Uncertianty should be considered when making decisions based on predictions

See [Blog post on Bayesian Decision making](https://pdh21.github.io/jekyll/update/2019/06/06/FACYNation_BDT.html)

## Calculating the Posterior
* Analytical approximation (often impossible)
* Grid approximation (not efficient)
* Quadratic approximation (limited)
* Markov chain Monte Carlo (intensive)

### [Comparing other MCMC techniques](https://chi-feng.github.io/mcmc-demo/app.html?algorithm=NaiveNUTS&target=banana)
e.g.
* Metropolis Hastings
* Gibbs
* Standard NUTS
* Nested Sampling

## Probabilistic Programming
*Define probability models & solve automatically*

[Good blog post on philosophy of Probabilistic Programming](https://www.oreilly.com/content/probabilistic-programming/)

# Stan

* Language for building probabilistic models
* Algorithms for fitting probabilistic models
* Interfaces for using probabilistic models

### Other PPLs:
* [Tensorflow probability](https://www.tensorflow.org/probability), formally Edward
* [PyMC3 (soon PyMC4)](https://docs.pymc.io/)
* [Pyro](http://docs.pyro.ai/en/stable/)

## Language
Specify probabilisitic model: the joint probability distribution function,

$$p(\theta,x)$$

* $\theta$ = parameters
* $x$ = data
* $p$ = model

Stan is a language:
* Statically typed, imperative
* User can define programs using parameters, data and  $\log p(\theta,x)$
* Any program can be written as long as $\log p(\theta,x)$ is differentiable

Statically typed like C, C++. i.e. types declared at compile time rather than run time like Python.

Imperative, you tell the compiler what you want to happen, step by step

## Algorithms
* Bayesian Inference: No U-Turn Hamiltonian Monte Carlo,
    - $p(\theta|x)$ approximated with $[\theta^1,\theta^2,..\theta^N]$
* Approximate Bayesian Inference: Variational Inference,
    - $\hat{p}(\theta|x) \approx q(\hat{\phi})$, where $\phi = argmin_{\phi} D_{KL}(q(\theta|\phi)||p(\theta,x))$
* Optimisation: Max
    - $\hat{\theta} = argmax_{\theta} p(\theta,x)$

### The advantages of Stan and HMC

* Uses gradient to traverse posterior more efficiently
* It knows when there is a problem
* Stan automatically tunes the parameters of HMC

## Interfaces
* CmdStan,PyStan,RStan
* C++ API
* C++ auto-diff library
* Software built with Stan: RStanArm, brms, prophet

## Writing a model in Stan
* A Stan program is organized into a sequence of named blocks

```python
example_model="""functions {
// ... function declarations and definitions ...
}
data {
// ... declarations ...
}
transformed data {
// ... declarations ... statements ...
}
parameters {
// ... declarations ...
}
transformed parameters {
// ... declarations ... statements ...}
model {
// ... declarations ... statements ...
}
generated quantities {
// ... declarations ... statements ...
}
"""
```

### Other things to note:
* Add comments with `\\`
* Sampling statements with `~`
* Can print output with `print()`
* `;` at end of each line

Documentation for Stan: https://mc-stan.org/docs/2_23/stan-users-guide/index.html

 As an example lets consider a question of if a coin is fair or not. We begin with a flat prior and update it as we flip a coin.

```python
coin_model ="""// Inferring a Rate
data {
  int<lower=1> n;
  int<lower=0> h;
  real<lower=0.0,upper=1.0> mu;
  real<lower=0.0,upper=10> sig;
}
parameters {
  real<lower=0,upper=1> theta;
}
model {
  // Prior Distribution for Rate Theta
  theta ~ normal(mu, sig);

  // Observed Counts
  h ~ binomial(n, theta);
}"""

sm=pystan.StanModel(model_code=coin_model)
```

    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_78d0d6b6c8de42c1347d66aa8351d4cc NOW.

```python
model_data={
        'n':100,
        'h':50,
        'mu':0.5,
        'sig':0.5
}

```

```python
fit=sm.sampling(data=model_data,chains=4,iter=1000,seed=12329)
```

```python
print(fit)

plt.plot(np.arange(0, fit['theta'].size), fit['theta'])
plt.xlabel('Sample')
plt.ylabel('Theta')
plt.show()

plt.hist(fit['theta'])
plt.xlabel('Theta')
plt.ylabel('Number of Samples')
plt.show
```

```python
model_data={
        'n':10,
        'h':7,
        'mu':0.5,
        'sig':0.5
}

fit=sm.sampling(data=model_data,chains=4,iter=1000,seed=12329)

print(fit)

plt.hist(fit['theta'])
plt.xlabel('Theta')
plt.ylabel('Number of Samples')
plt.show
```

### Exercise:
Using Python, generate some fake data for a coin tosses (i.e. ten flips) and fit the Stan model to sequential number of tosses and plot the results to see how the posterior changes as a function of the number of tosses.

```python

```

### The double coin toss game

 Sometimes our data may come in a more complex form. For instance imagine if you have a game that involves coin tosses. You flip a coin until you flip a heads and a tails in a row. Your opponent flips a coin until they flip two heads in a row. All the information you have is who won each game. From this you want to understand if one coin was biased or not.

 As HT and HH in two coin flips are equally likely we may run the code above and see how close our theta value is to 0.5

 After 2,000 flips your opponent has won 747 times but you have won 1,253.

```python
model_data={
        'n':2000,
        'h':747,
}

fit=sm.sampling(data=model_data,chains=4,iter=1000,seed=12329)

print(fit)

plt.hist(fit['theta'])
plt.xlabel('Theta')
plt.ylabel('Number of Samples')
plt.show
```

 So can we conclusively say that our coin was biased?

 Once again this is a case where we should have questioned our underlying assumptions, i.e. that both opponents had a 50% chance of winning. In future this will be covered in part by prior predictive checks which should give us intuition on how our model behaves. For now let us first generate data in the way described assuming both players have a fair coin

```python
bob_wins = 0
alice_wins = 0

for i in range(2000):
    bob_seq = np.array([])
    alice_seq = np.array([])
    bob_poss_win = 0
    alice_poss_win = 0
    while bob_wins + alice_wins < 2000:
        bob_seq = np.append(bob_seq, np.random.randint(2,size=1))
        alice_seq = np.append(alice_seq, np.random.randint(2,size=1))
        if np.prod((bob_seq.size > 1) and (bob_seq[-2:] == [0,0])):
            bob_poss_win = 1
        if np.prod((alice_seq.size > 1) and (alice_seq[-2:] == [0,1])):
            alice_poss_win = 1
        if (bob_poss_win == 1) and (alice_poss_win == 1):
            bob_seq = np.array([])
            alice_seq = np.array([])
            bob_poss_win = 0
            alice_poss_win = 0
        elif bob_poss_win == 1:
            bob_wins += 1
            bob_seq = np.array([])
            alice_seq = np.array([])
            bob_poss_win = 0
        elif alice_poss_win == 1:
            alice_wins += 1
            bob_seq = np.array([])
            alice_seq = np.array([])
            alice_poss_win = 0

print(bob_wins)
print(alice_wins)

```

    752
    1248

 Understanding a problem deeply is important when building a model. Misunderstanding a model can lead to misinterpretation. In this case there is a subtle difference, if you need to flip HT and first flip a H you will not win if on the next coin toss you flip another H - but you are still only one flip away. If you need to flip HH and after one H toss flip a T, you are again two flips away from winning.

## Exercise:
Create some fake data by sampling from a normal distribution with a known mean and standard deviation.

Build a Stan model that fits that data and estimates the mean and standard deviation.

* What did you choose as a prior?
* How did it affect your posterior?

### Exercise:
* Create some fake data for a linear regression model (i.e. $y=mx +c$)

* Build a Stan model to fit the data

* Look at results, compare posterior for slope and intercept

### So far:
* Introduced Probability theory
* Provided motivation as to we want to use for science and machine learning
* Stan is a PPL that we can use to build probabilisitic models
* Can build some simple Stan models

```python

```


# 10 Ordinary differential equations – Integrators
More about integrators to solve ODEs.

* Euler's Rule (and it's shortcomings)
* Runge-Kutta methods

## Euler's Rule

From the last lecture: Given the $n$-dimensional vectors from the ODE standard form

$$
\frac{d\mathbf{y}}{dt} = \mathbf{f}(t, \mathbf{y})
$$
the **Euler rule** amounts to

\begin{align}
\mathbf{f}(t, \mathbf{y}) = \frac{d\mathbf{y}(t)}{dt} &\approx \frac{\mathbf{y}(t_{n+1}) - \mathbf{y}(t_n)}{h}\\
\mathbf{y}_{n+1} &\approx \mathbf{y}_n + h \mathbf{f}(t_n, \mathbf{y}_n) \quad \text{with} \quad \mathbf{y}_n := \mathbf{y}(t_n)
\end{align}

Euler's rule cannot obtain high precision without very small $h$ but this leads to higher round-off error accumulation.

### Euler's Rule applied to the 1D harmonic oscillator (theory)

\begin{alignat}{3}
f^{(0)}(t, \mathbf{y}) &= y^{(1)},
    &\quad y^{(0)}_0 &= x_0,\\
f^{(1)}(t, \mathbf{y}) &=   - \frac{k}{m} y^{(0)},
    &\quad y^{(1)}_0 &= v_0.
\end{alignat}

with $k=1$; $x_0 = 0$ and $v_0 = +1$.

First step from $t=0$ to $t=h$:

\begin{align}
y^{(0)}_1 &= y^{(0)}_0 + h y^{(1)}_0 = x_0 + v_0 h\\
y^{(1)}_1 &= y^{(1)}_0 + h \left(-\frac{k}{m}y^{(0)}_0\right) = v_0 + \frac{-k x_0}{m} h
\end{align}

Compare to the equations of movement under a constant force (*ballistic motion*):

$$
x = x_0 + v_0 h + \frac{1}{2} a h^2, \quad v = v_0 + a h
$$

In Euler, the position does not even contain the $h^2$ term, i.e., the acceleration only contributes *indirectly* via the velocity of the *next* step. (Remember: Euler's error is $\mathcal{O}(h^2)$!)

### Euler's Rule: Application to 3 oscillators (numerical)
`git pull` new resources.

Put the notebook `10-ODE_integrators-students.ipynb` *and* the module `integrators.py` into the same directory, for instance a work directory `~/PHY494/10_ODEs`.

Open [integrators.py](https://github.com/ASU-CompMethodsPhysics-PHY494/PHY494-resources/blob/master/10_ODEs/integrators.py) to have a look at it: it contains code to integrate Newton's equations of motions for different forces and with different integrators (based on the work done in the last lecture).

```python
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
matplotlib.style.use('ggplot')

import integrators_solution as integrators
```

```python
t_max = 20
t_harm, y_harm = integrators.integrate_newton(t_max=t_max)
t_anharm, y_anharm = integrators.integrate_newton(t_max=t_max,
                                                  force=integrators.F_anharmonic)
t_power, y_power = integrators.integrate_newton(t_max=t_max,
                                                force=integrators.F_power)
```

```python
plt.plot(t_harm, y_harm[:, 0], label=r"harmonic $\frac{k}{2} x^2$")
plt.plot(t_anharm, y_anharm[:, 0], label=r"anharmonic $\frac{k}{2} x^2(1 - \frac{2}{3} \alpha x)$")
plt.plot(t_power, y_power[:, 0], label=r"power $\frac{k}{6} x^6$")
plt.legend(loc="best")
```

#### Phase-space portrait
Plot position against momentum ($x, p$):

```python
m = 1
x = y_harm[:, 0]
p = m * y_harm[:, 1]
ax = plt.subplot(1,1,1)
ax.plot(x, p)
ax.set_aspect(1)
```

```python
t, y = integrators.integrate_newton(t_max=40,
                                    force=integrators.F_harmonic,
                                    h=0.01)
t2, y2 = integrators.integrate_newton(t_max=40, h=0.001,
                                      force=integrators.F_harmonic)
```

```python
ax = plt.subplot(1,1,1)
ax.plot(y[:, 0], m * y[:, 1], 'b-', label=r"$h=0.01$")
ax.plot(y2[:, 0], m * y2[:, 1], 'r--', label="$h=0.001$")
ax.set_xlabel("position $x$")
ax.set_ylabel("momentum $p$")
ax.set_aspect(1)
ax.legend(loc="best")
```

#### Energy
Energy is conserved: stringent test for the integrator!

$$
E = T_\text{kin} + U\\
\frac{dE}{dt} = 0
$$
with $T_\text{kin} = \frac{1}{2} m v^2$.

Calculate and plot the three energy terms:

```python
KE = 0.5 * m * y_harm[:, 1]**2
PE = integrators.U_harmonic(y_harm[:, 0])
energy = KE + PE
plt.plot(t_harm, KE, 'r-', t_harm, PE, 'b-', t_harm, energy, 'k--')
```

Harmonic oscillator with larger timestep $h=0.01$

```python
m = 1
t, y = integrators.integrate_newton(t_max=500, force=integrators.F_harmonic, h=0.01, mass=m)
x, v = y.T
KE = 1/2 * m * v**2
PE = integrators.U_harmonic(x)
energy = KE + PE
```

```python
ax = plt.subplot(1,1,1)
ax.plot(t, KE, 'r-', t, PE, 'b-', t, energy, 'k--')
ax.set_xlim(0, 20)
ax.set_ylim(0, 0.7)
ax.set_xlabel("time")
ax.set_ylabel("energy")
ax.set_title("Euler: harmonic oscillator")
```

Note how energy conservation becomes worse when $h$ is increased.

For longer time scales, the energy just keeps on increasing — clearly, we are not solving the equations of motions in a physically meaningful way.

```python
ax.set_ylim(0, 80)
ax.set_xlim(0,500)
ax.figure
```

##### Quantify energy drift
Measure the relative departure from energy conservation

$$
\epsilon_E(t) = \log_{10}\left|\frac{E(t) - E(t=0)}{E(t=0)}\right|
$$

(By using the decadic logarithm, $-\epsilon_E$ is a direct measure of the precision.)

###### Harmonic oscillator
Analyze energy drift for the last harmonic oscillator example ($h=0.01$):

```python
energy_precision = np.log10(np.abs(energy/energy[0] - 1))
plt.plot(t, energy_precision)
plt.ylabel("log(relative error in $E$)")
plt.xlabel("time");
```

As we already know, with such a large time step, energy conservation is very bad with the *Euler* integrator: The relative (!) error quickly becomes 100% (after $t=200$).

##### Code re-use: energy analysis as a function
The module `integrators` contains the function `integrators.analyze_energies(t, y, U)` to automate the task of analysing the energy. You *must* provide the appropriate potential energy function `U`.

##### Problem: Make `analyze_energies()` work and analyze the power law oscillator
The code is *incomplete*.
1. Open `integrators.py` in your editor.
2. Complete the `kinetic_energy()` function (remember: $T_\text{kin} = \frac{1}{2} m v^2$)

Then `reload()` the module:

```python
# when working on the integrator module you need to reload() whenever you make changes
from importlib import reload   # for Python 3
reload(integrators);
```

and try it out for power law oscillator with a large time step $h = 0.01$. Can you find a value of $h$ that will give you energy precision to 4th decimal over 40 time units?

###### Energy stability analysis of the 6-th power oscillator
$x^6$ potential with $h=0.01$:

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    h=0.01)
```

```python
integrators.analyze_energies(t, y, integrators.U_power)
```

Trying 100 times smaller $h$ (takes 100 times longer!!)

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    h=0.0001)
integrators.analyze_energies(t, y, integrators.U_power)
```

*Reducing $h$ further does not seem to help and takes order of magnitude longer to run*. **We need a better integrator!**

## Runge-Kutta methods
Explicit RK methods propagate with the general form

$$
y_{n+1} = y_n + h \sum_{i=1}^s b_i k_i, \quad t_{n+1} = t_n + h
$$

where $s$ is the number of *stages* and the coefficients $a_{ij}$, $b_i$ and $c_i$ define the method.

\begin{align}
k_1 &= f(t_n, y_n)\\
k_2 &= f(t_n + c_2 h, y_n + h(a_{21} k_1))\\
k_3 &= f(t_n + c_3 h, y_n + h(a_{31} k_1 + a_{32} k_2))\\
 \vdots& \\
k_s &= f(t_n + c_s h, y_n + h(a_{s1} k_1 + a_{s2} k_2 + \cdots + a_{s,s-1} k_{s-1})).
\end{align}

Basic idea: improve estimate for the slope by using weighted averages of slopes over the interval.

### RK4
General purpose ODE solver.

$$
y_{n+1} = y_n + \frac{h}{6}\Big(k_1 + 2 k_2 + 2 k_3 + k_4\Big), \quad t_{n+1} = t_n + h
$$

and

\begin{align}
k_1 &= f(t_n, y_n)\\
k_2 &= f(t_n + \frac{1}{2} h, y_n + \frac{1}{2} h k_1)\\
k_3 &= f(t_n + \frac{1}{2} h, y_n + \frac{1}{2} h k_2)\\
k_4 &= f(t_n + h, y_n + h k_3)
\end{align}

`rk4` can obtain $\mathcal{O}(h^4)$ precision.

### RK2
`rk2` is simpler to derive and understand than `rk4` (which we will use more often).

Start with the exact solution

$$
\frac{dy}{dt} = f(t, y) \quad\text{and}\quad y(t) = \int f(t, y)dt\\
y_{n+1} = y_n + \int_{t_n}^{t_{n+1}} f(t, y)dt
$$

Expand $f(t, y)$ around *midpoint* $t_{n+1/2} = t + \frac{h}{2}$ of integration interval

\begin{gather}
f(t, y) \approx f(t_{n+1/2}, y_{n+1/2}) + (t - t_{n+1/2})\,  f^{(1)}(t_{n+1/2}) +
   \mathcal{O}(h^2)
\end{gather}

Integrate the *expansion* (Taylor series around midpoint):
\begin{align}
\int_{t_n}^{t_{n+1}} f(t, y) \, dt &\approx h\, f(t_{n+1/2}, y_{n+1/2}) + \mathcal{O}(h^3)\\
y_{n+1} &= y_n + h\, f(t_{n+1/2}, y_{n+1/2}) + \mathcal{O}(h^3)
\end{align}

I.e., use **slope at midpoint**! (The $\mathcal{O}(h^2)$ term in the integral is zero because $[(h/2)^2 - (h/2)^2] = 0$.)

But we don't have $y_{n+1/2}$ so we use Euler's rule to estimate it to $\mathcal{O}(h^2)$:

$$
y_{n+1/2} \approx y_n + \frac{1}{2} h\, f(t_n, y_n)
$$

and insert into $y_{n+1} = y_n + h\, f(t_{n+1/2}, y_{n+1/2})$ (so it is still $\mathcal{O}(h^3)$ accurate).

Altogether: **RK2** algorithm

$$
\mathbf{y}_{n+1} = \mathbf{y}_n + h \mathbf{k}_2, \quad t_{n+1} = t_n + h
$$

and

\begin{align}
\mathbf{k}_1 &= \mathbf{f}(t_n, \mathbf{y}_n)\\
\mathbf{k}_2 &= \mathbf{f}(t_n + \frac{1}{2} h, \mathbf{y}_n + \frac{1}{2} h \mathbf{k}_1)
\end{align}

with $\mathcal{O}(h^3)$ accuracy.

#### RK2 applied to the 1D harmonic oscillator (theory)

\begin{alignat}{3}
f^{(0)}(t, \mathbf{y}) &= y^{(1)},
    &\quad y^{(0)}_0 &= x_0,\\
f^{(1)}(t, \mathbf{y}) &=   - \frac{k}{m} y^{(0)},
    &\quad y^{(1)}_0 &= v_0.
\end{alignat}

with $k=1$; $x_0 = 0$ and $v_0 = +1$.

First step from $t=0$ to $t=h$:

\begin{align}
  y^{(0)}_{1} &=
    y^{(0)} _{0} + hf^{(0)} \Big(\frac{h}{2}, y^{(0)}_{0} + hk_{1} \Big)\\
    &= x_{0} + h\left[v_{0} + \frac{h}{2} \left(-\frac{k}{m} x_{0}\right)  \right]\\
    &= x_0 + v_0 h + \frac{1}{2} \left(-\frac{k}{m} x_{0}\right) h^2\\
  y^{(1)}_{1} &= y^{(1)}_{0} +
                h f^{(1)}\left(\frac{h}{2},
                     \mathbf{y}_{0} + \frac{h}{2} \mathbf{f}(0, \mathbf{y}_{0})\right) \\
             &= v_{0} + \frac{h}{m} \left[-k\left(x_{0} - \frac{h}{2} k x_{0}\right)\right]
\end{align}

Note that `rk2` contains a $h^2$ term in the position update.

#### RK2 applied in practice
Let's implement `rk2`

\begin{align}
\mathbf{y}_{n+1} &= \mathbf{y}_n + h \mathbf{k}_2\\
\mathbf{k}_1 &= \mathbf{f}(t_n, \mathbf{y}_n)\\
\mathbf{k}_2 &= \mathbf{f}(t_n + \frac{1}{2} h, \mathbf{y}_n + \frac{1}{2} h \mathbf{k}_1)
\end{align}

in our `integrators.py`.

```python
from importlib import reload
reload(integrators);
```

##### RK2 for the power potential

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    integrator=integrators.rk2, h=0.01)
```

###### phase-space portrait

```python
plt.plot(y[:, 0], m*y[:, 1])
```

###### energy drift

```python
integrators.analyze_energies(t, y, integrators.U_power)
```

#### RK4 applied in practice
Implement `rk4` in `integrator.py`:

\begin{align}
y_{n+1} &= y_n + \frac{h}{6}\Big(k_1 + 2 k_2 + 2 k_3 + k_4\Big)\\
k_1 &= f(t_n, y_n)\\
k_2 &= f(t_n + \frac{1}{2} h, y_n + \frac{1}{2} h k_1)\\
k_3 &= f(t_n + \frac{1}{2} h, y_n + \frac{1}{2} h k_2)\\
k_4 &= f(t_n + h, y_n + h k_3)
\end{align}

```python
reload(integrators)
```

    <module 'integrators_solution' from '/Volumes/Data/oliver/Documents/Teaching/ASU/CompPhys_PHY494/2018/PHY494-resources/10_ODEs/integrators_solution.py'>

and test the 6th power potential again. What level of energy precision can you obtain over 40 time units?

##### RK4 for the 6th power potential
Note that we can easily run `rk4` with a $h=0.01$!

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    integrator=integrators.rk4, h=0.01)
```

###### phase-space protrait

```python
plt.plot(y[:, 0], m*y[:, 1])
```

###### energy drift

```python
integrators.analyze_energies(t, y, integrators.U_power)
```

You should be able to get 8 - 11 decimals precision from `rk4` with a smaller $h$.

##### High precision RK4 for the 6th power potential
Try a 10 times smaller step $h=0.001$

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    integrator=integrators.rk4, h=0.001)
integrators.analyze_energies(t, y, integrators.U_power)
```

This looks like stable integration with an error of not worse than $10^{-12}$: *Looking very good!* (Especially when you compare it to *Euler* with $h=0.001$!)

The error spikes when the potential energy is dominant, which happens near the classical turning points where (in this case) the potential is the steepest and the forces are highest.

```python

```


```python
import numpy as np
import sympy
sympy.init_printing(use_unicode=True)
from sympy import *
from sympy.solvers import solve
from sympy.solvers.ode import dsolve
from IPython.display import display
from typing import Callable
from sympy.utilities.lambdify import lambdify, implemented_function

%matplotlib inline
import matplotlib.pyplot as plt

def simplified(exp, title=None):
    simp = simplify(exp)
    if simplified.LOG:
        if title: display(title,simp)
        else: display(simp)
    return simp
simplified.LOG = True

def firstOrderCondition(exp, var):
    diffExp = simplified(diff(exp, var))
    solutions = solve(diffExp, var)
    if firstOrderCondition.LOG:
        display(solutions)
    return solutions
firstOrderCondition.LOG = True

class Result(object):   # a class for holding results of calculations
    def __repr__(self): return self.__dict__.__repr__()
    def display(self):
        for k,v in sorted(self.__dict__.items()):
            display(k,v)
    def subs(self, params):
        ans = Result()
        for k,v in sorted(self.__dict__.items()):
            if hasattr(v,"subs"):
                ans.__dict__[k] = v.subs(params)
            else:
                ans.__dict__[k] = v
        return ans
```

# Symbolic calculations

```python
a,p,r,b,vmax,zmin,zmax,beta = symbols('a \\phi r z v_{\max} z_{\min} z_{\max} \\beta', positive=True,finite=True,real=True)
w,T,D,L,n,Supply = symbols('w T \\Delta \\ell n \\tau', positive=True,finite=True,real=True)
D,Supply,p
```

```python
def exactCostPerDay(T):
    return (a*p + w*b*( (1+r)**T - 1 )) / T

def approxCostPerDay(T):
    return a*p/T + w*b*r

def symmetricLifetime(w):
    return w**2/4/L

def asymmetricLifetime(w):
    return w / D

uniformPDF = Piecewise( (1 / zmax , b<zmax), (0, True) )
powerlawPDF =  Piecewise( (0 , b<zmin), (zmin / b**2, True) )

display(sympy.integrate(uniformPDF, (b, 0, sympy.oo)))  # should be 1
display(sympy.integrate(powerlawPDF, (b, 0, sympy.oo))) # should be 1
display(sympy.integrate(b*uniformPDF, (b, 0, sympy.oo)))  # should be zmax/2
display(sympy.integrate(b*powerlawPDF, (b, 0, sympy.oo))) # should be infinity!
```

```python
params = {
    L: 10,           # total transfers per day
    D: 6,            # delta transfers per day
    beta: 0.01,      # value / transfer-size
    r: 4/100/365,    # interest rate per day
    a: 1.1,          # records per reset tx
    Supply: 288000,  # records per day
    zmin: 0.001,     # min transfer size (for power law distribution)
    zmax: 1,         # max transfer size (for uniform distribution)
}
```

```python
x,y,t=symbols("x y t")
intt = 1 + simplified(sympy.integrate(t(y)/(x-y)**2, (y,0,x-1)) + sympy.integrate(t(y)/(x-y)**2, (y,x+1,w)))
intt.diff(x)
```

```python
intu = sympy.integrate(t(y), (y, sympy.Max(0,x-1),sympy.Min(w,x+1)))
intu, intu.diff(x)
```

```python
intr = sympy.integrate(t(y), (y, x-1, x+1))
intr, intr.diff(x)
```

```python
from sympy import *
x = symbols("x")
f = Function("f")
f_ = Derivative(f(x),x)
dsolve(f_ - f(x+1), f(x))
```

```python
### See solution in https://stackoverflow.com/a/46755444/827927
```


# Pulse stream recovery for arbitrary pulse shapes (noiseless)

In the previous notebook, we saw an example of recovering the pulse localtions and amplitudes for a signal of the following structure, after it has gone through the necessary sampling kernel and sampling:

\begin{align}
y(t) = \sum_{n\in\mathbb{Z}}\sum_{k=0}^{K-1}c_k \hspace{0.05cm} \delta(t-t_k-nT).  \nonumber
\end{align}

```python
import numpy as np
import sys
sys.path.append('..')
import plot_settings
from frius import create_pulse_param, sample_ideal_project, estimate_fourier_coeff, compute_ann_filt, estimate_time_param, estimate_amplitudes
```

# 1. Extension to arbitrary pulse shape

We now consider the situation in which our pulse shape may not be a Dirac. Similarly, our sampling kernel may not have a _flat_ response over its frequency support. This implies that we can "group" the pulse and the sampling kernel into a single function, which we will still denote by $h(t)$:

\begin{align}
y(t) = \sum_{n\in\mathbb{Z}}\sum_{k=0}^{K-1}c_k \hspace{0.05cm} h(t-t_k-nT). \nonumber
\end{align}

Our measured samples are then obtained from this signal which combines our pulse _and_ the effect of our sampling kernel.

Consequently, our recovery process requires knowledge about $h(t)$. In the previous notebook, we saw the following steps for recoverying the parameters of a _Dirac_ stream after it has gone through an ideal sampling kernel with a flat response:

1. With the samples, estimate the Fourier coefficients with the DFT.
2. From the Fourier coefficients, obtain the annihilating filter, i.e. the mask with zeros that coincide with the zeros of the Dirac stream.
3. Find the $K$ zeros of the mask corresponding to the annihilating filter. This will yield the locations $\{t_k\}_{k=0}^{K-1}$.
4. With the locations and measured samples, solve a least-squares for the pulse amplitudes $\{c_k\}_{k=0}^{K-1}$.

Please refer to [1] for a more detailed mathematical description.

For an arbitrary pulse shape, we need to remove the combined pulse and sampling effect; otherwise it will not be possible to determine an annihilating filter from the Fourier coefficients! This entails an additional step in our "mapping" phase before obtaining the annihilating filter:

1. With the samples, estimate the Fourier coefficients with the DFT.
2. Remove the effect of the pulse and/or sampling kernel from the estimated Fourier coefficients.
2. From the resulting Fourier coefficients, obtain the annihilating filter, i.e. the mask with zeros that coincide with the zeros of the Dirac stream.
3. Find the $K$ zeros of the mask corresponding to the annihilating filter. This will yield the locations $\{t_k\}_{k=0}^{K-1}$.
4. With the locations and measured samples, solve a least-squares for the pulse amplitudes $\{c_k\}_{k=0}^{K-1}$.

This additional step is often called **_equalization_**, as we are trying to "equalize" or balance out the level across frequencies. In the [Appendix](#app), we provide a more mathematical reasoning as to why this equalization step is necessary.

# 2. Example with code

We now consider a small example where either our pulse has a Gaussian shape and our sampling kernel is still an ideal lowpass filter with bandwidth $B=2K$.

We first define a function to compute the frequency response of our Gaussian shape [2]:

```python
def gaus_ft(freqs, alpha=1e4):
    return np.sqrt(np.pi/alpha) * np.exp( (np.pi*freqs)*(np.pi*freqs)/alpha )
```

We can then pass this function to our sampling utility.

```python
K = 6
period = 1
ck, tk = create_pulse_param(K=K, period=period)
y_samp, t_samp, fs_ind_base = sample_ideal_project(ck, tk, period=period, H=gaus_ft, viz=True, figsize=(10,5))
```

We first try to perform the recovery **_without_** the equalization step.

```python
freqs = fs_ind_base/period
fs_coeff_hat = estimate_fourier_coeff(y_samp, t_samp)
ann_filt = compute_ann_filt(fs_coeff_hat, K)
tk_hat = estimate_time_param(ann_filt, period)
tk_hat = np.sort(tk_hat)                                     # sort for comparison as `tk` are sorted
ck_hat = estimate_amplitudes(fs_coeff_hat, freqs, tk_hat, period)

print("||tk - tk_hat||_2 = %f " % np.linalg.norm(tk-tk_hat))
print("||ck - ck_hat||_2 = %f " % np.linalg.norm(ck-ck_hat))
```

    ||tk - tk_hat||_2 = 0.006075
    ||ck - ck_hat||_2 = 3.632765

We can observe that we are not able to recover the pulse locations precisely, and this also affects the recovery of the amplitudes! We now perform the recovery **_with_** the equalization step.

```python
freqs = fs_ind_base/period
fs_coeff_hat = estimate_fourier_coeff(y_samp, t_samp)
fs_coeff_hat /= gaus_ft(freqs)                               # equalization step
ann_filt = compute_ann_filt(fs_coeff_hat, K)
tk_hat = estimate_time_param(ann_filt, period)
tk_hat = np.sort(tk_hat)
ck_hat = estimate_amplitudes(fs_coeff_hat, freqs, tk_hat, period)

print("||tk - tk_hat||_2 = %f " % np.linalg.norm(tk-tk_hat))
print("||ck - ck_hat||_2 = %f " % np.linalg.norm(ck-ck_hat))
```

    ||tk - tk_hat||_2 = 0.000000
    ||ck - ck_hat||_2 = 0.000000

Perfect recovery! We could also pass the coefficients for equalization `gaus_ft(freqs)` directly to `estimate_fourier_coeff` as such:

```python
fs_coeff_hat = estimate_fourier_coeff(y_samp, t_samp, H=gaus_ft(freqs))
```

That's it for this notebook! If you would like to understand _why_ this equalization step is necessary, please check out the Appendix below.

In the next notebook, we investigate the performance of the pulse stream recovery procedure in the presence of noise and discuss techniques for coping with noise.

# Appendix: Why equalization? And what is meant by annihilation?

Mathematically, the equalization step can be written as:

\begin{align}
\hat{x}[m] = \dfrac{\hat{y}[m]}{\hat{h}(m/T)}, \text{ for } m\in[-K, K], \nonumber
\end{align}

where $\hat{y}[m]$ are the estimated Fourier coefficients obtained from applying the DFT to the measured samples, $K$ is the number of pulses which also means we require $N=2K+1$ samples (and Fourier coefficients) for the recovery, $\hat{h}(f)$ is the **_known_** frequency response of our pulse and sampling kernel, and (m/T) gives us the corresponding frequency of the $m$-th coefficient. The _equalized_ Fourier coefficients are given by $\hat{x}[m]$ and these values are used to obtain the annihilating filter.

If we consider the Fourier coefficients of the pulse stream, by taking the Fourier Series (FS) of one period:

\begin{equation}
y(t) = \sum_{k=0}^{K-1}c_k\cdot h(t-t_k) \xrightarrow{FS} \hat{y}[m] = \hat{h}(m/T) \sum_{k=0}^{K-1}c_k\cdot\exp(-j2\pi m t_k/T),
\end{equation}

we can observe that the only difference between the Fourier coefficients of an arbitrary pulse stream and a Dirac stream is the multiplication by a **_constant_** factor, i.e. constant across all $K$ pulses but potentially unique across all coefficient indices. Moreover, this constant factor is given by the frequency response of our pulse and sampling kernel at the corresponding frequency value.

Therefore, we can view equalization as an operation to "transform" our Fourier coefficients corresponding to an arbitrary pulse stream to Fourier coefficients corresponding to a Dirac stream. This operation is necessary because pulse stream Fourier coefficients are not necessarily "annihilate-able", i.e. we may not be able to determine our desired mask from these coefficients. However, Dirac stream Fourier coefficients are certainly annihilate-able. To see this, we first consider the Fourier coefficients for a single Dirac:

\begin{align}
\hat{x}_k[m] &= c_k \hspace{0.05cm} \exp(-j2\pi m t_k/T) \nonumber\\
&= c_k\hspace{0.05cm} u_k^m \nonumber\\
\Rightarrow \mathbf{\hat{x}}_k &= c_k  [\ldots, u_k^{-1}, 1, u_k, u_k^{2}, \ldots], \nonumber
\end{align}

where we set $u_k = \exp(-j2\pi t_k/T)$ for notational convenience.

For $ \mathbf{\hat{x}}_k=\{\hat{x}_k[m]\}_{m\in\mathbb{Z}} $, we denote by $\mathbf{a}_{k'} = \{a_{k'}[m]\}_{m=0}^{1} $ the two-tap filter that will annihilate $ \mathbf{\hat{x}}_k $. So what do we mean by annihilate? The term "annihilation" arises from the fact that the convolution between the Fourier coefficients and this filter results in  zero for all $ m $.

Moreover, $\mathbf{a}_{k'} $ must have a zero at $ u_k $ for this annihilation to occur, suggesting the following coefficients for our two-tap filter:

\begin{align}
\mathbf{a}_{k'} &= \begin{bmatrix} a_{k'}[0],\hspace{0.2cm} a_{k'}[1] \end{bmatrix}  \nonumber\\
&= [1, -u_k]. \nonumber
\end{align}

By writing out the convolution between $ \mathbf{\hat{x}}_k $ and our assumed $ \mathbf{a}_{k'} $, we can verify that the annihilation is fulfilled:

\begin{align}
&\vdots\nonumber\\
z_k[0] &= \begin{bmatrix} a_{k'}[1],\hspace{0.2cm} a_{k'}[0]  \end{bmatrix} \begin{bmatrix} \hat{x}_k[-1]\\ \hat{x}_k[0] \end{bmatrix} = c_k(-1 + 1) = 0  \nonumber\\
z_k[1] &= \begin{bmatrix} a_{k'}[1],\hspace{0.2cm} a_{k'}[0]\end{bmatrix} \begin{bmatrix} \hat{x}_k[0]\\ \hat{x}_k[1] \end{bmatrix} = c_k(-u_k + u_k) = 0 \nonumber\\
&\vdots&\nonumber\\
z_k[m] &= \begin{bmatrix} a_{k'}[1],\hspace{0.2cm} a_{k'}[0] \end{bmatrix} \begin{bmatrix} \hat{x}_k[\mathrel{{m}{-}{1}}]\\ \hat{x}_k[m] \end{bmatrix} = c_k(-u_k^m + u_k^m) = 0 \nonumber\\
&\vdots\nonumber
\end{align}

or in short:

\begin{equation}
\mathbf{z}_k = \mathbf{a}_{k'}  \ast \mathbf{\hat{x}}_k = \mathbf{0} \nonumber.
\end{equation}

From this annihilation, we can then observe the necessity for both the equalization **_and_** uniformly sampled coefficients. If we did not remove the effect of the pulse and and sampling kernel, we may have distinct weightings for adjacent Fourier coefficients and the annihilation would not hold. For example, if we did not equalize and computed the convolution for all $ m $:

\begin{align}
z_k[m] &= \begin{bmatrix} a_{k'}[1],\hspace{0.2cm} a_{k'}[0]  \end{bmatrix} \begin{bmatrix} \hat{h}_T[(\mathrel{{m}{-}{1}})/T] \hspace{0.05cm} \hat{x}_k[\mathrel{{m}{-}{1}}] \nonumber \\
\hat{h}_T[m/T] \hspace{0.05cm} \hat{x}_k[m] \end{bmatrix} \nonumber \\
&= c_k\Big(-\hat{h}_T[(\mathrel{{m}{-}{1}})/T] \hspace{0.05cm} u_k^m + \hat{h}_T[m/T]\hspace{0.05cm} u_k^m\Big)\nonumber\\
&= c_k u_k^m \underbrace{\Big(-\hat{h}_T[(\mathrel{{m}{-}{1}})/T] + \hat{h}_T[m/T]\hspace{0.05cm} \Big)}_{\Delta \hat{h}_T[m]}, \nonumber
\end{align}

we cannot be certain that $ \Delta \hat{h}_T[m]=0 $ so that the convolution is 0 for all $m$, which is why it is necessary to equalize. Furthermore, we require uniform samples so that adjacent coefficients of $ \mathbf{\hat{x}}_k $ have the same geometric ratio $ (\hat{x}_k[m]/\hat{x}_k[\mathrel{{m}{-}{1}}]=u_k)$. For $ u_k=\exp(-j2\pi f t_k) ,$ i.e. complex exponentials, this amounts to adjacent FS coefficients having the same phase difference, which is guaranteed by using uniform time-domain samples to estimate the FS coefficients with the DFT.

Now if we return to the sum of $ K $ complex exponentials $ \mathbf{\hat{x}}=\sum_{k=0}^{K-1}\mathbf{\hat{x}}_k$, the corresponding annihilating filter will be the convolution of $ K $ filters of the form $ \mathbf{a}_{k'} =[1, -u_k]$:

\begin{align}
\mathbf{a} &= \mathbf{a}_0 \ast \mathbf{a}_1 \ast \cdots \ast\mathbf{a}_{\mathrel{{K}{-}{1}}}.  \nonumber
\end{align}

Writing out the convolution yields:

\begin{align}
\mathbf{a} \ast \mathbf{\hat{x}} &= (\mathbf{a}_0 \ast \mathbf{a}_1 \ast \cdots \ast \mathbf{a}_{\mathrel{{K}{-}{1}}}) \ast (\sum_{k=0}^{K-1} \mathbf{\hat{x}}_k) \nonumber\\
&= \sum_{k=0}^{K-1} (\mathbf{a}_0 \ast \mathbf{a}_1 \ast \cdots \ast \mathbf{a}_{\mathrel{{K}{-}{1}}}) \ast\mathbf{\hat{x}}_k \nonumber\\
&= \sum_{k=0}^{K-1} (\mathbf{a}_0 \ast \mathbf{a}_1 \ast \cdots \ast \mathbf{a}_{\mathrel{{k'}{-}{1}}} \ast \mathbf{a}_{\mathrel{{k'}{+}{1}}} \ast\cdots \ast \mathbf{a}_{\mathrel{{K}{-}{1}}}) \ast \mathbf{a}_{k'}\ast\mathbf{\hat{x}}_k \nonumber\\
&= \sum_{k=0}^{K-1} (\mathbf{a}_0 \ast \mathbf{a}_1 \ast \cdots \ast \mathbf{a}_{\mathrel{{k'}{-}{1}}} \ast \mathbf{a}_{\mathrel{{k'}{+}{1}}} \ast\cdots \ast\mathbf{a}_{\mathrel{{K}{-}{1}}}) \ast \mathbf{0} \nonumber\\
&\Rightarrow \mathbf{a} \ast \mathbf{\hat{x}}  = \mathbf{0},\nonumber
\end{align}

where we use the linearity, associative, and commutative properties of convolution.

In summary, we require the annihilation step in order to "transform" our Fourier coefficients corresponding to an arbitrary pulse stream to Fourier coefficients corresponding to a Dirac stream, i.e. **_uniform samples of a sum of sinusoids_**. Moreover, as shown above, we can obtain an annihilating filter from uniform samples of a sum of sinusoids. In [1], it is discussed how we can then obtain the pulse parameters from the annihilating filter. Nevertheless, as can be observed above, the locations of the pulse parameters appear as zeros in the annihilating filter!

# References

    [1] M. Vetterli, P. Marziliano and T. Blu, "Sampling signals with finite rate of innovation," in IEEE Transactions on Signal Processing, vol. 50, no. 6, pp. 1417-1428, Jun 2002.
    [2] Fourier Transform of common functions (Wikipedia): https://en.wikipedia.org/wiki/Fourier_transform#Square-integrable_functions,_one-dimensional


# 3.2.3 Multiple Regression From Simple Univariate Regression

Suppose we have a *univariate* (p = 1) model with no intercept (3.23):
$$Y=X\beta+\varepsilon$$

The least squares estimate and residuals are (3.24):
$$
\begin{equation}
\hat{\beta} = \cfrac{\sum_1^N {x_iy_i}}{\sum_1^N {x_i^2}} \\
r_i = y_i - x_i\hat{\beta}
\end{equation}
$$

With the inner product:
$$
\begin{equation}
\hat{\beta} = \cfrac{\langle \mathbf{x}, \mathbf{y} \rangle}{\langle \mathbf{x}, \mathbf{x}\rangle}\\
\mathbf{r} = \mathbf{y} - \mathbf{x}\hat{\beta}
\end{equation}
$$

Suppose that the columns of the matrix **X** are orthogonal; that is $\langle \mathbf{x}_j, \mathbf{x}_k \rangle = 0$
then it is easy to check that $\hat{\beta_j} = \langle \mathbf{x}_j, \mathbf{y} \rangle / \langle \mathbf{x}_j, \mathbf{x}_j \rangle$, i.e the inputs have no effect on each other's parameter estimates.

Suppose next that we have an intercept and a single input x (3.27):
$$\hat{B}_1 = \cfrac{\langle \mathbf{x} - \overline{x}\mathbf{1}, \mathbf{y} \rangle}{ \langle \mathbf{x} - \overline{x}\mathbf{1}, \mathbf{x} - \overline{x}\mathbf{1} \rangle}$$

We can view the estimate as the result of two simple regression:

1. Regress **x** on **1** to produce the residual $\mathbf{z} = \mathbf{x} - \overline{x}\mathbf{1}$

2. Regress **y** on the residual **z** to give the coefficient $\hat{\beta}_1$.

Regress **b** on **a** means $\hat{\gamma}=\langle \mathbf{a},\mathbf{b} \rangle / \langle \mathbf{a}, \mathbf{a}\rangle$ and the residual vector $\mathbf{b} - \hat{\gamma}\mathbf{a}$.

This recipe generalizes to the case of *p* inputs, as shown in Algorithm 3.1.

**Algorithm 3.1 Regression by Successive Orthogonalization**
1. $\mathbf{z}_0 = \mathbf{x}_0 = \mathbf{1}$

2. For $j = 1, 2, \cdots, p$

   * Regress $\mathbf{x}_j$ on $\mathbf{z}_0,...,\mathbf{z}_{j - 1}$ to produce $\hat{\gamma}_{lj}=\langle \mathbf{z}_l, \mathbf{x}_j \rangle / \langle \mathbf{z}_l,\mathbf{z}_l \rangle$ $l=0,\cdots,j-1$, and residualt vector $\mathbf{z}_j=\mathbf{x}_j - \sum_{k=0}^{j-1} \hat{\gamma}_{kj}\mathbf{z}_k$

3. Regress $\mathbf{y}$ on the residual $\mathbf{z}_p$ to give the estimate $\hat{\beta}_p$

```python
import numpy as np
import pandas as pd
from scipy import stats, linalg

df = pd.read_csv('../data/prostate/prostate.data', delimiter='\t', index_col=0)
mask_train = df.pop('train')
df_y = df.pop('lpsa')
df = df.apply(stats.zscore)

def orthogonalize(X):
    p = X.shape[1]
    G = np.eye(p)
    Z = X.copy()
    for j in range(1, p):
        for l in range(j):
            G[l, j] = np.dot(Z[:, l], X[:, j]) / np.dot(Z[:, l], Z[:, l])
        for k in range(j):
            Z[:, j] -= G[k, j] * Z[:, k]
    return Z, G
```

The result of this algorithm is (3.28):

$$\hat{\beta}_p=\cfrac{\langle \mathbf{z}_p, \mathbf{y} \rangle}{\langle \mathbf{z}_p,\mathbf{z}_p \rangle}$$

If $\mathbf{x}_p$ is highly correlated with some of the other $\mathbf{x}_k$'s the residual vector $\mathbf{x}_p$ will be close to zero, and from (3.28) the coefficient $\hat{\beta}_p$ will be unstable.

From (3.28) we also obtain an alternative formula for the variance estimates, (3.29):

$$Var(\hat{\beta}_p) = \cfrac{\sigma^2}{\langle \mathbf{z}_p, \mathbf{z}_p \rangle}=\cfrac{\sigma^2}{||\mathbf{z}_p||^2}  $$

On other words, the precision with which we can estimate $\hat{\beta}_p$ depends on the lengths of the residual vector $\mathbf{z}_p$;

Algorithm 3.1 is known as the *Gram–Schmidt* procedure for multiple regression. We can represent step 2 of Algorithm 3.1 in matrix form (3.30):

$$\mathbf{X}=\mathbf{Z\Gamma}$$

where $\mathbf{Z}$ has as columns the $z_j$ (in order), and $\mathbf{\Gamma}$ is the upper triangular matrix
with entries $\hat{\gamma}_{kj}$. Introducing the diagonal matrix $\mathbf{D}$ with $D_{jj}=||z_j||$, we get (3.31):

$$\mathbf{X}=\mathbf{Z}\mathbf{D}^{-1}\mathbf{D}\mathbf{\Gamma}=\mathbf{QR}$$

the so-called QR decomposition of $\mathbf{X}$. Here $\mathbf{Q}$ is an N × (p +1) orthogonal
matrix, $\mathbf{Q}^T\mathbf{Q} = \mathbf{I}$, and **R** is a (p + 1) × (p + 1) upper triangular matrix.

The least squares solution is given by:

$$
\hat{\beta}=\mathbf{R}^{-1}\mathbf{Q}^T\mathbf{y}
$$

*Proof*:
$$
\begin{equation}
\mathbf{X}^T\mathbf{y}=\mathbf{X}^T\mathbf{X}\hat{\beta}\\
\mathbf{R}^T\mathbf{Q}^T\mathbf{y}=\mathbf{R}^T\mathbf{Q}^T\mathbf{Q}\mathbf{R}\hat{\beta}\\
\mathbf{R}^T\mathbf{Q}^T\mathbf{y}=\mathbf{R}^T\mathbf{R}\hat{\beta}\\
\mathbf{Q}^T\mathbf{y}=\mathbf{R}\hat{\beta}\\
\end{equation}
$$
And the predicted training values:

$$
\hat{\mathbf{y}}=\mathbf{QQ}^T\mathbf{y}
$$

*Proof*:

$$
\begin{align}
\hat{\mathbf{y}}&=\mathbf{X}\hat{\beta}\\
&=\mathbf{QR}\mathbf{R}^{-1}\mathbf{Q}^T\mathbf{y}\\
&=\mathbf{QQ}^T\mathbf{y}
\end{align}
$$

We can obtain from it not just $\hat{\beta}_p$, but also the entire multiple least squares fit.

*Proof*:
We can easily derive that:
$$
\mathbf{R}\hat{\beta}=\mathbf{Q}^T\mathbf{y}
$$

which can be expanded into:
$$
\begin{equation}
\begin{bmatrix}
    R_{0 0} & R_{02}  & \dots   & R_{0p} \\
    0       & R_{11}  & \dots   & R_{1p} \\
    \vdots  & \vdots  & \ddots  & \vdots \\
    0       & 0       & \dots   & R_{pp}
\end{bmatrix}
\begin{bmatrix}
    \hat{\beta_0} \\
    \hat{\beta_1} \\
    \vdots        \\
    \hat{\beta_p}
\end{bmatrix}
=
\begin{bmatrix}
    {Q_{0}}^T\mathbf{y} \\
    {Q_{1}}^T\mathbf{y} \\
    \vdots        \\
    {Q_{p}}^T\mathbf{y}
\end{bmatrix}
\end{equation}
$$

Now by applying the backward substitution it is possible to obtain the entire multiple least squares fit. For example to find the $\hat{\beta}_p$:
$$
\begin{equation}
R_{pp}\hat{\beta}_p = {Q_{p}}^T\mathbf{y}\\
\hat{\beta}_p = \cfrac{\langle Q_p, \mathbf{y} \rangle}{R_{pp}}=\cfrac{\langle \mathbf{z}_p, \mathbf{y} \rangle}{\langle \mathbf{z}_p,\mathbf{z}_p \rangle}
\end{equation}
$$

```python
def least_squares_qr(data_x, data_y):
    X = np.c_[np.ones((len(data_x), 1)), data_x]
    Z, G = orthogonalize(X)

    D = linalg.norm(Z, axis=0)
    Q = Z / D
    R = np.diag(D) @ G
    beta = linalg.solve_triangular(R, Q.T @ data_y)
    return beta

beta = least_squares_qr(df[mask_train == 'T'].as_matrix(), df_y[mask_train == 'T'].as_matrix())
print ("Coefficient: ", beta)
```

    Coefficient:  [ 2.46493292  0.67601634  0.26169361 -0.14073374  0.20906052  0.30362332
     -0.28700184 -0.02119493  0.26557614]


## Week 3 MA544
---
**Objectives and Plan**

1. Pseudo-inverse (Perron-Frobenius inverse) Properties
1. Linear Systems of Equations and Gaussian Elimination with pivoting
1. LU Decomposition of A
1. QR Decomposition of a matrix
1. Iterative solution of Linear Systems

```python

#IMPORT
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
%matplotlib inline

from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets

## Set a seed for the random number generator
np.random.seed(100)
```

## Linear System of Equations
---
Consider the following system of $m$ linear equations in $n$ variables.
\begin{align}
a_{11} x_1 + a_{12} x_2  + \cdots + a_{1n} x_n  &= b_1 \\
a_{21} x_1 + a_{22} x_2  + \cdots + a_{2n} x_n  &= b_2 \\
 \vdots \qquad \qquad   & \ \\
a_{m1} x_1 + a_{m2} x_2  + \cdots + a_{mn} x_n  &= b_ m,
\end{align}

-  The solution of a linear system represents the **point of intersection of hyperplanes**

$$
 x_1 \begin{pmatrix}a_{11}\\a_{21}\\ \vdots \\a_{m1}\end{pmatrix} +
 x_2 \begin{pmatrix}a_{12}\\a_{22}\\ \vdots \\a_{m2}\end{pmatrix} +
 \cdots +
 x_n \begin{pmatrix}a_{1n}\\a_{2n}\\ \vdots \\a_{mn}\end{pmatrix}
 =
 \begin{pmatrix}b_1\\b_2\\ \vdots \\b_m\end{pmatrix}
 $$

-  The solution also represents the **linear coding of the columns** of a matrix $A$ to get a vector $b$ in the column space ($\mathcal{R}(A)$).

- The system could be represented in a compact form as $Ax = b$, where
 $$
 A=
\begin{pmatrix}
a_{11} & a_{12} & \cdots & a_{1n} \\
a_{21} & a_{22} & \cdots & a_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
a_{m1} & a_{m2} & \cdots & a_{mn}
\end{pmatrix},\quad
\mathbf{x}=
\begin{pmatrix}
x_1 \\
x_2 \\
\vdots \\
x_n
\end{pmatrix},\quad
\mathbf{b}=
\begin{pmatrix}
b_1 \\
b_2 \\
\vdots \\
b_m
\end{pmatrix}
$$

>- The system is called...
>>- overdetermined when $m > n$.
>>- underdetermined when $ m < n$.

## Gaussian elimination with scaled-row partial pivoting
---
>- What is the need for pivoting?
>- How can you find a factorization of the matrix by using the followign code.
>- How can you modify the code to find the determinant of a matrix?
>- How can you modify the following code to solve a linear system?
>- How can you modify the code to find the inverse of a matrix?

```python
## Gaussian Elimination: Scaled Row Pivoting
## This function is based on the pseudo-code on page-148 in the Text by Kincaid and Cheney
def GE_rsp(A):
    '''
    This function returns the P'LU factorization of a square matrix A
    by scaled row partial pivoting.
    In place of returning L and U, elements of modified A are used to hold values of L and U.
    '''
    m,n = A.shape

    L = np.eye(n) # Not being used
    U = np.zeros_like(A) # Not being used
    if m !=n:
        sys.exit("This function needs a square matrix as an input.")

    # The initial ordering of rows
    p = list(range(n))

    # Scaling vector: absolute maximum elements of each row
    s = np.max(np.abs(A), axis=1)

    print("Scaling Vector: ",s)

    # Start the k-1 passes of Guassian Elimination on A
    for k in range(n-1):

        print("\n PASS {}: \n".format(k+1), A)
        # Find the pivot element and interchange the rows
        pivot_index = k + np.argmax(np.abs(A[p[k:], k])/s[p[k:]])

        # Interchange element in the permutation vector
        if pivot_index !=k:
            temp = p[k]
            p[k]=p[pivot_index]
            p[pivot_index] = temp
            print("permutation vector: ",p)

        print("\n Pivot Element: {0:.2f} \n".format(A[p[k],k]))
        if np.abs(A[p[k],k]) < 10**(-20):
             sys.exit("ERROR!! Provided matrix is singular.")

        # For the k-th pivot row Perform the Gaussian elimination on the following rows
        for i in range(k+1, n):
            # Find the multiplier
            z = A[p[i],k]/A[p[k],k]

            #Save z in A itself. You can save this in L also
            A[p[i],k] = z

            #Elimination operation: Changes all elements in a row simultaneously
            ##
            A[p[i],k+1:] -= z*A[p[k],k+1:]
    return A, p
```

```python
## Example on page number 146 (Kincaid Cheney).
## Example solved manually in class
A = np.array([[2, 3, -6], [1,-6,8], [3, -2, 1]], dtype=float)
print("\n Given A: \n ",A)
A,p =GE_rsp(A)
print("\n After Gaussian Elimination with RSPP: \n", A)
print("\n The permutation Vector is: \n", p)

```

     Given A:
      [[ 2.  3. -6.]
     [ 1. -6.  8.]
     [ 3. -2.  1.]]
    Scaling Vector:  [6. 8. 3.]

     PASS 1:
     [[ 2.  3. -6.]
     [ 1. -6.  8.]
     [ 3. -2.  1.]]
    permutation vector:  [2, 1, 0]

     Pivot Element: 3.00

     PASS 2:
     [[ 0.66666667  4.33333333 -6.66666667]
     [ 0.33333333 -5.33333333  7.66666667]
     [ 3.         -2.          1.        ]]
    permutation vector:  [2, 0, 1]

     Pivot Element: 4.33

     After Gaussian Elimination with RSPP:
     [[ 0.66666667  4.33333333 -6.66666667]
     [ 0.33333333 -1.23076923 -0.53846154]
     [ 3.         -2.          1.        ]]

     The permutation Vector is:
     [2, 0, 1]

### LU Factorization
---
One can use the Gaussian elimination to decompose the matrix A as follows
$$
P A = L U,\text{ or }\ A = P^T L U;
$$
where $P$ is a permutation matrix, $L$ is a unit lower-triangular matrix and $U$ is an upper triangular matrix.

```python
print("\n Upper triangular, U:\n ", np.triu(A[p,:]))
print("\n Lower triangular, L:\n", np.tril(A[p,:], -1)+np.eye(3))
print("The Permutation Matrix, P:\n",np.eye(3)[p,:])
```

     Upper triangular, U:
      [[ 3.         -2.          1.        ]
     [ 0.          4.33333333 -6.66666667]
     [ 0.          0.         -0.53846154]]

     Lower triangular, L:
     [[ 1.          0.          0.        ]
     [ 0.66666667  1.          0.        ]
     [ 0.33333333 -1.23076923  1.        ]]
    The Permutation Matrix, P:
     [[ 0.  0.  1.]
     [ 1.  0.  0.]
     [ 0.  1.  0.]]

## Iterative Methods for Linear Systems
---

### Jacobi Method

```python
# You can modify this code to answer the following
'''
Jacobi's iteration method for solving the system of equations Ax=b.
p0 is the initialization for the iteration.
'''
def jacobi(A, b, p0, tol, maxIter=100):
    n=len(A)
    p = p0

    for k in range(maxIter):
        p_old = p.copy() # In python assignment is not the same as copy

        # Update every component of iterant p
        for i in range(n):
            sumi = b[i];
            for j in range(n):
                if i==j: # Diagonal elements are not included in Jacobi
                    continue;
                sumi = sumi - A[i,j] * p_old[j]
            p[i] = sumi/A[i,i]

        rel_error = np.linalg.norm(p-p_old)/n
        # print("Relative error in iteration", k+1,":",rel_error)
        if rel_error<tol:
            print("TOLERANCE MET BEFORE MAX-ITERATION")
            break
    return p;
```

```python
# Example System
A = np.array([[10, -1, 2, 0],
              [-1, 11, -1, 3],
              [2, -1, 10, -1],
              [0, 3, -1, 8]],dtype=float)
b = np.array([6, 25, -11, 15],dtype=float)
```

```python

```


# Fitting a line to data - a full tutorial

_Boris Leistedt, September 2017_

This notebook is available at [this location on Github](https://github.com/ixkael/Prob-tools/blob/master/notebooks/Fitting%20a%20line%20to%20data%20-%20a%20full%20tutorial.ipynb).

This notebook It assumes some basic knowledge about Bayesian inference and data analysis. It is accompanied with a set of slides.

```python
%matplotlib inline
%config IPython.matplotlib.backend = 'retina'
%config InlineBackend.figure_format = 'retina'

from IPython.display import HTML
```

This is a tutorial.

Don't be like

```python
HTML('')
```

Play with the code! Try and do the exercises.

Please interrupt me if you are lost or if you disagree with what I say.

All questions are welcome, especially the ones that you find "simple" (they are often very good, and not simple!)

If you haven't done it, install those packages using conda and/or pip:

``conda install numpy scipy pandas matplotlib jupyter pip``

``pip install emcee corner``

start a jupyter kernel: ``jupyter notebook``

and open a copy of this notebook.

```python
import matplotlib
import matplotlib.pyplot as plt
from cycler import cycler
matplotlib.rc("font", family="serif", size=14)
matplotlib.rc("figure", figsize="10, 5")
colors = ['k', 'c', 'm', 'y']
matplotlib.rc('axes', prop_cycle=cycler("color", colors))

import scipy.optimize
import numpy as np
```

# Why Bayesian inference?

Constrain model parameters with data.

```python
HTML('')
```

# Fitting a line to data: basic setup

Our model will consist of a set of $N$ i.i.d. observations, including: a coordinate (fixed parameter) $x_i$, a noise level  $\sigma_i$, and observed variable $\hat{y}_i$ drawn from a Gaussian with mean $mx_i + b$ and variance $\sigma^2_i$, i.e. $\hat{y}_i \sim \mathcal{N}(mx_i+b;\sigma_i^2)$.

Let's generate a model:

```python
slope_true = np.random.uniform(0, 1)
intercept_true = np.random.uniform(0, 1)
print('Slopes:', slope_true)
print('Intercepts:', intercept_true)
# This notebook is ready for you to play with 2+ components and more complicated models.
```

    Slopes: 0.15916209709724805
    Intercepts: 0.2638003026824447

Let's generate some data drawn from that model:

```python
ndatapoints = 20
xis_true = np.random.uniform(0, 1, ndatapoints)
x_grid = np.linspace(0, 1, 100)

def model_linear(xs, slope, intercept): return xs * slope + intercept
yis_true = model_linear(xis_true, slope_true, intercept_true)
```

```python
sigma_yis = np.repeat(0.1, ndatapoints) * np.random.uniform(0.5, 2.0, ndatapoints)
yis_noisy = yis_true + np.random.randn(ndatapoints) * sigma_yis
```

```python
y_min, y_max = np.min(yis_noisy - sigma_yis), np.max(yis_noisy + sigma_yis)
y_min = np.min([y_min, np.min(model_linear(x_grid, slope_true, intercept_true))])
y_max = np.max([y_max, np.max(model_linear(x_grid, slope_true, intercept_true))])
```

```python
plt.plot(x_grid, model_linear(x_grid, slope_true, intercept_true), c=colors[0])
plt.errorbar(xis_true, yis_noisy, sigma_yis, fmt='o', c=colors[0])
plt.xlabel('$x$'); plt.ylabel('$y$');# plt.ylim([y_min, y_max])
```

We are going to pretend we don't know the true model.

Forget what you saw (please).

Here is the noisy data to be analyzed. Can you (mentally) fit a line through it?

```python
y_min, y_max = 0, 1
plt.errorbar(xis_true, yis_noisy, sigma_yis, fmt='o')
plt.xlabel('$x$'); plt.ylabel('$y$'); plt.ylim([y_min, y_max])
```

Let's define a loss/cost function: the total weighted squared error, also called chi-squared:
$$ \chi^2 = \sum_i \left( \frac{ \hat{y}_i - y_i^\mathrm{mod}(x_i, s, m) }{\sigma_i} \right)^2 $$

```python
def loss(observed_yis, yi_uncertainties, model_yis):
    scaled_differences = (observed_yis - model_yis) / yi_uncertainties
    return np.sum(scaled_differences**2, axis=0)
```

We want to minimize this chi-squared to obtain the best possible fit to the data.

Let us look at the fit for a couple of (random) sets of parameters.

```python
random_slopes = np.array([0.25, 0.25, 0.75, 0.75])
random_intercepts = np.array([0.25, 0.75, 0.25, 0.75])
```

```python
fig, axs = plt.subplots(1, 2, sharex=True)
axs[0].errorbar(xis_true, yis_noisy, sigma_yis, fmt='o')
for i, (slope, intercept) in enumerate(zip(random_slopes, random_intercepts)):
    axs[0].plot(x_grid, model_linear(x_grid, slope, intercept), c=colors[i])
    axs[1].scatter(slope, intercept,marker='x', c=colors[i])
    chi2 = loss(yis_noisy[:, None], sigma_yis[:, None],
                 model_linear(xis_true[:, None], slope, intercept))
    axs[1].text(slope, intercept+0.05, '$\chi^2 = %.1f$'% chi2,
                horizontalalignment='center')
axs[0].set_xlabel('$x$'); axs[0].set_ylabel('$y$')
axs[0].set_ylim([0, y_max]); axs[1].set_ylim([0, 1]);
axs[1].set_xlabel('slope'); axs[1].set_ylabel('intercept')
fig.tight_layout()
```

Let us try a brute-force search, and grid our 2D parameter space.

EXERCISE

Create a 100 x 100 grid covering our parameter space.

Evaluate the loss function on the grid, and plot exp(-0.5*loss).

Also find the point that has the minimal loss value.

```python
# SOLUTION
slope_grid, intercept_grid = np.meshgrid(np.linspace(0, 1, 100),
                                         np.linspace(0, 1, 100))
#np.mgrid[0:1:100j, 0:1:100j]
model_yis = model_linear(xis_true[:, None],
                         slope_grid.ravel()[None, :],
                         intercept_grid.ravel()[None, :])
loss_grid = loss(yis_noisy[:, None], sigma_yis[:, None], model_yis[:, :])
# Let's also find the grid point with minimum chi2:
ml_position = np.argmin(loss_grid)
slope_ml = slope_grid.ravel()[ml_position]
intercept_ml = intercept_grid.ravel()[ml_position]
loss_grid = loss_grid.reshape(slope_grid.shape)
```

```python
fig, axs = plt.subplots(1, 2, sharex=False, sharey=False)
axs[0].errorbar(xis_true, yis_noisy, sigma_yis, fmt='o')
axs[0].plot(x_grid, model_linear(x_grid, slope_ml, intercept_ml))
axs[0].set_xlabel('$x$'); axs[0].set_ylabel('$y$')
axs[0].set_ylim([y_min, y_max])
axs[1].set_xlabel('slope'); axs[1].set_ylabel('intercept')
axs[1].axvline(slope_ml, c=colors[1]); axs[1].axhline(intercept_ml, c=colors[1])
axs[1].pcolormesh(slope_grid, intercept_grid, np.exp(-0.5*loss_grid), cmap='ocean_r')
fig.tight_layout()
```

Why visualize $exp(-\frac{1}{2}\chi^2)$ and not simply the $\chi^2$?

Because the former is proportional to our likelihood:

$$\begin{align}
p(D| P, M) &= p(\{ \hat{y}_i \} \vert \{\sigma_i, x_i\}, \textrm{intercept}, \textrm{slope}) \\
&= \prod_{i=1}^{N} p(\hat{y}_i \vert x_i, \sigma_i, b, m)\\
&= \prod_{i=1}^{N} \mathcal{N}\left(\hat{y}_i - y^\mathrm{mod}(x_i; m, b); \sigma^2_i \right)
\ = \prod_{i=1}^{N} \mathcal{N}\left(\hat{y}_i - m x_i - b; \sigma^2_i \right) \\
&= \prod_{i=1}^{N} \frac{1}{\sqrt{2\pi}\sigma_i}\exp\left( - \frac{1}{2} \frac{(\hat{y}_i - m x_i - b)^2}{\sigma^2_i} \right)  \\
&\propto \ \exp\left( - \sum_{i=1}^{N} \frac{1}{2} \frac{(\hat{y}_i - m x_i - b)^2}{\sigma^2_i} \right) \ = \ \exp\left(-\frac{1}{2}\chi^2\right)
\end{align}
$$

Since the data points are independent and the noise is Gaussian.

Let's visualize the $\chi^2$ for individual objects

```python
model_yis = model_linear(xis_true, slope_ml, intercept_ml)
object_chi2s = 0.5*((yis_noisy - model_yis) / sigma_yis)**2
```

```python
fig, ax = plt.subplots(1, 1)
ax.plot(x_grid, model_linear(x_grid, slope_ml, intercept_ml))
v = ax.scatter(xis_true, yis_noisy, c=object_chi2s, cmap='coolwarm', zorder=0)
ax.errorbar(xis_true, yis_noisy, sigma_yis, fmt='o', zorder=-1)
ax.set_xlabel('$x$'); ax.set_ylabel('$y$'); ax.set_ylim([y_min, y_max])
plt.colorbar(v); fig.tight_layout()
```

## Digression: the limits of maximum likelihood

Is a line a good model?

Should we aiming at maximizing the likelihood only?

Here is a danger of Maximum Likelihood: there is always of model that perfectly fits all of the data.

This model does not have to be complicated...

EXERCISE (5 min): can you try to write a very flexible model that fits the data perfectly, i.e. go through every single point? What $\chi^2$ does it lead to?

NOTE: this might not be trivial, so just look for a model that goes through *most* of the data points.

HINT: numpy has good infrastructure for constructing and fitting polynomials... (try `?np.polyfit`).

If you pick a more complicated model you might need to use `scipy.optimize.minimize`.

```python
# SOLUTION
degree = 150
bestfit_polynomial_coefs = np.polyfit(xis_true, yis_noisy, degree)
bestfit_polynomial = np.poly1d(bestfit_polynomial_coefs)
chi2 = loss(yis_noisy, sigma_yis, bestfit_polynomial(xis_true))
print('The chi2 is', chi2)
```

    The chi2 is 3.94070876139e-09

    /Users/bl/anaconda3/lib/python3.5/site-packages/ipykernel_launcher.py:3: RankWarning: Polyfit may be poorly conditioned
      This is separate from the ipykernel package so we can avoid doing imports until

```python
plt.plot(x_grid, bestfit_polynomial(x_grid))
plt.errorbar(xis_true, yis_noisy, sigma_yis, fmt='o')
plt.ylim([y_min, y_max])
```

```python
HTML('')
# Copyright Daniela Huppenkothen, Astrohackweek 2015 in NYC
```

##  Bayes' theorem
with explicit Model and Fixed parameters conditioned on:

$$p(P | D, M, F) = \frac{p(D | P, M, F)\ p(P | M, F)}{p(D | M, F)}$$

In our case, if we omit the explicit dependence on a linear model:

$$p\bigl(m, b \ \bigl\vert \ \{ \hat{y}_i, \sigma_i, x_i\} \bigr) \ \propto \ p\bigl(\{ \hat{y}_i \} \ \bigl\vert \ m, b, \{\sigma_i, x_i\}\bigr) \  p\bigl(m, b\bigr) \ = \ \exp\bigl(-\frac{1}{2}\chi^2\bigr)\ p\bigl(m, b\bigr) $$

```python
# Let us play with Bayes theorem and pick some un-motivated prior:
prior_grid = np.exp(-slope_grid**-1) * np.exp(-intercept_grid**-1)
likelihood_grid = np.exp(-0.5*loss_grid)
posterior_grid = likelihood_grid * prior_grid
```

    /Users/bl/anaconda3/lib/python3.5/site-packages/ipykernel_launcher.py:2: RuntimeWarning: divide by zero encountered in reciprocal

```python
fig, axs = plt.subplots(1, 3)
for i in range(3):
    axs[i].set_ylabel('intercept'); axs[i].set_xlabel('slope');
axs[0].set_title('Prior'); axs[1].set_title('Likelihood'); axs[2].set_title('Posterior')
axs[1].axvline(slope_ml, c=colors[1]); axs[1].axhline(intercept_ml, c=colors[1])
axs[0].pcolormesh(slope_grid, intercept_grid, prior_grid, cmap='ocean_r')
axs[1].pcolormesh(slope_grid, intercept_grid, likelihood_grid, cmap='ocean_r')
axs[2].pcolormesh(slope_grid, intercept_grid, posterior_grid, cmap='ocean_r')
fig.tight_layout()
```

Discussion: what priors are adequate here?

Four common types of priors are:
- __Flat priors__: uniform probability in $[0, 1]$ for both the slope and intercept.
- __Conjugate priors__: Gaussians or inverse Gamma...
- __Empirical priors__: previous experiments told me that $p(m) = \mathcal{N}(0.5, 0.125)$ and $p(b) = \mathcal{N}(0.5, 0.125)$.
- __Non-informative priors__: rotationally invariance for the line: $p(m) \propto (1 + m^2)^{-1.5}$

```python

```

```python

```

## The Curse of Dimensionality (v1)

Problems with 'gridding': number of likelihood evaluations, resolution of the grids, etc

```python
fig, ax = plt.subplots(1, 1, figsize=(5, 4))
ax.set_xlabel('slope'); ax.set_ylabel('intercept');
ax.scatter(slope_grid.ravel(), intercept_grid.ravel(), marker='.', s=1)
ax.set_ylim([0, 1])
ax.set_xlim([0, 1])
fig.tight_layout()
print('Number of point/evaluations of the likelihood:', slope_grid.size)
```

Note to Boris: Go back to slides!

## Sampling posterior distributions with MCMC

We are going to approximate the posterior distribution with a set of samples (see slides).

EXERCISE

Write three functions returning:

- the log of the likelihood `ln_like(params, args...)`.

- the log of the prior `ln_prior(params, args...)`.

- the log of the posterior `ln_post(params, args...)`.

The likelihood is pretty much our previous loss function.

The prior should return `-np.inf` outside of our parameter space of interest. At this stage use a uniform prior in $[0, 1] \times [0, 1]$.

Think about what other priors could be used. Include the correct normalization in the prior and the likelihood if possible.

```python
def ln_like(params, xs, observed_yis, yi_uncertainties):
    model_yis = model_linear(xs, params[0], params[1])
    chi2s = ((observed_yis - model_yis) / yi_uncertainties)**2
    return np.sum(-0.5 * chi2s - 0.5*np.log(2*np.pi) - np.log(yi_uncertainties))

def ln_prior(params):
    if np.any(params < 0) or np.any(params > 1):
        return - np.inf
    return 0.

def ln_post(params, xs, observed_yis, yi_uncertainties):
    lnprior_val = ln_prior(params)
    if ~np.isfinite(lnprior_val):
        return lnprior_val
    else:
        lnlike_val = ln_like(params, xs, observed_yis, yi_uncertainties)
        return lnprior_val + lnlike_val
```

```python
x0 = np.array([0.5, 0.5])
print('Likelihood:', ln_like(x0, xis_true, yis_noisy, sigma_yis))
print('Prior:', ln_prior(x0))
print('Posterior:', ln_post(x0, xis_true, yis_noisy, sigma_yis))
```

    Likelihood: -119.927109309
    Prior: 0.0
    Posterior: -119.927109309

EXERCISE (2 min)

Find the maximum of the log posterior. Try different optimizers in `scipy.optimize.minimize`. Be careful about the sign of the objective function (is it plus or minus the log posterior?)

```python
# SOLUTION
def fun(p0):
    return - ln_post(p0, xis_true, yis_noisy, sigma_yis)

res = scipy.optimize.minimize(fun, np.random.uniform(0, 1, 2), method='Powell')
print(res)
best_parmas = res.x
```

       direc: array([[ 0.        ,  1.        ],
           [ 0.16723191, -0.08780883]])
         fun: -15.191867591126249
     message: 'Optimization terminated successfully.'
        nfev: 116
         nit: 3
      status: 0
     success: True
           x: array([ 0.21128909,  0.23426509])

    /Users/bl/anaconda3/lib/python3.5/site-packages/scipy/optimize/optimize.py:1927: RuntimeWarning: invalid value encountered in double_scalars
      tmp2 = (x - v) * (fx - fw)

```python

```

## Sampling strategy 1: Rejection Sampling

EXERCISE

Implement rejection sampling. Randomly draw points in our 2D parameter space. Keep each point with a probability proportional to the posterior distribution.

HINT: you will find that you need to normalize the posterior distribution in some way to make the sampling possible. Use the MAP solution we just found!

```python
# SOLUTION
normalization = ln_post(best_parmas, xis_true, yis_noisy, sigma_yis)
print(normalization)
num_draws = 10000
i_draw = 0
params_drawn = np.zeros((num_draws, 2))
params_vals = np.zeros((num_draws, ))
num_tot = 0
while i_draw < num_draws:
    params_drawn[i_draw, :] = np.random.uniform(0, 1, 2)
    params_vals[i_draw] = np.exp(
        ln_post(params_drawn[i_draw, :], xis_true, yis_noisy, sigma_yis)\
        - normalization)
    num_tot += 1
    if np.random.uniform(0, 1, 1) < params_vals[i_draw]:
        #print(params_vals[i_draw], i_draw)
        i_draw += 1
print(num_tot, num_draws)
```

    15.1918675911
    626560 10000

```python
fig, axs = plt.subplots(1, 2, sharex=True, sharey=True)
axs[0].pcolormesh(slope_grid, intercept_grid, likelihood_grid, cmap='ocean_r')
axs[1].hist2d(params_drawn[:, 0], params_drawn[:, 1], 30, cmap="ocean_r");
axs[0].set_title('Gridding'); axs[1].set_title('Rejection sampling');
axs[0].set_xlabel('slope'); axs[0].set_ylabel('intercept'); axs[1].set_xlabel('slope');
```

## Sampling strategy 2: Metropolis-Hastings

The Metropolis-Hastings algorithm

For a given target probability $p(\theta)$ and a (symmetric) proposal density $p(\theta_{i+1}|\theta_i)$. We repeat the following:
- draw a sample $\theta_{i+1}$ given $\theta_i$ from the proposal density,
- compute the acceptance probability ratio $a={p(\theta_{i+1})}/{p(\theta_i)}$,
- draw a random uniform number $r$ in $[0, 1]$ and accept $\theta_{i+1}$ if $r < a$.

EXERCISE

Use your implementation of the Metropolis-Hastings algorithm to draw samples from our 2D posterior distribution of interest.

Measure the proportion of parameter draws that are accepted: the acceptance rate.

Plot the chain and visualize the burn-in phase.

Compare the sampling to our previous gridded version.

Estimate the mean and standard deviation of the distribution from the samples. Are they accurate?

```python
# SOLUTION
num_draws = 1000
params_drawn = np.zeros((num_draws, 2))
i_draw = 1
num_draws_tot = 0
params_drawn[0, :] = np.random.uniform(0, 1, 2)
while i_draw < num_draws:
    num_draws_tot += 1
    params_drawn[i_draw, :] = params_drawn[i_draw-1, :] \
        + 0.05 * np.random.randn(2)
    a = np.exp(ln_post(params_drawn[i_draw, :], xis_true, yis_noisy, sigma_yis)\
                   - ln_post(params_drawn[i_draw-1, :], xis_true, yis_noisy, sigma_yis))
    if a >= 1 or np.random.uniform(0, 1, 1) < a:
        i_draw += 1
print('Acceptance rate:', num_draws/num_draws_tot)
```

```python
fig, axs = plt.subplots(1, 2, sharex=True, sharey=True)
axs[0].pcolormesh(slope_grid, intercept_grid, likelihood_grid, cmap='ocean_r')
axs[1].hist2d(params_drawn[:, 0], params_drawn[:, 1], 30, cmap="ocean_r");
axs[0].set_title('Gridding'); axs[1].set_title('Metropolis Hastings');
axs[0].set_xlabel('slope'); axs[0].set_ylabel('intercept'); axs[1].set_xlabel('slope');
```

Let's visualize the chains:

```python
fig, ax = plt.subplots(2, sharex=True)
for i in range(2):
    ax[i].plot(params_drawn[:, i]);
```

## Validation

MCMC is approximate and is only valid if it has converged. But we can't prove that a chain has converget - we can only show it hasn't.

What to do? ___Be paranoïd.__

Is it crucial to 1) run many chains in various setups, and 2) check that the results are stable, and 3) look at the auto-correlation time:

$$\rho_k = \frac{\mathrm{Covar}[X_t, X_{t+k}]}{\mathrm{Var}[X_t]\mathrm{Var}[X_{t+k}]]}$$

See http://rstudio-pubs-static.s3.amazonaws.com/258436_5c7f6f9a84bd47aeaa33ee763e57a531.html and  www.astrostatistics.psu.edu/RLectures/diagnosticsMCMC.pdf

EXERCISE

Visualize chains, autocorrelation time, etc, for short and long chains with different proposal distributions in the Metropolis Hastings algorithm.

```python
# SOLUTION
def autocorr_naive(chain, cutoff):
    auto_corr = np.zeros(cutoff-1)
    mu = np.mean(chain, axis=0)
    var = np.var(chain, axis=0)
    for s in range(1, cutoff-1):
        auto_corr[s] = np.mean( (chain[:-s] - mu) * (chain[s:] - mu) ) / var
    return auto_corr[1:]
```

```python
for i in range(2):
    plt.plot(autocorr_naive(params_drawn[:, i], 500))
plt.xscale('log'); plt.xlabel('$\Delta$'); plt.ylabel('Autocorrelation');
```

## Sampling strategy 3: affine-invariant ensemble sampler

EXERCISE

Let's use a more advanced sampler. Look at the documentation of the `emcee` package and use it to (again) draw samples from our 2D posterior distribution of interest. Make 2D plots with both `plt.hist2d` or `plt.contourf`. For the latter, add 68% and 95% confidence contours.

```python
# SOLUTION
import emcee

ndim = 2
nwalkers = 50

starting_params = np.random.uniform(0, 1, ndim*nwalkers).reshape((nwalkers, ndim))
sampler = emcee.EnsembleSampler(nwalkers, ndim, ln_post,
                                args=[xis_true, yis_noisy, sigma_yis])

num_steps = 100
pos, prob, state = sampler.run_mcmc(starting_params, num_steps)
```

```python
fig, ax = plt.subplots(2, sharex=True)
for i in range(2):
    ax[i].plot(sampler.chain[:, :, i].T, '-k', alpha=0.2);
```

```python
fig, axs = plt.subplots(1, 3, sharex=True, sharey=True)
for i in range(axs.size):
    axs[i].errorbar(sampler.chain[:, i, 0], sampler.chain[:, i, 1], fmt="-o", alpha=0.5, c='k');
```

```python
num_steps = 1000
sampler.reset()
pos, prob, state = sampler.run_mcmc(pos, num_steps)
```

```python
fig, ax = plt.subplots(2, sharex=True)
for i in range(2):
    ax[i].plot(sampler.chain[:, :, i].T, '-k', alpha=0.2);
```

```python
fig, axs = plt.subplots(1, 3, sharex=True, sharey=True)
for i in range(axs.size):
    axs[i].errorbar(sampler.chain[:, i, 0], sampler.chain[:, i, 1], fmt="-o", alpha=0.5, c='k');
```

```python
from corner import hist2d
fig, axs = plt.subplots(1, 2, sharex=True, sharey=True)
axs[0].hist2d(sampler.flatchain[:, 0], sampler.flatchain[:, 1], 30, cmap="ocean_r");
hist2d(sampler.flatchain[:, 0], sampler.flatchain[:, 1], ax=axs[1])
axs[0].set_xlabel('slope'); axs[0].set_ylabel('intercept'); axs[1].set_xlabel('slope');
fig.tight_layout()
```

```python
fig, axs = plt.subplots(1, 2, sharex=True, sharey=True)
axs[0].hist(sampler.flatchain[:, 0], histtype='step');
axs[1].hist(sampler.flatchain[:, 1], histtype='step');
axs[0].set_xlabel('slope'); axs[1].set_xlabel('intercept'); axs[0].set_ylabel('Marginal distribution');
fig.tight_layout()
```

It is extremely useful to plot the model in data space!

EXERCISE

Loop through the posterior samples (a random subset of them?) and over-plot them with the data, with some transparency.

```python
# SOLUTION
fig, axs = plt.subplots(1, 2, sharex=True, sharey=True)
axs[0].set_xlabel('$x$'); axs[1].set_xlabel('$x$'); axs[0].set_ylabel('$y$');
num = 1000
y_models = np.zeros((x_grid.size, num))
for j, i in enumerate(np.random.choice(np.arange(sampler.flatchain.shape[0]), num, replace=False)):
    y_models[:, j] = model_linear(x_grid, sampler.flatchain[i, 0], sampler.flatchain[i, 1])
    axs[0].plot(x_grid, y_models[:, j], c='gray', alpha=0.01, zorder=0)
axs[1].plot(x_grid, np.mean(y_models, axis=1), c='gray', alpha=1, zorder=0)
axs[1].fill_between(x_grid, np.mean(y_models, axis=1)-np.std(y_models, axis=1),
            np.mean(y_models, axis=1)+np.std(y_models, axis=1), color='gray', alpha=0.5, zorder=0)
axs[0].errorbar(xis_true, yis_noisy, sigma_yis, fmt='o', zorder=1)
axs[1].errorbar(xis_true, yis_noisy, sigma_yis, fmt='o', zorder=1)
```

```python
fig, axs = plt.subplots(1, 2, sharex=True, sharey=True)
axs[0].hist(sampler.flatchain[:, 0], histtype='step');
axs[1].hist(sampler.flatchain[:, 1], histtype='step');
axs[0].set_xlabel('slope'); axs[1].set_xlabel('intercept'); axs[0].set_ylabel('Marginal distribution');
fig.tight_layout()
```

## Parameter estimation

Often, we want to report summary statistics on our parameters, e.g. in a paper.

EXERCISE

Compute some useful summary statistics for our two parameters from the MCMC chains: mean, confidence intervals, etc

```python
# SOLUTION
thechain = sampler.flatchain
print('Mean values:', np.mean(thechain, axis=0))
print('Standard deviation:', np.std(thechain, axis=0))
print('Quantiles:', np.percentile(thechain, [5, 16, 50, 84, 95], axis=0))
```

NOTE: for any subsequent analysis, don't use the summary statistics, use the full MCMC chains if you can!

CONTROVERSIAL: if you are only ever going to report and use the mean of a parameter, maybe you don't need MCMC...

# Gibbs sampling

Because of the factorization $p(m, b) = p(m | b)p(b) = p(b|m) p(m)$ we can actually design an algorithm to create an MCMC chain where there is __no need to reject points__!

The procedure is simple: __given__ $(m_i, b_i)$, __draw__ $m_{i+1}$ from $p(m | b=b_i)$, then __draw__ $b_{i+1}$ from $p(b | m=m_{i+1})$.

The only __strong condition__ is to be able to draw directly from the conditional distributions, here $p(m | b)$ and $p(b|m)$.

Generalized to 3+ variables and blocks: draw from (block) conditional distributions in a sequence.

I will draw the way Gibbs sampling works on the board.

What conditional distributions do we need here? Recall that our full posterior distribution is

$$p\bigl(m, b \ \bigl\vert \ \{ \hat{y}_i, \sigma_i, x_i\} \bigr) \ \propto \ p\bigl(\{ \hat{y}_i \} \ \bigl\vert \ m, b, \{\sigma_i, x_i\}\bigr) \  p\bigl(m, b\bigr) $$
$$= \ \exp\left( - \sum_{i=1}^{N} \frac{1}{2} \frac{(\hat{y}_i - m x_i - b)^2}{\sigma^2_i} \right) \ p\bigl(m, b\bigr) $$

If we take uniform priors, $p\bigl(m, b\bigr) = C$, then this posterior distribution is Gaussian:

$$p\bigl(m, b \ \bigl\vert \ \{ \hat{y}_i, \sigma_i, x_i\} \bigr) \ \propto \ \mathcal{N}\left( [X^TX]^{-1}X^T Y ; [X^TX]^{-1} \Sigma \right) $$

$$= \mathcal{N}\left(
\left[\begin{matrix} \hat{m} \\ \hat{b} \end{matrix}\right];
\left[\begin{matrix} \Sigma_{mm} & \Sigma_{mb}\\ \Sigma_{mb} & \Sigma_{bb} \end{matrix}\right]
\right) $$

where I have put the $x_i$'s in a $N x 2$ vector $X$ (the first column is ones, and the second contains the $x_i$'s),  the $\hat{y}_i$'s in a vector $Y$, and the $\sigma_i$'s in a diagonal matrix $\Sigma$. This matches the classic maximum likelihood results for linear regression. The variables $(\hat{m}, \hat{b}, \Sigma_{mm}, \Sigma_{mb}, \Sigma_{mb}, \Sigma_{bb})$ are a convenient notation.

By using Gaussian identities (which can for example be found [here](https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf)), we can write the conditional distributions

$$p\bigl(m \ \bigl\vert \ b, \{ \hat{y}_i, \sigma_i, x_i\} \bigr) \ \propto \mathcal{N}\left(
\hat{m} + \Sigma_{mb}\Sigma_{bb}^{-1}(b - \hat{b});
\Sigma_{mm} - \Sigma_{mb}\Sigma_{bb}^{-1}\Sigma_{mb}
\right) $$

$$p\bigl(b \ \bigl\vert \ m, \{ \hat{y}_i, \sigma_i, x_i\} \bigr) \ \propto \mathcal{N}\left(
\hat{b} + \Sigma_{mb}\Sigma_{mm}^{-1}(m - \hat{m});
\Sigma_{bb} - \Sigma_{mb}\Sigma_{mm}^{-1}\Sigma_{mb}
\right) $$

__Important__: It is because we know how to draw random numbers from Gaussian distributions that we can use Gibbs sampling.

(I lied to you ; there is an analytic solution for this problem!)

BONUS question: look up online what "conjugate priors" are appropriate for linear regression models.

```python
X = np.vstack((xis_true, np.repeat(1, xis_true.size))).T
datacov = np.diag(sigma_yis**2.0)
A = np.dot(X.T, np.linalg.solve(datacov, X))
posterior_mean = np.linalg.solve(A, np.dot(X.T, np.linalg.solve(datacov, yis_noisy[:, None]))).ravel()
print('Posterior mean:', posterior_mean)
posterior_covariance = np.linalg.inv(np.dot(X.T,  np.linalg.solve(datacov, X)))
print('Posterior covariance:', posterior_covariance)
```

Important note: you might think sampling is useless here because we know the analytic posterior distribution...

But there is a very wide class of problems and models where you cannot find an analytic solution for the full posterior distribution, but __you can write the conditional posterior distributions on the parameters__!

In fact, in hierarchical models, in 99% of cases, we automatically have the conditional distributions, but it is how we construct the model. For this reason, and because Gibbs sampling has an acceptance rate of 1, it is very powerful and popular, and often a default solution!

EXERCISE: using those results, implement Gibbs sampling!

```python
# SOLUTION
num_draws = 10000
i_draw = 0
params_drawn = np.zeros((num_draws, 2))
params_drawn[0, :] = posterior_mean + 0.01*np.random.uniform(0, 1, 2)
for i_draw in range(1, num_draws):
    mu = posterior_mean[0] + posterior_covariance[1, 0] *\
        (params_drawn[i_draw-1, 1] - posterior_mean[1]) / posterior_covariance[1, 1]
    cov = posterior_covariance[0, 0] - posterior_covariance[1, 0]**2 / posterior_covariance[1, 1]
    params_drawn[i_draw, 0] = mu + np.random.randn() * cov**0.5

    mu = posterior_mean[1] + posterior_covariance[1, 0] *\
        (params_drawn[i_draw, 0] - posterior_mean[0]) / posterior_covariance[0, 0]
    cov = posterior_covariance[1, 1] - posterior_covariance[1, 0]**2 / posterior_covariance[0, 0]
    params_drawn[i_draw, 1] = mu + np.random.randn() * cov**0.5
```

```python
fig, axs = plt.subplots(1, 2, sharex=True, sharey=True)
axs[0].hist2d(sampler.flatchain[:, 0], sampler.flatchain[:, 1], 30, cmap="ocean_r");
axs[1].hist2d(params_drawn[:, 0], params_drawn[:, 1], 30, cmap="ocean_r");
axs[0].set_xlabel('slope'); axs[0].set_ylabel('intercept'); axs[1].set_xlabel('slope');
fig.tight_layout()
```

### Final words on Gibbs sampling
It is great to have an acceptance rate of 1, but we critically need nice conditional distributions, and no nasty degeneracies between the parameters.

Note to Boris: Go back to slides!

## Fitting data with both x and y errors

We observe a set of $\hat{x}_i$ which are noisified versions of the true $x_i$, with Gaussian noise $\gamma_i$.

```python
sigma_xis = np.repeat(0.1, ndatapoints) * np.random.uniform(0.2, 1.0, ndatapoints)
xis_noisy = xis_true + sigma_xis * np.random.randn(xis_true.size)
```

```python
plt.errorbar(xis_noisy, yis_noisy, xerr=sigma_xis, yerr=sigma_yis, fmt='o')
plt.xlabel('$x$'); plt.ylabel('$y$'); plt.ylim([y_min, y_max])
```

Our likelihood is now:

$$\begin{align}
p(D| P, M) &= p(\{ \hat{y}_i, \hat{x}_i \} \vert \{\sigma_i, \gamma_i, x_i\}, \textrm{intercept}, \textrm{slope}) \\
&= \prod_{i=1}^{N} p(\hat{y}_i \vert x_i, \sigma_i, b, m) \ p(\hat{x}_i \vert x_i, \gamma_i) \\
& = \prod_{i=1}^{N} \mathcal{N}\left(\hat{y}_i - m x_i - b; \sigma^2_i \right) \mathcal{N}\left(\hat{x}_i - x_i; \gamma^2_i \right)
\end{align}
$$

We now have $N$ extra parameters, the $x_i$'s!

The full posterior distribution:

$$  p\bigl( m, s, \{  x_i \} \bigl\vert  \{ \hat{y}_i, \hat{x}_i, \sigma_i, \gamma_i\} \bigr) \ \propto  \
p\bigl(\{ \hat{y}_i, \hat{x}_i \} \bigl\vert \{\sigma_i, \gamma_i, x_i\}, m, s\bigr) \ \ p\bigl(\{ x_i \}, m, s\bigr) $$

## This is the Curse of Dimensionality v2!

## One solution : Hamiltonian Monte Carlo

Neal's book chapter is a good starting point: https://arxiv.org/abs/1206.1901

Demo: https://chi-feng.github.io/mcmc-demo/app.html

Gradients (and hessians) needed! Three strategies:
- pen and paper, then home-made implementation
- automatic symbolic differentiation
- automatic numerical differentition

Always try auto-diff first (e.g., with `autograd`).

Large-scale inference (gazilion parameters): try `tensorflow`

## Analytic marginalization of latent variables

We are only truly interested in the marginalized posterior distribution:

$$p\bigl( m, s \bigl\vert  \{ \hat{y}_i, \hat{x}_i, \sigma_i, \gamma_i\} \bigr) \ = \ \int\mathrm{d}\{x_i\} p\bigl( m, s, \{  x_i \} \bigl\vert  \{ \hat{y}_i, \hat{x}_i, \sigma_i, \gamma_i\} \bigr) \\
 \propto \  \prod_{i=1}^{N} \int \mathrm{d}x_i \mathcal{N}\left(\hat{y}_i - m x_i - b; \sigma^2_i \right) \mathcal{N}\left(\hat{x}_i - x_i; \gamma^2_i \right) \ \ p\bigl(\{ x_i \}, m, s\bigr) \\
 \propto \  \prod_{i=1}^{N} \mathcal{N}\left(\hat{y}_i - m \hat{x}_i - b; \sigma^2_i + \gamma^2_i\right)  \ p(s, m) $$

with flat uninformative priors on $x_i$'s $p\bigl(x_i)$.

We have eliminated the $x_i$'s!

```python
HTML('')
```

Let us do a run with the x's fixed to their noisy values (which is wrong! This is ignoring the x noise).

```python
ndim = 2
nwalkers = 50

starting_params = np.random.uniform(0, 1, ndim*nwalkers).reshape((nwalkers, ndim))
sampler2 = emcee.EnsembleSampler(nwalkers, ndim, ln_post,
                                args=[yis_noisy, xis_noisy, sigma_yis])

num_steps = 100
pos, prob, state = sampler2.run_mcmc(starting_params, num_steps)
num_steps = 1000
sampler2.reset()
pos, prob, state = sampler2.run_mcmc(pos, num_steps)
```

```python
def ln_like(params, observed_yis, observed_xis, yi_uncertainties, xi_uncertainties):
    xyi_uncertainties = np.sqrt(xi_uncertainties**2. + yi_uncertainties**2.)
    model_yis = model_linear(observed_xis, params[0], params[1])
    return np.sum(-0.5 * ((observed_yis - model_yis) / xyi_uncertainties)**2
                   - 0.5*np.log(2*np.pi) - np.log(xyi_uncertainties))

def ln_prior(params):
    if np.any(params < 0) or np.any(params > 1):
        return - np.inf
    return 0.

def ln_post(params, observed_yis, observed_xis, yi_uncertainties, xi_uncertainties):
    lnprior_val = ln_prior(params)
    if ~np.isfinite(lnprior_val):
        return lnprior_val
    else:
        lnlike_val = ln_like(params, observed_yis, observed_xis, yi_uncertainties, xi_uncertainties)
        return lnprior_val + lnlike_val
```

```python
x0 = np.repeat(0.5, 2)
print('Likelihood:', ln_like(x0, yis_noisy, xis_noisy, sigma_yis, sigma_xis))
print('Prior:', ln_prior(x0))
print('Posterior:', ln_post(x0, yis_noisy, xis_noisy, sigma_yis, sigma_xis))
```

```python
ndim = 2
nwalkers = 50

starting_params = np.random.uniform(0, 1, ndim*nwalkers).reshape((nwalkers, ndim))
sampler3 = emcee.EnsembleSampler(nwalkers, ndim, ln_post,
                                args=[yis_noisy, xis_noisy, sigma_yis, sigma_xis])

num_steps = 100
pos, prob, state = sampler3.run_mcmc(starting_params, num_steps)
num_steps = 1000
sampler3.reset()
pos, prob, state = sampler3.run_mcmc(pos, num_steps)
```

```python
fig, axs = plt.subplots(1, 3, sharex=False, sharey=False)
axs[0].hist2d(sampler.flatchain[:, 0], sampler.flatchain[:, 1], 30, cmap="ocean_r");
axs[1].hist2d(sampler2.flatchain[:, 0], sampler2.flatchain[:, 1], 30, cmap="ocean_r");
axs[2].hist2d(sampler3.flatchain[:, 0], sampler3.flatchain[:, 1], 30, cmap="ocean_r");
axs[0].set_title('x true');
axs[1].set_title('x wrongly fixed');
axs[2].set_title('with x errors');
axs[0].set_xlabel('slope'); axs[0].set_ylabel('intercept');
axs[1].set_xlabel('slope');
axs[2].set_xlabel('slope');
for i in range(3):
    axs[i].axvline(slope_true)
    axs[i].axhline(intercept_true)
    axs[i].set_ylim([0, 1])
    axs[i].set_xlim([0, 1])
fig.tight_layout()
```

```python
fig, axs = plt.subplots(1, 2, sharex=True, sharey=True)
axs[0].set_xlabel('$x$'); axs[1].set_xlabel('$x$'); axs[0].set_ylabel('$y$');
num = 1000
y_models = np.zeros((x_grid.size, num))
for j, i in enumerate(np.random.choice(np.arange(sampler3.flatchain.shape[0]), num, replace=False)):
    y_models[:, j] = model_linear(x_grid, sampler3.flatchain[i, 0], sampler3.flatchain[i, 1])
    axs[0].plot(x_grid, y_models[:, j], c='gray', alpha=0.01, zorder=0)
axs[1].plot(x_grid, np.mean(y_models, axis=1), c='gray', alpha=1, zorder=0)
axs[1].fill_between(x_grid, np.mean(y_models, axis=1)-np.std(y_models, axis=1),
            np.mean(y_models, axis=1)+np.std(y_models, axis=1), color='gray', alpha=0.5, zorder=0)
axs[0].errorbar(xis_noisy, yis_noisy, xerr=sigma_xis, yerr=sigma_yis, fmt='o', zorder=1)
axs[1].errorbar(xis_noisy, yis_noisy, xerr=sigma_xis, yerr=sigma_yis, fmt='o', zorder=1)
```

We got around the number of parameters by analytically marginalizing the ones we don't really care about. Sometimes this is not possible!

# Extensions

- __Automatic differentiation__ with autograd, tensorflow, etc.
- __Nested Sampling for nasty distributions and model comparison__. Application: fitting multiple components/lines to a data set.
- __Model testing, model comparison__. I have multiple models. Which is the best? Example: fit multiple lines to data.
- __Hamiltonian Monte Carlo with quasi-auto-tuning for millions of parameters.__ Application: fitting a line with many latent parameters (x noise, outliers, etc).
- __Multi-stage hybrid sampling__: Application: non-linear models with many parameters and complicated gradients.
- __Connections to deep machine learning__: Bayesian interpretation of Convolution networks, Adversarial training, deep forward models, etc. TensorFlow.

Let me know if you are interested and we will organize a session. A few notebooks and mode advanced examples available on https://ixkael.github.io

# Final thoughts
With the right method you can solve problems/models that seem intractable. Don't underestimate yourself! Start small, but be ambitious.

```python
HTML('')
```

```python

```

notebooks/Fitting a line to data - a full tutorial - with solutions.ipynb


```python
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
```

```python
N = 2**6+1
```

```python
x = np.linspace(0.995,1,N)
```

```python
plt.plot(x, (1-x)**6)
```

```python
import sympy
```

```python
X = sympy.var('x')
sympy.expand((1-X)**6)
```

    x**6 - 6*x**5 + 15*x**4 - 20*x**3 + 15*x**2 - 6*x + 1

```python
x = np.linspace(0.995,1,N)
plt.plot(x, x**6 - 6*x**5 + 15*x**4 - 20*x**3 + 15*x**2 - 6*x + 1)
```

```python
0.01 * 1e2 #NOT what the computer does
```

    1.0

```python
#The computer knows how to do the sum between number which have the same powers:
#This is what the computer does:

1e-16 = 0.1 * 1e1 + 0.0000000000000001 * 1e1 = (0.1 + 0.0000000000000001) * 1e1

#if the condition number of the summation is larger that 10^15 the summation is not allowed.
```

```python
A=np.random.random((50,50))
```

```python
A.shape
K = np.linalg.cond(A)
A.shape, K
```

    ((50, 50), 258.66576624961664)

```python
#stiamo cercando di capire cosa significa avere un bad conditions number o un buon conditions
#number per semplici operazioni con la moltiplicazione
#cioe` vogliamo esplorare qual`e` il rounding error( l'errore dovuto al fatto che e` un numero finito)
#rounding error=10^15

#condition number:something that rappresent the inverse of signal to input ratio
```

```python
#repeating this few times the max error should reach k, the condition number of A

#np.linalg.cond(A)

estimated_k = 0
n_realizations = 1000

for j in range(n_realizations):

    x = np.random.random((50,))
    norm_x = np.linalg.norm(x)

    dx = np.random.random((50,))
    norm_dx = np.linalg.norm(dx)

    y = A.dot(x)
    norm_y = np.linalg.norm(y)

    dy = np.linalg.norm(A.dot(dx))
    norm_dy = np.linalg.norm(dy)

    kx = (norm_dy/norm_y) / (norm_dx/norm_x) #normalize perturb in output divided normalize perturb in input
    estimated_k = max(kx, estimated_k)

estimated_k
```

    1.1227069565500802

```python
kx, norm_x, norm_y, norm_dx, norm_dy
```

    (0.95504750163842733,
     4.1695476471279251,
     90.9175412389408,
     3.9136832605207146,
     81.502210666155406)

Now we try to estimate truncation error

```python
def FD(f, x, dx): #valuta la funzione f nel punto x usando dx come approx step
    return (f(x+dx)-f(x))/dx
```

```python
N = 2**6+1
x = np.linspace(0,1,N)
```

```python
def myf(x):
    return np.sin(np.pi * xi)

def supnorm(f_of_x):
    return max(abs(f_of_x))

y = myf(x)

yprime = np.pi * np.cos(np.pi * x)

FDyprime = FD(myf, x, 2**(-3))
```

```python
#plot(x,yprime 'ro')

dxs = []
errors = []

for i in range(2,51): #{2, 3, 4, 5}
    dx = 2**(-i)
    dxs.append(dx)
    FDprime = FD(myf, x, dx)
    # plot(x,FDprime)
    error = supnorm(yprime-FDprime)
    errors.append(error)

loglog(dxs, errors)

#plot(x,FD(myf,x,2**(-i)))
#plot(x,FDyprime)
yprime.shape
```

```python

```