About the author:
This notebook was forked from this [project](https://github.com/fonnesbeck/scipy2014_tutorial). The original author is Chris Fonnesbeck, Assistant Professor of Biostatistics. You can follow Chris on Twitter [@fonnesbeck](https://twitter.com/fonnesbeck).

#### Introduction

For most problems of interest, Bayesian analysis requires integration over multiple parameters, making the calculation of a [posterior](https://en.wikipedia.org/wiki/Posterior_probability) intractable whether via analytic methods or standard methods of numerical integration.

However, it is often possible to *approximate* these integrals by drawing samples
from posterior distributions. For example, consider the expected value (mean) of a vector-valued random variable $\mathbf{x}$:

$$
E[\mathbf{x}] = \int \mathbf{x} f(\mathbf{x}) \mathrm{d}\mathbf{x}\,, \quad
\mathbf{x} = \{x_1, \ldots, x_k\}
$$

where $k$ (dimension of vector $\mathbf{x}$) is perhaps very large.

If we can produce a reasonable number of random vectors $\{{\bf x_i}\}$, we can use these values to approximate the unknown integral. This process is known as [**Monte Carlo integration**](https://en.wikipedia.org/wiki/Monte_Carlo_integration). In general, Monte Carlo integration allows integrals against probability density functions

$$
I = \int h(\mathbf{x}) f(\mathbf{x}) \mathrm{d}\mathbf{x}
$$

to be estimated by finite sums

$$
\hat{I} = \frac{1}{n}\sum_{i=1}^n h(\mathbf{x}_i),
$$

where $\mathbf{x}_i$ is a sample from $f$. This estimate is valid and useful because:

- $\hat{I} \rightarrow I$ with probability $1$ by the [strong law of large numbers](https://en.wikipedia.org/wiki/Law_of_large_numbers#Strong_law);

- simulation error can be measured and controlled.

### Example (Negative Binomial Distribution)

We can use this kind of simulation to estimate the expected value of a random variable that is negative binomial-distributed. The [negative binomial distribution](https://en.wikipedia.org/wiki/Negative_binomial_distribution) applies to discrete positive random variables. It can be used to model the number of Bernoulli trials that one can expect to conduct until $r$ failures occur.

The [probability mass function](https://en.wikipedia.org/wiki/Probability_mass_function) reads

$$
f(k \mid p, r) = {k + r - 1 \choose k} (1 - p)^k p^r\,,
$$

where $k \in \{0, 1, 2, \ldots \}$ is the value taken by our non-negative discrete random variable and
$p$ is the probability of success ($0 < p < 1$).

Most frequently, this distribution is used to model *overdispersed counts*, that is, counts that have variance larger
than the mean (i.e., what would be predicted under a
[Poisson distribution](http://en.wikipedia.org/wiki/Poisson_distribution)).

In fact, the negative binomial can be expressed as a continuous mixture of Poisson distributions,
where a [gamma distributions](http://en.wikipedia.org/wiki/Gamma_distribution) act as mixing weights:

$$
f(k \mid p, r) = \int_0^{\infty} \text{Poisson}(k \mid \lambda) \,
\text{Gamma}_{(r, (1 - p)/p)}(\lambda) \, \mathrm{d}\lambda,
$$

where the parameters of the gamma distribution are denoted as (shape parameter, inverse scale parameter).

Let's resort to simulation to estimate the mean of a negative binomial distribution with $p = 0.7$ and $r = 3$:

```python
import numpy as np

r = 3
p = 0.7
```

```python
# Simulate Gamma means (r: shape parameter; p / (1 - p): scale parameter).
lam = np.random.gamma(r, p / (1 - p), size=100)
```

```python
# Simulate sample Poisson conditional on lambda.
sim_vals = np.random.poisson(lam)
```

```python
sim_vals.mean()
```

    6.3399999999999999

The actual expected value of the negative binomial distribution is $r p / (1 - p)$, which in this case is 7. That's pretty close, though we can do better if we draw more samples:

```python
lam = np.random.gamma(r, p / (1 - p), size=100000)
sim_vals = np.random.poisson(lam)
sim_vals.mean()
```

    7.0135199999999998

This approach of drawing repeated random samples in order to obtain a desired numerical result is generally known as **Monte Carlo simulation**.

Clearly, this is a convenient, simplistic example that did not require simuation to obtain an answer. For most problems, it is simply not possible to draw independent random samples from the posterior distribution because they will generally be (1) multivariate and (2) not of a known functional form for which there is a pre-existing random number generator.

However, we are not going to give up on simulation. Though we cannot generally draw independent samples for our model, we can usually generate *dependent* samples, and it turns out that if we do this in a particular way, we can obtain samples from almost any posterior distribution.

## Markov Chains

A Markov chain is a special type of *stochastic process*. The standard definition of a stochastic process is an ordered collection of random variables:

$$
\{X_t:t \in T\}
$$

where $t$ is frequently (but not necessarily) a time index. If we think of $X_t$ as a state $X$ at time $t$, and invoke the following dependence condition on each state:

\begin{align*}
&Pr(X_{t+1}=x_{t+1} | X_t=x_t, X_{t-1}=x_{t-1},\ldots,X_0=x_0) \\
&= Pr(X_{t+1}=x_{t+1} | X_t=x_t)
\end{align*}

then the stochastic process is known as a Markov chain. This conditioning specifies that the future depends on the current state, but not past states. Thus, the Markov chain wanders about the state space,
remembering only where it has just been in the last time step.

The collection of transition probabilities is sometimes called a *transition matrix* when dealing with discrete states, or more generally, a *transition kernel*.

It is useful to think of the Markovian property as **mild non-independence**.

If we use Monte Carlo simulation to generate a Markov chain, this is called **Markov chain Monte Carlo**, or MCMC. If the resulting Markov chain obeys some important properties, then it allows us to indirectly generate independent samples from a particular posterior distribution.

> ### Why MCMC Works: Reversible Markov Chains
>
> Markov chain Monte Carlo simulates a Markov chain for which some function of interest
> (e.g., the joint distribution of the parameters of some model) is the unique, invariant limiting distribution. An invariant distribution with respect to some Markov chain with transition kernel $Pr(y \mid x)$ implies that:
>
> $$\int_x Pr(y \mid x) \pi(x) dx = \pi(y).$$
>
> Invariance is guaranteed for any *reversible* Markov chain. Consider a Markov chain in reverse sequence:
> $\{\theta^{(n)},\theta^{(n-1)},...,\theta^{(0)}\}$. This sequence is still Markovian, because:
>
> $$Pr(\theta^{(k)}=y \mid \theta^{(k+1)}=x,\theta^{(k+2)}=x_1,\ldots ) = Pr(\theta^{(k)}=y \mid \theta^{(k+1)}=x)$$
>
> Forward and reverse transition probabilities may be related through Bayes theorem:
>
> $$\frac{Pr(\theta^{(k+1)}=x \mid \theta^{(k)}=y) \pi^{(k)}(y)}{\pi^{(k+1)}(x)}$$
>
> Though not homogeneous in general, $\pi$ becomes homogeneous if:
>
> -   $n \rightarrow \infty$
>
> -   $\pi^{(i)}=\pi$ for some $i < k$
>
> If this chain is homogeneous it is called reversible, because it satisfies the ***detailed balance equation***:
>
> $$\pi(x)Pr(y \mid x) = \pi(y) Pr(x \mid y)$$
>
> Reversibility is important because it has the effect of balancing movement through the entire state space. When a Markov chain is reversible, $\pi$ is the unique, invariant, stationary distribution of that chain. Hence, if $\pi$ is of interest, we need only find the reversible Markov chain for which $\pi$ is the limiting distribution.
> This is what MCMC does!

## Gibbs Sampling

The Gibbs sampler is the simplest and most prevalent MCMC algorithm. If a posterior has $k$ parameters to be estimated, we may condition each parameter on current values of the other $k-1$ parameters, and sample from the resultant distributional form (usually easier), and repeat this operation on the other parameters in turn. This procedure generates samples from the posterior distribution. Note that we have now combined Markov chains (conditional independence) and Monte Carlo techniques (estimation by simulation) to yield Markov chain Monte Carlo.

Here is a stereotypical Gibbs sampling algorithm:

1.  Choose starting values for states (parameters):
    ${\bf \theta} = [\theta_1^{(0)},\theta_2^{(0)},\ldots,\theta_k^{(0)}]$.

2.  Initialize counter $j=1$.

3.  Draw the following values from each of the $k$ conditional
    distributions:

    $$\begin{aligned}
    \theta_1^{(j)} &\sim& \pi(\theta_1 | \theta_2^{(j-1)},\theta_3^{(j-1)},\ldots,\theta_{k-1}^{(j-1)},\theta_k^{(j-1)}) \\
    \theta_2^{(j)} &\sim& \pi(\theta_2 | \theta_1^{(j)},\theta_3^{(j-1)},\ldots,\theta_{k-1}^{(j-1)},\theta_k^{(j-1)}) \\
    \theta_3^{(j)} &\sim& \pi(\theta_3 | \theta_1^{(j)},\theta_2^{(j)},\ldots,\theta_{k-1}^{(j-1)},\theta_k^{(j-1)}) \\
    \vdots \\
    \theta_{k-1}^{(j)} &\sim& \pi(\theta_{k-1} | \theta_1^{(j)},\theta_2^{(j)},\ldots,\theta_{k-2}^{(j)},\theta_k^{(j-1)}) \\
    \theta_k^{(j)} &\sim& \pi(\theta_k | \theta_1^{(j)},\theta_2^{(j)},\theta_4^{(j)},\ldots,\theta_{k-2}^{(j)},\theta_{k-1}^{(j)})\end{aligned}$$

4.  Increment $j$ and repeat until convergence occurs.

As we can see from the algorithm, each distribution is conditioned on the last iteration of its chain values, constituting a Markov chain as advertised. The Gibbs sampler has all of the important properties outlined in the previous section: it is aperiodic, homogeneous and ergodic. Once the sampler converges, all subsequent samples are from the target distribution. This convergence occurs at a geometric rate.

## Example: Inferring patterns in UK coal mining disasters

Let's try to model a more interesting example, a time series of recorded coal mining
disasters in the UK from 1851 to 1962.

Occurrences of disasters in the time series is thought to be derived from a
Poisson process with a large rate parameter in the early part of the time
series, and from one with a smaller rate in the later part. We are interested
in locating the change point in the series, which perhaps is related to changes
in mining safety regulations.

```python
disasters_array = np.array([4, 5, 4, 0, 1, 4, 3, 4, 0, 6, 3, 3, 4, 0, 2, 6,
                            3, 3, 5, 4, 5, 3, 1, 4, 4, 1, 5, 5, 3, 4, 2, 5,
                            2, 2, 3, 4, 2, 1, 3, 2, 2, 1, 1, 1, 1, 3, 0, 0,
                            1, 0, 1, 1, 0, 0, 3, 1, 0, 3, 2, 2, 0, 1, 1, 1,
                            0, 1, 0, 1, 0, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2,
                            3, 3, 1, 1, 2, 1, 1, 1, 1, 2, 4, 2, 0, 0, 1, 4,
                            0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1])

n_count_data = len(disasters_array)
```

```python
import plotly.plotly as py
import plotly.graph_objs as pgo
```

```python
data = pgo.Data([
        pgo.Scatter(
            x=[str(year) + '-01-01' for year in np.arange(1851, 1962)],
            y=disasters_array,
            mode='lines+markers'
    )
])
```

```python
layout = pgo.Layout(
    title='UK coal mining disasters (per year), 1851--1962',
    xaxis=pgo.XAxis(title='Year', type='date', range=['1851-01-01', '1962-01-01']),
    yaxis=pgo.YAxis(title='Disaster count')
)
```

```python
fig = pgo.Figure(data=data, layout=layout)
```

```python
py.iplot(fig, filename='coal_mining_disasters')
```

We are going to use Poisson random variables for this type of count data. Denoting year $i$'s accident count by $y_i$,

$$y_i \sim \text{Poisson}(\lambda).$$

For those unfamiliar, Poisson random variables look like this:

```python
data2 = pgo.Data([
        pgo.Histogram(
            x=np.random.poisson(l, 1000),
            opacity=0.75,
            name=u'λ=%i' % l
        ) for l in [1, 5, 12, 25]
])
```

```python
layout_grey_bg = pgo.Layout(
    xaxis=pgo.XAxis(zeroline=False, showgrid=True, gridcolor='rgb(255, 255, 255)'),
    yaxis=pgo.YAxis(zeroline=False, showgrid=True, gridcolor='rgb(255, 255, 255)'),
    paper_bgcolor='rgb(255, 255, 255)',
    plot_bgcolor='rgba(204, 204, 204, 0.5)'
)
```

```python
layout2 = layout_grey_bg.copy()
```

```python
layout2.update(
    barmode='overlay',
    title='Poisson Means',
    xaxis=pgo.XAxis(range=[0, 50]),
    yaxis=pgo.YAxis(range=[0, 400])
)
```

```python
fig2 = pgo.Figure(data=data2, layout=layout2)
```

```python
py.iplot(fig2, filename='poisson_means')
```

The modeling problem is about estimating the values of the $\lambda$ parameters. Looking at the time series above, it appears that the rate declines over time.

A **changepoint model** identifies a point (here, a year) after which the parameter $\lambda$ drops to a lower value. Let us call this point in time $\tau$. So we are estimating two $\lambda$ parameters:
$\lambda = \lambda_1$ if $t \lt \tau$ and $\lambda = \lambda_2$ if $t \geq \tau$.

We need to assign prior probabilities to both $\{\lambda_1, \lambda_2\}$. The gamma distribution not only provides a continuous density function for positive numbers, but it is also *conjugate* with the Poisson sampling distribution.

```python
lambda1_lambda2 = [(0.1, 100), (1, 100), (1, 10), (10, 10)]
```

```python
data3 = pgo.Data([
        pgo.Histogram(
            x=np.random.gamma(*p, size=1000),
            opacity=0.75,
            name=u'α=%i, β=%i' % (p[0], p[1]))
        for p in lambda1_lambda2
])
```

```python
layout3 = layout_grey_bg.copy()
layout3.update(
    barmode='overlay',
    xaxis=pgo.XAxis(range=[0, 300])
)
```

```python
fig3 = pgo.Figure(data=data3, layout=layout3)
```

```python
py.iplot(fig3, filename='gamma_distributions')
```

We will specify suitably vague hyperparameters $\alpha$ and $\beta$ for both priors:

\begin{align}
\lambda_1 &\sim \text{Gamma}(1, 10), \\
\lambda_2 &\sim \text{Gamma}(1, 10).
\end{align}

Since we do not have any intuition about the location of the changepoint (unless we visualize the data), we will assign a discrete uniform prior over the entire observation period [1851, 1962]:

\begin{align}
&\tau \sim \text{DiscreteUniform(1851, 1962)}\\
&\Rightarrow P(\tau = k) = \frac{1}{111}.
\end{align}

### Implementing Gibbs sampling

We are interested in estimating the joint posterior of $\lambda_1, \lambda_2$ and $\tau$ given the array of annnual disaster counts $\mathbf{y}$. This gives:

$$
 P( \lambda_1, \lambda_2, \tau | \mathbf{y} ) \propto P(\mathbf{y} | \lambda_1, \lambda_2, \tau ) P(\lambda_1, \lambda_2, \tau)
$$

To employ Gibbs sampling, we need to factor the joint posterior into the product of conditional expressions:

$$
 P(\lambda_1, \lambda_2, \tau | \mathbf{y}) \propto P(y_{t \lt \tau} | \lambda_1, \tau) P(y_{t \geq \tau} | \lambda_2, \tau) P(\lambda_1) P(\lambda_2) P(\tau)
$$

which we have specified as:

$$\begin{aligned}
P( \lambda_1, \lambda_2, \tau | \mathbf{y} ) &\propto \left[\prod_{t=1851}^{\tau} \text{Poi}(y_t|\lambda_1) \prod_{t=\tau+1}^{1962} \text{Poi}(y_t|\lambda_2) \right] \text{Gamma}(\lambda_1|\alpha,\beta) \text{Gamma}(\lambda_2|\alpha, \beta) \frac{1}{111} \\
&\propto \left[\prod_{t=1851}^{\tau} e^{-\lambda_1}\lambda_1^{y_t} \prod_{t=\tau+1}^{1962} e^{-\lambda_2} \lambda_2^{y_t} \right] \lambda_1^{\alpha-1} e^{-\beta\lambda_1} \lambda_2^{\alpha-1} e^{-\beta\lambda_2} \\
&\propto \lambda_1^{\sum_{t=1851}^{\tau} y_t +\alpha-1} e^{-(\beta+\tau)\lambda_1} \lambda_2^{\sum_{t=\tau+1}^{1962} y_i + \alpha-1} e^{-\beta\lambda_2}
\end{aligned}$$

So, the full conditionals are known, and critically for Gibbs, can easily be sampled from.

$$\lambda_1 \sim \text{Gamma}(\sum_{t=1851}^{\tau} y_t +\alpha, \tau+\beta)$$
$$\lambda_2 \sim \text{Gamma}(\sum_{t=\tau+1}^{1962} y_i + \alpha, 1962-\tau+\beta)$$
$$\tau \sim \text{Categorical}\left( \frac{\lambda_1^{\sum_{t=1851}^{\tau} y_t +\alpha-1} e^{-(\beta+\tau)\lambda_1} \lambda_2^{\sum_{t=\tau+1}^{1962} y_i + \alpha-1} e^{-\beta\lambda_2}}{\sum_{k=1851}^{1962} \lambda_1^{\sum_{t=1851}^{\tau} y_t +\alpha-1} e^{-(\beta+\tau)\lambda_1} \lambda_2^{\sum_{t=\tau+1}^{1962} y_i + \alpha-1} e^{-\beta\lambda_2}} \right)$$

Implementing this in Python requires random number generators for both the gamma and discrete uniform distributions. We can leverage NumPy for this:

```python
# Function to draw random gamma variate
rgamma = np.random.gamma

def rcategorical(probs, n=None):
    # Function to draw random categorical variate
    return np.array(probs).cumsum().searchsorted(np.random.sample(n))
```

Next, in order to generate probabilities for the conditional posterior of $\tau$, we need the kernel of the gamma density:

\\[\lambda^{\alpha-1} e^{-\beta \lambda}\\]

```python
dgamma = lambda lam, a, b: lam**(a - 1) * np.exp(-b * lam)
```

Diffuse hyperpriors for the gamma priors on $\{\lambda_1, \lambda_2\}$:

```python
alpha, beta = 1., 10
```

For computational efficiency, it is best to pre-allocate memory to store the sampled values. We need 3 arrays, each with length equal to the number of iterations we plan to run:

```python
# Specify number of iterations
n_iterations = 1000

# Initialize trace of samples
lambda1, lambda2, tau = np.empty((3, n_iterations + 1))
```

The penultimate step initializes the model paramters to arbitrary values:

```python
lambda1[0] = 6
lambda2[0] = 2
tau[0] = 50
```

Now we can run the Gibbs sampler.

```python
# Sample from conditionals
for i in range(n_iterations):

    # Sample early mean
    lambda1[i + 1] = rgamma(disasters_array[:tau[i]].sum() + alpha, 1./(tau[i] + beta))

    # Sample late mean
    lambda2[i + 1] = rgamma(disasters_array[tau[i]:].sum() + alpha,
                            1./(n_count_data - tau[i] + beta))

    # Sample changepoint: first calculate probabilities (conditional)
    p = np.array([dgamma(lambda1[i + 1], disasters_array[:t].sum() + alpha, t + beta) *
                  dgamma(lambda2[i + 1], disasters_array[t:].sum() + alpha, n_count_data - t + beta)
                  for t in range(n_count_data)])

    # ... then draw sample
    tau[i + 1] = rcategorical(p/p.sum())
```

Plotting the trace and histogram of the samples reveals the marginal posteriors of each parameter in the model.

```python
color = '#3182bd'
```

```python
trace1 = pgo.Scatter(
    y=lambda1,
    xaxis='x1',
    yaxis='y1',
    line=pgo.Line(width=1),
    marker=pgo.Marker(color=color)
)

trace2 = pgo.Histogram(
    x=lambda1,
    xaxis='x2',
    yaxis='y2',
    line=pgo.Line(width=0.5),
    marker=pgo.Marker(color=color)
)

trace3 = pgo.Scatter(
    y=lambda2,
    xaxis='x3',
    yaxis='y3',
    line=pgo.Line(width=1),
    marker=pgo.Marker(color=color)
)

trace4 = pgo.Histogram(
    x=lambda2,
    xaxis='x4',
    yaxis='y4',
    marker=pgo.Marker(color=color)
)

trace5 = pgo.Scatter(
    y=tau,
    xaxis='x5',
    yaxis='y5',
    line=pgo.Line(width=1),
    marker=pgo.Marker(color=color)
)

trace6 = pgo.Histogram(
    x=tau,
    xaxis='x6',
    yaxis='y6',
    marker=pgo.Marker(color=color)
)
```

```python
data4 = pgo.Data([trace1, trace2, trace3, trace4, trace5, trace6])
```

```python
import plotly.tools as tls
```

```python
fig4 = tls.make_subplots(3, 2)
```

    This is the format of your plot grid:
    [ (1,1) x1,y1 ]  [ (1,2) x2,y2 ]
    [ (2,1) x3,y3 ]  [ (2,2) x4,y4 ]
    [ (3,1) x5,y5 ]  [ (3,2) x6,y6 ]

```python
fig4['data'] += data4
```

```python
def add_style(fig):
    for i in fig['layout'].keys():
        fig['layout'][i]['zeroline'] = False
        fig['layout'][i]['showgrid'] = True
        fig['layout'][i]['gridcolor'] = 'rgb(255, 255, 255)'
    fig['layout']['paper_bgcolor'] = 'rgb(255, 255, 255)'
    fig['layout']['plot_bgcolor'] = 'rgba(204, 204, 204, 0.5)'
    fig['layout']['showlegend']=False
```

```python
add_style(fig4)
```

```python
fig4['layout'].update(
    yaxis1=pgo.YAxis(title=r'$\lambda_1$'),
    yaxis3=pgo.YAxis(title=r'$\lambda_2$'),
    yaxis5=pgo.YAxis(title=r'$\tau$'))
```

```python
py.iplot(fig4, filename='modelling_params')
```

## The Metropolis-Hastings Algorithm

The key to success in applying the Gibbs sampler to the estimation of Bayesian posteriors is being able to specify the form of the complete conditionals of
${\bf \theta}$, because the algorithm cannot be implemented without them. In practice, the posterior conditionals cannot always be neatly specified.

Taking a different approach, the Metropolis-Hastings algorithm generates ***candidate***  state transitions from an alternate distribution, and *accepts* or *rejects* each candidate probabilistically.

Let us first consider a simple Metropolis-Hastings algorithm for a single parameter, $\theta$. We will use a standard sampling distribution, referred to as the *proposal distribution*, to produce candidate variables $q_t(\theta^{\prime} | \theta)$. That is, the generated value, $\theta^{\prime}$, is a *possible* next value for
$\theta$ at step $t+1$. We also need to be able to calculate the probability of moving back to the original value from the candidate, or
$q_t(\theta | \theta^{\prime})$. These probabilistic ingredients are used to define an *acceptance ratio*:

$$\begin{gathered}
\begin{split}a(\theta^{\prime},\theta) = \frac{q_t(\theta^{\prime} | \theta) \pi(\theta^{\prime})}{q_t(\theta | \theta^{\prime}) \pi(\theta)}\end{split}\notag\\\begin{split}\end{split}\notag\end{gathered}$$

The value of $\theta^{(t+1)}$ is then determined by:

$$\theta^{(t+1)} = \left\{\begin{array}{l@{\quad \mbox{with prob.} \quad}l}\theta^{\prime} & \text{with probability } \min(a(\theta^{\prime},\theta^{(t)}),1) \\ \theta^{(t)} & \text{with probability } 1 - \min(a(\theta^{\prime},\theta^{(t)}),1) \end{array}\right.$$

This transition kernel implies that movement is not guaranteed at every step. It only occurs if the suggested transition is likely based on the acceptance ratio.

A single iteration of the Metropolis-Hastings algorithm proceeds as follows:

1.  Sample $\theta^{\prime}$ from $q(\theta^{\prime} | \theta^{(t)})$.

2.  Generate a Uniform[0,1] random variate $u$.

3.  If $a(\theta^{\prime},\theta) > u$ then
    $\theta^{(t+1)} = \theta^{\prime}$, otherwise
    $\theta^{(t+1)} = \theta^{(t)}$.

The original form of the algorithm specified by Metropolis required that
$q_t(\theta^{\prime} | \theta) = q_t(\theta | \theta^{\prime})$, which reduces $a(\theta^{\prime},\theta)$ to
$\pi(\theta^{\prime})/\pi(\theta)$, but this is not necessary. In either case, the state moves to high-density points in the distribution with high probability, and to low-density points with low probability. After convergence, the Metropolis-Hastings algorithm describes the full target posterior density, so all points are recurrent.

### Random-walk Metropolis-Hastings

A practical implementation of the Metropolis-Hastings algorithm makes use of a random-walk proposal.
Recall that a random walk is a Markov chain that evolves according to:

$$
\theta^{(t+1)} = \theta^{(t)} + \epsilon_t \\
\epsilon_t \sim f(\phi)
$$

As applied to the MCMC sampling, the random walk is used as a proposal distribution, whereby dependent proposals are generated according to:

$$\begin{gathered}
\begin{split}q(\theta^{\prime} | \theta^{(t)}) = f(\theta^{\prime} - \theta^{(t)}) = \theta^{(t)} + \epsilon_t\end{split}\notag\\\begin{split}\end{split}\notag\end{gathered}$$

Generally, the density generating $\epsilon_t$ is symmetric about zero,
resulting in a symmetric chain. Chain symmetry implies that
$q(\theta^{\prime} | \theta^{(t)}) = q(\theta^{(t)} | \theta^{\prime})$,
which reduces the Metropolis-Hastings acceptance ratio to:

$$\begin{gathered}
\begin{split}a(\theta^{\prime},\theta) = \frac{\pi(\theta^{\prime})}{\pi(\theta)}\end{split}\notag\\\begin{split}\end{split}\notag\end{gathered}$$

The choice of the random walk distribution for $\epsilon_t$ is frequently a normal or Student’s $t$ density, but it may be any distribution that generates an irreducible proposal chain.

An important consideration is the specification of the **scale parameter** for the random walk error distribution. Large values produce random walk steps that are highly exploratory, but tend to produce proposal values in the tails of the target distribution, potentially resulting in very small acceptance rates. Conversely, small values tend to be accepted more frequently, since they tend to produce proposals close to the current parameter value, but may result in chains that ***mix*** very slowly.

Some simulation studies suggest optimal acceptance rates in the range of 20-50%. It is often worthwhile to optimize the proposal variance by iteratively adjusting its value, according to observed acceptance rates early in the MCMC simulation .

## Example: Linear model estimation

This very simple dataset is a selection of real estate prices \\(p\\), with the associated age \\(a\\) of each house. We wish to estimate a simple linear relationship between the two variables, using the Metropolis-Hastings algorithm.

**Linear model**:

$$\mu_i = \beta_0 + \beta_1 a_i$$

**Sampling distribution**:

$$p_i \sim N(\mu_i, \tau)$$

**Prior distributions**:

$$\begin{aligned}
& \beta_i \sim N(0, 10000) \cr
& \tau \sim \text{Gamma}(0.001, 0.001)
\end{aligned}$$

```python
age = np.array([13, 14, 14,12, 9, 15, 10, 14, 9, 14, 13, 12, 9, 10, 15, 11,
                15, 11, 7, 13, 13, 10, 9, 6, 11, 15, 13, 10, 9, 9, 15, 14,
                14, 10, 14, 11, 13, 14, 10])

price = np.array([2950, 2300, 3900, 2800, 5000, 2999, 3950, 2995, 4500, 2800,
                  1990, 3500, 5100, 3900, 2900, 4950, 2000, 3400, 8999, 4000,
                  2950, 3250, 3950, 4600, 4500, 1600, 3900, 4200, 6500, 3500,
                  2999, 2600, 3250, 2500, 2400, 3990, 4600, 450,4700])/1000.
```

To avoid numerical underflow issues, we typically work with log-transformed likelihoods, so the joint posterior can be calculated as sums of log-probabilities and log-likelihoods.

This function calculates the joint log-posterior, conditional on values for each parameter:

```python
from scipy.stats import distributions
dgamma = distributions.gamma.logpdf
dnorm = distributions.norm.logpdf

def calc_posterior(a, b, t, y=price, x=age):
    # Calculate joint posterior, given values for a, b and t

    # Priors on a,b
    logp = dnorm(a, 0, 10000) + dnorm(b, 0, 10000)
    # Prior on t
    logp += dgamma(t, 0.001, 0.001)
    # Calculate mu
    mu = a + b*x
    # Data likelihood
    logp += sum(dnorm(y, mu, t**-2))

    return logp
```

The `metropolis` function implements a simple random-walk Metropolis-Hastings sampler for this problem. It accepts as arguments:

- the number of iterations to run
- initial values for the unknown parameters
- the variance parameter of the proposal distribution (normal)

```python
rnorm = np.random.normal
runif = np.random.rand

def metropolis(n_iterations, initial_values, prop_var=1):

    n_params = len(initial_values)

    # Initial proposal standard deviations
    prop_sd = [prop_var]*n_params

    # Initialize trace for parameters
    trace = np.empty((n_iterations+1, n_params))

    # Set initial values
    trace[0] = initial_values

    # Calculate joint posterior for initial values
    current_log_prob = calc_posterior(*trace[0])

    # Initialize acceptance counts
    accepted = [0]*n_params

    for i in range(n_iterations):

        if not i%1000: print('Iteration %i' % i)

        # Grab current parameter values
        current_params = trace[i]

        for j in range(n_params):

            # Get current value for parameter j
            p = trace[i].copy()

            # Propose new value
            if j==2:
                # Ensure tau is positive
                theta = np.exp(rnorm(np.log(current_params[j]), prop_sd[j]))
            else:
                theta = rnorm(current_params[j], prop_sd[j])

            # Insert new value
            p[j] = theta

            # Calculate log posterior with proposed value
            proposed_log_prob = calc_posterior(*p)

            # Log-acceptance rate
            alpha = proposed_log_prob - current_log_prob

            # Sample a uniform random variate
            u = runif()

            # Test proposed value
            if np.log(u) < alpha:
                # Accept
                trace[i+1,j] = theta
                current_log_prob = proposed_log_prob
                accepted[j] += 1
            else:
                # Reject
                trace[i+1,j] = trace[i,j]

    return trace, accepted
```

Let's run the MH algorithm with a very small proposal variance:

```python
n_iter = 10000
trace, acc = metropolis(n_iter, initial_values=(1,0,1), prop_var=0.001)
```

    Iteration 0
    Iteration 1000
    Iteration 2000
    Iteration 3000
    Iteration 4000
    Iteration 5000
    Iteration 6000
    Iteration 7000
    Iteration 8000
    Iteration 9000

We can see that the acceptance rate is way too high:

```python
np.array(acc, float)/n_iter
```

    array([ 0.9768,  0.9689,  0.961 ])

```python
trace1 = pgo.Scatter(
    y=trace.T[0],
    xaxis='x1',
    yaxis='y1',
    marker=pgo.Marker(color=color)
)

trace2 = pgo.Histogram(
    x=trace.T[0],
    xaxis='x2',
    yaxis='y2',
    marker=pgo.Marker(color=color)
)

trace3 = pgo.Scatter(
    y=trace.T[1],
    xaxis='x3',
    yaxis='y3',
    marker=pgo.Marker(color=color)
)

trace4 = pgo.Histogram(
    x=trace.T[1],
    xaxis='x4',
    yaxis='y4',
    marker=pgo.Marker(color=color)
)

trace5 = pgo.Scatter(
    y=trace.T[2],
    xaxis='x5',
    yaxis='y5',
    marker=pgo.Marker(color=color)
)

trace6 = pgo.Histogram(
    x=trace.T[2],
    xaxis='x6',
    yaxis='y6',
    marker=pgo.Marker(color=color)
)
```

```python
data5 = pgo.Data([trace1, trace2, trace3, trace4, trace5, trace6])
```

```python
fig5 = tls.make_subplots(3, 2)
```

    This is the format of your plot grid:
    [ (1,1) x1,y1 ]  [ (1,2) x2,y2 ]
    [ (2,1) x3,y3 ]  [ (2,2) x4,y4 ]
    [ (3,1) x5,y5 ]  [ (3,2) x6,y6 ]

```python
fig5['data'] += data5
```

```python
add_style(fig5)
```

```python
fig5['layout'].update(showlegend=False,
                     yaxis1=pgo.YAxis(title='intercept'),
                     yaxis3=pgo.YAxis(title='slope'),
                     yaxis5=pgo.YAxis(title='precision')
)
```

```python
py.iplot(fig5, filename='MH algorithm small proposal variance')
```

Now, with a very large proposal variance:

```python
trace_hivar, acc = metropolis(n_iter, initial_values=(1,0,1), prop_var=100)
```

    Iteration 0
    Iteration 1000
    Iteration 2000
    Iteration 3000
    Iteration 4000
    Iteration 5000
    Iteration 6000
    Iteration 7000
    Iteration 8000
    Iteration 9000

```python
np.array(acc, float)/n_iter
```

    array([ 0.003 ,  0.0001,  0.0009])

```python
trace1 = pgo.Scatter(
    y=trace_hivar.T[0],
    xaxis='x1',
    yaxis='y1',
    marker=pgo.Marker(color=color)
)

trace2 = pgo.Histogram(
    x=trace_hivar.T[0],
    xaxis='x2',
    yaxis='y2',
    marker=pgo.Marker(color=color)
)

trace3 = pgo.Scatter(
    y=trace_hivar.T[1],
    xaxis='x3',
    yaxis='y3',
    marker=pgo.Marker(color=color)
)

trace4 = pgo.Histogram(
    x=trace_hivar.T[1],
    xaxis='x4',
    yaxis='y4',
    marker=pgo.Marker(color=color)
)

trace5 = pgo.Scatter(
    y=trace_hivar.T[2],
    xaxis='x5',
    yaxis='y5',
    marker=pgo.Marker(color=color)
)

trace6 = pgo.Histogram(
    x=trace_hivar.T[2],
    xaxis='x6',
    yaxis='y6',
    marker=pgo.Marker(color=color)
)
```

```python
data6 = pgo.Data([trace1, trace2, trace3, trace4, trace5, trace6])
```

```python
fig6 = tls.make_subplots(3, 2)
```

    This is the format of your plot grid:
    [ (1,1) x1,y1 ]  [ (1,2) x2,y2 ]
    [ (2,1) x3,y3 ]  [ (2,2) x4,y4 ]
    [ (3,1) x5,y5 ]  [ (3,2) x6,y6 ]

```python
fig6['data'] += data6
```

```python
add_style(fig6)
```

```python
fig6['layout'].update(
    yaxis1=pgo.YAxis(title='intercept'),
    yaxis3=pgo.YAxis(title='slope'),
    yaxis5=pgo.YAxis(title='precision')
)
```

```python
py.iplot(fig6, filename='MH algorithm large proposal variance')
```

### Adaptive Metropolis

In order to avoid having to set the proposal variance by trial-and-error, we can add some tuning logic to the algorithm. The following implementation of Metropolis-Hastings reduces proposal variances  by 10% when the acceptance rate is low, and increases it by 10% when the acceptance rate is high.

```python
def metropolis_tuned(n_iterations, initial_values, f=calc_posterior, prop_var=1,
                     tune_for=None, tune_interval=100):

    n_params = len(initial_values)

    # Initial proposal standard deviations
    prop_sd = [prop_var] * n_params

    # Initialize trace for parameters
    trace = np.empty((n_iterations+1, n_params))

    # Set initial values
    trace[0] = initial_values
    # Initialize acceptance counts
    accepted = [0]*n_params

    # Calculate joint posterior for initial values
    current_log_prob = f(*trace[0])

    if tune_for is None:
        tune_for = n_iterations/2

    for i in range(n_iterations):

        if not i%1000: print('Iteration %i' % i)

        # Grab current parameter values
        current_params = trace[i]

        for j in range(n_params):

            # Get current value for parameter j
            p = trace[i].copy()

            # Propose new value
            if j==2:
                # Ensure tau is positive
                theta = np.exp(rnorm(np.log(current_params[j]), prop_sd[j]))
            else:
                theta = rnorm(current_params[j], prop_sd[j])

            # Insert new value
            p[j] = theta

            # Calculate log posterior with proposed value
            proposed_log_prob = f(*p)

            # Log-acceptance rate
            alpha = proposed_log_prob - current_log_prob

            # Sample a uniform random variate
            u = runif()

            # Test proposed value
            if np.log(u) < alpha:
                # Accept
                trace[i+1,j] = theta
                current_log_prob = proposed_log_prob
                accepted[j] += 1
            else:
                # Reject
                trace[i+1,j] = trace[i,j]

            # Tune every 100 iterations
            if (not (i+1) % tune_interval) and (i < tune_for):

                # Calculate aceptance rate
                acceptance_rate = (1.*accepted[j])/tune_interval
                if acceptance_rate<0.1:
                    prop_sd[j] *= 0.9
                if acceptance_rate<0.2:
                    prop_sd[j] *= 0.95
                if acceptance_rate>0.4:
                    prop_sd[j] *= 1.05
                elif acceptance_rate>0.6:
                    prop_sd[j] *= 1.1

                accepted[j] = 0

    return trace[tune_for:], accepted
```

```python
trace_tuned, acc = metropolis_tuned(n_iter*2, initial_values=(1,0,1), prop_var=5, tune_interval=25, tune_for=n_iter)
```

    Iteration 0
    Iteration 1000
    Iteration 2000
    Iteration 3000
    Iteration 4000
    Iteration 5000
    Iteration 6000
    Iteration 7000
    Iteration 8000
    Iteration 9000
    Iteration 10000
    Iteration 11000
    Iteration 12000
    Iteration 13000
    Iteration 14000
    Iteration 15000
    Iteration 16000
    Iteration 17000
    Iteration 18000
    Iteration 19000

```python
np.array(acc, float)/(n_iter)
```

    array([ 0.2888,  0.312 ,  0.3421])

```python
trace1 = pgo.Scatter(
    y=trace_tuned.T[0],
    xaxis='x1',
    yaxis='y1',
    line=pgo.Line(width=1),
    marker=pgo.Marker(color=color)
)

trace2 = pgo.Histogram(
    x=trace_tuned.T[0],
    xaxis='x2',
    yaxis='y2',
    marker=pgo.Marker(color=color)
)

trace3 = pgo.Scatter(
    y=trace_tuned.T[1],
    xaxis='x3',
    yaxis='y3',
    line=pgo.Line(width=1),
    marker=pgo.Marker(color=color)
)

trace4 = pgo.Histogram(
    x=trace_tuned.T[1],
    xaxis='x4',
    yaxis='y4',
    marker=pgo.Marker(color=color)
)

trace5 = pgo.Scatter(
    y=trace_tuned.T[2],
    xaxis='x5',
    yaxis='y5',
    line=pgo.Line(width=0.5),
    marker=pgo.Marker(color=color)
)

trace6 = pgo.Histogram(
    x=trace_tuned.T[2],
    xaxis='x6',
    yaxis='y6',
    marker=pgo.Marker(color=color)
)
```

```python
data7 = pgo.Data([trace1, trace2, trace3, trace4, trace5, trace6])
```

```python
fig7 = tls.make_subplots(3, 2)
```

    This is the format of your plot grid:
    [ (1,1) x1,y1 ]  [ (1,2) x2,y2 ]
    [ (2,1) x3,y3 ]  [ (2,2) x4,y4 ]
    [ (3,1) x5,y5 ]  [ (3,2) x6,y6 ]

```python
fig7['data'] += data7
```

```python
add_style(fig7)
```

```python
fig7['layout'].update(
    yaxis1=pgo.YAxis(title='intercept'),
    yaxis3=pgo.YAxis(title='slope'),
    yaxis5=pgo.YAxis(title='precision')
)
```

```python
py.iplot(fig7, filename='adaptive-metropolis')
```

50 random regression lines drawn from the posterior:

```python
# Data points
points = pgo.Scatter(
    x=age,
    y=price,
    mode='markers'
)

# Sample models from posterior
xvals = np.linspace(age.min(), age.max())
line_data = [np.column_stack([np.ones(50), xvals]).dot(trace_tuned[np.random.randint(0, 1000), :2]) for i in range(50)]

# Generate Scatter obejcts
lines = [pgo.Scatter(x=xvals, y=line, opacity=0.5, marker=pgo.Marker(color='#e34a33'),
                     line=pgo.Line(width=0.5)) for line in line_data]

data8 = pgo.Data([points] + lines)

layout8 = layout_grey_bg.copy()
layout8.update(
    showlegend=False,
    hovermode='closest',
    xaxis=pgo.XAxis(title='Age', showgrid=False, zeroline=False),
    yaxis=pgo.YAxis(title='Price', showline=False, zeroline=False)
)

fig8 = pgo.Figure(data=data8, layout=layout8)
py.iplot(fig8, filename='regression_lines')
```

## Exercise: Bioassay analysis

Gelman et al. (2003) present an example of an acute toxicity test, commonly performed on animals to estimate the toxicity of various compounds.

In this dataset `log_dose` includes 4 levels of dosage, on the log scale, each administered to 5 rats during the experiment. The response variable is `death`, the number of positive responses to the dosage.

The number of deaths can be modeled as a binomial response, with the probability of death being a linear function of dose:

<div style="font-size: 150%;">
$$\begin{aligned}
y_i &\sim \text{Bin}(n_i, p_i) \\
\text{logit}(p_i) &= a + b x_i
\end{aligned}$$
</div>

The common statistic of interest in such experiments is the **LD50**, the dosage at which the probability of death is 50%.

Use Metropolis-Hastings sampling to fit a Bayesian model to analyze this bioassay data, and to estimate LD50.

```python
# Log dose in each group
log_dose = [-.86, -.3, -.05, .73]

# Sample size in each group
n = 5

# Outcomes
deaths = [0, 1, 3, 5]
```

```python
from scipy.stats import distributions
dbin = distributions.binom.logpmf
dnorm = distributions.norm.logpdf

invlogit = lambda x: 1./(1 + np.exp(-x))

def calc_posterior(a, b, y=deaths, x=log_dose):

    # Priors on a,b
    logp = dnorm(a, 0, 10000) + dnorm(b, 0, 10000)
    # Calculate p
    p = invlogit(a + b*np.array(x))
    # Data likelihood
    logp += sum([dbin(yi, n, pi) for yi,pi in zip(y,p)])

    return logp
```

```python
bioassay_trace, acc = metropolis_tuned(n_iter, f=calc_posterior, initial_values=(1,0), prop_var=5, tune_for=9000)
```

    Iteration 0
    Iteration 1000
    Iteration 2000
    Iteration 3000
    Iteration 4000
    Iteration 5000
    Iteration 6000
    Iteration 7000
    Iteration 8000
    Iteration 9000

```python
trace1 = pgo.Scatter(
    y=bioassay_trace.T[0],
    xaxis='x1',
    yaxis='y1',
    marker=pgo.Marker(color=color)
)

trace2 = pgo.Histogram(
    x=bioassay_trace.T[0],
    xaxis='x2',
    yaxis='y2',
    marker=pgo.Marker(color=color)
)

trace3 = pgo.Scatter(
    y=bioassay_trace.T[1],
    xaxis='x3',
    yaxis='y3',
    marker=pgo.Marker(color=color)
)

trace4 = pgo.Histogram(
    x=bioassay_trace.T[1],
    xaxis='x4',
    yaxis='y4',
    marker=pgo.Marker(color=color)
)
```

```python
data9 = pgo.Data([trace1, trace2, trace3, trace4])
```

```python
fig9 = tls.make_subplots(2, 2)
```

    This is the format of your plot grid:
    [ (1,1) x1,y1 ]  [ (1,2) x2,y2 ]
    [ (2,1) x3,y3 ]  [ (2,2) x4,y4 ]

```python
fig9['data'] += data9
```

```python
add_style(fig9)
```

```python
fig9['layout'].update(
    yaxis1=pgo.YAxis(title='intercept'),
    yaxis3=pgo.YAxis(title='slope')
)
```

```python
py.iplot(fig9, filename='bioassay')
```

```python
from IPython.display import display, HTML

display(HTML('<link href="//fonts.googleapis.com/css?family=Open+Sans:600,400,300,200|Inconsolata|Ubuntu+Mono:400,700" rel="stylesheet" type="text/css" />'))
display(HTML('<link rel="stylesheet" type="text/css" href="http://help.plot.ly/documentation/all_static/css/ipython-notebook-custom.css">'))

! pip install publisher --upgrade
import publisher
publisher.publish(
    'montecarlo.ipynb', 'ipython-notebooks/computational-bayesian-analysis/',
    'Computational Methods in Bayesian Analysis',
    'Monte Carlo simulations, Markov chains, Gibbs sampling illustrated in Plotly',
    name='Computational Methods in Bayesian Analysis')
```

<link href="//fonts.googleapis.com/css?family=Open+Sans:600,400,300,200|Inconsolata|Ubuntu+Mono:400,700" rel="stylesheet" type="text/css" />

<link rel="stylesheet" type="text/css" href="http://help.plot.ly/documentation/all_static/css/ipython-notebook-custom.css">

    Requirement already up-to-date: publisher in /Users/chelsea/venv/venv2.7/lib/python2.7/site-packages

    /Users/chelsea/venv/venv2.7/lib/python2.7/site-packages/IPython/nbconvert.py:13: ShimWarning: The `IPython.nbconvert` package has been deprecated since IPython 4.0. You should import from nbconvert instead.
      "You should import from nbconvert instead.", ShimWarning)
    /Users/chelsea/venv/venv2.7/lib/python2.7/site-packages/publisher/publisher.py:53: UserWarning: Did you "Save" this notebook before running this command? Remember to save, always save.
      warnings.warn('Did you "Save" this notebook before running this command? '

```python

```


## 적분
- 미분과 반대되는 개념
- 부정적분(indefinite integral)
- 정적분(definite integral)

### 부정적분(indefinite integral)
- **정확하게 미분과 반대되는 개념, 반-미분(anti-derivative)**

- 부정적분으로 찾은 원래의 함수를 표기
    - $\int$ 기호(integral)로 나타내는 것이 일반적

- 도함수가 $f(x)$이므로 미분하기 전의 함수를 $F(x)$ 또는 $\int f(x)dx$로 쓴다.
- $dx$는 $x$라는 변수로 적분했다는 것을 나타내는 기호로 편미분에 대응하는 적분을 표기할 때 필요하다.

    $
    \dfrac{dF(x)}{dx} = f(x) \;\;\leftrightarrow\;\; F(x) = \int_{}^{} f(x) dx + C
    $

- $C$는 상수항을 의미
    - 상수항은 미분하면 0이 되므로 부정적분은 무한 개의 해가 존재
    - $C$는 당연하므로 생략하고 쓰는 경우도 있다.

### 편미분의 부정적분
- 편미분을 한 도함수에서 원래의 함수를 찾을 수도 있다.
- $f(x, y)$가 함수 $F(x, y)$를 $x$로 편미분한 함수일 때

    $
    \dfrac{\partial F(x, y)}{\partial x} = f(x, y) \; \leftrightarrow \; F(x, y) = \int_{}^{} f(x, y) dx + C(y)
    $

- $f(x, y)$가 함수 $F(x, y)$를 $y$로 편미분한 함수일 때

    $
    \dfrac{\partial F(x, y)}{\partial y} = f(x, y) \; \leftrightarrow \; F(x, y) = \int_{}^{} f(x, y) dy + C(x)
    $

### 다차 도함수와 다중적분
- 미분을 여러번 한 결과로 나온 다차 도함수로부터 원래의 함수를 찾아낼때

    $
    \iint f(x, y) dydx
    $

## 심파이를 이용한 부정적분
- 심파이의 `integrate()` 명령을 사용

```python
import sympy

sympy.init_printing(use_latex='mathjax')

x = sympy.symbols('x')
f = x * sympy.exp(x)
f
```

$$x e^{x}$$

```python
sympy.integrate(f)
```

$$\left(x - 1\right) e^{x}$$

### 정적분
- **독립변수 $x$가 어떤 구간 $[a, b]$ 사이일 때 그 구간에서 함수 $f(x)$의 값과 x 축이 이루는 면적**을 구하는 행위 or 값

    $
    \int_{a}^{b} f(x) dx
    $
- 정적분 구하는 방법
    - 미적분학의 기본 정리를 사용하여 푸는 방법
    - 수치적분(numerical integration)방법

#### 미적분학의 기본 정리(Fundamental Theorem of Calculus)
- 부정적분으로 구한 함수 $F(x)$를 이용하면 다음처럼 정적분의 값을 구할 수 있다.

    $
    \int_{a}^{b} f(x) dx = F(b) - F(a)
    $

### 수치적분
- 함수를 아주 작은 구간으로 나누어 실제 면적을 계산해 값을 구하는 방법.

```python
import scipy as sp
def f(x):
    return 2 * x ** 3 - 4 * x ** 2 + x + 8
sp.integrate.quad(f, 0, 2)  # 수치적분, Scipy의 integrate 서브패키지의 quad 명령
```

$$\left ( 15.333333333333332, \quad 1.7023419710919066e-13\right )$$

### 다변수 정적분
- 입력 변수가 2개인 2차원 함수 $ f(x, y) $
- 두 변수로 모두 적분하는 것은 2차원 평면에서 주어진 사각형 영역 아래의 부피를 구하는 것과 같다.

    $
    \int_{y=c}^{y=d} \int_{x=a}^{x=b} f(x, y) dx dy
    $

### 다차원 함수의 단일 정적분
- 2차원 함수이지만 이중적분을 하지 않고 단일 정적분을 하는 경우
    - 하나의 변수만 진짜 변수로 보고 나머지 하나는 상수라고 간주하는 경우

    $
    \int_a^b f(x, y) dx
    $


## The index of a saddle point

We consider a  *n* degree-of-freedom Hamiltonian of the following form:

\begin{equation}
H(q, p) = \sum_{i=1}^{n} \frac{p_i^2}{2} + V(q), \quad (q,p) \in \mathbb{R}^n \times \mathbb{R}^n,
\label{ham_int}
\end{equation}

where $q \in \mathbb{R}^n$ denote the configuration space variables and $p \in \mathbb{R}^n$ denote the corresponding conjugate momentum variables.  This Hamiltonian function gives rise to the corresponding Hamilton's differential equations (or just ''Hamilton's equations'') having the following form:

\begin{eqnarray}
\dot{q}_i & = & p_i, \nonumber \\
\dot{p}_i & = & -\frac{\partial V}{\partial q_i} (q), \quad i=1. \ldots , n.
\label{hameq_int}
\end{eqnarray}

These are a set of *2n* first order differential equations defined on the phase space
$\mathbb{R}^n \times \mathbb{R}^n$.

A critical point of the potential energy function is a point $\bar{q} \in \mathbb{R}^n$ satisfying the following equations:

\begin{equation}
\frac{\partial V}{\partial q_i} (\bar{q}) =0, \quad i=1, \ldots n.
\end{equation}

Once a critical point of the potential energy function is located, we want to ''classify'' it. This is done by examining the second derivative of the potential energy function evaluated at the critical point. The second derivative matrix  is referred to as the *Hessian matrix*, and it is given by:

\begin{equation}
\frac{\partial^2 V}{\partial q_i \partial q_j} (\bar{q}) =0, \quad i,j=1, \ldots n,
\label{hessian}
\end{equation}

which is a $n \times n$ symmetric matrix. Hence \eqref{hessian} has *n* real eigenvalues, which we denote by:

\begin{equation}
\sigma_k, \quad k=1, \ldots, n.
\label{eiv_Hess}
\end{equation}

However, returning to dynamics as  given by Hamilton's equations \eqref{hameq_int}, the point $(\bar{q}, 0)$ is an equilibrium point of Hamilton's equations, i.e. when this point is substituted into the right-hand-side of \eqref{hameq_int} we obtain $(\dot{q}_1, \ldots, \dot{q}_n, \dot{p}_1, \ldots, \dot{p}_n) = (0, \ldots, 0, 0, \ldots, 0)$, i.e. the point $(\bar{q}, 0)$ does not change in time.

Next, we want to determine the nature of the stability of this equilibrium point. Linearized stability is determined by computing the Jacobian of the right hand side of  \eqref{hameq_int}, which we will denote by $M$, evaluating it at the equilibrium point $(\bar{q}, 0)$, and determining its eigenvalues.  The following calculation  is from {% cite ezra2004impenetrable --file reaction_dynamics %}.
The Jacobian of the Hamiltonian vector field  \eqref{hameq_int} evaluated at $(\bar{q}, 0)$ is given by:

\begin{equation}
M =
\left(
\begin{array}{cc}
0_{n\times n} &  \rm{id}_{n \times n} \\
-\frac{\partial^2 V}{\partial q_i \partial q_j} (\bar{q}) & 0_{n\times n}
\end{array}
\right),
\end{equation}

which is a $2n \times 2n$ matrix.  The eigenvalues of $M$, denoted by $\lambda$,  are given by the solutions of the following characteristic equation:

\begin{equation}
{\rm det} \, \left( M - \lambda \, {\rm id}_{2n \times 2n} \right) =0,
\label{eivM}
\end{equation}

where ${\rm id}_{2n \times 2n}$ denoted the $2n \times 2n$ identity matrix. Writing \eqref{eivM} in detail (i.e. using the explicit expression for the Jacobian of \eqref{hameq_int}) gives:

\begin{equation}
{\rm det} \,
\left(
\begin{array}{cc}
-\lambda \, \rm{id}_{n \times n} & \rm{id}_{n \times n} \\
 -\frac{\partial^2 V}{\partial q_i \partial q_j} (\bar{q}) & -\lambda \rm{id}_{n \times n}
 \end{array}
 \right) =  {\rm det} \,  \left(\lambda^2 \, \rm{id}_{n \times n}  + \frac{\partial^2 V}{\partial q_i \partial q_j} (\bar{q})  \right) =0.
 \end{equation}

We can conclude from this calculation that the eigenvalues of the $n \times n$ symmetric matrix $\frac{\partial^2 V}{\partial q_i \partial q_j} (\bar{q})$ are $-\lambda^2$, where $\lambda$ are the eigenvalues of  the $n \times n$ matrix $M$. Hence, the eigenvalues of $M$ occur in pairs,  denoted by
 $\lambda_k, \, \lambda_{k+n}, \, k=1, \ldots n$, which have the form:

\begin{equation}
\lambda_k, \, \lambda_{k+n} = \pm \sqrt{-\sigma_k},  \quad k=1, \ldots, n,
\end{equation}

where $\sigma_k$ are the eigenvalues of the Hessian of the potential energy evaluated at the critical point $\bar{q}$ as denoted in \eqref{eiv_Hess}.  Hence,
we see that  the existence of equilibrium points of Hamilton's equations of ''saddle-like stability''  implies that there must be *at least* one negative eigenvalue of \eqref{hessian}. In fact, we have the following classification of the linearized stability of saddle-type equilibrium points of Hamilton's equations in terms of  the  critical points of the  potential energy surface.

+ **Index 1 saddle.** One eigenvalue of \eqref{hessian} is positive, the rest are  negative. We will assume that none of the eigenvalues of \eqref{hessian} are zero. Zero eigenvalues  give rise to special cases that must be dealt with separately. In the mathematics literature, these are often referred to as *saddle-center-$\cdots$-center equilibria*, with the number of center-$\cdots$-center terms equal to the number of pairs of pure imaginary eigenvalues.

+ **Index 2 saddle.**  Two eigenvalues of \eqref{hessian} are positive, the rest are  negative

and in general,

+ **Index k saddle.**  *k* eigenvalues of \eqref{hessian} are positive,thevrestvare negativev($k \le n$).

## References

{% bibliography --file reaction_dynamics --cited %}


# Exercise 2
Write a function to compute the roots of a mathematical equation of the form
\begin{align}
  ax^{2} + bx + c = 0.
\end{align}
Your function should be sensitive enough to adapt to situations in which a user might accidentally set $a=0$, or $b=0$, or even $a=b=0$.  For example, if $a=0, b\neq 0$, your function should print a warning and compute the roots of the resulting linear function.  It is up to you on how to handle the function header: feel free to use default keyword arguments, variable positional arguments, variable keyword arguments, or something else as you see fit.  Try to make it user friendly.

Your function should return a tuple containing the roots of the provided equation.

**Hint:** Quadratic equations can have complex roots of the form $r = a + ib$ where $i=\sqrt{-1}$ (Python uses the notation $j=\sqrt{-1}$).  To deal with complex roots, you should import the `cmath` library and use `cmath.sqrt` when computing square roots.  `cmath` will return a complex number for you.  You could handle complex roots yourself if you want, but you might as well use available libraries to save some work.

```python
import cmath
def find_root(a,b,c):
    if (a==0 and b==0 and c==0):
        print("warning!\n x has infinite numbers")
        return()
    elif (a==0 and b==0 and c!=0):
        print("error!\n no x")
        return()
    elif (a==0 and b!=0):
        print("warning!\n x=",-c/b)
        return(-c/b)
    else:
        x1=(-b+cmath.sqrt(b*b-4*a*c))/(2*a)
        x2=(-b-cmath.sqrt(b*b-4*a*c))/(2*a)
        print("x1=",x1)
        print("x2=",x2)
        return(x1,x2)

find_root(0,0,0)
```

    warning!
     x has infinite numbers

    ()

```python

```


# Neural Network Fundamentals

## Gradient Descent Introduction:
https://www.youtube.com/watch?v=IxBYhjS295w

```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression

np.random.seed(1)

%matplotlib inline
np.random.seed(1)
```

```python
N = 100
x = np.random.rand(N,1)*5
# Let the following command be the true function
y = 2.3 + 5.1*x
# Get some noisy observations
y_obs = y + 2*np.random.randn(N,1)
```

```python
plt.scatter(x,y_obs,label='Observations')
plt.plot(x,y,c='r',label='True function')
plt.legend()
plt.show()
```

## Gradient Descent

We are trying to minimise $\sum \xi_i^2$.

\begin{align}
\mathcal{L} & = \frac{1}{N}\sum_{i=1}^N (y_i-f(x_i,w,b))^2 \\
\frac{\delta\mathcal{L}}{\delta w} & = -\frac{1}{N}\sum_{i=1}^N 2(y_i-f(x_i,w,b))\frac{\delta f(x_i,w,b)}{\delta w} \\
& = -\frac{1}{N}\sum_{i=1}^N 2\xi_i\frac{\delta f(x_i,w,b)}{\delta w}
\end{align}
where $\xi_i$ is the error term $y_i-f(x_i,w,b)$ and
$$
\frac{\delta f(x_i,w,b)}{\delta w} = x_i
$$

Similar expression can be found for $\frac{\delta\mathcal{L}}{\delta b}$ (exercise).

Finally the weights can be updated as $w_{new} = w_{current} - \gamma \frac{\delta\mathcal{L}}{\delta w}$ where $\gamma$ is a learning rate between 0 and 1.

```python
# Helper functions
def f(w,b):
    return w*x+b

def loss_function(e):
    L = np.sum(np.square(e))/N
    return L

def dL_dw(e,w,b):
    return -2*np.sum(e*df_dw(w,b))/N

def df_dw(w,b):
    return x

def dL_db(e,w,b):
    return -2*np.sum(e*df_db(w,b))/N

def df_db(w,b):
    return np.ones(x.shape)
```

```python
# The Actual Gradient Descent
def gradient_descent(iter=100,gamma=0.1):
    # get starting conditions
    w = 10*np.random.randn()
    b = 10*np.random.randn()

    params = []
    loss = np.zeros((iter,1))
    for i in range(iter):
#         from IPython.core.debugger import Tracer; Tracer()()
        params.append([w,b])
        e = y_obs - f(w,b) # Really important that you use y_obs and not y (you do not have access to true y)
        loss[i] = loss_function(e)

        #update parameters
        w_new = w - gamma*dL_dw(e,w,b)
        b_new = b - gamma*dL_db(e,w,b)
        w = w_new
        b = b_new

    return params, loss

params, loss = gradient_descent()
```

```python
iter=100
gamma = 0.1
w = 10*np.random.randn()
b = 10*np.random.randn()

params = []
loss = np.zeros((iter,1))
for i in range(iter):
#         from IPython.core.debugger import Tracer; Tracer()()
    params.append([w,b])
    e = y_obs - f(w,b) # Really important that you use y_obs and not y (you do not have access to true y)
    loss[i] = loss_function(e)

    #update parameters
    w_new = w - gamma*dL_dw(e,w,b)
    b_new = b - gamma*dL_db(e,w,b)
    w = w_new
    b = b_new
```

```python
dL_dw(e,w,b)
```

    0.0078296405377948283

```python
plt.plot(loss)
```

```python
params = np.array(params)
plt.plot(params[:,0],params[:,1])
plt.title('Gradient descent')
plt.xlabel('w')
plt.ylabel('b')
plt.show()
```

```python
params[-1]
```

    array([ 4.98099133,  2.75130442])

## Multivariate case

We are trying to minimise $\sum \xi_i^2$. This time $ f = Xw$ where $w$ is Dx1 and $X$ is NxD.

\begin{align}
\mathcal{L} & = \frac{1}{N} (y-Xw)^T(y-Xw) \\
\frac{\delta\mathcal{L}}{\delta w} & = -\frac{1}{N} 2\left(\frac{\delta f(X,w)}{\delta w}\right)^T(y-Xw) \\
& = -\frac{2}{N} \left(\frac{\delta f(X,w)}{\delta w}\right)^T\xi
\end{align}
where $\xi_i$ is the error term $y_i-f(X,w)$ and
$$
\frac{\delta f(X,w)}{\delta w} = X
$$

Finally the weights can be updated as $w_{new} = w_{current} - \gamma \frac{\delta\mathcal{L}}{\delta w}$ where $\gamma$ is a learning rate between 0 and 1.

```python
N = 1000
D = 5
X = 5*np.random.randn(N,D)
w = np.random.randn(D,1)
y = X.dot(w)
y_obs = y + np.random.randn(N,1)
```

```python
w
```

    array([[-0.19670626],
           [-0.74645194],
           [ 0.93774813],
           [-2.62540124],
           [ 0.74616483]])

```python
X.shape
```

    (1000, 5)

```python
w.shape
```

    (5, 1)

```python
(X*w.T).shape
```

    (1000, 5)

```python
# Helper functions
def f(w):
    return X.dot(w)

def loss_function(e):
    L = e.T.dot(e)/N
    return L

def dL_dw(e,w):
    return -2*X.T.dot(e)/N
```

```python
def gradient_descent(iter=100,gamma=1e-3):
    # get starting conditions
    w = np.random.randn(D,1)
    params = []
    loss = np.zeros((iter,1))
    for i in range(iter):
        params.append(w)
        e = y_obs - f(w) # Really important that you use y_obs and not y (you do not have access to true y)
        loss[i] = loss_function(e)

        #update parameters
        w = w - gamma*dL_dw(e,w)

    return params, loss

params, loss = gradient_descent()
```

```python
plt.plot(loss)
```

```python
params[-1]
```

    array([[-0.18613581],
           [-0.74929568],
           [ 0.95245495],
           [-2.602146  ],
           [ 0.73246543]])

```python
model = LinearRegression(fit_intercept=False)
model.fit(X,y)
model.coef_.T
```

    array([[-0.19670626],
           [-0.74645194],
           [ 0.93774813],
           [-2.62540124],
           [ 0.74616483]])

```python
# compare parameters side by side
np.hstack([params[-1],model.coef_.T])
```

    array([[ 0.93191854,  0.93774813],
           [-2.62216904, -2.62540124],
           [ 0.74418434,  0.74616483],
           [ 0.64963419,  0.67411002],
           [ 1.00760672,  1.0142675 ]])

## Stochastic Gradient Descent

```python
def dL_dw(X,e,w):
    return -2*X.T.dot(e)/len(X)

def gradient_descent(gamma=1e-3, n_epochs=100, batch_size=20, decay=0.9):
    epoch_run = int(len(X)/batch_size)

    # get starting conditions
    w = np.random.randn(D,1)
    params = []
    loss = np.zeros((n_epochs,1))
    for i in range(n_epochs):
        params.append(w)

        for j in range(epoch_run):
            idx = np.random.choice(len(X),batch_size,replace=False)
            e = y_obs[idx] - X[idx].dot(w) # Really important that you use y_obs and not y (you do not have access to true y)
            #update parameters
            w = w - gamma*dL_dw(X[idx],e,w)
        loss[i] = e.T.dot(e)/len(e)
        gamma = gamma*decay #decay the learning parameter

    return params, loss

params, loss = gradient_descent()
```

```python
plt.plot(loss)
```

```python
np.hstack([params[-1],model.coef_.T])
```

    array([[-0.18387464, -0.19670626],
           [-0.74895223, -0.74645194],
           [ 0.9476572 ,  0.93774813],
           [-2.62055252, -2.62540124],
           [ 0.75009766,  0.74616483]])

```python

```

deepschool.io/Lesson 02 - GradientDescent.ipynb


```python
import sympy as sym
from sympy.polys import subresultants_qq_zz

sym.init_printing()
```

Resultant
----------

If $p$ and $q$ are two polynomials over a commutative ring with identity which can be factored into linear factors,

$$p(x)= a_0 (x - r_1) (x- r_2) \dots (x - r_m) $$

$$q(x)=b_0 (x - s_1)(x - s_2) \dots (x - s_n)$$

then the resultant $R(p,q)$ of $p$ and $q$ is defined as:

$$R(p,q)=a^n_{0}b^m_{0}\prod_{i=1}^{m}\prod_{j=1}^{n}(r_i - s_j)$$

Since the resultant is a symmetric function of the roots of the polynomials $p$ and $q$, it can be expressed as a polynomial in the coefficients of $p$ and $q$.

From the definition, it is clear that the resultant will equal zero if and only if $p$ and $q$ have at least one common root. Thus, the resultant becomes very useful in identifying whether common roots exist.

Sylvester's Resultant
---------------------

It was proven that the determinant of the Sylvester's matrix is equal to the resultant. Assume the two polynomials:

-  $$p(x) = a_0 x_m + a_1 x_{m-1}+\dots+a_{m-1}x+a_m$$

-  $$q(x)=b_0x_n + b_1x_{n-1}+\dots+b_{n-1}x+b_n$$

Then the Sylverster matrix in the $(m+n)\times(m+n)$ matrix:

$$
\left|
\begin{array}{cccccc}
a_{0} & a_{1} & a_{2} & \ldots & a_{m}    & 0     & \ldots &0 \\
0     & a_{0} & a_{1} & \ldots &a_{m-1}   & a_{m} & \ldots &0 \\
\vdots     &     \ddots & \ddots&       \ddots&         \ddots&      \ddots&       \ddots&\vdots \\
0     & 0     & \ddots     &       \ddots&         \ddots&      \ddots&       \ddots&a_{m}\\
b_{0} & b_{1} & b_{2} & \ldots & b_{n}    & 0    & \ldots & 0 \\
0     & b_{0} & b_{1} & \ldots & b_{n-1}  & b_{n} & \ldots & 0\\
\ddots     &\ddots      & \ddots&       \ddots&         \ddots&      \ddots&       \ddots&\ddots \\
0     & 0     &      \ldots&       \ldots&         \ldots&      \ldots&       \ldots& b_{n}\\
\end{array}
\right| = \Delta $$

Thus $\Delta$ is equal to the $R(p, q)$.

Example: Existence of common roots
------------------------------------------

Two examples are consider here. Note that if the system has a common root we are expecting the resultant/determinant to equal to zero.

```python
x = sym.symbols('x')
```

**A common root exists.**

```python
f = x ** 2 - 5 * x + 6
g = x ** 2 - 3 * x + 2
```

```python
f, g
```

```python
subresultants_qq_zz.sylvester(f, g, x)
```

```python
subresultants_qq_zz.sylvester(f, g, x).det()
```

**A common root does not exist.**

```python
z = x ** 2 - 7 * x + 12
h = x ** 2 - x
```

```python
z, h
```

```python
matrix = subresultants_qq_zz.sylvester(z, h, x)
matrix
```

```python
matrix.det()
```

Example: Two variables, eliminator
----------

When we have system of two variables we solve for one and the second is kept as a coefficient.Thus we can find the roots of the equations, that is why the resultant is often refeered to as the eliminator.

```python
y = sym.symbols('y')
```

```python
f = x ** 2 + x * y + 2 * x + y -1
g = x ** 2 + 3 * x - y ** 2 + 2 * y - 1
f, g
```

```python
matrix = subresultants_qq_zz.sylvester(f, g, y)
```

```python
matrix
```

```python
matrix.det().factor()
```

Three roots for x $\in \{-3, 0, 1\}$.

For $x=-3$ then $y=1$.

```python
f.subs({x:-3}).factor(), g.subs({x:-3}).factor()
```

```python
f.subs({x:-3, y:1}), g.subs({x:-3, y:1})
```

For $x=0$ the $y=1$.

```python
f.subs({x:0}).factor(), g.subs({x:0}).factor()
```

```python
f.subs({x:0, y:1}), g.subs({x:0, y:1})
```

For $x=1$ then $y=-1$ is the common root,

```python
f.subs({x:1}).factor(), g.subs({x:1}).factor()
```

```python
f.subs({x:1, y:-1}), g.subs({x:1, y:-1})
```

```python
f.subs({x:1, y:3}), g.subs({x:1, y:3})
```

Example: Generic Case
-----------------

```python
a = sym.IndexedBase("a")
b = sym.IndexedBase("b")
```

```python
f = a[1] * x + a[0]
g = b[2] * x ** 2 + b[1] * x + b[0]
```

```python
matrix = subresultants_qq_zz.sylvester(f, g, x)
```

```python
matrix.det()
```

```python

```


# Parameter Distributions

## Gamma Distribution: DD1
Histology studies show that axons do not have only one fixed diameter throughout the brain, but follow an axon diameter distribution *(Aboitiz et al. 1992)*.
In Microstructure Imaging, the true axon diameter distribution is usually modeled as a Gamma distribution $\Gamma(R;\alpha,\beta)$ with $R$ the axon radius as half its diameter and $\alpha$ and $\beta$ its scale and rate parameters *(Assaf et al. 2008)*.
The probability density of a Gamma distribution is given as

\begin{equation}
 \Gamma(R;\alpha,\beta)=\frac{\beta^\alpha R^{\alpha-1}e^{-R\beta}}{\Gamma(\alpha)}
\end{equation}

with $\Gamma(\alpha)$ a Gamma function.
However, we must take the cross-sectional area of these cylinders into account to relate this Gamma distribution of cylinder radii to the signal attenuation of this distribution of cylinders.
The reason for this is that it is not the cylinders themselves, but the (simulated) particles diffusing inside these cylinders that are contributing to the signal attenuation.
The final perpendicular, intra-cylindrical signal attenuation for a Gamma-distributed cylinder ensemble, using any of the previously describe cylinder representations, is then given as

\begin{equation}
E^{\Gamma}_\perp(q,\Delta,\delta;\alpha,\beta)=\frac{\int_{\mathbb{R}^+}\overbrace{\Gamma(R;\alpha,\beta)}^{\textrm{Gamma Distribution}}\times\overbrace{E_\perp(q,\Delta,\delta,R)}^{\textrm{Cylinder Signal Attenuation}}\times \overbrace{\pi R^2}^{\textrm{Surface Correction}} dR}{\underbrace{\int_{\mathbb{R}^+}\Gamma(R;\alpha,\beta)\times \textrm{N}(R) dR}_{\textrm{Normalization}}}.
\end{equation}

When modeling cylinders, $\textrm{N}(R)$ is the surface function $\textrm{N}(R)=\pi R^2$. But, if we were to model a Gamma distribution of spheres, then it is the volume function $\textrm{N}(R)=(4/3)\pi R^3$. In fact, Dmipy internally checks what model parameter it is distributing, and normalizes the Gamma distribution accordingly.

```python
from dmipy.distributions import distributions
gamma_standard = distributions.DD1Gamma(normalization='standard')
gamma_plane = distributions.DD1Gamma(normalization='plane')
gamma_cylinder = distributions.DD1Gamma(normalization='cylinder')
gamma_sphere = distributions.DD1Gamma(normalization='sphere')

radii_std, Pstd = gamma_standard(alpha=2., beta=1e-6)
radii_pln, Pstd = gamma_plane(alpha=2., beta=1e-6)
radii_cyl, Pcyl = gamma_cylinder(alpha=2., beta=1e-6)
radii_sph, Psph = gamma_sphere(alpha=2., beta=1e-6)

import matplotlib.pyplot as plt
%matplotlib inline
plt.plot(radii_std, Pstd, label='Gamma Standard')
plt.plot(radii_pln, Pstd, label='Gamma Plane')
plt.plot(radii_cyl, Pcyl, label='Gamma Cylinder')
plt.plot(radii_sph, Psph, label='Gamma Sphere')
plt.legend()
plt.xlabel('Radius [m]')
plt.ylabel('Probability Density [-]');
```

Notice that the normalization changes a lot in the apparent shape of the Gamma distribution. The distance between planes linearly with radius, the surface of cylinders grows quadratically with radius, and the volume of spheres grows cubically with radius. This is why the re-normalized distributions shift increasingly to the right.

Upon initialization the Gamma distribution pre-calculates between which radii there is 99% of the volume under the distribution, and only samples there. This is why the different distributions produce samples at different positions, depending on $\alpha, \beta$ and the normalization.

## References

- Aboitiz, Francisco, et al. "Fiber composition of the human corpus callosum." Brain research 598.1 (1992): 143-153.
- Assaf, Yaniv, et al. "AxCaliber: a method for measuring axon diameter distribution from diffusion MRI." Magnetic resonance in medicine 59.6 (2008): 1347-1354.


```python
import sympy as sm
sm.init_printing()
from sympy.abc import x # Normal assumptions
from sympy import pi, sin, cos, integrate, solve
```

```python
n = sm.symbols('n', positive=True, integer=True)
L = sm.symbols('L', positive=True)
integrand = A**2 * sin(n*pi*x/L)**2
integrand
```

```python
psi_integral = sm.integrate(integrand, (x, 0, L))
```

```python
sm.solve(psi_integral - 1, A)
```

```python
sin(x).diff(x)
```

```python

```


# sympy

*sympy* library can be used to handle mathematical formula. **mathemathica** and **matlab** are the famous language in this genre, but they are not open source and expensive software.

The advantage of sympy is that

- calculate algebra
- copy the formula in *latex* format, which can be paste in thesis or technical paper
- (and many more)

### fraction (Rational)

*sympy* supports fraction(Function name is *Rational*) such as 1/3.

```python
import sympy as smp

x = smp.Rational(1,3)
y = smp.Rational(1,2)

print(x, y, x+y)
print(x*3, (x+y)*6)
```

    1/3 1/2 5/6
    1 5

### Irrational Number

*sympy* support irrational number such as $\sqrt{2}, \pi,  e$

```python
root2 = smp.sqrt(2)
print('Root 2:', root2.evalf(100))
print('Pi:    ', smp.pi.evalf(100))
print('e:     ', smp.E.evalf(100))
```

    Root 2: 1.414213562373095048801688724209698078569671875376948073176679737990732478462107038850387534327641573
    Pi:     3.141592653589793238462643383279502884197169399375105820974944592307816406286208998628034825342117068
    e:      2.718281828459045235360287471352662497757247093699959574966967627724076630353547594571382178525166427

### algebra

*sympy* is powerful when solving algebra.

```python
x = smp.Symbol('x')  # Define 1 symbol
y, z = smp.symbols('y z')  # Define 2 or more symbols

f = x**2 - 2*x + 1
print('Factor', smp.factor(f))
print('Solve', smp.solve(f, x))
```

    Factor (x - 1)**2
    Solve [1]

```python
f = x + y -7     # x+y = 7
g = x * y -10    # x*y = 10
smp.solve([f,g])
```

    [{x: 2, y: 5}, {x: 5, y: 2}]

```python
from sympy import symbols, solve, factor

f = x**7 - 1
print(factor(f))
print(solve(f))
```

    (x - 1)*(x**6 + x**5 + x**4 + x**3 + x**2 + x + 1)
    [1, -cos(pi/7) - I*sin(pi/7), -cos(pi/7) + I*sin(pi/7), cos(2*pi/7) - I*sin(2*pi/7), cos(2*pi/7) + I*sin(2*pi/7), -cos(3*pi/7) - I*sin(3*pi/7), -cos(3*pi/7) + I*sin(3*pi/7)]

### Prime factorization

factorint() function of *sympy* performs prime factorization(decomposition).

```python
from sympy import factorint

years = [2017,2018,2019]

for y in years:
    print(factorint(y))
```

    {2017: 1}
    {2: 1, 1009: 1}
    {3: 1, 673: 1}

```python
# print years that are product of 3 different prims

for year in range(1900, 2019):
    primes = factorint(year)
    if sum(primes.values()) == 3 and len(primes) == 3:
        print('Year:', year, 'Prime', primes)
```

    Year: 1902 Prime {2: 1, 3: 1, 317: 1}
    Year: 1905 Prime {3: 1, 5: 1, 127: 1}
    Year: 1910 Prime {2: 1, 5: 1, 191: 1}
    Year: 1918 Prime {2: 1, 7: 1, 137: 1}
    Year: 1930 Prime {2: 1, 5: 1, 193: 1}
    Year: 1946 Prime {2: 1, 7: 1, 139: 1}
    Year: 1947 Prime {3: 1, 11: 1, 59: 1}
    Year: 1955 Prime {5: 1, 17: 1, 23: 1}
    Year: 1958 Prime {2: 1, 11: 1, 89: 1}
    Year: 1965 Prime {3: 1, 5: 1, 131: 1}
    Year: 1970 Prime {2: 1, 5: 1, 197: 1}
    Year: 1978 Prime {2: 1, 23: 1, 43: 1}
    Year: 1986 Prime {2: 1, 3: 1, 331: 1}
    Year: 1990 Prime {2: 1, 5: 1, 199: 1}
    Year: 2001 Prime {3: 1, 23: 1, 29: 1}
    Year: 2006 Prime {2: 1, 17: 1, 59: 1}
    Year: 2013 Prime {3: 1, 11: 1, 61: 1}
    Year: 2014 Prime {2: 1, 19: 1, 53: 1}
    Year: 2015 Prime {5: 1, 13: 1, 31: 1}

```python
factorint(1234567890123456789012345678913)
```

    {13: 1, 199: 1, 263: 1, 467: 1, 184567: 1, 67947437: 1, 309826661: 1}

```python

```


# **Una breve introducción al ecosisteam de Python**

*Por: Martin Vuelta ([zodiacfireworks](https://github.com/zodiacfireworks))*

*Email:* `martin.vuelta@softbutterfly.io`

## 1. ¿Qué es Python?

    Python is a programming language that lets you work more quickly and integrate your systems more effectively.

### *Ejemplos*

a. Hola Mundo

```python
print("Hola Mundo!")
```

b. Saludos

```python
name = input("What's your name? ")
greetings = f"Hello {name}!"
print(greetings)
```

## 2. Módulos

    Consider a module to be the same as a code library. A file or set of files containing a set of functions you want to include in your application.

### *Ejemplos*

a. Gravity

```python
from math import pi, sin, sqrt

def g(phi):
    sin2phi = sin(phi) ** 2
    return 9.7803253359 * ( 1 + 0.001931850400 * sin2phi ) / sqrt(1 - 0.006694384442 * sin2phi)

latitude_deg = input("What's your latitude? [deg]")
latitude_rad = float(latitude_deg) * pi / 180
gravity = f"La gravedad a {latitude_deg}deg es de {g(latitude_rad)} m/s^2"
print(gravity)
```

## 3. Paquetes

    A package, in essence, is a module or a set of modules prepared to be distributed. The most common way of distribution is through the Python Package Index (PyPI).

### 3.1. Paquetes comunes en el Python club

#### Numpy

    Base N-dimensional array package

##### *Ejemplos*

```python
import numpy as np

sample_array = np.array([(1.5,2,3), (4,5,6)])
sample_array
```

```python
import numpy as np

sample_array = np.array([(1.5,2,3), (4,5,6)])
print(sample_array)
```

```python
zeros_array = np.zeros((3,4))
print("Array of zeros")
print(zeros_array)
print()

ones_array = np.ones((2,3,4), dtype=np.int16)
print("Array of ones")
print(ones_array)
print()

empty_array = np.empty((2,3,4))
print("Empty array")
print(empty_array)
```

### Scipy

    Fundamental library for scientific computing

##### *Ejemplos*

```python
from numpy import poly1d

p = poly1d([3,4,5])
p
```

```python
p = poly1d([3,4,5])
print("Polinomio")
print(p)
print()
print("Coeficientes")
print(p.coeffs)
print()
print("Raices")
print(p.roots)
print()
print("Integrate")
print(p.integ())
print()
print("Detivative")
print(p.deriv())
```

```python
import numpy as np
from scipy.fft import fft, ifft
x = np.array([1.0, 2.0, 1.0, -1.0, 1.5])
y = fft(x)
yinv = ifft(y)

print(x)
print(y)
print(yinv)
```

#### Matplotlib y seaborn

    Matplotlib: Comprehensive 2-D plotting

    Seaborn: Statistical data visualization

##### *Ejemplos*

```python
%matplotlib inline
```

```python
import numpy as np
from scipy.fft import fft
import matplotlib.pyplot as plt

N = 600
T = 1.0 / 800.0
x = np.linspace(0.0, N*T, N)
y = np.sin(50.0*2.0*np.pi*x) + 0.5 * np.sin(80.0*2.0*np.pi*x)
yf = fft(y)
xf = np.linspace(0.0, 1.0/(2.0*T), N//2)

plt.plot(xf, 2.0/N * np.abs(yf[0:N//2]))
plt.grid()
plt.show()
```

```python
import seaborn as sns
import matplotlib.pyplot as plt
sns.set(style="whitegrid")

# Load the example diamonds dataset
diamonds = sns.load_dataset("diamonds")

# Draw a scatter plot while assigning point colors and sizes to different
# variables in the dataset
f, ax = plt.subplots(figsize=(6.5, 6.5))
sns.despine(f, left=True, bottom=True)
clarity_ranking = ["I1", "SI2", "SI1", "VS2", "VS1", "VVS2", "VVS1", "IF"]
sns.scatterplot(
    x="carat",
    y="price",
    hue="clarity",
    size="depth",
    palette="ch:r=-.2,d=.3_r",
    hue_order=clarity_ranking,
    sizes=(1, 8),
    linewidth=0,
    data=diamonds,
    ax=ax
)
plt.show()
```

#### Pandas

    Data structures & analysis

##### *Ejemplos*

```python
import pandas as pd

dataframe = pd.read_csv("../datasets/neos.csv")
dataframe
```

```python
dataframe["Perihelion Distance"].min() * 150 * (10 ** 6) / (12.7 * (10 ** 3) )
```

#### Sympy

    Symbolic mathematics

##### *Ejemplos*

```python
from sympy import *

init_printing(use_unicode=True)
```

```python
x = symbols('x')

solveset(3*x**2 + 4*x + 5, x)
```

```python
from sympy import *

f = symbols('f', cls=Function)
x = symbols('x')
```

```python
f(x)
```

```python
f(x).diff(x)
```

```python
diffeq = Eq(f(x).diff(x, x) - 2*f(x).diff(x) + f(x), sin(x))
diffeq
```

```python
dsolve(diffeq, f(x))
```

#### Scikit learn

    Machine Learning in Python

##### *Ejemplos*

```python
import numpy as np
import matplotlib.pyplot as plt
# Though the following import is not directly being used, it is required
# for 3D projection to work
from mpl_toolkits.mplot3d import Axes3D

from sklearn.cluster import KMeans
from sklearn import datasets

np.random.seed(5)

iris = datasets.load_iris()
X = iris.data
y = iris.target

estimators = [
    ('k_means_iris_8', KMeans(n_clusters=8)),
    ('k_means_iris_3', KMeans(n_clusters=3)),
    ('k_means_iris_bad_init', KMeans(n_clusters=3, n_init=1, init='random'))
]

fignum = 1
titles = ['8 clusters', '3 clusters', '3 clusters, bad initialization']

for name, est in estimators:
    fig = plt.figure(fignum, figsize=(10, 10))
    ax = Axes3D(fig, rect=[0, 0, .95, 1], elev=48, azim=134)

    est.fit(X)
    labels = est.labels_

    ax.scatter(
        X[:, 3],
        X[:, 0],
        X[:, 2],
        c=labels.astype(np.float),
        edgecolor='k'
    )

    ax.w_xaxis.set_ticklabels([])
    ax.w_yaxis.set_ticklabels([])
    ax.w_zaxis.set_ticklabels([])
    ax.set_xlabel('Petal width')
    ax.set_ylabel('Sepal length')
    ax.set_zlabel('Petal length')
    ax.set_title(titles[fignum - 1])
    ax.dist = 12
    fignum = fignum + 1

# Plot the ground truth
fig = plt.figure(fignum, figsize=(10, 10))
ax = Axes3D(fig, rect=[0, 0, .95, 1], elev=48, azim=134)

for name, label in [('Setosa', 0), ('Versicolour', 1), ('Virginica', 2)]:
    ax.text3D(
        X[y == label, 3].mean(),
        X[y == label, 0].mean(),
        X[y == label, 2].mean() + 2, name,
        horizontalalignment='center',
        bbox=dict(alpha=.2, edgecolor='w', facecolor='w')
    )

# Reorder the labels to have colors matching the cluster results
y = np.choose(y, [1, 2, 0]).astype(np.float)
ax.scatter(X[:, 3], X[:, 0], X[:, 2], c=y, edgecolor='k')

ax.w_xaxis.set_ticklabels([])
ax.w_yaxis.set_ticklabels([])
ax.w_zaxis.set_ticklabels([])
ax.set_xlabel('Petal width')
ax.set_ylabel('Sepal length')
ax.set_zlabel('Petal length')
ax.set_title('Ground Truth')
ax.dist = 12

plt.show()
```

```python

```


#Hydrogen functions
Start with some imports fro Symbolic Python library:

```
from sympy.physics.hydrogen import R_nl
from sympy.functions.special.spherical_harmonics import Ynm
from sympy import *
```

Define some variables, radial, polar, azimuthal, time, and two frequencies:

```
var("r theta phi t w1 w2")
```

    (r, theta, phi, t, w1, w2)

## Look at a few of the radial equations and the spherical harmonics
Notice that instead of Ylm the name is Ynm... the arguments to the function are still the quantum numbers `l` and `m`

```
R_nl(1, 0, r, 1)  # the n = 1, l = 0 radial function
```

    2*exp(-r)

```
Ynm(0,0,theta,phi).expand(func=True)  # the l = 0, m = 0 spherical harmonic
```

    1/(2*sqrt(pi))

Write the equation for the $|nlm\rangle = |100\rangle$ state. Use the sympy method `.expand(func=True)` to convert to the actual expression. To create this state, we combine the Radial function and the Ylm function. Make sure to set n, l, and m to the correct values. The fourth argument to `R_nl` is `Z` which we set to 1 since we are talking about a 1-proton nucleus.

The combination of R_nl and Ynm should look like the following (replace N, L, and M with the appropriate values):

`R_nl(N, L, r, 1)*Ynm(L, M, theta, phi).expand(func=True)`

```
# this is the |100> state:
psi100 = R_nl(1, 0, r, 1)*Ynm(0,0,theta,phi).expand(func=True)
```

```
psi100  # check to see how it looks as an expression
```

    exp(-r)/sqrt(pi)

##Integrating over all space
Remember spherical coordinate integrals of function $f(r,\theta,\phi)$ over all space look like: $$\int_0^\infty\int_0^\pi\int_0^{2\pi}r^2\sin(\theta)drd\theta d\phi \,\,f(r,\theta,\phi)$$ so you alwasy need to add a factor of `r**2*sin(theta)` and then integrate `r` from 0 to infinity, `theta` from $0-\pi$ and `phi` from $0-2\pi$. As a check, you should integrate the square of the `psi100` wavefunction over all space to see that it equals 1 (i.e. it is normalized)

```
integrate(r**2*sin(theta) * (psi100)**2 ,(r,0,oo),(theta,0,pi),(phi,0,2*pi))
```

    1

## Now do the $|210\rangle$ state:

```
psi210 = R_nl(2, 1, r, 1)*Ynm(1,0,theta,phi).expand(func=True)
```

```
psi210  # check how it looks
```

    sqrt(2)*r*exp(-r/2)*cos(theta)/(8*sqrt(pi))

Note, if you compare these to listed solutions (for example at http://hyperphysics.phy-astr.gsu.edu/hbase/quantum/hydwf.html#c3) you see that there are not any factors of $a_0$. This is because the `R_nl` function is defined in units of $a_0$. $a_0$ is the Bohr Radius: http://en.wikipedia.org/wiki/Bohr_radius

```
psi211 = R_nl(2, 1, r, 1)*Ynm(1,1,theta,phi).expand(func=True)
psi211
```

    -r*sqrt(-cos(theta)**2 + 1)*exp(-r/2)*exp(I*phi)/(8*sqrt(pi))

## Now calculate $\langle z \rangle$:
To calculate $\langle z \rangle$ we need to convert to spherical coordinates: $z = r\cos\theta$. The terms in the following integral are the $r^2\sin\theta$ then $z$ (in spherical coords) then the wave function squared.

```
expect = integrate(r**2*sin(theta)* (r*cos(theta)) * (psi100*psi100),(r,0,oo),(theta,0,pi),(phi,0,2*pi))
```

```
expect
```

    0

No surprise, the average z position of the electron in the hydrogen atom is 0.

## Now for problem 13.21
find $\langle z \rangle(t)$. Use the same integral, but add a time-dependent piece to each term in the wavefunction, add them together and multiply by the complex conjugate.

```
psi = 1/sqrt(2)*(psi100*exp(1j*w1*t) + psi210*exp(1j*w2*t))
psi_conj = 1/sqrt(2)*(psi100*exp(-1j*w1*t) + psi210*exp(-1j*w2*t))
```

```
expect2 = integrate(r**2*sin(theta)* (r*cos(theta)) * psi*psi_conj,(r,0,oo),(theta,0,pi),(phi,0,2*pi))
```

```
expect2
```

    2*pi*(32*sqrt(2)*exp(-1.0*I*t*w1)*exp(1.0*I*t*w2)/(243*pi) + 32*sqrt(2)*exp(1.0*I*t*w1)*exp(-1.0*I*t*w2)/(243*pi))

We need to interpret this result. First you should show that this expression is simply a constant amplitude factor times $\cos((w2-w1)t)$, in other words $\langle z \rangle$ oscillates at frequency `w2-w1`.

## Your assignment:

Explore other combinations of states and draw conclusions about the z behavior from the results. You may not be able to get these expressions to simplify, but the important thing is to look for the time dependence and simplify that part.

- Does $\langle z \rangle$ oscillate for any combination of two Hydrogen states $|nlm\rangle$?
- Are there restrictions on what n values give oscillating $\langle z \rangle$ expressions? (hint, to keep it simple, always let one state be the n=1 state)
- How does $\langle z \rangle$ change with different l and m values are used in the state?

Hints for interpreting your results:
- What are the relavant frequencies in your expression for $\langle z \rangle$ and why?
- Simplify one of your $\langle z \rangle$ expressions and write the time dependence in terms of the frequencies w2 and w1.

```

```


## TNK

Tanaka suggested the following two-variable problem:

**Definition**

\begin{equation}
\newcommand{\boldx}{\mathbf{x}}
\begin{array}
\mbox{Minimize} & f_1(\boldx) = x_1, \\
\mbox{Minimize} & f_2(\boldx) = x_2, \\
\mbox{subject to} & C_1(\boldx) \equiv x_1^2 + x_2^2 - 1 -
0.1\cos \left(16\arctan \frac{x_1}{x_2}\right) \geq 0, \\
& C_2(\boldx) \equiv (x_1-0.5)^2 + (x_2-0.5)^2 \leq 0.5,\\
& 0 \leq x_1 \leq \pi, \\
& 0 \leq x_2 \leq \pi.
\end{array}
\end{equation}

**Optimum**

Since $f_1=x_1$ and $f_2=x_2$, the feasible objective space is also
the same as the feasible decision variable space. The unconstrained
decision variable space consists of all solutions in the square
$0\leq (x_1,x_2)\leq \pi$. Thus, the only unconstrained Pareto-optimal
solution is $x_1^{\ast}=x_2^{\ast}=0$.
However, the inclusion of the first constraint makes this solution
infeasible. The constrained Pareto-optimal solutions lie on the boundary
of the first constraint. Since the constraint function is periodic and
the second constraint function must also be satisfied,
not all solutions on the boundary of the first constraint are Pareto-optimal. The
Pareto-optimal set is disconnected.
Since the Pareto-optimal
solutions lie on a nonlinear constraint surface, an optimization
algorithm may have difficulty in finding a good spread of solutions across
all of the discontinuous Pareto-optimal sets.

**Plot**

```python
from pymoo.factory import get_problem
from pymoo.util.plotting import plot

problem = get_problem("tnk")
plot(problem.pareto_front(), no_fill=True)
```

```python

```


```python
# imports
import sympy as sm
import numpy as np
import math
import matplotlib.pyplot as plt
%matplotlib inline
```

In order to recast the Huber loss as a valid likelihood function to be used in likelihood ratio tests, we need to derive the associated normalisation constant. We can achieve this using sympy, a symbolic computation package.

Using the same notation as in the PyTorchDIA paper, we can start by defining the class of distributions,

$f(x) = Q \times \frac{1}{\sigma} \times \exp({-\rho(x)})$.

$Q$ is the normalisation constant we're interested in deriving. In order to find $Q$, we need to integrate everything to its right over the entire range of $x$. Setting $Q$ equal to the reciprocal of the result of this integration will ensure that when $f(x)$ is integrated over the entire range of $x$ the result will always be $1$ i.e. $f(x)$ is now a valid probability distribution.

We'll start with the Gaussian case, where

$\rho(x) = \frac{1}{2}\frac{x^2}{\sigma^2}\;.$

N.B. Under the above definition for $f(x)$, we already have the $\sigma$ term that turns up in the usual derivation of the normalisation constant for a Gaussian, so integrating this expression should give us $\sqrt{2\pi}$. Let's use sympy to do this for us.

```python
# first, test this out for the gaussian case
x, sigma = sm.symbols("x, sigma", real=True, positive=True)
rho = ((x / sigma) ** 2 )/ 2
loss = sm.exp(-rho) / sigma
norm = 2*sm.integrate(loss, (x, 0, sm.oo))
sm.simplify(norm)
```

$\displaystyle \sqrt{2} \sqrt{\pi}$

OK great, that seems to work! Now let's do the same for the Huber loss.

\begin{equation}
    \rho_{\text{Huber}, c}(x) =
    \begin{cases}
    \frac{1}{2}\frac{x^2}{\sigma^2}, & \text{for } |\frac{x}{\sigma}|\leq c \\
    c(|\frac{x}{\sigma}| - \frac{1}{2}c), & \text{for } |\frac{x}{\sigma}| > c \;.
    \end{cases}
    \label{eq:huber_loss}
\end{equation}

I don't expect the result will be quite as tidy as above!

```python
# normalisation constant for Huber 'likelihood'
x, c, sigma = sm.symbols("x, c, sigma", real=True, positive=True)
rho = sm.Piecewise(
    ((x / sigma) ** 2 / 2, (x / sigma) <= c),
    (c * (sm.Abs(x / sigma) - c / 2), ((x / sigma) > c))
)
loss = sm.exp(-rho) / sigma
norm = 2 * sm.integrate(loss, (x, 0, sm.oo))
sm.simplify(norm)
```

$\displaystyle \sqrt{2} \sqrt{\pi} \operatorname{erf}{\left(\frac{\sqrt{2} c}{2} \right)} + \frac{2 e^{- \frac{c^{2}}{2}}}{c}$

```python
#const = (np.sqrt(2 * np.pi) * math.erf(c / np.sqrt(2))) + ((2 / c) * np.exp(-0.5 * c**2))
```

OK, on to the meat of this notebook. I'm not certain that it makes sense to compare a Huber likelihood against a Gaussian, even for instances in which there are no outlying data points. We can probe this question with a toy example; fitting a straight line to data with known Gaussian uncertainties.

It may be that the Huber likelihood (evaluated at the MLE) is strongly dependent on the tuning paramter, $c$. In the case where $c$ tends to infinity, we should expect the same value for the likelihood as for the Gaussian case. In this special case, not only are all residuals from the model treated as 'inliers', but note too that the normalisation constant we found above would also tend to $\sqrt{2\pi}$; the error function in the first term becomes unity and the latter term becomes neglibily small. However, for useful, smaller values of $c$ e.g. 1.345, in which 'outlying' residuals are treated linearly rather than quadratically, should we expect to get roughly the same values for the likelihoods? In other words, does the linear treatment of outliers balance with the change to the normalisation constant, which is itself dependent on $c$. I don't see how it could... but's let's verify this numerically.

```python
## Generate some mock data

# The linear model with slope 2 and intercept 1:
true_params = [2.0, 1.0]

# points drawn from true model
np.random.seed(42)
x = np.sort(np.random.uniform(-2, 2, 30))
yerr = 0.4 * np.ones_like(x)
y = true_params[0] * x + true_params[1] + yerr * np.random.randn(len(x))

# true line
x0 = np.linspace(-2.1, 2.1, 200)
y0 = np.dot(np.vander(x0, 2), true_params)

# plot
plt.errorbar(x, y, yerr=yerr, fmt=",k", ms=0, capsize=0, lw=1, zorder=999)
plt.scatter(x, y, marker="s", s=22, c="k", zorder=1000)
plt.plot(x0, y0, c='k')
```

```python
# analytic solution - MLE for a priori known Gaussian noise
def linear_regression(x, y, yerr):
    A = np.vander(x, 2)
    result = np.linalg.solve(np.dot(A.T, A / yerr[:, None]**2), np.dot(A.T, y / yerr**2))
    return result

res = linear_regression(x, y, yerr)
m, b = res
```

```python
# optimise Huber loss

import torch
# initialise model parameters (y = m*x + b)
m_robust = torch.nn.Parameter(1e-3*torch.ones(1), requires_grad = True)
b_robust = torch.nn.Parameter(1e-3*torch.ones(1), requires_grad = True)
params_robust = list([m_robust, b_robust])

# negative log-likelihood for Huber likelihood (excluding the irrelevant normalisation constant)
def nll_Huber(model, data, yerr, c):

    # ln_sigma is same as above
    ln_sigma = torch.log(yerr).sum()

    # define inliers and outliers with a threshold, c
    resid = torch.abs((model - data)/yerr)
    cond1 = resid <= c
    cond2 = resid > c

    inliers = ((model - data)/yerr)[cond1]
    outliers = ((model - data)/yerr)[cond2]

    # Huber loss can be thought of as a hybrid of l2 and l1 loss
    # apply l2 (i.e. normal) loss to inliers, and l1 to outliers
    l2 = 0.5*torch.pow(inliers, 2).sum()
    l1 = (c * torch.abs(outliers) - (0.5 * c**2)).sum()

    nll = ln_sigma + l2 + l1

    return nll

# pass paramterers to optimizer
optimizer_robust = torch.optim.Adam(params_robust, lr = 0.01)
```

```python
# tuning parameter, c
c = 1.345

xt, yt, yerrt = torch.from_numpy(x), torch.from_numpy(y), torch.from_numpy(yerr)

for epoch in range(2000):

    model = m_robust*xt + b_robust

    # negative loglikelihood
    loss = nll_Huber(model, yt, yerrt, c)

    optimizer_robust.zero_grad()
    loss.backward()
    optimizer_robust.step()

    if np.mod(epoch, 100) == 0:
        # You can see the alpha+scale parameters moving around.
        print('{:<4}: loss={:03f}  m={:03f}  b={:03f}'.format(
            epoch, loss.item(), m_robust.item(), b_robust.item()))
```

    0   : loss=-16.117150  m=2.061463  b=0.942204
    100 : loss=-16.138877  m=2.075715  b=0.943876
    200 : loss=-16.138879  m=2.075600  b=0.943839
    300 : loss=-16.138879  m=2.075600  b=0.943839
    400 : loss=-16.138879  m=2.075600  b=0.943839
    500 : loss=-16.138879  m=2.075600  b=0.943839
    600 : loss=-16.138879  m=2.075600  b=0.943839
    700 : loss=-16.138879  m=2.075600  b=0.943839
    800 : loss=-16.138879  m=2.075600  b=0.943839
    900 : loss=-16.138879  m=2.075600  b=0.943839
    1000: loss=-16.138879  m=2.075600  b=0.943839
    1100: loss=-16.138879  m=2.075600  b=0.943839
    1200: loss=-16.138879  m=2.075600  b=0.943839
    1300: loss=-16.138879  m=2.075600  b=0.943839
    1400: loss=-16.138879  m=2.075600  b=0.943839
    1500: loss=-16.138879  m=2.075600  b=0.943839
    1600: loss=-16.138879  m=2.075600  b=0.943839
    1700: loss=-16.138879  m=2.075600  b=0.943839
    1800: loss=-16.138879  m=2.075601  b=0.943839
    1900: loss=-16.138879  m=2.075601  b=0.943839

```python
m_r, b_r = m_robust.item(), b_robust.item()
```

```python
def evaluate_gaussian_log_likelihood(data, model, var):
    print('\nGaussian log-likelihood')
    chi2 = 0.5 * ((data - model)**2 / var).sum()
    lnsigma = np.log(np.sqrt(var)).sum()
    norm_constant = len(data.flatten()) * 0.5 * np.log(2 * np.pi)
    print('chi2, lnsigma, norm_constant:', chi2, lnsigma, norm_constant)
    return -(chi2 + lnsigma + norm_constant)

def evaluate_huber_log_likelihood(data, model, var, c):
    print('\nHuber log-likelihood')

    ## PyTorchDIA - 'Huber' likelihood
    sigma = np.sqrt(var)
    ln_sigma = np.log(sigma).sum()

    # gaussian when (model - targ)/sigma <= c
    # absolute deviation when (model - targ)/sigma > c
    cond1 = np.abs((model - data)/sigma) <= c
    cond2 = np.abs((model - data)/sigma) > c
    inliers = ((model - data)/sigma)[cond1]
    outliers = ((model - data)/sigma)[cond2]

    l2 = 0.5*(np.power(inliers, 2)).sum()
    l1 = (c *(np.abs(outliers)) - (0.5 * c**2)).sum()

    constant = (np.sqrt(2 * np.pi) * math.erf(c / np.sqrt(2))) + ((2 / c) * np.exp(-0.5 * c**2))
    norm_constant = len(data.flatten()) * np.log(constant)
    ll = -(l2 + l1 + ln_sigma + norm_constant)
    print('l2, l1, ln_sigma, norm_constant:', l2, l1, ln_sigma, norm_constant)
    return ll
```

```python
# MLE models
model = m*x + b # Gaussian
model_r = m_r*x + b_r # Huber
```

```python
ll_gaussian = evaluate_gaussian_log_likelihood(x, model, yerr**2)
print(ll_gaussian)
ll_huber = evaluate_huber_log_likelihood(x, model_r, yerr**2, c=c)
print(ll_huber)
```

    Gaussian log-likelihood
    chi2, lnsigma, norm_constant: 173.83346014331022 -27.488721956224655 27.56815599614018
    -173.91289418322575

    Huber log-likelihood
    l2, l1, ln_sigma, norm_constant: 4.538741913408627 81.53652455056039 -27.488721956224655 29.357945837155103
    -87.94449034489946

Unless I've blundered, it seems like the Huber log-likelihood is always going to exceed the Gaussian due to how the numerics are treated. If this is indeed correct, then any comparison between the two would be meaningless.

```python

```

Huber Loss recast as a valid Likelihood.ipynb