# System Properties

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Causality

The [principle of causality](https://en.wikipedia.org/wiki/Causality) states that an effect temporally has to follow its cause. A systems may violate this fundamental principle which limits its practical realization. Causality is hence an important aspect for the practical realization of systems. Conditions for causal linear time-invariant (LTI) systems for both the impulse response as well as the transfer function are derived in this section.

### Condition for the Impulse Response

The output signal $y(t) = \mathcal{H} \{ x(t) \}$ of an LTI system is given by convolving the input signal $x(t)$ with its impulse response $h(t)$

\begin{equation}
y(t) = \int_{-\infty}^{\infty} x(\tau) \cdot h(t-\tau) \; d\tau = \int_{-\infty}^{\infty} x(t - \tau) \cdot h(\tau) \; d\tau
\end{equation}

Analyzing the first integral reveals that the computation of the output signal $y(t)$ for some time instant $t=t_0$ requires knowledge of the input signal for all time instants. This includes also future time instants $t > t_0$. While this does not pose a problem for signals and impulse responses given in closed-form, this is not feasible in practice. Without imposing further restrictions, this would require the knowledge of the input signal for all future time instances.

Causality is not violated if we modify the integration limits of the convolution integral

\begin{equation}
y(t) = \int_{-\infty}^{t} x(\tau) \cdot h(t-\tau) \; d\tau = \int_{0}^{\infty} x(t - \tau) \cdot h(\tau) \; d\tau
\end{equation}

Now the output signal $y(t)$ for a given time instant $t=t_0$ depends only on the input signal $x(t)$ for $t \leq t_0$. Comparing the second equality with the second equality of the original convolution integral yields that the modified integration limits are equal to the assumption that  $h(t) = 0$ for $t < 0$.

A system is termed *causal system* [iff](https://en.wikipedia.org/wiki/If_and_only_if) its impulse response $h(t)$ is a causal signal

\begin{equation}
h(t) = 0 \qquad \text{for } t < 0
\end{equation}

Only causal systems can be realized practically due to above reasoning.

### Condition for the Transfer Function

The transfer function $H(j \omega) = \mathcal{F} \{ h(t) \}$ of a causal system shows specific symmetries. A causal impulse response $h(t)$ has to fulfill the following relation

\begin{equation}
h(t) = h(t) \cdot \epsilon(t)
\end{equation}

Fourier transformation of the left- and right-hand side yields

\begin{equation}
H(j \omega) = \frac{1}{\pi} H(j \omega) * \frac{1}{j \omega}
\end{equation}

by application of the [multiplication theorem](../fourier_transform/theorems.ipynb#Multiplication-Theorem) in conjunction with the [Fourier transform of the Heaviside signal](../fourier_transform/theorems.ipynb#Transformation-of-the-Heaviside-signal). Decomposing this result into the real and imaginary part of $H(j \omega)$ derives the following symmetry relations for a causal system

\begin{align}
\Re \{ H(j \omega) \} &= \frac{1}{\pi} \Im \{ H(j \omega) \} * \frac{1}{\omega} \\
\Im \{ H(j \omega) \} &= - \frac{1}{\pi} \Re \{ H(j \omega) \} * \frac{1}{\omega}
\end{align}

The convolution of a spectrum with $\frac{1}{\omega}$ is known as [Hilbert transform](https://en.wikipedia.org/wiki/Hilbert_transform). Above result states that the real and imaginary part of the spectrum $H(j \omega)$ of a causal system are related by the Hilbert transform. The Hilbert transform is used for instance in the theory of [single-sideband modulation](https://en.wikipedia.org/wiki/Single-sideband_modulation) in order to express the [analytic signal](https://en.wikipedia.org/wiki/Analytic_signal).

## Stability

The stability of a system can be evaluated with respect to various different criteria. The most common ones are [Lyapunov stability](https://en.wikipedia.org/wiki/Lyapunov_stability) and [bounded-input bounded-output stability](https://en.wikipedia.org/wiki/BIBO_stability). The latter is introduced in the following.

### Bounded-Input Bounded-Output Stability

With respect to the bounded-input bounded-output (BIBO) principle, an LTI system is termed stable if its output signal $y(t) = \mathcal{H} \{ x(t) \}$ is bounded for a bounded input signal $x(t)$. A signal is bounded if its magnitude does not exceed a given finite value. For the in- and output this condition is formulated as

\begin{align}
|x(t)| &< B_x \\
|y(t)| &< B_y
\end{align}

where $B_x, B_y < \infty$ denote constant finite bounds. The BIBO criterion is illustrated in the following

#### Condition for the Impulse Response

In order to derive a condition for the impulse response $h(t)$ of an LTI system that conforms to the BIBO criterion, the magnitude of the output signal $y(t)$ is expressed by convolving the input signal $x(t)$ with the impulse response

\begin{equation}
|y(t)| = \left| \int_{-\infty}^{\infty} x(\tau) \cdot h(t-\tau) \; d\tau \right|
\end{equation}

An upper bound for the magnitude of the output signal is found by applying the [triangle inequality](https://en.wikipedia.org/wiki/Triangle_inequality) together with the upper bound for $|x(t)|$

\begin{equation}
|y(t)| \leq \int_{-\infty}^{\infty} |x(\tau)| \cdot |h(t-\tau)| \; d\tau < \int_{-\infty}^{\infty} B_x \cdot |h(t-\tau)| \; d\tau
\end{equation}

Since the output signal $|y(t)|$ shall be bounded, it can be concluded that the impulse response needs to be an [absolutely integrable function](https://en.wikipedia.org/wiki/Absolutely_integrable_function)

\begin{equation}
\int_{-\infty}^{\infty} |h(t)| \; dt < B_h
\end{equation}

where $B_h < \infty$ denotes a constant finite bound.

An LTI system is stable in the sense of the BIBO stability criterion iff its impulse response is absolutely integrable. Since absolute integrability of a signal is sufficient for the [existence of its Fourier transform](../fourier_transform/definition.ipynb#Definition), this implies that the transfer function $H(j \omega) = \mathcal{F} \{ h(t) \}$ exists for stable systems.

#### Conditions for Rational Transfer Functions

For a rational transfer function $H(s) = \mathcal{L} \{ h(t) \}$, the BIBO stability of an LTI system implies constraints on the locations of its poles. These are derived in the following.

The inverse Laplace transform of a right-sided signal can be computed in closed form [using the partial fraction decomposition](../laplace_transform/inverse.ipynb#Basic-Procedure) of its Laplace transform. Applying this to the transfer function $H(s)$, the impulse response $h(t) = \mathcal{L}^{-1} \{ H(s) \}$ is given as

\begin{equation}
h(t) = A_0 \cdot \delta(t) + \epsilon(t) \sum_{\mu = 1}^{L} e^{s_{\infty \mu} t} \sum_{\nu = 1}^{R_\mu} \frac{A_{\mu \nu} \, t^{\mu - 1}}{(\nu -1)!}
\end{equation}

where $s_{\infty \mu}$ denotes the $\mu$-th unique pole of $H(s)$, $R_\mu$ its degree, $L$ the total number of different poles $\mu = 1 \dots L$, and $A_0$ and $A_{\mu \nu}$ the coefficients of the partial fraction decomposition. Above formula for the inverse Laplace transform holds if the order $M$ of the numerator is smaller or equal to the order $N$ of the denominator. In this case one has to check under which conditions the impulse response $h(t)$ is absolutely integrable. The first term $A_0 \delta(t)$ does not pose a problem for $A_0 < \infty$. Under the assumption $A_{\mu \nu} < \infty$, the absolute integrability of the second term is checked by evaluating

\begin{equation}
\int_{-\infty}^{\infty} |\epsilon(t) t^{\mu - 1} e^{s_{\infty \mu} t}| \; dt = \int_0^{\infty} t^{\mu - 1} e^{\sigma_{\infty \mu} t} \; dt
\end{equation}

where $s_{\infty \mu} = \sigma_{\infty \mu} + j \omega_{\infty \mu}$. It can be concluded that above integral is absolutely integrable if $\sigma_{\infty \mu} < 0$.

If $M > N$, the system is not stable in the BIBO sense. This statement can be proven by a counterexample. If $M > N$, a polynominal in $s$ with positive powers can be split off from the transfer function $H(s)$ by polynominal division. Lets consider one of these terms, for instance $H(s) = s$. It follows by application of the derivation theorem that $h(t) = \frac{d}{dt} \delta(t)$. For the bounded input signal $x(t) = \cos(\omega_0 t \cdot t)$, the output signal $y(t) = \cos(\omega_0 t \cdot t) * \frac{d}{dt} \delta(t) = \frac{d}{dt} \cos(\omega_0 t \cdot t) = - 2 \omega_0 t \sin(\omega_0 t \cdot t)$ is not bounded for $t \to \infty$. The same argumentation holds for positive powers of $s$ by repeated application of the derivation theorem.

Combining the results it can be concluded that a system with rational transfer function $H(s)$ and right-sided impulse response $h(t)$ is stable with respect to the BIBO criterion if

1. the order $M$ of its numerator is smaller or equal to the order $N$ of its numerator and
1. all poles are located in the left $s$-half-plane
    \begin{equation}
    \Re \{ s_{\infty \mu} \} < 0 \qquad \forall \mu
    \end{equation}

The locations of the zeros of the transfer function have no influence on the stability of a system.

In order to investigate the stability of a given system, the poles of its rational transfer function $H(s)$ have to be determined. This can be performed by computing the roots of the denominator polynomial. However, in order to avoid the computationally complex computation of roots, several stability tests have been developed. These determine the stability of a system from the coefficients of the denominator polynomial. An example of such a test is the [Routh–Hurwitz stability criterion](https://en.wikipedia.org/wiki/Routh%E2%80%93Hurwitz_stability_criterion).

**Example**

The impulse response $h(t) = \mathcal{L}^{-1} \{ H(s) \}$ of a 2nd-order system with transfer function

\begin{equation}
H(s) = \frac{1}{(s-s_\infty)(s-s^*_\infty)}
\end{equation}

is computed. First the case of a **stable system** is considered with $s_\infty = -1 + j$, hence $\Re \{ s_\infty \} < 0$. Its impulse response $h(t)$ is computed by inverse Laplace transform of the transfer function $h(t) = \mathcal{L}^{-1} \{ H(s) \}$

```python
import sympy as sym
sym.init_printing()

s = sym.symbols('s', complex=True)
t = sym.symbols('t', real=True)
s_inf = -1 + sym.I

H = 1/((s - s_inf)*(s - sym.conjugate(s_inf)))
h = sym.inverse_laplace_transform(H, s, t)
h
```

The impulse response is plotted for illustration

```python
sym.plot(h, (t, 0, 10), xlabel='$t$', ylabel='$h(t)$');
```

It can be observed that the amplitude of the impulse response decays exponentially with increasing time $t$. The absolute integrability of the impulse response is confirmed by noting that

\begin{equation}
\int_{-\infty}^{\infty} | \epsilon(t) e^{-t} \sin(t) | \; dt = \int_{0}^{\infty} | e^{-t} \sin(t) | \; dt < \int_{0}^{\infty} | e^{-t} | \; dt = 1
\end{equation}

Now the impulse response of an **instable system** is investigated with $s_\infty = 1 + j$, hence $\Re \{ s_\infty \} > 0$. Its impulse response is given as

```python
s_inf = 1 + sym.I

H = 1/((s - s_inf)*(s - sym.conjugate(s_inf)))
h = sym.inverse_laplace_transform(H, s, t)
h
```

Again the impulse response is plotted for illustration

```python
sym.plot(h, (t, 0, 10), xlabel='$t$', ylabel='$h(t)$');
```

The impulse response shows an exponentially increasing amplitude for increasing time. The impulse response is clearly not absolutely integrable and hence the system not stable in the BIBO sense. This may also be confirmed by computing the output signal $y(t)$ for the bounded input signal $x(t) = \epsilon(t) e^{s_0 t}$ with $\Re \{ s_0 \} < 0$ via inverse Laplace transform of $Y(s) = H(s) \cdot X(s)$ with $X(s) = \mathcal{L} \{ x(t) \}$

```python
s0 = sym.symbols('s_0')
Y = 1/((s - s_inf)*(s - sym.conjugate(s_inf))) * 1/(s+s0)

y = sym.inverse_laplace_transform(Y, s, t)
y
```

The output signal is plotted for illustration with $s_0 = -1 + 10 j$

```python
y1 = y.subs(s0, -1+10*sym.I)
sym.plot(sym.re(y1), (t, 0, 15), xlabel='$t$', ylabel='$y(t)$');
```

## Realizability

A system is said to be realizable if it is a causal and stable system. Realizability is a prerequisite for the practical implementation of systems. Causality implies that the impulse response $h(t)$ of the system has to be a causal signal. Hence the region of convergence (ROC) of its transfer function $H(s) = \mathcal{L} \{ h(t) \}$ is the half-plane located right of all poles. Stability and causality of a system with rational transfer function $H(s)$ implies that its poles are located in the left $s$-half-plane.

Summarizing, a system with rational transfer function $H(s)$ is realizable iff all poles of $H(s)$ are located in the left $s$-half-plane not including the imaginary axis and the order of the numerator is lower or equal to the order of the denominator. The ROC is given by a right-sided $s$-half-plane including the imaginary axis. The locations of the zeros of the transfer function have no influence on the realizability of systems.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


This is a tutorial training a [VAE](https://arxiv.org/abs/1312.6114) on MNIST and latent analysis

Structure:

  * create encoder and decoder, choose latent and model distributions.
  * VAE training
  * check likelihood for overfitting.
  * latent analysis

```
!pip install -q dm-sonnet
!pip install -q seaborn -U
```

## Imports

We will use Sonnet to define the encoder and decoder networks.

```
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import math

import tensorflow as tf
import sonnet as snt

import numpy as np

# Plotting library.
from matplotlib import pyplot as plt
import seaborn as sns
```

```
tf.logging.set_verbosity(tf.logging.ERROR)
```

```
sns.set(rc={"lines.linewidth": 2.8}, font_scale=2)
sns.set_style("whitegrid")
```

```
# Don't forget to select GPU runtime environment in Runtime -> Change runtime type
device_name = tf.test.gpu_device_name()
if device_name != '/device:GPU:0':
  raise SystemError('GPU device not found')
print('Found GPU at: {}'.format(device_name))
```

    Found GPU at: /device:GPU:0

## Hyperparameters

```
BATCH_SIZE = 64 # @param
NUM_LATENTS = 20 # @param
TRAINING_STEPS = 10000 # @param
```

## Get the data

We will use the MNIST dataset. Luckly, TensorFlow comes with a simple way to load it.

```
mnist = tf.contrib.learn.datasets.load_dataset("mnist")
```

    Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.
    Extracting MNIST-data/train-images-idx3-ubyte.gz
    Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.
    Extracting MNIST-data/train-labels-idx1-ubyte.gz
    Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.
    Extracting MNIST-data/t10k-images-idx3-ubyte.gz
    Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.
    Extracting MNIST-data/t10k-labels-idx1-ubyte.gz

```
print(mnist.train.images.shape)
print(type(mnist.train.images))
```

    (55000, 784)
    <type 'numpy.ndarray'>

### Transform the data from numpy arrays to graph tensors.

This allows us to use TensorFlow datasets, which ensure that a new batch from the data is being fed at each session.run. This means that we do not need to use feed_dicts to feed data to each session.

```
def make_tf_data_batch(np_data, shuffle=True):
  # Reshape the data to image size.
  images = np_data.reshape((-1, 28, 28, 1))

  # Create the TF dataset.
  dataset = tf.data.Dataset.from_tensor_slices(images)

  # Shuffle and repeat the dataset for training.
  # This is required because we want to do multiple passes through the entire
  # dataset when training.
  if shuffle:
    dataset = dataset.shuffle(100000).repeat()

  # Batch the data and return the data batch.
  one_shot_iterator = dataset.batch(BATCH_SIZE).make_one_shot_iterator()
  data_batch = one_shot_iterator.get_next()
  return data_batch
```

```
real_data = make_tf_data_batch(mnist.train.images)
print(real_data.shape)
```

    (?, 28, 28, 1)

## Define the encoder and decoder (generator) networks

### Decoder network

We use a simple Transpose convolutional network. Because the input to the model is 2 dimensional, we first use a linear layer to transform the input latents to a size suitable for the conv network.

```
class MnistDecoder(snt.AbstractModule):
  """The learned decoder distribution p(x|z).

    The output distribution will be a Bernoulli distribution for each pixel,
    with the mean of the distribution being learned by gradient descent.

    The variables of this module get a loss from the likelihood term and are
    trained using gradient descent.
  """

  def __init__(self, name='MnistDecoder'):
    super(MnistDecoder, self).__init__(name=name)

  def _build(self, inputs):
    """Constructs the generator graph.

    Args:
      inputs: `tf.Tensor` with the input of the generator.

    Returns:
      `tf.Tensor`, the generated samples.
    """
    linear = snt.Linear(7 * 7 * 64)
    inputs = linear(inputs)

    # Reshape the data to have rank 4.
    inputs = snt.BatchReshape((7, 7, 64))(inputs)
    inputs = tf.nn.relu(inputs)

    net = snt.nets.ConvNet2DTranspose(
        output_channels=[32, 1],
        output_shapes=[[14, 14], [28, 28]],
        strides=[2],
        paddings=[snt.SAME],
        kernel_shapes=[[5, 5]],
        use_batch_norm=False)

    # Wrap a Bernoulli distribution over the generated output.
    return tf.distributions.Bernoulli(net(inputs))
```

### Encoder network

We use a conv net encoder, and add a linear layer at the end - the latents will be 1D.

```
class MnistEncoder(snt.AbstractModule):
  """The learned encoder distribution q(z|x).

    The output distribution will be a Normal distribution for each latent,
    with the mean and variance of the distribution being learned by gradient
    descent.

    The variables of this module get a loss from both the likelihood and
    the KL term. We do not have to worry about the reparamtrization trick, since
    TensorFlow implements that for us, for the Normal distribution.
  """

  def __init__(self, name='MnistEncoder'):
    super(MnistEncoder, self).__init__(name=name)

  def _build(self, input_image):
    """Constructs the discriminator graph.

    Args:
      input_image: `tf.Tensor` input to be used by the discriminator.

    Returns:
      `tf.Tensor` output with the decision of the discriminator.
    """
    leaky_relu_activation = lambda x: tf.maximum(self._leaky_relu_coeff * x, x)

    conv2d = snt.nets.ConvNet2D(
        output_channels=[8, 16, 32, 64, 128],
        kernel_shapes=[[5, 5]],
        strides=[2, 1, 2, 1, 2],
        paddings=[snt.SAME],
        activate_final=True,
        activation=tf.nn.relu,
        use_batch_norm=False)

    convolved = conv2d(input_image)
    # Flatten the data to 2D for getting the mean and variance from the encoder.
    flat_data = snt.BatchFlatten()(convolved)

    mean = snt.Linear(NUM_LATENTS)(flat_data)
    scale = snt.Linear(NUM_LATENTS)(flat_data)

    return tf.distributions.Normal(loc=mean, scale=scale)

```

## VAE logic

Variational bound:
\begin{equation}
 \mathbb{E}_{p^*(x)} \mathbb{E}_{q(z|x)} \log p_\theta(x|z) - \mathbb{E}_{p^*(x)} KL(q(z|x)||p(z))
\end{equation}

The aim of the model is to maximize the variational lower bound, so the negative of the bound is minimized.

```
encoder = MnistEncoder()
decoder = MnistDecoder()

# Zero mean, unit variance prior.
prior_mean = tf.zeros(shape=(BATCH_SIZE, NUM_LATENTS), dtype=tf.float32)
prior_scale = tf.ones(shape=(BATCH_SIZE, NUM_LATENTS), dtype=tf.float32)
prior = tf.distributions.Normal(loc=prior_mean, scale=prior_scale)

posterior_distribution = encoder(real_data)
latent_sample = posterior_distribution.sample()

print('Latent samples {}'.format(latent_sample.shape))
```

    Latent samples (?, 20)

## Define the graph for the loss

Since we wnat to maximize the bound on the log probability of the data, we minimize the negative of the bound.

The terms of the bound are: the log prob term, and the KL term.

### Log prob term

Reminder: this term ensures reconstructions will look good.

\begin{equation}
 \mathbb{E}_{p^*(x)} \mathbb{E}_{q^(z|x) \log p_\theta(x|z)}
\end{equation}

### KL term

Reminder: this term ensures samples are close to reconstructions.

\begin{equation}
 - \mathbb{E}_{p^*(x)} KL(q(z|x)||p(z))
\end{equation}

```
def bound_terms(data_batch):
  posterior_distribution = encoder(data_batch)
  latent_sample = posterior_distribution.sample()

  all_log_probs = decoder(latent_sample).log_prob(data_batch)

  # Since we used broadcasting, log_prob of a Bernoulli will return a log_prob for
  # each dimension.
  all_log_probs.shape.assert_is_compatible_with([BATCH_SIZE, 28, 28, 1])

  # Reduce sum over the data dimensions
  likelihood_term = tf.reduce_sum(all_log_probs, axis=[1, 2, 3])

  # Reduce mean over the batch dimensions
  likelihood_term = tf.reduce_mean(likelihood_term)

  all_kls = tf.distributions.kl_divergence(posterior_distribution, prior)
  all_kls.shape.assert_is_compatible_with([BATCH_SIZE, NUM_LATENTS])

  # Reduce sum over the latent dimensions.
  kl_term = tf.reduce_sum(all_kls, axis=[1])

  # Reduce over the batch dimension.
  kl_term = tf.reduce_mean(kl_term)

  return likelihood_term, kl_term
```

## Total Loss

```
# Maximize the data likelihodd and minimize the KL divergence between the prior and posterior
likelihood_term, kl_term = bound_terms(real_data)
train_bound = likelihood_term - kl_term
loss =  - train_bound
```

## Define the samples

Note: VAEs have no direct cost on samples. The quality of the samples improves, because the likelihood term ensures that the model will reconstruct well, while the KL term brings samples close to reconstructions.

```
samples = decoder(prior.sample()).mean()
```

```
latent_sample = encoder(real_data).sample()
reconstructions = decoder(latent_sample).mean()

```

## Define the optimizer

We will train the encoder and decoder jointly, to minimize the same loss.

```
optimizer = tf.train.AdamOptimizer(0.001, beta1=0.9, beta2=0.9)
update_op = optimizer.minimize(loss)
```

## Create the tensorflow session

```
sess = tf.Session()

# Initialize all variables
sess.run(tf.global_variables_initializer())
```

## Training

```
losses = []

for i in xrange(TRAINING_STEPS):
  sess.run(update_op)

  if i % 100 == 0:
    iteration_loss = sess.run(loss)
    print('Iteration {}. Loss {}'.format(i, iteration_loss))
    losses.append(iteration_loss)
```

    Iteration 0. Loss 610.276123047
    Iteration 100. Loss 201.867080688
    Iteration 200. Loss 181.950180054
    Iteration 300. Loss 153.757537842
    Iteration 400. Loss 145.420257568
    Iteration 500. Loss 132.959655762
    Iteration 600. Loss 127.240966797
    Iteration 700. Loss 124.027557373
    Iteration 800. Loss 117.480369568
    Iteration 900. Loss 117.589767456
    Iteration 1000. Loss 114.042572021
    Iteration 1100. Loss 115.860397339
    Iteration 1200. Loss 115.532142639
    Iteration 1300. Loss 111.962844849
    Iteration 1400. Loss 116.609283447
    Iteration 1500. Loss 114.537094116
    Iteration 1600. Loss 118.786376953
    Iteration 1700. Loss 111.097732544
    Iteration 1800. Loss 113.713607788
    Iteration 1900. Loss 115.026344299
    Iteration 2000. Loss 109.995574951
    Iteration 2100. Loss 109.407897949
    Iteration 2200. Loss 109.415374756
    Iteration 2300. Loss 113.632843018
    Iteration 2400. Loss 106.374473572
    Iteration 2500. Loss 116.002853394
    Iteration 2600. Loss 101.022964478
    Iteration 2700. Loss 110.474555969
    Iteration 2800. Loss 104.975326538
    Iteration 2900. Loss 111.565422058
    Iteration 3000. Loss 108.190658569
    Iteration 3100. Loss 110.7526474
    Iteration 3200. Loss 107.815948486
    Iteration 3300. Loss 105.125762939
    Iteration 3400. Loss 110.461257935
    Iteration 3500. Loss 101.434020996
    Iteration 3600. Loss 110.245239258
    Iteration 3700. Loss 108.431915283
    Iteration 3800. Loss 101.121337891
    Iteration 3900. Loss 109.09034729
    Iteration 4000. Loss 102.14793396
    Iteration 4100. Loss 102.993904114
    Iteration 4200. Loss 104.034805298
    Iteration 4300. Loss 101.782730103
    Iteration 4400. Loss 98.9214630127
    Iteration 4500. Loss 104.594894409
    Iteration 4600. Loss 100.305435181
    Iteration 4700. Loss 103.436630249
    Iteration 4800. Loss 112.59274292
    Iteration 4900. Loss 106.438423157
    Iteration 5000. Loss 101.394233704
    Iteration 5100. Loss 101.886573792
    Iteration 5200. Loss 103.126480103
    Iteration 5300. Loss 104.068557739
    Iteration 5400. Loss 101.134399414
    Iteration 5500. Loss 92.9712524414
    Iteration 5600. Loss 104.605766296
    Iteration 5700. Loss 101.369316101
    Iteration 5800. Loss 101.803123474
    Iteration 5900. Loss 100.416854858
    Iteration 6000. Loss 101.946044922
    Iteration 6100. Loss 107.092697144
    Iteration 6200. Loss 106.329681396
    Iteration 6300. Loss 104.35508728
    Iteration 6400. Loss 103.333496094
    Iteration 6500. Loss 104.745407104
    Iteration 6600. Loss 103.936058044
    Iteration 6700. Loss 102.923782349
    Iteration 6800. Loss 101.658813477
    Iteration 6900. Loss 97.0839385986
    Iteration 7000. Loss 103.659080505
    Iteration 7100. Loss 97.0283813477
    Iteration 7200. Loss 102.28591156
    Iteration 7300. Loss 98.5410919189
    Iteration 7400. Loss 106.676506042
    Iteration 7500. Loss 91.8732376099
    Iteration 7600. Loss 98.99659729
    Iteration 7700. Loss 102.701965332
    Iteration 7800. Loss 101.746002197
    Iteration 7900. Loss 100.114768982
    Iteration 8000. Loss 105.365005493
    Iteration 8100. Loss 97.789352417
    Iteration 8200. Loss 101.698379517
    Iteration 8300. Loss 100.455055237
    Iteration 8400. Loss 96.4564819336
    Iteration 8500. Loss 101.938873291
    Iteration 8600. Loss 104.065986633
    Iteration 8700. Loss 107.216621399
    Iteration 8800. Loss 103.305397034
    Iteration 8900. Loss 99.7856292725
    Iteration 9000. Loss 99.9027557373
    Iteration 9100. Loss 98.6257324219
    Iteration 9200. Loss 104.862350464
    Iteration 9300. Loss 104.466560364
    Iteration 9400. Loss 101.520263672
    Iteration 9500. Loss 103.716140747
    Iteration 9600. Loss 101.349494934
    Iteration 9700. Loss 99.188873291
    Iteration 9800. Loss 102.41809082
    Iteration 9900. Loss 102.609298706

## Visualize the loss in time

```
plt.figure(figsize=(10,5))
plt.plot(losses, label='Negative log likelihood Bound')
plt.xlabel('Time')
plt.legend()
```

## Visualize samples, reconstructions

```
def gallery(array, ncols=10, rescale=True):
  """Code adapted from: https://stackoverflow.com/questions/42040747/more-idomatic-way-to-display-images-in-a-grid-with-numpy"""
    if rescale:
      array = (array + 1.) / 2
    nindex, height, width, intensity = array.shape
    nrows = nindex//ncols
    assert nindex == nrows*ncols
    # want result.shape = (height*nrows, width*ncols, intensity)
    result = (array.reshape(nrows, ncols, height, width, intensity)
              .swapaxes(1,2)
              .reshape(height*nrows, width*ncols, intensity))
    return result

```

```
real_data_examples = sess.run(real_data)
final_samples = sess.run(samples)
data_reconstructions = sess.run(reconstructions)
print(data_reconstructions.shape)
```

    (64, 28, 28, 1)

### Real data

```
plt.gray()
plt.axis('off')
plt.imshow(gallery(real_data_examples, ncols=8).squeeze(axis=2))
```

### Samples

```
plt.gray()
plt.axis('off')
plt.imshow(gallery(final_samples, ncols=8).squeeze(axis=2))
```

### Reconstructions

```
plt.gray()
plt.axis('off')
plt.imshow(gallery(data_reconstructions, ncols=8).squeeze(axis=2))
```

## Final training log prob aver multiple batches

```
NUM_TEST_BATCHES = 10 # @param
```

```
train_log_prob = []

for _ in xrange(NUM_TEST_BATCHES):
  train_log_prob.append(sess.run(train_bound))

train_avg_log_prob = np.mean(train_log_prob)

print('Averaged training log prob {}'.format(train_avg_log_prob))
```

    Averaged training log prob -101.639175415

## Checking for overfitting - test likelihoods

We want to ensure that the model has not overfitted - to do that, we see if it gives a higher likelihood to training data compared to test data. If this is the case, we can conclude the model is overfitting.

### Get test data

```
test_data_batch = make_tf_data_batch(mnist.test.images)
```

```
likelihood_term, kl_term = bound_terms(test_data_batch)
test_bound = likelihood_term - kl_term
```

```
test_log_prob = []

for _ in xrange(NUM_TEST_BATCHES):
  test_log_prob.append(sess.run(test_bound))

test_avg_log_prob = np.mean(test_log_prob)

print('Averaged test log prob {}'.format(test_avg_log_prob))

```

    Averaged test log prob -100.855674744

## Analyse the structure of the latent variables

VAEs are generative and inference models. So far we have looked at samples, and reconstructions, but we have not looked at the learned representations.

We will now analyze the learned VAE representations by looking at the KLs per latent, but also by doing latent space traversals.

Remember that the KL term is important, since the KL term ensures that the quality of the samples is going to be close to that of the reconstrunctions.

```
num_batches = int(mnist.train.images.shape[0] / BATCH_SIZE)
data_batch = make_tf_data_batch(mnist.train.images, shuffle=False)
```

```
posterior_distribution = encoder(data_batch)

all_kls = tf.distributions.kl_divergence(posterior_distribution, prior)
all_kls.shape.assert_is_compatible_with([BATCH_SIZE, NUM_LATENTS])

# Reduce over the batch dimension.
kls = tf.reduce_mean(all_kls, axis=[0])
kls.shape.assert_is_compatible_with([NUM_LATENTS])
```

```
aggregate_kl = np.zeros([NUM_LATENTS])
for _ in xrange(num_batches):
  aggregate_kl += sess.run(kls)

aggregate_kl /= num_batches
```

### Plot the kl divergences per latent

Because we used a factorizable prior (a Gaussian with unit covariance), the KL divergence
also factorizes, and we can see a KL divergence per unit). Here we analyze the KL divergence of each unit, to see how the model uses each latent unit.

We will see that if we increase the number of latent variables, the VAE "turns off" units, by setting their distribution to that of the prior.

```
def get_grid_size(num_latents):
  num_latents_sqrt = np.sqrt(num_latents)
  width = int(num_latents_sqrt)
  while num_latents % width !=0:
    width -= 1
  return width, int(num_latents / width)

aggregate_kl = np.reshape(aggregate_kl, get_grid_size(NUM_LATENTS))

np.random.shuffle(aggregate_kl)

if sns.__version__ == '0.7.1':
  cmap = 'BuGn'
else:
  cmap = sns.cubehelix_palette(4, start=2, rot=0, dark=0, light=.96)

sns.heatmap(aggregate_kl, vmin=0, vmax=2, linewidths=.05, cmap=cmap)
plt.axis('off')

```

## Latent space interpolation

By doing an interpolation in latent space, we can analyze the behaviour of the model.

```
a = np.linspace(0.0, 1.0, 100)
a = np.expand_dims(a, axis=1)

first_latents = tf.random_normal((1, NUM_LATENTS))
second_latents = tf.random_normal((1, NUM_LATENTS))

interpolations =  np.sqrt(a) * first_latents + np.sqrt(1 - a) * second_latents
# interpolations =  a * first_latents + (1 - a) * second_latents

samples_from_interpolations = decoder(interpolations).mean()

```

```
samples_from_interpolations_np = sess.run(samples_from_interpolations)
```

```
print(samples_from_interpolations_np.shape)
```

    (100, 28, 28, 1)

```
plt.gray()
plt.axis('off')
plt.imshow(gallery(samples_from_interpolations_np).squeeze(axis=2))
plt.show()
```

## Exercise: change the number of latents (to 100) and see what happens with the KL plot.

## Close session

```
# sess.close()
```

### VAE samples on other datasets

Note: these are samples with the vanilla algorithm, but they can be improved using better complex distributions.

#### CIFAR-10

#### CelebA
#


# Probability distributions and their stories

(c) 2019 Justin Bois. With the exception of pasted graphics, where the source is noted, this work is licensed under a [Creative Commons Attribution License CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/). All code contained herein is licensed under an [MIT license](https://opensource.org/licenses/MIT).

This document was prepared at [Caltech](http://www.caltech.edu) with financial support from the [Donna and Benjamin M. Rosen Bioengineering Center](http://rosen.caltech.edu).

*This tutorial was generated from an Jupyter notebook. If you want to use the interactivity to explore probability distributions, you will need to download the notebook and run it on your machine. You can download the notebook [here](probability_distribution.ipynb).*

```python
%load_ext autoreload
%autoreload 2

import numpy as np
import scipy.stats as st
import scipy.special

import distribution_explorer

import bokeh.io
import bokeh.plotting
import bokeh.application
import bokeh.application.handlers
bokeh.io.output_notebook()
```

    Loading BokehJS ...

Because this tutorial consists mainly of a list of probability distributions and their stories and properties, it is useful to have a little index of them up front.

* [Discrete distributions](#Discrete-distributions)
    - [Bernoulli distribution](#Bernoulli-distribution)
    - [Geometric distribution](#Geometric-distribution)
    - [Negative Binomial distribution](#Negative-Binomial-distribution)
    - [Binomial distribution](#Binomial-distribution)
    - [Poisson distribution](#Poisson-distribution)
    - [Hypergeometric distribution](#Hypergeometric-distribution)
    - [Categorical distribution](#Categorical-distribution)
    - [Discrete Uniform distribution](#Discrete-Uniform-distribution)
* [Continuous distributions](#Continuous-distributions)
    - [Uniform distribution](#Uniform-distribution)
    - [Gaussian (a.k.a. Normal) distribution](#Gaussian,-a.k.a.-Normal,-distribution)
    - [Half-Normal distribution](#Half-Normal-distribution)
    - [Log-Normal distribution](#Log-Normal-distribution)
    - [Chi-square distribution](#Chi-square-distribution)
    - [Student-t distribution](#Student-t-distribution)
    - [Cauchy distribution](#Cauchy-distribution)
    - [Exponential distribution](#Exponential-distribution)
    - [Gamma distribution](#Gamma-distribution)
    - [Inverse-Gamma distribution](#Inverse-Gamma-distribution)
    - [Weibull distribution](#Weibull-distribution)
    - [Beta distribution](#Beta-distribution)
* [Discrete multivariate distributions](#Discrete-multivariate-distributions)
    - [Multinomial distribution](#Multinomial-distribution)
* [Continuous multivariate distributions](#Continuous-Multivariate-distributions)
    - [Multivariate Gaussian (a.k.a. Multivariate Normal) distribution](#Multivariate-Gaussian,-a.k.a.-Multivariate-Normal,-distribution)
    - [Lewandowski-Kurowicka-Joe (LKJ) distribution](#Lewandowski-Kurowicka-Joe-%28LKJ%29-distribution)
    - [Dirichlet distribution](#Dirichlet-distribution)

This tutorial also features interactive plotting, which needs a running Jupyter notebook to utilize. You should therefore download and run the notebook. When you launch your notebook, take note of the URL. You can find this at the top of your browser. It will be something like `localhost:8888`. Before you work through the notebook, define your notebook_url in the code cell below.

### Motivation

In this class, we have talked about the three levels of building a model in the
biological sciences.
* **Cartoon model**: A cartoon or verbal phenomenological
  description of the system of interest.
* **Mathematical model**: A mathematization of a cartoon model. This gives quantitative predictions on how a biological system should behave.
* **Statistical model**: A description of what we expect from an
  experiment, given our mathematical model.  In the Bayesian context, this is
  specification of the likelihood and prior.

Take together, these models comprise a **generative model**, which describes how the data were generated. Specifying your model amounts to choosing a probability distributions that describe the process of data generation. In some cases, you need to derive the likelihood (or even numerically compute it when it cannot be written in closed form). In most practical cases, though, your model is composed of standard probability distributions.  These distributions all have **stories** associated with them.  Importantly, *if your data and model match the story of a distribution, you know that this is the distribution to choose for your likelihood.*

## Review of probability distributions, PMFs, PDFs, and CDFs

Before we begin talking about distributions, let's remind ourselves
what probability distributions are.  We cut some corners in our
definitions here, but these definitions are functional for most of our
data analysis purposes.

A **probability mass function** (PMF), $f(x)$, describes the
probability of a discrete variable obtaining value $x$.  The variable
$x$ takes on discrete values, so the **normalization condition**
is

\begin{align}
\sum_x f(x) = 1.
\end{align}

A **probability density function** (PDF), which we shall
call $f(x)$, is defined such that the probability that the value of a
continuous variable $x$ is $a \le x \le b$ is

\begin{align}
\int_a^b \mathrm{d}x\,f(x).
\end{align}

A **cumulative distribution function** (CDF), denoted $F(x)$ is defined such that $F(x)$ is the probability that the value of a variable $X$ is less than or equal to $x$. For a discrete distribution

\begin{align}
F(k) = \sum_{k'=k_\mathrm{min}}^k f(k'),
\end{align}

where $k_\mathrm{min}$ is the minimal value the variable can take, and for a continuous distribution,

\begin{align}
F(x) = \int_{-\infty}^x \mathrm{d}x'\,f(x').
\end{align}

If a probability mass or density function depends on parameters, say $n$ and $\theta$, we write it as $f(x;n, \theta)$. There does not seem to be consensus on the best notation, and you may see this same quantity written as $P(x\mid n, \theta)$, for example.

### Sampling

Given that we know a probability distribution, we can take
*samples* out of it.  This means that we can randomly draw
numbers and the probability that we draw a certain number $x$ is
proportional to the PMF or PDF, $f(x)$. Sampling
out of a distribution is often easier than computing the distribution
over a range of values because many of those values are zero.

The `numpy.random` module and [Stan](http://mc-stan.org/) are power tools for sampling out of distributions. For each distribution I describe, I show how to specify them using NumPy and Stan (and also with the `scipy.stats` module).

## Distributions and their stories

In what follows, I will present some commonly used probability distributions and their associated stories. I give the following.
- Name of the distribution.
- Its story.
- An example.
- The parameters of the distribution.
- The **support**, or the domain for which the probability is nonzero.
- The probability mass function (for discrete distributions) or probability density function (for continuous distributions). **I use the parametrization that Stan uses.**
- Syntax for how to use the distribution using `numpy.random`, `scipy.stats`, and Stan.
- Description of relations to other distributions.
- Miscellaneous notes.
- Interactive plots of the PMF/PDF and CDF of the distribution.

I omit analytical expression for the CDF because CDFs are often expressed as special functions, such as regularized incomplete beta functions or error functions. They can also be easily looked up. We mainly use them for comparing ECDFs in plots, and we can use the `scipy.stats` module to just compute them.

Along the way, we'll define other terms, such as **Bernoulli trial** and **Poisson process**. I will use the `scipy.stats` module (imported as `st`) to compute the PDF/PMF and CDF. The API for this module will be apparent as I use it.

Finally, there is often more than one way to define the parameters of a distribution. I will be using the definitions and variable names that are used in [Stan](http://mc-stan.org/).

## Visualization of PDF/PMF's and CDFs

I use [Bokeh](https://bokeh.pydata.org/) to construct interactive visualizations of the distribution. The `distribution_explorer.explore()` function creates Bokeh apps to visualize the distributions.

### A note about CDFs for discrete distributions

I plot the CDFs for discrete distributions as "staircases." As an example, here is a plot of the CDF of the [Binomial distribution](#Binomial-distribution) with parameters $N=10$ and $\theta = 0.5$.

```python
x = np.arange(0, 11)
x_size = x[-1] - x[0]
x_c = np.empty(2*len(x))
x_c[::2] = x
x_c[1::2] = x
x_c = np.concatenate(((max(x[0] - 0.05*x_size, x[0] - 0.95),),
                      x_c,
                      (min(x[-1] + 0.05*x_size, x[-1] + 0.95),)))
x_cdf = np.concatenate(((x_c[0],), x))

y = st.binom.cdf(x_cdf, 10, 0.5)
y_c = np.empty_like(x_c)
y_c[::2] = y
y_c[1::2] = y

p = bokeh.plotting.figure(plot_height=250,
                          plot_width=350,
                          x_axis_label='n',
                          y_axis_label='F(n; 10, 0.5)')
p.line(x_c, y_c, line_width=2)
bokeh.io.show(p)
```

The CDF appears to be multivalued at the vertical lines of the staircase. It is not. Furthermore, the lines at zero and one on the CDF axis should extend out to $-\infty$ and $\infty$, respectively along the horizontal axis. Strictly speaking, it should be plotted as follows.

```python
x = np.arange(0, 11)
y = st.binom.cdf(x, 10, 0.5)

p = bokeh.plotting.figure(plot_height=250,
                          plot_width=350,
                          x_axis_label='n',
                          y_axis_label='F(n; 10, 0.5)')
p.segment(x[:-1], y[:-1], x[1:], y[:-1], line_width=2)
p.ray(0, 0, angle=np.pi, length=0, line_width=2)
p.ray(x[-1], 1, angle=0, length=0, line_width=2)
p.circle([0], [0], fill_color='white')
p.circle(x[1:], y[:-1], fill_color='white')
p.circle(x, y)
bokeh.io.show(p)
```

However, since it is understood that the CDF is not multivalued, there should be no ambiguity in plotting the staircase, and indeed staircase style CDFs are commonly used. The staircase has less clutter and I find it is easier to look at and interpret. Furthemore, we know that all CDFs extend toward $x=-\infty$ with a value of zero and toward $x=\infty$ with a value of one. So, again, there is no ambiguity in cutting off the infinitely long tails of the CDF.

## Discrete distributions

### Bernoulli distribution

* **Story.** A *Bernoulli trial* is an experiment that has two outcomes that can be encoded as success ($y=1$) or failure ($y = 0$). The result $y$ of a Bernoulli trial is Bernoulli distributed.

* **Example.** Check to see if a given bacterium is competent, given that it has probability $\theta$ of being competent.

* **Parameter.** The Bernoulli distribution is parametrized by a
single value, $\theta$, the probability that the trial is successful.

* **Support.**
The Bernoulli distribution may be nonzero only for zero and one.

* **Probability mass function.**
\begin{align}
f(y;\theta) = \left\{ \begin{array}{ccc}
1-\theta & & y = 0 \\[0.5em]
\theta & & y = 1.
\end{array}
\right.
\end{align}

* **Usage**

| Package        | Syntax           |
|: ------------- | :------------- |
|**NumPy** | `np.random.choice([0, 1], p=[1-theta, theta])` |
|**SciPy** | `scipy.stats.bernoulli(theta)` |
|**Stan** | `bernoulli(theta)` |

* **Related distributions.**
    - The Bernoulli distribution is a special case of the [Binomial distribution](#Binomial-distribution) with $N=1$.

```python
bokeh.io.show(distribution_explorer.explore('bernoulli'))
```

### Geometric distribution

* **Story.** We perform a series of Bernoulli trials with probability of success $\theta$ until we
get a success.  The number of failures $y$ before the success is Geometrically distributed.

* **Example.** Consider actin polymerization.  At each time
step, an actin monomer may add to the filament ("failure"), or an
actin monomer may fall off ("success") with (usually very low)
probability $\theta$.  The length of actin filaments are Geometrically
distributed.

* **Parameter.** The Geometric distribution is parametrized by a
single value, $\theta$, the probability that the Bernoulli trial is
successful.

* **Support.** The Geometric distribution, as defined here, has support on the nonnegative integers.

* **Probability mass function.**

\begin{align}
f(y;\theta) = (1-\theta)^y \, \theta.
\end{align}

* **Usage**

| Package        | Syntax           |
|: ------------- | :------------- |
|**NumPy** | `np.random.geometric(theta)` |
|**SciPy** | `scipy.stats.geom(theta, loc=-1)` |
|**Stan** | `neg_binomial(1, theta/(1-theta))` |

* **Related distributions.**
    - The Geometric distribution is a special case of the [Negative Binomial distribution](#Negative-Binomial-distribution) in which $\alpha=1$ and $\theta = \beta/(1+\beta)$.
    - The continuous analog of the Geometric distribution is the [Exponential distribution](#Exponential-distribution).

* **Notes.**
    - The Geometric distribution is supported on non-negative integer $y$.
    - The Geometric distribution is not implemented in Stan. You can instead use a Negative Binomial distribution fixing the parameter $\alpha$ to be unity and relating the parameter $\beta$ of the Negative Binomial distribution to $\theta$ as $\theta = \beta/(1+\beta)$.
    - The Geometric distribution is defined differently in SciPy, instead replacing $y$ with $y-1$. In SciPy's parametrization the Geometric distribution describes the number of successive Bernoulli *trials* (not just the failures; the success is included) necessary to get a success. To adjust for this, we use the `loc=-1` kwarg.

```python
bokeh.io.show(distribution_explorer.explore('geometric'))
```

### Negative Binomial distribution

* **Story.** We perform a series of Bernoulli trials with probability $\beta/(1+\beta)$ of success. The number of failures, $y$, before we get $\alpha$
successes is Negative Binomially distributed. An equivalent story is that the sum of $\alpha$ independent and identically Gamma distributed variables is Negative Binomial distributed.

* **Example.** Bursty gene expression can give mRNA count
distributions that are Negative Binomially distributed.  Here,
"success" is that a burst in gene expression stops. In this case, the
parameter $1/\beta$ is the mean number of transcripts in a burst of expression.  The parameter $\alpha$ is related to the
frequency of the bursts.  If multiple bursts are possible within the
lifetime of mRNA, then $\alpha > 1$.  Then, the number of "failures" is
the number of mRNA transcripts that are made in the characteristic
lifetime of mRNA.

* **Parameters.** There are two parameters: $\alpha$, the desired number of
successes, and $\beta$, which is the mean of the $\alpha$ identical Gamma distributions that give the Negative Binomial. The probability of success of each Bernoulli trial is given by $\beta/(1+\beta)$.

* **Support.** The Negative-Binomial distribution is supported on the set of nonnegative integers.

* **Probability mass function.**
\begin{align}
\\ \phantom{blah}
f(y;\alpha,\beta) = \begin{pmatrix}
y+\alpha-1 \\
\alpha-1
\end{pmatrix}
\left(\frac{\beta}{1+\beta}\right)^\alpha \left(\frac{1}{1+\beta}\right)^y.
\\ \phantom{blah}
\end{align}
Here, we use a combinatorial notation;
\begin{align}
\\ \phantom{blah}
  \begin{pmatrix}
y+\alpha-1 \\
\alpha-1
\end{pmatrix} = \frac{(y+\alpha-1)!}{(\alpha-1)!\,y!}.
\\ \phantom{blah}
\end{align}
Generally speaking, $\alpha$ need not be an integer, so we may write the PMF as
\begin{align}
\\ \phantom{blah}
f(y;\alpha,\beta) = \frac{\Gamma(y+\alpha)}{\Gamma(\alpha) \, y!}\,\left(\frac{\beta}{1+\beta}\right)^\alpha \left(\frac{1}{1+\beta}\right)^y.
\end{align}

* **Usage**

| Package        | Syntax           |
|: ------------- | :------------- |
|**NumPy** | `np.random.negative_binomial(alpha, beta/(1+beta))` |
|**SciPy** | `scipy.stats.nbinom(alpha, beta/(1+beta))` |
|**Stan** | `neg_binomial(alpha, beta)` |
|**Stan with $(\mu, \phi)$ parametrization** | `neg_binomial_2(mu, phi)` |

* **Related distributions.**
    - The [Geometric distribution](#Geometric-distribution) is a special case of the Negative Binomial distribution in which $\alpha=1$ and $\theta = \beta/(1+\beta)$.
    - The continuous analog of the Negative Binomial distribution is the [Gamma distribution](#Gamma-distribution).
    - In a certain limit, which is easier implemented using the $(\mu,\phi)$ parametrization below, the Negative Binomial distribution becomes a [Poisson distribution](#Poisson-distribution).

* **Notes.**
    - The Negative Binomial distribution may be parametrized such that the probability mass function is
    \begin{align}
    \\ \phantom{blah}
       f(y;\mu,\phi) = \frac{\Gamma(y+\phi)}{\Gamma(\phi) \, y!}\,\left(\frac{\phi}{\mu+\phi}\right)^\phi\left(\frac{\mu}{\mu+\phi}\right)^y.
    \\ \phantom{blah}
    \end{align}
    These parameters are related to the parametrization above by $\phi = \alpha$ and $\mu = \alpha/\beta$. In the limit of $\phi\to\infty$, which can be taken for the PMF, the Negative Binomial distribution becomes Poisson with parameter $\mu$. This also gives meaning to the parameters $\mu$ and $\phi$. $\mu$ is the mean of the Negative Binomial, and $\phi$ controls extra width of the distribution beyond Poisson. The smaller $\phi$ is, the broader the distribution.
    - In Stan, the Negative Binomial distribution using the $(\mu,\phi)$ parametrization is called `neg_binomial_2`.
    - SciPy and NumPy use yet another parametrization. The PMF for SciPy is
    \begin{align}
    \\ \phantom{blah}
       f(y;n, p) = \frac{\Gamma(y+n)}{\Gamma(n) \, y!}\,p^n \left(1-p\right)^y.
    \\ \phantom{blah}
    \end{align}
    The parameter $p$ is the probability of success of a Bernoulli trial. The parameters are related to the others we have defined by $n=\alpha=\phi$ and $p=\beta/(1+\beta) = \phi/(\mu+\phi)$.

```python
# With α, β parametrization
bokeh.io.show(distribution_explorer.explore('negative_binomial'))
```

```python
# With μ, φ parametrization
bokeh.io.show(distribution_explorer.explore('negative_binomial_mu_phi'))
```

### Binomial distribution

* **Story.** We perform $N$ Bernoulli trials, each with probability $\theta$ of success.  The number of successes, $n$, is Binomially distributed.

* **Example.** Distribution of plasmids between daughter cells
in cell division.  Each of the $N$ plasmids as a chance $\theta$ of being
in daughter cell 1 ("success").  The number of plasmids, $n$, in
daughter cell 1 is binomially distributed.

* **Parameters.** There are two parameters: the probability $\theta$ of success for each Bernoulli trial, and the number of trials, $N$.

* **Support.** The Binomial distribution is supported on the set of nonnegative integers less than or equal to $N$.

* **Probability mass function.**

\begin{align}
f(n;N,\theta) = \begin{pmatrix}
N \\
n
\end{pmatrix}
\theta^n (1-\theta)^{N-n}.
\end{align}

* **Usage**

| Package        | Syntax           |
|: ------------- | :------------- |
|**NumPy** | `np.random.binomial(N, theta)` |
|**SciPy** | `scipy.stats.binom(N, theta)` |
|**Stan** | `binomial(N, theta)` |

* **Related distributions.**
    - The [Bernoulli distribution](#Bernoulli-distribution) is a special case of the Binomial distribution where $N=1$.
    - In the limit of $N\to\infty$ and $\theta\to 0$ such that the quantity $N\theta$ is fixed, the Binomial distribution becomes a [Poisson distribution](#Poisson-distribution) with parameter $N\theta$.
    - The Binomial distribution is a limit of the [Hypergeometric distribution](#Hypergeometric-distribution). Considering the Hypergeometric distribution and taking the limit of $a+b\to\infty$ such that $a/(a+b)$ is fixed, we get a [Binomial distribution](#Binomial-distribution) with parameters $N=N$ and $\theta = a/(a+b)$.

```python
bokeh.io.show(distribution_explorer.explore('binomial'))
```

### Poisson distribution

* **Story.**  Rare events occur with a rate $\lambda$ per unit
time.  There is no "memory" of previous events; i.e., that rate is
independent of time. A process that generates such events is called a *Poisson process*. The occurrence of a rare event in this context is referred to as an *arrival*. The number $n$ of arrivals in unit time is Poisson distributed.

* **Example.** The number of mutations in a strand of DNA per
unit length (since mutations are rare) are Poisson distributed.

* **Parameter.** The single parameter is the rate $\lambda$ of
the rare events occurring.

* **Support.** The Poisson distribution is supported on the set of nonnegative integers.

* **Probability mass function.**
\begin{align}
f(n;\lambda) = \frac{\lambda^n}{n!}\,\mathrm{e}^{-\lambda}.
\end{align}

* **Usage**

| Package        | Syntax           |
|: ------------- | :------------- |
|**NumPy** | `np.random.poisson(lam)` |
|**SciPy** | `scipy.stats.poisson(lam)` |
|**Stan** | `poisson(lam)` |

* **Related distributions.**
    - In the limit of $N\to\infty$ and $\theta\to 0$ such that the quantity $N\theta$ is fixed, the [Binomial distribution](#Binomial-distribution) becomes a Poisson distribution with parameter $N\theta$. Thus, for large $N$ and small $\theta$,
    \begin{align}
    \\ \phantom{blah}
    f_\mathrm{Poisson}(n;\lambda) \approx f_\mathrm{Binomial}(n;N, \theta),
    \\ \phantom{blah}
    \end{align}
    with $\lambda = N\theta$. Considering the biological example of mutations,
    this is Binomially distributed: There are $N$ bases, each with a
    probability $\theta$ of mutation, so the number of mutations, $n$ is
    binomially distributed.  Since $\theta$ is small and $N$ is large, it is approximately
    Poisson distributed.
    - Under the ($\mu, \phi$) parametrization of the [Negative Binomial distribution](#Negative-Binomial-distribution), taking the limit of large $\phi$ yields the Poisson distribution.

```python
bokeh.io.show(distribution_explorer.explore('poisson'))
```

### Hypergeometric distribution

* **Story.** Consider an urn with $a$ white balls and $b$ black
balls.  Draw $N$ balls from this urn without replacement.  The number
white balls drawn, $n$, is Hypergeometrically distributed.

* **Example.** There are $a+b$ finches on an island, and $a$ of
them are tagged (and therefore $b$ of them are untagged).  You capture $N$ finches.  The number of tagged
finches $n$ is Hypergeometrically distributed, $f(n;N, a, b)$,
as defined below.

* **Parameters.** There are three parameters: the number of
draws $N$, the number of white balls $a$, and the number of black
balls $b$.

* **Support.** The Hypergeometric distribution is supported on the set of integers between $\max(0, N-b)$ and $\min(N, a)$, inclusive.

* **Probability mass function.**
\begin{align}
f(n;N, a, b) = \frac{\begin{pmatrix}a\\n\end{pmatrix}\begin{pmatrix}b\\N-n\end{pmatrix}}
{\begin{pmatrix}a+b\\N\end{pmatrix}}.
\end{align}

* **Usage**

| Package        | Syntax           |
|: ------------- | :------------- |
|**NumPy** | `np.random.hypergeometric(a, b, N)` |
|**SciPy** | `scipy.stats.hypergeom(a+b, a, N)` |
|**Stan** | `hypergeometric(N, a, b)` |

* **Related distributions.**
    - In the limit of $a+b\to\infty$ such that $a/(a+b)$ is fixed, we get a [Binomial distribution](#Binomial-distribution) with parameters $N=N$ and $\theta = a/(a+b)$.

* **Notes.**
    - This distribution is analogous to the [Binomial distribution](#Binomial-distribution), except that the Binomial distribution describes draws from an urn *with* replacement.  In the analogy, $p = a/(a+b)$.
    - SciPy uses a different parametrization. Let $M = a+b$ be the total number of balls in the urn. Then, noting the order of the parameters, since this is what `scipy.stats.hypergeom` expects,
    \begin{align}
    \\ \phantom{blah}
    f(n;M, a, N) = \frac{\begin{pmatrix}a\\n\end{pmatrix}\begin{pmatrix}M-a\\N-n\end{pmatrix}}
    {\begin{pmatrix}M\\n\end{pmatrix}}.
        \\ \phantom{blah}
    \end{align}
    - The random number generator in `numpy.random` has a different parametrization than in the `scipy.stats` module. The `numpy.random.hypergeom()` function uses the same parametrization as Stan, except the parameters are given in the order `a, b, N`, not `N, a, b`, as in Stan.

```python
bokeh.io.show(distribution_explorer.explore('hypergeometric'))
```

### Categorical distribution

* **Story.** A probability is assigned to each of a set of discrete outcomes.

* **Example.** A hen will peck at grain A with probability $\theta_A$, grain B with probability $\theta_B$, and grain C with probability $\theta_C$.

* **Parameters.** The distribution is parametrized by the probabilities assigned to each event. We define $\theta_y$ to be the probability assigned to outcome $y$. The set of $\theta_y$'s are the parameters, and are constrained by

\begin{align}
\sum_y \theta_y = 1.
\end{align}

* **Support.** If we index the categories with sequential integers from 1 to *N*, the distribution is supported for integers 1 to _N_, inclusive.

* **Probability mass function.**
\begin{align}
f(y;\{\theta_y\}) = \theta_y.
\end{align}

* **Usage (with `theta` length `n`)**

| Package        | Syntax           |
|: -------------: | :------------- |
|**NumPy** | `np.random.choice(len(theta), p=theta)` |
|**SciPy** | `scipy.stats.rv_discrete(values=(range(len(theta)), theta)).rvs()` |
|**Stan** | `categorical(theta)` |

* **Related distributions.**
    - The [Discrete Uniform distribution](#Discrete-Uniform-distribution) is a special case where all $\theta_y$ are equal.
    - The Bernoulli distribution is a special case where there are two categories that can be encoded as having outcomes of zero or one. In this case, the parameter for the Bernoulli distribution is $\theta = \theta_0 = 1-\theta_1$.

* **Notes.**
    - This distribution must be manually constructed if you are using the `scipy.stats` module using `scipy.stats.rv_discrete()`. The categories need to be encoded by an index. For interactive plotting purposes, below, we need to specify a custom PMF and CDF.
    - To sample out of a Categorical distribution, use `numpy.random.choice()`, specifying the values of $\theta$ using the `p` kwarg.

```python
bokeh.io.show(distribution_explorer.explore('categorical'))
```

### Discrete Uniform distribution

* **Story.** A set of discrete outcomes that can be indexed with sequential integers each has equal probability, like rolling a fair die.

* **Example.** A monkey can choose from any of $n$ bananas with equal probability.

* **Parameters.** The distribution is parametrized by the high and low allowed values.

* **Support.** The Discrete Uniform distribution is supported on the set of integers ranging from $y_\mathrm{low}$ to $y_\mathrm{high}$, inclusive.

* **Probability mass function.**
\begin{align}
f(y;y_\mathrm{low}, y_\mathrm{high}) = \frac{1}{y_\mathrm{high} - y_\mathrm{low} + 1}
\end{align}

* **Usage**

| Package        | Syntax           |
|: -------------: | :------------- |
|**NumPy** | `np.random.randint(low, high+1)` |
|**SciPy** | `scipy.stats.randint(low, high+1)` |
|**Stan** | `categorical(theta)`, `theta` array with all equal values |

* **Related distributions.**
    - The [Discrete Uniform distribution](#Discrete-Uniform-distribution) is a special case of the [Categorical distribution](#Categorical-distribution) where all $\theta_y$ are equal.

* **Notes.**
    - This distribution is not included in Stan. Instead, use a [Categorical distribution](#Categorical-distribution) with equal probailities.
    - In SciPy, this distribution is know as `scipy.stats.randint`. The `high` parameter is not inclusive; i.e., the set of allowed values includes the `low` parameter, but not the `high`. The same is true for `numpy.random.randint()`, which is used for sampling out of this distribution.

```python
bokeh.io.show(distribution_explorer.explore('discrete_uniform'))
```

## Continuous distributions

### Uniform distribution

* **Story.** Any outcome in a given range has equal probability.

* **Example.** Anything in which all possibilities are equally
likely. This is, perhaps surprisingly, rarely encountered.

* **Parameters.** The Uniform distribution is not defined on an
infinite or semi-infinite domain, so lower and upper bounds, $\alpha$ and
$\beta$, respectively, are necessary parameters.

* **Support.** The Uniform distribution is supported on the interval $[\alpha, \beta]$.

* **Probability density function.**

\begin{align}
f(y;\alpha, \beta) = \left\{ \begin{array}{ccc}
\frac{1}{\beta - \alpha} & & \alpha \le y \le \beta \\[0.5em]
0 & & \text{otherwise.}
\end{array}
\right.
\end{align}

* **Usage**

| Package        | Syntax           |
|: -------------: | :------------- |
|**NumPy** | `np.random.uniform(alpha, beta)` |
|**SciPy** | `scipy.stats.uniform(alpha, beta)` |
|**Stan** | `uniform(alpha, beta)` |

* **Related distributions.**
    - The Uniform distribution on the interval [0, 1] (i.e., $\alpha=0$ and $\beta=1$) is a special case of the [Beta distribution](#Beta-distribution) where the parameters for the Beta distribution are $\alpha=\beta=1$ (not to be confused with the $\alpha$ and $\beta$ used to parametrize the Uniform distribution).

```python
bokeh.io.show(distribution_explorer.explore('uniform'))
```

### Gaussian, a.k.a. Normal, distribution

* **Story.** Any quantity that emerges as the sum of a large number of
subprocesses tends to be Gaussian distributed provided none of the
subprocesses is very broadly distributed.

* **Example.** We measure the length of many *C. elegans*
eggs.  The lengths are Gaussian distributed. Many biological measurements, like the height of people, are (approximately) Gaussian distributed. Many processes contribute to setting the length of an egg or the height of a person.

* **Parameters.** The Gaussian distribution has two parameters,
the mean $\mu$, which determines the location of its peak, and the
standard deviation $\sigma$, which is strictly positive (the
$\sigma\to 0$ limit defines a Dirac delta function) and determines the
width of the peak.

* **Support.** The Gaussian distribution is supported on the set of real numbers.

* **Probability density function.**

\begin{align}
f(y;\mu, \sigma) = \frac{1}{\sqrt{2\pi \sigma^2}}\,\mathrm{e}^{-(y-\mu)^2/2\sigma^2}.
\end{align}

* **Usage**

| Package        | Syntax           |
|: -------------: | :------------- |
|**NumPy** | `np.random.normal(mu, sigma)` |
|**SciPy** | `scipy.stats.norm(mu, sigma)` |
|**Stan** | `normal(mu, sigma)` |

* **Related distributions.** The Gaussian distribution is a limiting distribution in the sense of the
central limit theorem, but also in that many distributions have a
Gaussian distribution as a limit.  This is seen by formally taking
limits of, e.g., the Gamma, Student-t, Binomial distributions, which
allows direct comparison of parameters.

* **Notes.**
    - SciPy, NumPy, and Stan all refer to the Gaussian distribution as the Normal distribution.

```python
bokeh.io.show(distribution_explorer.explore('normal'))
```

### Half-Normal distribution

* **Story.** The Half-Normal distribution is a [Gaussian distribution](#Gaussian-distribution) with zero mean truncated to only have nonzero probability for positive real numbers.

* **Parameters.** Strictly speaking, the Half-Normal is parametrized by a single positive parameter $\sigma$. We could, however, translate it so that the truncated Gaussian has a maximum at $\mu$ and only values greater than or equal to $\mu$ have nonzero probability.

* **Probability density function.**

\begin{align}
f(y;\mu, \sigma) = \left\{\begin{array}{ccc}
\frac{2}{\sqrt{2\pi \sigma^2}}\,\mathrm{e}^{-(y-\mu)^2/2\sigma^2} & & y \ge \mu\\
0 && \text{otherwise.}
\end{array}\right.
\end{align}

* **Support.** The Half-Normal distribution is supported on the set $[\mu, \infty)$. Usually, we have $\mu = 0$, in which case the Half-Normal distribution is supported on the set of nonnegative real numbers.

* **Usage**

| Package        | Syntax           |
|: -------------: | :------------- |
|**NumPy** | `mu + np.abs(np.random.normal(0, sigma))` |
|**SciPy** | `scipy.stats.halfnorm(mu, sigma)` |
|**Stan sampling** | `real<lower=mu> y; y ~ normal(mu, sigma)` |
|**Stan rng** | `real<lower=mu> y; y = mu + abs(normal_rng(0, sigma))` |

* **Notes.**
    - In Stan, a Half-Normal is defined by putting bounds on the variable and then using a Normal distribution.
    - The Half-Normal distibution is a useful prior for nonnegative parameters that should not be too large and may be very close to zero.

```python
params = [dict(name='µ', start=-0.5, end=0.5, value=0, step=0.01),
          dict(name='σ', start=0.1, end=1.0, value=0.2, step=0.01)]
app = distribution_plot_app(x_min=-0.5,
                            x_max=2,
                            scipy_dist=st.halfnorm,
                            params=params,
                            x_axis_label='y',
                            title='Half-Normal')
bokeh.io.show(app, notebook_url=notebook_url)
```

### Log-Normal distribution

* **Story.** If $\ln y$ is Gaussian distributed, $y$ is
Log-Normally distributed.

* **Example.** A measure of fold change in gene expression can
be Log-Normally distributed.

* **Parameters.** As for a Gaussian, there are two parameters,
the mean, $\mu$, and the variance $\sigma^2$. Note that $\mu$ is the mean of $\ln y$, not of $y$ itself. That is, $\langle \ln y \rangle_\text{LogNorm} = \mu$. Similarly, $\sigma^2$ is the variance in $\ln y$, i.e, $\langle (\ln y -\mu)^2\rangle_\text{LogNorm} = \sigma^2$.

* **Support.** The Log-Normal distribution is supported on the set of positive real numbers.

* **Probability density function.**

\begin{align}
f(y;\mu, \sigma) = \frac{1}{y\sqrt{2\pi \sigma^2}}\,\mathrm{e}^{-(\ln y - \mu)^2/2\sigma^2}.
\end{align}

* **Usage**

| Package        | Syntax           |
|: -------------: | :------------- |
|**NumPy** | `np.random.lognormal(mu, sigma)` |
|**SciPy** | `scipy.stats.lognorm(sigma, loc=0, scale=np.exp(mu))` |
|**Stan** | `lognormal(mu, sigma)` |

* **Notes.**
    - Be careful not to get confused. The Log-Normal distribution describes the distribution of $y$ given that $\ln y$ is Gaussian distributed. It does not describe the distribution of $\ln y$.
    - The way location, scale, and shape parameters work in SciPy for the Log-Normal distribution is confusing. If you want to specify a Log-Normal distribution as we have defined it using `scipy.stats`, use a shape parameter equal to $\sigma$, a location parameter of zero, and a scale parameter given by $\mathrm{e}^\mu$. For example, to compute the PDF, you would use `scipy.stats.lognorm.pdf(x, sigma, loc=0, scale=np.exp(mu))`.
    - The definition of the Log_Normal in the `numpy.random` module matches what we have defined above and what is defined in Stan.

```python
params = [dict(name='µ', start=0.01, end=0.5, value=0.1, step=0.01),
          dict(name='σ', start=0.1, end=1.0, value=0.2, step=0.01)]
app = distribution_plot_app(x_min=0,
                            x_max=4,
                            scipy_dist=st.lognorm,
                            params=params,
                            transform=lambda mu, sigma: (sigma, 0, np.exp(mu)),
                            x_axis_label='y',
                            title='Log-Normal')

bokeh.io.show(app, notebook_url=notebook_url)
```

### Chi-square distribution

* **Story.** If $X_1$,
$X_2$, $\ldots$, $X_n$ are Gaussian distributed,
$X_1^2 + X_2^2 + \cdots + X_n^2$ is $\chi^2$-distributed. See also the story of the [Gamma distribution](#Gamma-distribution), below.

* **Example.** The sample variance of $\nu-1$ independent and identically distributed Gaussian random variables, after scaling, is Chi-square distributed. This is the most common use case of the Chi-square distribution.

* **Parameters.** There is only one parameter, the degrees of freedom $\nu$.

* **Support.** The Chi-square distribution is supported on the positive real numbers.

* **Probability density function.**
\begin{align}
  f(y;\nu) \equiv \chi^2_n(x;\nu) = \frac{1}{2^{\nu/2}\,\Gamma\left(\frac{\nu}{2}\right)}\,
x^{\frac{\nu}{2}-1}\,\mathrm{e}^{-y/2}.
\end{align}

* **Usage**

| Package        | Syntax           |
|: -------------: | :------------- |
|**NumPy** | `np.random.chisquare(nu)` |
|**SciPy** | `scipy.stats.chi2(nu)` |
|**Stan** | `chi_square(nu)` |

* **Related distributions.** The Chi-square distribution is a special case of the [Gamma distribution](#Gamma-distribution) with $\alpha = \nu/2$ and $\beta = 1/2$.

```python
params = [dict(name='ν', start=1, end=20, value=10, step=0.01)]
app = distribution_plot_app(x_min=0,
                            x_max=40,
                            scipy_dist=st.chi2,
                            params=params,
                            x_axis_label='y',
                            title='Chi-square')
bokeh.io.show(app, notebook_url=notebook_url)
```

### Student-t distribution

* **Story.** The story of the Student-t distribution largely derives from its relationships with other distributions. One way to think about it is as a Gaussian-like distribution with heavier tails.

* **Parameters.** The Student-t distribution is peaked, and its peak is located at $\mu$.  The peak's width is dictated by parameter $\sigma$. Finally, we define the "degrees of freedom" as $\nu$. This last parameter imparts the distribution with heavy tails.

* **Support.** The Student-t distribution is supported on the set of real numbers.

* **Probability density function.**

\begin{align}
f(y;\mu, \sigma, \nu) = \frac{\Gamma\left(\frac{\nu+1}{2}\right)}{\Gamma\left(\frac{\nu}{2}\right)\sqrt{\pi \nu \sigma^2}}\,
\left(1 + \frac{(y-\mu)^2}{\nu\sigma^2}\right)^{-\frac{\nu+1}{2}}.
\end{align}

* **Usage**

| Package        | Syntax           |
|: -------------: | :------------- |
|**NumPy** | `mu + sigma * np.random.standard_t(nu)` |
|**SciPy** | `scipy.stats.t(nu, mu, sigma)` |
|**Stan** | `student_t(nu, mu, sigma)` |

* **Related distributions.**
    - In the $\nu\to \infty$ limit, the Student-t distribution becomes as [Gaussian distribution](#Gaussian-distribution).
    - The [Cauchy distibution](#Cauchy-distribution) is a special case of the Student-t distribution in which $\nu = 1$.
    - Only the standard Student-t distribution ($\mu=0$ and $\sigma = 1$) is available in the `numpy.random` module. You can still draw out of the Student-t distribution by performing a transformation on the samples out of the standard Student-t distribution, as shown in the usage, above.
    - We get this distribution whenever we marginalize an unknown $\sigma$ out of a Gaussian distribution with an improper prior on $\sigma$ of $1/\sigma$.

```python
params = [dict(name='ν', start=1, end=50, value=10, step=0.01),
          dict(name='µ', start=-0.5, end=0.5, value=0, step=0.01),
          dict(name='σ', start=0.1, end=1.0, value=0.2, step=0.01)]
app = distribution_plot_app(x_min=-2,
                            x_max=2,
                            scipy_dist=st.t,
                            params=params,
                            x_axis_label='y',
                            title='Student-t')
bokeh.io.show(app, notebook_url=notebook_url)
```

### Cauchy distribution

* **Story.** The intercept on the *x*-axis of a beam of light coming from the point $(\mu, \sigma)$ is Cauchy distributed. This story is popular in physics (and is one of the first examples of Bayesian inference in [Sivia's book](https://global.oup.com/academic/product/data-analysis-9780198568322)), but is not particularly useful. You can think of it as a peaked distribution with enormously heavy tails.

* **Parameters.** The Cauchy distribution is peaked, and its peak is located at $\mu$.  The peak's width is dictated by parameter $\sigma$.

* **Support.** The Cauchy distribution is supported on the set of real numbers.

* **Probability density function.**

\begin{align}
f(y;\mu, \sigma) = \frac{1}{\pi \sigma}\,
\frac{1}{1 + (y-\mu)^2/\sigma^2}.
\end{align}

* **Usage**

| Package        | Syntax           |
|: -------------: | :------------- |
|**NumPy** | `mu + sigma * np.random.standard_cauchy()` |
|**SciPy** | `scipy.stats.cauchy(mu, sigma)` |
|**Stan** | `cauchy(mu, sigma)` |

* **Related distributions.**
    - The Cauchy distribution is a special case of the [Student-t distribution](Student-t-distribution) in which the degrees of freedom *ν* = 1.
    - The `numpy.random` module only has the Standard Cauchy distribution ($\mu = 0$ and $\sigma = 1$), but you can draw out of a Cauchy distribution using the transformation shown in the NumPy usage above.

```python
params = [dict(name='µ', start=-0.5, end=0.5, value=0, step=0.01),
          dict(name='σ', start=0.1, end=1.0, value=0.2, step=0.01)]
app = distribution_plot_app(x_min=-2,
                            x_max=2,
                            scipy_dist=st.cauchy,
                            params=params,
                            x_axis_label='y',
                            title='Cauchy')
bokeh.io.show(app, notebook_url=notebook_url)
```

### Exponential distribution

* **Story.** This is the waiting time for an arrival from a
Poisson process.  I.e., the inter-arrival time of a Poisson process is
Exponentially distributed.

* **Example.** The time between conformational switches in a
protein is Exponentially distributed (under simple mass action
kinetics).

* **Parameter.** The single parameter is the average arrival
*rate*, $\beta$. Alternatively, we can use $\tau=1/\beta$ as the parameter, in this case a characteristic arrival *time*.

* **Support.** The Exponential distribution is supported on the set of nonnegative real numbers.

* **Probability density function.**
\begin{align}
f(y;\beta) = \beta\, \mathrm{e}^{-\beta y}.
\end{align}

* **Related distributions.**
    - The Exponential distribution is the continuous analog of the [Geometric distribution](#Geometric-distribution).  The "rate" in the Exponential distribution is analogous to the probability of success of the Bernoulli trial. Note also that because they are uncorrelated, the amount of time between any two arrivals is independent of all other inter-arrival times.
    - The Exponential distribution is a special case of the [Gamma distribution](#Gamma-distribution) with parameter $\alpha = 1$.

* **Usage**

| Package        | Syntax           |
|: -------------: | :------------- |
|**NumPy** | `np.random.exponential(1/beta)` |
|**SciPy** | `scipy.stats.expon(loc=0, scale=1/beta)` |
|**Stan** | `exponential(beta)` |

* **Notes.**
    - Alternatively, we could parametrize the Exponential distribution in terms of an average time between arrivals of a Poisson process, $\tau$, as
    \begin{align}
    \\\phantom{blah}
    f(y;\tau) = \frac{1}{\tau}\, \mathrm{e}^{-y/\tau}.
    \\\phantom{blah}
    \end{align}
    - The implementation in the `scipy.stats` module also has a location parameter, which shifts the distribution left and right. For our purposes, you can ignore that parameter, but be aware that `scipy.stats` requires it. The `scipy.stats` Exponential distribution is parametrized in terms of the interarrival time, $\tau$, and not $\beta$.
    - The `numpy.random.exponential()` function does not need nor accept a location parameter.  It is also parametrized in terms of $\tau$.

```python
params = [dict(name='β', start=0.1, end=1, value=0.25, step=0.01)]
app = distribution_plot_app(0,
                            30,
                            st.expon,
                            params=params,
                            transform=lambda x: (0, 1/x),
                            x_axis_label='y',
                            title='Exponential')
bokeh.io.show(app, notebook_url=notebook_url)
```

### Gamma distribution

* **Story.** The amount of time we have to wait for $\alpha$ arrivals
of a Poisson process. More concretely, if we have events, $X_1$,
$X_2$, $\ldots$, $X_\alpha$ that are exponentially distributed,
$X_1 + X_2 + \cdots + X_\alpha$ is Gamma distributed.

* **Example.** Any multistep process where each step happens at
the same rate.  This is common in molecular rearrangements.

* **Parameters.** The number of arrivals, $\alpha$, and the rate of
arrivals, $\beta$.

* **Support.** The Gamma distribution is supported on the set of positive real numbers.

* **Probability density function.**

\begin{align}
f(y;\alpha, \beta) = \frac{1}{\Gamma(\alpha)}\,\frac{(\beta y)^\alpha}{y}\,\mathrm{e}^{-\beta y}.
\end{align}

* **Related distributions.**
    - The Gamma distribution is the continuous
analog of the [Negative Binomial distribution](#Negative-Binomial-distribution).
    - The special case where $\alpha=1$ is an [Exponential distribution](#Exponential-distribution).
    - The special case where $\alpha = \nu/2$ and $\beta = 1/2$ is a [Chi-square distribution](#Chi-square-distribution) parametrized by $\nu$.

* **Usage**

| Package        | Syntax           |
|: -------------: | :------------- |
|**NumPy** | `np.random.gamma(alpha, 1/beta)` |
|**SciPy** | `scipy.stats.gamma(alpha, loc=0, scale=1/beta)` |
|**Stan** | `gamma(alpha, beta)` |

* **Notes.**
    - The Gamma distribution is useful as a prior for positive parameters. It imparts a heavier tail than the [Half-Normal distribution](#Half-Normal-distribution) (but not too heavy; it keeps parameters from growing too large), and allows the parameter value to come close to zero.
    - SciPy has a location parameter, which should be set to zero, with $\beta$ being the scale parameter.

```python
params = [dict(name='α', start=1, end=5, value=2, step=0.01),
          dict(name='β', start=0.1, end=5, value=2, step=0.01)]
app = distribution_plot_app(x_min=0,
                            x_max=10,
                            scipy_dist=st.gamma,
                            params=params,
                            transform=lambda a, b: (a, 0, 1/b),
                            x_axis_label='y',
                            title='Gamma')
bokeh.io.show(app, notebook_url=notebook_url)
```

### Inverse Gamma distribution

* **Story.** If $x$ is [Gamma distributed](#Gamma-distribution), then $1/x$ is Inverse Gamma distributed.

* **Parameters.** The number of arrivals, $\alpha$, and the rate of
arrivals, $\beta$.

* **Support.** The Inverse Gamma distribution is supported on the set of positive real numbers.

* **Probability density function.**

\begin{align}
f(y;\alpha, \beta) = \frac{1}{\Gamma(\alpha)}\,\frac{\beta^\alpha}{y^{(\alpha+1)}}\,\mathrm{e}^{-\beta/ y}.
\end{align}

* **Usage**

| Package        | Syntax           |
|: -------------: | :------------- |
|**NumPy** | `1 / np.random.gamma(alpha, 1/beta)` |
|**SciPy** | `scipy.stats.invgamma(alpha, loc=0, scale=beta)` |
|**Stan** | `inv_gamma(alpha, beta)` |

* **Notes.**
    - The Inverse Gamma distribution is useful as a prior for positive parmeters. It imparts a quite heavy tail and keeps probability further from zero than the [Gamma distribution](#Gamma-distribution).
    - The `numpy.random` module does not have a function to sample directly from the Inverse Gamma distribution, but it can be achieved by sampling out of a Gamma distribution as shown in the NumPy usage above.

```python
params = [dict(name='α', start=0.01, end=2, value=0.5, step=0.01),
          dict(name='β', start=0.1, end=2, value=1, step=0.01)]
app = distribution_plot_app(x_min=0,
                            x_max=20,
                            scipy_dist=st.invgamma,
                            params=params,
                            transform=lambda alpha, beta: (alpha, 0, beta),
                            x_axis_label='y',
                            title='Inverse Gamma')

bokeh.io.show(app, notebook_url=notebook_url)
```

### Weibull distribution

* **Story.** Distribution of $x = y^\alpha$ if $y$ is
exponentially distributed.  For $\alpha > 1$, the longer we have
waited, the more likely the event is to come, and vice versa for
$\alpha < 1$.

* **Example.** This is a model for aging.  The longer an
organism lives, the more likely it is to die.

* **Parameters.** There are two parameters, both strictly
positive: the shape parameter $\alpha$, which dictates the shape of the
curve, and the scale parameter $\sigma$, which dictates the rate of
arrivals of the event.

* **Support.**
The Weibull distribution has support on the positive real numbers.

* **Probability density function.**

\begin{align}
f(y;\alpha, \sigma) = \frac{\alpha}{\sigma}\left(\frac{y}{\sigma}\right)^{\alpha - 1}\,
\mathrm{e}^{-(y/\sigma)^\alpha}.
\end{align}

* **Usage**

| Package        | Syntax           |
|: -------------: | :------------- |
|**NumPy** | `np.random.weibull(alpha) * sigma` |
|**SciPy** | `scipy.stats.weibull_min(alpha, loc=0, scale=sigma)` |
|**Stan** | `weibull(alpha, sigma)` |

* **Related distributions.**
    - The special case where $\alpha = 1$ is the [Exponential distribution](#Exponential-distribution) with parameter $\beta = 1/\sigma$.

* **Notes.**
    - SciPy has a location parameter, which should be set to zero, with $\sigma$ being the scale parameter.
    - NumPy only provides a version of the Weibull distribution with $\sigma = 1$. Sampling out of the Weibull distribution may be accomplished by multiplying the resulting samples by $\sigma$.

```python
params = [dict(name='α', start=0.1, end=5, value=1, step=0.01),
          dict(name='σ', start=0.1, end=3, value=1.5, step=0.01)]
app = distribution_plot_app(x_min=0,
                            x_max=8,
                            scipy_dist=st.weibull_min,
                            params=params,
                            transform=lambda a, s: (a, 0, s),
                            x_axis_label='y',
                            title='Weibull')
bokeh.io.show(app, notebook_url=notebook_url)
```

### Beta distribution

* **Story.** Say you wait for two multistep Poisson processes to happen. The individual steps of each process happen at the same rate, but the first multistep process requires $\alpha$ steps and the second requires $\beta$ steps. The fraction of the total waiting time taken by the first process is Beta distributed.

* **Parameters.** There are two parameters, both strictly
positive: $\alpha$ and $\beta$, defined in the above story.

* **Support.**
The Beta distribution has support on the interval [0, 1].

* **Probability density function.**
\begin{align}
f(\theta;\alpha, \beta) = \frac{\theta^{\alpha-1}(1-\theta)^{\beta-1}}{B(\alpha,\beta)},
\end{align}
where
\begin{align}
B(\alpha,\beta) = \frac{\Gamma(\alpha)\,\Gamma(\beta)}{\Gamma(\alpha + \beta)}
\end{align}
is the Beta function.

* **Usage**

| Package        | Syntax           |
|: -------------: | :------------- |
|**NumPy** | `np.random.beta(alpha, beta)` |
|**SciPy** | `scipy.stats.beta(alpha, beta)` |
|**Stan** | `beta(alpha, beta)` |

* **Related distributions.**
    - The [Uniform distribution](#Uniform-distribution) on the interval [0, 1] is a special case of the Beta distribution with $\alpha=\beta=1$.
    - The Beta distribution is a special case of the [Dirichlet distribution](#Dirichlet-distribution) with $K=2$.

* **Notes.**
    - The story of the Beta distribution is difficult to parse. Most importantly for our purposes, the Beta distribution allows us to put probabilities on unknown probabilities. It is only defined on $0\le x\le 1$, and $x$ here can be interpreted as a probability, say of a Bernoulli trial.
    - The case where $\alpha = \beta = 0$ is not technically a probability distribution because the PDF cannot be normalized. Nonetheless, it can be used as an improper prior, and this prior is known a Haldane prior, names after biologist J. B. S. Haldane. The case where $\alpha = \beta = 1/2$ is sometimes called a Jeffreys prior.
    - The Beta distribution may also be parametrized in terms of the mean $\phi$ and concentration $\kappa$, which are related to $\alpha$ and $\beta$ as
    \begin{align}
    \phi &= \frac{\alpha}{\alpha + \beta},\\[1em]
    \kappa &= \alpha + \beta.\\
    \end{align}
    The mean is the mean of the distribution and $\kappa$ is a measure of how broad it is. To convert back to an $\alpha, \beta$ parametrization from a $\phi, \kappa$ parametrization,
    \begin{align}
    \alpha &= \phi\kappa,\\[1em]
    \kappa &= (1-\phi)\kappa.\\
    \end{align}

```python
params = [dict(name='α', start=0.01, end=10, value=1, step=0.01),
          dict(name='β', start=0.01, end=10, value=1, step=0.01)]
app = distribution_plot_app(x_min=0,
                            x_max=1,
                            scipy_dist=st.beta,
                            params=params,
                            x_axis_label='θ',
                            title='Beta')
bokeh.io.show(app, notebook_url=notebook_url)
```

## Discrete multivariate distributions

So far, we have looked a univariate distributions, but we will consider multivariate distributions in class, and you will encounter them in your research. First, we consider a discrete multivariate distribution, the Multinomial.

### Multinomial distribution

* **Story.** This is a generalization of the Binomial distribution. Instead of a Bernoulli trial consisting of two outcomes, each trial has $K$ outcomes. The probability of getting $n_1$ of outcome 1, $n_2$ of outcome 2, ..., and $y_K$ of outcome $K$ out of a total of $N$ trials is Multinomially distributed.

* **Example.** There are two alleles in a population, A and a. Each individual may have genotype AA, Aa, or aa. The probability distribution describing having $y_1$ AA individuals, $y_2$ Aa individuals, and $n_3$ aa individuals in a population of $N$ total individuals is Multinomially distributed.

* **Parameters.** $N$, the total number of trials, and $\boldsymbol{\theta} = \{\theta_1, \theta_2, \ldots \theta_k\}$, the probabilities of each outcome. Note that $\sum_i \theta_i = 1$ and there is a further restriction that $\sum_i y_i = N$.

* **Support.** The _K_-nomial distribution is supported on $\mathbb{N}^K$.

* **Usage**
The usage below assumes that `theta` is a length _K_ array.

| Package        | Syntax           |
|: -------------: | :------------- |
|**NumPy** | `np.random.multinomial(N, theta)` |
|**SciPy** | `scipy.stats.multinomial(N, theta)` |
|**Stan sampling** | `multinomial(theta)` |
|**Stan rng** | `multinomial_rng(theta, N)` |

* **Probability density function.**

\begin{align}
f(\mathbf{y};\mathbf{\theta}, N) = \frac{N!}{y_1!\,y_2!\cdots y_k!}\,\theta_1^{y_1}\,\theta_2^{y_2}\cdots \theta_K^{y_K}.
\end{align}

* **Related distributions.**
    - The Multinomial distribution generalizes the [Binomial distribution](#Binomial-distribution) to multiple dimensions.

* **Notes.**
    - For a sampling statement in Stan, the value of `N` is implied.

## Continuous Multivariate distributions

We will consider two continuous multivariate distributions here, the multivariate Gaussian and the Dirichlet. Generally plotting multivariate PDFs is difficult, but bivariate PDFs may be conveniently plotted as **contour plots**. In our investigation of the multivariate Gaussian distribution, I will also demonstrate how to make plots of bivariate PDFs.

### Multivariate Gaussian, a.k.a. Multivariate Normal, distribution

* **Story.** This is a generalization of the [univariate Gaussian](#Gaussian,-a.k.a.-Normal,-distribution).

* **Example.** Finch beaks are measured for beak depth and beak length. The resulting distribution of depths and length is Gaussian distributed. In this case, the Gaussian is bivariate, with $\mu = (\mu_\mathrm{d}, \mu_\mathrm{l})$ and
\begin{align}
\mathsf{\Sigma} = \begin{pmatrix}
\sigma_\mathrm{d}^2 & \sigma_\mathrm{dl} \\
\sigma_\mathrm{dl} & \sigma_\mathrm{l}^2
\end{pmatrix}.
\end{align}

* **Parameters.** There is one vector-valued parameter, $\boldsymbol{\mu}$, and a matrix-valued parameter, $\mathsf{\Sigma}$, referred to respectively as the mean and covariance matrix. The covariance matrix is symmetric and strictly positive definite.

* **Support.** The _K_-variate Gaussian distribution is supported on $\mathbb{R}^K$.

* **Probability density function.**

\begin{align}
f(\mathbf{y};\boldsymbol{\mu}, \mathsf{\Sigma}) = \frac{1}{\sqrt{(2\pi)^K \det \mathsf{\Sigma}}}\,\mathrm{exp}\left[-\frac{1}{2}(\mathbf{y} - \boldsymbol{\mu})^T \cdot \mathsf{\Sigma}^{-1}\cdot (\mathbf{y} - \boldsymbol{\mu})\right].
\end{align}

* **Usage**
The usage below assumes that `mu` is a length _K_ array, `Sigma` is a _K_×_K_ symmetric positive definite matrix, and `L` is a _K_×_K_ lower-triangular matrix with strictly positive values on teh diagonal that is a Cholesky factor.

| Package        | Syntax           |
|: -------------: | :------------- |
|**NumPy** | `np.random.multivariate_normal(mu, Sigma)` |
|**SciPy** | `scipy.stats.multivariate_normal(mu, Sigma)` |
|**Stan** | `multi_normal(mu, Sigma)` |
|**NumPy Cholesky**| `np.random.multivariate_normal(mu, np.dot(L, L.T))` |
|**SciPy Cholesky**| `scipy.stats.multivariate_normal(mu, np.dot(L, L.T))` |
|**Stan Cholesky**| `multi_normal_cholesky(mu, L)` |

* **Related distributions.**
    - The Multivariate Gaussian is a generalization of the [univariate Gaussian](#Gaussian,-a.k.a.-Normal,-distribution).

* **Notes.**
    - The covariance matrix may also be written as
\begin{align}
\\\phantom{blah}
\mathsf{\Sigma} = \mathsf{S}\cdot \mathsf{C} \cdot \mathsf{S},
\\\phantom{blah}
\end{align}
where
\begin{align}
\\\phantom{blah}
\mathsf{S} = \sqrt{\text{diag}(\mathsf{\Sigma})},
\\\phantom{blah}
\end{align}
and entry $i, j$ in the **correlation matrix** $\mathsf{C}$ is
\begin{align}
\\\phantom{blah}
C_{ij} = \sigma_{ij} / \sigma_i \sigma_j.
\\\phantom{blah}
\end{align}
    - Furthermore, because $\mathsf{\Sigma}$ is symmetric and strictly positive definite, it can be uniquely defined in terms of its [Cholesky decomposition](https://en.wikipedia.org/wiki/Cholesky_decomposition), $\mathsf{L}$, which satisfies $\mathsf{\Sigma} = \mathsf{L}\cdot\mathsf{L}^T$. In practice, you will almost always use the Cholesky representation of the Multivariate Gaussian distribution in Stan.

### Lewandowski-Kurowicka-Joe (LKJ) distribution

* **Story.** Probability distribution for positive definite correlation matrices, or equivalanently for their Cholesky factors (which is what we use in practice).

* **Parameter.** There is one positive scalar parameter, $\eta$, which tunes the strength of the correlations. If $\eta = 1$, the density is uniform over all correlation matrix. If $\eta > 1$, matrices with a stronger diagonal (and therefore smaller correlations) are favored. If $\eta < 1$, the diagonal is weak and correlations are favored.

* **Support.** The LKJ distribution is supported over the set of $K\times K$ Cholesky factors of real symmetric positive definite matrices.

* **Probability density function.**
We cannot write the PDF in closed form.

* **Usage**

| Package        | Syntax           |
|: -------------: | :------------- |
|**NumPy** | not available |
|**SciPy** | not available |
|**Stan** | lkj_corr_cholesky(eta) |

* **Notes.**
    - The most common use case is as a prior for a covariance matrix. Note that LKJ distribution gives Cholesky factors for *correlation* matrices, not covariance matrices. To get the covariance Cholesky factor from the correlation Cholesky factor, you need to multiply the correlation Cholesky factor by a diagonal constructed from the variances of the individual variates. Here is an example for a multivariate Gaussian in Stan.

    ```
    parameters {
        // Vector of means
        vector[K] mu;

        // Cholesky factor for the correlation matrix
        cholesky_factor_corr[K] L_Omega;

        // Sqrt of variances for each variate
        vector<lower=0>[K] L_std;
    }
    model {
        // Cholesky factor for covariance matrix
        matrix[K, K] L_Sigma = diag_pre_multiply(L_std, L_Omega);

        // Prior on Cholesky decomposition of correlation matrix
        L_Omega ~ lkj_corr_cholesky(1);

        // Prior on standard deviations for each variate
        L_std ~ normal(0, 2.5);

        // Likelihood
        y ~ multi_normal_cholesky(mu, L_Sigma);
    }
    ```

### Dirichlet distribution

* **Story.** The Dirichlet distribution is a generalization of the [Beta distribution](#Beta-distribution). It is a probability distribution describing probabilities of outcomes. Instead of describing probability of one of two outcomes of a Bernoulli trial, like the Beta distribution does, it describes probability of $K-1$ of $K$ outcomes. The Beta distribution is the special case of $K = 2$.

* **Parameters.** The parameters are $\alpha_1, \alpha_2, \ldots \alpha_K$, all strictly positive, defined analogously to $\alpha$ and $\beta$ of the Beta distribution.

* **Support.** The Dirichlet distribution has support on the interval [0, 1] such that $\sum_{i=1}^K \theta_i = 1$.

* **Probability density function.**
\begin{align}
f(\boldsymbol{\theta};\boldsymbol{\alpha}) = \frac{1}{B(\boldsymbol{\alpha})}\,\prod_{i=1}^K \theta_i^{\alpha_i-1},
\end{align}
where
\begin{align}
B(\boldsymbol{\alpha}) = \frac{\prod_{i=1}^K\Gamma(\alpha_i)}{\Gamma\left(\sum_{i=1}^K \alpha_i\right)}
\end{align}
is the multivariate Beta function.

* **Usage**

| Package        | Syntax           |
|: -------------: | :------------- |
|**NumPy** | `np.random.dirichlet(alpha)` |
|**SciPy** | `scipy.stats.dirichlet(alpha)` |
|**Stan** | `dirichlet(alpha)` |

* **Related distributions.**
    - The special case where $K = 2$ is a [Beta distribution](#Beta-distribution) with parameters $\alpha = \alpha_1$ and $\beta = \alpha_2$.
    - If we have $K$ random variables that are Gamma distributed, $X_i \sim \text{Gamma}(\alpha_i, \beta)$, then $(X_1/V, X_2/,V \ldots) \sim \text{Dirichlet}(\alpha)$ with $\alpha = (\alpha_1, \alpha_2, \ldots)$ and $V = \sum_{i=1}^K X_i$. Note that the value of $\beta$ does not enter into the Dirichlet distribution.

* **Notes**
- In some cases, we may wish to specify the distribution of an *ordered* Dirichlet distributed vector $\theta$. That is $\theta \sim \text{Dirichlet}(\alpha_1, \alpha_2, \ldots, \alpha_K)$ with $\theta_i < \theta_{i+1}$ for all $\theta_i$. Because of the relationship of the Dirchlet distribution to a set of Gamma distributed random variable, we may specify this in Stan as follows.

```
data {
  int<lower=1> K;
}

parameters {
  vector<lower=0>[K] alpha;
  positive_ordered[K] lambda;
}

transformed parameters {
  simplex[K] theta = lambda / sum(lambda);
}

model {
  target += gamma_lpdf(lambda | alpha, 1);
}
```

## Conclusions

This document serves as a catalog of probability distributions that will be useful for you in your statistical modeling. As we will see, the mathematical expression of the PDF is not often needed. What is most important in your modeling is that you know the story of the distribution.

## Computing environment

```python
%load_ext watermark
```

```python
%watermark -v -p numpy,scipy,bokeh,jupyterlab
```