# A Rapid Introuction to Bayesian Modeling

This brief presentation will use a published case study to examine why Bayesian modeling is useful and how a Bayesian perspective informs data analysis.
Statistics is hard!
If this talk is successful in getting you interested in Bayesian analysis, you should learn more theory (see last slide) before diving into software.

* [James Doss-Gollin](https://jamesdossgollin.me)
* 2020-07-10

```python
# import required packages
import arviz as az
import matplotlib.pyplot as plt # plotting
import numpy as np # numeric data
import pandas as pd # standard for reading in tabular data
from scipy.stats import genextreme as gev
import xarray as xr # for gridded data -- I use whenever possible, it's awesome]
```

```python
# import pystan and a function to cache stan models to avoid recompilation
import pystan
import pickle
from hashlib import md5

def StanModel_cache(filename: str, model_name: str="", **kwargs):
    """Use just as you would `stan`"""
    with open(filename) as file:
        model_code = file.read()
    code_hash = md5(model_code.encode('ascii')).hexdigest()
    cache_fn = f'cached-{model_name}-{code_hash}.pkl'
    try:
        sm = pickle.load(open(cache_fn, 'rb'))
    except:
        sm = pystan.StanModel(model_code=model_code)
        with open(cache_fn, 'wb') as f:
            pickle.dump(sm, f)
    else:
        print("Using cached StanModel")
    return sm
```

```python
# read in data and create an xarray dataset with the areas and floods
areas = pd.read_table("areas.txt", names=["area"]).to_xarray().rename({"index": "site"})["area"]
ann_max_flood = pd.read_table("annmax.txt", names=[f"site_{i}" for i in range(20)], delim_whitespace=True)
ann_max_flood = xr.DataArray(
    ann_max_flood.T,
    coords = {
        "site": np.arange(ann_max_flood.shape[1]),
        "year": np.arange(ann_max_flood.shape[0]),
    },
    dims=["site", "year"]
)
flood_dat = xr.Dataset({"ann_max_flood": ann_max_flood, "area": areas})
n_obs = (1 - flood_dat.isnull()["ann_max_flood"]).sum(dim="year")
flood_dat["site"] = np.arange(len(flood_dat["site"])) + 1
flood_dat
```

<svg style="position: absolute; width: 0; height: 0; overflow: hidden">
<defs>
<symbol id="icon-database" viewBox="0 0 32 32">
<title>Show/Hide data repr</title>
<path d="M16 0c-8.837 0-16 2.239-16 5v4c0 2.761 7.163 5 16 5s16-2.239 16-5v-4c0-2.761-7.163-5-16-5z"></path>
<path d="M16 17c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z"></path>
<path d="M16 26c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z"></path>
</symbol>
<symbol id="icon-file-text2" viewBox="0 0 32 32">
<title>Show/Hide attributes</title>
<path d="M28.681 7.159c-0.694-0.947-1.662-2.053-2.724-3.116s-2.169-2.030-3.116-2.724c-1.612-1.182-2.393-1.319-2.841-1.319h-15.5c-1.378 0-2.5 1.121-2.5 2.5v27c0 1.378 1.122 2.5 2.5 2.5h23c1.378 0 2.5-1.122 2.5-2.5v-19.5c0-0.448-0.137-1.23-1.319-2.841zM24.543 5.457c0.959 0.959 1.712 1.825 2.268 2.543h-4.811v-4.811c0.718 0.556 1.584 1.309 2.543 2.268zM28 29.5c0 0.271-0.229 0.5-0.5 0.5h-23c-0.271 0-0.5-0.229-0.5-0.5v-27c0-0.271 0.229-0.5 0.5-0.5 0 0 15.499-0 15.5 0v7c0 0.552 0.448 1 1 1h7v19.5z"></path>
<path d="M23 26h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
<path d="M23 22h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
<path d="M23 18h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
</symbol>
</defs>
</svg>

xarray.Dataset

<input id='section-2a5f2b9e-7aee-4f3c-95eb-bdab20a1f3d0' class='xr-section-summary-in' type='checkbox' disabled >Dimensions:

site: 20

year: 75

<input id='section-08b76a58-48d5-48ee-a200-361365965c03' class='xr-section-summary-in' type='checkbox'  checked>Coordinates: (2)

site

(site)

int64

1 2 3 4 5 6 7 ... 15 16 17 18 19 20
<input id='attrs-1008f268-6179-4b2e-9e91-3432bbffdb3f' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-1d498dc5-4bb9-4b51-8238-e03ec3bc0ebc' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
       19, 20])

year

(year)

int64

0 1 2 3 4 5 6 ... 69 70 71 72 73 74
<input id='attrs-f84b78fe-9765-4a3b-908d-682f54bef405' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-9aa24ab6-0b8f-46d2-b607-6d48b1e05c2e' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
       36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,
       54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71,
       72, 73, 74])

<input id='section-a05781f5-3ca5-476d-a82c-2b82565508a8' class='xr-section-summary-in' type='checkbox'  checked>Data variables: (2)

ann_max_flood

(site, year)

float64

nan nan nan ... 86.12 92.56 342.4
<input id='attrs-b542c208-ecc7-456b-90f8-5d77b6b5e65a' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-15074d64-8a5c-4299-8685-3236082d5601' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([[         nan,          nan,          nan, ...,          nan,
                 nan,          nan],
       [2.900000e+00, 1.520000e+01, 3.070000e+01, ...,          nan,
                 nan,          nan],
       [         nan,          nan,          nan, ..., 1.559950e+03,
        1.871600e+03,          nan],
       ...,
       [1.939838e+03, 3.854242e+03, 1.987112e+03, ...,          nan,
                 nan,          nan],
       [         nan, 2.320000e+01, 9.500000e+00, ...,          nan,
                 nan,          nan],
       [1.726226e+02, 2.364919e+02, 1.388770e+02, ..., 8.611630e+01,
        9.256270e+01, 3.424200e+02]])

area

(site)

float64

1.61e+04 162.0 ... 77.4 1.16e+03
<input id='attrs-90f01034-5494-47ff-aca2-74695959941e' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-66d38957-e145-4855-a34a-956c6710d09d' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([1.61e+04, 1.62e+02, 2.42e+04, 7.64e+04, 4.05e+04, 8.21e+04,
       5.27e+03, 1.02e+04, 3.06e+03, 6.23e+03, 6.12e+04, 8.72e+03,
       5.51e+04, 9.77e+03, 1.01e+04, 1.28e+04, 2.42e+04, 6.12e+04,
       7.74e+01, 1.16e+03])

<input id='section-d286f487-fe1c-41a9-a83a-494080b9ccff' class='xr-section-summary-in' type='checkbox' disabled >Attributes: (0)

## Motivation: Why Bayesian

Bayesian Decision Theory: given an action space $\mathcal{A}$, state space $\mathcal{S}$, reward function (equivalently objective, cost, loss, etc) $R$, the expected reward for taking action $a \in \mathcal{A}$ is
$$\mathbb{E}[R | a] = \int_\mathcal{S} R(a, s)p(s) ds$$
allowing decision maker to select the best action
$$a^* := \arg \max_{a \in \mathcal{A}} \mathbb{E}[R | a]$$
A key point is that:
$$\mathbb{E}_s[R | a] \neq R(\mathbb{E}[s], a)$$
so we **need to fully represent our uncertainty**

## Example: Floods in Brazil

> Lima, C. H. R., Lall, U., Troy, T., & Devineni, N. (2016). A hierarchical Bayesian GEV model for improving local and regional flood quantile estimates. Journal of Hydrology, 541, 816–823. https://doi.org/10.1016/j.jhydrol.2016.07.042

We want to do flood frequency analysis in a single river basin

Quick look at data:

1. Data is sparse -- lots of missing observations
2. Large floods register at most sites

```python
flood_dat["ann_max_flood"].plot()
plt.show()
```

## Our goal

1. pool information across time and space to get accurate return period information at each site
2. transparently and credibly represent the uncertainty in these parameters (eg, site 0 should have very large uncertainty!)

How would you model this?

We're going to use a Bayesian hierarchical model.

## What is Bayesian statistics?

Bayesian statistics is all about **conditional probability!**

We typically want to conduct inference on the parameters $\theta$ of some model.
To do this we first need to write down a **joint probability model.**

Key insight:
$$ p(\theta, y) = p(\theta) p(y | \theta). $$
The terms on the right hand side of the equation are referred to as the prior and the likelihood functions, respectively.

## Bayes rule

Bayes rule relates joint and conditional probabilities:
$$
p(\theta | y) = \frac{p(\theta, y)}{p(y)}
$$
adding in the term from the previous slide
$$
p(\theta | y) = \frac{p(y | \theta) p(\theta)}{p(y)}
$$
for many problems, $p(\theta | y)$ doesn't depend on $p(y)$, so we can write
$$
p(\theta | y) \propto p(y | \theta) p(\theta)
$$
We call $p(\theta | y)$ the **posterior** distribution.

## Hierarchical models

> Many statistical applications involve multiple parameters that can be regarded as related or connected in some way by the structure of the problem, implying that a joint probability model for these parameters should reflect their dependence.
_(BDA3 p. 101)_

If we have some "hyperparameters" $\phi$ on which $\theta$ depend, then we can go a step farther:
$$
p(\phi, \theta) = p(\phi) p(\theta | \phi)
$$
And so the posterior distribution of is
$$
p(\phi, \theta | y) = p(\phi) p(\theta | \phi) p(y | \theta)
$$

## Lima _et. al_ model

Assume that local floods $Q$ in year $i$ at site $j$ follow a stationary GEV distribution
$$ Q_{i,j} \sim g(\mu_j, \sigma_j, \xi_j) $$
The data model assumes that $\mu_j$ and $\sigma_j$ depend on on catchment area $A_j$:
$$
\begin{align}
\log \mu_j &\sim \mathcal{N} \left( \alpha_1 + \alpha_2 \log A_j, \tau_\mu^2 \right) \\
\log \sigma_j &\sim \mathcal{N} \left( \beta_1 + \beta_2 \log A_j, \tau_\sigma^2 \right) \\
\xi_j &\sim \mathcal{N} \left( \overline{\xi}, \tau_\xi^2 \right)
\end{align}
$$
This is called a "hierarchical" or "partial pooling" model (*why?*)

Here the $Q_{i,j}$ the data ($y$ in earlier notation), the $\mu_j, \sigma_j, \xi_j$ are the parameters ($\theta$) and the hyperparameters $\alpha_1, \alpha_2, \beta_1, \beta_2, \overline{xi}, \tau_\mu, \tau_\sigma, \tau_\xi$ ($\theta$).

## Key assumptions

* Floods follow stationary GEV
* Floods are IID in space and time (no clustering!)
* Data missing at random

## Priors on hyperparameters

Lima _et al_ use uniform priors.
This places lots of weight on extremely implausible regions of the probability space.

We should use _weakly informative_ priors -- these should "capture what we know about rivers, but not the data we've collected about _this_ river".
A good way to check priors is to use your prior model to simulate fake data.
If it doesn't look completely absurd, you're probably OK.

The paper claims that they use uninformative priors.
However, that's not born out in their code -- instead we'll use weakly informative priors.

## Missing data

1. Missing data _don't pose a mechanistic challenge for our problem._ For places where we have very little information, we will rely more heavily on our model: $\log \mu_j \approx \alpha_1 + \alpha_2 \log A_j$.
2. That said, an analysis _should model the data collection process_. See BDA3 Chapter 8.

We'll assume that data are "missing at random."
This means that, the probability of data being observed (or missing) does not depend on the $y$.
In our case this clearly depends on time, but we have a stationary model -- we're going to pretend to believe in it completely!

## Stan

Stan is

* a modeling language for specifying Bayesian models (like the one above)
* a program that conducts inference on these models using Markov Chain Monte Carlo

Why [MCMC](https://chi-feng.github.io/mcmc-demo/app.html#NaiveNUTS,banana)?

* we can't compute our distributions analytically!
* we can't even compute them numerically
* however, thanks to cool math we _can_ sample from them!

We will approximate $p(\theta, \phi | y)$ by drawing many "Monte Carlo" samples of $\{\theta, \phi \}$ (using stan!).

## Compilation

Stan is a compiled language; it's slow to compile but fast to run.
The compiler will check the code for syntax errors, although it doesn't catch everything.
Then we can re-use the compiled model.

```python
stan_model = StanModel_cache("lima.stan", model_name="lima-2016")
```

    Using cached StanModel

```python
print(stan_model.model_code)
```

    functions{
        real gev_lpdf(real y, real mu, real sigma, real xi) {
            real t;
            real lp;
            t = xi==0 ? exp((mu - y) / sigma) : pow(1 + xi * ((y - mu ) / sigma), -1 / xi);
            lp = -log(sigma) + (xi + 1) * log(t) - t;
            return lp;
        }
        real gev_rng(real mu, real sigma, real xi){
            real p;
            real z;
            real epsilon;
            p = uniform_rng(0, 1);
            epsilon = 0.0001;
            z = fabs(xi) < epsilon ? -log(-log(p)): expm1(-xi * log(-log(p))) / xi;
            return mu + sigma * z;
        }
    }

    data {
        int<lower=0> J; // number of sites
        int<lower=0> N; // number of observations
        real<lower=0> log_area[J]; // area of each watershed
        real flood[N]; // the annual-maximum floods
        int site[N]; // the index of each data point
        // pass priors as data to avoid recompiling
        real alpha_1_std;
        real alpha_2_std;
        real beta_1_std;
        real beta_2_std;
        real xi_bar_std;
        real tau_mu_std;
        real tau_sigma_std;
        real tau_xi_std;
    }

    parameters {
        // scale, loc, shape: one per site
        real<lower=0> mu[J];
        real<lower=0> sigma[J];
        real xi[J];
        // hyperparameters
        real alpha_1;
        real alpha_2;
        real beta_1;
        real beta_2;
        real xi_bar;
        real<lower=0> tau_mu;
        real<lower=0> tau_sigma;
        real<lower=0> tau_xi;
    }

    model {
        // data model
        for (n in 1:N){
            flood[n] ~ gev(mu[site[n]], sigma[site[n]], xi[site[n]]);
        }
        // hierarchical model = note that stan uses stdev not variance for normal dist
        for (j in 1:J){
            mu[j] ~ lognormal(alpha_1 + alpha_2 * log_area[j], tau_mu);
            sigma[j] ~ lognormal(beta_1 + beta_2 * log_area[j], tau_sigma);
            xi[j] ~ normal(xi_bar, tau_xi);
        }
        // hyperpriors
        alpha_1 ~ normal(0, alpha_1_std);
        alpha_2 ~ normal(0, alpha_2_std);
        beta_1 ~ normal(0, beta_1_std);
        beta_2 ~ normal(0, beta_2_std);
        xi_bar ~ normal(0, xi_bar_std);
        tau_mu ~ normal(0, tau_mu_std);
        tau_sigma ~ normal(0, tau_sigma_std);
        tau_xi ~ normal(0, tau_xi_std);
    }

    generated quantities {
        real log_lik[N];
        real y_hat[N];
        for (n in 1:N) {
            log_lik[n] = gev_lpdf(flood[n] | mu[site[n]], sigma[site[n]], xi[site[n]]);
            y_hat[n] = gev_rng(mu[site[n]], sigma[site[n]], xi[site[n]]);
        }
    }

## Data

We need to pass the data to Stan in a format it understands (see the `data` block)

```python
non_missing_data = flood_dat.to_dataframe().reset_index().dropna()[["site", "ann_max_flood"]]
log_area = np.log(flood_dat["area"])
log_area = log_area - np.min(log_area)
stan_data = {
    "J": len(flood_dat["site"]),
    "N": len(non_missing_data),
    "log_area": np.array(log_area),
    "flood": np.array(non_missing_data["ann_max_flood"]),
    "site": np.array(non_missing_data["site"]),
    "alpha_1_std": 0.5,
    "alpha_2_std": 0.5,
    "beta_1_std": 0.5,
    "beta_2_std": 0.5,
    "xi_bar_std": 0.5,
    "tau_mu_std": 0.5,
    "tau_sigma_std": 0.5,
    "tau_xi_std": 0.5,
}
```

## Fitting the model

This model takes about 45 seconds to run on my computer:

```python
def get_inits():
    return {
        'mu': np.repeat(100, stan_data["J"]),
        'sigma': np.repeat(100, stan_data["J"]),
        'xi': np.repeat(0.1, stan_data["J"]),
        "alpha_1": stan_data["alpha_1_std"],
        "alpha_2": stan_data["alpha_2_std"],
        "beta_1": stan_data["beta_1_std"],
        "beta_2": stan_data["beta_2_std"],
        "xi_bar": stan_data["xi_bar_std"],
        "tau_mu": stan_data["tau_mu_std"],
        "tau_sigma": stan_data["tau_sigma_std"],
        "tau_xi": stan_data["tau_xi_std"],
    }
```

```python
fname = "posterior.nc"
try:
    az_data = az.from_netcdf(fname)
except FileNotFoundError:
    fit = stan_model.sampling(data=stan_data, iter=5000, chains=4, warmup=1000, init=get_inits)
    az_data = az.from_pystan(
        posterior=fit,
        posterior_predictive="y_hat",
        observed_data=["flood"],
        log_likelihood={"flood": "log_lik"},
        coords={"site": flood_dat["site"]},
        dims={
            "mu": ["site"],
            "sigma": ["site"],
            "xi": ["site"],
            "y_hat": ["obs"],
            "flood": ["obs"],
        },
    )

    az_data.to_netcdf(fname)
```

```python
az.summary(az_data)
```

      mean
      sd
      hdi_3%
      hdi_97%
      mcse_mean
      mcse_sd
      ess_mean
      ess_sd
      ess_bulk
      ess_tail
      r_hat

      mu[0]
      595.023
      140.351
      326.635
      856.530
      1.030
      0.728
      18577.0
      18577.0
      17843.0
      10777.0
      1.0

      mu[1]
      12.906
      1.756
      9.678
      16.263
      0.014
      0.010
      15140.0
      15140.0
      15030.0
      11506.0
      1.0

      mu[2]
      1191.436
      126.803
      958.371
      1434.289
      0.896
      0.633
      20034.0
      20034.0
      19778.0
      11747.0
      1.0

      mu[3]
      3575.507
      462.109
      2699.826
      4444.960
      3.621
      2.560
      16288.0
      16288.0
      17165.0
      9605.0
      1.0

      mu[4]
      1787.956
      148.938
      1523.037
      2077.553
      1.033
      0.731
      20772.0
      20772.0
      20661.0
      12134.0
      1.0

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      beta_2
      0.807
      0.048
      0.719
      0.897
      0.000
      0.000
      9673.0
      9459.0
      10094.0
      9198.0
      1.0

      xi_bar
      0.130
      0.050
      0.036
      0.225
      0.000
      0.000
      10689.0
      10689.0
      10698.0
      11754.0
      1.0

      tau_mu
      0.343
      0.077
      0.211
      0.485
      0.001
      0.001
      10129.0
      9715.0
      10762.0
      10749.0
      1.0

      tau_sigma
      0.344
      0.082
      0.202
      0.494
      0.001
      0.001
      8612.0
      8604.0
      8681.0
      10874.0
      1.0

      tau_xi
      0.143
      0.054
      0.049
      0.250
      0.001
      0.001
      2632.0
      2632.0
      2364.0
      2669.0
      1.0

68 rows × 11 columns

## Visualization

We can use the `arviz` package to visualize our model fit.
Visualization is a critical component of a Robust Bayesian Workflow

* Gabry, J., Simpson, D., Vehtari, A., Betancourt, M., & Gelman, A. (2019). Visualization in Bayesian workflow. Journal of the Royal Statistical Society: Series A (Statistics in Society), 182(2), 389–402. https://doi.org/10.1111/rssa.12378
* [M Betancourt: Towards a Principled Bayesian Workflow](https://github.com/betanalpha/jupyter_case_studies/blob/master/principled_bayesian_workflow/principled_bayesian_workflow.ipynb)

```python
az.plot_forest(
    az_data, combined=True, hdi_prob=0.94, var_names=["mu"],
    kind="forestplot", linewidth=0.5, ridgeplot_overlap=0.2, figsize=(8, 6)
)
plt.show()
```

```python
az.plot_forest(
    az_data, combined=True, hdi_prob=0.94,
    var_names=["alpha_1", "alpha_2", "beta_1", "beta_2", "xi_bar", "tau_mu", "tau_sigma", "tau_xi"],
    kind="forestplot", linewidth=0.5, ridgeplot_overlap=0.2, figsize=(8, 6)
)
plt.show()
```

```python
az.plot_pair(
    az_data, var_names=["alpha_1", "alpha_2", "beta_1", "beta_2", "xi_bar"],
    divergences=True, figsize=(8, 6)
)
plt.show()
```

```python
az.plot_pair(
    az_data, var_names=["tau_mu", "tau_sigma", "tau_xi"],
    divergences=True, figsize=(8, 6)
)
plt.show()
```

More sophisticated diagnostics:

> [BayesPlot R package](https://mc-stan.org/bayesplot/articles/visual-mcmc-diagnostics.html): Informally, the energy diagnostic for HMC quantifies the heaviness of the tails of the posterior distribution. The two histograms ideally look the same [(Betancourt, 2017)](https://arxiv.org/abs/1701.02434)

```python
az.plot_energy(az_data)
plt.show()
```

## Posterior predictive checks

Premise:

1. If our model is good, then simulations from our model should look like our data in dimensions that we care about
2. An iterative cycle of build -> check helps us improve our model over time

Since this is a quick test case, we're just going to run one posterior predictive check.
Specifically, let's compare the CDF distribution of simulated floods (from our GEV model, using the calibrated parameters and simulating at the same vector of sites as our input data) with observed.

```python
az.plot_ppc(az_data, kind="cumulative", num_pp_samples=1000, data_pairs={"flood": "y_hat"})
plt.xlim([-100, 7500])
plt.show()
```

## More Checks

In a more rigorous example, we would run additional simulation-based checks that make use of "test statistics" that map the estimated parameters $p(\theta | y)$ to $\mathbb{R}$

Some of these would require us to add to the `generated quantities` block

* calculate the CDF at each site and compare to observations
* with what frequency are the estimated 5, 10, 100 year floods at each site exceeded?

## Summary

Bayesian statistics means our goal is a joint probability distribution

1. Write down a model using conditional distributions (in Stan)
1. Sample from the posterior joint distribution $p(\theta | y)$ using MCMC
1. Check our model using quantitative, numerical summaries

Advantages

* combine data from different sources
* transparently and honestly quantify uncertainty
* make assumptions clear
* force yourself to write down and think about the full model

## Learn More

* [Stan Docs](https://mc-stan.org/)
* [PyStan Docs](http://pystan.readthedocs.io/)
* Gelman [BDA3](http://stat.columbia.edu/~gelman/book/BDA3.pdf) is the canonical textbook and used in graduate classes
* [Michael Betancourt](https://betanalpha.github.io/writing/) case studies
* The Stan developers publish a [guide to priors](https://github.com/stan-dev/stan/wiki/Prior-Choice-Recommendations) that you should read if you build a Bayesian model
* A good text on (complicated!) Bayesian models for space-time data. Cressie, N. A. C., & Wikle, C. K. (2011). Statistics for spatio-temporal data.