# Bayesian Statistics for Physicists

Note: This notebook was presented at the EMMI workshop "Uncertainty Quantification (UQ) at the Extremes (ISNET-6)" on 9-October-2018.
    It has been replaced for further development by the multipart BSFP_pn_xxx.ipynb notebooks.

## Overview

A brief, hands-on introduction to the basics of Bayesian statistics in a manner adapted to the general intuition and experience of physicists.  We use a Jupyter notebook with Python (scipy, numpy, mathplotlib) to allow for active visualization of examples, hands-on follow-ups, and readily extended content. You can find the notebook and associated files at
https://github.com/furnstahl/Bayes_for_physicists.

This is not an exhaustive guide to Bayesian statistics, but a selected sampling of topics that come up regularly in physics applications, with links to more information.
Most of the examples were adapted from code found on the web.
Please contribute suggestions, comments, links, code, ...

Last revised: 13-Oct-2018 by Dick Furnstahl [furnstahl.1@osu.edu].

## Contents

Overview

Python set up

Bayesian basics
        [Rules]
        [Networks]
        [Model discrepancy]

Choosing priors
        [Maximum entropy]
        [Conjugate priors]

Bayesian updating examples

Sampling
        [Multivariate gaussian]
        [MCMC]

Model selection: Bayes ratio and evidence

Gaussian processes

Appendices:
         [References]
         [Vocabulary]
         [Notation]

## Python/Jupyter set up

We recommend installing the standard Anaconda Python3 package (from https://www.anaconda.com/download), which is available for Windows, Mac OS X, and Linux.  Anaconda will provide scipy, numpy, matplotlib, and jupyter notebooks (and more!).

You can start this notebook from the Anaconda Navigator or from the command line (go to the directory with this notebook and type: jupyter notebook).

It is convenient to use the Jupyter notebook extensions "Code Folding" and "Collapsible Headings", which can be turned on using the jupyter_nbextensions_configurator extension after installing via

conda install -c conda-forge jupyter_contrib_nbextensions

 (see https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/ for a complete list and full instructions).

Other packages you will need to install for this notebook:

pymc3 [conda install -c conda-forge pymc3]

emcee [conda install -c astropy emcee]

corner [conda install -c astropy corner]

```python
# set up for plots in this notebook using matplotlib (there are other plotting choices)
%matplotlib inline
```

```python
import numpy as np

import scipy.stats as stats
from scipy.stats import norm, uniform

import matplotlib.pyplot as plt
#plt.style.use('seaborn') # pretty matplotlib plots

import corner
import pymc3 as pm

```

    /Users/furnstah/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      from ._conv import register_converters as _register_converters

```python
# make font adjustments
#plt.rcParams['font.size'] = 12
#plt.rcParams['legend.fontsize'] = 'medium'
#plt.rcParams['figure.titlesize'] = 'medium'
plt.rcdefaults()  # revert to defaults for now
```

```python
%%html
<!-- Use html cell magic to add css styling -->
<style>
  em {
      color: red;
  }
  dd {
      margin-left: 15px;
  }
  .red{color: red}
  .blue{color: blue}
</style>
```

```python
#%%javascript
#IPython.OutputArea.auto_scroll_threshold = 9999;
```

## Bayesian basics

### Why should physicists use Bayesian statistics?

cf. Why isn't every physicist a Bayesian? from 1995.

Includes conventional physics statistics (e.g., for parameter estimation)

Calculate what you really want, e.g., probability for some parameter vs. frequentist confidence interval

Assumptions are made explicit (in the form of priors)

Allows us to account for "nuisance parameters"

Well suited for theory errors, which are generally systematic

Clear prescription for combining different types of errors

Model selection: compare different theoretical models (or combine!)

Model checking: we can test if our UQ model works and study sensitivities

Statistics as diagnostic and discovery tools for physics

 **[add your own favorite reasons]**

### Everything is a pdf (probability density function)

Physicists are used to multidimensional normalized pdfs as wave functions squared, e.g. probability density for particle 1 at $x_1$ and particle 2 at $x_2$:

 $$
 |\Psi(x_1, x_2)|^2 \Longrightarrow p(x_1,x_2) \equiv p(\textbf{x})
   \quad \mbox{with}\quad \textbf{x}
 \equiv \{x_1,x_2\}
 $$

(Other notation for generic pdfs: $p(\textbf{x}) = P(\textbf{x}) = \textrm{pr}(\textbf{x}) = \textrm{prob}(\textbf{x}) = \ldots$ )

$p(x_1,x_2)$ is the joint probability density of $x_1$ and $x_2$.

What is the probability to find particle 1 at $x_1$ and particle 2 anywhere?  $\int\! |\Psi(x_1,x_2)|^2 dx_2$

The marginal probability density of $x_1$ is:
$\color{blue}{p(x_1) = \int\! p(x_1,x_2)\,dx_2}$.

"Marginalizing" = "integrating out" (eliminates "nuisance parameters" from posterior).

Just as with "Lagrangian", we will not always be careful about saying probability vs. probability density.

In Bayesian statistics there are pdfs (or pmfs if discrete) for data, experimental and theoretical uncertainties, fit parameters, hyperparameters (what?), events (Will it rain tomorrow?), etc.  Even if definite $x_0$, we can use $p(x) = \delta(x-x_0)$.

### Visualization of pdfs

#### Matplotlib plotting definitions

```python
def dist_stuff(dist):
    """
    Find the median, mean, and 68%/95% credible intervals
     for the given 1-d distribution (from stats).
    """
    median = [dist.median(), dist.pdf(dist.median())]
    mean = [dist.mean(), dist.pdf(dist.mean())]
    cred68 = dist.interval(0.68)
    cred95 = dist.interval(0.95)
    return median, mean, cred68, cred95

def dist_mode(dist, x):
    """
    Find the mode (maximum) of the 1-d distribution.
    """
    x_max_index = dist.pdf(x).argmax()
    mode = [x[x_max_index], dist.pdf(x[x_max_index])]
    return mode

def dist_plot(dist_label, x_dist, dist, plot_num):
    """
    Plot the distribution, indicating median, mean, mode
    and 68%/95% probability intervals.
    """
    colors = ('blue', 'blue', 'blue')
    median, mean, cred68, cred95 = dist_stuff(dist)
    mode = dist_mode(dist, x_dist)

    plt.subplot(1,3,plot_num)
    plt.plot(x_dist,dist.pdf(x_dist), label=dist_label, color=colors[plot_num-1])
    text_x = 0.2*(x_dist[-1]-x_dist[0])
    text_x_mid = (x_dist[-1]+x_dist[0])/2
    text_y = mode[1]*1.15
    plt.annotate('median', xy=median, xytext=(text_x_mid+text_x, text_y),
            arrowprops=dict(facecolor='black', shrink=0.05))
    plt.annotate('mode', xy=mode, xytext=(text_x_mid-text_x, text_y),
            arrowprops=dict(facecolor='red', shrink=0.05))
    plt.annotate('mean', xy=mean, xytext=(text_x_mid, text_y),
            arrowprops=dict(facecolor='blue', shrink=0.05))
    plt.xlabel('x')
    plt.ylabel('p(x)')
    plt.fill_between(x_dist, 0, dist.pdf(x_dist),
                 where=((x_dist > cred68[0]) & (x_dist < cred68[1])),
                 facecolor='blue', alpha=0.2)
    plt.fill_between(x_dist, 0, dist.pdf(x_dist),
                 where=((x_dist > cred95[0]) & (x_dist < cred95[1])),
                 facecolor='blue', alpha=0.1)
    plt.legend();

```

#### Some standard pdfs: normal and beta distributions

```python
%matplotlib inline

# Make some standard plots
plt.figure(figsize=(15,5))

# Standard normal distribution
x_norm = np.linspace(-4, 4, 500)
mu = 0     # mean
sigma = 1.0  # standard deviation
norm_dist = stats.norm(mu, sigma) # the normal distribution
norm_label='normal pdf' + '\n' + r'$\mu=${:1.1f},'.format(mu) \
             + '\n' + r'$\sigma=${:1.1f}'.format(sigma)
dist_plot(norm_label, x_norm, norm_dist, 1)

# beta distribution
x_beta = np.linspace(-0.1, 1.1, 500)
a1 = .5
b1 = 10
beta_dist = stats.beta(a1, b1)
beta1_label='beta pdf' + '\n' + r'$a=${:1.1f}'.format(a1) \
              + ',\n$b=${:1.1f}'.format(b1)
dist_plot(beta1_label, x_beta, beta_dist, 2)

# another beta distribution
#x_beta = np.linspace(-0.1, 1.1, 500)
a2 = 10
b2 = 10
beta2_dist = stats.beta(a2, b2)
beta2_label='beta pdf' + '\n' + r'$a=${:1.1f}'.format(a2) \
              + ',\n$b=${:1.1f}'.format(b2)
dist_plot(beta2_label, x_beta, beta2_dist, 3)

```

The 68%/95% probability regions are shown in dark/light shading.  When applied to posteriors, these are known as credible intervals or DoBs (degree of belief intervals) or Bayesian confidence intervals. The horizontal extent on the $x$-axis translates into the vertical extent of the error bar or error band for $x$.

#### More standard pdfs:  Student t

```python
%matplotlib inline

# Make some plots of the Student t distribution
plt.figure(figsize=(15,5))

x_t = np.linspace(-5, 5, 500)

nu1 = 1
t1_dist = stats.t(nu1) # the Student t distribution
t1_label='t pdf' + '\n' + r'$\nu=${:1.1f}'.format(nu1)
dist_plot(t1_label, x_t, t1_dist, 1)

nu2 = 5
t2_dist = stats.t(nu2) # the Student t distribution
t2_label='t pdf' + '\n' + r'$\nu=${:1.1f}'.format(nu2)
dist_plot(t2_label, x_t, t2_dist, 2)

nu3 = 100
t3_dist = stats.t(nu3) # the Student t distribution
t3_label='t pdf' + '\n' + r'$\nu=${:1.1f}'.format(nu3)
dist_plot(t3_label, x_t, t3_dist, 3)

```

Note the "heavy tails" in the t distribution as $\nu$ gets small.  As $\nu$ gets large, the distribution approaches a standard normal (Gaussian) distribution.

#### Projected posterior plots

Here we use the [corner package](https://corner.readthedocs.io/en/latest/api.html) to make some projected posterior plots.

```python
%matplotlib inline
# examples of corner plots
ndim, nsamples = 2, 100000
#np.random.seed(42)
# generate some fake data from a normal distribution
norm_samples = stats.norm.rvs(size=ndim * nsamples).reshape([nsamples, ndim])
#figure = corner.corner(norm_samples)
figure1 = corner.corner(norm_samples, labels=[r"$x$", r"$y$", r"$\log \alpha$", r"$\Gamma \, [\mathrm{parsec}]$"],
                       quantiles=[0.16, 0.5, 0.84],
                       show_titles=True, title_kwargs={"fontsize": 12})
ax = figure1.get_axes()
figure1.set_size_inches(5,5)

ndim, nsamples = 2, 100000
#np.random.seed(42)
# generate some fake data from a beta distribution
a = 4
b = 20
beta_samples = stats.beta(a,b).rvs(size=ndim * nsamples).reshape([nsamples, ndim])
#figure = corner.corner(beta_samples)
figure2 = corner.corner(beta_samples, labels=[r"$x$", r"$y$", r"$\log \alpha$", r"$\Gamma \, [\mathrm{parsec}]$"],
                       quantiles=[0.16, 0.5, 0.84],
                       show_titles=True, title_kwargs={"fontsize": 12})
figure2.set_size_inches(5,5)

```

```python
%matplotlib inline
# now more than one mode (all random)
ndim, nsamples = 4, 50000
np.random.seed(1234)
data1 = np.random.randn(ndim * 4 * nsamples // 5).reshape([4 * nsamples // 5, ndim])
mean = 4*np.random.rand(ndim)
data2 = (mean[None, :] + np.random.randn(ndim * nsamples // 5).reshape([nsamples // 5, ndim]))
samples = np.vstack([data1, data2])

#figure = corner.corner(samples)
figure = corner.corner(samples, labels=[r"$x$", r"$y$", r"$\log \alpha$", r"$\Gamma \, [\mathrm{parsec}]$"],
                       quantiles=[0.16, 0.5, 0.84],
                       show_titles=True, title_kwargs={"fontsize": 12})

figure.set_size_inches(5,5)

```

### Sampling of 1d pdfs in Python

```python
%matplotlib inline

def plot_hist(name, x_dist, dist, num_samples, num_bins, plot_num):
    plt.subplot(1,3,plot_num)
    samples = dist.rvs(size=num_samples)
    count, bins, ignored = plt.hist(samples, num_bins, density=True,
                                     color='blue', alpha=0.7)
    plt.plot(x_dist,dist.pdf(x_dist), linewidth=2, color='r')
    title_string = name + '  samples = {:d}'.format(num_samples)
    plt.title(title_string)

mu, sigma = 0, 1.0 # mean and standard deviation
x_dist = np.linspace(-4, 4, 500)
name = r'normal $\mu=${:1.1f}, $\sigma=${:1.1f}'.format(mu,sigma)
plt.figure(figsize=(15,5))
num_bins = 50

num_samples = 100
norm_dist = stats.norm(mu, sigma)
plot_hist(name, x_dist, norm_dist, num_samples, num_bins, 1)

num_samples = 1000
norm_dist = stats.norm(mu, sigma)
plot_hist(name, x_dist, norm_dist, num_samples, num_bins, 2)

num_samples = 10000
norm_dist = stats.norm(mu, sigma)
plot_hist(name, x_dist, norm_dist, num_samples, num_bins, 3)

```

### Bayes' Rule: Interaction of prior and likelihood

$A$ and $B$ are generic propositions and $I$ is "information" (things we know).  $p(A \mid B)$ means the probability of $A$ given $B$ (or contingent or conditional on $B$).

A particular case is a vector of parameters $\textbf{a} = \{a_1, a_2, \cdots\}$ for a theoretical model and some data it describes. Here Bayes' Rule is being used for parameter estimation.

$$
  \newcommand{\avec}{\textbf{a}}
  p(A \mid B,I) =
        \frac{p(B \mid A,I)\, p(A \mid I)}{p(B \mid I)}
  \ \Longrightarrow\
   \overbrace{p(\avec \mid \textrm{data},I)}^{\textrm{posterior}} =
  \frac{\color{red}{\overbrace{p(\textrm{data} \mid \avec,I)}^{\textrm{likelihood}}} \times
   \color{blue}{\overbrace{p(\avec \mid I)}^{\textrm{prior}}}}
   {\color{darkgreen}{\underbrace{p(\textrm{data} \mid I)}_{\textrm{evidence}}}}
$$

Common notation in statistics: $\boldsymbol{\theta}$ for parameters, $\mathcal{L}$ or $L$ for the likelihood, $\pi(\boldsymbol{\theta})$ for the prior.

$$\overbrace{p(\avec \mid \textrm{data},I)}^{\textrm{posterior}} \propto \color{red}{\overbrace{p(\textrm{data} \mid \avec,I)}^{\textrm{likelihood}}} \times
   \color{blue}{\overbrace{p(\avec \mid I)}^{\textrm{prior}}}$$

Left: likelihood overwhelms prior.  Right: prior is returned (restricts domain)

Note: these are one-dimensional projections of multi-dimensional pdfs.

Here we don't need to calculate the evidence separately; just normalize the numerator.

### Bayesian rules of probability as principles of logic

Notation: $p(x \mid I)$ is the probability (or pdf) of $x$ being true
given information $I$

 Sum rule: If set $\{x_i\}$ is exhaustive and exclusive,
  $$ \sum_i p(x_i  \mid  I) = 1   \quad \longrightarrow \quad       \color{red}{\int\!dx\, p(x \mid I) = 1}
  $$

 cf. complete and orthonormal

 implies marginalization (cf. inserting complete set of states or integrating out variables)
  $$
   p(x \mid  I) = \sum_j p(x,y_j \mid I)
    \quad \longrightarrow \quad
   \color{red}{p(x \mid I) = \int\!dy\, p(x,y \mid I)}
  $$

 Product rule: expanding a joint probability of $x$ and $y$
        $$
         \color{red}{ p(x,y \mid I) = p(x \mid y,I)\,p(y \mid I)
              = p(y \mid x,I)\,p(x \mid I)}
        $$

 If $x$ and $y$ are mutually independent:  $p(x \mid y,I)
      = p(x \mid I)$, then
    $$
       p(x,y \mid I) \longrightarrow p(x \mid I)\,p(y \mid I)
    $$

 Rearranging the second equality yields  Bayes' Rule (or Theorem)
     $$
      \color{blue}{p(x  \mid y,I) = \frac{p(y \mid x,I)\,
       p(x \mid I)}{p(y \mid I)}}
     $$

See Cox for the proof.

### Bayesian model checking: one example

How can you evaluate whether your Bayesian predictions are working?
Cf. checking whether a least-squares fit to data with  Gaussian noise misses about 1/3 of your 1-$\sigma$ error bars.
More generally: are the residuals normally distributed?

Are your Bayesian credible intervals consistent with observed successes? Check with a calibration or empirical coverage or credible interval diagnostic plot.

To be discussed: other ways to do Bayesian model checking.

### Networks

A Bayesian network is a graphical model that makes conditional dependence explicit through the edges in a directed graph.  (More on this soon!)

### Model discrepancy

$\newcommand{\yexp}{\textbf{y}_{\rm exp}}$
$\newcommand{\yth}{\textbf{y}_{\rm th}}$
$\newcommand{\ytrue}{\textbf{y}_{\rm true}}$
The main goal of Bayesian parameter estimation is the calculation of a joint posterior pdf for the model parameters given a set of experimental data and any other information we have.  This task begins with a statistical model:
$$ \yexp = \yth + \Delta \yth + \Delta \yexp $$
where $\yexp$ are the experimental measurements of an observable $y$
and $\yth$ are the corresponding theoretical (model) calculations.
In terms of the true results $\ytrue$, we can decompose this as
$$ \yexp = \ytrue + \Delta\yexp\;; \qquad
   \ytrue = \yth + \Delta\yth  \;.
$$
The model for the experimental uncertainty $\Delta\yexp$ is usually stochastic noise
$$ \Delta\yexp \sim \mathcal{N}(0,\Sigma_{\rm exp}) $$
with zero mean and $\Sigma_{\rm exp}$ typically uncorrelated (so the covariance matrix is diagonal).  Systematic uncertainties can also be incorporated.

The "new" feature here is $\Delta\yth$, which is the model discrepancy or model defect.  Its role is to account statistically for the deficiencies of the model.  It could be a truncation error from an expansion or a model of the observed residuals in a set of training data.
By including $\Delta\yth$, we can suppress overfitting and deal with underfitting.

[Return to Contents]

## Choosing priors

A key feature of Bayesian statistics is the choice of the prior probability distribution.  We should incorporate all of the information we have, but no more. What are the options for choosing priors?  What are the subtleties?  Places to use an informative prior:

your dataset is small, but there is related information available from other systems;

your model is very flexible but you want to prevent overfitting (so use priors that prefer
      values close to zero);

you want to stay away from particular regions of parameter space: maybe strictly unphysical such as
      negative cross sections or acausal, or maybe
      counter to well-motivated assumptions such as parameters are of order unity;

the posterior from a previous experiment can become a prior ==> Bayesian updating;

other examples??

### Subtlety with uniform prior

An example from the 2016 workshop on [Bayesian Methods in Astronomy](https://github.com/jakevdp/BayesianAstronomy/blob/master/Index.ipynb) considers fitting a straight line $y = mx+b$ to noisy data. If you take the prior on the slope $m$ to be uniform (flat), thinking that this is non-informative, then look at this plot, which samples lines with uniformly distributed slopes:

```python
%matplotlib inline
xx = np.linspace(-1, 1,11)
for slope in np.linspace(0, 20, 100):
    plt.plot(xx, slope * xx, '-k', linewidth=1)
plt.axis([-1, 1, -1, 1], aspect='equal');
```

```python
xx
```

    array([-1. , -0.8, -0.6, -0.4, -0.2,  0. ,  0.2,  0.4,  0.6,  0.8,  1. ])

The density of the lines indicates the relative probability of different slopes. Summary point: flat priors are not necessarily minimally informative.  For the slope we probably want a prior that does not artificially over-weight large slopes; see http://arxiv.org/abs/1411.5018 for some discussion. For example, we might use a flat prior on the *angle* the line makes with the x-axis, which implies
$$
P(m) \propto (1 + m^2)^{-3/2}
$$

### Maximum entropy for priors (appealing to physicists!)

 Basic idea: identify least informative $p(x)$ from maximizing entropy:
   $$
     S[p(x)] = -\int\!dx\, p(x)\, \log\left[\frac{p(x)}{m(x)}\right]
   $$
   subject to constraints from the prior information.

 $m(x)$ is an appropriate measure (often uniform, but see Sivia)

 Use Lagrange multiplier(s) to maximize

 One constraint is always normalization: $\int\!dx\, p(x) = 1$
     $\Longrightarrow$ alone it leads to uniform $p(x)$ (actually to $m(x)$)

   If the mean $\mu$ and variance $\sigma^2$ are known, then maximize (may be clearer to do with discretized variables)

   \begin{align}
    Q[p(x)] &= -\int\! dx\, p(x)\, \log\left[\frac{p(x)}{m(x)}\right]
    + \lambda_0 \left[ 1 - \int\! dx\, p(x) \right] \\
     & \quad + \lambda_1 \left[\sigma^2 -  \int\! dx\, (x-\mu)^2 p(x)  \right]
   \end{align}

   Then
   $$
    \frac{\delta Q}{\delta p(x)} = 0   \quad\mbox{and}\quad
     m(x) = \mbox{const.}
     \ \Longrightarrow\
     p(x \mid \mu,\sigma) = \frac{1}{\sigma\sqrt{2 \pi}}
     e^{-(x-\mu)^2/2\sigma^2}
   $$

    For $\textbf{x} = \{x_1, \cdots, x_N\}$, we find $p(\textbf{x})$
    is the familiar least-squares likelihood function.  See Sivia for many more details.

If you want a "scale invariant" prior, this often means a flat prior on the *logarithm* of the parameter.  This is an example of a [Jeffreys Prior](https://en.wikipedia.org/wiki/Jeffreys_prior).

[Return to Contents]

### Conjugate priors

Bayes theorem tells us that the posterior follows from multiplying the prior by the likelihood and normalizing:

$$
  p(\theta\,|\,x) = \frac{p(x\,|\,\theta)\, p(\theta)}
                          {\int\!d\theta'\, p(x\,|\,\theta')\, p(\theta')}
$$

If the resulting $p(\theta\,|\,x)$ is in the same family of pdfs as $p(\theta)$, then $p(\theta)$ is said to be a conjugate prior for the likelihood $p(x\,|\,\theta)$.

The likelihood is often a fixed form. If there is freedom to choose a conjugate prior, the Bayesian updating of the posterior is given in closed form.

Comments:

Suppose we are flipping a coin and seek the posterior for the probability of heads $\theta \in [0,1]$.  Let $x$ be the number of successes (heads) in $N$ trials. The likelihood is binomial:
   $$ p(x\mid\theta) = \binom{N}{x}
       \theta^{x} (1-\theta)^{N-x} \;.
   $$
If we choose the prior to be a beta distribution with hyperparameters $a$ and $b$ (which are parameters of the prior as opposed to the model),
   $$ p(\theta\mid a,b) = \textrm{Beta}(\theta\mid a,b)
      = \frac{\theta^{a-1}(1-\theta)^{b-1}}{B(a,b)}
   $$
with beta function $B(a,b)$, then the posterior is of the same form ($\bar x \equiv N - x$):
   $$ p(\theta\mid x,a,b) = \frac{\theta^{x+a-1}(1-\theta)^{\bar x+b-1}}{B(x+a,\bar x+b)} = \textrm{Beta}(\theta\mid x+a,\bar x +b)
   $$
This posterior can be used as the prior for more samples; the hyperparameters just add additional information (check that it doesn't matter if you analyze the data all at once or sequentially because the tosses are independent).

See the [Wikipedia article](https://en.wikipedia.org/wiki/Conjugate_prior) for a big table of conjugate prior pairs.

If the likelihood is a normal distribution with known variance, the conjugate prior is also normal.  If it is a normal distribution with known mean, the inverse gamma distribution is a conjugate prior.

[Return to Contents]

## Bayesian updating examples

### Determining the bias of a coin

```python
%matplotlib inline
# adapted from  https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers
from IPython.core.pylabtools import figsize
import numpy as np
from matplotlib import pyplot as plt
figsize(11, 15)

import scipy.stats as stats

# If the coin is fair, prob_heads = 0.5 but you can set it to what you want.
prob_heads = 0.7

# hyperparameters for several different priors
# prior 1 is uniform in [0,1]
alpha_1 = 1
beta_1 = 1
# prior 2 is concentrated near 0.5 with very small tails
alpha_2 = 30
beta_2 = 30
# prior 3 is peaked at ends, but allows for probability everywhere
alpha_3 = .2
beta_3 = .2

# Calculate Bayesian updating using the conjugate prior for binomial, which is a beta distribution
dist = stats.beta
n_trials = [0, 1, 2, 3, 4, 5, 8, 15, 50, 500, 1000, 2000]
data = stats.bernoulli.rvs(prob_heads, size=n_trials[-1])  # heads or tails, 1 or 0
x = np.linspace(0, 1, 301) # mesh for posterior plots

for k, N in enumerate(n_trials):  # enumerate creates a tuple with a counter for each n_trials entry
    heads = data[:N].sum()   # add up the number of 1s = number of heads
    # update using the conjugate prior
    y_1 = dist.pdf(x, alpha_1 + heads, beta_1 + N - heads)   # beta(x,alpha+heads,beta+(N-heads))
    y_2 = dist.pdf(x, alpha_2 + heads, beta_2 + N - heads)
    y_3 = dist.pdf(x, alpha_3 + heads, beta_3 + N - heads)
    y_max = np.max([y_1.max(), y_2.max()])

    # now make the plots!
    sx = plt.subplot(len(n_trials)/2, 2, k+1)
    plt.xlabel("$p$, probability of heads")
    plt.setp(sx.get_yticklabels(), visible=False)
    plt.yticks([])
    plt.plot(x, y_1, label="uniform prior")
    plt.fill_between(x, 0, y_1, color="blue", alpha=0.1)
    plt.plot(x, y_2, label="informative prior", color="r")
    plt.fill_between(x, 0, y_2, color="red", alpha=0.1)
    plt.plot(x, y_3, label="anti prior", color="g")
    plt.fill_between(x, 0, y_3, color="green", alpha=0.1)
    plt.vlines(prob_heads, 0, 1.1*y_max, color="k", linestyles="--", lw=2)
    plt.annotate("observe {:d} tosses,\n {:d} heads".format(N, heads), xy=(0.05,0.35),
                 xycoords='axes fraction', horizontalalignment='left',verticalalignment='top')
    leg = plt.legend()
    leg.get_frame().set_alpha(0.4)
    plt.autoscale(tight=True)

figure_title = "Bayesian updating of posterior probabilities for biased coin with actual p(heads) = %1.2f" % prob_heads
plt.suptitle(figure_title,
             y=1.02,
             fontsize=14)

plt.tight_layout()
```

## Sampling

### Sampling from multivariate normal distributions

Suppose we have a univariate normal distribution
$$
  x \sim \mathcal{N}(\mu,\sigma^2)
  \ \Longleftrightarrow\
  p(x\mid\mu,\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}
     e^{-(x-\mu)^2/2\sigma^2}
$$
If we have a way to generate standard normals $\mathcal{N}(0,1)$, then we can sample $x$ from
$$
    x \sim \mu + \sigma \mathcal{N}(0,1)  \;.
$$
So there is a simple shift by $\mu$ and then we scale the normal draw by $\sigma$, the square root of the variance.

$\newcommand{\xvec}{\textbf{x}}$
$\newcommand{\muvec}{\boldsymbol{\mu}}$
The general multivariate Gaussian distribution is
$$
   \xvec \sim \mathcal{N}(\muvec,\Sigma)
  \ \Longleftrightarrow\
 p(\xvec\mid \muvec,\Sigma) = \frac{1}{\sqrt{\det(2\pi\Sigma)}} e^{-\frac12(\xvec-\muvec)^{\rm T}\Sigma^{-1}(\xvec-\muvec)}
$$
The generalization to sample it will be to shift by $\muvec$ and scale by some square root of the covariance matrix $\Sigma$:
$$
    \xvec \sim \muvec + B \mathcal{N}(0,I) \;,
$$
where $I$ is the identity matrix and $B B^\intercal = \Sigma$ (e.g., a Cholesky decomposition).

[Return to Contents]

### MCMC sampling

Great examples of MCMC sampling are at http://elevanth.org/blog/2017/11/28/build-a-better-markov-chain/.

Here are the individual simplified simulations:

Metropolis-Hastings 2D Gaussian
     and donut

Hamiltonian Monte Carlo 2D Gaussian
     and donut

Hamiltonian Monte Carlo 2D Gaussian with
      U-turn

NUTS sampler 2D Gaussian
     and multi-modal

The detailed simulator can be found at https://chi-feng.github.io/mcmc-demo/, which also links to the github repository with the javascript source.

```javascript
%%javascript
IPython.OutputArea.auto_scroll_threshold = 9999;
```

    <IPython.core.display.Javascript object>

```python
# At present this adversely affects the menu bar
from IPython.display import display,HTML

display(HTML(filename="./MCMC_javascript_visualizations.html"))

```

<!DOCTYPE html>

	<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
	<meta name="viewport" content="width=device-width, initial-scale=1">
	<title>Markov Chains: Why Walk When You Can Flow? | Elements of Evolutionary Anthropology</title>
    <link rel="profile" href="http://gmpg.org/xfn/11">
    <link rel="pingback" href="http://elevanth.org/blog/xmlrpc.php">
    <link rel="dns-prefetch" href="http://s0.wp.com/">
<link rel="dns-prefetch" href="http://fonts.googleapis.com/">
<link rel="dns-prefetch" href="http://s.w.org/">
<link rel="alternate" type="application/rss+xml" title="Elements of Evolutionary Anthropology » Feed" href="http://elevanth.org/blog/feed/">
<link rel="alternate" type="application/rss+xml" title="Elements of Evolutionary Anthropology » Comments Feed" href="http://elevanth.org/blog/comments/feed/">

<link rel="stylesheet" id="wp-quicklatex-format-css" href="./MCMC_javascript_visualizations_files/quicklatex-format.css" type="text/css" media="all">
<link rel="stylesheet" id="tt-easy-google-fonts-css" href="./MCMC_javascript_visualizations_files/css" type="text/css" media="all">
<link rel="stylesheet" id="dashicons-css" href="./MCMC_javascript_visualizations_files/dashicons.min.css" type="text/css" media="all">
<link rel="stylesheet" id="crawford_oswald-css" href="./MCMC_javascript_visualizations_files/css(1)" type="text/css" media="all">
<link rel="stylesheet" id="crawford_domine-css" href="./MCMC_javascript_visualizations_files/css(2)" type="text/css" media="all">
<link rel="stylesheet" id="crawford_bootstrap_css-css" href="./MCMC_javascript_visualizations_files/bootstrap.min.css" type="text/css" media="all">
<link rel="stylesheet" id="crawford_style-css" href="./MCMC_javascript_visualizations_files/style.css" type="text/css" media="all">
<link rel="stylesheet" id="jetpack-widget-social-icons-styles-css" href="./MCMC_javascript_visualizations_files/social-icons.css" type="text/css" media="all">
<link rel="stylesheet" id="social-logos-css" href="./MCMC_javascript_visualizations_files/social-logos.min.css" type="text/css" media="all">
<link rel="stylesheet" id="jetpack_css-css" href="./MCMC_javascript_visualizations_files/jetpack.css" type="text/css" media="all">

<link rel="https://api.w.org/" href="http://elevanth.org/blog/wp-json/">
<link rel="EditURI" type="application/rsd+xml" title="RSD" href="http://elevanth.org/blog/xmlrpc.php?rsd">
<link rel="wlwmanifest" type="application/wlwmanifest+xml" href="http://elevanth.org/blog/wp-includes/wlwmanifest.xml">
<link rel="prev" title="Journal Clubbing: Accurate Age Estimation" href="http://elevanth.org/blog/2017/09/09/journal-clubbing-accurate-age-estimation/">
<link rel="next" title="Algebra and the Missing Oxen" href="http://elevanth.org/blog/2018/01/29/algebra-and-missingness/">
<meta name="generator" content="WordPress 4.9.8">
<link rel="canonical" href="http://elevanth.org/blog/2017/11/28/build-a-better-markov-chain/">
<link rel="shortlink" href="https://wp.me/p6D80w-bM">
<link rel="alternate" type="application/json+oembed" href="http://elevanth.org/blog/wp-json/oembed/1.0/embed?url=http%3A%2F%2Felevanth.org%2Fblog%2F2017%2F11%2F28%2Fbuild-a-better-markov-chain%2F">
<link rel="alternate" type="text/xml+oembed" href="http://elevanth.org/blog/wp-json/oembed/1.0/embed?url=http%3A%2F%2Felevanth.org%2Fblog%2F2017%2F11%2F28%2Fbuild-a-better-markov-chain%2F&format=xml">

<link rel="dns-prefetch" href="http://v0.wordpress.com/">

<meta property="og:type" content="article">
<meta property="og:title" content="Markov Chains: Why Walk When You Can Flow?">
<meta property="og:url" content="http://elevanth.org/blog/2017/11/28/build-a-better-markov-chain/">
<meta property="og:description" content="Abstract: If you are still using a Gibbs sampler, you are working too hard for too little result. Newer, better algorithms trade random walks for frictionless flow. In 1989, Depeche Mode was popula…">
<meta property="article:published_time" content="2017-11-28T20:50:01+00:00">
<meta property="article:modified_time" content="2018-04-26T08:15:56+00:00">
<meta property="og:site_name" content="Elements of Evolutionary Anthropology">
<meta property="og:image" content="http://elevanth.org/blog/wp-content/uploads/2017/11/KRDx8SR-1.gif">
<meta property="og:image:width" content="500">
<meta property="og:image:height" content="381">
<meta property="og:locale" content="en_US">
<meta name="twitter:site" content="@rlmcelreath">
<meta name="twitter:text:title" content="Markov Chains: Why Walk When You Can Flow?">
<meta name="twitter:card" content="summary">

<link rel="icon" href="http://elevanth.org/blog/wp-content/uploads/2015/08/elevanth_icon_box-150x150.png" sizes="32x32">
<link rel="icon" href="http://elevanth.org/blog/wp-content/uploads/2015/08/elevanth_icon_box-300x300.png" sizes="192x192">
<link rel="apple-touch-icon-precomposed" href="http://elevanth.org/blog/wp-content/uploads/2015/08/elevanth_icon_box-300x300.png">
<meta name="msapplication-TileImage" content="http://elevanth.org/blog/wp-content/uploads/2015/08/elevanth_icon_box-300x300.png">

Markov Chains: Comparing different samplers

Abstract: If you are still using a Gibbs sampler, you are working too hard for too little result. Newer, better algorithms trade random walks for frictionless flow.
            [Adapted from a blog entry
            by Richard McElreath (@rlmcelreath).]

Metropolis, Hastings, and the Random Walk

The simplest and least reliable way of building a Markov chain is the Metropolis-Hastings algorithm.
Below are embedded the MCMC simulations written by Chi Feng, found here. The target distribution is a benign two-dimensional Gaussian—a nice Gaussian hill. You are looking down on it, with its peak in the center. The Markov chain wanders around this hill, making random proposals to move away from its current position. These proposals are represented by the arrows. Green arrows are accepted proposals. The chain moves to the new location. Red arrows are rejections.

<object data="./MCMC_javascript_visualizations_files/applet.html#RandomWalkMH,standard" type="text/html" width="600" height="400"></object>

As samples build up, the target distribution takes shape, as seen in the histograms on the bottom and left margins. The algorithm works, even though it blindly stumbles around the target, doing a kind of random walk.

And that is exactly the problem. A major problem is that Metropolis-Hastings is a bit too random. So it spends a lot of time re-exploring the same parts of the target, and unless it is tuned just right, it will also reject a lot of proposals (the red arrows), wasting computer time.

Here’s another simulation, this time with a harder target: a donut. The donut target might look weird, but it represents a very common target, by analogy. In high dimensions—once there are many parameters—the target distribution occupies a narrow ring (in high-dimensional space). Most of the probability mass is not near the center. Now look what happens to Metropolis-Hastings:

<object data="./MCMC_javascript_visualizations_files/applet.html#RandomWalkMH,donut" type="text/html" width="600" height="400"></object>

Notice how the Markov chain tends to get stuck in specific regions of the donut. Not only that, but it rejects a lot of proposals (the red arrows), so it wastes a lot of computer time doing nothing. Given enough time, it can explore the entire target. But it might take a very long time indeed. When we don’t know what the target looks like in the first place, this kind of behavior is not only annoying, but also hazardous to inference.

The fundamental problem with Metropolis-Hastings, and with Gibbs-Sampling as a special case, is that it is just too random. In simple targets, that isn’t so bad. But in even moderately complex targets, it means inference often isn’t reliable. It tends to get stuck in narrow regions of the target. There must be a better way.

Better Living Through Physics

Can your Markov chain do this?

If there’s a random way to do something, there’s usually a less random way that is both better and requires more thought. Instead of making random proposals, suppose instead that you run a physics simulation. Your vector of parameters is now a particle in n-dimensional space. The surface in this space is a giant n-dimensional bowl. The shape of the bowl is determined by the shape of the logarithm of the target distribution. If the target is a nice Gaussian, for example, then the log-Gaussian is a smooth parabolic bowl like this (in one-dimension):

To make things a little crazier, suppose that this surface is frictionless. Now what we do is flick the particle in a random direction. It will frictionlessly flow across the bowl, eventually turning around. If we take samples of the particle’s position along its path, before flicking it off in another random trajectory, then we can learn about the shape of the surface.

This is the principle behind Hamiltonian Monte Carlo. It will be easier to see it in action. Here is another simulation, this time using Hamiltonian Monte Carlo, again on the two-dimensional Gaussian target. The paths are flicks of the particle, and the green arrows again represent accepted proposals.

<object data="./MCMC_javascript_visualizations_files/applet.html#HamiltonianMC,standard" type="text/html" width="600" height="400"></object>

Now the proposals are both within the high-probability region of the target—so many fewer proposals are rejected—and the proposals can get far away from their starting point, so that the chain efficiently explores the whole shape of the target in less time. Effectively, it flows across the target and maps out its whole shape much faster.

The cost of all this elegance is needing more information about the target. Hamiltonian Monte Carlo does a lot more calculation. But it also needs fewer samples to get a good image of the target. Where this really counts is with the donut. Let’s revisit it:

<object data="./MCMC_javascript_visualizations_files/applet.html#HamiltonianMC,donut" type="text/html" width="600" height="400"></object>

Now instead of getting stuck, the chain sweeps around the target. Even though all the chain knows at any moment in time is the local shape of the target—it can’t see the whole distribution like you can here—it still manages to glide smoothly around it. This shouldn’t be so amazing—a ball doesn’t know the shape of the surface it rolls on, yet its path is governed by it.

Stan is NUTS

There are still some improvements to be had. Hamiltonian Monte Carlo needs to be told how many steps to take in its simulated paths. The step number determines how long the path continues before a new flick is made in a new random direction. If it takes too few steps, then it ends up with samples too similar to one another. If it takes too many, it can also end up with samples too similar to one another. Why? Because the path eventually makes a U-turn. Here’s an example:

<object data="./MCMC_javascript_visualizations_files/applet.html#HamiltonianMC,standard,65" type="text/html" width="600" height="400"></object>

The path goes on long enough that it often eats its own tail—it makes an unfortunate U-turn. The algorithm still works, but it isn’t very efficient, because it again explores in local spaces. We can of course tune the number of steps by hand, but that is not so easy when the target distribution is complex.

The No U-Turn Sampler (NUTS) is an approach for adaptively finding a good number of steps. The NUTS algorithm tries to figure out when the path starts to turn around. In order to do this efficiently, it needs to simulate the path in both directions. It looks pretty weird:

<object data="./MCMC_javascript_visualizations_files/applet.html#NaiveNUTS,standard" type="text/html" width="600" height="400"></object>

Notice how the path grows in both directions. This is the algorithm figuring out when the path turns around. When the path starts to turn around, NUTS stops the simulation and takes a sample. Then it flicks the particle in another random direction and starts another simulation. There are lots of little adaptive nudges in these algorithms that help them explore the target more efficiently. An implementation like Stan (mc-stan.org) uses an advanced version of NUTS that is even slicker.

Problems Remain

Hamiltonian algorithms still have limitations. Some targets are still hard to explore. Here’s a example: a multimodal target. While the paths do a good job exploring each lump of probability mass, they have trouble transitioning among them.

<object data="./MCMC_javascript_visualizations_files/applet.html#NaiveNUTS,multimodal" type="text/html" width="600" height="400"></object>

Targets like this are not so unusual. They arise in many classification and latent variable models. With some clever coding, you can collapse some modes together. But you have to realize what is going on, first. Other issues arise with models that contain very steep changes in log-probability. Wizards are working on solutions to these problems. But even before solutions are found, Hamiltonian samplers are typically much better than Gibbs zombies.

Read More

The BUGS project’s history is summarized in: Lunn, Spiegelhalter, and Best. (2009). “The BUGS project: Evolution, critique and future directions.” doi:10.1002/sim.3680

Hamiltonian Monte Carlo was originally called “Hybrid Monte Carlo”: Duane, Kennedy, Pendleton, and Roweth (1987) “Hybrid Monte Carlo”. doi:10.1016/0370-2693(87)91197-X

The No U-Turn Sampler (NUTS) was first described by Hoffman and Gelman (2011) “The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo.” arxiv.org/abs/1111.4246

Michael Betancourt’s “Conceptual Introduction to Hamiltonian Monte Carlo” is only slightly technical and worth your time. arxiv.org/abs/1701.02434

		<form action="http://elevanth.org/blog/2017/11/28/build-a-better-markov-chain/" method="post">
			Send to Email Address
			<input type="email" name="target_email" id="target_email" value="">

				Your Name
				<input type="text" name="source_name" id="source_name" value="">

				Your Email Address
				<input type="email" name="source_email" id="source_email" value="">

						<input type="text" id="jetpack-source_f_name" name="source_f_name" class="input" value="" size="25" autocomplete="off" title="This field is for validation and should not be changed">

			<input type="submit" value="Send Email" class="sharing_send">
			Cancel

				Post was not sent - check your email addresses!

				Email check failed, please try again

				Sorry, your blog cannot share posts by email.

		</form>

[Return to Contents]

## Model selection and evidence

  $\newcommand{\thetavec}{\boldsymbol{\theta}}$
  Determine the evidence for different models $M_1$ and $M_2$ via marginalization by integrating over all possible sets of parameters ${\thetavec}$ in different models, with the same data $D$ and information $I$.

The evidence ratio for two different models:
$$
   \frac{p(M_1\mid D, I)}{p(M_2\mid D, I)}
   = \frac{p(D\mid M_1, I)\,p(M_1,I)}{p(D\mid M_2, I)\,p(M_2,I)}
$$

The Bayes Ratio (implements Occam’s Razor):
$$
\frac{p(D\mid M_1, I)}{p(D\mid M_2, I)}
  = \frac{\int\!d\thetavec_1\, p(D\mid\thetavec_1,M_1,I)
             \,p(\thetavec_1\mid M_1,I)}
         {\int\!d\thetavec_2\, p(D\mid\thetavec_2,M_2,I)
             \,p(\thetavec_2\mid M_2,I)}
$$

Example: what order polynomial underlies the noisy data?

[Return to Contents]

## Gaussian processes (GPs)

### Overview of GPs

GP: the natural generalization of multivariate Gaussian random variables to infinite (countably or continuous) index sets.  They look like random functions, but with characteristic degrees of smoothness, correlation lengths, and range.  Here are some examples with different "kernels" and different parameters that dictate those features (figure from J. Melendez):

#### Explanations of GPs from the web

The following is adapted from a blog entry from Kathleen Bailey at http://katbailey.github.io/post/gaussian-processes-for-dummies/.

"Here’s how Kevin Murphy explains it in the excellent textbook Machine Learning: A Probabilistic Perspective:"
'A GP defines a prior over functions, which can be converted into a posterior over functions once we have seen some data. Although it might seem difficult to represent a distribution over a function, it turns out that we only need to be able to define a distribution over the function’s values at a finite, but arbitrary, set of points, say $x_1, \ldots, x_N$.  A GP assumes that $p(f(x_1),\ldots,f(x_N))$ is jointly Gaussian, with some mean $\mu(x)$ and covariance $\Sigma(x)$ given by $\kappa_{ij} = \Sigma(x_i,x_j)$, where $\kappa$ is a positive definite kernel function. The key idea is that if $x_i$ and $x_j$ are deemed by the kernel to be similar, then we expect the output of the function at those points to be similar, too.'

So it is important to stress that we are really only dealing with a discrete set of points. Thus the physicist-friendly idea of a continuum limit of masses on springs may be preferred to more abstract notions in function space.

It should also be sufficient to consider the bivariate case, because the generalization from one to two variables is really where the new feature of correlation comes in.  Generalizing further really doesn't introduce anything new.

### Bivariate normal case

$\newcommand{\xvec}{\textbf{x}}$
$\newcommand{\muvec}{\boldsymbol{\mu}}$
The general multivariate Gaussian distribution is
$$
 p(\xvec\mid \muvec,\Sigma) = \frac{1}{\sqrt{\det(2\pi\Sigma)}} e^{-\frac12(\xvec-\muvec)^{\rm T}\Sigma^{-1}(\xvec-\muvec)}
$$

For the bivariate case we can parameterize the mean vector and covariance matrix as
$$
  \muvec = \left( \begin{array}{c}
                     \mu_x \\ \mu_y
                  \end{array}       \right)
    \;, \qquad
  \Sigma = \left( \begin{array}{cc}
                    \sigma_x^2 & \rho\sigma_x\sigma_y \\
                    \rho\sigma_x\sigma_y & \sigma_y^2
                  \end{array}
           \right)
$$
The covariance matrix must be positive definite, which implies $\color{red}{0\lt\rho^2\lt 1}$.

If take $\mu_x = \mu_y = 0$ and $\sigma_x = \sigma_y = \sigma$ for clarity,
so that
$$
  \Sigma = \sigma^2 \left(\begin{array}{cc}
                       1 & \rho \\
                       \rho & 1
                       \end{array}
                    \right)
$$
and
$$
  p(x,y\mid \sigma,\rho) = \frac{1}{2\pi\sigma^2}
        \exp\left(-\frac{x^2 + y^2 - 2\rho x y }{2\sigma^2\sqrt{1-\rho^2}}
          \right)
          \;.
$$
It's clear that contours of equal probability have $x^2 + y^2 - 2\rho xy = \mbox{constant}$, so they are ellipses.  The value of $\rho$ determines the eccentricity of the ellipse.
If $\rho=0$, $x$ and $y$ are independent (uncorrelated) and we have a circle.  As $\rho$ approaches $+1$, $x$ and $y$ are increasingly correlated (toward straight line at $45^\circ$), while for $\rho$ approaching $-1$ they become increasingly anti-correlated (toward straight line at $-45^\circ$).

For reference, the Cholesky decomposition of $\Sigma$ is
$$
  \Sigma = \sigma^2\left( \begin{array}{cc}
                    1 & \rho \\
                    \rho & 1
                  \end{array}
           \right)
           =
           \sigma^2\left( \begin{array}{cc}
                    1 & 0 \\
                    \rho &  \sqrt{1-\rho^2}
                  \end{array}
           \right)
           \left( \begin{array}{cc}
                    1 & \rho \\
                    0 &  \sqrt{1-\rho^2}
                  \end{array}
           \right)
$$

### Example code for generating and plotting GPs

The following code is adapted from a blog post by Katherine Bailey entitled Gaussian Processes for Dummies.  First we generate several instances of draws from a Gaussian process with a squared exponential kernel function, which is the covariance between $x$ and $x'$:
$$  \kappa_{\rm SE}(x,x') = \sigma^2 e^{-(x-x')^2/2l^2} $$

So we can see that $\sigma$ controls the vertical extent of the functions while $l$ controls
how rapidly they wiggle.  Comparing to our expression above for the bivariate normal case,
we see that $\rho$ is given by $e^{-(x-x')^2/2l^2}$.  So when $x$ and $x'$ are close,
$\rho \approx 1$ and the value of the function is highly correlated.  When $x$ and $x'$ are far apart, $\rho \rightarrow 0$, and they become independent (thus $l$ plays the role of a correlation length).

Let's generate some GPs with this kernel! For the function $f(x)$ we write draws as
$$
  f(x) \sim \mathcal{GP[\mu(x),\kappa(x,x')]}
$$
where $\mu(x)$ is the mean at each $x$ and $\kappa(x,x')$ is the covariance between $x$ and $x'$.  In practice we have a finite set of $N$ points $\textbf{x} = \{x_i\}_{i=1}^{N}$ with corresponding function values $\textbf{f}=\{f(x_i)\}_{i=1}^{N}$.
We form the mean vector $\boldsymbol{\mu} = m(\textbf{x})$ and the covariance matrix $K_{ij} = \kappa(x_i,x_j)$. Then
$$  \textbf{f} \mid \textbf{x} \sim \mathcal{N}(\boldsymbol{\mu},K)
$$
are draws from a multivariate normal distribution.  Try it:

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

# Define the squared exponential kernel function for the covariance
#   We take the variance to be 1.
def sqr_exp_kernel(a, b, length_param):
    sqdist = np.sum(a**2,1).reshape(-1,1) + np.sum(b**2,1) - 2*np.dot(a, b.T)
    return np.exp(-sqdist / (2*length_param**2))

# Grid of x points
npts = 500
x_min = -5
x_max = +5
Xtest = np.linspace(x_min, x_max, npts).reshape(-1,1)

length_param = .5  # this is "l" (correlation length)
K_ss = sqr_exp_kernel(Xtest, Xtest, length_param)

# Get Cholesky decomposition (square root) of the covariance matrix
nugget = 1e-12   # size of nugget will depend on how many points are used
                 #  1e-12 for 500; 1e-13 for 100; 1e-15 for 50
L = np.linalg.cholesky(K_ss + nugget*np.eye(npts))

# Sample 3 sets of standard normals for our test points,
# multiply them by the square root of the covariance matrix
# Note: mean mu = 0 here implicitly.
f_prior = np.dot(L, np.random.normal(size=(npts,3)))

# Now let's plot the 3 sampled functions.
plt.plot(Xtest, f_prior)
plt.axis([-5, 5, -3, 3])
plt.title('Three samples from the GP prior with l = {:1.1f}'.format(length_param))
plt.show()
```

Now we train it on some data (see references for details):

```python
# Noiseless training data
Xtrain = np.array([-4, -3, -2, -1, 1]).reshape(5,1)
ytrain = np.sin(Xtrain)
#ytrain = np.array([0,0,0,0,0]).reshape(5,1)

# Apply the same kernel function to our training points
nugget_train = 5e-5
K = sqr_exp_kernel(Xtrain, Xtrain, length_param)
L = np.linalg.cholesky(K + nugget_train*np.eye(len(Xtrain)))

# Compute the mean at our test points.
K_s = sqr_exp_kernel(Xtrain, Xtest, length_param)
Lk = np.linalg.solve(L, K_s)
mu = np.dot(Lk.T, np.linalg.solve(L, ytrain)).reshape((npts,))

# Compute the standard deviation so we can plot it
s2 = np.diag(K_ss) - np.sum(Lk**2, axis=0)
stdv = np.sqrt(s2)

# Draw samples from the posterior at our test points.
nugget_test = 1e-6
L = np.linalg.cholesky(K_ss + nugget_test*np.eye(npts) - np.dot(Lk.T, Lk))
f_post = mu.reshape(-1,1) + np.dot(L, np.random.normal(size=(npts,3)))

plt.plot(Xtrain, ytrain, 'bs', ms=8)
plt.plot(Xtest, f_post)
plt.gca().fill_between(Xtest.flat, mu-2*stdv, mu+2*stdv, color="#dddddd")
plt.plot(Xtest, mu, 'r--', lw=2)
plt.axis([-5, 5, -3, 3])
plt.title('Three samples from the GP posterior')
plt.show()
```

### Other demos for Gaussian Processes (and other regression)

Gaussian process regression, where you can add data points, play with the hyperparameters, and then see the inference for the curve. It’s by Tomi Peltola:
http://www.tmpl.fi/gp/

This simulation shows how a GP prior is a distribution over functions, and how observing data conditions the prior to obtain the GP posterior.
http://rpradeep.webhop.net/gpr/

[Return to Contents]

## Appendices

### References

Please suggest additional references (with links).

### Physics-oriented pedagogical articles and texts

R. Trotta,
        Bayes in the sky: Bayesian inference and model selection in cosmology, Contemp. Phys. 49, 71 (2008)
        [arXiv:0803.4089].

D.S. Sivia and J. Skilling,
       Data Analysis: A Bayesian Tutorial, 2nd edition, (Oxford University Press, 2006).

P. Gregory,
     Bayesian Logical Data Analysis for the Physical Sciences: A Comparative Approach with Mathematica® Support, (Cambridge University Press, 2010).

### Standard statistics references

A. Gelman et al.,
        Bayesian Data Analysis, 3rd edition, (Chapman and Hall/CRC, 2013).

### BUQEYE references

R.J. Furnstahl, D.R. Phillips and S. Wesolowski,
      A recipe for EFT uncertainty quantification in nuclear physics,
      J. Phys. G 42, 034028 (2015), [arXiv:1407.0657].

 R.J. Furnstahl, N. Klco, D.R. Phillips and S.Wesolowski,
    Quantifying truncation errors in effective field theory,
    Phys. Rev. C 92, 024005 (2015)
    [arXiv:1506.01343].

S. Wesolowski, N. Klco, R.J. Furnstahl, D.R. Phillips and A. Thapaliya,
  Bayesian parameter estimation for effective field theories,
    J. Phys. G 43, 074001 (2016)
    [arXiv:1511.03618].

  J.A. Melendez, S. Wesolowski and R.J. Furnstahl,
    Bayesian truncation errors in chiral effective field theory: nucleon-nucleon observables,
  Phys. Rev. C 96, 024003 (2017)
  [arXiv:1704.03308].

  S. Wesolowski, R.J. Furnstahl, J.A. Melendez and D.R. Phillips,
  Exploring Bayesian parameter estimation for chiral effective field theory using nucleon-nucleon phase shifts,
  [arXiv:1808.08211].

### Github repositories

Please suggest more!

https://github.com/jakevdp/BayesianAstronomy Materials for the Bayesian Methods in Astronomy workshop at the 227th American Astronomical Society meeting.  Includes Jupyter notebooks and useful exercises.

http://people.duke.edu/~ccc14/sta-663-2018/ STA 663: Computational Statistics and Statistical Computing (2018) at Duke University.  Lots of good things here!

### Vocabulary

Plan: build up a good set of definitions with appropriate links.  Please add more words/phrases!

conjugate prior

If the probability distribution family (e.g., beta distributions) for the posterior pdf is the same as for the prior pdf, the latter is said to be a conjugate prior. This means that the updating by Bayes' rule can be carried out analytically.   Some Bayesian practitioners are strongly opposed to the use of conjugate priors (see  comments here).

credible vs. confidence interval

This is a contrast between Bayesian and frequentist statistics.  For a frequentist, a parameter has a true value, which is fixed and not a distribution.  A 95% confidence interval mean that with a large number of repeated trials, 95% of the calculated confidence intervals would include the true value. This is clearly hard to think about!  A Bayesian 95% credible interval is the range of the posterior for the parameter (which is treated as a random variable) that has 95% of the probability.  So there is a 95% probability that the parameter is in that interval.

evidence

In the standard context of inferring parameters $\boldsymbol{\theta}$ given data $D$ and information $I$, the evidence is $p(D\mid I) = \int\! d\boldsymbol{\theta}\, p(D \mid \boldsymbol{\theta},I)\,p(\boldsymbol{\theta},I)$.  This is also called the Fully Marginalized Likelihood or FML. The expression shows that it is the integral over all $\boldsymbol{\theta}$ weighted by the likelihood.  This is typically an expensive integral to do.  In the context of model fitting (i.e., parameter estimation), it acts as a normalization constant and in most cases can be ignored because the normalization can be found directly (or only relative probabilities are needed).

gaussian process

From [Wikipedia](https://en.wikipedia.org/wiki/Gaussian_process): "In probability theory and statistics, a Gaussian process is a stochastic process (a collection of random variables indexed by time or space), such that every finite collection of those random variables has a multivariate normal distribution, i.e. every finite linear combination of them is normally distributed. The distribution of a Gaussian process is the joint distribution of all those (infinitely many) random variables, and as such, it is a distribution over functions with a continuous domain, e.g. time or space."

hierarchical model

A model with hyperparameters. See [Wikipedia](https://en.wikipedia.org/wiki/Bayesian_hierarchical_modeling).

hyperparameter

A parameter of a prior distribution.

iid (independently and identically distributed)

A set of random variables is iid (or i.i.d. or IID) if each random variable has the same probability distribution and all are mutually independent.

likelihood

Usually in the form $p(D\mid \boldsymbol{\theta},I)$, where $\boldsymbol{\theta}$ are the parameters of our model, $D$ is the data, and $I$ is any other information we use.  This is the probability of observing our actual data given the model (with the particular parameters $\boldsymbol{\theta}$). It is the same quantity that is maximized in frequentist maximum-likelihood approaches.

MAP estimate

Maximum a posteriori estimate. This is the mode (maximum) of the posterior distribution for the quantity of interest. If the prior is uniform, the MAP estimate equals the maximum likelihood estimate.

maximum entropy

A method used to determine priors.

MCMC

Markov-chain Monte Carlo. A generic name for stochastic sampling methods.

model selection and model averaging

nugget

For Gaussian process (GP) calculations or any sampling of a multivariate normal distribution, one typically needs to find the Cholesky decomposition of the covariance matrix.  However, this matrix can become ill-conditioned (too-small or negative eigenvalues).   A standard solution is to add a small number, called a nugget, to the diagonal of the covariance matrix. For GP regression, this is equivalent to adding (or increasing, if already present) the data noise.

nuisance parameter

A nuisance parameter is a parameter in your model whose value you don't care about for the posterior. So you integrate it out (marginalize).

overfitting and underfitting

This example from http://scikit-learn.org/stable/auto_examples/model_selection/plot_underfitting_overfitting.html of fitting polynomials to nonlinear functions illustrates overfitting and underfitting.  The true function is a cosine with noise added.  A polynomial of degree 1 is an inadequate model for this data; this is underfitting.  The polynomial of degree 15 tries to fit the noise; this is overfitting.

point estimate (cf. interval estimate)

A point estimate is a single value to characterize a
posterior.  It could be the mode, mean, median or something
else.  An interval estimate is more natural in Bayesian statistics, because the full posterior is the real target.  Giving a series of credible intervals often conveys much of the information about the posterior.

posterior

This is the quantity of the left side of Bayes' rule, the thing we want to compute.  Often in the form $p(\boldsymbol{\theta}\mid D,I)$, where $\boldsymbol{\theta}$ are the parameters of our model, $D$ is the data, and $I$ is any other information we use. It is our knowledge of the model given the data and any relevant background knowledge (which include the choice of model).

prior

A pdf that encodes what is known about the answer (e.g., parameters) before any data is used. The notation consistent with our definitions of posterior and likelihood is $p(\boldsymbol{\theta}\mid I)$, where $\boldsymbol{\theta}$ are the parameters of our model and $I$ is any other information we use (e.g., some of the parameters must be positive or less than a known magnitude because of physics reasons).
     See also conjugate prior and maximum entropy.

residual

The difference of theory prediction and experimental data.

### Notation    [still coming . . .]

Plan: build up a dictionary of notation with appropriate links and examples (with code).

univariate normal distribution
$$\mathcal{N}(\mu,\sigma^2)$$

[Return to Contents]

```python

```


*This notebook is part of  course materials for CS 345: Machine Learning Foundations and Practice at Colorado State University.
Original versions were created by Asa Ben-Hur.
The content is availabe [on GitHub](https://github.com/asabenhur/CS345).*

*The text is released under the [CC BY-SA license](https://creativecommons.org/licenses/by-sa/4.0/), and code is released under the [MIT license](https://opensource.org/licenses/MIT).*

<a href="https://colab.research.google.com/github//asabenhur/CS345/blob/master/notebooks/module03_02_derivatives_partial_derivatives.ipynb">

</a>

```python
%matplotlib inline
%autosave 0
import matplotlib.pyplot as plt
import numpy as np
```

    Autosave disabled

# Derivatives and partial derivatives

To solve linear regression we need to review the concept of derivatives, and how they can be used to find minima/maxima of a function of a single variable.  We will then extend these ideas to multivariate functions like the ones we deal with in machine learning.

### Derivatives

As you recall, the derivative of a function of a single variable indicates the steepness of the curve at a given point.
The following figure demonstrates that.

```python
plt.style.use('default')
plt.figure(figsize=(5,3))

def f(x):
    return x**3 + 10*x

def f_deriv(x):
    return (3 * x**2) + 10

x = np.linspace(0,9, 100)
y = f(x)

x1 = 5
y1 = f(x1)
deriv = f_deriv(x1)

plt.xlabel('x')
plt.ylabel('f(x)')

plt.plot(x, y)

plt.scatter(x1,y1, c='red')
plt.annotate(r'$(x^*,f(x^*))$',(x1,y1), xytext=(x1-0.5, y1-100))

# plot the tangent line at the point (x1,y1)
xMin = x1 - 2
yMin = y1 - (2*deriv)
xMax = x1 + 2
yMax = y1 + (2*deriv)
plt.plot([xMin,xMax],[yMin,yMax], color='magenta');

```

The line tangent to $f(x)$ at the point $x^*$ has a slope that is equal to the derivative of $f$ at that point.

#### Derivaties of common function

The following is a review of the derivative rules you will need to take the derivatives of the functions we will encounter in this course.

The *power rule*

\begin{equation}
\frac{d}{dx} x^{n} = nx^{n-1}
\end{equation}

The *product rule*

\begin{equation}
\frac{d}{dx}[f(x)g(x)] = f'(x)g(x) + f(x)g'(x)
\end{equation}

The *quotient rule*

\begin{equation}
\frac{d}{dx}\left[\frac{f(x)}{g(x)}\right] = \frac{f'(x)g(x) - f(x)g'(x)}{g(x)^2}
\end{equation}

The *chain rule*

\begin{equation}
\frac{d}{dx}[f(g(x))] = f'(g(x)) \cdot g'(x)
\end{equation}

which can be expressed as:

\begin{equation}
\frac{du}{dx} = \frac{d\!f}{dg}\frac{dg}{dx}
\end{equation}

Differentiating logs and exponentials:

$$
\frac{d}{dx}e^x = e^x
$$

$$
\frac{d}{dx}\log x = \frac{1}{x}
$$

### Exercises

Compute the derivaties of the following functions:

$$
\sigma(x) = \frac{1}{1 + e^{-x}}
$$

$$
f(x) = e^{-(x-\mu)^2}
$$

### Finding maxima and minima

To find the minimum/maximum of a function, look for points at which the derivative is 0.

Consider for example the function $f(x) = x^2 + 10x$.

The derivative:

$$
\frac{d\!f}{dx} = 2x + 10
$$

Setting to 0 and solving for $x$ we find that the minimum is at $x=-5$.
How did we know that it's a minimum and not a maximum?  That is based on the sign of the second derivative (positive for a minimum, negative for a maximum, zero for a saddle point).

Let's see a plot of the function:

```python
def f(x):
    return x**2 + 10*x

def f_deriv(x):
    return 2*x + 10

x = np.linspace(-15,5,100)
y = f(x)

plt.xlabel('x')
plt.ylabel('f(x)')

plt.plot(x, y);

```

In this case the function has a single global minimum.  Often, we are not so lucky.
Consider for example the function

$$
g(x) = x^4 + 7x^3 + 5x^2 - 17x
$$

As the following plot shows, this function has multiple minima.

```python
def g(x):
    return x**4 + 7*x**3 + 5*x**2 -17*x

def g_deriv(x):
    return 4*x**3 + 21*x**2 + 10*x - 17

x = np.linspace(-6, 2, 100)

plt.xlabel('x')
plt.ylabel('g(x)')

plt.plot(x, g(x));
```

Also, in this case solving for the minimum is not so easy, so we will consider an alternative method for this problem.

### Using derivatives to iteratively reach the minimum

For a function of a single variable, the derivative points us to the direction at which we would find a local minimum.  Well, actually it points at the opposite direction, so we would need to consider the negative of the derivative.
The idea is to take small steps in the direction of the negative of the derivative, which should converge to a local minimum.
This insight translates to the following algorithm:

* Initialize $x(0),\eta$
* for $t = 0,1,\ldots$
  * $x(t+1) = x(t) - \eta \frac{d\!f}{dx}(x(t))$
  * if converged, break

Let's illustrate this with the function $g$ we defined above:

```python
def g(x):
    return x**4 + 7*x**3 + 5*x**2 -17*x

def g_deriv(x):
    return 4*x**3 + 21*x**2 + 10*x - 17

eta = 0.01
x0 = -2.5 # the initial value
x_list = [x0]
y_list = [g(x0)]
for t in range(1, 20):
    x_new = x_list[-1] -eta * g_deriv(x_list[-1])
    x_list.append(x_new)
    y_list.append(g(x_new))

x_grid = np.linspace(-6, 2, 100)

plt.xlabel('x')
plt.ylabel('g(x)')

plt.plot(x_grid, g(x_grid));
plt.plot(x_list, y_list, 'ob', markersize=5, alpha=0.5);
```

In the example above, with the starting point chosen, we were able to find the global minimum of this function.  However, with a different choice, we might only find a **local minimum**:

```python
eta = 0.01
x0 = -1 # the initial value
x_list = [x0]
y_list = [g(x0)]
for t in range(1, 20):
    x_new = x_list[-1] -eta * g_deriv(x_list[-1])
    x_list.append(x_new)
    y_list.append(g(x_new))

x_grid = np.linspace(-6, 2, 100)

plt.xlabel('x')
plt.ylabel('g(x)')

plt.plot(x_grid, g(x_grid));
plt.plot(x_list, y_list, 'ob', markersize=5, alpha=0.5);
```

### Partial derivatives

Until now, we considered derivatives of functions of a single variable.
Next, we are going to consider functions of multiple variables, e.g.:

$$f(x_1,x_2) = x_1^2 + x_2^2$$

We can take a derivative with respect to either $x_1$ or $x_2$.
A derivative with respect to a variable while holding the others constant is called the **partial derivative**.  The partial derivative with respect to $x_1$ is denoted as

$$\frac{\partial f(x_1,x_2)}{\partial x_1} = \frac{\partial (x_1^2 + x_2^2)}{\partial x_1}$$

$f(x_1,x_2)$ is the sum of two terms.  The first is a polynomial in $x_1$ for which we use the power rule, while the second is constant when considered a function of $x_1$.  Therefore we have that:

$$\frac{\partial (x_1^2 + x_2^2)}{\partial x_1} = 2x_1$$

Overall we have two partial derivatives for this function:

$$\frac{\partial f(x_1,x_2)}{\partial x_1} = 2x_1 \\
\frac{\partial f(x_1,x_2)}{\partial x_2} = 2x_2$$

Setting the partial derivatives to 0 and solving for $x_1,x_2$ tells us that $(0,0)$ is an extremum point for $f$.  (An extremum point is a maximum, minimum or saddle point).
The condition for determining that an extremum point is indeed a minimum/maximum for multi-variate functions is more complex than for univariate functions, and will not be discussed in this course.

### Exercises

Compute the gradients of the following functions:

$$f(x_1,x_2) = 4x_1^2 x_2 + x_2^3$$

$$g(x_1,x_2) = - x_1 e^{-2(x_1 + x_2)}$$

$$
h(x_1, x_2) = \frac{1}{1 + e^{-(w_1 x_1 + w_2 x_2)}}
$$

### Finding the solution to univariate linear regression

Objective:  find $(w,b)$ that minimize
$$
J( w,b ) = \sum_{i=1}^N (y_i - \hat{y}_i)^2.
$$
For univariate linear regression this takes on the form:
$$
J( w,b ) = \sum_{i=1}^N (y_i - (w x_i + b))^2.
$$

To find the solution, we take the partial derivatives, set them to 0 and solve for $w$ and $b$.
Verify that
$$
\frac{\partial J}{\partial w} = -2 \sum_{i=1}^n x_i(y_i - (w x_i + b))
$$
and
$$
\frac{\partial J}{\partial b} = -2 \sum_{i=1}^n (y_i - (w x_i + b))
$$
Setting to 0 gives us two equations in two unknowns that can be solved for.
Ultimately, we are interested in the multi-variate version, so we will not seek the solution.

```python

```


This notebook contains an example for teaching.

## Introduction

In labor economics an important question is what determines the wage of workers. This is a causal question,
but we could begin to investigate from a predictive perspective.

In the following wage example, $Y$ is the hourly wage of a worker and $X$ is a vector of worker's characteristics, e.g., education, experience, gender. Two main questions here are:

* How to use job-relevant characteristics, such as education and experience, to best predict wages?

* What is the difference in predicted wages between men and women with the same job-relevant characteristics?

In this lab, we focus on the prediction question first.

## Data

The data set we consider is from the March Supplement of the U.S. Current Population Survey, year 2015.  We select white non-hispanic individuals, aged 25 to 64 years, and working more than 35 hours per week during at least 50 weeks of the year. We exclude self-employed workers; individuals living in group quarters; individuals in the military, agricultural or private household sectors;  individuals with inconsistent reports on earnings and employment status; individuals with allocated or missing information in any of the variables used in the analysis; and individuals with hourly wage below $3$.

The variable of interest $Y$ is the hourly wage rate constructed as the ratio of the annual earnings to the total number of hours worked, which is constructed in turn as the product of number of weeks worked and the usual number of hours worked per week. In our analysis, we also focus on single (never married) workers. The final sample is of size $n=5150$.

## Data analysis

We start by loading the data set.

```R
load("wage2015_subsample_inference.Rdata")
dim(data)
```

5150

20

Let's have a look at the structure of the data.

```R
str(data)
```

    'data.frame':	5150 obs. of  20 variables:
     $ wage : num  9.62 48.08 11.06 13.94 28.85 ...
     $ lwage: num  2.26 3.87 2.4 2.63 3.36 ...
     $ sex  : num  1 0 0 1 1 1 1 0 1 1 ...
     $ shs  : num  0 0 0 0 0 0 0 0 0 0 ...
     $ hsg  : num  0 0 1 0 0 0 1 1 1 0 ...
     $ scl  : num  0 0 0 0 0 0 0 0 0 0 ...
     $ clg  : num  1 1 0 0 1 1 0 0 0 1 ...
     $ ad   : num  0 0 0 1 0 0 0 0 0 0 ...
     $ mw   : num  0 0 0 0 0 0 0 0 0 0 ...
     $ so   : num  0 0 0 0 0 0 0 0 0 0 ...
     $ we   : num  0 0 0 0 0 0 0 0 0 0 ...
     $ ne   : num  1 1 1 1 1 1 1 1 1 1 ...
     $ exp1 : num  7 31 18 25 22 1 42 37 31 4 ...
     $ exp2 : num  0.49 9.61 3.24 6.25 4.84 ...
     $ exp3 : num  0.343 29.791 5.832 15.625 10.648 ...
     $ exp4 : num  0.24 92.35 10.5 39.06 23.43 ...
     $ occ  : Factor w/ 369 levels "10","20","40",..: 159 136 269 23 99 86 226 232 184 146 ...
     $ occ2 : Factor w/ 22 levels "1","2","3","4",..: 11 10 19 1 6 5 17 17 13 10 ...
     $ ind  : Factor w/ 236 levels "370","380","390",..: 204 117 12 165 231 176 171 135 210 201 ...
     $ ind2 : Factor w/ 21 levels "2","3","4","5",..: 17 8 3 11 21 13 13 8 18 17 ...

We are constructing the output variable $Y$ and the matrix $Z$ which includes the characteristics of workers that are given in the data.

```R
Y <- log(data$wage)
n <- length(Y)
Z <- data[-which(colnames(data) %in% c("wage","lwage"))]
p <- dim(Z)[2]

cat("Number of observation:", n, '\n')
cat( "Number of raw regressors:", p)
```

    Number of observation: 5150
    Number of raw regressors: 18

```R
p <- dim(Z)[2]
p
```

18

For the outcome variable *wage* and a subset of the raw regressors, we calculate the empirical mean to get familiar with the data.

```R
library(xtable)
Z_subset <- data[which(colnames(data) %in% c("lwage","sex","shs","hsg","scl","clg","ad","mw","so","we","ne","exp1"))]
table <- matrix(0, 12, 1)
table[1:12,1]   <- as.numeric(lapply(Z_subset,mean))
rownames(table) <- c("Log Wage","Sex","Some High School","High School Graduate","Some College","College Graduate", "Advanced Degree","Midwest","South","West","Northeast","Experience")
colnames(table) <- c("Sample mean")
tab<- xtable(table, digits = 2)
tab
```

A xtable: 12 × 1

	Sample mean

	<dbl>

Log Wage	 2.97078670

Sex	 0.44446602

Some High School	 0.02330097

High School Graduate	 0.24388350

Some College	 0.27805825

College Graduate	 0.31766990

Advanced Degree	 0.13708738

Midwest	 0.25961165

South	 0.29650485

West	 0.21611650

Northeast	 0.22776699

Experience	13.76058252

E.g., the share of female workers in our sample is ~44% ($sex=1$ if female).

Alternatively, we can also print the table as latex.

```R
print(tab, type="latex")
```

    % latex table generated in R 3.6.3 by xtable 1.8-4 package
    % Thu Feb 18 19:28:31 2021
    \begin{table}[ht]
    \centering
    \begin{tabular}{rr}
      \hline
     & Sample mean \\
      \hline
    Log Wage & 2.97 \\
      Sex & 0.44 \\
      Some High School & 0.02 \\
      High School Graduate & 0.24 \\
      Some College & 0.28 \\
      College Graduate & 0.32 \\
      Advanced Degree & 0.14 \\
      Midwest & 0.26 \\
      South & 0.30 \\
      West & 0.22 \\
      Northeast & 0.23 \\
      Experience & 13.76 \\
       \hline
    \end{tabular}
    \end{table}

## Prediction Question

Now, we will construct a prediction rule for hourly wage $Y$, which depends linearly on job-relevant characteristics $X$:

\begin{equation}\label{decompose}
Y = \beta'X+ \epsilon.
\end{equation}

Our goals are

* Predict wages  using various characteristics of workers.

* Assess the predictive performance using the (adjusted) sample MSE, the (adjusted) sample $R^2$ and the out-of-sample MSE and $R^2$.

We employ two different specifications for prediction:

1. Basic Model:   $X$ consists of a set of raw regressors (e.g. gender, experience, education indicators,  occupation and industry indicators, regional indicators).

2. Flexible Model:  $X$ consists of all raw regressors from the basic model plus occupation and industry indicators, transformations (e.g., ${exp}^2$ and ${exp}^3$) and additional two-way interactions of polynomial in experience with other regressors. An example of a regressor created through a two-way interaction is *experience* times the indicator of having a *college degree*.

Using the **Flexible Model**, enables us to approximate the real relationship by a
 more complex regression model and therefore to reduce the bias. The **Flexible Model** increases the range of potential shapes of the estimated regression function. In general, flexible models often deliver good prediction accuracy but give models which are harder to interpret.

Now, let us fit both models to our data by running ordinary least squares (ols):

```R
# 1. basic model
basic <- lwage~ (sex + exp1 + shs + hsg+ scl + clg + mw + so + we +occ2+ind2)
regbasic <- lm(basic, data=data)
regbasic # estimated coefficients
cat( "Number of regressors in the basic model:",length(regbasic$coef), '\n') # number of regressors in the Basic Model

```

    Call:
    lm(formula = basic, data = data)

    Coefficients:
    (Intercept)          sex         exp1          shs          hsg          scl
       3.722235    -0.072857     0.008568    -0.592798    -0.504337    -0.411994
            clg           mw           so           we        occ22        occ23
      -0.182216    -0.027541    -0.034454     0.017249    -0.076472    -0.034678
          occ24        occ25        occ26        occ27        occ28        occ29
      -0.096202    -0.187915    -0.414933    -0.045987    -0.377847    -0.215752
         occ210       occ211       occ212       occ213       occ214       occ215
      -0.010623    -0.455834    -0.307589    -0.361440    -0.499495    -0.464482
         occ216       occ217       occ218       occ219       occ220       occ221
      -0.233715    -0.412588    -0.340418    -0.241480    -0.212628    -0.288413
         occ222        ind23        ind24        ind25        ind26        ind27
      -0.422394    -0.116836    -0.244493    -0.273533    -0.249368    -0.139588
          ind28        ind29       ind210       ind211       ind212       ind213
      -0.242948    -0.387485    -0.193851    -0.169063    -0.077436    -0.172604
         ind214       ind215       ind216       ind217       ind218       ind219
      -0.187005    -0.325364    -0.315399    -0.304405    -0.335386    -0.374121
         ind220       ind221       ind222
      -0.551932    -0.316679    -0.118971

    Number of regressors in the basic model: 51

##### Note that the basic model consists of $51$ regressors.

```R
# 2. flexible model
flex <- lwage ~ sex + shs+hsg+scl+clg+occ2+ind2+mw+so+we + (exp1+exp2+exp3+exp4)*(shs+hsg+scl+clg+occ2+ind2+mw+so+we)
regflex <- lm(flex, data=data)
regflex # estimated coefficients
cat( "Number of regressors in the flexible model:",length(regflex$coef)) # number of regressors in the Flexible Model

```

    Call:
    lm(formula = flex, data = data)

    Coefficients:
    (Intercept)          sex          shs          hsg          scl          clg
      3.8602606   -0.0695532   -0.1233089   -0.5289024   -0.2920581   -0.0411641
          occ22        occ23        occ24        occ25        occ26        occ27
      0.1613397    0.2101514    0.0708570   -0.3960076   -0.2310611    0.3147249
          occ28        occ29       occ210       occ211       occ212       occ213
     -0.1875417   -0.3390270    0.0209545   -0.6424177   -0.0674774   -0.2329781
         occ214       occ215       occ216       occ217       occ218       occ219
      0.2562009   -0.1938585   -0.0551256   -0.4156093   -0.4822168   -0.2579412
         occ220       occ221       occ222        ind23        ind24        ind25
     -0.3010203   -0.4271811   -0.8694527   -1.2473654   -0.0948281   -0.5293860
          ind26        ind27        ind28        ind29       ind210       ind211
     -0.6221688   -0.5047497   -0.7295442   -0.8025334   -0.5805840   -0.9852350
         ind212       ind213       ind214       ind215       ind216       ind217
     -0.7375777   -1.0183283   -0.5860174   -0.3801359   -0.5703905   -0.8201843
         ind218       ind219       ind220       ind221       ind222           mw
     -0.7613604   -0.8812815   -0.9099021   -0.7586534   -0.4040775    0.1106834
             so           we         exp1         exp2         exp3         exp4
      0.0224244   -0.0215659   -0.0677247    1.6362944   -0.9154735    0.1429357
       shs:exp1     hsg:exp1     scl:exp1     clg:exp1   occ22:exp1   occ23:exp1
     -0.1919981   -0.0173433   -0.0664505   -0.0550346   -0.0736239   -0.0714859
     occ24:exp1   occ25:exp1   occ26:exp1   occ27:exp1   occ28:exp1   occ29:exp1
     -0.0723997    0.0946732   -0.0348928   -0.2279338   -0.0727459    0.0274143
    occ210:exp1  occ211:exp1  occ212:exp1  occ213:exp1  occ214:exp1  occ215:exp1
      0.0075628    0.1014221   -0.0862744    0.0067149   -0.1369153   -0.0400425
    occ216:exp1  occ217:exp1  occ218:exp1  occ219:exp1  occ220:exp1  occ221:exp1
     -0.0539314    0.0147277    0.1074099    0.0047165    0.0243156    0.0791776
    occ222:exp1   ind23:exp1   ind24:exp1   ind25:exp1   ind26:exp1   ind27:exp1
      0.1093246    0.4758891    0.0147304    0.1256987    0.1540275    0.1029245
     ind28:exp1   ind29:exp1  ind210:exp1  ind211:exp1  ind212:exp1  ind213:exp1
      0.2357669    0.1359079    0.1512578    0.3174885    0.2591089    0.3396094
    ind214:exp1  ind215:exp1  ind216:exp1  ind217:exp1  ind218:exp1  ind219:exp1
      0.1441411   -0.0568181    0.0847295    0.1728867    0.1565399    0.1516103
    ind220:exp1  ind221:exp1  ind222:exp1      mw:exp1      so:exp1      we:exp1
      0.1326629    0.2190905    0.1145814   -0.0279931   -0.0099678    0.0063077
       shs:exp2     hsg:exp2     scl:exp2     clg:exp2   occ22:exp2   occ23:exp2
      1.9005060    0.1171642    0.6217923    0.4096746    0.6632173    0.6415456
     occ24:exp2   occ25:exp2   occ26:exp2   occ27:exp2   occ28:exp2   occ29:exp2
      0.9748422   -0.9778823    0.1050860    3.1407119    0.6710877    0.0231977
    occ210:exp2  occ211:exp2  occ212:exp2  occ213:exp2  occ214:exp2  occ215:exp2
     -0.2692292   -1.0816539    0.8323737   -0.2209813    0.7511163   -0.0326858
    occ216:exp2  occ217:exp2  occ218:exp2  occ219:exp2  occ220:exp2  occ221:exp2
      0.3635814   -0.2659285   -2.5608762   -0.1291756   -0.3323297   -0.9099997
    occ222:exp2   ind23:exp2   ind24:exp2   ind25:exp2   ind26:exp2   ind27:exp2
     -0.8550536   -5.9368948   -1.1053411   -2.0149181   -2.2277748   -1.4648099
     ind28:exp2   ind29:exp2  ind210:exp2  ind211:exp2  ind212:exp2  ind213:exp2
     -2.9479949   -1.7796219   -2.1973300   -3.8776807   -3.1690425   -3.9651983
    ind214:exp2  ind215:exp2  ind216:exp2  ind217:exp2  ind218:exp2  ind219:exp2
     -2.0783289    0.1911692   -1.3265850   -2.2002873   -2.2006232   -1.9308536
    ind220:exp2  ind221:exp2  ind222:exp2      mw:exp2      so:exp2      we:exp2
     -1.9467267   -3.1127363   -1.8578340    0.2005611    0.0544354    0.0012717
       shs:exp3     hsg:exp3     scl:exp3     clg:exp3   occ22:exp3   occ23:exp3
     -0.6721239   -0.0179937   -0.1997877   -0.1025230   -0.2039403   -0.2369620
     occ24:exp3   occ25:exp3   occ26:exp3   occ27:exp3   occ28:exp3   occ29:exp3
     -0.4366958    0.3885298    0.0484737   -1.3949288   -0.2053899   -0.0909660
    occ210:exp3  occ211:exp3  occ212:exp3  occ213:exp3  occ214:exp3  occ215:exp3
      0.1854753    0.3931553   -0.2202559    0.0950356   -0.1443933    0.1477077
    occ216:exp3  occ217:exp3  occ218:exp3  occ219:exp3  occ220:exp3  occ221:exp3
     -0.0378548    0.1510497    1.4084443    0.0923425    0.1806994    0.3779083
    occ222:exp3   ind23:exp3   ind24:exp3   ind25:exp3   ind26:exp3   ind27:exp3
      0.2855058    2.6665808    0.7298431    0.9942250    1.0641428    0.7089089
     ind28:exp3   ind29:exp3  ind210:exp3  ind211:exp3  ind212:exp3  ind213:exp3
      1.2340948    0.8287315    1.0448162    1.6877578    1.3734455    1.6376669
    ind214:exp3  ind215:exp3  ind216:exp3  ind217:exp3  ind218:exp3  ind219:exp3
      1.0162910    0.1879483    0.6889680    1.0085540    1.0605598    0.8959865
    ind220:exp3  ind221:exp3  ind222:exp3      mw:exp3      so:exp3      we:exp3
      0.9768944    1.4415215    0.9687884   -0.0625771   -0.0115842   -0.0124875
       shs:exp4     hsg:exp4     scl:exp4     clg:exp4   occ22:exp4   occ23:exp4
      0.0777418    0.0004913    0.0210760    0.0078695    0.0176389    0.0303057
     occ24:exp4   occ25:exp4   occ26:exp4   occ27:exp4   occ28:exp4   occ29:exp4
      0.0584146   -0.0515181   -0.0170182    0.1905353    0.0196522    0.0190014
    occ210:exp4  occ211:exp4  occ212:exp4  occ213:exp4  occ214:exp4  occ215:exp4
     -0.0333347   -0.0465914    0.0110212   -0.0136895    0.0055582   -0.0327444
    occ216:exp4  occ217:exp4  occ218:exp4  occ219:exp4  occ220:exp4  occ221:exp4
     -0.0089706   -0.0256735   -0.2121372   -0.0169398   -0.0296125   -0.0524577
    occ222:exp4   ind23:exp4   ind24:exp4   ind25:exp4   ind26:exp4   ind27:exp4
     -0.0350646   -0.3851791   -0.1209478   -0.1441045   -0.1526110   -0.1001993
     ind28:exp4   ind29:exp4  ind210:exp4  ind211:exp4  ind212:exp4  ind213:exp4
     -0.1609664   -0.1178080   -0.1482842   -0.2322961   -0.1872911   -0.2155617
    ind214:exp4  ind215:exp4  ind216:exp4  ind217:exp4  ind218:exp4  ind219:exp4
     -0.1483524   -0.0532195   -0.1044336   -0.1427349   -0.1546248   -0.1269592
    ind220:exp4  ind221:exp4  ind222:exp4      mw:exp4      so:exp4      we:exp4
     -0.1468554   -0.2032619   -0.1480951    0.0062439    0.0003145    0.0017685

    Number of regressors in the flexible model: 246

Note that the flexible model consists of $246$ regressors.

Try Lasso next

```R
library(hdm)

lassoreg<- rlasso(flex, data=data)#

sumlasso<- summary(lassoreg)

```

    Call:
    rlasso.formula(formula = flex, data = data)

    Post-Lasso Estimation:  TRUE

    Total number of variables: 245
    Number of selected variables: 24

    Residuals:
         Min       1Q   Median       3Q      Max
    -2.03159 -0.29132 -0.01137  0.28472  3.63651

                Estimate
    (Intercept)    3.137
    sex            0.000
    shs           -0.480
    hsg           -0.404
    scl           -0.306
    clg            0.000
    occ22          0.041
    occ23          0.109
    occ24          0.000
    occ25          0.000
    occ26         -0.327
    occ27          0.000
    occ28         -0.257
    occ29          0.000
    occ210         0.038
    occ211        -0.398
    occ212         0.000
    occ213        -0.131
    occ214        -0.189
    occ215        -0.387
    occ216         0.000
    occ217        -0.280
    occ218         0.000
    occ219         0.000
    occ220         0.000
    occ221         0.000
    occ222        -0.209
    ind23          0.000
    ind24          0.000
    ind25          0.000
    ind26          0.000
    ind27          0.000
    ind28          0.000
    ind29         -0.169
    ind210         0.000
    ind211         0.000
    ind212         0.193
    ind213         0.000
    ind214         0.076
    ind215         0.000
    ind216         0.000
    ind217         0.000
    ind218         0.000
    ind219         0.000
    ind220        -0.225
    ind221         0.000
    ind222         0.140
    mw             0.000
    so             0.000
    we             0.000
    exp1           0.009
    exp2           0.000
    exp3           0.000
    exp4           0.000
    shs:exp1       0.000
    hsg:exp1       0.000
    scl:exp1       0.000
    clg:exp1       0.000
    occ22:exp1     0.000
    occ23:exp1     0.000
    occ24:exp1     0.000
    occ25:exp1     0.000
    occ26:exp1     0.000
    occ27:exp1     0.000
    occ28:exp1     0.000
    occ29:exp1     0.000
    occ210:exp1    0.003
    occ211:exp1    0.000
    occ212:exp1    0.000
    occ213:exp1   -0.011
    occ214:exp1   -0.009
    occ215:exp1    0.000
    occ216:exp1    0.000
    occ217:exp1    0.000
    occ218:exp1    0.000
    occ219:exp1    0.000
    occ220:exp1    0.000
    occ221:exp1    0.000
    occ222:exp1    0.000
    ind23:exp1     0.000
    ind24:exp1     0.000
    ind25:exp1     0.000
    ind26:exp1     0.000
    ind27:exp1     0.000
    ind28:exp1     0.000
    ind29:exp1     0.000
    ind210:exp1    0.000
    ind211:exp1    0.000
    ind212:exp1    0.000
    ind213:exp1    0.000
    ind214:exp1    0.004
    ind215:exp1    0.000
    ind216:exp1    0.000
    ind217:exp1    0.000
    ind218:exp1    0.000
    ind219:exp1    0.000
    ind220:exp1    0.000
    ind221:exp1    0.000
    ind222:exp1    0.000
    mw:exp1        0.000
    so:exp1        0.000
    we:exp1        0.000
    shs:exp2       0.000
    hsg:exp2       0.000
    scl:exp2       0.000
    clg:exp2       0.000
    occ22:exp2     0.000
    occ23:exp2     0.000
    occ24:exp2     0.000
    occ25:exp2     0.000
    occ26:exp2     0.000
    occ27:exp2     0.000
    occ28:exp2     0.000
    occ29:exp2     0.000
    occ210:exp2    0.000
    occ211:exp2    0.000
    occ212:exp2    0.000
    occ213:exp2    0.000
    occ214:exp2    0.000
    occ215:exp2    0.000
    occ216:exp2    0.000
    occ217:exp2    0.000
    occ218:exp2    0.000
    occ219:exp2    0.000
    occ220:exp2    0.000
    occ221:exp2    0.000
    occ222:exp2    0.000
    ind23:exp2     0.000
    ind24:exp2     0.000
    ind25:exp2     0.000
    ind26:exp2     0.000
    ind27:exp2     0.000
    ind28:exp2     0.000
    ind29:exp2     0.000
    ind210:exp2    0.000
    ind211:exp2    0.000
    ind212:exp2    0.000
    ind213:exp2    0.000
    ind214:exp2    0.000
    ind215:exp2    0.000
    ind216:exp2    0.000
    ind217:exp2    0.000
    ind218:exp2    0.000
    ind219:exp2    0.000
    ind220:exp2    0.000
    ind221:exp2    0.000
    ind222:exp2    0.000
    mw:exp2        0.000
    so:exp2        0.000
    we:exp2        0.000
    shs:exp3       0.000
    hsg:exp3       0.000
    scl:exp3       0.000
    clg:exp3       0.000
    occ22:exp3     0.000
    occ23:exp3     0.000
    occ24:exp3     0.000
    occ25:exp3     0.000
    occ26:exp3     0.000
    occ27:exp3     0.000
    occ28:exp3     0.000
    occ29:exp3     0.000
    occ210:exp3    0.000
    occ211:exp3    0.000
    occ212:exp3    0.000
    occ213:exp3    0.000
    occ214:exp3    0.000
    occ215:exp3    0.000
    occ216:exp3    0.000
    occ217:exp3    0.000
    occ218:exp3    0.000
    occ219:exp3    0.000
    occ220:exp3    0.000
    occ221:exp3    0.000
    occ222:exp3    0.000
    ind23:exp3     0.000
    ind24:exp3     0.000
    ind25:exp3     0.000
    ind26:exp3     0.000
    ind27:exp3     0.000
    ind28:exp3     0.000
    ind29:exp3     0.000
    ind210:exp3    0.000
    ind211:exp3    0.000
    ind212:exp3    0.000
    ind213:exp3    0.000
    ind214:exp3    0.000
    ind215:exp3    0.000
    ind216:exp3    0.000
    ind217:exp3    0.000
    ind218:exp3    0.000
    ind219:exp3    0.000
    ind220:exp3    0.000
    ind221:exp3    0.000
    ind222:exp3    0.000
    mw:exp3        0.000
    so:exp3        0.000
    we:exp3        0.000
    shs:exp4       0.000
    hsg:exp4       0.000
    scl:exp4       0.000
    clg:exp4       0.000
    occ22:exp4     0.000
    occ23:exp4     0.000
    occ24:exp4     0.000
    occ25:exp4     0.000
    occ26:exp4     0.000
    occ27:exp4     0.000
    occ28:exp4     0.000
    occ29:exp4     0.000
    occ210:exp4    0.000
    occ211:exp4    0.000
    occ212:exp4    0.000
    occ213:exp4    0.000
    occ214:exp4    0.000
    occ215:exp4    0.000
    occ216:exp4    0.000
    occ217:exp4    0.000
    occ218:exp4    0.000
    occ219:exp4    0.000
    occ220:exp4    0.000
    occ221:exp4    0.000
    occ222:exp4    0.000
    ind23:exp4     0.000
    ind24:exp4     0.000
    ind25:exp4     0.000
    ind26:exp4     0.000
    ind27:exp4     0.000
    ind28:exp4     0.000
    ind29:exp4     0.000
    ind210:exp4    0.000
    ind211:exp4    0.000
    ind212:exp4    0.000
    ind213:exp4    0.000
    ind214:exp4    0.000
    ind215:exp4    0.000
    ind216:exp4    0.000
    ind217:exp4    0.000
    ind218:exp4    0.000
    ind219:exp4    0.000
    ind220:exp4    0.000
    ind221:exp4    0.000
    ind222:exp4    0.000
    mw:exp4        0.000
    so:exp4        0.000
    we:exp4        0.000

    Residual standard error: 0.4847
    Multiple R-squared:  0.2779
    Adjusted R-squared:  0.2745
    Joint significance test:
     the sup score statistic for joint significance test is 89.09 with a p-value of 0.004

Now, we can evaluate the performance of both models based on the (adjusted) $R^2_{sample}$ and the (adjusted) $MSE_{sample}$:

```R
# Assess the predictive performance

sumbasic <- summary(regbasic)
sumflex <- summary(regflex)

#  R-squared
R2.1 <- sumbasic$r.squared
cat("R-squared for the basic model: ", R2.1, "\n")
R2.adj1 <- sumbasic$adj.r.squared
cat("adjusted R-squared for the basic model: ", R2.adj1, "\n")

R2.2 <- sumflex$r.squared
cat("R-squared for the flexible model: ", R2.2, "\n")
R2.adj2 <- sumflex$adj.r.squared
cat("adjusted R-squared for the flexible model: ", R2.adj2, "\n")

R2.L <- sumlasso$r.squared
cat("R-squared for the lasso with flexible model: ", R2.L, "\n")
R2.adjL <- sumlasso$adj.r.squared
cat("adjusted R-squared for the flexible model: ", R2.L, "\n")

# calculating the MSE
MSE1 <- mean(sumbasic$res^2)
cat("MSE for the basic model: ", MSE1, "\n")
p1 <- sumbasic$df[1] # number of regressors
MSE.adj1 <- (n/(n-p1))*MSE1
cat("adjusted MSE for the basic model: ", MSE.adj1, "\n")

MSE2 <-mean(sumflex$res^2)
cat("MSE for the flexible model: ", MSE2, "\n")
p2 <- sumflex$df[1]
MSE.adj2 <- (n/(n-p2))*MSE2
cat("adjusted MSE for the lasso flexible model: ", MSE.adj2, "\n")

MSEL <-mean(sumlasso$res^2)
cat("MSE for the lasso flexible model: ", MSEL, "\n")
pL <- length(sumlasso$coef)
MSE.adjL <- (n/(n-pL))*MSEL
cat("adjusted MSE for the lasso flexible model: ", MSE.adjL, "\n")
```

    R-squared for the basic model:  0.3100465
    adjusted R-squared for the basic model:  0.3032809
    R-squared for the flexible model:  0.3511099
    adjusted R-squared for the flexible model:  0.3186919
    R-squared for the lasso with flexible model:  0.2778653
    adjusted R-squared for the flexible model:  0.2778653
    MSE for the basic model:  0.2244251
    adjusted MSE for the basic model:  0.2266697
    MSE for the flexible model:  0.2110681
    adjusted MSE for the lasso flexible model:  0.221656
    MSE for the lasso flexible model:  0.2348928
    adjusted MSE for the lasso flexible model:  0.2466758

```R
library(xtable)
table <- matrix(0, 3, 5)
table[1,1:5]   <- c(p1,R2.1,MSE1,R2.adj1,MSE.adj1)
table[2,1:5]   <- c(p2,R2.2,MSE2,R2.adj2,MSE.adj2)
table[3,1:5]   <- c(pL,R2.L,MSEL,R2.adjL,MSE.adjL)
colnames(table)<- c("p","$R^2_{sample}$","$MSE_{sample}$","$R^2_{adjusted}$", "$MSE_{adjusted}$")
rownames(table)<- c("basic reg","flexible reg", "lasso flex")
tab<- xtable(table, digits =c(0,0,2,2,2,2))
print(tab,type="latex") # type="latex" for printing table in LaTeX
tab
```

    % latex table generated in R 3.6.3 by xtable 1.8-4 package
    % Thu Feb 18 19:28:38 2021
    \begin{table}[ht]
    \centering
    \begin{tabular}{rrrrrr}
      \hline
     & p & \$R\verb|^|2\_\{sample\}\$ & \$MSE\_\{sample\}\$ & \$R\verb|^|2\_\{adjusted\}\$ & \$MSE\_\{adjusted\}\$ \\
      \hline
    basic reg & 51 & 0.31 & 0.22 & 0.30 & 0.23 \\
      flexible reg & 246 & 0.35 & 0.21 & 0.32 & 0.22 \\
      lasso flex & 246 & 0.28 & 0.23 & 0.27 & 0.25 \\
       \hline
    \end{tabular}
    \end{table}

A xtable: 3 × 5

	p	$R^2_{sample}$	$MSE_{sample}$	$R^2_{adjusted}$	$MSE_{adjusted}$

	<dbl>	<dbl>	<dbl>	<dbl>	<dbl>

basic reg	 51	0.3100465	0.2244251	0.3032809	0.2266697

flexible reg	246	0.3511099	0.2110681	0.3186919	0.2216560

lasso flex	246	0.2778653	0.2348928	0.2744836	0.2466758

Considering all measures above, the flexible model performs slightly better than the basic model.

One procedure to circumvent this issue is to use **data splitting** that is described and applied in the following.

## Data Splitting

Measure the prediction quality of the two models via data splitting:

- Randomly split the data into one training sample and one testing sample. Here we just use a simple method (stratified splitting is a more sophiscticated version of splitting that we can consider).
- Use the training sample for estimating the parameters of the Basic Model and the Flexible Model.
- Use the testing sample for evaluation. Predict the $\mathtt{wage}$  of every observation in the testing sample based on the estimated parameters in the training sample.
- Calculate the Mean Squared Prediction Error $MSE_{test}$ based on the testing sample for both prediction models.

```R
#splitting the data

set.seed(1) # to make the results replicable (generating random numbers)
random <- sample(1:n, floor(n*4/5))
# draw (4/5)*n random numbers from 1 to n without replacing them
train <- data[random,] # training sample
test <- data[-random,] # testing sample
```

```R
# basic model
# estimating the parameters in the training sample
regbasic <- lm(basic, data=train)

# calculating the out-of-sample MSE
trainregbasic <- predict(regbasic, newdata=test)
y.test <- log(test$wage)
MSE.test1 <- sum((y.test-trainregbasic)^2)/length(y.test)
R2.test1<- 1- MSE.test1/var(y.test)

cat("Test MSE for the basic model: ", MSE.test1, " ")

cat("Test R2 for the basic model: ", R2.test1)
```

    Test MSE for the basic model:  0.1971044  Test R2 for the basic model:  0.3279559

In the basic model, the $MSE_{test}$ is quite closed to the $MSE_{sample}$.

```R
# flexible model
# estimating the parameters
options(warn=-1)
regflex <- lm(flex, data=train)

# calculating the out-of-sample MSE
trainregflex<- predict(regflex, newdata=test)
y.test <- log(test$wage)
MSE.test2 <- sum((y.test-trainregflex)^2)/length(y.test)
R2.test2<- 1- MSE.test2/var(y.test)

cat("Test MSE for the flexible model: ", MSE.test2, " ")

cat("Test R2 for the flexible model: ", R2.test2)
```

    Test MSE for the flexible model:  0.2064107  Test R2 for the flexible model:  0.2962252

In the flexible model, the discrepancy between the $MSE_{test}$ and the $MSE_{sample}$ is not large.

It is worth to notice that the $MSE_{test}$ vary across different data splits. Hence, it is a good idea average the out-of-sample MSE over different data splits to get valid results.

Nevertheless, we observe that, based on the out-of-sample $MSE$, the basic model using ols regression performs is about as well (or slightly better) than the flexible model.

Next, let us use lasso regression in the flexible model instead of ols regression. Lasso (*least absolute shrinkage and selection operator*) is a penalized regression method that can be used to reduce the complexity of a regression model when the number of regressors $p$ is relatively large in relation to $n$.

Note that the out-of-sample $MSE$ on the test sample can be computed for any other black-box prediction method as well. Thus, let us finally compare the performance of lasso regression in the flexible model to ols regression.

```R
# flexible model using lasso

# estimating the parameters
library(hdm)
reglasso <- rlasso(flex, data=train, post=FALSE)

# calculating the out-of-sample MSE
trainreglasso<- predict(reglasso, newdata=test)
MSE.lasso <- sum((y.test-trainreglasso)^2)/length(y.test)
R2.lasso<- 1- MSE.lasso/var(y.test)

cat("Test MSE for the lasso on flexible model: ", MSE.lasso, " ")

cat("Test R2 for the lasso flexible model: ", R2.lasso)
```

    Test MSE for the lasso on flexible model:  0.212698  Test R2 for the lasso flexible model:  0.2747882

Finally, let us summarize the results:

```R
table2 <- matrix(0, 3,2)
table2[1,1]   <- MSE.test1
table2[2,1]   <- MSE.test2
table2[3,1]   <- MSE.lasso
table2[1,2]   <- R2.test1
table2[2,2]   <- R2.test2
table2[3,2]   <- R2.lasso

rownames(table2)<- c("basic reg","flexible reg","lasso regression")
colnames(table2)<- c("$MSE_{test}$", "$R^2_{test}$")
tab2 <- xtable(table2, digits =3)
tab2
```

A xtable: 3 × 2

	$MSE_{test}$	$R^2_{test}$

	<dbl>	<dbl>

basic reg	0.1971044	0.3279559

flexible reg	0.2064107	0.2962252

lasso regression	0.2126980	0.2747882

```R
print(tab2,type="latex") # type="latex" for printing table in LaTeX
```

    % latex table generated in R 3.6.3 by xtable 1.8-4 package
    % Thu Feb 18 19:28:42 2021
    \begin{table}[ht]
    \centering
    \begin{tabular}{rrr}
      \hline
     & \$MSE\_\{test\}\$ & \$R\verb|^|2\_\{test\}\$ \\
      \hline
    basic reg & 0.197 & 0.328 \\
      flexible reg & 0.206 & 0.296 \\
      lasso regression & 0.213 & 0.275 \\
       \hline
    \end{tabular}
    \end{table}