# Scenario C - Peak Number Variation (multiple runs)

In this scenario the number of peaks in a generated dataset is varied from low to high,
the rest of the parameters is kept constant (noise level = 1%). The number of peaks expected by the probabilistic model is varied between the low and high peak number.

The model used in the inference of the parameters is formulated as follows:

\begin{equation}
\large y = f(x) = \sum\limits_{m=1}^M \big[A_m \cdot e^{-\frac{(x-\mu_m)^2}{2\cdot\sigma_m^2}}\big] + \epsilon
\end{equation}

This file runs a series of inference runs for a set of generated spectra. New spectra are generated for each run and stored. After running inference, only the summary statistics are stored and the next run is started.

```python
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import pymc3 as pm
import arviz as az

#az.style.use('arviz-darkgrid')

print('Running on PyMC3 v{}'.format(pm.__version__))
```

## Import local modules

```python
import os
import sys
sys.path.append('../../modules')
import datagen as dg
import models as mdl
import results as res
import figures as fig
import settings as cnf
```

## Local configuration

```python
# output for results and images
out_path      = './output_mruns_5x5'
file_basename = out_path + '/scenario_peaks'

# if dir does not exist, create it
if not os.path.exists(out_path):
    os.makedirs(out_path)

conf = {}

# scenario name
conf['scenario'] = 'peak variation'

# initialization method for sampler
conf['init_mode'] = 'adapt_diag'

# probabilistic model (priors)
conf['prior_model'] = 'lognormal'

# provide peak positions to the model as testvalues ('yes'/'no')
conf['peak_info'] = 'yes'

# data mode ('generate'/'preload')
conf['data_mode'] = 'generate'

# number of runs
conf['nruns'] = 4

# number of cores to run sampling chains on
conf['ncores'] = 2

# number of samples per chain
conf['nsamples'] = 2000
```

```python
conf
```

## Save configuration

```python
cnf.save(out_path, conf)
```

# Generate data and plot

```python
# list of wavelengths (x-values)
xval = [i for i in range(200, 400, 2)]

# number of spectra per peak number
nsets  = 4

# number of peaks in the spectrum
peak_numbers = [2,3,4,5,6]

# total number of datasets
tsets = nsets * len(peak_numbers)

# total number of inference runs (per run)
truns = nsets * len(peak_numbers)**2

# generate nruns sets of spectra
for r in range(conf['nruns']):
    print("Generating dataset {0} of {1}".format(r+1,conf['nruns']))

    ldata, lpeaks, lpeakdata  = [], [], []

    # create output directory for data
    out_dir = out_path + '/run_{0:02d}'.format(r+1)
    if not os.path.exists(out_dir):
        os.makedirs(out_dir)

    for pn in peak_numbers:
        for i in range(nsets):
            df, peaks, df_peakinfo = dg.data_generator(xvalues=xval, nsamples=15, npeaks=pn)
            ldata.append(df)
            lpeaks.append(peaks)
            lpeakdata.append(df_peakinfo)

    # save data and peak information to disk
    for i in range(len(ldata)):
        ldata[i].to_csv(out_dir + '/dataset_{0:02d}.csv'.format(i+1), index=False)
        lpeakdata[i].to_csv(out_dir + '/peakdata_{0:02d}.csv'.format(i+1), index=False)
    dg.data_save(out_dir + '/peakinfo.csv', lpeaks)

    # plot datasets
    filen = out_dir + '/scenario_peaks'
    fig.plot_datasets(ldata, lpeaks, dims=(int(tsets/2),2), figure_size=(12,int(tsets*(1.8))),
                                                        savefig='yes', fname=filen)
```

```python
print("total number of multiple runs                    : {0}".format(conf['nruns']))
print("total number of peak numbers                     : {0}".format(len(peak_numbers)))
print("total number of datasets per peak number         : {0}".format(nsets))
print("total number of datasets per model               : {0}".format(tsets))
print("total number of inference runs (per single loop) : {0}".format(truns))
```

# Load data, run inference, visualize, collect results and save

```python
# convert pandas data to numpy arrays
x_val = np.array(xval, dtype='float32')
```

```python
# dataframe to hold multiple run results
res_df = pd.DataFrame()

# run the whole loop of inference, posterior sampling, results collection and saving
for r in range(conf['nruns']):
    print("starting loop {0}/{1}".format(r+1,conf['nruns']))

    models, traces, lmodpeak = [], [], []

    # load datasets from disk
    data_dir = out_path + '/run_{0:02d}'.format(r+1)
    ldata, lpeaks, lpeakdata = dg.data_load(tsets, data_dir)

    # store dataset y-values in list
    cols = ldata[0].columns
    y_val = [ldata[i][cols].values for i in range(len(ldata))]

    # actual inference run number
    inf_run = 1

    for pn in peak_numbers:
        print("running {0}-peak model".format(pn))
        for i in range(len(ldata)):
            if conf['peak_info'] == 'yes':
                # Get the peak numbers from the list. If the actual peak number in the spectrum is
                # lower than what the model is expecting, then expand the list to the expected size,
                # duplicating the existing peak mu values, else truncate the list (taking the peaks
                # with the highest amplitude).
                plist = sorted(lpeaks[i])
                if len(plist) < pn:
                    pl = sorted(np.resize(plist, (1,pn)).flatten())
                else:
                    # sort peak info dataframe on amplitude value
                    l1 = lpeakdata[i].sort_values('amp', ascending=False)
                    # truncate list to expected peak number
                    pl = l1['mu'].values[:pn]

                model_g = mdl.model_pvoigt(xvalues=x_val, observations=y_val[i], npeaks=pn,
                                          mu_peaks=pl, pmodel=conf['prior_model'])
            else:
                model_g = mdl.model_pvoigt(xvalues=x_val, observations=y_val[i], npeaks=pn,
                                          pmodel=conf['prior_model'])
            models.append(model_g)

            with model_g:
                print("({6}:{2}/{3}) running inference on dataset #{0}/{1} [{4}-peak model:{5}-peak spectrum]"
                      .format(i+1,len(ldata),inf_run,truns,pn,len(plist),r+1))
                lmodpeak += [(pn,len(plist))]
                trace_g = pm.sample(conf['nsamples'], init=conf['init_mode'], cores=conf['ncores'])
                traces.append(trace_g)
                inf_run += 1

    # save model figure as image (once)
    if r == 0:
        img = pm.model_to_graphviz(models[0])
        img.render(filename=file_basename + '_model', format='png');

    # posterior predictive traces
    ppc = [pm.sample_posterior_predictive(traces[i], samples=500, model=models[i]) for i in range(len(traces))]

    # collect the results, concat single run result to overall result
    varnames = ['amp', 'mu', 'sigma', 'epsilon']
    lruns = ['{0}'.format(r+1) for i in range(truns)]
    df = res.get_results_summary(varnames, traces, ppc, y_val, epsilon_real=0.05, sets=tsets,
                                 labels=lmodpeak, runlist=lruns)

    res_df = res_df.append(df, ignore_index=True)
```

## Show results and save

```python
res_df
```

```python
# save results to .csv
res_df.to_csv(out_path + '/scenario_peaks_mruns.csv', index=False)
```

```python
cnf.close(out_path)
```


```python
import time
import sympy
sympy.init_printing()
x = sympy.Symbol("x")
```

```python
n = int(time.time())         # changes every second
```

The next cell nbval evalution should fail (in most cases), so we exclude this cell from the comparison with the `#NBVAL_IGNORE_OUTPUT` token:

```python
# NBVAL_IGNORE_OUTPUT
x**n
```

```python

```


# Estimating MI by the proposed  adaptive  label  smoothing classifier-based model (AdapLS)

## Setup

The following cell imports the necessary packages:

```python
import torch
import os
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np
import matplotlib.pyplot as plt
import argparse
from torch.autograd import Variable
import itertools
from tqdm import tqdm
from data.mix_gaussian import MixedGaussian
from data.gaussian import Gaussian
from model.utils import *
from torch.utils.tensorboard import SummaryWriter
```

Use GPU if GPU is available.

```python
cuda = True if torch.cuda.is_available() else False
FloatTensor = torch.cuda.FloatTensor if cuda else torch.FloatTensor
LongTensor = torch.cuda.LongTensor if cuda else torch.LongTensor
torch.set_default_tensor_type(FloatTensor)

name = './results/AdapLS' # filename
chkpt_name = name+'.pt'                         # checkpoint
writer = SummaryWriter('./results/log/AdapLS')
```

Set random seed for reproducibility.

```python
SEED = 0
torch.manual_seed(SEED)
torch.cuda.manual_seed(SEED)
np.random.seed(SEED)
```

Specify the hyper-parameters:

```python
parser = argparse.ArgumentParser()
parser.add_argument("--rho", type=float, default=0.9, help="coefficient of Gaussian")
parser.add_argument("--d", type=int, default=6, help="dimension of X & Y")
parser.add_argument("--sample_size", type=int, default=400, help="sample size")
parser.add_argument("--gamma", type=float, default=0, help="clipping parameter")
parser.add_argument("--b1", type=float, default=0.5, help="adam: decay of first order momentum of gradient")
parser.add_argument("--b2", type=float, default=0.999, help="adam: decay of first order momentum of gradient")
parser.add_argument("--n_iters", type=int, default=60000, help="number of epochs of training")
parser.add_argument("--batch_size", type=int, default=100, help="size of the batches")
parser.add_argument("--lr", type=float, default=1e-4, help="adam: learning rate")
parser.add_argument("--hidden_dim", type=int, default=100, help="Hidden dimension")
parser.add_argument("--c_0_1_ratio", type=float, default=1, help="Ratio of samples with label 0 and samples with label 1 ")
parser.add_argument("--ma_rate", type=float, default=0.1, help="move average rate")
parser.add_argument("--ma_ef", type=float, default=1, help="move average ef")
parser.add_argument("--alpha", type=float, default=1e-4, help="smooth parameter")

opt, unknown = parser.parse_known_args()
# adaptive LS parameters
a, b, c = 0.01, 1e-8, 1-1e-8  # a is \alpha_0, b is \tau and c is 1-\tau in the paper
```

Specify the distribution to generate the data samples:

```python
# Two choices here: 'Gaussian' and 'MixedGaussian
density = 'Gaussian'
```

## Model

Define the function `generate_data` for generating Gaussian or Mixed Gaussian distributions.

- When generating Gaussian distrition, ${X}$ and ${Y}$ are distributed as
\begin{align}
p_{XY} = \prod_{i=1}^d \mathcal{N} \left( \mathbf{0}, \begin{bmatrix}1 & \rho \\ \rho & 1\\ \end{bmatrix} \right)
\end{align}
- When generating Mixed Gaussian distributions, ${X}$ and ${Y}$ are distributed as
 \begin{align}
 p_{XY} = \frac{1}{2} \prod_{i=1}^d \mathcal{N} \left( \mathbf{0}, \begin{bmatrix}1 & \rho \\ \rho & 1\\ \end{bmatrix} \right) + \frac{1}{2} \prod_{i=1}^d \mathcal{N} \left( \mathbf{0}, \begin{bmatrix}1 & {-} \rho \\ {-} \rho & 1\\ \end{bmatrix} \right)
 \end{align}

```python
def generate_data(distribution='Gaussian', rho=0.9):
    # rho is the covariance for generating distributions
    # mu1 and mu2 are means for generating Mixed Gaussian distribution
    mu1 = 0
    mu2 = 0

    # mg is an object of Class Gaussian or MixedGaussian
    if distribution=='Gaussian':
        mg = Gaussian(sample_size=opt.sample_size,rho=rho)
    else:
        mg = MixedGaussian(sample_size=opt.sample_size, mean1=mu1, mean2=mu2,rho1=rho, rho2=-rho)

    # Calculate the ground truth MI between X and Y for (X, Y) from mg
    mi = mg.ground_truth * opt.d

    # Create X, Y for storing generated samples
    X = np.zeros((opt.sample_size,opt.d))
    Y = np.zeros((opt.sample_size,opt.d))

    # Generate samples of random variable X,Y and XY
    for j in range(opt.d):
        # In each iteration, mg.data will generate samples of two dimensions where one dimension for X and another for Y respectively
        data = mg.data
        X[:,j] = data[:,0]
        Y[:,j] = data[:,1]
    X = torch.Tensor(X)
    Y = torch.Tensor(Y)
    XY = torch.cat((X, Y), dim=1)

    return XY, X, Y, mi
```

The function `acti_func` below is for calculating the adaptive smoothing weight.
(Clearer motivation and definition of function `acti_func` will be given in the following parts.)

```python
def acti_func(x, a, b, c):
    # a is \alpha_0, b is \tau and c is 1-\tau in the paper
    alpha = torch.zeros_like(x)
    x_cpu = x.cpu()
    alpha[np.where(x_cpu.cpu()<=b)] = - a*x[np.where(x_cpu<=b)]/b + a
    alpha[np.where((x_cpu>b) & (x_cpu<c))] = 0
    alpha[np.where(x_cpu>=c)] = a*x[np.where(x_cpu>=c)]/(1-c) + a*c/(c-1)
    return alpha
```

Define the neural network.

```python
class Net(nn.Module):
    # Inner class that defines the neural network architecture
    def __init__(self, input_size=2, hidden_size=100, sigma=0.02):
        super().__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.fc2 = nn.Linear(hidden_size, hidden_size)
        self.fc3 = nn.Linear(hidden_size, 1)
        nn.init.normal_(self.fc1.weight, std=sigma)
        nn.init.constant_(self.fc1.bias, 0)
        nn.init.normal_(self.fc2.weight, std=sigma)
        nn.init.constant_(self.fc2.bias, 0)
        nn.init.normal_(self.fc3.weight, std=sigma)
        nn.init.constant_(self.fc3.bias, 0)

    def forward(self, input):
        output = F.elu(self.fc1(input))
        output = F.elu(self.fc2(output))
        output = F.sigmoid(self.fc3(output))

        alpha = acti_func(output, a, b, c)

        return output, alpha
```

The function `_resample` below is for resampling the given data samples for training the neural network.

```python
def _resample(data, batch_size, replace=False):
    # Resample the given data sample.
    index = np.random.choice(
        range(data.shape[0]), size=batch_size, replace=replace)
    batch = data[index]
    return batch
```

The function `mi_estimate` below returns the empirical estimate of MI given by Probabilistic Classifier Method (PCM):
\begin{align}
I(X, Y) = E \left[ \ln r(X, Y) \right],
\end{align}
where
\begin{align}
r(X, Y) := \frac{p_{C}(0)}{p_{C}(1)} \cdot \frac{p_{{C} | {\tilde{X}}{\tilde{Y}}} (1|x,y)}{p_{{C} | {\tilde{X}}{\tilde{Y}}} (0|x,y)},
\end{align}
and $p_{{C} | {\tilde{X}}{\tilde{Y}}}(1|x,y)$
is implemented by a neural network and may be clipped to a certain range before used for calculating MI estimation.

```python
def mi_estimate(model, XY, gamma, c_0_1_ratio):
    # c_0_1_ratio = p_{C}(0) / p_{C}(1)
    pre, _ = model(XY)
    pre = pre.clamp(min=gamma, max=1-gamma) # clip the output pre controled by gamma; gamma = 0 means no clipping
    MI_est = torch.log(c_0_1_ratio*pre/((1-pre).clamp(min=gamma, max=1-gamma))).mean()

    return MI_est
```

The function `smooth_ce_loss` returns the binary cross-entropy loss of the predicted value and the smoothed target.

```python
def smooth_ce_loss(pre_label, true_label, smoothing, num_classes):
    new_labels = (1.0 - smoothing) * true_label + smoothing / num_classes
    return torch.nn.BCELoss()(pre_label, new_labels)
```

The neural network `discriminator` below implements and outputs $p_{{C} | {\tilde{X}}{\tilde{Y}}}(1|x,y)$ before clipping:

```python
discriminator = Net(input_size=opt.d*2, hidden_size=100)
# move NN model GPU if GPU is available
if cuda:
    discriminator.cuda()

# Adam opetimizer
optimizer_D = torch.optim.Adam(discriminator.parameters(), lr=opt.lr, betas=(opt.b1, opt.b2))
```

Load previous results.

```python
load_available = True # set to False to prevent loading previous results
if load_available and os.path.exists(chkpt_name):
    checkpoint = torch.load(
        chkpt_name, map_location='cuda' if torch.cuda.is_available() else 'cpu')
    mi_list = checkpoint['mi_list']
    model_state = checkpoint['model_state']
    discriminator.load_state_dict(model_state)
    print('Previous results loaded.')
else:
    mi_list = [] # storing the mi estimation of each iteration
```

    Previous results loaded.

## Training

We want to obtain $p_{{C} | {\tilde{X}}{\tilde{Y}}}(1|x,y)$ by a neural network $\hat{p}_{\theta}$.

To train the neural network $\hat{p}_{\theta}$, instead of minimizing the cross-entropy of $\hat{p}_{\theta}$ relative to $p_{{C} | {\tilde{X}}{\tilde{Y}}}$, we propose minimizing the cross-entropy of $\hat{p}_{\theta}$ relative to the distribution
\begin{align}
p^{'}_{{C} | {\tilde{X}}{\tilde{Y}}} := (1-\alpha) \cdot p_{{C} | {\tilde{X}}{\tilde{Y}}} + \alpha \cdot p_{U},
\end{align}
where $p_{U} := \frac{1}{2}$ is the uniform over the labels $\{0, 1\}$ (in fact $p_{U}$ can be any distribution over labels), and $\alpha$ is the smoothing weight.

Here, we use adaptive smoothing weight $\alpha$ which will be given by function `acti_func` called in the forward pass of the neural network $\hat{p}_{\theta}$. The adaptive smoothing weight $\alpha$ depends on the input sample and the neural network, and is defined as follows:
\begin{equation*}
\alpha = \left\{
	 \begin{array}{ll}
	 -\frac{\alpha_{0}}{\tau}\hat{p}_{\theta}(1|\tilde{X},\tilde{Y})+\alpha_{0}, &
	 \hat{p}_{\theta}(1|\tilde{X},\tilde{Y})\in[0,\tau], \\
	 0, & \hat{p}_{\theta}(1|\tilde{X},\tilde{Y})\in(\tau,1-\tau), \\
	 \frac{\alpha_{0}}{\tau}\hat{p}_{\theta}(1|\tilde{X},\tilde{Y})+\frac{\tau-1}{\tau}\alpha_{0}, & \hat{p}_{\theta}(1|\tilde{X},\tilde{Y})\in[1-\tau,1],
	 \end{array}
\right.
\end{equation*}
where $\alpha_{0} \in [0, 1]$ is the maximum regularization weight and $\tau \in (0, \frac{1}{2})$ is the threshold to judge whether the classifier is starting to get overconfident.

We train the neural network with adative smoothing weight and $p^{'}_{{C} | {\tilde{X}}{\tilde{Y}}}$, while when we evaluate the MI estimate by the trained neural network, we use the function `mi_estimate` which is based on MI estimation by Probabilistic Classifier Method (PCM) method and uses $p_{{C} | {\tilde{X}}{\tilde{Y}}}$ instead of $p^{'}_{{C} | {\tilde{X}}{\tilde{Y}}}$.

The following cell train the neural network using the data samples.

```python
XY, X, Y, Ground_truth = generate_data(distribution=density, rho=opt.rho)
continue_train = False  # set to True to continue to train
if continue_train:
    for i in range(opt.n_iters):
        data_joint = _resample(XY, batch_size=opt.batch_size)
        X_ref = resample(X, batch_size=opt.batch_size)
        Y_ref = resample(Y, batch_size=opt.batch_size)
        data_margin = torch.cat((X_ref, Y_ref), dim=1)
        valid = Variable(torch.Tensor(opt.batch_size, 1).fill_(1.0), requires_grad=False)
        fake = Variable(torch.Tensor(opt.batch_size, 1).fill_(0.0), requires_grad=False)
        train_data = torch.cat((data_joint, data_margin), dim=0)
        labels = torch.cat((valid, fake), dim=0)
        pred_label, alpha = discriminator(train_data) # alpha is the adaptive label smoothing weight
        # print(alpha)
        c_0_1_ratio = data_margin.shape[0]/data_joint.shape[0] # c_0_1_ratio = p_{C}(0) / p_{C}(1)

        optimizer_D.zero_grad()
        loss = smooth_ce_loss(pred_label, labels, alpha.detach(), 2) # cross-entropy of $\hat{p}_{\theta}$ relative to the distribution p^{'}_{C}
        loss.backward()
        optimizer_D.step()

        with torch.no_grad():
            mi_est = mi_estimate(discriminator, XY, opt.gamma, c_0_1_ratio)
            mi_list.append(mi_est.item())
        writer.add_scalar('mi_list', mi_est.item(), i)
        writer.add_scalar('loss', loss, i)
        if i%500==0:
            print("Iternation: %d, loss: %f, mi_est: %f"%(i, loss.item(), mi_est))
writer.add_graph(discriminator, (XY,))
```

## MI estimates

Use moving average to smooth the estimated MI in each training iteration.

```python
ma_rate = 0.01
mi_copy = mi_list.copy()
for k in range(1,len(mi_list)):
    mi_copy[k] = (1-ma_rate) * mi_copy[k-1] + ma_rate * mi_copy[k]

```

Plot the MI estimation curve against the training iteration, together the ground truth.

```python
plt.plot(mi_copy, label='MI estimate')
plt.axhline(Ground_truth,label='ground truth',linestyle='--',color='red')
for t in range(len(mi_copy)):
    if (mi_copy[t]>.8*Ground_truth):
        plt.axvline(t,label='80% reached',linestyle=':',color='green')
        break
plt.xlabel('number of iterations')
plt.ylabel('MI estimation')
plt.legend()

```

Save the model.

```python
overwrite = True  # set to True to overwrite previously stored results
if overwrite or not os.path.exists(chkpt_name):
    model_state = discriminator.state_dict()
    torch.save({
        'mi_list': mi_list,
        'mi_copy': mi_copy,
        'model_state': model_state
    }, chkpt_name)
    writer.close()
    print('Current results saved.')
```

    Current results saved.

```python

```


# **Bayesian regression**

    Notebook version: 1.0 (Oct 01, 2015)

    Author: Jerónimo Arenas García (jarenas@tsc.uc3m.es)

    Changes: v.1.0 - First version

    Pending changes: * Include regression on the stock data

```python
# Import some libraries that will be necessary for working with data and displaying plots

# To visualize plots in the notebook
%matplotlib inline

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import scipy.io       # To read matlab files
import pylab
```

# 3. Bayesian regression

In the previous session we tackled the problem of fitting the following model using a LS criterion:

$${\hat s}({\bf x}) = f({\bf x}) = {\bf w}^\top {\bf z}$$

where ${\bf z}$ is a vector with components which can be computed directly from the observed variables. Such model is includes a linear regression problem, where ${\bf z} = [1; {\bf x}]$, as well as any other non-linear model as long as it can be expressed as a "linear in the parameters" model.

The LS solution was defined as the one minimizing the square of the residuals over the training set $\{{\bf x}^{(k)}, s^{(k)}\}_{k=1}^K$. As a result, a single parameter vector ${\bf w}^*$ was obtained, and correspondingly a single regression curve.

In this session, rather than trying to obtain the best single model, we will work with a family of models or functions, and model the problem probabilistically, so that we can assign a probability value to each of the possible functions.

## 3.1 Maximum Likelihood estimation of the weights

### 3.1.1 Limitations of the LS approach. The need for assumptions

Consider the same regression task of the previous session. We have a training dataset consisting of 15 points which are given, and depict the regression curves that would be obtained if adding an additional point at a fixed location, depending on the target value of that point:

(You can run this code fragment several times, to check also the changes in the regression curves between executions, and depending also on the location of the training points)

```python
n_points = 15
n_grid = 200
frec = 3
std_n = 0.2
n_val_16 = 5
degree = 12

X_tr = 3 * np.random.random((n_points,1)) - 0.5
S_tr = - np.cos(frec*X_tr) + std_n * np.random.randn(n_points,1)
X_grid = np.linspace(-.5,2.5,n_grid)
S_grid = - np.cos(frec*X_grid) #Noise free for the true model

X_16 = .3 * np.ones((n_val_16,))
S_16 = np.linspace(np.min(S_tr),np.max(S_tr),n_val_16)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(X_tr,S_tr,'b.',markersize=10)
ax.plot(X_16,S_16,'ro',markersize=6)
ax.plot(X_grid,S_grid,'r-',label='True model')

for el in zip(X_16,S_16):
    #Add point to the training set
    X_tr_iter = np.append(X_tr,el[0])
    S_tr_iter = np.append(S_tr,el[1])

    #Obtain LS regression coefficients and evaluate it at X_grid
    w_LS = np.polyfit(X_tr_iter, S_tr_iter, degree)
    S_grid_iter = np.polyval(w_LS,X_grid)
    ax.plot(X_grid,S_grid_iter,'g-')

ax.set_xlim(-.5,2.5)
ax.set_ylim(S_16[0]-2,S_16[-1]+2)
ax.legend(loc='best')
```

   - You can control the degree of the polynomia, and check that when the degree is set to 15 (16 weights) all points will be fitted perfectly
   - It seems obvious that we have not solved the problem ...
        * The regression curves overfit the training data
        * The regression curves change a lot when varying the label of just one pattern

The key missing ingredient is assumptions !!

### Open questions

   - Do we think that all models are equally probable... before we see any data? What does the term model probability mean?

   - Do we need to choose a single "best" model or can we consider several simultaneously?

   - Perhaps our training targets are contaminated with noise. What to do?

We will start postulating a generative model for the training data that includes the presence of noise contaminating the targets, and work on this model to partly answer the other two questions.

### 3.1.2 Generative model

Denoting by $f({\bf x}) = {{\bf w}}^\top {\bf z}$ the true function that we would like to obtain, we could assume that the observations in the training set are obtained as noisy values of the output of such function, i.e.,

$$s^{(k)} = f({\bf x}^{(k)}) + \varepsilon^{(k)}$$

We will further characterize the noise values as i.i.d. and normally distributed, with mean zero, and variance $\sigma_\varepsilon^2$, i.e.,

$$\varepsilon \sim {\cal N}\left(0, \sigma_\varepsilon^2\right)$$

### 3.1.3 The maximum likelihood solution

   - Joint distribution of the noise samples, ${\pmb \varepsilon} = \left[\varepsilon^{(1)}, \dots, \varepsilon^{(K)}\right]^\top$:

   $${\pmb \varepsilon} \sim \left( {\bf 0}, \sigma_{\varepsilon}^2 {\bf I}\right) \;\;\; p({\pmb \varepsilon}) = \left(\frac{1}{\sqrt{2\pi \sigma_{\varepsilon}^2}}\right)^K \exp\left(- \frac{{\pmb \varepsilon}^\top {\pmb \varepsilon}}{2 \sigma_{\varepsilon}^2}\right)$$

   - Denoting ${\bf s} = \left[s^{(1)}, \dots, s^{(K)} \right]^\top$ and ${\bf f} = \left[ f({\bf x}^{(1)}), \dots,f({\bf x}^{(K)})\right]^\top$, we have

   $${\bf s} = {\bf f} + {\pmb \varepsilon}$$

   - Conditioning on the values of the target function, ${\bf f}$, the pdf of the available targets is obtained as a shifted version of the distribution of the noise. More precisely:

   \begin{align}p({\bf s}|{\bf f}) & = \left(\frac{1}{\sqrt{2\pi \sigma_{\varepsilon}^2}}\right)^K \exp\left(- \frac{\|{\bf s} - {\bf f}\|^2}{2 \sigma_{\varepsilon}^2}\right) \end{align}

   - For the particular parametric selection of $f({\bf x})$, ${\bf f} = {\bf Z} {\bf w}$, conditioning on ${\bf f}$ is equivalent to conditioning on ${\bf w}$, so that:

   $$p({\bf s}|{\bf f}) = p({\bf s}|{\bf w}) = \left(\frac{1}{\sqrt{2\pi \sigma_{\varepsilon}^2}}\right)^K \exp\left(- \frac{\|{\bf s} - {\bf Z}{\bf w}\|^2}{2 \sigma_{\varepsilon}^2}\right)$$

   - The previous expression represents the probability of the observed targets given the weights, and is also known as the likelihood of the weights for a particular training set.

The maximum likelihood solution is then given by:

$${\bf w}_{ML} = \arg \max_{\bf w} p({\bf s}|{\bf w}) = \arg \min_{\bf w} \|{\bf s} - {\bf Z}{\bf w}\|^2$$

### 3.1.4 Multiple explanations of the data

With an additive Gaussian independent noise model, the maximum likelihood and the least squares solutions are the same. We have not improved much ...

However, we have already formulated the problem in a probabilistic way. This opens the door to reasoning in terms of a set of possible explanations, not just one. We believe more than one of our models could have generated the data.

   - We do not believe all models are equally likely to have generated the data
   - We may believe that a simpler model is more likely than a complex one

## 3.2 Bayesian Inference

### 3.2.1 Posterior distribution of weights

   - If we express our a priori belief of models using a prior distribution $p({\bf f})$, then we can infer the a posteriori distribution using Bayes' rule:

   $$p({\bf f}|{\bf s}) = \frac{p({\bf s}|{\bf f})~p({\bf f})}{p({\bf s})}$$

   In the previous expression:
      * $p({\bf s}|{\bf f})$: is the likelihood function
      * $p({\bf f})$: is the prior distribution of the models (assumptions are needed here)
      * $p({\bf s})$: is the marginal distribution of the observed data, which could be obtained integrating the numerator over all possible models. However, we normally do not need to explicitly compute $p({\bf s})$

   - For the parametric model ${\bf f} = {\bf Z} {\bf w}$, the previous expressions become:

   $$p({\bf w}|{\bf s}) = \frac{p({\bf s}|{\bf w})~p({\bf w})}{p({\bf s})}$$

   Where:
      * $p({\bf s}|{\bf w})$: is the likelihood function
      * $p({\bf w})$: is the prior distribution of the weights (assumptions are needed here)
      * $p({\bf s})$: is the marginal distribution of the observed data, which could be obtained integrating

### 3.2.2 Maximum likelihood vs Bayesian Inference.  Making predictions

   - Following a ML approach, we retain a single model, ${\bf w}_{ML} = \arg \max_{\bf w} p({\bf s}|{\bf w})$. Then, the predictive distribution of the target value for a new point would be obtained as:

   $$p({s^*}|{\bf w}_{ML},{\bf x}^*) $$

   For the generative model of Section 3.1.2 (additive i.i.d. Gaussian noise), this distribution is:

   $$p({s^*}|{\bf w}_{ML},{\bf x}^*) = \frac{1}{\sqrt{2\pi\sigma_\varepsilon^2}} \exp \left(-\frac{\left(s^* - {\bf w}_{ML}^\top {\bf z}^*\right)^2}{2 \sigma_\varepsilon^2} \right)$$

      * The mean of $s^*$ is just the same as the prediction of the LS model, and the same uncertainty is assumed independently of the observation vector (i.e., the variance of the noise of the model).

      * If a single value is to be kept, we would probably keep the mean of the distribution, which is equivalent to the LS prediction.

   - Using Bayesian inference, we retain all models. Then, the inference of the value $s^* = s({\bf x}^*)$ is carried out by mixing all models, according to the weights given by the posterior distribution.

   \begin{align}p({s^*}|{\bf x}^*,{\bf s}) & = \int p({s^*},{\bf w}~|~{\bf x}^*,{\bf s}) d{\bf w} \\
                         & = \int p({s^*}~|~{\bf w},{\bf x}^*,{\bf s}) p({\bf w}~|~{\bf x}^*,{\bf s}) d{\bf w} \\
                         & = \int p({s^*}~|~{\bf w},{\bf x}^*) p({\bf w}~|~{\bf s}) d{\bf w}\end{align}

   where:

      * $p({s^*}|{\bf w},{\bf x}^*) = \displaystyle\frac{1}{\sqrt{2\pi\sigma_\varepsilon^2}} \exp \left(-\frac{\left(s^* - {\bf w}^\top {\bf z}^*\right)^2}{2 \sigma_\varepsilon^2} \right)$
      * $p({\bf w}~|~{\bf s})$: Is the posterior distribution of the weights, that can be computed using Bayes' Theorem.

### 3.2.3 Example: Selecting a Gaussian prior for the weights

#### Prior distribution of the weights

In this section, we consider a particular example in which we assume the following prior for the weights:

$${\bf w} \sim {\cal N}\left({\bf 0},{\pmb \Sigma}_{p} \right)$$

The following figure shows functions which are generated by drawing points from this distribution

```python
n_points = 15
n_grid = 200
frec = 3
std_n = 0.2
degree = 12
nplots = 6

#Prior distribution parameters
sigma_eps = 0.1
mean_w = np.zeros((degree+1,))
sigma_w = 0.3
var_w = sigma_w * np.eye(degree+1)

X_tr = 3 * np.random.random((n_points,1)) - 0.5
S_tr = - np.cos(frec*X_tr) + std_n * np.random.randn(n_points,1)
X_grid = np.linspace(-.5,2.5,n_grid)
S_grid = - np.cos(frec*X_grid) #Noise free for the true model

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(X_tr,S_tr,'b.',markersize=10)

for k in range(nplots):

    #Draw weigths fromt the prior distribution
    w_iter = np.random.multivariate_normal(mean_w,var_w)
    S_grid_iter = np.polyval(w_iter,X_grid)
    ax.plot(X_grid,S_grid_iter,'g-')

ax.set_xlim(-.5,2.5)
ax.set_ylim(S_16[0]-2,S_16[-1]+2)

```

#### Likelihood of the weights

According to the generative model, ${\bf s} = {\bf Z}{\bf w} + {\pmb \varepsilon}$

$${\bf s}~|~{\bf w} \sim {\cal N}\left({\bf Z}{\bf w},\sigma_\varepsilon^2 {\bf I} \right)$$

#### Posterior distribution of the weights

$$p({\bf w}|{\bf s}) = \frac{p({\bf s}|{\bf w})~p({\bf w})}{p({\bf s})}$$

Since both $p({\bf s}|{\bf w})$ and $p({\bf w})$ follow a Gaussian distribution, we know also that the joint distribution and the posterior distribution of ${\bf w}$ given ${\bf s}$ are also Gaussian. Therefore,

$${\bf w}~|~{\bf s} \sim {\cal N}\left({\bar{\bf w}} , {\pmb\Sigma}_{\bf w}\right)$$

where the mean and the covariance matrix of the distribution are to be determined.

Exercise:

Show that the posterior mean and posterior covariance matrix of ${\bf w}$ given ${\bf s}$ are:

$${\pmb\Sigma}_{\bf w} = \left[\frac{1}{\sigma_\varepsilon^2} {\bf Z}^{\top}{\bf Z} + {\pmb \Sigma}_p^{-1}\right]^{-1}$$

$${\bar{\bf w}} = {\sigma_\varepsilon^{-2}} {\pmb\Sigma}_{\bf w} {\bf Z}^\top {\bf s}$$

The following fragment of code draws random vectors from $p({\bf w}|{\bf s})$, and plots the corresponding regression curve along with the training points. Compare these curves with those extracted from the prior distribution of ${\bf w}$.

```python
n_points = 15
n_grid = 200
frec = 3
std_n = 0.2
degree = 12
nplots = 6

#Prior distribution parameters
sigma_eps = 0.1
mean_w = np.zeros((degree+1,))
sigma_p = .3 * np.eye(degree+1)

X_tr = 3 * np.random.random((n_points,1)) - 0.5
S_tr = - np.cos(frec*X_tr) + std_n * np.random.randn(n_points,1)
X_grid = np.linspace(-.5,2.5,n_grid)
S_grid = - np.cos(frec*X_grid) #Noise free for the true model

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(X_tr,S_tr,'b.',markersize=10)

#Compute matrix with training input data for the polynomial model
Z = []
for x_val in X_tr.tolist():
    Z.append([x_val[0]**k for k in range(degree+1)])
Z=np.asmatrix(Z)

#Compute posterior distribution parameters
Sigma_w = np.linalg.inv(np.dot(Z.T,Z)/(sigma_eps**2) + np.linalg.inv(sigma_p))
posterior_mean = Sigma_w.dot(Z.T).dot(S_tr)/(sigma_eps**2)
posterior_mean = np.array(posterior_mean).flatten()

for k in range(nplots):

    #Draw weights from the posterior distribution
    w_iter = np.random.multivariate_normal(posterior_mean,Sigma_w)
    #Note that polyval assumes the first element of weight vector is the coefficient of
    #the highest degree term. Thus, we need to reverse w_iter
    S_grid_iter = np.polyval(w_iter[::-1],X_grid)
    ax.plot(X_grid,S_grid_iter,'g-')

#We plot also the least square solution
w_LS = np.polyfit(X_tr.flatten(), S_tr.flatten(), degree)
S_grid_iter = np.polyval(w_LS,X_grid)
ax.plot(X_grid,S_grid_iter,'m-',label='LS regression')

ax.set_xlim(-.5,2.5)
ax.set_ylim(S_16[0]-2,S_16[-1]+2)
ax.legend(loc='best')
```

#### Posterior distribution of the target

   - Since $f^* = f({\bf x}^*) = [{\bf x}^*]^\top {bf w}$, $f^*$ is also a Gaussian variable whose posterior mean and variance can be calculated as follows:

   $$\mathbb{E}\{{{\bf x}^*}^\top {\bf w}~|~{\bf s}, {\bf x}^*\} = {{\bf x}^*}^\top \mathbb{E}\{{\bf w}|{\bf s}\} = {\sigma_\varepsilon^{-2}} {{\bf x}^*}^\top  {\pmb\Sigma}_{\bf w} {\bf Z}^\top {\bf s}$$

   $$\text{Cov}\left[{{\bf x}^*}^\top {\bf w}~|~{\bf s}, {\bf x}^*\right] = {{\bf x}^*}^\top \text{Cov}\left[{\bf w}~|~{\bf s}\right] {{\bf x}^*} = {{\bf x}^*}^\top {\pmb \Sigma}_{\bf w} {{\bf x}^*}$$

   - Therefore, $f^*~|~{\bf s}, {\bf x}^* \sim {\cal N}\left({\sigma_\varepsilon^{-2}} {{\bf x}^*}^\top  {\pmb\Sigma}_{\bf w} {\bf Z}^\top {\bf s}, {{\bf x}^*}^\top {\pmb \Sigma}_{\bf w} {{\bf x}^*} \right)$

   - Finally, for $s^* = f^* + \varepsilon^*$, the posterior distribution is  $s^*~|~{\bf s}, {\bf x}^* \sim {\cal N}\left({\sigma_\varepsilon^{-2}} {{\bf x}^*}^\top  {\pmb\Sigma}_{\bf w} {\bf Z}^\top {\bf s}, {{\bf x}^*}^\top {\pmb \Sigma}_{\bf w} {{\bf x}^*} + \sigma_\varepsilon^2\right)$

```python
n_points = 15
n_grid = 200
frec = 3
std_n = 0.2
degree = 12
nplots = 6

#Prior distribution parameters
sigma_eps = 0.1
mean_w = np.zeros((degree+1,))
sigma_p = .5 * np.eye(degree+1)

X_tr = 3 * np.random.random((n_points,1)) - 0.5
S_tr = - np.cos(frec*X_tr) + std_n * np.random.randn(n_points,1)
X_grid = np.linspace(-.5,2.5,n_grid)
S_grid = - np.cos(frec*X_grid) #Noise free for the true model

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(X_tr,S_tr,'b.',markersize=10)

#Compute matrix with training input data for the polynomial model
Z = []
for x_val in X_tr.tolist():
    Z.append([x_val[0]**k for k in range(degree+1)])
Z=np.asmatrix(Z)

#Compute posterior distribution parameters
Sigma_w = np.linalg.inv(np.dot(Z.T,Z)/(sigma_eps**2) + np.linalg.inv(sigma_p))
posterior_mean = Sigma_w.dot(Z.T).dot(S_tr)/(sigma_eps**2)
posterior_mean = np.array(posterior_mean).flatten()

#Plot the posterior mean
#Note that polyval assumes the first element of weight vector is the coefficient of
#the highest degree term. Thus, we need to reverse w_iter
S_grid_iter = np.polyval(posterior_mean[::-1],X_grid)
ax.plot(X_grid,S_grid_iter,'g-',label='Predictive mean, BI')

#Plot confidence intervals for the Bayesian Inference
std_x = []
for el in X_grid:
    x_ast = np.array([el**k for k in range(degree+1)])
    std_x.append(np.sqrt(x_ast.dot(Sigma_w).dot(x_ast)[0,0]))
std_x = np.array(std_x)
plt.fill_between(X_grid, S_grid_iter-std_x, S_grid_iter+std_x,
    alpha=0.2, edgecolor='#1B2ACC', facecolor='#089FFF',
    linewidth=4, linestyle='dashdot', antialiased=True)

#We plot also the least square solution
w_LS = np.polyfit(X_tr.flatten(), S_tr.flatten(), degree)
S_grid_iter = np.polyval(w_LS,X_grid)
ax.plot(X_grid,S_grid_iter,'m-',label='LS regression')

ax.set_xlim(-.5,2.5)
ax.set_ylim(S_16[0]-2,S_16[-1]+2)
ax.legend(loc='best')
```

Not only do we obtain a better predictive model, but we also have confidence intervals (error bars) for the predictions.

```python

```


## Reconstructing Horndeski theories via the Gaussian Process: $H(z)$ reconstruction

This is part 1 of a two-part notebook on using the Gaussian process (GP) to reconstruct Horndeski theories ([2105.12970](https://arxiv.org/abs/2105.12970)). For this notebook, we reconstruct the Hubble function $H(z)$ given the combined Pantheon/MCT, cosmic chronometers (CC), and baryon acoustic oscillations (BAO) data. The output will be directly used in part 2 on model building.

References to the data and python packages used in this work can be found at end of this notebook.

### 0. Datasets: Pantheon/MCT, CC, BAO

To start, we import the datasets (Pantheon/MCT, CC, BAO) which will be used for the reconstruction.

```python
%matplotlib inline
import numpy as np
from numpy import loadtxt, savetxt
from scipy.constants import c
from matplotlib import pyplot as plt

c_kms = c/1000 # speed of light in km/s

# load pantheon + mct H(z) data
loc_sn = 'pantheon_mct.txt'
loc_sn_corr = 'pantheon_mct_corr.txt'
sn_data = loadtxt(loc_sn)
sn_corr = loadtxt(loc_sn_corr)

# setup snia observations
z_sn = sn_data[:, 0]
Ez_sn = sn_data[:, 1]
sigEz_sn_stat = sn_data[:, 2]

# construct snia cov matrix
covEz_sn_corr = np.diag(sigEz_sn_stat)@ \
                sn_corr@np.diag(sigEz_sn_stat)

# load pantheon compressed m(z) data
loc_lcparam = 'lcparam_DS17f.txt'
loc_lcparam_sys = 'sys_DS17f.txt'
lcparam = loadtxt(loc_lcparam, usecols = (1, 4, 5))
lcparam_sys = loadtxt(loc_lcparam_sys, skiprows = 1)

# setup pantheon samples
z_ps = lcparam[:, 0]
mz_ps = lcparam[:, 1]
sigmz_ps = lcparam[:, 2]

# pantheon samples systematics
covmz_ps_sys = lcparam_sys.reshape(40, 40)
covmz_ps_tot = covmz_ps_sys + np.diag(sigmz_ps**2)

# load cc dataset
loc_cc = 'cc_data.txt'
cc_data = loadtxt(loc_cc)

# setup cc observations
z_cc = cc_data[:, 0]
Hz_cc = cc_data[:, 1]
sigHz_cc = cc_data[:, 2]

# load bao's
loc_bao = 'bao_data.txt'
bao_data = loadtxt(loc_bao)

z_bao = bao_data[:, 0]
dmrs_bao = bao_data[:, 1] # dM/rs, rs = sound horizon
sigdmrs_bao = bao_data[:, 2]
dhrs_bao = bao_data[:, 3] # dH/rs
sigdhrs_bao = bao_data[:, 4]
```

The different datasets are visualized below.

```python
# pantheon/mct
plt.errorbar(z_sn, Ez_sn,
             yerr = np.sqrt(np.diag(covEz_sn_corr)),
             fmt = 'k^', markersize = 4,
             ecolor = 'red', elinewidth = 2, capsize = 2)
plt.xlabel('$z$')
plt.ylabel('$E(z)$')
plt.show()

# pantheon samples
plt.errorbar(np.log(z_ps), mz_ps,
             yerr = np.sqrt(np.diag(covmz_ps_tot)),
             fmt = 'k^', markersize = 4,
             ecolor = 'red', elinewidth = 2, capsize = 2)
plt.xlabel('$\ln(z)$')
plt.ylabel('$m(z)$')
plt.show()

# cosmic chronometers
plt.errorbar(z_cc, Hz_cc, yerr = sigHz_cc,
             fmt = 'bo', markersize = 2,
             ecolor = 'red', elinewidth = 2, capsize = 2)
plt.xlabel('$z$')
plt.ylabel('$H(z)$')
plt.show()

# bao
fig, ax = plt.subplots(1,2, figsize = (15,4))

ax[0].errorbar(z_bao, dmrs_bao, yerr = sigdmrs_bao,
               fmt = 'k*', markersize = 4,
               ecolor = 'red', elinewidth = 2, capsize = 2)
ax[0].set_xlabel('$z$')
ax[0].set_ylabel('$dM(z)/r_s$')

ax[1].errorbar(z_bao, dhrs_bao, yerr = sigdhrs_bao,
               fmt = 'k*', markersize = 2,
               ecolor = 'red', elinewidth = 2, capsize = 2)
ax[1].set_xlabel('$z$')
ax[1].set_ylabel('$dH(z)/r_s$')
plt.show()
```

### 1. Reconstructing $H(z)$

In this section, we shall use the combined datasets above to reconstruct the Hubble function $H(z)$. We shall also further complement the above datasets with an $H_0$ prior. But, first, let us import the GP function and the RBF kernel.

```python
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import (RBF, ConstantKernel)
```

In what follows, we will use the RBF kernel, also commonly known as the squared exponential kernel.

```python
kernels = {"SquaredExponential": ConstantKernel()*RBF()}
```

Select any one of the kernels...

```python
kern_name = "SquaredExponential"
kernel = kernels[kern_name]
```

The $H_0$ priors we shall consider for the reconstruction are given below.

```python
H0_priors = {'R19': {'ave': 74.03, 'std': 1.42},
             'TRGB': {'ave': 69.8, 'std': 1.9},
             'P18': {'ave': 67.4, 'std': 0.5}}
```

The Gaussian process is now performed in the next few lines. We begin with the CC dataset appended with the prior $H_0$.

```python
# set GP rec parameters
z_min = 1e-5
z_max = 2
n_div = 50

for H0_prior in H0_priors:

    H0_ave = H0_priors[H0_prior]['ave']
    H0_std = H0_priors[H0_prior]['std']

    z_cc_prior = np.append(np.array([z_min]), z_cc)
    Hz_cc_prior = np.append(np.array([H0_ave]), Hz_cc)
    sigHz_cc_prior = np.append(np.array([H0_std]), sigHz_cc)

    gp = GaussianProcessRegressor(kernel = kernel,
                                  alpha = sigHz_cc_prior**2,
                                  n_restarts_optimizer = 10)
    gp.fit(z_cc_prior.reshape(-1, 1), Hz_cc_prior)

    print("\nLearned kernel: %s" % gp.kernel_)
    print("Log-marginal-likelihood: %.3f"
          % gp.log_marginal_likelihood(gp.kernel_.theta))

    z_cc_rec = np.linspace(z_min, z_max, n_div)
    Hz_cc_rec, sigHz_cc_rec = gp.predict(z_cc_rec.reshape(-1, 1),
                                         return_std=True)

    fig = plt.figure()
    plt.errorbar(z_cc_prior, Hz_cc_prior, yerr = sigHz_cc_prior,
                 label = 'CC', fmt = 'kx', markersize = 4,
                 ecolor = 'red', elinewidth = 2, capsize = 2)
    plt.plot(z_cc_rec, Hz_cc_rec, 'b-', label = 'mean')
    plt.fill_between(z_cc_rec, Hz_cc_rec - sigHz_cc_rec,
                     Hz_cc_rec + sigHz_cc_rec,
                     alpha = .5, facecolor = 'b', edgecolor='None',
                     label= r'$1\sigma$')
    plt.fill_between(z_cc_rec, Hz_cc_rec - 2*sigHz_cc_rec,
                     Hz_cc_rec + 2*sigHz_cc_rec,
                     alpha = .2, facecolor = 'b', edgecolor='None',
                     label= r'$2\sigma$')
    plt.title(H0_prior)
    plt.xlabel('$z$')
    plt.ylabel('$H(z)$')
    plt.legend(loc = 'lower right', prop = {'size': 9.5})
    plt.xlim(min(z_cc_rec),
             max(z_cc_rec))
    plt.show()
```

The Pantheon/MCT dataset comes with a correlation matrix. A consistent usage of the lone Pantheon/MCT dataset therefore requires drawing many samples over the corresponding mean and covariance matrix of $E(z)$. However, regarding it as a minority in the overall dataset (CC + Pantheon/MCT + BAO dataset), we will just consider its mean for simplicity.

Moving on, the BAO $H(z)$ function can be obtained without relying on the sound horizon $r_s$ by using the Pantheon $m(z)$ samples. In particular, we take the ratio of $dM(z)/dH(z)$ and use the distance duality relation (DDR), $d_L(z) = d_A(z)(1 + z)^2$ where $d_L(z)$ and $d_A(z)$ are the luminosity distance and angular diameter distance, respectively, to obtain
\begin{equation}
H(z) = \dfrac{ dM(z) }{ dH(z) } 10^{(25 + M - m(z))/5} \ [ c / \text{Mpc} ].
\end{equation}
This requires a calibrated absolute magnitude $M$ which we determine this by sampling over the $z < 0.1$ data points in the compressed Pantheon samples. This assumes that for such very low redshifts the cosmological model-dependence should drop out and so the $\Lambda$CDM model can be taken to be a reasonably fair assumption. Furthermore, in using the DDR, we are also restricting our attention to spatially-flat and isotropic cosmologies.

The $\Lambda$CDM model is prepared below for the sampling. We import ``cobaya`` and ``getdist`` for the sampling and its statistical analysis.

```python
from scipy.integrate import quad
from cobaya.run import run

def E_lcdm(a, om0):
    '''returns the rescaled Hubble function of lcdm

    input:
    a = scale factor
    om0 = matter density
    ol0 = dark energy density
    '''

    ol0 = 1 - om0

    return np.sqrt((om0/(a**3)) + ol0)

def E_inv_lcdm(z, om0):

    a = 1/(z + 1)

    return 1/E_lcdm(a, om0)

def dl_lcdm(z, om0):

    '''returns dL = luminosity distance*H0/c'''

    rz = quad(E_inv_lcdm, 0, z, args = (om0))[0]

    return (1 + z)*rz

def m_lcdm(z, H0, om0, M):
    '''returns the apparent magnitude m'''

    return 5*np.log10(100000*(c_kms/H0)*dl_lcdm(z, om0)) + M

# prepare the log-likelihood, consider only z < 0.1 points
nr = 29
valid = [r for r in range(covmz_ps_tot.shape[0]) if r not in
         np.arange(len(z_ps) - nr, len(z_ps))]
covmz_ps_red = covmz_ps_tot[valid][:, valid] # covariance matrix
invcovmz_ps_red = np.linalg.inv(covmz_ps_red) # inverse C

def loglike_lcdm(H0, om0, M):
    '''returns the log-likelihood given (H0, om0, M)'''

    m_th = np.array([m_lcdm(z, H0, om0, M)
                     for z in z_ps[:11]])

    if om0 > 0:
        noise = mz_ps[:11] - m_th
        return -0.5*noise.T@invcovmz_ps_red@noise

    else: # rule out negative h and om0
        return -np.inf
```

Now, we sample over the parameter space $(H_0, \Omega_{m0}, M)$ using ``cobaya``.

*The next line will run for about five minutes*. (Optionally) skip and proceed to next line if the output has already been generated in the folder *chains*.

```python
M_calib = {}

for H0_prior in H0_priors:

    info_lcdm = {"likelihood": {"loglikeSNIa": loglike_lcdm}}

    info_lcdm["params"] = {"H0": {"prior": {"min": 0, "max": 100},
                                  "ref": {"min": 60, "max": 80},
                                  "proposal": 0.5,
                                  "latex": r"H_0"},
                           "om0": {"prior": {"min": -0.2, "max": 1},
                                   "ref": {"min": 0.2, "max":0.4},
                                   "proposal": 0.01,
                                   "latex": r"\Omega_{m0}"},
                           "M": {"prior": {"min": -22, "max": -16},
                                 "ref": {"min": -20, "max": -19},
                                 "proposal": 0.01,
                                 "latex": r"M"}}

    def ol0_lcdm(om0):
        '''returns the dark energy density parameter'''

        return 1 - om0

    info_lcdm["params"]["ol0"] = {"derived": ol0_lcdm,
                                  "latex": r"\Omega_{\Lambda}"}

    def H0_prior_loglike(H0):
        '''H0_prior: assume Gaussian'''

        H0_ave = H0_priors[H0_prior]['ave']
        H0_std = H0_priors[H0_prior]['std']

        return -0.5*((H0 - H0_ave)/H0_std)**2

    info_lcdm["prior"] = {"H0_prior": H0_prior_loglike}

    info_lcdm["sampler"] = {"mcmc": {"Rminus1_stop": 0.001,
                                     "max_tries": 1000}}

    # save mcmc chain
    info_lcdm["output"] = 'chains/M_calib_' + H0_prior

    # overwrite chain, if it exists
    info_lcdm["force"] = True

    # run MCMC
    updated_info_lcdm, sampler_lcdm = run(info_lcdm)
```

*Note*: Since the computation only included the $z < 0.1$ data points (the first 11 of 40 points in the compressed Pantheon samples), then it is to be expected that there is much less information available about the geometry of the expansion, i.e., large confidence intervals for the $\Omega$'s. Nonetheless, the justification of this methodology is that absolute magnitude $M$ (a nuisance parameter) has been constrained to subpercent precision since this depends less on the dynamics of cosmic expansion.

The calibrated $M$ for each of the $H_0$ priors is shown below.

```python
from getdist.mcsamples import loadMCSamples
import getdist.plots as gdplt
import os # requires *full path*

M_calib = {}
gdsamples_per_H0_prior = {}

for H0_prior in H0_priors:

    folder_file = 'chains/M_calib_' + H0_prior
    gdsamples = loadMCSamples(os.path.abspath(folder_file))

    # get statistics
    stats = gdsamples.getMargeStats()

    M_ave = stats.parWithName("M").mean
    M_std = stats.parWithName("M").err

    M_calib[H0_prior] = {'ave': M_ave, 'std': M_std}
    gdsamples_per_H0_prior[H0_prior] = gdsamples

# print calibrated M
for each in M_calib:
    print(each, 'M = ', M_calib[each]['ave'], '+/-', M_calib[each]['std'])
```

    [root] *WARNING* fine_bins not large enough to well sample smoothing scale - minuslogprior
    [root] *WARNING* fine_bins not large enough to well sample smoothing scale - minuslogprior__H0_prior
    [root] *WARNING* fine_bins not large enough to well sample smoothing scale - chi2
    [root] *WARNING* fine_bins not large enough to well sample smoothing scale - chi2__loglikeSNIa
    [root] *WARNING* fine_bins not large enough to well sample smoothing scale - chi2
    [root] *WARNING* fine_bins not large enough to well sample smoothing scale - chi2__loglikeSNIa
    [root] *WARNING* fine_bins not large enough to well sample smoothing scale - minuslogprior
    [root] *WARNING* fine_bins not large enough to well sample smoothing scale - minuslogprior__H0_prior
    [root] *WARNING* fine_bins not large enough to well sample smoothing scale - chi2
    [root] *WARNING* fine_bins not large enough to well sample smoothing scale - chi2__loglikeSNIa
    R19 M =  -19.24672146997805 +/- 0.047619802661079305
    TRGB M =  -19.374133105389465 +/- 0.06345405899637714
    P18 M =  -19.44973655096901 +/- 0.028209150641493812

The posteriors for each of the $H_0$ priors are superposed in the next plot.

```python
def plot_calib():
    '''plot posterior of cosmological parameters'''

    gdsamples_R16 = gdsamples_per_H0_prior['R19']
    gdsamples_TRGB = gdsamples_per_H0_prior['TRGB']
    gdsamples_P18 = gdsamples_per_H0_prior['P18']

    gdplot = gdplt.get_subplot_plotter()
    gdplot.triangle_plot([gdsamples_R16, gdsamples_TRGB, gdsamples_P18],
                         ["H0", "M"],
                          contour_ls = ['-', '--', '-.'],
                          contour_lws = [1.5, 1.5, 1.5],
                          contour_colors = [('blue'),
                                            ('red'),
                                            ('green')],
                          filled = True,
                          legend_loc = 'upper right',
                          legend_labels = ['R19', 'TRGB', 'P18'])

    gdplot = gdplt.get_subplot_plotter()
    gdplot.triangle_plot([gdsamples_R16, gdsamples_TRGB, gdsamples_P18],
                         ["H0", "om0"],
                          contour_ls = ['-', '--', '-.'],
                          contour_lws = [1.5, 1.5, 1.5],
                          contour_colors = [('blue'),
                                            ('red'),
                                            ('green')],
                          filled = True,
                          legend_loc = 'upper right',
                          legend_labels = ['R19', 'TRGB', 'P18'])

plot_calib()
```

Preparing the Hubble function given the BAO and reconstructed $m(z)$ from the Pantheon samples.

```python
def H_bao(z, dMrs, dHrs, m, M):
    '''Hubble function at redshift z
    dMrs = dM(z)/rs
    dHrs = dH(z)/rs
    rs = sound horizon
    (m, M) come from dA(z) -> DDR -> dL (pantheon samples)'''

    return c_kms*(dMrs/dHrs)*(1 + z)*(10**5)*(10**((M - m)/5))

def sigH_bao(z, dMrs, sigdMrs,
             dHrs, sigdHrs, m, sigm, M, sigM):
    '''uncertainty in the H_bao'''

    jac_dM = (10**((25 + M - m)/5))*c_kms*(1 + z)/dHrs
    jac_dH = (10**((25 + M - m)/5))*c_kms*dMrs*(1 + z)/(dHrs**2)
    jac_m = (20000*(10**((M - m)/5))*c_kms*dMrs*(1 + z)* \
             np.log(10))/dHrs
    jac_M = jac_m

    var_dM = (jac_dM*sigdMrs)**2
    var_dH = (jac_dH*sigdHrs)**2
    var_m = (jac_m*sigm)**2
    var_M = (jac_M*sigM)**2

    return np.sqrt(var_dM + var_dH + var_m + var_M)
```

The compressed Pantheon samples' $m(z)$ is reconstructed and plotted below.

```python
gp = GaussianProcessRegressor(kernel = kernel,
                              alpha = np.diag(covmz_ps_tot),
                              n_restarts_optimizer = 10)
gp.fit(np.log(z_ps).reshape(-1, 1), mz_ps)

print("\nLearned kernel: %s" % gp.kernel_)
print("Log-marginal-likelihood: %.3f"
      % gp.log_marginal_likelihood(gp.kernel_.theta))

logz_ps_rec = np.log(np.linspace(z_min, z_max, n_div*4))
z_ps_rec = np.exp(logz_ps_rec)
mz_ps_rec, sigmz_ps_rec = \
    gp.predict(logz_ps_rec.reshape(-1, 1),
               return_std = True)

def mz_approx(z):
    '''approximates the reconstruction mz_rec piecewise'''

    delta_logz = list(abs(np.log(z) - logz_ps_rec))
    min_index = delta_logz.index(min(delta_logz))

    return mz_ps_rec[min_index], sigmz_ps_rec[min_index]

# plot in log(z)
fig = plt.figure()
plt.errorbar(np.log(z_ps), mz_ps,
             yerr = np.sqrt(np.diag(covmz_ps_tot)),
             fmt = 'ko', markersize = 5,
             ecolor = 'red', elinewidth = 2,
             capsize = 2, label = 'Pantheon')
plt.plot(logz_ps_rec, mz_ps_rec, 'b--', label = 'mean')
plt.fill_between(logz_ps_rec,
                 mz_ps_rec - sigmz_ps_rec,
                 mz_ps_rec + sigmz_ps_rec,
                 alpha = .4, facecolor = 'b',
                 edgecolor = 'None', label = r'1$\sigma$')
plt.fill_between(logz_ps_rec,
                 mz_ps_rec - 2*sigmz_ps_rec,
                 mz_ps_rec + 2*sigmz_ps_rec,
                 alpha = .2, facecolor = 'b',
                 edgecolor = 'None', label = r'2$\sigma$')
plt.xlabel('$\ln(z)$')
plt.ylabel('$m(z)$')
plt.legend(loc = 'upper left', prop = {'size': 9.5})
plt.xlim(np.log10(z_min), np.log10(z_max))
plt.ylim(10, 30)
plt.show()

# plot in z
fig = plt.figure()
plt.errorbar(z_ps, mz_ps,
             yerr = np.sqrt(np.diag(covmz_ps_tot)),
             fmt = 'ko', markersize = 5,
             ecolor = 'red', elinewidth = 2,
             capsize = 2, label = 'Pantheon')
plt.plot(z_ps_rec, mz_ps_rec, 'b--', label = 'mean')
plt.fill_between(z_ps_rec,
                 mz_ps_rec - sigmz_ps_rec,
                 mz_ps_rec + sigmz_ps_rec,
                 alpha = .4, facecolor = 'b',
                 edgecolor='None', label = r'1$\sigma$')
plt.fill_between(z_ps_rec,
                 mz_ps_rec - 2*sigmz_ps_rec,
                 mz_ps_rec + 2*sigmz_ps_rec,
                 alpha = .2, facecolor = 'b',
                 edgecolor='None', label = r'2$\sigma$')
plt.xlabel('$z$')
plt.ylabel('$m(z)$')
plt.legend(loc = 'lower right', prop = {'size': 9.5})
plt.xlim(z_min, z_max)
plt.ylim(10, 30)
plt.show()
```

To simplify the BAO reconstructions, we setup the function below in order to obtain BAO-$H(z)$ corresponding to a given $H_0$ prior.

```python
def setup_BAO(H0_prior):
    '''prepares H(z) from BAO for a given H0 prior and calibrated M'''

    M_ave = M_calib[H0_prior]['ave']
    M_std = M_calib[H0_prior]['std']

    # setup H(z) BAO data
    Hz_bao = []
    sigHz_bao = []

    for i in np.arange(0, len(z_bao)):

        m, sigm = mz_approx(z_bao[i])

        Hz_i = H_bao(z = z_bao[i],
                     dMrs = dmrs_bao[i],
                     dHrs = dhrs_bao[i],
                     m = m,
                     M = M_ave)

        sigHz_i = sigH_bao(z = z_bao[i],
                           dMrs = dmrs_bao[i],
                           sigdMrs = sigdmrs_bao[i],
                           dHrs = dhrs_bao[i],
                           sigdHrs = sigdhrs_bao[i],
                           m = m,
                           sigm = sigm,
                           M = M_ave,
                           sigM = M_std)

        Hz_bao.append(Hz_i)
        sigHz_bao.append(sigHz_i)

    Hz_bao = np.array(Hz_bao)
    sigHz_bao = np.array(sigHz_bao)

    return Hz_bao, sigHz_bao
```

The CC + BAO GP reconstructed-$H(z)$ is obtained in the next line.

```python
for H0_prior in H0_priors:

    H0_ave = H0_priors[H0_prior]['ave']
    H0_std = H0_priors[H0_prior]['std']

    z_cc_prior = np.append(np.array([z_min]), z_cc)
    Hz_cc_prior = np.append(np.array([H0_ave]), Hz_cc)
    sigHz_cc_prior = np.append(np.array([H0_std]), sigHz_cc)

    Hz_bao, sigHz_bao = setup_BAO(H0_prior)

    z_bao_cc_prior = np.append(z_bao, z_cc_prior)
    Hz_bao_cc_prior = np.append(Hz_bao, Hz_cc_prior)
    sigHz_bao_cc_prior = np.append(sigHz_bao, sigHz_cc_prior)

    gp_bao_cc = GaussianProcessRegressor(kernel = kernel,
                                         alpha = sigHz_bao_cc_prior**2,
                                         n_restarts_optimizer = 10)
    gp_bao_cc.fit(z_bao_cc_prior.reshape(-1, 1), Hz_bao_cc_prior)

    print("\nLearned kernel: %s" % gp_bao_cc.kernel_)
    print("Log-marginal-likelihood: %.3f"
          % gp_bao_cc.log_marginal_likelihood(gp_bao_cc.kernel_.theta))

    z_bao_cc_rec = np.linspace(z_min, z_max, n_div)
    Hz_bao_cc_rec, sigHz_bao_cc_rec = \
        gp_bao_cc.predict(z_bao_cc_rec.reshape(-1, 1),
                          return_std=True)

    fig = plt.figure()
    plt.errorbar(z_bao, Hz_bao,
                 yerr = sigHz_bao,
                 fmt = 'k^', markersize = 4,
                 ecolor = 'red', elinewidth = 2,
                 capsize = 2, label = 'BAO')
    plt.errorbar(z_cc_prior, Hz_cc_prior, yerr = sigHz_cc_prior,
                 label = 'CC', fmt = 'ko', markersize = 5,
                 ecolor = 'red', elinewidth = 2, capsize = 2)
    plt.plot(z_bao_cc_rec,
             Hz_bao_cc_rec, 'b--', label = 'mean')
    plt.fill_between(z_bao_cc_rec,
                     Hz_bao_cc_rec - sigHz_bao_cc_rec,
                     Hz_bao_cc_rec + sigHz_bao_cc_rec,
                     alpha = .4, facecolor = 'b',
                     edgecolor='None', label=r'$1\sigma$')
    plt.fill_between(z_bao_cc_rec,
                     Hz_bao_cc_rec - 2*sigHz_bao_cc_rec,
                     Hz_bao_cc_rec + 2*sigHz_bao_cc_rec,
                     alpha = .2, facecolor = 'b',
                     edgecolor='None', label=r'$2\sigma$')

    plt.title(H0_prior)
    plt.xlabel('$z$')
    plt.ylabel('$H(z)$')
    plt.legend(loc = 'upper left', prop = {'size': 9.5})
    plt.xlim(z_min, z_max)

    plt.show()
```

The GP reconstruction given the combined CC + Pantheon/MCT + BAO is obtained in the next line. For comparison, this is shown together with the reconstructions obtained from the CC and CC + BAO datasets only.

```python
for H0_prior in H0_priors:

    H0_ave = H0_priors[H0_prior]['ave']
    H0_std = H0_priors[H0_prior]['std']

    # reconstruct H(z): CC + prior
    z_cc_prior = np.append(np.array([z_min]), z_cc)
    Hz_cc_prior = np.append(np.array([H0_ave]), Hz_cc)
    sigHz_cc_prior = np.append(np.array([H0_std]), sigHz_cc)

    gp = GaussianProcessRegressor(kernel = kernel,
                                  alpha = sigHz_cc_prior**2,
                                  n_restarts_optimizer = 10)
    gp.fit(z_cc_prior.reshape(-1, 1), Hz_cc_prior)
    z_cc_rec = np.linspace(z_min, z_max, n_div)
    Hz_cc_rec, sigHz_cc_rec = gp.predict(z_cc_rec.reshape(-1, 1),
                                         return_std = True)

    # reconstruct H(z): CC + BAO + Prior
    Hz_bao, sigHz_bao = setup_BAO(H0_prior)

    z_bao_cc_prior = np.append(z_bao, z_cc_prior)
    Hz_bao_cc_prior = np.append(Hz_bao, Hz_cc_prior)
    sigHz_bao_cc_prior = np.append(sigHz_bao, sigHz_cc_prior)

    gp_bao_cc = GaussianProcessRegressor(kernel = kernel,
                                         alpha = sigHz_bao_cc_prior**2,
                                         n_restarts_optimizer = 10)
    gp_bao_cc.fit(z_bao_cc_prior.reshape(-1, 1), Hz_bao_cc_prior)

    z_bao_cc_rec = np.linspace(z_min, z_max, n_div)
    Hz_bao_cc_rec, sigHz_bao_cc_rec = \
        gp_bao_cc.predict(z_bao_cc_rec.reshape(-1, 1),
                          return_std = True)

    # Pantheon/MCT + CC + BAO + Prior, subscripted sbc for 'SN', 'BAO', 'CC'
    Hz_pnmct = H0_ave*Ez_sn
    var_pnmct = (np.diag(covEz_sn_corr)* \
                 (H0_ave**2)) + (H0_std*Ez_sn)**2

    z_sbc = np.append(z_sn, z_bao_cc_prior)
    Hz_sbc = np.append(Hz_pnmct, Hz_bao_cc_prior)
    sigHz_sbc = np.append(np.sqrt(var_pnmct), sigHz_bao_cc_prior)

    # save full data for part 2
    sbc_data = np.stack((z_sbc, Hz_sbc, sigHz_sbc), axis = 1)
    savetxt('sbc_data_' + H0_prior + '.txt', sbc_data)

    gp_sbc = GaussianProcessRegressor(kernel = kernel,
                                      alpha = sigHz_sbc**2,
                                      n_restarts_optimizer = 10)
    gp_sbc.fit(z_sbc.reshape(-1, 1), Hz_sbc)
    z_sbc_rec = np.linspace(1e-5, 2, 100)
    Hz_sbc_rec, sigHz_sbc_rec = \
        gp_sbc.predict(z_sbc_rec.reshape(-1, 1), return_std = True)

    print("\nLearned kernel: %s" % gp_sbc.kernel_)
    print("Log-marginal-likelihood: %.3f"
          % gp_sbc.log_marginal_likelihood(gp_sbc.kernel_.theta))

    fig = plt.figure()

    # plot Pantheon/MCT
    plt.errorbar(z_sn, Hz_pnmct, yerr = np.sqrt(var_pnmct),
                 label = 'Pantheon/MCT', fmt = 'ko', markersize = 5,
                 ecolor = 'red', elinewidth = 2, capsize = 2)

    # plot BAO
    plt.errorbar(z_bao, Hz_bao, yerr = sigHz_bao,
                 fmt = 'k^', markersize = 5, ecolor = 'red',
                 elinewidth = 2, capsize = 2, label = 'BAO')

    # plot CC
    plt.errorbar(z_cc, Hz_cc, yerr = sigHz_cc, label = 'CC',
                 fmt = 'kx', markersize = 5,
                 ecolor = 'red', elinewidth = 2, capsize = 2)

    # GP reconstruction from SN, BAO, CC
    plt.plot(z_sbc_rec, Hz_sbc_rec, 'b--', label = 'mean')
    plt.fill_between(z_sbc_rec,
                     Hz_sbc_rec - sigHz_sbc_rec,
                     Hz_sbc_rec + sigHz_sbc_rec,
                     alpha = .4, facecolor = 'b',
                     edgecolor = 'None', label = r'1$\sigma$')
    plt.fill_between(z_sbc_rec,
                     Hz_sbc_rec - 2*sigHz_sbc_rec,
                     Hz_sbc_rec + 2*sigHz_sbc_rec,
                     alpha = .2, facecolor = 'b',
                     edgecolor = 'None', label = '2$\sigma$')

    plt.title(H0_prior)
    plt.xlabel('$z$')
    plt.ylabel('$H(z)$')
    plt.legend(loc = 'upper left', prop = {'size': 9.5})
    plt.xlim(min(z_sbc_rec), max(z_sbc_rec))
    plt.show()

    fig = plt.figure()

    # CC only
    plt.plot(z_cc_rec, Hz_cc_rec, 'r--', label = 'CC')
    plt.fill_between(z_cc_rec,
                     Hz_cc_rec - 2*sigHz_cc_rec,
                     Hz_cc_rec + 2*sigHz_cc_rec,
                     alpha = .2, facecolor = 'r',
                     edgecolor = 'r', hatch = '-')

    # CC + BAO
    plt.plot(z_bao_cc_rec, Hz_bao_cc_rec,
             'g-.', label = 'BAO + CC')
    plt.fill_between(z_bao_cc_rec,
                     Hz_bao_cc_rec - 2*sigHz_bao_cc_rec,
                     Hz_bao_cc_rec + 2*sigHz_bao_cc_rec,
                     alpha = .2, facecolor = 'g',
                     edgecolor ='g', hatch = '|')

    # CC + SN + BAO
    plt.plot(z_sbc_rec, Hz_sbc_rec, 'b-',
             label = 'Pantheon/MCT + BAO + CC')
    plt.fill_between(z_sbc_rec,
                     Hz_sbc_rec - 2*sigHz_sbc_rec,
                     Hz_sbc_rec + 2*sigHz_sbc_rec,
                     alpha = .2, facecolor = 'b',
                     edgecolor = 'b', hatch = 'x')

    plt.title(H0_prior)
    plt.xlabel('$z$')
    plt.ylabel('$H(z)$')
    plt.legend(loc = 'upper left', prop = {'size': 9.5})
    plt.xlim(min(z_sbc_rec), max(z_sbc_rec))
    plt.show()
```

The reconstructed Hubble function will be used in part 2 to draw Horndeski theories from the Hubble data.

### References

**Data sets** used in this work:

***Pantheon/MCT***: A. G. Riess et al., Type Ia Supernova Distances at Redshift > 1.5 from the Hubble Space
Telescope Multi-cycle Treasury Programs: The Early Expansion Rate, Astrophys. J. 853 (2018)
126 [[1710.00844](https://arxiv.org/abs/1710.00844)].

***Pantheon samples***: D. M. Scolnic et al., The Complete Light-curve Sample of Spectroscopically Confirmed SNe Ia
from Pan-STARRS1 and Cosmological Constraints from the Combined Pantheon Sample,
Astrophys. J. 859 (2018) 101 [[1710.00845](https://arxiv.org/abs/1710.00845)].

***Baryon Acoustic Oscillations***, from *various sources*:

(1) BOSS collaboration, The clustering of galaxies in the completed SDSS-III Baryon Oscillation
Spectroscopic Survey: cosmological analysis of the DR12 galaxy sample, Mon. Not. Roy.
Astron. Soc. 470 (2017) 2617 [[1607.03155](https://arxiv.org/abs/1607.03155)].

(2) J. E. Bautista et al., The Completed SDSS-IV extended Baryon Oscillation Spectroscopic
Survey: measurement of the BAO and growth rate of structure of the luminous red galaxy
sample from the anisotropic correlation function between redshifts 0.6 and 1, Mon. Not. Roy.
Astron. Soc. 500 (2020) 736 [[2007.08993](https://arxiv.org/abs/2007.08993)].

(3) H. Gil-Marin et al., The Completed SDSS-IV extended Baryon Oscillation Spectroscopic
Survey: measurement of the BAO and growth rate of structure of the luminous red galaxy
sample from the anisotropic power spectrum between redshifts 0.6 and 1.0, Mon. Not. Roy.
Astron. Soc. 498 (2020) 2492 [[2007.08994](https://arxiv.org/abs/2007.08994)].

(4) A. Tamone et al., The Completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey:
Growth rate of structure measurement from anisotropic clustering analysis in configuration
space between redshift 0.6 and 1.1 for the Emission Line Galaxy sample, Mon. Not. Roy.
Astron. Soc. 499 (2020) 5527 [[2007.09009](https://arxiv.org/abs/2007.09009)].

(5) A. de Mattia et al., The Completed SDSS-IV extended Baryon Oscillation Spectroscopic
Survey: measurement of the BAO and growth rate of structure of the emission line galaxy
sample from the anisotropic power spectrum between redshift 0.6 and 1.1, Mon. Not. Roy.
Astron. Soc. 501 (2021) 5616 [[2007.09008](https://arxiv.org/abs/2007.09008)].

(6) R. Neveux et al., The completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey:
BAO and RSD measurements from the anisotropic power spectrum of the quasar sample
between redshift 0.8 and 2.2, Mon. Not. Roy. Astron. Soc. 499 (2020) 210 [[2007.08999](https://arxiv.org/abs/2007.08999)].

(7) J. Hou et al., The Completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey:
BAO and RSD measurements from anisotropic clustering analysis of the Quasar Sample in
configuration space between redshift 0.8 and 2.2, Mon. Not. Roy. Astron. Soc. 500 (2020) 1201
[[2007.08998](https://arxiv.org/abs/2007.08998)].

(8) V. de Sainte Agathe et al., Baryon acoustic oscillations at z = 2.34 from the correlations of
Lyα absorption in eBOSS DR14, Astron. Astrophys. 629 (2019) A85 [[1904.03400](https://arxiv.org/abs/1904.03400)].

(9) M. Blomqvist et al., Baryon acoustic oscillations from the cross-correlation of Lyα absorption
and quasars in eBOSS DR14, Astron. Astrophys. 629 (2019) A86 [[1904.03430](https://arxiv.org/abs/1904.03430)].

***Cosmic Chronometers***, from *various sources*:

(1) M. Moresco, L. Pozzetti, A. Cimatti, R. Jimenez, C. Maraston, L. Verde et al., A 6%
measurement of the Hubble parameter at z ∼ 0.45: direct evidence of the epoch of cosmic
re-acceleration, JCAP 05 (2016) 014 [[1601.01701](https://arxiv.org/abs/1601.01701)].

(2) M. Moresco, Raising the bar: new constraints on the Hubble parameter with cosmic
chronometers at z ∼ 2, Mon. Not. Roy. Astron. Soc. 450 (2015) L16 [[1503.01116](https://arxiv.org/abs/1503.01116)].

(3) C. Zhang, H. Zhang, S. Yuan, S. Liu, T.-J. Zhang and Y.-C. Sun, Four new observational H(z)
data from luminous red galaxies in the Sloan Digital Sky Survey data release seven, Research in
Astronomy and Astrophysics 14 (2014) 1221 [[1207.4541](https://arxiv.org/abs/1207.4541)].

(4) D. Stern, R. Jimenez, L. Verde, M. Kamionkowski and S. A. Stanford, Cosmic chronometers:
constraining the equation of state of dark energy. I: H(z) measurements, JCAP 2010 (2010)
008 [[0907.3149](https://arxiv.org/abs/0907.3149)].

(5) M. Moresco et al., Improved constraints on the expansion rate of the Universe up to z ˜1.1 from
the spectroscopic evolution of cosmic chronometers, JCAP 2012 (2012) 006 [[1201.3609](https://arxiv.org/abs/1201.3609)].

(6) Ratsimbazafy et al. Age-dating Luminous Red Galaxies observed with the Southern African
Large Telescope, Mon. Not. Roy. Astron. Soc. 467 (2017) 3239 [[1702.00418](https://arxiv.org/abs/1702.00418)].

***R19 $H_0$ prior***: A. G. Riess, S. Casertano, W. Yuan, L. M. Macri and D. Scolnic, Large Magellanic Cloud
Cepheid Standards Provide a 1% Foundation for the Determination of the Hubble Constant and
Stronger Evidence for Physics beyond ΛCDM, Astrophys. J. 876 (2019) 85 [[1903.07603](https://arxiv.org/abs/1903.07603)].

***TRGB $H_0$ prior***: W. L. Freedman et al., The Carnegie-Chicago Hubble Program. VIII. An Independent
Determination of the Hubble Constant Based on the Tip of the Red Giant Branch, Astrophys.
J. 882 (2019) 34 [[1907.05922](https://arxiv.org/abs/1907.05922)].

***P18 $H_0$ prior***: Planck collaboration, Planck 2018 results. VI. Cosmological parameters, Astron. Astrophys. 641 (2020) A6 [[1807.06209](https://arxiv.org/abs/1807.06209)].

**Python packages** used in this work:

``scikit-learn``: F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel et al., Scikit-learn:
Machine learning in Python, [Journal of Machine Learning Research 12 (2011) 2825](https://www.jmlr.org/papers/volume12/pedregosa11a/pedregosa11a.pdf?source=post_page---------------------------).

``cobaya``: J. Torrado and A. Lewis, Cobaya: Code for Bayesian Analysis of hierarchical physical models (2020) [[2005.05290](https://arxiv.org/abs/2005.05290)].

``getdist``: A. Lewis, GetDist: a Python package for analysing Monte Carlo samples (2019) [[1910.13970](https://arxiv.org/abs/1910.13970)].

``numpy``: C. R. Harris et al., Array programming with NumPy, [Nature 585 (2020) 357–362](https://www.nature.com/articles/s41586-020-2649-2?fbclid=IwAR3qKNC7soKsJlgbF2YCeYQl90umdrcbM6hw8vnpaVvqQiaMdTeL2GZxUR0).

``scipy``: P. Virtanen et al., SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,
[Nature Methods 17 (2020) 261](https://www.nature.com/articles/s41592-019-0686-2).

``seaborn``: M. L. Waskom, seaborn: statistical data visualization, [Journal of Open Source Software 6
(2021) 3021](https://joss.theoj.org/papers/10.21105/joss.03021).

``matplotlib``: J. D. Hunter, Matplotlib: A 2d graphics environment, [Computing in Science Engineering 9
(2007) 90](https://ieeexplore.ieee.org/document/4160265).