```python
import seaborn as sns
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import pickle
import os
import warnings
warnings.filterwarnings('ignore')

from auxiliary.auxiliary import (simulate_data, get_results_regression, process_results, get_results_local_regression,
                                process_results_local)
```

# Stacking/ Jackknife Model Averaging and its Use in Regression Discontinuity Design
---

## Introduction
---

A common occurrence in data-driven Economics is that the researcher faces uncertainty about several facets of a statistical model as well as the results from its calibration. While the uncertainty in the parameter estimates of a model is typically reported by showing at least the standard deviation of the parameter estimates, it is not common practice in Economics to discuss the implications of uncertainty about the specific form of the statistical model. It is rather common to select one model among several competing models (maybe using some model selection techniques) from that on pretending that it is the true one while performing some robustness checks using other model specifications (Moral-Benito (2015)).

Fletcher (2019) argues that the typical problem with this negligence of remaining model uncertainty is that parameter estimates might be biased and uncertainty is underestimated. This approach yields confidence intervals of the estimated parameters that are too optimistic in the sense that the true coverage probability is below its nominal level when not accounting for this uncertainty. This might cause the resulting inference to be seriously flawed. This aspect is a major reason why model averaging might be useful in Economics. Model averaging takes information from all competing models into account and therefore explicitly includes model uncertainty into its parameter estimates as well as its confidence intervals. Steel (2019) summarizes the tradeoff between model selection and averaging by the question what the researcher is interested in and what the previous certainty of knowledge is. If one is interested in the true model and is certain that one of the competing models must be the true one, then model selection yields the best results as it can select the true one without confounding it with the others. Although the extent of this advantage is questionable as Zhang (2015) proves that several model avergaing estimators (among them Jackknife Model Averaging) are actually root-n consistent if the true model is among the ones that is averaged over. Further when there is considerable uncertainty about the model form, then an ensemble of different forms constructed by averaging yields more reliable results as it takes information from all of them into account.

In Economics, the application of model averaging is imaginable in many situations. For example, in a more structural framework, often there are competing theories that inform the statistical model such as human capital as opposed to signaling theory. Another situation commonly encountered is the question which covariates should be included in a linear regression model (compare Steel (2019)). The last situation is often faced in Regression Discontinuity Design (RDD). There is usually uncertainty about the polynomial with which the assignment variable affects the outcome variable. In the highly cited prationers' guide to RDD by Lee and Lemieux (2010) they argue that typically there is no theory available on which to base the decision for the choice of polynomial. This leads them to the suggestion to use model selection based on the Akaike information criterion (AIC) as a standard way to choose the polynomial. As there is no strong prior which polynomial to add and whether any of them is actually a good representation of the true model, for the above reasons, model averaging should be a suitable alternative to the suggested model selection procedure. It might be even superior in many situations as the actual coverage probability might be closer to its nominal one. In my first simulation, I therefore take a typical applied RDD paper (Bronzini and Iachini (2014)) that follows quite closely the suggestions in Lee and Lemieux (2010) to benchmark whether model averaging might actually perform better than selection based on the AIC (in the sense of bias, and root mean squared error in the treatment effect as well as the coverage probability of the 95% confidence intervals). Unfortunately, I noticed only after already having set up my first simulation that there is already a published paper that has a similar idea in mind (Button (2016)). In the light of that, I adapted my first simulation study to incorporate some of the author's good ideas. For the model averaging I focus on the approach of stacking (Wolpert (1992)) which has been rediscovered in the Econometrics literature as Jackknife Model Averaging (JMA) based on Hansen and Racine (2012). My setup is also an improvement over Button's in the sense that I allow for heteroskedasticity which generally works in favor of model averaging using JMA as opposed to other model averaging strategies (I will discuss this further later). In general, my simulation differs in a few dimensions which I will elaborate on later.

In a second simulation study, I go beyond the general ideas that are partly covered in Button (2016). As JMA is based on a leave one out cross validation procedure that aims at minimizing a cross validation criterion approximating the expected test mean squared error depending on the weights given to the competing models for averaging, I explore whether JMA can also be used for bandwidth selection in RDD in a self-sufficient way. As in RDD one is mainly interested in the comparison of individuals around a certain cutoff, it remains to determine which individuals to include around the cutoff to estimate the treatment effect. Lee and Lemieux (2010) suggest the procedure developed by Imbens and Lemieux (2008) involving the use of local linear regressions around the cutoff for several bandwidths, calculating a cross validation criterion for each and then choosing the bandwidth that minimizes it. The approach in JMA is similar while accounting for different possible model forms (such as the different orders of polynomials). Hence, JMA might be an alternative for bandwidth selection (especially when local linearity is a poor approximation) which I explore in the second simulation of my project.

The general structure of my project paper is the following. In the next section I introduce stacking in general and the specific implementation of it called Jackknife Model Averaging based on Hansen and Racine (2012) which involves a convenient calculation for linear regression models as needed in RDD. In section three, I introduce the general idea of Regression Discontinuity Design and explain the analysis of Bronzini and Iachini (2014) on which my different data generating processes are based. In section four I run my first simulation study in which I check the usefulness of JMA for determination of the treatment in the verge of uncertainty about the polynomial order of the assignment variable. In section five, I pick up the first simulation to see how JMA behaves as a tool for bandwidth selection. In the last section I shortly conclude with a summary of my findings.

## Stacking/ Jackknife Model Averaging
---

As already emphasized in the introduction, model averaging has its appeal in situations where the exact form of the model is unknown. It has an advantage over model selection in the sense that it incorporates the resulting uncertainty explicitly in the modeling by combining competing models and extracting information of each of them. This makes it possible to account for the model uncertainty in the uncertainty around the estimated model parameters making inference more stable in comparison to model selection (Fletcher (2019)). Generally, there are two main strands of model averaging present. The first follows the Bayesian paradigm and is consequently called Bayesian Model Averaging. It is characterized by the fact that priors on the true parameters and the models must be formed. The outcome is a weighted average of the posteriors from different models with the weight being based on the posterior probability that a model is true. The main objective of it is to rather identify the true model as opposed to improving prediction quality. While this is generally a desirable feature for Economists, the complexity to determine the priors and the dependence of the posterior model probabilities on it makes its use difficult in practice, though (Fletcher (2019)).

The second approach is Frequentist Model Averaging which is mainly concerned with improving model predictions and to obtain confidence intervals with good coverage. In this category falls the approach chosen in my project, Jackknife Model Averaging. It uses the idea of leave one out cross validation in order to obtain the weights that are used to build an optimal weighted average across different models. This idea was first introduced in Statistics as model-mixing formulated in Stone (1974). It was later picked up again by Wolpert (1992) in the realm of Machine Learning and he coined the term stacking for it. In the Econometrics literature it was Hansen and Racine (2012) who reintroduced it giving it a formal underpinning in the sense that they proved its capacity to improve model prediction. Wolpert (1992) only provides simulation evidence. In the following I will introduce the idea of stacking and especially Jackknife Model Averaging following Hansen and Racine (2012).

Assume that we are interested in averaging across different generalized linear models, i.e. linear models that do not necessarily have errors that are normally distributed. These competing models are denoted by $m = 1, ..., M$. The true data generating process takes the general linear form as following:

\begin{equation}
    \begin{split}
        y_i = \mu_i + e_i \\
        E(e_i|x_i) = 0.
    \end{split}
\end{equation}

The data is independently distributed $(x_i, y_i)$ for $n = 1, ..., N$. $\mu_i$ corresponds to $E(y_i|x_i)$. We determine $y = (y_1, ..., y_N)'$, $\mu = (\mu_1, ..., \mu_N)'$ and $e = (e_1, ..., e_N)'$. Furthermore there is, potentially, heteroskedasticity defined as $E(e_i^2|x_i)=\sigma_i^2$. Each of the competing models now differ in the sense that they yield possibly different linear estimators ${\hat\mu^1, ..., \hat\mu^M}$ for $\mu$.
Moral-Benito (2015) note that the general approach of Frequentist Model Averaging is to now find a weighted average among those estimators $\hat\mu^m$ sucht that the prediction of the ensemble is optimized. This means the general idea is to find weights $w^m$ per model that yield an ensemble estimator like the following:

\begin{equation}
    \hat\mu(w) = \sum_{m=1}^{M} w^m \hat\mu^m = \hat\mu w.
\end{equation}

The idea of Frequentist Model Averaging, hence, boils down to finding a weigthing vector $w=(w^1, ..., w^M)'$ based on optimizing some criterion that depends on the weights. In the case of stacking/ JMA this criterion is based on leave one out cross validation (as Fletcher (2019) points out that Hansen and Racine (2012) wrongly use the term jackknife for it, I refrain from applying their terminology in the following). Before I derive the exact cross validation procedure as well as the resulting cross validated criterion, let us introduce some further notation. Hansen and Racine focus their attention on the linear regression model for the application of the JMA and introduce an efficient calculation of their criterion for this special class of models. In general, in the class of linear models, the estimator $\hat\mu^m$ can also be written as a transformation of the vector $y$ using a projection matrix $P_m$ , i.e. $\hat\mu^m = P_m y$. It is well-known that for linear regression the matrix $P_m$ has the form $P_m = X^m(X^{m'}  X^m)^{-1} X^{m'} $. The ensemble estimator can in general be written as:

\begin{equation}
    \hat\mu(w) = P(w) y
\end{equation}

where

\begin{equation}
    P(w) = \sum_{m=1}^{M} w^m P_m.
\end{equation}

The first observation here is that $P(w)$ is linear in $w$ and $y$ is linear in $P(w)$ which means that it is also linear in $w$. Hansen and Racine (2012) restrict the weights to lie in a unit simplex, i.e. to lie in:

\begin{equation}
    H_n = \{w \in R^M: w^m \geq 0, \sum_{m=1}^{M} w^m = 1\}.
\end{equation}

The leave one out cross validation now comes into play in how the estimator for $\mu$ is calculated for each model. The linear estimator of the m*th* model is $\tilde \mu^m = (\tilde \mu_1^m, ..., \tilde \mu_N^m)'$. $\tilde \mu_i^m$ is the fitted value of model $m$ for observation $i$ when estimating the model on the data set without observation $i$ and then applying the estimated model on the i*th* observation. From this vector of linear estimates one can derive the cross validated residual vector $\tilde e^m = y - \tilde \mu^m$. As we have $M$ models in total, mixing them with some weight vector $w$ results in different cross validated linear estimators $\tilde \mu(w)$ and consequently different cross validated residuals $\tilde e(w)$ depending on the weight. This is depicted in the following two equations:

\begin{equation}
    \begin{split}
        \tilde \mu(w) = \sum_{m=1}^M w^m \tilde \mu^m = \tilde\mu w = \tilde P(w) y \\
        \tilde e(w) = y - \tilde \mu(w) = \sum_{m=1}^M w^m\tilde e^m = \tilde e w
    \end{split}
\end{equation}

with $\tilde \mu = (\tilde \mu^1, ..., \tilde \mu^M)'$, $\tilde P(w) = \sum_{m=1}^M w^m \tilde P_m$ and $\tilde e = (\tilde e^1, ..., \tilde e^M)'$.
Taking a step back to general stacking and how JMA relates to it, let us look at how stacking is typically built up. Stacking seeks to maximize the following expression by setting the weights $w^m$:

\begin{equation}
    \sum_{n=1}^N log L(\tilde \mu_i(w)|y_i)
\end{equation}

where $log L(.)$ describes the log likelihood function and $\tilde \mu_i(w) = \sum_{m=1}^M w^m \tilde \mu_i^m$ which is the averaged linear estimator for observation $i$ calculated using the data set without the i*th* observation. The approach taken by JMA is to now rely on another criterion than the log likelihood $log L(.)$ above but rather the least squares cross validation criterion as an estimate for the expected true error:

\begin{equation}
    CV(w) = \frac{1}{n} \tilde e(w)' \tilde e(w) = w' S w
\end{equation}

where $S = \frac{1}{n} \tilde e' \tilde e$. This means that Hansen and Racine (2012) construct a measure for the expected test mean squared error depending on the extent to which the individual models are mixed. $CV(w)$ is an $M \times M$ matrix covering the extreme cases that might occur on the diagonal which capture choosing weight of one for either of the models. As the choice of $w$ is restricted to lie in $H_n$ and the expected test error is supposed to be minimized this comes down to a constraint optimization problem like the following:

\begin{equation}
    \begin{split}
        \textrm{min}_w CV(w) \\
        \textrm{subject to } w \in H_n.
    \end{split}
\end{equation}

This is a quadratic programming problem yielding an optimal weight $\hat w$ which can then be used to derive the JMA estimator for $\mu$ which is $\hat \mu(\hat w) = \hat \mu \hat w$.

In order to summarize, the approach involves setting up each potential model indivdually. For each model based on leave one out cross validation the fitted value of each observation $i$ has to be obtained by estimating the model without that very observation. This means that per model $N$ regressions have to be run. Based on these fitted values, the cross validated residuals are derived which are then together with the potential weights $w$ used to build the least squares cross validation criterion. This serves an approximation of the expected test mean squared error of the resulting ensemble depending on the choice of $w$. Consequently, it is minimized by choosing optimal wights $\hat w$ under the constraint that each of them is nonnegative and sum up to one together.

For the case of linear regression, Hansen and Racine (2012) come up with a computationally light way of performing leave one out cross validation that reduces it to one operation per model as opposed to $N$ regressions. We already observed that in linear regressions the projection matrix has the form $P_m = X^m(X^{m'}  X^m)^{-1} X^{m'}$. This means consequently that the cross validated estimator for observation $i$ of a model $m$ is the following: $\tilde \mu_i^m = x_i^{m'} (X_{(-i)}^{m'} X_{(-i)}^{m})^{-1} X_{(-i)}^{m'} y_{(-i)}$ where $(-i)$ means that it is the data set without observation $i$ and $x_i^m$ is the i*th* row of the regressor matrix. The authors now make use of a derivation in Li (1987). Li derives that the cross validated projection matrix $\tilde P_m$ has the following form in linear regressions $\tilde P_m = D_m (P_m - I) + I$ with

\begin{equation}
    D_m = \begin{pmatrix} (1-h^m_{11})^{-1} & 0 & \cdots & 0 \\ 0 & \ddots & & \vdots \\ \vdots & & \ddots & 0 \\ 0 & \cdots & 0 & (1-h^m_{NN})^{-1} \end{pmatrix}
\end{equation}

and $h^m_{ii} = x_i^{m'} (X^{m'} X^{m})^{-1} x_i^{m}$ which is also the i*th* diagonal element of $P_m$. The cross validated residual vector of a single model $m$ can hence be written as $\tilde e^m = D_m(y - P_m y)$ which can be obtained in a single operation per model. This is exploited in the code I have written and exhibits a large speed advantage over the conventional way of performing leave one out cross validation which I will also explore in my last simulation.

While the Jackknife Model Averaging is a form of stacking, Hansen and Racine (2012) now add an entirely new discussion to the topic which is that the JMA weights $\hat w$ are asymptotically optimal drawing on two criteria. They define the training mean squared prediction error as $L_n(w) = \frac{1}{n} (\mu - \hat \mu(w))'(\mu - \hat \mu(w))$ and the expected test mean squared prediction error as $R_n(w) = E(L_n(w)|X)$. Under some regularity conditions, they prove that these criteria converge in probability to their minimal possible value across all feasible $w \in H_n$ when employing the JMA weights $\hat w$. Formally, this looks like the following:

\begin{equation}
    \frac{L_n(\hat w)}{\text{inf}_{w \in H_n} L_n(w)} \rightarrow_p 1
\end{equation}

\begin{equation}
    \frac{R_n(\hat w)}{\text{inf}_{w \in H_n} R_n(w)} \rightarrow_p 1.
\end{equation}

This shows that asymptotically the JMA weights yield the best in-sample and out-of-sample predictions across all feasible weights which also includes estimating just a single one of each model. This shows clearly that the focus of the approach is on improving predictive performance. This is achieved by a decrease in variance at the cost of increasing bias.
While we are generally more interested in measuring an accurate treatment effect in RDD, I will argue in the next section why JMA might still be beneficial for RDD although it might induce some bias in the treatment effect.

## Regression Discontinuity Design
---

My data generating process is based on a quite typical setup in RDD which is inspired by Bronzini and Iachini (2014). In their paper they pursue the question whether subsidy programs for Research & Development (R&D) for companies are effective in the sense that they actually induce firms to subsequently raise their investment in R&D. Essentially, they are interested in measuring the treatment effect of subsidies for R&D on the actual investment in it. For that they exploit a specific subsidy program that was launched in Northern Italy. The program began in 2003 and asked industrial companies to come up with project ideas that were funded by the program in case the idea gained a certain score determined by an independent commission. Every firm that received 75 points or more (on a scale from 0 to 100) for their idea was subsidized by a percentage of the total amount of the project.

The authors' identification strategy now involves a sharp RDD which postulates that those firms around the threshold of 75 points of score are comparable. It is argued that this has quasi-experimental character allowing the authors to identify the treatment effect by comparing some measure of R&D investment across firms around the threshold.
The authors draw from several measures while focussing on total investment into R&D relative to the sales before the subsidy program was launched in order to make it comparable. This is also the dependent variable I will focus on. In general in RDD, this dependent variable is a function of the assignment variable (that determines whether the firm is assigned to treatment) which is the score here. If the firms cannot control the assignment process, the RDD approach is valid. By assuming that the assignment around the threshold is somewhat random rendering the firms around it very similar apart form differeng in treatment and control, one can compare those at the threshold by measuring the discontinuity in the function of the dependent variable on the assignment variable. In this specific application, the authors allow for treatment effect heterogeneity. They allow the treatment effect to vary depending on whether the firm is classified as small or large. It is also quite common to further control for some variables that might affect both the dependent and the assignment variable which is not done in this paper. As previously noted, it is not ad-hoc clear whether the dependent variable is linear in the assignment variable or not. For this reason, it is common practice (promoted by Lee and Lemieux (2010)) to add several polynomials of the assignment variable to the function and deciding on the "true" model by choosing the one that has the lowest AIC. Formally, the described regression to obtain the treatment effect in the paper at hand looks like the following:

\begin{equation}
    Y_i = \sum_{k=1}^2 \sum_{p=0}^P \alpha_{kp} Size_i^k (X_i-c)^p + T_i \sum_{k=1}^2 \sum_{p=0}^P \beta_{kp} Size_i^k (X_i-c)^p +   \epsilon
\end{equation}

where $Size_i^1 = 1$ if the firm $i$ is small and zero otherwise and $Size_i^2 = 1$ if firm $i$ is large and zero otherwise. $Y_i$ is the investment in R&D divided by pre-program sales and $X_i$ is the assignment variable which is in our case the score a firm received for its project idea. $c$ captures the threshold for the assignment of the treatment which is equal to 75 as every firm having at least this score obtained the subsidy. $T_i$ is the treatment indicator which is one if $X_i \geq 75$ and zero otherwise. Withdrawing the threshold $c$ from the assignment variable $X_i$ is common practice as like this the coefficients $\beta_{10}$ and $\beta_{20}$ capture the discontinuity (the difference in intercept) at the threshold between treatment and non treatment for small and large firms, respectively. Consequently, they capture the treatment effect. The $P$ displays that the order of polynomial included in the regression function is unknown and potentially different ones might be included.

The authors now find themselves in a common situation when employing RDD. They cannot rely on any theory of how the assignment variable affects the dependent variable. Capturing the functional form as precisely as possible is of large interest, though, as it affects how accurately the difference in intercept and hence in treatment effect can be measured (compare Lee and Lemieux (2010)). For this reason, usually, it is allowed to have potentially different functional forms on each side of the cutoff. The flexibility in functional form is typically tested by including more and more polynomials of the assignment variable one at a time. The argument for this approach is that, when running only a polynomial of order one while the true relationship actually is more nonlinear might result in ommited variable bias. This is where model averaging and specifically JMA can come into play. Although model averaging increases bias while reducing variance in the estimates, by including information from several models (several polynomials) can decrease the danger of omitted variable bias. Further if the true model is among the candidates, the JMA is actually root-n consistent which means that with large n it does not do any harm to use JMA as opposed to luckily only running the true model. In comparison to model selection based on AIC as suggested in Lee and Lemieux (2010), it has the advantage that information from all models enter the parameter estimates and its uncertatinty. In the paper at hand, running the regression above with up to three polynomials and select the correct model via AIC suggests that the true model has polynomial of zero for small firms and the treatment effect is measured at 0.045 with a standard error of 0.018. This is just significant at the five percent level and rests heavily on the accurracy of the confidence intervals. As already established before, model selection tends to display too much certainty, understating confidence intervals. In the example above it might easily be the case that statistical significance cannot be maintained at the same level when incorporating model uncertainty. This is where JMA with its more accurate confidence intervals could be beneficial, although the bias might be increased but still it could give more accurate inference than AIC model selection and display less risk of ommited variable bias than single estimation of one model. In my first simulation study I will investigate the difference in estimated treatment effect between AIC model selection and JMA based on the setup of Bronzini and Iachini (2014).

Lee and Lemieux (2010) discuss a second major method of how to estimate the treatment effect in RDD which gives rise to my second simulation study. They argue that the above regression is not too appealing in the sense that it uses data far away from the cutoff point to predict the dependent variable at the cutoff (which one is solely interested in). It is therefore argued that an approach which reduces the data to closer to the cutoff might be prefered. Usually relied on when pursuing this is the nonparametric method suggested by Imbens and Lemieux (2008). The crucial part of this approach is the choice of the bandwidth $h$ around the cutoff $c$ for which the data is included for the estimation of the treatment effect. The idea is to restrict the data to close around the cutoff, i.e. including all $i$ for which $c-h \leq X_i \leq c+h$. Within this window, the data is used to run a kernel regression with a rectangular kernel which essentially boils down to running a regular linear regression as above with $P=1$ with the only difference that they run seperate regressions for each side of the cutoff. These regressions are run across a range of different bandwidths and for each of them the test mean squared error is calculated via leave one out cross validation. Averaging those across both sides of the cutoff results in a single criterion per bandwidth which looks like the following:

\begin{equation}
    CV_Y(h) = \frac{1}{N} \sum_{i=1}^N (Y_i - \hat Y(X_i))^2
\end{equation}

where $\hat Y$ is calculated as in JMA with leave one out cross validation based on a local linear regression on each side of the cutoff $c$ with a certain bandwidth $h$. The optimal bandwidth is then the $h$ that minimizes the above criterion, i.e. $h^* = \text{argmin}_h CV_Y(h)$. Clearly, there is a resemblence between this criterion and the one used in JMA. For this reason, I try whether JMA might be used to run local polynomial regressions in order to select the optimal bandwidth. This might be advantegeous in the case where the linear approximation of the local linear regression relied on is actually in parts not very accurate. I would generally expect that JMA might actually perform better than the above approach when there is polynomial nonlinearity to varying degree across the range of different bandwidths tested. This idea is investigated in my second simulation.

## Simulation Study One

In the first simulation study I mimic the true data of Bronzini and Iachini (2014) for the independent variables as closely as possible. The regressors in the paper are only the size of the firm and the score they obtained. As there are roughly 50 percent of small firms I draw the firm size from a binomial distribution with probability of 0.5. For $N$ observations this corresponds to:

\begin{equation}
    Size^1 \text{~} B(N, 0.5).
\end{equation}

There are some differences regarding the average distribution of the score depending on the size in the original data. So I simulate the score depending on the size of the firm. I use a right skewed normal distribution with $N(88, 12)$ for 80 percent of the small firms in combination with a uniform distribution $U(20, 55)$ for the rest. For large firms I only take a right skewed normal distribution of $N(92, 18)$ and flatten the peak by taking all firms with score between 80 and 90 and draw them again from $U(78, 92)$. As the scores can only take discrete values between 0 and 100, I round every number to the nearest integer and make sure that all observations lie in the interval by replacing values outside by a random choice of those values already drawn inside the interval. The comparison of the original data with my simulated data is shown in the graphs below (after having adjusted each score already by subtracting the cutoff $c=75$.

```python
# load original data
original_data = pd.read_stata("data/Bronzini-Iachini_dataset.dta")
# simulate data
np.random.seed(123)
num_obs = 360
coefficients = {"untreated": np.array([-0.05, -0.0016]),
                "treated": np.array([0.08, 0.0003])}
simulated_data = simulate_data(num_obs, coefficients, error_dist="random_cluster")[0]
```

```python
# plot scores of original and simulated data
fig, ax = plt.subplots(2, 1, sharex=True)
for size, number in [("Small Firms", 0), ("Large Firms", 1)]:
    ax[number].set_title(size)
    ax[1].set_xlabel("Score")
    sns.kdeplot(original_data.loc[original_data["largem"]==number, "s"], ax=ax[number], label="original")
    sns.kdeplot(simulated_data.loc[simulated_data["large"]==number, "score"], ax=ax[number], label="simulated")
```

In the original paper, the authors run polynomial regressions with $P \in \{0, 1, 2, 3\}$ and choose the optimal model according to the AIC which is readily available when running a linear regression. They find that there is no treatment effect for the large firms (see Table 5 Panel A left hand side in their paper). The dependent variable $Y$ (total investment in R&D divided by pre-program sales) I now create according to the following true data generating process:

\begin{equation}
    \begin{split}
        P &= 1 \\
        \alpha_{1p} &= (-0.05, -0.0016)' \\
        \alpha_{2p} &= (0, 0)' \\
        \beta_{1p} &= (0.08, 0.0003)' \\
        \beta_{2p} &= (0, 0)'.
    \end{split}
\end{equation}

This means that the true data generating process prescribes that for large firms there is no effect at all of the score on $Y$ and hence also no treatment effect. The dependent variable fluctuates randomly around 0. This is also what was found by the authors. The coefficients for small firms are also inspired by the authors' results when running a first order polynomial regression on the true data. The treatment effect for small firms is equal to 0.08 while score generally has an order one polynomial effect on $Y$. This data generating process results in the following typical RDD representation of the data in which $Y$ is plotted on the assignment variable $X$. Additionally, as the authors, I present this here only for small firms and average over $Y$ for each score.

```python
# plot small firms averaged scaled investment on the score (from previously simulated data)
small = simulated_data.loc[simulated_data["large"] == 0, :]
mean_small = small.groupby("score").mean()
mean_small.reset_index(inplace=True)
fig, ax = plt.subplots()
sns.scatterplot(mean_small["score"], mean_small["scaled_investment"], ax=ax)
ax.axvline()
sns.regplot(mean_small.loc[mean_small["score"]<0, "score"], mean_small.loc[mean_small["score"]<0, "scaled_investment"],
           order=1, scatter=False, ci=None)
sns.regplot(mean_small.loc[mean_small["score"]>=0, "score"], mean_small.loc[mean_small["score"]>=0, "scaled_investment"],
           order=1, scatter=False, ci=None, color=sns.color_palette("tab10")[0])
```

The error term is postulated as clustered by the authors (which is also assumed for the figure above). In my simulations, I control for different possible distributions of the error term. I show them graphically below. The homoskedastic is a $0.08 \times N(0, 1)$ distribution. The "normal" one is heteroskedastic in the sense that the error increases the closer the score is to the cutoff. In the "inverse" case this heteroskedasticity pattern is flipped around. In the "random_cluster" scenario I cluster the standard erros by allowing the error to come from a different uniform distribution centered around zero depending on the score and I add it to $0.05 \times N(0,1)$. The resulting error distributions are presented below.

```python
fig, ax = plt.subplots(2, 2, sharex=True, sharey=True, gridspec_kw={"wspace": 0})
for error_distr in [("homoskedastic", 0, 0), ("random_cluster", 0, 1) ,("normal", 1, 0), ("inverse", 1, 1)]:
    data, error = simulate_data(num_obs, coefficients, error_dist=error_distr[0])
    sns.scatterplot(data["score"], error, ax=ax[error_distr[1], error_distr[2]])
    ax[error_distr[1], error_distr[2]].set_title(error_distr[0])
```

In my simulation I now estimate the model using different functional forms and approaches. I always correctly exclude any regressors for large firms and only include different polynomials for the small firms. I run the polynomial regressions above with $P \in \{0, 1, 2, 3\}$ for small firms. Additionally, I separately extract which model would be selected when using the AIC. On top of that, I run Jackknife Model Averaging on the simulated data set as well. For the Jackknife Model Averaging I allow the procedure to average over the four different $P$. For each simulation run I report the estimated treatment effect, indicate whether the 95 percent confidence interval covers the true treatment effect parameter and how wide the confidence interval is. This is done for 1000 runs. The confidence interval for the polynomial regressions is readily available and consequently so is the one for the AIC model selection case. For JMA, per run, I bootstrap from the simulated data set 200 times by drawing with replacement and running JMA for the newly created data to obtain a confidence interval for the treatment effect. I opt for the simple percentile confidence interval (compare chapter 5 in Davidson and Hinkley (1997)).
In the next step I average the treatment effect over the 1000 runs, calculate the standard deviation and the root mean squared error to observe how well the different approaches recover the true treatment effect. Further I report the coverage probability and average width of the confidence interval to get a picture of how inference might be flawed and how large the increase in width should be to obtain good coverage.

I choose the order one polynomial model as the true underlying data generating process to see how JMA reacts to the unfavourable condition of having to face more flexible specifications which are not actually the true ones. As the true model is among the models to average over, though, the average treatment effect should converge to the true treatment effect with increasing number of observations (firms in the sample). This root-n consistency I check by running the above 1000 runs with different number of observations $N \in \{100, 200, 360, 600, 1000\}$. Generally, I expect the AIC to perform better in recovering the treatment effect as the true model is among the candidates and as opposed to JMA, the AIC does not confound it with the others when it is selected. In this situation where the number of observations is still somewhat small, JMA should have problems as it could easily find higher order effects of score in every run due to random disturbances (error term) which causes the bias to be quite substantial as the true linear form is confounded by higher order polynomials.
At the same time I expect the JMA confidence interval to have good coverage of the true treatment effect outperforming the AIC model selection due to its incorporation of model uncertainty. As opposed to Button (2016), I allow the error term not to be homoskedastic. As JMA is supposed to be robust to heterskedasticity, I do not expect large differences in performance across different error term specifications. The existence of heteroskedasticity gives an edge for JMA in comparison to the other model averaging procedures considered in Button (2016) as they are not heteroskedasticity robust (see Hansen and Racine (2012)). Button should have included a simulation with heteroskedasticity but this is only a side remark. I run the previously described simulation below.

```python
# If we are running on TRAVIS-CI we will simply load a file with existing results.
# just load the results after if statement if the simulation might take too long for you (I had to run it over night)...
if os.environ.get("TRAVIS"):
    results_linear_dgp = pickle.load(open("data/simulated/results_linear.pickle", "br"))
    processed_results_linear_dgp = pickle.load(
        open("data/simulated/processed_results_linear.pickle", "br"))
else:
    # set up simulations
    np.random.seed(123)
    num_runs = 1000
    num_bootstrap_runs = 200
    true_treatment_effect = 0.08
    true_model = {
        "polynomials": 1,
        "coefficients": {
            "untreated": np.array([-0.05, -0.0016]),
            "treated": np.array([0.08, 0.0003]),
        },
    }
    # run simulations
    results_linear_dgp = {}
    processed_results_linear_dgp = {}
    for error_distr in ["homoskedastic", "random_cluster" ,"normal", "inverse"]:
        results_linear_dgp[error_distr] = {}
        processed_results_linear_dgp[error_distr] = {}
        for num_obs in [100, 200, 360, 600, 1000]:
            results_linear_dgp[error_distr][str(num_obs)] = get_results_regression(
                num_runs, num_obs, num_bootstrap_runs, true_model, error_distr
            )
            processed_results_linear_dgp[error_distr][str(num_obs)] = process_results(
                results_linear_dgp[error_distr][str(num_obs)], true_treatment_effect
            )
```

Below I present the results of this simulation for the run in which I postulate the situation as in the paper. In the paper, $N$ equals to 360 and the error terms are assumed to be clustered across scores. In the other specifications with different error terms and sample sizes, the qualitative conclusion that I will make below stay valid which is way I do not present all the results here.

As expected the bias for JMA is larger than the one for model selection using AIC. Overall, though, JMA has a lower root mean squared error which is achieved by a reduction of the standard error of the estimated treatment effect. We can further see that the coverage probability of the 95 percent confidence interval almost reaches its nominal level while for AIC it is around nine percent below it. Rejection of the null hypothesis that the treatment effect equals zero in the case of AIC actually has a higher risk of type I error than the researcher nominally expects. This is different for JMA and in this specific example, a test would even be more conservative as the treatment effect is underestimated. The better coverage is obviously achieved for JMA by wider confidence intervals which compensate for the fact that the bias is generally larger. As a side remark it should be noted, that for the case of an "inverse" error, the bias encountered by JMA is larger than for the other cases which can be explained by the fact, that generally the error can be stronger than in other specifications (which can be seen in my figure with the different error distributions). This likely induces the JMA to find an influence of higher order polynomials more often which makes the confounding effect stronger.

```python
processed_results_linear_dgp["random_cluster"]["360"]
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>AIC</th>
      <th>JMA</th>
      <th>polynomial_0</th>
      <th>polynomial_1</th>
      <th>polynomial_2</th>
      <th>polynomial_3</th>
    </tr>
    <tr>
      <th>Statistic</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>Treatment Effect</th>
      <td>0.078921</td>
      <td>0.071887</td>
      <td>0.033971</td>
      <td>0.080486</td>
      <td>0.080926</td>
      <td>0.081253</td>
    </tr>
    <tr>
      <th>Bias</th>
      <td>-0.001079</td>
      <td>-0.008113</td>
      <td>-0.046029</td>
      <td>0.000486</td>
      <td>0.000926</td>
      <td>0.001253</td>
    </tr>
    <tr>
      <th>Standard Error</th>
      <td>0.028744</td>
      <td>0.025121</td>
      <td>0.011654</td>
      <td>0.020865</td>
      <td>0.028149</td>
      <td>0.036208</td>
    </tr>
    <tr>
      <th>RMSE</th>
      <td>0.028764</td>
      <td>0.026399</td>
      <td>0.047482</td>
      <td>0.020871</td>
      <td>0.028164</td>
      <td>0.036230</td>
    </tr>
    <tr>
      <th>95% Coverage</th>
      <td>0.859000</td>
      <td>0.944000</td>
      <td>0.037000</td>
      <td>0.946000</td>
      <td>0.942000</td>
      <td>0.940000</td>
    </tr>
    <tr>
      <th>CI Width</th>
      <td>0.082082</td>
      <td>0.107835</td>
      <td>0.048016</td>
      <td>0.078485</td>
      <td>0.105812</td>
      <td>0.135816</td>
    </tr>
  </tbody>
</table>
</div>

The root-n consistency I check, as suggested in Zhang (2015), by calculating $N^{0.95} \times MSE$. For root-n consistency we should see that with increasing $N$ the above measure should decrease. This is presented below again for "random_cluster" error term but generalizes across other error specifications.

```python
consistency = pd.DataFrame(index=["AIC", "JMA"], columns=[100, 200, 360, 600, 1000])
for num_obs in [100, 200, 360, 600, 1000]:
    consistency[[num_obs]] = processed_results_linear_dgp[
        "random_cluster"][str(num_obs)].loc["RMSE", ["AIC", "JMA"]].to_numpy().reshape((2, 1))**2 * num_obs**0.95
consistency
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>100</th>
      <th>200</th>
      <th>360</th>
      <th>600</th>
      <th>1000</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>AIC</th>
      <td>0.286025</td>
      <td>0.25876</td>
      <td>0.221922</td>
      <td>0.202671</td>
      <td>0.179699</td>
    </tr>
    <tr>
      <th>JMA</th>
      <td>0.179326</td>
      <td>0.188912</td>
      <td>0.18692</td>
      <td>0.177417</td>
      <td>0.1622</td>
    </tr>
  </tbody>
</table>
</div>

The theoretical result which is proven by Zhang (2015) for JMA seems to be confirmed in my simulation. It appears that $1000$ simulation runs are not sufficiently many runs to show this consistently from $N$ equal to 100 onwards but rather I obtain it solely from $N=200$ onwards. Although not proven by Zhang (2015) but maybe somewhere else, root-n consistency seems to hold for AIC model selection in my example.

I adapt the previous simulation study now such that the true data generating process is actually a fourth order polynomial. The coefficients now are the following:

\begin{equation}
    \begin{split}
        P &= 4 \\
        \alpha_{1p} &= (-0.05, -0.00016, -0.00006, -5e^{-6}, -5e^{-8})' \\
        \alpha_{2p} &= (0, 0)' \\
        \beta_{1p} &= (0.08, 0.00002, 0.00009, -8e^{-6}, -8e^{-8})' \\
        \beta_{2p} &= (0, 0)'.
    \end{split}
\end{equation}

In order to estimate the treatment effect, I run now the same Monte Carlo simulation as above with only the data generating process having changed. This means that the true model is now not among those considered in my estimation strategy anymore. The theoretical consistency proven by Zhang (2015) does not hold in this scenario. I want to check if we can still find it in practice, though, with this rather mild deviation of the largest model considered from the true data generating process. Further I want to check whether the difference in bias between AIC and JMA stays equally large or whether JMA catches up to AIC or might be even better as now confounding because of overfitting is not an issue anymore. JMA might be expected to even outperform AIC as JMA will likely always add some information from the "best" model which is the third order polynomial. AIC on the other hand will likely choose some other orders sometimes which leads to loss of information and stronger bias as when some information from the third order was included. Lastly, I want to see if the coverage probability for JMA stays higher and close to the nominal level, although now the true model is outside the space of those considered. Before we run the simulation study, let us have a look how the averaged dependent variable relates to the assignment variable for small firms now (again for $N=360$ and clustered error terms).

```python
# simulate data
np.random.seed(123)
num_obs = 360
polynomials = 4
coefficients = {
    "untreated": np.array([-0.05, -0.00016, -0.00006, -5e-6, -5e-8]),
    "treated": np.array([0.08, 0.00002, 0.00009, -8e-6, -8e-8])}
simulated_data = simulate_data(num_obs, coefficients, polynomials, error_dist="random_cluster")[0]
```

```python
# plot small firms averaged scaled investment on the score
small = simulated_data.loc[simulated_data["large"] == 0, :]
mean_small = small.groupby("score").mean()
mean_small.reset_index(inplace=True)
fig, ax = plt.subplots()
sns.scatterplot(mean_small["score"], mean_small["scaled_investment"], ax=ax)
ax.axvline()
sns.regplot(mean_small.loc[mean_small["score"]<0, "score"], mean_small.loc[mean_small["score"]<0, "scaled_investment"],
           order=3, scatter=False, ci=None)
sns.regplot(mean_small.loc[mean_small["score"]>=0, "score"], mean_small.loc[mean_small["score"]>=0, "scaled_investment"],
           order=3, scatter=False, ci=None, color=sns.color_palette("tab10")[0])
```

It is clearly visible now that the effect is nonlinear, when fitting a cubic polynomial. We are now ready to run the described simulation below.

```python
# If we are running on TRAVIS-CI we will simply load a file with existing results.
# just load the results after if statement if the simulation might take too long for you (I had to run it over night)...
if os.environ.get("TRAVIS"):
    results_nonlinear_dgp = pickle.load(open("data/simulated/results_nonlinear.pickle", "br"))
    processed_results_nonlinear_dgp = pickle.load(open(
        "data/simulated/processed_results_nonlinear.pickle", "br"))
else:
    # set up simulation
    np.random.seed(123)
    num_runs = 1000
    num_bootstrap_runs = 200
    true_treatment_effect = 0.08
    true_model = {
        "polynomials": 4,
        "coefficients": {
            "untreated": np.array([-0.05, -0.00016, -0.00006, -5e-6, -5e-8]),
            "treated": np.array([0.08, 0.00002, 0.00009, -8e-6, -8e-8]),
        },
    }
    # run simulation
    results_nonlinear_dgp = {}
    processed_results_nonlinear_dgp = {}
    for error_distr in ["homoskedastic", "random_cluster" ,"normal", "inverse"]:
        results_nonlinear_dgp[error_distr] = {}
        processed_results_nonlinear_dgp[error_distr] = {}
        for num_obs in [100, 200, 360, 600, 1000]:
            results_nonlinear_dgp[error_distr][str(num_obs)] = get_results_regression(
                num_runs, num_obs, num_bootstrap_runs, true_model, error_distr
            )
            processed_results_nonlinear_dgp[error_distr][str(num_obs)] = process_results(
                results_nonlinear_dgp[error_distr][str(num_obs)], true_treatment_effect
            )
```

Below I shortly present the examplary case of $N=360$ and clustered errors. What is interesting to see here is that the actual coverage probablity of JMA is higher than for any of the single models and that it still corresponds to its nominal level leaving AIC clearly behind. Also, here, JMA seems to also perform better than AIC looking at the bias for this specific case.

```python
processed_results_nonlinear_dgp["random_cluster"]["360"]
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>AIC</th>
      <th>JMA</th>
      <th>polynomial_0</th>
      <th>polynomial_1</th>
      <th>polynomial_2</th>
      <th>polynomial_3</th>
    </tr>
    <tr>
      <th>Statistic</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>Treatment Effect</th>
      <td>0.075707</td>
      <td>0.080547</td>
      <td>0.014510</td>
      <td>0.127161</td>
      <td>0.071150</td>
      <td>0.076685</td>
    </tr>
    <tr>
      <th>Bias</th>
      <td>-0.004293</td>
      <td>0.000547</td>
      <td>-0.065490</td>
      <td>0.047161</td>
      <td>-0.008850</td>
      <td>-0.003315</td>
    </tr>
    <tr>
      <th>Standard Error</th>
      <td>0.034648</td>
      <td>0.032097</td>
      <td>0.013089</td>
      <td>0.021502</td>
      <td>0.028219</td>
      <td>0.036201</td>
    </tr>
    <tr>
      <th>RMSE</th>
      <td>0.034913</td>
      <td>0.032102</td>
      <td>0.066785</td>
      <td>0.051832</td>
      <td>0.029575</td>
      <td>0.036352</td>
    </tr>
    <tr>
      <th>95% Coverage</th>
      <td>0.863000</td>
      <td>0.952000</td>
      <td>0.001000</td>
      <td>0.390000</td>
      <td>0.926000</td>
      <td>0.936000</td>
    </tr>
    <tr>
      <th>CI Width</th>
      <td>0.107287</td>
      <td>0.124099</td>
      <td>0.055363</td>
      <td>0.081085</td>
      <td>0.105990</td>
      <td>0.135847</td>
    </tr>
  </tbody>
</table>
</div>

To get a better picture, though, of how JMA and AIC actually perform here, let us compare the clustered arror case across different sample sizes. In the table below I present how the treatment effect and the coverage probability develops with increasing $N$ and whether there is any root-n consistency.

```python
statistics = ["Treatment Effect", "root-n MSE", "95% Coverage"]
approaches = ["AIC", "JMA"]
index = pd.MultiIndex.from_product([statistics, approaches], names=["Statistic", "Approach"])
consistency = pd.DataFrame(index=index, columns=[100, 200, 360, 600, 1000])
for num_obs in [100, 200, 360, 600, 1000]:
    consistency.loc["root-n MSE", num_obs] = processed_results_nonlinear_dgp[
        "random_cluster"][str(num_obs)].loc["RMSE", ["AIC", "JMA"]].to_numpy().reshape((2, 1))**2 * num_obs**0.95
    consistency.loc[["Treatment Effect", "95% Coverage"], num_obs] = processed_results_nonlinear_dgp[
        "random_cluster"][str(num_obs)].loc[["Treatment Effect", "95% Coverage"], ["AIC", "JMA"]].to_numpy().reshape(4, 1)
consistency
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th></th>
      <th>100</th>
      <th>200</th>
      <th>360</th>
      <th>600</th>
      <th>1000</th>
    </tr>
    <tr>
      <th>Statistic</th>
      <th>Approach</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th rowspan="2" valign="top">Treatment Effect</th>
      <th>AIC</th>
      <td>0.0912693</td>
      <td>0.080805</td>
      <td>0.0757073</td>
      <td>0.0717522</td>
      <td>0.0730202</td>
    </tr>
    <tr>
      <th>JMA</th>
      <td>0.0847633</td>
      <td>0.0823432</td>
      <td>0.0805469</td>
      <td>0.0759521</td>
      <td>0.0756255</td>
    </tr>
    <tr>
      <th rowspan="2" valign="top">root-n MSE</th>
      <th>AIC</th>
      <td>0.406662</td>
      <td>0.378382</td>
      <td>0.326926</td>
      <td>0.307912</td>
      <td>0.316256</td>
    </tr>
    <tr>
      <th>JMA</th>
      <td>0.2663</td>
      <td>0.280669</td>
      <td>0.276411</td>
      <td>0.270257</td>
      <td>0.276823</td>
    </tr>
    <tr>
      <th rowspan="2" valign="top">95% Coverage</th>
      <th>AIC</th>
      <td>0.733</td>
      <td>0.808</td>
      <td>0.863</td>
      <td>0.898</td>
      <td>0.898</td>
    </tr>
    <tr>
      <th>JMA</th>
      <td>0.95</td>
      <td>0.946</td>
      <td>0.952</td>
      <td>0.94</td>
      <td>0.938</td>
    </tr>
  </tbody>
</table>
</div>

We can see for treatment effect and $N^{0.95} \times MSE$, that for both approaches no clear convergence can be found. In this case, though, the treatment effect estimates of JMA seem to be more stable than those of AIC. Interestingly, the coverage probability of JMA decreases with larger sample size while it increases for AIC. Generally, JMA has again a coverage probability close to its nominal level outperforming AIC. Both of those findings are robust across the four different error distributions considered. For the other error distributions I also do not find any clear convergence pattern for treatment effect and the MSE criterion. For all four distributions, JMA performs on average a bit better than AIC regarding bias as it was expected.

Brought together this first simulation study shows that JMA outperforms AIC regarding coverage probability bringing the Type I error to its nominal level when testing for a nonzero treatment effect. Above that, the second part indicates that JMA seems to show a lower bias when the true model is not among those considered but one of the estimated models is rather close to the true one. This is a feature that might likely be found in RDD where a high order polynomial is often a good approximation of the true model but it probably is not exactly among those considered. This shows that JMA has some appealing features that might be useful to make the estimation of treatment effect and the following inference more reliable in comparison to the usually employed model selection based on the AIC.

## Simulation Study Two

As noted in my introduction, it is common practice in RDD to rather run local linear regressions to determine the best bandwidth around the cutoff instead of solely estimating polynomial regressions across the whole bandwidth. This is motivated by the intuitive notion that as one is interested in the difference in $Y$ close to the cutoff one should also only rely on data close to it. As described in the section on RDD, Imbens and Lemieux (2008) suggest a bandwidth selection that is based on a cross validation criterion which is supposed to approximate the expected test mean squared error across different bandwidths based on local linear regressions. The approach of Hansen and Racine (2012) has also the objective to approximate the expected test MSE with their least squares cross validation criterion. This observation induces me to try whether JMA and its criterion might be used to determine the optimal bandwidth allowing to average over polynomials of higher order than just running a regression with poylnomial of order (as done with local linear regressions). Hansen and Racine (2012) argue that their approach can serve for local polynomial regressions with fixed bandwidth while I want to go beyond it suggesting that it can even choose the bandwidth. As the linear regression is part of the models the JMA averages over, I expect the JMA approach to at least perform as well as the linear approach when the linear approximation is at least imperfect based on the observations from the previous simulation study.

My idea is that the approach could even perform better when there is a clear indication that the effect of the assignment variable is not linear locally around the cutoff (Imbens and Lemieux (2008) justify their approach by arguing that the linear approximation is typically a good one locally). In Bronzini and Iachini (2014), the authors opt for a sole robustness check based on a nonlinear kernel estimation for two fixed bandwidths, $h \in \{15, 30\}$. This indicates that also the authors did not necessarily believe in a good linear approximation locally. On the other hand, the fact that it is only a robustness check, though, displays that the setup in which they are is not typically well-suited for nonparametric methods such as local linear regressions. The reason is given by Lee and Lemieux (2010) who argue that RDD with discrete assignment variables (on top of that with a small sample, here) do not yield a good foundation for local methods as the amount of data around the cutoff is rather low. As I do not want to introduce another data generating process (due to size constraints) I run a comparison between local linear regression and JMA for bandwidth selection based on the setup in Bronzini and Iachini (2014) anyways with the idea of a good performance in this setup a good performance in more favourable setups should follow. I tested at which minimum bandwidth the resulting estimated treatment effects are reasonable and found that $h=10$ works well. Consequently, I run a bandwidth selection where I consider all integer bandwidths between 10 and 35. I add five more integers to each side of the bandwidths considered by Bronzini and Iachini. Again I run 1000 simulations in which I first simulate a data set with the same data generating process as in the last simulation, i.e. with $P=4$. Then I run local linear regressions as suggested in Imbens and Lemieux (2008) as well as JMA as specified before across all bandwidths. This yields the respective criterion per approach and per bandwidth among which I choose the bandwidth with the smallest criterion value. No matter which approach selected the bandwidth, I then estimate the treatment effect with JMA to not confound the results by using different estimation techniques for the treatment effect. For every run I report which bandwidth was selected, what the minimum of the criterion is, how high the estimated treatment effect is and how long it took to find the optimal bandwidth. As the effect of score on scaled investment is nonlinear and this seems to also persist locally (when looking abck at my figure of the last simulation), I expect JMA to be at least as good as local linear regressions in selecting the optimal bandwidth.

The main reason I added the different error distributions is for this second simulation study. I want to observe if JMA can follow local linear regressions with good selection across different specifications. In the inverse case one expects that both approaches should select a bandwidth close to the smallest possible ($h_{min}=10$) as the error is small there. For the "normal" case this should be the other way around. For homoskedasticity and especially for the clustered case I hope to obtain many different choices of bandwidth as the random error distribution across scores lets the effect of score appear more or less linear across scores. JMA might exploit the high order polynomials sometimes where the expected test MSE for the local linear regression is not the lowest due to an imperfect linear approximation for instance. I run the described simulation below.

```python
# If we are running on TRAVIS-CI we will simply load a file with existing results.
# just load the results after if statement if the simulation might take too long for you (I had to run it over night)...
if os.environ.get("TRAVIS"):
    results_local = pickle.load(open("data/simulated/results_local.pickle", "br"))
    processed_results_local = pickle.load(
        open("data/simulated/processed_results_local.pickle", "br"))
else:
    # set up simulation
    np.random.seed(123)
    num_runs = 1000
    num_obs = 360
    start_local = 10
    start_jma = 10
    width = 35
    true_model = {
        "polynomials": 4,
        "coefficients": {
            "untreated": np.array([-0.05, -0.00016, -0.00006, -5e-6, -5e-8]),
            "treated": np.array([0.08, 0.00002, 0.00009, -8e-6, -8e-8]),
        },
    }
    subset = np.array([np.arange(2), np.arange(4), np.arange(6), np.arange(8)])
    true_treatment_effect = 0.08

    # run simulation
    results = {}
    processed_results = {}
    for error_dist in ["homoskedastic", "random_cluster" ,"normal", "inverse"]:
        results[error_dist] = get_results_local_regression(
            num_runs,
            num_obs,
            true_model,
            start_local,
            start_jma,
            width,
            subset,
            error_dist=error_dist,
        )
        processed_results[error_dist] = process_results_local(
                results[error_dist], true_treatment_effect
            )
```

The results from the above simulation are processed such that I can present the average treatment effect, bias, standard error, mean squared error, expected MSE (according to the criterion of the approach), time, bandwidth and the percentage of times both approaches chose the same bandwidth. These processed results are shown below for all different error distributions.

```python
# show results
for error_dist in ["homoskedastic", "random_cluster" ,"normal", "inverse"]:
    display(error_dist)
    display(processed_results_local[error_dist])
```

    'homoskedastic'

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Treatment Effect</th>
      <th>Bias</th>
      <th>Standard Error</th>
      <th>MSE</th>
      <th>Expected MSE</th>
      <th>Time</th>
      <th>Same Bandwidth</th>
      <th>Bandwidth</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>JMA</th>
      <td>0.078796</td>
      <td>-0.001204</td>
      <td>0.020181</td>
      <td>0.000409</td>
      <td>0.006265</td>
      <td>0.747767</td>
      <td>0.617</td>
      <td>16.784</td>
    </tr>
    <tr>
      <th>local linear</th>
      <td>0.079480</td>
      <td>-0.000520</td>
      <td>0.019133</td>
      <td>0.000366</td>
      <td>0.006415</td>
      <td>3.125081</td>
      <td>0.617</td>
      <td>16.941</td>
    </tr>
  </tbody>
</table>
</div>

    'random_cluster'

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Treatment Effect</th>
      <th>Bias</th>
      <th>Standard Error</th>
      <th>MSE</th>
      <th>Expected MSE</th>
      <th>Time</th>
      <th>Same Bandwidth</th>
      <th>Bandwidth</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>JMA</th>
      <td>0.079207</td>
      <td>-0.000793</td>
      <td>0.021904</td>
      <td>0.000480</td>
      <td>0.005906</td>
      <td>0.743843</td>
      <td>0.611</td>
      <td>16.818</td>
    </tr>
    <tr>
      <th>local linear</th>
      <td>0.079815</td>
      <td>-0.000185</td>
      <td>0.020579</td>
      <td>0.000424</td>
      <td>0.006084</td>
      <td>3.125983</td>
      <td>0.611</td>
      <td>16.965</td>
    </tr>
  </tbody>
</table>
</div>

    'normal'

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Treatment Effect</th>
      <th>Bias</th>
      <th>Standard Error</th>
      <th>MSE</th>
      <th>Expected MSE</th>
      <th>Time</th>
      <th>Same Bandwidth</th>
      <th>Bandwidth</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>JMA</th>
      <td>0.078088</td>
      <td>-0.001912</td>
      <td>0.025937</td>
      <td>0.000676</td>
      <td>0.008272</td>
      <td>0.748410</td>
      <td>0.767</td>
      <td>33.006</td>
    </tr>
    <tr>
      <th>local linear</th>
      <td>0.078358</td>
      <td>-0.001642</td>
      <td>0.025119</td>
      <td>0.000634</td>
      <td>0.008485</td>
      <td>3.128192</td>
      <td>0.767</td>
      <td>32.404</td>
    </tr>
  </tbody>
</table>
</div>

    'inverse'

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Treatment Effect</th>
      <th>Bias</th>
      <th>Standard Error</th>
      <th>MSE</th>
      <th>Expected MSE</th>
      <th>Time</th>
      <th>Same Bandwidth</th>
      <th>Bandwidth</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>JMA</th>
      <td>0.078474</td>
      <td>-0.001526</td>
      <td>0.013493</td>
      <td>0.000184</td>
      <td>0.00427</td>
      <td>0.750192</td>
      <td>0.978</td>
      <td>10.029</td>
    </tr>
    <tr>
      <th>local linear</th>
      <td>0.078488</td>
      <td>-0.001512</td>
      <td>0.013525</td>
      <td>0.000185</td>
      <td>0.00429</td>
      <td>3.115280</td>
      <td>0.978</td>
      <td>10.035</td>
    </tr>
  </tbody>
</table>
</div>

We can see that indeed for the "normal" and the "inverse" case the frequency of equal bandwidths chosen is the highest. As expected both approaches choose small bandwidth for the "inverse" and large bandwidth for the "normal" error distribution. Apart from that both approaches give very similar results for the treatment effect across all distributions while the local linear regression approach is slightly better. Despite that for homoskedastic and clustered errors only in around 60 percent of the cases the same bandwidth is chosen, the average bandwidth for both approaches is very similar. Unfortunately, within my already nonlinear example, the local linear approximation seems to work reasonably well such that JMA cannot improve the results. At the same time, we can also see that JMA does not do harm here neither which might still underline its strength by being as performant as local linear regressions while displaying more flexibility regarding rather nonlinear relations. It would be interesting to see how JMA reacts when the underlying data generating process is actually linear. Even more interesting might be whether one can come up with an example that is locally nonlinear such that it actually outperforms the linear approach. As my simulations are already long I have to conclude with my last simulation, though, and maybe come back to that in a future project. As a side remark for this last simulation it is worth noting that the efficient way of performing leave one out cross validation for linear regressions based on Hansen and Racine (2012) makes the approach way faster than the local linear regression in which I coded up the cross validation conventionally.

One should mention, though, that despite the fact that the assignment variable is discrete, the local approaches work well and Bronzini and Iachini should have performed a proper bandwidth selection as opposed to simply selecting two bandwidths as a robustness check and arguing with non-local polynomial regressions selected via AIC as their main results. This can be seen by the fact how well both local approaches recover the true treatment effect here in comparison to the same data generating process estimated via polynomial regressions before. Further it would have given them more credibility as only more local data around the cutoff is used.

## Conclusion

My project shows that the typical method of choosing the optimal order of polynomial for the assignment variable in RDD based on model selection via AIC leads to too optimistic certainty of the estimated treatment effect. This results in clearly understated type I error. Jackknife Model Averaging can be a valid alternative in that regard as it appears that the confidence intervals reach their nominal coverage irrespective of whether the correct model is among those averaged over. At the same time, the bias encountered by AIC and JMA is similar when the true model is not among those considered by the researcher but at least one of the models is a good aproximation of the true one. This can be imagined to be a typical situation in RDD when deciding on the poylnomial order of the assignment variable. With this in mind, JMA should maybe be even the prefered approach.

The second finding of my project is that without relying on theory but solely on my simulation evidence, JMA seems to be an alternative to local linear regressions for bandwidth selection in RDD. I show that when the true model is not among the candidates (but one considered model is a good proxy) and the effect of the assignment variable is generally nonlinear, JMA yields very similar results to the local linear approach and both can recover the true treatment effect quite well on average. Although not shown, I argue that it could be imagineable to find situations in which JMA might be even prefered over the local linear approach. I imagine such a situation to involve considerable nonlinearity locally that potentially varies a lot with the selected bandwidth.

### References

Bronzini, R., & Iachini, E. (2014). Are incentives for R&D effective? Evidence from a regression discontinuity approach. *American Economic Journal: Economic Policy, 6*(4), 100-134.

Button, P. (2016). Model uncertainty and model averaging in regression discontinuity designs. *Journal of Econometric Methods, 5*(1), 103-116.

Davison, A. C. and D. V. Hinkley (1997). Bootstrap Methods and their Application. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press.

Fletcher, D. (2019). Model Averaging. Springer.

Hansen, B. E., & Racine, J. S. (2012). Jackknife model averaging. *Journal of Econometrics, 167*(1), 38-46.

Imbens, G. W., & Lemieux, T. (2008). Regression discontinuity designs: A guide to practice. *Journal of econometrics, 142*(2), 615-635.

Lee, D. S., & Lemieux, T. (2010). Regression discontinuity designs in economics. *Journal of economic literature, 48*(2), 281-355.

Li, K. C. (1987). Asymptotic optimality for Cp, CL, cross-validation and generalized cross-validation: discrete index set. *The Annals of Statistics, 958-975*.

Moral‐Benito, E. (2015). Model averaging in economics: An overview. *Journal of Economic Surveys, 29*(1), 46-75.

Steel, M. F. (forthcoming). Model Averaging and Its Use in Economics. *Journal of Economic Literature*.

Stone, M. (1974). Cross‐validatory choice and assessment of statistical predictions. *Journal of the Royal Statistical Society: Series B (Methodological), 36*(2), 111-133.

Wolpert, D. H. (1992). Stacked generalization. *Neural networks, 5*(2), 241-259.


# Start-to-Finish Example: Unit Testing `GiRaFFE_NRPy`: Interpolating Metric Face-Values

## Author: Patrick Nelson

## This module Validates the `FCVAL` routine for `GiRaFFE`.

**Notebook Status:** Validated

**Validation Notes:** This module will validate the routines in [Tutorial-GiRaFFE_NRPy-Metric_Face_Values](Tutorial-GiRaFFE_NRPy-Metric_Face_Values.ipynb).

### NRPy+ Source Code for this module:
* [GiRaFFE_NRPy/GiRaFFE_NRPy_Metric_Face_Values.py](../../edit/in_progress/GiRaFFE_NRPy/GiRaFFE_NRPy_Metric_Face_Values.py) [\[**tutorial**\]](Tutorial-GiRaFFE_NRPy-FCVAL.ipynb) Generates the C code to interpolate gridfunctions to cell faces.

## Introduction:

This notebook validates the code that will interpolate the metric gridfunctions on cell faces. These values, along with the reconstruction of primitive variables on the faces, are necessary for the Riemann solvers to compute the fluxes through the cell faces.

It is, in general, good coding practice to unit test functions individually to verify that they produce the expected and intended output. We will generate test data with arbitrarily-chosen analytic functions and calculate gridfunctions at the cell centers on a small numerical grid. We will then compute the values on the cell faces in two ways: first, with our interpolator, then second, we will shift the grid and compute them analytically. Then, we will rerun the function at a finer resolution. Finally, we will compare the results of the two runs to show fourth-order convergence.

When this notebook is run, the difference between the approximate and exact metric gridfunctions will be output to text files that can be found in the same directory as this notebook. These will be read in in [Step 3](#convergence), and used there to confirm convergence order of the algorithm.

# Table of Contents
$$\label{toc}$$

This notebook is organized as follows

1. [Step 1](#setup): Set up core functions and parameters for unit testing the FCVAL algorithm
    1. [Step 1.a](#expressions) Write expressions for the metric gridfunctions
    1. [Step 1.b](#ccodekernels) Generate C functions to calculate the gridfunctions
    1. [Step 1.c](#free_parameters) Set free parameters in the code
1. [Step 2](#mainc): `FCVAL_unit_test.c`: The Main C Code
    1. [Step 2.a](#compile_run): Compile and run the code
1. [Step 3](#convergence): Code validation: Verify that relative error in numerical solution converges to zero at the expected order
1. [Step 4](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Step 1: Set up core functions and parameters for unit testing the FCVAL algorithm \[Back to [top](#toc)\]
$$\label{setup}$$

We'll start by appending the relevant paths to `sys.path` so that we can access sympy modules in other places. Then, we'll import NRPy+ core functionality and set up a directory in which to carry out our test.

```python
import os, sys           # Standard Python modules for multiplatform OS-level functions
# First, we'll add the parent directory to the list of directories Python will check for modules.
nrpy_dir_path = os.path.join("..")
if nrpy_dir_path not in sys.path:
    sys.path.append(nrpy_dir_path)
nrpy_dir_path = os.path.join("..","..")
if nrpy_dir_path not in sys.path:
    sys.path.append(nrpy_dir_path)

from outputC import outCfunction, lhrh # NRPy+: Core C code output module
import sympy as sp               # SymPy: The Python computer algebra package upon which NRPy+ depends
import finite_difference as fin  # NRPy+: Finite difference C code generation module
import NRPy_param_funcs as par   # NRPy+: Parameter interface
import grid as gri               # NRPy+: Functions having to do with numerical grids
import indexedexp as ixp         # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support
import reference_metric as rfm   # NRPy+: Reference metric support
import cmdline_helper as cmd     # NRPy+: Multi-platform Python command-line interface

out_dir = "Validation/"
cmd.mkdir(out_dir)
subdir = "FCVAL"
cmd.mkdir(os.path.join(out_dir,subdir))

thismodule = "Start_to_Finish_UnitTest-GiRaFFE_NRPy-Metric_Face_Values"

# Set the finite-differencing order to 2
par.set_parval_from_str("finite_difference::FD_CENTDERIVS_ORDER", 2)

```

## Step 1.a: Write expressions for the metric gridfunctions \[Back to [top](#toc)\]
$$\label{expressions}$$

Now, we'll choose some functions with arbitrary forms to generate test data. We'll need to set ten gridfunctions, so expressions are being pulled from several previously written unit tests.

\begin{align}
\gamma_{xx} &= ax^3 + by^3 + cz^3 + dy^2 + ez^2 + f \\
\gamma_{yy} &= gx^3 + hy^3 + lz^3 + mx^2 + nz^2 + p \\
\gamma_{zz} &= px^3 + qy^3 + rz^3 + sx^2 + ty^2 + u. \\
\gamma_{xy} &= a \exp\left(-\left((x-b)^2+(y-c)^2+(z-d)^2\right)\right) \\
\gamma_{xz} &= f \exp\left(-\left((x-g)^2+(y-h)^2+(z-l)^2\right)\right) \\
\gamma_{yz} &= m \exp\left(-\left((x-n)^2+(y-o)^2+(z-p)^2\right)\right), \\
\beta^x &= \frac{2}{\pi} \arctan(ax + by + cz) \\
\beta^y &= \frac{2}{\pi} \arctan(bx + cy + az) \\
\beta^z &= \frac{2}{\pi} \arctan(cx + ay + bz) \\
\alpha &= 1 - \frac{1}{2+x^2+y^2+z^2} \\
\end{align}

```python
a,b,c,d,e,f,g,h,l,m,n,o,p,q,r,s,t,u = par.Cparameters("REAL",thismodule,["a","b","c","d","e","f","g","h","l","m","n","o","p","q","r","s","t","u"],1e300)
M_PI  = par.Cparameters("#define",thismodule,["M_PI"], "")

gammaDD = ixp.register_gridfunctions_for_single_rank2("AUXEVOL","gammaDD","sym01",DIM=3)
betaU = ixp.register_gridfunctions_for_single_rank1("AUXEVOL","betaU",DIM=3)
alpha = gri.register_gridfunctions("AUXEVOL","alpha")

par.set_parval_from_str("reference_metric::CoordSystem","Cartesian")
rfm.reference_metric()
x = rfm.xxCart[0]
y = rfm.xxCart[1]
z = rfm.xxCart[2]

gammaDD[0][0] = a*x**3 + b*y**3 + c*z**3 + d*y**2 + e*z**2 + f
gammaDD[1][1] = g*x**3 + h*y**3 + l*z**3 + m*x**2 + n*z**2 + o
gammaDD[2][2] = p*x**3 + q*y**3 + r*z**3 + s*x**2 + t*y**2 + u
gammaDD[0][1] = a * sp.exp(-((x-b)**2 + (y-c)**2 + (z-d)**2))
gammaDD[0][2] = f * sp.exp(-((x-g)**2 + (y-h)**2 + (z-l)**2))
gammaDD[1][2] = m * sp.exp(-((x-n)**2 + (y-o)**2 + (z-p)**2))

betaU[0] = (sp.sympify(2)/M_PI) * sp.atan(a*x + b*y + c*z)
betaU[1] = (sp.sympify(2)/M_PI) * sp.atan(b*x + c*y + a*z)
betaU[2] = (sp.sympify(2)/M_PI) * sp.atan(c*x + a*y + b*z)

alpha = sp.sympify(1) - sp.sympify(1) / (sp.sympify(2) + x**2 + y**2 + z**2)
```

## Step 1.b: Generate C functions to calculate the gridfunctions \[Back to [top](#toc)\]
$$\label{ccodekernels}$$

Here, we will use the NRPy+ function `outCfunction()` to generate C code that will calculate our metric gridfunctions over an entire grid. We will also call the function to generate the function we are testing.

```python
metric_gfs_to_print = [\
                       lhrh(lhs=gri.gfaccess("aux_gfs","gammaDD00"),rhs=gammaDD[0][0]),\
                       lhrh(lhs=gri.gfaccess("aux_gfs","gammaDD01"),rhs=gammaDD[0][1]),\
                       lhrh(lhs=gri.gfaccess("aux_gfs","gammaDD02"),rhs=gammaDD[0][2]),\
                       lhrh(lhs=gri.gfaccess("aux_gfs","gammaDD11"),rhs=gammaDD[1][1]),\
                       lhrh(lhs=gri.gfaccess("aux_gfs","gammaDD12"),rhs=gammaDD[1][2]),\
                       lhrh(lhs=gri.gfaccess("aux_gfs","gammaDD22"),rhs=gammaDD[2][2]),\
                       lhrh(lhs=gri.gfaccess("aux_gfs","betaU0"),rhs=betaU[0]),\
                       lhrh(lhs=gri.gfaccess("aux_gfs","betaU1"),rhs=betaU[1]),\
                       lhrh(lhs=gri.gfaccess("aux_gfs","betaU2"),rhs=betaU[2]),\
                       lhrh(lhs=gri.gfaccess("aux_gfs","alpha"),rhs=alpha),\
                      ]

desc = "Calculate the metric gridfunctions"
name = "calculate_metric_gfs"
outCfunction(
    outfile  = os.path.join(out_dir,name+".h"), desc=desc, name=name,
    params   ="const paramstruct *restrict params,REAL *restrict xx[3],REAL *restrict auxevol_gfs",
    body     = fin.FD_outputC("returnstring",metric_gfs_to_print,params="outCverbose=False").replace("IDX4","IDX4S"),
    loopopts="AllPoints,Read_xxs")

import GiRaFFE_NRPy.GiRaFFE_NRPy_Metric_Face_Values as FCVAL
FCVAL.GiRaFFE_NRPy_FCVAL(os.path.join(out_dir,subdir))

```

    Output C function calculate_metric_gfs() to file Validation/calculate_metric_gfs.h

## Step 1.c: Set free parameters in the code \[Back to [top](#toc)\]
$$\label{free_parameters}$$

We also need to create the files that interact with NRPy's C parameter interface.

```python
# Step 3.d.i: Generate declare_Cparameters_struct.h, set_Cparameters_default.h, and set_Cparameters[-SIMD].h
# par.generate_Cparameters_Ccodes(os.path.join(out_dir))

# Step 3.d.ii: Set free_parameters.h
with open(os.path.join(out_dir,"free_parameters.h"),"w") as file:
    file.write("""
// Override parameter defaults with values based on command line arguments and NGHOSTS.
params.Nxx0 = atoi(argv[1]);
params.Nxx1 = atoi(argv[2]);
params.Nxx2 = atoi(argv[3]);
params.Nxx_plus_2NGHOSTS0 = params.Nxx0 + 2*NGHOSTS;
params.Nxx_plus_2NGHOSTS1 = params.Nxx1 + 2*NGHOSTS;
params.Nxx_plus_2NGHOSTS2 = params.Nxx2 + 2*NGHOSTS;
// Step 0d: Set up space and time coordinates
// Step 0d.i: Declare \Delta x^i=dxx{0,1,2} and invdxx{0,1,2}, as well as xxmin[3] and xxmax[3]:
const REAL xxmin[3] = {-1.0,-1.0,-1.0};
const REAL xxmax[3] = { 1.0, 1.0, 1.0};

params.dxx0 = (xxmax[0] - xxmin[0]) / ((REAL)params.Nxx0);
params.dxx1 = (xxmax[1] - xxmin[1]) / ((REAL)params.Nxx1);
params.dxx2 = (xxmax[2] - xxmin[2]) / ((REAL)params.Nxx2);
printf("dxx0,dxx1,dxx2 = %.5e,%.5e,%.5e\\n",params.dxx0,params.dxx1,params.dxx2);
params.invdx0 = 1.0 / params.dxx0;
params.invdx1 = 1.0 / params.dxx1;
params.invdx2 = 1.0 / params.dxx2;
\n""")

# Generates declare_Cparameters_struct.h, set_Cparameters_default.h, and set_Cparameters[-SIMD].h
par.generate_Cparameters_Ccodes(os.path.join(out_dir))
```

# Step 2: `FCVAL_unit_test.c`: The Main C Code \[Back to [top](#toc)\]
$$\label{mainc}$$

```python
%%writefile $out_dir/FCVAL_unit_test.c
// These are common packages that we are likely to need.
#include "stdio.h"
#include "stdlib.h"
#include "math.h"
#include "string.h" // Needed for strncmp, etc.
#include "stdint.h" // Needed for Windows GCC 6.x compatibility
#include <time.h>   // Needed to set a random seed.

#define REAL double
#include "declare_Cparameters_struct.h"

const int NGHOSTS = 3;

REAL a,b,c,d,e,f,g,h,l,m,n,o,p,q,r,s,t,u;

// Standard NRPy+ memory access:
#define IDX4S(g,i,j,k) \
( (i) + Nxx_plus_2NGHOSTS0 * ( (j) + Nxx_plus_2NGHOSTS1 * ( (k) + Nxx_plus_2NGHOSTS2 * (g) ) ) )

const int kronecker_delta[4][3] = { { 0,0,0 },
                                    { 1,0,0 },
                                    { 0,1,0 },
                                    { 0,0,1 } };

// Give gridfunctions their names:
#define GAMMADD00GF 0
#define GAMMADD01GF 1
#define GAMMADD02GF 2
#define GAMMADD11GF 3
#define GAMMADD12GF 4
#define GAMMADD22GF 5
#define BETAU0GF 6
#define BETAU1GF 7
#define BETAU2GF 8
#define ALPHAGF 9
#define GAMMA_FACEDD00GF 10
#define GAMMA_FACEDD01GF 11
#define GAMMA_FACEDD02GF 12
#define GAMMA_FACEDD11GF 13
#define GAMMA_FACEDD12GF 14
#define GAMMA_FACEDD22GF 15
#define BETA_FACEU0GF 16
#define BETA_FACEU1GF 17
#define BETA_FACEU2GF 18
#define ALPHA_FACEGF 19
#define GAMMAUU00GF 20
#define GAMMAUU01GF 21
#define GAMMAUU02GF 22
#define GAMMAUU11GF 23
#define GAMMAUU12GF 24
#define GAMMAUU22GF 25
#define GAMMA_FACEUU00GF 26
#define GAMMA_FACEUU01GF 27
#define GAMMA_FACEUU02GF 28
#define GAMMA_FACEUU11GF 29
#define GAMMA_FACEUU12GF 30
#define GAMMA_FACEUU22GF 31
#define NUM_AUXEVOL_GFS 32

#include "calculate_metric_gfs.h"
#include "FCVAL/interpolate_metric_gfs_to_cell_faces.h"

int main(int argc, const char *argv[]) {
    paramstruct params;
#include "set_Cparameters_default.h"

    // Step 0c: Set free parameters, overwriting Cparameters defaults
    //          by hand or with command-line input, as desired.
#include "free_parameters.h"
#include "set_Cparameters-nopointer.h"

    // Step 0e: Set up cell-centered Cartesian coordinate grids
    REAL *xx[3];
    xx[0] = (REAL *)malloc(sizeof(REAL)*Nxx_plus_2NGHOSTS0);
    xx[1] = (REAL *)malloc(sizeof(REAL)*Nxx_plus_2NGHOSTS1);
    xx[2] = (REAL *)malloc(sizeof(REAL)*Nxx_plus_2NGHOSTS2);
    for(int j=0;j<Nxx_plus_2NGHOSTS0;j++) xx[0][j] = xxmin[0] + (j-NGHOSTS)*dxx0;
    for(int j=0;j<Nxx_plus_2NGHOSTS1;j++) xx[1][j] = xxmin[1] + (j-NGHOSTS)*dxx1;
    for(int j=0;j<Nxx_plus_2NGHOSTS2;j++) xx[2][j] = xxmin[2] + (j-NGHOSTS)*dxx2;

    //for(int i=0;i<Nxx_plus_2NGHOSTS0;i++) printf("xx[0][%d] = %.15e\n",i,xx[0][i]);

    // This is the array to which we'll write the NRPy+ variables.
    REAL *auxevol_gfs  = (REAL *)malloc(sizeof(REAL) * NUM_AUXEVOL_GFS * Nxx_plus_2NGHOSTS2 * Nxx_plus_2NGHOSTS1 * Nxx_plus_2NGHOSTS0);
    // And another for exact data:
    REAL *auxevol_exact_gfs  = (REAL *)malloc(sizeof(REAL) * NUM_AUXEVOL_GFS * Nxx_plus_2NGHOSTS2 * Nxx_plus_2NGHOSTS1 * Nxx_plus_2NGHOSTS0);

    // Generate some random coefficients. Leave the random seed on its default for consistency between trials.
    a = (double)(rand()%20)/5.0;
    f = (double)(rand()%20)/5.0;
    m = (double)(rand()%20)/5.0;
    b = (double)(rand()%10-5)/100.0;
    c = (double)(rand()%10-5)/100.0;
    d = (double)(rand()%10-5)/100.0;
    g = (double)(rand()%10-5)/100.0;
    h = (double)(rand()%10-5)/100.0;
    l = (double)(rand()%10-5)/100.0;
    n = (double)(rand()%10-5)/100.0;
    o = (double)(rand()%10-5)/100.0;
    p = (double)(rand()%10-5)/100.0;

    // First, calculate the test data on our grid:
    calculate_metric_gfs(&params,xx,auxevol_gfs);
    // Now, run our function:
    int flux_dirn = 0;
    interpolate_metric_gfs_to_cell_faces(&params,auxevol_gfs,flux_dirn+1);
    // For comparison, shift the grid half a gridpoint, then calculate the test data again.
    REAL dxx[3] = {dxx0,dxx1,dxx2};
    REAL Nxxp2NG[3] = {Nxx_plus_2NGHOSTS0,Nxx_plus_2NGHOSTS1,Nxx_plus_2NGHOSTS2};
    for(int i=0;i<Nxxp2NG[flux_dirn];i++) {
        xx[flux_dirn][i] -= dxx[flux_dirn]/2.0;
    }
    calculate_metric_gfs(&params,xx,auxevol_exact_gfs);

    char filename[100];
    sprintf(filename,"out%d-numer.txt",Nxx0);
    FILE *out2D = fopen(filename, "w");
    for(int i2=0;i2<Nxx_plus_2NGHOSTS2;i2++) for(int i1=0;i1<Nxx_plus_2NGHOSTS1;i1++) for(int i0=0;i0<Nxx_plus_2NGHOSTS0;i0++) {
        // We print the difference between approximate and exact numbers.
        fprintf(out2D,"%.16e\t %e %e %e\n",
                auxevol_gfs[IDX4S(BETA_FACEU0GF,i0,i1,i2)]-auxevol_exact_gfs[IDX4S(BETAU0GF,i0,i1,i2)],
                xx[0][i0],xx[1][i1],xx[2][i2]
                );
    }
    fclose(out2D);
}
```

    Writing Validation//FCVAL_unit_test.c

## Step 2.a: Compile and run the code \[Back to [top](#toc)\]
$$\label{compile_run}$$

Now that we have our file, we can compile it and run the executable.

```python
import time

print("Now compiling, should take ~2 seconds...\n")
start = time.time()
cmd.C_compile(os.path.join(out_dir,"FCVAL_unit_test.c"), os.path.join(out_dir,"FCVAL_unit_test"))
end = time.time()
print("Finished in "+str(end-start)+" seconds.\n\n")

print("Now running...\n")
start = time.time()
cmd.Execute(os.path.join("Validation","FCVAL_unit_test"), "10 10 10","out.txt")
# To do a convergence test, we'll also need a second grid with twice the resolution.
cmd.Execute(os.path.join("Validation","FCVAL_unit_test"), "20 20 20","out.txt")
end = time.time()
print("Finished in "+str(end-start)+" seconds.\n\n")

```

    Now compiling, should take ~2 seconds...

    Compiling executable...
    (EXEC): Executing `gcc -Ofast -fopenmp -march=native -funroll-loops Validation/FCVAL_unit_test.c -o Validation/FCVAL_unit_test -lm`...
    (BENCH): Finished executing in 0.20867443084716797 seconds.
    Finished compilation.
    Finished in 0.21781635284423828 seconds.

    Now running...

    (EXEC): Executing `taskset -c 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15 ./Validation/FCVAL_unit_test 10 10 10`...
    (BENCH): Finished executing in 0.20765209197998047 seconds.
    (EXEC): Executing `taskset -c 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15 ./Validation/FCVAL_unit_test 20 20 20`...
    (BENCH): Finished executing in 0.20685338973999023 seconds.
    Finished in 0.43360447883605957 seconds.

# Step 3: Code validation: Verify that relative error in numerical solution converges to zero at the expected order \[Back to [top](#toc)\]
$$\label{convergence}$$

Here, we import the data at two resolutions and wrote to text files. This data consists of the absolute error of a metric gridfunction at each in the grid. We'll plot a portion of this data along the axis at the lower resolution along with that same data at the higher resolution scaled to demonstrate that this error converges to 0 at the expected rate. Since our algorithm uses a third-order polynomial, we expect fourth-order convergence here.

```python
import numpy as np
import matplotlib.pyplot as plt

plt.rcParams.update({'font.size': 12})

Data1 = np.loadtxt("out10-numer.txt")
Data2 = np.loadtxt("out20-numer.txt")

def IDX4(i,j,k,Nxx_plus_2NGHOSTS0,Nxx_plus_2NGHOSTS1,Nxx_plus_2NGHOSTS2):
    return (i) + Nxx_plus_2NGHOSTS0 * ( (j) + Nxx_plus_2NGHOSTS1 * ( (k) + Nxx_plus_2NGHOSTS2 * (0) ) )

x1 = np.zeros(10)
a1 = np.zeros(10)
for i in range(10):
    x1[i] = Data1[IDX4(i+3,8,8,16,16,16),1]
    a1[i] = Data1[IDX4(i+3,8,8,16,16,16),0]
x2 = np.zeros(20)
a2 = np.zeros(20)
for i in range(20):
    x2[i] = Data2[IDX4(i+3,13,13,26,26,26),1]
    a2[i] = Data2[IDX4(i+3,13,13,26,26,26),0]

plt.figure()
a = plt.plot(x1,a1,'.',label="dx")
b = plt.plot(x2,a2*(2**4),label="dx/2, times (20/10)^4")
plt.legend()
plt.xlabel("x")
plt.ylabel("alpha")
plt.show()

convergence_satisfactory = np.log2(a1/a2[0:-1:2])>3
if not convergence_satisfactory.all():
    sys.exit(1)
```

# Step 4: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-Start_to_Finish_UnitTest-GiRaFFE_NRPy-Metric_Face_Values.pdf](Tutorial-Start_to_Finish_UnitTest-GiRaFFE_NRPy-Metric_Face_Values.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-Start_to_Finish_UnitTest-GiRaFFE_NRPy-Metric_Face_Values",location_of_template_file=os.path.join(".."))
```

    Created Tutorial-Start_to_Finish_UnitTest-GiRaFFE_NRPy-
        Metric_Face_Values.tex, and compiled LaTeX file to PDF file Tutorial-
        Start_to_Finish_UnitTest-GiRaFFE_NRPy-Metric_Face_Values.pdf


```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
import scipy.special
import scipy.ndimage
import scipy.optimize
```

```python
# helper for gradient checking
def approximate_gradient(x, func, eps=1e-5):
    res = np.zeros(x.size)

    for i in range(x.size):
        d = np.zeros(x.size)
        d[i] = eps

        res[i] = (func(x + d) - func(x - d)) / (2 * eps)

    return res
```

# Chapter 2

## Binary Variables

## Multinomial Variables

## The Gaussian distribution

Marginal and conditional Gaussians (see (2.113) - (2.117)):

$$
\begin{align}
    p(x) &= \mathcal{N}(x|\mu, \Lambda^{-1}) \\
    p(y|x) &= \mathcal{N}(y|A x + b, L^{-1}) \\
    p(y) &= \int \mathrm{d}x\; p(y|x)p(x)
        = \mathcal{N}(y|A\mu + b, L^{-1} + A \Lambda^{-1} A^T) \\
    p(x|y) &= \frac{p(y|x)p(x)}{p(y)}
        = \mathcal{N}(y|\Sigma \left\{ A^T L (y - b) + \Lambda \mu \right\}, \Sigma) \\
    \Sigma &= \left( \Lambda  + A^T L A \right)^{-1}
\end{align}
$$

### Bayesian Inference for Gaussians

$$
\begin{eqnarray}
    p(\{x\}|\mu) &\propto& \exp\left[ - \frac{1}{2 \sigma^2} \sum_n (x_n - \mu)^2 \right] \\
    p(\mu|\mu_0, \sigma_0) &\propto& \exp\left[ - \frac{1}{2 \sigma_0^2} (\mu - \mu_0)^2 \right] \\
    p(\mu|\{x\}) &=& \frac{p(\{x\}|\mu) p(\mu|\mu_0, \sigma_0)}{p(\{x\})}
\end{eqnarray}
$$

The posterior is given by
$$
\begin{eqnarray}
    -\log p(\mu|\{x\}) &=&
        \frac{1}{2 \sigma^2} \sum_n (x_n - \mu)^2 +
        \frac{1}{2 \sigma_0^2} (\mu - \mu_0)^2 +
        \mathrm{const}
\\
    &=&
        \frac{1}{2 \sigma^2} \sum_n \mu^2 + \frac{1}{2 \sigma_0^2} \mu^2 +
        \frac{1}{\sigma^2} \sum_n x_n \mu + \frac{1}{\sigma_0^2} \mu_0 \mu +
        \mathrm{const}
\\
    &=&
        \frac{1}{2 \sigma^{\prime2}} \mu^2 + \frac{1}{\sigma^{\prime^2}} \mu^\prime \mu + \mathrm{const}
\end{eqnarray}
$$

With:

$$
\begin{eqnarray}
    \sigma^{\prime2}
        &=& \left( \frac{1}{\sigma^2} \sum_n + \frac{1}{\sigma_0^2}  \right)^{-1}
        &=& \left( \frac{\sigma_0^2 N + \sigma^2}{\sigma_0^2 \sigma^2} \right)^{-1}
        &=& \frac{\sigma_0^2 \sigma^2}{\sigma_0^2 N + \sigma^2}
        &=& \frac{\sigma_0^2}{1 + N \sigma_0^2 / \sigma^2}
\end{eqnarray}
$$

and

$$
\begin{eqnarray}
    \mu^\prime
        &=& \frac{\sigma^{\prime^2}}{\sigma^2} \sum_n x_n + \frac{\sigma^{\prime^2}}{\sigma_0^2} \mu_0
        &=& \frac{\sigma^{\prime^2}}{\sigma^2} X + \frac{\sigma^{\prime^2}}{\sigma_0^2} \mu_0
        &=& \frac{\sigma_0^2 X + \sigma^2 \mu_0}{\sigma_0^2 N + \sigma^2}
        &=& \frac{\sigma_0^2 N \bar{x} + \sigma^2 \mu_0}{\sigma_0^2 N + \sigma^2}
\end{eqnarray}
$$

Note that for $N \rightarrow 0$:

$$
\begin{eqnarray}
    \sigma^{\prime2} &\rightarrow& \sigma_0
    &\quad&
    \mu^\prime &\rightarrow& \mu_0
\end{eqnarray}
$$

And for $N \rightarrow \infty$
$$
\begin{eqnarray}
    \sigma^{\prime2} &\rightarrow& 0
    &\quad&
    \mu^\prime &\rightarrow& \bar{x}
\end{eqnarray}
$$

### TODO: Summarize Student + Normal-Gamma Sections

### Periodic Variables

Von Mises-Fischer Distribution:

$$
    p(x|\mu, \kappa) =
        \frac{\kappa^{\nu}}{(2 \pi)^{\nu + 1} I_\nu(\kappa)} \exp \left[ \kappa \mu^T x \right]
$$

with $\nu = d / 2 - 1$. With $|\mu| = 1$. For maximum likelihood fitting note that:

$$
\begin{eqnarray}
    \mathcal{L} = p(\{x\}|\mu, \kappa)
    &=&
        \kappa \mu^T \sum_n x_n + N \log \kappa^{\nu}
        - N\log I_\nu(\kappa) + \mathrm{const}
\\
    \frac{\partial}{\partial \mu} \left( \mathcal{L} + \lambda (\mu^2 - 1) \right)
    &=&
        \kappa \sum_n x_n + \lambda \mu = 0
\\
    \mu &=& \frac{\sum_n x_n}{|\sum_n x_n|}
\\
    \frac{\partial}{\partial \kappa} \mathcal{L}
    &=&
        \mu^T \sum_n x_n
        - N \frac{\partial}{\partial \kappa} \log \kappa^{-\nu} I_\nu(\kappa) \\
    &=& \mu^T \sum_n x_n - N \frac{I_{\nu + 1}(\kappa)}{I_\nu(\kappa)} \\
    &=& |\sum_n x_n| - N \frac{I_{\nu + 1}(\kappa)}{I_\nu(\kappa)} = 0
\\
    \frac{I_{\nu + 1}(\kappa)}{I_\nu(\kappa)} &=& \frac{|\sum_n x_n|}{N}
\end{eqnarray}
$$

Note, the sign of $\mu$ stems from the maximization of the likelihood and the following identity was used:

$$
    \frac{\partial}{\partial \kappa} \left[ \kappa^{-\nu} I_\nu(\kappa) \right] =
    \kappa^{-\nu} I_{\nu + 1}(\kappa).
$$

The equation for $\kappa$ can be be solved, e.g., by bisection search.

```python
def vonmises_pdf(x, mu, kappa):
    return np.exp(kappa * (np.cos(x - mu) - 1)) / (2.0 * np.pi * scipy.special.ive(0, kappa))

def vonmises_fit(x):
    mu_est = np.arctan2(np.mean(np.sin(x)), np.mean(np.cos(x)))
    return mu_est, bisect_kappa(np.mean(np.cos(x - mu_est)))

def bisect_kappa(x):
    """Bisect the solution to ``I_1(kappa) / I_0(kappa) = x``
    """
    def eval(kappa):
        return scipy.special.ive(1, kappa) / scipy.special.ive(0, kappa)

    lower = -8
    upper = +8

    if x > eval(10 ** upper): return 10 ** upper
    if x < eval(10 ** lower): return 10 ** lower

    # perform logarithmic search
    for _ in range(10):
        cand = lower + 0.5 * (upper - lower)
        val = eval(10 ** cand)

        if val > x: upper = cand
        if val < x: lower = cand

        if (upper - lower) < 1:
            break

    # perform linear search
    lower = 10 ** lower
    upper = 10 ** upper

    for _ in range(20):
        cand = lower + 0.5 * (upper - lower)
        val = eval(cand)

        if val > x: upper = cand
        if val < x: lower = cand

    cand = lower + 0.5 * (upper - lower)
    return cand

```

```python
n_samples = 10_000
mu = 0.5* np.pi
kappa = 2

x = np.random.vonmises(mu, kappa, size=n_samples)
mu_est, kappa_est = vonmises_fit(x)

u = np.linspace(-np.pi, +np.pi, 100)
plt.hist(x, bins=51, normed=True, label='empirical', alpha=0.5)
plt.plot(u, vonmises_pdf(u, mu, kappa), label='analytical')
plt.plot(u, vonmises_pdf(u, mu_est, kappa_est), label='fitted', ls='--')
plt.legend(loc='best')
pass
```

## The exponential family

$$
\begin{eqnarray}
    p(x|\eta) &=& h(x) g(\eta) \exp \eta^T u(x) \\
    p(\eta|\nu, \chi) &=& f(\chi, \nu) g(\eta)^\nu \exp \nu \eta^T \chi
\end{eqnarray}
$$

Posterior:

$$
\begin{eqnarray}
    p(\eta|\{x\}) &=& \left( \prod_n p(x_n|\eta) \right) p(\eta|\nu, \chi) \\
        &\propto&
            g(\eta)^N \exp \left( \eta^T \sum_n u(x_n) \right)
            g(\eta)^\nu \exp \nu \eta^T \chi \\
        &=&
            g(\eta)^{\nu + N} \exp \left( (\nu + N) \frac{\nu \chi + \sum_n u(x_n)}{\nu + N} \right) \\
    p(\eta|\{x\}) &=& p(\eta|\nu^\prime, \chi^\prime) \\
    \nu^\prime &=& \nu + N \\
    \chi^\prime &=& \frac{\nu \chi + \sum_n u(x_n)}{\nu + N}
\end{eqnarray}
$$

### Maximum likelihood for exponential family

$$
\begin{eqnarray}
    \mathcal{L} &=& \sum_n \log p(x_n|\eta)
\\
    &=& \sum_n \eta^T u(x_n) + N \log g(\eta) + \mathrm{const}
\\
    \frac{\partial}{\partial \eta} \mathcal{L} &=&
        \sum_n u(x_n) + N \frac{\partial}{\partial \eta} \log g(\eta) = 0
\\
    -\frac{\partial}{\partial \eta} \log g(\eta) &=& \frac{1}{N} \sum_n u(x_n)
\end{eqnarray}
$$

For the Gaussian:

$$
\begin{eqnarray}
    p(x|\eta) = g(\eta) \exp\left[ \eta_1 u_1(x) + \eta_2 u_2(x) \right]
\\
    \eta_1 = \frac{\mu}{\sigma^2},\; \eta_2 = \frac{-1}{2 \sigma^2}
\\
    u_1(x) = x,\; u_2(x) = x^2
\\
    -\log g(\eta) = \frac{1}{2} \log \pi - \frac{1}{2} \log -\eta_2 - \frac{1}{4} \frac{\eta_1^2}{\eta_2}
\\
    -\log \frac{\partial}{\partial \eta_1} g(\eta) = \mu
\\
    -\log \frac{\partial}{\partial \eta_2} g(\eta) = \mu^2 + \sigma^2
\end{eqnarray}
$$

For Bernoulli

$$
\begin{eqnarray}
    p(x|\mu) &=& \mu^x (1 - \mu)^{1 - x}  = \left( \frac{\mu}{1 - \mu} \right)^x \left( 1 - \mu \right)
\\
    &=& g(\eta) \exp \eta x
\\
    \eta &=& \log \frac{\mu}{1 - \mu}
\\
    g(\eta) &=& \sigma(-\eta)
\\
    -\frac{\partial}{\partial \eta} \log g(\eta)
        &=& \frac{1}{\sigma(-\eta)} \sigma(\eta) \sigma(-\eta) = \sigma(\eta) = \mu
\end{eqnarray}
$$

```python

```