# Interpolating Metric Quantities on Cell Faces

## Author: Patrick Nelson

**Notebook Status:** Validated

**Validation Notes:** This module will be self-validated against [its module](../../edit/in_progress/GiRaFFE_NRPy/GiRaFFE_NRPy_FCVAL.py) and will also be validated against the corresponding algorithm in the old `GiRaFFE` code in [this tutorial](Tutorial-Start_to_Finish-GiRaFFE_NRPy-FCVAL.ipynb).

# This module presents the functionality of [GiRaFFE_NRPy_Metric_Face_Values.py](../../edit/in_progress/GiRaFFE_NRPy/GiRaFFE_NRPy_Metric_Face_Values.py) .
This notebook presents the macros from the original `GiRaFFE` that provide the values of the metric gridfunctions interpolated to the cell faces along with the code needed to implement this in NRPy.

# Table of Contents
$$\label{toc}$$

This notebook is organized as follows

0. [Step 0](#prelim): Preliminaries
1. [Step 1](#interpolator): The Interpolator
    1. [Step 1.a](#macros): Interpolator coefficients and definition
    1. [Step 1.b](#gf_struct): Create an array to easily define the gridfunctions we want to interpolate.
    1. [Step 1.c](#loops): Define the loop parameters and body
1. [Step 2](#code_validation): Code Validation against original C code
1. [Step 3](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Step 0: Preliminaries \[Back to [top](#toc)\]
$$\label{prelim}$$

This first block of code just sets up a subdirectory within `GiRaFFE_standalone_Ccodes/` to which we will write the C code and adds core NRPy+ functionality to `sys.path`.

```python
# Step 0: Add NRPy's directory to the path
# https://stackoverflow.com/questions/16780014/import-file-from-parent-directory
import os,sys
nrpy_dir_path = os.path.join("..")
if nrpy_dir_path not in sys.path:
    sys.path.append(nrpy_dir_path)

from outputC import *            # NRPy+: Core C code output module
import cmdline_helper as cmd     # NRPy+: Multi-platform Python command-line interface
Ccodesdir = "GiRaFFE_standalone_Ccodes/FCVAL"
cmd.mkdir(os.path.join(Ccodesdir))
```

# Step 1: The Interpolator \[Back to [top](#toc)\]
$$\label{interpolator}$$

Here, we we will write the code necessary to interpolate the metric gridfunction $\alpha, \beta^i, \gamma_{ij}$ onto the cell faces. These values will be necessary to compute fluxes of the Poynting vector and electric field through those faces.

## Step 1.a: Interpolator coefficients and definition \[Back to [top](#toc)\]
$$\label{macros}$$

First, we will define the functional form of our interpolator. At some point on our grid $i$, we will calculate the value of some gridfunction $Q$ at position $i-\tfrac{1}{2}$ with
$$
Q_{i-1/2} = a_{i-2} Q_{i-2} +a_{i-1} Q_{i-1} +a_{i} Q_{i} +a_{i+1} Q_{i+1}
$$
and the coefficients we will use for it,
\begin{align}
a_{i-2} &= -0.0625 \\
a_{i-1} &=  0.5625 \\
a_{i}   &=  0.5625 \\
a_{i+1} &= -0.0625 \\
\end{align}.

```python
%%writefile $Ccodesdir/interpolate_metric_gfs_to_cell_faces.h
// Side note: the following values could be used for cell averaged gfs:
//     am2=-1.0/12.0, am1=7.0/12.0, a0=7.0/12.0, a1=-1.0/12.0
// However, since the metric gfs store the grid point values instead of the cell average,
//     the following coefficients should be used:
//     am2 = -1/16, am1 = 9/16, a0 = 9/16, a1 = -1/16
// This will yield the third-order-accurate face values at m-1/2,
//      using values specified at {m-2,m-1,m,m+1}
#define AM2 -0.0625
#define AM1  0.5625
#define A0   0.5625
#define A1  -0.0625
#define COMPUTE_FCVAL(METRICm2,METRICm1,METRIC,METRICp1) (AM2*(METRICm2) + AM1*(METRICm1) + A0*(METRIC) + A1*(METRICp1))

```

    Overwriting GiRaFFE_standalone_Ccodes/FCVAL/interpolate_metric_gfs_to_cell_faces.h

## Step 1.b: Create an array to easily define the gridfunctions we want to interpolate. \[Back to [top](#toc)\]
$$\label{gf_struct}$$

We will need to apply this interpolation to each gridpoint for several gridfunctions: the lapse $\alpha$, the shift $\beta^i$, and the three-metric $\gamma_{ij}$. Consider that in NRPy+, each gridfunction is assigned an integer identifier with the C macro `#define`. So, the simplest (and shortest to write!) way to ensure we hit each of these is to create arrays that list each of these identifiers in order, so we can always hit the write gridfunction no matter where each gridfunction lies in the list. We use two arrays; the first identifies the usual gridfunctions from which we will read, and the second identifies the face-sampled gridfunctions to which we will write.

```python
%%writefile -a $Ccodesdir/interpolate_metric_gfs_to_cell_faces.h

const int metric_gfs_list[10] = {GAMMADD00GF,
                                 GAMMADD01GF,
                                 GAMMADD02GF,
                                 GAMMADD11GF,
                                 GAMMADD12GF,
                                 GAMMADD22GF,
                                 BETAU0GF,
                                 BETAU1GF,
                                 BETAU2GF,
                                 ALPHAGF};

const int metric_gfs_face_list[10] = {GAMMA_FACEDD00GF,
                                      GAMMA_FACEDD01GF,
                                      GAMMA_FACEDD02GF,
                                      GAMMA_FACEDD11GF,
                                      GAMMA_FACEDD12GF,
                                      GAMMA_FACEDD22GF,
                                      BETA_FACEU0GF,
                                      BETA_FACEU1GF,
                                      BETA_FACEU2GF,
                                      ALPHA_FACEGF};

const int num_metric_gfs = 10;
```

    Appending to GiRaFFE_standalone_Ccodes/FCVAL/interpolate_metric_gfs_to_cell_faces.h

## Step 1.c: Define the loop parameters and body \[Back to [top](#toc)\]
$$\label{loops}$$

Next, we will write the function that loops over the entire grid. One additional parameter to consider here is the direction in which we need to do the interpolation. This direction exactly corresponds to the parameter `flux_dirn` used in the calculation of the flux of the [Poynting vector](Tutorial-GiRaFFE_NRPy-Stilde-flux.ipynb) and [electric field](Tutorial-GiRaFFE_NRPy-Induction_Equation.ipynb).

The outermost loop will iterate over the gridfunctions we listed above. Nested inside of that, there will be three loops that go through the grid in the usual way. However, the upper bound will be a little unusual. Instead of covering all points or all interior points, we will write these loops to cover all interior points *and one extra past that*. This is because we have define our interpolation on the $i-\tfrac{1}{2}$ face of a cell, but any given calculation will require both that and an interolation on the $i+\tfrac{1}{2}$ face as well.

```python
%%writefile -a $Ccodesdir/interpolate_metric_gfs_to_cell_faces.h
// Kronecker delta: all zero-offset
const int kronecker_delta[3][3] = { { 1,0,0 },
                                    { 0,1,0 },
                                    { 0,0,1 } };

```

    Appending to GiRaFFE_standalone_Ccodes/FCVAL/interpolate_metric_gfs_to_cell_faces.h

```python
desc = "Interpolate metric gridfunctions to cell faces"
name = "interpolate_metric_gfs_to_cell_faces"
interp_Cfunc = outCfunction(
    outfile  = "returnstring", desc=desc, name=name,
    params   ="const paramstruct *params,REAL *auxevol_gfs,const int flux_dirn",
    preloop  ="""    int in_gf,out_gf;
    REAL Qm2,Qm1,Qp0,Qp1;

""" ,
    body     ="""    for(int gf = 0;gf < num_metric_gfs;gf++) {
        in_gf  = metric_gfs_list[gf];
        out_gf = metric_gfs_face_list[gf];
        for (int i2 = NGHOSTS;i2 < Nxx2+NGHOSTS+1;i2++) {
            for (int i1 = NGHOSTS;i1 < Nxx2+NGHOSTS+1;i1++) {
                for (int i0 = NGHOSTS;i0 < Nxx2+NGHOSTS+1;i0++) {
                    Qm2 = auxevol_gfs[IDX4S(in_gf,i0-2*kronecker_delta[0][flux_dirn],i1-2*kronecker_delta[1][flux_dirn],i2-2*kronecker_delta[2][flux_dirn])];
                    Qm1 = auxevol_gfs[IDX4S(in_gf,i0-kronecker_delta[0][flux_dirn],i1-kronecker_delta[1][flux_dirn],i2-kronecker_delta[2][flux_dirn])];
                    Qp0 = auxevol_gfs[IDX4S(in_gf,i0,i1,i2)];
                    Qp1 = auxevol_gfs[IDX4S(in_gf,i0+kronecker_delta[0][flux_dirn],i1+kronecker_delta[1][flux_dirn],i2+kronecker_delta[2][flux_dirn])];
                    auxevol_gfs[IDX4S(out_gf,i0,i1,i2)] = COMPUTE_FCVAL(Qm2,Qm1,Qp0,Qp1);
                }
            }
        }
    }

""")

with open(os.path.join(Ccodesdir,"interpolate_metric_gfs_to_cell_faces.h"),"a") as file:
    file.write(interp_Cfunc)
```

# Step 2: Code Validation against `GiRaFFE_NRPy_FCVAL.py`  \[Back to [top](#toc)\]
$$\label{code_validation}$$

Now, we will confirm that the code we have written here is the same as that generated by the module [`GiRaFFE_NRPy_FCVAL.py`](../../edit/in_progress/GiRaFFE_NRPy/GiRaFFE_NRPy_FCVAL.py).

```python
# Define the directory that we wish to validate against:
valdir = "GiRaFFE_NRPy/GiRaFFE_Ccode_library/FCVAL/"

import GiRaFFE_NRPy.GiRaFFE_NRPy_Metric_Face_Values as FCVAL
FCVAL.GiRaFFE_NRPy_FCVAL(valdir)

import difflib
import sys

print("Printing difference between original C code and this code...")
# Open the files to compare
file = "interpolate_metric_gfs_to_cell_faces.h"

print("Checking file " + file)
with open(os.path.join(valdir,file)) as file1, open(os.path.join(Ccodesdir,file)) as file2:
    # Read the lines of each file
    file1_lines = file1.readlines()
    file2_lines = file2.readlines()
    num_diffs = 0
    for line in difflib.unified_diff(file1_lines, file2_lines, fromfile=os.path.join(valdir+file), tofile=os.path.join(Ccodesdir+file)):
        sys.stdout.writelines(line)
        num_diffs = num_diffs + 1
    if num_diffs == 0:
        print("No difference. TEST PASSED!")
    else:
        print("ERROR: Disagreement found with .py file. See differences above.")
```

    Printing difference between original C code and this code...
    Checking file interpolate_metric_gfs_to_cell_faces.h
    No difference. TEST PASSED!

# Step 3: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-GiRaFFE_NRPy-FCVAL.pdf](Tutorial-GiRaFFE_NRPy-FCVAL.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
!jupyter nbconvert --to latex --template latex_nrpy_style.tplx --log-level='WARN' Tutorial-GiRaFFE_NRPy-Metric_Face_Values.ipynb
!pdflatex -interaction=batchmode Tutorial-GiRaFFE_NRPy-Metric_Face_Values.tex
!pdflatex -interaction=batchmode Tutorial-GiRaFFE_NRPy-Metric_Face_Values.tex
!pdflatex -interaction=batchmode Tutorial-GiRaFFE_NRPy-Metric_Face_Values.tex
!rm -f Tut*.out Tut*.aux Tut*.log
```

    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode


## Burgers Equation
#### Parameter estimation for Burgers' Equation using Gaussian processes (Backward Euler scheme)

#### Problem Setup

$u_t + u u_{x} - (\nu - e) u_{xx} = 0$

$u(x,t) = \frac{x}{1+t}$

$u_0(x) := u(x,0) = x$

$x \in [0, 1], t \in \{0, \tau \}$

Using the backward Euler scheme, the equation can be re-written as:

$\frac{u_n - u_{n-1}}{\tau} + u_n \frac{d}{dx}u_n - (\nu - e) \frac{d^2}{dx^2}u_n = 0$

and setting $u_n = \mu_n$ to deal with the non-linearity:

$u_n + \tau \mu_n \frac{d}{dx}u_n - \tau (\nu - e) \frac{d^2}{dx^2}u_n = u_{n-1}$

Consider $u_n$ to be a Gaussian processes.

$u_n \sim \mathcal{GP}(0, k_{uu}(x_i, x_j, \theta))$

And the linear operator:

$\mathcal{L}_x^\nu = \cdot + \tau \mu_n \frac{d}{dx}\cdot - \tau (\nu - e) \frac{d^2}{dx^2} \cdot$

so that

$\mathcal{L}_x^\nu u_n = u_{n-1}$

Problem at hand: estimate $\nu$ (should be $\nu = e$ in the end).

For the sake of simplicity, take $u := u_n$ and $f := u_{n-1}$.

#### step 1: Simulate data

Take data points at $t = 0$ for $(u_{n-1})$ and $t = \tau$ for $(u_n)$, where $\tau$ is the time step.

```python
import numpy as np
import sympy as sp
from scipy.optimize import minimize
import matplotlib.pyplot as plt
```

```python
tau = 0.01
def get_simulated_data(tau):
    x = np.random.rand(20)
    y_u = x/(1+tau)
    y_f = x
    return (x, y_u,y_f)

(x, y_u, y_f) = get_simulated_data(tau)
```

```python
plt.plot(x, y_u, 'ro')
plt.show()
```

```python
plt.plot(x, y_f, 'bo')
plt.show()
```

#### Step 2:Evaluate kernels

$k_{nn}(x_i, x_j; \theta) = \theta exp(-\frac{1}{2}(x_i-x_j)^2)$

```python
x_i, x_j, theta, mu_n, nu = sp.symbols('x_i x_j theta mu_n nu')
kuu_sym = theta*sp.exp(-1/(2)*((x_i - x_j)**2))
kuu_fn = sp.lambdify((x_i, x_j, theta), kuu_sym, "numpy")
def kuu(x, theta):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kuu_fn(x[i], x[j], theta)
    return k
```

$k_{ff}(x_i,x_j;\theta,\phi) \\
= \mathcal{L}_{x_i}^\nu \mathcal{L}_{x_j}^\nu k_{uu}(x_i, x_j; \theta) \\
= \mathcal{L}_{x_i}^\nu \left( k_{uu} + \tau \mu_n \frac{d}{dx_j}k_{uu} - \tau(\nu - e)\frac{d^2}{dx_j^2}k_{uu} \right) \\
 = k_{uu} + \tau \mu_n \frac{d}{dx_i}k_{uu} - \tau(\nu-e)\frac{d^2}{dx_i^2}k_{uu} + \tau\mu_n\frac{d}{dx_j}k_{uu} + \tau\mu_n\frac{d}{dx_i}\tau\mu_n\frac{d}{dx_j}k_{uu} - \tau(\nu - e)\frac{d^2}{dx_i^2}\tau\mu_n\frac{d}{dx_j}k_{uu} - \tau(\nu-e)\frac{d^2}{dx_j^2}k_{uu} - \tau\mu_n\frac{d}{dx_i}\tau(\nu-e)\frac{d^2}{dx_j^2}k_{uu} + \tau(\nu - e)\frac{d^2}{dx_i^2}\tau(\nu-e)\frac{d^2}{dx_j^2}k_{uu}$

```python
kff_sym = kuu_sym \
        + tau*mu_n*(sp.diff(kuu_sym, x_i)) \
        - tau*(nu-np.e)*sp.diff(kuu_sym, x_i, x_i) \
        + tau*mu_n*sp.diff(kuu_sym, x_j) \
        + tau**2*mu_n**2*sp.diff(kuu_sym, x_i, x_j) \
        - tau**2*(nu-np.e)*mu_n*sp.diff(kuu_sym, x_j, x_i, x_i) \
        - tau*(nu-np.e)*sp.diff(kuu_sym, x_j, x_j) \
        - tau**2*mu_n*(nu-np.e)*sp.diff(kuu_sym, x_j, x_j, x_i) \
        + tau**2*(nu-np.e)**2*sp.diff(kuu_sym, x_j, x_j, x_i, x_i)
kff_fn = sp.lambdify((x_i, x_j, theta, mu_n, nu), kff_sym, "numpy")
def kff(x, theta, mu_n, nu):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kff_fn(x[i], x[j], theta, mu_n, nu)
    return k
```

$k_{fu}(x_i,x_j;\theta,\phi) \\
= \mathcal{L}_{x_i}^\nu k_{uu}(x_i, x_j; \theta) \\
= k_{uu} + \tau \mu_n \frac{d}{dx_i}k_{uu} - \tau(\nu - e) \frac{d^2}{dx_i^2}k_{uu}$

```python
kfu_sym = kuu_sym + tau*mu_n*sp.diff(kuu_sym, x_i) - tau*(nu-np.e)*sp.diff(kuu_sym, x_i, x_i)
kfu_fn = sp.lambdify((x_i, x_j, theta, mu_n, nu), kfu_sym, "numpy")
def kfu(x, theta, mu_n, nu):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kfu_fn(x[i], x[j], theta, mu_n, nu)
    return k
```

```python
def kuf(x, theta, mu_n, nu):
    return kfu(x,theta,mu_n, nu).T
```

#### Step 3: Compute NLML

```python
def nlml(params, x, y1, y2, s):
    theta_exp = np.exp(params[0]) # don't see why mu or nu must be positive
    K = np.block([
        [kuu(x, theta_exp) + s*np.identity(x.size), kuf(x, theta_exp, params[1], params[2])],
        [kfu(x, theta_exp, params[1], params[2]), kff(x, theta_exp, params[1], params[2]) + s*np.identity(x.size)]
    ])
    y = np.concatenate((y1, y2))
    val = 0.5*(np.log(abs(np.linalg.det(K))) + np.mat(y) * np.linalg.inv(K) * np.mat(y).T)
    return val.item(0)
```

```python
nlml((np.exp(-2.42), 0.53, np.e), x, y_u, y_f, 1e-6)

```

    -205.47437331669116

Mapping w.r.t. theta:

```python
z = np.linspace(-10, 10)
z_out = [nlml((q, 0.53, 6.22), x, y_u, y_f, 1e-6) for q in z]
plt.plot(z, z_out)

nlml((0.53, 0.53, 6.22), x, y_u, y_f, 1e-6)
```

Mapping w.r.t. nu:

```python
nlml((-2.4, 0.5, np.e) ,x ,y_u,y_f,1e-6) # mu should be roughly 0.5

z = np.linspace(-10, 10)
z_out = [nlml((0.53, 0.53, q), x, y_u, y_f, 1e-6) for q in z]
plt.plot(z, z_out)

# Relatively consistently we get the info, that vu = np.e is the maximizer, not the minimizer of the function:
print(nlml((0.53, 0.525, np.e - 0.2) ,x ,y_u,y_f,1e-6))
print(nlml((0.53, 0.525, np.e) ,x ,y_u,y_f,1e-6))
print(nlml((0.53, 0.525, np.e + 0.2) ,x ,y_u,y_f,1e-6))

# This is possibly hinting at a sign error somewhere
```

#### Step 4: Optimise hyperparameters

```python
m = minimize(nlml, (np.random.rand(1), 0.5, np.e), args=(x, y_u, y_f, 1e-6), method="Nelder-Mead", options = {'maxiter' : 1000})
```

```python
m.x[2]
```

    6.2198395417518855

```python
m
```

     final_simplex: (array([[-0.63551394,  0.52538897,  6.21983954],
           [-0.63551501,  0.52538874,  6.21974606],
           [-0.63553883,  0.5253878 ,  6.21976136],
           [-0.63553225,  0.52539213,  6.21980002]]), array([-239.48495151, -239.48495151, -239.48495151, -239.48495151]))
               fun: -239.48495151445627
           message: 'Optimization terminated successfully.'
              nfev: 163
               nit: 93
            status: 0
           success: True
                 x: array([-0.63551394,  0.52538897,  6.21983954])


# Statistics

## Introduction

In this chapter, you'll learn about how to do statistics with code. We already saw some statistics in the chapter on probability and random processes: here we'll focus on computing basic statistics and using statistical tests. We'll make use of the excellent [*pingouin*](https://pingouin-stats.org/index.html) statistics package and its documentation for many of the examples and methods in this chapter {cite}`vallat2018pingouin`. This chapter also draws on Open Intro Statistics {cite}`diez2012openintro`.

### Notation and basic definitions

Greek letters, like $\beta$, are the truth and represent parameters. Modified Greek letters are an estimate of the truth, for example $\hat{\beta}$. Sometimes Greek letters will stand in for vectors of parameters. Most of the time, upper case Latin characters such as $X$ will represent random variables (which could have more than one dimension). Lower case letters from the Latin alphabet denote realised data, for instance $x$ (which again could be multi-dimensional).  Modified Latin alphabet letters denote computations performed on data, for instance $\bar{x} = \frac{1}{n} \displaystyle\sum_{i} x_i$ where $n$ is number of samples. Parameters are given following a vertical bar, for example if $f(x|\mu, \sigma)$ is a probability density function, the vertical line indicates that its parameters are $\mu$ and $\sigma$. The set of distributions with densities $f_\theta(x)$, $\theta \in \Theta$ is called a parametric family, eg there is a family of different distributions that are parametrised by $\theta$.

A **statistic** $T(x)$ is a function of the data $x=(x_1, \dots, x_n)$.

An **estimator** of a parameter $\theta$ is a function $T=T(x)$ which is used to estimate $\theta$ based on observations of data. $T$ is an unbiased estimator if $\mathbb{E}(T) = \theta$.

If $X$ has PDF $f(x|\theta)$ then, given the observed value $x$ of $X$, the **likelihood** of $\theta$ is defined by $\text{lik}(\theta) = f(x | \theta)$. For independent and identically distributed observed values, then $\text{lik}(\theta) = f(x_1, \dots, x_n| \theta) = \Pi_{i=1}^n f(x_i | \theta)$. The $\hat{\theta}$ such that this function attains its maximum value is the **maximum likelihood estimator (MLE)** of $\theta$.

Given an MLE $\hat{\theta}$ of $\theta$, $\hat{\theta}$ is said to be **consistent** if $\mathbb{P}(\hat{\theta} - \theta > \epsilon) \rightarrow 0$ as $n\rightarrow \infty$.

An estimator *W* is **efficient** relative to another estimator $V$ if $\text{Var}(W) < \text{Var}(V)$.

Let $\alpha$ be the 'significance level' of a test statistic $T$.

Let $\gamma(X)$ and $\delta(X)$ be two statistics satisfying $\gamma(X) < \delta(X)$ for all $X$. If on observing $X = x$, the inference can be made that $\gamma(x) \leq \theta \leq \delta(x)$. Then $[\delta(x), \gamma(x)]$ is an **interval estimate** and $[\delta(X), \gamma(X)]$ is an **interval estimator**. The random interval (random because the *endpoints* are random variables) $[\delta(X), \gamma(X)]$ is called a $100\cdot\alpha \%$ **confidence interval** for $\theta$. Of course, there is a true $\theta$, so either it is in this interval or it is not. But if the confidence interval was constructed many times over using samples, $\theta$ would be contained within it $100\cdot\alpha \%$ of the times.

A **hypothesis test** is a conjecture about the distribution of one or more random variables, and a test of a hypothesis is a procedure for deciding whether or not to reject that conjecture. The **null hypothesis**, $H_0$, is only ever conservatively rejected and represents the default positiion. The **alternative hypothesis**, $H_1$, is the conclusion contrary to this.

A type I error occurs when $H_0$ is rejected when it is true, ie when a *true* null hypothesis is rejected. Mistakenly failing to reject a false null hypothesis is called a type II error.

In the most simple situations, the upper bound on the probability of a type I error is called the size or **significance level** of the *test*. The **p-value** of a random variable $X$ is the smallest value of the significance level (denoted $\alpha$) for which $H_0$ would be rejected on the basis of seeing $x$. The p-value is sometimes called the significance level of $X$. The probability that a test will reject the null is called the power of the test. The probability of a type II error is equal to 1 minus the power of the test.

Recall that there are two types of statistics out there: parametrised, eg by $\theta$, and non-parametrised. The latter are often distribution free (ie don't involve a PDF) or don't require parameters to be specified.

### Imports

First we need to import the packages we'll be using

```python
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
import pandas as pd
import pingouin as pg
import statsmodels.formula.api as smf
from numpy.random import Generator, PCG64

# Set seed for random numbers
seed_for_prng = 78557
prng = Generator(PCG64(seed_for_prng))
```

## Basic statistics

Let's start with computing the simplest statistics you can think of using some synthetic data. Many of the functions have lots of extra options that we won't explore here (like weights or normalisation); remember that you can see these using the `help()` method.

We'll generate a vector with 100 entries:

```python
data = np.array(range(100))
data
```

```python
from myst_nb import glue
import sympy
import warnings

warnings.filterwarnings("ignore")
dict_fns = {
    "mean": np.mean(data),
    "std": np.std(data),
    "mode": stats.mode([0, 1, 2, 3, 3, 3, 5])[0][0],
    "median": np.median(data),
}

for name, eval_fn in dict_fns.items():
    glue(name, f"{eval_fn:.1f}")

# Set max rows displayed for readability
pd.set_option("display.max_rows", 6)
# Plot settings
plt.style.use("plot_style.txt")
```

Okay, let's see how some basic statistics are computed. The mean is `np.mean(data)=` {glue:}`mean`, the standard deviation is `np.std(data)=` {glue:}`std`, and the median is given by `np.median(data)= `{glue:}`median`. The mode is given by `stats.mode([0, 1, 2, 3, 3, 3, 5])[0]=` {glue:}`mode` (access the counts using `stats.mode(...)[1]`).

Less famous quantiles than the median are given by, for example for $q=0.25$,

```python
np.quantile(data, 0.25)
```

As with **pandas**, **numpy** and **scipy** work on scalars, vectors, matrices, and tensors: you just need to specify the axis that you'd like to apply a function to:

```python
data = np.fromfunction(lambda i, j: i + j, (3, 6), dtype=int)
data
```

```python
np.mean(data, axis=0)
```

Remember that, for discrete data points, the $k$th (unnormalised) moment is

$$
\hat{m}_k = \frac{1}{n}\displaystyle\sum_{i=1}^{n} \left(x_i - \bar{x}\right)^k
$$

To compute this use scipy's `stats.moment(a, moment=1)`. For instance for the kurtosis ($k=4$), it's

```python
stats.moment(data, moment=4, axis=1)
```

Covariances are found using `np.cov`.

```python
np.cov(np.array([[0, 1, 2], [2, 1, 0]]))
```

Note that, as expected, the $C_{01}$ term is -1 as the vectors are anti-correlated.

## Parametric tests

Reminder: parametric tests assume that data are effectively drawn a probability distribution that can be described with fixed parameters.

### One-sample t-test

The one-sample t-test tells us whether a given parameter for the mean, i.e. a suspected $\mu$, is likely to be consistent with the sample mean. The null hypothesis is that $\mu = \bar{x}$. Let's see an example using the default `alternative='two-sided'` option. Imagine we have data on the number of hours people spend working each day and we want to test the (alternative) hypothesis that $\bar{x}$ is not $\mu=$8 hours:

```python
x = [8.5, 5.4, 6.8, 9.6, 4.2, 7.2, 8.8, 8.1]

pg.ttest(x, 8).round(2)
```

(The returned object is a **pandas** dataframe.) We only have 8 data points, and so that is a great big confidence interval! It's worth remembering what a t-statistic and t-test really are. In this case, the statistic that is constructed to test whether the sample mean is different from a known parameter $\mu$ is

$$
T = \frac{\sqrt{n}(\bar{x}-\mu)}{\hat{\sigma}} \thicksim t_{n-1}
$$

where $t_{n-1}$ is the student's t-distribution and $n-1$ is the number of degrees of freedom. The $100\cdot(1-\alpha)\%$ test interval in this case is given by

$$
1 - \alpha = \mathbb{P}\left(-t_{n-1, \alpha/2} \leq \frac{\sqrt{n}(\bar{x} - \mu)}{\hat{\sigma}} \leq t_{n-1,\alpha/2}\right)
$$

where we define $t_{n-1, \alpha/2}$ such that $\mathbb{P}(T > t_{n-1, \alpha/2}) = \alpha/2$. For $\alpha=0.05$, implying confidence intervals of 95%, this looks like:

```python
import scipy.stats as st

def plot_t_stat(x, mu):
    T = np.linspace(-7, 7, 500)
    pdf_vals = st.t.pdf(T, len(x) - 1)

    sigma_hat = np.sqrt(np.sum((x - np.mean(x)) ** 2) / (len(x) - 1))
    actual_T_stat = (np.sqrt(len(x)) * (np.mean(x) - mu)) / sigma_hat

    alpha = 0.05
    T_alpha_over_2 = st.t.ppf(1.0 - alpha / 2, len(x) - 1)

    interval_T = T[((T > -T_alpha_over_2) & (T < T_alpha_over_2))]
    interval_y = pdf_vals[((T > -T_alpha_over_2) & (T < T_alpha_over_2))]

    fig, ax = plt.subplots()
    ax.plot(T, pdf_vals, label=f"Student t: dof={len(x)-1}", zorder=2)
    ax.fill_between(
        interval_T, 0, interval_y, alpha=0.2, label=r"95% interval", zorder=1
    )
    ax.plot(
        actual_T_stat,
        st.t.pdf(actual_T_stat, len(x) - 1),
        "bo",
        ms=15,
        label=r"$\sqrt{n}(\bar{x} - \mu)/\hat{\sigma}}$",
        color="orchid",
        zorder=4,
    )
    ax.vlines(
        actual_T_stat, 0, st.t.pdf(actual_T_stat, len(x) - 1), color="orchid", zorder=3
    )
    ax.set_xlabel("Value of statistic T")
    ax.set_ylabel("PDF")
    ax.set_xlim(-7, 7)
    ax.set_ylim(0.0, 0.4)
    ax.legend(frameon=False)
    plt.show()

mu = 8
plot_t_stat(x, mu)
```

In this case, we would reject the alternative hypothesis. You can see why from the plot; the test statistic we have constructed lies within the interval where we cannot reject the null hypothesis. $\bar{x}-\mu$ is close enough to zero to give us cause for concern. (You can also see from the plot why this is a two-tailed test: we don't care if $\bar{x}$ is greater or less than $\mu$, just that it's different--and so the test statistic could appear in either tail of the distribution for us to accept $H_1$.)

We accept the null here, but about if there were many more data points? Let's try adding some generated data (pretend it is from making extra observations).

```python
# 'Observe' extra data
extra_data = prng.uniform(5.5, 8.5, size=(30))
# Add it in to existing vector
x_prime = np.concatenate((np.array(x), extra_data), axis=None)
# Run t-test
pg.ttest(x_prime, 8).round(2)
```

Okay, what happened? Our extra observations have seen the confidence interval shrink considerably, and the p-value is effectively 0. There's a large negative t-statistic too. Unsurprisingly, as we chose a uniform distribution that only just included 8 but was centered on $(8-4.5)/2$ *and* we had more points, the test now rejects the null hypothesis that $\mu=8$ . Because the alternative hypothesis is just $\mu\neq8$, and these tests are conservative, we haven't got an estimate of what the mean actually is; we just know that our test rejects that it's $8$.

We can see this in a new version of the chart that uses the extra data:

```python
plot_t_stat(x_prime, mu)
```

Now our test statistic is safely outside the interval.

#### Connection to linear regression

Note that testing if $\mu\neq0$ is equivalent to having the alternative hypothesis that a single, non-zero scalar value is a good expected value for $x$, i.e. that $\mathbb{E}(x) \neq 0$. Which may sound familiar if you've run **linear regression** and, indeed, this t-test has an equivalent linear model! It's just regressing $X$ on a constant--a single, non-zero scalar value. In general, t-tests appear in linear regression to test whether any coefficient $\beta \neq 0$.

We can see this connection by running a hypothesis test of whether the sample mean is not zero. Note the confidence interval, t-statistic, and p-value.

```python
pg.ttest(x, 0).round(3)
```

And, as an alternative, regressing x on a constant, again noting the interval, t-stat, and p-value:

```python
import statsmodels.formula.api as smf

df = pd.DataFrame(x, columns=["x"])

res = smf.ols(formula="x ~ 1", data=df).fit()
# Show only the info relevant to the intercept (there are no other coefficients)
print(res.summary().tables[1])
```

Many tests have an equivalent  linear model.

#### Other information provided by **Pingouin** tests

We've covered the degrees of freedom, the T statistic, the p-value, and the confidence interval. So what's all that other gunk in our t-test? Cohen's d is a measure of whether the difference being measured in our test is large or not (this is important; you can have statistically significant differences that are so small as to be inconsequential). Cohen suggested that $d = 0.2$ be considered a 'small' effect size, 0.5 represents a 'medium' effect size and 0.8 a 'large' effect size. BF10 represents the Bayes factor, the ratio (given the data) of the likelihood of the alternative hypothesis relative to the null hypothesis. Values greater than unity therefore favour the alternative hypothesis. Finally, power is the achieved power of the test, which is $1 - \mathbb{P}(\text{type II error})$. A common default to have in mind is a power greater than 0.8.

### Two-sample t-test

The two-sample t-test is used to determine if two population means are equal (with the null being that they *are* equal). Let's look at an example with synthetic data of equal length, which means we can use the *paired* version of this. We'll imagine we are looking at an intervention with a pre- and post- dataset.

```python
pre = [5.5, 2.4, 6.8, 9.6, 4.2, 5.9]
post = [6.4, 3.4, 6.4, 11.0, 4.8, 6.2]
pg.ttest(pre, post, paired=True, alternative="two-sided").round(2)
```

In this case, we cannot reject the null hypothesis that the means are the same pre- and post-intervention.

### Pearson correlation

The Pearson correlation coefficient measures the linear relationship between two datasets. Strictly speaking, it requires that each dataset be normally distributed.

```python
mean, cov = [4, 6], [(1, 0.5), (0.5, 1)]
x, y = prng.multivariate_normal(mean, cov, 30).T
# Compute Pearson correlation
pg.corr(x, y).round(3)
```

### Welch's t-test

In the case where you have two samples with unequal variances (or, really, unequal sample sizes too), Welch's t-test is appropriate. With `correction='true'`, it assumes that variances are not equal.

```python
x = prng.normal(loc=7, size=20)
y = prng.normal(loc=6.5, size=15)
pg.ttest(x, y, correction="true")
```

### One-way ANOVA

Analysis of variance (ANOVA) is a technique for testing hypotheses about means, for example testing the equality of the means of $k>2$ groups. The model would be

$$
X_{ij} = \mu_i + \epsilon_{ij} \quad j=1, \dots, n_i \quad i=1, \dots, k.
$$

so that the $i$th group has $n_i$ observations. The null hypothesis of one-way ANOVA is that $H_0: \mu_1 = \mu_2 = \dots = \mu_k$, with the alternative hypothesis that this is *not* true.

```python
df = pg.read_dataset("mixed_anova")
df.head()
```

```python
# Run the ANOVA
pg.anova(data=df, dv="Scores", between="Group", detailed=True)
```

### Multiple pairwise t-tests

There's a problem with running multiple t-tests: if you run enough of them, something is bound to come up as significant! As such, some *post-hoc* adjustments exist that correct for the fact that multiple tests are occurring simultaneously. In the example below, multiple pairwise comparisons are made between the scores by time group. There is a corrected p-value, `p-corr`, computed using the Benjamini/Hochberg FDR correction.

```python
pg.pairwise_ttests(
    data=df,
    dv="Scores",
    within="Time",
    subject="Subject",
    parametric=True,
    padjust="fdr_bh",
    effsize="hedges",
).round(3)
```

### One-way ANCOVA

Analysis of covariance (ANCOVA) is a general linear model which blends ANOVA and regression. ANCOVA evaluates whether the means of a dependent variable (dv) are equal across levels of a categorical independent variable (between) often called a treatment, while statistically controlling for the effects of other continuous variables that are not of primary interest, known as covariates or nuisance variables (covar).

```python
df = pg.read_dataset("ancova")
df.head()
```

```python
pg.ancova(data=df, dv="Scores", covar="Income", between="Method")
```

### Power calculations

Often, it's quite useful to know what sample size is needed to avoid certain types of testing errors. **Pingouin** offers ways to compute effect sizes and test powers to help with these questions.

As an example, let's assume we have a new drug (`x`) and an old drug (`y`) that are both intended to reduce blood pressure. The standard deviation of the reduction in blood pressure of those receiving the old drug is 12 units. The null hypothesis is that the new drug is no more effective than the new drug. But it will only be worth switching production to the new drug if it reduces blood pressure by more than 3 units versus the old drug. In this case, the effect size of interest is 3 units.

Let's assume for a moment that the true difference is 3 units and we want to perform a test with $\alpha=0.05$. The problem is that, for small differences in the effect, the distribution of effects under the null and the distribution of effects under the alternative have a great deal of overlap. So the chances of making a Type II error - accepting the null hypothesis when it is actually false - is quite high. Let's say we'd ideally have at most a 20% chance of making a Type II error: what sample size do we need?

We can compute this, but we need an extra piece of information first: a normalised version of the effect size called Cohen's $d$. We need to transform the difference in means to compute this. For independent samples, $d$ is:

$$ d = \frac{\overline{X} - \overline{Y}}{\sqrt{\frac{(n_{1} - 1)\sigma_{1}^{2} + (n_{2} - 1)\sigma_{2}^{2}}{n_1 + n_2 - 2}}}$$

(If you have real data samples, you can compute this using `pg.compute_effsize`.)

For this case, $d$ is $-3/12 = -1/4$ if we assume the standard deviations are the same across the old (`y`) and new (`x`) drugs. So we will plug that $d$ in and look at a range of possible sample sizes along with a standard value for $alpha$ of 0.05. In the below `alternative=less` tests the alternative that `x` has a smaller mean than `y`.

```python
cohen_d = -0.25  # Fixed effect size
sample_size_array = np.arange(1, 500, 50)  # Incrementing sample size
# Compute the achieved power
pwr = pg.power_ttest(
    d=cohen_d, n=sample_size_array, alpha=0.05, contrast="two-samples", alternative="less"
)
fig, ax = plt.subplots()
ax.plot(sample_size_array, pwr, "ko-.")
ax.axhline(0.8, color="r", ls=":")
ax.set_xlabel("Sample size")
ax.set_ylabel("Power (1 - type II error)")
ax.set_title("Achieved power of a T-test")
plt.show()
```

From this, we can see we need a sample size of at least 200 in order to have a power of 0.8.

The `pg.power_ttest` function takes any three of the four of `d`, `n`, `power`, and `alpha` (ie leave one of these out), and then returns what the missing parameter should be. We passed in `d`, `n`, and `alpha`, and so the `power` was returned.

## Non-parametric tests

Reminder: non-parametrics tests do not make any assumptions about the distribution from which data are drawn or that it can be described by fixed parameters.

### Wilcoxon Signed-rank Test

This tests the null hypothesis that two related paired samples come from the same distribution. It is the non-parametric equivalent of the t-test.

```python
x = [20, 22, 19, 20, 22, 18, 24, 20, 19, 24, 26, 13]
y = [38, 37, 33, 29, 14, 12, 20, 22, 17, 25, 26, 16]
pg.wilcoxon(x, y, alternative="two-sided").round(2)
```

### Mann-Whitney U Test (aka Wilcoxon rank-sum test)

The Mann–Whitney U test is a non-parametric test of the null hypothesis that it is equally likely that a randomly selected value from one sample will be less than or greater than a randomly selected value from a second sample. It is the non-parametric version of the two-sample T-test.

Like many non-parametric **pingouin** tests, it can take values of tail that are 'two-sided', 'one-sided', 'greater', or 'less'. Below, we ask if a randomly selected value from `x` is greater than one from `y`, with the null that it is not.

```python
x = prng.uniform(low=0, high=1, size=20)
y = prng.uniform(low=0.2, high=1.2, size=20)
pg.mwu(x, y, alternative="greater")
```

### Spearman Correlation

The Spearman correlation coefficient is the Pearson correlation coefficient between the rank variables, and does not assume normality of data.

```python
mean, cov = [4, 6], [(1, 0.5), (0.5, 1)]
x, y = prng.multivariate_normal(mean, cov, 30).T
pg.corr(x, y, method="spearman").round(2)
```

### Kruskal-Wallace

The Kruskal-Wallis H-test tests the null hypothesis that the population median of all of the groups are equal. It is a non-parametric version of ANOVA. The test works on 2 or more independent samples, which may have different sizes.

```python
df = pg.read_dataset("anova")
df.head()
```

```python
pg.kruskal(data=df, dv="Pain threshold", between="Hair color")
```

### The Chi-Squared Test

The chi-squared test is used to determine whether there is a significant difference between the expected frequencies and the observed frequencies in one or more categories. This test can be used to evaluate the quality of a categorical variable in a classification problem or to check the similarity between two categorical variables.

There are two conditions for a chi-squared test:

- Independence: Each case that contributes a count to the table must be independent of all the other cases in the table.

- Sample size or distribution: Each particular case (ie cell count) must have at least 5 expected cases.

Let's see an example from the **pingouin** docs: whether gender is a good predictor of heart disease. First, let's load the data and look at the gender split in total:

```python
chi_data = pg.read_dataset("chi2_independence")
chi_data["sex"].value_counts(ascending=True)
```

If gender is *not* a predictor, we would expect a roughly similar split between those who have heart disease and those who do not. Let's look at the observerd versus the expected split once we categorise by gender and 'target' (heart disease or not).

```python
expected, observed, stats = pg.chi2_independence(chi_data, x="sex", y="target")
observed - expected
```

So we have fewer in the 0, 0 and 1, 1 buckets than expected but more in the 0, 1 and 1, 0 buckets. Let's now see how the test interprets this:

```python
stats.round(3)
```

From these, it is clear we can reject the null and therefore it seems like gender is a good predictor of heart disease.

### Shapiro-Wilk Test for Normality

Note that the null here is that the distribution *is* normal, so normality is only rejected when the p-value is sufficiently small.

```python
x = prng.normal(size=20)
pg.normality(x)
```

The test can also be run on multiple variables in a dataframe:

```python
df = pg.read_dataset("ancova")
pg.normality(df[["Scores", "Income", "BMI"]], method="normaltest").round(3)
```