# Tutorial-IllinoisGRMHD: add_fluxes_and_source_terms_to_hydro_rhss.C

## Authors: Leo Werneck & Zach Etienne

**This module is currently under development**

## In this tutorial module we explain how we add the flux and source terms to the right-hand side of the hydrodynamic variables

### Required and recommended citations:

* **(Required)** Etienne, Z. B., Paschalidis, V., Haas R., Mösta P., and Shapiro, S. L. IllinoisGRMHD: an open-source, user-friendly GRMHD code for dynamical spacetimes. Class. Quantum Grav. 32 (2015) 175009. ([arxiv:1501.07276](http://arxiv.org/abs/1501.07276)).
* **(Required)** Noble, S. C., Gammie, C. F., McKinney, J. C., Del Zanna, L. Primitive Variable Solvers for Conservative General Relativistic Magnetohydrodynamics. Astrophysical Journal, 641, 626 (2006) ([astro-ph/0512420](https://arxiv.org/abs/astro-ph/0512420)).
* **(Recommended)** Del Zanna, L., Bucciantini N., Londrillo, P. An efficient shock-capturing central-type scheme for multidimensional relativistic flows - II. Magnetohydrodynamics. A&A 400 (2) 397-413 (2003). DOI: 10.1051/0004-6361:20021641 ([astro-ph/0210618](https://arxiv.org/abs/astro-ph/0210618)).

If using the version of `IllinoisGRMHD` with piecewise polytropic *or* tabulated (coming soon!) EOS support, then the following citation is also required:

* **(Required)** Etienne, Z. B., Werneck, L., Paschalidis, V., Haas R., Mösta P., and Shapiro, S. L., *IllinoisGRMHD github repository* (2019). Source Code URL: https://github.com/zachetienne/nrpytutorial/tree/master/IllinoisGRMHD/.

### Dependencies

The files generated in this tutorial notebook depends on the following files:

*  [GRHD/equations NRPy+ module](/edit/NRPyIGM/GRHD/equations.py) \[[**tutorial**](/notebooks/NRPyIGM/Tutorial-GRHD_Equations-Cartesian.ipynb)\]
*  [GRFFE/equations NRPy+ module](/edit/NRPyIGM/GRFFE/equations.py) \[[**tutorial**](/notebooks/NRPyIGM/Tutorial-GRFFE_Equations-Cartesian.ipynb)\]
*  [GRMHD/equations NRPy+ module](/edit/NRPyIGM/GRMHD/equations.py) \[[**tutorial**](/notebooks/NRPyIGM/Tutorial-GRMHD_Equations-Cartesian.ipynb)\]
* `IllinoisGRMHD_EoS_lowlevel_functs.C` \[[**tutorial**](Tutorial-IllinoisGRMHD__EoS_lowlevel_functs.ipynb)\]

# Table of Contents
$$\label{toc}$$

This module is organized as follows

0. [Step 0](#src_dir): **Source directory creation**
1. [Step 1](#mhdflux_terms): **Generating MHD flux terms using NRPy+**
    1. [Step 1.a](#characteristic_speeds): *Computing characteristic speeds*
        1. [Step 1.a.i](#compute_v0_squared): Computing $v_{0}^{2}$
        1. [Step 1.a.ii](#compute_cp_and_cm): Computing $c_{+}$ and $c_{-}$
    1. [Step 1.b](#compute_cmax_and_cmin): *Computing $c_{\max}$ and $c_{\min}$*
    1. [Step 1.c](#computing_fhll): *Computing $F^{\rm HLL}$*
    1. [Step 1.d](#grmhd_flux): *Computing the GRMHD flux*
    1. [Step 1.e](#flux_dirn_i__h): *The `smallb4R_and_L.h`, `flux_dirn_x.h`, `flux_dirn_y.h`, and `flux_dirn_z.h` files*
1. [Step 2](#add_fluxes_and_source_terms_to_hydro_rhss__c): **`add_fluxes_and_source_terms_to_hydro_rhss.C`**
    1. [Step 2.a](#reading_everything_in): *Reading in input variables*
    1. [Step 2.b](#facevals): *Computing face values*
    1. [Step 2.c](#mhdflux): *Computing MHD flux terms*
    1. [Step 2.d](#partial_i_gmunu_alpha): *Computing $\partial_{i}g_{\mu\nu}$ and $\partial_{i}\alpha$*
    1. [Step 2.e](#tau_stilde_source_terms): *The $\tilde{\tau}$ and $\tilde{S}_{i}$ source terms*
    1. [Step 2.f](#add_flux_terms): *Adding fluxes to the hydro RHSs*
1. [Step 3](#code_validation): **Code validation**
1. [Step 4](#latex_pdf_output): **Output this notebook to $\LaTeX$-formatted PDF file**

# Step 0: Source directory creation \[Back to [top](#toc)\]
$$\label{src_dir}$$

We will now use the [cmdline_helper.py NRPy+ module](Tutorial-Tutorial-cmdline_helper.ipynb) to create the source directory within the `IllinoisGRMHD` NRPy+ directory, if it does not exist yet.

```python
# Step 0: Creation of the IllinoisGRMHD source directory
# Step 0a: Add NRPy's directory to the path
# https://stackoverflow.com/questions/16780014/import-file-from-parent-directory
import os,sys
nrpy_dir_path = os.path.join("..","..")
if nrpy_dir_path not in sys.path:
    sys.path.append(nrpy_dir_path)

# Step 0b: Load up cmdline_helper and create the directory
import cmdline_helper as cmd
outdir = os.path.join("..","src")
cmd.mkdir(outdir)

# Step 0c: Set up header file output path
NRPy_headers_dir_path = os.path.join(outdir,"NRPy_generated_headers")
cmd.mkdir(NRPy_headers_dir_path)
```

# Step 1: Generating MHD flux terms using NRPy+ \[Back to [top](#toc)\]
$$\label{mhdflux_terms}$$

We will now use NRPy+ and its modules to generate the MHD flux terms for the conservative variables. We will make extensive use of the [GRFFE/equations](/edit/NRPyIGM/GRFFE/equations.py) \[[**tutorial**](/notebooks/NRPyIGM/Tutorial-GRFFE_Equations-Cartesian.ipynb)\], [GRHD/equations](/edit/NRPyIGM/GRHD/equations.py) \[[**tutorial**](/notebooks/NRPyIGM/Tutorial-GRHD_Equations-Cartesian.ipynb)\], and [GRMHD/equations](/edit/NRPyIGM/GRMHD/equations.py) \[[**tutorial**](/notebooks/NRPyIGM/Tutorial-GRMHD_Equations-Cartesian.ipynb)\] NRPy+ modules, so we encourage the reader to go through their tutorial notebooks carefully.

## Step 1.a: Computing characteristic speeds \[Back to [top](#toc)\]
$$\label{characteristic_speeds}$$

In this function we will implement the algorithm discussed in the [inlined_functions.C tutorial notebook](Tutorial-IllinoisGRMHD__inlined_functions.ipynb), repeated here for the sake of the reader.

We approximate the general GRMHD dispersion relation (eq. 27 of [Gammie & McKinney (2003)](https://arxiv.org/pdf/astro-ph/0301509.pdf)) by the simpler expression

$$
\omega_{\rm cm}^{2} = \left[v_{\rm A}^{2} + c_{\rm s}^{2}\left(1-v_{\rm A}^{2}\right)\right]k_{\rm cm}^{2}\ ,
$$

where $\omega_{\rm cm}=-k_{\mu}u^{\mu}$ is the frequency and $k_{\rm cm}^{2} = K_{\mu}K^{\mu}$ the wavenumber of an MHD wave mode in the frame comoving with the fluid, where $K_{\mu}$ is defined as the projection of the wave vector $k^{\nu}$ onto the direction normal to $u^{\nu}$: $K_{\mu} = \left(g_{\mu\nu}+u_{\mu}u_{\nu}\right)k^{\nu}$. $c_{\rm s}$ is the sound speed, and $v_{\rm A}$ is the Alfvén speed, given by

$$
v_{\rm A} = \sqrt{\frac{b^{2}}{\rho_{b}h + b^{2}}}\ .
$$

With these definitions, we may then solve the approximate dispersion relation above along direction $i$, noting that in the comoving frame $k_{\mu} = \left(-\omega,k_{j}\delta^{j}_{\ i}\right)$ and the wave (phase) velocity is $c_{\pm} = \left.\omega\middle/\left(k_{j}\delta^{j}_{\ i}\right)\right.$. The dispersion can then be written as a quadratic equation for $c_{\pm}$:

$$
ac_{\pm}^{2} + bc_{\pm} + c = 0\ ,
$$

with

$$
\boxed{
\begin{align}
v_{\rm A}^{2} &= \frac{b^{2}}{\rho_{b}h + b^{2}}\\
c_{\rm s}^{2} &= \left.\left[\frac{dP_{\rm cold}}{d\rho_{b}} + \Gamma_{\rm th}\left(\Gamma_{\rm th}-1\right)\epsilon_{\rm th}\right]\middle/h\right.\\
v_{0}^{2} &= v_{\rm A}^{2} + c_{\rm s}^{2}\left(1-v_{\rm A}^{2}\right)\\
a &= \left(1-v_{0}^{2}\right)\left(u^{0}\right)^{2} - v_{0}^{2}g^{00}\\
b &= 2v_{0}^{2}g^{i0} - 2u^{i}u^{0}\left(1-v^{2}_{0}\right)\\
c &= \left(1-v_{0}^{2}\right)\left(u^{i}\right)^{2} - v_{0}^{2}g^{ii}\\
c_{+} &= \max\left(\frac{-b \pm \sqrt{b^{2}-4ac}}{2a}\right)\\
c_{-} &= \min\left(\frac{-b \pm \sqrt{b^{2}-4ac}}{2a}\right)
\end{align}
}\ .
$$

We now implement the boxed equations below.

### Step 1.a.i: Computing $v_{0}^{2}$ \[Back to [top](#toc)\]
$$\label{compute_v0_squared}$$

We start by defining a function that returns $v_{0}^{2}$. To this end, we compute the following quantities:

$$
\boxed{
\begin{align}
v_{\rm A}^{2} &= \frac{b^{2}}{\rho_{b}h + b^{2}}\\
c_{\rm s}^{2} &= \left.\left[\frac{dP_{\rm cold}}{d\rho_{b}} + \Gamma_{\rm th}\left(\Gamma_{\rm th}-1\right)\epsilon_{\rm th}\right]\middle/h\right.\\
v_{0}^{2} &= v_{\rm A}^{2} + c_{\rm s}^{2}\left(1-v_{\rm A}^{2}\right)
\end{align}
}\ .
$$

```python
def compute_v0_squared( smallbsquared, rhob,dPcold_drho,h,Gamma_th,eps_th ):
    # Compute v_{A}^{2}
    vA_squared = smallbsquared/( rhob*h + smallbsquared )

    # Compute c_{s}^{2}
    c_s_squared = ( dPcold_drho + Gamma_th*(Gamma_th - 1)*eps_th ) / h

    # Compute v_{0}^{2}
    v0_squared = vA_squared + (1 - vA_squared) * c_s_squared

    return v0_squared
```

### Step 1.a.ii: Computing $c_{+}$ and $c_{-}$ \[Back to [top](#toc)\]
$$\label{compute_cp_and_cm}$$

We now compute $c_{+}$ and $c_{-}$, using the [equations above](#characteristic_speeds). Note, however, that because we are using symbolic expressions, we cannot simply compute minimum and maximum values of expressions. Instead, we evaluate

$$
\Delta^{2} \equiv b^{2} - 4ac\ ,
$$

and then

$$
\Delta = \sqrt{\frac{1}{2}\left(\Delta^{2} + \left|\Delta^{2}\right|\right)}\ ,
$$

which ensures that $\Delta \geq 0$ and hence guaratees that $c_{\pm}$ are determined by considering $\pm\Delta$ in their respective expressions. Thus we have:

$$
\boxed{
\begin{align}
a &= \left(1-v_{0}^{2}\right)\left(u^{0}\right)^{2} - v_{0}^{2}g^{00}\\
b &= 2v_{0}^{2}g^{i0} - 2u^{i}u^{0}\left(1-v^{2}_{0}\right)\\
c &= \left(1-v_{0}^{2}\right)\left(u^{i}\right)^{2} - v_{0}^{2}g^{ii}\\
c_{+} &= \frac{-b + \Delta}{2a}\\
c_{-} &= \frac{-b - \Delta}{2a}
\end{align}
}\ .
$$

```python
def compute_cp_and_cm(flux_dirn, alpha, g4UU, smallbsquared, rhob,dPcold_drho,h,Gamma_th,eps_th,u4U):

    # Compute v_{0}^{2}
    v0_squared = compute_v0_squared( smallbsquared,rhob,dPcold_drho,h,Gamma_th,eps_th )

    # Compute a = (1-v_{0}^{2})*(u^{0})^{2} - v_{0}^{2} g^{00}
    a = (1 - v0_squared) * u4U[0]**2 + v0_squared / alpha**2

    # Compute b = 2v_{0}^{2} g^{i0} - 2u^{i}u^{0}(1-v_{0}^{2})
    # and     c = (1-v_{0}^{2})(u^{i})^{2} - v_{0}^{2}g^{ii}
    b = 2 * v0_squared * g4UU[flux_dirn+1][0] - 2 * u4U[flux_dirn+1] * u4U[0] * (1 - v0_squared)
    c = (1 - v0_squared) * u4U[flux_dirn+1]**2 - v0_squared * g4UU[flux_dirn+1][flux_dirn+1]

    # Compute cplus and cminus, making sure we do
    # not use a negative number in the square root
    detm = b*b - 4*a*c
    detm = sp.sqrt(sp.Rational(1,2)*(detm + nrpyAbs(detm)))
    cplus  = sp.Rational(1,2)*(-b/a + detm/a)
    cminus = sp.Rational(1,2)*(-b/a - detm/a)

    return cplus, cminus
```

## Step 1.b: Computing $c_{\max}$ and $c_{\min}$ \[Back to [top](#toc)\]
$$\label{compute_cmax_and_cmin}$$

We now compute $c_{\max}$ and $c_{\min}$ based on $c_{+,R}$, $c_{-,R}$, $c_{+,L}$, and $c_{-,L}$. The basic idea would be to simply find

\begin{align}
c_{\max} &= \max\left(0,c_{+,R},c_{+,L}\right)\ ,\\
c_{\min} &= -\min\left(0,c_{-,R},c_{-,L}\right)\ .
\end{align}

However, because we are working with symbolic expressions, sympy would not be able to discover whether $c_{+,R}>c_{+,L}$ or otherwise (analogously for $c_{-,L,R}$). Therefore, we use the following trick:

\begin{align}
c_{\max} &= \frac{1}{2}\left(c_{+,R} + c_{+,L} + \left|c_{+,R}-c_{+,L}\right|\right) \to \frac{1}{2}\left(c_{\max} + \left|c_{\max}\right|\right)\ ,\\
c_{\min} &= \frac{1}{2}\left(c_{-,R} + c_{-,L} - \left|c_{-,R}-c_{-,L}\right|\right) \to -\frac{1}{2}\left(c_{\min} - \left|c_{\min}\right|\right)\ .
\end{align}

```python
def find_cmax_and_cmin(flux_dirn, alpha_face,beta_faceU,gamma_faceDD,
                       smallbsquared_R, smallbsquared_L,
                       rhob_R,dPcold_drho_R,h_R,eps_th_R,u4_RU,
                       rhob_L,dPcold_drho_L,h_L,eps_th_L,u4_LU,
                       Gamma_th):

    # Compute g^{\mu\nu} from input
    gamma_faceUU,gamma_facedet = ixp.generic_matrix_inverter3x3(gamma_faceDD)
    AB4m.g4UU_ito_BSSN_or_ADM("ADM",betaU=beta_faceU,alpha=alpha_face,gammaUU=gamma_faceUU)
    g4_faceUU = AB4m.g4UU

    # Compute cplus_{R,L} and cminus_{R,L}
    cpR,cmR = compute_cp_and_cm(flux_dirn, alpha_face,g4_faceUU, smallbsquared_R, rhob_R,dPcold_drho_R,h_R,Gamma_th,eps_th_R,u4_RU)
    cpL,cmL = compute_cp_and_cm(flux_dirn, alpha_face,g4_faceUU, smallbsquared_L, rhob_L,dPcold_drho_L,h_L,Gamma_th,eps_th_L,u4_LU)

    # Compute cmax = max(0,cpR,cpL)
    cmax = sp.Rational(1,2)*(cpR+cpL+nrpyAbs(cpR-cpL))
    cmax = sp.Rational(1,2)*(cmax+nrpyAbs(cmax))

    # Compute cmin = min(0,cmR,cmL)
    cmin = sp.Rational(1,2)*(cmR+cmL-nrpyAbs(cmR-cmL))
    # From above expression, suppose cmin = -0.1, then
    #   the following expression yields:
    #   -1/2 (-0.1 - 0.1) = 0.1 , which indeed is
    #   -min(cmin,0)
    cmin = -sp.Rational(1,2)*(cmin-nrpyAbs(cmin))

    return cmax, cmin
```

## Step 1.c: Computing $F^{\rm HLL}$ \[Back to [top](#toc)\]
$$\label{computing_fhll}$$

We also set up a function to compute the standard [Harten-Lax-van Lee (HLL) approximate Riemann solver](https://epubs.siam.org/doi/10.1137/1025002) (see equation (25) of [Etienne *et al.*](https://arxiv.org/pdf/1501.07276.pdf) for the exact expression implemented below), which will be used to compute fluxes for the conservative variables:

$$
\boxed{F^{\rm HLL} = \frac{c^{-}F_{r} + c^{+}F_{l} - c^{+}c^{-}\left(U_{r} - U_{l}\right)}{c^{+} + c^{-}}}\ .
$$

In the expression above, $F_{r,l}$ represent the flux terms for the the conservative variables $U_{r,l}$ on the right,left face, respectively.

```python
# The code below has been adapted from the HLLE_solver() found in
# Tutorial-GiRaFFE_NRPy_Ccode_library-Stilde-flux.ipynb.
def compute_FHLL(cp,cm,F_R,F_L,U_R,U_L):
    # Compute F^{HLL} = ( cm * Fr + cp * Fl - cm*cp*(Ur - Ul) ) / (cp + cm)
    return ( cm * F_R + cp * F_L - cm*cp*(U_R - U_L) ) / ( cp + cm )
```

## Step 1.d: Computing the GRMHD flux \[Back to [top](#toc)\]
$$\label{grmhd_flux}$$

We now compute the GRMHD flux terms for the conservative variables $\left(\rho_{\star},\tilde{\tau},\tilde{S}_{i}\right)$, namely

$$
\begin{aligned}
F^{j}_{\rho_{\star}} &= \rho_{\star} v^{j}\ ,\\
F^{j}_{\tilde{\tau}} &= \alpha^{2}\sqrt{\gamma}T^{0j} - \rho_{\star}v^{j}\ ,\\
F_{\tilde{S}_{\ i}}^{j} &= \alpha\sqrt{\gamma}T^{j}_{\ i}\ ,
\end{aligned}
$$

where $j$ represents the flux direction (${\rm flux\_dirn}$) and $i$ is the standard spatial index. After this, we compute the $F^{\rm HLL}$ flux terms for the conservative variables. These quantities are computed using the [GRFFE/equations](/edit/NRPyIGM/GRFFE/equations.py), [GRHD/equations](/edit/NRPyIGM/GRHD/equations.py), and [GRMHD/equations](/edit/NRPyIGM/GRMHD/equations.py) NRPy+ modules, specifically the functions:

* `GRFFE.compute_smallbsquared()`: computes $b^{2}$
* `GRHD.compute_enthalpy()`: computes $h$
* `GRMHD.compute_GRMHD_T4UU()`: computes $\left(T_{\rm GRMHD}\right)^{\mu\nu}$
* `GRMHD.compute_GRMHD_T4UD()`: computes $\left(T_{\rm GRMHD}\right)^{\mu}_{\ \ \nu}$
* `GRHD.compute_rho_star()`: computes $\rho_{\star}$
* `GRHD.compute_tau_tilde()`: computes $\tilde{\tau}$
* `GRHD.compute_S_tildeD()`: computes $\tilde{S}_{i}$
* `GRHD.compute_rho_star_fluxU()`: computes $F^{j}_{\rho_{\star}}$
* `GRHD.compute_tau_tilde_fluxU()`: computes $F^{j}_{\tilde{\tau}}$
* `GRHD.compute_S_tilde_fluxUD()`: computes $F_{\tilde{S}_{\ i}}^{j}$

```python
def mhdflux(flux_dirn, alpha,betaU,gammaDD, sqrtgammaDET, Gamma_th,
            rhob_R,P_R,dPcold_drho_R,eps_cold_R,epsilon_R,vRU, u4_RU,smallb4UR,
            rhob_L,P_L,dPcold_drho_L,eps_cold_L,epsilon_L,vLU, u4_LU,smallb4UL):

    # Compute g_{\mu\nu}
    AB4m.g4DD_ito_BSSN_or_ADM("ADM",gammaDD=gammaDD,alpha=alpha,betaU=betaU)
    g4DD = AB4m.g4DD

    # Compute b^{2}_{R}
    GRFFE.compute_smallbsquared(gammaDD,betaU,alpha, smallb4UR)
    smallbsquared_R = GRFFE.smallbsquared

    # Compute b^{2}_{R}
    GRFFE.compute_smallbsquared(gammaDD,betaU,alpha, smallb4UL)
    smallbsquared_L = GRFFE.smallbsquared

    # Compute h_{R}
    GRHD.compute_enthalpy(rhob_R,P_R,epsilon_R)
    h_R = GRHD.h

    # Compute h_{L}
    GRHD.compute_enthalpy(rhob_L,P_L,epsilon_L)
    h_L = GRHD.h

    # Compute epsilon_thermal_{R,L}
    eps_th_R = epsilon_R - eps_cold_R
    eps_th_L = epsilon_L - eps_cold_L

    # Compute cmax, cmin
    cmax, cmin = find_cmax_and_cmin(flux_dirn, alpha,betaU,gammaDD,          \
                                    smallbsquared_R, smallbsquared_L,        \
                                    rhob_R,dPcold_drho_R,h_R,eps_th_R,u4_RU, \
                                    rhob_L,dPcold_drho_L,h_L,eps_th_L,u4_LU, \
                                    Gamma_th)

    # Compute GRMHD T^{\mu\nu}_{R}
    GRMHD.compute_GRMHD_T4UU(gammaDD, betaU, alpha, rhob_R, P_R, epsilon_R, u4_RU, smallb4_RU, smallbsquared_R)
    GRMHD.compute_GRMHD_T4UD(gammaDD, betaU, alpha, GRMHD.GRHDT4UU,GRMHD.GRFFET4UU)
    T4R_UU = GRMHD.T4UU
    T4R_UD = GRMHD.T4UD

    # Compute GRMHD T^{\mu\nu}_{L}
    GRMHD.compute_GRMHD_T4UU(gammaDD, betaU, alpha, rhob_L, P_L, epsilon_L, u4_LU, smallb4_LU, smallbsquared_L)
    GRMHD.compute_GRMHD_T4UD(gammaDD, betaU, alpha, GRMHD.GRHDT4UU,GRMHD.GRFFET4UU)
    T4L_UU = GRMHD.T4UU
    T4L_UD = GRMHD.T4UD

    # Compute Conservatives from Primitives (R)
    GRHD.compute_rho_star( alpha, sqrtgammaDET, rhob_R,u4_RU)
    GRHD.compute_tau_tilde(alpha, sqrtgammaDET, T4R_UU,GRHD.rho_star)
    GRHD.compute_S_tildeD( alpha, sqrtgammaDET, T4R_UD)
    rho_star_R  = GRHD.rho_star
    tau_tilde_R = GRHD.tau_tilde
    S_tilde_RD  = GRHD.S_tildeD

    # Compute Conservatives from Primitives (L)
    GRHD.compute_rho_star( alpha, sqrtgammaDET, rhob_L,u4_LU)
    GRHD.compute_tau_tilde(alpha, sqrtgammaDET, T4L_UU,GRHD.rho_star)
    GRHD.compute_S_tildeD( alpha, sqrtgammaDET, T4L_UD)
    rho_star_L  = GRHD.rho_star
    tau_tilde_L = GRHD.tau_tilde
    S_tilde_LD  = GRHD.S_tildeD

    # Compute flux terms (R)
    GRHD.compute_rho_star_fluxU(                      vRU,        rho_star_R)
    GRHD.compute_tau_tilde_fluxU(alpha, sqrtgammaDET, vRU,T4R_UU, rho_star_R)
    GRHD.compute_S_tilde_fluxUD( alpha, sqrtgammaDET,     T4R_UD)
    rho_star_fluxRU  = GRHD.rho_star_fluxU
    tau_tilde_fluxRU = GRHD.tau_tilde_fluxU
    S_tilde_fluxRUD  = GRHD.S_tilde_fluxUD

    # Compute flux terms (L)
    GRHD.compute_rho_star_fluxU(                      vLU,        rho_star_L)
    GRHD.compute_tau_tilde_fluxU(alpha, sqrtgammaDET, vLU,T4L_UU, rho_star_L)
    GRHD.compute_S_tilde_fluxUD( alpha, sqrtgammaDET,     T4L_UD)
    rho_star_fluxLU  = GRHD.rho_star_fluxU
    tau_tilde_fluxLU = GRHD.tau_tilde_fluxU
    S_tilde_fluxLUD  = GRHD.S_tilde_fluxUD

    # Compute F^{HLL} for rho_star
    rho_star_FHLLU_fluxdirn = compute_FHLL(cmax,cmin,rho_star_fluxRU[flux_dirn],rho_star_fluxLU[flux_dirn],
                                           rho_star_R,rho_star_L)

    # Compute F^{HLL} for tau_tilde
    tau_tilde_FHLLU_fluxdirn = compute_FHLL(cmax,cmin,tau_tilde_fluxRU[flux_dirn],tau_tilde_fluxLU[flux_dirn],
                                            tau_tilde_R,tau_tilde_L)

    S_tilde_FHLLU_fluxdirn_D = ixp.zerorank1()
    for i in range(DIM):
        S_tilde_FHLLU_fluxdirn_D[i] = compute_FHLL(cmax,cmin,
                                                  S_tilde_fluxRUD[flux_dirn][i],
                                                  S_tilde_fluxLUD[flux_dirn][i],
                                                  S_tilde_RD[i],S_tilde_LD[i])

    return rho_star_FHLLU_fluxdirn,tau_tilde_FHLLU_fluxdirn,S_tilde_FHLLU_fluxdirn_D,cmax,cmin
#     return rho_star_FHLL_fluxdirn_U,tau_tilde_FHLL_fluxdirn_U,S_tilde_FHLL_fluxdirn_UD,cmax,cmin,smallbsquared_R,\
#            S_tilde_RD[0],S_tilde_LD[0],S_tilde_fluxRUD[flux_dirn][0],S_tilde_fluxLUD[flux_dirn][0], \
#            S_tilde_RD[1],S_tilde_LD[1],S_tilde_fluxRUD[flux_dirn][1],S_tilde_fluxLUD[flux_dirn][1], \
#            S_tilde_RD[2],S_tilde_LD[2],S_tilde_fluxRUD[flux_dirn][2],S_tilde_fluxLUD[flux_dirn][2], \
#            tau_tilde_fluxRU[flux_dirn],tau_tilde_fluxLU[flux_dirn],tau_tilde_R,tau_tilde_L
```

## Step 1.e: The `smallb4R_and_L.h`, `flux_dirn_x.h`, `flux_dirn_y.h`, and `flux_dirn_z.h` files \[Back to [top](#toc)\]
$$\label{flux_dirn_i__h}$$

We now write down the C files that will compute $b^{\mu}_{R,L}$ and the fluxes in the $(x,y,z)$-directions.

```python
# Import the necessary Python/NRPy+ modules
import time                                     # Python module: allows us to time code evaluation
import sympy as sp                              # Python module: symbolic expressions functionality
from outputC import *                           # NRPy+  module: C code output routines
import indexedexp as ixp                        # NRPy+  module: funcionality to work with vectors/tensors
import BSSN.ADMBSSN_tofrom_4metric as AB4m      # NRPy+  module: computes ADM 4-metric from ADM or BSSN quantities
import IllinoisGRMHD_output_functions as IGMout # NRPy+  module: IllinoisGRMHD output file functions
import GRHD.equations  as GRHD                  # NRPy+  module: Collection GRHD related functions
import GRFFE.equations as GRFFE                 # NRPy+  module: Collection GRFFE related functions
import GRMHD.equations as GRMHD                 # NRPy+  module: Collection GRMHD related functions

# Step 1: Set spatial dimension to 3
DIM = 3

# Step 1: Set up alpha
alpha_ito_FACEVAL  = sp.sympify(1) + sp.Symbol("FACEVAL[CM_LAPM1]",real=True)
# Step 2: Set up beta^{i}
betaU_ito_FACEVAL   = ixp.zerorank1()
for i in range(DIM):
    betaU_ito_FACEVAL[i] = sp.Symbol('FACEVAL[CM_SHIFT'+chr(ord('X')+i)+"]",real=True)
# Step 3: Set up \bar\gamma_{ij}
gammabarfaceDD = ixp.zerorank2()
for i in range(DIM):
    for j in range(i,DIM):
        gammabarfaceDD[i][j] = gammabarfaceDD[j][i] = sp.Symbol('FACEVAL[CM_GAMMATILDE'+chr(ord('X')+i)+chr(ord('X')+j)+"]",real=True)

# Step 4: Set up \bar\gamma^{ij}
gammabarfaceUU, gammabardet = ixp.generic_matrix_inverter3x3(gammabarfaceDD)

# Step 6: Set up psi^{4} and psi^{-4}
# psi   = sp.Symbol("FACEVAL[CM_PSI]",real=True)
psi2  = sp.Symbol("FACEVAL_PSI2_and_PSI4[0]",real=True)
psi4  = sp.Symbol("FACEVAL_PSI2_and_PSI4[1]",real=True)
sqrtgamma = psi2*psi4
sqrtgammaDET = sqrtgamma

# Step 7: Set up \gamma_{ij}, \gamma^{ij}, and sqrt(\gamma)
gamma_ito_gammabarfaceDD = ixp.zerorank2()
for i in range(DIM):
    for j in range(DIM):
        gamma_ito_gammabarfaceDD[i][j] = psi4  * gammabarfaceDD[i][j]

# Step 8: Declare needed variables, named according to IllinoisGRMHD
Gamma_th = sp.Symbol("Gamma_th",real=True)
rhob_L,P_L,P_cold_L,dPcold_drho_L,eps_cold_L = sp.symbols("Ul[RHOB] Ul[PRESSURE] P_coldl dPcold_drhol eps_coldl",real=True)
rhob_R,P_R,P_cold_R,dPcold_drho_R,eps_cold_R = sp.symbols("Ur[RHOB] Ur[PRESSURE] P_coldr dPcold_drhor eps_coldr",real=True)

# Step 9: Compute epsilon
epsilon_L = eps_cold_L + (P_L - P_cold_L)/(rhob_L * (Gamma_th - 1))
epsilon_R = eps_cold_R + (P_R - P_cold_R)/(rhob_R * (Gamma_th - 1))

# Step 10: Compute u^{\mu} from v^{i}
vLU = [sp.Symbol("Ul[VX]",real=True),sp.Symbol("Ul[VY]",real=True),sp.Symbol("Ul[VZ]",real=True)]
vRU = [sp.Symbol("Ur[VX]",real=True),sp.Symbol("Ur[VY]",real=True),sp.Symbol("Ur[VZ]",real=True)]

# Step 11: Compute u^{\mu}_{R,L} from v^{i}_{R,L}
GRHD.u4U_in_terms_of_vU__rescale_vU_by_applying_speed_limit(alpha_ito_FACEVAL,betaU_ito_FACEVAL,gamma_ito_gammabarfaceDD, vRU)
u4_RU = GRHD.u4U_ito_vU
GRHD.u4U_in_terms_of_vU__rescale_vU_by_applying_speed_limit(alpha_ito_FACEVAL,betaU_ito_FACEVAL,gamma_ito_gammabarfaceDD, vLU)
u4_LU = GRHD.u4U_ito_vU

# Step 12: Define B^{i}_{R,L} and sqrt4pi
sqrt4pi      = sp.symbols('sqrt4pi', real=True)
B_notilde_RU = [sp.Symbol("Ur[BX_CENTER]",real=True),sp.Symbol("Ur[BY_CENTER]",real=True),sp.Symbol("Ur[BZ_CENTER]",real=True)]
B_notilde_LU = [sp.Symbol("Ul[BX_CENTER]",real=True),sp.Symbol("Ul[BY_CENTER]",real=True),sp.Symbol("Ul[BZ_CENTER]",real=True)]

# Step 13: Compute b^{\mu}_{R,L}
GRFFE.compute_smallb4U(gamma_ito_gammabarfaceDD,betaU_ito_FACEVAL,alpha_ito_FACEVAL, u4_RU,B_notilde_RU, sqrt4pi)
smallb4_RU = GRFFE.smallb4U
GRFFE.compute_smallb4U(gamma_ito_gammabarfaceDD,betaU_ito_FACEVAL,alpha_ito_FACEVAL, u4_LU,B_notilde_LU, sqrt4pi)
smallb4_LU = GRFFE.smallb4U

# Step 14: Output result to file
string = outputC([smallb4_RU[0],smallb4_RU[1],smallb4_RU[2],smallb4_RU[3],
                 smallb4_LU[0],smallb4_LU[1],smallb4_LU[2],smallb4_LU[3]],
                 ["smallb4U0_R","smallb4U1_R","smallb4U2_R","smallb4U3_R",
                  "smallb4U0_L","smallb4U1_L","smallb4U2_L","smallb4U3_L"],
                  filename="returnstring", params="outCverbose=False")

filename="smallb4R_and_L.h"
filepath = os.path.join(NRPy_headers_dir_path,filename)
IGMout.NRPy_IGM_write_to_file(filepath,filename,string)

smallb4UR = [sp.Symbol("smallb4U0_R",real=True),
             sp.Symbol("smallb4U1_R",real=True),
             sp.Symbol("smallb4U2_R",real=True),
             sp.Symbol("smallb4U3_R",real=True)]
smallb4UL = [sp.Symbol("smallb4U0_L",real=True),
             sp.Symbol("smallb4U1_L",real=True),
             sp.Symbol("smallb4U2_L",real=True),
             sp.Symbol("smallb4U3_L",real=True)]
```

    Just generated the file: ../src/NRPy_generated_headers/smallb4R_and_L.h

```python
start = time.time()
def flux_terms(flux_dirn):

    start = time.time()

    rho_star_FHLLU_fluxdirn,tau_tilde_FHLLU_fluxdirn,\
    S_tilde_FHLLU_fluxdirn_D,cmax,cmin = mhdflux(int(flux_dirn), alpha_ito_FACEVAL,betaU_ito_FACEVAL,gamma_ito_gammabarfaceDD, sqrtgammaDET,
                                                 Gamma_th,
                                                 rhob_R,P_R,dPcold_drho_R,eps_cold_R,epsilon_R,vRU,u4_RU, smallb4UR,
                                                 rhob_L,P_L,dPcold_drho_L,eps_cold_L,epsilon_L,vLU,u4_LU, smallb4UL)

    exprlist = [rho_star_FHLLU_fluxdirn,
                tau_tilde_FHLLU_fluxdirn,
                S_tilde_FHLLU_fluxdirn_D[0],
                S_tilde_FHLLU_fluxdirn_D[1],
                S_tilde_FHLLU_fluxdirn_D[2],
                cmax,cmin]
    varlist   = ["rho_star_flux[index]",
                 "tau_flux[index]",
                 "st_x_flux[index]",
                 "st_y_flux[index]",
                 "st_z_flux[index]",
                 "cmax[index]","cmin[index]"]

#------------------------------------------FOR DEBUGGING PURPOSES------------------------------------------
#     rho_star_FHLLU_fluxdirn,tau_tilde_FHLLU_fluxdirn,S_tilde_FHLLUD_fluxdirn,cmax,cmin,smallbsquared_R, \
#     st_x_r,st_x_l,F_x_r,F_x_l,st_y_r,st_y_l,F_y_r,F_y_l,st_z_r,st_z_l,F_z_r,F_z_l, \
#     tau_tilde_fluxRU_fluxdirn,tau_tilde_fluxLU_fluxdirn,tau_tilde_R,tau_tilde_L = \
#         mhdflux(int(flux_dirn), alpha_ito_FACEVAL,betaU_ito_FACEVAL,gamma_ito_gammabarfaceDD, sqrtgammaDET,
#                 Gamma_th,
#                 rhob_R,P_R,dPcold_drho_R,eps_cold_R,epsilon_R,vRU,u4_RU, smallb4UR,
#                 rhob_L,P_L,dPcold_drho_L,eps_cold_L,epsilon_L,vLU,u4_LU, smallb4UL)
#     exprlist = [rho_star_FHLLU_fluxdirn,
#                 tau_tilde_FHLLU_fluxdirn,
#                 S_tilde_FHLLUD_fluxdirn[0],S_tilde_FHLLUD_fluxdirn[1],S_tilde_FHLLUD_fluxdirn[2],
#                 cmax,cmin,smallbsquared_R,
#                 st_x_r,st_x_l,F_x_r,F_x_l,st_y_r,st_y_l,F_y_r,F_y_l,st_z_r,st_z_l,F_z_r,F_z_l,
#                 tau_tilde_fluxRU_fluxdirn,tau_tilde_fluxLU_fluxdirn,tau_tilde_R,tau_tilde_L]
#     varlist = ["rho_star_flux","tau_flux",
#                "st_x_flux","st_y_flux","st_z_flux",
#                "cmax","cmin","smallb2r",
#                "st_x_r","st_x_l","F_x_r","F_x_l",
#                "st_y_r","st_y_l","F_y_r","F_y_l",
#                "st_z_r","st_z_l","F_z_r","F_z_l",
#                "tau_tilde_fluxRU_fluxdirn","tau_tilde_fluxLU_fluxdirn","tau_tilde_R","tau_tilde_L"]
#------------------------------------------FOR DEBUGGING PURPOSES------------------------------------------

    string = outputC(exprlist,varlist,filename="returnstring", params="outCverbose=False")

    filename="flux_dirn_"+chr(ord('x')+flux_dirn)+".h"
    filepath = os.path.join(NRPy_headers_dir_path,filename)
    IGMout.NRPy_IGM_write_to_file(filepath,filename,string)
    print("Generated "+chr(ord('x')+flux_dirn)+" direction flux file in %5.2lf seconds"%(time.time()-start))
```

```python
# Generate flux terms C code in parallel, if possible
try:
    if os.name == 'nt':
        # It's a mess to get working in Windows, so we don't bother. :/
        #  https://medium.com/@grvsinghal/speed-up-your-python-code-using-multiprocessing-on-windows-and-jupyter-or-ipython-2714b49d6fac
        raise Exception("Parallel codegen currently not available in Windows")
    # Step 1.a: Import the multiprocessing module.
    import multiprocessing

    # Step 1.b: Evaluate flux_terms in parallel if possible;
    #           otherwise fallback to serial evaluation:
    pool = multiprocessing.Pool()
    pool.map(flux_terms,range(3))
except:
    # Steps 1.b, alternate: As fallback, evaluate functions in serial.
    #                       This will happen on Android and Windows systems
    for flux_dirn in range(3):
        flux_terms(flux_dirn)
```

    Just generated the file: ../src/NRPy_generated_headers/flux_dirn_y.h
    Generated y direction flux file in 17.99 seconds
    Just generated the file: ../src/NRPy_generated_headers/flux_dirn_x.h
    Generated x direction flux file in 18.04 seconds
    Just generated the file: ../src/NRPy_generated_headers/flux_dirn_z.h
    Generated z direction flux file in 18.06 seconds

# Step 2: `add_fluxes_and_source_terms_to_hydro_rhss.C` \[Back to [top](#toc)\]
$$\label{add_fluxes_and_source_terms_to_hydro_rhss__c}$$

We now start documenting the `add_fluxes_and_source_terms_to_hydro_rhss.C` file from `IllinoisGRMHD`. We will start by writing the preamble of the file, which contains a few useful macros that are used to

1. Compute the facevalue of a metric quantity, say $\lambda$, via:

$$
\lambda_{\rm FACE} = -\frac{1}{16}\lambda_{i-2} + \frac{9}{16}\lambda_{i-1} + \frac{9}{16}\lambda_{i} - \frac{1}{16}\lambda_{i+1}\ ,
$$

1. Compute the ADM 4-metric $g_{\mu\nu}$,

$$
g_{\mu\nu} =
\begin{pmatrix}
-\alpha^{2} + \beta^{\ell}\beta_{\ell} & \beta_{i}\\
\beta_{j} & \gamma_{ij}
\end{pmatrix}\ ,
$$

where $\gamma_{ij} = \psi^{4}\bar{\gamma}_{ij}$ is the physical spatial metric, $\bar{\gamma}_{ij}$ the conformal metric, $\alpha$ the lapse function, and $\beta_{i} = \gamma_{ij}\beta^{j} = \psi^{4}\bar\gamma_{ij}\beta^{j}$ the shift vector.

```python
%%writefile $outdir/add_fluxes_and_source_terms_to_hydro_rhss.C
// Side note: the following values could be used for cell averaged gfs:
//     am2=-1.0/12.0, am1=7.0/12.0, a0=7.0/12.0, a1=-1.0/12.0
// However, since the metric gfs store the grid point values instead of the cell average,
//     the following coefficients should be used:
//     am2 = -1/16, am1 = 9/16, a0 = 9/16, a1 = -1/16
// This will yield the third-order-accurate face values at m-1/2,
//      using values specified at {m-2,m-1,m,m+1}
#define AM2 -0.0625
#define AM1  0.5625
#define A0   0.5625
#define A1  -0.0625
#define COMPUTE_FCVAL(METRICm2,METRICm1,METRIC,METRICp1) (AM2*(METRICm2) + AM1*(METRICm1) + A0*(METRIC) + A1*(METRICp1))

#define COMPUTE_FOURMETRIC(g4tt,g4tx,g4ty,g4tz,g4xx,g4xy,g4xz,g4yy,g4yz,g4zz,CONF_METRIC,psi4)  ( { \
      /* g_{0i} = beta_i */                                             \
      g4tx = psi4*(CONF_METRIC[CM_GAMMATILDEXX]*CONF_METRIC[CM_SHIFTX] + CONF_METRIC[CM_GAMMATILDEXY]*CONF_METRIC[CM_SHIFTY] + CONF_METRIC[CM_GAMMATILDEXZ]*CONF_METRIC[CM_SHIFTZ]); \
      g4ty = psi4*(CONF_METRIC[CM_GAMMATILDEXY]*CONF_METRIC[CM_SHIFTX] + CONF_METRIC[CM_GAMMATILDEYY]*CONF_METRIC[CM_SHIFTY] + CONF_METRIC[CM_GAMMATILDEYZ]*CONF_METRIC[CM_SHIFTZ]); \
      g4tz = psi4*(CONF_METRIC[CM_GAMMATILDEXZ]*CONF_METRIC[CM_SHIFTX] + CONF_METRIC[CM_GAMMATILDEYZ]*CONF_METRIC[CM_SHIFTY] + CONF_METRIC[CM_GAMMATILDEZZ]*CONF_METRIC[CM_SHIFTZ]); \
      /* g_{00} = -alpha^2 + beta^i beta^j gamma_{ij} = -alpha^2 + beta^i beta_i = -alpha^2 + beta^i g_{0i} */ \
      g4tt = -SQR((CONF_METRIC[CM_LAPM1]) + 1.0) + g4tx*CONF_METRIC[CM_SHIFTX] + g4ty*CONF_METRIC[CM_SHIFTY] + g4tz*CONF_METRIC[CM_SHIFTZ]; \
      g4xx = psi4*CONF_METRIC[CM_GAMMATILDEXX];                              \
      g4xy = psi4*CONF_METRIC[CM_GAMMATILDEXY];                              \
      g4xz = psi4*CONF_METRIC[CM_GAMMATILDEXZ];                              \
      g4yy = psi4*CONF_METRIC[CM_GAMMATILDEYY];                              \
      g4yz = psi4*CONF_METRIC[CM_GAMMATILDEYZ];                              \
      g4zz = psi4*CONF_METRIC[CM_GAMMATILDEZZ];                              \
    } )
```

    Overwriting ../src/add_fluxes_and_source_terms_to_hydro_rhss.C

## Step 2.a: Reading in input variables \[Back to [top](#toc)\]
$$\label{reading_everything_in}$$

We now start reading in all variables needed to compute the flux terms. These variables include:

1. $\bar\gamma_{ij}$, the conformal metric
1. $U_{R,L}$, primitive variables on the Right,Left faces
1. $T^{\mu\nu}$, the energy-momentum tensor

```python
%%writefile -a $outdir/add_fluxes_and_source_terms_to_hydro_rhss.C

static void add_fluxes_and_source_terms_to_hydro_rhss(const int flux_dirn,const cGH *cctkGH,const int *cctk_lsh,const int *cctk_nghostzones,CCTK_REAL *dX,
                                                      CCTK_REAL **conf_metric,gf_and_gz_struct *IN_PRIMS,CCTK_REAL **TUPMUNU,
                                                      int numvars_reconstructed,gf_and_gz_struct *OUT_PRIMS_R,gf_and_gz_struct *OUT_PRIMS_L,eos_struct &eos,
                                                      CCTK_REAL *cmax,CCTK_REAL *cmin,
                                                      CCTK_REAL *rho_star_flux,CCTK_REAL *tau_flux,CCTK_REAL *st_x_flux,CCTK_REAL *st_y_flux,CCTK_REAL *st_z_flux,
                                                      CCTK_REAL *rho_star_rhs,CCTK_REAL *tau_rhs,CCTK_REAL *st_x_rhs,CCTK_REAL *st_y_rhs,CCTK_REAL *st_z_rhs) {

  DECLARE_CCTK_PARAMETERS;

  CCTK_REAL dxi[4] = { 1e100,1.0/dX[0],1.0/dX[1],1.0/dX[2] };

  // Notice in the loop below that we go from 3 to cctk_lsh-2 for i, j, AND k, even though
  //   we are only computing the flux in one direction at a time. This is because in the end,
  //   we only need the rhs's from 3 to cctk_lsh-3 for i, j, and k.
#pragma omp parallel for
  for(int k=cctk_nghostzones[2];k<cctk_lsh[2]-(cctk_nghostzones[2]-1);k++) for(int j=cctk_nghostzones[1];j<cctk_lsh[1]-(cctk_nghostzones[1]-1);j++) for(int i=cctk_nghostzones[0];i<cctk_lsh[0]-(cctk_nghostzones[0]-1);i++) {
    int index = CCTK_GFINDEX3D(cctkGH,i,j,k);

    // Set metric and associated variables
    CCTK_REAL CONF_METRIC[NUMVARS_FOR_CONF_METRIC_FACEVALS]; for(int ii=0;ii<NUMVARS_FOR_CONF_METRIC_FACEVALS;ii++) CONF_METRIC[ii] = conf_metric[ii][index];

    CCTK_REAL Ur[MAXNUMVARS]; for(int ii=0;ii<numvars_reconstructed;ii++) Ur[ii] = OUT_PRIMS_R[ii].gf[index];
    CCTK_REAL Ul[MAXNUMVARS]; for(int ii=0;ii<numvars_reconstructed;ii++) Ul[ii] = OUT_PRIMS_L[ii].gf[index];

    // Read the T^{\mu \nu} gridfunction from memory, since computing T^{\mu \nu} is expensive
    CCTK_REAL TUP[4][4]; int counter=0;
    for(int ii=0;ii<4;ii++) for(int jj=ii;jj<4;jj++) { TUP[ii][jj] = TUPMUNU[counter][index]; counter++; }
```

    Appending to ../src/add_fluxes_and_source_terms_to_hydro_rhss.C

## Step 2.b: Computing face values \[Back to [top](#toc)\]
$$\label{facevals}$$

To begin computing the facevalues, we start by setting up the indices $i-2$, $i-1$, $i+1$, and $i+2$, where $i$ indicates the flux direction. Then the metric quantities are evaluated at these points. Finally, using the macro COMPUTE_FCVAL described [above](#add_fluxes_and_source_terms_to_hydro_rhss__c), we determine the face values of the metric quantities at the points $i-1$, $i$, and $i+1$.

```python
%%writefile -a $outdir/add_fluxes_and_source_terms_to_hydro_rhss.C

    // Next set metric on the faces, applying a 3rd-order lopsided stencil.
    int indexm2 = CCTK_GFINDEX3D(cctkGH,i-2*kronecker_delta[flux_dirn][0],j-2*kronecker_delta[flux_dirn][1],k-2*kronecker_delta[flux_dirn][2]);
    int indexm1 = CCTK_GFINDEX3D(cctkGH,i-  kronecker_delta[flux_dirn][0],j-  kronecker_delta[flux_dirn][1],k-  kronecker_delta[flux_dirn][2]);
    int indexp1 = CCTK_GFINDEX3D(cctkGH,i+  kronecker_delta[flux_dirn][0],j+  kronecker_delta[flux_dirn][1],k+  kronecker_delta[flux_dirn][2]);
    int indexp2 = CCTK_GFINDEX3D(cctkGH,i+2*kronecker_delta[flux_dirn][0],j+2*kronecker_delta[flux_dirn][1],k+2*kronecker_delta[flux_dirn][2]);
    // The "vector" METRIC stores needed metric-related quantities.
    CCTK_REAL CONF_METRICm2[NUMVARS_FOR_CONF_METRIC_FACEVALS]; for(int ii=0;ii<NUMVARS_FOR_CONF_METRIC_FACEVALS;ii++) CONF_METRICm2[ii] = conf_metric[ii][indexm2];
    CCTK_REAL CONF_METRICm1[NUMVARS_FOR_CONF_METRIC_FACEVALS]; for(int ii=0;ii<NUMVARS_FOR_CONF_METRIC_FACEVALS;ii++) CONF_METRICm1[ii] = conf_metric[ii][indexm1];
    CCTK_REAL CONF_METRICp1[NUMVARS_FOR_CONF_METRIC_FACEVALS]; for(int ii=0;ii<NUMVARS_FOR_CONF_METRIC_FACEVALS;ii++) CONF_METRICp1[ii] = conf_metric[ii][indexp1];
    CCTK_REAL CONF_METRICp2[NUMVARS_FOR_CONF_METRIC_FACEVALS]; for(int ii=0;ii<NUMVARS_FOR_CONF_METRIC_FACEVALS;ii++) CONF_METRICp2[ii] = conf_metric[ii][indexp2];

    // Next compute the metric values at the {i,j,k} +/- 1/2 faces (i.e., the "face values" of the metric)
    CCTK_REAL FACEVAL[NUMVARS_FOR_CONF_METRIC_FACEVALS],FACEVALp1[NUMVARS_FOR_CONF_METRIC_FACEVALS];
    for(int w=0;w<NUMVARS_FOR_CONF_METRIC_FACEVALS;w++) FACEVAL[w]   = COMPUTE_FCVAL(CONF_METRICm2[w],CONF_METRICm1[w],CONF_METRIC[w],CONF_METRICp1[w]);
    for(int w=0;w<NUMVARS_FOR_CONF_METRIC_FACEVALS;w++) FACEVALp1[w] = COMPUTE_FCVAL(CONF_METRICm1[w],CONF_METRIC[w],CONF_METRICp1[w],CONF_METRICp2[w]);
    // The original IllinoisGRMHD interpolates phi to faces, and then computes directly psi2 & psi4 based on that.
    //   We do the same here to ensure roundoff-level agreement with the original IllinoisGRMHD.
    CCTK_REAL FACEVAL_PSI2_and_PSI4[2];
    CCTK_REAL FACEVALp1_PSI2_and_PSI4[2];
    const int FVPSI2 = 0;
    const int FVPSI4 = 1;
    FACEVAL_PSI2_and_PSI4[FVPSI2] = exp(2.0*FACEVAL[CM_PHI]);
    FACEVAL_PSI2_and_PSI4[FVPSI4] = FACEVAL_PSI2_and_PSI4[FVPSI2]*FACEVAL_PSI2_and_PSI4[FVPSI2];
    FACEVALp1_PSI2_and_PSI4[FVPSI2] = exp(2.0*FACEVALp1[CM_PHI]);
    FACEVALp1_PSI2_and_PSI4[FVPSI4] = FACEVALp1_PSI2_and_PSI4[FVPSI2]*FACEVALp1_PSI2_and_PSI4[FVPSI2];
```

    Appending to ../src/add_fluxes_and_source_terms_to_hydro_rhss.C

## Step 2.c: Computing MHD flux terms \[Back to [top](#toc)\]
$$\label{mhdflux}$$

We now move on to make use of the NRPy+ headers we have generated [above](mhdflux_terms) to compute the MHD flux terms.

```python
%%writefile -a $outdir/add_fluxes_and_source_terms_to_hydro_rhss.C

    //-----------------------------------------------------------------------------
    // Next compute fluxes for \tilde{S}_i, tau, and rho_*

    if(i==14 && j==14 && k==14) {
      printf("dddd0 %d ",flux_dirn);
      for(int ii=0;ii<numvars_reconstructed;ii++) {
        printf("%d %e %e || ",ii,Ur[ii],Ul[ii]);
      }
      printf("\n");

      printf("gggg0 %d ",flux_dirn);
      for(int ii=0;ii<NUMVARS_FOR_CONF_METRIC_FACEVALS;ii++) {
        printf("%e ",CONF_METRIC[ii]);
      }
      printf("\n");

      printf("ffff0 %d ",flux_dirn);
      for(int ii=0;ii<NUMVARS_FOR_CONF_METRIC_FACEVALS;ii++) {
        printf("%e ",FACEVAL[ii]);
      }
      printf("\n");

      printf("cmaxmin %e %e\n",cmax[index],cmin[index]);
    }

    // We removed mhdflux() and replaced it by the NRPy+ generated code below
    // mhdflux(i,j,k,flux_dirn,Ul  ,Ur  ,FACEVAL ,FACEVAL_PSI2_and_PSI4, eos, cmax[index],cmin[index],
    //         rho_star_flux[index],tau_flux[index],st_x_flux[index],st_y_flux[index],st_z_flux[index]);

    // First compute P_{cold}, \epsilon_{cold}, dP_{cold}/drho, \epsilon_{th}, h, and \Gamma_{cold},
    // for right and left faces:
    CCTK_REAL P_coldr,eps_coldr,dPcold_drhor=0,eps_thr=0,h_r=0,Gamma_coldr;
    compute_P_cold__eps_cold__dPcold_drho__eps_th__h__Gamma_cold(Ur,eos,Gamma_th,P_coldr,eps_coldr,dPcold_drhor,eps_thr,h_r,Gamma_coldr);
    CCTK_REAL P_coldl,eps_coldl,dPcold_drhol=0,eps_thl=0,h_l=0,Gamma_coldl;
    compute_P_cold__eps_cold__dPcold_drho__eps_th__h__Gamma_cold(Ul,eos,Gamma_th,P_coldl,eps_coldl,dPcold_drhol,eps_thl,h_l,Gamma_coldl);

    // Declare variables needed by the NRPy+ generated code
    CCTK_REAL smallb4U0_R,smallb4U1_R,smallb4U2_R,smallb4U3_R;
    CCTK_REAL smallb4U0_L,smallb4U1_L,smallb4U2_L,smallb4U3_L;
    const CCTK_REAL sqrt4pi = sqrt(4.0*M_PI);
#include "NRPy_generated_headers/smallb4R_and_L.h"

    switch(flux_dirn)
      {
      case 1:
        {
#include "NRPy_generated_headers/flux_dirn_x.h"
          break;
        }
      case 2:
        {
#include "NRPy_generated_headers/flux_dirn_y.h"
          break;
        }
      case 3:
        {
#include "NRPy_generated_headers/flux_dirn_z.h"
          break;
        }
      }

    /*
    if(i==14 && j==14 && k==14) {
      printf("dddd1 %e %e %e %e %e %e %e\n",cmax[index],cmin[index],
             rho_star_flux[index],tau_flux[index],st_x_flux[index],st_y_flux[index],st_z_flux[index]);
    }
    */
```

    Appending to ../src/add_fluxes_and_source_terms_to_hydro_rhss.C

## Step 2.d: Computing $\partial_{i}g_{\mu\nu}$ and $\partial_{i}\alpha$ \[Back to [top](#toc)\]
$$\label{partial_i_gmunu}$$

Having the flux terms computed, we move on the the evaluation of the source terms. We start by computing $\partial_{i}g_{\mu\nu}$, which are required for the evaluation of the right-hand side of the evolution equation for $\tilde{S}_{i}$. We compute the derivatives using centered finite differences, namely

\begin{align}
\left[\partial_{x}g_{\mu\nu}\right]_{i+1/2,j,k} &= \frac{\left[g_{\mu\nu}\right]_{i+1,j,k}-\left[g_{\mu\nu}\right]_{i,j,k}}{dx}\ ,\\
\left[\partial_{y}g_{\mu\nu}\right]_{i,j+1/2,k} &= \frac{\left[g_{\mu\nu}\right]_{i,j+1,k}-\left[g_{\mu\nu}\right]_{i,j,k}}{dy}\ ,\\
\left[\partial_{z}g_{\mu\nu}\right]_{i,j,k+1/2} &= \frac{\left[g_{\mu\nu}\right]_{i,j,k+1}-\left[g_{\mu\nu}\right]_{i,j,k}}{dz}\ .
\end{align}

We also evaluate $\partial_{i}\alpha$, which is given analogously by

\begin{align}
\left[\partial_{x}\alpha\right]_{i+1/2,j,k} &= \frac{\alpha_{i+1,j,k}-\alpha_{i,j,k}}{dx}\ ,\\
\left[\partial_{y}\alpha\right]_{i,j+1/2,k} &= \frac{\alpha_{i,j+1,k}-\alpha_{i,j,k}}{dy}\ ,\\
\left[\partial_{z}\alpha\right]_{i,j,k+1/2} &= \frac{\alpha_{i,j,k+1}-\alpha_{i,j,k}}{dz}\ ,
\end{align}

so that we can evaluate right-hand side of the evolution equation for $\tilde{\tau}$.

```python
%%writefile -a $outdir/add_fluxes_and_source_terms_to_hydro_rhss.C

    //-----------------------------------------------------------------------------
    // If we are not in the ghostzones, then add third-order accurate curvature terms to \tilde{S}_i RHS's
    //    Without this if() statement, _rhs variables are in general set to nonzero values in ghostzones, which messes up frozen BC's.
    //    Also, this if() statement should speed up the computation slightly.
    if(k<cctk_lsh[2]-cctk_nghostzones[2] && j<cctk_lsh[1]-cctk_nghostzones[1] && i<cctk_lsh[0]-cctk_nghostzones[0]) {

      CCTK_REAL Psi6 = exp(6.0*CONF_METRIC[CM_PHI]);
      CCTK_REAL half_alpha_sqrtgamma = 0.5*(CONF_METRIC[CM_LAPM1] + 1.0)*Psi6;

      // First compute four metric.
      CCTK_REAL psi4 = FACEVAL_PSI2_and_PSI4[FVPSI4];
      CCTK_REAL g4tt_f,g4tx_f,g4ty_f,g4tz_f,g4xx_f,g4xy_f,g4xz_f,g4yy_f,g4yz_f,g4zz_f;
      COMPUTE_FOURMETRIC(g4tt_f,g4tx_f,g4ty_f,g4tz_f,g4xx_f,g4xy_f,g4xz_f,g4yy_f,g4yz_f,g4zz_f,FACEVAL,psi4);

      CCTK_REAL psi4p1 = FACEVALp1_PSI2_and_PSI4[FVPSI4];
      CCTK_REAL g4tt_fp1,g4tx_fp1,g4ty_fp1,g4tz_fp1,g4xx_fp1,g4xy_fp1,g4xz_fp1,g4yy_fp1,g4yz_fp1,g4zz_fp1;
      COMPUTE_FOURMETRIC(g4tt_fp1,g4tx_fp1,g4ty_fp1,g4tz_fp1,g4xx_fp1,g4xy_fp1,g4xz_fp1,g4yy_fp1,g4yz_fp1,g4zz_fp1,FACEVALp1,psi4p1);

      // Compute \partial_i g_{\mu \nu} at m+1/2
      CCTK_REAL partial_i_gmunu[4][4];
      partial_i_gmunu[0][0] = (g4tt_fp1 - g4tt_f)*dxi[flux_dirn];
      partial_i_gmunu[0][1] = (g4tx_fp1 - g4tx_f)*dxi[flux_dirn];
      partial_i_gmunu[0][2] = (g4ty_fp1 - g4ty_f)*dxi[flux_dirn];
      partial_i_gmunu[0][3] = (g4tz_fp1 - g4tz_f)*dxi[flux_dirn];
      partial_i_gmunu[1][1] = (g4xx_fp1 - g4xx_f)*dxi[flux_dirn];
      partial_i_gmunu[1][2] = (g4xy_fp1 - g4xy_f)*dxi[flux_dirn];
      partial_i_gmunu[1][3] = (g4xz_fp1 - g4xz_f)*dxi[flux_dirn];
      partial_i_gmunu[2][2] = (g4yy_fp1 - g4yy_f)*dxi[flux_dirn];
      partial_i_gmunu[2][3] = (g4yz_fp1 - g4yz_f)*dxi[flux_dirn];
      partial_i_gmunu[3][3] = (g4zz_fp1 - g4zz_f)*dxi[flux_dirn];

      // Needed for tau_rhs computation:
      CCTK_REAL lapse_deriv[4] = { 0,0,0,0 };
      lapse_deriv[flux_dirn] = (FACEVALp1[CM_LAPM1] - FACEVAL[CM_LAPM1])*dxi[flux_dirn];
```

    Appending to ../src/add_fluxes_and_source_terms_to_hydro_rhss.C

## Step 2.e: The $\tilde{\tau}$ and $\tilde{S}_{i}$ source terms \[Back to [top](#toc)\]
$$\label{tau_stilde_source_terms}$$

We now compute the $\tilde{S}_{i}$ source terms according to equation (43) in [](http://arxiv.org/pdf/astro-ph/0503420.pdf):

$$
s_{\tilde{S}_{i}} = \frac{1}{2}\alpha\sqrt{\gamma}T^{\mu\nu}\partial_{i}g_{\mu\nu}\ .
$$

We also compute the last piece that was missing from the $\tilde{\tau}$ source term, namely

$$
s_{\tilde{\tau}} = \cdots - \left(T^{00}\beta^{i} + T^{0i}\right)\partial_{i}\alpha\ .
$$

```python
%%writefile -a $outdir/add_fluxes_and_source_terms_to_hydro_rhss.C

      // Needed for st_i_rhs computation:
      CCTK_REAL st_i_curvature_terms[4] = { 0,0,0,0 };
      // add \frac{1}{2} \alpha \sqrt{\gamma} T^{\mu \nu} \partial_i g_{\mu \nu} . Note that i is given by the flux direction.
      //   (Source term of Eq 43 in http://arxiv.org/pdf/astro-ph/0503420.pdf)
      st_i_curvature_terms[flux_dirn] = half_alpha_sqrtgamma * (TUP[0][0]*partial_i_gmunu[0][0]      +
                                                                TUP[1][1]*partial_i_gmunu[1][1]      +
                                                                TUP[2][2]*partial_i_gmunu[2][2]      +
                                                                TUP[3][3]*partial_i_gmunu[3][3]      +
                                                                2.0*(TUP[0][1]*partial_i_gmunu[0][1] +
                                                                TUP[0][2]*partial_i_gmunu[0][2]      +
                                                                TUP[0][3]*partial_i_gmunu[0][3]      +
                                                                TUP[1][2]*partial_i_gmunu[1][2]      +
                                                                TUP[1][3]*partial_i_gmunu[1][3]      +
                                                                TUP[2][3]*partial_i_gmunu[2][3]) );

      // add - ( T^{00} \beta^i + T^{0i} ) \partial_i \alpha.
      //   (Last part of Eq. 39 source term in http://arxiv.org/pdf/astro-ph/0503420.pdf)
      CCTK_REAL alpha_sqrtgamma = 2.0*half_alpha_sqrtgamma;
      tau_rhs[index]  += alpha_sqrtgamma*(-(TUP[0][0]*CONF_METRIC[CM_SHIFTX+(flux_dirn-1)] + TUP[0][flux_dirn])*lapse_deriv[flux_dirn]);

      // Eq 43 in http://arxiv.org/pdf/astro-ph/0503420.pdf:
      // \partial_t \tilde{S}_i = - \partial_i (\alpha \sqrt{\gamma} T^j_i) + \frac{1}{2}\alpha \sqrt{\gamma} T^{\mu \nu}g_{\mu \nu,i}
      // Notice that st_i_curvature_terms[N]=0 for N!=flux_dirn.
      st_x_rhs[index] += st_i_curvature_terms[1];
      st_y_rhs[index] += st_i_curvature_terms[2];
      st_z_rhs[index] += st_i_curvature_terms[3];
    }

  }
```

    Appending to ../src/add_fluxes_and_source_terms_to_hydro_rhss.C

## Step 2.f: Adding fluxes to the hydro RHSs \[Back to [top](#toc)\]
$$\label{add_flux_terms}$$

Finally, we add the flux terms

$$
\left(
\partial_{i}F_{\rho_{\star}},
\partial_{i}F_{\tilde{\tau}},
\partial_{j}F^{j}_{\tilde{S_{i}}}
\right)\ ,
$$

to the hydro RHSs. We again use centered finite differences. Notice that because these terms appear with a negative sign in the RHSs, we flip the order of the terms in the finite difference approximation below, e.g.

$$
-\partial_{x}\left[F_{\rho_{\star}}\right]_{i+1/2,j,k} = \frac{\left[F_{\rho_{\star}}\right]_{i,j,k}-\left[F_{\rho_{\star}}\right]_{i+1,j,k}}{dx}\ .
$$

```python
%%writefile -a $outdir/add_fluxes_and_source_terms_to_hydro_rhss.C

  // Notice in the loop below that we go from 3 to cctk_lsh-3 for i, j, AND k, even though
  //   we are only computing the flux in one direction. This is because in the end,
  //   we only need the rhs's from 3 to cctk_lsh-3 for i, j, and k.
#pragma omp parallel for
  for(int k=cctk_nghostzones[2];k<cctk_lsh[2]-cctk_nghostzones[2];k++) for(int j=cctk_nghostzones[1];j<cctk_lsh[1]-cctk_nghostzones[1];j++) for(int i=cctk_nghostzones[0];i<cctk_lsh[0]-cctk_nghostzones[0];i++) {
    int index = CCTK_GFINDEX3D(cctkGH,i,j,k);
    int indexp1 = CCTK_GFINDEX3D(cctkGH,i+kronecker_delta[flux_dirn][0],j+kronecker_delta[flux_dirn][1],k+kronecker_delta[flux_dirn][2]);

    rho_star_rhs[index] += (rho_star_flux[index] - rho_star_flux[indexp1]) * dxi[flux_dirn];
    tau_rhs[index]      += (tau_flux[index]      - tau_flux[indexp1]     ) * dxi[flux_dirn];
    st_x_rhs[index]     += (st_x_flux[index]     - st_x_flux[indexp1]    ) * dxi[flux_dirn];
    st_y_rhs[index]     += (st_y_flux[index]     - st_y_flux[indexp1]    ) * dxi[flux_dirn];
    st_z_rhs[index]     += (st_z_flux[index]     - st_z_flux[indexp1]    ) * dxi[flux_dirn];
  }
}
```

    Appending to ../src/add_fluxes_and_source_terms_to_hydro_rhss.C

# Step 3: Code validation \[Back to [top](#toc)\]
$$\label{code_validation}$$

First we download the original `IllinoisGRMHD` source code and then compare it to the source code generated by this tutorial notebook.

```python
# # Verify if the code generated by this tutorial module
# # matches the original IllinoisGRMHD source code

# # First download the original IllinoisGRMHD source code
# import urllib
# from os import path

# original_IGM_file_url  = "https://bitbucket.org/zach_etienne/wvuthorns/raw/5611b2f0b17135538c9d9d17c7da062abe0401b6/IllinoisGRMHD/src/add_fluxes_and_source_terms_to_hydro_rhss.C"
# original_IGM_file_name = "add_fluxes_and_source_terms_to_hydro_rhss-original.C"
# original_IGM_file_path = os.path.join(IGM_src_dir_path,original_IGM_file_name)

# # Then download the original IllinoisGRMHD source code
# # We try it here in a couple of ways in an attempt to keep
# # the code more portable
# try:
#     original_IGM_file_code = urllib.request.urlopen(original_IGM_file_url).read().decode("utf-8")
#     # Write down the file the original IllinoisGRMHD source code
#     with open(original_IGM_file_path,"w") as file:
#         file.write(original_IGM_file_code)
# except:
#     try:
#         original_IGM_file_code = urllib.urlopen(original_IGM_file_url).read().decode("utf-8")
#         # Write down the file the original IllinoisGRMHD source code
#         with open(original_IGM_file_path,"w") as file:
#             file.write(original_IGM_file_code)
#     except:
#         # If all else fails, hope wget does the job
#         !wget -O $original_IGM_file_path $original_IGM_file_url

# # Perform validation
# Validation__add_fluxes_and_source_terms_to_hydro_rhss__C  = !diff $original_IGM_file_path $outfile_path__add_fluxes_and_source_terms_to_hydro_rhss__C

# if Validation__add_fluxes_and_source_terms_to_hydro_rhss__C == []:
#     # If the validation passes, we do not need to store the original IGM source code file
#     !rm $original_IGM_file_path
#     print("Validation test for add_fluxes_and_source_terms_to_hydro_rhss.C: PASSED!")
# else:
#     # If the validation fails, we keep the original IGM source code file
#     print("Validation test for add_fluxes_and_source_terms_to_hydro_rhss.C: FAILED!")
#     # We also print out the difference between the code generated
#     # in this tutorial module and the original IGM source code
#     print("Diff:")
#     for diff_line in Validation__add_fluxes_and_source_terms_to_hydro_rhss__C:
#         print(diff_line)
```

# Step 4: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-IllinoisGRMHD__add_fluxes_and_source_terms_to_hydro_rhss.pdf](Tutorial-IllinoisGRMHD__add_fluxes_and_source_terms_to_hydro_rhss.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means).

```python
latex_nrpy_style_path = os.path.join(nrpy_dir_path,"latex_nrpy_style.tplx")
#!jupyter nbconvert --to latex --template $latex_nrpy_style_path --log-level='WARN' Tutorial-IllinoisGRMHD__add_fluxes_and_source_terms_to_hydro_rhss.ipynb
#!pdflatex -interaction=batchmode Tutorial-IllinoisGRMHD__add_fluxes_and_source_terms_to_hydro_rhss.tex
#!pdflatex -interaction=batchmode Tutorial-IllinoisGRMHD__add_fluxes_and_source_terms_to_hydro_rhss.tex
#!pdflatex -interaction=batchmode Tutorial-IllinoisGRMHD__add_fluxes_and_source_terms_to_hydro_rhss.tex
!rm -f Tut*.out Tut*.aux Tut*.log
```