# Solving a transformed system of first order ordinary differential equations
In this notebook we explore how we can use different convenient subclasses of ``SymbolicSys`` to reformulate a problem (for other stability/accuracy/performance characteristics)

```python
from __future__ import print_function, division, absolute_import
import numpy as np
import matplotlib.pyplot as plt
import sympy as sp
from pyodesys import OdeSys
from pyodesys.symbolic import SymbolicSys, symmetricsys
sp.init_printing()
%matplotlib inline
print(sp.__version__)
```

We will consider a chain of three coupled decays:

```python
f = lambda t, x, k: [-k[0]*x[0], k[0]*x[0] - k[1]*x[1], k[1]*x[1] - k[2]*x[2], k[2]*x[2] - k[3]*x[3]]
k = [7, 3, 2, 0]
y0 = [1, 0, 0, 0]
tend = 1.0
```

The above equation has an analytic solution, evaluated at t = 1 it is:

```python
ref = [9.11881965554516244e-04, 8.55315762040415040e-02, 3.07983556726319885e-01, 6.05572985104084083e-01]
def rmsd(x):
    diff = np.array(ref[4-len(x):]) - x
    return np.sum(diff**2)**0.5
```

We can integrate this using e.g. lsoda from scipy:

```python
odesys = OdeSys(f)
res = odesys.integrate(np.linspace(0, tend), y0, params=k)
res.plot(names='abcd')
plt.legend()
{k: v for k, v in res.info.items() if not k.startswith('internal')}, rmsd(res.yout[-1, :])
```

For a stiff system (requiring an implicit method) we would have needed to define the jacobian too, then using symbolic manipulation can be very useful (less error prone and less user code needed to be written):

```python
odesys = SymbolicSys.from_callback(f, 4, 4, names='abcd')
odesys.exprs, odesys.get_jac(), odesys.get_dfdx()
```

Let's define a convienience function for plotting both with linear and logartithmic scale:

```python
def integrate_and_plot(odesys, integrator, tout, y0, k, interpolate=False, stiffness=False, **kwargs):
    plt.figure(figsize=(14,5))
    xout, yout, info = odesys.integrate(tout, y0, k, integrator=integrator, **kwargs)
    plt.subplot(1, 3 if stiffness else 2, 1)
    odesys.plot_result(interpolate=interpolate)
    plt.legend(loc='best')

    plt.subplot(1, 3 if stiffness else 2, 2)
    plt.gca().set_xscale('log')
    plt.gca().set_yscale('log')
    odesys.plot_result(interpolate=interpolate)
    plt.legend(loc='best')

    if stiffness:
        ratios = odesys.stiffness()
        plt.subplot(1, 3, 3)
        plt.yscale('linear')
        plt.plot(odesys._internal[0], ratios)
    info.pop('internal_xout')
    info.pop('internal_yout')
    return len(xout), info, rmsd(yout[-1, :])
```

Let's use the ``vode`` integrator this time and an implicit algorithm:

```python
integrate_and_plot(odesys, 'scipy', np.linspace(0, tend), y0, k, first_step=1e-14, name='vode', method='bdf')
```

We see that the jacobian was evaluated twice. The final error is slightly higher (which might be expected since the system is not particularly stiff)

SymbolicSys has provided us with callbacks if we manually want to evaluate f or its jacobian:

```python
print(odesys.f_cb(0, y0, k))
print()
print(odesys.j_cb(0, y0, k))
```

We can use the [cvode](https://computation.llnl.gov/casc/sundials/description/description.html#descr_cvode) integrator (through the use of [pycvodes](https://pypi.python.org/pypi/pycvodes))

```python
integrate_and_plot(odesys, 'cvode', tend, y0, k, atol=1e-8, rtol=1e-8, method='bdf')
```

Since we are using symbolic manipulation it is very easy to perform a variable transformation.
We will look at how this system behaves in logarithmic space (we need $y>0$ so we'll add a $\delta$ much smaller than our original linear absolute tolerance)

First we need to define our transformation, we use the helper function ``symmetricsys`` for this:

```python
y0aug = [1, 1e-20, 1e-20, 1e-20]
logexp = sp.log, sp.exp
LogLogSys = symmetricsys(
    logexp, logexp, exprs_process_cb=lambda exprs: [
    sp.powsimp(expr.expand(), force=True) for expr in exprs])
```

Alright, so our newly defined `LogLogSys` class can now take the same input and give back transformed expressions:

```python
tsys = LogLogSys.from_callback(f, 4, 4, names=True)
tsys.exprs, tsys.get_jac(), tsys.get_dfdx()
```

Now let us integrate the transformed system:

```python
integrate_and_plot(tsys, 'cvode', [1e-12, tend], y0aug, k, atol=1e-7, rtol=1e-7, first_step=1e-4, nsteps=5000)
```

```python
integrate_and_plot(tsys, 'odeint', [1e-12, tend], y0aug, k, atol=1e-8, rtol=1e-8, first_step=1e-4, nsteps=50000)
```

Substantially more work was needed to solve this transformed system, also the accuracy suffered. So this transformation was not of any help for this particular problem, choice of initial conditions and length of integration. There may be situations where it is useful though.

## Stiffness

```python
def f2(t, x, p):
    y0 = p[0]*sp.exp(-p[1]*t)
    return [p[1]*y0 - p[2]*x[0]] + [
        p[i+2]*x[i] - p[i+3]*x[i+1] for i in range(len(x) - 1)
    ]
tsys2 = LogLogSys.from_callback(f2, 3, 5, names=True)
tsys2.exprs, tsys2.get_jac()
```

```python
integrate_and_plot(tsys2, 'cvode', [1e-12, tend], y0aug[1:], [y0aug[0], 7, 3, 2, 1],
                   atol=1e-7, rtol=1e-7, first_step=1e-4, stiffness=True, nsteps=5000)
```

```python
integrate_and_plot(odesys, 'cvode', [1e-12, tend], y0aug, [7, 3, 2, 1], atol=1e-7, rtol=1e-7, first_step=1e-4, stiffness=True)
```

# Using PartiallySolvedSys
Sometimes we can solve a system paritally by integrating some of the dependent variables. The ODE system then needs to be reformulated in terms of the new anayltic function. ``PartiallySolvedSystem`` helps us with this task:

```python
from pyodesys.symbolic import PartiallySolvedSystem
psys = PartiallySolvedSystem(odesys, lambda x0, y0, p0: {odesys.dep[0]: y0[0]*sp.exp(-p0[0]*(odesys.indep-x0))})
psys.exprs, psys.get_jac()
```

```python
integrate_and_plot(psys, 'cvode', [1e-12, tend], y0aug, [7, 3, 2, 1], atol=1e-7, rtol=1e-7, first_step=1e-4, stiffness=True)
```

## Using fixed RK4 stepper
pyodesys provides an example class of an integrator (``RK4_example_integrator``) in ``pyodesys.integrators``. It can be used as a model integrator when designing custom steppers.

```python
from pyodesys.integrators import RK4_example_integrator
```

```python
integrate_and_plot(odesys, RK4_example_integrator, [1e-12, tend], y0aug, k, atol=1e-8, rtol=1e-8, first_step=1e-1)
```

```python
integrate_and_plot(psys, RK4_example_integrator, [1e-12, tend], y0aug, [7, 3, 2, 1], atol=1e-8, rtol=1e-8, first_step=1e-1)
```

```python

```


# North American Einstein Toolkit School 2021: NRPy+ Tutorial
### Authors: Leo Werneck, Terrence Pierre Jacques, Patrick Nelson, Zach Etienne, & Thiago Assumpção

## You can access this tutorial by going to https://tinyurl.com/ETKSchoolNRPytutorial

# Table of Contents
$$\label{toc}$$

This tutorial notebook is organized as follows:

0. [Step 0](#introduction): **Introduction**
    1. [Step 0.a](#maxwell_equations): Maxwell's equations in vacuum, flat space, and Cartesian coordinates
    1. [Step 0.b](#potential_formulation_maxwell_equations): Potential formulation of Maxwell's equations
    1. [Step 0.c](#hyperbolicity_maxwell_equations): Improving the hyperbolicity of Maxwell's equations
    1. [Step 0.d](#systems_I_and_II): Summary
1. [Step 1](#implementation): **Part I - Implementing Maxwell's equations using NRPy+**
    1. [Step 1.a](#initializenrpy): Initialize core Python/NRPy+ modules
    1. [Step 1.b](#gridfunction_declaration): Gridfunction declaration
    1. [Step 1.c](#finite_differences): Finite difference derivatives
    1. [Step 1.d](#system_I_implementation): Implementation of system I
        1. [Step 1.d.i](#system_I_evolution_eqs): *System I evolution equations*
        1. [Step 1.d.ii](#system_I_constraint_eqs): *System I constraint equation*
    1. [Step 1.e](#system_II_implementation): Implementation of system II
        1. [Step 1.e.i](#system_II_evolution_eqs): *System II evolution equations*
        1. [Step 1.e.ii](#system_II_constraint_eqs): *System II constraint equations*
1. [Step 2](#thorn_writing): **Part II - Einstein Toolkit thorn writing**
    1. [Step 2.a](#generating_c_code_kernels): Generating C code kernels for Maxwell's equations
        1. [Step 2.a.i](#system_I_c_code_generation): *System I C code generation*
        1. [Step 2.a.ii](#system_II_c_code_generation): *System II C code generation*
        1. [Step 2.a.iii](#zero_rhss): *C code to initialize right-hand sides to zero*
    1. [Step 2.b](#rhs_driver): The right-hand side driver function
    1. [Step 2.c](#constraints_driver): The constraints driver function
    1. [Step 2.d](#mol_registration): Registering gridfunctions for the Method of Lines
    1. [Step 2.e](#gridfunction_symmetries): Set gridfunction symmetries
    1. [Step 2.f](#boundary_conditions): Boundary condition configuration
    1. [Step 2.g](#nrpy_banner): `NRPy+` banner
    1. [Step 2.h](#ccl_files): Thorn configuration files
        1. [Step 2.h.i](#param_ccl): *Generating the `param.ccl` file*
        1. [Step 2.h.ii](#interface_ccl): *Generating the `interface.ccl` file*
        1. [Step 2.h.iii](#schedule_ccl): *Generating the `schedule.ccl` file*
        1. [Step 2.h.iv](#configuration_ccl): *Generating the `configuration.ccl` file*
        1. [Step 2.h.v](#make_code_defn): *Generating the `make.code.defn` file*
1. [Step 3](#results): **Part III - Results**
1. [Step 4](#latex_pdf_output): **Output this notebook to $\LaTeX$-formatted PDF file**

# Step 0: Introduction \[Back to [top](#toc)\]
$$\label{introduction}$$

In this tutorial we will provide a hands-on introduction on how to use the [NRPy+ infrastructure](http://nrpyplus.net) to write thorns for the [Einstein Toolkit (ETK)](https://einsteintoolkit.org). Although we will give an overview of NRPy+ and provide a high level description of some of its modules, we do not aim to thoroughly describe every aspect of the infrastructure. For more details on NRPy+'s inner workings, we refer the reader to:

* [NRPy+'s webpage](http://nrpyplus.net)
* [NRPy+ tutorial from the 2020 ETK workshop by Zach Etienne](https://www.youtube.com/watch?v=TIPiW5-mPOM)

The material covered by this tutorial notebook is also based on many of the pedagogical `Jupyter` notebooks that form `NRPy+`'s documentation. We mention here the ones that are most useful as **additional reading material**:

* [Terrence's presentation on `MaxwellVacuum` thorns](https://lsu.app.box.com/s/9hrfzrh8dkmvpba4uqatnthjwafwahnv)
* [NRPy+ indexed expressions tutorial notebook](Tutorial-Indexed_Expressions.ipynb)
* [NRPy+ finite differences tutorial notebook](Tutorial-Finite_Difference_Derivatives.ipynb)
* [How NRPy+ computs finite differences coefficients tutorial notebook](Tutorial-How_NRPy_Computes_Finite_Difference_Coeffs.ipynb)
* [NRPy+ MaxwellVacuum formulation in Cartesian coordinates tutorial notebook](Tutorial-VacuumMaxwell_formulation_Cartesian.ipynb)
* [NRPy+ MaxwellVacuum formulation in Curvilinear coordinates tutorial notebook](Tutorial-VacuumMaxwell_formulation_Curvilinear.ipynb)
* [NRPy+ MaxwellVacuum right-hand sides tutorial notebook](Tutorial-VacuumMaxwell_Cartesian_RHSs.ipynb)
* [NRPy+ MaxwellVacuum ETK thorn tutorial notebook](Tutorial-ETK_thorn-MaxwellVacuum.ipynb)

## Step 0.a: Maxwell's equations in vacuum, flat space, and Cartesian coordinates \[Back to [top](#toc)\]
$$\label{maxwell_equations}$$

In this tutorial we will generate an ETK thorn to solve [Maxwell's equations](https://en.wikipedia.org/wiki/Maxwell%27s_equations) in flat space using Cartesian coordinates. In [Gaussian](https://en.wikipedia.org/wiki/Gaussian_units) and $c = 1$ units, the system of equations we are interested in is

$$
\begin{align}
\vec{\nabla} \cdot \vec{E} &= 0, \\
\vec{\nabla} \cdot \vec{B} &= 0, \\
\frac{\partial \vec{E}}{\partial t} &= \vec{\nabla} \times \vec{B}, \\
\frac{\partial \vec{B}}{\partial t} &= -\vec{\nabla} \times \vec{E},
\end{align}
$$

where $\vec{E}$ is the electric field, $\vec{B}$ is the magnetic field, and $\vec{\nabla}\cdot\vec{V}$ and $\vec{\nabla}\times\vec{V}$ are the divergence and the curl of the vector $\vec{V}$, respectively.

The last two equations involve time derivatives of $\vec{E}$ and $\vec{B}$ and therefore are referred to as *evolution* equations. The first two equations must be satisfied for all times, and are referred to as *constraint* equations.

## Step 0.b: Potential formulation of Maxwell's equations \[Back to [top](#toc)\]
$$\label{potential_formulation_maxwell_equations}$$

A formulation of Maxwell's equations that is particularly useful for numerical integration can be found by introducing auxiliary variables. We start by introducing a new vector quantity, $\vec{A}$, known as the *magnetic* or *vector* potential, such that

$$
\vec{B} = \vec{\nabla}\times\vec{A}.
$$

Note that the "no-magnetic monopole" constraint is automatically satisfied:

$$
\vec{\nabla} \cdot \vec{B} = \vec{\nabla} \cdot \bigl(\vec{\nabla}\times\vec{A}\bigr) = 0.
$$

[Ampère's law](https://en.wikipedia.org/wiki/Ampère%27s_circuital_law), which is the evolution equation for $\vec{B}$, can then be written as

$$
\frac{\partial}{\partial t}\bigl(\vec{\nabla}\times\vec{A}\bigr) = -\vec{\nabla} \times \vec{E} \implies \vec{\nabla}\times\left(\frac{\partial\vec{A}}{\partial t} + \vec{E}\right) = 0 \implies \frac{\partial\vec{A}}{\partial t} + \vec{E} = -\vec{\nabla}\Phi,
$$

where $\Phi$ is an arbitrary scalar function known as the *electric* (or *scalar*) potential. Finally, [Faraday's law](https://en.wikipedia.org/wiki/Faraday%27s_law_of_induction), the evolution equation for $\vec{E}$, can be written as

$$
\frac{\partial\vec{E}}{\partial t} = \vec{\nabla}\times\bigl(\vec{\nabla}\times\vec{A}\bigr) = -\nabla^{2}\vec{A} + \vec{\nabla}\bigl(\vec{\nabla}\cdot\vec{A}\bigr).
$$

Thus Maxwell's equations now read

$$
\begin{align}
\vec{\nabla} \cdot \vec{E} &= 0, \\
\frac{\partial\vec{E}}{\partial t} &= -\nabla^{2}\vec{A} + \vec{\nabla}\bigl(\vec{\nabla}\cdot\vec{A}\bigr),\\
\frac{\partial\vec{A}}{\partial t} &= -\vec{E} - \vec{\nabla}\Phi.
\end{align}
$$

Note that we now end up with 7 dynamical fields, namely $\left(\Phi,\vec{A},\vec{E}\right)$, but only 6 evolution equations. We can remedy this by adopting a particular gauge. We will choose here the [Lorenz gauge](https://en.wikipedia.org/wiki/Lorenz_gauge_condition), which reads

$$
\frac{\partial\Phi}{\partial t} = -\vec{\nabla}\cdot\vec{A}.
$$

Thus we arrive at the evolution equations

$$
\boxed{
\begin{align}
\frac{\partial\vec{E}}{\partial t} &= -\nabla^{2}\vec{A} + \vec{\nabla}\bigl(\vec{\nabla}\cdot\vec{A}\bigr)\\
\frac{\partial\vec{A}}{\partial t} &= -\vec{E} - \vec{\nabla}\Phi\\
\frac{\partial\Phi}{\partial t} &= -\vec{\nabla}\cdot\vec{A}
\end{align}
}\ ,
$$

plus the constraints

$$
\boxed{ \vec{\mathcal{C}} \equiv \vec{\nabla} \cdot \vec{E} = 0}\ ,
$$

which must be satisfied at all times.

## Step 0.c: Improving the hyperbolicity of Maxwell's equations \[Back to [top](#toc)\]
$$\label{hyperbolicity_maxwell_equations}$$

If we take a time derivative of the evolution equation for the vector potential and plug in the evolution equation for the electric field and the scalar potential we find

$$
\frac{\partial^{2}\vec{A}}{\partial t^{2}} = -\frac{\partial\vec{E}}{\partial t} - \vec{\nabla}\left(\frac{\partial\Phi}{\partial t}\right) = \nabla^{2}\vec{A} - \vec{\nabla}\bigl(\vec{\nabla}\cdot\vec{A}\bigr) - \vec{\nabla}\left(\frac{\partial\Phi}{\partial t}\right).
$$

Notice that this is almost a wave equation for the vector potential $\vec{A}$, but we have a mixed derivative term on the right-hand side that spoils this behaviour. We can thus introduce a new auxiliary variable defined as

$$
\Gamma \equiv \vec{\nabla}\cdot\vec{A},
$$

so that we have

$$
\frac{\partial^{2}\vec{A}}{\partial t^{2}} = \nabla^{2}\vec{A} - \vec{\nabla}\Gamma - \vec{\nabla}\left(\frac{\partial\Phi}{\partial t}\right).
$$

The resulting system of equations now becomes

$$
\boxed{
\begin{align}
\frac{\partial\vec{E}}{\partial t} &= -\nabla^{2}\vec{A} + \vec{\nabla}\Gamma\\
\frac{\partial\vec{A}}{\partial t} &= -\vec{E} - \vec{\nabla}\Phi\\
\frac{\partial\Phi}{\partial t} &= -\Gamma\\
\frac{\partial\Gamma}{\partial t} &= -\nabla^{2}\Phi
\end{align}
}\ ,
$$

while the constraints that must satisfied are

$$
\boxed{
\begin{align}
\mathcal{C} &\equiv \vec{\nabla} \cdot \vec{E} = 0\\
\mathcal{G} &\equiv \Gamma - \vec{\nabla}\cdot\vec{A} = 0
\end{align}
}\ .
$$

## Step 0.d: Summary \[Back to [top](#toc)\]
$$\label{systems_I_and_II}$$

Switching to index notation, we write

$$
\frac{\partial}{\partial t} \equiv \partial_{t}\ ;\ \frac{\partial}{\partial x^{i}} \equiv \partial_{i},
$$

where $\vec{x} \equiv x^{i} = (x,y,z)$.

We are thus interested in solving Maxwell's equations in vacuum (i.e. without source terms), in flat space, and in Cartesian coordinates, and we will do so using:

$$
\text{System I:}\
\boxed{
\begin{align}
\color{blue}{\partial_{t}E^{i}} &\, \color{blue}{= -\partial^{j}\partial_{j}A^{i} + \partial^{i}\partial_{j}A^{j}}\\
\color{blue}{\partial_{t}A^{i}} &\, \color{blue}{= -E^{i} - \partial^{i}\Phi}\\
\color{blue}{\partial_{t}\Phi}  &\, \color{blue}{= -\partial_{i}A^{i}}\\
\color{red}{\partial_{i}E^{i}} &\, \color{red}{= 0}
\end{align}
}
\ ,
$$

and

$$
\text{System II:}\
\boxed{
\begin{align}
\color{blue}{\partial_{t}E^{i}}  &\, \color{blue}{= -\partial^{j}\partial_{j}A^{i} + \partial^{i}\Gamma}\\
\color{blue}{\partial_{t}A^{i}}  &\, \color{blue}{= -E^{i} - \partial^{i}\Phi}\\
\color{blue}{\partial_{t}\Phi}   &\, \color{blue}{= -\Gamma}\\
\color{blue}{\partial_{t}\Gamma} &\, \color{blue}{= -\partial^{i}\partial_{i}\Phi}\\
\color{red}{\partial_{i}E^{i}}   &\, \color{red}{= 0}\\
\color{red}{\Gamma}              &\, \color{red}{= \partial_{i}A^{i}}
\end{align}
}\ .
$$

In the colorcode above, $\color{blue}{\text{blue}}$ equations are the $\color{blue}{\text{evolution}}$ equations, while $\color{red}{\text{red}}$ equations are the $\color{red}{\text{constraint}}$ equations.

# Step 1: Part I - Implementing Maxwell's equations using NRPy+ \[Back to [top](#toc)\]
$$\label{implementation}$$

In this part of the tutorial we will focus on the implementation of Maxwell's equations. At the end of this part, we will have written a series of `Python` functions which can be used to generate symbolic expressions for Maxwell's equations. We will go over:

* Basic `NRPy+` syntax
* Handling indexed expressions
* Finite difference derivatives
* Implementation of Maxwell's equations

## Step 1.a: Initialize core Python/NRPy+ modules \[Back to [top](#toc)\]
$$\label{initializenrpy}$$

Let's start by importing all the needed modules from NRPy+:

```python
# Step 1.a: Import core Python/NRPy+ modules
# Step 1.a.i: Import needed Python modules
import shutil, os, sys # Standard Python modules for multiplatform OS-level functions
import sympy as sp     # SymPy: a Python library for symbolic mathematics

# Step 1.a.ii: Add the "nrpy_core" directory to the path
base_dir      = os.getcwd()
nrpy_core_dir = os.path.join(base_dir,"..","..")
sys.path.append(nrpy_core_dir)

# Step 1.b: Load needed NRPy+ modules
from outputC import lhrh, outCfunction  # NRPy+: Core C code output module
import finite_difference as fin         # NRPy+: Finite difference C code generation module
import NRPy_param_funcs as par          # NRPy+: Parameter interface
import grid as gri                      # NRPy+: Functions having to do with numerical grids
import indexedexp as ixp                # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support
import reference_metric as rfm          # NRPy+: Reference metric support
import cmdline_helper as cmd            # NRPy+: Multi-platform Python command-line interface
import NRPy_logo as logo                # NRPy+: contains NRPy+ logo in ASCII art
```

## Step 1.b: Gridfunction declaration \[Back to [top](#toc)\]
$$\label{gridfunction_declaration}$$

When solving Maxwell's equations numerically using `NRPy`, we will *discretize* space as a *grid*. Assuming we choose a cube of length $L$ and discretize the $(x,y,z)$-directions using $(N_{x},N_{y},N_{z})$ points, respectively, we introduce the notation

$$
\begin{alignat}{2}
x \to x_{i} &\equiv x_{\rm min} + i \cdot \Delta x,\ &i=0,1,\ldots,N_{x}-1\\
y \to y_{j} &\equiv y_{\rm min} + j \cdot \Delta y,\ &j=0,1,\ldots,N_{y}-1\\
z \to z_{k} &\equiv z_{\rm min} + k \cdot \Delta z,\ &k=0,1,\ldots,N_{z}-1
\end{alignat}
$$

where $(\Delta x,\Delta y,\Delta z)$ are known as the *step sizes*. Our particular choice of discretization does not avoid the origin and is known as *vertex-centered*.

A particular function $f(x,y,z)$ can be evaluated on the points in the grid. We introduce the notation

$$
f_{ijk} \equiv f(x_{i},y_{j},z_{k}) = f(i\cdot\Delta x,j\cdot\Delta y,k\cdot\Delta z).
$$

$f_{ijk}$, i.e. the function $f(x,y,z)$ evaluated at the points of our numerical grid, is referred to as a *gridfunction*. Notice that the indices $i,j,k$ in this case are *not* indices associated with spacetime, but simply provide a useful bookeeping of where we are in our numerical grid.

In order to implement systems I and II, we will need the following *gridfunctions*:

* The electric field, $E^{i} = \bigl(E^{x},E^{y},E^{z}\bigr)$;
* The vector potential, $A^{i} = \bigl(A^{x},A^{y},A^{z}\bigr)$;
* The scalar potential, $\Phi$;
* The auxiliary scalar, $\Gamma$, which is only used in system II.

This means we need a total of 7 gridfunctions for system I and 8 gridfunctions for system II. We will now learn how to register [`SymPy`](https://www.sympy.org) variables which correspond to the gridfunctions above.

Notice first that the gridfunctions above fall into two distinct categories: scalar gridfunctions, $(\Phi,\Gamma)$, and 3-vector gridfunctions, $(E^{i},A^{i})$. Because of this, their declaration within `NRPy+` is also slightly different.

>Scalar gridfunctions are registered using the `NRPy+` module [`grid.py`](#FIXME)

>Indexed expression gridfunctions are registered using the `NRPy+` module [`indexedexp.py`](#FIXME)

We handle contravariant ("Up") and covariant ("Down") indices by appending "U"'s and "D"'s to the names of indexed expression variables. For example, in standard `NRPy+` notation, the variable that represents the tensor $M^{ab}_{\ \ \ \ \ \ cd}$ should be defined using

$$
\underbrace{M^{ab}_{\ \ \ \ \ \ cd}}_{\text{Indexed expression}} \leftrightarrow \underbrace{\text{MUUDD}}_{\text{NRPy+ variable}}.
$$

Variables that represent 4-dimensional indexed expressions should have a "4" in their names. For example, the 4-dimensional indexed expression $V_{\mu}$ is declared as

$$
\underbrace{V_{\mu}}_{\text{Indexed expression}} \leftrightarrow \underbrace{\text{V4D}}_{\text{NRPy+ variable}}.
$$

```python
# Step 1.b: Gridfunction declaration
def declare_MaxwellVacuum_gridfunctions_if_not_declared_already():
    # Step 1.b.i: This if statement simply prevents multiple
    #             declarations of the same gridfunctions,
    #             which would result in errors.
    for i in range(len(gri.glb_gridfcs_list)):
        if "Phi" in gri.glb_gridfcs_list[i].name:
            Phi,Gamma = sp.symbols("Phi Gamma",real=True)
            EU        = ixp.declarerank1("EU",DIM=3)
            AU        = ixp.declarerank1("AU",DIM=3)
            return Phi,Gamma,EU,AU

    # Step 1.b.i: Scalar gridfunctions: Phi and Gamma
    Phi,Gamma = gri.register_gridfunctions("EVOL",["Phi","Gamma"])

    # Step 1.b.ii: 3-vector gridfunctions: E^{i} and A^{i}
    EU = ixp.register_gridfunctions_for_single_rank1("EVOL","EU",DIM=3)
    AU = ixp.register_gridfunctions_for_single_rank1("EVOL","AU",DIM=3)

    return Phi,Gamma,EU,AU

Phi,Gamma,EU,AU = declare_MaxwellVacuum_gridfunctions_if_not_declared_already()
```

## Step 1.c: Finite difference derivatives \[Back to [top](#toc)\]
$$\label{finite_differences}$$

`NRPy+` handles derivatives numerically using [finite differences](https://en.wikipedia.org/wiki/Finite_difference). For example, to obtain the expression for $\partial_{x}f(x)$, we can use the following:

$$
f(x\pm\Delta x) = f(x) \pm \Delta x \partial_{x}f(x) + \frac{(\Delta x)^{2}}{2}\partial_{x}^{2}f(x) + \mathcal{O}\bigl(\Delta x^{3}\bigr),
$$

from which we can construct the *second-order accurate centered finite difference* approximation

$$
\underbrace{\partial_{x}f(x)}_{\text{Derivative}} = \underbrace{\frac{f(x+\Delta x) - f(x-\Delta x)}{2\Delta x}}_{\text{FD approximation}} + \mathcal{O}\bigl(\Delta x^{2}\bigr).
$$

In `NRPy+` the derivatives are replaced by the finite difference approximation only when we are generating the C code from the symbolic expressions. So let us begin our discussion by explaining how to represent derivatives in our symbolic expressions.

From the point of view of the programmer, a derivative of an object should be considered simply as another indexed expression. For example, we can think of the second derivative of the vector $P^{i}$ as the new object $p^{k}_{\ \ ij}$, i.e.

$$
p^{k}_{\ \ ij} \equiv \partial_{i}\partial_{j}P^{k}.
$$

This means that the declaration of derivatives is also performed using the [`indexedexp.py`](FIXME) module. However, there is a special notation to distinguish "derivative indices" ($i$ and $j$ in the example above) from "regular indices" ($k$ in the example above). This distinction must be introduced so that the [`outputC.py`](FIXME) module correctly identifies derivatives and replaces them with the appropriate finite difference expression. This special notation consists of appending "_d" (lowercase!) to the end of the variable's name, followed by the derivative indices. For example:

$$
\partial_{i}\partial_{j}P^{k} \equiv \underbrace{P^{k}_{\ ,ij}}_{\text{Indexed expression}} \leftrightarrow \underbrace{\text{PU_dDD}}_{\text{NRPy+ variable}}.
$$

The underscore preceding the derivative indices is introduced to distinguish *numerical* derivatives (i.e. those which we evaluate using finite differences) from *analytic* derivatives (i.e. those which we evaluate using `SymPy` and exact, symbolic differentiation).

Here we provide a simple example that evaluates the derivative of a scalar $\psi$ using finite differences. This variable will not be used again, so we prepend its name with an underscore to identify it as a dummy variable. Note also that we will remove $\psi$ from the list of gridfunctions after using it, thus avoiding undesireable gridfunctions when generating the ETK thorn.

[Using fourth-order accurate finite differences, the expressions we should obtain for the first and second derivatives](https://en.wikipedia.org/wiki/Finite_difference_coefficient) of $\psi$ along the $x$-direction are, respectively,

$$
\begin{align}
\partial_{x}\psi &= \frac{1}{\Delta x}\left(\frac{1}{12}\psi_{i-2} - \frac{2}{3}\psi_{i-1} + \frac{2}{3}\psi_{i+1} - \frac{1}{12}\psi_{i+2}\right),\\
\partial_{x}^{2}\psi &= \frac{1}{\Delta x^{2}}\left(-\frac{1}{12}\psi_{i-2} + \frac{4}{3}\psi_{i-1} - \frac{5}{2}\psi_{i} + \frac{4}{3}\psi_{i+1} - \frac{1}{12}\psi_{i+2}\right).
\end{align}
$$

Let us now demonstrate that `NRPy+` gives us the above expressions.

```python
# Step 1.c: Finite differences
# Step 1.c.i: Declare the gridfunction psi and
#             its first and second derivatives
_psi     = gri.register_gridfunctions("EVOL","psi")
_psi_dD  = ixp.declarerank1("psi_dD")
_psi_dDD = ixp.declarerank2("psi_dDD","sym01") # <- symmetric under i <-> j

# Step 1.c.ii: Now get the finite difference expression
par.set_parval_from_str("finite_difference::FD_CENTDERIVS_ORDER",4)

# Step 1.c.iii: Generate the C code
print(fin.FD_outputC("returnstring",
                     [lhrh(lhs="First__derivative",rhs=_psi_dD[0]),
                      lhrh(lhs="Second_derivative",rhs=_psi_dDD[0][0])],
                     params="outCverbose=False"))

# Step 1.c.iv: Remove the gridfunction psi from the list of gridfunctions
for gf in gri.glb_gridfcs_list:
    if gf.name == "psi":
        gri.glb_gridfcs_list.remove(gf)
```

    {
       /*
        * NRPy+ Finite Difference Code Generation, Step 1 of 2: Read from main memory and compute finite difference stencils:
        */
       const double psi_i0m2_i1_i2 = in_gfs[IDX4(PSIGF, i0-2,i1,i2)];
       const double psi_i0m1_i1_i2 = in_gfs[IDX4(PSIGF, i0-1,i1,i2)];
       const double psi = in_gfs[IDX4(PSIGF, i0,i1,i2)];
       const double psi_i0p1_i1_i2 = in_gfs[IDX4(PSIGF, i0+1,i1,i2)];
       const double psi_i0p2_i1_i2 = in_gfs[IDX4(PSIGF, i0+2,i1,i2)];
       const double FDPart1_Rational_2_3 = 2.0/3.0;
       const double FDPart1_Rational_1_12 = 1.0/12.0;
       const double FDPart1_Rational_5_2 = 5.0/2.0;
       const double FDPart1_Rational_4_3 = 4.0/3.0;
       const double psi_dD0 = invdx0*(FDPart1_Rational_1_12*(psi_i0m2_i1_i2 - psi_i0p2_i1_i2) + FDPart1_Rational_2_3*(-psi_i0m1_i1_i2 + psi_i0p1_i1_i2));
       const double psi_dDD00 = ((invdx0)*(invdx0))*(FDPart1_Rational_1_12*(-psi_i0m2_i1_i2 - psi_i0p2_i1_i2) + FDPart1_Rational_4_3*(psi_i0m1_i1_i2 + psi_i0p1_i1_i2) - FDPart1_Rational_5_2*psi);
       /*
        * NRPy+ Finite Difference Code Generation, Step 2 of 2: Evaluate SymPy expressions and write to main memory:
        */
       First__derivative = psi_dD0;
       Second_derivative = psi_dDD00;
    }

## Step 1.d: Implementation of system I \[Back to [top](#toc)\]
$$\label{system_I_implementation}$$

We now implement system I, derived in [Step 0.b](#potential_formulation_maxwell_equations) above, which read

$$
\text{System I:}\
\boxed{
\begin{align}
\color{blue}{\partial_{t}E^{i}} &\, \color{blue}{= -\partial^{j}\partial_{j}A^{i} + \partial^{i}\partial_{j}A^{j}}\\
\color{blue}{\partial_{t}A^{i}} &\, \color{blue}{= -E^{i} - \partial^{i}\Phi}\\
\color{blue}{\partial_{t}\Phi}  &\, \color{blue}{= -\partial_{i}A^{i}}\\
\color{red}{\partial_{i}E^{i}} &\, \color{red}{= 0}
\end{align}
}
\ .
$$

In the colorcode above, $\color{blue}{\text{blue}}$ equations are $\color{blue}{\text{evolution}}$ equations, while $\color{red}{\text{red}}$ equations are $\color{red}{\text{constraint}}$ equations.

### Step 1.d.i: System I evolution equations \[Back to [top](#toc)\]
$$\label{system_I_evolution_eqs}$$

We will now implement the evolution equations of system I, namely

$$
\begin{align}
\partial_{t}E^{i} &= -\partial^{j}\partial_{j}A^{i} + \partial^{i}\partial_{j}A^{j},\\
\partial_{t}A^{i} &= -E^{i} - \partial^{i}\Phi,\\
\partial_{t}\Phi  &= -\partial_{i}A^{i}.
\end{align}
$$

```python
# Step 1.d: Implementation of system I
# Step 1.d.i: System I evolution equations
def MaxwellVacuum_system_I_evolution_equations():
    # Step 1.d.i.1: Declare Maxwell gridfunctions
    _Phi,_Gamma,_EU,_AU = declare_MaxwellVacuum_gridfunctions_if_not_declared_already()

    # Step 1.d.i.2: Declare all derivatives appearing on the
    #               right-hand sides of the evolution equations
    #               of system I
    AU_dDD = ixp.declarerank3("AU_dDD","sym12")
    AU_dD  = ixp.declarerank2("AU_dD","nosym")
    Phi_dD = ixp.declarerank1("Phi_dD")

    # Step 1.d.i.3: Right-hand side of E^{i}:
    #
    # partial_{t}E^{i} = -partial^{j}partial_{j}A^{i} + partial^{i}partial_{j}A^{j}
    ErhsU = ixp.zerorank1()
    for i in range(DIM):
        for j in range(DIM):
            ErhsU[i] += -AU_dDD[i][j][j] + AU_dDD[j][j][i]

    # Step 1.d.i.4: Right-hand side of A^{i}:
    #
    # partial_{t}A^{i} = -E^{i} - partial^{i}Phi
    ArhsU = ixp.zerorank1()
    for i in range(DIM):
        ArhsU[i] = -EU[i] - Phi_dD[i]

    # Step 1.d.i.5: Right-hand side of Phi:
    #
    # partial_{t}Phi = -partial_{i}A^{i}
    Phi_rhs = sp.sympify(0)
    for i in range(DIM):
        Phi_rhs += -AU_dD[i][i]

    return ErhsU,ArhsU,Phi_rhs
```

### Step 1.d.ii: System I constraint equation \[Back to [top](#toc)\]
$$\label{system_I_constraint_eqs}$$

We now implement the constraint equation

$$
\mathcal{C} = \partial_{i}E^{i}.
$$

Notice that $\mathcal{C}=0$ *analytically*, but this constraint *is not* enforced during an evolution. It can instead be used as a diagnostic of how well our numerical solution satisfies Maxwell's equations.

```python
# Step 1.d.ii: System I constraint equation
def MaxwellVacuum_system_I_constraint_equation():
    # Step 1.d.ii.1: Declare Maxwell gridfunctions
    _Phi,_Gamma,_EU,_AU = declare_MaxwellVacuum_gridfunctions_if_not_declared_already()

    # Step 1.d.ii.2: Declare all derivatives appearing on the
    #                right-hand sides of the constraint equations
    #                of system I
    EU_dD = ixp.declarerank2("EU_dD", "nosym")

    # Step 1.d.ii.3: Constraint equation (Gauss' law in vacuum):
    #
    # C = partial_{i}E^{i}
    C = sp.sympify(0)
    for i in range(DIM):
        C += EU_dD[i][i]

    return C
```

## Step 1.e: Implementation of system II \[Back to [top](#toc)\]
$$\label{system_II_implementation}$$

We now implement system II, derived in [Step 0.c](#hyperbolicity_maxwell_equations) above, which read

$$
\text{System II:}\
\boxed{
\begin{align}
\color{blue}{\partial_{t}E^{i}}  &\, \color{blue}{= -\partial^{j}\partial_{j}A^{i} + \partial^{i}\Gamma}\\
\color{blue}{\partial_{t}A^{i}}  &\, \color{blue}{= -E^{i} - \partial^{i}\Phi}\\
\color{blue}{\partial_{t}\Phi}   &\, \color{blue}{= -\Gamma}\\
\color{blue}{\partial_{t}\Gamma} &\, \color{blue}{= -\partial^{i}\partial_{i}\Phi}\\
\color{red}{\partial_{i}E^{i}}   &\, \color{red}{= 0}\\
\color{red}{\Gamma}              &\, \color{red}{= \partial_{i}A^{i}}
\end{align}
}\ .
$$

In the colorcode above, $\color{blue}{\text{blue}}$ equations are $\color{blue}{\text{evolution}}$ equations, while $\color{red}{\text{red}}$ equations are $\color{red}{\text{constraint}}$ equations.

### Step 1.e.i: System II evolution equations \[Back to [top](#toc)\]
$$\label{system_II_evolution_eqs}$$

We will now implement the evolution equations of system II, namely

$$
\begin{align}
\partial_{t}E^{i}  &= -\partial^{j}\partial_{j}A^{i} + \partial^{i}\Gamma,\\
\partial_{t}A^{i}  &= -E^{i} - \partial^{i}\Phi,\\
\partial_{t}\Phi   &= -\Gamma,\\
\partial_{t}\Gamma &= -\partial^{i}\partial_{i}\Phi.
\end{align}
$$

```python
# Step 1.e: Implementation of system II
# Step 1.e.i: System II evolution equations
def MaxwellVacuum_system_II_evolution_equations():
    # Step 1.e.i.1: Declare Maxwell gridfunctions
    _Phi,Gamma,_EU,_AU = declare_MaxwellVacuum_gridfunctions_if_not_declared_already()

    # Step 1.e.i.2: Declare all derivatives appearing on the
    #               right-hand sides of the evolution equations
    #               of system II
    AU_dDD   = ixp.declarerank3("AU_dDD","sym12")
    Phi_dD   = ixp.declarerank1("Phi_dD")
    Phi_dDD  = ixp.declarerank2("Phi_dDD","sym01")
    Gamma_dD = ixp.declarerank1("Gamma_dD")

    # Step 1.e.i.3: Right-hand side of E^{i}:
    #
    # partial_{t}E^{i} = -partial^{j}partial_{j}A^{i} + partial^{i}Gamma
    ErhsU = ixp.zerorank1()
    for i in range(DIM):
        ErhsU[i] += Gamma_dD[i]
        for j in range(DIM):
            ErhsU[i] -= AU_dDD[i][j][j]

    # Step 1.e.i.4: Right-hand side of A^{i}:
    #
    # partial_{t}A^{i} = -E^{i} - partial^{i}Phi
    ArhsU = ixp.zerorank1()
    for i in range(DIM):
        ArhsU[i] = -EU[i] - Phi_dD[i]

    # Step 1.e.i.5: Right-hand side of Phi:
    #
    # partial_{t}Phi = -Gamma
    Phi_rhs = -Gamma

    # Step 1.e.i.6: Right-hand side of Gamma:
    #
    # partial_{t}Gamma = -partial^{i}partial_{i}Phi
    Gamma_rhs = sp.sympify(0)
    for i in range(DIM):
        Gamma_rhs -= Phi_dDD[i][i]

    return ErhsU,ArhsU,Phi_rhs,Gamma_rhs
```

### Step 1.e.ii: System II constraint equations \[Back to [top](#toc)\]
$$\label{system_II_constraint_eqs}$$

We now implement the constraint equations

$$
\begin{align}
\mathcal{C} &= \partial_{i}E^{i},\\
\mathcal{G} &= \Gamma - \partial_{i}A^{i}.
\end{align}
$$

Notice that $\mathcal{C}=0$ and $\mathcal{G}=0$ *analytically*, but these constraints *are not* enforced during an evolution. They can instead be used as diagnostics of how well our numerical solution satisfies Maxwell's equations.

```python
# Step 1.e.ii: System I constraint equations
def MaxwellVacuum_system_II_constraint_equations():
    # Step 1.e.ii.1: Declare Maxwell gridfunctions
    _Phi,_Gamma,_EU,_AU = declare_MaxwellVacuum_gridfunctions_if_not_declared_already()

    # Step 1.e.ii.2: Declare all derivatives appearing on the
    #                right-hand sides of the constraint equations
    #                of system II
    EU_dD = ixp.declarerank2("EU_dD", "nosym")
    AU_dD = ixp.declarerank2("AU_dD", "nosym")

    # Step 1.e.ii.3: Gauss' law in vacuum:
    #
    # C = partial_{i}E^{i}
    C = sp.sympify(0)
    for i in range(DIM):
        C += EU_dD[i][i]

    # Step 1.e.ii.4: Gamma-constraint:
    #
    # G = Gamma - partial_{i}A^{i}
    G = Gamma
    for i in range(DIM):
        G += -AU_dD[i][i]

    return C,G
```

# Step 2: Part II - Einstein Toolkit thorn writing \[Back to [top](#toc)\]
$$\label{thorn_writing}$$

In this part of the tutorial we will focusing on writing the `MaxwellVacuum` ETK thorn. We will go over:

* Generating C code kernels from the symbolic expressions, including a function to initialize the right-hand sides to zero
* Writing all additional functions that are needed by the thorn:
    * The right-hand sides driver function
    * The constraints driver function
    * Registering the evolved, and respective right-hand sides, gridfunctions so that they can be used by the [Method of Lines](https://einsteintoolkit.org/thornguide/CactusNumerical/MoL/documentation.html) thorn
    * Specifying gridfunctions symmetries
    * Boundary conditions
    * Printing the `NRPy+` logo when using this thorn
* Writing the thorn's configuration (`*.ccl`) files

## Step 2.a: Generating C code kernels for Maxwell's equations \[Back to [top](#toc)\]
$$\label{generating_c_code_kernels}$$

We now focus on generating the C code kernels that are needed by the `MaxwellVacuum` thorn. These will be source files containing a single C function each. The functions can be used to compute, for example, the right-hand sides of Maxwell's evolution equations, and the source code which composes the core of the function's body is the result of converting the `SymPy` symbolic expressions into highly optimized C code.

We will then focus on how to generate this highly optimized C code in `NRPy+` using the `outputC.py` module. Because we want our symbolic derivatives to be replaced by finite differences approximations, we will be using the wrapper function `FD_outputC()`, defined in the `finite_differences.py` module.

```python
# Step 2.a: Generating C code kernels for Maxwell's equations

# Step 2.a.0.i: Set the thorn's directories
Thorndir  = "MaxwellVacuum"
Ccodesdir = os.path.join(Thorndir,"src")
shutil.rmtree(Thorndir, ignore_errors=True)
cmd.mkdir(Thorndir)
cmd.mkdir(Ccodesdir)

# Step 2.a.0.ii: Copy SIMD compiler intrinsics files
cmd.mkdir(os.path.join(Ccodesdir,"SIMD"))
shutil.copy(os.path.join(nrpy_core_dir,"SIMD","SIMD_intrinsics.h"),os.path.join(Ccodesdir,"SIMD"))

# Step 2.a.0.iii: Enable rfm precompute
cmd.mkdir(os.path.join(Ccodesdir,"rfm_files"))
par.set_parval_from_str("reference_metric::enable_rfm_precompute","True")
par.set_parval_from_str("reference_metric::rfm_precompute_Ccode_outdir",os.path.join(Ccodesdir,"rfm_files"))

# Step 2.a.0.iv: Set gridfunction memory access mode
par.set_parval_from_str("grid::GridFuncMemAccess","ETK")

# Step 2.a.0.v: Set coordinate system and generate reference_metric files
CoordSystem = "Cartesian"
par.set_parval_from_str("reference_metric::CoordSystem",CoordSystem)
rfm.reference_metric()

# Step 2.a.0.vi: Set number of spatial dimensions
par.set_parval_from_str("grid::DIM",3)
DIM = par.parval_from_str("grid::DIM")

# Step 2.a.0.vii: Master C code generating function
def MaxwellVacuum_C_code_generation_function(name,desc,gf_and_expr_list,make_code_defn_list=None):

    outfile = os.path.join(Ccodesdir,name+".c")
    outCfunction(
        includes = ["math.h","cctk.h","cctk_Arguments.h","cctk_Parameters.h","SIMD/SIMD_intrinsics.h"],
        outfile  = outfile,
        desc     = desc,
        name     = name,
        enableCparameters = False,
        params   = "CCTK_ARGUMENTS",
        preloop  = """

    DECLARE_CCTK_ARGUMENTS;
    const CCTK_REAL NOSIMDinvdx0 = 1.0/CCTK_DELTA_SPACE(0);
    const REAL_SIMD_ARRAY invdx0 = ConstSIMD(NOSIMDinvdx0);
    const CCTK_REAL NOSIMDinvdx1 = 1.0/CCTK_DELTA_SPACE(1);
    const REAL_SIMD_ARRAY invdx1 = ConstSIMD(NOSIMDinvdx1);
    const CCTK_REAL NOSIMDinvdx2 = 1.0/CCTK_DELTA_SPACE(2);
    const REAL_SIMD_ARRAY invdx2 = ConstSIMD(NOSIMDinvdx2);

#pragma omp parallel for
    for(int i2=cctk_nghostzones[2];i2<cctk_lsh[2]-cctk_nghostzones[2];i2++) {
        #include "rfm_files/rfm_struct__SIMD_inner_read2.h"
        for(int i1=cctk_nghostzones[1];i1<cctk_lsh[1]-cctk_nghostzones[1];i1++) {
            #include "rfm_files/rfm_struct__SIMD_inner_read1.h"
            for(int i0=cctk_nghostzones[0];i0<cctk_lsh[0]-cctk_nghostzones[0];i0+=SIMD_width) {
                #include "rfm_files/rfm_struct__SIMD_inner_read0.h"
""",
        body     =fin.FD_outputC("returnstring",gf_and_expr_list,
                                 params="outCverbose=False,includebraces=False,enable_SIMD=True,preindent=8"),
        postloop = """
            } // for(int i0=cctk_nghostzones[0];i0<cctk_lsh[0]-cctk_nghostzones[0];i0+=SIMD_width)
        } // for(int i1=cctk_nghostzones[1];i1<cctk_lsh[1]-cctk_nghostzones[1];i1++)
    } // for(int i2=cctk_nghostzones[2];i2<cctk_lsh[2]-cctk_nghostzones[2];i2++)
""")

    if make_code_defn_list is not None:
        make_code_defn_list.append(name+".c")
```

### Step 2.a.i: System I C code generation \[Back to [top](#toc)\]
$$\label{system_I_c_code_generation}$$

We will now generate the C code kernels for [System I](#systems_I_and_II). We will generate a total of 8 source files containing:

* Functions to evaluate the right-hand sides of the evolution equations in System I using:
    * 2nd order finite differences
    * 4th order finite differences
    * 6th order finite differences
    * 8th order finite differences
* Functions to evaluate the constraint equation in System I using:
    * 2nd order finite differences
    * 4th order finite differences
    * 6th order finite differences
    * 8th order finite differences

```python
# Step 2.a.i: System I C code generation
# Step 2.a.i.1: Clear list of registered gridfunctions
gri.glb_gridfcs_list = []

# Step 2.a.i.2: Declare an auxiliary gridfunction "C" to store Gauss' law
_CGF = gri.register_gridfunctions("AUX","C")

# Step 2.a.i.3: Store the current (dfault) finite difference order
FD_order_orig = par.parval_from_str("finite_difference::FD_CENTDERIVS_ORDER")

# Step 2.a.i.4: Set which finite difference orders we want
FD_order_min  = 2
FD_order_max  = 8
FD_order_step = 2

# Step 2.a.i.5: Generate the C code kernels
make_code_defn_list = []
for FD_order in range(FD_order_min,FD_order_max+1,FD_order_step):

    # Update finite difference order
    par.set_parval_from_str("finite_difference::FD_CENTDERIVS_ORDER",FD_order)

    # Evolution equations
    name                = "MaxwellVacuum_System_I_RHSs_FD_order_%d"%(FD_order)
    desc                = "Right-hand sides of the evolution equations in system I - FD order = %d"%(FD_order)
    ErhsU,ArhsU,Phi_rhs = MaxwellVacuum_system_I_evolution_equations()
    gf_and_expr_list    = [lhrh(lhs=gri.gfaccess("rhs_gfs","EU0"),rhs=ErhsU[0]),
                           lhrh(lhs=gri.gfaccess("rhs_gfs","EU1"),rhs=ErhsU[1]),
                           lhrh(lhs=gri.gfaccess("rhs_gfs","EU2"),rhs=ErhsU[2]),
                           lhrh(lhs=gri.gfaccess("rhs_gfs","AU0"),rhs=ArhsU[0]),
                           lhrh(lhs=gri.gfaccess("rhs_gfs","AU1"),rhs=ArhsU[1]),
                           lhrh(lhs=gri.gfaccess("rhs_gfs","AU2"),rhs=ArhsU[2]),
                           lhrh(lhs=gri.gfaccess("rhs_gfs","Phi"),rhs=Phi_rhs)]
    MaxwellVacuum_C_code_generation_function(name,desc,gf_and_expr_list,make_code_defn_list=make_code_defn_list)

    # Constraint equations
    name             = "MaxwellVacuum_System_I_constraints_FD_order_%d"%(FD_order)
    desc             = "Constraint equations in system I - FD order = %d"%(FD_order)
    C                = MaxwellVacuum_system_I_constraint_equation()
    gf_and_expr_list = lhrh(lhs=gri.gfaccess("aux_gfs","C"),rhs=C)
    MaxwellVacuum_C_code_generation_function(name,desc,gf_and_expr_list,make_code_defn_list=make_code_defn_list)

# Step 2.a.i.6: Restore original FD_order
par.set_parval_from_str("finite_difference::FD_CENTDERIVS_ORDER",FD_order_orig)
```

    Output C function MaxwellVacuum_System_I_RHSs_FD_order_2() to file MaxwellVacuum/src/MaxwellVacuum_System_I_RHSs_FD_order_2.c
    Output C function MaxwellVacuum_System_I_constraints_FD_order_2() to file MaxwellVacuum/src/MaxwellVacuum_System_I_constraints_FD_order_2.c
    Output C function MaxwellVacuum_System_I_RHSs_FD_order_4() to file MaxwellVacuum/src/MaxwellVacuum_System_I_RHSs_FD_order_4.c
    Output C function MaxwellVacuum_System_I_constraints_FD_order_4() to file MaxwellVacuum/src/MaxwellVacuum_System_I_constraints_FD_order_4.c
    Output C function MaxwellVacuum_System_I_RHSs_FD_order_6() to file MaxwellVacuum/src/MaxwellVacuum_System_I_RHSs_FD_order_6.c
    Output C function MaxwellVacuum_System_I_constraints_FD_order_6() to file MaxwellVacuum/src/MaxwellVacuum_System_I_constraints_FD_order_6.c
    Output C function MaxwellVacuum_System_I_RHSs_FD_order_8() to file MaxwellVacuum/src/MaxwellVacuum_System_I_RHSs_FD_order_8.c
    Output C function MaxwellVacuum_System_I_constraints_FD_order_8() to file MaxwellVacuum/src/MaxwellVacuum_System_I_constraints_FD_order_8.c

### Step 2.a.ii: System II C code generation \[Back to [top](#toc)\]
$$\label{system_II_c_code_generation}$$

We will now generate the C code kernels for [System II](#systems_I_and_II). We will generate a total of 8 source files containing:

* Functions to evaluate the right-hand sides of the evolution equations in System II using:
    * 2nd order finite differences
    * 4th order finite differences
    * 6th order finite differences
    * 8th order finite differences
* Functions to evaluate the constraint equation in System II using:
    * 2nd order finite differences
    * 4th order finite differences
    * 6th order finite differences
    * 8th order finite differences

```python
# Step 2.a.ii: System II C code generation
# Step 2.a.ii.1: Clear list of registered gridfunctions
gri.glb_gridfcs_list = []

# Step 2.a.ii.2: Declare an auxiliary gridfunction "C" to store Gauss' law
_CGF,_GGF = gri.register_gridfunctions("AUX",["C","G"])

# Step 2.a.ii.3: Store the current (dfault) finite difference order
FD_order_orig = par.parval_from_str("finite_difference::FD_CENTDERIVS_ORDER")

# Step 2.a.ii.4: Set which finite difference orders we want
FD_order_min  = 2
FD_order_max  = 8
FD_order_step = 2

# Step 2.a.ii.5: Generate the C code kernels
for FD_order in range(FD_order_min,FD_order_max+1,FD_order_step):

    # Update finite difference order
    par.set_parval_from_str("finite_difference::FD_CENTDERIVS_ORDER",FD_order)

    # Evolution equations
    desc                          = "Right-hand sides of the evolution equations in system II - FD order = %d"%(FD_order)
    name                          = "MaxwellVacuum_System_II_RHSs_FD_order_%d"%(FD_order)
    ErhsU,ArhsU,Phi_rhs,Gamma_rhs = MaxwellVacuum_system_II_evolution_equations()
    gf_and_expr_list              = [lhrh(lhs=gri.gfaccess("rhs_gfs","EU0"  ),rhs=ErhsU[0]),
                                     lhrh(lhs=gri.gfaccess("rhs_gfs","EU1"  ),rhs=ErhsU[1]),
                                     lhrh(lhs=gri.gfaccess("rhs_gfs","EU2"  ),rhs=ErhsU[2]),
                                     lhrh(lhs=gri.gfaccess("rhs_gfs","AU0"  ),rhs=ArhsU[0]),
                                     lhrh(lhs=gri.gfaccess("rhs_gfs","AU1"  ),rhs=ArhsU[1]),
                                     lhrh(lhs=gri.gfaccess("rhs_gfs","AU2"  ),rhs=ArhsU[2]),
                                     lhrh(lhs=gri.gfaccess("rhs_gfs","Phi"  ),rhs=Phi_rhs),
                                     lhrh(lhs=gri.gfaccess("rhs_gfs","Gamma"),rhs=Gamma_rhs)]
    MaxwellVacuum_C_code_generation_function(name,desc,gf_and_expr_list,make_code_defn_list=make_code_defn_list)

    # Constraint equations
    desc             = "Constraint equations in system II - FD order = %d"%(FD_order)
    name             = "MaxwellVacuum_System_II_constraints_FD_order_%d"%(FD_order)
    C,G              = MaxwellVacuum_system_II_constraint_equations()
    gf_and_expr_list = [lhrh(lhs=gri.gfaccess("aux_gfs","C"),rhs=C),
                        lhrh(lhs=gri.gfaccess("aux_gfs","G"),rhs=G)]
    MaxwellVacuum_C_code_generation_function(name,desc,gf_and_expr_list,make_code_defn_list=make_code_defn_list)

# Step 2.a.ii.6: Restore original FD_order
par.set_parval_from_str("finite_difference::FD_CENTDERIVS_ORDER",FD_order_orig)
```

    Output C function MaxwellVacuum_System_II_RHSs_FD_order_2() to file MaxwellVacuum/src/MaxwellVacuum_System_II_RHSs_FD_order_2.c
    Output C function MaxwellVacuum_System_II_constraints_FD_order_2() to file MaxwellVacuum/src/MaxwellVacuum_System_II_constraints_FD_order_2.c
    Output C function MaxwellVacuum_System_II_RHSs_FD_order_4() to file MaxwellVacuum/src/MaxwellVacuum_System_II_RHSs_FD_order_4.c
    Output C function MaxwellVacuum_System_II_constraints_FD_order_4() to file MaxwellVacuum/src/MaxwellVacuum_System_II_constraints_FD_order_4.c
    Output C function MaxwellVacuum_System_II_RHSs_FD_order_6() to file MaxwellVacuum/src/MaxwellVacuum_System_II_RHSs_FD_order_6.c
    Output C function MaxwellVacuum_System_II_constraints_FD_order_6() to file MaxwellVacuum/src/MaxwellVacuum_System_II_constraints_FD_order_6.c
    Output C function MaxwellVacuum_System_II_RHSs_FD_order_8() to file MaxwellVacuum/src/MaxwellVacuum_System_II_RHSs_FD_order_8.c
    Output C function MaxwellVacuum_System_II_constraints_FD_order_8() to file MaxwellVacuum/src/MaxwellVacuum_System_II_constraints_FD_order_8.c

### Step 2.a.iii: C code to initialize right-hand sides to zero \[Back to [top](#toc)\]
$$\label{zero_rhss}$$

This is a simple function that is used to initialize the right-hand side gridfunctions to zero.

```python
# Step 2.a.iii: C code to initialize right-hand sides to zero
zero                          = sp.sympify(0)
name                          = "MaxwellVacuum_zero_rhss"
desc                          = "Set right-hand sides of the evolution equations to zero"
gf_and_expr_list              = [lhrh(lhs=gri.gfaccess("rhs_gfs","EU0"  ),rhs=zero),
                                 lhrh(lhs=gri.gfaccess("rhs_gfs","EU1"  ),rhs=zero),
                                 lhrh(lhs=gri.gfaccess("rhs_gfs","EU2"  ),rhs=zero),
                                 lhrh(lhs=gri.gfaccess("rhs_gfs","AU0"  ),rhs=zero),
                                 lhrh(lhs=gri.gfaccess("rhs_gfs","AU1"  ),rhs=zero),
                                 lhrh(lhs=gri.gfaccess("rhs_gfs","AU2"  ),rhs=zero),
                                 lhrh(lhs=gri.gfaccess("rhs_gfs","Phi"  ),rhs=zero),
                                 lhrh(lhs=gri.gfaccess("rhs_gfs","Gamma"),rhs=zero)]

outfile = os.path.join(Ccodesdir,name+".c")
outCfunction(
    includes = ["math.h","cctk.h","cctk_Arguments.h","cctk_Parameters.h","SIMD/SIMD_intrinsics.h"],
    outfile  = outfile,
    desc     = desc,
    name     = name,
    enableCparameters=False,
    params   = "CCTK_ARGUMENTS",
    preloop  = """

    DECLARE_CCTK_ARGUMENTS;

#pragma omp parallel for
    for(int i2=0;i2<cctk_lsh[2];i2++) {
        for(int i1=0;i1<cctk_lsh[1];i1++) {
            for(int i0=0;i0<cctk_lsh[0];i0+=SIMD_width) {
""",
    body     =fin.FD_outputC("returnstring",gf_and_expr_list,
                             params="outCverbose=False,includebraces=False,enable_SIMD=True,preindent=8"),
    postloop = """
            } // for(int i0=0;i0<cctk_lsh[0];i0+=SIMD_width)
        } // for(int i1=0;i1<cctk_lsh[1];i1++)
    } // for(int i2=0;i2<cctk_lsh[2];i2++)
""")

make_code_defn_list.append(name+".c")
```

    Output C function MaxwellVacuum_zero_rhss() to file MaxwellVacuum/src/MaxwellVacuum_zero_rhss.c

## Step 2.b: The right-hand side driver function \[Back to [top](#toc)\]
$$\label{rhs_driver}$$

This function is used to select which of the right-hand sides functions we have defined above should be called. There are two parameters in the `MaxwellVacuum` thorn (which we will declare below) that steer the behaviour of the right-hand side driver:

* `FD_order`
* `which_system`

The first of them, `FD_order`, selects which finite difference order we want to use when evaluating the right-hand sides of the evolution equations. The second of them, `which_system`, selects whether we will use the evolution equations in System I or System II.

```python
# Step 2.b: The right-hand side driver function
# Step 2.b.i: Get the names of the RHSs functions
fct_names = []
for fct in make_code_defn_list:
    if "RHSs" in fct:
        fct_names.append(fct.split(".")[0])

# Step 2.b.ii: Includes we will need in our driver funtion
include_string = ""
for fct_name in fct_names:
    include_string += "extern void "+fct_name+"(CCTK_ARGUMENTS);\n"

# Step 2.b.iii: Right-hand side evaluation driver function
driver_function_string = """
#include <math.h>
#include "cctk.h"
#include "cctk_Arguments.h"
#include "cctk_Parameters.h"
#include "SIMD/SIMD_intrinsics.h"
"""+include_string+"""
void MaxwellVacuum_RHSs(CCTK_ARGUMENTS) {

    DECLARE_CCTK_ARGUMENTS;
    DECLARE_CCTK_PARAMETERS;

    if( CCTK_EQUALS(which_system,"SystemI")) {
        if(FD_order == 2) {
            MaxwellVacuum_System_I_RHSs_FD_order_2(CCTK_PASS_CTOC);
        }
        else if(FD_order == 4) {
            MaxwellVacuum_System_I_RHSs_FD_order_4(CCTK_PASS_CTOC);
        }
        else if(FD_order == 6) {
            MaxwellVacuum_System_I_RHSs_FD_order_6(CCTK_PASS_CTOC);
        }
        else if(FD_order == 8) {
            MaxwellVacuum_System_I_RHSs_FD_order_8(CCTK_PASS_CTOC);
        }
        else {
            CCTK_VError(__LINE__,__FILE__,CCTK_THORNSTRING,
                        "Error: unsupported FD_order: %d",FD_order);
        }
    }
    else if( CCTK_EQUALS(which_system,"SystemII")) {
        if(FD_order == 2) {
            MaxwellVacuum_System_II_RHSs_FD_order_2(CCTK_PASS_CTOC);
        }
        else if(FD_order == 4) {
            MaxwellVacuum_System_II_RHSs_FD_order_4(CCTK_PASS_CTOC);
        }
        else if(FD_order == 6) {
            MaxwellVacuum_System_II_RHSs_FD_order_6(CCTK_PASS_CTOC);
        }
        else if(FD_order == 8) {
            MaxwellVacuum_System_II_RHSs_FD_order_8(CCTK_PASS_CTOC);
        }
        else {
            CCTK_VError(__LINE__,__FILE__,CCTK_THORNSTRING,
                        "Error: unsupported FD_order: %d. Supported orders are: 2, 4, 6, and 8.",FD_order);
        }
    }
    else {
        CCTK_VError(__LINE__,__FILE__,CCTK_THORNSTRING,
                        "Error: unsupported which_system: %s. Supported systems are \\"SystemI\\" and \\"SystemII\\".",which_system);
    }
}
"""

# Step 2.b.iv: Write the right-hand side evaluation driver function to file
name    = "MaxwellVacuum_RHSs.c"
outfile = os.path.join(Ccodesdir,name)
with open(outfile,"w") as file:
    file.write(driver_function_string)
print("Wrote file \"%s\""%outfile)

make_code_defn_list.append(name)
```

    Wrote file "MaxwellVacuum/src/MaxwellVacuum_RHSs.c"

## Step 2.c: The constraints driver function \[Back to [top](#toc)\]
$$\label{constraints_driver}$$

This function is used to select which of the constraint functions we have generated above should be called. There are two parameters in the `MaxwellVacuum` thorn (which we will declare below) that steer the behaviour of the constraints driver:

* `FD_order`
* `which_system`

The first of them, `FD_order`, selects which finite difference order we want to use when evaluating the constraint equations. The second of them, `which_system`, selects whether we will use the constraint equations in System I or System II.

```python
# Step 2.c: The constraints driver function
# Step 2.c.i: Get the names of the constraints functions
fct_names = []
for fct in make_code_defn_list:
    if "constraint" in fct:
        fct_names.append(fct.split(".")[0])

# Step 2.c.ii: Includes we will need in our driver funtion
include_string = ""
for fct_name in fct_names:
    include_string += "extern void "+fct_name+"(CCTK_ARGUMENTS);\n"

# Step 2.c.iii: Constraints driver function
driver_function_string = """
#include <math.h>
#include "cctk.h"
#include "cctk_Arguments.h"
#include "cctk_Parameters.h"
#include "SIMD/SIMD_intrinsics.h"
"""+include_string+"""
void MaxwellVacuum_constraints(CCTK_ARGUMENTS) {

    DECLARE_CCTK_ARGUMENTS;
    DECLARE_CCTK_PARAMETERS;

    if( CCTK_EQUALS(which_system,"SystemI")) {
        if(FD_order == 2) {
            MaxwellVacuum_System_I_constraints_FD_order_2(CCTK_PASS_CTOC);
        }
        else if(FD_order == 4) {
            MaxwellVacuum_System_I_constraints_FD_order_4(CCTK_PASS_CTOC);
        }
        else if(FD_order == 6) {
            MaxwellVacuum_System_I_constraints_FD_order_6(CCTK_PASS_CTOC);
        }
        else if(FD_order == 8) {
            MaxwellVacuum_System_I_constraints_FD_order_8(CCTK_PASS_CTOC);
        }
        else {
            CCTK_VError(__LINE__,__FILE__,CCTK_THORNSTRING,
                        "Error: unsupported FD_order: %d",FD_order);
        }
    }
    else if( CCTK_EQUALS(which_system,"SystemII")) {
        if(FD_order == 2) {
            MaxwellVacuum_System_II_constraints_FD_order_2(CCTK_PASS_CTOC);
        }
        else if(FD_order == 4) {
            MaxwellVacuum_System_II_constraints_FD_order_4(CCTK_PASS_CTOC);
        }
        else if(FD_order == 6) {
            MaxwellVacuum_System_II_constraints_FD_order_6(CCTK_PASS_CTOC);
        }
        else if(FD_order == 8) {
            MaxwellVacuum_System_II_constraints_FD_order_8(CCTK_PASS_CTOC);
        }
        else {
            CCTK_VError(__LINE__,__FILE__,CCTK_THORNSTRING,
                        "Error: unsupported FD_order: %d. Supported orders are: 2, 4, 6, and 8.",FD_order);
        }
    }
    else {
        CCTK_VError(__LINE__,__FILE__,CCTK_THORNSTRING,
                        "Error: unsupported which_system: %s. Supported systems are \\"SystemI\\" and \\"SystemII\\".",which_system);
    }
}
"""

# Step 2.c.iv: Write the constraints driver function to file
name    = "MaxwellVacuum_constraints.c"
outfile = os.path.join(Ccodesdir,name)
with open(outfile,"w") as file:
    file.write(driver_function_string)
print("Wrote file \"%s\""%outfile)

make_code_defn_list.append(name)
```

    Wrote file "MaxwellVacuum/src/MaxwellVacuum_constraints.c"

## Step 2.d: Registering gridfunctions for the Method of Lines \[Back to [top](#toc)\]
$$\label{mol_registration}$$

We now register the gridfunctions from the `MaxwellVacuum` thorn so that they can be used the [Method of Lines](https://einsteintoolkit.org/thornguide/CactusNumerical/MoL/documentation.html) ETK thorn.

```python
make_code_defn_list.append("MaxwellVacuum_MoL_registration.c")
```

```python
%%writefile $Ccodesdir/MaxwellVacuum_MoL_registration.c
//--------------------------------------------------------------------------
// Register with the Method of Lines time stepper
// (MoL thorn, found in arrangements/CactusBase/MoL)
// MoL documentation located in arrangements/CactusBase/MoL/doc
//--------------------------------------------------------------------------
#include <stdio.h>

#include "cctk.h"
#include "cctk_Parameters.h"
#include "cctk_Arguments.h"

#include "Symmetry.h"

void MaxwellVacuum_MoL_registration(CCTK_ARGUMENTS)
{
  DECLARE_CCTK_ARGUMENTS;
  DECLARE_CCTK_PARAMETERS;

  CCTK_INT group, rhs, ierr=0;

  // Register evolution & RHS gridfunction groups with MoL, so it knows

  group = CCTK_GroupIndex("MaxwellVacuum::evol_variables");
  rhs   = CCTK_GroupIndex("MaxwellVacuum::evol_variables_rhs");
  ierr += MoLRegisterEvolvedGroup(group, rhs);

  if(ierr) CCTK_ERROR("Problems registering with MoL");
}
```

    Writing MaxwellVacuum/src/MaxwellVacuum_MoL_registration.c

## Step 2.e: Set gridfunction symmetries \[Back to [top](#toc)\]
$$\label{gridfunction_symmetries}$$

This function tells the [`CartGrid3D`](https://einsteintoolkit.org/thornguide/CactusBase/CartGrid3D/documentation.html) ETK thorn what are the symmetry properties of each of the gridfunctions used by the `MaxwellVacuum` thorn. This is used to know the parity of the gridfunctions, for example

$$
f(-x,y,z) = s f(x,y,z),
$$

with $s=+1\ (-1)$ meaning the function is even (odd) in $x$.

```python
# Step 2.e: Set gridfuntion symmetries
# Step 2.e.i: Get list of evolved and auxiliary gridfunctions
evol_gfs_list = []
aux_gfs_list  = []
for i in range(len(gri.glb_gridfcs_list)):
    if gri.glb_gridfcs_list[i].gftype == "EVOL":
        evol_gfs_list.append(gri.glb_gridfcs_list[i].name+"GF")

    if gri.glb_gridfcs_list[i].gftype == "AUX":
        aux_gfs_list.append( gri.glb_gridfcs_list[i].name+"GF")

# Step 2.e.ii: Sort gridfunction lists
evol_gfs_list.sort()
aux_gfs_list.sort()

# Step 2.e.iii: Get list of all (evol+aux) gridfunctions
full_gfs_list = []
full_gfs_list.extend(evol_gfs_list)
full_gfs_list.extend(aux_gfs_list)

# Step 2.e.iv: Write the function to a string
outstr = """
#include "cctk.h"
#include "cctk_Arguments.h"
#include "cctk_Parameters.h"
#include "Symmetry.h"

void MaxwellVacuum_Symmetry_registration(CCTK_ARGUMENTS) {

  DECLARE_CCTK_ARGUMENTS;
  DECLARE_CCTK_PARAMETERS;

  // Stores gridfunction parity across x=0, y=0, and z=0 planes, respectively
  int sym[3];

  // Next register parities for each gridfunction based on its name
  //    (to ensure this algorithm is robust, gridfunctions with integers
  //     in their base names are forbidden in NRPy+).
"""
outstr += ""
for gfname in full_gfs_list:
    gfname_without_GFsuffix = gfname[:-2]
    outstr += """
  // Default to scalar symmetry:
  sym[0] = 1; sym[1] = 1; sym[2] = 1;
  // Now modify sym[0], sym[1], and/or sym[2] as needed
  //    to account for gridfunction parity across
  //    x=0, y=0, and/or z=0 planes, respectively
"""
    # If gridfunction name does not end in a digit, by NRPy+ syntax, it must be a scalar
    if gfname_without_GFsuffix[len(gfname_without_GFsuffix) - 1].isdigit() == False:
        outstr += "      // (this gridfunction is a scalar -- no need to change default sym[]'s!)\n"
    elif len(gfname_without_GFsuffix) > 2:
        # Rank-1 indexed expression (e.g., vector)
        if gfname_without_GFsuffix[len(gfname_without_GFsuffix) - 2].isdigit() == False:
            if int(gfname_without_GFsuffix[-1]) > 2:
                print("Error: Found invalid gridfunction name: "+gfname)
                sys.exit(1)
            symidx = gfname_without_GFsuffix[-1]
            if int(symidx) < 3:   outstr += "  sym[" + symidx + "] = -1;\n"
        # Rank-2 indexed expression
        elif gfname_without_GFsuffix[len(gfname_without_GFsuffix) - 2].isdigit() == True:
            if len(gfname_without_GFsuffix) > 3 and gfname_without_GFsuffix[len(gfname_without_GFsuffix) - 3].isdigit() == True:
                print("Error: Found a Rank-3 or above gridfunction: "+gfname+", which is at the moment unsupported.")
                print("It should be easy to support this if desired.")
                sys.exit(1)
            symidx0 = gfname_without_GFsuffix[-2]
            if int(symidx0) >= 0: outstr += "  sym[" + symidx0 + "] *= -1;\n"
            symidx1 = gfname_without_GFsuffix[-1]
            if int(symidx1) >= 0: outstr += "  sym[" + symidx1 + "] *= -1;\n"
    else:
        print("Don't know how you got this far with a gridfunction named "+gfname+", but I'll take no more of this nonsense.")
        print("   Please follow best-practices and rename your gridfunction to be more descriptive")
        sys.exit(1)
    outstr += "  SetCartSymVN(cctkGH, sym, \"MaxwellVacuum::" + gfname + "\");\n"
outstr += "}\n"

# Step 2.e.v: Write the function string to file
name    = "MaxwellVacuum_Symmetry_registration.c"
outfile = os.path.join(Ccodesdir,name)
with open(outfile,"w") as file:
    file.write(outstr)
print("Wrote file \"%s\""%outfile)

# Step 2.e.vi: Add function name to the list of functions
#              to be added to the make.code.defn file
make_code_defn_list.append(name)
```

    Wrote file "MaxwellVacuum/src/MaxwellVacuum_Symmetry_registration.c"

## Step 2.f: Boundary condition configuration \[Back to [top](#toc)\]
$$\label{boundary_conditions}$$

```python
# Step 2.f: Register with the boundary conditions thorns.
# Step 2.f.i Set BC type to "none" for all variables
# Since we choose NewRad boundary conditions, we must register all
#   gridfunctions to have boundary type "none". This is because
#   NewRad is seen by the rest of the Toolkit as a modification to the
#   RHSs.

# Step 2.f.ii: This code is based on Kranc's McLachlan/ML_Maxwell/src/Boundaries.cc code.
outstr = """
#include "cctk.h"
#include "cctk_Arguments.h"
#include "cctk_Parameters.h"
#include "cctk_Faces.h"
#include "util_Table.h"
#include "Symmetry.h"

// Set `none` boundary conditions on Maxwell RHSs, as these are set via NewRad.
void MaxwellVacuum_BoundaryConditions_evolved_gfs(CCTK_ARGUMENTS) {

  DECLARE_CCTK_ARGUMENTS;
  DECLARE_CCTK_PARAMETERS;

  CCTK_INT ierr CCTK_ATTRIBUTE_UNUSED = 0;
"""
for gf in evol_gfs_list:
    outstr += """
  ierr = Boundary_SelectVarForBC(cctkGH, CCTK_ALL_FACES, 1, -1, "MaxwellVacuum::"""+gf+"""", "none");
  if (ierr < 0) CCTK_ERROR("Failed to register BC for MaxwellVacuum::"""+gf+"""!");
"""
outstr += """
}

// Set `none` boundary conditions on Maxwell constraints
void MaxwellVacuum_BoundaryConditions_aux_gfs(CCTK_ARGUMENTS) {
  DECLARE_CCTK_ARGUMENTS;
  DECLARE_CCTK_PARAMETERS;

  CCTK_INT ierr CCTK_ATTRIBUTE_UNUSED = 0;

"""
for gf in aux_gfs_list:
    outstr += """
  ierr = Boundary_SelectVarForBC(cctkGH, CCTK_ALL_FACES, cctk_nghostzones[0], -1, "MaxwellVacuum::"""+gf+"""", "none");
  if (ierr < 0) CCTK_ERROR("Failed to register BC for MaxwellVacuum::"""+gf+"""!");
"""
outstr += "}\n"

# Step 2.f.iii: Write the function string to file
name    = "MaxwellVacuum_BoundaryConditions_evolved_gfs.c"
outfile = os.path.join(Ccodesdir,name)
with open(outfile,"w") as file:
    file.write(outstr)
print("Wrote file \"%s\""%outfile)

# Step 2.f.iv: Add function name to the list of functions
#              to be added to the make.code.defn file
make_code_defn_list.append(name)

# Step 2.f.v: Set C code for calling NewRad BCs
#   As explained in lean_public/LeanMaxwellMoL/src/calc_mwev_rhs.F90,
#   the function NewRad_Apply takes the following arguments:
#   NewRad_Apply(cctkGH, var, rhs, var0, v0, radpower),
#     which implement the boundary condition:
#       var  =  var_at_infinite_r + u(r-var_char_speed*t)/r^var_radpower
#  Obviously for var_radpower>0, var_at_infinite_r is the value of
#    the variable at r->infinity. var_char_speed is the propagation
#    speed at the outer boundary, and var_radpower is the radial
#    falloff rate.

outstr = """
#include <math.h>

#include "cctk.h"
#include "cctk_Arguments.h"
#include "cctk_Parameters.h"

void MaxwellVacuum_NewRad(CCTK_ARGUMENTS) {
  DECLARE_CCTK_ARGUMENTS;
  DECLARE_CCTK_PARAMETERS;

"""
for gf in evol_gfs_list:
    var_at_infinite_r = "0.0"
    var_char_speed    = "1.0"
    var_radpower      = "3.0"

    outstr += "  NewRad_Apply(cctkGH, "+gf+", "+gf.replace("GF","")+"_rhsGF, "+var_at_infinite_r+", "+var_char_speed+", "+var_radpower+");\n"
outstr += "}\n"

# Step 2.f.vi: Write the function string to file
name    = "MaxwellVacuum_NewRad.c"
outfile = os.path.join(Ccodesdir,name)
with open(outfile,"w") as file:
    file.write(outstr)
print("Wrote file \"%s\""%outfile)

# Step 2.f.vii: Add function name to the list of functions
#               to be added to the make.code.defn file
make_code_defn_list.append(name)
```

    Wrote file "MaxwellVacuum/src/MaxwellVacuum_BoundaryConditions_evolved_gfs.c"
    Wrote file "MaxwellVacuum/src/MaxwellVacuum_NewRad.c"

## Step 2.g: `NRPy+` banner \[Back to [top](#toc)\]
$$\label{nrpy_banner}$$

```python
# Step 2.g: Function that prints the NRPy+ banner
# Step 2.g.i: Write the function to a string
outstr = """
#include <stdio.h>

void MaxwellVacuum_Banner() {
"""
logostr = logo.print_logo(print_to_stdout=False)
outstr += "    printf(\"MaxwellVacuum: another Einstein Toolkit thorn generated by\\n\");\n"
for line in logostr.splitlines():
    outstr += "    printf(\""+line+"\\n\");\n"
outstr += "}\n"

# Step 2.g.ii: Write the function string to file
name    = "MaxwellVacuum_Banner.c"
outfile = os.path.join(Ccodesdir,name)
with open(outfile,"w") as file:
    file.write(outstr)
print("Wrote file \"%s\""%outfile)

# Step 2.g.iii: Add function name to the list of functions
#               to be added to the make.code.defn file
make_code_defn_list.append(name)
```

    Wrote file "MaxwellVacuum/src/MaxwellVacuum_Banner.c"

## Step 2.h: Thorn configuration files \[Back to [top](#toc)\]
$$\label{ccl_files}$$

### Step 2.h.i: Generating the `make.code.defn` file \[Back to [top](#toc)\]
$$\label{make_code_defn}$$

The `make.code.defn` file specifies which files must be compiled during the ETK build. The general format is:

```
# Files to be compiled
SRCS = file_1.ext \
       file_2.ext \
       file_3.ext
# Subdirectories containing source files
SUBDIRS = subdirectory_1 \
          subdirectory_2
```

or, equivalently,

```
# Files to be compiled
SRCS = file_1.ext file_2.ext file_3.ext
# Subdirectories containing source files
SUBDIRS = subdirectory_1 subdirectory_2
```

```python
# Step 2.c: Thorn configuration files
# Step 2.c.i: Generating the make.code.defn file
# Step 2.c.i.1: Write the make.code.defn file to string
make_code_defn_list.sort()
make_code_defn_string = """
# Main make.code.defn file for thorn MaxwellVacuum

# Source files in this directory
SRCS = """
for i in range(len(make_code_defn_list)):
    fct = make_code_defn_list[i]
    if i == 0:
        make_code_defn_string += fct+" \\\n"
    elif i > 0 and i < len(make_code_defn_list)-1:
        make_code_defn_string += "       "+fct+" \\\n"
    else:
        make_code_defn_string += "       "+fct

# Step 2.c.i.2: Write the make.code.defn file
outfile = os.path.join(Ccodesdir,"make.code.defn")
with open(outfile,"w") as file:
    file.write(make_code_defn_string)
print("Wrote file \"%s\""%outfile)
```

    Wrote file "MaxwellVacuum/src/make.code.defn"

### Step 2.h.ii: Generating the `param.ccl` file \[Back to [top](#toc)\]
$$\label{param_ccl}$$

The `param.ccl` file constains the parameters defined by our thorn and those which we wish to use in our thorn but are defined by other thorns. The general structure of an entry in the `param.ccl` file is:

PARAMETER_TYPE parameter_name "Description of the parameter"
{
  <allowed or disallowed values> :: "Description of the value"
} parameter_default_value

Typical values of `PARAMETER_TYPE` are `CCTK_REAL`, `CCTK_INT`, `CCTK_STRING`, and `BOOLEAN`. The `<allowed or disallowed>` values can actually span multiple lines.

Let us now look at the simplest of examples. Let us define a parameter `key`, which is an `integer`, can have any value, and defaults to 1. This is achieved by doing:

CCTK_INT key "A very simple integer parameter"
{
  *:* :: "Can be anything"
} 1

As mentioned before, our thorn contains two parameters: `FD_order` and `which_system`, which are of types `CCTK_INT` and `CCTK_STRING`, respectively. It also shares parameters with the `Method of Lines` ETK thorn.

```python
%%writefile $Thorndir/param.ccl
# This param.ccl file was automatically generated by NRPy+.
#   You are advised against modifying it directly; instead
#   modify the Python code that generates it.

shares: MethodOfLines

restricted:

CCTK_INT FD_order "Finite-differencing order"
{
 2:2   :: "finite-differencing order = 2"
 4:4   :: "finite-differencing order = 4"
 6:6   :: "finite-differencing order = 6"
 8:8   :: "finite-differencing order = 8"
} 4

CCTK_STRING which_system "Which system to evolve"
{
    "SystemI"  :: "Evolve system I"
    "SystemII" :: "Evolve system II"
} "SystemII"
```

    Writing MaxwellVacuum/param.ccl

### Step 2.h.iii: Generating the `interface.ccl` file \[Back to [top](#toc)\]
$$\label{interface_ccl}$$

We will now generate the `interface.ccl` file for the `MaxwellVacuum` thorn. This file specifies how the thorn interacts with the other thorns in the toolkit. For example, if we want to implement a function that can be used by other thorns, then we would include its prototype to this file. Similarly, if you want to use a function or parameter defined by another thorn, you have to mention that in this file.

This is also the place to define the gridfunctions that we need for our thorn, i.e. $\bigl(E^{i},A^{i},\Phi,\Gamma,\mathcal{C},\mathcal{G}\bigr)$.

The [official Einstein Toolkit documentation](https://einsteintoolkit.org/usersguide/UsersGuide.html#x1-179000D2.2) defines what must/should be included in an `interface.ccl` file in detail.

```python
%%writefile $Thorndir/interface.ccl
# This interface.ccl file was automatically generated by NRPy+.
#   You are advised against modifying it directly; instead
#   modify the Python code that generates it.

# With "implements", we give our thorn its unique name.
implements: MaxwellVacuum

# By "inheriting" other thorns, we tell the Toolkit that we
#   will rely on variables/function that exist within those
#   functions.
inherits: Boundary grid MethodofLines

# Needed functions and #include's:
USES INCLUDE: Symmetry.h
USES INCLUDE: Boundary.h

# Needed Method of Lines function
CCTK_INT FUNCTION MoLRegisterEvolvedGroup(CCTK_INT IN EvolvedIndex, CCTK_INT IN RHSIndex)
REQUIRES FUNCTION MoLRegisterEvolvedGroup

# Needed Boundary Conditions function
CCTK_INT FUNCTION GetBoundarySpecification(CCTK_INT IN size,                  \
                                           CCTK_INT OUT ARRAY nboundaryzones, \
                                           CCTK_INT OUT ARRAY is_internal,    \
                                           CCTK_INT OUT ARRAY is_staggered,   \
                                           CCTK_INT OUT ARRAY shiftout)
USES FUNCTION GetBoundarySpecification

CCTK_INT FUNCTION SymmetryTableHandleForGrid(CCTK_POINTER_TO_CONST IN cctkGH)
USES FUNCTION SymmetryTableHandleForGrid

CCTK_INT FUNCTION Boundary_SelectVarForBC(CCTK_POINTER_TO_CONST IN GH, \
                                          CCTK_INT IN faces,           \
                                          CCTK_INT IN boundary_width,  \
                                          CCTK_INT IN table_handle,    \
                                          CCTK_STRING IN var_name,     \
                                          CCTK_STRING IN bc_name)
USES FUNCTION Boundary_SelectVarForBC

# Needed for EinsteinEvolve/NewRad outer boundary condition driver:
CCTK_INT FUNCTION NewRad_Apply(CCTK_POINTER_TO_CONST IN cctkGH, \
                               CCTK_REAL ARRAY IN var,          \
                               CCTK_REAL ARRAY INOUT rhs,       \
                               CCTK_REAL IN var0,               \
                               CCTK_REAL IN v0,                 \
                               CCTK_INT IN radpower)
REQUIRES FUNCTION NewRad_Apply

# Tell the Toolkit that we want all gridfunctions
#    to be visible to other thorns by using
#    the keyword "public". Note that declaring these
#    gridfunctions *does not* allocate memory for them;
#    that is done by the schedule.ccl file.

public:
CCTK_REAL evol_variables type = GF Timelevels=3
{
    AU0GF,AU1GF,AU2GF,EU0GF,EU1GF,EU2GF,GammaGF,PhiGF
} "Maxwell evolved gridfunctions"

CCTK_REAL evol_variables_rhs type = GF Timelevels=1 TAGS='InterpNumTimelevels=1 prolongation="none"'
{
    AU0_rhsGF,AU1_rhsGF,AU2_rhsGF,EU0_rhsGF,EU1_rhsGF,EU2_rhsGF,Gamma_rhsGF,Phi_rhsGF
} "right-hand-side storage for Maxwell evolved gridfunctions"

CCTK_REAL aux_variables type = GF Timelevels=3
{
    CGF,GGF
} "Auxiliary gridfunctions for Maxwell diagnostics"
```

    Writing MaxwellVacuum/interface.ccl

### Step 2.h.iv: Generating the `schedule.ccl` file \[Back to [top](#toc)\]
$$\label{schedule_ccl}$$

We will now generate the `schedule.ccl` file for the `MaxwellVacuum` thorn. This file specifies:

* Which variables/gridfunctions we need to allocate memory to
* What function calls are performed by the toolkit scheduler and the order in which these function calls happen
* Function scope (global or local)
* (Optional for now) which gridfunctions our functions read from and write to

Official documentation on constructing ETK `schedule.ccl` files is found [here](https://einsteintoolkit.org/usersguide/UsersGuide.html#x1-187000D2.4).

```python
%%writefile $Thorndir/schedule.ccl
# This schedule.ccl file was automatically generated by NRPy+.
#   You are advised against modifying it directly; instead
#   modify the Python code that generates it.

# Next allocate storage for all 3 gridfunction groups used in MaxwellVacuum
STORAGE: evol_variables[3]     # Evolution variables
STORAGE: evol_variables_rhs[1] # Variables storing right-hand-sides
STORAGE: aux_variables[3]      # Diagnostics variables

# The following scheduler is based on Lean/LeanMaxwellMoL/schedule.ccl

schedule MaxwellVacuum_Banner at STARTUP
{
  LANG: C
  OPTIONS: meta
} "Output ASCII art banner"

schedule MaxwellVacuum_Symmetry_registration at BASEGRID
{
  LANG: C
  OPTIONS: Global
} "Register symmetries, the CartGrid3D way."

schedule MaxwellVacuum_zero_rhss at BASEGRID after MaxwellVacuum_Symmetry_registration
{
  LANG: C
} "Idea from Lean: set all rhs functions to zero to prevent spurious nans"

# MoL: registration

schedule MaxwellVacuum_MoL_registration in MoL_Register
{
  LANG: C
  OPTIONS: META
} "Register variables for MoL"

# MoL: compute RHSs, etc

schedule MaxwellVacuum_RHSs in MoL_CalcRHS as MaxwellVacuum_RHS
{
  LANG: C
} "MoL: Evaluate Maxwell RHSs"

schedule MaxwellVacuum_NewRad in MoL_CalcRHS after MaxwellVacuum_RHS
{
  LANG: C
} "NewRad boundary conditions, scheduled right after RHS eval."

schedule MaxwellVacuum_BoundaryConditions_evolved_gfs in MoL_PostStep
{
  LANG: C
  OPTIONS: LEVEL
  SYNC: evol_variables
} "Apply boundary conditions and perform AMR+interprocessor synchronization"

schedule GROUP ApplyBCs as MaxwellVacuum_ApplyBCs in MoL_PostStep after MaxwellVacuum_BoundaryConditions_evolved_gfs
{
} "Group for applying boundary conditions"

# Compute divergence and Gamma constraints

schedule MaxwellVacuum_constraints in MoL_PseudoEvolution
{
  LANG: C
  OPTIONS: Local
} "Compute Maxwell (divergence and Gamma) constraints"
```

    Writing MaxwellVacuum/schedule.ccl

# Step 3: Part III - Code validation (Courtesy: Terrence Pierre Jacques) \[Back to [top](#toc)\]
$$\label{code_validation}$$

**Code tests adopting fourth-order finite differencing, coupled to 2nd order Iterative Crank-Nicholson method-of-lines for time integration**

Inside the directory *`test_results/`* are the files used for this convergence test:
**maxwell_toroidaldipole-0.125_OB4.par & maxwell_toroidaldipole-0.0625_OB4.par** : ETK parameter files needed for performing the tests. These parameter files set up a toroidal dipole field propagating in a 3D numerical grid that extends from -4. to +4. along the x-, y-, and z-axes (in units of $c=1$). The parameter files are identical, except the latter has grid resolution that is twice as high as the former (so the errors should drop in the higher resolution case by a factor of $2^2$, since we adopt fourth-order finite differencing coupled to 2nd order Iterative Crank-Nicholson time integration.)

**Second-order code validation test results:**

The plot below shows the discrepancy between numerical and exact solutions to x-components of system I $\vec{E}$ and $\vec{A}$ at two different resolutions, at t = 2.0 (to not have errors at the boundary propagate too far inward): dashed is low resolution ($\Delta x_{\rm low}=0.125$) and solid is high resolution ($\Delta x_{\rm high}=0.0625$). Since this test adopts **fourth**-order finite differencing for spatial derivatives and **second**-order Iterative Crank-Nicholson timestepping, we would expect this error to drop by a factor of approximately $(\Delta x_{\rm low}/\Delta x_{\rm high})^2 = (0.125/0.0625)^2 = 2^2=4$ when going from low to high resolution, and after rescaling the error in the low-resolution case, we see that indeed it overlaps the high-resolution result quite nicely, confirming second-order convergence. We note that we also observe convergence for all other evolved variables (in both systems) with a nonzero exact solutions.

```python
from IPython.display import Image
Image("test_results/Ex-convergence.png", width=700, height=700)
```

```python
Image("test_results/Ax-convergence.png", width=700, height=700)
```

Because System I is weakly hyperbolic (see [Tutorial-VacuumMaxwell_formulation_Cartesian](Tutorial-VacuumMaxwell_formulation_Cartesian.ipynb) for more discussion), zero speed error nodes of the constraint violation sit on our numerical grid, adding to the errors of our evolution variables. In contrast, System II is strongly hyperbolic, and the error nodes propagate away at the speed of light, leading to more stable evolution of the evolution variables. The plot below demostrates the qualitative behavior for both systems.

Contrast these plots to Figure 1 in [Knapp, Walker, & Baumgarte (2002)](https://arxiv.org/abs/gr-qc/0201051); we observe excellent qualitative agreement.

```python
Image("test_results/constraintviolation.png")
```

# Step 4: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[ETK_Workshop_2021-NRPy_tutorial.pdf](ETK_Workshop_2021-NRPy_tutorial.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
# Step 4: Generate a PDF version of this tutorial notebook
# Step 4.a: First copy the latex_nrpy_style.tplx from the
#           nrpy_core directory to the base directory
src_file = os.path.join(nrpy_core_dir,"latex_nrpy_style.tplx")
dst_file = os.path.join(base_dir     ,"latex_nrpy_style.tplx")
shutil.copyfile(src_file,dst_file)

# Step 4.b: Now generate the PDF file
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("ETK_Workshop_2021-NRPy_tutorial")

# Step 4.c: Clean up by removing the latex_nrpy_style.tplx from
#           the base directory
cmd.delete_existing_files(dst_file)
```

    Created ETK_Workshop_2021-NRPy_tutorial.tex, and compiled LaTeX file to PDF
        file ETK_Workshop_2021-NRPy_tutorial.pdf