# Start-to-Finish Example: Applying Boundary Conditions in Curvilinear Coordinates in Three Dimensions

## Author: Zach Etienne

## This module documents and validates basic boundary condition algorithms for curvilinear coordinate systems (e.g., Spherical, Cylindrical), based on prescription in the [SENR/NRPy+ paper](https://arxiv.org/abs/1712.07658).

## Introduction: Applying boundary conditions in curvilinear coordinates
$$\label{intro}$$

Here is a list of four challenges we face when solving PDEs in curvilinear coordinates:

1. [Challenge 1](#innerouterbcs): Unlike ordinary Cartesian coordinate grids, not all boundary points on uniform curvilinear coordinate grids are outer boundary points.
1. [Challenge 2](#coordinversion): Figuring out the locations to which boundary points map requires a coordinate inversion, but some coordinate systems are not easily invertible.
1. [Challenge 3](#parity): Tensors and vectors in curvilinear coordinates can change directions across "inner" boundaries, changing sign as a result.
1. [Challenge 4](#coordsingularities): Coordinate singularities can appear, causing a stiffening or divergence of terms in PDEs.

### Challenge 1: Inner versus outer boundary points
$$\label{innerouterbcs}$$

Unlike Cartesian coordinates, the boundaries of our grid in generic curvilinear coordinates are not all outer boundaries.

Consider first a computational grid in Cartesian coordinates, with $(x_0,x_1,x_2)=(x,y,z)$, that is *uniform* (i.e., with $\Delta x$, $\Delta y$, and $\Delta z$ all set to constant values). This grid may extend over arbitrary coordinate ranges $x_i \in [x_{i, \rm min},x_{i, \rm max}]$.

By contrast, consider now a uniform grid in spherical coordinates $(x_0,x_1,x_2)=(r,\theta,\phi)$ with constant spacing $\Delta r$, $\Delta \theta$, and $\Delta \phi$ between grid points in $r$, $\theta$, and $\phi$, respectively. Further, let's assume that these grids span all possible values of $\theta$ and $\phi$, with $r=0$ included in the domain. Then our numerical domain must satisfy the following relations

+ $x_0 = r \in [0,{\rm RMAX}]$,
+ $x_1 = \theta \in [0,\pi]$, and
+ $x_2 = \phi \in [-\pi,\pi]$. (Notice how we do not choose $x_2= \phi \in [0,2\pi]$ so that our conversion from Cartesian to spherical coordinates is compatible with the output range from the ${\rm atan2}(y,x)$ function: $\phi={\rm atan2}(y,x)\in[-\pi,\pi]$.)

Notice that unlike Cartesian coordinates, the boundaries of this numerical grid in spherical coordinates are not all outer boundaries. For example, data stored at $\phi=-\pi$ will be identical to data at $\phi=\pi$, *regardless of $r$ or $\theta$*. I.e., $\phi$ satisfies *periodic* boundary conditions only. Further, $\theta=0$ presents a more complicated boundary condition, in which points with negative $\theta$ map to points with $|\theta|$ but at an angle of $\phi\to \phi+\pi$. Finally, negative $r$ points will map to postive $r$ points on the other side of the origin. We call these boundaries *inner* boundaries, as they generally map to other points in the interior (as opposed to the outer boundaries) of the grid.

As we generally cannot apply an outer boundary condition to the inner boundaries, these boundaries will need to be treated differently.

On our numerical grids, this poses some difficulty, as finite difference derivatives we compute within the numerical domain require that the grid be extended beyond the domain boundaries. In spherical coordinates, this means that we need additional grid points at, e.g.,  $r<0$, $\theta<0$, and $\phi>\pi$, just to name a few. Whether they be on outer or inner boundaries, we call grid points in the extended region *ghost zones*.

Numerical grids of $N$th order accuracy generally possess $N/2$ ghost zone points in the boundary regions (i.e., $x_i < x_{i,\rm min}$ and $x_i > x_{i, \rm max}$). While in Cartesian coordinates, these ghost zone points map to regions outside the grid domain $x_i \in [x_{i, \rm min},x_{i, \rm max}]$, in spherical coordinates, *most* ghost zone points map to regions *inside* the grid domain. For example, for some $\tilde{r}\in [0,{\rm RMAX}]$ and $\tilde{\theta}\in[0,\pi]$, the ghost zone point $(\tilde{r},\tilde{\theta},2\pi+\Delta \phi/2)$ would map to the interior point $(\tilde{r},\tilde{\theta},\Delta \phi/2)$ because the $\phi$ coordinate is periodic. Thus when given a ghost zone point in some arbitrary curvilinear coordinate system, we are faced with the problem of addressing the following two questions:
1. Does a given ghost point map to an interior point, or is it an outer boundary point (i.e., a point exterior to the domain)?
1. If the ghost zone point maps to an interior point, to which interior point does it map?

### Challenge 2: Inverting coordinates
$$\label{coordinversion}$$

Coordinate systems within NRPy+ are generally Spherical-like, Cylindrical-like, SymTP-like (where SymTP is a prolate-spheroidal coordinate system), or Cartesian-like. For example, SinhSphericalv2 coordinates are exactly the same as Spherical coordinates, except we choose an odd function for the radial coordinate $r$ as a function of $x_0$:

$$
r(x_0) = {\rm AMPL} \left[ {\rm const\_dr} x_0 + \sinh\left(\frac{x_0}{\rm SINHW}\right) / \sinh\left(\frac{1}{\rm SINHW}\right) \right].
$$

While this coordinate choice exhibits nice properties for certain cases, the function $x_0(r)$ is not a closed-form expression. Thus finding the mapping of ghost zone points in the radial direction would require a root finder.

*Is there an easier way of dealing with this problem than with a root finder?*

### Challenge 3: Parity: changes of direction in vectors and tensors across inner boundaries
$$\label{parity}$$

When applying inner boundary conditions to vectors and tensors, we must consider how the direction or *parity* of vector and tensor components change across the inner boundary.

Suppose we have a vector $v^\rho$ defined at ghost zone $(-\rho,\phi,z)$ ($\rho>0$) in cylindrical coordinates. This will map to an interior point at $(\rho,\phi+\pi,z)$. At this point, the direction of the $\hat{\rho}$ unit vector flips sign. Thus we cannot simply set the value of $v^\rho$ to the value it possesses at interior point $(\rho,\phi+\pi,z)$; that would result in a sign error. Instead we have
\begin{align}
v^\rho(-\rho,\phi,z)&=-v^\rho(\rho,\phi+\pi,z) \\
&= \mathbf{e}^\rho\left(-\rho,\phi,z\right) \cdot \mathbf{e}^\rho\left(\rho,\phi+\pi,z\right)v^\rho(\rho,\phi+\pi,z),
\end{align}
where $\mathbf{e}^\rho\left(\rho,\phi,z\right)$ is the $\rho$ unit vector evaluated at point $(\rho,\phi,z)$, and $\mathbf{e}^\rho\left(-\rho,\phi,z\right) \cdot \mathbf{e}^\rho\left(\rho,\phi+\pi,z\right)$ is the dot product of the two unit vectors, which must evaluate to $\pm 1$ (i.e., the **parity**). Contrast this with scalars, which do not possess a sense of direction/parity.

### Challenge 4: Coordinate singularities
$$\label{coordsingularities}$$

Most non-Cartesian, orthogonal coordinate systems (like spherical coordinates) possess *coordinate singularities*.

For example, coordinate singularities in spherical coordinates lie along $\theta=0$ and $\theta=\pi$; these are points where the coordinate system focuses to a single point. For example, the coordinate singularity at the North Pole is the reason why all directions are south there. Critically, these singularities manifest as points where the reference metric or its inverse crosses through zero or diverges to $\infty$. As we derived in a [previous module](Tutorial-ScalarWaveCurvilinear.ipynb), the Laplacian in spherical polar coordinates takes the form
$$
\nabla^2 u = \partial_r^2 u + \frac{1}{r^2} \partial_\theta^2 u + \frac{1}{r^2 \sin^2 \theta} \partial_\phi^2 u +  \frac{2}{r} \partial_r u + \frac{\cos\theta}{r^2 \sin\theta} \partial_\theta u,
$$
which diverges at $r=0$ and $\sin\theta=0-$precisesly at the $\theta=0$ and $\theta=\pi$ coordinate singularity.

To avoid this divergence, we simply choose that our numerical grids be **cell-centered**. For example, if we choose a numerical grid with 3 outer ghost zone points (needed due to 6th-order-accurate centered finite differencing), and we want the grid interior to be sampled with ${\rm Nr}\times{\rm Ntheta}\times{\rm Nphi}$ grid points, then the cell-centered grid points can be indexed via three *even* integers $\left(\rm{i0}\in[0,{\rm Nr}+6),\rm{i1}\in[0,{\rm Ntheta}+6),\rm{i2}\in[0,{\rm Ntheta}+6)\right)$:
+ $r_{\rm i0}      = r_{\rm min}      + \left({\rm i0} + \frac{1}{2}\right)\Delta r$, where $r_{\rm min}=\left(-3+\frac{1}{2}\right)\Delta r$;
+ $\theta_{\rm i1} = \theta_{\rm min} + \left({\rm i1} + \frac{1}{2}\right)\Delta \theta$, where $\theta_{\rm min}= \left(-3+\frac{1}{2}\right)\Delta \theta$; and
+ $\phi_{\rm i2}   = \phi_{\rm min}   + \left({\rm i2} + \frac{1}{2}\right)\Delta \phi$, where $\phi_{\rm min}=-\pi+ \left(-3+\frac{1}{2}\right)\Delta \phi$.

**Exercise to student**: Given the prescription above, why do the integers $\left(\rm{i0},\rm{i1},\rm{i2}\right)$ need to be even?

As Laplacians like these appear on the right-hand sides of, e.g., the scalar wave equation in curvilinear coordinates, we still have a problem of some terms becoming quite large as the coordinate singularity is approached. This issue manifests as a stiffening of the PDE, requiring that we be very careful about the precise [Method of Lines](Tutorial-Method_of_Lines-C_Code_Generation.ipynb) timestepping algorithm used. See [Cordero-Carrión & Cerdá-Durán](https://arxiv.org/abs/1211.5930) for information on dealing with this subtlety in a second-order Runge-Kutta Method of Lines context; it was later found that the standard RK4 method maintain stable solutions to PDEs affected by this sort of stiffening.

The above discussion focuses primarily on scalar fields. However, when solving PDEs involving vectors and tensors, the vectors and tensors themselves can exhibit divergent behavior at coordinate singularities. The good news is, this singular behavior is well-understood in terms of the scale factors of the reference metric, enabling us to define rescaled version of these quantities that are well behaved (so that, e.g., they can be finite differenced).

For example, given a smooth vector *in a 3D Cartesian basis* $\bar{\Lambda}^{i}$, all components $\bar{\Lambda}^{x}$, $\bar{\Lambda}^{y}$, and $\bar{\Lambda}^{z}$ will be smooth (by assumption). When changing the basis to spherical coordinates (applying the appropriate Jacobian matrix transformation), we will find that since $\phi = \arctan(y/x)$, $\bar{\Lambda}^{\phi}$ is given by

\begin{align}
\bar{\Lambda}^{\phi} &= \frac{\partial \phi}{\partial x} \bar{\Lambda}^{x} +
\frac{\partial \phi}{\partial y} \bar{\Lambda}^{y} +
\frac{\partial \phi}{\partial z} \bar{\Lambda}^{z} \\
&= -\frac{y}{\sqrt{x^2+y^2}} \bar{\Lambda}^{x} +
\frac{x}{\sqrt{x^2+y^2}} \bar{\Lambda}^{y} \\
&= -\frac{y}{r \sin\theta} \bar{\Lambda}^{x} +
\frac{x}{r \sin\theta} \bar{\Lambda}^{y}.
\end{align}

Thus $\bar{\Lambda}^{\phi}$ diverges at all points where $r\sin\theta=0$ due to the $\frac{1}{r\sin\theta}$ that appear in the Jacobian transformation.

This divergence might pose no problem on cell-centered grids that avoid $r \sin\theta=0$, except that the BSSN equations require that *first and second derivatives* of these quantities be taken. Usual strategies for numerical approximation of these derivatives (e.g., finite difference methods) will "see" these divergences and errors generally will not drop to zero with increased numerical sampling of the functions at points near where the functions diverge.

However, notice that if we define $\lambda^{\phi}$ such that

$$\bar{\Lambda}^{\phi} = \frac{1}{r\sin\theta} \lambda^{\phi},$$

then $\lambda^{\phi}$ will be smooth as well.

Avoiding such singularities can be generalized to arbitrary coordinate systems, so long as $\lambda^i$ is defined as:

$$\bar{\Lambda}^{i} = \frac{\lambda^i}{\text{scalefactor[i]}} ,$$

where scalefactor\[i\] is the $i$th scale factor in the given coordinate system. This idea can be extended to covariant (lowered-index) vectors and arbitrary tensors, as described in [the BSSN quantities tutorial notebook](Tutorial-BSSN_quantities.ipynb#rescaling_tensors).

**In summary, Challenge 3 is addressed via a combination of cell-centered grids, tensor rescaling, and a stable Method of Lines time stepping algorithm. This tutorial notebook will therefore focus on addressing Challenges 1 and 2, which, coincidentally, are addressed via an appropriate boundary condition algorithm.**

# Table of Contents
$$\label{toc}$$

This notebook is organized as follows

1. [Step 1](#basic_algorithm): Overview of boundary condition algorithm in curvilinear coordinates
    1. [Step 1.a](#challenge1): Addressing Challenge 1: Distinguishing inner from outer boundary points
    1. [Step 1.b](#challenge2): Addressing Challenge 2: Eigen-Coordinate systems
    1. [Step 1.c](#challenge3): Addressing Challenge 3: Applying parity conditions to arbitrary-rank tensors
1. [Step 2](#ccode_bc_struct): `bc_struct` and other C data structures used for storing boundary condition information
1. [Step 3](#nrpycodegen): NRPy+-based C code generation for parity conditions
    1. [Step 3.a](#dotproducts): Set up unit-vector dot products (=parity) for each of the 10 parity condition types, store to `parity_conditions_symbolic_dot_products.h`
    1. [Step 3.b](#set_parity_type): Set parity type for each gridfunction, based on the digits at the end of its name, output to `dirname+gridfunction_defines.h`
1. [Step 4](#set_up__bc_gz_map_and_parity_condns): `set_up__bc_gz_map_and_parity_condns()`: C function for distinguishing inner from outer boundary points, and setting parity conditions
1. [Step 5](#set_bc_struct): `set_bcstruct()`: Using information from `set_up__bc_gz_map_and_parity_condns()` as input, set `bcstruct`
1. [Step 6](#bcstruct_c_code_driver): `driver_bcstruct.h`: C code driver for declaring `bc_struct` data type, the `bcstruct` instance of said data type, and calling `set_up__bc_gz_map_and_parity_condns()` and `set_bcstruct()` to fill `bcstruct`
1. [Step 7](#apply_bcs_curvilinear): `apply_bcs_curvilinear()` C function: quickly apply boundary and parity conditions with information from `bcstruct`
1. [Step 8](#start2finish): `CurviBC_Playground.c`: Start-to-Finish C code module for testing & validating curvilinear boundary conditions
    1. [Step 8.a](#ccode_numgridgen): Generate C code for generating numerical grids
    1. [Step 8.b](#register_gfs): Register gridfunctions of all 10 parity types in NRPy+; output gridfunction aliases to `CurviBoundaryConditions/gridfunction_defines.h`
    1. [Step 8.c](#validate): Set up test data for Curvilinear Boundary Conditions code validation
    1. [Step 8.d](#mainc): `CurviBC_Playground.c` : The Main C Code
1. [Step 9](#senr_compare): Validation: Compare with original SENR results
1. [Step 10](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Step 1: Overview of boundary condition algorithm in curvilinear coordinates \[Back to [top](#toc)\]
$$\label{basic_algorithm}$$

Here we **review** the basic algorithm for addressing Challenges [1](#innerouterbcs) [2](#coordinversion), and [3](#parity) discussed in the [Introduction](#intro) above.

The algorithm itself is **implemented** as C code in Steps [**2**](#bc_struct) (data structures), [**3**](#set_up__bc_gz_map_and_parity_condns) (searching entire grid for inner and outer boundary points, and setting parities), [**4**](#set_bcstruct) (setting data structures for quick and efficient implementation of outer boundaries), and [**5**](#apply_bcs_curvilinear) (function to apply inner & outer boundary conditions).

## Step 1.a: Addressing Challenge 1: Distinguishing inner from outer boundary points \[Back to [top](#toc)\]
$$\label{challenge1}$$

At each ghost zone grid point $\mathbf{d}_{\rm gz}=(x_0,x_1,x_2)$, we will do the following:

1. Evaluate the Cartesian coordinate $\left(x(x_0,x_1,x_2),y(x_0,x_1,x_2),z(x_0,x_1,x_2)\right)$, corresponding to this grid point. Then evaluate the inverse $\mathbf{d}_{\rm new}=\left(x_0(x,y,z),x_1(x,y,z),x_2(x,y,z)\right)$.
    1. If $\mathbf{d}_{\rm new} \ne \mathbf{d}_{\rm gz}$, then the ghost zone grid point maps to a point in the grid interior, *which is exactly the case described in the above section*. To distinguish this case from an "outer boundary condition", we shall henceforth refer to it variously as an application of an "interior", "inner", or "parity" boundary condition.
    1. Ghost zone points for which $\mathbf{d}_{\rm new} \equiv \mathbf{d}_{\rm gz}$ are on the outer boundary of the grid, and standard outer boundary conditions should be applied.

In detail, the algorithm is as follows:

1. Convert the coordinate $(x_0,x_1,x_2)$ for the ghost zone point to Cartesian coordinates $\left(x(x_0,x_1,x_2),y(x_0,x_1,x_2),z(x_0,x_1,x_2)\right)$. For example, if we choose ordinary spherical coordinates $(x_0,x_1,x_2)=(r,\theta,\phi)$, then
    + $x(r,\theta,\phi) = r \sin(\theta) \cos(\phi) = x_0 \sin(x_1) \cos(x_2)$
    + $y(r,\theta,\phi) = r \sin(\theta) \sin(\phi) = x_0 \sin(x_1) \sin(x_2)$
    + $z(r,\theta,\phi) = r \cos(\theta)            = x_0 \cos(x_1)$
1. Once we have $(x,y,z)$, we then find the corresponding value $(x_0,x_1,x_2)_{\rm in/OB}=(r,\theta,\phi)_{\rm in/OB}$ *in the grid interior or outer boundary*, via the simple inverse formula:
    + $r_{\rm in/OB}      = x_{0, \rm in/OB} = \sqrt{x^2+y^2+z^2} \in [0,\infty)$
    + $\theta_{\rm in/OB} = x_{1, \rm in/OB} = {\rm acos}\left(\frac{z}{\sqrt{x^2+y^2+z^2}}\right) \in [0,\pi]$
    + $\phi_{\rm in/OB}   = x_{2, \rm in/OB} = {\rm atan2}(y,x) \in [-\pi,\pi]$ [Wikipedia article on atan2()](https://en.wikipedia.org/w/index.php?title=Atan2&oldid=859313982)

1. If $(x_0,x_1,x_2)_{\rm in/OB}$ is the same as the original $(x_0,x_1,x_2)$, then we know $(x_0,x_1,x_2)$ is an outer boundary point (in spherical coordinates, at $r>{\rm RMAX}$), and we store `(i0,i1,i2)`$_{\rm in/OB} = (-1,-1,-1)$. Otherwise, we know that $(x_0,x_1,x_2)$ maps to some interior point at index `(i0,i1,i2)`, which we store:
    + $\rm{i0}_{\rm in/OB}=\frac{r_{\rm in/OB}      -      r_{\rm min}}{\Delta r}      - \frac{1}{2}$
    + $\rm{i1}_{\rm in/OB}=\frac{\theta_{\rm in/OB} - \theta_{\rm min}}{\Delta \theta} - \frac{1}{2}$
    + $\rm{i2}_{\rm in/OB}=\frac{\phi_{\rm in/OB}   -   \phi_{\rm min}}{\Delta \phi}   - \frac{1}{2}$

1. When updating a ghost zone point `(i0,i1,i2)` in the domain exterior, if the corresponding `(i0,i1,i2)`$_{\rm in/OB}$ was set to $(-1,-1,-1)$, then we apply outer boundary conditions. Otherwise, we simply copy the data from the interior point at `(i0,i1,i2)`$_{\rm in/OB}$ to `(i0,i1,i2)`.

Following the prescription in the [SENR/NRPy+ paper](https://arxiv.org/abs/1712.07658), we will implement curvilinear boundary conditions for rank-0, rank-1, and symmetric rank-2 tensors in three dimensions; as this is the same dimension and highest rank needed for BSSN.

## Step 1.b: Addressing Challenge 2: Eigen-Coordinate Systems \[Back to [top](#toc)\]
$$\label{challenge2}$$

Suppose we were to rewrite the spherical coordinate $r$ as an arbitrary odd function of $x_0$ instead of $x_0$ itself. In that case, $r(-x_0)=-r(x_0)$, and all parity conditions remain unchanged. However the inverse function, $x_0(r)$, may not be writable as a closed-form expression, requiring a Newton-Raphson root finder to find the appropriate boundary mappings.

To greatly simplify the algorithm in the case of arbitrary $r(x_0)$ in Spherical-like coordinates, or $\rho(x_0)$ or $z(x_2)$ in Cylindrical-like coordinates, we note that the coordinate mappings *and* parities for all Spherical-like coordinate systems are identical to the mappings and parities for ordinary Spherical coordinates. The same holds true for Cylindrical-like and SymTP-like coordinate systems. Thus so long as we know the correct "Eigen-Coordinate system" (i.e., Spherical in the case of SinhSpherical or SinhSphericalv2; Cylindrical in the case of SinhCylindrical; SymTP in the case of SinhSymTP; etc.) there is no need for a Newton-Raphson root finder to set up the boundary conditions.

## Step 1.c: Addressing Challenge 3: Applying parity conditions to arbitrary-rank tensors \[Back to [top](#toc)\]
$$\label{challenge3}$$

Above we presented the strategy for applying parity boundary conditions to a single component of a vector. Here we outline the generic algorithm for arbitrary-rank tensors.

Continuing the discussion from the previous section, we assume $\mathbf{d}_{\rm new} \ne \mathbf{d}_{\rm gz}$ (otherwise we would apply the *outer* boundary condition algorithm). Next suppose we are given a generic rank-$N$ tensor ($N>0$).

1. The first component of the rank-$N$ tensor corresponds to some direction with unit vector $\mathbf{e}^i$; e.g., $v^r$ corresponds to the $\mathbf{e}^r$ direction. Compute the dot product of the unit vector $\mathbf{e}^i$ evaluated at points $\mathbf{d}_{\rm gz}$ and $\mathbf{d}_{\rm new}$. Define this dot product as $P_1$ ("$P$" for "parity"):
$$
P_1 = \mathbf{e}^i\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^i\left(\mathbf{d}_{\rm new}\right).
$$
1. $P_1$ will take the value of $\pm 1$, depending on the unit-vector direction and the points $\mathbf{d}_{\rm gz}$ and $\mathbf{d}_{\rm new}$
1. Repeat the above for the remaining components of the rank-$N$ tensor $j\in \{2,3,...,N\}$, storing each $P_j$.
1. The tensor mapping from $\mathbf{d}_{\rm gz}$ to $\mathbf{d}_{\rm new}$ for this tensor $T^{ijk...}_{mnp...}$ will be given by
$$
T^{ijk...}_{lmn...}(x_0,x_1,x_2)_{\rm gz} = \prod_{\ell=1}^N P_\ell T^{ijk...}_{mnp...}(x_0,x_1,x_2)_{\rm new}.
$$

In this formulation of BSSN, we only need to deal with rank-0, rank-1, and *symmetric* rank-2 tensors. Further, our basis consists of 3 directions, so there are a total of
+ 1 parity condition (the trivial +1) for scalars (rank-0 tensors)
+ 3 parity conditions for all rank-1 tensors (corresponding to each direction)
+ 6 parity conditions for all *symmetric* rank-2 tensors (corresponding to the number of elements in the lower or upper triangle of a $3\times3$ matrix, including the diagonal)

Thus we must keep track of the behavior of **10 separate parity conditions**, which can be evaluated once the numerical grid has been set up, for all time. The following Table outlines the correct conditions for each:

The appropriate dot products determining parity condition are assigned to each gridfunction based on the following numbering:

Tensor type | Parity type | Dot product(s) determining parity condition (see equation above)
--- | --- | ---
Scalar (Rank-0 tensor) | 0 | (*none*)
Rank-1 tensor in **i0** direction | 1 | $\mathbf{e}^0\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^0\left(\mathbf{d}_{\rm new}\right)$
Rank-1 tensor in **i1** direction | 2 | $\mathbf{e}^1\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^1\left(\mathbf{d}_{\rm new}\right)$
Rank-1 tensor in **i2** direction | 3 | $\mathbf{e}^2\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^2\left(\mathbf{d}_{\rm new}\right)$
Rank-2 tensor in **i0-i0** direction | 4 | $\left[\mathbf{e}^0\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^0\left(\mathbf{d}_{\rm new}\right)\right]^2 = 1$
Rank-2 tensor in **i0-i1** direction | 5 | $\left[\mathbf{e}^0\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^0\left(\mathbf{d}_{\rm new}\right)\right]\left[\mathbf{e}^1\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^1\left(\mathbf{d}_{\rm new}\right)\right]$
Rank-2 tensor in **i0-i2** direction | 6 | $\left[\mathbf{e}^0\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^0\left(\mathbf{d}_{\rm new}\right)\right]\left[\mathbf{e}^2\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^2\left(\mathbf{d}_{\rm new}\right)\right]$
Rank-2 tensor in **i1-i1** direction | 7 | $\left[\mathbf{e}^1\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^1\left(\mathbf{d}_{\rm new}\right)\right]^2 = 1$
Rank-2 tensor in **i1-i2** direction | 8 | $\left[\mathbf{e}^1\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^1\left(\mathbf{d}_{\rm new}\right)\right]\left[\mathbf{e}^2\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^2\left(\mathbf{d}_{\rm new}\right)\right]$
Rank-2 tensor in **i2-i2** direction | 9 | $\left[\mathbf{e}^2\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^2\left(\mathbf{d}_{\rm new}\right)\right]^2 = 1$

In the following few steps, we will document the data structures for and implementation of this boundary condition algorithm.

# Step 2:  `bc_struct`: Data structure for storing boundary condition information \[Back to [top](#toc)\]
$$\label{ccode_bc_struct}$$

Here we define `bc_struct`, a C data structure that stores all information needed to apply boundary conditions at all boundary points.

The information needed to fill a ghost zone depends on whether it exists on an inner or an outer boundary, and the `bc_struct` data structure is constructed to account for this fact:

```C
typedef struct __bcstruct__ {
    outer_bc **outer; // Array of 1D arrays, of length
                      //   [NGHOSTS][num_ob_gz_pts[which_outer_ghostzone_point]]

    inner_bc **inner; // Array of 1D arrays, of length
                      //   [NGHOSTS][num_ib_gz_pts[which_inner_ghostzone_point]]

    // Arrays storing number of outer/inner boundary ghostzone points at each ghostzone,
    //   of length NGHOSTS:
    int     *num_ob_gz_pts;
    int     *num_ib_gz_pts;
} bc_struct;

```

Ghost zones must be filled in from the inside outward, as e.g., the outermost ghost zones may depend on ghost zones closer to the grid interior being set. We thus store ghost zones in arrays that point to a particular layer of ghost zones. This explains the fact that we declare `inner` and `outer` in `bc_struct` with the `**` prefix, denoting that these are "pointers to pointers".

For example `outer[0]` points to the set of ghost zone cells on the *outer* boundary that are in the layer of ghost zones just adjacent to the grid interior. Further, `outer[0][0]` points to a `outer_bc` struct containing all information needed to fill "outer boundary point zero" with valid ata. Note that the numbering of the boundary points (the `j` index in `outer[i][j]`) is rather arbitrary, but each point has a unique label, and there are no duplicates. This ensures efficiency and locality in memory.

The same reasoning holds when considering ghost zones that are not outer boundary points.

Next we summarize all basic data structures that appear within `bc_struct`.

* `inner_bc`:
```c
typedef struct __inner_bc__ {
    gz_map inner_bc_dest_pt;
    gz_map inner_bc_src_pt;
    int8_t parity[10]; // We store the 10 parity conditions in 10 int8_t integers,
                       // one for each condition. Note that these conditions can
                       // only take one of two values: +1 or -1, hence the use of
                       // int8_t, the smallest C data type.
} inner_bc;
```
    * `inner_bc_dest_pt.{i0,i1,i2}`: Location `(i0,i1,i2)` of inner ghost zone point `which_pt` on the `which_gz` ghost zone layer.
    * `inner_bc_src_pt.{i0,i1,i2}`: Location of the interior grid point to which the `inner_bc_dest_pt.{i0,i1,i2}` inner ghost zone maps.
    * `parity[10]` Parity information ($\pm 1$) at the given inner ghost zone, for all 10 gridfunction parity types.
* `outer_bc`:
```c
typedef struct __outer_bc__ {
    gz_map outer_bc_dest_pt;
    int8_t FACEi0,FACEi1,FACEi2; // FACEi* takes values of -1, 0, and +1 only,
                                 // corresponding to MAXFACE, NUL, and MINFACE
                                 // respectively.
                                 // Thus int8_t (one byte each, the smallest C
                                 // type) is sufficient.
} outer_bc;
```
    * outer_bc_dest_pt.{i0,i1,i2}`: Location `(i0,i1,i2)` of outer ghost zone point `which_pt` on the `which_gz` ghost zone layer
    * `int8_t FACEi0,FACEi1,FACEi2`: Specifies to which face of the numerical domain outer boundary point `which_pt` on the `which_gz` ghost zone layer corresponds. Many outer boundary conditions depend on some extrapolation from inner points. Thus knowing on which face a given outer boundary point lies provides needed direction for extrapolation.
* `num_ib/ob_gz_pts[which_gz]`: The number of inner/outer boundary points on ghost zone layer `which_gz`

```python
# Step P1: Import needed NRPy+ core modules:
from outputC import *            # NRPy+: Core C code output module
import NRPy_param_funcs as par   # NRPy+: Parameter interface
import grid as gri               # NRPy+: Functions having to do with numerical grids
import indexedexp as ixp         # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support
import reference_metric as rfm   # NRPy+: Reference metric support
import cmdline_helper as cmd     # NRPy+: Multi-platform Python command-line interface
import shutil, os, sys           # Standard Python modules for multiplatform OS-level functions

# Step P2: Set spatial dimension (must be 3 for BSSN)
DIM = 3
par.set_parval_from_str("grid::DIM",DIM)

# Step P3: Create C code output directory:
Ccodesdir = os.path.join("CurviBoundaryConditions_Ccodes/")
# First remove C code output directory if it exists
# Courtesy https://stackoverflow.com/questions/303200/how-do-i-remove-delete-a-folder-that-is-not-empty
# !rm -r ScalarWaveCurvilinear_Playground_Ccodes
shutil.rmtree(Ccodesdir, ignore_errors=True)
# Then create a fresh directory
cmd.mkdir(Ccodesdir)
cmd.mkdir(os.path.join(Ccodesdir,"boundary_conditions"))

# Step P4: Set path for set_Cparameters.h, relative to Ccodesdir/boundary_conditions/
Cparamspath = os.path.join("../")
# Step P5: Output correct #include for set_Cparameters.h to
#          Ccodesdir/boundary_conditions/RELATIVE_PATH__set_Cparameters.h
with open(os.path.join(Ccodesdir,"boundary_conditions","RELATIVE_PATH__set_Cparameters.h"), "w") as file:
    file.write("#include \"" + Cparamspath + "/set_Cparameters.h\"\n") # #include's may include forward slashes for paths, even in Windows.
```

```python
%%writefile $Ccodesdir/boundary_conditions/BCs_data_structs.h

typedef struct __ghostzone_map__ {
  short i0,i1,i2; // i0,i1,i2 stores values from -1 (used to indicate outer boundary)
                  // to Nxx_plus_2NGHOSTS*. We assume that grid extents beyond the
                  // limits of short (i.e., beyond about 32,000) are unlikely. This
                  // can be easily extended if needed, though.
} gz_map;

const int8_t MAXFACE = -1;
const int8_t NUL     = +0;
const int8_t MINFACE = +1;

typedef struct __parity__ {
  int8_t parity[10]; // We store the 10 parity conditions in 10 int8_t integers,
                     // one for each condition. Note that these conditions can
                     // only take one of two values: +1 or -1, hence the use of
                     // int8_t, the smallest C data type.
} parity_condition;

typedef struct __inner_bc__ {
    gz_map inner_bc_dest_pt;
    gz_map inner_bc_src_pt;
    int8_t parity[10]; // We store the 10 parity conditions in 10 int8_t integers,
                       // one for each condition. Note that these conditions can
                       // only take one of two values: +1 or -1, hence the use of
                       // int8_t, the smallest C data type.
} inner_bc;

typedef struct __outer_bc__ {
    gz_map outer_bc_dest_pt;
    int8_t FACEi0,FACEi1,FACEi2; // FACEi* takes values of -1, 0, and +1 only,
                                 // corresponding to MAXFACE, NUL, and MINFACE
                                 // respectively.
                                 // Thus int8_t (one byte each, the smallest C
                                 // type) is sufficient.
} outer_bc;

typedef struct __bcstruct__ {
    outer_bc **outer; // Array of 1D arrays, of length
                      //   [NGHOSTS][num_ob_gz_pts[which_outer_ghostzone_point]]

    inner_bc **inner; // Array of 1D arrays, of length
                      //   [NGHOSTS][num_ib_gz_pts[which_inner_ghostzone_point]]

    // Arrays storing number of outer/inner boundary ghostzone points at each ghostzone,
    //   of length NGHOSTS:
    int     *num_ob_gz_pts;
    int     *num_ib_gz_pts;
} bc_struct;
```

    Writing CurviBoundaryConditions_Ccodes//boundary_conditions/BCs_data_structs.h

#  Step 3: NRPy+-based C code generation for parity conditions \[Back to [top](#toc)\]
$$\label{nrpycodegen}$$

Much of the algorithm needed for setting up `bcstruct` requires a loop over all gridpoints on the numerical grid. As the precise numerical grids are chosen at C runtime, that part of the algorithm must be run entirely within a static C code.

However, there are two parts to the overall algorithm that must be generated by NRPy+, namely

1. [Step 3.a](#dotproducts): `parity_conditions_symbolic_dot_products.h`: Based on the chosen reference metric, sets up the needed unit-vector dot products for each of the 10 parity condition types.
1. [Step 3.b](#set_parity_type): Set parity type for each gridfunction registered within NRPy+, based on the digits at the end of each gridfunction name, append result to `dirname+gridfunction_defines.h`

## Step 3.a: Set up unit-vector dot products (=parity) for each of the 10 parity condition types, store to `parity_conditions_symbolic_dot_products.h` \[Back to [top](#toc)\]
$$\label{dotproducts}$$

Next we generate the C code necessary to perform needed dot products for filling in the parity condition arrays inside `bcstruct`.

Using the unit vectors defined in `rfm.UnitVectors[][]` (in `reference_metric.py`), each unit vector takes as input either $\mathbf{d}_{\rm gz} = (x_0,x_1,x_2)_{\rm IB}$=`(xx0,xx1,xx2)` or $\mathbf{d}_{\rm new} = (x_0,x_1,x_2)_{\rm in}$=`(xx0_inbounds,xx1_inbounds,xx2_inbounds)` as summarized in the table above in [Step 1](#challenge2). We paste the table here again, for quick reference:

The appropriate symbolic dot products determining parity condition are assigned to each gridfunction based on the following numbering:

Tensor type | Parity type | Dot product(s) determining parity condition (
--- | --- | ---
Scalar (Rank-0 tensor) | 0 | (*none*)
Rank-1 tensor in **i0** direction | 1 | $\mathbf{e}^0\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^0\left(\mathbf{d}_{\rm new}\right)$
Rank-1 tensor in **i1** direction | 2 | $\mathbf{e}^1\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^1\left(\mathbf{d}_{\rm new}\right)$
Rank-1 tensor in **i2** direction | 3 | $\mathbf{e}^2\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^2\left(\mathbf{d}_{\rm new}\right)$
Rank-2 tensor in **i0-i0** direction | 4 | $\left[\mathbf{e}^0\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^0\left(\mathbf{d}_{\rm new}\right)\right]^2 = 1$
Rank-2 tensor in **i0-i1** direction | 5 | $\left[\mathbf{e}^0\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^0\left(\mathbf{d}_{\rm new}\right)\right]\left[\mathbf{e}^1\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^1\left(\mathbf{d}_{\rm new}\right)\right]$
Rank-2 tensor in **i0-i2** direction | 6 | $\left[\mathbf{e}^0\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^0\left(\mathbf{d}_{\rm new}\right)\right]\left[\mathbf{e}^2\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^2\left(\mathbf{d}_{\rm new}\right)\right]$
Rank-2 tensor in **i1-i1** direction | 7 | $\left[\mathbf{e}^1\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^1\left(\mathbf{d}_{\rm new}\right)\right]^2 = 1$
Rank-2 tensor in **i1-i2** direction | 8 | $\left[\mathbf{e}^1\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^1\left(\mathbf{d}_{\rm new}\right)\right]\left[\mathbf{e}^2\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^2\left(\mathbf{d}_{\rm new}\right)\right]$
Rank-2 tensor in **i2-i2** direction | 9 | $\left[\mathbf{e}^2\left(\mathbf{d}_{\rm gz}\right) \cdot \mathbf{e}^2\left(\mathbf{d}_{\rm new}\right)\right]^2 = 1$

Looping over all 10 parity types, the corresponding symbolic expressions for dot product(s) is output to C code. For example, in Spherical coordinates, parity type 1's dot product output as C code is given by:

```
parity[1] = sin(xx1)*sin(xx1_inbounds)*sin(xx2)*sin(xx2_inbounds) + sin(xx1)*sin(xx1_inbounds)*cos(xx2)*cos(xx2_inbounds) + cos(xx1)*cos(xx1_inbounds);
```

To wit (as described above), there are 10 parity types for BSSN evolved variables, which include tensors up to and including rank-2.

```python
# Set unit-vector dot products (=parity) for each of the 10 parity condition types
def parity_conditions_symbolic_dot_products(Ccodesdir="CurviBoundaryConditions/boundary_conditions/"):
    parity = ixp.zerorank1(DIM=10)
    UnitVectors_inner = ixp.zerorank2()
    xx0_inbounds,xx1_inbounds,xx2_inbounds = sp.symbols("xx0_inbounds xx1_inbounds xx2_inbounds", real=True)
    for i in range(3):
        for j in range(3):
            UnitVectors_inner[i][j] = rfm.UnitVectors[i][j].subs(rfm.xx[0],xx0_inbounds).subs(rfm.xx[1],xx1_inbounds).subs(rfm.xx[2],xx2_inbounds)
    # Type 0: scalar
    parity[0] = sp.sympify(1)
    # Type 1: i0-direction vector or one-form
    # Type 2: i1-direction vector or one-form
    # Type 3: i2-direction vector or one-form
    for i in range(3):
        for Type in range(1,4):
            parity[Type] += rfm.UnitVectors[Type-1][i]*UnitVectors_inner[Type-1][i]
    # Type 4: i0i0-direction rank-2 tensor
    # parity[4] = parity[1]*parity[1]
    # Type 5: i0i1-direction rank-2 tensor
    # Type 6: i0i2-direction rank-2 tensor
    # Type 7: i1i1-direction rank-2 tensor
    # Type 8: i1i2-direction rank-2 tensor
    # Type 9: i2i2-direction rank-2 tensor
    count = 4
    for i in range(3):
        for j in range(i,3):
            parity[count] = parity[i+1]*parity[j+1]
            count = count + 1

    lhs_strings = []
    for i in range(10):
        lhs_strings.append("parity["+str(i)+"]")
    outputC(parity,lhs_strings, os.path.join(Ccodesdir,"parity_conditions_symbolic_dot_products.h"))

#     print("\n\nExample: parity type 1's dot product is given by: \n"+lhs_strings[1]+" = "+str(parity[1]))
```

## Step 3.b: Set parity type for each gridfunction, based on the digits at the end of its name, output to `dirname+gridfunction_defines.h` \[Back to [top](#toc)\]
$$\label{set_parity_type}$$

For example, if the gridfunction name ends with "01", then (based on the table above) the `set_parity_types()` function below will set the parity_type of that gridfunction to 5. We can be assured this is a robust algorithm, because `gri.register_gridfunctions()` in [grid.py](../edit/grid.py) will throw an error if a gridfunction base name ends in an integer. This strict syntax was added with the express purpose of making it easier to set parity types based on the gridfunction name.

After each parity type is found, we store the parity type of each gridfunction to `const int8_t arrays` `evol_gf_parity` and `aux_gf_parity`, appended to the end of `gridfunction_defines.h`.

```python
def set_gridfunction_defines_with_parity_types(Ccodesdir="CurviBoundaryConditions/",verbose=True):

    # First generate Ccodesdir/gridfunction_defines.h file,
    #       containing human-readable gridfunction aliases
    evolved_variables_list, auxiliary_variables_list, auxevol_variables_list = gri.output__gridfunction_defines_h__return_gf_lists(Ccodesdir)

    # Step 3.b: set the parity conditions on all gridfunctions in gf_list,
    #       based on how many digits are at the end of their names
    def set_parity_types(list_of_gf_names):
        parity_type = []
        for name in list_of_gf_names:
            for gf in gri.glb_gridfcs_list:
                if gf.name == name:
                    parity_type__orig_len = len(parity_type)
                    if gf.DIM < 3 or gf.DIM > 4:
                        print("Error: Cannot currently specify parity conditions on gridfunctions with DIM<3 or >4.")
                        sys.exit(1)
                    if gf.rank == 0:
                        parity_type.append(0)
                    elif gf.rank == 1:
                        if gf.DIM == 3:
                            parity_type.append(int(gf.name[-1]) + 1)  # = 1 for e.g., beta^0; = 2 for e.g., beta^1, etc.
                        elif gf.DIM == 4:
                            parity_type.append(int(gf.name[-1]))  # = 0 for e.g., b4^0; = 1 for e.g., beta^1, etc.
                    elif gf.rank == 2:
                        if gf.DIM == 3:
                            # element of a list; a[-2] the
                            # second-to-last element, etc.
                            idx0 = gf.name[-2]
                            idx1 = gf.name[-1]
                            if idx0 == "0" and idx1 == "0":
                                parity_type.append(4)
                            elif (idx0 == "0" and idx1 == "1") or (idx0 == "1" and idx1 == "0"):
                                parity_type.append(5)
                            elif (idx0 == "0" and idx1 == "2") or (idx0 == "2" and idx1 == "0"):
                                parity_type.append(6)
                            elif idx0 == "1" and idx1 == "1":
                                parity_type.append(7)
                            elif (idx0 == "1" and idx1 == "2") or (idx0 == "2" and idx1 == "1"):
                                parity_type.append(8)
                            elif idx0 == "2" and idx1 == "2":
                                parity_type.append(9)
                        elif gf.DIM == 4:
                            idx0 = gf.name[-2]
                            idx1 = gf.name[-1]
                            # g4DD00 = g_{tt} : parity type = 0
                            # g4DD01 = g_{tx} : parity type = 1
                            # g4DD02 = g_{ty} : parity type = 2
                            # g4DD0a = g_{ta} : parity type = a
                            if idx0 == "0":
                                parity_type.append(int(idx1))
                            elif idx1 == "0":
                                parity_type.append(int(idx0))
                            if idx0 == "1" and idx1 == "1":
                                parity_type.append(4)
                            elif (idx0 == "1" and idx1 == "2") or (idx0 == "2" and idx1 == "1"):
                                parity_type.append(5)
                            elif (idx0 == "1" and idx1 == "3") or (idx0 == "3" and idx1 == "1"):
                                parity_type.append(6)
                            elif idx0 == "2" and idx1 == "2":
                                parity_type.append(7)
                            elif (idx0 == "2" and idx1 == "3") or (idx0 == "3" and idx1 == "2"):
                                parity_type.append(8)
                            elif idx0 == "3" and idx1 == "3":
                                parity_type.append(9)
                    if len(parity_type) == parity_type__orig_len:
                        print("Error: Could not figure out parity type for "+gf.gftype+" gridfunction: " + gf.name,gf.DIM,gf.name[-2],gf.name[-1],gf.rank)
                        sys.exit(1)
        if len(parity_type) != len(list_of_gf_names):
            print("Error: For some reason the length of the parity types list did not match the length of the gf list.")
            sys.exit(1)
        return parity_type

    evol_parity_type = set_parity_types(evolved_variables_list)
    aux_parity_type = set_parity_types(auxiliary_variables_list)
    auxevol_parity_type = set_parity_types(auxevol_variables_list)

    # Output all gridfunctions to Ccodesdir/gridfunction_defines.h
    # ... then append to the file the parity type for each gridfunction.
    with open(os.path.join(Ccodesdir, "gridfunction_defines.h"), "a") as file:
        file.write("\n\n/* PARITY TYPES FOR ALL GRIDFUNCTIONS.\n")
        file.write(
            "   SEE \"Tutorial-Start_to_Finish-Curvilinear_BCs.ipynb\" FOR DEFINITIONS. */\n")
        if len(evolved_variables_list) > 0:
            file.write("const int8_t evol_gf_parity[" + str(len(evolved_variables_list)) + "] = { ")
            for i in range(len(evolved_variables_list) - 1):
                file.write(str(evol_parity_type[i]) + ", ")
            file.write(str(evol_parity_type[len(evolved_variables_list) - 1]) + " };\n")

        if len(auxiliary_variables_list) > 0:
            file.write("const int8_t aux_gf_parity[" + str(len(auxiliary_variables_list)) + "] = { ")
            for i in range(len(auxiliary_variables_list) - 1):
                file.write(str(aux_parity_type[i]) + ", ")
            file.write(str(aux_parity_type[len(auxiliary_variables_list) - 1]) + " };\n")

        if len(auxevol_variables_list) > 0:
            file.write("const int8_t auxevol_gf_parity[" + str(len(auxevol_variables_list)) + "] = { ")
            for i in range(len(auxevol_variables_list) - 1):
                file.write(str(auxevol_parity_type[i]) + ", ")
            file.write(str(auxevol_parity_type[len(auxevol_variables_list) - 1]) + " };\n")

    if verbose == True:
        for i in range(len(evolved_variables_list)):
            print("Evolved gridfunction \"" + evolved_variables_list[i] + "\" has parity type " + str(
                evol_parity_type[i]) + ".")
        for i in range(len(auxiliary_variables_list)):
            print("Auxiliary gridfunction \"" + auxiliary_variables_list[i] + "\" has parity type " + str(
                aux_parity_type[i]) + ".")
        for i in range(len(auxevol_variables_list)):
            print("AuxEvol gridfunction \"" + auxevol_variables_list[i] + "\" has parity type " + str(
                auxevol_parity_type[i]) + ".")
```

# Step 4:  `set_up__bc_gz_map_and_parity_condns()`: C function for distinguishing inner from outer boundary points, and setting parity conditions \[Back to [top](#toc)\]
$$\label{set_up__bc_gz_map_and_parity_condns}$$

This step is performed using only the [Eigen-Coordinate](#challenge2) corresponding to the chosen `reference_metric::CoordSystem`.

`set_up__bc_gz_map_and_parity_condns()` loops over *all* points on the numerical grid (interior and ghost zone points included), with the aim of filling `gz_map *bc_gz_map` and `parity_condition *bc_parity_conditions` at each point. To do so, the function implements the algorithm described above in [Step 1](#basic_algorithm).

That is to say, at each coordinate point $(x_0,x_1,x_2)$ situated at grid index `(i0,i1,i2)`:

1. Convert the curvilinear coordinate $(x_0,x_1,x_2)$ to the corresponding Cartesian coordinate $(x,y,z)$
1. Find the Cartesian grid point `(i0_inbounds,i1_inbounds,i2_inbounds)` in the grid interior or outer boundary corresponding to this Cartesian coordinate $(x,y,z)$.
1. If and only if we are on an outer boundary ghost zone or in the grid interior, `i0_inbounds==i0`, `i1_inbounds==i1`, and `i2_inbounds==i2`, and inner boundary conditions do not apply: set `bc_gz_map` to $(-1,-1,-1)$, and for all 10 gridfunction parities, set `parity=1`.
1. If `i0_inbounds==i0`, `i1_inbounds==i1`, and `i2_inbounds==i2`, does not hold true, then `(i0,i1,i2)` *is* an inner boundary point: set `bc_gz_map` to `(i0_inbounds,i1_inbounds,i2_inbounds)`, and for all 10 gridfunction parities, evaluate all dot products of the unit vectors evaluated at `(i0_inbounds,i1_inbounds,i2_inbounds)` and `(i0,i1,i2)`, as prescribed above in [Step 1](#challenge2). The C code for computing the needed symbolic dot products is generated above in [Step 2](#dotproducts), and is imported via an `#include` statement in below C code function `eval_symbolic_dot_products_to_set_parity_conditions()`.

```python
%%writefile $Ccodesdir/boundary_conditions/set_up__bc_gz_map_and_parity_condns.h

// set_parity_conditions_from_symbolic_dot_products():
// Evaluate dot products needed for setting parity
//     conditions at a given point (xx0,xx1,xx2),
//     using C code generated by NRPy+ function
//     parity_conditions_symbolic_dot_products().
void eval_symbolic_dot_products_to_set_parity_conditions(const paramstruct *restrict params,  REAL parity[10],
                                    const REAL xx0,const REAL xx1,const REAL xx2,
                                    const REAL xx0_inbounds,const REAL xx1_inbounds,const REAL xx2_inbounds) {
#include "RELATIVE_PATH__set_Cparameters.h" /* Header file containing correct #include for set_Cparameters.h;
                                             * accounting for the relative path */
#include "parity_conditions_symbolic_dot_products.h"
}

void set_up__bc_gz_map_and_parity_condns(const paramstruct *restrict params,
                                         REAL *xx[3], gz_map *bc_gz_map,parity_condition *bc_parity_conditions) {
#include "RELATIVE_PATH__set_Cparameters.h" /* Header file containing correct #include for set_Cparameters.h;
                                             * accounting for the relative path */
  // xx[0][j] = xxmin[0] + ((REAL)(j-NGHOSTS) + (1.0/2.0))*dxx0;
  // -> xxmin[0] = xx[0][0] - ((REAL)(0-NGHOSTS) + (1.0/2.0))*dxx0
  const REAL xxmin[3] = { xx[0][0] - ((REAL)(0-NGHOSTS) + (1.0/2.0))*dxx0,
                          xx[1][0] - ((REAL)(0-NGHOSTS) + (1.0/2.0))*dxx1,
                          xx[2][0] - ((REAL)(0-NGHOSTS) + (1.0/2.0))*dxx2 };
  //fprintf(stderr,"hey inside setbc: %e %e %e | %e %e\n",xxmin[0],xxmin[1],xxmin[2],xx[0][0],dxx0);
  LOOP_REGION(0,Nxx_plus_2NGHOSTS0,0,Nxx_plus_2NGHOSTS1,0,Nxx_plus_2NGHOSTS2) {
    // Step 1: Convert the (curvilinear) coordinate (x0,x1,x2) to Cartesian coordinates
    REAL xCart[3];
    EigenCoord_xxCart(params, xx, i0,i1,i2, xCart);
    REAL Cartx = xCart[0];
    REAL Carty = xCart[1];
    REAL Cartz = xCart[2];

    // Step 2: Find the (i0_inbounds,i1_inbounds,i2_inbounds) corresponding to the above Cartesian coordinate.
    //   If (i0_inbounds,i1_inbounds,i2_inbounds) is in a ghost zone, then it must equal (i0,i1,i2), and
    //      the point is an outer boundary point.
    //   Otherwise (i0_inbounds,i1_inbounds,i2_inbounds) is in the grid interior, and data at (i0,i1,i2)
    //      must be replaced with data at (i0_inbounds,i1_inbounds,i2_inbounds), but multiplied by the
    //      appropriate parity condition (+/- 1).
    REAL Cart_to_xx0_inbounds,Cart_to_xx1_inbounds,Cart_to_xx2_inbounds;
#include "EigenCoord_Cart_to_xx.h"
    int i0_inbounds = (int)( (Cart_to_xx0_inbounds - xxmin[0] - (1.0/2.0)*dxx0 + ((REAL)NGHOSTS)*dxx0)/dxx0 + 0.5 );
    int i1_inbounds = (int)( (Cart_to_xx1_inbounds - xxmin[1] - (1.0/2.0)*dxx1 + ((REAL)NGHOSTS)*dxx1)/dxx1 + 0.5 );
    int i2_inbounds = (int)( (Cart_to_xx2_inbounds - xxmin[2] - (1.0/2.0)*dxx2 + ((REAL)NGHOSTS)*dxx2)/dxx2 + 0.5 );

    // Step 2.a: (Sanity/validation check) Convert the interior point
    //           x0(i0_inbounds),x1(i1_inbounds),x2(i2_inbounds) to Cartesian coordinates,
    //           make sure that the Cartesian coordinate matches the Cartesian coordinate of
    //           x0(i0),x1(i1),x2(i2). If not, error out!
    REAL xCart_orig[3]; for(int ii=0;ii<3;ii++) xCart_orig[ii] = xCart[ii];
    EigenCoord_xxCart(params, xx, i0_inbounds,i1_inbounds,i2_inbounds, xCart);

//fprintf(stderr,"Cartesian agreement: ( %.15e %.15e %.15e ) ?= ( %.15e %.15e %.15e )\n",
// (double)xCart_orig[0],(double)xCart_orig[1],(double)xCart_orig[2],
// (double)xCart[0],(double)xCart[1],(double)xCart[2]);

#define EPS_ABS 1e-8
    if(fabs( (double)(xCart_orig[0] - xCart[0]) ) > EPS_ABS ||
       fabs( (double)(xCart_orig[1] - xCart[1]) ) > EPS_ABS ||
       fabs( (double)(xCart_orig[2] - xCart[2]) ) > EPS_ABS) {
      fprintf(stderr,"Error. Cartesian disagreement: ( %.15e %.15e %.15e ) != ( %.15e %.15e %.15e )\n",
             (double)xCart_orig[0],(double)xCart_orig[1],(double)xCart_orig[2],
             (double)xCart[0],(double)xCart[1],(double)xCart[2]);
      exit(1);
    }

    // Step 3: Set bc_gz_map and bc_parity_conditions.
    if(i0_inbounds-i0 == 0 && i1_inbounds-i1 == 0 && i2_inbounds-i2 == 0) {
      // Step 3.a: Iff we are on an outer boundary point or in the grid
      //           interior, i0_inbounds==i0, i1_inbounds==i1, and
      //           i2_inbounds==i2, and inner boundary conditions do not
      //           apply: set bc_gz_map to -1, and parity=1.
      bc_gz_map[IDX3S(i0,i1,i2)].i0=-1;
      bc_gz_map[IDX3S(i0,i1,i2)].i1=-1;
      bc_gz_map[IDX3S(i0,i1,i2)].i2=-1;
      for(int which_parity=0; which_parity<10; which_parity++) {
        bc_parity_conditions[IDX3S(i0,i1,i2)].parity[which_parity] = 1;
      }
    } else {
      // Step 3.b: If we are on an *inner* boundary point:
      // 1. Set bc_gz_map at (i0,i1,i2) to the point
      //    in the interior to which this boundary
      //    point maps, and
      // 2. Perform the unit vector dot products
      //    necessary to set all 10 possible parity
      //    conditions, calling function
      //    set_parity_from_unit_vector_dot_product()
      bc_gz_map[IDX3S(i0,i1,i2)].i0=i0_inbounds;
      bc_gz_map[IDX3S(i0,i1,i2)].i1=i1_inbounds;
      bc_gz_map[IDX3S(i0,i1,i2)].i2=i2_inbounds;
      const REAL xx0 = xx[0][i0];
      const REAL xx1 = xx[1][i1];
      const REAL xx2 = xx[2][i2];
      const REAL xx0_inbounds = xx[0][i0_inbounds];
      const REAL xx1_inbounds = xx[1][i1_inbounds];
      const REAL xx2_inbounds = xx[2][i2_inbounds];
      REAL REAL_parity_array[10];
      eval_symbolic_dot_products_to_set_parity_conditions(params,  REAL_parity_array,  xx0,xx1,xx2,
                                                          xx0_inbounds,xx1_inbounds,xx2_inbounds);
      for(int whichparity=0;whichparity<10;whichparity++) {
          //printf("Good? Parity %d evaluated to %e\n",whichparity,(double)REAL_parity_array[whichparity]);
          // Perform sanity check on parity array output: should be +1 or -1 to within 8 significant digits:
          if( (REAL_parity_array[whichparity]  > 0 && fabs(REAL_parity_array[whichparity] - (+1)) > 1e-8) ||
              (REAL_parity_array[whichparity] <= 0 && fabs(REAL_parity_array[whichparity] - (-1)) > 1e-8) ) {
              fprintf(stderr,"Error. Parity evaluated to %e , which is not within 8 significant digits of +1 or -1.",
                      REAL_parity_array[whichparity]);
              exit(1);
          }
          if(REAL_parity_array[whichparity] < 0.0) bc_parity_conditions[IDX3S(i0,i1,i2)].parity[whichparity] = -1;
          if(REAL_parity_array[whichparity] > 0.0) bc_parity_conditions[IDX3S(i0,i1,i2)].parity[whichparity] = +1;
      }
    }
  }
}
```

    Writing CurviBoundaryConditions_Ccodes//boundary_conditions/set_up__bc_gz_map_and_parity_condns.h

# Step 5: `set_bcstruct()`: Using information from `set_up__bc_gz_map_and_parity_condns()` as input, set `bc_struct` \[Back to [top](#toc)\]
$$\label{set_bc_struct}$$

As described above, `set_up__bc_gz_map_and_parity_condns()` sets `gz_map *bc_gz_map` and `parity_condition *bc_parity_conditions` at *all* grid points. While this information could be used directly to apply boundary conditions, it is more efficient (both in memory and CPU time) to instead store only the needed information to the `bcstruct` array, so that when applying boundary conditions, we can simply loop over `bcstruct`.

The algorithm follows the `bc_struct` data type defined above in [Step 2](#bc_struct), and loops over all boundary ghost zones, from the innermost layer (`which_gz=0`) outward (to `which_gz=NGHOSTS-1`). This is necessary, as, for example, some ghost zone points on the `which_gz=1` layer depend on ghost zones being set on the `which_gz=0` layer.

1. Count the number of outer and inner boundary points, store to `num_ob_pts` and `num_ib_pts`, respectively.
1. Now that we know the number of outer boundary points on this ghostzone layer, allocate memory needed for storing the outer and inner boundary condition data.
    1. At all outer boundary ghost zones, allocate memory a single member of the `outer_bc` data type.
    1. At all inner boundary ghost zones, allocate memory a single member of the `inner_bc` data type.
1. Store the number of outer and inner boundary points on each ghostzone layer, where e.g.,`which_gz==0` corresponds to the innermost ghostzones on the numerical domain.
1. Store information needed for outer boundary conditions, to `outer_bc_dest_pt` and `outer_bc_face` arrays.
1. Store information needed for inner boundary conditions, including interior point to which inner ghost zone maps, and parity conditions for all 10 gridfunction types.

```python
%%writefile $Ccodesdir/boundary_conditions/set_bcstruct.h

// set_bcstruct() loops from the innermost boundary
//      ghostzones on the cube ("which_gz==0",
//      corresponding to the single layer of ghostzones
//      closest to the interior data), and at each
//      ghostzone layer, we apply the following 5-step
//      algorithm:
// Step 1: Count the number of outer and inner
//         boundary points, store to
//         num_ob_pts and num_ib_pts, respectively.
// Step 2: Now that we know the number of outer
//         boundary points on this ghostzone layer,
//         allocate memory needed for storing the
//         outer and inner boundary condition data.
// Step 2.a: At all outer boundary ghost zones, allocate
//           memory for a single member of the outer_bc
//           data type.
// Step 2.b: At all inner boundary ghost zones, allocate
//           memory for a single member of the inner_bc
//           data type.
// Step 3: Store the number of outer and inner boundary
//         points on each ghostzone layer, where e.g.,
//         which_gz==0 corresponds to the innermost
//         ghostzones on the numerical domain.
// Step 4: Store information needed for outer boundary
//         conditions, to outer_bc_dest_pt and
//         outer_bc_face arrays.
// Step 5: Store information needed for inner boundary
//         conditions, including interior point to which
//         inner ghost zone maps, and parity conditions
//         for all 10 gridfunction types.
void set_bcstruct(const paramstruct *restrict params,
                  gz_map *restrict bc_gz_map,
                  parity_condition *bc_parity_conditions,
                  bc_struct *restrict bcstruct) {
#include "RELATIVE_PATH__set_Cparameters.h" /* Header file containing correct #include for set_Cparameters.h;
                                             * accounting for the relative path */
    int imin[3] = { NGHOSTS, NGHOSTS, NGHOSTS };
    int imax[3] = { Nxx_plus_2NGHOSTS0-NGHOSTS, Nxx_plus_2NGHOSTS1-NGHOSTS, Nxx_plus_2NGHOSTS2-NGHOSTS };

    // Loop from the innermost ghostzone on the cube (which_gz==0) and work outward.
    //      This ordering is necessary, as ghostzones at which_gz==1 will generally
    //      depend on ghostzones at which_gz==0 being already set.
    for(int which_gz = 0; which_gz < NGHOSTS; which_gz++) {

    // Step 1: Count the number of outer and inner
    //         boundary points, store to
    //         num_ob_pts and num_ib_pts, respectively.
#define COUNT_INNER_OR_OUTER if(bc_gz_map[IDX3S(i0,i1,i2)].i0==-1) { num_ob_pts++;} else { num_ib_pts++; }
        int num_ob_pts = 0;
        int num_ib_pts = 0;
        LOOP_REGION(imin[0]-1,imin[0], imin[1],imax[1], imin[2],imax[2]) { COUNT_INNER_OR_OUTER } imin[0]--;
        LOOP_REGION(imax[0],imax[0]+1, imin[1],imax[1], imin[2],imax[2]) { COUNT_INNER_OR_OUTER } imax[0]++;
        LOOP_REGION(imin[0],imax[0], imin[1]-1,imin[1], imin[2],imax[2]) { COUNT_INNER_OR_OUTER } imin[1]--;
        LOOP_REGION(imin[0],imax[0], imax[1],imax[1]+1, imin[2],imax[2]) { COUNT_INNER_OR_OUTER } imax[1]++;
        LOOP_REGION(imin[0],imax[0], imin[1],imax[1], imin[2]-1,imin[2]) { COUNT_INNER_OR_OUTER } imin[2]--;
        LOOP_REGION(imin[0],imax[0], imin[1],imax[1], imax[2],imax[2]+1) { COUNT_INNER_OR_OUTER } imax[2]++;

        // Step 2: Now that we know the number of outer boundary points on this ghostzone
        //    layer, we allocate memory needed for storing the outer and inner boundary
        //     condition data.

        // Step 2.a: At all outer boundary ghost zones, allocate memory for a single member of the outer_bc
        //           data type.
        bcstruct->outer[which_gz] = (outer_bc *)malloc(sizeof(outer_bc)*num_ob_pts);
        // Step 2.b: At all inner boundary ghost zones, allocate memory for a single member of the inner_bc
        //           data type.
        bcstruct->inner[which_gz] = (inner_bc *)malloc(sizeof(inner_bc)*num_ib_pts);

        // Step 3: Store the number of outer and inner boundary points on each ghostzone layer, where e.g.,
        //         which_gz==0 corresponds to the innermost ghostzones on the numerical domain.
        bcstruct->num_ob_gz_pts[which_gz] = num_ob_pts;
        bcstruct->num_ib_gz_pts[which_gz] = num_ib_pts;

        // Reset imin[] and imax[], to prepare for the next step.
        for(int ii=0;ii<3;ii++) {imin[ii]++; imax[ii]--;}

        // Step 4: Store information needed for outer boundary conditions, to outer_bc_dest_pt[which_gz][]
        //         and outer_bc_face[which_gz][] arrays:
#define OB_SET(facei0,facei1,facei2) if(bc_gz_map[IDX3S(i0,i1,i2)].i0==-1) {  \
    bcstruct->outer[which_gz][pt].outer_bc_dest_pt.i0 = i0;     \
    bcstruct->outer[which_gz][pt].outer_bc_dest_pt.i1 = i1;     \
    bcstruct->outer[which_gz][pt].outer_bc_dest_pt.i2 = i2;     \
    bcstruct->outer[which_gz][pt].FACEi0= facei0; \
    bcstruct->outer[which_gz][pt].FACEi1= facei1; \
    bcstruct->outer[which_gz][pt].FACEi2= facei2; \
    pt++; }

        int pt = 0;
        LOOP_REGION(imin[0]-1,imin[0], imin[1],imax[1], imin[2],imax[2]) {OB_SET(MINFACE,NUL,NUL)} imin[0]--;
        LOOP_REGION(imax[0],imax[0]+1, imin[1],imax[1], imin[2],imax[2]) {OB_SET(MAXFACE,NUL,NUL)} imax[0]++;
        LOOP_REGION(imin[0],imax[0], imin[1]-1,imin[1], imin[2],imax[2]) {OB_SET(NUL,MINFACE,NUL)} imin[1]--;
        LOOP_REGION(imin[0],imax[0], imax[1],imax[1]+1, imin[2],imax[2]) {OB_SET(NUL,MAXFACE,NUL)} imax[1]++;
        LOOP_REGION(imin[0],imax[0], imin[1],imax[1], imin[2]-1,imin[2]) {OB_SET(NUL,NUL,MINFACE)} imin[2]--;
        LOOP_REGION(imin[0],imax[0], imin[1],imax[1], imax[2],imax[2]+1) {OB_SET(NUL,NUL,MAXFACE)} imax[2]++;
        // fprintf(stderr,"num OB points with which_gz = %d: %d | should be: %d\n",which_gz,pt,num_ob_gz_pts[which_gz]);

        // Reset imin[] and imax[], to prepare for the next step.
        for(int ii=0;ii<3;ii++) {imin[ii]++; imax[ii]--;}

        // Step 5: Store information needed for inner boundary conditions, including interior point to which
        //         inner ghost zone maps, and parity conditions for all 10 gridfunction types.
#define IB_SET if(bc_gz_map[IDX3S(i0,i1,i2)].i0!=-1) { \
    bcstruct->inner[which_gz][pt].inner_bc_dest_pt.i0=i0; \
    bcstruct->inner[which_gz][pt].inner_bc_dest_pt.i1=i1; \
    bcstruct->inner[which_gz][pt].inner_bc_dest_pt.i2=i2; \
    bcstruct->inner[which_gz][pt].inner_bc_src_pt.i0 =bc_gz_map[IDX3S(i0,i1,i2)].i0; \
    bcstruct->inner[which_gz][pt].inner_bc_src_pt.i1 =bc_gz_map[IDX3S(i0,i1,i2)].i1; \
    bcstruct->inner[which_gz][pt].inner_bc_src_pt.i2 =bc_gz_map[IDX3S(i0,i1,i2)].i2; \
    for(int ii=0;ii<10;ii++) { \
      bcstruct->inner[which_gz][pt].parity[ii] = \
                              (int8_t)bc_parity_conditions[IDX3S(i0,i1,i2)].parity[ii]; } \
    pt++; }

        pt = 0;
        LOOP_REGION(imin[0]-1,imin[0], imin[1],imax[1], imin[2],imax[2]) {IB_SET} imin[0]--;
        LOOP_REGION(imax[0],imax[0]+1, imin[1],imax[1], imin[2],imax[2]) {IB_SET} imax[0]++;
        LOOP_REGION(imin[0],imax[0], imin[1]-1,imin[1], imin[2],imax[2]) {IB_SET} imin[1]--;
        LOOP_REGION(imin[0],imax[0], imax[1],imax[1]+1, imin[2],imax[2]) {IB_SET} imax[1]++;
        LOOP_REGION(imin[0],imax[0], imin[1],imax[1], imin[2]-1,imin[2]) {IB_SET} imin[2]--;
        LOOP_REGION(imin[0],imax[0], imin[1],imax[1], imax[2],imax[2]+1) {IB_SET} imax[2]++;

    } // END for(int which_gz = 0; which_gz < NGHOSTS; which_gz++)
} // END function
```

    Writing CurviBoundaryConditions_Ccodes//boundary_conditions/set_bcstruct.h

# Step 6: `driver_bcstruct.h`: C code driver for declaring `bc_struct` data type, the `bcstruct` instance of said data type, and calling `set_up__bc_gz_map_and_parity_condns()` and `set_bcstruct()` to fill `bcstruct` \[Back to [top](#toc)\]
$$\label{bcstruct_c_code_driver}$$

`CurviBoundaryConditions/driver_bcstruct.h` is meant to be #include'd in a C main() function. It allocates memory for boundary condition pointers and calls functions referenced above, including `set_up__bc_gz_map_and_parity_condns()` and `set_bcstruct()`. It then frees unneeded memory after bcstruct has been fully initialized.

For completeness, we also output

* `CurviBoundaryConditions/bcstruct_freemem.h`, which frees memory allocated within `bcstruct`. It is meant to be #include'd at the end of a C main() function, and
* `CurviBoundaryConditions/CurviBCs_Cfunctions.h`, which #include's all the needed C source functions in order.

```python
%%writefile $Ccodesdir/boundary_conditions/driver_bcstruct.h

// Step 1: Allocate memory storage for bc_gz_map, which
//         in the case a boundary point is a *parity*
//         boundary, is set to the interior, non-
//         boundary point corresponding to the same
//         Cartesian gridpoint. Otherwise bc_gz_map
//         is set to (i0,i1,i2) = (-1,-1,-1).
gz_map *bc_gz_map = (gz_map *)malloc(sizeof(gz_map)*Nxx_plus_2NGHOSTS_tot);

// Step 2: Allocate memory storage for bc_parity_conditions,
//         which store parity conditions for all 10
//         gridfunction types at all grid points.
parity_condition *bc_parity_conditions = (parity_condition *)malloc(sizeof(parity_condition)*Nxx_plus_2NGHOSTS_tot);

// Step 3: Set bc_gz_map and bc_parity_conditions at *all*
//         points; on the boundary and otherwise.
set_up__bc_gz_map_and_parity_condns(&params, xx, bc_gz_map,
                                    bc_parity_conditions
                                   );

// Step 4: Declare and allocate memory for bcstruct,
//         which will store all information needed for
//         applying the boundary conditions.
bcstruct.outer = (outer_bc **)malloc(sizeof(outer_bc *)*NGHOSTS);
bcstruct.inner = (inner_bc **)malloc(sizeof(inner_bc *)*NGHOSTS);
bcstruct.num_ob_gz_pts = (    int *)malloc(sizeof(int)*NGHOSTS);
bcstruct.num_ib_gz_pts = (    int *)malloc(sizeof(int)*NGHOSTS);

// Step 4: Store all information needed to quickly and
//         efficiently apply boundary conditions. This
//         function transfers all information from
//         bc_gz_map (defined at *all gridpoints*) into
//         bcstruct (defined only at boundary points).
//         Thus when this function has finished,
//         bc_gz_map is no longer needed.
set_bcstruct(&params,bc_gz_map,
             bc_parity_conditions,
             &bcstruct);

// Step 5: As described in Step 4, bc_gz_map is no
//         longer needed at this point, so we free its
//         memory. Farewell, friend!
free(bc_gz_map);
free(bc_parity_conditions);
```

    Writing CurviBoundaryConditions_Ccodes//boundary_conditions/driver_bcstruct.h

```python
%%writefile $Ccodesdir/boundary_conditions/bcstruct_freemem.h

for(int i=0;i<NGHOSTS;i++) { free(bcstruct.outer[i]);  free(bcstruct.inner[i]); }
free(bcstruct.num_ob_gz_pts); free(bcstruct.num_ib_gz_pts);
```

    Writing CurviBoundaryConditions_Ccodes//boundary_conditions/bcstruct_freemem.h

```python
%%writefile $Ccodesdir/boundary_conditions/CurviBC_include_Cfunctions.h

#include "BCs_data_structs.h"
#include "EigenCoord_xxCart.h"
#include "set_up__bc_gz_map_and_parity_condns.h"
#include "set_bcstruct.h"
#include "apply_bcs_curvilinear.h"
```

    Writing CurviBoundaryConditions_Ccodes//boundary_conditions/CurviBC_include_Cfunctions.h

# Step 7: `apply_bcs_curvilinear()` C function: quickly apply boundary and parity conditions with information from `bcstruct` \[Back to [top](#toc)\]
$$\label{apply_bcs_curvilinear}$$

`apply_bcs_curvilinear()` loops over all `NUM_GFS` gridfunctions in the `gfs` `IDX4` array and, using a `bcstruct` filled in by the `set_bcstruct()` function above, applies boundary conditions to each ghost zone layer, starting with the innermost layer and working outward.

This function is meant to be called within a [Method of Lines timestepping algorithm](Tutorial-Method_of_Lines-C_Code_Generation.ipynb), at or near the end of each substep.

```python
%%writefile $Ccodesdir/boundary_conditions/apply_bcs_curvilinear.h

// Declare boundary condition BC_UPDATE_OUTER macro,
//          which updates a single outer boundary face
//          of the 3D grid cube using quadratic polynomial
//          extrapolation.

#define BC_UPDATE_OUTER(which_gf, i0,i1,i2, FACEX0,FACEX1,FACEX2) {     \
    const int idx3 = IDX3S(i0,i1,i2);                                   \
    gfs[IDX4S(which_gf,i0,i1,i2)] =                                     \
        +3.0*gfs[IDX4S(which_gf,i0+1*FACEX0,i1+1*FACEX1,i2+1*FACEX2)]   \
        -3.0*gfs[IDX4S(which_gf,i0+2*FACEX0,i1+2*FACEX1,i2+2*FACEX2)]   \
        +1.0*gfs[IDX4S(which_gf,i0+3*FACEX0,i1+3*FACEX1,i2+3*FACEX2)];  \
    }

// Curvilinear boundary condition driver routine: Apply BCs to all six
//          boundary faces of the 3D numerical domain, filling in the
//          innermost ghost zone layer first, and moving outward.
void apply_bcs_curvilinear(const paramstruct *restrict params, const bc_struct *restrict bcstruct,
                           const int NUM_GFS, const int8_t *restrict gfs_parity, REAL *restrict gfs) {
#pragma omp parallel for
    for(int which_gf=0;which_gf<NUM_GFS;which_gf++) {
#include "RELATIVE_PATH__set_Cparameters.h" /* Header file containing correct #include for set_Cparameters.h;
                                             * accounting for the relative path */
        for(int which_gz = 0; which_gz < NGHOSTS; which_gz++) {
            // First apply OUTER boundary conditions,
            //   in case an INNER (parity) boundary point
            //   needs data at the outer boundary:
            // After updating each face, adjust imin[] and imax[]
            //   to reflect the newly-updated face extents.
            for(int pt=0;pt<bcstruct->num_ob_gz_pts[which_gz];pt++) {
                BC_UPDATE_OUTER(which_gf,
                                bcstruct->outer[which_gz][pt].outer_bc_dest_pt.i0,
                                bcstruct->outer[which_gz][pt].outer_bc_dest_pt.i1,
                                bcstruct->outer[which_gz][pt].outer_bc_dest_pt.i2,
                                bcstruct->outer[which_gz][pt].FACEi0,
                                bcstruct->outer[which_gz][pt].FACEi1,
                                bcstruct->outer[which_gz][pt].FACEi2);
            }

            // Then apply INNER (parity) boundary conditions:
            for(int pt=0;pt<bcstruct->num_ib_gz_pts[which_gz];pt++) {
                const int i0dest = bcstruct->inner[which_gz][pt].inner_bc_dest_pt.i0;
                const int i1dest = bcstruct->inner[which_gz][pt].inner_bc_dest_pt.i1;
                const int i2dest = bcstruct->inner[which_gz][pt].inner_bc_dest_pt.i2;
                const int i0src  = bcstruct->inner[which_gz][pt].inner_bc_src_pt.i0;
                const int i1src  = bcstruct->inner[which_gz][pt].inner_bc_src_pt.i1;
                const int i2src  = bcstruct->inner[which_gz][pt].inner_bc_src_pt.i2;
                const int8_t *prty= bcstruct->inner[which_gz][pt].parity;
//                printf("%d\n",bcstruct->inner_bc_parity[which_gz][pt].parity[gfs_parity[which_gf]]);
                gfs[IDX4S(which_gf,i0dest,i1dest,i2dest)] =
                        bcstruct->inner[which_gz][pt].parity[gfs_parity[which_gf]] * gfs[IDX4S(which_gf, i0src,i1src,i2src)];
            } // END for(int pt=0;pt<num_ib_gz_pts[which_gz];pt++)
        } // END for(int which_gz = 0; which_gz < NGHOSTS; which_gz++)
    } // END for(int which_gf=0;which_gf<NUM_GFS;which_gf++)
} // END function
```

    Writing CurviBoundaryConditions_Ccodes//boundary_conditions/apply_bcs_curvilinear.h

# Step 8: `CurviBC_Playground.c`: Start-to-Finish C code module for testing & validating curvilinear boundary conditions \[Back to [top](#toc)\]
$$\label{start2finish}$$

```python
# Set the finite differencing order to 4; although this module doesn't compute
#   finite difference derivatives, it does need to set the number of ghost zone
#   cells, which is generally based on NGHOSTS, which itself depends
#   on finite_difference::FD_CENTDERIVS_ORDER.
import finite_difference as fin  # NRPy+: Finite difference C code generation module
par.set_parval_from_str("finite_difference::FD_CENTDERIVS_ORDER",4)

# Then we set the coordinate system for the numerical grid
par.set_parval_from_str("reference_metric::CoordSystem","SinhSphericalv2")
rfm.reference_metric()
```

## Step 8.a: Generate C code for generating numerical grids \[Back to [top](#toc)\]
$$\label{ccode_numgridgen}$$

Next we generate the C code needed for applying boundary conditions in generic coordinate systems, using the [Eigen-Coordinate](#challenge2) approach described above:
1. $\left(x(x_0,x_1,x_2),y(x_0,x_1,x_2),z(x_0,x_1,x_2)\right)$, (`CurviBoundaryConditions/EigenCoord_xxCart.h`)
1. $\left(x_0(x,y,z),x_1(x,y,z),x_2(x,y,z)\right)$, (`CurviBoundaryConditions/EigenCoord_Cart_to_xx.h`):

```python
# Step 1: Find the Eigen-Coordinate and set up the Eigen-Coordinate's reference metric:
CoordSystem_orig = par.parval_from_str("reference_metric::CoordSystem")
par.set_parval_from_str("reference_metric::CoordSystem",rfm.get_EigenCoord())
rfm.reference_metric()

# Step 2: Output C code for the Eigen-Coordinate mapping from xx->Cartesian:
rfm.xxCart_h("EigenCoord_xxCart",os.path.join(Cparamspath,"set_Cparameters.h"),os.path.join(Ccodesdir,"boundary_conditions","EigenCoord_xxCart.h"))

# Step 3: Output the Eigen-Coordinate mapping from Cartesian->xx:
# Step 3.a: Sanity check: First make sure that rfm.Cart_to_xx has been set. Error out if not!
if rfm.Cart_to_xx[0] == 0 or rfm.Cart_to_xx[1] == 0 or rfm.Cart_to_xx[2] == 0:
    print("ERROR: rfm.Cart_to_xx[], which maps Cartesian -> xx, has not been set for")
    print("       reference_metric::CoordSystem = "+par.parval_from_str("reference_metric::CoordSystem"))
    print("       Boundary conditions in curvilinear coordinates REQUiRE this be set.")
    sys.exit(1)
# Step 3.b: Output C code for the Eigen-Coordinate mapping from Cartesian->xx:
outputC([rfm.Cart_to_xx[0],rfm.Cart_to_xx[1],rfm.Cart_to_xx[2]],
        ["Cart_to_xx0_inbounds","Cart_to_xx1_inbounds","Cart_to_xx2_inbounds"],
        os.path.join(Ccodesdir,"boundary_conditions","EigenCoord_Cart_to_xx.h"))

# Step 4: Restore reference_metric::CoordSystem back to the original CoordSystem
par.set_parval_from_str("reference_metric::CoordSystem",CoordSystem_orig)
rfm.reference_metric()

# Step 5: Generate set_Nxx_dxx_invdx_params__and__xx.h, which sets
#         params Nxx,Nxx_plus_2NGHOSTS,dxx,invdx, and xx[] for the
#         chosen (not necessarily Eigen-) CoordSystem.
rfm.set_Nxx_dxx_invdx_params__and__xx_h(Ccodesdir)

# Step 6: Generate declare_Cparameters_struct.h, set_Cparameters_default.h, and set_Cparameters[-SIMD].h
par.generate_Cparameters_Ccodes(Ccodesdir)
```

    Wrote to file "CurviBoundaryConditions_Ccodes/boundary_conditions/EigenCoord_Cart_to_xx.h"

## Step 8.b: Register gridfunctions of all 10 parity types; output gridfunction aliases to `CurviBoundaryConditions/gridfunction_defines.h` \[Back to [top](#toc)\]
$$\label{register_gfs}$$

Here we

1. Register within NRPy+ one gridfunction per each of the 10 parity conditions, and then
1. output to `CurviBoundaryConditions/gridfunction_defines.h` the corresponding gridfunction aliases and parity info, so the C code can
    1. Access each gridfunction by its human-friendly alias (e.g., `test_gfs[RANKONEU0GF][idx]` instead of `test_gfs[6][idx]`), and
    1. Access each gridfunction parity by the same alias (e.g., `bcstruct->inner_bc_parity[which_gz][pt].parity[gf_parity[RANKONEU0GF]]`)

```python
# Step 8.b.1: Register gridfunctions of all 10 parity types

# 6 gridfunctions, corresponding to all unique rank-2 tensor components:
ranktwosymmDD = ixp.register_gridfunctions_for_single_rank2("AUX","ranktwosymmDD", "sym01")
# 3 gridfunctions, corresponding to all unique rank-1 tensor components:
rankoneU = ixp.register_gridfunctions_for_single_rank1("AUX","rankoneU")
# 1 rank-0 (scalar) gridfunction
rankzero = ixp.gri.register_gridfunctions("AUX","rankzero")

# Step 8.b.2. Output gridfunction #define aliases to file:

# import parity_conditions_symbolic_dot_products as parity
parity_conditions_symbolic_dot_products(os.path.join(Ccodesdir,"boundary_conditions/"))
# import set_gridfunction_defines_with_parity_types as gfdefineparity
set_gridfunction_defines_with_parity_types(os.path.join(Ccodesdir,"boundary_conditions/"),verbose=True)
```

    Wrote to file "CurviBoundaryConditions_Ccodes/boundary_conditions/parity_conditions_symbolic_dot_products.h"
    Auxiliary gridfunction "rankoneU0" has parity type 1.
    Auxiliary gridfunction "rankoneU1" has parity type 2.
    Auxiliary gridfunction "rankoneU2" has parity type 3.
    Auxiliary gridfunction "ranktwosymmDD00" has parity type 4.
    Auxiliary gridfunction "ranktwosymmDD01" has parity type 5.
    Auxiliary gridfunction "ranktwosymmDD02" has parity type 6.
    Auxiliary gridfunction "ranktwosymmDD11" has parity type 7.
    Auxiliary gridfunction "ranktwosymmDD12" has parity type 8.
    Auxiliary gridfunction "ranktwosymmDD22" has parity type 9.
    Auxiliary gridfunction "rankzero" has parity type 0.

## Step 8.c: Set up test data for Curvilinear Boundary Conditions code validation \[Back to [top](#toc)\]
$$\label{validate}$$

We will validate this curvilinear boundary condition module by comparing its results with the original (trusted) SENR code, as follows:

* **Discrete data test**:
    1. Fill all 10 gridfunctions at each gridpoint with the unique gridpoint integer index `IDX3S(i0,i1,i2)`
    1. Apply curvilinear boundary conditions
    1. Compare output data at all gridpoints with those from the original SENR code. Agreement should be perfect.

Another (future, to-be-implemented) test, which will enable us to validate coordinate systems that do not exist within the original SENR code, is described below:

* **Smooth data test** (TODO):
    1. Fill all 10 gridfunctions with data that are smooth in the Cartesian basis.
    1. Apply Jacobian transformation to all data points, to convert to curvilinear basis
    1. Apply curvilinear boundary conditions
    1. Apply Jacobian transformation to all data points, to convert back to Cartesian basis
    1. Compute difference between original Cartesian data and transformed data. Difference should be zero (to within roundoff) at all points except those that are influenced by outer boundary conditions.

```python
%%writefile $Ccodesdir/CurviBC_Discrete_Test.h

void Discrete_initial_data(paramstruct *params,REAL *in_gfs) {
#include "set_Cparameters.h"
#pragma omp parallel for
    for(int i=0;i<Nxx_plus_2NGHOSTS0*Nxx_plus_2NGHOSTS1*Nxx_plus_2NGHOSTS2*NUM_AUX_GFS;i++) {
        in_gfs[i] = (REAL)i;
    }
}
```

    Writing CurviBoundaryConditions_Ccodes//CurviBC_Discrete_Test.h

# Step 8.d: `CurviBC_Playground.c`: The Main C Code \[Back to [top](#toc)\]
$$\label{mainc}$$

```python
# Part P0: Set the number of ghost cells, from NRPy+'s FD_CENTDERIVS_ORDER
with open(os.path.join(Ccodesdir,"NGHOSTS.h"), "w") as file:
    file.write("// Part P0: Set the number of ghost zones, from NRPy+'s FD_CENTDERIVS_ORDER\n")
    # Upwinding in BSSN requires that NGHOSTS = FD_CENTDERIVS_ORDER/2 + 1 <- Notice the +1.
    file.write("#define NGHOSTS "+str(int(par.parval_from_str("finite_difference::FD_CENTDERIVS_ORDER")/2)+1)+"\n")
```

```python
%%writefile $Ccodesdir/CurviBC_Playground.c

// Step P0: Define REAL and NGHOSTS; and declare CFL_FACTOR. This header is generated in NRPy+.
#include "NGHOSTS.h"
#define REAL double

#include "declare_Cparameters_struct.h"

// Step P1: Import needed header files
#include "stdio.h"
#include "stdlib.h"
#include "math.h"
#include "string.h" // Needed for strncmp, etc.
#include "stdint.h" // Needed for Windows GCC 6.x compatibility
#ifndef M_PI
#define M_PI 3.141592653589793238462643383279502884L
#endif
#ifndef M_SQRT1_2
#define M_SQRT1_2 0.707106781186547524400844362104849039L
#endif

// Step P2: Declare the IDX4S(gf,i,j,k) macro, which enables us to store 4-dimensions of
//           data in a 1D array. In this case, consecutive values of "i"
//           (all other indices held to a fixed value) are consecutive in memory, where
//           consecutive values of "j" (fixing all other indices) are separated by
//           Nxx_plus_2NGHOSTS0 elements in memory. Similarly, consecutive values of
//           "k" are separated by Nxx_plus_2NGHOSTS0*Nxx_plus_2NGHOSTS1 in memory, etc.
#define IDX4S(g,i,j,k) \
( (i) + Nxx_plus_2NGHOSTS0 * ( (j) + Nxx_plus_2NGHOSTS1 * ( (k) + Nxx_plus_2NGHOSTS2 * (g) ) ) )
#define IDX3S(i,j,k) ( (i) + Nxx_plus_2NGHOSTS0 * ( (j) + Nxx_plus_2NGHOSTS1 * ( (k) ) ) )
#define LOOP_REGION(i0min,i0max, i1min,i1max, i2min,i2max) \
  for(int i2=i2min;i2<i2max;i2++) for(int i1=i1min;i1<i1max;i1++) for(int i0=i0min;i0<i0max;i0++)

// Step P3: Set #define's for all gridfunctions registered within NRPy+. C code generated above
#include "boundary_conditions/gridfunction_defines.h"

#include "set_Nxx_dxx_invdx_params__and__xx.h"

// Step P4: Include basic functions needed to impose curvilinear
//          parity and boundary conditions.
#include "boundary_conditions/CurviBC_include_Cfunctions.h"

// Step P5: Declare the function for the exact solution. time==0 corresponds to the initial data.
#include "CurviBC_Discrete_Test.h"

// main() function:
// Step 0: Read command-line input, set up grid structure, allocate memory for gridfunctions, set up coordinates
// Step 1: Write test data to gridfunctions
// Step 2: Overwrite all data in ghost zones with NaNs
// Step 3: Output relative error between numerical and exact solution.
// Step 4: Print gridfunction data after curvilinear boundary conditions have been applied:
// Step 5: Free all allocated memory
int main(int argc, const char *argv[]) {
    paramstruct params;
#include "set_Cparameters_default.h"

    // Step 0a: Read command-line input, error out if nonconformant
    if(argc != 5 || atoi(argv[1]) < NGHOSTS || atoi(argv[2]) < NGHOSTS || atoi(argv[3]) < NGHOSTS) {
        fprintf(stderr,"Error: Expected one command-line argument: ./CurviBC_Playground Nx0 Nx1 Nx2 [test type: Smooth or Discrete],\n");
        fprintf(stderr,"where Nx[0,1,2] is the number of grid points in the 0, 1, and 2 directions.\n");
        fprintf(stderr,"Nx[] MUST BE larger than NGHOSTS (= %d)\n",NGHOSTS);
        exit(1);
    }

    // Step 0b: Set up numerical grid structure...
    const int Nxx[3] = { atoi(argv[1]), atoi(argv[2]), atoi(argv[3]) };
    if(Nxx[0]%2 != 0 || Nxx[1]%2 != 0 || Nxx[2]%2 != 0) {
        fprintf(stderr,"Error: Cannot guarantee a proper cell-centered grid if number of grid cells not set to even number.\n");
        fprintf(stderr,"       For example, in case of angular directions, proper symmetry zones will not exist.\n");
        exit(1);
    }

    // Step 0c: Set test type to Smooth or Discrete
    char test_type[100];
    snprintf(test_type,100,"%s",argv[4]);
    if(strncmp("Smooth",test_type,100)!=0 && strncmp("Discrete",test_type,100)!=0) {
        fprintf(stderr,"Error: test type = %s not supported. Choose Smooth or Discrete.\n",test_type);
        exit(1);
    }

    // Step 0d: Uniform coordinate grids are stored to *xx[3]
    REAL *xx[3];
    // Step 0d.i: Set bcstruct
    bc_struct bcstruct;
    {
        int EigenCoord = 1;
        // Step 0d.ii: Call set_Nxx_dxx_invdx_params__and__xx(), which sets
        //             params Nxx,Nxx_plus_2NGHOSTS,dxx,invdx, and xx[] for the
        //             chosen Eigen-CoordSystem.
        set_Nxx_dxx_invdx_params__and__xx(EigenCoord, Nxx, &params, xx);
        // Step 0d.iii: Set Nxx_plus_2NGHOSTS_tot
#include "set_Cparameters-nopointer.h"
        const int Nxx_plus_2NGHOSTS_tot = Nxx_plus_2NGHOSTS0*Nxx_plus_2NGHOSTS1*Nxx_plus_2NGHOSTS2;
        // Step 0e: Find ghostzone mappings; set up bcstruct
#include "boundary_conditions/driver_bcstruct.h"
        // Step 0e.i: Free allocated space for xx[][] array
        for(int i=0;i<3;i++) free(xx[i]);
    }

    // Step 0f: Call set_Nxx_dxx_invdx_params__and__xx(), which sets
    //          params Nxx,Nxx_plus_2NGHOSTS,dxx,invdx, and xx[] for the
    //          chosen (non-Eigen) CoordSystem.
    int EigenCoord = 0;
    set_Nxx_dxx_invdx_params__and__xx(EigenCoord, Nxx, &params, xx);

    // Step 0g: Set all C parameters "blah" for params.blah, including
    //          Nxx_plus_2NGHOSTS0 = params.Nxx_plus_2NGHOSTS0, etc.
#include "set_Cparameters-nopointer.h"
    const int Nxx_plus_2NGHOSTS_tot = Nxx_plus_2NGHOSTS0*Nxx_plus_2NGHOSTS1*Nxx_plus_2NGHOSTS2;

    // Step 0h: Allocate memory for gridfunctions
    REAL *test_gfs = (REAL *)malloc(sizeof(REAL) * NUM_AUX_GFS * Nxx_plus_2NGHOSTS_tot);

    // Step 1: Write test data to gridfunctions
    if(strncmp("Discrete",test_type,100)==0) {
        Discrete_initial_data(&params, test_gfs);
    } else {
        fprintf(stderr,"Sorry, curvilinear boundary conditions test = %s not yet supported. Feel free to contribute!\n",test_type);
    }

    // Step 2: Overwrite all data in ghost zones with NaNs
    LOOP_REGION(0,Nxx_plus_2NGHOSTS0, 0,Nxx_plus_2NGHOSTS1, 0,Nxx_plus_2NGHOSTS2) {
        for(int gf=0;gf<NUM_AUX_GFS;gf++) {
            const int idx4 = IDX4S(gf,i0,i1,i2);
            if(i0 < NGHOSTS || i0 >= Nxx_plus_2NGHOSTS0-NGHOSTS) test_gfs[idx4] = +(0.0 / 0.0);
            if(i1 < NGHOSTS || i1 >= Nxx_plus_2NGHOSTS1-NGHOSTS) test_gfs[idx4] = +(0.0 / 0.0);
            if(i2 < NGHOSTS || i2 >= Nxx_plus_2NGHOSTS2-NGHOSTS) test_gfs[idx4] = +(0.0 / 0.0);
        }
    }

    // Step 3: Apply curvilinear boundary conditions
    apply_bcs_curvilinear(&params, &bcstruct, NUM_AUX_GFS, aux_gf_parity, test_gfs);

    // Step 4: Print gridfunction data after curvilinear boundary conditions have been applied:
    LOOP_REGION(0,Nxx_plus_2NGHOSTS0, 0,Nxx_plus_2NGHOSTS1, 0,Nxx_plus_2NGHOSTS2) {
        printf("%d %d %d | ",i0,i1,i2);
        for(int gf=0;gf<NUM_AUX_GFS;gf++) {
            const int idx4 = IDX4S(gf,i0,i1,i2);
            if(!isnan(test_gfs[idx4])) {
                printf("%d ",(int)test_gfs[idx4]);
            } else {
                fprintf(stderr,"ERROR: found NaN %d %d %d %d %d\n",gf,i0,i1,i2,NUM_AUX_GFS);
                exit(1);
            }
        }
        printf("\n");
    }

    // Step 5: Free all allocated memory
#include "boundary_conditions/bcstruct_freemem.h"
    free(test_gfs);
    for(int i=0;i<3;i++) free(xx[i]);
    return 0;
}
```

    Writing CurviBoundaryConditions_Ccodes//CurviBC_Playground.c

# Step 9: Validation: Compile & compare with original (trusted) SENR results \[Back to [top](#toc)\]
$$\label{senr_compare}$$

```python
import cmdline_helper as cmd
cmd.C_compile(os.path.join(Ccodesdir,"CurviBC_Playground.c"), "CurviBC_Playground", compile_mode="safe")
cmd.Execute("CurviBC_Playground", "4 4 4 Discrete", os.path.join(Ccodesdir,"out4x4x4-Spherical-NGHOSTS4oFD.txt"))
print("Finished this code cell.")
```

    Compiling executable...
    Executing `gcc -O2 -g -fopenmp CurviBoundaryConditions_Ccodes/CurviBC_Playground.c -o CurviBC_Playground -lm`...
    Finished executing in 0.4106414318084717 seconds.
    Finished compilation.
    Executing `taskset -c 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15 ./CurviBC_Playground 4 4 4 Discrete`...
    Finished executing in 0.20898818969726562 seconds.
    Finished this code cell.

```python
import filecmp
if filecmp.cmp(os.path.join(Ccodesdir,"out4x4x4-Spherical-NGHOSTS4oFD.txt"),
               os.path.join("CurviBoundaryConditions","SENRout4x4x4-Spherical_NGHOSTS4oFD.txt")) == False:
    print("ERROR: Spherical boundary conditions do not agree!")
    sys.exit(1)
else:
    print("Spherical boundary condition comparison test between this tutorial notebook & trusted original SENR code: PASSED")
```

    Spherical boundary condition comparison test between this tutorial notebook & trusted original SENR code: PASSED

```python
import filecmp
import CurviBoundaryConditions.CurviBoundaryConditions as cbcs
cbcs.Set_up_CurviBoundaryConditions(os.path.join(Ccodesdir,"boundary_conditions/"),verbose=False,
                                   enable_copy_of_static_Ccodes=False) # Use static C codes generated above instead!

print("Running discrete CurviBC_Playground test on Spherical data, using NRPy+ CurviBoundaryConditions.CurviBoundaryConditions module.")
cmd.C_compile(os.path.join(Ccodesdir,"CurviBC_Playground.c"), "CurviBC_Playground")
cmd.Execute("CurviBC_Playground", "4 4 4 Discrete",
            os.path.join(Ccodesdir,"CBCmodule-out4x4x4-Spherical-NGHOSTS4oFD.txt"))

# Then we set the coordinate system for the numerical grid
par.set_parval_from_str("reference_metric::CoordSystem","SinhCylindricalv2")
rfm.reference_metric() # Not necessary to call this.
cbcs.Set_up_CurviBoundaryConditions(os.path.join(Ccodesdir,"boundary_conditions/"),verbose=False,
                                    enable_copy_of_static_Ccodes=False) # Use static C codes generated above instead!

# Step 5: Generate set_Nxx_dxx_invdx_params__and__xx.h, which sets
#         params Nxx,Nxx_plus_2NGHOSTS,dxx,invdx, and xx[] for the
#         chosen (not necessarily Eigen-) CoordSystem.
rfm.set_Nxx_dxx_invdx_params__and__xx_h(Ccodesdir)
# Step 6: Generate declare_Cparameters_struct.h, set_Cparameters_default.h, and set_Cparameters[-SIMD].h
par.generate_Cparameters_Ccodes(Ccodesdir)
print("Running discrete CurviBC_Playground test on Cylindrical data, using NRPy+ CurviBoundaryConditions.CurviBoundaryConditions module.")
cmd.C_compile(os.path.join(Ccodesdir,"CurviBC_Playground.c"), "CurviBC_Playground")
cmd.Execute("CurviBC_Playground", "4 4 4 Discrete", os.path.join(Ccodesdir,"CBCmodule-out4x4x4-Cylindrical-NGHOSTS4oFD.txt"))
print("Finished this code cell.")

print("\n")

if filecmp.cmp(os.path.join(Ccodesdir,"CBCmodule-out4x4x4-Spherical-NGHOSTS4oFD.txt"),
               os.path.join("CurviBoundaryConditions","SENRout4x4x4-Spherical_NGHOSTS4oFD.txt")) == False:
    print("ERROR: Spherical boundary conditions do not agree!")
    sys.exit(1)
else:
    print("Spherical boundary condition comparison test between CBC & trusted original SENR code: PASSED")

if filecmp.cmp(os.path.join(Ccodesdir,"CBCmodule-out4x4x4-Cylindrical-NGHOSTS4oFD.txt"),
               os.path.join("CurviBoundaryConditions","SENRout4x4x4-Cylindrical_NGHOSTS4oFD.txt")) == False:
    print("ERROR: Cylindrical boundary conditions do not agree!")
    sys.exit(1)
else:
    print("Cylindrical boundary condition comparison test between CBC & trusted original SENR code: PASSED")
```

    Wrote to file "CurviBoundaryConditions_Ccodes/boundary_conditions/parity_conditions_symbolic_dot_products.h"
    Wrote to file "CurviBoundaryConditions_Ccodes/boundary_conditions/EigenCoord_Cart_to_xx.h"
    Running discrete CurviBC_Playground test on Spherical data, using NRPy+ CurviBoundaryConditions.CurviBoundaryConditions module.
    Compiling executable...
    Executing `gcc -Ofast -fopenmp -march=native -funroll-loops CurviBoundaryConditions_Ccodes/CurviBC_Playground.c -o CurviBC_Playground -lm`...
    Finished executing in 0.8112778663635254 seconds.
    Finished compilation.
    Executing `taskset -c 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15 ./CurviBC_Playground 4 4 4 Discrete`...
    Finished executing in 0.20944476127624512 seconds.
    Wrote to file "CurviBoundaryConditions_Ccodes/boundary_conditions/parity_conditions_symbolic_dot_products.h"
    Wrote to file "CurviBoundaryConditions_Ccodes/boundary_conditions/EigenCoord_Cart_to_xx.h"
    Running discrete CurviBC_Playground test on Cylindrical data, using NRPy+ CurviBoundaryConditions.CurviBoundaryConditions module.
    Compiling executable...
    Executing `gcc -Ofast -fopenmp -march=native -funroll-loops CurviBoundaryConditions_Ccodes/CurviBC_Playground.c -o CurviBC_Playground -lm`...
    Finished executing in 0.8110315799713135 seconds.
    Finished compilation.
    Executing `taskset -c 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15 ./CurviBC_Playground 4 4 4 Discrete`...
    Finished executing in 0.20937156677246094 seconds.
    Finished this code cell.

    Spherical boundary condition comparison test between CBC & trusted original SENR code: PASSED
    Cylindrical boundary condition comparison test between CBC & trusted original SENR code: PASSED

# Step 10: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-Start_to_Finish-Curvilinear_BCs.pdf](Tutorial-Start_to_Finish-Curvilinear_BCs.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
!jupyter nbconvert --to latex --template latex_nrpy_style.tplx --log-level='WARN' Tutorial-Start_to_Finish-Curvilinear_BCs.ipynb
!pdflatex -interaction=batchmode Tutorial-Start_to_Finish-Curvilinear_BCs.tex
!pdflatex -interaction=batchmode Tutorial-Start_to_Finish-Curvilinear_BCs.tex
!pdflatex -interaction=batchmode Tutorial-Start_to_Finish-Curvilinear_BCs.tex
!rm -f Tut*.out Tut*.aux Tut*.log
```

    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode


```python
from pycalphad.tests.datasets import *
from pycalphad.core.solver import InteriorPointSolver
from pycalphad import equilibrium, Database, Model, variables as v
import sympy

ALFE_DBF = Database(ALFE_TDB)
ISSUE43_DBF = Database('issue43.tdb')

class ProblemSaver(InteriorPointSolver):
    saved_problem = [None]
    def solve(self, prob):
        self.saved_problem[0] = prob
        self.verbose = True
        return super(ProblemSaver, self).solve(prob)

my_phases = ['FCC_A1', 'GAMMA_PRIME']
comps = ['AL', 'NI', 'CR', 'VA']
comps = sorted(comps)
conds = dict({v.T: 800, v.P: 101325, v.N: 1})

mod = Model(ISSUE43_DBF, comps, 'FCC_A1')

NP = sympy.Symbol('NP', real=True)
total_moles = sum([NP*mod.moles(c) for c in comps])
total_moles = NP

variables = [v.N, v.P, v.T] + mod.site_fractions + [NP]

mass_cons = [v.N, v.P, v.T]
mass_cons.extend(mod.get_internal_constraints())
mass_cons.extend(NP*mod.moles(c) for c in comps)

mass_jac = []
for cons in mass_cons:
    mass_jac.append([cons.diff(x) for x in variables])

energy_grad = [(total_moles*mod.GM).diff(x) for x in variables]
```

```python
mass_cons
```

    [N,
     P,
     T,
     FCC_A10AL + FCC_A10CR + FCC_A10NI - 1,
     FCC_A11VA - 1,
     1.0*FCC_A10AL*NP/(1.0*FCC_A10AL + 1.0*FCC_A10CR + 1.0*FCC_A10NI),
     1.0*FCC_A10CR*NP/(1.0*FCC_A10AL + 1.0*FCC_A10CR + 1.0*FCC_A10NI),
     1.0*FCC_A10NI*NP/(1.0*FCC_A10AL + 1.0*FCC_A10CR + 1.0*FCC_A10NI),
     0]

```python
mass_jac
```

    [[1, 0, 0, 0, 0, 0, 0, 0],
     [0, 1, 0, 0, 0, 0, 0, 0],
     [0, 0, 1, 0, 0, 0, 0, 0],
     [0, 0, 0, 1, 1, 1, 0, 0],
     [0, 0, 0, 0, 0, 0, 1, 0],
     [0,
      0,
      0,
      -1.0*FCC_A10AL*NP/(1.0*FCC_A10AL + 1.0*FCC_A10CR + 1.0*FCC_A10NI)**2 + 1.0*NP/(1.0*FCC_A10AL + 1.0*FCC_A10CR + 1.0*FCC_A10NI),
      -1.0*FCC_A10AL*NP/(1.0*FCC_A10AL + 1.0*FCC_A10CR + 1.0*FCC_A10NI)**2,
      -1.0*FCC_A10AL*NP/(1.0*FCC_A10AL + 1.0*FCC_A10CR + 1.0*FCC_A10NI)**2,
      0,
      1.0*FCC_A10AL/(1.0*FCC_A10AL + 1.0*FCC_A10CR + 1.0*FCC_A10NI)],
     [0,
      0,
      0,
      -1.0*FCC_A10CR*NP/(1.0*FCC_A10AL + 1.0*FCC_A10CR + 1.0*FCC_A10NI)**2,
      -1.0*FCC_A10CR*NP/(1.0*FCC_A10AL + 1.0*FCC_A10CR + 1.0*FCC_A10NI)**2 + 1.0*NP/(1.0*FCC_A10AL + 1.0*FCC_A10CR + 1.0*FCC_A10NI),
      -1.0*FCC_A10CR*NP/(1.0*FCC_A10AL + 1.0*FCC_A10CR + 1.0*FCC_A10NI)**2,
      0,
      1.0*FCC_A10CR/(1.0*FCC_A10AL + 1.0*FCC_A10CR + 1.0*FCC_A10NI)],
     [0,
      0,
      0,
      -1.0*FCC_A10NI*NP/(1.0*FCC_A10AL + 1.0*FCC_A10CR + 1.0*FCC_A10NI)**2,
      -1.0*FCC_A10NI*NP/(1.0*FCC_A10AL + 1.0*FCC_A10CR + 1.0*FCC_A10NI)**2,
      -1.0*FCC_A10NI*NP/(1.0*FCC_A10AL + 1.0*FCC_A10CR + 1.0*FCC_A10NI)**2 + 1.0*NP/(1.0*FCC_A10AL + 1.0*FCC_A10CR + 1.0*FCC_A10NI),
      0,
      1.0*FCC_A10NI/(1.0*FCC_A10AL + 1.0*FCC_A10CR + 1.0*FCC_A10NI)],
     [0, 0, 0, 0, 0, 0, 0, 0]]

```python
A = sympy.Matrix(mass_jac).T.pinv()
x = A * sympy.Matrix(energy_grad)
```

```python
from pycalphad.codegen.sympydiff_utils import build_functions

mu_a = build_functions(x[4], variables, include_grad=True, include_hess=False)
mu_b = build_functions(x[5], variables, include_grad=True, include_hess=False)
mu_c = build_functions(x[6], variables, include_grad=True, include_hess=False)

energy = build_functions(mod.GM, variables, include_grad=True)
```

```python
print(x[4].free_symbols)
```

    {TEST0A, TEST0B, NP, T, TEST0C}

```python
mu_a.func([1, 1e5, 1000, 0.4, 0.6, 1e-12, 1e-6])
```

    array(-7618.14919886)

```python
np.array(mu_a.grad([1, 1e5, 1000, 0.4, 0.6, 1e-12, 1])) - np.array(mu_b.grad([1, 1e5, 1000, 0.4, 0.6, 1e-12, 1]))
```

    array([ 0.00000000e+00,  0.00000000e+00, -3.37123964e+00,  2.07862500e+04,
           -1.38575000e+04,  9.23094935e-01, -9.44368765e-10])

```python
from pycalphad.core.solver import InteriorPointSolver

class ProblemSaver(InteriorPointSolver):
    saved_problem = [None]
    def solve(self, prob):
        self.saved_problem[0] = prob
        self.verbose = True
        return super(ProblemSaver, self).solve(prob)

eq = equilibrium(dbf, ['A', 'B', 'C'], ['TEST'],
                     {v.MU('B'): -1000, v.X('A'): 0.1, v.T: 800, v.P: 101325}, solver=ProblemSaver())
```

    Chemical Potentials [-15315.87500456  -1000.         -21480.14360388]
    [0. 0. 0. 0. 0. 0. 0.]
    [1.00000000e+00 1.01325000e+05 8.00000000e+02 1.00000000e-01
     8.60415585e-01 3.95844148e-02 1.00000000e+00]
    Status: 0 b'Algorithm terminated successfully at a locally optimal point, satisfying the convergence tolerances (can be specified by options).'

```python
ProblemSaver.saved_problem[0].jacobian([1, 1e5, 800, 0.1, 8.60415585e-01, 3.95844148e-2, 1.0])[-1]
```

    array([ 0.00000000e+00,  0.00000000e+00, -1.25000000e+00,  2.59387889e-09,
            7.73068284e+03,  1.93540473e-09, -8.77662387e-10])

```python
mu_b.grad([1, 1e5, 800, 0.1, 8.60415585e-01, 3.95844148e-2, 1.0])
```

    [array(0.),
     array(0.),
     array(-1.25),
     array(-3.52429197e-12),
     array(7730.68284206),
     array(2.79669621e-11),
     array(2.33626452e-11)]

```python

```