# NRPy+'s Reference Metric Interface

## Author: Zach Etienne
### Formatting improvements courtesy Brandon Clark

### NRPy+ Source Code for this module: [reference_metric.py](../edit/reference_metric.py)

## Introduction:
### Why use a reference metric? Benefits of choosing the best coordinate system for the problem

When solving a partial differential equation on the computer, it is useful to first pick a coordinate system well-suited to the geometry of the problem. For example, if we are modeling a spherically-symmetric star, it would be hugely wasteful to model the star in 3-dimensional Cartesian coordinates ($x$,$y$,$z$). This is because in Cartesian coordinates, we would need to choose high sampling in all three Cartesian directions. If instead we chose to model the star in spherical coordinates  ($r$,$\theta$,$\phi$), so long as the star is centered at $r=0$, we would not need to model the star with more than one point in the $\theta$ and $\phi$ directions!

A similar argument holds for stars that are *nearly* spherically symmetric. Such stars may exhibit density distributions that vary slowly in $\theta$ and $\phi$ directions (e.g., isolated neutron stars or black holes). In these cases the number of points needed to sample the angular directions will still be much smaller than in the radial direction.

Thus choice of an appropriate reference metric may directly mitigate the [Curse of Dimensionality](https://en.wikipedia.org/wiki/Curse_of_dimensionality).

# Table of Contents
$$\label{toc}$$

This notebook is organized as follow

1. [Step 1](#define_ref_metric): Defining a reference metric, [`reference_metric.py`](../edit/reference_metric.py)
1. [Step 2](#define_geometric): Defining geometric quantities, **`ref_metric__hatted_quantities()`**
1. [Step 3](#prescribed_ref_metric): Prescribed reference metrics in [`reference_metric.py`](../edit/reference_metric.py)
    1. [Step 3.a](#sphericallike): Spherical-like coordinate systems
        1. [Step 3.a.i](#spherical): **`reference_metric::CoordSystem = "Spherical"`**
        1. [Step 3.a.ii](#sinhspherical): **`reference_metric::CoordSystem = "SinhSpherical"`**
        1. [Step 3.a.iii](#sinhsphericalv2): **`reference_metric::CoordSystem = "SinhSphericalv2"`**
    1. [Step 3.b](#cylindricallike): Cylindrical-like coordinate systems
        1. [Step 3.b.i](#cylindrical): **`reference_metric::CoordSystem = "Cylindrical"`**
        1. [Step 3.b.ii](#sinhcylindrical): **`reference_metric::CoordSystem = "SinhCylindrical"`**
        1. [Step 3.b.iii](#sinhcylindricalv2): **`reference_metric::CoordSystem = "SinhCylindricalv2"`**
    1. [Step 3.c](#cartesianlike): Cartesian-like coordinate systems
        1. [Step 3.c.i](#cartesian): **`reference_metric::CoordSystem = "Cartesian"`**
        1. [Step 3.c.ii](#sinhcartesian): **`reference_metric::CoordSystem = "SinhCartesian"`**
    1. [Step 3.d](#prolatespheroidal): Prolate spheroidal coordinates
        1. [Step 3.d.i](#symtp): **`reference_metric::CoordSystem = "SymTP"`**
        1. [Step 3.d.ii](#sinhsymtp): **`reference_metric::CoordSystem = "SinhSymTP"`**
1. [Step 4](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Step 1: Defining a reference metric, [`reference_metric.py`](../edit/reference_metric.py) \[Back to [top](#toc)\]
$$\label{define_ref_metric}$$

***Note that currently only orthogonal reference metrics of dimension 3 or fewer are supported. This can be extended if desired.***

NRPy+ assumes all curvilinear coordinate systems map directly from a uniform, Cartesian numerical grid with coordinates $(x,y,z)$=(`xx[0]`,`xx[1]`,`xx[2]`). Thus when defining reference metrics, all defined coordinate quantities must be in terms of the `xx[]` array. As we will see, this adds a great deal of flexibility

For example,  [**reference_metric.py**](../edit/reference_metric.py) requires that the *orthogonal coordinate scale factors* be defined. As described [here](https://en.wikipedia.org/wiki/Curvilinear_coordinates), the $i$th scale factor is the positive root of the metric element $g_{ii}$. In ordinary spherical coordinates $(r,\theta,\phi)$, with line element $ds^2 = g_{ij} dx^i dx^j = dr^2+ r^2 d \theta^2 + r^2 \sin^2\theta \ d\phi^2$, we would first define
* $r = xx_0$
* $\theta = xx_1$
* $\phi = xx_2$,

so that the scale factors are defined as
* `scalefactor_orthog[0]` = $1$
* `scalefactor_orthog[1]` = $r$
* `scalefactor_orthog[2]` = $r \sin \theta$

Here is the corresponding code:

```python
import sympy as sp               # SymPy: The Python computer algebra package upon which NRPy+ depends
import NRPy_param_funcs as par   # NRPy+: parameter interface
import reference_metric as rfm   # NRPy+: Reference metric support

r = rfm.xx[0]
th = rfm.xx[1]
ph = rfm.xx[2]

rfm.scalefactor_orthog[0] = 1
rfm.scalefactor_orthog[1] = r
rfm.scalefactor_orthog[2] = r*sp.sin(th)

# Notice that the scale factor will be given
#    in terms of the fundamental Cartesian
#    grid variables, and not {r,th,ph}:
print("r*sin(th) = "+str(rfm.scalefactor_orthog[2]))
```

    r*sin(th) = xx0*sin(xx1)

Next suppose we wish to modify our radial coordinate $r(xx_0)$ to be an exponentially increasing function, so that our numerical grid $(xx_0,xx_1,xx_2)$ will map to a spherical grid with radial grid spacing ($\Delta r$) that *increases* with $r$. Generally we will find it useful to define $r(xx_0)$ to be an odd function, so let's choose

$$r(xx_0) = a \sinh(xx_0/s),$$

where $a$ is an overall radial scaling factor, and $s$ denotes the scale (in units of $xx_0$) over which exponential growth will take place. In our implementation below, note that we use the relation

$$\sinh(x) = \frac{e^x - e^{-x}}{2},$$

as SymPy finds it easier to evaluate exponentials than hyperbolic trigonometric functions.

```python
a,s = sp.symbols('a s',positive=True)
xx0_rescaled = rfm.xx[0] / s
r = a*(sp.exp(xx0_rescaled) - sp.exp(-xx0_rescaled))/2

# Must redefine the scalefactors since 'r' has been updated!
rfm.scalefactor_orthog[0] = 1
rfm.scalefactor_orthog[1] = r
rfm.scalefactor_orthog[2] = r*sp.sin(th)

print(rfm.scalefactor_orthog[2])
```

    a*(exp(xx0/s) - exp(-xx0/s))*sin(xx1)/2

Often we will find it useful to also define the appropriate mappings from (`xx[0]`,`xx[1]`,`xx[2]`) to Cartesian coordinates (for plotting purposes) and ordinary spherical coordinates (e.g., in case initial data when solving a PDE are naturally written in spherical coordinates). For this purpose, reference_metric.py also declares lists **`xx_to_Cart[]`** and **`xxSph[]`**, which in this case are defined as

```python
rfm.xxSph[0] = r
rfm.xxSph[1] = th
rfm.xxSph[2] = ph

rfm.xx_to_Cart[0] = r*sp.sin(th)*sp.cos(ph)
rfm.xx_to_Cart[1] = r*sp.sin(th)*sp.sin(ph)
rfm.xx_to_Cart[2] = r*sp.cos(th)

# Here we show off SymPy's pretty_print()
#   and simplify() functions. Nice, no?
sp.pretty_print(sp.simplify(rfm.xx_to_Cart[0]))
```

                            ⎛xx₀⎞
    a⋅sin(xx₁)⋅cos(xx₂)⋅sinh⎜───⎟
                            ⎝ s ⎠

# Step 2: Define geometric quantities, `ref_metric__hatted_quantities()` \[Back to [top](#toc)\]
$$\label{define_geometric}$$

Once `scalefactor_orthog[]` has been defined, the function **`ref_metric__hatted_quantities()`** within [reference_metric.py](../edit/reference_metric.py) can be called to define a number of geometric quantities useful for solving PDEs in curvilinear coordinate systems.

Adopting the notation of [Baumgarte, Montero, Cordero-Carrión, and Müller, PRD 87, 044026 (2012)](https://arxiv.org/abs/1211.6632), geometric quantities related to the reference metric are named "hatted" quantities, . For example, the reference metric is defined as $\hat{g}_{ij}$=`ghatDD[i][j]`:

```python
rfm.ref_metric__hatted_quantities()

sp.pretty_print(sp.Matrix(rfm.ghatDD))
```

    ⎡1           0                         0              ⎤
    ⎢                                                     ⎥
    ⎢                     2                               ⎥
    ⎢      ⎛ xx₀    -xx₀ ⎞                                ⎥
    ⎢      ⎜ ───    ─────⎟                                ⎥
    ⎢    2 ⎜  s       s  ⎟                                ⎥
    ⎢   a ⋅⎝ℯ    - ℯ     ⎠                                ⎥
    ⎢0  ───────────────────                0              ⎥
    ⎢            4                                        ⎥
    ⎢                                                     ⎥
    ⎢                                          2          ⎥
    ⎢                           ⎛ xx₀    -xx₀ ⎞           ⎥
    ⎢                           ⎜ ───    ─────⎟           ⎥
    ⎢                         2 ⎜  s       s  ⎟     2     ⎥
    ⎢                        a ⋅⎝ℯ    - ℯ     ⎠ ⋅sin (xx₁)⎥
    ⎢0           0           ─────────────────────────────⎥
    ⎣                                      4              ⎦

In addition to $\hat{g}_{ij}$, **`ref_metric__hatted_quantities()`** also provides:
* The rescaling "matrix" `ReDD[i][j]`, used for separating singular (due to chosen coordinate system) pieces of smooth rank-2 tensor components from the smooth parts, so that the smooth parts can be used within temporal and spatial differential operators.
* Inverse reference metric: $\hat{g}^{ij}$=`ghatUU[i][j]`.
* Reference metric determinant: $\det\left(\hat{g}_{ij}\right)$=`detgammahat`.
* First and second derivatives of the reference metric: $\hat{g}_{ij,k}$=`ghatDD_dD[i][j][k]`; $\hat{g}_{ij,kl}$=`ghatDD_dDD[i][j][k][l]`
* Christoffel symbols associated with the reference metric, $\hat{\Gamma}^i_{jk}$ = `GammahatUDD[i][j][k]` and their first derivatives $\hat{\Gamma}^i_{jk,l}$ = `GammahatUDD_dD[i][j][k][l]`

For example, the Christoffel symbol $\hat{\Gamma}^{xx_1}_{xx_2 xx_2}=\hat{\Gamma}^1_{22}$ is given by `GammahatUDD[1][2][2]`:

```python
sp.pretty_print(sp.simplify(rfm.GammahatUDD[1][2][2]))
```

    -sin(2⋅xx₁)
    ────────────
         2

Given the trigonometric identity $2\sin(x)\cos(x) = \sin(2x)$, notice that the above expression is equivalent to Eq. 18 of [Baumgarte, Montero, Cordero-Carrión, and Müller, PRD 87, 044026 (2012)](https://arxiv.org/abs/1211.6632). This is expected since the sinh-radial spherical coordinate system is equivalent to ordinary spherical coordinates in the angular components.

# Step 3: Prescribed reference metrics in [`reference_metric.py`](../edit/reference_metric.py) \[Back to [top](#toc)\]
$$\label{prescribed_ref_metric}$$

One need not manually define scale factors or other quantities for reference metrics, as a number of prescribed reference metrics are already defined in [reference_metric.py](../edit/reference_metric.py). These can be accessed by first setting the parameter **reference_metric::CoordSystem** to one of the following, and then calling the function **`rfm.reference_metric()`**.

```python
import indexedexp as ixp    # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support
import grid as gri          # NRPy+: Functions having to do with numerical grids

# Step 0a: Initialize parameters
thismodule = __name__
par.initialize_param(par.glb_param("char", thismodule, "CoordSystem", "Spherical"))

# Step 0b: Declare global variables
xx = gri.xx
xx_to_Cart = ixp.zerorank1(DIM=4) # Must be set in terms of xx[]s
Cart_to_xx = ixp.zerorank1(DIM=4) # Must be set in terms of xx[]s
Cartx,Carty,Cartz = sp.symbols("Cartx Carty Cartz", real=True)
Cart = [Cartx,Carty,Cartz]
xxSph  = ixp.zerorank1(DIM=4) # Must be set in terms of xx[]s
scalefactor_orthog = ixp.zerorank1(DIM=4) # Must be set in terms of xx[]s
have_already_called_reference_metric_function = False

CoordSystem = par.parval_from_str("reference_metric::CoordSystem")
M_PI,M_SQRT1_2 = par.Cparameters("#define",thismodule,["M_PI","M_SQRT1_2"],"")

global xxmin
global xxmax

global UnitVectors
UnitVectors = ixp.zerorank2(DIM=3)
```

We will find the following plotting function useful for analyzing coordinate systems in which the radial coordinate is rescaled.

```python
def create_r_of_xx0_plots(CoordSystem, r_of_xx0,rprime_of_xx0):
    import matplotlib.pyplot as plt  # matplotlib: Python module specializing in plotting capabilities
    plt.clf()
    Nr = 20
    dxx0 = 1.0 / float(Nr)
    xx0s    = []
    rs      = []
    deltars = []
    rprimes = []
    for i in range(Nr):
        xx0 = (float(i) + 0.5)*dxx0
        xx0s.append(xx0)
        rs.append(     sp.sympify(str(r_of_xx0     ).replace("xx0",str(xx0))))
        rprimes.append(sp.sympify(str(rprime_of_xx0).replace("xx0",str(xx0))))
        if i>0:
            deltars.append(sp.log(rs[i]-rs[i-1],10))
        else:
            deltars.append(sp.log(2*rs[0],10))

    # fig, ax = plt.subplots()
    fig = plt.figure(figsize=(12,12)) # 8 in x 8 in

    ax = fig.add_subplot(221)
    ax.set_title('$r(xx_0)$ for '+CoordSystem,fontsize='x-large')
    ax.set_xlabel('$xx_0$',fontsize='x-large')
    ax.set_ylabel('$r(xx_0)$',fontsize='x-large')
    ax.plot(xx0s, rs, 'k.', label='Spacing between\nadjacent gridpoints')
    # legend = ax.legend(loc='lower right', shadow=True, fontsize='x-large')
    # legend.get_frame().set_facecolor('C1')

    ax = fig.add_subplot(222)
    ax.set_title('Grid spacing for '+CoordSystem,fontsize='x-large')
    ax.set_xlabel('$xx_0$',fontsize='x-large')
    ax.set_ylabel('$\log_{10}(\Delta r)$',fontsize='x-large')
    ax.plot(xx0s, deltars, 'k.', label='Spacing between\nadjacent gridpoints\nin $r(xx_0)$ plot')
    legend = ax.legend(loc='lower right', shadow=True, fontsize='x-large')
    legend.get_frame().set_facecolor('C1')

    ax = fig.add_subplot(223)
    ax.set_title('$r\'(xx_0)$ for '+CoordSystem,fontsize='x-large')
    ax.set_xlabel('$xx_0$',fontsize='x-large')
    ax.set_ylabel('$r\'(xx_0)$',fontsize='x-large')
    ax.plot(xx0s, rprimes, 'k.', label='Nr=96')
    # legend = ax.legend(loc='upper left', shadow=True, fontsize='x-large')
    # legend.get_frame().set_facecolor('C1')

    plt.tight_layout(pad=2)
    plt.show()
```

## Step 3.a: Spherical-like coordinate systems \[Back to [top](#toc)\]
$$\label{sphericallike}$$

### Step 3.a.i: **`reference_metric::CoordSystem = "Spherical"`** \[Back to [top](#toc)\]
$$\label{spherical}$$

Standard spherical coordinates, with $(r,\theta,\phi)=(xx_0,xx_1,xx_2)$

```python
if CoordSystem == "Spherical":
    # Adding assumption real=True can help simplify expressions involving xx[0] & xx[1] below.
    xx[0] = sp.symbols("xx0", real=True)
    xx[1] = sp.symbols("xx1", real=True)

    RMAX = par.Cparameters("REAL", thismodule, ["RMAX"],10.0)
    xxmin = [sp.sympify(0), sp.sympify(0), -M_PI]
    xxmax = [         RMAX,          M_PI,  M_PI]

    r  = xx[0]
    th = xx[1]
    ph = xx[2]

    Cart_to_xx[0] = sp.sqrt(Cartx ** 2 + Carty ** 2 + Cartz ** 2)
    Cart_to_xx[1] = sp.acos(Cartz / Cart_to_xx[0])
    Cart_to_xx[2] = sp.atan2(Carty, Cartx)

    xxSph[0] = r
    xxSph[1] = th
    xxSph[2] = ph

    # Now define xCart, yCart, and zCart in terms of x0,xx[1],xx[2].
    # Note that the relation between r and x0 is not necessarily trivial in SinhSpherical coordinates. See above.
    xx_to_Cart[0] = xxSph[0]*sp.sin(xxSph[1])*sp.cos(xxSph[2])
    xx_to_Cart[1] = xxSph[0]*sp.sin(xxSph[1])*sp.sin(xxSph[2])
    xx_to_Cart[2] = xxSph[0]*sp.cos(xxSph[1])

    scalefactor_orthog[0] = sp.diff(xxSph[0],xx[0])
    scalefactor_orthog[1] = xxSph[0]
    scalefactor_orthog[2] = xxSph[0]*sp.sin(xxSph[1])

    # Set the unit vectors
    UnitVectors = [[ sp.sin(xxSph[1])*sp.cos(xxSph[2]), sp.sin(xxSph[1])*sp.sin(xxSph[2]),  sp.cos(xxSph[1])],
                   [ sp.cos(xxSph[1])*sp.cos(xxSph[2]), sp.cos(xxSph[1])*sp.sin(xxSph[2]), -sp.sin(xxSph[1])],
                   [                 -sp.sin(xxSph[2]),                  sp.cos(xxSph[2]),  sp.sympify(0)   ]]
```

Now let's analyze $r(xx_0)$ for **"Spherical"** coordinates.

```python
%matplotlib inline

CoordSystem = "Spherical"
par.set_parval_from_str("reference_metric::CoordSystem",CoordSystem)
rfm.reference_metric()

RMAX     = 10.0
r_of_xx0      = sp.sympify(str(rfm.xxSph[0]                   ).replace("RMAX",str(RMAX)))
rprime_of_xx0 = sp.sympify(str(sp.diff(rfm.xxSph[0],rfm.xx[0])).replace("RMAX",str(RMAX)))

create_r_of_xx0_plots(CoordSystem, r_of_xx0,rprime_of_xx0)
```

### Step 3.a.ii: **`reference_metric::CoordSystem = "SinhSpherical"`** \[Back to [top](#toc)\]
$$\label{sinhspherical}$$

Spherical coordinates, but with $$r(xx_0) = \text{AMPL} \frac{\sinh\left(\frac{xx_0}{\text{SINHW}}\right)}{\sinh\left(\frac{1}{\text{SINHW}}\right)}.$$

SinhSpherical uses two parameters: `AMPL` and `SINHW`. `AMPL` sets the outer boundary distance; and `SINHW` sets the focusing of the coordinate points near $r=0$, where a small `SINHW` ($\sim 0.125$) will greatly focus the points near $r=0$ and a large `SINHW` will look more like an ordinary spherical polar coordinate system.

```python
if CoordSystem == "SinhSpherical":
    xxmin = [sp.sympify(0), sp.sympify(0), -M_PI]
    xxmax = [sp.sympify(1),          M_PI,  M_PI]

    AMPL, SINHW = par.Cparameters("REAL",thismodule,["AMPL","SINHW"],[10.0,0.2])
    # Set SinhSpherical radial coordinate by default; overwrite later if CoordSystem == "SinhSphericalv2".
    r = AMPL * (sp.exp(xx[0] / SINHW) - sp.exp(-xx[0] / SINHW)) / \
               (sp.exp(1 / SINHW) - sp.exp(-1 / SINHW))
    th = xx[1]
    ph = xx[2]

    Cart_to_xx[0] = SINHW*sp.asinh(sp.sqrt(Cartx ** 2 + Carty ** 2 + Cartz ** 2)*sp.sinh(1/SINHW)/AMPL)
    Cart_to_xx[1] = sp.acos(Cartz / sp.sqrt(Cartx ** 2 + Carty ** 2 + Cartz ** 2))
    Cart_to_xx[2] = sp.atan2(Carty, Cartx)

    xxSph[0] = r
    xxSph[1] = th
    xxSph[2] = ph

    # Now define xCart, yCart, and zCart in terms of x0,xx[1],xx[2].
    # Note that the relation between r and x0 is not necessarily trivial in SinhSpherical coordinates. See above.
    xx_to_Cart[0] = xxSph[0]*sp.sin(xxSph[1])*sp.cos(xxSph[2])
    xx_to_Cart[1] = xxSph[0]*sp.sin(xxSph[1])*sp.sin(xxSph[2])
    xx_to_Cart[2] = xxSph[0]*sp.cos(xxSph[1])

    scalefactor_orthog[0] = sp.diff(xxSph[0],xx[0])
    scalefactor_orthog[1] = xxSph[0]
    scalefactor_orthog[2] = xxSph[0]*sp.sin(xxSph[1])

    # Set the unit vectors
    UnitVectors = [[ sp.sin(xxSph[1])*sp.cos(xxSph[2]), sp.sin(xxSph[1])*sp.sin(xxSph[2]),  sp.cos(xxSph[1])],
                   [ sp.cos(xxSph[1])*sp.cos(xxSph[2]), sp.cos(xxSph[1])*sp.sin(xxSph[2]), -sp.sin(xxSph[1])],
                   [                 -sp.sin(xxSph[2]),                  sp.cos(xxSph[2]),  sp.sympify(0)   ]]
```

Now we explore $r(xx_0)$ for `SinhSpherical` assuming `AMPL=10.0` and `SINHW=0.2`:

```python
%matplotlib inline

CoordSystem = "SinhSpherical"
par.set_parval_from_str("reference_metric::CoordSystem",CoordSystem)
rfm.reference_metric()

AMPL     = 10.0
SINHW    = 0.2
r_of_xx0      = sp.sympify(str(rfm.xxSph[0]                   ).replace("AMPL",str(AMPL)).replace("SINHW",str(SINHW)))
rprime_of_xx0 = sp.sympify(str(sp.diff(rfm.xxSph[0],rfm.xx[0])).replace("AMPL",str(AMPL)).replace("SINHW",str(SINHW)))

create_r_of_xx0_plots(CoordSystem, r_of_xx0,rprime_of_xx0)
```

### Step 3.a.iii: **`reference_metric::CoordSystem = "SinhSphericalv2"`** \[Back to [top](#toc)\]
$$\label{sinhsphericalv2}$$

The same as SinhSpherical coordinates, but with an additional `AMPL*const_dr*xx_0` term:
$$r(xx_0) = \text{AMPL} \left[\text{const_dr}\ xx_0 + \frac{\sinh\left(\frac{xx_0}{\text{SINHW}}\right)}{\sinh\left(\frac{1}{\text{SINHW}}\right)}\right].$$

```python
if CoordSystem == "SinhSphericalv2":
    # SinhSphericalv2 adds the parameter "const_dr", which allows for a region near xx[0]=0 to have
    # constant radial resolution of const_dr, provided the sinh() term does not dominate near xx[0]=0.
    xxmin = [sp.sympify(0), sp.sympify(0), -M_PI]
    xxmax = [sp.sympify(1),          M_PI,  M_PI]

    AMPL, SINHW = par.Cparameters("REAL",thismodule,["AMPL","SINHW"],[10.0,0.2])
    const_dr = par.Cparameters("REAL",thismodule,["const_dr"],0.0625)

    r = AMPL*( const_dr*xx[0] + (sp.exp(xx[0] / SINHW) - sp.exp(-xx[0] / SINHW)) /
               (sp.exp(1 / SINHW) - sp.exp(-1 / SINHW)) )
    th = xx[1]
    ph = xx[2]

    # NO CLOSED-FORM EXPRESSION FOR RADIAL INVERSION.
    # Cart_to_xx[0] = "NewtonRaphson"
    # Cart_to_xx[1] = sp.acos(Cartz / sp.sqrt(Cartx ** 2 + Carty ** 2 + Cartz ** 2))
    # Cart_to_xx[2] = sp.atan2(Carty, Cartx)

    xxSph[0] = r
    xxSph[1] = th
    xxSph[2] = ph

    # Now define xCart, yCart, and zCart in terms of x0,xx[1],xx[2].
    # Note that the relation between r and x0 is not necessarily trivial in SinhSpherical coordinates. See above.
    xx_to_Cart[0] = xxSph[0]*sp.sin(xxSph[1])*sp.cos(xxSph[2])
    xx_to_Cart[1] = xxSph[0]*sp.sin(xxSph[1])*sp.sin(xxSph[2])
    xx_to_Cart[2] = xxSph[0]*sp.cos(xxSph[1])

    scalefactor_orthog[0] = sp.diff(xxSph[0],xx[0])
    scalefactor_orthog[1] = xxSph[0]
    scalefactor_orthog[2] = xxSph[0]*sp.sin(xxSph[1])

    # Set the unit vectors
    UnitVectors = [[ sp.sin(xxSph[1])*sp.cos(xxSph[2]), sp.sin(xxSph[1])*sp.sin(xxSph[2]),  sp.cos(xxSph[1])],
                   [ sp.cos(xxSph[1])*sp.cos(xxSph[2]), sp.cos(xxSph[1])*sp.sin(xxSph[2]), -sp.sin(xxSph[1])],
                   [                 -sp.sin(xxSph[2]),                  sp.cos(xxSph[2]),  sp.sympify(0)   ]]
```

Now we explore $r(xx_0)$ for `SinhSphericalv2` assuming `AMPL=10.0`, `SINHW=0.2`, and `const_dr=0.05`. Notice that the `const_dr` term significantly increases the grid spacing near $xx_0=0$ relative to `SinhSpherical` coordinates.

```python
%matplotlib inline

CoordSystem = "SinhSphericalv2"
par.set_parval_from_str("reference_metric::CoordSystem",CoordSystem)
rfm.reference_metric()

AMPL     = 10.0
SINHW    = 0.2
const_dr = 0.05
r_of_xx0      = sp.sympify(str(rfm.xxSph[0]                   ).replace("AMPL",str(AMPL)).replace("SINHW",str(SINHW)).replace("const_dr",str(const_dr)))
rprime_of_xx0 = sp.sympify(str(sp.diff(rfm.xxSph[0],rfm.xx[0])).replace("AMPL",str(AMPL)).replace("SINHW",str(SINHW)).replace("const_dr",str(const_dr)))

create_r_of_xx0_plots(CoordSystem, r_of_xx0,rprime_of_xx0)
```

## Step 3.b: Cylindrical-like coordinate systems \[Back to [top](#toc)\]
$$\label{cylindricallike}$$

### Step 3.b.i: **`reference_metric::CoordSystem = "Cylindrical"`** \[Back to [top](#toc)\]
$$\label{cylindrical}$$

Standard cylindrical coordinates, with $(\rho,\phi,z)=(xx_0,xx_1,xx_2)$

```python
if CoordSystem == "Cylindrical":
    # Assuming the cylindrical radial coordinate
    #   is positive makes nice simplifications of
    #   unit vectors possible.
    xx[0] = sp.symbols("xx0", real=True)

    RHOMAX,ZMIN,ZMAX = par.Cparameters("REAL",thismodule,["RHOMAX","ZMIN","ZMAX"],[10.0,-10.0,10.0])
    xxmin = [sp.sympify(0), -M_PI, ZMIN]
    xxmax = [       RHOMAX,  M_PI, ZMAX]

    RHOCYL = xx[0]
    PHICYL = xx[1]
    ZCYL   = xx[2]

    Cart_to_xx[0] = sp.sqrt(Cartx ** 2 + Carty ** 2)
    Cart_to_xx[1] = sp.atan2(Carty, Cartx)
    Cart_to_xx[2] = Cartz

    xx_to_Cart[0] = RHOCYL*sp.cos(PHICYL)
    xx_to_Cart[1] = RHOCYL*sp.sin(PHICYL)
    xx_to_Cart[2] = ZCYL

    xxSph[0] = sp.sqrt(RHOCYL**2 + ZCYL**2)
    xxSph[1] = sp.acos(ZCYL / xxSph[0])
    xxSph[2] = PHICYL

    scalefactor_orthog[0] = sp.diff(RHOCYL,xx[0])
    scalefactor_orthog[1] = RHOCYL
    scalefactor_orthog[2] = sp.diff(ZCYL,xx[2])

    # Set the unit vectors
    UnitVectors = [[ sp.cos(PHICYL), sp.sin(PHICYL), sp.sympify(0)],
                   [-sp.sin(PHICYL), sp.cos(PHICYL), sp.sympify(0)],
                   [ sp.sympify(0),  sp.sympify(0),  sp.sympify(1)]]
```

Next let's plot **"Cylindrical"** coordinates.

```python
%matplotlib inline

import numpy as np                  # NumPy: A numerical methods module for Python
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

R = np.linspace(0, 2, 24)
h = 2
u = np.linspace(0,  2*np.pi, 24)

x = np.outer(R, np.cos(u))
y = np.outer(R, np.sin(u))
z = h * np.outer(np.ones(np.size(u)), np.ones(np.size(u)))

r = np.arange(0,2,0.25)
theta = 2*np.pi*r*0

fig = plt.figure(figsize=(12,12)) # 8 in x 8 in
fig, (ax1, ax2) = plt.subplots(1, 2)

ax1 = plt.axes(projection='polar')

ax1.set_rmax(2)

ax1.set_rgrids(r,labels=[])

thetas = np.linspace(0,360,24, endpoint=True)
ax1.set_thetagrids(thetas,labels=[])

# ax.grid(True)
ax1.grid(True,linewidth='1.0')
ax1.set_title("Top Down View")
plt.show()

ax2 = plt.axes(projection='3d', xticklabels=[], yticklabels=[], zticklabels=[])
#ax2.plot_surface(x,y,z, alpha=.75, cmap = 'viridis') # z in case of disk which is parallel to XY plane is constant and you can directly use h

x=np.linspace(-2, 2, 100)
z=np.linspace(-2, 2, 100)
Xc, Zc=np.meshgrid(x, z)
Yc = np.sqrt(4-Xc**2)

rstride = 10
cstride = 10
ax2.plot_surface(Xc, Yc, Zc, alpha=1.0, rstride=rstride, cstride=cstride, cmap = 'viridis')
ax2.plot_surface(Xc, -Yc, Zc, alpha=1.0, rstride=rstride, cstride=cstride, cmap = 'viridis')
ax2.set_title("Standard Cylindrical Grid in 3D")
ax2.grid(False)
plt.axis('off')

plt.show()
```

### Step 3.b.ii" **`reference_metric::CoordSystem = "SinhCylindrical"`** \[Back to [top](#toc)\]
$$\label{sinhcylindrical}$$

Cylindrical coordinates, but with
$$\rho(xx_0) = \text{AMPLRHO} \frac{\sinh\left(\frac{xx_0}{\text{SINHWRHO}}\right)}{\sinh\left(\frac{1}{\text{SINHWRHO}}\right)}$$
and
$$z(xx_2) = \text{AMPLZ} \frac{\sinh\left(\frac{xx_2}{\text{SINHWZ}}\right)}{\sinh\left(\frac{1}{\text{SINHWZ}}\right)}$$

```python
if CoordSystem == "SinhCylindrical":
    # Assuming the cylindrical radial coordinate
    #   is positive makes nice simplifications of
    #   unit vectors possible.
    xx[0] = sp.symbols("xx0", real=True)

    xxmin = [sp.sympify(0), -M_PI, sp.sympify(-1)]
    xxmax = [sp.sympify(1),  M_PI, sp.sympify(+1)]

    AMPLRHO, SINHWRHO, AMPLZ, SINHWZ = par.Cparameters("REAL",thismodule,
                                                       ["AMPLRHO","SINHWRHO","AMPLZ","SINHWZ"],
                                                       [     10.0,       0.2,   10.0,    0.2])

    # Set SinhCylindrical radial & z coordinates by default; overwrite later if CoordSystem == "SinhCylindricalv2".
    RHOCYL = AMPLRHO * (sp.exp(xx[0] / SINHWRHO) - sp.exp(-xx[0] / SINHWRHO)) / (sp.exp(1 / SINHWRHO) - sp.exp(-1 / SINHWRHO))
    # phi coordinate remains unchanged.
    PHICYL = xx[1]
    ZCYL   = AMPLZ   * (sp.exp(xx[2] / SINHWZ)   - sp.exp(-xx[2] / SINHWZ))   / (sp.exp(1 / SINHWZ)   - sp.exp(-1 / SINHWZ))
    Cart_to_xx[0] = SINHWRHO*sp.asinh(sp.sqrt(Cartx ** 2 + Carty ** 2)*sp.sinh(1/SINHWRHO)/AMPLRHO)
    Cart_to_xx[1] = sp.atan2(Carty, Cartx)
    Cart_to_xx[2] = SINHWZ*sp.asinh(Cartz*sp.sinh(1/SINHWZ)/AMPLZ)

    xx_to_Cart[0] = RHOCYL*sp.cos(PHICYL)
    xx_to_Cart[1] = RHOCYL*sp.sin(PHICYL)
    xx_to_Cart[2] = ZCYL

    xxSph[0] = sp.sqrt(RHOCYL**2 + ZCYL**2)
    xxSph[1] = sp.acos(ZCYL / xxSph[0])
    xxSph[2] = PHICYL

    scalefactor_orthog[0] = sp.diff(RHOCYL,xx[0])
    scalefactor_orthog[1] = RHOCYL
    scalefactor_orthog[2] = sp.diff(ZCYL,xx[2])

    # Set the unit vectors
    UnitVectors = [[ sp.cos(PHICYL), sp.sin(PHICYL), sp.sympify(0)],
                   [-sp.sin(PHICYL), sp.cos(PHICYL), sp.sympify(0)],
                   [ sp.sympify(0),  sp.sympify(0),  sp.sympify(1)]]
```

Next let's plot **"SinhCylindrical"** coordinates.

```python
fig=plt.figure()

plt.clf()

fig = plt.figure()

ax = plt.subplot(1,1,1, projection='polar')

ax.set_rmax(2)

Nr = 20
xx0s = np.linspace(0,2,Nr, endpoint=True) + 1.0/(2.0*Nr)

rs = []
AMPLRHO = 1.0
SINHW = 0.4
for i in range(Nr):
    rs.append(AMPLRHO * (np.exp(xx0s[i] / SINHW) - np.exp(-xx0s[i] / SINHW)) / \
                        (np.exp(1.0 / SINHW) - np.exp(-1.0 / SINHW)))

ax.set_rgrids(rs,labels=[])

thetas = np.linspace(0,360,25, endpoint=True)
ax.set_thetagrids(thetas,labels=[])

# ax.grid(True)
ax.grid(True,linewidth='1.0')

plt.show()

```

### Step 3.b.iii: **`reference_metric::CoordSystem = "SinhCylindricalv2"`** \[Back to [top](#toc)\]
$$\label{sinhcylindricalv2}$$

Cylindrical coordinates, but with
$$\rho(xx_0) = \text{AMPLRHO} \left[\text{const_drho}\ xx_0 + \frac{\sinh\left(\frac{xx_0}{\text{SINHWRHO}}\right)}{\sinh\left(\frac{1}{\text{SINHWRHO}}\right)}\right]$$
and
$$z(xx_2) = \text{AMPLZ} \left[\text{const_dz}\ xx_2 + \frac{\sinh\left(\frac{xx_2}{\text{SINHWZ}}\right)}{\sinh\left(\frac{1}{\text{SINHWZ}}\right)}\right]$$

```python
if CoordSystem == "SinhCylindricalv2":
    # Assuming the cylindrical radial coordinate
    #   is positive makes nice simplifications of
    #   unit vectors possible.
    xx[0] = sp.symbols("xx0", real=True)

    # SinhCylindricalv2 adds the parameters "const_drho", "const_dz", which allows for regions near xx[0]=0
    # and xx[2]=0 to have constant rho and z resolution of const_drho and const_dz, provided the sinh() terms
    # do not dominate near xx[0]=0 and xx[2]=0.
    xxmin = [sp.sympify(0), -M_PI, sp.sympify(-1)]
    xxmax = [sp.sympify(1),  M_PI, sp.sympify(+1)]
    AMPLRHO, SINHWRHO, AMPLZ, SINHWZ = par.Cparameters("REAL",thismodule,
                                                       ["AMPLRHO","SINHWRHO","AMPLZ","SINHWZ"],
                                                       [     10.0,       0.2,   10.0,    0.2])
    const_drho, const_dz = par.Cparameters("REAL",thismodule,["const_drho","const_dz"],[0.0625,0.0625])

    RHOCYL = AMPLRHO * ( const_drho*xx[0] + (sp.exp(xx[0] / SINHWRHO) - sp.exp(-xx[0] / SINHWRHO)) / (sp.exp(1 / SINHWRHO) - sp.exp(-1 / SINHWRHO)) )
    PHICYL = xx[1]
    ZCYL   = AMPLZ   * ( const_dz  *xx[2] + (sp.exp(xx[2] / SINHWZ  ) - sp.exp(-xx[2] / SINHWZ  )) / (sp.exp(1 / SINHWZ  ) - sp.exp(-1 / SINHWZ  )) )

    # NO CLOSED-FORM EXPRESSION FOR RADIAL OR Z INVERSION.
    # Cart_to_xx[0] = "NewtonRaphson"
    # Cart_to_xx[1] = sp.atan2(Carty, Cartx)
    # Cart_to_xx[2] = "NewtonRaphson"

    xx_to_Cart[0] = RHOCYL*sp.cos(PHICYL)
    xx_to_Cart[1] = RHOCYL*sp.sin(PHICYL)
    xx_to_Cart[2] = ZCYL

    xxSph[0] = sp.sqrt(RHOCYL**2 + ZCYL**2)
    xxSph[1] = sp.acos(ZCYL / xxSph[0])
    xxSph[2] = PHICYL

    scalefactor_orthog[0] = sp.diff(RHOCYL,xx[0])
    scalefactor_orthog[1] = RHOCYL
    scalefactor_orthog[2] = sp.diff(ZCYL,xx[2])

    # Set the unit vectors
    UnitVectors = [[ sp.cos(PHICYL), sp.sin(PHICYL), sp.sympify(0)],
                   [-sp.sin(PHICYL), sp.cos(PHICYL), sp.sympify(0)],
                   [ sp.sympify(0),  sp.sympify(0),  sp.sympify(1)]]
```

For example, let's set up **`SinhCylindricalv2`** coordinates and output the Christoffel symbol $\hat{\Gamma}^{xx_2}_{xx_2 xx_2}$, or more simply $\hat{\Gamma}^2_{22}$:

```python
par.set_parval_from_str("reference_metric::CoordSystem","SinhCylindricalv2")

rfm.reference_metric()

sp.pretty_print(sp.simplify(rfm.GammahatUDD[2][2][2]))
```

                             ⎛ 2⋅xx₂     ⎞    1
                             ⎜ ──────    ⎟  ──────
                             ⎜ SINHWZ    ⎟  SINHWZ
                            -⎝ℯ       - 1⎠⋅ℯ
    ────────────────────────────────────────────────────────────────────────
           ⎛                  ⎛   2       ⎞   xx₂     ⎛ 2⋅xx₂     ⎞    1   ⎞
           ⎜                  ⎜ ──────    ⎟  ──────   ⎜ ──────    ⎟  ──────⎟
           ⎜                  ⎜ SINHWZ    ⎟  SINHWZ   ⎜ SINHWZ    ⎟  SINHWZ⎟
    SINHWZ⋅⎝- SINHWZ⋅const_dz⋅⎝ℯ       - 1⎠⋅ℯ       - ⎝ℯ       + 1⎠⋅ℯ      ⎠

As we will soon see, defining these "hatted" quantities will be quite useful when expressing hyperbolic ([wave-equation](https://en.wikipedia.org/wiki/Wave_equation)-like) PDEs in non-Cartesian coordinate systems.

## Step 3.c: Cartesian-like coordinate systems \[Back to [top](#toc)\]
$$\label{cartesianlike}$$

### Step 3.c.i: **`reference_metric::CoordSystem = "Cartesian"`** \[Back to [top](#toc)\]
$$\label{cartesian}$$

Standard Cartesian coordinates, with $(x,y,z)=$ `(xx0,xx1,xx2)`

```python
if CoordSystem == "Cartesian":
    xmin, xmax, ymin, ymax, zmin, zmax = par.Cparameters("REAL",thismodule,
                                                         ["xmin","xmax","ymin","ymax","zmin","zmax"],
                                                         [ -10.0,  10.0, -10.0,  10.0, -10.0,  10.0])
    xxmin = ["xmin", "ymin", "zmin"]
    xxmax = ["xmax", "ymax", "zmax"]

    xx_to_Cart[0] = xx[0]
    xx_to_Cart[1] = xx[1]
    xx_to_Cart[2] = xx[2]

    xxSph[0] = sp.sqrt(xx[0] ** 2 + xx[1] ** 2 + xx[2] ** 2)
    xxSph[1] = sp.acos(xx[2] / xxSph[0])
    xxSph[2] = sp.atan2(xx[1], xx[0])

    Cart_to_xx[0] = Cartx
    Cart_to_xx[1] = Carty
    Cart_to_xx[2] = Cartz

    scalefactor_orthog[0] = sp.sympify(1)
    scalefactor_orthog[1] = sp.sympify(1)
    scalefactor_orthog[2] = sp.sympify(1)

    # Set the transpose of the matrix of unit vectors
    UnitVectors = [[sp.sympify(1), sp.sympify(0), sp.sympify(0)],
                   [sp.sympify(0), sp.sympify(1), sp.sympify(0)],
                   [sp.sympify(0), sp.sympify(0), sp.sympify(1)]]
```

```python
%matplotlib inline

import numpy as np               # NumPy: A numerical methods module for Python
import matplotlib.pyplot as plt  # matplotlib: Python module specializing in plotting capabilities

plt.clf()

fig = plt.figure()
ax = fig.gca()
Nx = 16
ax.set_xticks(np.arange(0, 1., 1./Nx))
ax.set_yticks(np.arange(0, 1., 1./Nx))

for tick in ax.get_xticklabels():
    tick.set_rotation(60)

# plt.scatter(x, y)
ax.set_aspect('equal')

plt.grid()

# plt.savefig("Cartgrid.png",dpi=300)
plt.show()
# plt.close(fig)
```

### Step 3.c.ii: **`reference_metric::CoordSystem = "SinhCartesian"`** \[Back to [top](#toc)\]
$$\label{sinhcartesian}$$

In this coordinate system, all three coordinates behave like the $z$-coordinate in SinhCylindrical coordinates, i.e.

$$
\begin{align}
x(xx_0) &= \text{AMPLXYZ} \left[\frac{\sinh\left(\frac{xx_0}{\text{SINHWXYZ}}\right)}{\sinh\left(\frac{1}{\text{SINHWXYZ}}\right)}\right]\ ,\\
y(xx_1) &= \text{AMPLXYZ} \left[\frac{\sinh\left(\frac{xx_1}{\text{SINHWXYZ}}\right)}{\sinh\left(\frac{1}{\text{SINHWXYZ}}\right)}\right]\ ,\\
z(xx_2) &= \text{AMPLXYZ} \left[\frac{\sinh\left(\frac{xx_2}{\text{SINHWXYZ}}\right)}{\sinh\left(\frac{1}{\text{SINHWXYZ}}\right)}\right]
\end{align}
$$

```python
if CoordSystem == "SinhCartesian":
    # SinhCartesian coordinates allows us to push the outer boundary of the
    # computational domain a lot further away, while keeping reasonably high
    # resolution towards the center of the computational grid.

    # Set default values for min and max (x,y,z)
    xxmin = [sp.sympify(-1), sp.sympify(-1), sp.sympify(-1)]
    xxmax = [sp.sympify(+1), sp.sympify(+1), sp.sympify(+1)]

    # Declare basic parameters of the coordinate system and their default values
    AMPLXYZ, SINHWXYZ = par.Cparameters("REAL", thismodule,
                                        ["AMPLXYZ", "SINHWXYZ"],
                                        [     10.0,        0.2])

    # Compute (xx_to_Cart0,xx_to_Cart1,xx_to_Cart2) from (xx0,xx1,xx2)
    for ii in [0, 1, 2]:
        xx_to_Cart[ii] = AMPLXYZ*(sp.exp(xx[ii]/SINHWXYZ) - sp.exp(-xx[ii]/SINHWXYZ))/(sp.exp(1/SINHWXYZ) - sp.exp(-1/SINHWXYZ))

    # Compute (r,th,ph) from (xx_to_Cart2,xx_to_Cart1,xx_to_Cart2)
    xxSph[0] = sp.sqrt(xx_to_Cart[0] ** 2 + xx_to_Cart[1] ** 2 + xx_to_Cart[2] ** 2)
    xxSph[1] = sp.acos(xx_to_Cart[2] / xxSph[0])
    xxSph[2] = sp.atan2(xx_to_Cart[1], xx_to_Cart[0])

    # Compute (xx0,xx1,xx2) from (Cartx,Carty,Cartz)
    Cart_to_xx[0] = SINHWXYZ*sp.asinh(Cartx*sp.sinh(1/SINHWXYZ)/AMPLXYZ)
    Cart_to_xx[1] = SINHWXYZ*sp.asinh(Carty*sp.sinh(1/SINHWXYZ)/AMPLXYZ)
    Cart_to_xx[2] = SINHWXYZ*sp.asinh(Cartz*sp.sinh(1/SINHWXYZ)/AMPLXYZ)

    # Compute scale factors
    scalefactor_orthog[0] = sp.diff(xx_to_Cart[0], xx[0])
    scalefactor_orthog[1] = sp.diff(xx_to_Cart[1], xx[1])
    scalefactor_orthog[2] = sp.diff(xx_to_Cart[2], xx[2])

    # Set the transpose of the matrix of unit vectors
    UnitVectors = [[sp.sympify(1), sp.sympify(0), sp.sympify(0)],
                   [sp.sympify(0), sp.sympify(1), sp.sympify(0)],
                   [sp.sympify(0), sp.sympify(0), sp.sympify(1)]]

```

```python
%matplotlib inline

import numpy as np               # NumPy: A numerical methods module for Python
import matplotlib.pyplot as plt  # matplotlib: Python module specializing in plotting capabilities

plt.clf()

fig = plt.figure(dpi=160)
ax = fig.gca()

# Set plot title
ax.set_title(r"$z=0$ slice of the 3D grid")

# Set SINH parameters. Here we assume:
#
# AMPLX  = AMPLY  = SINHA
# SINHWX = SINHWY = SINHW
SINHA = 10.0
SINHW = 0.45

# Set number of points. We assume the same point
# distribution along the (x,y)-directions
Nxxs = 24
xxis = np.linspace(-1,1,Nxxs, endpoint=True)

# Compute axis ticks by evaluating x and y using SinhCartesian coordinates
axis_ticks = []
for i in range(Nxxs):
    axis_ticks.append(SINHA * (np.exp(xxis[i] / SINHW) - np.exp(-xxis[i] / SINHW)) / \
                        (np.exp(1.0 / SINHW) - np.exp(-1.0 / SINHW)))

# Set the axis ticks
ax.set_xticks(axis_ticks)
ax.set_yticks(axis_ticks)

# Set x and y labels. Initialize array with empty strings
labelsx = ["" for i in range(Nxxs)]
labelsy = ["" for i in range(Nxxs)]

# Set x_min and x_max tick label
labelsx[0] = r"-AMPLX"
labelsx[-1] = r"AMPLX"

# Set y_min and y_max tick label
labelsy[0] = r"-AMPLY"
labelsy[-1] = r"AMPLY"

# Set tick labels
ax.set_xticklabels(labelsx)
ax.set_yticklabels(labelsy)

# Rotate x labels by 60 degrees
for tick in ax.get_xticklabels():
    tick.set_rotation(60)

# Draw the x=0 and y=0 ticklabel
ax.text(0,-11,"0",ha="center",va="center")
ax.text(-11,0,"0",ha="center",va="center")

# plt.scatter(x, y)
ax.set_aspect('equal')

plt.grid(color='black',linewidth=0.3)

plt.show()
# plt.savefig("Cartgrid.png",dpi=400)
# plt.close(fig)
```

## Step 3.d: [Prolate spheroidal](https://en.wikipedia.org/wiki/Prolate_spheroidal_coordinates)-like coordinate systems \[Back to [top](#toc)\]
$$\label{prolatespheroidal}$$

### Step 3.d.i: **`reference_metric::CoordSystem = "SymTP"`** \[Back to [top](#toc)\]
$$\label{symtp}$$

The Symmetric TwoPuncture (SymTP) coordinate system is obtained by slightly modifying [prolate spheroidal coordinates](https://en.wikipedia.org/wiki/Prolate_spheroidal_coordinates) (PSC). Standard PSC are related to Cartesian coordinates $(x,y,z)$ via

$$
\begin{aligned}
x &= a\sinh\mu\sin\nu\cos\varphi,\\
y &= a\sinh\mu\sin\nu\sin\varphi,\\
z &= a\cosh\mu\cos\nu = \left(a^{2}\sinh^{2}\mu + a^{2}\right)^{1/2}\cos\nu,
\end{aligned}
$$

where $\mu\in[0,\infty)$, $\nu\in[0,\pi]$, and $\varphi\in[0,2\pi]$, and we have used the [identity](https://en.wikipedia.org/wiki/Hyperbolic_functions)

$$
\cosh^{2}\mu - \sinh^{2}\mu = 1.
$$

PSC have two foci located at $z=\pm a$, where the grid lines get more dense and the grid resolution increases. However, note that the parameter $a$ controls both the foci position *and* the grid scaling, which is not a particularly interesting property.

In order to remedy this, we introduce new coordinates $(xx_{0},xx_{1},xx_{2})$, such that

$$
\begin{aligned}
xx_{0} &= \frac{1}{a}\sinh\mu,\\
xx_{1} &= \nu,\\
xx_{2} &= \varphi,
\end{aligned}
$$

and change $a\to \text{bScale}$, so that we obtain

$$
\begin{aligned}
x &= xx_{0}\sin(xx_{1})\cos(xx_{2}),\\
y &= xx_{0}\sin(xx_{1})\sin(xx_{2}),\\
z &= \left(xx_{0}^{2}+\text{bScale}^{2}\right)\cos(xx_{1}),
\end{aligned}
$$

Note that, for numerical convenience, we change the range of $xx_{2}$ to $[-\pi,\pi]$. Comparing SymTP coordinates with Cylindrical coordinates, we find that $(\rho,\phi,z)=(xx_{0}\sin(xx_{1}), xx_{2}, \sqrt{xx_{0}^2 + \text{bScale}^2}\cos(xx_{1}))$.

```python
if CoordSystem == "SymTP":

    var1, var2= sp.symbols('var1 var2',real=True)
    bScale, AW, AMAX, RHOMAX, ZMIN, ZMAX = par.Cparameters("REAL",thismodule,
                                                           ["bScale","AW","AMAX","RHOMAX","ZMIN","ZMAX"],
                                                           [0.5,     0.2,   10.0,    10.0, -10.0,  10.0])

    # Assuming xx0, xx1, and bScale
    #   are positive makes nice simplifications of
    #   unit vectors possible.
    xx[0],xx[1] = sp.symbols("xx0 xx1", real=True)

    xxmin = [sp.sympify(0), sp.sympify(0),-M_PI]
    xxmax = [         AMAX,          M_PI, M_PI]

    AA = xx[0]

    var1 = sp.sqrt(AA**2 + (bScale * sp.sin(xx[1]))**2)
    var2 = sp.sqrt(AA**2 + bScale**2)

    RHOSYMTP = AA*sp.sin(xx[1])
    PHSYMTP = xx[2]
    ZSYMTP = var2*sp.cos(xx[1])

    xx_to_Cart[0] = AA  *sp.sin(xx[1])*sp.cos(xx[2])
    xx_to_Cart[1] = AA  *sp.sin(xx[1])*sp.sin(xx[2])
    xx_to_Cart[2] = ZSYMTP

    xxSph[0] = sp.sqrt(RHOSYMTP**2 + ZSYMTP**2)
    xxSph[1] = sp.acos(ZSYMTP / xxSph[0])
    xxSph[2] = PHSYMTP

    rSph  = sp.sqrt(Cartx ** 2 + Carty ** 2 + Cartz ** 2)
    thSph = sp.acos(Cartz / rSph)
    phSph = sp.atan2(Carty, Cartx)

    # Mathematica script to compute Cart_to_xx[]
    #             AA = x1;
    #             var2 = Sqrt[AA^2 + bScale^2];
    #             RHOSYMTP = AA*Sin[x2];
    #             ZSYMTP = var2*Cos[x2];
    #             Solve[{rSph == Sqrt[RHOSYMTP^2 + ZSYMTP^2],
    #                    thSph == ArcCos[ZSYMTP/Sqrt[RHOSYMTP^2 + ZSYMTP^2]],
    #                    phSph == x3},
    #                   {x1, x2, x3}]
    Cart_to_xx[0] = sp.sqrt(-bScale**2 + rSph**2 +
                            sp.sqrt(bScale**4 + 2*bScale**2*rSph**2 + rSph**4 -
                                    4*bScale**2*rSph**2*sp.cos(thSph)**2))*M_SQRT1_2 # M_SQRT1_2 = 1/sqrt(2); define this way for UnitTesting

    # The sign() function in the following expression ensures the correct root is taken.
    Cart_to_xx[1] = sp.acos(sp.sign(Cartz)*(
                              sp.sqrt(1 + rSph**2/bScale**2 -
                                      sp.sqrt(bScale**4 + 2*bScale**2*rSph**2 + rSph**4 -
                                              4*bScale**2*rSph**2*sp.cos(thSph)**2)/bScale**2)*M_SQRT1_2)) # M_SQRT1_2 = 1/sqrt(2); define this way for UnitTesting

    Cart_to_xx[2] = phSph
```

```python
%matplotlib inline

Nxxs   = 24
xx0    = np.linspace(-2,2,Nxxs,endpoint=True)
xx1    = np.linspace(0,np.pi,Nxxs,endpoint=True)
xx2    = 0
bScale = 1

def x(xx0,xx1):
    return xx0 * np.sin(xx1)
def z(xx0,xx1,bScale):
    return np.sqrt(xx0**2 + bScale**2) * np.cos(xx1)

import numpy as np               # NumPy: A numerical methods module for Python
import matplotlib.pyplot as plt  # matplotlib: Python module specializing in plotting capabilities

plt.clf()

fig = plt.figure(dpi=160)
ax = fig.gca()

# Set plot title
ax.set_title(r"""SymTP Coordinates: zx-plane ($xx_{2}$=0 and $xx_{2}=\pi$)
Blue (red) lines have constant $xx_{0}$ ($xx_{1}$)""")

ax.set_xlim(-2.5,2.5)
ax.set_ylim(-2.5,2.5)
ax.set_aspect('equal')
ax.axis('off')

for i0 in range(Nxxs):
    plt.plot(x(xx0[i0],xx1),z(xx0[i0],xx1,bScale),'b',lw=0.75)

for i1 in range(Nxxs):
    plt.plot(x(xx0,xx1[i1]),z(xx0,xx1[i1],bScale),'r',lw=0.75)

plt.show()
```

### Step 3.d.ii: **`reference_metric::CoordSystem = "SinhSymTP"`** \[Back to [top](#toc)\]
$$\label{sinhsymtp}$$

SinhSymTP coordinates are obtained from SymTP coordinates by making the substitution

$$
xx0 \to \mathcal{A}\frac{\sinh(xx_{0}/w)}{\sinh(1/w)},
$$

with the further modification that $xx_{0}\in[0,1]$. The parameter $\mathcal{A}$ controls the scale of the grid, while the parameter $w$ controls how densily sampled the region around the foci are.

```python
if CoordSystem == "SinhSymTP":

    var1, var2= sp.symbols('var1 var2',real=True)
    bScale, AW, AMAX, RHOMAX, ZMIN, ZMAX = par.Cparameters("REAL",thismodule,
                                                           ["bScale","AW","AMAX","RHOMAX","ZMIN","ZMAX"],
                                                           [0.5,     0.2,   10.0,    10.0, -10.0,  10.0])

    # Assuming xx0, xx1, and bScale
    #   are positive makes nice simplifications of
    #   unit vectors possible.
    xx[0],xx[1] = sp.symbols("xx0 xx1", real=True)

    xxmin = [sp.sympify(0), sp.sympify(0),-M_PI]
    xxmax = [sp.sympify(1),          M_PI, M_PI]

    AA = AMAX * (sp.exp(xx[0]/SINHWAA) - sp.exp(-xx[0]/SINHWAA)) / (sp.exp(1/SINHWAA) - sp.exp(-1/SINHWAA))

    var1 = sp.sqrt(AA**2 + (bScale * sp.sin(xx[1]))**2)
    var2 = sp.sqrt(AA**2 + bScale**2)

    RHOSYMTP = AA*sp.sin(xx[1])
    PHSYMTP = xx[2]
    ZSYMTP = var2*sp.cos(xx[1])

    xx_to_Cart[0] = AA*sp.sin(xx[1])*sp.cos(xx[2])
    xx_to_Cart[1] = AA*sp.sin(xx[1])*sp.sin(xx[2])
    xx_to_Cart[2] = ZSYMTP

    xxSph[0] = sp.sqrt(RHOSYMTP**2 + ZSYMTP**2)
    xxSph[1] = sp.acos(ZSYMTP / xxSph[0])
    xxSph[2] = PHSYMTP

    rSph  = sp.sqrt(Cartx ** 2 + Carty ** 2 + Cartz ** 2)
    thSph = sp.acos(Cartz / rSph)
    phSph = sp.atan2(Carty, Cartx)

    # Mathematica script to compute Cart_to_xx[]
    #             AA = x1;
    #             var2 = Sqrt[AA^2 + bScale^2];
    #             RHOSYMTP = AA*Sin[x2];
    #             ZSYMTP = var2*Cos[x2];
    #             Solve[{rSph == Sqrt[RHOSYMTP^2 + ZSYMTP^2],
    #                    thSph == ArcCos[ZSYMTP/Sqrt[RHOSYMTP^2 + ZSYMTP^2]],
    #                    phSph == x3},
    #                   {x1, x2, x3}]
    Cart_to_xx[0] = sp.sqrt(-bScale**2 + rSph**2 +
                            sp.sqrt(bScale**4 + 2*bScale**2*rSph**2 + rSph**4 -
                                    4*bScale**2*rSph**2*sp.cos(thSph)**2))*M_SQRT1_2 # M_SQRT1_2 = 1/sqrt(2); define this way for UnitTesting

    # The sign() function in the following expression ensures the correct root is taken.
    Cart_to_xx[1] = sp.acos(sp.sign(Cartz)*(
                              sp.sqrt(1 + rSph**2/bScale**2 -
                                      sp.sqrt(bScale**4 + 2*bScale**2*rSph**2 + rSph**4 -
                                              4*bScale**2*rSph**2*sp.cos(thSph)**2)/bScale**2)*M_SQRT1_2)) # M_SQRT1_2 = 1/sqrt(2); define this way for UnitTesting

    Cart_to_xx[2] = phSph

    scalefactor_orthog[0] = sp.diff(AA,xx[0]) * var1 / var2
    scalefactor_orthog[1] = var1
    scalefactor_orthog[2] = AA * sp.sin(xx[1])

    # Set the transpose of the matrix of unit vectors
    UnitVectors = [[sp.sin(xx[1]) * sp.cos(xx[2]) * var2 / var1,
                    sp.sin(xx[1]) * sp.sin(xx[2]) * var2 / var1,
                    AA * sp.cos(xx[1]) / var1],
                   [AA * sp.cos(xx[1]) * sp.cos(xx[2]) / var1,
                    AA * sp.cos(xx[1]) * sp.sin(xx[2]) / var1,
                        -sp.sin(xx[1]) * var2 / var1],
                   [-sp.sin(xx[2]), sp.cos(xx[2]), sp.sympify(0)]]
```

```python
%matplotlib inline

import numpy as np               # NumPy: A numerical methods module for Python
import matplotlib.pyplot as plt  # matplotlib: Python module specializing in plotting capabilities

Nxx0   = 12
Nxx1   = 24
xx0    = np.linspace(0,1,Nxx0,endpoint=True)
xx1    = np.linspace(0,np.pi,Nxx1,endpoint=True)
xx2    = 0
bScale = 1
sinhA  = 2.2
sinhW  = 0.3

def rtilde(xx0,sinhA,sinhW):
    return sinhA * np.sinh(xx0/sinhW) / np.sinh(1.0/sinhW)
def x(xx0,xx1,sinhA,sinhW):
    return rtilde(xx0,sinhA,sinhW) * np.sin(xx1)
def z(xx0,xx1,bScale,sinhA,sinhW):
    return np.sqrt(rtilde(xx0,sinhA,sinhW)**2 + bScale**2) * np.cos(xx1)

plt.clf()

fig = plt.figure(dpi=160)
ax = fig.gca()

# Set plot title
ax.set_title(r"""SinhSymTP Coordinates: zx-plane ($xx_{2}$=0 and $xx_{2}=\pi$)
Blue (red) lines have constant $xx_{0}$ ($xx_{1}$)""")

ax.set_xlim(-2.5,2.5)
ax.set_ylim(-2.5,2.5)
ax.set_aspect('equal')
ax.axis('off')

for i0 in range(Nxx0):
    plt.plot(x(xx0[i0],xx1,sinhA,sinhW),z(xx0[i0],xx1,bScale,sinhA,sinhW),'b',lw=0.75)
    plt.plot(-x(xx0[i0],xx1,sinhA,sinhW),z(xx0[i0],xx1,bScale,sinhA,sinhW),'b',lw=0.75)

for i1 in range(Nxx1):
    plt.plot(x(xx0,xx1[i1],sinhA,sinhW),z(xx0,xx1[i1],bScale,sinhA,sinhW),'r',lw=0.75)
    plt.plot(-x(xx0,xx1[i1],sinhA,sinhW),z(xx0,xx1[i1],bScale,sinhA,sinhW),'r',lw=0.75)

plt.show()
```

# Step 4: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-Reference_Metric.pdf](Tutorial-Reference_Metric.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-Reference_Metric")
```

    Created Tutorial-Reference_Metric.tex, and compiled LaTeX file to PDF file
        Tutorial-Reference_Metric.pdf


# Tetracycline Resistance Model

(c) 2021 Tom Röschinger. This work is licensed under a
[Creative Commons Attribution License CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/).
All code contained herein is licensed under an
[MIT license](https://opensource.org/licenses/MIT).

```julia
using SymPy, Polynomials, Jedi, ColorSchemes, Turing, Distributions, LinearAlgebra

# Comment this out if Jedi.jl is not installed
#Jedi.default_gr!()
```

In this notebook we go through everything bla bla

```julia
@syms a λ λ_0 κ_t K_d Δr j V_0 a_ex K_M P_out P_in
```

    (a, λ, λ_0, κ_t, K_d, Δr, j, V_0, a_ex, K_M, P_out, P_in)

```julia
eq1 = (a * λ) / (κ_t * K_d) - Δr * (1 - λ/λ_0)
```

$\begin{equation*}- Δr \left(- \frac{λ}{λ_{0}} + 1\right) + \frac{a λ}{K_{d} κ_{t}}\end{equation*}$

```julia
eq2 = a^2 * (-λ - P_out) + a * (-λ * K_M  + P_in * a_ex - P_out * K_M - V_0 * λ/λ_0 ) + P_in * a_ex * K_M
```

$\begin{equation*}K_{M} P_{in} a_{ex} + a^{2} \left(- P_{out} - λ\right) + a \left(- K_{M} P_{out} - K_{M} λ + P_{in} a_{ex} - \frac{V_{0} λ}{λ_{0}}\right)\end{equation*}$

```julia
expr1 = solve(eq1, a)[1]
```

$\begin{equation*}\frac{K_{d} Δr κ_{t} \left(- λ + λ_{0}\right)}{λ λ_{0}}\end{equation*}$

```julia
expr2 = subs(eq2, a=>expr1)
```

$\begin{equation*}K_{M} P_{in} a_{ex} + \frac{K_{d}^{2} Δr^{2} κ_{t}^{2} \left(- P_{out} - λ\right) \left(- λ + λ_{0}\right)^{2}}{λ^{2} λ_{0}^{2}} + \frac{K_{d} Δr κ_{t} \left(- λ + λ_{0}\right) \left(- K_{M} P_{out} - K_{M} λ + P_{in} a_{ex} - \frac{V_{0} λ}{λ_{0}}\right)}{λ λ_{0}}\end{equation*}$

```julia
expr3 = expr2 * λ^2 * λ_0^2
```

$\begin{equation*}λ^{2} λ_{0}^{2} \left(K_{M} P_{in} a_{ex} + \frac{K_{d}^{2} Δr^{2} κ_{t}^{2} \left(- P_{out} - λ\right) \left(- λ + λ_{0}\right)^{2}}{λ^{2} λ_{0}^{2}} + \frac{K_{d} Δr κ_{t} \left(- λ + λ_{0}\right) \left(- K_{M} P_{out} - K_{M} λ + P_{in} a_{ex} - \frac{V_{0} λ}{λ_{0}}\right)}{λ λ_{0}}\right)\end{equation*}$

```julia
expr4 = factor(expr3)
```

$\begin{equation*}K_{M} K_{d} P_{out} Δr κ_{t} λ^{2} λ_{0} - K_{M} K_{d} P_{out} Δr κ_{t} λ λ_{0}^{2} + K_{M} K_{d} Δr κ_{t} λ^{3} λ_{0} - K_{M} K_{d} Δr κ_{t} λ^{2} λ_{0}^{2} + K_{M} P_{in} a_{ex} λ^{2} λ_{0}^{2} - K_{d}^{2} P_{out} Δr^{2} κ_{t}^{2} λ^{2} + 2 K_{d}^{2} P_{out} Δr^{2} κ_{t}^{2} λ λ_{0} - K_{d}^{2} P_{out} Δr^{2} κ_{t}^{2} λ_{0}^{2} - K_{d}^{2} Δr^{2} κ_{t}^{2} λ^{3} + 2 K_{d}^{2} Δr^{2} κ_{t}^{2} λ^{2} λ_{0} - K_{d}^{2} Δr^{2} κ_{t}^{2} λ λ_{0}^{2} - K_{d} P_{in} a_{ex} Δr κ_{t} λ^{2} λ_{0} + K_{d} P_{in} a_{ex} Δr κ_{t} λ λ_{0}^{2} + K_{d} V_{0} Δr κ_{t} λ^{3} - K_{d} V_{0} Δr κ_{t} λ^{2} λ_{0}\end{equation*}$

```julia
expr5 = collect(expr4, λ)
```

$\begin{equation*}- K_{d}^{2} P_{out} Δr^{2} κ_{t}^{2} λ_{0}^{2} + λ^{3} \left(K_{M} K_{d} Δr κ_{t} λ_{0} - K_{d}^{2} Δr^{2} κ_{t}^{2} + K_{d} V_{0} Δr κ_{t}\right) + λ^{2} \left(K_{M} K_{d} P_{out} Δr κ_{t} λ_{0} - K_{M} K_{d} Δr κ_{t} λ_{0}^{2} + K_{M} P_{in} a_{ex} λ_{0}^{2} - K_{d}^{2} P_{out} Δr^{2} κ_{t}^{2} + 2 K_{d}^{2} Δr^{2} κ_{t}^{2} λ_{0} - K_{d} P_{in} a_{ex} Δr κ_{t} λ_{0} - K_{d} V_{0} Δr κ_{t} λ_{0}\right) + λ \left(- K_{M} K_{d} P_{out} Δr κ_{t} λ_{0}^{2} + 2 K_{d}^{2} P_{out} Δr^{2} κ_{t}^{2} λ_{0} - K_{d}^{2} Δr^{2} κ_{t}^{2} λ_{0}^{2} + K_{d} P_{in} a_{ex} Δr κ_{t} λ_{0}^{2}\right)\end{equation*}$

```julia
println(expr5.coeff(λ, 4))
expr5.coeff(λ, 4)
```

    0

$\begin{equation*}0\end{equation*}$

```julia
println(expr5.coeff(λ, 3))
factor(expr5.coeff(λ, 3), K_d)
```

    K_M*K_d*Δr*κ_t*λ_0 - K_d^2*Δr^2*κ_t^2 + K_d*V_0*Δr*κ_t

$\begin{equation*}- K_{d} Δr κ_{t} \left(- K_{M} λ_{0} + K_{d} Δr κ_{t} - V_{0}\right)\end{equation*}$

```julia
println(expr5.coeff(λ, 2))
factor(expr5.coeff(λ, 2), K_d)
```

    K_M*K_d*P_out*Δr*κ_t*λ_0 - K_M*K_d*Δr*κ_t*λ_0^2 + K_M*P_in*a_ex*λ_0^2 - K_d^2*P_out*Δr^2*κ_t^2 + 2*K_d^2*Δr^2*κ_t^2*λ_0 - K_d*P_in*a_ex*Δr*κ_t*λ_0 - K_d*V_0*Δr*κ_t*λ_0

$\begin{equation*}K_{M} P_{in} a_{ex} λ_{0}^{2} - K_{d}^{2} \left(P_{out} Δr^{2} κ_{t}^{2} - 2 Δr^{2} κ_{t}^{2} λ_{0}\right) - K_{d} \left(- K_{M} P_{out} Δr κ_{t} λ_{0} + K_{M} Δr κ_{t} λ_{0}^{2} + P_{in} a_{ex} Δr κ_{t} λ_{0} + V_{0} Δr κ_{t} λ_{0}\right)\end{equation*}$

```julia
println(expr5.coeff(λ, 1))
factor(expr5.coeff(λ, 1), K_d)
```

    -K_M*K_d*P_out*Δr*κ_t*λ_0^2 + 2*K_d^2*P_out*Δr^2*κ_t^2*λ_0 - K_d^2*Δr^2*κ_t^2*λ_0^2 + K_d*P_in*a_ex*Δr*κ_t*λ_0^2

$\begin{equation*}K_{d} Δr κ_{t} λ_{0} \left(- K_{M} P_{out} λ_{0} + K_{d} \left(2 P_{out} Δr κ_{t} - Δr κ_{t} λ_{0}\right) + P_{in} a_{ex} λ_{0}\right)\end{equation*}$

```julia
println(expr5.coeff(λ, 0))
expr5.coeff(λ, 0)
```

    -K_d^2*P_out*Δr^2*κ_t^2*λ_0^2

$\begin{equation*}- K_{d}^{2} P_{out} Δr^{2} κ_{t}^{2} λ_{0}^{2}\end{equation*}$

```julia
expr5
```

$\begin{equation*}- K_{d}^{2} P_{out} Δr^{2} κ_{t}^{2} λ_{0}^{2} + λ^{3} \left(K_{M} K_{d} Δr κ_{t} λ_{0} - K_{d}^{2} Δr^{2} κ_{t}^{2} + K_{d} V_{0} Δr κ_{t}\right) + λ^{2} \left(K_{M} K_{d} P_{out} Δr κ_{t} λ_{0} - K_{M} K_{d} Δr κ_{t} λ_{0}^{2} + K_{M} P_{in} a_{ex} λ_{0}^{2} - K_{d}^{2} P_{out} Δr^{2} κ_{t}^{2} + 2 K_{d}^{2} Δr^{2} κ_{t}^{2} λ_{0} - K_{d} P_{in} a_{ex} Δr κ_{t} λ_{0} - K_{d} V_{0} Δr κ_{t} λ_{0}\right) + λ \left(- K_{M} K_{d} P_{out} Δr κ_{t} λ_{0}^{2} + 2 K_{d}^{2} P_{out} Δr^{2} κ_{t}^{2} λ_{0} - K_{d}^{2} Δr^{2} κ_{t}^{2} λ_{0}^{2} + K_{d} P_{in} a_{ex} Δr κ_{t} λ_{0}^{2}\right)\end{equation*}$

```julia

function solve_polynomial(aex, λ0, κt, Kd, KM, j, V0, Δr)
    c = zeros(4)
    c1 = κt * Kd * Δr
    c[4] = c1 * (KM * λ0 - c1 + V0)
    c[3] = KM * j * aex * λ0^2 - c1^2 * (j - 2λ0) - c1 * λ0 * (-KM * j  + KM * λ0 + j * aex + V0)
    c[2] = c1 * λ0 * (-KM * j * λ0 + c1 * (2j - λ0) + j * aex * λ0)
    c[1] = -Kd^2*j*Δr^2*κt^2*λ0^2
    pol = Polynomial(c)
    return Polynomials.roots(pol)
end

```

    solve_polynomial (generic function with 1 method)

```julia
_aex = 0  # µM
_λ0 = 0.68  # h**-1
_κt = 0.06  # µM**-1 h**-1
_Kd = .1  # µM
_KM = 100 # µM
_j = 10# h**-1
Pin = 10
Pout = 3
_V0 = 0
_Δr = 46.5  # µM

a_ex_range = [0, 1]
sol_list = zeros(ComplexF64, 3, length(a_ex_range))
for (i, x) in enumerate(a_ex_range)
    # Pack parameters together
    args = (x, _λ0, _κt, _Kd, _KM, _j, _V0, _Δr)
    # Find roots
    sol_list[:, i] = solve_polynomial(args...)
end
```

```julia
sol_list[:, 2]
```

    3-element Vector{ComplexF64}:
         -33.89338895304312 + 0.0im
     -0.0028014943665923423 + 0.0im
        0.20062909641230975 + 0.0im

```julia
p = scatter(
    ylim=[0, 1],
    xscale=:log,
    legendtitle = "V0",
    ylabel="λ/λ0",
    xlabel="a_ex",
    title="Is this the right font?"
)

V_0_list = [0, 1000, 5000, 10000]
for (_V0, c) in zip(V_0_list, palette(:BuPu_6)[2:end])
    y = []
    x = Float64[]
    for i in 1:length(a_ex_range)
        args = (a_ex_range[i], _λ0, _κt, _Kd, _KM, _j, _V0, _Δr)
        solutions = solve_polynomial(args...)
        _y = [imag(x) == 0 ? real(x)/_λ0 : missing for x in solutions]
        push!(y, _y...)
        _x = ones(length(_y)) .* a_ex_range[i]
        push!(x, _x...)
    end
    scatter!(p, x, y, color=c, label="$_V0")
end
savefig(p, "res_model.pdf")
p
```

## Writing Generative Model

Now we are going to write down the model in a Bayesian way and try out various priors.

\begin{align}
\lambda_0 &= 0.68 h^{-1},\\[.5em]
\kappa_t &= 0.06 \mu M^{-1} h^{-1},\\[.5em]
K_d &= 0.1 \mu M,\\[.5em]
K_M &= 10 \mu M,\\[.5em]
\Delta r &= 46.5 \mu M,\\[.5em]
\log j &\sim \text{Norm}(2, 2),\\[.5em]
j &= 10^{\log j} \times 1 s^{-1},\\[.5em]
\log P_\mathrm{in} &\sim \text{Norm}(2, 2),\\[.5em]
P_\mathrm{in} &= 10^{\log P_\mathrm{in}} \times 1 s^{-1},\\[.5em]
\log P_\mathrm{out} &\sim \text{Norm}(2, 2),\\[.5em]
P_\mathrm{out} &= 10^{\log P_\mathrm{out}} \times 1 s^{-1},\\[.5em]
\log V_0 &\sim \text{Norm}(2, 2),\\[.5em]
V_0  &= 10^{\log V_0 } \times 1 \mu M s^{-1}
\end{align}

```julia
@model function fit_growth(growth_rates, aex_arr)

    log_j ~ Normal(2, .2)
    log_V0 ~ Normal(2, .2)
    log_sigma ~ Normal(-3, 1)

    sigma = float(10^log_sigma)
    j = 10^log_j
    V0 = 10^log_V0

    λ0 = 0.68
    κt = 0.06
    Kd = .1
    KM = 10
    Δr = 46.5

    for (i, aex) in enumerate(aex_arr)
        λ = growth_rates[i]
        println(i)
        println(typeof(V0))
        x = [aex, λ0, κt, Kd, KM, j, V0, Δr]
        solutions = solve_polynomial(x)

        #y = [imag(x) == 0 ? real(x)/λ0 : missing for x in solutions]
        #y = filter(x -> !ismissing(x) && (x > 0), y)
        #if length(y) == 3
            #deleteat!(y, 2)
            #opt_y = y[argmin([(λ - root)^2 for root in y])]
        #else
            #opt_y = y[1]
       #end

        #println(opt_y)
        #println(typeof(opt_y))
        #println()
        #growth_rates[i] ~ Normal(opt_y / λ0, sigma)
    end

end
```

    fit_growth (generic function with 3 methods)

```julia
y = Float64[]
for i in 1:length(a_ex_range)
    args = (a_ex_range[i], _λ0, _κt, _Kd, _KM, _j, 0, _Δr)
    solutions = solve_polynomial(args...)
    _y = [imag(x) == 0 ? real(x)/_λ0 : missing for x in solutions]
    push!(y, _y[1])
end
```

```julia
chn = sample(fit_growth(y, a_ex_range), NUTS(0.65), 100)

```

```julia
chn
```

```julia
?ForwardDiff.Dual
```

```julia
?eigvals!
```

    search: eigvals! eigvals

```
eigvals!(A; permute::Bool=true, scale::Bool=true, sortby) -> values
```

Same as [`eigvals`](@ref), but saves space by overwriting the input `A`, instead of creating a copy. The `permute`, `scale`, and `sortby` keywords are the same as for [`eigen`](@ref).

!!! note
    The input matrix `A` will not contain its eigenvalues after `eigvals!` is called on it - `A` is used as a workspace.

# Examples

```jldoctest
julia> A = [1. 2.; 3. 4.]
2×2 Matrix{Float64}:
 1.0  2.0
 3.0  4.0

julia> eigvals!(A)
2-element Vector{Float64}:
 -0.3722813232690143
  5.372281323269014

julia> A
2×2 Matrix{Float64}:
 -0.372281  -1.0
  0.0        5.37228
```

---

```
eigvals!(A, B; sortby) -> values
```

Same as [`eigvals`](@ref), but saves space by overwriting the input `A` (and `B`), instead of creating copies.

!!! note
    The input matrices `A` and `B` will not contain their eigenvalues after `eigvals!` is called. They are used as workspaces.

# Examples

```jldoctest
julia> A = [1. 0.; 0. -1.]
2×2 Matrix{Float64}:
 1.0   0.0
 0.0  -1.0

julia> B = [0. 1.; 1. 0.]
2×2 Matrix{Float64}:
 0.0  1.0
 1.0  0.0

julia> eigvals!(A, B)
2-element Vector{ComplexF64}:
 0.0 - 1.0im
 0.0 + 1.0im

julia> A
2×2 Matrix{Float64}:
 -0.0  -1.0
  1.0  -0.0

julia> B
2×2 Matrix{Float64}:
 1.0  0.0
 0.0  1.0
```

---

```
eigvals!(A::Union{SymTridiagonal, Hermitian, Symmetric}, irange::UnitRange) -> values
```

Same as [`eigvals`](@ref), but saves space by overwriting the input `A`, instead of creating a copy. `irange` is a range of eigenvalue *indices* to search for - for instance, the 2nd to 8th eigenvalues.

---

```
eigvals!(A::Union{SymTridiagonal, Hermitian, Symmetric}, vl::Real, vu::Real) -> values
```

Same as [`eigvals`](@ref), but saves space by overwriting the input `A`, instead of creating a copy. `vl` is the lower bound of the interval to search for eigenvalues, and `vu` is the upper bound.

```julia

```


# Brief introduction to [PyMC3](https://docs.pymc.io/)

```
%matplotlib inline
import sklearn
import scipy.stats as stats
import scipy.optimize
import matplotlib.pyplot as plt
import seaborn as sns
import time
import numpy as np
import os
import pandas as pd
```

```
!pip install pymc3==3.8
import pymc3 as pm
pm.__version__

!pip install arviz
import arviz as az
```

    Collecting pymc3==3.8
      Downloading https://files.pythonhosted.org/packages/32/19/6c94cbadb287745ac38ff1197b9fadd66500b6b9c468e79099b110c6a2e9/pymc3-3.8-py3-none-any.whl (908kB)
         |████████████████████████████████| 911kB 5.7MB/s
    Requirement already satisfied: pandas>=0.18.0 in /usr/local/lib/python3.6/dist-packages (from pymc3==3.8) (1.1.5)
    Collecting arviz>=0.4.1
      Downloading https://files.pythonhosted.org/packages/a9/05/54183e9e57b0793eceb67361dbf4a7c4ed797ae36a04a3287791a564568c/arviz-0.10.0-py3-none-any.whl (1.5MB)
         |████████████████████████████████| 1.5MB 18.1MB/s
    Requirement already satisfied: scipy>=0.18.1 in /usr/local/lib/python3.6/dist-packages (from pymc3==3.8) (1.4.1)
    Requirement already satisfied: theano>=1.0.4 in /usr/local/lib/python3.6/dist-packages (from pymc3==3.8) (1.0.5)
    Requirement already satisfied: tqdm>=4.8.4 in /usr/local/lib/python3.6/dist-packages (from pymc3==3.8) (4.41.1)
    Requirement already satisfied: patsy>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from pymc3==3.8) (0.5.1)
    Requirement already satisfied: numpy>=1.13.0 in /usr/local/lib/python3.6/dist-packages (from pymc3==3.8) (1.19.4)
    Requirement already satisfied: h5py>=2.7.0 in /usr/local/lib/python3.6/dist-packages (from pymc3==3.8) (2.10.0)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.18.0->pymc3==3.8) (2018.9)
    Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.18.0->pymc3==3.8) (2.8.1)
    Requirement already satisfied: setuptools>=38.4 in /usr/local/lib/python3.6/dist-packages (from arviz>=0.4.1->pymc3==3.8) (51.0.0)
    Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from arviz>=0.4.1->pymc3==3.8) (20.8)
    Collecting xarray>=0.16.1
      Downloading https://files.pythonhosted.org/packages/10/6f/9aa15b1f9001593d51a0e417a8ad2127ef384d08129a0720b3599133c1ed/xarray-0.16.2-py3-none-any.whl (736kB)
         |████████████████████████████████| 737kB 41.5MB/s
    Collecting netcdf4
      Downloading https://files.pythonhosted.org/packages/1a/71/a321abeee439caf94850b4f68ecef68d2ad584a5a9566816c051654cff94/netCDF4-1.5.5.1-cp36-cp36m-manylinux2014_x86_64.whl (4.7MB)
         |████████████████████████████████| 4.7MB 47.9MB/s
    Requirement already satisfied: matplotlib>=3.0 in /usr/local/lib/python3.6/dist-packages (from arviz>=0.4.1->pymc3==3.8) (3.2.2)
    Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from theano>=1.0.4->pymc3==3.8) (1.15.0)
    Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->arviz>=0.4.1->pymc3==3.8) (2.4.7)
    Collecting cftime
      Downloading https://files.pythonhosted.org/packages/66/60/bad8525d2c046eb2964911bc412a85ba240b31c7b43f0c19377233992c6c/cftime-1.3.0-cp36-cp36m-manylinux1_x86_64.whl (295kB)
         |████████████████████████████████| 296kB 46.2MB/s
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=3.0->arviz>=0.4.1->pymc3==3.8) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=3.0->arviz>=0.4.1->pymc3==3.8) (1.3.1)
    Installing collected packages: xarray, cftime, netcdf4, arviz, pymc3
      Found existing installation: xarray 0.15.1
        Uninstalling xarray-0.15.1:
          Successfully uninstalled xarray-0.15.1
      Found existing installation: pymc3 3.7
        Uninstalling pymc3-3.7:
          Successfully uninstalled pymc3-3.7
    Successfully installed arviz-0.10.0 cftime-1.3.0 netcdf4-1.5.5.1 pymc3-3.8 xarray-0.16.2
    Requirement already satisfied: arviz in /usr/local/lib/python3.6/dist-packages (0.10.0)
    Requirement already satisfied: scipy>=0.19 in /usr/local/lib/python3.6/dist-packages (from arviz) (1.4.1)
    Requirement already satisfied: numpy>=1.12 in /usr/local/lib/python3.6/dist-packages (from arviz) (1.19.4)
    Requirement already satisfied: netcdf4 in /usr/local/lib/python3.6/dist-packages (from arviz) (1.5.5.1)
    Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from arviz) (20.8)
    Requirement already satisfied: setuptools>=38.4 in /usr/local/lib/python3.6/dist-packages (from arviz) (51.0.0)
    Requirement already satisfied: xarray>=0.16.1 in /usr/local/lib/python3.6/dist-packages (from arviz) (0.16.2)
    Requirement already satisfied: pandas>=0.23 in /usr/local/lib/python3.6/dist-packages (from arviz) (1.1.5)
    Requirement already satisfied: matplotlib>=3.0 in /usr/local/lib/python3.6/dist-packages (from arviz) (3.2.2)
    Requirement already satisfied: cftime in /usr/local/lib/python3.6/dist-packages (from netcdf4->arviz) (1.3.0)
    Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->arviz) (2.4.7)
    Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.23->arviz) (2.8.1)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.23->arviz) (2018.9)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=3.0->arviz) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=3.0->arviz) (1.3.1)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.7.3->pandas>=0.23->arviz) (1.15.0)

## Example: 1d Gaussian with unknown mean.

We use the simple example from the [Pyro intro](https://pyro.ai/examples/intro_part_ii.html#A-Simple-Example). The goal is to infer the weight $\theta$ of an object, given noisy measurements $y$. We assume the following model:
$$
\begin{align}
\theta &\sim N(\mu=8.5, \tau^2=1.0)\\
y \sim &N(\theta, \sigma^2=0.75^2)
\end{align}
$$

Where $\mu=8.5$ is the initial guess.

By Bayes rule for Gaussians, we know that the exact posterior,
given a single observation $y=9.5$, is given by

$$
\begin{align}
\theta|y &\sim N(m, s^s) \\
m &=\frac{\sigma^2 \mu + \tau^2 y}{\sigma^2 + \tau^2}
  = \frac{0.75^2 \times 8.5 + 1 \times 9.5}{0.75^2 + 1^2}
  = 9.14 \\
s^2 &= \frac{\sigma^2 \tau^2}{\sigma^2  + \tau^2}
= \frac{0.75^2 \times 1^2}{0.75^2 + 1^2}= 0.6^2
\end{align}
$$

```
mu = 8.5; tau = 1.0; sigma = 0.75; y = 9.5
m = (sigma**2 * mu + tau**2 * y)/(sigma**2 + tau**2)
s2 = (sigma**2 * tau**2)/(sigma**2 + tau**2)
s = np.sqrt(s2)
print(m)
print(s)
```

    9.14
    0.6

```

with pm.Model() as model:
  theta = pm.Normal('theta', mu=m, sd=tau)
  obs = pm.Normal('obs', mu=theta, sd=sigma, observed=y)
  trace = pm.sample(1000)

az.summary(trace)
```

    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Sequential sampling (2 chains in 1 job)
    NUTS: [theta]
    Sampling chain 0, 0 divergences: 100%|██████████| 1500/1500 [00:01<00:00, 1287.93it/s]
    Sampling chain 1, 0 divergences: 100%|██████████| 1500/1500 [00:00<00:00, 2366.47it/s]
    /usr/local/lib/python3.6/dist-packages/arviz/data/io_pymc3.py:91: FutureWarning: Using `from_pymc3` without the model will be deprecated in a future release. Not using the model will return less accurate and less useful results. Make sure you use the model argument or call from_pymc3 within a model context.
      FutureWarning,

      mean
      sd
      hdi_3%
      hdi_97%
      mcse_mean
      mcse_sd
      ess_mean
      ess_sd
      ess_bulk
      ess_tail
      r_hat

      theta
      9.384
      0.595
      8.169
      10.434
      0.019
      0.013
      1004.0
      1004.0
      995.0
      1152.0
      1.0

```
samples = trace['theta']
print(samples.shape)
post_mean = np.mean(samples)
post_std = np.std(samples)
print([post_mean, post_std])
```

    (2000,)
    [9.383772725220332, 0.5950231559728297]

```

```


# Neuromatch Academy: Week 1, Day 5, Tutorial 1
# Regularization

__Content creators:__ Lyle Ungar, Ravi Teja Konkimalla, Mohitrajhu Lingan Kumaraian, Kevin Machado Gamboa, Kelson Shilling-Scrivo

__Content reviewers:__ Piyush Chauhan, Kelson Shilling-Scrivo

__Content editors:__ Roberto Guidotti

__Production editors:__ Saeed Salehi, Spiros Chavlis

---
# Tutorial Objectives

1.   **Big ANNs are efficient universal approximators due to adaptive basis functions**
2.   **ANN’s memorize some but generalize well**
3.   Regularization as shrinkage of overparameterized models: L1, L2, early stopping
4.   Regularization by Dropout
5. Regularization by Data Augmentation
6. Perils of Hyper-Parameter Tuning
7.   Rethinking generalization

```
#@title Video : Introduction to Regularization
import time
from IPython.display import YouTubeVideo

try: t1;
except NameError: t1=time.time()

video = YouTubeVideo(id="bc1nsP4htVg", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)

video
```

A key idea of neural nets, is that they use models that are "too complex" - complex enough to fit all the noise in the data. One then needs to "regularize" them to make the models fit complex enough, but not too complex. The more complex the model, the better it fits the training data, but if it is too complex, it generalizes less well; it memorizes the training data but is less accurate on future test data.

```
#@markdown Tutorial slides

#@markdown **Do not read them now.**
# you should link the slides for all tutorial videos here (we will store pdfs on osf)

from IPython.display import HTML
HTML('')
```

---
# Setup
Note that some of the code for today can take up to an hour to run. We have therefore "hidden" that code and shown the resulting outputs.

**Ensure you're running a GPU notebook:**
From "Runtime" in the drop-down menu above, click "Change runtime type". Ensure that "Hardware Accelerator" says "GPU".

**Ensure you can save:** From "File", click "Save a copy in Drive"

```
#@title Imports
# Imports
from __future__ import print_function
import torch
import pathlib
import random
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torchvision.datasets import ImageFolder
from torch.utils.data import DataLoader, TensorDataset
import torch.nn.utils.prune as prune
from torch.optim.lr_scheduler import StepLR
import time
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
import copy
from tqdm.auto import tqdm
from IPython.display import HTML, display
```

```
# @title Figure Settings
import ipywidgets as widgets

%matplotlib inline
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```
# @title Loading Animal Faces data
%%capture
!rm -r AnimalFaces32x32/
!git clone https://github.com/arashash/AnimalFaces32x32
!rm -r afhq/
!unzip ./AnimalFaces32x32/afhq_32x32.zip
```

```
# @title Loading Animal Faces Randomized data
%%capture
!rm -r Animal_faces_random/
!git clone https://github.com/Ravi3191/Animal_faces_random.git
!rm -r afhq_random_32x32/
!unzip ./Animal_faces_random/afhq_random_32x32.zip
!rm -r afhq_10_32x32/
!unzip ./Animal_faces_random/afhq_10_32x32.zip
```

```
#@title Seeding for Reproducibility
seed = 90108
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.enabled = False
torch.torch.use_deterministic_algorithms(True)
def seed_worker(worker_id):
  worker_seed = seed % (worker_id+1)
  np.random.seed(worker_seed)
  random.seed(worker_seed)
```

```
# @title Plotting functions
def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    plt.axis(False)
    plt.show()

def plot_weights(norm, labels, ws, title='Weight Size Measurement'):

  plt.figure(figsize=[8,6])
  plt.title(title)
  plt.ylabel('Frobenius Norm Value')
  plt.xlabel('Model Layers')
  plt.bar(labels, ws)
  plt.axhline(y=norm, linewidth=1, color='r', ls='--', label='Total Model F-Norm')
  plt.legend()
  plt.show()
```

Now, lets define a Animal Net model, train, test and main functions which we will use frequently today.

```
##Network Class - Animal Faces
class Animal_Net(nn.Module):
  def __init__(self):
    torch.manual_seed(104)
    super(Animal_Net, self).__init__()
    self.fc1 = nn.Linear(3*32*32, 128)
    self.fc2 = nn.Linear(128, 32)
    self.fc3 = nn.Linear(32, 3)

  def forward(self, x):
    x = x.view(x.shape[0],-1)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    output = F.log_softmax(x, dim=1)
    return output
```

The train function takes in the current model along with the train_loader and loss function and updates the parameters for a single pass of the entire dataset. The test function takes in the current model after every epoch and calculates the accuracy on the test dataset.

```
def train(args, model, device, train_loader, optimizer, epoch,
          reg_function1=None, reg_function2=None, criterion=F.nll_loss):
  """
  Trains the current inpur model using the data
  from Train_loader and Updates parameters for a single pass
  """
  model.train()
  for batch_idx, (data, target) in enumerate(train_loader):
    data, target = data.to(device), target.to(device)
    optimizer.zero_grad()
    output = model(data)
    if reg_function1 is None:
      loss = criterion(output, target)
    elif reg_function2 is None:
      loss = criterion(output, target)+args['lambda']*reg_function1(model)
    else:
      loss = criterion(output, target)+args['lambda1']*reg_function1(model)+args['lambda2']*reg_function2(model)
    loss.backward()
    optimizer.step()

def test(model, device, test_loader, loader = 'Test', criterion=F.nll_loss):
  """
  Tests the current Model
  """
  model.eval()
  test_loss = 0
  correct = 0
  with torch.no_grad():
    for data, target in test_loader:
      data, target = data.to(device), target.to(device)
      output = model(data)
      test_loss += criterion(output, target, reduction='sum').item()  # sum up batch loss
      pred = output.argmax(dim=1, keepdim=True)  # get the index of the max log-probability
      correct += pred.eq(target.view_as(pred)).sum().item()

  test_loss /= len(test_loader.dataset)
  return 100. * correct / len(test_loader.dataset)

def main(args, model, train_loader, val_loader, test_data, reg_function1=None,
         reg_function2=None, criterion=F.nll_loss):
  """
  Trains the model with train_loader and tests the learned model using val_loader
  """

  use_cuda = not args['no_cuda'] and torch.cuda.is_available()
  device = torch.device('cuda' if use_cuda else 'cpu')

  model = model.to(device)
  optimizer = optim.SGD(model.parameters(), lr=args['lr'], momentum=args['momentum'])

  val_acc_list, train_acc_list,param_norm_list = [], [], []
  for epoch in tqdm(range(args['epochs'])):
    train(args, model, device, train_loader, optimizer, epoch,reg_function1=reg_function1,reg_function2=reg_function2)
    train_acc = test(model,device,train_loader, 'Train')
    val_acc = test(model,device,val_loader, 'Val')
    param_norm = calculate_frobenius_norm(model)
    train_acc_list.append(train_acc)
    val_acc_list.append(val_acc)
    param_norm_list.append(param_norm)

  return val_acc_list, train_acc_list, param_norm_list, model, 0
```

---
#Section 1: Regularization is Shrinkage

```
#@title Video : Regularization as Shrinkage
try: t2;
except NameError: t2=time.time()

video = YouTubeVideo(id="B4CsCKViB3k", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)

video
```

One way to think about Regularization is to think in terms of the magnitude of the overall weights of the model. A model with big weights can fit more data perfectly. Wheras a model with smaller weights tend to underperform on the train set but can suprisingly do very well on the test set. Too small of a weights can also be as issue an it can the underfit the model.

This week we use the sum of Frobenius Norm of all the tensors in the model as a measure of the "size of the model".

##Exercise 1: Frobenius Norm
Before we start, let's define the Frobenius norm, sometimes also called the Euclidean norm of an m×n matrix $A$  as the square root of the sum of the absolute squares of its elements.
\begin{equation}
||A||_F= \sqrt{\sum_{i=1}^m\sum_{j=1}^n|a_{ij}|^2}
\end{equation}

This is just a measure of how big the matrix is, analagous to how big a vector is.

 **Hint:** Use functions model.parameters() or model.named_parameters()

```
def calculate_frobenius_norm(model):

  ####################################################################
  # Fill in all missing code below (...),
  # then remove or comment the line below to test your function
  raise NotImplementedError("Define `calculate_frobenius_norm` function")
  ####################################################################

  norm = 0.0

  # Sum the square of all parameters
  for param in model.parameters():
      norm += ...

  # Take a square root of the sum of squares of all the parameters
  norm = ...
  return norm

# # uncomment to run
# net = nn.Linear(10, 1)
# print(f'Frobenius Norm of Single Linear Layer: {calculate_frobenius_norm(net)}')
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D5_Regularization/solutions/W1D5_Tutorial1_Solution_7d499503.py)

Apart from calculating the weight size for an entire model, we could also determine the weight size in every layer. For this, we can modify our `calculate_frobenius_norm` function as shown below.

**Have a look how it works!!**

```
#@markdown Frobenius Norm per Layer
def calculate_frobenius_norm(model):

  # initialization of variables
  norm, ws, labels = 0.0, [], []

  # Sum all the parameters
  for name, parameters in model.named_parameters():
    p = torch.sum(parameters**2)
    norm += p

    ws.append((p**0.5).cpu().detach().numpy())
    labels.append(name)

  # Take a square root of the sum of squares of all the parameters
  norm = (norm**0.5).cpu().detach().numpy()

  return norm, ws, labels

#uncomment to run
net = nn.Linear(10,1)
norm, ws, labels = calculate_frobenius_norm(net)
print(f'Frobenius Norm of Single Linear Layer: {norm}')
# Plots the weights
plot_weights(norm, labels, ws)
```

Using the last function `calculate_frobenius_norm`, we can also obtain the Frobenius Norm per layer for a whole NN model and use the `plot_weigts` function to visualize them.

```
# Creates a new model
model = Animal_Net()

# Calculates the forbenius norm per layer
norm, ws, labels = calculate_frobenius_norm(model)
print(f'Frobenius Norm of Models weights: {norm}')

# Plots the weights
plot_weights(norm, labels, ws)
```

---
#Section 2: Overfitting

```
#@title Video : Overfitting
try: t3;
except NameError: t3=time.time()

from IPython.display import YouTubeVideo
video = YouTubeVideo(id="RlaGyRKP2nY", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)

video
```

##Visualizing Overfitting

Let's create some synthetic dataset that we will use to illustrate overfitting in neural networks.

```
#creating train data
X = torch.rand((10, 1))
X.sort(dim = 0)
Y = 2*X + 2*torch.empty((X.shape[0], 1)).normal_(mean=0, std=1) #adding small error in the data

X = X.unsqueeze_(1)
Y = Y.unsqueeze_(1)

#visualizing trian data
plt.figure(figsize=(8, 6))
plt.scatter(X.numpy(),Y.numpy())
plt.xlabel('input (x)')
plt.ylabel('output(y)')
plt.title('toy dataset')
plt.show()

#creating test dataset
X_test = torch.linspace(0, 1, 40)
X_test = X_test.reshape((40, 1, 1))
```

Let's create an overparametrized NN that can fit on the dataset we just created and train it.

First, let's build the model architecture:

```
##Network Class - 2D
class Net(nn.Module):
  def __init__(self):
    super(Net, self).__init__()

    self.fc1 = nn.Linear(1, 300)
    self.fc2 = nn.Linear(300, 500)
    self.fc3 = nn.Linear(500, 1)

  def forward(self, x):
    x = F.leaky_relu(self.fc1(x))
    x = F.leaky_relu(self.fc2(x))
    output = self.fc3(x)
    return output
```

Next, let's define the different parameters for training our model:

```
#train the network on toy dataset
model = Net()

criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr = 1e-4)
max_epochs = 10000
iters = 0
# Calculates frobenius before training
normi, wsi, label = calculate_frobenius_norm(model)
```

At this point, we can now train our model.

```
# Initializing variables to store weights
per_layer = []

running_predictions = np.empty((40, (int)(max_epochs/500 + 1)))

train_loss = []
test_loss = []
model_norm = []

for epoch in tqdm(range(max_epochs)):

  norm, pl, layer_names = calculate_frobenius_norm(model)

  #training
  model_norm.append(norm)
  per_layer.append(pl)
  model.train()
  optimizer.zero_grad()
  predictions = model(X)
  loss = criterion(predictions, Y)
  loss.backward()
  optimizer.step()

  train_loss.append(loss.data)
  model.eval()
  Y_test = model(X_test)
  loss = criterion(Y_test, 2*X_test)
  test_loss.append(loss.data)

  if(epoch % 500 == 0 or epoch == max_epochs - 1):
    running_predictions[:, iters] = Y_test[:, 0, 0].detach().numpy()
    iters += 1
```

Now that we have finished training, let's see how the model has evolved over the training process.

```
#@title Animation (Run Me!)

# create a figure and axes
fig = plt.figure(figsize=(14, 5))
ax1 = plt.subplot(1,2,1)
ax2 = plt.subplot(1,2,2)
# organizing subplots
plot1, = ax1.plot([],[])
plot2 = ax2.bar([], [])

def frame(i):
  ax1.clear()
  title1 = ax1.set_title('')
  ax1.set_xlabel("Input(x)")
  ax1.set_ylabel("Output(y)")

  ax2.clear()
  ax2.set_xlabel('Layer names')
  ax2.set_ylabel('Frobenius norm')
  title2 = ax2.set_title('Weight Measurement: Forbenius Norm')

  ax1.scatter(X[:,0,:].numpy(),Y[:,0,:].numpy())
  plot1 = ax1.plot(X_test[:,0,:].detach().numpy(),running_predictions[:,i])
  title1.set_text(f'Epochs: {i * 500}')
  plot2 = ax2.bar(label, per_layer[i*500])
  plt.axhline(y=model_norm[i*500], linewidth=1,
              color='r', ls='--', label=f'Norm: {model_norm[i*500]:.2f}')
  plt.legend()

  return plot1, plot2

anim = animation.FuncAnimation(fig, frame, frames=range(20),
                               blit=False, repeat=False, repeat_delay=10000)
html_anim = HTML(anim.to_html5_video());
plt.close()
display(html_anim)
```

```
#@title Plot the train and test losses
plt.figure(figsize=(8, 6))
plt.plot(train_loss,label='train_loss')
plt.plot(test_loss,label='test_loss')
plt.ylabel('loss')
plt.xlabel('epochs')
plt.title('loss vs epoch')
plt.legend()
plt.show()
```

Now let's visualize the Frobenious norm of the model as we trained. You should see that the value of weights increases over the epochs.

```
#@title Frobenious norm of the model
plt.figure(figsize=(8, 6))
plt.plot(model_norm)
plt.ylabel('norm of the model')
plt.xlabel('epochs')
plt.title('Size of the model vs Epochs') ## Change title to Frobenious norm of the model
plt.show()
```

Finally, you can compare the Frobenius norm for every layer in the model, before and after training.

```
#@title Frobenius norm per layer before and after training
normf, wsf, label = calculate_frobenius_norm(model)

plot_weights(float(normi), label, wsi, title='Weight Size Before Training')
plot_weights(float(normf), label, wsf, title='Weight Size After Training')
```

What trend do you see w.r.t to train and test losses ( Where do you see the minimum of these losses?) What does it tell about the model we trained? Discuss among yourselves.

## Overfitting on Test Dataset

In principle, we should not touch our test set until after we have chosen all our hyperparameters. Were we to use the test data in the model selection process, there is a risk that we might overfit the test data. Then we would be in serious trouble. If we overfit our training data, there is always the evaluation on test data to keep us honest. But if we overfit the test data, how would we ever know?

Note that there is another kind of overfitting: you do "honest" fitting on one set of images or posts, or medical records, but it may not generalize to other sets of images, posts or medical records.

##Validation Dataset
A common practice to address this problem is to split our data three ways, using a validation dataset (or validation set) to tune the hyperparameters. Ideally, we would only touch the test data once, to assess the very best model or to compare a small number of models to each other, real-world test data is seldom discarded after just one use.

---
# Section 3: Memorization

Given sufficiently large networks and enough training, Neural Networks can acheive almost 100% train accuracy.

In this section we train three MLP's one each on:

1.   Animal Faces Dataset
2.   A Completely Noisy Dataset (Random Shuffling of all labels)
3.   Partially Noisy Dataset (Random Shuffling of 15% labels)

Now, think for a couple of minutes as to what the train and test accuracies of each of these models might be, given you train for sufficient time and use a powerful network.

First, create the required dataloaders for all three datasets. Notice how we split the data. We train on a fraction of the dataset as it will be faster to train and will overfit more clearly.

```
##Dataloaders for the Dataset
batch_size = 128
classes = ('cat', 'dog', 'wild')

train_transform = transforms.Compose([
     transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
     ])
data_path = pathlib.Path('.')/'afhq' # using pathlib to be compatible with all OS's
img_dataset = ImageFolder(data_path/'train', transform=train_transform)

####################################################

##Dataloaders for the  Original Dataset

img_train_data, img_val_data,_ = torch.utils.data.random_split(img_dataset, [100,100,14430])

#Creating train_loader and Val_loader
train_loader = torch.utils.data.DataLoader(img_train_data,
                                           batch_size=batch_size,
                                           worker_init_fn=seed_worker)
val_loader = torch.utils.data.DataLoader(img_val_data,
                                         batch_size=1000,
                                         worker_init_fn=seed_worker)

#creating test dataset
test_transform = transforms.Compose([
     transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
     ])
img_test_dataset = ImageFolder(data_path/'val', transform=test_transform)

####################################################

##Dataloaders for the  Random Dataset

#splitting randomized data into training and validation data
data_path = pathlib.Path('.')/'afhq_random_32x32/afhq_random' # using pathlib to be compatible with all OS's
img_dataset = ImageFolder(data_path/'train', transform=train_transform)
random_img_train_data, random_img_val_data,_ = torch.utils.data.random_split(img_dataset, [100, 100, 14430])

#Randomized train and validation dataloader
rand_train_loader = torch.utils.data.DataLoader(random_img_train_data,
                                                batch_size=batch_size,
                                                num_workers=0,
                                                worker_init_fn=seed_worker)
rand_val_loader = torch.utils.data.DataLoader(random_img_val_data,
                                              batch_size=1000,
                                              num_workers=0,
                                              worker_init_fn=seed_worker)

####################################################

##Dataloaders for the Partially Random Dataset

#Splitting data between training and validation dataset for partially randomized data
data_path = pathlib.Path('.')/'afhq_10_32x32/afhq_10' # using pathlib to be compatible with all OS's
img_dataset = ImageFolder(data_path/'train', transform=train_transform)
partially_random_train_data, partially_random_val_data,_ = torch.utils.data.random_split(img_dataset, [100, 100, 14430])

#Training and Validation loader for partially randomized data
partial_rand_train_loader = torch.utils.data.DataLoader(partially_random_train_data,
                                                        batch_size=batch_size,
                                                        num_workers=0,
                                                        worker_init_fn=seed_worker)
partial_rand_val_loader = torch.utils.data.DataLoader(partially_random_val_data,
                                                      batch_size=1000,
                                                      num_workers=0,
                                                      worker_init_fn=seed_worker)
```

Now let's define a model which has many parameters compared to the training dataset size, and train it on these datasets.

```
# # Network Class - Animal Faces
class Big_Animal_Net(nn.Module):
  def __init__(self):
    torch.manual_seed(104)
    super(Big_Animal_Net, self).__init__()
    self.fc1 = nn.Linear(3*32*32, 124)
    self.fc2 = nn.Linear(124, 64)
    self.fc3 = nn.Linear(64, 3)

  def forward(self, x):
    x = x.view(x.shape[0], -1)
    x = F.leaky_relu(self.fc1(x))
    x = F.leaky_relu(self.fc2(x))
    x = self.fc3(x)
    output = F.log_softmax(x, dim=1)
    return output
```

Before training our `Big_AnimalNet()`, calculate the Frobenius norm again.

```
model = Big_Animal_Net()
normi, wsi, label = calculate_frobenius_norm(model)
```

Now, train our `Big_AnimalNet()` model

```
# # Here we have 100 true train data.
args = {'test_batch_size': 1000,
        'epochs': 200,
        'lr': 5e-3,
        'momentum': 0.9,
        'no_cuda': False,
        }

acc_dict = {}

start_time = time.time()
val_acc_pure, train_acc_pure, _, model ,_ = main(args,
                                                 model,
                                                 train_loader,
                                                 val_loader,
                                                 img_test_dataset)
end_time = time.time()

print("Time to memorize the dataset:",end_time - start_time)

# # Train and Test accuracy plot
plt.figure(figsize=(8, 6))
plt.plot(val_acc_pure, label='Val Accuracy Pure', c='red', ls='dashed')
plt.plot(train_acc_pure, label='Train Accuracy Pure', c='red', ls='solid')
plt.axhline(y=max(val_acc_pure), c='green', ls='dashed')
plt.title('Memorization')
plt.ylabel('Accuracy (%)')
plt.xlabel('Epoch')
plt.legend()
plt.show()
```

```
#@title Frobenius norm for AnimalNet before and after training
normf, wsf, label = calculate_frobenius_norm(model)

plot_weights(float(normi), label, wsi, title='Weight Size Before Training')
plot_weights(float(normf), label, wsf, title='Weight Size After Training')
```

##Exercise 2: Data Visualizer
Before we train the model on a data with random labels, let's visualize and verify for ourselves that the data is random. Here, we have classes = ("cat", "dog", "wild").

**Hint:** Use .permute() method. plt.imshow() expects imput to be in numpy format and in the format (Px,Py,3), where Px and Py are the number of pixels along axis x and y respectively.

```
def Visualize_data(dataloader):
  ####################################################################
  # Fill in all missing code below (...),
  # then remove or comment the line below to test your function
  # The dataloader here gives out mini batches of 100 data points.
  raise NotImplementedError("Complete the Visualize_random_data function")
  ####################################################################

  for idx, (data,label) in enumerate(dataloader):
    plt.figure(idx)
    # Choose the datapoint you would like to visualize
    index = ...

    # choose that datapoint using index and permute the dimensions
    # and bring the pixel values between [0,1]
    data = ...

    # Convert the torch tensor into numpy
    data = ...

    plt.imshow(data)
    plt.axis(False)
    image_class = classes[...]
    print(f'The image belongs to : {image_class}')

  plt.show()

# # uncomment to run the function
# Visualize_data(rand_train_loader)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D5_Regularization/solutions/W1D5_Tutorial1_Solution_529efdce.py)

*Example output:*

Now let's train the network on the shuffled data and see if it memorizes.

```
##Here we have 100 completely shuffled train data.
args = {'epochs': 200,
        'lr': 5e-3,
        'momentum': 0.9,
        'no_cuda': False
        }

acc_dict = {}
model = Big_Animal_Net()

val_acc_random, train_acc_random, _,model,_ = main(args,
                                                   model,
                                                   rand_train_loader,
                                                   val_loader,
                                                   img_test_dataset)

##Train and Test accuracy plot
plt.figure(figsize=(8, 6))
plt.plot(val_acc_random, label='Val Accuracy random', c='red', ls='dashed')
plt.plot(train_acc_random, label='Train Accuracy random', c='red', ls='solid')
plt.axhline(y=max(val_acc_random), c='green', ls='dashed')
plt.title('Memorization')
plt.ylabel('Accuracy (%)')
plt.xlabel('Epoch')
plt.legend()
plt.show()
```

Finally lets train on a parially shuffled dataset where 15% of the labels are noisy.

```
##Here we have 100 partially shuffled train data.
args = {'test_batch_size': 1000,
        'epochs': 200,
        'lr': 5e-3,
        'momentum': 0.9,
        'no_cuda': False,
        }

acc_dict = {}
model = Big_Animal_Net()

val_acc_shuffle, train_acc_shuffle, _,_,_ = main(args,
                                                 model,
                                                 partial_rand_train_loader,
                                                 val_loader,
                                                 img_test_dataset)

#train and test acc plot
plt.figure(figsize=(8, 6))
plt.plot(val_acc_shuffle, label='Val Accuracy shuffle', c='red', ls='dashed')
plt.plot(train_acc_shuffle, label='Train Accuracy shuffle', c='red', ls='solid')
plt.axhline(y=max(val_acc_shuffle), c='green', ls='dashed')
plt.title('Memorization')
plt.ylabel('Accuracy (%)')
plt.xlabel('Epoch')
plt.legend()
plt.show()
```

```
#@title Plotting them all together (Run Me!)
plt.figure(figsize=(8, 6))
plt.plot(val_acc_pure,label='Val - Pure',c='red',ls = 'dashed')
plt.plot(train_acc_pure,label='Train - Pure',c='red',ls = 'solid')
plt.plot(val_acc_random,label='Val - Random',c='blue',ls = 'dashed')
plt.plot(train_acc_random,label='Train - Random',c='blue',ls = 'solid')
plt.title('Memorization')
plt.ylabel('Accuracy (%)')
plt.xlabel('Epoch')
plt.legend()
plt.show()
```

Given that the NN fit/memorize the training data perfectly, Do you think it generalizes well? What makes you think it does or doesn't?

Isn't it supprising to see that the NN was able to acheive 100% training accuracy on randomly shuffled labels. This is one of the reasons why training accuracy is not a good indicator of model performance.

Also it is interesting to note that sometimes the model trained on slightly shuffled data does slightly better than the one trained on pure data.  Shuffling some of the data is a form of regularization--one of many ways of adding noise to the training data.

---
# Section 4: Early Stopping

```
#@title Video : Early Stopping
try: t4;
except NameError: t4=time.time()

from IPython.display import YouTubeVideo
video = YouTubeVideo(id="GA6J-50GCWs", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)

video
```

Now that we have established that the validation accuracy reaches the peak well before the model overfits we want to somehow stop the training early. You should have also observed from the above plots that the train/test loss on real data is not very smooth and hence you might guess that the choice of epoch can play a very large role on the val/test accuracy of your model.

Early stopping stops training when the validation accuracies stop increasing.

## Exercise 3: Early Stopping
Reimplement the main function to include early stopping as described above. Then run the code below to validate your implementation.

```
def early_stopping_main(args, model, train_loader, val_loader, test_data):

  ####################################################################
  # Fill in all missing code below (...),
  # then remove or comment the line below to test your function
  raise NotImplementedError("Complete the early_stopping_main function")
  ####################################################################

  use_cuda = not args['no_cuda'] and torch.cuda.is_available()
  device = torch.device('cuda' if use_cuda else 'cpu')

  model = model.to(device)
  optimizer = optim.SGD(model.parameters(),
                        lr=args['lr'],
                        momentum=args['momentum'])

  best_acc = 0.0
  best_epoch = 0

  # Number of successive epochs that you want to wait before stopping training process
  patience = 20

  # Keps track of number of epochs during which the val_acc was less than best_acc
  wait = 0

  val_acc_list, train_acc_list = [], []
  for epoch in tqdm(range(args['epochs'])):
    # train the model
    ...
    # calculate training accuracy
    train_acc = ...
    # calculate validation accuracy
    val_acc = ...
    if (val_acc > best_acc):
      best_acc = val_acc
      best_epoch = epoch
      best_model = copy.deepcopy(model)
      wait = 0
    else:
      wait += 1
    if (wait > patience):
      print(f'early stopped on epoch:{epoch}')
      break
    train_acc_list.append(train_acc)
    val_acc_list.append(val_acc)

  return val_acc_list, train_acc_list, best_model, best_epoch
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D5_Regularization/solutions/W1D5_Tutorial1_Solution_08f7091a.py)

```
args = {'epochs': 200,
        'lr': 5e-4,
        'momentum': 0.99,
        'no_cuda': False,
        }

model = Animal_Net()
val_acc_earlystop, train_acc_earlystop, _, best_epoch = early_stopping_main(args,
                                                                            model,
                                                                            train_loader,
                                                                            val_loader,
                                                                            img_test_dataset)

print("Maximum Validation Accuracy is reached at epoch:%2d"%(best_epoch))
```

Do you think early stopping can be harmful for the training of your network? Discuss among your pod why or why not?