# Demo - LISA Horizon Distance
This demo shows how to use ``LEGWORK`` to compute the horizon distance for a collection of sources.

```python
%matplotlib inline
```

```python
import legwork as lw
import numpy as np
import astropy.units as u
import matplotlib.pyplot as plt
```

```python
%config InlineBackend.figure_format = 'retina'

plt.rc('font', family='serif')
plt.rcParams['text.usetex'] = False
fs = 24

# update various fontsizes to match
params = {'figure.figsize': (12, 8),
          'legend.fontsize': fs,
          'axes.labelsize': fs,
          'xtick.labelsize': 0.9 * fs,
          'ytick.labelsize': 0.9 * fs,
          'axes.linewidth': 1.1,
          'xtick.major.size': 7,
          'xtick.minor.size': 4,
          'ytick.major.size': 7,
          'ytick.minor.size': 4}
plt.rcParams.update(params)
```

## Horizon distance of circular binaries
The horizon distance for a source is the maximum distance at which the SNR of a source is still above some detectable threshold. The horizon distance can be computed from the SNR as follows since it is inversely proportional to the distance.
$$\begin{equation}
    \label{eq:snr_to_hor_dist}
    D_{\rm hor} = \frac{\rho(D)}{\rho_{\rm detect}} \cdot D,
\end{equation}$$
Where $\rho(D)$ is the SNR at some distance $D$ and $\rho_{\rm detect}$ is the SNR above which we consider a source detectable.

Let's start doing this by creating a grid of chirp masses and orbital frequencies and creating a Source class from them.

```python
# create a list of masses and frequencies
m_c_grid = np.logspace(-1, np.log10(50), 500) * u.Msun
f_orb_grid = np.logspace(np.log10(4e-5), np.log10(3e-1), 400) * u.Hz

# turn the two lists into grids
MC, FORB = np.meshgrid(m_c_grid, f_orb_grid)

# flatten grids
m_c, f_orb = MC.flatten(), FORB.flatten()

# convert chirp mass to individual masses for source class
q = 1.0
m_1 = m_c / q**(3/5) * (1 + q)**(1/5)
m_2 = m_1 * q

# use a fixed distance and circular binaries
dist = np.repeat(1, len(m_c)) * u.kpc
ecc = np.zeros(len(m_c))

# create the source class
sources = lw.source.Source(m_1=m_1, m_2=m_2, dist=dist, f_orb=f_orb, ecc=ecc, gw_lum_tol=1e-3)
```

Next, we can use `LEGWORK` to compute their merger times and SNRs for the contours.

```python
# calculate merger times and then SNR
sources.get_merger_time()
sources.get_snr(verbose=True)
```

    Calculating SNR for 200000 sources
    	125169 sources are stationary
    		125169 sources are stationary and circular
    	74831 sources are evolving
    		74831 sources are evolving and circular

    array([5.37039831e-04, 5.48303595e-04, 5.59803603e-04, ...,
           1.06420933e+04, 1.07531750e+04, 1.08654183e+04])

We flattened the grid to fit into the Source class but now we can reshape the output to match the original grid.

```python
# reshape the output into grids
t_merge_grid = sources.t_merge.reshape(MC.shape)
snr_grid = sources.snr.reshape(MC.shape)
```

Now we can define a couple of functions for formatting the time, distance and galaxy name contours.

```python
def fmt_time(x):
    if x == 4:
        return r"$t_{\rm merge} = T_{\rm obs}$"
    elif x >= 1e9:
        return "{0:1.0f} Gyr".format(x / 1e9)
    elif x >= 1e6:
        return "{0:1.0f} Myr".format(x / 1e6)
    elif x >= 1e3:
        return "{0:1.0f} kyr".format(x / 1e3)
    elif x >= 1:
        return "{0:1.0f} yr".format(x)
    elif x >= 1/12:
        return "{0:1.0f} month".format(x * 12)
    else:
        return "{0:1.0f} week".format(x * 52)

def fmt_dist(x):
    if x >= 1e9:
        return "{0:1.0f} Gpc".format(x / 1e9)
    elif x >= 1e6:
        return "{0:1.0f} Mpc".format(x / 1e6)
    elif x >= 1e3:
        return "{0:1.0f} kpc".format(x / 1e3)
    else:
        return "{0:1.0f} pc".format(x)

def fmt_name(x):
    if x == np.log10(8):
        return "MW Centre"
    elif x == np.log10(50):
        return "SMC/LMC"
    elif x == np.log10(800):
        return "Andromeda"
    elif x == np.log10(40000):
        return "GW170817"
```

Finally, we put it all together to create a contour plot with all of the information.

```python
# create a square figure plus some space for a colourbar
size = 12
cbar_space = 2
fig, ax = plt.subplots(figsize=(size + cbar_space, size))

# set up scales early so contour labels show up nicely
ax.set_xscale("log")
ax.set_yscale("log")

# set axes labels and lims
ax.set_xlabel(r"Orbital Frequency, $f_{\rm orb} \, [\rm Hz]$")
ax.set_ylabel(r"Chirp Mass, $\mathcal{M}_c \, [\rm M_{\odot}]$")
ax.set_xlim(4e-5, 3e-1)

# calculate the horizon distance
snr_threshold = 7
horizon_distance = (snr_grid / snr_threshold * 1 * u.kpc).to(u.kpc)

# set up the contour levels
distance_levels = np.arange(-3, 6 + 0.5, 0.5)
distance_tick_levels = distance_levels[::2]

# plot the contours for horizon distance
distance_cont = ax.contourf(FORB, MC, np.log10(horizon_distance.value), levels=distance_levels)

# hide edges that show up in rendered PDFs
for c in distance_cont.collections:
    c.set_edgecolor("face")

# create a colour with custom formatted labels
cbar = fig.colorbar(distance_cont, ax=ax, pad=0.02, ticks=distance_tick_levels, fraction=cbar_space / (size + cbar_space))
cbar.ax.set_yticklabels([fmt_dist(np.power(10, distance_tick_levels + 3)[i]) for i in range(len(distance_tick_levels))])
cbar.set_label(r"Horizon Distance", fontsize=fs)
cbar.ax.tick_params(axis="both", which="major", labelsize=0.7 * fs)

# annotate the colourbar with some named distances
named_distances = np.log10([8, 50, 800, 40000])
for name in named_distances:
    cbar.ax.axhline(name, color="white", linestyle="dotted")

# plot the same names as contours
named_cont = ax.contour(FORB, MC, np.log10(horizon_distance.value), levels=named_distances,
                        colors="white", alpha=0.8, linestyles="dotted")
ax.clabel(named_cont, named_cont.levels, fmt=fmt_name, use_clabeltext=True, fontsize=0.7*fs,
          manual=[(1.1e-3, 2e-1), (4e-3, 2.2e-1), (4e-3,1e0), (3e-3, 1.2e1)])

# add a line for when the merger time becomes less than the inspiral time
time_cont = ax.contour(FORB, MC, t_merge_grid.to(u.yr).value, levels=[4],
                       colors="black", linewidths=2, linestyles="dotted") #[1/52, 1/12, 4, 1e2, 1e3, 1e4, 1e5, 1e6, 1e7, 1e8, 1e9, 1e10]
ax.clabel(time_cont, time_cont.levels, fmt=fmt_time, fontsize=0.7*fs, use_clabeltext=True, manual=[(2.5e-2, 5e0)])

# plot a series of lines and annotations for average DCO masses
for m_1, m_2, dco in [(0.6, 0.6, "WDWD"), (1.4, 1.4, "NSNS"),
                      (10, 1.4, "BHNS"), (10, 10, "BHBH"), (30, 30, "BHBH")]:
    # find chirp mass
    m_c_val = lw.utils.chirp_mass(m_1, m_2)

    # plot lines before and after bbox
    ax.plot([4e-5, 4.7e-5], [m_c_val, m_c_val],
            color="black", lw=0.75, zorder=1, linestyle="--")
    ax.plot([1.2e-4, 1e0], [m_c_val, m_c_val],
            color="black", lw=0.75, zorder=1, linestyle="--")

    # plot name and bbox, then masses below in smaller font
    ax.annotate(dco + "\n", xy=(7.5e-5, m_c_val), ha="center", va="center", fontsize=0.7*fs,
               bbox=dict(boxstyle="round", fc="white", ec="white", alpha=0.25))
    ax.annotate(r"${{{}}} + {{{}}}$".format(m_1, m_2),
               xy=(7.5e-5, m_c_val * 0.95), ha="center", va="top", fontsize=0.5*fs)

# ensure that everyone knows this only applies for circular sources
ax.annotate(r"$e = 0.0$", xy=(0.03, 0.04), xycoords="axes fraction", fontsize=0.8*fs,
            bbox=dict(boxstyle="round", fc="white", ec="white", alpha=0.25))

ax.set_facecolor(plt.get_cmap("viridis")(0.0))

plt.show()
```


# Tudo que você precisa saber sobre: Regressão Linear

> *Note que o título é: "Tudo que você precisa saber", mas entenda que o objetivo principal aqui é explicar conceitos com uma abordagem bem simples, dar alguns exemplos e referências. Isto não é um estudo profundo do assunto.*

## Conceitos básicos
Regressão Linear é um método estatístico para encontrar relacionamentos entre variáveis dependentes e independentes. Pode ser usado para machine learning já que alguns problemas simples podem ser resolvidos com regressão linear.

Encontramos este relacionamento achando a reta que se encaixa melhor nos dados atuais, dando-nos poder para prever novos dados.

Nós podemos minimizar o erro entre a linha e os dados atuais para aprimorar nossa capacidade de predição

A forma como minimizamos o erro é mudando a posição da reta usando a equação:

\begin{equation}
y = c + b*x
\end{equation}

Where:
+ y = valor estimado da variável dependente
+ c = constante
+ b = coeficiente de regressão
+ x = valor da variável dependente

## Exemplos

Um bom exemplo de Regressão Linear é prever o peso de um animal tendo como referência o peso de seu cérebro.

Podemos fazer isso facilmente usando o scikit-learn:

```python
%matplotlib inline

## Importando dependencias
import pandas as pd
from sklearn import linear_model
import matplotlib.pyplot as plt

## Carregando dados
df = pd.read_fwf('brain_body.txt')

## Usando peso do cérebro como variável independente
x_values = df[['Brain']].values.tolist()
## Usando peso do corpo como variável dependente
y_values = df[['Body']].values.tolist()

## Treinando modelo nos dados
body_weight_reg = linear_model.LinearRegression()
body_weight_reg.fit(x_values, y_values)

## Imprimindo resultados
plt.scatter(x_values, y_values)
plt.plot(x_values, body_weight_reg.predict(x_values))
plt.show()
```

## Recursos

### Artigos
+ [Artigo do Wikipédia](https://pt.wikipedia.org/wiki/Regress%C3%A3o_linear)
+ [Artigo da Statistics Solutions](http://www.statisticssolutions.com/what-is-linear-regression/)
+ [Aula do PennState College of Science](https://onlinecourses.science.psu.edu/stat501/node/250)
+ [Exemplos de regressão linear do Scikit-learn](http://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html)


```
import numpy as np

#######################################################
######                                           ######
#####   Denne koden skal tilnærme kvadratroten    #####
#####          av et heltall n med                #####
#####          et gitt antall desimaler           #####
######                                           ######
#######################################################

################################################
########                                ########
###### funksjonen tar inn to parametre    ######
#####     n og s som gir ut sqrt(n) med    #####
#####         1/10**s feilmargin og        #####
######  forteller hvor mange iterasjoner  ######
########                                ########
################################################

def herons(n,s): # s betegner hvor mange desimaler man ønsker at skal stemme
  ref = np.sqrt(n) # bruker kvadratrota som referanse

  # lagrer sidelengdene i matriser X og Y
  X = [1]    # starter med sidelengde lik 1
  Y = [n]    # den andre sidelengden er n
  diff = 1   # definerer hvor lenge vi skal holde på
  i = 0      # teller antall iterasjoner
  while diff>10**(-s):
    x = (X[-1]+Y[-1])/2    # ny approksimasjon av n
    y = n/x                #
    X.append(x)            # tildeler nye verdier for x og y
    Y.append(y)            # i matrisene X og Y
    diff = np.abs(ref-x)   # sjekker om svaret er presist nok
    i = i+1                # i = i+1 teller hvor mange runder som trengs
  print(                   # printer et svar
        "Etter å ha brukt Herons metode "
        + str(i)
        + (" gang" if i == 1 else " ganger")
        + ", har vi tilnærmet kvadratroten til "
        + str(n)
        + " med minst "
        + str(s)
        + (" desimal" if s==1 else " desimaler")
        )

  print("")
  print(                  # skriver den siste approksimasjonen for n
      "Tilnærmingen er da "+str(X[-1]
                                )
        )
  print(X)
  print(Y)
```

# New Section

Blabla
\begin{equation}
x^2 = 5 t
  \end{equation}

```
herons(2,1)
```

    Etter å ha brukt Herons metode 1 gang,⋅  har vi tilnærmet kvadratroten til 2 med minst 1 desimal

    Tilnærmingen er da 1.5
    [1, 1.5]
    [2, 1.3333333333333333]

# New Section


# Spatial preprocessing
*If you haven't gone through the `temporal_preprocessing.ipynb` notebook, please do so first!*

In the previous notebook, we've looked at how to temporally preprocess our signal through high-pass filtering, prewhitening, and nuisance regression. Importantly, these operations are *performed on the timeseries of each voxel separately*! Essentially, given our 4D fMRI data ($X \times Y \times Z \times T$), temporal filtering as we discussed here is only applied to the fourth dimension (the time-dimension, $T$).

In addition to these temporal operations, there are several spatial operations performed on (f)MRI data, including spatial smoothing (a form of filtering), distortion correction, and spatial registration/resampling, which are discussed in this notebook.

**What you'll learn**: after this lab, you'll ...

- explain the necessity of motion correction and the advantage of motion regression
- know how to implement the concepts above in Python

**Estimated time needed to complete**: 4-6 hours

```python
# import some functionality
import os.path as op
import numpy as np
import nibabel as nib
import matplotlib.pyplot as plt
from scipy.ndimage import gaussian_filter
```

## Spatial filtering (blurring/smoothing)
Spatial filtering refers to operations applied to the *spatial* dimensions ($X$, $Y$, and $Z$) of our 4D data. The most common spatial filtering operation — spatial smoothing — is usually implemented using a "3D gaussian (low-pass) smoothing kernel". Sounds familiar? Well, it should, because it's essentially the same type of filter as we used for our temporal high-pass filter! Only this time, it's not a 1-dimensional gaussian ("kernel") that does the high-pass filtering, but it's a 3-dimensional gaussian that does *low-pass* filtering. Just like the temporal gaussian-weighted running line smoother is applied across time, we apply the 3D gaussian across space (i.e., the 3 spatial dimensions, $X$, $Y$, and $Z$). The figure below schematically visualized the process\*:

(Note that we show the spatial data in 2D, simply because it's easier to visualize, but in reality this is always 3D!)

In fact, because both (high-pass) temporal filtering and (low-pass) spatial filtering in most fMRI applications depend on the same "gaussian filtering" principle, we can even use the same Python function: `gaussian_filter`! However, as we mentioned before, spatial smoothing in fMRI is used as a *low-pass filter*, which means that the (spatial!) frequencies *higher* than a certain cutoff are filtered out, while the (spatial!) frequencies *lower* than this cut-off are *passed*. Therefore, we don't need to subtract the output from the `gaussian_filter` function from the (spatial) data! (If we'd would that, than we're effectively high-pass filtering the data!)

---
\* 3D gaussian figure from Philipp Klaus.

Before we go on and demonstrate smoothing on fMRI data, we need to determine the sigma of our (3D) gaussian kernel that we'll use to smooth the data. Annoyingly, smoothing kernels in the fMRI literature (and software packages!) are usually not reported in terms of sigma, but as "full-width half maximum" (FWHM), which refers to the width of the gaussian at half the maximum height:

For example, you might read in papers something like "We smoothed our data with a gaussian kernel with a FWHM of 3 millimeter". So, we need to convert our desired FWHM (in millimeters) to sigma. In fact, this is the same conversion that we needed to do for our temporal Gaussian running line high-pass filter, but this time applied to the spatial domain (i.e., FWHM in mm) instead of the temporal domain (i.e., TR in seconds). We can use the same formula to convert FWHM (in mm.) to sigma:

\begin{align}
\sigma_{kernel} = \frac{\mathrm{FWHM}_{mm}}{\sqrt{8 \ln{2}} \cdot \mathrm{voxel\ size}_{mm}}
\end{align}

So, for example, if I want to smooth at FWHM = 6 mm and my voxels are 3 mm in size (assuming equal size in the three spatial dimensions, which is common), my sigma becomes:

\begin{align}
\sigma_{kernel} = \frac{6}{\sqrt{8 \ln{2}} \cdot 3} \approx 0.85
\end{align}

The entire conversion process is a bit annoying, but it's simply necessary due to conventions in the fMRI literature/software packages.

Anyway, having dealt with the conversion issue, let's look at an example. We'll use the 4D fMRI data from before (the `data_4d` variable) and we'll extract a single 3D volume which we'll smooth using the `gaussian_filter` function. This particular data has a voxel-size of $6\ mm^{3}$; given that we want to smooth at an FWHM of, let's say, 10 millimeter, we need a sigma of $\frac{10}{\sqrt{8 \ln{2}} \cdot 6} \approx 0.7$.

```python
f = op.join(op.dirname(op.abspath('')), 'week_1', 'func.nii.gz')
data_4d = nib.load(f).get_fdata()
vol = data_4d[:, :, :, 20] # We'll pick the 21st volume (Python is 0-indexed, remember?)

fwhm = 10
voxelsize = 6

sigma = fwhm / (np.sqrt(8 * np.log(2)) * voxelsize)
smoothed_vol = gaussian_filter(vol, sigma=sigma)

# Let's plot both the unsmoothed and smoothed volume
plt.figure(figsize=(12, 5))
plt.subplot(1, 2, 1)
plt.imshow(vol[:, :, 30], cmap='gray') # And we'll pick the 11th axial slice to visualize
plt.axis('off')
plt.title("Unsmoothed volume\n", fontsize=15)

plt.subplot(1, 2, 2)
plt.imshow(smoothed_vol[:, :, 30], cmap='gray')
plt.axis('off')
plt.title('Smoothed volume\n($\sigma = %.1f; FWHM = %s_{mm}$)' % (sigma, fwhm), fontsize=15)
plt.show()
```

ToDo (2 points): Now, in the above example we only smoothed a single volume, but in your analyses you would of course smooth all volumes in your fMRI run! (Just like with temporal filtering you need to filter the timeseries of all voxels separately.) In this ToDo, you need to loop through all (50) volumes from the <tt>data_4d</tt> variable and smooth them separately. Store the smoothed data in the already pre-allocated variable <tt>data_4d_smoothed</tt>. Use a sigma of 0.7 for the gaussian filter.

```python
# Implement your ToDo here

data_4d_smoothed = np.zeros(data_4d.shape)

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the ToDo above '''
from niedu.tests.nii.week_4 import test_smoothing_todo
test_smoothing_todo(data_4d, data_4d_smoothed)
```

    ToThink (1 point): Since the <tt>gaussian_filter</tt> works for any $N$-dimensional array, one could argue that you don't have to loop through all volumes and apply a 3D filter, but you could equally well skip the loop and use a 4D filter straightaway. Explain (concisely) why this is a bad idea (for fMRI data).

YOUR ANSWER HERE

## Susceptibility distortion correction (SDC)
Most functional MRI sequences use a technique called "gradient-echo echo planar imaging" (GE-EPI), which is a relatively sensitive and well-established technique for BOLD-MRI. For reasons beyond the scope of this course, GE-EPI scans unfortunately show signs of **"geometric distortion"** (sometimes called susceptibility distortion) and **signal loss** in some areas of the brain. Signal loss, which is most apparent in areas with B0 (static magnetic field) inhomogeneities caused by sudden air-tissue transitions (like around the ears, earduct → brain, and the ventromedial cortex, frontal sinus → brain). Geometric distortion is most visible in the phase-encoding direction of the scan (usually posterior → anterior or left → right). These geometric distorions are usually visible as tissue "folding" inwards or outwards (depending on the exact phase-encoding direction, e.g., posterior → anterior or anterior → posterior).

In the gif below, you can see where signal loss (in ventromedial cortex and around the ears, best visible in slice `z=12`) and geometric distortion is most apparent. The "before" image is the one *with* geometric distortion (most apparent as folding inwards of the cerebellum and frontal cortex). Hover your cursor above the image to see it changing from uncorrected ("before") to corrected ("after").

While there is (currently) no way to correct for signal loss, there *are* ways to correct for geometric distortion (as you can see in the gif above). Most distortion correction techniques require you to acquire additional scans that can be used to correct (or "unwarp") your functional MRI scans. One technique is to acquire a "fieldmap" scan to image the homogeneity of the magnetic field (based on two scans with slightly different TEs) which can then be used for geometric correction. Another technique, often called "topup unwarping" (or PEPOLAR unwarping), uses one or more volumes of a separate functional MRI scan with the opposite phase-encoding direction as your to-be-correction functional MRI scan. In the "topup" scan, the distortion is expected to be in the complete opposite direction as in your to-be-correction fMRI scan (e.g., inward folding of tissue where the fMRI scan would show outward folding and vice versa), which can then be used to estimate an undistorted shape (which is "in the middle" of the fMRI and topup scan).

The mathematics and implementation of distortion correction is beyond the scope of this course, but we just wanted it to show you so that you're familiar with the term!

## Motion correction / realignment
Participant movement during scanning arguably affects your data quality most, and should be taken care of accordingly. As such, *motion correction* (or sometimes called *motion realignment*) is an important step in any preprocessing pipeline. Motion correction makes sure that all volumes (i.e., 3D fMRI images) are spatially aligned. Before discussing motion correction in more detail, let's take a look at how a motion corrupted scan looks like. Actually, the data that we've been using so far (`data_4d`) actually has very little motion, so we're going to add some motion to it (with a custom function `add_motion_to_vols`). Then, we'll plot it volume by volume as a short movie:

```python
from niedu.utils.nii import add_motion_to_vols, animate_volumes

vols = add_motion_to_vols(data_4d)
animate_volumes(vols)
```

### Rigid body registration
Motion correction aligns all volumes by picking a reference volume (usually the first or middle one) and subsequently "moving" all the other volumes such that they are spatially aligned with the reference volume. Specifically, motion correction uses *translation* (movement along the primary axes: left/right, up/down, front/back) and *rotation* (around the center of the image).

*Image from [Arcoverde Neto et al. (2014)](https://www.researchgate.net/figure/Orientation-of-the-head-in-terms-of-pitch-roll-and-yaw-movements-describing-the-three_fig1_279291928).*

This specific transformation of the image has six parameters (translation in the X, Y, and Z direction, and rotation in the X, Y, and Z direction) and is often called a "rigid body registration". Given these parameters, the to-be-moved volume can be spatially resampled at the location of the reference volume.

A rigid body registration is an example of a more general *affine registration*: a linear transformation of an image that may involve translation, rotation, scaling, and shearing. For motion correction, scaling and shearing is *not* applied.

    ToThink (0 points): Why do you think motion correction not use scaling and shearing transformations?

### Affine registrations
For affine registrations, the way the image should be moved is defined in the affine matrix, a $4 \times 4$ matrix. This matrix is similar to the one we discussed in week 1, but instead of describing how each image coordinate ($i, j, k$) relates to world coordinates ($x, y, z$), the affine matrix, here, describes how the coordinates from the original (non-corrected) image related to the corrected image:

\begin{align}
\begin{bmatrix} i_{corr}, j_{corr}, k_{corr}, 1 \end{bmatrix} = A\begin{bmatrix}
           i \\
           j \\
           k \\
           1
         \end{bmatrix}
\end{align}

For example, suppose I have a particular affine matrix (`A_example`, below) that encodes a translation of 3 voxels downwards (superior → inferior). Then, for any coordinate $(x, y, z, 1)$, I can compute the location of the coordinate in the corrected image as follows:

```python
A_example = np.array([
    [1, 0, 0, 0],
    [0, 1, 0, 0],
    [0, 0, 1, 3],
    [0, 0, 0, 1]
])

# Note that we always have to append a 1, just like in week 1
coord = np.array([0, 0, 0, 1])  # the middle of the image!
corr_coord = A_example @ coord
print(corr_coord[:3])
```

As expected, the affine transformation above shows us that the middle coordinate $(0, 0, 0)$ refers to the corrected coordinate $(0, 0, 3)$ after translating the image 3 voxels downwards. But how is this particular translation encoded in the affine? For translation, this is relatively straightforward. For any translation $(x, y, z)$, the affine matrix looks like:

\begin{align}
\begin{bmatrix} i_{\mathrm{trans}} & j_{\mathrm{trans}} & k_{\mathrm{trans}} & 1 \end{bmatrix} =
\begin{bmatrix}
           1 & 0 & 0 & x \\
           0 & 1 & 0 & y \\
           0 & 0 & 1 & z \\
           0 & 0 & 0 & 1
\end{bmatrix}
\begin{bmatrix}
           i \\
           j \\
           k \\
           1
         \end{bmatrix}
\end{align}

For rotation, however, it is slightly more complicated! For rotation transformations (with rotations $x, y, z$ in radians), the affine is computed by the matrix

\begin{align}
\begin{bmatrix} i_{\mathrm{rot}} & j_{\mathrm{rot}} & k_{\mathrm{rot}} & 1 \end{bmatrix} =
    \begin{bmatrix}
           1 & 0 & 0 & 0 \\
           0 & \cos(x) & -\sin(x) & 0 \\
           0 & \sin(x) & \cos(x) & 0 \\
           0 & 0 & 0 & 1
    \end{bmatrix}
    \begin{bmatrix}
           \cos(y) & 0 & \sin(y) & 0 \\
           0 & 1 & 0 & 0 \\
           -\sin(y) & 0 & \cos(y) & 0 \\
           0 & 0 & 0 & 1
    \end{bmatrix}
    \begin{bmatrix}
           \cos(z) & -\sin(z) & 0 & 0 \\
           \sin(z) & \cos(z) & 0 & 0 \\
           0 & 0 & 1 & 0 \\
           0 & 0 & 0 & 1
    \end{bmatrix}
    \begin{bmatrix}
    i \\
    j \\
    k \\
    1
    \end{bmatrix}
\end{align}

As you can see, rotations are actually encoded in the affine matrix as the result of matrix multiplication of the the separate x, y, and z rotation matrices. Don't worry, you don't have to program this rotation-to-affine operation. Using a custom function (`get_rotation_matrix`), let's take a look at what the affine matrix would look like for a rotation of 45 degrees in every direction:

```python
def get_rotation_matrix(x=0, y=0, z=0):
    """ Computes the rotation matrix.

    Parameters
    ----------
    x : float
        Rotation in the x (first) dimension in degrees
    y : float
        Rotation in the y (second) dimension in degrees
    z : float
        Rotation in the z (third) dimension in degrees

    Returns
    -------
    rot_mat : numpy ndarray
        Numpy array of shape 4 x 4
    """

    x = np.deg2rad(x)
    y = np.deg2rad(y)
    z = np.deg2rad(z)

    rot_roll = np.array([
        [1, 0, 0, 0],
        [0, np.cos(x), -np.sin(x), 0],
        [0, np.sin(x), np.cos(x), 0],
        [0, 0, 0, 1]
    ])

    rot_pitch = np.array([
        [np.cos(y), 0, np.sin(y), 0],
        [0, 1, 0, 0],
        [-np.sin(y), 0, np.cos(y), 0],
        [0, 0, 0, 1]
    ])

    rot_yaw = np.array([
        [np.cos(z), -np.sin(z), 0, 0],
        [np.sin(z), np.cos(z), 0, 0],
        [0, 0, 1, 0],
        [0, 0, 0, 1]
    ])

    rot_mat = rot_roll @ rot_pitch @ rot_yaw
    return rot_mat

A_rot = get_rotation_matrix(x=45, y=45, z=45)
print(A_rot)
```

Now, if we want to combine both rotations and translations, we multiply the translation matrix ($T$) with our rotation matrix ($R = R_{x}R_{y}R_{z}$):

\begin{align}
A = TR
\end{align}

    ToDo (1 point): Below, we also define a function to get the translation matrix. Now, suppose I want to rotate my image with 45 degrees in the z-direction and translate my image with 10 voxels in the x-direction and 5 voxels in the y-direction. Calculate the corresponding affine matrix, and store this in a variable named <tt>affine_todo</tt>.

```python
def get_translation_matrix(x=0, y=0, z=0):
    """ Computes the translation matrix.

    Parameters
    ----------
    x : float
        Translation in the x (first) dimension in voxels
    y : float
        Rotation in the y (second) dimension in voxels
    z : float
        Rotation in the z (third) dimension in voxels

    Returns
    -------
    trans_mat : numpy ndarray
        Numpy array of shape 4 x 4
    """

    trans_mat = np.eye(4)
    trans_mat[:, -1] = [x, y, z, 1]
    return trans_mat

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nii.week_4 import test_affine_todo
test_affine_todo(affine_todo)
```

Now, we're not going to discuss the intricacies of how to actually resample an image given a particular affine. Instead, we defined a function called `resample_image` that resamples an image given a particular translation and rotation matrix. It's a bit more complicated than we discussed so far, but you get the idea.

```python
from scipy.ndimage import affine_transform

def resample_image(image, trans_mat, rot_mat):
    """ Resamples an image given a translation and rotation matrix.

    Parameters
    ----------
    image : numpy array
        A 3D numpy array with image data
    trans_mat : numpy array
        A numpy array of shape 4 x 4
    rot_mat : numpy array
        A numpy array of shape 4 x 4

    Returns
    -------
    image_reg : numpy array
        A transformed 3D numpy array
    """

    # We need to rotate around the origin, not (0, 0), so
    # add a "center" translation
    center = np.eye(4)
    center[:3, -1] = np.array(image.shape) // 2 - 0.5
    A = center @ trans_mat @ rot_mat @ np.linalg.inv(center)

    # affine_transform does "pull" resampling by default, so
    # we need the inverse of A
    image_corr = affine_transform(image, matrix=np.linalg.inv(A))

    return image_corr
```

    ToDo (0 points): Try out different transformations of an actual brain image by changing the arguments given to <tt>get_translation_matrix</tt> and <tt>get_rotation_matrix</tt>. You can also change the axis (<tt>PLOT_AXIS</tt>) and slice number (<tt>SLICE_IDX</tt>) that is plotted.

```python
trans_mat = get_translation_matrix(5, 5, 0)
rot_mat = get_rotation_matrix(0, 0, 40)

vols = data_4d
VOL = vols[:, :, :, 0]
PLOT_AXIS = 2
SLICE_IDX = 20

plt.figure(figsize=(10, 5))
plt.subplot(1, 2, 1)
plt.imshow(np.take(VOL, SLICE_IDX, PLOT_AXIS).T, origin='lower', cmap='gray')
plt.axis('off')
plt.title('Original image', fontsize=20)

# Resample
vol_res = resample_image(VOL, trans_mat, rot_mat)

plt.subplot(1, 2, 2)
plt.imshow(np.take(vol_res, SLICE_IDX, PLOT_AXIS).T, origin='lower', cmap='gray')
plt.axis('off')
plt.title('Transformed image', fontsize=20)

plt.tight_layout()
plt.show()
```

Now, suppose that we want to align two volumes with each other. How would we go about it? Visually, it's quite hard to assess whether a particular affine matrix works well. In practice, most neuroimaging software packages use automatic algorithms that try to find a affine matrix that *minimizes some cost function*. In other words, it defines a function to evaluate the mismatch (or cost) of the alignment of two volumes given an affine matrix, and then it tries to minimize this function by (in a smart way) trying out different rotation and translation parameters.

One cost function that is often used for rigid body registration is "least squares", which is defined as the sum over the squared (voxel-wise, $v$) differences between the reference image (sometimes called the "fixed" image) and the realigned image (sometimes called the "moving" image):

\begin{align}
\mathrm{cost}_{LS} = \sum_{v}(\mathrm{fixed}_{v} - \mathrm{moving}_{v})^2
\end{align}

    ToDo (1 point): Define a function named <tt>least_squares_cost</tt> below, that takes two input parameters — <tt>fixed</tt> and <tt>moving</tt> (both 3D numpy arrays) — and outputs the least-squares cost. Note: you don't need a for loop!

```python
# YOUR CODE HERE
raise NotImplementedError()
```

```python
""" Tests the above ToDo """
fixed = np.ones((10, 10, 10))
moving = np.zeros((10, 10, 10))

np.testing.assert_almost_equal(least_squares_cost(fixed, moving), 1000)

fixed = np.arange(1000).reshape((10, 10, 10))
moving = fixed[::-1]
np.testing.assert_almost_equal(least_squares_cost(fixed, moving), 330000000)

moving = np.roll(fixed.ravel(), 1).reshape((10, 10, 10))
np.testing.assert_almost_equal(least_squares_cost(fixed, moving), 999000)

print("Well done!")
```

    ToDo (1 point): For this ToDo, you are going to be the optimizer! Below, we have loaded two (brain) images: <tt>fixed</tt> and <tt>moving</tt>. Use the <tt>resample_image</tt> function to translate/rotate the <tt>moving</tt> image. Make sure you name the output (i.e., the transformed volume) <tt>moved</tt>. This is then plotted on top of the reference volume to give you an idea of how well your transformation worked. Keep doing this until you're satisfied.

Hint 1: you need rotation in the z dimension and translation in the x and y dimension.

Hint 2: rotation is counter clockwise (if you define negative rotation, e.g. -20, it will be rotated clockwise).

```python
with np.load('moco_todo.npz') as moco_data:
    moving = moco_data['moving']
    fixed = moco_data['fixed']

# YOUR CODE HERE
raise NotImplementedError()

### Plotting starts here
# Only works if you've defined a variable named 'moved'
# Do not change the code below
plt.figure(figsize=(10, 10))
plt.imshow(fixed[:, :, 25].T, origin='lower', cmap='gray')
moved4plot = np.ma.masked_where(moved < 2, moved)
plt.imshow(moved4plot[:, :, 25].T, origin='lower', alpha=0.8)
plt.axis('off')
plt.show()
```

## Anatomical realignment/registration and normalization
So far, we discussed how to register volumes to a reference volume using rigid body registrations (with a six parameter affine) in the context of motion correction. This is, however, only the first step in the complete registration protocol of (most) studies. If your goal is, in the end, to perform some kind of group analysis (in which you pool together data from multiple subjects), you want to make sure that each voxel (at location $(X, Y, Z)$) is in (roughly) the same location in the brain. The problem is that people's brains differ quite a bit in terms of anatomy! Therefore, you need to register (transform) your subject-level (functional) data to some kind of common template. This process is often done in two stages: first, you compute a registration from (a reference volume of) your functional data ("moving") to your anatomical data (your high-resolution T1-weighted scan; "fixed"), and second you compute a registration from your anatomical data ("moving") to a common template ("fixed"). Then, you can register your functional data to the common template by sequentially applying the first and second stage registration parameters. This is visualized in the figure below.

In the figure above, you can see that the first stage (BOLD → T1w) uses the same registration protocol as we used with motion-correction: an affine transformation with 6 parameters (translation and rotation in 3 directions; a "rigid body registration"). The second stage (T1w → template) also uses an affine transformation, but this time with 12 parameters. In addition to the 6 rigid body registration parameters, a 12 parameter affine (sometimes reffered to as an affine with 12 degrees-of-freedom, DOF) also includes global scaling in 3 directions (i.e., making the brain bigger of smaller) and shearing in 3 directions (i.e., "stretching" in 3 directions). These extra parameters attempt to overcome the anatomical differences between the individual subjects' brains and the template.

However, only a 12 DOF affine is usually not sufficient for proper alignment. Therfore, most neuroimaging software packages also offer the option to apply a non-linear registration step ("warping") *on top of the 12 DOF affine registration*. Instead of globally transforming your "moving" volume (i.e., the transformation applies to each voxel equally), non-linear registrations allow for *local* transformations (i.e., some voxels might be transformed "more" than others). Usually, the combination of an initial 12 DOF affine and a non-linear ("warping") registration operation lead to a proper alignment of the subject's anatomical volume and the template.

    ToThink (2 points): When we discussed motion correction, we used the "least squares" cost function to optimize our rigid body registration parameters. Is it a good idea to use this cost function for the BOLD → T1w registration procedure? And for our initial T1w → template registration procedure? Argue for both cases why (not).

YOUR ANSWER HERE

## Motion filtering
Even after motion realignment, your data is still 'contaminated' by motion. This is because movement itself influences the measured activity. For example, suppose that you measure a single voxel in someone's brain; then, this person moves his/her head 2 centimeters. Now, we can do motion realigment to make sure we measure the same voxel before and after the movement, but *this does not change the fact that this particular voxel was originally measured at two different locations*. It could be that after the movement, the voxel was actually a little bit closer to the headcoil, which results in a (slight) increase in signal compared to before the movement (this is also known as "spin history effects").

Ideally, you want to account for these interactions between motion and the measured activity. One way to do this is through "motion filtering", of which one popular approach is to simply add the 6 realignment parameters (rotation and translation in 3 directions) to the design-matrix ($X$)! In other words, we treat the motion realignment parameters as "nuisance regressors" that are aimed to explain activity that is related to motion. Realize that this is, thus, as temporal preprocessing step (which we discuss here instead of the previous notebook because we needed to explain motion correction first).

Alright, let's load some realigment parameters (6 in total) from an fMRI run of 200 volumes. We'll plot them below:

```python
""" This data has been motion-corrected using the FSL tool 'MCFLIRT', which outputs a file
ending in *.par that contains the 6 motion parameters (rotation/translation in 3 directions each).
We'll load in this file and plot these motion parameters. """

motion_params = np.loadtxt('func_motion_pars_new.txt')
rotation_params = motion_params[:, :3]
translation_params = motion_params[:, 3:]

plt.figure(figsize=(15, 7))
plt.subplot(2, 1, 1)
plt.title('Rotation', fontsize=25)
plt.plot(rotation_params)
plt.xlim(0, motion_params.shape[0])
plt.legend(['x', 'y', 'z'], fontsize=15)
plt.ylabel('Rotation in radians', fontsize=15)
plt.grid()

plt.subplot(2, 1, 2)
plt.title('Translation', fontsize=25)
plt.plot(translation_params)
plt.legend(['x', 'y', 'z'], fontsize=15)
plt.ylabel('Translation in mm', fontsize=15)
plt.xlim(0, motion_params.shape[0])
plt.xlabel('Time (TR)', fontsize=15)
plt.grid()

plt.tight_layout()
plt.show()
```

So, is this good data or not? In general, you could say that higher values (in either translation or rotation) are worse. That said, from the translation and rotation parameters alone it is hard to judge whether the participant moved *too* much, partly because we're dealing with 6 parameter traces here. One way to make this decision process a little easier is to convert these parameters to a single "framewise displacement" (FD) trace. FD aims to represent "bulk motion" and is a summary statistic of motion that combines all motion parameters into a single metric. It is calculated as follows:

\begin{align}
FD_{t} = |r(\mathrm{rot}_{tx} - \mathrm{rot}_{(t-1)x})| + |r(\mathrm{rot}_{ty} - \mathrm{rot}_{(t-1)y})| + |r(\mathrm{rot}_{tz} - \mathrm{rot}_{(t-1)z})| + |\mathrm{trans}_{xt} - \mathrm{trans}_{x(t-1)}| + |\mathrm{trans}_{yt} - \mathrm{trans}_{y(t-1)}| + |\mathrm{trans}_{zt} - \mathrm{trans}_{z(t-1)}|
\end{align}

where $|x|$ means the absolute value of $x$ and $r$ stands for a predefined radius in millimeters (which is usually taken to be 50). So, basically it is a sum across the absolute successive differences ($x_{t} - x_{t-1}$) of parameter traces.

ToDo (0 points; optional): Calculate the FD trace corresponding to the motion parameters in the <tt>motion_params</tt> variable. Assume a radius of 50. Store the result in a variable named <tt>fd_trace</tt>. Note that, just like with the RMSSD, the first value is undefined (because at $t=0$, there is no $t-1$). Therefore, prepend a 0 to your FD trace.

```python
# Implement the optional ToDo here
# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nii.week_4 import test_fd_calc
test_fd_calc(motion_params, fd_trace)
```

ToThink (0 points): Looking at the plots above (before the optional ToDo), can you deduce which volume was used as a reference volume? You can also use the <tt>motion_params</tt> variable to find out.

As said in the beginning of this section, motion parameters (translation/rotation in 3 dimensions) are used as nuisance regressors in subject-level analyses. The reason for this is that, even after motion correction, the actual movement may have influenced the signal intensity through "spin-history effects", which occurs when a voxel is excited by an RF pulse slightly earlier/later due to movement (i.e., it may have moved up or down one slice), which changes how much T1 relaxation occurs (which affect the signal intensity; read more [here](http://imaging.mrc-cbu.cam.ac.uk/imaging/CommonArtefacts)).

There are two ways to go about this, which are not mutually exclusive. The most common way (in task-based fMRI) is to add the six motion parameters (rotation/translation in three directions) to your design. Any variance of your signal that covaries with (one or more of) these motion parameters will be accounted for (and will thus *not* end up in your noise term).

ToDo (0 points; optional): Sometimes, in addition to the original six rigid body registration parameters, people add the lag-1 shiften parameters (i.e., $t-1$) to their design, as well as the squares (i.e., $x^{2}$) of those twelve regressors, yielding a set of 24 nuisance regressors (sometimes called the "Friston24" set). Calculate this extended set of motion parameters and store it in a variable named <tt>friston24</tt>. Hint: because (again) the timepoint $t-1$ is not defined at $t=0$, prepend a row of zeros to the lag-1 shifted parameters.

```python
# Implement your ToDo here
# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nii.week_4 import test_friston24
test_friston24(motion_params, friston24)
```

Apart from directly adding the (extended) motion parameters to your design, you can also perform "motion scrubbing". Motion scrubbing is usually performed in resting-state analyses and is conceptually very similar to the "despiking" we did earlier. With motion scrubbing, you usually set a predefined threshold for your FD trace (for example 1 mm) and any timepoint above this threshold will be "scrubbed out". This is done by defining a separate regressor for each timepoint above the threshold containing all zeros except for the above-threshold timepoint, which contains a zero (just like we did with despiking). Sometimes, even the timepoint before and/or after the "bad" timepoint are additionally added to the design matrix. As this course focuses mostly on task-based fMRI analyses, we won't discuss motion scrubbing any further (but if you want to know more about the different motion correction strategies, we recommend the following article: [Power et al., 2015](https://www.sciencedirect.com/science/article/pii/S1053811914008702)).

ToDo (2 points)

For this ToDo, you have to compare two models and the resulting (normalized) effects (t-values): a model *without* the six motion parameters and a model *with* the six motion parameters. You should use the <tt>motion_params</tt> variable which we defined earlier as your motion parameters.

We provide you with a design-matrix (<tt>X</tt>) with an intercept and a single stimulus-predictor and the signal of a single voxel (<tt>sig</tt>). Calculate the t-value for the contrast of the predictor-of-interest against baseline for both the original design-matrix (only intercept + predictor-of-interest; store this in a variable named <tt>tval_simple</tt>)
and the design-matrix extended with the six motion parameters (which thus has 8 predictors; store this in a variable named <tt>tval_ext</tt>).

```python
# First, we'll load the data
with np.load('data_motion_filtering.npz') as data_last_todo:
    X = data_last_todo['X']
    sig = data_last_todo['sig']

print("Shape of original X: %s" % (X.shape,))
print("Shape of signal: %s" % (sig.shape,))
```

```python
""" Implement the ToDo here. """

# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nii.week_4 import test_tvals_motion
test_tvals_motion(X, sig, motion_params, tval_simple, tval_ext)
```

ToThink (2 points): If you did the above ToDo correctly, you should have found that the $t$-value of the extended model (with motion parameters) is actually smaller than the simple model (without motion parameters) ... What caused the t-value of this predictor to become smaller when the motion-parameters were included in your design-matrix? Inspect the different elements of the $t$-value formula carefully.

YOUR ANSWER HERE

Alright, that's it for preprocessing! If you want to know more about a preprocessing software package that we particularly like, check out the next (optional) notebook about [Fmriprep](https://fmriprep.readthedocs.io/) (`fmriprep.ipynb`)!

    Tip!
    Before handing in your notebooks, we recommend restarting your kernel (Kernel → Restart & Clear Ouput) and running all your cells again (manually, or by Cell → Run all). By running all your cells one by one (from "top" to "bottom" of the notebook), you may spot potential errors that are caused by accidentally overwriting your variables or running your cells out of order (e.g., defining the variable 'x' in cell 28 which you then use in cell 15).


```python
from IPython.core.display import HTML
HTML("<style>.container { width:95% !important; }</style>")
```

# Lecture 4: unconstrained optimization- gradient-based methods

## Optimization problem to be studied

We are still studying functions of multiple variables in unconstrained optimization problems
$$
\begin{align}
\min \quad &f(x)\\
\text{s.t.}\quad &x\in \mathbb R^n
\end{align}
$$

## Let us define the same function as on the previous lecture for testing

$$
\begin{align}
\min \quad & (x_1-10)^2+(x_2+5)^2+x_1^2\\
\text{s.t.}\quad &x_1,x_2\in\mathbb R
\end{align}
$$

```python
def f_simple(x):
    return (x[0] - 10.0)**2 + (x[1] + 5.0)**2+x[0]**2
```

## Reminder: Model algorithm for unconstrained minimization (an iterative approach)

 Let $x_k$ be the current approximation for a local minimum $x^*$:

* (1) **Check the convergency** If conditions are satisfied, stop. The solution is $x_k$.

* (2) **Identify a search direction**  Find a nonzero vector $d_k \in \mathbb R^n$ as a search direction.

* (3) **Identify a step length** Find a $\alpha_k > 0$ as a step length, for which $f(x_k + \alpha_k d_k) < f(x_k)$ (i.e., leads us to a better approximation)

* (4) **Update the current approximation for the local minimum** Set $x_{k+1} = x_k + \alpha_k d_k, k =  k + 1$ and go to (1).

## Gradient-based optimization

If gradients are available, they should always be used --> improve convergence

Gradient-based methods usually converge to a local minimum which is nearest to the starting point (**Why?**)

If analytical formulas of the gradients are not available, one way is to numerically approximate them by using finite differences ($ℎ>0$)
* forward difference: $\frac{f(x+h)-f(x)}{h}=\nabla f(x)+O(h)$
* central difference: $\frac{f(x+h)-f(x-h)}{2h}=\nabla f(x)+O(h^2)$
* central difference is more accurate but requires twice as many function evaluations (in $𝑥+ℎ$ and $𝑥−ℎ$)

## Automatic differentiation in Python

An alternative way is to use automatic differentiation

https://en.wikipedia.org/wiki/Automatic_differentiation

Import automatic differentiation package for Python

Needs to be installed typing
```
(!) pip install ad
```

You can ask for gradient and hessian using the
ad.gh
 function.

 Let us use automatic differentiation for the function <it>f</it>  that we defined.

```python
import ad
grad_f, hess_f = ad.gh(f_simple)
```

```python
print("At the point (1,2) gradient is ", grad_f([1,2]), " and hessian is ",hess_f([1,2]))
```

    At the point (1,2) gradient is  [-16.0, 14.0]  and hessian is  [[4.0, 0.0], [0.0, 2.0]]

### Let us visualize the gradient

```python
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import numpy as np
from pylab import meshgrid
def visualize_gradient(f,point,x_lim,y_lim):
    grad_point = np.array(ad.gh(f)[0](point)) # computes the gradient
    grad_point = 10.0*grad_point/np.linalg.norm(grad_point) # scale the gradient vector so that it is easier to see
    X,Y,Z = point[0],point[1],f(point) # point that we are interested in and the function value in that point
    U,V,W = grad_point[0],grad_point[1],0 # gradient vector, last component is zero since the gradient has 2 components
    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    x = np.arange(x_lim[0],x_lim[1],0.1) # divide interval for x
    y = np.arange(y_lim[0],y_lim[1],0.1) # divide interval for y
    X2,Y2 = meshgrid(x, y) # make a grid for x and y
    Z2 = [f([x,y]) for (x,y) in zip (X2,Y2)] # evaluation of the function on the grid
    Z2 = np.asarray(Z2)
    surf = ax.plot_surface(X2, Y2, Z2,alpha=0.5) # surface plot for the function values
    ax.quiver(X,Y,Z,U,V,W,color='red',linewidth=2.5) # plots a 2d arrow in a 3d figure
    return plt

```

```python
visualize_gradient(f_simple,[1,-2],[0,10],[-10,0]).show()
```

With the lambda function we can easily visualize gradients of various functions without a user-defined function

```python
import math
visualize_gradient(lambda x:5*x[0]+3*x[1],[1,1],[0,2],[0,2]).show()
```

# Gradient-based Methods

## Steepest descent and Newton's method

### Reminder: Model algorithm for unconstrained minimization (an iterative approach)

Let $x_k$ be the current approximation for a local minimum $x^*$:

* (1) **Check the convergency** If conditions are satisfied, stop. The solution is $x_k$.

* (2) **Identify a search direction**  Find a nonzero vector $d_k \in \mathbb R^n$ as a search direction.

* (3) **Identify a step length** Find a $\alpha_k > 0$ as a step length, for which $f(x_k + \alpha_k d_k) < f(x_k)$ (i.e., leads us to a better approximation)

* (4) **Update the current approximation for the local minimum** Set $x_{k+1} = x_k + \alpha_k d_k, k =  k + 1$ and go to (1).

## Base algorithm for the steepest descent and Newton's algorithms
**Input:** function $f$ to be optimized, starting point $x_0$, step length rule $alpha$, stopping rule $stop$
**Output:** A solution $x^*$ that is close to a locally optimal solution
```
set f_old as a big number and f_new as f(x0)
while a stopping criterion has not been met:
    f_old = f_new
    determine search direction d_k according to the method
    determine the step length alpha
    set x = x + alpha *d_k
    f_new = f(x)
return x
```

The way to determine search direction distinguishes steepest descent algorithm and the Newton algorithm. Different stopping rules and step sizes can be mixed and matched with both algorithms.

When moving along the gradient direction from any point to n-dimensional space, the function values change at the **fastest rate**.

## Steepest Descent algorithm for unconstrained optimization

Also known as *gradient descent* or *gradient method*

## Desent direction:

* If $d_k$ is a descent direction for $f$ at $x_k$: $f(x_{k+1}) < f(x_k) $.

So, if $x_{k+1} = x_k + \alpha_k d_k$:

* $\rightarrow  f(x_k + \alpha_k d_k) < f(x_k)$

Using Taylor series expansion of $f$:

* $\rightarrow  f(x_k) + \alpha_k ( \nabla f(x_k) . d_k) < f(x_k) $

* $\rightarrow \alpha_k ( \nabla f(x_k) . d_k) < 0 $, ($\alpha_k > 0$)

* $\rightarrow \nabla f(x_k) . d_k < 0 $ (**condition for the descent direction**)

## Steepest Descent algorithm for unconstrained optimization

Also known as *gradient descent* or *gradient method*

In the steepest descent algorithm, the search direction $d_k$ is determined by the negative of the gradient $-\nabla f(x_k)$ which (locally) gives the direction of the steepest descent of $f$ at point $x_k$.

* $x_{k+1} = x_k - \alpha_k \nabla f(x_k)$

 Indeed, the objective function decreases fastest in $x_k$ (you can check it out yourself using the condition for the descent direction if you are interested)

 * Very simple example of a univariant function

 An example of a function with two variables:

Another example of a function with two variables:

### Code in Python

Let us use a simple stopping rule, where we stop when the change is not bigger than precision and we have a fixed step size.

```python
import numpy as np
import ad
def steepest_descent(f,start,step,precision):
    f_old = float('Inf')
    x = np.array(start)
    steps = []
    f_new = f(x)
    while abs(f_old-f_new)>precision:
    #while np.linalg.norm(ad.gh(f)[0](x))>precision: # an alternative stopping rule
        f_old = f_new # store value at the current point
        d = -np.array(ad.gh(f)[0](x)) # search direction
        x = x+d*step # take a step
        f_new = f(x) # compute function value at the new point
        steps.append(list(x)) # save step
    return x,f_new,steps
```

### Solve the problem using the Python function

```python
start = [2.0,-10.0]
step_size = 0.1
precision = 0.01
(x_value,f_value,steps) = steepest_descent(f_simple,start,step_size,precision)
print("Optimal solution is ",x_value)
print(len(steps))
print(steps)
```

    Optimal solution is  [ 4.9994922  -5.11258999]
    17
    [[3.2, -9.0], [3.92, -8.2], [4.352, -7.56], [4.6112, -7.048], [4.76672, -6.6384], [4.860032, -6.31072], [4.9160192, -6.048576], [4.94961152, -5.8388608], [4.969766912, -5.67108864], [4.9818601472, -5.5368709119999995], [4.98911608832, -5.429496729599999], [4.993469652992, -5.34359738368], [4.9960817917952, -5.274877906944], [4.9976490750771205, -5.2199023255552], [4.998589445046273, -5.17592186044416], [4.999153667027763, -5.140737488355328], [4.999492200216658, -5.1125899906842625]]

Plot the steps of solving

```python
import matplotlib.pyplot as plt

def plot_2d_steps(steps,start):
    myvec = np.array([start]+steps).transpose()
    #print(myvec)
    plt.plot(myvec[0,],myvec[1,],'ro')
    for label,x,y in zip([str(i) for i in range(len(steps)+1)],myvec[0,],myvec[1,]):
        plt.annotate(label,xy = (x, y))
    return plt
```

```python
plot_2d_steps(steps,start).show()
```

## Other stopping rules
* change in function values: $|f_{new}-f_{old}| \leq $precision

This one we already used. Others?

* norm of the gradient: $||\nabla f(x_{new})|| \leq $precision
* change in variable values: $||x_{new}-x_{old}|| \leq $precision
* maximum number of iterations

```python
# Another example
def f_simple2(x):
    return (x[0] - 2.0)**4 + (x[0] - 2.0*x[1])**2
# Optimal solution is (2,1)
```

```python
start = [0.0,3.0]
step_size = .01
precision = 0.0001
(x_value,f_value,steps) = steepest_descent(f_simple2,start,step_size,precision)
print("Optimal solution is ",x_value)
print(len(steps))
#print(steps)
```

    Optimal solution is  [1.82510975 0.92136469]
    52

```python
plot_2d_steps(steps,start).show()
```

Compare steepest descent with optimized step size:

*From Miettinen: Nonlinear optimization, 2007 (in Finnish)*

## Newton's method

Based on setting the search direction as $d_k=-[H(x_k)]^{-1}\nabla f(x_k).\hspace{2.0cm}$   --> **uses second derivatives!**

In one-dimensional case by the Taylor series:
$$f(x+\Delta x)\approx f(x)+f'(x)\Delta x+\frac12f''(x)\Delta x^2$$

We want to find $x$ such that $f(x)$ is at minimum and, thus, we seek to solve the equation that sets the derivative of this expression with respect to $\Delta x$ equal to zero (Necessary condition):

$$ 0 = \frac{d}{d\Delta x} \left(f(x_k)+f'(x_k)\Delta x+\frac 1 2 f''(x_k) \Delta x^2\right) = f'(x_k)+f'' (x_k) \Delta x.$$

The solution of the above equation is $\Delta x=-f'(x_k)/f''(x_k)$. Thus, the best approximation of $x_{k+1}$ as the minimum is $x_k-f''(x_k)^{-1}f'(x_k)$.

```python
def newton(f,start,step,precision):
    f_old = float('Inf')
    x = np.array(start)
    steps = []
    f_new = f(x)
    while abs(f_old-f_new)>precision:
        f_old = f_new # store value of f at the current point
        H_inv = np.linalg.inv(np.matrix(ad.gh(f)[1](x))) # compute the inverse of the Hessian
        d = (-H_inv*(np.matrix(ad.gh(f)[0](x)).transpose())).transpose() # compute search direction
        #Change the type from np.matrix to np.array so that we can use it in our function
        x = np.array(x+d*step)[0] # take the step
        f_new = f(x) # compute function value at the new point
        steps.append(list(x)) # store step
    return x,f_new,steps
```

```python
start = [2.0,-10.0]
step_size = 0.5
precision = 0.001
(x_value,f_value,steps) = newton(f_simple,start,step_size,precision)
print("Optimal solution is ",x_value)
print(len(steps))
```

    Optimal solution is  [ 4.99414062 -5.00976562]
    9

```python
plot_2d_steps(steps,start).show()
```

## Another example

```python
start = [0.0,3.0]
step_size = 0.3
precision = 0.001
(x_value,f_value,steps) = newton(f_simple2,start,step_size,precision)
print("Optimal solution is ",x_value)
print(len(steps))
```

    Optimal solution is  [1.80304582 0.90269586]
    22

```python
plot_2d_steps(steps,start).show()
```

Compare Newton's method with step size = 1.0:

*From Miettinen: Nonlinear optimization, 2007 (in Finnish)*

# Summary
Steepest descent
* Descent direction (does the search direction guarantee decrease in $f$): yes
* Global convergence (converges from any starting point): yes
* Local convergence: zig-zag near optimum
* Computational bottle neck: step length
* Memory consumption: $𝑂(𝑛)$

Newton’s method
* Descent direction: only if $𝐻(𝑥)^{−1}$ positive definite
* Global convergence: no
* Local convergence: yes, good if 𝑓 quadratic
* Computational bottle neck: $𝐻(𝑥 )^{−1}$ --> can be relaxed by using approximations of $H(x)^{-1}$ --> quasi-Newton
* Memory consumption: $𝑂(𝑛^2)$ --> can be reduced e.g. by using conjugate gradient methods

Lecture 4, Steepest descent and Newton's method for unrestricted optimization.ipynb


# Removing `if` Statements from Expressions

## Author: Patrick Nelson

### NRPy+ Source Code for this module:
* [Min_Max_and_Piecewise_Expressions.py](../edit/Min_Max_and_Piecewise_Expressions.py) Contains functions that can be used to compute the minimum or maximum of two values and to implement piecewise-defined expressions

## Introduction:

Conditional statements are a critical tool in programming, allowing us to control the flow through a program to avoid pitfalls, code piecewise-defined functions, and so forth. However, there are times when it is useful to work around them. It takes a processor time to evaluate the whether or not to execute the code block, so for some expressions, performance can be improved by rewriting the expression to use an absolute value function in a manner upon which we will expand in this tutorial. Even more relevant to NRPy+ are piecewise-defined functions. These inherently involve `if` statements, but NRPy+'s automatic code generation cannot handle these by itself, requiring hand-coding to be done. However, if it is possible to rewrite the expression in terms of absolute values, then NRPy+ can handle the entire thing itself.

The absolute value is a function that simply returns the magnitude of its argument, a positive value. That is,
\begin{align}
|x|&= \left \{ \begin{array}{lll}x & \mbox{if} & x \geq 0 \\
-x & \mbox{if} & x \leq 0 \end{array} \right. \\
\end{align}

In C, this is implemented as `fabs()`, which merely has to make the first bit of a double-precision floating point number 0, and is thus quite fast.

There are myriad uses for these tricks in practice. One example comes from GRMHD (and, by extension, the special cases of GRFFE and GRHD), in which it is necessary to limit the velocity of the plasma in order to keep the simulations stable. This is done by calculating the Lorentz factor $\Gamma$ of the plasma and comparing to some predefined maximum $\Gamma_\max$. Then, if
$$
R = 1-\frac{1}{\Gamma^2} > 1-\frac{1}{\Gamma_{\max}^2} = R_\max,
$$
we rescale the velocities by $\sqrt{R_\max/R}$. In NRPy+, we instead always rescale by
$$
\sqrt{\frac{\min(R,R_\max)}{R+\epsilon}},
$$
which has the same effect while allowing the entire process to be handled by NRPy+'s automatic code generation. ($\epsilon$ is some small number chosen to avoid division by zero without affecting the results otherwise.) See [here](Tutorial-GRHD_Equations-Cartesian.ipynb#convertvtou) for more information on this.

# Table of Contents
$$\label{toc}$$

This notebook is organized as follows

1. [Step 1](#min_max): Minimum and Maximum
    1. [Step 1.a](#confirm): Confirm that these work for real numbers
1. [Step 2](#piecewise): Piecewise-defined functions
1. [Step 3](#sympy): Rewrite functions to work with symbolic expressions
1. [Step 4](#validation): Validation against `Min_Max_and_Piecewise_Expressions` NRPy+ module
1. [Step 5](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Step 1: Minimum and Maximum \[Back to [top](#toc)\]
$$\label{min_max}$$

Our first job will be to rewrite minimum and maximum functions without if statements. For example, the typical implementation of `min(a,b)` will be something like this:
```python
def min(a,b):
    if a<b:
        return a
    else:
        return b
```
However, to take full advantage of NRPy+'s automated function generation capabilities, we want to write this without the `if` statements, replacing them with calls to `fabs()`. We will define these functions in the following way:
$$\boxed{
\min(a,b) = \tfrac{1}{2} \left( a+b - \lvert a-b \rvert \right)\\
\max(a,b) = \tfrac{1}{2} \left( a+b + \lvert a-b \rvert \right).}
$$

## Step 1.a: Confirm that these work for real numbers \[Back to [top](#toc)\]
$$\label{confirm}$$

For real numbers, these operate exactly as expected. In the case $a>b$,
\begin{align}
\min(a,b) &= \tfrac{1}{2} \left( a+b - (a-b) \right) = b \\
\max(a,b) &= \tfrac{1}{2} \left( a+b + (a-b) \right) = a, \\
\end{align}
and in the case $a<b$, the reverse holds:
\begin{align}
\min(a,b) &= \tfrac{1}{2} \left( a+b - (b-a) \right) = a \\
\max(a,b) &= \tfrac{1}{2} \left( a+b + (b-a) \right) = b, \\
\end{align}

In code, we will represent this as:
```
min_noif(a,b) = sp.Rational(1,2)*(a+b-nrpyAbs(a-b))
max_noif(a,b) = sp.Rational(1,2)*(a+b+nrpyAbs(a-b))
```
For demonstration purposes, we will use `np.absolute()` and floating point numbers.

```python
import numpy as np
import matplotlib.pyplot as plt

thismodule = "Min_Max_and_Piecewise_Expressions"
# First, we'll write the functions. Note that we are not using sympy right now. For NRPy+ code generation,
# use the expressions above.
def min_noif(a,b):
    return 0.5 * (a+b-np.absolute(a-b))

def max_noif(a,b):
    return 0.5 * (a+b+np.absolute(a-b))

# Now, let's put these through their paces.
a_number = 5.0
another_number = 10.0
print("The minimum of "+str(a_number)+" and "+str(another_number)+" is "+str(min_noif(a_number,another_number)))
```

    The minimum of 5.0 and 10.0 is 5.0

Feel free to test other cases above if you'd like. Note that we use a suffix, `_noif`, to avoid conflicts with other functions. When using this in NRPy+, make sure you use `sp.Rational()` and the `nrpyAbs()` function, which will always be interpreted as the C function `fabs()` (Sympy's `sp.Abs()` may get interpreted as $\sqrt{zz^*}$, for instance).

# Step 2: Piecewise-defined functions \[Back to [top](#toc)\]
$$\label{piecewise}$$

Next, we'll define functions to represent branches of a piecewise-defined function. For example, consider the function
\begin{align}
f(x) &= \left \{ \begin{array}{lll} \frac{1}{10}x^2+1 & \mbox{if} & x \leq 0 \\
\exp(\frac{x}{5}) & \mbox{if} & x > 0 \end{array} \right. , \\
\end{align}
which is continuous, but not differentiable at $x=0$.

To solve this problem, let's add the two parts together, multiplying each part by a function that is either one or zero depending on $x$. To define $x \leq 0$, this can be done by multiplying by the minimum of $x$ and $0$. We also will need to normalize this. To avoid putting a zero in the denominator, however, we will add some small $\epsilon$ to the denominator, i.e.,
$$
\frac{\min(x,0)}{x-\epsilon}
$$
This $\epsilon$ corresponds `TINYDOUBLE` in NRPy+; so, we will define the variable here with its default value, `1e-100`. Additionally, to get the correct behavior on the boundary, we shift the boundary by $\epsilon$, giving us
$$
\frac{\min(x-\epsilon,0)}{x-\epsilon}
$$

The corresponding expression for $x > 0$ can be written as
$$
\frac{\max(x,0)}{x+\epsilon},
$$
using a positive small number to once again avoid division by zero.
When using these for numerical relativity codes, it is important to consider the relationship between $\epsilon$, or `TINYDOUBLE`, and the gridpoints in the simulation. As long as $\epsilon$ is positive and large enough to avoid catastrophic cancellation, these functional forms avoid division by zero, as proven [below](#proof).

So, we'll code NumPy versions of these expressions below. Naturally, there are many circumstances in which one will want the boundary between two pieces of a function to be something other than 0; if we let that boundary be $x^*$, this can easily be done by passing $x-x^*$ to the maximum/minimum functions. For the sake of code readability, we will write the functions to pass $x$ and $x^*$ as separate arguments. Additionally, we code separate functions for $\leq$ and $<$, and likewise for $\geq$ and $>$. The "or equal to" versions add a small offset to the boundary to give the proper behavior on the desired boundary.

```python
TINYDOUBLE = 1.0e-100
def coord_leq_bound(x,xstar):
    # Returns 1.0 if x <= xstar, 0.0 otherwise.
    # Requires appropriately defined TINYDOUBLE
    return min_noif(x-xstar-TINYDOUBLE,0.0)/(x-xstar-TINYDOUBLE)

def coord_geq_bound(x,xstar):
    # Returns 1.0 if x >= xstar, 0.0 otherwise.
    # Requires appropriately defined TINYDOUBLE
    return max_noif(x-xstar+TINYDOUBLE,0.0)/(x-xstar+TINYDOUBLE)

def coord_less_bound(x,xstar):
    # Returns 1.0 if x < xstar, 0.0 otherwise.
    # Requires appropriately defined TINYDOUBLE
    return min_noif(x-xstar,0.0)/(x-xstar-TINYDOUBLE)

def coord_greater_bound(x,xstar):
    # Returns 1.0 if x > xstar, 0.0 otherwise.
    # Requires appropriately defined TINYDOUBLE
    return max_noif(x-xstar,0.0)/(x-xstar+TINYDOUBLE)

# Now, define our the equation and plot it.
x_data = np.arange(start = -10.0, stop = 11.0, step = 1.0)
y_data = coord_less_bound(x_data,0.0)*(0.1*x_data**2.0+1.0)\
        +coord_geq_bound(x_data,0.0)*np.exp(x_data/5.0)

plt.figure()
a = plt.plot(x_data,y_data,'k',label="Piecewise function")
b = plt.plot(x_data,0.1*x_data**2.0+1.0,'b.',label="y=0.1*x^2+1")
c = plt.plot(x_data,np.exp(x_data/5.0),'g.',label="y=exp(x/5)")
plt.legend()
plt.xlabel("x")
plt.ylabel("y")
plt.show()

```

The plot above shows the expected piecewise-defined function. It is important in applying these functions that each greater-than be paired with a less-than-or-equal-to, or vice versa. Otherwise, the way these are written, a point on the boundary will be set to zero or twice the expected value.

These functions can be easily combined for more complicated piecewise-defined functions; if a piece of a function is defined as $f(x)$ on $x^*_- \leq x < x^*_+$, for instance, simply multiply by both functions, e.g.
```
coord_geq_bound(x,x_star_minus)*coord_less_bound(x,x_star_plus)*f(x)
```

# Step 3: Rewrite functions to work with symbolic expressions \[Back to [top](#toc)\]
$$\label{sympy}$$

In order to use this with sympy expressions in NRPy+, we will need to rewrite the `min` and `max` functions with slightly different syntax. Critically, we will change `0.5` to `sp.Rational(1,2)` and calls to `np.absolute()` to `nrpyAbs()`. We will also need to import `outputC.py` here for access to `nrpyAbs()`. The other functions will not require redefinition, because they only call specific combinations of the `min` and `max` function.

In practice, we want to use `nrpyAbs()` and *not* `sp.Abs()` with our symbolic expressions, which will force `outputC` to use the C function `fabs()`, and not try to multiply the argument by its complex conjugate and then take the square root.

```python
from outputC import *            # NRPy+: Core C code output module

def min_noif(a,b):
    # Returns the minimum of a and b
    return sp.Rational(1,2) * (a+b-nrpyAbs(a-b))

def max_noif(a,b):
    # Returns the maximum of a and b
    return sp.Rational(1,2) * (a+b+nrpyAbs(a-b))
```

# Step 4: Validation against `Min_Max_and_Piecewise_Expressions` NRPy+ module \[Back to [top](#toc)\]
$$\label{validation}$$

As a code validation check, we will verify agreement in the SymPy expressions for plane-wave initial data for the Scalar Wave equation between
1. this tutorial and
2. the NRPy+ [Min_Max_and_Piecewise_Expressions](../edit/Min_Max_and_Piecewise_Expressions.py) module.

```python
# Reset & redefine TINYDOUBLE for proper comparison
%reset_selective -f TINYDOUBLE
TINYDOUBLE = par.Cparameters("REAL", thismodule, "TINYDOUBLE", 1e-100)

import Min_Max_and_Piecewise_Expressions as noif
all_passed=0

def comp_func(expr1,expr2,basename,prefixname2="noif."):
    passed = 0
    if str(expr1-expr2)!="0":
        print(basename+" - "+prefixname2+basename+" = "+ str(expr1-expr2))
        passed = 1
    return passed

a,b = sp.symbols("a b")

here = min_noif(a,b)
there = noif.min_noif(a,b)
all_passed += comp_func(here,there,"min_noif")

here = max_noif(a,b)
there = noif.max_noif(a,b)
all_passed += comp_func(here,there,"max_noif")

here = coord_leq_bound(a,b)
there = noif.coord_leq_bound(a,b)
all_passed += comp_func(here,there,"coord_leq_bound")

here = coord_geq_bound(a,b)
there = noif.coord_geq_bound(a,b)
all_passed += comp_func(here,there,"coord_geq_bound")

here = coord_less_bound(a,b)
there = noif.coord_less_bound(a,b)
all_passed += comp_func(here,there,"coord_less_bound")

here = coord_greater_bound(a,b)
there = noif.coord_greater_bound(a,b)
all_passed += comp_func(here,there,"coord_greater_bound")

import sys
if all_passed==0:
    print("ALL TESTS PASSED!")
else:
    print("ERROR: AT LEAST ONE TEST DID NOT PASS")
    sys.exit(1)
```

    ALL TESTS PASSED!

# Step 5: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-Min_Max_and_Piecewise_Expressions.pdf](Tutorial-Min_Max_and_Piecewise_Expressions.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
!jupyter nbconvert --to latex --template latex_nrpy_style.tplx --log-level='WARN' Tutorial-Min_Max_and_Piecewise_Expressions.ipynb
!pdflatex -interaction=batchmode Tutorial-Min_Max_and_Piecewise_Expressions.tex
!pdflatex -interaction=batchmode Tutorial-Min_Max_and_Piecewise_Expressions.tex
!pdflatex -interaction=batchmode Tutorial-Min_Max_and_Piecewise_Expressions.tex
!rm -f Tut*.out Tut*.aux Tut*.log
```

    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode


# Flow driven by an oscillating solid cylinder and bounded by a stationary outer hollow cylinder

\begin{equation}\label{p1introhpa} \Psi _{\eta \eta \tau }  = \frac{1}{{R_s }}\Psi _{\eta \eta \eta \eta }  + \Psi _{\eta \eta \eta } \Psi _\theta   - \Psi _{\eta \eta \theta } \Psi _\eta, \\
\quad \Psi(\eta=\mp 1) = 0, \qquad \Psi_\eta(\eta=\mp 1) = - \mu sin\theta, \qquad
-1 \le \eta \le1, \quad 0 \le \theta \le \pi
\end{equation}

## Calculating $\Psi(\theta,\eta)$ the steady state solution $\theta$-marching
The governing equation is solved using:

* Half domain $\eta$-domain $0 \le \eta \le 1$
* $\theta$ marching with some of the nonlinear terms being explicit

## Summary of results

## Small $R_s$ series solution

```julia
using DifferentialEquations
using Plots
using BenchmarkTools
using DelimitedFiles
using LaTeXStrings
using SparseArrays
using LinearAlgebra
using StaticArrays
using Crayons.Box
using Printf
```

```julia
const N = 128;  # Total number of intervals (steps-1) in the η direction
const M = 180;  # Total number of intervals (steps-1) in the θ direction
ψ_j = zeros(N+1,M+1); # Initialise the matrix for the streamfunction at the current time step
const Δη = 1.0/N;
const Δθ = pi/M;
const c    =  Δη/Δθ;
const μ = 1.0
```

    WARNING: redefinition of constant N. This may fail, cause incorrect answers, or produce other errors.
    WARNING: redefinition of constant Δη. This may fail, cause incorrect answers, or produce other errors.
    WARNING: redefinition of constant c. This may fail, cause incorrect answers, or produce other errors.

    1.0

```julia
function ψ_ηηη(n ::Int64, m:: Int64, ψ::Matrix{Float64}, meshsize::SVector{2, Int64}, delta_sizes::SVector{2, Float64}, paramtrs::SVector{1, Float64})
     μ = paramtrs[1]
     N = meshsize[1]
    (Δθ, Δη) = delta_sizes

    if  n > 2 && n < N

        return(-0.5*ψ[n-2,m] + ψ[n-1,m] - ψ[n+1,m] + 0.5*ψ[n+2,m])

    elseif n == 2

        return(-0.5*(-ψ[n,m]) + ψ[n-1,m] - ψ[n+1,m] + 0.5*ψ[n+2,m])

    elseif n == N

        return(-0.5*ψ[n-2,m] + ψ[n-1,m] - ψ[n+1,m] + 0.5*(ψ[n,m] - 2.0*Δθ*μ*sin(Δθ*m)))
    else
        println("Error in ψ_ηηη n =$n")
        return 0.0
    end
end

function ψ_ηη(n ::Int64, m:: Int64, ψ::Matrix{Float64}, meshsize::SVector{2, Int64})
    N = meshsize[1]
    if  n > 1 && n < N + 1

        return(  ψ[n-1,m] -2*ψ[n,m]+ ψ[n+1,m] )
    else
        println("Error in ψ_ηη n =$n")
        return 0.0
    end
end

function ψ_η(n ::Int64, m:: Int64, ψ::Matrix{Float64},  meshsize::SVector{2, Int64})
    N = meshsize[1]
    if  n >= 2 && n <= N

        return(  -0.5*ψ[n-1,m] + 0.5*ψ[n+1,m] )
    else
        println("Error in ψ_η n =$n")
        return 0.0
    end
end

meshsize = SA{Int64}[N,M]
delta_sizes= SA{Float64}[Δθ, Δη]
paramtrs =  SA{Float64}[μ]

rey_number = 5.0

d3  =  1.0/rey_number          #Diagonal entries for the matrix problem
d2  = -4.0/rey_number
d1  =  6.0/rey_number

for m = 2:M+1
    rhs = @view ψ_j[2:N,m]
    A =spdiagm(-2=>zeros(N-3), -1 => zeros(N-2), 0 => zeros(N-1), 1 => zeros(N-2), 2 => zeros(N-3))
    row = 1
    n = row + 1
    #A[row,row-2]  = nuemann boundary condition applied and add 0 to rhs
    #A[row,row-1] = 0 dirichlet boundary condition applied, this term is zero
    A[row,row]    = -d3 + d1 + ψ_ηηη(n, m-1, ψ_j, meshsize, delta_sizes, paramtrs)*c +2*ψ_η(n, m-1, ψ_j, meshsize)*c
    A[row,row+1]  = d2 -ψ_η(n, m-1, ψ_j, meshsize)*c
    A[row,row+2]  = d3

    rhs[row]      = ψ_ηηη(n, m-1, ψ_j, meshsize, delta_sizes, paramtrs)*ψ_j[n, m-1]*c +
                        -ψ_η(n, m-1, ψ_j, meshsize)*ψ_ηη(n, m-1, ψ_j, meshsize)*c

    row = 2
    n = row + 1
    #A[row,row-2]  = d3 dirichlet boundary condition applied, this term is zero
    A[row,row-1]  = d2 - ψ_η(n, m-1, ψ_j, meshsize)*c
    A[row,row]    = d1 + ψ_ηηη(n, m-1, ψ_j, meshsize, delta_sizes, paramtrs)*c +2*ψ_η(n, m-1, ψ_j, meshsize)*c
    A[row,row+1]  = d2 -ψ_η(n, m-1, ψ_j, meshsize)*c
    A[row,row+2]  = d3

    rhs[row]      = ψ_ηηη(n, m-1, ψ_j, meshsize, delta_sizes, paramtrs)*ψ_j[n, m-1]*c +
                    -ψ_η(n, m-1, ψ_j, meshsize)*ψ_ηη(n, m-1, ψ_j, meshsize)*c

    for row = 3:N-3
        n = row + 1
        A[row,row-2]  = d3
        A[row,row-1]  = d2 - ψ_η(n, m-1, ψ_j, meshsize)*c
        A[row,row]    = d1 + ψ_ηηη(n, m-1, ψ_j, meshsize, delta_sizes, paramtrs)*c +2*ψ_η(n, m-1, ψ_j, meshsize)*c
        A[row,row+1]  = d2 -ψ_η(n, m-1, ψ_j, meshsize)*c
        A[row,row+2]  = d3

        rhs[row]      = ψ_ηηη(n, m-1, ψ_j, meshsize, delta_sizes, paramtrs)*ψ_j[n, m-1]*c +
                        -ψ_η(n, m-1, ψ_j, meshsize)*ψ_ηη(n, m-1, ψ_j, meshsize)*c
    end

    row = N-2
    n = row + 1
    A[row,row-2]  = d3
    A[row,row-1]  = d2 - ψ_η(n, m-1, ψ_j, meshsize)*c
    A[row,row]    = d1 + ψ_ηηη(n, m-1, ψ_j, meshsize, delta_sizes, paramtrs)*c +2*ψ_η(n, m-1, ψ_j, meshsize)*c
    A[row,row+1]  = d2 -ψ_η(n, m-1, ψ_j, meshsize)*c
    #A[row,row+2]  = d3 dirichlet boundary condition applied, this term is zero

    rhs[row]      = ψ_ηηη(n, m-1, ψ_j, meshsize, delta_sizes, paramtrs)*ψ_j[n, m-1]*c +
                        -ψ_η(n, m-1, ψ_j, meshsize)*ψ_ηη(n, m-1, ψ_j, meshsize)*c

    row = N-1
    n = row + 1
    A[row,row-2]  = d3
    A[row,row-1]  = d2 - ψ_η(n, m-1, ψ_j, meshsize)*c
    A[row,row]    = d3 + d1 + ψ_ηηη(n, m-1, ψ_j, meshsize, delta_sizes, paramtrs)*c +2*ψ_η(n, m-1, ψ_j, meshsize)*c
    #A[row,row+1]  = d2 -ψ_η(n, m-1, ψ_j, meshsize)*c  dirichlet boundary condition applied, this term is zero
    #A[row,row+2]  = d3 nuemann boundary condition applied and add +2Δημsin(θ)*d3 to rhs

    rhs[row]      = ψ_ηηη(n, m-1, ψ_j, meshsize, delta_sizes, paramtrs)*ψ_j[n, m-1]*c +
                        -ψ_η(n, m-1, ψ_j, meshsize)*ψ_ηη(n, m-1, ψ_j, meshsize)*c + 2*Δη*μ*sin(Δθ*(m-1))*d3

    LUA = lu(A)
    ldiv!(LUA,rhs)
    A = nothing       #This frees up the memory on the next automated garbage collection GC.gc()
end

```

```julia

```

```julia

```

```julia
"""The symmetric (odd in η) asymptotic solution for the streamfunction for small \$R_s\$ with 5 terms"""
function Ψ_smallr(θ::Float64, η::Float64 ,r::Float64, number_of_terms::Int)
    terms = [
        -η*(η^2-1)*sin(θ)/2.0,

        r*η*(η^2-1)^2.0*(2.0+η^2)*sin(2*θ)/560.0,

        r^2.0*η*(η^2-1)^2*((-591 -2294*η^2 + 161*η^4 + 1428*η^6)*sin(θ) +
            +3.0*(423 + 166*η^2 - 553*η^4 - 84*η^6)sin(3*θ))/62092800.0,

        r^3.0*η*(η - 1)^2*(η + 1)^2*(2184*η^10*cos(θ)^2 + 1540*η^10 + 24213*η^8*cos(θ)^2 - 44205*η^8 +
            + 18942*η^6*cos(θ)^2 + 33446*η^6 - 101444*η^4*cos(θ)^2 + 66052*η^4 - 23086*η^2*cos(θ)^2 -
            6746*η^2 - 14889*cos(θ)^2 - 9383)*sin(θ)*cos(θ)/28252224000,

        r^4.0*η*(η - 1)^2*(η + 1)^2*(968647680*η^14*sin(θ)^5 + 14774464320*η^14*sin(θ) + 3510571680*η^14*sin(3*θ) -
            393513120*η^14*sin(5*θ) - 39671478000*η^12*sin(θ)^5 - 75537556710*η^12*sin(θ) -
            28276357725*η^12*sin(3*θ) - 3752563815*η^12*sin(5*θ) - 181021246560*η^10*sin(θ)^5 +
            357286638324*η^10*sin(θ) + 75142631718*η^10*sin(3*θ) - 1180776366*η^10*sin(5*θ) +
            677920708080*η^8*sin(θ)^5 - 645834258762*η^8*sin(θ) - 123813530379*η^8*sin(3*θ) -
            1711397457*η^8*sin(5*θ) + 193167076032*η^6*sin(θ)^5 - 82870720296*η^6*sin(θ) +
            215941919508*η^6*sin(3*θ) + 59820477948*η^6*sin(5*θ) + 153978403824*η^4*sin(θ)^5 -
            123221433370*η^4*sin(θ) + 181687022685*η^4*sin(3*θ) - 60662329497*η^4*sin(5*θ) -
            500785425504*η^2*sin(θ)^5 + 367229681332*η^2*sin(θ) - 134582502330*η^2*sin(3*θ) -
            33288605262*η^2*sin(5*θ) - 1507832445168*sin(θ)^5 + 969877856394*sin(θ) - 482955016725*sin(3*θ) -
            9092499471*sin(5*θ))/16189310893916160000
        ]
    if number_of_terms < 6
        return sum(terms[1:number_of_terms])
    else
        return term[1]
    end
end
```

    Ψ_smallr

```julia
psi_sol = reshape(ψ_j,length(ψ_j));
η_mesh = [(-1 +(n-1)*Δη)  for m=1:M+1 for n=1:N+1];
θ_mesh = [((m-1)*Δθ) for m=1:M+1 for n=1:N+1 ];
η_vector = range(-1,stop = 1,length = N+1);
θ_vector = range(0,stop = pi,length = M+1);
```

```julia

wireframe( range(-1.0,stop = 1.0,length = round(Int,N/8)+1),range(0.0,stop = pi,length = round(Int,M/8)+1), transpose(ψ_j[1:8:end,1:8:end]), colour=:greys, fmt = :svg)
yticks!(([0:π/6:π;], ["0", "\\pi/6", "\\pi/3", "\\pi/2", "2\\pi/3", "5\\pi/6","\\pi"]))
plot!(xlabel=L"\eta", ylabel=L"\theta", zlabel=L"\Psi", title=L"R_s = %$rey_number"*"\n"*L"N_η=%$N, M_θ=%$M", size=(1000,800),camera=(-40,30) )
plot!(xtickfont = font(12), ytickfont = font(12), ztickfont = font(12), titlefont = font(11))
#yticks!(([0:π/6:π;], ["0", "\\pi/6", "\\pi/3", "\\pi/2", "2\\pi/3", "5\\pi/6","\\pi"]))
#plot!(xlabel=L"η", ylabel=L"θ", zlabel=L"Ψ", title=L"R_s = %$rey_number"*"\n"*L"N_η=%$N, M_θ=%$M")
#plot!(xtickfont = font(12), ytickfont = font(12), ztickfont = font(12), titlefont=font(10))
```

```julia
plot(ψ_j[:,2])
#plot!(ψ_j[:,20])
#plot!(ψ_j[:,40])
```

```julia
pyplot()
m=80
eta=range(0, stop=1,length= N+1)
plot(eta, ψ_j[:,m], label="Steady numerical ", linecolor=:red,    linestyle=:dot,   linewidth =2, legend=:outertopright, size=(1100,500))
plot!(x->Ψ_smallr(Δθ*(m-1),x,rey_number,3),0,1, label="Small "*L"R_s"*" asymptotic",linestyle=:dot,   linewidth =3, linecolor=:blue)
```

```julia

```


# Propagation of uncertainty (error)

> Marcos Duarte
> Laboratory of Biomechanics and Motor Control ([http://demotu.org/](http://demotu.org/))
> Federal University of ABC, Brazil

> The result of a measurement is only an approximation or estimate of the value of the measurand and thus is complete only when accompanied by a statement of the uncertainty of that estimate.
>
> Uncertainty (of measurement) is a parameter, associated with the result of a measurement, that characterizes the dispersion of the values that could reasonably be attributed to the measurand.
>
> [Evaluation of measurement data - Guide to the expression of uncertainty in measurement (2008)](http://www.bipm.org/utils/common/documents/jcgm/JCGM_100_2008_E.pdf)

For instance, to calculate the density of a material someone made the following measurements of mass and volume (m=d/V):

$$ m = 4.0 \pm 0.5 \; g \\
V = 2.0 \pm 0.2 \; cm^3 $$

Where 0.5 g and 0.2 cm$^3$ each represent a value of one standard deviation and express the errors in the measured weight and volume.
Based on these numbers, the material density is 2 g/cm$^3$; but how much is the error (uncertainty) of the density?

One way to estimate the error in the density is by propagation of uncertainty:

> Propagation of uncertainty (or propagation of error) is the effect of variables' uncertainties (or errors) on the uncertainty of a function based on them. When the variables are the values of experimental measurements they have uncertainties due to measurement limitations (e.g., instrument precision and noise) which propagate to the combination of variables in the function ([Wikipedia](http://en.wikipedia.org/wiki/Propagation_of_uncertainty)).

## Propagation of uncertainty by linear approximation

The classic way for propagation of uncertainty is through linear approximation of error propagation theory. We start with the mathematical function that relates the physical quantities and estimate the contribution of their variations by means of partial derivatives (the rate of variation) of each of these primary quantities. The linearity of the mathematical function in the vicinity of the obtained value and the statistical correlations between these quantities are also taken into consideration in the propagation of uncertainty.
If the measurand $f$ is defined in terms of the variables $x, y, z, ...$ by a general expression $f(x, y, z, ...)$, a first order approximation of the propagation of uncertainty ignoring correlations between these variables is given by the well known formula:

$$ \sigma _{f}\;=\;\sqrt{\left ( \frac{\partial f}{\partial x}  \right)^2\sigma _{x}^{2} + \left ( \frac{\partial f}{\partial y} \right)^2 \sigma _{y}^{2} + \left ( \frac{\partial f}{\partial z} \right)^2 \sigma _{z}^{2}\: + \ldots} $$

Where $ \partial f/\partial i $ is the first partial derivative of $ f $ with respect to variable $ i $ and $ \sigma_i $ is the uncertainty of the measurement of the variable $ i $ (one standard deviation).

Using the formula above, the estimation of the uncertainty of the density is given by:

$$ d=\frac{m}{V} \\
\sigma _{d}\;=\;\sqrt{ \left ( \frac{\partial d}{\partial m}  \right)^2 \sigma _{m}^{2}+\left ( \frac{\partial d}{\partial V} \right)^2 \sigma _{V}^{2} } \\
\sigma _{d}\;=\;\sqrt{ \left ( \frac{1}{V} \right)^2 \sigma _{m}^{2}+\left ( -\frac{m}{V^2} \right)^2 \sigma _{V}^{2} } $$

And considering the values of mass and volume given earlier:

$$ \sigma_{d}\;=\;\sqrt{\left(\frac{1}{2}\right)^2\times 0.5\:^2+\left(-\frac{4}{2^2}\right)^2\times 0.2\:^2}\;\;\approx\; 0.32 $$

Then, the estimated value of the density can now be expressed in a complete form: d = 2.00±0.32 g/cm$^3$.

## Using a software to calculate uncertainty

It is essential that any result of a measurement is accompanied by the estimate of its uncertainty as above, this way we can better understand the value of the measure and how reliable it is. However, it is unlikely that anyone will repeat these calculations in the day-to-day work (although there are tables with partial derivatives for the most common expressions). Therefore, it is useful a software that can perform these calculations (at least for checking the results manually obtained). Fortunately, there are software for the propagation of uncertainty, [see this list on Wikipedia](http://en.wikipedia.org/wiki/List_of_uncertainty_propagation_software).

One such software, free, open source, and cross-platform, is the library for [Python](http://python.org) called [uncertainties](http://pythonhosted.org/uncertainties/). If you don't know or don't have Python, the easiest way to get it is to install the [Anaconda distribution](https://www.continuum.io/downloads), which is also free and cross-platform. Read the notebook [How to install Python](http://nbviewer.ipython.org/github/demotu/BMC/blob/master/notebooks/PythonInstallation.ipynb) if you need help with that.

With [uncertainties](http://pythonhosted.org/uncertainties/) installed, let's calculate again the uncertainty of the density:

```python
# Import the necessary libraries
from IPython.display import display, Math  # IPython formatting
from uncertainties import ufloat           # to define variables with uncertainty
```

Let's input the values:

```python
m = ufloat(nominal_value=4.0, std_dev=.5)  # value with uncertainty
V = ufloat(2.0, 0.2)                       # we can pass values directly
d = m/V                                    # calculate density

print(d)
```

    2.00+/-0.32

The result we obtained before, but without having to deduct the partial derivatives because [uncertainties](http://pythonhosted.org/uncertainties/) does this with an [algorithm for automatic differentiation](http://en.wikipedia.org/wiki/Automatic_differentiation).

And we can display this output more elegantly:

```python
display(Math('d = ' + r'{:.3L}'.format(d) + r'\; g/cm^3'))
display(Math('d = ' + r'{:.2eL}'.format(d) + r'\; g/cm^3'))
display(Math('d = ' + r'{:.2uS}'.format(d) + r'\; g/cm^3'))
```

$$d = 2.00 \pm 0.32\; g/cm^3$$

$$d = \left(2.00 \pm 0.32\right) \times 10^{0}\; g/cm^3$$

$$d = 2.00(32)\; g/cm^3$$

To evince how simple is to perform propagation of uncertainty in this way, the operation above could have been performed in a single line (after you imported the necessary libraries at the first time using them):

```python
print(ufloat(4.0, 0.5) / ufloat(2.0, 0.2))
```

    2.00+/-0.32

And how much each variable mass and volume contributed to the final uncertainty (i.e., the partial derivative of density in relation to the variable mass or volume times its standard deviation (each term on the right side of the equation above without squaring)) is given by:

```python
d.error_components().items()
```

    dict_items([(2.0+/-0.2, 0.2), (4.0+/-0.5, 0.25)])

By definition, according to the formula above, the square root of the quadratic sum of these values is the total uncertainty for the density.

The value of each partial derivative of the density function with respect to the mass and volume variables is given by:

```python
d.derivatives
```

    defaultdict(None, {2.0+/-0.2: -1.0, 4.0+/-0.5: 0.5})

Stepping back, we can even deduce the formula for the error propagation using the partial derivatives in case we want the actual formula using [Sympy](http://sympy.org/en/index.html):

```python
from sympy import symbols, Symbol, sqrt, Add, init_printing, latex
from IPython.display import display, Math, Latex
init_printing()

def stdform(formula, *errors):
    """
    Calculate the symbolic formula for error propagation using partial derivatives.
    Based on http://code.activestate.com/recipes/535164/
    """
    from functools import reduce

    formula = sqrt(reduce(Add, (formula.diff(var)**2 * Symbol('\sigma_' + var.name)**2
                          for var in formula.atoms(Symbol) if var.name in errors), 0.))

    return formula
```

```python
m, V = symbols('m, V')
d = m/V
formula = stdform(d, 'm', 'V')
display(Math('\sigma_d = ' + latex(formula)))
```

$$\sigma_d = \sqrt{\frac{\sigma_m^{2}}{V^{2}} + \frac{\sigma_V^{2} m^{2}}{V^{4}}}$$

Same as before, but Sympy deduced for us.

### Another example

Let's estimate the uncertainty in the measurement of the volume of a cylinder given its diameter ($d=2.0\pm0.1$) cm and height $h=10.0\pm0.5$ cm. But now, let's go straight to the solution:

```python
from sympy import pi
d, h = symbols('d, h')
V = h*pi*(d/2)**2

formula = stdform(V, 'd', 'h')
display(Math('\sigma_V = ' + latex(formula)))
```

$$\sigma_V = \sqrt{\frac{\pi^{2} \sigma_d^{2}}{4} d^{2} h^{2} + \frac{\pi^{2} \sigma_h^{2}}{16} d^{4}}$$

```python
from math import pi
h = ufloat(nominal_value=10.0, std_dev=0.5)
d = ufloat(nominal_value=2.0, std_dev=0.1)
V = h*pi*(d/2)**2
display(Math('V = ' + r'{:.3L}'.format(V) + r'\; cm^3'))
```

$$V = 31.4 \pm 3.5\; cm^3$$

## In conclusion

There is no excuse to not perform propagation of uncertainty of the result of a measurement.

**Remember, your work is complete only when you report the uncertainty.**

## References

- [Evaluation of measurement data - Guide to the expression of uncertainty in measurement](http://www.bipm.org/utils/common/documents/jcgm/JCGM_100_2008_E.pdf)
JCGM 100:2008.
- [Avaliação de dados de medição - Guia para a expressão de incerteza de medição](http://www.inmetro.gov.br/inovacao/publicacoes/gum_final.pdf) (GUM 2008 in Portuguese).
- [Uncertainties](http://packages.python.org/uncertainties): a Python package for calculations with uncertainties, Eric O. Lebigot.