# Easy authoring with Jupyter  notebooks

## Sagemath Notebook

## Key concepts

1.  Notebooks are stored in version control system (git) without output.
1.  ``Makefile`` is ised to keep all prescriptions for bulting
  pdf/html etc.
1.  ``nbconvert`` is used to automatically execute and clean notebooks.
1.  It will work with different Jupyter kenrels: we tested wit
  against SageMath and Python.
1.  Notebook will use ``PDF`` environment variable to distinguish
  between interactive output or static one. In the `Python` process is
  it easily accesible as ``os.environ['PDF']``. This can be
  interesting when using interactive features like `@interact`
  in Sage, which should be replaces by static figures in pdf format.
1.  All cells with tags ``nbtest`` will be removed from pdf or html
  output. This mechanism can be used for regression testing.
1.  We will include `Dockerfile` which allows to run a given
  repo on mybinder service.

```python
factor(12345678)
```

## TESTS

In building process all notebooks are,

 - stripped of output,
 - executed
 - stripped of cells with `nbtest` tags.

Tags can be edites if following mode is activated:

Stripped notebooks can be found in `notebooks4pdf` directory.

It will raise exception in compilation but will be invisible in pdf/html output.

## This cell will be removed after execution

```python
#  This cell will be removed after execution
assert 1==1
```

## Figure

- one should use markdown for figures (not html tags):

gives:

## Passing environment variable to notebooks

If compiled for pdf, in Makefile we set `PDF=1`. It can be read inside the notebook using following conditional:

    if  'PDF' in os.environ.keys():

There is an example below:

```python
import os
def fun(a):
    plot(sin(a*x),(x,-4,4)).show(figsize=3)

if  not 'PDF' in os.environ.keys():
    @interact
    def _(a = slider(1,2,0.1)):
        fun(a)
else:
    for a in [1,1.5,2]:
        fun(a)
```

## References to equations

We use $\LaTeX$ labels in equations, they will be used in PDF output:

\begin{equation}
\label{eq1}
2+2=5
\end{equation}

in $\ref{eq1}$ there is a problem!

## Problem with SageMath `show`  and $\LaTeX$

There is an issue in Sagemath which prevent from corect display of formulas in nbconvert. We circumvent it by using showmath function whereaver applicable

```python
expr = (x^2+1)/x^3*sin(x)+2^x
```

```python
show(expr)
```

```python
from IPython.display import Math
def showmath(expr):
    return Math(latex(expr))
```

```python
showmath(expr)
```

## Plotting

 - if using matplotlib, it is recommended to call `plt.show()`

```python
%matplotlib inline
import matplotlib.pyplot as plt
plt.plot([2,3,4])
plt.show()
```

```python
circle((0,1),0.2)
```

### Unsolved problem with long output

```python
expr = expand((1+x)^21)
```

```python
showmath( expr )
```

One can force newpage in $\LaTeX$ by:

\newpage


```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
```

#### Param Estimation

* We randomly count the number of observed moving traffic citations within a 5 mile radius of UH Manoa during 90-days, spread across 12 months
  * We want to even out sampling to avoid any bias
   * Ex. We may underestimate # citations during summer or overestimate # citations during the holiday season

* What does it mean to estimate the distribution parameter based on maximum likelihood?

```python
citations_data = pd.read_csv("https://www.dropbox.com/s/eg7wyjur1ff6d63/citations_counts.tsv?dl=1", index_col="Day")
citations_data.head()
```

      Counts

      Day

      0
      27

      1
      18

      2
      30

      3
      13

      4
      20

```python
plt.figure(figsize=(12,8))

plt.hist(citations_data["Counts"], bins=8, density=True, edgecolor='black', linewidth=1.2, alpha=0.5)
x = np.arange(30)

```

### Fitting the Data

* Since this is count data, we'll try to fit it using a Poisson distribution

* Form the data, we see that the most frequent value for the number of citations is $\approx$ 17
  * Poisson distribtion  with a mean ($\lambda$) of 8  is not likely to have generate this dataset
  * Poisson distribtion  with a mean ($\lambda$) of 18  is more likely to have generate this dataset that a value of $\lambda =8$

### Question

* Why do the two distributions have different heights?

```python
plt.figure(figsize=(12,8))

from scipy.stats import poisson

np.random.seed(4)

plt.hist(citations_data["Counts"], bins=8, density=True, edgecolor='black', linewidth=1.2)
x = np.arange(30)

p_y = poisson.pmf(x, 9)
plt.plot(x, p_y, color= 'r', linewidth=4, label="$\lambda=9$")
p_y = poisson.pmf(x, 18)
plt.plot(x, p_y, color= 'k', linewidth=4, label="$\lambda=18$")
plt.legend()
```

```python
plt.figure(figsize=(12,8))

plt.hist(citations_data["Counts"], bins=8, density=True, edgecolor='black', linewidth=1.2)

for i in [12,13,14,15,16, 17,18,19,20]:
    p_y = poisson.pmf(x, i)
    plt.plot(x, p_y, color= 'r', linewidth=0.5, label="$\lambda=%sa$" % i, alpha=.9)

p_y = poisson.pmf(x, 17)
plt.plot(x, p_y, color= 'r', linewidth=1.5, label="$\lambda=%sa$" % i)

```

### Question

* The graph below shows clearly that some plots better fit the data.
* How could we formally (or informally) decide which one is the best fit for the data?

### Computing the Probability of Observing these Data

* We know how to compute the probability of each point given a function.
  * We know how to compute the probability (pmf) of a single event
          `poisson.pmf(x, lambda)`
  * Since all events are independent, the probability of the complete dataset is simply the product of all probabilities
              `np.product(poisson.pmf(x, lambda))`
* We can compute the probability given each possible parameter $\lambda$
  * We limit ourselves to reasonable values

### Example: Compound probability of Independent Events

* Given a fair coin $p(H)=p(T) = 1/2$
* If we flip the coin twice, the sample sapce is \{HH, HT, TH, TT\} and  probabilities of the different outcomes is:

$$
p(HH) = p(H) * p(H) = 1/4 \\
p(HT) = p(H) * p(T) = 1/4 \\
p(TH) = p(T) * p(H) = 1/4 \\
p(TT) = p(T) * p(T) = 1/4 \\
$$

* As expected, all the probabilities sum to one

* This only works if the probabilities are independent; i.e., obtaining `head` in the first toss does not affect the outcome of the second coin flip

### Computing the Likelihood (Probability)

* If the events are independent, the probability of the data is is the product of the probabilities

* This is calld the the likelihood $L$ of the data given the parameter $\lambda$

$$
\begin{align}
L([x_1,x_2,...x_{90}] | \lambda) &= pmf(x_1, \lambda) \times  pmf(x_2, \lambda)  ....  pmf(x_{90}, \lambda) \\
  &=  \prod_{i=1}^{90} pmf(x_i, \lambda)
\end{align}
$$

```python
data_point = 12
_lambda = 16
poisson.pmf(data_point, _lambda)
```

    0.06612887239464967

```python
data_point = 16
_lambda = 16
poisson.pmf(data_point, _lambda)
```

    0.09921753162215541

```python

poisson.pmf(citations_data["Counts"], _lambda)

```

    array([0.00335386, 0.08300552, 0.00056393, 0.08138938, 0.05591951,
           0.04959665, 0.09921753, 0.06612887, 0.09338121, 0.06989938,
           0.09921753, 0.04959665, 0.09921753, 0.09921753, 0.09921753,
           0.06989938, 0.09301644, 0.08300552, 0.09338121, 0.0309857 ,
           0.08300552, 0.09921753, 0.02155527, 0.05591951, 0.0309857 ,
           0.09301644, 0.09338121, 0.09338121, 0.09921753, 0.01198747,
           0.04959665, 0.09301644, 0.00599374, 0.08300552, 0.09338121,
           0.09921753, 0.09921753, 0.08300552, 0.09301644, 0.09301644,
           0.09921753, 0.09338121, 0.08138938, 0.09921753, 0.08300552,
           0.09301644, 0.09338121, 0.06989938, 0.09921753, 0.06612887,
           0.09921753, 0.00105737, 0.09921753, 0.09921753, 0.08300552,
           0.09921753, 0.09301644, 0.06989938, 0.06612887, 0.02131106,
           0.05591951, 0.09301644, 0.09301644, 0.09301644, 0.02131106,
           0.09338121, 0.0309857 , 0.00919692, 0.06612887, 0.08138938,
           0.0340977 , 0.01437018, 0.09338121, 0.05591951, 0.0309857 ,
           0.09921753, 0.09301644, 0.04260534, 0.06612887, 0.04260534,
           0.09921753, 0.0340977 , 0.08138938, 0.06612887, 0.00919692,
           0.00919692, 0.09338121, 0.09921753, 0.09921753, 0.06612887])

```python
np.product(poisson.pmf(citations_data["Counts"], _lambda))

```

    1.1152800100792753e-113

### Question

* How can the likelihood of citation data be so low?

```python
>>> np.product(poisson.pmf(citations_data["Counts"], _lambda))

1.1152800100792753e-113
```

### Understanding the Likelihood (Probability)

* Why is the resulting probability so small?
  * Thik, for instance about a sample form 3 days where the citations were (16,15,21)
  * The sample space is large and includes (0,1,8), (1,10,10), (21,11,28), ...
  * If the maximimum number of citations daily is 50, then the sample space has a size 132,651.
     * The probability of observing (12,11,11) is just one event (possibility)  among  many (countably infinite) other events.

* The sample space for 90 days is $5.15377520732010019344215834280 \times 10^{29}$
    * It therefore makes sense that the computed likelihoos is so small

```python
_lambda = 16
np.product(poisson.pmf((16,15,16), _lambda))
```

    0.0009767091466218046

```python
np.product(poisson.pmf((9,11,11), 16))
```

    5.24215504845479e-05

```python
total_prob = 0
for i in range(0, 35):
    for j in range(0, 35):
        for k in range(0, 35):
            total_prob += np.product(poisson.pmf((i,j,k), 16))

print(total_prob)
```

    0.9999192928464893

```python
for i in range(12, 21):
    likelihood_data_i = np.product(poisson.pmf(citations_data["Counts"], i))
    print("The likelihood of the data given lambda=%s is %s" % (i, likelihood_data_i))
```

    The likelihood of the data given lambda=12 is 7.225758340021846e-142
    The likelihood of the data given lambda=13 is 1.3062039910366992e-129
    The likelihood of the data given lambda=14 is 3.6837061119261494e-121
    The likelihood of the data given lambda=15 is 5.438292115885739e-116
    The likelihood of the data given lambda=16 is 1.1152800100792753e-113
    The likelihood of the data given lambda=17 is 7.058672617915774e-114
    The likelihood of the data given lambda=18 is 2.670870493915664e-116
    The likelihood of the data given lambda=19 is 1.051303302700141e-120
    The likelihood of the data given lambda=20 is 6.877960001068855e-127

```python
plt.figure(figsize=(12,4))

x = range(12, 21)
l_x = [np.product(poisson.pmf(citations_data["Counts"], i)) for i in x]

frame1 = plt.gca()
frame1.axes.yaxis.set_ticklabels([])
plt.scatter(x, l_x)

_= plt.ylim(-1.1152800100792753e-113, 0.2e-112)

```

### Understanding the Likelihood - Cont'd

* Working with small numbers can lead to likelihood to be very small
  * Can lead to arithmetic floating point underflow
    * Results of a calculation smaller than what can actually be represented in memory
* Rather than computing likelihood, we can calculate Log-likelihood
  * Scale changes but share of the function does not

### Understanding the Likelihood - Cont'd

We know that:

$$
log(x \cdot y) = log(x) + log(y)
$$

Therefore:

\begin{align}
\mbox{log}(L([x_1,x_2,..., x_{90}] | \lambda)) &= \mbox{log}(\mbox{pmf}(x_1, \lambda)) \times  \mbox{log}(\mbox{pmf}(x_2, \lambda))  ....  \mbox{log}(\mbox{pmf}(x_{90}, \lambda)) \\
  &=  \sum_{i=1}^{90} \mbox{log}(\mbox{pmf}(x_i, \lambda))
\end{align}

* The resulting values are no longer probabilities as they don't sum to 1

```python
plt.figure(figsize=(12,8))

x = range(12, 21)
l_x = [np.sum(np.log(poisson.pmf(citations_data["Counts"], i))) for i in x]

plt.scatter(x, l_x)

```

### Maximum likelihood For the Poisson Distribution

* We hypothesize that the data was generated using a parameter $\lambda$
  * Among all possible lambda values, the data is most likely under one of those
  * We say that this  value maximizes the likelihood

* We can quickly show that the value that maximized the likelihood merely is the sample mean!

```python
>>> np.mean(citations_data["Counts"])
16.41
```
* While we assumed that the data is best represented using a Poisson, we could have manually tested other distributions and selected the one that provides the highest likelihood
  * Therefore, this approach is not only suitable for estimating a parameter, but also making decisions

# Example: A/B Testing

* In an A/B test, we are interested in comparing two versions of a webpage
  * We would like to determine which one performs better at a task `X`

* Ex.: Is Version A (new) or Version B (old) better at signing up new users?

### Maximum Likelihood For a Binomial Distribution

* For now, we are only concerned with version A.

* We test 8 individuals (unrealistically small sample) using version A
 * 5 sign up for the newsletter

*  What can we infer about the type of data and parameter that gave rise to this outcome

### Maximum Likelihood For a Binomial Distribution

1. The data is distributed according to a binomial distribution
   * $n=8$ and $p \in [0,1]$

2 Intuitively, we know that parameter cannot too small or too big
  * From the data, p is somewhere in the upper  side of the range
  * Ex the data is more likely with a $p=0.8$ than with  $p=0.03$

* We can determine $p$ using the same approach used with traffic citations data

```python
from scipy.stats import binom
n = 8
p = 0.5
binom.pmf(3,n ,p)
```

    0.21875000000000014

```python
binom.pmf(4, n ,p)
```

    0.27343750000000017

```python
x = np.arange(0.001,1,0.025)
l_x_vals = []
for p in x:
    l_x_i = np.sum(np.log(binom.pmf(5, n , p)))
    l_x_vals.append(l_x_i)

plt.scatter(x, l_x_vals)
plt.scatter(x[np.argmax(l_x_vals)], np.max(l_x_vals))

plt.axvline(0.625, color='r')

```

### Maximum Likelihood For a Binomial Distribution - Cont'd

* If we toss a coin 100 times and observe 85 heads, then we assume the probability of heads is 0.85
* Since 5 out of 8 clients signed up, we assume that $p = 5/ 8 =  0.625$
  * Therefor the maximum likelihood ($p=0.625$) result is consistent with our intuition $p = 5/ 8 =  0.625$

###  Maximum Likelihood Estimation

* The maximum likelihood estimators for parameters of common distributions can be computed analytically
  * Ex. The maximum estimator of the mean in a gaussian is simply the sample mean and the estimator of the standard deviation is the sample's standard deviation

$$
\hat{\mu} = \frac{1}{n} \sum_{i=1}^{n}x_i,
$$
and
$$
\hat{\mu} = \frac{1}{n} \sum_{i=1}^{n}{(x_i - \bar{x})^2}.
$$

* Recall that ML estimators are simply point-estimates that maximize the likelihood of data
* Do not give us a probability of the value we generate
 * We do not have the probability or confidence that it is the correct parameter of the population
 * they don't even provide confidence interfals, as the bootstrap approach did.

* The questions is then: ** "How different could this estimate have been, if the sample had come out differently?"**


# Analysis of the difference equations

## Properties of the solution of the wave equation

The wave equation

$$
\frac{\partial^2 u}{\partial t^2} =
c^2 \frac{\partial^2 u}{\partial x^2}
$$

has solutions of the form

$$
\begin{equation}
u(x,t) = g_R(x-ct) + g_L(x+ct),
\label{wave:pde1:gensol} \tag{1}
\end{equation}
$$

for any functions $g_R$ and $g_L$ sufficiently smooth to be differentiated
twice. The result follows from inserting ([1](#wave:pde1:gensol))
in the wave equation. A function of the form $g_R(x-ct)$ represents a
signal
moving to the right in time with constant velocity $c$.
This feature can be explained as follows.
At time $t=0$ the signal looks like $g_R(x)$. Introducing a
moving horizontal coordinate $\xi = x-ct$, we see the function
$g_R(\xi)$ is "at rest"
in the $\xi$ coordinate system, and the shape is always
the same. Say the $g_R(\xi)$ function has a peak at $\xi=0$. This peak
is located at $x=ct$, which means that it moves with the velocity
$dx/dt=c$ in the $x$ coordinate system. Similarly, $g_L(x+ct)$
is a function, initially with shape $g_L(x)$, that moves in the negative
$x$ direction with constant velocity $c$ (introduce $\xi=x+ct$,
look at the point $\xi=0$, $x=-ct$, which has velocity $dx/dt=-c$).

With the particular initial conditions

$$
u(x,0)=I(x),\quad \frac{\partial}{\partial t}u(x,0) =0,
$$

we get, with $u$ as in ([1](#wave:pde1:gensol)),

$$
g_R(x) + g_L(x) = I(x),\quad -cg_R'(x) + cg_L'(x) = 0\thinspace .
$$

The former suggests $g_R=g_L$, and the former then leads to
$g_R=g_L=I/2$. Consequently,

$$
\begin{equation}
u(x,t) = \frac{1}{2} I(x-ct) + \frac{1}{2} I(x+ct) \thinspace .
\label{wave:pde1:gensol2} \tag{2}
\end{equation}
$$

The interpretation of ([2](#wave:pde1:gensol2)) is that
the initial shape of $u$ is split into two parts, each with the same
shape as $I$ but frac{1}{2}
of the initial amplitude. One part is traveling to the left and the
other one to the right.

The solution has two important physical features: constant amplitude
of the left and right wave, and constant velocity of these two waves.
mathcal{I}_t turns out that the numerical solution will also preserve the
constant amplitude, but the velocity depends on the mesh parameters
$\Delta t$ and $\Delta x$.

The solution ([2](#wave:pde1:gensol2)) will be influenced by
boundary conditions when the parts
$\frac{1}{2} I(x-ct)$ and $\frac{1}{2} I(x+ct)$ hit the boundaries and get, e.g.,
reflected back into the domain. However, when $I(x)$ is nonzero
only in a small part in the middle
of the spatial domain $[0,L]$, which means that the
boundaries are placed far away from the initial disturbance of $u$,
the solution ([2](#wave:pde1:gensol2)) is very clearly observed
in a simulation.

A useful representation of solutions of wave equations is a linear
combination of sine and/or cosine waves. Such a sum of waves is a
solution if the governing PDE is linear and each sine or cosine
wave fulfills the
equation.  To ease analytical calculations by hand we shall work with
complex exponential functions instead of real-valued sine or cosine
functions. The real part of complex expressions will typically be
taken as the physical relevant quantity (whenever a physical relevant
quantity is strictly needed).
The idea now is to build $I(x)$ of complex wave components
$e^{ikx}$:

$$
\begin{equation} I(x) \approx \sum_{k\in K} b_k e^{ikx} \thinspace .
\label{wave:Fourier:I} \tag{3}
\end{equation}
$$

Here, $k$ is the frequency of a component,
$K$ is some set of all the discrete
$k$ values needed to approximate $I(x)$ well,
and $b_k$ are
constants that must be determined. We will very seldom
need to compute the $b_k$ coefficients: most of the insight
we look for, and the understanding of the numerical methods we want to
establish, come from
investigating how the PDE and the scheme treat a single
component $e^{ikx}$ wave.

Letting the number of $k$ values in $K$ tend to infinity, makes the sum
([3](#wave:Fourier:I)) converge to $I(x)$. This sum is known as a
*Fourier series* representation of $I(x)$.  Looking at
([2](#wave:pde1:gensol2)), we see that the solution $u(x,t)$, when
$I(x)$ is represented as in ([3](#wave:Fourier:I)), is also built of
basic complex exponential wave components of the form $e^{ik(x\pm
ct)}$ according to

$$
\begin{equation}
u(x,t) = \frac{1}{2} \sum_{k\in K} b_k e^{ik(x - ct)}
+ \frac{1}{2} \sum_{k\in K} b_k e^{ik(x + ct)} \thinspace .
\label{wave:Fourier:u1} \tag{4}
\end{equation}
$$

mathcal{I}_t is common to introduce the frequency in time $\omega = kc$ and
assume that $u(x,t)$ is a sum of basic wave components
written as $e^{ikx -\omega t}$.
(Observe that inserting such a wave component in the governing PDE reveals that
$\omega^2 = k^2c^2$, or $\omega =\pm kc$, reflecting the
two solutions: one ($+kc$) traveling to the right and the other ($-kc$)
traveling to the left.)

## More precise definition of Fourier representations

The above introduction to function representation by sine and cosine
waves was quick and intuitive, but will suffice as background
knowledge for the following material of single wave component
analysis.
However, to understand
all details of how different wave components sum up to the analytical
and numerical solutions, a more precise mathematical treatment is helpful
and therefore summarized below.

mathcal{I}_t is well known that periodic functions can be represented by
Fourier series. A generalization of the Fourier series idea to
non-periodic functions defined on the real line is the *Fourier transform*:

$$
\begin{equation}
I(x) = \int_{-\infty}^\infty A(k)e^{ikx}dk,
\label{wave:pde1:Fourier:I} \tag{5}
\end{equation}
$$

$$
\begin{equation}
A(k) = \int_{-\infty}^\infty I(x)e^{-ikx}dx\thinspace .
\label{wave:pde1:Fourier:A} \tag{6}
\end{equation}
$$

The function $A(k)$ reflects the weight of each wave component $e^{ikx}$
in an infinite sum of such wave components. That is, $A(k)$
reflects the frequency content in the function $I(x)$. Fourier transforms
are particularly fundamental for analyzing and understanding time-varying
signals.

The solution of the linear 1D wave PDE can be expressed as

$$
u(x,t) = \int_{-\infty}^\infty A(k)e^{i(kx-\omega(k)t)}dx\thinspace .
$$

In a finite difference method, we represent $u$ by a mesh function
$u^n_q$, where $n$ counts temporal mesh points and $q$ counts
the spatial ones (the usual counter for spatial points, $i$, is
here already used as imaginary unit). Similarly, $I(x)$ is approximated by
the mesh function $I_q$, $q=0,\ldots,N_x$.
On a mesh, it does not make sense to work with wave
components $e^{ikx}$ for very large $k$, because the shortest possible
sine or cosine wave that can be represented uniquely
on a mesh with spacing $\Delta x$
is the wave with wavelength $2\Delta x$. This wave has its peaks
and throughs at every two mesh points. That is, the wave "jumps up and down"
between the mesh points.

The corresponding $k$ value for the shortest possible wave in the mesh is
$k=2\pi /(2\Delta x) = \pi/\Delta x$. This maximum frequency is
known as the *Nyquist frequency*.
Within the range of
relevant frequencies $(0,\pi/\Delta x]$ one defines
the [discrete Fourier transform](http://en.wikipedia.org/wiki/Discrete_Fourier_transform), using $N_x+1$ discrete frequencies:

$$
\begin{equation}
I_q = \frac{1}{N_x+1}\sum_{k=0}^{N_x} A_k e^{i2\pi k q/(N_x+1)},\quad
q=0,\ldots,N_x,
\label{_auto1} \tag{7}
\end{equation}
$$

$$
\begin{equation}
A_k = \sum_{q=0}^{N_x} I_q e^{-i2\pi k q/(N_x+1)},
\quad k=0,\ldots,N_x\thinspace .
\label{_auto2} \tag{8}
\end{equation}
$$

The $A_k$ values represent the discrete Fourier transform of the $I_q$ values,
which themselves are the inverse discrete Fourier transform of the $A_k$
values.

The discrete Fourier transform is efficiently computed by the
*Fast Fourier transform* algorithm. For a real function $I(x)$,
the relevant Python code for computing and plotting
the discrete Fourier transform appears in the example below.

```python
%matplotlib inline

import numpy as np
from numpy import sin, pi

def I(x):
    return sin(2*pi*x) + 0.5*sin(4*pi*x) + 0.1*sin(6*pi*x)

# Mesh
L = 10; Nx = 100
x = np.linspace(0, L, Nx+1)
dx = L/float(Nx)

# Discrete Fourier transform
A = np.fft.rfft(I(x))
A_amplitude = np.abs(A)

# Compute the corresponding frequencies
freqs = np.linspace(0, pi/dx, A_amplitude.size)

import matplotlib.pyplot as plt
plt.plot(freqs, A_amplitude)
plt.show()
```

## Stability

The scheme

$$
\begin{equation}
[D_tD_t u = c^2 D_xD_x u]^n_q
\label{wave:pde1:analysis:scheme} \tag{9}
\end{equation}
$$

for the wave equation $u_{tt} = c^2u_{xx}$ allows basic wave components

$$
u^n_q=e^{i(kx_q - \tilde\omega t_n)}
$$

as solution, but it turns out that
the frequency in time, $\tilde\omega$, is not equal to
the exact frequency $\omega = kc$.  The goal now is to
find exactly what $\tilde \omega$ is. We ask two key
questions:

 * How accurate is $\tilde\omega$
   compared to $\omega$?

 * Does the amplitude of such a wave component
   preserve its (unit) amplitude, as it should,
   or does it get amplified or damped in time (because of a complex $\tilde\omega$)?

The following analysis will answer these questions. We shall
continue using $q$ as an identifier for a certain mesh point in
the $x$ direction.

### Preliminary results

A key result needed in the investigations is the finite difference
approximation of a second-order derivative acting on a complex
wave component:

$$
[D_tD_t e^{i\omega t}]^n = -\frac{4}{\Delta t^2}\sin^2\left(
\frac{\omega\Delta t}{2}\right)e^{i\omega n\Delta t}
\thinspace .
$$

By just changing symbols ($\omega\rightarrow k$,
$t\rightarrow x$, $n\rightarrow q$) it follows that

$$
[D_xD_x e^{ikx}]_q = -\frac{4}{\Delta x^2}\sin^2\left(
\frac{k\Delta x}{2}\right)e^{ikq\Delta x} \thinspace .
$$

### Numerical wave propagation

Inserting a basic wave component $u^n_q=e^{i(kx_q-\tilde\omega t_n)}$ in
([9](#wave:pde1:analysis:scheme)) results in the need to
evaluate two expressions:

$$
\lbrack D_tD_t e^{ikx}e^{-i\tilde\omega t}\rbrack^n_q = \lbrack D_tD_t e^{-i\tilde\omega t}\rbrack^ne^{ikq\Delta x}\nonumber
$$

$$
\begin{equation}  = -\frac{4}{\Delta t^2}\sin^2\left(
\frac{\tilde\omega\Delta t}{2}\right)e^{-i\tilde\omega n\Delta t}e^{ikq\Delta x}
\label{_auto3} \tag{10}
\end{equation}
$$

$$
\lbrack D_xD_x e^{ikx}e^{-i\tilde\omega t}\rbrack^n_q = \lbrack D_xD_x e^{ikx}\rbrack_q e^{-i\tilde\omega n\Delta t}\nonumber
$$

$$
\begin{equation}  = -\frac{4}{\Delta x^2}\sin^2\left(
\frac{k\Delta x}{2}\right)e^{ikq\Delta x}e^{-i\tilde\omega n\Delta t} \thinspace .   \label{_auto4} \tag{11}
\end{equation}
$$

Then the complete scheme,

$$
\lbrack D_tD_t e^{ikx}e^{-i\tilde\omega t} = c^2D_xD_x e^{ikx}e^{-i\tilde\omega t}\rbrack^n_q
$$

leads to the following equation for the unknown numerical
frequency $\tilde\omega$
(after dividing by $-e^{ikx}e^{-i\tilde\omega t}$):

$$
\frac{4}{\Delta t^2}\sin^2\left(\frac{\tilde\omega\Delta t}{2}\right)
= c^2 \frac{4}{\Delta x^2}\sin^2\left(\frac{k\Delta x}{2}\right),
$$

or

$$
\begin{equation}
\sin^2\left(\frac{\tilde\omega\Delta t}{2}\right)
= C^2\sin^2\left(\frac{k\Delta x}{2}\right),
\label{wave:pde1:analysis:sineq1} \tag{12}
\end{equation}
$$

where

$$
\begin{equation}
C = \frac{c\Delta t}{\Delta x}
\label{_auto5} \tag{13}
\end{equation}
$$

is the Courant number.
Taking the square root of ([12](#wave:pde1:analysis:sineq1)) yields

$$
\begin{equation}
\sin\left(\frac{\tilde\omega\Delta t}{2}\right)
= C\sin\left(\frac{k\Delta x}{2}\right),
\label{wave:pde1:analysis:sineq2} \tag{14}
\end{equation}
$$

Since the exact $\omega$ is real it is reasonable to look for a real
solution $\tilde\omega$ of ([14](#wave:pde1:analysis:sineq2)).
The right-hand side of
([14](#wave:pde1:analysis:sineq2)) must then be in $[-1,1]$ because
the sine function on the left-hand side has values in $[-1,1]$
for real $\tilde\omega$. The magnitude of the sine function on
the right-hand side attains the value 1 when

$$
\frac{k\Delta x}{2} = \frac{\pi}{2} + m\pi,\quad m\in\Integer
\thinspace .
$$

With $m=0$ we have $k\Delta x = \pi$, which means that
the wavelength $\lambda = 2\pi/k$ becomes $2\Delta x$. This is
the absolutely shortest wavelength that can be represented on the mesh:
the wave jumps up and down between each mesh point. Larger values of $|m|$
are irrelevant since these correspond to $k$ values whose
waves are too short to be represented
on a mesh with spacing $\Delta x$.
For the shortest possible wave in the mesh, $\sin\left(k\Delta x/2\right)=1$,
and we must require

$$
\begin{equation}
C\leq 1 \thinspace .
\label{wave:pde1:stability:crit} \tag{15}
\end{equation}
$$

Consider a right-hand side in ([14](#wave:pde1:analysis:sineq2)) of
magnitude larger
than unity. The solution $\tilde\omega$ of ([14](#wave:pde1:analysis:sineq2))
must then be a complex number
$\tilde\omega = \tilde\omega_r + i\tilde\omega_i$ because
the sine function is larger than unity for a complex argument.
One can show that for any $\omega_i$  there will also be a
corresponding solution with $-\omega_i$. The component with $\omega_i>0$
gives an amplification factor $e^{\omega_it}$ that grows exponentially
in time. We cannot allow this and must therefore require $C\leq 1$
as a *stability criterion*.

**Remark on the stability requirement.**

For smoother wave components with longer wave lengths per length $\Delta x$,
([15](#wave:pde1:stability:crit)) can in theory be relaxed. However,
small round-off errors are always present in a numerical solution and these
vary arbitrarily from mesh point to mesh point and can be viewed as
unavoidable noise with wavelength $2\Delta x$. As explained, $C>1$
will for this very small noise lead to exponential growth of
the shortest possible wave component in the mesh. This noise will
therefore grow with time and destroy the whole solution.

## Numerical dispersion relation

Equation ([14](#wave:pde1:analysis:sineq2)) can be solved with respect
to $\tilde\omega$:

$$
\begin{equation}
\tilde\omega = \frac{2}{\Delta t}
\sin^{-1}\left( C\sin\left(\frac{k\Delta x}{2}\right)\right) \thinspace .
\label{wave:pde1:disprel} \tag{16}
\end{equation}
$$

The relation between the numerical frequency $\tilde\omega$ and
the other parameters $k$, $c$, $\Delta x$, and $\Delta t$ is called
a *numerical dispersion relation*. Correspondingly,
$\omega =kc$ is the *analytical dispersion relation*.
In general, dispersion refers to the phenomenon where the wave
velocity depends on the spatial frequency ($k$, or the
wave length $\lambda = 2\pi/k$) of the wave.
Since the wave velocity is $\omega/k =c$, we realize that the
analytical dispersion relation reflects the fact that there is no
dispersion. However, in a numerical scheme we have dispersive waves
where the wave velocity depends on $k$.

The special case $C=1$ deserves attention since then the right-hand side
of ([16](#wave:pde1:disprel)) reduces to

$$
\frac{2}{\Delta t}\frac{k\Delta x}{2} = \frac{1}{\Delta t}
\frac{\omega\Delta x}{c} = \frac{\omega}{C} = \omega \thinspace .
$$

That is, $\tilde\omega = \omega$ and the numerical solution is exact
at all mesh points regardless of $\Delta x$ and $\Delta t$!
This implies that the numerical solution method is also an analytical
solution method, at least for computing $u$ at discrete points (the
numerical method says nothing about the
variation of $u$ *between* the mesh points, and employing the
common linear interpolation for extending the discrete solution
gives a curve that in general deviates from the exact one).

For a closer examination of the error in the numerical dispersion
relation when $C<1$, we can study
$\tilde\omega -\omega$, $\tilde\omega/\omega$, or the similar
error measures in wave velocity: $\tilde c - c$ and $\tilde c/c$,
where $c=\omega /k$ and $\tilde c = \tilde\omega /k$.
mathcal{I}_t appears that the most convenient expression to work with is $\tilde c/c$,
since it can be written as a function of just two parameters:

$$
\frac{\tilde c}{c} = \frac{1}{Cp}{\sin}^{-1}\left(C\sin p\right),
$$

with $p=k\Delta x/2$ as a non-dimensional measure of the spatial frequency.
In essence, $p$ tells how many spatial mesh points we have per
wave length in space for the wave component with frequency $k$ (recall
that the wave
length is $2\pi/k$). That is, $p$ reflects how well the
spatial variation of the wave component
is resolved in the mesh. Wave components with wave length
less than $2\Delta x$ ($2\pi/k < 2\Delta x$) are not visible in the mesh,
so it does not make sense to have $p>\pi/2$.

We may introduce the function $r(C, p)=\tilde c/c$ for further investigation
of numerical errors in the wave velocity:

$$
\begin{equation}
r(C, p) = \frac{1}{Cp}{\sin}^{-1}\left(C\sin p\right), \quad C\in (0,1],\ p\in (0,\pi/2] \thinspace .
\label{wave:pde1:disprel2} \tag{17}
\end{equation}
$$

This function is very well suited for plotting since it combines several
parameters in the problem into a dependence on two dimensionless
numbers, $C$ and $p$.

The fractional error in the wave velocity for different Courant numbers.

Defining

        def r(C, p):
            return 2/(C*p)*asin(C*sin(p))

we can plot $r(C,p)$ as a function of $p$ for various values of
$C$, see [Figure](#wave:pde1:fig:disprel). Note that the shortest
waves have the most erroneous velocity, and that short waves move
more slowly than they should.

We can also easily make a Taylor series expansion in the
discretization parameter $p$:

```python
import sympy as sym
C, p = sym.symbols('C p')
# Compute the 7 first terms around p=0 with no O() term
rs = r(C, p).series(p, 0, 7).removeO()
rs
```

```python
# Pick out the leading order term, but drop the constant 1
rs_error_leading_order = (rs - 1).extract_leading_order(p)
rs_error_leading_order
```

```python
# Turn the series expansion into a Python function
rs_pyfunc = lambdify([C, p], rs, modules='numpy')
```

```python
# Check: rs_pyfunc is exact (=1) for C=1
rs_pyfunc(1, 0.1)
```

Note that without the `.removeO()` call the series gets an `O(x**7)` term
that makes it impossible to convert the series to a Python function
(for, e.g., plotting).

From the `rs_error_leading_order` expression above, we see that the leading
order term in the error of this series expansion is

$$
\begin{equation}
\frac{1}{6}\left(\frac{k\Delta x}{2}\right)^2(C^2-1)
= \frac{k^2}{24}\left( c^2\Delta t^2 - \Delta x^2\right),
\label{_auto6} \tag{18}
\end{equation}
$$

pointing to an error $\Oof{\Delta t^2, \Delta x^2}$, which is
compatible with the errors in the difference approximations ($D_tD_tu$
and $D_xD_xu$).

We can do more with a series expansion, e.g., factor it to see how
the factor $C-1$ plays a significant role.
To this end, we make a list of the terms, factor each term,
and then sum the terms:

```python
rs = r(C, p).series(p, 0, 4).removeO().as_ordered_terms()
rs
```

```python
rs = [factor(t) for t in rs]
rs
```

```python
rs = sum(rs)  # Python's sum function sums the list
rs
```

We see from the last expression
that $C=1$ makes all the terms in `rs` vanish.
Since we already know that the numerical solution is exact for $C=1$, the
remaining terms in the Taylor series expansion
will also contain factors of $C-1$ and cancel for $C=1$.

## Extending the analysis to 2D and 3D

The typical analytical solution of a 2D wave equation

$$
u_{tt} = c^2(u_{xx} + u_{yy}),
$$

is a wave traveling in the direction of $\kk = k_x\ii + k_y\jj$, where
$\ii$ and $\jj$ are unit vectors in the $x$ and $y$ directions, respectively
($\ii$ should not be confused with $i=\sqrt{-1}$ here).
Such a wave can be expressed by

$$
u(x,y,t) = g(k_xx + k_yy - kct)
$$

for some twice differentiable function $g$, or with $\omega =kc$, $k=|\kk|$:

$$
u(x,y,t) = g(k_xx + k_yy - \omega t)\thinspace .
$$

We can, in particular, build a solution by adding complex Fourier components
of the form

$$
e^{(i(k_xx + k_yy - \omega t))}
\thinspace .
$$

A discrete 2D wave equation can be written as

$$
\begin{equation}
\lbrack D_tD_t u = c^2(D_xD_x u + D_yD_y u)\rbrack^n_{q,r}
\thinspace .
\label{wave:pde1:analysis:scheme2D} \tag{19}
\end{equation}
$$

This equation admits a Fourier component

$$
\begin{equation}
u^n_{q,r} = e^{\left( i(k_x q\Delta x + k_y r\Delta y -
\tilde\omega n\Delta t)\right)},
\label{wave:pde1:analysis:numsol2D} \tag{20}
\end{equation}
$$

as solution. Letting the operators $D_tD_t$, $D_xD_x$, and $D_yD_y$
act on $u^n_{q,r}$ from ([20](#wave:pde1:analysis:numsol2D)) transforms
([19](#wave:pde1:analysis:scheme2D)) to

$$
\begin{equation}
\frac{4}{\Delta t^2}\sin^2\left(\frac{\tilde\omega\Delta t}{2}\right)
= c^2 \frac{4}{\Delta x^2}\sin^2\left(\frac{k_x\Delta x}{2}\right)
+ c^2 \frac{4}{\Delta y^2}\sin^2\left(\frac{k_y\Delta y}{2}\right) \thinspace .   \label{_auto7} \tag{21}
\end{equation}
$$

or

$$
\begin{equation}
\sin^2\left(\frac{\tilde\omega\Delta t}{2}\right)
= C_x^2\sin^2 p_x
+ C_y^2\sin^2 p_y,  \label{_auto8} \tag{22}
\end{equation}
$$

where we have eliminated the factor 4 and introduced the symbols

$$
C_x = \frac{c\Delta t}{\Delta x},\quad
C_y = \frac{c\Delta t}{\Delta y}, \quad
p_x = \frac{k_x\Delta x}{2},\quad
p_y = \frac{k_y\Delta y}{2}\thinspace .
$$

For a real-valued $\tilde\omega$ the right-hand side
must be less than or equal to unity in absolute value, requiring in general
that

$$
\begin{equation}
C_x^2 + C_y^2 \leq 1 \thinspace .
\label{wave:pde1:analysis:2DstabC} \tag{23}
\end{equation}
$$

This gives the stability criterion, more commonly expressed directly
in an inequality for the time step:

$$
\begin{equation}
\Delta t \leq \frac{1}{c} \left( \frac{1}{\Delta x^2} +
\frac{1}{\Delta y^2}\right)^{-\frac{1}{2}i}
\label{wave:pde1:analysis:2Dstab} \tag{24}
\end{equation}
$$

A similar, straightforward analysis for the 3D case leads to

$$
\begin{equation}
\Delta t \leq \frac{1}{c}\left( \frac{1}{\Delta x^2} +
\frac{1}{\Delta y^2} + \frac{1}{\Delta z^2}\right)^{-\frac{1}{2}i}
\label{_auto9} \tag{25}
\end{equation}
$$

In the case of a variable coefficient $c^2=c^2(\xpoint)$, we must use
the worst-case value

$$
\begin{equation}
\bar c = \sqrt{\max_{\xpoint\in\Omega} c^2(\xpoint)}
\label{_auto10} \tag{26}
\end{equation}
$$

in the stability criteria. Often, especially in the variable wave
velocity case, it is wise to introduce a safety factor $\beta\in (0,1]$ too:

$$
\begin{equation}
\Delta t \leq \beta \frac{1}{\bar c}
\left( \frac{1}{\Delta x^2} +
\frac{1}{\Delta y^2} + \frac{1}{\Delta z^2}\right)^{-\frac{1}{2}i}
\label{_auto11} \tag{27}
\end{equation}
$$

The exact numerical dispersion relations in 2D and 3D becomes, for constant $c$,

$$
\begin{equation}
\tilde\omega = \frac{2}{\Delta t}\sin^{-1}\left(
\left( C_x^2\sin^2 p_x + C_y^2\sin^2 p_y\right)^\frac{1}{2}\right),
\label{_auto12} \tag{28}
\end{equation}
$$

$$
\begin{equation}
\tilde\omega = \frac{2}{\Delta t}\sin^{-1}\left(
\left( C_x^2\sin^2 p_x + C_y^2\sin^2 p_y + C_z^2\sin^2 p_z\right)^\frac{1}{2}\right)\thinspace .
\label{_auto13} \tag{29}
\end{equation}
$$

We can visualize the numerical dispersion error in 2D much like we did
in 1D. To this end, we need to reduce the number of parameters in
$\tilde\omega$. The direction of the wave is parameterized by the
polar angle $\theta$, which means that

$$
k_x = k\sin\theta,\quad k_y=k\cos\theta\thinspace .
$$

A simplification is to set $\Delta x=\Delta y=h$.
Then $C_x=C_y=c\Delta t/h$, which we call $C$. Also,

$$
p_x=\frac{1}{2} kh\cos\theta,\quad p_y=\frac{1}{2} kh\sin\theta\thinspace .
$$

The numerical frequency $\tilde\omega$
is now a function of three parameters:

  * $C$, reflecting the number of cells a wave is displaced during a time step,

  * $p=\frac{1}{2} kh$, reflecting the number of cells per wave length in space,

  * $\theta$, expressing the direction of the wave.

We want to visualize the error in the numerical frequency. To avoid having
$\Delta t$ as a free parameter in $\tilde\omega$, we work with
$\tilde c/c = \tilde\omega/(kc)$. The coefficient in front of the
$\sin^{-1}$ factor is then

$$
\frac{2}{kc\Delta t} = \frac{2}{2kc\Delta t h/h} =
\frac{1}{Ckh} = \frac{2}{Cp},
$$

and

$$
\frac{\tilde c}{c} = \frac{2}{Cp}
\sin^{-1}\left(C\left(\sin^2 (p\cos\theta)
+ \sin^2(p\sin\theta) \right)^\frac{1}{2}\right)\thinspace .
$$

We want to visualize this quantity as a function of
$p$ and $\theta$ for some values of $C\leq 1$. mathcal{I}_t is
instructive
to make color contour plots of $1-\tilde c/c$ in
*polar coordinates* with $\theta$ as the angular coordinate and
$p$ as the radial coordinate.

The stability criterion ([23](#wave:pde1:analysis:2DstabC))
becomes $C\leq C_{\max} = 1/\sqrt{2}$ in the present 2D case with the
$C$ defined above. Let us plot $1-\tilde c/c$ in polar coordinates
for $C_{\max}, 0.9C_{\max}, 0.5C_{\max}, 0.2C_{\max}$.
The program below does the somewhat tricky
work in Matplotlib, and the result appears
in [Figure](#wave:pde1:fig:disprel2D). From the figure we clearly
see that the maximum $C$ value gives the best results, and that
waves whose propagation direction makes an angle of 45 degrees with
an axis are the most accurate.

```python
def dispersion_relation_2D(p, theta, C):
    arg = C*sqrt(sin(p*cos(theta))**2 +
                 sin(p*sin(theta))**2)
    c_frac = 2./(C*p)*arcsin(arg)

    return c_frac

import numpy as np
from numpy import \
     cos, sin, arcsin, sqrt, pi  # for nicer math formulas

r = p = np.linspace(0.001, pi/2, 101)
theta = np.linspace(0, 2*pi, 51)
r, theta = np.meshgrid(r, theta)

# Make 2x2 filled contour plots for 4 values of C
import matplotlib.pyplot as plt
C_max = 1/sqrt(2)
C = [[C_max, 0.9*C_max], [0.5*C_max, 0.2*C_max]]
fix, axes = plt.subplots(2, 2, subplot_kw=dict(polar=True))
for row in range(2):
    for column in range(2):
        error = 1 - dispersion_relation_2D(
            p, theta, C[row][column])
        print error.min(), error.max()
        # use vmin=error.min(), vmax=error.max()
        cax = axes[row][column].contourf(
            theta, r, error, 50, vmin=-1, vmax=-0.28)
        axes[row][column].set_xticks([])
        axes[row][column].set_yticks([])

# Add colorbar to the last plot
cbar = plt.colorbar(cax)
cbar.ax.set_ylabel('error in wave velocity')
plt.savefig('disprel2D.png');  plt.savefig('disprel2D.pdf')
plt.show()
```

Error in numerical dispersion in 2D.