 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli

```python
from __future__ import print_function

%matplotlib inline
import numpy
import matplotlib.pyplot as plt
```

# Sources of Error

Error can come from many sources when using applying a numerical method:
 - Model/Data Error
 - Discretization Error
 - Convergence Error
 - Floating Point Error

**Goal:** Categorize and understand each type of error and explore some simple approaches to analyzing error.

## Model and Data Error

Errors in fundamental formulation
 - Lotka-Volterra - fractional rabbits, no extinctions, etc.
 - Data Error - Inaccuracy in measurement or uncertainties in parameters

Unfortunatley we cannot control model and data error directly but we can use methods that may be more robust in the presense of these types of errors.

## Discretization or Truncation Error

Errors arising from approximating a function with a simpler function, e.g. Using the approximation $\sin(x) \approx x$ when $|x| \approx 0$.

## Floating Point Error

Errors arising from approximating real numbers with finite-precision numbers and arithmetic.

## Convergence Error

In some instances an algorithm is developed that will take a current approximation and then find an improvement on the current approximation. In some instances the errors generated in each indivudal step can accumulate or become magnified after repeating the algorithm a number of times.

## Basic Definitions

Before exploring the different kinds of error, it is important to first define the ways that error is measured. Given a true value of a function $f$ and an approximate solution $F$ define:

Absolute Error:
$$
    e = | f - F |
$$

Relative Error:
$$
    r = \frac{e}{|f|} = \frac{|f - F|}{|f|}
$$

### Decimal Precision

This definition of relative error provides a convenient estimate for the number of digits of decimal precision $p$

given a relative error $r$,  the precision $p$ is the largest integer such that
$$
    r \leq 5\times 10^{-p}
$$

Example
* if $r = 0.001 < 5\times10^{-3}$ has $p=3$ significant digits
* if $r = 0.006 < 5\times10^{-2}$ has $p=2$ significant digits (because this error would cause rounding up)

### Example

let
$$
    f = e^1,\quad F=2.71
$$

```python
f = numpy.exp(1.)
F = 2.71
e = numpy.abs(f - F)
r = e/numpy.abs(f)
p = int(-numpy.log10(r/5.))
print('f = {}'.format(f))
print('F = {}'.format(F))

print('Absolute Error: {}'.format(e))
print('Relative Error: {}'.format(r))
print('Decimal precision: {}'.format(p))
```

    f = 2.718281828459045
    F = 2.71
    Absolute Error: 0.008281828459045126
    Relative Error: 0.0030467144253912685
    Decimal precision: 3

### Big-O Notation

In many situations an approximation will have a parameter associated with it, and the value of the parameter is often chosen to insure that the error is reasonable in a given situation. In such circumstances we often want to know the impact on the error if we change the value of the parameter. This leads to the definition of Big-O notation:
$$
    f(x) =  O(g(x)) \quad \text{as} \quad x \rightarrow a
$$
if and only if
$$
    |f(x)| \leq M |g(x)| \quad \text{as}\quad  |x - a| < \delta \quad \text{where} \quad M,a > 0.
$$

In practice we use Big-O notation to say something about how the terms we may have left out of a series might behave.  We saw an example earlier of this with the Taylor's series approximations.

#### Example:
given $f(x) = \sin x$ with $x_0 = 0$,  then the Taylor polynomial (truncated Taylor series) of $f(x)$ expanded around $x_0=0$ is

$$T_N(x) = \sum^N_{n=0} (-1)^{n} \frac{x^{2n+1}}{(2n+1)!}$$

For $N=2$, we can then write $f(x)$  as

$$f(x) = x - \frac{x^3}{6} + \frac{x^5}{120} + O(x^7)$$

This becomes more useful when we look at this using  $x = x_0 + \Delta x$:

$$f(x) = \Delta x - \frac{\Delta x^3}{6} + \frac{\Delta x^5}{120} + O(\Delta x^7)$$

**We can also develop rules for error propagation based on Big-O notation:**

In general, there are two theorems that do not need proof and hold when the value of x is large:

Let
$$\begin{aligned}
    f(x) &= p(x) + O(x^n) \\
    g(x) &= q(x) + O(x^m) \\
    k &= \max(n, m)
\end{aligned}$$
then
$$
    f+g = p + q + O(x^k)
$$
and
\begin{align}
    f \cdot g &= p \cdot q + p O(x^m) + q O(x^n) + O(x^{n + m}) \\
    &= p \cdot q + O(x^{n+m})
\end{align}

On the other hand, if we are interested in small values of x, say $\Delta x$, the above expressions can be modified as follows:

\begin{align}
    f(\Delta x) &= p(\Delta x) + O(\Delta x^n) \\
    g(\Delta x) &= q(\Delta x) + O(\Delta x^m) \\
    r &= \min(n, m)
\end{align}
then
$$
    f+g = p + q + O(\Delta x^r)
$$
and
\begin{align}
    f \cdot g &= p \cdot q + p \cdot O(\Delta x^m) + q \cdot O(\Delta x^n) + O(\Delta x^{n+m}) \\
    &= p \cdot q + O(\Delta x^r)
\end{align}

**Note:** In this case we suppose that at least the polynomial with $k = \max(n, m)$ has the following form:

$$
    p(\Delta x) = 1 + p_1 \Delta x + p_2 \Delta x^2 + \ldots
$$
or
$$
    q(\Delta x) = 1 + q_1 \Delta x + q_2 \Delta x^2 + \ldots
$$

so that there is an $\mathcal{O}(1)$ term that guarantees the existence of $\mathcal{O}(\Delta x^r)$ in the final product.

To get a sense of why we care most about the power on $\Delta x$ when considering convergence the following figure shows how different powers on the convergence rate can effect how quickly we converge to our solution.  Note that here we are plotting the same data two different ways.  Plotting the error as a function of $\Delta x$ is a common way to show that a numerical method is doing what we expect and exhibits the correct convergence behavior.  Since errors can get small quickly it is very common to plot these sorts of plots on a log-log scale to easily visualize the results.  Note that if a method was truly of the order $n$ that they will be a linear function in log-log space with slope $n$.

#### Behavior of error as a function of $\Delta x$

```python
dx = numpy.linspace(1.0, 1e-4, 100)

fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2.0)
axes = []
axes.append(fig.add_subplot(1, 2, 1))
axes.append(fig.add_subplot(1, 2, 2))

for n in range(1, 5):
    axes[0].plot(dx, dx**n, label="$\Delta x^%s$" % n)
    axes[1].loglog(dx, dx**n, label="$\Delta x^%s$" % n)

axes[0].legend(loc=2)
axes[1].set_xticks([10.0**(-n) for n in range(5)])
axes[1].set_yticks([10.0**(-n) for n in range(16)])
axes[1].legend(loc=4)
for n in range(2):
    axes[n].set_title("Growth of Error vs. $\Delta x^n$")
    axes[n].set_xlabel("$\Delta x$")
    axes[n].set_ylabel("Estimated Error")

plt.show()
```

## Discretization Error

**Taylor's Theorem:**  Let $f(x) \in C^{m+1}[a,b]$ and $x_0 \in [a,b]$, then for all $x \in (a,b)$ there exists a number $c = c(x)$ that lies between $x_0$ and $x$ such that

$$ f(x) = T_N(x) + R_N(x)$$

where $T_N(x)$ is the Taylor polynomial approximation

$$T_N(x) = \sum^N_{n=0} \frac{f^{(n)}(x_0)\cdot(x-x_0)^n}{n!}$$

and $R_N(x)$ is the residual (the part of the series we left off)

$$R_N(x) = \frac{f^{(n+1)}(c) \cdot (x - x_0)^{n+1}}{(n+1)!}$$

### Note

The residual:

$$
    R_N(x) = \frac{f^{(n+1)}(c) \cdot (x - x_0)^{n+1}}{(n+1)!}
$$

depends on the $n+1$ order derivative of $f$ evaluated at an **unknown** value $c\in[x,x_0]$.

If we knew the value of $c$ we would know the exact value of $R_N(x)$ and therefore the function $f(x)$.  In general we don't know this value but we can use $R_N(x)$ to put upper bounds on the error **and** to understand how the error changes as we move away from $x_0$.

Start by replacing $x - x_0$ with $\Delta x$.  The primary idea here is that the residual $R_N(x)$ becomes smaller as $\Delta x \rightarrow 0$ (at which point $T_N(x) = f(x_0)$).

$$
    T_N(x) = \sum^N_{n=0} \frac{f^{(n)}(x_0)\cdot\Delta x^n}{n!}
$$

and $R_N(x)$ is the residual (the part of the series we left off)

$$
    R_N(x) = \frac{f^{(n+1)}(c) \cdot \Delta x^{n+1}}{(n+1)!} \leq M \Delta x^{n+1} = O(\Delta x^{n+1})
$$

where $M$ is an upper bound on
$$
    \frac{f^{(n+1)}(c)}{(n+1)!}
$$

#### Example 1

$f(x) = e^x$ with $x_0 = 0$ on the interval $x\in(-1,1)$

Using this we can find expressions for the relative and absolute error as a function of $x$ assuming $N=2$.

Derivatives:
$$\begin{aligned}
    f'(x) &= e^x \\
    f''(x) &= e^x \\
    f^{(n)}(x) &= e^x
\end{aligned}$$

Taylor polynomials:
$$\begin{aligned}
    T_N(x) &= \sum^N_{n=0} e^0 \frac{x^n}{n!} \Rightarrow \\
    T_2(x) &= 1 + x + \frac{x^2}{2}
\end{aligned}$$

Remainders:
$$\begin{aligned}
    R_N(x) &= e^c \frac{x^{n+1}}{(n+1)!} = e^c \cdot \frac{x^3}{6} \quad \Rightarrow \\
    R_2(x) &\leq \frac{e^1}{6} \approx 0.5
\end{aligned}$$

Accuracy:
\begin{align}
    \exp(1) &= 2.718\ldots \\
    T_2(1) &= 2.5
\end{align}

$$
\Rightarrow e \approx 0.2,\quad r \approx 0.08,\quad p = 1
$$

We can also use the package sympy which has the ability to calculate Taylor polynomials built-in!

```python
import sympy
sympy.init_printing(pretty_print=True)
x = sympy.symbols('x')
f = sympy.exp(x)
f.series(x0=0, n=3)
```

```python
x = 1.
Tn = 1 + x + x**2/2
f = numpy.exp(1.)
e = numpy.abs(f-Tn)
r = e/f
p = int(-numpy.log10(r/5.))
print("T_n = {}, e = {}, r = {}, p = {}".format(Tn,e,r,p))
```

    T_n = 2.5, e = 0.2182818284590451, r = 0.08030139707139415, p = 1

Lets plot this numerically for a section of $x$.

```python
x = numpy.linspace(-1, 1, 100)
f = numpy.exp(x)
T_N = 1.0 + x + x**2 / 2.0
R_N = numpy.exp(1) * x**3 / 6.0

plt.figure(figsize=(8,6))
plt.plot(x, T_N, 'r', x, f, 'k', x, R_N, 'b')
plt.plot(x,numpy.exp(x)-T_N,'g--')
plt.plot(0.0, 1.0, 'o', markersize=10)
plt.grid()
plt.xlabel("x",fontsize=16)
plt.ylabel("$f(x)$, $T_N(x)$, $R_N(x)$", fontsize=16)
plt.legend(["$T_N(x)$", "$f(x)$", "$R_N(x)$", "e(x)"], loc=2)
plt.show()
```

#### Example 2

Approximate
$$
    f(x) = \frac{1}{x} \quad x_0  = 1,
$$
using $x_0 = 1$ to the 3rd Taylor series term.

$$
\begin{matrix}
f'(x) = -\frac{1}{x^2}, &  f''(x) = \frac{2}{x^3}, & f'''(x) = -\frac{6}{x^4}, & \ldots, & f^{(n)}(x) &= \frac{(-1)^n n!}{x^{n+1}}
\end{matrix}
$$

$$
\begin{aligned}
    T_N(x) &= \sum^N_{n=0} (-1)^n (x-1)^n \Rightarrow \\
    T_2(x) &= 1 - (x - 1) + (x - 1)^2
\end{aligned}
$$

$$
\begin{aligned}
    R_N(x) &= \frac{(-1)^{n+1}(x - 1)^{n+1}}{c^{n+2}} \Rightarrow \\
    R_2(x) &= \frac{-(x - 1)^{3}}{c^{4}}
\end{aligned}
$$

### plot this problem

```python
x = numpy.linspace(0.8, 2, 100)
f = 1.0 / x
T_N = 1.0 - (x-1) + (x-1)**2
R_N = -(x-1.0)**3 / (1.**4)

plt.figure(figsize=(8,6))
plt.plot(x, T_N, 'r', x, f, 'k', x, R_N, 'b')
plt.plot(x,f - T_N,'g--')
plt.plot(1.0, 1.0, 'o', markersize=10)
plt.grid(True)
plt.xlabel("x",fontsize=16)
plt.ylabel("$f(x)$, $T_N(x)$, $R_N(x)$",fontsize=16)
plt.title('$f(x) = 1/x$',fontsize=18)
plt.legend(["$T_N(x)$", "$f(x)$", "$R_N(x)$", '$e(x)$'], loc='best')
plt.show()
```

### Computational Issue #1: Accuracy... how many terms?

Given a Taylor Polynomial approximation of an arbitrary function $f(x)$,  how do we determine how many terms are required such that $R_N(x)<tol$.  And how do we determine the tolerance?

### Computational Issue #2 Efficiency... Operation counts for polynomial evaluation

Given

$$P_N(x) = a_0 + a_1 x + a_2 x^2 + \ldots + a_N x^N$$

or

$$P_N(x) = p_0 x^N + p_1 x^{N-1} + p_2 x^{N-2} + \ldots + p_{N}$$

what is the most **efficient way**  to evaluate $P_N(x)$? (i.e. minimize number of floating point operations)

Consider two ways to write $P_3$

* The standard way:

$$ P_3(x) = p_0 x^3 + p_1 x^2 + p_2 x + p_3$$

* using nested multiplication (aka **Horner's Method**):

$$ P_3(x) = ((p_0 x + p_1) x + p_2) x + p_3$$

Consider how many operations it takes for each...

$$ P_3(x) = p_0 x^3 + p_1 x^2 + p_2 x + p_3$$

$$P_3(x) = \overbrace{p_0 \cdot x \cdot x \cdot x}^3 + \overbrace{p_1\cdot x \cdot x}^2 + \overbrace{p_2 \cdot x}^1 + p_3$$

Note:  here we're just counting multiplications as they will dominate the flop count

Adding up all the operations we can in general think of this as a pyramid

Thus we can estimate that the algorithm written this way will take approximately $O(N^2 / 2)$ operations to complete.

Looking at nested iteration, however:

$$ P_3(x) = ((p_0 x + p_1) x + p_2) x + p_3$$

Here we find that the method is $O(N)$ compared to the first evaluation which $O(N^2)$ (we usually drop the 2 in these cases).  That's a huge difference for large $N$!

#### Algorithm

Fill in the function and implement Horner's method:
```python
def eval_poly(p, x):
    """Evaluates polynomial given coefficients p at x

    Function to evaluate a polynomial in order N operations.  The polynomial is defined as

    P(x) = p[0] x**n + p[1] x**(n-1) + ... + p[n-1] x + p[n]

    The value x should be a float.
    """
    pass
```

```python
def eval_poly(p, x):
    """Evaluates polynomial given coefficients p at x

    Function to evaluate a polynomial in order N operations.  The polynomial is defined as

    P(x) = p[0] x**n + p[1] x**(n-1) + ... + p[n-1] x + p[n]

    The value x should be a float.
    """
    ### ADD CODE HERE
    pass
```

```python
p = [1, -3, 10, 4, 5, 5]
x = numpy.linspace(-1, 1, 100)
eval_poly(p,x)
```

```python
# Scalar version
def eval_poly(p, x):
    """Evaluates polynomial given coefficients p at x

    Function to evaluate a polynomial in order N operations.  The polynomial is defined as

    P(x) = p[0] x**n + p[1] x**(n-1) + ... + p[n-1] x + p[n]

    The value x should be a float.
    """

    y = p[0]
    for coefficient in p[1:]:
        y = y * x + coefficient

    return y
```

```python
# Vectorized version
def eval_poly(p, x):
    """Evaluates polynomial given coefficients p at x

    Function to evaluate a polynomial in order N operations.  The polynomial is defined as

    P(x) = p[0] x**n + p[1] x**(n-1) + ... + p[n-1] x + p[n]

    The value x can by a NumPy ndarray.
    """

    y = numpy.ones(x.shape) * p[0]
    for coefficient in p[1:]:
        y = y * x + coefficient

    return y
```

```python
p = [1, -3, 10, 4, 5, 5]
x = numpy.linspace(-10, 10, 100)
plt.plot(x, eval_poly(p, x))
plt.grid()
plt.show()
```

## Convergence Error

In some circumstances a formula or algorithm is applied repeatedly as a way to obtain a final approximation. Usually, the errors that occur at each individual step are small. By repeating the algorithm, though, the errors can sometimes grow or become magnified.

As example of this phenomena is given below. The values of the terms in a difference equation are calculated,
$$
   \begin{align}
      y_0 &= 1, \\
      y_1 &= \frac{1}{5}, \\
      y_{n+1} &= \frac{16}{5} y_n - \frac{3}{5} y_{n-1}.
   \end{align}
$$

The true solution to the difference equation is $y_n = \left(\frac{1}{5}\right)^n$, where $n=$0, 1, 2, $\ldots$

```python
# Choose the number of iterations
N = 40
y = numpy.empty(N+1)            # Allocate an empty vector with N+1 entries

# Now use the difference equation to generate the numbers in the sequence
y[0] = 1
y[1] = 1/5
for n in range(2,N+1):
    y[n] = 16/5*y[n-1] - 3/5*y[n-2]

```

And plot the result

```python
# Now plot the result
n = numpy.arange(0,N+1)
fig = plt.figure(figsize=(10.0, 5.0))
axes = fig.add_subplot(1, 1, 1)
axes.semilogy(n,y, 'rx', markersize=5, label='$y_n$')
axes.semilogy(n,(1/5)**n,'b.', label='$y_{true}$')
axes.grid()
axes.set_title("Calculated Values Of A Difference Equation",fontsize=18)
axes.set_xlabel("$n$",fontsize=16)
axes.set_ylabel("$y_n$",fontsize=16)
axes.legend(loc='best', shadow=True)
plt.show()
```

Simply looking at the exact solution, the sequence of numbers generated by the difference equation above should get very close to zero. Instead, the numbers in the sequence initially get closer to zero, but at some point they begin to grow and get larger. An underlying problem is that the computer is not able to store the numbers exactly. The second number in the sequence, $y_1=\frac{1}{5}$ has a small error, and the computer stores it as $y_1 = \frac{1}{5}+\epsilon$ where $\epsilon$ is some small error.

Each time a new number in the loop is generated, the error is multiplied. For example, after the first iteration $y_2$ is
$$
    \begin{align}
       y_2 &= \frac{16}{5} \left( \frac{1}{5}+\epsilon \right)
               - \frac{3}{5} \left( 1 \right), \\
           &= \frac{1}{5^2} + \frac{16}{5} \epsilon.
    \end{align}
$$

After the second time through the loop, the value of $y_3$ is

$$
y_3=\frac{1}{5^3} + \frac{241}{25}\epsilon
$$

Even though the value of $\epsilon$ is very close to zero, every iteration makes the error grow.  Repeated multiplication will result in a very large number.

The error associated with the initial representation of the number $\frac{1}{5}$ is a problem with the way a digital computer stores floating point numbers. In most instances the computer cannot represent a number exactly, and the small error in approximating a given number can give rise to other problems.

## Floating Point Error

Errors arising from approximating real numbers with finite-precision numbers

$$\pi \approx 3.14$$

or $\frac{1}{3} \approx 0.333333333$ in decimal, results form finitely number of registers to represent each number.

### Floating Point Systems

Numbers in floating point systems are represented as a series of bits that represent different pieces of a number.  In *normalized floating point systems* there are some standard conventions for what these bits are used for.  In general the numbers are stored by breaking them down into the form

$$F = \pm d_1 . d_2 d_3 d_4 \ldots d_p \times \beta^E$$

where
1. $\pm$ is a single bit and of course represents the sign of the number
2. $d_1 . d_2 d_3 d_4 \ldots d_p$ is called the *mantissa*.  Note that technically the decimal could be moved but generally, using scientific notation, the decimal can always be placed at this location.  The digits $d_2 d_3 d_4 \ldots d_p$ are called the *fraction* with $p$ digits of precision.  Normalized systems specifically put the decimal point in the front like we have and assume $d_1 \neq 0$ unless the number is exactly $0$.
3. $\beta$ is the *base*.  For binary $\beta = 2$, for decimal $\beta = 10$, etc.
4. $E$ is the *exponent*, an integer in the range $[E_{\min}, E_{\max}]$

The important points on any floating point system is that
1. There exist a discrete and finite set of representable numbers
2. These representable numbers are not evenly distributed on the real line
3. Arithmetic in floating point systems yield different results from infinite precision arithmetic (i.e. "real" math)

#### Properties of Floating Point Systems
All floating-point systems are characterized by several important numbers
 - Smalled normalized number (underflow if below - related to subnormal numbers around zero)
 - Largest normalized number (overflow if above)
 - Zero
 - Machine $\epsilon$ or $\epsilon_{\text{machine}}$
 - `inf` and `nan`, infinity and **N**ot **a** **N**umber respectively

##### Example:  Toy System
Consider the toy 2-digit precision decimal system (normalized)
$$f = \pm d_1 . d_2 \times 10^E$$
with $E \in [-2, 0]$.

**Number and distribution of numbers**
1. How many numbers can we represent with this system?

2. What is the distribution on the real line?

3. What is the underflow and overflow limits?

How many numbers can we represent with this system?

$$
    f = \pm d_1 . d_2 \times 10^E ~~~ \text{with} E \in [-2, 0]
$$

$$
    2 \times 9 \times 10 \times 3 + 1 = 541
$$

What is the distribution on the real line?

```python
d_1_values = [1, 2, 3, 4, 5, 6, 7, 8, 9]
d_2_values = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
E_values = [0, -1, -2]

fig = plt.figure(figsize=(10.0, 1.0))
axes = fig.add_subplot(1, 1, 1)

for E in E_values:
    for d1 in d_1_values:
        for d2 in d_2_values:
            axes.plot( (d1 + d2 * 0.1) * 10**E, 0.0, 'r+', markersize=20)
            axes.plot(-(d1 + d2 * 0.1) * 10**E, 0.0, 'r+', markersize=20)

axes.plot(0.0, 0.0, '+', markersize=20)
axes.plot([-10.0, 10.0], [0.0, 0.0], 'k')

axes.set_title("Distribution of Values")
axes.set_yticks([])
axes.set_xlabel("x")
axes.set_ylabel("")
axes.set_xlim([-1, 1])
plt.show()
```

What is the underflow and overflow limits?

Smallest number that can be represented is the underflow:  $1.0 \times 10^{-2} = 0.01$
Largest number that can be represented is the overflow:  $9.9 \times 10^0 = 9.9$

### Binary Systems
Consider the 2-digit precision base 2 system:

$$
    f=\pm d_1 . d_2 \times 2^E \quad \text{with} \quad E \in [-1, 1]
$$

#### Number and distribution of numbers
1. How many numbers can we represent with this system?

2. What is the distribution on the real line?

3. What is the underflow and overflow limits?

How many numbers can we represent with this system?

$$f=\pm d_1 . d_2 \times 2^E ~~~~ \text{with} ~~~~ E \in [-1, 1]$$

$$ 2 \times 1 \times 2 \times 3 + 1 = 13$$

What is the distribution on the real line?

```python
d_1_values = [1]
d_2_values = [0, 1]
E_values = [1, 0, -1]

fig = plt.figure(figsize=(10.0, 1.0))
axes = fig.add_subplot(1, 1, 1)

for E in E_values:
    for d1 in d_1_values:
        for d2 in d_2_values:
            axes.plot( (d1 + d2 * 0.5) * 2**E, 0.0, 'r+', markersize=20)
            axes.plot(-(d1 + d2 * 0.5) * 2**E, 0.0, 'r+', markersize=20)

axes.plot(0.0, 0.0, 'r+', markersize=20)
axes.plot([-4.5, 4.5], [0.0, 0.0], 'k')

axes.set_title("Distribution of Values")
axes.set_yticks([])
axes.set_xlabel("x")
axes.set_ylabel("")
axes.set_xlim([-5, 5])
plt.show()
```

Smallest number that can be represented is the underflow:  $1.0 \times 2^{-1} = 0.5$

Largest number that can be represented is the overflow:  $1.1 \times 2^1 = 3$

Note that these numbers are in a binary system.

Quick rule of thumb:
$$
    2^3 2^2 2^1 2^0 . 2^{-1} 2^{-2} 2^{-3}
$$
correspond to
8s, 4s, 2s, 1s . halves, quarters, eighths, ...

### Real Systems - IEEE 754 Binary Floating Point Systems

#### Single Precision
 - Total storage alloted is 32 bits
 - Exponent is 8 bits $\Rightarrow E \in [-126, 127]$
 - Fraction 23 bits ($p = 24$)

```
s EEEEEEEE FFFFFFFFFFFFFFFFFFFFFFF
0 1      8 9                     31
```
* Overflow $= 2^{127} \approx 3.4 \times 10^{38}$
* Underflow $= 2^{-126} \approx 1.2 \times 10^{-38}$
* $\epsilon_{\text{machine}} = 2^{-23} \approx 1.2 \times 10^{-7}$

#### Double Precision
 - Total storage alloted is 64 bits
 - Exponent is 11 bits $\Rightarrow E \in [-1022, 1024]$
 - Fraction 52 bits ($p = 53$)

```
s EEEEEEEEEE FFFFFFFFFF FFFFFFFFFF FFFFFFFFFF FFFFFFFFFF FFFFFFFFFF FF
0 1       11 12                                                      63
```
* Overflow $= 2^{1024} \approx 1.8 \times 10^{308}$
* Underflow $= 2^{-1022} \approx 2.2 \times 10^{-308}$
* $\epsilon_{\text{machine}} = 2^{-52} \approx 2.2 \times 10^{-16}$

### Python Access to IEEE Numbers

Access many important parameters, such as machine epsilon:

```python
import numpy
numpy.finfo(float).eps
```

```python
print(numpy.finfo(numpy.float16))
print(numpy.finfo(numpy.float32))
```

```python
print(numpy.finfo(float))
print(numpy.finfo(numpy.float128))
```

### Examples

```python
eps = numpy.finfo(float).eps
MAX = numpy.finfo(float).max
print('eps = {}'.format(eps))
print('MAX = {}'.format(MAX))
```

Show that $(1 + \epsilon_{mach}) > 1$

```python
print(MAX*1)
print(MAX*(1+eps))
```

## Why should we care about this?

 - Floating point arithmetic is not commutative or associative
 - Floating point errors compound, do not assume even double precision is enough!
 - Mixing precision is very dangerous

### Example 1: Simple Arithmetic

Simple arithmetic $\delta < \epsilon_{\text{machine}}$

   $$(1+\delta) - 1 = 1 - 1 = 0$$

   $$1 - 1 + \delta = \delta$$

```python
eps = numpy.finfo(float).eps
delta = 0.5*eps
print('eps =   {},  1 + eps   > 1 is {}'.format(eps,((1. + eps) > 1.)))
print('delta = {}, 1 + delta > 1 is {}'.format(delta,((1. + delta) > 1.)))
x = 1+delta  -1
y = 1 - 1 + delta
print()
print('1 + delta - 1 = {}'.format(x))
print('1 - 1 + delta = {}'.format(y))
```

### Example 2: Catastrophic cancellation

Let us examine what happens when we add two numbers $x$ and $y$ where $x + y \neq 0$.  We can actually estimate these bounds by doing some error analysis.  Here we need to introduce the idea that each floating point operation introduces an error such that

$$
    \text{fl}(x ~\text{op}~ y) = (x ~\text{op}~ y) (1 + \delta)
$$

where $\text{fl}(\cdot)$ is a function that returns the floating point representation of the expression enclosed, $\text{op}$ is some operation (e.g. $+, -, \times, /$), and $\delta$ is the floating point error due to $\text{op}$.

Back to our problem at hand.  The floating point error due to addition is

$$
    \text{fl}(x + y) = (x + y) (1 + \delta).
$$

Comparing this to the true solution using a relative error we have

$$\begin{aligned}
    \frac{(x + y) - \text{fl}(x + y)}{x + y} &= \frac{(x + y) - (x + y) (1 + \delta)}{x + y} = \delta.
\end{aligned}$$

so that if $\delta = \mathcal{O}(\epsilon_{\text{machine}})$ we are not too concerned.

What if instead we consider a floating point error on the representations of $x$ and $y$, $x \neq y$, and say $\delta_x$ and $\delta_y$ are the magnitude of the errors in their representation.  Here we will assume this constitutes the floating point error rather than being associated with the operation itself.

Given all of this we then would have

$$\begin{aligned}
    \text{fl}(x + y) &= x (1 + \delta_x) + y (1 + \delta_y) \\
    &= x + y + x \delta_x + y \delta_y \\
    &= (x + y) \left(1 + \frac{x \delta_x + y \delta_y}{x + y}\right)
\end{aligned}$$

Again computing the relative error we then have

$$\begin{aligned}
    \frac{x + y - (x + y) \left(1 + \frac{x \delta_x + y \delta_y}{x + y}\right)}{x + y} &= 1 - \left(1 + \frac{x \delta_x + y \delta_y}{x + y}\right) \\
    &= \frac{x}{x + y} \delta_x + \frac{y}{x + y} \delta_y \\
    &= \frac{1}{x + y} (x \delta_x + y \delta_y)
\end{aligned}$$

The important distinction here is that now the error is dependent on the values of $x$ and $y$ and more importantly, their sum.  Of particular concern is the relative size of $x + y$.  As it approaches zero relative to the magnitudes of $x$ and $y$ the error could be arbitrarily large.  This is known as **catastrophic cancellation**.

```python
dx = numpy.array([10**(-n) for n in range(1, 16)])
x = 1.0 + dx
y = -numpy.ones(x.shape)
error = numpy.abs(x + y - dx) / (dx)
```

```python
fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2)

axes = fig.add_subplot(1, 2, 1)
axes.loglog(dx, x + y, 'o-')
axes.set_xlabel("$\Delta x$")
axes.set_ylabel("$x + y$")
axes.set_title("$\Delta x$ vs. $x+y$")

axes = fig.add_subplot(1, 2, 2)
axes.loglog(dx, error, 'o-')
axes.set_xlabel("$\Delta x$")
axes.set_ylabel("$|x + y - \Delta x| / \Delta x$")
axes.set_title("Difference between $x$ and $y$ vs. Relative Error")

plt.show()
```

### Example 3: Function Evaluation

Consider the function
$$
    f(x) = \frac{1 - \cos x}{x^2}
$$
with $x\in[-10^{-4}, 10^{-4}]$.

Taking the limit as $x \rightarrow 0$ we can see what behavior we would expect to see from evaluating this function:
$$
    \lim_{x \rightarrow 0} \frac{1 - \cos x}{x^2} = \lim_{x \rightarrow 0} \frac{\sin x}{2 x} = \lim_{x \rightarrow 0} \frac{\cos x}{2} = \frac{1}{2}.
$$

What does floating point representation do?

```python
x = numpy.linspace(-1e-3, 1e-3, 100, dtype=numpy.float32)
error = (0.5 - (1.0 - numpy.cos(x)) / x**2) / 0.5

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, error, 'o')
axes.set_xlabel("x")
axes.grid()
axes.set_ylabel("Relative Error")
plt.show()
```

### Example 4: Evaluation of a Polynomial

   $$f(x) = x^7 - 7x^6 + 21 x^5 - 35 x^4 + 35x^3-21x^2 + 7x - 1$$

Note: $f(1) = 0$ (and will be close to zero for $x\approx 1$)

Here we compare polynomial evaluation using naive powers compared to Horner's method as implemented in `eval_poly(p,x)` defined above.

```python
x = numpy.linspace(0.988, 1.012, 1000, dtype=numpy.float16)
y = x**7 - 7.0 * x**6 + 21.0 * x**5 - 35.0 * x**4 + 35.0 * x**3 - 21.0 * x**2 + 7.0 * x - 1.0

# repeat using Horner's method from above
p = numpy.array([1, -7, 21, -35, 35, -21, 7, -1 ])
yh = eval_poly(p,x)
```

```python
fig = plt.figure(figsize=(8,6))
fig.set_figwidth(fig.get_figwidth() * 2)

axes = fig.add_subplot(1, 2, 1)
axes.plot(x, y, 'r',label='naive')
axes.plot(x, yh, 'b',label='horner')
axes.set_xlabel("x")
axes.set_ylabel("y")
axes.set_ylim((-0.1, 0.1))
axes.set_xlim((x[0], x[-1]))
axes.grid()
axes.legend()

axes = fig.add_subplot(1, 2, 2)
axes.plot(x,yh-y,'g')
axes.grid()
axes.set_xlabel('x')
axes.set_ylabel('$f_{horner} - f_n$')
axes.set_title('error')
plt.show()
```

### Example 5: Rational Function Evaluation
Compute $f(x) = x + 1$ by the function $$F(x) = \frac{x^2 - 1}{x - 1}$$.

Do you expect there to be issues?

```python
x = numpy.linspace(0.5, 1.5, 101, dtype=numpy.float16)
f_hat = (x**2 - 1.0) / (x - 1.0)
f = (x + 1.0)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, numpy.abs(f - f_hat)/numpy.abs(f))
axes.set_xlabel("$x$")
axes.set_ylabel("Relative Error")
plt.show()
```

## Combination of Error

In general we need to concern ourselves with the combination of both discretization error and floating point error.

Discretization error:  Errors arising from approximation of a function, truncation of a series...

$$\sin x \approx x - \frac{x^3}{3!} + \frac{x^5}{5!} + O(x^7)$$

Floating-point Error:  Errors arising from approximating real numbers with finite-precision numbers

$$\pi \approx 3.14$$

or $\frac{1}{3} \approx 0.333333333$ in decimal, results form finitely number of registers to represent each number.

### Example 1

Consider the finite difference approximation where $f(x) = e^x$ and we are evaluating at $x=1$?

$$f'(x) \approx \frac{f(x + \Delta x) - f(x)}{\Delta x}$$

Compare the error between decreasing $\Delta x$ and the true solution $f'(1) = e$.

```python
delta_x = numpy.array([2.0**(-n) for n in range(1, 60)])
x = 1.0
f_hat_1 = (numpy.exp(x + delta_x) - numpy.exp(x)) / (delta_x)
f_hat_2 = (numpy.exp(x + delta_x) - numpy.exp(x - delta_x)) / (2.0 * delta_x)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.loglog(delta_x, numpy.abs(f_hat_1 - numpy.exp(1)), 'o-', label="One-Sided")
axes.loglog(delta_x, numpy.abs(f_hat_2 - numpy.exp(1)), 's-', label="Centered")
axes.legend(loc=3)
axes.set_xlabel("$\Delta x$")
axes.set_ylabel("Absolute Error")
plt.show()
```

### Example 2

Evaluate $e^x$ with its Taylor series.

$$e^x = \sum^\infty_{n=0} \frac{x^n}{n!}$$

Can we pick $N < \infty$ that can approximate $e^x$ over a give range $x \in [a,b]$ such that the relative error $E$ satisfies $E < 8 \cdot \varepsilon_{\text{machine}}$?

We can try simply evaluating the Taylor polynomial directly for various $N$

```python
from scipy.special import factorial

def my_exp(x, N=10):
    value = 0.0
    for n in range(N + 1):
        value += x**n / numpy.float(factorial(n))

    return value
```

And test this

```python
eps = numpy.finfo(float).eps

x = numpy.linspace(-2., 50., 100, dtype=numpy.float)
for N in range(1, 300):
    error = numpy.abs((numpy.exp(x) - my_exp(x, N=N)) / numpy.exp(x))
    if numpy.all(error < 8.0 * eps):
        break
```

```python
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, error/eps)
axes.set_xlabel("x")
axes.set_ylabel("Relative Error/eps")
axes.set_title('N = {} terms'.format(N))
plt.show()
```

### Can we do better?

Note:

```python
print(numpy.log(numpy.finfo(float).max))
print(numpy.exp(709))
print(numpy.exp(-709))
```

Your homework:  the great Exp Challenge

### Example 3 - Relative Error

Say we wanted to compute the relative error between two values $x$ and $y$ using $x$ as the normalizing value.  Algebraically the forms
$$
    E = \frac{x - y}{x}
$$
and
$$
    E = 1 - \frac{y}{x}
$$
are equivalent.  In finite precision what form might be expected to be more accurate and why?

*Example based on a [blog](https://nickhigham.wordpress.com/2017/08/14/how-and-how-not-to-compute-a-relative-error/) post by Nick Higham*

Using this model the original definition contains two floating point operations such that
$$\begin{aligned}
    E_1 = \text{fl}\left(\frac{x - y}{x}\right) &= \text{fl}(\text{fl}(x - y) / x) \\
    &= \left[ \frac{(x - y) (1 + \delta_1)}{x} \right ] (1 + \delta_2) \\
    &= \frac{x - y}{x}  (1 + \delta_1) (1 + \delta_2)
\end{aligned}$$

For the other formulation we have
$$\begin{aligned}
    E_2 = \text{fl}\left( 1 - \frac{y}{x} \right ) &= \text{fl}\left(1 - \text{fl}\left(\frac{y}{x}\right) \right) \\
    &= \left(1 - \frac{y}{x} (1 + \delta_1) \right) (1 + \delta_2)
\end{aligned}$$

If we assume that all $\text{op}$s have similar error magnitudes then we can simplify things by letting
$$
    |\delta_\ast| \le \epsilon.
$$

To compare the two formulations we again use the relative error between the true relative error $e_i$ and our computed versions $E_i$.

Original definition:
$$\begin{aligned}
    \frac{e - E_1}{e} &= \frac{\frac{x - y}{x} - \frac{x - y}{x}  (1 + \delta_1) (1 + \delta_2)}{\frac{x - y}{x}} \\
    &\le 1 - (1 + \epsilon) (1 + \epsilon) = 2 \epsilon + \epsilon^2
\end{aligned}$$

Manipulated definition:

$$\begin{aligned}
    \frac{e - E_2}{e} &= \frac{e - \left[1 - \frac{y}{x}(1 + \delta_1) \right] (1 + \delta_2)}{e} \\
    &= \frac{e - \left[e - \frac{y}{x} \delta_1 \right] (1 + \delta_2)}{e} \\
    &= \frac{e - \left[e + e\delta_2 - \frac{y}{x} \delta_1 - \frac{y}{x} \delta_1 \delta_2)) \right] }{e} \\
    &= - \delta_2 + \frac{1}{e} \frac{y}{x} \left(\delta_1 + \delta_1 \delta_2 \right) \\
    &= - \delta_2 + \frac{1 -e}{e} \left(\delta_1 + \delta_1 \delta_2 \right) \\
    &\le \epsilon + \left |\frac{1 - e}{e}\right | (\epsilon + \epsilon^2)
\end{aligned}$$

We see then that our floating point error will be dependent on the relative magnitude of $e$

#### Comparison of Relative Errors of estimates of Relative Error ;^)

```python
# Based on the code by Nick Higham
# https://gist.github.com/higham/6f2ce1cdde0aae83697bca8577d22a6e
# Compares relative error formulations using single precision and compared to double precision

N = 501    # Note: Use 501 instead of 500 to avoid the zero value
d = numpy.finfo(numpy.float32).eps * 1e4
a = 3.0
x = a * numpy.ones(N, dtype=numpy.float32)
y = [x[i] + numpy.multiply((i - numpy.divide(N, 2.0, dtype=numpy.float32)), d, dtype=numpy.float32) for i in range(N)]

# Compute errors and "true" error
relative_error = numpy.empty((2, N), dtype=numpy.float32)
relative_error[0, :] = numpy.abs(x - y) / x
relative_error[1, :] = numpy.abs(1.0 - y / x)
exact = numpy.abs( (numpy.float64(x) - numpy.float64(y)) / numpy.float64(x))

# Compute differences between error calculations
error = numpy.empty((2, N))
for i in range(2):
    error[i, :] = numpy.abs((relative_error[i, :] - exact) / numpy.abs(exact))

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.semilogy(y, error[0, :], '.', markersize=10, label="$|x-y|/|x|$")
axes.semilogy(y, error[1, :], '.', markersize=10, label="$|1-y/x|$")

axes.grid(True)
axes.set_xlabel("y")
axes.set_ylabel("Relative Error")
axes.set_xlim((numpy.min(y), numpy.max(y)))
axes.set_ylim((5e-9, numpy.max(error[1, :])))
axes.set_title("Relative Error Comparison")
axes.legend()
plt.show()
```

Some other links that might be helpful regarding IEEE Floating Point:
 - [What Every Computer Scientist Should Know About Floating-Point Arithmetic](http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html)
 - [IEEE 754 Floating Point Calculator](http://babbage.cs.qc.edu/courses/cs341/IEEE-754.html)
 - [Numerical Computing with IEEE Floating Point Arithmetic](http://epubs.siam.org/doi/book/10.1137/1.9780898718072)

## Operation Counting

Discretization Error:  **Why not use more terms in the Taylor series?**

Floating Point Error: **Why not use the highest precision possible?**

### Example 1: Matrix-Vector Multiplication

Let $A, B \in \mathbb{R}^{N \times N}$ and $x \in \mathbb{R}^N$.

1. Count the approximate number of operations it will take to compute $A x$.
2. Do the same for $A B$.

Matrix-vector product:  Defining $[A]_i$ as the $i$th row of $A$ and $A_{ij}$ as the $i$, $j$th entry then
$$
    A x = \sum^N_{i=1} [A]_i \cdot x = \sum^N_{i=1} \sum^N_{j=1} A_{ij} x_j
$$

Take an explicit case, say $N = 3$, then the operation count is
$$
    A x = [A]_1 \cdot v + [A]_2 \cdot v + [A]_3 \cdot v = \begin{bmatrix}
        A_{11} \times v_1 + A_{12} \times v_2 + A_{13} \times v_3 \\
        A_{21} \times v_1 + A_{22} \times v_2 + A_{23} \times v_3 \\
        A_{31} \times v_1 + A_{32} \times v_2 + A_{33} \times v_3
    \end{bmatrix}
$$

This leads to 15 operations (6 additions and 9 multiplications).

Take another case, say $N = 4$, then the operation count is
$$
    A x = [A]_1 \cdot v + [A]_2 \cdot v + [A]_3 \cdot v = \begin{bmatrix}
        A_{11} \times v_1 + A_{12} \times v_2 + A_{13} \times v_3 + A_{14} \times v_4 \\
        A_{21} \times v_1 + A_{22} \times v_2 + A_{23} \times v_3 + A_{24} \times v_4 \\
        A_{31} \times v_1 + A_{32} \times v_2 + A_{33} \times v_3 + A_{34} \times v_4 \\
        A_{41} \times v_1 + A_{42} \times v_2 + A_{43} \times v_3 + A_{44} \times v_4 \\
    \end{bmatrix}
$$

This leads to 28 operations (12 additions and 16 multiplications).

Generalizing this there are $N^2$ multiplications and $N (N -1)$ additions for a total of

$$
    \text{operations} = N (N - 1) + N^2 = \mathcal{O}(N^2).
$$

Matrix-Matrix product ($AB$):  Defining $[B]_j$ as the $j$th column of $B$ then
$$
    (A B)_{ij} = \sum^N_{i=1} \sum^N_{j=1} [A]_i \cdot [B]_j
$$
The inner product of two vectors is represented by
$$
    a \cdot b = \sum^N_{i=1} a_i b_i
$$
leading to $\mathcal{O}(3N)$ operations.  Since there are $N^2$ entries in the resulting matrix then we would have $\mathcal{O}(N^3)$ operations.

There are methods for performing matrix-matrix multiplication faster.  In the following figure we see a collection of algorithms over time that have been able to bound the number of operations in certain circumstances.  Here
$$
    \mathcal{O}(N^\omega)
$$


# Advection

We start our study with the scalar, linear hyperbolic PDE: the advection equation.  In this chapter we consider one of the most simple hyperbolic conservation laws, the scalar constant-coefficient advection equation. The solution to this equation simply consists of the initial condition propagating with constant velocity. This equation is extremely important to understand before tackling linear hyerbolic systems, which can be solved by decoupling the system into a set of advection equations. We first describe the physical origin of this equation and then investigate the solution to its Riemann problem. This equation is studied in more detail in Chapter 2 of (LeVeque, 2002).

## Conservation of mass

Imagine a fluid flowing in a long, narrow tube.  We'll use $q$ to indicate the density of the fluid and $u$ to indicate its velocity.  Both of these are functions of space and time: $q = q(x,t)$; $u=u(x,t)$.  The total mass in the section of tube $[x_1,x_2]$ is
\begin{equation*}
\int_{x_1}^{x_2} q(x,t) dx.
\end{equation*}
This total mass can change in time due to fluid flowing in or out of this section of the tube.  We call the rate of flow the *flux*, and represent it with the function $f(q)$.  Thus the net rate of flow of mass into (or out of) the interval $[x_1,x_2]$ at time $t$ is
$$f(q(x_1,t)) - f(q(x_2,t)).$$
We just said that this rate of flow must equal the time rate of change of total mass; i.e.
$$\frac{d}{dt} \int_{x_1}^{x_2} q(x,t) dx = f(q(x_1,t)) - f(q(x_2,t)).$$
Now since $\int_{x_1}^{x_2} \frac{\partial}{\partial x} f(q) dx = f(q(x_2,t)) - f(q(x_1,t))$, we can rewrite this as
$$\frac{d}{dt} \int_{x_1}^{x_2} q(x,t) dx = -\int_{x_1}^{x_2} \frac{\partial}{\partial x} f(q) dx.$$
Under certain smoothness assumptions on $q$, we can move the time derivative inside the integral.  We'll also put everything on the left side, to obtain
$$\int_{x_1}^{x_2} \left(\frac{\partial}{\partial t}q(x,t) + \frac{\partial}{\partial x} f(q)\right) dx = 0.$$
Since this integral is zero for *any* choice of $x_1,x_2$, it must be that the integrand (the expression in parentheses) is actually zero *everywhere*!  Therefore we can write the **differential conservation law**
$$q_t + f(q)_x = 0.$$
This equation expresses the fact that the total mass is conserved, since locally the mass can change only due to a net inflow or outflow.

## Advection
In order to solve the conservation law above, we need an expression for the flux, $f$.  The rate of flow is just mass times velocity: $f=u q$.  Thus we obtain the **continuity equation**
$$q_t + (uq)_x = 0.$$

In general, we need another equation to determine the velocity $u(x,t)$.  In later chapters we'll examine models in which the velocity depends on the density (or other properties), but for now let's consider the simplest case, in which all of the fluid flows at a single, constant velocity $u(x,t)=a$.  Then the continuity equation becomes the **advection equation**
\begin{align} \label{adv:advection}
    q_t + a q_x = 0.
\end{align}
This equation has a very simple solution.  If we are given the density $q(x,0)=q_0(x)$ at time zero, then the solution is just
\begin{align} \label{adv:solution}
    q(x,t) = q_0(x-at).
\end{align}

```python
%matplotlib inline
```

```python
%config InlineBackend.figure_format = 'svg'
from ipywidgets import interact
from exact_solvers import advection
```

To examine the Python code for this chapter, see:

 - [exact_solvers/advection.py](exact_solvers/advection.py)

## Characteristics
Notice that the solution is constant along the line $x-at=C$, for each value of $C$.  These are parallel, straight lines in the $x-t$ plane, with slope $1/a$, as shown in the figure below.

```python
interact(advection.characteristics);
```

We can think of the initial values $q_0(x)$ being transmitted along these lines; we sometimes say that *information* is transmitted along characteristics.

The next figure shows how an initial hump propagates to the right along characteristics.

```python
interact(advection.solution);
```

For more complicated hyperbolic problems, we may have multiple sets of characteristics, they may not be parallel, and the solution may not be constant along them.  But it will still be the case that information is propagated along characteristics.  The idea that information propagates at finite speed is an essential property of hyperbolic PDEs.

## The Riemann problem
The Riemann problem consists of a hyperbolic PDE, such as (\ref{adv:advection}), together with piecewise constant initial data consisting of two states.  For convenience, we place the interface (or jump) at $x=0$ and refer to the left state (for $x<0$) as $q_l$ and the right state (for $x>0$) as $q_r$.  We thus have
\begin{align*}
    q_0(x) & = \begin{cases} q_l & x<0 \\ q_r & x>0. \end{cases}
\end{align*}
For the advection equation, the solution to the Riemann problem is immediately obvious; it is simply a special case of (\ref{adv:solution}).  The discontinuity initially at $x=0$ moves at speed $a$.  We have
\begin{align*}
    q(x,t) & = \begin{cases} q_l & x-at<0 \\ q_r & x-at>0. \end{cases}
\end{align*}

```python
interact(advection.riemann_demo);
```

Notice how the initial discontinuity follows the characteristic coming from $x=0$ at $t=0$.

Here's another way to look at the Riemann solution that will be very useful for more complicated equations.

```python
q_l = 1.
q_r = 0.
advection.plot_riemann_solution(q_l, q_r, a=1.);
```

The left plot shows (again) the characteristics in the x-t plane; the characteristic along which the discontinuity propagates is darker and thicker.  The right plot shows the solution.  Try modifying the initial states and the velocity $a$ to see how this simple solution changes.


```python
from IPython.display import HTML
from matplotlib.animation import FuncAnimation
from scipy.integrate import solve_ivp
import matplotlib.pyplot as plt
import numpy as np
import sympy as sm
import sympy.physics.mechanics as me

me.init_vprinting(use_latex='mathjax')
```

```python
q1, q2, q3 = me.dynamicsymbols('q1, q2, q3')
u1, u2, u3 = me.dynamicsymbols('u1, u2, u3')
la, lb, lc, ln = sm.symbols('l_a, l_b, l_c, l_n')
m, g = sm.symbols('m, g')
t = me.dynamicsymbols._t

p = sm.Matrix([la, lb, lc, ln, m, g])

q = sm.Matrix([q1])
qr = sm.Matrix([q2, q3])
qN = q.col_join(qr)

u = sm.Matrix([u1])
ur = sm.Matrix([u2, u3])
uN = u.col_join(ur)

qdN = qN.diff(t)
ud = u.diff(t)

p, q, qr, qN, u, ur, uN, qdN, ud
```

```python
N = me.ReferenceFrame('N')
A = me.ReferenceFrame('A')
B = me.ReferenceFrame('B')
C = me.ReferenceFrame('C')

A.orient_axis(N, q1, N.z)
B.orient_axis(A, q2, A.z)
C.orient_axis(B, q3, B.z)

P1 = me.Point('P1')
P2 = me.Point('P2')
P3 = me.Point('P3')
P4 = me.Point('P4')

P2.set_pos(P1, la*A.x)
P3.set_pos(P2, lb*B.x)
P4.set_pos(P3, lc*C.x)
```

```python
loop = P4.pos_from(P1) - ln*N.x

fh = sm.Matrix([loop.dot(N.x), loop.dot(N.y)])
fh = sm.trigsimp(fh)
fh
```

```python
fk = uN - qdN
fk
```

```python
Mk = fk.jacobian(qdN)
gk = uN
qd_sol = -Mk.LUsolve(gk)
qd_repl = dict(zip(qdN, qd_sol))
qd_repl
```

```python
fhd = fh.diff(t).xreplace(qd_repl)
fhd
```

```python
me.find_dynamicsymbols(fhd)
```

```python
Mhd = fhd.jacobian(ur)
ur_zero = {uri: 0 for uri in ur}
ghd = fhd.xreplace(ur_zero)
ur_sol = -Mhd.LUsolve(ghd)
ur_repl = dict(zip(ur, ur_sol))
```

```python
ur_repl
```

```python
A.set_ang_vel(N, u1*N.z)
B.set_ang_vel(A, ur_repl[u2]*A.z)
C.set_ang_vel(B, ur_repl[u3]*B.z)
```

```python
P1.set_vel(N, 0)
P2.v2pt_theory(P1, N, A)
P3.v2pt_theory(P2, N, B)
P4.v2pt_theory(P3, N, C)
```

```python
me.find_dynamicsymbols(P4.vel(N), reference_frame=N)
```

```python
gk = gk.xreplace(ur_repl)
```

```python
me.find_dynamicsymbols(gk)
```

```python
R_P2 = -m*g*N.y
R_P3 = -m*g*N.y
```

```python
F1 = P2.vel(N).diff(u1, N).dot(R_P2) + P3.vel(N).diff(u1, N).dot(R_P3)
F1
```

```python
me.find_dynamicsymbols(P3.acc(N).xreplace(qd_repl).xreplace(ur_repl), reference_frame=N)
```

```python
N_a_P2 = P2.acc(N).xreplace(qd_repl).xreplace(ur_repl)
N_a_P3 = P3.acc(N).xreplace(qd_repl).xreplace(ur_repl)
```

```python
Rs_P2 = -m*N_a_P2
Rs_P3 = -m*N_a_P3
```

```python
F1s = P2.vel(N).diff(u1, N).dot(Rs_P2) + P3.vel(N).diff(u1, N).dot(Rs_P3)
```

```python
me.find_dynamicsymbols(F1)
```

```python
me.find_dynamicsymbols(F1s)
```

```python
fd = F1 + F1s
```

```python
fh
```

```python
fk.xreplace(ur_repl)
```

```python
p_vals = np.array([
    0.8,  # la [m]
    2.0,  # lb [m]
    1.0,  # lc [m]
    2.0,  # ln [m]
    1.0,  # m [kg]
    9.81,  # g [m/s^2]
])

```

```python
q1_0 = np.deg2rad(10.0)
u1_0 = 0.0
```

```python
from scipy.optimize import fsolve
```

```python
fsolve?
```

```python
eval_fh = sm.lambdify((qN, p), fh)
```

```python
eval_fh([1.0, 2.0, 3.0], p_vals)
```

```python
def fsolve_fh(qr, q1, p):
    return np.squeeze(eval_fh(np.hstack((q1, qr)), p))
```

```python
fsolve_fh([2.0, 3.0], 1.0, p_vals)
```

```python
qr_guess = np.deg2rad([10.0, -150.0])
```

```python
q2_0, q3_0 = fsolve(fsolve_fh, qr_guess, args=(q1_0, p_vals), xtol=1e-14)
```

```python
q1_0, q2_0, q3_0
```

```python
eval_fh([q1_0, q2_0, q3_0], p_vals)
```

```python
me.find_dynamicsymbols(fd)
```

```python
Md = fd.diff(u1.diff())
gd = fd.xreplace({u1.diff(): 0})
u1_sol = -gd/Md
eval_u1 = sm.lambdify((u1, qN, p), u1_sol)
```

```python
u1d_0 = eval_u1(u1_0, [q1_0, q2_0, q3_0], p_vals)
u1d_0
```

```python
q1_0, q2_0, q3_0, u1_0, u1d_0
```

```python
from scikits.odes import dae
```

```python
dae?
```

```python
u1d = sm.symbols('u1d')
eval_fd = sm.lambdify((qN, u1, u1d, p), fd.xreplace({u1.diff(): u1d}))
eval_fd([q1_0, q2_0, q3_0], 1.0, 3.0, p_vals)
```

```python
eval_fh?
```

```python
def eval_eom(t, x, xd, residual, p):
    qN = x[:3]
    u1 = x[3]
    q1d = xd[0]
    u1d = xd[3]

    residual[0:2] = eval_fh(qN, p).squeeze()  # shape(2,)
    residual[2] = u1 - q1d  # shape(1,)
    residual[3] = eval_fd(qN, u1, u1d, p).squeeze()  # shape(1,)
```

```python
res = np.ones(4)
res
```

```python
eval_eom(1.0, # t
         [q1_0, q2_0, q3_0, u1_0], # x: q1, q2, q3, u1
         [0.0, 0.0, 0.0, u1d_0], # xd: q1d, q2q, q3d, u1d
         res,
         p_vals)
```

```python
res
```

```python
solver = dae('ida',  # which solver to use
             eval_eom,
             rtol=1e-10,
             atol=1e-10,
             algebraic_vars_idx=[0, 1],
             user_data=p_vals,
             old_api=False)  # gives us new output format
```

```python
x0 = np.array([q1_0, q2_0, q3_0, u1_0])
xd0 = np.array([0.0, 0.0, 0.0, u1d_0])
ts = np.linspace(0.0, 4.0, num=401)
```

```python
solution = solver.solve(ts, x0, xd0)
```

```python
solution.values.y
```

```python
fig, ax = plt.subplots()

ax.plot(solution.values.t, solution.values.y)
```

```python
def eval_constraints(xs, p):
    """Returns the value of the left hand side of the holonomic constraints
    at each time instance.

    Parameters
    ==========
    xs : ndarray, shape(n, 4)
        States at each of n time steps.
    p : ndarray, shape(6,)
        Constant parameters.

    Returns
    =======
    con : ndarray, shape(n, 2)
        fh evaluated at each xi in xs.

    """
    con = []
    for xi in xs:  # xs is shape(n, 4)
        con.append(eval_fh(xi[:3], p).squeeze())
    return np.array(con)
```

```python
con = eval_constraints(solution.values.y, p_vals)
```

```python
def plot_results(ts, xs, con):
    """Returns the array of axes of a 4 panel plot of the state trajectory
    versus time.

    Parameters
    ==========
    ts : array_like, shape(n,)
       Values of time.
    xs : array_like, shape(n, 4)
       Values of the state trajectories corresponding to ``ts`` in order
       [q1, q2, q3, u1].
    con : array_like, shape(n, 2)
       x and y constraint residuals of P4 at each time in ``ts``.

    Returns
    =======
    axes : ndarray, shape(3,)
       Matplotlib axes for each panel.

    """
    fig, axes = plt.subplots(3, 1, sharex=True)

    fig.set_size_inches((10.0, 6.0))

    axes[0].plot(ts, np.rad2deg(xs[:, :3]))  # q1(t), q2(t), q3(t)
    axes[1].plot(ts, np.rad2deg(xs[:, 3]))  # u1(t)
    axes[2].plot(ts, np.squeeze(con))  # fh(t)

    axes[0].legend(['$q_1$', '$q_2$', '$q_3$'])
    axes[1].legend(['$u_1$'])
    axes[2].legend([r'$\cdot\hat{n}_x$', r'$\cdot\hat{n}_y$'])

    axes[0].set_ylabel('Angle [deg]')
    axes[1].set_ylabel('Angular Rate [deg/s]')
    axes[2].set_ylabel('Distance [m]')
    axes[2].set_xlabel('Time [s]')

    fig.tight_layout()

    return axes

```

```python
plot_results(solution.values.t, solution.values.y, con)
```

```python
coordinates = P2.pos_from(P1).to_matrix(N)
for point in [P3, P4, P1, P2]:
    coordinates = coordinates.row_join(point.pos_from(P1).to_matrix(N))
eval_point_coords = sm.lambdify((qN, p), coordinates)
```

```python
def setup_animation_plot(ts, xs, p):
    """Returns objects needed for the animation.

    Parameters
    ==========
    ts : array_like, shape(n,)
       Values of time.
    xs : array_like, shape(n, 4)
       Values of the state trajectories corresponding to ``ts`` in order
       [q1, q2, q3, u1].
    p : array_like, shape(6,)

    """

    x, y, z = eval_point_coords(xs[0, :3], p)

    fig, ax = plt.subplots()
    fig.set_size_inches((10.0, 10.0))
    ax.set_aspect('equal')
    ax.grid()

    lines, = ax.plot(x, y, color='black',
                     marker='o', markerfacecolor='blue', markersize=10)

    title_text = ax.set_title('Time = {:1.1f} s'.format(ts[0]))
    ax.set_xlim((-1.0, 3.0))
    ax.set_ylim((-1.0, 1.0))
    ax.set_xlabel('$x$ [m]')
    ax.set_ylabel('$y$ [m]')

    return fig, ax, title_text, lines
```

```python
def animate_linkage(ts, xs, p):
    """Returns an animation object.

    Parameters
    ==========
    ts : array_like, shape(n,)
    xs : array_like, shape(n, 4)
       x = [q1, q2, q3, u1]
    p : array_like, shape(6,)
       p = [la, lb, lc, ln, m, g]

    """
    # setup the initial figure and axes
    fig, ax, title_text, lines = setup_animation_plot(ts, xs, p)

    # precalculate all of the point coordinates
    coords = []
    for xi in xs:
        coords.append(eval_point_coords(xi[:3], p))
    coords = np.array(coords)

    # define the animation update function
    def update(i):
        title_text.set_text('Time = {:1.1f} s'.format(ts[i]))
        lines.set_data(coords[i, 0, :], coords[i, 1, :])

    # close figure to prevent premature display
    plt.close()

    # create and return the animation
    return FuncAnimation(fig, update, len(ts))

```

```python
xs = solution.values.y
fps = 100
HTML(animate_linkage(ts, xs, p_vals).to_jshtml(fps=fps))
```

```python

```


# Занятие 7
# Прикладная алгебра и численные методы
## Многочлены Чебышева

```
!python -m pip install sympy --upgrade
```

    Collecting sympy
      Downloading https://files.pythonhosted.org/packages/ff/69/b16fc81b939d3efdd0b552f2e3e54f7fa1423d0c320cced2e69e675dde26/sympy-1.7.1-py3-none-any.whl (5.9MB)
         |████████████████████████████████| 5.9MB 6.4MB/s
    Requirement already satisfied, skipping upgrade: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy) (1.2.1)
    Installing collected packages: sympy
      Found existing installation: sympy 1.1.1
        Uninstalling sympy-1.1.1:
          Successfully uninstalled sympy-1.1.1
    Successfully installed sympy-1.7.1

```
!python -m pip install numpy --upgrade
```

    Requirement already up-to-date: numpy in /usr/local/lib/python3.6/dist-packages (1.19.5)

```
import numpy as np
import sympy
from sympy import S
from sympy.functions.special.polynomials import chebyshevt, chebyshevu
from numpy.polynomial.chebyshev import chebinterpolate
import matplotlib.pyplot as plt
import pandas as pd
from google.colab import files
%matplotlib inline
```

```
import sympy
import numpy
sympy.__version__, numpy.__version__
```

    ('1.7.1', '1.19.5')

https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.norm.html#scipy.linalg.norm

https://numpy.org/doc/stable/reference/generated/numpy.linalg.norm.html

## Многочлен Чебышева первого рода
$T_0(x) = 1$, $T_1(x) = x$, $T_(x) = 2xT_{n - 1}(x) - T_{n - 2}(x)$, $n\ge 2$.
## Пример 1
Построим графики многочленов Чебышева первого рода при $n = 4, 5$.

```
x = S('x')
def T(n, x):
  if n == 0:
    return 1
  if n == 1:
    return x
  return sympy.expand(sympy.simplify(2*x*T(n - 1, x) - T(n - 2, x)))

display(T(4, x), T(5, x))
p = sympy.plot(T(4, x), (x, -1, 1), line_color='green', show=False)
p.extend(sympy.plot(T(5, x), (x, -1, 1), line_color='red', show=False))
p.show()
```

## Пример 2.
$$
T_n(x) = \frac{(x + \sqrt{x^2 - 1})^n + (x - \sqrt{x^2 - 1})^n}{2}, \quad |x| \ge 1.
$$
Построим многочлен Чебышева первого рода порядка 15 по приведенной выше формуле и по рекурсивной формуле Примера 1.

На сетке значений $x$ от 2 до 3 с шагом 0.001 вычислить нормы разностей значений многочлена Чебышева, полученных двумя способами.

```
def T_new(n, x):
  return ((x + sympy.sqrt(x**2 - 1))**n + (x - sympy.sqrt(x**2 - 1))**n)/2

X = np.linspace(2, 3, 1001)
Y1 = sympy.lambdify(x, T(15, x))(X)
Y2 = sympy.lambdify(x, T_new(15, x))(X)

print(*[np.linalg.norm(Y1 - Y2, k) for k in [1, 2, 3]])
plt.plot(X, Y1, 'g-', label=sympy.latex(sympy.Eq(S('T15(x)'), T(15, x)), mode='inline'))
plt.plot(X, Y2, 'r:', label=sympy.latex(sympy.Eq(S('newT15(x)'), T_new(15, x)), mode='inline'))
plt.legend()
```

## Пример 3
Многочлен Чебышева второго рода:
$$
U_n = \frac{1}{n + 1}T'_{n + 1}(x), \quad n \ge 0.
$$
$$
U_n(x) = \frac{(x + \sqrt{x^2 - 1})^{n + 1} - (x - \sqrt{x^2 - 1})^{n + 1}}{2\sqrt{x^2 - 1}}, \quad |x| \ge 1.
$$

Построим многочлен Чебышева степени 7 двумя способами, сравним нормы разностей на сетке как в Примере 2.

```
def U(n, x):
    return sympy.expand(sympy.simplify(T(n + 1, x).diff(x)/(n + 1)))
def U_new(n, x):
    return sympy.expand(sympy.simplify(sympy.expand(((x + sympy.sqrt(x**2 - 1))**(n + 1) - (x - sympy.sqrt(x**2 - 1))**(n + 1)))/(2*sympy.sqrt(x**2 - 1))))
def U1(n, x):
    return sympy.expand(sympy.simplify(T_new(n + 1, x).diff(x)/(n + 1)))
display(U(7, x), U_new(7, x), U1(7, x))

```

$\displaystyle 128 x^{7} - 192 x^{5} + 80 x^{3} - 8 x$

$\displaystyle 128 x^{7} - 192 x^{5} + 80 x^{3} - 8 x$

$\displaystyle 128 x^{7} - 192 x^{5} + 80 x^{3} - 8 x$

## Многочлен Чебышева в Sympy.
В sympy.functions.special.polynomials есть chebyshevt и chebyshevu,
возвращающие соответственно многочлен Чебышева первого и второго рода, аргументы - порядок многочлена и переменная.

## Пример 4.
Построим полиномы Чебышева первого и второго рода порядка 5:

```
display(chebyshevt(5, x), chebyshevu(5, x))
```

$\displaystyle 16 x^{5} - 20 x^{3} + 5 x$

$\displaystyle 32 x^{5} - 32 x^{3} + 6 x$

## Пример 5
Норма Чебышева (максимум модуля на данном отрезке):
$$
|f|_0 = \max_{[-1, 1]}|f(x)|.
$$
Площадь под графиком функции на данном отрезке:
$$
|f|_1 = \int_{-1}^1|f(x)|\,dx.
$$
Вычислить норму Чебышева и площадь под графиком для $f(x) = \frac{x^3}{e^x}$ на отрезке [-1, 1].

```
x = S('x')

def f5(x):
  return x**3/sympy.exp(x)

f5_norm0 = sympy.calculus.util.maximum(f5(x), x, domain=sympy.Interval(-1, 1))
f5_norm1 = sympy.Abs(f5(x)).integrate((x, -1, 1))
display('|f4|0 = {0} = {1}'.format(f5_norm0, round(f5_norm0, 3)),
        '|f4|1 = {0} = {1}'.format(f5_norm1, round(f5_norm1, 3)))
```

    '|f4|0 = exp(-1) = 0.368'

    '|f4|1 = -16*exp(-1) - 2*E + 12 = 0.677'

## Пример 6
Наилучшее приближение функции $f(x)$ многочленом степени $\le n$:
$$
\tilde{f}(x) = \sum_{i=0}^n \frac{<T_i, f>}{<T_i, T_i>}T_i(x),
$$
$T_i(x)$ многочлен Чебышева первого рода степени $i$,
$$
<f, g> = \int_{-1}^1 \frac{f(x)g(x)}{\sqrt{1 - x^2}}\,dx, \quad
|f| = \sqrt{<f, f>} = \sqrt{\int_{-1}^1 \frac{f^2(x)}{\sqrt{1 - x^2}}\,dx}
$$

Построим многочлен степени не выше 3, наилучшим образом приближающий $x^5 - 1$ на [-1, 1]

```
x = S('x')
def f6(x):
    return x**5 - 1

def dot_prod_cheb(f, g, x):
    return (f*g/sympy.sqrt(1 - x**2)).integrate((x, -1, 1))

def f_cheb(f, n, x):
    res = 0
    for k in range(n + 1):
        cheb_k = chebyshevt(k, x)
        coef = dot_prod_cheb(cheb_k, f, x)/dot_prod_cheb(cheb_k, cheb_k, x)
        print(coef)
        res += coef*cheb_k
    return res
res6 = f_cheb(f6(x), 3, x)
sympy.plot((res6, (x, -1, 1)), (f6(x), (x, -1, 1)))
display(res6)
```

## Наилучшее приближение функции $f(x)$ многочленом Чебышева, Numpy.

https://numpy.org/doc/stable/reference/generated/numpy.polynomial.chebyshev.Chebyshev.html

Для построения многочлена Чебышева используем chebinterpolate из
numpy.polynomial.chebyshev

Первый (обязательный) аргумент  chebinterpolate - имя функции, второй - порядок многочлена.

Возвращает chebinterpolate коэффициенты $a_i$ при многочленах Чебышева, в порядке возрастания степени:
$$
f(x) = a_0T_0 + a_1T_1 + ... + a_nT_n.
$$
## Пример 7
Построим многочлен Чебышева для Примера 6 с помощью chebinterpolate.

```
res7 = chebinterpolate(f6, 3)
res7
```

    array([-1.00000000e+00,  6.25000000e-01,  8.32667268e-17,  2.50000000e-01])

Представим функцию $f_6$ с помощью полиномов Чебышева:

```
res7poly = res7[0]
for k in range(1, len(res7)):
    res7poly += res7[k]*chebyshevt(k, x)
display(res7poly)
```

$\displaystyle 1.0 x^{3} + 1.66533453693773 \cdot 10^{-16} x^{2} - 0.125 x - 1.0$

Изобразим на одном графике функцию $f_6$ и ее представление полиномами Чебышева, полученными с помощью Sympy и Numpy:

```
p = sympy.plot(res7poly, (x, -1, 1), line_color='green', show=False)
p.extend(sympy.plot(f6(x), (x, -1, 1), line_color='black', show=False))
p.extend(sympy.plot(res6, (x, -1, 1), line_color='red', show=False))
p.show()
```

Представления $f_6$, полученные с помощью Sympy и Numpy несколько различаются.

Сравним нормы разностей функции $f_6$ и ее разложений:

```
for item in (res7poly, res6):
    f7_norm0 = sympy.calculus.util.maximum(f6(x) - item, x, domain=sympy.Interval(-1, 1))
    f7_norm1 = sympy.Abs(f6(x) - item).integrate((x, -1, 1))
    display('|f7|0 = {0}'.format(round(f7_norm0, 3)),
        '|f7|1 = {0}'.format(round(f7_norm1, 3)))

```

    '|f7|0 = 0.125'

    '|f7|1 = 0.076'

    '|f7|0 = 0.062'

    '|f7|1 = 0.079'

Как видно, у разложения Numpy меньше максимум модуля отклонения от функции на отрезке [-1, 1], а у разложения Sympy - площадь между графиками функции и ее разложения.

Посмотрим, как Numpy и Sympy раскладывают по полиномам Чебышева $x^5 - 1$ в полином степени не больше 5 (должно получиться ровно $x^5 - 1$).

```
res10= chebinterpolate(lambda x: x**5 - 1, 5)
res10poly = res10[0]
for k in range(1, len(res10)):
    res10poly += res10[k]*chebyshevt(k, x)
res10poly
```

$\displaystyle 0.999999999999991 x^{5} + 7.99360577730113 \cdot 10^{-15} x^{4} + 9.32587340685131 \cdot 10^{-15} x^{3} - 7.38298311375729 \cdot 10^{-15} x^{2} - 1.27675647831893 \cdot 10^{-15} x - 0.999999999999999$

```
x = S('x')
def f0(x):
    return x**5 - 1

res = f_cheb(f0(x), 5, x)
sympy.plot((res, (x, -1, 1)), (f0(x), (x, -1, 1)))
display(res)
```

## Дополнение по полиномам Чебышева

```
x = S('x')

def cos_n(n, x):
    return(chebyshevt(n, sympy.cos(x)))

for k in range(5):
    display(cos_n(k, x))

```

$\displaystyle 1$

$\displaystyle \cos{\left(x \right)}$

$\displaystyle 2 \cos^{2}{\left(x \right)} - 1$

$\displaystyle 4 \cos^{3}{\left(x \right)} - 3 \cos{\left(x \right)}$

$\displaystyle 8 \cos^{4}{\left(x \right)} - 8 \cos^{2}{\left(x \right)} + 1$


# 03 - Full-Waveform Inversion (FWI)

This notebook is the third in a series of tutorial highlighting various aspects of seismic inversion based on Devito operators. In this second example we aim to highlight the core ideas behind seismic inversion, where we create an image of the subsurface from field recorded data. This tutorial follows on the modelling tutorial and will reuse the modelling and velocity model.

## Inversion requirement

Seismic inversion relies on two known parameters:

- **Field data** - or also called **recorded data**. This is a shot record corresponding to the true velocity model. In practice this data is acquired as described in the first tutorial. In order to simplify this tutorial we will fake field data by modelling it with the true velocity model.

- **Initial velocity model**. This is a velocity model that has been obtained by processing the field data. This model is a rough and very smooth estimate of the velocity as an initial estimate for the inversion. This is a necessary requirement for any optimization (method).

## Inversion computational setup

In this tutorial, we will introduce the gradient operator. This operator corresponds to the imaging condition introduced in the previous tutorial with some minor modifications that are defined by the objective function (also referred to in the tutorial series as the *functional*, *f*) and its gradient, *g*. We will define these two terms in the tutorial too.

## Notes on the operators

As we have already described the creation of a forward modelling operator, we will only call a wrapper function here. This wrapper already contains all the necessary operators for seismic modeling, imaging and inversion. Operators introduced for the first time in this tutorial will be properly described.

```python
import numpy as np
%matplotlib inline

from devito import configuration
configuration['log-level'] = 'WARNING'
```

## Computational considerations

As we will see, FWI is computationally extremely demanding, even more than RTM. To keep this tutorial as lightwight as possible we therefore again use a very small demonstration model. We also define here a few parameters for the final example runs that can be changed to modify the overall runtime of the tutorial.

```python
nshots = 9  # Number of shots to create gradient from
nreceivers = 101  # Number of receiver locations per shot
fwi_iterations = 5  # Number of outer FWI iterations
```

# True and smooth velocity models

We will use a very simple model domain, consisting of a circle within a 2D domain. We will again use the "true" model to generate our synthetic shot data and use a "smooth" model as our initial guess. In this case the smooth model is very smooth indeed - it is simply a constant background velocity without any features.

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import demo_model, plot_velocity, plot_perturbation

# Define true and initial model
shape = (101, 101)  # Number of grid point (nx, nz)
spacing = (10., 10.)  # Grid spacing in m. The domain size is now 1km by 1km
origin = (0., 0.)  # Need origin to define relative source and receiver locations

model = demo_model('circle-isotropic', vp_circle=3.0, vp_background=2.5,
                    origin=origin, shape=shape, spacing=spacing, nbl=40)

model0 = demo_model('circle-isotropic', vp_circle=2.5, vp_background=2.5,
                     origin=origin, shape=shape, spacing=spacing, nbl=40,
                     grid = model.grid)

plot_velocity(model)
plot_velocity(model0)
plot_perturbation(model0, model)
```

```python
assert model.grid == model0.grid
assert model.vp.grid == model0.vp.grid
```

## Acquisition geometry

In this tutorial, we will use the easiest case for inversion, namely a transmission experiment. The sources are located on one side of the model and the receivers on the other side. This allows to record most of the information necessary for inversion, as reflections usually lead to poor inversion results.

```python
#NBVAL_IGNORE_OUTPUT
# Define acquisition geometry: source
from examples.seismic import AcquisitionGeometry

t0 = 0.
tn = 1000.
f0 = 0.010
# First, position source centrally in all dimensions, then set depth
src_coordinates = np.empty((1, 2))
src_coordinates[0, :] = np.array(model.domain_size) * .5
src_coordinates[0, 0] = 20.  # Depth is 20m

# Define acquisition geometry: receivers

# Initialize receivers for synthetic and imaging data
rec_coordinates = np.empty((nreceivers, 2))
rec_coordinates[:, 1] = np.linspace(0, model.domain_size[0], num=nreceivers)
rec_coordinates[:, 0] = 980.

# Geometry

geometry = AcquisitionGeometry(model, rec_coordinates, src_coordinates, t0, tn, f0=f0, src_type='Ricker')
# We can plot the time signature to see the wavelet
geometry.src.show()
```

```python
#NBVAL_IGNORE_OUTPUT
# Plot acquisition geometry
plot_velocity(model, source=geometry.src_positions,
              receiver=geometry.rec_positions[::4, :])
```

## True and smooth data

We can generate shot records for the true and smoothed initial velocity models, since the difference between them will again form the basis of our imaging procedure.

```python
# Compute synthetic data with forward operator
from examples.seismic.acoustic import AcousticWaveSolver

solver = AcousticWaveSolver(model, geometry, space_order=4)
true_d, _, _ = solver.forward(vp=model.vp)
```

```python
# Compute initial data with forward operator
smooth_d, _, _ = solver.forward(vp=model0.vp)
```

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import plot_shotrecord

# Plot shot record for true and smooth velocity model and the difference
plot_shotrecord(true_d.data, model, t0, tn)
plot_shotrecord(smooth_d.data, model, t0, tn)
plot_shotrecord(smooth_d.data - true_d.data, model, t0, tn)
```

# Full-Waveform Inversion

## Formulation

Full-waveform inversion (FWI) aims to invert an accurate model of the discrete wave velocity, $\mathbf{c}$, or equivalently the square slowness of the wave, $\mathbf{m} = \frac{1}{\mathbf{c}^2}$, from a given set of measurements of the pressure wavefield $\mathbf{u}$. This can be expressed as the following optimization problem [1, 2]:

\begin{aligned}
	\mathop{\hbox{minimize}}_{\mathbf{m}} \Phi_s(\mathbf{m})&=\frac{1}{2}\left\lVert\mathbf{P}_r
	\mathbf{u} - \mathbf{d}\right\rVert_2^2 \\
	\mathbf{u} &= \mathbf{A}(\mathbf{m})^{-1} \mathbf{P}_s^T \mathbf{q}_s,
\end{aligned}

where $\mathbf{P}_r$ is the sampling operator at the receiver locations, $\mathbf{P}_s^T$ is the injection operator at the source locations, $\mathbf{A}(\mathbf{m})$ is the operator representing the discretized wave equation matrix, $\mathbf{u}$ is the discrete synthetic pressure wavefield, $\mathbf{q}_s$ is the corresponding pressure source and $\mathbf{d}$ is the measured data. It is worth noting that $\mathbf{m}$ is the unknown in this formulation and that multiple implementations of the wave equation operator $\mathbf{A}(\mathbf{m})$ are possible.

We have already defined a concrete solver scheme for $\mathbf{A}(\mathbf{m})$ in the first tutorial, including appropriate implementations of the sampling operator $\mathbf{P}_r$ and source term $\mathbf{q}_s$.

To solve this optimization problem using a gradient-based method, we use the
adjoint-state method to evaluate the gradient $\nabla\Phi_s(\mathbf{m})$:

\begin{align}
 \nabla\Phi_s(\mathbf{m})=\sum_{\mathbf{t} =1}^{n_t}\mathbf{u}[\mathbf{t}] \mathbf{v}_{tt}[\mathbf{t}] =\mathbf{J}^T\delta\mathbf{d}_s,
\end{align}

where $n_t$ is the number of computational time steps, $\delta\mathbf{d}_s = \left(\mathbf{P}_r \mathbf{u} - \mathbf{d} \right)$ is the data residual (difference between the measured data and the modelled data), $\mathbf{J}$ is the Jacobian operator and $\mathbf{v}_{tt}$ is the second-order time derivative of the adjoint wavefield solving:

\begin{align}
 \mathbf{A}^T(\mathbf{m}) \mathbf{v} = \mathbf{P}_r^T \delta\mathbf{d}.
\end{align}

We see that the gradient of the FWI function is the previously defined imaging condition with an extra second-order time derivative. We will therefore reuse the operators defined previously inside a Devito wrapper.

## FWI gradient operator

To compute a single gradient $\nabla\Phi_s(\mathbf{m})$ in our optimization workflow we again use `solver.forward` to compute the entire forward wavefield $\mathbf{u}$ and a similar pre-defined gradient operator to compute the adjoint wavefield `v`. The gradient operator provided by our `solver` utility also computes the correlation between the wavefields, allowing us to encode a similar procedure to the previous imaging tutorial as our gradient calculation:

- Simulate the forward wavefield with the background velocity model to get the synthetic data and save the full wavefield $\mathbf{u}$
- Compute the data residual
- Back-propagate the data residual and compute on the fly the gradient contribution at each time step.

This procedure is applied to multiple source positions and summed to obtain a gradient image of the subsurface. We again prepare the source locations for each shot and visualize them, before defining a single gradient computation over a number of shots as a single function.

```python
#NBVAL_IGNORE_OUTPUT

# Prepare the varying source locations sources
source_locations = np.empty((nshots, 2), dtype=np.float32)
source_locations[:, 0] = 30.
source_locations[:, 1] = np.linspace(0., 1000, num=nshots)

plot_velocity(model, source=source_locations)
```

```python
from devito import Eq, Operator

# Computes the residual between observed and synthetic data into the residual
def compute_residual(residual, dobs, dsyn):
    if residual.grid.distributor.is_parallel:
        # If we run with MPI, we have to compute the residual via an operator
        # First make sure we can take the difference and that receivers are at the
        # same position
        assert np.allclose(dobs.coordinates.data[:], dsyn.coordinates.data)
        assert np.allclose(residual.coordinates.data[:], dsyn.coordinates.data)
        # Create a difference operator
        diff_eq = Eq(residual, dsyn.subs({dsyn.dimensions[-1]: residual.dimensions[-1]}) -
                               dobs.subs({dobs.dimensions[-1]: residual.dimensions[-1]}))
        Operator(diff_eq)()
    else:
        # A simple data difference is enough in serial
        residual.data[:] = dsyn.data[:] - dobs.data[:]

    return residual
```

```python
# Create FWI gradient kernel
from devito import Function, TimeFunction, norm
from examples.seismic import Receiver

import scipy
def fwi_gradient(vp_in):
    # Create symbols to hold the gradient
    grad = Function(name="grad", grid=model.grid)
    # Create placeholders for the data residual and data
    residual = Receiver(name='residual', grid=model.grid,
                        time_range=geometry.time_axis,
                        coordinates=geometry.rec_positions)
    d_obs = Receiver(name='d_obs', grid=model.grid,
                     time_range=geometry.time_axis,
                     coordinates=geometry.rec_positions)
    d_syn = Receiver(name='d_syn', grid=model.grid,
                     time_range=geometry.time_axis,
                     coordinates=geometry.rec_positions)
    objective = 0.
    for i in range(nshots):
        # Update source location
        geometry.src_positions[0, :] = source_locations[i, :]

        # Generate synthetic data from true model
        _, _, _ = solver.forward(vp=model.vp, rec=d_obs)

        # Compute smooth data and full forward wavefield u0
        _, u0, _ = solver.forward(vp=vp_in, save=True, rec=d_syn)

        # Compute gradient from data residual and update objective function
        compute_residual(residual, d_obs, d_syn)

        objective += .5*norm(residual)**2
        solver.gradient(rec=residual, u=u0, vp=vp_in, grad=grad)

    return objective, grad
```

Having defined our FWI gradient procedure we can compute the initial iteration from our starting model. This allows us to visualize the gradient alongside the model perturbation and the effect of the gradient update on the model.

```python
# Compute gradient of initial model
ff, update = fwi_gradient(model0.vp)
assert np.isclose(ff, 57283, rtol=1e0)
```

```python
#NBVAL_IGNORE_OUTPUT
from devito import mmax
from examples.seismic import plot_image

# Plot the FWI gradient
plot_image(-update.data, vmin=-1e4, vmax=1e4, cmap="jet")

# Plot the difference between the true and initial model.
# This is not known in practice as only the initial model is provided.
plot_image(model0.vp.data - model.vp.data, vmin=-1e-1, vmax=1e-1, cmap="jet")

# Show what the update does to the model
alpha = .5 / mmax(update)
plot_image(model0.vp.data + alpha*update.data, vmin=2.5, vmax=3.0, cmap="jet")
```

We see that the gradient and the true perturbation have the same sign, therefore, with an appropriate scaling factor, we will update the model in the correct direction.

```python
from devito import Min, Max
# Define bounding box constraints on the solution.
def update_with_box(vp, alpha, dm, vmin=2.0, vmax=3.5):
    """
    Apply gradient update in-place to vp with box constraint

    Notes:
    ------
    For more advanced algorithm, one will need to gather the non-distributed
    velocity array to apply constrains and such.
    """
    update = vp + alpha * dm
    update_eq = Eq(vp, Max(Min(update, vmax), vmin))
    Operator(update_eq)()
```

```python
#NBVAL_SKIP

from devito import mmax

# Run FWI with gradient descent
history = np.zeros((fwi_iterations, 1))
for i in range(0, fwi_iterations):
    # Compute the functional value and gradient for the current
    # model estimate
    phi, direction = fwi_gradient(model0.vp)

    # Store the history of the functional values
    history[i] = phi

    # Artificial Step length for gradient descent
    # In practice this would be replaced by a Linesearch (Wolfe, ...)
    # that would guarantee functional decrease Phi(m-alpha g) <= epsilon Phi(m)
    # where epsilon is a minimum decrease constant
    alpha = .05 / mmax(direction)

    # Update the model estimate and enforce minimum/maximum values
    update_with_box(model0.vp , alpha , direction)

    # Log the progress made
    print('Objective value is %f at iteration %d' % (phi, i+1))
```

    Objective value is 39293.834132 at iteration 1
    Objective value is 24506.003329 at iteration 2
    Objective value is 14386.446012 at iteration 3
    Objective value is 7907.553497 at iteration 4
    Objective value is 3960.108621 at iteration 5

```python
#NBVAL_IGNORE_OUTPUT

# Plot inverted velocity model
plot_velocity(model0)
```

```python
#NBVAL_SKIP
import matplotlib.pyplot as plt

# Plot objective function decrease
plt.figure()
plt.loglog(history)
plt.xlabel('Iteration number')
plt.ylabel('Misift value Phi')
plt.title('Convergence')
plt.show()
```

## References

[1] _Virieux, J. and Operto, S.: An overview of full-waveform inversion in exploration geophysics, GEOPHYSICS, 74, WCC1–WCC26, doi:10.1190/1.3238367, http://library.seg.org/doi/abs/10.1190/1.3238367, 2009._

[2] _Haber, E., Chung, M., and Herrmann, F. J.: An effective method for parameter estimation with PDE constraints with multiple right hand sides, SIAM Journal on Optimization, 22, http://dx.doi.org/10.1137/11081126X, 2012._

This notebook is part of the tutorial "Optimised Symbolic Finite Difference Computation with Devito" presented at the Intel® HPC Developer Conference 2017.


```python
import numpy as np
import pandas as pd
import linearsolve as ls
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Class 15: Prescott's Real Business Cycle Model II

In this notebook, we continue to examine the centralized version of the model from pages 11-17 in Edward Prescott's article "Theory Ahead of Business Cycle Measurement in the Fall 1986 of the Federal Reserve Bank of Minneapolis' *Quarterly Review* (link to article: [https://www.minneapolisfed.org/research/qr/qr1042.pdf](https://www.minneapolisfed.org/research/qr/qr1042.pdf)). In this notebook, we:

1. Look at the effect of changing $\rho$ (the autoregressive coefficient on log TFP) on the simulated impulse responses of model variables to a TFP shock
2. Compute a stochastic simulation of the model and compute summary statistics.

## Example: Effect of Changing $\rho$ on Impulse Responses

Recall that the equilibrium conditions for Prescott's RBC model are:

\begin{align}
\frac{1}{C_t} & = \beta E_t \left[\frac{\alpha A_{t+1}K_{t+1}^{\alpha-1}L_{t+1}^{1-\alpha} +1-\delta }{C_{t+1}}\right]\\
\frac{\varphi}{1-L_t} & = \frac{(1-\alpha)A_tK_t^{\alpha}L_t^{-\alpha}}{C_t} \\
Y_t & = A_t K_t^{\alpha}L_t^{1-\alpha}\\
K_{t+1} & = I_t + (1-\delta) K_t\\
Y_t & = C_t + I_t\\
\log A_{t+1} & = \rho \log A_t + \epsilon_{t+1}
\end{align}

where $\epsilon_{t+1} \sim \mathcal{N}(0,\sigma^2)$.

The objective is use `linearsolve` to simulate impulse responses to a TFP shock using the following parameter values for the simulation for $\rho = 0.5,0.75,0.9,0.99$. Other parameter values are given in the table below:

| $\sigma$ | $\beta$ | $\varphi$ | $\alpha$ | $\delta $ |
|----------|---------|-----------|----------|-----------|
| 0.006    | 0.99    | 1.7317    | 0.35     |  0.025    |

## Model Preparation

As usual, we recast the model in the form required for `linearsolve`. Write the model with all variables moved to the left-hand side of the equations and dropping the expecations operator $E_t$ and the exogenous shock $\epsilon_{t+1}$:

\begin{align}
0 & = \beta\left[\frac{\alpha A_{t+1}K_{t+1}^{\alpha-1}L_{t+1}^{1-\alpha} +1-\delta }{C_{t+1}}\right] - \frac{1}{C_t}\\
0 & = \frac{(1-\alpha)A_tK_t^{\alpha}L_t^{-\alpha}}{C_t} - \frac{\varphi}{1-L_t}\\
0 & = A_t K_t^{\alpha}L_t^{1-\alpha} - Y_t\\
0 & = I_t + (1-\delta) K_t - K_{t+1}\\
0 & = C_t + I_t - Y_t\\
0 & = \rho \log A_t - \log A_{t+1}
\end{align}

Remember, capital and TFP are called *state variables* because they're $t+1$ values are predetermined. Output, consumption, and investment are called a *costate* or *control* variables. Note that the model as 5 equations in 5 endogenous variables.

## Initialization, Approximation, and Solution

The next several cells initialize the model in `linearsolve` and then approximate and solve it.

```python
# Create a variable called 'parameters' that stores the model parameter values in a Pandas Series. CELL PROVIDED
# Note that a value for rho is absent
parameters = pd.Series()

parameters['beta'] = 0.99
parameters['phi'] = 1.7317
parameters['alpha'] = 0.35
parameters['delta'] = 0.025

# Print the model's parameters
print(parameters)
```

```python
# Create a variable called 'sigma' that stores the value of sigma

```

```python
# Create variable called 'varNames' that stores the variable names in a list with state variables ordered first. CELL PROVIDED
var_names = ['a','k','y','c','i','l']

# Create variable called 'shockNames' that stores an exogenous shock name for each state variable. CELL PROVIDED
shock_names = ['e_a','e_k']
```

```python
# Define a function that evaluates the equilibrium conditions of the model solved for zero. CELL PROVIDED
def equilibrium_equations(variables_forward,variables_current,parameters):

    # Parameters. PROVIDED
    p = parameters

    # Current variables. PROVIDED
    cur = variables_current

    # Forward variables. PROVIDED
    fwd = variables_forward

    # Euler equation
    mpk = p.alpha*fwd.a*fwd.k**(p.alpha-1)*fwd.l**(1-p.alpha)
    euler_equation = p.beta*(mpk+1-p.delta)/fwd.c - 1/cur.c

    # Labor-labor choise
    mpl = (1-p.alpha)*cur.a*cur.k**p.alpha*cur.l**(-p.alpha)
    labor_leisure = mpl/cur.c - p.phi/(1-cur.l)

    # Production function
    production_function = cur.a*cur.k**p.alpha*cur.l**(1-p.alpha) - cur.y

    # Capital evolution
    capital_evolution = cur.i + (1 - p.delta)*cur.k - fwd.k

    # Market clearing
    market_clearing = cur.c+cur.i - cur.y

    # Exogenous tfp
    tfp_process = p.rho*np.log(cur.a) - np.log(fwd.a)

    # Stack equilibrium conditions into a numpy array
    return np.array([
        euler_equation,
        labor_leisure,
        production_function,
        capital_evolution,
        market_clearing,
        tfp_process
        ])
```

Next, initialize the model using `ls.model` which takes the following required arguments:

* `equations`
* `n_states`
* `var_names`
* `shock_names`
* `parameters`

```python
# Initialize the model into a variable named 'rbc_model'. CELL PROVIDED.
rbc_model = ls.model(equations = equilibrium_equations,
                 n_states=2,
                 var_names=var_names,
                 shock_names=shock_names,
                 parameters=parameters)
```

### SImulation and Plotting

The objective is to create a $2\times 2$ grid of plots containing the impulse responses of TFP, output, labor, and consumption to a one percent shock to TFP for each of the values for $\rho$: 0.5,0.75,0.9,0.99. Here are the steps that we'll take:

1. Initilize figure and axes for plotting.
2. Iterate over each desired value for $\rho$.
    1. Set `rbc_model.parameters['rho']` equal to current value of $\rho$.
    2. Use `rbc_model.compute_ss()` to compute the steady state with `guess` equal to `[1,4,1,1,1,0.5]`.
    3. Use `rbc_model.approximate_and_solve()` to approximate and solve the model with the current value of $\rho$.
    4. Use `rbc_model.impulse()` to compute the 51 period impulse response to a 0.01 unit shock to TFP in period 5.
    5. Add the computed impulse responses to the axes.

```python
# Create a 12x8 figure

# Create four axis variables: 'ax1', 'ax2', 'ax3', 'ax4'

# Create an axis equal to the size of the figure. PROVIDED
ax0 = fig.add_subplot(1,1,1)
# Turn off the axis so that the underlying axes are visible. PROVIDED
ax0.set_frame_on(False)
# Hide the x-axis. PROVIDED
ax0.get_xaxis().set_visible(False)
# Hide the y-axis. PROVIDED
ax0.get_yaxis().set_visible(False)

# Create variable called 'rho_values' that stores the desired values of rho

# Iterate over the elements of rho_values

    # Update the value of rho in rbc_model.parameters

    # Compute the steady state with initial guess equal to [1,4,1,1,1,0.5]

    # Approximate the model and solve

    # Compute the impulse responses to a 0.01 unit shock to TFP

    # Add plots of TFP, output, labor, and consumption to ax1, ax2, ax3, and ax4

    # Plot the point 0,0 on ax0 with the same line properties used for the other plotted lines and provide a label

# Set axis 1 title. PROVIDED
ax1.set_title('TFP')

# Set axis 2 title. PROVIDED
ax2.set_title('Output')

# Set axis 3 title. PROVIDED
ax3.set_title('Labor')

# Set axis 4 title. PROVIDED
ax4.set_title('Consumption')

# Add grid to the axis 1. PROVIDED
ax1.grid()

# Add grid to the axis 2. PROVIDED
ax2.grid()

# Add grid to the axis 3. PROVIDED
ax3.grid()

# Add grid to the axis 4. PROVIDED
ax4.grid()

# Set ax1 y-axis limits to [0,2]. PROVIDED
ax1.set_ylim([0,2])
# Set ax2 y-axis limits to [0,2]. PROVIDED
ax2.set_ylim([0,2])
# Set ax3 y-axis limits to [-0.5,1.25]. PROVIDED
ax3.set_ylim([-0.5,1.5])
# Set ax4 y-axis limits to [-0.5,1.5]. PROVIDED
ax4.set_ylim([-0.5,1.5])

# Add legend below the figure. PROVIDED
legend = ax0.legend(loc='upper center',bbox_to_anchor=(0.5,-0.075), ncol=4,fontsize=15)
```

## Example: Stochastic Simulation

Compute a 401 period stochastic simulation of the model. Set $\rho=0.75$ to match US business cycle data. And set the seed for the simulation to 126.

```python
# Set the value of rho in rbc_model.parameters to 0.75

# Compute the steady state with initial guess equal to [1,4,1,1,1,0.5]

# Approximate the model and solve

```

```python
# Compute the stochastic simulation using the .stoch_sim() method of rbc_model

```

On a single axis, plot the simulated values for output, consumption, investment, and labor.

```python
# Create 12x4 figure. PROVIDED
fig = plt.figure(figsize=(12,4))

# Create axis. PROVIDED
ax = fig.add_subplot(1,1,1)

# Plot output, consumption, investment, labor

# Add legend outside plot. PROVIDED
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))

```

```python
# Print the standard deviations (times 100) of output, consumption, investment, and labor in the simulated data

```

```python
# Print the correlations of output, consumption, investment, and labor in the simulated data

```


Stochastic Processes:

Data Analysis and Computer Simulation

Distribution function and random number

-Random walk-

#### Note 1

- In this plot, you will learn the interesting properties of random walk as a very basic example of stochastic processes.

- After summarizing the theoretical knowledge we already learn and can be applicable to the random walk, we will conduct computer simulations of a random walk and examine the results with the theoretical knowledge.

- At the end of this plot, we will understand the close relationship between random walk and the diffusion process.

# 1D random walk

## The model system

- Because the word "random walk" is translated as "drunken steps" in Japanese, I will use this analogy to explain it.

- Let us consider the probability distribution function ${P}(l,m)$ of finding a drunken man at the location $L=l$ after $M=m$ steps.

- He was initially at the origin ($l=0$ at $m=0$) and made $m$ random steps $s_i=-1\ {\rm or}\ 1$ until now, where $s_i=-1$ means to take a step to the left and $+1$ to the right.

- Therefore $l=\sum_{i=1}^m s_i$.

#### Note 2

- To avoid the complications, we will consider a 1-dimensional random walk here.

- The goal of this part is to obtain the probability distribution function $P(l,m)$ of finding the drunken man at the location $L=l$ after $m$ steps.

- ...

- Therefore the the location $l$ of the drunken man is given by the sum of single steps $s_i$ from the 1st step $i=1$ to the $m$-th step.

## As binomial distribution

- The total number of steps to the right: $n_+$
- The total number of steps to the left: $n_-$
- The total number of steps: $m=n_+ + n_-$
- The current location: $l=n_+ - n_-$

\begin{equation}
\therefore \ \ \ n_+=\frac{m+l}{2}, \ \ \ \ \ \ n_-=\frac{m-l}{2}\tag{E1}
\end{equation}

\begin{equation}
%\tilde
{P}(l,m)\rightarrow P_{\rm Binomial}(n_+;m)
%\frac{dn_+}{dl}
=P_{\rm Binomial}(n_-;m)
%\frac{dn_-}{dl}
\tag{E2}
\end{equation}

\begin{equation}
\hspace{20mm}
=
%\frac{1}{2}
\frac{m!}{n_+!(m-n_+)!}{{p}^{n_+}}{{(1-p)}^{m-n_+}}\tag{E3}
\end{equation}

#### Note 3

- Let us try to evaluate $P(l,m)$ using binomial distribution function.

- To this end, we introduce two variables namely $n_+$, the total number of steps to the right, and $n_-$, The total number of steps to the left.

- Then the total number of steps $m$ is represented as $n_+ + n_−$, and the current location $l$ is represented as $n_+ - n_−$.

- Solving for $n_+$ and $n_-$, we obtain Eq.(E1).

- Using  $n_+$ or $n_-$, the probability distribution function $P(l,m)$ can be estimated using binomial distribution function $P_{Binomial}$ of obtaining $n_+$ right steps in $m$ total steps or equivalently obtaining $n_-$ steps in $m$ steps.

- We already know that such probabilities are given explicitly by the form of Eq.(E3).

## As normal distribution (for $n_+,m\gg 1$)

\begin{equation}
P_{\rm Binomial}(n_+;m)
%\xrightarrow[n_+\to cont.]{n_+,m\gg 1}\
\simeq
\frac{1}{\sqrt{2\pi {{\sigma'}^{2}}}}\exp \left[ -\frac{{{(n_+-{{\mu' }_{1}})}^{2}}}{2{{\sigma' }^{2}}} \right]\tag{E4}
\end{equation}

\begin{equation}
{\rm with}\hspace{5mm}
\mu_1'=\langle n_+\rangle=mp,\ \ \
\sigma'^2=\langle n_+^2\rangle - \langle n_+\rangle^2=mp(1-p) \tag{E5, E6}
\end{equation}

- Recall that $n_+=(m+l)~/~2$

\begin{equation}
P_{\rm Binomial}(n_+;m)
%\rightarrow\frac{1}{\sqrt{2\pi mp(1-p)}}
\simeq\exp \left[ -\frac{{{(l-{m(2p-1)})}^{2}}}{8mp(1-p)} \right]\tag{E7}
\end{equation}

\begin{equation}
\therefore \ \
%\tilde
{P}(l,m)= P_{\rm Binomial}(n_+;m)\frac{dn_+}{dl}
= P_{\rm Binomial}(n_+;m)\frac{1}{2}\tag{E8}
\end{equation}

\begin{equation}
\simeq\frac{1}{\sqrt{2\pi \sigma''^2}}
\exp \left[ -\frac{{{(l-{\mu''_1})}^{2}}}{2\sigma''^2} \right]\tag{E9}
\end{equation}

\begin{equation}
%\hspace{20mm}
{\rm with}\hspace{5mm}
\mu''_1=\langle l\rangle=m(2p-1),\ \ \
\sigma''^2=\langle l^2\rangle - \langle l\rangle^2=4mp(1-p) \tag{E10, E11}
\end{equation}

#### Note 4

- When $n_+$ and $m$ are much larger than 1, we know that the binomial distribution function becomes equivalent to normal distribution shown in Eq.(E4) with the average and the variance for $n_+$ shown in Eqs.(E5) and (E6).

- Recall that $n_+$ is equal to $m+l$ devided by 2, binomial distribution function appeared in Eq.(E4) can be rewritten to Eq.(E7) s a function of $l$.

- Therefore the aimed probability distribution function $P(l,m)$ is obtained as Eq.(E9) with the average and the variance for $l$ shown in Eqs.(E10) and (E11).

## By computer simulation

```python
% matplotlib inline
import numpy as np # import numpy library as np
import math # use mathematical functions defined by the C standard
import matplotlib.pyplot as plt # import pyplot library as plt
plt.style.use('ggplot') # use "ggplot" style for graphs
```

#### Note 5

- Now let us perform computer simulations for the random walk using a random number generator.

- First import libraries as usual.

```python
p = 0.5 # set p, propability to take a step to the right
M = 1000 # M = number of total steps
N = 100000 # N = number of independent random walkers
ave = M*(2*p-1) # average of the location L after M steps Eq.(E10)
std = np.sqrt(4*M*p*(1-p)) # standard deviation of L after M steps Eq.(E11)
print('p =',p,'M =',M)
L = np.zeros(N)
np.random.seed(0) # initialize the random number generator with seed=0
for i in range(N): # repeat independent random walks N times
    step=np.random.choice([-1,1],M) # generate random sampling from -1 or +1 M times and store them as step
    L[i]=np.sum(step) # calculate l after making M random steps and store it as L
nmin=np.int(ave-std*5)
nmax=np.int(ave+std*5)
nbin=np.int((nmax-nmin)/4)
plt.hist(L,range=[nmin,nmax],bins=nbin,normed=True) # plot normalized histgram of L
x = np.arange(nmin,nmax,0.01/std) # create array of x from nmin to nmax with increment 0.01/std
y = np.exp(-(x-ave)**2/(2*std**2))/np.sqrt(2*np.pi*std**2) # calculate the theoretical distribution Eq.(E9)
plt.plot(x,y,lw=2,color='b') # plot y vs. x with blue line
plt.xlabel(r'$l$',fontsize=16) # set x-label
plt.ylabel(r'$P(l,m)$',fontsize=16) # set y-label
plt.legend([r'Gauss',r'histgram'], fontsize=16) # set legends
plt.xlim(ave-250,ave+250) # set x-range
plt.show() # display plots
```

#### Note 6

- The main part of the simulation code is shown here.

- In the beginning, we set $p=0.5$ representing that the drunken man has equal probabilities for moving right or left, then set the number of total steps $M=1,000$ and the number of independent random walkers $N=100,000$.

- The average and the standard deviation for the current location $l$ are calculated using Eqs.(E10) and (E11).

- After initialize the component of an array $L$ to be all zero, we generate random sampling from -1 or +1 M times and store them as step at the 10th line, then calculate the location $l$ by accumulating all steps and store it as L at the 11th line.

- The 9th line represents that this operation is repeated N times.

- The remaining part of the code is to draw plots.

(Run cell)

- By running this code example, you see that the aimed probability distribution function $P(l,m)$ is perfectly represented by Eqs.(E9) to (E11).

- You may repeat the same simulation by choosing different values of total steps, for example $M=100$, $1,000$, $10,000$, and $100,000$ to see how the distribution changes with the number of total steps.

## In connection with the diffusion constant $D$

### $P(x,t)$ from random walk

- Define $a$ as the length of a single step and $t_s$ as the time between subsequent steps.

- Define $x=al$ as the position of the random walker and $t=t_s m$ as the duration of time for making $m$ steps.

- Here we consider a drift free case $p=0.5$, i.e., $\mu_1=\langle l\rangle=m(2p-1)=0$.

\begin{equation}
P(x,t)=P(l,m)\frac{dl}{dx}=P(l,m)\frac{1}{a}
\tag{E12}
\end{equation}

\begin{equation}
%P_{\rm Binomial}(n_+;m)\rightarrow
=\frac{1}{a\sqrt{8\pi mp(1-p)}}
\exp \left[ -\frac{{l^2}}{8mp(1-p)} \right]\tag{E13}
\end{equation}

\begin{equation}
%P_{\rm Binomial}(n_+;m)\rightarrow
=\frac{1}{\sqrt{8\pi a^2 p(1-p)t/t_s}}
\exp \left[ -\frac{{x^2}}{8a^2p(1-p)t/t_s} \right]\tag{E14}
\end{equation}

\begin{equation}
\hspace{-10mm}
{\rm with}\hspace{5mm}
\mu_1=\langle x\rangle=0,\ \
\sigma^2=\langle x^2\rangle - \langle x\rangle^2=4a^2p(1-p)t/t_s \tag{E15, E16}
\end{equation}

#### Note 7

- Let us nest discuss the property of random walk in connection with the diffusion constant $D$.

- To this end, we define two constants, namely the length of a single step "a" and the time between subsequent steps $t_s$.

- Here we consider a drift free case where p=0.5 and thus the average of $l$ is 0.

- Using "a" and $t_s$, we define the position of the random walker as $x$, which is equal to "a" multiplied with the current location "l", and the duration of time for making m steps as $t_s$, which is equal to $t_s$ multiplied with the number of total steps "m".

- Taking care of the normalization as shown in Eq.(E12), the probability distribution function $P(l,m)$ can be converted to $P(x,t)$ as shown in Eqs.(E13) to (E16).

- Note that the variance of the position $x$ is now given by a function of time $t$ not the number of steps $m$.

### $P(x,t)$ from the diffusion equation

- If we solve the diffusion equation defined with a diffusion constant $D$ as follows.

\begin{equation}
\frac{\partial}{\partial t} P(x,t)=D\frac{\partial ^2}{\partial x^2} P(x,t)
\tag{E17}
\end{equation}

\begin{equation}
{\rm with}\hspace{5mm}P(x,t=0)=\delta(x)
\tag{E18}
\end{equation}

- The solution is given by

\begin{equation}
P(x,t)=\frac{1}{\sqrt{4\pi Dt}}
\exp \left[ -\frac{{{x^2}}}{4Dt} \right]\tag{E19}
\end{equation}

- By comparing Eqs.(E14) and (E19), we finally obtain the following general formula which relates the diffusion constant $D$ and the variance of the position of random walkers $\sigma^2$, which is also called as the mean square displacement.

\begin{equation}
D=\frac{2a^2p(1-p)}{t_s}=\frac{\sigma^2}{2t}\tag{E20}
\end{equation}

#### Note 8

- Turn our attention to the diffusion equation shown in Eq.(E17) with a diffusion constant $D$.

- If we solve this diffusion equation using the initial condition shown in Eq.(18), the solution is given by Eq.(E19) which has an identical form to Eq.(E14).

- By comparing Eqs.(E14) and (E19), we finally obtain the following general formula which relates the diffusion constant $D$ and the variance of the position of random walkers $\sigma^2$, which is also called as the mean square displacement.

# Homework

```python

```