# 15 PDEs: Solution with Time Stepping

## Heat Equation
The **heat equation** can be derived from Fourier's law and energy conservation (see the [lecture notes on the heat equation (PDF)](15_PDEs_LectureNotes_HeatEquation.pdf))

$$
\frac{\partial T(\mathbf{x}, t)}{\partial t} = \frac{K}{C\rho} \nabla^2 T(\mathbf{x}, t),
$$

## Problem: insulated metal bar (1D heat equation)
A metal bar of length $L$ is insulated along it lengths and held at 0ºC at its ends. Initially, the whole bar is at 100ºC. Calculate $T(x, t)$ for $t>0$.

### Analytic solution
Solve by separation of variables and power series: The general solution that obeys the boundary conditions $T(0, t) = T(L, t) = 0$ is

$$
T(x, t) = \sum_{n=1}^{+\infty} A_n \sin(k_n x)\, \exp\left(-\frac{k_n^2 K t}{C\rho}\right), \quad k_n = \frac{n\pi}{L}
$$

The specific solution that satisfies $T(x, 0) = T_0 = 100^\circ\text{C}$ leads to $A_n = 4 T_0/n\pi$ for $n$ odd:

$$
T(x, t) = \sum_{n=1,3,5,\dots}^{+\infty} \frac{4 T_0}{n \pi} \sin(k_n x)\, \exp\left(-\frac{k_n^2 K t}{C\rho}\right)
$$

```python
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('ggplot')
```

Analytical solution:

```python
def T_bar(x, t, T0, L, K=237, C=900, rho=2700, nmax=1000):
    T = np.zeros_like(x)
    eta = K / (C*rho)
    for n in range(1, nmax, 2):
        kn = n*np.pi/L
        T += 4*T0/(np.pi * n) * np.sin(kn*x) * np.exp(-kn*kn * eta * t)
    return T
```

```python
T0 = 100.
L = 1.0
X = np.linspace(0, L, 100)
for t in np.linspace(0, 3000, 50):
    plt.plot(X, T_bar(X, t, T0, L))
plt.xlabel(r"$x$ (m)")
plt.ylabel(r"$T$ ($^\circ$C)");
```

### Numerical solution: Leap frog
Discretize (finite difference):

For the time domain we only have the initial values so we use a simple forward difference for the time derivative:

$$
\frac{\partial T(x,t)}{\partial t} \approx \frac{T(x, t+\Delta t) - T(x, t)}{\Delta t}
$$

For the spatial derivative we have initially all values so we can use the more accurate central difference approximation:

$$
\frac{\partial^2 T(x, t)}{\partial x^2} \approx \frac{T(x+\Delta x, t) + T(x-\Delta x, t) - 2 T(x, t)}{\Delta x^2}
$$

Thus, the heat equation can be written as the finite difference equation

$$
\frac{T(x, t+\Delta t) - T(x, t)}{\Delta t} = \frac{K}{C\rho} \frac{T(x+\Delta x, t) + T(x-\Delta x, t) - 2 T(x, t)}{\Delta x^2}
$$

which can be reordered so that the RHS contains only known terms and the LHS future terms. Index $i$ is the spatial index, and $j$ the time index: $x = x_0 + i \Delta x$, $t = t_0 + j \Delta t$.

$$
T_{i, j+1} = (1 - 2\eta) T_{i,j} + \eta(T_{i+1,j} + T_{i-1, j}), \quad \eta := \frac{K \Delta t}{C \rho \Delta x^2}
$$

Thus we can step forward in time ("leap frog"), using only known values.

### Solve the 1D heat equation numerically for an iron bar
* $K = 237$ W/mK
* $C = 900$ J/K
* $\rho = 2700$ kg/m3
* $L = 1$ m
* $T_0 = 373$ K and $T_b = 273$ K
* $T(x, 0) = T_0$ and $T(0, t) = T(L, t) = T_b$

#### Key considerations

The key line is the computation of the new temperature field at time step $j+1$ from the temperature distribution at time step $j$. It can be written purely with numpy array operations (see last lecture!):

```python
T[1:-1] = (1 - 2*eta) * T[1:-1] + eta * (T[2:] + T[:-2])
```

Note that the range operator `T[start:end]` *excludes* `end`, so in order to include `T[1], T[2], ..., T[-2]` (but not the rightmost `T[-1]`) we have to use `T[1:-1]`.

The *boundary conditions* are fixed for all times:
```python
T[0] = T[-1] = Tb
```

The *initial conditions* (at time step `j=0`)
```python
T[1:-1] = T0
```
are only used to compute the distribution of temperatures at the next step `j=1`.

#### Solution

```python
import numpy as np

import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

For 3D rotatable output:

```python
%matplotlib widget
```

For HTML/nbviewer output, use inline:

```python
%matplotlib inline
```

Numerical solution:

```python
L_rod = 1.    # m
t_max = 3000. # s

Dx = 0.02   # m
Dt = 2    # s

Nx = int(L_rod // Dx)
Nt = int(t_max // Dt)

Kappa = 237 # W/(m K)
CHeat = 900 # J/K
rho = 2700  # kg/m^3

T0 = 373 # K
Tb = 273 # K

eta = Kappa * Dt / (CHeat * rho * Dx**2)
eta2 = 1 - 2*eta

step = 20  # plot solution every n steps

print("Nx = {0}, Nt = {1}".format(Nx, Nt))
print("eta = {0}".format(eta))

T = np.zeros(Nx)
T_plot = np.zeros((Nt//step + 1, Nx))

# initial conditions
T[1:-1] = T0
# boundary conditions
T[0] = T[-1] = Tb

t_index = 0
T_plot[t_index, :] = T
for jt in range(1, Nt):
    T[1:-1] = eta2 * T[1:-1] + eta*(T[2:] + T[:-2])
    if jt % step == 0 or jt == Nt-1:
        t_index += 1
        T_plot[t_index, :] = T
        print("Iteration {0:5d}".format(jt), end="\r")
else:
    print("Completed {0:5d} iterations: t={1} s".format(jt, jt*Dt))
```

    Nx = 49, Nt = 1500
    eta = 0.4876543209876543
    Completed  1499 iterations: t=2998 s

#### Visualization
Visualize (you can use the code as is).

Note how we are making the plot use proper units by mutiplying with `Dt * step` and `Dx`.

```python
X, Y = np.meshgrid(range(T_plot.shape[0]), range(T_plot.shape[1]))
Z = T_plot[X, Y]
fig = plt.figure()
ax = fig.add_subplot(111, projection="3d")
ax.plot_wireframe(X*Dt*step, Y*Dx, Z)
ax.set_xlabel(r"time $t$ (s)")
ax.set_ylabel(r"position $x$ (m)")
ax.set_zlabel(r"temperature $T$ (K)")
fig.tight_layout()
```

2D as above for the analytical solution…

```python
X = Dx * np.arange(T_plot.shape[1])
plt.plot(X, T_plot.T)
plt.xlabel(r"$x$ (m)")
plt.ylabel(r"$T$ (K)");
```

## Stability of the solution

### Empirical investigation of the stability
Investigate the solution for different values of `Dt` and `Dx`. Can you discern patters for stable/unstable solutions?

Report `Dt`, `Dx`, and `eta`
* for 3 stable solutions
* for 3 unstable solutions

```python
def calculate_T(L_rod=1, t_max=3000, Dx=0.02, Dt=2, T0=373, Tb=273,
               step=20):
    Nx = int(L_rod // Dx)
    Nt = int(t_max // Dt)

    Kappa = 237 # W/(m K)
    CHeat = 900 # J/K
    rho = 2700  # kg/m^3

    eta = Kappa * Dt / (CHeat * rho * Dx**2)
    eta2 = 1 - 2*eta

    print("Nx = {0}, Nt = {1}".format(Nx, Nt))
    print("eta = {0}".format(eta))

    T = np.zeros(Nx)
    T_plot = np.zeros((int(np.ceil(Nt/step)) + 1, Nx))

    # initial conditions
    T[1:-1] = T0
    # boundary conditions
    T[0] = T[-1] = Tb

    t_index = 0
    T_plot[t_index, :] = T
    for jt in range(1, Nt):
        T[1:-1] = eta2 * T[1:-1] + eta*(T[2:] + T[:-2])
        if jt % step == 0 or jt == Nt-1:
            t_index += 1
            T_plot[t_index, :] = T
            print("Iteration {0:5d}".format(jt), end="\r")
    else:
        print("Completed {0:5d} iterations: t={1} s".format(jt, jt*Dt))
    return T_plot

def plot_T(T_plot, Dx, Dt, step):
    X, Y = np.meshgrid(range(T_plot.shape[0]), range(T_plot.shape[1]))
    Z = T_plot[X, Y]
    fig = plt.figure()
    ax = fig.add_subplot(111, projection="3d")
    ax.plot_wireframe(X*Dt*step, Y*Dx, Z)
    ax.set_xlabel(r"time $t$ (s)")
    ax.set_ylabel(r"position $x$ (m)")
    ax.set_zlabel(r"temperature $T$ (K)")
    fig.tight_layout()
    return ax
```

```python
Dx, Dt, step = 0.01, 2, 20
T_plot = calculate_T(Dx=Dx, Dt=Dt, step=step)
plot_T(T_plot, Dx, Dt, step)
```

Note that *decreasing* the value of $\Delta x$ made the solution *unstable*. This is strange, we have gotten used to the idea that working on a finer mesh will increase the detail (until we hit round-off error) and just become computationally more expensive. But here the algorithm suddenly becomes unstable (and it is not just round-off).

For certain combination of values of $\Delta t$ and $\Delta x$ the solution become unstable. Empirically, bigger $\eta$ leads to instability. (In fact, $\eta \geq \frac{1}{2}$ is unstable for the leapfrog algorithm as we will see.)

### Von Neumann stability analysis

If the difference equation solution diverges then we *know* that we have a bad approximation to the original PDE.

Von Neumann stability analysis starts from the assumption that *eigenmodes* of the difference equation can be written as

$$
T_{m,j} = \xi(k)^j e^{ikm\Delta x}, \quad t=j\Delta t,\ x=m\Delta x
$$

with the unknown wave vectors $k=2\pi/\lambda$ and unknown complex functions – the *amplification factors* – $\xi(k)$.

Solutions of the difference equation can be written as linear superpositions of these basis functions. But they are only stable if the eigenmodes are stable, i.e., will not grow in time (with $j$). This is the case when

$$
|\xi(k)| < 1
$$

for all $k$.

Insert the eigenmodes into the finite difference equation

$$
T_{m, j+1} = (1 - 2\eta) T_{m,j} + \eta(T_{m+1,j} + T_{m-1, j})
$$

to obtain

\begin{align}
\xi(k)^{j+1} e^{ikm\Delta x} &= (1 - 2\eta) \xi(k)^{j} e^{ikm\Delta x}
    + \eta(\xi(k)^{j} e^{ik(m+1)\Delta x} + \xi(k)^{j} e^{ik(m-1)\Delta x})\\
\xi(k) &= (1 - 2\eta) + \eta(e^{ik\Delta x} + e^{-ik\Delta x})\\
\xi(k) &= 1 - 2\eta + 2\eta \cos k\Delta x\\
\xi(k) &= 1 + 2\eta\big(\cos k\Delta x - 1\big)
\end{align}

For $|\xi(k)| < 1$ (and all possible $k$):

\begin{align}
|\xi(k)| < 1 \quad &\Leftrightarrow \quad \xi^2(k) < 1\\
(1 + 2y)^2 = 1 + 4y + 4y^2 &< 1 \quad \text{with}\ \  y = \eta(\cos k\Delta x - 1)\\
y(1 + y) &< 0 \quad \Leftrightarrow \quad -1 < y < 0\\
\eta(\cos k\Delta x - 1) &\leq 0 \quad \forall k \quad (\eta > 0, -1 \leq \cos x \leq 1)\\
\eta(\cos k\Delta x - 1) &> -1\\
\eta &< \frac{1}{1 - \cos k\Delta x}\\
\eta = \frac{K \Delta t}{C \rho \Delta x^2} &< \frac{1}{2} \le \frac{1}{1 - \cos k\Delta x}
\end{align}

Thus, solutions are only stable for $\eta < 1/2$. In particular, decreasing $\Delta t$ will always improve stability, But decreasing $\Delta x$ requires an quadratic *increase* in $\Delta t$!

Note
* Perform von Neumann stability analysis when possible (depends on PDE and the specific discretization).
* Test different combinations of $\Delta t$ and $\Delta x$.
* Not guarantee that decreasing both will lead to more stable solutions!

Check my inputs:

This was stable and it conforms to the stability criterion:

```python
Dt = 2
Dx = 0.02
eta = Kappa * Dt /(CHeat * rho * Dx*Dx)
print(eta)
```

    0.4876543209876543

... and this was unstable, despite a seemingly small change:

```python
Dt = 2
Dx = 0.01
eta = Kappa * Dt /(CHeat * rho * Dx*Dx)
print(eta)
```

    1.9506172839506173

```python

```


## Performance Indicator

It is fundamental for any algorithm to measure the performance. In a multi-objective scenario, we can not calculate the distance to the true global optimum but must consider a set of solutions. Moreover, sometimes the optimum is not even known, and other techniques must be used.

First, let us consider a scenario where the Pareto-front is known:

```python
import numpy as np
from pymoo.factory import get_problem
from pymoo.visualization.scatter import Scatter

# The pareto front of a scaled zdt1 problem
pf = get_problem("zdt1").pareto_front()

# The result found by an algorithm
A = pf[::10] * 1.1

# plot the result
Scatter(legend=True).add(pf, label="Pareto-front").add(A, label="Result").show()
```

### Generational Distance (GD)

The GD performance indicator  measure the distance from solution to the Pareto-front. Let us assume the points found by our algorithm are the objective vector set $A=\{a_1, a_2, \ldots, a_{|A|}\}$ and the reference points set (Pareto-front) is $Z=\{z_1, z_2, \ldots, z_{|Z|}\}$. Then,

\begin{align}
\begin{split}
\text{GD}(A) & = & \; \frac{1}{|A|} \; \bigg( \sum_{i=1}^{|A|} d_i^p \bigg)^{1/p}\\[2mm]
\end{split}
\end{align}

where $d_i$ represents the Euclidean distance (p=2) from $a_i$ to its nearest reference point in $Z$. Basically, this results in the average distance from any point $A$ to the closest point in the Pareto-front.

```python
from pymoo.factory import get_performance_indicator

gd = get_performance_indicator("gd", pf)
print("GD", gd.calc(A))
```

    GD 0.05497689467314528

### Generational Distance Plus (GD+)

Ishibushi et. al. proposed in  GD+:

\begin{align}
\begin{split}
\text{GD}^+(A) & = & \; \frac{1}{|A|} \; \bigg( \sum_{i=1}^{|A|} {d_i^{+}}^2 \bigg)^{1/2}\\[2mm]
\end{split}
\end{align}

where for minimization $d_i^{+} = max \{ a_i - z_i, 0\}$ represents the modified distance from $a_i$ to its nearest reference point in $Z$ with the corresponding value $z_i$.

```python
from pymoo.factory import get_performance_indicator

gd_plus = get_performance_indicator("gd+", pf)
print("GD+", gd_plus.calc(A))
```

    GD+ 0.05497689467314528

### Inverted Generational Distance (IGD)

The IGD performance indicator  inverts the generational distance and measures the distance from any point in $Z$ to the closest point in $A$.

\begin{align}
\begin{split}
\text{IGD}(A) & = & \; \frac{1}{|Z|} \; \bigg( \sum_{i=1}^{|Z|} \hat{d_i}^p \bigg)^{1/p}\\[2mm]
\end{split}
\end{align}

where $\hat{d_i}$ represents the euclidean distance (p=2) from $z_i$ to its nearest reference point in $A$.

```python
from pymoo.factory import get_performance_indicator

igd = get_performance_indicator("igd", pf)
print("IGD", igd.calc(A))
```

    IGD 0.06690908300327662

### Inverted Generational Distance Plus (IGD+)

In  Ishibushi et. al. proposed IGD+ which is weakly Pareto compliant wheres the original IGD is not.

\begin{align}
\begin{split}
\text{IGD}^{+}(A) & = & \; \frac{1}{|Z|} \; \bigg( \sum_{i=1}^{|Z|} {d_i^{+}}^2 \bigg)^{1/2}\\[2mm]
\end{split}
\end{align}

where for minimization $d_i^{+} = max \{ a_i - z_i, 0\}$ represents the modified distance from $z_i$ to the closest solution in $A$ with the corresponding value $a_i$.

```python
from pymoo.factory import get_performance_indicator

igd_plus = get_performance_indicator("igd+", pf)
print("IGD+", igd_plus.calc(A))
```

    IGD+ 0.06466828842775944

### Hypervolume

For all performance indicators showed so far, a target set needs to be known. For Hypervolume only a reference point needs to be provided. First, I would like to mention that we are using the Hypervolume implementation from [DEAP](https://deap.readthedocs.io/en/master/). It calculates the area/volume, which is dominated by the provided set of solutions with respect to a reference point.

This image is taken from  and illustrates a two objective example where the area which is dominated by a set of points is shown in grey.
Whereas for the other metrics, the goal was to minimize the distance to the Pareto-front, here, we desire to maximize the performance metric.

```python
from pymoo.factory import get_performance_indicator

hv = get_performance_indicator("hv", ref_point=np.array([1.2, 1.2]))
print("hv", hv.calc(A))
```

    hv 0.9631646448182305


**1**. Making wallpaper with `fromfunction`

Adapted from [Circle Squared](http://igpphome.ucsd.edu/~shearer/COMP233/SciAm_Mandel.pdf)

Create a $400 \times 400$ array using the function `lambda i, j:  0.27**2*(i**2 + j**2) % 1.5`. Use `imshow` from `matplotlib.pyplot` with `interpolation='nearest'` and the `YlOrBr` colormap to display the resulting array as an image.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
```

```python

```

**2**. Find t least squares solution for $\beta_0, \beta_1, \beta_2$ using the normal equations $\hat{\beta} = (X^TX)^{-1}x^Ty$.

\begin{align}
10 &= \beta_0 + 3 \beta_1  + 7 \beta_2 \\
11 &= \beta_0 + 2 \beta_1  + 8 \beta_2 \\
9 &= \beta_0 + 3 \beta_1 + 7 \beta_2 \\
10 &= \beta_0 + 1 \beta_1 + 9 \beta_2 \\
\end{align}

You can find the inverse of a matrix by using `np.linalg.inv` and the transpose with `X.T`

```python

```


# Time Series - III (MA Models)

Moving average models are very similar to Auto regressive models, difference being that instead of using past observations of the time series, we try to estimate the value using past error terms.

\begin{align}
x_{t} &= w_{t} + \beta_{1}.w_{t-1} + \beta_{2}.w_{t-2} +   ..
\end{align}

In the above equation, x is the value of the time series that we want to estimate, $w_{t}$ are the error terms of the previous values, which is assumed to be normally distributed since it is white noise and betas are the coefficients. In this, we will try to explain the time series values using the previous noise terms.

For MA(q) models, ACF should be zero for lags > q by definition.

```python
#Importing the required packages

import pandas as pd
import numpy as np

from matplotlib import pyplot as plt
%matplotlib inline

import statsmodels.tsa.api as smt
import statsmodels.api as sm
import scipy.stats as scs
import statsmodels.stats as sms

import warnings
warnings.filterwarnings('ignore')
```

```python
#Defining a function to visualize and analyze the time series

def tsplot(y, lags=None, figsize=(15, 10), style='bmh'):
    '''
    Prepares a (3,2) dimensional plot for the visualization of time series values, autocorrelation and partial
    autocorrelation plots and QQ and probability plots for comparision with normal distribution.

    Args:
    y: time series values
    lags: How many lagging values are to be considered.
    '''
    if not isinstance(y, pd.Series):
        y = pd.Series(y)
    with plt.style.context(style):
        fig = plt.figure(figsize=figsize)

        layout = (3, 2)
        ts_ax = plt.subplot2grid(layout, (0, 0), colspan=2)
        acf_ax = plt.subplot2grid(layout, (1, 0))
        pacf_ax = plt.subplot2grid(layout, (1, 1))
        qq_ax = plt.subplot2grid(layout, (2, 0))
        pp_ax = plt.subplot2grid(layout, (2, 1))

        y.plot(ax=ts_ax)
        ts_ax.set_title('Time Series Analysis Plots')
        smt.graphics.plot_acf(y, lags=lags, ax=acf_ax, alpha=0.05)
        smt.graphics.plot_pacf(y, lags=lags, ax=pacf_ax, alpha=0.05)
        sm.qqplot(y, line='s', ax=qq_ax)
        qq_ax.set_title('QQ Plot')
        scs.probplot(y, sparams=(y.mean(), y.std()), plot=pp_ax)

        plt.tight_layout()
    return
```

**Simulating MA(1) process using beta = 0.6 and specifying the AR(p) alpha = 0 **

```python
n = int(1000)

#Setting AR(p) alphas = 0
alphas = np.array([0.])
betas = np.array([0.6])

#Adding zero-lag and negating alphas
ar = np.r_[1, -alphas]
ma = np.r_[1, betas]

ma1 = smt.arma_generate_sample(ar = ar, ma = ma, nsample = n)
#print(ma1.shape)
_ = tsplot(ma1, lags = 30)
```

Since this is a first order model, i.e, q = 1, there is a peak at lag 1 in ACF plot and rest of the peaks are insignificant.  By looking at the ACF of the series we can see how many sequential non zero lags exist. If there are q such lags then we can say that MA(q) model fits well at the data.
This is similar to looking at the PACF plot for AR(p) models.

The above plot shows that MA(1) model could be an appropriate fit for the series.

**Fitting a MA(1) model on the above simulated data. **

```python
max_lag = 30
model = smt.ARMA(ma1, order = (0,1)).fit(maxlag = max_lag,
                                        method = 'mle',
                                        trend = 'nc')
print(model.summary())
```

                                  ARMA Model Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                 1000
    Model:                     ARMA(0, 1)   Log Likelihood               -1459.048
    Method:                           mle   S.D. of innovations              1.041
    Date:                Sat, 23 Jun 2018   AIC                           2922.095
    Time:                        02:01:19   BIC                           2931.911
    Sample:                             0   HQIC                          2925.826

    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    ma.L1.y        0.6090      0.025     24.219      0.000       0.560       0.658
                                        Roots
    =============================================================================
                      Real          Imaginary           Modulus         Frequency
    -----------------------------------------------------------------------------
    MA.1           -1.6421           +0.0000j            1.6421            0.5000
    -----------------------------------------------------------------------------

The lag coefficient as calculated by the model is 0.67 which is very close to real value 0.6

```python
from statsmodels.stats.stattools import jarque_bera

score, pvalue, _, _ = jarque_bera(model.resid)

if pvalue < 0.10:
    print('We have reason to suspect the residuals are not normally distributed.')
else:
    print('The residuals seem normally distributed.')
```

    The residuals seem normally distributed.

**Simulating and fitting a MA(3) process to obtain the correct betas where betas1,2,3 are equal to 0.3, 0.2 and 0.1.**

```python
#We should be expecting peaks at lag 1,2,3 and the insignificant peaks beyond 3 lags in ACF plot

n = int(500)
alphas = np.array([0.])
betas = np.array([0.3, 0.2, 0.1])
ar = np.r_[1, -alphas]
ma = np.r_[1, betas]

ma3 = smt.arma_generate_sample(ar = ar, ma = ma, nsample = n)
#print(ma3.shape)
_ = tsplot(ma3, lags = 30)
```

**Fitting the MA(3) simulated time series **

```python
max_lag = 30
model = smt.ARMA(ma3, order=(0, 3)).fit(maxlag=max_lag,
                                        method='mle',
                                        trend='nc')
print(model.summary())
```

                                  ARMA Model Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                  500
    Model:                     ARMA(0, 3)   Log Likelihood                -724.881
    Method:                           mle   S.D. of innovations              1.031
    Date:                Sat, 23 Jun 2018   AIC                           1457.762
    Time:                        02:01:21   BIC                           1474.621
    Sample:                             0   HQIC                          1464.377

    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    ma.L1.y        0.2972      0.045      6.647      0.000       0.210       0.385
    ma.L2.y        0.1626      0.047      3.447      0.001       0.070       0.255
    ma.L3.y        0.0358      0.042      0.855      0.393      -0.046       0.118
                                        Roots
    =============================================================================
                      Real          Imaginary           Modulus         Frequency
    -----------------------------------------------------------------------------
    MA.1           -0.1913           -2.5844j            2.5915           -0.2618
    MA.2           -0.1913           +2.5844j            2.5915            0.2618
    MA.3           -4.1625           -0.0000j            4.1625           -0.5000
    -----------------------------------------------------------------------------

```python
from statsmodels.stats.stattools import jarque_bera

score, pvalue, _, _ = jarque_bera(model.resid)

if pvalue < 0.10:
    print('We have reason to suspect the residuals are not normally distributed.')
else:
    print('The residuals seem normally distributed.')
```

    We have reason to suspect the residuals are not normally distributed.

The model was roughly able to calculate the coefficients of errors

## Application - Fitting a MA(1) model on Tesla log returns

In this section, we apply the above technique to fit a MA(1) model on the log returns of TSLA (Tesla Inc.) stock data and see how good can we explain the changes. We use last 5 years EOD stock price data of the TSLA stock which could be downloaded from Yahoo Finance.

```python
#Downloading the data from the directory

tsla_market_data = pd.read_csv("C:\\Users\ku.kulshrestha\Downloads\TSLA.csv")
print(tsla_market_data.head(5))
```

             Date        Open        High         Low       Close   Adj Close  \
    0  2013-06-20  104.650002  107.129997   99.449997  100.650002  100.650002
    1  2013-06-21  103.699997  103.699997   97.500000   99.550003   99.550003
    2  2013-06-24   96.500000  102.870003   95.300003  101.489998  101.489998
    3  2013-06-25  103.099998  104.199997  100.550003  102.400002  102.400002
    4  2013-06-26  103.800003  105.870003  102.660004  105.720001  105.720001

         Volume
    0  10106500
    1  11718600
    2   7119800
    3   5848700
    4   6602600

Since we would only be dealing with the price at the end of the day, hence we only need to take care about Close column of the above dataset and then calculate the log returns of the closing price.

```python
tsla_ts_data = tsla_market_data[['Date', 'Close']]

#Adding a new col in the dataframe
tsla_ts_data['Log_return'] = np.nan
tsla_ts_data.head()
for i in range(1, len(tsla_ts_data)):
    tsla_ts_data.loc[i, 'Log_return'] = np.log(np.divide(tsla_ts_data.loc[i,'Close'], tsla_ts_data.loc[i-1, 'Close']))

#removing the first row as Log_return cannot be defined for it
tsla_ts_data = tsla_ts_data.dropna()
tsla_ts_data.head()
```

      Date
      Close
      Log_return

      1
      2013-06-21
      99.550003
      -0.010989

      2
      2013-06-24
      101.489998
      0.019300

      3
      2013-06-25
      102.400002
      0.008926

      4
      2013-06-26
      105.720001
      0.031907

      5
      2013-06-27
      109.250000
      0.032845

```python
#Plotting the distribution of the log returns
tsla_ts_data.Log_return.plot(figsize = (10,8))
```

Now we have our data ready, we will fit a simple MA(1) model on the Log_return data.

```python
# Fitting a MA(1) model on the TSLA log returns

max_lag = 30
model = smt.ARMA(tsla_ts_data.Log_return, order = (0,1)).fit(maxlag = max_lag,
                                                            method = 'mle',
                                                            trend = 'nc')
print(model.summary())
_ = tsplot(model.resid, lags = max_lag)
```

We can see some peaks in ACF plot at k = 8,9,12,16,19. Trying MA(2) now.

```python
# Fitting MA(2) model on TSLA log returns

model2 = smt.ARMA(tsla_ts_data.Log_return, order = (0,2)).fit(maxlag = max_lag,
                                                             method = 'mle',
                                                             trend = 'nc')

print(model2.summary())
_ = tsplot(model2.resid, lags = max_lag)
```

We see very marginal peaks at k = 16, 19. This concludes that MA(2) model is capturing a lot of the autocorrelation, but not all long-memory effects. Even if we keep increasing the order of the model, we would still see the peaks because we'll be adding a new parameter to a model that has seemingly explained away much of the correlations at shorter lags, but that won't have much of an effect on the longer term lags.

It is highly unlikely that a simple MA(q) model explains all the serial correlation for this data.

```python
from statsmodels.stats.stattools import jarque_bera

score, pvalue, _, _ = jarque_bera(model2.resid)

if pvalue < 0.10:
    print('We have reason to suspect the residuals are not normally distributed.')
else:
    print('The residuals seem normally distributed.')
```

    We have reason to suspect the residuals are not normally distributed.

**Conclusion: **

We have analyzed fitting Tesla Log returns to the AR(p) and MA(q) models where p and q are the order of the AR/MQ models respectively. Both of these models are capable in explaining some of the autocorrelation in the residuals. But some long-memory effects still remains and there is still scope of improvement.


 The Bare Minimum about Floating-Point

Warning: Reading this requires care.

To a first approximation, no-one likes thinking carefully about floating-point numbers.  Most people think that floats are finicky, tedious, and boring. Isn't that why we have computers, to do that sort of thing? Most everyone just wants floating-point arithmetic to work, fast and without giving problems.  But there comes a time when you have to think about them.  This notebook tries to give the absolute minimum that you need to know, so that (most of the time) you can forget about them (but still be ready when their peculiarities strike). In fact, we're going to assume that you have already seen something weird about decimal arithmetic on a computer, and are looking here for answers.  You'll get them, but it's not that simple.

To make it more complicated, there is a standard for floating-point arithmetic laid out by the IEEE that is now over 35 years old : it's very well-known, and very very good; but unfortunately,  not everyone knows about it or follows it, and so not everyone supports it (we do---it makes life so much easier). Indeed a lot of machines and computer languages do, and so we use it here, because you need to know something about it.  But when it is not supported, it's like barking your shin on a piece of furniture in the dark: not fun.  Sadly, we will see an example below, with Python native floating-point arithmetic (which is, quite surprisingly, not the same as NumPy floating-point arithmetic, which is different from SymPy floating-point arithmetic.  Yay.).

Reading about this topic requires close attention.  You have to read every word, and not just the words, but also the actual code and its output.  You have to look at the graphs and think about why they look the way they do (instead of how they would look if we used theoretical real arithmetic, or arbitrary precision arithmetic).  You have to look at the fine details of the numerical output (and sometimes you can only see what's going on if you use hexadecimal notation).  Understanding floating-point arithmetic even requires a painful reversal of understanding and unlearning some things.  Sometimes, this will not seem like fun.  (...although, there's a certain satisfaction in understanding what's really going on.)

Most writers don't like it, either.  Most start off thinking that they can convey the essentials of floating point in a really short time and space with a minimum of fuss, concentrating on just exactly what you can use to avoid learning all the horrible details. Um, no. We can't do it either.  This section wound up being much longer than we wanted to write, and it's almost certainly longer than you want to read.  And, it's hard to read, too, being full of ugly details.  Well, we'll try to point you at some useful videos, too.  You can start with one by  Computerphile , which is very good.  That may help you to read what follows.  But, be warned: you'll need to take a deep breath each time you start to read again.

So if (when!) you want a break from learning about floating-point arithmetic, go for it.  Take a break, then come back and re-read. For whimsical relaxation that's still (sort of) on topic, we suggest listening to the music of  Floating Points . We think it's seriously good: Sam Shepard, the artist, has a PhD in Neuroscience from University College London; whether he chose the name with any reference to floating-point arithmetic we don't know, but maybe.  Listening might help you to come back in the mood for more details.  Or, well, there's always  Rachmaninov, played by the great Leslie Howard.

 What you will want to get from this notebook

 Why we use floating-point arithmetic, even though it causes headaches (In short: it forestalls or alleviates much worse trouble)

 A sense in which floats are better models of reality than the real numbers are

 How to recognize when floating-point effects ("rounding errors") are adding up and making a difference

 What's a rounding error, anyway?

 What happens when your significant figures cancel each other, and other ways to lose accuracy

 The difference between accuracy and precision

 How to predict when problems might arise, and when instead to be confident (actually, most of the time!)

 How to model floating-point arithmetic with the epsilon-delta tools from real analysis (yes, that's right: the tools of real analysis make numerical analysis easier---don't be dismayed, this is actually kind of cool).

 The basic constants: unit roundoff $\mu$, machine epsilon $\varepsilon$, the largest representable real number (often called realmax), the smallest representable positive real number (often called realmin)

 What happens when the numbers overflow (get bigger than realmax) or underflow (get smaller than realmin)

 Why machine epsilon $\varepsilon$ is usually more important than the (much smaller) realmin.

 Where to look for more information (when you wind up really needing it).

Yes,  Wikipedia  is a really good place to start (although, good grief, there is a lot of information densely packed there).

Then there is the Python floating-point tutorial

and the old classic What every computer scientist should know about Floating-Point Arithmetic.

An excellent slim textbook (maybe the best reference) is Michael Overton, Numerical Computing with IEEE Floating-Point Arithmetic.

 Many references at  William (Velvel) Kahan's page

 The wonderful book Accuracy and Stability of Numerical Algorithms by Nicholas J. Higham

 The book  A Graduate Introduction to Numerical Methods  by Corless and Fillion.

 Floats versus the Real World

Let's begin by thinking of something really, really small: the nanofibers making up the fine strands of the finest spider silk, that of the (venomous, of course) brown recluse spider.  According to  this article from ACS Macro Letters  which has some really cool pictures (of fine silk fibres, not so many of spiders, thankfully enough if you don't like spiders much), the nanofibers average about $20$nm in width---in scientific notation, that is about $20\cdot 10^{-9}$ metres wide.  Very hard to see, even with the best equipment.

As an aside, if you do like spiders, we recommend some really cool videos with dancing Peacock Spiders (genus Maratus) by Jürgen Otto.  The spiders are typically smaller than your little fingernail (length 4--5mm), absolutely harmless, and really cool to watch.  Anyway, back to work. (Sorry if we sent you down a rabbit hole (spider hole)).

Machine epsilon (we'll learn what that means) for IEEE Standard 754 double precision floating point arithmetic is $2^{-52} \approx 2.2\cdot 10^{-16}$, or about $20 \cdot 10^{-17}$.  Machine epsilon, compared to the unit $1$, is therefore $10^8$ (one hundred million) times finer than those nanofibers of spider silk are when compared to a metre.  You can compare double precision machine epsilon to the size of a proton, even, which has a width about $10^{-15}$m.  In other words, double precision can resolve things about ten times smaller than a proton, compared to everyday scales.

 A basic fact:  Floating-point numbers are represented by a fixed number of bits.  IEEE double-precision numbers use just $64$ bits, which means that there are at most $2^{64} \approx 1.85\cdot 10^{19}$ distinct such numbers.  (Actually there are fewer than that, because there are some special codes sprinkled in there).  That sounds like a lot, but it means that there is a largest representable number, a smallest positive number, and other strange things that are not true of the real number system, including weirdnesses such as nonzero numbers $x$ that are so small that adding them to $1$ doesn't change it: $1 + x = 1$ with $x$ not being zero.  This causes brain-hurt, so if you want to take a break for a bit, go ahead.

(In fact, the machine epsilon $\varepsilon$ is often (sort of) defined as the smallest machine number for which $1+\varepsilon$ winds up being bigger than $1$; it's actually a bit trickier than that, as you will see below.)

```python
import numpy as np
import sys

unity = 1.0
small = 1.0e-22
ouch = unity + small
print( 'one plus 10**(-22) equals ', ouch, ' in floats; in hexadecimal, we can see it really is:', ouch.hex() )
eps = sys.float_info.epsilon
print('machine epsilon in decimal is ', eps, 'and in hex it is 2**(-52): ', eps.hex() )
ah = unity + eps
print( 'one plus machine epsilon equals ', ah, 'in floats: in hex we can see it exactly:', ah.hex())
eh = unity + 2*eps/3
print( 'adding 2/3 of epsilon to one gets', eh.hex())
uh = unity + eps/2
print( 'adding half of epsilon to one gets ', uh.hex() )
```

    one plus 10**(-22) equals  1.0  in floats; in hexadecimal, we can see it really is: 0x1.0000000000000p+0
    machine epsilon in decimal is  2.220446049250313e-16 and in hex it is 2**(-52):  0x1.0000000000000p-52
    one plus machine epsilon equals  1.0000000000000002 in floats: in hex we can see it exactly: 0x1.0000000000001p+0
    adding 2/3 of epsilon to one gets 0x1.0000000000001p+0
    adding half of epsilon to one gets  0x1.0000000000000p+0

Most floating-point arithmetic nowadays uses double precision; sometimes double-double (or "quad") precision is used, which is absurdly more precise; less frequently but sometimes just single precision is used, which has machine epsilon $2^{-23} \approx 1.2\cdot 10^{-7}$ (so about six times the width of an average spider nanofibre); and nowadays especially in applications where there is a lot of data, half-precision can be used, which has machine epsilon $2^{-10} \approx 10^{-3}$; note that the average width of a human hair is about $1.8\cdot 10^{-4}$m, so half precision has a fineness about five human hairs' width.  Compare that to spider silk, and we see that the comparison human hair::half precision is more or less the same as spider fibre::single precision.  Double precision takes that down to nuclear structure fineness.

So how could that degree of fineness ever cause problems in ordinary arithmetic?  Well, most of the time---really, very much most of the time---it does not.  The IEEE 754 standard was designed to do as good a job as possible, given the constraint that we want memory access to be predictable (and therefore memory operations on computers will be fast).

But sometimes it goes off the rails, and we need to understand how, and why.

 Rounding Errors Can Grow in Relative Importance (sometimes quickly)

The fundamental reason it goes off the rails is that it has to.  We give you no less an Authority than Aristotle himself: in "On the Heavens" written 350BCE we find

 Admit, for instance, the existence of a minimum magnitude, and you will find that the minimum which you have introduced, small as it is, causes the greatest truths of mathematics to totter.

What could this mean for modern computers?  Consider the following simple little bits of arithmetic.

```python
 ( -7.35e22 + 7.35e22) + 100.0
```

    100.0

```python
 -7.35e22 + (7.35e22 + 100.0 )
```

    0.0

Those answers are different, although to your eyes those instructions should give the same answer.

All we did there was do the operations in a different order.  Real arithmetic is associative: (a+b)+c = a+(b+c), end of story.  But in floating-point, this is not so.  Both those answers are correct (even IEEE standard!), and actually what we want when we do floating-point arithmetic (that may be hard to believe, when the second answer is so clearly "wrong" in real arithmetic).

But ok, you say.  Big deal, you say. Those numbers are different on a nuclear scale.  Or, rather, on an astronomical scale: if the $100$ represents the mass of a person in kilograms, then $7.35e22$ is about the mass of the Earth's Moon.  If we threw an anti-moon at the moon and it all vanished in a blaze of energy, it wouldn't matter much if there was a person standing on the moon or not when that happened; but if you put the human there after the anti-moon destroyed the moon, you'd still have the person. (Well, if the spacesuit was good, anyway.)

Incidentally, the main benefit of floating-point arithmetic (as opposed to fixed-point arithmetic) is the dynamic range. It is equally useful talking about things on astronomical scales, normal scales, or nuclear scales, or indeed even more scales: there is an exponent, as in scientific notation (which floating-point translates to, readily), and a number of significant "digits" (actually, bits) called the significand or mantissa.  More soon.

The really annoying thing is that floating-point differences (called "rounding errors") can sometimes add up rapidly and show up unexpectedly, even with very few innocuous-looking operations.  Let's look at another example, namely plotting $(x-1/3)^2$,  and let's zoom in on the region around $x=1/3$.  We all remember from high school (right, we all remember) what the graph  should look like, and indeed on a "human" scale it does:

```python
from matplotlib import pyplot as plt
```

```python
poly = np.poly1d([1.0, -2.0/3.0, 1.0/9.0])
x = np.linspace(-1/3,1)
y = poly(x)
plt.plot(x,y)
plt.show()
```

Now let's zoom in: instead of a nice smooth curve (even on a nuclear scale) we see flat spots---on the spider silk scale in the x-direction!

```python
poly = np.poly1d([1.0, -2.0/3.0, 1.0/9.0])
x = np.linspace(1/3-1.0e-8,1/3+1.0e-8)
y = poly(x)
plt.plot(x,y,".")
plt.show()
```

That's actually not bad, and kind of pretty.  You can see that the floating-point numbers, which are discrete, seem to be trying to represent a nice smooth function. There are no double-precision floating point numbers on the vertical scale between the flat layers, but in spite of that the sense of the curve is conveyed fairly well.

Let's try a more complicated polynomial (after all, for that last example we hardly needed a computer).  This next one one is $(3x-1)^5(x-2)$, expanded.  On a human scale, all looks more or less well.

```python
poly = np.poly1d([243., -891., 1080., -630., 195., -31., 2.])
x = np.linspace(0,1)
y = poly(x)
plt.plot(x,y)
plt.show()
```

Now zoom in on the zero at $x=1/10$.

```python
poly = np.poly1d([243., -891., 1080., -630., 195., -31., 2.])
c = 0.3333
delt = 4.0e-4
x = np.linspace(c-delt,c+delt,2021)
y = poly(x)
plt.plot(x,y,"k,")
plt.xticks([c-delt, c, c+delt])
plt.show()
```

Now, on the scale of about the width of four or five human hairs (even though we are using nuclear-fine arithmetic!), we see all kinds of weirdness.  This is what rounding error looks like, when you zoom in. (Don't connect the dots, it just looks like a mess.)

Even though the values are all fairly small on the $y$-axis, this is kind of a disaster.  Evaluating that polynomial takes only six operations at each point; we have evaluated it at each of $2021$ points, near to the zero at $x=1/3$.  But deciding from the plot where the zero is is now pretty difficult.  If we didn't know ahead of time where it was, we'd be out of luck.  Even next-door points might go seemingly unpredictably to one flat layer or another.  The x-scale is $\pm 4\cdot 10^{-4}$, about the width of a few human hairs, as we said.

And we haven't gone beyond a degree six polynomial, and the coefficients are all modestly sized.  How much more difficulty are we going to find in real-life-sized problems?

Actually, there are some real difficulties with floating-point in real-life problems. Maybe the most famous is the loss of the Ariane 5 although that was a conversion error that caused a cascading failure, not a rounding error.  But floating-point doesn't cause all that many problems, really, and most of those are going to show up in obvious ways.  But there are a nasty few that can cause the worst kinds of problems and give you wrong but plausible answers.  These are the ones  you need to know about.

 An example with no subtractions

Here's a nasty one.  Take a simple number, such as 6.0.  Take its square root, then take the square root of that, and so on 52 times.  Then take the result of that, and square it.  Then square that, then square the result, and so on 52 times, again.  Square root and square are inverse operations, so we ought to get back to where we started from.  We don't, and this time, even though we are using nuclear-scale precision, we get answers that are wrong on a human scale (never mind being off by a hair, we are off by almost fifty percent).  This result, being done with such simple operations, ought to shock you.

The appearance of a number so close to $e$, the base of the natural logarithms, is not an accident; but we won't stop to explain that here.

```python
x = 6.0
for i in range(52):
  x = np.sqrt(x)
  print( 'After ', i+1, 'square roots, x is ', x, 'or in hex ', x.hex() )

for i in range(52):
  x = x*x

print( 'x started as 6.0, and is now ', x)
```

    After  1 square roots, x is  2.44948974278 or in hex  0x1.3988e1409212ep+1
    After  2 square roots, x is  1.56508458007 or in hex  0x1.90a9620ee37f6p+0
    After  3 square roots, x is  1.25103340486 or in hex  0x1.4043b9a812c08p+0
    After  4 square roots, x is  1.11849604597 or in hex  0x1.1e55c1c229b25p+0
    After  5 square roots, x is  1.05758973424 or in hex  0x1.0ebe3369277b9p+0
    After  6 square roots, x is  1.02839181942 or in hex  0x1.0744afafddae3p+0
    After  7 square roots, x is  1.0140965533 or in hex  0x1.039bd4eb69864p+0
    After  8 square roots, x is  1.00702361109 or in hex  0x1.01cc4ca3f2b22p+0
    After  9 square roots, x is  1.00350566072 or in hex  0x1.00e5bf3a22ebfp+0
    After  10 square roots, x is  1.00175129684 or in hex  0x1.0072c5e2a60ccp+0
    After  11 square roots, x is  1.00087526537 or in hex  0x1.00395c8428e1ap+0
    After  12 square roots, x is  1.00043753697 or in hex  0x1.001caca6f7f55p+0
    After  13 square roots, x is  1.00021874456 or in hex  0x1.000e55ecba9cfp+0
    After  14 square roots, x is  1.0001093663 or in hex  0x1.00072adcadaf2p+0
    After  15 square roots, x is  1.00005468165 or in hex  0x1.00039567eb06cp+0
    After  16 square roots, x is  1.00002734045 or in hex  0x1.0001cab25a921p+0
    After  17 square roots, x is  1.00001367013 or in hex  0x1.0000e558c68d1p+0
    After  18 square roots, x is  1.00000683504 or in hex  0x1.000072ac49979p+0
    After  19 square roots, x is  1.00000341752 or in hex  0x1.000039561e601p+0
    After  20 square roots, x is  1.00000170876 or in hex  0x1.00001cab0d952p+0
    After  21 square roots, x is  1.00000085438 or in hex  0x1.00000e558663dp+0
    After  22 square roots, x is  1.00000042719 or in hex  0x1.0000072ac3184p+0
    After  23 square roots, x is  1.00000021359 or in hex  0x1.000003956185bp+0
    After  24 square roots, x is  1.0000001068 or in hex  0x1.000001cab0c14p+0
    After  25 square roots, x is  1.0000000534 or in hex  0x1.000000e558604p+0
    After  26 square roots, x is  1.0000000267 or in hex  0x1.00000072ac300p+0
    After  27 square roots, x is  1.00000001335 or in hex  0x1.0000003956180p+0
    After  28 square roots, x is  1.00000000667 or in hex  0x1.0000001cab0c0p+0
    After  29 square roots, x is  1.00000000334 or in hex  0x1.0000000e55860p+0
    After  30 square roots, x is  1.00000000167 or in hex  0x1.000000072ac30p+0
    After  31 square roots, x is  1.00000000083 or in hex  0x1.0000000395618p+0
    After  32 square roots, x is  1.00000000042 or in hex  0x1.00000001cab0cp+0
    After  33 square roots, x is  1.00000000021 or in hex  0x1.00000000e5586p+0
    After  34 square roots, x is  1.0000000001 or in hex  0x1.0000000072ac3p+0
    After  35 square roots, x is  1.00000000005 or in hex  0x1.0000000039561p+0
    After  36 square roots, x is  1.00000000003 or in hex  0x1.000000001cab0p+0
    After  37 square roots, x is  1.00000000001 or in hex  0x1.000000000e558p+0
    After  38 square roots, x is  1.00000000001 or in hex  0x1.00000000072acp+0
    After  39 square roots, x is  1.0 or in hex  0x1.0000000003956p+0
    After  40 square roots, x is  1.0 or in hex  0x1.0000000001cabp+0
    After  41 square roots, x is  1.0 or in hex  0x1.0000000000e55p+0
    After  42 square roots, x is  1.0 or in hex  0x1.000000000072ap+0
    After  43 square roots, x is  1.0 or in hex  0x1.0000000000395p+0
    After  44 square roots, x is  1.0 or in hex  0x1.00000000001cap+0
    After  45 square roots, x is  1.0 or in hex  0x1.00000000000e5p+0
    After  46 square roots, x is  1.0 or in hex  0x1.0000000000072p+0
    After  47 square roots, x is  1.0 or in hex  0x1.0000000000039p+0
    After  48 square roots, x is  1.0 or in hex  0x1.000000000001cp+0
    After  49 square roots, x is  1.0 or in hex  0x1.000000000000ep+0
    After  50 square roots, x is  1.0 or in hex  0x1.0000000000007p+0
    After  51 square roots, x is  1.0 or in hex  0x1.0000000000003p+0
    After  52 square roots, x is  1.0 or in hex  0x1.0000000000001p+0
    x started as 6.0, and is now  2.71828180818

It's easier to see what's happening in hex; after a while, not enough decimals are printed, base 10, but in hex we see everything. We'll see in the next cell just what those hex symbols mean.

We did not print the results when squaring---they're kind of the same but backwards, and different in detail.  We have all we need, now, to explain what is going on.

The problem is loss of precision causing loss of accuracy.  The information about where we started from ($x=6.0$) is carried in hex digits further and further to the right, as we progressively take square roots, as you can see above as the zeros creep out from the "decimal" point.  A similar thing happens in decimal:  That information drops off the right end of the numbers each time the result is stored in the variable $x$.  When we start squaring again, we are starting from a number that has lost almost all information about where it started.  Indeed, one more square root and it would have rounded down to exactly $1$, and then when we started squaring we wouldn't have got anywhere.

This loop is a simple iteration, executing a very simple dynamical system such as are studied frequently in real-life applications.  There are other examples of similar disasters.

 What are the rules?

Computers work in binary.  Hexadecimal is basically the same thing except four times shorter: one hex digit takes four bits, so 64 bits in binary is only 16 hexadecimal places.  An IEEE 754 double precision float occupies 64 bits, or 16 hex digits.  Let's try to follow the addition $\sqrt{2} + \pi$ in double precision.

```python
import math
root2float = np.sqrt(2.0)
pifloat = math.pi
modest = 100.35
print( root2float.hex() )
print( pifloat.hex() )
print( modest.hex() )
```

    0x1.6a09e667f3bcdp+0
    0x1.921fb54442d18p+1
    0x1.9166666666666p+6

What you are looking at there is a human-readable version of the internal representation of the two floating-point approximations to $\sqrt2$ and to $\pi$, together with a modestly bigger number chosen more or less at random.  Neither of the first two numbers can be represented exactly as a finite decimal, binary, or hexadecimal number, of course (although both are "computable" in a technical sense: we can get as many digits of either as we could possibly want).  The third has a terminating decimal representation but a nonterminating hex representation (this was a lucky accident). Asking Matlab to use "format hex" actually shows the internal representations: $\sqrt2$ is represented as "3ff6a09e667f3bcd" using exactly 16 hexadecimal digits, and $\pi$ is represented as "400921fb54442d18" which at first glance look baffling, and bafflingly different to those representations above.  But the first three hex "digits" of each of those are used for the sign bit and the 11-bit exponent in a biased format which we'll ignore for a moment.  After those three "digits" we see the sequence "6a09e667f3bcd" from Matlab and that's exactly the string after the "decimal" point for $\sqrt2$; similarly, the $13$ hexadecimal digits after the "decimal" point for $\pi$ show up in the Matlab string as the final $13$ hexadecimal digits in Python.

Hexadecimal digits are in a one-to-one correspondence with the following numbers.  We put them in decimal--hex--binary.
Here they are typed by hand:

00--1--0000

01--1--0001

02--2--0010

03--3--0011

04--4--0100

05--5--0101

06--6--0110

07--7--0111

08--8--1000

09--9--1001

10--a--1010

11--b--1011

12--c--1100

13--d--1101

14--e--1110

15--f--1111

and here they are tabulated  automatically (with decorations 0x meaning hex and 0b meaning binary).  The leading binary 0 bits are trimmed below, which makes the correspondence with hex somewhat less intelligible: four bits to a hex digit is the point.

```python
from tabulate import tabulate

results = [(n, hex(n), bin(n)) for n in range(16)]
print(tabulate(results, headers=["decimal", "hex", "binary"]))

```

      decimal  hex    binary
    ---------  -----  --------
            0  0x0    0b0
            1  0x1    0b1
            2  0x2    0b10
            3  0x3    0b11
            4  0x4    0b100
            5  0x5    0b101
            6  0x6    0b110
            7  0x7    0b111
            8  0x8    0b1000
            9  0x9    0b1001
           10  0xa    0b1010
           11  0xb    0b1011
           12  0xc    0b1100
           13  0xd    0b1101
           14  0xe    0b1110
           15  0xf    0b1111

You can also use capital letters, A (hex) for 10 (decimal) and 1010 (binary), and so on, to F for 15.  16 in decimal is thus 10 in hex or 10000 in binary.  Translating groups of four bits to hex is easy, and translating a hex "digit" to binary is just as easy.  Binary to decimal and decimal to binary (with fractional parts), on the other hand, is annoyingly full of special cases that cause headaches.

So what does the Python format 0x1.6a09e667f3bcdp+0 mean? This seems weird at first, and then gets a bit weirder, because it actually uses binary in the exponent, p+0 means times 2 to the 0 and p+1 means $\cdot 2^1$ and p+6 means $\cdot 2^6$ so that number is bigger than $64$ (which we knew, of course). As we said before, the 0x means hex.  The 1. means there is a 1 before the "decimal" point.  (For "normal" numbers this is always true so we don't even have to store it---this gives us what is called a "hidden bit").  Then after the "decimal" point comes a string of hex digits, meaning $6/16 + a/16^2 + 0/16^3 + 9/16^4 + e/16^5 + 6/16^6 + 6/16^7 + 7/16^8 + f/16^9 + 3/16^{10} + b/16^{11} + c/16^{12} + d/16^{13}$.  Using exact rational arithmetic, that adds up to $6369051672525773/4503599627370496$ (including the leading 1).

Ick.  There's the other reason we like scientific notation.  It's hard even to see if that exact rational is bigger than $1$ or not.  We'll convert it to decimal in a minute, and we'll see that it makes more sense to humans that way.  If we square it exactly as a rational number (mercifully keeping the computation to ourselves, but go ahead and do it yourself if you like), we get a rational number just a little bigger than $2$; it is $2$ plus $5545866846675497/20282409603651670423947251286016$, which is about $2.73\cdot 10^{-16}$.  It's no coincidence that this is about the same size as the machine epsilon for double precision.  If we convert the rational number $6369051672525773/4503599627370496$ to a decimal number, we get $1.4142135623730951\ldots$ which, if we were going to report only to five significant figures, we would round to $1.4142$. That accuracy after rounding is enough to get us the distance corner-to-corner of a square one kilometer on a side accurate to better than $10$cm.  If we hadn't rounded, and instead used all figures, it would be precise to a tenth of a billionth of a millimeter.  We don't say accurate there because how could we measure the angles of a square to that much accuracy? That's about 100 protons wide!  Precision can be spurious.

But keeping all precision during the computation guards against growth of rounding errors.  So, as a general rule, try to convert to decimals and round your answers only at the very end.

Now that we have talked about binary, hexadecimal, and the fact that we have to approximate we can talk about simple arithmetic.  Here's the IEEE rule:

 Do the arithmetic (+, -, *, divide) on two operands exactly correctly, then round the results to the nearest machine representable number.  In the case of a tie, round to the nearest machine representable number whose last bit is 0.

This tie-breaking rule (which only rarely happens) is called "round-to-even" and is done to eliminate bias in long computations. (There is only one place in this notebook where we know that it happened.  Good exercise to find it.)

This is a pretty good principle, and really, how could it be better?  It's saying that the only error that's allowed is the one at the end, which you can't escape, once you have decided to store your numbers in just 64 bits (or 32 if you're using single precision, or 16 if you're using half precision).

```python
root2float = np.sqrt(2.0)
pifloat = math.pi
added = root2float + pifloat
print( root2float.hex() )
print( pifloat.hex() )
print( 'sqrt(2) + pi = ', added, 'which is a few decimals translated from ', added.hex(), 'exactly in the internal hex' )
```

    0x1.6a09e667f3bcdp+0
    0x1.921fb54442d18p+1
    sqrt(2) + pi =  4.55580621596 which is a few decimals translated from  0x1.2392543c1e57fp+2 exactly in the internal hex

You should notice that Python only prints 12 significant figures there in decimal (seems to be a default printing width. In Maple, working to 20 decimal digits, we get $4.5558062159628882873$).  In order to reliably convert binary (or hex) to and from IEEE double precision and back again, to get exactly the same number, you have to keep about $18$ decimal places (not just sixteen as suggested by the machine epsilon; we're not sure where "twelve" comes from in that example).  There are worse problems with conversion to decimal, such as the one called "The Table-Maker's Dilemma" which has to do with how many bits you have to carry when evaluating an elementary function just to be sure you are going to round correctly: it's more than you think, and (perhaps very surprisingly) for some functions the full answer is not even known! But we are not going to go into that.  That's a hard problem for mathematical software designers.  We're just going to walk by, whistling.

Back to the sum $\sqrt2 + \pi$ above.  That hex answer is the exact sum of the two rational numbers represented in hex by 0x1.6a09e667f3bcdp+0 and 0x1.921fb54442d18p+1, correctly rounded back into the IEEE double precision format.  You can see already that there have been some changes from real arithmetic.  First, $\sqrt2$ had to be rounded, to $13$ hex digits plus a hidden bit; then $\pi$ had to suffer the same kind of rounding.  Then even though the arithmetic is carried out correctly, we aren't adding $\sqrt2$ and $\pi$ any more, we're adding the rational approximations thereof.  Then one final rounding happens when we put the result of that addition into a double precision float.

The details of how computer manufacturers guarantee that the arithmetic is done correctly, and the correct rounding is done, are beyond the scope of almost anyone's interest.  We are willing to trust the manufacturers to do it right.  But  remembering the Intel Pentium bug uncovered by Professor Thomas R. Nicely  we remember that sometimes that trust is misplaced.  Intel paid over a third of a billion US dollars at the time on the recall.  Ow.

Translating the IEEE guarantee into the language of real analysis

We have the following rules: if $\varepsilon$ is the machine epsilon, which is the distance to the next larger representable machine number (after $1$, although we can talk about this for other numbers as well), and $\mu$ is the unit roundoff, which is half of that, namely $\mu = \varepsilon/2$, then we have the mathematical statements: if the results do not overflow or underflow then

 fl$(a+b) = (a+b)(1+\delta)$ where $\delta$ is some real number at most as big as $\mu$: $|\delta| \le \mu$

 fl$(a-b) = (a-b)(1+\delta)$ where $\delta$ is some real number at most as big as $\mu$: $|\delta| \le \mu$

 fl$(a*b) = (a\cdot b)(1+\delta)$ where $\delta$ is some real number at most as big as $\mu$: $|\delta| \le \mu$

 fl$(a/b) = (a/b)(1+\delta)$ where $\delta$ is some real number at most as big as $\mu$: $|\delta| \le \mu$

That last rule applies only if $b \ne 0$.  The notation fl$(\cdot)$ means "the floating-point result of".

A result overflows if it is larger than realmax.  A result underflows if it is positive but smaller than realmin. On reflection, the same thing happens for negative numbers.  (Complex numbers are, well, slightly more complex.)

Using these rules, we can prove some facts about floating-point arithmetic.  For instance, if $a$, $b$ and $c$ are three positive real numbers, then if operations are performed left-to-right fl$(a+b+c) = $fl(fl$(a+b)+c) = ((a+b)(1+\delta_1)+c)(1+\delta_2)$.  From there you can expand to get $(a+b+c) + $ rounding error; the rounding error can be written as a formula, which is $(a+b+c)\delta_2 + (a+b)\delta_1(1+\delta_2) $. By the triangle inequality, the absolute value of the error is less than $|(a+b+c)\delta_2| + |(a+b)\delta_1(1+\delta_2)|$. Since $a$, $b$, and $c$ are positive, this is less than $(a+b+c)(|\delta_2| + |\delta_1(1+\delta_2)|$, which by hypothesis is less than $(a+b+c)(\mu + \mu(1+\mu))$ or $(a+b+c)(2\mu + \mu^2)$.  This means that the relative error in adding three positive numbers is, ignoring the $\mu^2$ term, $2\mu$.  This is a nice result and says that for adding positive numbers, the errors grow essentially only linearly.

That kind of analysis is not to everyone's taste.  You should try to see what happens when one of the $a$, $b$, or $c$ is zero, though.  In particular, if $a=-b$, say.  You should be able to convince yourself that the resulting relative error might be infinite!  In fact, this is what happened in our first example, above.

```python
import sys

print( 'The largest representable double precision number is ', sys.float_info.max, 'which in hex is', sys.float_info.max.hex() )
print( 'The smallest normal positive double precision number is ', sys.float_info.min, 'which in hex is ', sys.float_info.min.hex() )
```

    The largest representable double precision number is  1.7976931348623157e+308 which in hex is 0x1.fffffffffffffp+1023
    The smallest normal positive double precision number is  2.2250738585072014e-308 which in hex is  0x1.0000000000000p-1022

 NaN, or what you get when the limits are exceeded

A NaN, or "Not a Number," is a special code which results when you perform something like $0/0$ or $\infty-\infty$ or $0\cdot \infty$.  Maybe surprisingly, IEEE arithmetic can do some useful things with infinity (which can result from overflow).  And even NaNs can be useful: if you compute a whole array of results, with a few NaNs in them, and then plot the results, you still get a plot (maybe missing some spots---the NaNs won't be plotted and will leave holes).

What happens when you try to make a number smaller than realmin?  Say, by squaring realmin?  You get underflow (default behaviour is to replace it by zero).  This is frequently, but not always, what you want.

Sadly, Python actually gives an error message if you try to do 0/0 in its native arithmetic, which we think is nasty.  Doubtless there will be a way to alter that behaviour, but we won't fuss now (numpy helps).  The following statement is a mistake of this author (RMC) caused by his forgetting that ^ in Python does not mean power: We think that Python's $0^0$ is worse---Donald Knuth argues very persuasively that it should be $1$! See  the Wikipedia article  which even says "There do not seem to be any authors assigning a specific value of $0^0$ other than $1$".  End of mistake   When we use the power operator ** correctly, Python is fine.  In my defence (this is RMC speaking, not EYSC) in markdown the ^ is power, as it is in Maple and Matlab and every other language I use.  Just Python now insists on $**$ (I think even Fortran allows ^ now, although it's been decades since my last Fortan program).  Grump, grump.

```python
big = float.fromhex('0x1.fffffffffffffp+1023')
small = float.fromhex('0x1.0000000000000p-1022')
print('Yes, it\'s big: ', big, 'in hex ', big.hex())
# Cut and paste the hex string, then edit the last f to be an e (this is the next smaller number)
notquiteasbig = float.fromhex('0x1.ffffffffffffep+1023')
stepback = big - notquiteasbig
print( 'distance to next smaller number is ', stepback, 'in hex ', stepback.hex() )
print( 'exponent difference ', 1023 - 971 )
jumpingoffthecliff = big + stepback
print('jump over the cliff, whee! ', jumpingoffthecliff, '---in hex: ', jumpingoffthecliff.hex() )

infminusinf = jumpingoffthecliff - jumpingoffthecliff
print('Surely it must be zero: but (this is good) 0*inf is: ', infminusinf )
print('small*big should actually be ok: ', small*big )
print('Here is an underflow: ', small*small )
bad = 0
worse = bad*jumpingoffthecliff
print('What\'s 0*inf = ', worse )
print('what\'s 0^0 (We don\'t like this one)', bad^bad )
print('Well if we use the proper syntax, 0**0 is ', bad**bad )
badfloat = 0.0
print('what\'s 0.0**0.0? ', badfloat**badfloat )
print('But 0/0 generates an error (commented out, so the notebook runs to completion)')
# bad/bad

```

    Yes, it's big:  1.7976931348623157e+308 in hex  0x1.fffffffffffffp+1023
    distance to next smaller number is  1.99584030953472e+292 in hex  0x1.0000000000000p+971
    exponent difference  52
    jump over the cliff, whee!  inf ---in hex:  inf
    Surely it must be zero: but (this is good) 0*inf is:  nan
    small*big should actually be ok:  3.9999999999999996
    Here is an underflow:  0.0
    What's 0*inf =  nan
    what's 0^0 (We don't like this one) 0
    Well if we use the proper syntax, 0**0 is  1
    what's 0.0**0.0?  1.0
    But 0/0 generates an error (commented out, so the notebook runs to completion)

We also wanted to show the internal representation of "inf" when we used .hex(), but instead of "7ff0000000000000" it just gave inf again.  Anyway inf is the normal result when you overflow: you get a "machine infinity".

We see that realmax is about 4 times bigger than realmin is small (their product is 4, but not exactly 4).  That seems weird, until you realize that realmax has a whole string of F's in its expansion and is not quite a pure power of $2$.

You will at first hate NaNs, especially when your output is nothing but NaNs, but eventually come to accept them. They actually are quite useful.

 Revealing earlier rounding errors by subtractive cancellation

Consider the familiar quadratic formula for the roots of a quadratic equation.  Let's make it simpler by considering a quadratic of the form $x^2 - 2b x + 1 = 0$.  Factoring this gets $(x-r_1)(x-r_2)$ where (say) $r_1 = b + \sqrt{b^2-1}$ and $r_2 = b - \sqrt{b^2-1}$.  Feel free to check the algebra, maybe by adding the roots to get $2b$ (as we should) or multiplying them to get $(b+\sqrt{b^2-1})(b-\sqrt{b^2-1}) = b^2 - (b^2-1) = 1$, again as we should.

```python
n = 100
barr = np.logspace( 1.0, 2.0, num=n )
unity = np.array(range(n),dtype=float)
for i in range(n):
    b = barr[i]
    delt = np.sqrt(b*b-1)
    r1 = b + delt
    r2 = b - delt
    unity[i] = r1*r2

plt.plot( barr, unity, '.')
plt.show()
```

```python
n = 250
barr = np.logspace( 6.0, 8.0, num=n )
unity = np.array(range(n),dtype=float)
for i in range(n):
    b = barr[i]
    delt = np.sqrt(b*b-1)
    r1 = b + delt
    r2 = b - delt
    unity[i] = r1*r2

plt.plot( barr, unity, '.')
plt.show()
```

Inspection of those two plots ought to surprise you.  The first one is ok---the results are all $1$ to plus or minus $10^{-12}$, which is maybe a bit bigger rounding error than the $10^{-15}$ or so that we might be expecting, but it's not all that bad.  The product of the two computed roots is $1 \pm O(10^{-12})$.  Fine.

The second plot is, however, a shocker.  The vertical scale runs from $0$ to $3$.  Some of the products of the computed roots are as big as $3$, almost; and many of them are actually zero (which means that one of the computed roots is zero). Some of those blasted straight lines overlap, too.  Those results do not look to the unskilled eye like they reveal any rounding errors.  But that's exactly what's happening.

This, from the venerable quadratic formula, known since Babylonian times.

In detail: for large $b$, $b^2$ will be even larger.  Then, subtracting $1$ from $b^2$ will leave it close to the same; taking the square root again gets it nearly back to $b$, but not quite (even in exact arithmetic).  Then, subtracting this from $b$ will leave a small number.  Let's take a specific example, and track this in hex.

```python
b = 2.0e5
print( 'b is ', b, 'which is ', b.hex(), ' in hex ')
delt = np.sqrt(b*b-1)
print( 'delt is ', delt, 'which is ', delt.hex(), ' in hex ')
r1 = b + delt
print( 'r1 is ', r1, 'which is ', r1.hex(), 'in hex')
r2 = b - delt
print( 'r2 is ', r2, 'which is ', r2.hex(), 'in hex')

```

    b is  200000.0 which is  0x1.86a0000000000p+17  in hex
    delt is  199999.999998 which is  0x1.869fffffeb075p+17  in hex
    r1 is  399999.999997 which is  0x1.869ffffff583ap+18 in hex
    r2 is  2.4999899324e-06 which is  0x1.4f8b000000000p-19 in hex

We see that even for $b = 2\cdot 10^5$ that $\Delta = \sqrt{b^2-1}$ is not so different to $b$ itself, being about $2\cdot 10^5 (1- 10^{-11})$ (if we have counted the decimal 9s correctly).  In hex we see the digits 1.86a000000000000 and 1.869ffffffeb075. The f's are all in binary 1111, so we have a string of binary 1s that is 24 bits long; adding a 1 to the end of that causes a carry to chain all along those 24 bits, making the 9 turn into an a.  Subtracting those two things (which we apparently have to do to get $r_2$) will cancel out the 1.869ffffff with the 1.86a000000 which will cost us nine hexadecimal digits of significance and bring the lower 3 hexadecimal digits up to the front of the new number, the result. That is, we subtracted two nearly equal numbers, each about $2^{17}$, leaving a result about the size of $2^{-19}$ (to see where we got the $17$ and the $19$ just look at the values in the +p parts).  This means we have lost about 36 bits of precision (leaving just approximately 16 bits left).  We have made rounding errors earlier in computing $\Delta$: one when we multiplied the b's together, another when we subtracted the 1, and more when we took the square root.  All that information that would have wound up sitting below the 13 hexadecimal digits was lost when we formed $\Delta$.  Now when we subtract, the first nine hexadecimal digits are the same and cancel, leaving only four good hex digits and those rounding errors to move upward in significance.

This is called catastrophic cancellation when subtraction reveals earlier rounding errors.

Trying it for larger $b$, say $b=10^8$, gets zero for $r_2$ because all the good digits cancel out; then there are no significant digits left.

It might seem like a cheat, but the solution to this for this particular problem is not to do the subtraction at all.  Compute $r_1$ which adds instead of subtracts---there's still a rounding error in $\Delta$ but it's not revealed.  Then compute $r_2$ by $r_2 = 1/r_1$.  This gets us a perfectly accurate small root.

This trick is important enough to be part of the canon.  For more general problems, we have to use other tricks.

Welcome to the seamy underbelly of numerical analysis. (Don't worry too much---most of the time we can treat floats as if they were just real numbers; it's only occasionally that weird stuff happens.)

```python
b = 1.0e16
print( 'b in decimal is ', b, 'and in hex is ', b.hex() )
bigger = float.fromhex('0x1.1c37937e08001p+53')
print( 'The next machine representable number is ', bigger )
b = 1.0
print( 'b in decimal is ', b, 'and in hex is ', b.hex() )
bigger = float.fromhex('0x1.0000000000001p+0')
print( 'The next machine representable number is ', bigger )
print('The machine epsilon at 1 is ', bigger - b )
```

    b in decimal is  1e+16 and in hex is  0x1.1c37937e08000p+53
    The next machine representable number is  1.0000000000000002e+16
    b in decimal is  1.0 and in hex is  0x1.0000000000000p+0
    The next machine representable number is  1.0000000000000002
    The machine epsilon at 1 is  2.220446049250313e-16

```python
#  This will work in Python 3.9; until installed, commented out
#from math import nextafter, inf
#b = 1.0e16
#print( 'b in decimal is ', b, 'and in hex is ', b.hex() )
#bigger = nextafter(b, inf )
#print( 'next floating point number after b is ', bigger, 'and in hex is ', bigger.hex() )

```

Now let's try an example from a 1981 paper by W. Kahan, "Why do we need a floating-point arithmetic standard?" namely the evaluation of the rational function

$$
R(z) = 1 - \frac{3}{z-2 - \frac{1}{z-7 + \frac{10}{z-2 - \frac{2}{z-3}}}}
$$

which we may wish to write as $1 -  1/(z-2 - 1/(z-7 + 10/(z-2 - 2/(z-3))))$ which ought to translate nicely into Python.  Kahan calls this a one-liner, that ought to work just fine if the conventions described above are followed.

Unfortunately, we walk straight into a bog.  Python apparently does not follow the IEEE 754 standard, but instead breaks it on purpose.  On one discussion board from 2008, we find quotes that make RMC groan in pain: "If you're doing such serious number-crunching that you really want to handle NANs, you're probably not writing in Python anyway."

This isn't serious number crunching, it's just wanting to plot something that has removable discontinuities :(

and "For the vast majority of programming, division by zero is a mistake and not merely a degenerate case, so Python decided to treat it like one."

This attitude just makes life harder for Python programmers :(

The Matlab one-line program

R = @(z) 1 -  1/(z-2 - 1/(z-7 + 10/(z-2 - 2/(z-3))))

and the Maple one-line program

R := z -> 1 - 1/(z - 2 - 1/(z - 7 + 10/(z - 2 - 2/(z - 3))))

both work perfectly.  The Maple one is maybe the best, because if you call it with floats, e.g. R(1.0), you get the right answer by the IEEE arithmetic (to the last bit) whereas if you call it with integers, e.g. R(1), you get the error message "division by zero"

However, the plot worked even though it called R(1), R(2), R(3), R(4), and R(5).  This is because numpy (as opposed to plain Python) actually does use NaNs intelligently.  Whew!  And, looking at the plot, numpy got the right answers.  The difference is in the use of arrays, apparently (Thanks to Jack Betteridge for pointing this out).

```python
R = lambda z: 1.0-1.0/(z-2.0-1.0/(z-7.0+10.0/(z-2.0-2.0/(z-3.0))))
n = 13
x = np.linspace(0,6,num=n)
y = np.array(range(n),dtype=float)
for i in range(n):
    y[i] = R(x[i])
    #print( x[i], y[i])
plt.plot( x, y, '.' )
plt.show()

r1 = R( x[2] )
r2 = R( x[4] )
r3 = R( x[6] )
r4 = R( x[8] )
print( 'R(1) = ', r1, 'R(2) = ', r2, 'R(3) = ', r3, 'R(4) = ', r4 )
#np.seterr(divide='ignore')  # This command seems to be ignored its own self :(
worked = R( np.float64(1.0) )
print( 'It worked with NumPy', worked )
print( 'But would fail with native Python floats, which are different for some reason.  Commented out')
#didntwork = R( 1.0 )
# Should have been 2
#R(2)
# Should have been 1
#R(3)
# Should have been 1/5
#R(4)
# Should have been 1/2

```

```python
#
# Here's an example where the 0/0 occurs naturally in a plot
# and we DON'T want to fuss with special cases.  It's nice
# that NaNs allow this to work (in NumPy)

n = 201
x = np.linspace(-4*math.pi,4*math.pi,num=n)
y = np.array(range(n),dtype=float)
#np.seterr(divide='ignore')
for i in range(n):
    y[i] = math.sin(x[i])/x[i]
    #print( x[i], y[i])
plt.plot( x, y  )
plt.show()

```

The Bare Minimum about Floating Point Arithmetic.ipynb


# INF-510, v0.22, Claudio Torres, ctorres@inf.utfsm.cl. DI-UTFSM
## Textbook: Lloyd N. Trefethen, Spectral Methods in MATLAB, SIAM, Philadelphia, 2000

# Differentiation by Interpolation

```python
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import scipy.sparse.linalg as sp
import sympy as sym
from scipy.linalg import toeplitz
import ipywidgets as widgets
import matplotlib as mpl
mpl.rcParams['font.size'] = 14
mpl.rcParams['axes.labelsize'] = 20
mpl.rcParams['xtick.labelsize'] = 14
mpl.rcParams['ytick.labelsize'] = 14
sym.init_printing()
```

Defining symbolic variables

```python
xj, h, x = sym.symbols('xj h x', reals=True)
u1, u2, u3, u4, u5 = sym.symbols('u1 u2 u3 u4 u5', reals=True)
```

## Do you remember Lagrange interpolation? Please do

```python
# The little Lagrange polynomials
def l_Lagrange(X,x,i):
    l=1
    n=len(X)
    for k in np.arange(i):
        l*=(x-X[k])
    for k in np.arange(i+1,n):
        l*=(x-X[k])
    return l
# The Lagrange polynomials
def L_Lagrange(X,x,i):
    num = l_Lagrange(X,x,i)
    den = l_Lagrange(X,X[i],i)
    L=num/den
    return L
# The Lagrange interpolation
# X : x - data points
# x : the symbolic variable
# Y : y - data points
def P_Lagrange(X,x,Y):
    P=0
    n=len(X)
    for i in np.arange(n):
        P+=Y[i]*L_Lagrange(X,x,i)
    return P
```

## Just a small example of a interpolation of 2 points

```python
# x - data points
X=(0,1,2,3,4,5,6)
# y - data points
Y=(0,1,-2,1,0,2,7)
# The interpolation
P=P_Lagrange(X,x,Y)
# Just showing the polynomial. Do you see the Lagrange polynomials?
P
```

```python
# And a simplified version of the polinomial!
sym.simplify(P)
```

Plotting what we got

```python
# We need to plot in a finer grid
xx=np.linspace(min(X),max(X),1000)
# Here we create a lambda function for our interpolation
Pl=sym.lambdify(x,P)
# The next line vectorize the lambda function, it is not need here but it can be used
# Plv=np.vectorize(Pl)
plt.figure()
# Plotting the interpolated data
plt.plot(xx,Pl(xx),label='$P(x)$')
# Plotting the data points
plt.plot(np.array(X),np.array(Y),'k.',markersize=20,label='Data points')
plt.grid(True)
plt.ylim([np.min(Pl(xx))-1,np.max(Pl(xx))+1])
plt.xlabel('$x$')
plt.ylabel('$y$')
plt.legend(loc='best')
plt.show()
```

## Let's find the 4th order differentiation approximation

```python
# x - data points. Notice that they are all symbolic variables!
X=(xj-2*h,xj-h,xj,xj+h,xj+2*h)
# y - data points. Also symbolic variables!
Y=(u1,u2,u3,u4,u5)
## In-class
#X=(xj-h,xj,xj+h)
#Y=(u1,u2,u3)
# The Lagrange interpolation still works!
P=P_Lagrange(X,x,Y)
P
```

```python
# Here we compute the derivative of the polynomian P(x), i.e. we obtain P'(x) and we definy it as Pp (P-prime).
Pp=sym.diff(P,x)
Pp
```

```python
# Now we evaluate P'(x) at x=x_j.
PpE=Pp.subs({x:xj})
PpE
```

# Class work

## How to you find a 6th order approximation? Please do it

```python

```

## How do you find a second order approximation on a left boundary? i.e. say you have $u_1(x_1)$, $u_2(x_2)$ and $u_3(x_3)$, interpolate them and compute the derivative at $x_1$.

```python

```

SC5/01 Differentiation by Interpolation.ipynb


In this notebook, we use the package [`bseries.py`](https://github.com/ketch/bseries) to derive modifying integrators for certain Runge-Kutta methods applied to first-order ODEs, and study how well the solution of the truncated equations approximates the exact solution.

```python
import numpy as np
from BSeries import trees, bs
import matplotlib.pyplot as plt
from nodepy import rk, ivp
from IPython.display import display, Math
import sympy
from sympy import symbols, simplify, lambdify, dsolve, Eq, Function
from sympy import Derivative as D
from sympy.abc import t
cf = trees.canonical_forest
one = sympy.Rational(1)
from sympy import sin
from scipy.integrate import solve_ivp
h = sympy.Symbol('h')
```

# Rigid body Euler equations

Here we reproduce the first example from CHV2007.  First we set up the system of 3 ODEs corresponding to the rigid body problem (CHV2007 eqn. (9)):

```python
from sympy.abc import alpha, beta, gamma
from sympy.abc import x, y, z
y = [symbols('y%d' % i) for i in range(1,4)]

u = [y[0],y[1],y[2]]
u
```

```python
f = np.array([alpha*y[1]*y[2],beta*y[2]*y[0],gamma*y[0]*y[1]])
for rhs in f:
    display(rhs)
```

We will derive a modifying integrator ODE system for the implicit midpoint method.

```python
# Implicit Midpoint Method
A = np.array([[one/2]])
b = np.array([one])
```

```python
series = bs.modifying_integrator(u, f, A, b, order=5)
simplify(series)
```

We can verify that the results match Eqn. (12) of CHV2007:

```python
s3 = sympy.Poly((series[0]-f[0])/(f[0]),h).coeffs()[1]
s3
```

```python
s5 = sympy.Poly((series[0]-f[0])/(f[0]),h).coeffs()[0]
simplify(s5 - 6*one/5*s3**2)
```

# Lotka-Volterra

Next we consider using the explicit Euler method to solve the Lotka-Volterra model:

$$
    p'(t) = (2-q)p \quad \quad q'(t)=(p-1)q.
$$

```python
p, q = symbols('p,q')
u = [p,q]
f = np.array([p*(2-q),q*(p-1)])

FE1 = rk.loadRKM('FE')

dt = 0.35
T = 15.
IC = [1.5,2.25]

f_ = lambdify([p,q],f)

def f_vec(t,u):
    return f_(*u)

myivp = ivp.IVP(f=f_vec,u0=np.array(IC),T=T)
t0, y0 = FE1(myivp,dt=dt)
y0 = np.array(y0)

f_ex = lambdify([p,q],f)

def f_vec(t,u):
    return f_ex(*u)

myivp = ivp.IVP(f=f_vec,u0=np.array(IC),T=T)
BS5 = rk.loadRKM('BS5')

t, y = BS5(myivp,errtol=1.e-10,dt=1.e-3)
y_exact = np.array(y)
```

The exact solution of this problem is periodic, but Euler's method produces an unstable trajectory.  Here we use an especially large timestep in order to more clearly illustrate what will follow, but the qualitative behavior is the same for any step size.

```python
plt.figure(figsize=(9,6))
plt.plot(y_exact[:,1],y_exact[:,0],'-k',lw=2)
plt.plot(y0[:,1],y0[:,0],'--b')
plt.xlim(0,4)
plt.ylim(0,2.5)
plt.legend(['Exact solution','Explicit Euler, dt=0.1'],fontsize=15);
```

Now we will derive a "modifying integrator".  What this means is that we will determine a perturbed RHS such that when Euler's method is applied to the perturbed RHS, the result is the exact solution to the original Lotka-Volterra system.  The perturbed system takes the form of a power series in $h$, and in order to compute with it we will truncate it at a certain order.  We can compare the accuracy (and qualitative behavior) obtained by truncating at different orders.

```python
A = FE1.A
b = FE1.b
max_order = 4
series = bs.modifying_integrator(u, f, A, b, order=max_order)
simplify(series)
```

```python
ymod = []
for order in range(2,max_order+1):
    fs = simplify(np.array([term.series(h,0,order).removeO() for term in series]))
    f_ = lambdify([p,q,h],fs)

    def f_p_vec(t,u,h=dt):
        return f_(*u,h)

    myivp = ivp.IVP(f=f_p_vec,u0=np.array(IC),T=T)
    _, y1 = FE1(myivp,dt=dt)
    ymod.append(np.array(y1))
```

```python
colors = 'rgc'
plt.figure(figsize=(9,6))
plt.plot(y_exact[:,1],y_exact[:,0],'-k',lw=2)
plt.plot(y0[:,1],y0[:,0],'--b')
for j in range(max_order-1):
    plt.plot(ymod[j][:,1],ymod[j][:,0],'--'+colors[j],lw=3,alpha=0.5)
plt.xlim(0,4)
plt.ylim(0,2.5)
plt.legend(['Exact solution','Explicit Euler, dt='+str(dt),
            'EE with modified flow to O(h)',
           'EE with modified flow to $O(h^2)$',
           'EE with modified flow to $O(h^3)$'],fontsize=15);
```

We see that if we include one additional term, the resulting trajectory still grows, while with two additional terms the solution appears to be dissipative.  With each additional term the solution gets closer to the exact solution of the original problem, and with three added terms it is hard to see the difference between them at this scale.


```
from funciones_forma_lagrange import lagrangiana_1D, lagrangiana_2D, lagrangiana_3D
from funciones_forma_lagrange import recorrer_matriz_espiral, guardar_matriz_espiral
from funciones_forma_lagrange import recorrer_matriz_3d_espiral, guardar_matriz_3d_espiral
from funciones_forma_lagrange import hermitiana_1D
from sympy import Symbol, lambdify, latex
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import axes3d
from mayavi import mlab

x = Symbol('x')
y = Symbol('y')
z = Symbol('z')
```

```
A1 = hermitiana_1D(x, 2)

for i in range(len(A1)):
    print A1[i]

X1 = np.arange(-1, 1, 0.01)

for i in range(len(A1)):
    Y1 = lambdify(x, A1[i])
    plt.plot(X1, Y1(X1))
    plt.title('Funcion de forma de ' + str(len(A1) / 2) + ' nodos')
plt.show()
```

Las ecuaciones obtenidas son:

\begin{eqnarray}
0.125 x^{3} + 0.125 x^{2} - 0.625 x + 0.375 \\\
0.25 x^{3} - 0.25 x^{2} - 0.25 x + 0.25 \\\
- 0.125 x^{3} + 0.125 x^{2} + 0.625 x + 0.375 \\\
0.25 x^{3} + 0.25 x^{2} - 0.25
\end{eqnarray}

```
A2 = hermitiana_1D(x, 3)

for i in range(len(A2)):
    print latex(A2[i])

X2 = np.arange(-1, 1, 0.01)

for i in range(len(A2)):
    Y2 = lambdify(x, A2[i])
    plt.plot(X2, Y2(X2))
    plt.title('Funcion de forma de ' + str(len(A2) / 2) + ' nodos')
plt.show()
```

Las ecuaciones obtenidas son:

\begin{eqnarray}
0.375 x^{5} - 0.125 x^{4} - 0.875 x^{3} + 0.625 x^{2} \\\
0.25 x^{5} - 0.25 x^{4} - 0.25 x^{3} + 0.25 x^{2} \\\
1.0 x^{4} - 2.0 x^{2} + 1.0 \\\
1.0 x^{5} - 2.0 x^{3} + 1.0 x \\\
- 0.375 x^{5} - 0.125 x^{4} + 0.875 x^{3} + 0.625 x^{2} \\\
0.25 x^{5} + 0.25 x^{4} - 0.25 x^{3} - 0.25 x^{2}
\end{eqnarray}


# Check CFHT Zero Point Value

## Theoretical Background

The Zero Point value (in ADU/sec) for the `r` filter in the [CFHT specifications page](https://www.cfht.hawaii.edu/Instruments/Imaging/Megacam/generalinformation.html) is 26.22 and it seems to be over estimated. In the [LSST GitHub code](https://github.com/LSSTDESC/WeakLensingDeblending/blob/9f851f79f6f820f815528d11acabf64083b6e111/descwl/survey.py#L288) it is recomputed and its new value is 10.72.

In the following we are going to recompute the Zero Point value using the formulas in the [LSST GitHub issue #1](https://github.com/LSSTDESC/WeakLensingDeblending/issues/1).

We have:

\begin{equation}
z_p = m_0+2.5 \log_{10}(s_0[R,m_0]) \quad,
\end{equation}

where $z_p$ is the Zero Point AB magnitude, $m_0$ is a fiducial source of AB magnitude of value 24 and $s_0[R,m_0]$ the corresponding flux, such that:

\begin{equation}
s_0[R,m_0]=s[f_{AB},R]\cdot 10^{-0.4m_0} \quad,
\end{equation}

with $s[f_{AB},R]$ is defined as:

\begin{equation}
s[f_{AB},R]=A \int_0^{+\infty}f(\lambda)\cdot\frac{\lambda}{hc}\cdot R(\lambda) d\lambda \quad ,
\end{equation}

here $A=8.022$ is telescope effective area in $m^2$, $f(\lambda)=f_{AB}(\lambda)=1.08855 \cdot 10^{-5} (J/s/m^2/nm)$, $h$ is the Planck constant, $c$ is the speed of light and $\lambda$ (in nm) and $R(\lambda)$ value are given in the R.MP9602 third generation [filter response table](http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/en/megapipe/docs/filt.html).

## Numerical Application

First, extract and convert $\lambda$ and $R(\lambda)$ values, load other constants and compute $s[f_{AB},R]$.
> In the table, $\lambda$ values are in Angstroms and should be converted to nm.

```python
import numpy as np
from scipy.constants import h,c

file_name = 'r_lambda.txt'
lamb, r = np.loadtxt(file_name, unpack=True)
lamb /= 10 #convert from Angstrom to nm
m0 = 24
f = 1.08855/(lamb**2) * 10 ** (-5) # J/s/m^2/nm^-1
A = 8.022 #m^2

# use the  trapezoidal rule to perform a numerical integration
sAB = A * np.trapz(f*lamb*r/(h*c), lamb)

print(r'The value of s[f_AB,R] is {}'.format(np.format_float_scientific(sAB)))
```

    The value of s[f_AB,R] is 5.155817453958427e+19

Deduce the values of $s_0[R,m_0]$ then $z_p$.

```python
s0 = sAB * 10 ** (-0.4*m0)
zp = m0 + 2.5*np.log10(s0)
print(r'The value of s_0[R, m_0] is {}'.format(np.format_float_scientific(s0)))
print(r'The magnitude of the zero point is {}'.format(np.format_float_scientific(zp)))
```

    The value of s_0[R, m_0] is 1.2950827905938398e+10
    The magnitude of the zero point is 4.928074383098289e+01