# Computational Astrophysics
## Interpolation 01

---
## Eduard Larrañaga

Observatorio Astronómico Nacional\
Facultad de Ciencias\
Universidad Nacional de Colombia

---

### About this notebook

In this notebook we present some of the interpolation techniques.

---

## Interpolation

Experimental astrophysical data usually consist of a discrete set of data points $(x_j, f_j)$ which represent the value of a function $f(x)$ for a finite set of arguments $\{ x_1, x_2, ..., x_n \}$. However, it is usually needed to know the value of the function at aditional points and **interpolation** is the method used to obtain those values.

**Interpolation** corresponds to define a function $g(x)$, using the known discrete information and such that $g(x_j) = f(x_j)$, to approximate the value of $f$ at any point $x \in [x_{min}, x_{max}]$, where $x_{min} = \min [x_j]$ and $x_{max} = \max \{ x_j \}$.

**Extrapolation** will correspond to approximate the value of $f$ at a point $x \notin [x_{min}, x_{max}]$}.

---
## Simple Polynomial Interpolation

The simplest method of interpolation is called **Polynomial Interpolation** and consist in finding a polynomial $p(x)$ of degree $n$ that passes through $n+1$ points $x_j$ with values $p(x_j) = f(x_j)$, where $j=0,1,2,...,n$.

The polynomial is written

$p_n(x) = a_0 + a_1 x + a_2 x^2 + \cdots + a_n x^n$

where $a_i$ are $n+1$-real constants to be determined by the conditions

$\left(
\begin{array}{ccccc}
1&x_0^1&x_0^2&\cdots&x_0^n\\
\vdots&\vdots&\vdots&\vdots&\vdots\\
\vdots&\vdots&\vdots&\vdots&\vdots\\
1&x_n^1&x_n^2&\cdots&x_n^n\\
\end{array}
\right)
\left(\begin{array}{c}
a_0\\
\vdots\\
\vdots\\
a_n
\end{array}\right)
=
\left(\begin{array}{c}
f(x_0)\\
\vdots\\
\vdots\\
f(x_n)
\end{array}\right)$

Solving this system is straightforward for simple cases such as linear ($n=1$) and quadratic ($n=2$) interpolation,which we show below, but can be complicated for large $n$.

---
### Linear Interpolation

The linear interpolation ($n=1$) of a function $f(x)$ in an interval
$[x_i,x_{i+1}]$ requires information from just two points (the lower and upper limits of the interval).

Solving the linear system or equivalently, using the forward difference approximation defined for numerical derivation, we obtain the linear polynomial

$p_1(x) = f(x_i) + \frac{f(x_{i+1}) - f(x_i)}{h} (x-x_i) + \mathcal{O}(h^2)$,

where $h=x_{i+1} - x_i$.

The linear interpolation method provides a polynomial with second order accuracy and that can be differentiated once, but this derivative is not continuous at the endpoints $x_i$ and $x_{i+1}$.

#### Example. Piecewise Linear Interpolation

We will read a data set from a .txt file and interpolate linearly between each pair of points (piecewise interpolation)

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Reading the data
data = np.loadtxt('data_points.txt', comments='#', delimiter=',')
x = data[:,0]
f = data[:,1]

plt.figure()
plt.scatter(x,f)
plt.xlabel(r'$x$')
plt.ylabel(r'$f(x)$')
plt.show()
```

```python
def linearInterpolation(x1, x2, f1, f2, x):
    p1 = f1 + ((f2-f1)/(x2-x1))*(x-x1)
    return p1

N = len(x)

plt.figure(figsize=(7,5))
plt.scatter(x, f, color='black')

for i in range(N-1):
    x_interval = np.linspace(x[i],x[i+1],3)
    # Note that the number 3 in thie above line indeicates the number of
    # points interpolated in each interval !
    # (including the extreme points of the interval)
    y_interval = linearInterpolation(x[i], x[i+1], f[i], f[i+1], x_interval)
    plt.plot(x_interval, y_interval,'r')

plt.title(r'Linear Piecewise Interpolation')
plt.xlabel(r'$x$')
plt.ylabel(r'$p_1(x)$')
plt.show()
```

---
### Quadratic Interpolation

The quadratic interpolation ($n=2$) requires information of three points and the resulting polynomial will be sensitive to which three points are chosen.

Choosing the points $x_i$ , $x_{i+1}$ and $x_{i+2}$ for interpolating $f(x)$ in the range $[x_{i},x_{i+1}]$ and solving the corresponding system from the linear system gives

$p_2(x) = \frac{(x-x_{i+1})(x-x_{i+2})}{(x_i - x_{i+1})(x_i - x_{i+2})} f(x_i)
+ \frac{(x-x_{i})(x-x_{i+2})}{(x_{i+1} - x_{i})(x_{i+1} - x_{i+2})} f(x_{i+1})
+ \frac{(x-x_i)(x-x_{i+1})}{(x_{i+2} - x_i)(x_{i+2} - x_{i+1})} f(x_{i+2}) + \mathcal{O}(h^3)$,

where $h = \max \{ x_{i+2}-x_{i+1},x_{i+1}-x_i \}$.

This time the interpolating polynomial $p(x)$ is twice differentiable, but although its first derivative will be continuous, the second derivative will have finite-size steps.

#### Example. Piecewise Quadratic Interpolation

We will read a data set from a .txt file and interpolate a second order polynomial between each pair of points (piecewise interpolation)

```python
import numpy as np
import matplotlib.pyplot as plt

# Reading the data
data = np.loadtxt('data_points.txt', comments='#', delimiter=',')
x = data[:,0]
f = data[:,1]

def quadraticInterpolation(x1, x2, x3, f1, f2, f3, x):
    p2 = (((x-x2)*(x-x3))/((x1-x2)*(x1-x3)))*f1 + (((x-x1)*(x-x3))/((x2-x1)*(x2-x3)))*f2 +\
         (((x-x1)*(x-x2))/((x3-x1)*(x3-x2)))*f3
    return p2

N = len(x)

plt.figure(figsize=(7,5))
plt.scatter(x, f, color='black')

for i in range(N-2):
    x_interval = np.linspace(x[i],x[i+1],6) # 6 interpolate points in each interval
    y_interval = quadraticInterpolation(x[i], x[i+1], x[i+2], f[i], f[i+1], f[i+2], x_interval)
    plt.plot(x_interval, y_interval,'r')

plt.title(r' Quadratic Polynomial Piecewise Interpolation')
plt.xlabel(r'$x$')
plt.ylabel(r'$p_2(x)$')
plt.show()
```

**Note:** Tthe form of this quadratic interpolation leaves the last interval without information (because we need three points to apply the numerical method).

---
## Lagrange Interpolation

**Lagrange Interpolation** also finds an interpolating polynomial of degree $n$ using data at $n+1$
points, but uses an alternative method for finding the coefficients. First, we re-write the interpolating linear polynomial as

\begin{equation}
p_1(x) = \frac{x-x_{i+1}}{x_i - x_{i+1}} f(x_i) + \frac{x-x_i}{x_{i+1}-x_i} f(x_{i+1}) + \mathcal{O}(h^2),
\end{equation}
or as
\begin{equation}
p_1(x) = \sum_{j=i}^{i+1} f(x_j) L_{1j}(x) + \mathcal{O}(h^2)
\end{equation}

where we introduce the Lagrange coefficients
\begin{equation}
L_{1j}(x) = \frac{x-x_k}{x_j-x_k}\bigg|_{k\ne j}
\end{equation}

Note that these coefficients ensure that the polynomial passes through the two data points, i.e. $p_1(x_i) = f(x_i)$ and $p_1(x_{i+1}) = f(x_{i+1})$

**Lagrange interpolation** generalizes these expressions to give a polynomial of degree $n$ that
passes through all the $n+1$ data points. It is defined by
\begin{equation}
p_n (x) = \sum_{j=0}^{n} f(x_j) L_{nj}(x) + \mathcal{O}(h^{n+1})\,, \label{eq:LagrangeInterpolation}
\end{equation}
where the coefficients are generalized to
\begin{equation}
L_{nj}(x) = \prod_{k\ne j}^{n} \frac{x-x_k}{x_j - x_k}\,.
\end{equation}

Again, it is important to note that these coefficients ensure that $p(x_j) = f(x_j)$ for the $n+1$ data points.

```python
%load lagrangeInterpolation
```

```python
import numpy as np
import matplotlib.pyplot as plt
import lagrangeInterpolation as lagi
import sys

# Reading the data
data = np.loadtxt('data_points.txt', comments='#', delimiter=',')
x = data[:,0]
f = data[:,1]

N = len(x)

# Degree of the polynomial to be interpolated piecewise
n = 3

# Check if the number of point is enough to interpolate such a polynomial
if n>=N:
    print('\nThere are not enough points to interpolate this polynomial.')
    print(f'Using {N:.0f} points it is possible to interpolate polynomials up to order n={N-1:.0f}')
    sys.exit()

plt.figure(figsize=(7,5))
plt.title(f'Lagrange Polynomial Piecewise Interpolation n={n:.0f}')
plt.scatter(x, f, color='black')

# Piecewise Interpolation Loop
for i in range(N-n):
    xi = x[i:i+n+1]
    fi = f[i:i+n+1]
    x_interval = np.linspace(x[i],x[i+1],3*n)
    y_interval = lagi.p(x_interval,xi,fi)
    plt.plot(x_interval, y_interval,'r')

plt.xlabel(r'$x$')
plt.ylabel(r'$p_n(x)$')
plt.show()
```

Note that the las $N-n$ points can be interpolated. What can we do?

```python
import numpy as np
import matplotlib.pyplot as plt
import lagrangeInterpolation as lagi
import sys

# Reading the data
data = np.loadtxt('data_points.txt', comments='#', delimiter=',')
x = data[:,0]
f = data[:,1]

N = len(x)

# Degree of the polynomial to be interpolated piecewise
n = 3

# Check if the number of point is enough to interpolate such a polynomial
if n>=N:
    print('\nThere are not enough points to interpolate this polynomial.')
    print(f'Using {N:.0f} points it is possible to interpolate polynomials up to order n={N-1:.0f}')
    sys.exit()

plt.figure(figsize=(7,5))
plt.title(f'Lagrange Polynomial Piecewise Interpolation n={n:.0f}')
plt.scatter(x, f, color='black')

# Piecewise Interpolation Loop
for i in range(N-n):
    xi = x[i:i+n+1]
    fi = f[i:i+n+1]
    x_interval = np.linspace(x[i],x[i+1],3*n)
    y_interval = lagi.p(x_interval,xi,fi)
    plt.plot(x_interval, y_interval,'r')

# Piecewise Interpolation for the final N-n points,
# using a lower degree polynomial
while n>1:
    m = n-1
    for i in range(N-n,N-m):
        xi = x[i:i+m+1]
        fi = f[i:i+m+1]
        x_interval = np.linspace(x[i],x[i+1],3*m)
        y_interval = lagi.p(x_interval,xi,fi)
        plt.plot(x_interval, y_interval,'r')
    n=n-1

plt.xlabel(r'$x$')
plt.ylabel(r'$p_n(x)$')
plt.show()
```

### Runge's Phenomenon

Why to interpolate piecewise? It is ususal to hink that interpolating a high order polynomial may be better than lower order polynomials. However, for oscillating function this is ususally not a good idea due to Runge's phenomenon.

For example, for a dataset with $N$-points we can interpolate a $19$-degree polynomial:

```python
import numpy as np
import matplotlib.pyplot as plt
import lagrangeInterpolation as lagi
import sys

# Reading the data
data = np.loadtxt('data_points.txt', comments='#', delimiter=',')
x = data[:,0]
f = data[:,1]

N = len(x)

# Higher Degree polynomial to be interpolated
n = N-1

plt.figure(figsize=(7,5))
plt.title(f'Lagrange Polynomial Piecewise Interpolation n={n:.0f}')
plt.scatter(x, f, color='black')

#Interpolation of the higher degree polynomial
x_int = np.linspace(x[0],x[N-1],3*n)
y_int = lagi.p(x_int,x,f)
plt.plot(x_int, y_int,'r')

plt.xlabel(r'$x$')
plt.ylabel(r'$p_n(x)$')
plt.show()
```

It is clear that the high order polynomial interpolated (not piecewise), has a bad behavior, specially at the borders of the dataset interval. The bad behavior is worst for higly oscillating functions!

---
## Piecewise Cubic Hermite Interpolation

Hermite interpolation is a special form of polynomial interpolation which uses data points and the derivatives of the data to obtain the interpolating polynomial. Incorporation of the first derivative reduces the unwanted oscillations. The inclusion of the derivative also permits to interpolate a high-order polynomial with less data points than Lagrange interpolation.

Piecewise third-order Hermite interpolation is one of the most used cases. In this method, for each domain interval $[x_i , x_{i+1}]$, in which we know (or evaluate) $f(x_i)$, $f(x_{i+1})$, $f'(x_i)$ and $f'(x_{i+1})$, one interpolates a cubic Hermite polynomial given by

\begin{equation}
H_3(x) = f(x_i)\psi_0(z) + f(x_{i+1})\psi_0(1-z)+ f'(x_i)(x_{i+1} - x_{i})\psi_1(z) - f'(x_{i+1})(x_{i+1}-x_i)\psi_1 (1-z),
\end{equation}

where

\begin{equation}
z = \frac{x-x_i}{x_{i+1}-x_i}
\end{equation}
and
\begin{align}
\psi_0(z) =&2z^3 - 3z^2 + 1 \\
\psi_1(z) =&z^3-2z^2+z\,\,.
\end{align}

Note that it is possible to interpolate a third-order polynomial in an interval with only two points! This fact makes possible to interpolate the third-order polynomial between all intervals, even the last one.

```python
%load HermiteInterpolation
```

```python
import numpy as np
import matplotlib.pyplot as plt
import HermiteInterpolation as heri

def Derivative(x, f):
    '''
    ------------------------------------------
    Derivative(x, f)
    ------------------------------------------
    This function returns the numerical
    derivative of a discretely-sample function
    using one-side derivatives in the extreme
    points of the interval and second order
    accurate derivative in the middle points.
    The data points may be evenly or unevenly
    spaced.
    ------------------------------------------
    '''
    # Number of points
    N = len(x)
    dfdx = np.zeros([N, 2])
    dfdx[:,0] = x

    # Derivative at the extreme points
    dfdx[0,1] = (f[1] - f[0])/(x[1] - x[0])
    dfdx[N-1,1] = (f[N-1] - f[N-2])/(x[N-1] - x[N-2])

    #Derivative at the middle points
    for i in range(1,N-1):
        h1 = x[i] - x[i-1]
        h2 = x[i+1] - x[i]
        dfdx[i,1] = h1*f[i+1]/(h2*(h1+h2)) - (h1-h2)*f[i]/(h1*h2) -\
                    h2*f[i-1]/(h1*(h1+h2))

    return dfdx

# Loading the data
data = np.loadtxt('data_points.txt', comments='#', delimiter=',')
x = data[:,0]
f = data[:,1]
N = len(x)

# Calling the derivative function and chosing only the second column
dfdx = Derivative(x,f)[:,1]

plt.figure(figsize=(7,5))
plt.title(f'Cubic Hermite Polynomial Piecewise Interpolation')
plt.scatter(x, f, color='black')

# Piecewise Hermite Interpolation Loop
for i in range(N-1):
    xi = x[i:i+2]
    fi = f[i:i+2]
    dfidx = dfdx[i:i+2]
    x_interval = np.linspace(x[i],x[i+1],4)
    y_interval = heri.H3(x_interval, xi, fi, dfidx)
    plt.plot(x_interval, y_interval,'r')

plt.xlabel(r'$x$')
plt.ylabel(r'$H_3(x)$')
plt.show()

```


# Discrete Signals

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Comunications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Standard Signals

A discrete signal $x[k] \in \mathbb{C}$ is a sequence of values that depends on the discrete index $k \in \mathbb{Z}$. It may have been derived by [sampling](../sampling/ideal.ipynb) $x[k] := x(k T)$ from a continuous signal $x(t)$. As for the case of continuous signals, certain [standard signals](../continuous_signals/standard_signals.ipynb) play an important role in the theory and practical application of discrete signals and systems. These standard signals are introduced and illustrated in the following. Some of the signals are derived by sampling their continuous counterparts, some of them by postulating equivalent properties. Discrete signals are commonly denoted by using square brackets for their arguments.

### Complex Exponential Signal

The discrete complex exponential signal is defined by the [complex exponential function](https://en.wikipedia.org/wiki/Exponential_function#Complex_plane)

\begin{equation}
x[k] = e^{(\Sigma + j \Omega) \, k} = z^k
\end{equation}

where $z = e^{\Sigma + j \Omega}$ denotes the complex frequency $z \in \mathbb{C}$ with $\Sigma, \Omega \in \mathbb{R}$. The discrete complex exponential signal can be related to the [continuous complex exponential signal](../continuous_signals/standard_signals.ipynb#Complex-Exponential-Signal) $x(t) = e^{(\sigma + j \omega) t}$ by sampling

\begin{equation}
x[k] = x(k T) = e^{(\sigma + j \omega) k T} = e^{( \sigma T + j \omega T ) k}
\end{equation}

where $T$ denotes the sampling interval. Comparison to above definition of the discrete signal reveals that $\Sigma = \sigma T$ and $\Omega = \omega T$. Due to this relation, the latter is termed as *normalized frequency* $\Omega$. Using [Euler's formula](https://en.wikipedia.org/wiki/Euler's_formula), the definition of the complex exponential signal can be reformulated as

\begin{equation}
x[k] = e^{\Sigma k} \cos[\Omega k] + j e^{\Sigma k} \sin[\Omega k]
\end{equation}

The real/imaginary part of the exponential signal is given by a weighted discrete cosine/sine with normalized frequency $\Omega$. The normalized frequency $\Omega$ is ambiguous due to the periodicity of the cosine/sine function for discrete $k$. For instance

\begin{equation}
\cos[\Omega k] = \cos[(\Omega + n \cdot 2 \pi) \cdot k]
\end{equation}

with $n \in \mathbb{Z}$. It can be concluded that the normalized frequency $\Omega$ is unique for $-\pi < \Omega < \pi$. This also becomes evident when considering the sampling of a continuous exponential signal, as shown above. For [critical sampling](../sampling/ideal.ipynb#Sampling-Theorem-for-Low-Pass-Signals) the sampling frequency $\omega_\text{s} = 2 \cdot \omega$. From $T = \frac{2 \pi}{\omega_\text{s}}$ follows $\Omega = \omega T = \pi$. It can be concluded that the normalized frequency $\Omega = \pm \pi$ represents the highest/lowest normalized frequency a sampled signal can represent.

The complex exponential function is only periodic with respect to the discrete index $k$, if

\begin{equation}
x[k] = x[k + n \cdot N_\text{p}]
\end{equation}

holds for $n \in \mathbb{Z}$ and $N_\text{p} \in \mathbb{N}$. The periodicity of the complex exponential function is given as

\begin{equation}
N_\text{p} = \frac{2 \pi}{\Omega}
\end{equation}

It follows from the requirement $N_\text{p} \in \mathbb{N}$ for a periodic sequence, that not all normalized frequencies $\Omega$ result in a periodic discrete complex exponential signal. Only $\Omega = \frac{2 \pi}{N_\text{p}}$ with $N_\text{p} \in \mathbb{N}$ is periodic with period $N_\text{p}$. Sampling of a continuous complex exponential signal may result in an aperiodic discrete complex exponential signal.

The complex exponential signal can be expressed in terms of its magnitude and phase

\begin{equation}
x[k] = z^k = |z|^k \cdot e^{j \varphi(z) k}
\end{equation}

where by comparison with its definition $|z| = |e^\Sigma|$ and $\varphi(z) = \Omega$. This finding allows an interpretation of the complex frequency $z$. Its phase $\varphi(z)$ is equal to the normalized frequency $\Omega$ of its harmonic part $e^{j \Omega k}$. This is weighted by the magnitude $|z|^k$. With increasing index $k >0$, the magnitude of the complex exponential signal is

* exponentially decaying for $|z| < 1$ ($\Sigma < 0$)
* constantly one for $|z| = 1$ ($\Sigma = 0$)
* exponentially growing for $|z| > 1$ ($\Sigma > 0$).

**Example**

This example illustrates the discrete complex exponential signal and its parameters. The Python module [NumPy](http://www.numpy.org/) provides functionality for numerical mathematics. In discrete signal processing it is common  to define discrete signals as vectors holding the values $x[k]$ for a given set of indexes $k$. Two functions are defined as prerequisite. One for the generation of the exponential signal and one to plot its real and imaginary part.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

def exponential_signal(k, Sigma, Omega):
    return np.exp((Sigma + 1j * Omega) * k)

def plot_signal(k, x):
    plt.figure(figsize=(10, 4))
    plt.subplot(121)
    plt.stem(k, np.real(x))
    plt.xlabel('$k$')
    plt.ylabel(r'$\Re \{ x[k] \}$')

    plt.subplot(122)
    plt.stem(k, np.imag(x))
    plt.xlabel('$k$')
    plt.ylabel(r'$\Im \{ x[k] \}$')
    plt.tight_layout()
```

The vector `x` is populated by the samples of the exponential signal for a given set of indexes $k$, $\Sigma = 0.025$ and $\Omega = 0.5$. Its first ten values are printed for illustration.

```python
k = np.arange(31)
x = exponential_signal(k, 0.025, 0.5)
x[:10]
```

    array([ 1.        +0.j        ,  0.89979867+0.49156225j,
            0.5680042 +0.88461412j,  0.07624651+1.07518404j,
           -0.45991338+1.00492907j, -0.90781465+0.67815778j,
           -1.15020718+0.16395806j, -1.11555049-0.41786919j,
           -0.79836212-0.92436066j, -0.26398437-1.22418317j])

Discrete signals are commonly visualized by '*stem*'-plots. The Python module [`matplotlib`](http://matplotlib.org/) is used for this purpose. The real and imaginary part of the complex exponential signal is shown by the left and right plot, respectively.

```python
plot_signal(k, x)
```

Now the case of an aperiodic harmonic exponential signal with $N_\text{p} \notin \mathbb{N}$ is illustrated for $\Sigma = 0$. Again the left and right plot show the real and imaginary part.

```python
Np = 20.55
x = exponential_signal(k, 0, 2 * np.pi / Np)
plot_signal(k, x)
```

**Exercise**

* Change the values of $\Sigma$ in the first example in order to create a signal with increasing/constant/decaying amplitude.
* Check if the stated non-uniqueness of the normalized frequency $\Omega$ holds by adding multiples of $2 \pi$ and plotting the resulting signal.

### Dirac Impulse

The discrete Dirac impulse $\delta[k]$ cannot be derived by sampling from its [continuous counterpart](../continuous_signals/standard_signals.ipynb#Dirac-Impulse), since the [Dirac delta function](https://en.wikipedia.org/wiki/Dirac_delta_function) is a distribution and not a function in the conventional sense. The discrete Dirac impulse is defined as

\begin{equation}
\delta[k] = \begin{cases}
1 & \text{for } k = 0 \\
0 & \text{otherwise}
\end{cases}
\end{equation}

This function is also known as [*Kronecker delta*](https://en.wikipedia.org/wiki/Kronecker_delta), the signal as *impulse sequence* or *unit impulse*. The discrete Dirac impulse $\delta[k]$ maintains the essential properties of its continuous counterpart $\delta(t)$. In particular

1. **Sifting property**
    \begin{equation}
    \sum_{\kappa = -\infty}^{\infty} x[\kappa] \cdot \delta[k - \kappa] = x[k]
    \end{equation}
    The sifting property implies $\sum_{\kappa = -\infty}^{\infty} x[\kappa] = 1$.

2. **Multiplication**
    \begin{equation}
    x[k] \cdot \delta[k - \kappa] = x[\kappa] \cdot \delta[k - \kappa]
    \end{equation}

3. **Linearity**
    \begin{equation}
    a \cdot \delta[k] + b \cdot \delta[k] = (a+b) \cdot \delta[k]
    \end{equation}

The discrete Dirac impulse is an important signal in the theory of signals and systems. It is used for instance for the characterization of discrete LTI systems by their impulse response.

**Example**

The properties of the discrete Dirac impulse are illustrated. First a function is defined for the discrete Dirac impulse and $\delta[k]$ is plotted

```python
def dirac(k):
    return np.where(k == 0, 1.0, 0.0)

k = np.arange(-10, 11)
x = dirac(k)
plt.stem(k, x)
plt.xlabel('$k$')
plt.ylabel('$\delta[k]$')
plt.ylim([-0.1, 1.1])
```

    (-0.1, 1.1)

Now let's check the multiplication property by defining a cosine signal $y[k] = \cos[k]$ and computing the signal $w[k] = y[k] \cdot \delta[k-6]$

```python
x = dirac(k - 6)
y = np.cos(k)
w = x*y

plt.figure(figsize=(10, 4))
plt.subplot(121)
plt.stem(k, y)
plt.xlabel('$k$')
plt.ylabel(r'$\cos[\Omega k]$')
plt.ylim([-1.1, 1.1])

plt.subplot(122)
plt.stem(k, w)
plt.xlabel('$k$')
plt.ylabel('$\cos[\Omega k] \cdot \delta[k - \kappa]$')
plt.ylim([-1.1, 1.1])
plt.tight_layout()
```

### Heaviside Signal

The discrete Heaviside signal is defined similar to its [continuous Heaviside signal](../continuous_signals/standard_signals.ipynb#Heaviside-Signal) as

\begin{equation}
\epsilon[k] = \begin{cases} 1 & k \geq 0 \\  0 & k < 0 \end{cases}
\end{equation}

The Heaviside signal is used to represent a signal that switches on at a specified time and stays on indefinitely. The Heaviside signal can be related to the Dirac impulse by

\begin{equation}
\epsilon[k] = \sum_{\kappa = -\infty}^{k} \delta[\kappa]
\end{equation}

The Dirac impulse can be expressed in terms of the Heaviside signal

\begin{equation}
\delta[k] = \epsilon[k] - \epsilon[k-1]
\end{equation}

**Example**

In the following, a function is defined for the Heaviside signal $\epsilon[k]$ and the signal is plotted for illustration.

```python
def heaviside(k):
    return np.where(k >= 0, 1.0, 0.0)

k = np.arange(-5, 20)
x = heaviside(k)

plt.stem(k, x)
plt.xlabel('$k$')
plt.ylabel('$\epsilon[k]$')
plt.ylim([-0.1, 1.1])
```

    (-0.1, 1.1)

### Rectangular Signal

The discrete rectangular signal is defined as

\begin{equation}
\text{rect}_N[k] = \begin{cases} 1 & \text{for } 0 \leq k < N \\ 0 & \text{otherwise}  \end{cases}
\end{equation}

where $N \in \mathbb{N}$ denotes the number of its non-zero samples. Note that the discrete rectangular signal is not even symmetric as the [continuous rectangular signal](../continuous_signals/standard_signals.ipynb#Rectangular-Signal). The rectangular signal is used to represent signals which are non-zero for a limited period of time. The rectangular signal can be related to the Heaviside signal by

\begin{equation}
\text{rect}[k] = \epsilon[k] - \epsilon[k - N]
\end{equation}

**Example**

A function is defined for the rectangular signal and $\text{rect}_5[k]$ is plotted for illustration.

```python
def rect(k, N):
    return np.where((0 <= k) & (k < N), 1.0, 0.0)

k = np.arange(-5, 20)
x = rect(k, 10)

plt.stem(k, x)
plt.xlabel('$k$')
plt.ylabel('$\mathrm{rect}[k]$')
plt.ylim([-0.1, 1.1])
```

    (-0.1, 1.1)

**Exercise**

* Use $\text{rect}_N[t]$ to construct a cosine signal $x[k] = \cos[\Omega k] \cdot \text{rect}_N[t]$ of one period length with $\Omega= \frac{2 \pi}{10}$.

### Sign Signal

The discrete sign signal is defined analogously to the [continuous sign signal](../continuous_signals/standard_signals.ipynb#Sign-Signal) as

\begin{equation}
\text{sgn}[k] = \begin{cases} 1 & k>0 \\ 0 & k=0 \\ -1 & k < 0  \end{cases}
\end{equation}

The sign signal is used to represent the absolute value of a signal $x[k]$ as

\begin{equation}
|x[k]| = x[k] \cdot \text{sgn}(x[k])
\end{equation}

It is related to the Heaviside signal by

\begin{equation}
\text{sgn}[k] = \epsilon[k] - \epsilon[-k]
\end{equation}

**Example**

The sign signal is realized by using the [`numpy.sign`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.sign.html#numpy.sign) function. It is plotted for illustration.

```python
k = np.arange(-10, 11)
x = np.sign(k)

plt.stem(k, x)
plt.xlabel('$k$')
plt.ylabel('$\mathrm{sgn}[k]$')
plt.ylim([-1.1, 1.1])
```

    (-1.1, 1.1)

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


# Hamiltonian flows and symplectic maps

(sec:Ham2)=
## Hamiltonian flows

A Hamiltonian system with $N$ degrees of freedom is described by the Hamiltonian function $H(\mathbf{q},\mathbf{p},t)$, which depends on the generalized coordinates $\mathbf{q} = (q_1,\ldots,q_N)$, their canonically conjugate momenta $\mathbf{p} = (p_1,\ldots,p_N)$, and time. This scalar function gives rise to a $2N$-dimensional dynamical system by means of Hamilton's equations of motion

```{math}
---
label: eq:hamiltons_eom
---
\dot{q}_i = \frac{\partial H}{\partial p_i} \quad,\quad \dot{p}_i = -\frac{\partial H}{\partial q_i} \;, \quad \text{for} \; i = 1, 2, \ldots, N.
```

where the dot symbol represents the total time derivative, that is,  $\cdot \equiv d/dt$. A state of the system is represented by a point in the $2N$-dimensional _phase space_, Any solution of this system is a trajectory of the form

```{math}
---
label: eq:solution
---
\mathbf{x}(t) = \left(q_1(t), \ldots, q_N(t), p_1(t), \ldots, p_N(t)\right) \,,
```

starting at the initial condition

```{math}
---
label: eq:init_condition
---
\mathbf{x}(0) = \mathbf{x}_0 = \left(q_1^0, \ldots, q_N^0, p_1^0, \ldots, p_N^0\right) \, .
```

We will assume that $H$ is sufficiently smooth to guarantee the uniqueness of solutions. If the Hamiltonian is time independent, that is $H(\mathbf{q},\mathbf{p})$, then the resulting dynamical system is autonomous (ref to part 1) and thus the solutions of Eq. {eq}`eq:hamiltons_eom` form a one-parameter family of diffeomorphisms, known as a _Hamiltonian flow_, given by $\{\phi_t\}_{t\in\mathbb{R}}$ so that

```{math}
---
label: eq:flow
---
\phi_s(\mathbf{x}(t))=\mathbf{x}(s+t) \;.
```

The family is a group because it satisfies the properties

* $\phi_0=Id$,
* $\phi_s\circ \phi_t=\phi_{t+s}$,
* $\phi_{-t} \circ \phi_t=\phi_0$.

## Symplectic maps

A linear transformation $C$ is symplectic if it satisfies

```{math}
---
label: eq:symp_cond
---
\begin{equation}
C \mathcal{J} C^T = \mathcal{J} =
    \begin{pmatrix}
    0_N & I_N \\
    -I_N & 0_N
    \end{pmatrix}
\end{equation}
```

where $\mathcal{J}$ is a 2N $\times$ 2N matrix known as the symplectic matrix, and $I_N$ denotes the $N \times N$ identity matrix. The symplectic transformation $C$ maps the original coordinates $(x_1,\ldots, x_N, p_{x_1}, \ldots, p_{x_N})$ to the coordinates $(q_1, \ldots, q_N, p_1, \ldots, p_N)$ and Hamilton's equations in Eq, {eq}`eq:hamiltons_eom` as follows:

```{math}
\begin{equation}
    \begin{bmatrix}
    q_1 \\
    \vdots \\
    q_N \\
    p_1 \\
    \vdots \\
    q_N
    \end{bmatrix}
    = C
    \begin{bmatrix}
    x_1 \\
    \vdots \\
    x_N \\
    p_{x_1} \\
    \vdots \\
    p_{x_N}
    \end{bmatrix}
    \implies
    \begin{bmatrix}
    \dot{q}_1 \\
    \vdots \\
    \dot{q}_N \\
    \dot{p}_1 \\
    \vdots \\
    \dot{p}_N
    \end{bmatrix}
    = C
    \begin{bmatrix}
    \dot{x}_1 \\
    \vdots \\
    \dot{x}_N \\
    \dot{p}_{x_1} \\
    \vdots \\
    \dot{p}_{x_N}
    \end{bmatrix}
\end{equation}
```

From {eq}`eq:symp_cond` we find the inverse transformation $C^{-1}=\mathcal{J}C^T\mathcal{J}^{-1}$, such that

```{math}
\begin{equation}
\begin{bmatrix}
\dot{x}_1 \\
\vdots \\
\dot{x}_N \\
\dot{p}_{x_1} \\
\vdots \\
\dot{p}_{x_N}
\end{bmatrix}
= C^{-1}
\begin{bmatrix}
\dot{q}_1 \\
\vdots \\
\dot{q}_N \\
\dot{p}_1 \\
\vdots \\
\dot{p}_N
\end{bmatrix}
\end{equation}
```

Linear symplectic transformations are $C^\infty$ diffeomorphisms, which means that they have continuous derivatives of all orders and they also have a continuous inverse. This group of transformations is important because they preserve the form of the equations of motion {eq}`eq:hamiltons_eom`, and also phase space structures, such as normally hyperbolic invariant manifolds and their associated stable and unstable invariant manifolds. Furthermore, symplectic transformations preserve the stability of invariant sets and Lyapunov exponents of trajectories.

In Ref {cite}`naik_finding_2019`, we used the symplectic transformation to transform the uncoupled (separable) quadratic normal form to a coupled (non-separable) quadratic normal form Hamiltonian for two and three degrees of freedom systems.

We can define nonlinear symplectic transformations using the _Poisson bracket_ (see {eq}`eq:poisson`) defined by

```{math}
---
label: eq:poisson
---
\begin{equation}
 \{A, B\} = \langle D A,\mathcal{J} D B \rangle = \sum\limits_{i=1}^{N} \left( \frac{\partial A}{\partial q_i}\frac{\partial B}{\partial p_i} - \frac{\partial A}{\partial p_i} \frac{\partial B}{\partial q_i} \right),
\end{equation}
```

where $D A$ and $D B$ are the gradients of $A$ and $B$, and $\mathcal{J}$ is defined in Eq. {eq}`eq:symp_cond`. For example, the Poisson bracket allows us to rewrite the equations of motion in Eq. {eq}`eq:hamiltons_eom` as

```{math}
---
label: eq:poisson_eqn
---
\begin{equation}
\dot{q}_i = \{q_i, H\} \quad,\quad \dot{p}_i = \{p_i, H\} \; , \quad \text{for } i = 1,\ldots,N
\end{equation}
```

Following {cite}`Wiggins2003`, a  symplectic transformations is a sufficiently smooth diffeomorphism defined on phase space $f:~\mathbb{R}^{2N}\rightarrow \mathbb{R}^{2N}$, such that

```{math}
---
label: eq:symplectic
---
\begin{equation}
 \langle u, \mathcal{J}v \rangle = \langle u, (Df(x))^T\mathcal{J}Df(x) v \rangle,
\end{equation}
```

for any $x,u,v\in \mathbb{R}^{2N}$.

Under $f$, phase space points $(q,p)$ are transformed into $f(q,p)$ and the Hamiltonian $H(q,p)$ into the new Hamiltonian $G(f(q,p))$, that satisfies $Df(q,p)DH = DG$. The equations of motion {eq}`eq:poisson_eqn` then remain

```{math}
\begin{align*}
 \{(q,p),H\} = \langle D(q,p),\mathcal{J}DH \rangle &= \langle D(q,p),\;(Df(q,p))^T\mathcal{J}Df(q,p)DH \rangle \\
 &= \langle Df(q,p)D(q,p),\;\mathcal{J}Df(q,p)DH \rangle = \{f(q,p), G\}.
\end{align*}
```

For a coordinate free formulation using differential forms see {cite}`Arnold76`.

The flow of an autonomous Hamiltonian system is a symplectic transformation for each $t$. Another common example of a symplectic map is the return map associated with a Poincarè surface of section (See Chapter {ref}`3<sec:LDs>`), to be discussed in the next section. Other well known examples of symplectic maps include the cat map, standard map, and Hénon map.
Symplectic maps arise in many applications in physics and chemistry, for example in celestial mechanics and dynamical astronomy, particle accelerators, plasma, and chemical reactions {cite}`meiss_symplectic_1992` to name but a few.

## Coordinates, ($q, p)$, $(q_i , p_i)$, significance of canonically conjugate pairs

The canonical coordinates $(q, p)$ satisfy the following Poisson bracket relations

```{math}
---
label: eq:pb_canonical
---
\begin{equation}
\{q_i, q_j\} = 0 \quad,\quad \{p_i, p_j\} = 0 \quad,\quad
\{q_i, p_j\} = \delta_{ij} \; , \quad \text{for } i,j \in \{1,\ldots,N\}
\end{equation}
```

where $\delta_{ij}$ is the Kronecker delta. The pair $(q_i , p_i)$ is referred to as canonically conjugate. The relations above are referred to as the fundamental Poisson bracket equations and are preserved if and only if the coordinates are canonical. Further details on this can be found in the section 9.4 and 9.5 of the textbook {cite}`Goldstein2001`.

An important feature of canonically conjugate pairs lies in their role in defining Poincarè surfaces of section. A surface defined by $q_j = K$ with $\dot{q}_j \geq 0$ or $\dot{q}_j \leq 0$, or $p_j = K$ with $\dot{p}_j \geq 0$ or $\dot{p}_j \leq 0$, admits a symplectic (Poincarè) return map, provided it is a surface of section in the sense of Birkhoff {cite}`Birkhoff27`. Such a surface must be transverse to the flow and all trajectories that cross it must return. Suitably-chosen Poincarè surfaces of section and the associated return maps allow up to study the dynamics in phase space on a lower dimensional surface, for example they facilitate finding periodic orbits of the system as fixed points of the return maps. Note that such surfaces may not exist in every system.

## Phase space slices, phase space transversal slice

The phase space structures that govern the dynamics of a Hamiltonian system with $N$ degrees of freedom are of codimension-$1$ and codimension-$2$ in $2N$-dimensional phase space, in the case of an autonomous system codimension-$1$ and codimension-$2$ in the $2N-1$-dimensional energy surface {cite}`Waalkens08` (refer to hyperbolic section). A Poincarè surface of section is a $(2N - 2)$-dimensional surface, for example one specified by $q_j = K$ with $\dot{q}_j \geq 0$ or $\dot{q}_j \leq 0$,
where $j \in 1, 2, \ldots, N$ and $K$ is constant. For this Poincarè surface of section to be transverse to the flow, we require

```{math}
\begin{equation}
\dot{q}_j = \dfrac{\partial H}{\partial p_j} \neq 0 \, ,
\end{equation}
```

on the entire surface of section.

We note here that the complete $(2N - 2)$-dimensional surface cannot be visualized without projecting onto a $2$ or $3$ dimensional space. Even a three dimensional projection can be difficult to visualize due to the limitation of the viewing medium such as the screen or paper. However, if we consider a two-dimensional slice of the $2N-1$ dimensional energy surface, then the intersection of the slice with a codimension-$1$ or codimension-$2$ object is either one- or zero-dimensional provided they intersect. This follows  the rule

\begin{equation*}
\text{dimension of intersection = (dimension of surface + dimension of object) - dimension of the space.}
\end{equation*}

Visualising one- or zero-dimensional intersections on a two-dimensional surface makes high-dimensional phase space structures accessible again. This visualization may not give us the whole picture of these structures in high-dimensional phase space but it is useful in order to obtain important dynamical information about these high-dimensional structures, for example to detect the intersections of the invariant manifolds and many others.

## Liouville's Theorem

> This subsection and concepts herein are adapted from Classical Mechanics by Goldstein, 3rd edition.

Suppose some property of the system is denoted by $A = A(\mathbf{x}, t)$, then the total time rate of change of this property along the solutions of an autonomous Hamiltonian system $H$ is

```{math}
\frac{d A}{d t} = \frac{\partial A}{\partial t} + \sum\limits_{i=1}^{N} \left( \frac{\partial A}{\partial q_i}\dot{q}_i + \frac{\partial A}{\partial p_i} \dot{p}_i \right) = \frac{\partial A}{\partial t} + \sum\limits_{i=1}^{N} \left( \frac{\partial A}{\partial q_i}\frac{\partial H}{\partial p_i} - \frac{\partial A}{\partial p_i} \frac{\partial H}{\partial q_i} \right)
```

Using the definition of the _Poisson bracket_:

```{math}
\begin{equation}
\{A, H\} = \sum\limits_{i=1}^{N} \left( \frac{\partial A}{\partial q_i}\frac{\partial H}{\partial p_i} - \frac{\partial A}{\partial p_i} \frac{\partial H}{\partial q_i} \right)
\end{equation}
```

we can write the time evolution of the property as

```{math}
---
label: eq:property_conserved
---
\begin{equation}
\frac{d A}{d t} = \frac{\partial A}{\partial t} + \{A, H\}
\end{equation}
```

Therefore, if we assume that the Hamiltonian does not depend explicitly on time and use $A = H$ in the above equation, we get

```{math}
---
label: eq:energy_conserved
---
\begin{equation}
\frac{d H}{d t} = 0 + \{H, H\} = 0
\end{equation}
```

which implies that the Hamiltonian, $H$, is a constant of the motion for conservative systems, that is the total energy is fixed. Thus, the states for a specified energy are restricted to the $(2N - 1)$-dimensional surface embedded in the $2N$-dimensional phase space.

We use this set-up to present the mathematical statement of the Liouville's theorem which states: _when a classical ensemble evolves its location and shape in phase space may change but its volume will be conserved_.

A classical ensemble consists of a set of points in phase space, with each point representing a system in a specified state. Let $\rho(\mathbf{x},t)$ be the density of the set of points in a given volume. As each member of ensemble moves through phase space along a trajectory specified by Hamilton's equations of motion, the phase space density evolves in time.

Using Poisson brackets {eq}`eq:property_conserved`, we can express Liouville's theorem as

```{math}
\begin{equation}
\frac{d \rho}{d t} = \frac{\partial \rho}{\partial t} + \{\rho, H\} = 0,
\end{equation}
```

which means that the phase space density $\rho(\mathbf{x}, t)$ is conserved along the solutions of $H$. Consequently when an ensemble evolves, its location and shape in phase space change but its volume is conserved. It also follows that all autonomous Hamiltonian systems are volume preserving due to Liouville's theorem.

## Poincarè recurrence

> This section is adapted from the book Applied Nonlinear Dynamical systems by S. Wiggins  (Springer, 2003) and the review article Symplectic maps, variational principles, and transport by J.D. Meiss  (Rev. Mod. Phys. 1992).

Hamiltonian systems with a bounded energy surface have a recurrence property discovered by Henri Poincarè (1890). The Poincarè recurrence theorem states that every state of a system will return arbitrarily close to its initial state in finite (potentially very long) time.

We know from Eq. {eq}`eq:energy_conserved` that autonomous Hamiltonian systems conserve energy and thus for $N$ degrees of freedom system, motion is constrained to a $(2N - 1)$-dimensional energy surface

\begin{equation*}
E = H(q_j, p_j),\quad \text{where } j \in \{1, 2, \ldots, N\}
\end{equation*}

If the energy surface is bounded (compact), then the Poincarè recurrence theorem states that almost all trajectories (all but a set of zero volume) that begin on a Poincarè surface of section will return to it. For a formal statement and proof, see Theorem 7.6.3 in {cite}`Wiggins2003`.

Combining Liouville's theorem and Poincarè recurrence theorem leads to the conclusion, that a set of points of a given volume on a Poincarè surface of section will return to the surface and intersect it in a set of points of equal volume. The return map associated with this surface is therefore volume preserving.

As pointed out by Arnold {cite}`Arnold76`, _"The following prediction is a paradoxical conclusion from the theorems of Poincarè and Liouville: if you open a partition separating a chamber containing gas and a chamber with a vacuum, then after a while the gas molecules will again collect in the first chamber. The resolution of the paradox lies in the fact that"_ a while  may be longer than the duration of the solar system's existence.

# References

```{bibliography} bibliography/chapter1.bib
```


# AST 341: Stars and Radiation
## homework #1 solutions

I'm using SymPy -- the symbolic math library for python -- to do the algebra here.  The next 2 cells load SymPy for interactive work and define the variables we are going to use (and treat as math symbols)

```python
from sympy import init_session
init_session()
%matplotlib inline
```

    IPython console for SymPy 1.6.2 (Python 3.8.5-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.6.2/

```python
rho = symbols('rho', cls=Function)
qc = symbols('q_c')
pc = symbols('p_c')
G = symbols('G')
Mstar, Rstar = symbols('M_\star R_\star')
r = symbols('r')
xi = symbols('xi')
beta = symbols('beta')
```

# 1.

Consider a mass profile in a star:
$$m(r) = 4\pi R^3 \beta \left [ \frac{1}{9} \left (\frac{r}{R} \right )^3 - \frac{1}{18} \left ( \frac{r}{R} \right)^6 \right ]$$
where $\beta$ is a constant

## a.

What is the density profile, $\rho(r)$, that gives rise to this mass?

First let's plot the function $m(r)$

```python
m = 4*pi*Rstar**3*beta*(Rational(1, 9) * (r/Rstar)**3 - Rational(1, 18) * (r/Rstar)**6)
m
```

```python
# to plot it, we need to make it dimensionless
z = (m/(4*pi*Rstar**3*beta)).subs(r, xi*Rstar)
plot(z, (xi, 0, 1), xlabel=r"$r/R_\star$", ylabel=r"$m(r)/(4\pi R_\star^3\beta)$")
```

Now we get the density as
\begin{equation}
\rho = \frac{1}{4\pi r^2} dM/dr
\end{equation}

```python
rho = m.diff(r, 1)/(4*pi*r**2)
rho = simplify(rho)
rho
```

## b.

What is $\beta$ in terms of the central density?

The central density is just $\rho(0)$

```python
rhoc = rho.subs(r, 0)
rhoc
```

So $\beta = 3 \rho_c$.  We can rewrite the density in terms of $\rho_c$ now

```python
rhoc = symbols("rho_c")
rho = simplify(rho.subs(beta, 3*rhoc))
rho
```

We can also rewrite the mass, $m(r)$, in terms of $\rho_c$:

```python
m = simplify(m.subs(beta, 3*rhoc))
m
```

Notice that the density vanishes at $r = R_\star$.  Now let's plot this.

```python
z = simplify((rho/rhoc).subs(r, xi*Rstar))
plot(z, (xi, 0, 1), xlabel=r"$r/R_\star$", ylabel=r"$\rho(r)/\rho_c$")
```

## c.

What is the ratio of the central density to the average density, $\rho_c / \bar{\rho}$?

The average density is just the total mass divided by the volume of the star.  First we get the total mass, $M_\star = m(R_\star)$

```python
Mstar = m.subs(r, Rstar)
Mstar
```

and now we compute
\begin{equation}
\bar{\rho} = \frac{M_\star}{(4/3) \pi R_\star^3}
\end{equation}

```python
rhobar = symbols(r"\bar{\rho}")
rhobar = Mstar / (Rational(4,3)*pi*Rstar**3)
rhobar
```

So $\bar{\rho} = \rho_c/2$ and the ratio of $\rho_c/\bar{\rho}$ is 2.

## d.

What is the central pressure of the star for this mass distribution?

Now we can integrate HSE.  We will do this as

$$p = p_c + \int_0^{R_\star} \frac{-Gm(r)}{r^2} \rho dr$$

Here I've written the integral in terms of $r$.
We'll substitute in our expressions for $m(r)$ and $\rho(r)$ in the integrand.

```python
pc = symbols("p_c")
p = pc + integrate(-G*m/r**2*rho, (r, 0, Rstar))
p
```

This was the integral to the surface, so this result is the surface pressure, but we know that $p(R_\star) = 0$,
so we can enforce that here to find $p_c$

```python
pc = solve(Eq(p, 0), pc)[0]
pc
```

This is $p_c$ in terms of $\rho_c$, but we can substitute in $\rho_c = 2 \bar{\rho}$ to find the central pressure in terms of $M_\star$ and $R_\star$.
(Note to make the code use $M_\star$ and not its equivalent in terms of $\rho_c$ found above, I am defining a new symbol $M$ here that will represent the total mass of the star.)

```python
M = symbols("M")
pc.subs(rhoc, 2*M/(Rational(4,3)*pi*Rstar**3))
```

From this expression, we see that $f = 63/(80\pi)$.

## e.

What is the total gravitational potential energy, $\Omega$, of the star?

We integrate

$$\Omega = -\int_0^{M_\star} \frac{Gm(r)}{r} dm = -\int_0^{R_\star} \frac{G m(r)}{r} 4\pi r^2 \rho dr$$

To do this, we use our expression for $m(r)$ and $\rho(r)$

```python
Omega = symbols("Omega")
Omega = integrate(-G*m*4*pi*r*rho, (r, 0, Rstar))
```

```python
Omega
```

We can rewrite this in terms of mass, by using $\rho_c = 2 \bar{\rho}$ that we found for this model above

```python
Omega = Omega.subs(rhoc, 2*M/(Rational(4,3)*pi*Rstar**3))
Omega
```

So this shows that $\alpha = 153/220$.

```python

```


 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli

```python
from __future__ import print_function
from __future__ import absolute_import

%matplotlib inline
import numpy
import matplotlib.pyplot as plt
```

# Numerical Differentiation

**GOAL:**  Given a set of $N+1$ points $(x_i, y_i)$ compute the derivative of a given order to a specified accuracy.

**Approach:** Find the interpolating polynomial $P_N(x)$ and differentiate that.

### Newton Polynomials

While the interpolating polynomial $P_N(x)$ through $N+1$ distinct points is unique,  it can be expressed in terms of many bases.

So far we have considered the monomial basis $< 1,x,x^2,\ldots,x^n>$ and the Lagrange basis $\ell_i(x)$

However, for our purposes here it's somewhat more convenient to use the *Newton Polynomial* Basis

\begin{align}
    n_0 &= 1\\
    n_j(x) &= \prod^{j-1}_{i=0} (x - x_i)\quad\mathrm{for}\quad j>0 \\
\end{align}

### Newton Polynomials

Unrolling them, we see
\begin{align}
    n_0(x) &= 1\\
    n_1(x) &= (x-x_0) \\
    n_2(x) &= (x - x_0)(x-x_1)\\
        &\vdots \\
    n_N(x) &= (x - x_0)(x - x_1)\cdots(x-x_{N-1})\\
\end{align}

Note:  The Newton polynomials have some features common to both the monomials and the Lagrange polynomials

* Like the monomials they are monic polynomials of increasing degree in $x$, i.e. $n_2(x)$ is quadratic
* Like the Lagrange polynomials, some Newton polynomials vanish identically at the interpolation points $x_i$.  In particular

$$
    n_j(x_i) = 0 \quad \mathrm{for}\quad j> i
$$

### Newton's Form

This last feature makes it convenient to solve for the interpolating polynomial.

$$P_N(x) = \sum^N_{j=0} a_j n_j(x)$$

as the $(N+1)\times (N+1)$ linear system

$$
    P_N(x_i) = y_i\quad\mathrm{for}\quad i=0,\ldots,N
$$

#### Becomes the lower triangular system

$$
\begin{bmatrix}
1 & 0 & 0 & \ldots & 0 \\
1 & (x_1-x_0) & 0 & \ldots & 0 \\
1 & (x_2-x_0) & (x_2-x_0)(x_2-x_1) & \ldots & 0 \\
\vdots & \vdots & \vdots & \vdots & \vdots  \\
1 & n_1(x_n) & n_2(x_n) & \ldots & n_n(x_n) \\
\end{bmatrix}\begin{bmatrix}
a_0 \\
a_1 \\
a_2 \\
\vdots \\
a_n \\
\end{bmatrix}=
\begin{bmatrix}
y_0 \\
y_1 \\
y_2 \\
\vdots \\
y_n \\
\end{bmatrix}
$$

Which can be solved sequentially starting with

\begin{align}
    a_0 & = y_0\\
    a_1 & = \frac{y_1 - y_0}{x_1 - x_0} \\
    a_2 & = \frac{y_2 - y_0 - a_1(x_2 - x_0)}{(x_2-x_0)(x_2-x_1)} \\
    &\vdots \\
\end{align}

### The divided differences

The $a_j = [y_0, \ldots, y_j]$ are also known as *the divided differences* which are defined recursively as

$$[y_i] = y_i \quad i \in \{0,\ldots, N+1\}$$

and

$$[y_i, \ldots , y_{i+j}] = \frac{[y_{i+1}, \ldots , y_{i + j}] - [y_{i},\ldots,y_{i+j-1}]}{x_{i+j} - x_{i}} \quad i \in \{0,\ldots,N+1 - j\} \quad j \in \{1,\ldots, N+1\}$$

These formulas are recursively defined but not so helpful, here are a few examples to start out with:

$$[y_0] = y_0$$

$$[y_0, y_1] = \frac{y_1 - y_0}{x_1 - x_0}$$

$$[y_0, y_1, y_2] = \frac{[y_1, y_2] - [y_0, y_1]}{x_{2} - x_{0}} = \frac{\frac{y_2 - y_1}{x_2 - x_1} - \frac{y_1 - y_0}{x_1 - x_0}}{x_2 - x_0} = \frac{y_2 - y_1}{(x_2 - x_1)(x_2 - x_0)} - \frac{y_1 - y_0}{(x_1 - x_0)(x_2 - x_0)}$$

The benefit of writing a polynomial like this is that it isolates the $x$ dependence (we can easily take derivatives of this form).

In general then $P_N(x)$ can be written in Newton's form as

$$P_N(x) = y_0 + (x-x_0)[y_0, y_1] + (x - x_0) (x - x_1) [y_0, y_1, y_2] + \cdots + (x-x_0) (x-x_1) \cdots (x-x_{N-1}) [y_0, y_1, \ldots, y_{N}]$$

Coefficients of Newton form are the divided differences.

Note: Much simpler in Lagrange basis,

```python
# Recursive function (calls itself until it gets to last iteration) (else-->if)
def divided_difference(x, y, N=50):
    """Compute the Nth divided difference using *x* and *y*"""
    if N == 0:
        raise Exception("Reached recursion limit!")

    # Reached the end of the recurssion
    if y.shape[0] == 1:
        return y[0]
    elif y.shape[0] == 2:
        return (y[1] - y[0]) / (x[1] - x[0])
    else:
        return (divided_difference(x[1:], y[1:], N=N-1) - divided_difference(x[:-1], y[:-1], N=N-1)) / (x[-1] - x[0])

def newton_basis(x, x0):
    """ construct newton basis for the interpolating polynomial given interpolation points x0"""
    basis = numpy.ones((len(x0), len(x)))
    for j in range(len(x0)):
        for i in range(j):
            basis[j, :] *= (x - x0[i])
    return basis

def P_newton(x, data):
    """ construct the interpolating polynomial of order N through the N+1 data points data using a Newton basis"""
    P = numpy.zeros(x.shape)
    basis = newton_basis(x, data[:,0])
    for j in range(data.shape[0]):
        P += divided_difference(data[:j + 1, 0], data[:j + 1, 1]) * basis[j, :]

    return P
```

```python
# Calculate a polynomial in Newton Form
data = numpy.array([[-2.0, 1.0], [-1.5, -1.0], [-0.5, -3.0], [0.0, -2.0], [1.0, 3.0], [2.0, 1.0]])
x = numpy.linspace(-2.0, 2.0, 100)
basis = newton_basis(x, data[:,0])
P = P_newton(x, data)
N = data.shape[0] - 1
```

```python
# Plot basis and interpolant
fig = plt.figure(figsize = (16,6))
fig.subplots_adjust(hspace=.5)

axes = [None, None]
axes[0] = fig.add_subplot(1, 2, 1)
axes[1] = fig.add_subplot(1, 2, 2)

for j in range(N + 1):
    axes[0].plot(x, basis[j, :], label='Basis {}'.format(j))
    axes[1].plot(data[j, 0], data[j, 1],'ko')
axes[1].plot(x, P)

axes[0].set_title("Newton Polynomial Basis")
axes[0].set_xlabel("x")
axes[0].set_ylabel("$n_j(x)$")
axes[0].grid()
axes[0].legend()

axes[1].set_title("Interpolant $P_%s(x)$" % N)
axes[1].set_xlabel("x")
axes[1].set_ylabel("$P_%s(x)$" % N)
axes[1].grid()

plt.show()
```

As another concrete example consider the general quadratic polynomial written in Newton's form

$$P_2(x) = [y_0] + (x - x_0) [y_0, y_1] + (x - x_0)(x - x_1) [y_0, y_1, y_2] \\= y_0 + (x - x_0) \frac{y_1 - y_0}{x_1 - x_0} + (x - x_0)(x - x_1) \left ( \frac{y_2 - y_1}{(x_2 - x_1)(x_2 - x_0)} - \frac{y_1 - y_0}{(x_1 - x_0)(x_2 - x_0)} \right )$$

Recall that the interpolating polynomial of degree $N$ through these points is unique!

### Error Analysis

Given $N + 1$ points we can form an interpolant $P_N(x)$ of degree $N$ where

$$f(x) = P_N(x) + R_N(x)$$

And we know from Lagrange's Theorem that the remainder term looks like

$$R_N(x) = (x - x_0)(x - x_1)\cdots (x - x_{N})(x - x_{N+1}) \frac{f^{(N+1)}(c)}{(N+1)!}$$

noting that we need to require that $f(x) \in C^{N+1}$ on the interval of interest.  Taking the derivative of the interpolant $P_N(x)$ then leads to

$$P_N'(x) = [y_0, y_1] + ((x - x_1) + (x - x_0)) [y_0, y_1, y_2] + \cdots + \left(\sum^{N-1}_{i=0}\left( \prod^{N-1}_{j=0,~j\neq i} (x - x_j) \right )\right ) [y_0, y_1, \ldots, y_N]$$

Similarly we can find the derivative of the remainder term $R_N(x)$ as

$$R_N'(x) = \left(\sum^{N}_{i=0} \left( \prod^{N}_{j=0,~j\neq i} (x - x_j) \right )\right ) \frac{f^{(N+1)}(c)}{(N+1)!}$$

Now if we consider the approximation of the derivative evaluated at one of our data points $(x_k, y_k)$ these expressions simplify such that

$$f'(x_k) = P_N'(x_k) + R_N'(x_k)$$

If we let $\Delta x = \max_i |x_k - x_i|$ we then know that the remainder term will be $\mathcal{O}(\Delta x^N)$ as $\Delta x \rightarrow 0$ thus showing that this approach converges and we can find arbitrarily high order approximations (ignoring floating point error).

### Caution

High order does not necessarily imply high-accuracy!

```python
def PN_prime(x, data):
    """ calculate derivative of interpolating polynomail at point x"""
    # General form of derivative of P_N'(x)
    P_prime = numpy.zeros(x.shape)
    newton_basis_prime = numpy.empty(x.shape)
    product = numpy.empty(x.shape)
    N = data.shape[0] - 1
    for n in range(N + 1):
        newton_basis_prime = 0.0
        for i in range(n):
            product = 1.0
            for j in range(n):
                if j != i:
                    product *= (x - data[j, 0])
            newton_basis_prime += product
        P_prime += divided_difference(data[:n+1, 0], data[:n+1, 1]) * newton_basis_prime

    return P_prime
```

```python
num_points = 4
xmin = -1.
xmax = 1.
data = numpy.empty((num_points, 2))
data[:, 0] = numpy.linspace(xmin, xmax, num_points)
data[:, 1] = numpy.sin(data[:, 0])
x = numpy.linspace(xmin, xmax, 100)
P_prime = PN_prime(x, data)
```

```python
fig = plt.figure(figsize=(8, 6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, numpy.cos(x), 'k', label="$f'(x)$")
axes.plot(x, P_prime, 'r--', label="$Pn'(x)$")
axes.plot(data[:,0], numpy.cos(data[:,0]), 'ro')
axes.legend(loc='best')
axes.set_title("$f'(x)$")
axes.set_xlabel("x")
axes.set_ylabel("$f'(x)$ and $\hat{f}'(x)$")
axes.grid()

plt.show()
```

- H refinement: decrease $\Delta x$
- P refinement: increase polynomial order (number of points interpolated from)

### Examples

Often in practice we only use a small number of data points to derive a differentiation formula.  In the context of differential equations we also often have $f(x)$ so that $f(x_k) = y_k$ and we can approximate the derivative of a known function $f(x)$.

### Example 1:  1st order Forward and Backward Differences

Using 2 points we can get an approximation that is $\mathcal{O}(\Delta x)$:

$$f'(x) \approx P_1'(x) = [y_0, y_1] = \frac{y_1 - y_0}{x_1 - x_0} = \frac{y_1 - y_0}{\Delta x} = \frac{f(x_1) - f(x_0)}{\Delta x}$$

We can also calculate the error as

$$R_1'(x) = -\Delta x \frac{f''(c)}{2}$$

We can also derive the "forward" and "backward" formulas by considering the question slightly differently.  Say we want $f'(x_n)$, then the "forward" finite-difference ($x_{n+1} > x_n$) can be written as

$$f'(x_n) \approx D_1^+ = \frac{f(x_{n+1}) - f(x_n)}{\Delta x}$$

and the "backward" finite-difference ($x_{n-1} < x_n$) as

$$f'(x_n) \approx D_1^- = \frac{f(x_n) - f(x_{n-1})}{\Delta x}$$

Note these approximations should be familiar to use as the limit as $\Delta x \rightarrow 0$ these are no longer approximations but equivalent definitions of the derivative at $x_n$.

```python
f = lambda x: numpy.sin(x)
f_prime = lambda x: numpy.cos(x)

# Use uniform discretization
x = numpy.linspace(-2 * numpy.pi, 2 * numpy.pi, 1000)
N = 10
x_hat = numpy.linspace(-2 * numpy.pi, 2 * numpy.pi, N)
delta_x = x_hat[1] - x_hat[0]

# Compute forward difference using a loop
f_prime_hat = numpy.empty(x_hat.shape)
for i in range(N - 1):
    f_prime_hat[i] = (f(x_hat[i+1]) - f(x_hat[i])) / delta_x
f_prime_hat[-1] = (f(x_hat[-1]) - f(x_hat[-2])) / delta_x

# Vector based calculation (one line)
# f_prime_hat[:-1] = (f(x_hat[1:]) - f(x_hat[:-1])) / (delta_x)

# Use first-order differences for points at edge of domain
f_prime_hat[-1] = (f(x_hat[-1]) - f(x_hat[-2])) / delta_x  # Backward Difference at x_N
```

```python
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)

axes.plot(x, f_prime(x), 'k', label='actual f\'(x)')
axes.plot(x_hat + 0.5 * delta_x, f_prime_hat, 'ro', label='approximate f\'(x)')
axes.set_xlim((x[0], x[-1]))
axes.set_ylim((-1.1, 1.1))
axes.grid()
axes.legend()

plt.show()
```

- Errors are largest where slope of f(x) is steepest ($\left|f'(x)\right|$ is large)
- Error as vector of errors as each point: can compute overall error using sum of abs value ($L^1$ norm) or sum of squares ($L^2$ norm) or $L^\infty$ (max(error))

#### Aside:  Computing Order of Convergence

Say we had the error $E(\Delta x)$ and we wanted to make a statement about the rate of convergence (note we can replace $E$ here with the $R$ from above).  Then we can do the following:
$$\begin{aligned}
    E(\Delta x) &= C \Delta x^n \\
    \log E(\Delta x) &= \log C + n \log \Delta x
\end{aligned}$$

The slope of the line is $n$ when modeling the error like this!  We can also match the first point by solving for $C$:

$$
    C = e^{\log E(\Delta x) - n \log \Delta x}
$$

```python
f = lambda x: numpy.sin(x)
f_prime = lambda x: numpy.cos(x)

# Compute the error as a function of delta_x
N_range = numpy.logspace(1, 4, 10, dtype=int)
delta_x = numpy.empty(N_range.shape)
error = numpy.empty((N_range.shape[0], 4))
for (i, N) in enumerate(N_range):
    x_hat = numpy.linspace(-2 * numpy.pi, 2 * numpy.pi, N)
    delta_x[i] = x_hat[1] - x_hat[0]

    # Compute forward difference
    f_prime_hat = numpy.empty(x_hat.shape)
    f_prime_hat[:-1] = (f(x_hat[1:]) - f(x_hat[:-1])) / (delta_x[i])

    # Use first-order differences for points at edge of domain
    f_prime_hat[-1] = (f(x_hat[-1]) - f(x_hat[-2])) / delta_x[i]  # Backward Difference at x_N

    # The differences in error computations is interesting here.  Note that the L_\infty norm returns a single
    # point-wise like error where as the L_2 and L_1 are adding error up so they exhbit less convergence
    error[i, 0] = numpy.linalg.norm(numpy.abs(f_prime(x_hat) - f_prime_hat), ord=numpy.infty)
    error[i, 1] = numpy.linalg.norm(numpy.abs(f_prime(x_hat + 0.5 * delta_x[i]) - f_prime_hat), ord=numpy.infty)
    error[i, 2] = numpy.linalg.norm(numpy.abs(f_prime(x_hat) - f_prime_hat), ord=1)
    error[i, 3] = numpy.linalg.norm(numpy.abs(f_prime(x_hat) - f_prime_hat), ord=2)

error = numpy.array(error)
delta_x = numpy.array(delta_x)

fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2)
fig.set_figheight(fig.get_figheight() * 2)
# plt.rc('legend', fontsize=6)
# plt.rc('font', size=6)

error_type = ['$L^\infty$', 'offset $L^\infty$', '$L^1$', '$L^2$']
for i in range(error.shape[1]):
    axes = fig.add_subplot(2, 2, i + 1)
    axes.loglog(delta_x, error[:, i], 'ko', label="Error")

    order_C = lambda delta_x, error, order: numpy.exp(numpy.log(error) - order * numpy.log(delta_x))
    axes.loglog(delta_x, order_C(delta_x[0], error[0, i], 1.0) * delta_x**1.0, 'r--', label="1st Order")
    axes.loglog(delta_x, order_C(delta_x[0], error[0, i], 2.0) * delta_x**2.0, 'b--', label="2nd Order")
    axes.legend(loc=4)
    axes.set_title("Convergence of 1st Order Differences - %s" % error_type[i])
    axes.set_xlabel("$\Delta x$")
    axes.set_ylabel("$|f'(x) - \hat{f}'(x)|$")

plt.show()
```

### Example 2: 2nd Order Centered Difference

Now lets use 3 points to calculate the 2nd order accurate finite-difference.  Consider the points $(x_{n}, y_{n})$, $(x_{n-1}, y_{n-1})$, and $(x_{n+1}, y_{n+1})$, from before we have

$$\begin{aligned}
    P_2(x) &= [f(x_0)] + (x - x_0) [f(x_0), f(x_1)] + (x - x_0)(x - x_1) [f(x_0), f(x_1), f(x_2)] \\
    &= f(x_0) + (x - x_0) \frac{f(x_1) - f(x_0)}{x_1 - x_0} + (x - x_0)(x - x_1) \left ( \frac{f(x_2) - f(x_1)}{(x_2 - x_1)(x_2 - x_0)} - \frac{f(x_1) - f(x_0)}{(x_1 - x_0)(x_2 - x_0)} \right )
\end{aligned}$$
Compute the formula for the derivative.  Assume that the distance between the $x_i$ are equal.

$$\begin{aligned}
    P_2'(x) &= [f(x_n), f(x_{n+1})] + ((x - x_n) + (x - x_{n+1})) [f(x_n), f(x_{n+1}), f(x_{n-1})] \\
    &= \frac{f(x_{n+1}) - f(x_n)}{x_{n+1} - x_n}  + ((x - x_n) + (x - x_{n+1}))\\ &* \left ( \frac{f(x_{n-1}) - f(x_{n+1})}{(x_{n-1} - x_{n+1})(x_{n-1} - x_n)} - \frac{f(x_{n+1}) - f(x_n)}{(x_{n+1} - x_n)(x_{n-1} - x_n)} \right )
\end{aligned}$$

Note: $x_{n-1} - x_{n+1} = 2\Delta x$ and $x_{n-1} - x_n = \Delta x$ and $x_{n+1} - x_n = \Delta x$

Evaluating at $x_n$ and assuming the points $x_{n-1}, x_n, x_{n+1}$ are evenly spaced leads to

$$\begin{aligned}
    P_2'(x_n) &= \frac{f(x_{n+1}) - f(x_n)}{\Delta x} - \Delta x \left ( \frac{f(x_{n-1}) - f(x_{n+1})}{2\Delta x^2} + \frac{f(x_{n+1}) - f(x_n)}{\Delta x^2} \right ) \\
    &=\frac{f(x_{n+1}) - f(x_n)}{\Delta x} - \left ( \frac{f(x_{n+1}) - 2f(x_n) + f(x_{n-1})}{2\Delta x}\right ) \\
    &=\frac{2f(x_{n+1}) - 2f(x_n) - f(x_{n+1}) + 2f(x_n) - f(x_{n-1})}{2 \Delta x} \\
    &=\frac{f(x_{n+1}) - f(x_{n-1})}{2 \Delta x}
\end{aligned}$$

This finite-difference is second order accurate and is centered about the point it is meant to approximate ($x_n$).  We can show that it is second order by again considering the remainder term's derivative

$$\begin{aligned}
    R_2'(x) &= \left(\sum^{2}_{i=0} \left( \prod^{2}_{j=0,~j\neq i} (x - x_j) \right )\right ) \frac{f'''(c)}{3!} \\
    &= \left ( (x - x_{n+1}) (x - x_{n-1}) + (x-x_n) (x-x_{n-1}) + (x-x_n)(x-x_{n+1}) \right ) \frac{f'''(c)}{3!}
\end{aligned}$$

Again evaluating this expression at $x = x_n$ and assuming evenly space points we have

$$R_2'(x_n) = -\Delta x^2 \frac{f'''(c)}{3!}$$

showing that our error is $\mathcal{O}(\Delta x^2)$.

```python
f = lambda x: numpy.sin(x)
f_prime = lambda x: numpy.cos(x)

# Use uniform discretization
x = numpy.linspace(-2 * numpy.pi, 2 * numpy.pi, 1000)
N = 11
x_hat = numpy.linspace(-2 * numpy.pi, 2 * numpy.pi, N)
delta_x = x_hat[1] - x_hat[0]

# Compute derivative
f_prime_hat = numpy.empty(x_hat.shape)
f_prime_hat[1:-1] = (f(x_hat[2:]) - f(x_hat[:-2])) / (2. * delta_x)

# Use first-order differences for points at edge of domain
f_prime_hat[0] = (f(x_hat[1]) - f(x_hat[0])) / delta_x     # Forward Difference at x_0
f_prime_hat[-1] = (f(x_hat[-1]) - f(x_hat[-2])) / delta_x  # Backward Difference at x_N
#f_prime_hat[0] = (-3.0 * f(x_hat[0]) + 4.0 * f(x_hat[1]) - f(x_hat[2])) / (2.0 * delta_x)
#f_prime_hat[-1] = (3.0 * f(x_hat[-1]) - 4.0 * f(x_hat[-2]) + f(x_hat[-3])) / (2.0 * delta_x)
```

```python
fig = plt.figure()
plt.rcdefaults()
axes = fig.add_subplot(1, 1, 1)

axes.plot(x, f_prime(x), 'k',label="$f'(x)$")
axes.plot(x_hat, f_prime_hat, 'ro')
axes.plot(x, f(x),'g--',label="$f(x)$")
axes.plot(x_hat, f(x_hat), 'go')
axes.set_xlim((x[0], x[-1]))
# axes.set_ylim((-1.1, 1.1))
axes.grid()
axes.legend()

plt.show()
```

```python
# Compute the error as a function of delta_x
delta_x = []
error = []
# for N in range(2, 101):
for N in range(50, 1000, 50):
    x_hat = numpy.linspace(-2 * numpy.pi, 2 * numpy.pi, N + 1)
    delta_x.append(x_hat[1] - x_hat[0])

    # Compute derivative
    f_prime_hat = numpy.empty(x_hat.shape)
    f_prime_hat[1:-1] = (f(x_hat[2:]) - f(x_hat[:-2])) / (2 * delta_x[-1])

    # Use first-order differences for points at edge of domain
    f_prime_hat[0] = (f(x_hat[1]) - f(x_hat[0])) / delta_x[-1]
    f_prime_hat[-1] = (f(x_hat[-1]) - f(x_hat[-2])) / delta_x[-1]
    # Use second-order differences for points at edge of domain
    f_prime_hat[0] =  (-3.0 * f(x_hat[0])  +  4.0 * f(x_hat[1])  + - f(x_hat[2]))  / (2.0 * delta_x[-1])
    f_prime_hat[-1] = ( 3.0 * f(x_hat[-1]) + -4.0 * f(x_hat[-2]) +   f(x_hat[-3])) / (2.0 * delta_x[-1])

    # Compute Error
    error.append(numpy.linalg.norm(numpy.abs(f_prime(x_hat) - f_prime_hat), ord=numpy.infty))
#     error.append(numpy.linalg.norm(numpy.abs(f_prime(x_hat) - f_prime_hat), ord=2))

error = numpy.array(error)
delta_x = numpy.array(delta_x)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)

axes.loglog(delta_x, error, "ro", label="Approx. Derivative")

order_C = lambda delta_x, error, order: numpy.exp(numpy.log(error) - order * numpy.log(delta_x))
axes.loglog(delta_x, order_C(delta_x[0], error[0], 1.0) * delta_x**1.0, 'b--', label="1st Order error")
axes.loglog(delta_x, order_C(delta_x[0], error[0], 2.0) * delta_x**2.0, 'r--', label="2nd Order error")
axes.legend(loc=4)
axes.set_title("Convergence of 2nd Order Differences")
axes.set_xlabel("$\Delta x$")
axes.set_ylabel("$|f'(x) - \hat{f}'(x)|$")
axes.grid()

plt.show()
```

### Example 3: Alternative Derivations

An alternative method for finding finite-difference formulas is by using Taylor series expansions about the point we want to approximate.  The Taylor series about $x_n$ is

$$f(x) = f(x_n) + (x - x_n) f'(x_n) + \frac{(x - x_n)^2}{2!} f''(x_n) + \frac{(x - x_n)^3}{3!} f'''(x_n) + \mathcal{O}((x - x_n)^4)$$

Note: slightly different error function

Say we want to derive the second order accurate, first derivative approximation that we just did, this requires the values $(x_{n+1}, f(x_{n+1})$ and $(x_{n-1}, f(x_{n-1})$.  We can express these values via our Taylor series approximation above as

\begin{aligned}
    f(x_{n+1}) &= f(x_n) + (x_{n+1} - x_n) f'(x_n) + \frac{(x_{n+1} - x_n)^2}{2!} f''(x_n) + \frac{(x_{n+1} - x_n)^3}{3!} f'''(x_n) + \mathcal{O}((x_{n+1} - x_n)^4) \\
\end{aligned}

or
\begin{aligned}
&= f(x_n) + \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) + \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)
\end{aligned}

and

\begin{align}
f(x_{n-1}) &= f(x_n) + (x_{n-1} - x_n) f'(x_n) + \frac{(x_{n-1} - x_n)^2}{2!} f''(x_n) + \frac{(x_{n-1} - x_n)^3}{3!} f'''(x_n) + \mathcal{O}((x_{n-1} - x_n)^4)
\end{align}

\begin{align}
&= f(x_n) - \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) - \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)
\end{align}

Or all together (__for regularly spaced points__),
\begin{align}
f(x_{n+1}) &= f(x_n) + \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) + \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)\\
f(x_{n-1})&= f(x_n) - \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) - \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)
\end{align}

Now to find out how to combine these into an expression for the derivative we assume our approximation looks like

$$
    f'(x_n) + R'(x_n) = A f(x_{n+1}) + B f(x_n) + C f(x_{n-1})
$$

where $R(x_n)$ is our error.

Plugging in the Taylor series approximations we find

$$\begin{aligned}
    f'(x_n) + R'(x_n) &= A \left ( f(x_n) + \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) + \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)\right ) \\
    & + B f(x_n) \\
    & + C \left ( f(x_n) - \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) - \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4) \right )
\end{aligned}$$

__Method of undetermined coefficients__

Since we want $R(x_n) = \mathcal{O}(\Delta x^2)$ we want all terms lower than this to cancel __*except for those multiplying $f'(x_n)$ as those should sum to 1*__ {since we want to find $f'(x_n)$} to give us our approximation.  Collecting the terms with common evaluations of the derivatives on $f(x_n)$ we get a series of expressions for the coefficients $A$, $B$, and $C$ based on the fact we want an approximation to $f'(x_n)$.  The $n=0$ terms collected are $A + B + C$ and are set to 0 as we want the $f(x_n)$ term to also cancel.

$$\begin{aligned}
    f(x_n):&  &A + B + C &= 0 \\
    f'(x_n): & &A \Delta x - C \Delta x &= 1 \\
    f''(x_n): & &A \frac{\Delta x^2}{2} + C \frac{\Delta x^2}{2} &= 0
\end{aligned} $$

This last equation $\Rightarrow A = -C$, using this in the second equation gives $A = \frac{1}{2 \Delta x}$ and $C = -\frac{1}{2 \Delta x}$.  The first equation then leads to $B = 0$.  Putting this altogether then gives us our previous expression including an estimate for the error:

$$\begin{aligned}
    f'(x_n) + R(x_n) &= \quad \frac{1}{2 \Delta x} \left ( f(x_n) + \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) + \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)\right ) \\
    & \quad + 0 \cdot f(x_n) \\
    & \quad - \frac{1}{2 \Delta x} \left ( f(x_n) - \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) - \frac{\Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4) \right ) \\
    &=  f'(x_n) + \frac{1}{2 \Delta x} \left ( \frac{2 \Delta x^3}{3!} f'''(x_n) + \mathcal{O}(\Delta x^4)\right )
\end{aligned}$$
so that we find
$$
    R(x_n) = \frac{\Delta x^2}{3!} f'''(x_n) + \mathcal{O}(\Delta x^3) = \mathcal{O}(\Delta x^2)
$$

#### Another way...

There is one more way to derive the second order accurate, first order finite-difference formula.  Consider the two first order forward and backward finite-differences averaged together:

$$\frac{D_1^+(f(x_n)) + D_1^-(f(x_n))}{2} = \frac{f(x_{n+1}) - f(x_n) + f(x_n) - f(x_{n-1})}{2 \Delta x} = \frac{f(x_{n+1}) - f(x_{n-1})}{2 \Delta x}$$

### Example 4: Higher Order Derivatives

Using our Taylor series approach lets derive the second order accurate second derivative formula.  Again we will use the same points and the Taylor series centered at $x = x_n$ so we end up with the same expression as before:

$$\begin{aligned}
    f''(x_n) + R''(x_n) &= \quad A \left ( f(x_n) + \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) + \frac{\Delta x^3}{3!} f'''(x_n) + \frac{\Delta x^4}{4!} f^{(4)}(x_n) + \mathcal{O}(\Delta x^5)\right ) \\
    &+ \quad B \cdot f(x_n) \\
    &+ \quad C \left ( f(x_n) - \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) - \frac{\Delta x^3}{3!} f'''(x_n) + \frac{\Delta x^4}{4!} f^{(4)}(x_n) + \mathcal{O}(\Delta x^5) \right )
\end{aligned}$$

except this time we want to leave $f''(x_n)$ on the right hand side.

Try out the same trick as before and see if you can setup the equations that need to be solved.

Doing the same trick as before we have the following expressions:

$$\begin{aligned}
    f(x_n): & & A + B + C &= 0\\
    f'(x_n): & & A \Delta x - C \Delta x &= 0\\
    f''(x_n): & & A \frac{\Delta x^2}{2} + C \frac{\Delta x^2}{2} &= 1
\end{aligned}$$

The second equation implies $A = C$ which combined with the third implies

$$A = C = \frac{1}{\Delta x^2}$$

Finally the first equation gives

$$B = -2A = -\frac{2}{\Delta x^2}$$

leading to the final expression

$$\begin{aligned}
    f''(x_n) + R(x_n) &= \quad \frac{1}{\Delta x^2} \left ( f(x_n) + \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) + \frac{\Delta x^3}{3!} f'''(x_n) + \frac{\Delta x^4}{4!} f^{(4)}(x_n) + \mathcal{O}(\Delta x^5)\right ) \\
    &+ \quad -\frac{2}{\Delta x^2} \cdot f(x_n) \\
    &+ \quad \frac{1}{\Delta x^2} \left ( f(x_n) - \Delta x f'(x_n) + \frac{\Delta x^2}{2!} f''(x_n) - \frac{\Delta x^3}{3!} f'''(x_n) + \frac{\Delta x^4}{4!} f^{(4)}(x_n) + \mathcal{O}(\Delta x^5) \right ) \\
    &= f''(x_n) + \frac{1}{\Delta x^2} \left(\frac{2 \Delta x^4}{4!} f^{(4)}(x_n) + \mathcal{O}(\Delta x^5) \right )
\end{aligned}
$$
so that

$$
    R(x_n) = \frac{\Delta x^2}{12} f^{(4)}(x_n) + \mathcal{O}(\Delta x^3)
$$

```python
f = lambda x: numpy.sin(x)
f_dubl_prime = lambda x: -numpy.sin(x)

# Use uniform discretization
x = numpy.linspace(-2 * numpy.pi, 2 * numpy.pi, 1000)
N = 40
x_hat = numpy.linspace(-2 * numpy.pi, 2 * numpy.pi, N)
delta_x = x_hat[1] - x_hat[0]

# Compute derivative
f_dubl_prime_hat = numpy.empty(x_hat.shape)
f_dubl_prime_hat[1:-1] = (f(x_hat[2:]) -2.0 * f(x_hat[1:-1]) + f(x_hat[:-2])) / (delta_x**2)

# Use first-order differences for points at edge of domain
f_dubl_prime_hat[0] = (2.0 * f(x_hat[0]) - 5.0 * f(x_hat[1]) + 4.0 * f(x_hat[2]) - f(x_hat[3])) / delta_x**2
f_dubl_prime_hat[-1] = (2.0 * f(x_hat[-1]) - 5.0 * f(x_hat[-2]) + 4.0 * f(x_hat[-3]) - f(x_hat[-4])) / delta_x**2
```

```python
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)

axes.plot(x, f_dubl_prime(x), 'k')
axes.plot(x_hat, f_dubl_prime_hat, 'ro')
axes.set_xlim((x[0], x[-1]))
axes.set_ylim((-1.1, 1.1))
axes.grid()

plt.show()
```

```python
f = lambda x: numpy.sin(x)
f_dubl_prime = lambda x: -numpy.sin(x)

# Compute the error as a function of delta_x
delta_x = []
error = []
# for N in xrange(2, 101):
for N in range(50, 1000, 50):
    x_hat = numpy.linspace(-2 * numpy.pi, 2 * numpy.pi, N)
    delta_x.append(x_hat[1] - x_hat[0])

    # Compute derivative
    f_dubl_prime_hat = numpy.empty(x_hat.shape)
    f_dubl_prime_hat[1:-1] = (f(x_hat[2:]) -2.0 * f(x_hat[1:-1]) + f(x_hat[:-2])) / (delta_x[-1]**2)

    # Use second-order differences for points at edge of domain
    f_dubl_prime_hat[0] = (2.0 * f(x_hat[0]) - 5.0 * f(x_hat[1]) + 4.0 * f(x_hat[2]) - f(x_hat[3])) / delta_x[-1]**2
    f_dubl_prime_hat[-1] = (2.0 * f(x_hat[-1]) - 5.0 * f(x_hat[-2]) + 4.0 * f(x_hat[-3]) - f(x_hat[-4])) / delta_x[-1]**2

    error.append(numpy.linalg.norm(numpy.abs(f_dubl_prime(x_hat) - f_dubl_prime_hat), ord=numpy.infty))

error = numpy.array(error)
delta_x = numpy.array(delta_x)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)

# axes.plot(delta_x, error)
axes.loglog(delta_x, error, "ko", label="Approx. Derivative")
order_C = lambda delta_x, error, order: numpy.exp(numpy.log(error) - order * numpy.log(delta_x))
axes.loglog(delta_x, order_C(delta_x[2], error[2], 1.0) * delta_x**1.0, 'b--', label="1st Order")
axes.loglog(delta_x, order_C(delta_x[2], error[2], 2.0) * delta_x**2.0, 'r--', label="2nd Order")
axes.legend(loc=4)
axes.grid()

plt.show()
```


# Systems of ODEs
## CH EN 2450 - Numerical Methods
**Prof. Tony Saad (www.tsaad.net)
Department of Chemical Engineering
University of Utah**

```python
import numpy as np
from numpy import *
%matplotlib inline
%config InlineBackend.figure_format = 'svg'

import matplotlib.pyplot as plt
from odeintegrate import *
from scipy.integrate import odeint
```

# Kinetics Example
Solve the system of ODEs:
\begin{equation}
\frac{\text{d}A}{\text{d}t} = -k_1 A + k_3 B C \\
\frac{\text{d}B}{\text{d}t} = k_1 A  - k_2 B^2 - k_3 B C \\
\frac{\text{d}C}{\text{d}t} = k_2 B^2
\end{equation}

```python
def forward_euler_system(rhsvec, f0vec, tend, dt):
    '''
    Solves a system of ODEs using the Forward Euler method
    '''
    nsteps = int(tend/dt)
    neqs = len(f0vec)
    f = np.zeros( (neqs, nsteps) )
    f[:,0] = f0vec
    time = np.linspace(0,tend,nsteps)
    for n in np.arange(nsteps-1):
        t = time[n]
        f[:,n+1] = f[:,n] + dt * rhsvec(f[:,n], t)
    return time, f

```

```python
def rhs_kinetics(f,t):
    A = f[0]
    B = f[1]
    C = f[2]
    k1 = 0.04
    k2 = 3e7
    k3 = 1e4
    rhs1 = - k1*A + k3*B*C
    rhs2 = k1*A - k2*B*B - k3*B*C
    rhs3  = k2*B*B
    return np.array([rhs1,rhs2,rhs3])
```

```python
import time
tic = time.clock()
toc = time.clock()
toc - tic
tend = 10
dt = 1e-5
t = np.linspace(0,10,10)
y0 = np.array([1,0,0])
sol = odeint(rhs_kinetics,y0,t) # use odeint
timefe, solfe = forward_euler_system(rhs_kinetics,y0,tend,dt)
plt.plot(t,sol[:,0],t,sol[:,1],t,sol[:,2],timefe,solfe[0])
plt.grid()
```

topics/initial-value-problems/System of Equations - Kinetics.ipynb


# Análisis de Estado Estable

## Probabilidades de estado estable

Podemos utilizar la ecuación de Chapman-Kolgomorov para analizar la evolución de las probabilidades de transición de $n$-pasos. Utilicemos los datos de la parte anterior:

```python
import numpy as np
p = np.array([[0.7, 0.1, 0.2], [0.2, 0.7, 0.1], [0.5, 0.2, 0.3]])
p
```

    array([[0.7, 0.1, 0.2],
           [0.2, 0.7, 0.1],
           [0.5, 0.2, 0.3]])

```python
p2 = p @ p
p2
```

    array([[0.61, 0.18, 0.21],
           [0.33, 0.53, 0.14],
           [0.54, 0.25, 0.21]])

```python
p4 = p2 @ p2
p4
```

    array([[0.5449, 0.2577, 0.1974],
           [0.4518, 0.3753, 0.1729],
           [0.5253, 0.2822, 0.1925]])

```python
p8 = p4 @ p4
p8
```

    array([[0.51703909, 0.29284182, 0.19011909],
           [0.50657073, 0.30607133, 0.18735794],
           [0.51485418, 0.29560297, 0.18954285]])

```python
p16 = p8 @ p8
p16
```

    array([[0.51355812, 0.29724092, 0.18920096],
           [0.51342566, 0.29740832, 0.18916602],
           [0.51353048, 0.29727586, 0.18919366]])

En este caso, es notable observar como los valores en cada columna van convergiendo a un mismo valor.

Esta es una propiedad que tienen todas las cadenas de Markov que forman un conjunto irreducible no periódico. Estas cadenas también se conocen como **ergódicas**.

```python
import matplotlib.pyplot as plt

fig, ax = plt.subplots()
nt = 15
pt = np.empty(3 * 3 * nt)
pt.shape = (3, 3, nt)
p_aux = p
n, m = p.shape
for t in range(nt):
    pt[:, :, t] = p_aux
    p_aux = p_aux @ p

for j in range(n):
    for k in range(m):
        plt.plot(pt[k, j], color=f"#{format(k*96, '02x')}{format(j*96, '02x')}32", label=f'$p_{{{k}{j}}}$')
ax.set_xlabel('n-ésima potencia de $\mathbf{P}$')
ax.set_ylabel('Probabilidad')
plt.legend(loc='lower right', bbox_to_anchor=(1.2, 0))
plt.show()
```

Se puede observar que cada grupo de líneas que corresponden a las probabilidades de n-ésimo paso de una misma columna convergen. Estos valores a los que convergen todos los valores de una misma columna se conocen como **Probabilidades de estado estable** que se denotan por $\mathbf{\Pi} = \lim\limits_{n \to \infty}\mathbf{P}^n$, y pueden calcularse directamente observando que en algún momento:

\begin{equation}
    \mathbf{\Pi} \cdot \mathbf{P} = \mathbf{\Pi}
\end{equation}

Lo que implica que:

\begin{equation}
    \pi_j = \sum_{i=1}^{N}\pi_i p_{ij}~\forall j=1, 2, \dots, N
\end{equation}

Esto conduce al sistema de ecuaciones:

\begin{equation}
     [\mathbf{P}^t - \mathbf{I}]~\mathbf{\Pi} = \mathbf{0}
\end{equation}

Como todas las ecuaciones quedan igualadas a cero se genera una dependencia lineal, por lo que se debe sustituir una de estas ecuaciones (es decir, una fila del sistema anterior) por el hecho que:

\begin{equation}
    \sum_{i=1}^{N}\pi_i = 1
\end{equation}

Así, podemos resolverlo matricialmente mediante la siguiente función:

```python
def steady_state(p):
    n = p.shape[0]
    a = p.T - np.eye(n)
    a[n - 1, :] = np.ones(n)
    b = np.append(np.zeros(n - 1), 1)
    return np.linalg.solve(a, b)
```

```python
from IPython.display import display, Markdown

pi = steady_state(p)
display(Markdown(rf'$\pi_{{1}} =$ {pi[0]:.6f}'))
display(Markdown(rf'$\pi_{{2}} =$ {pi[1]:.6f}'))
display(Markdown(rf'$\pi_{{3}} =$ {pi[2]:.6f}'))
```

$\pi_{1} =$ 0.513514

$\pi_{2} =$ 0.297297

$\pi_{3} =$ 0.189189

La forma básica en la que pueden interpretarse estos valores es la probabilidad de que el sistema (representado por la cadena de markov) se encuentre en uno estado específico después de un número suficientemente grande de transiciones.

En este caso, $\pi_{1} = 0.513514$ indica que la probabilidad de conseguir un día soleado dentro de 100 o 200 días es aproximadamente $0.51$, independientemente del estado del clima de hoy.

Hay otra interpretación probablemente más interesante de las probabilidades de estado estable, a partir del propio concepto de la probabilidad como frecuencia relativa, y es que de un número suficientemente grande de días, estas probabilidades representan el número de estimado de ocurrencias de un estado.

En este caso, en un año el número de días soleados estaría dado por:

```python
365 * pi[0]
```

    187.4324324324324

Esto indica que poco más de $187$ días al año serán soleados. Y en general, que el $51.35 \%$ de los días serán soleados.