# Random Signals and LTI-Systems

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing.

## The Wiener Filter

The [Wiener filter](https://en.wikipedia.org/wiki/Wiener_filter), named after [*Nobert Wiener*](https://en.wikipedia.org/wiki/Norbert_Wiener), aims at estimating an unknown random signal by filtering a noisy distorted observation of the signal. It has a wide range of applications in signal processing. For instance, noise reduction, system identification, deconvolution and signal detection. The Wiener filter is frequently used as basic building block for algorithms that denoise audio signals, like speech, or remove noise from a image.

### Signal Model

The following signal model is underlying the Wiener filter

The random signal $s[k]$ is subject to distortion by the linear time-invariant (LTI) system $G(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and additive noise $n[k]$, resulting in the observed signal $x[k] = s[k] * g[k] + n[k]$. The additive noise $n[k]$ is assumed to be uncorrelated from $s[k]$. It is furthermore assumed that all random signals are wide-sense stationary (WSS). This distortion model holds for many practical problems, like e.g. the measurement of a physical quantity by a sensor.

The basic concept of the Wiener filter is to apply the LTI system $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ to the observed signal $x[k]$ such that the output signal $y[k] = x[k] * h[k]$ matches $s[k]$ as close as possible. In order to quantify the deviation between $y[k]$ and $s[k]$, the error signal

\begin{equation}
e[k] = y[k] - s[k]
\end{equation}

is introduced. The error $e[k]$ equals zero, if the output signal $y[k]$ perfectly matches $s[k]$. In general, this goal cannot be achieved in a strict sense. As alternative one could aim at minimizing the linear average of the error $e[k]$. However, this quantity is not very well suited for optimization since it is signed. Instead, the quadratic average of the error $e[k]$ is used in the Wiener filter and other techniques. This measure is known as [*mean squared error*](https://en.wikipedia.org/wiki/Mean_squared_error) (MSE). It is defined as

\begin{equation}
E \{ |e[k]|^2 \} = \frac{1}{2 \pi} \int\limits_{-\pi}^{\pi} \Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \,\mathrm{d}\Omega = \varphi_{ee}[\kappa] \Big\vert_{\kappa = 0}
\end{equation}

the equalities are deduced from the properties of the power spectral density (PSD) and the auto-correlation function (ACF). Above equation is referred to as [*cost function*](https://en.wikipedia.org/wiki/Loss_function). We aim at minimizing the cost function, hence minimizing the MSE between the original signal $s[k]$ and its estimate $y[k]$. The solution is referred to as [minimum mean squared error](https://en.wikipedia.org/wiki/Minimum_mean_square_error) (MMSE) solution.

### Transfer Function of the Wiener Filter

At first, the Wiener filter shall only have access to the observed signal $x[k]$ and selected statistical measures. It is assumed that the cross-power spectral density $\Phi_{xs}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ between the observed signal $x[k]$ and the original signal $s[k]$, and the PSD of the observed signal $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ are known. This knowledge can either be gained by estimating both from measurements taken at an actual system or by using suitable statistical models.

The optimal filter is found by minimizing the MSE $E \{ |e[k]|^2 \}$ with respect to the transfer function $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. The solution of this optimization problem goes beyond the scope of this notebook and can be found in the literature, e.g. [[Girod et. al](../index.ipynb#Literature)]. The transfer function of the Wiener filter is given as

\begin{equation}
H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{\Phi_{sx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}{\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})} = \frac{\Phi_{xs}(\mathrm{e}^{\,-\mathrm{j}\,\Omega})}{\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}
\end{equation}

for $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \neq 0$.
No knowledge on the actual distortion process is required besides the assumption of an LTI system and additive noise. Only the PSDs $\Phi_{sx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ have to be known in order to estimate $s[k]$ from $x[k]$ in the MMSE sense. Care has to be taken that the filter $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is causal and stable in practical applications.

**Example - Denoising of an Deterministic Signal**

The following example considers the estimation of the original signal from a distorted observation. It is assumed that the original signal is a deterministic signal $s[k] = \sin[\Omega_0\,k]$ which is distorted by an LTI system and additive wide-sense ergodic normal distributed zero-mean white noise with PSD $N_0 = 0.1$. The PSDs $\Phi_{sx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ are estimated from $s[k]$ and $x[k]$ using the [Welch technique](../spectral_estimation_random_signals/welch_method.ipynb). The Wiener filter is applied to the observation $x[k]$ in order to compute the estimate $y[k]$ of $s[k]$. Note that the discrete signals have been illustrated by continuous lines for ease of illustration.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

N = 8129  # number of samples
M = 256  # length of Wiener filter
Om0 = 0.1*np.pi  # frequency of original signal
N0 = 0.1  # PSD of additive white noise

# generate original signal
s = np.cos(Om0 * np.arange(N))
# generate observed signal
g = 1/20*np.asarray([1, 2, 3, 4, 5, 4, 3, 2, 1])
np.random.seed(1)
n = np.random.normal(size=N, scale=np.sqrt(N0))
x = np.convolve(s, g, mode='same') + n
# estimate (cross) PSDs using Welch technique
f, Pxx = sig.csd(x, x, nperseg=M)
f, Psx = sig.csd(s, x, nperseg=M)
# compute Wiener filter
H = Psx/Pxx
H = H * np.exp(-1j*2*np.pi/len(H)*np.arange(len(H))*(len(H)//2))  # shift for causal filter
h = np.fft.irfft(H)
# apply Wiener filter to observation
y = np.convolve(x, h, mode='same')

# plot (cross) PSDs
Om = np.linspace(0, np.pi, num=len(H))

plt.figure(figsize=(10, 4))
plt.subplot(121)
plt.plot(Om, 20*np.log10(np.abs(.5*Pxx)), label=r'$| \Phi_{xx}(e^{j \Omega}) |$ in dB')
plt.plot(Om, 20*np.log10(np.abs(.5*Psx)), label=r'$| \Phi_{sx}(e^{j \Omega}) |$ in dB')
plt.title('(Cross) PSDs')
plt.xlabel(r'$\Omega$')
plt.legend()
plt.axis([0, np.pi, -60, 40])
plt.grid()

# plot transfer function of Wiener filter
plt.subplot(122)
plt.plot(Om, 20*np.log10(np.abs(H)))
plt.title('Transfer function of Wiener filter')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$| H(e^{j \Omega}) |$ in dB')
plt.axis([0, np.pi, -150, 3])
plt.grid()
plt.tight_layout()

# plot signals
idx = np.arange(500, 600)
plt.figure(figsize=(10, 4))
plt.plot(idx, x[idx], label=r'observed signal $x[k]$')
plt.plot(idx, s[idx], label=r'original signal $s[k]$')
plt.plot(idx, y[idx], label=r'estimated signal $y[k]$')
plt.title('Signals')
plt.xlabel(r'$k$')
plt.axis([idx[0], idx[-1], -1.5, 1.5])
plt.legend()
plt.grid()
```

**Exercise**

* Take a look at the PSDs and the resulting transfer function of the Wiener filter. How does the Wiener filter remove the noise from the observed signal?
* Change the frequency `Om0` of the original signal $s[k]$ and the noise power `N0` of the additive noise. What changes?

Solution: The cross-PSD $\Phi_{sx}(e^{j \Omega})$ has its maximum at the frequency $\Omega_0$ of the original signal $s[k]$, besides this frequency the PSD $\Phi_{xx}(e^{j \Omega})$ has approximately the value of the PSD $\Phi_{nn}(e^{j \Omega}) = N_0$ of the additive noise. The resulting Wiener filter $G(e^{j \Omega})$ has an attenuation of 0 dB at $\Omega_0$ which increases off this frequency. It has the characteristics of a band-pass filter centered at $\Omega_0$. By filtering the observed signal $x[k]$ with the Wiener filter, the major portion of the additive white noise is removed. However, around $\Omega_0$ both the original signal $s[k]$ and noise pass the filter. This explains the remaining small deviations between the estimated signal $y[k]$ and the original signal $y[k]$. The adaption of the Wiener filter to the original signal and additive noise changes when changing its frequency and PSD.

### Wiener Deconvolution

As discussed above, the general formulation of the Wiener filter is based on the knowledge of the PSDs $\Phi_{sx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ characterizing the distortion process and the observed signal, respectively. These PSDs can be derived from the PSDs of the original signal $\Phi_{ss}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and the noise $\Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$, and the transfer function $G(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of the distorting system.

Under the assumption that $n[k]$ is uncorrelated from $s[k]$, the PSD $\Phi_{sx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ can be derived as

\begin{equation}
\Phi_{sx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \Phi_{ss}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot G(\mathrm{e}^{\,-\mathrm{j}\,\Omega})
\end{equation}

and

\begin{equation}
\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \Phi_{ss}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot  |G(\mathrm{e}^{\,\mathrm{j}\,\Omega})|^2 + \Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})
\end{equation}

Introducing these results into the general formulation of the Wiener filter yields

\begin{equation}
H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{\Phi_{ss}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot G(\mathrm{e}^{\,-\mathrm{j}\,\Omega})}{\Phi_{ss}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot  |G(\mathrm{e}^{\,\mathrm{j}\,\Omega})|^2 + \Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}
\end{equation}

This specialization is also known as [*Wiener deconvolution filter*](https://en.wikipedia.org/wiki/Wiener_deconvolution). The filter can be derived from the PSDs of the original signal $s[k]$ and the noise $n[k]$, and the transfer function $G(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of the distorting system. This form is especially useful when the PSDs can be modeled reasonably well by analytic probabilty density functions (PSDs). In many cases, the additive noise can be modeled as white noise $\Phi_{nn}(e^{j \Omega}) = N_0$.

### Interpretation

The Wiener deconvolution filter derived above is also useful for an interpretation of the operation of the Wiener filter. It can be rewritten by introducing the frequency dependent [signal-to-noise ratio](https://en.wikipedia.org/wiki/Signal-to-noise_ratio) $\text{SNR}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{\Phi_{ss}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}{\Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}$ between the original signal and the noise as

\begin{equation}
H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{1}{G(\mathrm{e}^{\,\mathrm{j}\,\Omega})} \cdot \left(   \frac{|G(\mathrm{e}^{\,\mathrm{j}\,\Omega})|^2}{|G(\mathrm{e}^{\,\mathrm{j}\,\Omega})|^2 + \frac{1}{\text{SNR}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}} \right)
\end{equation}

Now two special cases are discussed:

1. If there is no additive noise $\Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = 0$, the bracketed expression is equal to 1. Hence, the Wiener filter is simply given as the inverse system to the distorting system $$H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{1}{G(\mathrm{e}^{\,\mathrm{j}\,\Omega})}$$

2. If the distorting system is just a pass-through $G(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = 1$, the Wiener filter is given as

    $$H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{\text{SNR}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}{\text{SNR}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) + 1}=\frac{\Phi_{ss}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}{\Phi_{ss}(\mathrm{e}^{\,\mathrm{j}\,\Omega})+\Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}$$
    Two extreme cases are considered
    * for $\Phi_{ss}(\mathrm{e}^{\,\mathrm{j}\,\Omega})\gg \Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ (high $\text{SNR}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$) at a given frequency $\Omega$ the transfer function approaches 1
    * for $\Phi_{ss}(\mathrm{e}^{\,\mathrm{j}\,\Omega})\ll \Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ (low $\text{SNR}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$) at a given frequency $\Omega$ the transfer function approaches 0

The discussed cases explain the operation of the Wiener filter for specific situations, the general operation is a combination of these.

**Example - Denoising of a deterministic signal with the Wiener deconvolution filter**

The preceding example of the general Wiener filter will now be reevaluated with the Wiener deconvolution filter.

```python
N = 8129  # number of samples
M = 256  # length of Wiener filter
Om0 = 0.1*np.pi  # frequency of original signal
N0 = .1  # PSD of additive white noise

# generate original signal
s = np.cos(Om0 * np.arange(N))
# generate observed signal
g = 1/20*np.asarray([1, 2, 3, 4, 5, 4, 3, 2, 1])
np.random.seed(1)
n = np.random.normal(size=N, scale=np.sqrt(N0))
x = np.convolve(s, g, mode='same') + n
# estimate PSD
f, Pss = sig.csd(s, s, nperseg=M)
f, Pnn = sig.csd(n, n, nperseg=M)
# compute Wiener filter
G = np.fft.rfft(g, M)
H = 1/G * (np.abs(G)**2 / (np.abs(G)**2 + N0/Pss))
H = H * np.exp(-1j*2*np.pi/len(H)*np.arange(len(H))*(len(H)//2-8))  # shift for causal filter
h = np.fft.irfft(H)
# apply Wiener filter to observation
y = np.convolve(x, h, mode='same')

# plot (cross) PSDs
Om = np.linspace(0, np.pi, num=len(H))

plt.figure(figsize=(10, 4))
plt.subplot(121)
plt.plot(Om, 20*np.log10(np.abs(.5*Pss)), label=r'$| \Phi_{ss}(e^{j \Omega}) |$ in dB')
plt.plot(Om, 20*np.log10(np.abs(.5*Pnn)), label=r'$| \Phi_{nn}(e^{j \Omega}) |$ in dB')
plt.title('PSDs')
plt.xlabel(r'$\Omega$')
plt.legend()
plt.axis([0, np.pi, -60, 40])
plt.grid()

# plot transfer function of Wiener filter
plt.subplot(122)
plt.plot(Om, 20*np.log10(np.abs(H)))
plt.title('Transfer function of Wiener filter')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$| H(e^{j \Omega}) |$ in dB')
plt.axis([0, np.pi, -150, 10])
plt.grid()
plt.tight_layout()

# plot signals
idx = np.arange(500, 600)
plt.figure(figsize=(10, 4))
plt.plot(idx, x[idx], label=r'observed signal $x[k]$')
plt.plot(idx, s[idx], label=r'original signal $s[k]$')
plt.plot(idx, y[idx], label=r'estimated signal $y[k]$')
plt.title('Signals')
plt.xlabel(r'$k$')
plt.axis([idx[0], idx[-1], -1.5, 1.5])
plt.legend()
plt.grid()
```

**Exercise**

* Compare the transfer function of the Wiener deconvolution filter to the transfer function of the general Wiener filter from the preceding example.
* What is different compared to the general Wiener filter? Why?

Solution: The transfer function $H(e^{j \Omega})$ of the Wiener deconvolution filter is much smoother than the transfer function of the general Wiener filter from the preceding example. This is due to the knowledge of the transfer function $G(e^{j \Omega})$ of the disturbing system. The general Wiener filter inherently estimates this from the noisy observations.


# Spring-mass problem: tracking based on range and range rate

Created on 01 January 2018

Example 4.8.2 Spring-mass problem
pg 216 in  Statistical Orbit Determination, Tapley, Born, Schutz.
pg 271 problem 15

@author: Ashiv Dhondea

```python
# Import generic libraries
import numpy as np
import math
```

```python
# Importing what's needed for nice plots.
import matplotlib.pyplot as plt
from matplotlib import rc
rc('font', **{'family': 'serif', 'serif': ['Helvetica']})
rc('text', usetex=True)
params = {'text.latex.preamble' : [r'\usepackage{amsmath}', r'\usepackage{amssymb}']}
plt.rcParams.update(params)
from mpl_toolkits.axes_grid.anchored_artists import AnchoredText
```

Object of mass $m$ is connected to two fixed walls by springs with spring constants $k_1$ and $k_2$. The sensor used to measure the range $\rho$ and range-rate $\dot{\rho}$ is located at a point $P$ which is at a height of $h$ on the wall to which spring $k_1$ is attached.

The initial position and velocity of the object are assumed to be $x_0 = 3.0~\mathrm{m}$ and $v_0 = 0.0~\mathrm{m/s}$.

The EOM (Equation of Motion) of the body is given by
\begin{equation}
\ddot{x} = - (k_1 + k_2)(x-\bar{x})/m = -\omega^2 (x-\bar{x})
\end{equation}
where $x$ is the position of the object, $\bar{x}$ is its static equilibrium position and $\omega$ is the angular frequency. $\bar{x}$ is to 0.

```python
# Parameters for the given problem
k1 = 2.5;
k2 = 3.7;
m = 1.5;
h = 5.4;
x0 = 3.0;
v0 = 0.0;

omega2 = (k1+k2)/m;
omega = math.sqrt(omega2);
```

From the EOM, it is inferred that an appropriate choice of state vector is
\begin{equation}
\mathbf{X} =
\begin{bmatrix}
x \\
v
\end{bmatrix}
\end{equation}

The relationship between position $x$ and velocity $v$ easily leads to a differential equation
\begin{equation}
\frac{dx}{dt} = v
\end{equation}
while the relationship between velocity and acceleration leads to another DE.
\begin{equation}
\frac{dv}{dt} = -\omega^2 x
\end{equation}

These two DEs lead to:
\begin{equation}
\frac{d}{dt}\begin{bmatrix}
x \\
v
\end{bmatrix}
=
\begin{bmatrix}
v \\
-\omega^2 x
\end{bmatrix}
\end{equation}

The state space formulation is thus
\begin{equation}
\dot{\mathbf{X}} =
\begin{bmatrix}
0 & 1 \\
-\omega^2 & 0
\end{bmatrix}
\mathbf{X} = \mathbf{A} \mathbf{X}
\end{equation}

Function fn_A implements $\mathbf{A}$.

An analysis of the state space formulation leads to the state transition matrix $\Phi$ given by
\begin{equation}
\Phi =
\begin{bmatrix}
\cos (\omega t) & \frac{1}{\omega}\sin (\omega t) \\
-\omega \sin (\omega t) & \cos (\omega t)
\end{bmatrix}
\end{equation}

Check: the determinant of this STM is indeed 1.

Function fn_STM implements the STM.

The sensor measures the range and range-rate to the object.
The measurement vector $y$ is related to the state vector $x$ by the measurement function $g$.
\begin{equation}
y = \begin{bmatrix}
\rho \\
\dot{\rho}
\end{bmatrix}
=
\begin{bmatrix}
\sqrt{x^2 + h^2}\\
\frac{xv}/{\rho}
\end{bmatrix}
\end{equation}

Functin fn_G implements the measurement function $g$.

Both the range and range-rate are nonlinear functions of the system state. Therefore, the Jacobian matrix of the measurement function needs to be found for inclusion in the nonlinear tracking filter.
\begin{equation}
\tilde{\mathbf{H}}(\mathbf{X}) =
\begin{bmatrix}
\frac{x}{\rho} & 0 \\
\big( \frac{v}{\rho} -\frac{x^2 v}{\rho^3} \big) & \frac{x}{\rho}
\end{bmatrix}
\end{equation}

This is implemented in function fn_Htilde.

```python
# Define the functions given by the stated equations
def fn_A(x_state,t,omega):
    # eqn 4.8.3
    A = np.array([[0.,1.],[-omega**2,0.]],dtype=np.float64);
    return A

def fn_G(x_state,h):
    # eqn 4.8.4
    rho = math.sqrt(x_state[0]**2+h**2);
    G = np.array([rho,x_state[0]*x_state[1]/rho],dtype=np.float64);
    return G

def fn_Htilde(x_state,h):
    # Jacobian matrix of G
    dx = np.shape(x_state)[0];
    x = x_state[0];v=x_state[1];
    rhovel = fn_G(x_state,h);
    dy=np.shape(rhovel)[0];
    rho = rhovel[0];#rhodot=rhovel[1];
    Htilde=np.zeros([dy,dx],dtype=np.float64);
    Htilde[0,0]=x/rho;
    Htilde[1,0]= v/rho - x**2*v/rho**3;
    Htilde[1,1]=x/rho;
    return Htilde

def fn_STM(t,omega):
    stm = np.zeros([2,2],dtype=np.float64);
    stm[0,0]=math.cos(omega*t);
    stm[0,1]=(1/omega)*math.sin(omega*t);
    stm[1,0]=-omega*math.sin(omega*t);
    stm[1,1]=math.cos(omega*t);
    return stm
```

```python
# Define the time step and time vector
delta_t = 1.0; # [s]
timevec = np.arange(0.,10.+delta_t,delta_t,dtype=np.float64);

# Dimensionality of the measurement vector
dy = 2; # 2 quantities measured
y_meas = np.zeros([dy,len(timevec)],dtype=np.float64); # array of measurements
# Load measurements
import pandas as pd

dframe = pd.read_excel("sod_book_pg218_table_4_8_1_data.xlsx",sheetname="Sheet1")
dframe = dframe.reset_index()
data_time = dframe['Time'][0:len(timevec)+1]
data_range = dframe['Range'][0:len(timevec)+1]
data_range_rate = dframe['Range rate'][0:len(timevec)+1]

y_meas[0,:] = data_range;
y_meas[1,:] = data_range_rate;
```

```python
# a priori values
x_nom_ini = np.array([4.,0.2],dtype=np.float64); # a priori state vector
P_ini = np.diag(np.array([1000.,100.],dtype=np.float64)); # a priori covariance matrix
dx = np.shape(x_nom_ini)[0]; # dimensionality of the state vector
delta_x_ini = np.zeros([dx],dtype=np.float64); # a priori perturbation vector

```

```python
def fn_BatchProcessor(delta_x,xnom,Pnom,timevec,y_meas,R_meas,omega,h):
    dy = np.shape(R_meas)[0];
    # Total observation matrix
    TotalObsvMat = np.linalg.inv(Pnom)#np.zeros([dx,dx],dtype=np.float64);
    # total observation vector
    TotalObsvVec = np.dot(TotalObsvMat,delta_x)#np.zeros([dx],dtype=np.float64);
    # simulated perturbation vector
    delta_Y = np.zeros([dy],dtype=np.float64);
    Rinv = np.linalg.inv(R_meas);

    for index in range(len(timevec)):
        stm_nom = fn_STM(timevec[index],omega)
        x_state_nom = np.dot(stm_nom,xnom);
        H = fn_Htilde(x_state_nom,h)
        Hi = np.dot(H,stm_nom);
        HiT = np.transpose(Hi);
        Ynom = fn_G(x_state_nom,h);

        delta_Y = np.subtract(y_meas[:,index],Ynom);
        TotalObsvMat = TotalObsvMat + np.dot(HiT,np.dot(Rinv,Hi));
        TotalObsvVec = TotalObsvVec + np.dot(HiT,np.dot(Rinv,delta_Y));

    S_hat = np.linalg.inv(TotalObsvMat);
    delta_x = np.dot(S_hat,TotalObsvVec);
    x0_hat = xnom + delta_x;
    return x0_hat,delta_x,S_hat
```

```python
sd_rho = 1.;
sd_rhodot = 1.;
R_meas = np.diag(np.array([sd_rho**2,sd_rhodot**2],dtype=np.float64));
```

```python
num_iter = 4;
x0_hat = np.zeros([dx,num_iter],dtype=np.float64);
S_hat = np.zeros([dx,dx,num_iter],dtype=np.float64);
obsv_error = np.zeros([dy,num_iter],dtype=np.float64);

xnom_in = x_nom_ini;
Pnom_in = P_ini;
delta_x_in = delta_x_ini;
for i_iter in range (0,num_iter):
    x0_hat[:,i_iter],delta_x,S_hat[:,:,i_iter] = fn_BatchProcessor(delta_x_in,xnom_in,Pnom_in,timevec,y_meas,R_meas,omega,h);
    xnom_in = x0_hat[:,i_iter];
    delta_x_in = delta_x_in - delta_x;

print x0_hat[:,num_iter-1]
print S_hat[:,:,num_iter-1]

sigma_x0 = math.sqrt(S_hat[0,0,num_iter-1]);
sigma_v0 = math.sqrt(S_hat[1,1,num_iter-1]);
sigma_x0v0 = math.sqrt(S_hat[0,1,num_iter-1]);
print sigma_x0
print sigma_v0
print sigma_x0v0
print math.sqrt(S_hat[1,0,num_iter-1]);
```

    [  3.00019485e+00   1.18181268e-03]
    [[ 0.16934803  0.01277539]
     [ 0.01277539  0.58448166]]
    0.411519179131
    0.764514003907
    0.113028287431
    0.113028287431

See page 219 in SOD book:
The state estimate x0_hat[:,num_iter-1] is identical to the estimate in the book.

The standard deviations are identical to those in the book. The correlation coefficient is incorrect.

```python

```


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
np.random.seed(404)
```

## Basic MCMC sampling

This notebook shows the following MCMC sampling methods for a multivariate Gaussian distribution

1. Random-Walk-Metropolis-Hastings (RWMH)
2. Gibbs sampler
3. Metropolis adjusted Langevin algorithm

## Data

We will use 500 samples of a multivaraite Gaussian distribution with the following mean vector

\begin{equation}
\mu = \begin{pmatrix} 1 \\ 2\\ 3 \end{pmatrix}
\end{equation}

and covariance matrix

\begin{equation}
\Sigma = \begin{pmatrix} 1 & -0.7 & -0.7 \\ -0.7 & 1.5 & 0.7 \\ -0.7 & 0.7 & 1 \end{pmatrix}
\end{equation}

```python
#covariance matrix
Σ = np.array([[1,-0.7,-0.7],[-0.7,1.5,0.7],[-0.7,0.7,1]])
print("Eigenvlaues of Σ : {}".format(np.linalg.eigvals(Σ)))
#mean vector
μ = np.array([1,2,3])
#Generaate 500 samples
Y = pd.DataFrame(np.random.multivariate_normal(μ,Σ,size=500),columns=['x_1','x_2','x_3'])
```

    Eigenvlaues of Σ : [2.59498744 0.3        0.60501256]

## Summary of generated samples

```python
Y.describe().round(2)
```

      x_1
      x_2
      x_3

      count
      500.00
      500.00
      500.00

      mean
      0.98
      2.01
      3.01

      std
      1.04
      1.24
      1.07

      min
      -2.64
      -1.94
      -0.09

      25%
      0.20
      1.19
      2.30

      50%
      1.01
      1.92
      3.06

      75%
      1.73
      2.86
      3.76

      max
      4.10
      5.76
      6.64

## Define density

\begin{equation}
        f(y|\mu,\Sigma) = (2\pi)^{-\frac{3}{2}}|\Sigma|^{-\frac{1}{2}} \exp{\left(-\frac{1}{2}(y-\mu)^{T}\Sigma^{-1}(y-\mu)\right)}
\end{equation}

```python
def mvn_dens(x,Σ,μ):
    return 1/np.sqrt((2*np.pi)**3*np.linalg.det(Σ))*np.exp(-0.5*(x-μ).T@np.linalg.inv(Σ)@(x-μ))
```

## 1) RWMH algorithm

```python
#initial guess
init = np.array([0.5,1.5,2.5])
n= 500
MCMC = [init]
s = init
accepted = 0
for i in range(n):
    s = MCMC[-1]
    #generate proposal sample
    y = np.random.multivariate_normal(s,1.0*np.eye(3))
    #compute acceptance rate
    α = mvn_dens(y,Σ,μ)/mvn_dens(s,Σ,μ)
    #accepte or reject
    if α > np.random.uniform(0,1):
        #if accepted append proposal sample to list of samples
        MCMC.append(y)
        accepted +=1
    else:
        #if rejected append current state to list of samples
        MCMC.append(s)
```

```python
samples_hm = pd.DataFrame(np.asarray(MCMC),columns=['x_1','x_2','x_3'])
```

## Helper function to plot MCMC samples and compare it to generated data

```python
def plot(samples):
    fig = plt.figure(figsize=(10,10))
    ax1=plt.subplot2grid( (3,3), [0,0], 1, 1 )
    sns.histplot(data=Y,x="x_1",ax=ax1,color='black',alpha=0.4,label='True values')
    sns.histplot(data=samples,x="x_1",color='royalblue',ax=ax1,alpha=0.6,label='Sampled values')
    ax1.set_xlabel('')
    #ax1.legend()
    ax2=plt.subplot2grid( (3,3), [1,0], 1, 1 )
    sns.kdeplot(data=Y,x='x_1',y='x_2',ax=ax2,color='black',alpha=0.6)
    sns.scatterplot(data=samples,x='x_1',y='x_2',ax=ax2,color='royalblue',linewidth=0.25)
    ax2.set_xlabel('')
    #plt.plot( x2, y2 )
    #plt.subplot2grid( (3,3), [1,0], 1, 1 )
    ax3=plt.subplot2grid( (3,3), [1,1], 1, 1 )
    sns.histplot(data=Y,x="x_2",ax=ax3,color='black',alpha=0.4)
    sns.histplot(data=samples,x="x_2",color='royalblue',ax=ax3,alpha=0.6)
    ax3.set_ylabel('')
    ax3.set_xlabel('')
    ax4=plt.subplot2grid( (3,3), [2,0], 1, 1 )
    sns.kdeplot(data=Y,x='x_1',y='x_3',ax=ax4,color='black',alpha=0.6)
    sns.scatterplot(data=samples,x='x_1',y='x_3',ax=ax4,color='royalblue',linewidth=0.25)
    ax5=plt.subplot2grid( (3,3), [2,1], 1, 1 )
    sns.kdeplot(data=Y,x='x_2',y='x_3',ax=ax5,color='black',alpha=0.6)
    sns.scatterplot(data=samples,x='x_2',y='x_3',ax=ax5,color='royalblue',linewidth=0.25)
    ax5.set_ylabel('')
    ax6=plt.subplot2grid( (3,3), [2,2], 1, 1 )
    sns.histplot(data=Y,x="x_3",ax=ax6,color='black',alpha=0.4)
    sns.histplot(data=samples,x="x_3",color='royalblue',ax=ax6,alpha=0.6,linewidth=1)
    ax6.set_ylabel('')
    handles, labels = ax1.get_legend_handles_labels()
    fig.legend(handles, labels,loc='center right',bbox_to_anchor=[0.82,0.59]);

    return fig
```

## Sampled values with RWMH vs generated data

```python
f=plot(samples_hm)
```

# 2) Gibbs sampler

## Define conditional distributions for Gibbs sampler

```python
def coonditional(x,mu,Sigma,ind):
    if ind==2:
        x=np.array([x[1],x[0],x[2]])
        μ = np.array([mu[1],mu[0],mu[2]])
        Σ = np.array([[Sigma[1,1],Sigma[1,0],Sigma[1,2]],[Sigma[0,1],Sigma[0,0],Sigma[0,2]],[Sigma[2,1],Sigma[2,0],Sigma[2,2]]])
    elif ind==3:
        x=np.array([x[2],x[0],x[1]])
        μ = np.array([mu[2],mu[0],mu[1]])
        Σ = np.array([[Sigma[2,2],Sigma[2,0],Sigma[2,1]],[Sigma[0,2],Sigma[0,0],Sigma[0,1]],[Sigma[1,2],Sigma[1,0],Sigma[1,1]]])
    else:
        μ = mu
        Σ = Sigma
    new_mean=μ[0]+Σ[0,1:]@np.linalg.inv(Σ[1:,1:])@(x[1:]-μ[1:])
    cov = Σ[0,0]-Σ[0,1:]@np.linalg.inv(Σ[1:,1:])@Σ[1:,0]
    return np.random.normal(new_mean,cov)
```

```python
init = np.array([0.5,1.5,2.5])
n= 500
MCMC = [init]
s = init
for i in range(n):
    s = MCMC[-1]
    a = coonditional(s,μ,Σ,1)
    s[0]=a
    b = coonditional(s,μ,Σ,2)
    s[1]=b
    c = coonditional(s,μ,Σ,3)
    s[2]=c
    MCMC.append(np.array([a,b,c]))
```

```python
samples_Gibbs = pd.DataFrame(np.asarray(MCMC),columns=['x_1','x_2','x_3'])
```

## Sampled values with Gibbs sampler vs generated data

```python
f_2=plot(samples_Gibbs)
```

## 3) Metropolis adjusted Langevin algorithm (MALA)

### Define gradient of log-likelihood wrt. to x

```python
def d_log(x,μ,Σ):
    return -np.linalg.inv(Σ)@(x-μ)
```

### Genrate proposal sample $y$ according to

\begin{equation}
y=s+\frac{h}{2}\nabla \log{\pi(s)} + hz.
\end{equation}

### where $s$ is the current state and $z \sim N(0,I_{3})$.

### The acceptance rate is

\begin{equation}
\alpha = \min\left[1, \frac{f(y|\mu,\Sigma)q(s|y)}{f(s|\mu,\Sigma)q(y|s)}\right]
\end{equation}

### with

\begin{equation}
q(s|y) \propto \exp{\left( -\frac{1}{4h}||s-y-h\nabla \log{f(y|\mu,\Sigma)}||_{2}^{2}\right)}
\end{equation}

```python
h = 0.4
init = np.array([0.5,1.5,2.5])
n= 1000
MCMC = [init]
s = init
accepted = 0
for i in range(n):
    s = MCMC[-1]
    #generate proposal sample y
    y = s + h/2*d_log(s,μ,Σ) + (2*h)**0.5*np.random.multivariate_normal(np.zeros(3),np.eye(3))
    #compute q(y|s)
    rxy = np.exp(-np.linalg.norm(y-s-h*d_log(s,μ,Σ))**2/(4*h))
    #compute q(s|y)
    ryx = np.exp(-np.linalg.norm(s-y-h*d_log(y,μ,Σ))**2/(4*h))
    #compute acceptance rate
    α = (mvn_dens(y,Σ,μ)*ryx)/(mvn_dens(s,Σ,μ)*rxy)
    #accepting/rejecting
    if α > np.random.uniform(0,1):
        MCMC.append(y)
        accepted +=1
    else:
        MCMC.append(s)
print(accepted/1000)
```

    0.659

```python
samples_LHM = pd.DataFrame(np.asarray(MCMC[500:]),columns=['x_1','x_2','x_3'])
```

## Sampled values with MALA (stepsize 0.4) vs generated data

```python
f_3 = plot(samples_LHM)
```


## 20 Sept 2019

RULES

Date: Level 2 heading ##

Example Heading: Level 3 heading ###

Method Heading: Level 4 heading ####

### References

1. [Taan_S_Elali]_Discrete_Systems_and_Digital_Signal(z-lib.org)

### Imports

```python
import numpy as np
from sympy import oo
import math
import sympy as sp
import matplotlib.pyplot as plt
import matplotlib as mpl
from mpl_toolkits.mplot3d import Axes3D
from IPython.display import display
from IPython.display import display_latex
from sympy import latex
import math
from scipy import signal
```

### Setup

```python
sp.init_printing(use_latex = True)

z, f, i = sp.symbols('z f i')
x, k = sp.symbols('x k')
theta = sp.Symbol('theta')
```

```python
# Usage: display_equation('u_x', x)
def display_equation(idx, symObj):
    if(isinstance(idx, str)):
        eqn = '\\[' + idx + ' = ' + latex(symObj) + '\\]'
        display_latex(eqn, raw=True)
    else:
        eqn = '\\[' + latex(idx) + ' = ' + latex(symObj) + '\\]'
        display_latex(eqn, raw=True)
    return
```

```python
def sum_of_GP(a, r):
    return sp.simplify(a/(1-r))
```

```python
def display_ztrans(f, k, limits = (-4, 4)):
    F = sp.summation(f/z**k, ( k, -oo, oo ))
    display_equation('f(k)', f)
    display_equation('F(k)_{\infty}', F)

    F = sp.summation(f/z**k, (k, limits[0], limits[1]))
    display_equation('F(k)_{'+ str(limits[0]) + ',' + str(limits[1]) + '}', F)
    return
```

```python
# Credit: https://www.dsprelated.com/showcode/244.php
import numpy as np
import matplotlib.pyplot as plt
from  matplotlib import patches
from matplotlib.figure import Figure
from matplotlib import rcParams

def zplane(b,a,filename=None):
    """Plot the complex z-plane given a transfer function.
    """

    # get a figure/plot
    ax = plt.subplot(111)

    # create the unit circle
    uc = patches.Circle((0,0), radius=1, fill=False,
                        color='black', ls='dashed')
    ax.add_patch(uc)

    # The coefficients are less than 1, normalize the coeficients
    if np.max(b) > 1:
        kn = np.max(b)
        b = b/float(kn)
    else:
        kn = 1

    if np.max(a) > 1:
        kd = np.max(a)
        a = a/float(kd)
    else:
        kd = 1

    # Get the poles and zeros
    p = np.roots(a)
    z = np.roots(b)
    k = kn/float(kd)

    # Plot the zeros and set marker properties
    t1 = plt.plot(z.real, z.imag, 'go', ms=10)
    plt.setp( t1, markersize=10.0, markeredgewidth=1.0,
              markeredgecolor='k', markerfacecolor='g')

    # Plot the poles and set marker properties
    t2 = plt.plot(p.real, p.imag, 'rx', ms=10)
    plt.setp( t2, markersize=12.0, markeredgewidth=3.0,
              markeredgecolor='b', markerfacecolor='b')

    ax.spines['left'].set_position('center')
    ax.spines['bottom'].set_position('center')
    ax.spines['right'].set_visible(False)
    ax.spines['top'].set_visible(False)

    # set the ticks
    r = 1.5; plt.axis('scaled'); plt.axis([-r, r, -r, r])
    ticks = [-1, -.5, .5, 1]; plt.xticks(ticks); plt.yticks(ticks)

    if filename is None:
        plt.show()
    else:
        plt.savefig(filename)

    return z, p, k
```

### [1] Example 3.8

```python
n = sp.Symbol('n')
f1 = 0.5 ** n
f = f1 * sp.Heaviside(n)
# or
#f = sp.Piecewise((0, n < 0), (f1, n >= 0))
n = 20
display_equation('f(n)', f)
q = sum_of_GP(1, 0.5 * sp.exp(-theta))
display_equation('X(\\theta)', q)

t = np.arange(0, n, 1)
f = np.power(0.5, t)
plt.stem(t, f)
plt.xlabel('t')
plt.ylabel('Magnitude')

```

```python

```


# 2021-09-15 Advection and stability

## Last time

* Variable coefficients
* Conservative/divergence form vs non-divergence forms
* Verification with discontinuities

## Today
* Advection and boundary layers
* The (cell) Péclet number and oscillations

```julia
using Plots
using LinearAlgebra

function vander(x, k=nothing)
    if k === nothing
        k = length(x)
    end
    V = ones(length(x), k)
    for j = 2:k
        V[:, j] = V[:, j-1] .* x
    end
    V
end

function fdstencil(source, target, k)
    "kth derivative stencil from source to target"
    x = source .- target
    V = vander(x)
    rhs = zero(x)'
    rhs[k+1] = factorial(k)
    rhs / V
end

function poisson(x, spoints, forcing; left=(0, zero), right=(0, zero))
    n = length(x)
    L = zeros(n, n)
    rhs = forcing.(x)
    for i in 2:n-1
        jleft = min(max(1, i-spoints÷2), n-spoints+1)
        js = jleft : jleft + spoints - 1
        L[i, js] = -fdstencil(x[js], x[i], 2)
    end
    L[1,1:spoints] = fdstencil(x[1:spoints], x[1], left[1])
    L[n,n-spoints+1:n] = fdstencil(x[n-spoints+1:n], x[n], right[1])
    rhs[1] = left[2](x[1])
    rhs[n] = right[2](x[n])
    L, rhs
end

CosRange(a, b, n) = (a + b)/2 .+ (b - a)/2 * cos.(LinRange(-pi, 0, n))

function vander_chebyshev(x, n=nothing)
    if isnothing(n)
        n = length(x) # Square by default
    end
    m = length(x)
    T = ones(m, n)
    if n > 1
        T[:, 2] = x
    end
    for k in 3:n
        T[:, k] = 2 * x .* T[:,k-1] - T[:, k-2]
    end
    T
end

function chebdiff(x, n=nothing)
    T = vander_chebyshev(x, n)
    m, n = size(T)
    dT = zero(T)
    dT[:,2:3] = [one.(x) 4*x]
    for j in 3:n-1
        dT[:,j+1] = j * (2 * T[:,j] + dT[:,j-1] / (j-2))
    end
    ddT = zero(T)
    ddT[:,3] .= 4
    for j in 3:n-1
        ddT[:,j+1] = j * (2 * dT[:,j] + ddT[:,j-1] / (j-2))
    end
    T, dT, ddT
end

function poisson_cheb(n, rhsfunc, leftbc=(0, zero), rightbc=(0, zero))
    x = CosRange(-1, 1, n)
    T, dT, ddT = chebdiff(x)
    dT /= T
    ddT /= T
    T /= T
    L = -ddT
    rhs = rhsfunc.(x)
    for (index, deriv, func) in
            [(1, leftbc...), (n, rightbc...)]
        L[index,:] = (T, dT)[deriv+1][index,:]
        rhs[index] = func(x[index])
    end
    x, L, rhs
end
```

    ┌ Info: Precompiling Plots [91a5bcdd-55d7-5caf-9e0b-520d859cae80]
    └ @ Base loading.jl:1342

    poisson_cheb (generic function with 3 methods)

# Variable coefficients

* Heat conduction: steel, brick, wood, foam
* Electrical conductivity: copper, rubber, air
* Elasticity: steel, rubber, concrete, rock
* Linearization of nonlinear materials
  * ketchup, glacier ice, rocks (mantle/lithosphere)

```julia
kappa_step(x) = .1 + .9 * (x > 0)
kappa_tanh(x, epsilon=.1) = .55 + .45 * tanh(x / epsilon)
plot([kappa_step, kappa_tanh], xlims=(-1, 1), ylims=(0, 1), label="κ")
```

    ┌ Info: Precompiling GR_jll [d2c73de3-f751-5644-a686-071e5b155ba9]
    └ @ Base loading.jl:1342

# A naive finite difference solver

| Conservative (divergence) form | Non-divergence form |
| ------------------------------ | ------------------- |
| $-(\kappa u_x)_x = 0$           | $-\kappa u_{xx} - \kappa_x u_x = 0$ |

```julia
function poisson_nondivergence(x, spoints, kappa, forcing; leftbc=(0, zero), rightbc=(0, zero))
    n = length(x)
    L = zeros(n, n)
    rhs = forcing.(x)
    kappax = kappa.(x)
    for i in 2:n-1
        jleft = min(max(1, i-spoints÷2), n-spoints+1)
        js = jleft : jleft + spoints - 1
        kappa_x = fdstencil(x[js], x[i], 1) * kappax[js]
        L[i, js] = -fdstencil(x[js], x[i], 2) .* kappax[i] - fdstencil(x[js], x[i], 1) * kappa_x
    end
    L[1,1:spoints] = fdstencil(x[1:spoints], x[1], leftbc[1])
    if leftbc[1] == 1
        L[1, :] *= kappax[1]
    end
    L[n,n-spoints+1:n] = fdstencil(x[n-spoints+1:n], x[n], rightbc[1])
    if rightbc[1] == 1
        L[n, :] *= kappax[n]
    end
    rhs[1] = leftbc[2](x[1])
    rhs[n] = rightbc[2](x[n])
    L, rhs
end
```

    poisson_nondivergence (generic function with 1 method)

# Discretizing in conservative form

| Conservative (divergence) form | Non-divergence form |
| ------------------------------ | ------------------- |
| $-(\kappa u_x)_x = 0$           | $-\kappa u_{xx} - \kappa_x u_x = 0$ |

```julia
function poisson_conservative(n, kappa, forcing; leftbc=(0, zero), rightbc=(0, zero))
    x = LinRange(-1, 1, n)
    xstag = (x[1:end-1] + x[2:end]) / 2
    L = zeros(n, n)
    rhs = forcing.(x)
    kappa_stag = kappa.(xstag)
    for i in 2:n-1
        flux_L = kappa_stag[i-1] * fdstencil(x[i-1:i], xstag[i-1], 1)
        flux_R = kappa_stag[i] * fdstencil(x[i:i+1], xstag[i], 1)
        js = i-1:i+1
        weights = -fdstencil(xstag[i-1:i], x[i], 1)
        L[i, i-1:i+1] = weights[1] * [flux_L..., 0] + weights[2] * [0, flux_R...]
    end
    if leftbc[1] == 0
        L[1, 1] = 1
        rhs[1] = leftbc[2](x[1])
        rhs[2:end] -= L[2:end, 1] * rhs[1]
        L[2:end, 1] .= 0
    end
    if rightbc[1] == 0
        L[end,end] = 1
        rhs[end] = rightbc[2](x[end])
        rhs[1:end-1] -= L[1:end-1,end] * rhs[end]
        L[1:end-1,end] .= 0
    end
    x, L, rhs
end
```

    poisson_conservative (generic function with 1 method)

# Compare conservative vs non-divergence forms

```julia
forcing = zero # one
kappa_tanh(x, epsilon=.01) = .55 + .45 * tanh(x / epsilon)
x, L, rhs = poisson_conservative(20, kappa_tanh,
    forcing, leftbc=(0, zero), rightbc=(0, one))
u = L \ rhs
plot(x, u, marker=:auto, legend=:bottomright)
```

```julia
x = LinRange(-1, 1, 20)
L, rhs = poisson_nondivergence(x, 3, kappa_tanh,
    forcing, leftbc=(0, zero), rightbc=(0, one))
u = L \ rhs
plot(x, u, marker=:auto, legend=:bottomright)
```

# Continuity of flux

```julia
forcing = zero
x, L, rhs = poisson_conservative(20, kappa_step,
    forcing, leftbc=(0, zero), rightbc=(0, one))
u = L \ rhs
plot(x, u, marker=:auto, legend=:bottomright)
```

```julia
xstag = (x[1:end-1] + x[2:end]) ./ 2
du = (u[1:end-1] - u[2:end]) ./ diff(x)
plot(xstag, [du .* kappa_step.(xstag)], marker=:auto, ylims=[-1, 1])
```

# Manufactured solutions with discontinuous coefficients

* We need to be able to evaluate derivatives of the flux $-\kappa u_x$.
* A physically-realizable solution would have continuous flux, but we we'd have to be making a physical solution to have that in verification.
* Idea: replace the discontinuous function with a continuous one with a rapid transition.

```julia
kappa_tanh(x, epsilon=.1) = .55 + .45 * tanh(x / epsilon)
d_kappa_tanh(x, epsilon=.1) = .45/epsilon * cosh(x/epsilon)^-2
plot([kappa_tanh])
```

# Solving with the smoothed step $\kappa$

```julia
kappa_tanh(x, epsilon=.01) = .55 + .45 * tanh(x / epsilon)
d_kappa_tanh(x, epsilon=.01) = .45/epsilon * cosh(x/epsilon)^-2
manufactured(x) = tanh(2*x)
d_manufactured(x) = 2*cosh(2*x).^-2
flux_manufactured_kappa_tanh(x) = kappa_tanh(x) * d_manufactured(x)
function forcing_manufactured_kappa_tanh(x)
    8 * tanh(2x) / cosh(2x)^2 * kappa_tanh(x) -
    d_kappa_tanh(x) * d_manufactured(x)
end
x, L, rhs = poisson_conservative(200, kappa_tanh,
    forcing_manufactured_kappa_tanh,
    leftbc=(0, manufactured), rightbc=(0, manufactured))
u = L \ rhs
plot(x, u, marker=:auto, legend=:bottomright, title="Error $(norm(u - manufactured.(x), Inf))")
plot!([manufactured flux_manufactured_kappa_tanh forcing_manufactured_kappa_tanh kappa_tanh])
```

# Convergence

```julia
function poisson_error(n, spoints=3)
    x, L, rhs = poisson_conservative(n, kappa_tanh,
        forcing_manufactured_kappa_tanh,
        leftbc=(0, manufactured), rightbc=(0, manufactured))
    u = L \ rhs
    norm(u - manufactured.(x), Inf)
end
ns = 2 .^ (3:10)
plot(ns, poisson_error.(ns, 3), marker=:auto, xscale=:log10, yscale=:log10)
plot!(n -> n^-2, label="\$1/n^2\$")
```

# Advection

Advection represents transport in a "wind" $w$. The time-dependent model is
$$ u_t + (w u)_x = 0 $$
for which $u(t,x) = u(0, x - tw)$.

```julia
u0(x) = @. tanh(2*x)
usolution(t,x; w=1) = @. u0(x - t*w)
x = LinRange(-1, 1, 50)
plot(x, [usolution(t, x) for t in 0:.2:2])
```

## What boundary conditions can be specified on the steady-state ($t\to\infty$) problem?

## What is the steady state solution?

# Advection-diffusion with Chebyshev
$$ -u_{xx} + (wu)_x = f $$

```julia
function advdiff_cheb(n, wind, rhsfunc, leftbc=(0, zero), rightbc=(0, zero))
    x = CosRange(-1, 1, n)
    T, dT, ddT = chebdiff(x)
    dT /= T
    ddT /= T
    T /= T
    L = -ddT + wind * dT
    rhs = rhsfunc.(x)
    for (index, deriv, func) in
            [(1, leftbc...), (n, rightbc...)]
        L[index,:] = (T, dT)[deriv+1][index,:]
        rhs[index] = func(x[index])
    end
    x, L, rhs
end
```

    advdiff_cheb (generic function with 3 methods)

```julia
x, L, rhs = advdiff_cheb(25, 100, one)
@show minimum(diff(x))
plot(x, L \ rhs, legend=:none, marker=:auto)
```

    minimum(diff(x)) = 0.008555138626189618

# Advection-diffusion with conservative FD

$$(- \kappa u_x + wu)_x = f$$

```julia
function advdiff_conservative(n, kappa, wind, forcing)
    x = LinRange(-1, 1, n)
    xstag = (x[1:end-1] + x[2:end]) / 2
    L = zeros(n, n)
    rhs = forcing.(x)
    kappa_stag = kappa.(xstag)
    for i in 2:n-1
        flux_L = -kappa_stag[i-1] * fdstencil(x[i-1:i], xstag[i-1], 1) + wind * [.5 .5]
        flux_R = -kappa_stag[i] * fdstencil(x[i:i+1], xstag[i], 1) + wind * [.5 .5]
        js = i-1:i+1
        weights = fdstencil(xstag[i-1:i], x[i], 1)
        L[i, i-1:i+1] = weights[1] * [flux_L..., 0] + weights[2] * [0, flux_R...]
    end
    L[1, 1] = 1
    rhs[1] = 0
    L[2:end, 1] .= 0
    L[end,end] = 1
    rhs[end] = 0
    L[1:end-1,end] .= 0
    x, L, rhs
end
```

    advdiff_conservative (generic function with 1 method)

# Experiments, The (cell) [Péclet number](https://en.wikipedia.org/wiki/P%C3%A9clet_number)

\begin{align}
(- \kappa u_x + wu)_x &= f & \mathrm{Pe}_h &= \frac{h \lvert w \rvert}{\kappa}
\end{align}

```julia
n = 30; h = 2/n
kappa = 1
wind = 10
x, L, rhs = advdiff_conservative(n, x -> kappa, wind, one)
@show minimum(diff(x))
plot(x, L \ rhs, marker=:auto, legend=:none, title="ratio $(wind*h/kappa)")
```

    minimum(diff(x)) = 0.06896551724137923

# Upwinded discretization

Idea: incoming **advective flux** should depend only on upwind value, outgoing should depend only on my value.

```julia
function advdiff_upwind(n, kappa, wind, forcing)
    x = LinRange(-1, 1, n)
    xstag = (x[1:end-1] + x[2:end]) / 2
    L = zeros(n, n)
    rhs = forcing.(x)
    kappa_stag = kappa.(xstag)
    for i in 2:n-1
        flux_L = -kappa_stag[i-1] * fdstencil(x[i-1:i], xstag[i-1], 1) +
            wind * (wind > 0 ? [1 0] : [0 1])
        flux_R = -kappa_stag[i] * fdstencil(x[i:i+1], xstag[i], 1) +
            wind * (wind > 0 ? [1 0] : [0 1])
        js = i-1:i+1
        weights = fdstencil(xstag[i-1:i], x[i], 1)
        L[i, i-1:i+1] = weights[1] * [flux_L..., 0] + weights[2] * [0, flux_R...]
    end
    L[1, 1] = 1
    rhs[1] = 0
    L[2:end, 1] .= 0
    L[end,end] = 1
    rhs[end] = 0
    L[1:end-1,end] .= 0
    x, L, rhs
end
```

    advdiff_upwind (generic function with 1 method)

# Try it for robustness

```julia
n = 30; h = 2/n
kappa = 1
wind = 10
x, L, rhs = advdiff_upwind(n, x -> kappa, wind, one)
@show minimum(diff(x))
plot(x, L \ rhs, marker=:auto, legend=:none, title="ratio $(wind*h/kappa)")
```

    minimum(diff(x)) = 0.06896551724137923


# Calculate historical Vapor Pressure Deficit (VPD) grids.

Do this for gridded ERA-Interim data using the estimated RH% field. VPD is a simple forula.

\begin{align}
VPD & = e_{sat} - e_{actual}
\end{align}

where:

\begin{align}
e_{actual}=\frac{e_{sat}*RH)}{100}
\end{align}

where the saturation water vopor pressure ($e_{sat}$) is expressed with the Teten's formula:
Teton's formula, eq. 7.5
https://www.ecmwf.int/sites/default/files/elibrary/2015/9211-part-iv-physical-processes.pdf

\begin{align}
e_{sat}(T) & = a_{1}exp(a_{3}\frac{T - T_{0}}{T-a_{4}})\\
\\
a_{1} & = 611.21Pa \\
a_{3} & = 17.502 \\
a_{4} & = 32.19 \\
T_{0} & = 273.16
\end{align}

```python
import numpy as np
from netCDF4 import Dataset
import os
import glob

def calculate_es(T):
    """Calculate saturation vapor pressure using Tetons formula"""
    a1 = 611.21 # Pa
    a3 = 17.502
    a4 = 32.19
    T0 = 273.16 # K
    e_sat = a1 * np.exp( a3 * (T-T0)/(T-a4) )

    return e_sat
```

```python
def get_nc(var, year, dataDir):
    loadFile = os.path.join(dataDir, var + "_" + str(year) + ".nc")
    nc = Dataset(loadFile)
    vals = nc.variables[var][:]
    t = nc.variables["time"][:]
    lon = nc.variables["longitude"][:]
    lat = nc.variables["latitude"][:]
    nc.close()
    return vals, t, lon, lat
```

```python
def write_VPD_nc(VALS, t, x, y, year, dataDir):

    outputFile = os.path.join(dataDir, "VPD_" + str(year) + ".nc")

    ncFile = Dataset(outputFile, 'w', format='NETCDF4')
    ncFile.description = 'VPD (saturation pressure relative to water, Tetons eq.)'
    ncFile.location = 'Global'
    ncFile.createDimension('time',  len(t) )
    ncFile.createDimension('latitude', len(y) )
    ncFile.createDimension('longitude', len(x) )

    VAR_ = ncFile.createVariable("VPD", 'f4',('time', 'latitude','longitude'))
    VAR_.long_name = "Vapor Pressure Deficit"
    VAR_.units = "Pa"

    # Create time variable
    time_ = ncFile.createVariable('time', 'i4', ('time',))
    time_.units = "hours since 1900-01-01 00:00:0.0"
    time_.calendar = "gregorian"

    # create lat variable
    latitude_ = ncFile.createVariable('latitude', 'f4', ('latitude',))
    latitude_.units = 'degrees_north'

    # create longitude variable
    longitude_ = ncFile.createVariable('longitude', 'f4', ('longitude',))
    longitude_.units = 'degrees_east'

    # Write the actual data to these dimensions
    VAR_[:]       = VALS
    time_[:]      = t
    latitude_[:]  = y
    longitude_[:] = x

    ncFile.close()
```

### Use above functions to create VPD file for era-interim.

```python
dataDir = os.path.join("..","..", "metSpreadData", "ERA-Interim", 'merged_time')
year = "1983-2017"

# Get temperature to calculate es
t2m,t,x,y = get_nc("t2m", year, dataDir)

# Get RH% to calculate e_actual
RH,t,x,y = get_nc("RH", year, dataDir)

# get saturation and actual vapor pressure
e_sat = calculate_es(t2m)
e_actual = e_sat * (RH / 100.)

# Calculate VPD using what is known as subtraction
VPD = e_sat - e_actual

write_VPD_nc(VPD, t, x, y, year, dataDir)
```

```python
import cdo as cdo
cdo = cdo.Cdo()

dataDir = os.path.join("..","..", "metSpreadData", "ERA-Interim", 'merged_time')
common_grid_txt = os.path.join("..","..", "metSpreadData", "ERA-Interim", "COMMON_GRID.txt")
f_in = os.path.join(dataDir, "VPD_1983-2017.nc")
f_out = os.path.join("..","..", "metSpreadData", "ERA-Interim", 'merged_t_COMMON_GRID', "VPD_1983-2017.nc")
cdo.remapbil(common_grid_txt, input=f_in, output=f_out, options="-b F64")
```

    '../../metSpreadData/ERA-Interim/merged_t_COMMON_GRID/VPD_1983-2017.nc'

Use cdo remapbil,COMMON_GRID.txt to create the needed file for comparing to CMIP5. Move that file to the metSpread directory.
- Done most recently for 1983-2017 VPD data via bash cdo on Nov 24 2018

# Do the same for CMIP5 output.

## NOTE:
This script creates VPD files for merged_common_grid variables only, so only run after Python/format_cmip5_data.py has been most recently executed.

## TODO:
- Update for new files and the fact that we have huss now and do longer need to estimate using RH%
- Be more careful in the VPD file creation such that the month spaned that the data represent time span is better preserved. The output and calculations are correct but there is something sloppy happenning with the time stamp when I quickly rewrite the netcdf file using write_VPD_nc which takes t.units and time.data from a tas file.
- Use available files, loop through them, vs. checking for a T and RH file for each attempted model.

```python
def get_CMIP_nc(var, loadFile):

    nc = Dataset(loadFile)
    vals = np.array(nc.variables[var])
    t = np.array(nc.variables["time"])
    t_units = nc.variables["time"].units
    lon = np.array(nc.variables["lon"])
    lat = np.array(nc.variables["lat"])
    nc.close()

    return vals, t, t_units, lon, lat
```

```python
def write_VPD_CMIP5_nc(VALS, model, rcp, t, t_units, x, y, dataDir):
    """
    Needs to take rcp files that have been merged with history files, so that these operations
    can be done only onces and not repeated at several points on the data pipeline.

        DataSource: ../r1i1p1_rcp_COMMON_GRID/

    """

    outputFile = os.path.join(dataDir, "VPD_Amon_"+model+"_"+rcp+"_r1i1p1_198301-210012.nc")

    ncFile = Dataset(outputFile, 'w', format='NETCDF4')
    ncFile.description = 'VPD (saturation pressure relative to water, Tetons eq.)'
    ncFile.location = 'Global'
    ncFile.createDimension('time',  len(t) )
    ncFile.createDimension('lat', len(y) )
    ncFile.createDimension('lon', len(x) )

    VAR_ = ncFile.createVariable("VPD", 'f4',('time', 'lat','lon'))
    VAR_.long_name = "Vapor Pressure Deficit"
    VAR_.units = "Pa"

    # Create time variable
    time_ = ncFile.createVariable('time', 'i4', ('time',))
    time_.units = t_units
    #time_.calendar = "gregorian"

    # create lat variable
    latitude_ = ncFile.createVariable('lat', 'f4', ('lat',))
    latitude_.units = 'degrees_north'
    latitude_.axis = "Y"

    # create longitude variable
    longitude_ = ncFile.createVariable('lon', 'f4', ('lon',))
    longitude_.units = 'degrees_east'
    longitude_.axis = "X"

    # Write the actual data to these dimensions
    VAR_[:]       = VALS
    time_[:]      = t
    latitude_[:]  = y
    longitude_[:] = x

    ncFile.close()
```

### Loop through the unique models based on the data that are actually available at the common_grid_level

```python
dataDir = os.path.join("..","..", "metSpreadData", "getCMIP5", 'r1i1p1_rcp_COMMON_GRID')

all_model_names = []
all_files = glob.glob(dataDir+"/tas_Amon*")
for f in all_files :
    all_model_names.append(f.split("/")[5].split("_")[2])
unique_model_names = np.unique(all_model_names)
```

```python
unique_model_names
```

    array(['ACCESS1-0', 'ACCESS1-3', 'CCSM4', 'CMCC-CESM', 'CMCC-CM',
           'CMCC-CMS', 'CNRM-CM5', 'CSIRO-Mk3-6-0', 'CanESM2', 'FGOALS-g2',
           'GFDL-CM3', 'GFDL-ESM2G', 'GFDL-ESM2M', 'GISS-E2-H',
           'GISS-E2-H-CC', 'GISS-E2-R', 'GISS-E2-R-CC', 'HadGEM2-AO',
           'HadGEM2-CC', 'HadGEM2-ES', 'IPSL-CM5A-LR', 'IPSL-CM5A-MR',
           'IPSL-CM5B-LR', 'MIROC-ESM', 'MIROC-ESM-CHEM', 'MIROC5',
           'MPI-ESM-LR', 'MPI-ESM-MR', 'MRI-CGCM3', 'MRI-ESM1', 'NorESM1-M',
           'NorESM1-ME', 'bcc-csm1-1-m', 'inmcm4'], dtype='<U14')

```python
dataDir = os.path.join("..","..", "metSpreadData", "getCMIP5", 'r1i1p1_rcp_COMMON_GRID')

print(dataDir)

# Need VPD for both rcps
rcps = ['rcp45', 'rcp85']

# list the files

for rcp in rcps :

    print("Working on: " + rcp)

    for model in unique_model_names :

        print("start----------------------------")
        print(model)

        # see if Temperature and RH file exist, both are required
        # TODO: replace hurs (RH) with use of estimated huss (e) and that will increase accuracy
        # TODO: and simpliciity though make it slightly different from how things are done for
        # TODO: calculating era-interim VPD.
        T_file = os.path.join(dataDir, "tas_Amon_"+model+"_"+rcp+"_r1i1p1_198301-210012.nc")
        RH_file = os.path.join(dataDir, "hurs_Amon_"+model+"_"+rcp+"_r1i1p1_198301-210012.nc")

        # If these files exist (they don't for all models and rcps), continue
        if os.path.isfile(T_file) & os.path.isfile(RH_file) :

            # Get the values to calculate e
            tas,t,t_units,x,y = get_CMIP_nc('tas', T_file)
            RH,t,_, x,y  = get_CMIP_nc('hurs', RH_file)

            # get saturation and actual vapor pressure
            e_sat = calculate_es(tas)
            e_actual = e_sat * (RH / 100.)

            # Calculate VPD using what is known as subtraction
            VPD = e_sat - e_actual

            write_VPD_CMIP5_nc(VPD, model, rcp, t, t_units, x, y,  dataDir)

            print("Writing VPD esetimate")

        else :

            print("*********************************")
            print("Missing a file: ")
            print("T_file present: " + str(os.path.isfile(T_file)))
            print("RH_file present: " + str(os.path.isfile(RH_file)))
            print("*********************************")

        print("------------------------------end")
        print(" ")
```

    ../../metSpreadData/getCMIP5/r1i1p1_rcp_COMMON_GRID
    Working on: rcp45
    start----------------------------
    ACCESS1-0
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    ACCESS1-3
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    CCSM4
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    CMCC-CESM
    *********************************
    Missing a file:
    T_file present: False
    RH_file present: False
    *********************************
    ------------------------------end

    start----------------------------
    CMCC-CM
    *********************************
    Missing a file:
    T_file present: True
    RH_file present: False
    *********************************
    ------------------------------end

    start----------------------------
    CMCC-CMS
    *********************************
    Missing a file:
    T_file present: True
    RH_file present: False
    *********************************
    ------------------------------end

    start----------------------------
    CNRM-CM5
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    CSIRO-Mk3-6-0
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    CanESM2
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    FGOALS-g2
    *********************************
    Missing a file:
    T_file present: True
    RH_file present: False
    *********************************
    ------------------------------end

    start----------------------------
    GFDL-CM3
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    GFDL-ESM2G
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    GFDL-ESM2M
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    GISS-E2-H
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    GISS-E2-H-CC
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    GISS-E2-R
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    GISS-E2-R-CC
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    HadGEM2-AO
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    HadGEM2-CC
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    HadGEM2-ES
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    IPSL-CM5A-LR
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    IPSL-CM5A-MR
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    IPSL-CM5B-LR
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    MIROC-ESM
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    MIROC-ESM-CHEM
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    MIROC5
    *********************************
    Missing a file:
    T_file present: True
    RH_file present: False
    *********************************
    ------------------------------end

    start----------------------------
    MPI-ESM-LR
    *********************************
    Missing a file:
    T_file present: True
    RH_file present: False
    *********************************
    ------------------------------end

    start----------------------------
    MPI-ESM-MR
    *********************************
    Missing a file:
    T_file present: True
    RH_file present: False
    *********************************
    ------------------------------end

    start----------------------------
    MRI-CGCM3
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    MRI-ESM1
    *********************************
    Missing a file:
    T_file present: False
    RH_file present: False
    *********************************
    ------------------------------end

    start----------------------------
    NorESM1-M
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    NorESM1-ME
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    bcc-csm1-1-m
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    inmcm4
    Writing VPD esetimate
    ------------------------------end

    Working on: rcp85
    start----------------------------
    ACCESS1-0
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    ACCESS1-3
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    CCSM4
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    CMCC-CESM
    *********************************
    Missing a file:
    T_file present: True
    RH_file present: False
    *********************************
    ------------------------------end

    start----------------------------
    CMCC-CM
    *********************************
    Missing a file:
    T_file present: True
    RH_file present: False
    *********************************
    ------------------------------end

    start----------------------------
    CMCC-CMS
    *********************************
    Missing a file:
    T_file present: True
    RH_file present: False
    *********************************
    ------------------------------end

    start----------------------------
    CNRM-CM5
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    CSIRO-Mk3-6-0
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    CanESM2
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    FGOALS-g2
    *********************************
    Missing a file:
    T_file present: True
    RH_file present: False
    *********************************
    ------------------------------end

    start----------------------------
    GFDL-CM3
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    GFDL-ESM2G
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    GFDL-ESM2M
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    GISS-E2-H
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    GISS-E2-H-CC
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    GISS-E2-R
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    GISS-E2-R-CC
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    HadGEM2-AO
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    HadGEM2-CC
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    HadGEM2-ES
    *********************************
    Missing a file:
    T_file present: False
    RH_file present: False
    *********************************
    ------------------------------end

    start----------------------------
    IPSL-CM5A-LR
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    IPSL-CM5A-MR
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    IPSL-CM5B-LR
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    MIROC-ESM
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    MIROC-ESM-CHEM
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    MIROC5
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    MPI-ESM-LR
    *********************************
    Missing a file:
    T_file present: True
    RH_file present: False
    *********************************
    ------------------------------end

    start----------------------------
    MPI-ESM-MR
    *********************************
    Missing a file:
    T_file present: True
    RH_file present: False
    *********************************
    ------------------------------end

    start----------------------------
    MRI-CGCM3
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    MRI-ESM1
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    NorESM1-M
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    NorESM1-ME
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    bcc-csm1-1-m
    Writing VPD esetimate
    ------------------------------end

    start----------------------------
    inmcm4
    Writing VPD esetimate
    ------------------------------end


```python
from sympy import *
y, x, a, s, u = symbols('ry rx amp sigma2 mu')
simplify(diff((y-(a*exp(-(x-u)**2/(2*s**2))))**2, s, u))
```

    2*amp*(mu - rx)*(2*sigma2**2*(amp - ry*exp((mu - rx)**2/(2*sigma2**2))) - (2*amp - ry*exp((mu - rx)**2/(2*sigma2**2)))*(mu - rx)**2)*exp(-(mu - rx)**2/sigma2**2)/sigma2**5

```python
if 1:
    y, x, a, s, u = symbols('ly lx amp sigma1 mu')
else:
    y, x, a, s, u = symbols('ry rx amp sigma2 mu')
hessian((y-(a*exp(-(x-u)**2/(2*s**2))))**2, (a, s, u))
```

    Matrix([
    [                                                                                2*exp(-(lx - mu)**2/sigma1**2),                                                                                2*amp*(lx - mu)**2*exp(-(lx - mu)**2/sigma1**2)/sigma1**3 - 2*(lx - mu)**2*(-amp*exp(-(lx - mu)**2/(2*sigma1**2)) + ly)*exp(-(lx - mu)**2/(2*sigma1**2))/sigma1**3,                                                                                -amp*(-2*lx + 2*mu)*exp(-(lx - mu)**2/sigma1**2)/sigma1**2 + (-2*lx + 2*mu)*(-amp*exp(-(lx - mu)**2/(2*sigma1**2)) + ly)*exp(-(lx - mu)**2/(2*sigma1**2))/sigma1**2],
    [ 2*amp*(lx - mu)**2*exp(-(lx - mu)**2/sigma1**2)/sigma1**3 - 2*(lx - mu)**2*(-amp*exp(-(lx - mu)**2/(2*sigma1**2)) + ly)*exp(-(lx - mu)**2/(2*sigma1**2))/sigma1**3,                              2*amp**2*(lx - mu)**4*exp(-(lx - mu)**2/sigma1**2)/sigma1**6 + 6*amp*(lx - mu)**2*(-amp*exp(-(lx - mu)**2/(2*sigma1**2)) + ly)*exp(-(lx - mu)**2/(2*sigma1**2))/sigma1**4 - 2*amp*(lx - mu)**4*(-amp*exp(-(lx - mu)**2/(2*sigma1**2)) + ly)*exp(-(lx - mu)**2/(2*sigma1**2))/sigma1**6, -amp**2*(-2*lx + 2*mu)*(lx - mu)**2*exp(-(lx - mu)**2/sigma1**2)/sigma1**5 - 2*amp*(-2*lx + 2*mu)*(-amp*exp(-(lx - mu)**2/(2*sigma1**2)) + ly)*exp(-(lx - mu)**2/(2*sigma1**2))/sigma1**3 + amp*(-2*lx + 2*mu)*(lx - mu)**2*(-amp*exp(-(lx - mu)**2/(2*sigma1**2)) + ly)*exp(-(lx - mu)**2/(2*sigma1**2))/sigma1**5],
    [-amp*(-2*lx + 2*mu)*exp(-(lx - mu)**2/sigma1**2)/sigma1**2 + (-2*lx + 2*mu)*(-amp*exp(-(lx - mu)**2/(2*sigma1**2)) + ly)*exp(-(lx - mu)**2/(2*sigma1**2))/sigma1**2, -amp**2*(-2*lx + 2*mu)*(lx - mu)**2*exp(-(lx - mu)**2/sigma1**2)/sigma1**5 - 2*amp*(-2*lx + 2*mu)*(-amp*exp(-(lx - mu)**2/(2*sigma1**2)) + ly)*exp(-(lx - mu)**2/(2*sigma1**2))/sigma1**3 + amp*(-2*lx + 2*mu)*(lx - mu)**2*(-amp*exp(-(lx - mu)**2/(2*sigma1**2)) + ly)*exp(-(lx - mu)**2/(2*sigma1**2))/sigma1**5,                             amp**2*(-2*lx + 2*mu)**2*exp(-(lx - mu)**2/sigma1**2)/(2*sigma1**4) + 2*amp*(-amp*exp(-(lx - mu)**2/(2*sigma1**2)) + ly)*exp(-(lx - mu)**2/(2*sigma1**2))/sigma1**2 - amp*(-2*lx + 2*mu)**2*(-amp*exp(-(lx - mu)**2/(2*sigma1**2)) + ly)*exp(-(lx - mu)**2/(2*sigma1**2))/(2*sigma1**4)]])

```python
if 0:
    y, x, a, s, u = symbols('ly lx amp sigma1 mu')
else:
    y, x, a, s, u = symbols('ry rx amp sigma2 mu')
hessian((y-(a*exp(-(x-u)**2/(2*s**2))))**2, (a, s, u))
```

    Matrix([
    [                                                                                2*exp(-(-mu + rx)**2/sigma2**2),                                                                                2*amp*(-mu + rx)**2*exp(-(-mu + rx)**2/sigma2**2)/sigma2**3 - 2*(-mu + rx)**2*(-amp*exp(-(-mu + rx)**2/(2*sigma2**2)) + ry)*exp(-(-mu + rx)**2/(2*sigma2**2))/sigma2**3,                                                                                -amp*(2*mu - 2*rx)*exp(-(-mu + rx)**2/sigma2**2)/sigma2**2 + (2*mu - 2*rx)*(-amp*exp(-(-mu + rx)**2/(2*sigma2**2)) + ry)*exp(-(-mu + rx)**2/(2*sigma2**2))/sigma2**2],
    [2*amp*(-mu + rx)**2*exp(-(-mu + rx)**2/sigma2**2)/sigma2**3 - 2*(-mu + rx)**2*(-amp*exp(-(-mu + rx)**2/(2*sigma2**2)) + ry)*exp(-(-mu + rx)**2/(2*sigma2**2))/sigma2**3,                          2*amp**2*(-mu + rx)**4*exp(-(-mu + rx)**2/sigma2**2)/sigma2**6 + 6*amp*(-mu + rx)**2*(-amp*exp(-(-mu + rx)**2/(2*sigma2**2)) + ry)*exp(-(-mu + rx)**2/(2*sigma2**2))/sigma2**4 - 2*amp*(-mu + rx)**4*(-amp*exp(-(-mu + rx)**2/(2*sigma2**2)) + ry)*exp(-(-mu + rx)**2/(2*sigma2**2))/sigma2**6, -amp**2*(-mu + rx)**2*(2*mu - 2*rx)*exp(-(-mu + rx)**2/sigma2**2)/sigma2**5 - 2*amp*(2*mu - 2*rx)*(-amp*exp(-(-mu + rx)**2/(2*sigma2**2)) + ry)*exp(-(-mu + rx)**2/(2*sigma2**2))/sigma2**3 + amp*(-mu + rx)**2*(2*mu - 2*rx)*(-amp*exp(-(-mu + rx)**2/(2*sigma2**2)) + ry)*exp(-(-mu + rx)**2/(2*sigma2**2))/sigma2**5],
    [   -amp*(2*mu - 2*rx)*exp(-(-mu + rx)**2/sigma2**2)/sigma2**2 + (2*mu - 2*rx)*(-amp*exp(-(-mu + rx)**2/(2*sigma2**2)) + ry)*exp(-(-mu + rx)**2/(2*sigma2**2))/sigma2**2, -amp**2*(-mu + rx)**2*(2*mu - 2*rx)*exp(-(-mu + rx)**2/sigma2**2)/sigma2**5 - 2*amp*(2*mu - 2*rx)*(-amp*exp(-(-mu + rx)**2/(2*sigma2**2)) + ry)*exp(-(-mu + rx)**2/(2*sigma2**2))/sigma2**3 + amp*(-mu + rx)**2*(2*mu - 2*rx)*(-amp*exp(-(-mu + rx)**2/(2*sigma2**2)) + ry)*exp(-(-mu + rx)**2/(2*sigma2**2))/sigma2**5,                              amp**2*(2*mu - 2*rx)**2*exp(-(-mu + rx)**2/sigma2**2)/(2*sigma2**4) + 2*amp*(-amp*exp(-(-mu + rx)**2/(2*sigma2**2)) + ry)*exp(-(-mu + rx)**2/(2*sigma2**2))/sigma2**2 - amp*(2*mu - 2*rx)**2*(-amp*exp(-(-mu + rx)**2/(2*sigma2**2)) + ry)*exp(-(-mu + rx)**2/(2*sigma2**2))/(2*sigma2**4)]])


# Variational Methods

The variational method is an approximate method used in quantum mechanics. Compared to perturbation theory, the variational method can be more robust in situations where it is hard to determine a good unperturbed Hamiltonian (i.e., one which makes the perturbation small but is still solvable). On the other hand, in cases where there is a good unperturbed Hamiltonian, perturbation theory can be more efficient than the variational method.

The basic idea of the variational method is to guess a "trial" wavefunction for the problem, which consists of some adjustable parameters called "variational parameters". These parameters are adjusted until the energy of the trial wavefunction is minimized. The resulting trial wavefunction and its corresponding energy are variational method approximations to the exact wavefunction and energy.

Why would it make sense that the best approximate trial wavefunction is the one with the lowest energy? This results from the Variational Theorem, which states that the energy of any trial wavefunction $E$ is always an upper bound to the exact ground state energy $E_0$. This can be proven easily. Let the trial wavefunction be denoted $\phi$. Any trial function can formally be expanded as a linear combination of the exact eigenfunctions $\psi_i$. Of course, in practice, we do not know the $\psi_i$, since we are assuming that we are applying the variational method to a problem we can not solve analytically. Nevertheless, that does not prevent us from using the exact eigenfunctions in our proof, since they certainly exist and form a complete set, even if we do not happen to know them:
\begin{equation}
\sum_{i,j}\langle\psi_{i}|\psi_{j}\rangle=\delta_{i,j}
\end{equation}
where $\delta_{i,j}$ is the Kronecker delta.
\begin{equation}
H \left|\psi_i\right\rangle = E_i\left|\psi_i \right\rangle .
\end{equation}

We are asuming that the physical states are normalized, * i.e. * their norm is equal to unity (we can always make it to do so).
Let us assume that we have a candidate wavefunction to describe the ground-state, that we call $|\phi\rangle$, and that this function deppends on a set of parameters ${c_i}$, that are complex numbers.
Ignoring complications involved with a continuous spectrum of H, suppose that the spectrum is bounded from below and that its greatest lower bound is $E_0$.
So, the approximate energy corresponding to this wavefunction is
the expectation value of $H$:
\begin{eqnarray}
\left\langle\phi|H|\phi\right\rangle & = & \sum_{i,j} \left\langle\phi|\psi_{i}\right\rangle \left\langle\psi_{i}|H|\psi_{j}\right\rangle \left\langle\psi_{j}|\phi\right\rangle \\
             & = & \sum_{i} E_i \left|\left\langle\psi_i|\phi\right\rangle\right|^2\ge\sum_{i}E_0 \left|\left\langle\psi_i|\phi\right\rangle\right|^2=E_0
\end{eqnarray}

In other words, the energy of any approximate wavefunction is always greater than or equal to the exact ground state energy ${E}_0$. This explains the strategy of the variational method: since the energy of any approximate trial function is always above the true energy, then any variations in the trial function which lower its energy are necessarily making the approximate energy closer to the exact answer. (The trial wavefunction is also a better approximation to the true ground state wavefunction as the energy is lowered, although not necessarily in every possible sense unless the limit $\phi = \psi_0$ is reached).

Frequently, the trial function is written as a linear combination of basis functions, such as

$$ |\phi\rangle = \sum_i c_i |\phi_i\rangle. $$

This leads to the linear variation method, and the variational parameters are the expansion coefficients $c_i$.
We shall assume that the possible solutions are restricted to a subspace of the Hilbert space, and we shall seek the best possible solution in this subspace.

The energy for this approximate wavefunction is just
\begin{equation} E[\phi] = \frac{\sum_{ij} c_i^* c_j \langle\phi_i |H| \phi_j\rangle}{ \sum_{ij} c_i^* c_j \langle\phi_i|\phi_j\rangle},
\label{evar}
\end{equation}
which can be simplified using the notation
$H_{ij} = \langle \psi_i|H|\psi_j\rangle = \int \phi_i^* {H} \phi_j,$
$S_{ij} = \langle \psi_i|H|\psi_j\rangle = \int \phi_i^* \phi_j,$
to yield
$$ E[\phi] = \frac{\sum_{ij} c_i^* c_j H_{ij}}{\sum_{ij} c_i^* c_j S_{ij}}. $$

In order to find the optimial solution, we need to minimize this energy functional with respect to the variational parameters $c_i$, or calculate the variation such that:
\begin{equation}
\delta E({c_i})=0.
\end{equation}
We will calculate only the variation with respect to $c_i^*$, since the variation with respect to $c_j$ will yield the same result:
\begin{eqnarray}
\delta E & = & \sum_i \frac{\left(\sum_j c_j H_{ij} \sum_{i',j'} c_{i'}^*c_{j'} S_{i'j'} - \sum_j c_j S_{ij} \sum_{i',j'} c_{i'}^*c_{j'} H_{i'j'}\right)}{\left(\sum_{i',j'} c_{i'}^*c_{j'} S_{i'j'} \right)^2} \delta c_i^*.
\end{eqnarray}
Reordering some terms we can rewrite it as:
\begin{equation}
\sum_i \left(\frac{\sum_j c_j H_{ij} - E \sum_j c_j S_{ij}}{\sum_{i'j'} c_{i'}^*c_{j'} S_{i'j'}}\right) \delta c_i^*,
\end{equation}
where $E$ is given by Eq.(\ref{evar}).
This should be satisfied for all $c_i$'s, and we find that the coefficients should satisfy the following conditions:
\begin{equation}
\sum_{j=1}^N{(H_{ij}-ES_{ij})c_j} = 0 \,\,\,\,\, i=1,2,...,N.
\end{equation}
This is a generalized eigenvalue problem, that can be rewritten:
\begin{equation}
{\bf Hc}=E{\bf Sc},
\end{equation}
where ${\bf H}$ is the Hamiltonian matrix, and ${\bf S}$ is the so-called "overlap matrix". The main difference with an ordinary eigenvalue problem is the marix ${\bf S}$ on the right hand side. We'll see how to solve this problem in the exercises.
The solution to thsi problem is obtained by solving the following "secular determinant"
\begin{equation}
\left \vert
\begin{array}{cccc}
H_{11} - E S_{11} & H_{12} - E S_{12} & \cdots & H_{1N}-ES_{1N} \\
H_{21} - E S_{21} & H_{22} - E S_{22} & \cdots & H_{2N}-ES_{2N} \\
\vdots & \vdots & \vdots & \vdots \\
H_{N1} - E S_{N1} & H_{2N} - E S_{2N} & \cdots & H_{NN} - E S_{NN}
\end{array}\right\vert = 0.
\end{equation}

If an orthonormal basis is used, the secular equation is greatly simplified because $S_{ij} = \delta_{ij}$ (1 for $i=j$ and 0 for $i \neq j$), and we obtain:
\begin{equation}
{\bf Hc}=E{\bf c},
\end{equation}
which is nothing else but the stationary Schr\"odinger's equation, formulated in this basis.
In this case, the secular determinant is
\begin{equation}
\left \vert
\begin{array}{cccc}
H_{11} - E & H_{12} & \cdots & H_{1N} \\
H_{21} & H_{22} - E & \cdots & H_{2N} \\
\vdots & \vdots & \vdots & \vdots \\
H_{N1} & H_{2N} & \cdots & H_{NN} - E
\end{array}\right\vert = 0,
\end{equation}

In either case, the secular determinant for $N$ basis functions gives an $N$-th order polynomial in $E$ which is solved for $N$ different roots, each of which approximates a different eigenvalue.
These equations can be easily solved using readily available library routines, such as those in Numerical Recipes, or Lapack.

These equations can be easily solved using readily available library routines, such as those in Numerical Recipes, or Lapack.
At this point one may wonder where the approximation is: aren't we solving the problem exactly? If we take into account a complete basis set, the answer is "yes, we are solving the problem exactly". But as we said before, teh Hilbert space is very large, and we therefore have to limit the basis size to a number that is easily tractable with a computer. Therefore, we have to work in a constrained Hilbert space with a relatively small number of basis states kept, which makes the result variational.
Because of the computer time needed for numerical diagonalizations scales as the third power of the linear matrix size, we would want to keep the basis size as small as possible. Therefore, the basis wavefunctions must be choses carefully: it should be possible to approximate the exact solution to the full problem with a small number of basis states. Inorder to do that, we need some good intuition about the underlying physics of the problem.

We have used a linear parametrization of the wave function. This greatly simplifies the calculations.
However, nonlinear parametrizations are also possible, such as in the case of hartree-Fock theory.

The variational method lies behind hartree-Fock theory and the configuration interaction method for the electronic structure of atoms and molecules, as we will see in the following chapter.

## Examples of linear variational calculations
### The infinite potential well

The potential well with inifinite barriers is defined:
\begin{equation}
V(x) = \left\{
\begin{array}{ccc}
 \infty & {\mathrm {for}} & |x| > |a| \\
 0  & {\mathrm {for}} & |x| \leq  |a|
\end{array}
\right.
\end{equation}
and it forces the wave function to vanish at the boundaries of the well at $x=\pm a$. The exact solutioon for this problems is known and treated in introductory quantum mechanics courses. Here we discuss a linear variational approach to be compared with the exact solution. We take $a=1$ and use natural units such that $\hbar^2/2m=1$.

As basis functions we take simple polynomials that vanish on the boundaries of the well:
\begin{equation}
\psi_n(x)=x^n(x-1)(x+1), n=0,1,2,3...
\end{equation}
The reason for choosing this particular form of basis functions is that the relevant matrix elements can easily be calculated analytically. We start we the overlap matrix:
\begin{equation}
S_{mn}=\langle \psi_n|\psi_m \rangle = \int_{-1}^1 \psi_n(x) \psi_m(x) dx.
\end{equation}
Working out the integrals, one obtains
\begin{equation}
S_{mn}=\frac{2}{n+m+5} - \frac{4}{n+m+3} + \frac{2}{n+m+1}
\end{equation}
for $n+m$ even, and zero otherwise.

We can also calculate the Hamiltonian matrix elements:
\begin{eqnarray}
H_{mn}=\langle \psi_n | p^2 | \psi_m \rangle = \int_{-1}^1 \psi_n(x) \left(-\frac{d^2}{dx^2} \right) \psi_m(x) dx \\
= -8 \left[ \frac{1-m-n-2mn}{(m+n+3)(m+n+1)(m+n-1)} \right]
\end{eqnarray}
for $m+n$ even, and zero otherwise.

#### Exercise 16.1: Infinite potential well

$\bullet$ Write a computer program in which you fill the overlap and Hamiltonian matrices for this problem. Use standard software to solve the generalized eigenvalue problem. Notice that the matrices are Hermitian, so only the upper, or lower triangular parts have to be calculated (including the diagonal).

$\bullet$ Compare results with exact analytic solutions. These are given by
\begin{equation}
\psi_n(x) = \left\{
\begin{array}{ccc}
\cos{(k_nx)} & n & {\mathrm {odd}} \\
\sin{(k_nx)} & n & {\mathrm{even \,\, and \,\, positive}}
\end{array}\right.
\end{equation}
with $k_n=n\pi/2, n=1,2...$, and the corresponding energies are given by
\begin{equation}
E_n = k_n^2=\frac{n^2\pi^2}{4}
\end{equation}
For each eigenvector ${\bf c}$, the function $\sum_{p=1}^{N} = c_p\phi_p(x)$ should approximate an exact eigenfunction. They can be compared by displaying both graphically. Carry out the comparison for various numbers of basis states kept.

```python
import numpy as np
from scipy import linalg

N=10
h=np.zeros((N,N), dtype=float)
s=np.zeros((N,N), dtype=float)

for m in range(N):
    for n in range(m,N):
        if((m+n)%2 == 0):
            h[n,m]=-8*(1-m-n-2*m*n)/((m+n+3)*(m+n+1)*(m+n-1))
            s[n,m] = 2./(m+n+5)-4./(m+n+3)+2./(n+m+1)
# No need for these lines below: we only have to define the lower triangle
#            h[m,n] = h[n,m]
#            s[m,n] = s[n,m]

w, v = linalg.eigh(h,s)
for n in range(N):
    print(n,np.pi**2*(n+1)**2/4,w[n])
```

    0 2.4674011002723395 2.467401100272339
    1 9.869604401089358 9.869604401093914
    2 22.206609902451056 22.2066123442276
    3 39.47841760435743 39.47850458931857
    4 61.68502750680849 61.76067928958051
    5 88.82643960980423 89.16437778460741
    6 120.90265391334464 132.91796389990037
    7 157.91367041742973 181.68712260391723
    8 199.8594891220595 463.1473433601288
    9 246.74011002723395 642.3003906576474

### Hydrogen atom

One example of the variational method would be using the Gaussian function $\chi(r) = e^{- \alpha r^2}$ as a trial function for the hydrogen atom ground state. This problem could be solved by the variational method by obtaining the energy of $\chi(r)$ as a function of the variational parameter $\alpha$, and then minimizing $E(\alpha)$ to find the optimum value $\alpha_{min}$. The variational theorem's approximate wavefunction and energy for the hydrogen atom would then be $\chi(r) = e^{- \alpha_{min} r^2}$ and $E(\alpha_{min})$.

This is a one electron problem, so we do not have to worry about electron-electron interactions, or antisymmetrization of the wave function.
The Schr\"odinger's equation reads:
\begin{equation}
\left[ -\frac{\hbar^2}{2m}\nabla^2 - \frac{e^2}{4\pi\epsilon_0}\frac{1}{r} \right] \psi(x) = E \psi(x)
\end{equation}
where the second term is the Coulomb interaction with the positive nucleus (remember, this is a charged particle in a central potential). The mass $m$ is the reduced mass of the proton-electron system, which is approximately equal to the electron mass. The ground state has energy
\begin{equation}
E=-\frac{N}{\hbar^2} \left(\frac {{\mathrm e}^2}{4\pi\epsilon_0} \right)^2 \approx -13.6058 {\mathrm eV}
\end{equation}
and the wave function is given by
\begin{equation}
\psi({\bf x}) = \frac{2}{a_0^{3/2}} \exp{(-x/a_0)}
\label{H_exact}
\end{equation}
where $a_0$ is Bohr's radius
\begin{equation}
a_0=\frac{4\pi\epsilon_0\hbar^2}{m{\mathrm e}^2}.
\end{equation}

It is convenient to use units such that equations take on a simpler form. These are the so-called *standard units* in electronic structure: the unit of distance is Bohr's radius, masses are expressed in units of the electon mass $m_{\mathrm e}$, and charge in units of the electron charge e. The energy is finally given in "hartrees", equal to $E_H=m_{\mathrm e}c^2\alpha^2$ (where $\alpha$ is the fine structure constant). In these units the Schr\"odinger equation for the hydrogen atom assumes the following simpler form:
\begin{equation}
\left[ -\frac{1}{2}\nabla^2 - \frac{1}{r} \right] \psi(x) = E \psi(x).
\end{equation}

To approximate the ground state energy and wave function of the hydrogen atom in a linear variational procedure, we will use Gaussian basis functions. For the ground state, we only need angular momentum $l=0$ wave functions ($s$-orbitals), which have the form:
\begin{equation}
\chi_p(r)=\exp{(-\alpha_p r^2)}
\end{equation}
centered on the nucleus (whis is thus placed at the origin). We have to specify the values of the exponents $\alpha_p$, which are our variational parameters. Optimal values of these exponents have been previously found by other means, and
in our case, we will keep these values fixed:
\begin{eqnarray}
\alpha_1 &=& 13.00773 \\
\alpha_2 &=& 1.962079 \\
\alpha_3 &=& 0.444529 \\
\alpha_4 &=& 0.1219492.
\end{eqnarray}
If the program works correctly, it should shield a value of the energy close to the exact results $E_0=-1/2 E_H$.

It remains to determine the coefficients of the linear expansion, by solving the generalized eigenvalue problem, as we did in the previous example.
The matrix elements of
the overlap matrix ${\bf S}$, the kinetic energy matrix ${\bf T}$, and the Coulomb interaction ${\bf V}$ are given by:
\begin{eqnarray}
S_{pq} &=& \int d^3r e^{-\alpha_pr^2}e^{-\alpha_qr^2} = \left(\frac{\pi}{\alpha_p+\alpha_q}\right)^{3/2} \\
T_{pq} &=& \int d^3r e^{-\alpha_pr^2}\nabla^2 e^{-\alpha_qr^2} = 3 \frac{\alpha_p\alpha_q\pi^{3/2}}{(\alpha_p+\alpha_q)^{5/2}} \\
V_{pq} &=& \int d^3r e^{-\alpha_pr^2} \frac{1}{r} e^{-\alpha_qr^2} = - \frac{2\pi}{(\alpha_p+\alpha_q)}.
\end{eqnarray}
Using these expressions, one can fill the overlap and Hamiltonian matrices and solve the problem numerically.

#### Exercise 16.2: Hydrogen atom

$\bullet$ Solve the problem stated in the previous section. If your program has no errors, you should obtain $E=-0.499278 E_H$, which is remarkable considering that only four basis states have been taken into account.

$\bullet$ Compare graphically the variational ground-state to the exact one, given by
\begin{equation}
\psi({\bf x}) = \frac{2}{a_0^{3/2}} \exp{(-x/a_0)}
\end{equation}

```python
import numpy as np
from scipy import linalg

N=4
h=np.zeros((N,N), dtype=float)
s=np.zeros((N,N), dtype=float)

𝛼 = np.array([13.00773,1.962079,0.444529,0.1219492])
for m in range(N):
    for n in range(m,N):
        t=3*𝛼[n]*𝛼[m]*np.pi**1.5/(𝛼[n]+𝛼[m])**2.5
        v=-2*np.pi/(𝛼[n]+𝛼[m])
        h[n,m]=t+v
        s[n,m]=(np.pi/(𝛼[n]+𝛼[m]))**1.5

w, v = linalg.eigh(h,s)
for n in range(N):
    print(n,w[n])
```

    0 -0.49927840566748505
    1 0.1132139204579877
    2 2.5922995719598165
    3 21.144365190122503

```python

```


## Problem 7.2. Chi-squared Test.

In this problem, you will calculate the chi-squared statstiic and
  the p-value to accept or reject the hypothesis that the number
  of flight cancellations depends on the month of the year.

```
import numpy as np
import pandas as pd
from scipy import stats
```

Suppose you are studying the number of flight cancellations
  at a small regional airport and observe the following pattern:

    January           23
    February          12
    March             10
    April             10
    May               13
    June               7
    July              12
    August            11
    September         22
    October            6
    November           6
    December          14

These were the actual 2001 flight cancellations at the Willard airport (CMI), which you should be able to check easily from 2001.csv. It seems like there are a lot of cancellations in January and September, and fewer than average cancellations in June, October, and December.

In this problem, we will compute the chi-squared statistic to test the hypothesis that the number of flight cancellations depends on the month of the year.
In other words, we will accept or reject the null hypothesis (the hypothesis that the fluctuations in flight cancellations is entirely due to random fluctuations) using the chi-squared test.
The chi-squared statistic is defined as

\begin{equation}
\chi^2 = \sum_i \frac{\left(O_i - E_i\right)^2}{E_i}.
\end{equation}

We will also compute the corresponding p-value of this distribution.
For example, if the p-value is, say, 50%, then it means that there's a 50% chance and that the apparent pattern is due to chance. On the other hand, if the p-value is something small like 0.1%, then this pattern happens only one time in 1000 when the number of cancellations does not depend on the month of the year.
In this problem, we will compute the $\chi^2$ statistic defined as

and the corresponding $p$-value of this distribution.

I summarized the number of flight cancellations in the following array:

```
cancelled = np.array([23, 12, 10, 10, 13,  7, 12, 11, 22,  6,  6, 14])
```

I will break down each step in separate functions to make sure you know
  how to calculate $\chi^2$. But note that you would normally combine
  all the steps in a single function.
  Also note that there's a SciPy function
  that does this for you:
  [`scipy.stats.chisquare()`](http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.chisquare.html).
  You may **not** use this function to write your code, but you can use it to check your answer.

The $\chi^2$ test is covered in Section 7.7 of *Think Stats*.

First, we define the cells to be the twelve months of the year.

### get\_expected()

Second, write a function named `get_expected()` that takes a Numpy array
  and returns a Numpy array with the number of flight cancellations
  expected under the *null hypothesis*.
  Under the null hypothesis, we expect a uniform distribution,
  i.e. the same number of flight cancellations in each month.
  However, we also want our function to handle different types of data,
  so do not simply add up all the numbers and divide by 12.
  Instead, you should use the length of the array.
  For example, if the input array is `np.array([1.0, 2.0, 3.0, 4.0, 5.0]`, the `get_expected()`
  should return `np.array([3.0, 3.0, 3.0, 3.0, 3.0])`.

```
def get_expected(x):
    '''
    Takes a Numpy array and returns a Numpy array of the same length
    that represents the expected values (uniform distribution).

    Parameters
    ----------
    x: A Numpy array.

    Returns
    -------
    A Numpy array of the same length as x.

    Example
    -------
    >>> import numpy as np
    >>> x = get_expected([1., 2.])
    >>> x = get_expected(np.array([2., 1., 3., 4.]))
    >>> x
    array([ 2.5,  2.5,  2.5,  2.5])
    '''

    # your code goes here
    expected = np.ones(len(x)) * sum(x) / len(x)
    # my code ends here

    return expected

```

### get\_diff()

Next, write a function named `get_diff()` that takes two Numpy arrays,
   an array of the observed values and an array of the expected values.
   For each cell, the funtion calculates the difference between the observed value $O_i$ and the expected value $E_i$.
   Thus, it should return a Numpy array of $O_i$ - $E_i$.

```
def get_diff(observed, expected):
    '''
    Takes two Numpy arrays, the observed values and the expected values,
    and returns a Numpy array of the differences.

    Parameters
    ----------
    observed: A Numpy array.
    expected: A Numpy array.

    Returns
    -------
    A Numpy array.

    Example
    -------
    >>> import numpy as np
    >>> x = np.array([2., 4., 6.])
    >>> y = np.array([3., 6., 9.])
    >>> z = get_diff(x, y)
    >>> z
    array([-1., -2., -3.])
    '''

    # your code goes here
    diff = observed - expected
    # my code ends here

    return diff
```

### get_chi2()

Next, write a function named `get_chi2()` that takes two Numpy arrays,
  an array of the difference between the observed values and the expected values
  and an array of the expected values.
  It should return a single float that represents the $\chi^2$ statistic.

```
def get_chi2(diff, expected):
    '''
    Takes O - E and computes chi-squared.

    Parameters
    ----------
    diff: A Numpy array. O - E
    expected: A numpy array. E

    Returns
    -------
    A float. The chi-squared value.

    Example
    -------
    >>> import numpy as np
    >>> x = np.array([1., 2., 3.])
    >>> y = np.array([1., 2., 3.])
    >>> z = get_chi2(x, y)
    >>> z
    '''

    # your code goes here
    chi2 = sum(diff * diff / expected)
    # my code ends here

    return chi2
```

### get\_p\_value()

Finally, write a function named `get_p_value()` that takes two arguments,
  the $\chi^2$ value (a float) and a degree of freedom (an int).
  The $p$-value can easily be calculated by using
  [scipy.stats.chi2.cdf()](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.chi2.html).
  The $p$-value is given simply by 1 minus the CDF of the $\chi^2$ distribution
  (See [Chi-squared distribution](http://en.wikipedia.org/wiki/Chi-squared_distribution)).
  When using `scipy.stats.chi2.cdf()`, you should specify the `df` option,
  which corresponds to the degrees of freedom.
  The [degrees of freedom](http://en.wikipedia.org/wiki/Degrees_of_freedom_%28statistics%29)
  is the number of cells minus 1, which in our case is 11
  (the twelve months of the year minus 1).

```
def get_p_value(chi2, dof):
    '''
    Takes chi-squared and degrees of freedom
    and computes the p-value.

    Parameters
    ----------
    chi2: A float.
    dof: An int. Degrees of freedom.

    Returns
    -------
    >>> import numpy as np
    >>> x = 1.0
    >>> y = 2
    >>> z = get_p_value(x, y)
    0.60653065971263342
    '''

    # your code goes here
    p = 1 - stats.chi2.cdf(chi2, df = dof)
    # my code ends here

    return p
```

When you run the following cell, you should get

    The chi square value is 27.3 and the p-value is 4.2e-03.

That means there's a 0.4% chance that the observed pattern is due to chance.
I think we can safely reject the null hypothesis.

```
expected = get_expected(cancelled)
diff = get_diff(cancelled, expected)
chi2 = get_chi2(diff, expected)
p = get_p_value(chi2, len(expected) - 1)
print('The chi square value is {0:.1f} and the p-value is {1:.1e}.'.format(chi2, p))
```

    The chi square value is 27.3 and the p-value is 4.2e-03.

```

```


importamos la libreria sympy
esto tambien nos permite tener un salida en LATEX

```python
from sympy import init_session
```

Inicializamos para la salida en LATEX

```python
init_session(use_latex=True)
```

    IPython console for SymPy 1.7.1 (Python 3.7.10-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.7.1/

Las variables debes ser declaradas

```python
a = symbols('a')
b = symbols('b')
a,b
```

Utilizando PI

```python
(b + pi)
```

```python
(a + pi) ** 2
```

tipo de variable

```python
type(a)
```

    sympy.core.symbol.Symbol

Pasar los simbolos a variables

```python
a = 2
b = 3
(a + b) * pi
```

Para saber el valor de a ( anterior era symbol)

```python
type(a)
```

    int