# Linear algebra games including SVD for PCA

Some parts adapted from [Computational-statistics-with-Python.ipynb](https://github.com/cliburn/Computational-statistics-with-Python), which is itself from a course taught at Duke University; other parts from Peter Mills' [blog](https://blog.statsbot.co/singular-value-decomposition-tutorial-52c695315254).

The goal here is to practice some linear algebra manipulations by hand and with Python, and to gain some experience and intuition with the Singular Value Decomposition (SVD).
$\newcommand{\Amat}{\mathbf{A}} \newcommand{\AmatT}{\mathbf{A^\top}}
\newcommand{\thetavec}{\boldsymbol{\theta}}
\newcommand{\Sigmamat}{\mathbf{\Sigma}}
\newcommand{\Yvec}{\mathbf{Y}}
$

## Preliminary exercise: manipulations using the index form of matrices

If you haven't already done this earlier, prove that the Maximum Likelihood Estimate (MLE) for $\chi^2$ given by

$$
\chi^2 = (\Yvec - \Amat\thetavec)^{\mathbf{\top}} \Sigmamat^{-1} (\Yvec - \AmatT\thetavec)
$$

is

$$
\thetavec_{\mathrm{MLE}} = (\AmatT \Sigmamat^{-1} \Amat)^{-1} (\AmatT \Sigmamat^{-1} \Yvec)  \;.
$$

Here $\thetavec$ is a $m\times 1$ matrix of parameters (i.e., there are $m$ parameters), $\Sigmamat$ is the $m\times m$ covariance matrix, $\Yvec$ is a $N\times 1$ matrix of observations (data), and $\Amat$ is an $N\times m$ matrix

$$
\Amat =
\left(
\begin{array}{cccc}
   1  & x_1  & x_1^2 & \cdots \\
   1  & x_2  & x_2^2 & \cdots \\
   \vdots & \vdots & \vdots &\cdots \\
   1  & x_N  & x_N^2 & \cdots
\end{array}
\right)
$$

where $N$ is the number of observations.  The idea is to do this with explicit indices for vectors and matrices, using the Einstein summation convention.

A suggested approach:
* Write $\chi^2$ in indices: $\chi^2 = (Y_i - A_{ij}\theta_j)\Sigma^{-1}_{ii'}(Y_{i'}- A_{i'j'}\theta_{j'})$, where summations over repeated indices are implied (be careful of transposes).  *How do we see that $\chi^2$ is a scalar?*
* Find $\partial\chi^2/\partial \theta_k = 0$ for all $k$, using $\partial\theta_j/\partial\theta_k = 0$. Isolate the terms with one component of $\thetavec$ from those with none.
* You should get the matrix equation $ (\AmatT \Sigmamat^{-1} \Yvec) = (\AmatT \Sigmamat^{-1} \Amat)\thetavec$. At this point you can directly solve for $\thetavec$. *Why can you do this now?*
* If you get stuck, see Dick's notes from the Parameter Estimation III lecture.

## SVD basics

A singular value decomposition (SVD) decomposes a matrix $A$ into three other matrices (we'll skip the boldface font here):

$$
A = U S V^\top
$$

where (take $m > n$ for now)
* $A$ is an $m\times n$ matrix;
* $U$ is an $m\times n$ (semi)orthogonal matrix;
* $S$ is an $n\times n$ diagonal matrix;
* $V$ is an $n\times n$ orthogonal matrix.

Comments and tasks:
* *Verify that these dimensions are compatible with the decomposition of $A$.*
* The `scipy.linalg` function `svd` has a Boolean argument `full_matrices`.  If `False`, it returns the decomposition above with matrix dimensions as stated.  If `True`, then $U$ is $m\times m$, $S$ is $m \times n$, and $V$ is $n\times n$.  We will use the `full_matrices = False` form here.  *Can you see why this is ok?*
* Recall that orthogonal means that $U^\top U = I_{n\times n}$ and $V V^\top = I_{n\times n}$.  *Are $U U^\top$ and $V^\top V$ equal to identity matrices?*
* In index form, the decomposition of $A$ is $A_{ij} = U_{ik} S_k V_{jk}$, where the diagonal matrix elements of $S$ are
$S_k$ (*make sure you agree*).
* These diagonal elements of $S$, namely the $S_k$, are known as **singular values**.  They are ordinarily arranged from largest to smallest.
* $A A^\top = U S^2 U^\top$, which implies (a) $A A^\top U = U S^2$.
* $A^\top A = V S^2 V^\top$, which implies (b) $A^\top A V = V S^2$.
* If $m > n$, we can diagonalize $A^\top A$ to find $S^2$ and $V$ and then find $U = A V S^{-1}$.  If $m < n$ we switch the roles of $U$ and $V$.

Quick demonstations for you to do or questions to answer:
* *Show from equations (a) and (b) that both $U$ and $V$ are orthogonal and that the eigenvalues, $\{S_i^2\}$, are all positive.*
* *Show that if $m < n$ there will be at most $m$ non-zero singular values.*
* *Show that the eigenvalues from equations (a) and (b) must be the same.*

A key feature of the SVD for us here is that the sum of the squares of the singular values equals the total variance in $A$, i.e., the sum of squares of all matrix elements (squared Frobenius norm). Thus the size of each says how much of the total variance is accounted for by each singular vector.  We can create a truncated SVD containing a percentage (e.g., 99%) of the variance:

$$
  A_{ij} \approx \sum_{k=1}^{p} U_{ik} S_k V_{jk}
$$

where $p < n$ is the number of singular values included. Typically this is not a large number.

### Geometric interpretation of SVD

- Geometric interpretation of SVD
    - rotate orthogonal frame $V$ onto standard frame
    - scale by $S$
    - rotate standard frame into orthogonal frame $U$

Consider the two-dimensional case: $\mathbf{x_1} = (x_1, y_1)$, $\mathbf{x_2} = (x_2, y_2)$.  We can fit these to an ellipse with major axis $a$ and minor axis $b$, made by stretching and rotating a unit circle.  Let $\mathbf{x'} = (x', y')$ be the transformed coordinates:

$$
\mathbf{x'} = \mathbf{x} R M^{-1} \quad\mbox{with}\quad
    R = \left(\begin{array}{cc}
               \cos\theta & \sin\theta \\
               -\sin\theta & \cos\theta
              \end{array}
        \right)
    \quad\mbox{and}\quad
    M = \left(\begin{array}{cc}
               a & 0 \\
               0 & b
              \end{array}
        \right)
$$

In index form this is $x'_j = \frac{1}{m_j} x_i R_{ij}$ or (clockwise rotation):

$$\begin{align}
x' &= \frac{x \cos\theta - y\sin\theta}{a} \\
y' &= \frac{x \sin\theta + y\cos\theta}{b} \\
\end{align}$$

The equation for a unit circle $\mathbf{x' \cdot x'} = 1$ becomes

$$
(M^{-1} R^\top \mathbf{x}) \cdot (\mathbf{x} R M^{-1}) = 1.
$$

With $X = \left(\begin{array}{cc}
               x_1 & y_1 \\
               x_2 & y_2
              \end{array}
        \right)$ we find the matrix equation:

$$
M^{-1} R^\top X^\top X R M^{-1}= 1.
$$

which is just a rearrangement of the equation from above, $A^\top A V = V S^2$.

**Interpretation:** If $A$ is considered to be a collection of points, then the singular values are the axes of a least-squares fitted ellipsoid while $V$ is its orientation.  The matrix $U$ is the projection of each of the points in $A$ onto the axes.

### Solving matrix equations with SVD

We can solve for $\mathbf{x}$:

$$\begin{align}
  A \mathbf{x} &= b \\
  \mathbf{x} &= V S^{-1} U^\top b
\end{align}$$

or $x_i = \sum_j \frac{V_{ij}}{S_j} \sum_k U_{kj} b_k$.  The value of this solution method is when we have an ill-conditioned matrix, meaning that the smallest eigenvalues are zero or close to zero.  We can throw away the corresponding components and all is well! See [also](https://personalpages.manchester.ac.uk/staff/timothy.f.cootes/MathsMethodsNotes/L3_linear_algebra3.pdf).

Comments:
- If we have a non-square matrix, it still works. If $m\times n$ with $m > n$, then only $n$ singular values.
- If $m < n$, then only $n$ singular values.
- This is like solving

$$A^\top A \mathbf{x} = A^\top b$$

which is called the *normal equation*.  It produces the solution to $\mathbf{x}$ that is closest to the origin, or

$$
  \min_{\mathbf{x}} |A\mathbf x - b| \;.
$$

**Task:** *prove these results (work backwards from the last equation as a least-squares minimization)*.

### Data reduction

For machine learning (ML), there might be several hundred variables but the algorithms are made for a few dozen.  We can use SVD in ML for variable reduction.  This is also the connection to sloppy physics models.  In general, our matrix $A$ can be closely approximated by only keeping the largest of the singular values.  We'll see that visually below using images.

## Python imports

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.linalg as la

from sklearn.decomposition import PCA
```

*Generate random matrices and verify the properties for SVD given above.  Check what happens when $m > n$.*

```python
A = np.random.rand(9, 4)
print('A = ', A)

Ap = np.random.randn(5, 3)
print('Ap = ', Ap)
```

Check the definition of `scipy.linalg.svd` with shift-tab-tab.

```python
# SVD from scipy.linalg
U, S, V_trans = la.svd(A, full_matrices=False)
Up, Sp, Vp_trans = la.svd(Ap, full_matrices=False)
```

```python
print(U.shape, S.shape, V_trans.shape)
```

```python
# Transpose with T, matrix multiplication with @
print(U.T @ U)
```

```python
# Here's one way to suppress small numbers from round-off error
np.around(U.T @ U, decimals=15)
```

```python
# Predict this one before evaluating!
print(U @ U.T)
```

Go on and check the other claimed properties.

For example, is $A = U S V^\top$? (Note: you'll need to make $S$ a matrix with `np.diag(S)`.)

```python
# Check the other properties, changing the matrix size and shapes.

```

For a square matrix, compare the singular values in $S$ to the eigenvalues from `la.eig`.  What do you conclude?  Now try this for a symmetric matrix (note that a matrix plus its transpose is symmetric).

## SVD applied to images for compression

Read in `figs/elephant.jpg` as a gray-scale image. The image has $1066 \times 1600$ values. Using SVD, recreate the image with a relative error of less than 0.5%. What is the relative size of the compressed image as a percentage?

```python
from skimage import io

img = io.imread('figs/elephant.jpg', as_gray=True)
plt.imshow(img, cmap='gray');
print('shape of img: ', img.shape)
```

```python
# turn off axis
plt.imshow(img, cmap='gray')
plt.gca().set_axis_off()
```

```python
# Do the svg
U, S, Vt = la.svd(img, full_matrices=False)
```

```python
# Check the shapes
U.shape, S.shape, Vt.shape
```

```python
# Check that we can recreate the image
img_orig = U @ np.diag(S) @ Vt
print(img_orig.shape)
plt.imshow(img_orig, cmap='gray')
plt.gca().set_axis_off()
```

Here's how we can efficiently reduce the size of the matrices.  Our SVD should be sorted, so we are keeping only the largest singular values up to a point.

```python
# Pythonic way to figure out when we've accumulated 99.5% of the result
k = np.sum(np.cumsum((S**2)/(S**2).sum()) <= 0.995)
```

#### Aside: dissection of the Python statement to find the index for accumulation

```python
test = np.array([5, 4, 3, 2, 1])
threshold = 0.995
print('initial matrix, in descending magnitude: ', test)
print( 'fraction of total sum of squares: ', (test**2) / (test**2).sum() )
print( 'cumulative fraction: ', np.cumsum((test**2) / (test**2).sum()) )
print( 'mark entries as true if less than or equal to threshold: ',
       (np.cumsum((test**2) / (test**2).sum()) <= threshold) )
print( 'sum up the Trues: ',
       np.sum(np.cumsum((test**2) / (test**2).sum()) <= threshold) )
print( 'The last result is the index we are looking for.')
```

```python
# Let's plot the eigenvalues and mark where k is
fig = plt.figure(figsize=(8,6))
ax = fig.add_subplot(1,1,1)
ax.semilogy(S, color='blue', label='eigenvalues')
ax.axvline(k, color='red', label='99.5% of the variance');
ax.set_xlabel('eigenvalue number')
ax.legend()
fig.tight_layout()
```

Now keep only the most significant eigenvalues (those up to k).

```python
img2 = U[:,:k] @ np.diag(S[:k])@ Vt[:k, :]
img2.shape
```

```python
plt.imshow(img2, cmap='gray')
plt.gca().set_axis_off();
```

```python
k99 = np.sum(np.cumsum((S**2)/(S**2).sum()) <= 0.99)
img99 = U[:,:k99] @ np.diag(S[:k99])@ Vt[:k99, :]
```

```python
plt.imshow(img99, cmap='gray')
plt.gca().set_axis_off();
```

Let's try another interesting picture . . .

```python
fraction_kept = 0.995

def svd_shapes(U, S, V, k=None):
    if k is None:
        k = len(S)
    U_shape = U[:,:k].shape
    S_shape = S[:k].shape
    V_shape = V[:,:k].shape
    print(f'U shape: {U_shape}, S shape: {S_shape}, V shape: {V_shape}')

img_orig = io.imread('figs/Dick_in_tailcoat.jpg')
img = io.imread('figs/Dick_in_tailcoat.jpg', as_gray=True)

U, S, V = la.svd(img)
svd_shapes(U, S, V)

k995 = np.sum(np.cumsum((S**2)/(S**2).sum()) <= fraction_kept)
print(f'k995 = {k995}')
img995 = U[:,:k995] @ np.diag(S[:k995])@ V[:k995, :]
print(f'img995 shape = {img995.shape}')
svd_shapes(U, S, V, k995)

fig = plt.figure(figsize=(12,6))
ax1 = fig.add_subplot(1,3,1)
ax1.imshow(img_orig)
ax1.set_axis_off()

ax2 = fig.add_subplot(1,3,2)
ax2.imshow(img, cmap='gray')
ax2.set_axis_off()

ax3 = fig.add_subplot(1,3,3)
ax3.imshow(img995, cmap='gray')
ax3.set_axis_off()

fig.tight_layout()
```

```python
# Let's plot the eigenvalues and mark where k is
fig = plt.figure(figsize=(8,6))
ax = fig.add_subplot(1,1,1)
ax.semilogy(S, color='blue', label='eigenvalues')
ax.axvline(k995, color='red', label='99.5% of the variance');
ax.set_xlabel('eigenvalue number')
ax.legend()
fig.tight_layout()
```

### Things to do:

* Get your own figure and duplicate these results.  Then play!
* As you reduce the percentage of the variance kept, what features of the image are retained and what are lost?
* See how small you can make the percentage and still recognize the picture.
* How is this related to doing a spatial Fourier transform, applying a low-pass filter, and transforming back.  (Experts: try this!)

```python

```

## Covariance, PCA and SVD

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import scipy.linalg as la

np.set_printoptions(precision=3)
```

Recall the formula for covariance

$$
\text{Cov}(X, Y) = \frac{\sum_{i=1}^n(X_i - \bar{X})(Y_i - \bar{Y})}{n-1}
$$

where $\text{Cov}(X, X)$ is the sample variance of $X$.

```python
def cov(x, y):
    """Returns covariance of vectors x and y)."""
    xbar = x.mean()
    ybar = y.mean()
    return np.sum((x - xbar)*(y - ybar))/(len(x) - 1)
```

```python
X = np.random.random(10)
Y = np.random.random(10)
```

```python
np.array([[cov(X, X), cov(X, Y)], [cov(Y, X), cov(Y,Y)]])
```

```python
np.cov(X, Y)  # check against numpy
```

```python
# Extension to more variables is done in a pair-wise way
Z = np.random.random(10)
np.cov([X, Y, Z])
```

### Eigendecomposition of the covariance matrix

```python
# Zero mean but off-diagonal correlation matrix
mu = [0,0]
sigma = [[0.6,0.2],[0.2,0.2]]
n = 1000
x = np.random.multivariate_normal(mu, sigma, n).T
plt.scatter(x[0,:], x[1,:], alpha=0.2);
```

```python
# Find the covariance matrix of the matrix of points x
A = np.cov(x)
```

```python
# m = np.array([[1,2,3],[6,5,4]])
# ms = m - m.mean(1).reshape(2,1)
# np.dot(ms, ms.T)/2
```

```python
# Find the eigenvalues and eigenvectors
e, v = la.eigh(A)
```

```python
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(1,1,1)
ax.scatter(x[0,:], x[1,:], alpha=0.2)
for e_, v_ in zip(e, v.T):
    ax.plot([0, 3*e_*v_[0]], [0, 3*e_*v_[1]], 'r-', lw=2)
ax.axis([-3,3,-3,3])
ax.set_aspect(1)
ax.set_title('Eigenvectors of covariance matrix scaled by eigenvalue.');
```

### PCA (from Duke course)

"Principal Components Analysis" (PCA) basically means to find and rank all the eigenvalues and eigenvectors of a covariance matrix. This is useful because high-dimensional data (with $p$ features) may have nearly all their variation in a small number of dimensions $k<p$, i.e. in the subspace spanned by the eigenvectors of the covariance matrix that have the $k$ largest eigenvalues. If we project the original data into this subspace, we can have a dimension reduction (from $p$ to $k$) with hopefully little loss of information.

Numerically, PCA is typically done using SVD on the data matrix rather than eigendecomposition on the covariance matrix. Numerically, the condition number for working with the covariance matrix directly is the square of the condition number using SVD, so SVD minimizes errors."

For zero-centered vectors,

\begin{align}
\text{Cov}(X, Y) &= \frac{\sum_{i=1}^n(X_i - \bar{X})(Y_i - \bar{Y})}{n-1} \\
  &= \frac{\sum_{i=1}^nX_iY_i}{n-1} \\
  &= \frac{XY^T}{n-1}
\end{align}

and so the covariance matrix for a data set $X$ that has zero mean in each feature vector is just $XX^T/(n-1)$.

In other words, we can also get the eigendecomposition of the covariance matrix from the positive semi-definite matrix $XX^T$.

Note: Here $x$ is a matrix of **row** vectors.

```python
X = np.random.random((5,4))
X
```

```python
Y = X - X.mean(axis=1)[:, None]  # eliminate the mean
print(Y.mean(axis=1))
```

```python
np.around(Y.mean(1), 5)
```

```python
Y
```

Check that the covariance matrix is unaffected by removing the mean:

```python
np.cov(X)
```

```python
np.cov(Y)
```

```python
# Find the eigenvalue and eigenvectors
e1, v1 = np.linalg.eig(np.dot(x, x.T)/(n-1))
```

#### Principal components

Principal components are simply the eigenvectors of the covariance matrix used as basis vectors. Each of the original data points is expressed as a linear combination of the principal components, giving rise to a new set of coordinates.

```python
# Check that we reproduce the previous result
fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(1,1,1)
ax.scatter(x[0,:], x[1,:], alpha=0.2)
for e_, v_ in zip(e1, v1.T):
    ax.plot([0, 3*e_*v_[0]], [0, 3*e_*v_[1]], 'r-', lw=2)
ax.axis([-3,3,-3,3]);
ax.set_aspect(1)

```

### Using SVD for PCA

SVD is a decomposition of the data matrix $X = U S V^T$ where $U$ and $V$ are orthogonal matrices and $S$ is a diagonal matrix.

Recall that the transpose of an orthogonal matrix is also its inverse, so if we multiply on the right by $X^T$, we get the following simplification

\begin{align}
X &= U S V^T \\
X X^T &= U S V^T (U S V^T)^T \\
 &= U S V^T V S U^T \\
 &= U S^2 U^T
\end{align}

Compare with the eigendecomposition of a matrix $A = W \Lambda W^{-1}$, we see that SVD gives us the eigendecomposition of the matrix $XX^T$, which as we have just seen, is basically a scaled version of the covariance for a data matrix with zero mean, with the eigenvectors given by $U$ and eigenvalues by $S^2$ (scaled by $n-1$)..

```python
u, s, v = np.linalg.svd(x)
```

```python
# reproduce previous results yet again!
e2 = s**2/(n-1)
v2 = u
plt.scatter(x[0,:], x[1,:], alpha=0.2)
for e_, v_ in zip(e2, v2):
    plt.plot([0, 3*e_*v_[0]], [0, 3*e_*v_[1]], 'r-', lw=2)
plt.axis([-3,3,-3,3]);
```

```python
v1 # from eigenvectors of covariance matrix
```

```python
v2 # from SVD
```

```python
e1 # from eigenvalues of covariance matrix
```

```python
e2 # from SVD
```

```python

```

## Exercises: covariance matrix manipulations in Python (taken from the Duke course)

Given the following covariance matrix
```python
A = np.array([[2,1],[1,4]])
```
use Python to do these basic tasks (that is, do not do them by hand but use `scipy.linalg` functions).

1. Show that the eigenvectors of $A$ are orthogonal.
1. What is the vector representing the first principal component direction?
1. Find $A^{-1}$ without performing a matrix inversion.
1. What are the coordinates of the data points (0, 1) and (1, 1) in the standard basis expressed as coordinates of the principal components?
1. What is the proportion of variance explained if we keep only the projection onto the first principal component?

We'll give you a headstart on the Python manipulations (you should take a look at the `scipy.linalg` documentation).

```python
A = np.array([[2,1],[1,4]])
eigval, eigvec = la.eig(A)
```

```python

```

- Find the matrix $A$ that results in rotating the standard vectors in $\mathbb{R}^2$ by 30 degrees counter-clockwise and stretches $e_1$ by a factor of 3 and contracts $e_2$ by a factor of $0.5$.
- What is the inverse of this matrix? How you find the inverse should reflect your understanding.

The effects of the matrix $A$ and $A^{-1}$ are shown in the figure below:

```python

```

We observe some data points $(x_i, y_i)$, and believe that an appropriate model for the data is that

$$
f(x) = ax^2 + bx^3 + c\sin{x}
$$

with some added noise. Find optimal values of the parameters $\beta = (a, b, c)$ that minimize $\Vert y - f(x) \Vert^2$

1. using `scipy.linalg.lstsq`
2. solving the normal equations $X^TX \beta = X^Ty$
3. using `scipy.linalg.svd`

In each case, plot the data and fitted curve using `matplotlib`.

Data
```
x = array([ 3.4027718 ,  4.29209002,  5.88176277,  6.3465969 ,  7.21397852,
        8.26972154, 10.27244608, 10.44703778, 10.79203455, 14.71146298])
y = array([ 25.54026428,  29.4558919 ,  58.50315846,  70.24957254,
        90.55155435, 100.56372833,  91.83189927,  90.41536733,
        90.43103028,  23.0719842 ])
```

```python
x = np.array([ 3.4027718 ,  4.29209002,  5.88176277,  6.3465969 ,  7.21397852,
        8.26972154, 10.27244608, 10.44703778, 10.79203455, 14.71146298])
y = np.array([ 25.54026428,  29.4558919 ,  58.50315846,  70.24957254,
        90.55155435, 100.56372833,  91.83189927,  90.41536733,
        90.43103028,  23.0719842 ])
```

```python

```


## Inference over a BN using Belief Propagation

Consider the following joint probability distribution:
\begin{align}
p(x_1,x2,x3)=p(x_1)p(x_2|x_1)p(x_3|x_2)p(x_4|x_2)
\end{align}

where $x_1\in\{0,1\}$, $x_2\in\{0,1\}$,$x_3\in\{0,1,2\}$ and $x_4\in\{0,1,2,3\}$

The values of the CPD factors are know. Here we simply worry about inference. The goal is to learn how to use and interpret the provided library.

```python
import numpy as np
from lib import Message_passing_BN as mp
import warnings
warnings.filterwarnings('ignore')
from IPython.display import Image

#p(x1)
CPD_1=np.array([0.5, 0.5])

#p(x2|x1)
CPD_2=np.array([0.75, 0.25, 0.3, 0.7])

#p(x3|x2)
CPD_3=np.array([0.15, 0.8,0.05,0.1,0.3,0.6])

#p(x4|x2)
CPD_4=np.array([0.05,0.8,0.05,0.1,0.2,0.2,0.5,0.1])
```

A CPD of the form $p(x_c|x_a,x_b)$ is a table of $c_c \times c_a \times c_b$ entries, where $c_j$ is the cardinality of the R.V. $x_j$. We store the complete CPD in a vector of $c_c \times c_a \times c_b$ and we use a tree encoding to associate each position of the vector with a configuration of $(x_c,x_a,x_b)$.

For instance, assume they are binary variables. For the tree encoding, we assume a given encoding order. In our case $x_a\rightarrow x_b\rightarrow x_c\rightarrow x_i$. The following plot illustrates the encoding.

```python
Image(filename='files/tree.png')
```

** We provide two functions for easy encoding/decoding in CPD tables**

Let use the function mp.CPD_position_to_variable_index to find the encoding for each position of CPD_3, which stores the values $p(x_3|x_2)$. The sintax of the function is

``var_indexes=mp.CPD_position_to_variable_index(position,v_card,CPD_length)``

where v_card is the vector of cardinalities **in the encoding order**. In this case $x_2 \rightarrow x_3$ and thus `v_card=[2,3]`.

Similarly the function variable_index_to_CPD_position returns the CPD position for a given configuration of the variables. The sintax of the function is

``CPD_position=variable_index_to_CPD_position(var_index,v_card,CPD_size)``

where var_index is the variable array of indexes.

```python
values_x1=[0,1]           #Note that the actual values of the variables are irrelevant!
values_x2=[0,1]           #All the information is in the CPDs! We just care about indexes
values_x3=[0,1,2]
values_x4=[0,1,2,3]

for position,cpd_val in enumerate(CPD_3):

    var_index=mp.CPD_position_to_variable_index(position,[2,3],6)

    print("Position %d of CPD_3 correspond to p(x_3=%s|x_2=%d)\n"
          %(position,values_x3[var_index[1]],values_x2[var_index[0]]))

print("The CPD position corresponding to p(x3=%d|x2=%d) is %d\n"
     %(values_x3[2],values_x2[1],mp.variable_index_to_CPD_position([1,2],[2,3],6)))
```

    Position 0 of CPD_3 correspond to p(x_3=0|x_2=0)

    Position 1 of CPD_3 correspond to p(x_3=1|x_2=0)

    Position 2 of CPD_3 correspond to p(x_3=2|x_2=0)

    Position 3 of CPD_3 correspond to p(x_3=0|x_2=1)

    Position 4 of CPD_3 correspond to p(x_3=1|x_2=1)

    Position 5 of CPD_3 correspond to p(x_3=2|x_2=1)

    The CPD position corresponding to p(x3=2|x2=1) is 5

### Defining the factor graph associated to the BN

Before running inference using BP, we will define the variable nodes and the graph. The factor graph associated to the BN is shown below

```python
Image(filename='files/BN1.png')
```

```python
v_card=[2,2,3,4]         #Vector of cardinalities

node_x1=mp.create_var_node(ID=1,cardinality=v_card[0],neighbor_order=[1,2],observed_value_index=-1)
node_x2=mp.create_var_node(ID=2,cardinality=v_card[1],neighbor_order=[2,3,4],observed_value_index=-1)
node_x3=mp.create_var_node(ID=3,cardinality=v_card[2],neighbor_order=[3],observed_value_index=-1)
node_x4=mp.create_var_node(ID=4,cardinality=v_card[3],neighbor_order=[4],observed_value_index=-1)

list_var_nodes=[node_x1,node_x2,node_x3,node_x4]

factor_1=mp.create_factor_node(ID=1,neighbors=[node_x1],CPD=CPD_1)
factor_2=mp.create_factor_node(ID=2,neighbors=[node_x1,node_x2],CPD=CPD_2)
factor_3=mp.create_factor_node(ID=3,neighbors=[node_x2,node_x3],CPD=CPD_3)
factor_4=mp.create_factor_node(ID=4,neighbors=[node_x2,node_x4],CPD=CPD_4)

list_factor_nodes=[factor_1,factor_2,factor_3,factor_4]
```

### Running BP message passing

This is straightforward given the functions provided. As the graph is a tree, we know the algorithm will converge in a few iterations

```python
L=5 #BP iterations

for l in range(L):

    #Factor update
    for factor_node in list_factor_nodes:
        mp.update_factor_to_var(factor_node)

    #Variable update
    for var_node in list_var_nodes:
        mp.update_var_to_factor(var_node)

list_marg=[]

for var_node in list_var_nodes:
    marg=mp.compute_var_marginal(var_node)
    list_marg.append(marg)
    print("The marginal pmf of node x_%d is=" %(var_node['ID']))
    print(marg)

```

    The marginal pmf of node x_1 is=
    [[ 0.5]
     [ 0.5]]
    The marginal pmf of node x_2 is=
    [[ 0.525]
     [ 0.475]]
    The marginal pmf of node x_3 is=
    [[ 0.12625]
     [ 0.5625 ]
     [ 0.31125]]
    The marginal pmf of node x_4 is=
    [[ 0.12125]
     [ 0.515  ]
     [ 0.26375]
     [ 0.1    ]]

### Inference with partial observations

Imagine we observe x_3 and we are interested in computing the marginal distribution of the rest of variables. This is straightforward to implement.

```python
# The function mp.initialize_variable set the value to their current observed state

mp.initialize_variable(var_node=node_x3,observed_value_index=0)

# Equivalently we can define again the variable_node

#node_x3=mp.create_var_node(ID=3,cardinality=v_card[2],neighbor_order=[3],observed_value_index=1)

#IMPORTANT, we have to re-initialize all factor nodes!! Intermediate messages were not erased from the previous BP

mp.initialize_factor_msgs(factor_1,neighbors=[node_x1])
mp.initialize_factor_msgs(factor_2,neighbors=[node_x1,node_x2])
mp.initialize_factor_msgs(factor_3,neighbors=[node_x2,node_x3])
mp.initialize_factor_msgs(factor_4,neighbors=[node_x2,node_x4])
```

And now we can run BP again ...

```python
L=5 #BP iterations

for l in range(L):

    #Factor update
    for factor_node in list_factor_nodes:
        mp.update_factor_to_var(factor_node)

    #Variable update
    for var_node in list_var_nodes:
        mp.update_var_to_factor(var_node)

list_marg=[]

for var_node in list_var_nodes:
    marg=mp.compute_var_marginal(var_node)
    list_marg.append(marg)
    print("The marginal pmf of node x_%d is=" %(var_node['ID']))
    print(marg)
```

    The marginal pmf of node x_1 is=
    [[ 0.54455446]
     [ 0.45544554]]
    The marginal pmf of node x_2 is=
    [[ 0.62376238]
     [ 0.37623762]]
    The marginal pmf of node x_3 is=
    [[  1.00000000e+00]
     [  6.16587561e-87]
     [  3.41178451e-87]]
    The marginal pmf of node x_4 is=
    [[ 0.125]
     [ 0.5  ]
     [ 0.275]
     [ 0.1  ]]

### Inference of a joint distribution

One of the reasons not to erase intermediate BP messages after the algorithm finishes it that they can be used to compute joint probability distributions. In general, to compute the joint probability of two variable nodes $x_i,x_j$ connected to a shared factor node $f$, we have to multiply and store accordingly the factor $f$ and the incoming messages to the joint supernode $(x_i,x_j)$. In the plot we illustrate the computation of the joint pdf $(x_2,x_4)$.

```python
Image(filename='files/BN1-2.png')
```

```python
# Computation of the joint pdf. Store in a table using the same tree-factorization than the used for CPDs!

msg_IN=factor_4['input_msgs'][0]['table'].reshape(-1,)

marg24=np.zeros(CPD_4.shape)

for position,cpd_value in enumerate(CPD_4):

    #Store order x_2-->x4

    indexes=mp.CPD_position_to_variable_index(position,[v_card[1],v_card[3]],v_card[1]*v_card[3])

    marg24[position]+=cpd_value*np.exp(msg_IN[indexes[0]])  #Msgs defined in the logaritmic domain for numerical stability

#We have to normalize!

marg24/=np.sum(marg24)

print("The joint marginal of nodes x_2 and x_4 is=")
print(marg24)

```

    The joint marginal of nodes x_2 and x_4 is=
    [ 0.025  0.4    0.025  0.05   0.1    0.1    0.25   0.05 ]

```python

```

Notebooks/First Example Message Passing.ipynb


Computational Seismology

Numerical first derivative

---

This notebook is part of the supplementary material
to [Computational Seismology: A Practical Introduction](https://global.oup.com/academic/product/computational-seismology-9780198717416?cc=de&lang=en&#),
Oxford University Press, 2016.

---

##### Authors:
* Kristina Garina
* Heiner Igel ([@heinerigel](https://github.com/heinerigel))

---

### Excercise:
Initialise arbitrary analytical function and calculate numerical first derivative with 2-point operator, compare with analytical solution. **Task:** extend scheme to 4 point operator and demonstrate improvement **(see Chapter 4)**

---

This exercise covers the following aspects:

* Calculation of numerical first derivative
* Comparison with analytical solution

---

**Please, execute first!**

```
# Import Libraries
%matplotlib notebook
import numpy as np
import math
import matplotlib.pyplot as plt

plt.style.use("ggplot")
```

We initialise a Gaussian function

\begin{equation}
f(t)=\dfrac{1}{\sqrt{2 \pi a}}e^{-\dfrac{t^2}{2a}}
\end{equation}

```
# Initial parameters
tmax=10.0                       # maximum time
nt=100                       # number of time sample
a=1                           # half-width
dt=tmax/nt                    # defining dt
t0 = tmax/2                   # defining t0

time=np.linspace(0,tmax,nt)   # defining time

# Initialization gaussian function with zeros
f=(1./np.sqrt(2*np.pi*a))*np.exp(-(((time-t0)**2)/(2*a)))  # eq.(4.32) p. 80

# Plotting of gaussian
plt.figure()
plt.plot(time, f)
plt.title('Gaussian function')
plt.xlabel('Time, s')
plt.ylabel('Amplitude')
plt.xlim((0, tmax))
plt.grid()
plt.show()
```

In the cell below we calculate numerical derivative using two points

\begin{equation}
f^{\prime}(t)=\dfrac{f(t+dt)-f(t-dt)}{2dt}
\end{equation}

and analytical derivative
\begin{equation}
f^{\prime}(t)=-\dfrac{t}{a}\dfrac{1}{\sqrt{2\pi a}}e^{-\dfrac{t^2}{2a}}
\end{equation}

```
# First derivative with two points

# Initiation of numerical and analytical derivatives
nder=np.zeros(nt)          # numerical derivative
ader=np.zeros(nt)          # analytical derivative

# Numerical derivative of the given function
for it in range (1, nt-1):
    nder[it]=(f[it+1]-f[it-1])/(2*dt)

# Analytical derivative of the given function
ader=(-(time-t0)/a)*(1/np.sqrt(2*np.pi*a))*np.exp(-(time-t0)**2/(2*a))

# Plot of the first derivative and analytical derivative
plt.figure()
plt.plot (time, nder,label="Numerical derivative", lw=2, color="yellow")
plt.plot (time, ader, label="Analytical derivative", ls="--",lw=2, color="red")
plt.title('First derivative')
plt.xlabel('Time, s')
plt.ylabel('Amplitude')
plt.legend()
plt.grid()
plt.show()
```

In the cell below calculation of the first derivative with four points is provided with the following weights:

\begin{equation}
f^{\prime}(t)=\dfrac{\dfrac{1}{12}f(t-2dt)-\dfrac{2}{3}f(t-dt)+\dfrac{2}{3}f(t+dt)-\dfrac{1}{12}f(t+2dt)}{dt}
\end{equation}

```
# First derivative with four points

# Initiation of derivative
ffder=np.zeros(nt)

# Type your code here:
for it in range (2, nt-2):
    #ffder[it]=

# Plotting
plt.figure()
plt.plot (time, nder,label="Derivative, 2 points", lw=2, color="violet")
plt.plot (time, ffder, label="Derivative, 4 points", lw=2, ls="--")
plt.title('First derivative')
plt.xlabel('Time, s')
plt.ylabel('Amplitude')
plt.legend()
plt.grid()
plt.show()
```

```

```

notebooks/Computational Seismology/The Finite-Difference Method/fd_first_derivative.ipynb


Jonah M. Miller, Scientist, CCS-2, Los Alamos  National Lab

# Introduction

In this brief tutorial,  I will describe the simplest way of modeling a deadly infectious disease, such as COVID-19.  I will show that at early times, t he disease grows exponentially, but at late times, falls off like a Bell curve. I will solve the equations numerically with Python

A common mathematical model is the so-called [compartmental model](https://en.wikipedia.org/wiki/Compartmental_models_in_epidemiology), where diseases can move people between "categories" such as healthy, infectious, or truly sick. This is a large, well established field and there are many good resources on this topic. I started with the SIAM review article [The mathematics of infectious diseases](https://epubs.siam.org/doi/10.1137/S0036144500371907) (Hethcote, 2000). However, there are many excellent textbooks and more  modern reviews as well.

```python
# Import python modules we need
%matplotlib inline
import numpy as np
import scipy as sp
import matplotlib as  mpl
from matplotlib import pyplot as plt
mpl.rc('font',size=16)
```

# The SIR Model

We will use the SIR model, where the categories are:
- **S**usceptible
- **I**nfectious
- **R**ecovered

This model was first proposed by Kermack and McKendrick in their [seminal work](https://royalsocietypublishing.org/doi/10.1098/rspa.1927.0118), now known as Kermack-McKendrick theory.

We use a system of differential equations to model the number of people in each category. First some definitions.  We use the variables $S$, $I$, and $R$ to represent the number of people in each category. The total number of people is $N$. Therefore, the sum of people over all three categories must equal the total population:
$$S + I + R = N$$
In general, $S$, $I$, $R$, and $N$ can all be functions of time and space. An important assumption of this model is that **people who recover from the disease are immune and can't carry it or be re-infected.** This appears to be true for COVID-19 but it is not always true.

For simplicity, we assume that they are a function only of time, which we will call $t$. This allows us to treat the problem with ordinary, rather than partial, differential equations. Further, we will begin our analysis by assuming there are no births or deaths and $N$ is a constant. By the end of the tutorial, however, we will move beyond this simplistic assumption.

## No Deaths

Assuming no births or deaths, our equations are:
\begin{align}
\frac{dS}{dt} &= - \beta \frac{S I}{N}\\
\frac{dI}{dt} &= \beta \frac{S I}{N} - \gamma I\\
\frac{dR}{dt} &= \gamma I
\end{align}

The symbols here are:
- $\beta$ is the infection rate. An infected individual infects $\beta N$ other individuals per unit time.
- $\gamma$ is the average recovery rate such that $1/\gamma$ is the average time it takes  for an individual to recover  from the disease

A common metric of a disease the *power of infection*, which is the ratio of $\beta$ and $\gamma$:
$$R_0 = \frac{\beta}{\gamma}$$

These equations are often simplified by working with the *fractions* of the population that are  infected, susceptible, etc. These are denoted with lower-case variables
\begin{align}
s &= S/N\\
i &= I/N\\
r &= R/N
\end{align}

so that the SIR equations become

\begin{align}
\frac{ds}{dt} &= - \beta s i\\
\frac{di}{dt} &= \beta s i - \gamma i\\
\frac{dr}{dt} &= \gamma i
\end{align}

### The values  of the parameters

Let's try this out and see if we can model COVID-19. We will work in units of days. So, days to recovery, for example, is $1/\gamma$. The official time to self-quarantine for COVID-19 is 14 days, so I will set
$$\gamma = 1/14.$$
We will set $\beta$ as
$$\beta  = \gamma R_0$$
The infectious power of the disease is between 2 and 5. It depends on countermeasures implemented. i.e., $R_0 = 5$ if nothing is done, and it can be as low as 2 if people are careful. Let's encode that in Python:

```python
def get_beta_gamma(R=5,recovery_time=14):
    """Returns beta and gamma based
    on recovery time and infectious power R
    """
    gamma = 1./recovery_time
    beta = R*gamma
    return beta,gamma

beta,gamma = get_beta_gamma()
print("beta = {}\ngamma = {}".format(beta,gamma))
```

    beta = 0.3571428571428571
    gamma = 0.07142857142857142

Our differential conditions need initial data. At the beginning, there has to be at least one infected person, since the disease has to start somewhere. In the early stages of the infection in the United States, there were 10s of cases, out of 327 million. That's something like
$$i(t=0) = 10^{-8}.$$
We'll assume
$$r(t=0) = 0$$
to start, and that
$$s(t=0) = 1 - i(t=0).$$
Let's encode that in Python:

```python
i0 = 1e-8
r0 = 0
s0 = 1 - i0
```

Let's try solving this. We will use scipy's ODE integrator to solve the equations. It solves vector equations of the form
$$\frac{dU}{dt} = f(t,U)$$
in our case
$$U = \left(s, i, r\right)$$
and
$$f(t,U) = \left(-\beta s i, \beta s i-\gamma s i, \gamma i\right)$$
We'll encode the solution as a two-dimensional numpy array, where the row is time, and each column is $U$.

```python
num_times = 180 # simulate for half a year
times = np.arange(num_times)
Us = np.empty((num_times,3))
Us[0] = np.array([s0,i0,r0])
```

We'll encode the right-hand-side $f$ as a Python function

```python
def rhs(t, U, beta, gamma):
    s,i,r = U[0],U[1],U[2]
    return np.array([-beta*s*i,beta*s*i-gamma*i,gamma*i])
```

Let's load up the integrator code and set the initial conditions and parameters $\beta$ and $\gamma$

```python
# import the tool
from scipy.integrate  import ode
# create the integrator
integrator = ode(rhs)
# set initial conditions and parameters
beta,gamma = get_beta_gamma(R=5,recovery_time=14)
integrator.set_initial_value(Us[0],0)
integrator.set_f_params(beta,gamma)
```

    <scipy.integrate._ode.ode at 0x7f9e8b3a9710>

And let's integrate!

```python
for i in range(len(times)-1):
    t = times[i+1]
    integrator.integrate(t)
    Us[i+1] = integrator.y
```

Let's see what it looks like. We can get s, i, and r as the columns of matrix:

```python
s,i,r = Us[:,0],Us[:,1],Us[:,2]
```

And let's plot:

```python
plt.plot(times,s,label='susceptible')
plt.plot(times,i,label='infected')
plt.plot(times,r,label='recovered')
plt.legend()
plt.xlabel('time (days)')
plt.ylabel('fraction of total population')
```

You can see the "bell curve" people talk about in the news, which we want to flatten by reducing the infectious power of the disease. Making $R_0$ smaller will accomplish that. (Try it yourself! Change $R_0$ above. Try setting it to 3 or 4. You may also need to play with the duration of the simulation.) Something terrifying you should note. **In the worst case scenario, everyone gets the disease and half the population gets it at once.** That would completely overwhelm a country's medical system. Which we currently see happening in real time in Italy.

It's informative to look at this plot with the y-axis on a log scale:

```python
plt.semilogy(times,s,label='susceptible')
plt.plot(times,i,label='infected')
plt.plot(times,r,label='recovered')
plt.legend()
plt.xlabel('time (days)')
plt.ylabel('fraction of total population')
```

While the population is susceptible to the disease, the number of infected cases grows exponentially. After a critical number of people are infected and begin to recover, the number of cases *decays* exponentially. This exponential growth is the reason why things can get out of hand so quickly. Monday is normal. On Friday, everyone is buying all the toilet paper. Changing the infectious power of the disease changes the rate of exponential growth, which is something we want.

## Including Deaths: The SIRD Model

Up until now, we have assumed that the disease is non-fatal. What about a fatal disease? We include this by adding a new category, $D$ for deceased, with
$$d = D/N$$
Now $N$ is the the population at the beginning of the outbreak. We also add a new rate,
$$\sigma,$$
which is the case-fatality rate---the number of infected people who die per unit time. Our fraction equations now become:
\begin{align}
\frac{ds}{dt} &= - \beta s i\\
\frac{di}{dt} &= \beta s i - (\gamma + \sigma) i \\
\frac{dr}{dt} &= \gamma i\\
\frac{dd}{dt} &= \sigma i
\end{align}
We can relate $\sigma$ to $\gamma$ as
$$\sigma = R_1 \gamma,$$
where $R_1$ is the fraction of people who have the disease who die. For COVID-19, that's bout 3%. Or:
$$\sigma \approx 3\gamma/100$$
Let's code  it up.

```python
R1 = 3./100.
```

To start, nobody has died yet.

```python
d0 = 0
```

New initial conditions:

```python
num_times = 180 # simulate for half a year
times = np.arange(num_times)
Us = np.empty((num_times,4))
Us[0] = np.array([s0,i0,r0,d0])
```

New right-hand-side:

```python
def rhs(t, U, beta, gamma, sigma):
    s,i,r,d = U[0],U[1],U[2],U[3]
    return np.array([-beta*s*i,
                     beta*s*i-(gamma+sigma)*i,
                     gamma*i,
                     sigma*i
                    ])
```

Set up the integrator

```python
integrator = ode(rhs)
# set initial conditions and parameters
beta,gamma = get_beta_gamma(R=3.4,recovery_time=14)
sigma = R1*gamma
integrator.set_initial_value(Us[0],0)
integrator.set_f_params(beta,gamma,sigma)
```

    <scipy.integrate._ode.ode at 0x7f9e8aab6e10>

Integrate

```python
for i in range(len(times)-1):
    t = times[i+1]
    integrator.integrate(t)
    Us[i+1] = integrator.y
```

And plot:

```python
s,i,r,d = Us[:,0],Us[:,1],Us[:,2],Us[:,3]
```

```python
plt.plot(times,s,label='susceptible')
plt.plot(times,i,label='infected')
plt.plot(times,r,label='recovered')
plt.plot(times,d,label='dead')
plt.legend(bbox_to_anchor=[0.985,0.7])
plt.xlabel('time (days)')
plt.ylabel('fraction of total population')
```

```python
plt.semilogy(times,s,label='susceptible')
plt.plot(times,i,label='infected')
plt.plot(times,r,label='recovered')
plt.plot(times,d,label='dead')
plt.legend(bbox_to_anchor=[0.985,0.7])
plt.xlabel('time (days)')
plt.ylabel('fraction of total population')
```

The story roughly the same. But now that red line reflects people who never recover.

# Limitations

There are of course significant limitations to this toy model. We neglect:
- Births
- How the disease spreads or does not spread throughout space. Social distancing will help!
- How the global health care systems can react or fail to react as cases grow
- How different people and different age groups are more or less susceptible to the disease

So take all this analysis with a grain of salt. But it helped me understand at least a little bit of what's going on and build a better intuition for what's happening right now.

# Further reading

Here are some resources you may find helpful as you look at epidemiology and disease modeling:

- [Triplebyte's Tutorial on disease modeling](https://triplebyte.com/blog/modeling-infectious-diseases)
- [Kai Sasaki's ML fit of COVID-19 cases to the analytic solution of the SIR model](https://www.lewuathe.com/covid-19-dynamics-with-sir-model.html)
- [SciPy's SIR example](https://scipython.com/book/chapter-8-scipy/additional-examples/the-sir-epidemic-model/)
- [The Mathematics of Infectious Diseases, Hethcote, 2000](https://epubs.siam.org/doi/10.1137/S0036144500371907)
- [A Contribution to the Mathemat of Epidemics, Kermack and McKendrick, 1927](https://royalsocietypublishing.org/doi/10.1098/rspa.1927.0118)
- [Compartmental Models in Epidemiology](https://link.springer.com/chapter/10.1007/978-3-540-78911-6_2)

```python

```

The Differential Equations That Model Diseases.ipynb


# Python and Jupyter notebooks: part 01

Last revised: 01-Jan-2020 by Dick Furnstahl [furnstahl.1@osu.edu]

This is a whirlwind tour of just the minimum we need to know about Python and Jupyter notebooks to get started doing mechanics problems.  We'll add more features and details as we proceed.

A Jupyter notebook is displayed on a web browser on a computer, tablet (e.g., IPad), or even your smartphone.  The notebook is divided into *cells*, of which two types are relevant for us:
* Markdown cells: These have headings, text, and mathematical formulas in $\LaTeX$ using a simple form of HTML (language of webpages) called markdown.
* Code cells: These have Python code (or other languages, but we'll stick to Python).

Either type of cell can be selected with your cursor and will be highlighted in color when active.  You evaluate an active cell with shift-return (as with Mathematica) or by pressing `Run` on the toolbar.  Some notes:
* When a new cell is inserted, by default it is a Code cell and will have `In []:` in front.  You can type Python expressions or entire programs in a cell.  How you break up code between cells is your choice and you can always put Markdown cells in between.  When you evaluate a cell it gets the next number, e.g., `In [5]:`.
* On the menu bar is a pulldown menu that lets you change back and forth between Code and Markdown cells.  Once you evaluate a Markdown cell, it gets formatted (and has a blue border).  To edit the Markdown cell, double click in it.

**Try double-clicking on this cell and then shift-return.**  You will see that a bullet list is created just with an asterisk and a space at the beginning of lines (without the space you get *italics* and with two asterisks you get **bold**).  **Double click on the title header above and you'll see it starts with a single #.**  Headings of subsections are made using ## or ###.  See this [Markdown cheatsheet](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet) for a quick tour of the Markdown language (including how to add links!).

**Now try turning the next (empty) cell to a Markdown cell and type:** `Einstein says $E=mc^2$` **and then evaluate it.**  This is $\LaTeX$! (If you forget to convert to Markdown and get `SyntaxError: invalid syntax`, just select the cell and convert to Markdown with the menu.)

```python

```

The menus enable you to rename your notebook file (always ending in `.ipynb`) or `Save and Checkpoint` to save the changes to your notebook.  You can insert and delete cells (use the up and down arrows in the toolbar to easily move cells).  You will often use the `Kernel` menu to `Restart` the notebook (and possibly clear output).

## Ok, time to try out Python expressions and numpy

We can use the Jupyter notebook as a super calculator much like Mathematica and Matlab.  **Try some basic operations, modifying and evaluating the following cells, noting that exponentiation is with** `**` **and not** `^`.

```python
 1 + 1  # Everything after a number sign / pound sign / hashtag)
        #  is a comment
```

    2

```python
3.2 * 4.713
```

    15.081600000000002

If we want a floating point number (which will be the same as a `double` in C++), we *always* include a decimal point (even when we don't have to) while a number without a decimal point is an integer. (Note: in the Python we use there is no problem with 1/2 being evaluated as zero, as in C++, but we will still keep the convention of including decimal points.)

```python
3.**2
```

    9.0

We can define integer, floating point, and string variables, perform operations on them, and print them.  Note that we don't have to predefine the type of a variable (unlike C++) and we can use underscores in the names (unlike Mathematica).  **Evaluate the following cells and then try your own versions.**

```python
x = 5.
print(x)
x   # If the last line of a cell returns a value, it is returned as output.
```

    5.0

    5.0

```python
y = 3.*x**2 - 2.*x + 7.
print('y = ', y)           # Strings delimited by ' 's (or " "s)
print(f'y = {y:.0f}')      # Just a preview: more on formatting later
print(f'y = {y:.2f}')      #  (note: this uses an "fstring")
```

    y =  72.0
    y = 72
    y = 72.00

```python
first_name = 'Dick'     # Strings delimited by ' 's
last_name = 'Furnstahl'
full_name = first_name + ' ' + last_name  # you can concatenate strings
print(full_name)
```

    Dick Furnstahl

Ok, how about square roots and trigonometric functions and ...

```python
ans = np.sqrt(2)
print(f'The answer is {ans: .4f}')
```

    The answer is  1.4142

```python
np.sin(np.pi)
```

    1.2246467991473532e-16

We need to `import` these functions through the numpy library. There are other choices, but numpy works with the arrays we will use.  Note: *Never* use `from numpy import *` instead of `import numpy as np`.  Here `np` is just a abbreviation for numpy (which we can choose to be anything, but `np` is conventional).

```python
import numpy as np
```

```python
print(np.cos(0.))
```

    1.0

Now functions and constants like `np.sqrt` and `np.pi` will work.  **Go back and fix the square root and sine.**

### Debugging aside . . .

Suppose you try to import and it fails (**go ahead and evaluate the cell**):

```python
import numpy
```

When you get a `ModuleNotFoundError`, the first thing to check is whether you have misspelled the name. Try using Google, e.g., search for "python numpie". In this case (and in most others), Google will suggest the correct name (here it is numpy).  If the name does exist, check whether it sounds like the package you wanted.

If you have the correct spelling, check whether you have installed the relevant package.  If you installed Python with Anaconda (which we will assume you did -- if not, do it!), then use `conda list`, e.g., `conda list numpy` in a Terminal window (on a Mac or Linux box) or in an Anaconda Prompt window (on a Windows PC).

### numpy arrays

At first, we will most often use numpy arrays so we'll start with those.  They are *like* lists delimited by square brackets, i.e., `[]`s, and we will construct them with `np.arange(min, max, step)` to get an array from `min` to `max` in steps of `step`. Examples:

```python
t_pts = np.arange(0., 10., .1)
t_pts
```

    array([0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. , 1.1, 1.2,
           1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2. , 2.1, 2.2, 2.3, 2.4, 2.5,
           2.6, 2.7, 2.8, 2.9, 3. , 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8,
           3.9, 4. , 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7, 4.8, 4.9, 5. , 5.1,
           5.2, 5.3, 5.4, 5.5, 5.6, 5.7, 5.8, 5.9, 6. , 6.1, 6.2, 6.3, 6.4,
           6.5, 6.6, 6.7, 6.8, 6.9, 7. , 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7,
           7.8, 7.9, 8. , 8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.7, 8.8, 8.9, 9. ,
           9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 9.7, 9.8, 9.9])

If we give a numpy array to a function, each term in the list is evaluated with that function:

```python
x = np.arange(1., 5., 1.)
print(x)
print(x**2)
print(np.sqrt(x))
```

    [1. 2. 3. 4.]
    [ 1.  4.  9. 16.]
    [1.         1.41421356 1.73205081 2.        ]

We can pick out elements of the list.  **Why does the last one fail?**

```python
# List index is out range.
# List elements begin with indices at 0.
print(x[0])
print(x[3])
print(x[2])
```

    1.0
    4.0
    3.0

An alternative to `np.arange` is `np.linspace(min, max, number)` to get an array from `min` to `max` with `number` elements.  Example:

```python
u_pts = np.linspace(0., 10., 10)
u_pts
```

    array([ 0.        ,  1.11111111,  2.22222222,  3.33333333,  4.44444444,
            5.55555556,  6.66666667,  7.77777778,  8.88888889, 10.        ])

**Change the last statement for `u_pts` to get an array from 0 to 10 spaced by 1.**

## Getting help

You will often need help identifying the appropriate Python (or numpy or scipy or ...) command or you will need an example of how to do something or you may get an error message you can't figure out.  In all of these cases, Google (or equivalent) is your friend. Always include "python" in the search string (or "numpy" or "matplotlib" or ...) to avoid getting results for a different language. You will usually get an online manual as one of the first responses if you ask about a function; these usually have examples if you scroll down. Otherwise, answers from *Stack Overflow* queries are your best bet to find a useful answer.

## Functions

There are many Python language features that we will use eventually, but in the short term what we need first are functions.  Here we first see the role of *indentation* in Python in place of {}s or ()s in other languages.  We'll always indent four spaces (never tabs!).  We know a function definition is complete when the indentation stops.

To find out about a Python function or one you define, put your cursor on the function name and hit shift+Tab+Tab. **Go back and try it on `np.arange`.**

```python
# Use "def" to create new functions.
#  Note the colon and indentation (4 spaces).
def my_function(x):
    """This function squares the input.  Always include a brief description
        at the top between three starting and three ending quotes.  We will
        talk more about proper documentation later.
        Try shift+Tab+Tab after you have evaluated this function.
    """
    return x**2

print(my_function(5.))

# We can pass an array to the function and it is evaluated term-by-term.
x_pts = np.arange(1., 10., 1.)
print(my_function(x_pts))
```

    25.0
    [ 1.  4.  9. 16. 25. 36. 49. 64. 81.]

```python
# Two variables, with a default for the second
def add(x, y=4.):
    """Add two numbers."""
    print("x is {} and y is {}".format(x, y))
    return x + y  # Return values with a return statement

# Calling functions with parameters
print('The sum is ', add(5, 6))  # => prints out "x is 5 and y is 6" and returns 11

# Another way to call functions is with keyword arguments
add(y=6, x=5)  # Keyword arguments can arrive in any order.

```

    x is 5 and y is 6
    The sum is  11
    x is 5 and y is 6

    11

**How do you explain the following result?**

```python
add(2)
```

    x is 2 and y is 4.0

    6.0

### Debugging aside . . .

There are two bugs in the following function.  **Note the line where an error is first reported and fix the bugs sequentially (so you see the different error messages).**

```python
def hello_function():
    msg = "hello, world!"
    print(msg)
    return msg
```

## Plotting with Matplotlib

Matplotlib is the plotting library we'll use, at least at first.  We'll follow convention and abbreviate the module we need as `plt`.  The `%matplotlib inline` line tells the Jupyter notebook to make inline plots (we'll see other possibilities later).

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
```

Procedure we'll use to make the skeleton plot:
0. Generate some data to plot in the form of arrays.
1. Create a figure;
2. add one or more subplots;
3. make a plot and display it.

```python
t_pts = np.arange(0., 10., .1)     # step 0.
x_pts = np.sin(t_pts)  # More often this would be from a function
                       #  *we* write.

my_fig = plt.figure()              # step 1.
my_ax = my_fig.add_subplot(1,1,1)  # step 2: rows=1, cols=1, 1st subplot
my_ax.plot(t_pts, x_pts)           # step 3: plot x vs. t
```

NOTE: When making just a single plot, you will more usually see steps 1 to 3 compressed into `plt.plot(t_pts, np.sin(t_pts))`.  *Don't do this.*  It saves a couple of lines but restricts your ability to easily extend the plot, which is what we want to make easy.

We can always go back and dress up the plot:

```python
my_fig = plt.figure()
my_ax = my_fig.add_subplot(1,1,1)  # nrows=1, ncols=1, first plot
my_ax.plot(t_pts, x_pts, color='blue', linestyle='--', label='sine')

my_ax.set_xlabel('t')
my_ax.set_ylabel(r'$\sin(t)$')  # use $...$ to get LaTeX and r to render it
my_ax.set_title('Sine wave')

# here we'll put the function in the call to plot!
my_ax.plot(t_pts, np.cos(t_pts), label='cosine')  # just label the plot

my_ax.legend();  # turn on legend (try removing the semicolon)

```

Now make two subplots:

```python
y_pts = np.exp(t_pts)         # another function for a separate plot

fig = plt.figure(figsize=(10,5))  # allow more room for two subplots

# call the first axis ax1
ax1 = fig.add_subplot(1,2,1)  # one row, two columns, first plot
ax1.plot(t_pts, x_pts, color='blue', linestyle='--', label='sine')
ax1.plot(t_pts, np.cos(t_pts), label='cosine')  # just label the plot
ax1.legend()

ax2 = fig.add_subplot(1,2,2)  # one row, two columns, second plot
ax2.plot(t_pts, np.exp(t_pts), label='exponential')
ax2.legend();

```

## Solving ordinary differential equations (ODEs)

Newton's 2nd Law for one particle:

$\begin{align}
  \mathbf{F} = m\mathbf{a} = m\frac{d\mathbf{v}}{dt} = m\frac{d^2\mathbf{x}}{dt^2}
\end{align}$

is a first-order differential equation in the velocity vector and a second-order differential equation in the position vector. Here we assume that $\mathbf{F} = \mathbf{F}(\mathbf{x}, \mathbf{v}, t)$ does not have higher derivatives. So we need to know how to solve such differential equations, usually with initial conditions (as opposed to boundary conditions).

Here is how to do it with the Scipy function `odeint`.  (Note that `solve_ivp` is now preferred to `odeint`; we'll switch to that later but it is easier to get started with `odeint`.)

### First-order ODE

```python
# Import the required modules
import numpy as np
import matplotlib.pyplot as plt

from scipy.integrate import odeint  # Get only odeint from scipy.integrate
# for the future:
from scipy.integrate import solve_ivp   # Now preferred to odeint
```

Let's try a one-dimensional first-order ODE, say:

$\begin{align}
\quad
\frac{dv}{dt} = -g, \quad \mbox{with} \quad v(0) = 10
\end{align}$

in some appropriate units (we'll use MKS units by default).  This ODE can be separated and directly integrated:

$\begin{align}
  \int_{v_0=10}^{v} dv' = - g \int_{0}^{t} dt'
  \quad\Longrightarrow\quad
    v - v_0 = - g (t - 0)
  \quad\Longrightarrow\quad
   v(t) = 10 - gt
\end{align}$

```python
# Define a function which calculates the derivative
def dv_dt(v, t, g=9.8):
    """Returns the right side of a simple first-order ODE with default g."""
    return -g

t_pts = np.linspace(0., 10., 101)     # 101 points from t=0 and t=10.
v_0 = 10.0  # the initial condition
v_pts = odeint(dv_dt, v_0, t_pts)  # odeint( function for rhs,
                                   #         initial value of v(t),
                                   #         array of t values )

```

```python

```

Let's check the output $v(t)$.  Does it make sense?

```python
print(v_pts[0:10])
v_pts.shape
```

    [[10.  ]
     [ 9.02]
     [ 8.04]
     [ 7.06]
     [ 6.08]
     [ 5.1 ]
     [ 4.12]
     [ 3.14]
     [ 2.16]
     [ 1.18]]

    (101, 1)

Now plot $v(t)$ calculated numerically and the analytic solution:

```python
g = 9.8
v_pts_exact = v_0 - g*t_pts

fig = plt.figure(figsize=(8,4))

ax = fig.add_subplot(1,2,1)
ax.plot(t_pts, v_pts, label='numerical', color='blue', lw=4)
ax.plot(t_pts, v_pts_exact, label='exact', color='red')
ax.set_xlabel('t [sec]')
ax.set_ylabel('v(t) [meters/sec]')
ax.legend();

diff = v_pts_exact - v_pts.flatten()
ax2 = fig.add_subplot(1,2,2)
ax2.semilogy(t_pts, diff, label='difference', color='red')
ax2.set_xlabel('t [sec]')
ax2.set_ylabel('v(t) [meters/sec]')
ax2.legend();

fig.tight_layout()
```

We adjusted the colors and linewidths so you could see that the lines are on top of each other. **Why are the differences around $10^{-15}$?**

Try solving instead:

$\begin{align}
\quad
\frac{dx}{dt} = -x \quad \mbox{with} \quad x(0) = 1
\end{align}$

both analytically and numerically.

```python
def dx_dt(x, t):
    """Function returns the right side of the above ODE."""
    return -x

t_pts = np.linspace(0, 10, 101)
x_0 = 1
x_pts = odeint(dx_dt, x_0, t_pts)

print(x_pts[0:10])
print(x_pts.shape)
```

    [[1.        ]
     [0.90483745]
     [0.81873077]
     [0.7408182 ]
     [0.67032006]
     [0.60653067]
     [0.54881165]
     [0.49658532]
     [0.44932898]
     [0.40656968]]
    (101, 1)

### Second-order ODE

Suppose we have a second-order ODE such as:

$$
\quad y'' + 2 y' + 2 y = \cos(2x), \quad \quad y(0) = 0, \; y'(0) = 0
$$

We can turn this into two first-order equations by defining a new dependent variable equal to $y'$ (we'll call it $z$):

$$
\quad z \equiv y' \quad \Rightarrow \quad z' + 2 z + 2y = \cos(2x)
  \quad\mbox{or}\quad z' = -2z -2y + \cos(2x), \quad\mbox{with}
\quad z(0)=y(0) = 0.
$$

Now introduce the vector

$$
  \mathbf{U}(x) = \left(\begin{array}{c}
                         y(x) \\
                         z(x)
                        \end{array}
                  \right)
        \quad\Longrightarrow\quad
    \frac{d\mathbf{U}}{dx} =
                             \left(\begin{array}{c}
                                    y' \\
                                    z'
                                   \end{array}
                             \right)
                           =
                             \left(\begin{array}{c}
                                    z \\
                                    -2 y' - 2 y + \cos(2x)
                                   \end{array}
                             \right)
              \quad\mbox{with}\quad
    \mathbf{U}(0) = \left(\begin{array}{c}
                         0 \\
                         0
                        \end{array}
                  \right)
$$

We can solve this system of ODEs using `odeint` with lists, as follows:

```python
# Define a function for the right side
def dU_dx(U, x):
    """Right side of the differential equation to be solved.
    U is a two-component vector with y=U[0] and z=U[1].
    Thus this function should return [y', z']
    """
    return [U[1], -2*U[1] - 2*U[0] + np.cos(2*x)]

# initial condition U_0 = [y(0)=0, z(0)=y'(0)=0]
U_0 = [0., 0.]

x_pts = np.linspace(0, 15, 200)  # Set up the mesh of x points
U_pts = odeint(dU_dx, U_0, x_pts)  # U_pts is a 2-dimensional array
y_pts = U_pts[:, 0]   # Ok, this is tricky.  For each x, U_pts has two
                      #  components.  We want the upper component for all
                      #  x, which is y(x).  The : means all of the first
                      #  index, which is x, and the 0 means the first
                      #  component in the other dimension.
                      # What is U_pts[:, 1]? (Try plotting it.)
```

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(x_pts, y_pts)
ax.set_xlabel('x')
ax.set_ylabel('y')
```

## Widgets!

A widget is an object such as a slider or a check box or a pulldown menu.  We can use them to make it easy to explore different parameter values in a problem we're solving, which is invaluable for building intuition.  They act on the argument of a function.  We'll look at a simple case here but plan to explore this much more as we proceed.

The set of widgets we'll use here (there are others!) is from `ipywidgets`; we'll conventionally import the module as `import ipywidgets as widgets` and we'll also often use `display` from `Ipython.display`.

```python
import numpy as np
import matplotlib.pyplot as plt

import ipywidgets as widgets
from IPython.display import display

%matplotlib inline
```

The simplest form is to use `interact`, which we pass a function name and the variables with ranges.  By default this makes a *slider*, which takes on integer or floating point values depending on whether you put decimal points in the range. **Try it! Then modify the function and try again.**

```python
# We can do this to any function
def test_f(x=5.):
    """Test function that prints the passed value and its square.
       Note that there is no return value in this case."""
    print ('x = ', x, ' and  x^2 = ', x**2)

widgets.interact(test_f, x=(0.,10.));
```

    interactive(children=(FloatSlider(value=5.0, description='x', max=10.0), Output()), _dom_classes=('widget-inte…

```python
# Explicit declaration of the widget (here FloatSlider) and details
def test_f(x=5.):
    """Test function that prints the passed value and its square.
       Note that there is no return value in this case."""
    print ('x = ', x, ' and  x^2 = ', x**2)

widgets.interact(test_f,
                 x = widgets.FloatSlider(min=-10,max=30,step=1,value=10));
```

    interactive(children=(FloatSlider(value=10.0, description='x', max=30.0, min=-10.0, step=1.0), Output()), _dom…

Here's an example with some bells and whistles for a plot.  **Try making changes!**

```python
def plot_it(freq=1., color='blue', lw=2, grid=True, xlabel='x',
            function='sin'):
    """ Make a simple plot of a trig function but allow the plot style
        to be changed as well as the function and frequency."""
    t = np.linspace(-1., +1., 1000)  # linspace(min, max, total #)

    fig = plt.figure(figsize=(8,6))
    ax = fig.add_subplot(1,1,1)

    if function=='sin':
        ax.plot(t, np.sin(2*np.pi*freq*t), lw=lw, color=color)
    elif function=='cos':
        ax.plot(t, np.cos(2*np.pi*freq*t), lw=lw, color=color)
    elif function=='tanh':
        ax.plot(t, np.tanh(2*np.pi*freq*t), lw=lw, color=color)

    ax.grid(grid)
    ax.set_xlabel(xlabel)

widgets.interact(plot_it,
                 freq=(0.1, 60.), color=['blue', 'red', 'green'],
                 lw=(1, 10), xlabel=['x', 't', 'dog'],
                 function=['sin', 'cos', 'tanh'])

```

    interactive(children=(FloatSlider(value=1.0, description='freq', max=60.0, min=0.1), Dropdown(description='col…

    <function __main__.plot_it(freq=1.0, color='blue', lw=2, grid=True, xlabel='x', function='sin')>

```python

```


# Random Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Auto-Power Spectral Density

The (auto-) [power spectral density](https://en.wikipedia.org/wiki/Spectral_density#Power_spectral_density) (PSD) is defined as the Fourier transformation of the [auto-correlation function](correlation_functions.ipynb) (ACF).

### Definition

For a continuous-amplitude, real-valued, wide-sense stationary (WSS) random signal $x[k]$ the PSD is given as

\begin{equation}
\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \mathcal{F}_* \{ \varphi_{xx}[\kappa] \},
\end{equation}

where $\mathcal{F}_* \{ \cdot \}$ denotes the [discrete-time Fourier transformation](https://en.wikipedia.org/wiki/Discrete-time_Fourier_transform) (DTFT) and $\varphi_{xx}[\kappa]$ the ACF of $x[k]$. Note that the DTFT is performed with respect to $\kappa$. The ACF of a wide-sense ergodic random signal of finite length $N$ can be expressed by way of a linear convolution

\begin{equation}
\varphi_{xx}[\kappa] = \frac{1}{N} \cdot x_N[k] * x_N[-k].
\end{equation}

Taking the DTFT of the left- and right-hand side yields

\begin{equation}
\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{1}{N} \, X_N(\mathrm{e}^{\,\mathrm{j}\,\Omega})\, X_N(\mathrm{e}^{-\,\mathrm{j}\,\Omega}) =
\frac{1}{N} \, | X_N(\mathrm{e}^{\,\mathrm{j}\,\Omega}) |^2.
\end{equation}

The last equality results from the definition of the magnitude and the symmetry of the DTFT for real-valued signals. The spectrum $X_N(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ quantifies the amplitude density of the signal $x_N[k]$. It can be concluded that the PSD quantifies the squared amplitude or power density of a random signal. This is reflected by the term power spectral density.

### Properties

The properties of the PSD can be deduced from the properties of the ACF and the DTFT as:

1. From the link between the PSD $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and the spectrum $X_N(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ derived above it can be concluded that the PSD is real valued

    $$\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \in \mathbb{R}$$

2. From the even symmetry $\varphi_{xx}[\kappa] = \varphi_{xx}[-\kappa]$ of the ACF it follows that

    $$ \Phi_{xx}(\mathrm{e}^{\,\mathrm{j} \, \Omega}) = \Phi_{xx}(\mathrm{e}^{\,-\mathrm{j}\, \Omega}) $$

3. The PSD of an uncorrelated random signal is given as

    $$ \Phi_{xx}(\mathrm{e}^{\,\mathrm{j} \, \Omega}) = \sigma_x^2 + \mu_x^2 \cdot {\bot \!\! \bot \!\! \bot}\left( \frac{\Omega}{2 \pi} \right) ,$$

    which can be deduced from the [ACF of an uncorrelated signal](correlation_functions.ipynb#Properties).

4. The quadratic mean of a random signal is given as

    $$ E\{ x[k]^2 \} = \varphi_{xx}[\kappa=0] = \frac{1}{2\pi} \int\limits_{-\pi}^{\pi} \Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\, \Omega}) \,\mathrm{d} \Omega $$

    The last relation can be found by expressing the ACF via the inverse DTFT of $\Phi_{xx}$ and considering that $\mathrm{e}^{\mathrm{j} \Omega \kappa} = 1$ when evaluating the integral for $\kappa=0$.

### Example - Power Spectral Density of a Speech Signal

In this example the PSD $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j} \,\Omega})$ of a speech signal of length $N$ is estimated by applying a discrete Fourier transformation (DFT) to its ACF. For a better interpretation of the PSD, the frequency axis $f = \frac{\Omega}{2 \pi} \cdot f_s$ has been chosen for illustration, where $f_s$ denotes the sampling frequency of the signal. The speech signal constitutes a recording of the vowel 'o' spoken from a German male, loaded into variable `x`.

In Python the ACF is stored in a vector with indices $0, 1, \dots, 2N - 2$ corresponding to the lags $\kappa = (0, 1, \dots, 2N - 2)^\mathrm{T} - (N-1)$. When computing the discrete Fourier transform (DFT) of the ACF numerically by the fast Fourier transform (FFT) one has to take this shift into account. For instance, by multiplying the DFT $\Phi_{xx}[\mu]$ by $\mathrm{e}^{\mathrm{j} \mu \frac{2 \pi}{2N - 1} (N-1)}$.

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.io import wavfile

# read audio file
fs, x = wavfile.read('../data/vocal_o_8k.wav')
x = np.asarray(x, dtype=float)
N = len(x)

# compute ACF
acf = 1/N * np.correlate(x, x, mode='full')
# compute PSD
psd = np.fft.fft(acf)
psd = psd * np.exp(1j*np.arange(2*N-1)*2*np.pi*(N-1)/(2*N-1))
f = np.fft.fftfreq(2*N-1, d=1/fs)

# plot PSD
plt.figure(figsize=(10, 4))
plt.plot(f, np.real(psd))
plt.title('Estimated power spectral density')
plt.ylabel(r'$\hat{\Phi}_{xx}(e^{j \Omega})$')
plt.xlabel(r'$f / Hz$')
plt.axis([0, 500, 0, 1.1*max(np.abs(psd))])
plt.grid()
```

**Exercise**

* What does the PSD tell you about the average spectral contents of a speech signal?

Solution: The speech signal exhibits a harmonic structure with the dominant fundamental frequency $f_0 \approx 100$ Hz and a number of harmonics $f_n \approx n \cdot f_0$ for $n > 0$. This due to the fact that [vowels](https://en.wikipedia.org/wiki/Vowel) generate random signals which are in good approximation periodic. To generate vowels, the sound produced by the periodically vibrating [vocal folds](https://en.wikipedia.org/wiki/Vocal_cords) is filtered by the resonance volumes and articulators above the voice box. The spectrum of periodic signals is a line spectrum.

## Cross-Power Spectral Density

The cross-power spectral density is defined as the Fourier transformation of the [cross-correlation function](correlation_functions.ipynb#Cross-Correlation-Function) (CCF).

### Definition

For two continuous-amplitude, real-valued, wide-sense stationary (WSS) random signals $x[k]$ and $y[k]$, the cross-power spectral density is defined as

\begin{equation}
\Phi_{xy}(\mathrm{e}^{\,\mathrm{j} \, \Omega}) = \mathcal{F}_* \{ \varphi_{xy}[\kappa] \},
\end{equation}

where $\varphi_{xy}[\kappa]$ denotes the CCF of $x[k]$ and $y[k]$. Note again, that the DTFT is performed with respect to $\kappa$. The CCF of two wide-sense ergodic random signals of finite length $N$ and $M$ can be expressed by way of a linear convolution

\begin{equation}
\varphi_{xy}[\kappa] = \frac{1}{N} \cdot x_N[k] * y_M[-k].
\end{equation}

Note, the $\frac{1}{N}$ normalization corresponds to the length of the signal $x$. If $N \neq M$, care should be taken on the interpretation of this normalization. In case of $N=M$ the $\frac{1}{N}$-averaging yields a [biased estimator](https://en.wikipedia.org/wiki/Bias_of_an_estimator) of the CCF, which consistently should be denoted with $\hat{\varphi}_{xy,\mathrm{biased}}[\kappa]$.

Taking the DTFT of the left- and right-hand side from above cross-correlation results in

\begin{equation}
\Phi_{xy}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{1}{N} \, X_N(\mathrm{e}^{\,\mathrm{j}\,\Omega})\, Y_M(\mathrm{e}^{-\,\mathrm{j}\,\Omega}).
\end{equation}

### Properties

1. The symmetries of $\Phi_{xy}(\mathrm{e}^{\,\mathrm{j}\, \Omega})$ can be derived from the symmetries of the CCF and the DTFT as

    $$ \underbrace {\Phi_{xy}(\mathrm{e}^{\,\mathrm{j}\, \Omega}) = \Phi_{xy}^*(\mathrm{e}^{-\,\mathrm{j}\, \Omega})}_{\varphi_{xy}[\kappa] \in \mathbb{R}} =
\underbrace {\Phi_{yx}(\mathrm{e}^{\,- \mathrm{j}\, \Omega}) = \Phi_{yx}^*(\mathrm{e}^{\,\mathrm{j}\, \Omega})}_{\varphi_{yx}[-\kappa] \in \mathbb{R}},$$

    from which $|\Phi_{xy}(\mathrm{e}^{\,\mathrm{j}\, \Omega})| = |\Phi_{yx}(\mathrm{e}^{\,\mathrm{j}\, \Omega})|$ can be concluded.

2. The cross PSD of two uncorrelated random signals is given as

    $$ \Phi_{xy}(\mathrm{e}^{\,\mathrm{j} \, \Omega}) = \mu_x^2 \mu_y^2 \cdot {\bot \!\! \bot \!\! \bot}\left( \frac{\Omega}{2 \pi} \right) $$

    which can be deduced from the CCF of an uncorrelated signal.

### Example - Cross-Power Spectral Density

The following example estimates and plots the cross PSD $\Phi_{xy}(\mathrm{e}^{\,\mathrm{j}\, \Omega})$ of two random signals $x_N[k]$ and $y_M[k]$ of finite lengths $N = 64$ and $M = 512$.

```python
N = 64  # length of x
M = 512  # length of y

# generate two uncorrelated random signals
np.random.seed(1)
x = 2 + np.random.normal(size=N)
y = 3 + np.random.normal(size=M)
N = len(x)
M = len(y)

# compute cross PSD via CCF
acf = 1/N * np.correlate(x, y, mode='full')
psd = np.fft.fft(acf)
psd = psd * np.exp(1j*np.arange(N+M-1)*2*np.pi*(M-1)/(2*M-1))
psd = np.fft.fftshift(psd)
Om = 2*np.pi * np.arange(0, N+M-1) / (N+M-1)
Om = Om - np.pi

# plot results
plt.figure(figsize=(10, 4))
plt.stem(Om, np.abs(psd), basefmt='C0:', use_line_collection=True)
plt.title('Biased estimator of cross power spectral density')
plt.ylabel(r'$|\hat{\Phi}_{xy}(e^{j \Omega})|$')
plt.xlabel(r'$\Omega$')
plt.grid()
```

**Exercise**

* What does the cross PSD $\Phi_{xy}(\mathrm{e}^{\,\mathrm{j} \, \Omega})$ tell you about the statistical properties of the two random signals?

Solution: The cross PSD $\Phi_{xy}(\mathrm{e}^{\,\mathrm{j} \, \Omega})$ is essential only non-zero for $\Omega=0$. It hence can be concluded that the two random signals are not mean-free and uncorrelated to each other.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples*.