# Gaussian Mixture Models

A Gaussian mixture model (GMM) is a density model where we combine a finite number of K Gaussian distributions $\mathcal{N}(x|\mu_k,\Sigma_k)$ so that the probability density of a random variable $x$ is expressed as:

$$ p(x|\theta) = \sum_{k=1}^K \pi_k \mathcal{N}(x|\mu_k,\Sigma_k) \\
0 \le \pi_k \le 1, \sum_{k=1}^{K}{\pi_k} = 1$$
where $\theta:=\{\mu_k,\Sigma_k,\pi_k:k=1,2,\dots,K\}$ is the collection of all parameters of the model.

Mixture models allow relatively complex marginal distributions to be expressed interms of more tractable joint distributions over an expanded space by the inclusion of latent (or unobserved) variables. In addition to providing a framework for building complex probability distributions, mixture models can be used to cluster data.

## Theory
In density estimation, we estimate data compactly using a density from a parametric family of distributions. Typically, a Gaussian, this choice might make for a poor approximation, in which case a more expressive model to consider would be mixture models.

Gaussian Mixture models have the advantage to
- enable multi-modal data representations with multiple clusters.
- compute the parameters, $\theta$ computed/learned from data via a maximum likelihood approach using the Expectation-Maximization algorithm.

Given a dataset for random variable $x$, we introduce a $K$ dimensional binary random variable $z$ having a 1-of-$K$ representation in which a particular element $z_k = 1$  with $z_i = 0 \text{ } \forall i\ne k$ i.e., $\sum{z_k} = 1$.
The marginal distribution $p(z,x)$ is defined as $ p(z,x) = p(x|z)\cdot p(z)$.

If $p(z_k = 1) = \pi_k$ and $p(x|z_k=1) = \mathcal{N}(x|\mu_k,\Sigma_k)$ then, $$\begin{align} p(z) &= \prod_{k=1}^K \pi_k^{z_k} \\
p(x|z) &= \prod_{k=1}^K \Bigg(\mathcal{N}(x|\mu_k,\Sigma_k)\Bigg)^{z_k} \end{align}$$

The marginal distribution of $x$ is therefore,

$$ p(x) = \sum_z p(x,z) = \sum_z p(z) p(x|z) = \sum_{k=1}^K{\pi_k \mathcal{N}(x|\mu_k,\Sigma_k) }$$

### Maximum Likelihood Approach
In the maximum likelihood (MLE) approach, we wish to model a dataset of observations $\{x_1,x_2,\dots,x_N\}$ using a mixture of Gaussians. In the MLE approach, we seek to compute the parameters $\pi$,$\mu$,$\Sigma$ of each of the $K$ Gaussians per the graphical model below.

The log-likelihood for the entire dataset is expressed as:

$$ \ln{p(X|\pi,\mu,\Sigma)} = \sum_{n=1}^N {\ln {\Bigg(\sum_{k=1}^K {\pi_k \mathcal{N}(x|\mu_k,\Sigma_k)}\Bigg)}} $$

To determine the optimal MLE parameters, we seek to compute $\pi$,$\mu$,$\Sigma$ which maximize the likelihood of the data. Note however that this term is intractable due to the presence of the summation inside the logarithm, so that the logarithm no longer directly acts on the Gaussians.

### Expectation Maximization for GMM
An elegant and powerful method for finding MLE solutions for models with latent variables is the EM algorithm.

At maximum likelihood, the derivative of the log-likelihood w.r.t each of the parameters is zero.

#### MLE of $\mu$
$$ \begin{align}
0 &= \frac{d}{d \mu_k} \ln{p(X|\pi,\mu,\Sigma)} \\
&= -\sum_{n=1}^N {\frac{\pi_k \mathcal{N}(x_n|\mu_k,\Sigma_k)}{\sum_{k=1}^K {\pi_k \mathcal{N}(x|\mu_k,\Sigma_k)}}\Sigma_k (x_n - \mu_k)}\\
&= -\sum_{n=1}^N{\gamma(z_{nk})\Sigma_k (x_n - \mu_k)}
\end{align}$$

where $\gamma(z_{nk}) = \frac{\pi_k \mathcal{N}(x_n|\mu_k,\Sigma_k)}{\sum_{k=1}^K {\pi_k \mathcal{N}(x|\mu_k,\Sigma_k)}}$ defined as the `responsibility that component` $k$ `takes for explaining the observation` $x$, i.e., $\gamma(z_{nk}) = p(z_k=1|x_n)$ is the posterior probability of the latent variable.

The MLE estimate of $\mu_k$ is therefore

$$\mu_k = \frac{\sum_{n=1}^N{\gamma(z_{nk}) x_n}}{\sum_{n=1}^N{\gamma(z_{nk})}} = \frac{1}{N_k}\sum_{n=1}^N{\gamma(z_{nk}) x_n}$$
where $N_k = \sum_{n=1}^N{\gamma(z_{nk})}$ is the effective number of points assigned to the cluster $k$.

#### MLE of $\Sigma$
$$ \begin{align}
0 &= \frac{d}{d \Sigma_k} \ln{p(X|\pi,\mu,\Sigma)} \\
&= \frac{d}{d \Sigma_k} |\Sigma_k| -\sum_{n=1}^N {\frac{\pi_k \mathcal{N}(x_n|\mu_k,\Sigma_k)}{\sum_{k=1}^K {\pi_k \mathcal{N}(x|\mu_k,\Sigma_k)}} (x_n - \mu_k)(x_n - \mu_k)^T}\\
&= \Sigma_k -\sum_{n=1}^N{\gamma(z_{nk}) (x_n - \mu_k)(x_n - \mu_k)^T}
\end{align}$$

Note that the $ \frac{d}{dA} \ln{\left(\det{A}\right)^{-1}} = A$. More details on this identity can be found [here](https://statisticaloddsandends.wordpress.com/2018/05/24/derivative-of-log-det-x/).

The MLE estimate of $\Sigma_k$ is therefore

$$\Sigma_k = \frac{1}{N_k}\sum_{n=1}^N{\gamma(z_{nk}) (x_n - \mu_k)(x_n - \mu_k)^T}$$

#### MLE of $\pi$
To compute the MLE of $\pi$, the constraint $\sum_k \pi_k = 1$ has to be accounted for. This can be done with a lagrange multipler $\lambda$ and maximizing the quantity $\ln{p(X|\pi,\mu,\Sigma)} + \lambda \sum_k{\pi_k} - 1$.

$$ \begin{align}
0 &= \frac{d}{d \pi_k} \Bigg(\ln{p(X|\pi,\mu,\Sigma)} + \lambda \sum_k{\pi_k} - 1 \Bigg)\\
&= \sum_{n=1}^N {\frac{\mathcal{N}(x_n|\mu_k,\Sigma_k)}{\sum_{k=1}^K {\pi_k \mathcal{N}(x|\mu_k,\Sigma_k)}} + \lambda} \end{align} $$

Multiplying both sides by $\pi_k$ and summing over k,

$$ \begin{align} 0 &= \sum_{n=1}^N {\sum_{k=1}^K{\frac{\pi_k \mathcal{N}(x_n|\mu_k,\Sigma_k)}{\sum_{k=1}^K {\pi_k \mathcal{N}(x|\mu_k,\Sigma_k)}}}} + \sum_{k=1}^K {\pi_k \lambda} \\
&= \sum_{n=1}^N {N_k} + \lambda \\
\Rightarrow \lambda &= -\sum_{n=1}^N {N_k} = -N
\end{align}$$

Substituting for $\lambda$ above, the MLE of $\pi_k$ is derived as:

$$\pi_k = \frac{N_k}{N}$$

The above solutions do not constitute a closed-form solution for the parameters of the mixture model because of the dependence of the responsibilities $\gamma(z_{nk})$ on the same parameters. An iterative scheme of these steps constitutes the EM algorithm can be applied to solve for the parameters.

Each iteration of the EM algorithm involves two updates - `the E step and the M step`.
- In the `E step`, we use the current values of the parameters to evaluate the responsibilities.
- In the `M step`, we maximize the parameters with respect to the responsibilities computed in the `E step`.

## Example
The palmer penguins dataset released by [4] and obtained from [5] is used as an example. Two features - Flipper Length & Culmen Length are used as the features to cluster the dataset into the 3 categories of penguins - Adelie, Chinstrap and Gentoo. The dataset is plotted below.

```python
import pandas as pd
import requests
import io
import numpy as np
from scipy.stats import multivariate_normal as gaussian
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from matplotlib.patches import Ellipse
import matplotlib.transforms as transforms
import matplotlib.colors as mcolors

def getCSV(url):
    download = requests.get(url).content
    df = pd.read_csv(io.StringIO(download.decode('utf-8')))
    return df

file = "https://raw.githubusercontent.com/mcnakhaee/palmerpenguins/master/palmerpenguins/data/penguins-raw.csv"
df = getCSV(file)
```

```python
txt_labels = np.unique(df['Species'])
lbl = txt_labels[0]
fig,ax = plt.subplots(1,2,figsize=(10,5))
df_data = [None]*len(txt_labels)
img = mpimg.imread('../../img/lter_penguins.png')
ax[0].imshow(img)
ax[0].axis('off')
color = ['tomato','mediumorchid','seagreen','aqua','black','magenta']
for i,lbl in enumerate(txt_labels):
    df_data[i] = df[df['Species'] ==  lbl]
#     print(df_data[i].columns)
    ax[1].scatter(df_data[i]['Flipper Length (mm)'],df_data[i]['Culmen Length (mm)'],color=color[i])
# ax[1].axis('off')
ax[1].set_xlabel('Flipper Length');
ax[1].set_ylabel('Culmen Length');
```

```python
## Number of classes
K = 3#len(txt_labels)

flp_len = np.mean(df['Flipper Length (mm)'])
clm_len = np.mean(df['Culmen Length (mm)'])

df = df[df['Flipper Length (mm)'].notna()]
df = df[df['Culmen Length (mm)'].notna()]
data = np.matrix(np.c_[df['Flipper Length (mm)'],df['Culmen Length (mm)']].T)
# print(data)

x_mean = np.array([flp_len,clm_len])
d = data - np.reshape(x_mean,(2,1))
cov = np.matmul(d,d.T)/float(data.shape[1])
## Init
pi_init = [1/float(K) for i in range(K)]#[float(df_i.shape[0])/float(df.shape[0]) for df_i in df_data]
mu_init = [np.ravel(data[:,k]) for k in range(K)]
# sigma_init = [np.eye(x_mean.shape[0]) for k in range(K)]
sigma_init = [cov for k in range(K)]
```

```python
def E_Step(X,pi,mu,sigma):
    gamma = np.zeros((X.shape[1],K))
    for n in range(X.shape[1]):
        tot = 0.0
        for k in range(K):
            gamma[n,k] = pi[k]*gaussian.pdf(X[:,n].T,mean=mu[k],cov=sigma[k])
            tot = tot + gamma[n][k]
        gamma[n,:] = gamma[n,:]/tot
    return gamma

def M_step(X,gamma):
    Nk = [0 for i in range(K)]
    pi = [0 for i in range(K)]
    for k in range(K):
        Nk[k] = np.sum(gamma[:,k])
        pi[k] = Nk[k]/float(X.shape[1])
    mu_mat = np.matmul(X,gamma)/Nk
    mu = []
    for k in range(K):
        mu.append(np.array([mu_mat[0,k],mu_mat[1,k]]))
    sigma = [np.zeros((2,2)) for i in range(K)]
    for k in range(K):
        for n in range(X.shape[1]):
            del_x = X[:,n] - np.reshape(mu[k],(X.shape[0],1))
            cov = np.matmul(del_x,del_x.T)
            sigma[k] = sigma[k] + gamma[n,k]/Nk[k]*cov

    return mu,sigma,pi

def getLogLikelihood(X,pi,mu,sigma):
    logLikelihood = 0
    for n in range(X.shape[1]):
        prob = 0
        for k in range(K):
            if np.linalg.det(sigma[k]) < 1E-6:
                print(sigma[k])
                continue
            prob = prob + pi[k]*gaussian.pdf(X[:,n].T,mean=mu[k].T,cov=sigma[k])
        logLikelihood = logLikelihood + np.log(prob)
    return logLikelihood

def EM(X,pi0,mu0,sigma0,iter_max = 250,tol=1E-6):
    pi_curr = pi0
    mu_curr = mu0
    sigma_curr = sigma0
    iter = 0
    post_prob = []
    pi = pi0.copy()
    mu = mu0.copy()
    sigma = sigma0.copy()
    while iter < iter_max:
        max_del = np.finfo('f').min
        iter = iter + 1
        gamma = E_Step(X,pi,mu,sigma)
        mu1,sigma1,pi1 = M_step(X,gamma)
#         print(pi)
#         print(mu)
#         print(sigma)
        prob = 0
        for k in range(len(mu)):
            del_mu  = np.max(np.abs(mu[k]-mu1[k]))
            del_pi  = np.max(np.abs(pi[k]-pi1[k]))
            del_sig = np.max(np.max(np.abs(sigma[k]-sigma1[k])))
            max_del = max(max_del,max(del_mu,max(del_pi,del_sig)))
        pi = pi1.copy()
        mu = mu1.copy()
        sigma = sigma1.copy()
        pi_curr = pi
        mu_curr = mu
        sigma_curr = sigma
        post_prob.append(getLogLikelihood(X,pi,mu,sigma))
#         print(iter,max_del)
        if (max_del <= tol):
            print("Converged after Iteration: " + str(iter))
            break

    return mu,sigma,pi,post_prob

def confidence_ellipse(ax, mu, cov, n_std=3.0, facecolor='none', **kwargs):
    """
    Create a plot of the covariance confidence ellipse of `x` and `y`

    Parameters
    ----------
    cov : Covariance matrix
        Input data.

    ax : matplotlib.axes.Axes
        The axes object to draw the ellipse into.

    n_std : float
        The number of standard deviations to determine the ellipse's radiuses.

    Returns
    -------
    matplotlib.patches.Ellipse

    Other parameters
    ----------------
    kwargs : `~matplotlib.patches.Patch` properties
    """
#     if cov != cov.T:
#         raise ValueError("Not a valid covariance matrix")

#     cov = np.cov(x, y)
    pearson = cov[0, 1]/np.sqrt(cov[0, 0] * cov[1, 1])
    # Using a special case to obtain the eigenvalues of this
    # two-dimensionl dataset.
    ell_radius_x = np.sqrt(1 + pearson)
    ell_radius_y = np.sqrt(1 - pearson)
    ellipse = Ellipse((0, 0),
        width=ell_radius_x * 2,
        height=ell_radius_y * 2,
        facecolor=facecolor,
        **kwargs)

    # Calculating the stdandard deviation of x from
    # the squareroot of the variance and multiplying
    # with the given number of standard deviations.
    scale_x = np.sqrt(cov[0, 0]) * n_std
    mean_x = mu[0]

    # calculating the stdandard deviation of y ...
    scale_y = np.sqrt(cov[1, 1]) * n_std
    mean_y = mu[1]

    transf = transforms.Affine2D() \
        .rotate_deg(45) \
        .scale(scale_x, scale_y) \
        .translate(mean_x, mean_y)

    ellipse.set_transform(transf + ax.transData)
    return ax.add_patch(ellipse)
```

```python
mu,sigma,pi,post_prob = EM(data,pi_init,mu_init,sigma_init)
```

    Converged after Iteration: 110

```python
fig,ax = plt.subplots(1,3,figsize=(20,5))
gamma = E_Step(data,pi,mu,sigma)
for n in range(data.shape[1]):
    rgb = np.array([0,0,0])
    for k in range(K):
        rgb = rgb+gamma[n,k]*np.array(mcolors.to_rgb(color[k]))
    ax[1].scatter(data[0,n],data[1,n],color=rgb)
ax[1].set_title('Classification as a function of responsibilities')
for k in range(K):
    if k < 3:
        ax[0].scatter(df_data[k]['Flipper Length (mm)'],df_data[k]['Culmen Length (mm)'],color=color[k])
    ax[0].plot(mu[k][0],mu[k][1],'kx')
    for i in range(3):
        confidence_ellipse(ax[0],mu[k],sigma[k],i+1,edgecolor=color[k],linestyle='dashed')

ax[0].set_title('Confidence bounds of the clusters from EM')
ax[0].set_xlabel('Flipper Length (mm)');
ax[0].set_ylabel('Culmen Length (mm)');

ax[2].plot(range(len(post_prob)),post_prob);
ax[2].set_title('Learning curve');
ax[2].set_ylabel('Log Likehood');
ax[2].set_xlabel('Iteration Number');
```

## References
[1]: Bishop, Christopher M. 2006. Pattern Recognition and Machine Learning. Springer.

[2]: M. P. Deisenroth, A. A. Faisal, and C. S. Ong, 2021. https://mml-book.com

[3]: Refer to https://statisticaloddsandends.wordpress.com/2018/05/24/derivative-of-log-det-x/ for an explanation of identity for derivative of log of a matrix determinant

[4]: Horst AM, Hill AP, Gorman KB (2020). palmerpenguins: Palmer Archipelago (Antarctica) penguin data. R package version 0.1.0. https://allisonhorst.github.io/palmerpenguins/.

[5]: CSV data downloaded from https://github.com/mcnakhaee/palmerpenguins

[6]: Code for plotting confidence ellipses from https://matplotlib.org/3.1.0/gallery/statistics/confidence_ellipse.html


# Worksheet 3

```
%matplotlib inline
```

## Question 1

Apply Simpson's rule to compute

\begin{equation}
  \int_0^{\pi/2} \cos (x) \, dx
\end{equation}

using 3 points (so $h = \pi/4$) and 5 points (so $h = \pi/8$).

### Answer Question 1

The exact solution is, of course, 1.

Simpson’s rule (composite version) is

\begin{equation}
  I = \frac{h}{3} \left[ f(a) + f(b) + 2 \sum_{j = 1}^{N/2 - 1} f(x_{2 j}) + 4 \sum_{j = 1}^{N/2} f(x_{2 j-1})  \right]
\end{equation}

where we are using $N + 1$ points with $N$ even, with $x_0 = a, x_N = b$, equally spaced with grid spacing $h = (b − a)/N$.

With 3 points we have $N = 2$ and $h = (\pi/2)/2 = \pi/4$, and so we have nodes and samples given by

\begin{equation}
  \begin{array}{c|c|c}
    i & x_i & f(x_i) \\ \hline
    0 & 0 & 1 \\
    1 & \pi/4 & 1 / \sqrt{2} \\
    2 & \pi/2 & 0
  \end{array}
\end{equation}

Using Simpson's rule we then get

\begin{align}
  I &= \frac{h}{3} \left[ f_0 + f_2 + 4 f_1 \right] \\
    & = \frac{\pi}{12} \left[ 1 + 2 \sqrt{2} \right] \\
    & \approx 1.0023.
\end{align}

With 5 points we have $N = 4$ and $h = (\pi/2)/4 = \pi/8$, and so we have nodes and samples given by

\begin{equation}
  \begin{array}{c|c|c}
    i & x_i & f(x_i) \\ \hline
    0 & 0 & 1 \\
    1 & \pi/8 & \cos ( \pi / 8 ) \approx 0.9239 \\
    2 & \pi/4 & 1 / \sqrt{2} \\
    3 & 3\pi/8 & \cos ( 3 \pi / 8 ) \approx 0.9239 \\
    4 & \pi/2 & 0
  \end{array}
\end{equation}

Using Simpson's rule we then get

\begin{align}
  I &= \frac{h}{3} \left[ f_0 + f_4 + 4 (f_1 + f_3) + 2 f_2 \right] \\
    & = \frac{\pi}{24} \left[ 1 + 4 \left( \cos ( \pi / 8 ) + \cos ( 3 \pi / 8 ) \right) + \sqrt{2} \right] \\
    & \approx 1.00013.
\end{align}

## Question 2

Apply Richardson extrapolation to the result above; does the answer improve?

### Answer Question 2

Simpson's rule has order of accuracy 4. We note that we have just computed the result using 3 ($N = 2$) and 5 ($N = 4$) points. Richardson extrapolation gives the result

\begin{align}
  R_4 &= \frac{2^4 I_{N=4} - I_{N=2}}{2^4 - 1} \\
      &\approx 0.999992.
\end{align}

We note that the error has gone from $2.3 \times 10^{−3}$ for $I_{N=2}$ to $1.3 \times 10^{−4}$ for $I_{N=4}$ and now to $8.4 \times 10^{−6}$ for the Richardson extrapolation $R_4$, a good improvement.

## Question 3

State the rate of convergence of the trapezoidal rule and Simpson’s rule, and sketch (or explain in words) the proof.

### Answer Question 3

For the trapezoidal rule the error converges as $h^2$. For Simpson’s rule the error converges as $h^4$.

In both cases the proof takes a similar path. Consider the quadrature over a single subinterval. Taylor series expand the quadrature rule about a suitable point $x_j$ (left edge for trapezoidal rule, centre for Simpson’s rule) to get an expression for the quadrature of the interval in terms of $h$ and the function $f$ and its derivatives as evaluated at $x_j$.

Next write down the anti-derivative $F(t)$ of $f$ for the interval as a function of the width of the interval $t$. This, when evaluated at $t = h$, is the exact solution for the quadrature of the subinterval. Taylor series expand $F$ about $t = 0$ to get an expression for the exact result in terms of $h$ and the function $f$ and its derivatives as evaluated at $x_j$.

By comparing the two expressions we have a bound on the error in terms of $h$ and derivatives of $f$. By summing over all intervals (note that at this stage we lose a power of $h$ as we have $N$ subintervals with $N \propto h^{−1}$) we can bound the global error in terms of $h$ and the maximum value of a derivative of $f$.

## Question 4

Explain in words adaptive and Gaussian quadrature, in particular the aims of each and the times when one or the other is more useful.

### Answer Question 4

Adaptive quadrature uses any standard quadrature method and some error estimator, such as Richardson extrapolation, to place additional nodes wherever required to ensure that the error is less than some desired tolerance. Each subinterval is tested to ensure that its (appropriately weighted) contribution to the total error is sufficiently small. If it is not, the subinterval is further subdivided by introducing more nodes in a fashion appropriate for the quadrature method used. This is a straightforward way of getting high accuracy for low computational cost using standard quadrature algorithms.

Gaussian quadrature aims to get the best result for a *generic* function by allowing both the choice of nodes and weights to vary. The location of the nodes and the value of the weights is given by ensuring that the quadrature is exact for as many polynomials as possible; i.e., if we have $N$ nodes (and hence $N$ weights) we should be able to exactly integrate $x^s$ for $0 \le s \le 2 N − 1$. By introducing a weighting function we can also deal with integrands that are (mildly) singular at the boundaries of the domain, or unbounded domains. Provided the function can be evaluated anywhere this is an effective way of getting high accuracy with few function evaluations for most functions.

## Question 5

Show how the speed of convergence of a nonlinear root finding method depends and the derivatives of the map $g(x)$ near the fixed point $s$.

### Answer Question 5

We assume we are constructing an iterative sequence $x_n$ where $x_{n+1} = g(x_n)$, and that the error at step $n$ is $e_n = x_n − s$. Then if we assume that the step $x_{n+1}$ is sufficiently close to the root $s$ then we can write

\begin{align}
  e_{n+1} &= x_{n+1} − s \\
  & = g(x_n) − g(s)
\end{align}
which, using the definition of the sequence and the fixed point, gives
\begin{align}
  e_{n+1} & = g(s) (x_n − s) + \frac{g''(s)}{2!} (x_n − s)^2 + {\mathcal O} \left( (x_n − s)^3 \right)
\end{align}
and, by Taylor expanding
\begin{align}
  e_{n+1} & = g(s) e_n + \frac{g''(s)}{2!} e_n^2 + {\mathcal O} \left( e_n^3 \right).
\end{align}

Hence if $g(s) = 0$ we have that the error reduces by a constant amount proportional to the derivative at each step. If the derivative does vanish the error at each iteration is proportional to the square of the previous error which leads to faster convergence.

## Question 6

Use Newton's method to find the root in $[0, 1]$ of

\begin{equation}
  f(x) = \sin (x) − e^x + 0.9 + x.
\end{equation}

Start from $x_0 = 1/2$ and retain 3 significant figures. Take 3 steps.

### Answer Question 6

For Newton's method we have

\begin{equation}
  x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}.
\end{equation}

So first we compute the derivative,

\begin{equation}
f(x) = \cos (x) − e^x + 1.
\end{equation}

It follows that the iterative scheme is given by

\begin{equation}
  x_{n+1} = x_n − \frac{ \sin (x_n) − e^{x_n} + 0.9 + x_n}{\cos (x_n) − e^{x_n} + 1}.
\end{equation}

We start from $x_0 = 1/2$ and compute with full precision but only retain 3 significant figures for the values of the $x_n$:

\begin{align}
  x_1 & = x_0 − \frac{ \sin (x_0) − e^{x_0} + 0.9 + x_0}{\cos (x_0) − e^{x_0} + 1} \\
      & \approx -0.508;
\end{align}

retaining 3 s.f. we set $x_1 = −0.508$, and find

\begin{align}
  x_2 & = x_1 − \frac{ \sin (x_1) − e^{x_1} + 0.9 + x_1}{\cos (x_1) − e^{x_1} + 1} \\
      & \approx 0.0393;
\end{align}

retaining 3 s.f. we set $x_2 = 0.0393$, and find

\begin{align}
  x_3 & = x_2 − \frac{ \sin (x_2) − e^{x_2} + 0.9 + x_2}{\cos (x_2) − e^{x_2} + 1} \\
      & \approx 0.103.
\end{align}

After 5 steps you would see, to 3 s.f., that it has converged to 0.106, so after 3 steps it does quite
well; a better approximation to the solution is $0.106022965\dots$ .

```
def Newton(f, df, x0, tolerance = 1e-10, MaxSteps = 100):
    """Implementing Newton's method to solve f(x) = 0, where df is the derivative of f, starting from the guess x_0."""

    import numpy as np

    x = np.zeros(MaxSteps)
    x[0] = x0

    # Set up the map g
    g = lambda x: x - f(x) / df(x)

    for i in range(1, MaxSteps):
        x[i] = g(x[i-1])
        if (np.absolute(f(x[i])) < tolerance):
            break
    return x[:i+1]

def fn_q6(x):
    """Simple function defined in question, f(x) = sin(x) - e^x + 0.9 + x."""

    import numpy as np

    return np.sin(x) - np.exp(x) + 0.9 + x

def d_fn_q6(x):
    """Derivative of simple function defined in question, f(x) = sin(x) - e^x + 0.9 + x."""

    import numpy as np

    return np.cos(x) - np.exp(x) + 1.0

x = Newton(fn_q6, d_fn_q6, 0.5, tolerance = 1e-15)
print "The root is approximately {} where f is after {} steps.\n".format(x[-1], fn_q6(x[-1]), len(x))
print "The first three steps are {}\n".format(x[1:4])
print "The fifth step is ", x[5]

import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (12, 8), dpi = 50)
plt.semilogy(range(len(x)-1), np.absolute(x[:-1] - x[-1]), 'kx')
plt.xlabel('Iteration', size = 16)
plt.ylabel('$|x_i - x_{final}|$', size = 16)

plt.show()
```

## Coding Question 1

Write a single function that, depending on an input argument, computes the integral of an input function $f(x)$ between the input arguments $a, b$, using either

1. Simpson's rule, 3 points
2. Trapezoidal rule, 3 points
3. Gaussian Quadrature, 3 nodes.

Test your code on

\begin{align}
  \int_0^1 \sin^2 ( \pi x ) & = \frac{1}{2}, \\
  \int_0^1 e^{-x} \sinh ( x ) d x & \approx 0.283833821 \\
  \int_0^1 \frac{1}{\sqrt{x}} d x & = 2.
\end{align}

Note that there are good reasons for some of the methods to fail on the final test!

### Answer Coding Question 1

```
def integrate(f, a, b, method = 'Simpson'):
    """Integrate a given function f over [a, b] using 3 points/nodes using one of three methods (Simpson, Trapezoidal, Gauss)."""
    import numpy as np

    if method == 'Simpson':
        h = (b - a) / 2.0
        I = h / 3.0 * (f(a) + f(b) + 4.0 * f( (a + b) / 2.0 ))
    elif method == 'Trapezoidal':
        h = (b - a) / 2.0
        I = h / 2.0 * (f(a) + f(b) + 2.0 * f( (a + b) / 2.0 ))
    elif method == 'Gauss':
        nodes = np.array([-np.sqrt(3.0/5.0), 0.0, np.sqrt(3.0/5.0)])
        weights = np.array([5.0/9.0, 8.0/9.0, 5.0/9.0])
        # Remap [-1, 1] to the given interval
        nodes = (nodes + 1.0) * (b - a) / 2.0 + a
        I = 0
        for i in range(len(nodes)):
            I += weights[i] * f(nodes[i])
        # Reweight
        I *= (b - a) / 2.0
    else:
        raise Exception("method parameter unknown: must be one of ['Simpson', 'Trapezoidal', 'Gauss']")

    return I

def f1(x):
    """First integrand"""
    import numpy as np

    return np.sin(np.pi * x)**2

def f2(x):
    """Second integrand"""
    import numpy as np

    return np.exp(-x) * np.sinh(x)

def f3(x):
    """Third integrand"""
    import numpy as np

    return 1.0 / np.sqrt(x)

# Now look at the results
import numpy as np

exact_solutions = [0.5, 0.283833821, 2.0]

integrand = 0
for i in [f1, f2, f3]:
    integrand +=1
    for m in ['Simpson', 'Trapezoidal', 'Gauss']:
        print "For integrand number {} using method {} the result is {} (exact solution is {})\n".format(integrand, m, integrate(i, 0, 1, m), exact_solutions[integrand-1])

```

    For integrand number  1  using method  Simpson  the result is  0.666666666667  (exact solution is 0.5 )

    For integrand number  1  using method  Trapezoidal  the result is  0.5  (exact solution is 0.5 )

    For integrand number  1  using method  Gauss  the result is  0.511227100236  (exact solution is 0.5 )

    For integrand number  2  using method  Simpson  the result is  0.282762246006  (exact solution is 0.283833821 )

    For integrand number  2  using method  Trapezoidal  the result is  0.266113229303  (exact solution is 0.283833821 )

    For integrand number  2  using method  Gauss  the result is  0.283839841028  (exact solution is 0.283833821 )

    For integrand number  3  using method  Simpson  the result is  inf  (exact solution is 2.0 )

    For integrand number  3  using method  Trapezoidal  the result is  inf  (exact solution is 2.0 )

    For integrand number  3  using method  Gauss  the result is  1.75086317797  (exact solution is 2.0 )

    -c:42: RuntimeWarning: divide by zero encountered in double_scalars

The results for the final integral will fail for those methods that evaluate the integral at $x = 0$.

## Coding Question 2

Implement the secant method to find the root of

\begin{equation}
  f(x) = \tan (x) - e^{-x}, \quad x \in [0, 1].
\end{equation}

### Answer Coding Question 2

```
def Secant(f, x0, x1, tolerance = 1e-10, MaxSteps = 100):
    """Implement the secant method to find the root of the equation f(x) = 0, starting from the initial guesses x^{(0,1)} = x0, x1"""

    import numpy as np

    x = np.zeros(MaxSteps)
    x[0] = x0
    x[1] = x1

    # There is no map!
    for i in range(2, MaxSteps):
        x[i] = x[i-1] - f(x[i-1]) * (x[i-1] - x[i-2]) / (f(x[i-1]) - f(x[i-2]))
        if (np.absolute(f(x[i])) < tolerance):
            break
    return x[:i+1]

# Now define the function whose root is to be found
def fn_q2(x):
    """Simple function defined in question, f(x) = tan(x) - exp(-x)."""

    import numpy as np

    return np.tan(x) - np.exp(-x)

x = Secant(fn_q2, 0.0, 1.0)
print "The root is approximately {} where f is {} after {} steps.\n".format(x[-1], fn_q2(x[-1]), len(x))

import numpy as np
import matplotlib.pyplot as plt
fig = plt.figure(figsize = (12, 8), dpi = 50)
plt.semilogy(range(len(x)-1), np.absolute(x[:-1] - x[-1]), 'kx')
plt.xlabel('Iteration', size = 16)
plt.ylabel('$|x_i - x_{final}|$', size = 16)

plt.show()
```

```
from IPython.core.display import HTML
def css_styling():
    styles = open("../../IPythonNotebookStyles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: Verdana, Arial, Helvetica, sans-serif;
    }
    h2 {
        font-family: Verdana, Arial, Helvetica, sans-serif;
    }
    h3 {
        font-family: Verdana, Arial, Helvetica, sans-serif;
    }
    div.text_cell_render{
        font-family: Gill, Verdana, Arial, Helvetica, sans-serif;
        line-height: 110%;
        font-size: 120%;
        width:700px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
/*    .prompt{
        display: None;
    }*/
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 12pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

> (The cell above executes the style for this notebook. It closely follows the style used in the [12 Steps to Navier Stokes](http://lorenabarba.com/blog/cfd-python-12-steps-to-navier-stokes/) course.)


# Numerical Evaluation of Integrals

```python
%matplotlib inline

import matplotlib.pyplot as plt
import numpy as np
```

Integration problems are common in statistics whenever we are dealing with continuous distributions. For example the expectation of a function is an integration problem

$$
E[f(x)] = \int{f(x) \, p(x) \, dx}
$$

In Bayesian statistics, we need to solve the integration problem for the marginal likelihood or evidence

$$
p(X \mid \alpha) = \int{p(X \mid \theta) \, p(\theta \mid \alpha) d\theta}
$$

where $\alpha$ is a hyperparameter and $p(X \mid \alpha)$ appears in the denominator of Bayes theorem

$$
p(\theta | X) = \frac{p(X \mid \theta) \, p(\theta \mid \alpha)}{p(X \mid \alpha)}
$$

In general, there is no closed form solution to these integrals, and we have to approximate them numerically. The first step is to check if there is some **reparameterization** that will simplify the problem. Then, the general approaches to solving integration problems are

1. Numerical quadrature
2. Importance sampling, adaptive importance sampling and variance reduction techniques (Monte Carlo swindles)
3. Markov Chain Monte Carlo
4. Asymptotic approximations (Laplace method and its modern version in variational inference)

This lecture will review the concepts for quadrature and Monte Carlo integration.

Quadrature
----

You may recall from Calculus that integrals can be numerically evaluated using quadrature methods such as Trapezoid and Simpson's's rules. This is easy to do in Python, but has the drawback of the complexity growing as $O(n^d)$ where $d$ is the dimensionality of the data, and hence infeasible once $d$ grows beyond a modest number.

### Integrating functions

```python
from scipy.integrate import quad
```

```python
def f(x):
    return x * np.cos(71*x) + np.sin(13*x)
```

```python
x = np.linspace(0, 1, 100)
plt.plot(x, f(x))
pass
```

#### Exact solution

```python
from sympy import sin, cos, symbols, integrate

x = symbols('x')
integrate(x * cos(71*x) + sin(13*x), (x, 0,1)).evalf(6)
```

    0.0202549

#### Using quadrature

```python
y, err = quad(f, 0, 1.0)
y
```

    0.02025493910239419

#### Multiple integration

Following the `scipy.integrate` [documentation](http://docs.scipy.org/doc/scipy/reference/tutorial/integrate.html), we integrate

$$
I=\int_{y=0}^{1/2}\int_{x=0}^{1-2y} x y \, dx\, dy
$$

```python
x, y = symbols('x y')
integrate(x*y, (x, 0, 1-2*y), (y, 0, 0.5))
```

    0.0104166666666667

```python
from scipy.integrate import nquad

def f(x, y):
    return x*y

def bounds_y():
    return [0, 0.5]

def bounds_x(y):
    return [0, 1-2*y]

y, err = nquad(f, [bounds_x, bounds_y])
y
```

    0.010416666666666668

## Monte Carlo integration

The basic idea of Monte Carlo integration is very simple and only requires elementary statistics. Suppose we want to find the value of
$$
I = \int_a^b f(x) dx
$$
in some region with volume $V$. Monte Carlo integration estimates this integral by estimating the fraction of random points that fall below $f(x)$ multiplied by $V$.

In a statistical context, we use Monte Carlo integration to estimate the expectation
$$
E[g(X)] = \int_X g(x) p(x) dx
$$

with

$$
\bar{g_n} = \frac{1}{n} \sum_{i=1}^n g(x_i)
$$
where $x_i \sim p$ is a draw from the density $p$.

We can estimate the Monte Carlo variance of the approximation as
$$
v_n = \frac{1}{n^2} \sum_{o=1}^n (g(x_i) - \bar{g_n})^2)
$$

Also, from the Central Limit Theorem,

$$
\frac{\bar{g_n} - E[g(X)]}{\sqrt{v_n}} \sim \mathcal{N}(0, 1)
$$

The convergence of Monte Carlo integration is $\mathcal{0}(n^{1/2})$ and independent of the dimensionality. Hence Monte Carlo integration generally beats numerical integration for moderate- and high-dimensional integration since numerical integration (quadrature) converges as $\mathcal{0}(n^{d})$. Even for low dimensional problems, Monte Carlo integration may have an advantage when the volume to be integrated is concentrated in a very small region and we can use information from the distribution to draw samples more often in the region of importance.

An elementary, readable description of Monte Carlo integration and variance reduction techniques can be found [here](https://www.cs.dartmouth.edu/~wjarosz/publications/dissertation/appendixA.pdf).

### Intuition behind Monte Carlo integration

We want to find some integral

$$I = \int{f(x)} \, dx$$

Consider the expectation of a function $g(x)$ with respect to some distribution $p(x)$. By definition, we have

$$
E[g(x)] = \int{g(x) \, p(x) \, dx}
$$

If we choose $g(x) = f(x)/p(x)$, then we have

$$
\begin{align}
E[g(x)] &= \int{\frac{f(x}{p(x)} \, p(x) \, dx} \\
&= \int{f(x) dx} \\
&= I
\end{align}
$$

By the law of large numbers, the average converges on the expectation, so we have

$$
I \approx \bar{g_n} = \frac{1}{n} \sum_{i=1}^n g(x_i)
$$

If $f(x)$ is a proper integral (i.e. bounded), and $p(x)$ is the uniform distribution, then $g(x) = f(x)$ and this is known as ordinary Monte Carlo. If the integral of $f(x)$ is improper, then we need to use another distribution with the same support as $f(x)$.

```python
from scipy import stats
```

```python
x = np.linspace(-3,3,100)
dist = stats.norm(0,1)
a = -2
b = 0
plt.plot(x, dist.pdf(x))
plt.fill_between(np.linspace(a,b,100), dist.pdf(np.linspace(a,b,100)), alpha=0.5)
plt.text(b+0.1, 0.1, 'p=%.4f' % (dist.cdf(b) - dist.cdf(a)), fontsize=14)
pass
```

#### Using quadrature

```python
y, err = quad(dist.pdf, a, b)
y
```

    0.47724986805182085

#### Simple Monte Carlo integration

If we can sample directly from the target distribution $N(0,1)$

```python
n = 10000
x = dist.rvs(n)
np.sum((a < x) & (x < b))/n
```

    0.4816

If we cannot sample directly from the target distribution $N(0,1)$ but can evaluate it at any point.

Recall that $g(x) = \frac{f(x)}{p(x)}$. Since $p(x)$ is $U(a, b)$, $p(x) = \frac{1}{b-a}$. So we want to calculate

$$
\frac{1}{n} \sum_{i=1}^n (b-a) f(x)
$$

```python
n = 10000
x = np.random.uniform(a, b, n)
np.mean((b-a)*dist.pdf(x))
```

    0.4783397843683427

### Intuition for error rate

We will just work this out for a proper integral $f(x)$ defined in the unit cube and bounded by $|f(x)| \le 1$. Draw a random uniform vector $x$ in the unit cube. Then

$$
\begin{align}
E[f(x_i)] &= \int{f(x) p(x) dx} = I \\
\text{Var}[f(x_i)] &= \int{(f(x_i) - I )^2 p(x) \, dx} \\
&= \int{f(x)^2 \, p(x) \, dx} - 2I \int(f(x) \, p(x) \, dx + I^2 \int{p(x) \, dx} \\
&= \int{f(x)^2 \, p(x) \, dx}  + I^2 \\
& \le \int{f(x)^2 \, p(x) \, dx} \\
& \le \int{p(x) \, dx} = 1
\end{align}
$$

Now consider summing over many such IID draws $S_n = f(x_1) + f(x_2) + \cdots + f(x_n)$, \ldots, x_n$. We have

$$
\begin{align}
E[S_n] &= nI \\
\text{Var}[S_n] & \le n
\end{align}
$$

and as expected, we see that $I \approx S_n/n$. From Chebyshev's inequality,

$$
\begin{align}
P \left( \left| \frac{s_n}{n} - I \right| \ge \epsilon \right)  &=
P \left( \left| s_n - nI \right| \ge n \epsilon \right) & \le \frac{\text{Var}[s_n]}{n^2 \epsilon^2} & \le
\frac{1}{n \epsilon^2} = \delta
\end{align}
$$

Suppose we want 1% accuracy and 99% confidence - i.e. set $\epsilon = \delta = 0.01$. The above inequality tells us that we can achieve this with just $n = 1/(\delta \epsilon^2) = 1,000,000$ samples, regardless of the data dimensionality.

### Example

We want to estimate the following integral $\int_0^1 e^x dx$.

```python
x = np.linspace(0, 1, 100)
plt.plot(x, np.exp(x))
plt.xlim([0,1])
plt.ylim([0, np.exp(1)])
pass
```

#### Analytic solution

```python
from sympy import symbols, integrate, exp

x = symbols('x')
expr = integrate(exp(x), (x,0,1))
expr.evalf()
```

    1.71828182845905

#### Using quadrature

```python
from scipy import integrate

y, err = integrate.quad(exp, 0, 1)
y
```

    1.7182818284590453

#### Monte Carlo integration

```python
for n in 10**np.array([1,2,3,4,5,6,7,8]):
    x = np.random.uniform(0, 1, n)
    sol = np.mean(np.exp(x))
    print('%10d %.6f' % (n, sol))
```

            10 2.016472
           100 1.717020
          1000 1.709350
         10000 1.719758
        100000 1.716437
       1000000 1.717601
      10000000 1.718240
     100000000 1.718152

### Monitoring variance in Monte Carlo integration

We are often interested in knowing how many iterations it takes for Monte Carlo integration to "converge". To do this, we would like some estimate of the variance, and it is useful to inspect such plots. One simple way to get confidence intervals for the plot of Monte Carlo estimate against number of iterations is simply to do many such simulations.

For the example, we will try to estimate the function (again)

$$
f(x) = x \cos 71 x + \sin 13x, \ \  0 \le x \le 1
$$

```python
def f(x):
    return x * np.cos(71*x) + np.sin(13*x)
```

```python
x = np.linspace(0, 1, 100)
plt.plot(x, f(x))
pass
```

#### Single MC integration estimate

```python
n = 100
x = f(np.random.random(n))
y = 1.0/n * np.sum(x)
y
```

    -0.15505102485636882

#### Using multiple independent sequences to monitor convergence

We vary the sample size from 1 to 100 and calculate the value of $y = \sum{x}/n$ for 1000 replicates. We then plot the 2.5th and 97.5th percentile of the 1000 values of $y$ to see how the variation in $y$ changes with sample size. The blue lines indicate the 2.5th and 97.5th percentiles, and the red line a sample path.

```python
n = 100
reps = 1000

x = f(np.random.random((n, reps)))
y = 1/np.arange(1, n+1)[:, None] * np.cumsum(x, axis=0)
upper, lower = np.percentile(y, [2.5, 97.5], axis=1)
```

```python
plt.plot(np.arange(1, n+1), y, c='grey', alpha=0.02)
plt.plot(np.arange(1, n+1), y[:, 0], c='red', linewidth=1);
plt.plot(np.arange(1, n+1), upper, 'b', np.arange(1, n+1), lower, 'b')
pass
```

#### Using bootstrap to monitor convergence

If it is too expensive to do 1000 replicates, we can use a bootstrap instead.

```python
xb = np.random.choice(x[:,0], (n, reps), replace=True)
yb = 1/np.arange(1, n+1)[:, None] * np.cumsum(xb, axis=0)
upper, lower = np.percentile(yb, [2.5, 97.5], axis=1)
```

```python
plt.plot(np.arange(1, n+1)[:, None], yb, c='grey', alpha=0.02)
plt.plot(np.arange(1, n+1), yb[:, 0], c='red', linewidth=1)
plt.plot(np.arange(1, n+1), upper, 'b', np.arange(1, n+1), lower, 'b')
pass
```

## Variance Reduction

With independent samples, the variance of the Monte Carlo estimate is

$$
\begin{align}
\text{Var}[\bar{g_n}] &= \text{Var} \left[ \frac{1}{N}\sum_{i=1}^{N} \frac{f(x_i)}{p(x_i)} \right] \\
&= \frac{1}{N^2} \sum_{i=1}^{N}  \text{Var} \left[ \frac{f(x_i)}{p(x_i)} \right] \\
&= \frac{1}{N^2} \sum_{i=1}^{N} \text{Var}[Y_i] \\
&= \frac{1}{N} \text{Var}[Y_i]
\end{align}
$$

where $Y_i = f(x_i)/p(x_i)$. In general, we want to make $\text{Var}[\bar{g_n}]$ as small as possible for the same number of samples. There are several variance reduction techniques (also colorfully known as Monte Carlo swindles) that have been described - we illustrate the change of variables and importance sampling techniques here.

### Change of variables

The Cauchy distribution is given by
$$
f(x) = \frac{1}{\pi (1 + x^2)}, \ \ -\infty \lt x \lt \infty
$$

Suppose we want to integrate the tail probability $P(X > 3)$ using Monte Carlo. One way to do this is to draw many samples form a Cauchy distribution, and count how many of them are greater than 3, but this is extremely inefficient.

#### Only 10% of samples will be used

```python
import scipy.stats as stats

h_true = 1 - stats.cauchy().cdf(3)
h_true
```

    0.10241638234956674

```python
n = 100

x = stats.cauchy().rvs(n)
h_mc = 1.0/n * np.sum(x > 3)
h_mc, np.abs(h_mc - h_true)/h_true
```

    (0.1, 0.02359370926927643)

#### A change of variables lets us use 100% of draws

We are trying to estimate the quantity

$$
\int_3^\infty \frac{1}{\pi (1 + x^2)} dx
$$

Using the substitution $y = 3/x$ (and a little algebra), we get

$$
\int_0^1 \frac{3}{\pi(9 + y^2)} dy
$$

Hence, a much more efficient MC estimator is

$$
\frac{1}{n} \sum_{i=1}^n \frac{3}{\pi(9 + y_i^2)}
$$

where $y_i \sim \mathcal{U}(0, 1)$.

```python
y = stats.uniform().rvs(n)
h_cv = 1.0/n * np.sum(3.0/(np.pi * (9 + y**2)))
h_cv, np.abs(h_cv - h_true)/h_true
```

    (0.10252486615772155, 0.0010592427272478476)

### Importance sampling

Suppose we want to evaluate

$$
I = \int{h(x)\,p(x) \, dx}
$$

where $h(x)$ is some function and $p(x)$ is the PDF of $y$. If it is hard to sample directly from $p$, we can introduce a new density function $q(x)$ that is easy to sample from, and write

$$
I = \int{h(x)\, p(x)\, dx} = \int{h(x)\, \frac{p(x)}{q(x)} \, q(x) \, dx}
$$

In other words, we sample from $h(y)$ where $y \sim q$ and weight it by the likelihood ratio $\frac{p(y)}{q(y)}$, estimating the integral as

$$
\frac{1}{n}\sum_{i=1}^n \frac{p(y_i)}{q(y_i)} h(y_i)
$$

Sometimes, even if we can sample from $p$ directly, it is more efficient to use another distribution.

#### Example

Suppose we want to estimate the tail probability of $\mathcal{N}(0, 1)$ for $P(X > 5)$. Regular MC integration using samples from $\mathcal{N}(0, 1)$ is hopeless since nearly all samples will be rejected. However, we can use the exponential density truncated at 5 as the importance function and use importance sampling. Note that $h$ here is simply the identify function.

```python
x = np.linspace(4, 10, 100)
plt.plot(x, stats.expon(5).pdf(x))
plt.plot(x, stats.norm().pdf(x))
pass
```

#### Expected answer

We expect about 3 draws out of 10,000,000 from $\mathcal{N}(0, 1)$ to have a value greater than 5. Hence simply sampling from $\mathcal{N}(0, 1)$ is hopelessly inefficient for Monte Carlo integration.

```python
%precision 10
```

    '%.10f'

```python
v_true = 1 - stats.norm().cdf(5)
v_true
```

    0.0000002867

#### Using direct Monte Carlo integration

```python
n = 10000
y = stats.norm().rvs(n)
v_mc = 1.0/n * np.sum(y > 5)
# estimate and relative error
v_mc, np.abs(v_mc - v_true)/v_true
```

    (0.0000000000, 1.0000000000)

#### Using importance sampling

```python
n = 10000
y = stats.expon(loc=5).rvs(n)
v_is = 1.0/n * np.sum(stats.norm().pdf(y)/stats.expon(loc=5).pdf(y))
# estimate and relative error
v_is, np.abs(v_is- v_true)/v_true
```

    (0.0000002850, 0.0056329867)

```python

```


## Learning Objectives

By the end of this session you should be able to...

1. Take the derivative of a function over one variable
1. Take the partial derivative of a function over all of its variables
1. Find the minimum of the function to obtain the best line that represents relationships between two variables in a dataset

## Why are derivatives important?

Derivatives are the foundation for Linear Regression (a topic we'll cover later in the course) that allows us to obtain the best line that represents relationships between two variables in a dataset.

## Introduction to Derivatives

The process of fidning a derivative is called **Differentiation**, which is a technique used to calculate the slope of a graph at different points.

### Activity - Derivative Tutorial:

1. Go through this [Derivative tutorial from Math Is Fun](https://www.mathsisfun.com/calculus/derivatives-introduction.html) (15 min)
1. When you're done, talk with a partner about topics you still have questions on. See if you can answer each other's questions. (5 min)
1. We'll then go over questions on the tutorial as a class (10 min)

### Review Diagram

Review the below diagram as a class, and compare with what you just learned in the above Derivative Tutorial. Note that a Gradient Function is just another name for the Derivative of a function:

## Derivative Formula

- Choose small $\Delta x$

- $f^\prime(x) = \frac{d}{dx}f(x) = \frac{\Delta y}{\Delta x}  = \frac{f(x + \Delta x) - f(x)}{\Delta x}$

Remember that $\Delta x$ approaches 0. So if plugging in a value in the above formula, choose a _very_ small number, or simplify the equation further such that all $\Delta x = 0$, like we saw in the tutorial

## Activity: Write a Python function that calculates the gradient of $x^2$ at $x = 3$ and $x = -2$ using the above definition

```python
def f(x):
    return x**2

eps = 1e-6
x = 3
print((f(x + eps) - f(x)) / eps)
x = -2
print((f(x + eps) - f(x)) / eps)
```

    6.000001000927568
    -3.999998999582033

Note that these values match $2x$, our derivative of $x^2$:

$2*3 = 6$

$2 * -2 = -4$

## Derivative Table

As a shortcut, use the second page of this PDF to find the derivative for common formulas. Utilize this as a resource going forward!

- https://www.qc.edu.hk/math/Resource/AL/Derivative%20Table.pdf

## Extend Gradient into Two-Dimensional Space

Now we know how to calculate a derivative of one variable. But what if we have two?

To do this, we need to utilize **Partial Derivatives**. Calculating a partial derivative is essentially calculating two derivatives for a function: one for each variable, where they other variable is set to a constant.

### Activity - Partial Derivative Video

Lets watch this video about Partial Derivative Intro from Khan Academy: https://youtu.be/AXqhWeUEtQU

**Note:** Here are some derivative shortcuts that will help in the video:

$\frac{d}{dx}x^2 = 2x$

$\frac{d}{x}sin(x) = cos(x)$

$\frac{d}{dx}x = 1$

### Activity - Now You Try!
Consider the function $f(x, y) = \frac{x^2}{y}$

- Calculate the first order partial derivatives ($\frac{\partial f}{\partial x}$ and $\frac{\partial f}{\partial y}$) and evaluate them at the point $P(2, 1)$.

## We can use the Symbolic Python package (library) to compute the derivatives and partial derivatives

```python
from sympy import symbols, diff
# initialize x and y to be symbols to use in a function
x, y = symbols('x y', real=True)
f = (x**2)/y
# Find the partial derivatives of x and y
fx = diff(f, x, evaluate=True) # partial derivative of f(x,y) with respect to x
fy = diff(f, y, evaluate=True) # partial derivative of f(x,y) with respect to y
print(fx)
print(fy)
# print(f.evalf(subs={x: 2, y: 1}))
print(fx.evalf(subs={x: 2, y: 1}))
print(fy.evalf(subs={x: 2, y: 1}))
```

    2*x/y
    -x**2/y**2
    4.00000000000000
    -4.00000000000000

## Optional Reading: Tensorflow is a powerful package from Google that calculates the derivatives and partial derivatives numerically

```python
import tensorflow as tf

x = tf.Variable(2.0)
y = tf.Variable(1.0)

with tf.GradientTape(persistent=True) as t:
    z = tf.divide(tf.multiply(x, x), y)

# Use the tape to compute the derivative of z with respect to the
# intermediate value x and y.
dz_dx = t.gradient(z, x)
dz_dy = t.gradient(z, y)

print(dz_dx)
print(dz_dy)

# All at once:
gradients = t.gradient(z, [x, y])
print(gradients)

del t
```

## Optional Reading: When x and y are declared as constant, we should add `t.watch(x)` and `t.watch(y)`

```python
import tensorflow as tf

x = tf.constant(2.0)
y = tf.constant(1.0)

with tf.GradientTape(persistent=True) as t:
    t.watch(x)
    t.watch(y)
    z = tf.divide(tf.multiply(x, x), y)

# Use the tape to compute the derivative of z with respect to the
# intermediate value y.
dz_dx = t.gradient(z, x)
dz_dy = t.gradient(z, y)
```

# Calculate Partial Derivative from Definition

```python
def f(x, y):
    return x**2/y

eps = 1e-6
x = 2
y = 1
print((f(x + eps, y) - f(x, y)) / eps)
print((f(x, y + eps) - f(x, y)) / eps)
```

    4.0000010006480125
    -3.9999959997594203

Looks about right! This works rather well, but it is just an approximation. Also, you need to call `f()` at least once per parameter (not twice, since we could compute `f(x, y)` just once). This makes this approach difficult to control for large systems (for example neural networks).

## Why Do we need Partial Gradients?

In many applications, more specifically DS applications, we want to find the Minimum of a cost function

- **Cost Function:** a function used in machine learning to help correct / change behaviour to minimize mistakes. Or in other words, a measure of how wrong the model is in terms of its ability to estimate the relationship between x and y. [Source](https://towardsdatascience.com/machine-learning-fundamentals-via-linear-regression-41a5d11f5220)

Why do we want to find the minimum for a cost function? Given that a cost function mearues how wrong a model is, we want to _minimize_ that error!

In Machine Learning, we frequently use models to run our data through, and cost functions help us figure out how badly our models are performing. We want to find parameters (also known as **weights**) to minimize our cost function, therefore minimizing error!

We find find these optimal weights by using a **Gradient Descent**, which is an algorithm that tries to find the minimum of a function (exactly what we needed!). The gradient descent tells the model which direction it should take in order to minimize errors, and it does this by selecting more and more optimal weights until we've minimized the function! We'll learn more about models when we talk about Linear Regression in a future lesson, but for now, let's review the Gradient Descent process with the below images, given weights $w_0$ and $w_1$:

Look at that bottom right image. Looks like we're using partial derivatives to find out optimal weights. And we know exactly how to do that!

## Finding minimum of a function

Assume we want to minimize the function $J$ which has two weights $w_0$ and $w_1$

We have two options to find the minimum of $J(w_0, w_1)$:

1. Take partial derivatives of $J(w_0, w_1)$ with relation to $w_0$ and $w_1$:

$\frac{\partial J(w_0, w_1)}{\partial w_0}$

$\frac{\partial J(w_0, w_1)}{\partial w_1}$

And find the appropriate weights such that the partial derivatives equal 0:

$\frac{\partial J(w_0, w_1)}{\partial w_0} = 0$

$\frac{\partial J(w_0, w_1)}{\partial w_1} = 0$

In this approach we should solve system of linear or non-linear equation

2. Use the Gradient Descent algorithm:

First we need to define two things:

- A step-size alpha ($\alpha$) -- also called the *learning rate* -- as a small number (like $1.e-5$)
- An arbitrary random initial value for $w_0$ and $w_1$: $w_0 = np.random.randn()$ and $w_1 = np.random.randn()$

Finally, we need to search for the most optimal $w_0$ and $w_1$ by using a loop to update the weights until we find the most optimal weights. We'll need to establish a threshold to compare weights to know when to stop the loop. For example, if the weight update -- the change in the weight parameter -- from one iteration is within 0.0001 of the weight from the previous iteration, we can stop the loop (0.0001 is our threshold here)

Let's review some pseudocode for how to implement this algorithm:

```
# initialization
initialize the following:
    a starting weight value -- an initial guess, could be random
    the learning rate (alpha), a small number (we'll choose 1.e-5)
    the threshold -- set this to 1.e-4
    the current weight update -- initialize to 1

# weight update loop
while the weight update is greater than the threshold:
    store the current values of the weights into a previous value variable
    set the weight values to new values based on the algorithm, by adding the weight updates
```

How do we `set the weight values to new values based on the algorithm`? by using the below equations:

$w_0 = w_0 - \alpha \frac{\partial J(w_0, w_1)}{\partial w_0}$

$w_1 = w_1 - \alpha \frac{\partial J(w_0, w_1)}{\partial w_1}$

Finish the "starter code" block below, creating real code from the pseudocode!

**Stretch Challenge:** We may also want to limit the number of loops we do, in addition to checking the threshold. Determine how we may go about doing that

## Resources

- [Derivative tutorial from Math Is Fun](https://www.mathsisfun.com/calculus/derivatives-introduction.html)
- [Derivative Table](https://www.qc.edu.hk/math/Resource/AL/Derivative%20Table.pdf)
- [Khan Academy - Partial Derivatives video](https://www.youtube.com/watch?v=AXqhWeUEtQU&feature=youtu.be)
- [Towards Data Science - Machine Learning Fundamentals: cost functions and gradient Descent](https://towardsdatascience.com/machine-learning-fundamentals-via-linear-regression-41a5d11f5220)

## Gradient descent in one dimension

```python
# pseudo-code
def minimize(f):
    # Initialize

    # run the weight update loop until it terminates

    # return the current weights
```

```python
import matplotlib.pyplot as plt
import numpy as np
np.random.seed(42)
eps = 1.e-10

# define an interesting function to minimize
x = np.linspace(-10,10, 100)
fx =lambda x: (1/50)*x**4 - 2*x**2 + x + 1
df_dx = lambda x: (4/50)*x**3 - 4*x + 1
plt.plot(x,fx(x),label = 'function f(x)')
plt.plot(x,df_dx(x),'r--',label = 'derivative df/dx')
plt.legend()
plt.grid()
```

```python
def derivative(fx, x):
    delta_x = 1e-10
    df_dx = (fx(x + delta_x) - fx(x)) / delta_x
    return df_dx
```

```python
derivative(fx, 7.300008)
```

    2.921467512351228

### gradient descent function

```python
def minimize(fx,x_init):
    # Initialize
    alpha = 1.e-6
    thresh = 1.e-4
    weight_update = 1.
    x_values = []
    max_iter = 1000
    n_iter = 0
    x = x_init
    eps = 1.e-10

    # run the weight update loop until it terminates
    while np.abs(weight_update) > thresh: #and n_iter < max_iter:
        n_iter+=1
        df_dx = (fx(x+eps) - fx(x))/eps
        weight_update = -alpha*df_dx
        x = x + weight_update
        x_values.append(x)

    # return the final value of the weight -- which should correspond to the minimum of the function
    return x, x_init, x_values, n_iter
```

### Choose an initial value for x, then run gradient descent

```python
import numpy as np

np.random.seed(42)

x_init = np.random.uniform(-10,10,1) # choose a random starting point
print(x_init)
x_star, x_init, x_values, n_iter = minimize(f,x_init)
print(f'Started at {x_init}, found minimum {x_star} after {n_iter} iterations')
```

### Check that the derivative of the function is indeed zero at the minimum found by gradient descent

```python
# derivative, from calculus
print(f'derivative from calculus: {df_dx(x_star)}')

# derivative, from definition
print(f'derivative from calculus:  {(fx(x_star + eps) - fx(x_star) )/eps}')
```

    derivative from calculus: [10.01321418]
    derivative from calculus:  [10.01321692]

```python
x_values = np.array(x_values)
plt.plot(x,fx(x),label = 'function f(x)')
plt.plot(np.array(x_values),fx(x_values),'.',markersize = 1, label='gradient descent updates')
plt.plot(x_init,fx(x_init),'r.',markersize = 10, label = '$x_{init}$')
plt.plot(x_star,fx(x_star),'k*',markersize = 10, label = '$x_{star}$')

plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid()
plt.legend(loc='best');
```

```python

```


# Symbolic Computing with SymPy

Use of Computer Algebra Systems (CAS) or symbolic computing relates to symbolic mathematical operations to be contrasted with the floating point mathematics of numerical computation.  Symbolic computing should be viewed as complementary to numerical computing, and both can be extremely helpful in the course of your research.  We will focus here on symbolic computing in SymPy, a Python based CAS.

There are many other CAS packages avaialble (see for example: https://en.wikipedia.org/wiki/List_of_computer_algebra_systems).  We will work with SymPy because it is free, open-source, built in Python, and accessible from anywhere.  SageMath is a related, but more complete solution with similar advantages but more overhead.

This lesson is developed from the official SymPy tutorial, available with more detail at http://docs.sympy.org/latest/tutorial/index.html.

A working version of SymPy can be accessed from anywhere by going to http://live.sympy.org

To get started with SymPy, we need to import the library.  If the library is not avaialble, you need to run `pip install sympy --user` in your terminal.

```python
%pylab inline
from sympy import *
```

One advantage of a CAS such as SymPy is that irrational numbers are treated exactly, rather than approximately.

```python
sqrt(8)
```

Compare this to what we would have gotten using the definition of `sqrt()` in the math package.

```python
import math
math.sqrt(8)
```

Another advantage of SymPy is that it can produce pretty output, including LaTeX.  Let us now turn on the LaTeX printer for SymPy.

```python
init_printing(latex)
```

Just like elsewhere in Python, one needs to be careful about how division is treated due to the assumed data types of the input.  Try evaluating the follwing cells.

```python
1/3
```

```python
1./3
```

```python
S(1)/3
```

Though these expressions look similar, they give different results.  The result of the first cell even depends on your version of Python.  The third cell coverts the 1 from an `int` to a SymPy object, and returns a SymPy `Rational`.  You can always check your data type with the `type()` command.

```python
type(1)
```

```python
type(1.)
```

```python
type(S(1))
```

```python
type(S(1)/3)
```

The main advantage of any CAS, however is its ability to deal with variables, which are called symobls in SymPy.  We first have to define symbols that will then be treated as SymPy objects.

```python
x, y, z, t, nu, sigma = symbols('x y z t nu sigma')
```

```python
expr = x**2+2*y
expr
```

```python
expr + 1
```

```python
expr - y
```

```python
expr * x
```

```python
expr**2
```

```python
expand(expr**2)
```

```python
x**nu+sigma
```

You can see from these examples that SymPy does some straightforward simplifications automatically, but exressions are neither factored nor expanded unless explicit commands are given.  There are also several built-in characters which will produce nice LaTeX output.

### Exercise 1

Write $(w+1/3)^3 + (w^2-2w+12)^2 - (w/4)^4$ in expanded polynomial form.

```python

```

```python

```

# Quick Examples of SymPy Capabilities

## Derivatives

```python
diff(exp(x**2), x)
```

```python
diff(x**7, x)
```

```python
diff(x**7, x, x)
```

```python
diff(x**7, x, 3)
```

```python
diff(x**2 * y**3, x, y)
```

```python
diff(sin(x)**2, x)
```

```python
diff(exp(x)*cos(x), x)
```

Multiple derivatives can be taken by repeating the variable, or by indicating the order of the derivative with an integer.

### Exercise 2

Find the first and second derivatives of $\frac{1}{\log(w^2+1/2)}$ with respect to $w$.

```python

```

```python

```

```python

```

## Integrals

```python
integrate(2*x, x)
```

```python
integrate(sin(x)*cos(x), x)
```

```python
integrate(exp(x)*cos(x)**2, x)
```

```python
integrate(y*x*z, x, y)
```

```python
integrate(cos(x), (x, 1, 2))
```

```python
integrate(exp(-x**2), (x, -oo, oo))
```

Note that infinity is written as two small 'o' characters.

### Exercise 3

Calculate the following definte integral: $\int_0^{e^4} \log(w) \, \mathrm{d}w$.

```python

```

```python

```

## Limits

```python
limit( (x**2 - 1)/(x - 1), x, 1)
```

```python
limit( (5*x**4 + 3*x**2)/(2*x**4 - 20*x**3 + 4), x, oo)
```

```python
limit( sin(x)/x, x, 0)
```

```python
limit(1/x, x, 0, '+')
```

```python
limit(1/x, x, 0, '-')
```

### Exercise 4

Calculate the limit: $$\lim_{w \rightarrow 1} \left(\frac{w}{3w-3} - \frac{1}{3\log w} \right) $$

```python

```

```python

```

```python

```

## Solving Algebraic Equations

The function `solve()` assumes that the expression is equal to zero unless the SymPy `Eq()` is used.

```python
solve(x**2 - 9, x)
```

```python
solve( x**2 + y*x + z, x)
```

```python
solve(10**x - y**3, x)
```

```python
solve(tan(x) - y**2, x)
```

```python
solve(x**2 + 1, x)
```

```python
solve(x**3 + 1, x)
```

### Exercise 5

Solve the following expression for $w$: $\sqrt{5}w + 3w^2 = 2/7$

```python

```

```python

```

## Solving Differential Equations

The function `dsolve()` can be used to solve ordinary differential equations after defining `Function` symbols.

```python
f, g, h = symbols('f g h', cls=Function)
```

```python
dsolve(diff(f(x), x) - f(x), f(x))
```

```python
dsolve(diff(f(x), x, x) + f(x), f(x))
```

```python
dsolve(diff(f(x), x, x) + 3 * diff(f(x), x ) + 6 * f(x), f(x))
```

There is also a `pdsolve()` for partial differential equations.

```python
pdsolve(2*diff(f(x,y),x)/f(x,y) + 3*diff(f(x,y), y)/f(x,y) + 1, f(x,y))
```

### Exercise 6

Solve $f'(x) = f^2(x)$ where $f(1/3) = 5/2$.

```python

```

```python

```

# Basic Manipulation

## Substitution and Numerical Evaluation

All instances of a variable in an expression can be replaced with the subs() command.

```python
expr = 4*x**2 + 3*x + 1
expr
```

```python
expr.subs(x, 2)
```

```python
expr.subs(x, y+z)
```

```python
expr = x**2 * y**2 + 3 * x * y**2 + 4*x - 12*z
expr
```

```python
expr.subs([(x,3), (y,-1), (z,2)])
```

The `evalf()` command is used to evaluate an expression as a floating point number to an arbitrary number of digits.

```python
expr = sqrt(8)
expr
```

```python
expr.evalf()
```

```python
expr = sqrt(x)
expr
```

```python
expr.evalf(subs={x: 2})
```

```python
pi.evalf(100)
```

### Exercise 7

Find the numerical value of the expression $x^{1/3} + \sqrt{5} x^2 - \log x$ for $x=1,2,\ldots,10$

```python

```

```python

```

## Simplification

The function `simplify()` attempts to give the simplest form of an expression.

```python
simplify(sin(x)**2 + cos(x)**2)
```

```python
simplify((x**3 + x**2 - x - 1)/(x**2 + 2*x + 1))
```

```python
simplify(gamma(x)/gamma(x - 2))
```

Expressions are typically not expanded or factored by `simplify()`, but can be forced to do so by `expand()` and `factor()`.

```python
expand((x + 1)**4)
```

```python
factor(x**2 + 2*x + 1)
```

`collect()` can be used to group common powers of a given term in an expression.

```python
collect(4*x**2 + 3*x + 6*z*x**2 + y*x + 1, x)
```

# Calculus

There are tools for performing calculus operations as well as creating unevaluated derivatives, integrals, and series, which can be evaluated with the `doit()` command.

```python
diff(log(x), x)
```

```python
expr = sin(x)*y**2
expr
```

```python
expr.diff(x, x, y)
```

```python
deriv = Derivative(exp(x**2*y),x,y)
deriv
```

```python
deriv.doit()
```

```python
integrate(exp(x)*x**2, x)
```

```python
integ = Integral(exp(-x**2 - y**2), (x, -oo, oo), (y, -oo, oo))
integ
```

```python
integ.doit()
```

```python
limit(tan(x)/x, x, 0)
```

```python
expr = Limit(sqrt(sin(x))/sqrt(x),x,0)
expr
```

```python
expr.doit()
```

## Series Expansions

SymPy can perform Taylor series expansions of a given function about a specified point to a desired order (which defaults to 6).

```python
series(exp(x), x, 0, 8)
```

```python
expr = atan(x)
expr.series(x, 0)
```

```python
series(f(x), x, 1)
```

```python
series(exp(1/x), x, oo)
```

```python
series(1/sin(x), x, 0)
```

### Exercise 8

Find the series expansion of $\mathrm{sinc}(x)$ about $x=0$

```python

```

```python

```

## Linear Algebra

SymPy can also handle symbolic computing of vectors and matrices.

```python
A = Matrix([[sqrt(2),0],[pi,E]])
A
```

```python
B = Matrix([[x,y],[z,w]])
B
```

```python
B.inv()
```

```python
B**(-1)
```

```python
B**2
```

```python
B.T
```

```python
3*B
```

```python
V = Matrix([2*x,5*z])
V
```

```python
B*V
```

```python
(V.T)*B
```

```python
B.det()
```

```python
B.rref()
```

# Printing

We have so far seen that SymPy can produce nice LaTeX output for easy reading.  It can also produce output which is easily transferrable for other uses.

```python
expr = Piecewise((x + 1, x > 0), (x, True))
expr
```

```python
print(latex(expr))
```

```python
print(ccode(expr))
```

```python
print(fcode(expr))
```

```python
print(python(expr))
```

```python
expr = Eq(x**2 + 2*log(3*y), z)
expr
```

```python
print(latex(expr))
```

```python
print(ccode(expr))
```

```python
print(fcode(expr))
```

```python
expr = Integral(exp(-x**2-y**2), (x, 0, oo), (y, 0, oo))
expr
```

```python
print(latex(expr))
```

```python
print(python(expr))
```

# Detailed Example: The Quantum Harmonic Oscillator

Let us now see how we can use what we have learned about SymPy to treat a realistic physical problem: the 1-d quantum harmonic oscillator (following Griffiths Quantum Mechanics).  In particular, we wish to find the wavefunctions and energy levels of a particle of mass $m$ in a potential $V(x) = \frac{1}{2} m \omega^2 x^2$.  We need to solve the time-independent Schrodinger equation.
$$-\frac{\hbar^2}{2m}\frac{d^2\phi}{dx^2} + \frac{1}{2} m \omega^2 x^2 \phi = E \phi$$
Let's put this in a slightly more convenient form
$$ \frac{d^2\phi}{d\xi^2} = (\xi^2 - K)\phi $$
where we have defined $\xi \equiv \sqrt{\frac{m\omega}{\hbar}}x$ and $K \equiv \frac{2E}{\hbar\omega} $.  We will now define the relevant quantities for SymPy manipulation.

```python
phi, f = symbols('phi f', cls=Function)
xi, K, c = symbols('xi K c')
```

Notice that for large $\xi$, we can neglect the $K$ on the right hand side of the equation.  In this regime, we can easily find approximate solutions to the approximate version of Schrodinger's equation.

```python
schrod_approx = Eq(diff(phi(xi),xi,xi), xi**2*phi(xi))
schrod_approx
```

```python
A, B = symbols('A B')
sol_approx = A*exp(xi**2/2) + B*exp(-xi**2/2)
sol_approx
```

Let's check that this is indeed an approximate solution.

```python
diff(sol_approx,xi,xi)
```

We see that in the regime $\xi\rightarrow\infty$, this is indeed an approximate solution to Schrodinger's equation.  We can also see that the term proportional to $A$ blows up as $\xi\rightarrow\infty$ and is thus unphysical.  We therefore take the term proportional to $B$ to be our approximate solution in this regime.  We can the simplify the subsequent steps by separating out this exponential piece of the solution.  Let us define
$$\phi(\xi) \equiv h(\xi) e^{-\xi^2/2}$$

```python
h = symbols('h', cls=Function)
phi_sub = h(xi)*exp(-xi**2/2)
phi_sub
```

```python
phi_prime = diff(phi_sub, xi)
phi_prime
```

```python
phi_prime_prime = diff(phi_prime, xi)
phi_prime_prime
```

```python
schrod_h = simplify(phi_prime_prime - (xi**2-K)*phi_sub)
schrod_h
```

We can now solve the simpler differential equation without the exponential part.

```python
schrod_h_simp = simplify(schrod_h * exp(xi**2/2))
schrod_h_simp
```

This type of differential equation can typically be solved as a power series.

```python
j, n = symbols('j n', integer=True)
schrod_series = schrod_h_simp.subs(h(xi), Sum(f(j)*xi**j,(j,0,oo)))
schrod_series
```

```python
schrod_series.doit()
```

Now we can read off the coefficient of $\xi^j$ which gives a recursive expression for $f(j)$.

```python
schrod_recursive = Eq((j+1)*(j+2)*f(j+2) - 2*j*f(j) + (K - 1) * f(j),0)
schrod_recursive
```

```python
Eq(f(j+2),solve(schrod_recursive, f(j+2))[0])
```

This defines a recursive relation for the coefficients of the power series solution which depends on the two arbitrary constants $f(0)$ and $f(1)$.  In order for our soution to be normalizable, we must require that the series truncates, and so $f(n) = 0$ for some finite $n$ and also that $f(0) = 0$ for $n$ odd and $f(1) = 0$ for $n$ even.  We therefore find that $K = 2n+1$ and so
$$ E_n = \left( n+\frac{1}{2}\right) \hbar \omega \, $$
With this choice of $K$, we have
$$ f(j+2) = \frac{-2(n-j)}{(j+1)(j+2)}f(j) $$

```python
def h_sol(n):
    g = symbols('g', cls=Function)
    F = 0
    if n % 2 == 0:
        F = c
        g = F
        for j in range (2, n+1, 2):
            F *= S(-2)*(n-(j-2))/((j-1)*j)
            g += F*xi**j
    else:
        F = c
        g = F*xi
        for j in range (3, n+1, 2):
            F *= S(-2)*(n-(j-2))/((j-1)*j)
            g += F*xi**j
    return g
```

```python
h_sol(5)
```

Finally, we arrive at the wavefunctions for the stationary states of the quantum harmonic oscillator.

```python
phi_sol = h_sol(5)*exp(-xi**2/2)
phi_sol
```

```python
norm_c = solve(integrate(phi_sol**2, (xi, -oo, oo))-1,c)[1]
norm_c
```

```python
norm_phi_sol = simplify(phi_sol.subs(c, norm_c))
norm_phi_sol
```

```python
for i in range(0,6):
    phi_sol = h_sol(i)*exp(-xi**2/2)
    norm_c = solve(integrate(phi_sol**2, (xi, -oo, oo))-1,c)[1]
    norm_phi_sol = simplify(phi_sol.subs(c, norm_c))
    p1 = plot(norm_phi_sol, norm_phi_sol**2, (xi, -5, 5))
```

```python

```


```python
%matplotlib inline
from IPython.display import display,Math
from sympy import *
init_session()
```

    IPython console for SymPy 1.8 (Python 3.7.10-64-bit) (ground types: gmpy)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.8/

```python
%%time
# ユークリッドの互除法
a, b = 20210608, 80601202
r = a%b
count = 1
while r != 0:
    a,b = b,r
    r = a%b
    count += 1
print(b,count)
```

    22 16
    CPU times: user 57 µs, sys: 0 ns, total: 57 µs
    Wall time: 60.3 µs

```python
%%time
# ユークリッドの互除法　余りの改良
a, b = 20210608, 80601202
r = a%b
count = 1
while r != 0:
    a,b = b,r
    r = a%b
    count += 1
    if b < 2*r:
        r = b-r
print(b,count)
```

    22 11
    CPU times: user 239 µs, sys: 56 µs, total: 295 µs
    Wall time: 208 µs

```python
# 拡張ユークリッドの互除法
def myexgcd(a,b):
    x, y, u, v = 1, 0, 0, 1
    while b != 0:
        q = a // b
        x -= q * u
        y -= q * v
        x, u = u, x
        y, v = v, y
        a, b = b, a % b
    return x, y
```

```python
# 動作確認
a,b = 123,321
x,y = myexgcd(a,b)
print("{:d}*{:d}{:+d}*{:d}={:d}".format(x,a,y,b,x*a+y*b))
```

    47*123-18*321=3

```python
from ipywidgets import interact
import time
def myexgcd(a,b):
    x, y, u, v = 1, 0, 0, 1
    while b != 0:
        q = a // b
        x -= q * u
        y -= q * v
        x, u = u, x
        y, v = v, y
        a, b = b, a % b
    return x, y
@interact
def _(a="314159265",b="35"):
    digits = len(b)
    a,b = int(a),int(b)
    x,y = myexgcd(a,b)
    return display(Math("{:d}*{:d}{:+d}*{:d}={:d}".format(x,a,y,b,x*a+y*b)))
```

    interactive(children=(Text(value='314159265', description='a'), Text(value='35', description='b'), Output()), …

```python

```


# Mathematical Theory and Intuition behind Our Improved Model

This markdown explains the mathematical theory behind our model. Note that the actual implementation might be different. For simplicity, we are also not including all the datasets we used (otherwise the equations might be too long).

Let $i$ denote the $i$th date since the beginning of our dataset and $n$ denote the cutoff date for training-testing . Let the range from $n+1$ to $m$ be the range for testing (forecast).

For a given date $i$, let $\textbf{asp}(i)$ be the actual index for S\&P500; $\textbf{fbsp}(i)$ be the index for S\&P500 bases on a training set of the first $i-1$ many days' market data of S\&P500; $\textbf{div}(i)$ be the dividend yield rates of S\&P500; $\textbf{eps}(i)$ be the dividen yield rates of S\&P500; $\textbf{tby}(i)$ be the 10-year treasury bond yield; $\textbf{ffr}(i)$ be the fed fund rates;  $\textbf{fta}(i)$ be the fed total assets.

## Linear Model

\begin{equation}
    \textbf{fb}_1(i,a,b,c,d,e):=\textbf{fbsp}(i)+a*\textbf{tby}(i)+b*\textbf{ffr}(i)+c*\textbf{fta}(i)+d*\textbf{div}(i)+e*\textbf{eps}(i)
\end{equation}

\begin{equation}
    \textbf{E}_n(a,b,c,d,e):= \frac{1}{n-1000}\sum_{i=1000}^{n}
    (\textbf{fb}_1(i,a,b,c,d,e)-\textbf{asp}(i))^2
\end{equation}

attains its minimum. Namely, finding

\begin{equation}
    (a_n,b_n,c_n,d_n,e_n):=\text{argmin} \textbf{E}_n(a,b,c,d,e)
\end{equation}

Note that it doesn't make sense to start with $i=1$ since the fbprophet itself need to use the first $i-1$ of the data for training

## Nonlinear Model

All notations are the same as above.

Here is a different model (nonlinear revision of fbprophet). In this model, we will the dividend yield rate of S\&P 500.

First, what is the dividend yield rate? Dividend is the money that publicly listed companies pay you (usually four times a year) for holding their stocks (until the so-called ex-dividend dates). It's like the interest paid to your saving accounts from the banks. Some companies pay while some don't, especially for the growth tech stocks. From my experience, the impact of bond rates and fed fund rates on the stock market will change when they rise above or fall below the dividend yield rate}. Stock prices fall when those rates rise above the dividend yield rate of SP500 (investor are selling their stocks to buy bonds or save more money in their bank account!).

Based on this idea, it might be useful to consider the differences of those rates and the dividend yield rate of SP500.

Normally an increase in the federal fund rate will result in an increase in bank loan interest rate, which will in turn result in an decrease in the net income of S\&P500-listed companies since they have to pay a higher interest when borrowing money from banks. Based on this thought, I believe it is reasonable to make a correction to $c*\textbf{eps}(i)$ by replacing the term by $c*\textbf{eps}(i)(1+d*\textbf{ffr}(i))$. If my intuition is correct, the generated constant $d$ from the optimization should be a negative number.

$$\textbf{fb}_2(i,a,b,c,d,e):= \textbf{fbsp}(i)*\big[1+a*(\textbf{div}(i)-\textbf{tby}(i))\big]\big[1+b*(\textbf{div}(i)-\textbf{ffr}(i))\big]\\
+c*\textbf{eps}(i)(1+d*\textbf{ffr}(i)+e*\textbf{fta}(i))$$

and consider

$$E_n(a,b,c,d,e) := \frac{1}{n-1000}\sum_{i=1000}^n(\textbf{fb}_2(i,a,b,c,d,e)-\textbf{asp}(i))^2$$

Now find (by approximation, SGD, etc.) $(a_n,b_n,c_n,d_n,e_n):=\text{argmin} E_n(a,b,c,d,e)$

Using $(a_n,b_n,c_n,d_n,e_n)$ as constants, our output will be $\textbf{fb}_2(i,a_n,b_n,c_n,d_n,e_n)$

### The actual implementation of the nonlinear model

For the actual implementation of the nonlinear model, we threw away the higher order terms (the products of three things) since they are relatively smaller quantities

### The mathematical theory behind our nonlinear model:

#### The Taylor's theorem for multivariate functions

Let $f :\mathbb{R}^n → \mathbb R$ be a $k$-times-differentiable function at the point $a ∈ \mathbb{R}^n$. Then there exists $h_a:\mathbb{R}^n → \mathbb R$ such that:

<math>\begin{align}
& f(\boldsymbol{x}) = \sum_{|\alpha|\leq k} \frac{D^\alpha f(\boldsymbol{a})}{\alpha!} (\boldsymbol{x}-\boldsymbol{a})^\alpha  + \sum_{|\alpha|=k} h_\alpha(\boldsymbol{x})(\boldsymbol{x}-\boldsymbol{a})^\alpha, \\
& \mbox{and}\quad \lim_{\boldsymbol{x}\to \boldsymbol{a}}h_\alpha(\boldsymbol{x})=0.
\end{align}</math>

This approximation is also optimal in some sense.

In our model, we think of $\textbf{asp}$ as a function of $\textbf{fbsp}, \textbf{tby}, \textbf{div}, \textbf{ffr}, \textbf{fta}$, etc. Say

$$\textbf{asp}:=F(\textbf{fbsp}, \textbf{tby}, \textbf{div}, \textbf{ffr}, \textbf{fta},\cdots)$$

With the assumption that $F$ here is regular, we can apply the Taylor's theorem for multivariate functions from above to $F$. Ideally, we have to consider all possible products in our implementation. But in our implementation, we chose to make a balance between our intuitive nonlinear model and Taylor's theorem.

## A faster computation scheme
One drawback of the models we proposed about is that we have to call fbprophet thousands of times when we implement it.

Here is a method that reduces the number of times calling fbprophet:

Say in the training process we are considering from i=1,000 to 11,000. Namely based on the current scheme, we have to call fbprophet for 10,000 times.

Instead of this, we make a break-down 10,000=100*100 as follows:

For i=1,000 to 1,100, we only use the first 999 dates for training;

For i=1,100 to 1,200, we only use the first 1,099 dates for training;

.............................................

For i=10,900 to 11,000, we only use the first 10,899 dates for training;

In this way, it seems that we only need to call fbprophet for 100 times. And this doesn't seem harm the accuracy too much.

```python
import pandas as pd
import numpy as np

## For plotting
import matplotlib.pyplot as plt
from matplotlib import style
import datetime as dt
import seaborn as sns
sns.set_style("whitegrid")
```

```python
path = '../Data/dff2.csv'
```

```python
df= pd.read_csv(path, parse_dates=['ds'])
# df = df.rename(columns = {"Date":"ds","Close":"y"})
df = df[['ds', 'y','fbsp', 'diff','tby', 'ffr', 'fta', 'eps', 'div', 'une', 'wti', 'ppi',
       'rfs']]
# df
```

```python
df['fbsp_tby'] = df['fbsp'] * df['tby']
df['fbsp_ffr'] = df['fbsp'] * df['ffr']
df['fbsp_div'] = df['fbsp'] * df['div']
df['eps_tby'] = df['eps'] * df['tby']
df['eps_ffr'] = df['eps'] * df['ffr']
df['eps_div'] = df['eps'] * df['div']
```

```python
# cutoff between test and train data
cutoff = len(df) - 252
df_train = df[:cutoff].copy()
df_test = df[cutoff:].copy()
print(cutoff)
```

    2300

```python
df_train.columns
```

    Index(['ds', 'y', 'fbsp', 'diff', 'tby', 'ffr', 'fta', 'eps', 'div', 'une',
           'wti', 'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby',
           'eps_ffr', 'eps_div'],
          dtype='object')

```python
possible_features = ['tby', 'ffr', 'fta', 'eps', 'div', 'une', 'wti',
       'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby',
       'eps_ffr', 'eps_div']

from itertools import chain, combinations

def powerset(iterable):
    #"powerset([1,2,3]) --> () (1,) (2,) (3,) (1,2) (1,3) (2,3) (1,2,3)"
    s = list(iterable)
    return chain.from_iterable(combinations(s, r) for r in range(len(s)+1))

#print(list(powerset(possible_features)))
```

```python
len(possible_features)
```

    15

```python
from statsmodels.regression.linear_model import OLS

reg_new = OLS((df_train['diff']).copy(),df_train[possible_features].copy()).fit()
print(reg_new.params)

#from the output, we can see it's consistent with sklearn output
```

    tby          99.694834
    ffr         298.736913
    fta          -0.000051
    eps          -9.542566
    div        -575.728680
    une         -57.231481
    wti           2.970708
    ppi          -8.724101
    rfs           0.008933
    fbsp_tby     -0.099985
    fbsp_ffr     -0.269213
    fbsp_div     -0.276994
    eps_tby       0.947387
    eps_ffr       2.839324
    eps_div       6.401360
    dtype: float64

```python
new_coef = reg_new.params
new_possible_feats = new_coef[abs(new_coef)>0].index

power_feats = list(powerset(new_possible_feats))
power_feats.remove(())

power_feats = [ list(feats) for feats in power_feats]
len(power_feats)

```

    32767

```python
AIC_scores = []
parameters = []

for feats in power_feats:
    tmp_reg = OLS((df_train['diff']).copy(),df_train[feats].copy()).fit()
    AIC_scores.append(tmp_reg.aic)
    parameters.append(tmp_reg.params)

Min_AIC_index = AIC_scores.index(min(AIC_scores))
Min_AIC_feats = power_feats[Min_AIC_index]
Min_AIC_params  = parameters[Min_AIC_index]
print(Min_AIC_feats)
print(Min_AIC_params)
```

    ['tby', 'ffr', 'fta', 'eps', 'div', 'une', 'wti', 'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby', 'eps_ffr', 'eps_div']
    tby          99.694834
    ffr         298.736913
    fta          -0.000051
    eps          -9.542566
    div        -575.728680
    une         -57.231481
    wti           2.970708
    ppi          -8.724101
    rfs           0.008933
    fbsp_tby     -0.099985
    fbsp_ffr     -0.269213
    fbsp_div     -0.276994
    eps_tby       0.947387
    eps_ffr       2.839324
    eps_div       6.401360
    dtype: float64

```python
len(Min_AIC_feats)
```

    15

```python
###After selecting the best features, we report the testing error, and make the plot
AIC_df_test = df_test[Min_AIC_feats]
AIC_pred_test = AIC_df_test.dot(Min_AIC_params)+df_test.fbsp

AIC_df_train = df_train[Min_AIC_feats]
AIC_pred_train = AIC_df_train.dot(Min_AIC_params)+ df_train.fbsp

```

```python
from sklearn.metrics import mean_squared_error as MSE

mse_train = MSE(df_train.y, AIC_pred_train)
mse_test = MSE(df_test.y, AIC_pred_test)

#compare with fbprophet()

fb_mse_train = MSE(df_train.y, df_train.fbsp)
fb_mse_test = MSE(df_test.y, df_test.fbsp)

print(mse_train,fb_mse_train)

print(mse_test,fb_mse_test)
```

    2946.8595927656197 18713.80955029442
    325321.3852693186 98769.80247438994

```python
df_train.ds
```

    0      2010-11-16
    1      2010-11-17
    2      2010-11-18
    3      2010-11-19
    4      2010-11-22
              ...
    2295   2020-01-22
    2296   2020-01-23
    2297   2020-01-24
    2298   2020-01-27
    2299   2020-01-28
    Name: ds, Length: 2300, dtype: datetime64[ns]

```python
plt.figure(figsize=(18,10))

# plot the training data
plt.plot(df_train.ds,df_train.y,'b',
            label = "Training Data")

plt.plot(df_train.ds, AIC_pred_train,'r-',
            label = "Improved Fitted Values by Best_AIC")

# # plot the fit
plt.plot(df_train.ds, df_train.fbsp,'g-',
            label = "FB Fitted Values")

# # plot the forecast
plt.plot(df_test.ds, df_test.fbsp,'g--',
            label = "FB Forecast")
plt.plot(df_test.ds, AIC_pred_test,'r--',
            label = "Improved Forecast by Best_AIC")
plt.plot(df_test.ds,df_test.y,'b--',
            label = "Test Data")

plt.legend(fontsize=14)

plt.xlabel("Date", fontsize=16)
plt.ylabel("SP&500 Close Price", fontsize=16)

plt.show()
```

```python
plt.figure(figsize=(18,10))
plt.plot(df_test.y,label="Training Data")
plt.plot(df_test.fbsp,label="FB Forecast")
plt.plot(AIC_pred_test,label="Improved Forecast by Best_AIC")
plt.legend(fontsize = 14)
plt.show()
```

```python

```

```python
column = ['tby', 'ffr', 'fta', 'eps', 'div', 'une',
                                                  'wti', 'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby', 'eps_ffr', 'eps_div']
```

```python
from sklearn import preprocessing
df1_train = df_train[['diff', 'tby', 'ffr', 'fta', 'eps', 'div', 'une', 'wti', 'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby', 'eps_ffr', 'eps_div']]

X = preprocessing.scale(df1_train)
from statsmodels.regression.linear_model import OLS

reg_new = OLS((X[:,0]).copy(),X[:,1:].copy()).fit()
print(reg_new.params)
```

    [ 0.65150054  1.70457239 -0.1573802  -0.18007979 -0.15221931 -0.62326075
      0.45065894 -0.38972706  2.87210843 -1.17604495 -4.92858316 -2.15459111
      0.11418468  2.74829778  0.55520382]

```python
# Before Covid
# pd.Series(reg_new.params, index=['tby', 'ffr', 'fta', 'eps', 'div', 'une',
#                                                   'wti', 'ppi', 'rfs', 'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby', 'eps_ffr', 'eps_div'] )
```

```python
# before covid
coef1 = [ 1.50405129,  1.03228322,  0.27409454,  1.17073571,  0.31243092,
       -0.75747342,  0.46988206, -0.39944639,  2.10369448, -0.69112943,
       -2.1804296 , -2.38576385, -1.14196633,  1.41832903, -0.34501927]
# include covid
coef2 = [ 0.65150054,  1.70457239, -0.1573802 , -0.18007979, -0.15221931,
       -0.62326075,  0.45065894, -0.38972706,  2.87210843, -1.17604495,
       -4.92858316, -2.15459111,  0.11418468,  2.74829778,  0.55520382]
```

```python
# Include Covid
# pd.Series( np.append( ['coefficients (before covid)'], np.round(coef1,3)),  index= np.append(['features'], column) )

```

```python
index1 = ['10 Year U.S Treasury Bond Yield Rates (tby)', 'Federal Funds Rates (ffr)',
        'Federal Total Assets (fta)', 'Earning-Per-Share of S&P 500 (eps)', 'Dividend Yield of S&P 500 (div)',
        'Unemployment Rates (une) ', 'West Texas Intermediate oil index (wit)', 'Producer Price Index (ppi)',
         'Retail and Food Services Sales (rfs)',
         'fbsp_tby', 'fbsp_ffr', 'fbsp_div', 'eps_tby', 'eps_ffr', 'eps_div'
        ]
```

```python
len(index1)
```

    15

```python
pd.Series(coef2, index =index1)
```

    10 Year U.S Treasury Bond Yield Rates (tby)    0.651501
    Federal Funds Rates (ffr)                      1.704572
    Federal Total Assets (fta)                    -0.157380
    Earning-Per-Share of S&P 500 (eps)            -0.180080
    Dividend Yield of S&P 500 (div)               -0.152219
    Unemployment Rates (une)                      -0.623261
    West Texas Intermediate oil index (wit)        0.450659
    Producer Price Index (ppi)                    -0.389727
    Retail and Food Services Sales (rfs)           2.872108
    fbsp_tby                                      -1.176045
    fbsp_ffr                                      -4.928583
    fbsp_div                                      -2.154591
    eps_tby                                        0.114185
    eps_ffr                                        2.748298
    eps_div                                        0.555204
    dtype: float64

```python
df3 = pd.DataFrame(coef1, index = index1, columns = ['coefficients (before covid)'])
df3['coefficients (include covid)'] =pd.Series(coef2, index =index1)
df3
```

      coefficients (before covid)
      coefficients (include covid)

      10 Year U.S Treasury Bond Yield Rates (tby)
      1.504051
      0.651501

      Federal Funds Rates (ffr)
      1.032283
      1.704572

      Federal Total Assets (fta)
      0.274095
      -0.157380

      Earning-Per-Share of S&P 500 (eps)
      1.170736
      -0.180080

      Dividend Yield of S&P 500 (div)
      0.312431
      -0.152219

      Unemployment Rates (une)
      -0.757473
      -0.623261

      West Texas Intermediate oil index (wit)
      0.469882
      0.450659

      Producer Price Index (ppi)
      -0.399446
      -0.389727

      Retail and Food Services Sales (rfs)
      2.103694
      2.872108

      fbsp_tby
      -0.691129
      -1.176045

      fbsp_ffr
      -2.180430
      -4.928583

      fbsp_div
      -2.385764
      -2.154591

      eps_tby
      -1.141966
      0.114185

      eps_ffr
      1.418329
      2.748298

      eps_div
      -0.345019
      0.555204

```python

```

Code/Scenario2 including covid period.ipynb


```python
import matplotlib.pyplot as plt
import numpy as np
```

# Motivation

During my master studies working through assignments in probability theory I spend a lot of time struggling with a task, which seemed to be very easy at the first glance. Here it is:

Given two independent uniformly distributed random variables $X$ and $Y$ (i.e. $X ⊥ Y, X,Y \sim U([0,1 ])$), determine the probability density function p(z) of $Z=X+Y$.

After a quick web search I found the theoretical tool, which is required need to accomplish the task - [Convolution of probability distributions](https://en.wikipedia.org/wiki/Convolution_of_probability_distributions). It states:

The general formula for the distribution of the sum $Z=X+Y$ of two independent integer-valued (and hence discrete) random variables is

\begin{equation}
P(Z=z) = \sum_{k=-\infty}^\infty P(X=k)P(Y=z-k)
\end{equation}

The counterpart for independent continuously distributed random variables with density functions $f, g$ is

\begin{equation}
h(z)=(f*g)(z)=\int_{-\infty}^\infty f(z-t)g(t) dt = \int_{-\infty}^\infty f(t)g(z-t) dt
\end{equation}

My first thought was "oh, I saw those integrals back in control theory lectures years ago" - but years went by and I had no idea anymore how the integral

\begin{equation}
(f*g)(z) = \int_{-\infty}^\infty f(t)g(z-t) dt
\end{equation}

is calculated - there are some fancy visualitions in wikipedia like this one, but for me it was hard to decipher the meaning of the different variables within the equation.

The goal of this blogpost is to go step by step over the basics - in the end you should feel comfortable with convolutions plus you could calculate them in Python, too.

Enjoy!

# Basics

## Altering fuction graphs

You probably had this chapter in school, teachers do that usually with quadratic functions. If you know what happens for function $f$ if you alter $f(x)$ to $f(-x+3)$ feel free to skip this section.

The motivation of all this is to understand the term $g(z-x)$ which is changed from $g(x)$. Note that, I replaced the $t$ with $x$ in order not to confuse the reader with $t$ as time.

Let's define two simple functions to work through this tutorial. The first will be a simple quadratic function

\begin{equation}
f_1(x)=x^2
\end{equation}

As I want to solve my task, too - we will have a uniform distribution $U$ as our second function

\begin{equation}
f_2(x)=\begin{cases}
    \frac{1}{b - a} & \mathrm{for}\ a \le x \le b, \\[8pt]
    0 & \mathrm{for}\ xb
\end{cases}
\end{equation}

Let's implement them in Python:

```python
def f1(x):
    return x**2
```

```python
def f2(x, a, b):
    const = 1/(b-a)
    if a <= x and x <= b:
        return const
    else:
        return 0
```

```python
x_vec = np.linspace(-10, 10, 10000)
f1_vec = f1(x_vec)
f2_vec = np.asarray([f2(x, 0, 1) for x in np.nditer(x_vec)])

plt.figure(figsize=(15,5))
plt.plot(x_vec, f1_vec, label="$f_1(x)=x^2$")
plt.plot(x_vec, f2_vec, label="$f_2(x)=U([0, 1])(x)$")
plt.xlim([-4, 4])
plt.ylim([-0.1, 2])
plt.grid()
plt.legend()
plt.xlabel("x")
plt.ylabel("function value")
plt.title("functions graphs $f_1, f_2$ visualized")
plt.savefig("001_f1_and_f2.png", dpi=300)
```

Now what happens when we add a constant $z$ to the function argument (i.e. $f(x+3)$)? Let's see what happens for different values of $z$.

```python
k_array = [-2, 1]
```

```python
x_vec = np.linspace(-10, 10, 10000)

plt.figure(figsize=(15,5))
plt.plot(x_vec, f1(x_vec), label="$f_1(x)$")
for k in k_array:
    label_text = "$f_1(x%+d)$ with z=%d"%(k,k)
    plt.plot(x_vec, f1(x_vec+k), label=label_text, alpha=0.8, ls="--")
plt.xlim([-4, 4])
plt.ylim([-0.1, 2])
plt.grid()
plt.legend()
plt.xlabel("x")
plt.ylabel("function value")
plt.title("Altering $f_1$ by adding a constant $z$ to the function argument $f(x+z)$")
plt.savefig("002_f1_shift_z.png", dpi=300)
```

```python
x_vec = np.linspace(-10, 10, 10000)
f2_vec = np.asarray([f2(x, 0, 1) for x in np.nditer(x_vec)])

plt.figure(figsize=(15,5))
plt.plot(x_vec, f2_vec, label="$f_2(x)$")
for k in k_array:
    label_text = "$f_2(x%+d)$ with z=%d"%(k,k)
    f2_vec = np.asarray([f2(x, 0-k, 1-k) for x in np.nditer(x_vec)])
    plt.plot(x_vec, f2_vec, label=label_text, alpha=0.8, ls="--")
plt.xlim([-4, 4])
plt.ylim([-0.1, 2])
plt.grid()
plt.legend()
plt.xlabel("x")
plt.ylabel("function value")
plt.title("Altering $f_2$ by adding a constant $z$ to the function argument $f(x+z)$")
plt.savefig("003_f2_shift_z.png", dpi=300)
```

Adding a positive number $z$, we shift the graph to the left, with a negative $z$ to the right.

What happens if we multiply $-1$ to the function argument $x$?

```python
x_vec = np.linspace(-10, 10, 10000)

plt.figure(figsize=(15,5))
plt.plot(x_vec, f1(x_vec), label="$f_1(x)$")

label_text = "$f_1(-x)$"
plt.plot(x_vec, f1(-x_vec), label=label_text, alpha=0.8, ls="--")
plt.xlim([-4, 4])
plt.ylim([-0.1, 2])
plt.axvline(0, color="k", linewidth=0.7)
plt.grid()
plt.legend()
plt.xlabel("x")
plt.ylabel("function value")
plt.title("Altering $f_1$ by inverting the sign of the function argument $f(-x)$")
plt.savefig("004_f1_invert_x.png", dpi=300)
```

```python
x_vec = np.linspace(-10, 10, 10000)
f2_vec = np.asarray([f2(x, 0, 1) for x in np.nditer(x_vec)])

plt.figure(figsize=(15,5))
plt.plot(x_vec, f2_vec, label="$f_2(x)$")

label_text = "$f_2(-x)$"
f2_vec_minus = np.asarray([f2(x, -1, 0) for x in np.nditer(x_vec)])
plt.plot(x_vec, f2_vec_minus, label=label_text, alpha=0.8, ls="--")
plt.xlim([-4, 4])
plt.ylim([-0.1, 2])
plt.grid()
plt.legend()
plt.axvline(0, color="k", linewidth=0.7)
plt.xlabel("x")
plt.ylabel("function value")
plt.title("Altering $f_2$ by inverting the sign of the function argument $f(-x)$")
plt.savefig("005_f2_invert_x.png", dpi=300)
```

Apperently, this change "mirrors" the function graph on the y-axis.

Now we can understand the tranformation $g(t)$ to $g(z-t)$. It is adding a constant $z$ and multiplying $-1$ to $t$.

```python
x_vec = np.linspace(-10, 10, 10000)
f1_vec = f1(-x_vec+2)
f2_vec = np.asarray([f2(-x, 0-2, 1-2) for x in np.nditer(x_vec)])

plt.figure(figsize=(15,5))

f1_vec = f1(x_vec)
f2_vec = np.asarray([f2(x, 0, 1) for x in np.nditer(x_vec)])
plt.plot(x_vec, f1_vec, label="$f_1(x)$")
plt.plot(x_vec, f2_vec, label="$f_2(x)$")

"""
f1_vec = f1(x_vec+2)
f2_vec = np.asarray([f2(x, 0-2, 1-2) for x in np.nditer(x_vec)])
plt.plot(x_vec, f1_vec, label="$f_1(2+x)$", color="blue", ls="-.")
plt.plot(x_vec, f2_vec, label="$f_2(2+x)$", color="orange", ls="-.")
"""
f1_vec = f1(-x_vec+2)
f2_vec = np.asarray([f2(-x, 0-2, 1-2) for x in np.nditer(x_vec)])
plt.plot(x_vec, f1_vec, label="$f_1(2-x)$", color="blue", ls="--")
plt.plot(x_vec, f2_vec, label="$f_2(2-x)$", color="orange", ls="--")
plt.xlim([-4, 4])
plt.ylim([-0.1, 2])
plt.grid()
plt.legend()
plt.xlabel("x")
plt.ylabel("function value")
plt.title("functions graphs $f_1, f_2$ altered to $f(z-x)$")
plt.savefig("006_f1_f2_altered.png", dpi=300)
```

## Multiplying two functions

Let's continue to work through the convolution calculation formula - our next task is to undestand the product of two functions (or in my case density distributions):

\begin{equation}
f(x)g(z-x)
\end{equation}

What happens if we multiply them?

Let's define a proxy function $a$ as a product of $f_2(x)$ and $f_2(0.5-x)$ and visualize it

\begin{equation}
a(x) = f_2(x)f_2(0.5-x)
\end{equation}

```python
x_vec = np.linspace(-10, 10, 10000)
f2_vec = np.asarray([f2(x, 0, 1) for x in np.nditer(x_vec)])

plt.figure(figsize=(15,5))
plt.plot(x_vec, f2_vec, label="$f_2(x)$", linewidth=0.9)

label_text = "$f_2(0.5-x)$"
f2_vec2 = np.asarray([f2(-x, -1.5, -0.5) for x in np.nditer(x_vec)])
plt.plot(x_vec, f2_vec2, label=label_text, linewidth=0.9)

h_t = np.multiply(f2_vec, f2_vec2)
plt.plot(x_vec, h_t, label="$a(x)$", alpha=0.6, ls="--", color="k")
plt.xlim([-1, 4])
plt.ylim([-0.1, 1.5])
plt.grid()
plt.legend()
plt.axvline(0, color="k", linewidth=0.7)
plt.xlabel("x")
plt.ylabel("function value")
plt.title("Product $a(t)$ of two functions $f_2(x)f_2(0.5-x)$")
plt.savefig("007_a_x_f1_f2_multiplied.png", dpi=300)
```

The black dotted graph is the product we we integrate on in the next step.

## Integral

An integral is the area between the function graph and the x-axis. This is the last part of our convolution equation

\begin{equation}
\int_{-\infty}^\infty a(x) dx = \int_{-\infty}^\infty f(x)g(z-x) dx
\end{equation}

```python
f2_vec = np.asarray([f2(x, 0, 1) for x in np.nditer(x_vec)])
f2_vec2 = np.asarray([f2(-x, -1.5, -0.5) for x in np.nditer(x_vec)])
h_t = np.multiply(f2_vec, f2_vec2)

plt.figure(figsize=(15,5))
plt.plot(x_vec, h_t, label="$a(x)$", ls="-")

plt.fill_between(x_vec, 0, h_t, hatch="/", facecolor='yellow', label="$\int_{-\infty}^\infty a(x) dx = 0.5$")
plt.xlim([-1, 4])
plt.ylim([-0.1, 1.5])
plt.grid()
plt.legend()
plt.axvline(0, color="k", linewidth=0.7)
plt.xlabel("x")
plt.ylabel("function value")
plt.title("Integral $\int_{-\infty}^\infty a(x) dx$")
plt.savefig("008_a_x_integrated.png", dpi=300)
```

# Building blocks in one picture

## Principle

Now we are ready to understand the convolution equation

\begin{equation}
h(z) = (f*g)(z) = \int_{-\infty}^\infty f(x)g(z-x) dx
\end{equation}

In order to calculate $h(z)$ for a particular $z$ we have to do following steps:

1. Shift function $g(x)$ to the left $z$, i.e. $g(x+z)$
2. Mirror the result relative to the y-axis, i.e. $g(-x+z)=g(z-x)$
3. Calculate the product of $f(x)g(z-x)=a(x)$
4. Calculate the infinite integral over $a(x) = i_z$
5. The result of the convolution at particulat position $z$ is $i_z$, i.e. we calculated $h(z)=\int_{-\infty}^\infty f(x)g(z-x) dx$.

## Back to homework

Now I believe we can solve our random variable sum assignment $Z=X+Y$:

\begin{equation}
p_Z(Z=z)=\int_{-\infty}^\infty p_X(X=(z-x))p_Y(Y=x) dx
\end{equation}

Let's apply our algorithm developed above and see what will be the result.

```python
from scipy.integrate import trapz
```

```python
def p_Z(z):
    x_vec = np.linspace(-5, 5, 1000)
    px_vec = np.asarray([f2(x, 0, 1) for x in np.nditer(x_vec)])
    py_vec_altered = np.asarray([f2(-x, 0-z, 1-z) for x in np.nditer(x_vec)])
    a_x = np.multiply(px_vec, py_vec_altered)
    return trapz(a_x, x_vec)
```

```python
z_vec = np.linspace(-1, 3, 200)
pz_vec= np.asarray([p_Z(x) for x in np.nditer(z_vec)])
```

```python
plt.figure(figsize=(15,5))
plt.plot(z_vec, pz_vec, label="$p_Z(Z=z)$", alpha=0.6, ls="--", color="k")
plt.xlim([-1, 3])
plt.ylim([-0.1, 1.5])
plt.grid()
plt.legend()
#plt.axvline(0, color="k", linewidth=0.7)
plt.xlabel("z")
plt.ylabel("function value")
plt.title("Probability density of $Z=Y+X$ with $X ⊥ Y, X,Y \sim U([0,1 ])$")
plt.savefig("009_px_py_convolution.png", dpi=300)
```

Nice! We can see that the sum of two equally distributed random variables will lead to a triangle formed probability density! It also sounds natural because the expected value of $X$ and $Y$ are both $0.5$ - so the expected value of $Z$ (the focal point of the density) will be at $z=1$!

```python

```