# Interpolation and reconstruction

It's not obvious how to represent a continuous-time signal $x(t)$ in a way that can be manipulated by a computer.  The time index is real-valued, so even if we only care about the signal value over a finite interval there are infinitely many function values to consider and store.

One approach is to assume that $x(t)$ varies slowly, and hope that sampling $x(t)$ at $t=nT$ for $n$ integer and $T$ small is sufficient to characterise the signal.  This leads to the discrete-time signal $x[n] = x(nT)$.  When this assumption is formalised and holds then the method works, and we use it a lot.  The first part of this workbook explores how a discrete signal $x[n]$ can be used to represent or parameterise a continuous-time $x(t)$.  Understanding this link and its limitations lets us process "analog" signals $x(t)$ using digital processing of the corresponding $x[n]$.  See https://en.wikipedia.org/wiki/Digital_signal_processing.

A more general option is to represent the continuous-time signal $x(t)$ as a linear combination of a fixed set of known basis functions $b_n(t)$:
$$x(t) = \sum_{n=0}^{N-1} c_n b_n(t).$$
Here the signal $x(t)$ is defined for all $t \in \mathbb{R}$, but as written it is completely specified by the finite and discrete set of values $c_n, n=0, \ldots, N-1$.  Not every signal $x(t)$ can be written in this way, and those that can depends on the choice of functions $b_n(t)$.

One simple example uses a polynomial basis with $b_n(t) = t^n$.  The signals parameters $c_n$ are then just the coefficients for each polynomial order.  Another is the cosine basis $b_n(t) = \cos(n \omega_0 t)$, which can represent all periodic even functions $x(t)$ if $N$ is large enough.  Instead of summing from $0$ to $N-1$ in the representation we might also have infinite bounds like $0$ to $\infty$, or $-\infty$ to $\infty$.  Even then, the coefficients $c_n$ are still a countable set and can be carefully used to represent or process the continuous-time $x(t)$ that they represent.  The second part of this workbook explores this concept.

## Reconstruction from discrete samples

Suppose that $b_0(t)$ is an even function centered on the origin $t=0$, and for each $n$ the basis function $b_n(t)$ in the representation equation above is $b_0(t)$ shifted until it is centered on some point $t=nT$:
$$x(t) = \sum_{n=0}^{N-1} c_n b_0(t-nT).$$
This representation is particularly useful if we add the requirement
$$b_0(t) = \begin{cases}
  1 \qquad & t=0 \\
  0 \qquad & t=nT \quad \text{for integer $n$}.
  \end{cases}
$$

Suppose now that we have access to a discrete set of regular samples $x[n] = x(nT)$ from $x(t)$, and want to determine the corresponding coefficients $c_n$.  Note that
$$x(kT) = \sum_{n=0}^{N-1} c_n b_0(kT-nT) = \sum_{n=0}^{N-1} c_n b_0((k-n)T) = c_k.$$
The last step above follows because $b_0((k-n)T)=1$ only if $k-n=0$, so all the terms in the sum are zero except one.  You could also note that $b_0((k-n)T) = \delta(k-n)$ and use the sifting property.  Either way $x[k] = x(kT) = c_k$, and the expansion takes the form
$$x(t) = \sum_{n=0}^{N-1} x[n] b_0(t-nT).$$

We can view this as a reconstruction formula.  It takes as input the sample values in the form of a discrete signal $x[n]$ for integer $n$, and produces the interpolated or reconstructed continuous-time signal $x(t)$.  The nature of the reconstruction depends on the prototype basis function $b_0(t)$, and different choices lead to interpolation with different properties.

### Basic code

We start off with an instance of a continuous-time signal and sample it according to $x[n] = x(nT)$ for some $T$.  The first subplot below shows a continuous-time signal $x(t)$, along with dots indicating a set of discrete samples of the signal.  The second subplot shows the corresponding discrete signal $x[n]$.

```python
import numpy as np
import sympy as sp
import matplotlib.pyplot as plt
%matplotlib notebook

# Set up symbolic signal to approximate
t = sp.symbols('t');
#x = sp.cos((t-5)/5) - ((t-5)/5)**3;
x = sp.exp(sp.cos(t)+t/10)
lam_x = sp.lambdify(t, x, modules=['numpy']);

# Discrete samples of signal over required range
T = 2;  tmax = 20;
nm = np.floor(tmax/T);
tnv = T*np.arange(0,nm+1);
xnv = lam_x(tnv);

# Dense set of points for plotting "continuous" signals
tv = np.linspace(-2, tmax+2, 2500);
xv = lam_x(tv);

# Plots
fh, ax = plt.subplots(2);
ax[0].plot(tv,xv,'r-');  ax[0].scatter(tnv, xnv, c='r');
ax[0].set_ylabel('$x(t)$ and samples');  ax[0].set_xlabel('$t$');
ax[1].stem(tnv, xnv);  ax[1].set_ylabel('$x[n]$');  ax[1].set_xlabel('$n$');
```

    <IPython.core.display.Javascript object>

    /Users/nicolls/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:26: UserWarning: In Matplotlib 3.3 individual lines on a stem plot will be added as a LineCollection instead of individual lines. This significantly improves the performance of a stem plot. To remove this warning and switch to the new behaviour, set the "use_line_collection" keyword argument to True.

The code below defines and plots a triangular reconstruction or interpolation kernel $b_0(t)$ that satisfies the required conditions, namely that $b_0(nT) = \delta(n)$ for integer $n$.  Following that is the entire set of basis functions shown over the interval of interest.

```python
# Interpolation kernel
#b0 = sp.Piecewise( (0, t/T<=-1/2), (1, t/T<1/2), (0, True));  # rectangular
#b0 = sp.Piecewise( (0, t/T<=-1), (1+t/T, t<0), (1-t/T, t/T<1), (0, True));  # triangular
#at = sp.Abs(t/T);  a = -0.5;  b0 = sp.Piecewise( ((a+2)*at**3 - (a+3)*at**2 + 1 , at<1), (a*at**3 - 5*a*at**2 + 8*a*at - 4*a, at<2), (0, True));  # cubic
at = sp.Abs(t/T);  w = 3;  b0 = sp.Piecewise( (sp.sin(sp.pi*at)/(sp.pi*at), at<=w), (0, True) );  # windowed sinc
lam_b0 = sp.lambdify(t, b0.simplify(), modules=['numpy']);

# Plot kernel and reconstruction basis
tbv = np.linspace(-4*T, 4*T, 1500);
b0v = lam_b0(tbv);
fh, ax = plt.subplots(2);
ax[0].plot(tbv, b0v, 'r-');
for tn in tnv:
    ax[1].plot(tv, lam_b0(tv-tn));
ax[1].set_xlabel('t');
```

The first subplot below shows a continuous-time signal $x(t)$, the dots indicating a set of discrete samples of the signal, and the signal reconstructed from the dots using the reconstruction kernel above.  The second subplot shows the separate scaled components that are summed to form the reconstruction.  We observe that the reconstruction corresponds to first-order linear interpolation between the sample points, so $b_0(t)$ in this case is the linear interpolation reconstruction kernel.

```python
# Numerically evaluate linear interpolation approximation
xav = np.zeros(tv.shape);
for i in range(0,len(xnv)):
    xav = xav + xnv[i]*lam_b0(tv - tnv[i]);

# Plots
fh, ax = plt.subplots(2);
ax[0].plot(tv,xv,'r-',tv,xav,'b-');
ax[0].scatter(tnv, xnv, c='r');
for i in range(0,len(xnv)):
    ax[1].plot(tv,xnv[i]*lam_b0(tv - tnv[i]));
ax[1].scatter(tnv, xnv, c='r');  ax[1].set_xlabel('t');
```

The reconstruction above is quite good, but the condition that $b_0(nT) = \delta(n)$ for integer $n$ is not sufficient for it to be a good reconstruction kernel.  The code below generates a reconstruction using another basis.

```python
# Bad interpolation
b0a = sp.Piecewise( (0, t<=-T), (1+t/T, t<0), (1-t/T, t<T), (0, True))**5;
lam_b0a = sp.lambdify(t, b0a, modules=['numpy']);

# Numerically evaluate function and approximation
xava = np.zeros(tv.shape);
for i in range(0,len(xnv)):
    xava = xava + xnv[i]*lam_b0a(tv - tnv[i]);

# Plot
fh, ax = plt.subplots(2);
ax[0].plot(tv,xv,'r-',tv,xava,'b-');
ax[0].scatter(tnv, xnv, c='r');
for i in range(0,len(xnv)):
    ax[1].plot(tv,xnv[i]*lam_b0a(tv - tnv[i]));
ax[1].scatter(tnv, xnv, c='r');
```

This kernel again leads to perfect reconstruction of the sampled points, but the approximation is bad between them.  Evidently we also require that the interpolation kernel $b_0(t)$ also be smooth in some sense.  The ideal bandlimited interpolation kernel with no high-frequency components is the *sinc* function
$$b_0(t) = \frac{\sin(\pi t/T)}{\pi t/T},$$
which leads to the reconstruction below.  In this case if the signal $x(t)$ were actually bandlimited then the reconstruction would be exact.

```python
# Sinc interpolation
b0b = sp.sin(sp.pi*t/T)/(sp.pi*t/T);
lam_b0b = sp.lambdify(t, b0b, modules=['numpy']);

# Numerically evaluate function and approximation
xavb = np.zeros(tv.shape);
for i in range(0,len(xnv)):
    xavb = xavb + xnv[i]*lam_b0b(tv - tnv[i]);

# Plot
fh, ax = plt.subplots(2);
ax[0].plot(tv,xv,'r-',tv,xavb,'b-');
ax[0].scatter(tnv, xnv, c='r');
for i in range(0,len(xnv)):
    ax[1].plot(tv,xnv[i]*lam_b0b(tv - tnv[i]));
ax[1].scatter(tnv, xnv, c='r');
```

One usually has some difficulty making the approximation accurate near the boundary of the observation interval.  Dealing with the boundary of a signal that is only observed for a finite time is a difficult part of signal processing:  the models usually assume time invariance, which means that functions are observed and go on forever.

### Interpretation

The representation formula used above suggests that we can reconstruct $x(t)$ using the expression
$$x(t)= \sum_{n=-\infty}^{\infty} x[n] b_0(t-nT).$$
Here we've increased the range of the summation to include all $n$.  This can also be written as the convolution
$$x_r(t) = x_s(t) \ast b_0(t)$$
where $x_s(t)$ is the modulated impulse train
$$x_s(t) = \sum_{n=-\infty}^{\infty} x[n] \delta(t-nT).$$

The interpretation is this:  given a discrete signal $x[n]$ obtained from regular samples of $x(t)$, we can generate a reconstruction $x_r(t)$ by forming a modulated impulse train $x_s(t)$ and filtering it with the reconstruction filter $b_0(t)$.

If $x(t)$ is bandlimited with a sufficiently small sample spacing, and $b_0(t)$ is the ideal sinc interpolation kernel, then we will have $x_r(t) = x(t)$ and the representation or reconstruction is perfect.

## Radial basis function approximation

Another interesting example that is investigated in this section is the use of *radial basis functions* (RBFs).

Suppose that $b_0(t)$ is again an even function centered on the origin $t=0$, and for each $n$ the basis function $b_n(t)$ is just $b_0(t)$ shifted until it is centered on some point $t=t_n$.  Then
$$x(t) = \sum_{n=0}^{N-1} c_n \, b_0(t - t_n).$$
The function $b_0(t)$ and the centers $t_0, \ldots, t_{N-1}$ are specified in advance, and the coefficients $c_n$ are chosen so that the approximation matches the actual $x(t)$.

For demonstration we use the prototype basis $b_0(t) = e^{-t^2/\sigma^2}$ with $\sigma=1$.  Earlier in this workbook we assumed $t_n=nT$, and while we do so again this is not in general required for an RBF network.  We also do not assume that $b_0(nT) = \delta(n)$, and this makes it more difficult to specify the coefficients.  The prototype basis $b_0(t)$ and example basis functions $b_n(t) = b_0(t - t_n)$ in the representation are evaluated and plotted by the code below.

```python
import numpy as np
import sympy as sp
import matplotlib.pyplot as plt
%matplotlib notebook

# Prototype basis function
sigma = 1;
t = sp.symbols('t');
b0r = sp.exp(-t**2/sigma**2);
lam_b0r = sp.lambdify(t, b0r, modules=['numpy']);

# Discretisation details and basis centers
tmax = 12;
Tc = 1;  nmc = int(np.floor(tmax/Tc));
tnv = Tc*np.arange(0,nmc+1);
#tnv = np.random.uniform(0, tmax, nm+1);  # random basis centers

# Plot prototype and reconstruction basis
tv = np.linspace(-2, tmax+2, 2500);
tbv = np.linspace(-3*sigma, 3*sigma, 1500);  b0v = lam_b0r(tbv);
fh, ax = plt.subplots(2);
ax[0].plot(tbv, b0v, 'r-');
for tn in tnv:
    ax[1].plot(tv, lam_b0r(tv-tn));
ax[1].set_xlabel('t');
```

    <IPython.core.display.Javascript object>

Finding the required values for $c_n$ given a function $x(t)$ to be approximated depends on the particular problem.  In this case we again simply require that the function $x(t)$ and the approximation agree on their values at $t=kT$ for some set of integers $k$.  As long as this set has more than $N$ elements then we have more than $N$ equations in $N$ unknowns, which can be solved if the resulting system is not underdetermined.

The code below calculates the coefficients for the function
$$x(t) = \cos\left( \frac{t-5}{5} \right) - \left( \frac{t-5}{5} \right)^3$$
by formulating a matrix equation and solving.  Each value of $k$ leads to a single sample point and an equation of the form
$$x(kT) = \sum_{n=0}^{N-1} c_n b_0(kT - t_n)
= \begin{pmatrix} b_0(kT-t_0) & \cdots & b_0(kT-t_{N-1}) \end{pmatrix}
\begin{pmatrix} c_0 \\ \vdots \\ c_{N-1} \end{pmatrix}.$$
If we consider $k=0, \ldots, M-1$ this leads to the system of equations
$$\begin{pmatrix} x(0T) \\ \vdots \\ x((M-1)T) \end{pmatrix}
= \begin{pmatrix}
b_0(0T-t_0) & \cdots & b_0(0T-t_{N-1}) \\
\vdots & \ddots & \vdots \\
b_0((M-1)T-t_0) & \cdots & b_0((M-1)T-t_{N-1})
\end{pmatrix}
\begin{pmatrix} c_0 \\ \vdots \\ c_{N-1} \end{pmatrix},$$
which is of the form $\mathbf{b} = \mathbf{B} \mathbf{c}$.  In general if this system is overdetermined (which is usually true if $M>N$) then the least squares solution is often appropriate:
$$\mathbf{c} = (\mathbf{B}^T \mathbf{B})^{-1} \mathbf{B}^T \mathbf{b}.$$
See https://en.wikipedia.org/wiki/Least_squares for details if interested.

```python
from numpy.linalg import pinv

# Set up signal to approximate
x = sp.cos((t-5)/5) - ((t-5)/5)**3;
#x = sp.sin(t+0.5) + 5*(1 - sp.exp(-(t+0.5)));  # reconstruction points
lam_x = sp.lambdify(t, x, modules=['numpy']);

# Evaluate basis matrix and value for periodic observations
M = int(1.0*len(tnv));  # M = 2*len(tnv);
T = tmax/(M-1);
B = np.zeros((M,len(tnv)));  b = np.zeros(M);
for k in range(0,M):
    b[k] = lam_x(k*T);
    for n in range(0,len(tnv)):
        B[k,n] = lam_b0r(k*T - tnv[n]);

# Solve for coefficients
cv = pinv(B).dot(b);
print(cv);
```

    [ 1.35834594  0.46235455  0.64432842  0.5208424   0.56564884  0.56715441
      0.55934324  0.50895771  0.34921072  0.18851101 -0.36759327 -0.33474544
     -2.44417749]

The coefficient vector `cv` now specifies the particular instance of the approximation function.  The code below plots both the approximation and the actual function for a dense set of time points, and then shows the scaled basis contributions.

```python
# Numerically evaluate approximation (same as before)
xav = np.zeros(tv.shape);
for n in range(0,len(tnv)):
    xav = xav + cv[n]*lam_b0r(tv - tnv[n]);

# Plot
fh, ax = plt.subplots(2);
ax[0].plot(tv,lam_x(tv),'k-',tv,xav,'r-');
ax[0].scatter(np.arange(0,M)*T, b, c='r');
for n in range(0,len(tnv)):
    ax[1].plot(tv,cv[n]*lam_b0r(tv - tnv[n]));
```

    <IPython.core.display.Javascript object>

For the case where $M=N$ the series approximation will match the actual function values at all the points used to estimate the coefficients.  However, we have no control of the quality of the approximation between these points.  Also observe that if we increase $\sigma$ then the approximation becomes smoother, and vice versa.  Without knowing the degree of smoothness of $x(t)$ it is however difficult to know what value is appropriate, and in general we might not even have an expression for it that we can plot.

The radial basis function approach shown here is a local representation:  each basis function only influences on a small neighbourhood of points around its center.  This is convenient but isn't required.  A polynomial basis, for example, does not have this property, and neither does a sinusoidal basis.

Finding the coefficients for a general basis, even for the sampling-based method presented above which is not without flaws, is computationally complex:  for a $N$ element basis we have to invert a $N \times N$ matrix.  With a different basis we can improve on this situation.  For example, the requirement that $b_0(nT) = \delta(n)$ used earlier in this workbook imposes an *orthogonality* condition on the problem that makes it easier to find the coefficients.

# Tasks

These tasks involve writing code, or modifying existing code, to meet the objectives described.

1.  Suppose we have samples $x[n] = x(nT)$ of the signal
$$x(t) = \cos\left( \frac{t-5}{5} \right) - \left( \frac{t-5}{5} \right)^3$$
for $n=0, \ldots, N-1$, with $N=10$ and $T = 1.2$.  Consider the reconstruction equation
$$x_r(t) = \sum_{n=0}^{N-1} x[n] b_0(t-nT).$$
On the same set of axes plot both $x(t)$ and $x_r(t)$ over the range $t = 0$ to $t=(N-1)T$ for the case of a box filter interpolant $b_0(t) = p_T(t) = p_1(t/T)$, where $p_T(t)$ is the unit pulse of total width $T$ centered on the origin.

2.  Generate two new plots with the same specifications as for the previous task, but using these interpolation kernels:

    A.  Cubic kernel $b_0(tT) =
    \begin{cases}
      (a+2)|t|^3 - (a+3)|t|^2 + 1 \qquad & 0 \leq |t| < 1 \\
      a|t|^3 - 5a|t|^2 + 8a|t| - 4a \qquad & 1 \leq |t| < 2 \\
      0 \qquad & 2 \leq |t|
    \end{cases}
    $

    with $a=-1/2$.

    B.  Truncated or windowed sinc function $b_0(tT) = \frac{\sin(\pi t)}{\pi t} p_{2w}(t)$ for $w=3$.

3.  Use a representation of the form
$$x_r(t) = \sum_{n=0}^{N-1} c_n b_n(t)$$
with a polynomial basis $b_n(t) = t^n$ to approximate the signal $x(t) = e^{(t/10 + cos(t))}$.  The coefficients should be calculated using least squares from the samples $x[n] = x(nT)$ for $T=1$ and $n=0, \ldots, M-1$.  On a single set of axes show the signal $x(t)$, the reconstruction $x_r(t)$, and the sampled points, for the case of $N=9$ and $M=11$.  The domain of the plot should be from $t=0$ to $t=10$.

4.  (optional) Repeat the previous task using a cosine basis $b_n(t) = \cos(\omega_0 n t)$ with $\omega_0 = 2\pi/10$ and $N = 6$.

# Hints and guidance

The remainder of this notebook provides a more detailed walkthrough of possible ways of addressing the given tasks.  You may not need them.

## Task 1

```python
# 1.A) Set up symbolic signal x
# Create a symbolic variable 't', and specify the given x in terms of t. Use 'lambdify' to create a function 'lam_x'
# that takes t as an input and gives x(t) out.
```

```python
# 1.B) Get the continuous x(t)
# Create an array tv of 2500 time points over the range you're interested in. Get xv, the continuous approximation of the signal
# x, by applying the lambda function you just created to each value in tv.
```

```python
# 1.C) Get the discrete x[n]
# Create the an array 'tnv' that contains the values of nT for n = 0 to n = N. Sample x by applying the lam_x function to each
# value in tnv. Call the resulting array xnv.
```

```python
# 1.D) Get b0
# Use 'piecewise' to create the rectangular pulse b0. (Hint: the line of code you'll need should be in the 'Interpolation
# kernel' cell.) Create a lambda function lam_b0 that takes t as the input and returns b0(t).

# OPTIONAL: create an array 'b0v' by applying lam_b0 to a time array (tv or some new array, tbv, over a preferable range.)
# plot b0v to see what each individual basis function looks like. You can also use a for loop to plot all the basis functions
# over the range of interest.
```

```python
# 1.E) Reconstruct
# Create an array 'xav' to hold the reconstructed signal. It should be the same size as tv. Use a for loop to add each scaled
# basis function to the reconstruction.
```

```python
# 1.F) Output
# Plot the original signal xv and the reconstructed signal xav on the same set of axes vs tv. If you like, you can use subplot
# to create a second set of axes and plot each basis function on it using a for loop to better visualize what's happening.
```

## Task 2.A

```python
# 2.A.A) Get b0
# You've already set up the signal and created continuous and discrete versions of it in task 1.
# Use 'piecewise' to create the cubic interpolation function b0 and lambdify it. (Again, Prof's done it for you. You just have to
# uncomment it and change the parameters.)

# OPTIONAL: plot b0 as described in 1.D
```

```python
# 2.A.B) Reconstruct and output
# Create the reconstructed signal xav and plot it on the same set of axes as the original signal, as described in 1.E and 1.F
```

## Task 2.B

```python
# 2.B.A) Get b0
# Use 'piecewise' to create the sinc interpolation function b0 and lambdify it.

# OPTIONAL: plot b0 as described in 1.D
```

```python
# 2.B.B) Reconstruct and output
# Create the reconstructed signal xav and plot it on the same set of axes as the original signal, as described in 1.E and 1.F
```

## Task 3

The idea of reconstruction from basis functions and the way this task is formulated probably seem unfamiliar to you, but if you expand out that sum, you get $x(t) = c_0 b_0 + c_1 b_1 + ... + c_N b_N $

Substituting in $b_n = t^n$, you get $x(t) = c_0 + c_1 t + c_2 t^2 ... + c_N t^N $ so all we're actually trying to do here is come up with an Nth order polynomial we can use to approximate $x(t)$

```python
# 3.A) Set up symbolic x
# Specify the signal x in terms of the symbolic variable t and lambdify it to create lam_x
# Create the continuous approximation x(t) as described in 1.B

t = sp.symbols('t');
x = sp.exp(sp.cos(t)+t/10)
lam_x = sp.lambdify(t, x, modules=['numpy']);t = sp

tv = np.linspace(0,10,2500);
xv = lam_x(tv);
```

## 3.B Finding coefficients with least squares
We're going to use samples of $x(t)$ at $M$ points to evaluate the coefficients.

At each individual sample, $x(kT) = \sum_{n=0}^{N-1} c_n b_n(kT) = c_0 + c_1 (kT) + c_2 (kT)^2 + ... c_N (kT)^N$, which can be written as the matrix expression: $$x(kT)
= \begin{pmatrix} 1 & kT & (kT)^2 & \cdots & (kT)^N) \end{pmatrix}
\begin{pmatrix} c_0 \\ c_1 \\ c_2\\ \vdots \\ c_N \end{pmatrix}.$$

By repeating this for k values from 0 through M, we get $$\begin{pmatrix} x(0T) \\ x(1T) \\ \vdots \\ x(MT) \end{pmatrix}
= \begin{pmatrix} 1 & 0T & \cdots & (0T)^N
\\ 1 & 1T & \cdots & (1T)^N
\\ \vdots
\\ 1 & MT & \cdots & (MT)^N
\end{pmatrix}
\begin{pmatrix} c_0 \\ c_1 \\ \vdots \\ c_N \end{pmatrix}$$ or $b = Bc$

We can calculate each term in $b$ using the lam_x function. The matrix $B$ is also simple to evaluate. From there, we just have to invert $B$ and multiply it by $b$ to get the coefficient matrix $c$ out.

```python
# 3.B)
# Use 'zeros' to create the M x 1 matrix 'b' and the M x N matrix 'B'. Populate these matrices with the values described above
# using for loops. Solve for the coefficient matrix cv by taking the dot product of the inverse of B and b.
# (Hint: you can get an idea of how to do each of these tasks by looking at the least squares code.)
```

```python
# 3.C) Reconstruct and output
# Create an array xav, the same size as tv, to use for your polynomial approximation. Use a for loop to build the polynomial by
# adding each c_n*t**n term from 0 up to N.
# plot the original signal and the polynomial on the same set of axes.
```


**Create Plots**

**Plot with Symbolic Plotting Functions**

MATLAB® provides many techniques for plotting numerical data. Graphical capabilities of MATLAB include plotting tools, standard plotting functions, graphic manipulation and data exploration tools, and tools for printing and exporting graphics to standard formats. Symbolic Math Toolbox™ expands these graphical capabilities and lets you plot symbolic functions using:

- fplot to create 2-D plots of symbolic expressions, equations, or functions in Cartesian coordinates.
- fplot3 to create 3-D parametric plots.
- ezpolar to create plots in polar coordinates.
- fsurf to create surface plots.
- fcontour to create contour plots.
- fmesh to create mesh plots.

Plot the symbolic expression $sin(6x)$ by using **fplot**. By default, **fplot** uses the range $−5<x<5$.

```python
from sympy import *
x = symbols('x')
plot(sin(6*x),(x,-5,5))
```

Plot a symbolic expression or function in polar coordinates $r$ (radius) and $\theta$ (polar angle) by using **ezpolar**. By default, **ezpolar** plots a symbolic expression or function over the interval $0<\theta<2\pi$.

Plot the symbolic expression $sin(6t)$ in polar coordinates.

```python
#syms t
#ezpolar(sin(6*t))
import matplotlib.pyplot as plt
import numpy as np

t = symbols('t')
eqf = lambdify(t,sin(6*t))
angle = np.arange(0,2*np.pi,1/100)
plt.polar(angle,np.abs(eqf(angle)))
plt.title('$r=sin(6t)$')
```

**Plot Functions Numerically**

As an alternative to plotting expressions symbolically, you can substitute symbolic variables with numeric values by using **subs**. Then, you can use these numeric values with plotting functions in MATLAB™.

In the following expressions **u** and **v**, substitute the symbolic variables **x** and **y** with the numeric values defined by **meshgrid**.

```python
x,y = symbols('x y')
u = sin(x**2+y**2)
v = cos(x*y)
```

Now, you can plot **U** and **V** by using standard MATLAB plotting functions.

Create a plot of the vector field defined by the functions $U(X,Y)$ and $V(X,Y)$ by using the MATLAB **quiver** function.

```python
eqfU = lambdify((x,y),u)
eqfV = lambdify((x,y),v)
X,Y = np.meshgrid(np.arange(-1,1,0.1),np.arange(-1,1,0.1))
plt.quiver(X,Y,eqfU(X,Y),eqfV(X,Y))
```

**Plot Multiple Symbolic Functions in One Graph**

Plot several functions on one graph by adding the functions sequentially. After plotting the first function, add successive functions by using the **hold** on command. The **hold on** command keeps the existing plots. Without the **hold on** command, each new plot replaces any existing plot. After the **hold on** command, each new plot appears on top of existing plots. Switch back to the default behavior of replacing plots by using the **hold off** command.

Plot $f=e^x sin(20x)$ using **fplot**. Show the bounds of **f** by superimposing plots of $e^x$ and $-e^x$ as dashed red lines. Set the title by using the **DisplayName** property of the object returned by **fplot**.

```python
x,y = symbols('x y')
f = exp(x)*sin(20*x)
```

$f=sin(20x)e^x$

```python
p1 = plot(f,exp(x),-exp(x),(x,0,3))
```

**Plot Multiple Symbolic Functions in One Figure**

Display several functions side-by-side in one figure by dividing the figure window into several subplots using **subplot**. The command **subplot(m,n,p)** divides the figure into a **m** by **n** matrix of subplots and selects the subplot **p**. Display multiple plots in separate subplots by selecting the subplot and using plotting commands. Plotting into multiple subplots is useful for side-by-side comparisons of plots.

Compare plots of $sin\left(\left(x^2+y^2\right)/a\right)$ for $a=10,20,50,100$ by using subplot to create side-by-side subplots.

```python
import mpl_toolkits.mplot3d

x,y,a = symbols('x y a')
eqf3 = lambdify((x,y,a),sin((x**2+y**2)/a))
X,Y = np.meshgrid(np.arange(-5,5,0.1),np.arange(-5,5,0.1))

fig = plt.figure(constrained_layout=True)
ax0 = fig.add_subplot(2,2,1,projection='3d')
ax0.plot_surface(X,Y,eqf3(X,Y,10),cmap=plt.cm.viridis) #使用viridis色谱
ax0.set_title('$a=10$',loc='left')
ax1 = fig.add_subplot(2,2,2,projection='3d')
ax1.plot_surface(X,Y,eqf3(X,Y,20),cmap=plt.cm.viridis) #使用viridis色谱
ax1.set_title('$a=20$',loc='left')
ax2 = fig.add_subplot(2,2,3,projection='3d')
ax2.plot_surface(X,Y,eqf3(X,Y,50),cmap=plt.cm.viridis) #使用viridis色谱
ax2.set_title('$a=50$',loc='left')
ax3 = fig.add_subplot(2,2,4,projection='3d')
ax3.plot_surface(X,Y,eqf3(X,Y,100),cmap=plt.cm.viridis) #使用viridis色谱
ax3.set_title('$a=100$',loc='left')
```

**Combine Symbolic Function Plots and Numeric Data Plots**

Plot numeric and symbolic data on the same graph by using MATLAB and Symbolic Math Toolbox functions together.

For numeric values of **x** between $[−5,5]$, return a noisy sine curve by finding $y=sin(x)$ and adding random values to **y**. View the noisy sine curve by using **scatter** to plot the points $(x1,y1),(x2,y2),⋯$.

```python
x = np.arange(-5,5,1/10)
y = np.sin(x)+((-1)*np.random.randint(10,size=100)*np.random.rand(100))/8
fig,ax = plt.subplots()
ax.scatter(x,y,c='w',edgecolors='#1f77b4')
```

Show the underlying structure in the points by superimposing a plot of the sine function. First, use **hold on** to retain the **scatter** plot. Then, use **fplot** to plot the sine function.

```python
#hold on
#syms t
#fplot(sin(t))
#hold off
t = symbols('t')
eqft = lambdify(t,sin(t))
fig,ax = plt.subplots()
ax.scatter(x,y,c='w',edgecolors='#1f77b4')
ax.plot(x,eqft(x))
```

**Combine Numeric and Symbolic Plots in 3-D**

Combine symbolic and numeric plots in 3-D by using MATLAB and Symbolic Math Toolbox plotting functions. Symbolic Math Toolbox provides these 3-D plotting functions:

- fplot3 creates 3-D parameterized line plots.
- fsurf creates 3-D surface plots.
- fmesh creates 3-D mesh plots.

Create a spiral plot by using **fplot3** to plot the parametric line
$$ x=(1-t)sin(100t)$$
$$ y=(1-t)cos(100t)$$
$$ z=\sqrt{1-x^2-y^2}$$

```python
t = symbols('t')
x = (1-t)*sin(100*t)
y = (1-t)*cos(100*t)
z = sqrt(1-x**2-y**2)
eqfx = lambdify(t,x)
eqfy = lambdify(t,y)
eqfz = lambdify(t,z)
X = eqfx(np.arange(0,1,1/1000))
Y = eqfy(np.arange(0,1,1/1000))
Z = eqfz(np.arange(0,1,1/1000))

fig = plt.figure()
ax = fig.add_subplot(projection='3d')
ax.plot(X,Y,Z,linewidth=0.6)
ax.set_title('Symbolic 3-D Parametric Line')
```

Superimpose a plot of a sphere with radius 1 and center at $(0, 0, 0)$. Find points on the sphere numerically by using **sphere**. Plot the sphere by using **mesh**. The resulting plot shows the symbolic parametric line wrapped around the top hemisphere.

```python
#hold on
#[X,Y,Z] = sphere;
#mesh(X, Y, Z)
#colormap(gray)
#title('Symbolic Parametric Plot and a Sphere')
#hold off
theta,phi = np.meshgrid(np.linspace(0,2*np.pi,30),np.linspace(0,np.pi,30))
X_sphere = np.sin(phi)*np.cos(theta)
Y_sphere = np.sin(phi)*np.sin(theta)
Z_sphere = np.cos(phi)

fig = plt.figure()
ax = fig.add_subplot(projection='3d')
ax.plot_wireframe(X_sphere,Y_sphere,Z_sphere,linewidth=0.2,color='black')
ax.plot(X,Y,Z)
```

Symbolic Math Toolbox/Create Plots.ipynb


# 15.6. Finding a Boolean propositional formula from a truth table

```python
from sympy import *
init_printing()
```

```python
var('x y z')
```

```python
P = x & (y | ~z)
P
```

```python
P.subs({x: True, y: False, z: True})
```

```python
minterms = [[1, 0, 1], [1, 0, 0], [0, 0, 0]]
dontcare = [[1, 1, 1], [1, 1, 0]]
```

```python
Q = SOPform(['x', 'y', 'z'], minterms, dontcare)
Q
```

```python
Q.subs({x: True, y: False, z: False}), Q.subs(
    {x: False, y: True, z: True})
```


---
author: Nathan Carter (ncarter@bentley.edu)
---

This answer assumes you have imported SymPy as follows.

```python
from sympy import *                   # load all math functions
init_printing( use_latex='mathjax' )  # use pretty math output
```

Let's create an equation with many variables.

```python
var('P V n R T')
ideal_gas_law = Eq( P*V, n*R*T )
ideal_gas_law
```

$\displaystyle P V = R T n$

To isolate one variable, call the `solve` function, and pass that variable
as the second argument.

```python
solve( ideal_gas_law, R )
```

$\displaystyle \left[ \frac{P V}{T n}\right]$

The brackets surround a list of all solutions---in this case, just one.
That solution is that $R=\frac{PV}{Tn}$.

database/tasks/How to isolate one variable in an equation/Python, using SymPy.ipynb


# Brusselator

The Brusselator is a simplified model for chemical reactions that oscillate. The reaction scheme is as follows:
$$
\begin{align}
A  &\overset{k_1}\longrightarrow X \tag{1} \\
B + X &\overset{k_2}\longrightarrow Y + D \tag{2}\\
2X + Y &\overset{k_3}\longrightarrow 3X \tag{3}\\
X &\overset{k_4}\longrightarrow E \tag{4}\\
\end{align}
 $$

`A, B, D, E, X, Y` are the reactants species.
We will use the general formulation of the mass action in differential form to derive the differentiatl equations that govern the dynamics of the system.
$$
\begin{align}
\frac{\mathrm{d} X}{\mathrm{d} t}&= (B-A)^T \cdot K \cdot X^A \tag{53}\\
\end{align}
$$
where  `A` and `B` are the matrices with the stoichiometric coefficients, `X` is the state vector $X=[X_1,X_2,X_3]^T$, and `K` is a matrix in the form:

$$
A=\begin{pmatrix}
 1 & 0  & 0  & 0 & 0 \\
 0 & 1 & 1  & \dots & 0\\
 0 & 0  & k_3 &\dots  &0 \\
 \vdots &  \vdots   &  \vdots  &  \vdots  &  \vdots  \tag{54}\\
 0 & 0  & 0  &  \dots & k_r
\end{pmatrix}
$$

The system is a open reactor where we maintain a constant concentration of the reactants `A,B,D,E`.

Therefore, the  equations for the evolution of `[X]` and `[Y]` are as follows:

$$\begin{align}
            \frac{ d[X] }{dt} &= k_1[A] − k_2[B][X] + k_3[X]^2[Y] − k_4[X]\tag{5} \\
            \frac{ d[Y] }{dt} &= k_2[B][X] − k_3[X]^2[Y]  \tag{6}
            \end{align}            $$

We assume as initial conditions:

$$\begin{align}
            [X (0)] &= 0 \tag{7} \\
            [Y (0)] &= 0   \tag{8}
            \end{align}            $$

To calculate the equilibrium, we just set eqs. 5 and 6 to zero and solve for `X` and `Y`.

$$\begin{align}
            [X]_{eq} &= \frac{k_1 [A]}{k_4}\tag{5} \\
            [Y]_{eq} &= \frac{k_4  k_2 [B]}{k_3  k_1 [A]} \tag{6}
            \end{align}            $$

To evaluate stability, we will  evaluate the Jacobian at the stationary state $([X]_{eq},[Y]_{eq})$.

The parameters sets to try are:

$$
k_1=1\\
k_2=1\\
k_3=1\\
k_4=1\\
A=1\\
B=3
$$

```julia
using DifferentialEquations
```

```julia
using Plots; gr()
```

    Plots.GRBackend()

```julia
brusselator! = @ode_def BR begin
    dX = k_1 * A - k_2 * B * X + k_3 * X^2 * Y - k_4 * X
    dY = k_2 * B * X - k_3 * X^2 * Y
    end k_1 k_2 k_3 k_4 A B
```

    (::BR{getfield(Main, Symbol("##3#7")),getfield(Main, Symbol("##4#8")),getfield(Main, Symbol("##5#9")),Nothing,Nothing,getfield(Main, Symbol("##6#10")),Expr,Expr}) (generic function with 2 methods)

```julia

```

```julia
tspan = (0.0,50.0)
k_1=1.
k_2=1.
k_3=1.
k_4=1.
A=1.
B=3.
u₀=[0.0,0.0]
p=[k_1,k_2,k_3,k_4,A,B];
```

```julia
prob1 = ODEProblem(brusselator!,u₀,tspan,p)
sol1 = solve(prob1)
plot(sol1,label=["X","Y"])
title!("Brusselator")
xlabel!("Time [s]")
ylabel!("Concentration [a.u]")
```

```julia
plot(sol1,vars=(1,2),label=["limit cycle plot"])
title!("Brusselator ")
xlabel!("X [a.u.]")
ylabel!("Y [a.u.]")
```

```julia

```


# Box-Cox Transformation

Normality is an important assumption for many statistical techniques. A **Box-Cox transformation** is a transformation of non-normal dependent variables into a normal shape, which depends on the parameter $\lambda$ and is defined as follow :

\begin{equation}
w_t=
\begin{cases}
      \displaystyle \ln(y_t) \text{ if } \lambda=0 \text{ , } \\
      \displaystyle \frac{(y_t^\lambda-1)}{\lambda} \text{ otherwise.}
\end{cases}
\end{equation}

* For all $\lambda \neq 1$, the time series will change shape.

* If $\lambda=0$, natural logarithms are used.

* If $\lambda=1$, then $w_t=y_t-1$, so the transformed data is shifted downwards but there is no change in the shape of the time series.

* Having chosen a transformation, we need to forecast the transformed data. Then, we need to reverse the transformation (or back-transform) to obtain forecasts on the original scale.

* Features of power transformations
  * Often no transformation is needed.
  * The forecasting results are relatively insensitive(感覺遲鈍的) to the value of $\lambda$.
  * Transformations sometimes make little difference to the forecasts but **have a large effect on prediction intervals**.

Reference : [Mathematical transformations](https://otexts.com/fpp2/transformations.html#mathematical-transformations)


# Optimization Exercise 1

## Imports

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import scipy.optimize as opt
```

## Hat potential

The following potential is often used in Physics and other fields to describe symmetry breaking and is often known as the "hat potential":

$$ V(x) = -a x^2 + b x^4 $$

Write a function `hat(x,a,b)` that returns the value of this function:

```python
# YOUR CODE HERE
def hat(x, a, b):
    return (-a * x**2) + (b * x**4)
```

```python
assert hat(0.0, 1.0, 1.0)==0.0
assert hat(0.0, 1.0, 1.0)==0.0
assert hat(1.0, 10.0, 1.0)==-9.0
```

Plot this function over the range $x\in\left[-3,3\right]$ with $b=1.0$ and $a=5.0$:

```python
a = 5.0
b = 1.0
x = np.linspace(-3, 3, 100)
plt.plot(x, hat(x, a, b))
```

```python
# YOUR CODE HERE
raise NotImplementedError()
```

```python
assert True # leave this to grade the plot
```

Write code that finds the two local minima of this function for $b=1.0$ and $a=5.0$.

* Use `scipy.optimize.minimize` to find the minima. You will have to think carefully about how to get this function to find both minima.
* Print the x values of the minima.
* Plot the function as a blue line.
* On the same axes, show the minima as red circles.
* Customize your visualization to make it beatiful and effective.

```python
# YOUR CODE HERE
#specifies a number of divisions which the function will look for a minimum on each.
#more divisions = more accurate
def minima(divisions, function, a, b):
    critpoints = []
    for n in range(0, divisions):
        sectionx = np.linspace(n*(6/divisions) - 3, (n+1)*(6/divisions) - 3, 100)
        sectiony = function(np.linspace(n*(6/divisions) - 3, (n+1)*(6/divisions) - 3, 100), a, b)
        #make sure the minimum is not on the ends
        if np.amin(sectiony) != sectiony[0] and np.amin(sectiony) != sectiony[-1]:
            minpt = np.argmin(sectiony)
            critpoints.append(sectionx[minpt])

    return critpoints

```

```python
minpts = minima(100, hat, a, b)
```

```python
x = np.linspace(-3, 3, 100)
plt.plot(x, hat(x, a, b))
plt.scatter(minpts[0], hat(minpts[0], a, b), color = "r")
plt.scatter(minpts[1], hat(minpts[1], a, b), color = "r")
plt.xlabel("X")
plt.ylabel("V(x)")
print("Minimums: ", minpts)
```

```python
assert True # leave this for grading the plot
```

To check your numerical results, find the locations of the minima analytically. Show and describe the steps in your derivation using LaTeX equations. Evaluate the location of the minima using the above parameters.

To find the minima, we can take the derivative of the function and set it to zero, finding the critical points.

\begin{equation}
V'(x) = -2ax + 4bx^3 = 0
\end{equation}

Dividing both sides by x gives us a quadratic, and we can put in the given values of a and b:

\begin{equation}
-10 + 4x^2 = 0
\end{equation}

Solving this gives us the intercepts, where V'(x) = 0.

\begin{equation}
x = \pm \sqrt{\frac{5}{2}}
\end{equation}

Which approximates to:
\begin{equation}
x = \pm 1.5811
\end{equation}

And this agrees with the program's results.

```python

```


If $x_1, ..., x_n$ are an iid sample from the double-exponential (Laplace) distribution with density:

$$p_{\theta}(x) = (2 \theta)^{-1} exp (-|x|/\theta)$$

then from the factorization theorem it is obvious that $$t(x) = \sum_i |x_i|$$ is sufficient.

We want to show that the conditional distribution of the data given T = t is free of $\theta$. According to the following discussion, it seems that we can infer that the [distribution of T](https://math.stackexchange.com/questions/199460/distribution-of-sum-of-absolute-value-of-random-variable) is given by $T \sim Gamma(n \theta, n)$. Then using Bayes theorem we have:

$$
\begin{align}
P(X_1 = x_1, ..., X_n = x_n | T = t) & = \frac{P(X_1 = x_1, ..., X_n = x_n, T = t)}{P(T = t)} \\
& = \frac{P(X_1 = x_1, ..., X_n = x_n)}{P(T = t)} \\
& = \frac{(2 \theta)^{-n} exp (-t/\theta)}{\Gamma (n)^{-1} (n \theta)^{-n} t^{n - 1 } exp (-t/\theta)} \\
& = \frac{\Gamma (n)}{t^{n - 1}} \bigg( \frac{2}{n} \bigg)^{-n}
\end{align}
$$

which is free of $\theta$ as expected.


# Linear regression
- It's a way to model relationship between 2 models
- Also known as the slope formula
- equation form: \begin{align}Y = a + bX\end{align}
  - where X is the independent variable and Y is dependent
  - b = slope
  - a = y-intercept

#### Equations
\begin{align}
a = \dfrac{(\sum y)(\sum x^2)-(\sum x)(\sum xy)}{n(\sum x^2) - (\sum x)^2}
\end{align}

\begin{align}
b = \dfrac{n(\sum xy)-(\sum x)(\sum y)}{n(\sum x^2) - (\sum x)^2}
\end{align}

```python
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
```

```python
import os
path = os.getcwd() + '\data\ex1data1.txt'
data = pd.read_csv(path, header=None, names=['Population', 'Profit'])
data.head()
```

      Population
      Profit

      0
      6.1101
      17.5920

      1
      5.5277
      9.1302

      2
      8.5186
      13.6620

      3
      7.0032
      11.8540

      4
      5.8598
      6.8233

```python
data.describe()
```

      Population
      Profit

      count
      97.000000
      97.000000

      mean
      8.159800
      5.839135

      std
      3.869884
      5.510262

      min
      5.026900
      -2.680700

      25%
      5.707700
      1.986900

      50%
      6.589400
      4.562300

      75%
      8.578100
      7.046700

      max
      22.203000
      24.147000

> Let's plot it together to get a better idea of what the data looks like

```python
plt.scatter(x=data['Population'], y=data['Profit'])
plt.show()
```

> Implementing linear regression using gradient descent to minimize cost function

> Creating a function to compute the cost of a given solution

```python
def compute_cost(X, y, theta):
    inner = np.power(((X * theta.T) - y), 2)
    return np.sum(inner) / (2 * len(X))
```

Inserting a column of ones to use a vectorized solution to computing cost and gradients.

```python
data.insert(0, 'Ones', 1)
```

Checking the new table

```python
data.head()
```

      Ones
      Population
      Profit

      0
      1
      6.1101
      17.5920

      1
      1
      5.5277
      9.1302

      2
      1
      8.5186
      13.6620

      3
      1
      7.0032
      11.8540

      4
      1
      5.8598
      6.8233

Variable initialization

```python
# set X (training data) and y (target variable)
cols = data.shape[1]
X = data.iloc[:, 0:cols-1]
y = data.iloc[:, cols-1:cols]
```

Checking values

```python
X.head()
```

      Ones
      Population

      0
      1
      6.1101

      1
      1
      5.5277

      2
      1
      8.5186

      3
      1
      7.0032

      4
      1
      5.8598

```python
y.head()
```

      Profit

      0
      17.5920

      1
      9.1302

      2
      13.6620

      3
      11.8540

      4
      6.8233

Convert to numpy matrices

```python
X = np.matrix(X.values)
y = np.matrix(y.values)
theta = np.matrix(np.array([0, 0]))
```

```python
theta
```

    matrix([[0, 0]])

Shape of matrices

```python
X.shape, y.shape, theta.shape
```

    ((97, 2), (97, 1), (1, 2))

Compute cost for initial solution (0 values for theta)

```python
compute_cost(X, y, theta)
```

    32.072733877455676

---
Defining a function to perform gradient descent on paramaters theta using the update rules defined in the text

```python
def gradient_descent(X, y, theta, alpha, iters):
    temp = np.matrix(np.zeros(theta.shape))
    parameters = int(theta.ravel().shape[1])
    cost = np.zeros(iters)

    for i in range(iters):
        error = (X * theta.T) - y

        for j in range(parameters):
            term = np.multiply(error, X[:, j])
            temp[0, j] = theta[0, j] - ((alpha / len(X)) * np.sum(term))

        theta = temp
        cost[i] = compute_cost(X, y, theta)

    return theta, cost
```

Init some additional variables: learning rate $\alpha$ (alpha), and number of iterations to perform

```python
alpha = 0.01
iters = 1000
```

Running the gradient descent algo to ift our parameters theta to the training set

```python
g, cost = gradient_descent(X, y, theta, alpha, iters)
g
```

    matrix([[-3.24140214,  1.1272942 ]])

Now we can compute our cost (error) of trained model using our fitted parameters.

```python
compute_cost(X, y, g)
```

    4.5159555030789118

Let's plot the linear model along with the data to see how it fits

```python
x = np.linspace(data.Population.min(), data.Population.max(), 100)
f = g[0, 0] + (g[0, 1] * x)

fig, ax = plt.subplots(figsize=(12, 8))
ax.plot(x, f, 'r', label='Prediction')
ax.scatter(data.Population, data.Profit, label='Traning Data')
ax.legend(loc=2)
ax.set_xlabel('Population')
ax.set_ylabel('Profit')
ax.set_title('Predicted Profit vs. Population Size')
fig
```

Observations:
- The cost always decreases. This is an example of convex optimization problem
- function outputs a vector with the cost at each iteration, so we can plot that as well

```python
fig, ax = plt.subplots(figsize=(12,8))
ax.plot(np.arange(iters), cost, 'r')
ax.set_xlabel('Iterations')
ax.set_xlim([0, iters])
ax.set_ylabel('Cost')
ax.set_title('Error vs. Training Epoch')
fig
```

---
# Mulitple variables
---

Let's try 2 variables.

Example: a housing price data set with 2 variables
- size of house, no. of bedrooms and
- target price of house.

```python
path = os.getcwd() + '\data\ex1data2.txt'
data2 = pd.read_csv(path, header=None, names=['Size', 'Bedrooms', 'Price'])
data2.head()
```

      Size
      Bedrooms
      Price

      0
      2104
      3
      399900

      1
      1600
      3
      329900

      2
      2400
      3
      369000

      3
      1416
      2
      232000

      4
      3000
      4
      539900

> add another step: normalizing the features

```python
data2 = (data2 - data2.mean()) / data2.std()
data2.head()
```

      Size
      Bedrooms
      Price

      0
      0.130010
      -0.223675
      0.475747

      1
      -0.504190
      -0.223675
      -0.084074

      2
      0.502476
      -0.223675
      0.228626

      3
      -0.735723
      -1.537767
      -0.867025

      4
      1.257476
      1.090417
      1.595389

Repeating steps from previous part

```python
# Add ones column
data2.insert(0, 'Ones', 1)
```

```python
# set X(training data) and y(target)
cols = data2.shape[1]
X2 = data2.iloc[:, 0:cols-1]
y2 = data2.iloc[:, cols-1:cols]
```

```python
# convert to np array
X2 = np.matrix(X2.values)
y2 = np.matrix(y2.values)
theta2 = np.matrix(np.array([0,0,0]))
```

```python
# perform linear regression
g2, cost2 = gradient_descent(X2, y2, theta2, alpha, iters)

# compute cost
compute_cost(X2, y2, g2)
```

    0.13070336960771892

```python
fig, ax = plt.subplots(figsize=(12,8))
ax.plot(np.arange(iters), cost2, 'r')
ax.set_xlabel('Iterations')
ax.set_ylabel('Cost')
ax.set_title('Error vs. Training Epoch')
fig
```

---
# Using scikit learn's linear regression function
---

```python
from sklearn import linear_model
```

```python
model = linear_model.LinearRegression()
model.fit(X, y)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)

```python
x = np.array(X[:, 1].A1)
f = model.predict(X).flatten()

fig, ax = plt.subplots(figsize=(12,8))
ax.plot(x, f, 'r', label='Prediction')
ax.scatter(data.Population, data.Profit, label='Traning Data')
ax.legend(loc=2)
ax.set_xlabel('Population')
ax.set_ylabel('Profit')
ax.set_title('Predicted Profit vs. Population Size')
fig
```


# Gaussian Process

In this tutorial, we expose what gaussian processes are, and how to use the [GPy library](http://sheffieldml.github.io/GPy/). We first provide a gentle reminder about Gaussian distributions and their properties.

```python
# Import all the important libraries
import math
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm
from scipy.stats import multivariate_normal
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.mlab as mlab
from ipywidgets import widgets as wg
from matplotlib import cm

import GPy
#%matplotlib inline
%matplotlib notebook
```

## 1D Gaussian distribution

```python
# Plot
def plot_gaussian(mu=0, sigma=1):
    x = np.linspace(-3, 3, 100)
    plt.plot(x, norm.pdf(x, mu, sigma))
    plt.xlabel('x')
    plt.ylabel('p(x)')

wg.interact(plot_gaussian, mu=(-2,2,0.1), sigma=(-2,2,0.1))
plt.show()
```

## Multivariate Gaussian distribution (2D)

The multivariable Gaussian distribution is a generalization of the Gaussian distribution to vectors. See [wikipedia](https://en.wikipedia.org/wiki/Multivariate_normal_distribution) for more info.

```python
# moments
mu = np.array([0,0])
Sigma = np.array([[1,0],
                  [0,1]])
Sigma1 = np.array([[1,0.5],
                   [0.5,1]])
Sigma2 = np.array([[1,-0.5],
                   [-0.5,1]])
Sigmas = [Sigma, Sigma1, Sigma2]

pts = []
for S in Sigmas:
    pts.append(np.random.multivariate_normal(mu, S, 1000).T)

# Plotting
width = 16
height = 4
plt.figure(figsize=(width, height))

# make plot
for i in range(len(Sigmas)):
    plt.subplot(1,3,i+1)
    plt.title('Plot '+str(i+1))
    plt.ylim(-4,4)
    plt.xlim(-4,4)
    plt.xlabel('x1')
    plt.ylabel('x2')
    plt.plot(pts[i][0], pts[i][1],'o')
    #plt.scatter(pts[i][0], pts[i][1])

plt.show()
```

The 1st plot above is described by:
\begin{equation}
    \left[ \begin{array}{c} x_1\\x_2 \end{array}\right] \sim \mathcal{N} \left(\left[ \begin{array}{c} 0\\0 \end{array}\right], \left[ \begin{array}{cc} 1 & 0\\ 0 & 1 \end{array}\right] \right)
\end{equation}

The 2nd plot is given by:
\begin{equation}
     \left[ \begin{array}{c} x_1\\x_2 \end{array}\right] \sim \mathcal{N}\left(\left[ \begin{array}{c} 0\\0 \end{array}\right], \left[ \begin{array}{cc} 1 & 0.5\\ 0.5 & 1 \end{array}\right]\right)
\end{equation}

Finally, the 3rd plot is given by:
\begin{equation}
    \left[ \begin{array}{c} x_1\\x_2 \end{array}\right] \sim \mathcal{N}\left(\left[ \begin{array}{c} 0\\0 \end{array}\right], \left[ \begin{array}{cc} 1 & -0.5\\ -0.5 & 1 \end{array}\right]\right)
\end{equation}

The covariance (and the dot product) measures the similarity.

For the 2nd and 3rd plots, $x_1$ is **correlated** with $x_2$, i.e. knowing $x_1$ gives us information about $x_2$.

### Joint distribution $p(x_1,x_2)$

The joint distribution $p(x_1, x_2)$ is given by:

\begin{equation}
    \left[ \begin{array}{c} x_1\\x_2 \end{array}\right] \sim \mathcal{N}\left(\left[ \begin{array}{c} \mu_1 \\ \mu_2 \end{array}\right], \left[ \begin{array}{cc} \Sigma_{11} & \Sigma_{12} \\ \Sigma_{21} & \Sigma_{22} \end{array}\right] \right) = \mathcal{N}(\pmb{\mu}, \pmb{\Sigma})
\end{equation}

```python
# Reference: http://stackoverflow.com/questions/38698277/plot-normal-distribution-in-3d

# moments
mu = np.array([0,0])
Sigma = np.array([[1,0], [0,1]])

# Create grid and multivariate normal
step = 500
bound = 10
x = np.linspace(-bound,bound,step)
y = np.linspace(-bound,bound,step)
X, Y = np.meshgrid(x,y)
pos = np.empty(X.shape + (2,))
pos[:, :, 0] = X; pos[:, :, 1] = Y
pdf = multivariate_normal(mu, Sigma).pdf(pos)

# Plot
fig = plt.figure(figsize=plt.figaspect(0.5)) # Twice as wide as it is tall.

# 1st subplot (3D)
ax = fig.add_subplot(1, 2, 1, projection='3d')
ax.plot_surface(X, Y, pdf, cmap='viridis', linewidth=0)
ax.set_xlabel('x1')
ax.set_ylabel('x2')
ax.set_zlabel('p(x1, x2)')

# 2nd subplot (2D)
ax = fig.add_subplot(1, 2, 2)
ax.contourf(x, y, pdf)
#ax.colorbar()
ax.set_xlabel('x1')
ax.set_ylabel('x2')

fig.tight_layout()

plt.show()
```

### Normalization

In order to be a valid probability distribution, the volume under the surface should equal to 1.

\begin{equation}
    \int \int p(x_1,x_2) dx_1 dx_2 = 1
\end{equation}

```python
# p(x1,x2) = pdf
# dx = 2.*bound/step
# dx1 dx2 = (2.*bound/step)**2
print("Summation: {}".format((2.*bound/step)**2 * pdf.sum()))
```

### Conditional distribution $p(x_2|x_1)$

What is the mean $\mu_{2|1}$ and the variance $\Sigma_{2|1}$ of the conditional distribution $p(x_2|x_1) = \mathcal{N}(\mu_{2|1}, \Sigma_{2|1})$?

We know the mean $\pmb{\mu}$ and the covariance $\pmb{\Sigma}$ of the joint distribution $p(x_1,x_2)$. Using the [Schur complement](https://en.wikipedia.org/wiki/Schur_complement), we obtain:

\begin{align}
    \mu_{2|1} &= \mu_{2} + \Sigma_{21}\Sigma_{22}^{-1}(x_2 - \mu_2) \\
    \Sigma_{2|1} &= \Sigma_{22} - \Sigma_{21}\Sigma_{22}^{-1}\Sigma_{12}
\end{align}

For the demo, check Murphy's book "Machine Learning: A Probabilistic Perspective", section 4.3.4

```python
fig = plt.figure(figsize=plt.figaspect(0.5)) # Twice as wide as it is tall.

x1_value = 0
z_max = pdf.max()

# 1st subplot
ax = fig.add_subplot(1, 2, 1, projection='3d')
ax.plot_surface(X, Y, pdf, cmap='viridis', linewidth=0)
ax.set_xlabel('x1')
ax.set_ylabel('x2')
ax.set_zlabel('p(x1, x2)')
y1 = np.linspace(-bound,bound,2)
z = np.linspace(0,z_max,2)
Y1, Z = np.meshgrid(y1,z)
ax.plot_surface(x1_value, Y1, Z, color='red', alpha=0.2)
#cset = ax.contourf(X, Y, pdf, zdir='x', offset=-bound, cmap=cm.coolwarm)

# 2nd subplot
ax = fig.add_subplot(1, 2, 2)
ax.plot(x, pdf[step//2 + x1_value*step//(2*bound)])
ax.set_xlabel('x2')
ax.set_ylabel('p(x2|x1)')

fig.tight_layout()

plt.show()
```

### Marginal distribution $p(x_1)$ and $p(x_2)$

\begin{align}
    p(x_1) &= \int p(x_1, x_2) dx_2 = \mathcal{N}(\mu_1, \Sigma_{11}) \\
    p(x_2) &= \int p(x_1, x_2) dx_1 = \mathcal{N}(\mu_2, \Sigma_{22})
\end{align}

```python
fig = plt.figure(figsize=plt.figaspect(0.5))
plt.subplot(1,2,1)
plt.title('By summing')
dx = 2. * bound / step
plt.plot(x, pdf.sum(0) * dx, color='blue')
plt.xlabel('x2')
plt.ylabel('p(x2)')

plt.subplot(1,2,2)
plt.title('by using the normal distribution')
plt.plot(x, norm.pdf(x, mu[1], Sigma[1,1]), color='red')
plt.xlabel('x2')
plt.ylabel('p(x2)')

fig.tight_layout()
plt.show()
```

## Gaussian Processes (GPs)

A Gaussian process is a Gaussian distribution over functions. That is, it is a generalization of the multivariable Gaussian distribution to infinite vectors.

It will become clearer with an example.

```python
x = [0.5,0.8,1.4]
f = [1,2,6]

plt.plot(x,f,'o')
for i in range(len(x)):
    plt.annotate('f'+str(i+1), (x[i],f[i]))
plt.xlim(0,2)
plt.ylim(0,6.5)
plt.ylabel('f(x)')
plt.xlabel('x')
plt.xticks(x, ['x'+str(i+1) for i in range(len(x))])
plt.show()
```

\begin{align}
    \left[ \begin{array}{c} f_1 \\ f_2 \\ f_3 \end{array}\right]
    &\sim \mathcal{N}\left( \left[ \begin{array}{c} 0 \\ 0 \\ 0 \end{array}\right], \left[ \begin{array}{ccc} K_{11} & K_{12} & K_{13} \\ K_{21} & K_{22} & K_{23} \\ K_{31} & K_{32} & K_{33} \end{array}\right] \right) \\
    &\sim \mathcal{N}\left( \left[ \begin{array}{c} 0 \\ 0 \\ 0 \end{array}\right], \left[ \begin{array}{ccc} 1 & 0.7 & 0.2 \\ 0.7 & 1 & 0.6 \\ 0.2 & 0.6 & 1 \end{array}\right] \right)
\end{align}

Similarity measure: $K_{ij} = \exp(- ||x_i - x_j||^2) = \left\{ \begin{array}{ll} 0 & ||x_i - x_j|| \rightarrow \infty \\ 1 & x_i = x_j \end{array} \right.$

Prediction (noiseless GP regression): given data $\mathcal{D} = \{(x_1,f_1), (x_2,f_2), (x_3,f_3)\}$, and new point $x_*$ (e.g. $x_*$=1.4), what is the value of $f_*$?

\begin{equation}
    \pmb{f} \sim \mathcal{N}(\pmb{0}, \pmb{K}) \qquad \mbox{and} \qquad f_* \sim \mathcal{N}(0, K(x_*,x_*)) = \mathcal{N}(0, K_{**})
\end{equation}

In this case, $K_{**} = K(x_*,x_*) = \exp(- ||x_* - x_*||^2) = 1$.

Now, we can write:

\begin{equation}
        \left[ \begin{array}{c} \pmb{f} \\ f_* \end{array} \right] \sim \mathcal{N}\left(\pmb{0}, \left[\begin{array}{cc} \left[ \begin{array}{ccc} K_{11} & K_{12} & K_{13} \\ K_{21} & K_{22} & K_{23} \\ K_{31} & K_{32} & K_{33} \end{array} \right] & \left[ \begin{array}{c} K_{1*} \\ K_{2*} \\ K_{3*} \end{array} \right] \\ \left[ \begin{array}{ccc} K_{*1} & K_{*2} & K_{*3} \end{array} \right] & \left[\begin{array}{c} K_{**} \end{array} \right] \end{array} \right] \right) = \mathcal{N}\left(\pmb{0}, \left[\begin{array}{cc} \pmb{K} & \pmb{K}_* \\ \pmb{K}_*^T & \pmb{K}_{**} \end{array}\right]\right)
\end{equation}

Using the formula for the conditional probability $p(f_*|f)$, we have:

\begin{align}
    \mu_* &= \mathbb{E}[f_*] = \pmb{K}_*^T \pmb{K}^{-1}\pmb{f} \\
    c_* &= K_{**} - \pmb{K}_*^T \pmb{K}^{-1}\pmb{K}_*
\end{align}

We can thus predict the mean $\mu_*$ and the variance $c_*$ for the test point $x_*$.

```python
x = [0.5,0.8,1.4]
f = [1,2,6]

x_new = 1.3
f_new = 5.2

plt.plot(x+[x_new],f+[f_new],'o')
for i in range(len(x)):
    plt.annotate('f'+str(i+1), (x[i],f[i]))
plt.errorbar(x_new, f_new, yerr=1)
plt.annotate('f*', (x_new+0.02, f_new))
plt.xlim(0,2)
plt.ylim(0,6.5)
plt.ylabel('f(x)')
plt.xlabel('x')
plt.xticks(x+[x_new], ['x'+str(i+1) for i in range(len(x))]+['x*'])
plt.show()
```

### Generalization

A GP defines a distribution over functions $p(f)$ (i.e. it is the joint distribution over all the infinite function values).

Definition: $p(f)$ is a GP if for any finite subset $\{x_1,...,x_n\} ⊂ X$, the marginal distribution over that finite subset $p(f)$ has a multivariate Gaussian distribution.

Prior on $f$:
\begin{equation}
    \pmb{f}|\pmb{x} \sim \mathcal{GP}(\pmb{\mu}(\pmb{x}), \pmb{K}(\pmb{x}, \pmb{x}))
\end{equation}
with
\begin{align*}
    \pmb{\mu}(\pmb{x}) &= \mathbb{E}_f \lbrack \pmb{x} \rbrack \\
    k(\pmb{x}, \pmb{x'}) &= \mathbb{E}_f \lbrack (\pmb{x} - \pmb{\mu}(\pmb{x})) (\pmb{x'} - \pmb{\mu}(\pmb{x'})) \rbrack
\end{align*}

Often written as:
\begin{equation}
    \pmb{f} \sim \mathcal{GP}(\pmb{0}, \pmb{K})
\end{equation}

Concretely, assume $\pmb{x} \in \mathbb{R}^{50}$, then $\pmb{K}(\pmb{x}, \pmb{x}) \in \mathbb{R}^{50 \times 50}$, then $\pmb{f} \sim \mathcal{GP}(\pmb{0}, \pmb{K})$ means:

\begin{equation}
    \left[ \begin{array}{c} f_1 \\ \vdots \\ f_{50} \end{array}\right] := \left[ \begin{array}{c} f(x_1) \\ \vdots \\ f(x_{50}) \end{array}\right]  \sim \mathcal{N}\left( \left[ \begin{array}{c} 0 \\ \vdots \\ 0 \end{array}\right], \left[ \begin{array}{ccc} k(x_1,x_1) & \cdots & k(x_1, x_{50}) \\ \vdots & \ddots & \vdots \\ k(x_{50},x_1) & \cdots & k(x_{50}, x_{50}) \end{array} \right] \right)
\end{equation}

#### RBF kernel

Let's choose a RBF (a.k.a Squared Exponential, Gaussian) kernel:

\begin{equation}
\pmb{K} = \left[ \begin{array}{ccc} k(x_1,x_1) & \cdots & k(x_1, x_d) \\ \vdots & \ddots & \vdots \\ k(x_d,x_1) & \cdots & k(x_d, x_d) \end{array} \right]
\end{equation}
with
\begin{equation}
    k(x_i, x_j) = \alpha^2 \exp \left( - \frac{(x_i - x_j)^2}{2l} \right) \qquad \mbox{ and hyperparameters } \pmb{\Phi} =  \left\{ \begin{array}{l} \alpha \mbox{: amplitude} \\ l \mbox{: the lengthscale} \end{array} \right.
\end{equation}

This function $k$ is infinitely differentiable.

```python
# Reference: https://www.youtube.com/watch?v=4vGiHC35j9s&t=51s

# Hyperparameters
alpha = 1
l = 2

# Parameters
n = 50 # nb of points
n_func = 10 # nb of fct to draw
x_bound = 5 # bound on the x axis

def RBF_kernel(a,b):
    sqdist = np.sum(a**2,1).reshape(-1,1) + np.sum(b**2,1) - 2*np.dot(a,b.T)
    return alpha**2 * np.exp(-1/l * sqdist)

n = 50
X = np.linspace(-x_bound, x_bound, n).reshape(-1,1)
K = RBF_kernel(X, X) # dim(K) = n x n

L = np.linalg.cholesky(K + 1e-6 * np.eye(n))
f_prior = np.dot(L, np.random.normal(size=(n, n_func)))

# Plotting
width = 16
height = 4
plt.figure(figsize=(width, height))

# plot f_prior
plt.subplot(1,3,1)
plt.title('GP: prior on f')
plt.plot(X, f_prior)
plt.plot(X, f_prior.mean(1), linewidth=3, color='black')
plt.ylabel('f(x)')
plt.xlabel('x')

# plot Kernel
plt.subplot(1,3,2)
plt.title('Kernel matrix')
plt.pcolor(K[::-1])
plt.colorbar()

plt.subplot(1,3,3)
plt.title('Kernel function')
plt.plot(X, RBF_kernel(X, np.array([[1.0]])))
plt.show()
```

### Kernel (prior knowledge)

By choosing a specific kernel, we can incorporate prior knowledge that we have about the function $f$, such as, if the function is:
* periodic
* smooth
* symmetric
* etc.

The hyperparameters for each kernel are also very intuitive/interpretable.

Note: kernels can be combined!

Indeed, if $k(x,y)$, $k_1(x,y)$ and $k_2(x,y)$ are valid kernels then:
* $\alpha k(x,y) $ with $\alpha \geq 0$
* $k_1(x,y) + k_2(x,y)$
* $k_1(x,y) k_2(x,y)$
* $p(k(x,y))$ with $p$ being a polynomial function with non-negative coefficients
* $exp(k(x,y))$
* $f(x) k(x,y) \overline{f(y)}$ with $\overline{f} = $ complex conjugate
* $k(\phi(x),\phi(y))$

are all valid kernels!

##### Periodic Exponential kernel

```python
variance = 1.
lengthscale = 1.
period = 2.*np.pi

#K = periodic_kernel(X, X) # dim(K) = n x n
kern = GPy.kern.PeriodicExponential(variance=variance, lengthscale=lengthscale, period=period)
K1 = kern.K(X)

L = np.linalg.cholesky(K1 + 1e-6 * np.eye(n))
f_prior = np.dot(L, np.random.normal(size=(n, 1)))

# Plotting
width = 16
height = 4
plt.figure(figsize=(width, height))

# plot f_prior
plt.subplot(1,3,1)
plt.title('GP: prior on f')
plt.plot(X, f_prior)
plt.plot(X, f_prior.mean(1), linewidth=3, color='black')
plt.ylabel('f(x)')
plt.xlabel('x')

# plot Kernel
plt.subplot(1,3,2)
plt.title('Kernel matrix')
plt.pcolor(K1[::-1])
plt.colorbar()

plt.subplot(1,3,3)
plt.title('Kernel function')
plt.plot(X, kern.K(X, np.array([[1.0]])))
plt.show()
```

##### addition and multiplication of 2 kernels (SE and PE)

```python
K_add = K + K1

L = np.linalg.cholesky(K_add + 1e-6 * np.eye(n))
f_prior = np.dot(L, np.random.normal(size=(n, n_func)))

# Plotting
width = 16
height = 8
plt.figure(figsize=(width, height))

# plot f_prior
plt.subplot(2,2,1)
plt.title('GP: prior on f with K_add')
plt.plot(X, f_prior)
plt.plot(X, f_prior.mean(1), linewidth=3, color='black')
plt.ylabel('f(x)')
plt.xlabel('x')

# plot Kernel
plt.subplot(2,2,2)
plt.title('Kernel matrix: K_add')
plt.pcolor(K_add[::-1])
plt.colorbar()

K_prod = K * K1

L = np.linalg.cholesky(K_prod + 1e-6 * np.eye(n))
f_prior = np.dot(L, np.random.normal(size=(n, n_func)))

# plot f_prior
plt.subplot(2,2,3)
plt.title('GP: prior on f with K_prod')
plt.plot(X, f_prior)
plt.plot(X, f_prior.mean(1), linewidth=3, color='black')
plt.ylabel('f(x)')
plt.xlabel('x')

# plot Kernel
plt.subplot(2,2,4)
plt.title('Kernel matrix: K_prod')
plt.pcolor(K_prod[::-1])
plt.colorbar()
plt.show()
```

### GP Posterior

Given $\mathcal{D}=\{(x_i, y_i)\}_{i=1}^{i=N} = (\pmb{X}, \pmb{y})$, we have:

\begin{equation}
    p(f|\mathcal{D}) = \frac{p(\mathcal{D}|f)p(f)}{p(\mathcal{D})}
\end{equation}

### GP Regression

\begin{equation}
    y_i = f(\pmb{x}_i) + \epsilon_i \qquad
	\left\{ \begin{array}{l}
		f \sim \mathcal{GP}(\pmb{0}, \pmb{K}) \\
		\epsilon_i \sim \mathcal{N}(0, \sigma^2)
	\end{array} \right.
\end{equation}

* Prior $f$ is a GP $\Leftrightarrow p(\pmb{f}|\pmb{X}) = \mathcal{N}(\pmb{0}, \pmb{K})$
* Likelihood is Gaussian $\Leftrightarrow p(\pmb{y}|\pmb{X},\pmb{f}) = \mathcal{N}(\pmb{f}, \sigma^2\pmb{I})$
* $\rightarrow p(f|\mathcal{D})$ is also a GP.

#### Predictive distribution:
$$p(\pmb{y}_*|\pmb{x}_*, \pmb{X}, \pmb{y}) = \int p(\pmb{y}_{*}| \pmb{x}_{*}, \pmb{f}, \pmb{X}, \pmb{y}) p(f|\pmb{X}, \pmb{y}) d\pmb{f} = \mathcal{N}(\pmb{\mu}_*, \pmb{\Sigma}_*)$$
\begin{align}
    \pmb{\mu}_* &= \pmb{K}_{*N} (\pmb{K}_N + \sigma^2 \pmb{I})^{-1} \pmb{y} \\
    \pmb{\Sigma}_* &= \pmb{K}_{**} - \pmb{K}_{*N} (\pmb{K}_N + \sigma^2 \pmb{I})^{-1} \pmb{K}_{N*}
\end{align}

### Learning a GP
#### Marginal likelihood:
\begin{equation}
    p(\pmb{y}|\pmb{X}) = \int p(\pmb{y}|\pmb{f},\pmb{X}) p(\pmb{f}|\pmb{X}) d\pmb{f} = \mathcal{N}(\pmb{0}, \pmb{K} + \sigma^2\pmb{I})
\end{equation}

By taking the logarithm, and setting $\pmb{K}_y = (\pmb{K} + \sigma^2\pmb{I})$, we have:

\begin{equation}
    \mathcal{L} = \log p(\pmb{y}|\pmb{X}; \pmb{\Phi}) = \underbrace{-\frac{1}{2} \pmb{y}^T \pmb{K}_y^{-1} \pmb{y}}_{\mbox{data fit}} \underbrace{-\frac{1}{2} \log |\pmb{K}_y^{-1}|}_{\mbox{complexity penalty}} - \frac{n}{2} \log 2\pi
\end{equation}

The marginal likelihood (i.e. ML-II) is used to optimize the hyperparameters $\pmb{\Phi}$ that defines the covariance function and thus the GP.

\begin{equation}
    \pmb{\Phi}^* = argmax_{\pmb{\Phi}} \log p(\pmb{y}|\pmb{X}; \pmb{\Phi})
\end{equation}

Optimizing the marginal likelihood is more robust than the likelihood as it tries to optimize the complexity of the model, and the fitting of this last one to the observed data.

### GPy

```python
# GP Regression
# Based on the tutorial: https://github.com/SheffieldML/notebook/blob/master/GPy/GPyCrashCourse.ipynb

# Create dataset
X = np.random.uniform(-3.0, 3.0, (20,1))
Y = np.sin(X) + np.random.randn(20,1) * 0.05

# Create the kernel
# Reminder 1: The sum of valid kernels gives a valid kernel.
# Reminder 2: The product of valid kernels gives a valid kernel.
# Available kernels: RBF, Exponential, Matern32, Matern52, Brownian, Bias, Linear, PeriodicExponential, White.
kernel = GPy.kern.RBF(input_dim=1, variance=1.0, lengthscale=1.0)

# Create the model
gp_model = GPy.models.GPRegression(X, Y, kernel)

# Display and plot
print("Before optimization: ", gp_model)
gp_model.plot()
plt.show()

# Optimize the model (that is find the 'best' hyperparameters of the kernel matrix)
# By default, the optimizer is a 2nd order algo: lbfgsb. Others are available such as the scg, ...
gp_model.optimize(messages=False)

# Display and plot
print("After optimization: ", gp_model)
gp_model.plot()
plt.show()
```

### Gaussian Process Latent Variable Model (GP-LVM)

```python
# GPLVM
# Based on the tutorials:
# http://nbviewer.jupyter.org/github/SheffieldML/notebook/blob/master/GPy/MagnificationFactor.ipynb
# https://github.com/SheffieldML/notebook/blob/master/lab_classes/gprs/lab4-Copy0.ipynb

# Create dataset
N = 100
k1 = GPy.kern.RBF(5, variance=1, lengthscale=1./np.random.dirichlet(np.r_[10,10,10,0.1,0.1]), ARD=True)
k2 = GPy.kern.RBF(5, variance=1, lengthscale=1./np.random.dirichlet(np.r_[0.1,10,10,10,0.1]), ARD=True)
X = np.random.normal(0, 1, (N,5))
A = np.random.multivariate_normal(np.zeros(N), k1.K(X), 10).T
B = np.random.multivariate_normal(np.zeros(N), k2.K(X), 10).T

Y = np.vstack((A,B))

# latent space dimension
latent_dim = 2

# Create the kernel
kernel = GPy.kern.RBF(input_dim=latent_dim, variance=1.0, lengthscale=1.0)

# Create the GPLVM model
gplvm_model = GPy.models.GPLVM(Y, latent_dim, init='PCA', kernel=kernel)

# Display and plot
print("Before optimization: ", gplvm_model)
gplvm_model.plot_latent()
plt.show()

# Optimize the model (that is find the 'best' hyperparameters of the kernel matrix)
# By default, the optimizer is a 2nd order algo: lbfgsb. Others are available such as the scg, ...
gplvm_model.optimize(messages=False)

# Display and plot
print("After optimization: ", gplvm_model)
gplvm_model.plot_latent()
plt.show()
```


# Solving differential equations in Julia

## Define your model and find parameters

The concentration of a decaying nuclear isotope could be described as an exponential decay:

$$
\frac{d}{dt}C(t) = - \lambda C(t)
$$

**State variable**
- $C(t)$: The concentration of a decaying nuclear isotope.

**Parameter**
- $\lambda$: The rate constant of decay. The half-life $t_{\frac{1}{2}} = \frac{ln2}{\lambda}$

Take a more complex model, the spreading of an contagious disease can be described by the [SIR model](https://www.maa.org/press/periodicals/loci/joma/the-sir-model-for-spread-of-disease-the-differential-equation-model):

$$
\begin{align}
\frac{d}{dt}S(t) &= - \beta S(t)I(t)  \\
\frac{d}{dt}I(t) &= \beta S(t)I(t)  - \gamma I(t)  \\
\frac{d}{dt}R(t) &= \gamma I(t)
\end{align}
$$

**State variables**

- $S(t)$ : the fraction of susceptible people
- $I(t)$ : the fraction of infectious people
- $R(t)$ : the fraction of recovered (or removed) people

**Parameters**

- $\beta$ : the rate of infection when susceptible and infectious people meet
- $\gamma$ : the rate of recovery of infectious people

## Make a solver by yourself

### Forward Euler method

The most straightforward approach to numerically solve differential equations is the forward Euler's (FE) method[^Euler].

In each step, the next state variables ($\vec{u}_{n+1}$) is accumulated by the product of the size of time step (dt) and the derivative at the current state ($\vec{u}_{n}$):

$$
\vec{u}_{n+1} = \vec{u}_{n} + dt \cdot f(\vec{u}_{n}, t_{n})
$$

```julia
# The ODE model. Exponential decay in this example
# The input/output format is compatible to Julia DiffEq ecosystem
expdecay(u, p, t) = p * u

# Forward Euler stepper
step_euler(model, u, p, t, dt) = u .+ dt .* model(u, p, t)

# In house ODE solver
function mysolve(model, u0, tspan, p; dt=0.1, stepper=step_euler)
    # Time points
    ts = tspan[1]:dt:tspan[end]
    # State variable at those time points
    us = zeros(length(ts), length(u0))
    # Initial conditions
    us[1, :] .= u0
    # Iterations
    for i in 1:length(ts)-1
        us[i+1, :] .= stepper(model, us[i, :], p, ts[i], dt)
    end
    # Results
    return (t = ts, u = us)
end

tspan = (0.0, 2.0)
p = -1.0
u0 = 1.0

sol = mysolve(expdecay, u0, tspan, p, dt=0.1, stepper=step_euler)

# Visualization
using Plots
Plots.gr(lw=2)

# Numericalsolution
plot(sol.t, sol.u, label="FE method")

# True solution
plot!(x -> exp(-x), 0.0, 2.0, label="Analytical solution")
```

```julia
# SIR model
function sir(u, p ,t)
	s, i, r = u
	β, γ = p
	v1 = β * s * i
	v2 = γ * i
	return [-v1, v1-v2, v2]
end

p = (β = 1.0, γ = 0.3)
u0 = [0.99, 0.01, 0.00]  # s, i, r
tspan = (0.0, 20.0)

sol = mysolve(sir, u0, tspan, p, dt=0.5, stepper=step_euler)

plot(sol.t, sol.u, label=["S" "I" "R"], legend=:right)
```

### The fourth order Runge-Kutta (RK4) method

One of the most popular ODE-solving methods is the fourth order Runge-Kutta ([RK4](https://en.wikipedia.org/wiki/Runge%E2%80%93Kutta_methods)) method.

In each step, the next state is calculated in 5 steps, 4 of which are intermediate steps.

$$
\begin{align}
k_1 &= dt \cdot f(\vec{u}_{n}, t_n)  \\
k_2 &= dt \cdot f(\vec{u}_{n} + 0.5k_1, t_n + 0.5dt)  \\
k_3 &= dt \cdot f(\vec{u}_{n} + 0.5k_2, t_n + 0.5dt)  \\
k_4 &= dt \cdot f(\vec{u}_{n} + k_3, t_n + dt)  \\
u_{n+1} &= \vec{u}_{n} + \frac{1}{6}(k_1 + 2k_2 + 2k_3 + k_4)
\end{align}
$$

In homework 1, you are going to replace the Euler stepper with the RK4 one:

```julia
step_rk4(f, u, p, t, dt) = """TODO"""
```

## Using DifferentialEquations.jl

Documentation: <https://diffeq.sciml.ai/dev/index.html>

```julia
using Plots, DifferentialEquations
Plots.gr(linewidth=2)
```

### Exponential decay model

```julia
# Parameter of exponential decay
p = -1.0
u0 = 1.0
tspan = (0.0, 2.0)

# Define a problem
prob = ODEProblem(expdecay, u0, tspan, p)

# Solve the problem
sol = solve(prob)

# Visualize the solution
plot(sol, legend=:right)
```

### SIR model

```julia
# Parameters of the SIR model
p = (β = 1.0, γ = 0.3)
u0 = [0.99, 0.01, 0.00]  # s, i, r
tspan = (0.0, 20.0)

# Define a problem
prob = ODEProblem(sir, u0, tspan, p)

# Solve the problem
sol = solve(prob)

# Visualize the solution
plot(sol, label=["S" "I" "R"], legend=:right)
```

```julia
plot(sol, vars=(0, 2), legend=:right)
```

```julia
plot(sol, vars=(1, 2), legend=:right)
```

## Using ModelingToolkit.jl

[ModelingToolkit.jl](https://mtk.sciml.ai/dev/) is a high-level package for symbolic-numeric modeling and simulation ni the Julia DiffEq ecosystem.

```julia
using DifferentialEquations
using ModelingToolkit
using Plots
Plots.gr(linewidth=2)
```

### Exponential decay model

```julia
@parameters λ       # Decaying rate constant
@variables t C(t)   # Time and concentration

D = Differential(t) # Differential operator

# Make an ODE system
@named expdecaySys = ODESystem([D(C) ~ -λ*C ])
```

```julia
u0 = [C => 1.0]
p = [λ => 1.0]
tspan = (0.0, 2.0)

prob = ODEProblem(expdecaySys, u0, tspan, p)
sol = solve(prob)

plot(sol)
```

### SIR model

```julia
@parameters β γ
@variables t s(t) i(t) r(t)

D = Differential(t) # Differential operator

# Make an ODE system
@named sirSys = ODESystem(
    [D(s) ~ -β * s * i,
     D(i) ~ β * s * i - γ * i,
     D(r) ~ γ * i])
```

```julia
# Parameters of the SIR model
p = [β => 1.0, γ => 0.3]
u0 = [s => 0.99, i => 0.01, r => 0.00]
tspan = (0.0, 20.0)

prob = ODEProblem(sirSys, u0, tspan, p)
sol = solve(prob)

plot(sol)
```

## Using Catalyst.jl

[Catalyst.jl](https://github.com/SciML/Catalyst.jl) is a domain-specific language (DSL) package to solve "law of mass action" problems.

```julia
using Catalyst
using DifferentialEquations
using Plots
Plots.gr(linewidth=2)
```

### Exponential decay model

```julia
decayModel = @reaction_network begin
    λ, C --> 0
end λ
```

```julia
p = [1.0]
u0 = [1.0]
tspan = (0.0, 2.0)

prob = ODEProblem(decayModel, u0, tspan, p)
sol = solve(prob)

plot(sol)
```

### SIR model

```julia
sirModel = @reaction_network begin
    β, S + I --> 2I
    γ, I --> R
end β γ
```

```julia
# Parameters of the SIR model
p = (1.0, 0.3)
u0 = [0.99, 0.01, 0.00]
tspan = (0.0, 20.0)

prob = ODEProblem(sirModel, u0, tspan, p)
sol = solve(prob)

plot(sol, legend=:right)
```


# For today's code challenge you will be reviewing yesterdays lecture material. Have fun!

### if you get done early check out [these videos](https://www.3blue1brown.com/neural-networks).

# The Perceptron

The first and simplest kind of neural network that we could talk about is the perceptron. A perceptron is just a single node or neuron of a neural network with nothing else. It can take any number of inputs and spit out an output. What a neuron does is it takes each of the input values, multplies each of them by a weight, sums all of these products up, and then passes the sum through what is called an "activation function" the result of which is the final value.

I really like figure 2.1 found in this [pdf](http://www.uta.fi/sis/tie/neuro/index/Neurocomputing2.pdf) even though it doesn't have bias term represented there.

If we were to write what is happening in some verbose mathematical notation, it might look something like this:

\begin{align}
 y = sigmoid(\sum(weight_{1}input_{1} + weight_{2}input_{2} + weight_{3}input_{3}) + bias)
\end{align}

Understanding what happens with a single neuron is important because this is the same pattern that will take place for all of our networks.

When imagining a neural network I like to think about the arrows as representing the weights, like a wire that has a certain amount of resistance and only lets a certain amount of current through. And I like to think about the node itselef as containing the prescribed activation function that neuron will use to decide how much signal to pass onto the next layer.

# Activation Functions (transfer functions)

In Neural Networks, each node has an activation function. Each node in a given layer typically has the same activation function. These activation functions are the biggest piece of neural networks that have been inspired by actual biology. The activation function decides whether a cell "fires" or not. Sometimes it is said that the cell is "activated" or not. In Artificial Neural Networks activation functions decide how much signal to pass onto the next layer. This is why they are sometimes referred to as transfer functions because they determine how much signal is transferred to the next layer.

## Common Activation Functions:

# Implementing a Perceptron from scratch in Python

### Establish training data

```python
import numpy as np

np.random.seed(812)

inputs = np.array([
    [0, 0, 1],
    [1, 1, 1],
    [1, 0, 1],
    [0, 1, 1]
])

correct_outputs = [[0], [1], [1], [0]]
```

### Sigmoid activation function and its derivative for updating weights

```python
def sigmoid(x):
  return 1 / (1 + np.exp(-x))

def sigmoid_derivative(x):
  sx = sigmoid(x)
  return sx * (1 - sx)
```

## Updating weights with derivative of sigmoid function:

### Initialize random weights for our three inputs

```python

```

### Calculate weighted sum of inputs and weights

```python

```

### Output the activated value for the end of 1 training epoch

```python

```

### take difference of output and true values to calculate error

```python

```

### Put it all together

```python

```


# Solution {-}
The random variable $X$ is completely described by its mean and covariance matrix:
\begin{equation*}
  \mu=
  \begin{bmatrix}
    1 \\
    2 \\
  \end{bmatrix} \quad
  C_X=
  \begin{bmatrix}
    4 &1\\
    1 &1\\
  \end{bmatrix}
\end{equation*}

Now consider another random variable $Y$ that is functionally related to $X$ by:
\begin{equation*}
  y=Ax+b
\end{equation*}

where
\begin{equation*}
  A=
  \begin{bmatrix}
    2 &1\\
    1 &-1\\
  \end{bmatrix} \quad
  b=
  \begin{bmatrix}
    1\\
    1\\
  \end{bmatrix}
\end{equation*}

Find the mean and the covariance matrix for $Y$:

```python
from sympy import Matrix

mx = Matrix([[1],
             [2]])
Cx = Matrix([[4, 1],
             [1, 1]])

A = Matrix([[2,  1],
            [1, -1]])
b = Matrix([[1],
            [1]])

my = A@mx + b
Cy = A@Cx@A.T

display(my)
display(Cy)
```

$\displaystyle \left[\begin{matrix}5\\0\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}21 & 6\\6 & 3\end{matrix}\right]$


# Computing the Z-Normalized Euclidean Distance from Dot Products

In the [Matrix Profile I](https://www.cs.ucr.edu/~eamonn/STOMP_GPU_final_submission_camera_ready.pdf) and [Matrix Profile II](https://www.cs.ucr.edu/~eamonn/PID4481997_extend_Matrix%20Profile_I.pdf) papers, the Z-normalized Euclidean distance between a query subsequence, $Q_{i,m}=(q_i, q_{i+1}, q_{i+2}\ldots, q_{i+m-1})$, and the $i^{th}$ subsequence, $T_{i,m}=(t_i, t_{i+1}, t_{i+2}, \ldots, t_{i+m-1})$, with window size, $m$, in the time series, $T$, can be computed following:

\begin{align}
    D(Q_{i,m}, T_{i,m}) ={}&
        \sqrt{
            \sum \limits _{0 \leq {j} \lt m}
                \left(
                    \frac{t_{i+j}-M_{T_{i,m}}}{\Sigma_{T_{i,m}}}
                    -
                    \frac{q_{i+j}-\mu_{Q_{i,m}}}{\sigma_{Q_{i,m}}}
                \right)^2
        }
    \\
    ={}&
        \sqrt{
            \sum \limits _{0 \leq {j} \lt m}
                \left[
                    \left(
                        \frac{t_{i+j}-M_{T_{i,m}}}{\Sigma_{T_{i,m}}}
                    \right)
                    -
                    2
                    \left(
                        \frac{t_{i+j}-M_{T_{i,m}}}{\Sigma_{T_{i,m}}}
                    \right)
                    \left(
                        \frac{q_{i+j}-\mu_{Q_{i,m}}}{\sigma_{Q_{i,m}}}
                    \right)
                    +
                    \left(
                        \frac{q_{i+j}-\mu_{Q_{i,m}}}{\sigma_{Q_{i,m}}}
                    \right)^2
                \right]
        }
    \\
    ={}&
        \sqrt{
            \sum \limits _{0 \leq {j} \lt m}
                \left(
                    \frac{t_{i+j}-M_{T_{i,m}}}{\Sigma_{T_{i,m}}}
                \right)^2
                -
            \sum \limits _{0 \leq {j} \lt m}
                2
                \left(
                    \frac{t_{i+j}-M_{T_{i,m}}}{\Sigma_{T_{i,m}}}
                \right)
                \left(
                    \frac{q_j-\mu_{Q_{i,m}}}{\sigma_{Q_{i,m}}}
                \right)
                +
            \sum \limits _{0 \leq {j} \lt m}
                \left(
                    \frac{q_{i+j}-\mu_{Q_{i,m}}}{\sigma_{Q_{i,m}}}
                \right)^2
        }
    \\
    ={}&
        \sqrt{
            m
            -
            \sum \limits _{0 \leq {j} \lt m}
                2
                \left(
                    \frac{t_{i+j}-M_{T_{i,m}}}{\Sigma_{T_{i,m}}}
                \right)
                \left(
                    \frac{q_{i+j}-\mu_{Q_{i,m}}}{\sigma_{Q_{i,m}}}
                \right)
            +
            m
        }
    \\
    ={}&
        \sqrt{
            2m
            -
            2
            \sum \limits _{0 \leq {j} \lt m}
                \left(
                    \frac{t_{i+j}-M_{T_{i,m}}}{\Sigma_{T_{i,m}}}
                \right)
                \left(
                    \frac{q_{i+j}-\mu_{Q_m}}{\sigma_{Q_{i,m}}}
                \right)
        }
    \\
    ={}&
        \sqrt{
            2m
            \left[
                1
                -
                \frac{1}{m}
                \sum \limits _{0 \leq {j} \lt m}
                    \left(
                        \frac{t_{i+j}-M_{T_{i,m}}}{\Sigma_{T_{i,m}}}
                    \right)
                    \left(
                        \frac{q_{i+j}-\mu_{Q_{i,m}}}{\sigma_{Q_{i,m}}}
                    \right)
            \right]
        }
    \\
    ={}&
        \sqrt{
            2m
            \left[
                1
                -
                \sum \limits _{0 \leq {j} \lt m}
                    \frac{
                        \left(
                            t_{i+j}-M_{T_{i,m}}
                        \right)
                        \left(
                            q_{i+j}-\mu_{Q_{i,m}}
                        \right)
                    }{
                        m \sigma_{Q_{i,m}} \Sigma_{T_{i,m}}
                    }
            \right]
        }
    \\
    ={}&
        \sqrt{
            2m
            \left[
                1
                -
                \sum \limits _{0 \leq {j} \lt m}
                    \frac{
                        t_{i+j}q_j
                        -t_{i+j}\mu_{Q_{i,m}}
                        -M_{T_{i,m}}q_{i+j}
                        +M_{T_{i,m}}\mu_{Q_{i+m}}
                    }{
                        m \sigma_{Q_{i+m}} \Sigma_{T_{i,m}}
                    }
            \right]
        }
    \\
    ={}&
        \sqrt{
            2m
            \left[
                1
                -
                \frac{
                    \sum \limits _{0 \leq {j} \lt m}
                        q_{i+j}t_{i+j}
                    -
                    \sum \limits _{0 \leq {j} \lt m}
                        t_{i+j}\mu_{Q_{i,m}}
                    -
                    \sum \limits _{0 \leq {j} \lt m}
                        \left(
                            M_{T_{i,m}}q_{i+j}
                            -M_{T_{i,m}}{\mu_{Q_{i,m}}}
                        \right)
                }{
                    m \sigma_{Q_{i,m}} \Sigma_{T_{i,m}}
                }
            \right]
        }
    \\
    ={}&
        \sqrt{
            2m
            \left[
                1
                -
                \frac{
                    Q_{i,m}\cdot{T_{i,m}}
                    -
                    \sum \limits _{0 \leq {j} \lt m}
                        t_{i+j}\mu_{Q_{i,m}}
                    -
                    M_{T_{i,m}}
                    \sum \limits _{0 \leq {j} \lt m}
                        \left(
                            q_{i+j}-{\mu_{Q_{i,m}}}
                        \right)
                }{
                    m \sigma_{Q_{i,m}} \Sigma_{T_{i,m}}
                }
            \right]
        }
    \\
    ={}&
        \sqrt{
            2m
            \left[
                1
                -
                \frac{
                    Q_{i,m}\cdot{T_{i,m}}
                    -
                    \mu_{Q_{i,m}}
                    \sum \limits _{0 \leq {j} \lt m}
                        t_{i+j}
                    -
                    0
                }{
                    m \sigma_{Q_{i,m}} \Sigma_{T_{i,m}}
                }
            \right]
        }
    \\
    ={}&
        \sqrt{
            2m
            \left[
                1
                -
                \frac{
                    Q_{i,m}\cdot{T_{i,m}}
                    -
                    \mu_{Q_{i,m}}
                    \sum \limits _{0 \leq {j} \lt m}
                        t_{i+j}
                }{
                    m \sigma_{Q_{i,m}} \Sigma_{T_{i,m}}
                }
            \right]
        }
    \\
    ={}&
        \sqrt{
            2m
            \left[
                1
                -
                \frac{
                    Q_{i,m}\cdot{T_{i,m}}
                    -
                    \mu_{Q{i,m}}m
                    \sum \limits _{0 \leq {j} \lt m}
                        \frac{t_{i+j}}{m}
                }{
                    m \sigma_{Q_{i,m}} \Sigma_{T_{i,m}}
                }
            \right]
        }
    \\
    ={}&
        \sqrt{
            2m
            \left[
                1
                -
                \frac{
                    Q_{i,m}\cdot{T_{i,m}}
                    -
                    m\mu_{Q_{i,m}}M_{T_{i,m}}
                }{
                    m \sigma_{Q_{i,m}} \Sigma_{T_{i,m}}
                }
            \right]
        }
    \\
\end{align}

# Computing the  Z-Normalized Euclidean Distance from the Pearson Correlation

Based on the fact that the Pearson Correlation, $\rho$, can be written as (see Equation 4 in [this paper](https://www.cs.unm.edu/~mueen/Projects/JOCOR/joinICDM.pdf) or Equation 3 in [this paper](https://arxiv.org/pdf/1601.02213.pdf)):

\begin{align}
    \rho(Q_{i,m}, T_{i,m}) ={}& \frac{E
        \left[
            \left(
                Q_{i,m}-\mu_{Q_{i,m}}
            \right)
            \left(
                T_{i,m}-M_{T_{i,m}}
            \right)
        \right]
    }{\sigma_{Q_{i,m}}\Sigma_{T_{i,m}}}
    \\
    ={}&
        \frac{
            \langle
            \left(
                Q_{i,m}-\mu_{Q_{i,m}}
            \right)
            ,
            \left(
                T_{i,m}-M_{T_{i,m}}
            \right)
            \rangle
        }{\sigma_{Q_{i,m}}\Sigma_{T_{i,m}}}
    \\
    ={}&
        \frac{1}{m}
        \sum \limits _{0 \leq j \lt m}
            \frac{
                \left(
                    q_{i+j}-\mu_{Q_{i,m}}
                \right)
                \left(
                    t_{i+j}-M_{T_{i,m}}
                \right)
            }{\sigma_{Q_{i,m}}\Sigma_{T_{i,m}}}
    \\
    ={}&
        \frac{1}{m}
        \sum \limits _{0 \leq j \lt m}
            \left(
                \frac{
                    q_{i+j}-\mu_{Q_{i,m}}
                }{\sigma_{Q_{i,m}}}
            \right)
            \left(
                \frac{
                    t_{i+j}-M_{T_{i,m}}
                }{\Sigma_{T_{i,m}}}
            \right)
    \\
\end{align}

Similar to above, the Z-normalized Euclidean distance can be computed from $\rho$ following:

\begin{align}
    D(Q_{i,m}, T_{i,m}) ={}&
        \sqrt{
            \sum \limits _{0 \leq {j} \lt m}
                \left(
                    \frac{t_{i+j}-M_{T_{i,m}}}{\Sigma_{T_{i,m}}}
                    -
                    \frac{q_{i+j}-\mu_{Q_{i,m}}}{\sigma_{Q_{i,m}}}
                \right)^2
        }
        \\
    \vdots
        \\
    ={}&
        \sqrt{
            2m
            \left[
                1
                -
                \frac{1}{m}
                \sum \limits _{0 \leq {j} \lt m}
                    \left(
                        \frac{t_{i+j}-M_{T_{i,m}}}{\Sigma_{T_{i,m}}}
                    \right)
                    \left(
                        \frac{q_{i+j}-\mu_{Q_{i,m}}}{\sigma_{Q_{i,m}}}
                    \right)
            \right]
        }
        \\
    ={}&
        \sqrt{
            2m
            \left[
                1
                -
                \rho(Q_{i,m},T_{i,m})
            \right]
        }
        \\
\end{align}

Thus, by employing the most efficient way to compute $\rho(Q_{i,m},T_{i,m})$, then we'd also have an efficient way to directly compute $D(Q_{i,m},T_{i,m})$. Recall that:

\begin{align}
    \rho(Q_{i,m},T_{i,m}) = \frac{cov(Q_{i,m},T_{i,m})}{\sigma_{Q_{i,m}}\Sigma_{T_{i,m}}}
\end{align}

Thus, it follows that finding the most efficient way to compute the covariance matrix, $cov(Q_{i,m},T_{i,m})$ would result in the most efficient way to compute the distance. Also, remember that we would like to traverse our distance matrix along each diagonal rather than along each row/column.

# Covariance

Recall that the covariance, $cov(Q_{i,m},T_{i,m})$, can be written as:

\begin{align}
    cov(Q_{i,m},T_{i,m}) ={}& E
        \left[
            \left(
                Q-\mu_{Q_{i,m}}
            \right)
            \left(
                T_{i,m}-M_{T_{i,m}}
            \right)
        \right]
    \\
    ={}&
        \langle
            \left(
                Q-\mu_{Q_{i,m}}
            \right)
            ,
            \left(
                T_{i,m}-M_{T_{i,m}}
            \right)
        \rangle
    \\
    ={}&
        \frac{1}{m}
        \sum \limits _{0 \leq j \lt m}
            \left(
                q_{i+j}-\mu_{Q_{i,m}}
            \right)
            \left(
                t_{i+j}-M_{T_{i,m}}
            \right)
    \\
\end{align}

Note that we've explicitly called out the fact that the means, $\mu_{Q_{{i,m}}}$ and $M_{T_{i,m}}$, are computed with the subsequences of length $m$. Additionally, according to Welford, we can express these means with respect to the means of the same subsequences that have their last elements removed (i.e., $\mu_{Q_{i,m-1}}$ and $M_{T_{i,m-1}}$).

\begin{align}
    cov(Q_{i,m},T_{i,m})
    ={}&
        \frac{1}{m}
        \sum \limits _{0 \leq j \lt m}
            \left(
                q_{i+j}-\mu_{Q_{i,m}}
            \right)
            \left(
                t_{i+j}-M_{T_{i,m}}
            \right)
    \\
    ={}&
        \frac{
            S(Q_{i,m-1}, T_{i,m-1})
            +
            \left(
                \frac{m-1}{m}
            \right)
            \left(
                q_{i+m-1} - \mu_{Q_{i,m-1}}
            \right)
            \left(
                t_{i+m-1} - M_{T_{i,m-1}}
            \right)
        }{m}
    \\
    ={}&
        \frac{
            \frac{m-1}{m-1}S(Q_{i,m-1}, T_{i,m-1})
            +
            \left(
                \frac{m-1}{m}
            \right)
            \left(
                q_{i+m-1} - \mu_{Q_{i,m-1}}
            \right)
            \left(
                t_{i+m-1} - M_{T_{i,m-1}}
            \right)
        }{m}
    \\
    ={}&
        \frac{
            cov(Q_{i,m-1},T_{i,m-1}) (m-1)
            +
            \left(
                \frac{m-1}{m}
            \right)
            \left(
                q_{i+m-1} - \mu_{Q_{i,m-1}}
            \right)
            \left(
                t_{i+m-1} - M_{T_{i,m-1}}
            \right)
        }{m}
    \\
    ={}&
        \frac{m-1}{m}
        \left[
            cov(Q_{i,m-1},T_{i,m-1})
            +
            \frac{
                \left(
                    q_{i+m-1} - \mu_{Q_{i,m-1}}
                \right)
                \left(
                    t_{i+m-1} - M_{T_{i,m-1}}
                \right)
            }{m}
        \right]
    \\
\end{align}

Similarly, $cov(Q_{i-1,m},T_{i-1,m})$ can also be expressed with respect to $cov(Q_{i,m-1},T_{i,m-1})$:

\begin{align}
    cov(Q_{i-1,m},T_{i-1,m})
    ={}&
        \frac{1}{m}
        \sum \limits _{0 \leq j \lt m}
            \left(
                q_{i+j-1}-\mu_{Q_{i-1,m}}
            \right)
            \left(
                t_{i+j-1}-M_{T_{i-1,m}}
            \right)
    \\
    ={}&
        \frac{
            S(Q_{i,m-1},T_{i,m-1})
            +
            \frac{m-1}{m}
            \left(
                q_{i-1}
                -
                \mu_{Q_{i,m-1}}
            \right)
            \left(
                t_{i-1}
                -
                M_{T_{i,m-1}}
            \right)
        }{m}
    \\
    ={}&
        \frac{
            \frac{m-1}{m-1}S(Q_{i,m-1},T_{i,m-1})
            +
            \frac{m-1}{m}
            \left(
                q_{i-1}
                -
                \mu_{Q_{i,m-1}}
            \right)
            \left(
                t_{i-1}
                -
                M_{T_{i,m-1}}
            \right)
        }{m}
    \\
    ={}&
        \frac{
            cov(Q_{i,m-1},T_{i,m-1}) (m-1)
            +
            \left(
                \frac{m-1}{m}
            \right)
            \left(
                q_{i-1}
                -
                \mu_{Q_{i,m-1}}
            \right)
            \left(
                t_{i-1}
                -
                M_{T_{i,m-1}}
            \right)
        }{m}
    \\
    ={}&
        \frac{m-1}{m}
        \left[
            cov(Q_{i,m-1},T_{i,m-1})
            +
            \frac{
                \left(
                    q_{i-1}
                    -
                    \mu_{Q_{i,m-1}}
                \right)
                \left(
                    t_{i-1}
                    -
                    M_{T_{i,m-1}}
                \right)
            }{m}
        \right]
    \\
\end{align}

Now, we can rearrange write this and write represent $cov(Q_{i,m-1},T_{i,m-1})$ as a function of $cov(Q_{i-1,m},T_{i-1,m})$:

\begin{align}
    cov(Q_{i-1,m},T_{i-1,m})
    ={}&
        \frac{m-1}{m}
        \left[
            cov(Q_{i,m-1},T_{i,m-1})
            +
            \frac{
                \left(
                    q_{i-1}
                    -
                    \mu_{Q_{i,m-1}}
                \right)
                \left(
                    t_{i-1}
                    -
                    M_{T_{i,m-1}}
                \right)
            }{m}
        \right]
    \\
    \frac{m}{m-1}
    cov(Q_{i-1,m},T_{i-1,m})
    -
    \frac{
        \left(
            q_{i-1}
            -
            \mu_{Q_{i,m-1}}
        \right)
        \left(
            t_{i-1}
            -
            M_{T_{i,m-1}}
        \right)
    }{m}
    ={}&
        cov(Q_{i,m-1},T_{i,m-1})
    \\
\end{align}

And we can then substitute this representation of $cov(Q_{i,m-1},T_{i,m-1})$ into our $cov(Q_{i,m},T_{i,m})$ equation from above and get:

\begin{align}
    cov(Q_{i,m},T_{i,m})
    ={}&
        \frac{m-1}{m}
        \left[
            cov(Q_{i,m-1},T_{i,m-1})
            +
            \frac{
                \left(
                    q_{i+m-1} - \mu_{Q_{i,m-1}}
                \right)
                \left(
                    t_{i+m-1} - M_{T_{i,m-1}}
                \right)
            }{m}
        \right]
    \\
    ={}&
        \frac{m-1}{m}
        \left[
            \frac{m}{m-1}
            cov(Q_{i-1,m},T_{i-1,m})
            -
            \frac{
                \left(
                    q_{i-1}
                    -
                    \mu_{Q_{i,m-1}}
                \right)
                \left(
                    t_{i-1}
                    -
                    M_{T_{i,m-1}}
                \right)
            }{m}
            +
            \frac{
                \left(
                    q_{i+m-1} - \mu_{Q_{i,m-1}}
                \right)
                \left(
                    t_{i+m-1} - M_{T_{i,m-1}}
                \right)
            }{m}
        \right]
    \\
    ={}&
        cov(Q_{i-1,m},T_{i-1,m})
        +
        \frac{m-1}{m^2}
        \left[
            \left(
                q_{i+m-1} - \mu_{Q_{i,m-1}}
            \right)
            \left(
                t_{i+m-1} - M_{T_{i,m-1}}
            \right)
            -
            \left(
                q_{i-1}
                -
                \mu_{Q_{i,m-1}}
            \right)
            \left(
                t_{i-1}
                -
                M_{T_{i,m-1}}
            \right)
        \right]
    \\
\end{align}

# Pearson Correlation

\begin{align}
    \rho(Q_{i,m},T_{i,m})
    &{}=
        \frac{cov(Q_{i,m},T_{i,m})}{\sigma_{Q_{i,m}}\Sigma_{T_{i,m}}}
    \\
    &{}=
        \frac{
            cov(Q_{i-1,m},T_{i-1,m})
        +
        \frac{m-1}{m^2}
        \left[
            \left(
                q_{i+m-1} - \mu_{Q_{i,m-1}}
            \right)
            \left(
                t_{i+m-1} - M_{T_{i,m-1}}
            \right)
            -
            \left(
                q_{i-1}
                -
                \mu_{Q_{i,m-1}}
            \right)
            \left(
                t_{i-1}
                -
                M_{T_{i,m-1}}
            \right)
        \right]
        }{\sigma_{Q_{i,m}}\Sigma_{T_{i,m}}}
    \\
\end{align}

# Z-Normalized Distance

```python

```


### simple *sqrt*

```python
import math
import sympy

math.sqrt(9)
math.sqrt(8)

sympy.sqrt(9)
sympy.sqrt(8)
sympy.sqrt(3)
```

### actual computation

```python
from sympy import *

x,t,z,nu = symbols('x t z nu')
init_printing(use_unicode=True)
```

```python
limit(sin(x)/x,x,0),
limit((3*x**2)/(x-3),x,3)
```

```python
diff(sin(x)),
diff(5*x**3,x),
diff(5*x**3,x,2)
```

```python
solve(x**2-10,x)
```

### *expand* / *factor*

```python
from sympy import symbols
from sympy import expand,factor

x,y = symbols('x y')
expr = x**2 + 2*y
```

```python
expr
expr + 1
expr - x
```

```python
expand(x*expr),
factor(x*expr)
```

### about *symbols*

```python
from sympy import *

x = symbols('x')
y,z = symbols('y z')

x,y,z
```

```python
h = symbols('h')
expr = h ** 2

# a python variable
h = 5

# the sympy symbol 'h'
expr * 2
```

```python
x = symbols('x')
expr = x + 1

expr.subs(x,1)
expr
```

```python
# both '=' and '==' cannot be used in here!
Eq(1,1)
Eq(x**2+y**2,z**2)
Eq((x+1)**2,x**2+2*x+1)
```

```python
a = (x+1)**2
b = x**2 + 2*x + 1
c = x**2 - 2*x + 1

simplify(a-b)
simplify(a-c)
simplify(b-c)
```

```python
a = cos(x)**2 + sin(x)**2
b = a/a

a.equals(b)
a.equals(b+1)
```

    True

    False

### *type* thing

```python
type(1+1)
type(Integer(1)+1)

1+1 == Integer(1)+1
```

    int

    sympy.core.numbers.Integer

    True

```python
# Nah
1/3 , 1//3

# Oh yeah
1/Integer(3), 1//Integer(3)

Rational(1,3) == Integer(1)/3
```

### basic opts

mostly subs

```python
from sympy import *
x,y,z = symbols('x y z')
```

```python
expr = cos(x) + 1

expr.subs(x,sin(0))
expr.subs(x,0)
expr.subs(x,expr.subs(x,0))
```

```python
expr = sin(2*x) + cos(2*x)

expand(expr)
expand_trig(expr)

expr.subs(sin(2*x),2*sin(x)*cos(x))
```

```python
expr = x**2 + y**2

expr.subs([(x,3),(y,4)])
expr.subs([(x,30),(y,40)])
```

```python
expr = x**4 - 4*x**3 + 4*x**2 - 2*x +3

# replace all inst of x that have an even power
#     x**2 -> y**2
#     x**4 -> y**4
replace_evenpow_as_y = [ (x**i,y**i) for i in range(5) if i %2 ==0 ]

expr
expr.subs(replace_evenpow_as_y)
```

```python
# do not confused with 'simplify' (down-below)
sympify("x**2+y**2")
sympify("x**2 + y**2").subs([(x,3),(y,4)])

simplify("(x+1)**2 + (x**2 + 2*x + 1)")
```

```python
expr = sqrt(8)

expr
expr.evalf()
expr.evalf(20)
```

```python
expr = cos(2*x)

expr.evalf(subs={x:0})
expr.evalf(subs={x:pi/2})
expr.evalf(subs={x:pi/6})
```

```python
one = cos(x)**2

one.evalf(subs={x:pi/2})
one.evalf(subs={x:pi/2},chop=True)
```

```python
import numpy
a = numpy.arange(10)
```

```python
f = lambdify(x,sin(x))

f(-30)
f(30)
```

```python
f1 = lambdify(x,cos(x),"numpy")
f2 = lambdify(x,sin(x),"math")

f1(a)

f2(pi/6)
f2(pi/4)
f2(pi/3)
```

### Printing

```python
from sympy import init_printing

# It'll enable the best printer available
init_printing(use_unicode=True)
```

```python
# Or you're in an interactive shell (ipython or qtconsole)
#   >> from sympy import init_session
#   >> init_session()
#   >> init_printing()

# See more at sympy.org
```