© 2018 Suzy Beeler and Vahe Galstyan. This work is licensed under a [Creative Commons Attribution License CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/). All code contained herein is licensed under an [MIT license](https://opensource.org/licenses/MIT)

This exercise was generated from a Jupyter notebook. You can download the notebook [here](diffusion_master_equation.ipynb).
___

# Objective

[Previously](diffusion_via_coin_flips.html) we used a random number generator to obtain stochastic trajectories of diffusive particles. By overlaying lots of such trajectories, we got intuition about the evolution of the probability distribution of walker's position over time. In this tutorial, we will get at this probability distribution more directly by using numerical integration of a master equation for diffusion.

# Master equation for diffusion

As covered in lecture, we can write down the following "master equation" to describe diffusion:

\begin{align}
\overbrace{p(n,t + \Delta t)}^{\text{new probability}} =
\underbrace{p(n, t)}_{\text{old probability}} +
\overbrace{k \Delta t p(n-1, t)}^{\text{jumping from the left}} +
\underbrace{k \Delta t p(n+1, t)}_{\text{jumping from the right}} -
\overbrace{2k \Delta t p(n,t)}^{\text{jumping to the left and right}}. \tag{1}
\end{align}

This master equation allows us to calculate the probability of the walker being at position $n$ at the next time step $t + \Delta t$, given the probabilities at the current time step $t$. $k$ is the hopping rate between neighboring positions, and $\Delta t$ is the time step used in the numerical calculation.

The master equation at the boundaries is slightly modified and is given by

\begin{align}
\overbrace{p(0,t + \Delta t)}^{\text{new probability}} =
\underbrace{p(0, t)}_{\text{old probability}} +
\overbrace{k \Delta t p(1, t)}^{\text{jumping from the right}} -
\underbrace{k \Delta t p(0,t)}^{\text{jumping to the right}}. \tag{2}
\end{align}

for the left edge, and

\begin{align}
\overbrace{p(N,t + \Delta t)}^{\text{new probability}} =
\underbrace{p(N, t) }_{\text{old probability}} +
\overbrace{k \Delta t p(N-1, t)}^{\text{jumping from the right}} -
\underbrace{k \Delta t p(N,t)}^{\text{jumping to the left}}. \tag{3}
\end{align}

for the right edge, where $N$ is the index of the right-most position. These boundary conditions are specified so that particles at the left-most edge can only move to the right and particles at the right-most edge can only move to the left.

```python
# Import modules
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# for nice plots
import seaborn as sns
rc={'lines.linewidth': 2, 'axes.labelsize': 14, 'axes.titlesize': 14, \
    'xtick.labelsize' : 14, 'ytick.labelsize' : 14}
sns.set(rc=rc)

# Import the utilities file
import pboc_utils as pboc
```

# Diffusion of a point source

For our first numerical integration, let's see how an initial point source of particles diffuse over time. To do this, we will set up an array of boxes through which our particles will diffuse, and we will specify the box in the center to initially have probability $1$ (i.e. *all* the particles are found here) and all other boxes will have probability $0$. To do that, we specify the number of boxes, the time increment, the number of time steps and the hopping rate. In general, the time increment needs to be smaller than the hopping timescale to ensure accurate estimation, as we discussed in the [forward euler](forward_euler.ipynb) exercise.

```python
# number of boxes
n_boxes = 40

# time increment (Delta t) in seconds
dt = 0.1

# number of steps
n_steps = 100

# hopping rate constant (k) in sec^-1
k = 1

# initialize the probability distribution
p = np.zeros([n_boxes, n_steps])

# find the middle box
n_center = int(n_boxes / 2)

# set the middle box to have probability 1 at the initial time step
p[n_center, 0] = 1
```

Now we can solve the master equation iteratively, by using the probabilities at the previous time step $t-1$ to evaluate those at the current time step $t$.

```python
# loop through the number of steps
for t in range(1, n_steps):

    # update probability array in the interior region
    for b in range(1,n_boxes-1):
        p[b,t] = p[b,t-1] + k*dt*p[b-1,t-1] + k*dt*p[b+1,t-1] - 2*k*dt*p[b,t-1]

    # update probability array for the left-most box
    p[0,t] = p[0,t-1] + k*dt*p[1,t-1] - k*dt*p[0,t-1]

    # update probability array for the right-most box
    # note that -1 corresponds to the last item
    p[-1, t] = p[-1, t-1] + k*dt*p[-2,t-1] - k*dt*p[-1, t-1]
```

To display the probabilities at different points in time, we will use the custom-written `bar_plot` function in the [`pboc_utilities.py`](pboc_utilities.py) utilities file. As inputs, it takes the 2-dimensional probability array `p`, the number of time slices `n_slices`, the time increment `dy` and the axis labels `x_label` and `y_label`.

```python
pboc.bar_plot(p, n_slices = 6, dy = dt, x_label='Box number', y_label='time (sec)')
```

As we can see, as time progresses, the probability distribution becomes wider and remains centered around the original position of the walker. This result is similar to what we found [previously](diffusion_via_coin_flips.html), but here we've gotten the solution explicitly rather than stochastically. That is, at every time point instead of relying on coin flips, we explicitly calculated how much of the probability should be moving to the left and to right.

# FRAP

Having developed the numerical method of integrating the master equation, let's consider its application to the problem of fluorescence recovery after photobleaching (FRAP). In FRAP experiments, the fluorescently labeled molecules initially have a uniform distribution. Applying high intensity light, the fluorescent molecules in the desired region get photobleached, meaning that they no longer fluoresce. Over time, the fluorescent molecules that were outside the photobleached region will diffuse into it, recovering the uniform fluorescence distribution. By estimating the timescale of fluorescence recovery, experimentalists can determine the diffusion constant of fluorescently labeled molecules.

In this part of the tutorial we will implement the 1-dimensional version of the FRAP experiment. We will initialize the probability distribution as zero in the middle region and uniform outside it. We will when observe the process of recovery as the gap gets "filled" with probability from the two flanking regions.

```python
# need more time steps this time
n_steps = 1000

# initialize the probability distribution
p = np.zeros([n_boxes, n_steps])

# assign non-zero values to the first time step
p[:,0] = 1

# number boxes to photobleach
n_bleach = 10

# find start and end positions for the photobleached region
start = n_center - int(n_bleach/2)
end = n_center + int(n_bleach/2)

# do the "photobleach" by setting the probabilty to 0
p[start:end,0] = 0

# re-normalization so probability sums to 1
p[:,0] = p[:,0] / np.sum(p[:,0])
```

Now we can run the same code from before, just with this new initial condition specified in the probability array.

```python
# loop through the number of steps
for t in range(1, n_steps):

    # update probability array in the interior region
    for b in range(1,n_boxes-1):
        p[b,t] = p[b,t-1] + k*dt*p[b-1,t-1] + k*dt*p[b+1,t-1] - 2*k*dt*p[b,t-1]

    # update probability array for the left-most box
    p[0,t] = p[0,t-1] + k*dt*p[1,t-1] - k*dt*p[0,t-1]

    # update probability array for the right-most box
    # noteL p[-1,t] chooses the last row and the t-th column
    p[-1, t] = p[-1, t-1] + k*dt*p[-2,t-1] - k*dt*p[-1, t-1]
```

Now let's use the utility function `bar_plot` to show the process of recovery.

```python
pboc.bar_plot(p, n_slices = 10, dy = dt, x_label='box number', y_label='time (sec)')
```


# Principal Component Analysis: lecture

## 1. Introduction

Up until now, We have focused on supervised learning. This group of methods aims at predicting labels based on training data that is labeled as well. Principal Componant Analysis is our first so-called "unsupervised" estimator. Generally, the aim of unsupervised estimators is to reveal interesting data patterns without having any reference labels.

The first unsupervised learning algorithm, is Principal Component Analysis, also referred to as PCA. PCA is a dimensionality reduction technique which is often used in practice for visualization, feature extraction, noise filtering, etc.

Generally, PCA would be applied on data sets with many variables. PCA creates new variables that are linear combinations of the original variables. The idea is to reduce the dimension of the data considerably while maintaining as much information as possible. While the purpose is to significantly reduce the dimensionality, the maximum amount of new variables that can possibly be created is equal to the number of original variables. A nice feature of PCA is that the newly created variables are uncorrelated.

**A simple example start** : Imagine that a data set consists of the height and weight of a group of people. One could imagine that these 2 metrics are heavily correlated, so we could basically summarize these 2 metrics in one variable, a linear combination of the two. This one variable will contain most of the information from 2 variables in one. It is important to note that the effectiveness of PCA strongly depends on the structure of the correlation matrix of the existing variables!

## 2. Intermezzo: Eigenvalues and eigenvectors

An eigenvector is a vector that after transformation hasn't changed, except by a scalar value known as the *eigenvalue*.

### 2.1 Definition

If there exists a square matrix $A$ (an n x n matrix) , then a scalar $\lambda$ is called the **eigenvalue** of $A$ if there is a non-zero vector $v$ such that

$$Av = \lambda v$$.

This vector $v$ is thene called the **eigenvector** of A corresponding to $\lambda$.

Eigenvalues and eigenvectors are very useful and have tons of applications!

Imagine you have a matrix

\begin{equation}
A = \begin{bmatrix}
   3 & 2 \\
   3 & -2
\end{bmatrix}
\end{equation}

We have an eigenvector
\begin{equation}
v = \begin{bmatrix}
    2 \\
    1
\end{bmatrix}
\end{equation}

Let's perform the multiplication $A v$

\begin{equation}
Av  = \begin{bmatrix}
   3 & 2 \\
   3 & -2
\end{bmatrix}
\begin{bmatrix}
    2 \\
    1
\end{bmatrix} = \begin{bmatrix}
    3*2+2*1 \\
   3*2+(-2*1)
\end{bmatrix}
= \begin{bmatrix}
    8 \\
   4
\end{bmatrix}
\end{equation}

Now we want to see if we can find a $\lambda$ such that

\begin{equation}
Av = \begin{bmatrix}
   8 \\
   4
\end{bmatrix}= \lambda \begin{bmatrix}
   2 \\
   1
\end{bmatrix}
\end{equation}

Turns out $\lambda =4$ is the eigenvalue for our proposed eigenvector!

### 2.2 But how can you find values of eigenmatrices?

An $n xn$ matrix has n eigenvectors and n eigenvalues! How to find the eigenvalues?

$ det(A- \lambda I)= 0$

\begin{equation}
det(A- \lambda I) = det\begin{bmatrix}
   3-\lambda & 2 \\
   3 & -2-\lambda
\end{bmatrix}
\end{equation}

This way we indeed find that 4 is an eigenvalue, and so is -3! You'll learn about the connection between eigenvalues and eigenmatrices in a second.

https://georgemdallas.wordpress.com/2013/10/30/principal-component-analysis-4-dummies-eigenvectors-eigenvalues-and-dimension-reduction/

https://www.youtube.com/watch?v=ue3yoeZvt8E

"toepassingen van de statistiek" sl 53-63 from the PCA chapter!

## 3.  PCA: some notation

### 3.1 The data matrix

Let's say we have P variables $X_1, X_2, \dots, X_p$ and $n$ observations $1,...,n$. Or data looks like this:

\begin{bmatrix}
    X_{11} & X_{12} & X_{13} & \dots  & X_{1p} \\
    X_{21} & X_{22} & X_{23} & \dots  & X_{2p} \\
    \vdots & \vdots & \vdots & \ddots & \vdots \\
    X_{n1} & X_{n2} & X_{n3} & \dots  & X_{np}
\end{bmatrix}

For 2 variables, this is what our data could look like:

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np

np.random.seed(123)
X = np.random.normal(2, 1.5, 50)
Y = np.random.normal(3, 0.6, 50)
fig, ax = plt.subplots()
ax.axhline(y=0, color='k')
ax.axvline(x=0, color='k')
plt.ylim(-5,5)
plt.xlim(-7,7)

plt.scatter(X, Y, s = 6)
```

### 3.2 The mean & mean-corrected data

The **mean** of the $j$-th variable:

$\bar{X}_j = \dfrac{\sum_{i=1}^n X_{ij}}{n}= \bar X_{.j}$

To get to the **mean-corrected** data: substract the mean from each $X_{ij}$

$x_{ij} = X_{ij}-\bar X_{.j}$

Going back to our two variables example, this is how the data would be shifted:

```python
fig, ax = plt.subplots()
ax.axhline(y=0, color='k')
ax.axvline(x=0, color='k')
X_mean = X- np.mean(X)
Y_mean = Y- np.mean(Y)
plt.ylim(-5,5)
plt.xlim(-7,7)
plt.scatter(X_mean, Y_mean, s = 6)
```

### 3.3 The variance & standardized data

$s^2_j = \dfrac{\sum_{i=1}^n (X_{ij}-\bar X_{.j})^2}{n-1}= \dfrac{\sum_{i=1}^n x_{ij}^2}{n-1}$

To get to the **standardized** data: divide the mean-corrected data by the standard deviation $s_j$.

$z_{ij} = \dfrac{x_{ij}}{s_{j}}$

Going back to the example with 2 variables, this is what standardized data would look like:

```python
fig, ax = plt.subplots()
ax.axhline(y=0, color='k')
ax.axvline(x=0, color='k')
X_mean = X- np.mean(X)
Y_mean = Y- np.mean(Y)
X_std= np.std(X)
Y_std = np.std(Y)
X_stdized = X_mean / X_std
Y_stdized = Y_mean / Y_std
plt.ylim(-5,5)
plt.xlim(-7,7)
plt.scatter(X_stdized, Y_stdized, s=6)
```

### 3.4 The covariance

The covariance for two variables $X_j$ and $X_k$:

$s_{jk} = \dfrac{\sum_{i=1}^n (X_{ij}-\bar X_{.j})(X_{ij}-\bar X_{.k})}{n-1}= \dfrac{\sum_{i=1}^n x_{ij}x_{ik}}{n-1}$

Denote $\mathbf{S}$ the sample covariance matrix

\begin{equation}
\mathbf{S} = \begin{bmatrix}
    s^2_{1} & s_{12}  & \dots  & s_{1p} \\
    s_{21} & s^2_{2}  & \dots  & s_{2p} \\
    \vdots & \vdots & \ddots & \vdots \\
    s_{p1} & s_{p2} & \dots  & s^2_{p}
\end{bmatrix}
\end{equation}

When you do the same computation with standardized variables, you get the **correlation**. Remember that the correlation $r_{jk}$ always lies between -1 and 1.

$r_{jk} = \dfrac{\sum_{i=1}^n z_{ij}z_{ik}}{n-1}$

Then, $\mathbf{R}$ is the correlation matrix

\begin{equation}
\mathbf{R} = \begin{bmatrix}
    1 & r_{12} & \dots  & r_{1p} \\
    r_{21} & 1 & \dots  & r_{2p} \\
    \vdots & \vdots  & \ddots & \vdots \\
    r_{p1} & r_{p2}  & \dots  & 1
\end{bmatrix}
\end{equation}

## 4. How does PCA work? Matrices and eigendecomposition

### 4.1 Finding principal components

$ \mathbf{X}= (X_1, X_2, \ldots, X_p)$ is a random variable.
Then the principal components of $\mathbf{X}$, denoted by $PC_1, \ldots, PC_p$ satisfy these 3 conditions:

-  $(PC_1, PC_2, \ldots, PC_p)$ are mutually uncorrelated
- $var(PC_1)\geq var(PC_2) \geq \ldots \geq var(PC_p)$
- $PC_j =  c_{j1} X_1+c_{j2} X_2+\ldots+c_{jp} X_p$

Note that for  $j=1,\ldots,p$ and $c_j = (c_{j1}, c_{j2}, \ldots, c_{jp})$' is a vector of constants satisfying $ \lVert{\mathbf{c_j} \rVert^2 = \mathbf{c'_j}\mathbf{c_j}} = \displaystyle\sum^p_{k=1} c^2_{kj}=1 $

The variance of $PC$ is then:
$var(PC_j) = var( c_{j1} X_1+c_{j2} X_2+\ldots+c_{jp} X_p) \\
  = c_{j1}^2 var(X_1) +c_{j2}^2 var(X_2) + \ldots + c_{jp}^2 var(X_p) + 2 \displaystyle\sum_k\sum_{l \neq k}c_{jk}c_{jl} cov(X_k,  X_l) \\  = c_j' \Sigma c_j$

In words, this means that variances can easily be computed using the coefficients used while making the linear combinations.

We can prove that  $var(PC_1)\geq var(PC_2) \geq \ldots \geq var(PC_p)$ is actually given by the eigenvalues $\lambda_1\geq \lambda_2 \geq \ldots \geq \lambda_3$ and the eigenvectors are given by  $c_j = (c_{j1}, c_{j2}, \ldots, c_{jp})$. From here on, we'll denote the eigenvectors by $e_j$ instead of $c_j$.

# Sources

http://www.bbk.ac.uk/ems/faculty/brooms/teaching/SMDA/SMDA-02.pdf

https://stackoverflow.com/questions/13224362/principal-component-analysis-pca-in-python

https://georgemdallas.wordpress.com/2013/10/30/principal-component-analysis-4-dummies-eigenvectors-eigenvalues-and-dimension-reduction/


# SymPyLab

**INTEGRANTES**

Ivonne Nathalia Uribe Lara

David Antonio Garzón Avendaño

Juan Carlos Sanchez Orjuela

SymPy’s documentation
- https://docs.sympy.org/latest/index.html

## SymPy’s polynomials
- https://docs.sympy.org/latest/modules/polys/basics.html#polynomials

- (x-1)(x-2)(x-3)(x-4)(x-5) = x^5 - 15 x^4  + 85 x^3 - 225 x^2 + 274 x - 120

- (x^5 - 15 x^4  + 85 x^3 - 225 x^2 + 274 x - 120) / (x-1) = x^4  - 14 x^3  + 71 x^2  - 154 x + 120

```
from sympy import Symbol
from sympy import div

x = Symbol('x')

p = x**5 - 15*x**4  + 85*x**3 - 225*x**2 + 274*x - 120

p, r = div(p,  x-1)

print(p)
print(r)

p, r = div(p,  x-2)

print(p)
print(r)

p, r = div(p,  x-3)

print(p)
print(r)

p, r = div(p,  x-4)

print(p)
print(r)

```

    x**4 - 14*x**3 + 71*x**2 - 154*x + 120
    0
    x**3 - 12*x**2 + 47*x - 60
    0
    x**2 - 9*x + 20
    0
    x - 5
    0

Edición

 Polinomio:

 (x-1)(x-2)(x-3)(x-4)(x-5)(x-6)(x-7)(x-8)(x-9)(x-10) = x^10 - 55x^9 = 1320x^8 - 18150x^7 + 157773x^6 - 902055x^5 + 3416930x^4 + 8409500x^3 - 10628640x + 3628800

(x-1)(x-2)(x-3)(x-4)(x-5)(x-6)(x-7)(x-8)(x-9)(x-10)/(x-1) = x^9 - 54x^8 + 1266x^7 - 16884x^6 + 140889x^5 - 761166x^4 + 2655764^x^3 - 5753736^x^2 + 6999840^x - 3628800

wolfram:

```
#edicion

from sympy import Symbol
from sympy import div

x = Symbol('x')

p = (x-1)*(x-2)*(x-3)*(x-4)*(x-5)*(x-6)*(x-7)*(x-8)*(x-9)*(x-10)

p, r = div(p,  x-1)

print(p)
print(r)

p, r = div(p,  x-2)

print(p)
print(r)

p, r = div(p,  x-3)

print(p)
print(r)

p, r = div(p,  x-4)

print(p)
print(r)

p, r = div(p,  x-5)

print(p)
print(r)

p, r = div(p,  x-6)

print(p)
print(r)

p, r = div(p,  x-7)

print(p)
print(r)

p, r = div(p,  x-8)

print(p)
print(r)

p, r = div(p,  x-9)

print(p)
print(r)

```

    x**9 - 54*x**8 + 1266*x**7 - 16884*x**6 + 140889*x**5 - 761166*x**4 + 2655764*x**3 - 5753736*x**2 + 6999840*x - 3628800
    0
    x**8 - 52*x**7 + 1162*x**6 - 14560*x**5 + 111769*x**4 - 537628*x**3 + 1580508*x**2 - 2592720*x + 1814400
    0
    x**7 - 49*x**6 + 1015*x**5 - 11515*x**4 + 77224*x**3 - 305956*x**2 + 662640*x - 604800
    0
    x**6 - 45*x**5 + 835*x**4 - 8175*x**3 + 44524*x**2 - 127860*x + 151200
    0
    x**5 - 40*x**4 + 635*x**3 - 5000*x**2 + 19524*x - 30240
    0
    x**4 - 34*x**3 + 431*x**2 - 2414*x + 5040
    0
    x**3 - 27*x**2 + 242*x - 720
    0
    x**2 - 19*x + 90
    0
    x - 10
    0

## SymPy’s polynomial simple univariate polynomial factorization
- https://docs.sympy.org/latest/modules/polys/wester.html#simple-univariate-polynomial-factorization
- factor(x\*\*5 - 15\*x\*\*4  + 85\*x\*\*3 - 225\*x\*\*2 + 274\*x - 120)

```
from sympy import *
x = Symbol('x')
factor(x**5 - 15*x**4  + 85*x**3 - 225*x**2 + 274*x - 120)
```

    (x - 5)*(x - 4)*(x - 3)*(x - 2)*(x - 1)

Edición

factor(x^4 - 46x^3 + 233x^2 + 5656x + 9996)

```
from sympy import *
x = Symbol('x')
factor(x**4 - 46*x**3  +233*x**2 + 5656*x + 9996)
```

    (x - 34)*(x - 21)*(x + 2)*(x + 7)

## SymPy’s solvers
- https://docs.sympy.org/latest/tutorial/solvers.html
- x\*\*5 - 15\*x\*\*4  + 85\*x\*\* 3 - 225\*x\*\* 2 + 274\*x - 120 = 0

```
from sympy import *
x = Symbol('x')
solveset(Eq(x**5 - 15*x**4  + 85*x**3 - 225*x**2 + 274*x - 120, 0), x)

```

    {1, 2, 3, 4, 5}

Edición

resolve(x^4 - e^x - pi = 0)

Wolfram:

```
#edicion

from sympy import *
import numpy as np
x = Symbol('x')
solveset(Eq(x**4 -32*x**3 - np.pi , 0), x)
```

    {-0.459132099076072, 32.0000958729375, 0.229518113069273 - 0.401431993025616*I, 0.229518113069273 + 0.401431993025616*I}

## SymPy’s Symbolic and Numercical Complex Evaluations
- https://docs.sympy.org/latest/modules/evalf.html](https://)
- x = x1 + I*x2,y = y1 + I*y2, z = z1 + I*z2, x*y*z

```
from sympy import *
x1, x2, y1, y2, z1, z2 = symbols("x1 x2 y1 y2 z1 z2", real=True)
x = x1 + I*x2
y = y1 + I*y2
z = z1 + I*z2

print(x*y*z)
print(expand(x*y*z))
print(expand((x*y)*z))
print(expand(x*(y*z)))

w=N(1/(pi + I), 20)
print('w=',w)
```

    (x1 + I*x2)*(y1 + I*y2)*(z1 + I*z2)
    x1*y1*z1 + I*x1*y1*z2 + I*x1*y2*z1 - x1*y2*z2 + I*x2*y1*z1 - x2*y1*z2 - x2*y2*z1 - I*x2*y2*z2
    x1*y1*z1 + I*x1*y1*z2 + I*x1*y2*z1 - x1*y2*z2 + I*x2*y1*z1 - x2*y1*z2 - x2*y2*z1 - I*x2*y2*z2
    x1*y1*z1 + I*x1*y1*z2 + I*x1*y2*z1 - x1*y2*z2 + I*x2*y1*z1 - x2*y1*z2 - x2*y2*z1 - I*x2*y2*z2
    w= 0.28902548222223624241 - 0.091999668350375232456*I

Edición

x = x1 + I*x2, y = y1 + I*y2, z = z1 + I*z2, a = a1 + I*a2, x*y*z/a

Wolfram

```
from sympy import *
x1, x2, y1, y2, z1, z2, a1, a2 = symbols("x1 x2 y1 y2 z1 z2 a1 a2", real=True)
x = x1 + I*x2
y = y1 + I*y2
z = z1 + I*z2
a = a1 + I*a2

print(x*y*(z/a))
print(expand(x*y*(z/a)))
print(expand((x*y)*(z/a)))
print(expand((z/a)*x*y))

wres =N(pi**2 - sin(2), 128)
print('Wres=',wres)

```

    (x1 + I*x2)*(y1 + I*y2)*(z1 + I*z2)/(a1 + I*a2)
    x1*y1*z1/(a1 + I*a2) + I*x1*y1*z2/(a1 + I*a2) + I*x1*y2*z1/(a1 + I*a2) - x1*y2*z2/(a1 + I*a2) + I*x2*y1*z1/(a1 + I*a2) - x2*y1*z2/(a1 + I*a2) - x2*y2*z1/(a1 + I*a2) - I*x2*y2*z2/(a1 + I*a2)
    x1*y1*z1/(a1 + I*a2) + I*x1*y1*z2/(a1 + I*a2) + I*x1*y2*z1/(a1 + I*a2) - x1*y2*z2/(a1 + I*a2) + I*x2*y1*z1/(a1 + I*a2) - x2*y1*z2/(a1 + I*a2) - x2*y2*z1/(a1 + I*a2) - I*x2*y2*z2/(a1 + I*a2)
    x1*y1*z1/(a1 + I*a2) + I*x1*y1*z2/(a1 + I*a2) + I*x1*y2*z1/(a1 + I*a2) - x1*y2*z2/(a1 + I*a2) + I*x2*y1*z1/(a1 + I*a2) - x2*y1*z2/(a1 + I*a2) - x2*y2*z1/(a1 + I*a2) - I*x2*y2*z2/(a1 + I*a2)
    Wres= 8.9603069742636769234384711339644062926114444357929003580343763646890775208784216983816467147935928515140580170316236578164913950

## SymPy’s integrals
- https://docs.sympy.org/latest/modules/integrals/integrals.html
- [risk-engineering.org](https://risk-engineering.org/notebook/monte-carlo-LHS.html)

Let’s start with a simple integration problem in 1D,

$$\int_1^5 x^2 dx$$

This is easy to solve analytically, and we can use the SymPy library in case you’ve forgotten how to resolve simple integrals.

```
import sympy
# we’ll save results using different methods in this data structure, called a dictionary
result = {}
x = sympy.Symbol("x")
i = sympy.integrate(x**2)
print(i)
result["analytical"] = float(i.subs(x, 5) - i.subs(x, 1))
print("Analytical result: {}".format(result["analytical"]))
```

    x**3/3
    Analytical result: 41.333333333333336

Edición

Inetgral:

$$\int_3^8 x^3 + 8 dx$$

```
#Edicion

import sympy
resultado = {}
x = sympy.Symbol("x")
i = sympy.integrate(x**3 + 8)
print(i)
resultado["ant"] = float(i.subs(x, 8) - i.subs(x, 3))
print("Resiltado: {}".format(resultado["ant"]))
```

    x**4/4 + 8*x
    Resiltado: 1043.75

**Integrating with Monte Carlo**
[risk-engineering.org](https://risk-engineering.org/notebook/monte-carlo-LHS.html)

We can estimate this integral using a standard Monte Carlo method, where we use the fact that the expectation of a random variable is related to its integral

$$\mathbb{E}(f(x)) = \int_I f(x) dx $$

We will sample a large number N of points in I and calculate their average, and multiply by the range over which we are integrating.

```
import numpy
N = 1_000_000
accum = 0
for i in range(N):
    x = numpy.random.uniform(1, 5)
    accum += x**2
volume = 5 - 1
result["MC"] = volume * accum / float(N)
print("Standard Monte Carlo result: {}".format(result["MC"]))
```

    Standard Monte Carlo result: 41.32998559988266

Edición

Estimamos la siguiente integral

$$\int_3^8 x^3 + 8 dx$$

por Monte Carlo

```
import numpy
N = 10_000_000
temp = 0
for i in range(N):
    x = numpy.random.uniform(3, 8)
    temp += x**3 + 8
volume = 8 - 3
resultado["MontCar"] = volume * temp / float(N)
print("Resultado por Monte Carlo: {}".format(resultado["MontCar"]))
```

    Resultado por Monte Carlo: 1043.9422026850486

- integrate(x\*\*2 * sin(x)\*\*3)

```
import sympy
x = Symbol("x")
i = integrate(x**2 * sin(x)**3)
print(i)
print(float(i.subs(x, 5) - i.subs(x, 1)))
```

    -x**2*sin(x)**2*cos(x) - 2*x**2*cos(x)**3/3 + 14*x*sin(x)**3/9 + 4*x*sin(x)*cos(x)**2/3 + 14*sin(x)**2*cos(x)/9 + 40*cos(x)**3/27
    -15.42978215330555

Edición

Inetgramos : sin(x) + (cos(x)^2)

Wolfram:

```
#Edicion
x = Symbol("x")
i = integrate(sin(x) + (cos(x)**2))
print(i)
print(float(i.subs(x, 8) - i.subs(x, 3)))

```

    x/2 + sin(x)*cos(x)/2 - cos(x)
    1.6533855825916333

```
import numpy
N = 100_000
accum = 0
l =[]
for i in range(N):
    x = numpy.random.uniform(1, 5)
    accum += x**2 * sin(x)**3
volume = 5 - 1
result["MC"] = volume * accum / float(N)
print("Standard Monte Carlo result: {}".format(result["MC"]))
```

    Standard Monte Carlo result: -15.3350781160391

```
#Edicion

import numpy
N = 1_000_000
temp = 0
for i in range(N):
    x = numpy.random.uniform(3, 8)
    temp += numpy.sin(x) + (numpy.cos(x))**2
volume = 8 - 3
resultado["MontCar2"] = volume * temp / float(N)
print("Resultado por Monte Carlo: {}".format(resultado["MontCar2"]))
```

    Resultado por Monte Carlo: 1.644611681543184

**A higher dimensional integral** [risk-engineering.org](https://risk-engineering.org/notebook/monte-carlo-LHS.html)

Let us now analyze an integration problem in dimension 4, the Ishigami function. This is a well-known function in numerical optimization and stochastic analysis, because it is very highly non-linear.

```
import sympy

x1 = sympy.Symbol("x1")
x2 = sympy.Symbol("x2")
x3 = sympy.Symbol("x3")
expr = sympy.sin(x1) + 7*sympy.sin(x2)**2 + 0.1 * x3**4 * sympy.sin(x1)
res = sympy.integrate(expr,
                      (x1, -sympy.pi, sympy.pi),
                      (x2, -sympy.pi, sympy.pi),
                      (x3, -sympy.pi, sympy.pi))
# Note: we use float(res) to convert res from symbolic form to floating point form
result = {}
result["analytical"] = float(res)
print("Analytical result: {}".format(result["analytical"]))
```

    Analytical result: 868.175747048395

```
#Edicion

import sympy

x = sympy.Symbol("x")
y = sympy.Symbol("y")
z = sympy.Symbol("z")
expr = sympy.sin(x) + 8*sympy.sin(y)**2 + 0.34 * z**4 * sympy.sin(x)
res = sympy.integrate(expr,
                      (x, -sympy.pi, sympy.pi),
                      (y, -sympy.pi, sympy.pi),
                      (z, -sympy.pi, sympy.pi))

resultados = {}
resultados["analytical"] = float(res)
print("Analytical result: {}".format(resultados["analytical"]))

```

    Analytical result: 992.2008537695942

```
N = 10_000
accum = 0
for i in range(N):
    xx1 = numpy.random.uniform(-numpy.pi, numpy.pi)
    xx2 = numpy.random.uniform(-numpy.pi, numpy.pi)
    xx3 = numpy.random.uniform(-numpy.pi, numpy.pi)
    accum += numpy.sin(xx1) + 7*numpy.sin(xx2)**2 + 0.1 * xx3**4 * numpy.sin(xx1)
volume = (2 * numpy.pi)**3
result = {}
result["MC"] = volume * accum / float(N)
print("Standard Monte Carlo result: {}".format(result["MC"]))
```

    Standard Monte Carlo result: 860.6796782529639

```
#Edicion

import numpy
N = 100_000
temp = 0
for i in range(N):
  xx = numpy.random.uniform(-numpy.pi, numpy.pi)
  yy = numpy.random.uniform(-numpy.pi, numpy.pi)
  zz = numpy.random.uniform(-numpy.pi, numpy.pi)
  temp += np.sin(xx) + 8*np.sin(yy)**2 + 0.34 * zz**4 * np.sin(xx)

volumen = (2 * numpy.pi)**3
resultados["MonteCarlo"] = volumen * temp / float(N)
print("Resultado Usando Montecarlo: {}".format(resultados["MonteCarlo"]))

```

    Resultado Usando Montecarlo: 983.0796811421585

```
import math
import numpy
# adapted from https://mail.scipy.org/pipermail/scipy-user/2013-June/034744.html
def halton(dim: int, nbpts: int):
    h = numpy.full(nbpts * dim, numpy.nan)
    p = numpy.full(nbpts, numpy.nan)
    P = [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31]
    lognbpts = math.log(nbpts + 1)
    for i in range(dim):
        b = P[i]
        n = int(math.ceil(lognbpts / math.log(b)))
        for t in range(n):
            p[t] = pow(b, -(t + 1))

        for j in range(nbpts):
            d = j + 1
            sum_ = math.fmod(d, b) * p[0]
            for t in range(1, n):
                d = math.floor(d / b)
                sum_ += math.fmod(d, b) * p[t]

            h[j*dim + i] = sum_
    return h.reshape(nbpts, dim)
```

```
import matplotlib.pyplot as plt
N = 1000
seq = halton(2, N)
plt.title("2D Halton sequence")
# Note: we use "alpha=0.5" in the scatterplot so that the plotted points are semi-transparent
# (alpha-transparency of 0.5 out of 1), so that we can see when any points are superimposed.
plt.axes().set_aspect('equal')
plt.scatter(seq[:,0], seq[:,1], marker=".", alpha=0.5);
```

```
#Edicion
import numpy
import math

def halton2(dimencion: int, nbpts: int):
    h = numpy.full(nbpts * dimencion, numpy.nan)
    p = numpy.full(nbpts, numpy.nan)
    P = [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47]
    lognbpts = math.log(nbpts + 1)
    for i in range(dimencion):
        b = P[i]
        n = int(math.ceil(lognbpts / math.log(b)))
        for t in range(n):
            p[t] = b**(-(t + 1))

        for j in range(nbpts):
            d = j + 1
            sum_ = math.fmod(d, b) * p[0]
            for t in range(1, n):
                d = math.floor(d / b)
                sum_ += math.fmod(d, b) * p[t]

            h[j*dimencion + i] = sum_
    return h.reshape(nbpts, dimencion)

```

```
#Edicion

import matplotlib.pyplot as plt
N = 1000
seq = halton2(2, N)
plt.title("Secuencia de Halton")
plt.axes().set_aspect('equal')
plt.scatter(seq[:,0], seq[:,1], marker=".", alpha=0.5, color="green");
```

```
N = 10_000

seq = halton(3, N)
accum = 0
for i in range(N):
    xx1 = -numpy.pi + seq[i][0] * numpy.pi * 2
    xx2 = -numpy.pi + seq[i][1] * numpy.pi * 2
    xx3 = -numpy.pi + seq[i][2] * numpy.pi * 2
    accum += numpy.sin(xx1) + 7*numpy.sin(xx2)**2 + 0.1 * xx3**4 * numpy.sin(xx1)
volume = (2 * numpy.pi)**3
result = {}
result["MC"] = volume * accum / float(N)
print("Qausi Monte Carlo Halton Sequence result: {}".format(result["MC"]))
```

    Qausi Monte Carlo Halton Sequence result: 868.238928030592

```
#Edicion

import numpy as np
N = 100_000
temp = 0
seq2 = halton2(3,N)
for i in range(N):
  xx = -numpy.pi + seq2[i][0] * numpy.pi * 2
  yy = -numpy.pi + seq2[i][1] * numpy.pi * 2
  zz = -numpy.pi + seq2[i][2] * numpy.pi * 2
  temp += np.sin(xx) + 8*np.sin(yy)**2 + 0.34 * zz**4 * np.sin(xx)

resultados ={}
volumen = (2 * numpy.pi)**3
resultados["MonteCarlo"] = volumen * temp / float(N)
print("Resultado Usando Montecarlo: {}".format(resultados["MonteCarlo"]))
```

    Resultado Usando Montecarlo: 992.1898731825128

Como punto adicional, vamos a hacer algunas integrales dobles y triples (no se anexan fotos de Wolfram debido a tiene problemas para resolver este tipo de integrales)

- $$\int_2^5 \int_3^6 x^2 + sin^2(y) dx dy$$

- $$\int_3^5 \int_6^7 ln(x) + cos(y) dx dy$$

- $$\int_1^5 \int_1^2 \int_2^3 e^x + ln(y) - cos(z) dx dy dz $$

```
#Edicion

import sympy

x = sympy.Symbol("x")
y = sympy.Symbol("y")
expr = x**2 + sin(y)**2
res = sympy.integrate(expr,
                      (x, 2, 5),
                      (y, 3, 6))
result = {}
result["analytical"] = float(res)
print("Analytical result: {}".format(result["analytical"]))
```

    Analytical result: 121.69286806485113

```
#Edicion

import sympy

x = sympy.Symbol("x")
y = sympy.Symbol("y")
expr = ln(x) + cos(x)
res = sympy.integrate(expr,
                      (x, 3, 5),
                      (y, 6, 7))
result = {}
result["analytical"] = float(res)
print("Analytical result: {}".format(result["analytical"]))
```

    Analytical result: 1.651308413443167

```
#Edicion
import math
x = sympy.Symbol("x")
y = sympy.Symbol("y")
z = sympy.Symbol("z")
expr = math.e**x + ln(y) - cos(z)
res = sympy.integrate(expr,
                      (x, 1, 5),
                      (y, 1, 2),
                      (z, 2, 3))

result = {}
result["analytical"] = float(res)
print("Analytical result: {}".format(result["analytical"]))
```

    Analytical result: 150.31276439366033

## Wolfram alpha answers quastion in natural languaje
- What is the average temperature in Bogota in September?

Edición


# Discrete Fourier Transform

## Always Keep In Mind:
- Why does the algorithm converge/not converge?
- Could the algorithm be more efficient? Why/why not?
- Are there alternative algorithms? Do they converge better?
- Try to explain your observations either mathematically or in a rigorous computational fashion.
- What do you observe? What is your personal interpretation of the results?
***

### DFT Definition

The Discrete Fourier Transform (DFT) is often used for numerical computation in digital signal processing. There exists a multitude of applications, which be optimised by using the various properties of the DFT, namely:

- Periodicity
- Complex Conjugate
- Multiplication
- Parseval's Theorem
- Symmetry

These will be explored and applied to optimise the DFT throughout this report. In real world applications, signals are sampled using sensors to measure various signals, this sampling rate is extremely important and will be explored further in this report. These discretized value are then used to shift from a time-domain to a frequency-domain using the DFT represented below.

$X_k= \sum_{n=0}^{N-1} x_n . e^{{-i 2 \pi k n}/{N}}$

The signal can be sent back from the frequency domain to the time domain using the Inverse Direct Fourier Transform represented below.

$x_n= \frac{1}{N} \sum_{k=0}^{N-1} X_k . e^{{i 2 \pi n  k}/{N}}$

***

### Question 1
Implement the discrete Fourier transform of a set of pairs outlined below. Test on easy cases and plot the corresponding interpolating trigonometric polynomial.

 $(x_j,y_j = f(x_j)), $        $ x_j = \frac{2 j \pi}{n+1} , $       $ 0 \leq j \leq n $

### !!!!!!!!STILL NEED TO DO THE INTERPOLATING POLY

```python
import numpy as np

#Evaluation of the Discrete Fourier Transform
def DFT(x):
    #Variables Initiated
    N = len(x)
    y_s = []
    y_a = []
    y_b = []

    for k in range(N):  # For each output element
        s = complex(0)
        for n in range(N):  # For each input element
            angle = 2j * np.pi * n * k / N
            s += x[n] * np.exp(-angle)
        y_s.append(s)
        #y_a.append(round(s.real,2))
        #y_b.append(round(s.imag,2))

    #return [y_s, y_a, y_b];
    return y_s;
```

```python
np.allclose(DFT(x), np.fft.fft(x))
```

    True

```python
import numpy as np
import matplotlib.pyplot as plt

n = 10

#Discretization of the input signal
x_j = []
for j in range(0,n):
    x_j.append((2*np.pi*j)/(n+1))

#Input signal generation
x = []
for j in range(0,n):
    x.append(0)
x[4] = 1
x[n-5] = 1

#DFT function called
X = DFT(x)
#X = np.fft.fft(x)
#np.allclose(DFT(x), np.fft.fft(x))

#Figure Size set
fig, axs = plt.subplots(3)
fig.set_size_inches(12,15)
#fig.suptitle('Sharing both axes')

#Graphs Plotted Signal[1], Real Coefficients[2] & Imaginary Coefficients[2]
axs[0].title.set_text('Signal (Time Domain)')
axs[0].stem(x_j,x, 'b-', use_line_collection=True)
axs[0].set_xlabel('x[n] [0;2pi]')
axs[0].set_ylabel('y[n]')
axs[1].title.set_text('Real Coefficients (Frequency Domain)')
axs[1].stem(x_j,np.real(X), 'b-', use_line_collection=True);
axs[1].set_xlabel('xj [0;2pi]')
axs[1].set_ylabel('Real [ck]')
axs[2].title.set_text('Imaginary Coefficients (Frequency Domain)')
axs[2].stem(x_j,np.imag(X), 'b-', use_line_collection=True);
axs[2].set_xlabel('xj [0;2pi]')
axs[2].set_ylabel('Imag [ck]')

#plt.subplot_tool()

plt.show()
```

    <IPython.core.display.Javascript object>

# This part is to help visualize the data, can be ignored

```python
import pandas as pd
df = pd.DataFrame ({'x_j': x_j})
df['DFT(x)']= X
df['fft']=np.fft.fft(x)
df
```

      x_j
      DFT(x)
      fft

      0
      0.000000
      2.000000e+00+0.000000e+00j
      2.000000e+00+0.000000e+00j

      1
      0.571199
      -1.809017e+00-5.877853e-01j
      -1.809017e+00-5.877853e-01j

      2
      1.142397
      1.309017e+00+9.510565e-01j
      1.309017e+00+9.510565e-01j

      3
      1.713596
      -6.909830e-01-9.510565e-01j
      -6.909830e-01-9.510565e-01j

      4
      2.284795
      1.909830e-01+5.877853e-01j
      1.909830e-01+5.877853e-01j

      5
      2.855993
      0.000000e+00-1.224647e-16j
      -1.110223e-16+5.551115e-17j

      6
      3.427192
      1.909830e-01-5.877853e-01j
      1.909830e-01-5.877853e-01j

      7
      3.998391
      -6.909830e-01+9.510565e-01j
      -6.909830e-01+9.510565e-01j

      8
      4.569589
      1.309017e+00-9.510565e-01j
      1.309017e+00-9.510565e-01j

      9
      5.140788
      -1.809017e+00+5.877853e-01j
      -1.809017e+00+5.877853e-01j

```python
np.allclose(DFT(x), np.fft.fft(x))
```

    True

# Interpolating Polynomial

### $$P_M(x)= \sum_{k=-M}^{M} c_k*e^{ikx_j}  $$

### $$ c_k= \frac{1}{n+1} \sum_{j=0}^{n} f(x_j)e^{-ikx_j} $$

Here's the approach I was thinking about. I haven't tried it yet,I'm not so sure if it'd work.

1. Find the matrix of $ c_k $ values: (in other words) if $Ax_j = DFT(x)$  then $x_j = A^{-1} DFT(x)$ , can we find A? since we have $x_j$ and $DFT(x)$ ?

2. Compute polynomials using $c_k$ and formula 1?

```python

```

```python
%timeit DFT(x)
%timeit np.fft.fft(x)
```

    214 µs ± 5.83 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
    8.89 µs ± 291 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

## !!!!!!!!!!!ENTER Q1 CONCLUSION HERE
DFT with for loop (Tom version) very slow - fft 25 times faster. **Apllied to binary pairs**

STUDY:
1. Speed by experimenting and seeing how computation time increases with n
2. Deduce Complexity of it
3. Do it for a **binary input** signal

numpy fft is super fast as its written in fortran, a low-level algorithm compared to python, on top of this we're usinf a lot of python-stack recursion and allocation of temporary arrays.

To make faster, one should vectorize repeated computations to remove the recursive functions that are currently used.

***

### Question 2
Apply it to the function $f:[0, 2\pi] \to \mathbb{R}$ be defined by

$$f(0) = f(2\pi) = 1$$

$f(x) = 0$  otherwise

Let $ M \in \mathbb{N}, n = 2M,$ and

$x_j = \frac{2 j \pi}{n+1}$   $j=0,...,n$

What are your conclusions?

```python
import numpy as np
import matplotlib.pyplot as plt

n = 10

x_j = []
for j in range(0,n):
    x_j.append((2*np.pi*j)/(n+1))

x = []
for j in range(0,n):
    x.append(0)
x[0] = 1
x[n-1] = 1

X = DFT(x)
#X = np.fft.fft(x)
#np.allclose(DFT(x), np.fft.fft(x))

#Figure Size set
fig, axs = plt.subplots(3)
fig.set_size_inches(12,15)
#fig.suptitle('Sharing both axes')

#Graphs Plotted Signal[1], Real Coefficients[2] & Imaginary Coefficients[2]
axs[0].title.set_text('Signal (Time Domain)')
axs[0].stem(x_j,x, 'b-', use_line_collection=True)
axs[0].set_xlabel('x[n] [0;2pi]')
axs[0].set_ylabel('y[n]')
axs[1].title.set_text('Real Coefficients (Frequency Domain)')
axs[1].stem(x_j,np.real(X), 'b-', use_line_collection=True);
axs[1].set_xlabel('xj [0;2pi]')
axs[1].set_ylabel('Real [ck]')
axs[2].title.set_text('Imaginary Coefficients (Frequency Domain)')
axs[2].stem(x_j,np.imag(X), 'b-', use_line_collection=True);
axs[2].set_xlabel('xj [0;2pi]')
axs[2].set_ylabel('Imag [ck]')

#plt.subplot_tool()

plt.show()
```

    <IPython.core.display.Javascript object>

```python
np.allclose(DFT(x), np.fft.fft(x))
```

    True

```python
%timeit DFT(x)
%timeit np.fft.fft(x)
```

    220 µs ± 6.36 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
    8.45 µs ± 139 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

## !!!!!!!!!!!ENTER Q2 CONCLUSION HERE
DFT with for loop (Tom version) very slow - fft 25 times faster. **Apllied to binary pairs**
Computation time increases with more binary pairs - as expected

***

### Question 3
Apply it to the function $f:[0, 2\pi] \to \mathbb{R}$ be defined by $f(x) = \sin(2x).$   $For M \in \mathbb{N}, n=2M, and$

$x_j = \frac{2 j \pi}{n+1}$   $j=0,...,n$

Discuss Shannon’s theorem. Illustrate the aliasing phenomenon for $ f(x)  = \sin(x) + \sin(5x)$ with 9 nodes. How many nodes do we need for an exact representation of $ f(x)  = \cos(x) + \cos(5x)$

#### Shannon Theorem

Claude Shannon furthered research in 1948 carried out by Harry Nyquist. He conclusively proved that there exists a limit to the amount of noise that can be present in a signal for it to be succesfully transmitted across a communications channel experiencing random data transmission errors (Nyquist-Shannon limit).

Together Nyquist & Shannon provided us with the notions to sample a continuous sample, transmit a discrete signal across a communications channel and reconstruct it successfully upon reception. This sampling rate can be described as:

 $Sampling Frequency$        $(f_s) > 2(B)$       $Highest Frequency$

A continuous-time signal of finite bandwidth can then sampled without aliasing issues as shown in the diagrams below.

Aliased signal (Reconstruction Impossible)<h/3>

Correctly Sampled Signal (Reconstruction Possible)<h/3>

```python
import numpy as np
import matplotlib.pyplot as plt

n = 20

x_j = []
for j in range(0,n):
    x_j.append((2*np.pi*j)/(n+1))

x = []
for j in range(0,n):
    x.append(np.sin(2000000*x_j[j]))

X = DFT(x)
#X = np.fft.fft(x)
#np.allclose(DFT(x), np.fft.fft(x))

#Figure Size set
fig, axs = plt.subplots(3)
fig.set_size_inches(12,15)
#fig.suptitle('Sharing both axes')

#Graphs Plotted Signal[1], Real Coefficients[2] & Imaginary Coefficients[2]
axs[0].title.set_text('Signal (Time Domain)')
axs[0].stem(x_j,x, 'b-', use_line_collection=True)
axs[0].set_xlabel('x[n] [0;2pi]')
axs[0].set_ylabel('y[n]')
axs[1].title.set_text('Real Coefficients (Frequency Domain)')
axs[1].stem(x_j,np.real(X), 'b-', use_line_collection=True);
axs[1].set_xlabel('xj [0;2pi]')
axs[1].set_ylabel('Real [ck]')
axs[2].title.set_text('Imaginary Coefficients (Frequency Domain)')
axs[2].stem(x_j,np.imag(X), 'b-', use_line_collection=True);
axs[2].set_xlabel('xj [0;2pi]')
axs[2].set_ylabel('Imag [ck]')

#plt.subplot_tool()

plt.show()
```

    <IPython.core.display.Javascript object>

## !!!!!!!!!!!ENTER Q3 (sinx ) CONCLUSION HERE
STUDY:
1. Speed by experimenting and seeing how computation time increases with n
2. Deduce Complexity of it
3. Do it for a **periodic** signal
***

```python
import numpy as np
import matplotlib.pyplot as plt

n = 9

x_j = []
for j in range(0,n):
    x_j.append((2*np.pi*j)/(n+1))

x = []
for j in range(0,n):
    x.append(np.sin(x_j[j])+np.sin(5*x_j[j]))
    #x.append(np.sin(50*x_j[j]))
X = DFT(x)
#X = np.fft.fft(x)
#np.allclose(DFT(x), np.fft.fft(x))

#Figure Size set
fig, axs = plt.subplots(3)
fig.set_size_inches(12,15)
#fig.suptitle('Sharing both axes')

#Graphs Plotted Signal[1], Real Coefficients[2] & Imaginary Coefficients[2]
axs[0].title.set_text('Signal (Time Domain)')
axs[0].stem(x_j,x, 'b-', use_line_collection=True)
axs[0].set_xlabel('x[n] [0;2pi]')
axs[0].set_ylabel('y[n]')
axs[1].title.set_text('Real Coefficients (Frequency Domain)')
axs[1].stem(x_j,np.real(X), 'b-', use_line_collection=True);
axs[1].set_xlabel('xj [0;2pi]')
axs[1].set_ylabel('Real [ck]')
axs[2].title.set_text('Imaginary Coefficients (Frequency Domain)')
axs[2].stem(x_j,np.imag(X), 'b-', use_line_collection=True);
axs[2].set_xlabel('xj [0;2pi]')
axs[2].set_ylabel('Imag [ck]')

#plt.subplot_tool()

plt.show()

```

    <IPython.core.display.Javascript object>

## !!!!!!!!!!!ENTER Q3 (cumulative (+) sinx)  CONCLUSION HERE
STUDY:
1. Shown aliasing
***

```python
import numpy as np
import matplotlib.pyplot as plt

n = 64

x_j = []
for j in range(0,n):
    x_j.append((2*np.pi*j)/(n+1))

x = []
for j in range(0,n):
    x.append(np.cos(x_j[j])*np.cos(5*x_j[j]))

X = DFT(x)
#X = np.fft.fft(x)
#np.allclose(DFT(x), np.fft.fft(x))

#Figure Size set
fig, axs = plt.subplots(3)
fig.set_size_inches(12,15)
#fig.suptitle('Sharing both axes')

#Graphs Plotted Signal[1], Real Coefficients[2] & Imaginary Coefficients[2]
axs[0].title.set_text('Signal (Time Domain)')
axs[0].stem(x_j,x, 'b-', use_line_collection=True)
axs[0].set_xlabel('x[n] [0;2pi]')
axs[0].set_ylabel('y[n]')
axs[1].title.set_text('Real Coefficients (Frequency Domain)')
axs[1].stem(x_j,np.real(X), 'b-', use_line_collection=True);
axs[1].set_xlabel('xj [0;2pi]')
axs[1].set_ylabel('Real [ck]')
axs[2].title.set_text('Imaginary Coefficients (Frequency Domain)')
axs[2].stem(x_j,np.imag(X), 'b-', use_line_collection=True);
axs[2].set_xlabel('xj [0;2pi]')
axs[2].set_ylabel('Imag [ck]')

#plt.subplot_tool()

plt.show()

```

    <IPython.core.display.Javascript object>

## !!!!!!!!!!!ENTER Q3 (cosx multiplication) CONCLUSION HERE
STUDY:
1. Identify Nyquist limit of signal
2. Should abide by multiplication property, show it in graph. A multiplication in time represents a circular convoution in frequency domain

$$x(n)h(n) \leftrightarrow \frac{1}{N}X(k) \ast H(k) $$
***

#### Cooley-Tukey

Friedrich Gauss developed the numerical algorithm in 1805, it was then rediscovered and applied to digital signals by J.W. Cooley & John Tukey in 1965. The algorithm capitalizes on the symmetry present in the DFT equation to reduce the computational costs of evaluating it.

The first step involves, splitting the DFT calculation into even and odd terms as shown below:

\begin{align}
X_k &= \sum_{n=0}^{N-1} x_n \cdot e^{-i~2\pi~k~n~/~N} \\
    &= \sum_{m=0}^{N/2 - 1} x_{2m} \cdot e^{-i~2\pi~k~(2m)~/~N} + \sum_{m=0}^{N/2 - 1} x_{2m + 1} \cdot e^{-i~2\pi~k~(2m + 1)~/~N} \\
    &= \sum_{m=0}^{N/2 - 1} x_{2m} \cdot e^{-i~2\pi~k~m~/~(N/2)} + e^{-i~2\pi~k~/~N} \sum_{m=0}^{N/2 - 1} x_{2m + 1} \cdot e^{-i~2\pi~k~m~/~(N/2)}
\end{align}

The second step involves using the fact that $M = \frac{N}{2}$ to only compute half of the sums. This reduces our computation from order $N^2$ to order $M^2$.

Finally, we repeat the steps above as long as M is even within the subsystems, this effectively halves the computational costs until the arrays reduce in size so much that it is no longer beneficial. The final computational speed is of order $Nlog(N)$, the same as a Fast Fourier Transform (FFT).

***

### Question 4
Improve the implementation using a simple form of Cooley-Tukey’s algorithm (Ref 1). Hint: assume that $n + 1$ is a power of 2, in particular $n$ is odd ($n = 2M - 1$), see (Ref 19), and using periodicity compute $ck$ for indices$ k = 0; ...; n$. Discuss the computer effort in comparison with the basic implementation.

```python
def Cooley_Tukey(x):
    #Cooley-Tukey FFT (Recursive)
    x = np.asarray(x, dtype=float)
    N = x.shape[0]

    if N % 2 > 0:
        raise ValueError("size of x must be a power of 2")
    elif N <= 32:  # this cutoff should be optimized dependent on the size of x
        return DFT(x)
    else:
        X_even = Cooley_Tukey(x[::2])
        X_odd = Cooley_Tukey(x[1::2])
        factor = np.exp(-2j * np.pi * np.arange(N) / N)
        return np.concatenate([X_even + factor[:N // 2] * X_odd,
                               X_even + factor[N // 2:] * X_odd])

```

```python
import numpy as np
import matplotlib.pyplot as plt

n = 1024

x_j = []
for j in range(0,n):
    x_j.append((2*np.pi*j)/(n+1))

x = []
for j in range(0,n):
    x.append(np.cos(x_j[j])*np.cos(5*x_j[j]))

#np.allclose(Cooley_Tukey(x), np.fft.fft(x))
%timeit DFT(x)
%timeit Cooley_Tukey(x)
%timeit np.fft.fft(x)
```

    5.91 s ± 333 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
    171 ms ± 8.02 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
    117 µs ± 793 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

## !!!!!!!!!!!ENTER Q4 CONCLUSION HERE
STUDY:
1. Optimise the cutoff for 3 different sizes of x
2.

***

### Question 5
Illustrate the effect of truncating the discrete Fourier transform (i.e. omitting high frequencies). Discuss some applications in signal compression.

## !!!!!!!!!!!!!CHANGE BELOW!!!!!!!!!!
The high frequencies contribute to the fast varying parts of the signal(the sharp transitions), while the low frequencies contribute to the slow variations of the signal in the time domain.

```python
The high frequencies contribute to the fast varying parts of the signal(the sharp transitions), while the low frequencies contribute to the slow variations of the signal in the time domain.
```

### Question 6
Implement the discrete Fourier transform in two dimensions Ref 1), first in a basic way then optionally in an improved way inspired from question 4.

```python

```

### Question 7
Show the effect of truncating the DFT on greyscale images. Give illustrations in the context of image denoising / compression.

```python

```

### Question 8
Possible continuation: FFT for solving linear PDEs with applications in image denoising. To be discussed with us.

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```


# 3D Grid on GPU with Kernel Tuner

In this tutorial we are going to see how to map a series of Gaussian functions, each located at a different point on a 3D a grid. We are going to optimize the GPU code and compare its performance with the CPU implementation.

<div class="alert alert-info">

**Note:** If you are reading this tutorial on the Kernel Tuner's documentation pages, note that you can actually run this tutorial as a Jupyter Notebook. Just clone the Kernel Tuner's [GitHub repository](http://github.com/benvanwerkhoven/kernel_tuner). Install the Kernel Tuner and Jupyter Notebooks and you're ready to go! You can start the tutorial by typing "jupyter notebook" in the "kernel_tuner/tutorial" directory.

</div>

## Let's start on the CPU

Before delving into the GPU implementation, let's start with a simple CPU implementation of the problem. The problem at hand is to compute the values of the following function

\begin{equation} \nonumber
f = \sum_{i=1}^{N}\exp\left(-\beta \sqrt{(x-x_i)^2+(y-y_i)^2+(z-z_i)^2}\right)
\end{equation}

on a 3d grid. The $x$, $y$ and $z$ vectors contain the coordinate of the points in the Cartesian space. We can define a simple Python function that computes the value of the function $f$ for one given Gaussian. Don't forget to execute all the code cells, like the one below, as you read through this notebook by selecting the cell and pressing *shift+enter*.

```python
import numpy as np
import numpy.linalg as la
from time import time

def compute_grid(center,xgrid,ygrid,zgrid):
    x0,y0,z0 = center
    beta = -0.1
    f = np.sqrt( (xgrid-x0)**2 + (ygrid-y0)**2 + (zgrid-z0)**2 )
    f = np.exp(beta*f)
    return f
```

For a given center, this function returns the values of the corresponding Gaussian function mapped on the 3D grid. The grid points are here defined by the variables `xgrid`, `ygrid` and `zgrid`. These variables are themselves 3D grids obtained, as we will see in an instant, using the `numpy.meshgrid` function.

To use this function we simply have to create the grid, defined by the vectors x, y, and z. Since we want to later on send these vectors to the GPU we define them as 32-bit floats. For simplicity, we here select the interval $[-1:1]$ to define our grid. We use $n=256$ grid points in order to have a sufficiently large problem without requiring too long calculations. We then create meshgrids to be passed to the function above. We define here 100 gaussian centers that are randomly distributed within the 3D space.

```python
# dimension of the problem
n = 256

# define the vectors
x = np.linspace(-1,1,n).astype(np.float32)
y = np.linspace(-1,1,n).astype(np.float32)
z = np.linspace(-1,1,n).astype(np.float32)

# create meshgrids
xgrid,ygrid,zgrid = np.meshgrid(x,y,z)
cpu_grid = np.zeros_like(xgrid)

# centers
npts = 100
center = (-1 + 2*np.random.rand(npts,3)).astype(np.float32)

# compute the grid and time the operation
t0 = time()
for xyz in center:
    cpu_grid += compute_grid(xyz,xgrid,ygrid,zgrid)
print('CPU Execution time %f ms' %( (time()-t0)*1000) )
```

    CPU Execution time 52320.160627 ms

Depending on your hardware it might take a few seconds for the calculations above to finish.

## Let's move to the GPU

Let's see now how that will look like on the GPU. We first write a kernel that does the same calculation as the above function. As you can see see below, the variables `block_size_x`, `block_size_y` and `block_size_z` are not yet defined here. These variables are used to set the number of threads per thread block on the GPU and are the main parameters that we will optimize in this tutorial. During tuning, Kernel Tuner will automatically insert `#define` statements for these parameters at the top of the kernel code. So for now we don't have to specify their values.

The dimensions of the problem `nx`, `ny`, and `nz`, are the number of grid points in the x, y, and z dimensions. We can again use Kernel Tuner to insert these parameters into the code.

```python
# define a kernel template
# several parameters are available
# block sizes : bx, by, bz
# dimensions  : nx, ny, nz
kernel_code = """
#include <math.h>

// a simple gaussian function
__host__ __device__ float f(float d){
    float b = 0.1;
    float x = exp(-b*d);
    return x;
}

// the main function called below
__global__ void AddGrid(float x0, float y0, float z0, float *xvect, float *yvect, float *zvect, float *out)
{

    // 3D thread
    int x = threadIdx.x + block_size_x * blockIdx.x;
    int y = threadIdx.y + block_size_y * blockIdx.y;
    int z = threadIdx.z + block_size_z * blockIdx.z;

    if ( ( x < nx ) && (y < ny) && (z < nz) )
    {

        float dx = xvect[x]-x0;
        float dy = yvect[y]-y0;
        float dz = zvect[z]-z0;
        float d = sqrt(dx*dx + dy*dy + dz*dz);
        out[y * nx * nz + x * nz + z] = f(d);
    }
}
"""
```

### Tune the kernel

We can now use the tuner to optimize the thread block dimensions on our GPU. To do so we define the tunable parameters of our kernel using the `tune_params` dictionary, which assigns to each block size the values we want the tuner to explore. We also use the tunable parameters to insert the domain dimensions `nx`, `ny`, and `nz`.

We also define a list containing the arguments of the CUDA function (AddGrid) above. Since we only want to optimize the performance of the kernel we only consider here one center in the middle of the grid. Note that Kernel Tuner needs either `numpy.ndarray` or `numpy.scalar` as arguments of the kernel. Hence we need to be specific on the types of the Gaussians positions.

```python
from collections import OrderedDict
from kernel_tuner import tune_kernel

# create the dictionary containing the tune parameters
tune_params = OrderedDict()
tune_params['block_size_x'] = [2,4,8,16,32]
tune_params['block_size_y'] = [2,4,8,16,32]
tune_params['block_size_z'] = [2,4,8,16,32]
tune_params['nx'] = [n]
tune_params['ny'] = [n]
tune_params['nz'] = [n]

# define the final grid
grid = np.zeros_like(xgrid)

# arguments of the CUDA function
x0,y0,z0 = np.float32(0),np.float32(0),np.float32(0)
args = [x0,y0,z0,x,y,z,grid]

# dimensionality
problem_size = (n,n,n)
```

As mentioned earlier, the tuner will automatically insert `#define` statements at the top of the kernel to define the block sizes and domain dimensions, so we don't need to specify them here. Then, we simply call the `tune_kernel` function.

```python
# call the kernel tuner
result = tune_kernel('AddGrid', kernel_code, problem_size, args, tune_params)
```

    Using: GeForce GTX 1080 Ti
    block_size_x=2, block_size_y=2, block_size_z=2, time=3.56833920479
    block_size_x=2, block_size_y=2, block_size_z=4, time=1.80796158314
    block_size_x=2, block_size_y=2, block_size_z=8, time=0.940044796467
    block_size_x=2, block_size_y=2, block_size_z=16, time=0.855628800392
    block_size_x=2, block_size_y=2, block_size_z=32, time=0.855359995365
    block_size_x=2, block_size_y=4, block_size_z=2, time=4.16174077988
    block_size_x=2, block_size_y=4, block_size_z=4, time=2.11877760887
    block_size_x=2, block_size_y=4, block_size_z=8, time=1.01592960358
    block_size_x=2, block_size_y=4, block_size_z=16, time=0.849273598194
    block_size_x=2, block_size_y=4, block_size_z=32, time=0.849235200882
    block_size_x=2, block_size_y=8, block_size_z=2, time=4.19029750824
    block_size_x=2, block_size_y=8, block_size_z=4, time=2.16199679375
    block_size_x=2, block_size_y=8, block_size_z=8, time=1.40401918888
    block_size_x=2, block_size_y=8, block_size_z=16, time=1.39618558884
    block_size_x=2, block_size_y=8, block_size_z=32, time=1.39508478642
    block_size_x=2, block_size_y=16, block_size_z=2, time=5.31647996902
    block_size_x=2, block_size_y=16, block_size_z=4, time=2.31470079422
    block_size_x=2, block_size_y=16, block_size_z=8, time=1.50787198544
    block_size_x=2, block_size_y=16, block_size_z=16, time=1.53760001659
    block_size_x=2, block_size_y=16, block_size_z=32, time=1.56709756851
    block_size_x=2, block_size_y=32, block_size_z=2, time=5.34500494003
    block_size_x=2, block_size_y=32, block_size_z=4, time=2.25130877495
    block_size_x=2, block_size_y=32, block_size_z=8, time=1.50662400723
    block_size_x=2, block_size_y=32, block_size_z=16, time=1.55267841816
    block_size_x=4, block_size_y=2, block_size_z=2, time=4.17987194061
    block_size_x=4, block_size_y=2, block_size_z=4, time=2.12309756279
    block_size_x=4, block_size_y=2, block_size_z=8, time=1.01125121117
    block_size_x=4, block_size_y=2, block_size_z=16, time=0.849631989002
    block_size_x=4, block_size_y=2, block_size_z=32, time=0.853708791733
    block_size_x=4, block_size_y=4, block_size_z=2, time=4.17051515579
    block_size_x=4, block_size_y=4, block_size_z=4, time=2.15584001541
    block_size_x=4, block_size_y=4, block_size_z=8, time=1.40074241161
    block_size_x=4, block_size_y=4, block_size_z=16, time=1.39547519684
    block_size_x=4, block_size_y=4, block_size_z=32, time=1.39331197739
    block_size_x=4, block_size_y=8, block_size_z=2, time=5.30295038223
    block_size_x=4, block_size_y=8, block_size_z=4, time=2.28725762367
    block_size_x=4, block_size_y=8, block_size_z=8, time=1.39589118958
    block_size_x=4, block_size_y=8, block_size_z=16, time=1.38867840767
    block_size_x=4, block_size_y=8, block_size_z=32, time=1.37724158764
    block_size_x=4, block_size_y=16, block_size_z=2, time=5.34344320297
    block_size_x=4, block_size_y=16, block_size_z=4, time=2.26213116646
    block_size_x=4, block_size_y=16, block_size_z=8, time=1.38793599606
    block_size_x=4, block_size_y=16, block_size_z=16, time=1.3775359869
    block_size_x=4, block_size_y=32, block_size_z=2, time=4.74003200531
    block_size_x=4, block_size_y=32, block_size_z=4, time=2.13276162148
    block_size_x=4, block_size_y=32, block_size_z=8, time=1.37233917713
    block_size_x=8, block_size_y=2, block_size_z=2, time=4.18835201263
    block_size_x=8, block_size_y=2, block_size_z=4, time=2.15777277946
    block_size_x=8, block_size_y=2, block_size_z=8, time=1.40247042179
    block_size_x=8, block_size_y=2, block_size_z=16, time=1.39366400242
    block_size_x=8, block_size_y=2, block_size_z=32, time=1.39439997673
    block_size_x=8, block_size_y=4, block_size_z=2, time=5.23719043732
    block_size_x=8, block_size_y=4, block_size_z=4, time=2.28542718887
    block_size_x=8, block_size_y=4, block_size_z=8, time=1.39207677841
    block_size_x=8, block_size_y=4, block_size_z=16, time=1.38956804276
    block_size_x=8, block_size_y=4, block_size_z=32, time=1.3778496027
    block_size_x=8, block_size_y=8, block_size_z=2, time=5.29814395905
    block_size_x=8, block_size_y=8, block_size_z=4, time=2.26398081779
    block_size_x=8, block_size_y=8, block_size_z=8, time=1.38625922203
    block_size_x=8, block_size_y=8, block_size_z=16, time=1.3754431963
    block_size_x=8, block_size_y=16, block_size_z=2, time=4.72981758118
    block_size_x=8, block_size_y=16, block_size_z=4, time=2.12483196259
    block_size_x=8, block_size_y=16, block_size_z=8, time=1.37322881222
    block_size_x=8, block_size_y=32, block_size_z=2, time=4.61618566513
    block_size_x=8, block_size_y=32, block_size_z=4, time=2.2194111824
    block_size_x=16, block_size_y=2, block_size_z=2, time=5.17600002289
    block_size_x=16, block_size_y=2, block_size_z=4, time=2.27082881927
    block_size_x=16, block_size_y=2, block_size_z=8, time=1.38787200451
    block_size_x=16, block_size_y=2, block_size_z=16, time=1.3835711956
    block_size_x=16, block_size_y=2, block_size_z=32, time=1.37543039322
    block_size_x=16, block_size_y=4, block_size_z=2, time=5.30227203369
    block_size_x=16, block_size_y=4, block_size_z=4, time=2.23127679825
    block_size_x=16, block_size_y=4, block_size_z=8, time=1.38627202511
    block_size_x=16, block_size_y=4, block_size_z=16, time=1.37677440643
    block_size_x=16, block_size_y=8, block_size_z=2, time=4.64358406067
    block_size_x=16, block_size_y=8, block_size_z=4, time=2.12255358696
    block_size_x=16, block_size_y=8, block_size_z=8, time=1.37474560738
    block_size_x=16, block_size_y=16, block_size_z=2, time=4.61655673981
    block_size_x=16, block_size_y=16, block_size_z=4, time=2.19179515839
    block_size_x=16, block_size_y=32, block_size_z=2, time=4.99912958145
    block_size_x=32, block_size_y=2, block_size_z=2, time=5.213971138
    block_size_x=32, block_size_y=2, block_size_z=4, time=2.16430072784
    block_size_x=32, block_size_y=2, block_size_z=8, time=1.38772480488
    block_size_x=32, block_size_y=2, block_size_z=16, time=1.3735104084
    block_size_x=32, block_size_y=4, block_size_z=2, time=4.54432649612
    block_size_x=32, block_size_y=4, block_size_z=4, time=2.05524477959
    block_size_x=32, block_size_y=4, block_size_z=8, time=1.36935677528
    block_size_x=32, block_size_y=8, block_size_z=2, time=4.42449922562
    block_size_x=32, block_size_y=8, block_size_z=4, time=2.10455036163
    block_size_x=32, block_size_y=16, block_size_z=2, time=4.67516155243
    best performing configuration: block_size_x=2, block_size_y=4, block_size_z=32, time=0.849235200882

The `tune_kernel` function explores all the possible combinations of tunable parameters (here only the block size). For each possible kernel configuration, the tuner compiles the code and its measures execution time (by default using 7 iterations). At the end of the the run, the `tune_kernel` outputs the optimal combination of the tunable parameters. But the measured execution time of all benchmarked kernels is also returned by `tune_kernel` for programmatic access to the data.

As you can see the range of performances is quite large. With our GPU (GeForce GTX 1080 Ti) we obtained a maximum time of 5.30 ms and minimum one of 0.84 ms. The performance of the kernel varies by a factor 6 depending on the thread block size!

## Using the optimized parameters

Now that we have determined which parameters are the best suited for our application we can specify them in our kernel and run it.  In our case, the optimal grid size determined by the tuner were  *block_size_x = 4, block_size_y = 2, block_size_z=16*. We therefore use these parameters here to define the block size. The grid size is simply obtained by dividing the dimension of the problem by the corresponding block size.

```python
from pycuda import driver, compiler, gpuarray, tools
import pycuda.autoinit

# optimal values of the block size
block = [4, 2, 16]

# corresponding grid size
grid_dim = [int(np.ceil(n/b)) for b, n in zip(block, problem_size)]
```

Before using the kernel we need to specify the block size in its definition. There are different ways of doing this, we here simply replace the `block_size_x`, `block_size_y` and `block_size_z` by their values determined by the tuner. In order to do that we create a dictionary that associates the name of the block size and their values and simply make the substitution. Once the block size are specified, we can compile the kernel ourselves and get the function.

```python
# change the values of the block sizes in the kernel
fixed_params = OrderedDict()
fixed_params['block_size_x'] = block[0]
fixed_params['block_size_y'] = block[1]
fixed_params['block_size_z'] = block[2]
fixed_params['nx'] = n
fixed_params['ny'] = n
fixed_params['nz'] = n

for k,v in fixed_params.items():
    kernel_code = kernel_code.replace(k,str(v))

# compile the kernel_code and extract the function
mod = compiler.SourceModule(kernel_code)
addgrid = mod.get_function('AddGrid')
```

We now have to manually create the gpuarrays that correspond to the vector x, y and z as well as the 3D grid. Once all these are defined we can call the `addgrid` function using the gpuarrays and the block and grid size in argument. We also time the execution to compare it with the one outputed by the kernel tuner. Note that we exlicitly synchronize the CPU and GPU to obtain an accurate timing.

```python
# create the gpu arrays
xgpu = gpuarray.to_gpu(x)
ygpu = gpuarray.to_gpu(y)
zgpu = gpuarray.to_gpu(z)
grid_gpu = gpuarray.zeros((n,n,n), np.float32)

# compute the grid and time the performance
t0 = time()
for xyz in center:
    x0,y0,z0 = xyz
    addgrid(x0,y0,z0,xgpu,ygpu,zgpu,grid_gpu,block = tuple(block),grid=tuple(grid_dim))
driver.Context.synchronize()
print('Final GPU time : %f ms' %((time()-t0)*1000))
```

    Final GPU time : 80.133200 ms

As you can see the GPU execution time is much lower than than the CPU execution time obtained above. In our case it went from roughly 40000 ms to just 80 ms !


## LiDAR Basics with VLP-16

LIDAR—Light Detection and Ranging—is used to find the precise distance of objects in relation to us. Velodyne LiDAR sensors use time-of-flight methodology for doing this.

When a laser pulse is emited, its time-of-shooting and direction are registered. The laser pulse travels through air until it hits an obstacle which reflects some of the energy. The time-of-acquisition and power received are registered by sensor after recieving the portion of energy. The spherical coordinates of the obstacle is calculated using time-of-acquisition which is returned along with power received(as relectance) after each scan.

As LiDAR sensor returns reading in spherical coordinates, let's brush up with the spherical coordinate system.
___
To know more on LiDAR's history and how it works this [blog](https://news.voyage.auto/an-introduction-to-lidar-the-key-self-driving-car-sensor-a7e405590cff) by Oliver Cameron is helpful.

### Spherical Coordinate System
In a spherical coordinate system, a point is defined by a distance and two angles. To represent the two angles we use azimuth($\theta$) and polar angle($\phi$) convention. Thus a point is defined by $(\text{r}, \theta, \phi)$.

As you can see from above diagram, the azimuth angle is in X-Y plane measured from X-axis and polar angle is in Z-Y plane measured from Z axis.

From above diagram, we can get the following equations for converting a cartesian coordinate to spherical coordinates.

<math>\begin{align} r&=\sqrt{x^2 + y^2 + z^2} \\ \theta &= \arccos\frac{z}{\sqrt{x^2 + y^2 + z^2}} = \arccos\frac{z}{r} \\ \varphi &= \arctan \frac{y}{x} \end{align}</math>

We can derive cartesian coordinates from spherical coordinates using below equations.

<math>\begin{align} x&=r \, \sin\theta \, \cos\varphi \\ y&=r \, \sin\theta \, \sin\varphi \\ z&=r \, \cos\theta\end{align}</math>

___
Read [more](https://en.wikipedia.org/wiki/Spherical_coordinate_system) about Spherical coordinate system at Wikipedia.

## VLP-16 coordinate system
Velodyne VLP-16 returns reading in spherical coordinates. But there is a slight difference from the above discussed convention.

In sensor coordinate system, a point is defined by (radius $\text{r}$, elevation $\omega$, azimuth $\alpha$). Elevation angle, $\omega$ is in Z-Y plane measured from Y-axis. Azimuth angle, $\alpha$ is in X-Y plane measured from Y-axis. Below is the diagram.

Cartesian coordinates can be derived by following equations.

<math>\begin{align} x&=r \, \cos\omega \, \sin\alpha \\ y&=r \, \cos\omega \, \cos\alpha \\ z&=r \, \sin\omega\end{align}</math>

A computation is necessary to convert the spherical data from the sensor to Cartesian coordinates using above equations. This can be done using a ros package.

Above diagram shows the coordinate system of sensor mounted on a car.
___
This [manual](https://velodynelidar.com/docs/manuals/63-9243%20REV%20D%20MANUAL,USERS,VLP-16.pdf) is a good start to know more about Velodyne VLP-16.

## Data format
An unstructured point cloud is returned after each scan by sensor. Even though LiDAR returns reading in spherical coordinates, widely used coordinate system is Cartesian.

A point in point cloud is defined by it's coordinates and reflectance. Reflectance tells us the reflectivity of the surface. A zero value in reflectance denotes that the laser pulse didn't result in a measurement.

There are many formats to store and process point clouds like PCD, PCL but we can treat point cloud as a `Numpy` array. Each element of the array will have `(x,y,z,r)`. For processing point cloud we can use `Numpy`.

tutorial/LiDAR Basics  with VLP-16.ipynb

tutorial/LiDAR Basics  with VLP-16.ipynb

tutorial/LiDAR Basics  with VLP-16.ipynb


# Regression

### Preliminaries

- Goal
  - Introduction to Bayesian (Linear) Regression
- Materials
  - Mandatory
    - These lecture notes
  - Optional
    - Bishop pp. 152-158

### Regression - Illustration

Given a set of (noisy) data measurements, find the 'best' relation between an input variable $x \in \mathbb{R}^D$ and input-dependent outcomes $y \in \mathbb{R}$

### Regression vs Density Estimation

- Observe $N$ IID data **pairs** $D=\{(x_1,y_1),\dotsc,(x_N,y_N)\}$ with $x_n \in \mathbb{R}^M$ and $y_n \in \mathbb{R}$.

- Assume that we are interested in (a model for) the responses $y_n$ for **given inputs** $x_n$?, I.o.w. we are interested in building a model for the conditional distribution $p(y|x)$.

- Note that, since $p(x,y)=p(y|x)\, p(x)$, building a model $p(y|x)$ is similar to density estimation with the assumption that $x$ is drawn from a uniform distribution.

### Bayesian Linear Regression

- Next, we discuss (1) model specification, (2) Inference and (3) a prediction application for a Bayesian linear regression problem.

#####  1. Model Specification

- In a _regression_ model, we try to 'explain the data' by a purely deterministic term $f(x_n,w)$, plus a purely random term $\epsilon_n$ for 'unexplained noise',

    $$
    y_n  = f(x_n,w) + \epsilon_n
    $$

-  In a _linear regression_ model, i.e., linear w.r.t. the parameters $w$, we assume that
$$f(x_n,w)= \sum_{j=0}^{M-1} w_j \phi_j(x_n) = w^T \phi(x_n)$$
where $\phi_j(x)$ are called basis functions.
  - For notational simplicity, from now on we will assume $f(x_n,w) = w^T x_n$, with $x_n \in \mathbb{R}^M$.

-  In _ordinary linear regression_ , the noise process $\epsilon_n$ is zero-mean Gaussian with constant variance, i.e.
$$
y_n  = w^T x_n  + \mathcal{N}(0,\beta^{-1}) \,.
$$

- Hence, given a data set  $D=\{(x_1,y_1),\dotsc,(x_N,y_N)\}$, the likelihood for an ordinary linear regression model is

$$\begin{align*}
p(y\,|\,\mathbf{X},w,\beta) &= \mathcal{N}(y\,|\,\mathbf{X} w,\beta^{-1} \mathbf{I}) \\
  &= \prod_n \mathcal{N}(y_n\,|\,w^T x_n,\beta^{-1}) \tag{B-3.10}
\end{align*}$$

where $w = \left(\begin{matrix} w_1 \\ w_2 \\ \vdots \\ w_{M} \end{matrix} \right)$, the $(N\times M)$-dim matrix $\mathbf{X}  = \left(\begin{matrix}x_1^T \\ x_2^T \\ \vdots \\ x_N^T \end{matrix} \right) = \left(\begin{matrix}x_{11},x_{12},\dots,x_{1M}\\ x_{21},x_{22},\dots,x_{2M} \\ \vdots \\ x_{N1},x_{N2},\dots,x_{NM} \end{matrix} \right) $  and $y = \left(\begin{matrix} y_1 \\ y_2 \\ \vdots \\ y_N \end{matrix} \right)$.

- For full Bayesian learning we should also choose a prior $p(w)$:

$$\begin{equation}
p(w\,|\,\alpha) = \mathcal{N}(w\,|\,0,\alpha^{-1}\mathbf{I}) \tag{B-3.52}
\end{equation}$$

- For simplicity, we will assume that $\alpha$ and $\beta$ are known.

##### 2. Inference

- We'll do Bayesian inference for the parameters $w$.

$$\begin{align*}
p(w|D) &\propto p(D|w)\cdot p(w) \\
   &= \mathcal{N}(y\,|\,\mathbf{X} w,\beta^{-1} \mathbf{I}) \cdot \mathcal{N}(w\,|\,0,\alpha^{-1}\mathbf{I}) \\
   &\propto \exp \big( -\frac{\beta}{2} \big( {y - \mathbf{X}w } \big)^T \big( {y - \mathbf{X}w } \big)  - \frac{\alpha}{2}w^T w \big)\tag{B-3.55} \\
   &= \exp\big( -\frac{1}{2} w^T\big(\underbrace{\beta \mathbf{X}^T\mathbf{X} + \alpha \mathbf{I}}_{\Lambda_N}\big)w + \big(\underbrace{\beta \mathbf{X}^Ty}_{\eta_N}\big)^T w - \frac{\beta}{2}y^T y \big) \\
     &\propto \mathcal{N}_c\left(w\,|\,\eta_N,\Lambda_N \right)
\end{align*}$$

with natural parameters (see the [natural parameterization of Gaussian](https://nbviewer.jupyter.org/github/bertdv/BMLIP/blob/master/lessons/notebooks/The-Gaussian-Distribution.ipynb#natural-parameterization)):

$$\begin{align*}
\eta_N &= \beta\mathbf{X}^Ty \\
\Lambda_N &= \beta \mathbf{X}^T\mathbf{X} + \alpha \mathbf{I}
\end{align*}$$

- Or equivalently (in the moment parameterization of the Gaussian):
$$\begin{align*}
p(w|D) &= \mathcal{N}\left(w\,|\,m_N,S_N \right) \tag{B-3.49} \\
m_N &= \beta S_N \mathbf{X}^T y  \tag{B-3.53}\\
S_N &= \left(\alpha \mathbf{I} + \beta \mathbf{X}^T \mathbf{X}\right)^{-1} \tag{B-3.54}
\end{align*}$$

- Note that B-3.53 and B-3.54 combine to
$$
m_N = \left(\frac{\alpha}{\beta}\mathbf{I} + \mathbf{X}^T \mathbf{X} \right)^{-1} \mathbf{X}^T y\,.
$$

 - (Bishop Fig.3.7) Illustration of sequential Bayesian learning for a simple linear model of the form $y(x, w) =
w_0 + w_1 x$. (Bishop Fig.3.7, detailed description at Bishop, pg.154.)

##### 3. Application: predictive distribution

- Assume we are interested in the distribution $p(y_\bullet \,|\, x_\bullet, D)$ for a new input $x_\bullet$. This can be worked out as (exercise B-3.10)

$$\begin{align*}
p(y_\bullet \,|\, x_\bullet, D) &= \int p(y_\bullet \,|\, x_\bullet, w) p(w\,|\,D)\,\mathrm{d}w \\
&= \int \mathcal{N}(y_\bullet \,|\, w^T x_\bullet, \beta^{-1}) \mathcal{N}(w\,|\,m_N,S_N)\,\mathrm{d}w \\
&= \int \mathcal{N}(y_\bullet \,|\, z, \beta^{-1}) \mathcal{N}(z\,|\,x_\bullet^T m_N,x_\bullet^T S_N x_\bullet)\,\mathrm{d}z \quad \text{(sub. }z=x_\bullet^T w \text{)} \\
&= \int \underbrace{\mathcal{N}(z \,|\, y_\bullet, \beta^{-1}) \mathcal{N}(z\,|\,x_\bullet^T m_N,x_\bullet^T S_N x_\bullet)}_{\text{Use Gaussian product formula SRG-6}}\,\mathrm{d}z  \\
&= \int \mathcal{N}\left(y_\bullet\,|\, m_N^T x_\bullet, \sigma_N^2(x_\bullet) \right) \underbrace{\mathcal{N}\left(z\,|\, m_z, S_z\right)}_{\text{integrate this out}} \mathrm{d}z\\
&= \mathcal{N}\left(y_\bullet\,|\, m_N^T x_\bullet, \sigma_N^2(x_\bullet) \right)
\end{align*}$$
with
$$\begin{align*}
\sigma_N^2(x_\bullet) = \beta^{-1} + x^T_\bullet S_N x_\bullet \tag{B-3.59}
\end{align*}$$

- So, the uncertainty $\sigma_N^2(x_\bullet)$ about the output contains comprises both uncertainty about the process ($\beta^{-1}$) and about the model parameters ($x^T_\bullet S_N x_\bullet$).

### Example Predictive Distribution

- As an example, let's do Bayesian Linear Regression for a synthetic sinusoidal data set and a model with 9 Gaussian basis functions
$$\begin{align*}
y &=\sum_{m=1}^9 w_m \phi_m(x_n) + \mathcal{N}(0,\beta^{-1}) \\
\phi_m(x_n) &= \exp\left( \frac{x_n-\mu_m}{\sigma^2}\right)
\end{align*}$$

- The predictive distributions for $y$ are shown in the following plots (Bishop, Fig.3.8)

- And some plots of draws of posteriors for the functions $w^T \phi(x)$ (Bishop, Fig.3.9)

###  Maximum Likelihood Estimation for Linear Regression Model

- Recall the posterior mean for the weight vector
$$
m_N = \left(\frac{\alpha}{\beta}\mathbf{I} + \mathbf{X}^T \mathbf{X} \right)^{-1} \mathbf{X}^T y
$$
where $\alpha$ is the prior precision for the weights.

- The Maximum Likelihood solution for $w$ is obtained by letting $\alpha \rightarrow 0$, which leads to

$$\begin{equation*}
\hat w_{\text{ML}} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T y
\end{equation*}$$

-  The matrix $\mathbf{X}^\dagger \equiv  (\mathbf{X}^T \mathbf{X})^{-1}\mathbf{X}^T$ is also known as the **Moore-Penrose pseudo-inverse** (which is sort-of-an-inverse for non-square matrices).

-  Note that if we have fewer training samples than input dimensions, i.e., if $N<M$, then $\mathbf{X}^T \mathbf{X}$  will not be invertible and maximum likelihood blows up. The Bayesian solution does not suffer from this problem.

###  Least-Squares Regression

-  (You may say that) we don't need to work with probabilistic models. E.g., there's also the deterministic **least-squares** solution: minimize sum of squared errors,

$$\begin{align*} \hat w_{\text{LS}} &= \arg\min_{w} \sum_n {\left( {y_n  - w ^T x_n } \right)} ^2
  = \arg\min_{w} \left( {y - \mathbf{X}w } \right)^T \left( {y - \mathbf{X} w } \right)
\end{align*}$$

-  Setting the gradient
$ \frac{\partial \left( {y - \mathbf{X}w } \right)^T \left( {y - \mathbf{X}w } \right)}{\partial w} = -2 \mathbf{X}^T \left(y - \mathbf{X} w  \right)
$ to zero yields the so-called  **normal equations**
$\mathbf{X}^T\mathbf{X} \hat w_{\text{LS}} = \mathbf{X}^T y$  and consequently

$$
\hat w_{\text{LS}} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T y
$$

which is the same answer as we got for the maximum likelihood weights $\hat w_{\text{ML}}$.

- $\Rightarrow$ Least-squares regression ($\hat w_{\text{LS}}$) corresponds to the (probabilistic) maximum likelihood solution ($\hat w_{\text{ML}}$) if the probabilistic model includes the following assumptions:
  1. The observations are independently and identically distributed (**IID**) (this determines how errors are combined), and
  1. The noise signal $\epsilon_n \sim \mathcal{N}(0,\,\beta^{-1})$ is **Gaussian** distributed (determines the error metric)

- If you use the Least-Squares method, you cannot see (nor modify) these assumptions. The probabilistic method forces you to state all assumptions explicitly!

###  Not Identically Distributed Data

-  Let's do an example regarding changing our assumptions. What if we assume that the variance of the measurement error varies with the sampling index,  $\epsilon_n \sim \mathcal{N}(0,\beta_n^{-1})$?

- The likelihood is now (using $\Lambda \triangleq \mathrm{diag}(\beta_n)$ )
$$
p(y\,|\,\mathbf{X},w,\Lambda) = \mathcal{N}(y\,|\,\mathbf{X} w,\Lambda^{-1} ) \,.
$$

- Combining this likelihood with the prior $p(w) = \mathcal{N}(w\,|\,0,\alpha^{-1}\mathbf{I})$ leads to a posterior

$$\begin{align*}
p(w|D) &\propto p(D|w)\cdot p(w) \\
   &= \mathcal{N}(y\,|\,\mathbf{X} w,\Lambda^{-1} \mathbf{I}) \cdot \mathcal{N}(w\,|\,0,\alpha^{-1}\mathbf{I}) \\
   &\propto \exp \left\{ \frac{1}{2} \left( {y - \mathbf{X}w } \right)^T \Lambda \left( {y - \mathbf{X}w } \right)  + \frac{\alpha}{2}w^T w \right\} \\
   &\propto \mathcal{N}\left(w\,|\,m_N,S_N \right)
\end{align*}$$
with
$$\begin{align*}
m_N &=  S_N \mathbf{X}^T \Lambda y  \\
S_N &= \left(\alpha \mathbf{I} +  \mathbf{X}^T \Lambda \mathbf{X}\right)^{-1}
\end{align*}$$

- And maximum likelihood solution
$$
\hat{w}_{\text{ML}} = \left. m_N\right|_{\alpha \rightarrow 0} = \left(\mathbf{X}^T \Lambda \mathbf{X}\right)^{-1} \mathbf{X}^T \Lambda y
$$

-  This maximum likelihood solution is also called the **Weighted Least Squares** (WLS) solution. (Note that we just stumbled upon it, the crucial aspect is appropriate model specification!)

-  Note also that the dimension of $\Lambda$ grows with the number of data points. In general, models for which the number of parameters grow as the number of observations increase are called **non-parametric models**.

#### CODE EXAMPLE

We'll compare the Least Squares and Weighted Least Squares solutions for a simple linear regression model with input-dependent noise:

$$\begin{align*}
x &\sim \text{Unif}[0,1]\\
y|x &\sim \mathcal{N}(f(x), v(x))\\
f(x) &= 5x - 2\\
v(x) &= 10e^{2x^2}-9.5\\
\mathcal{D} &= \{(x_1,y_1),\ldots,(x_N,y_N)\}
\end{align*}$$

```julia
using Pkg;Pkg.activate("probprog/workspace/");Pkg.instantiate();
IJulia.clear_output();
```

```julia
using PyPlot, LinearAlgebra

# Model specification: y|x ~ 𝒩(f(x), v(x))
f(x) = 5*x .- 2
v(x) = 10*exp.(2*x.^2) .- 9.5 # input dependent noise variance
x_test = [0.0, 1.0]
plot(x_test, f(x_test), "k--") # plot f(x)

# Generate N samples (x,y), where x ~ Unif[0,1]
N = 50
x = rand(N)
y = f(x) + sqrt.(v(x)) .* randn(N)
plot(x, y, "kx"); xlabel("x"); ylabel("y") # Plot samples

# Add constant to input so we can estimate both the offset and the slope
_x = [x ones(N)]
_x_test = hcat(x_test, ones(2))

# LS regression
w_ls = pinv(_x) * y
plot(x_test, _x_test*w_ls, "b-") # plot LS solution

# Weighted LS regression
W = Diagonal(1 ./ v(x)) # weight matrix
w_wls = inv(_x'*W*_x) * _x' * W * y
plot(x_test, _x_test*w_wls, "r-") # plot WLS solution
ylim([-5,8]); legend(["f(x)", "D", "LS linear regr.", "WLS linear regr."],loc=2);
```

```julia
open("../../styles/aipstyle.html") do f
    display("text/html", read(f, String))
end
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>

```julia

```


```python
from sympy import init_printing; init_printing();
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
```

```python
from sympy import symbols, cos, sin, Array
from silkpy.symbolic.geometry_map import GeometryMap
from silkpy.symbolic.curve.curve import ParametricCurve

a, b = symbols("a, b", positive=True); t = symbols("t", real=True)
line_on_cylind = ParametricCurve(t, [a*cos(t), a*sin(t), b*t])
```

```python
line_on_cylind.T_N_B
line_on_cylind.jacobian
```

    /home/wenyin/pyvenv/vsym/lib/python3.8/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The to_png function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    /home/wenyin/pyvenv/vsym/lib/python3.8/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The to_rgba function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    /home/wenyin/pyvenv/vsym/lib/python3.8/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The to_mask function was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)
    /home/wenyin/pyvenv/vsym/lib/python3.8/site-packages/IPython/lib/latextools.py:126: MatplotlibDeprecationWarning:
    The MathtextBackendBitmap class was deprecated in Matplotlib 3.4 and will be removed two minor releases later. Use mathtext.math_to_image instead.
      mt.to_png(f, s, fontsize=12, dpi=dpi, color=color)

$\displaystyle \left( \left[\begin{matrix}- \frac{a \sin{\left(t \right)}}{\sqrt{a^{2} + b^{2}}} & \frac{a \cos{\left(t \right)}}{\sqrt{a^{2} + b^{2}}} & \frac{b}{\sqrt{a^{2} + b^{2}}}\end{matrix}\right], \  \left[\begin{matrix}- \cos{\left(t \right)} & - \sin{\left(t \right)} & 0\end{matrix}\right], \  \left[\begin{matrix}\frac{b \sin{\left(t \right)}}{\sqrt{a^{2} + b^{2}}} & - \frac{b \cos{\left(t \right)}}{\sqrt{a^{2} + b^{2}}} & \frac{a}{\sqrt{a^{2} + b^{2}}}\end{matrix}\right]\right)$

$\displaystyle \left[\left[\begin{matrix}- a \sin{\left(t \right)} & a \cos{\left(t \right)} & b\end{matrix}\right]\right]$

```python
from silkpy.symbolic.curve.transform import curve_normalize
line_on_cylind_normalized = curve_normalize(line_on_cylind)
```

```python
line_on_cylind_normalized.T_N_B
```

$\displaystyle \left( \left[\begin{matrix}- \frac{a \sin{\left(\frac{s}{\sqrt{a^{2} + b^{2}}} \right)}}{\sqrt{a^{2} + b^{2}}} & \frac{a \cos{\left(\frac{s}{\sqrt{a^{2} + b^{2}}} \right)}}{\sqrt{a^{2} + b^{2}}} & \frac{b}{\sqrt{a^{2} + b^{2}}}\end{matrix}\right], \  \left[\begin{matrix}- \cos{\left(\frac{s}{\sqrt{a^{2} + b^{2}}} \right)} & - \sin{\left(\frac{s}{\sqrt{a^{2} + b^{2}}} \right)} & 0\end{matrix}\right], \  \left[\begin{matrix}\frac{b \sin{\left(\frac{s}{\sqrt{a^{2} + b^{2}}} \right)}}{\sqrt{a^{2} + b^{2}}} & - \frac{b \cos{\left(\frac{s}{\sqrt{a^{2} + b^{2}}} \right)}}{\sqrt{a^{2} + b^{2}}} & \frac{a}{\sqrt{a^{2} + b^{2}}}\end{matrix}\right]\right)$

```python

```

```python

```


## affine transformation matrix (the augmented matrix in equation)

\begin{equation}
    \begin{bmatrix}
        v_x\\
        v_y\\
        1
    \end{bmatrix}
    =
    \begin{bmatrix}
        c & 0 & l\\
        0 & -c & t\\
        0 & 0 & 1
    \end{bmatrix}
    \begin{bmatrix}
        r_x\\
        r_y\\
        1
    \end{bmatrix}
\end{equation}

```python
import matplotlib.pyplot as plt
%matplotlib inline
from matplotlib import patches
import numpy as np
import pandas as pd
import matplotlib
```

```python
fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(8, 4))

axes[0].set_aspect("equal")
axes[0].set_xlim(4950, 6050)
axes[0].set_ylim(3450, 4550)
axes[0].set_xticks(np.arange(5000,6100,500))
axes[0].set_xticks(np.arange(5000,6100,250), minor=True)
axes[0].set_yticks(np.arange(3500,4600,500))
axes[0].set_yticks(np.arange(3500,4600,250), minor=True)
lines = [(5200, 5800), (3700, 4300), "red",
         (5400, 5400), (4400, 3600), "red"]
axes[0].plot(*lines)

a = np.zeros((11,11))
a[1:10, 4] = 1
a[np.arange(2, 9), np.arange(8,1,-1)] = 1

axes[1].set_xticks(np.arange(0,11,5))
axes[1].set_xticks(np.arange(0,11,2.5), minor=True)
axes[1].set_yticks(np.arange(0,11,5))
axes[1].set_yticks(np.arange(0,11,2.5), minor=True)
axes[1].set_aspect("equal")

levels = [0, 1, 2]
colors = ['white', 'grey']
cmap, norm = matplotlib.colors.from_levels_and_colors(levels, colors)

axes[1].imshow(a, cmap=cmap, interpolation='none', vmin=0, vmax=1, aspect='equal')

def rect(pos, axisid=1, background=False):
    if background is False:
        r = plt.Rectangle(pos-0.5, 1, 1, facecolor="none", edgecolor="k", linewidth=2, linestyle="--")
    else:
        r = plt.Rectangle(pos-0.5, 1, 1, facecolor="none", edgecolor="k", linewidth=1, linestyle=":")
    axes[axisid].add_patch(r)

x, y = np.meshgrid(np.arange(a.shape[1]), np.arange(a.shape[0]))

m = np.c_[x[a.astype(bool)], y[a.astype(bool)]]

b = np.ones((11,11))
n = np.c_[x[b.astype(bool)], y[b.astype(bool)]]

for pos in n:
    rect(pos, background=True)

# vectors = np.array([(5200, 3700, 1), (5800, 4300, 1), (5400, 4400, 1), (5400, 3600, 1)])
# vectors = vectors.transpose()

# affine_mtx = np.array([[100, 0, 5000],
#                        [0, -100, 4500],
#                        [0, 0, 1]])
# result = np.linalg.inv(affine_mtx).dot(vectors)

plt.tight_layout()
fig.savefig("./affine.png", dpi=600)
```


```python
# Models from Introduction to Algorithmic Marketing
# https://algorithmicweb.wordpress.com/
#
# Adstock model helps to estimate the contribution of
# individual campaigns to the overall financial performance
```

```python
%matplotlib inline
import sympy as sy
import numpy as np
from mpmath import *
from scipy.optimize import minimize
import matplotlib.pyplot as plt
```

```python
# The input of the model is a sequence of samples (time series)
# where each sample is represented by three values:
#  - Total revenue
#  - Intensity of campaign 01
#  - Intensity of campaign 02
revenue_series = [37, 89, 82, 58, 110, 77, 103, 78, 95, 106, 98, 96, 68, 96, 157, 198, 145, 132, 96, 135]
campaign_series_01 = [6, 27, 0, 0, 20, 0, 20, 0, 0, 18, 9, 0, 0, 0, 13, 25, 0, 15, 0, 0]
campaign_series_02 = [3, 0, 4, 0, 5, 0, 0, 0, 8, 0, 0, 5, 0, 11, 16, 11, 5, 0, 0, 15]
decay_length = 3
time = range(1, len(revenue_series)+1)
plt.plot(time, revenue_series, time, campaign_series_01, time, campaign_series_02);
```

```python
# Adds a time lag (memory effect) to the campaign intensity samples
def lag(campaing_series, alpha, decay_lenght):
    lag_series = []
    for t in range(1, len(campaing_series)+1):
        lag_series.append(
            campaing_series[t-1] +
            nsum(lambda j: alpha**j * campaing_series[int(t-1-j)], [1, min(t, decay_lenght)])
        )
    return lag_series
```

```python
# Apply a time lag to the campaign intensity data, and then
# define the Adstock model as a linear function of lagged samples.
# The model training objective is to minimize the approximation error
symbols = sy.symbols("b w1 w2 a1 a2")
b, w1, w2, a1, a2 = symbols
campaing_lag_01 = sy.Matrix(lag(campaign_series_01, a1, decay_length))
campaing_lag_02 = sy.Matrix(lag(campaign_series_02, a2, decay_length))
revenue_series_model = sy.Matrix([b]*len(time)) + w1*campaing_lag_01 + w2*campaing_lag_02
objective = (sy.Matrix(revenue_series) - revenue_series_model).norm()
```

```python
# Each sample of the revenue series is approximated by a polynomial of several
# campaing intensity samples
revenue_series_model
```

    Matrix([
    [                                                b + 6.0*w1 + w2*(15*a2**1.0 + 3)],
    [                           b + w1*(6*a1**1.0 + 27) + w2*(3*a2**1.0 + 15*a2**2.0)],
    [               b + w1*(27*a1**1.0 + 6*a1**2.0) + w2*(3*a2**2.0 + 15*a2**3.0 + 4)],
    [                    b + w1*(27*a1**2.0 + 6*a1**3.0) + w2*(4*a2**1.0 + 3*a2**3.0)],
    [                                   b + w1*(27*a1**3.0 + 20) + w2*(4*a2**2.0 + 5)],
    [                                  20*a1**1.0*w1 + b + w2*(5*a2**1.0 + 4*a2**3.0)],
    [                                         5*a2**2.0*w2 + b + w1*(20*a1**2.0 + 20)],
    [                                 5*a2**3.0*w2 + b + w1*(20*a1**1.0 + 20*a1**3.0)],
    [                                                      20*a1**2.0*w1 + b + 8.0*w2],
    [                                         8*a2**1.0*w2 + b + w1*(20*a1**3.0 + 18)],
    [                                          8*a2**2.0*w2 + b + w1*(18*a1**1.0 + 9)],
    [                            b + w1*(9*a1**1.0 + 18*a1**2.0) + w2*(8*a2**3.0 + 5)],
    [                                  5*a2**1.0*w2 + b + w1*(9*a1**2.0 + 18*a1**3.0)],
    [                                          9*a1**3.0*w1 + b + w2*(5*a2**2.0 + 11)],
    [                                  b + 13.0*w1 + w2*(11*a2**1.0 + 5*a2**3.0 + 16)],
    [                    b + w1*(13*a1**1.0 + 25) + w2*(16*a2**1.0 + 11*a2**2.0 + 11)],
    [b + w1*(25*a1**1.0 + 13*a1**2.0) + w2*(11*a2**1.0 + 16*a2**2.0 + 11*a2**3.0 + 5)],
    [b + w1*(25*a1**2.0 + 13*a1**3.0 + 15) + w2*(5*a2**1.0 + 11*a2**2.0 + 16*a2**3.0)],
    [                  b + w1*(15*a1**1.0 + 25*a1**3.0) + w2*(5*a2**2.0 + 11*a2**3.0)],
    [                                         15*a1**2.0*w1 + b + w2*(5*a2**3.0 + 15)]])

```python
# Adstock model fitting using numerical optimization
objective_n = sy.lambdify(symbols, objective, modules='numpy')
def objective_v(x):
    return objective_n(*tuple(x))

solution = minimize(objective_v, np.zeros(len(symbols)))
bn, w1n, w2n, a1n, a2n = solution.x
```

```python
# Plot the difference between the actual revenue data and adstock model
revenue_predicted = sy.lambdify(symbols, revenue_series_model)
revenue_predicted_series = revenue_predicted(*tuple(solution.x))
plt.plot(time, revenue_series, time, revenue_predicted_series);
```

```python
# Evaluate and plot model components that correspond to individual campaigns
# (i.e. estimate contribution of individual campaigns)
def eval_campaing_lag(campaing_lag, a, aval, w):
    return sy.matrix2numpy(w*campaing_lag.evalf(subs={a:aval}), dtype=float).flatten().tolist()

plt.stackplot(time, [
    [bn]*len(time),
    eval_campaing_lag(campaing_lag_01, a1, a1n, w1n),
    eval_campaing_lag(campaing_lag_02, a2, a2n, w2n)
] );
```


# Discrete call-option pricing model

The goal of this assignment is formulate a script that correctly prices discrete call options.

The main extension to the model is to analyze how an investor could combine the asset and the risk-free asset to replicate the option.

### NB! The extension is only viable if the market is arbitrage-free and complete, which will be explained during the script

## Option value

For a market $(\Omega , \mathbb{P})$ where the market states are defined as $\Omega = ({w_{1} , w_{2} , ... , w_{s}})$ an option has different prices at different times $t$.

The options initial price $S_{0}$ can either go up $S_{up}$ or down $S_{down}$ with a given probability.
The up and down prices at time $t$ depends on the factors $u , d$ , which are defined as follows:

### $u = \exp^{\sigma \cdot \sqrt({t}/{n})}$ , $d = \exp^{-\sigma \cdot \sqrt({t}/{n})} = 1 / u$

These state-factors combined with the interest rate $R = (1+r)$ makes it possible to calculate the $q$ probability defined as:

### $q = \frac{(R - u)}{(u - d)}$

For a call option the option value depends on the strike price, and the option value in a certain state can be defined as :

### $C_{u} = (uS_{0}-K)^+$ , $C_{d} = (dS_{0}-K)^+$

All these equations can be combined to find the option value at $t=0$ denoted as $C_{0}$

### $C_{0} = \frac{1}{R} \cdot (q\cdot C_{u} + (1-q)\cdot C_{d})$

## No arbitrage and complete market?

If the market is complete and arbitrage-free, an investor is able to replicate the option by allocating between the asset and risk-free asset.

It is therefore beneficial to examine whether or not the market is complete and arbitrage-free.

A market $(\Omega , \mathbb{P})$ is arbitrage-free and complete if the probability measure $q\in [0 , 1]$ and $\sum q_{i} = 1$.

This ensures that q is the unconditional martingale-measure and makes the market Arbitrage-free and complete.

A quick way in our framework to test whether the market is Arbitrage-free and complete is to examine if $u>R>d$.

To replicate the option in an arbitrage-free and complete market (2-period) , we need to know the price-vector $\pi$ and payment matrix $D^T$ which are defined as the following, note that these definitions can be extended to further dimensions:

$\pi = \begin{bmatrix}
C_{u} \\\
C_{d}
\end{bmatrix} , $ $D^T = \begin{bmatrix}
S_{1u} & R \\\
S_{1d} & R
\end{bmatrix} $

To replcate the option you have to multiply the inverse payment matrix with the price vector :

$\Phi = (D^T)^{-1}\cdot \pi = \begin{bmatrix}
Assets \\\
Risk free
\end{bmatrix} $

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sp
```

```python
def option_tree(N , T , S0 , sigma , r , K , array_out=False):

    """This function generates a price-tree , option-pricing-tree and returns option price at time t = 0
        "array_out = False" will only show the option price a t = 0.
        "Array_out = True" will show details from function"""

#======================================================

    # Defining variables
    global price_tree , option , R
    dt = T / N                      # Expiration time / periods
    u = np.exp(sigma * np.sqrt(dt)) # Up-state
    d = 1/u                         # Down-State
    R = 1 + r                       # Interest
    q = (R - d) / (u - d)           # q-prob

#=======================================================

    # Price-Tree
    price_tree = np.zeros([N+1 , N+1]) # Empty grid

    for i in range(N + 1):
        for j in range(i + 1):
            price_tree[j , i] = S0 * (d ** j) * (u ** (i - j)) # Generating prices in grid

#=======================================================

    # Option value
    option = np.zeros([N + 1 , N + 1])
    option[: , N] = np.maximum(np.zeros(N + 1) , price_tree[: , N] - K) # Finds max when strike price has been accounted for

#=======================================================

    # Calculates option price at t = 0
    for i in np.arange(N - 1 , -1 , -1):
        for j in np.arange(0 , i + 1):
            option[j , i] = (1 / R) * (q * option[j , i + 1] + (1 - q) * option[j + 1 , i + 1]) # Calculates C0

#=======================================================
    answer1 = "The model is Arbitrage-free and complete"
    answer2 = "The model is not Arbitrage-free or complete, replication NOT possible"

    # Return the price
    if array_out:
        if u > R > d:
            return [str(answer1) , option[0 , 0] , price_tree , option]
        else:
            return [str(answer2) , option[0 , 0] , price_tree , option]
    else:
        return option[0 , 0]
```

```python
#Calling the function by selecting parameters
option_tree(2 , 2 , 50 , 0.182575441 , 0.05 , 45 , True)
```

    ['The model is Arbitrage-free and complete',
     10.74842362739621,
     array([[50.        , 60.01523499, 72.03656861],
            [ 0.        , 41.6560895 , 50.        ],
            [ 0.        ,  0.        , 34.70459585]]),
     array([[10.74842363, 17.15809213, 27.03656861],
            [ 0.        ,  2.81264011,  5.        ],
            [ 0.        ,  0.        ,  0.        ]])]

```python
#Plotting Option price
plt.xlim(0,1)
plt.figtext(0.18,0.55,"{:.2f}".format((price_tree[0 , 0])))
plt.figtext(0.22,0.1,"t=0")
plt.figtext(0.45,0.1,"t=1")
plt.figtext(0.65,0.1,"t=2")
plt.figtext(0.45,0.5+0.16,"{:.2f}".format((price_tree[0 , 1])))
plt.figtext(0.45,0.5-0.17,"{:.2f}".format((price_tree[1 , 1])))
plt.figtext(0.65,0.5+0.25,"{:.2f}".format((price_tree[0 , 2])))
plt.figtext(0.65,0.5+0.05,"{:.2f}".format((price_tree[1 , 2])))
plt.figtext(0.65,0.5-0.25,"{:.2f}".format((price_tree[2 , 2])))

plt.annotate('',xy=(0.5,0.5+0.15), xytext=(0.1,0.5), arrowprops=dict(facecolor='dimgrey',shrink=0.01))
plt.annotate('',xy=(0.5,0.5-0.15), xytext=(0.1,0.5), arrowprops=dict(facecolor='dimgrey',shrink=0.01))
plt.annotate('',xy=(0.75,0.5+0.24), xytext=(0.1,0.5), arrowprops=dict(facecolor='dimgrey',shrink=0.62))
plt.annotate('',xy=(0.75,0.5-0.24), xytext=(0.1,0.5), arrowprops=dict(facecolor='dimgrey',shrink=0.62))
plt.annotate('',xy=(0.75,.5), xytext=(0.1,0.5), arrowprops=dict(facecolor='dimgrey',shrink=0.68))
plt.title('Option Price')
plt.axis('off')
plt.show()
```

### The middle arrow shows if the price goes up then down, or down then up

```python
#Plotting Option Value
plt.xlim(0,1)
plt.figtext(0.18,0.55,"{:.2f}".format((option[0 , 0])))
plt.figtext(0.22,0.1,"t=0")
plt.figtext(0.45,0.1,"t=1")
plt.figtext(0.65,0.1,"t=2")
plt.figtext(0.45,0.5+0.16,"{:.2f}".format((option[0 , 1])))
plt.figtext(0.45,0.5-0.17,"{:.2f}".format((option[1 , 1])))
plt.figtext(0.65,0.5+0.25,"{:.2f}".format((option[0 , 2])))
plt.figtext(0.65,0.5+0.05,"{:.2f}".format((option[1 , 2])))
plt.figtext(0.65,0.5-0.25,"{:.2f}".format((option[2 , 2])))

plt.annotate('',xy=(0.5,0.5+0.15), xytext=(0.1,0.5), arrowprops=dict(facecolor='dimgrey',shrink=0.01))
plt.annotate('',xy=(0.5,0.5-0.15), xytext=(0.1,0.5), arrowprops=dict(facecolor='dimgrey',shrink=0.01))
plt.annotate('',xy=(0.75,0.5+0.24), xytext=(0.1,0.5), arrowprops=dict(facecolor='dimgrey',shrink=0.62))
plt.annotate('',xy=(0.75,0.5-0.24), xytext=(0.1,0.5), arrowprops=dict(facecolor='dimgrey',shrink=0.62))
plt.annotate('',xy=(0.75,.5), xytext=(0.1,0.5), arrowprops=dict(facecolor='dimgrey',shrink=0.68))
plt.title('Option Value')
plt.axis('off')
plt.show()
```

### The middle arrow shows if the value goes up then down, or down then up

## Replicating option by mixing asset and risk-free asset

```python
pi = sp.Matrix([[option[0 , 1]] , [option[1 , 1]]]) # price-vector
Dt = sp.Matrix([[price_tree[0 , 1] , R] , [price_tree[1 , 1] , R]]) # payment matrix

Dti = Dt**-1 # Inverting the payment matrix

phi = Dti * pi

display(phi , "Investment in asset {0:.3f}".format(phi[0]) , "Investment in Risk-Free asset {0:.3f}".format(phi[1]))

# Check if the calculations are correct

#display(Dt * phi) # Should be equal to the price-vector
```

$\displaystyle \left[\begin{matrix}0.781379069461348\\-28.3205298456712\end{matrix}\right]$

    'Investment in asset 0.781'

    'Investment in Risk-Free asset -28.321'