# Random Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing.

## Independent Processes

The independence of random signals is a desired property in many applications of statistical signal processing, as well as uncorrelatedness and orthogonality. The concept of independence is introduced in the following together with a discussion of the links to uncorrelatedness and orthogonality.

### Definition

Two stochastic events are said to be [independent](https://en.wikipedia.org/wiki/Independence_(probability_theory%29) if the probability of occurrence of one event is not affected by the occurrence of the other event. Or more specifically, if their joint probability equals the product of their individual probabilities. In terms of the bivariate probability density function (PDF) of two continuous-amplitude real-valued random processes $x[k]$ and $y[k]$ this reads

\begin{equation}
p_{xy}(\theta_x, \theta_y, k_x, k_y) = p_x(\theta_x, k_x) \cdot p_y(\theta_y, k_y)
\end{equation}

where $p_x(\theta_x, k_x)$ and $p_y(\theta_y, k_y)$ denote the univariate ([marginal](https://en.wikipedia.org/wiki/Marginal_distribution)) PDFs of the random processes for the time-instances $k_x$ and $k_y$, respectively. The bivariate PDF of two independent random processes is given by the multiplication of their univariate PDFs. It follows that the [second-order ensemble average](ensemble_averages.ipynb#Second-Order-Ensemble-Averages) for a linear mapping is given as

\begin{equation}
E\{ x[k_x] \cdot y[k_y] \} = E\{ x[k_x] \} \cdot E\{ y[k_y] \}
\end{equation}

The linear second-order ensemble average of two independent random signals is equal to the multiplication of their linear first-order ensemble averages. For jointly wide-sense stationary (WSS) processes, the bivariate PDF does only depend on the difference $\kappa = k_x - k_y$ of the time instants. Hence, two jointly WSS random signals are independent if

\begin{equation}
\begin{split}
p_{xy}(\theta_x, \theta_y, \kappa) &= p_x(\theta_x, k_x) \cdot p_y(\theta_y, k_x - \kappa) \\
&= p_x(\theta_x) \cdot p_y(\theta_y, \kappa)
\end{split}
\end{equation}

Above bivariate PDF is rewritten using the definition of [conditional probabilities](https://en.wikipedia.org/wiki/Conditional_probability) in order to specialize the definition of independence to one WSS random signal $x[k]$

\begin{equation}
p_{xy}(\theta_x, \theta_y, \kappa) = p_{y|x}(\theta_x, \theta_y, \kappa) \cdot p_x(\theta_x)
\end{equation}

where $p_{y|x}(\theta_x, \theta_y, \kappa)$ denotes the conditional probability that $y[k - \kappa]$ takes the amplitude value $\theta_y$ under the condition that $x[k]$ takes the amplitude value $\theta_x$. Under the assumption that $y[k-\kappa] = x[k-\kappa]$ and substituting $\theta_x$ and $\theta_y$ by $\theta_1$ and $\theta_2$, independence for one random signal is defined as

\begin{equation}
p_{xx}(\theta_1, \theta_2, \kappa) =
\begin{cases}
p_x(\theta_1) \cdot \delta(\theta_2 - \theta_1) & \text{for } \kappa = 0  \\
p_x(\theta_1) \cdot p_x(\theta_2, \kappa) & \text{for } \kappa \neq 0
\end{cases}
\end{equation}

since the conditional probability $p_{x[k]|x[k-\kappa]}(\theta_1, \theta_2, \kappa) = \delta(\theta_2 - \theta_1)$ for $\kappa = 0$ since this represents a sure event. The bivariate PDF of an independent random signal is equal to the product of the univariate PDFs of the signal and the time-shifted signal for $\kappa \neq 0$. A random signal for which this condition does not hold shows statistical dependencies between samples. These dependencies can be exploited for instance for coding or prediction.

#### Example - Comparison of bivariate PDF and product of marginal PDFs

The following example estimates the bivariate PDF $p_{xx}(\theta_1, \theta_2, \kappa)$ of a WSS random signal $x[k]$ by computing its two-dimensional histogram. The univariate PDFs $p_x(\theta_1)$ and $p_x(\theta_2, \kappa)$ are additionally estimated. Both the estimated bivariate PDF and the product of the two univariate PDFs $p_x(\theta_1) \cdot p_x(\theta_2, \kappa)$ are plotted for different $\kappa$.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

N = 10000000  # number of random samles
M = 50  # number of bins for bivariate/marginal histograms

def compute_plot_histograms(kappa):
    # shift signal
    x2 = np.concatenate((x1[kappa:], np.zeros(kappa)))

    # compute bivariate and marginal histograms
    pdf_xx, x1edges, x2edges = np.histogram2d(x1, x2, bins=(M,M), range=((-1.5, 1.5),(-1.5, 1.5)), normed=True)
    pdf_x1, _ = np.histogram(x1, bins=M, range=(-1.5, 1.5), density=True)
    pdf_x2, _ = np.histogram(x2, bins=M, range=(-1.5, 1.5), density=True)

    # plot results
    fig = plt.figure(figsize=(10, 10))

    plt.subplot(121, aspect='equal')
    plt.pcolormesh(x1edges, x2edges, pdf_xx)
    plt.xlabel(r'$\theta_1$')
    plt.ylabel(r'$\theta_2$')
    plt.title(r'Bivariate PDF $p_{{xy}}(\theta_1, \theta_2, \kappa)$')
    plt.colorbar(fraction=0.046)

    plt.subplot(122, aspect='equal')
    plt.pcolormesh(x1edges, x2edges, np.outer(pdf_x1, pdf_x2))
    plt.xlabel(r'$\theta_1$')
    plt.ylabel(r'$\theta_2$')
    plt.title(r'Product of PDFs $p_x(\theta_1) \cdot p_x(\theta_2, \kappa)$')
    plt.colorbar(fraction=0.046)

    fig.suptitle('Shift $\kappa =$ {:<2.0f}'.format(kappa), y=0.72)
    fig.tight_layout()

# generate signal
x = np.random.normal(size=N)
x1 = np.convolve(x, [1, .5, .3, .7, .3], mode='same')

# compute and plot the PDFs for various shifts
compute_plot_histograms(0)
compute_plot_histograms(2)
compute_plot_histograms(20)
```

**Exercise**

* With the given results, how can you evaluate the independence of the random signal?
* Can the random signal assumed to be independent?

Solution: According to the definition of independence, the bivariate PDF and the product of the univariate PDFs has to be equal for $\kappa \neq 0$. This is obviously not the case for $\kappa=2$. Hence, the random signal is not independent in a strict sense. However for $\kappa=20$ the condition for independence is sufficiently fulfilled, considering the statistical uncertainty due to a finite number of samples.

### Independence versus Uncorrelatedness

Two continuous-amplitude real-valued jointly WSS random processes $x[k]$ and $y[k]$ are termed as [uncorrelated](correlation_functions.ipynb#Properties) if their cross-correlation function (CCF) is equal to the product of their linear means, $\varphi_{xy}[\kappa] = \mu_x \cdot \mu_y$. If two random signals are independent then they are also uncorrelated. This can be proven by introducing above findings for the linear second-order ensemble average of independent random signals into the definition of the CCF

\begin{equation}
\varphi_{xy}[\kappa] = E \{ x[k] \cdot y[k - \kappa] \} = E \{ x[k] \} \cdot E \{ y[k - \kappa] \} = \mu_x \cdot \mu_y
\end{equation}

where the last equality is a consequence of the assumed wide-sense stationarity. The reverse, that two uncorrelated signals are also independent does not hold in general from this result.

The auto-correlation function (ACF) of an [uncorrelated signal](correlation_functions.ipynb#Properties) is given as $\varphi_{xx}[\kappa] = \mu_x^2 + \sigma_x^2 \cdot \delta[\kappa]$. Introducing the definition of independence into the definition of the ACF yields

\begin{equation}
\begin{split}
\varphi_{xx}[\kappa] &= E \{ x[k] \cdot x[k - \kappa] \} \\
&=
\begin{cases}
E \{ x^2[k] \} & \text{for } \kappa = 0 \\
E \{ x[k] \} \cdot E \{ x[k - \kappa] \} & \text{for } \kappa \neq 0
\end{cases} \\
&=
\begin{cases}
\mu_x^2 + \sigma_x^2 & \text{for } \kappa = 0 \\
\mu_x^2 & \text{for } \kappa \neq 0
\end{cases} \\
&= \mu_x^2 + \sigma_x^2 \delta[\kappa]
\end{split}
\end{equation}

where the result for $\kappa = 0$ follows from the bivariate PDF $p_{xx}(\theta_1, \theta_2, \kappa)$ of an independent signal, as derived above. It can be concluded from this result that an independent random signal is also uncorrelated. The reverse, that an uncorrelated signal is independent does not hold in general.

### Independence versus Orthogonality

In geometry, two vectors are said to be [orthogonal](https://en.wikipedia.org/wiki/Orthogonality) if their dot product equals zero. This definition is frequently applied to finite-length random signals by interpreting them as vectors. The relation between independence, correlatedness and orthogonality is derived in the following.

Let's assume two continuous-amplitude real-valued jointly wide-sense ergodic random signals $x_N[k]$ and $y_M[k]$ with finite lengths $N$ and $M$, respectively. The CCF $\varphi_{xy}[\kappa]$ between both can be reformulated as follows

\begin{equation}
\begin{split}
\varphi_{xy}[\kappa] &= \frac{1}{N} \sum_{k=0}^{N-1} x_N[k] \cdot y_M[k-\kappa] \\
&= \frac{1}{N} < \mathbf{x}_N, \mathbf{y}_M[\kappa] >
\end{split}
\end{equation}

where $<\cdot, \cdot>$ denotes the [dot product](https://en.wikipedia.org/wiki/Dot_product). The $(N+2M-2) \times 1$ vector $\mathbf{x}_N$ is defined as

$$\mathbf{x}_N = \left[ \mathbf{0}^T_{(M-1) \times 1}, x[0], x[1], \dots, x[N-1], \mathbf{0}^T_{(M-1) \times 1} \right]^T$$

where $\mathbf{0}_{(M-1) \times 1}$ denotes the zero vector of length $M-1$. The $(N+2M-2) \times 1$ vector $\mathbf{y}_M[\kappa]$ is defined as

$$\mathbf{y}_M = \left[ \mathbf{0}^T_{\kappa \times 1}, y[0], y[1], \dots, y[M-1], \mathbf{0}^T_{(N+M-2-\kappa) \times 1}  \right]^T$$

It follows from above definition of orthogonality that two finite-length random signals are orthogonal if their CCF is zero. This implies that at least one of the two signals has to be mean free. It can be concluded further that two independent random signals are also orthogonal and uncorrelated if at least one of them is mean free. The reverse, that orthogonal signals are independent, does not hold in general.

The concept of orthogonality can also be extended to one random signal by setting $\mathbf{y}_M[\kappa] = \mathbf{x}_N[\kappa]$. Since a random signal cannot be orthogonal to itself for $\kappa = 0$, the definition of orthogonality has to be extended for this case. According to the ACF of a mean-free uncorrelated random signal $x[k]$, self-orthogonality may be defined as

\begin{equation}
\frac{1}{N} < \mathbf{x}_N, \mathbf{x}_N[\kappa] > =
\begin{cases}
\sigma_x^2 & \text{for } \kappa = 0 \\
0 & \text{for } \kappa \neq 0
\end{cases}
\end{equation}

An independent random signal is also orthogonal if it is zero-mean. The reverse, that an orthogonal signal is independent does not hold in general.

#### Example - Computation of cross-correlation by dot product

This example illustrates the computation of the CCF by the dot product. First, a function is defined which computes the CCF by means of the dot product

```python
def ccf_by_dotprod(x, y):
    N = len(x)
    M = len(y)
    xN = np.concatenate((np.zeros(M-1), x, np.zeros(M-1)))
    yM = np.concatenate((y, np.zeros(N+M-2)))

    return np.fromiter([np.dot(xN, np.roll(yM, kappa)) for kappa in range(N+M-1)], float)
```

Now the CCF is computed using different methods: computation by the dot product and by the built-in correlation function. The CCF is plotted for the computation by the dot product, as well as the difference (magnitude) between both methods. The resulting difference is in the typical expected range due to numerical inaccuracies.

```python
N = 32  # length of signals

# generate signals
np.random.seed(1)
x = np.random.normal(size=N)
y = np.convolve(x, [1, .5, .3, .7, .3], mode='same')

# compute CCF
ccf1 = 1/N * np.correlate(x, y, mode='full')
ccf2 = 1/N * ccf_by_dotprod(x, y)
kappa = np.arange(-N+1, N)

# plot results
plt.figure(figsize=(10, 4))

plt.subplot(121)
plt.stem(kappa, ccf1)
plt.xlabel('$\kappa$')
plt.ylabel(r'$\varphi_{xy}[\kappa]$')
plt.title('CCF by dot product')
plt.grid()

plt.subplot(122)
plt.stem(kappa, np.abs(ccf1-ccf2))
plt.xlabel('$\kappa$')
plt.title('Difference (magnitude)')
plt.tight_layout()
```


# PHY321: Classical Mechanics 1

**First midterm project, due Wednesday March 12**

Date: **An attempt at a solution**

### Part 1, Particle in a one-dimensional  potential

We consider a particle (for example an atom) of mass $m$ moving in a one-dimensional potential,

$$
V(x)=\frac{V_0}{d^4}\left(x^4-2x^2d^2+d^4\right).
$$

We will assume all other forces on the particle are small in
comparison, and neglect them in our model.  The parameters $V_0$ and
$d$ are known constants.

1. (5pt) Plot the potential and find the  equilibrium points (stable and unstable) by requiring that the first derivative of the potential is zero. Make an energy diagram (see for example Malthe-Sørenssen chapter 11.3) and mark the equilibrium points on the diagram and characterize their stability. The position of the particle is $x$.

We have chosen values $d=1$ and $V_0=4$.
The following Python code gives a plot of potential

```python
%matplotlib inline

# Common imports
import numpy as np
from math import *
import matplotlib.pyplot as plt

Deltax = 0.01
#set up arrays
xinitial = -2.0
xfinal = 2.0
n = ceil((xfinal-xinitial)/Deltax)
x = np.zeros(n)
for i in range(n):
    x[i] = xinitial+i*Deltax
V = np.zeros(n)
# Initial conditions as compact 2-dimensional arrays
d = 1.0
V0 = 4.0
V = (V0/d**4)*(x**4-2*x*x*d*d+d**4)
# Plot potential as function of position
fig, ax = plt.subplots()
#ax.set_xlim(0, tfinal)
ax.set_xlabel('x')
ax.set_ylabel('V')
ax.plot(x, V)
fig.tight_layout()
plt.show()
```

When we take the derivative of the potential we have

$$
\frac{dV}{dx} = \frac{V_0}{d^4}\left(4x^3-4xd^2\right).
$$

We see that we have minima or maxima when the derivative is zero. This leads to

$$
\frac{V_0}{d^4}\left(4x^3-4xd^2\right)=0,
$$

which gives us as solutions $x=0$ and $x=\pm d$.

Taking the second derivative we have that

$$
\frac{d^2V}{dx^2} = \frac{V_0}{d^4}\left(12x^2-4d^2\right).
$$

If we use $x=\pm d$ we find that the second derivative is given by

$$
\frac{d^2V}{dx^2}\vert_{x=\pm d} = \frac{8V_0}{d^2},
$$

and assuming $V_0 > 0$, these points corresponds to stable equilibrium
points since the the second derivative is posit ive, meaning that
$x=\pm d$ are minima. For $x=0$ we get

$$
\frac{d^2V}{dx^2}\vert_{x=0} = -\frac{4V_0}{d^2},
$$

and our point represents a maximum and we $x=0$ is an unstable point.

1. (5pt) Choose two different energies that give two distinct types of motions, draw them into the energy diagram, and describe the motion in each case.

We select an energy where the particle starts at a potential less then
$V_0$. Therefore, it oscillates in its starting well.

For the second energy, the particle starts at a potential greater than
$V_0$. Therefore, it oscillates through one, goes over the hump at $x=0$
and then oscillates the second well up to the same y-coordinate as
the starting point. Then, it oscillates back to the original point.

1. (5pt) If the particle  starts at rest at $x=2d$, what is the velocity of the particle at the point $x=d$?

Below we show that the above potential fulfills the requirements about
being an energy conserving potential. Since energy is conserved, we
must have that the sum of potential and kinetic energies in the two
positions has to be conserved. We have At $x=2d$ we have that the
velocity $v_{2d}=0$ (we skip units) and at $x=d$ we have the unknown
velocity $v_d$. Inserting the values for the kinetic and potential
energies we get

$$
E_{2d}=0+V(x=2d)=9V_0=E_d=\frac{1}{2}mv^2_d+V(x=d)=\frac{1}{2}mv^2_d+0,
$$

which gives us

$$
v_d=\pm 3\sqrt{\frac{2V_0}{m}}.
$$

1. (5pt) If the particle  starts at $x=d$ with velocity $v_0$, how large must $v_0$ be for the  particle to reach the point $x=−d$?

At $x=d$ the potential energy is zero. In order to reach the othr side
of the well, the particle must have a kinetic energy which can
overcome the potential energy at the hump at $x=0$. There the
potential is $V_0$. Assuming that the velocity is zero there,
conservation of energy gives us

$$
E_{d}=\frac{1}{2}mv^2_d+=E_0=0+V_0,
$$

meaning we have

$$
v_d=\pm\sqrt{\frac{2V_0}{m}}.
$$

1. (5pt) Use the above potential to set up the total forces acting on the particle.  Find the acceleration acting on the particle. Is this a conservative force? Calculate also the **curl** of the force  $\boldsymbol{\nabla}\times \boldsymbol{F}$ in order to validate your conclusion.

The potential depends only on position and the **curl** of the force is
zero. The latter is easily seen since the components of the **curl** in
the $x$-, $y$- and $z$-z directions are, respectively,

$$
\frac{\partial F_y}{\partial z} -\frac{\partial F_z}{\partial y} =0,
$$

and

$$
\frac{\partial F_x}{\partial z} -\frac{\partial F_z}{\partial x} =0,
$$

and

$$
\frac{\partial F_x}{\partial y} -\frac{\partial F_y}{\partial x} =0,
$$

which means the **curl** of the force is zero.

1. (5pt) Are linear momentum and angular momentum conserved? You need to show this by calculating the quantities.

Because there only exists an external force, linear momentum is not
conserved.  The time-derivative of the momentum $\boldsymbol{p}$ is given by

$$
\frac{d\boldsymbol{p}}{dt}=\boldsymbol{F},
$$

and since the problem is a one-dimensional we have

$$
\frac{dp}{dt}=F=-\frac{dV(x)}{dx}=-\frac{V_0}{d^4}\left(12x^2-4d^2\right).
$$

Linear momentum is thus not conserved. Angular momentum is defined as

$$
\boldsymbol{l} = \boldsymbol{r}\times \boldsymbol{p},
$$

and its time derivative is

$$
\frac{d\boldsymbol{l}}{dt} = \boldsymbol{r}\times \frac{d\boldsymbol{p}}{dt}=\boldsymbol{r} \times \boldsymbol{F},
$$

which gives us the following $x$, $y$ and $z$ components of the derivative of angular momentum
(and since we are in one dimension only we have $y=z=0$ and $F_y=F_z=0$)

$$
F_yz -F_zy =0,
$$

and

$$
F_xz -F_zx =0,
$$

and

$$
F_xy -F_yx =0.
$$

Angular momentum is thus conserved.

1. (10pt) Write a numerical algorithm to find the position and velocity of the particle at a time $t+\Delta t$ given the position and velocity at a time $t$. Here you can use either the standard forward Euler, or the Euler-Cromer or the Velocity Verlet algorithms.   You need to justify your choice here (hint: consider energy conservation).

Since the force conserves energy, using either the Euler-Cromer or the
Velocity Verlet method, preserves the energy as function of time. The
Velocity Verlet method has also a better mathematical truncation error
$O(\Delta t^3)$, to be compared with $O(\Delta t^2)$ in the
Euler-Cromer method. The main difference is the new evaluation of the
acceleration at a time $t+1$. Thus, the Velocity Verlet method is the
preferred one here. The code examples in part 1 and part 2 focus on this latter method.

Setting up such a code has been discussed during the lectures and we refer to the slides from week 7 and week 8 at
<" and URL:>", respectively.

1. (10pt) Use now your program to find the position of the particle as function of time from $t=0$ to $t=30$ s using a mass  $m=1.0$ kg, the parameter $V_0=1$ J and $d=0.1$ m. Make a plot of three distinct positions with initial conditions $x_0=d$ and $v_0=0.5$ m/s, $x_0=d$ and $v_0=1.5$ m/s, and $x_0=d$ and $v_0=2.5$ m/s. Plot also the velocity.  Perform calculations with and without the term $x^4$ in the potential. Do you see a difference? Compare and discuss the results obtained with the Velocity Verlet algorithm and the Euler-Cromer algorithm.

2. (10pt) Describe the behavior of the particle for the three initial conditions  and sketch the motion in an energy diagram. Is energy conserved in your simulations?
    We also see that the particle exhibits an oscillatory motion in all

cases. In the first case $v_0=0.5$m/s, it oscillates around one
potential well around a stable equilibrium point at $x=d$. For the
other two cases ($v_0=$1.5m/s,2.5m/s), the particle exhibits an
periodic oscillatory motion around the unstable equilbrium point and
across both potential wells at $x=\pm d$. And the energy is
conserved in all three initial cases.

### Part 2, two dimensions

We move then to two dimensions. Our particle/object interacts with a surface potential given by

$$
V(r)=\frac{V_0}{d^4}\left(r^4-2r^2d^2+d^4\right),
$$

where $r=\sqrt{x^2+y^2}$ is the distance to the origin.

1. (5pt) Show that the acceleration is now $\boldsymbol{a}=-\frac{4V_0}{md^4}\left(r^3-rd^2\right)\frac{\boldsymbol{r}}{r}$.

We move then to two dimensions. Our particle/object interacts with a
surface potential given by

$$
V(r)=\frac{V_0}{d^4}\left(r^4-2r^2d^2+d^4\right),
$$

where $r=\sqrt{x^2+y^2}$ is the distance to the origin.

The gradient of $r$ is thus

$$
\boldsymbol{\nabla} r=\frac{\partial r}{\partial x}\boldsymbol{e}_1+\frac{\partial r}{\partial y}\boldsymbol{e}_2,
$$

which becomes

$$
\boldsymbol{\nabla} r=\frac{x}{\sqrt{x^2+y^2}}\boldsymbol{e}_1+\frac{y}{\sqrt{x^2+y^2}}\boldsymbol{e}_2,
$$

or

$$
\boldsymbol{\nabla} r=\frac{\boldsymbol{r}}{r}.
$$

The force is given by the negative gradient of the potential, that is

$$
\boldsymbol{F}=-\boldsymbol{\nabla}V(\boldsymbol{r})=-\boldsymbol{\nabla}\left[\frac{V_0}{d^4}\left(r^4-2r^2d^2+d^4\right)\right],
$$

and using the chain rule and the above gradient of $r$, we have

$$
\boldsymbol{F}=-\frac{V_0}{d^4}\left(4r^3-4rd^2\right)\frac{\boldsymbol{r}}{r},
$$

and dividing by the mass $m$ we find the acceleration

$$
\boldsymbol{a}=-\frac{V_0}{md^4}\left(4r^3-4rd^2\right)\frac{\boldsymbol{r}}{r},
$$

1. (10pt) Rewrite your program to find the velocity and position of the atom using the new expression for the force $\boldsymbol{F}$. Use vectorized expressions in your code as you did in homework 6 for the Earth-Sun system. See eventually the code from the [lectures](https://mhjensen.github.io/Physics321/doc/pub/energyconserv/html/energyconserv.html).  We recommend to revisit the Earth-Sun problem from homework 6 since it has several similarities with the problem here.

2. (10pt) Plot the motion of a particle starting in $\boldsymbol{r}_0=(d,0)$ from $t=0$ s to $t=20$ s for the initial velocities $\boldsymbol{v}_0=(0,0.5)$ m/s, $\boldsymbol{v}_0=(0,1)$ m/s, and $\boldsymbol{v}_0=(0,1.5)$ m/s. The parameters $d$ and $V_0$ are as before.

3. (5pt) Is energy conserved?

4. (10pt) Can you choose initial conditions $r_0$ and $v_0$ in such a manner that the particle moves in a circular orbit with a constant radius? If so, what initial conditions are those? Plot the motion for these conditions.

From the centripetal force, we have (for a velocity perpendicular to the
radius vector)

$$
\begin{equation}
\frac{v^2}{r}\frac{\boldsymbol{r}}{r}  = \frac{4 V_0}{m d^4}(r^3 - r d^2) \frac{\boldsymbol{r}}{r}
\label{_auto1} \tag{1}
\end{equation}
$$

$$
\begin{equation}  v^2  = \frac{4 V_0}{m d^4}(r^4 - r^2d^2)
\label{_auto2} \tag{2}
\end{equation}
$$

$$
\begin{equation}  v_0  = \sqrt{\frac{4 V_0}{m d^4}(r_0^4 - r_0^2d^2)} \text{ for } r_0 \neq d
\label{_auto3} \tag{3}
\end{equation}
$$


```python
%pylab inline
import pandas as pd

import numpy as np
from __future__ import division
import itertools

import matplotlib.pyplot as plt
import seaborn as sns

import logging
logger = logging.getLogger()
```

    Populating the interactive namespace from numpy and matplotlib

15 Dynamic Programming
============

for optimization problems:

1. divide-and-conquer
   partition into **disjoint** subproblems, solve recursively, and then combine.

2. dynamic programming
   subploems **overlap**
   developing by four steps:
   1. Characterize the **structure** of an optimal solution.
   2. **Recursively** define the value of an optimal solution.
   3. Compute the value of an optimal solution, typically in a **bottom-up** fashion.
   4. [opt] Construct an optimal solution from computed information.

### 15.1 Rod cutting

##### rod-cutting problem
Given:
1. a rod of length $n$ inches.
2. a table of prices $p_i$ for $i = 1, 2, \cdots, n$

Problem:
determine the __maximum revenue__ $r_n$ obtainable by cutting up the rod and selling the pieces.

Solution:
$$r_n = max_{1 \leq i \leq n} (p_i + r_{n-i})$$

Implementation:
1. recursive top-down
2. dynamic programming
    + **optimal substructure**: optimal solutions to a problem incorporate optimal solutions to related subproblems.
    + uses additional memory to save computation time
    + two ways:
        1. top-down with memorization
            + it “remembers” what results it has computed previously.
        2. bottom-up method.
            + We sort the subproblems by size and solve them in size order, smallest first.
            + The bottom-up approach often has much better constant factors, since it has less overhead for procedure calls.
    + **subproblems graphs**, as shown in Fig (15.4).
        - A dynamic-programming approach runs in **polynomial** time when **the number of distinct subproblems** involved is **polynomial** in the input size and we can solve each such subproblem in **polynomial** time.
        - namely, the running time of dynamic programming is **linear** in the number of vertices and edges.
    + **Reconstructing a solution**:
      We can extend the dynamic-programming approach to record not only the optimal value computed for each subproblem, but also a choice that led to the optimal value. With this information, we can readily print an optimal solution.
￼

```python
price = np.array([1, 5, 8, 9, 10, 17, 17, 20, 24, 30])

print('length i: {}'.format(range(len(price))))
print('price p_i: {}'.format(price))

def bottom_up_cut_rod(p_, n):
    """Solve cut_rod problem by bottom-up dynamic programming.

    """
    assert (len(p_) >= n), 'n should be less than the max length {}'.format(len(p))

    p = np.insert(p_, 0, 0) #set price_0 = 0

    r = np.zeros(n+1)

    for k in range(1, n+1):
        q = -np.inf
        for m in range(1, k+1):
            q = max(q, p[m]+r[k-m])
        r[k] = q

    return r[-1]

print('max price: {}'.format([bottom_up_cut_rod(price, n) for n in range(1, len(price)+1)]))
```

    length i: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    price p_i: [ 1  5  8  9 10 17 17 20 24 30]
    max price: [1.0, 5.0, 8.0, 10.0, 13.0, 17.0, 18.0, 22.0, 25.0, 30.0]

### 15.2 Martrix-chain multiplication

##### matrix-chain multiplication problem
**given** a chain $<A_1, A_2, \cdots, A_n>$ of $n$ matrices, where for $i = 1, 2, \cdots, n$, matrix $A_i$ has dimension $p_{i-1} \times p_i$,
**problem**: fully parenthesize the product $A_1 A_2 \cdots A_n$ in a way that minimizes the number of scalar multiplicatons.

**solutions**:

1. exhaustively checking all possible parenthesizations. $\Omega(2^n)$
    \begin{equation}
        P(n) = \begin{cases}
            1 & \quad \text{if } n = 1 \\
            \sum_{k=1}^{n-1} P(k) P(n-k) & \quad \text{if } n \geq 2 \\
        \end{cases}
    \end{equation}

2. Applying dynamic programming. $\Omega (n^3)$
    define $A_{i \dotso j} = A_i A_{i+1} \dotsm A_j$

    1. The structure of an optimal parenthesization.
       suppose $A_{i \dotso k} A_{k \dotso j}$ is the optimal solution of $<A_i, \dotsm, A_j>$, where $i \leq k < j$.
       then $A_{i \dotso k}$ must be the optimal solution of $<A_i, \dotsm, A_k>$, as well as $A_{k \dots j}$.

       Proof:
       if existing $Cost(\hat{A}_{i \dotso k}) < Cost(A_{i \dotso k})$,
       then $\hat{A}_{i \dotso k} A_{k \dotso j}$ should be the optimal solution ==> contradiction.

    2. A recursive solution.
       Let $m[i,j]$ be the minimum number of scalar multiplications needed to compute the matrix $A_{i \dotso j}$.

       \begin{equation}
           m[i,j] = \begin{cases}
               0 & \quad \text{if } i = j \\
               min_{i \leq k < j} {m[i,k] + m[k+1,j] + p_{i-1} p_k p_j} & \quad \text{if } i < j
           \end{cases}
       \end{equation}

    3. Computing the optimal costs.
       $m[i,j]$ depends on all $m[i,k], m[k+1,j]$.
       hence, the algorithm should fill in the talbe $m$ in a manner that corresponds to solving the parenthesizaiton problem on matrix chains of incresing lenghth.

    4. Constructing an optimal solution.
       construct table $s$ whose rows is $1, \dotsm, n-1$ and columns is $2, \dotsm, n$.
       Each entry $s[i,j]$ records the optimal solution $k$ for $A_{i \dotso j}$.

```python
#todo
```

### 15.3 Elements of dynamic programming

Two keys: optimal substructure and overlapping subproblems.

#### Optimal substructure
##### common pattern
1. A solution to the problem consists of making a choice.

2. Suppose: for a given problem, you are given the choice

3. Given the choice, you determine which subproblems ensue and how to best characterize the resulting space of subproblems.

4. By using "cut-and-paste" technique, you show that the solutions to the subproblems used within an optimal solution to the problem must themselves be optimal.

To characterize the space of subproblems, a good rule of thumb says to try to keep the space **as simple as possible** and then **expand it as necessary**.

Optimal substructure **varies** across problem domains in two ways:

1. how many subproblems an optimal solution to the original problem use.

2. how many choices we have in detering which subproblem(s) to use in an optimal solution.

the **running time** of a dynamic-programming algorithm depends on the product of two factors:

1. the number of subproblems overall.

2. how many choices we look at for each subproblem.

Dynamic programming often uses optimal substructure in a **bottom-up fashion**.

The **cost** of the problem solution is usually the subproblem costs plus a cost that is directly attributable to the choice itself.

##### subtleites
Be careful NOT to assume that the optimal substructure applies when it does not.
eg: Unweighted shortest path VS Unweighted longest simple path.

subproblems are **independent**:
The solution to one subproblem does not affect the solution to another subproblem of the same problem.
in another word, **the subproblems do not share resources**.

#### Overlapping subproblems

#### Reconstructing an optimal solution
store which choice we made in each subproblem in a table.

#### Memoization
maintains an entry in a table for the solution to each subproblem.

In general practice, we prefer to use bottom-up rather than top-down memoized algorithm. While if some subproblems need not be solved at all, the memoized solution has the advantage.

### 15.4 Longest common subsequence

Steps:

1. Characterizing the optimal substructure:
   Let $X = <x_1, x_2, \dotsc, x_m>$ and $Y = <y_1, y_2, \dotsc, y_n>$ be sequences, and let $Z = <z_1, z_2, \dotsc, z_k>$ be any LCS of $X$ and $Y$.

   1. If $x_m = y_n$, then $z_k = x_m = y_n$, and $Z_{k-1}$ is an LCS of $X_{m-1}$ and $Y_{n-1}$.

   2. If $X_m \neq y_n$, then $z_k \neq x_m$ implies that $Z$ is an LCS of $X_{m-1}$ and $Y$.

   3. If $x_m \neq y_n$, then $z_k \neq y_n$ implies that $Z$ is an LCS of $X$ and $Y_{n-1}$.

2. A recursive solution.

   \begin{equation}
       c[i,j] = \begin{cases}
           0 \quad & \text{if } i = 0 \text{ or } j = 0 \\
           c[i-1,j-1] + 1 \quad & \text{if } i, j > 0 \text{ and } x_i  = y_j \\
           max(c[i,j-1], c[i-1,j]) \quad & \text{if } i, j > 0 \text{ and } x_i \neq y_j
       \end{cases}
   \end{equation}

3. Computing the length of an LCS.
   code.

4. Constructing.
   $b[i,j]$ points to the table entry corresponding to the optimal subproblem solution chosen when computing $c[i,j]$.

##### improving the code
1. eliminate the $b$ table.

2. leave only two rows of table $c$ at a time.

```python
#todo
```

### 15.5 Optimal binary search trees

**Given**:
a sequence $K = <k_1, k_2, \dotsc, k_n>$ of $n$ distinct keys in sorted order. For each $k_i$, we have a probability $p_i$ that a serch will be for $k_i$. And we also have $n+1$ "dummy keys" $<d_0, d_1, \dotsc, d_n>$ representing values not in $K$, and the probability $q_i$ for each $d_i$.

$$\sum_{i=1}^{n} p_i + \sum_{i=0}^{n} q_i = 1 $$

We want to build a search trees $T$ for $K$, and define the expected cost of a search in $T$ is:
\begin{align}
    E[cost] &= \sum_{i=1}^n (depth_T(k_i) + 1) p_i  +  \sum_{i=0}^n (depth_T(d_i) + 1) q_i \\
            &= 1 + \sum_{i=1}^n depth_T(k_i) p_i + \sum_{i=0}^n depth_T(d_i) q_i
\end{align}

**Problem**:
We wish the expected cost of $T$ is the smallest.

#### steps
1. Optimal substructure
   If an optimal binary search tree $T$ has a subtree $T'$ containing keys $k_i, \dotsc, k_j$, then this subtree $T'$ must be optimal as well for the subproblem with keys $k_i, \dotsc, k_j$ and dummy keys $d_{i-1}, \dotsc, d_j$.

2. A recursive solution

   define: $w[i,j] = \sum_{l=i}^{j} p_l + \sum_{l=i-1}^{j} q_l$

   \begin{equation}
       e[i,j] = \begin{cases}
           q_{i-1} \quad & \text{if } j = i - 1 \\
           min_{i \leq r \leq j} \{e[i,r-1] + e[r+1,j] + w[i,j]\}  \quad & \text{if } i \leq j
       \end{cases}
   \end{equation}

3. Computing
   $$w[i,j] = w[i,j-1] + p_j + q_j$$

```python
#todo
```


# Exercises

We have prepared five exercises in this chapter:

1. Modify the HCM code to work for three groups. This exercise can be divded into four tasks:
    - modify the parameters,
    - modify the calculate_u function,
    - execute the clustering,
    - plot the results.
2. For density clustering, plot the feature space with all element marked with different color, depending on the cluster that it's assigned to. You should do the following tasks:
    - fill the get_color method,
    - fill the plot code.
3. Build a method that plot baed on dendrograms_history and pydot, a dendrogram for the divisive clustering method. You should base on agglomerative method, but keep in mind that it works top-down instead of bottom-up. This exercise need just one function to be implemented:
    - show_tree_divisive.
    You should loop over the dendrogram_history variable and loop over childs.
4. Implement the $s_{2}$ metric
5. Draw the borders between clusters in the output image (for 5.0 grade)

## Libraries

To solve the exercises, we need the following libraries to load in the first place.

```python
import numpy
import random
import numpy as np
import pandas as pd
from math import sqrt

import matplotlib.image as img
from PIL import Image

from matplotlib import pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

from IPython.display import Image
```

## Exercise 1: Modify the HCM code to work for three groups

The obvious part is the variable ```groups```, but the most changes needs to be done here:

```python
%store -r data_set

### change here:
groups = 3

error_margin = 0.01
m=2
assignation=np.zeros((len(data_set),groups))

centers = np.array([[0.01229673, 0.25183492],
       [0.3689626 , 0.61904127],
       [0.95732769, 0.45059586]])

centers = np.array([[0.01229673, 0.25183492],
       [0.3689626 , 0.61904127],
       [0.95732769, 0.45059586]])

def calculate_distance(x,v):
    return sqrt((x[0]-v[0])**2+(x[1]-v[1])**2)

def calculate_new_centers(u):
    new_centers=[]
    for c in range(groups):
        u_x_vector=np.zeros(2)
        u_scalar=0.0
        for i in range(len(data_set)):
            u_scalar = u_scalar+(u[i][c]**m)
            u_x_vector=np.add(u_x_vector,np.multiply(u[i][c]**m,data_set[i]))
        new_centers.append(np.divide(u_x_vector,u_scalar))
    return new_centers

def calculate_differences(new_assignation, assignation):
    return np.sum(np.abs(np.subtract(assignation,new_assignation)))

def cluster_hcm(assignation,centers):
    difference_limit_not_achieved=True
    new_centers = centers
    iter=0
    while difference_limit_not_achieved:
        new_assignation=[]
        for i in range(len(data_set)):
            new_assignation.append(calculate_u_three(data_set[i]))
        new_centers = calculate_new_centers(new_assignation)
        if iter>0:
            if calculate_differences(new_assignation, assignation) < error_margin:
                difference_limit_not_achieved=False
        assignation=new_assignation
        iter=iter+1
    return new_assignation, new_centers
```

### Modify the ``calculate_u`` function

Fill the gap below to make the function working for more groups than two. The goal here is to calculate the distance between ``x`` and the center of a given group and append the value to ``minimal_distance``.

```python
def calculate_u_three(x):
    u_array = np.zeros(groups)
    minimal_distance = []
    for group in range(groups):
        minimal_distance.append(calculate_distance(x, centers[group]))
    min_group_id = np.argmin(minimal_distance)
    u_array[min_group_id] = 1
    return u_array
```

### Execute the clustering

As in the previous example we need to cluster it.

```python
new_assignation_hcm3, new_centers_hcm3 = cluster_hcm(assignation, centers)
pd.DataFrame(new_centers_hcm3)
```

      0
      1

      0
      0.078585
      0.178323

      1
      0.496071
      0.692516

      2
      0.958743
      0.955892

### Plot the results

```python
red = data_set[np.where(np.array(new_assignation_hcm3)[:,0]==1)]
blue = data_set[np.where(np.array(new_assignation_hcm3)[:,1]==1)]
green = data_set[np.where(np.array(new_assignation_hcm3)[:,2]==1)]

fig, ax = plt.subplots()

ax.scatter(blue[:,0],blue[:,1],c='blue')
ax.scatter(red[:,0],red[:,1],c='red')
ax.scatter(green[:,0],green[:,1],c='green')
ax.scatter(np.array(new_centers_hcm3)[:,0],np.array(new_centers_hcm3)[:,1],c='black')
ax.set(xlabel='Seats count', ylabel='Distance range (km)',
       title='Aircrafts (clusters)')
ax.grid()
plt.show()
```

## Exercise 2: Plot the density clusters

Use the code below to plot the results. You can play with the max_distance variable to get more or less groups.

```python
%store -r new_assignation_density
%store -r data_set
```

### Fill the ``get_group_objects`` method

Only one line needs to be updated. The ``get_group_objects`` function should return the objects of a given group.

```python
colors = ['red','blue','green','orange','black','yellow']

def get_group_objects(color_id):
    return data_set[new_assignation_density == color_id]
```

### Fill the plot code

If done properly the code below should return a plot of two clusters and the noise.

```python
colors = ['red','blue','green','orange','black','yellow']

fig, ax = plt.subplots()

assigned_groups = new_assignation_density
for group in np.unique(assigned_groups):
    small_set = get_group_objects(group)
    ax.scatter(small_set[:,0],small_set[:,1],c=colors.pop(0))
    for circle in small_set:
        circle1 = plt.Circle((circle[0],circle[1]), 0.25, color='green', fill=False)
        ax.add_artist(circle1)

ax.set(xlabel='Seats count', ylabel='Distance range (km)',title='Aircrafts (clusters)')
ax.grid()
plt.show()
```

## Exercise 3: Build a dendrogram using dendrograms_history and pydot (done)

In this exercise we gonna use the variable dendrograms_history and pydot. Below we restore the variable and initialize the dendrogram graph.

```python
%store -r dendrogram_hist_diana
```

### Fill ``show_tree_divisive`` function

The function show_tree_divisive goes through each child node and build and edge between.

```python
import pydot
tree = pydot.Dot(graph_type='graph')
def show_tree_divisive():
    for item in dendrogram_hist_diana:
        for child in item[0]["childs"]:
            new_edge = pydot.Edge(str(item[0]["acesor"]),str(child))
            tree.add_edge(new_edge)
    tree.write('tree_diana.png',format='png')

show_tree_divisive()
```

Take a look if you did it properly:

```python
from IPython.display import Image
Image(filename='tree_diana.png')
```

## Exercise 4: Implement the $s_{2}$ metric

The $s_{2}$ metric gives a better understanding of the distances between centers.

\begin{equation}
 s_{2}(c_{i},c_{j})=d(c_{i},c_{j}).
\end{equation}

Let's restore the centers from HCM grouped by two and initialize the values for three groups as below.

```python
%store -r new_centers_hcm

new_centers_hcm = np.array(new_centers_hcm)
new_centers_hcm3 = np.array([[0.42239686, 0.38503185],[0.07858546, 0.17832272],[0.82907662, 0.97059448]])
```

Measure the distance between each center.

```python
def calculate_s_2(centers):
    s2 = []
    for center_1 in range(len(centers)):
        for center_2 in range(len(centers)):
            if center_1 == center_2:
                break
            s2.append(calculate_distance(centers[center_1], centers[center_2]))
    return s2
```

## Exercise 5: Modify the output image with borders between clusters

We use the ``Segmentation`` class as in previous example.

```python
class Segmentation:

    def __init__(self, feature_matrix, groups):
        self.__data_set = feature_matrix
        self.__groups = groups
        self.__space=[[0, 255], [0, 255], [0, 255]]
        self.__error_margin = 0.5
        self.assignation = np.zeros((len(self.__data_set), self.__groups))
        self.centers = []
        self.select_centers()

    def select_centers(self):
        if len(self.centers) == 0:
            iter=0
            while iter<self.__groups:
                self.centers.append(((random.randrange(0, 255)*1.0/255),
                                     (random.randrange(0, 255)*1.0/255),
                                     (random.randrange(0, 255)*1.0/255)))
                iter=iter+1

    def calculate_distance(self, x, v):
        return sqrt((x[0]-v[0])**2+(x[1]-v[1])**2+(x[2]-v[2])**2)

    def calculate_u(self, x, i):
        smallest_distance = float(self.calculate_distance(x, self.centers[0]))
        smallest_id = 0
        for i in range(1, self.__groups):
            distance = self.calculate_distance(x, self.centers[i])
            if distance < smallest_distance:
                smallest_id = i
                smallest_distance = distance
        distance = np.zeros(self.__groups)
        distance[smallest_id]=1
        return distance

    def calculate_new_centers(self, u):
        new_centers=[]
        for c in range(self.__groups):
            u_x_vector = np.zeros(len(self.centers[0]))
            u_scalar = 0
            for i in range(len(u)):
                u_scalar = u_scalar + u[i][c]
                u_x_vector = np.add(u_x_vector, np.multiply(u[i][c], self.__data_set[i]))
            new_centers.append(np.divide(u_x_vector,u_scalar))
        self.centers = new_centers

    def calculate_differences(self,new_assignation):
        diff=0
        for i in range(len(self.assignation)):
            for j in range(self.__groups):
                diff = diff + abs(float(new_assignation[i][j]) - float(self.assignation[i][j]))
        return diff

    def do_segmentation(self):
        difference_limit_not_achieved = True
        iter = 0
        while difference_limit_not_achieved:
            new_assignation = []
            for i in range(len(self.__data_set)):
                new_assignation.append(self.calculate_u(self.__data_set[i], iter))
            self.calculate_new_centers(new_assignation)

            if iter > 0:
                if self.calculate_differences(new_assignation) < self.__error_margin:
                    difference_limit_not_achieved=False
            self.assignation = new_assignation
            iter = iter + 1

    def get_results(self):
        return self.centers, self.assignation
```

### Change ``save_image`` method

Add an if statement in the code below. It should consider the change of ``current_pixel`` variable. Please keep in mind that there should be three states considered.

```python
from PIL import Image
class ImageConversion:

    def get_image_from_url(self, img_url):
        image = open(img_url,'rb')
        return img.imread(image)

    def get_unique_colours(self, image_matrix):
        feature_matrix = []
        for i in range(len(image_matrix)):
            for j in range(len(image_matrix[0])):
                feature_matrix.append(image_matrix[i, j])
        feature_matrix_np = np.array(feature_matrix)
        uniques, index = np.unique([str(i) for i in feature_matrix_np], return_index=True)
        return feature_matrix_np[index], feature_matrix

    def save_image(self, size, pixel_matrix, unique_matrix, assignation_matrix, colours, output):
        image_out = Image.new("RGB", size)
        pixels = []
        current_pixel = None ## added
        for i in range(len(pixel_matrix)):
            pixel_list = pixel_matrix[i].tolist()
            for j in range(len(unique_matrix)):
                if (pixel_list == unique_matrix[j].tolist()):
                    for k in range(len(colours)):
                        if assignation_matrix[j][k] == 1:
                            segmented_colours = [int(i) for i in (colours[k] * 255)]
                            if current_pixel is not None and current_pixel == segmented_colours:
                                pixels.append(tuple(segmented_colours))
                            else:
                                #print(segmented_colours)
                                current_pixel = segmented_colours
                                pixels.append((255, 255, 255))

        image_out.putdata(pixels)
        image_out.save(output)

    def do_segmentation(self):
        difference_limit_not_achieved = True
        iter = 0
        while difference_limit_not_achieved:
            new_assignation = []
            for i in range(len(self.__data_set)):
                new_assignation.append(self.calculate_u(self.__data_set[i], iter))
            self.calculate_new_centers(new_assignation)
            if iter > 0:
                if self.calculate_differences(new_assignation) < self.__error_margin:
                    difference_limit_not_achieved=False
            self.assignation = new_assignation
            iter = iter + 1

    def get_results(self):
        return self.centers, self.assignation
```

Execute segmentation without any changes:

```python
image_to_segment = "../images/logo_krakow.png"
image_converter = ImageConversion()
image_data = image_converter.get_image_from_url(image_to_segment)
unique_image_data, image_data_list = image_converter.get_unique_colours(image_data)

groups = 3

if image_data.shape[2] > 3:
    image_data = image_data[:,:,[0,1,2]]
    unique_image_data = unique_image_data[:,[0,1,2]]
    image_data_list = numpy.array(image_data_list)[:,[0,1,2]]

segmentation = Segmentation(unique_image_data, groups)
segmentation.do_segmentation()
centers, assignation_matrix = segmentation.get_results()

image_size = (232, 258)
image_converter.save_image(image_size, image_data_list, unique_image_data, assignation_matrix, centers, "../images/segmented.png")
```

    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:44: RuntimeWarning: invalid value encountered in true_divide

The image should have black broders between one and the other segment.

```python
from IPython.display import Image
Image("../images/segmented.png")
```

```python

```


#### _Speech Processing Labs 2020: Signals: Module 1_

```python
## Run this first!
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

import cmath
from math import floor
from matplotlib.animation import FuncAnimation
from IPython.display import HTML
plt.style.use('ggplot')

## some helper functions:
from dspMisc import *
```

# 4 The Discrete Fourier Transform

### Learning Outcomes
* Understand the DFT equation: what input does it take and what outputs does it produce.

### Need to know
* Topic Videos: Series Expansion, Fourier Analysis, Frequency Domain
* Vector operations: dot product
* [Digital Signals: Sampling sinusoids](./sp-m1-3-sampling-sinusoids.ipynb)

     Optional-ish:  This notebook goes through the DFT equation in quite a lot of detail - more than is strictly necessary for this course.  It's perfectly fine to just focus on the visualizations or to skip it for now.

That said, you might like to run the code anyway and look at the visualization of the different phasors (i.e. basis functions) that correspond to the different DFT outputs in the DFT work through section called [Generate Phasors](#genphasors), and the corresponding magnitude and phase response graphs for that example

Do you remember learning a bunch of math (trig, algebra, calculus) in high-school and thinking you'll never have to use this? While you might not use it directly, a lot of the technology you use everyday depends on it.  Pretty much all current speech technology depends on the Fourier Transform in some way! It has been (and continues to be) used to solve problems in many different fields from from image processing to quantum mechanics

In the previous notebook, you looked at how you can add 'pure' sinusoids together to form complicated waveforms. The Fourier Transform gives us a way of doing the opposite: taking sequence of measurements over time (like a speech wave) and decomposing it into sinusoids of different frequencies, magnitudes, and phases.

Since we are interested in digital signals (i.e. measuresment over discrete time steps), we'll need to use the **Discrete Fourier Transform**.  This makes it possible to actually do these calculations on a computer, but it does also lead to some unexpected side-effects.

Equation alert: If you're viewing this on github, please note that the equation rendering is not always perfect. You should view the notebooks through a jupyter notebook server for an accurate view.

## 4.1 The DFT equation

You can think of the DFT as a machine that takes $N$ amplitude samples over time and returns the frequency spectrum of the input signal (just like you when you view a spectral slice in praat).

You can treat this as a black box. But, now that we've gone through complex numbers, phasors, etc we can look at how the machine works and, more importantly, what it's limitations are.

* Input: $x[n]$, for $n=1..N$ (i.e. a time series of $N$ samples)
* Output: N complex numbers $\mapsto$ magnitude and phases of specific phasors:

$$
\begin{align}
DFT[k] &= \sum_{n=0}^{N-1} x[n] e^{-j \frac{2\pi n}{N} k} \\
DFT[k] &= \sum_{n=0}^{N-1} x[n]\big[\cos(\frac{2\pi n}{N} k) - j \sin(\frac{2\pi n}{N} k) \big]
\end{align}
$$

You'll usually see this written as $X[k]$ in signal processing textbooks, but we'll just say $DFT[k]$ to be a bit more transparent (and reduce the probability of typos!)

Notice the $-j$ in the phasor! This means as $n$ increases the 'clockhand' of the phasor is rotating clockwise!

## 4.2 The DFT Step-by-Step

But what does the DFT do for us? Very broadly, the DFT takes a series of $N$ values in time as input (e.g. waveform amplitudes over time for our sound waves) and outputs the **correlations** between the input and $N$ pure cosine waves with specific frequencies.  That is, it tells you how you would weight (and shift) these cosine waves so that you can add them all up and recover the original input.

_Let's break it down!_

### The DFT for $k=0$
Let's first look at the DFT for $k=0$:

$$
\begin{align}
DFT[0] = \sum_{n=0}^{N-1} x[n] e^{-j \frac{2\pi n}{N} 0}
\end{align}
$$

This is usually referred to as the **bias** term.  It tells us whether the input function is shifted up or down the amplitude axis (in a time v amplitude plot of the waveform).

### Exercise:

* Q: Why doesn't DFT[0] tell us anything about the frequency of the input?

```python

```

### The DFT for $k=1$:

Now we can start to look at the correlation between the input sequence and some phasors of different frequencies.  For $k=1$, we have:

$$ DFT[1] = \sum_{n=0}^{N-1} x[n] e^{-j \frac{2 \pi n}{N}}$$

Each of the $e^{j 2\pi n/N}$ terms in the sum is a step of $2\pi n/N$ radians around the circle.  That is, our $k=1$ phasor $e^{-j \frac{2 \pi}{N}n}$ makes completes 1 full circle every $N$ time steps.

Let's plot the the $k=1$ phasor for $N=16$:

```python
## plot complex numbers in the DFT[1] equation
N=16

## make an array of N steps: 0,...,N
nsteps = np.array(range(N))
tsteps = 2*np.pi*nsteps/N
Tmin = np.min(tsteps)
Tmax = np.max(tsteps)

## Corresponding complex numbers z[n] = e^{j 2 pi n/N}
zn_k1 = np.exp(-1j*2*np.pi*nsteps/N)

```

```python
## Plot the phasor corresponding to DFT[k=1]
X_k1, Y_k1, fig, phasor, iproj, rproj = plot_phasor_static(zn_k1, tsteps, plot_real=True, plot_imag=True)
```

```python
## Animate the phasor!
phasor1_vid = get_phasor_animation(X_k1, Y_k1, tsteps, phasor, iproj, rproj, fig)
phasor1_vid
```

What this animation shows is that for $k=1$, the $e^{-j\varphi}$ terms in the DFT represent $N$ evenly space samples around the unit circle aka one period of a sinusoid. So, The term $x[n] e^{-j2\pi n/N}$ represents multiplying the $n$th step around the unit circle with the $n$th input value.

### Exercise (optional)

We saw from Euler's formula previously that projecting the phasor (left fig) onto the imaginary (vertical) axis, would give us a sine wave, i.e. $\sin(t)$.  This isn't quite what we see above.  Instead the sinusoid on the right, looks like the mirror image of $\sin(t)$, i.e. $-\sin(t)$.  Why is this?

### Notes

```python

```

So, we can interpret the DFT as taking the dot product between the input and a cosine (and sine) wave of specific frequency.  That is, it tells us if the input sequence repeats itself  with a specific frequency.

### The dot product with the input $x[n]$
It's useful at this point to break the DFT equation into two parts using Euler's rule. This esssentially gives us two dot products (i.e. elementwise multiplication and summation). This allows us to calculate the real and imaginary components of the DFT output separately as a dot product with a cosine (the real part) and a dot product with a sine (the imaginary part):

$$
\begin{align}
DFT[1] &= \sum_{n=0}^{N-1} x[n] \big(\cos(2\pi n/N) -j\sin(2\pi n/N)\big)\\
&= \sum_{n=0}^{N-1} x[n] \cos(2\pi n/N) - j \sum_{n=0}^{N-1} x[n] \sin(2\pi n/N)
\end{align}
$$

### Example:
Let's look at what's happening in these sum terms using a simple sampled square wave with the same period as the $k=1$ phasor for input.  The following visualizes the second sum (i.e. the dot product with $\sin(2\pi n/N)$).

```python

## Make the background and plot the k=1 phasor
X_k1, Y_k1, fig, phasor, iproj, rproj = plot_phasor_static(zn_k1, tsteps, plot_real=True, plot_imag=True)

## Now, let's say our input x looks like this square wave:
xn = np.array([-1]*8 + [1]*8)

## Exercise: Try some other inputs later
#xn = np.array([-1, -1,-1,-1, 1, 1,1,1]*2)
#xn = np.array([-1]*4 + [1]*8 + [-1]*4)
#xn = np.array([1] + [-1, -1,-1, 1, 1,1,1]*2 + [-1])
#xn = zn_imag_k1
#xn = np.concatenate([Ys[np.arange(0, 16, 2)], Ys[np.arange(0, 16, 2)]])
#xn = np.array([1] + [-1, -1,-1, 1, 1,1,1]*2 + [-1])

print("x[n]:", xn)

## Plot the input x[n]:

# with respect to the imaginary component (top right)
iproj.scatter(tsteps, xn, color='C1')

# and the real component (bottom left)
rproj.scatter(xn, tsteps, color='C1')

## Do the elementwise multiplication of input (xn) and the k=1 phasor (zn_k1)
xn_times_zn = xn * zn_k1

## Get the real and imaginary parts
real_xn_zn = np.real(xn_times_zn)
imag_xn_zn = np.imag(xn_times_zn)

## Plot the imaginary parts = x[n]*sin(theta) (top right) in yellow
iproj.plot(tsteps, imag_xn_zn, color='C4')
iproj.scatter(tsteps, imag_xn_zn, color='C4', s=100, alpha=0.5)

## Plot the real parts = x[n]*cos(theta) (bottom left) in yello
rproj.plot(real_xn_zn, tsteps,  color='C4')
rproj.scatter(real_xn_zn, tsteps, color='C4', s=100, alpha=0.5)
```

In the plot above, you can see that:
* The grey dots show our samples from the $k=1$ phasor (top left), and projected imaginary component, i.e. the sine in the DFT sum (top right), and the projected real component, i.e. the cosine (bottom left)
* The blue dots show our input sequence (top right and bottom left)
* the yellow dots show the element wise multiplication of the phasor values and the input, projected on the imaginary and real axes.

Let's just look at the imaginary (sine) part of the $x[n] \cdot z_1[n]$ multiplication (i.e., `xn_times_zn`):
When we multiply the values in both sequences together, we can see that (1)  the values in the input and phasor don't exactly match, but (2) they are always the same sign. That is, the input and the sine component of the $k=1$ phasor are correlated to some extent. In this case, this means that the multiplication terms (in yellow) in this case are all positive. So, adding them all up will give us a positive term for the imaginary component of the DFT.

We can also see that, even though we basically chose this example to match the sine component, we do also get a positive correlation with the real (cos) component.

The following cell shows that you get the same result from doing the dot products with the real (cos) and imaginary (sin) parts of the phasor separately, or doing the dot product with the phasor and then projecting the real and imaginary parts.

```python
## The dot product: sum all the elementwise multiplications
dft_1_real = np.sum(real_xn_zn)
dft_1_imag = np.sum(imag_xn_zn)

print("* projection and then two separate products: DFT[k] = %f + i%f" %(dft_1_real, dft_1_imag))

## check these are the same!
dft_1 = np.sum(xn*zn_k1)
print("* one dot product and then projection: DFT[k] = %f + i%f" %(dft_1.real, dft_1.imag))
```

    * projection and then two separate products: DFT[k] = -2.000000 + i10.054679
    * one dot product and then projection: DFT[k] = -2.000000 + i10.054679

### Exercise: Change the input
What happens when we change the input `xn` so that:

* it has a different period
* exactly matches the sine component of the k=1 phasor
* is out of phase with the sine component of the k=1 phasor
* has a different magnitude
* something else ...

There are some commented out options that you can try in the cell above

### Notes

```python

```

### Example: Phase shifted input

Let's see what happens when our input matches the imaginary component of $k=1$ phasor but has it's phase shifted a bit.  Remember, we can shift the start point of our phasor by multiply the whole phasor sequence a complex number.

$$ \sin(\theta + \phi) = Imag(e^{j\phi}e^{j\theta}) = Imag(e^{j\theta+\phi})$$

```python
## Let's generate our DFT[1] phasor for an input of N=16 points:
N=16

## make an array of N steps: 0,...,N
nsteps = np.array(range(N))
tsteps = 2*np.pi*nsteps/N

## The k=1 phasor:
zn_k1 = np.exp(-1j*tsteps)
```

Now let's create an input that's the same as the sine component of the DFT[1] phasor, but shifted by $\pi/3$ radians.

```python
## Plot the DFT[1] phasor
X_k1, Y_k1, fig, phasor, iproj, rproj = plot_phasor_static(zn_k1, tsteps, plot_real=False, plot_imag=True)

## Now, for the input let's use a phase shifted version of the sine component of our phasor, zn (defined above)
# Remember that multiplying a complex number by e^{j theta} rotates it by theta degrees
# (anticlockwise if theta is positive)
# So to shift the whole sequence we just multiply everything in the sequence by the same complex number

## For this example, let's shift our -sin(2 pi n/N) by pi/3
zn_shift = np.exp(1j*np.pi/3) * zn_k1

## And take as input just the imaginary component of this shifted sine wave
xn = np.imag(zn_shift)

## Plot the phase shifted sine wave it in blue!
iproj.scatter(2*np.pi*nsteps/N, xn, color='C1')
iproj.plot(2*np.pi*nsteps/N, xn, color='C1')

```

In the plot above, you should see the input (in blue) is the same as the sine component of the $k=1$ phasor but *phase shifted*. By multiplying everything by $e^{j\pi/3}$ _delays_ the our $-sin(\theta)$ function by $\pi/3$ seconds (in this example we're assuming that it takes $\theta$ seconds to travel $\theta$ degrees around the phasor).

Now let's see how this effects the DFT output:

```python
 ## Elementwise multiplication of the input and the k=1 phasor
xn_times_zn = xn * zn_k1

## Add it all up to get the dot product
dft_1 = np.sum(xn_times_zn)

print("DFT[1] = %f + i%f" %(dft_1.real, dft_1.imag))
print("in polar coordinates: magnitude=%f, phase=%f" %cmath.polar(dft_1))

## Plot the sequence of multiplications (yellow)
iproj.plot(2*np.pi*nsteps/N, np.imag(xn_times_zn), color='C4')
iproj.scatter(2*np.pi*nsteps/N, np.imag(xn_times_zn), color='C4')
fig
```

The result of the DFT is 6.93 + j4, which in polar coordinates has a magnitude=8 and phase angle=0.52

* Non-zero magnitude means that the input has a frequency component that matches the frequency of the $k=1$ phasor
* Non-zero phase means that the input is like the $k=1$ phasor in frequency, but starting from a different starting angle.

### How does the detected phase angle relate to the phase we used to shift the input?

The magnitude of the DFT output is relatively straightforward to interpret.  The bigger the magnitude, the bigger the amplitude of this frequency component in the input.

When we convert the $DFT[1]$ output to polar coordinates we can, as before, interpret $DFT[1]$ as a scaling (magnitude) and rotation (phase) factor.  We can think of the phase as signalling a bigger amplitude of this frequency in the the input.   We can think of the phase angle as rotating the starting position of the DFT[1] phasor 'clock hand' by that angle. When we convert this to time vs amplitude, this essential means starting from a future point in the sinusoid for positive phase (or a past point for negative phase) relative to the direction the phasor is rotating.

###  DFT phase is relative to cosine!

An important thing to note is that the phase output of the DFT is relative to the **cosine** function with the same frequency as the phasor. This is why the phase value we calculated isn't actually the same as the angle we used to shift the input sequence ($\pi/3=1.047$ radians), since that input sequence was actually based on a sine function.

### Exercise (optional)

Show that our input above $-\sin(t-\pi/3)$ is the same as a cosine wave with the phase shift derived above $\pi/6$.
You'll need to use these trig identities:
$$ \cos(\alpha - \pi/2) = \sin(\alpha) $$
$$\sin(t+\pi) = -\sin(t)$$

**Note** We definitely won't be asking you do these sorts of calculations for assessment, but going through it will help consolidate your intutions about sinusoids and the relationship between cosine and sine waves.

### Notes

```python

```

Even if you don't do the exercise above, we can see that shifting a cosine function by the DFT output phase gives us the same sinusoid as our input.

That is:
$$ \cos(t + \pi/6) = -\sin(t-\pi/3)$$

just by plotting them:

```python
## Plot the input from above and the equivalent cos and sin function based on DFT phase output.
_, _, fig, phasor, iproj, rproj = plot_phasor_static(zn_k1, tsteps, plot_real=False, plot_imag=True)

#fig, phasor, iproj, rproj = create_phasor_iproj_bkg(Tmin, Tmax, ymax=1.5)
#phasor.scatter(zn_real_k1, zn_imag_k1)

## Our input (C1=blue)
tsteps = 2*np.pi*nsteps/N
iproj.scatter(tsteps, xn, color='C1', s=300)
iproj.plot(tsteps, xn, color='C1')

## cos(t + pi/6) (C0=red)
iproj.scatter(tsteps, np.cos(tsteps+np.pi/6), color='C0', s=200)
iproj.plot(tsteps, np.cos(tsteps+np.pi/6), color='C0')

## -sin(t-pi/3) (C5=green)
iproj.scatter(tsteps, -np.sin(tsteps-np.pi/3), color='C5', s=80)
iproj.plot(tsteps, -np.sin(tsteps-np.pi/3), color='C5')
```

In the plot above, you should see:
* the DFT[1] phasor in grey, i.e. $-\sin(t)$
* Our phase shifted input in blue
* The phase shift determined by DFT[1] applied as a cosine wave: $\cos(t + \pi/6)$ in red
* Our phase shifted input generated directly using the `np.sin` function: $-\sin(t-\pi/3)$ in green

You should see is that that the last three functions are all the same! We can always write a sine wave as a cosine wave with a phase shift.

### The DFT for k = 2 and beyond

We can think of DFT[2] as representing the contribution of a phasor that spins around the unit circle
twice as fast as the $k=1$ DFT phasor:

* For $k=2$, Each $e^{i \frac{2\pi n}{N}k}$ is a step of $\frac{2\pi}{N}\times 2$ radians around the unit circle
    * i.e. we only sample every second point compared to the $k=1$ case.

* This means it only takes half the time to make a full cycle. So, this phasor represents a sinusoid with twice the frequency of the one for $k=1$

### Exercise:
Plot sinusoids for different values of `k` using the code below
* What happens when $k=N/2$?
* What happens when $k > N/2$?
* What if you increase $N$?
* How many actual frequencies can the DFT actually tell us about?
* How does this relate to the aliasing problem we saw in the previously?

```python
## Plot the phasor and sinusoid for different values of k

## Number of samples
N=16

## DFT output we're looking at: a sinusoid with k times the freq of the k=1 phasor
k=15

## indices of points in sample
nsteps = np.array(range(N))

## times of samples
tsteps = 2*np.pi*nsteps/N

## N sample values of the kth DFT phasor
zn_k = np.exp(k*-1j*tsteps)

X_k, Y_k, fig, phasor, iproj, rproj = plot_phasor_static(zn_k, tsteps, plot_real=False, plot_imag=True)

```

```python
kphasor_anim= get_phasor_animation(X_k, Y_k, tsteps, phasor, iproj, rproj, fig)
kphasor_anim
```

### Notes

```python

```

## 4.3 The full DFT

Now that we've seen what happens when we calculate the individual DFT components, lets do the whole thing!

We need

1. **Input:** a sequence of $N$ samples
1. generate $N$ phasors, with $N$ sampled points
1. generate $N$ dot products between the input and the phasors
1. **Output:** $N$ complex numbers representing magnitude and phase

The magnitudes and phases give us the decomposition of the input into pure cosine waves

### Set the input $x[n]$

Let's use the same input as above $x[n] = -\sin(2\pi n/N -\pi/3)$

```python
## 1) Input: -sin(t-pi/3) as above
N=16
nsteps = np.array(range(N))
theta_step = 2*np.pi/N
theta_steps = theta_step * nsteps

## set the phase shift
phase_in = np.pi/3

## set the input as -sin with phase shift
x = -np.sin(theta_steps-phase_in)

## Plot the input x
fig, tdom = plt.subplots(figsize=(16, 4))
tdom.scatter(tsteps, x, color='magenta')
tdom.plot(tsteps, x, color='magenta')

```

You should see the input (16 points), `x`, plotted in magenta

### Generate phasors

Let's use some functions to generate all the DFT phasors in one go:

```python
## 2) Generate the phasors
# Given a sequence length N, return dictionary representing N DFT outputs
# the kth element of the return dictionary contains the complex numbers sampled around the unit circle after
# N steps, their real and imaginary parts, and the angles of those complex numbers (magnitude is always 1)
def get_dft_phasors(N, centered=False):

    ## N input steps, N phasors
    nsteps = np.array(range(N))

    ## DFT works for negatively indexed samples, i.e. x[-n], but we don't need this right now!
    if centered:
        nsteps = nsteps - floor(N/2)
        print(nsteps)

    # We know the smallest step around the unit circle we can take is 2pi/N
    theta_step = 2*np.pi/N
    # And we're going to take N steps
    theta_steps = theta_step * nsteps

    ## Generate N phasors
    phasors = {}
    for k in range(N):
        ## samples around the unit circle
        zs = np.exp(k*-1j*theta_steps)
        real = np.real(zs)
        imag = np.imag(zs)

        ## Since we're here, let's return all these things for convenience
        # zs: the phasor samples, real: the real component, imag: the imaginary component
        # theta_steps: the angles for each phasor sample, theta_step: the smallest angle step size

        phasors[k] = {'zs':zs, 'real':real, 'imag':imag, 'thetas':theta_steps, 'theta_step':theta_step}

    return phasors

```

```python
## get the list of phasors
phasors = get_dft_phasors(N)
```

```python
## plot the different phasors and sine (imaginary) components
## You should be able to see the frequency relationship between each DFT component
for k in range(N):
    X_k, Y_k, fig, phasor, iproj, rproj = plot_phasor_static(phasors[k]['zs'], tsteps, plot_real=False, plot_imag=True, color='C'+str(k))

```

You should see in the plots above that phasors $k > N/2$ repeat the frequencies we see for $k < N/2$. For example,
the $k=1$ phasor is equivalent in frequency to the $k=15$ phasor!

### Calculate dot products (i.e.  input and phasor correlations)

```python
## 3) Get the dot products for each for each phasor and the input

def get_dft_outputs(x, phasors):
    DFT = []

    ## Go through our list of N phasors
    for k, phasor in phasors.items():

        ## Do the dot product between the input and each phasor sequence
        DFT.append(np.sum(x * phasor['zs']))

    return DFT

```

```python
## do the DFT
dft = get_dft_outputs(x, phasors)
```

### Get the output  magnitudes and phases

Now we convert the results of the dot products we just calculated into polar form to get magnitudes and phases. We can then plot them!

```python
## convert to polar coordinates to get magnitude and phase
dft_polar = [cmath.polar(z) for z in dft]
mags = [m for m, _ in dft_polar]
phases = [ph if mag > 0.00001 else 0 for mag,ph   in dft_polar]

## we need to put a condition into the phase calculation because phase is calculated by
## a ratio of the imaginary component and the real component. Both of these values might be very, very small.
## This makes it susceptible to floating point errors.
## Then, the ratio of two very small things can actually end up to be quite (spuriously) big!

```

```python
## Plot the magnitudes
#print(mags)
fig, freqdom = plt.subplots(figsize=(16, 4))
freqdom.set(xlim=(-1, N), ylim=(-1, 10))
freqdom.scatter(range(N), mags)
```

```python
## plot the phase angle for each DFT component
#print(phases)
fig, fdom = plt.subplots(figsize=(16, 4))
fdom.set(xlim=(-1, N), ylim=(-2, 2))
fdom.scatter(range(N), phases)
```

### How do we interpret the magnitude and phase graphs?

The top plot shows the magnitude response of the DFT with respect to our input.  The bottom plot shows the phase respose.

The magnitude plot shows positive magnitudes for the 1st and the N-1th DFT components, and zero for all the rest.  This means that the input is a sinusoid with the same frequency as the 1st DFT component.

The positive phase for the first component indicates a phase shift.  We see the opposite phase shift for the 15th component because the phasor for that has the same frequency but rotates the opposite way.

So, this is what we expect since we generated the input as exactly the sine projection of the k=1 phasor, just shifted by $\pi/3$ radians!

### Plotting DFT outputs on the complex plane
So far, we've looked at the real and imaginary components of the DFT outputs separately.  But we can also think about their properties on the complex plane.

The following plots the DFT outputs for the example above on the complex plane.

```python
## Plot the actual DFT outputs as complex numbers on the complex plane
## For the example above, we can see that most of the outputs are at (0,0), i.e. not contribution
## but we get positive magnitudes for the 1st and 15th component, with the same
## magnitude by 'mirrored' phase angle:

ymax=(N/2) + 2
fig, ax = plt.subplots(figsize=(5, 5))
ax.set(xlim=(-10, 10), ylim=(-10, 10))
ax.plot([-ymax,ymax], [0,0], color='grey')
ax.plot([0,0],[-ymax,ymax], color='grey')
circle1 = plt.Circle((0, 0), 1, color='grey',fill=False)

ax.add_artist(circle1)

for k in range(N):
    zs = phasors[k]['zs']
    xz = x * zs
    dftk = sum(xz)
    #print(dftk, dftk.real, cmath.polar(dftk))
    ax.plot(dftk.real, dftk.imag, 'o')

    if dftk.real > 0.0001:
        print("k=%d, %f + j%f" % (k, dftk.real, dftk.imag))

```

You can see from this plot that all but two of the DFT outputs have zero magnitude.  The two that have non-zero magnitude are the 1st and 15th components.  You can see that they have the same magnitude, but mirror each other in phase (they are complex conjugates). This symmetry is something you see a lot of in digital signal processing.  We won't go into it in this class, but for one thing we can take advantage of to save some computations!

### Exercise (optional)

You can use the code below to visualize the dot products of the input with with cos (real) and sin (imaginary) parts of each DFT component by varying $k$:

* What's the relationship between the DFT phasors for $k=1$ and $k=N-1$ in terms of the dot product between the real (left plot) and imaginary (right plot) components.
* What's happen for components that don't have the same frequency as the input? e.g. $k=2$

```python
## 1) Input: -sin(t-pi/3) as above
nsteps = np.array(range(N))
theta_step = 2*np.pi/N
theta_steps = theta_step * nsteps

Tmin = np.min(theta_steps)
Tmax = np.max(theta_steps)

phase_in = np.pi/3
x = -np.sin(theta_steps-phase_in)

fig, tdom = plt.subplots(figsize=(16, 4))
tdom.scatter(tsteps, x, color='magenta')
tdom.plot(tsteps, x, color='magenta')

```

```python
## let's break it down into cos and sin components
k=15

## zcos is our sampling of the real part of the kth phasor
zcos = phasors[k]['real']
zsin = phasors[k]['imag']

## the elementwise multiplication between the input x and the real part of the phasor, zcos
xz_real = x * zcos
xz_imag = x * zsin

## This initializes our plots, makes them the appropriate size etc, but leave the plot empty
_, _, fig, phasor, iproj, rproj = plot_phasor_static(phasors[k]['zs'], tsteps, plot_phasor=False, plot_real=True, plot_imag=True, color='grey')

## plot the zero line: if the sequence we get after the elementwise multiplication is
## symmetric around zero, we know that the dot product will be zero since we're adding all the
## values together
#inusoid.plot([Tmin-1,Tmax+1], [0,0], color='grey')

## Plot the input in magenta
rproj.plot(tsteps, x, color='magenta')
rproj.scatter(tsteps, x, color='magenta')

## Plot the elementwise multiplication in orange
rproj.plot(tsteps, xz_real, color='orange')
rproj.scatter(tsteps, xz_real, color='orange')

## Plot the input in magenta
iproj.plot(tsteps, x, color='magenta')
iproj.scatter(tsteps, x, color='magenta')

## Plot the elementwise multiplication in orange
iproj.plot(tsteps, xz_imag, color='orange')
iproj.scatter(tsteps, xz_imag, color='orange')

## Add it all up to get the dot product
dftk_real, dftk_imag = sum(xz_real), sum(xz_imag)
print("dft[%d] = %f + j%f "  % (k, dftk_real, dftk_imag))
```

### Notes

```python

```

```python

```

```python

```

```python

```

```python

```