Trusted Notebook" width="500 px" align="left">

## _*The Simon Algorithm*_

The latest version of this notebook is available on https://github.com/qiskit/qiskit-tutorial.

The Simon algorithm is an example that shows a quantum algorithm can solve a problem exponentially efficient than any classical algorithms. Like the Grover search, it depends on the existence of a blackbox (or, oracle) function that returns a predefined output over specific input or query. In the query-complexity setting, one cares only about how many queries are required to solve a specific problem, but does not care how the blackbox is realized. However, in this tutorial we have to implement it using the unit gates available in QISKit, just like we have done with the Grover search.

We first describe the problem addressed by the Simon algorithm, show the steps of the algorithm and the construction of the blackbox function, and present the experimental results on simulators and real devices.

***
### Contributors
Rudy Raymond

## The Problem

The Simon algorithm deals with finding a hidden integer $s \in \{0,1\}^n$ from an oracle $f_s$ that satisfies $f_s(x) = f_s(y)$ if and only if $y = x \oplus s$ for all $x \in \{0,1\}^n$.  Here, the $\oplus$ is the bitwise XOR operation. Thus, if $s = 0\ldots 0$, i.e., the all-zero bitstring, then $f_s$ is a 1-to-1 (or, permutation) function. Otherwise, if $s \neq 0\ldots 0$, then $f_s$ is a 2-to-1 function.

The Simon algorithm can find the hidden integer using only $O(n)$ queries to the blackbox function, while any classical algorithms require $\Omega(\sqrt{2^n})$ queries.

## The Algorithm to Find the Hidden Integer

The Simon algorithm finds the hidden integer by combining quantum algorithm with postprocessing on classical computers as below.

1. Prepare two quantum registers each of length $n$ that are initialized to all-zero bitstring: the first one as input and the second one as output of the blackbox function.
$$
|0\rangle |0\rangle
$$

2. Apply Hadamard gates to the first register to create superposition of all possible inputs.
$$
H^{\otimes n} |0\rangle |0\rangle = \frac{1}{\sqrt{2^n}} \sum_{x=0}^{2^n-1} |x\rangle |0\rangle
$$

3. Query the blackbox function to obtain the answer to queries on the second register.
$$
\frac{1}{\sqrt{2^n}} \sum_{x=0}^{2^n-1} U_{f_s}|x\rangle |0\rangle = \frac{1}{\sqrt{2^n}} \sum_{x=0}^{2^n-1} |x\rangle |f_s(x)\rangle
$$

4. Apply Hadamard gates to the first register.
$$
\frac{1}{\sqrt{2^n}} \sum_{x=0}^{2^n-1} H^{\otimes n}|x\rangle |f_s(x)\rangle = \frac{1}{2^n} \sum_{y=0}^{2^n-1}\sum_{x=0}^{2^n-1} (-1)^{x \cdot y}|y\rangle |f_s(x)\rangle = \frac{1}{2^n} \sum_{y=0}^{2^n-1} |y\rangle \sum_{x=0}^{2^n-1} ( (-1)^{x \cdot y} + (-1)^{(x\oplus s) \cdot y} )  |f_s(x)\rangle
$$

    Notice that at the right-hand side of the above equation, because $(-1)^{(x\oplus s) \cdot y} = (-1)^{x\cdot y +  s \cdot y}$ we can conclude that the probability amplitude of the basis state $|y\rangle |f_s(x)\rangle$ is $(-1)^{x\cdot y} (1 + (-1)^{s \cdot y}  )$, which is zero if and only if $s \cdot y = 1$. Thus, measuring the first register will always give $y$ such that $s \cdot y = 0$. Moreover, we can obtain many different $y$'s by repeating Step 1 to 4.

5. Repeat Step 1 to 4 for $m$ times to obtain $y_1, y_2, \ldots, y_m$.

6. **(Classical post-processing)** Let $\mathbf{Y}$ be an $m\times n$ matrix whose $i$-th row is $y_i$ in Step 5, and $\vec{s}$ be the column vector whose $j$-th element is the $j$-th bit of $s$. Solve the following system of linear equations to obtain $s$.
$$
\mathbf{Y} \vec{s} = 0
$$

## The Circuit

We now implement the Simon algorithm with Qiskit by first preparing the environment.

```python
#initialization
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# importing Qiskit
from qiskit import Aer, IBMQ
from qiskit import QuantumCircuit, ClassicalRegister, QuantumRegister
from qiskit import available_backends, execute, register, get_backend, compile
from qiskit.wrapper.jupyter import *

# import basic plot tools
from qiskit.tools.visualization import plot_histogram
```

```python
# Load the saved IBMQ accounts
IBMQ.load_accounts()
```

We then set the hidden bitstring $s$ that will be used to construct the circuit of the blackbox function (whose details will be given later). The number of qubits used in the experiment is twice the length of the bitstring $s$.

```python
s = "010101" # the hidden bitstring

assert 1 < len(s) < 20, "The length of s must be between 2 and 19"

for c in s:
    assert c == "0" or c == "1", "s must be a bitstring of '0' and '1'"

n = len(s) #the length of the bitstring
```

We then use Qiskit to create the circuit of the Simon algorithm prior the querying the blackbox function.

```python
# Step 1
# Creating registers
# qubits for querying the oracle and recording its output
qr = QuantumRegister(2*n)
# for recording the measurement on the first register of qr
cr = ClassicalRegister(n)

circuitName = "Simon"
simonCircuit = QuantumCircuit(qr, cr)

# Step 2
# Apply Hadamard gates before querying the oracle
for i in range(n):
    simonCircuit.h(qr[i])

# Apply barrier to mark the beginning of the blackbox function
simonCircuit.barrier()
```

    <qiskit.extensions.standard.barrier.Barrier at 0xa18623390>

### Constructing a Circuit for the Blackbox Function

We now details the construction of 1-to-1 and 2-to-1 permutation circuit of the blackbox function. Let us assume the blackbox function receive $|x\rangle|0\rangle$ as input. With regards to a predetermined $s$, the blackbox function writes its output to the second register so that it transforms the input to $|x\rangle|f_s(x)\rangle$ such that $f(x) = f(x\oplus s)$ for all $x \in \{0,1\}^n$.

Such a blackbox function can be realized by the following procedures.

-  Copy the content of the first register to the second register.
$$
|x\rangle|0\rangle \rightarrow |x\rangle|x\rangle
$$

-  **(Creating 1-to-1 or 2-to-1 mapping)** If $s$ is not all-zero, then there is the least index $j$ so that $s_j = 1$. If $x_j = 0$, then XOR the second register with $s$. Otherwise, do not change the second register.
$$
|x\rangle|x\rangle \rightarrow |x\rangle|x \oplus s\rangle~\mbox{if}~x_j = 0~\mbox{for the least index j}
$$

-  **(Creating random permutation)** Randomly permute and flip the qubits of the second register.
$$
|x\rangle|y\rangle \rightarrow |x\rangle|f_s(y)\rangle
$$

Below is the circuit of the blacbox function based on the above procedures.

```python
# Step 3 query the blackbox function
# copy the content of the first register to the second register
for i in range(n):
    simonCircuit.cx(qr[i], qr[n+i])

# get the least index j such that s_j is "1"
j = -1
for i, c in enumerate(s):
    if c == "1":
        j = i
        break

# Creating 1-to-1 or 2-to-1 mapping with the j-th qubit of x as control to XOR the second register with s
for i, c in enumerate(s):
    if c == "1" and j >= 0:
        simonCircuit.cx(qr[j], qr[n+i]) #the i-th qubit is flipped if s_i is 1

# get random permutation of n qubits
perm = list(np.random.permutation(n))

#initial position
init = list(range(n))
i = 0
while i < n:
    if init[i] != perm[i]:
        k = perm.index(init[i])
        simonCircuit.swap(qr[n+i], qr[n+k]) #swap qubits
        init[i], init[k] = init[k], init[i] #marked swapped qubits
    else:
        i += 1

# randomly flip the qubit
for i in range(n):
    if np.random.random() > 0.5:
        simonCircuit.x(qr[n+i])

# Apply the barrier to mark the end of the blackbox function
simonCircuit.barrier()
```

    <qiskit.extensions.standard.barrier.Barrier at 0xa1863a668>

Now we can continue with the steps of the Simon algorithm: applying the Hadamard gates to the first register and measure.

```python
# Step 4 apply Hadamard gates to the first register
for i in range(n):
    simonCircuit.h(qr[i])

# Step 5 perform measurement on the first register
for i in range(n):
    simonCircuit.measure(qr[i], cr[i])

#draw the circuit
from qiskit.tools.visualization import circuit_drawer
circuit_drawer(simonCircuit)
```

## Experimenting with Simulators

We show the experiments of finding the hidden integer with simulators.

```python
# use local simulator
backend = Aer.get_backend("qasm_simulator")
# the number of shots is twice the length of the bitstring
shots = 2*n
job = execute(simonCircuit, backend=backend, shots=shots)
answer = job.result().get_counts()

plot_histogram(answer)
```

We can see that the results of the measurements are the basis whose inner product with the hidden string $s$ are zero.

*(Notice that the basis on the label of the x-axis in the above plot are numbered from right to left instead of from left to right that we used for $s$)*

Gathering the measurement results, we proceed to post-processing with computations that can be done on classical computers.

### Post Processing with Gaussian Elimination

The post processing is done with Gaussian elimination to solve the system of linear equations to determine $s$.

```python
# Post-processing step
# Constructing the system of linear equations Y s = 0
# By k[::-1], we reverse the order of the bitstring
lAnswer = [ (k[::-1],v) for k,v in answer.items() if k != "0"*n  ] #excluding the trivial all-zero
#Sort the basis by their probabilities
lAnswer.sort(key = lambda x: x[1], reverse=True)

Y = []
for k, v in lAnswer:
    Y.append( [ int(c) for c in k ] )

#import tools from sympy
from sympy import Matrix, pprint, MatrixSymbol, expand, mod_inverse

Y = Matrix(Y)

#pprint(Y)

#Perform Gaussian elimination on Y
Y_transformed = Y.rref(iszerofunc=lambda x: x % 2==0) # linear algebra on GF(2)

#to convert rational and negatives in rref of linear algebra on GF(2)
def mod(x,modulus):
    numer, denom = x.as_numer_denom()
    return numer*mod_inverse(denom,modulus) % modulus

Y_new = Y_transformed[0].applyfunc(lambda x: mod(x,2)) #must takecare of negatives and fractional values
#pprint(Y_new)

print("The hidden bistring s[ 0 ], s[ 1 ]....s[",n-1,"] is the one satisfying the following system of linear equations:")
rows, cols = Y_new.shape
for r in range(rows):
    Yr = [ "s[ "+str(i)+" ]" for i, v in enumerate(list(Y_new[r,:])) if v == 1 ]
    if len(Yr) > 0:
        tStr = " + ".join(Yr)
        print(tStr, "= 0")

```

    The hidden bistring s[ 0 ], s[ 1 ]....s[ 5 ] is the one satisfying the following system of linear equations:
    s[ 0 ] = 0
    s[ 1 ] + s[ 5 ] = 0
    s[ 2 ] = 0
    s[ 3 ] + s[ 5 ] = 0
    s[ 4 ] = 0

As seen above, the system of linear equations is satisfied by the hidden integer $s$. Notice that there can be more than one solutions to the system. In fact, all-zero bitstring is a trivial solution to the system of linear equations. But by having more samples one can narrow down the candidates of the solution, and then test the solution by querying the blackbock in the classical manner.

## Experimenting with Real Devices

We see how one can still find out the hidden integer by running the Simon algorithm on real devices. Due to imperfect quantum computers, obtaining the conclusion is not as easy as done with the simulator of perfect quantum computers.

We first register the token and query the available backends.

```python
Aer.backends()
```

    [<QasmSimulator('qasm_simulator') from Aer()>,
     <QasmSimulatorPy('qasm_simulator_py') from Aer()>,
     <StatevectorSimulator('statevector_simulator') from Aer()>,
     <StatevectorSimulatorPy('statevector_simulator_py') from Aer()>,
     <UnitarySimulator('unitary_simulator') from Aer()>,
     <CliffordSimulator('clifford_simulator') from Aer()>]

```python
IBMQ.backends()
```

    [<IBMQBackend('ibmqx4') from IBMQ()>,
     <IBMQBackend('ibmqx5') from IBMQ()>,
     <IBMQBackend('ibmqx2') from IBMQ()>,
     <IBMQBackend('ibmq_16_melbourne') from IBMQ()>,
     <IBMQBackend('ibmq_qasm_simulator') from IBMQ()>]

```python
%%qiskit_job_status
#Use one of the available backends
backend = IBMQ.get_backend("ibmq_16_melbourne")

# show the status of the backend
print("Status of", backend, "is", get_backend(backend).status())

shots = 10*n #run more experiments to be certain
max_credits = 3          # Maximum number of credits to spend on executions.

job_exp = execute(simonCircuit, backend=backend, shots=shots, max_credits=max_credits)
```

    Status of ibmq_16_rueschlikon is {'pending_jobs': 36, 'name': 'ibmqx5', 'operational': True}

    VBox(children=(HTML(value="
Job Status : job is being initialized
"),))

```python
results = job_exp.result()
answer = results.get_counts(simonCircuit)

plot_histogram(answer, options={'width': 12, 'height': 6})

# Post-processing step
# Constructing the system of linear equations Y s = 0
# By k[::-1], we reverse the order of the bitstring
lAnswer = [ (k[::-1][:n],v) for k,v in answer.items() ] #excluding the qubits that are not part of the inputs
#Sort the basis by their probabilities
lAnswer.sort(key = lambda x: x[1], reverse=True)

Y = []
for k, v in lAnswer:
    Y.append( [ int(c) for c in k ] )

Y = Matrix(Y)

#Perform Gaussian elimination on Y
Y_transformed = Y.rref(iszerofunc=lambda x: x % 2==0) # linear algebra on GF(2)
Y_new = Y_transformed[0].applyfunc(lambda x: mod(x,2)) #must takecare of negatives and fractional values
#pprint(Y_new)

print("The hidden bistring s[ 0 ], s[ 1 ]....s[",n-1,"] is the one satisfying the following system of linear equations:")
rows, cols = Y_new.shape
for r in range(rows):
    Yr = [ "s[ "+str(i)+" ]" for i, v in enumerate(list(Y_new[r,:])) if v == 1 ]
    if len(Yr) > 0:
        tStr = " + ".join(Yr)
        print(tStr, "= 0")
```

## References

[1] "[On the power of quantum computation](https://epubs.siam.org/doi/abs/10.1137/S0097539796298637)", Daniel R. Simon, SIAM J. Comput., 26(5), 1474–1483 (1997)


https://towardsdatascience.com/quantile-regression-from-linear-models-to-trees-to-deep-learning-af3738b527c3

>[Quantile regression from OLS to TensorFlow](#scrollTo=PQdJbWCS9N3G)

>>[Setup](#scrollTo=VjvK4vP2-X_D)

>>>[Graph options](#scrollTo=sm3RTN850S70)

>>>[Data](#scrollTo=vdaqXkSCe7gG)

>>>>[Load](#scrollTo=vdaqXkSCe7gG)

>>>>[Normalize](#scrollTo=xuphFNDqtEiW)

>>>>[Make DataFrames](#scrollTo=HDT3Cs7ptIYG)

>>>[Initialize result](#scrollTo=utzAT-gdBTqj)

>>[Quantile loss example](#scrollTo=OZ7dkNLfyZ3r)

>>[OLS](#scrollTo=6dRLoTDhCrpO)

>>[QuantReg](#scrollTo=esTAKyTyG1TS)

>>[Random forests](#scrollTo=qCt2S4V9Uepc)

>>[Gradient boosted trees](#scrollTo=6ssXXxsCA7FS)

>>[Keras](#scrollTo=H55y_74W97jH)

>>>[Quantiles 0.1, 0.5, 0.9](#scrollTo=SeMIPNsO-qxI)

>>[TensorFlow](#scrollTo=PEy2M6X89tbD)

>>[Visualize quantiles](#scrollTo=NG7Oqjp-7htm)

>>[Compare quantile loss](#scrollTo=42LIBXl933YI)

>>[All features](#scrollTo=ig_Wywm05sjA)

>>>[OLS](#scrollTo=ImnTiUft-Yhq)

>>>[Quantreg](#scrollTo=sC29BePy_dtj)

>>>[Random forests](#scrollTo=Z3g3cXQVDvBJ)

>>>[Gradient boosting](#scrollTo=kfHp4qvBDeA3)

>>>[Keras](#scrollTo=tLm8TjoV_qap)

>>>[TensorFlow](#scrollTo=G_lApuMnAoR9)

>>>[Compare](#scrollTo=hn3sxw4yDXdz)

# Quantile regression from OLS to TensorFlow

Use the Boston housing dataset.

Based on blog posts using:
* Keras: https://towardsdatascience.com/deep-quantile-regression-c85481548b5a
* Tensorflow: https://towardsdatascience.com/deep-quantile-regression-in-tensorflow-1dbc792fe597
* statsmodels quantile regression: https://www.statsmodels.org/dev/examples/notebooks/generated/quantile_regression.html

## Setup

```python
import pandas as pd
import numpy as np

import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns

import statsmodels.api as sm

from scipy.stats import norm

from sklearn import ensemble

import tensorflow as tf
from tensorflow import keras
```

Keras and TensorFlow options

```python
EPOCHS = 200
BATCH_SIZE = 32
UNITS = 512
```

### Graph options

```python
# !wget https://github.com/MaxGhenis/random/raw/master/Roboto-Regular.ttf -P /usr/local/lib/python3.6/dist-packages/matplotlib/mpl-data/fonts/ttf
# mpl.font_manager._rebuild()
```

```python
sns.set_style('white')
DPI = 200
mpl.rc('savefig', dpi=DPI)
mpl.rcParams['figure.dpi'] = DPI
mpl.rcParams['figure.figsize'] = 6.4, 4.8  # Default.
mpl.rcParams['font.sans-serif'] = 'Roboto'
mpl.rcParams['font.family'] = 'sans-serif'

# Set title text color to dark gray (https://material.io/color) not black.
TITLE_COLOR = '#212121'
mpl.rcParams['text.color'] = TITLE_COLOR

# Axis titles and tick marks are medium gray.
AXIS_COLOR = '#757575'
mpl.rcParams['axes.labelcolor'] = AXIS_COLOR
mpl.rcParams['xtick.color'] = AXIS_COLOR
mpl.rcParams['ytick.color'] = AXIS_COLOR
```

### Data

#### Load

```python
boston_housing = keras.datasets.boston_housing

(train_data, train_labels), (test_data, test_labels) = \
    boston_housing.load_data()
```

```python
train_data.shape
```

    (404, 13)

```python
test_data.shape
```

    (102, 13)

#### Make DataFrames

```python
column_names = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD',
                'TAX', 'PTRATIO', 'B', 'LSTAT']

train_df = pd.DataFrame(train_data, columns=column_names)
test_df = pd.DataFrame(test_data, columns=column_names)
```

#### Normalize

```python
train_df.describe()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      count
      404.000000
      404.000000
      404.000000
      404.000000
      404.000000
      404.000000
      404.000000
      404.000000
      404.000000
      404.000000
      404.000000
      404.000000
      404.000000

      mean
      3.745111
      11.480198
      11.104431
      0.061881
      0.557356
      6.267082
      69.010644
      3.740271
      9.440594
      405.898515
      18.475990
      354.783168
      12.740817

      std
      9.240734
      23.767711
      6.811308
      0.241238
      0.117293
      0.709788
      27.940665
      2.030215
      8.698360
      166.374543
      2.200382
      94.111148
      7.254545

      min
      0.006320
      0.000000
      0.460000
      0.000000
      0.385000
      3.561000
      2.900000
      1.129600
      1.000000
      188.000000
      12.600000
      0.320000
      1.730000

      25%
      0.081437
      0.000000
      5.130000
      0.000000
      0.453000
      5.874750
      45.475000
      2.077100
      4.000000
      279.000000
      17.225000
      374.672500
      6.890000

      50%
      0.268880
      0.000000
      9.690000
      0.000000
      0.538000
      6.198500
      78.500000
      3.142300
      5.000000
      330.000000
      19.100000
      391.250000
      11.395000

      75%
      3.674808
      12.500000
      18.100000
      0.000000
      0.631000
      6.609000
      94.100000
      5.118000
      24.000000
      666.000000
      20.200000
      396.157500
      17.092500

      max
      88.976200
      100.000000
      27.740000
      1.000000
      0.871000
      8.725000
      100.000000
      10.710300
      24.000000
      711.000000
      22.000000
      396.900000
      37.970000

```python
mean = train_df.mean(axis=0)
std = train_df.std(axis=0)
train_df = (train_df - mean) / std
test_df = (test_df - mean) / std
```

```python
x_train = train_df.AGE
x_test = test_df.AGE
```

Denormalize.

```python
def denorm_AGE(x):
    return x * std.AGE + mean.AGE

x_train_denorm = denorm_AGE(x_train)
x_test_denorm = denorm_AGE(x_test)
```

Reformat data for `statsmodels`.

```python
X_train = sm.add_constant(x_train)
X_test = sm.add_constant(x_test)
```

Reformat data for `tensorflow`.

```python
x_train_expanded = np.expand_dims(x_train, 1)
x_test_expanded = np.expand_dims(x_test, 1)
train_labels_expanded = np.expand_dims(train_labels, 1)
```

### Initialize result

Dataset per method, quantile, and `x` value.

```python
METHODS = ['OLS', 'QuantReg', 'Random forests', 'Gradient boosting', 'Keras',
           'TensorFlow']

QUANTILES = [0.1, 0.3, 0.5, 0.7, 0.9]
# QUANTILES.reverse()  # Test out to see if we're getting different results.

quantiles_legend = [str(int(q * 100)) + 'th percentile' for q in QUANTILES]
```

```python
# sns.set_palette(sns.color_palette('Blues', len(QUANTILES)))
sns.set_palette(sns.color_palette('Blues'))
# Set dots to a light gray
dot_color = sns.color_palette('coolwarm', 3)[1]
```

```python
preds = np.array([(method, q, x)
                  for method in METHODS
                  for q in QUANTILES
                  for x in x_test])
preds = pd.DataFrame(preds)
preds.columns = ['method', 'q', 'x']
preds = preds.apply(lambda x: pd.to_numeric(x, errors='ignore'))
```

```python
preds['label'] = np.resize(test_labels, preds.shape[0])
```

## Quantile loss example

```python
# pandas version rather than Keras.
def quantile_loss(q, y, f):
    # q: Quantile to be evaluated, e.g., 0.5 for median.
    # y: True value.
    # f: Fitted or predicted value.
    e = y - f
    return np.maximum(q * e, (q - 1) * e)
```

```python
quantile_loss_example_e = np.linspace(-1, 1, 1000)
quantile_loss_example_loss_10 = quantile_loss(0.1, 0, quantile_loss_example_e)
quantile_loss_example_loss_50 = quantile_loss(0.5, 0, quantile_loss_example_e)
quantile_loss_example_loss_90 = quantile_loss(0.9, 0, quantile_loss_example_e)
```

```python
with sns.color_palette('Blues', 3):
    plt.plot(quantile_loss_example_e, quantile_loss_example_loss_10)
    plt.plot(quantile_loss_example_e, quantile_loss_example_loss_50)
    plt.plot(quantile_loss_example_e, quantile_loss_example_loss_90)
    plt.legend([str(int(q * 100)) + 'th percentile' for q in [0.1, 0.5, 0.9]])
    sns.despine(left=True, bottom=True)
    plt.xlabel('Error')
    plt.ylabel('Quantile loss')
    plt.title('Quantile loss by error and quantile', loc='left');
```

```python
quantile_loss_example_q = np.linspace(0.01, 0.99, 99)
quantile_loss_example_loss_neg1 = quantile_loss(quantile_loss_example_q, 0, -1)
quantile_loss_example_loss_pos1 = quantile_loss(quantile_loss_example_q, 0, 1)
```

```python
with sns.color_palette('Blues', 2):
    plt.plot(quantile_loss_example_q, quantile_loss_example_loss_neg1)
    plt.plot(quantile_loss_example_q, quantile_loss_example_loss_pos1)
    plt.legend(['Error of -1', 'Error of +1'])
    sns.despine(left=True, bottom=True)
    plt.xlabel('Quantile')
    plt.ylabel('Quantile loss')
    plt.title('Quantile loss by quantile and error', loc='left');
```

### Scatter plot of AGE

```python
mean.AGE
```

    69.01064356435643

```python
std.AGE
```

    27.940664897596555

```python
from matplotlib.ticker import FuncFormatter

ax = plt.scatter(x_train_denorm, train_labels, color=dot_color)
plt.title('Age vs. value, Boston housing dataset (training slice)', loc='left')
sns.despine(left=True, bottom=True)
ax.axes.xaxis.set_major_formatter(FuncFormatter(
    lambda x, _: '{:.0%}'.format(x / 100)))
ax.axes.yaxis.set_major_formatter(FuncFormatter(
    lambda y, _: '${:.0f}k'.format(y)))
plt.xlabel('Proportion of owner-occupied units built prior to 1940')
plt.ylabel('Median value of owner-occupied homes')
plt.show()
```

## OLS

https://stackoverflow.com/questions/17559408/confidence-and-prediction-intervals-with-statsmodels

```python
ols = sm.OLS(train_labels, X_train).fit()
```

```python
def ols_quantile(m, X, q):
    # m: OLS model.
    # X: X matrix.
    # q: Quantile.
    #
    # Set alpha based on q. Vectorized for different values of q.
    mean_pred = m.predict(X)
    se = np.sqrt(m.scale)
    return mean_pred + norm.ppf(q) * se
```

```python
preds.loc[preds.method == 'OLS', 'pred'] = np.concatenate(
    [ols_quantile(ols, X_test, q) for q in QUANTILES])
```

## QuantReg

https://www.statsmodels.org/dev/examples/notebooks/generated/quantile_regression.html

```python
quantreg = sm.QuantReg(train_labels, X_train)  # Don't fit yet, since we'll fit once per quantile.
```

```python
preds.loc[preds.method == 'QuantReg', 'pred'] = np.concatenate(
    [quantreg.fit(q=q).predict(X_test) for q in QUANTILES])
```

## Random forests

```python
N_ESTIMATORS = 1000
rf = ensemble.RandomForestRegressor(n_estimators=N_ESTIMATORS,
                                    min_samples_leaf=1, random_state=3,
                                    verbose=True,
                                    n_jobs=-1)  # Use maximum number of cores.
rf.fit(X_train, train_labels)
```

    [Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  34 tasks      | elapsed:    0.0s
    [Parallel(n_jobs=-1)]: Done 184 tasks      | elapsed:    0.0s
    [Parallel(n_jobs=-1)]: Done 434 tasks      | elapsed:    0.2s
    [Parallel(n_jobs=-1)]: Done 784 tasks      | elapsed:    0.4s
    [Parallel(n_jobs=-1)]: Done 1000 out of 1000 | elapsed:    0.5s finished

    RandomForestRegressor(bootstrap=True, ccp_alpha=0.0, criterion='mse',
                          max_depth=None, max_features='auto', max_leaf_nodes=None,
                          max_samples=None, min_impurity_decrease=0.0,
                          min_impurity_split=None, min_samples_leaf=1,
                          min_samples_split=2, min_weight_fraction_leaf=0.0,
                          n_estimators=1000, n_jobs=-1, oob_score=False,
                          random_state=3, verbose=True, warm_start=False)

```python
def rf_quantile(m, X, q):
    rf_preds = []
    for estimator in m.estimators_:
        rf_preds.append(estimator.predict(X))
    rf_preds = np.array(rf_preds).transpose()  # One row per record.
    return np.percentile(rf_preds, q * 100, axis=1)
```

```python
preds.loc[preds.method == 'Random forests', 'pred'] = np.concatenate(
    [rf_quantile(rf, X_test, q) for q in QUANTILES])
```

## Gradient boosted trees

```python
def gb_quantile(X_train, train_labels, X, q):
    gbf = ensemble.GradientBoostingRegressor(loss='quantile', alpha=q,
                                             n_estimators=N_ESTIMATORS,
                                             max_depth=3,
                                             learning_rate=0.1, min_samples_leaf=9,
                                             min_samples_split=9)
    gbf.fit(X_train, train_labels)
    return gbf.predict(X)
```

```python
preds.loc[preds.method == 'Gradient boosting', 'pred'] = np.concatenate(
    [gb_quantile(X_train, train_labels, X_test, q) for q in QUANTILES])
```

## Keras

From https://github.com/sachinruk/KerasQuantileModel/blob/master/Keras%20Quantile%20Model.ipynb

One area that Deep Learning has not explored extensively is the uncertainty in estimates. However, as far as decision making goes, most people actually require quantiles as opposed to true uncertainty in an estimate. eg. For a given age the weight of an individual will vary. What would be interesting is the (for arguments sake) the 10th and 90th percentile. The uncertainty of the estimate of an individuals weight is less interesting.

### Quantiles 0.1, 0.5, 0.9

The loss for an individual data point is defined as:
$$
\begin{align}
\mathcal{L}(\xi_i|\alpha)=\begin{cases}
\alpha \xi_i &\text{if }\xi_i\ge 0, \\
(\alpha-1) \xi_i &\text{if }\xi_i< 0.
\end{cases}
\end{align}
$$
where $\alpha$ is the required quantile and $\xi_i = y_i - f(\mathbf{x}_i)$ and, $f(\mathbf{x}_i)$ is the predicted (quantile) model. The final overall loss is defines as:
$$\mathcal{L}(\mathbf{y},\mathbf{f}|\alpha)=\frac{1}{N} \sum_{i=1}^N \mathcal{L}(y_i-f(\mathbf{x}_i)|\alpha)$$

The following function defines the loss function for a quantile model.

**Note**: The following 4 lines is ALL that you change in comparison to a normal Deep Learning method, i.e. The loss function is all that changes.

```python
def tilted_loss(q, y, f):
    e = (y - f)
    return keras.backend.mean(keras.backend.maximum(q * e, (q - 1) * e),
                              axis=-1)
```

```python
# optimizer = tf.train.AdamOptimizer(0.001)
optimizer = tf.optimizers.Adam(0.001)
early_stop = keras.callbacks.EarlyStopping(monitor='val_loss', patience=20)
```

```python
def keras_pred(x_train, train_labels, x_test, q):
    print(q)
    # Set input_dim for the number of features.
    if len(x_train.shape) == 1:
        input_dim = 1
    else:
        input_dim = x_train.shape[1]
    model = keras.Sequential([
      keras.layers.Dense(UNITS, activation=tf.nn.relu,
                         input_dim=input_dim),
      keras.layers.Dense(UNITS, activation=tf.nn.relu),
      keras.layers.Dense(1)
    ])

    model.compile(loss=lambda y, f: tilted_loss(q, y, f), optimizer=optimizer)
    model.fit(x_train, train_labels, epochs=EPOCHS, batch_size=BATCH_SIZE,
              verbose=0, validation_split=0.2, callbacks=[early_stop])

    # Predict the quantile
    return model.predict(x_test)
```

```python
preds.loc[preds.method == 'Keras', 'pred'] = np.concatenate(
    [keras_pred(x_train_expanded, train_labels, x_test_expanded, q)
     for q in QUANTILES])
```

    0.1
    0.3
    0.5
    0.7
    0.9

## TensorFlow

Adapted from https://github.com/strongio/quantile-regression-tensorflow/blob/master/Quantile%20Loss.ipynb

```python
# # Initialize session
# # sess = tf.Session()
# sess = tf.compat.v1.Session()
```

```python
# # Create network
# class q_model:
#     def __init__(self,
#                  sess,
#                  quantiles,
#                  in_shape=1,
#                  out_shape=1,
#                  batch_size=32):

#         self.sess = sess

#         self.quantiles = quantiles
#         self.num_quantiles = len(quantiles)

#         self.in_shape = in_shape
#         self.out_shape = out_shape
#         self.batch_size = batch_size

#         self.outputs = []
#         self.losses = []
#         self.loss_history = []

#         self.build_model()

#     def build_model(self, scope='q_model', reuse=tf.AUTO_REUSE):
#         with tf.variable_scope(scope, reuse=reuse) as scope:
#             self.x = tf.placeholder(tf.float32, shape=(None, self.in_shape))
#             self.y = tf.placeholder(tf.float32, shape=(None, self.out_shape))

#             self.layer0 = tf.layers.dense(self.x,
#                                           units=UNITS,
#                                           activation=tf.nn.relu)
#             self.layer1 = tf.layers.dense(self.layer0,
#                                           units=UNITS,
#                                           activation=tf.nn.relu)

#             # Create outputs and losses for all quantiles
#             for i, q in enumerate(self.quantiles):
#                 # Get output layers
#                 output = tf.layers.dense(self.layer1, self.out_shape,
#                                          name="{}_q{}".format(i, int(q * 100)))
#                 self.outputs.append(output)

#                 # Create losses
#                 error = tf.subtract(self.y, output)
#                 loss = tf.reduce_mean(tf.maximum(q * error, (q - 1) * error),
#                                       axis=-1)

#                 self.losses.append(loss)

#             # Create combined loss
#             self.combined_loss = tf.reduce_mean(tf.add_n(self.losses))
#             self.train_step = tf.train.AdamOptimizer().minimize(
#                 self.combined_loss)

#     def fit(self, x, y, epochs=EPOCHS):
#         for epoch in range(epochs):
#             epoch_losses = []
#             for idx in range(0, x.shape[0], self.batch_size):
#                 batch_x = x[idx : min(idx + self.batch_size, x.shape[0]), :]
#                 batch_y = y[idx : min(idx + self.batch_size, y.shape[0]), :]

#                 feed_dict = {self.x: batch_x,
#                              self.y: batch_y}

#                 _, c_loss = self.sess.run([self.train_step, self.combined_loss],
#                                           feed_dict)
#                 epoch_losses.append(c_loss)

#             epoch_loss =  np.mean(epoch_losses)
#             self.loss_history.append(epoch_loss)
#             if epoch % 100 == 0:
#                 print("Epoch {}: {}".format(epoch, epoch_loss))

#     def predict(self, x):
#         # Run model to get outputs
#         feed_dict = {self.x: x}
#         predictions = sess.run(self.outputs, feed_dict)

#         return predictions
```

```python
# # Instantiate model
# tf_model = q_model(sess, QUANTILES, in_shape=1, out_shape=1,
#                    batch_size=BATCH_SIZE)

# # Initialize all variables
# init_op = tf.global_variables_initializer()
# sess.run(init_op)
```

```python
# # Run training
# tf_model.fit(x_train_expanded, train_labels_expanded, EPOCHS)
```

    Epoch 0: 54.29452133178711
    Epoch 100: 10.774564743041992

```python
# preds.loc[preds.method == 'TensorFlow', 'pred'] = \
#     np.array([item for sublist in tf_model.predict(x_test_expanded)
#               for item in sublist])
```

Denormalize `x`

```python
# preds['x_denorm'] = denorm_AGE(preds.x)
```

## Visualize quantiles

```python
# for i, method in enumerate(METHODS):
#     ax = plt.scatter(x_test_denorm, test_labels, color=dot_color)
#     plt.plot(preds[preds.method == method].pivot_table(
#         index='x_denorm', columns='q', values='pred'))
#     plt.legend(quantiles_legend)
#     # Reversing legend isn't working, possibly because of multiple plots.
# #     handles, labels = ax.get_legend_handles_labels()
# #     ax.legend(handles[::-1], labels[::-1])
#     plt.xlim((0, 100))
#     ax.axes.xaxis.set_major_formatter(FuncFormatter(
#         lambda x, _: '{:.0%}'.format(x / 100)))
#     ax.axes.yaxis.set_major_formatter(FuncFormatter(
#         lambda y, _: '${:.0f}k'.format(y)))
#     plt.xlabel('Proportion of owner-occupied units built prior to 1940')
#     plt.ylabel('Median value of owner-occupied homes')
#     plt.title(method + ' quantiles', loc='left')
#     sns.despine(left=True, bottom=True)
#     plt.show()
```

## Compare quantile loss

```python
preds['quantile_loss'] = quantile_loss(preds.q, preds.label, preds.pred)
```

```python
def plot_loss_comparison(preds):
    overall_loss_comparison = preds[~preds.quantile_loss.isnull()].\
      pivot_table(index='method', values='quantile_loss').\
      sort_values('quantile_loss')
    # Show overall table.
    print(overall_loss_comparison)

    # Plot overall.
    with sns.color_palette('Blues', 1):
        ax = overall_loss_comparison.plot.barh()
        plt.title('Total quantile loss', loc='left')
        sns.despine(left=True, bottom=True)
        plt.xlabel('Quantile loss')
        plt.ylabel('')
        ax.legend_.remove()

    # Per quantile.
    per_quantile_loss_comparison = preds[~preds.quantile_loss.isnull()].\
        pivot_table(index='q', columns='method', values='quantile_loss')
    # Sort by overall quantile loss.
    per_quantile_loss_comparison = \
        per_quantile_loss_comparison[overall_loss_comparison.index]
    print(per_quantile_loss_comparison)

    # Plot per quantile.
    with sns.color_palette('Blues'):
        ax = per_quantile_loss_comparison.plot.barh()
        plt.title('Quantile loss per quantile', loc='left')
        sns.despine(left=True, bottom=True)
        handles, labels = ax.get_legend_handles_labels()
        plt.xlabel('Quantile loss')
        plt.ylabel('Quantile')
        # Reverse legend.
        ax.legend(reversed(handles), reversed(labels))
```

```python
plot_loss_comparison(preds)
```

## All features

Same analysis but using all `x` features.

```python
preds_full = preds[['method', 'q', 'label']].copy(deep=True)
```

```python
x_train_full = train_df.copy(deep=True)
x_test_full = test_df.copy(deep=True)
```

Reformat data for `statsmodels`.

```python
X_train_full = sm.add_constant(x_train_full)
X_test_full = sm.add_constant(x_test_full)
```

### OLS

```python
ols_full = sm.OLS(train_labels, X_train_full).fit()
```

```python
preds_full.loc[preds_full.method == 'OLS', 'pred'] = np.concatenate(
    [ols_quantile(ols_full, X_test_full, q) for q in QUANTILES])
```

### Quantreg

```python
# Don't fit yet, since we'll fit once per quantile.
quantreg_full = sm.QuantReg(train_labels, X_train_full)
```

```python
preds_full.loc[preds_full.method == 'QuantReg', 'pred'] = np.concatenate(
    [quantreg_full.fit(q=q).predict(X_test_full) for q in QUANTILES])
```

### Random forests

```python
rf_full = ensemble.RandomForestRegressor(n_estimators=N_ESTIMATORS,
                                         min_samples_leaf=1, random_state=3,
                                         n_jobs=-1)
rf_full.fit(X_train_full, train_labels)
```

    RandomForestRegressor(bootstrap=True, ccp_alpha=0.0, criterion='mse',
                          max_depth=None, max_features='auto', max_leaf_nodes=None,
                          max_samples=None, min_impurity_decrease=0.0,
                          min_impurity_split=None, min_samples_leaf=1,
                          min_samples_split=2, min_weight_fraction_leaf=0.0,
                          n_estimators=1000, n_jobs=-1, oob_score=False,
                          random_state=3, verbose=0, warm_start=False)

```python
preds_full.loc[preds_full.method == 'Random forests', 'pred'] = np.concatenate(
    [rf_quantile(rf_full, X_test_full, q) for q in QUANTILES])
```

### Gradient boosting

```python
preds_full.loc[preds_full.method == 'Gradient boosting', 'pred'] = \
    np.concatenate([gb_quantile(X_train_full, train_labels, X_test_full, q)
                    for q in QUANTILES])
```

### Keras

```python
preds_full.loc[preds_full.method == 'Keras', 'pred'] = np.concatenate(
    [keras_pred(x_train_full, train_labels, x_test_full, q) for q in QUANTILES])
```

    0.1
    0.3
    0.5
    0.7
    0.9

### TensorFlow

Reset graph to avoid conflicts with previous model.

```python
# tf.reset_default_graph()
# sess = tf.Session()
```

```python
# # Instantiate model
# tf_model_full = q_model(sess, QUANTILES, in_shape=train_data.shape[1],
#                         batch_size=BATCH_SIZE)

# # Initialize all variables
# init_op = tf.global_variables_initializer()
# sess.run(init_op)

# # Run training
# tf_model_full.fit(train_data, train_labels_expanded, EPOCHS)

# preds_full.loc[preds_full.method == 'TensorFlow', 'pred'] = np.array(
#     [item for sublist in tf_model_full.predict(test_data)
#      for item in sublist]
# )
```

    Epoch 0: 28.826343536376953
    Epoch 100: 4.595376968383789

### Compare

```python
preds_full['quantile_loss'] = quantile_loss(preds_full.q, preds_full.label,
                                            preds_full.pred)
```

```python
plot_loss_comparison(preds_full)
```

Save everything manually using `pickle` (`shelve` and `dill` fail for some objects ([issue](https://github.com/uqfoundation/dill/issues/155))).

```python
# import pickle

# !mkdir quantile_regression_comparison

# for i in ['preds']:
#     pickle.dump(eval(i),
#                 open('quantile_regression_comparison/' + i + '.p', 'wb'))
```


# The Dynamic Factor Model

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import metran

metran.show_versions()
```

    Python version: 3.8.8 (default, Apr 13 2021, 19:58:26)
    [GCC 7.3.0]
    numpy version: 1.20.1
    scipy version: 1.6.2
    pandas version: 1.2.4
    matplotlib version: 3.3.4
    pastas version: 0.17.1
    numba version: 0.53.1
    lmfit version: 1.0.2

Tip: To run this notebook and the related metran model, it is strongly recommended to install Numba (http://numba.pydata.org). This Just-In-Time (JIT) compiler compiles the computationally intensive part of metran model.

When modeling multiple groundwater time series within the same hydrological system, it often appears that these components show distinct correlations between locations. Usually large part of the correlation is caused by common input stresses like precipitation and evapotranspiration, which shows up within the deterministic components of the models.

The residual components of the univariate TFN models are often correlated as well. This means that there is spatial correlation
which has not been captured by the deterministic component, e.g. because of errors in common input data or due to simplification of the hydrological model leading to misspecification of the deterministic component. We can exploit these correlations by modeling the series simultaneously with a dynamic factor model. Dynamic factor modeling (DFM) is a multivariate timeseries analysis technique used to describe the variation among many variables in terms of a few underlying but unobserved variables called factors.

This notebook explains the Dynamic Factor Model (DFM) as presented in [Berendrecht and Van Geer, 2016](#References). It describes the model, model parameters and how the results may be interpreted.

## 1. Basic multivariate AR(1) model
A general univariate AR(1) model can be written as:

$$
\begin{align}
 {x}_t&=\phi x_{t-1}+\eta_t\\
 {n}_t&={x}_t+\varepsilon_t
\end{align}
$$

with $\phi$ the AR(1) parameter, $\eta_t$ a zero mean white noise process, and $\varepsilon_t$ the measurement noise. In the current version of `metran` the measurement noise is assumed to be zero, so that $n_t=x_t$.

The multivariate extension of this model is:

$$
 \left[\begin{array}{c}x_{1}\\x_{2}\end{array}\right]_t = \left[\begin{array}{cc}\phi_{1} & 0\\0 & \phi_{2}\end{array}\right] \left[\begin{array}{c}x_{1}\\x_{2}\end{array}\right]_{t-1} + \left[\begin{array}{c}\eta_{1}\\\eta_{2}\end{array}\right]_t
$$

Or:

$$
 \mathbf{x}_t=\mathbf{\Phi} \mathbf{x}_{t-1}+\mathbf{\eta}_t
$$

## 2. Generate synthetic correlated time series
Let us generate time series based on the 2-dimensional model given above. We use the AR(1) model to generate three time series with the AR(1) parameter $\phi$: two series as the specific dynamic factor and one series as the common dynamic factor. Combining the specific and common dynamic factors results in two time series which are mutually correlated.

```python
# seed numpy.random
np.random.seed(20210505)

# define mean and scale (standard deviation for noise series)
mean = np.zeros(3)
scale = [1, 0.6, 2]

# generate noise series that are mutually uncorrelated
noise = np.random.multivariate_normal(mean, np.diag(np.square(scale)), 2001)

# generate AR(1) processes
phi = np.array([0.80, 0.95, 0.90])
a = np.zeros_like(noise)
for i in range(1, noise.shape[0]):
    a[i] = noise[i] + np.multiply(a[i - 1], phi)

# add AR(1) processes to construct two correlated series
s1 = np.add(a[1:, 0], a[1:, 2])
s2 = np.add(a[1:, 1], a[1:, 2])

s = pd.DataFrame(data=np.array([s1, s2]).T,
                 index=pd.date_range(start='1-1-2000', periods=2000),
                 columns=['series 1', 'series 2'])

s.plot(figsize=(10, 2), xlabel='Date');
```

We can calculated the mean and standard deviation of the generated series and test the correlation between these series. The correlation must be close to the desired correlation defined above.

```python
print('Mean:')
print(s.mean())
print('\nStandard deviation:')
print(s.std())
print('\nCorrelation:')
print(s.corr())
```

    Mean:
    series 1   -0.847064
    series 2   -0.868526
    dtype: float64

    Standard deviation:
    series 1    4.769178
    series 2    4.928326
    dtype: float64

    Correlation:
              series 1  series 2
    series 1  1.000000  0.872126
    series 2  0.872126  1.000000

## 3. The Dynamic Factor Model
With the Dynamic Factor Model (DFM) we try to decompose series into latent (unobserved) factors describing common and specific dynamics. For the example above, the common dynamic factor describe the all variation that is found in both series. The remaining part of each series is described by the specific dynamic factor.

Mathematically, this can be written as:

$$
 \left[\begin{array}{c}n_{1,t}\\n_{2,t}\end{array}\right] = \left[\begin{array}{c}x_{s,1}\\x_{s,2}\end{array}\right]_t + \left[\begin{array}{c}\gamma_{1}\\ \gamma_{2}\end{array}\right] x_{c,t}
$$

where $\gamma_1$ and $\gamma_2$ are the factor loadings for series 1 resp. series 2. These factor loadings describe how the series $n_1$ and $n_2$ are related to the common dynamic factor.

The specific dynamic factors $x_s$ and common dynamic factor $x_c$ can be described by an AR(1) model as:

$$
\begin{align}
\mathbf{x}_{s,t}&=\left[\begin{array}{cc}\phi_{s,1} & 0\\0 & \phi_{s,2}\end{array}\right]\mathbf{x}_{s,t-1}+\left[\begin{array}{c}\eta_{s,1}\\\eta_{s,2}\end{array}\right]_t\\
x_{c,t}&=\phi_c x_{c,t-1}+\eta_{c,t}
\end{align}
$$

The model can also be written in a single matrix notation as:

$$
\begin{align}
\mathbf{x}_{t}&=\Phi \mathbf{x}_{t-1}+\mathbf{\eta}_{t}\\
\mathbf{n}_{t}&=\mathbf{Z} \mathbf{x}_{t}
\end{align}
$$

with the state vector
$
\mathbf{x}=\left[\begin{array}{c}x_{s,1}\\x_{s,2}\\x_{c,1}\end{array}\right]$,
the transition matrix $\mathbf{\Phi}=\left[\begin{array}{ccc}\phi_{s,1} & 0 & 0\\0 & \phi_{s,2} & 0\\0 & 0 & \phi_{c}\end{array}\right]$, the transition noise vector $\mathbf{\eta}=\left[\begin{array}{c}\eta_{s,1}\\ \eta_{s,2}\\ \eta_{c}\end{array}\right]$, and the observation matrix $\mathbf{Z}=\left[\begin{array}{ccc}1&0&\gamma_1\\0&1&\gamma_2\end{array}\right]$.

When analyzing more than two series, multiple common dynamic factors may be used. In that case, the equation for the common dynamic factor also becomes a vector equation.

## 4. Standardization
With the DFM we want to describe the common and specifc dynamics based on the correlation  rather than the covariance structure. Therefore, all series are standardized as:

$$\tilde{n}_{i,t} = \frac{n_{i,t}-\mu_{n_i}}{\sigma_{n_i}}$$

This standardization is done internally in `metran`, so there is no need to perform any standardization beforehand. However, as an illustration, the code below shows the standardized series.

```python
mt = metran.Metran(s)
series_std = mt.standardize(s)
series_std.plot(figsize=(10, 2), xlabel='Date').set_ylim(-4, 4);
```

## 5. Running the model
Let us now run the model for the generate time series. In this example, we solve the model with `report=False`. This means that no report is shown. Instead, we analyze the results step by step.

```python
mt = metran.Metran(s)
mt.solve(report=False)
```

    INFO: Number of factors according to Velicer's MAP test: 1

### 5.1 Factors, communality and specificity

Metran first determines the optimal number of common dynamic factors based on the correlation structure of the time series. For this, the Minimum Average Partial (MAP) test is used ([Velicer, 1976](#References); [Velicer et al., 2000](#References)). If this test results in 0 factors, then a second test is done based on the Kaiser criterion ([Kaiser, 1960](#References)). In this case, as we can see above, 1 factor has been selected to describe the common dynamics.

Besides, Metran estimates the factor loadings $\gamma_1$ and $\gamma_2$ using the minimum residual (minres) algorithm ([Harman and Jones, 1966](#References)).

```python
print('Factors:\n', mt.factors)
```

    Factors:
     [[0.93540765]
     [0.93540765]]

As described in [section 3](#dfm), the factor loadings show the degree to which a factor elaborates a variable (observed series). The sum of squared factor loadings for all common factors for a given series is referred to as the communality. The communality measures the fraction of variance in a given variable explained by all common factors jointly, or in our case, one common factor.

```python
print('Communality:', mt.get_communality())
```

    Communality: [0.87498746 0.87498746]

The fraction that is unique/specific for each series is referred to as the specificity and is calculated as (1 - communality).

```python
print('Specificity:', mt.get_specificity())
```

    Specificity: [0.12501254 0.12501254]

### 5.2 Estimating AR(1) parameters
After the number of factors and associated factor loadings have been estimated, Metran uses an optimization algorithm to estimate the AR(1) model parameters $\phi_{s,1}$, $\phi_{s,2}$, and $\phi_{c}$. Similar to the AR parameter is `pastas`, $\phi$ is written as:

$$
\phi_k=e^{−\Delta t_i/\alpha_k}
$$

and $\alpha_k$ is being estimated.

As all series have been standardized, the variance of each series is equal to 1. In addtion, we know the communality (and specificity) for each series, which means that we know the variance of the specific and common dynamic factors. As a result, the noise variance parameters of the AR(1) model do not need to be estimated. Instead, Metran calculates them as:

$$
\begin{align}
q_{s,1} &= \left(1-\phi_{s,1}^2\right) \cdot s_1 \\
q_{s,2} &= \left(1-\phi_{s,2}^2\right) \cdot s_2 \\
q_{c} &= \left(1-\phi_{c}^2\right)
\end{align}
$$

with $s_1$ and $s_2$ the specificity of series 1 resp. series 2.

The results of the parameter estimation process can be shown using `mt.fit_report()`.

```python
print(mt.fit_report())
```

    Fit report Cluster                Fit Statistics
    ====================================================
    tmin     None                     obj     2262.45
    tmax     None                     nfev         64
    freq     D                        AIC     2268.45
    solver   ScipySolve

    Parameters (3 were optimized)
    ====================================================
                          optimal   stderr initial  vary
    series 1_sdf_alpha   4.480161   ±7.14%      10  True
    series 2_sdf_alpha  19.342573  ±10.38%      10  True
    cdf1_alpha            9.24052   ±3.60%      10  True

    Parameter correlations |rho| > 0.5
    ====================================================
    series 1_sdf_alpha cdf1_alpha -0.63

### 5.3 Metran report
Further output of the Metran model parameters and statistics is given by `mt.metran_report()`. The following results are shown:
- nfct: number of factors
- fep: percentage of total variance explained by these factors
- communality for each series: percentage of variance that a series has in common with other series.
- state parameters:
 - AR(1) parameter $\phi$, calculated from the optimized parameter $\alpha$
 - variance $q$ of white noise process $\eta$
- observation parameters:
 - factor loadings $\gamma$ for each factor and series
 - scale: standard deviation $\sigma_n$ of each series (used for standardization, see [section 4](#standardization))
 - mean: mean $\mu_n$ of each series (used for standardization, see [section 4](#standardization))
- state correlations: correlation between specific and/or common dynamic factors

```python
print(mt.metran_report())
```

    Metran report Cluster        Factor Analysis
    ============================================
    tmin     None                nfct    1
    tmax     None                fep     93.61%
    freq     D

    Communality
    ============================================

    series 1      87.50%
    series 2      87.50%

    State parameters
    ============================================
                       phi         q
    series 1_sdf  0.799950  0.045015
    series 2_sdf  0.949614  0.012280
    cdf1          0.897431  0.194618

    Observation parameters
    ============================================
                    gamma1     scale      mean
    series 1      0.935408  4.769178 -0.847064
    series 2      0.935408  4.928326 -0.868526

    State correlations |rho| > 0.5
    ============================================
    series 1_sdf series 2_sdf -0.6

The statistic `fep` is based on the eigenvalues of the correlation matrix. The eigenvalues can be retrieved from the `metran` class.

```python
mt.eigval
```

    array([1.87212635, 0.12787365])

The sum of the eigenvalues always equals the dimension of the correlation matrix, in this case 2.

```python
round(mt.eigval.sum())
```

    2

As we have used 1 eigenvalue (`nfct` = 1), the statistic `fep` is calculated as:

```python
round(100 * mt.eigval[0] / mt.eigval.sum(), 2)
```

    93.61

## 6. Checking the estimated AR(1) parameters
We can compare the estimate AR(1) parameters $\phi$ with the AR(1) parameters used to generate the time series.

```python
print(np.round(np.diagonal(mt.get_transition_matrix()), 2), 'vs', phi)
```

    [0.8  0.95 0.9 ] vs [0.8  0.95 0.9 ]

The estimated parameters are close to those being used to generate the synthetic series, which means that the model has estimated the autoregression of the latent components well.

## 7. Decomposition of series
The specific dynamic components (sdf's) $x_{s,1}$ and $x_{s,2}$ can be retrieved from the state vector $\mathbf{x}$.

```python
mt.get_state_means().plot(figsize=(10, 2),
                          xlabel='Date',
                          title='Specific and common dynamic factors'
                          ).set_ylim(-4, 4);
```

Note that the common factor need to be multiplied by the factor loadings, to get the common factor for each series. Furthermore, these results are for the standardized series and need to be rescaled to obtain the unstandardized dynamic factors. Metran has a specific method to obtain the specific and common dynamic factors for each series.

```python
mt.decompose_simulation(name='series 1').plot(
    figsize=(10, 2), xlabel='Date',
    title='Specific and common dynamic factor for series 1');
```

We can compare the calculated specificity with the variance of the specific dynamic component divided by the series variance (which is the sum of the specific and common dynamic factor).

```python
sim1 = mt.decompose_simulation(name='series 1')
sdf1_variance = sim1['sdf'].var() / sim1.sum(axis=1).var()
print('Variance sdf series 1:', "{:.2f}%".format(100 * sdf1_variance))
print('Specificity series 1 :', "{:.2f}%".format(100 * mt.get_specificity()[0]))
```

    Variance sdf series 1: 8.51%
    Specificity series 1 : 12.50%

Theoretically, these values must be equal. In practice, they may slightly differ, e.g. due to some correlation between the specific and common dynamic factor. We can test this by calculating the correlation.

```python
sim1.corr()
```

      sdf
      cdf1

      sdf
      1.000000
      0.162154

      cdf1
      0.162154
      1.000000

Similar to series 1, we can decompose series 2 and compare the associated specificity and communality.

```python
mt.decompose_simulation(name='series 2').plot(
    figsize=(10, 2), xlabel='Date',
    title='Specific and common dynamic factor for series 2');
```

```python
sim2 = mt.decompose_simulation(name='series 2')
sdf2_variance = sim2['sdf'].var() / sim2.sum(axis=1).var()
print('Variance sdf series 2:', "{:.2f}%".format(100 * sdf2_variance))
print('Specificity series 2 :', "{:.2f}%".format(100 * mt.get_specificity()[1]))
```

    Variance sdf series 2: 7.48%
    Specificity series 2 : 12.50%

```python
sim2.corr()
```

      sdf
      cdf1

      sdf
      1.000000
      0.193686

      cdf1
      0.193686
      1.000000

## References

- Berendrecht, W.L., F.C. van Geer, 2016. A dynamic factor modeling framework for analyzing multiple groundwater head series simultaneously, Journal of Hydrology, 536, pp. 50-60, [DOI](http://dx.doi.org/10.1016/j.jhydrol.2016.02.028).
- Harman, H., Jones, W., 1966. Factor analysis by minimizing residuals (minres). Psychometrika 31, 351–368.
- Kaiser, H.F., 1960. The application of electronic computers to factor analysis. Educ. Psychol. Meas. 20, 141–151.
- Velicer, W.F., 1976. Determining the number of components from the matrix of partial correlations. Psychometrika 41, 321–327.
- Velicer, W.F., Eaton, C.A., Fava, J.L., 2000. Construct explication through factor or component analysis: a review and evaluation of alternative procedures for determining the number of factors or components. In: Goffin, R., Helmes, E. (Eds.), Problems and Solutions in Human Assessment. Springer, US, pp. 41–71.

```python

```


```
from sympy import I, Matrix, symbols, Symbol, eye
from datetime import datetime
import numpy as np
import pandas as pd

```

```
'''
Rotas
'''
R1 = Matrix([[0,"L1_p1",0,0,0,0,0],[0,0,"L1_p3",0,0,0,0],[0,0,0,"L1_v1",0,0,0],[0,0,0,0,"L1_p4",0,0],[0,0,0,0,0,"L1_v3",0],[0,0,0,0,0,0,"L1_v4"],[0,0,0,0,0,0,0]])
R2 = Matrix([[0,"L2_p2",0,0,0,0,0],[0,0,"L2_p3",0,0,0,0],[0,0,0,"L2_v2",0,0,0],[0,0,0,0,"L2_p5",0,0],[0,0,0,0,0,"L2_v3",0],[0,0,0,0,0,0,"L2_v5"],[0,0,0,0,0,0,0]])
R3 = Matrix([[0,"L3_p5",0,0,0,0,0],[0,0,"L3_p3",0,0,0,0],[0,0,0,"L3_v5",0,0,0],[0,0,0,0,"L3_p1",0,0],[0,0,0,0,0,"L3_v3",0],[0,0,0,0,0,0,"L3_v1"],[0,0,0,0,0,0,0]])
```

```
'''
Seções de bloqueio
'''
T1 = Matrix([[0, "p1"],["v1", 0]])
T2 = Matrix([[0, "p2"],["v2", 0]])
T3 = Matrix([[0, "p3"],["v3", 0]])
T4 = Matrix([[0, "p4"],["v4", 0]])
T5 = Matrix([[0, "p5"],["v5", 0]])
```

```
def kronSum(A,B):
  m = np.size(A,1)
  n = np.size(B,1)
  A = np.kron(A, np.eye(n))
  B = np.kron(np.eye(m),B)
  return A + B
```

```
momento_inicio = datetime.now()
'''
Algebra de rotas
'''
rotas = kronSum(R1,R2)
rotas = kronSum(rotas,R3)

'''
Algebra de seções
'''
secoes = kronSum(T1,T2)
secoes = kronSum(secoes,T3)
secoes = kronSum(secoes,T4)
secoes = kronSum(secoes,T5)
'''
Algebra de sistema
'''
sistema = np.kron(rotas, secoes)
# calcula tempo de processamento
tempo_processamento = datetime.now() - momento_inicio
```

```
sistema = pd.DataFrame(data=sistema,index=list(range(1,np.size(sistema,0)+1)), columns=list(range(1,np.size(sistema,1)+1)))
```

```
sistema.shape
```

    (10976, 10976)

```
print(tempo_processamento)
```

    0:03:10.028234

```
sistema
```

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      ...
      10937
      10938
      10939
      10940
      10941
      10942
      10943
      10944
      10945
      10946
      10947
      10948
      10949
      10950
      10951
      10952
      10953
      10954
      10955
      10956
      10957
      10958
      10959
      10960
      10961
      10962
      10963
      10964
      10965
      10966
      10967
      10968
      10969
      10970
      10971
      10972
      10973
      10974
      10975
      10976

      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      1.0*L3_p5*p5
      1.0*L3_p5*p4
      0
      1.0*L3_p5*p3
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      2
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      1.0*L3_p5*v5
      0
      0
      1.0*L3_p5*p4
      0
      1.0*L3_p5*p3
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      3
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      1.0*L3_p5*v4
      0
      0
      1.0*L3_p5*p5
      0
      0
      1.0*L3_p5*p3
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      4
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      1.0*L3_p5*v4
      1.0*L3_p5*v5
      0
      0
      0
      0
      1.0*L3_p5*p3
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      5
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      1.0*L3_p5*v3
      0
      0
      0
      0
      1.0*L3_p5*p5
      1.0*L3_p5*p4
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      10972
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      10973
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      10974
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      10975
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      10976
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

10976 rows × 10976 columns

```
momento_inicio = datetime.now()
colunas = ['denode', 'paranode', 'aresta']
grafo = pd.DataFrame(columns=colunas)
r = 1
c = 1
for j in range(np.size(sistema,0)):
  for i in range(np.size(sistema,0)):
    if sistema.loc[r,c]==0 and c < np.size(sistema,0):
      c += 1
    elif c < np.size(sistema,0):
      grafo.loc[len(grafo)+1] = (r, c, sistema.loc[r,c])
      c += 1
    else:
      c = 1
      r += 1
tempo_processamento = datetime.now() - momento_inicio
print(tempo_processamento)
```

    1:08:39.152899

```
grafo
```

      denode
      paranode
      aresta

      1
      1
      34
      1.0*L3_p5*p5

      2
      1
      35
      1.0*L3_p5*p4

      3
      1
      37
      1.0*L3_p5*p3

      4
      1
      41
      1.0*L3_p5*p2

      5
      1
      49
      1.0*L3_p5*p1

      ...
      ...
      ...
      ...

      141101
      10944
      10960
      1.0*L3_v1*v1

      141102
      10944
      10968
      1.0*L3_v1*v2

      141103
      10944
      10972
      1.0*L3_v1*v3

      141104
      10944
      10974
      1.0*L3_v1*v4

      141105
      10944
      10975
      1.0*L3_v1*v5

141105 rows × 3 columns

```
grafo['aresta'] = grafo['aresta'].astype('str')
grafo
```

      denode
      paranode
      aresta

      1
      1
      34
      1.0*L3_p5*p5

      2
      1
      35
      1.0*L3_p5*p4

      3
      1
      37
      1.0*L3_p5*p3

      4
      1
      41
      1.0*L3_p5*p2

      5
      1
      49
      1.0*L3_p5*p1

      ...
      ...
      ...
      ...

      141101
      10944
      10960
      1.0*L3_v1*v1

      141102
      10944
      10968
      1.0*L3_v1*v2

      141103
      10944
      10972
      1.0*L3_v1*v3

      141104
      10944
      10974
      1.0*L3_v1*v4

      141105
      10944
      10975
      1.0*L3_v1*v5

141105 rows × 3 columns

```
new = grafo["aresta"].str.split("*", n = -1, expand = True)
grafo["aresta"]=new[1]
grafo["semaforo_secao"]=new[2]
new = grafo["aresta"].str.split("_", n = -1, expand = True)
grafo["trem"]=new[0]
grafo["semaforo_trem"]=new[1]
grafo['coincide'] = np.where(grafo['semaforo_secao']==grafo['semaforo_trem'], True, False)
grafo
```

      denode
      paranode
      aresta
      semaforo_secao
      trem
      semaforo_trem
      coincide

      1
      1
      34
      L3_p5
      p5
      L3
      p5
      True

      2
      1
      35
      L3_p5
      p4
      L3
      p5
      False

      3
      1
      37
      L3_p5
      p3
      L3
      p5
      False

      4
      1
      41
      L3_p5
      p2
      L3
      p5
      False

      5
      1
      49
      L3_p5
      p1
      L3
      p5
      False

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      141101
      10944
      10960
      L3_v1
      v1
      L3
      v1
      True

      141102
      10944
      10968
      L3_v1
      v2
      L3
      v1
      False

      141103
      10944
      10972
      L3_v1
      v3
      L3
      v1
      False

      141104
      10944
      10974
      L3_v1
      v4
      L3
      v1
      False

      141105
      10944
      10975
      L3_v1
      v5
      L3
      v1
      False

141105 rows × 7 columns

```
grafo = pd.DataFrame(data=grafo)
```

```
grafo.drop(grafo[grafo.coincide == False].index, inplace=True)
grafo.reset_index(drop = True)
grafo.index = np.arange(1, len(grafo) + 1)
grafo
```

      denode
      paranode
      aresta
      semaforo_secao
      trem
      semaforo_trem
      coincide

      1
      1
      34
      L3_p5
      p5
      L3
      p5
      True

      2
      1
      233
      L2_p2
      p2
      L2
      p2
      True

      3
      1
      1585
      L1_p1
      p1
      L1
      p1
      True

      4
      2
      234
      L2_p2
      p2
      L2
      p2
      True

      5
      2
      1586
      L1_p1
      p1
      L1
      p1
      True

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      14108
      10940
      10956
      L3_v1
      v1
      L3
      v1
      True

      14109
      10941
      10957
      L3_v1
      v1
      L3
      v1
      True

      14110
      10942
      10958
      L3_v1
      v1
      L3
      v1
      True

      14111
      10943
      10959
      L3_v1
      v1
      L3
      v1
      True

      14112
      10944
      10960
      L3_v1
      v1
      L3
      v1
      True

14112 rows × 7 columns

```
# Passo 1:
alcancavel = [1]
r = 1
for i in range(np.size(grafo,0)):
  de = grafo.loc[r]['denode']
  para = grafo.loc[r]['paranode']
  if (de in alcancavel):
    alcancavel.append(para)
    r = r + 1
  else:
    r = r + 1
```

```
alcancavel.sort()
```

```
grafo = grafo[grafo.denode.isin(alcancavel)]
grafo
```

      denode
      paranode
      aresta
      semaforo_secao
      trem
      semaforo_trem
      coincide

      1
      1
      34
      L3_p5
      p5
      L3
      p5
      True

      2
      1
      233
      L2_p2
      p2
      L2
      p2
      True

      3
      1
      1585
      L1_p1
      p1
      L1
      p1
      True

      52
      34
      70
      L3_p3
      p3
      L3
      p3
      True

      53
      34
      266
      L2_p2
      p2
      L2
      p2
      True

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      14034
      10786
      10822
      L3_p3
      p3
      L3
      p3
      True

      14051
      10822
      10853
      L3_v5
      v5
      L3
      v5
      True

      14069
      10853
      10901
      L3_p1
      p1
      L3
      p1
      True

      14089
      10901
      10929
      L3_v3
      v3
      L3
      v3
      True

      14097
      10929
      10945
      L3_v1
      v1
      L3
      v1
      True

300 rows × 7 columns

```
grafo.reset_index(drop = True)
grafo.index = np.arange(1, len(grafo) + 1)
grafo
```

      denode
      paranode
      aresta
      semaforo_secao
      trem
      semaforo_trem
      coincide

      1
      1
      34
      L3_p5
      p5
      L3
      p5
      True

      2
      1
      233
      L2_p2
      p2
      L2
      p2
      True

      3
      1
      1585
      L1_p1
      p1
      L1
      p1
      True

      4
      34
      70
      L3_p3
      p3
      L3
      p3
      True

      5
      34
      266
      L2_p2
      p2
      L2
      p2
      True

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      296
      10786
      10822
      L3_p3
      p3
      L3
      p3
      True

      297
      10822
      10853
      L3_v5
      v5
      L3
      v5
      True

      298
      10853
      10901
      L3_p1
      p1
      L3
      p1
      True

      299
      10901
      10929
      L3_v3
      v3
      L3
      v3
      True

      300
      10929
      10945
      L3_v1
      v1
      L3
      v1
      True

300 rows × 7 columns

```
# Passo 2:
ultimo = alcancavel[-1]
alcancavel = [ultimo]
r = np.size(grafo,0)
for i in range(np.size(grafo,0),1,-1):
  para = grafo.loc[r]['paranode']
  de = grafo.loc[r]['denode']
  if (para in alcancavel):
    alcancavel.append(de)
    r = r - 1
  else:
    r = r - 1

```

```
grafo = grafo[grafo.paranode.isin(alcancavel)]
grafo
```

      denode
      paranode
      aresta
      semaforo_secao
      trem
      semaforo_trem
      coincide

      1
      1
      34
      L3_p5
      p5
      L3
      p5
      True

      2
      1
      233
      L2_p2
      p2
      L2
      p2
      True

      3
      1
      1585
      L1_p1
      p1
      L1
      p1
      True

      4
      34
      70
      L3_p3
      p3
      L3
      p3
      True

      5
      34
      266
      L2_p2
      p2
      L2
      p2
      True

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      296
      10786
      10822
      L3_p3
      p3
      L3
      p3
      True

      297
      10822
      10853
      L3_v5
      v5
      L3
      v5
      True

      298
      10853
      10901
      L3_p1
      p1
      L3
      p1
      True

      299
      10901
      10929
      L3_v3
      v3
      L3
      v3
      True

      300
      10929
      10945
      L3_v1
      v1
      L3
      v1
      True

266 rows × 7 columns

```
grafo.reset_index(drop = True)
grafo.index = np.arange(1, len(grafo) + 1)
grafo

```

      denode
      paranode
      aresta
      semaforo_secao
      trem
      semaforo_trem
      coincide

      1
      1
      34
      L3_p5
      p5
      L3
      p5
      True

      2
      1
      233
      L2_p2
      p2
      L2
      p2
      True

      3
      1
      1585
      L1_p1
      p1
      L1
      p1
      True

      4
      34
      70
      L3_p3
      p3
      L3
      p3
      True

      5
      34
      266
      L2_p2
      p2
      L2
      p2
      True

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      262
      10786
      10822
      L3_p3
      p3
      L3
      p3
      True

      263
      10822
      10853
      L3_v5
      v5
      L3
      v5
      True

      264
      10853
      10901
      L3_p1
      p1
      L3
      p1
      True

      265
      10901
      10929
      L3_v3
      v3
      L3
      v3
      True

      266
      10929
      10945
      L3_v1
      v1
      L3
      v1
      True

266 rows × 7 columns

```
# Passo 3
contra = [["v1","p1"],["v2","p2"],["v3","p3"],["v4","p4"],["v5","p5"]]
contra = pd.DataFrame(contra,columns=["operador_v","operador_p"])
v = grafo
v = v.loc[v['semaforo_trem'].isin(contra['operador_v'])]
v = v.drop(columns=["aresta", "semaforo_secao", "coincide"])
v = v.rename(columns={"denode": "noh_anterior","semaforo_trem": "operador_v", "paranode":"denode"})
v = v.merge(contra, on="operador_v")
v = grafo.merge(v, on="denode")
v['coincide'] = np.where(v['semaforo_secao']==v['operador_p'], True, False)
v.drop(v[v.coincide == False].index, inplace=True)
v.reset_index(drop = True)
v.index = np.arange(1, len(v) + 1)
v = v.drop(columns=["aresta","semaforo_secao", "coincide", "operador_p"])
v = v.rename(columns={"trem_x": "trem_atual", "semaforo_trem":"semaforo_atual", "trem_y": "trem_anterior", "operador_v": "semaforo_anterior"})
nohs_sincroniza = v
```

```
nohs_sincroniza
```

      denode
      paranode
      trem_atual
      semaforo_atual
      noh_anterior
      trem_anterior
      semaforo_anterior

      1
      193
      1777
      L1
      p1
      177
      L3
      v1

      2
      409
      637
      L2
      p3
      381
      L3
      v3

      3
      425
      2009
      L1
      p1
      409
      L3
      v1

      4
      653
      2237
      L1
      p1
      637
      L3
      v1

      5
      869
      2453
      L1
      p1
      853
      L3
      v1

      6
      1094
      2678
      L1
      p1
      1078
      L3
      v1

      7
      1314
      2898
      L1
      p1
      1298
      L3
      v1

      8
      1345
      1378
      L3
      p5
      1122
      L2
      v5

      9
      1537
      3121
      L1
      p1
      1521
      L3
      v1

      10
      2706
      4278
      L1
      p3
      2486
      L2
      v3

      11
      2898
      4470
      L1
      p3
      2678
      L2
      v3

      12
      2929
      2962
      L3
      p5
      2706
      L2
      v5

      13
      4501
      4534
      L3
      p5
      4278
      L2
      v5

      14
      6053
      6086
      L3
      p5
      5830
      L2
      v5

      15
      7623
      7656
      L3
      p5
      7400
      L2
      v5

      16
      7876
      7912
      L3
      p3
      6312
      L1
      v3

      17
      8075
      8303
      L2
      p3
      6511
      L1
      v3

      18
      8108
      8144
      L3
      p3
      6544
      L1
      v3

      19
      8251
      8479
      L2
      p3
      8223
      L3
      v3

      20
      8267
      8495
      L2
      p3
      6703
      L1
      v3

      21
      9187
      9220
      L3
      p5
      8964
      L2
      v5

      22
      9220
      9256
      L3
      p3
      7656
      L1
      v3

      23
      9817
      10045
      L2
      p3
      9789
      L3
      v3

      24
      10753
      10786
      L3
      p5
      10530
      L2
      v5

```
# Passo 4
```

```
'''grafo.to_csv('grafo.csv', sep=";")
from google.colab import files
files.download('grafo.csv')'''
```

    'grafo.to_csv(\'grafo.csv\', sep=";")\nfrom google.colab import files\nfiles.download(\'grafo.csv\')'


```python
%matplotlib inline
from sympy import *
from sympy import besselk
import numpy as np
from scipy.special import kn, jv
from scipy.integrate import quad
import matplotlib.pyplot as plt
from matplotlib.pyplot import cm
init_printing(use_unicode=True)
```

```python
x, y, r, R, theta_s, phi = symbols('x y r R theta_s phi')
```

```python
theta_s = 0.2
R = 1
```

```python
def nx(x, y, R, theta_s):
    return -theta_s*(R*y/(x**2 + y**2))

def ny(x, y, R, theta_s):
    return theta_s*(R*x/(x**2 + y**2))
```

```python
nx(x, y, R, theta_s), ny(x, y, R, theta_s)
```

```python
Y, X = np.mgrid[-3:3:15j, -3:3:15j]
NX = -theta_s*(R*Y/(X**2 + Y**2))
NY = theta_s*(R*X/(X**2 + Y**2))
U = np.sqrt(NX**2 + NY**2)
```

    /usr/local/lib/python3.4/dist-packages/IPython/kernel/__main__.py:2: RuntimeWarning: invalid value encountered in true_divide
      from IPython.kernel.zmq import kernelapp as app
    /usr/local/lib/python3.4/dist-packages/IPython/kernel/__main__.py:3: RuntimeWarning: invalid value encountered in true_divide
      app.launch_new_instance()

```python
plot1 = plt.figure(figsize=(11,8))
plt.quiver(X, Y, NX/U, NY/U, U, cmap=cm.copper, headlength=3)

plt.colorbar()

plt.title('Vector Field')
plt.show(plot1)
```

```python

```


#### _Speech Processing Labs 2020: Module 1_

```python
## Run this first!
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

import cmath
from math import floor
from matplotlib.animation import FuncAnimation
from IPython.display import HTML
plt.style.use('ggplot')

from dspMisc import *
```

# 5 Interpreting the  Discrete Fourier Transform

### Learning Outcomes
* Understand how sampling rate effects the DFT output
* Understand what the DFT leakage is.

### Need to know
* Topic Videos: Fourier Analysis, Frequency Domain
* [Digital Signals: Sampling sinusoids](./sp-m1-3-sampling-sinusoids.ipynb)
* [The Discrete Fourier Transform](./sp-m1-4-discrete-fourier-transform.ipynb) _(desirable)_

Equation alert: If you're viewing this on github, please note that the equation rendering is not always perfect. You should view the notebooks through a jupyter notebook server for an accurate view.

## 5.1 A Very Quick DFT recap
The previous worksheet went through the mechanics of the Discrete Fourier Transform (DFT).  The DFT input and output are broadly:

* **Input:**  $N$ amplitude samples over time
    * $x[n]$, for $n=1..N$ (i.e. a time series of $N$ samples)

* **Output:** the dot product (i.e., the similiarity) between the input and $N$ sinusoids with different frequencies
    * DFT[k] $= Me^{j\phi}$, i.e. a complex number with **magnitude** $M$ and **phase** angle $\phi$

The outputs are calculated using the following formula for $k=0,...N-1$.

$$
\begin{align}
DFT[k] &= \sum_{n=0}^{N-1} x[n] e^{-j \frac{2\pi nk}{N}} \\
&= \sum_{n=0}^{N-1} x[n]\big[\cos\big(\frac{2\pi nk}{N} \big) - j \sin\big(\frac{2\pi nk}{N} \big) \big]
\end{align}
$$

We saw how the complex numbers in this equation $e^{-j2\pi nk/N}$ for $k=1...N-1$ represent sinusoids with specific frequencies as projections of phasors. You can just think of a phasor as an analogue clock with one hand (vector) ticking (rotating) around a clockface (i.e. a circle), where the length of the hand is the amplitude of that wave, and how fast it goes around the clock is it's frequency.

Each DFT[k] output essentially tells us how strongly the input is repeats itself (i.e. is periodic) at the frequency of the $k$th DFT phasor. So, we talk about the DFT outputs as providing a **frequency response**.

Since the the DFT outputs are complex numbers, we can talk about them in terms of magnitude and phase angle: The magnitude of DFT[k] tells us how much we'd weight the $k$-th phasor if we were to try to reconstruct the original input by adding all the DFT phasors together.  The phase of DFT[k] tells use whether we need to shift that wave along the time axis.

## 5.2 The DFT Frequency Response:  Which Frequencies?

You will have noticed in the previous worksheet that we only talked about relative frequencies above, e.g. the $e^{-j2\pi nk/N}$ phasor has $k$ times the frequency of the $e^{-j2\pi n/N}$ phasor.

Since the frequencies of the DFT phasors are all multiples of the $k=1$ phasor, we just have to figure out the frequency of that phasor and we'll know all the rest. Now, the $k=1$ phasor takes $N$ steps to complete a full cycle of the unit circle.  So, it's frequency depends on how long each of these steps takes.  That is, the **sampling rate**.

Assume we are sampling at a rate of $f_s$ samples per second. Then, we can calculate:

* How long between each sample (i.e. the **sampling time**):
    * $t_s = 1/f_s$  (seconds)

* How long does it take to take $N$ samples?
    * $t_s \times N$  (seconds x samples = seconds)

* How long will it take the $k=1$ phasor to make 1 complete cycle?
    * $T_1 = t_s \times N$  (seconds)
    * This is the **period** or **wavelength** of the phasor

* What is the **frequency** of the $k=1$ phasor?
    * $f_{min} = 1/T_1 = 1/(t_s N) $ (cycles/second)

* What is the **frequency** of the $k$th phasor?
    * $kf_{min}$ (cycles/second)

So, the important thing to remember is that the DFT outputs depend on:

* The **number of samples** in the input sequence, $N$
* The **sampling rate**, $f_s$ samples/second

### Example
Assume we have a sampling rate of $f_s = 32$ samples/second, and an input length of $N=16$.

We can then calculate the $k=1$ DFT output frequency by noting the following:

* The sampling time is: $t_s = 1/f_s = 1/32$ seconds
* So, the time it takes to complete one cycle is: $T_1 = t_sN = 1/32 \times 16 = 1/2$ seconds
* So, the frequency of DFT[1] is $f_{min} = 1/1/2 = 2$ cycles/second

### Exercise

* What's the frequency of the $k=5$ phasor in the previous example?

### Notes

```python

```

### Exercise

Try running this code which sets the input to be a sine wave with the same frequency as the the 1st DFT component. That is, `f_in = 1/wavlen_in`.

```python
N=64

#sampling rate:
f_s = 128

## sample time
t_s = 1/f_s

## Notice that the DFT component frequencies as completely determined by the number of samples N
## and the sampling frequency f_s.
## So if we wanted the input frequency to exactly match the frequency of the 1st DFT component we
## could just do this:

wavlen_in = t_s*N
f_in = 1/wavlen_in

print("f_s: %f\nt_s: %f\nwavlen: %f\nf_in %f" % (f_s, t_s, wavlen_in, f_in))

## indices of input sequence of size N
nsteps = np.array(range(N))

samples_per_cycle = wavlen_in/t_s
print("samples per cycle %f" % samples_per_cycle)

## Generate a sine wave with f_in matching the frequency of the k=1 DFT phasor
## You can look at the code for this function in the file XXX.py

x, time_steps = gen_sinusoid(frequency=f_in, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.sin)

## Plot the input
fig, timedom = plt.subplots(figsize=(16, 4))
timedom.scatter(time_steps, x, color='magenta')
timedom.plot(time_steps, x, color='magenta')
timedom.set_xlabel("time (s)")
timedom.set_ylabel("Amplitude")
timedom.set_title("Input: $x[n]$ in the time domain")
```

```python
## Calculate the frequencies associated with each DFT output
def get_dft_freqs_all(sample_rate, seq_len):
    ## Get the minimum frequency
    f_min = sample_rate/seq_len

    ## All the other frequencies are just integer multiples of the minimum frequency
    ## We have 0,...,seq_len-1 outputs
    dft_outs = np.arange(seq_len)

    return f_min * dft_outs
```

```python
## Find the DFT output frequencies for our current sample rate and input length
dft_freqs = get_dft_freqs_all(sample_rate=f_s, seq_len=N)
print(dft_freqs)

```

```python
## Apply the DFT to the input and return the magnitudes and phases for each DFT output
mags, phases = get_dft_mag_phase(x, seq_len=N)

## Plot the magnitudes
fig, fdom = plt.subplots(figsize=(16, 4))
fdom.scatter(dft_freqs, mags)
fdom.set_xlabel("Frequency (Hz)")
fdom.set_ylabel("Magnitude")
fdom.set_title("Magnitude response")

#print(mags)

## Plot the phases
fig, fdom = plt.subplots(figsize=(16, 4))
fdom.scatter(dft_freqs, phases)
fdom.set_xlabel("Frequency (Hz)")
fdom.set_ylabel("Phase angle (radians)")
fdom.set_title("Phase response")

#print(phases)

```

### Exercise

* Is the output what you'd expect given the input?
* Why is the phase non-zero for the DFT output's 1 and 63?
* What happens if you change the `amplitude` of the input sine wave? What about the `phase`?
* What happens to the range of DFT output frequencies if you increase the sampling rate `f_s` but keep the input sequence length `N` fixed?
* What happens to the range of DFT output frequencies for $k > N/2$? Can the DFT actually model those frequencies?

### Notes

```python

```

## 5.3 DFT on a compound waveform

Now, let's see what happens when we use a compound waverform as input to the DFT.  First we need to set the length of the sequence and the sample rate.

```python
## Set the number of samples N, sampling rate f_s
N=64

#sampling rate:
f_s = 64

## sample time
t_s = 1/f_s

print("N: %d\nf_s: %f\nt_s: %f" % (N, f_s, t_s))
```

Let's create a compound waveform as input with frequency components of 4Hz and 12Hz:

```python
## Now let's create a compound waveform as input
## We're setting gen_function=np.cos, meaning that each of our sinusoids is a cosine wave
x1, time_steps = gen_sinusoid(frequency=4, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)

## Let's shift the phase of this one by 90 degrees (pi/2 rads)
x2, time_steps = gen_sinusoid(frequency=12, phase=np.pi/2, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)

x_compound = x1 + x2

## Plot the input
fig, timedom = plt.subplots(figsize=(16, 4))
timedom.scatter(time_steps, x_compound, color='magenta')
timedom.plot(time_steps, x_compound, color='magenta')
timedom.set_xlabel("time (s)")
timedom.set_ylabel("Amplitude")
timedom.set_title("A compound waveform: $x_compound$ in the time domain")
```

You should see a somewhat complicated waveform whose overall pattern repeats 4 times in a second.  Now we do the DFT and look at the magnitude and phase values for the different DFT outputs.

```python
## Do the DFT on the compound waveform as above:
mags, phases = get_dft_mag_phase(x_compound, seq_len=N)
dft_freqs = get_dft_freqs_all(sample_rate=f_s, seq_len=N)

## Plot the magnitudes
fig, fdom = plt.subplots(figsize=(16, 4))
#timedom.scatter(dft_freqs[:round(N)], mags[:round(N)])
fdom.scatter(dft_freqs[:round(N/2)], mags[:round(N/2)])
fdom.set_xlabel("Frequency (Hz)")
fdom.set_ylabel("Magnitude")
fdom.set_title("Magnitude response")
#print(mags)

## Plot the phases - let's just skip this for the moment
fig, fdom = plt.subplots(figsize=(16, 4))
fdom.scatter(dft_freqs[:round(N/2)], phases[:round(N/2)])
fdom.set_xlabel("Frequency (Hz)")
fdom.set_ylabel("Phase (rad)")
fdom.set_title("Phase response")
#print(phases)

```

### Exercise

* Do you see what you would expect given the input?
* Why does the number of non-zero phases differ from the number of non-zero magnitudes?

### Notes

```python

```

### Exercise

Explore the outputs of the DFT when you change the following parameters
* sampling rate `f_s`
* sequence length `N`
* frequency of the input sequence(s) `frequency` (in the function `gen_sinusoid`)

**Question**
* What happens if you set the frequency of one of the input components to be greater than the Nyquist frequency?
* What happens when the frequency of the input components doesn't match any of the DFT component frequencies?
(e.g. for the example above what if one of the input frequencies is not a whole number?)

### Notes

```python

```

## 5.4  Leakage

One of the main things to remember about the DFT is that you're calculating the correlation between the input and phasors with specific frequencies.  If your input exactly matches one of those phasor frequencies the magnitude response will show a positive magnitude for that phasor and zero for everything else.  However, if the input frequency falls between phasor frequencies (i.e. doesn't correspond to a specific DFT output), then you'll see spectral **leakage**.  The DFT outputs close to the input frequency will also get positive magnitudes, with the DFT output closest to the input frequency getting the highest magnitude.

The following code gives an example

```python
N=64
f_s = 64
f_in1 = 4.5 ## In between DFT output frequencies
f_in2 = 12  ## One of the DFT outputs

x1, time_steps = gen_sinusoid(frequency=f_in1, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)
x2, time_steps = gen_sinusoid(frequency=f_in2, phase=np.pi/2, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)
x_compound = x1  + x2

## Plot the input
fig, timedom = plt.subplots(figsize=(16, 4))
timedom.scatter(time_steps, x_compound, color='magenta')
timedom.plot(time_steps, x_compound, color='magenta')
timedom.set_xlabel("time (s)")
timedom.set_ylabel("Amplitude")
timedom.set_title("Leakage example (time domain)")

```

```python
## Do the DFT on the compound waveform as above:
mags, phases = get_dft_mag_phase(x_compound, seq_len=N)
dft_freqs = get_dft_freqs_all(sample_rate=f_s, seq_len=N)

## Plot the magnitudes
fig, fdom = plt.subplots(figsize=(16, 4))

## Just plot the first N/2 frequencies since we know that they are the mirrored for k>N/2
fdom.scatter(dft_freqs[:round(N/2)], mags[:round(N/2)])
fdom.set_xlabel("Frequency (Hz)")
fdom.set_ylabel("Magnitude")
fdom.set_title("Leakage example: Magnitude response")
#print(mags)

```

### Leakage as the normalized sinc function

Leakage makes the DFT harder to interpret. However, we can derive the shape that leakage will have from the the DFT equation and some algebra about rectangular functions.  It turns out that leakage for a particular frequency follows the normalized **sinc** function:

$$
\begin{align}
X(m) &= \Big|\frac{AN}{2} \cdot \mathop{sinc}(c-m)\Big|\\
&= \Big|\frac{AN}{2} \cdot \frac{\sin(\pi(c-m))}{2\pi(c-m))}\Big|\\
\end{align}
$$

Where $A$ is the peak amplitude of the input, $N$ is the input sequence length, $c$ is the number of cycles completed in the input sequence time.  If $c$ is a whole number we just get the usual DFT response, but if $c$ is not a whole number, we get a spread across output frequency bins.

Let's check whether the sinc function matches what we get in the DFT. First we write a function to evaluate the leakage function in between our DFT outputs.

```python
## Calculate the approximated leakage as the sinc function
def calc_leakage(freq,  sample_rate, seqlen, amplitude=1):
    sequence_time = (1/sample_rate)*seqlen

    ## number of cycles in input for component
    c = freq * sequence_time
    print("c=", c)

    ## Interpolate between DFT ouput indices
    ms = np.arange(0, seqlen, 0.05)

    ## Approximated response - we could actually just return a function here, but
    ## let's just keep things concrete for now.
    leakage = np.abs(amplitude * seqlen * 0.5 * np.sinc((c-ms)))

    return leakage, ms * (sample_rate/seqlen)

```

Now let's plot the leakage predicted for our two input components separately (top) and added together (bottom)

```python
## Calculate the leakage function for our know input wave components
leakage1, ms = calc_leakage(f_in1, f_s, N)
leakage2, ms = calc_leakage(f_in2, f_s, N)

## Plot the magnitude response and the leakage function for each of our input components
fig, fdom = plt.subplots(figsize=(16, 4))
fdom.set(xlim=(-1, N/2), ylim=(-1, N))
fdom.scatter(dft_freqs, mags)

fdom.plot(ms, leakage1)
fdom.plot(ms, leakage2)
fdom.set_xlabel("Frequency (Hz)")
fdom.set_ylabel("Magnitude")
fdom.set_title("Leakage function for each input component frequency")

## Plot the magnitude response and the sum of the leakage functions
fig, fdom = plt.subplots(figsize=(16, 4))
fdom.set(xlim=(-1, N/2), ylim=(-1, N))
fdom.scatter(dft_freqs, mags)
fdom.plot(ms, leakage1 + leakage2, color='C5')
fdom.set_xlabel("Frequency (Hz)")
fdom.set_ylabel("Magnitude")
fdom.set_title("Sum of leakage functions for input components")

## It fits, though not perfectly!
```

In the top figure, you should see that peaks (**main lobes**) of each leakage function are aligned with our input component frequencies. The peaks are at the same points as the DFT outputs when the sinusoidal component has a frequency matching the DFT output frequency (i.e. 12 above Hz).  Otherwise we see the spread of leakage around the input component frequency (i.e. around 4.5 Hz).

You'll also notice that our DFT magnitudes points don't always line up perfectly with our sinc curves.  Part of this is because the leakage function is an _approximation_.  Nevertheless, it's a very good approximation!

### Exercise

* What happens if the frequencies of the two components making the compound waveform are very close together?
    * e.g. make `f_in2=6`
* What if one of the components has a relatively small amplitude?
    * e.g. change `amplitude` of the second input to 0.4

### Notes

```python

```

### Shaping the lobes
The leakage sinc function has a peak around a specific frequency. If we want our DFT to be better able to distinguish between close frequencies, we need that peak, the **main lobe** to be narrower.  We also want the other peaks, the **side lobes** to be flatter.  We can achieve this using different **windowing methods** on our input.  This is why you see 'Hanning' as the default option for window method in the advanced spectrogram settings in praat.

We'll see some more examples of this when we look at different types of filters later.  But for now the main thing to observe is that leakage might give you the impression that specific frequency components are present in your waveform, when what's actually happening is that your waveform has frequencies that don't match the DFT phasors.  Another thing that can happen is that the peak for a low amplitude component gets subsumed into the main lobe of a nearby frequency component.  This might make you miss frequency components in your input!

```python

```

## 5.6 Extra:

Here's the composition and decomposition for the compound waveform one you saw in Notebook 2!

```python
N=1028
f_s = 1028
f_in1 = 8 ## In between DFT output frequencies
f_in2 = 20  ## One of the DFT outputs
f_in3 = 36  ## One of the DFT outputs

x1, time_steps = gen_sinusoid(frequency=f_in1, phase=0, amplitude=0.5, sample_rate=f_s, seq_length=N, gen_function=np.cos)
x2, time_steps = gen_sinusoid(frequency=f_in2, phase=np.pi/2, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)
x3, time_steps = gen_sinusoid(frequency=f_in3, phase=0, amplitude=0.3, sample_rate=f_s, seq_length=N, gen_function=np.cos)

x_compound = x1  + x2 + x3

## Plot the input
fig, timedom = plt.subplots(figsize=(16, 4))
#timedom.scatter(time_steps, x_compound, color='magenta')
#timedom.set(xlim=(0, 0.2))
timedom.plot(time_steps, x_compound, color='magenta')
```

```python
## Plot the input

fig = plt.figure(figsize=(15,15))
gs = fig.add_gridspec(3,2)

ymax=2
timedom = fig.add_subplot(gs[0, 0])
timedom.set(xlim=(-0.1, 1), ylim=(-ymax,ymax))
timedom.plot(time_steps, x_compound, color='magenta')
timedom.set_title("waveform made from adding 3 sine waves")

s1 = fig.add_subplot(gs[0, 1])
s1.set(xlim=(-0.1, 1), ylim=(-ymax,ymax))
s1.plot(time_steps, x1)
s1.set_title("component 1: 8 Hz")

s2 = fig.add_subplot(gs[1, 1])
s2.set(xlim=(-0.1, 1), ylim=(-ymax,ymax))
s2.plot(time_steps, x2)
s2.set_title("component 2: 20 Hz")

s3 = fig.add_subplot(gs[2, 1])
s3.set(xlim=(-0.1, 1), ylim=(-ymax,ymax))
s3.plot(time_steps, x3)
s3.set_title("component 3: 36 Hz")

fig.savefig("./fig/compound_waveform.png")
```

```python
## Do the DFT on the compound waveform as above:
mags, phases = get_dft_mag_phase(x_compound, seq_len=N)
dft_freqs = get_dft_freqs_all(sample_rate=f_s, seq_len=N)

## Plot the magnitudes
fig,fdom = plt.subplots(figsize=(16, 4))
fdom.set(xlim=(-1, N/2))
## Just plot the first N/2 frequencies since we know that they are the mirrored for k>N/2
fdom.scatter(dft_freqs[:round(N/2)], mags[:round(N/2)])
fdom.set_xlabel("Frequency (Hz)")
fdom.set_ylabel("Magnitude")
fdom.set_title("Magnitude Response")
#print(mags)

```

```python

```

```python

```