# Unit 9: Discrete Fourier Transform (DFT)

Overview and Learning Objectives

Inner Product

Definition of DFT

DFT Matrix

Fast Fourier Transform (FFT)

Exercise 1: Interpretation of Frequency Indices

Exercise 2: Missing Time Localization

Exercise 3: Chirp Signal

Exercise 4: Inverse DFT

Overview and Learning Objectives

The Fourier transform is one of the most important tools for a wide range of engineering and computer science applications. The general idea of Fourier analysis is to decompose a given signal into a weighted superposition of sinusoidal functions. Since these functions possess an explicit physical meaning regarding their frequencies, the decomposition is typically more accessible for subsequent processing steps than the original signal. Assuming that you are familiar with the Fourier transform and its applications in signal processing, we review in this unit the discrete variant of the Fourier transform known as Discrete Fourier Transform (DFT). We define the inner product that allows for comparing two vectors (e.g., discrete-time signals of finite length). The DFT can be thought of as comparing a given signal of finite length with a specific set of exponential signals (a complex variant of sinusoidal signals), each comparison yielding a complex-valued Fourier coefficient. Then, using suitable visualizations, we show how you can interpret the amplitudes and phases of these coefficients. Recall that one can express the DFT as a complex-valued square matrix. We show how separately plotting the real and imaginary parts leads to beautiful and insightful images. Applying a DFT boils down to computing a matrix–vector product, which we implement via the standard NumPy function np.dot. Since the number of operations for computing a DFT via a simple matrix–vector product is quadratic in the input length, the runtime of this approach becomes problematic with increasing length. This issue is exactly where the fast Fourier transform (FFT) comes into the game. We present this famous divide-and-conquer algorithm and provide a Python implementation. Furthermore, we compare the runtime behavior between the FFT implementation and the naive DFT implementation. We will further deepen your understanding of the Fourier transform by considering further examples and visualization in the exercises. In Exercise 1, you will learn how to interpret and plot frequency indices in a physically meaningful way. In Exercise 2, we discuss the issue of loosing time information when applying the Fourier transform, which is the main motivation for the short-time Fourier transform. In Exercise 3, you will apply the DFT to a chirp signal, which yields another illustrative example of the DFT's properties. Finally, in Exercise 4, we will invite you to explore the relationship between the DFT and its inverse. Again, an overarching goal of this unit is to apply and deepen your Python programming skills within the context of a central topic for signal processing.

## Inner Product

In this notebook, we consider [discrete-time (DT) signals](PCP_Signal.html) of finite length $N\in\mathbb{N}$, which we represent as vector

$$
x=(x(0),x(1),...,x(N-1))^\top\in\mathbb{R}^N
$$

with samples $x(n)\in\mathbb{R}^N$ for $n\in[0:N-1]$. Note that $\top$ indicates the transpose of a vector, thus converting a row vector into a column vector. Furthermore, note that we start indexing with the index $0$ (thus adapting our mathematical notation to Python conventions). A general concept for comparing two vectors (or signals) is the **inner product**. Given two vectors $x, y \in \mathbb{R}^N$, the inner product between $x$ and $y$ is defined as follows:

$$
\langle x | y \rangle := \sum_{n=0}^{N-1} x(n) y(n).
$$

The absolute value of the inner product may be interpreted as a measure of similarity between $x$ and $y$. If $x$ and $y$  are similar (i.e., if they point to more or less the same direction), the inner product $|\langle x | y \rangle|$ is large. If $x$ and $y$ are dissimilar (i.e., if $x$ and $y$ are more or less orthogonal to each other), the inner product $|\langle x | y \rangle|$ is close to zero.

One can extend this concept to **complex-valued** vectors $x,y\in\mathrm{C}^N$, where the inner product is defined as

$$
\langle x | y \rangle := \sum_{n=0}^{N-1} x(n) \overline{y(n)}.
$$

In the case of real-valued signals, the complex conjugate does not play any role and the definition of the complex-valued inner product reduces to the real-valued one. In the following code cell, we give some examples.

Note:
One can use the NumPy function np.vdot to compute the inner product. However, opposed to the mathematical convention to conjugate the second argument, this function applies complex conjugation on the first argument. Therefore, for computing $\langle x | y \rangle$ as defined above, one has to call np.vdot(y, x).

In the following, we generate and visualize three signals $x_1$, $x_2$, $x_3$. Then, we compute and discuss different inner products using the signals.

```python
import numpy as np
from matplotlib import pyplot as plt
import libpcp.signal
%matplotlib inline

Fs = 64
dur = 1
x1, t = libpcp.signal.generate_example_signal(Fs=Fs, dur=dur)
x2, t = libpcp.signal.generate_sinusoid(dur=dur, Fs=Fs, amp=1, freq=2, phase=0.3)
x3, t = libpcp.signal.generate_sinusoid(dur=dur, Fs=Fs, amp=1, freq=6, phase=0.1)

def plot_inner_product(ax, t, x, y, color_x='k', color_y='r', label_x='x', label_y='y'):
    """Plot inner product

    Notebook: PCP_dft.ipynb

    Args:
        ax: Axis handle
        t: Time axis
        x: Signal x
        y: Signal y
        color_x: Color of signal x (Default value = 'k')
        color_y: Color of signal y (Default value = 'r')
        label_x: Label of signal x (Default value = 'x')
        label_y: Label of signal y (Default value = 'y')
    """
    ax.plot(t, x, color=color_x, linewidth=1.0, linestyle='-', label=label_x)
    ax.plot(t, y, color=color_y, linewidth=1.0, linestyle='-', label=label_y)
    ax.set_xlim([0, t[-1]])
    ax.set_ylim([-1.5, 1.5])
    ax.set_xlabel('Time (seconds)')
    ax.set_ylabel('Amplitude')
    sim = np.vdot(y, x)
    ax.set_title(r'$\langle$ %s $|$ %s $\rangle = %.1f$' % (label_x, label_y, sim))
    ax.legend(loc='upper right')

plt.figure(figsize=(8, 5))
ax = plt.subplot(2, 2, 1)
plot_inner_product(ax, t, x1, x1, color_x='k', color_y='k', label_x='$x_1$', label_y='$x_1$')
ax = plt.subplot(2, 2, 2)
plot_inner_product(ax, t, x1, x2, color_x='k', color_y='r', label_x='$x_1$', label_y='$x_2$')
ax = plt.subplot(2, 2, 3)
plot_inner_product(ax, t, x1, x3, color_x='k', color_y='b', label_x='$x_1$', label_y='$x_3$')
ax = plt.subplot(2, 2, 4)
plot_inner_product(ax, t, x2, x3, color_x='r', color_y='b', label_x='$x_2$', label_y='$x_3$')
plt.tight_layout()
```

In the above example, one can make the following observations:

* The signal $x_1$ is similar to itself, leading to a large value of $\langle x_1 | x_1 \rangle=40.0$.
* The overall course of the signal $x_1$ strongly correlates with the sinusoid $x_2$, which is reflected by a relatively large value of $\langle x_1 | x_2 \rangle=29.9$.
* There are some finer oscillations of $x_1$ that are captured by $x_3$, leading to a still noticeable value of  $\langle x_1 | x_3 \rangle=14.7$.
* The two sinusoids $x_2$ and $x_3$ are more or less uncorrelated, which is revealed by the value of $\langle x_2 | x_3 \rangle\approx 0$.

In other words, the above comparison reveals that the signal $x_1$ has a strong signal component of $2~\mathrm {Hz}$ (frequency of $x_2$) and $6~\mathrm {Hz}$ (frequency of $x_3$). Measuring correlations between an arbitrary signal and sinusoids of different frequencies is exactly the idea of performing a Fourier (or spectral) analysis.

## Definition of DFT

Let $x\in \mathbb{C}^N$ be a vector of length $N\in\mathbb{N}$. The **discrete Fourier transform** (DFT) of $x$ is defined by:

$$ X(k) := \sum_{n=0}^{N-1} x(n) \exp(-2 \pi i k n / N) $$

for $k \in [0:N-1]$. The vector $X\in\mathbb{C}^N$ can be interpreted as a frequency representation of the time-domain signal $x$.   To obtain a geometric interpretation of the DFT, we define the vector $\mathbf{e}_k \in\mathbb{C}^N$ with real part $\mathbf{c}_k=\mathrm{Re}(\mathbf{e}_k)$ and imaginary part $\mathbf{s}_k=\mathrm{Im}(\mathbf{e}_k)$ by

$$\mathbf{e}_k(n) :=  \exp(2 \pi i k n / N) = \cos(2 \pi i k n / N) + i \sin(2 \pi i k n / N)
= \mathbf{c}_k(n) + i \mathbf{s}_k(n)$$

for each $k \in [0:N-1]$.

This vector can be regarded as a [sampled version](PCP_signal.html) of the [exponential function](PCP_exp.html) of frequency $k/N$. Using inner products, the DFT can be expressed as

$$ X(k) = \sum_{n=0}^{N-1} x(n) \overline{\mathbf{e}_k}(n) = \langle x | \mathbf{e}_k \rangle,$$

thus measuring the similarity between the signal $x$ and the sampled exponential functions $\mathbf{e}_k$. The absolute value $|X(k)|$ indicates the degree of similarity between the signal $x$ and $\mathbf{e}_k$. In the case that $x\in \mathbb{R}^N$ is a real-valued vector (which is typically the case for audio signals), we obtain:

$$
X(k) = \langle x |\mathrm{Re}(\mathbf{e}_k) \rangle - i\langle x | \mathrm{Im}(\mathbf{e}_k) \rangle
= \langle x |\mathbf{c}_k \rangle - i\langle x | \mathbf{s}_k \rangle
$$

The following plot shows an example signal $x$ compared with functions $\overline{\mathbf{e}_k}$ for various frequency parameters $k$. The real and imaginary part of $\overline{\mathbf{e}_k}$ are shown in  red and  blue, respectively.

```python
def plot_signal_e_k(ax, x, k, show_e=True, show_opt=False):
    """Plot signal and k-th DFT sinusoid

    Notebook: PCP_dft.ipynb

    Args:
        ax: Axis handle
        x: Signal
        k: Index of DFT
        show_e: Shows cosine and sine (Default value = True)
        show_opt: Shows cosine with optimal phase (Default value = False)
    """
    N = len(x)
    time_index = np.arange(N)
    ax.plot(time_index, x, 'k', marker='.', markersize='10', linewidth=2.0, label='$x$')
    plt.xlabel('Time (samples)')
    e_k = np.exp(2 * np.pi * 1j * k * time_index / N)
    c_k = np.real(e_k)
    s_k = np.imag(e_k)
    X_k = np.vdot(e_k, x)

    plt.title(r'k = %d: Re($X(k)$) = %0.2f, Im($X(k)$) = %0.2f, $|X(k)|$=%0.2f' %
              (k, X_k.real, X_k.imag, np.abs(X_k)))
    if show_e is True:
        ax.plot(time_index, c_k, 'r', marker='.', markersize='5',
                 linewidth=1.0, linestyle=':', label='$\mathrm{Re}(\overline{\mathbf{u}}_k)$')
        ax.plot(time_index, s_k, 'b', marker='.', markersize='5',
                 linewidth=1.0, linestyle=':', label='$\mathrm{Im}(\overline{\mathbf{u}}_k)$')
    if show_opt is True:
        phase_k = - np.angle(X_k) / (2 * np.pi)
        cos_k_opt = np.cos(2 * np.pi * (k * time_index / N - phase_k))
        d_k = np.sum(x * cos_k_opt)
        ax.plot(time_index, cos_k_opt, 'g', marker='.', markersize='5',
                 linewidth=1.0, linestyle=':', label='$\cos_{k, opt}$')
    plt.grid()
    plt.legend(loc='lower right')

N = 64
x, t = libpcp.signal.generate_example_signal(Fs=N, dur=1)

plt.figure(figsize=(8, 15))
for k in range(1, 8):
    ax = plt.subplot(7, 1, k)
    plot_signal_e_k(ax, x, k=k)

plt.tight_layout()
```

## DFT Phase

At first sight, the DFT may be a bit confusing: Why is a real-valued signal $x$ compared with a complex-valued sinusoid $\mathbf{e}_k$? What does the resulting complex-valued Fourier coefficient

$$
c_k:= X(k) := \langle x |\mathrm{Re}(\mathbf{e}_k) \rangle - i\langle x | \mathrm{Im}(\mathbf{e}_k) \rangle.
$$

encode? To understand this, we represent the complex number $c_k$ in form of its [polar representation](PCP_complex.html#polar)

$$
c_k = |c_k| \cdot \mathrm{exp}(i \gamma_k),
$$

where $\gamma_k$ is the [angle](PCP_complex.html) (given in radians).  Furthermore, let $\mathbf{cos}_{k,\varphi}:[0:N-1]\to\mathbb{R}$ be a sampled sinusoid with frequency parameter $k$ and phase $\varphi\in[0,1)$, defined by

$$
   \mathbf{cos}_{k,\varphi}(n) = \mathrm{cos}\big( 2\pi (kn/N - \varphi) \big)
$$

for $n\in[0,N-1]$. Defining $\varphi_k := - \frac{\gamma_k}{2 \pi}$, one obtains the following remarkable property of the Fourier coefficient $c_k$:

\begin{eqnarray}
|c_k| &=& \mathrm{max}_{\varphi\in[0,1)} \langle x | \mathbf{cos}_{k,\varphi} \rangle,\\
\varphi_k  &=& \mathrm{argmax}_{\varphi\in[0,1)} \langle x | \mathbf{cos}_{k,\varphi} \rangle.
\end{eqnarray}

In other words, the phase $\varphi_k$ maximizes the correlation between $x$ and all possible sinusoids $\mathbf{cos}_{k,\varphi}$ with $\varphi\in[0,1)$. Furthermore, the magnitude $|c_k|$ yields this maximal value. Thus, computing a single correlation between $x$ and the complex-valued function $\mathbf{e}_k$ (which real part coincides with $\mathbf{cos}_{k,0}$, and its imaginary part with $\mathbf{cos}_{k,0.25}$) solves an optimization problem. In the following code cell, we demonstrate this optimality property, where the $\mathbf{cos}_{k,\varphi}$ with optimal phase $\varphi=\varphi_k$ is shown in green.

```python
plt.figure(figsize=(8, 15))
for k in range(1, 8):
    ax = plt.subplot(7, 1, k)
    plot_signal_e_k(ax, x, k=k, show_e=False, show_opt=True)

plt.tight_layout()
```

## DFT Matrix

Being a linear operator $\mathbb{C}^N \to \mathbb{C}^N$, the DFT can be expressed by some $N\times N$-matrix. This leads to the famous DFT matrix $\mathrm{DFT}_N \in \mathbb{C}^{N\times N}$ matrix, which is given by

$$\mathrm{DFT}_N(n, k) = \mathrm{exp}(-2 \pi i k n / N)$$

for $n\in[0:N-1]$ and $k\in[0:N-1]$. Let $\rho_N:=\exp(2 \pi i / N)$ be the primitive $N^\mathrm{th}$ [root of unity](PCP_exp.html#roots). Then

$$\sigma_N:= \overline{\rho_N} = \mathrm{exp}(-2 \pi i / N)$$

also defines a primitive $N^\mathrm{th}$ [root of unity](PCP_exp.html#roots). From the [properties of exponential functions](PCP_exp.html), one obtains that

$$ \sigma_N^{kn} = \mathrm{exp}(-2 \pi i / N)^{kn} = \mathrm{exp}(-2 \pi i k n / N)$$

From this, one obtains:

$$
\mathrm{DFT}_N =
\begin{pmatrix}
    1 & 1 & 1 & \dots  & 1 \\
    1 & \sigma_N & \sigma_N^2 & \dots  & \sigma_N^{N-1} \\
    1 & \sigma_N^2 & \sigma_N^4 & \dots  & \sigma_N^{2(N-1)} \\
    \vdots & \vdots & \vdots & \ddots & \vdots \\
    1 & \sigma_N^{N-1} & \sigma_N^{2(N-1)} & \dots  & \sigma_N^{(N-1)(N-1)} \\
\end{pmatrix}
$$

In the following visualization, the real and imaginary part of $\mathrm{DFT}_N$ are shown, where the values are encoded by suitable colors. Note that the $k^\mathrm{th}$ row of $\mathrm{DFT}_N$ corresponds to the vector $\mathbf{e}_k$ as defined above.

```python
def generate_matrix_dft(N, K):
    """Generate a DFT (discete Fourier transfrom) matrix

    Notebook: PCP_dft.ipynb

    Args:
        N: Number of samples
        K: Number of frequency bins

    Returns:
        dft: The DFT matrix
    """
    dft = np.zeros((K, N), dtype=np.complex128)
    time_index = np.arange(N)
    for k in range(K):
        dft[k, :] = np.exp(-2j * np.pi * k * time_index / N)
    return dft

N = 32
dft_matrix = generate_matrix_dft(N, N)

plt.figure(figsize=(10, 4))

plt.subplot(1, 2, 1)
plt.title('$\mathrm{Re}(\mathrm{DFT}_N)$')
plt.imshow(np.real(dft_matrix), origin='lower', cmap='seismic', aspect='equal')
plt.xlabel('Time (sample, index $n$)')
plt.ylabel('Frequency (index $k$)')
plt.colorbar()

plt.subplot(1, 2, 2)
plt.title('$\mathrm{Im}(\mathrm{DFT}_N)$')
plt.imshow(np.imag(dft_matrix), origin='lower', cmap='seismic', aspect='equal')
plt.xlabel('Time (samples, index $n$)')
plt.ylabel('Frequency (index $k$)')
plt.colorbar()
plt.tight_layout()
```

We now write a function that computes the discrete Fourier transform $X = \mathrm{DFT}_N \cdot x$ of a signal $x\in\mathbb{C}^N$. We apply the function from above sampled at $N=64$ time points. The peaks of the magnitude Fourier transform $|X|$ correspond to the main frequency components the signal is composed of. Note that the magnitude Fourier transform is symmetrical around the center. Why? For the interpretation of the time and frequency axis, see also Exercise 1: Interpretation of Frequency Indices

```python
def dft(x):
    """Compute the discete Fourier transfrom (DFT)

    Notebook: PCP_dft.ipynb

    Args:
        x: Signal to be transformed

    Returns:
        X: Fourier transform of x
    """
    x = x.astype(np.complex128)
    N = len(x)
    dft_mat = generate_matrix_dft(N, N)
    return np.dot(dft_mat, x)

N = 64
x, t = libpcp.signal.generate_example_signal(Fs=N, dur=1)
X = dft(x)

def plot_signal_dft(t, x, X, ax_sec=False, ax_Hz=False, freq_half=False, figsize=(10, 2)):
    """Plotting function for signals and its magnitude DFT

    Notebook: PCP_dft.ipynb

    Args:
        t: Time axis (given in seconds)
        x: Signal
        X: DFT
        ax_sec: Plots time axis in seconds (Default value = False)
        ax_Hz: Plots frequency axis in Hertz (Default value = False)
        freq_half: Plots only low half of frequency coefficients (Default value = False)
        figsize: Size of figure (Default value = (10, 2))
    """
    N = len(x)
    if freq_half is True:
        K = N // 2
        X = X[:K]
    else:
        K = N

    plt.figure(figsize=figsize)
    ax = plt.subplot(1, 2, 1)
    ax.set_title('$x$ with $N=%d$' % N)
    if ax_sec is True:
        ax.plot(t, x, 'k', marker='.', markersize='3', linewidth=0.5)
        ax.set_xlabel('Time (seconds)')
    else:
        ax.plot(x, 'k', marker='.', markersize='3', linewidth=0.5)
        ax.set_xlabel('Time (samples)')
    ax.grid()

    ax = plt.subplot(1, 2, 2)
    ax.set_title('$|X|$')
    if ax_Hz is True:
        Fs = 1 / (t[1] - t[0])
        ax_freq = Fs * np.arange(K) / N
        ax.plot(ax_freq, np.abs(X), 'k', marker='.', markersize='3', linewidth=0.5)
        ax.set_xlabel('Frequency (Hz)')

    else:
        ax.plot(np.abs(X), 'k', marker='.', markersize='3', linewidth=0.5)
        ax.set_xlabel('Frequency (index)')
    ax.grid()
    plt.tight_layout()
    plt.show()

plot_signal_dft(t, x, X)
plot_signal_dft(t, x, X, ax_sec=True, ax_Hz=True)
plot_signal_dft(t, x, X, ax_sec=True, ax_Hz=True, freq_half=True)
```

## Fast Fourier Transform (FFT)

Next, we discuss the famous fast Fourier transform (FFT), which is a fast algorithm to compute the DFT. The FFT algorithm was originally found by Gauss in about 1805 and  then rediscovered by Cooley and Tukey in 1965. The FFT algorithm is based on the observation that applying a DFT of even size $N=2M$ can be expressed in terms of  applying two DFTs of half the size $M$. It exploits the fact that there are algebraic relations between the entries $\sigma_N^{kn} = \mathrm{exp}(-2 \pi i / N)^{kn}$ of DFT matrices. In particular, one has

$$\sigma_M = \sigma_N^2$$

In the FFT algorithm, one computes the DFT of the even-indexed and the uneven-indexed entries of $x$:

\begin{align}
(A(0), \dots, A(N/2-1)) &= \mathrm{DFT}_{N/2} \cdot (x(0), x(2), x(4), \dots, x(N-2))\\
(B(0), \dots, B(N/2-1)) &= \mathrm{DFT}_{N/2} \cdot (x(1), x(3), x(5), \dots, x(N-1))
\end{align}

With these two DFTs of size $N/2$, one can compute the full DFT of size $N$ via:

\begin{eqnarray}
C(k) &=& \sigma_N^k \cdot B(k)\\
X(k) &=& A(k) + C(k)\\
X(N/2 + k) &=& A(k) - C(k)\\
\end{eqnarray}

for $k \in [0: N/2 - 1]$. The numbers $\sigma_N^k$ are also called *twiddle factors*. If $N$ is a power of two, this idea can be applied recursively until one reaches the computation of $\mathrm{DFT}_{1}$ (the case $N=1$), which is simply multiplication by one (i.e. just returning the signal of length $N=1$). For further details, we refer to Section 2.4.3 of [Müller, FMP, Springer 2015]) (see also Table 2.1).

In the following code, we provide a function `fft` that implements the FFT algorithm. We test the function  `fft` by comparing its output with the one when applying the `dft` on a test signal `x`. For the comparison of result matrices, we use the NumPy functions [`np.array_equal`](https://numpy.org/doc/stable/reference/generated/numpy.array_equal.html) and [`np.allclose`](https://numpy.org/doc/stable/reference/generated/numpy.allclose.html#numpy.allclose).

```python
def fft(x):
    """Compute the fast Fourier transform (FFT)

    Notebook: PCP_dft.ipynb

    Args:
        x: Signal to be transformed

    Returns:
        X: Fourier transform of x
    """
    x = x.astype(np.complex128)
    N = len(x)
    log2N = np.log2(N)
    assert log2N == int(log2N), 'N must be a power of two!'
    X = np.zeros(N, dtype=np.complex128)

    if N == 1:
        return x
    else:
        this_range = np.arange(N)
        A = fft(x[this_range % 2 == 0])
        B = fft(x[this_range % 2 == 1])
        range_twiddle_k = np.arange(N // 2)
        sigma = np.exp(-2j * np.pi * range_twiddle_k / N)
        C = sigma * B
        X[:N//2] = A + C
        X[N//2:] = A - C
        return X

N = 64
x, t = libpcp.signal.generate_example_signal(Fs=N, dur=1)

X_via_dft = dft(x)
X_via_fft = fft(x)
X_via_fft_numpy = np.fft.fft(x)

is_equal = np.array_equal(X_via_dft, X_via_fft)
is_equal_tol = np.allclose(X_via_dft, X_via_fft)
is_equal_tol_np = np.allclose(X_via_dft, X_via_fft_numpy)

print('Equality test for dft(x) and fft(x) using np.array_equal:    ', is_equal)
print('Equality test for dft(x) and fft(x) using np.allclose:       ', is_equal_tol)
print('Equality test for dft(x) and np.fft.fft(x) using np.allclose:', is_equal_tol_np)
```

    Equality test for dft(x) and fft(x) using np.array_equal:     False
    Equality test for dft(x) and fft(x) using np.allclose:        True
    Equality test for dft(x) and np.fft.fft(x) using np.allclose: True

Note: The test shows that our dft and fft implementations do not yield the same result (due to numerical issues). However, the results are numerically very close, which is verified by the test using np.allclose.

The FFT reduces the overall number of operations from the order of $N^2$ (needed when computing the usual matrix–vector product $\mathrm{DFT}_N \cdot x$) to the order of $N\log_2N$. The savings are enormous. For example, using $N=2^{10}=1024$, the FFT requires roughly $N\log_2N=10240$ instead of $N^2=1048576$ operations in the naive approach. Using the module `timeit`, which provides a simple way to time small bits of Python code, the following code compares the running time when using the naive approach and the FFT. Furthermore, we compare the running time with the highly optimized NumPy implementation np.fft.fft.

```python
import timeit

rep = 3
for N in [256, 512, 1024, 2048, 4096]:
    time_index = np.arange(N)
    x = np.sin(2 * np.pi *  time_index / N )
    t_DFT = 1000 * timeit.timeit(lambda: dft(x), number=rep)/rep
    t_FFT = timeit.timeit(lambda: fft(x), number=rep*5)/(rep*5)
    t_FFT_np = timeit.timeit(lambda: np.fft.fft(x), number=rep*100)/(rep*100)
    print('Runtime (ms) for N = %4d : DFT %10.2f,  FFT %.5f,  FFT_np %.8f'%(N, t_DFT, t_FFT, t_FFT_np))
```

    Runtime (ms) for N =  256 : DFT      19.22,  FFT 0.01186,  FFT_np 0.00001728
    Runtime (ms) for N =  512 : DFT      31.76,  FFT 0.02365,  FFT_np 0.00001604
    Runtime (ms) for N = 1024 : DFT     104.20,  FFT 0.03962,  FFT_np 0.00002435
    Runtime (ms) for N = 2048 : DFT     376.25,  FFT 0.08224,  FFT_np 0.00004206
    Runtime (ms) for N = 4096 : DFT    1418.35,  FFT 0.15875,  FFT_np 0.00007802

## Exercises and Results

```python
import libpcp.dft
show_result = True
```

Exercise 1: Interpretation of Frequency Indices

Given a dimension $N\in\mathbb{N}$, the $\mathrm{DFT}_N$ transform a vector $x\in\mathbb{C}^N$ into another vector $X\in\mathbb{C}^N$. Assuming that $x$ represents a time-domain signal sampled with a sampling rate $F_\mathrm{s}$, one can associate the index $n\in[0:N-1]$ of the sample $x(n)$ with the physical time point $t = n/F_\mathrm{s}$ given in seconds. In case of the vector $X$, the index $k\in[0:N-1]$ of the coefficient $X(k)$ can be associated to a physical frequency value

$$
  \omega=\frac{k \cdot F_\mathrm{s}}{N}.
$$

Furthermore, using a real-valued signal $x\in\mathbb{R}^N$, the upper part of $X\in\mathbb{C}^N$ becomes redundant, and it suffices to consider the first $K$ coefficients with $K=N/2$.

Find explanations why these properties apply.

Find out how the function plot_signal_dft uses these properties to convert and visualize the time and frequency axes.

Using the signal x, t = libpcp.signal.generate_example_signal(Fs=64, dur=2), plot the signal and its magnitude Fourier transform once using axes given in indices and once using axes given in physical units (seconds, Hertz). Discuss the results.

Do the same for the signal x, t = libpcp.signal.generate_example_signal(Fs=32, dur=2). What is going wrong and why?

```python
#<solution>
# Your Solution
#</solution>
```

```python
libpcp.dft.exercise_freq_index(show_result=show_result)
```

Exercise 2: Missing Time Localization

The Fourier transform yields frequency information that is averaged over the entire time axis. However, the information on when these frequencies occur is hidden in the transform. To demonstrate this phenomenon, construct the following two different signals defined on a common time axis $[0, T]$ with $T$ given in seconds (e.g., $T=6~\mathrm{sec}$).

A superposition of two sinusoids $f_1+f_2$ defined over the entire time interval $[0, T]$, where the first sinusoid $f_1$ has a frequency $\omega_1=1~\mathrm{Hz}$ and an amplitude of $1$, while the second sinusoid $f_2$ has a frequency $\omega_2=5~\mathrm{Hz}$ and an amplitude of $0.5$.

A concatenation of two sinusoids, where $f_1$ (specified as before) is now defined only on the subinterval $[0, T/2]$, and $f_2$ is defined on the subinterval $[T/2, T]$.

Sample the interval $[0,T]$ to obtain $N$ samples (use np.linspace), with $N\in\mathbb{N}$ being power of two (e.g., $N=256$). Define DT-signals of the superposition and the concatenation and compute the DFT for each of the signals. Plot the signals as well as the resulting magnitude Fourier transforms and discuss the result.

```python
#<solution>
# Your Solution
#</solution>
```

```python
libpcp.dft.exercise_missing_time(show_result=show_result)
```

Exercise 3: Chirp Signal

The function $f(t)=\sin\left(\pi t^2\right)$ defines a chirp signal (also called sweep signal), in which the frequency increases with time. The instantaneous frequency $\omega_t$ of the chirp signal at time $t$ is the derivate of the sinusoid's argument divided by $2\pi$, thus $\omega_t = t$.

Let $[t_0,t_1]$ be a time interval (given in seconds) with $0\leq t_0<t_1$ and $N\in\mathbb{N}$ be power of two. Implement a function generate_chirp that outputs a sampled chirp signal x over the interval $[t_0,t_1]$ with $N$ samples (use np.linspace).

Compute the DFT of x for various input parameters $t_0$, $t_1$, and $N$. Plot the chirp signal as well as the resulting magnitude Fourier transform. Discuss the result.

```python
#<solution>
# Your Solution
#</solution>
```

```python
libpcp.dft.exercise_chirp(show_result=show_result)
```

Exercise 4: Inverse DFT

The discrete Fourier transform given by the matrix $\mathrm{DFT}_N \in \mathbb{C}^{N\times N}$ is an invertible operation, given by the inverse DFT matrix $\mathrm{DFT}_N^{-1}$.

There is an explicit relation between $\mathrm{DFT}_N$ and its inverse $\mathrm{DFT}_N^{-1}$. Which one?

Write a function generate_matrix_dft_inv that explicitly generates $\mathrm{DFT}_N^{-1}$.

Check your function by computing $\mathrm{DFT}_N \cdot \mathrm{DFT}_N^{-1}$ and $\mathrm{DFT}_N^{-1} \cdot \mathrm{DFT}_N$ (using np.matmul) and comparing these products with the identity matrix (using np.eye and np.allclose).

Furthermore, compute the inverse DFT by using np.linalg.inv. Compare the result with your function using np.allclose.

Similar to fft, implement a fast inverse Fourier transform fft_inv

```python
#<solution>
# Your Solution
#</solution>
```

```python
libpcp.dft.exercise_inverse(show_result=show_result)
```

    Comparison between DFT * DFT_inv and I: True
    Comparison between DFT_inv * DFT and I: True
    Comparison between DFT_inv and DFT_inv_via_np: True
    Example signal x: [ 0.  1.  2.  3.  4.  5.  6.  7.  8.  9. 10. 11. 12. 13. 14. 15.]
    Signal y = fft_inv(fft(x)):
    [4.4408921e-16-3.33066907e-16j 1.0000000e+00-2.16323416e-16j
     2.0000000e+00-9.95799250e-17j 3.0000000e+00+2.16323416e-16j
     4.0000000e+00-8.81375476e-17j 5.0000000e+00-2.16323416e-16j
     6.0000000e+00+5.43669135e-16j 7.0000000e+00+2.16323416e-16j
     8.0000000e+00+1.11022302e-16j 9.0000000e+00+2.27765794e-16j
     1.0000000e+01-9.95799250e-17j 1.1000000e+01+2.16323416e-16j
     1.2000000e+01+3.10182152e-16j 1.3000000e+01+2.27765794e-16j
     1.4000000e+01-3.44509285e-16j 1.5000000e+01-6.71855004e-16j]
    Comparison between x and y: True


# Frequentist Inference Case Study - Part B

## Learning objectives

Welcome to Part B of the Frequentist inference case study! The purpose of this case study is to help you apply the concepts associated with Frequentist inference in Python. In particular, you'll practice writing Python code to apply the following statistical concepts:
* the _z_-statistic
* the _t_-statistic
* the difference and relationship between the two
* the Central Limit Theorem, including its assumptions and consequences
* how to estimate the population mean and standard deviation from a sample
* the concept of a sampling distribution of a test statistic, particularly for the mean
* how to combine these concepts to calculate a confidence interval

In the previous notebook, we used only data from a known normal distribution. **You'll now tackle real data, rather than simulated data, and answer some relevant real-world business problems using the data.**

## Hospital medical charges

Imagine that a hospital has hired you as their data scientist. An administrator is working on the hospital's business operations plan and needs you to help them answer some business questions.

In this assignment notebook, you're going to use frequentist statistical inference on a data sample to answer the questions:
* has the hospital's revenue stream fallen below a key threshold?
* are patients with insurance really charged different amounts than those without?

Answering that last question with a frequentist approach makes some assumptions, and requires some knowledge, about the two groups.

We are going to use some data on medical charges obtained from [Kaggle](https://www.kaggle.com/easonlai/sample-insurance-claim-prediction-dataset).

For the purposes of this exercise, assume the observations are the result of random sampling from our single hospital. Recall that in the previous assignment, we introduced the Central Limit Theorem (CLT), and its consequence that the distributions of sample statistics approach a normal distribution as $n$ increases. The amazing thing about this is that it applies to the sampling distributions of statistics that have been calculated from even highly non-normal distributions of data! Recall, also, that hypothesis testing is very much based on making inferences about such sample statistics. You're going to rely heavily on the CLT to apply frequentist (parametric) tests to answer the questions in this notebook.

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import t
from numpy.random import seed
medical = pd.read_csv('insurance2.csv')

```

```python
medical.shape

```

    (1338, 8)

```python
medical.head()

```

      age
      sex
      bmi
      children
      smoker
      region
      charges
      insuranceclaim

      0
      19
      0
      27.900
      0
      1
      3
      16884.92400
      1

      1
      18
      1
      33.770
      1
      0
      2
      1725.55230
      1

      2
      28
      1
      33.000
      3
      0
      2
      4449.46200
      0

      3
      33
      1
      22.705
      0
      0
      1
      21984.47061
      0

      4
      32
      1
      28.880
      0
      0
      1
      3866.85520
      1

__Q1:__ Plot the histogram of charges and calculate the mean and standard deviation. Comment on the appropriateness of these statistics for the data.

__A:__

```python
# _ = plt.plot('hist','charges',data=medical)

medical.hist('charges',bins=40)

plt.show()
```

```python
# summary stats
med_mean = np.mean(medical.charges)
med_std = np.std(medical.charges)

med_mean, med_std

```

    (13270.422265141257, 12105.484975561605)

```python

```

The data has a long right tail and no left to, so this data set is as far from a gausian distribution as possible

```python
med_std=np.std(medical.charges,ddof=1)
med_std
```

    12110.011236693994

__Q2:__ The administrator is concerned that the actual average charge has fallen below 12,000, threatening the hospital's operational model. On the assumption that these data represent a random sample of charges, how would you justify that these data allow you to answer that question? And what would be the most appropriate frequentist test, of the ones discussed so far, to apply?

__A:__ The standard deviation is greater than the value by which we want to know if there is a change by so I do not think we can use this data to answer the question.

__Q3:__ Given the nature of the administrator's concern, what is the appropriate confidence interval in this case? A ***one-sided*** or ***two-sided*** interval? (Refresh your understanding of this concept on p. 399 of the *AoS*). Calculate the critical value and the relevant 95% confidence interval for the mean, and comment on whether the administrator should be concerned.

__A:__ as we are only concered with if the price change is below one value we can use a one-sided confidence interval.

```python
# import stats
import scipy
scipy.stats.norm.ppf(0.95,loc =med_mean,scale=med_std )

```

    33189.61817024046

```python
# critical value  =  t score for 95% conf and 1337 df
from scipy import stats

print (stats.t.ppf(.95, df= 1337))

```

    1.6459941145571317

```python
#margin of error  = t (a,df ) * s/sqrt(n)
                # for one sided

me =(stats.t.ppf(.95, df= 1337))* (med_std/np.sqrt(1338))

me

```

    544.9350813250251

```python
cofidence_interval = (med_mean-me,med_mean+me)
cofidence_interval

```

    (12725.48718381623, 13815.357346466282)

```python
help(scipy.stats.norm.ppf)

```

    Help on method ppf in module scipy.stats._distn_infrastructure:

    ppf(q, *args, **kwds) method of scipy.stats._continuous_distns.norm_gen instance
        Percent point function (inverse of `cdf`) at q of the given RV.

        Parameters
        ----------
        q : array_like
            lower tail probability
        arg1, arg2, arg3,... : array_like
            The shape parameter(s) for the distribution (see docstring of the
            instance object for more information)
        loc : array_like, optional
            location parameter (default=0)
        scale : array_like, optional
            scale parameter (default=1)

        Returns
        -------
        x : array_like
            quantile corresponding to the lower tail probability q.

```python
help(scipy.stats.norm.cdf)
```

    Help on method cdf in module scipy.stats._distn_infrastructure:

    cdf(x, *args, **kwds) method of scipy.stats._continuous_distns.norm_gen instance
        Cumulative distribution function of the given RV.

        Parameters
        ----------
        x : array_like
            quantiles
        arg1, arg2, arg3,... : array_like
            The shape parameter(s) for the distribution (see docstring of the
            instance object for more information)
        loc : array_like, optional
            location parameter (default=0)
        scale : array_like, optional
            scale parameter (default=1)

        Returns
        -------
        cdf : ndarray
            Cumulative distribution function evaluated at `x`

```python
help(scipy.stats.t.ppf)

```

    Help on method ppf in module scipy.stats._distn_infrastructure:

    ppf(q, *args, **kwds) method of scipy.stats._continuous_distns.t_gen instance
        Percent point function (inverse of `cdf`) at q of the given RV.

        Parameters
        ----------
        q : array_like
            lower tail probability
        arg1, arg2, arg3,... : array_like
            The shape parameter(s) for the distribution (see docstring of the
            instance object for more information)
        loc : array_like, optional
            location parameter (default=0)
        scale : array_like, optional
            scale parameter (default=1)

        Returns
        -------
        x : array_like
            quantile corresponding to the lower tail probability q.

```python
null is there is no difference between pay for the two groups

```

The administrator then wants to know whether people with insurance really are charged a different amount to those without.

__Q4:__ State the null and alternative hypothesis here. Use the _t_-test for the difference between means, where the pooled standard deviation of the two groups is given by:
\begin{equation}
s_p = \sqrt{\frac{(n_0 - 1)s^2_0 + (n_1 - 1)s^2_1}{n_0 + n_1 - 2}}
\end{equation}

and the *t*-test statistic is then given by:

\begin{equation}
t = \frac{\bar{x}_0 - \bar{x}_1}{s_p \sqrt{1/n_0 + 1/n_1}}.
\end{equation}

(If you need some reminding of the general definition of ***t-statistic***, check out the definition on p. 404 of *AoS*).

What assumption about the variances of the two groups are we making here?

__A:__

__Q5:__ Perform this hypothesis test both manually, using the above formulae, and then using the appropriate function from [scipy.stats](https://docs.scipy.org/doc/scipy/reference/stats.html#statistical-tests) (hint, you're looking for a function to perform a _t_-test on two independent samples). For the manual approach, calculate the value of the test statistic and then its probability (the p-value). Verify you get the same results from both.

__A:__

```python
no_in = medical.loc[medical['insuranceclaim'] == 0 ]
no_in.info()
insur = medical.loc[medical.insuranceclaim == 1]
insur.info()
```

    <class 'pandas.core.frame.DataFrame'>
    Int64Index: 555 entries, 2 to 1336
    Data columns (total 8 columns):
     #   Column          Non-Null Count  Dtype
    ---  ------          --------------  -----
     0   age             555 non-null    int64
     1   sex             555 non-null    int64
     2   bmi             555 non-null    float64
     3   children        555 non-null    int64
     4   smoker          555 non-null    int64
     5   region          555 non-null    int64
     6   charges         555 non-null    float64
     7   insuranceclaim  555 non-null    int64
    dtypes: float64(2), int64(6)
    memory usage: 39.0 KB
    <class 'pandas.core.frame.DataFrame'>
    Int64Index: 783 entries, 0 to 1337
    Data columns (total 8 columns):
     #   Column          Non-Null Count  Dtype
    ---  ------          --------------  -----
     0   age             783 non-null    int64
     1   sex             783 non-null    int64
     2   bmi             783 non-null    float64
     3   children        783 non-null    int64
     4   smoker          783 non-null    int64
     5   region          783 non-null    int64
     6   charges         783 non-null    float64
     7   insuranceclaim  783 non-null    int64
    dtypes: float64(2), int64(6)
    memory usage: 55.1 KB

```python
df_no_in = len(no_in)-1
df_insur = len(insur)-1

numerator = (df_no_in*np.var(no_in))+(df_insur*np.var(insur))

denominator = len(no_in)+len(insur)-2

sp = np.sqrt(numerator/denominator)
sp
```

    age                  13.953843
    sex                   0.499723
    bmi                   5.628061
    children              1.099324
    smoker                0.380504
    region                1.104229
    charges           11512.282899
    insuranceclaim        0.000000
    dtype: float64

```python
df_no_in = len(no_in)-1
df_insur = len(insur)-1

numerator = (df_no_in*np.var(no_in.charges))+(df_insur*np.var(insur.charges))

denominator = len(no_in)+len(insur)-2

sp = np.sqrt(numerator/denominator)
sp
```

    11512.282899205744

```python
## check variance of two samples

np.var(no_in.charges),np.var(insur.charges)
```

    (41482614.44965368, 197036140.77172482)

```python
numerator = np.mean(no_in.charges)+ np.mean(insur.charges)

denominator = sp*np.sqrt((1/len(no_in.charges))+(1/len(insur.charges)))
t_stat = numerator/denominator
t_stat
```

    39.52021163443274

```python
# my calc was never going to be negative compared to the stats.ttest_ind
numerator, denominator
```

    (25245.350168843957, 638.7959255473335)

```python
# cal the probability of t statistic caluclated

scipy.stats.t.cdf(39.52,df=denominator)
```

    1.0

```python
stats.ttest_ind(a=no_in.charges,
                   b=insur.charges,
               equal_var=False)
```

    Ttest_indResult(statistic=-13.298031957975649, pvalue=1.1105103216309125e-37)

```python
stats.ttest_ind(a=insur.charges,
                   b=no_in.charges,
               equal_var=False)
```

    Ttest_indResult(statistic=13.298031957975649, pvalue=1.1105103216309125e-37)

```python
## probablity of t stat from .ttest_ind

scipy.stats.t.cdf(13.298031957975649,df= numerator)
```

    1.0

```python
#?? this means there is a 100% chance that the observed difference is in the distribution ??

```

Congratulations! Hopefully you got the exact same numerical results. This shows that you correctly calculated the numbers by hand. Secondly, you used the correct function and saw that it's much easier to use. All you need to do is pass your data to it.

__Q6:__ Conceptual question: look through the documentation for statistical test functions in scipy.stats. You'll see the above _t_-test for a sample, but can you see an equivalent one for performing a *z*-test from a sample? Comment on your answer.

__A:__

## Learning outcomes

Having completed this project notebook, you now have good hands-on experience:
* using the central limit theorem to help you apply frequentist techniques to answer questions that pertain to very non-normally distributed data from the real world
* performing inference using such data to answer business questions
* forming a hypothesis and framing the null and alternative hypotheses
* testing this using a _t_-test

Frequentist Case Study/Frequentist Inference Case Study - Part B (2).ipynb


# Design of Digital Filters

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Design of Non-Recursive Filters using the Window Method

The design of non-recursive filters with a finite impulse response (FIR) is a frequent task in practical applications. The designed filter should approximate a desired frequency response as close as possible. First, the design of causal filters is considered. For many applications the resulting filter should have a linear phase characteristic since this results in a constant (and thus frequency independent) group delay. We therefore specialize the design to causal linear-phase filters.

### Causal Filters

Let's assume that the desired frequency characteristics of the filter are given by its frequency response (i.e. discrete-time Fourier transform) $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. The corresponding impulse response is computed by its inverse discrete-time Fourier transform (IDTFT)

\begin{equation}
h_\text{d}[k] = \frac{1}{2 \pi} \int\limits_{- \pi}^{\pi} H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \, \mathrm{e}^{\,\mathrm{j}\,\Omega\,k} \; \mathrm{d}\Omega
\end{equation}

In the general case, $h_\text{d}[k]$ will not be a causal FIR. The [Paley-Wiener theorem](https://en.wikipedia.org/wiki/Paley%E2%80%93Wiener_theorem) states, that a causal system $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ can only have zeros at single frequencies. This is not the case for idealized filters, like e.g. the [ideal low-pass filter](https://en.wikipedia.org/wiki/Low-pass_filter#Ideal_and_real_filters). The basic idea of the window method is to truncate the impulse response $h_\text{d}[k]$ in order to derive a causal FIR filter. This can be achieved by applying a window $w[k]$ of finite length $N$ to $h_\text{d}[k]$

\begin{equation}
h[k] = h_\text{d}[k] \cdot w[k]
\end{equation}

where $h[k]$ denotes the impulse response of the designed filter and $w[k] = 0$ for $k < 0$. Its frequency response $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is given by the multiplication theorem of the discrete-time Fourier transform (DTFT)

\begin{equation}
H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{1}{2 \pi} \; H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \circledast W(\mathrm{e}^{\,\mathrm{j}\,\Omega})
\end{equation}

where $W(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ denotes the DTFT of the window function $w[k]$. The frequency response $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of the filter is given as the periodic convolution of the desired frequency response $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and the frequency response of the window function $W(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. The frequency response $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is equal to the desired frequency response $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ only if $W(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = 2 \pi \cdot \delta(\Omega)$. This would require that $w[k] = 1$ for $k = -\infty, \dots, \infty$. Hence for a window $w[k]$ of finite length deviations from the desired frequency response are to be expected.

In order to investigate the effect of truncation on the frequency response $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$, a particular window is considered. A straightforward choice is the rectangular window $w[k] = \text{rect}_N[k]$ of length $N$. Its DTFT is given as

\begin{equation}
W(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \mathrm{e}^{-\mathrm{j} \, \Omega \,\frac{N-1}{2}} \cdot \frac{\sin(\frac{N \,\Omega}{2})}{\sin(\frac{\Omega}{2})}
\end{equation}

The frequency-domain properties of the rectangular window have already been discussed for the [leakage effect](../spectral_analysis_deterministic_signals/leakage_effect.ipynb). The rectangular window features a narrow main lobe at the cost of relative high sidelobe level. The main lobe gets narrower with increasing length $N$. The convolution of the desired frequency response with the frequency response of the window function effectively results in smoothing and ringing. While the main lobe will smooth discontinuities of the desired transfer function, the sidelobes result in undesirable ringing effects. The latter can be alleviated by using other window functions. Note that typical [window functions](../spectral_analysis_deterministic_signals/window_functions.ipynb) decay towards their ends and are symmetric with respect to their center. This may cause problems for desired impulse responses with large values towards their ends.

#### Example - Causal approximation of ideal low-pass

The design of an ideal low-pass filter using the window technique is illustrated in the following example. For $|\Omega| < \pi$ the transfer function of the ideal low-pass is given as

\begin{equation}
H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \begin{cases}
1 & \text{for } |\Omega| \leq \Omega_\text{c} \\
0 & \text{otherwise}
\end{cases}
\end{equation}

where $\Omega_\text{c}$ denotes the corner frequency of the low-pass. An inverse DTFT of the desired transfer function yields

\begin{equation}
h_\text{d}[k] = \frac{\Omega_\text{c}}{\pi} \cdot \text{si}[\Omega_\text{c} \, k]
\end{equation}

The impulse response $h_\text{d}[k]$ is not causal nor FIR. In order to derive a causal and FIR approximation a rectangular window $w[k]$ of length $N$ is applied

\begin{equation}
h[k] = h_\text{d}[k] \cdot \text{rect}_N[k]
\end{equation}

The resulting magnitude and phase response is computed numerically in the following.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

N = 32  # length of filter
Omc = np.pi/2

# compute impulse response
k = np.arange(N)
hd = Omc/np.pi * np.sinc(k*Omc/np.pi)
# windowing
w = np.ones(N)
h = hd * w

# frequency response
Om, H = sig.freqz(h)

# plot impulse response
plt.figure(figsize=(10, 3))
plt.stem(h)
plt.title('Impulse response')
plt.xlabel(r'$k$')
plt.ylabel(r'$h[k]$')
# plot magnitude responses
plt.figure(figsize=(10, 3))
plt.plot([0, Omc, Omc], [0, 0, -100], 'r--')
plt.plot(Om, 20 * np.log10(abs(H)))
plt.title('Magnitude response')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$|H(e^{j \Omega})|$ in dB')
plt.axis([0, np.pi, -20, 3])
plt.grid()
# plot phase responses
plt.figure(figsize=(10, 3))
plt.plot(Om, np.unwrap(np.angle(H)))
plt.title('Phase')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$\varphi (\Omega)$ in rad')
plt.grid()
```

**Exercises**

* Does the resulting filter have the desired phase?
* Increase the length `N` of the filter. What changes?

Solution: The desired filter has zero-phase for all frequencies, hence $\varphi_\text{d}(\Omega) = 0$. The phase of the resulting filter is not zero as can be concluded from the lower illustration. The small local variations (ripples) in the magnitude $|H(\mathrm{e}^{\,\mathrm{j}\,\Omega})|$ of the transfer function of the resulting filter decrease with an increasing number `N` of filter coefficients. The achievable attenuation in the stop-band of the low-pass does not change.

### Zero-Phase Filters

Above results show that an ideal-low pass cannot be realized very well by the window method. The reason is that an ideal-low pass has zero-phase, as most of the idealized filters.

Lets assume a general zero-phase filter with transfer function $H_d(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = A(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ with amplitude $A(\mathrm{e}^{\,\mathrm{j}\,\Omega})  \in \mathbb{R}$. Its impulse response $h_d[k] = \mathcal{F}_*^{-1} \{ H_d(e^{j \Omega} \}$ is conjugate complex symmetric

\begin{equation}
h_d[k] = h_d^*[-k]
\end{equation}

due to the symmetry relations of the DTFT. Removal of the anti-causal part by windowing will result in large deviations between the desired transfer function and the designed filter. Hence, zero-phase filters cannot be realized very well by simple windowing of the desired impulse response. This observation motivates to shift the desired impulse response to the center of the window in order to limit the effect of windowing. This can be achieved by replacing the zero-phase with a linear-phase, as illustrated in the following.

### Causal Linear-Phase Filters

The design of non-recursive FIR filters with a linear phase is often desired due to their constant group delay. Let's assume a system with generalized linear phase. For $|\Omega| < \pi$ its transfer function is given as

\begin{equation}
H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot \mathrm{e}^{-\mathrm{j} \alpha \Omega + \mathrm{j} \beta}
\end{equation}

where $A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \in \mathbb{R}$ denotes the amplitude of the filter, $\alpha$ its linear phase and $\beta$ a constant phase offset. Such a system can be decomposed into two cascaded systems: a zero-phase system with transfer function $A(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and an all-pass with phase $\varphi(\Omega) = - \alpha \Omega + \beta$. The linear phase term $- \alpha \Omega$ results in the desired constant group delay $t_g(\Omega) = \alpha$.

The impulse response $h[k]$ of a linear-phase system shows a specific symmetry which can be deduced from the symmetry relations of the DTFT for odd/even symmetry of $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ as

\begin{equation}
h[k] = \pm h[N-1-k]
\end{equation}

for $k=0, 1, \dots, N-1$ where $N \in \mathbb{N}$ denotes the length of the (finite) impulse response. The transfer function of a linear phase filter is given by its DTFT

\begin{equation}
H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sum_{k=0}^{N-1} h[k] \, \mathrm{e}^{\,-\mathrm{j}\,\Omega\,k}
\end{equation}

Introducing the symmetry relations of the impulse response $h[k]$ into the DTFT and comparing the result with above definition of a generalized linear phase system reveals four different types of linear-phase systems. These can be discriminated with respect to their phase and magnitude characteristics

| Type | Length $N$ | Impulse Response $h[k]\;$ | Group Delay $\alpha$ in Samples| Constant Phase $\beta$ | Transfer Function $A(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ |
| :---:  | :---: | :---: | :---:| :---: | :---: |
| 1 | odd | $h[k] = h[N-1-k]$ | $\alpha = \frac{N-1}{2} \in \mathbb{N}$ | $\beta = \{0, \pi\}$ | $A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = A(\mathrm{e}^{-\,\mathrm{j}\,\Omega})$, all filter characteristics|
| 2 | even| $h[k] = h[N-1-k]$ | $\alpha = \frac{N-1}{2} \notin \mathbb{N}$ | $\beta = \{0, \pi\}$ | $A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = A(\mathrm{e}^{-\,\mathrm{j}\,\Omega})$, $A(\mathrm{e}^{\,\mathrm{j}\,\pi}) = 0$, only lowpass or bandpass|
| 3 | odd | $h[k] = -h[N-1-k]$ | $\alpha = \frac{N-1}{2} \in \mathbb{N}$ | $\beta = \{ \frac{\pi}{2}, \frac{3 \pi}{2} \} $ | $A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = - A(\mathrm{e}^{-\,\mathrm{j}\,\Omega})$, $A(\mathrm{e}^{\,\mathrm{j}\,0}) = A(\mathrm{e}^{\,\mathrm{j}\,\pi}) =  0$, only bandpass|
| 4 | even | $h[k] = -h[N-1-k]$ | $\alpha = \frac{N-1}{2} \notin \mathbb{N}$ | $\beta = \{ \frac{\pi}{2}, \frac{3 \pi}{2} \} $ | $A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = - A(\mathrm{e}^{-\,\mathrm{j}\,\Omega})$, $A(\mathrm{e}^{\,\mathrm{j}\,0}) = 0$, only highpass or bandpass|

These relations have to be considered in the design of a causal linear phase filter. Depending on the desired frequency characteristics $A(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ the suitable type is chosen. The odd/even length $N$ of the filter and the phase (or group delay) is chosen accordingly for the design of the filter. It is also abovious that a filter with zero-phase $\alpha = 0$, e.g. an ideal low-pass, would result in $N=1$.

#### Example - Causal linear-phase approximation of ideal low-pass

We aim at the design of a causal linear-phase low-pass using the window technique. According to the previous example, the desired frequency response has an even symmetry $A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = A(\mathrm{e}^{\,-\mathrm{j}\,\Omega})$ with $A(\mathrm{e}^{\mathrm{j}\,0}) = 1$. This could be realized by a filter of type 1 or 2. We choose type 1, since the resulting filter then exhibits an integer group delay of $t_g(\Omega) = \frac{N-1}{2}$ samples. Consequently the length of the filter $N$ has to be odd.

The impulse response $h_\text{d}[k]$ is given by the inverse DTFT of $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ as

\begin{equation}
h_\text{d}[k] = \frac{\Omega_\text{c}}{\pi} \cdot \text{si} \left[ \Omega_\text{c} \left(k-\frac{N-1}{2} \right) \right]
\end{equation}

The impulse response fulfills the desired symmetry for $k=0,1, \dots, N-1$. A causal and FIR approximation is obtained by applying a window function of length $N$ to the impulse response $h_\text{d}[k]$

\begin{equation}
h[k] = h_\text{d}[k] \cdot w[k]
\end{equation}

Note that the window function $w[k]$ also has to fulfill the desired symmetries.

As already outlined, the chosen window determines the properties of the transfer function $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. The [spectral properties of commonly applied windows](../spectral_analysis_deterministic_signals/window_functions.ipynb) have been discussed previously. The width of the main lobe will generally influence the  smoothing of the desired transfer function $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$, while the sidelobes influence the typical ringing artifacts. This is illustrated in the following.

```python
N = 33  # length of filter
Omc = np.pi/2

# compute impulse response
k = np.arange(N)
hd = Omc/np.pi * np.sinc((k-(N-1)/2)*Omc/np.pi)
# windowing
w1 = np.ones(N)
w2 = np.blackman(N)
h1 = hd * w1
h2 = hd * w2

# frequency responses
Om, H1 = sig.freqz(h1)
Om, H2 = sig.freqz(h2)

# plot impulse response
plt.figure(figsize=(10, 3))
plt.stem(h1)
plt.title('Impulse response (rectangular window)')
plt.xlabel(r'$k$')
plt.ylabel(r'$h[k]$')
# plot magnitude responses
plt.figure(figsize=(10, 3))
plt.plot([0, Omc, Omc], [0, 0, -300], 'r--')
plt.plot(Om, 20 * np.log10(abs(H1)), label='rectangular window')
plt.plot(Om, 20 * np.log10(abs(H2)), label='Blackmann window')
plt.title('Magnitude response')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$|H(e^{j \Omega})|$ in dB')
plt.axis([0, np.pi, -120, 3])
plt.legend(loc=3)
plt.grid()
# plot phase responses
plt.figure(figsize=(10, 3))
plt.plot(Om, np.unwrap(np.angle(H1)), label='rectangular window')
plt.plot(Om, np.unwrap(np.angle(H2)), label='Blackmann window')
plt.title('Phase')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$\varphi (\Omega)$ in rad')
plt.legend(loc=3)
plt.grid()
```

**Exercises**

* Does the impulse response fulfill the required symmetries for a type 1 filter?
* Can you explain the differences between the magnitude responses $|H(\mathrm{e}^{\,\mathrm{j}\,\Omega})|$ for the different window functions?
* What happens if you increase the length `N` of the filter?

Solution: Inspection of the impulse response reveals that it shows the symmetry $h[k] = h[N-1-k]$ of a type 1 filter for odd `N`. The rectangular window features a narrow main lobe at the cost of a high level of the side lobes, the main lobe of the Blackmann window is wider but the level of the side lobes is lower compared to the rectangular window. This explains the behavior of the magnitude responses $|H(\mathrm{e}^{\,\mathrm{j}\,\Omega})|$ in the stop-band of the realized low-passes. The distance between the local minima in the magnitude responses $|H(\mathrm{e}^{\,\mathrm{j}\,\Omega})|$ decreases with increasing length `N` and the attenuation for frequencies towards the Nyquist frequency increases.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2018*.


```python
def downloadDriveFile(file_id,file_name,file_extension):
  '''
  Allows charge of public files into colab's workspace
  '''
  !wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id='$file_id -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id="$file_id -O "$file_name"."$file_extension" && rm -rf /tmp/cookies.txt

if 'google.colab' in str(get_ipython()):
    downloadDriveFile('1KkHZjTDShCz0uBEHWAZ9xx98F6ENMvrq','optimize','py')
```

```python
import numpy as np
import sympy as sp
import matplotlib.pyplot as plt

import optimize
```

Steepest Descent Method
=========

Example 1
----
Find the minimizer of $f(\mathbf{x}) = x_1^{e−(x_1^2 + x_2^2)}$.

```python
x = sp.symbols('x1 x2')
func = x[0]*sp.exp(-x[0]**2 - x[1]**2)
x0 = [-1, 1]

min, search_steps = optimize.steepest_descent(func, x, start=x0, full_output=True)
min_fix, search_steps_fix = optimize.steepest_descent(func, x, start=x0, step_type='fix', step=0.25, full_output=True)
```

# Sección nueva

```python
fig = plt.figure()

step = 0.01
x_vals = np.arange(-1.6, 0, step)
y_vals = np.arange(-1.6, 1.6, step)
X, Y = np.meshgrid(x_vals, y_vals)

function = sp.lambdify([*x], func, modules='numpy')
func_eval = function(X,Y)

ax1 = fig.add_subplot(1,2,1)
ax1.contour(X, Y, func_eval)
ax1.plot(search_steps[:, 0],search_steps[:, 1],'-k.', linewidth='0.8', markersize=6)
ax1.set_xlabel('$x_1$')
ax1.set_ylabel('$x_2$')

ax2 = fig.add_subplot(1,2,2)
ax2.contour(X, Y, func_eval)
ax2.plot(search_steps_fix[:, 0],search_steps_fix[:, 1],'-k.', linewidth='0.8' , markersize=6)
ax2.set_xlabel('$x_1$')
ax2.set_ylabel('$x_2$')

fig_dpi = 150
fig_width = 18.8/2.54
fig_height = 9.4/2.54

fig.dpi=fig_dpi
fig.set_size_inches((fig_width,fig_height))
fig.tight_layout()

plt.show()
```

Example 2
----
Find the minimizer of $f(\mathbf{x}) = 0.06e^{(2x_1+x_2)} + 0.05e^{(x_1-2x_2)} + e^{-x_1}$.

```python
x = sp.symbols('x1 x2')
func = 0.06*sp.exp(2*x[0]+x[1]) + 0.05*sp.exp(x[0]-2*x[1]) + sp.exp(-x[0])
x0 = [-0.5, 2]

min, search_steps = optimize.steepest_descent(func, x, start=x0, full_output=True)
min_fix, search_steps_fix = optimize.steepest_descent(func, x, start=x0, step_type='fix', step=0.25, full_output=True)
```

```python
fig = plt.figure()

step = 0.001
x_vals = np.arange(-1, 1.75, step)
y_vals = np.arange(-2, 3.5, step)
X, Y = np.meshgrid(x_vals, y_vals)

function = sp.lambdify([*x], func, modules='numpy')
func_eval = function(X,Y)

ax1 = fig.add_subplot(1,2,1)
ax1.contour(X, Y, func_eval)
ax1.plot(search_steps[:, 0],search_steps[:, 1],'-k.', linewidth='0.8', markersize=6)
ax1.set_xlabel('$x_1$')
ax1.set_ylabel('$x_2$')

ax2 = fig.add_subplot(1,2,2)
ax2.contour(X, Y, func_eval)
ax2.plot(search_steps_fix[:, 0],search_steps_fix[:, 1],'-k.', linewidth='0.8' , markersize=6)
ax2.set_xlabel('$x_1$')
ax2.set_ylabel('$x_2$')

fig_dpi = 150
fig_width = 18.8/2.54
fig_height = 9.4/2.54

fig.dpi=fig_dpi
fig.set_size_inches((fig_width,fig_height))
fig.tight_layout()

plt.show()
```

```python

```


# Deterministic Variational Inference: Toy example
Code to accompany [**Fixing Variational Bayes: Deterministic Variational Inference for Bayesian Neural Networks**](https://arxiv.org/abs/1810.03958)

```python
%matplotlib inline

import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
from IPython.display import clear_output
import os, json

import gaussian_variables as gv
import utils as u
import plot_utils as pu
import bayes_layers as bnn
from bayes_models import MLP, PointMLP, AdaptedMLP

np.random.seed(3)
```

## Dataset

First we generate a toy dataset according to:

\begin{equation}
    y = -(x+0.5)\sin(3\pi x) + \eta
\end{equation}

Where the noise is generated according to:

\begin{equation}
    \eta = 0.45(x + 0.5)^2
\end{equation}

```python
def base_model(x):
    return -(x+0.5)*np.sin(3 * np.pi *x)

def noise_model(x):
    return 0.45*(x+0.5)**2

def sample_data(x):
    return base_model(x) + np.random.normal(0, noise_model(x))

data_size = {'train': 500, 'valid': 100, 'test': 100}
toy_data = []
for section in ['train', 'valid', 'test']:
    x = (np.random.rand(data_size['train'], 1) - 0.5)
    toy_data.append([x, sample_data(x).reshape(-1)])
x = np.arange(-1,1,1/100)
toy_data.append([[[_] for _ in x], base_model(x)])

pu.toy_results_plot(toy_data, {'mean':base_model, 'std':noise_model})
```

# Model

Here we set the hyperparameters and provide a function for constructing a Bayesian Neural Network.

```python
hypers = {
    "x_dim": 1,                                "y_dim": 2,
    "hidden_dims": [128,128],                  "nonlinearity": "relu",
    "adapter": {
        'in' : {"scale": [[1.0]], "shift": [[0.0]]},
        'out': {"scale": [[1.0, 0.83]], "shift": [[0.0, -3.5]]}
    },

    "method": "bayes",                         "style": "heteroskedastic",
    "homo_logvar_scale": 2*np.log(0.2),        "prior_type": ["empirical", "wider_he", "wider_he"],

    "n_epochs": 20000,                         "batch_size": 500,
    "learning_rate": 0.001,                    "lambda": 1.0,
    "warmup_updates": {'lambda': 14000.0},     "anneal_updates": {'lambda': 1000.0},
    "optimizer": "adam",                       "gradient_clip": 0.1,

    "data_fraction": 1.0,                      "sections_to_run": ["train"]
}
```

```python
def make_model(hypers):
    if hypers['method'].lower().strip() == 'bayes':
        MLP_factory = MLP
        prediction = lambda y: tf.reshape(y.mean[:,0], [-1])
        loss = bnn.regression_loss
    else:
        MLP_factory = PointMLP
        prediction = lambda y: tf.reshape(y.mean[:,0], [-1])
        loss = bnn.point_regression_loss

    mlp = MLP_factory(hypers['x_dim'], hypers['y_dim'], hypers)
    mlp = AdaptedMLP(mlp)
    mlp.make_placeholders()
    ipt = mlp.placeholders['ipt_mean']
    y = mlp(ipt)

    target = tf.placeholder(tf.float32, [None])
    mlp.placeholders['target'] = target
    global_step = tf.Variable(0, trainable=False, name='global_step')
    loss, logprob, all_surprise = loss(y, target, mlp, hypers, global_step)

    accuracy = tf.reduce_mean(tf.abs(target - prediction(y)))

    return {
        'model': mlp,
        'metrics': {
            'accuracy': accuracy,   'loss': loss,
            'logprob': logprob,     'all_surprise': all_surprise
            },
        'global_step': global_step}
```

## Training loop

For the first `warmup_updates` epochs we run with the KL term in the elbo turned off. You will observe that the model starts to overfit the data. The KL term is then ramped back on and the model is trained with the full ELBO to convergence.

```python
run_id = u.start_run()

data = toy_data

restricted_training_set = u.restrict_dataset_size(data[0], hypers['data_fraction'])
hypers['dataset_size'] = len(restricted_training_set[0])

device_id = 0
device_string = u.get_device_string(device_id)
print(hypers)
with tf.device(device_string):
    if True:
        model_and_metrics = make_model(hypers)
        train_op = u.make_optimizer(model_and_metrics, hypers)
        sess = u.get_session()
        saver = tf.train.Saver()

        all_summaries = []
        best_valid_accuracy = np.inf

    for epoch in range(hypers['n_epochs']):
        verbose = (epoch % 20 == 0)
        if verbose:
            print("Epoch %i:        " % epoch, end='')
        epoch_summary, accuracies = u.train_valid_test(
            {
                'train': restricted_training_set,
                'valid': data[1],
                'test': data[2]
            },
            sess, model_and_metrics, train_op, hypers, verbose)

        if epoch % 200 == 0:
            clear_output()
            predictions = u.get_predictions(data[3], sess, model_and_metrics['model'], hypers)
            pu.toy_results_plot(toy_data, {'mean':base_model, 'std':noise_model}, hypers, predictions)
            plt.show()

        # dump log file
        all_summaries.append(epoch_summary)
        #with open(os.path.join("logs", run_id, "results.json"), 'w') as f:
        #    json.dump(all_summaries, f, indent=4, cls=u.NumpyEncoder)
```

## Comparison with MC

We compare the output activations of the BNN as predicted using our deterministic approximation with the activations predicted by Monte Carlo approximation (see Figure 3 of the paper)

```python
from matplotlib import rc
rc('text', usetex=True)

n_samp = 20000
x=0.25
ipt = [[[x]] for _ in range(n_samp)]
sample_op = model_and_metrics['model'].run_with_MC(ipt, n_sample=n_samp)
approx_op = model_and_metrics['model'](x)

samples = sess.run(sample_op)
approx = sess.run([approx_op.mean, approx_op.var])

def gaussian1d(mean, var):
    x_axis = np.linspace(-5,5,1000)
    return x_axis, 1.0 / np.sqrt(2.0 * np.pi * var) * np.exp(-1.0 / (2.0 * var) * (x_axis - mean)**2)

samples.shape
plt.figure(figsize=(3,2))
plt.hist(samples[:,0,0], int(np.round(200/(1-0.3))), density=True, edgecolor='k', facecolor='#b4c7e7')
plt.plot(*gaussian1d(approx[0][0,0], approx[1][0,0,0]), 'b')
plt.xlim([-1,-0.3])
plt.yticks([])
plt.xlabel('$m$')
plt.ylabel('$q(m)$')
plt.show()

plt.figure(figsize=(3,2))
plt.hist(samples[:,0,1], int(np.round(200/(3.5-1.9))), density=True, edgecolor='k', facecolor='#b4c7e7', label="MC")
plt.plot(*gaussian1d(approx[0][0,1], approx[1][0,1,1]), 'b', label="ours")
plt.xlim([-3.5,-1.9])
plt.yticks([])
plt.xlabel('$\ell$')
plt.ylabel('$q(\ell)$')
plt.legend()
plt.show()
```

```python

```


Stitch Fix, Jupyter, GitHub, and the Long Tail
==

At Stitch Fix we are avid users of Jupyter for research at both the personal and team scales. At the personal level, Jupyter is a great interface to research the question at hand. It captures the workflow of the research where we can take detailed notes on the code and explain models with written content and mathematical equations.

At the team level, Jupyter is a great tool for communication. Notebooks allow one to fluidly mix text, code, equations, data, and plots in whatever way makes the most sense to explain something. You can organize your explanation around your thought process rather than around the artificial lines determined by the tools you’re using.  You have a single Jupyter notebook instead of a bunch of disconnected files, one containing SQL code, another Python, a third LaTeX to typeset equations.

When the analysis is finished, the Jupyter notebook remains a “living” interactive research notebook.  Re-doing the analysis using new data or different assumptions is simple.

Stitch Fix provides clothing to its clients based on their style preferences.  We send a client five pieces of clothing we predict they’ll like, and the client chooses what to keep. Inevitably, some pieces of clothing will be more popular than others. In some cases, a few select items may be unpopular.

The largest benefit from adding a single style of clothing to our line of inventory comes from the most popular one. Each of the less popular styles, by itself, contributes less.  However, there are *many* of the less popular ones, reflecting the fact that our clients are unique in their fashion preferences. Together, the value in the "long tail" can match or exceed the value of the few products in the "head."  Catering to the long tail allows us to save our clients the time they would otherwise spend searching through many retail stores to find clothing that’s unique to their tastes.

But, where do we draw the line on how far into the long tail we should support? Below we investigate this question using the Jupyter Notebook. The portability and flexibility of the Notebooks allows us to easily share the analysis with others.  GitHub integration allows a great new possibility: other researches can fork the notebook to extend or alter the analysis according to their own particular interests!

Is the value in the head or the tail?
--

We will approximate the number of each style of clothing sold as a power law of the rank $r$ by sales volume.  The most popular style has $r=1$, the second most popular $r=2$, and the least popular has $r=N$.  Consumer preferences dictate the shape of the curve.

Even though we may want to carry an infinite number of styles of clothing, it's important to keep $N$ finite so that the integrals converge!  For the moment we will consider a scaled-down version of Stitch Fix that only carrys 100 styles of clothing and sells a volume of $V=5,000$ units per year.

The volume of each style sold is
\begin{equation}
v(r) = \frac{A}{r^n}
\end{equation}
where $A$ is a normalization constant and $n$ is the index of the power law.  The value of $n$ will determine how much value is in the head versus the tail.

Approximating the product distribution as continuous so it can be written as an integral, the normalization constant is set by the constraint
\begin{equation}
\int_1^N \frac{A\,dr}{r^n} = V
\end{equation}
so
\begin{equation}
A = \frac{(n-1) V}{1-N^{1-n}}
\end{equation}

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
sns.set_context('talk')
sns.set_style('darkgrid')
```

```python
inventory = 100.0
volume = 5000.0

rr = np.linspace(1,inventory,100)
ns = [0.25, 0.75, 1.25, 1.75]

fig, ax = plt.subplots(figsize=(10, 6))

for nn in ns:
    norm = (nn-1)*volume/(1-inventory**(1-nn))
    ax.plot(rr, norm/rr**nn, label='$n=%g$' % nn)

ax.legend()
ax.set_xlabel('Rank by Sales Volume $r$')
ax.set_ylabel('Units Sold')
ax.set_title('Sales volume of each product by rank')
ax.set_ylim(0,100)
```

All of these distributions have the same area under the curve, so they represent the same total number of units sold.  Smaller values of $n$ give flatter distributions (less head, more tail) and larger values of $n$ give more head-heavy distributions.

What is the _total_ value in the head versus the tail?  Define the head to be the 10% of styles with the largest sales volume, the tail to be the 50% of styles with the lowest sales volumes, and the middle to be those in between.

That is, the head, tail, and middle look like this:

```python
# Same plot as above
fig, ax = plt.subplots(figsize=(10, 6))

for nn in ns:
    norm = (nn-1)*volume/(1-inventory**(1-nn))
    ax.plot(rr, norm/rr**nn, label='$n=%g$' % nn)

ax.set_xlabel('Rank by Sales Volume $r$')
ax.set_ylabel('Units Sold')
ax.set_title('Sales volume of each product by rank')
ax.set_ylim(0,100)

# Ask seaborn for some pleasing colors
c1, c2, c3 = sns.color_palette(n_colors=3)

# Add transparent rectangles
head_patch = plt.matplotlib.patches.Rectangle((1,0), 9, 100, alpha=0.25, color=c1)
middle_patch = plt.matplotlib.patches.Rectangle((11,0), 39, 100, alpha=0.25, color=c2)
tail_patch = plt.matplotlib.patches.Rectangle((51,0), 48, 100, alpha=0.25, color=c3)
ax.add_patch(head_patch)
ax.add_patch(middle_patch)
ax.add_patch(tail_patch)

# Add text annotations
ax.text(5,50,"Head", color=c1, fontsize=16, rotation=90)
ax.text(25,80,"Middle", color=c2, fontsize=16)
ax.text(75,80,"Tail", color=c3, fontsize=16)
```

How many units from the head, tail, and middle are sold?  Integrate over the sales rank distribution to get the sales volume in the head:

\begin{equation}
V_H = \int_1^{f_H N} \frac{A\, dr}{r^n} = \frac{V(N^{n-1} - f_H^{1-n})}{N^{n-1} - 1}
\end{equation}
where $f_H=0.1$

The volume in the tail is

\begin{equation}
V_T = \int_{f_T N}^N \frac{A\, dr}{r^n} = \frac{V(f_T^{1-n}-1)}{N^{n-1}-1}
\end{equation}
where $f_T=0.5$

And the middle:

\begin{equation}
V_M = \int_{f_H N}^{f_T N} \frac{A\, dr}{r^n} = \frac{V(f_H^{1-n} - f_T^{1-n})}{N^{n-1}-1}
\end{equation}

```python
f_head = 0.1
f_tail = 0.5

ns = np.linspace(0,2,100)
nm1 = ns-1.0

head = volume*(inventory**nm1 - f_head**-nm1)/(inventory**nm1-1)
middle = volume*(f_head**-nm1 - f_tail**-nm1)/(inventory**nm1-1)
tail = volume*(f_tail**-nm1 - 1)/(inventory**nm1-1)

fig, ax = plt.subplots(figsize=(10, 6))

ax.plot(ns, head, label='Head')
ax.plot(ns, middle, label='Middle')
ax.plot(ns, tail, label='Tail')
ax.legend(loc='upper left')
ax.set_ylabel('Units Sold')
ax.set_xlabel('Power law index $n$')
```

For $n>1$, the head has most of value.  As $n$ falls, the middle and tail become important.

How many styles of clothing should we carry?
--

We can choose expand our inventory from $N$ to $N+1$ styles of clothing.  How many additional units will we sell?

This is just the sales volume distribution $n(r)$ evaluated at $r=N+1$

\begin{equation}
\frac{d V}{d N} = \frac{(n-1) V}{(1-N^{1-n})(N+1)^n}
\end{equation}

```python
marginal_benefit = ((ns-1)*volume)/((1-inventory**(1-ns))*(inventory+1)**ns)

fig, ax = plt.subplots(figsize=(10, 6))

ax.plot(ns, marginal_benefit)
ax.set_ylabel('Additional Units Sold')
ax.set_xlabel('Power law index $n$')
ax.set_title('Marginal Benefit of Expanding Inventory')
```

Our scaled down version of Stitch Fix can expect to sell an additional $\simeq 40$ pieces of clothing per year if the sales distribution has $n=0.25$, or an additional $\simeq 3$ units per year if $n=1.5$  Whether expanding the inventory is worth it to get those additional sales depends entirely on how much it costs to bring one additional product into the inventory.


```python
import pandas as pd
from matplotlib import pyplot as plt
import numpy as np
import sympy
import matplotlib as mpl
```

```python
mpl.rc("text", usetex=False)
```

# Punto A #

```python
p=pd.read_csv("DataSet_Resolution_50",header=None)
```

```python
fig1,ax1=plt.subplots(figsize=(8,7))
(n1, bins1, patches1)=ax1.hist(p[0],bins=60,range=(5, 14))
ax1.set_xlabel("logE",fontsize=12)
ax1.set_ylabel("Counts",fontsize=12)
#ax.legend("Mean"+str(p.mean()))
ax1.set_title("Raw Counts",fontsize=15,pad=15)
plt.savefig("Raw_Counts_1.png")
```

```python
cost1=2*np.pi*3.15 * 10**7
```

```python
fig2,ax2=plt.subplots(figsize=(8,7))
(n2, bins2, patches2)=ax2.hist(p[0],bins=60,range=(5, 14),weights=(np.zeros_like(p)+1)/cost1)
ax2.set_xlabel("logE",fontsize=12)
ax2.set_ylabel("Counts",fontsize=12)
#ax.legend("Mean"+str(p.mean()))
ax2.set_title("Raw Counts 2",fontsize=15,pad=15)
plt.savefig("Raw_Counts_2.png")
```

```python
data1=[n2[i-1]/(10**bins1[i]-10**bins1[i-1]) for i in range(1,len(bins1))]
```

```python
fig3,ax3=plt.subplots(figsize=(8,7))
ax3.plot(bins1[:-1],data1)
#ax.plot(energy[:-1],n5)
#ax.set_xscale('log')
ax3.set_yscale("log")
ax3.set_ylabel("J (1/(km^2*yr*sr*eV))",fontsize=12)
ax3.set_xlabel("logE",fontsize=12)
ax3.set_title("Raw Spectrum",fontsize=15,pad=15)
plt.savefig("Raw_Spectrum_plot.png")
```

```python
fig4,ax4=plt.subplots(figsize=(8,7))
(n4, bins4, patches4)=ax4.hist(p[0],bins=60,range=(8, 14), rwidth=0.85,color='#0504aa')
ax4.set_xlabel("logE",fontsize=12)
ax4.set_ylabel("Counts",fontsize=12)
ax4.set_yscale("log")
ax4.grid(axis='y', alpha=0.75)
#ax4.set_ylim(0,10**(-9))
#ax.legend("Mean"+str(p.mean()))
ax4.set_title("Raw Binned Event Rate",fontsize=15,pad=15)
plt.savefig("Raw Binned Event Rate.png")
```

```python
fig5,ax5=plt.subplots(figsize=(8,7))
(n5, bins5, patches5)=ax5.hist(p[0],bins=60,range=(8, 14),weights=(np.zeros_like(p)+1)/cost1, rwidth=0.85,color='#0504aa')
ax5.set_xlabel("logE",fontsize=12)
ax5.set_ylabel("Flux [1/(m^2*s*sr)]",fontsize=12)
ax5.set_yscale("log")
ax5.grid(axis='y', alpha=0.75)
#ax.legend("Mean"+str(p.mean()))
ax5.set_title("Integral Flux",fontsize=15,pad=15)
plt.savefig("Integral Flux.png")
```

```python
data2=[n5[i-1]/(10**bins5[i]-10**bins5[i-1]) for i in range(1,len(bins5))]
```

```python
fig6,ax6=plt.subplots(figsize=(8,7))
(n6, bins6, patches6)=ax6.hist(bins5[:-1],bins=60,range=(8, 14),weights=data2, rwidth=0.85,color='#0504aa')
ax6.set_xlabel("logE",fontsize=12)
ax6.set_ylabel("Flux [1/(m^2*s*sr*eV)]",fontsize=12)
ax6.set_yscale("log")
ax6.grid(axis='y', alpha=0.75)
#ax.legend("Mean"+str(p.mean()))
ax6.set_title("J Raw",fontsize=15,pad=15)
plt.savefig("J Raw.png")
```

# Punto C#

```python
J0=6.57523*10**(-9)
gamma=-2.29989
exposure=np.pi*10**7*2*np.pi
```

## True Binned Event Rate ##

```python
def counts_fit(x):
    return J0*pow(10,gamma*(x-8))*exposure
```

```python
real_counts=[counts_fit((bins4[i]+bins4[i-1])/2)*(10**bins4[i]-10**bins4[i-1]) for i in range(1,len(bins4))]
```

```python
fig7,ax7=plt.subplots(figsize=(8,7))
(n7, bins7, patches7)=ax7.hist(bins5[:-1],bins=60,range=(8, 14),weights=real_counts, rwidth=0.85,color='red')
ax7.set_xlabel("logE",fontsize=12)
ax7.set_ylabel("Counts",fontsize=12)
ax7.set_yscale("log")
ax7.grid(axis='y', alpha=0.75)
#ax.legend("Mean"+str(p.mean()))
ax7.set_title("True Binned Event Rate",fontsize=15,pad=15)
plt.savefig("True Binned Event Rate.png")
```

```python
fig8,ax8=plt.subplots(figsize=(8,7))
ax8.hist(bins5[:-1],bins=60,range=(8, 14),weights=real_counts, rwidth=0.85,color='red',alpha=0.8,label="True")
ax8.hist(p[0],bins=60,range=(8, 14), rwidth=0.85,color='#0504aa',alpha=0.8,label="Raw")
ax8.set_xlabel("logE",fontsize=12)
ax8.set_ylabel("Counts",fontsize=12)
ax8.set_yscale("log")
ax8.grid(axis='y', alpha=0.75)
ax8.legend()
ax8.set_title("Binned Event Rate",fontsize=15,pad=15)
plt.savefig("Binned Event Rate.png")
```

## True Spectrum ##

```python
def J_fit(x):
    return J0*pow(10,gamma*(x-8))
```

```python
real_J=[J_fit((bins4[i]+bins4[i-1])/2) for i in range(1,len(bins4))]
```

```python
energy=[10**((bins4[i]+bins4[i-1])/2) for i in range(1,len(bins4))]
```

```python
fig7,ax7=plt.subplots(figsize=(7,7))
ax7.plot(energy,real_J)
ax7.set_xscale('log')
ax7.set_yscale("log")
ax7.set_ylabel("J (1/(km^2*yr*sr*eV))",fontsize=12)
ax7.set_xlabel("Energy [eV]",fontsize=12)
ax7.set_title("True Spectrum",fontsize=15,pad=15)
plt.savefig("True Spectrum.png")
```

# Punto D #

```python
fig8,ax8=plt.subplots(figsize=(7,7))
ax8.plot(energy,real_J,label="True")
ax8.plot(energy,data2,label="Raw")
ax8.set_xscale('log')
ax8.set_yscale("log")
ax8.set_ylabel("J (1/(m^2*s*sr*eV))",fontsize=12)
ax8.set_xlabel("Energy [eV]",fontsize=12)
ax8.set_title("Spectrum",fontsize=15,pad=15)
ax8.legend()
plt.savefig("Spectrum.png")
```

```python
true_raw_fraction=[real_J[i]/data2[i] for i in range(0,len(n5))]
```

    /home/andry/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:1: RuntimeWarning: divide by zero encountered in double_scalars
      """Entry point for launching an IPython kernel.

```python
fig9,ax9=plt.subplots(figsize=(6,6))
ax9.plot(energy,true_raw_fraction)
ax9.set_xscale('log')
#ax9.set_yscale('log')
ax9.set_ylim(0.8,1.1)
ax9.set_xlabel("Energy [eV]",fontsize=12)
ax9.set_ylabel("True J/Raw J",fontsize=12)
plt.savefig("True J_Raw J.png")
```

# Punto E #

```python
x = sympy.symbols('x')
```

```python
y=J0*pow(10,gamma*(x-8))*2*np.pi
```

```python
c=sympy.integrate(y,(x,9,9.1))
```

```python
c.evalf()*10**(9.05)
```

$\displaystyle 0.0180413613773858$

```python
J0*pow(10,gamma*(9.05-8))*2*np.pi*(10**9.1-10**9)
```

    0.041151056811310156

```python

```


```python
import numpy as np
import random
import statsmodels.api as sm
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.linear_model import LinearRegression
from sympy.stats import Bernoulli, sample_iter
```

15. Consider the following procedure.

• Set n = 100 and draw n continuous values x_i uniformly distributed between 0 and 10. Then simulate data from the model `y_i = a + bx_i + error_i`, for `i = 1,…,n, with a = 2, b = 3`, and independent errors from a normal distribution.

• Regress y on x. Look at the median and mad sd of b. Check to see if the interval formed by the median ± 2 mad sd includes the true value, b = 3.

• Repeat the above two steps 1000 times.

(a) True or false: You would expect the interval to contain the true value approximately 950 times. Explain your answer (in one sentence).

(b) Same as above, except the error distribution is bimodal, not normal. True or false: You would expect the interval to contain the true value approximately 950 times. Explain your answer (in one sentence).

```python
x = [random.randint(0, 10) for i in range(100)]
y = [2 + 3 * _x + random.normalvariate(0, 1) for _x in x]
```

```python
mod = sm.OLS(y, sm.add_constant(x))

res = mod.fit(q=0.5)

print(res.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:                      y   R-squared:                       0.986
    Model:                            OLS   Adj. R-squared:                  0.986
    Method:                 Least Squares   F-statistic:                     7151.
    Date:                Wed, 17 Jun 2020   Prob (F-statistic):           2.12e-93
    Time:                        09:08:45   Log-Likelihood:                -150.87
    No. Observations:                 100   AIC:                             305.7
    Df Residuals:                      98   BIC:                             310.9
    Df Model:                           1
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const          2.1504      0.202     10.620      0.000       1.749       2.552
    x1             2.9645      0.035     84.561      0.000       2.895       3.034
    ==============================================================================
    Omnibus:                        1.686   Durbin-Watson:                   2.324
    Prob(Omnibus):                  0.430   Jarque-Bera (JB):                1.232
    Skew:                           0.257   Prob(JB):                        0.540
    Kurtosis:                       3.177   Cond. No.                         10.8
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

```python
# 'robust' mad
print(sm.robust.scale.mad(x, axis=0))
```

    4.447806655516806

```python
fig, ax = plt.subplots()
fig = sm.graphics.plot_fit(res, 1, ax=ax)
```

#### Boostrap

```python
x = np.array([random.randint(0, 10) for i in range(100)])
y = np.array([2 + 3 * _x + random.normalvariate(0, 1) for _x in x])

X = np.vstack([x, np.ones(len(x))]).T

plt.figure(figsize=(12,8))

for i in range(0, 1000):
    sample_index = np.random.choice(range(0, len(y)), len(y))

    X_samples = X[sample_index]
    y_samples = y[sample_index]

    lr = LinearRegression()
    lr.fit(X_samples, y_samples)
    plt.plot(x, lr.predict(X), color='grey', alpha=0.1, zorder=1)

plt.scatter(x,y, marker='o', color='orange', zorder=4)

lr = LinearRegression()
lr.fit(X, y)
plt.plot(x, lr.predict(X), color='green', linewidth=3, zorder=5)
plt.show()
```

### Responses

a.) True. Two standard deviations from the median should result in a confidence interval where 95% of estimates contain the actual value.

b.) True because of the CLT


## 미분

예측모형 함수의 계수 = 모수

우리가 결정할 수 있는 것은 선형예측모형의 가중치 $w$

이 가중치 $w$의 값이 모수이다.

- 목적함수(objective function): 성능함수, 손실함수, 비용함수, 오차함수 등 최적화의 대상이 되는 함수를 통틀은 것

- 최적화(optmization) : 목적함수가 주어졌을 때 이 목적함수의 값을 가장 크게 혹은 적게 할 수 있는 모수를 구하는 것

- **미분(differentiation)** : 최적화를 하기 위해서는 입력값이 변했을 때 함수의 출력값이 어떻게 달라지는지를 정량적으로 알아야 한다. 이러한 과정.

### 기울기
**기울기(slope)** $dx$값이 0으로 근접할 때의 변화율을 말한다.

$
\text{slope} = \lim_{\Delta x \rightarrow 0} \dfrac{f(x + \Delta x) - f(x)}{\Delta x}
$

### 수치미분(numerical differentiation)
- scipy.misc 패키지의 ``derivative()`` 명령으로 수치적으로 대략적인 기울기를 계산가능.

대략적인 기울기 수식

$
\text{slope} \approx \dfrac{f\left(x + \dfrac{1}{2}dx\right) - f\left(x - \dfrac{1}{2}dx\right)}{dx}
$

이렇게 기울기를 구하는 방법을 **수치미분**이라고 한다.

```python
from scipy.misc import derivative

def f(x):
    return x ** 4 - 4 * x ** 3 + x ** 2 + x

print(derivative(f, 0, dx=1e-6))
```

    0.9999999999960001

### 미분
- **어떤 함수로부터 그 함수 기울기를 출력하는 새로운 함수를 만들어내는 작업**
- **도함수(derivative)** : 미분으로 만들어진 함수

    -    $
         f' = \dfrac{d}{dx}(f) = \dfrac{df}{dx}
         $

- 읽는 방법

    - $f'$ =  "f 프라임(prime)"
    - $\dfrac{df}{dx}$는 "df 오버(over) dx"

### 미분가능
- 함수가 $x=0$인 위치에서 기울기를 정의할 수 없는 경우
    - 이런 경우에는 $x=0$에서 **미분 불가능(not differentiable)**
- 함수가 $x=0$인 위치에서 기울기를 정의할 수 있는 경우
    - **미분 가능(differentiable)**

### 미분공식

#### 1. 기본 미분공식
- 상수, 거듭제곱, 로그함수, 지수함수 등의 간단한 함수에 대한 미분공식
    - (1) 상수
        - 상수를 미분하면 0이 된다.

            $
            \dfrac{d}{dx}(c) = 0
            $

    - (2) 거듭제곱
        - $x$의 $n$ 제곱을 미분하면 $n-1$ 제곱으로 제곱수가 1씩 감소. $n=0$일 때는 성립하지 않는다.

            $
            \dfrac{d}{dx}(x^n) = n x^{n-1}
            $

    - (3) 로그
        - 로그함수를 미분하면 $x^{-1}$이 된다.

            $
            \dfrac{d}{dx}(\log x) = \dfrac{1}{x}
            $

    - (4) 지수

        - 밑이 오일러 수인 지수함수는 미분해도 변하지 않음.

            $
            \dfrac{d}{dx}(e^x) = e^x
            $

#### 2. 선형조합법칙
- 상수를 곱한 후 더한 **선형조합(linear combination)**은 각 함수의 도함수를 선형조합한 것과 같다.

$
\dfrac{d}{dx}\left(a_1 f_1 + a_2 f_2 \right) = a_1 \dfrac{df_1}{dx} + a_2 \dfrac{df_2}{dx}
$

#### 3. 곱셈법칙
- **곱셈법칙** : 어떤 함수의 형태가 두 함수를 곱한 것과 같을 때, 각 개별 함수의 도함수를 사용하여 원래 함수의 도함수를 구하는 것

$
\dfrac{d}{dx}\big( f  \cdot g \big) = f \cdot  \dfrac{dg}{dx} + \dfrac{df}{dx} \cdot g
$

#### 4. 연쇄법칙
- 미분하고자 하는 함수의 입력 변수가 다른 함수의 출력 변수인 경우 적용가능.

$
f(x) = h(g(x))
$

일 때, 도함수는

$
\dfrac{df}{dx} = \dfrac{dh}{dg} \cdot \dfrac{dg}{dx}
$

- 로그 함수에 연쇄법칙을 적용.

$
\dfrac{d}{dx} \log f(x) = \dfrac{f'(x)}{f(x)}
$

### 2차 도함수
- 도함수를 한 번 더 미분하여 만들어진 함수
- 2차 도함수는 2개의 prime 기호($''$)를 붙이거나 $\dfrac{d^2}{dx^2}$ 기호로 표시.

$
f'' = \dfrac{d^2}{dx^2}(f) = \dfrac{d^2f}{dx^2}
$

- 2차 도함수는 도함수의 기울기를 나타낸다
    - 도함수 값이 증가하면 2차 도함수 값은 양수 : 2차 도함수 값이 양수인 경우를 **볼록(convex)**
    - 도함수 값이 감소하면 2차 도함수 값은 음수 : 2차 도함수 값이 음수인 경우를 **오목(concave)**
    - 그래서 2차 도함수값을 **볼록도(convexity)**라고도 부른다.

### 편미분
- 함수가 둘 이상의 독립변수를 가지는 다변수 함수인 경우의 미분

- 편미분의 결과로 하나의 함수에 대해 여러 개의 도함수가 나올 수 있다.

- 편미분의 결과 즉 도함수는 독립 변수를 함수의 아랫첨자로 써서 표기하거나 $\partial$(round) 기호를 사용하여 표기.

    $
    f_x(x,y) = \dfrac{\partial f}{\partial x}
    $

    $
    f_y(x,y) = \dfrac{\partial f}{\partial y}
    $

- 편미분 방법
    - 변수가 하나인 함수의 미분과 같다.
    - **어떤 하나의 독립 변수에 대해 미분할 때는 다른 독립 변수를 상수로 생각한다**.
    - ex)

        $
        f(x,y) = x^2 + 4xy + 2y^2
        $

        $
        f_x(x,y) = \dfrac{\partial f}{\partial x} = 2x + 4y
        $

        $
        f_y(x,y) = \dfrac{\partial f}{\partial y} = 4x + 4y
        $

### 다변수 함수의 연쇄법칙
$
y_N = f_N(x)
$

$
z = g(y_1, y_2, \ldots, y_N)
$

이때 변수 $x$ 값의 변화에 따른 $z$ 값의 변화는 다음처럼 계산.

$
\dfrac{dz}{dx}
= \dfrac{\partial z}{\partial y_1}\dfrac{dy_1}{dx} + \cdots
+ \dfrac{\partial z}{\partial y_N}\dfrac{dy_N}{dx}
$

### 심파이(SymPy)를 이용한 미분
- **심볼릭 연산(symbolic operation)**을 지원하는 파이썬 패키지
    - 심볼릭 연산 : 사람이 연필로 계산하는 미분/적분과 동일한 형태의 연산

```python
import sympy
sympy.init_printing(use_latex='mathjax') # Juypter notebook에서 수학식의 LaTeX 표현을 위해
```

```python
x = sympy.symbols('x')
x
```

$$x$$

```python
f = x * sympy.exp(x)
f
```

$$x e^{x}$$


# Performance analysis of a uniform linear array

We compare the MSE of MUSIC with the CRB for a uniform linear array (ULA).

```python
import numpy as np
import doatools.model as model
import doatools.estimation as estimation
import doatools.performance as perf
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
wavelength = 1.0 # normalized
d0 = wavelength / 2

# Create a 12-element ULA.
ula = model.UniformLinearArray(12, d0)
# Place 8 sources uniformly within (-pi/3, pi/4)
sources = model.FarField1DSourcePlacement(
    np.linspace(-np.pi/3, np.pi/4, 8)
)
# All sources share the same power.
power_source = 1 # Normalized
source_signal = model.ComplexStochasticSignal(sources.size, power_source)
# 200 snapshots.
n_snapshots = 200
# We use root-MUSIC.
estimator = estimation.RootMUSIC1D(wavelength)
```

We vary the SNR from -20 dB to 20 dB. Here the SNR is defined as:
\begin{equation}
    \mathrm{SNR} = 10\log_{10}\frac{\min_i p_i}{\sigma^2_{\mathrm{n}}},
\end{equation}
where $p_i$ is the power of the $i$-th source, and $\sigma^2_{\mathrm{n}}$ is the noise power.

```python
snrs = np.linspace(-20, 10, 20)
# 300 Monte Carlo runs for each SNR
n_repeats = 300

mses = np.zeros((len(snrs),))
crbs_sto = np.zeros((len(snrs),))
crbs_det = np.zeros((len(snrs),))
crbs_stouc = np.zeros((len(snrs),))

for i, snr in enumerate(snrs):
    power_noise = power_source / (10**(snr / 10))
    noise_signal = model.ComplexStochasticSignal(ula.size, power_noise)
    # The squared errors and the deterministic CRB varies
    # for each run. We need to compute the average.
    cur_mse = 0.0
    cur_crb_det = 0.0
    for r in range(n_repeats):
        # Stochastic signal model.
        A = ula.steering_matrix(sources, wavelength)
        S = source_signal.emit(n_snapshots)
        N = noise_signal.emit(n_snapshots)
        Y = A @ S + N
        Rs = (S @ S.conj().T) / n_snapshots
        Ry = (Y @ Y.conj().T) / n_snapshots
        resolved, estimates = estimator.estimate(Ry, sources.size, d0)
        # In practice, you should check if `resolved` is true.
        # We skip the check here.
        cur_mse += np.mean((estimates.locations - sources.locations)**2)
        B_det = perf.ecov_music_1d(ula, sources, wavelength, Rs, power_noise,
                                   n_snapshots)
        cur_crb_det += np.mean(np.diag(B_det))
    # Update the results.
    B_sto = perf.crb_sto_farfield_1d(ula, sources, wavelength, power_source,
                                     power_noise, n_snapshots)
    B_stouc = perf.crb_stouc_farfield_1d(ula, sources, wavelength, power_source,
                                         power_noise, n_snapshots)
    mses[i] = cur_mse / n_repeats
    crbs_sto[i] = np.mean(np.diag(B_sto))
    crbs_det[i] = cur_crb_det / n_repeats
    crbs_stouc[i] = np.mean(np.diag(B_stouc))
    print('Completed SNR = {0:.2f} dB'.format(snr))
```

    Completed SNR = -20.00 dB
    Completed SNR = -18.42 dB
    Completed SNR = -16.84 dB
    Completed SNR = -15.26 dB
    Completed SNR = -13.68 dB
    Completed SNR = -12.11 dB
    Completed SNR = -10.53 dB
    Completed SNR = -8.95 dB
    Completed SNR = -7.37 dB
    Completed SNR = -5.79 dB
    Completed SNR = -4.21 dB
    Completed SNR = -2.63 dB
    Completed SNR = -1.05 dB
    Completed SNR = 0.53 dB
    Completed SNR = 2.11 dB
    Completed SNR = 3.68 dB
    Completed SNR = 5.26 dB
    Completed SNR = 6.84 dB
    Completed SNR = 8.42 dB
    Completed SNR = 10.00 dB

We plot the results below.

* The MSE should approach the stochastic CRBs in high SNR regions.
* The stochastic CRB should be tighter than the deterministic CRB.
* With the additional assumption of uncorrelated sources, we expect a even lower CRB.
* All three CRBs should converge together as the SNR approaches infinity.

```python
plt.figure(figsize=(8, 6))
plt.semilogy(
    snrs, mses, '-x',
    snrs, crbs_sto, '--',
    snrs, crbs_det, '--',
    snrs, crbs_stouc, '--'
)
plt.xlabel('SNR (dB)')
plt.ylabel(r'MSE / $\mathrm{rad}^2$')
plt.grid(True)
plt.legend(['MSE', 'Stochastic CRB', 'Deterministic CRB',
           'Stochastic CRB (Uncorrelated)'])
plt.title('MSE vs. CRB')
plt.margins(x=0)
plt.show()
```