Sascha Spors,
Professorship Signal Theory and Digital Signal Processing,
Institute of Communications Engineering (INT),
Faculty of Computer Science and Electrical Engineering (IEF),
University of Rostock,
Germany

# Tutorial Digital Signal Processing

**Random Signals and LTI Systems**,
Winter Semester 2021/22 (Course #24505)

- lecture: https://github.com/spatialaudio/digital-signal-processing-lecture
- tutorial: https://github.com/spatialaudio/digital-signal-processing-exercises

Feel free to contact lecturer frank.schultz@uni-rostock.de

# Fundamentals

## Cross Correlation

The cross correlation function (CCF) for two (complex-valued) signals $x[k]$ and $y[k]$ of finite length $N$ may be defined as

\begin{equation}
\varphi_{xy}[\kappa]=\frac{1}{N}\sum\limits_{k=0}^{N-1} x[k+\kappa]\cdot y^*[k] = \frac{1}{N}\sum\limits_{k=0}^{N-1} x[k]\cdot y^*[k-\kappa],
\end{equation}

following the notation convention of the lecture and denoting $^*$ as complex-conjugate.
A very important property is

\begin{equation}
\varphi_{xy}[\kappa]=\varphi^*_{yx}[-\kappa].
\end{equation}

Thus, only if the CCF is real-valued - which is the case if $x[k]$ and $y[k]$ are real valued - the equality $\varphi_{xy}[\kappa]=\varphi_{yx}[-\kappa]$ holds. Otherwise $\varphi_{xy}[\kappa]$ and $\varphi_{yx}[\kappa]$ are linked by time-mirroring **and** complex-conjugate.

## Cross Power Spectral Density

The so called cross power spectral density (CPSD) is the DTFT of the CCF, given as

\begin{equation}
\Phi_{xy}(\mathrm{e}^{\mathrm{j}\Omega})=\text{DTFT}\{\varphi_{xy}[\kappa]\}=\sum\limits_{\kappa=-\infty}^\infty\varphi_{xy}[\kappa] \cdot \mathrm{e}^{-\mathrm{j}\Omega\kappa}.
\end{equation}

Consequently, the inverse DTFT links the (cross)-PSD with the CCF as

\begin{equation}
\varphi_{xy}[\kappa]=\text{IDTFT}\{\Phi_{xy}(\mathrm{e}^{\mathrm{j}\Omega})\}=\frac{1}{2\pi}\int\limits_{-\pi}^\pi\Phi_{xy}(\mathrm{e}^{\mathrm{j}\Omega})\cdot\mathrm{e}^{+\mathrm{j}\Omega\kappa}\mathrm{d}\Omega.
\end{equation}

## Auto Correlation and Auto Power Spectral Density

If $x$ and $y$ are the same signal, then so called auto correlation function (ACF) $\varphi_{xx}[\kappa]$ and (auto)-PSD $\Phi_{xx}(\mathrm{e}^{\mathrm{j}\Omega})$ are evaluated.
These are defined and linked as

\begin{equation}
\begin{split}
\varphi_{xx}[\kappa]&=\frac{1}{N}\sum\limits_{k=0}^{N-1} x[k+\kappa]\cdot x^*[k] = \frac{1}{N}\sum\limits_{k=0}^{N-1} x[k]\cdot x^*[k-\kappa]\\
\varphi_{xx}[\kappa]&=\text{IDTFT}\{\Phi_{xx}(\mathrm{e}^{\mathrm{j}\Omega})\}=\frac{1}{2\pi}\int\limits_{-\pi}^\pi\Phi_{xx}(\mathrm{e}^{\mathrm{j}\Omega})\cdot\mathrm{e}^{+\mathrm{j}\Omega\kappa}\mathrm{d}\Omega\\
\Phi_{xx}(\mathrm{e}^{\mathrm{j}\Omega})&=\text{DTFT}\{\varphi_{xx}[\kappa]\}=\sum\limits_{\kappa=-\infty}^\infty\varphi_{xx}[\kappa] \cdot \mathrm{e}^{-\mathrm{j}\Omega\kappa}
\end{split}
\end{equation}

The following link

\begin{equation}
\varphi_{xx}[\kappa]=\varphi^*_{xx}[-\kappa]
\end{equation}

with respect to time mirroring and complex-conjugate holds, since it is a special case of $\varphi_{xy}[\kappa]=\varphi^*_{yx}[-\kappa]$ for $x=y$.
For $x\in\mathbb{R}$, an additional important property can be deduced from

\begin{equation}
\varphi_{xx}[\kappa]\leq\varphi_{xx}[\kappa=0]=\text{E}\left\{x[k]\cdot x[k]\right\}=\sigma_x^2+\mu_x^2.
\end{equation}

This means, that evaluation of the ACF at $\kappa=0$ results in the quadratic mean, i.e. the mean power. For all other values of $\kappa$, the ACF can only be less or equal than the mean power.
Due to the IDTFT definition, also the relation

\begin{equation}
\varphi_{xx}[\kappa=0]=\frac{1}{2\pi}\int\limits_{-\pi}^\pi\Phi_{xx}(\mathrm{e}^{\mathrm{j}\Omega})\mathrm{d}\Omega
\end{equation}

holds for the mean power, i.e. it is proportional to the area of the power spectral density.

## Reaction of an LTI System on Random Signals

The relation between an input signal $x[k]$, the impulse response $h[k]$ of an LTI system, and  output signal $y[k]$, as well as the relation between their DTFT spectra $X(\mathrm{e}^{\mathrm{j}\Omega})$, $H(\mathrm{e}^{\mathrm{j}\Omega})$ and $Y(\mathrm{e}^{\mathrm{j}\Omega})$ are the fundamentals of DSP:

\begin{equation}
y[k]=x[k]\ast h[k] \circ - \bullet Y(\mathrm{e}^{\mathrm{j}\Omega})=X(\mathrm{e}^{\mathrm{j}\Omega})\cdot H(\mathrm{e}^{\mathrm{j}\Omega}).
\end{equation}

Similarly, the following relations can be stated for an input signal, drawn from a stationary random process, acting on an LTI system:

\begin{equation}
\mathrm{I:}\quad\varphi_{yx}[\kappa]=\varphi_{xx}[\kappa]\ast h[\kappa] \circ - \bullet \Phi_{yx}(\mathrm{e}^{\mathrm{j}\Omega})=\Phi_{xx}(\mathrm{e}^{\mathrm{j}\Omega})\cdot H(\mathrm{e}^{\mathrm{j}\Omega}).
\end{equation}

Interchanging of indices allows to write

\begin{equation}
\varphi_{xy}[\kappa]=\varphi_{xx}[\kappa]\ast h^*[-\kappa] \circ - \bullet \Phi_{xy}(\mathrm{e}^{\mathrm{j}\Omega})=\Phi_{xx}(\mathrm{e}^{\mathrm{j}\Omega})\cdot H^*(\mathrm{e}^{\mathrm{j}\Omega}).
\end{equation}

The general rule for PSDs with $a$, $b$ either $x$ or $y$ \& $A$, $B$ either $X$ or $Y$

$$\Phi_{ab}(\mathrm{e}^{\mathrm{j}\Omega})= A(\mathrm{e}^{\mathrm{j}\Omega}) \cdot B^*(\mathrm{e}^{\mathrm{j}\Omega}),$$

and the fact that

\begin{equation}
\Phi_{aa}(\mathrm{e}^{\mathrm{j}\Omega})= A(\mathrm{e}^{\mathrm{j}\Omega}) \cdot A^*(\mathrm{e}^{\mathrm{j}\Omega}) = |A(\mathrm{e}^{\mathrm{j}\Omega})|^2,
\end{equation}

and the further useful relations (actually from these two everything else can be deduced)

\begin{equation}
\begin{split}
\mathrm{II:}\quad\Phi_{ya}(\mathrm{e}^{\mathrm{j}\Omega})=&\Phi_{xa}(\mathrm{e}^{\mathrm{j}\Omega})H(\mathrm{e}^{\mathrm{j}\Omega})\\
\mathrm{III:}\quad\Phi_{ay}(\mathrm{e}^{\mathrm{j}\Omega})=&\Phi_{ax}(\mathrm{e}^{\mathrm{j}\Omega})H^*(\mathrm{e}^{\mathrm{j}\Omega}).
\end{split}
\end{equation}

are important to show the links between the functions.

The Wiener-Lee theorem links the input, the system response and output ACFs as

\begin{equation}
\mathrm{IV:}\quad\varphi_{yy}[\kappa]=\varphi_{xx}[\kappa]\ast \varphi_{hh}[\kappa] \circ - \bullet \Phi_{yy}(\mathrm{e}^{\mathrm{j}\Omega})=\Phi_{xx}(\mathrm{e}^{\mathrm{j}\Omega})\cdot |H(\mathrm{e}^{\mathrm{j}\Omega})|^2,
\end{equation}

which can be also written as

\begin{equation}
\varphi_{yy}[\kappa] = \varphi_{xx}[\kappa] \ast \varphi_{hh}[\kappa] \circ - \bullet |Y(\mathrm{e}^{\mathrm{j}\Omega})|^2=|X(\mathrm{e}^{\mathrm{j}\Omega})|^2\cdot |H(\mathrm{e}^{\mathrm{j}\Omega})|^2.
\end{equation}

The auto correlation function
\begin{equation}
\varphi_{hh}[\kappa]=h[\kappa]\ast h^*[-\kappa]
\end{equation}
is called *filter ACF*.

Equations $\mathrm{I}$, $\mathrm{II}$, $\mathrm{III}$ and $\mathrm{IV}$ describe the fundamentals of random signal processing with LTI systems.

## Packages / Functions

We import the required packages first.
Furthermore, we define a cross-correlation function `my_xcorr2(x,y)` that conveniently returns the ACF with time shifts/lags $\kappa$ and specific scaling. This is not available out of the box with the `numpy` or `signal` packages.

```python
# most common used packages for DSP, have a look into other scipy submodules
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
from scipy import signal

def my_xcorr2(x, y, scaleopt='none'):
    N = len(x)
    M = len(y)
    kappa = np.arange(0, N+M-1) - (M-1)
    ccf = signal.correlate(x, y, mode='full', method='auto')
    if N == M:
        if scaleopt == 'none' or scaleopt == 'raw':
            ccf /= 1
        elif scaleopt == 'biased' or scaleopt == 'bias':
            ccf /= N
        elif scaleopt == 'unbiased' or scaleopt == 'unbias':
            ccf /= (N - np.abs(kappa))
        elif scaleopt == 'coeff' or scaleopt == 'normalized':
            ccf /= np.sqrt(np.sum(x**2) * np.sum(y**2))
        else:
            print('scaleopt unknown: we leave output unnormalized')
    return kappa, ccf
```

# Exercise 1: Auto Correlation for Sine Signal

## Task

Let us shortly consider continuous-time signals, since analytical calculus is a little bit simpler to show the intended key facts.

Calculate the continuous-time auto-correlation function

\begin{equation}
\phi_{xx}(\tau)=\lim\limits_{T\rightarrow\infty}\frac{1}{2T}\int\limits_{-T}^T x(t+\tau)\,x^*(t)\,\mathrm{d}t
\end{equation}

for the power signal (i.e. a signal with a finite mean power)

\begin{equation}
x(t)=A\cdot\sin(\omega\,t+\phi)
\end{equation}

with $A<\infty, \in \mathbb{R}$ and $\phi\in\mathbb{R}$ and $\omega\neq 0$.

## Solution

In the first step, just insert $x(t)$ into the ACF definition
\begin{equation}
\phi_{xx}(\tau)=\lim\limits_{T\rightarrow\infty}\frac{1}{2T}\int\limits_{-T}^T A^2\sin(\omega\,(t+\tau)+\phi)\,\sin(\omega\,t+\phi)\,\mathrm{d}t.
\end{equation}

With $\sin(\alpha)\sin(\beta)=\frac{1}{2}\left(\cos(\alpha-\beta)-\cos(\alpha+\beta)\right)$ we can derive

\begin{equation}
\begin{split}
\phi_{xx}(\tau)&=\lim_{T\to\infty}\frac{A^2}{4T}\int\limits_{-T}^T\cos(\omega\tau) - \cos(2\omega t+\omega\tau+2\phi)\,\mathrm{d}t\\
&=\lim\limits_{T\to\infty}\left(\frac{A^2}{4T}\cos(\omega\tau)\bigg[t\bigg]_{-T}^T-\underbrace{\frac{A^2}{4T}\int\limits_{-T}^T\cos(2\omega t+\omega\tau+2\phi)\mathrm{d}t}_{\to0\quad\text{for}\quad T\to\infty}\right) \\
&=\lim_{T\to\infty}\frac{A^2}{4T}\cos(\omega\tau)\cdot(T+T)\\
&=\lim_{T\to\infty}\frac{A^2}{2}\cos(\omega\tau)\\
&=\frac{A^2}{2}\cos(\omega\tau)
\end{split}
\end{equation}

The integral that tends $\to0\quad\text{for}\quad T\to\infty$

\begin{equation}
-\lim_{T\to\infty} \frac{A^2}{4 T} \int\limits_{-T}^{T} \cos(2 \omega t + a)\,\mathrm{d}t
\end{equation}

with $a = \omega \tau + 2 \phi$ can be evaluated as

\begin{equation}
\begin{split}
&-\frac{A^2}{4} \lim_{T\to\infty} \frac{1}{T} \int\limits_{-T}^{T} \cos(2 \omega t + a)\,\mathrm{d}t=
-\frac{A^2}{4} \lim_{T\to\infty} \frac{1}{T} \left[\frac{\sin(2 \omega t + a)}{2 \omega} \right]_{-T}^T\\
&-\frac{A^2}{4 \cdot 2 \omega} \lim_{T\to\infty} \frac{1}{T} \left(\sin(2 \omega T + a) - \sin(-2 \omega T + a)\right).
\end{split}
\end{equation}

We observe that $-2 \leq \sin(2 \omega T + a) - \sin(-2 \omega T + a) \leq +2$, and thus bounded, which means
that $\lim_{T\to\infty} \frac{1}{T} (\dots) \to0$ and the pre-factors not longer matter.

## Interpretation

We derived that

\begin{equation}
x(t)=A\cdot\sin(\omega\,t+\phi)
\end{equation}

has the auto correlation function

\begin{equation}
\phi_{xx}(\tau) = \frac{A^2}{2}\cos(\omega\tau).
\end{equation}

Let us plot this with an example of quasi-continuous signals, i.e. with very high sampling of the sine.

```python
w = 2*np.pi*1  # f = 1 Hz
tend = 4  # theoretically the sine has infinite duration, here only 4s to plot
N = tend * 2**8  # 256 samples per period -> very sufficient oversampling
t = np.arange(N)/N * tend - tend//2  # here: [-2s...+2s)
A = np.sqrt(8)  # this choice matches an ACF amplitude of 4
phi = np.pi/4  # arbitrary choice, note that ACF is not affected by phase
x = A*np.sin(w*t + phi)  # create the sine signal

# estimate the sampling frequency from the time interval between two samples
fs = 1 / (t[1]-t[0])
print('fs = {0:f} Hz'.format(fs))
plt.figure(figsize=(9, 3))
plt.plot(t, x)
plt.xlim(-2, 2)
plt.ylim(-3, 3)
plt.xlabel(r'$t$ / s')
plt.ylabel(r'$x(t)$')
plt.title('sine signal')
plt.grid(True)
```

```python
kappa, phixx = my_xcorr2(x, x, 'unbiased')

plt.figure(figsize=(9, 3))
plt.plot(kappa/fs, phixx)
plt.xlim(-2, 2)
plt.ylim(-4, 4)
plt.xlabel(r'$\tau$ / s')
plt.ylabel(r'ACF $\phi_{xx}(\tau)$')
plt.title('auto correlation function of sine signal')
plt.grid(True)
```

- We see that maximum **positive** auto correlation is obtained when sine signals exactly overlay each other. This is the case for $\tau= m \frac{2\pi}{\omega}$, $m\in\mathbb{Z}$, i.e. multiples of the sine signal's period.

- The maximum **negative** auto correlation is obtained when the two signals exhibit exact inverted polarity.
For a single sine signal this is the case for a phase shift of 180 deg or given in time by half of the period, thus at $\tau = m \frac{\pi}{\omega}$, $|m|\in 1,3,5,7,\dots$.

- Correlation is zero when the phase shift exhibits $\pm$90 deg, i.e. a quarter of a period.
Then the autocorrelation is evaluated for a sine and a cosine function (the sign of the cosine depends on the actual shift) of same frequency.
Since these two signals are known to be orthogonal, the zero correlation is not surprising, cf. the [Fourier series](http://mathworld.wolfram.com/FourierSeries.html).

# Exercise 2: CCFs

Let us deal with some correlations of discrete-time signals that can be conveniently discussed in terms of shifting and correlation properties.

Since we perform the same evaluation repeatedly, it is meaningful to program a little function that calculates and plots the correlation. See below:

```python
def my_ccf_plot(x, y, scaleopt):
    kappa, ccf = my_xcorr2(x, y, scaleopt)
    plt.figure(figsize=(9, 9))

    plt.subplot(3, 1, 1)
    plt.plot(x, label='1st signal')
    plt.xlim(0, 2000)
    plt.ylim(-4, 4)
    plt.xlabel(r'$k$')
    plt.ylabel(r'$x[k]$')
    plt.legend()
    plt.grid(True)

    plt.subplot(3, 1, 2)
    plt.plot(y, label='2nd signal')
    plt.xlim(0, 2000)
    plt.ylim(-4, 4)
    plt.xlabel(r'$k$')
    plt.ylabel(r'$y[k]$')
    plt.legend()
    plt.grid(True)

    plt.subplot(3, 1, 3)
    plt.plot(kappa, ccf, label='correlation(1st,2nd)')
    plt.xlim(-1000, 1000)
    plt.ylim(-1, 1)
    plt.xlabel(r'$\kappa$')
    plt.ylabel(r'$\phi_{xy}[\kappa]$')
    plt.legend()
    plt.grid(True)
```

Here are the first subtasks:

a) Generate a sine signal $x[k]=\sin(\Omega_1k)$ for $0\leq k\leq N-1$ with $N=5000$ and $\Omega_1=5\cdot\frac{2\pi}{N}$.

b) Generate a sine signal $y[k]=\sin\left(\Omega_1k-\frac{\pi}{2}\right)$ with the same parameters as in a).

c) Calculate the unbiased estimator of the CCF between $x$ and $y$.

d) Plot $x[k]$, $y[k]$ and $\varphi_{xy}[\kappa]$.

e) Interpret $\varphi_{xy}[\kappa]$ with respect to the extreme cases when $x$ is shifted by $\pm\frac{\pi}{2}$ and not at all.

```python
N = 5000
k = np.arange(N)
Omega1 = 5 * 2*np.pi/N
Omega2 = 20 * 2*np.pi/N  # needed later
```

```python
# a)
x = np.sin(Omega1*k)
# b)
y = np.sin(Omega1*k - np.pi/2)
# c) and d)
my_ccf_plot(x, y, scaleopt='unbiased')
```

Interpretation for subtask e)

Recall

\begin{equation}
\varphi_{xy}[\kappa]=\frac{1}{N}\sum\limits_{k=0}^{N-1} x[k+\kappa]\cdot y^*[k] = \frac{1}{N}\sum\limits_{k=0}^{N-1} x[k]\cdot y^*[k-\kappa].
\end{equation}

Then we can state:
- Period is 1000 samples, so phase shift of $\frac{\pi}{2}$ corresponds to 250 samples
- For $\kappa=0$ we have a positive sine signal and a negative cosine signal to be correlated. Since these two sequences are orthogonal, the CCF is zero here.
- ACF is also periodic, it is a negative sine signal actually
- If $x$ is shifted by $\kappa=-250$, i.e. to the right: signals lie on top of each other $\to$ maximum value in the ACF.
- If $y$ is shifted by $\kappa=-250$, i.e.  to the left: signals lie on top of each other $\to$ maximum value in the ACF.
- If $x$ is shifted by $\kappa=+250$, i.e.  to the left: signals are exactly out of phase $\to$ minimum value in the ACF.
- If $y$ is shifted by $\kappa=+250$, i.e.  to the right: signals are exactly out of phase $\to$ minimum value in the ACF.

Subtask f) Repeat the previous steps a) to e) for $y[k]=x[k]$, i.e. generate the ACF $\varphi_{xx}[\kappa]$.

```python
# f)
x = np.sin(Omega1*k)
y = x
my_ccf_plot(x, y, scaleopt='unbiased')
```

We see the same ACF characteristics as in exercise 1.
Full period is 1000 samples, in-phase case, positive maximum correlation.
Half period is 500 samples, out-of-phase case, negative maximum correlation.
Quarter period is 250 samples, orthogonal case, zero correlation.

Subtask g) Repeat the previous steps  a) to e) for the signals $x[k]=\sin(\Omega_1k)$ with $\Omega_1=5\cdot\frac{2\pi}{N}$ and $y[k]=\sin(\Omega_2k)$ with $\Omega_2=20\cdot\frac{2\pi}{N}$.

```python
# g)
x = np.sin(Omega1*k)
y = np.sin(Omega2*k)
my_ccf_plot(x, y, scaleopt='unbiased')
```

The CCF is zero for all $\kappa$.
The signals correspond to harmonics in a Fourier series (when considering time-continuous signals) or a DFT.
In our case we could think of dealing with a DFT of length $N$, where we created sine signals for the
5th and 20th DFT eigenfrequency.

Recall: It is a key feature of DFT that the $\mathrm{e}^{\mathrm{j}\frac{2\pi}{N}k\mu}$ DFT eigensignals are orthogonal, i.e.

\begin{equation}
\frac{1}{N}\sum_{k}^{N-1}\mathrm{e}^{\mathrm{j}\frac{2\pi}{N}k\mu_1} \cdot\mathrm{e}^{\mathrm{j}\frac{2\pi}{N}k\mu_2}=
\begin{cases}
1\quad\mathrm{for}\quad\mu_1=\mu_2\\
0\quad\mathrm{for}\quad\mu_1\neq\mu_2
\end{cases}
\end{equation}

For the case of $\mu_1\neq\mu_2$, a time shift by arbitrary $\kappa$ does not change the result of zero correlation:

\begin{equation}
\frac{1}{N}\sum_{k}^{N-1}\mathrm{e}^{\mathrm{j}\frac{2\pi}{N}(k+\kappa)\mu_1} \cdot\mathrm{e}^{\mathrm{j}\frac{2\pi}{N}k\mu_2}=0\quad\mathrm{if}\quad\mu_1\neq\mu_2.
\end{equation}

For the case of $\mu_1=\mu_2$, we can observe that for arbitrary $\kappa$

\begin{equation}
\bigg|\frac{1}{N}\sum_{k}^{N-1}\mathrm{e}^{\mathrm{j}\frac{2\pi}{N}(k+\kappa)\mu_1} \cdot\mathrm{e}^{\mathrm{j}\frac{2\pi}{N}k\mu_2}\bigg|=1\quad\mathrm{if}\quad\mu_1=\mu_2
\end{equation}
holds.

The case plotted above with the two sine signals is just a special case of evaluating only the imaginary part of $\mathrm{e}^{\mathrm{j}\frac{2\pi}{N}k\mu}$-signals for the CCF.

Subtask h) Generate a white noise signal of length $N$, whose amplitude values are drawn from the standard normal distribution (i.e. ideally $\mu_x=0, \sigma^2=1$, $\varphi_{xx}=\delta[\kappa]$, $\Phi_{xx}(\mathrm{e}^{\mathrm{j}\Omega})=1$ holds).

```python
# h)
np.random.seed(2)  # arbitrary choice
x = np.random.randn(N)
y = x
my_ccf_plot(x, y, scaleopt='biased')
```

We see that the ACF is a Dirac-like function, not ideally a Dirac since we are dealing with finite length sequences and thus only an estimator for the ACF based on the random sample function $x=y$.

We will deal with this Dirac property a lot in the subsequent exercises.

# Exercise 3: ACF of a Short Sequence

## Task

- Evaluate the ACF

\begin{equation}
\varphi_{xx}[\kappa]=\frac{1}{N}\sum\limits_{k=0}^{N-1} x[k+\kappa]\cdot x^*[k]
\end{equation}

of the energy signal

\begin{equation}
x[k] = \delta[k] + 2 \delta[k-1] -3 \delta[k-2].
\end{equation}

- Make a sketch of $x[k]$ and $\varphi_{xx}[\kappa]$.

- Calculate the signal energy from the ACF.

## Solution

We make sure that we get the equation right.

We have $N=3$.
A general time shift / lag by $\kappa$ can be written as

\begin{equation}
x[k+\kappa] = \delta[k+\kappa] + 2 \delta[k+\kappa-1] -3 \delta[k+\kappa-2].
\end{equation}

Then the sifting property of the Dirac can be used to find the actual values of the ACF as follows (note that we don't need the summation equation since the Dirac representation is already a sum).

For $\kappa=0$ we get

\begin{equation}
\varphi_{xx}[\kappa=0] = \frac{1}{3} (\delta[k+0] + 2 \delta[k+0-1] -3 \delta[k+0-2]) \cdot (\delta[k] + 2 \delta[k-1] -3 \delta[k-2]) = \frac{1}{3}(1^2 + 2^2 + (-3)^2) = \frac{14}{3} = 4.6667
\end{equation}

For $\kappa=-1$ we get

\begin{equation}
\varphi_{xx}[\kappa=-1] = \frac{1}{3} (\delta[k-1] + 2 \delta[k-1-1] -3 \delta[k-1-2]) \cdot (\delta[k] + 2 \delta[k-1] -3 \delta[k-2]) = \frac{1}{3}(1\cdot2 + 2\cdot(-3)) = \frac{-4}{3} = -1.333
\end{equation}

For $\kappa=-2$ we get

\begin{equation}
\varphi_{xx}[\kappa=-2] = \frac{1}{3} (\delta[k-2] + 2 \delta[k-2-1] -3 \delta[k-2-2]) \cdot (\delta[k] + 2 \delta[k-1] -3 \delta[k-2]) = \frac{1}{3}(1\cdot (-3)) = \frac{-3}{3} = -1
\end{equation}

Of course, the results are also obtained by drawing the two signals on a paper, shift one by a certain $\kappa$, multiply the values that overlap and sum them up (just as we learned for convolution but without the time reversal step).

From above we know that

\begin{equation}
\varphi_{xx}[\kappa]=\varphi^*_{xx}[-\kappa]
\end{equation}

and since $\varphi_{xx}[\kappa]\in\mathbb{R}$ we just deal with axial-symmetry with respect to $\kappa=0$

\begin{equation}
\varphi_{xx}[\kappa]=\varphi_{xx}[-\kappa].
\end{equation}

Thus,

\begin{equation}
\varphi_{xx}[\kappa=+2] = -1
\end{equation}

and

\begin{equation}
\varphi_{xx}[\kappa=+1] = -\frac{4}{3},
\end{equation}

which could be also be derived when using the manual calculus with the Dirac sifting that was deployed for $\kappa=0,-1,-2$.

For completeness we should state that

\begin{equation}
\varphi_{xx}[\kappa] = 0\quad\mathrm{for}\quad |\kappa|>2.
\end{equation}

Let us plot the signal $x[k]$ and $\varphi_{xx}[\kappa]$.

```python
x = (+1., +2., -3.)
kappa, acf = my_xcorr2(x, x, 'biased')  # use own function defined above

for i in acf:
    print('{:f}'.format(i), end=', ')

plt.figure(figsize=(9, 3))
plt.subplot(1, 2, 1)
plt.stem(x, basefmt='C0:')
plt.xticks(np.arange(0, 3))
plt.yticks(np.arange(-3, 3, 1))
plt.xlabel(r'$k$')
plt.ylabel(r'$x[k]$')
plt.title('signal')
plt.grid(True)

plt.subplot(1, 2, 2)
# note that we plot 3*ACF ! this gives simple integer results in the plot
plt.stem(kappa, acf*3, basefmt='C0:')
plt.xticks(np.arange(-2, 3))
plt.yticks(np.arange(-4, 16, 2))
plt.xlabel(r'$\kappa$')
plt.ylabel(r'$3 \cdot \phi_{xx}[\kappa]$')
plt.title('ACF, note the normalization by 3!')
plt.grid(True)
```

The signal energy is contained in
\begin{equation}
\varphi_{xx}[\kappa=0]=\text{E}\left\{x[k]\cdot x[k]\right\}=\sigma_x^2+\mu_x^2 = \frac{14}{3}.
\end{equation}
Trivial fact, but worth to recapitulate: this is always a positive number.

# Exercise 4: Auto Correlation Function of LTI System's Output

## Task

An LTI system with the DTFT transfer function

\begin{equation}
H(\mathrm{e}^{\mathrm{j}\Omega})=\begin{cases}
	1, & |\Omega|<\Omega_c\\
	0, & \Omega_c\leq|\Omega|\leq\pi
\end{cases}
\end{equation}

for $\Omega_c > 0$ is excited by white noise $x[k]$ that exhibits the input (auto)-PSD $\Phi_{xx}(\mathrm{e}^{\mathrm{j}\Omega}) = \Phi_0$.
We assume that $x[k]$ is drawn from a stationary, ergodic random process.

- Make a sketch of the amplitude response of the transfer function and characterize the system characteristics.
- Calculate the ACF, the linear mean and the variance of the output signal $y[k]$.

## Solution

```python
Omegac = np.pi/4  # arbitrary choice, must be <pi
N = 2**10
Omega = np.arange(N) * 2*np.pi/N - np.pi  # [-pi...pi)
H = np.ones(N)
H[Omegac < np.abs(Omega)] = 0
plt.figure(figsize=(9, 3))
plt.plot(Omega, H)
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$H(\mathrm{e}^{\mathrm{j}\Omega})$')
plt.title('ideal lowpass')
plt.xticks([-np.pi, -Omegac, 0, +Omegac, +np.pi],
           [r'-$\pi$', r'-$\Omega_c$', '0', r'+$\Omega_c$', r'+$\pi$'])
plt.xlim(-np.pi, +np.pi)
plt.ylim(-0.1, 1.1)
plt.grid(True)
```

- ACF of output:

Make use of the Wiener-Lee theorem in DTFT domain

\begin{equation}
\begin{split}
\Phi_{yy}(\mathrm{e}^{\mathrm{j}\Omega})&=\Phi_{xx}(\mathrm{e}^{\mathrm{j}\Omega})\cdot|H(\mathrm{e}^{\mathrm{j}\Omega})|^2\\
&=\Phi_{0}\cdot|H(\mathrm{e}^{\mathrm{j}\Omega})|^2.
\end{split}
\end{equation}

The squared transfer function of LTI system is simply

\begin{equation}
|H(\mathrm{e}^{\mathrm{j}\Omega})|^2=\begin{cases}
	1, & |\Omega|<\Omega_c \\
	0, & \Omega_c\leq|\Omega|\leq\pi.
\end{cases}
\end{equation}

Make use of inverse DTFT

\begin{equation}
\begin{split}
\varphi_{yy}[\kappa]& \circ - \bullet \Phi_{yy}(\mathrm{e}^{\mathrm{j}\Omega})\\
\varphi_{yy}[\kappa]&=\frac{1}{2\pi}\int\limits_{-\pi}^\pi\Phi_{yy}(\mathrm{e}^{\mathrm{j}\Omega})\mathrm{e}^{\mathrm{j}\Omega\kappa}\mathrm{d}\Omega.
\end{split}
\end{equation}

Insert the given relations

\begin{equation}
\begin{split}
\varphi_{yy}[\kappa]&=\frac{\Phi_0}{2\pi}\int\limits_{-\Omega_c}^{\Omega_c}\mathrm{e}^{\mathrm{j}\Omega\kappa}\mathrm{d}\Omega \\
&=\frac{\Phi_0}{2\pi}\left[\frac{1}{\mathrm{j}\kappa}\mathrm{e}^{\mathrm{j}\Omega\kappa}\right]_{-\Omega_c}^{\Omega_c} \\
&=\frac{\Phi_0}{2\pi\mathrm{j}\kappa}\left(\mathrm{e}^{\mathrm{j}\Omega_c\kappa}-\mathrm{e}^{-\mathrm{j}\Omega_c\kappa}\right).
\end{split}
\end{equation}

With $\sin(x)=\frac{1}{2\mathrm{j}}\left(\mathrm{e}^{\mathrm{j} x}-\mathrm{e}^{-\mathrm{j} x}\right)$ we can simplify and obtain our final result

\begin{equation}
\varphi_{yy}[\kappa]=\Phi_0\frac{\sin(\Omega_c\kappa)}{\kappa\pi}
\end{equation}

Rewritten to make a weighted sinc-function more obvious
\begin{equation}
\varphi_{yy}[\kappa]=\Phi_0\,\frac{\Omega_c}{\pi}\,\frac{\sin(\Omega_c\kappa)}{\Omega_c\kappa}
\end{equation}

- Linear mean of output:

For the output signal it is fair to assume that

\begin{equation}
E\{y[k] y[k+\kappa]\} = E\{y[k]\}\cdot E\{y[k+\kappa]\}\quad\mathrm{for}\quad \kappa\to \infty,
\end{equation}

meaning that the signal $y$ for $k$ and the time shift $\kappa$ are not correlated.
Since we applied a white noise that should exhibit this characteristics, the output signal fulfills this assumption as well, since it was only shaped by the ideal lowpass filter.
Furthermore the process is assumed to be stationary, which means that the mean $\mu_y[k]$ is not time-dependent.
Thus,
\begin{equation}
E\{y[k]\} = E\{y[k+\kappa]\} = \mu_y,
\end{equation}
which results in
\begin{equation}
\varphi_{yy}[\kappa] = E\{y[k]\}\cdot E\{y[k+\kappa]\} = \mu_y^2\quad\mathrm{for}\quad \kappa\to \infty.
\end{equation}
Applying results in the final result
\begin{equation}
\mu_y^2 = \varphi_{yy}[\kappa\to \infty]=\lim_{\kappa\to\infty}\Phi_0\frac{\sin(\Omega_c\kappa)}{\kappa\pi} = \frac{\mathrm{bounded}}{\infty} = 0.
\end{equation}

Since $\mu_x = 0$ by definition of our input signal, we should expect this $\mu_y=0$ for the output signal as well, since the low pass filter as LTI system will not create a DC offset.

- Variance of output:

Generally, the ACF evaluated at $\kappa=0$ contains the quadratic mean, this is $\varphi_{yy}[\kappa=0]=\sigma_y^2+\mu_y^2$. Evaluating this yields

\begin{equation}
\begin{split}
\varphi_{yy}[\kappa=0]&=\sigma_y^2+\mu_y^2\\
\sigma^2_y&=\varphi_{yy}[0]-\mu^2_y \\
&=\varphi_{yy}[0]-0 \\
&=\lim\limits_{\kappa\to0}\Phi_0\frac{\sin(\Omega_c\kappa)}{\kappa\pi} \\
&=\lim\limits_{\kappa\to0}\Phi_0\frac{\Omega_c\cos(\Omega_c\kappa)}{\pi} \\
&=\Phi_0\frac{\Omega_c}{\pi}
\end{split}
\end{equation}

## Interpretation

- The Dirac ACF of input $\varphi_{xx}[\kappa]=\Phi_0 \delta[\kappa]$ is shaped to the output ACF $\varphi_{yy}[\kappa] = \Phi_0\frac{\sin(\Omega_c\kappa)}{\kappa\pi}$, i.e. a sinc-like shape.
This is typical for lowpass filter characteristics.

- The squared mean of input $\mu_x^2 = \varphi_{xx}[\kappa\to\infty]=\lim_{\kappa\to\infty}\Phi_0 \delta[\kappa]=0$ tells us that the input signal is mean free (DC free).
Since, the (time-independent) means of input/output signal are linked by $\mu_y = \mu_x \cdot H(\mathrm{e}^{\mathrm{j}\Omega})\big|_{\Omega=0}$, we obtain $\mu_y=0$ here. Note that in this example, this result is independent of the specific $H(\mathrm{e}^{\mathrm{j}\Omega})\big|_{\Omega=0}$.

- If $\Omega_c=\pi$, i.e. no low-pass characteristic at all, the complete power is transmitted through the system.
For this case, verify that $h[k] = \delta[k]$, which furthermore implies $y = x$, $\varphi_{yy} = \varphi_{xx}$.

- For $\Omega_c<\pi$, i.e. certain low-pass characteristic, only the fraction $\frac{\Omega_c}{\pi}$ of the whole power $\Phi_0$ is transmitted.

- Plot $\varphi_{yy}[\kappa] = \Phi_0\frac{\sin(\Omega_c\kappa)}{\kappa\pi}$ by yourself for different $0\leq \Omega_c\leq \pi$. What should be expected and is this reflected by the plots?

# Exercise 5: Transfer Function of LTI System, Output PSD

This exercise is very similar to the exercise 4, just that the LTI system is slightly more complicated and we discuss the spectral domain in more detail. The key principles remain the same, since we are again dealing with white noise input signal.

## Task

Calculate the PSD $\Phi_{yy}(\mathrm{e}^{\mathrm{j}\Omega})$ of the output signal $y[k]=x[k]\ast h[k]$. Make a sketch of the PSD $\Phi_{yy}(\Omega)$ for $0\leq\Omega<2\pi$.

- $x[k]$ is a stationary random input signal for which the (ideal) auto-correlation function is given as

\begin{equation}
\varphi_{xx}[\kappa]=\sigma_x^2\cdot\delta[\kappa]
\end{equation}

- the impulse response describing the LTI system is given as

\begin{equation}
h[k]=\left(\frac{3}{4}\right)^k\cdot\epsilon[k]
\end{equation}

In signals & systems we learned how to discuss the system characteristics in terms of impulse/step response, frequency response, pole/zero plot, bode plot.
We will need some of this stuff here again.

## Solution

We make use of the Wiener-Lee theorem, introduce the filter-ACF and find the links in spectral (DTFT) domain:

\begin{equation}
\varphi_{yy}[\kappa]=\varphi_{hh}[\kappa]\ast\varphi_{xx}[\kappa]=(h[\kappa]\ast h^*[-\kappa])\ast\varphi_{xx}[\kappa] \circ - \bullet \Phi_{yy}(\mathrm{e}^{\mathrm{j}\Omega})=|H(\mathrm{e}^{\mathrm{j}\Omega})|^2\cdot\Phi_{xx}(\mathrm{e}^{\mathrm{j}\Omega}).
\end{equation}

We see, that for solving $\Phi_{yy}(\mathrm{e}^{\mathrm{j}\Omega})$, we need $\Phi_{xx}(\mathrm{e}^{\mathrm{j}\Omega})$ and $|H(\mathrm{e}^{\mathrm{j}\Omega})|^2$.

The (auto)-PSD for input signal $x$ is with $\delta[k] \circ - \bullet 1$ derived as

\begin{equation*}
\Phi_{xx}(\mathrm{e}^{\mathrm{j}\Omega})=\text{DTFT}\{\varphi_{xx}[\kappa]\}=\text{DTFT}\{\sigma_x^2\cdot\delta[\kappa]\}=\sigma_x^2.
\end{equation*}

The squared magnitude of the transfer function can be derived by the DTFT of the impulse response $h[k]$.
The well known relation (have a look at signals and systems lecture / exercise)

\begin{equation}
\text{DTFT}\{a^k\cdot\epsilon[k]\}=\frac{1}{1-a\cdot\mathrm{e}^{-\mathrm{j}\Omega}} \quad\text{for}\quad|a|<1
\end{equation}

can be used to find that

\begin{equation}
h[k] \circ - \bullet H(\mathrm{e}^{\mathrm{j}\Omega}) = \frac{1}{1-\frac{3}{4}\cdot\mathrm{e}^{-\mathrm{j}\Omega}}.
\end{equation}

Thus,

\begin{equation}
\begin{split}
\Phi_{yy}(\mathrm{e}^{\mathrm{j}\Omega})=&|H(\mathrm{e}^{\mathrm{j}\Omega})|^2\cdot\Phi_{xx}(\mathrm{e}^{\mathrm{j}\Omega})\\
\Phi_{yy}(\mathrm{e}^{\mathrm{j}\Omega})=&\left|\frac{1}{1-\frac{3}{4}\cdot\mathrm{e}^{-\mathrm{j}\Omega}}\right|^2 \cdot \sigma_x^2.
\end{split}
\end{equation}

Taking the usual mathematical transformations for complex numbers, we get

\begin{equation}
\begin{split}
\Phi_{yy}(\mathrm{e}^{\mathrm{j}\Omega})&=\frac{1}{\left|1-\frac{3}{4}\cos(\Omega)+\mathrm{j}\frac{3}{4}\sin(\Omega)\right|^2}\cdot\sigma_x^2\\
&=\frac{\sigma_x^2}{\left(1-\frac{3}{4}\cos(\Omega)\right)^2+\left(\frac{3}{4}\sin(\Omega)\right)^2}\\
&=\frac{\sigma_x^2}{1-2\cdot\frac{3}{4}\cos(\Omega)+\frac{9}{16}\cos^2(\Omega)+\frac{9}{16}\sin^2(\Omega)}\\
&=\frac{\sigma_x^2}{1-\frac{3}{2}\cos(\Omega)+\frac{9}{16}}\\
&=\frac{\sigma_x^2}{\frac{25}{16}-\frac{3}{2}\cos(\Omega)}
\end{split}
\end{equation}
which brings the final result
\begin{equation}
\begin{split}
\Phi_{yy}(\mathrm{e}^{\mathrm{j}\Omega})=\frac{2}{\frac{25}{8}-3\cos(\Omega)} \cdot \sigma_x^2.
\end{split}
\end{equation}

We should plot this normalized by $\sigma_x^2$, so in fact we plot $|H(\mathrm{e}^{\mathrm{j}\Omega})|^2$ then for our example.
This also gives us a nice way to validate our analytic calculus. Instead of the DTFT, we can describe the LTI system response in the $z$-domain, i.e. with its $z$-transform

\begin{equation}
h[k] \circ - \bullet H(z) = \frac{1}{1-\frac{3}{4} z^{-1}},
\end{equation}

where we recognize a recursive system with difference equation $a_0 y[k] = b_0 x[k] - a_1 y[k-1]$ using the coefficients $b_0=1$, $a_0=1$ and $a_1=-\frac{3}{4}$.
The `freqz()` function conveniently allows to evaluate the magnitude response of (non)-recursive systems, i.e. it evaluates the DTFT numerically with certain frequency resolution. Here we use $\Delta\Omega = \frac{2\pi}{256}$.

```python
N = 2**8
Omega = np.arange(N) * 2*np.pi/N
H2 = 2 / (25/8 - 3*np.cos(Omega))  # analytic
Omega, H_IIR = signal.freqz(b=(1), a=(1, -3/4), worN=Omega)  # numeric

plt.figure(figsize=(9, 3))
plt.plot(Omega/np.pi, H2, lw=3, label='analytic solution')
plt.plot(Omega/np.pi, np.abs(H_IIR)**2, label='IIR filter freqz()-solution')
plt.plot(Omega/np.pi, Omega*0+1,
         label=r'$\Phi_{xx}(\mathrm{e}^{\mathrm{j}\Omega})\,/\,\sigma_x^2$')
plt.xlabel(r'$\frac{\Omega}{\pi}$')
plt.ylabel(
    r'$\Phi_{yy}(\mathrm{e}^{\mathrm{j}\Omega})\,/\,\sigma_x^2 = |H(\mathrm{e}^{\mathrm{j}\Omega})|^2$')
plt.title('PSD of output')
plt.xlim(0, 2)
plt.xticks(np.arange(0, 20, 2)/10)
plt.legend()
plt.grid(True)
```

## Interpretation

- In the plot we see, that analytic and numeric solution of $|H(\mathrm{e}^{\mathrm{j}\Omega})|^2$ are the same, so our calculus is very likely correct if we assume that freqz() is correctly implemented.

- Furthermore in the plot, we sketched the $\Phi_{xx}(\mathrm{e}^{\mathrm{j}\Omega})$ normalized by $\sigma_x^2$, which yields a line, i.e. magnitude one for all frequencies.

- We see that the input PSD $\Phi_{xx}(\mathrm{e}^{\mathrm{j}\Omega})$, which is constant over frequency, is spectrally shaped by the LTI system towards the output PSD $\Phi_{yy}(\mathrm{e}^{\mathrm{j}\Omega})$ .

- The output PSD $\Phi_{yy}(\mathrm{e}^{\mathrm{j}\Omega})$ is thus not longer constant over frequency, but amplified at lower frequencies and attenuated at higher frequencies.
This filter exhibits certain lowpass characteristics with a DC gain of 12 dB, -3 dB cut frequency approximately at $\Omega=\frac{1}{10} \pi$ and unity gain (0 dB) at about $\frac{4}{10} \pi$.

# Exercise 6: Impulse Response Estimation with Random Signal in Time Domain

This exercise deals with estimating the impulse response of an LTI system. We make use of the same concepts from the two previous exercises.

## Task

In this programming task we want to elaborate how to identify the impulse response of an LTI system with random signal in the time domain.
If white noise is used as input signal, the task becomes very convenient in terms of required signal processing steps. Thus, we

- generate white noise signal $x[k]$ drawn from gaussian PDF
- create finite impulse response $h[k]$ of a simple LTI system, i.e. a lowpass (in practice this would be unknown)
- apply convolution $y[k] = x[k] \ast h[k]$
- estimate the impulse response $\hat{h}[k]$ based on the concept of correlation functions.

## Solution

### Generate White Noise

We aim at an ACF

$$\varphi_{xx}[\kappa]\approx \delta[\kappa]$$

and thus constant (Auto)-PSD for an input signal block $x[k]$ that was drawn
- from Gaussian PDF with zero mean and unit variance (i.e. standard normal distribution)
- with N=$2^{16}$ samples length

```python
np.random.seed(2)  # arbitrary choice
Nx = 2**16
k = np.arange(Nx)
x = np.random.randn(Nx)
kappa, phixx = my_xcorr2(x, x, 'biased')  # we use biased here, i.e. 1/N normalization
idx = np.where(kappa==0)[0][0]

plt.figure(figsize=(9, 3))
plt.subplot(1, 2, 1)
plt.plot(k, x)
plt.xlabel('$k$')
plt.ylabel('$x[k]$')
plt.title('input signal')
plt.grid(True)
plt.subplot(1, 2, 2)
plt.plot(kappa, phixx)
plt.xlim(-1000, +1000)
plt.xlabel('$\kappa$')
plt.ylabel('ACF $\phi_{xx}[\kappa]$')
plt.title('ACF of input signal, Dirac amplitude=%4.3f' % phixx[idx])
plt.grid(True)
```

### Create a Finite Impulse Response LTI System

- exemplary lowpass with FIR least squares design
- cut frequency 4.8 kHz
- sampling frequency 48 kHz
- 45 taps
- one tap weight is manipulated manually to break axial-symmetry on purpose

```python
fs = 48000  # sampling frequency in Hz
fc = 4800  # cut frequency in Hz
number_fir_coeff = 45  # FIR taps
h = signal.firls(numtaps=number_fir_coeff,  # example for demo
                 bands=(0, fc, fc+1, fs//2),
                 desired=(1, 1, 0, 0),
                 fs=fs)
Nh = h.size
k = np.arange(Nh)
# make the IR unsymmetric by arbitray choice for demonstration purpose
idx = 30
h[idx] = 0  # then FIR is not longer linear-phase, see the spike in the plot

print('h[0]={0:4.3f}, DC={1:4.3f} dB'.format(h[0], 20*np.log10(np.sum(h))))

N = 2**8
Omega = np.arange(0, N) * 2*np.pi/N
_, H = signal.freqz(b=h, a=1, worN=Omega)

plt.figure(figsize=(9, 6))
plt.subplot(2, 1, 1)
plt.stem(k, h, basefmt='C0:')
plt.plot(k, h, 'C0-', lw=0.5)
plt.xlabel(r'$k$')
plt.ylabel(r'$h[k]$')
plt.title(str(Nh)+'-coeff FIR, note the bad guy at k=%d destroying the symmetry of FIR' % idx)
plt.grid(True)
plt.subplot(2, 1, 2)
plt.semilogx(Omega / (2*np.pi) * fs, 20*np.log10(np.abs(H)))
plt.xlabel(r'$f$ / Hz')
plt.ylabel(r'$|H(\mathrm{e}^{\mathrm{j}\Omega})|$ / dB')
plt.xlim(1, fs//2)
plt.ylim(-40, 10)
plt.grid(True)
```

## Convolution

```python
y = np.convolve(x, h, mode='full')  # signal x through system h returns output y
```

## Estimate the Impulse Response by Cross Correlation

The (most simple, and in practice rather not longer used) method here is based on the fact that the ACF of the input signal is ideally a Dirac impulse.

In essence we deal with:

- CCF of $y[k]$ and $x[k]$ yields **either** $\varphi_{yx}$ **or** $\varphi_{xy}$ (note the index exchange) depending on the order of calling `my_xcorr2()`
- check $\varphi_{yx}[\kappa]=h[+\kappa]$
- check $\varphi_{xy}[\kappa]=h[-\kappa]$

More detailed, the relationships between CCF, ACF and IR are known from above

\begin{equation}
\begin{split}
\varphi_{yx}[\kappa]=&\varphi_{xx}[\kappa]\ast h[+\kappa]\\
\varphi_{xy}[\kappa]=&\varphi_{xx}[\kappa]\ast h^*[-\kappa].
\end{split}
\end{equation}

For $h\in\mathbb{R}$ this yields
\begin{equation}
\varphi_{xy}[\kappa]=\varphi_{xx}[\kappa]\ast h[-\kappa].
\end{equation}

For above assumption that the random input signal fulfills

$$\varphi_{xx}[\kappa]\approx \delta[\kappa]$$

we get
\begin{equation}
\begin{split}
\varphi_{yx}[\kappa]=\delta[\kappa]\ast h[+\kappa]\\
\varphi_{xy}[\kappa]=\delta[\kappa]\ast h[-\kappa]
\end{split}
\end{equation}

and since the Dirac impulse behaves neutral in convolution we have

\begin{equation}
\begin{split}
\varphi_{yx}[\kappa]=h[+\kappa]\\
\varphi_{xy}[\kappa]=h[-\kappa]
\end{split}
\end{equation}

as estimators for the unknown LTI system impulse response $h[k]$.

So, let us check the case $\varphi_{yx}[\kappa]=h[+\kappa]$ first:

```python
kappa, phiyx = my_xcorr2(y, x, 'biased')  # get cross correlation in order y,x

# find the index for kappa=0, the IR starts here
idx = np.where(kappa == 0)[0][0]
# cut out the IR, since we know the numtaps this is easy to decide here
h_est = phiyx[idx:idx+Nh] / len(y)
# get DTFT estimate of PSD
_, Phiyx = signal.freqz(b=h_est, a=1, worN=Omega)

plt.figure(figsize=(9, 6))
plt.subplot(2, 1, 1)
plt.stem(h, basefmt='C0:', label='True IR: h[k]')
plt.plot(kappa, phiyx / len(y), 'C1o-',
         label=r'IR Estimate: CCF $\phi_{yx}[\kappa]$')
plt.xlim(0, Nh-1)
plt.xlabel(r'sample index $\rightarrow$')
plt.ylabel(r'amplitude $\rightarrow$')
plt.title(str(Nh)+'-Coeff FIR')
plt.legend()
plt.grid(True)
plt.subplot(2, 1, 2)
plt.semilogx(Omega/2/np.pi*fs, 20*np.log10(np.abs(H)), lw=3,
             label=r'True TF: $|\mathrm{H}(\mathrm{e}^{\mathrm{j}\Omega})|$')
plt.semilogx(Omega/2/np.pi*fs, 20*np.log10(np.abs(Phiyx)),
             label='TF Estimate: $|\Phi_{yx}(\mathrm{e}^{\mathrm{j}\Omega})|$')
plt.xlabel(r'$f$ / Hz')
plt.ylabel(r'$\rightarrow$ dB')
plt.xlim(1, fs//2)
plt.ylim(-40, 10)
plt.legend()
plt.grid(True)
```

By interchanging the order for cross correlation, i.e. `my_xcorr2(x,y)`,

$$\varphi_{xy}[\kappa]=h[-\kappa]$$

we obtain the mirrored impulse response.
To conveniently see this, we implemented the `h[30] = 0` manipulation, to make slightly IR asymmetric.

```python
kappa, phixy = my_xcorr2(x, y, 'biased')  # get cross correlation x,y

plt.figure(figsize=(10, 3))
plt.plot(kappa, phixy/len(y), 'C3o-',
         label=r'IR Estimate: CCF $\phi_{xy}[\kappa]$')
plt.xlim(-(Nh-1), 0)
plt.xlabel(r'sample index $\rightarrow$')
plt.ylabel(r'amplitude $\rightarrow$')
plt.title('mirrored impulse response')
plt.legend()
plt.grid(True)
```

## Interpretation

The estimated impulse can be compared against the reference impulse response (we only can do this here, since we have built one for demonstration. In practice this response is of course unknown, and one needs to be sure about the chosen method to obtain a valid estimation.).
We can take the absolute error on the impulse response itself, here as

\begin{equation}
e_h[k] = \hat{h}[k] - h[k]
\end{equation}

Furthermore, it is of interest how the magnitude spectrum (and in many applications also the phase) of the transfer function differs. We can establish an error in terms of relative dB

\begin{equation}
e_H(\mathrm{e}^{\mathrm{j}\Omega}) = 20 \log_{10}|\hat{H}(\mathrm{e}^{\mathrm{j}\Omega})| - 20 \log_{10}|H(\mathrm{e}^{\mathrm{j}\Omega})|.
\end{equation}

Let us plot these two errors below.

```python
plt.figure(figsize=(9, 6))
plt.subplot(2, 1, 1)
plt.plot(h_est - h, 'o-')
plt.xlabel('k')
plt.ylabel(r'$e_h[k]$')
plt.grid(True)
plt.subplot(2, 1, 2)
plt.semilogx(Omega / (2*np.pi) * fs, 20 *
             np.log10(np.abs(Phiyx)) - 20*np.log10(np.abs(H)))
plt.xlabel('f / Hz')
plt.ylabel(r'$e_H(\mathrm{e}^{\mathrm{j}\Omega})$ / dB')
plt.xlim(1, fs//2)
plt.ylim(-2, 4)
plt.grid(True)
```

Without an application in mind, we cannot tell if these errors are sufficiently small or too large.

However, we can observe for that specific example, that the error $e_h$ somehow follows the shape of the impulse response, i.e. a sinc-like pattern.
Furthermore, we can state that for low frequencies there is a constant offset about 0.5 dB in the magnitude spectrum, slowly decreasing to zero dB in the region of the cut frequency. Above the cut frequency (except the strong peak) the error is somehow 'noisy' around 0 dB.

We might play around with the length of the input signal and the specific seed of the random number generator, to obtain different pictures.
For example instead of `N = 2**16` check what happens with `N = 2**8`.
It is very likely that the estimated impulse response is not precise enough for any application, since the deviations to the reference are too large.

Make yourself comfortable, why and how the parameter `N` changes the result and how the assumption of *ACF of $x$ being a Dirac* is affected.

# **Copyright**

The notebooks are provided as [Open Educational Resources](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebooks for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Frank Schultz, Digital Signal Processing - A Tutorial Featuring Computational Examples* with the URL https://github.com/spatialaudio/digital-signal-processing-exercises


# Realization of Recursive Filters

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Quantization of Variables and Operations

As for [non-recursive filters](../nonrecursive_filters/quantization_effects.ipynb#Quantization-Effects), the practical realization of recursive filters may suffer from the quantization of variables and algebraic operations. The effects of [coefficient quantization](quantization_of_coefficients.ipynb) were already discussed. This section takes a look at the quantization of variables.  We limit the investigations to the recursive part of a second-order section (SOS), since any recursive filter of order $N \geq 2$ can be [decomposed into SOSs](cascaded_structures.ipynb).

The computation of the output signal $y[k] = \mathcal{H}\{ x[k] \}$ by a difference equation involves a number of multiplications and additions. As discussed already for [non-recursive filters](../nonrecursive_filters/quantization_effects.ipynb#Quantization-of-Signals-and-Operations), multiplying two numbers in a binary representation (e.g. [two's complement](https://en.wikipedia.org/wiki/Two's_complement) or [floating point](https://en.wikipedia.org/wiki/Floating_point)) requires requantization of the result to keep the word length constant. The addition of two numbers may fall outside the maximum/minimum values of the representation and may suffer from clipping.

The resulting round-off and clipping errors depend on the number and sequence of algebraic operations. These depend on the structure used for implementation of the SOSs. For ease of illustration we limit our discussion to the [direct form I and II](direct_forms.ipynb). Similar insights can be achieved in a similar manner for other structures.

### Analysis of Round-Off Errors

Round-off errors are a consequence of reducing the word length after a multiplication. In order to investigate the influence of these errors on a recursive filter, the statistical model for [round-off errors in multipliers](../nonrecursive_filters/quantization_effects.ipynb#Model-for-round-off-errors-in-multipliers) as introduced for non-recursive filters is used. We furthermore neglect clipping.

The difference equation for the recursive part of a SOS realized in direct form I or II is given as

\begin{equation}
y[k] =  x[k] - a_1 \, y[k-1] - a_2 \, y[k-2]
\end{equation}

where $a_0 = 1$, $a_1$ and $a_2$ denote the coefficients of the recursive part. Introducing the requantization after the multipliers into the difference equation yields the output signal $y_Q[k]$

\begin{equation}
y_Q[k] = x[k] - \mathcal{Q} \{ a_{1} \, y[k-1] \} - \mathcal{Q} \{ a_{2} \, y[k-2] \}
\end{equation}

where $\mathcal{Q} \{ \cdot \}$ denotes the requantizer. Requantization is a non-linear process which results in a requantization error. If the value to be requantized is much larger that the quantization step $Q$, the average statistical properties of this error can be modeled as additive uncorrelated white noise. Introducing the error into above difference equation gives

\begin{equation}
y_Q[k] = x[k] - a_1 \, y[k-1] - e_1[k] - a_2 \, y[k-2] - e_2[k]
\end{equation}

where the two white noise sources $e_1[k]$ and $e_2[k]$ are assumed to be uncorrelated to each other. This difference equation can be split into a set of two difference equations

\begin{align}
y_Q[k] &= y[k] + e[k] \\
y[k] &=  x[k] - a_1 \, y[k-1] - a_2 \, y[k-2] \\
e[k] &= - e_1[k] - e_2[k] - a_1 \, e[k-1] - a_2 \, e[k-2]
\end{align}

The first difference equation computes the desired output signal $y[k]$ as a result of the input signal $x[k]$. The second one the additive error $e[k]$ due to requantization as a result of the requantization error $- (e_1[k] + e_2[k])$ injected into the recursive filter.
The power spectral density (PSD) $\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of the error $e[k]$ is then given as

\begin{equation}
\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = | H(\mathrm{e}^{\,\mathrm{j}\,\Omega})|^2 \cdot (\Phi_{e_1 e_1}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) + \Phi_{e_2 e_2}(\mathrm{e}^{\,\mathrm{j}\,\Omega}))
\end{equation}

According to the model for the requantization errors, their PSDs are given as $\Phi_{e_1 e_1}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \Phi_{e_2 e_2}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{Q^2}{12}$. Introducing this together with the transfer function of the SOS yields

\begin{equation}
\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \left| \frac{1}{1 + a_1 \, \mathrm{e}^{\,-\mathrm{j}\,\Omega} + a_2 \, \mathrm{e}^{\,-\mathrm{j}\,2\,\Omega}} \right|^2 \cdot \frac{Q^2}{6}
\end{equation}

#### Example - Round-off error of a SOS

The following example evaluates the error $e[k] = y_Q[k] - y[k]$ for a SOS which only consists of a recursive part. The desired system response $y[k]$ is computed numerically by floating point operations with double precision, $y_Q[k]$ is computed by applying a uniform midtread quantizer after the multiplications. The system is excited by uniformly distributed white noise. Besides the PSD $\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$, the signal-to-noise ratio (SNR) $10 \cdot \log_{10} \left( \frac{\sigma_y^2}{\sigma_e^2} \right)$ in dB of the filter is evaluated.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

N = 8192  # length of signals
w = 8  # wordlength for requantization of multiplications

def uniform_midtread_quantizer(x):
    # linear uniform quantization
    xQ = Q * np.floor(x/Q + 1/2)

    return xQ

def no_quantizer(x):

    return x

def sos_df1(x, a, requantize=None):
    y = np.zeros(len(x)+2)  # initial value appended
    for k in range(len(x)):
        y[k] = x[k] - requantize(a[1]*y[k-1])  - requantize(a[2]*y[k-2])

    return y[0:-2]

# cofficients of the SOS
p = 0.90*np.array([np.exp(1j*np.pi/3), np.exp(-1j*np.pi/3)])
a = np.poly(p)
# quantization step
Q = 1/(2**(w-1))

# compute input signal
x = np.random.uniform(low=-1, high=1, size=N)
# compute output signals w and w/o requantization
yQ = sos_df1(x, a, requantize=uniform_midtread_quantizer)
y = sos_df1(x, a, requantize=no_quantizer)
# compute requantization error
e = yQ-y
# Signal-to-noise ratio
SNR = 10*np.log10(np.var(y)/np.var(e))
print('SNR due to requantization: %f dB'%SNR)

# estimate PSD of requantization error
nf, Pxx = sig.welch(e, window='hamming', nperseg=256, noverlap=128)
Pxx = .5*Pxx  # due to normalization in scipy.signal
Om = 2*np.pi*nf
# compute frequency response of system
w, H = sig.freqz([1,0,0], a)

# plot results
plt.figure(figsize=(10,4))
plt.plot(Om, Pxx/Q**2 * 12, 'b', label=r'$|\hat{\Phi}_{ee}(e^{j \Omega})|$')
plt.plot(w, np.abs(H)**2 * 2, 'g', label=r'$|H(e^{j \Omega})|^2$')
plt.title('Estimated PSD and transfer function of requantization noise')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$Q^2/12$')
plt.axis([0, np.pi, 0, 100])
plt.legend()
plt.grid();
```

### Small Limit Cycles

Besides the requantization noise, recursive filters may be subject to periodic oscillations present at the output. These undesired oscillations are termed *limit cycles*. Small limit cycles emerge from the additive round-off noise due to requantization after a multiplication. The feedback in a recursive filter leads to a feedback of the requantization noise. This may lead to a periodic output signal with an amplitude range of some quantization steps $Q$, even after the input signal is zero. The presence, amplitude and frequency of small limit cycles depends on the location of poles and the structure of the filter. A detailed treatment of this phenomenon is beyond the scope of this notebook and can be found in the literature.

#### Example - Small limit cycles of a SOS

The following example illustrates small limit cycles for the system investigated in the previous example. The input signal is uniformly distributed white noise till time-index $k=256$ and zero for the remainder.

```python
# compute input signal
x = np.random.uniform(low=-1, high=1, size=256)
x = np.concatenate((x, np.zeros(1024)))
# compute output signal
yQ = sos_df1(x, a, requantize=uniform_midtread_quantizer)

# plot results
np.seterr(divide='ignore')
plt.figure(figsize=(10, 3))
plt.plot(20*np.log10(np.abs(yQ)))
plt.title('Level of output signal')
plt.xlabel(r'$k$')
plt.ylabel(r'$|y_Q[k]|$ in dB')
plt.grid()

plt.figure(figsize=(10, 3))
k = np.arange(1000, 1050)
plt.stem(k, yQ[k]/Q)
plt.title('Output signal for zero input')
plt.xlabel(r'$k$')
plt.ylabel(r'$y_Q[k] / Q$ ')
plt.axis([k[0], k[-1], -3, 3])
plt.grid();
```

**Exercise**

* Estimate the period of the small limit cycles. How is it related to the poles of the system?
* What amplitude range is spanned?

Solution: The period of the small limit cycles can be estimated from the second illustration as $P = 6$. The normalized frequency of a harmonic exponential signal with the same periodicity is given as $\Omega_0 = \frac{2 \pi}{P} = \frac{\pi}{3}$. The poles of the system can be extracted from the code of the first example as $z_{\infty 0,1} = 0.9 \cdot e^{\pm j \frac{\pi}{3}}$. The periodicity of the small limit cycles is hence linked to the normalized frequency of the poles. The amplitude range spanned by the small limit cycles is $\pm 2 Q$.

### Large Limit Cycles

Large limit cycles are periodic oscillations of a recursive filter due to overflows in the multiplications/additions. As for small limit cycles, large limit cycles may be present even after the input signal is zero. Their level is typically in the range of the minimum/maximum value of the requantizer. Large limit cycles should therefore be avoided in a practical implementation. The presence of large limit cycles depends on the scaling of input signal and coefficients, as well as the strategy used to cope for clipping. Amongst others, they can be avoided by proper scaling of the coefficients to prevent overflow. Again, a detailed treatment of this phenomenon is beyond the scope of this notebook and can be found in the literature.

#### Example - Large limit cycles of a SOS

The following example illustrates large limit cycles for the system investigated in the first example. In order to trigger large limit cycles, the coefficients of the filter have been doubled. The input signal is uniformly distributed white noise till time-index $k=256$ and zero for the remainder.

```python
def uniform_midtread_quantizer(x, xmin=1):
    # limiter
    x = np.copy(x)
    if x <= -xmin:
        x = -1
    if x > xmin - Q:
        x = 1 - Q
    # linear uniform quantization
    xQ = Q * np.floor(x/Q + 1/2)

    return xQ

# compute input signal
x = np.random.uniform(low=-1, high=1, size=256)
x = np.concatenate((x, np.zeros(1024)))
# compute output signal
yQ = sos_df1(x, 2*a, requantize=uniform_midtread_quantizer)

# plot results
plt.figure(figsize=(10, 3))
plt.plot(20*np.log10(np.abs(yQ)))
plt.title('Level of output signal')
plt.xlabel(r'$k$')
plt.ylabel(r'$|y_Q[k]|$ in dB')
plt.grid()

plt.figure(figsize=(10, 3))
k = np.arange(1000, 1050)
plt.stem(k, yQ[k])
plt.title('Output signal for zero input')
plt.xlabel(r'$k$')
plt.ylabel(r'$y_Q[k]$ ')
plt.grid();
```

**Exercise**

* Determine the period of the large limit cycles. How is it related to the poles of the system?

Solution: The period of the large limit cycles can be estimated from the second illustration as $P = 6$. The normalized frequency of a harmonic exponential signal with the same periodicity is given as $\Omega_0 = \frac{2 \pi}{P} = \frac{\pi}{3}$. The poles of the system can be extracted from the code of the first example as $z_{\infty 0,1} = 0.9 \cdot e^{\pm j \frac{\pi}{3}}$. The periodicity of the large limit cycles is hence linked to the normalized frequency of the poles.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2018*.


```
# default_exp block_diagrams
```

```
#hide
%load_ext autoreload
%autoreload 2
```

```
#export
import numpy as np
import matplotlib.pyplot as plt
```

```
#hide
%matplotlib inline
```

# Block diagrams

This notebook will talk about:
- Basic components of block diagrams
- How to compose block diagrams together
- Block diagrams and system properties (e.g., Stability, etc)

------------------

## Block Diagrams

- One of the control system engineer’s favorite tools for organizing, communicating, simulating and solving problems
- Standard representation of interconnected systems and subsystems using Transfer Functions
- Makes it easier to identify inputs, outputs and dynamic systems
- Block diagrams show us the interrelationship of systems and how signals flow between them.

   - variables or signals: arrows ($\rightarrow$)
   - systems: blocks with a transfer function, and arrows going in (inputs) and out (outputs)
   - summation node: circle with arrows going in (with the appropriate signs) and an arrow going out
   - branching points: represents two or more variables are replicas of the variable before the branching point

- Always useful during the analysis.

For example:

## Block Diagram Albegra

- Block diagrams can be quite complex, and when that happens it is useful to know how to calculate the transfer function between a specific input and a specific output.

- For this, it is better to start with some easy cases: **Cascaded, Parallel and Feedback Loop.**

- More complex cases are handled recursively applying these rules. Note that, given that we are restricting the analysis to linear systems, the impact of each input on a specific output is independent of other inputs that can be present (they can all be set to 0).

### Cascaded Blocks

This can be easily calculated using algebric manipulation:

$$Y(s)=G_2(s)U(s)$$
$$U(s)=G_1(s)E(s)$$

which means:

$$Y(s)=G_2(s)G_1(s)E(s) \Rightarrow G(s) = G_1(s)G_2(s)$$

### Parallel

This can be easily calculated using algebric manipulation:

$$Y(s)=G_1(s)R(s) + G_2(s)R(s) + G_3(s)R(s) = (G_1(s) + G_2(s) + G_3(s))R(s)$$

which means:

$$ G(s) = G_1(s) + G_2(s) + G_3(s)$$

Note the presence of:
- _take off points_: allow an unaltered signal to go along multiple paths.
- _summing junctions_: sum or subtract signals

### Feedback Loop

So, how do we do this one?

$$Y(s)=G_1(s)*E(s)$$
$$E(s)=R(s)-Y_m(s)$$
$$Y_m(s)=G_2(s)Y(s)$$

or

$$Y(s)=G_1(s)*(R(s)-Y_m(s)) \rightarrow Y(s)=G_1(s)*(R(s)-G_2(s)Y(s)) $$

and finally:

$$Y(s)+G_1(s)G_2(s)Y(s) = G_1(s)*R(s) $$

$$ G(s) = \frac{G_1(s)}{1 + G_1(s)G_2(s)} $$

## Zero-Pole Cancellations

- When we compose systems together we expect the order of the system being equal to the sum of the order of each individual system.

- However, when we compose systems together and calculate the transfer function it might happen that the resulting function has less poles.
   - In this case, we have cancellations that correspond to "hidden" parts of the system.

Let's consider an example:

where

\begin{equation}
G_1(s)=\frac{s+1}{s+2}\;\; \text{and}\;\; G_2(s)=\frac{1}{s}
\end{equation}

We reduce the feedback loop first:

$$G_3(s)=\frac{1/s}{1+1/s} = \frac{1/s}{\frac{s+1}{s}} = \frac{1}{s+1}$$

Then the serie:
$$G(s) = \frac{s+1}{s+2} \frac{1}{s+1} = \frac{1}{s+2} $$

This connection between two systems determined an "hidden" part, which is not visible in the input/output relationship and the system behaves like a first-order one.

### Observability and Controllobality of Interconnected Systems

- Controllability measures the ability of a particular actuator configuration to control all the states of the system;

- Observability measures the ability of the particular sensor configuration to supply all the information necessary to estimate all the states of the system.

#### Cascaded Blocks

$$G(s) = G_1(s)G_2(s) = \frac{N_1(s)}{D_1(s)}\frac{N_2(s)}{D_2(s)}$$

- If $N_1(s)$ and $D_2(s)$ have roots in common, then $G(s)$ is not controllable (zero-pole cancellation)

- If $N_2(s)$ and $D_1(s)$ have roots in common, then $G(s)$ is not observable (pole-zero cancellation)

#### Parallel Blocks

$$G(s) = G_1(s) + G_2(s) = \frac{N_1(s)}{D_1(s)} + \frac{N_2(s)}{D_2(s)} = \frac{N_1(s)D_2(s) + N_2(s)D_1(s)}{D_1(s)D_2(s)}$$

- Cancellations can happen only when $D_1(s)$ and $D_2(s)$ have common roots (you can bring them out together in the numerator and then cancel them out with those in the denumerator). When this happens, we have a **non-observable and non-controllable** part of the system.

#### Feedback Loop

$$G(s) = \frac{G_1(s)}{1 + G_1(s)G_2(s)} =\frac{\frac{N_1(s)}{D_1(s)}}{1 + \frac{N_1(s)}{D_1(s)}\frac{N_2(s)}{D_2(s)}} = \frac{N_1(s)D_2(s)}{D_1(s)D_2(s) + N_1(s)N_2(s)}$$

- We have the equivalent of a serie interconnection: $G_1(s)G_2(s)$, same rules apply
- Cancellations occur when $N_1(s)$ and $D_2(s)$ have common factors (you can bring them out together in the numerator and then cancel them out with those in the denumerator):
   - When zero of $G_1(s)$ cancels a pole of $G_2(s)$ then we have a **non-observable and non-controllable** part of the system.
   - When pole of $G_1(s)$ cancels a zero of $G_2(s)$ then we still have a **fully observable and fully controllable** system. This is because the denominator of $G(s)$ does not lose its degree.

Example:

- zero of $G_1(s)$ cancels a pole of $G_2(s)$
$$G_1(s)=\frac{s+1}{s+2},\;\; G_2(s)=\frac{s+3}{s+1}$$

$$\Rightarrow G(s)= \frac{\frac{s+1}{s+2}}{1+\frac{s+3}{s+2}} = \frac{s+3}{1+(s+3)}$$

- pole of $G_1(s)$ cancels a zero of $G_2(s)$
$$G_1(s)=\frac{s+1}{s+2},\;\; G_2(s)=\frac{s+2}{s+3}$$

$$\Rightarrow G(s)= \frac{\frac{s+1}{s+2}}{1+\frac{s+1}{s+3}} = \frac{\frac{(s+1)(s+3)}{s+2}}{(s+3)+(s+1)} = \frac{(s+1)(s+3)}{(s+2)((s+3)+(s+1))}$$

### Stability of Interconnected Systems

Let's now analyse the stability of a system which is obtained through a composition of subsystems.

Question:
- Is the asymptotic stability of each indidual subsystem necessary/sufficient to guarantee the stability of the entire system?

Assumption:
-  Each subsystem are represented by rational functions with numerator and denominator prime between them.

#### Cascaded Blocks

$$G(s) = G_1(s)G_2(s) = \frac{N_1(s)}{D_1(s)}\frac{N_2(s)}{D_2(s)}$$

**No zero-pole cancellations**
- The denominator of $G(s)$ is the product of each denominator, and the poles of $G(s)$ are the union of the poles of each subsystem.
- If and only if, the poles of $G_1(s)$ and $G_2(s)$ have Re $<0$, then $G(s)$ is asymptotically stable.

**Zero-pole/pole-zero cancellations**
- In this case, the overall system has an hidden part corresponding to the zero-pole or pole-zero cancellation. In this case, _if the cancellation is of a pole with Re $\ge0$ then the hidden part is not stable and the entire system is not stable_.
- We call these types of cancellations, **critical**.
- Note that this is not visible directly from the transfer function!

#### Parallel Blocks

$$G(s) = G_1(s) + G_2(s) = \frac{N_1(s)}{D_1(s)} + \frac{N_2(s)}{D_2(s)} = \frac{N_1(s)D_2(s) + N_2(s)D_1(s)}{D_1(s)D_2(s)}$$

The analysis is similar to the previous case (serie).

If there are no cancellations, the poles of $G(s)$ are the product of the poles of $G_1(s)$ and of $G_2(s)$.
- $G(s)$ is asymptotically stable, if and only if, the poles of $G_1(s)$ and $G_2(s)$ have Re $<0$.

If there are cancellations, then the overall system has an hidden part corresponding to the zero-pole or pole-zero cancellation. If the cancellation is of a pole with Re $>=0$ then the hidden part is not stable and the entire system is not stable.

Again, this is not visible analysing the final T.F.

#### Feedback Loop

$$G(s) = \frac{G_1(s)}{1 + G_1(s)G_2(s)} =\frac{\frac{N_1(s)}{D_1(s)}}{1 + \frac{N_1(s)}{D_1(s)}\frac{N_2(s)}{D_2(s)}} = \frac{N_1(s)D_2(s)}{D_1(s)D_2(s) + N_1(s)N_2(s)}$$

- In this case, the poles of the entire system are not only dependednt on the poles of the two subsytems (the numerators are also playing a role).

If we do not have cancellations, then the poles of the systems are the roots of the equation:

$$D_1(s)D_2(s) + N_1(s)N_2(s) = 0$$

_this is called characteristic equation of the feedback system._

We can also write the equation above as:

$$ 1 + \frac{N_1(s)D_2(s)}{D_1(s)D_2(s)} = 0$$

The feedback system is asymptotically stable if and only if the roots of the previous equation are with Re $<0$, but this does not depend on the stability property of the individual systems.

**Critical cancellations**

- Note that when we calculate $\frac{N_1(s)D_2(s)}{D_1(s)D_2(s)}$ we can have _critical cancellations_ and these would correspond to hidden parts of the system (Note that we are assuming to use $y$ or $y_m$ as output variables of the system).
- As before, if these critical cancellations are of poles with Re $\ge0$, these hidden parts are not asymptotically stable and neither is the full system.
- This is not visible from the analysis of the T.F.
- We have to verify if there are _critical_ cancellations before we analyse the stability of the feedback system.

Since the poles of the feedback system are, in general, different from the poles of the individual subsystems, we can use this connection to effectively modify the dynamics of the system, placing the poles where we need them. For example, it is possible to stabilise a non-stable system, or obtain desired performance.

Note that the opposite is also true: we can have a non-stable system even if we the individual subsystems are stable.

## Equivalent block diagrams

## BIBO Stability

- A system is BIBO stable (Bounded Input, Bounded Output) if for each limited input, there is a limited output
- For linear systems: BIBO stability if and only if poles of the transfer functions have Re $<0$
- **Caution! The poles of the T.F. are only those that are controllable and observable!**
- BIBO stability only depends on the forced response of the system.

# Example: Car driving uphill

We can model a car moving uphill using a linear model (small angle approximation) as:

\begin{equation}
A=\frac{df}{dx}=
\begin{bmatrix}
0 & 1\\
0 & -\frac{\beta}{m}
\end{bmatrix}
\end{equation}

\begin{equation}
B=\frac{df}{du}=
\begin{bmatrix}
0 & 0\\
\frac{\gamma}{m} & -g
\end{bmatrix}
\end{equation}

and

$$\dot{\mathbf{x}}=A\mathbf{x}+B\mathbf{u}$$
$$y=[0 \; 1]\mathbf{x}$$

Note that:
- If $\theta = 0$ and $u=0$ (no slope, and no input)
- then the equilibrium is $\forall x_1$, $x_2=0$ (the position does not matter, speed is 0).
- this system has two inputs: the pedal and gravity

We can calculate the Matrix Transfer Function as:

$$G(s)=C(sI-A)^{-1}Bu$$

\begin{equation}
(sI-A) =
\begin{bmatrix}
s & -1\\
0 & s+\frac{\beta}{m}
\end{bmatrix}
\end{equation}

\begin{equation}
(sI-A)^{-1} = \frac{1}{s(s+ \frac{\beta}{m})}
\begin{bmatrix}
s+ \frac{\beta}{m} & 1\\
0 & s
\end{bmatrix}
\end{equation}

\begin{equation}
G(s) = \frac{1}{s(s+ \frac{\beta}{m})}
\begin{bmatrix}
s \frac{\gamma}{m} & -sg
\end{bmatrix} =
\begin{bmatrix}
\frac{\frac{\gamma}{m}}{s+\frac{\beta}{m}} & \frac{-g}{s+\frac{\beta}{m}}
\end{bmatrix}
\end{equation}

If we want to know the T.F. from the pedal to the velocity:

$$G(s)=\frac{\frac{\gamma}{m}}{s+\frac{\beta}{m}}$$

What happens when we apply a step onto the pedal?

Let's study the transfer function and include the input:

$$Y(s) = G(s)U(s)=\frac{\frac{\gamma}{m}}{s+\frac{\beta}{m}} \frac{1}{s} = \frac{\alpha_1}{s+\frac{\beta}{m}} \frac{\alpha_2}{s}$$

where $$\alpha_1=-\frac{\gamma}{\beta}, \; \alpha_2=\frac{\gamma}{\beta}$$

and the inverse transform is:

$$ y(t) = \alpha_1 e^{-\frac{\beta}{m}t} + \alpha_2 1(t)$$

We can implement this in Python:

Let's define the step function first

```
from classical_control_theory.intro_to_control_theory import step

# def step(t, step_time=0):
#     """Heaviside step function"""
#     return 1 * (t >= step_time)
```

And now we can plot the response of the system $y(t)$ to the step input:

```
# We decide the parameters of the car
m = 1
beta = 1
gamma = 1

# We set the simulation parameters
t = np.linspace(0, 15)

# Now we can calculate the output of the system
alpha_1 = -gamma/beta
alpha_2 = gamma/beta

y = alpha_1*np.exp(-beta/m*t) + alpha_2*step(t)
```

And we can plot it:

```
plt.plot(t, y, linewidth=4)
plt.xlabel('time (s)')
plt.ylabel('speed (m/s)')
plt.grid()
```

Essentially we predict that the car will increase its speed to try and match our step input.

But what does the actual car do?

We implemented a car already in notebook `02_Intro_to_control_theory`, but here it is again for convenience:

```
class Car:
    _g = 9.8 # Gravity

    def __init__(self, x0, params):
        self._x_1 = x0[0] # position (along the road)
        self._x_2 = x0[1] # velocity (along the road)
        self._m, self._alpha, self._beta, self._gamma = params

    def step(self, dt, u, theta):
        self.theta = theta
        self._x_1 = self._x_1 + dt*self._x_2
        self._x_2 = self._x_2 + dt*(-self._alpha/self._m*abs(self._x_2)*self._x_2 - \
                                    self._beta/self._m*self._x_2 + self._gamma/self._m*u - \
                                    Car._g*np.sin(theta))

    def speedometer(self):
        v = self._x_2
        return (v,)

    # Utility function to simplify plotting
    def sensor_i(self):
        # Rotation matrix to get back to the main frame.
        R = np.array(((np.cos(theta), -np.sin(theta)), (np.sin(theta), np.cos(theta))))
        x_i, y_i = R.dot(np.array([[self._x_1],[0]]))
        v = self._x_2
        return (x_i, y_i, v)
```

We can now run it:

```
# We define the parameters of the car
m = 1
alpha = 1
beta = 1
gamma = 1
params = (m, alpha, beta, gamma)

# Let's define its initial conditions
x_0 = (0,0) # position and speed
theta = np.radians(0) # no slope

# Define the input: step function
u = 1 # input (e.g. 0 or 1)

# Create the car
car = Car(x_0, params)

# run it!
t0, tf, dt = 0, 10, 0.1

#rotation to get back to the inertial frame..
c, s = np.cos(theta), np.sin(theta)
R = np.array(((c, -s), (s, c)))

position = []
velocity = []
time = []
for t in np.arange(t0, tf, dt):
    car.step(dt, u, theta)
    v = car.speedometer() # this is in the along road frame
    velocity.append(v)
    time.append(t)

```

```
plt.plot(np.arange(t0, tf, dt), velocity, linewidth=3)
plt.xlabel('time (s)')
plt.ylabel('speed (m/s)')
plt.grid()
```

Hmm, the speed has a step response but it does not go to 1...

The reason is that the actual car does behave differently from our linear model. Let's try another car:

```
#export
class SimplerCar:
    _g = 9.8

    def __init__(self, x0, params):
        self._x_1 = x0[0] # position (along the road)
        self._x_2 = x0[1] # velocity
        self._m, self._alpha, self._beta, self._gamma = params

    def step(self, dt, u, theta):
        self.theta = theta
        self._x_1 = self._x_1 + dt*self._x_2
        self._x_2 = self._x_2 + dt*(- self._beta/self._m*self._x_2 + \
                                    self._gamma/self._m*u - Car._g*np.sin(theta))

    def speedometer(self):
        v = self._x_2
        return (v,)
```

And now we can run it again,

```
# We define the parameters of the car
m = 1
alpha = 1
beta = 1
gamma = 1
params = (m, alpha, beta, gamma)

# Let's define its initial conditions
x_0 = (0,0) # position and speed
theta = np.radians(0) # no slope

# Define the input: step function
u = 1 # input (e.g. 0 or 1)

# Create the car
car = SimplerCar(x_0, params)

# run it!
t0, tf, dt = 0, 10, 0.1

#rotation to get back to the inertial frame..
c, s = np.cos(theta), np.sin(theta)
R = np.array(((c, -s), (s, c)))

position = []
velocity = []
time = []
for t in np.arange(t0, tf, dt):
    car.step(dt, u, theta)
    v = car.speedometer() # this is in the along road frame
    velocity.append(v)
    time.append(t)

```

```
plt.plot(time, velocity, linewidth=3)
plt.xlabel('time (s)')
plt.ylabel('speed (m/s)')
plt.grid()
```

Alright! Our linear model represents this second car better.

------------------------------------

## Simplifying block diagrams

Let's now apply the block diagram algebra to some examples:

### Example 1:

Let's suppose that we have a classic negative feedback:

$B$ and $C$ are in parallel and we can combine them into a single block:

We can now reduce the feedback loop:

And lastly we can combine the two blocks in series:

### Creating Unity Feedback

- Sometimes when we manipulate block diagrams we do not want to remove blocks to simplify the diagram, but we would like to restructure the diagram to have it in a specific form.

- For example we would like to make the following equivalent:

- We need to determine an $R(s)$ value that makes the two systems equivalent.
- We need to write the transfer function for each system, set the two equations equal to each other, and then solving for $R(s)$.

$$
\frac{G(s)}{1+G(s)H(s)} = \frac{R(s)}{1+R(s)}
$$

$$ \Downarrow $$

$$
\frac{G(s) (1+R(s))}{1+G(s)H(s)} = R(s)
$$

$$ \Downarrow $$

$$
\frac{G(s)}{1+G(s)H(s)} = R(s) \bigg( 1- \frac{G(s)}{1+G(s)H(s)} \bigg)
$$

$$ \Downarrow $$

$$
R(s) = \frac{G(s)}{1+G(s)H(s)-G(s)}
$$

Note that we would not be able to easily use this method to go from a unity feedback system to a non-unity feedback system (there will be two unknown variables).

------------------------


```python
from __future__ import print_function
from dolfin import *
import numpy as np
import matplotlib.pyplot as plt
# %matplotlib notebook
%matplotlib inline
import mshr
```

The mesh provided by the class **mshr** and its method *generate_mesh* automatically produces periodic mesh. This can be justified by examining the nodes of boundaries and easily seen for the simple case of UnitSquareMesh. Therefore, we will use the class **mshr** for our mesh generation and do not need to rely on the **gmsh** at this point.

```python
# Create domain, mark subdomains and generate mesh
domain = mshr.Polygon([Point(0, 0), Point(1, 0.25), Point(1.25, 1.25), Point(0.25, 1)])
inclusion = mshr.Circle(Point(1.25/2, 1.25/2), 0.2)
domain.set_subdomain(1, inclusion)

mesh = mshr.generate_mesh(domain, 12)
subdomain_marker = MeshFunction('size_t', mesh, mesh.topology().dim(),
                               mesh.domains())

plt.figure(figsize=(8, 4), dpi = 80)
plt.subplot(1,2,1)
plot(mesh); plt.grid()
plt.subplot(1,2,2)
plot(subdomain_marker); plt.grid()
```

```python
a = z[0,:] = [10, 1]
print(z)
```

```python
np.allclose(np.array([0, 1]), np.array([1, 1]), 2)
```

```python
# classed used to define the periodic boundary map
class PeriodicBoundary(SubDomain):
    # Constructor
    def __ini__(self, vertices, TOL=DOLFIN_EPS_LARGE):
        # verticies stores the coordinates of the 4 unit cell corners
        super().__init__()
        self.TOL = TOL
        self.vertices = vertices
        # vectors generating periodicity
        self.a = np.zeros((2, 2))
        self.a[0,:] = vertices[1,:] - vertices[0,:]
        self.a[1,:] = vertices[3,:] - vertices[0,:]

        # vectors normal to vectors a[0,:] and a[1,:]
        self.n = np.zeros((2,2))
        self.n[0,:] = [ -self.a[0,1], self.a[0,0] ]
        self.n[1,:] = [ -self.a[1,1], self.a[1,0] ]

        assert np.dot(self.n[0,:], self.a[0,:]) < self.TOL
        assert np.dot(self.n[1,:], self.a[1,:]) < self.TOL

    # method "inside" overide the superclass method
    def inside(self, x, on_boundary):
        # return True if x is on the left or bottom boundaries
        # AND NOT on one of the bottom-right or top-left vertices
        top_left = np.allclose(x, self.vertices[3,:], self.TOL)
        bottom_right = np.allclose(x, self.vertices[1,:], self.TOL)
        bottom = np.dot(self.n[0,:], x - vertices[0,:]) < self.TOL

```

We now define the constitutive law for both phases. We employ the energy density for linear isotropic elasicity
\begin{equation}
    \psi = \frac{1}{2}\lambda \mathrm{trace}(\boldsymbol{\epsilon})^2 + \mu \mathrm{trace}(\boldsymbol{\epsilon}^2).
\end{equation}
The Lame parameters $\lambda$ and $\mu$ are computed in terms of the Young's modulus and Poisson ratio as follows:
\begin{equation}
    \lambda = \frac{E \nu}{(1 + \nu) (1 - 2\nu}, \quad \mu = \frac{E}{2(1 + \nu)}.
\end{equation}
Since both $\lambda$ and $\mu$

The stress tensor $\boldsymbol{\sigma}$ is derived as
\begin{equation}
    \boldsymbol{\sigma} = \lambda \mathrm{trace}(\boldsymbol{\epsilon})\mathbf{I} + 2\mu \boldsymbol{\epsilon}
\end{equation}
The tangent stiffness, which is linear in this case, is given by
\begin{equation}
    C_{ijkl} = \lambda \delta_{ij}\delta_{kl} + \mu(\delta_{il}\delta_{jk} + \delta_{ik}\delta_{jl})
\end{equation}
Due to the symmetries of the $\boldsymbol{\delta}$, the stiffness is (1) majorly symmetric: $C_{ijkl} = C_{klij}$ and (2) minorly symmetric: $C_{ijkl} = C_{jikl} = C_{ijlk}$.

```python
Em = 50e3
num = 0.2
Er = 210e3
nur = 0.3
material_parameters = [(Em, num), (Er, nur)]
nphases = len(material_parameters)
def eps(v):
    return sym(grad(v))
def sigma(v, i, Eps):
    E, nu = material_parameters[i]
    lmbda = E*nu/(1+nu)/(1-2*nu)
    mu = E/2/(1+nu)
    return lmbda*tr(eps(v) + Eps)*Identity(2) + 2*mu*(eps(v)+Eps)
```

## Variational formulation

The previous problem is very similar to a standard linear elasticity problem, except for the periodicity constraint which has now been included in the FunctionSpace definition and for the presence of an eigenstrain term $\boldsymbol{E}$. It can easily be shown that the variational formulation of the previous problem reads as: Find $\boldsymbol{v}\in V$ such that:

$$\begin{equation}
F(\boldsymbol{v},\widehat{\boldsymbol{v}}) = \int_{\mathcal{A}} (\boldsymbol{E}+\nabla^s\boldsymbol{v}):\mathbb{C}(\boldsymbol{y}):\nabla^s\widehat{\boldsymbol{v}}\text{ d} \Omega = 0 \quad \forall \widehat{\boldsymbol{v}}\in V
\end{equation}$$

The above problem is not well-posed because of the existence of rigid body translations. One way to circumvent this issue would be to fix one point but instead we will add an additional constraint of zero-average of the fluctuation field $v$ as is classically done in homogenization theory. This is done by considering an additional vectorial Lagrange multiplier $\lambda$ and considering the following variational problem (see the [pure Neumann boundary conditions FEniCS demo](https://fenicsproject.org/docs/dolfin/2019.1.0/python/demos/neumann-poisson/demo_neumann-poisson.py.html) for a similar formulation): Find $(\boldsymbol{v},\boldsymbol{\lambda})\in V\times \mathbb{R}^2$ such that:

$$\begin{equation}
\int_{\mathcal{A}} (\boldsymbol{E}+\nabla^s\boldsymbol{v}):\mathbb{C}(\boldsymbol{y}):\nabla^s\widehat{\boldsymbol{v}}\text{ d} \Omega + \int_{\mathcal{A}} \boldsymbol{\lambda}\cdot\widehat{\boldsymbol{v}} \text{ d} \Omega + \int_{\mathcal{A}} \widehat{\boldsymbol{\lambda}}\cdot\boldsymbol{v} \text{ d} \Omega = 0 \quad \forall (\widehat{\boldsymbol{v}}, \widehat{\boldsymbol{\lambda}})\in V\times\mathbb{R}^2
\end{equation}$$

Which can be summarized as:

$$\begin{equation}
a(\boldsymbol{v},\widehat{\boldsymbol{v}}) + b(\boldsymbol{\lambda},\widehat{\boldsymbol{v}}) + b(\widehat{\boldsymbol{\lambda}}, \boldsymbol{v}) = L(\widehat{\boldsymbol{v}}) \quad \forall (\widehat{\boldsymbol{v}}, \widehat{\boldsymbol{\lambda}})\in V\times\mathbb{R}^2
\end{equation}$$

This readily translates into the following FEniCS code:

```python
Ve = VectorElement("CG", mesh.ufl_cell(), 2)
Re = VectorElement("R", mesh.ufl_cell(), 0)
W = FunctionSpace(mesh, MixedElement([Ve, Re]), constrained_domain=PeriodicBoundary(vertices, tolerance=1e-10))
V = FunctionSpace(mesh, Ve)

v_,lamb_ = TestFunctions(W)
dv, dlamb = TrialFunctions(W)
w = Function(W)
dx = Measure('dx')(subdomain_data=subdomains)
```

We have used a general implementation using a sum over the different phases for the functional `F`. We then used the `lhs` and `rhs` functions to respectively extract the corresponding bilinear $a$ and linear $L$ forms.

## Resolution

The resolution of the auxiliary problem is performed for elementary load cases consisting of uniaxial strain and pure shear sollicitations by assigning unit values of the correspnonding $E_{ij}$ components. For each load case, the average stress $\boldsymbol{\Sigma}$ is computed components by components and the macroscopic stiffness components $\mathbb{C}^{hom}$ are then printed.

```python
def macro_strain(i):
    """returns the macroscopic strain for the 3 elementary load cases"""
    Eps_Voigt = np.zeros((3,))
    Eps_Voigt[i] = 1
    return np.array([[Eps_Voigt[0], Eps_Voigt[2]/2.],
                    [Eps_Voigt[2]/2., Eps_Voigt[1]]])
def stress2Voigt(s):
    return as_vector([s[0,0], s[1,1], s[0,1]])

Chom = np.zeros((3, 3))
for (j, case) in enumerate(["Exx", "Eyy", "Exy"]):
    print("Solving {} case...".format(case))
    F = sum([inner(sigma(dv, i, Constant(macro_strain(j))), eps(v_))*dx(i) for i in range(nphases)]) + dot(lamb_,dv)*dx + dot(dlamb,v_)*dx
    a, L = lhs(F), rhs(F)
    solve(a == L, w, [])
    (v, lamb) = split(w)
    Sigma = np.zeros((3,))
    for k in range(3):
        Sigma[k] = assemble(sum([stress2Voigt(sigma(v, i, Eps))[k]*dx(i) for i in range(nphases)]))/vol
    Chom[j, :] = Sigma

print(np.array_str(Chom, precision=2))
```

```python
for (j, case) in enumerate(["Exx", "Eyy", "Exy"]):
    print("Solving {} case...".format(case))
    print(j)
```

It can first be verified that the obtained macroscopic stiffness is indeed symmetric and that the corresponding behaviour is quasi-isotropic (up to the finite element discretization error). Indeed, if $\lambda^{hom} = \mathbb{C}_{xxyy}$ and $\mu^{hom} = \mathbb{C}_{xyxy}$ we have that $\mathbb{C}_{xxxx}\approx\mathbb{C}_{yyyy}\approx \mathbb{C}_{xxyy}+2\mathbb{C}_{xyxy} = \lambda^{hom}+2\mu^{hom}$.

> **Note:** The macroscopic stiffness is not exactly symmetric because we computed it from the average stress which is not stricly verifying local equilibrium on the unit cell due to the FE discretization. A truly symmetric version can be obtained from the computation of the bilinear form for a pair of solutions to the elementary load cases.

```python
lmbda_hom = Chom[0, 1]
mu_hom = Chom[2, 2]
print(Chom[0, 0], lmbda_hom + 2*mu_hom)
```

We thus deduce that $E^{hom} = \mu^{hom}\dfrac{3\lambda^{hom}+2\mu^{hom}}{\lambda^{hom}+\mu^{hom}}$ and $\nu^{hom} = \dfrac{\lambda^{hom}}{2(\lambda^{hom}+\mu^{hom})}$ that is:

```python
E_hom = mu_hom*(3*lmbda_hom + 2*mu_hom)/(lmbda_hom + mu_hom)
nu_hom = lmbda_hom/(lmbda_hom + mu_hom)/2
print("Apparent Young modulus:", E_hom)
print("Apparent Poisson ratio:", nu_hom)
```

```python
# plotting deformed unit cell with total displacement u = Eps*y + v
y = SpatialCoordinate(mesh)
plt.figure()
p = plot(0.5*(dot(Eps, y)+v), mode="displacement", title=case)
plt.colorbar(p)
plt.show()
```

```python

```


```python
import tensorflow as tf
#import tensorflow_quantum as tfq

import cirq
import sympy
import numpy as np
import seaborn as sns
import collections

# visualization tools
%matplotlib inline
import matplotlib.pyplot as plt
from cirq.contrib.svg import SVGCircuit
from PIL import Image
```

```python

```

```python

```

```python
def convert_to_circuit(image):
    values = np.ndarray.flatten(image)
    qubits = cirq.GridQubit.rect(4, 4)
    circuit = cirq.Circuit()
    for i, value in enumerate(values):
        if value:
            circuit.append(cirq.X(qubits[i]))
    return circuit
```

```python
image=np.random.randint(255,size=(2,2))
#image=np.ones(255,size=(2,2))
plt.imshow(image,cmap='gray',vmin=0,vmax=255)
plt.show()
plt.imsave('grayimg.jpeg',image,cmap='gray')
```

```python
FILENAME='grayimg.jpeg'
image2 = Image.open(FILENAME)
pixels=np.asarray(image2)
pixels=pixels.astype('float32')
pixels2=pixels/255.0
print(pixels2)
THRESHOLD = 0.5
image_new_bin=np.array(pixels2 > THRESHOLD, dtype=np.float32)
image_new_bin
image_circuit=convert_to_circuit(image_new_bin)
```

```python
pix_val = list(image2.getdata())
pix_val
pix_val_flat = [x for sets in pix_val for x in sets]
pix_val_flat
```

```python
pixel_mat = np.array(image2.getdata())
width = image2.size[0]

pixel_ind = np.where((pixel_mat[:, :3] ==pix_val_flat[0]).any(axis=1))[0]
coordinate = np.concatenate(
    [
        (pixel_ind % width).reshape(-1, 1),
        (pixel_ind // width).reshape(-1, 1),
    ],
    axis=1,
)
coord=coordinate[0]
#xprime=list(coordinate[[0],coordinate[1])
xprime=coord.tolist()
xprime
```

```python
SVGCircuit(image_circuit)
```

```python
"""Get qubits to use in the circuit for Grover's algorithm."""
# Number of qubits n.
nqubits = 2

# Get qubit registers.
qubits = cirq.LineQubit.range(nqubits)
ancilla = cirq.NamedQubit("Ancilla")
```

```python
def make_oracle(qubits, ancilla, xprime):
    """Implements the function {f(x) = 1 if x == x', f(x) = 0 if x != x'}."""
    # For x' = (1, 1), the oracle is just a Toffoli gate.
    # For a general x', we negate the zero bits and implement a Toffoli.

    # Negate zero bits, if necessary.
    yield (cirq.X(q) for (q, bit) in zip(qubits, xprime) if not bit)

    # Do the Toffoli.
    yield (cirq.TOFFOLI(qubits[0], qubits[1], ancilla))

    # Negate zero bits, if necessary.
    yield (cirq.X(q) for (q, bit) in zip(qubits, xprime) if not bit)
```

```python
def grover_iteration(qubits, ancilla, oracle):
    """Performs one round of the Grover iteration."""
    circuit = cirq.Circuit()

    # Create an equal superposition over input qubits.
    circuit.append(cirq.H.on_each(*qubits))

    # Put the output qubit in the |-⟩ state.
    circuit.append([cirq.X(ancilla), cirq.H(ancilla)])

    # Query the oracle.
    circuit.append(oracle)

    # Construct Grover operator.
    circuit.append(cirq.H.on_each(*qubits))
    circuit.append(cirq.X.on_each(*qubits))
    circuit.append(cirq.H.on(qubits[1]))
    circuit.append(cirq.CNOT(qubits[0], qubits[1]))
    circuit.append(cirq.H.on(qubits[1]))
    circuit.append(cirq.X.on_each(*qubits))
    circuit.append(cirq.H.on_each(*qubits))

    # Measure the input register.
    circuit.append(cirq.measure(*qubits, key="result"))

    return circuit
```

```python
"""Select a 'marked' bitstring x' at random."""
xprime = [np.random.randint(0, 1) for _ in range(nqubits)]
xprime=list(coordinate)
xprime=coord.tolist()
print(f"Marked bitstring: {xprime}")

```

```python
"""Create the circuit for Grover's algorithm."""
# Make oracle (black box)
oracle = make_oracle(qubits, ancilla, xprime)

# Embed the oracle into a quantum circuit implementing Grover's algorithm.
circuit = grover_iteration(qubits, ancilla, oracle)
print("Circuit for Grover's algorithm:")
print(circuit)
```

```python
"""Simulate the circuit for Grover's algorithm and check the output."""
# Helper function.
def bitstring(bits):
    return "".join(str(int(b)) for b in bits)

# Sample from the circuit a couple times.
simulator = cirq.Simulator()
result = simulator.run(circuit, repetitions=10)

# Look at the sampled bitstrings.
frequencies = result.histogram(key="result", fold_func=bitstring)
print('Sampled results:\n{}'.format(frequencies))

# Check if we actually found the secret value.
most_common_bitstring = frequencies.most_common(1)[0][0]
print("\nMost common bitstring: {}".format(most_common_bitstring))
print("Found a match? {}".format(most_common_bitstring == bitstring(xprime)))
```


Non Linear Regression Analysis

If the data shows a curvy trend, then linear regression will not produce very accurate results when compared to a non-linear regression because, as the name implies, linear regression presumes that the data is linear.
Let's learn about non linear regressions and apply an example on python. In this notebook, we fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014.

Importing required libraries

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Though Linear regression is very good to solve many problems, it cannot be used for all datasets. First recall how linear regression, could model a dataset. It models a linear relation between a dependent variable y and independent variable x. It had a simple equation, of degree 1, for example y = $2x$ + 3.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 2*(x) + 3
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
#plt.figure(figsize=(8,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

Non-linear regressions are a relationship between independent variables $x$ and a dependent variable $y$ which result in a non-linear function modeled data. Essentially any relationship that is not linear can be termed as non-linear, and is usually represented by the polynomial of $k$ degrees (maximum power of $x$).

$$ \ y = a x^3 + b x^2 + c x + d \ $$

Non-linear functions can have elements like exponentials, logarithms, fractions, and others. For example: $$ y = \log(x)$$

Or even, more complicated such as :
$$ y = \log(a x^3 + b x^2 + c x + d)$$

Let's take a look at a cubic function's graph.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 1*(x**3) + 1*(x**2) + 1*x + 3
y_noise = 20 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

As you can see, this function has $x^3$ and $x^2$ as independent variables. Also, the graphic of this function is not a straight line over the 2D plane. So this is a non-linear function.

Some other types of non-linear functions are:

### Quadratic

$$ Y = X^2 $$

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

y = np.power(x,2)
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Exponential

An exponential function with base c is defined by $$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number. The base, c, is constant and the exponent, x, is a variable.

```python
X = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

Y= np.exp(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Logarithmic

The response $y$ is a results of applying logarithmic map from input $x$'s to output variable $y$. It is one of the simplest form of __log()__: i.e. $$ y = \log(x)$$

Please consider that instead of $x$, we can use $X$, which can be polynomial representation of the $x$'s. In general form it would be written as
\begin{equation}
y = \log(X)
\end{equation}

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = np.log(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Sigmoidal/Logistic

$$ Y = a + \frac{b}{1+ c^{(X-d)}}$$

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = 1-4/(1+np.power(3, X-2))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

# Non-Linear Regression example

For an example, we're going to try and fit a non-linear model to the datapoints corresponding to China's GDP from 1960 to 2014. We download a dataset with two columns, the first, a year between 1960 and 2014, the second, China's corresponding annual gross domestic income in US dollars for that year.

```python
import numpy as np
import pandas as pd

#downloading dataset
#!wget -nv -O china_gdp.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/china_gdp.csv

df = pd.read_csv("china_gdp.csv")
df.head(10)
```

      Year
      Value

      0
      1960
      5.918412e+10

      1
      1961
      4.955705e+10

      2
      1962
      4.668518e+10

      3
      1963
      5.009730e+10

      4
      1964
      5.906225e+10

      5
      1965
      6.970915e+10

      6
      1966
      7.587943e+10

      7
      1967
      7.205703e+10

      8
      1968
      6.999350e+10

      9
      1969
      7.871882e+10

__Did you know?__ When it comes to Machine Learning, you will likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: [Sign up now for free](http://cocl.us/ML0101EN-IBM-Offer-CC)

### Plotting the Dataset ###
This is what the datapoints look like. It kind of looks like an either logistic or exponential function. The growth starts off slow, then from 2005 on forward, the growth is very significant. And finally, it decelerate slightly in the 2010s.

```python
plt.figure(figsize=(8,5))
x_data, y_data = (df["Year"].values, df["Value"].values)
plt.plot(x_data, y_data, 'ro')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

### Choosing a model ###

From an initial look at the plot, we determine that the logistic function could be a good approximation,
since it has the property of starting with a slow growth, increasing growth in the middle, and then decreasing again at the end; as illustrated below:

```python
X = np.arange(-5.0, 5.0, 0.1)
Y = 1.0 / (1.0 + np.exp(-X))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

The formula for the logistic function is the following:

$$ \hat{Y} = \frac1{1+e^{\beta_1(X-\beta_2)}}$$

$\beta_1$: Controls the curve's steepness,

$\beta_2$: Slides the curve on the x-axis.

### Building The Model ###
Now, let's build our regression model and initialize its parameters.

```python
def sigmoid(x, Beta_1, Beta_2):
     y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))
     return y
```

Lets look at a sample sigmoid line that might fit with the data:

```python
beta_1 = 0.10
beta_2 = 1990.0

#logistic function
Y_pred = sigmoid(x_data, beta_1 , beta_2)

#plot initial prediction against datapoints
plt.plot(x_data, Y_pred*15000000000000.)
plt.plot(x_data, y_data, 'ro')
```

Our task here is to find the best parameters for our model. Lets first normalize our x and y:

```python
# Lets normalize our data
xdata =x_data/max(x_data)
ydata =y_data/max(y_data)
```

#### How we find the best parameters for our fit line?
we can use __curve_fit__ which uses non-linear least squares to fit our sigmoid function, to data. Optimal values for the parameters so that the sum of the squared residuals of sigmoid(xdata, *popt) - ydata is minimized.

popt are our optimized parameters.

```python
from scipy.optimize import curve_fit
popt, pcov = curve_fit(sigmoid, xdata, ydata)
#print the final parameters
print(" beta_1 = %f, beta_2 = %f" % (popt[0], popt[1]))
```

    beta_1 = 690.451711, beta_2 = 0.997207

Now we plot our resulting regression model.

```python
x = np.linspace(1960, 2015, 55)
x = x/max(x)
plt.figure(figsize=(8,5))
y = sigmoid(x, *popt)
plt.plot(xdata, ydata, 'ro', label='data')
plt.plot(x,y, linewidth=3.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

## Practice
Can you calculate what is the accuracy of our model?

```python
# write your code here
msk = np.random.rand(len(df)) < 0.8
train_x = xdata[msk]
test_x = xdata[~msk]
train_y = ydata[msk]
test_y = ydata[~msk]

# build the model using train set
popt, pcov = curve_fit(sigmoid, train_x, train_y)

# predict using test set
y_hat = sigmoid(test_x, *popt)

# evaluation
print("Mean absolute error: %.2f" % np.mean(np.absolute(y_hat - test_y)))
print("Residual sum of squares (MSE): %.2f" % np.mean((y_hat - test_y) ** 2))
from sklearn.metrics import r2_score
print("R2-score: %.2f" % r2_score(y_hat , test_y) )

```

    Mean absolute error: 0.18
    Residual sum of squares (MSE): 0.13
    R2-score: -224542884463096208820659027968.00

Double-click __here__ for the solution.

Want to learn more?

IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: SPSS Modeler

Also, you can use Watson Studio to run these notebooks faster with bigger datasets. Watson Studio is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, Watson Studio enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of Watson Studio users today with a free account at Watson Studio

Thanks for completing this lesson!

Author:  Saeed Aghabozorgi

Saeed Aghabozorgi, PhD is a Data Scientist in IBM with a track record of developing enterprise level applications that substantially increases clients’ ability to turn data into actionable knowledge. He is a researcher in data mining field and expert in developing advanced analytic methods like machine learning and statistical modelling on large datasets.

Copyright © 2018 Cognitive Class. This notebook and its source code are released under the terms of the MIT License.

Machine Learning With Python/w2/ML0101EN-Reg-NoneLinearRegression-py-v1.ipynb


\begin{equation}
\begin{aligned} \log p^{*}(x) &=\log p^{*}(z)+\log \left|\operatorname{det} \frac{d F}{d x}\right| \\ &=\log p^{*}(z)+\sum_{i=1}^{K} \log \left|\operatorname{det} \frac{d h_{i}}{d h_{i-1}}\right| \end{aligned}
\end{equation}

# Preface

```python
import pandas as pd
import seaborn as sns
import torch

import pyro
import pyro.distributions as dist

from rethinking import LM, coef
```

### Code 0.1

```python
print("All models are wrong, but some are useful.")
```

    All models are wrong, but some are useful.

### Code 0.2

```python
x = torch.arange(1., 3)
x = x * 10
x = x.log()
x = x.sum()
x = x.exp()
x
```

    tensor(200.0000)

### Code 0.3

```python
print(torch.tensor(0.01).pow(200).log())
print(200 * torch.tensor(0.01).log())
```

    tensor(-inf)
    tensor(-921.0341)

### Code 0.4

```python
# Load the data:
# car braking distances in feet paired with speeds in km/h
# see cars.info() for details
cars = pd.read_csv("../data/cars.csv")

# fit a linear regression of distance on speed
m = LM("dist ~ speed", data=cars).run()

# estimated coefficients from the model
print(coef(m))

# plot residuals against speed
y = coef(m)["Intercept"].item() + coef(m)["speed"].item() * cars["speed"]
resid = cars["dist"] - y
ax = sns.scatterplot(cars["speed"], resid)
ax.set(xlabel="speed", ylabel="residual");
```

### Code 0.5

```sh
pip install jupyter pandas pyro-ppl seaborn torch
```