# Transformée de Fourier

```python
%matplotlib inline

import matplotlib
matplotlib.rcParams['figure.figsize'] = (6, 6)

import math
import cmath          # math functions for complex numbers
import numpy as np
import matplotlib.pyplot as plt

import ipywidgets
from ipywidgets import interact

import sympy as sp

# See: http://docs.sympy.org/latest/tutorial/printing.html
sp.init_printing()

t = sp.symbols("t")
```

**TODO**:
* https://cdn.uclouvain.be/public/Exports%20reddot/fsa/documents/Travail_Complet.pdf <- semble très intéressant !
* Exemples en notation exponentielle
* Cas d'un signal 2D
* Discrete Fourier Transform + algo pour implémentation concrète (du calcul des coefs) en Python
* FFT

The following notations come from the following book: *Toutes les mathématiques et les bases de l'informatique*, Horst Stöcker (Dunod, 2002)

## Cas simple: fonction périodique de période $2\pi$

### Définitions

#### Série de Fourier

$$
f(t) = \frac{a_0}{2} + \sum^{\infty}_{n=1} [a_n \cos(n t) + b_n \sin(n t)]
$$

#### Calcul des coefficients de Fourier

\begin{eqnarray*}
    \forall n \in \mathbb{N},   ~~~~~ a_n & = & \frac{1}{\pi} \int^{\pi}_{-\pi} f(t) \cos(n t) ~ dt \\
    \forall n \in \mathbb{N}^*, ~~~~~ b_n & = & \frac{1}{\pi} \int^{\pi}_{-\pi} f(t) \sin(n t) ~ dt \\
\end{eqnarray*}

Cf. section \ref{sec:exemples} pour des exemples de calculs de coefficients de Fourier.

Rappel: $\mathbb{N}^*$ est l'ensemble $\mathbb{N}$ privé de $0$.

#### Condition de Dirichlet

...

rem: si on avait que des cos (ou que des sin) dans la série de fourier, elle ne pourrait coder définir que des fonctions qui s'annulent tous les pi/2...

### Questions / notes / remarques

* Que représente le terme $\frac{a_0}{2}$ dans la définition de la série de Fourier ?

...

* Que représente le facteur $\frac{1}{\pi}$ dans les coeffients de Fourier ?

...

## Cas général: fonction périodique de période $T$

### Définition

#### Série de Fourier

$$
f(t) = \frac{a_0}{2} + \sum^{\infty}_{n=1} [a_n \cos(\omega n t) + b_n \sin(\omega n t)]
$$

#### Calcul des *coefficients de Fourier*

\begin{eqnarray*}
    \forall n \in \mathbb{N},   ~~~~~ a_n & = & \frac{2}{T} \int^{T/2}_{-T/2} f(t) \cos(\omega n t) ~ dt \\
    \forall n \in \mathbb{N}^*, ~~~~~ b_n & = & \frac{2}{T} \int^{T/2}_{-T/2} f(t) \sin(\omega n t) ~ dt \\
\end{eqnarray*}

## Représentation spectrale

**TODO**: rendre plus claire la définition suivante...

### Série de Fourier

$$
f(t) = \frac{a_0}{2} + \sum^{\infty}_{n=1} [A_n \sin(\omega n t + \phi_n)]
$$
avec
$$A_n = \sqrt{a^2_n + b^2_n} \quad \text{ et } \quad \tan \phi_n = \frac{a_n}{b_n}$$

rem:
ne fait qu'écrire autrement les coefs (passage d'une notation carthésienne à une notation polaire)...
* $a_1$ et $b_1$ en $A_1$ et $\phi_1$,
* $a_2$ et $b_2$ en $A_2$ et $\phi_2$,
* ...

Pour chaque $n$, les coefs $a_n$ et $b_n$ peuvent être vus comme un point dans le plan.

Ces points peuvent aussi être définis avec une notation polaire (c'est ce qui est fait ici): $A_n$ et $\phi_n$.

Qu'est-ce que ça apporte d'un point de vue pratique ? Il serait intéressant de visualiser la différence entre ces deux espaces de coefficients... Le filtrage d'un signal est peut-être plus facile dans le 2e ?

## Représentation complexe (notation exponentielle)

**TODO**: rendre plus claire la définition suivante...

### Définitions pour une fonction $2\pi$ periodique

#### Série de Fourier

$$
f(t) = \sum^{\infty}_{n=\color{red}{-\infty}} c_n e^{i n t}
$$

#### Calcul des *coefficients de Fourier*

$$
c_n = \frac{1}{2\pi} \int^{\pi}_{-\pi} f(t) e^{-i n t} dt
$$

### Définitions pour une fonction $T$ periodique

#### Série de Fourier

$$
f(t) = \sum^{\infty}_{n=\color{red}{-\infty}} c_n e^{i\omega n t}
$$

#### Calcul des *coefficients de Fourier*

$$
c_n = \frac{1}{T} \int^{T/2}_{-T/2} f(t) e^{-i \omega n t} dt
$$

### Relation entre les coefficients $a_n$, $b_n$ et $c_n$

$$
c_n =
\left\{
\begin{align}
\frac{a_0}{2} & \quad n = 0 \\
\\
\frac{a_n - ib_n}{2} & \quad n > 0 \\
\\
\frac{a_{-n} - ib_{-n}}{2} & \quad n < 0
\end{align}
\right.
$$
ou
$$
\left.
\begin{align}
a_n & = c_n + c_{-n} \\
b_n & = i(c_n - c_{-n})
\end{align}
\right\}
n > 0
$$

Les valeurs $\omega_n = \omega n$ sont appelées le *spectre* de $f(t)$.

rem:
ne fait que regrouper les coefs
* $a_1$ et $b_1$ dans un nombre complexe $z_1$,
* $a_2$ et $b_2$ dans un nombre complexe $z_2$,
* ...

Pour chaque $n$, les coefs $a_n$ et $b_n$ peuvent être vus comme un point dans le plan et donc comme un nombre complexe.

Du coup, ces points peuvent aussi être définis avec une notation exponentielle.

motivation: "ça simplifie les calculs et les notations"

## Conclusion

TODO...

### Synthèse

Définitions à connaitre:
* Série de Fourier
* Coefficients de Fourier
* Condition de Dirichlet

## Annexes

### Rappels de maths

TODO: add plots...

\begin{eqnarray*}
    \forall c \in \mathbb{N}^*, \int^{\pi}_{-\pi} c ~ dt         & = & c \times 2\pi \\
    \forall c \in \mathbb{N}^*, \int^{\pi}_{-\pi} c \cos(t) ~ dt & = & 0 \\
    \forall c \in \mathbb{N}^*, \int^{\pi}_{-\pi} c \sin(t) ~ dt & = & 0 \\
\end{eqnarray*}

$$
\forall m \in \mathbb{N}^*, n \in \mathbb{N}^*,
\int^{\pi}_{-\pi} \cos(n ~ t) \cos(m ~ t) ~ dt =
\left\{
    \begin{array}{l l}
        \pi  & \quad \text{si $n = m$}\\
        0    & \quad \text{si $n \neq m$}
    \end{array} \right.
$$

Par exemple:
\begin{eqnarray*}
    \int^{\pi}_{-\pi} \cos(t) \cos(t) ~ dt   & = & \pi \\
    \int^{\pi}_{-\pi} \sin(t) \sin(t) ~ dt   & = & \pi \\
    \int^{\pi}_{-\pi} \cos(t) \sin(t) ~ dt   & = & 0 \\
    \int^{\pi}_{-\pi} \cos(t) \cos(2t) ~ dt  & = & 0 \\
    \int^{\pi}_{-\pi} \cos(t) \sin(2t) ~ dt  & = & 0 \\
    \int^{\pi}_{-\pi} \cos(2t) \cos(2t) ~ dt & = & \pi \\
    \int^{\pi}_{-\pi} \sin(2t) \sin(2t) ~ dt & = & \pi \\
\end{eqnarray*}

### Exemples détaillés

#### Calcule des coefficients de Fourier pour la fonction $f(t) = 3$

\begin{eqnarray*}
    a_0 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} 3 \cos(0) ~ dt \\
        & = & \frac{1}{\pi} \int^{\pi}_{-\pi} 3 ~ dt \\
        & = & \frac{1}{\pi} \times 3 \times 2 \pi \\
        & = & 6 \\
\end{eqnarray*}

\begin{eqnarray*}
    a_1 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} 3 \cos(t) ~ dt \\
        & = & 0 \\
\end{eqnarray*}

\begin{eqnarray*}
    b_1 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} 3 \sin(t) ~ dt \\
        & = & 0 \\
\end{eqnarray*}

De même, les coefficients $a_3$, $b_3$, $a_4$, $b_4$, etc. sont tous nuls.

##### Vérification

On a bien:
\begin{eqnarray*}
    f(t) & = & \frac{a_0}{2} + \sum^{\infty}_{n=1} (a_n \cos(n t) + b_n \sin(n t)) \\
         & = & \frac{6}{2} + 0 \times \cos(t) + 0 \times \sin(t) + 0 \times \cos(2t) + 0 \times \sin(2t) + ... \\
         & = & 3 \\
\end{eqnarray*}

#### Calcule des coefficients de Fourier pour la fonction $f(t) = \cos(t)$

TODO: add plots...

\begin{eqnarray*}
    a_0 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} \cos(t) \cos(0) ~ dt \\
        & = & \frac{1}{\pi} \int^{\pi}_{-\pi} \cos(t) ~ dt \\
        & = & 0 \\
\end{eqnarray*}

\begin{eqnarray*}
    a_1 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} \cos(t) \cos(t) ~ dt \\
        & = & \frac{1}{\pi} \times \pi \\
        & = & 1 \\
\end{eqnarray*}

\begin{eqnarray*}
    b_1 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} \cos(t) \sin(t) ~ dt \\
        & = & \frac{1}{\pi} \times 0 \\
        & = & 0 \\
\end{eqnarray*}

\begin{eqnarray*}
    a_2 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} \cos(t) \cos(2t) ~ dt \\
        & = & \frac{1}{\pi} \times 0 \\
        & = & 0 \\
\end{eqnarray*}

\begin{eqnarray*}
    b_2 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} \cos(t) \sin(2t) ~ dt \\
        & = & \frac{1}{\pi} \times 0 \\
        & = & 0 \\
\end{eqnarray*}

De même, les coefficients $a_3$, $b_3$, $a_4$, $b_4$, etc. sont tous nuls.

##### Vérification

On a bien:
\begin{eqnarray*}
    f(t) & = & \frac{a_0}{2} + \sum^{\infty}_{n=1} (a_n \cos(n t) + b_n \sin(n t)) \\
         & = & \frac{0}{2} + 1 \times \cos(t) + 0 \times \sin(t) + 0 \times \cos(2t) + 0 \times \sin(2t) + ... \\
         & = & \cos(t) \\
\end{eqnarray*}

#### Calcule des coefficients de Fourier pour la fonction $f(t) = 3 \cos(t)$

\begin{eqnarray*}
    a_0 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} f(t) \cos(n t) ~ dt \\
        & = & ... \\
\end{eqnarray*}

\begin{eqnarray*}
    a_1 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} f(t) \cos(n t) ~ dt \\
        & = & ... \\
\end{eqnarray*}

\begin{eqnarray*}
    b_1 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} f(t) \sin(n t) ~ dt \\
        & = & ... \\
\end{eqnarray*}

#### Calcule des coefficients de Fourier pour la fonction $f(t) = \cos(t) + 3$

\begin{eqnarray*}
    a_0 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} f(t) \cos(n t) ~ dt \\
        & = & ... \\
\end{eqnarray*}

\begin{eqnarray*}
    a_1 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} f(t) \cos(n t) ~ dt \\
        & = & ... \\
\end{eqnarray*}

\begin{eqnarray*}
    b_1 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} f(t) \sin(n t) ~ dt \\
        & = & ... \\
\end{eqnarray*}

#### Calcule des coefficients de Fourier pour la fonction $f(t) = \cos(t + 3)$

**TODO: ???**

\begin{eqnarray*}
    a_0 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} f(t) \cos(n t) ~ dt \\
        & = & ... \\
\end{eqnarray*}

\begin{eqnarray*}
    a_1 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} f(t) \cos(n t) ~ dt \\
        & = & ... \\
\end{eqnarray*}

\begin{eqnarray*}
    b_1 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} f(t) \sin(n t) ~ dt \\
        & = & ... \\
\end{eqnarray*}

#### Calcule des coefficients de Fourier pour la fonction $f(t) = \cos(t) + 2 \sin(t) - 3 \sin(2t) + 4$

TODO: add plots...

```python
integ = sp.Integral((sp.cos(t) + 2 * sp.sin(t) - 3 * sp.sin(2*t) + 4) * sp.cos(0), (t, -sp.pi, sp.pi))
sp.Eq(integ, integ.doit())
```

\begin{eqnarray*}
    a_0 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} (\cos(t) + 2 \sin(t) - 3 \sin(2t) + 4) \cos(0) ~ dt \\
        & = & \frac{1}{\pi} \int^{\pi}_{-\pi} \cos(t) ~ dt \\
        & = & \frac{1}{\pi} \times 8\pi\\
        & = & 8\\
\end{eqnarray*}

```python
integ = sp.Integral((sp.cos(t) + 2 * sp.sin(t) - 3 * sp.sin(2*t) + 4) * sp.cos(t), (t, -sp.pi, sp.pi))
sp.Eq(integ, integ.doit())
```

\begin{eqnarray*}
    a_1 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} (\cos(t) + 2 \sin(t) - 3 \sin(2t) + 4) \cos(t) ~ dt \\
        & = & \frac{1}{\pi} \times \pi \\
        & = & 1 \\
\end{eqnarray*}

```python
integ = sp.Integral((sp.cos(t) + 2 * sp.sin(t) - 3 * sp.sin(2*t) + 4) * sp.sin(t), (t, -sp.pi, sp.pi))
sp.Eq(integ, integ.doit())
```

\begin{eqnarray*}
    b_1 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} (\cos(t) + 2 \sin(t) - 3 \sin(2t) + 4) \sin(t) ~ dt \\
        & = & \frac{1}{\pi} \times 2\pi \\
        & = & 2 \\
\end{eqnarray*}

```python
integ = sp.Integral((sp.cos(t) + 2 * sp.sin(t) - 3 * sp.sin(2*t) + 4) * sp.cos(2*t), (t, -sp.pi, sp.pi))
sp.Eq(integ, integ.doit())
```

\begin{eqnarray*}
    a_2 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} (\cos(t) + 2 \sin(t) - 3 \sin(2t) + 4) \cos(2t) ~ dt \\
        & = & \frac{1}{\pi} \times 0 \\
        & = & 0 \\
\end{eqnarray*}

```python
integ = sp.Integral((sp.cos(t) + 2 * sp.sin(t) - 3 * sp.sin(2*t) + 4) * sp.sin(2*t), (t, -sp.pi, sp.pi))
sp.Eq(integ, integ.doit())
```

\begin{eqnarray*}
    b_2 & = & \frac{1}{\pi} \int^{\pi}_{-\pi} (\cos(t) + 2 \sin(t) - 3 \sin(2t) + 4) \sin(2t) ~ dt \\
        & = & \frac{1}{\pi} \times -3\pi \\
        & = & -3 \\
\end{eqnarray*}

Les coefficients $a_3$, $b_3$, $a_4$, $b_4$, etc. sont tous nuls.

##### Vérification

On a bien:
\begin{eqnarray*}
    f(t) & = & \frac{a_0}{2} + \sum^{\infty}_{n=1} (a_n \cos(n t) + b_n \sin(n t)) \\
         & = & \frac{8}{2} + 1 \times \cos(t) + 2 \times \sin(t) + 0 \times \cos(2t) + (-3) \times \sin(2t) + \dots \\
         & = & \cos(t) + 2 \sin(t) - 3 \sin(2t) + 4 \\
\end{eqnarray*}

#### Calcule des coefficients de Fourier pour la fonction $f(t) = \cos(t) + 2 \sin(t) - 3 \sin(2t) + 4$ en utilisant la notation exponentielle

TODO: add plots...

**TODO**

$$
c_n = \frac{1}{2\pi} \int^{\pi}_{-\pi} f(t) e^{-i n t} dt
$$

```python
sp.cos(t) + 2 * sp.sin(t) - 3 * sp.sin(2*t) + 4
```

```python
n = 0
sp.plot(sp.cos(t) + 2 * sp.sin(t) - 3 * sp.sin(2*t) + 4, (t, -sp.pi, sp.pi));
```

```python
import mpmath

n = 2
mpmath.cplot(lambda t: mpmath.exp(-mpmath.j * n * t), points=100000)
```

```python
n = 0
integ = sp.Integral((sp.cos(t) + 2 * sp.sin(t) - 3 * sp.sin(2*t) + 4) * sp.exp(-sp.I * n * t), (t, -sp.pi, sp.pi))
integ_res = integ.doit()
sp.Eq(integ, integ_res)
```

```python
sp.simplify(integ_res / sp.pi)
```

\begin{eqnarray*}
    c_0 & = & \frac{1}{2\pi} \int^{\pi}_{-\pi} (\cos(t) + 2 \sin(t) - 3 \sin(2t) + 4) e^{0} ~ dt \\
        & = & \frac{1}{\pi} \times 8\pi\\
        & = & 8\\
\end{eqnarray*}

```python
n = 1
integ = sp.Integral((sp.cos(t) + 2 * sp.sin(t) - 3 * sp.sin(2*t) + 4) * sp.exp(-sp.I * n * t), (t, -sp.pi, sp.pi))
integ_res = integ.doit()
sp.Eq(integ, integ_res)
```

```python
sp.simplify(integ_res / sp.pi)
```

\begin{eqnarray*}
    c_1 & = & \frac{1}{2\pi} \int^{\pi}_{-\pi} (\cos(t) + 2 \sin(t) - 3 \sin(2t) + 4) e^{-it} ~ dt \\
        & = & \frac{1}{\pi} \times \pi - 2i\pi\\
        & = & 1 - 2i\\
\end{eqnarray*}

```python
n = -1
integ = sp.Integral((sp.cos(t) + 2 * sp.sin(t) - 3 * sp.sin(2*t) + 4) * sp.exp(-sp.I * n * t), (t, -sp.pi, sp.pi))
integ_res = integ.doit()
sp.Eq(integ, integ_res)
```

```python
sp.simplify(integ_res / sp.pi)
```

\begin{eqnarray*}
    c_{-1} & = & \frac{1}{2\pi} \int^{\pi}_{-\pi} (\cos(t) + 2 \sin(t) - 3 \sin(2t) + 4) e^{it} ~ dt \\
        & = & \frac{1}{\pi} \times \pi + 2i\pi\\
        & = & 1 + 2i\\
\end{eqnarray*}

```python
n = 2
integ = sp.Integral((sp.cos(t) + 2 * sp.sin(t) - 3 * sp.sin(2*t) + 4) * sp.exp(-sp.I * n * t), (t, -sp.pi, sp.pi))
integ_res = integ.doit()
sp.Eq(integ, integ_res)
```

```python
sp.simplify(integ_res / sp.pi)
```

\begin{eqnarray*}
    c_2 & = & \frac{1}{2\pi} \int^{\pi}_{-\pi} (\cos(t) + 2 \sin(t) - 3 \sin(2t) + 4) e^{-2it} ~ dt \\
        & = & \frac{1}{\pi} \times 3i\pi\\
        & = & 3i\\
\end{eqnarray*}

```python
n = -2
integ = sp.Integral((sp.cos(t) + 2 * sp.sin(t) - 3 * sp.sin(2*t) + 4) * sp.exp(-sp.I * n * t), (t, -sp.pi, sp.pi))
integ_res = integ.doit()
sp.Eq(integ, integ_res)
```

```python
sp.simplify(integ_res / sp.pi)
```

\begin{eqnarray*}
    c_{-2} & = & \frac{1}{2\pi} \int^{\pi}_{-\pi} (\cos(t) + 2 \sin(t) - 3 \sin(2t) + 4) e^{2it} ~ dt \\
        & = & \frac{1}{\pi} \times -3i\pi\\
        & = & -3i\\
\end{eqnarray*}

```python
n = 3
integ = sp.Integral((sp.cos(t) + 2 * sp.sin(t) - 3 * sp.sin(2*t) + 4) * sp.exp(-sp.I * n * t), (t, -sp.pi, sp.pi))
integ_res = integ.doit()
sp.Eq(integ, integ_res)
```

```python
sp.simplify(integ_res / sp.pi)
```

\begin{eqnarray*}
    c_3 & = & \frac{1}{2\pi} \int^{\pi}_{-\pi} (\cos(t) + 2 \sin(t) - 3 \sin(2t) + 4) e^{-3it} ~ dt \\
        & = & \frac{1}{\pi} \times 0 \\
        & = & 0\\
\end{eqnarray*}

```python
n = -3
integ = sp.Integral((sp.cos(t) + 2 * sp.sin(t) - 3 * sp.sin(2*t) + 4) * sp.exp(-sp.I * n * t), (t, -sp.pi, sp.pi))
integ_res = integ.doit()
sp.Eq(integ, integ_res)
```

```python
sp.simplify(integ_res / sp.pi)
```

\begin{eqnarray*}
    c_{-3} & = & \frac{1}{2\pi} \int^{\pi}_{-\pi} (\cos(t) + 2 \sin(t) - 3 \sin(2t) + 4) e^{3it} ~ dt \\
        & = & \frac{1}{\pi} \times 0 \\
        & = & 0 \\
\end{eqnarray*}

Les coefficients $c_4$, $b_-4$, $a_5$, $b_-5$, etc. sont tous nuls.

##### Vérification

On a bien:
\begin{eqnarray*}
    f(t) & = & \sum^{\infty}_{n=-\infty} c_n e^{i n t} \\
         & = & \underbrace{8 \times e^{0}} + \underbrace{1-2i \times e^{-it}} + \underbrace{1+2i \times e^{it}} + \underbrace{3i \times e^{-2it}} + \underbrace{-3i \times e^{2it}} + \dots \\
         & = & TODO !!!!!!! \\
         & = & \cos(t) + 2 \sin(t) - 3 \sin(2t) + 4 \\
\end{eqnarray*}

```python
eq1 = sp.simplify(8 * sp.exp(0) \
                  + 1 - 2 * sp.I * sp.exp(-sp.I * t) \
                  + 1 + 2 * sp.I * sp.exp( sp.I * t) \
                  + 3 * sp.I * sp.exp(-2* sp.I * t) \
                  - 3 * sp.I * sp.exp( 2* sp.I * t) )
eq1
```

```python
sp.simplify(eq1 - sp.cos(t) + 2 * sp.sin(t) - 3 * sp.sin(2 * t) + 4)
```

**TODO**

L'exemple précédent est foireux... Regarder plutôt les 3 exemples du site suivant pour s'entrainer avec la notation complexe: https://www.math24.net/complex-form-fourier-series/

Voir aussi https://www.youtube.com/watch?v=FIKPlRsADL0

Intérêt de la notation exponentielle: certains signaux tel que celui présenté dans
http://www.thefouriertransform.com/series/complexcoefficients.php
nécessiterait de calculer une infinité de coefficients avec la notation trigonométrique... Avec la notation exponentielle, deux formules suffisent à donner tous les coefficients $\neq$ 0 !

**TODO**:
* step 1: calcul pour retrouver la série réelle ($\mathbb{R}$) à partir des coefs complexes -> prendre un de ces exemples, partir du résultat (les coefs. $c_n$) et calculer la série -> il y a un ex dans https://www.math24.net/complex-form-fourier-series/
* step 2: calcul des coefs.

**TODO**: DFT (p.794) et FFT (p.797)


## Topics covered in this notebook:
1. What is K-Nearest Neighbors(kNN) mean?
2. Implementation.
3. How to choose K?
4. Common Issues & Fix.
5. Where kNN can fail?
6. References.

## 1. K - Nearest Neighbors:

1. The idea is to make prediction using the closest know data points.
2. Look at the image below:
    1. There are 2 categories: Star & Triangle.
    2. Consider a new test data point -> Green square. What is this point classified as?
        1. K = 3 -> 3-nearest neighbor -> Pick star.
        2. K = 5 -> 5-nearest neighbor -> Pick traingle.
        3. We classify by calcualting eulidean distance between nearby K points & their corresponding classes.
3. Forms complex decision boundaries; adapts to data density.
4. These type of models are called non-parametric models.
5. These type of classifiers are also called as lazy classifiers.
    1. train(X,Y) doesn't do anything. Just stores X & Y.
    2. predict(X') does all the work by looking through stored X & Y.

6. Few assumptions:
    1. Output varies smoothly with input.
    2. Data occupies sub-space of high-dimensional input space.

## 2. Implementation:
3. Idea is simple but implementation can be tricky.
4. Keeping track of arbitrary number of distances not so easy.
5. First -> need to look through all of the training data -> O(N).
6. Then need to look through the closest distances you have stored so far -> O(K).
\begin{align}
\large \lvert\lvert \large x^{a} \,-\, x^{b}\rvert\rvert_2 \, &= \large \sqrt {\Sigma_{j=1}^d (x_j^{a} \,-\, x_j^{b})^2} \\
\end{align}
7. Total O(NK).
    8. Searching through a sorted list would be O(log K), a little better.
    9. Even better: Ball Tree, K-D Tre etc.
8. Once we have the k-nearest neighbors we need to turn them into votes which means we need to store the class as well.
    9. {dist1: class1, dist2:class2,...}
9. Count up the values:
    1. {class1: num_class1,class2:num_class2...}
10. Pick the class that has the highest votes.
    1. What if there is tie?
        1. Use whatever argmax(votes) outputs.
        2. Pick one at random.
        3. Weight by distance to neighbors.

## 3. How to choose K?:
1. No easy answer.
2. K is hyperparameter.
3. Use cross-validation.
4. Larger k may lead to better performance.
5. But if we set k too large we may look at samples that are not neighbors.
6. Rule of thumb: k < sqrt(n), where n is number of training examples.

## 4. Common Issues & Fix:
1. If some attribute have larger ranges, they are treated as more important:
    1. Normalize scale:
        1. Linear transformatio to be between [0,1].
        2. Scale to have 0 mean and 1 variance.
        3. Caution: Sometimes scale matters.
2. Irrelevant attributes can add noise to distance measure.
    1. Remove attributes.
    2. Adapt weights using regularization techniques (using othere types of classifier).
3. Computation:O(NK)
    1. Use subset of dimensions.
    2. Pre-sort training examples into fast data structures(e.g. kd-trees) - Need to read about it.
    3. Compute only approximate distance(e.g. LSH).
    4. Remove redundant data(e.g., condensing).
4. High Dimensional Data: 'Curse of dimensionality'
    1. Required amount of data increases exponentially with dimension.
    2. Computation cost also increases.

## 5. Where kNN can fail?
1. Grid of alteranting dots.
    1. If you choose K=3, there will always 2/3 vote from wrong class.
        1. Can fix by choosing K = 1.
        2. Weighing each points by distance.

## 6. References:

1. An Introduction to Statistical Learning Textbook by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani.
2. University of Michigan EECS 445 - Machine Learning Course (https://github.com/eecs445-f16/umich-eecs445-f16).

3. University of Toronto CSC 411 - Intro. to Machine Learning (http://www.cs.toronto.edu/~urtasun/courses/CSC411_Fall16/CSC411_Fall16.html).

4. Stanford CS109 - Intro. to proabability for computer scientists (https://web.stanford.edu/class/archive/cs/cs109/cs109.1166/).

5. Few online courses on Udemy, Coursera etc.


# Semi-Monocoque Theory

```python
from pint import UnitRegistry
import sympy
import networkx as nx
#import numpy as np
import matplotlib.pyplot as plt
#import sys
%matplotlib inline
#from IPython.display import display
```

    /home/claudio/anaconda2/envs/py35/lib/python3.5/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
      warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
    /home/claudio/anaconda2/envs/py35/lib/python3.5/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
      warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

Import **Section** class, which contains all calculations

```python
from Section import Section
```

Initialization of **sympy** symbolic tool and **pint** for dimension analysis (not really implemented rn as not directly compatible with sympy)

```python
ureg = UnitRegistry()
sympy.init_printing()
```

Define **sympy** parameters used for geometric description of sections

```python
A, A0, t, t0, a, b, h, L = sympy.symbols('A A_0 t t_0 a b h L', positive=True)
```

We also define numerical values for each **symbol** in order to plot scaled section and perform calculations

```python
values = [(A, 150 * ureg.millimeter**2),(A0, 250  * ureg.millimeter**2),(a, 80 * ureg.millimeter), \
          (b, 20 * ureg.millimeter),(h, 35 * ureg.millimeter),(L, 2000 * ureg.millimeter)]
datav = [(v[0],v[1].magnitude) for v in values]
```

# First example: Closed section

Define graph describing the section:

1) **stringers** are **nodes** with parameters:
- **x** coordinate
- **y** coordinate
- **Area**

2) **panels** are **oriented edges** with parameters:
- **thickness**
- **lenght** which is automatically calculated

```python
stringers = {1:[(sympy.Integer(0),h),A],
             2:[(a/2,h),A],
             3:[(a,h),A],
             4:[(a-b,sympy.Integer(0)),A],
             5:[(b,sympy.Integer(0)),A]}

panels = {(1,2):t,
          (2,3):t,
          (3,4):t,
          (4,5):t,
          (5,1):t}
```

Define section and perform first calculations

```python
S1 = Section(stringers, panels)
```

## Verify that we find a simply closed section

```python
S1.cycles
```

## Plot of **S1** section in original reference frame

Define a dictionary of coordinates used by **Networkx** to plot section as a Directed graph.
Note that arrows are actually just thicker stubs

```python
start_pos={ii: [float(S1.g.node[ii]['ip'][i].subs(datav)) for i in range(2)] for ii in S1.g.nodes() }
```

```python
plt.figure(figsize=(12,8),dpi=300)
nx.draw(S1.g,with_labels=True, arrows= True, pos=start_pos)
plt.arrow(0,0,20,0)
plt.arrow(0,0,0,20)
#plt.text(0,0, 'CG', fontsize=24)
plt.axis('equal')
plt.title("Section in starting reference Frame",fontsize=16);
```

Expression of **Inertial properties** wrt Center of Gravity in with original rotation

```python
S1.Ixx0, S1.Iyy0, S1.Ixy0, S1.α0
```

## Plot of **S1** section in inertial reference Frame

Section is plotted wrt **center of gravity** and rotated (if necessary) so that *x* and *y* are principal axes.
**Center of Gravity** and **Shear Center** are drawn

```python
positions={ii: [float(S1.g.node[ii]['pos'][i].subs(datav)) for i in range(2)] for ii in S1.g.nodes() }
```

```python
x_ct, y_ct = S1.ct.subs(datav)

plt.figure(figsize=(12,8),dpi=300)
nx.draw(S1.g,with_labels=True, pos=positions)
plt.plot([0],[0],'o',ms=12,label='CG')
plt.plot([x_ct],[y_ct],'^',ms=12, label='SC')
#plt.text(0,0, 'CG', fontsize=24)
#plt.text(x_ct,y_ct, 'SC', fontsize=24)
plt.legend(loc='lower right', shadow=True)
plt.axis('equal')
plt.title("Section in pricipal reference Frame",fontsize=16);
```

Expression of **inertial properties** in *principal reference frame*

```python
S1.Ixx, S1.Iyy, S1.Ixy, S1.θ
```

## **Shear center** expression

```python
S1.ct
```

## Analisys of symmetry properties of the section

For **x** and **y** axes pair of symmetric **nodes** and **edges** are searched for

```python
S1.symmetry
```

    [{'edges': [((1, 2), (2, 3)), ((3, 4), (5, 1)), (4, 5)],
      'nodes': [(1, 3), (2, 2), (4, 5)]},
     {'edges': [], 'nodes': []}]

## Compute axial loads in Stringers in **S1**

We first define some symbols:

```python
Tx, Ty, Nz, Mx, My, Mz, F, ry, ry, mz = sympy.symbols('T_x T_y N_z M_x M_y M_z F r_y r_x m_z')
```

Set **loads** on the section:

**Example 1**: _shear_ in **y** direction and _bending moment_ in **x** direction

```python
S1.set_loads(_Tx=0, _Ty=Ty, _Nz=0, _Mx=Mx, _My=0, _Mz=0)
```

Compute _axial loads_ in **stringers** and _shear flows_ in **panels**

```python
S1.compute_stringer_actions()
S1.compute_panel_fluxes();
```

## Axial loads

```python
S1.N
```

## Shear flows

```python
S1.q
```

**Example 2**: _twisting moment_ in **z** direction

```python
S1.set_loads(_Tx=0, _Ty=0, _Nz=0, _Mx=0, _My=0, _Mz=Mz)
S1.compute_stringer_actions()
S1.compute_panel_fluxes();
```

**Axial loads**

```python
S1.N
```

**Panel fluxes**

```python
S1.q
```

Set **loads** on the section:

**Example 3**: _shear_ in **x** direction and _bending moment_ in **y** direction

```python
S1.set_loads(_Tx=Tx, _Ty=0, _Nz=0, _Mx=0, _My=My, _Mz=0)
S1.compute_stringer_actions()
S1.compute_panel_fluxes();
```

**Axial loads**

```python
S1.N
```

**Panel fluxes**
Not really an easy expression

```python
S1.q
```

## Compute **Jt**

Computation of **torsional moment of inertia**:

```python
S1.compute_Jt()
S1.Jt
```

# Second example: Open section

```python
stringers = {1:[(sympy.Integer(0),h),A],
             2:[(sympy.Integer(0),sympy.Integer(0)),A],
             3:[(a,sympy.Integer(0)),A],
             4:[(a,h),A]}

panels = {(1,2):t,
          (2,3):t,
          (3,4):t}
```

Define section and perform first calculations

```python
S2 = Section(stringers, panels)
```

Verify that the section is open

```python
S2.cycles
```

## Plot of **S2** section in original reference frame

Define a dictionary of coordinates used by **Networkx** to plot section as a Directed graph.
Note that arrows are actually just thicker stubs

```python
start_pos={ii: [float(S2.g.node[ii]['ip'][i].subs(datav)) for i in range(2)] for ii in S2.g.nodes() }
```

```python
plt.figure(figsize=(12,8),dpi=300)
nx.draw(S2.g,with_labels=True, arrows= True, pos=start_pos)
plt.arrow(0,0,20,0)
plt.arrow(0,0,0,20)
#plt.text(0,0, 'CG', fontsize=24)
plt.axis('equal')
plt.title("Section in starting reference Frame",fontsize=16);
```

Expression of **Inertial properties** wrt Center of Gravity in with original rotation

```python
S2.Ixx0, S2.Iyy0, S2.Ixy0, S2.α0
```

## Plot of **S2** section in inertial reference Frame

Section is plotted wrt **center of gravity** and rotated (if necessary) so that *x* and *y* are principal axes.
**Center of Gravity** and **Shear Center** are drawn

```python
positions={ii: [float(S2.g.node[ii]['pos'][i].subs(datav)) for i in range(2)] for ii in S2.g.nodes() }
```

```python
x_ct, y_ct = S2.ct.subs(datav)

plt.figure(figsize=(12,8),dpi=300)
nx.draw(S2.g,with_labels=True, pos=positions)
plt.plot([0],[0],'o',ms=12,label='CG')
plt.plot([x_ct],[y_ct],'^',ms=12, label='SC')
#plt.text(0,0, 'CG', fontsize=24)
#plt.text(x_ct,y_ct, 'SC', fontsize=24)
plt.legend(loc='lower right', shadow=True)
plt.axis('equal')
plt.title("Section in pricipal reference Frame",fontsize=16);
```

Expression of **inertial properties** in *principal reference frame*

```python
S2.Ixx, S2.Iyy, S2.Ixy, S2.θ
```

## **Shear center** expression

```python
S2.ct
```

## Analisys of symmetry properties of the section

For **x** and **y** axes pair of symmetric **nodes** and **edges** are searched for

```python
S2.symmetry
```

    [{'edges': [((1, 2), (3, 4)), (2, 3)], 'nodes': [(1, 4), (2, 3)]},
     {'edges': [(1, 2), (3, 4)], 'nodes': [(1, 2), (3, 4)]}]

## Compute axial loads in Stringers in **S2**

Set **loads** on the section:

**Example 2**: _shear_ in **y** direction and _bending moment_ in **x** direction

```python
S2.set_loads(_Tx=0, _Ty=Ty, _Nz=0, _Mx=Mx, _My=0, _Mz=0)
```

Compute _axial loads_ in **stringers** and _shear flows_ in **panels**

```python
S2.compute_stringer_actions()
S2.compute_panel_fluxes();
```

## Axial loads

```python
S2.N
```

## Shear flows

```python
S2.q
```

Set **loads** on the section:

**Example 2**: _shear_ in **x** direction and _bending moment_ in **y** direction

```python
S2.set_loads(_Tx=Tx, _Ty=0, _Nz=0, _Mx=0, _My=My, _Mz=0)
S2.compute_stringer_actions()
S2.compute_panel_fluxes();
```

```python
S2.N
```

```python
S2.q
```

# Second example (2): Open section

```python
stringers = {1:[(a,h),A],
             2:[(sympy.Integer(0),h),A],
             3:[(sympy.Integer(0),sympy.Integer(0)),A],
             4:[(a,sympy.Integer(0)),A]}

panels = {(1,2):t,
          (2,3):t,
          (3,4):t}
```

Define section and perform first calculations

```python
S2_2 = Section(stringers, panels)
```

## Plot of **S2** section in original reference frame

Define a dictionary of coordinates used by **Networkx** to plot section as a Directed graph.
Note that arrows are actually just thicker stubs

```python
start_pos={ii: [float(S2_2.g.node[ii]['ip'][i].subs(datav)) for i in range(2)] for ii in S2_2.g.nodes() }
```

```python
plt.figure(figsize=(12,8),dpi=300)
nx.draw(S2_2.g,with_labels=True, arrows= True, pos=start_pos)
plt.arrow(0,0,20,0)
plt.arrow(0,0,0,20)
#plt.text(0,0, 'CG', fontsize=24)
plt.axis('equal')
plt.title("Section in starting reference Frame",fontsize=16);
```

Expression of **Inertial properties** wrt Center of Gravity in with original rotation

```python
S2_2.Ixx0, S2_2.Iyy0, S2_2.Ixy0, S2_2.α0
```

## Plot of **S2** section in inertial reference Frame

Section is plotted wrt **center of gravity** and rotated (if necessary) so that *x* and *y* are principal axes.
**Center of Gravity** and **Shear Center** are drawn

```python
positions={ii: [float(S2_2.g.node[ii]['pos'][i].subs(datav)) for i in range(2)] for ii in S2_2.g.nodes() }
```

```python
x_ct, y_ct = S2_2.ct.subs(datav)

plt.figure(figsize=(12,8),dpi=300)
nx.draw(S2_2.g,with_labels=True, pos=positions)
plt.plot([0],[0],'o',ms=12,label='CG')
plt.plot([x_ct],[y_ct],'^',ms=12, label='SC')
#plt.text(0,0, 'CG', fontsize=24)
#plt.text(x_ct,y_ct, 'SC', fontsize=24)
plt.legend(loc='lower right', shadow=True)
plt.axis('equal')
plt.title("Section in pricipal reference Frame",fontsize=16);
```

Expression of **inertial properties** in *principal reference frame*

```python
S2_2.Ixx, S2_2.Iyy, S2_2.Ixy, S2_2.θ
```

## **Shear center** expression

```python
S2_2.ct
```


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Class 6: Pandas and Statsmodels

## Background

In their article "The New Kaldor Facts: Ideas, Institutions, Population, and Human Capital" in the January 2010 issue of the *American Economic Journal: Macroeconomics*, Charles Jones and Paul Romer document six empirical features about macroeconomics that are yet to be adequately explained by a general theory of economic growth (link to article: [https://www.jstor.org/stable/25760291](https://www.jstor.org/stable/25760291)). Among these facts is the observation that total factor productivity is positively correlated with a country's GDP per worker.

Recall the Cobb-Douglas production function with human capital:

\begin{align}
Y & = A K^{\alpha}\left(hL\right)^{1-\alpha}, \label{eqn:production}
\end{align}

where $Y$ denotes the production of final goods and services, $K$ is the stock of physical capital, $L$ is the labor force (either number of workers or number of worker hours), $h$ is human capital per worker, and $A$ is *total factor productivity* or TFP which represents all other factors that might affect production that are not captured by $K$, $L$, or $h$.

For most countries, we can measure $Y$, $K$, $L$ and $h$. Measurements may not be perfect; for example, we often associate $h$ with years of schooling in a country even though, strictly speaking, on-the-job experience and training also contribute to human capital accumulation. Regardless, the point is that the production function *implies* a value for TFP:

\begin{align}
A & = \frac{Y}{K^{\alpha}(hL)^{1-\alpha}}
\end{align}

TFP is like a residual in a linear regression: it is the part of production that is not explained by observable inputs for production. Remarkably, Jones and Romer show that TFP explains a lot of the variation in GDP per worker across countries. This fact is captured in Figure 1.

Figure 1. TFP and GDP per capita across countries  relative to the US. Source: Jones and Romer (2010)

In this Notebook, we will replicate Figure 1 and use a linear regression procedure to estimate the relationship between TFP and GDP per worker. The data required for the exercise is named `cross_country_production.csv` and is available here: https://github.com/letsgoexploring/econ126/raw/master/Data/Csv/cross_country_production.csv
The file contains production data in 2014 for 144 countries. In addition to the name and country code, the file contains production data for 144 countries in 2014:

| Name               | Description               | Units                       |
|--------------------|---------------------------|-----------------------------|
| `gdp`              | real GDP                  | Millions of 2001 US dollars |
| `labor`            | Average number of workers | Millions                    |
| `physical_capital` | Physical capital stock    | Millions of 2001 US dollars |
| `human_capital`    | Human capital per worker  | Index (unitless)            |

We can use the data to compute TFP for each country in the sample and then to replicate Figure 1.

## Replicate Figure from Jones and Romer (2010)

Import the production data. Add new columns containing GDP per worker, physical capital per worker, and TFP. Construct three scatter plots:
1. GDP per worker against physical capital per worker
2. GDP per worker against human capital per worker
3. TFP against GDP per worker

```python
# Import data into a DataFrame called 'df'. Use the leftmost column as the index
df = pd.read_csv('https://github.com/letsgoexploring/econ126/raw/master/Data/Csv/cross_country_production.csv',index_col=0)

# Print the first five rows of the imported data
df.head()
```

      country
      gdp
      labor
      human_capital
      physical_capital

      country_code

      AGO
      Angola
      205371.265625
      8.343885
      1.445598
      7.960541e+05

      ALB
      Albania
      31419.962891
      0.883679
      2.930766
      1.127928e+05

      ARE
      United Arab Emirates
      618051.187500
      5.938807
      2.730336
      2.429822e+06

      ARG
      Argentina
      859908.500000
      18.098450
      2.945663
      2.148251e+06

      ARM
      Armenia
      27410.783203
      1.201936
      3.122178
      4.400342e+04

```python
# Add a column to df called 'gdp_pw' equal to GDP divided by labor divided by 1000 so units are in thousands of $
df['gdp_pw'] = df['gdp']/df['labor']/1000

# Add a column to df called 'physical_capital_pw' equal to physical capital divided by labor divided by 1000 so units are in thousands of $
df['physical_capital_pw'] = df['physical_capital']/df['labor']/1000

# Set alpha to 1/3
alpha =1/3

# Add a column to df called 'tfp' equal to TFP computed using the production function
df['tfp'] = df['gdp']/df['physical_capital']**alpha/(df['labor']*df['human_capital'])**(1-alpha)

# Sort the rows of df so that the GDP per worker column is in ascending order
df = df.sort_values('gdp_pw')
```

```python
# Construct a scatter plot of real GDP per worker against human capital per worker with:
#    1. All countries as blue circles, size 75, opacity (alpha) 0.25, label: 'All countries'
#    2. USA as red square, size 75, opacity (alpha) 0.5, label='USA'
#    3. Add a legend in the upper left corner with scatterpoints=1
#    4. y-axis limits: [-10,250]
#    5. x-axis label: 'Human capital per worker\n(Index)'
#    6. y-axis label: 'Real GDP per worker\n(Thousands of 2001 $)'
#    7. Add a grid if you'd like

fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.scatter(df['human_capital'],df['gdp_pw'],s=75,alpha=0.25,label='All countries')
ax.scatter(df['human_capital'].loc['USA'],df['gdp_pw'].loc['USA'],marker='s',c='r',s=75,alpha=0.5,label='USA')
ax.legend(scatterpoints=1,loc='upper left')
ax.set_ylim([-10,250])
ax.set_xlabel('Human capital per worker\n(Index)')
ax.set_ylabel('Real GDP per worker\n(Thousands of 2001 $)')
ax.grid()
```

```python
# Construct a scatter plot of real GDP per worker against physical capital per worker with:
#    1. All countries as blue circles, size 75, opacity (alpha) 0.25, label: 'All countries'
#    2. USA as red square, size 75, opacity (alpha) 0.5, label='USA'
#    3. Add a legend in the upper left corner with scatterpoints=1
#    4. y-axis limits: [-10,250]
#    5. x-axis label: 'Physical capital per worker\n(Thousands of 2001 $)'
#    6. y-axis label: 'Real GDP per worker\n(Thousands of 2001 $)'
#    7. Add a grid if you'd like

fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.scatter(df['physical_capital_pw'],df['gdp_pw'],s=75,alpha=0.25,label='All countries')
ax.scatter(df['physical_capital_pw'].loc['USA'],df['gdp_pw'].loc['USA'],marker='s',c='r',s=75,alpha=0.5,label='USA')
ax.legend(scatterpoints=1,loc='upper left')
ax.set_ylim([-10,250])
ax.set_xlabel('Physical capital per worker\n(Thousands of 2001 $)')
ax.set_ylabel('Real GDP capital per worker\n(Thousands of 2001 $)')
ax.grid()
```

```python
# Construct a scatter plot of TFP against real GDP per worker with:
#    1. All countries as blue circles, size 75, opacity (alpha) 0.25, label: 'All countries'
#    2. USA as red square, size 75, opacity (alpha) 0.5, label='USA'
#    3. Add a legend in the upper left corner with scatterpoints=1
#    4. x-axis limits: [-10,250]
#    4. y-axis limits: [-10,1600]
#    6. x-axis label: 'Real GDP capital per worker\n(Thousands of 2001 $)'
#    7. y-axis label: 'TFP (Unitless)'
#    8. Add a grid if you'd like

fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.scatter(df['gdp_pw'],df['tfp'],s=75,alpha=0.25,label='All countries')
ax.scatter(df['gdp_pw'].loc['USA'],df['tfp'].loc['USA'],marker='s',c='r',s=75,alpha=0.5,label='USA')
ax.legend(scatterpoints=1,loc='upper left')
ax.set_xlim([-10,250])
ax.set_ylim([-10,1600])
ax.set_xlabel('Real GDP per worker\n(Thousands of 2001 $)')
ax.set_ylabel('TFP (Unitless)')
ax.grid()
```

The strong positive relationship between GDP per worker and TFP means that *even after accounting for their lower levels of human capital per worker and physical capital per worker*, workers in lower-income countries are less productive because they use what human and physical capital the *do* have less efficiently than workers in higher-income countries. And since TFP isn't directly observable, we still don't know exactly why.

## Estimate Relationship Between TFP and GDP per Worker

The Statsmodels module is an excellent tool for a broad range of statistical applications. Learn more about Statsmodels here: https://www.statsmodels.org/stable/index.html. We can use the module to estimate the relationship between TFP and GDP per worker in our data.

Since the scatter plot reveals some curvature in the relationship between TFP and GDP per worker, let's specifiy the linear regression model with the variables in logs:

\begin{align}
\log \text{TFP}_i & = \beta_0 + \beta_1 \log \text{GDP}_i + \epsilon_i
\end{align}

Once we've obtained the estimated coefficients $\hat{\beta}_0$ and $\hat{\beta}_1$, we can compute predicted values for TFP $\widehat{TFP}_i$ using by setting $\epsilon_i=0$ and solving the regression equation for TFP:

\begin{align}
\widehat{TFP}_i & = \exp\left(\hat{\beta}_0 + \hat{\beta}_1 \log \text{GDP}_i\right)
\end{align}

```python
# Import the Statsmodels module
import statsmodels.api as sm
```

```python
# Create a variable called 'y' that will be to the dependent variable in the regression
y = np.log(df['tfp'])

# Create a variable called 'x' that will be to the independent variable in the regression
x = np.log(df['gdp_pw'])

# Use the function sm.add_constant() to add a constant to x
x = sm.add_constant(x)

# Print the first five rows of x
print(x.head())
```

                  const    gdp_pw
    country_code
    CAF             1.0  0.315622
    BDI             1.0  0.611815
    NER             1.0  0.893346
    MWI             1.0  0.913269
    LBR             1.0  0.960224

    /Users/bcjenkin/opt/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py:2495: FutureWarning: Method .ptp is deprecated and will be removed in a future version. Use numpy.ptp instead.
      return ptp(axis=axis, out=out, **kwargs)

```python
# Create an OLS model using Statsmodels
model = sm.OLS(y,x)

# Fit the OLS model to data
results = model.fit()

# Print the summary2() method of results to print summary of regression results:
print(results.summary2())
```

                     Results: Ordinary least squares
    =================================================================
    Model:              OLS              Adj. R-squared:     0.905
    Dependent Variable: tfp              AIC:                -78.3129
    Date:               2020-01-21 15:11 BIC:                -72.3732
    No. Observations:   144              Log-Likelihood:     41.156
    Df Model:           1                F-statistic:        1359.
    Df Residuals:       142              Prob (F-statistic): 1.36e-74
    R-squared:          0.905            Scale:              0.033524
    -------------------------------------------------------------------
                 Coef.    Std.Err.      t      P>|t|    [0.025   0.975]
    -------------------------------------------------------------------
    const        4.1585     0.0463   89.8804   0.0000   4.0670   4.2499
    gdp_pw       0.4936     0.0134   36.8661   0.0000   0.4671   0.5200
    -----------------------------------------------------------------
    Omnibus:              6.608        Durbin-Watson:           1.890
    Prob(Omnibus):        0.037        Jarque-Bera (JB):        6.185
    Skew:                 0.460        Prob(JB):                0.045
    Kurtosis:             3.430        Condition No.:           11
    =================================================================

Estimated coefficients are stored in the `params` attribute of `results`. Residuals of the regression are stored in the attribute `resid`. And predicted values of the dependent variable are stored in `fittedvalues`.

```python
# Print estimated coefficients of linear regression
print(results.params)
```

    const     4.158469
    gdp_pw    0.493573
    dtype: float64

```python
# Print R-squared of the regression
print(results.rsquared)
```

    0.9054032510077362

**Question**

1. What is the interpretation of the coefficient on log GDP per worker?
2. What is the R-squared of the regression? What does it mean?

**Answer**

1. A one percent increase in GDP per worker increases TFP by one half of a percent.
2. The R-squared of the regression is 0.905 and means that GDP per worker explains over 90 percent of the variation in TFP.

```python
# Print the first five rows of residuals.
print(results.resid.head())
```

    country_code
    CAF   -0.368663
    BDI    0.253711
    NER   -0.036212
    MWI   -0.222537
    LBR   -0.093016
    dtype: float64

```python
# Print the mean of the residuals of the regression.
print(results.resid.mean())
```

    2.917419392487217e-15

```python
# Create a variable called 'yhat' equal to the fitted values of y:
yhat = results.fittedvalues

# Print the first five rows of yhat
print(yhat.head())
```

    country_code
    CAF    4.314252
    BDI    4.460445
    NER    4.599401
    MWI    4.609234
    LBR    4.632410
    dtype: float64

Recall that we can use estimated parameters can be used to compute out-of-sample predictions:

\begin{align}
\widehat{TFP}_i & = \exp\left(\hat{\beta}_0 + \hat{\beta}_1 \log \text{GDP}_i\right)
\end{align}

```python
# Create variable called 'gdp_values' that is an array from 0.1 to 300
gdp_values = np.arange(0.1,300,0.001)

# Create variable called 'beta0' equal to the constant of the regression
beta0 = results.params['const']

# Create variable called 'beta1' equal to the coefficient on log GDP per worker from the regression
beta1 = results.params['gdp_pw']

# Create a variable called 'ahat' that is equal to the predicted values of TFP corresponding to gdp_values
ahat = np.exp(beta0 + beta1*np.log(gdp_values))
```

```python
# Recreate the scatter plot of TFP against real GDP per worker with:
#    1. All features and settings from the previous plot
#    2. A plot of ahat against gdp_values; linewidth: 3, opacity: 0.5, label: 'OLS', color: magenta (c='m')

fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.scatter(df['gdp_pw'],df['tfp'],s=75,alpha=0.25,label='All countries')
ax.scatter(df['gdp_pw'].loc['USA'],df['tfp'].loc['USA'],marker='s',c='r',s=75,alpha=0.5,label='USA')
ax.plot(gdp_values,ahat,c='m',lw=3,alpha=0.5,label='OLS')
ax.legend(scatterpoints=1,loc='upper left')
ax.set_xlim([-10,250])
ax.set_ylim([-10,1600])
ax.set_xlabel('Real GDP per worker\n(Thousands of 2001 $)')
ax.set_ylabel('TFP (Unitless)')
ax.grid()
```


## Constraint Handling

### Inequality Constraints

**If somebody is interested in implementing or willing to make contribution(s) regarding constraint handling of inequality constraints please let us know.
The G problem is suite is already available to experiment with different algorithms. So far, mostly parameter-less constraint handling is used for our algorithms.**

### Equality Constraints

We got a couple of questions of how equality constraints should be handled in a genetic algorithm. In general, functions without any smoothness are challenging to handle for genetic algorithms. An equality constraint is basically an extreme case, where the constraint violation is 0 at exactly one point and otherwise 1.

Let us consider the following constraint $g(x)$ where $x$ represents a variable:

$g(x): x = 5$

An equality constraint can be expressed by an inequality constraint:

$g(x): |x - 5| \leq 0$

or

$g(x): (x-5)^2   \leq 0$

However, all of constraints above are very strict and make **most of the search space infeasible**. Without providing more information to the algorithm those constraint are very difficult to satisfy.

For this reason, the constraint can be smoothed by adding an epsilon to it and, therefore, having two inequality constraints:

$g'(x): 5 - \epsilon \leq x \leq 5 + \epsilon$

Also, it can be simply expressed in one inequality constraint by:

$g'(x): (x-5)^2  - \hat{\epsilon} \leq 0$

Depending on the $\epsilon$ the solutions will be more or less close to the desired value. However, the genetic algorithm does not know anything about the problem itself which makes it difficult to handle and focus the search in the infeasible space.

**Constraint Handling Through Repair**

A simple approach is to handle constraints through a repair function. This is only possible if the equation of the constraint is known. The repair makes sure every solution that is evaluated is in fact feasible. Let us consider the following example where
the equality constraints need to consider more than one variable:

\begin{align}
\begin{split}
\min \;\; & f_1(x) = (x_1^2 + x_2^2) \\
\max \;\; & f_2(x) = -(x_1-1)^2 - x_2^2 \\[1mm]
\text{s.t.} \;\; & g_1(x_1, x_3) : x_1 + x_3 = 2\\[2mm]
& -2 \leq x_1 \leq 2 \\
& -2 \leq x_2 \leq 2 \\
& -2 \leq x_3 \leq 2
\end{split}
\end{align}

We implement the problem using by squaring the term and using an $\epsilon$ as we have explained above. The source code for the problem looks as follows:

```python
import numpy as np

from pymoo.model.problem import Problem

class MyProblem(Problem):

    def __init__(self):
        super().__init__(n_var=3,
                         n_obj=2,
                         n_constr=1,
                         xl=np.array([-2, -2, -2]),
                         xu=np.array([2, 2, 2]))

    def _evaluate(self, x, out, *args, **kwargs):
        f1 = x[:, 0] ** 2 + x[:, 1] ** 2
        f2 = (x[:, 0] - 1) ** 2 + x[:, 1] ** 2
        g1 = (x[:, 0] + x[:, 2] - 2) ** 2 - 1e-5

        out["F"] = np.column_stack([f1, f2])
        out["G"] = g1

```

As you might have noticed the problem has similar characteristics to problem in our getting started.
Before a solution is evaluated a repair function is called. To make sure a solution is feasible, an approach would be to either set $x_3 = 2 - x_1$ or $x_1 = 2 - x_3$. Additionally, we need to consider that this repair might produce a variable to be out of bounds.

```python
from pymoo.model.repair import Repair

class MyRepair(Repair):

    def _do(self, problem, pop, **kwargs):
        for k in range(len(pop)):
            x = pop[k].X
            if np.random.random() < 0.5:

                x[2] = 2 - x[0]
                if x[2] > 2:
                    val = x[2] - 2
                    x[0] += val
                    x[2] -= val

            else:
                x[0] = 2 - x[2]
                if x[0] > 2:
                    val = x[0] - 2
                    x[2] += val
                    x[0] -= val

        return pop
```

Now the algorithm object needs to be initialized with the repair operator and then can be run to solve the problem:

```python
from pymoo.algorithms.nsga2 import NSGA2

algorithm = NSGA2(pop_size=100, repair=MyRepair(), eliminate_duplicates=True)

from pymoo.optimize import minimize
from pymoo.visualization.scatter import Scatter

res = minimize(MyProblem(),
               algorithm,
               ('n_gen', 20),
               seed=1,
               verbose=True)

plot = Scatter()
plot.add(res.F, color="red")
plot.show()
```

In our case it is easy to verify if the constraint is violated or not:

```python
print(res.X[:, 0] + res.X[:, 2])
```

    [2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2.
     2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2.
     2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2.
     2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2.
     2. 2. 2. 2.]

If you would like to compare the solution without a repair you will see how searching only in the feasible space helps:

```python
algorithm = NSGA2(pop_size=100, eliminate_duplicates=True)

res = minimize(MyProblem(),
               algorithm,
               ('n_gen', 20),
               seed=1,
               verbose=True)

plot = Scatter()
plot.add(res.F, color="red")
plot.show()

```

```python
print(res.X[:, 0] + res.X[:, 2])
```

    [1.99825371]

Here in fact the $\epsilon$ term is necessary to find any feasible solution at all.


# Deterministic Variational Inference: Toy example
Code to accompany [**Fixing Variational Bayes: Deterministic Variational Inference for Bayesian Neural Networks**](https://arxiv.org/abs/1810.03958)

```python
!rm -rf deterministic-variational-inference
```

```python
!git clone https://github.com/EmptyJackson/deterministic-variational-inference
%cd deterministic-variational-inference/
```

    Cloning into 'deterministic-variational-inference'...
    remote: Enumerating objects: 10, done.
    remote: Counting objects: 100% (10/10), done.
    remote: Compressing objects: 100% (9/9), done.
    remote: Total 33 (delta 1), reused 5 (delta 1), pack-reused 23
    Unpacking objects: 100% (33/33), done.
    /content/deterministic-variational-inference

```python
import subprocess
import glob

!mkdir frames
vid_folder = "frames"
```

```python
%matplotlib inline

import numpy as np
import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()

import matplotlib.pyplot as plt
from IPython.display import clear_output
import os, json

import gaussian_variables as gv
import utils as u
import plot_utils as pu
import bayes_layers as bnn
from bayes_models import MLP, PointMLP, AdaptedMLP

np.random.seed(3)
```

    WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
    Instructions for updating:
    non-resource variables are not supported in the long term

```python
print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))
```

    Num GPUs Available:  1

## Dataset

First we generate a toy dataset according to:

\begin{equation}
    y = -(x+0.5)\sin(3\pi x) + \eta
\end{equation}

Where the noise is generated according to:

\begin{equation}
    \eta = 0.45(x + 0.5)^2
\end{equation}

```python
def base_model(x):
    return -(x+0.5)*np.sin(3 * np.pi *x)

def noise_model(x):
    return 0.45*(x+0.5)**2

def sample_data(x):
    return base_model(x) + np.random.normal(0, noise_model(x))

data_size = {'train': 500, 'valid': 100, 'test': 100}
toy_data = []
for section in ['train', 'valid', 'test']:
    x = (np.random.rand(data_size['train'], 1) - 0.5)
    toy_data.append([x, sample_data(x).reshape(-1)])
x = np.arange(-1,1,1/100)
toy_data.append([[[_] for _ in x], base_model(x)])

pu.toy_results_plot(toy_data, {'mean':base_model, 'std':noise_model})
```

# Model

Here we set the hyperparameters and provide a function for constructing a Bayesian Neural Network.

```python
hypers = {
    "x_dim": 1,                                "y_dim": 2,
    "hidden_dims": [128,128],                  "nonlinearity": "relu",
    "adapter": {
        'in' : {"scale": [[1.0]], "shift": [[0.0]]},
        'out': {"scale": [[1.0, 0.83]], "shift": [[0.0, -3.5]]}
    },

    "method": "bayes",                         "style": "heteroskedastic",
    "homo_logvar_scale": 2*np.log(0.2),        "prior_type": ["empirical", "wider_he", "wider_he"],

    "n_epochs": 7500,                         "batch_size": 500,
    "learning_rate": 0.001,                    "lambda": 1.0,
    "warmup_updates": {'lambda': 14000.0},     "anneal_updates": {'lambda': 1000.0},
    "optimizer": "adam",                       "gradient_clip": 0.1,

    "data_fraction": 1.0,                      "sections_to_run": ["train"]
}
```

```python
def make_model(hypers):
    if hypers['method'].lower().strip() == 'bayes':
        MLP_factory = MLP
        prediction = lambda y: tf.reshape(y.mean[:,0], [-1])
        loss = bnn.regression_loss
    else:
        MLP_factory = PointMLP
        prediction = lambda y: tf.reshape(y.mean[:,0], [-1])
        loss = bnn.point_regression_loss

    mlp = MLP_factory(hypers['x_dim'], hypers['y_dim'], hypers)
    mlp = AdaptedMLP(mlp)
    mlp.make_placeholders()
    ipt = mlp.placeholders['ipt_mean']
    y = mlp(ipt)

    target = tf.placeholder(tf.float32, [None])
    mlp.placeholders['target'] = target
    global_step = tf.Variable(0, trainable=False, name='global_step')
    loss, logprob, all_surprise = loss(y, target, mlp, hypers, global_step)

    accuracy = tf.reduce_mean(tf.abs(target - prediction(y)))

    return {
        'model': mlp,
        'metrics': {
            'accuracy': accuracy,   'loss': loss,
            'logprob': logprob,     'all_surprise': all_surprise
            },
        'global_step': global_step}
```

## Training loop

For the first `warmup_updates` epochs we run with the KL term in the elbo turned off. You will observe that the model starts to overfit the data. The KL term is then ramped back on and the model is trained with the full ELBO to convergence.

```python
run_id = u.start_run()

data = toy_data

restricted_training_set = u.restrict_dataset_size(data[0], hypers['data_fraction'])
hypers['dataset_size'] = len(restricted_training_set[0])

device_id = 0
device_string = u.get_device_string(device_id)
print(hypers)
with tf.device(device_string):
    if True:
        model_and_metrics = make_model(hypers)
        train_op = u.make_optimizer(model_and_metrics, hypers)
        sess = u.get_session()
        saver = tf.train.Saver()

        all_summaries = []
        best_valid_accuracy = np.inf

    for epoch in range(hypers['n_epochs']):
        verbose = (epoch % 20 == 0)
        if verbose:
            print("Epoch %i:        " % epoch, end='')
        epoch_summary, accuracies = u.train_valid_test(
            {
                'train': restricted_training_set,
                'valid': data[1],
                'test': data[2]
            },
            sess, model_and_metrics, train_op, hypers, verbose)

        if epoch % 1000 == 0:
            clear_output()

        frame_break = 50
        if epoch % frame_break == 0 and epoch != 0:
            predictions = u.get_predictions(data[3], sess, model_and_metrics['model'], hypers)
            pu.toy_results_plot(toy_data, {'mean':base_model, 'std':noise_model}, hypers, predictions)
            plt.savefig(vid_folder + "/file%02d.png" % (epoch/frame_break), dpi=200)
            plt.clf()
            #plt.show()

        # dump log file
        all_summaries.append(epoch_summary)
        #with open(os.path.join("logs", run_id, "results.json"), 'w') as f:
        #    json.dump(all_summaries, f, indent=4, cls=u.NumpyEncoder)

os.chdir(vid_folder)
subprocess.call([
    'ffmpeg', '-framerate', '15', '-i', 'file%02d.png', '-r', '30', '-pix_fmt', 'yuv420p',
    'train.mp4'
])
for file_name in glob.glob("*.png"):
    os.remove(file_name)
```

    Epoch 7020:         train accuracy = 0.1150 | logprob = 0.9254 | KL term = 15.3170673828125
    Epoch 7040:         train accuracy = 0.1149 | logprob = 0.9309 | KL term = 15.328740234375
    Epoch 7060:         train accuracy = 0.1149 | logprob = 0.9362 | KL term = 15.3406376953125
    Epoch 7080:         train accuracy = 0.1148 | logprob = 0.9414 | KL term = 15.35257421875
    Epoch 7100:         train accuracy = 0.1148 | logprob = 0.9465 | KL term = 15.3643720703125
    Epoch 7120:         train accuracy = 0.1148 | logprob = 0.9515 | KL term = 15.3759638671875
    Epoch 7140:         train accuracy = 0.1147 | logprob = 0.9564 | KL term = 15.3873935546875
    Epoch 7160:         train accuracy = 0.1147 | logprob = 0.9612 | KL term = 15.398646484375
    Epoch 7180:         train accuracy = 0.1147 | logprob = 0.9659 | KL term = 15.4097412109375
    Epoch 7200:         train accuracy = 0.1146 | logprob = 0.9704 | KL term = 15.4207041015625
    Epoch 7220:         train accuracy = 0.1146 | logprob = 0.9748 | KL term = 15.4315185546875
    Epoch 7240:         train accuracy = 0.1146 | logprob = 0.9791 | KL term = 15.44221484375
    Epoch 7260:         train accuracy = 0.1146 | logprob = 0.9833 | KL term = 15.4528203125
    Epoch 7280:         train accuracy = 0.1145 | logprob = 0.9874 | KL term = 15.46330078125
    Epoch 7300:         train accuracy = 0.1145 | logprob = 0.9913 | KL term = 15.473619140625
    Epoch 7320:         train accuracy = 0.1145 | logprob = 0.9953 | KL term = 15.4837490234375
    Epoch 7340:         train accuracy = 0.1145 | logprob = 0.9991 | KL term = 15.4937646484375
    Epoch 7360:         train accuracy = 0.1145 | logprob = 1.0029 | KL term = 15.5036796875
    Epoch 7380:         train accuracy = 0.1145 | logprob = 1.0066 | KL term = 15.5135009765625
    Epoch 7400:         train accuracy = 0.1144 | logprob = 1.0103 | KL term = 15.52321875
    Epoch 7420:         train accuracy = 0.1144 | logprob = 1.0139 | KL term = 15.5328427734375
    Epoch 7440:         train accuracy = 0.1144 | logprob = 1.0175 | KL term = 15.5423984375
    Epoch 7460:         train accuracy = 0.1144 | logprob = 1.0211 | KL term = 15.5518896484375
    Epoch 7480:         train accuracy = 0.1144 | logprob = 1.0246 | KL term = 15.561322265625

    <Figure size 432x288 with 0 Axes>

## Comparison with MC

We compare the output activations of the BNN as predicted using our deterministic approximation with the activations predicted by Monte Carlo approximation (see Figure 3 of the paper)

```python
from matplotlib import rc
rc('text', usetex=False)

n_samp = 20000
x=0.25
ipt = [[[x]] for _ in range(n_samp)]
sample_op = model_and_metrics['model'].run_with_MC(ipt, n_sample=n_samp)
approx_op = model_and_metrics['model'](x)

samples = sess.run(sample_op)
approx = sess.run([approx_op.mean, approx_op.var])

def gaussian1d(mean, var):
    x_axis = np.linspace(-5,5,1000)
    return x_axis, 1.0 / np.sqrt(2.0 * np.pi * var) * np.exp(-1.0 / (2.0 * var) * (x_axis - mean)**2)

samples.shape
plt.figure(figsize=(3,3))
plt.hist(samples[:,0,0], int(np.round(15/0.5)), density=True, edgecolor='k', facecolor='#b4c7e7')
plt.plot(*gaussian1d(approx[0][0,0], approx[1][0,0,0]), 'b')
plt.xlim([-0.8,-0.5])
plt.yticks([])
plt.xlabel('$m$')
plt.ylabel('$q(m)$')
plt.savefig("empirical_m.png", dpi=100, bbox_inches="tight")
plt.show()

plt.figure(figsize=(3,3))
plt.hist(samples[:,0,1], int(np.round(15/0.9)), density=True, edgecolor='k', facecolor='#b4c7e7', label="MC")
plt.plot(*gaussian1d(approx[0][0,1], approx[1][0,1,1]), 'b', label="DVI")
plt.xlim([-3.2,-2.3])
plt.yticks([])
plt.xlabel('$\ell$')
plt.ylabel('$q(\ell)$')
plt.legend()
plt.savefig("empirical_l.png", dpi=100, bbox_inches="tight")
plt.show()
```

```python

```


# ***Introduction to Radar Using Python and MATLAB***
## Andy Harrison - Copyright (C) 2019 Artech House
<br/>

# Rounded Nose Cone Radar Cross Section
***

Referring to Section 7.4.1.6, for cones with rounded nose tips, as shown in Figure 7.12, the physical optics approximation with axial incidence is given by (Equation 7.55)

$$
    \sigma = \pi b^2\Bigg[ 1 - \frac{\sin\big(2kb(1-\sin\alpha)\big)}{kb\cos^2\alpha} + \frac{1 + \cos^4\alpha}{4(kb)^2\cos^4\alpha}  - \frac{\cos\big(2kb(1-\sin\alpha)\big)}{2(kb^2)\cos^2\alpha} \Bigg] \hspace{0.5in} \text{(m}^2\text{)},
$$

where $b$ is the radius of the rounded nose tip. For incident angles other than axial, but less than the cone angle ($\theta_i \le \alpha$),  (Equation 7.56)

$$
    \sigma =  \pi b^2 \frac{1 + \theta_i^2}{4(kb)^2} \Big[A_1 + A_2 \cos\big(2k\cos\theta_i (1 - \sin\alpha)\big)  + A_3\sin\big(2k\cos\theta_i(1-\sin\alpha)\big)\Big]\hspace{0.25in} \text{(m}^2\text{)},
$$

where

\begin{align}
    A_1 = \; &2 + 2\alpha^2 - 2\theta_i^2 + \alpha^4 - \alpha^2\theta_i^2 + 0.5\theta_i^4 + 2\alpha^4\theta_i^2 + 4(kb)^2 - 2(kb)^2\theta_i^2 \nonumber \\[5pt]
    &- 8(kb)^3\theta_i^2 + (kb)^2\theta_i^4 + 6(kb)^2\alpha^2\theta_i^2 + 8(kb)^3\theta_i^4 + 13(kb)^4\theta_i^4, \\  \nonumber \\
    A_2 = &-2 -2\alpha^2 + 2\theta_i^2 + \alpha^2\theta_i^2 - 0.5\theta_i^4 - 6(kb)^2\theta_i^2 + 8(kb)^4\theta_i^3 + 3(kb)^2\theta_i^4, \\  \nonumber \\
    A_3 = &-4\big(1 + \alpha^2 - 0.5\theta_i^2 + 3(kb\theta_i)^2\big)\big(kb - kb\theta_i^2 - (kb\theta_i)^2\big) - 4(kb\theta_i)^3.\nonumber
\end{align}
***

Begin by getting the library path

```python
import lib_path
```

Set the operating frequency (Hz), the cone half angle (radians), and the nose radius (m)

```python
from numpy import radians

frequency = 1e9

cone_half_angle = radians(20.0)

nose_radius = 1.4
```

Set the incident angles using the `linspace` routine from `scipy`

```python
from numpy import linspace

incident_angle = linspace(0, cone_half_angle, 1801)
```

Calculate the radar cross section (m^2) for the rounded nose cone

```python
from Libs.rcs.rounded_nose_cone import radar_cross_section

from numpy import array

rcs = array([radar_cross_section(frequency, cone_half_angle, nose_radius, ia) for ia in incident_angle])
```

Display the radar cross section (m^2) for the rounded nose cone using the routines from `matplotlib`

```python
from matplotlib import pyplot as plt

from numpy import log10, degrees

# Set the figure size

plt.rcParams["figure.figsize"] = (15, 10)

# Display the results

plt.plot(degrees(incident_angle), 10.0 * log10(abs(rcs)), '')

# Set the plot title and labels

plt.title('RCS vs Incident Angle', size=14)

plt.ylabel('RCS (dBsm)', size=12)

plt.xlabel('Incident Angle (deg)', size=12)

# Set the tick label size

plt.tick_params(labelsize=12)

# Turn on the grid

plt.grid(linestyle=':', linewidth=0.5)
```

```python

```


```python
import cirq
```

We are now going to explore the quantum Fourier transformation, which is an important ingredient for many quantum algorithms with superpolynomial speed-up.

Rading Ch 7.1, RC 19642 (07/12/94) (arXiv:quant-ph/0201067)

Before specifying the algorithm, we want to define the desired outcome and also establish some notation.
We define the Quantum Fourier Transform (QFT) with respect to the basis states $\{|0\rangle\},|1\rangle, ..., |m-1\rangle$, where we now understand $|y\rangle$ as the state corresponding to state $|\mathbf{y}\rangle$ where $\mathbf{y}$ is the binary encoding of $y$, in the following way
\begin{definition}
\begin{equation}
\mathrm{QFT}_m: |x\rangle \rightarrow \frac{1}{\sqrt{m}}\sum^{m-1}_{y=0}e^{2\pi i \frac{x}{m} y }|y\rangle
\end{equation}
\end{definition}
Here we understand $xy$ as normal integer multiplication. In order to understand how this translates to the binary setting of qubit system let us also sepcify, how we understand a general phase $\omega \in [0,1)$. Here we view $\omega$ as written in binary, i.e. $\omega = 0.x_1x_2x_3... = \sum 2^{-i}x_i $. This immediedtly leads to the identity
\begin{equation}
e^{2\pi i 2^k \omega} =e^{2\pi i  x_1x_2...x_k.x_{k+1}...} = e^{0.x_{k+1}x_{k+2}...}
\end{equation}
Lastly we want to state a useful identity, which also hopefully clearifies the notation further. Before we state the identity we note that $\sum^n_{k=0} 2^k = 2^n-1$ and note that to encode numbers up to $2^{n-1}$ in binary we need $n$-bits. This allows us now for example to write $e^{2\pi i \omega 2^n-1}| 2^n-1\rangle = e^{2\pi i \omega 2^n-1}|1,1,...1\rangle = (e^{2\pi i \omega 2^{n-1}}|1\rangle) \otimes  (e^{2\pi i \omega 2^{n-2}}|1\rangle) \otimes...(e^{2\pi i \omega}|1\rangle) $ in the same way we can now rewrite all states as products, (just to illustrate the point a bit further $e^{2\pi i \omega 2^n-2}| 2^n-2\rangle =  (e^{2\pi i \omega 2^{n-1}}|1\rangle) \otimes  (e^{2\pi i \omega 2^{n-2}}|1\rangle) \otimes...|0\rangle)$) which then immediatley leads to the identity
\begin{equation}
\frac{1}{\sqrt{2}}\sum^{2^n-1}_{y = 0} e^{2\pi i \omega y }|y\rangle =  \frac{1}{\sqrt{2}} \left(|0\rangle +(e^{2\pi i \omega 2^{n-1}}|1\rangle)\right)\otimes\left(|0\rangle +(e^{2\pi i \omega 2^{n-2}}|1\rangle)\right)\otimes...\otimes \left(|0\rangle +e^{2\pi i \omega}|1\rangle\right)
\end{equation}
this can now be further simplified using (2).

\begin{equation}
\frac{1}{\sqrt{2}}\sum^{2^n-1}_{y = 0} e^{2\pi i \omega y }|y\rangle =\frac{1}{\sqrt{2^n}} \left(|0\rangle+e^{2\pi i 0.x_nx_{n+1}...}|1\rangle \right)\otimes \left(|0\rangle+e^{2\pi i 0.x_{n-1}x_{n}...}|1\rangle \right)\otimes...\otimes \left(|0\rangle+e^{2\pi i 0.x_{1}x_{2}...}|1\rangle \right)
\end{equation}
We can now see that the QFT in (1) can be understood as the direct product in (4). We can now devise a strategy to implement the QFT. If we start out with n registers in the state $|x\rangle = |x_1x_2...x_n\rangle =|x_1\rangle \otimes |x_2\rangle \otimes...\otimes |x_n\rangle $ we want to build the circuit in such a way, that $|x_i\rangle \rightarrow \frac{1}{\sqrt{2}}\big(|0\rangle + e^{2\pi i 0.x_ix_{i+1},...x_n}|1\rangle\big)$, which then in turn gives (4)

Before turning to the details of the curcuit implementing the QFT we want to introduced the controlled phase rotation operator, which is defined as
\begin{equation}
cR_\phi = |0\rangle\langle 0| \otimes I + |0\rangle\langle 0| \otimes \begin{bmatrix}
1 & 0\\
0 & e^{i\phi}
\end{bmatrix}
\end{equation}
where  $R_\phi$ is also expressed in the computational basis. We will write $R_n$ to refer to $\phi= 2^{-n}$

We are now able to devise a strategy for implementing the $QFT_{2^n}$. We start by applying a Hadamard gate to every register, which leads to
\begin{equation}
H |x_i\rangle = \frac{1}{\sqrt{2}} \left(|0\rangle + (-1)^{x_i}\right|1\rangle) = \frac{1}{\sqrt{2}} \left(|0\rangle + e^{\pi i x_i}\right|1\rangle) = \frac{1}{\sqrt{2}} \left(|0\rangle + e^{2\pi i 0.x_i}\right|1\rangle)
\end{equation}
This shows that we already have the right first element for each phase. If we now apply a decreasing controlled phase gates (i.e. $cR_2, \, cR_3,\, ...,cR_{n-{i+1}}$) controlled by all the registers below $i$ ($i+1,...,n$) it is easy to see that we obtain exactly the right factors in (4), just in reverse order.

```python
#input x list in bit format
def qft(x):
    n = len(x)
    c = cirq.Circuit()
    ##initialise n qubits
    q_l = cirq.LineQubit.range(n)
    # bring qubits into state x
    c.append([cirq.X(q_l[i]) for i in range(n) if x[i]])
    # add H gates and add phase gates note that here the phase of z is \pi hence the
    #index is shifted by one e.g. R_2 corresponds to 2**{-1}
    for i in range(n):
        c.append([cirq.H(q_l[i])])
        c.append([(cirq.Z(q_l[i])**(2**(-(j-i)))).controlled_by(q_l[j]) for j in range(i+1,n)])
    return c
### fix order
```

```python
qft([0,1,1])
```

              ┌──┐
0: ───H───S────T─────────────
          │    │
1: ───X───@────┼H────S───────
               │     │
2: ───X────────@─────@───H───
              └──┘

```python
s=cirq.Simulator()
c = qft([0,1,1])
```

```python
s.simulate(c)
```

    measurements: (no measurements)
    output vector: 0.354|000⟩ - 0.354|001⟩ - 0.354j|010⟩ + 0.354j|011⟩ + (-0.25+0.25j)|100⟩ + (0.25-0.25j)|101⟩ + (0.25+0.25j)|110⟩ + (-0.25-0.25j)|111⟩

```python
import numpy as np
```

```python
for y in range(0,2**3):
    print(bin(y))
    print(np.exp(2*np.pi*1j*11*y/2**3)/np.sqrt(2**3))
```

    0b0
    (0.35355339059327373+0j)
    0b1
    (-0.2499999999999997+0.2500000000000003j)
    0b10
    (-8.661748989382706e-16-0.35355339059327373j)
    0b11
    (0.2500000000000009+0.24999999999999906j)
    0b100
    (-0.35355339059327373+1.7323497978765411e-15j)
    0b101
    (0.24999999999999847-0.2500000000000015j)
    0b110
    (2.5985246968148118e-15+0.35355339059327373j)
    0b111
    (-0.2500000000000021-0.24999999999999784j)

```python
#Note that we have to relabel as mentoined earlier, e.g. |100> <-> 001 etc.
#This is to follow Ch.7, one could of course also just implement the circuit "upside down"
```


## Chapter 3

### Notes
* Python functions are written with very narrow scope: they only address the individual problem-at-hand and _probably_ won't generalize.

```python
import sys
from importlib import reload
sys.path.append("..")
```

```python
from sympy import *
init_printing(use_latex='mathjax')
s = symbols('s')
import Ch3.utilities as ch3_utils
reload(ch3_utils)
```

    <module 'Ch3.utilities' from '../Ch3/utilities.py'>

### Problem 3.1

#### Notes
* The `sympy` package does algebra symbolically (check the `environment.yml` file)
* The state transition matrix, STM, is represented by $\Phi(t)$
* The helper function `computeResolvent` (see Ch3/utilities.py) does the heavy-lifting

```python
def helper3p1(A):
    res = ch3_utils.computeResolvent(A)
    pprint('Matrix A')
    pprint(A)
    pprint('has resolvent')
    pprint(res)
```

#### a)

```python
A = Matrix([[-1, 0, 0],[1, -2, 0], [1, 2, -3]])
helper3p1(A)
```

    Matrix A
    ⎡-1  0   0 ⎤
    ⎢          ⎥
    ⎢1   -2  0 ⎥
    ⎢          ⎥
    ⎣1   2   -3⎦
    has resolvent
    ⎡              1                                      ⎤
    ⎢            ─────                     0           0  ⎥
    ⎢            s + 1                                    ⎥
    ⎢                                                     ⎥
    ⎢           1       1                  1              ⎥
    ⎢       - ───── + ─────              ─────         0  ⎥
    ⎢         s + 2   s + 1              s + 2            ⎥
    ⎢                                                     ⎥
    ⎢    1         2         3          2       2      1  ⎥
    ⎢───────── - ───── + ─────────  - ───── + ─────  ─────⎥
    ⎣2⋅(s + 3)   s + 2   2⋅(s + 1)    s + 3   s + 2  s + 3⎦

The inverse Laplace transform of the resolvent of A is:
\begin{eqnarray}
\Phi(t) = \begin{pmatrix}
e^{-t} & 0 & 0 \\
e^{-t} - e^{-2t} & e^{-2t} & 0 \\
\frac{1}{2}e^{-3t} - 2 e^{-2t} + \frac{3}{2} e^{-t} & 2(e^{-2t} - e^{-3t}) & e^{-3t} \\
\end{pmatrix}
\end{eqnarray}

#### b)

```python
A = Matrix([[-1, 0, 0],[1, -1, 0], [0, 1, -1]])
helper3p1(A)
```

    Matrix A
    ⎡-1  0   0 ⎤
    ⎢          ⎥
    ⎢1   -1  0 ⎥
    ⎢          ⎥
    ⎣0   1   -1⎦
    has resolvent
    ⎡   1                     ⎤
    ⎢ ─────       0        0  ⎥
    ⎢ s + 1                   ⎥
    ⎢                         ⎥
    ⎢   1         1           ⎥
    ⎢────────   ─────      0  ⎥
    ⎢       2   s + 1         ⎥
    ⎢(s + 1)                  ⎥
    ⎢                         ⎥
    ⎢   1         1        1  ⎥
    ⎢────────  ────────  ─────⎥
    ⎢       3         2  s + 1⎥
    ⎣(s + 1)   (s + 1)        ⎦

The inverse Laplace transform of the resolvent of A is:
\begin{eqnarray}
\Phi(t) = \begin{pmatrix}
e^{-t} & 0 & 0 \\
t e^{-t} & e^{-t} & 0 \\
\frac{t^2}{2} e^{-t} & t e^{-t} & e^{-t} \\
\end{pmatrix}
\end{eqnarray}
The terms involving repeated roots require using the shift property of Laplace transforms, namely that
\begin{eqnarray}
\mathcal{L}^{-1} \bigl [ F(s-a) \bigr ] = e^{at} \mathcal{L}^{-1} \bigl [ F(s) \bigr ]
\end{eqnarray}

#### c)

```python
A = Matrix([[-2, 1, 1],[1, -2, 1], [1, 1, -2]])
helper3p1(A)
```

    Matrix A
    ⎡-2  1   1 ⎤
    ⎢          ⎥
    ⎢1   -2  1 ⎥
    ⎢          ⎥
    ⎣1   1   -2⎦
    has resolvent
    ⎡     2        1          1        1         1        1 ⎤
    ⎢ ───────── + ───   - ───────── + ───  - ───────── + ───⎥
    ⎢ 3⋅(s + 3)   3⋅s     3⋅(s + 3)   3⋅s    3⋅(s + 3)   3⋅s⎥
    ⎢                                                       ⎥
    ⎢      1        1        2        1          1        1 ⎥
    ⎢- ───────── + ───   ───────── + ───   - ───────── + ───⎥
    ⎢  3⋅(s + 3)   3⋅s   3⋅(s + 3)   3⋅s     3⋅(s + 3)   3⋅s⎥
    ⎢                                                       ⎥
    ⎢      1        1         1        1        2        1  ⎥
    ⎢- ───────── + ───  - ───────── + ───   ───────── + ─── ⎥
    ⎣  3⋅(s + 3)   3⋅s    3⋅(s + 3)   3⋅s   3⋅(s + 3)   3⋅s ⎦

The inverse Laplace transform of the resolvent of A is:
\begin{eqnarray}
\Phi(t) = \frac{1}{3} \begin{pmatrix}
2 e^{-3t} + 1 & 1 - e^{-3t} & 1 - e^{-3t} \\
1 - e^{-3t} & 2 e^{-3t} + 1 & 1 - e^{-3t} \\
1 - e^{-3t} & 1 - e^{-3t} & 2 e^{-3t} + 1 \\
\end{pmatrix}
\end{eqnarray}

### Problem 3.2

#### Notes
* The `sympy` package does algebra symbolically, as in previous problem
* The helper function `firstCompanionForm` computes the first companion form described by Eqns. 3.88 and 3.94
* The helper function `PFDforJordanForm` computes the partial fraction decomposition of an input transfer function.  This makes the Jordan form easier to calculate by Eqns. 3.108 and 3.116

```python
def helper3p2(num, den):
    pprint('Transfer function H(s):')
    pprint(factor(num)/factor(den))
    pprint('First Companion Form matrices A, B, C, D:')
    A, B, C, D = ch3_utils.firstCompanionForm(num, den)
    pprint(A)
    pprint(B)
    pprint(C)
    pprint(D)
    try:
        A, B, C, D = ch3_utils.jordanForm(num, den)
        pprint('Jordan (canonical) Form matrices A, B, C, D:')
        pprint(A)
        pprint(B)
        pprint(C)
        pprint(D)
    except NotImplementedError:
        pprint('Jordan form condition not implemented.')
        pprint('Here is the partial fraction decomposition to compute Jordan form manually.')
        num = factor(num)
        den = factor(den)
        pd = apart(num/den)
        pprint(pd)
```

#### a)

```python
num = expand((s+2)*(s+4)).as_poly()
den = expand((s+1)*(s+3)*(s+5)).as_poly()
helper3p2(num, den)
```

    Transfer function H(s):
        (s + 2)⋅(s + 4)
    ───────────────────────
    (s + 1)⋅(s + 3)⋅(s + 5)
    First Companion Form matrices A, B, C, D:
    ⎡ 0    1   0 ⎤
    ⎢            ⎥
    ⎢ 0    0   1 ⎥
    ⎢            ⎥
    ⎣-15  -23  -9⎦
    ⎡0⎤
    ⎢ ⎥
    ⎢0⎥
    ⎢ ⎥
    ⎣1⎦
    [8  6  1]
    [0]
    3/(8*(s + 5)) + 1/(4*(s + 3)) + 3/(8*(s + 1))
    Jordan (canonical) Form matrices A, B, C, D:
    ⎡-3  0   0 ⎤
    ⎢          ⎥
    ⎢0   -1  0 ⎥
    ⎢          ⎥
    ⎣0   0   -5⎦
    ⎡1⎤
    ⎢ ⎥
    ⎢1⎥
    ⎢ ⎥
    ⎣1⎦
    [1/4  3/8  3/8]
    [0]

There are no repeated roots, so the Jordan form (Eqn. 3.108) is:
\begin{eqnarray}
A &=& \begin{pmatrix}
-5 & 0 & 0 \\
0 & -3 & 0 \\
0 & 0 & -1 \\
\end{pmatrix}, \
B = \begin{pmatrix}
1 & 1 & 1 \\
\end{pmatrix}^T, \\
C &=& \frac{1}{8} \begin{pmatrix}
3 & 2 & 3
\end{pmatrix}, \
D = \begin{pmatrix}
0
\end{pmatrix}
\end{eqnarray}

_NOTE_: the form above is a bit different than the one returned by `jordanForm`.  A simple transformation to reorder the poles from smallest to largest was done offline because it was easiest.

#### b)

```python
num = expand((s+2)).as_poly()
den = expand(s*((s+1)**2 + 4)).as_poly()
helper3p2(num, den)
```

    Transfer function H(s):
         s + 2
    ────────────────
      ⎛ 2          ⎞
    s⋅⎝s  + 2⋅s + 5⎠
    First Companion Form matrices A, B, C, D:
    ⎡0  1   0 ⎤
    ⎢         ⎥
    ⎢0  0   1 ⎥
    ⎢         ⎥
    ⎣0  -5  -2⎦
    ⎡0⎤
    ⎢ ⎥
    ⎢0⎥
    ⎢ ⎥
    ⎣1⎦
    [2  1  0]
    [0]
    -(2*s - 1)/(5*(s**2 + 2*s + 5)) + 2/(5*s)
    Jordan (canonical) Form matrices A, B, C, D:
    ⎡0   0     0  ⎤
    ⎢             ⎥
    ⎢0   0     1  ⎥
    ⎢             ⎥
    ⎣0  -1/5  -2/5⎦
    ⎡1⎤
    ⎢ ⎥
    ⎢0⎥
    ⎢ ⎥
    ⎣1⎦
    [2/5  1/5  -2/5]
    [0]

#### c)

```python
num = expand((s+3)).as_poly()
den = expand((s+1)**2*(s+2)).as_poly()
helper3p2(num, den)
```

    Transfer function H(s):
         s + 3
    ────────────────
           2
    (s + 1) ⋅(s + 2)
    First Companion Form matrices A, B, C, D:
    ⎡0   1   0 ⎤
    ⎢          ⎥
    ⎢0   0   1 ⎥
    ⎢          ⎥
    ⎣-2  -5  -4⎦
    ⎡0⎤
    ⎢ ⎥
    ⎢0⎥
    ⎢ ⎥
    ⎣1⎦
    [3  1  0]
    [0]
    Jordan form condition not implemented.
    Here is the partial fraction decomposition to compute Jordan form manually.
      1       1        2
    ───── - ───── + ────────
    s + 2   s + 1          2
                    (s + 1)

We can use the partial fraction decomposition to do the Jordan form construction manually with $-1$ as a repeated root.

The system can be decomposed into two subsystems.

##### Subsystem 1: $H_1(s) = \frac{1}{s+2}$
\begin{eqnarray}
A_1 &=& \begin{pmatrix}
-2
\end{pmatrix}, B_1 = \begin{pmatrix}
1
\end{pmatrix}, \\
C_1 &=& \begin{pmatrix}
1
\end{pmatrix}
\end{eqnarray}

##### Subsystem 2: $H_2(s) = \frac{1}{(s+1)^2} - \frac{1}{s+1}$
Using Eqn 3.116, with $s_1 = -1$ and $r_{11}, r_{21} = -1, 1$:
\begin{eqnarray}
A_2 &=& \begin{pmatrix}
-1 & 0 \\
1 & -1 \\
\end{pmatrix}, \
B_2 = \begin{pmatrix}
1 & 0 \\
\end{pmatrix}^T, \\
C_2 &=& \begin{pmatrix}
-1 & 1
\end{pmatrix}
\end{eqnarray}

Both systems are (again) combined block-diagonally:
\begin{eqnarray}
A &=& \begin{pmatrix}
A_1 & 0_{1x2} \\
0_{2x1} & A_2 \\
\end{pmatrix}, \
B = \begin{pmatrix}
B_1^T & B_2^T \\
\end{pmatrix}^T, \\
C &=& \begin{pmatrix}
C_1 & C_2
\end{pmatrix}, \
D = \begin{pmatrix}
b_0
\end{pmatrix}
\end{eqnarray}

which, for this problem, are:
\begin{eqnarray}
A &=& \begin{pmatrix}
-2 & 0 & 0 \\
0 & -1 & 0 \\
0 & 1 & -1 \\
\end{pmatrix}, \
B = \begin{pmatrix}
1 & 1 & 0 \\
\end{pmatrix}^T, \\
C &=& \begin{pmatrix}
1 & -1 & 1
\end{pmatrix}, \
D = \begin{pmatrix}
0
\end{pmatrix}
\end{eqnarray}

### Problem 3.3
#### a)
The matrices can be derived by just looking at the block-diagram as:

\begin{eqnarray}
A &=& \begin{pmatrix}
-s_1 & 1 & 0 & \dots \\
0 & -s_2 & 1 & 0 \dots \\
0 & \dots & \ddots & \dots 1 \\
0 & \dots & 0 & -s_k
\end{pmatrix}, \
B = \begin{pmatrix}
0_{k-1} \\
1
\end{pmatrix}, \\
C &=& \begin{pmatrix}
C_1 & C_2 & \dots & C_k
\end{pmatrix}, \
D = \begin{pmatrix}
0
\end{pmatrix}
\end{eqnarray}

#### b)
The dynamics of the system transformed to the normal form are _similar_ to the tandem form with $\mathbf{z}= T \mathbf{x}$ (where I have reversed the book's convention and am using $\mathbf{z}$ to refer to the normal form).  This amounts to finding a tranformation matrix, $T$ s.t. $T^{-1} \Lambda T = A$ from above, which is a Schur decomposition.  The upper-triangular Schur matrix, $U$, is simply $\Lambda$.  The Schur transformation matrix, $T$, is obtained by finding the eigenvectors of $A$ using $(A + s_iI)\mathbf{t}_i = \mathbf{0}$.  Since $A$ is upper-triangular, the eigenvalues are simply the diagonal elements.  To find the columns $\{\mathbf{t}_i \}_{1 \le i \le k}$ of $T$, it is necessary to solve linear equations of the form
\begin{eqnarray}
(s_i - s_1)t_i^1 + t_i^2 &=& 0 \\
(s_i - s_2)t_i^2 + t_i^3 &=& 0 \\
\vdots
\end{eqnarray}
for $0 \le i < k$.  **Note:  The superscripts above are not powers, they represent the numbered elements of the eigenvector $\mathbf{t}_i$**

To find the first eigenvector, there is only a single equation, leaving $t_1^1$ free.  Choosing $t_1^1=1$ is fine (the application of $T^{-1}$ would rescale correctly regardless), so $\mathbf{t}_1 = (1 \ 0 \ \dots 0)^T$.  Setting the free parameter to 1 will be done subsequently for the same reason.

To find the second eigenvector ($t_2^2 = 1$):
\begin{eqnarray}
(s_2 - s_1) t_2^1 + 1 &=& 0
\end{eqnarray}
which means $\mathbf{t}_2 = (-\frac{1}{s_2 - s_1} \ 1 \ 0 \ \dots 0)^T$

To find the third eigenvector ($t_3^3 = 1$):
\begin{eqnarray}
(s_3 - s_1) t_3^1 + t_3^2 &=& 0 \\
(s_3 - s_2) t_3^2 + 1 &=& 0
\end{eqnarray}
which means $\mathbf{t}_3 = (\frac{1}{(s_3 - s_2)(s_2 - s_1)} \ -\frac{1}{s_3 - s_2} \ 1 \ \dots 0)^T$

To nail down the pattern, the fourth eigenvector is computed from the following set of linear equations:
\begin{eqnarray}
(s_4 - s_1) t_4^1 + t_4^2 &=& 0 \\
(s_4 - s_2) t_4^2 + t_4^3 &=& 0 \\
(s_4 - s_3) t_4^3 + 1 &=& 0
\end{eqnarray}
which means $\mathbf{t}_4 = (-\frac{1}{(s_4 - s_3)(s_4 - s_2)(s_4-s_1)} \ \frac{1}{(s_4 - s_3)(s_4-s_2)} \ -\frac{1}{s_4-s_3} \ 1 \dots 0)^T$

Finally, putting it all together (and extending where necessary):

\begin{eqnarray}
T = \begin{pmatrix}
1 & -\frac{1}{s_2 - s_1} & \frac{1}{(s_3 - s_2)(s_3-s_1)} & -\frac{1}{(s_4-s_3)(s_4 - s_2)(s_4-s_1)} & \dots & (-1)^{k-1} \prod_{i = 1}^k \frac{1}{(s_i - s_1)}\\
0 & 1 & -\frac{1}{s_3 - s_2} & \frac{1}{(s_4 - s_3)(s_4-s_2)} & \dots & (-1)^{k-2} \prod_{i = 2}^k \frac{1}{(s_i - s_2)}\\
0 & 0 & 1 & -\frac{1}{s_4 - s_3} & \dots & (-1)^{k-3} \prod_{i = 3}^k \frac{1}{(s_i - s_3)}\\
0 & 0 & 0 & 1 & \dots & (-1)^{k-4} \prod_{i = 4}^k \frac{1}{(s_i - s_4)}\\
0 & 0 & 0 & \ddots & \ddots & \vdots \\
\vdots & \vdots & \vdots & \dots & \ddots & \vdots \\
0 & 0 & 0 & \dots & \dots & 1
\end{pmatrix}
\end{eqnarray}

```python
# perform a quick check of the above for a 3x3
S = numbered_symbols('s')
next(S)
s_array = [next(S) for i in range(3)]
A = Matrix([[-s_array[0], 1, 0], [0, -s_array[1], 1], [0, 0, -s_array[2]]])
T = Matrix([[1, -1/(s_array[1]-s_array[0]), 1/((s_array[2]-s_array[1])*(s_array[2]-s_array[0]))],
            [0, 1, -1/(s_array[2]-s_array[1])],
            [0, 0, 1]])
result = T**-1*A*T  # makes sure that T \Lambda T**-1 = A
for i in range(result.shape[0]):
    for j in range(result.shape[1]):
        result[i, j] = simplify(result[i, j])
pprint(result)
```

    ⎡-s₁   0    0 ⎤
    ⎢             ⎥
    ⎢ 0   -s₂   0 ⎥
    ⎢             ⎥
    ⎣ 0    0   -s₃⎦

The above check equals $\Lambda$ for the $3x3$ case, indicating the correct solution for $T$ was found.

### Problem 3.4

From Eqn. 3.25:
\begin{eqnarray}
\frac{\partial \Phi(\tau, t)}{\partial \tau} = A(\tau) \Phi(\tau, t)
\end{eqnarray}
Using the fact that $\Phi(t, \tau) = \Phi^{-1}(\tau, t)$ and the hint given, it follows that:
\begin{eqnarray}
\frac{\partial \Phi(t, \tau)}{\partial \tau} &=& \frac{\partial \Phi^{-1}(\tau, t)}{\partial \tau} \\
&=& -\Phi^{-1}(\tau, t) \frac{\partial \Phi(\tau, t)}{\partial \tau} \Phi^{-1}(\tau, t) \\
&=& -\Phi^{-1}(\tau, t) A(\tau) \Phi(\tau, t) \Phi^{-1}(\tau, t) \\
&=& -\Phi^{-1}(\tau, t) A(\tau) \\
&=& -\Phi(t, \tau) A(\tau)
\end{eqnarray}
QED

### Problem 3.5

The state space matrices can be quickly computed from the block-diagram:
\begin{eqnarray}
A &=& \begin{pmatrix}
-a_1 & 1 & 0 & \dots \\
-a_2 & 0 & 1 & 0 \dots \\
\vdots & \dots & \ddots & \dots 1 \\
-a_k & \dots & 0 & 0
\end{pmatrix}, \
B = \begin{pmatrix}
0_{k-1} \\
1
\end{pmatrix}, \\
C &=& \begin{pmatrix}
q_0 - \sum_{i=1}^k q_i a_i & q_1 & \dots & q_{k-1}
\end{pmatrix}, \
D = \begin{pmatrix}
q_k
\end{pmatrix}
\end{eqnarray}

I used these matrices to find the transfer function via $H(s) = C (sI - A)^{-1} B + D$ (Eqn. 3.65) and the algorithm from Fig. 3.1 for computing the $E_i$ matrices in $\text{adj}(sI - A)$ (see Eqn. 3.67).  Some observations (I'm not going to work through all the details):

* The form of $A$ means that $|sI - A|$, which is the denominator of $H(s)$, will be $s^k + a_1 s^{k-1} + a_2 s^{k-2} + \cdots + a_k$
* The algorithm for computing $E_i$ recursively, combined with the sparseness of $B$, lead to incrementing of the index on the $q$ terms in lower powers of $s$ in the numerator of $H(s)$.

The transfer function $H(s)$ is:
\begin{eqnarray}
H(s) = \frac{q_{k-1} s^{k-1} + [a_1 q_{k-1} + q_{k-2}]s^{k-2} + [(a_1 + a_2)q_{k-1} + q_{k-2} + q_{k-3}]s^{k-2} + \cdots}{s^k + a_1 s^{k-1} + a_2 s^{k-2} + \cdots + a_k}
\end{eqnarray}

The relationship between the $q_i$'s and the numerator coefficients of the standard transfer function representation (using $b_i$'s) is:
\begin{eqnarray}
b_0 &=& q_{k-1} \\
b_1 &=& a_1 q_{k-1} + q_{k-2} \\
b_2 &=& (a_1 + a_2)q_{k-1} + q_{k-2} + q_{k-3} \\
\vdots \\
b_j &=& q_{k-1} \sum_{i=1}^j a_i + \sum_{i=2}^{j+1} q_{k-i} \\
\vdots \\
b_k &=& q_{k-1} \sum_{i=1}^k a_i + \sum_{i=2}^k q_{k-i}
\end{eqnarray}

### Problem 3.6

#### a)
The differential equations are:
\begin{eqnarray}
\ddot x + \frac{k^2}{Mr^2R} \dot x + \frac{mg}{M} \theta &=& \frac{k}{MRr} e \\
\ddot \theta - \frac{M+m}{M \ell} g \dot \theta - \frac{k^2}{Mr^2R \ell} \dot x &=& -\frac{k}{MRr \ell} e
\end{eqnarray}
The 2 second-order ODEs above can be cast as 4 first-order ODEs with $x_1 \equiv x, x_2 \equiv \dot x, \theta_1 \equiv \theta, \theta_2 \equiv \dot \theta$:
\begin{eqnarray}
\dot x_1 &=& x_2 \\
\dot x_2 &=& -\frac{k^2}{Mr^2R} x_2 - \frac{mg}{M} \theta_1 + \frac{k}{MRr} e \\
\dot \theta_1 &=& \theta_2 \\
\dot \theta_2 &=& \frac{M+m}{M \ell} g \dot \theta + \frac{k^2}{Mr^2R \ell} x_2 -\frac{k}{MRr \ell} e
\end{eqnarray}
With $\mathbf{x} \equiv [x_1 \ x_2 \ \theta_1 \ \theta_2]^T$, the state-space model (with output $\mathbf{y} = [x_1 \ \theta_1]^T$) can be written:
\begin{eqnarray}
\mathbf{\dot x} &=& A \mathbf{x} + B e \\
\mathbf{y} &=& C \mathbf{x} + D e,
\end{eqnarray}
with
\begin{eqnarray}
A &=& \begin{pmatrix}
0 & 1 & 0 & 0 \\
0 & -\frac{k^2}{Mr^2R} & -\frac{mg}{M} & 0 \\
0 & 0 & 0 & 1 \\
0 & \frac{k^2}{Mr^2R \ell} & \frac{M+m}{M \ell} g & 0
\end{pmatrix}, \
B = \frac{k}{MRr}\begin{pmatrix}
0 \\
1 \\
0 \\
-\frac{1}{\ell}
\end{pmatrix}, \\
C &=& \begin{pmatrix}
1 & 0 & 0 & 0 \\
0 & 0 & 1 & 0
\end{pmatrix}, \
D = \begin{pmatrix}
0 \\
0
\end{pmatrix}
\end{eqnarray}

#### b)

#### c)

```python
a, b, c, d = symbols('a b c d')
# a = -k^2 / (Mr^2R)
# b = -mg / M
# c = k^2 / (Mr^2Rl)
# d = (M+m)g / (Ml)
A = Matrix([[0, 1, 0, 0],[0, a, b, 0], [0, 0, 0, 1], [0, c, d, 0]])
pprint(A)
res = simplify(ch3_utils.computeResolvent(A))
pprint(res)
```

    ⎡0  1  0  0⎤
    ⎢          ⎥
    ⎢0  a  b  0⎥
    ⎢          ⎥
    ⎢0  0  0  1⎥
    ⎢          ⎥
    ⎣0  c  d  0⎦
    ⎡                     2
    ⎢1              -d + s                            b
    ⎢─  ───────────────────────────────  ───────────────────────────   ───────────
    ⎢s    ⎛         2                3⎞           2                3     ⎛
    ⎢   s⋅⎝a⋅d - a⋅s  - b⋅c - d⋅s + s ⎠  a⋅d - a⋅s  - b⋅c - d⋅s + s    s⋅⎝a⋅d - a⋅
    ⎢
    ⎢                    2
    ⎢               d - s                            b⋅s
    ⎢0   ────────────────────────────    ───────────────────────────     ─────────
    ⎢              2                3             2                3
    ⎢    -a⋅d + a⋅s  + b⋅c + d⋅s - s     a⋅d - a⋅s  - b⋅c - d⋅s + s      a⋅d - a⋅s
    ⎢
    ⎢                  c                          s⋅(a - s)
    ⎢0    ───────────────────────────    ────────────────────────────   ──────────
    ⎢              2                3              2                3
    ⎢     a⋅d - a⋅s  - b⋅c - d⋅s + s     -a⋅d + a⋅s  + b⋅c + d⋅s - s    -a⋅d + a⋅s
    ⎢
    ⎢                 c⋅s                      a⋅d - b⋅c - d⋅s                   s
    ⎢0    ───────────────────────────    ────────────────────────────   ──────────
    ⎢              2                3              2                3
    ⎣     a⋅d - a⋅s  - b⋅c - d⋅s + s     -a⋅d + a⋅s  + b⋅c + d⋅s - s    -a⋅d + a⋅s

                        ⎤
        b               ⎥
    ────────────────────⎥
     2                3⎞⎥
    s  - b⋅c - d⋅s + s ⎠⎥
                        ⎥
                        ⎥
        b               ⎥
    ──────────────────  ⎥
    2                3  ⎥
      - b⋅c - d⋅s + s   ⎥
                        ⎥
     a - s              ⎥
    ──────────────────  ⎥
    2                3  ⎥
      + b⋅c + d⋅s - s   ⎥
                        ⎥
    ⋅(a - s)            ⎥
    ──────────────────  ⎥
    2                3  ⎥
      + b⋅c + d⋅s - s   ⎦

To get the STM, do a table-lookup for the inverse Laplace transform element-by-element (or evaluate numerically).  This would be tedious, so I am not going to do it.

#### d)
Split the matrices $C, D$ into two components: $C_x, D_x$ correspond to the first rows and $C_\theta$ the second rows.  The transfer function from $e$ to output $x$ is $H_x(s) = C_x(sI-A)^{-1}B + D_x$.  Similarly, the transfer function from $e$ to $\theta$ is $H_\theta(s) = C_\theta(sI-A)^{-1}+D_\theta$.  The resolvent $(sI-A)^{-1}$ was computed in part c) above.

### Problem 3.7

#### a)
From Problem 2.3's solution:
\begin{eqnarray}
A &=& \frac{1}{RC} \begin{pmatrix}
-3 & 1 & 0 \\
1 & -2 & 1 \\
0 & 1 & -3
\end{pmatrix}, \
B &=& \frac{1}{RC} \begin{pmatrix}
2 \\
0 \\
0
\end{pmatrix}, \
E &=& \frac{1}{RC} \begin{pmatrix}
0 \\
0 \\
2
\end{pmatrix}
\end{eqnarray}

#### b)
The solution to b) follows exactly the same as part c) above.

```python
R, C = symbols('R C')
A = Matrix([[-3/(R*C), 1/(R*C), 0],[1/(R*C), -2/(R*C), 1/(R*C)], [0, 1/(R*C), -3/(R*C)]])
pprint(A)
res = simplify(ch3_utils.computeResolvent(A))
pprint(res)
```

    ⎡-3    1      ⎤
    ⎢───  ───   0 ⎥
    ⎢C⋅R  C⋅R     ⎥
    ⎢             ⎥
    ⎢ 1   -2    1 ⎥
    ⎢───  ───  ───⎥
    ⎢C⋅R  C⋅R  C⋅R⎥
    ⎢             ⎥
    ⎢      1   -3 ⎥
    ⎢ 0   ───  ───⎥
    ⎣     C⋅R  C⋅R⎦
    ⎡         ⎛ 2  2  2              ⎞
    ⎢     C⋅R⋅⎝C ⋅R ⋅s  + 5⋅C⋅R⋅s + 5⎠                C⋅R
    ⎢─────────────────────────────────────  ───────────────────────  ─────────────
    ⎢ 3  3  3      2  2  2                  (C⋅R⋅s + 1)⋅(C⋅R⋅s + 4)   3  3  3
    ⎢C ⋅R ⋅s  + 8⋅C ⋅R ⋅s  + 19⋅C⋅R⋅s + 12                           C ⋅R ⋅s  + 8⋅
    ⎢
    ⎢                 C⋅R                       C⋅R⋅(C⋅R⋅s + 3)
    ⎢       ───────────────────────         ───────────────────────         ──────
    ⎢       (C⋅R⋅s + 1)⋅(C⋅R⋅s + 4)         (C⋅R⋅s + 1)⋅(C⋅R⋅s + 4)         (C⋅R⋅s
    ⎢
    ⎢                                                                         ⎛ 2
    ⎢                 C⋅R                             C⋅R                 C⋅R⋅⎝C ⋅
    ⎢─────────────────────────────────────  ───────────────────────  ─────────────
    ⎢ 3  3  3      2  2  2                  (C⋅R⋅s + 1)⋅(C⋅R⋅s + 4)   3  3  3
    ⎣C ⋅R ⋅s  + 8⋅C ⋅R ⋅s  + 19⋅C⋅R⋅s + 12                           C ⋅R ⋅s  + 8⋅

                            ⎤
        C⋅R                 ⎥
    ────────────────────────⎥
     2  2  2                ⎥
    C ⋅R ⋅s  + 19⋅C⋅R⋅s + 12⎥
                            ⎥
        C⋅R                 ⎥
    ─────────────────       ⎥
     + 1)⋅(C⋅R⋅s + 4)       ⎥
                            ⎥
     2  2              ⎞    ⎥
    R ⋅s  + 5⋅C⋅R⋅s + 5⎠    ⎥
    ────────────────────────⎥
     2  2  2                ⎥
    C ⋅R ⋅s  + 19⋅C⋅R⋅s + 12⎦

#### c)
The transfer function from $e_0$ to $v_3$ is $H(s) = C_{v_3}(sI - A)^{-1}B + \frac{2}{RC}$, with $C_{v_3} = [0 \ 0 \ 1]^T$.

### Problem 3.8

With only resistors, capacitors, and voltage/current sources, any node in the network will look like the above.  By Kirchoff's loop law, the current through node $i$ must be the sum of the two currents due to the current source and $\frac{v}{R_i}$ ($v$ will be due to adjacent loops) minus the current drop across the capacitor.  This results in a differential equation $\dot v_i = \frac{1}{R_i C_i}(iR_i + v - v_i)$.  So, in the construction of the block submatrix in the full system matrix A, the entry at $(i,i)$ will be $-\frac{1}{R_i C_i}$, which is _negative_ clearly.  A consequence of Kirchoff's loop law is symmetry of the system matrix $A$: positive voltage difference across resistor $R_i$ into node $v_i$ is seen as a negative voltage difference into an adjacent node.

From the problem above, the structure of the block due node $v_i$ becomes clear.  The magnitude of the element on the diagonal will be larger than the magnitude of the off-diagonal elements (terms due to current sources will be an additive bias not captured in the $A$ matrix).  This means that reduction to row-echelon form will result in an upper-triangular matrix with negative diagonal.  This proves that all eigenvalues lie on the _negative_ real axis, as required.  QED

### Problem 3.9

#### a)
From Problem 2.5's solution:
\begin{eqnarray}
A &=& \begin{pmatrix}
0 & 0 & 1 & 0 \\
0 & 0 & 0 & 1 \\
-\frac{K}{M} & \frac{K}{M} & -\frac{k^2}{r^2R} & 0 \\
\frac{K}{M} & -\frac{K}{M} & 0 & -\frac{k^2}{r^2R}
\end{pmatrix}, \
B = \frac{k}{MRr}\begin{pmatrix}
0 & 0 \\
0 & 0 \\
\frac{k}{Rr} & 0 \\
0 & \frac{k}{Rr}
\end{pmatrix}, \\
C &=& \begin{pmatrix}
1 & 0 & 0 & 0 \\
0 & 1 & 0 & 0
\end{pmatrix}, \
D = \begin{pmatrix}
0 \\
0
\end{pmatrix}
\end{eqnarray}

#### b)

```python
# with K, M, k, R, r = 40, 1, 2, 100, 0.02
A = Matrix([[0, 0, 1, 0], [0, 0, 0, 1], [-40, 40, -100, 0], [40, -40, 0, -100]])
res = computeResolvent(A)
C_x1 = Matrix([1, 0, 0, 0])
C_x2 = Matrix([0, 1, 0, 0])
B = Matrix([[0, 0], [0, 0], [1, 0], [0, 1]])
H_x1 = C_x1.T * res * B
H_x2 = C_x2.T * res * B
# H_x1
pprint(H_x1[0])
# H_x2
pprint(H_x2[0])
```

             1                  1           1
    ─────────────────── - ───────────── + ─────
      ⎛ 2             ⎞   200⋅(s + 100)   200⋅s
    2⋅⎝s  + 100⋅s + 80⎠
               1                  1           1
    - ─────────────────── - ───────────── + ─────
        ⎛ 2             ⎞   200⋅(s + 100)   200⋅s
      2⋅⎝s  + 100⋅s + 80⎠

```python
from numpy import sqrt
r1 = -100
r2 = 0
a, b, c = 1., 100., 80.
r3 = (-b + sqrt(b**2 - 4.*a*c)) / (2.*a)
r4 = (-b - sqrt(b**2 - 4.*a*c)) / (2.*a)
print('Roots of both transfer functions are nearly the same: {}, {}, {}, {}'.format(r1, r2, r3, r4))
```

    Roots of both transfer functions are nearly the same: -100, 0, -0.8065044950046243, -99.19349550499538