### Preparation

* [`numpy`](http://www.numpy.org) is a module for scientific computing (i.e. *vector* and *matrix* processing) with Python.
* [`matplotlib`](https://matplotlib.org) is a module for plotting.

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn
```

```python
from operator import mul
from IPython.core.interactiveshell import InteractiveShell

InteractiveShell.ast_node_interactivity = "all"
plt.style.use('fivethirtyeight')
```

## An example

Let us look at a hypothetical appartment rent data. We will take a look only at the price per square meter ($m^2$): `price = w0 * 1  + w1 * square_meters`. We call `price` a response variable; in literature usually denoted as $y$. We call `square_meters` a predictor variable. We call `w0` and `w1` weights - it is basically *intercept* and *slope* of a line. `w0` has a special meaning and is called a `bias`. In our example we can interpret it as base fixed price you would pay for 0 sqaure meters of living area.

Here is a graph that visualizes a price with 200 CU (currency units) base price and 10 CU increase per square meter: `price = 200 + 10 * square_meters`. We also add some *observations*, i.e., some data that we read in a finctional announce or whereever. These data points have some *noise* - factors that we cannot account for. For example some renters may have good or bad negotiation skills that will make the price vary around the line.

```python
square_meters = np.linspace(0, 150)
price = 200 + 10 * square_meters

# a random number between 0 and 1 will be multiplied by 150 to
# kind of represent a spectrum of square meters.
number_of_points = 20
random_square_meters_data = np.random.sample(size = number_of_points) * 150
price_with_noise = 200 + 10 * random_square_meters_data + np.random.normal(size = number_of_points, scale = 50)

# plot data
_ = plt.plot(square_meters, price, color = 'grey')
_ = plt.plot(random_square_meters_data, price_with_noise, 'o', color = 'blue')
_ = plt.xlabel('Square meters')
_ = plt.ylabel('Price in CU')
plt.ylim(0, 2000)
plt.show()
```

The data here has one dimension: the square meters. But we rarely have only one-dimensional data (also see [problems with many dimensions](https://en.wikipedia.org/wiki/Curse_of_dimensionality)). For example, square meters are actually two variables: length and width. We can also add height as a third dimension that may be an additional influence to the price. So is the city, weather and anything else. We usually abstract from the actual names and simly call dimensions by their numbered variable: dimension $i$ is simply encoded in variable $x_i$. For example $x_1$ is `length`, $x_2$ is `width`, $x_3$ is `height` and so on. *Bias* has usually a special notation: $x_0$. We almost always need to take care for *bias* variable and we will do it down bellow.

We now have a conceptual understanding of dimensions and can jump into coding stuff.

The goal of the following is to explain how basic regression works.

Imagine we have gathered 20 samples of 'data' where for each price (response) we have `length`, `width`, `height`, and `distance` from downtown (predictors).

The weights influence the price as follows: we have `basic_price` ($w_0$), `length` ($x_1$), `width` ($x_2$), `height` ($x_3$), `distance` ($x_4$), and the last one `square_meters` ($x_5$). Thus the price results from the following:
$$price = w_0 + w_1 \cdot x_1 + w_2 \cdot x_2 + w_3 \cdot x_3 + w_4 \cdot x_4 + w_5 \cdot x_5$$

```python
weights = (np.random.uniform(size = 6) + 1) * [200, .5, .7, 1.5, -.05, 3.5]
weights
```

```python
n_samples = 20
np.set_printoptions(2, suppress=True)
tmp = np.column_stack((1.5 + np.random.sample(size = n_samples) * 10, # length
                       1.5 + np.random.sample(size = n_samples) * 10, # width
                       np.round(2   + np.random.sample(size = n_samples) * 2, decimals=1), # height
                       np.random.sample(size = n_samples) * 3000)) # distance from center in meters
samples_data_X = np.column_stack((tmp, tmp[:,0]*tmp[:,1])) # square meters
del(tmp)
samples_data_X
```

```python
samples_data_price_Y = np.dot(np.column_stack((np.ones(samples_data_X.shape[0]), samples_data_X)), weights)
# we add some noise to the response that adds unpredictable variance to the price.
samples_data_price_Y = samples_data_price_Y + np.random.normal(scale = 2.5, size = np.prod(samples_data_price_Y.shape))
samples_data_price_Y
```

Let's say, we want to find *weights* of a *linear model* to predict the price of an offer (the six $w_i$ from the equation above). Meaning we got data from a makler that send us data for `length=10`, `width=10`, `height=3`. Furthermore we looked at some map data and got the `distance=3044` meters from downtown. The makler didn't tell us the price yet - but lets see if we can estimate the price by ourselfes. In such a way we will go into negotiations and can back up our claims with data.

Having gathered the data, we compute the following:

$$\mathbf{\hat{w}} = (\mathbf{X}^T \mathbf{X})^{-1}\mathbf{X}^T \mathbf{y} $$

```python
bold_X = np.column_stack((np.ones(samples_data_X.shape[0]), samples_data_X))
estimated_weights = np.dot(np.dot(np.linalg.inv(np.dot(bold_X.T, bold_X)), bold_X.T), samples_data_price_Y)
estimated_weights
weights
```

Looks like the estimated weight from noisy data are not so far off. Of course we wouldn't know the true weights in reality. And so, we would have to bargain with a makler for a price around:

```python
np.round(np.dot(estimated_weights, [1, 10, 10, 3, 3044, 10*10]), 2)
```

### Here be dragons

**What is this matrix vodoo above? And why does it looks like it looks like?**

We will derive the above equation based on amazingly clear [tutorial of Mark L. J. Orr](https://www.cc.gatech.edu/~isbell/tutorials/rbf-intro.pdf). And we will do it in the least mathematical way possible - mostly with **code**.

## Appendix A.1

A *vector* is simply an array of $n$ numbers $\mathbf{x}^T = [x_1, x_2, \ldots, x_n]$. The number $n$ represents the number of *dimensions*. The T-operator $\cdot^T$ simply tells us to treat the vector as a *column* vector - more to that in a second.

```python
x = np.arange(5)
x
```

A matrix is simply a 'collection' of vectors with the same dimension. Most of the time the matrix will have $m$ number of rows and $n$ number of columns.

**Example**: We want to have two (2) arrays. Each of array contains five (5) numbers, i.e., is a 5-dimensional vector. The resulting matrix has $m=2$ rows and $n=5$ columns.

```python
v1 = np.arange(1, 6)
v1
v2 = np.arange(1, 6) * 2
v2
simple_matrix = np.stack([v1, v2])
simple_matrix
```

For computational purpuses we seldomly use $x$ as single vector. Instead we treat it as *row* entry in a matrix, i.e. as a *row vector*. Hence, in matrix notation, a vector has one (1) row and $n$ columns. (This is usually denoted as a *tuple* `(1, n)`)

```python
v1 = np.reshape(np.arange(1,6), (1,5))
v1
```

Remember our T-operator? Now, if we use $\cdot^T$ it will *[T]ranspose* the matrix, i.e. rows will be collums and collumns will be rows.

```python
v1.T
```

The same applies to matrices:

```python
simple_matrix = np.stack([v1, v2])
simple_matrix
simple_matrix.T
```

Which bring us directly to matrix and vector operations. We denote a $k$-th vector with bold $\mathbf{x}_k$ (e.g. vector 1 is $\mathbf{x}_1$). We denote a matrix with a bold capital letter, for example $M$. As discussed, we can access $j$-th vector (i.e. $\mathbf{x}_j$) and $i$-th value (value in dimension $i$) - or in short: the value of $M_{ji}.

We have the following operations: (TODO implement in actual code; compare timings with numpy)

* vector addition
* vector and scalar multiplication
* vector matrix product
* matrix product

About matrix product: you can only multiply matrices when the 'inner' shape matches: `(shape_1, inner_shape) x (inner_shape, shape_2)`. The resulting shape of the matrix will be `(shape_1, shape_2)`. Here the T-operator comes in play because it reverses the shape. This allow us to multiply matrix and its transpose form. The resulting shape will have the same number of rows and columns and is termed quadratic matrix. One of the important properties of the quadratic matrix is that it is inversible. The inverse of a matrix describes the same effect for simple number. If you have a number $a$ - the inverse is $\frac{1}{a}$ or $a^{-1}$ which means that $a \cdot a^{-1} = 1$.
The same is true for matrices where $\mathbf{M} \cdot \mathbf{M}^{-1} = \mathbf{I}$. $\mathbf{I}$ is the unity matrix - a matrix where the only non-zero entries are in the diagonal with value 1, or in other terms $I_{ii} = 1$. The identity matrix' purpose is to have an operation which yields the same 'object' when multiplied.

```python
vector = np.arange(5)
vector
vector + 5 # 5 is 'broadcasted' into the array - i.e. + 5 is repeated for every entry
vector + vector # adding vectors element wise
vector * 3 # multiply every entry with 3
vector * vector # element wise multiplication | THAT IS NOT A VECTOR PRODUCT | it is called Hadaman-Product
np.dot(vector, vector) # that is a vector product; it results in a scalar
mat_1 = np.reshape(np.random.randint(1,100, size = 2*3), (2,3))
mat_2 = np.reshape(np.random.randint(1,100, size = 2*3), (2,3))
mat_1
mat_2
np.dot(mat_1, mat_2.T)
np.dot(mat_1, mat_2.T).shape
np.dot(mat_1.T, mat_2)
np.dot(mat_1.T, mat_2).shape
```

#### Why do we use matrix operations ?

In linear models we usually compute.
$$y = w_0 \cdot 1 + w_1 \cdot x_1 + w_2 \cdot x_2 + w_3 \cdot x_3 + w_4 \cdot x_4 + w_5 \cdot x_5$$
$$y = w_0 + \sum\limits_{i=1}^n w_i \cdot x_i$$

For efficiency reasons we transform the vector $\mathbf{x} = (x_1, x_2, x_3, x_4, x_5)$ into $\mathbf{x} = (1, x_1, x_2, x_3, x_4, x_5)$

```python
example_weights = np.array([20,4,6])
example_weights
example_x = np.array([1, 4, - 5])
example_x
```

The equation above can basically be implemented like this (don't! it is highly ineffcient!):

```python
def get_y(w, x):
    res = 0
    for i in range(len(x)):
        res = res + w[i]*x[i]
    return res
```

```python
get_y(example_weights, example_x)
```

A *vectorized* version is actually 'better':

```python
sum(example_weights * example_x)
```

For matrix operation we need to treat the vectors as tuples (1 row of 3 dimensions, `(1,3)`). The ultimate operation is the *dot-product* and since we need matching of inner dimensions, we need to prepare the vectors to return one number, i.e. the outer shape is 1. Thus the dot-product between these two matrix-vectors can be written as $\mathbf{w}\cdot\mathbf{x}^T$ for *row-vectors* (and actually in mathematical terms for *column vectors*: $\mathbf{w}^T\mathbf{x}$, but never mind).

```python
example_weights2 = np.reshape(example_weights, (1, 3))
example_x2 = np.reshape(example_x, (1, 3))
np.dot(example_weights2, example_x2.T)
```

The timings speak for themselfes:

```python
%timeit get_y(example_weights, example_x)
%timeit sum(example_weights * example_x)
%timeit np.dot(example_weights2, example_x2.T)
```

And that, just for one single small sample. We usually are dealing with more. How about 10000 samples with 1000 dimensions?

```python
example_weights2 = np.reshape(np.random.sample(size = 1*1000), (1, 1000))
example_x2 = np.reshape(np.random.sample(size = 10000*1000), (10000, 1000))
```

```python
%timeit y1 = [get_y(example_weights2[0], x) for x in example_x2]
%timeit y2 = [sum(example_weights2[0] * x) for x in example_x2]
%timeit y3 = np.dot(example_weights2, example_x2.T)
```

**This is the reason why matrix operations are so important!** (not only because of math)!

## Appendix A.4 The optimal weight vector

Now we are getting into the bottom of things: So given data in $\mathbf{X}$ (m rows of n dimensional vectors) we want to determine weights (n (+1) dimensional vector) of a linear model. As [exemplarly shown](https://www.geogebra.org/m/xC6zq7Zv), the goal is to find such a weight vector that minimizes the '*reconstruction error*' of the estimation represented by a *sum of squared errors* (SSE):
$$E = \frac{1}{2}\sum\limits_{\mathbf{x}_i \in \mathbf{X}, \hat{y}_i} (\hat{y}_i - f(\mathbf{x}_i))^2$$
with $f(\mathbf{x}_i)$ beeing our estimator:
$$f(\mathbf{x}) = \sum\limits_{k=0}^n w_k \cdot x_k$$
(we add $\frac{1}{2}$ to make our equations nicer ;-)

Now, to find a minimum of a function (and we want to find parameters such that the error $E$ is minimal) math tells us to
* differentiate the function with respect to the variables (in our case $w_i$ from $\mathbf{w}$)
* eqaute the result with zero
* solve the equation for the variables

Let's to just that:
$$\frac{\delta E}{\delta w_j} = \frac{\delta \frac{1}{2} \sum\limits_{\mathbf{x}_i \in \mathbf{X}, \hat{y}_i} (\hat{y}_i - f(\mathbf{x}_i))^2}{\delta w_j}$$
which is equally valid to
$$\frac{\delta E}{\delta w_j} = \frac{1}{2} \sum\limits_{\mathbf{x}_i \in \mathbf{X}, \hat{y}_i} \frac{\delta (\hat{y}_i - f(\mathbf{x}_i))^2}{\delta w_j}$$

Let us focus on the term $\frac{\delta (\hat{y}_i - f(\mathbf{x}_i))^2}{\delta w_j}$. A simple application of [chain rule](https://en.wikipedia.org/wiki/Chain_rule) tells us that $(h(g(x)))' =  h'(g(x))\cdot g'(x)$ with $h(x)~~\hat{=}~~x^2$ and $g(x)~~\hat{=}~~\hat{y}_i - f(\mathbf{x}_i)$. The derivate of $h(x)$ is $h'(x) = 2\cdot x$. And the derivate of $g(x)$ is $g'(x) = - x_{ij}$.

Substituting the variables yields:
$$\frac{\delta (\hat{y}_i - f(\mathbf{x}_i))^2}{\delta w_j} = 2 \cdot (\hat{y}_i - f(\mathbf{x}_i)) \cdot {-  x_{ij}}$$

Inserting into the error $E$ we get
$$\frac{\delta E}{\delta w_j} =  \frac{1}{2} \sum\limits_{\mathbf{x}_i \in \mathbf{X}, \hat{y}_i} 2 \cdot (\hat{y}_i - f(\mathbf{x}_i)) \cdot {-  x_{ij}}$$
and equating to zero we get:
\begin{align}
0 & = \sum\limits_{\mathbf{x}_i \in \mathbf{X}, \hat{y}_i} (\hat{y}_i - f(\mathbf{x}_i)) \cdot {-  x_{ij}} \\
0 & = \sum\limits_{\mathbf{x}_i \in \mathbf{X}, \hat{y}_i} - \hat{y}_i \cdot x_{ij} + f(\mathbf{x}_i) \cdot x_{ij} \\
0 & = - \sum\limits_{\mathbf{x}_i \in \mathbf{X}, \hat{y}_i} \hat{y}_i \cdot x_{ij} + \sum\limits_{\mathbf{x}_i \in \mathbf{X}, \hat{y}_i} f(\mathbf{x}_i) \cdot x_{ij} \\
\sum\limits_{\mathbf{x}_i \in \mathbf{X}, \hat{y}_i} \hat{y}_i \cdot x_{ij} & = \sum\limits_{\mathbf{x}_i \in \mathbf{X}, \hat{y}_i} f(\mathbf{x}_i) \cdot x_{ij} \\
\end{align}

Now, as we saw earlier, we can rewrite the sums over samples in *matrix form*:

$$\mathbf{X}^T\cdot \hat{\mathbf{y}} = \mathbf{X}^T\cdot \mathbf{f}$$

Our $\mathbf{f}$ is a vector of linear model estimation of $\mathbf{x}_j$:
$$f_j = f(\mathbf{x}_j) = \sum\limits_{i=0}^n w_i \cdot x_{i}$$
(with $x_0 = 1$) and hence can be rewritten to $\mathbf{x}_j^T\hat{\mathbf{w}}$ as we saw earlier.

$$
\mathbf{f} = \begin{bmatrix}f_1\\\ldots\\f_m\end{bmatrix} = \begin{bmatrix}\mathbf{x}_1^T\hat{\mathbf{w}}\\\ldots\\\mathbf{x}_m^T\hat{\mathbf{w}}\end{bmatrix} = \mathbf{X}\mathbf{\hat{w}}
$$

Then, from above, we have:
\begin{align}
\mathbf{X}^T \hat{\mathbf{y}} & = \mathbf{X}^T \cdot \mathbf{f} \\
\mathbf{X}^T \hat{\mathbf{y}} & = \mathbf{X}^T \mathbf{X} \cdot \mathbf{\hat{w}}
\end{align}

And solving for $\hat{w}$ (i.e. 'dividing' by $\mathbf{X}^T \mathbf{X}$ or better yet: multiplying by the inverse $(\mathbf{X}^T \mathbf{X})^{-1}$) finally yields:

$$ \mathbf{\hat{w}} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \hat{\mathbf{y}}$$

This is our solution from above. Simply coded as `np.dot(np.dot(np.linalg.inv(np.dot(X.T, X)), X.T), Y)`.

## Runtime information

Numpy, BLAS, and LAPACK.

```python
np.__file__
np.__version__
np.show_config()
```

```bash
%%bash
ldd ~/anaconda3/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so
```


## Implizite Abbildung durch Kerne

a. Geben Sie eine allgemeine Formel für die Abbildung der Ebene in den Raum der geordneten Monome
zweiter Ordnung (d.h. Produkte in unterschiedlicher Reihenfolge bekommen eine eigene Dimension,
obwohl das Resultat identisch ist, z.B. x1·x2 und x2·x1 bilden unterschiedliche Einträge im Vektorφ) an.

```python
import scipy as sy
import numpy as np
```

```python
def monome(x1,x2):
    phi = [x1*x1,x1*x2,x2*x1,x2*x2]
    return phi
```

$$
\begin{equation}
 \begin{aligned}
\Phi_1 :R^2->R^3 : \Phi_1(x) &= (x_1^2, \sqrt2 x_1 x_2, x_2^2) \\
\\
 \Phi_2 :R^2->R^4 : \Phi_2(x) &= (x_1^2, x_1 x_2, x_2 x_1, x_2^2) \\
  \end{aligned}
\end{equation}
$$

b. Berechnen Sie das komponentenweise Standard-Skalarprodukt im Raum der Monome zweiter
Ordnung zwischen zwei unterschiedlichen zweidimensionalen Eingabevektoren (x1,x2) und (y1,y2).

$$ x = (x_1,x_2), y = (y_1,y_2): $$

$$ f(x)=(x_1^2,x_1x_2,x_2x_1,x_2^2)$$

$$f(y)=(y_1^2,y_1y_2,y_2y_1,y_2^2) $$

$$ \\
\begin{equation}
 \begin{aligned}
\langle\Phi(x), \Phi(y)\rangle  &=  x_1^2 y_1^2 + 2x_1 x_2 y_1 y_2 + x_2^2 y_2^2 \\
%% &= (x_1 y_1 + x_2 y_2)^2 \\
%% &= \langle x, y\rangle^2
 \end{aligned}
\end{equation}
\\ $$

```python
# Eingabevektor erzeugen
def create_vector(x,y):
    vec = [x,y]

    return vec
```

```python
# Beispielvektoren
x1 = 4
x2 = 1

y1 = 5
y2 = 2

vector1 = create_vector(x1,x2)
vector2 = create_vector(y1,y2)
```

```python
vector1, vector2
```

    ([4, 1], [5, 2])

```python
def scalar_in_monome(v1,v2):
    x1=v1[0]
    x2=v1[1]
    y1=v2[0]
    y2=v2[1]
    mon1 = monome(x1,x2)
    mon2 = monome(y1,y2)
    #Bildung des Skalarprodukts
    scal  = mon1[0]*mon2[0]+mon1[1]*mon2[1]+mon1[2]*mon2[2]+mon1[3]*mon2[3]
    return scal
```

```python
res = scalar_in_monome(vector1,vector2)
res
```

    484

c. Berechnen Sie den homogenen Polynomkern zweiter Ordnung

$$ k_\Phi(x,y) = \langle x * y \rangle^2 \\$$

$$ k_\Phi(x,y) = \langle f(x),f(y) \rangle \\
          = \langle(x_1^2, 2x_1x_2, x_2^2), (y_1^2, 2y_1y_2, y_2^2) \rangle \\
          = x_1^2y_1^2 + 2((x_1x_2) * (y_1y_2)) + x_2^2y_2^2 $$

Daraus folgt:

$$ k_\Phi(x,y) = x_1^2y_1^2 + 2 ((x_1x_2) * (y_1y_2)) + x_2^2y_2^2 \\
          = (x_1y_1 + x_2y_2)^2 \\
          = \langle x,y \rangle^2 $$

```python
def polynom(v1,v2):
    poly = np.dot(v1,v2)**2
    return poly

```

```python
res = polynom(vector1,vector2)
res
```

    484

d. Geben Sie eine Formel für die Perzeptron-Entscheidungsfunktion im Raum der Monome
zweiter Ordnung in ihrer dualen Form an.

Ebenengleichung Perzeptron:

$$ g(x) \rightarrow \langle w , x\rangle + b \\ $$

Gewichtungsvektor w (Anpassungsschritten):

$$ w = w + y_i*x_i   $$

Linearkombination:

$$ w = \sum_{j=1}^\ell y_j x_j $$

Raum der Monome 2. Ordnung:

$$ w = \sum_{j=1}^\ell y_j f(x_j) $$

Perzeptron:

$$ g(x) = \langle \sum_{j=1}^\ell y_j f(x_j),f(x) \rangle + b \\
        = \sum_{j=1}^\ell y_j k_\Phi(x_j,x) + b \\
        = \sum_{j=1}^\ell y_j \langle x_j,x \rangle ^2 + b $$


### Prof. Dr. -Ing. Gerald Schuller
 Jupyter Notebook: Renato Profeta

# Filters

```python
%%html

```

Filters are linear, time-invariant systems. This means they fulfil the following properties:

If $F(x(n))$ is our filter function of input signal $x(n)$, then we have:

**Linearity:** for 2 signals $x_1(n)$ and $x_2(n)$,

$$\large
F( x_1(n)+x_2(n) )=F( x_1(n) )+F( x_2(n) )$$

With a factor $a$:

$$\large F( a \cdot x(n) )= a \cdot F( x(n) )$$

which means we can **“draw out” sums and factors out of our function.**

**Time-Invariance:** if

$$\large y(n)=F(x(n))$$

then we have, for a delay of $n_0$:

$$\large
y(n+n_0)=F(x(n+n_0))
$$

which means our function **stays the same no matter when** we apply it.

## FIR Filters

```python
%%html

```

A simple Finite Impulse Response (**FIR**) filter has a difference equation like the following, with x(n) the input of our filter, and y(n) its output:

$$\large
y(n)=\sum _ {m=0}^ L b(m) x(n-m)
$$

Observe that this is the **convolution** of the signal x(n) with b(n). Here, the b(m) are the coefficients of the filter, or its **impulse response**. These are commonly also referred to as **“taps”**, because this system can be viewed as “tapping” a delay line, as seen in the picture below.

This difference equation is also how usually filters are implemented in Matlab or Python and other programming languages. A typical block diagram of an FIR filter is as follows:

(From: http://en.wikipedia.org/wiki/Finite_impulse_response)

Observe that here the blocks with $z^{-1}$ are implemented with a delay by 1 sampling interval, not a multiplication with $z^{-1}$, as we would in the z-domain!

After the first delay block $z^{-1}$ we have $x[n-1]$, after the second delay block we have $x[n-2]$, and so on. Each delay block “memorizes” the value from the left for one sample clock cycle, and releases it to the right at the next sample clock cycle.
Hence they delay samples by 1 sample clock cycle.

The z-transform of our difference equation of our convolution

$$\large
y(n)=\sum _ {m=0} ^L b(m) x(n-m)
$$

 is (using the linearty of the z-Transform):

$$\large
Y(z)= \sum _{m=0} ^ L b(m) \cdot z^{-m} \cdot X(z)= X(z) \cdot \sum _ {m=0} ^ L b(m) \cdot z^{-m}
$$

Now we can compute the **transfer function**, defined as the output divided by the input,

$$\large
H(z):= \frac{Y(z)} {X(z)}=\sum _ {m=0} ^ L b(m) \cdot z^{-m}
$$

**Observe**  that this is the z-transform of the coefficients b(m)! This is the **z-transform of the impulse response** of the FIR filter!

Now we can obtain the **frequency response** (so that we can see which frequencies are attenuated and which are not) from our transfer function of the filter by just replacing $z$ by $e^{j \Omega}$:

$$\large
H(e^{j\Omega})=\sum _{m=0} ^L b(m) \cdot e^{-j\Omega \cdot m}
$$

Since $e^{j\Omega}$ is a complex number, our frequency response $H$ is also complex. Hence is a complex number for each frequency $\Omega$. Usually it is plotted as a **magnitude** plot and a **phase** plot over frequency. Its magnitude tells us the attenuation at each frequency, and the phase its phase shift for each frequency. Using those 2 plots or properties, we can also design our filters with desired properties (for instance a stop-band at given frequencies). The Matlab and Python  function to generate a magnitude and phase plot of a transfer function or signal is “freqz”, which we already saw.

**Recommended**: https://github.com/GuitarsAI/MRSP_Notebooks - 03 Frequency Response

```python
%%html

```

## IIR Filters

```python
%%html

```

Their difference equation is:

$$\large
\begin{equation}
y(n)= \sum _{m=0} ^L b(m) \cdot x(n-m)+ \sum _ {r=1} ^R a(r) \cdot y(n-r)
\end{equation}$$

(See also: Oppenheim, Schafer: “Discrete Time Signal Processing”, Chapter 6 in Ed. 1989)

Here we have **2 convolutions**. Observe the feedback from the output y back to the input in this sum. Also observe that the feedback part starts with a delay of r=1. This is because we want to avoid so-called delayless loops. We cannot use the value y(n) before we computed it! Again, this difference equation is the usual implementation using Matlab or Octave or Python.

The following image shows the corresponding block diagram of our filter,

Again, here the boxes with $z^{-1}$ symbolize a delay of 1 sampling period, and the triangles symbolize a multiplication with the factor written next to it.

We can simplify this structure by combining the summations,

The z-transform of its difference equation (1) is

$$ \large
Y(z)= \sum _{m=0} ^ L b(m) \cdot X(z) \cdot z^{-m} + \sum _{r=1} a(r) \cdot Y(z) \cdot z^{-r}$$

**Observe:** **Matlab and Octave** and **Python scipy.signal.lfilter** are defining the coefficients a with **opposite signs** as we and Oppenheim/Schafer are defining. See for instance “help filter” or “help(scipy.signal.lfilter)” in Python.

To obtain its transfer function, we first move the Y(z) to one side:

$$\large
Y(z)(1-\sum _ {r=1} ^R a(r) \cdot z^{-r})= X(z) \cdot \sum _{m=0} ^ L b(m) \cdot  z^{-m}
$$

Hence the resulting transfer function is

$$ \large
H(z)=\frac{Y(z)}  {X(z)}= \frac{\sum _ {m=0} ^L b(m)  \cdot z^{-m}}{1-\sum _ {r=1} ^ R a(r) \cdot z^{-r}}
$$

**Observe:** With the help of the z-Transform we were able to find a closed form solution for the transfer function, even though we have a feedback loop in our system! This is a **big advantage** for the z-Transform.

Here we can see that we obtain a polynomial in the **denominator** of our transfer function, and hence poles. And the filter can potentially become unstable! The zeros of this denominator polynomial become the poles of the transfer function. If these poles are all **inside the unit circle**, we have a **stable filter**! This also shows that we just need to design our coefficients a(n) such that the **poles are inside the unit circle for stabiltity**.

## Combined FIR-IIR Structure used in the Python “lfilter” Function:

```python
%%html

```

Since a delay is a linear operator, we can shift it after a summation, and hence can combine the delay chain for the FIR and IIR part. This reduces the memory requirement for an implementation, and leads to the following structure,

## Filter Example

```python
%%html

```

Going back to our simple example of an exponential decaying signal, this shows how to implement it. We just need a system with a pole at position p. In the above equation we obtain it by setting b(0)=1 and a(1)=p. Hence we obtain a simple difference equation

$$\large
y(n)=1\cdot x(n)+p \cdot y(n-1)$$

Here you can see: if x(n) is the unit pulse, the output is the exponential decaying sequence:

$$\large
1,p,p^{2},p^{3},...
$$

which is an infinitely long impulse response. Hence the name IIR.

This can also be written in the form of a block diagram:

(also compare it with the Python implementaion block diagram above, for $b[0]=1$ and $a[1]=p$).

In the z-domain this is

$$\large
Y(z)=X(z)+p \cdot z^{-1} \cdot Y(z)$$

$$\large
\rightarrow H(z)=\frac{Y(z)}{X(z)}= \frac{1}  {1-p \cdot z^{-1}}
$$

In this structure we can now see the feedback loop. Observe that this is the same as the z-Transform of our exponential series before. This means, when we transform this back to the time domain, we obtain an exponential function, which is the **filters impulse response**, which confirms what we just saw by using an impulse in the time domain for our filter. So the result of the inverse z-Transform of our transfer function is indeed,

$$\large
1,p,p^{2},p^{3},...
$$.

### Computing the Resulting Frequency Response

**Example:** The Matlab or Octave or Python function “freqz” (we have a Python freqz function also in Moodle) can be used to plot the magnitude and phase plot of the transfer function of this filter. Its input are directly the coefficients a and b of the transfer function H(z), in the form:  ```freqz(B,A)```, where B and A are vectors containing the coefficients.

If we choose a(1)=p=0.9 in our example, we obtain $A=[1,-0.9]$ and $B=[1].$

```python
%%html

```

```python
%matplotlib notebook
from scipy.signal import freqz
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.ticker import FuncFormatter, MultipleLocator
```

```python
A=np.array([1,-0.9])
B=np.array([1])
w, h = freqz(B, A, worN=512, whole=False)
```

```python
fig = plt.figure()
plt.title('Digital filter frequency response')
plt.subplot(2,1,1)
plt.plot(w, 20 * np.log10(abs(h)), 'b')
plt.ylabel('Amplitude (dB)')
plt.xlabel('Normalized Frequency')
plt.grid()
ax = plt.gca()
ax.xaxis.set_major_formatter(FuncFormatter(
   lambda val,pos: '{:.0g}$\pi$'.format(val/np.pi) if val !=0 else '0'
))
ax.xaxis.set_major_locator(MultipleLocator(base=0.5))
plt.subplot(2,1,2)
angles = np.angle(h)
plt.plot(w, angles, 'g')
plt.ylabel('Angle (radians)')
plt.xlabel('Normalized Frequency')
ax = plt.gca()
ax.xaxis.set_major_formatter(FuncFormatter(
   lambda val,pos: '{:.0g}$\pi$'.format(val/np.pi) if val !=0 else '0'
))
ax.xaxis.set_major_locator(MultipleLocator(base=0.5))
plt.grid()
plt.tight_layout()

```

    <IPython.core.display.Javascript object>

Observe that the horizontal axis is the normalized frequency (see last lecture), its right hand side is $\pi$, which is the Nyquist frequency or half the sampling frequency.

The frequency response we see here has a low pass characteristic.

We can use the command “zplane” (also in Moodle) to plot the location of the zeros and poles in the complex z-plane.
For that we first need to calculate the pole an zero positions with Pythons function np.roots.

We have
$$\large
H(z) = \dfrac{1}{1-p \cdot z^{-1}} $$

and roots expects as argument the ordered coefficients of our polinomial in $z^-1$ (which makes it compatible with *lfilter*). Hence our pole positions are the `np.roots(a)` and the zeros at `np.roots(b)`.

```python
def zplane(B,A):
    # Plot Poles and Zeros
    from  matplotlib import patches
    plt.figure(figsize=(6,4))
    ax = plt.subplot(111)
    r = 1.5; plt.axis('scaled'); plt.axis([-r, r, -r, r])
    #ticks = [-1, 1]; plt.xticks(ticks); plt.yticks(ticks)
    # Unit Circle
    uc = patches.Circle((0,0), radius=1, fill=False, color='black', ls='dashed')
    ax.add_patch(uc)
    ax.spines['left'].set_position('center')
    ax.spines['bottom'].set_position('center')
    ax.spines['right'].set_visible(False)
    ax.spines['top'].set_visible(False)
    plt.xlabel('Re', horizontalalignment='right', x=1.0)
    plt.ylabel('Im',  y=1.0)
    plt.title('Complex z-Plane', loc='right')
    plt.grid()
    plt.plot(np.real(A),np.imag(A),'rx')
    plt.plot(np.real(B),np.imag(B),'bo')
```

```python
zplane(np.roots(B),np.roots(A))
```

    <IPython.core.display.Javascript object>

**Zeros** are marked with an **“o”**, and **poles** are marked with an **“x”**. Here we see the pole at location z=0.9.

In general, the **closer a pole** is to the **unit circle**, the larger is the corresponding **peak** in the **magnitude** of the frequency response at a **normalized** frequency identical to the **angle of the pole** to the origin.

This can be seen by turning the z-transform into a DTFT by replacing z by $e^{j\Omega}$. Hence the frequency response is obtained by “running” on this unit circle, $\Omega$ is our angle and also normalized frequency, and the closer we get to a pole, the higher the magnitude of the frequency response becomes. This is opposite for zeros, the closer we come to a zero, the smaller magnitude of the frequency response becomes.

Here our pole has an angle of 0 degrees. In the example we can indeed observe a peak in the magnitude response at normalized frequency $\Omega=0$ above.

The **filtering** operation itself works similarly in Matlab or Octave or Python, in the time domain. The function is “filter” or “lfilter”. Given an input signal in the vector x, and filter coefficients in vectors A and B, the filtered output y of our filter is simply:

```y=lfilter(B,A,x);```

Often used orders for the zeros and poles L and R are a few to a few dozen coefficients.

**Python example:**

Use the function *`lfilter`* to obtain the impulse response of this IIR filter.

```python
%%html

```

```python
import scipy.signal
import matplotlib.pyplot as plt
#Start with a unit pulse as input x:
x=np.zeros(30)
x[0] = 1
#B and A are given as before:
A=[1,-0.9];
B=[1];
#Now calculate the impulse response:
y=scipy.signal.lfilter(B,A,x);
plt.figure()
plt.stem(y, use_line_collection=True)
plt.xlabel('Sample')
plt.ylabel('Value')
plt.grid()

```

    <IPython.core.display.Javascript object>

Here we can see the indeed exponetial decaying function (the sequence $ir(n)=p^{n}$ for p=0.9). In this way we can also test more complicated IIR filters.

This exponential decaying impulse response again shows the stability of the filter, which was to be expected because the pole of its transfer function in the z-domain is placed at z=0.9, and hence inside the unit-circle!

```python

```


# Seasonality in time series data

Consider the problem of modeling time series data with multiple seasonal components with different perioidicities.  Let us take the time series $y_t$ and decompose it explicitly to have a level component and two seasonal components.

$$
y_t = \mu_t + \gamma^{(1)}_t + \gamma^{(2)}_t
$$

where $\mu_t$ represents the trend or level, $\gamma^{(1)}_t$ represents a seasonal component with a relatively short period, and $\gamma^{(2)}_t$ represents another seasonal component of longer period. We will have a fixed intercept term for our level and consider both $\gamma^{(2)}_t$ and $\gamma^{(2)}_t$ to be stochastic so that the seasonal patterns can vary over time.

In this notebook, we will generate synthetic data conforming to this model and showcase modeling of the seasonal terms in a few different ways under the unobserved components modeling framework.

```python
%matplotlib notebook
```

```python
import numpy as np
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
```

### Synthetic data creation

We will create data with multiple seasonal patterns by following equations (3.7) and (3.8) in Durbin and Koopman (2012).  We will simulate 300 periods and two seasonal terms parameterized in the frequency domain having periods 10 and 100, respectively, and 3 and 2 number of harmonics, respectively.  Further, the variances of their stochastic parts are 4 and 9, respecively.

```python
# First we'll simulate the synthetic data
def simulate_seasonal_term(periodicity, total_cycles, noise_std=1.,
                           harmonics=None):
    duration = periodicity * total_cycles
    assert duration == int(duration)
    duration = int(duration)
    harmonics = harmonics if harmonics else int(np.floor(periodicity / 2))

    lambda_p = 2 * np.pi / float(periodicity)

    gamma_jt = noise_std * np.random.randn((harmonics))
    gamma_star_jt = noise_std * np.random.randn((harmonics))

    total_timesteps = 100 * duration # Pad for burn in
    series = np.zeros(total_timesteps)
    for t in range(total_timesteps):
        gamma_jtp1 = np.zeros_like(gamma_jt)
        gamma_star_jtp1 = np.zeros_like(gamma_star_jt)
        for j in range(1, harmonics + 1):
            cos_j = np.cos(lambda_p * j)
            sin_j = np.sin(lambda_p * j)
            gamma_jtp1[j - 1] = (gamma_jt[j - 1] * cos_j
                                 + gamma_star_jt[j - 1] * sin_j
                                 + noise_std * np.random.randn())
            gamma_star_jtp1[j - 1] = (- gamma_jt[j - 1] * sin_j
                                      + gamma_star_jt[j - 1] * cos_j
                                      + noise_std * np.random.randn())
        series[t] = np.sum(gamma_jtp1)
        gamma_jt = gamma_jtp1
        gamma_star_jt = gamma_star_jtp1
    wanted_series = series[-duration:] # Discard burn in

    return wanted_series
```

```python
duration = 100 * 3
periodicities = [10, 100]
num_harmonics = [3, 2]
std = np.array([2, 3])
np.random.seed(8678309)

terms = []
for ix, _ in enumerate(periodicities):
    s = simulate_seasonal_term(
        periodicities[ix],
        duration / periodicities[ix],
        harmonics=num_harmonics[ix],
        noise_std=std[ix])
    terms.append(s)
terms.append(np.ones_like(terms[0]) * 10.)
series = pd.Series(np.sum(terms, axis=0))
df = pd.DataFrame(data={'total': series,
                        '10(3)': terms[0],
                        '100(2)': terms[1],
                        'level':terms[2]})
h1, = plt.plot(df['total'])
h2, = plt.plot(df['10(3)'])
h3, = plt.plot(df['100(2)'])
h4, = plt.plot(df['level'])
plt.legend(['total','10(3)','100(2)', 'level'])
plt.show()
```

### Unobserved components (frequency domain modeling)

The next method is an unobserved components model, where the trend is modeled as a fixed intercept and the seasonal components are modeled using trigonometric functions with primary periodicities of 10 and 100, respectively, and number of harmonics 3 and 2, respecively.  Note that this is the correct, generating model. The process for the time series can be written as:

$$
\begin{align}
y_t & = \mu_t + \gamma^{(1)}_t + \gamma^{(2)}_t + \epsilon_t\\
\mu_{t+1} & = \mu_t \\
\gamma^{(1)}_{t} &= \sum_{j=1}^2 \gamma^{(1)}_{j, t} \\
\gamma^{(2)}_{t} &= \sum_{j=1}^3 \gamma^{(2)}_{j, t}\\
\gamma^{(1)}_{j, t+1} &= \gamma^{(1)}_{j, t}\cos(\lambda_j) + \gamma^{*, (1)}_{j, t}\sin(\lambda_j) + \omega^{(1)}_{j,t}, ~j = 1, 2, 3\\
\gamma^{*, (1)}_{j, t+1} &= -\gamma^{(1)}_{j, t}\sin(\lambda_j) + \gamma^{*, (1)}_{j, t}\cos(\lambda_j) + \omega^{*, (1)}_{j, t}, ~j = 1, 2, 3\\
\gamma^{(2)}_{j, t+1} &= \gamma^{(2)}_{j, t}\cos(\lambda_j) + \gamma^{*, (2)}_{j, t}\sin(\lambda_j) + \omega^{(2)}_{j,t}, ~j = 1, 2\\
\gamma^{*, (2)}_{j, t+1} &= -\gamma^{(2)}_{j, t}\sin(\lambda_j) + \gamma^{*, (2)}_{j, t}\cos(\lambda_j) + \omega^{*, (2)}_{j, t}, ~j = 1, 2\\
\end{align}
$$
$$

where $\epsilon_t$ is white noise, $\omega^{(1)}_{j,t}$ are i.i.d. $N(0, \sigma^2_1)$, and  $\omega^{(2)}_{j,t}$ are i.i.d. $N(0, \sigma^2_2)$, where $\sigma_1 = 2.$

```python
model = sm.tsa.UnobservedComponents(series.values,
                                    level='fixed intercept',
                                    freq_seasonal=[{'period': 10,
                                                    'harmonics': 3},
                                                   {'period': 100,
                                                    'harmonics': 2}])
res_f = model.fit(disp=False)
print(res_f.summary())
# The first state variable holds our estimate of the intercept
print("fixed intercept estimated as {0:.3f}".format(res_f.smoother_results.smoothed_state[0,-1:][0]))

res_f.plot_components()
plt.show()

```

```python
model.ssm.transition[:, :, 0]
```

Observe that the fitted variances are pretty close to the true variances of 4 and 9.  Further, the individual seasonal components look pretty close to the true seasonal components.  The smoothed level term is kind of close to the true level of 10.  Finally, our diagnostics look solid; the test statistics are small enough to fail to reject our three tests.

### Unobserved components (mixed time and frequency domain modeling)

The second method is an unobserved components model, where the trend is modeled as a fixed intercept and the seasonal components are modeled using 10 constants summing to 0 and trigonometric functions with a primary periodicities of 100 with 2 harmonics total.  Note that this isn't the generating model, as it presupposes that there are more state errors for the shorter seasonal component than in reality. The process for the time series can be written as:

$$
\begin{align}
y_t & = \mu_t + \gamma^{(1)}_t + \gamma^{(2)}_t + \epsilon_t\\
\mu_{t+1} & = \mu_t \\
\gamma^{(1)}_{t + 1} &= - \sum_{j=1}^9 \gamma^{(1)}_{t + 1 - j} + \omega^{(1)}_t\\
\gamma^{(2)}_{j, t+1} &= \gamma^{(2)}_{j, t}\cos(\lambda_j) + \gamma^{*, (2)}_{j, t}\sin(\lambda_j) + \omega^{(2)}_{j,t}, ~j = 1, 2\\
\gamma^{*, (2)}_{j, t+1} &= -\gamma^{(2)}_{j, t}\sin(\lambda_j) + \gamma^{*, (2)}_{j, t}\cos(\lambda_j) + \omega^{*, (2)}_{j, t}, ~j = 1, 2\\
\end{align}
$$

where $\epsilon_t$ is white noise, $\omega^{(1)}_{t}$ are i.i.d. $N(0, \sigma^2_1)$, and  $\omega^{(2)}_{j,t}$ are i.i.d. $N(0, \sigma^2_2)$.

```python
model = sm.tsa.UnobservedComponents(series,
                                    level='fixed intercept',
                                    seasonal=10,
                                    freq_seasonal=[{'period': 100,
                                                    'harmonics': 2}])
res_tf = model.fit()
print(res_tf.summary())
# The first state variable holds our estimate of the intercept
print("fixed intercept estimated as {0:.3f}".format(res_tf.smoother_results.smoothed_state[0,-1:][0]))

res_tf.plot_components()
plt.show()
```

The plotted components look good.  However, the estimated variance of the second seasonal term is inflated from reality.  Additionally, we reject the Ljung-Box statistic, indicating we may have remaining autocorrelation after accounting for our components.

### Unobserved components (lazy frequency domain modeling)

The third method is an unobserved components model with a fixed intercept and one seasonal component, which is modeled using trigonometric functions with primary periodicity 100 and 50 harmonics. Note that this isn't the generating model, as it presupposes that there are more harmonics then in reality.  Because the variances are tied together, we are not able to drive the estimated covariance of the non-existent harmonics to 0.  What is lazy about this model specification is that we have not bothered to specify the two different seasonal components and instead chosen to model them using a single component with enough harmonics to cover both.  We will not be able to capture any differences in variances between the two true components.  The process for the time series can be written as:

$$
\begin{align}
y_t & = \mu_t + \gamma^{(1)}_t + \epsilon_t\\
\mu_{t+1} &= \mu_t\\
\gamma^{(1)}_{t} &= \sum_{j=1}^{50}\gamma^{(1)}_{j, t}\\
\gamma^{(1)}_{j, t+1} &= \gamma^{(1)}_{j, t}\cos(\lambda_j) + \gamma^{*, (1)}_{j, t}\sin(\lambda_j) + \omega^{(1}_{j,t}, ~j = 1, 2, \dots, 50\\
\gamma^{*, (1)}_{j, t+1} &= -\gamma^{(1)}_{j, t}\sin(\lambda_j) + \gamma^{*, (1)}_{j, t}\cos(\lambda_j) + \omega^{*, (1)}_{j, t}, ~j = 1, 2, \dots, 50\\
\end{align}
$$

where $\epsilon_t$ is white noise, $\omega^{(1)}_{t}$ are i.i.d. $N(0, \sigma^2_1)$.

```python
model = sm.tsa.UnobservedComponents(series,
                                    level='fixed intercept',
                                    freq_seasonal=[{'period': 100}])
res_lf = model.fit()
print(res_lf.summary())
# The first state variable holds our estimate of the intercept
print("fixed intercept estimated as {0:.3f}".format(res_lf.smoother_results.smoothed_state[0,-1:][0]))

res_lf.plot_components()
plt.show()
```

Note that one of our diagnostic tests would be rejected at the .05 level.

### Unobserved components (lazy time domain seasonal modeling)

The fourth method is an unobserved components model with a fixed intercept and a single seasonal component modeled using a time-domain seasonal model of 100 constants. The process for the time series can be written as:

$$
\begin{align}
y_t & =\mu_t + \gamma^{(1)}_t + \epsilon_t\\
\mu_{t+1} &= \mu_{t} \\
\gamma^{(1)}_{t + 1} &= - \sum_{j=1}^{99} \gamma^{(1)}_{t + 1 - j} + \omega^{(1)}_t\\
\end{align}
$$

where $\epsilon_t$ is white noise, $\omega^{(1)}_{t}$ are i.i.d. $N(0, \sigma^2_1)$.

```python
model = sm.tsa.UnobservedComponents(series,
                                    level='fixed intercept',
                                    seasonal=100)
res_lt = model.fit(disp=False)
print(res_lt.summary())
# The first state variable holds our estimate of the intercept
print("fixed intercept estimated as {0:.3f}".format(res_lt.smoother_results.smoothed_state[0,-1:][0]))

res_lt.plot_components()
plt.show()
```

The seasonal component itself looks good--it is the primary singal.  The estimated variance of the seasonal term is very high ($>10^5$), leading to a lot of uncertainty in our one-step-ahead predictions and slow responsiveness to new data, as evidenced by large errors in one-step ahead predictions and observations. Finally, all three of our diagnostic tests were rejected.

### Comparison of filtered estimates

The plots below show that explicitly modeling the individual components results in the filtered state being close to the true state within roughly half a period.  The lazy models took longer (almost a full period) to do the same on the combined true state.

```python
# Assign better names for our seasonal terms
true_seasonal_10_3 = terms[0]
true_seasonal_100_2 = terms[1]
true_sum = true_seasonal_10_3 + true_seasonal_100_2

```

```python
time_s = np.s_[:50]  # After this they basically agree
fig1 = plt.figure()
ax1 = fig1.add_subplot(111)
h1, = ax1.plot(series.index[time_s], res_f.freq_seasonal[0].filtered[time_s], label='Double Freq. Seas')
h2, = ax1.plot(series.index[time_s], res_tf.seasonal.filtered[time_s], label='Mixed Domain Seas')
h3, = ax1.plot(series.index[time_s], true_seasonal_10_3[time_s], label='True Seasonal 10(3)')
plt.legend([h1, h2, h3], ['Double Freq. Seasonal','Mixed Domain Seasonal','Truth'], loc=2)
plt.title('Seasonal 10(3) component')
plt.show()
```

```python
time_s = np.s_[:50]  # After this they basically agree
fig2 = plt.figure()
ax2 = fig2.add_subplot(111)
h21, = ax2.plot(series.index[time_s], res_f.freq_seasonal[1].filtered[time_s], label='Double Freq. Seas')
h22, = ax2.plot(series.index[time_s], res_tf.freq_seasonal[0].filtered[time_s], label='Mixed Domain Seas')
h23, = ax2.plot(series.index[time_s], true_seasonal_100_2[time_s], label='True Seasonal 100(2)')
plt.legend([h21, h22, h23], ['Double Freq. Seasonal','Mixed Domain Seasonal','Truth'], loc=2)
plt.title('Seasonal 100(2) component')
plt.show()
```

```python
time_s = np.s_[:100]

fig3 = plt.figure()
ax3 = fig3.add_subplot(111)
h31, = ax3.plot(series.index[time_s], res_f.freq_seasonal[1].filtered[time_s] + res_f.freq_seasonal[0].filtered[time_s], label='Double Freq. Seas')
h32, = ax3.plot(series.index[time_s], res_tf.freq_seasonal[0].filtered[time_s] + res_tf.seasonal.filtered[time_s], label='Mixed Domain Seas')
h33, = ax3.plot(series.index[time_s], true_sum[time_s], label='True Seasonal 100(2)')
h34, = ax3.plot(series.index[time_s], res_lf.freq_seasonal[0].filtered[time_s], label='Lazy Freq. Seas')
h35, = ax3.plot(series.index[time_s], res_lt.seasonal.filtered[time_s], label='Lazy Time Seas')

plt.legend([h31, h32, h33, h34, h35], ['Double Freq. Seasonal','Mixed Domain Seasonal','Truth', 'Lazy Freq. Seas', 'Lazy Time Seas'], loc=1)
plt.title('Seasonal components combined')
plt.show()
```

##### Conclusions

In this notebook, we simulated a time series with two seasonal components of different periods.  We modeled them using structural time series models with (a) two frequency domain components of correct periods and numbers of harmonics, (b) time domain seasonal component for the shorter term and a frequency domain term with correct period and number of harmonics, (c) a single frequency domain term with the longer period and full number of harmonics, and (d) a single time domain term with the longer period.  We saw a variety of diagnostic results, with only the correct generating model, (a), failing to reject any of the tests.  Thus, more flexible seasonal modeling allowing for multiple components with specifiable harmonics can be a useful tool for time series modeling.  Finally, we can represent seasonal components with fewer total states in this way, allowing for the user to attempt to make the bias-variance tradeoff themselves instead of being forced to choose "lazy" models, which use a large number of states and incur additional variance as a result.

```python

```


# Simulating Vibratory Systems with Non-Linearities

This notebook requires [`eng122.py`](https://moorepants.github.io/eng122/materials/notebooks/eng122.py). Download that file to the same folder as this notebook.

## Introduction

In this notebook, we'll explore numerically simulating the motion of **non-linear** systems for which we can't (or can't easily) derive the exact solutions analytically. After completing the notebook, you will be able to:

- Manipulate second-order differential equations (equations of motion) to obtain corresponding *systems of first-order differential equations*
- Become familiar with the programming interface for numerically integrating systems of first-order ODEs
- Simulate the free response of systems with non-linear characteristics

### Systems of First-Order ODEs

To simulate non-linear vibratory systems, we manipulate the equation(s) of motion into a specific form that allows generic numerical integration routines to work. This is achieved by selecting *state variables* for which we can write first-order ODEs corresponding to the equations of motion. That is, the goal is to write a system of first-order differential equations of the form:

$$\begin{align} \dot{x}_1 &= f(x_1,\, x_2,\, \ldots) \\ \dot{x}_2 &= g(x_1,\, x_2,\, \ldots) \\ \vdots \end{align}$$

When we derive the equations of motion using the Newton-Euler method, we get second-order differential equations. For one-dimensional motion, a simple choice of variables is $x_1 = x$ (position) and $x_2 = \dot{x}$ (velocity). Then we get $\dot{x}_1 = x_2$ and $\dot{x}_2 = \ddot{x}$, so we can write our system of equations as

$$\begin{align} \dot{x}_1 &= x_2 \\ \dot{x}_2 &= \frac{1}{m} \sum F \end{align}$$

The first equation is trivial (because of our selection of state variables), and the second equation is just $F = ma$.

### Numerically Integrating First-Order ODEs

In our code, we define a Python function that takes:

- the current values of our state variables ($x_1,\, x_2,\, \ldots$) in an array
- the current time value
- (optional) additional parameters such as masses, spring constants, etc.

The function uses these inputs to compute the derivatives of the state variables $\left(\dot{x}_1(t),\,\dot{x}_2(t),\,\ldots)\right)$ and return them. This function is passed to an ordinary differential equation solver with the initial conditions and an array of time values, and the solver repeatedly runs our function and numerically integrates to evaulate the state variables at the requested time points.

### Example: Linear mass-spring-damper system

As an example, let's simulate the mass-spring-damper system with linear spring and damping elements. The equation of motion is:

$$m\ddot{x} + c\dot{x} + k x = 0$$

Since we're dealing with 1D motion, our selection of state variables is straightforward: $x$ (position) and $v = \dot{x}$ (velocity). In this case, our state equations are:

$$\begin{align} \dot{x} &= v \\ \dot{v} &= -\frac{k}{m}x - \frac{c}{m} v \end{align}$$

Let's start with some imports.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# this is a Python module providing an integration routine
from eng122 import rk4int
```

Now let's write the function that returns the derivatives of the state variables. The function takes the current values of the state variables (as an array), the current time, and any additional parameters we need (system parameters $m$, $k$, and $c$).

```python
def msd_eqns(current_state, time, m, k, c):
    """Equations for a linear mass-spring-damper system"""
    # unpack the state variables array
    x, v = current_state

    # compute the derivatives
    xdot = v
    vdot = -(k/m)*x - (c/m)*v

    # pack the state derivatives together to return (same order as current_state!)
    state_derivative = xdot, vdot

    return state_derivative
```

Now we can use an integration routine (here we'll use the Runge-Kutta method provided as `rk4int`) to integrate the equations we specified at a set of time values. Here's what `rk4int` expects:

```python
help(rk4int)
```

    Help on function rk4int in module eng122:

    rk4int(func, x0, t, args=None)
        4th-order Runge-Kutta integration.

        Parameters
        ----------
        func : callable(x, t, *args)
            Function that returns the derivatives of the state variables at time t.
        x0 : array_like, shape(n,)
            Initial values of the state variables.
        t : array, shape(m,)
            Array of time values at which to solve.
        args : tuple, optional
            Additional arguments to `func`.

        Returns
        -------
        x : ndarray, shape(m, n)
            Array containing the values of the state variables at the
            specified time values in `t`.

So we set things up by specifying the system parameters ($m$, $k$, and $c$), initial values of the state variables (position and velocity), and an array of time values to simulate over. Each state variable is then a column of the output of `rk4int`.

```python
# parameters
m = 1
k = 1000
c = 10

# initial conditions
x0 = 5
v0 = 0

# array of time values
t = np.arange(0, 1, 0.01)

# call integration routine
state_trajectory = rk4int(msd_eqns, [x0, v0], t, args=(m, k, c))
#                         |         |         |  |
#                         |         |         |  |_additional parameters for msd_eqns
#                         |         |         |
#                         |         |         |_time array
#                         |         |
#                         |         |_initial conditions
#                         |
#                         |_function returning state variable derivatives

# unpack the columns of the state variable array
x = state_trajectory[:, 0]
v = state_trajectory[:, 1]

plt.plot(t, x)
plt.xlabel('$t$')
plt.ylabel('$x(t)$')
```

## Exercise 1: Coulomb Friction

A common source of nonlinearity is Coulomb friction, which is specific type of damping. In viscous damping, there is a force that is linearly proportional to the velocity:

$$F_v = c \dot{x}$$

This simple source of energy dissipation allows us to derive the exact solution to the equations of motion for a mass-spring-damper system. With Coulomb friction, the damping force takes the form:

$$F_c = \begin{cases} -\mu N & \dot{x} > 0 \\ 0 & \dot{x} = 0 \\ \mu N & \dot{x} < 0 \end{cases}$$

where $\mu$ is the coefficient of sliding friction and $N$ is the normal force. Here the damping force is constant, always working against the motion of the system.

The form of the equation of motion is the same as in the linear damping case, where we have

$$m \ddot{x} + F_{\text{damping}} + F_{\text{spring}} = 0$$

But now the equation will have a nonlinearity because the damping force depends on the sign of $\dot{x}$:

$$m \ddot{x} + \mu m g \,\text{sgn}\left( \dot{x} \right) + kx = 0$$

Fill in the function definition below to compute the derivatives of the state variables $x$ and $v = \dot{x}$.

```python
def coulomb_eqns(state_vars, t, m, k, mu):
    """Returns the derivative of the states.

    Parameters
    ==========
    state_vars : array_like, shape(n,)
        A 1D array containing the current state values.
    t : float
        A value for the current time.
    m : float
        A value for the mass.
    k : float
        A value for the stiffness.
    mu : float
        A value for the coefficient of kinetic friction.

    Returns
    =======
    xdot : ndarray, shape(n,)
        The derivative of the states at the current time.
    """
    x, v = state_vars

    xdot = v
    vdot = -mu*9.81*np.sign(v) - (k/m)*x

    return xdot, vdot
```

Use the function above to simulate a mass-spring system with Coulomb friction using the following parameters:

- $m = 1000$ kg
- $k = 5000$ N/m
- $\mu = 0.2$
- duration: 10 seconds

Use zero initial velocity and plot the position of the mass for two different initial positions: 5 m and 4.5 m. Add a grid to the plot and take note of where the system comes to rest.

```python
# system parameters
m = 1000 # mass [kg]
k = 5000 # spring stiffness [N/m]
mu = 0.2 # coulomb friction coefficient
d = 10   # duration [s]

# initial conditions
xinits = np.array([[5, 0],    # simulation 1
                   [4.5, 0]]) # simulation 2
linestyles = ['-', '--']

# time array with a fairly small time step
t = np.arange(0, 10, 0.01)

for (x0, v0), linestyle in zip(xinits, linestyles):
    state_trajectory = rk4int(coulomb_eqns, [x0, v0], t, args=(m, k, mu))
    x = state_trajectory[:, 0]
    v = state_trajectory[:, 1]
    plt.plot(t, x, linestyle=linestyle,
             color='k', label="x0 = {:.1f}m".format(x0))

plt.xlabel(r'$t$ [s]')
plt.ylabel(r'$x(t)$ [m]')
plt.legend()
plt.grid()
```

## Questions

1) This system behaves similiarly to a vicously damped system, but there are some differences. What differences can you find when exploring the simulation? Think about how the oscillation decays over time and any unexpected behavior. You may want to plot both state variables to see more information. Describe these differences in the cell below.

2) With $v_0=0$ try to find the value of $x_0$ that bounds the equilibrium range by simulating the system with various initial positions. What is the significance of this boundary? What does it represent?

## Exercise 2: Pendulum

In class, we've looked at the simple pendulum, which has the following equation of motion:

$$\ddot{\theta} + \frac{g}{l} \sin \theta = 0$$

To solve the equation of motion for this system, we linearized to get rid of a $\sin\theta$ term. This was done by noting that for small angles, $\sin\theta \approx \theta$. With this approximation, we find that the system oscillates at a natural frequency of $\omega_n = \sqrt{\frac{g}{l}}$. Let's see what happens if we do not make the small angle approximation.

Start by writing the function to return the (non-linear) derivatives of the state variables $\theta$ and $\omega = \dot{\theta}$.

```python
def pend_eqns(state_vars, t, l, g):
    """Returns the derivative of the states.

    Parameters
    ==========
    state_vars : array_like, shape(n,)
        A 1D array containing the current state values: theta and omega.
    t : float
        A value for the current time.
    l : float
        A value for the length of the pendulum.
    g : float
        A value for the acceleration due to gravity.

    Returns
    =======
    xdot : ndarray, shape(n,)
        The derivative of the states at the current time, i.e. theta' and omega'.

    """
    theta, omega = state_vars

    thetadot = omega
    omegadot = - g / l * np.sin(theta)

    return thetadot, omegadot
```

Use `interact` to plot $\theta(t)$ such that you can interactively manipulate the initial angle between 0 and 90 degrees (use zero initial velocity). Use a value of $l$ which makes the linearized system oscillate at 1 Hz and simulate for 4 seconds.

In addition to the response of the non-linear system, plot the response of the linearized system as well. Since we're using zero initial velocity, you may use the following equation to compute the response of the linearized system:

$$\theta_l(t) = \theta_0 \cos\left(\sqrt{\frac{g}{l}} t\right)$$

```python
from ipywidgets import interact

# gravitational constant
g = 9.81
# oscillation frequency in Hz
f = 1

# convert frequency to rad/s
omega_n = 2 * np.pi * f
# compute length to get desired natural frequency
l = g / omega_n**2

t = np.arange(0, 10, 0.01)

def simulate(ang_init=10):
    # linearized response
    theta_lin = ang_init * np.cos(np.sqrt(g / l)*t)

    # non-linear response
    x = rk4int(pend_eqns, [np.deg2rad(ang_init), 0], t, args=(l, g))
    x = np.rad2deg(x)
    theta_nonlin = x[:, 0]

    plt.plot(t, theta_lin, color='k', linestyle='-', label='linear')
    plt.plot(t, theta_nonlin, color='k', linestyle='--', label='nonlinear')
    plt.ylim(-90, 90)
    plt.xlabel(r'$t$ [s]')
    plt.ylabel(r'$\theta(t)$ [deg]')
    plt.legend()
    plt.grid()

interact(simulate, ang_init=(0, 90, 5))
```

## Question

Describe what happens when the initial angle is large, i.e. when the small angle approximation no longer holds?

# Exercise 3: Unstable Pendulum

In class we derived the non-linear and linear equations of motion for a pendulum that was held veritcal by two springs in series. We predicted that it would be unstable if:

$$
\frac{k}{2 m} < \frac{g}{l}
$$

Here is the second order non-linear equation of motion:

$$
g l m \operatorname{sin}\left(\theta\right) - \frac{k l^{2}}{2} \operatorname{sin}\left(\theta\right) \operatorname{cos}\left(\theta\right) - l^{2} m \ddot{\theta} = 0
$$

Simulate this non-linear system along with the linear system using $m = 1.0, l=1.0, g=9.81$ and various values of $k$ that will cause the system to be stable and/or unstable. Initial conditions can be $\theta=\pi/10,\dot{\theta}=0$.

```python
def unstable_pend_eqns(state_vars, t, m, l, g):
    """Returns the derivative of the states.

    Parameters
    ==========
    state_vars : array_like, shape(n,)
        A 1D array containing the current state values: theta and omega.
    t : float
        A value for the current time.
    m : float
        A value for the mass of the pendulum.
    l : float
        A value for the length of the pendulum.
    g : float
        A value for the acceleration due to gravity.

    Returns
    =======
    xdot : ndarray, shape(n,)
        The derivative of the states at the current time, i.e. theta' and omega'.

    """
    theta, omega = state_vars

    thetadot = omega
    omegadot = (1 / m*l**2) * \
        (g*l*m*np.sin(theta) - 0.5*k*l**2*np.sin(theta)*np.cos(theta))

    return thetadot, omegadot
```

Here we'll just set up the system parameters.

```python
m = 1.0  # mass [kg]
l = 1.0  # length [m]
g = 9.81 # gravitational constant

# intial conditions
theta_0 = np.pi / 10
omega_0 = 0

# array of time values to simulate over
t = np.arange(0, 10, 0.01)
```

Now we can specify some values of $k$ that will result in a stable and unstable system. To do this, we can use the stability criterion $\frac{k}{2 m} < \frac{g}{l}$ to find a critical value of $k_c = \frac{2mg}{l}$ with coefficients greater than or less than 1 to control whether or not the criterion holds.

We'll create a set of $k$ values for each condition (stable and unstable) and plot them on separate axes.

```python
k_crit = 2 * m * g / l

k_stable = np.array([2, 1.5]) * k_crit
k_unstable = np.array([0.5, 0.1]) * k_crit

k_sets = [k_stable, k_unstable]
fig, axes = plt.subplots(nrows=2, sharex=True)
label = ['stable', 'unstable']

for k_set, ax, label in zip(k_sets, axes, label):
    for k in k_set:
        x = rk4int(unstable_pend_eqns, [theta_0, omega_0], t, args=(m, l, g))
        x = np.rad2deg(x)
        theta = x[:, 0]

        ax.set_title(label)
        ax.plot(t, theta)
        ax.set_ylabel(r'$\theta(t)$ [deg]')

axes[-1].set_xlabel(r'$t$ [s]')
```

## Question

1) Is the system unstable with specific values of $k$? If so what are those values?

2) Does this do what you expect it should do? Make sure to plot the state trajectories to see what is going on.


# Time evolution of the entanglement entroy of a product state

This project is part of the Computational Physics course held at Perimeter, Fall 2018.

```julia
using LinearAlgebra
using Arpack
using Plots
```

The following functions are used to compute the entanglement entropy, given a state separated in sub-states A and B.

```julia
function compute_rdmA(PsiAB)
    @assert ndims(PsiAB) == 2 # did we input a matrix?
    @assert size(PsiAB,1) < 2^13 # make sure the resulting matrix is not too large!
    rdm = PsiAB*PsiAB'
end

function compute_entropy(dm)
    p = eigvals(dm) # we diagonalize and use the eigenvalues. the more Julia way
    # has a problem when the log is very large
    entropy = 0.0
    #entropy = - p'*log2.(p) # problems when p[n] is of order 0 (possibly negative)
    for n in 1:size(p,1)
        if abs(p[n]) > 1e-12
            entropy = entropy - p[n]*log2(p[n])
        end
    end
    entropy
end
```

    compute_entropy (generic function with 1 method)

## Product state

We use the outer product of 1 spin states to construct a separable state:

\begin{equation}
| \Psi_{\text{sep}} > = |\Psi>_1 \otimes |\Psi>_2 \otimes \cdots \otimes |\Psi>_N
\end{equation}

```julia
N=2*5 # > 2

function normalizeVec(Vec)
    Vec = Vec/sqrt(abs(Vec'*Vec))
end

# here we implement the outer product
function separable(N)
    PsiInitial= reshape( (randn(2) + im*randn(2))*(randn(2) + im*randn(2))', 2^2 )
    for i in 3:N
        PsiInitial = reshape( PsiInitial*(randn(2) + im*randn(2))', 2^i )
    end
    return normalizeVec(PsiInitial)
end

PsiInitial = separable(N);
```

```julia
# checking separability
nA = Int(N/2) # number spins/qubits in A
nB = Int(N/2) # number of spins/qubits in B

PsiAB = reshape(PsiInitial, (2^(nA),2^(nB)))

rdmA = compute_rdmA(PsiAB)

isapprox(compute_entropy(rdmA), 0, atol=10^-14)  # checking that entropy is zero to 10^-14 accuracy
```

    true

## Time evolution

To obtain $|\Psi(t)\rangle \equiv \exp (-i t H) |\Psi(0)\rangle$, we can use
 $|\Psi(t+\delta t)\rangle = |\Psi(t)\rangle - i \delta t H |\Psi(t)\rangle$. We want to keep the accuracy within $10^{-5}$ at each time step.

We want the code to be able to deal with large spin chains, so we do not want to record the full Hamiltonian matrix.

Here we look at the evolution generated by a modified Ising model Hamiltonian,

\begin{equation}
H = - ( \cos(\theta) \sigma_x \otimes \sigma_x + \sin(\theta)/2 (\sigma_z \otimes I + I \otimes \sigma_z ) ) + 1/2 (\sigma_z \otimes \sigma_y + \sigma_y \otimes \sigma_z)
\end{equation}

```julia
# Ising model
function buildIsing(theta=pi/4) # Ising model with transverse magnetic field h (critical theta=pi/4 by default)
    X = [0. 1; 1 0]
    Y = [0. im; -im 0]
    Z = [1. 0; 0 -1]
    E = diagm(0=>ones(2))
    XX = kron(X,X)
    HXX = XX
    HZ = kron(Z,E) + kron(E,Z)
    H2 = -(cos(theta)*XX + sin(theta)/2*HZ) + 0.5 * ( kron(Z,Y) + kron(Y,Z) )
    return H2
end

# Energy shift
theta = pi/4 # critical magnetic field
H2 = buildIsing(theta) # Hamiltonian
D,U = eigen(H2)
shiftE = D[end]
H2 = H2 - shiftE*diagm(0=>ones(4))

# Sparse multiplication
function multiplyHPsi(Psi, H2)
    L = length(Psi)  # Dimension of the vector space
    N = convert(Int64,log2(L))  # Number of spins
    HPsi = zeros(L)
    for n=1:N                              # This multiplies by the Hamiltonian H
        Psi = reshape(Psi,(4,2^(N-2)))
        HPsi = reshape(HPsi,(4,2^(N-2)))
        HPsi += H2*Psi
        Psi = reshape(Psi,(2,2^(N-1)))
        HPsi = reshape(HPsi,(2,2^(N-1)))
        Psi = permutedims(Psi,(2,1))
        HPsi = permutedims(HPsi,(2,1))
    end
    Psi = reshape(Psi,L)
    HPsi = reshape(HPsi, L)
    return HPsi
end
```

    multiplyHPsi (generic function with 1 method)

## Example: N=12

```julia
# Initialization
init_steps = 1
entropy = zeros(1)
normdiff = zeros(1)
initial_step=1
Nsteps = 0;

N = 2 * 6
Psi = separable(N);
deltat = 0.0001
```

    0.0001

The for loop implements $|\Psi(t+\delta t)\rangle = |\Psi(t)\rangle - i \delta t H |\Psi(t)\rangle$.

```julia
# Loop over the time steps
initial_step += Nsteps
Nsteps = 100000  # change this number if the entanglement entropy has not reached its plateau
final_step = initial_step + Nsteps-1

for n=initial_step:final_step
    Psi = Psi .- im * deltat .* multiplyHPsi(Psi,H2)
    if n%400==0
    normdiff = [normdiff; abs(1 - sqrt(abs(Psi'*Psi)))]
    #Psi = normalizeVec(Psinew)
    PsiAB = reshape(Psi, (2^(Int(N/2)),2^(Int(N/2))))
    rdmA = compute_rdmA(PsiAB)
    entropynew = compute_entropy(rdmA)
    entropy = [entropy; entropynew]
    end
    if n%10000==0 print(n-initial_step+1, ":", Nsteps, " ") end
end
```

    10000:100000 20000:100000 30000:100000 40000:100000 50000:100000 60000:100000 70000:100000 80000:100000 90000:100000 100000:100000

We plot the difference from 1 of the normalization, to check the precision of our time integration.

```julia
tarray = collect(range(0, stop = deltat * Nsteps, length = length(normdiff)))
standard = [10^-5 for i in 1:length(tarray)]
plot(tarray, standard, label = "goal", xlabel = "t", ylabel = "normalization loss")
plot!(tarray, normdiff, label = "numerical", markershape = :hexagon, markerstrokealpha = 0,
    markersize = 2)
```

```julia
savefig("12spins_modIsing_norm.png")
```

## Evolution of the entanglement entropy

We can finally plot the evolution of the entanglement entropy of the half chain with time.

```julia
tarray = collect(range(0, stop = deltat * Nsteps, length = length(normdiff)))
asympt = [N/2 for i in 1:length(tarray)]
plot(tarray, asympt, xlabel = "t", ylabel = "entanglement entropy", label = "N/2")
plot!(tarray, entropy, label = "numerical", markershape = :hexagon, markerstrokealpha = 0,
    markersize = 2)#, markersize = 10, markerstrokewidth = 2)
#(length(tarray),length(entropy))
```

```julia
savefig("12spins_modIsing_ee.png")
```

The entanglement entropy of the initially separable state saturates at $N/2$, as expected.


```python
import numpy as np
from sympy import *
init_printing(use_latex='mathjax')
```

```python
x, h = symbols("x h")
f = Function("f")
f(x).series(x, x0=h, n=3)
```

$$f{\left (h \right )} + \left(- h + x\right) \left. \frac{d}{d \xi_{1}} f{\left (\xi_{1} \right )} \right|_{\substack{ \xi_{1}=h }} + \frac{1}{2} \left(- h + x\right)^{2} \left. \frac{d^{2}}{d \xi_{1}^{2}}  f{\left (\xi_{1} \right )} \right|_{\substack{ \xi_{1}=h }} + \mathcal{O}\left(\left(- h + x\right)^{3}; x\rightarrow h\right)$$

```python
x = symbols('x')
f = exp(x ** 2)
series(f, x, n = 12)
```

$$1 + x^{2} + \frac{x^{4}}{2} + \frac{x^{6}}{6} + \frac{x^{8}}{24} + \frac{x^{10}}{120} + \mathcal{O}\left(x^{12}\right)$$

```python
x,p = symbols('x p')
f = 1 / x
t = series(f, x, x0=p, n = 5)
t.subs({p:4})
```

$$\frac{1}{2} + \frac{1}{64} \left(x - 4\right)^{2} - \frac{1}{256} \left(x - 4\right)^{3} + \frac{1}{1024} \left(x - 4\right)^{4} - \frac{x}{16} + \mathcal{O}\left(\left(x - 4\right)^{5}; x\rightarrow 4\right)$$

```python
x,p = symbols('x p')
f = ln(x)
t = series(f, x, x0=p, n = 2)
t.subs({p:10})
```

$$\log{\left (10 \right )} - 1 + \frac{x}{10} + \mathcal{O}\left(\left(x - 10\right)^{2}; x\rightarrow 10\right)$$

```python
x,p = symbols('x p')
f = 1 / ((1 - x) ** 2 )
t = series(f, x, x0=p, n = 4)
t.subs({p:0})
```

$$1 + 2 x + 3 x^{2} + 4 x^{3} + \mathcal{O}\left(x^{4}\right)$$

```python
x,p = symbols('x p')
f = sqrt(4 - x)
t = series(f, x, x0=p, n = 4).removeO()
t.subs({p:0})
```

$$- \frac{x^{3}}{512} - \frac{x^{2}}{64} - \frac{x}{4} + 2$$

```python
x,y, dx, dy = symbols('x y dx dy')
f = x * (y ** 2 ) * exp(- (x ** 4 ) - ((y ** 2) / 2))

e = f.series(x, x0=dx, n=1).removeO().series(y, x0=dx, n=1).removeO()
simplify(e)
```

$$dx^{3} e^{- dx^{4} - \frac{dx^{2}}{2}}$$

```python
x = symbols('x')
f = exp(x) + x + sin(x)
f.series(x, x0=0, n=6).removeO()
```

$$\frac{x^{5}}{60} + \frac{x^{4}}{24} + \frac{x^{2}}{2} + 3 x + 1$$

```python
x = symbols('x')
f = 2 / (x ** 2 - x)
f.series(x, x0=0.5, n=6).removeO()
```

$$- 128.0 \left(x - 0.5\right)^{4} - 32.0 \left(x - 0.5\right)^{2} - 8.0$$

```python
x = symbols('x')
f = exp(-2 * x)
f.series(x, x0=2, n=2)
```

$$e^{-4} - \frac{2}{e^{4}} \left(x - 2\right) + \mathcal{O}\left(\left(x - 2\right)^{2}; x\rightarrow 2\right)$$

```python

```

Certification 2/Week4.1 - Taylor series.ipynb


```python
import sympy
```

```python
x = sympy.Symbol('x')
y = sympy.Symbol('y')
```

```python
print(type(x))
```

    <class 'sympy.core.symbol.Symbol'>

```python
expr = x**2 + y + 1

print(expr)
```

    x**2 + y + 1

```python
z = sympy.Symbol('ZZZZ')
```

```python
expr_z = z**2 + 3 * z

print(expr_z)
```

    ZZZZ**2 + 3*ZZZZ

```python
print(expr)
```

    x**2 + y + 1

```python
print(expr.subs(x, 1))
```

    y + 2

```python
print(expr.subs(x, y))
```

    y**2 + y + 1

```python
print(expr.subs([(x, 1), (y, 2)]))
```

    4

```python
expr = (x + 1)**2
```

```python
print(expr)
```

    (x + 1)**2

```python
expr_ex = sympy.expand(expr)
```

```python
print(expr_ex)
```

    x**2 + 2*x + 1

```python
expr_factor = sympy.factor(expr_ex)
```

```python
print(expr_factor)
```

    (x + 1)**2

```python
print(sympy.factor(x**3 - x**2 - 3 * x + 3))
```

    (x - 1)*(x**2 - 3)

```python
print(sympy.factor(x * y + x + y + 1))
```

    (x + 1)*(y + 1)

```python
print(sympy.solve(x**2 - 3 * x + 2))
```

    [1, 2]

```python
print(sympy.solve(x**2 + x + 1))
```

    [-1/2 - sqrt(3)*I/2, -1/2 + sqrt(3)*I/2]

```python
expr = x + y**2 - 4
```

```python
print(sympy.solve(expr, x))
```

    [-y**2 + 4]

```python
print(sympy.solve(expr, y))
```

    [-sqrt(-x + 4), sqrt(-x + 4)]

```python
expr1 = 3 * x + 5 * y - 29
expr2 = x + y - 7

print(sympy.solve([expr1, expr2]))
```

    {x: 3, y: 4}

```python
print(sympy.diff(x**3 + 2 * x**2 + x))
```

    3*x**2 + 4*x + 1

```python
expr = x**3 + y**2 - y
```

```python
print(sympy.diff(expr, x))
```

    3*x**2

```python
print(sympy.diff(expr, y))
```

    2*y - 1

```python
print(sympy.integrate(3 * x**2 + 4 * x + 1))
```

    x**3 + 2*x**2 + x

```python
print(sympy.diff(sympy.cos(x)))
```

    -sin(x)

```python
print(sympy.diff(sympy.exp(x)))
```

    exp(x)

```python
print(sympy.diff(sympy.log(x)))
```

    1/x

```python
print(sympy.integrate(sympy.cos(x)))
```

    sin(x)

```python
print(sympy.integrate(sympy.exp(x)))
```

    exp(x)

```python
print(sympy.integrate(sympy.log(x)))
```

    x*log(x) - x


# Density operator and matrix

```
from sympy import *
from sympy.physics.quantum import *
from sympy.physics.quantum.density import *
from sympy.physics.quantum.spin import (
    Jx, Jy, Jz, Jplus, Jminus, J2,
    JxBra, JyBra, JzBra,
    JxKet, JyKet, JzKet,
)
from IPython.core.display import display_pretty
```

```
%load_ext sympyprinting
```

Create n density matrix using symbolic states

```
psi = Ket('psi')
phi = Ket('phi')
```

```
d = Density((psi,0.5),(phi,0.5)); d
```

$$\rho(\begin{pmatrix}{\left|\psi\right\rangle }, & 0.5\end{pmatrix},\begin{pmatrix}{\left|\phi\right\rangle }, & 0.5\end{pmatrix})$$

```
display_pretty(d)
```

    ρ((❘ψ⟩, 0.5),(❘φ⟩, 0.5))

```
d.states()
```

$$\begin{pmatrix}{\left|\psi\right\rangle }, & {\left|\phi\right\rangle }\end{pmatrix}$$

```
d.probs()
```

$$\begin{pmatrix}0.5, & 0.5\end{pmatrix}$$

```
d.doit()
```

```
Dagger(d)
```

$$\rho(\begin{pmatrix}{\left|\psi\right\rangle }, & 0.5\end{pmatrix},\begin{pmatrix}{\left|\phi\right\rangle }, & 0.5\end{pmatrix})$$

```
A = Operator('A')
```

```
d.apply_op(A)
```

$$\rho(\begin{pmatrix}A {\left|\psi\right\rangle }, & 0.5\end{pmatrix},\begin{pmatrix}A {\left|\phi\right\rangle }, & 0.5\end{pmatrix})$$

Now create a density matrix using spin states

```
up = JzKet(S(1)/2,S(1)/2)
down = JzKet(S(1)/2,-S(1)/2)
```

```
d2 = Density((up,0.5),(down,0.5)); d2
```

$$\rho(\begin{pmatrix}{\left|\frac{1}{2},\frac{1}{2}\right\rangle }, & 0.5\end{pmatrix},\begin{pmatrix}{\left|\frac{1}{2},- \frac{1}{2}\right\rangle }, & 0.5\end{pmatrix})$$

```
represent(d2)
```

$$\left[\begin{smallmatrix}0.5 & 0\\0 & 0.5\end{smallmatrix}\right]$$

```
d2.apply_op(Jz)
```

$$\rho(\begin{pmatrix}J_z {\left|\frac{1}{2},\frac{1}{2}\right\rangle }, & 0.5\end{pmatrix},\begin{pmatrix}J_z {\left|\frac{1}{2},- \frac{1}{2}\right\rangle }, & 0.5\end{pmatrix})$$

```
qapply(_)
```

$$\rho(\begin{pmatrix}\frac{1}{2} \hbar {\left|\frac{1}{2},\frac{1}{2}\right\rangle }, & 0.5\end{pmatrix},\begin{pmatrix}- \frac{1}{2} \hbar {\left|\frac{1}{2},- \frac{1}{2}\right\rangle }, & 0.5\end{pmatrix})$$

```
qapply((Jy*d2).doit())
```


---
author: Nathan Carter (ncarter@bentley.edu)
---

This answer assumes you have imported SymPy as follows.

```python
from sympy import *                   # load all math functions
init_printing( use_latex='mathjax' )  # use pretty math output
```

Here we define a simple mathematical formula, $\frac{x^2-x-2}{x-2}$,
and compute the limit as $x$ approaches 2.  We use SymPy's built-in
`limit` function, which takes the formula $f(x)$, the variable $x$,
and the value $a$.

```python
var( 'x' )
formula = ( x**2 - x - 2 ) / ( x - 2 )
limit( formula, x, 2 )
```

$\displaystyle 3$

You can also compute one-sided limits.  For instance,
the limit of $\frac{\vert x\vert}{x}$ is $1$ as $x$ approaches 0 from the right,
but it is $-1$ as $x$ approaches 0 from the left.

```python
limit( abs(x)/x, x, 0, "-" )  # "-" means from the left
```

$\displaystyle -1$

```python
limit( abs(x)/x, x, 0, "+" )  # "+" means from the right
```

$\displaystyle 1$

database/tasks/How to compute the limit of a function/Python, using SymPy.ipynb


# the German tank problem

the Germans have a population of $n$ tanks labeled with serial numbers $1,2,...,n$ on the tanks. The number of tanks $n$ is unknown and of interest to the Allied forces. The Allied forces randomly capture $k$ tanks from the Germans with replacement and observe their serial numbers $\{x_1, x_2, ..., x_k\}$. The goal is to, from observing the serial numbers on this random sample of the tanks, estimate $n$.

the *estimator* of $n$ maps an outcome of the experiment to an estimate of $n$, $\hat{n}$.

```julia
using StatsBase
using PyPlot
using Statistics
using Printf

PyPlot.matplotlib.style.use("seaborn-pastel")

rcParams = PyPlot.PyDict(PyPlot.matplotlib."rcParams")
rcParams["font.size"] = 16
```

    16

## data structure for a tank

for elegance

```julia

```

## visualizing the captured tanks and their serial numbers

```julia
function viz_tanks(tanks::Array{Tank}, savename=nothing)
    nb_tanks = length(tanks)

    img = PyPlot.matplotlib.image.imread("tank.png")

    fig, ax = subplots(1, nb_tanks)
    for (t, tank) in enumerate(tanks)
        ax[t].imshow(img)
        ax[t].set_title(tank.serial_no)
        ax[t].axis("off")
    end
    tight_layout()

    if ! isnothing(savename)
        savefig(savename * ".png", format="png", dpi=300)
        # Linux command line tool to trim white space
        run(`convert $savename.png -trim $savename.png`)
    end
end
```

## simulating tank capture

write a function `capture_tanks` to simulate the random sampling of `nb_tanks_captured` tanks from all `nb_tanks` tanks the Germans have (without replacement). return a random sample of tanks.

```julia

```

## defining different estimators

an estimator maps an outcome $\{x_1, x_2, ..., x_k\}$ to an estimate for $n$, $\hat{n}$.

### estimator (1): maximum serial number

this is the maximum likelihood estimator.

\begin{equation}
\hat{n} = \max_i x_i
\end{equation}

```julia

```

### estimator (2): maximum serial number plus initial gap

\begin{equation}
\hat{n} = \max_i x_i + \bigl(\min_i x_i -1\bigr)
\end{equation}

```julia

```

### estimator (3): maximum serial number plus gap if samples are evenly spaced

\begin{equation}
\hat{n} = \max_i x_i + \bigl( \max_i x_i / k -1 \bigr)
\end{equation}

```julia

```

## assessing the bias and variance of different estimators

say the Germans have `nb_tanks` tanks, and we randomly capture `nb_tanks_captured`. what is the distribution of the estimators (over different outcomes of this random experiment), and how does the distribution compare to the true `nb_tanks`?

```julia

```

```julia

```

```julia

```

notes:

## what happens as we capture more and more tanks, i.e. increase $k$?

assess estimator (3).

```julia

```

## one-sided confidence interval

how confident are we that the Germans don't have *more* tanks?

significance level: $\alpha$

test statistic = estimator (3) = $\hat{n} = \max_i x_i + \bigl( \max_i x_i / k -1 \bigr)$

**null hypothesis**: the number of tanks is $n=n_0$<br>
**alternative hypothesis**: the number of tanks is less than $n_0$

we reject the null hypothesis (say, "the data does not support the null hypothesis") that the number of tanks is $n=n_0$ if the p-value is less than $\alpha$. the p-value is the probability that, if the null hypothesis is true, we get a test statistic equal to or smaller than we observed.

we want to find the highest $n_0$ such that we have statistical power to reject the null hypothesis in favor of the alternative hypothesis. this is the upper bound on the confidence interval!

then the idea is that, if the null hypothesis is that the number of tanks is absurdly large compared to the largest serial number we saw in our sample, it would be very unlikely that we would see such small serial numbers compared to the number of tanks, so we'd reject the null hypothesis.

```julia

```

say $\alpha=0.05$ and we seek a 95% one-sided confidence interval

```julia

```

```julia

```


```python
# Ejemplo lagrange
from sympy import *
import matplotlib.pyplot as plt
import numpy as np

x = [1950, 1960, 1970, 1980, 1990, 2000]
y = [123.5, 131.2, 150.7, 141.3, 203.2, 240.5]

pL = ''
for k in range(len(x)):
    pL +=  str(y[k]) + '* ('
    Lxk = 1
    for j in range(len(x)):
        if (j == k):
            continue
        pL += '(x - %f)*'%(x[j])
        Lxk *= x[k] - x[j]
    pL = pL[:-1] + '/%f) +'%(Lxk)
pL = pL[:-1]

expr = sympify(pL)
expr = expand(expr)
print(expr)

plt.plot(x,y,'ro')
x2 = np.linspace(1950,2000,100)
y2 = []
x = symbols('x')
for i in range(len(x2)):
    y2.append(expr.subs(x,x2[i]))
plt.plot(x2,y2)

estimation = expr.subs(x,1965);
print("Año 1965: {0}".format(estimation))
```

```python

```


# Math

Some essential math

## Vectors

Represented by a matrix `r` as:

\begin{equation*}
r = \begin{bmatrix} i \\ j \end{bmatrix}
\end{equation*}

The vector above has 2 components `i` and `j`.

### Vector Operations

#### Scalar Multiplication

\begin{equation*}
A \times r = A \begin{bmatrix} r_i \\ r_j \end{bmatrix} = \begin{bmatrix} A \times r_i \\ A \times r_j \end{bmatrix}
\end{equation*}

**NOTE:** Reversing direction of Vector
Is the same as $-1 \times \vec{\mathbf{r}} $

#### Addition to 2 vectors

```
            x
   x       /
    \     / r
   s \   /

 r + s =
       x
       | \ s
  r+s  |   X
       |  /
       | / r
       |/

```

\begin{equation}
r = \begin{bmatrix} r_i \\ r_j \end{bmatrix}
s = \begin{bmatrix} s_i \\ s_j \end{bmatrix}
\end{equation}

`r + s`  will be

\begin{equation}
r + s = \begin{bmatrix} r_i + s_i \\ r_j + s_j \end{bmatrix}
\end{equation}

which would be
\begin{equation*}
\mathbf{V}_1 \times \mathbf{V}_2 =  \begin{vmatrix}
\mathbf{i} & \mathbf{j} & \mathbf{k} \\
\frac{\partial X}{\partial u} &  \frac{\partial Y}{\partial u} & 0 \\
\frac{\partial X}{\partial v} &  \frac{\partial Y}{\partial v} & 0
\end{vmatrix}
\end{equation*}

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

fig, ax = plt.subplots()
x = np.linspace(0,1)
ax.plot(x, x, 'o')

ax.annotate('default line', xy=(0.15,0.1), xytext=(0.6,0.1),
            arrowprops={'arrowstyle': '-'}, va='center')
ax.annotate('dashed line', xy=(0.25,0.2), xytext=(0.6,0.2),
            arrowprops={'arrowstyle': '-', 'ls': 'dashed'}, va='center')
ax.annotate('default arrow', xy=(0.35,0.3), xytext=(0.6,0.3),
            arrowprops={'arrowstyle': '->'}, va='center')
ax.annotate('thick blue arrow', xy=(0.45,0.4), xytext=(0.6,0.4),
            arrowprops={'arrowstyle': '->', 'lw': 4, 'color': 'blue'},
            va='center')
ax.annotate('double-headed arrow', xy=(0.45,0.5), xytext=(0.01,0.5),
            arrowprops={'arrowstyle': '<->'}, va='center')
ax.annotate('arrow with closed head', xy=(0.55,0.6), xytext=(0.1,0.6),
            arrowprops={'arrowstyle': '-|>'}, va='center')
ax.annotate('a really thick red arrow\nwith not much space', xy=(0.65,0.7),
            xytext=(0.1,0.7), va='center', multialignment='right',
            arrowprops={'arrowstyle': '-|>', 'lw': 8, 'ec': 'r'})
ax.annotate('a really thick red arrow\nwith space between\nthe tail and the'
            'label', xy=(0.85,0.9), xytext=(0.1,0.9), va='center',
            multialignment='right',
            arrowprops={'arrowstyle': '-|>', 'lw': 8, 'ec': 'r', 'shrinkA': 10})

plt.show()

```