```python
from IPython.core.display import HTML
HTML("<style>.container { width:95% !important; }</style>")
```

# Lecture 3: Unconstrained optimization: Direct search methods

## Optimization problem to be studied

We will start studying functions of multiple variables by studying unconstrained optimization problems
$$
\begin{align}
\min \quad &f(x)\\
\text{s.t.}\quad &x\in \mathbb R^n
\end{align}
$$

As an example, we study the optimization problem
$$
\begin{align}
\min \quad & (x_1-10)^2+(x_2+5)^2+x_1^2\\
\text{s.t.}\quad &x_1,x_2\in\mathbb R
\end{align}
$$
This problem is unconstrained, because there are no constraints.

## Optimality conditions
(*Necessary condition*) Let $f$ be twice differentiable at $x^*\in\mathbb R^n$. If $x^*$ is a local minimizer, then $\nabla f(x^*)=0$ and the Hessian matrix $H(x^*)$ is positively semidefinite.

What is a Hessian matrix?

What means definiteness for matrices?

(*Sufficient condition*) Let $f$ be twice continuously differentiable at $x^*\in\mathbb R^n$. If $\nabla f(x^*)=0$ and $H(x^*)$ is positively definite, then $x^*$ is a strict local minimizer.

When $\nabla f(x^*)=0$, then $x^*$ is said to be a **critical point**.

Let's consider our example problem:

$\nabla \left [(x_1-10)^2+(x_2+5)^2+x_1^2\right] = (2x_1-20+2x_1,2x_2+10)$

$=(4x_1-20,2x_2+10)=0 \iff x=(5,-5)$

By the sufficient condition, $(5,-5)$ is a strict local minimizer. (...*verify that $H(x)$ is positive definite*)

Now we need to define a function in Python that evaluates that function. That is, we define a two-variable function
$$f:(x_1,x_2)\to (x_1-10)^2+(x_2+5)^2+x_1^2$$

```python
def f_simple(x):
    return (x[0] - 10.0)**2 + (x[1] + 5.0)**2 + x[0]**2

```

Now we can call that function

```python
print("At point (3,-8) the value of the function is ",f_simple([3,-8]))

```

    At point (3,-8) the value of the function is  67.0

```python
print("At point (5,-5) the value of the function is ",f_simple([5,-5]))

```

    At point (5,-5) the value of the function is  50.0

We can also plot that function

```python
import numpy as np
from pylab import meshgrid
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

def plot_2d_function(lb1,lb2,ub1,ub2,f):
    x = np.arange(lb1,ub1,0.5) # divides the interval [lb1,ub1] into set of equally divided points
    y = np.arange(lb2,ub2,0.5) # divides the interval [lb2,ub2] into set of equally divided points
    X,Y = meshgrid(x, y) # grid of points (x_i, y_j)
    Z = [f([x,y]) for (x,y) in zip (X,Y)] # evaluation of the function f on the grid
    Z = np.asarray(Z)
    fig = plt.figure() # initializes figure
    ax = fig.gca(projection='3d') # 3D
    surf = ax.plot_surface(X, Y, Z) # surface plot of f
    return plt

```

```python
plot_2d_function(3,-7,7,-3,f_simple).show() # plot f when 3<x1<7 and -7<x2<-3

```

# Direct search methods

Direct search methods (also called pattern search methods) just rely on the function values to find a (local) optimum. Direct search methods consist of a set of
1. **exploratory moves** that acquire information about the function $f$ in the neighbourhood of the current solution $x$, and
2. **pattern moves** that attempt to speed up the search using the information acquired in the exploratory moves (e.g., moving in the direction that found in the exploratory moves).

## Iterative approach

* At each iteration $k$ we have the current approximation of the optimal solution $x_k$ and a search pattern to find the next apprximation

*   $x_{k+1} = x_k + \alpha_k d_k$, where

*             $\alpha_k > 0$ : step lenght
*             $d_k$ : search direction

## Simplest method: Coordinate descent
Optimize function towards each coordinate direction separately, *does not utilize pattern moves*.

$f(x)=2x_1^2+2x_1x_2+x_2^2+x_1-x_2$

* Slow convergence, especially when the level curves of the function are not along any of the coordinate axes

## The idea of the pattern search

## Hooke&Jeeves (1961)

**input:** a minimum step length $L>0$, initial step length $\epsilon_0$, constant $0<\delta<1$ for reducing the step length, exploratory step multiplier $\gamma>1$, and starting solution $x_0$
**output:** an approximation of a local optimum (no guarantees of quality in general cases)
```
set epsilon as the initial step length epsilon0
set x as the starting solution
while epsilon is greater than L:
    for each coordinate direction i:
        find the smallest of function values by incrementing and reducing the
        variable value in that coordinate by epsilon, let this value be xi*
    if x is the same as (x1*,...,xn*):
        reduce epsilon to delta*epsilon
    else:
        if the value of f at (x1*,...,xn*) is smaller than at x+gamma*((x1*,...,xn*)-x):
            set x as (x1*,...xn*)
        else:
            set x as x+gamma*((x1*,...,xn*)-x)
return x
```

Thus, the exploratory step of Hooke&Jeeves is performed by incrementing and reducing the variable to each coordinate direction and the pattern move is just a multiplication of the exploratory move.

```python
import copy #Copying vectors
import numpy as np #Import vector calculus and much more!
def hookejeeves(L,epsilon0,delta,gamma,x0,f):
    #Set up the initial values
    epsilon = epsilon0
    x = np.array(x0)
    #Loop while step length epsilon is greater than L:
    while epsilon>L:
        #our exploratory move is initially [0,..,0]
        xtest = np.zeros(len(x))
        for coordinate in range(len(x)):
            #First points to be explored are the all x, to be changed
            exp_points = [copy.copy(x) for _ in range(3)] #points to be explored
            #Change exp_points[0] and exp_points[1] to reflect
            #moving along the coordinate
            exp_points[0][coordinate]-=epsilon
            exp_points[1][coordinate]+=epsilon
            #Assign the function values given by exp_points to a list
            f_exp_points = [f(exp_point) for exp_point in exp_points]
            #pick the smallest one of them
            min_value = min(f_exp_points)
            #The exploratory move to the coordinate direction is given by the
            #move giving the smallest value of f
            xtest[coordinate] = exp_points[f_exp_points.index(min_value)][coordinate] #The coordinate value is the one where the minimum is attained
        #If no move at all, then reduce the exploratory move step size
        if all(xtest==x):
            epsilon = delta*epsilon
        else:
            #if exploratory move is better than pattern move
            if f(xtest)<f(x+gamma*(xtest-x)):
                #...set x as the exploratory move
                x = xtest
            else:
                #Otherwise we take the pattern move
                x = x+gamma*(xtest-x)
    return x
```

```python
L = 0.001
epsilon0 = 1
delta = 0.1
gamma = 2.0
start = [2.0,3.0]
#%timeit hookejeeves(L,epsilon0,delta,gamma,x0,f_simple)
x = hookejeeves(L,epsilon0,delta,gamma,start,f_simple); print("Optimal solution is " + str(x))
```

    Optimal solution is [ 5. -5.]

### How does it actually work?

Let us define a function that can plot a set of steps:

```python
import matplotlib.pyplot as plt

def plot_2d_steps(steps,start):
    myvec = np.array([start]+steps).transpose()
    plt.plot(myvec[0,],myvec[1,],'ro') # plotting only for problems with 2 variables
    for label,x,y in zip([str(i) for i in range(len(steps)+1)],myvec[0,],myvec[1,]):
        plt.annotate(label,xy = (x, y))
    return plt
```

Now we need to save the steps of the algorithm. For that, let us modify the algorithm slightly:

```python
import copy #Copying vectors
import numpy as np #Import vector calculus and much more!
def hookejeeves_savesteps(L,epsilon0,delta,gamma,x0,f):
    epsilon = epsilon0
    x = np.array(x0)
    steps = []
    while epsilon>L:
        xtest = np.zeros(len(x))
        # exploratory move
        for coordinate in range(len(x)):
            exp_points = [copy.copy(x) for _ in range(3)] #points to be explored
            exp_points[1][coordinate]-=epsilon
            exp_points[2][coordinate]+=epsilon
            f_exp_points = [f(exp_point) for exp_point in exp_points]
            min_value = min(f_exp_points)
            xtest[coordinate] = exp_points[f_exp_points.index(min_value)][coordinate] #The coordinate value is the one where the minimum is attained
        if all(xtest==x):
            # exploratory move is zero
            epsilon = delta*epsilon
            print(epsilon)
        else:
            # check if pattern move helps
            if f(xtest)<f(x+gamma*(xtest-x)):
                x = xtest
            else:
                x = x+gamma*(xtest-x)
            steps.append(x)
            #print(x)
    return x,steps
```

```python
L = 0.001
epsilon0 =  0.3
delta = 0.1
gamma = 2
start = [2,3]
(x,steps) = hookejeeves_savesteps(L,epsilon0,delta,gamma,start,f_simple)
print(x)
print(len(steps))

```

    0.03
    0.003
    0.00030000000000000003
    [ 5. -5.]
    11

```python
plot_2d_steps(steps,start).show()
```

## The role of the step length
The above implementation uses a fixed step length. It is easier to implement but not as efficient if the step length would be optimized by using a line search. The difference is illustrated in the figures below.

Fixed                    |Optimized
:-----------------------:|:----------------------------:
 |

**Note: You can implement a version optimizing the step length by line search e.g. when solving future excercises**

## Powell's method (1964)

Powell's method is similar to Hooke&Jeeves, but the first step in exploratory moves is taken to the direction of the last pattern move. This speeds up the convergence in most cases (**highly depending on how the step size is computed!**).

```python
import copy
import numpy as np
def powell(L,epsilon0,delta,gamma,x0,f):
    epsilon = epsilon0
    exp_direction = np.array([0,1])
    x = np.array(x0)
    while epsilon>L:
        exp_direction=epsilon*exp_direction # to take steps of length epsilon
        #Comparing among exploratory points to first exploratory direction:
        if f(x+exp_direction)<f(x):
            exp_step1=exp_direction
        elif f(x-exp_direction)<f(x):
            exp_step1=-exp_direction
        else:
            exp_step1 = np.zeros(2)
        #The following only works in 2d!!
        exp_direction2 = np.array([exp_direction[1],-exp_direction[0]]) # conjugate directions
        if f(x+exp_direction2)<f(x):
            exp_step2=exp_direction2
        elif f(x-exp_direction2)<f(x):
            exp_step2=-exp_direction2
        else:
            exp_step2 = np.zeros(2)
        if all(exp_step1+exp_step2==0):
            epsilon = delta*epsilon
        else:
            if f(x+(exp_step1+exp_step2))<f(x+gamma*(exp_step1+exp_step2)):
                x = x+(exp_step1+exp_step2)
            else:
                x = x+gamma*(exp_step1+exp_step2)
            exp_direction = epsilon*(exp_step1+exp_step2)/np.linalg.norm(exp_step1+exp_step2)
    return x
```

```python
L = 0.001
epsilon0 = 1.
delta = 0.1
gamma = 2.0
start = [2.,3.]
#%timeit powell(L,epsilon0,delta,gamma,x0,f_simple)
x = powell(L,epsilon0,delta,gamma,start,f_simple)
print("Optimal solution is " + str(x))
```

    Optimal solution is [ 5.00000029 -5.00000031]

### How does it actually work?

Let us again modify the algorithm slightly:

```python
import copy
import numpy as np
def powell_savesteps(L,epsilon0,delta,gamma,x0,f):
    epsilon = epsilon0
    exp_direction = np.array([0,1])
    x = np.array(x0)
    steps = []
    while epsilon>L:
        exp_direction=epsilon*exp_direction
        #Comparing among exploratory points to firtst exploratory direction:
        if f(x+exp_direction)<f(x):
            exp_step1=exp_direction
        elif f(x-exp_direction)<f(x):
            exp_step1=-exp_direction
        else:
            exp_step1 = np.zeros(2)
        #The following only works in 2d!!
        exp_direction2 = np.array([exp_direction[1],-exp_direction[0]])
        if f(x+exp_direction2)<f(x):
            exp_step2=exp_direction2
        elif f(x-exp_direction2)<f(x):
            exp_step2=-exp_direction2
        else:
            exp_step2 = np.zeros(2)
        if all(exp_step1+exp_step2==0):
            epsilon = delta*epsilon
            print(epsilon)
        else:
            if f(x+(exp_step1+exp_step2))<f(x+gamma*(exp_step1+exp_step2)):
                x = x+(exp_step1+exp_step2)
            else:
                x = x+gamma*(exp_step1+exp_step2)
            steps.append(x)
            exp_direction = (exp_step1+exp_step2)/np.linalg.norm(exp_step1+exp_step2)
            #print(x)
    return x,steps
```

```python
L = 0.001
epsilon0 = 0.3
delta = 0.1
gamma = 2
start = [2,3]
(x,steps) = powell_savesteps(L,epsilon0,delta,gamma,start,f_simple)
print(x)
print(len(steps))
```

    0.03
    0.003
    0.00030000000000000003
    [ 5.         -5.00146069]
    17

```python
plot_2d_steps(steps,start).show()
```

Another example
$$
\begin{align}
\min \quad & (x_1-2)^4+(x_1 - 2x_2)^2\\
\text{s.t.}\quad &x_1,x_2\in\mathbb R
\end{align}
$$
Optimal solution clearly is $x^*=(2,1)^T$

```python
def f_simple2(x):
    return (x[0] - 2.0)**4 + (x[0] - 2.0*x[1])**2
```

```python
plot_2d_function(-2,-6,6,4,f_simple2).show()
```

```python
# Hooke & Jeeves
L = 0.0001
epsilon0 = .8
delta = 0.1
gamma = 2
start = [-1.,3.]
(x,steps) = hookejeeves_savesteps(L,epsilon0,delta,gamma,start,f_simple2)
print(x)
print(len(steps))
```

    0.08000000000000002
    0.008000000000000002
    0.0008000000000000003
    8.000000000000003e-05
    [1.916  0.9576]
    90

```python
plot_2d_steps(steps,start).show()
```

```python
# Powell
L = 0.0001
epsilon0 = .8
delta = 0.1
gamma = 2.
start = [-1.,3.]
(x,steps) = powell_savesteps(L,epsilon0,delta,gamma,start,f_simple2)
print(x)
print(len(steps))
```

    0.08000000000000002
    0.008000000000000002
    0.0008000000000000003
    8.000000000000003e-05
    [1.94316485 0.97120742]
    21

```python
plot_2d_steps(steps,start).show()
```

Just another example
$$
\begin{align}
\min \quad & x_1^2+2x_1x_2+x_2^2+x_1-x_2\\
\text{s.t.}\quad &x_1,x_2\in\mathbb R
\end{align}
$$
Optimal solution is $x^*=(-1,1.5)^T$

```python
def f_simple3(x):
    return 2.0*x[0]**2 + 2.0*x[0]*x[1] + x[1]**2 + x[0] - x[1]
# opt = (-1,1.5)
```

```python
plot_2d_function(-4,-4,2,6,f_simple3).show()

```

```python
print(f_simple3([-1.,1.4]))
print(f_simple3([-.9,1.5]))
print(f_simple3([-1.,1.5]))

```

    -1.24
    -1.23
    -1.25

```python
# Hooke & Jeeves
L = 0.001
epsilon0 = 1.
delta = 0.1
gamma = 2
start = [-2.0,3.0]
(x,steps) = hookejeeves_savesteps(L,epsilon0,delta,gamma,start,f_simple3)
print(x)
print(len(steps))

```

    0.1
    0.010000000000000002
    0.0010000000000000002
    0.00010000000000000003
    [-1.   1.5]
    18

```python
plot_2d_steps(steps,start).show()

```

```python
L = 0.001
epsilon0 = 1.
delta = 0.1
gamma = 2.0
start = [-2.,3.]
(x,steps) = powell_savesteps(L,epsilon0,delta,gamma,start,f_simple3)
print(x)
print(len(steps))

```

    0.1
    0.010000000000000002
    0.0010000000000000002
    0.00010000000000000003
    [-1.00060323  1.50036488]
    23

```python
plot_2d_steps(steps,start).show()

```

## Nelder Mead method (by J. Nelder & R. Mead in 1965)

A direct search method based on a simplex, i.e. $n+1$ points in $n$-dimensional space. The points for a simplex are iteratively updated by replacing the worst point based on the objective function value. Further information and illustration can be found e.g. in Nelder-Mead

## Nice review and wrap-up papers about the direct search methods:

* https://www.cs.wm.edu/~va/research/sirev.pdf
* https://www.sciencedirect.com/science/article/pii/S0377042700004234

Lecture 3, Direct search methods; Hooke&Jeeves and Powell's method.ipynb


# Trends and cycles in unemployment

Here we consider three methods for separating a trend and cycle in economic data. Supposing we have a time series $y_t$, the basic idea is to decompose it into these two components:

$$
y_t = \mu_t + \eta_t
$$

where $\mu_t$ represents the trend or level and $\eta_t$ represents the cyclical component. In this case, we consider a *stochastic* trend, so that $\mu_t$ is a random variable and not a deterministic function of time. Two of methods fall under the heading of "unobserved components" models, and the third is the popular Hodrick-Prescott (HP) filter. Consistent with e.g. Harvey and Jaeger (1993), we find that these models all produce similar decompositions.

This notebook demonstrates applying these models to separate trend from cycle in the U.S. unemployment rate.

```python
%matplotlib inline
```

```python
import numpy as np
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
```

```python
from pandas_datareader.data import DataReader
endog = DataReader('UNRATE', 'fred', start='1954-01-01')
endog.index.freq = endog.index.inferred_freq
```

### Hodrick-Prescott (HP) filter

The first method is the Hodrick-Prescott filter, which can be applied to a data series in a very straightforward method. Here we specify the parameter $\lambda=129600$ because the unemployment rate is observed monthly.

```python
hp_cycle, hp_trend = sm.tsa.filters.hpfilter(endog, lamb=129600)
```

### Unobserved components and ARIMA model (UC-ARIMA)

The next method is an unobserved components model, where the trend is modeled as a random walk and the cycle is modeled with an ARIMA model - in particular, here we use an AR(4) model. The process for the time series can be written as:

$$
\begin{align}
y_t & = \mu_t + \eta_t \\
\mu_{t+1} & = \mu_t + \epsilon_{t+1} \\
\phi(L) \eta_t & = \nu_t
\end{align}
$$

where $\phi(L)$ is the AR(4) lag polynomial and $\epsilon_t$ and $\nu_t$ are white noise.

```python
mod_ucarima = sm.tsa.UnobservedComponents(endog, 'rwalk', autoregressive=4)
# Here the powell method is used, since it achieves a
# higher loglikelihood than the default L-BFGS method
res_ucarima = mod_ucarima.fit(method='powell', disp=False)
print(res_ucarima.summary())
```

                            Unobserved Components Results
    ==============================================================================
    Dep. Variable:                 UNRATE   No. Observations:                  804
    Model:                    random walk   Log Likelihood                -459.536
                                  + AR(4)   AIC                            931.072
    Date:                Tue, 02 Feb 2021   BIC                            959.202
    Time:                        06:51:34   HQIC                           941.876
    Sample:                    01-01-1954
                             - 12-01-2020
    Covariance Type:                  opg
    ================================================================================
                       coef    std err          z      P>|z|      [0.025      0.975]
    --------------------------------------------------------------------------------
    sigma2.level     0.0906      0.161      0.564      0.573      -0.224       0.405
    sigma2.ar        0.0890      0.164      0.541      0.588      -0.233       0.411
    ar.L1            1.0672      0.123      8.701      0.000       0.827       1.308
    ar.L2           -0.2113      0.397     -0.532      0.595      -0.990       0.567
    ar.L3            0.0904      0.216      0.419      0.675      -0.332       0.513
    ar.L4           -0.0082      0.081     -0.101      0.919      -0.166       0.150
    ===================================================================================
    Ljung-Box (L1) (Q):                   0.01   Jarque-Bera (JB):           5940381.74
    Prob(Q):                              0.94   Prob(JB):                         0.00
    Heteroskedasticity (H):               9.45   Skew:                            17.27
    Prob(H) (two-sided):                  0.00   Kurtosis:                       422.94
    ===================================================================================

    Warnings:
    [1] Covariance matrix calculated using the outer product of gradients (complex-step).

### Unobserved components with stochastic cycle (UC)

The final method is also an unobserved components model, but where the cycle is modeled explicitly.

$$
\begin{align}
y_t & = \mu_t + \eta_t \\
\mu_{t+1} & = \mu_t + \epsilon_{t+1} \\
\eta_{t+1} & = \eta_t \cos \lambda_\eta + \eta_t^* \sin \lambda_\eta + \tilde \omega_t \qquad & \tilde \omega_t \sim N(0, \sigma_{\tilde \omega}^2) \\
\eta_{t+1}^* & = -\eta_t \sin \lambda_\eta + \eta_t^* \cos \lambda_\eta + \tilde \omega_t^* & \tilde \omega_t^* \sim N(0, \sigma_{\tilde \omega}^2)
\end{align}
$$

```python
mod_uc = sm.tsa.UnobservedComponents(
    endog, 'rwalk',
    cycle=True, stochastic_cycle=True, damped_cycle=True,
)
# Here the powell method gets close to the optimum
res_uc = mod_uc.fit(method='powell', disp=False)
# but to get to the highest loglikelihood we do a
# second round using the L-BFGS method.
res_uc = mod_uc.fit(res_uc.params, disp=False)
print(res_uc.summary())
```

                                Unobserved Components Results
    =====================================================================================
    Dep. Variable:                        UNRATE   No. Observations:                  804
    Model:                           random walk   Log Likelihood                -461.513
                       + damped stochastic cycle   AIC                            931.027
    Date:                       Tue, 02 Feb 2021   BIC                            949.770
    Time:                               06:51:35   HQIC                           938.227
    Sample:                           01-01-1954
                                    - 12-01-2020
    Covariance Type:                         opg
    ===================================================================================
                          coef    std err          z      P>|z|      [0.025      0.975]
    -----------------------------------------------------------------------------------
    sigma2.level        0.1262      0.026      4.867      0.000       0.075       0.177
    sigma2.cycle        0.0435      0.025      1.772      0.076      -0.005       0.092
    frequency.cycle     0.3491      0.150      2.326      0.020       0.055       0.643
    damping.cycle       0.7870      0.048     16.377      0.000       0.693       0.881
    ===================================================================================
    Ljung-Box (L1) (Q):                   1.38   Jarque-Bera (JB):           6086584.12
    Prob(Q):                              0.24   Prob(JB):                         0.00
    Heteroskedasticity (H):               9.22   Skew:                            17.51
    Prob(H) (two-sided):                  0.00   Kurtosis:                       428.61
    ===================================================================================

    Warnings:
    [1] Covariance matrix calculated using the outer product of gradients (complex-step).

### Graphical comparison

The output of each of these models is an estimate of the trend component $\mu_t$ and an estimate of the cyclical component $\eta_t$. Qualitatively the estimates of trend and cycle are very similar, although the trend component from the HP filter is somewhat more variable than those from the unobserved components models. This means that relatively mode of the movement in the unemployment rate is attributed to changes in the underlying trend rather than to temporary cyclical movements.

```python
fig, axes = plt.subplots(2, figsize=(13,5));
axes[0].set(title='Level/trend component')
axes[0].plot(endog.index, res_uc.level.smoothed, label='UC')
axes[0].plot(endog.index, res_ucarima.level.smoothed, label='UC-ARIMA(2,0)')
axes[0].plot(hp_trend, label='HP Filter')
axes[0].legend(loc='upper left')
axes[0].grid()

axes[1].set(title='Cycle component')
axes[1].plot(endog.index, res_uc.cycle.smoothed, label='UC')
axes[1].plot(endog.index, res_ucarima.autoregressive.smoothed, label='UC-ARIMA(2,0)')
axes[1].plot(hp_cycle, label='HP Filter')
axes[1].legend(loc='upper left')
axes[1].grid()

fig.tight_layout();
```


# Bayesian Statistics for Physicists: 03 Choosing Priors

## Contents of the BSFP series

01 Overview and Setup

02 Bayesian basics

03 Choosing priors

04 Examples of Bayesian updating

05 Sampling

06 Model selection, mixing, ...

07 Gaussian processes

08 Machine learning

A01 Appendices: reference, vocabulary, notation

## Overview of Choosing priors

A key feature of Bayesian statistics is the choice of the prior probability distribution.  We should incorporate all of the information we have, but no more. What are the options for choosing priors?  What are the subtleties?  Places to use an informative prior:

your dataset is small, but there is related information available from other systems;

your model is very flexible but you want to prevent overfitting (so use priors that prefer
      values close to zero);

you want to stay away from particular regions of parameter space: maybe strictly unphysical such as
      negative cross sections or acausal, or maybe
      counter to well-motivated assumptions such as parameters are of order unity;

the posterior from a previous experiment can become a prior ==> Bayesian updating;

other examples??

Last revised: 14-Oct-2018 by Dick Furnstahl [furnstahl.1@osu.edu].

## Contents of this notebook on Choosing priors

 Maximum entropy

 Conjugate priors

## Python/Jupyter set up

See Part 01 for overall installation and setup.

```python
# set up for plots in this notebook using matplotlib (there are other plotting choices)
%matplotlib inline
```

```python
import numpy as np

import scipy.stats as stats
from scipy.stats import norm, uniform

import matplotlib.pyplot as plt
#plt.style.use('seaborn') # pretty matplotlib plots

import corner
import pymc3 as pm

```

    /Users/furnstah/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      from ._conv import register_converters as _register_converters

```python
# make font adjustments
#plt.rcParams['font.size'] = 12
#plt.rcParams['legend.fontsize'] = 'medium'
#plt.rcParams['figure.titlesize'] = 'medium'
plt.rcdefaults()  # revert to defaults for now
```

```python
%%html
<!-- Use html cell magic to add css styling -->
<style>
  em {
      color: red;
  }
  dd {
      margin-left: 15px;
  }
  .red{color: red}
  .blue{color: blue}
</style>
```

### Subtlety with uniform prior

An example from the 2016 workshop on [Bayesian Methods in Astronomy](https://github.com/jakevdp/BayesianAstronomy/blob/master/Index.ipynb) considers fitting a straight line $y = mx+b$ to noisy data. If you take the prior on the slope $m$ to be uniform (flat), thinking that this is non-informative, then look at this plot, which samples lines with uniformly distributed slopes:

```python
%matplotlib inline
xx = np.linspace(-1, 1,11)
fig, ax = plt.subplots(1)
ax.axis([-1, 1, -1, 1])
for slope in np.linspace(0, 20, 100):
    ax.plot(xx, slope * xx, '-k', linewidth=1)
ax.set_title("Weighting of slopes");

```

The density of the lines indicates the relative probability of different slopes. Summary point: flat priors are not necessarily minimally informative.  For the slope we probably want a prior that does not artificially over-weight large slopes; see http://arxiv.org/abs/1411.5018 for some discussion. For example, we might use a flat prior on the *angle* the line makes with the x-axis, which implies
$$
P(m) \propto (1 + m^2)^{-3/2}
$$

### Maximum entropy for priors (appealing to physicists!)

 Basic idea: identify least informative $p(x)$ from maximizing entropy:
   $$
     S[p(x)] = -\int\!dx\, p(x)\, \log\left[\frac{p(x)}{m(x)}\right]
   $$
   subject to constraints from the prior information.

 $m(x)$ is an appropriate measure (often uniform, but see Sivia)

 Use Lagrange multiplier(s) to maximize

 One constraint is always normalization: $\int\!dx\, p(x) = 1$
     $\Longrightarrow$ alone it leads to uniform $p(x)$ (actually to $m(x)$)

   If the mean $\mu$ and variance $\sigma^2$ are known, then maximize (may be clearer to do with discretized variables)

   \begin{align}
    Q[p(x)] &= -\int\! dx\, p(x)\, \log\left[\frac{p(x)}{m(x)}\right]
    + \lambda_0 \left[ 1 - \int\! dx\, p(x) \right] \\
     & \quad + \lambda_1 \left[\sigma^2 -  \int\! dx\, (x-\mu)^2 p(x)  \right]
   \end{align}

   Then
   $$
    \frac{\delta Q}{\delta p(x)} = 0   \quad\mbox{and}\quad
     m(x) = \mbox{const.}
     \ \Longrightarrow\
     p(x \mid \mu,\sigma) = \frac{1}{\sigma\sqrt{2 \pi}}
     e^{-(x-\mu)^2/2\sigma^2}
   $$

    For $\textbf{x} = \{x_1, \cdots, x_N\}$, we find $p(\textbf{x})$
    is the familiar least-squares likelihood function.  See Sivia for many more details.

If you want a "scale invariant" prior, this often means a flat prior on the *logarithm* of the parameter.  This is an example of a [Jeffreys Prior](https://en.wikipedia.org/wiki/Jeffreys_prior).

[Return to Contents]

### Conjugate priors

Bayes theorem tells us that the posterior follows from multiplying the prior by the likelihood and normalizing:

$$
  p(\theta\,|\,x) = \frac{p(x\,|\,\theta)\, p(\theta)}
                          {\int\!d\theta'\, p(x\,|\,\theta')\, p(\theta')}
$$

If the resulting $p(\theta\,|\,x)$ is in the same family of pdfs as $p(\theta)$, then $p(\theta)$ is said to be a conjugate prior for the likelihood $p(x\,|\,\theta)$.

The likelihood is often a fixed form. If there is freedom to choose a conjugate prior, the Bayesian updating of the posterior is given in closed form.

Comments:

Suppose we are flipping a coin and seek the posterior for the probability of heads $\theta \in [0,1]$.  Let $x$ be the number of successes (heads) in $N$ trials. The likelihood is binomial:
   $$ p(x\mid\theta) = \binom{N}{x}
       \theta^{x} (1-\theta)^{N-x} \;.
   $$
If we choose the prior to be a beta distribution with hyperparameters $a$ and $b$ (which are parameters of the prior as opposed to the model),
   $$ p(\theta\mid a,b) = \textrm{Beta}(\theta\mid a,b)
      = \frac{\theta^{a-1}(1-\theta)^{b-1}}{B(a,b)}
   $$
with beta function $B(a,b)$, then the posterior is of the same form ($\bar x \equiv N - x$):
   $$ p(\theta\mid x,a,b) = \frac{\theta^{x+a-1}(1-\theta)^{\bar x+b-1}}{B(x+a,\bar x+b)} = \textrm{Beta}(\theta\mid x+a,\bar x +b)
   $$
This posterior can be used as the prior for more samples; the hyperparameters just add additional information (check that it doesn't matter if you analyze the data all at once or sequentially because the tosses are independent).

See the [Wikipedia article](https://en.wikipedia.org/wiki/Conjugate_prior) for a big table of conjugate prior pairs.

If the likelihood is a normal distribution with known variance, the conjugate prior is also normal.  If it is a normal distribution with known mean, the inverse gamma distribution is a conjugate prior.

[Return to Contents]


# Обратная задача динамики в пространстве

Для начала определим необходимый матан:

```python
from sympy import *
from IPython import display
init_printing()

class Vector(Expr):

    def __init__(self, x, y, z):
        super(Vector, self).__init__()
        self.x = x
        self.y = y
        self.z = z

    @staticmethod
    def zero():
        return Vector(0, 0, 0)

    @staticmethod
    def lerp(start, end, t):
        return start + t * (end + -1 * start)

    def normalized(self):
        return 1 / self.magnitude() * self

    def magnitude(self):
        return (self.x ** 2 + self.y ** 2 + self.z ** 2) ** 0.5

    def times(self, other):
        return Vector(
            other * self.x,
            other * self.y,
            other * self.z
        )

    def cross(self, other):
        return Vector(
            self.y * other.z - self.z * other.y,
            self.z * other.x - self.x * other.z,
            self.x * other.y - self.y * other.x,
        )

    def dot(self, other):
        return self.x * other.x + self.y * other.y + self.z * other.z

    def angle_to(self, other, axis):
        crossed = self.cross(other)
        collinearity = crossed.dot(axis)
        return arctan2(
            crossed.magnitude(),
            self.dot(other)
        ) * (1 if collinearity > 0.0 else -1)

    def __str__(self):
        return "[{}, {}, {}]".format(self.x, self.y, self.z)

    def __repr__(self):
        return "Vector[{}, {}, {}]".format(self.x, self.y, self.z)

    def __add__(self, other):
        if isinstance(other, Vector):
            return Vector(
                self.x + other.x,
                self.y + other.y,
                self.z + other.z
            )
        return Add(self, other)

    def __mul__(self, other):
        return self.times(other)

    def __rmul__(self, other):
        return self.times(other)

    def _repr_latex_(self):
        return r"$\left[{}, {}, {}\right]$".format(
            printing.latex(self.x),
            printing.latex(self.y),
            printing.latex(self.z),
        )

    def _latex(self, printer):
        return r"\left[{}, {}, {}\right]".format(
            printing.latex(self.x),
            printing.latex(self.y),
            printing.latex(self.z),
        )

    def simplify(self, **kwargs):
        return Vector(
            simplify(self.x, **kwargs),
            simplify(self.y, **kwargs),
            simplify(self.z, **kwargs)
        )

    def velocity(self, t):
        return sqrt(
            diff(self.x, t) ** 2 +
            diff(self.y, t) ** 2 +
            diff(self.z, t) ** 2
        )

    def __pow__(self, other):
        return Vector(
            self.x ** other,
            self.y ** other,
            self.z ** other
        )

    def diff(self, t):
        return Vector(
            diff(self.x, t),
            diff(self.y, t),
            diff(self.z, t)
        )

class Quaternion(Expr):

    def __init__(self, w, x, y, z):
        super(Quaternion, self).__init__()
        self.w = w
        self.x = x
        self.y = y
        self.z = z

    @staticmethod
    def identity():
        return Quaternion(1, 0, 0, 0)

    def rotate(self, vector):
        pure = Quaternion(0, vector.x, vector.y, vector.z)
        rotated = self * pure * self.conjugate()
        return Vector(rotated.x, rotated.y, rotated.z)

    def magnitude(self):
        return (self.w ** 2 + self.x ** 2 + self.y ** 2 + self.z ** 2) ** 0.5

    def normalized(self):
        return 1 / self.magnitude() * self

    def dot(self, other):
        return self.w * other.w + self.x * other.x + self.y * other.y + self.z * other.z

    @staticmethod
    def slerp(start, end, t):
        dot = start.dot(end)
        if dot < 0.0:
            start = -1 * start
            dot = -dot
        if dot >= 1.0:
            return start
        theta = arccos(dot) * t
        q = (end + -dot * start).normalized()
        return cos(theta) * start + sin(theta) * q

    def conjugate(self):
        return Quaternion(self.w, -self.x, -self.y, -self.z)

    def multiply(self, other):
        return Quaternion(
            self.w * other.w - self.x * other.x - self.y * other.y - self.z * other.z,
            self.w * other.x + self.x * other.w + self.y * other.z - self.z * other.y,
            self.w * other.y - self.x * other.z + self.y * other.w + self.z * other.x,
            self.w * other.z + self.x * other.y - self.y * other.x + self.z * other.w
        )

    def plus(self, other):
        return Quaternion(
            self.w + other.w,
            self.x + other.x,
            self.y + other.y,
            self.z + other.z
        )

    def times(self, other):
        return Quaternion(
            other * self.w,
            other * self.x,
            other * self.y,
            other * self.z
        )

    @staticmethod
    def from_angle_axis(angle, axis, unit=True):
        if not unit:
            axis = axis.normalized()
        c = cos(angle / 2)
        s = sin(angle / 2)
        return Quaternion(
            c,
            s * axis.x,
            s * axis.y,
            s * axis.z
        )

    def __str__(self):
        return "[{}, {}, {}, {}]".format(self.w, self.x, self.y, self.z)

    def __repr__(self):
        return "Quaternion[{}, {}, {}, {}]".format(self.w, self.x, self.y, self.z)

    def __mul__(self, other):
        if isinstance(other, Vector):
            return self.rotate(other)
        if isinstance(other, Quaternion):
            return self.multiply(other)
        if isinstance(other, MatrixSymbol):
            return Mul(self, other)
        return self.times(other)

    def __add__(self, other):
        return self.plus(other)

    def __rmul__(self, other):
        return self.times(other)

    def diff(self, t):
        return Quaternion(
            diff(self.w, t),
            diff(self.x, t),
            diff(self.y, t),
            diff(self.z, t)
        )

    def angular_velocity(self, t):
        velocity = 2 * diff(self, t) * self.conjugate()
        return Vector(
            velocity.x,
            velocity.y,
            velocity.z
        )

    def local_angular_velocity(self, t):
        velocity = 2 * self.conjugate() * diff(self, t)
        return Vector(
            velocity.x,
            velocity.y,
            velocity.z
        )

    def _repr_latex_(self):
        return r"$\left[{}, {}, {}, {}\right]$".format(
            printing.latex(self.w),
            printing.latex(self.x),
            printing.latex(self.y),
            printing.latex(self.z),
        )

    def _latex(self, printer):
        return r"\left[{}, {}, {}, {}\right]".format(
            printing.latex(self.w),
            printing.latex(self.x),
            printing.latex(self.y),
            printing.latex(self.z),
        )

    def simplify(self):
        return Quaternion(
            simplify(self.w),
            simplify(self.x),
            simplify(self.y),
            simplify(self.z)
        )

class Transform(Expr):
    def __init__(self, translation: Vector, rotation: Quaternion):
        super(Transform, self).__init__()
        self.translation = translation
        self.rotation = rotation

    @staticmethod
    def identity():
        return Transform(
            Vector.zero(),
            Quaternion.identity()
        )

    def inverse(self):
        conj = self.rotation.conjugate()
        return Transform(
            conj * (-1 * self.translation),
            conj
        )

    @staticmethod
    def lerp(start, end, t):
        return Transform(
            Vector.lerp(start.translation, end.translation, t),
            Quaternion.slerp(start.rotation, end.rotation, t)
        )

    def __str__(self):
        return "[\n\t{}\n\t{}\n]".format(str(self.translation), str(self.rotation))

    def __repr__(self):
        return "Transform[\n\t{}\n\t{}\n]".format(str(self.translation), str(self.rotation))

    def __add__(self, other):
        if isinstance(other, Transform):
            return Transform(
                self.translation + self.rotation * other.translation,
                self.rotation * other.rotation
            )
        if isinstance(other, Vector):
            return self.translation + self.rotation * other

    def _repr_latex_(self):
        return r"$\left[{}, {}\right]$".format(
            printing.latex(self.translation),
            printing.latex(self.rotation),
        )

    def simplify(self):
        return Transform(
            simplify(self.translation),
            simplify(self.rotation)
        )
```

Зададим основные переменные:
- время
- ускорение свободного падения
- моменты инерции звеньев и объекта
- центры масс звеньев и объекта
- массы звеньев и объекта
- длины звеньев
- обобщенные координаты

```python
t = Symbol("t")
g = Symbol("g")
j = symbols("J_0:4", cls=MatrixSymbol, n=3, m=3)
cmx = symbols("c_mx0:4")
cmy = symbols("c_my0:4")
cmz = symbols("c_mz0:4")
m = symbols("m_0:4")
l = symbols("l_0:3")
q_ = symbols("q_0:3", cls=Function)
q = [q_[0](t), q_[1](t), q_[2](t)]
```

Укажем положения звеньев:

```python
def transform0(q):
    return Transform(
        Vector(0, 0, 0),
        Quaternion.from_angle_axis(q[0], Vector(0, 0, 1))
    )

def transform1(q):
    return transform0(q) + Transform(
        Vector(0, 0, l[0]),
        Quaternion.from_angle_axis(q[1], Vector(1, 0, 0))
    )

def transform2(q):
    return transform1(q) + Transform(
        Vector(0, 0, l[1]),
        Quaternion.from_angle_axis(q[2], Vector(1, 0, 0))
    )

def transform3(q):
    return transform2(q) + Transform(
        Vector(0, 0, l[2]),
        Quaternion.identity()
    )
```

И положения центров масс:

```python
def mass_position0(q):
    return transform0(q) + Vector(cmx[0], cmy[0], cmz[0])

def mass_position1(q):
    return transform1(q) + Vector(cmx[1], cmy[1], cmz[1])

def mass_position2(q):
    return transform2(q) + Vector(cmx[2], cmy[2], cmz[2])

def mass_position3(q):
    return transform3(q) + Vector(cmx[3], cmy[3], cmz[3])
```

Рассчитаем кинетическую энергию:

```python
def kinetic_energy0(q):
    return mass_position0(q).velocity(t) ** 2 * m[0] / 2 +\
        transform0(q).rotation.local_angular_velocity(t) ** 2 * j[0] / 2

def kinetic_energy1(q):
    return mass_position1(q).velocity(t) ** 2 * m[1] / 2 +\
        transform1(q).rotation.local_angular_velocity(t) ** 2 * j[1] / 2

def kinetic_energy2(q):
    return mass_position2(q).velocity(t) ** 2 * m[2] / 2 +\
        transform2(q).rotation.local_angular_velocity(t) ** 2 * j[2] / 2

def kinetic_energy3(q):
    return mass_position3(q).velocity(t) ** 2 * m[3] / 2 +\
        transform3(q).rotation.local_angular_velocity(t) ** 2 * j[3] / 2

def total_kinetic_energy(q):
    return kinetic_energy0(q) +\
        kinetic_energy1(q) +\
        kinetic_energy2(q) +\
        kinetic_energy3(q)
```

И потенциальную:

```python
def potential_energy0(q):
    return mass_position0(q).z * m[0] * g

def potential_energy1(q):
    return mass_position1(q).z * m[1] * g

def potential_energy2(q):
    return mass_position2(q).z * m[2] * g

def potential_energy3(q):
    return mass_position3(q).z * m[3] * g

def total_potential_energy(q):
    return potential_energy0(q) +\
        potential_energy1(q) +\
        potential_energy2(q) +\
        potential_energy3(q)
```

Дальше все просто:

```python
def lagrangian(q):
    return total_kinetic_energy(q) - total_potential_energy(q)
```

```python
def force_calculation0(q):
    lgr = lagrangian(q)
    return lgr.diff(q[0].diff(t)).diff(t) - lgr.diff(q[0])

def force_calculation1(q):
    lgr = lagrangian(q)
    return lgr.diff(q[1].diff(t)).diff(t) - lgr.diff(q[1])

def force_calculation2(q):
    lgr = lagrangian(q)
    return lgr.diff(q[2].diff(t)).diff(t) - lgr.diff(q[2])
```

```python
force0 = force_calculation0(q)
force1 = force_calculation1(q)
force2 = force_calculation2(q)
```

Посмотрим силы в простом случае:
- $q_0(t) = t$
- $q_1(t) = \frac{\pi}{2}$
- $q_1(t) = -\frac{\pi}{2}$

```python
force0.replace(q[0], t).replace(q[1], pi/2).replace(q[2], -pi/2).simplify()
```

```python
force1.replace(q[0], t).replace(q[1], pi/2).replace(q[2], -pi/2).simplify()
```

```python
force2.replace(q[0], t).replace(q[1], pi/2).replace(q[2], -pi/2).simplify()
```


<h1>  (some) LaTeX environments for Jupyter notebook </h1>

```javascript
%%javascript
IPython.utils.load_extensions('calico-document-tools');
```

    <IPython.core.display.Javascript object>

```python
%%html
<style>
    .prompt{
        display: none;
</style>
```

# Presentation and main features

This extension for IPython 3.x or Jupyter enables to use some LaTeX commands and environments in the notebook's markdown cells.

\begin{enumerate}
\item **LaTeX commands and environments**
\begin{itemize}
\item support for some LaTeX commands within markdown cells, *e.g.* `\textit`, `\textbf`, `\underline`
\item support for **theorems-like environments**
\item support for **lists**: *enumerate, itemize*,
\item limited support for a **figure environment**,
\item support for an environment *listing*,
\item additional *textboxa* environment
\end{itemize}
\item **Citations and bibliography**
\begin{itemize}
\item support for `\cite` with creation of a References section
\end{itemize}
\item **Document-wide numbering of equations, support for `\label` and `\ref`**
\item **Configuration toolbar**
\item Styles can be customized in the *latex\_env.css* stylesheet
\end{enumerate}

A simple illustration is as follows: on can type the following in a markdown cell
\begin{listing}
\begin{theorem} \label{theo:dotp}
Let $u$ and $v$ be two vectors of $\mathbb{R}^n$. The dot product can be expressed as
\begin{equation}
\label{eq:dotp}
u^Tv = |u||v| \cos \theta,
\end{equation}
where $\theta$ is the angle between $u$ and $v$ ...
\end{theorem}
\end{listing}
and have it rendered as

\begin{theorem} \label{theo:dotp}
Let $u$ and $v$ be two vectors of $\mathbb{R}^n$. The dot product can be expressed as
\begin{equation}
\label{eq:dotp}
u^Tv = |u||v| \cos \theta,
\end{equation}
where $\theta$ is the angle between $u$ and $v$ ...
\end{theorem}

## Implementation principle

The main idea is to override the standard Markdown renderer in order to add a *small* parsing of LaTeX expressions and environments. This heavily uses regular expressions. The LaTeX expression are then rendered using an html version. For instance `\underline {something}` is rendered as `<u> something </u>`, that is \underline{something}. The environments are replaced by an html tag with a class derived from the name of the environment. For example, a `definition` denvronment will be replaced by an html rendering corresponding to the class `latex_definition`. The styles associated with the different classes are sepcified in `latex_env.css`. These substitutions are implemented in `thsInNb4.js`.

## Support for simple LaTeX commands

We also added some LaTeX commands (e.g. `\textit`, `\textbf`, `\underline`) -- this is useful in the case of copy-paste from a LaTeX document. Labels and references are supported, including for equations.

## Available environments

- **theorems-like environments**: *property, theorem, lemma, corollary, proposition, definition,remark, problem,           exercise, example*,
- **lists**: *enumerate, itemize*,
- limited support for a *figure* environment,
- an environment *listing*,
- *textboxa*, wich is a `textbox` environment defined as a demonstration (see below).

More environments can be added easily in the javascript source file `thmsInNb.js`. The rendering is done according to the stylesheet `latex_env.css`, which can be customized.

## Automatic numerotation, labels and references

Counters for numerotation are implemented: one for theorems-like environments, a second for exercises-like environments and a third one for numbering figures.
Mathjax-equations with a label are also numbered document-wide (in contrast with standard notebook/mathjax numbering where the scope of numbering is limited to cells).
An anchor is created for any label which enables to links things within the document: `\label` and `\ref` are both supported. A limitation is that numbering is updated (incremented) each time a cell is rendered. A toolbar button is provided to reset the counters and refresh the rendering of the whole document.

A simple example is as follows, featuring automatic numerotation, and the use of labels and references. Also note that standard markdown can be present in the environment and is interpreted.
*The rendering is done according to the stylesheet `latex_env.css`, which of course, can be tailored to specific uses and tastes*.

\begin{listing}
\begin{definition} \label{def:FT}
Let $x[n]$ be a sequence of length $N$. Then, its **Fourier transform** is given by
\begin{equation}
\label{eq:FT}
X[k]= \frac{1}{N} \sum_{n=0}^{N-1} x[n] e^{-j2\pi \frac{kn}{N}}
\end{equation}
\end{definition}
\end{listing}

\begin{definition} \label{def:FT}
Let $x[n]$ be a sequence of length $N$. Then, its **Fourier transform** is given by
\begin{equation}
\label{eq:FT2}
X[k]= \frac{1}{N} \sum_{n=0}^{N-1} x[n] e^{-j2\pi \frac{kn}{N}}
\end{equation}
\end{definition}

It is now possible to refer to the definition and to the equation by their labels,  as in:
\begin{listing}
As an example of Definition \ref{def:FT}, consider the Fourier transform (\ref{eq:FT2}) of a pure cosine wave given by
$$
x[n]= \cos(2\pi k_0 n/N),
$$
where $k_0$ is an integer.
\end{listing}

As an example of Definition \ref{def:FT}, consider the Fourier transform (\ref{eq:FT2}) of a pure cosine wave given by
$$
x[n]= \cos(2\pi k_0 n/N),
$$
where $k_0$ is an integer.
Its Fourier transform is given by
$$
X[k] = \frac{1}{2} \left( \delta[k-k_0] + \delta[k-k_0] \right),
$$
modulo $N$.

## Bibliography

### Usage

It is possible to cite bibliographic references using the standard LaTeX `\cite` mechanism. The extension looks for the references in a bibTeX file, by default `biblio.bib` in the same directory as the notebook. The name of this file can be modified in the configuration toolbar. It is then possible to cite works in the notebook, e.g.
\begin{listing}
The main paper on IPython is definitively \cite{PER-GRA:2007}. Other interesting references are certainly \cite{mckinney2012python, rossant2013learning}. Interestingly, a presentation of the IPython notebook has also be published recently in Nature \cite{shen2014interactive}.
\end{listing}
The main paper on IPython is definitively \cite{PER-GRA:2007}. Other interesting references are certainly \cite{mckinney2012python, rossant2013learning}. Interestingly, a presentation of the IPython notebook has also be published recently in Nature \cite{shen2014interactive}.

### Implementation

The implemention uses several snippets from the nice [icalico-document-tools](https://bitbucket.org/ipre/calico/downloads/) extension that also considers the rendering of citations in the notebook. We also use a modified version of the [bibtex-js](https://code.google.com/p/bibtex-js/) parser for reading the references in the bibTeX file. The different functions are implemented in `bibInNb4.js`. The rendering of citations calls can adopt three styles (Numbered, by key or apa-like) -- this can be selected in the configuration toolbar. It is also possible to customize the rendering of references in the reference list. A citation template is provided in the beginning of file `latex_envs.js`:
```
var cit_tpl = {
// feel free to add more types and customize the templates
    'INPROCEEDINGS': '%AUTHOR:InitialsGiven%, ``_%TITLE%_\'\', %BOOKTITLE%, %MONTH% %YEAR%.',
    ... etc
```
The keys are the main types of documents, eg inproceedings, article, inbook, etc. To each key is associated a string where the %KEYWORDS% are the fields of the bibtex entry. The keywords are replaced by the correponding bibtex entry value. The template string can formatted with additional words and effects (markdown or LaTeX are commands are supported)

## Figure environment

Finally, it is sometimes useful to integrate a figure within a markdown cell. The standard markdown markup for that is ``, but a limitation is that the image can not be resized, can not be referenced and is not numbered. Furthermore it can be useful for re-using existing code. Threfore we have added a limited support for the `figure` environment. This enables to do something like
\begin{listing}
\begin{figure}[H]
\centerline{\includegraphics[width=10cm]{example.png}}
\caption{\label{fig:example} This is an example of figure included using LaTeX commands.}
\end{figure}
\end{listing}
which renders as
\begin{figure}[H]
\centerline{\includegraphics[width=10cm]{example.png}}
\caption{\label{fig:example} This is an example of figure included using LaTeX commands.}
\end{figure}
Of course, this Figure can now be referenced:
\begin{listing}
Figure \ref{fig:example} shows a second filter with input $X_2$, output $Y_2$  and an impulse response denoted as $h_2(n)$
\end{listing}
Figure \ref{fig:example} shows a second filter with input $X_2$, output $Y_2$  and an impulse response denoted as $h_2(n)$

## Other features

- It is possible to mix LaTeX and markdown markup in environments
- Environments can be nested. However, this is not always perfect...

## User interface

### Buttons on main toolbar

On the main toolbar, the extension provides three buttons
\begin{figure}[H]
\centerline{\includegraphics[width=10cm]{main_toolbar.png}}
\end{figure}
The first one  can be used to refresh the numerotation of equations and references in all the document. The second one fires the reading of the bibliography bibtex file and creates (or updates) the reference section. Finally the third one is a toogle button that opens or closes the configuration toolbar.

### Configuration toolbar

The configuration toolbar
\begin{figure}[H]
\centerline{\includegraphics[width=10cm]{configuration_toolbar.png}}
\end{figure}
 enables to enter  some configuration options for the extension. First, one can indicate the name of the bibtex file. If this file is not found and the user creates the reference section, then this section will indicate that the file was not found. The references drop-down menu enables to choose the type of reference calls. The Equations input box enable to initiate numbering of equations at the given number (this may be useful for complex documents in several files/parts). Finally the last drop-down menu let the user choose to number equation or to display their label instead.  These configuration options are then stored in the notebook's metadata (and restored on reload).

# Installation, usage and further examples

## Installation

The extension consists in several javascript scripts: `latex_envs.js`, `thmsInNb4.js`, `bibInNb4.js` and `initNb.js`,  together with a stylesheet `latex_envs.css`.
With Jupyter, you may also simply install the extension with
```python
from notebook.nbextensions import install_nbextension, check_nbextension
install_nbextension("https://rawgit.com/jfbercher/latex_envs/master/latex_envs.zip", user=True)
```

An even more simple procedure is to issue
```
jupyter nbextension install https://rawgit.com/jfbercher/latex_envs/master/latex_envs.zip  --user

```
at the command line.

If you are still with IPython 3, you can install with
```python
from IPython.html.nbextensions import install_nbextension, check_nbextension
install_nbextension("https://rawgit.com/jfbercher/latex_envs/master/latex_envs.zip", user=True)
```

You may then try the extension.  Load it by typing
```python
%%javascript
require("base/js/utils").load_extensions("latex_envs/latex_envs")
```
in a code cell of the notebook.

If you want to automatically load the extension for any notebook, you may use
```python
from IPython.html.services.config import ConfigManager
ip = get_ipython()
cm = ConfigManager(parent=ip, profile_dir=ip.profile_dir.location)
cm.update('notebook', {"load_extensions": {"latex_envs/latex_envs": True}})
```

and replace the `True` by `None if you want to unload the extension.

Alternatively you may also do this in javascript via
```python
%%javascript
IPython.notebook.config.update({"load_extensions":{"latex_envs/latex_envs":true}})
```

replace the `true` by `null` to unload.

The last alternative is
```
jupyter nbextension enable latex_envs/latex_envs

```

and `disable` to disable it, of course.

You may follow the instructions in the [wiki](https://github.com/ipython-contrib/IPython-notebook-extensions/wiki) to install the extension.

## First example (continued)

We continue the first example on fthe Fourier transform definition \ref{def:FT} in order to show that, of course, we can illustrate things using a simple code. Since the Fourier transform is an essential tool in signal processing, We put this in evidence using the `textboxa` environment -- which is defined here in the css, and that one should define in the LaTeX counterpart:
\begin{listing}
\begin{textboxa}
The Fourier transform is an extremely useful tool to have in your toolbox!
\end{textboxa}
\end{listing}

\begin{textboxa}
The Fourier transform is an extremely useful tool to have in your toolbox!
\end{textboxa}

The Fourier transform of a pure cosine is given by
$$
X[k] = \frac{1}{2} \left( \delta[k-k_0] + \delta[k-k_0] \right),
$$
modulo $N$. This is illustrated in the following simple script:

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from numpy.fft import fft
k0=4; N=128; n=np.arange(N); k=np.arange(N)
x=np.sin(2*np.pi*k0*n/N)
X=fft(x)
plt.stem(k,np.abs(X))
plt.xlim([0, 20])
plt.title("Fourier transform of a cosine")
_=plt.xlabel("Frequency index (k)")
```

## Second example

This example shows a series of environments, with different facets; **links, references, markdown or/and LaTeX formatting within environments**. Again, the rendering is done according to the stylesheet `latex_env.css`, which can be customized. The listing of environments below is typed using the environment *listing*...

\begin{listing}
\begin{definition} \label{def:diffeq}
We call \textbf{difference equation} an equation of the form
$$
\label{eq:diffeq}
y[n]= \sum_{k=1}^{p} a_k y[n-k] + \sum_{i=0}^q b_i x[n-i]
$$
\end{definition}

\begin{property}
If all the $a_k$ in equation (\ref{eq:diffeq}) of definition \ref{def:diffeq} are zero, then the filter has a **finite impulse response**.
\end{property}

\begin{proof}
Let $\delta[n]$ denote the Dirac impulse. Take $x[n]=\delta[n]$ in (\ref{eq:diffeq}). This yields, by definition, the impulse response:
$$
\label{eq:fir}
h[n]= \sum_{i=0}^q b_i \delta[n-i],
$$
which has finite support.
\end{proof}

\begin{theorem}
The poles of a causal stable filter are located within the unit circle in the complex plane.
\end{theorem}

\begin{example} \label{ex:IIR1}
Consider $y[n]= a y[n-1] +  x[n]$. The pole of the transfer function is $z=a$. The impulse response $h[n]=a^n$ has infinite support.
\end{example}

In the following exercise, you will check that the filter is stable iff $a$<1.

\begin{exercise}\label{ex:exofilter}
Consider the filter defined in Example \ref{ex:IIR1}. Using the **function** `lfilter` of scipy, compute and plot the impulse response for several values of $a$.
\end{exercise}

\end{listing}

The lines above are rendered as follows (of course everything can be tailored in the stylesheet):

\begin{definition} \label{def:diffeq}
We call \textbf{difference equation} an equation of the form
\begin{equation}
\label{eq:diffeq}
y[n]= \sum_{k=1}^{p} a_k y[n-k] + \sum_{i=0}^q b_i x[n-i]
\end{equation}
\end{definition}
Properties of the filter are linked to the coefficients of the difference equation. For instance, an immediate property is

\begin{property}
If all the $a_k$ in equation (\ref{eq:diffeq}) of definition \ref{def:diffeq} are zero, then the filter has a **finite impulse response**.
\end{property}

\begin{proof}
Let $\delta[n]$ denote the Dirac impulse. Take $x[n]=\delta[n]$ in (\ref{eq:diffeq}). This yields, by definition, the impulse response:
\begin{equation}
\label{eq:fir}
h[n]= \sum_{i=0}^q b_i \delta[n-i],
\end{equation}
which has finite support.
\end{proof}

\begin{theorem}
The poles of a causal stable filter are located within the unit circle in the complex plane.
\end{theorem}

\begin{example} \label{ex:IIR1}
Consider $y[n]= a y[n-1] +  x[n]$. The pole of the transfer function is $z=a$. The impulse response $h[n]=a^n$ has infinite support.
\end{example}

In the following exercise, you will check that the filter is stable iff $a$<1.

\begin{exercise}\label{ex:exofilter}
Consider the filter defined in Example \ref{ex:IIR1}. Using the **function** `lfilter` of scipy, compute and plot the impulse response for several values of $a$.
\end{exercise}

\begin{listing}
The solution of exercise \ref{ex:exofilter}, which uses a difference equation as in Definition \ref{def:diffeq}:
\end{listing}
The solution of exercise \ref{ex:exofilter}, which uses a difference equation as in Definition \ref{def:diffeq}:

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from scipy.signal import lfilter
d=np.zeros(100); d[0]=1 #dirac impulse
alist=[0.2, 0.8, 0.9, 0.95, 0.99, 0.999, 1.001, 1.01]
for a in alist:
    h=lfilter([1], [1, -a],d)
    _=plt.plot(h, label="a={}".format(a))
plt.ylim([0,1.5])
plt.xlabel('Time')
_=plt.legend()
```

## Third example:

This example shows that environments like itemize or enumerate are also available. As already indicated, this is useful for copying text from a TeX file. Following the same idea, text formating commands `\textit`, `\textbf`, `\underline`, etc are also available.

\begin{listing}
The following \textit{environments} are available:
\begin{itemize}
    \item \textbf{Theorems and likes}
    \begin{enumerate}
        \item theorem,
        \item lemma,
        \item corollary
        \item ...
    \end{enumerate}
    \item \textbf{exercises}
    \begin{enumerate}
        \item problem,
        \item example,
        \item exercise
    \end{enumerate}
\end{itemize}
\end{listing}

which gives...

The following \textit{environments} are available:
\begin{itemize}
\item \textbf{Theorems and likes}
\begin{enumerate}
\item theorem,
\item lemma,
\item corollary
\item ...
\end{enumerate}
\item \textbf{exercises}
\begin{enumerate}
\item problem,
\item example,
\item exercise
\end{enumerate}
\end{itemize}

# (post)-Converters

The extension works in the live-notebook. Since it relies on a bunch of javascript, the notebook does not render as is in very nice services such as `nbviewer` or `github` viewer. Similarly, `nbconvert` does not know of the LaTeX constructs which are used and therefore do not fully convert notebooks making use of this extension. Therefore, it is necessary to add a post conversion step to conversions provided by `nbconvert`. Though an interface exists for adding post-converters to nbconvert, this (first) author was too lazy and not enough strong to implement the post conversion along these lines. What has be done are simple `bash` and `python` scripts that perform this conversion.

## Installation

Copy the scripts files to a directory in your search path, or launch the scripts with the complete path. The two main scripts are `ipynb_thms_to_html` (conversion to html, of course:) and `ipynb_thms_to_latex`  (conversion to LaTeX!).

## Conversion to html

**Requirements**: You will need `perl`, `nodejs`, and `ipython3` (the script calls `ipython3`; if your interpreter is `ipython`, edit the script and replace the different occurences).

**Configuration**:

- If you still use IPython 3.x (not Jupyter), edit `ipynb_thms_to_html` and set the variable `stillIPython3=true`. IPython 3.x will also have to change the `nbextensionsDir` to `/.ipython/nbextensions` in the file `post_html_thms.js`.
- Finally **all users** should check the path for their `marked.js` library as indicated in `post_html_thms.js`. On Debian, it is located
`/usr/share/jupyter/notebook/static/components/marked/lib/marked.js`. It shoud be similar for other flavors of linux
(may be `/usr/local/share/...`).
You may locate the right library by issuing `locate -eb0P marked.js | xargs -r0 ls -ald` on the command line. Please update the variable `marked` for your system in `post_html_thms.js`.

The conversion to html is done by something like

        [path/]ipynb_thms_to_html filename
or a list of files such as

        [path/]ipynb_thms_to_html *.ipynb
In turn, this script makes somes substitutions using `perl`, and then uses the `nodesj` javascript interpreter to make the very same substitutions that are done in the live notebook.
The conversion uses the template `thmsInNb.tpl` (located in the script directory). It also copies the css `latex_env.css` in the directory of the output html file (it must be copied with html files in the case of web upload).

## Conversion to LaTeX

**Requirements**: You will need `perl` and `ipython3`.

The conversion to LaTeX is done by something like

        [path/]ipynb_thms_to_latex filename
or a list of files such as

        [path/]ipynb_thms_to_latex *.ipynb

The script makes some substitutions and cleaning in markdown cells, then calls the legacy `nbconvert`. Afterward, it runs through the LaTeX environments and converts their contents (which can contain markdown markup) to  LaTeX. Note that the script contains a list of the LaTeX environments to process. In the case of the addition of an environment in the main javascript (`thmsInNb.js`), this list must also be updated.

Finally, the script removes the header and footer in the LaTeX file. This is a personnal choice, and the corresponding line can be safely commented.

\begin{example}
As for an example, the present document has been converted using

    ipynb_thms_to_latex latex_env_doc.ipynb

Then the resulting file (without header/footer) has been included in the main file `documentation.tex`, where some LaTeX definitions of environments are done (namely listings, colors, etc) and compiled using

    xelatex documentation

The output can be consulted [here](documentation.pdf).
\end{example}

# Disclaimer, sources and thanks

This is a not-quick but certainly dirty hack. I am a complete beginner in javascript and of course there are obviously a large amount of possible improvements of the code, in cleaning, factorizing, etc! Language also needs improvement.

**Contributions will be welcome and deeply appreciated.**

Originally, I used a piece of code from the nice online markdown editor `stackedit` [https://github.com/benweet/stackedit/issues/187](https://github.com/benweet/stackedit/issues/187), where the authors also considered the problem of incorporating LaTeX markup in their markdown. I also used examples and code from [https://github.com/ipython-contrib/IPython-notebook-extensions](https://github.com/ipython-contrib/IPython-notebook-extensions).

# References

(<a id="cit-PER-GRA:2007" href="#call-PER-GRA:2007">P\'erez and Granger, 2007</a>) P\'erez Fernando and Granger Brian E., ``_IPython: a System for Interactive Scientific Computing_'', Computing in Science and Engineering, vol. 9, number 3, pp. 21--29, May 2007.  [online](http://ipython.org)

(<a id="cit-mckinney2012python" href="#call-mckinney2012python">McKinney, 2012</a>) Wes McKinney, ``_Python for data analysis: Data wrangling with Pandas, NumPy, and IPython_'',  2012.

(<a id="cit-rossant2013learning" href="#call-rossant2013learning">Rossant, 2013</a>) Cyrille Rossant, ``_Learning IPython for interactive computing and data visualization_'',  2013.

(<a id="cit-shen2014interactive" href="#call-shen2014interactive">Shen, 2014</a>) Shen Helen, ``_Interactive notebooks: Sharing the code_'', Nature, vol. 515, number 7525, pp. 151--152,  2014.


I thought it would be interesting to experiment with creating an image that changed each time it was drawn.
Now this isn't the most exciting realisation of that idea, but it does also touch on an idea I want to look at
developing further as part of `arlunio` itself. Applying the concept of interfaces to arlunio images I think
could be very powerful - imagine having some generic "human" that knew how to walk and all you had to
do was tell it how to draw one...

Anyway back to this image, as you've probably noticed by now it's a clock - not a very interesting clock
but it tells the time, in fact it's frozen at the time that it was when the image was rendered.

```python
import arlunio as ar
import numpy as np

from datetime import datetime

from arlunio.image import Image, fill
from arlunio.mask import all_
from arlunio.math import R, T
from arlunio.shape import Circle
```

As the clock is made up from a collection of distinct shapes you need to ensure that each
component is defined with a consistent scale otherwise it won't look right. I found
it useful to say that the radius of the clockface is `1` and then define everything else relative
to that. However since the default scale of the grid an image is defined on is also `1` the
clock would look a bit cramped as it would run right up againt the edge.

Rather than work to a more awkard scale I'm defining a `scale` paramaeter that I'm going to pass
to each shape. This changes the size of the underlying cartesian grid that is mapped onto the
image when drawn. By enlarging this grid each shape will appear smaller and fit better into the
final image

```python
scale = 1.1
```

As you've probably noticed the hands of the clock are just rectangles, but instead of
using the built in `Rectangle` shape in the standard library I've decided to define
a custom `ClockHand` shape. This is because the standard `Rectangle` shape is hard to
control in this scenario (perhaps this should be improved 🤔...) You will also see that
I'm using [polar coordinates](https://en.wikipedia.org/wiki/Polar_coordinate_system)
instead of the conventional \\(x\\) and \\(y\\)

By defining the `ClockHand` shape with polar coordinates adding the ability to set the
rotation becomes easy since all we have to do is take an offset `t0` and subtract it from
the angle parameter `t`. Something to note is that when `arlunio` maps polar coordinates onto
an image it follows the mathematical convention of having the line `t = 0` be at
3 o'clock, so in order to make positioning the hands easier later on I also shift this
coordinate by \\(\frac{\pi}{2}\\) (90 degrees) so when `t0 = 0` the hand will be pointing at
12 o'clock

Unfortunately while nice for expressing rotations, polar coordinates are certainly not the
best at expressing a rectangle. Thankfully the conversion from polar to standard \\(x\\) and
\\(y\\) coordinates is quite straightforward

\begin{align}
    x &= r\cos{(t)} \\\\
    y &= r\sin{(t)}
\end{align}

Once we have our \\(x\\) and \\(y\\) coordinates it's easy enough to define a rectangle as
(ar.)all the \\(x\\) values between `0` and the `length` **and** all the \\(y\\) values
between `-size` and `size`.

```python
@ar.definition
def ClockHand(r: R, t: T, *, length=1, size=0.025):
    t -= np.pi/2

    x = r * np.cos(t)
    y = r * np.sin(t)

    return all_(
        x > 0, x < length,
        np.abs(y) < size
    )
```

Next we look at the shape I've called `Numeral`, since it's used to indicate the hours on the
clockface having the ability to rotate it easily is important so it is also defined with respect
to polar coordinates. In fact it's definition is almost identical to the `ClockHand` shape
with one small difference. To try and make the clock a little more interesting to look at the sides
of a numeral are defined to be those underneath a small slope, instead of having flat sides
like a rectangle.

```python
@ar.definition
def Numeral(r: R, t: T):
    t -= np.pi/2

    x = r * np.cos(t)
    y = r * np.sin(t)

    return all_(
        x > 0.8, x < 0.95,
        np.abs(y) < (0.1 * x) - 0.07
    )
```

With all the shapes that we need defined it's time to focus on how we're going to position
the hands in order to tell the time. This is where we get into that "interfaces" idea I was talking
about earlier, as you can see there's nothing formal yet instead we're just taking advantage of
Python's duck typing. This function only requires that the shapes you give it all have a
`t0` property that controls its angle - which is all you need to implement a clock poser(?),
positioner(?), director(?)... 🤔 we'll have to think of a name later!

Calculating the angle itself is quite straightforward - assuming you're familiar with
[radians](https://en.wikipedia.org/wiki/Radian) that is. The `make_clock` function is
written with the assumption that `t0 = 0` is going to correpsond with the hand pointing at 12 o'clock,
which we ensured earlier when shifting the `t` parameter in the `ClockHand` shape by \\(\frac{\pi}{2}\\)
radians (90 degrees).

From there it's case of unpacking our Python [datetime](https://docs.python.org/3/library/datetime.html#datetime.datetime)
object and calculating the percentage each respective hand has travelled around the clockface
by dividing it by the maximum value each hand can represent (60 minutes in an hour, 12 hours
on a clock etc). Once we have the percentage we can multiply it by \\(2\pi\\)) (360 degrees)
to get the angle. Finally we add each of the shapes representing the hands together so that
we return a single object representing the clock.

```python
def make_clock(dt, width, height, image, hour_hand, minute_hand, second_hand):
    hour = dt.hour if dt.hour <= 12 else dt.hour - 12
    minutes = dt.minute
    seconds = dt.second

    second_hand.t0 = -2 * np.pi * (seconds / 60)
    minute_hand.t0 = -2 * np.pi * (minutes / 60)
    hour_hand.t0 = -2 * np.pi * ((hour / 12) + (minutes / 600))

    image = fill(second_hand(width=width, height=height), foreground="#f00", image=image)
    image = fill(minute_hand(width=width, height=height), image=image)
    image = fill(hour_hand(width=width, height=height), image=image)

    center = Circle(r=0.2)
    image = fill(center(width=width, height=height), image=image)

    return image
```

A few other points of interest to make about the above implementation

- `datetime` objects use 24hr time so it feels neater to convert it to 12hr time even if it's
  not strictly necessary
- When calculating the angle for the hour hand we also add a small contribution from the minute
  hand so that we replicate the effect of the hour hand moving between numbers throughout the
  hour. - However it probably requires further thought in order to produce a more accurate result...

Last but not least we come to the final assembly, I've gone on long enough as it is so here
is a few highlights

- We create three instances of the `ClockHand`, hours, minutes and seconds
- Using a for loop we "spin" an instance of the `Numeral` shape around the clockface one for each
  hour.
- Since our `make_clock` function was written to take any `datetime` instance, having the clock
  frozen at the time it was drawn is a simple as passing the result of `datetime.now()` to the
  function.

```python
@ar.definition
def Clock(width: int, height: int) -> Image:

    clock = Circle(r=1, pt=0.01, scale=scale)
    image = fill(clock(width=width, height=height), background="white")

    for i in range(12):
        t0 = i * (np.pi / 6)
        numeral = Numeral(t0=t0, scale=scale)
        image = fill(numeral(width=width, height=height), image=image)

    hour_hand = ClockHand(size=0.02, length=0.5, scale=scale)
    minute_hand = ClockHand(size=0.02, length=0.8, scale=scale)
    second_hand = ClockHand(size=0.01, length=.95, scale=scale)

    return make_clock(datetime.now(), width, height, image, hour_hand, minute_hand, second_hand)
```

```python
clock = Clock()
image = clock(width=1920, height=1080)
image
```


[Sascha Spors](https://orcid.org/0000-0001-7225-9992),
Professorship Signal Theory and Digital Signal Processing,
[Institute of Communications Engineering (INT)](https://www.int.uni-rostock.de/),
Faculty of Computer Science and Electrical Engineering (IEF),
[University of Rostock, Germany](https://www.uni-rostock.de/en/)

# Tutorial Signals and Systems (Signal- und Systemtheorie)

Summer Semester 2021 (Bachelor Course #24015)

- lecture: https://github.com/spatialaudio/signals-and-systems-lecture
- tutorial: https://github.com/spatialaudio/signals-and-systems-exercises

WIP...
The project is currently under heavy development while adding new material for the summer semester 2021

Feel free to contact lecturer [frank.schultz@uni-rostock.de](https://orcid.org/0000-0002-3010-0294)

## Übung / Exercise 5 Magnitude Bode Plot Approximation for LTI Systems

### References:

* Norbert Fliege (1991): "*Systemtheorie*", Teubner, Stuttgart (GER), cf. chapter 4.3.5

* Alan V. Oppenheim, Alan S. Willsky with S. Hamid Nawab (1997): "*Signals & Systems*", Prentice Hall, Upper Saddle River NJ (USA), 2nd ed., cf. chapter 6

* Bernd Girod, Rudolf Rabenstein, Alexander Stenger (2001): "*Signals and Systems*", Wiley, Chichester (UK), cf. chapter 10

* Bernd Girod, Rudolf Rabenstein, Alexander Stenger (2005/2007): "*Einführung in die Systemtheorie*", Teubner, Wiesbaden (GER), 3rd/4th ed., cf. chapter 10

```python
import matplotlib.pyplot as plt
import numpy as np
import scipy.signal as signal
decades = 2  # plot number of decades left and right from cut frequency

print('log10 axis spacing:')
print(np.log10(np.arange(1, 11, 1)))
print('log2 axis spacing:')
print(np.log2(np.arange(1, 2.1, 0.1)))
```

## 2nd Order Lowpass, PT2

The 2nd order lopwass from `laplace_transform/solving_2nd_order_ode.tex`
\begin{align}
H_\mathrm{Low}(s) = \frac{1}{\frac{16}{25}s^2+\frac{24}{25}s +1} = [\frac{16}{25}s^2+\frac{24}{25}s +1]^{-1}
\end{align}
is to be visualized as approximated and exact magnitude Bode plot.
See `laplace_system_analysis/frequency_response_2nd_order_ode.tex` for detailed calculus and discussion.

\begin{align}
H(s) = H_0 \frac{1}{(s-s_{\infty,1}) (s-s_{\infty,2})}
\end{align}

The poles

$s_{\infty,1,2} = -\frac{3}{4} \pm \mathrm{j}$

constitute a complex conjugate pair, thus

$|s_{\infty,1}| = |s_{\infty,2}| = \frac{5}{4}$, $|s_{\infty,1}|^2 = |s_{\infty,2}|^2 = \frac{25}{16}$

$H_0 = 25/16$.

\begin{align}
\tilde{H_0} = \frac{H_0}{|s_{\infty,1}|^2} = \frac{\frac{25}{16}}{\frac{25}{16}} = 1 \rightarrow 0 \mathrm{dB}
\end{align}

```python
w = np.arange(1e-2, 1e+2, 0.01)
s = 1j*w

# define LTI system by zeros and poles:
soo1 = (-3/4+1j)
soo2 = (-3/4-1j)
H0 = 25/16
# get transfer function Python/Matlab-like handling:
sz = 0  # we need this zero and the pole at the origin for signal.lti()
sp = 0, soo1, soo2
sys = signal.lti(sz, sp, H0)
w, Hmag, Hphase = sys.bode(w)

# Bode approximation:
# gain -> straight line at 20 dB, blue:
H0tilde = H0 / np.abs(soo1)**2
H0t = np.zeros_like(w) + 20*np.log10(np.abs(H0tilde))
# conj complex pole ->
Hsoo1 = -2*20*np.log10(np.abs(s/soo1))  # line with -40dB/decade for w>5/4, red
Hsoo1[np.abs(s) <= np.abs(soo1)] = 0  # 0 dB for w<5/4, red
# add dB
Happrox = H0t + Hsoo1  # add magnitude frequency responses in dB

plt.figure(figsize=(7, 5))
plt.semilogx(w, H0t, color='C0', lw=5,
             label=r'$|\tilde{H_0}|=1\rightarrow 0$ dB')
plt.semilogx(w, Hsoo1, color='C3', lw=4,
             label=r'conj complex pole $|s_{\infty,1,2}|=\frac{5}{4}$')
plt.semilogx(w, Happrox, color='k', lw=2, label='Bode plot approximation')
plt.semilogx(w, Hmag, lw=2, color='C8', label='Bode plot exact')

plt.xticks((1e-1, 2e-1, 5e-1, 1e0, 5/4, 2, 2.5, 5, 10, 12.5, 20, 50, 100),
           ['0.1', '0.2', '0.5', '1', r'$\frac{5}{4}$', '2', r'$\frac{5}{2}$', '5', '10', '12.5', '20', '50', '100'])
plt.yticks((-40, -36, -30, -24, -20, -12, -10, -6, -3, 0, 3, 6, 10))
plt.xlim(0.1, 13)
plt.ylim(-40, 3)
plt.xlabel(r'$\omega$ / (rad/s)')
plt.ylabel('level: 20 lg|H| / dB')
plt.title(
    r'magnitude frequency response of $H(s) = [\frac{16}{25}s^2+\frac{24}{25}s +1]^{-1}$')
plt.legend()
plt.grid(True, which='both')
plt.savefig('bodeplot_example_approximations_level_44EB4169E9.pdf')
```

```python
w = np.arange(1e-2, 1e+2, 0.01)
s = 1j*w

sz = 0
sp = 0, -3/4-1j, -3/4+1j
H0 = 25/16
sys = signal.lti(sz, sp, H0)
w, mag, phase = sys.bode(np.arange(1e-2, 1e2, 1e-2))

plt.figure(figsize=(7, 5))
plt.semilogx(w, phase, 'C8', linewidth=3)
plt.xticks((1e-1, 2e-1, 5e-1, 1e0, 5/4, 2, 2.5, 5, 10, 12.5, 20, 50, 100),
           ['0.1', '0.2', '0.5', '1', r'$\frac{5}{4}$', '2', r'$\frac{5}{2}$', '5', '10', '12.5', '20', '50', '100'])
plt.yticks(np.arange(-180, 225, 45))
plt.xlim(0.1, 13)
plt.ylim(-180, 0)
plt.xlabel('$\omega$ / (rad/s)')
plt.ylabel(r'phase: $\angle(H)$ / deg')
plt.title(
    r'phase frequency response of $H(s) = [\frac{16}{25}s^2+\frac{24}{25}s +1]^{-1}$')
plt.grid(True, which='both')
plt.savefig('bodeplot_example_approximations_phase_44EB4169E9.pdf')
```

## 2nd Order Lowpass, PT2 -> Varying Pole Quality

When using cut frequency $\omega_0$, damping factor $0 < D \leq 1$ or pole quality $Q>0.5$ we can write the Laplace transfer function
of the 2nd order lowpass filter as
\begin{align}
H(s) = \frac{1}{\frac{1}{\omega_0^2} s^2 + \frac{2 D}{\omega_0} s + 1}=
\frac{1}{\frac{1}{\omega_0^2} s^2 + \frac{1}{\omega_0 Q_\infty} s + 1}.
\end{align}
With the example below you can play around with w0 (i.e. $\omega_0$) and D.

Note that this semi-logarithmic plot is over the **frequency** $f$ in Hz, rather than over **angular frequency** $\omega$ in rad/s.

```python
# user input:
w0 = 2*np.pi*1  # cut frequency in rad/s
# 0 < D <= 1:
# D = 1/100  # near to the jw axis, more resonsant
#D = 1/10
D = 3/5
#D = 1/np.sqrt(2)
# D = 1  # far from jw axis, less resonsant
###

Q = 1/(2*D)
soo1 = - D*w0 + 1j*w0*np.sqrt(1-D**2)
soo2 = - D*w0 - 1j*w0*np.sqrt(1-D**2)
print('D = %4.3f, Q = %4.3f' % (D, Q))
print('poles s001,2 = %5.4f +- j %5.4f' % (np.real(soo1), np.imag(soo1)))

w = np.arange(w0/10**decades, w0*10**decades, w0/100)
s = 1j*w

# exact:
H = 1/(s**2/w0**2 + s/w0/Q + 1)
Hlevel = 20*np.log10(np.abs(H))

# approximation:
wconst = w[np.abs(s) < w0]
Hconst = 0*wconst
wslope = np.array([w0, w[-1]])
Hslope = (0, -40*decades)

# plot
plt.figure(figsize=(7, 5))
plt.semilogx(wslope/2/np.pi, Hslope, color='k', lw=3)
plt.semilogx(wconst/2/np.pi, Hconst, color='k',
             label='Bode plot approximation', lw=3)
plt.semilogx(w/2/np.pi, Hlevel, color='C8',
             label=r'Bode plot exact for $Q_{\infty,1}=$%5.4f' % Q, lw=2)
plt.xlabel('f / Hz')
plt.ylabel('level: 20 lg|H| / dB')
plt.yticks((-60, -40, -20, -12, 0, 6))
plt.title(r'magnitude frequency response of 2nd order lowpass')
plt.legend()
plt.grid(True, which='both')
plt.xlim(w[0]/2/np.pi, w[-1]/2/np.pi)
```

## 2nd Order Bandpass

We assume the causal LTI system with

* zero in origin $s_0=0$
* pole at $s_{\infty,1}=-0.1$
* pole at $s_{\infty,2}=-10$
* gain $H_0=10$

and want to create the magnitude Bode plot approximation.

\begin{align}
H(s) = H_0\frac{s-s_{0,1}}{(s-s_{\infty,1})(s-s_{\infty,2})} = 10\frac{(s-0)}{(s-(-0.1))(s-(-10))}
=\frac{100 s}{10 s^2 + 101 s + 10}
\end{align}

\begin{align}
20\mathrm{lg}|H(s)| = 20\mathrm{lg}|\tilde{H_0}| + 20\mathrm{lg}|s| - 20\mathrm{lg}|\frac{s}{|s_{\infty,1}|}-1|
- 20\mathrm{lg}|\frac{s}{|s_{\infty,2}|}-1|
\end{align}

\begin{align}
\tilde{H_0} = \frac{H_0}{|s_{\infty,1}| \cdot |s_{\infty,2}|} = 10 \rightarrow 20\mathrm{lg}|\tilde{H_0}| = 20 \mathrm{dB}
\end{align}

```python
w = np.arange(1e-2, 1e+2, 0.01)
s = 1j*w
# define LTI system by zeros and poles:
s01 = 0
soo1 = -1/10
soo2 = -10
H0 = 10

# get transfer function Python/Matlab-like handling:
sz = s01
sp = soo1, soo2
sys = signal.lti(sz, sp, H0)
w, Hexact, Hphase = sys.bode(w)
# tf = sys.to_tf()  # get b=num,a=den coefficients
# H0 = tf.num[0] / tf.den[0]  # get H0 from the polynomial version

# Bode approximation:

# gain -> straight line at 20 dB, blue
H0tilde = H0 / np.abs(soo1) / np.abs(soo2)
H0t = np.zeros_like(w) + 20*np.log10(np.abs(H0tilde))

# zero in origin -> line with +20dB/decade through 1 rad/s, orange
Hs01 = +20*np.log10(np.abs(s))

# single pole ->
Hsoo1 = -20*np.log10(np.abs(s/soo1))  # line with -20dB/decade for w>0.1, green
Hsoo1[np.abs(s) <= np.abs(soo1)] = 0  # 0 dB for w<0.1, green

# single pole ->
Hsoo2 = -20*np.log10(np.abs(s/soo2))  # line with -20dB/decade for w>10, red
Hsoo2[np.abs(s) <= np.abs(soo2)] = 0  # 0 dB for w<10, red

Happrox = H0t + Hs01 + Hsoo1 + Hsoo2  # add magnitude frequency responses in dB

Hexact2 = 20*np.log10(H0tilde) + 20*np.log10(np.abs(s)) - 20*np.log10(
    np.abs(s/np.abs(soo1)-1)) - 20*np.log10(np.abs(s/np.abs(soo2)-1))

plt.figure(figsize=(7, 5))
plt.semilogx(w, H0t, color='C0', lw=2,
             label=r'$|\tilde{H_0}|=10\rightarrow 20$ dB')
plt.semilogx(w, Hs01, color='C1', lw=2, label='one zero in origin')
plt.semilogx(w, Hsoo1, color='C2', lw=5,
             label=r'real, single pole at $|s_{\infty,1}|=\frac{1}{10}$')
plt.semilogx(w, Hsoo2, color='C3', lw=3,
             label=r'real, single pole at $|s_{\infty,2}|=10$')
plt.semilogx(w, Happrox, color='k', label='Bode plot approximation')
plt.semilogx(w, Hexact2, lw=1, color='C8')
plt.semilogx(w, Hexact, lw=1, color='C8',
             label='Bode plot exact')

plt.xlim(1e-2, 1e2)
plt.yticks((-60, -40, -20, -10, -3, -6, 0, 6, 20, 40))
plt.ylim(-60, 40)
plt.xlabel(r'$\omega$ / (rad/s)')
plt.ylabel('20 lg|H| / dB')
plt.title(r'magnitude frequency response of $H(s) = 100s\,/\,(10s^2+101s+10)$')
plt.legend()
plt.grid(True, which='both')
plt.savefig('bodeplot_example_approximations_590A7AFD51.pdf')
```

## Highpass 2nd Order with Slight Resonance

We assume the causal LTI system
\begin{align}H(s) = \frac{s^2+10 s}{s^2+\sqrt{2} s +1}\end{align}
and want to create the magnitude Bode plot approximation.

\begin{align}
&s_{0,1} = 0\rightarrow |s_{0,1}| = 0\\
&s_{0,2} = -10 \rightarrow |s_{0,2}| = 10\\
&s_{\infty,1,2} = -\frac{\sqrt{2}}{2} \pm \sqrt{\frac{2}{4}-1} = -\frac{1}{\sqrt{2}} \pm \text{j} \frac{1} {\sqrt{2}} \rightarrow |s_{\infty,1,2}|^2 = 1\\
&|H_0| = 1\\
&|\tilde{H}_0| = 1 \cdot \frac{|s_{0,2}|}{|s_{\infty,1,2}|^2} = 10 \rightarrow 20\text{lg}|\tilde{H}_0|=20 \text{dB}
\end{align}

```python
w = np.arange(1e-2, 1e+2, 0.01)
s = 1j*w
# define LTI system by zeros and poles:
s01 = 0
s02 = 10
soo1 = (-1+1j)/np.sqrt(2)
soo2 = (-1-1j)/np.sqrt(2)
H0 = 1
# get transfer function Python/Matlab-like handling:
sz = s01, s02
sp = soo1, soo2
sys = signal.lti(sz, sp, H0)
w, Hmag, Hphase = sys.bode(w)
# tf = sys.to_tf()  # get b=num,a=den coefficients
# H0 = tf.num[0] / tf.den[0]  # get H0 from the polynomial version

# Bode approximation:

# gain -> straight line at 20 dB, blue
H0tilde = H0 * np.abs(s02) / np.abs(soo1)**2
H0t = np.zeros_like(w) + 20*np.log10(np.abs(H0tilde))

# single zero in origin -> line with +20dB/decade through 1 rad/s, orange
Hs01 = +20*np.log10(np.abs(s))

# single zero ->
Hs02 = +20*np.log10(np.abs(s/s02))  # line with +20dB/decade for w>10, green
Hs02[np.abs(s) <= np.abs(s02)] = 0  # 0 dB for w<10, green

# conj complex pole ->
Hsoo1 = -2*20*np.log10(np.abs(s/soo1))  # line with -40dB/decade for w>1, red
Hsoo1[np.abs(s) <= np.abs(soo1)] = 0  # 0 dB for w<1, red

Happrox = H0t + Hs01 + Hs02 + Hsoo1  # add magnitude frequency responses in dB

plt.figure(figsize=(7, 5))
plt.semilogx(w, H0t, color='C0', lw=2,
             label=r'$|\tilde{H_0}|=10\rightarrow 20$ dB')
plt.semilogx(w, Hs01, color='C1', lw=2, label=r'zero in origin $s_{0,1}$')
plt.semilogx(w, Hs02, color='C2', lw=3, label=r'zero $|s_{0,2}|=10$')
plt.semilogx(w, Hsoo1, color='C3', lw=2,
             label='conj complex pole $|s_{oo,1,2}|=1$')
plt.semilogx(w, Happrox, color='k', lw=2, label='Bode Plot Approximation')
plt.semilogx(w, Hmag, lw=1, color='C8', label='Bode Plot Exact')
plt.xlim(1e-2, 1e2)
plt.ylim(-60, 40)
plt.yticks((-60, -40, -20, -12, -6, 0, 6, 12, 20, 40))
plt.xlabel(r'$\omega$ / (rad/s)')
plt.ylabel('20 lg|H| / dB')
plt.title(
    r'magnitude frequency response of $H(s) = (s^2+10 s)\,/\,(s^2 + \sqrt{2} s +1)$')
plt.legend()
plt.grid(True, which='both')
plt.savefig('bodeplot_example_approximations_1EC316D8B2.pdf')
```

## Copyright

This tutorial is provided as Open Educational Resource (OER), to be found at
https://github.com/spatialaudio/signals-and-systems-exercises
accompanying the OER lecture
https://github.com/spatialaudio/signals-and-systems-lecture.
Both are licensed under a) the Creative Commons Attribution 4.0 International
License for text and graphics and b) the MIT License for source code.
Please attribute material from the tutorial as *Frank Schultz,
Continuous- and Discrete-Time Signals and Systems - A Tutorial Featuring
Computational Examples, University of Rostock* with
``main file, github URL, commit number and/or version tag, year``.