```python
import numpy as np
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Homework 4

**Instructions:** Complete the notebook below. Download the completed notebook in HTML format. Upload assignment using Canvas.

**Due:** Feb. 2 at **2pm.**

## Exercise: First-Order Difference Equation - Temporary Shocks

For each of the first-order difference equations below, compute $y$ for $31$ periods (i.e., $t=0,\ldots, 30$) with $y_0=0$ and  $w$ equal to one in the second period ($t=1$) and zero otherwise (i.e., $w_1 = 1$ and $w_0=w_2=w_3 \cdots = w_{30} = 0$). Construct a well-labeled plot of $y$ for each simulation.

1. $y_{t} = 0.56 y_{t-1} + w_t$
2. $y_{t} = y_{t-1} + w_t$
3. $y_{t} = 0.24 y_{t-1} + w_t$
4. $y_{t} = -.9 y_{t-1} + w_t$
5. $y_{t} = 1.45 y_{t-1} + w_t$

You may use the function from the Notebook for Lecture 7, write your own, or take a more brute force approach.

### $\rho = 0.56$

```python

```

### $\rho =1$

```python

```

### $\rho=0.24$

```python

```

### $\rho = -0.9$

```python

```

### $\rho = 1.45$

```python

```

**Questions**

For each value for $\rho$, identify whether the first-order difference equation is stable, explosive, or neither.

1. $\rho =  0.56$
2. $\rho = 1$
3. $\rho =  0.24$
4. $\rho =  -.9$
5. $\rho =  1.45$

**Answers**

1.

2.

3.

4.

5.

## Exercise: First-Order Difference Equation - Permanent Shocks

For each of the first-order difference equations below, compute $y$ for $31$ periods (i.e., $t=0,\ldots, 30$) with $y_0 = 0$ and $w$ equal to zero in the first period ($t=0$) and one otherwise (i.e., $w_1 = 0$ and $w_1=w_2=w_3 \cdots = w_{30} = 1$). Note that you can use the NumPy function `ones()` to create an array of ones analagously to how you use `zeros()` to create an array of zeros.

Construct a well-labeled plot of $y$ for each simulation.

1. $y_{t} = 0.37 y_{t-1} + w_t$
2. $y_{t} = y_{t-1} + w_t$
3. $y_{t} = -0.8 y_{t-1} + w_t$

You may use the function from the Notebook for Lecture 7, write your own, or take a more brute force approach.

### $\rho = 0.37$

```python

```

### $\rho = 1$

```python

```

### $\rho = -0.8$

```python

```

## Exercise: Hall's Model of Unemployment

In the 1979 article "A theory of the natural unemployment rate and the duration of employment" Robert Hall proposed a dynamic model of unemployment that is the foundation of contemporary models of unemployment (link to article: [https://www.sciencedirect.com/science/article/abs/pii/0304393279900011](https://www.sciencedirect.com/science/article/abs/pii/0304393279900011)).

Let $U_t$ denote the number of people unemployed in month $t$ and let $E_t$ denote the number of employed workers. Assume that the labor force is fixed at size $L$. By definition, the labor force equals the sum of the number of employed and unemployed workers:

\begin{align}
L & = E_t + U_t.
\end{align}

Each month, a randomly-determined fraction $f$ of unemployed workers find jobs and a fraction $s$ of employed workers lose, or become separated from, their jobs. We call $f$ the *job-finding rate* and $s$ the *separation rate*. The implied law of motion for the number of unemployed people is:

\begin{align}
U_{t+1} & = (1-f)U_t + sE_t.
\end{align}

Use the definition of the labor force to elimiate $E_t$ from the law of motion for unemployment:

\begin{align}
U_{t+1} & = (1-f)U_t + s(L - U_t),
\end{align}

and then divide by $L$ to express the law of motion in terms of the unemployment rate $u_t$:

\begin{align}
u_{t+1} & = (1-f)u_t + s(1-u_t),
\end{align}

where $u_t \equiv U_t/L$.

In this exercise, you will:

1. Calibrate $f$ so that the model predicts average unemployment duration equal to the US average.
2. Calibrate $s$ so that the steady state unemployment rate in the model equals the average unemployment rate of the US.
3. Compute a dynamic simulation of the model.

Follow the instructions in the following cells.

### Part (a): Mean Unemployment Duration

A constant job-finding rate $f$ implies that the average number of months that an unemployed worker is inversely proportional to $f$:

\begin{align}
\text{Average duration} & = \frac{1}{f}
\end{align}

See page 27 of my paper with Paul Jackson and Argya Bhattacharya for an explanation of why this makes sense: https://www.tandfonline.com/doi/full/10.1080/00220485.2017.1397574. But, for example, if the job-finding rate were 0.4, then a worker who become unemployed will wait 2.5 months on average before finding a new job.

For this part, use mean unemployment duration data from FRED (https://fred.stlouisfed.org/series/UEMPMEAN/) to compute the average job-finding rate for the US.

```python
# Download mean unemployment duration data from FRED. PROVIDED
duration = pd.read_csv('https://fred.stlouisfed.org/data/UEMPMEAN.txt',skiprows=13,sep='\s+',index_col=0,parse_dates=True)

# Set duration equal to itself divided by 4 to convert weekly units into monthly units

```

```python
# Construct a well-labeled plot of mean unemployment duration

```

```python
# Create a variable called 'f' equal to one divided by the mean of mean unemployment duration

# Print the value of f

```

**Questions**

1. On average, how many months has the average unemployed worker in the US remained unemployed?
2. What was the highest unemployment duration and in which month and year did this occur?

**Answers**

1.

2.

### Part (b): Average Unemployment Rate

In the steady state, $u_t = u_{t+1} = u^*$:

\begin{align}
u^* & = \frac{1}{1+f/s}
\end{align}

Unlike the job-finding rate $f$, it is harder to find data that would directly pin down a value for the separation rate $s$. So we will **calibrate** $s$ by selecting a value implies that the steady state unemployment rate equals the average unemployment rate for the US.

For this part, use unemployment rate data from FRED (https://fred.stlouisfed.org/series/UNRATE/) to compute the average unemployment rate for the US and then the implied value for $s$.

```python
# Download unemployment rate data from FRED. PROVIDED
unemployment = pd.read_csv('https://fred.stlouisfed.org/data/UNRATE.txt',skiprows=23,sep='\s+',index_col=0)

# Set unemployment equal to itself divided by 100

# Create a variable called 'u_star' that is equal to the mean unemployment rate for the US

# Print u_star

```

```python
# Create a variable called 's' equal to the separation rate implied by the computed values f and u_star

# Print s

```

### Part (c): Simulation of Hall's Model

Use Hall's model to simulate the unemployment rate for 26 periods (i.e., $t=0, \ldots, 25$) for $u_0= 0.01$ and $u_0 = 0.05$. Plot both simulations on the same set of axes and be sure to add a legend so that I can see which line corresponds to which value of $u_0$.

```python

```

## Exercise: Solow Model with Exogenous Labor and Technology

Consider the Solow growth model with exogenous labor and TFP growth. That is, let $L_t$ denote the quantity of labor and suppose that the population (and therefore the labor supply) grows at the constant rate $n$. This means:

\begin{align}
L_{t+1} & = (1+n)L_t,
\end{align}

where the intial value of labor $L_0$ must be given. And let $A_t$ denote exogenous TFP that grows at rate $g$:

\begin{align}
A_{t+1} & = (1+g)A_t,
\end{align}

with $A_0$ given. The rest of the model in aggregate terms is:

\begin{align}
Y_t & = A_tK_t^{\alpha}L_t^{1-\alpha}\\
K_{t+1} & = I_t + (1-\delta)K_{t}\\
Y_t & = C_t + I_t\\
C_t & = (1-s)Y_t,
\end{align}

where the intial value of capital $K_0$ is also given.

Since capital *and* labor *and* TFP are all determined in the previous period by their respective laws of motion, the Solow model with exogenous labor and TFP growth has *three* state variables: $K_t$, $L_t$, and $A_t$. We can solve the capital law of motion for capital in terms of only the variables capital, labor , and TFP and so the three equations that determine how the state of the economy evolves are:

\begin{align}
K_{t+1} & = sA_tK_t^{\alpha}L_t^{1-\alpha} + (1-\delta)K_{t}\\
L_{t+1} & = (1+n)L_t\\
A_{t+1} & = (1+g)A_t
\end{align}

If we iterate on these two to compute simulations of $K_t$ and $L_t$, then we can compute $Y_t$, $C_t$, and $I_t$ easily.

Let's suppose the following values for a simulation:

| $$L_0$$ | $$n$$  | $$A_0$$ | $$g$$  | $$K_0$$ | $$s$$  | $$\alpha$$ | $$\delta $$ | $$T$$ |
|---------|--------|---------|--------|---------|--------|------------|-------------|-------|
| 1       | 0.01   | 1       | 0.02   | 10      | 0.1    | 0.35       | 0.1         | 101   |

Do the following:
1. Compute simulated values for $K_t$ and $L_t$ and $A_t$.
2. Use simulated values of $K_t$ and $L_t$ and $A_t$ to compute and plot simulated series for output $Y_t$ and output *per worker* $Y_t/L_t$ side-by-side in the same figure.

You may use the function from the Notebook for Lecture 8, write your own, or take a more brute force approach.

```python
# Create variables 'K0', 'L0', 'A0', 'n', 'g', 's', 'alpha','delta','T' to store parameter values for the simulation

# Simulate the model

# Create a figure that is wide enough for two axes

# Construct a plot of simulated output

# Construct a plot of simulated output per worker

```


# **[HW5] Practice_3**
1. Install packages
2. Determinant
3. Change of basis
4. Least Square

선형대수 실습은, exercise 혹은 coding 문제풀이에 집중하기 보다는, 수업시간에 배웠던 개념들을 visualize 하면서 이를 폭넓게 이해하는 데에 초점을 맞추고 있습니다. 실습에서 사용한 예시 외에도, 다양한 matrix들을 visualize 하면서 개념을 익혀보시기 바랍니다.

# 1. Install packages

> 필요한 package를 설치하고 import합니다

```python
# visualization을 위한 helper code입니다.
from urllib.request import urlretrieve
URL = 'https://go.gwu.edu/engcomp4plot'
urlretrieve(URL, 'plot_helper.py')

import sys
sys.path.append('../scripts/')

# 다음 세 custom function (1)plot_vector, (2)plot_linear_transformation, (3) plot_linear_transformations
# 을 사용할 것입니다.
from plot_helper import *
```

```python
import matplotlib.pyplot as plt
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
import scipy as sp
import scipy.linalg
import sympy as sy

sy.init_printing()
np.set_printoptions(precision=3)
np.set_printoptions(suppress=True)
```

```python
def plotVectors(vecs, colors, alpha=1):
    """
    Determinant 섹션의 plot에 사용하기 위한 custom function입니다.

    Parameters
    ----------
    vecs : plot할 vector(numpy array)의 list  (e.g., [[1, 3], [2, 2]] )
    colors : 각 vector의 color  (e.g., ['red', 'blue'])
    alpha : 투명도

    Returns
    -------
    fig : figure
    """
    plt.axvline(x=0, color='#A9A9A9', zorder=0)
    plt.axhline(y=0, color='#A9A9A9', zorder=0)

    for i in range(len(vecs)):
        if (isinstance(alpha, list)):
            alpha_i = alpha[i]
        else:
            alpha_i = alpha
        x = np.concatenate([[0,0],vecs[i]])
        plt.quiver([x[0]], [x[1]], [x[2]], [x[3]], angles='xy',
                   scale_units='xy', scale=1, color=colors[i],alpha=alpha_i)
```

# 2. Determinant

> 강의에서는 간략하게 다루고 넘어간 determinant에 대해 좀더 알아보겠습니다. 저번 수업 및 실습에서 배웠듯, matrix는 linear transformation으로 생각할 수 있습니다. Determinant는, 이러한 transformation이 일어날 때, "multiplicative change"가 일어나는 정도" 라고 생각할 수 있습니다. Multiplicative change는 크게 두 가지를 생각하면 됩니다: (1) 어떠한 input이 matrix $A$를 통과하게 되면 "부피(volume)"가 얼마나 변하는가, (2) 어떠한 input이 matrix $A$를 통과하게 되면 "방향(orientation)"이 변하는가. 두 성질에 대해 각각 알아보겠습니다.

 출처: Ref. [1]

### Example 1

> 위 그림은 $\mathbf{i}, \mathbf{j}$로 이루어진 unit cube 입니다. $\mathbf{i}, \mathbf{j}$가 각각 길이가 1이므로, 넓이도 1임을 알 수 있습니다.

```python
orange = '#FF9A13'
blue = '#1190FF'

i = [0, 1]
j = [1, 0]

fig = plt.figure(figsize = (4,4))
plotVectors([i, j], [[blue], [orange]], alpha=1)
plt.plot([0, 1], [1, 1], ls = '--', color = 'black', alpha = .5)
plt.plot([1, 1], [0, 1], ls = '--', color = 'black', alpha = .5)

plt.xlim(-0.5, 3)
plt.ylim(-0.5, 3)
plt.show()
```

> 이제 위 vector를 matrix $A = \begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix}$에 통과시켜 보겠습니다.

```python
A = np.array([[2, 0], [0, 2]])

new_i = A.dot(i)
new_j = A.dot(j)

fig = plt.figure(figsize = (4,4))
plotVectors([new_i, new_j], [[blue], [orange]])
plt.plot([0, 2], [2, 2], ls = '--', color = 'black', alpha = .5)
plt.plot([2, 2], [0, 2], ls = '--', color = 'black', alpha = .5)

plt.xlim(-0.5, 3)
plt.ylim(-0.5, 3)
plt.show()
```

> 예상하셨겠지만, $A$는 diagonal matrix이기 때문에 $\mathbf{i}, \mathbf{j}$ 를 각각 2배로 rescale 함을 알 수 있습니다 ($\mathbf{i,j} \rightarrow \mathbf{2i,2j}$). 따라서 주황색, 파랑색 vector는 rotate, reflection 등이 없이 각자 2배 길어지는 것이 전부입니다. 따라서 넓이는 4(사각형 넓이 = 밑변 * 높이)가 됐습니다. 이제 determinant를 계산해봅시다.

```python
np.linalg.det(A)
```

> 넓이가 1에서 4, 즉 volume 4배가 됐는데 이는 $A$의 determinant와 일치함을 알 수 있습니다.

### Example 2

이제 두 번째 성질, 방향(orientation)에 대해 알아보겠습니다. 이번 예시에선 $\mathbf{i}, \mathbf{j}$로 이루어진 unit cube에 $B = \begin{bmatrix} -2 & 0 \\ 0 & 2 \end{bmatrix}$를 통과시켜 보겠습니다.

```python
orange = '#FF9A13'
blue = '#1190FF'

i = [0, 1]
j = [1, 0]

fig = plt.figure(figsize = (4,4))
plotVectors([i, j], [[blue], [orange]])
plt.xlim(-0.5, 3)
plt.ylim(-0.5, 3)
plt.show()
```

```python
B = np.array([[-2, 0], [0, 2]])
new_i_1 = B.dot(i)
new_j_1 = B.dot(j)

fig = plt.figure(figsize = (4,4))
plotVectors([new_i_1, new_j_1], [['#1190FF'], ['#FF9A13']])
plt.xlim(-3, 0.5)
plt.ylim(-0.5, 3)
plt.show()
```

> 이번에도 각 주황색, 파랑색 vector의 길이가 2배가 됐음은 동일합니다. 따라서 determinant의 절대값은 4가 나와야 합니다.

```python
np.linalg.det(B)
```

> 절대값이 4이긴 하지만, 앞에 - 부호가 붙어있음을 알 수 있습니다. 이는 방향(orientation)을 보존하지 못했다는 의미입니다.

 출처: Ref. [1]

> 좌측 그림의 경우, 회전 혹은 길이의 조절을 통해서 변환이 가능한 경우입니다. 하지만 우측 그림의 경우, 오른쪽 h를 회전하거나 길이를 조정한다고 해서 왼쪽 h를 얻을 수 없고, reflection이 필요합니다. 이에 대해 더 구체적인 내용은, 다음 wiki(https://en.wikipedia.org/wiki/Determinant) 를 참조하시기 바랍니다.

> 이와 같이, determinant의 절대값은 크기(volume)가 변화하는 정도, 부호는 방향(orientation)의 변화 여부를 알려줍니다.

### Exercise

> 어떤 matrix의 determinant가 0일 경우, 해당 matrix는 invertible하지 않습니다. 위의 "multiplicative change"로 해석했을 때, 이는 어떤 의미일까요?

예를들어, 2x2 matrix $A$의 rank가 1일 경우, column space는 1-dimensional subspace일 것입니다. 이 말은, input 2-dimensional $R^2$가 $A$를 거치면서 1-dimensional subspace로 squish된다는 것을 의미합니다.

즉 위의 길이가 1인 정사각형 예시로 따지자면, $A$를 거친 이후에 높이가 0이 된다고 생각할 수 있습니다 (2차원 짜리 사각형이 1차원 직선이 됐으니까). 따라서 넓이가 $1 \rightarrow 0$이 됐으니까 determinant는 0이 됩니다.

이 설명으로부터 non-invertible에 대해 이해해볼 수 있습니다. $A$를 **함수**라고 생각해본다면, single input에 대해 single output이 나와야 합니다. 그런데 $A$의 역행렬은, 1-dimensional subspace를 2-dimensional $R^2$로 복원해야 하는데, 즉 sinlge input에 대해 multiple output이 나와야 하므로, $A^{-1}$라는  function이 존재할 수 없는 것입니다.

# Change of basis

> 저번 실습 시간에 basis를 배웠고, change of basis도 간략하게 다뤘습니다. "왜 change of basis를 해야하는가?"에 대한 대답은 eigen decomposition 시간에 자세히 알아보기로 하고, 오늘은 이를 하는 방법에 대해 예시를 들어 알아보겠습니다.

> 모든 vector는 basis vector의 linear combination으로 표현할 수 있습니다. 예를 들어, 다음과 같은 vector $v = \begin{bmatrix}-3\\0\end{bmatrix}$를 $R^2$의 basis $\Big\{ \mathbf{a} = \begin{bmatrix}2\\1\end{bmatrix}$, $\mathbf{b} = \begin{bmatrix}1\\2\end{bmatrix} \Big\}$의 linear combination으로 나타내고 싶다고 합시다.

```python
C = np.array([[2,1],[1,2]])  # 각 column이 a,b인 행렬
```

> 이전 실습에서 배웠던것처럼, 같은 vector는 basis를 어떻게 잡냐에 따라 다른 좌표를 갖게 됩니다. Standard basis를 사용한다면 ($\Big\{\mathbf{i}, \mathbf{j}\Big\}$), vector $v$의 좌표는 (-3,0)이 될 것입니다. 저희가 궁금한 것은, basis basis $\Big\{ \mathbf{a} = \begin{bmatrix}2\\1\end{bmatrix}$, $\mathbf{b} = \begin{bmatrix}1\\2\end{bmatrix} \Big\}$를 사용했을 때의 좌표입니다. 즉,

$$Cx = v $$

 (component 1의 좌표 * $[2,1]^T$) + (component 2의 좌표 * $[1,2]^T$) = v

> 를 만족하는 x를 찾고 싶은 것입니다.

```python
v = np.array([-3,0])
x = np.linalg.solve(C, v)  # numpy의 solve 함수를 이용해서도 Ax=b의 해를 구할 수 있습니다
x
```

    array([-2.,  1.])

> Vector $v$는, $\Big\{ \mathbf{a}, \mathbf{b} \Big\}$의 basis에서 좌표가 (-2,1)이 됨을 알 수 있습니다.

> 이를 visualize 해봅시다.

```python
a = np.array([2,1])
b = np.array([1,2])
plot_change_basis (a, b, v)  # standard basis에서 {a,b} basis로 바꿔주는 custom function 입니다.
```

> 빨간색 vector는 $v$,  갈색 vector는 $i$와 $a$,  파란색 vector는 $j$와 $b$ 입니다. 왼쪽 plot에선, $v$는 -3*$\mathbf{i}$ + 0*$\mathbf{j}$임을 알 수 있고, 따라서 좌표가 (-3,0)이 됩니다 (저희가 일반적으로 알고 있는 standard basis에서의 좌표입니다). 오른쪽 plot에선, **같은 vector $v$**가, -2*$\mathbf{a}$ + 1*$\mathbf{b}$로 표현됨을 알 수 있고, 따라서 좌표가 (-2,1)이 됩니다.

* New basis로 삼을 $\mathbf{a}, \mathbf{b}$를 각 column으로 하는 matrix $C$는, $\Big\{ \mathbf{a}, \mathbf{b} \Big\}$ basis에서의 좌표 $x$를 $\rightarrow$ standard basis $\Big\{ \mathbf{i}, \mathbf{j} \Big\}$에서의 좌표 $v$로 변환해주는 matrix이다 ($Cx = v$, 즉 x는 {a,b}의 좌표, v는 {i,j}의 좌표).

> 그럼 반대로, $\Big\{\mathbf{i}, \mathbf{j}\Big\}$의 좌표를 $\Big\{\mathbf{a}, \mathbf{b}\Big\}$의 좌표로 변환시키려면 어떤 matrix를 곱하면 될까요? 당연하게도, $C^{-1}$임을 알 수 있습니다.

$$ x = C^{-1}v$$

> 하나만 더 나아가 생각해봅시다. 만약 우리가 어떤 vector를 standard basis 상에서(즉 왼쪽 plot에서) $x$를 반시계 방향으로 90도 회전하고 싶은데, $x$의 $\Big\{\mathbf{a}, \mathbf{b}\Big\}$에서의 좌표 (즉 오른쪽 plot에서의 좌표) 만이 주어져 있다고 합시다. 그러면 어떻게 해야 할까요?

> 이를 계산하기 위해선, 다음과 같은 단계를 거쳐야 합니다.

1.   $\Big\{\mathbf{a}, \mathbf{b}\Big\}$의 좌표를 $\Big\{\mathbf{i}, \mathbf{j}\Big\}$로 바꾼다 (change basis: $C$).
2.   $\Big\{\mathbf{i}, \mathbf{j}\Big\}$ 상에서 반시계 방향으로 90도 회전을 한다 (linear transformation: $R$).
3.   다시 $\Big\{\mathbf{i}, \mathbf{j}\Big\}$의 좌표를 $\Big\{\mathbf{a}, \mathbf{b}\Big\}$의 좌표로 옮긴다 (change basis: $C^{-1}$) .

> 이 셋을 연달아 수행하게 되면, $\Big\{\mathbf{a}, \mathbf{b}\Big\}$ basis에서의 linearly transformed 된 좌표를 얻을 수 있게 됩니다.

$$ \text{새로운 좌표} = C^{-1}RCx$$

```python
x = np.array([-2, 1])

# 1
x_1 = C@x
print('1. x_ij:',x_1)

# 2
R = np.array([[0,-1],[1,0]])
x_2 = R@x_1
print('2. rotated_x_ij:',x_2)

# 3
C_inv = np.linalg.inv(C)
x_3 = C_inv@x_2
print('3: rotated_x_ab:', x_3)
```

    1. x_ij: [-3  0]
    2. rotated_x_ij: [ 0 -3]
    3: rotated_x_ab: [ 1. -2.]

### Exercise

$\Big\{\mathbf{a}, \mathbf{b}\Big\}$ basis에서 좌표가 (-7, 11)인 vector $v$가 있습니다. $v$에 대해 $\Big\{\mathbf{i}, \mathbf{j}\Big\}$ 상에서 $T = \begin{bmatrix} 3 & 5 \\ 2 & 11 \end{bmatrix}$로 표현되는 linear transformation을 적용했을 때, 변환된 좌표를 구하시오. ($\mathbf{a}, \mathbf{b}$는 위 예시와 같습니다)

```python
C
```

    array([[2, 1],
           [1, 2]])

```python
#answer

T = np.array([[3,5],[2,11]])

v = np.array([-7, 11])

# 1. a,b 좌표계로 바꾼다
v_1 = C@v
print(v_1)

#2. a,b 좌표계에서 T를 수행한다
v_2 = T@v_1
print(v_2)

#3. i,j 좌표계로 다시 되돌린다
v_3 = C_inv@v_2
print(v_3)
```

    [-3 15]
    [ 66 159]
    [-9. 84.]

```python
C_inv@T@C@v
```

    array([-9., 84.])

### Exercise

$\Big\{\mathbf{i}, \mathbf{j}\Big\}$ basis에서 좌표가 (-7, 11)인 vector $v$가 있습니다. $v$에 대해 $\Big\{\mathbf{a}, \mathbf{b}\Big\}$ 상에서 $T = \begin{bmatrix} 3 & 5 \\ 2 & 11 \end{bmatrix}$로 표현되는 linear transformation을 적용했을 때, 변환된 좌표를 구하시오. ($\mathbf{a}, \mathbf{b}$는 위 예시와 같습니다)

```python
#answer

T = np.array([[3,5],[2,11]])

v = np.array([-7, 11])

# 1. a,b 좌표계로 바꾼다
v_1 = C_inv@v
print(v_1)

#2. a,b 좌표계에서 T를 수행한다
v_2 = T@v_1
print(v_2)

#3. i,j 좌표계로 다시 되돌린다
v_3 = C@v_2
print(v_3)
```

    [-8.333  9.667]
    [23.333 89.667]
    [136.333 202.667]

```python
C@T@C_inv@v
```

    array([136.333, 202.667])

# Least square

> 강의에서 배웠듯, least square 문제는 $Ax = b$의 해가 존재하지 않을 때, 그나마 $Ax$의 값이 $b$와 가장 가까운 (i.e., 둘 사이의 L2 norm을 가장 작게 하는) $x$를 찾으려고 하는 것입니다.

$$
Ax = \hat{y} \approx b \\
\begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \\ a_{41} & a_{42} & a_{43} \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} = \begin{bmatrix} \hat{y_1} \\ \hat{y_2} \\ \hat{y_3} \\ \hat{y_4} \end{bmatrix} \approx  \begin{bmatrix} b_1 \\ b_2 \\ b_3 \\ b_4 \end{bmatrix}
$$

> 구체적으로는, Error $E = ||b-Ax||^2_2 = ||b-\hat{y}||^2_2$를 minimize하는 x를 찾는 문제가 least square 문제입니다. 강의 및 실습에서 배웠듯, $b$가 $A$의 column space에 존재한다면 $E = 0$을 만드는 $x$가 존재할 것입니다. 만약 $b$가 $A$의 column space 밖에 있다면, $E$는 0보다 크게 됩니다.

> 강의의 geometric interpretation of least square에서 알 수 있듯, 해당 $x$는

$$A^T(b - A\hat{x}) = 0 \\
A^TA\hat{x} = A^Tb$$

> 으로 구할 수 있습니다. 만약 $A^TA$가 invertible 하다면, $x = (A^TA)^{-1}A^Tb$ 로 구할 수 있습니다.

# 4. Least square practice

> 다음과 같은 data가 있다고 합시다. 몸무게, 키, 흡연 여부로 life-span을 맞추는 문제입니다.

* data = $\begin{array}{ccc}
& Weight & Height & Is \; smoking & Life-span\\
Person 1 & 60 & 177 & 1(=yes) & 66\\
Person 2 & 65 & 170 & 0(=no) & 74\\
Person 3 & 55 & 175 & 0 & 78\\
Person 4 & 70 & 180 & 1 & 72\\
Person 5 & 45 & 155 & 1 & 70 \\
Person 5 & 50 & 160 & 0 & 80
\end{array}$

> Least square를 이용해서 $E$를 최소화하는 x를 구해봅시다.

### Exercise

```python
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('seaborn-poster')
```

```python
b = np.array([66,74,78,72, 70,80])

# (1) Weight만 사용해서 life-span을 예측해봅시다.

A_1 = np.array([60,65,55,70,45,50]).reshape(6,1)
x_1 = np.dot((np.dot(np.linalg.inv(np.dot(A_1.T,A_1)),A_1.T)),b)
error_1 = ((A_1@x_1 - b)**2).sum()

# (2) Weight, Height 2개를 이용해 life-span을 예측해봅시다.

A_2 = np.array([[60,177],[65,170],[55,175],[70,180],[45,155],[50,160]])
x_2 = np.dot((np.dot(np.linalg.inv(np.dot(A_2.T,A_2)),A_2.T)),b)
error_2 = ((A_2@x_2 - b)**2).sum()

# (2) Weight, Height, Is smoking 모두 사용해서 life-span을 예측해봅시다.

A_3 = np.array([[60,177,1],[65,170,0],[55,175,0],[70,180,1],[45,155,1],[50,160,0]])
x_3 = np.dot((np.dot(np.linalg.inv(np.dot(A_3.T,A_3)),A_3.T)),b)
error_3 = ((A_3@x_3 - b)**2).sum()

print("error_1: %.4f, error_2: %.4f, error_3: %.4f"%(error_1, error_2, error_3))
```

    error_1: 954.2540, error_2: 205.7798, error_3: 102.9521

> 위 코드를 완성시키고, 왜 error에 차이가 나는지 생각해봅시다.

> life-span을 맞추는데 도움이 되는 정보를 더 많이 사용할수록, 예측의 정확도가 올라간다.

>Quiz 1처럼 우리가 배운 식을 이용해 구할 수도 있지만, 이미 만들어진 함수들을 활용할 수도 있습니다.

### Exercise

> 2-1) Use np.linalg.lstsq (https://numpy.org/doc/stable/reference/generated/numpy.linalg.lstsq.html)

```python
A = A_3
x_4 = np.linalg.lstsq(A,b,rcond=None)
error_4 = ((A@x_4[0] - b)**2).sum()
error_4
```

> 2-2) Use optimize.curve_fit from scipy (https://docs.scipy.org/doc/scipy/reference/reference/generated/scipy.optimize.curve_fit.html?highlight=curve_fit#scipy.optimize.curve_fit)

```python
from scipy import optimize

# 주의: 이 방식은 속도 등을 고려하지 않은, 예시 코드입니다.
def func(data, x_1, x_2, x_3):
  return data[0]*x_1 + data[1]*x_2 + data[2]*x_3
```

```python
weight = []
height = []
smoke = []
for item in A:
  weight.append(item[0])
  height.append(item[1])
  smoke.append(item[2])
```

```python
x_5 = optimize.curve_fit(func, xdata = [weight,height,smoke], ydata = list(b))[0]
```

```python
error_5 = ((A@x_5 - b)**2).sum()
error_5
```

# Reference

1. https://github.com/akhilvasvani/Linear-Algebra-Basics
2. https://github.com/jclosure/EngComp4_landlinear

3. https://github.com/MacroAnalyst/Linear_Algebra_With_Python


# Random Signals and LTI-Systems

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Auto-Correlation Function

The auto-correlation function (ACF) $\varphi_{yy}[\kappa]$ of the output signal of an LTI system $y[k] = \mathcal{H} \{ x[k] \}$ is derived. It is assumed that the input signal is a wide-sense stationary (WSS) real-valued random process and that the LTI system has a real-valued impulse repsonse $h[k] \in \mathbb{R}$.

Introducing the output relation $y[k] = h[k] * x[k]$ of an LTI system into the definition of the ACF and rearranging terms yields

\begin{equation}
\begin{split}
\varphi_{yy}[\kappa] &= E \{ y[k+\kappa] \cdot y[k] \} \\
&= E \left\{  \sum_{\mu = -\infty}^{\infty} h[\mu] \; x[k+\kappa-\mu] \cdot
\sum_{\nu = -\infty}^{\infty} h[\nu] \; x[k-\nu] \right\} \\
&= \underbrace{h[\kappa] * h[-\kappa]}_{\varphi_{hh}[\kappa]} * \varphi_{xx}[\kappa]
\end{split}
\end{equation}

where the ACF $\varphi_{hh}[\kappa]$ of the deterministic impulse response $h[k]$ is commonly termed as *filter ACF*. This is related to the [link between ACF and convolution](../random_signals/correlation_functions.ipynb#Definition). The relation above is known as the *Wiener-Lee theorem*. It states that the ACF of the output $\varphi_{yy}[\kappa]$ of an LTI system is given by the convolution of the input signal's ACF $\varphi_{xx}[\kappa]$ with the filter ACF $\varphi_{hh}[\kappa]$. For a system which just attenuates the input signal $y[k] = A \cdot x[k]$ with $A \in \mathbb{R}$, the ACF at the output is given as $\varphi_{yy}[\kappa] = A^2 \cdot \varphi_{xx}[\kappa]$.

### Example - System Response to White Noise

Let's assume that the wide-sense ergodic input signal $x[k]$ of an LTI system with impulse response $h[k] = \text{rect}_N[k]$ is normal distributed white noise. Introducing $\varphi_{xx}[\kappa] = N_0\, \delta[\kappa]$ and $h[k]$ into the Wiener-Lee theorem yields

\begin{equation}
\varphi_{yy}[\kappa] = N_0 \cdot \varphi_{hh}[\kappa] = N_0 \cdot (\text{rect}_N[\kappa] * \text{rect}_N[-\kappa])
\end{equation}

The example is evaluated numerically for $N_0 = 1$ and $N=5$

```python
import numpy as np
import matplotlib.pyplot as plt

L = 10000  # number of samples
K = 30  # limit for lags in ACF

# generate input signal (white Gaussian noise)
np.random.seed(2)
x = np.random.normal(size=L)
# compute system response
y = np.convolve(x, [1, 1, 1, 1, 1], mode='full')

# compute and truncate ACF
acf = 1/len(y) * np.correlate(y, y, mode='full')
acf = acf[len(y)-K-1:len(y)+K-1]
kappa = np.arange(-K, K)

# plot ACF
plt.figure(figsize=(10, 6))
plt.stem(kappa, acf, use_line_collection=True)
plt.title('Estimated ACF of output signal $y[k]$')
plt.ylabel(r'$\hat{\varphi}_{yy}[\kappa]$')
plt.xlabel(r'$\kappa$')
plt.axis([-K, K, 1.2*min(acf), 1.1*max(acf)])
plt.grid()
```

**Exercise**

* Derive the theoretic result for $\varphi_{yy}[\kappa]$ by calculating the filter-ACF $\varphi_{hh}[\kappa]$.
* Why is the estimated ACF $\hat{\varphi}_{yy}[\kappa]$ of the output signal not exactly equal to its theoretic result $\varphi_{yy}[\kappa]$?
* Change the number of samples `L` and rerun the example. What changes?

Solution: The filter-ACF is given by $\varphi_{hh}[\kappa] = \text{rect}_N[\kappa] * \text{rect}_N[-\kappa]$. The convolution of two rectangular signals $\text{rect}_N[\kappa]$ results in a triangular signal. Taking the time reversal into account yields

\begin{equation}
\varphi_{hh}[\kappa] = \begin{cases}
N - |\kappa| & \text{for } -N < \kappa \leq N \\
0 & \text{otherwise}
\end{cases}
\end{equation}

for even $N$. The estimated ACF $\hat{\varphi}_{yy}[\kappa]$ differs from its theoretic value due to the statistical uncertainties when using random signals of finite length. Increasing its length `L` lowers the statistical uncertainties.

## Cross-Correlation Function

The cross-correlation functions (CCFs) $\varphi_{xy}[\kappa]$ and $\varphi_{yx}[\kappa]$ between the in- and output signal of an LTI system $y[k] = \mathcal{H} \{ x[k] \}$ are derived. As for the ACF it is assumed that the input signal originates from a wide-sense stationary real-valued random process and that the LTI system's impulse response is real-valued, i.e. $h[k] \in \mathbb{R}$.

Introducing the convolution into the definition of the CCF and rearranging the terms yields

\begin{equation}
\begin{split}
\varphi_{xy}[\kappa] &= E \{ x[k+\kappa] \cdot y[k] \} \\
&= E \left\{ x[k+\kappa] \cdot \sum_{\mu = -\infty}^{\infty} h[\mu] \; x[k-\mu] \right\} \\
&= \sum_{\mu = -\infty}^{\infty} h[\mu] \cdot E \{ x[k+\kappa] \cdot x[k-\mu] \} \\
&= h[-\kappa] * \varphi_{xx}[\kappa]
\end{split}
\end{equation}

The CCF $\varphi_{xy}[\kappa]$ between in- and output is given as the time-reversed impulse response of the system convolved with the ACF of the input signal.

The CCF between out- and input is yielded by taking the symmetry relations of the CCF and ACF into account

\begin{equation}
\varphi_{yx}[\kappa] = \varphi_{xy}[-\kappa] = h[\kappa] * \varphi_{xx}[\kappa]
\end{equation}

The CCF $\varphi_{yx}[\kappa]$ between out- and input is given as the impulse response of the system convolved with the ACF of the input signal.

For a system which just attenuates the input signal $y[k] = A \cdot x[k]$, the CCFs between input and output are given as $\varphi_{xy}[\kappa] = A \cdot \varphi_{xx}[\kappa]$ and $\varphi_{yx}[\kappa] = A \cdot \varphi_{xx}[\kappa]$.

## System Identification by Cross-Correlation

The process of determining the impulse response or transfer function of a system is referred to as *system identification*. The CCFs of an LTI system play an important role in the estimation of the impulse response $h[k]$ of an unknown system. This is illustrated in the following.

The basic idea is to use a specific measurement signal as input signal to the system. Let's assume that the unknown LTI system is excited by [white noise](../random_signals/white_noise.ipynb). The ACF of the wide-sense stationary input signal $x[k]$ is then given as $\varphi_{xx}[\kappa] = N_0 \cdot \delta[\kappa]$. According to the relation derived above, the CCF between out- and input for this special choice of the input signal becomes

\begin{equation}
\varphi_{yx}[\kappa] = h[\kappa] * N_0 \cdot \delta[\kappa] = N_0 \cdot h[\kappa]
\end{equation}

For white noise as input signal $x[k]$, the impulse response of an LTI system can be estimated by estimating the CCF between its out- and input signals. Using noise as measurement signal instead of a Dirac impulse is beneficial since its [crest factor](https://en.wikipedia.org/wiki/Crest_factor) is limited.

### Example

The application of the CCF to the identification of a system is demonstrated. The system is excited by wide-sense ergodic normal distributed white noise with $N_0 = 1$. The ACF of the in- and output, as well as the CCF between out- and input is estimated and plotted.

```python
import scipy.signal as sig

N = 10000  # number of samples for input signal
K = 50  # limit for lags in ACF

# generate input signal
# normally distributed (zero-mean, unit-variance) white noise
np.random.seed(5)
x = np.random.normal(size=N)
# impulse response of the system
h = np.concatenate((np.zeros(10), sig.triang(10), np.zeros(10)))
# output signal by convolution
y = np.convolve(h, x, mode='full')

# compute correlation functions
acfx = 1/len(x) * np.correlate(x, x, mode='full')
acfy = 1/len(y) * np.correlate(y, y, mode='full')
ccfyx = 1/len(y) * np.correlate(y, x, mode='full')

def plot_correlation_function(cf):
    '''Plot correlation function.'''
    cf = cf[N-K-1:N+K-1]
    kappa = np.arange(-len(cf)//2, len(cf)//2)
    plt.stem(kappa, cf, use_line_collection=True)
    plt.xlabel(r'$\kappa$')
    plt.axis([-K, K, -0.2, 1.1*max(cf)])

# plot ACFs and CCF
plt.rc('figure', figsize=(10, 3))
plt.figure()
plot_correlation_function(acfx)
plt.title('Estimated ACF of input signal')
plt.ylabel(r'$\hat{\varphi}_{xx}[\kappa]$')

plt.figure()
plot_correlation_function(acfy)
plt.title('Estimated ACF of output signal')
plt.ylabel(r'$\hat{\varphi}_{yy}[\kappa]$')

plt.figure()
plot_correlation_function(ccfyx)
plt.plot(np.arange(len(h)), h, 'g-')
plt.title('Estimated and true impulse response')
plt.ylabel(r'$\hat{h}[k]$, $h[k]$');
```

**Exercise**

* Why is the estimated CCF $\hat{\varphi}_{yx}[k]$ not exactly equal to the true impulse response $h[k]$ of the system?
* What changes if you change the number of samples `N` of the input signal?

Solution: The derived relations for system identification hold for the case of a wide-sense ergodic input signal of infinite duration. Since we can only numerically simulate signals of finite duration, the observed deviations are a result of the resulting statistical uncertainties. Increasing the length `N` of the input signal improves the estimate of the impulse response.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples*.


$\newcommand{\vct}[1]{\boldsymbol{#1}}
\newcommand{\mtx}[1]{\mathbf{#1}}
\newcommand{\tr}{^\mathrm{T}}
\newcommand{\reals}{\mathbb{R}}
\newcommand{\lpa}{\left(}
\newcommand{\rpa}{\right)}
\newcommand{\lsb}{\left[}
\newcommand{\rsb}{\right]}
\newcommand{\lbr}{\left\lbrace}
\newcommand{\rbr}{\right\rbrace}
\newcommand{\fset}[1]{\lbr #1 \rbr}
\newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}}$

# Single layer models

In this lab we will implement a single-layer network model consisting of solely of an affine transformation of the inputs. The relevant material for this was covered in [the slides of the first lecture](http://www.inf.ed.ac.uk/teaching/courses/mlp/2016/mlp01-intro.pdf).

We will first implement the forward propagation of inputs to the network to produce predicted outputs. We will then move on to considering how to use gradients of a cost function evaluated on the outputs to compute the gradients with respect to the model parameters to allow us to perform an iterative gradient-descent training procedure.

## A general note on random number generators

It is generally a good practice (for machine learning applications **not** for cryptography!) to seed a pseudo-random number generator once at the beginning of each experiment. This makes it easier to reproduce results as the same random draws will produced each time the experiment is run (e.g. the same random initialisations used for parameters).

Therefore generally when we need to generate random values during this course, we will create a seeded random number generator object as follows:

```python
import matplotlib.pyplot as plt
import math
import sys
```

```python
import numpy as np

# Seed a random number generator to be used later
seed = 27092016
rng = np.random.RandomState(seed)
```

## Exercise 1: linear and affine transforms

Any *linear transform* (also called a linear map) of a finite-dimensional vector space can be parametrised by a matrix. So for example if we consider $\vct{x} \in \reals^{M}$ as the input space of a model with $M$ dimensional real-valued inputs, then a matrix $\mtx{W} \in \reals^{N\times M}$ can be used to define a prediction model consisting solely of a linear transform of the inputs

\begin{equation}
    \vct{y} = \mtx{W} \vct{x}
    \qquad
    \Leftrightarrow
    \qquad
    y_i = \sum_{j=1}^M \lpa W_{ij} x_j \rpa \qquad \forall i \in \fset{1 \dots N}
\end{equation}

with here $\vct{y} \in \reals^N$ the $N$-dimensional real-valued output of the model. Geometrically we can think of a linear transform doing some combination of rotation, scaling, reflection and shearing of the input.

An *affine transform* consists of a linear transform plus an additional translation parameterised by a vector $\vct{b} \in \reals^N$. A model consisting of an affine transformation of the inputs can then be defined as

\begin{equation}
    \vct{y} = \mtx{W}\vct{x} + \vct{b}
    \qquad
    \Leftrightarrow
    \qquad
    y_i = \sum_{j=1}^M \lpa W_{ij} x_j \rpa + b_i \qquad \forall i \in \fset{1 \dots N}
\end{equation}

In machine learning we will usually refer to the matrix $\mtx{W}$ as a *weight matrix* and the vector $\vct{b}$ as a *bias vector*.

Implement *forward propagation* for a single-layer model consisting of an affine transformation of the inputs. This should work for a batch of inputs of shape `(batch_size, input_dim)` producing a batch of outputs of shape `(batch_size, output_dim)`.

You may find the NumPy [`dot`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.dot.html) function and [broadcasting features](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html) useful.

```python
def fprop(inputs, weights, biases):
    """Forward propagates activations through the layer transformation.

    For inputs `x`, outputs `y`, weights `W` and biases `b` the layer
    corresponds to `y = W x + b`.

    Args:
        inputs: Array of layer inputs of shape (batch_size, input_dim).
        weights: Array of weight parameters of shape
            (output_dim, input_dim).
        biases: Array of bias parameters of shape (output_dim, ).

    Returns:
        outputs: Array of layer outputs of shape (batch_size, output_dim).
    """
    #return weights.dot(inputs.T).T + biases
    return inputs.dot(weights.T) + biases
```

Check your implementation by running the test cell below

```python
inputs = np.array([[0., -1., 2.], [-6., 3., 1.]])
weights = np.array([[2., -3., -1.], [-5., 7., 2.]])
biases = np.array([5., -3.])
true_outputs = np.array([[6., -6.], [-17., 50.]])

if not np.allclose(fprop(inputs, weights, biases), true_outputs):
    print('Wrong outputs computed.')
else:
    print('All outputs correct!')
```

    All outputs correct!

## Exercise 2: visualising random models

In this exercise you will use your `fprop` implementation to visualise the outputs of a single-layer affine transform model with two-dimensional inputs and a one-dimensional output. In this simple case we can visualise the joint input-output space on a 3D axis.

For this task and the learning experiments later in the notebook we will use a regression dataset from the [UCI machine learning repository](http://archive.ics.uci.edu/ml/index.html). In particular we will use a version of the [Combined Cycle Power Plant dataset](http://archive.ics.uci.edu/ml/datasets/Combined+Cycle+Power+Plant), where the task is to predict the energy output of a power plant given observations of the local ambient conditions (e.g. temperature, pressure and humidity).

The original dataset has four input dimensions and a single target output dimension. We have preprocessed the dataset by [whitening](https://en.wikipedia.org/wiki/Whitening_transformation) it, a common preprocessing step. We will only use the first two dimensions of the whitened inputs (corresponding to the first two principal components of the inputs) so we can easily visualise the joint input-output space.

The dataset has been wrapped in the `CCPPDataProvider` class in the `mlp.data_providers` module. Running the cell below will initialise an instance of this class, get a single batch of inputs and outputs and import the necessary `matplotlib` objects.

```python
#import mlp.data_providers_pligor as dps
from mpl_toolkits.mplot3d import Axes3D
from itertools import permutations, islice
from mlp.data_providers import CCPPDataProvider
%matplotlib notebook
```

```python
#dp = dps.CCPPDataProvider(
dp = CCPPDataProvider(
    which_set='train',
    input_dims=[0, 1],
    batch_size=5000,
    max_num_batches=1,
    shuffle_order=False
)

#print dp[0]

#inputs = None
#target = None
#for ins, targ in dp:
#    inputs = ins
#    target = targ
inputs, targets = dp.next()

#print target

def nth(iterable, n, default=None):
    "Returns the nth item or a default value"
    return next(islice(iterable, n, None), default)

#print nth(dp, 2)
#    print
x = inputs[:, 0]
y = inputs[:, 1]

X, Y = np.meshgrid(x, y)

#for x in permutations(range(2),2):    print x

#print next(islice(dp, 5, None))

#plt.clf()
#fig = plt.figure()
#ax = fig.gca(projection='3d')
#ax.plot_surface(X,Y,target)
#plt.show()
```

```python
input_dim, output_dim = 2, 1

print inputs.shape
print targets.shape
```

    (5000, 2)
    (5000, 1)

Now run the cell below to plot the predicted outputs of a randomly initialised model across the two dimensional input space as well as the true target outputs. This sort of visualisation can be a useful method (in low dimensions) to assess how well the model is likely to be able to fit the data and to judge appropriate initialisation scales for the parameters. Each time you re-run the cell a new set of random parameters will be sampled

Some questions to consider:

  * How do the weights and bias initialisation scale affect the sort of predicted input-output relationships?
  * Does the linear form of the model seem a good fit for the data here?

```python
weights_init_range = 0.5
biases_init_range = 0.1
```

```python
# Randomly initialise weights matrix
weights = rng.uniform(
    low=-weights_init_range,
    high=weights_init_range,
    size=(output_dim, input_dim)
)
weights
```

    array([[ 0.39135876, -0.1522873 ]])

```python
#  Randomly initialise biases vector
biases = rng.uniform(
    low=-biases_init_range,
    high=biases_init_range,
    size=output_dim
)
```

```python
# Calculate predicted model outputs
outputs = fprop(inputs = inputs, weights = weights, biases=biases)
```

```python
# Plot target and predicted outputs against inputs on same axis
fig = plt.figure(figsize=(8, 8))
ax = fig.add_subplot(111, projection='3d')
ax.plot(inputs[:, 0], inputs[:, 1], targets[:, 0], 'r.', ms=2)
ax.plot(inputs[:, 0], inputs[:, 1], outputs[:, 0], 'b.', ms=2)
ax.set_xlabel('Input dim 1')
ax.set_ylabel('Input dim 2')
ax.set_zlabel('Output')
ax.legend(['Targets', 'Predictions'], frameon=False)
fig.tight_layout()
```

    <IPython.core.display.Javascript object>

## Exercise 3: computing the SSE cost function and its gradient

\begin{equation}
    C = \frac{1}{2} \sum_{i=1}^N \lbr (y_i - t_i)^2 \rbr
\end{equation}

\begin{equation}
    \pd{C}{y_k} = y_k - t_k
\end{equation}

   * Implement sum squared error cost function and gradient with respect to outputs

```python
def cost(targets, outputPredictions):
    """SSE_costFunction"""
    #print targets.shape
    #print outputPredictions.shape
    sumByRow = np.sum((targets - outputPredictions)**2, axis=1)
    #consider the outputs equally important and take the mean
    return 0.5 * np.mean(sumByRow)

def cost2(targets, predictions):
    return 0.5 * np.sum((targets - predictions)**2) / len(targets)

outputs = np.array([[1., 2.], [-1., 0.], [6., -5.], [-1., 1.]])
targets = np.array([[0., 1.], [3., -2.], [7., -3.], [1., -2.]])

print cost(targets = targets, outputPredictions = outputs)
print cost2(targets = targets, predictions = outputs)

#print targets.shape
#print np.sum(targets, axis=1)

def cost_grad(y_real, y_predicted):
    return y_real - y_predicted

#print gradient.shape
a =np.array([[0, 5],
             [1, 3]])
b =np.array([[-3, 4],
             [2, 3]])

print getGradient(a,b)
```

    5.0
    5.0
    [[ 3  1]
     [-1  0]]

```python
def _cost(outputs, targets):
    """Calculates cost function given a batch of outputs and targets.

    Args:
        outputs: Array of model outputs of shape (batch_size, output_dim).
        targets: Array of target outputs of shape (batch_size, output_dim).

    Returns:
        Scalar cost function value.
    """

    #return 0.5 * np.mean(np.sum((outputs - targets)**2, axis=1))
    #Why not make it like this
    return 0.5 * np.sum((targets - predictions)**2) / len(targets)

def _cost_grad(outputs, targets):
    """Calculates gradient of cost function with respect to outputs.

    Args:
        outputs: Array of model outputs of shape (batch_size, output_dim).
        targets: Array of target outputs of shape (batch_size, output_dim).

    Returns:
        Gradient of cost function with respect to outputs.
    """
    return outputs - targets
```

Check your implementation by running the test cell below

```python
outputs = np.array([[1., 2.], [-1., 0.], [6., -5.], [-1., 1.]])
targets = np.array([[0., 1.], [3., -2.], [7., -3.], [1., -2.]])
true_cost = 5.
true_cost_grad = np.array([[1., 1.], [-4., 2.], [-1., -2.], [-2., 3.]])

if not cost(outputs, targets) == true_cost:
    print('Cost calculated incorrectly.')
elif not np.allclose(cost_grad(outputs, targets), true_cost_grad):
    print('Cost gradient calculated incorrectly.')
else:
    print('Cost function and gradient computed correctly!')
```

    Cost function and gradient computed correctly!

## Exercise 4: computing gradients with respect to the parameters

\begin{equation}
    \pd{C}{W_{ij}} = \sum_{k=1}^N \lbr \pd{C}{y_k} \pd{y_k}{W_{ij}} \rbr
    \qquad
    \pd{y_k}{W_{ij}} = \delta_{ik} x_j
\end{equation}

\begin{equation}
    \pd{C}{b_{i}} = \sum_{k=1}^N \lbr \pd{C}{y_k} \pd{y_k}{b_{i}} \rbr
    \qquad
    \pd{y_k}{b_i} = \delta_{ik}
\end{equation}

   * Implement function to calculate gradient with respect to weight and bias parameters given gradient with respect to outputs

```python
inputs = np.array([[1., 2., 3.], [-1., 4., -9.]])
grads_wrt_outputs = np.array([[-1., 1.], [2., -3.]])

true_grads_wrt_weights = np.array([[-3., 6., -21.], [4., -10., 30.]])
true_grads_wrt_biases = np.array([1., -2.])

def grads_wrt_params_analytically(inputs, grads_wrt_outputs):
    #gradient of the cost in respect to the output C/yk
    #delta means one above at the theory
    #we have to get the C/yk which for all k is the j-th column of grads_wrt_outputs
    #we have to get the yk/Wij which is equal to xj which is the j-th column of inputs
    #so we multiply element-wise the two columns and we sum them up
    #We are doing it analytically here to be more explicit

    result = np.zeros(inputs.shape)

    for col in range(inputs.shape[1]):
        for row in range(inputs.shape[0]):
            result[row, col] = np.sum(inputs[:, col] * grads_wrt_outputs[:, row])

    return result

print grads_wrt_params(inputs, grads_wrt_outputs)
```

    [array([[ -3.,   6., -21.],
           [  4., -10.,  30.]]), array([ 1., -2.])]

```python
def grads_wrt_params(inputs, grads_wrt_outputs):
    """Calculates gradients with respect to model parameters.

    Args:
        inputs: array of inputs to model of shape (batch_size, input_dim)
        grads_wrt_to_outputs: array of gradients with respect to the model
            outputs of shape (batch_size, output_dim)

    Returns:
        list of arrays of gradients with respect to the model parameters
        `[grads_wrt_weights, grads_wrt_biases]`.
    """
    grads_wrt_weights = np.dot(grads_wrt_outputs.T, inputs)
    grads_wrt_biases = np.sum(grads_wrt_outputs, axis=0)
    return [grads_wrt_weights, grads_wrt_biases]
```

Check your implementation by running the test cell below

```python
inputs = np.array([[1., 2., 3.], [-1., 4., -9.]])
grads_wrt_outputs = np.array([[-1., 1.], [2., -3.]])
true_grads_wrt_weights = np.array([[-3., 6., -21.], [4., -10., 30.]])
true_grads_wrt_biases = np.array([1., -2.])

grads_wrt_weights, grads_wrt_biases = grads_wrt_params(
    inputs, grads_wrt_outputs)

if not np.allclose(true_grads_wrt_weights, grads_wrt_weights):
    print('Gradients with respect to weights incorrect.')
elif not np.allclose(true_grads_wrt_biases, grads_wrt_biases):
    print('Gradients with respect to biases incorrect.')
else:
    print('All parameter gradients calculated correctly!')
```

    All parameter gradients calculated correctly!

## Exercise 5: wrapping the functions into reusable components

In the previous exercises you implemented methods to compute the predicted outputs of our model, evaluate the cost function and its gradient on the outputs and finally to calculate the gradients of the cost with respect to the model parameters. Together they constitute all the basic ingredients we need to implement a gradient-descent based iterative learning procedure for the model.

Although you could implement training code which directly uses the functions you defined, this would only be usable for this particular model architecture. In subsequent labs we will want to use the affine transform functions as the basis for more interesting multi-layer models. We will therefore wrap the implementations you just wrote in to reusable components that we can build more complex models with later in the course.

  * In the `mlp.layers` module, use your implementations of `fprop` and `grad_wrt_params` above to implement the corresponding methods in the skeleton `AffineLayer` class provided.
  * In the `mlp.costs` module use your implementation of `cost` and `cost_grad` to implement the `__call__` and `grad` methods respectively of the skeleton `MeanSquaredErrorCost` class provided. Note `__call__` is a special Python method that allows an object to be used with a function call syntax.

```python

```

Run the cell below to use your completed `AffineLayer` and `MeanSquaredErrorCost` implementations to train a single-layer model using gradient descent on the CCCP dataset.

```python
from mlp.layers import AffineLayer
from mlp.costs import MeanSquaredErrorCost
from mlp.models import SingleLayerModel
from mlp.initialisers import UniformInit, ConstantInit
from mlp.learning_rules import GradientDescentLearningRule
from mlp.optimisers import Optimiser
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.addHandler(logging.StreamHandler())

train_data = CCPPDataProvider('train', [0, 1], batch_size=100)
valid_data = CCPPDataProvider('train', [0, 1], batch_size=100)
input_dim, output_dim = 2, 1

layer = AffineLayer(input_dim, output_dim, UniformInit(-0.1, 0.1, rng=rng), ConstantInit(0))
model = SingleLayerModel(layer)
cost = MeanSquaredErrorCost()
learning_rule = GradientDescentLearningRule(learning_rate=1e-3)
optimiser = Optimiser(model, cost, learning_rule, train_data, valid_data)
stats = optimiser.train(5, 1)
```

    Epoch 1: 0.0s to complete
        cost(train)=7.66e-02, cost(valid)=7.66e-02, cost(param)=0.00e+00
    Epoch 1: 0.0s to complete
        cost(train)=7.66e-02, cost(valid)=7.66e-02, cost(param)=0.00e+00
    Epoch 2: 0.0s to complete
        cost(train)=7.67e-02, cost(valid)=7.66e-02, cost(param)=0.00e+00
    Epoch 2: 0.0s to complete
        cost(train)=7.67e-02, cost(valid)=7.66e-02, cost(param)=0.00e+00
    Epoch 3: 0.0s to complete
        cost(train)=7.66e-02, cost(valid)=7.66e-02, cost(param)=0.00e+00
    Epoch 3: 0.0s to complete
        cost(train)=7.66e-02, cost(valid)=7.66e-02, cost(param)=0.00e+00
    Epoch 4: 0.0s to complete
        cost(train)=7.65e-02, cost(valid)=7.67e-02, cost(param)=0.00e+00
    Epoch 4: 0.0s to complete
        cost(train)=7.65e-02, cost(valid)=7.67e-02, cost(param)=0.00e+00
    Epoch 5: 0.0s to complete
        cost(train)=7.67e-02, cost(valid)=7.68e-02, cost(param)=0.00e+00
    Epoch 5: 0.0s to complete
        cost(train)=7.67e-02, cost(valid)=7.68e-02, cost(param)=0.00e+00

Using similar code to previously we can now visualise the joint input-output space for the trained model. If you implemented the required methods correctly you should now see a much improved fit between predicted and target outputs when running the cell below.

```python
data_provider = CCPPDataProvider(
    which_set='train',
    input_dims=[0, 1],
    batch_size=5000,
    max_num_batches=1,
    shuffle_order=False
)

inputs, targets = data_provider.next()

# Calculate predicted model outputs
outputs = model.fprop(inputs)[-1]

# Plot target and predicted outputs against inputs on same axis
fig = plt.figure(figsize=(8, 8))
ax = fig.add_subplot(111, projection='3d')
ax.plot(inputs[:, 0], inputs[:, 1], targets[:, 0], 'r.', ms=2)
ax.plot(inputs[:, 0], inputs[:, 1], outputs[:, 0], 'b.', ms=2)
ax.set_xlabel('Input dim 1')
ax.set_ylabel('Input dim 2')
ax.set_zlabel('Output')
ax.legend(['Targets', 'Predictions'], frameon=False)
fig.tight_layout()
```

    <IPython.core.display.Javascript object>

## Optional exercise: visualising training trajectories in parameter space

Running the cell below will display an interactive widget which plots the trajectories of gradient-based training of the single-layer affine model on the CCPP dataset in the three dimensional parameter space (two weights plus bias) from random initialisations. Also shown on the right is a plot of the evolution of the cost function (evaluated on the current batch) over training. By moving the sliders you can alter the training hyperparameters to investigate the effect they have on how training procedes.

Some questions to explore:

  * Are there multiple local minima in parameter space here? Why?
  * What happens to learning for very small learning rates? And very large learning rates?
  * How does the batch size affect learning?

```python
from ipywidgets import interact
%matplotlib inline

def setup_figure():
    # create figure and axes
    fig = plt.figure(figsize=(12, 6))
    ax1 = fig.add_axes([0., 0., 0.5, 1.], projection='3d')
    ax2 = fig.add_axes([0.6, 0.1, 0.4, 0.8])
    # set axes properties
    ax2.spines['right'].set_visible(False)
    ax2.spines['top'].set_visible(False)
    ax2.yaxis.set_ticks_position('left')
    ax2.xaxis.set_ticks_position('bottom')
    ax2.set_yscale('log')
    ax1.set_xlim((-2, 2))
    ax1.set_ylim((-2, 2))
    ax1.set_zlim((-2, 2))
    #set axes labels and title
    ax1.set_title('Parameter trajectories over training')
    ax1.set_xlabel('Weight 1')
    ax1.set_ylabel('Weight 2')
    ax1.set_zlabel('Bias')
    ax2.set_title('Batch costs over training')
    ax2.set_xlabel('Batch update number')
    ax2.set_ylabel('Batch cost')
    return fig, ax1, ax2

def visualise_training(n_epochs=1, batch_size=200, log_lr=-3.5, n_inits=5,
                       w_scale=1., b_scale=1., elev=30., azim=0.):
    fig, ax1, ax2 = setup_figure()
    # create seeded random number generator
    rng = np.random.RandomState(1234)
    # create data provider
    data_provider = CCPPDataProvider(
        input_dims=[0, 1],
        batch_size=batch_size,
        shuffle_order=False,
    )
    learning_rate = 10 ** log_lr
    n_batches = data_provider.num_batches
    weights_traj = np.empty((n_inits, n_epochs * n_batches + 1, 1, 2))
    biases_traj = np.empty((n_inits, n_epochs * n_batches + 1, 1))
    costs_traj = np.empty((n_inits, n_epochs * n_batches))
    # randomly initialise parameters
    weights = rng.uniform(-w_scale, w_scale, (n_inits, 1, 2))
    biases = rng.uniform(-b_scale, b_scale, (n_inits, 1))
    # store initial parameters
    weights_traj[:, 0] = weights
    biases_traj[:, 0] = biases
    # iterate across different initialisations
    for i in range(n_inits):
        # iterate across epochs
        for e in range(n_epochs):
            # iterate across batches
            for b, (inputs, targets) in enumerate(data_provider):
                outputs = fprop(inputs, weights[i], biases[i])
                costs_traj[i, e * n_batches + b] = cost(outputs, targets)
                grad_wrt_outputs = cost_grad(outputs, targets)
                weights_grad, biases_grad = grads_wrt_params(inputs, grad_wrt_outputs)
                weights[i] -= learning_rate * weights_grad
                biases[i] -= learning_rate * biases_grad
                weights_traj[i, e * n_batches + b + 1] = weights[i]
                biases_traj[i, e * n_batches + b + 1] = biases[i]
    # choose a different color for each trajectory
    colors = plt.cm.jet(np.linspace(0, 1, n_inits))
    # plot all trajectories
    for i in range(n_inits):
        lines_1 = ax1.plot(
            weights_traj[i, :, 0, 0],
            weights_traj[i, :, 0, 1],
            biases_traj[i, :, 0],
            '-', c=colors[i], lw=2)
        lines_2 = ax2.plot(
            np.arange(n_batches * n_epochs),
            costs_traj[i],
            c=colors[i]
        )
    ax1.view_init(elev, azim)
    plt.show()

w = interact(
    visualise_training,
    elev=(-90, 90, 2),
    azim=(-180, 180, 2),
    n_epochs=(1, 5),
    batch_size=(100, 1000, 100),
    log_lr=(-5., -2.),
    w_scale=(0., 2.),
    b_scale=(0., 2.),
    n_inits=(1, 10)
)

for child in w.widget.children:
    child.layout.width = '100%'
```

Running the cell below will display an interactive widget which plots the trajectories of gradient-based training of the single-layer affine model on the CCPP dataset in the three dimensional parameter space (two weights plus bias) from random initialisations. Also shown on the right is a plot of the evolution of the cost function (evaluated on the current batch) over training. By moving the sliders you can alter the training hyperparameters to investigate the effect they have on how training procedes.

Some questions to explore:

  * Are there multiple local minima in parameter space here? Why?
  * What happens to learning for very small learning rates? And very large learning rates?
  * How does the batch size affect learning?

My answers:

Yes indeed there are multiple local minima in parameter space. We see that for different initializations the final vectors of weight1, weight2 and bias have different directions and amplitudes. If there was only one minima these vectors would be very similar

Actually the step size of the (stochastic) gradient descent algorithm is called learning rate. Source: https://en.wikipedia.org/wiki/Stochastic_gradient_descent
So for really low learning rate it take too long to converge.
For medium learning rate the convergence occurs much faster
At some point around -2.70 of learning rate the converge is not the optimal if the initialization is not correct and even if we got a "lucky"(good) initialization then algorithm cannot converge to the actual local minima and keeps going back and forth.
Now if we increase the step even more to -2.60 or more then the error rate is increasing constantly the step is so large that the algorithm cannot converge

By increasing-decreasing the batch size we see that with small batch size the cost is more of high frequency more noisy since we do not have so much data, while with high batch size then the cost is smoothed since we are always better of getting an averaged-error

```python

```


```python
import sympy as sym
%matplotlib inline
sym.init_printing()

import matplotlib.pyplot as plt
from sympy.plotting.plot import MatplotlibBackend

def logx_plot(Hw, wrange, **kwargs):
    'create plot with logarithmic x-axis'
    p = sym.plot(Hw, wrange, show=False, **kwargs)
    backend = MatplotlibBackend(p)
    backend.process_series()
    backend.ax[0].spines['left'].set_position(('axes', 0))
    backend.ax[0].spines['bottom'].set_position(('axes', 0))
    plt.xscale('log')
    plt.grid(which='both')

def db(x):
    'compute dB value'
    return 20 * sym.log(sym.Abs(x), 10)
```

# Caracterización de Sistemas en Dominio de la Frecuencia

## Función de Transferencia

Hay distintas representaciones para modelar sistemas LTI. Algunas de estas son:
- Los coeficientes constantes de las Ecuaciones Diferenciales(ODEs).
- La [respuesta impulsional](../systems_time_domain/impulse_response.ipynb).
- La [respuesta al escalón](../systems_time_domain/step_response.ipynb).

La [Función de transferencia](../systems_time_domain/eigenfunctions.ipynb#Transfer-Function) usa la forma $e^{s t}$ como eigenfunción para establecer combinaciones de $e^{s t}$ al ser excitado un sistema LTI. Se usa la transformada de Laplace para esta representación.

### Respuesta al Impulso y Señales de Entrada y Salida

La operación de convolución en el dominio temporal tiene equivalencia en el dominio de la frecuencia.

En el dominio temporal, la salida $y(t)$ de un sistema LTI está dada por la convolución entre la entrada $x(t)$ y la [respuesta al impulso](../systems_time_domain/impulse_response.ipynb) $h(t)$.

\begin{equation}
y(t) = x(t) * h(t)
\end{equation}

En el dominio transformado se tiene

\begin{equation}
Y(s) = X(s) \cdot H(s)
\end{equation}

donde $X(s) = \mathcal{L} \{ x(t) \}$ y $Y(s) = \mathcal{L} \{ y(t) \}$ son las transformadas de Laplace de las señales de entrada y salida.

La función de transferencia $H(s)$ está dada por la transformada de Laplace de la respuesta al impulso $h(t)$

\begin{equation}
H(s) = \mathcal{L} \{ h(t) \}
\end{equation}

Así, un sistema LTI queda caracterizado mediante la respuesta impulsional en el dominio temporal y en el dominio frecuencial.

Note también que puede encontrarse la función de transferencia como el cociente entre las señales de salida $Y(s)$ y  $X(s)$ en el dominio de Laplace.

\begin{equation}
H(s) = \frac{Y(s)}{X(s)}
\end{equation}

O como el cociente entre las señales de salida $Y(\omega)$ y  $X(\omega)$ en el dominio de Fourier.

\begin{equation}
H(\omega) = \frac{Y(\omega)}{X(\omega)}
\end{equation}

En cuanto a señales:

- La transformada de Laplace permite analizar formas transitorias y estáticas de las señales.
- La transformada de Fourier permite analizar solamente formas estáticas de las señales oscilatorias, composición espectral.

En cuanto a sistemas:
- La transformada de Laplace permite analizar comportamientos transitorios y estáticos de los sistemas ante cualquier señal de entrada.
- La transformada de Fourier permite analizar solamente la respuesta de los sistemas anre excitaciones oscilatorias, respuesta frecuencial.

Escoger una u otra transformada depende de la aplicación.

Puede obtener la respuesta frecuencial de un sistema modelado como función de transferencia de Laplace con la equivalencia $s = j\omega$.

### Función de Transferencia y Ecuaciones Diferenciales

Suponga una ecuación diferencial con coeficientes constantes $\alpha_n$ y $\beta_m$.

\begin{equation}
\alpha_N \frac{d^N}{d t^N} y(t) + \alpha_{N-1} \frac{d^{N-1}}{d t^{N-1}} y(t) + ... + \alpha_{2} \frac{d^{2}}{d t^{2}} y(t) + \alpha_{1} \frac{d}{d t} y(t) + \alpha_{0} y(t) = \beta_M \frac{d^M}{d t^M} x(t) + \beta_{M-1} \frac{d^{M-1}}{d t^{M-1}} x(t) + ... + \beta_{2} \frac{d^{2}}{d t^{2}} x(t) + \beta_{1} \frac{d}{d t} x(t) + \beta_{0} x(t)
\end{equation}

\begin{equation}
\sum_{n=0}^{N} \alpha_n \frac{d^n}{d t^n} y(t) = \sum_{m=0}^{M} \beta_m \frac{d^m}{d t^m} x(t)
\end{equation}

donde $\alpha_N \neq 0$ y $N$ es el orden del modelo.

Suponga también que los valores iniciales son iguales a cero.

Al aplicar la Transformada de Laplace a la ecuación se tiene:

\begin{equation}
Y(s) \sum_{n=0}^{N} \alpha_n s^n = X(s) \sum_{m=0}^{M} \beta_m s^m
\end{equation}

De donde se obtiene la función de transferencia como:

\begin{equation}
H(s) = \frac{Y(s)}{X(s)} = \frac{\sum_{m=0}^{M} \beta_m s^m}{\sum_{n=0}^{N} \alpha_n s^n}
\end{equation}

Observe que la forma es **polinomio** sobre **polinomio**.

**Ejemplo**

Los coeficientes del filtro RLC analizado antes se pueden obtener de Ecuación Diferencial como:

\begin{equation}
C L \frac{d^2 y(t)}{dt^2} + C R \frac{d y(t)}{dt} + y(t) = x(t)
\end{equation}

\begin{align}
\beta_0 &= 1 & \alpha_0 &= 1 \\
& & \alpha_1 &= C R \\
& & \alpha_2 &= C L
\end{align}

A partir de estos coeficientes se define la función de transferencia $H(s)$ en `SymPy`.

```python

s = sym.symbols('s', complex=True)
w = sym.symbols('omega', real=True)

R = 1
L = 0.5
C = 0.4

beta = 1
alpha = L*C, R*C, 1

Num_s = sym.Poly(beta, s)
Den_s = sym.Poly(alpha, s)

```

```python
H = Num_s / Den_s
H
```

```python
Hw = H.subs(s,w*1j)
Hw
```

Varios programas definen la función de transferencia únicamente a partir de los coeficientes $\alpha$ y $\beta$.Este es el caso del Toolbox [`scipy.signal`](http://docs.scipy.org/doc/scipy/reference/signal.html) para python donde se puede evaluar numéricamente $H(j \omega)$ en un conjunto de valores de frecuencia angular $\omega_i$ a partir de los coeficientes del modelo.

```python
from scipy import signal

R = 1
L = 0.5
C = 0.4
beta = 1
alpha = L*C, R*C, 1

w, H = signal.freqresp((beta, alpha))

import matplotlib.pyplot as plt

plt.plot(w, abs(H))
plt.xlabel('$\omega$ in $s^{-1}$')
plt.ylabel('$|H(j \omega)|$')
plt.grid();
```

## Gráficas de Bode

La función de transferencia en el dominio de Fourier caracteriza la transmisión de un sistema LTI con respecto a una señal armónica $e^{j \omega t}$ con frecuencia angular $\omega = 2 \pi f$.

A menudo se trabajan las funciones magnitud $| H(j \omega) |$ y fase $\varphi_H(j \omega)$ de forma separada.

\begin{align}
| Y(j \omega) | &= | X(j \omega) | \cdot | H(j \omega) | \\
\varphi_Y(j \omega) &= \varphi_X(j \omega) + \varphi_H(j \omega)
\end{align}

donde $X(j \omega)$ es la señal de entrada y $| X(j \omega) |$ and $\varphi_X(j \omega)$ su magnitud y fase, respectivamente.

Observe que $| H(j \omega) |$ muestra la atenuación o amplificación en función de la frecuencia $| X(j \omega) |$ de la señal de entrada. Por otra parte, $\varphi_H(j \omega)$ muestra la fase que se introduce en el sistema.

Las [gráficas de Bode](https://en.wikipedia.org/wiki/Bode_plot) se usan para representar gráficamente las propiedades del sistema en términos de magnitud y fase en función de la frecuencia. Se trata de una gráfica logarítmica de la magnitud de la función de transferencia en [decibeles](https://en.wikipedia.org/wiki/Decibel)  $20 \log_{10} | H(j \omega) |$ contra el logaritmo de la frecuencia $\omega$ o $f$.

La fase $\varphi_H(j \omega)$ se grafica linealmente contra el logaritmo de la frecuencia.

Como $H(s)$ es racional respecto a $s$, se puede representar en términos de polos y ceros.

\begin{equation}
H(s) = K \cdot \frac{\prod_{\mu=1}^{M} (s - s_{0 \mu})}{\prod_{\nu=1}^{N} (s - s_{\infty \nu})}
\end{equation}

considerando que $s=j \omega$
\begin{equation}
H(j \omega) = K \cdot \frac{\prod_{\mu=1}^{M} (j \omega - s_{0 \mu})}{\prod_{\nu=1}^{N} (j \omega - s_{\infty \nu})}
\end{equation}

donde $s_{0 \mu}$ y $s_{\infty \nu}$ son el $\mu$-ecimo cero and $\nu$-ecimo polo de $H(s)$. El logaritmo de la magnitud y la fase son:

\begin{align}
\log_{10} | H(j \omega) | &= \sum_{\mu=0}^{M} \log_{10} |j \omega - s_{0 \mu}| - \sum_{\nu=0}^{N} \log_{10} |j \omega - s_{\infty \nu}| + \log_{10} |K|   \\
\varphi_H(j \omega) &= \sum_{\mu=0}^{M} \arg (j \omega - s_{0 \mu}) - \sum_{\nu=0}^{N} \arg (j \omega - s_{\infty \nu})
\end{align}

donde $\arg(\cdot)$ es el [argumento](https://en.wikipedia.org/wiki/Argument_%28complex_analysis%29) (fase) de una función compleja.

Los resultados anteriores permiten aproximar las gráficas de Bode a mano. Se recomienda revisar la [literatura](https://en.wikipedia.org/wiki/Bode_plot#Rules_for_handmade_Bode_plot).

**Ejemplo**

La gráfica de Bode del filtro anterior

La función de transferencia $H(j \omega)$ se obtiene a partir de $H(s)$ como

\begin{equation}
H(j \omega) = \frac{1}{C L s^2 + C R s + 1} \bigg\rvert_{s = j \omega}
\end{equation}

```python
s = sym.symbols('s')
w = sym.symbols('omega', real=True)

R = 1
L = 0.5
C = 0.4

Hs = 1/(C*L*s**2 + C*R*s + 1)
H = Hs.subs(s, sym.I * w)
H

```

```python
plt.rcParams['figure.figsize'] = 6, 3

logx_plot(db(H), (w, 0.1, 10),
          xlabel='$\omega$', ylabel='$20 \log_{10} | H(j \omega) |$ in dB')
logx_plot(sym.arg(H), (w, 0.1, 1000),
          xlabel='$\omega$', ylabel=r'$\varphi(j \omega)$')
```


```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('ggplot')
```

# Midterm Study Guide

So far in this class, we have worked with the following topics.

* Rootfinding
  * Midpoint Methods
  * Newton and Secant Methods
  * Convergence classes of iterative methods (q-linear and q-quadratic)
* Conditioning of problems
* Stability of algorithms
* Numerical Linear Algebra
  * Thinking abstractly about vectors
  * Inner products and norms
  * Projectors $P = I - v v^T$ where $\lVert v \rVert = 1$
  * Reflectors $F = I - 2 v v^T$ where $\lVert v \rVert = 1$
  * Gram-Schmidt algorithm for QR factorization
  * Householder algorithm for QR factorization
  * Leading order cost of algorithms (like big-$O$, but we quantify the leading coefficient)
  * Solution of linear equations and linear least squares problems

I recommend reviewing the class notebooks, in-class activities, and homeworks.  Below, I'll give some discussion of the sort of questions that I could ask.

## Rootfinding

All rootfinding methods attempt to find $x$ such that $f(x) = 0$, but they require different information about $f$ and/or where to look.
If we know $f'(x) < 0$, which methods can be used?  What if we also know that $f(a) > 0 > f(b)$?  Does any method guarantee that we find a unique solution?

```python
def iterate(f, x0, n):
    """Apply n iterations of x_{i+1} = f(x_i),
    return resulting history as an array.
    """
    hist = [x0]
    for i in range(n):
        hist.append(f(hist[i]))
    return np.array(hist)

def f1(x):
    return np.cos(x)

hist1 = iterate(f1, 1, 80)
x_star = hist1[-1] # Suppose the last estimate is best
plt.semilogy(np.abs(hist1[:30] - x_star), 'o');
plt.semilogy(np.sin(x_star)**np.arange(30));
```

```python
np.sin(x_star)
```

    0.6736120291832185

What convergence class is the above figure evidence of?

```python
def f2(x):
    return x - (np.cos(x) - x) / (-np.sin(x) - 1)

hist2 = iterate(f2, 1, 15)
plt.semilogy(np.abs(hist2 - x_star), 'o');
```

What convergence class is the above plot evidence of?

What happens after the fourth iteration?

Can you sketch a function such that Newton exhibits a certain behavior, such as error decreasing while residual increases?

## Conditioning

You should be able to estimate (relative) condition numbers (a finite number or $\infty$) for differentiable functions such as $\sin x$ at a given value of $x$.  Recall that for differentiable functions, the relative condition number is
$$ \kappa = \lvert f'(x) \rvert \frac{\lvert x \rvert}{\lvert f(x) \rvert} . $$
You should be able to estimate condition number for a $2\times 2$ diagonal matrix, for example.

## Stability

You should be able to recognize unstable algorithms, such as in the activity where we computed $\exp(-20)$ by directly evaluating the Taylor series.

```python
def g1(theta):
    return np.array([np.cos(theta),
                     np.sin(theta)])

def g2(theta):
    return np.array([np.cos(theta),
                     np.cos(theta - np.pi/2)])

theta = np.linspace(1, 1.3)
plt.plot(*g1(theta), '.')
plt.plot(*g2(theta), '-.')
plt.plot(*g1(theta + np.pi*2e14), 'o')
plt.plot(*g2(theta + np.pi*2e14), 's');
```

Can you explain the plot above?  Is there evidence that one of these functions (`g1` or `g2`) is stable or backward stable?

## Numerical Linear Algebra

If $u$ and $v$ are vectors and I propose a function $h(u, v)$, can you determine whether it is an inner product?

There are two Moodle quizes about projections and matrix transformations.  If I ask a similar question, can you understand the terminology and show whether a property is true?

Is an orthogonal projector an orthogonal matrix?

If $Q$ has orthonormal columns, is $Q^T Q = Q Q^T$?  What if $Q$ is square?

```python
def factor(A):
    A = A.copy()
    n = len(A)
    for i in range(n):
        ai = 1/A[i, i]
        A[i+1:, i] *= ai
        A[i+1:, i+1:] -= np.outer(A[i+1:, i], A[i, i+1:])
    return np.tril(A, -1) + np.eye(n), np.triu(A)

A = np.random.randn(4, 4)
L, U = factor(A)
np.linalg.norm(L @ U - A)
```

    3.3422138886441676e-16

Which line in the above function is most expensive?

How much does it cost on the first iteration?

Can you write a sum of the cost over all loop iterations?

Can you write a $2\times 2$ matrix such that the above code fails?

Given a collection of vectors, how can you use QR factorization to build an orthogonal basis that spans the same space and where all but one vector has mean zero?

Given a matrix $A$, can you devise an elementary reflector $F = I - 2 v v^T$ such that the first column of the product $F A$ is a constant vector (all entries the same)?

### Sample question: QR

Solving a linear system using QR factorization requires solving $R x = y$ (where $R$ is upper triangular).
The "back substitution" algorithm starts with the last row (a scalar equation) and works up, solving for one additional entry in $x$ at a time.
The algorithm can be implemented in Python as
```python
  def R_solve(R, b):
      # Solve Rx = b using back substitution.
      x = b.copy()
      n = len(b)
      for i in reversed(range(n)): # from n-1 down to 0
          x[i] -= R[i,i+1:] @ x[i+1:] # dot product of length n-i-1
          x[i] /= R[i,i]
      return x
```
1. Count the number of flops performed by this algorithm and report the dominant term as a function of $n$ when $n$ is large.
      (This is like big-$O$ notation, but we keep track of multiplicative constants.  Count all elementary operations together.)
2. QR factorization of an $m\times n$ matrix using Gram-Schmidt requires $2 m n^2$ operations.  Multiplying $Q^T$ with a vector requires $2 m n$ operations.
To solve a linear (or least squares) problem $A x = b$ using QR, we compute $y = Q^T b$ and then solve $R x = y$.
Of course we can reuse the QR factorization for many right hand sides $b$.
How many right hand sides $k$ must we solve before the cost of solving all $k$ right hand sides equals the cost of QR factorization?

### Sample question: Newton
1. Sketch a figure of a strictly monotonic, differentiable function $f(x)$ with one root $x_* = 0$ and an initial guess $x_0$ for which one step of Newton's method (without line search) computes an $x_1$ with larger error and larger residual.  In other words,
$$ \lvert x_1 - x_* \rvert > \lvert x_0 - x_* \rvert \quad \text{and} \quad \lvert f(x_1)\rvert > \lvert f(x_0) \rvert .$$
    Mark the points $(x_0, f(x_0))$ and $(x_1, f(x_1))$ on the graph.
2. Same question, but choose the function and initial guess so that the error increases while the residual decreases:
    $$ \lvert x_1 - x_* \rvert > \lvert x_0 - x_* \rvert \quad \text{and} \quad \lvert f(x_1)\rvert < \lvert f(x_0) \rvert .$$
3. Same question, but choose the function and initial guess so that the error decreases while the residual increases:
    $$ \lvert x_1 - x_* \rvert < \lvert x_0 - x_* \rvert \quad \text{and} \quad \lvert f(x_1)\rvert > \lvert f(x_0) \rvert .$$

### Sample question: Convergence

We learned that a fixed point iteration $x_{i+1} = g(x_i)$ converges with rate $\lvert g'(x) \rvert$ for $x$ near a fixed point $x_*$.
1. Consider $g(x) = \cos x$ and estimate the q-linear convergence factor $\rho$ such that
      $$ \lim_{i \to \infty} \frac{\lvert x_{i+1} \rvert}{\lvert x_i \rvert} = \rho . $$
2. Write a function $h(x)$ such that $x_{i+1} = h(x_i)$ converges to the same $x_*$ at a faster rate (smaller $\rho$).

### Sample question: Linear algebra
An $m \times n$ matrix $A$, $m > n$, is said to have a pseudoinverse $G$ if the following conditions hold
    \begin{align}
      AGA &= A \\
      GAG &= G \\
      (AG)^T &= AG \\
      (GA)^T &= GA
    \end{align}

1. What is the shape of the matrix $G$?
2. If $A$ is full rank, we can compute its $QR$ factorization.  Which of the above conditions hold for $G = R^{-1} Q^T$?
3. Let $y = G b$.  Show that the residual $A y - b$ is orthogonal to the range of $Q$.  (Hint: Every point in the range of $Q$ can be represented as $Q x$ for some $x$. Since the range of $Q$ is equal to the range of $A$, this also proves that $x = y$ minimizes $\lVert A x - b \rVert$.)

### Sample question: Conditioning
The condition number of a function or algorithm is the relative sensitivity of the output to the input:
    $$\kappa = \max_{\delta x} \frac{|\delta f|/|f|}{|\delta x|/|x|}$$
where $\delta f = f(x + \delta x)$.  If $f$ is differentiable, we can take the limit of $\delta x \to 0$ and write as
    $$\kappa = |f'(x)| \frac{|x|}{|f|}.$$
Estimate the condition number of the following functions:
1. `log(x)` for $x = 1 + 10^{-10}$
2. `log1p(x)` for $x = 10^{-10}$.  (`log1p` is a function that accurately evaluates $\log(1+x)$, i.e., without computing $1+x$.)
3. `expm1(x)` for $x = 10^{-10}$.  (`expm1` is a function that accurately evaluates $e^x - 1$.)
4. `exp(x)` for $x = 10^{-10}$.


```python
import numpy as np
import sympy as sy
```

```python
x, y, z, k = sy.symbols('x y z k', commutative=False)
A = sy.Function('A')(x)
Psi = sy.Function('Psi')(x)

expr = k*A*k*Psi
expr2 = k*A*k*Psi + k*Psi
```

```python
def discretize(expr):
    expr = sy.expand(expr)

    if expr.func == sy.Add:
        terms = []
        for term in expr.args:
            terms.append(discretize(term))
        res = sy.Add(*tuple(terms))
    else:
        # First find a k (in terms of an index for the expr.args list)
        index = 'None'
        for i in range(len(expr.args)):
            if expr.args[i] == k:
                index = i

        # Only change anything if there was at least one k in the expression.
        if not index == 'None':
            #Rafal wants three terms: left side, right side, derivative operators!
            # Take everything behind the k
            temp = sy.Mul(*expr.args[index+1:])
            # apply the derivative operator
            temp = (temp.subs(x, x+1) - temp.subs(x, x-1))/2
            # multiply by whatever was in front of the derivative operator
            res = sy.Mul(sy.Mul(*expr.args[:index]), temp)
            # expand to get standard form
            res = sy.expand(res)
            # in order to check whether there are derivatives left, run on the discretizer on the result again
            res = discretize(res)
        else:
            res = expr

    return res
```

```python
expr3 = (k**2+k)*Psi
```

```python
expr3 = sy.expand(expr3)
```

```python
sy.srepr(expr3)
```

    "Add(Mul(Symbol('k'), Function('Psi')(Symbol('x'))), Mul(Pow(Symbol('k'), Integer(2)), Function('Psi')(Symbol('x'))))"

```python

```


```python
import numpy as np
import sympy as sym
import numba
import pydae.build as db
```

```python

```

\begin{eqnarray}
\dot \delta &=& \Omega_b \left(\omega - \omega_s\right)\\
\dot \omega &=& 1/(2 H) \left(p_m - p_e - D (\omega - \omega_s) \right)\\
\end{eqnarray}

$$ \sf
\Omega_{b} \left(\sf \omega - \omega_{s}\right)
$$

## System definition

```python
params = {'S_b':500.0,
          'X_d':1.81,'X1d':0.3,'T1d0':8.0,   # synnchronous machine d-axis parameters
          'X_q':1.76,'X1q':0.65,'T1q0':1.0,  # synnchronous machine q-axis parameters
          'R_a':0.003,'X_l': 0.05,
          'H':3.5,'D':1.0,
          'Omega_b':2*np.pi*50,'omega_s':1.0,
          'v_0':1.0,'theta_0':0.0,
          'K_a':100, 'T_r':0.1, 'v_pss':0.0,
          'T_c':2.0,'T_b':10.0,'Kpgov':10.0,'Kigov':2.0,'Droop':0.05,'Kimw':0.0,'S_b_g':500.0}

u_ini_dict = {'P_t':0.8, 'Q_t':0.2}  # for the initialization problem
u_run_dict = {'p_m':0.8,'v_ref':1.0}  # for the running problem (here initialization and running problem are the same)

x_list_syn = ['delta','omega','e1q','e1d','v_c']    # [inductor current, PI integrator]
y_ini_list = ['i_d','i_q','v_1','theta_1','p_m','v_ref','v_f'] # for the initialization problem
y_run_list = ['i_d','i_q','v_1','theta_1','P_t','Q_t',  'v_f'] # for the running problem (here initialization and running problem are the same)

x_list_gov = ['xi_gov','xi_mw','x_gov','p_m_meas']    # [inductor current, PI integrator]

sys_vars = {'params':params,
            'u_list':u_run_dict,
            'x_list':x_list_syn + x_list_gov,
            'y_list':y_run_list}

exec(db.sym_gen_str())  # exec to generate the required symbolic varables and constants
```

```python
v_d = v_1*sin(delta - theta_1)
v_q = v_1*cos(delta - theta_1)

p_e = i_d*(v_d + R_a*i_d) + i_q*(v_q + R_a*i_q)

ddelta = Omega_b*(omega - omega_s)
domega = 1/(2*H)*(p_m - p_e - D*(omega - omega_s))
de1q = 1/T1d0*(-e1q - (X_d - X1d)*i_d + v_f)
de1d = 1/T1q0*(-e1d + (X_q - X1q)*i_q)
dv_c =   (v_1 - v_c)/T_r

g_1 = v_q + R_a*i_q + X1d*i_d - e1q
g_2 = v_d + R_a*i_d - X1q*i_q - e1d
g_3 = P_t - (v_1*v_0*sin(theta_1 - theta_0))/X_l
g_4 = Q_t + (v_1*v_0*cos(theta_1 - theta_0))/X_l - v_1**2/X_l
g_5 = i_d*v_d + i_q*v_q - P_t
g_6 = i_d*v_q - i_q*v_d - Q_t
g_7 = K_a*(v_ref - v_c + v_pss) - v_f

sys = {'name':'smib_milano_ex8p1_4ord_avr',
       'params_dict':params,
       'f_list':[ddelta,domega,de1q,de1d,dv_c],
       'g_list':[g_1,g_2,g_3,g_4,g_5,g_6,g_7],
       'x_list':x_list,
       'y_ini_list':y_ini_list,
       'y_run_list':y_run_list,
       'u_run_dict':u_run_dict,
       'u_ini_dict':u_ini_dict,
       'h_dict':{'p_m':p_m}}

sys = db.system(sys)
db.sys2num(sys)

```

```python

omega_droop = Droop*p_m_meas
p_gov= Kpgov*(omega_ref - omega - omega_droop + Kimw*xi_mw) + Kigov*xi_gov
p_m = x_gov + T_c_1/T_b_1*(p_gov - x_gov)
f1 =  1.0 - omega_1 - omega_droop_1 + Kimw_1*xi_mw_1
f2 = p_ref - p_m
f3 = (p_gov_1 - x_gov_1)/T_b_1
f4 = 1/0.1*(p_m_1 - p_m_meas_1)

g_grid_p = P_t_1*S_b_1/S_b - (v_1*v_0*sin(theta_1 - theta_0))/X_l_1
g_grid_q = Q_t_1*S_b_1/S_b + (v_1*v_0*cos(theta_1 - theta_0))/X_l_1 - v_1**2/X_l_1
```

```python

```


# Polar and Cilindrical Frame of Reference
Renato Naville Watanabe

Consider that we have the position vector $\bf\vec{r}$ of a particle, moving in a circular path indicated in the figure below by a dashed line. This vector $\bf\vec{r(t)}$ is described in a fixed reference frame as:

\begin{equation}
{\bf\hat{r}}(t) = {x}{\bf\hat{i}}+{y}{\bf\hat{j}} + {z}{\bf\hat{k}}
\end{equation}

Naturally, we could describe all the kinematic variables in the fixed reference frame. But in circular motions, is convenient to define a basis with a vector in the direction of the position vector  $\bf\vec{r}$. So, the vector  $\bf\hat{e_R}$ is defined as:

\begin{equation}
{\bf\hat{e_R}} =  \frac{\bf\vec{r}}{\Vert{\bf\vec{r} }\Vert}
\end{equation}

The second vector of the basis can be obtained by the cross multiplication between $\bf\hat{k}$ and $\bf\hat{e_R}$:

\begin{equation}
{\bf\hat{e_\theta}} = {\bf\hat{k}} \times {\bf\hat{e_R}}
\end{equation}

The third vector of the basis is the conventional ${\bf\hat{k}}$ vector.

This basis can be used also for non-circular movements. For a 3D movement, the versor ${\bf\hat{e_R}}$ is obtained by removing the projection of the vector ${\bf\vec{r}}$  onto the versor ${\bf\hat{k}}$:

\begin{equation}
{\bf\hat{e_R}} =  \frac{\bf\vec{r} - ({\bf\vec{r}.{\bf\hat{k}}){\bf\hat{k}}}}{\Vert\bf\vec{r} - ({\bf\vec{r}.{\bf\hat{k}}){\bf\hat{k}}\Vert}}
\end{equation}

## Time-derivative of the versors ${\bf\hat{e_R}}$ and ${\bf\hat{e_\theta}}$

To obtain the expressions of the velocity and acceleration vectors, it is necessary to obtain the expressions of the time-derivative of the vectors ${\bf\hat{e_R}}$ and ${\bf\hat{e_\theta}}$.

This can be done by noting that:

\begin{align}
{\bf\hat{e_R}} &= \cos(\theta){\bf\hat{i}} + \sin(\theta){\bf\hat{j}}\\
{\bf\hat{e_\theta}} &= -\sin(\theta){\bf\hat{i}} + \cos(\theta){\bf\hat{j}}
\end{align}

Deriving ${\bf\hat{e_R}}$ we obtain:

\begin{equation}
\frac{d{\bf\hat{e_R}}}{dt} = -\sin(\theta)\dot\theta{\bf\hat{i}} + \cos(\theta)\dot\theta{\bf\hat{j}} = \dot{\theta}{\bf\hat{e_\theta}}
\end{equation}

Similarly, we obtain the time-derivative of ${\bf\hat{e_\theta}}$:

\begin{equation}
    \frac{d{\bf\hat{e_\theta}}}{dt} = -\cos(\theta)\dot\theta{\bf\hat{i}} - \sin(\theta)\dot\theta{\bf\hat{j}} = -\dot{\theta}{\bf\hat{e_R}}
\end{equation}

## Position, velocity and acceleration

### Position

The position vector $\bf\vec{r}$, from the definition of $\bf\hat{e_R}$, is:

\begin{equation}
{\bf\vec{r}} = R{\bf\hat{e_R}} + z{\bf\hat{k}}
\end{equation}

where $R = \Vert\bf\vec{r} - ({\bf\vec{r}.{\bf\hat{k}}){\bf\hat{k}}\Vert}$.

### Velocity

The velocity vector $\bf\vec{v}$ is obtained by deriving the vector $\bf\vec{r}$:

\begin{equation}
{\bf\vec{v}} = \frac{d(R{\bf\hat{e_R}})}{dt} + \dot{z}{\bf\hat{k}} = \dot{R}{\bf\hat{e_R}}+R\frac{d\bf\hat{e_R}}{dt}=\dot{R}{\bf\hat{e_R}}+R\dot{\theta}{\bf\hat{e_\theta}}+ \dot{z}{\bf\hat{k}}
\end{equation}

### Acceleration

The acceleration vector $\bf\vec{a}$ is obtained by deriving the velocity vector:

\begin{align}
{\bf\vec{a}} =& \frac{d(\dot{R}{\bf\hat{e_R}}+R\dot{\theta}{\bf\hat{e_\theta}}+\dot{z}{\bf\hat{k}})}{dt}=\\
    =&\ddot{R}{\bf\hat{e_R}}+\dot{R}\frac{d\bf\hat{e_R}}{dt} + \dot{R}\dot{\theta}{\bf\hat{e_\theta}} + R\ddot{\theta}{\bf\hat{e_\theta}} + R\dot{\theta}\frac{d{\bf\hat{e_\theta}}}{dt} + \ddot{z}{\bf\hat{k}}=\\
    =&\ddot{R}{\bf\hat{e_R}}+\dot{R}\dot{\theta}{\bf\hat{e_\theta}} + \dot{R}\dot{\theta}{\bf\hat{e_\theta}} + R\ddot{\theta}{\bf\hat{e_\theta}} - R\dot{\theta}^2{\bf\hat{e_R}}+ \ddot{z}{\bf\hat{k}} =\\
    =&\ddot{R}{\bf\hat{e_R}}+2\dot{R}\dot{\theta}{\bf\hat{e_\theta}}+ R\ddot{\theta}{\bf\hat{e_\theta}} - {R}\dot{\theta}^2{\bf\hat{e_R}}+ \ddot{z}{\bf\hat{k}} =\\
    =&(\ddot{R}-R\dot{\theta}^2){\bf\hat{e_R}}+(2\dot{R}\dot{\theta} + R\ddot{\theta}){\bf\hat{e_\theta}}+ \ddot{z}{\bf\hat{k}}
\end{align}

+ The term $\ddot{R}$ is  an acceleration in the radial direction.

+ The term $R\ddot{\theta}$ is an angular acceleration.

+ The term $\ddot{z}$ is an acceleration in the $\bf\hat{k}$ direction.

+ The term $-R\dot{\theta}^2$ is the well known centripetal acceleration.

+ The term $2\dot{R}\dot{\theta}$ is known as Coriolis acceleration. This term may be difficult to understand. It appears when there is displacement in the radial and angular directions at the same time.

## Important to note

The reader must bear in mind that the use of a different basis to represent the position, velocity or acceleration vectors is only a different representation of the same vector. For example, for the acceleration vector:

\begin{equation}
{\bf\vec{a}} = \ddot{x}{\bf\hat{i}}+ \ddot{y}{\bf\hat{j}} + \ddot{z}{\bf\hat{k}}=(\ddot{R}-R\dot{\theta}^2){\bf\hat{e_R}}+(2\dot{R}\dot{\theta} + R\ddot{\theta}){\bf\hat{e_\theta}}+ \ddot{z}{\bf\hat{k}}=\dot{\Vert\bf\vec{v}\Vert}{\bf\hat{e}_t}+{\Vert\bf\vec{v}\Vert}^2\Vert{\bf\vec{C}} \Vert{\bf\hat{e}_n}
\end{equation}

In which the last equality is the acceleration vector represented in the path-coordinate of the particle (see http://nbviewer.jupyter.org/github/BMClab/bmc/blob/master/notebooks/Time-varying%20frames.ipynb).

## Example

Consider a particle following the spiral path described below:

\begin{equation}
{\bf\vec{r}}(t) = (2\sqrt(t)\cos(t)){\bf\hat{i}}+ (2\sqrt(t)\sin(t)){\bf\hat{j}}
\end{equation}

```python
import numpy as np
import sympy as sym
from sympy.plotting import plot_parametric,plot3d_parametric_line
from sympy.vector import CoordSys3D
import matplotlib.pyplot as plt
# from matplotlib import rc
# rc('text', usetex=True)
sym.init_printing()
```

### Solving numerically

```python
t = np.linspace(0.01,10,30).reshape(-1,1) #create a time vector and reshapes it to a column vector
R = 2*np.sqrt(t)
theta = t
rx = R*np.cos(t)
ry = R*np.sin(t)
r = np.hstack((rx, ry)) # creates the position vector by stacking rx and ry horizontally
```

```python
e_r = r/np.linalg.norm(r, axis=1, keepdims=True) # defines e_r vector
e_theta = np.cross([0,0,1],e_r)[:,0:-1] # defines e_theta vector

```

```python
dt = t[1] #defines delta_t
Rdot = np.diff(R, axis=0)/dt #find the R derivative
thetaDot = np.diff(theta, axis=0)/dt #find the angle derivative
v = Rdot*e_r[0:-1,:] +R[0:-1]*thetaDot*e_theta[0:-1,:] # find the linear velocity.
```

```python
Rddot = np.diff(Rdot, axis=0)/dt
thetaddot = np.diff(thetaDot, axis=0)/dt
```

```python
a = ((Rddot - R[1:-1]*thetaDot[0:-1]**2)*e_r[1:-1,:]
     + (2*Rdot[0:-1]*thetaDot[0:-1] + Rdot[0:-1]*thetaddot)*e_theta[1:-1,:])
```

```python
from matplotlib.patches import FancyArrowPatch
%matplotlib inline
plt.rcParams['figure.figsize']=10,10
fig = plt.figure()
plt.plot(r[:,0],r[:,1],'.')
ax = fig.add_axes([0,0,1,1])
for i in np.arange(len(t)-2):
    vec1 = FancyArrowPatch(r[i,:],r[i,:]+e_r[i,:],mutation_scale=30,color='r', label='e_r')
    vec2 = FancyArrowPatch(r[i,:],r[i,:]+e_theta[i,:],mutation_scale=30,color='g', label='e_theta')
    ax.add_artist(vec1)
    ax.add_artist(vec2)
plt.xlim((-10,10))
plt.ylim((-10,10))
plt.grid()
plt.legend([vec1, vec2],[r'$\vec{e_r}$', r'$\vec{e_{\theta}}$'])
plt.show()
```

```python
from matplotlib.patches import FancyArrowPatch
%matplotlib inline
plt.rcParams['figure.figsize']=10,10
fig = plt.figure()
plt.plot(r[:,0],r[:,1],'.')
ax = fig.add_axes([0,0,1,1])
for i in np.arange(len(t)-2):
    vec1 = FancyArrowPatch(r[i,:],r[i,:]+v[i,:],mutation_scale=10,color='r')
    vec2 = FancyArrowPatch(r[i,:],r[i,:]+a[i,:],mutation_scale=10,color='g')
    ax.add_artist(vec1)
    ax.add_artist(vec2)
plt.xlim((-10,10))
plt.ylim((-10,10))
plt.grid()
plt.legend([vec1, vec2],[r'$\vec{v}$', r'$\vec{a}$'])
plt.show()
```

### Solved simbolically (extra reading)

```python
O = sym.vector.CoordSys3D(' ')
t = sym.symbols('t')
```

```python
r = 2*sym.sqrt(t)*sym.cos(t)*O.i+2*sym.sqrt(t)*sym.sin(t)*O.j
r
```

```python
plot_parametric(r.dot(O.i),r.dot(O.j),(t,0,10))
```

```python
e_r = r - r.dot(O.k)*O.k
e_r = e_r/sym.sqrt(e_r.dot(O.i)**2+e_r.dot(O.j)**2+e_r.dot(O.k)**2)
```

```python
e_r
```

```python
e_theta = O.k.cross(e_r)
e_theta
```

```python
from matplotlib.patches import FancyArrowPatch
plt.rcParams['figure.figsize']=10,10
fig = plt.figure()
ax = fig.add_axes([0, 0, 1, 1])
ax.axis("on")
time = np.linspace(0,10,30)
for instant in time:
    vt = FancyArrowPatch([float(r.dot(O.i).subs(t,instant)),float(r.dot(O.j).subs(t,instant))],
                         [float(r.dot(O.i).subs(t,instant))+float(e_r.dot(O.i).subs(t,instant)), float(r.dot(O.j).subs(t, instant))+float(e_r.dot(O.j).subs(t,instant))],
                         mutation_scale=20,
                         arrowstyle="->",color="r",label='${{e_r}}$')
    vn = FancyArrowPatch([float(r.dot(O.i).subs(t, instant)),float(r.dot(O.j).subs(t,instant))],
                         [float(r.dot(O.i).subs(t, instant))+float(e_theta.dot(O.i).subs(t, instant)), float(r.dot(O.j).subs(t, instant))+float(e_theta.dot(O.j).subs(t, instant))],
                         mutation_scale=20,
                         arrowstyle="->",color="g",label='${{e_{theta}}}$')
    ax.add_artist(vn)
    ax.add_artist(vt)
plt.xlim((-10,10))
plt.ylim((-10,10))
plt.legend(handles=[vt,vn],fontsize=20)
plt.grid()
plt.show()
```

```python
R = 2*sym.sqrt(t)
```

```python
Rdot = sym.diff(R,t)
Rddot = sym.diff(Rdot,t)
Rddot
```

```python
v = Rdot*e_r + R*e_theta
```

```python
v
```

```python
a = (Rddot - R)*e_r + (2*Rdot*1+0)*e_theta
aCor = 2*Rdot*1*e_theta
aCor
```

```python
a
```

```python
from matplotlib.patches import FancyArrowPatch
plt.rcParams['figure.figsize'] = 10,10
fig = plt.figure()
ax = fig.add_axes([0, 0, 1, 1])
ax.axis("on")
time = np.linspace(0.1,10,30)
for instant in time:
    vt = FancyArrowPatch([float(r.dot(O.i).subs(t,instant)),float(r.dot(O.j).subs(t,instant))],
                         [float(r.dot(O.i).subs(t,instant))+float(v.dot(O.i).subs(t,instant)), float(r.dot(O.j).subs(t, instant))+float(v.dot(O.j).subs(t,instant))],
                         mutation_scale=20,
                         arrowstyle="->",color="r",label='${{v}}$')
    vn = FancyArrowPatch([float(r.dot(O.i).subs(t, instant)),float(r.dot(O.j).subs(t,instant))],
                         [float(r.dot(O.i).subs(t, instant))+float(a.dot(O.i).subs(t, instant)), float(r.dot(O.j).subs(t, instant))+float(a.dot(O.j).subs(t, instant))],
                         mutation_scale=20,
                         arrowstyle="->",color="g",label='${{a}}$')
    vc = FancyArrowPatch([float(r.dot(O.i).subs(t, instant)),float(r.dot(O.j).subs(t,instant))],
                         [float(r.dot(O.i).subs(t, instant))+float(aCor.dot(O.i).subs(t, instant)), float(r.dot(O.j).subs(t, instant))+float(aCor.dot(O.j).subs(t, instant))],
                         mutation_scale=20,
                         arrowstyle="->",color="b",label='${{a_{Cor}}}$')
    ax.add_artist(vn)
    ax.add_artist(vt)
    ax.add_artist(vc)
plt.xlim((-10,10))
plt.ylim((-10,10))
plt.legend(handles=[vt,vn,vc],fontsize=20)
plt.grid()
plt.show()
```

## Problems

1. Problems from 15.1.1 to 15.1.14 from Ruina and Rudra's book,
2. Problems from 18.1.1 to 18.1.8 and 18.1.10 from Ruina and Rudra's book.

## Reference

- Ruina A, Rudra P (2019) [Introduction to Statics and Dynamics](http://ruina.tam.cornell.edu/Book/index.html). Oxford University Press.

```python

```