#  Taylor Examples 5.4 and 5.5

last revised: 17-Jan-2019 by Dick Furnstahl [furnstahl.1@osu.edu]

Here we are exploring the Fourier series for a waveform defined to be even about the origin, so $f(-t) = +f(t)$, with period $\tau$.  That means that the integrand for the $b_m$ coefficients is odd and so all of the corresponding integrals vanish.

The particular wave of interest here is a periodic rectangular pulse, such that in the interval $-\tau/2 \leq t \leq \tau/2$, the function takes the form:

$\newcommand{\fmax}{f_{\textrm{max}}}$
$\begin{align}
  f(t) = \left\{ \begin{array}{lc}
                   \fmax & -\Delta\tau/2 < t < \Delta\tau/2 \\
                   0 & |t| > \Delta\tau/2
                 \end{array}
                   \right.
\end{align}$

As usual, we do the $a_0$ integral separately and then $a_m$ for $m \geq 1$.  Here $\omega \equiv 2\pi/\tau$.  The results are (also see the text):

$\begin{align}
  a_0 = \frac{1}{\tau} \int_{-\tau/2}^{\tau/2} f(t)\, dt = \frac{\fmax\Delta\tau}{\tau}
\end{align}$

$\begin{align}
  a_m = \frac{2}{\tau} \int_{-\tau/2}^{\tau/2} \cos(m\omega t) f(t)\, dt =
      \frac{4\fmax}{\tau} \int_0^{\Delta\tau/2} \cos(2\pi m t/\tau) \, dt
    %  &= - \frac{2\fmax}{m\pi}\left[\cos(m\pi t)\right]^1_0 \\
      =
        \frac{2 \fmax}{m\pi} \sin\left(\frac{\pi m \Delta\tau}{\tau}\right)
\end{align}$

## Define the functions we'll need

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import quad
```

We start by defining a function for the rectangular pulse at any $t$.  The definition here is for a scalar function.  That is, it won't work to call it with $t$ and array of time points, unlike other functions we have defined.  It is possible to make it work, but then the function will be much less clear.  When we need to evaluate it for all elements of an array `t_pts`, we will use the construction: <br/> `np.array([rectangular(t) for t in t_pts])`.

```python
def rectangular(t, tau=1., Delta_tau=0.25, f_max=1.):
    """Returns the triangular wave of amplitude f_max and even about the
        origin at time t.  The period is tau.  It is defined as a scalar
        function (i.e., only one value of t can be passed at a time).
        **There must be a better way to define this!**
    """
    t_adjust = t - round(t)  # shift t so we can test for being within Delta_tau

    if np.fabs(t_adjust) < Delta_tau/2.:
        return f_max
    else:
        return 0.
```

Now a function that creates an array of Fourier coefficients for the triangular wave up to order N_max.

```python
def rectangular_coeffs_by_hand(N_max, tau=1., Delta_tau=0.25, f_max=1):
    """Fourier coefficients calculated by hand and loaded into an array
        as pairs in the form (a_n, b_n).
    """
    # initialize the array with the vaues of a_0 and b_0
    coeffs_array = [(f_max * Delta_tau / tau, 0)]
    for n in np.arange(1, N_max, 1):
        # these are pre-calcualated values of a_n and b_n
        a_n = 2.*f_max/(n * np.pi) * np.sin((np.pi * n * Delta_tau) / tau)
        b_n = 0.
        coeffs_array.append((a_n, b_n))  # add them to the end of the array
    return np.array(coeffs_array)  # convert to a numpy array to be safe

```

We would like a general way to construct the array of Fourier coefficients given any periodic function.  Our first pass at that uses a class definition and the scipy integration function quad.

```python
class FourierSeries():
    """
    Fourier series class finds the coefficients in a Fourier series with
    period tau up to a specified order.
    Assume these imports:
        from scipy.integrate import quad
        import numpy as np
    ***Need to add args to the function.***
    """
    def __init__(self,
                 function,
                 tau=1.,
                 N_max=10
                ):
        self.function = function
        self.tau = tau
        self.omega = 2. * np.pi / tau
        self.N_max = N_max
        # add something to quit if Nmax < 0 or not an integer (try and except)

    def a0_calc(self):
        """Calculate the constant Fourier coefficient by integration"""
        answer, error = quad(self.function, -tau/2., tau/2., args=(tau,))
        return (1./self.tau) * answer

    def an_integrand(self, t, n):
        """Integrand for the nth cosine coefficient"""
        return self.function(t, tau) * np.cos(n * self.omega * t)

    def an_calc(self, n):
        """Calculate the nth cosine coefficient (n > 0)"""
        # note comma after n in args
        answer, error = quad(self.an_integrand, -tau/2., tau/2., args=(n,))
        return (2./self.tau) * answer

    def bn_integrand(self, t, n):
        """Integrand for the nth cosine coefficient"""
        return self.function(t, tau) * np.sin(n * self.omega * t)

    def bn_calc(self, n):
        """Calculate the nth cosine coefficient (n > 0)"""
        answer, error = quad(self.bn_integrand, -tau/2., tau/2., args=(n,))
        return (2./self.tau) * answer

    def coeffs_upto_Nmax(self):
        """Calculate the Fourier series up to Nmax"""
        # first generate the coefficient
        coeffs_array = [(self.a0_calc(), 0)]   # a_0 and b_0

        for n in np.arange(1, N_max, 1):
            a_n = self.an_calc(n)
            b_n = self.bn_calc(n)
            coeffs_array.append((a_n, b_n))  # append a tuple of coefficients

        return np.array(coeffs_array)  # convert to a numpy array

```

Finally, we need a function that can take as input an array of t values and an array of Fourier coefficients and return the function at those t values with terms up to order N_max.

```python

def Fourier_reconstruct(t_pts, coeffs_array, tau, N_max):
    """Sum up the Fourier series up to n = N_max terms."""
    omega = 2. * np.pi / tau   # definition of omega
    result = 0.
    # iterate over coefficients but only up to N_max
    for n, (a,b) in enumerate(coeffs_array[:N_max+1]):
        result = result + a * np.cos(n * omega * t_pts) \
                        + b * np.sin(n * omega * t_pts)
    return result
```

## Example 5.4

Ok, now we can do Example 5.4.  Calculate the coefficients both ways.

```python
N_max = 20
tau = 1.
f_max = 1.
Delta_tau = 0.25
coeffs_by_hand = rectangular_coeffs_by_hand(N_max, tau, Delta_tau, f_max)

fs = FourierSeries(rectangular, tau, N_max)
coeffs_by_quad = fs.coeffs_upto_Nmax()
```

Let's check that the exact and numerical calculation of the coefficients agree.

(Note the space in the formats below, e.g., `{a1: .6f}`.  This means to leave an extra space for a positive number so that it aligns at the decimal point with negative numbers.)

```python
print(' n   a_exact     a_quad      b_exact     b_quad')
for n, ((a1,b1), (a2,b2)) in enumerate(zip(coeffs_by_hand, coeffs_by_quad)):
    print(f'{n:2d}  {a1: .6f}  {a2: .6f}    {b1: .6f}  {b2: .6f}')

# Lists don't need to be indexed in order to be zipped!
```

     n   a_exact     a_quad      b_exact     b_quad
     0   0.250000   0.250000     0.000000   0.000000
     1   0.450158   0.450158     0.000000   0.000000
     2   0.318310   0.318310     0.000000   0.000000
     3   0.150053   0.150053     0.000000   0.000000
     4   0.000000   0.000000     0.000000   0.000000
     5  -0.090032  -0.090032     0.000000   0.000000
     6  -0.106103  -0.106103     0.000000   0.000000
     7  -0.064308  -0.064308     0.000000   0.000000
     8  -0.000000   0.000000     0.000000   0.000000
     9   0.050018   0.050018     0.000000   0.000000
    10   0.063662   0.063662     0.000000   0.000000
    11   0.040923   0.040923     0.000000   0.000000
    12   0.000000   0.000000     0.000000   0.000000
    13  -0.034628  -0.034628     0.000000   0.000000
    14  -0.045473  -0.045473     0.000000   0.000000
    15  -0.030011  -0.030011     0.000000   0.000000
    16  -0.000000   0.000000     0.000000   0.000000
    17   0.026480   0.026480     0.000000   0.000000
    18   0.035368   0.035368     0.000000   0.000000
    19   0.023693   0.023693     0.000000   0.000000

Make the comparison plot requested: N_max = 2 vs. N_max = 6.

```python
t_pts = np.arange(-2., 4., .01)
f_pts_2 = Fourier_reconstruct(t_pts, coeffs_by_hand, tau, 2)
f_pts_6 = Fourier_reconstruct(t_pts, coeffs_by_hand, tau, 50)

# Python way to evaluate the rectangular function at an array of points:
#   * np.array creates a numpy array;
#   * note the []s around the inner statement;
#   * rectangular(t) for t in t_pts
#       means step through each element of t_pts, call it t, and
#       evaluate rectangular at that t.
#   * This is called a list comprehension.  There are more compact ways,
#      but this is clear and easy to debug.
rectangular_t_pts = np.array([rectangular(t, tau, Delta_tau, f_max) \
                              for t in t_pts])
```

```python
# It looks like we should define a function for the axis!

fig_1 = plt.figure(figsize=(10,5))

ax_1 = fig_1.add_subplot(1,2,1)
ax_1.plot(t_pts, f_pts_2, label='N = 2', color='blue')
ax_1.plot(t_pts, rectangular_t_pts, label='exact', color='red')
ax_1.set_xlim(-1.1,4.1)
ax_1.set_xlabel('t')
ax_1.set_ylabel('f(t)')
ax_1.set_title('N = 2')
ax_1.legend()

ax_2 = fig_1.add_subplot(1,2,2)
ax_2.plot(t_pts, f_pts_6, label='N = 6', color='blue')
ax_2.plot(t_pts, rectangular_t_pts, label='exact', color='red')
ax_2.set_xlim(-1.1,4.1)
ax_2.set_xlabel('t')
ax_2.set_ylabel('f(t)')
ax_2.set_title('N = 6')
ax_2.legend();

fig_1.tight_layout()
fig_1.savefig('example_5.4.png')
```

## Example 5.5

We consider a driven oscillator with an even driving force so only cosine terms are non-zero in the Fourier transform of $f(t)$.  Linearity of the equation then means that the solution takes the form

$\begin{align}
  x(t) = \sum_{n=0}^{\infty} A_n \cos(n\omega t - \delta_n) \;,
\end{align}$

where

$\begin{align}
  A_n &= \frac{f_n}{\sqrt{(\omega_0^2 - n^2 \omega^2)^2 + 4\beta^2 n^2 \omega^2}}  \\
  \delta_n &= \arctan\left(\frac{2\beta n\omega}{\omega_0^2 - n^2 \omega^2}  \right)
\end{align}$

For Example 5.5 we apply this to the rectangular driving force, so we can use the coefficients from Example 5.4, which play the role of the $f_n$ coefficients here.

```python
def driven_coefficients(tau_0, tau, beta, coeffs):
    """Calculate the coefficients A_n and delta_n for a driven oscillator."""
    omega_0 = 2. * np.pi / tau_0
    omega = 2. * np.pi / tau

    A_n = []
    delta_n = []

    # the second entry in the tuple is called dummy because it is not used
    for n, (f_n, dummy) in enumerate(coeffs):
        A_n.append(f_n / np.sqrt((omega_0**2 - n**2 * omega**2)**2 +
                                  (2. * beta * n *omega)**2))
        delta_n.append(np.arctan2(2. * beta * n *omega,
                                  omega_0**2 - n**2 * omega**2 ))
    return np.array(A_n), np.array(delta_n)

def x_drive(t_pts, tau_0, tau, beta, N_max, coeffs):
    """Sum up the Fourier series for x(t) up to n = N_max terms."""
    omega = 2. * np.pi / tau
    A_n_array, delta_n_array = driven_coefficients(tau_0, tau, beta, coeffs)

    result = 0.
    # iterate over coefficients but only up to N_max
    for n, (A_n, delta_n) in enumerate(zip(A_n_array[:N_max+1],
                                           delta_n_array[:N_max+1])):
        result = result + A_n * np.cos( n * omega * t_pts - delta_n)
    return result
```

Ok, let's do Example 5.5!

```python
N_max = 20
tau = 1.
f_max = 1.
Delta_tau = 0.25
coeffs_by_hand = rectangular_coeffs_by_hand(N_max, tau, Delta_tau, f_max)

```

```python
t_pts = np.arange(0., 7., .01)

# first set of parameters
N_max = 20
f_max = 1.
Delta_tau = 0.25
tau = 1.
coeffs_by_hand_1 = rectangular_coeffs_by_hand(N_max, tau, Delta_tau, f_max)

tau_0_1 = 1.
beta = 0.2
A_n_array_1, delta_n_array_1 = driven_coefficients(tau_0_1, tau, beta,
                                                   coeffs_by_hand_1)
x_pts_1 = x_drive(t_pts, tau_0_1, tau, beta, N_max, coeffs_by_hand_1)

# now only tau is varied
tau = 1.5
coeffs_by_hand_2 = rectangular_coeffs_by_hand(N_max, tau, Delta_tau, f_max)

tau_0_2 = 1.
beta = 0.2
A_n_array_2, delta_n_array_2 = driven_coefficients(tau_0_2, tau, beta,
                                                   coeffs_by_hand_2)
x_pts_2 = x_drive(t_pts, tau_0_2, tau, beta, N_max, coeffs_by_hand_2)

# now only tau is varied
tau = 2.0
coeffs_by_hand_3 = rectangular_coeffs_by_hand(N_max, tau, Delta_tau, f_max)

tau_0_3 = 1.
beta = 0.2
A_n_array_3, delta_n_array_3 = driven_coefficients(tau_0_3, tau, beta,
                                                   coeffs_by_hand_3)
x_pts_3 = x_drive(t_pts, tau_0_3, tau, beta, N_max, coeffs_by_hand_3)

# now only tau is varied
tau = 2.5
coeffs_by_hand_4 = rectangular_coeffs_by_hand(N_max, tau, Delta_tau, f_max)

tau_0_4 = 1.
beta = 0.2
A_n_array_4, delta_n_array_4 = driven_coefficients(tau_0_4, tau, beta,
                                                   coeffs_by_hand_4)
x_pts_4 = x_drive(t_pts, tau_0_4, tau, beta, N_max, coeffs_by_hand_4)

```

How would you code the last cell differently so that you could just give an array of $\tau$ values?

Compare coefficients for the cases from parts 1 to 4: large $A_n$ coefficients when that component is in resonance.

```python
print(' n      case 1        case 2        case 3        case 4')
for n, (A_n_1, A_n_2, A_n_3, A_n_4) in enumerate(zip(A_n_array_1, A_n_array_2,
                                                     A_n_array_3, A_n_array_4)):
    print(f'{n:2d}   {A_n_1: .8f}   {A_n_2: .8f}   {A_n_3: .8f}   {A_n_4: .8f}')
```

     n      case 1        case 2        case 3        case 4
     0    0.00633257    0.00422172    0.00316629    0.00253303
     1    0.17911224    0.01447102    0.00822068    0.00592958
     2    0.00268521    0.00892472    0.08955612    0.01303477
     3    0.00047497    0.00179014    0.00396133    0.00973767
     4    0.00000000    0.00057109    0.00134261    0.00245255
     5   -0.00009501    0.00015945    0.00056729    0.00107408
     6   -0.00007678    0.00000000    0.00023749    0.00053672
     7   -0.00003393   -0.00005543    0.00007835    0.00027238
     8   -0.00000000   -0.00006360    0.00000000    0.00012820
     9    0.00001584   -0.00005119   -0.00003562    0.00004629
    10    0.00001629   -0.00003214   -0.00004751    0.00000000
    11    0.00000864   -0.00001389   -0.00004630   -0.00002467
    12    0.00000000   -0.00000000   -0.00003839   -0.00003583
    13   -0.00000522    0.00000837   -0.00002778   -0.00003854
    14   -0.00000591    0.00001158   -0.00001697   -0.00003608
    15   -0.00000339    0.00001086   -0.00000745   -0.00003071
    16   -0.00000000    0.00000774   -0.00000000   -0.00002399
    17    0.00000233    0.00000372    0.00000509   -0.00001696
    18    0.00000277    0.00000000    0.00000792   -0.00001036
    19    0.00000167   -0.00000266    0.00000879   -0.00000462

Now for the graphs in Figure 5.25:

```python
fig_2 = plt.figure(figsize=(10,5))

ax_1 = fig_2.add_subplot(2,2,1)
ax_1.plot(t_pts, x_pts_1, color='blue')
ax_1.set_xlim(0., 7.)
ax_1.set_ylim(-0.2, 0.2)
ax_1.set_xlabel('t')
ax_1.set_ylabel('x(t)')
ax_1.set_title(r'case (1): $\tau = 1$')

ax_2 = fig_2.add_subplot(2,2,2)
ax_2.plot(t_pts, x_pts_2, color='blue')
ax_2.set_xlim(0., 7.)
ax_2.set_ylim(-0.2, 0.2)
ax_2.set_xlabel('t')
ax_2.set_ylabel('x(t)')
ax_2.set_title(r'case (2): $\tau = 1.5$')

ax_3 = fig_2.add_subplot(2,2,3)
ax_3.plot(t_pts, x_pts_3, color='blue')
ax_3.set_xlim(0., 7.)
ax_3.set_ylim(-0.2, 0.2)
ax_3.set_xlabel('t')
ax_3.set_ylabel('x(t)')
ax_3.set_title(r'case (3): $\tau = 2.0$')

ax_4 = fig_2.add_subplot(2,2,4)
ax_4.plot(t_pts, x_pts_4, color='blue')
ax_4.set_xlim(0., 7.)
ax_4.set_ylim(-0.2, 0.2)
ax_4.set_xlabel('t')
ax_4.set_ylabel('x(t)')
ax_4.set_title(r'case (4): $\tau = 2.5$')

fig_2.tight_layout()
fig_2.savefig('Example_.png')
```

To add additional graphs with more $\tau$ values, we should really define a function that produces the plot given a $\tau$.

```python

```


```python
from IPython.display import Image
from IPython.core.display import HTML
```

```python
Image(url= "https://i.imgur.com/OKzEasd.png")
```

```python
from sympy import *
from sympy import euler
x = symbols('x')
c = symbols('c', real = True)
sympify(c).is_real
```

    True

```python
eq5 = c*E**(x)
Eq(eq5,100) #solving for c when x = 10 because the plot looks nice
```

$\displaystyle c e^{x} = 100$

```python
factorial??
```

```python
eq = x**x
Eq(Derivative(eq).doit(),E**(log(x)*x))
```

$\displaystyle x^{x} \left(\log{\left(x \right)} + 1\right) = e^{x \log{\left(x \right)}}$

```python
E**log(x)
```

$\displaystyle x$

```python
for x in range(1,10):
   print(x**x)
```

    1
    4
    27
    256
    3125
    46656
    823543
    16777216
    387420489

```python
for x in range(1,10):
    print((x**x)*(log(x)+1).evalf())
```

    1.00000000000000
    6.77258872223978
    56.6625317940390
    610.891356446692
    8154.49347635656
    130252.329796304
    2426083.68188346
    51664455.9041356
    1238670309.19442

```python
import math
x = symbols('x')
f = (x**x)*(log(x)+1)
#plot(x**x, f,yscale='log',xlim = (-1,10),line_color=1,legend=True)

p1 = plot(x**x, show=False,line_color=lambda x:math.cos(x), title="Derivatives of x**x",yscale='log',xscale='log',size = (9,5))
p2 = plot(f, show = False,line_color=lambda x:math.cos(x))
p3 = plot(diff(x**x,x,2),show=False,line_color=lambda x:math.cos(x))
p4 = plot(diff(x**x,x,3),show=False,line_color=lambda x:math.cos(x))

p5 = plot(diff(x**x,x,4),show=False,line_color=lambda x:math.cos(x))
p6 = plot(diff(x**x,x,5),show=False,line_color=lambda x:math.cos(x))
p7 = plot(diff(x**x,x,6),show=False,line_color=lambda x:math.cos(x))
p8 = plot(diff(x**x,x,7),show=False,line_color=lambda x:math.cos(x))
p9 = plot(diff(x**x,x,8),show=False,line_color=lambda x:math.cos(x))

p1.extend(p2)
p1.extend(p3)
p1.extend(p4)
p1.extend(p5)
p1.extend(p6)
p1.extend(p7)
p1.extend(p8)
p1.extend(p9)

p1.show()
```

```python
Derivative(x**x) > x**x
```

$\displaystyle \frac{d}{d x} x^{x} > x^{x}$

```python
(Derivative(x**x,x,2)> Derivative(x**x,x,1))
```

$\displaystyle \frac{d^{2}}{d x^{2}} x^{x} > \frac{d}{d x} x^{x}$

```python
(Derivative(x**x,x,2)> Derivative(x**x,x,1)).doit()
```

$\displaystyle x^{x} \left(\left(\log{\left(x \right)} + 1\right)^{2} + \frac{1}{x}\right) > x^{x} \left(\log{\left(x \right)} + 1\right)$

```python
(Derivative(x**x,x,4)> Derivative(x**x,x,3)).doit()
```

$\displaystyle x^{x} \left(\left(\log{\left(x \right)} + 1\right)^{4} + \frac{6 \left(\log{\left(x \right)} + 1\right)^{2}}{x} - \frac{4 \left(\log{\left(x \right)} + 1\right)}{x^{2}} + \frac{3}{x^{2}} + \frac{2}{x^{3}}\right) > x^{x} \left(\left(\log{\left(x \right)} + 1\right)^{3} + \frac{3 \left(\log{\left(x \right)} + 1\right)}{x} - \frac{1}{x^{2}}\right)$

```python
from sympy import oo
n = symbols('n')
Limit(Derivative(x**x,x,n),n,oo)  # this does not display what I want accurately in LaTeX
```

$\displaystyle \lim_{n \to \infty} \frac{d^{2}}{d nd x} x^{x}$

```python
log(x,x**x)
```

$\displaystyle \frac{\log{\left(x \right)}}{\log{\left(x^{x} \right)}}$

```python
for i in range(2,5):
    print(log(x,i**i))
```

    log(x)/log(4)
    log(x)/log(27)
    log(x)/log(256)

```python
Eq(Derivative(log(x,x**x)),Derivative(log(x,x**x)).doit())
```

$\displaystyle \frac{d}{d x} \frac{\log{\left(x \right)}}{\log{\left(x^{x} \right)}} = \frac{\left(- \log{\left(x \right)} - 1\right) \log{\left(x \right)}}{\log{\left(x^{x} \right)}^{2}} + \frac{1}{x \log{\left(x^{x} \right)}}$

```python

```

```python
Limit(x,x,2)
```

$\displaystyle \lim_{x \to 2^+} x$

```python
Eq(Derivative(E**(log(x)*x)),Derivative(E**(log(x)*x)).doit())
```

```python
Eq(((x**x)/(E**x)),simplify(((x**x)/(E**x))))
```

```python
((E**x)/(x**x))
```

```python
factorial(x).compare(simplify(((x**x)/(E**x))))
# -1 for smaller, 0 for equal and 1 for greater
```

```python
(factorial(x) > ((x**x)/(E**x))).simplify()
```

```python
plot(x**x,Derivative(eq).doit(),yscale='log') # after plotting this I noticed that it would be nice if this y= 100 right when x = 10
                 # the final solution may be useful for conceptualizing percentage of exponential growths
```

```python
log(10**4).evalf()
```

```python
Image(url= "https://i.imgur.com/F2uZGSM.png")
```

```python
eq11 = Eq(c*E**(10),100)
eq11
# This is what I want to solve.
```

```python
eq1 = 0.00453999208646404*exp(x)
Eq(eq1,eq1.subs(x,10).evalf())
```

```python
eq2 = 0.00454*exp(x)
Eq(eq2,eq2.subs(x,10).evalf())
```

```python
p0 = plot(eq1,eq2,show = False,xlim = (0,10),size = (12,5),legend = True,autoscale = True)
p0[0].line_color = 'r'
p0[1].line_color = 'b'
p0.show()
```

```python
s = solve(eq11)[0]
s # one possible solution for real number c when x = 10
```

```python
s.evalf()
```

```python
final_solution = (s*E**x)
phi = symbols('phi')
Eq(phi,final_solution)
```

```python
final_solution.subs(x,10)
```

```python
final_equation = s.evalf()*E**x
Eq(final_equation,100)
```

```python
(0.00453999297624849*E**10).evalf()
```

```python
p0 = plot(eq1,eq2,final_solution,show = False,xlim = (0,10),size = (12,5),legend = True,autoscale = True)
p0[0].line_color = 'r'
p0[1].line_color = 'b'
p0[2].line_color = 'g'
p0.show()
```

```python
eq14 = Eq(0.00453999297624849*E**(x),100)
solve(eq14)
```

```python
0.00453999208646404 < 0.00453999297624849 < 0.00454
```


```python
from sympy.physics.mechanics import ReferenceFrame,Point,dynamicsymbols
from sympy.physics.mechanics import Point
from sympy import latex,pprint,symbols,init_printing
from sympy.algebras.quaternion import Quaternion
import numpy as np
import sys
sys.path.append("../tools")
from vis import Visualizer
%matplotlib notebook
init_printing() # Para visualizar simbolos
# Definicion del modelo para el ejemplo de atraccion de parque de diversiones
a=ReferenceFrame('A')
#Defina el punto O
o=Point('O')
# Parametros del modelo
l,r,h=symbols('L,R,h')
# Variables de movimiento
q1,q2=dynamicsymbols('q1,q2')
# Marco de referencia B
b=a.orientnew('B','Axis',(q1,a.x))
p=o.locatenew('P',l*b.y)
# Marco de referencia C
c=b.orientnew('C','Axis',(q2,b.z))
q=p.locatenew('Q',r*c.y+h*c.z)
# Visualizacion (Incluyendo superficies en stl)

# Construya un objecto de visualizacion con el marco de referencia inercial y punto de origen
vis=Visualizer(a,o)
# Agrege marcos y puntos para ser visualizados (marco,punto,geometria)
vis.add(a,o,shape='assets/Atraccion_Parque_base.stl')
vis.add(b,o,shape='assets/Atraccion_Parque_link1.stl')
vis.add(c,p,shape='assets/Atraccion_Parque_disk.stl')
vis.add(a,o,frame_scale=50)
vis.add(b,o,frame_scale=50)
vis.add(c,q,frame_scale=50)

# Modifique las variables de movimiento y verifique el cambio en la posicion y
# orientacion de los marcos
vis.plot({l:35,r:30,h:12,q1:0.5,q2:0})
```

    <IPython.core.display.Javascript object>

```python
#Modifique las variables de movimiento para cambiar la configuracion cinematica del modelo
#Puede hacerlo desde esta celda para actualizar el modelo sin tener que volver a generar la figura.
vis.plot({l:35,r:30,h:12,q1:0.5,q2:1.2})
```

```python
#Esto permite crear animaciones facilmente
#Ejecute esta celda y de click en la figura 1 para observar la animacion del movimiento.
import matplotlib
from matplotlib.animation import FuncAnimation
tt=np.linspace(0,10,500)
qq1=np.sin(2*np.pi*0.1*tt)
qq2=np.linspace(0,np.pi*2*10,len(tt))

def animfunc(i,qq1,qq2):
    vis.plot({l:35,r:30,h:12,q1:qq1[i],q2:qq2[i]})

FuncAnimation(vis.fig,animfunc,fargs=(qq1,qq2),interval=20)
```

    <matplotlib.animation.FuncAnimation at 0x15a6062bc40>


# Numerical Solution of Ramsey/Cass-Koopmans model

[](https://mybinder.org/v2/gh/llorracc/Jupyter/master?filepath=notebooks%2FRamseyCassKoopmans.ipynb)

## by [Mateo Velásquez-Giraldo](https://github.com/Mv77)

This notebook implements a class representing Ramsey's growth model. Current capacities include:
- Numerically finding the consumption rule using the 'time elimination' method as originally implemented by Alexander Tabarrok and updated by Christopher D. Carroll in this [Wolfram Mathematica notebook](www.econ2.jhu.edu/people/ccarroll/public/LectureNotes/Growth/RamseyNumericSolve.zip)
- Drawing the phase diagram of the model.
- Simulating optimal dynamics from a given starting point.

A formal treatment of the exact version of the model implemented in this notebook can be found in [Christopher Carroll's graduate macroeconomics lecture notes](http://www.econ2.jhu.edu/people/ccarroll/public/LectureNotes/Growth/RamseyCassKoopmans/).

## Time Elimination

The 'time elimination' method is a straightforward way to obtain the slope of the saddle path of the model (the 'consumption function').

The idea is simple.  The model has two dynamic equations:
   * $dc/dt$ is given by the consumption Euler equation
   * $dk/dt$ is given by the dynamic budget constraint

To produce a phase diagram $\{k,c\}$ space, though, we need $dc/dk$.  The 'time elimination' method is simply to note that

$\left(\frac{dc/dt}{dk/dt}\right)=\left(\frac{dc}{dk}\right)$

If we begin with some point that is on the saddle path (the consumption "policy function"), the equation for $dc/dk$ will tell us how to construct the function at other points along the saddle path

```python
# Setup
import numpy as np
import matplotlib.pyplot as plt

from scipy.integrate import odeint
from scipy import interpolate

from numpy import linalg as LA

# This is a jupytext paired notebook
# which can be executed from a terminal command line via "ipython [notebookname].py"
# But a terminal does not permit inline figures, so we need to test jupyter vs terminal
# Google "how can I check if code is executed in the ipython notebook"
def in_ipynb():
    try:
        if str(type(get_ipython())) == "<class 'ipykernel.zmqshell.ZMQInteractiveShell'>":
            return True
        else:
            return False
    except NameError:
        return False

# Determine whether to make the figures inline (for spyder or jupyter)
# vs whatever is the automatic setting that will apply if run from the terminal
if in_ipynb():
    # %matplotlib inline generates a syntax error when run from the shell
    # so do this instead
    get_ipython().run_line_magic('matplotlib', 'inline')
else:
    get_ipython().run_line_magic('matplotlib', 'auto')

# Import the plot-figure library matplotlib

import matplotlib
import matplotlib.pyplot as plt
matplotlib.rcParams['text.usetex'] = False
```

```python
# Implement the RCKmod class

class RCKmod:
    """
    A class representing Ramsey/Cass-Koopmans growth models.
    """

    def __init__(self,rho,alpha,theta,xi,delta,phi):
        """
        Inputs:
        - rho:   relative risk aversion coefficient for CRRA utility.
        - alpha: capital's share of production in Cobb-Douglas output function.
        - theta: time preference/discount rate.
        - xi:    population growth rate.
        - delta: capital depreciation rate.
        - phi:   labor productivity growth rate.
        """
        # Assign parameter values
        self.rho = rho
        self.alpha = alpha
        self.theta = theta
        self.xi = xi
        self.delta = delta
        self.phi = phi

        # Create empty consumption function
        self.cFunc = None

        # Maximum capital
        self.kmax = (1/(self.phi + self.xi + self.delta))**(1/(1-self.alpha))

        # Steady state capital
        self.kss = (alpha/(theta + xi + delta + rho*phi))**(1/(1-alpha))
        # Steady state consumption
        self.css = self.kss**alpha - (xi + delta + phi)*self.kss

        # Solve the model to create its consumption function
        self.solve()

    def output(self,k):
        """
        Cobb-Douglas normalized production function
        """
        return(k**self.alpha)

    def dcdt(self,c,k):
        """
        Consumption differential equation
        """
        dc = c/self.rho*(self.alpha*k**(self.alpha - 1) - self.theta -\
                         (self.xi + self.delta) -self.rho*self.phi)
        return(dc)

    def dkdt(self,c,k):
        """
        Capital differential equation
        """
        dk = self.output(k) - c - (self.phi + self.xi + self.delta)*k
        return(dk)

    def dcdk(self,c,k):
        """
        Differential equation for the time elimination method.
        This corresponds to dc/dk = (dc/dt)/(dk/dt)
        """
        return(self.dcdt(c,k)/self.dkdt(c,k))

    def solve(self, eps = 10**(-8), npoints = 400):
        """
        Solves for the model's consumption rule through the time elimination
        method.

        Parameters:
        - eps:     disturbance used to prevent dc/dk from becoming 0/0 at
                   the steady state value of capital.
        - npoints: number of points used on each side of the steady
                   state capital for solving the dc/dk equation.
        """
        # K ranges, avoiding kss through a small disturbance
        k_below = np.linspace(self.kss-eps,0.0001,npoints)
        k_above = np.linspace(self.kss+eps,self.kmax,npoints)
        k = np.concatenate((k_below,k_above)).flatten()

        # Solve for c on each side of the steady state capital,
        # Using the slope of the saddle path to approximate initial
        # conditions
        c_below = odeint(self.dcdk,
                         self.css - eps*self.slope_ss(), k_below)
        c_above = odeint(self.dcdk,
                         self.css + eps*self.slope_ss(), k_above)

        c = np.concatenate((c_below,c_above)).flatten()

        # Create consumption function as an interpolation of the
        # numerical solutions.
        self.cFunc = interpolate.interp1d(k,c)

    def dkdt_opt(self,k,t):
        """
        Differential equation for k assuming optimal c decisions.
        """
        return(self.dkdt(self.cFunc(k),k))

    def k_dynamics(self,k0,t):
        """
        Simulates optimal capital dynamics from a given starting point.
        Parameters:
        - t : vector of time points at which to solve for capital
        - k0: value of capital at t[0]
        """
        k = odeint(self.dkdt_opt, k0, t)
        return(k)

    def k0locus(self,k):
        """
        Returns the consumption value that leaves a given ammount of
        effective capital unchanged.
        """
        return(self.output(k) - (self.phi + self.xi + self.delta)*k)

    def phase_diagram(self, npoints = 200, arrows = False, n_arrows = 5, labels = True, legend = True):
        """
        Plots the model's phase diagram.
        - npoints:  number of ticks in the k axis.
        - arrows:   boolean to indicate whether or not to draw arrow
                    grid.
        - n_arrows: controls the number of arrows in the grid
        """

        k = np.linspace(0.01,self.kmax,npoints)

        # Create plot
        plt.figure()

        # Plot k0 locus
        plt.plot(k,self.k0locus(k),label = '$\\dot{k}=0$ locus')
        # Plot c0 locus
        plt.axvline(x = self.kss,linestyle = '--',
                    label = '$\\dot{c}=0$ locus')
        # Plot saddle path
        plt.plot(k,self.cFunc(k), label = 'Saddle path')
        # Plot steady state
        plt.plot(self.kss,self.css,'*r', label = 'Steady state')

        # Add arrows ilustrating behavior in different parts of
        # the diagram.
        # Taken from:
        # http://systems-sciences.uni-graz.at/etextbook/sw2/phpl_python.html
        if arrows:
            x = np.linspace(k[0],k[-1],n_arrows)
            y = np.linspace(self.cFunc(k[0]),self.cFunc(k[-1]),n_arrows)

            X, Y = np.meshgrid(x,y)
            dc = self.dcdt(Y,X)
            dk = self.dkdt(Y,X)

            M = (np.hypot(dk, dc))
            M[ M == 0] = 1.
            dk /= M
            dc /= M
            plt.quiver(X, Y, dk, dc, M, pivot='mid', alpha = 0.3)

        # Labels
        if labels:
            plt.title('Phase diagram and consumption rule\n(normalized by efficiency units)')
            plt.xlabel('k')
            plt.ylabel('c')
        if legend:
            plt.legend()

    def J_matrix(self,c,k):
        """
        Returns the matrix of first derivatives of the solution's dynamic system
        evaluated at the point (c,k).
        This matrix is used for linear approximations of the system around point
        (c,k).
        """

        J = np.array([[1/self.rho*(self.alpha*k**(self.alpha - 1)-\
                                   self.theta-self.xi-self.delta-self.phi),\
                       c/self.rho*\
                       self.alpha*(self.alpha - 1)*k**(self.alpha - 2)],
                      [-1,
                       self.alpha*k**(self.alpha-1) -\
                       (self.phi + self.xi +self.delta)]])

        return(J)

    def slope_ss(self):
        """
        Finds the slope of the saddle path at the steady state.
        """
        J = self.J_matrix(self.css,self.kss)

        # Find eigenvalues and eigenvectors
        w, v = LA.eig(J)
        # Find position of smallest eigenvalue
        min_eig = np.argsort(w)[0]

        # The slope of the saddle path is that
        # generated by the eigenvector of the
        # negative eigenvalue.
        slope = v[0,min_eig]/v[1,min_eig]

        return(slope)
```

## _Example_

This is a quick example of how the class is used.

An instance of the model is first created by assigning the required parameter values.

The model needs to be solved in order to find the consumption rule or 'saddle path'.

```python
# Create and solve model
RCKmodExample = RCKmod(rho = 2,alpha = 0.3,theta = 0.02,xi = 0.01,
                       delta = 0.08,phi = 0.03)
RCKmodExample.solve()

# Test the consumption rule
print('Consumption at k = %1.2f is c = %1.2f'\
      % (RCKmodExample.kss/2, RCKmodExample.cFunc(RCKmodExample.kss/2)))
```

    Consumption at k = 1.13 is c = 0.75

The model's phase diagram can then be generated.

```python
RCKmodExample.phase_diagram(arrows= True, n_arrows = 12)
```

The $\texttt{RCKmod}$ class can also be used to simulate the dynamics of capital given a starting point.

```python
# Create grid of time points
t = np.linspace(0,100,100)

# Find capital dynamics at the desired time points and with
# a given starting capital
k0 = 4
k = RCKmodExample.k_dynamics(k0,t)

# Plot
plt.figure()
plt.plot(t,k)
plt.axhline(y = RCKmodExample.kss,linestyle = '--',color = 'k',
            label = '$\\bar{k}$')
plt.title('Capital')
plt.xlabel('Time')
plt.legend()
plt.show()
```

With capital, the consumption rule can be used to find the dynamics of consumption.

```python
# Find consumption
c = RCKmodExample.cFunc(k)

# Plot
plt.figure()
plt.plot(t,c)
plt.axhline(y = RCKmodExample.css,linestyle = '--',color = 'k',
            label = '$\\bar{c}$')
plt.title('Consumption')
plt.xlabel('Time')
plt.legend()
plt.show()
```

# Appendix 1: finding the slope of the saddle path at the steady state

From the solution of the model, we know that the system of differential equations that describes the dynamics of $c$ and $k$ is

\begin{align}
\begin{bmatrix}
\dot{c_t}\\
\dot{k_t}
\end{bmatrix}
=
\begin{bmatrix}
f(c_t,k_t)\\
g(c_t,k_t)
\end{bmatrix}
=
\begin{bmatrix}
\frac{c_t}{\rho}(\alpha k_t^{\alpha - 1} - \theta - \xi - \delta) - \phi\\
k_t^\alpha - c_t - (\phi + \xi + \delta)*k_t
\end{bmatrix}
\end{align}

We seek to approximate this system around the steady state $(\bar{c},\bar{k})$ through

\begin{align}
\begin{bmatrix}
\dot{c_t}\\
\dot{k_t}
\end{bmatrix}
\approx
\begin{bmatrix}
f(\bar{c},\bar{k}) + f_c(\bar{c},\bar{k})(c_t - \bar{c}) + f_k(\bar{c},\bar{k})(k_t - \bar{k})\\
g(\bar{c},\bar{k}) + g_c(\bar{c},\bar{k})(c_t - \bar{c}) + g_k(\bar{c},\bar{k})(k_t - \bar{k})
\end{bmatrix}
\end{align}

For this we find the system's matrix of first derivatives

\begin{align}
J(c,k) =
\begin{bmatrix}
f_c(c,k) & f_k(c,k)\\
g_c(c,k) & g_k(c,k)
\end{bmatrix} =
\begin{bmatrix}
\frac{1}{\rho}(\alpha k^{\alpha - 1} - \theta - \xi - \delta) - \phi & \frac{c}{\rho}\alpha (\alpha-1) k^{\alpha - 2}\\
-1 & \alpha k^{\alpha - 1} - (\phi + \xi + \delta)
\end{bmatrix}
\end{align}

Given the saddle-path stability of the system, $J(c_{ss},k_{ss})$ will have a positive and a negative eigenvalue. The slope of the saddle path at the steady state capital is given by the eigenvector associated with the negative eigenvalue.

To understand why, rewrite the ODE system as

\begin{align}
\begin{bmatrix}
\dot{c_t}\\
\dot{k_t}
\end{bmatrix}
\approx
\begin{bmatrix}
f(\bar{c},\bar{k})\\
g(\bar{c},\bar{k})
\end{bmatrix}+ J(\bar{c},\bar{k})\times
\begin{bmatrix}
c_t - \bar{c}\\
k_t - \bar{k}
\end{bmatrix}.
\end{align}

Since $(\bar{c},\bar{k})$ are the steady state values, we know $f(\bar{c},\bar{k}) = g(\bar{c},\bar{k}) = 0$. Now define $\hat{c} = c - \bar{c}$ and $\hat{k} = k - \bar{k}$, and notice $\dot{c} = \dot{\hat{c}}$ and $\dot{k} = \dot{\hat{k}}$. With these facts and writting $J(\bar{c},\bar{k}) = J$ to reduce clutter, we have

\begin{align}
\begin{bmatrix}
\dot{\hat{c}_t}\\
\dot{\hat{k}_t}
\end{bmatrix}
\approx
J\times
\begin{bmatrix}
\hat{c}_t\\
\hat{k}_t
\end{bmatrix},
\end{align}

which is a first-order homogeneous ODE system (see ['Matrix differential equation' in Wikipedia](https://en.wikipedia.org/wiki/Matrix_differential_equation)). As discussed in the Wikipedia entry, these systems have solutions of the form

\begin{align}
\begin{bmatrix}
\hat{c}_t\\
\hat{k}_t
\end{bmatrix}
=
c_1 e^{\lambda_1 t}\mathbf{u_1} + c_2 e^{\lambda_2 t}\mathbf{u_2}
\end{align}

where $c_1$ and $c_2$ are constants, $\lambda_1$ and $\lambda_2$ are the eigenvalues of $J$, and $\mathbf{u_1}$ and $\mathbf{u_2}$ are their associated eigenvectors.

For this model, it can be shown that (for all reasonable parameter values) $J$ has a positive and a negative eigenvalue. Thus, assume $\lambda_1 < 0$ and $\lambda_2 > 0$.

We are interested in solutions that approximate to the steady state ($\hat{c}_t = \hat{k}_t =0$) as $t \rightarrow \infty$. For these solutions, we must set $u_2 = 0$: else, given $\lambda_2 > 0$, $e^{\lambda_2 t}\rightarrow \infty$ as $t \rightarrow \infty$ and the system diverges. Therefore, we are left with a solution of the type

\begin{align}
\begin{bmatrix}
\hat{c}_t\\
\hat{k}_t
\end{bmatrix}
=
c_1 e^{\lambda_1 t}\mathbf{u_1} =
\begin{bmatrix}
c_1 e^{\lambda_1 t} u_{1,1}\\
c_1 e^{\lambda_1 t} u_{1,2}
\end{bmatrix}.
\end{align}

Note that, given $\lambda_1<0$, as $t \rightarrow \infty$, $e^{\lambda_1 t}\rightarrow 0$ and $[\hat{c}_t,\hat{k}_t] = [0,0]$ which is precisely what we require.

From the previous solution, we know that in our linear approximation of the dynamic system around $[\hat{c}_t, \hat{k}_t] = [0,0]$, the ratio $\hat{c}_t/\hat{k}_t$ will be the constant $u_{1,1}/u_{1,2}$. Therefore, we can conclude that the slope of the tangent to the saddle path (in k-c coordinates) at the steady state capital $\bar{k}$ will be exactly $u_{1,1}/u_{1,2}$ where $\mathbf{u_1}$ is the eigenvector associated with the negative eigenvalue of the Jacobian matrix J.  This corresponds to the "time elimination" step, because $u_{1,1}$ corresponds to $dc/dt$ and $u_{2,1}$ corresponds to $dk/dt$, so $u_{1,1}/u_{1,2}$ corresponds to $(dc/dt)/(dk/dt)=dc/dk$, which is the slope of the consumption function at the steady state.

# Appendix 2: Figures for Christopher D. Carroll's lecture notes

```python
# Figure RamseySSPlot

labels = ['$\phi$ low','$\phi$ high']
colors = ['red','blue']
g = [0.01,0.1]
npoints = 100

plt.figure()
for i in range(len(g)):

    # Create model
    RCKmodExample = RCKmod(rho = 2,alpha = 0.3,theta = 0.02,xi = 0.01,
                           delta = 0.08,phi = g[i])

    # K range
    k = np.linspace(0,RCKmodExample.kmax,npoints)

    # Plot k0 locus
    plt.plot(k,RCKmodExample.k0locus(k),color = colors[i])

    k_dummy = (RCKmodExample.kmax + RCKmodExample.kss) / 2

    plt.annotate('$\\dot{k}=0$, ' + labels[i],
                 xy=(k_dummy, RCKmodExample.k0locus(k_dummy)),
                 xytext=(k_dummy, RCKmodExample.k0locus(k_dummy)*1.3),
                 arrowprops = dict(color = 'black', width = 0.05,
                                   headwidth = 5, headlength = 7))
    # Plot c0 locus
    plt.axvline(x = RCKmodExample.kss,linestyle = '--',color = colors[i])

    plt.annotate('$\\dot{c} = 0$, ' + labels[i],
                 xy=(RCKmodExample.kss, RCKmodExample.css/3),
                 xytext=(RCKmodExample.kss + 2, RCKmodExample.css/3),
                 arrowprops = dict(color = 'black', width = 0.05,
                                   headwidth = 5, headlength = 7))

# Labels
plt.title('$\\dot{c}/c = 0$ and $\\dot{k} = 0$ Loci')
plt.xlabel('k')
plt.ylabel('c')
fig = plt.gcf() # Get the figure in order to save it
fig.savefig('./RamseyCassKoopmans-Figures/RamseySSPlot.svg')
fig.savefig('./RamseyCassKoopmans-Figures/RamseySSPlot.png')
fig.savefig('./RamseyCassKoopmans-Figures/RamseySSPlot.pdf')

plt.show()
```

```python
# Figure RamseySaddlePlot
npoints = 100

# Create and solve model
RCKmodExample = RCKmod(rho = 2,alpha = 0.3,theta = 0.02,xi = 0.01,
                       delta = 0.08,phi = 0.03)
RCKmodExample.solve()
pd = RCKmodExample.phase_diagram(arrows= True, n_arrows = 12, labels = False)

# Set initial k for off-path trajectories
k0 = RCKmodExample.kss * 0.35

# Find values of c: one too high, one too low
c_high = RCKmodExample.cFunc(k0) * 1.25
c_low = RCKmodExample.cFunc(k0) * 0.75
init_cs = [c_low, c_high]

# Trajectories with high consumption values become problematic much faster
# than those with low consumption, thus, different time intervals are used
t_final = [20,3.4602]

for i in range(len(init_cs)):

    # Create dynamic system for c and k
    dck_dt = lambda ck, t : [RCKmodExample.dcdt(ck[0],ck[1]),
                             RCKmodExample.dkdt(ck[0],ck[1])]

    # Create time vector
    t = np.linspace(0,t_final[i],npoints)

    # Solve for c and k
    ck = odeint(dck_dt, [init_cs[i], k0], t)

    # Plot the trajectory in k-c space
    plt.plot(ck[:,1], ck[:,0], 'k')

plt.title('Transition to the Steady State')
plt.xlabel('k')
plt.ylabel('c')
fig = plt.gcf() # Get the figure in order to save it
fig.savefig('./RamseyCassKoopmans-Figures/RamseySaddlePlot.svg')
fig.savefig('./RamseyCassKoopmans-Figures/RamseySaddlePlot.png')
fig.savefig('./RamseyCassKoopmans-Figures/RamseySaddlePlot.pdf')
plt.show()
```


```python
from sklearn.linear_model import LinearRegression
from sklearn.datasets import make_regression
import numpy as np
from sklearn.svm import SVR
from sklearn import tree
from sklearn.model_selection import  train_test_split
from sklearn.preprocessing import PolynomialFeatures
%matplotlib inline
from matplotlib import pyplot as plt

```

# Q1.
## Q1.1
Using make_regression to generate data, create and train the following models:
- Linear Regression
- Support Vector Regression ([see here](https://scikit-learn.org/stable/modules/svm.html#regression))
- Decision Tree Regression ([see here](https://scikit-learn.org/stable/modules/tree.html#regression))

Make sure to use the same data for each model. Plot the data labels and overlay model predictions. Which model performs better? For the SVR, change the `kernel` parameter between 'rbf','linear', and 'poly'.

## Q1.2
Create a new function, similar to "create_data" in Section 1 exercises. Instead of a weighted sum, implement the following:

\begin{equation}
y = A*sin(f x) + V*noise
\end{equation}

The function should meet the following specifications:
- x is a user-specified [n by 1] array, where n is the number of elements.
- A is a user-specified scalar. It corresponds to the amplitude of the sine.
- f is a user-specified scalar. It corresponds to the frequency of the sine.
- V is a user-specified sclar. It corresponds to the variance of the noise.
- noise is a normally-distributed random value added to each sample.
- y is the returned [n by 1] array.

Re-initialize models from Q1.1 and train them on (x, y). Plot the data labels and overlay predictions. Which models perform better?
Hint: numpy.sin ; numpy.random.randn

## Q1.3
Using the noisy sine function from Q1.2, generate new data. Use `PolynomialFeatures` as we did in Section 1 to create powers of our features (`x`). Training the models from Q1.2 using our new `x_poly` features, how do the models trained on `x_poly` compare to the ones trained on `x` in Q1.2?

```python
# Generate new data using make_regression
'''
x_data, y_label, coef = make_regression(n_samples=500, n_features=10, n_informative=7,
                                        noise=50, coef=True, random_state=50)

x_train, x_test, y_train, y_test =  train_test_split(x_data, y_label, random_state=42)
print(x_train.shape)
print(x_test.shape)
'''

def create_data(X_n_1, A, f, V):
    y_list = []
    for i in X_n_1:
        rand = numpy.random.randn()
        sin = numpy.sin(i*f)
        y =   A*sin + V*rand
        y_list.append(y)
    return y_list

x_data = list(numpy.random.random((500,1)))
A = 5
f = 2
V = 3
y_label = create_data(x_data, A, f, V)
assert len(y_label) == len(x_data)

poly_feat_maker = PolynomialFeatures(degree=3)  # Initialize object
x_data = poly_feat_maker.fit_transform(x_data)  # Apply transformation

plt.plot(x_data[:,0], 'k')
plt.plot(x_data[:,1], 'b')
plt.plot(x_data[:,2], 'r')
plt.plot(x_data[:,3], 'y')
plt.legend(['degree 0','degree 1 ','degree 2','degree 3'])

x_train, x_test, y_train, y_test =  train_test_split(x_data, y_label, random_state=42)

print(len(x_train))
print(len(x_test))
print(len(y_train))
print(len(y_test))
```

```python
# Plot data using different colors
plt.figure()
plt.plot(x_train, y_train, '.', color='blue')
plt.plot(x_test, y_test, '.', color='red')
plt.xlabel('feature value')
plt.ylabel('label value')
```

```python
mdl = LinearRegression()
mdl.fit(x_train, y_train)
print(mdl.coef_)
print(mdl.intercept_)
plt.figure()
plt.plot(x_train, y_train,'.')
```

```python
import sklearn.metrics

y_train_pred = mdl.predict(x_train)
r2 = sklearn.metrics.r2_score(y_train, y_train_pred)
mse = sklearn.metrics.mean_squared_error(y_train, y_train_pred)
print("Trained")
print(r2)
print(mse)
print()
plt.figure()
plt.plot(y_train_pred, y_train,'.')
print()

y_test_pred = mdl.predict(x_test)
r2 = sklearn.metrics.r2_score(y_test, y_test_pred)
mse = sklearn.metrics.mean_squared_error(y_test, y_test_pred)
print("Predicted")
print(r2)
print(mse)

plt.figure()
plt.plot(y_test_pred, y_test,'.')
print()
```

```python
svr_rbf = SVR(kernel='rbf', C=100, gamma=0.1, epsilon=.1)
svr_lin = SVR(kernel='linear', C=100, gamma='auto')
svr_poly = SVR(kernel='poly', C=100, gamma='auto', degree=3, epsilon=.1, coef0=1)
svr_models = [svr_rbf, svr_lin, svr_poly]
```

```python
for m in svr_models:
    m.fit(numpy.array(x_train), numpy.array(y_train).ravel())

    r2 = sklearn.metrics.r2_score(y_train, y_train_pred)
    mse = sklearn.metrics.mean_squared_error(y_train, y_train_pred)
    print("Trained")
    print(r2)
    print(mse)
    plt.figure()
    plt.plot(y_train_pred, y_train,'.')

    y_pred_test = m.predict(x_test)
    r2 = sklearn.metrics.r2_score(y_test, y_pred_test)
    mse = sklearn.metrics.mean_squared_error(y_test, y_pred_test)
    print("Predicted")
    print(r2)
    print(mse)
    print()
    plt.figure()
    plt.plot(y_pred_test,y_test,'.')

```

```python

```


```python
import sys

import numpy as np
import matplotlib.pyplot as plt

sys.path.append('../src')
import utils
```

# Machine Learning Models

Many problems in machine learing seek to build a model

$$g(a; x) \approx y$$

given a data set

$$\{(a_1, y_1), \dots, (a_m, y_m)\},$$

with components
* $a_i \in \mathbf{R}^n$ - data features,
* $y_i \in \mathbf{R}$ or $\{0, 1\}$ - data value or class,
* $g: \mathbf{R}^n \to \mathbf{R}$ or $\{0, 1\}$ - prediction function,
* $x \in \mathbf{R}^n$ - model parameters,
* $m$ - number of data points, and
* $n$ - number of data features.

We can fit a model to the given data by solving an optimization problem of the form

$$\min_x \sum_{i=1}^m f_i(g(a_i; x), y_i) + r(x)$$

with components
* $x \in \mathbf{R}^n$ - model parameters,
* $f_i: \mathbf{R}^n \to \mathbf{R}$ - functions that measure how well the model fits the data for a given set of parameters, and
* $r(x): \mathbf{R}^n \to \mathbf{R}$ - regularization function.

# Linear Regression

In the linear regression problem, we would like to find a linear predictor

$$g(a_i; x) = x_1 a_{i1} + \dots + x_n a_{in} = a_i^T x \approx y_i,$$

where both $a_i$ and $y_i$ are continuous.
One approach for deriving the [loss functions](https://en.wikipedia.org/wiki/Loss_function) $f_i$ is to assume a statistical model for the error in the data set, and then develop a [maximum likelihood](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation) formulation.
Here we will assume that errors are independently drawn from a normal distribution with mean zero and variance $\sigma^2$, that is,

$$y_i = a_i^Tx + \epsilon_i, \quad \epsilon_i \sim \mathcal{N}(0, \sigma^2).$$

This means that the probability density function for an observation $(a_i, y_i)$ given model parameters $x$ is

$$p((a_i, y_i); x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(y_i - a_i^Tx)^2}{2\sigma^2}\right),$$

and the PDF of all $m$ i.i.d. observations is

\begin{align}
p\big(\{(a_1, y_1), \dots, (a_m, y_m)\}; x\big) &= \prod_{i=1}^m p((a_i, y_i); x) \\
&= \prod_{i=1}^m \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(y_i - a_i^Tx)^2}{2\sigma^2}\right) \\
&= \left(\frac{1}{2\pi\sigma^2}\right)^{m/2} \exp\left(-\frac{\sum_{i=1}^m (y_i - a_i^Tx)^2}{2\sigma^2}\right).
\end{align}

Alternatively, we can consider the likelihood of a set of model parameters given our $m$ observations,

$$\mathcal{L}\big(x; \{(a_1, y_1), \dots, (a_m, y_m)\}\big) = p\big(\{(a_1, y_1), \dots, (a_m, y_m)\}; x\big),$$

so that we can solve an optimization problem to find the parameters with the maximum likelihood. In practice, this is often done by minimizing the negative log-likelihood, which, ignoring the constant coefficient, results in our linear least-squares problem

$$\min_x \sum_{i=1}^m(y_i - a_i^Tx)^2.$$

# Regularization

Many problems in machine learning add a [regularization](https://en.wikipedia.org/wiki/Regularization_(mathematics)) term $r(x)$ to the objective function as a means of
* improving the condition number of ill-posed problems,
* incorporating prior knowledge (e.g., smoothness or sparsity), or
* preventing [overfitting](https://en.wikipedia.org/wiki/Overfitting).

Here are some examples of commonly used regularization functions:
* [Lasso](https://en.wikipedia.org/wiki/Lasso_(statistics)) - $\lambda \|x\|_1$
* [Ridge regression (Tikhonov regularization)](https://en.wikipedia.org/wiki/Tikhonov_regularization) - $\lambda \|x\|_2^2$
* [Elastic net](https://en.wikipedia.org/wiki/Elastic_net_regularization) - $\lambda_1 \| x \|_1 + \lambda_2 \|x\|_2^2$
* [Total variation](https://en.wikipedia.org/wiki/Total_variation_denoising) - $\lambda \|x\|_{TV}$
* [Early stopping](https://en.wikipedia.org/wiki/Early_stopping)

Here we will focus on the Lasso problem

$$\min_x \frac{1}{2} \|Ax - y\|_2^2 + \lambda \|x\|_1,$$

which is used to find sparse solutions (e.g., few nonzeros in $x$).

# Proximal Gradient Descent

Because the 1-norm isn't smooth (the absolute value function isn't differentiable at zero), we need to modify the gradient descent algorithm before we can use it to solve the Lasso problem.
One approach is to combine the gradient step of the differentiable least-squares term with the [proximal operator](https://en.wikipedia.org/wiki/Proximal_operator)

$$\text{prox}_r(x) = \text{argmin}_y \frac{1}{2} \| y - x \|_2^2 + r(y),$$

which generalizes the projection operator.
For input $x$, the prox computes output $y$ that is both small with respect to the regularizer, while at the same time being close to $x$.
When $r$ is the [0-$\infty$ indicator function](https://en.wikipedia.org/wiki/Characteristic_function_(convex_analysis)) of a set, the prox returns the projection of $x$ onto the set.

For an optimization problem

$$\min_x f(x) + r(x),$$

where $f$ is convex and differentiable, and $r$ is convex but possibly not differentiable, the [proximal gradient descent](https://en.wikipedia.org/wiki/Proximal_gradient_methods_for_learning) algorithm with initial guess $x^0$ and step size $\alpha$ is:

> **input** $x^0$, $\alpha$
> **set** $k = 0$
> **while** not converged **do**
>     $x^{k+1} \leftarrow \text{prox}_{\lambda r}\Big(x^k - \alpha \nabla f(x^k)\Big)$
>     $k \leftarrow k + 1$
> **end while**

The prox of the 1-norm is known as the soft-thresholding operator, and can be computed elementwise:

$$\Big(\text{prox}_{\lambda \|\cdot\|_1}(x)\Big)_i = \begin{cases} x_i - \lambda & x_i > \lambda \\ 0 & |x_i| \leq \lambda \\ x_i + \lambda & x_i < -\lambda \end{cases}$$

Proximal gradient descent for 1-norm regularized problems is also refered to as an iterative soft-thresholding algorithm or an iterative shrinkage-thresholding algorithm (ISTA).
As seen in the plot below, the soft-thresholding operator encourages sparsity by setting small elements of the input to zero, while reducing the magnitude of the rest.
This effect becomes more pronounced as the parameter $\lambda$ is increased.

```python
x = np.linspace(-5, 5, 100)
plt.plot(x, x, '--')
plt.plot(x, utils.prox(x, 1))
plt.plot(x, utils.prox(x, 2))
plt.plot(x, utils.prox(x, 3))
plt.xlabel('$x$')
plt.ylabel('$S_{\lambda}(x)$')
plt.title('Soft-thresholding Operator')
plt.legend(['$x$', '$\lambda = 1$', '$\lambda = 2$', '$\lambda = 3$']);
```

# Exercise 1: Sparse Signal Recovery

Suppose we want to solve a linear regression problem where we know that the true solution should be sparse.
Without any regularization ($\lambda = 0$), we are able to find a low function value, but we cannot correctly identify the nonzero model parameters. In the example below, experiment with the size of $\lambda$ to answer the following questions:
* Using regularization parameters 0, 0.1, 1, 10, and 100, fill out the table below. Can you find a relationship between $\lambda$, the minimum function value, and the number of nonzero elements in the solution?

$\lambda$ | $f(x)$ | # nonzeros
-|-|-
0 |  |
0.01 |  |
0.1 |  |
1 |  |
10 |  |
100 |  |

* From the table, which value of $\lambda$ did the best job at approximating the true number (14) of nonzero elements in the solution?
* From the plot, which value of $\lambda$ did the best job at approximating the true value of the nonzero elements in the solution?

```python
m = 50; n = 128; k = 14
np.random.seed(0)
A = np.random.randn(m,n)
p = np.random.permutation(n)[:k]
x_true = np.zeros(n)
x_true[p] = np.random.randn(k)
y = A.dot(x_true) + 0.1*np.random.randn(A.shape[0])
```

```python
lam = 0 # modify this variable
results = utils.prox_descent(A, y, lam=lam)
print(f'Number of nonzeros: {np.count_nonzero(results[0])}')
utils.plot_pgd1(A, x_true, y, results)
```

Below, we can see the results for a wider range of regularization parameters, with similar results:
* For small $\lambda$, the least-squares term dominates the objective function, so we get low model error but dense solution vectors.
* For large $\lambda$, the regularization term dominates the objective function, so we have sparser solutions but higher model error.

```python
utils.plot_lam1(A, y, x_true, lam_vals=10.**np.arange(-4, 5))
```

The regularization parameter $\lambda$ is an example of a [hyperparameter](https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)), a value related to the learing process rather than the model itself.
We have already seen a few examples of hyperparameters in the previous notebook: the step size in gradient descent, and the mini-batch size and decay parameter for the learning rate schedule in stochastic gradient descent.
While tuning hyperparameters can sometimes feel like an art form, there are a variety of [methods](https://en.wikipedia.org/wiki/Hyperparameter_optimization) that can make the process easier and more systematic.

# Exercise 2: Dictionary Learning

Suppose we would like to fit a nonlinear model to some data, but we don't know which functions to use.
For example, we may suspect that our data was generated by a sum of cosines, but we don't know how many terms to use or their frequencies.
Our goal will be to fit the model

$$g(x_i; a, b) = \sum_{j=1}^k a_j \cos(b_jx_i),$$

for a given set of points $(x_i, y_i)$, where $k$, $a_j$, and $b_j$ are unknown.
Note that in this example, we have changed notation so that scalars $x_i$ are input, while vectors $a$ and $b$ are model parameters.

For this example, we will split our data into a training set $(x_{train}, y_{train})$ and a test set $(x_{test}, y_{test})$.
We will fit our model parameters using the training set, and then we will compare our model error on both the training set and test set.
This will give us an idea of not only how well our model fits the given data, but also how well our model generalizes to new data.

```python
n = 50; m = 25; k = 5
np.random.seed(0)
a = np.random.rand(k); b = np.random.choice(n, k, replace=False)
x_train = np.linspace(0, 2, m)
y_train = utils.g(x_train, a, b) + 0.1*np.random.randn(len(x_train))
x_test = np.linspace(2, 4, m)[1:]
y_test = utils.g(x_test, a, b) + 0.1*np.random.randn(len(x_test))
plt.plot(np.linspace(0, 4, 1000), utils.g(np.linspace(0, 4, 1000), a, b))
plt.plot(x_train, y_train, '.', markersize=10)
plt.plot(x_test, y_test, '.', markersize=10)
plt.xlabel('$x_i$')
plt.ylabel('$y_i$')
plt.legend(['$g(x)$', 'Train', 'Test']);
```

Even though our model $g$ is nonlinear, we can use linear least-squares to estimate parameter vectors $a$ and $b$ by first creating dictionary matrix $D$ that contains cosine evaluated at our data for various candidate frequencies $b_i$:

$$D = \begin{bmatrix}
    \cos(b_1 x_1) & \cos(b_2 x_1) & \dots & \cos(b_n x_1) \\
    \cos(b_1 x_2) & \cos(b_2 x_2) & \dots & \cos(b_n x_2) \\
    \vdots & \vdots & \ddots & \vdots \\
    \cos(b_1 x_m) & \cos(b_2 x_m) & \dots & \cos(b_n x_m)
\end{bmatrix}$$

Now we can evaluate $g$ at our data for estimated coefficients $a_i$ using the matrix-vector product $Da$.

```python
D = np.array([np.cos(ii*x_train) for ii in range(n)]).T
```

Since we expect that $g$ only has a few terms, we will solve the Lasso problem

$$\min_a \frac{1}{2} \|Da - y \|_2^2 + \lambda \| a \|_1$$

to find a sparse vector of coefficients $a_i$.

* Using regularization parameters 0, 0.1, 1, and 10, fill out the table below. Can you find a relationship between $\lambda$, the minimum function value, the function value on the test set, and the number of nonzero elements in the solution?

$\lambda$ | $f(x_{train})$ | $f(x_{test})$ | # terms
-|-|-|-
0 | |  |
0.01 |  |  |
0.1 |  |  |
1 |  |   |
10 |  |  |

* From the table, which value of $\lambda$ did the best job at approximating the true number (5) of terms?
* From the plot, which value of $\lambda$ did the best job at approximating the true value of the nonzero elements in the solution?
* From the plot, which value of $\lambda$ did the best job at predicting new data?

```python
lam = 0 # modify this parameter
results = utils.prox_descent(D, y_train, lam=lam)
f_test = np.linalg.norm(utils.g(x_test, results[0]) - y_test)**2/2 + lam*np.linalg.norm(results[0], 1)
print(f'Function value on test set: {f_test:.2f}')
print(f'Number of terms: {np.count_nonzero(results[0])}')
utils.plot_pgd2(a, b, x_train, y_train, x_test, y_test, results)
```

Below, we can see the results for a wider range of regularization parameters, with similar results:
* For small $\lambda$, the least-squares term dominates the objective function, so we get low model error on the training data but dense coefficient vectors. This is an example of overfitting: by using more terms, our model fits the training data extremely well, but it cannot generalize to new data, so we have high model error on the test data.
* For large $\lambda$, the regularization term dominates the objective function, so we have sparser coefficient vectors but higher model error on both the training and test data. This is an example of underfitting: we don't have enough terms, so our model cannot predict any of our data well.
* The best trade-off occurs where our test error achieves its minimum value.

```python
utils.plot_lam2(D, a, b, x_train, y_train, x_test, y_test, lam_vals=10.**np.arange(-4, 3))
```

# Further Reading

* Beck, A., & Teboulle, M. (2009). "A fast iterative shrinkage-thresholding algorithm for linear inverse problems." *SIAM Journal on Imaging Sciences*, 2(1), 183-202.
* Tibshirani, R. (2011). "Regression shrinkage and selection via the lasso: a retrospective." *Journal of the Royal Statistical Society: Series B (Statistical Methodology)*, 73(3), 273-282.
* Parikh, N., & Boyd, S. (2014). "Proximal algorithms". *Foundations and Trends in optimization*, 1(3), 127-239.
* Boyd, S., & Vandenberghe, L. (2018). *Introduction to Applied Linear Algebra: Vectors, Matrices, and Least Squares*. Cambridge University Press.


Sascha Spors,
Professorship Signal Theory and Digital Signal Processing,
Institute of Communications Engineering (INT),
Faculty of Computer Science and Electrical Engineering (IEF),
University of Rostock,
Germany

# Tutorial Digital Signal Processing

**DFT Fundamentals**,
Winter Semester 2021/22 (Course #24505)

- lecture: https://github.com/spatialaudio/digital-signal-processing-lecture
- tutorial: https://github.com/spatialaudio/digital-signal-processing-exercises

Feel free to contact lecturer frank.schultz@uni-rostock.de

```python
import numpy as np
import matplotlib.pyplot as plt
from numpy.linalg import inv
from numpy.fft import fft, ifft
#from scipy.fft import fft, ifft
```

# Discrete Fourier Transform (DFT)

## Input Signal

Let us first define a **complex-valued signal** $x[k]$ of a certain block length $N$ ranging from $0\leq k\leq N-1$.

The variable `tmpmu` defines the frequency of the signal. We will see later how this is connected to the DFT.
For now on, leave it with `tmpmu=1`. This results in exactly one period of cosine and sine building the complex signal. If `tmpmu=2` we get exactly two periods of cos/sin. We'll get get an idea of `tmpmu`...

```python
N = 2**3  # signal block length
k = np.arange(N)  # all required sample/time indices
A = 10  # signal amplitude

tmpmu = 2-1/2  # DFT eigenfrequency worst case
tmpmu = 1  # DFT eigenfrequency best case

x = A * np.exp(tmpmu * +1j*2*np.pi/N * k)

# plot
plt.stem(k, np.real(x), markerfmt='C0o',
         basefmt='C0:', linefmt='C0:', label='real')
plt.stem(k, np.imag(x), markerfmt='C1o',
         basefmt='C1:', linefmt='C1:', label='imag')
# note that connecting the samples by lines is actually wrong, we
# use it anyway for more visual convenience:
plt.plot(k, np.real(x), 'C0-', lw=0.5)
plt.plot(k, np.imag(x), 'C1-', lw=0.5)
plt.xlabel(r'sample $k$')
plt.ylabel(r'complex-valued input signal $x[k]$')
plt.legend()
plt.grid(True)
```

We will now perform an DFT of $x[k]$ since we are interested in the frequency spectrum of it.

## DFT Definition

The discrete Fourier transform pair for a discrete-time signal $x[k]$ with sample index $k$ and the corresponding DFT spectrum $X[\mu]$ with frequency index $\mu$ is given as
\begin{align}
\text{DFT}: X[\mu]=&\sum_{k=0}^{N-1}x[k]\cdot\mathrm{e}^{-\mathrm{j}\frac{2\pi}{N}k\mu}\\
\text{IDFT}: x[k]=\frac{1}{N}&\sum_{\mu=0}^{N-1}X[\mu]\cdot\mathrm{e}^{+\mathrm{j}\frac{2\pi}{N}k\mu}
\end{align}

Note the sign reversal in the exp()-function and the $1/N$ normalization in the IDFT. This convention is used by the majority of DSP text books and also in Python's `numpy.fft.fft()`, `numpy.fft.ifft()` and Matlab's `fft()`, `ifft()` routines.

## DFT and IDFT with For-Loops

We are now going to implement the DFT and IDFT with for-loop handling. While this might be helpful to validate  algorithms in its initial development phase, this should be avoided for practical used code in the field: for-loops are typically slow and very often more complicated to read than appropriate set up matrices and vectors. Especially for very large $N$ the computation time is very long.

Anyway, the for-loop concept is: the DFT can be implemented with an outer for loop iterating over $\mu$ and an inner for loop summing over all $k$ for a specific $\mu$.

We use variable with _ subscript here, in order to save nice variable names for the matrix based calculation.

```python
# DFT with for-loop:
X_ = np.zeros((N, 1), dtype=complex)  # alloc RAM, init with zeros
for mu_ in range(N):  # do for all DFT frequency indices
    for k_ in range(N):  # do for all sample indices
        X_[mu_] += x[k_] * np.exp(-1j*2*np.pi/N*k_*mu_)
```

IDFT with outer and inner looping reads as follows.

```python
# IDFT with for-loop:
x_ = np.zeros((N, 1), dtype=complex)  # alloc RAM, init with zeros
for k_ in range(N):
    for mu_ in range(N):
        x_[k_] += X_[mu_] * np.exp(+1j*2*np.pi/N*k_*mu_)
x_ *= 1/N  # normalization in the IDFT stage
```

Besides exchanged variables, main differences are sign reversal in exp() and the $1/N$ normalization. This is expected due to the DFT/IDFT equation pair given above.

## DFT and IDFT with Matrix Multiplication

Now we do a little better: We should think of the DFT/IDFT in terms of a matrix operation setting up a set of linear equations.

For that we define a column vector containing the samples of the discrete-time signal $x[k]$
\begin{equation}
\mathbf{x}_k = (x[k=0], x[k=1], x[k=2], \dots , x[k=N-1])^\mathrm{T}
\end{equation}

and a column vector containing the DFT coefficients $X[\mu]$

\begin{equation}
\mathbf{x}_\mu = (X[\mu=0], X[\mu=1], X[\mu=2], \dots, X[\mu=N-1])^\mathrm{T}
\end{equation}

Then, the matrix operations

\begin{align}
\text{DFT:   } & \mathbf{x}_\mu = \mathbf{W}^* \mathbf{x}_k\\
\text{IDFT:   } & \mathbf{x}_k = \frac{1}{N} \mathbf{W} \mathbf{x}_\mu
\end{align}

hold.

$()^\mathrm{T}$ is the transpose, $()^*$ is the conjugate complex.

The $N\times N$ Fourier matrix is defined as (element-wise operation $\odot$)
\begin{equation}
\mathbf{W} = \mathrm{e}^{+\mathrm{j}\frac{2\pi}{N} \odot \mathbf{K}}
\end{equation}
using the so called twiddle factor (note that the sign in the exp() is our convention)
\begin{equation}
W_N = \mathrm{e}^{+\mathrm{j}\frac{2\pi}{N}}
\end{equation}
and the outer product
\begin{equation}
\mathbf{K} =
\begin{bmatrix}
0\\
1\\
2\\
\vdots\\
N-1
\end{bmatrix}
\cdot
\begin{bmatrix}
0 & 1 & 2 & \cdots & N-1
\end{bmatrix}
\end{equation}
containing all possible products $k\,\mu$ in a suitable arrangement.

For the simple case $N=4$ these matrices are
\begin{align}
\mathbf{K} = \begin{bmatrix}
0 & 0 & 0 & 0\\
0 & 1 & 2 & 3\\
0 & 2 & 4 & 6\\
0 & 3 & 6 & 9
\end{bmatrix}
\rightarrow
\mathbf{W} = \begin{bmatrix}
1 & 1 & 1 & 1\\
1 & +\mathrm{j} & -1 & -\mathrm{j}\\
1 & -1 & 1 & -1\\
1 & -\mathrm{j} & -1 & +\mathrm{j}
\end{bmatrix}
\end{align}

```python
# k = np.arange(N)  # all required sample/time indices, already defined above

# all required DFT frequency indices, actually same entries like in k
mu = np.arange(N)

# set up matrices
K = np.outer(k, mu)  # get all possible entries k*mu in meaningful arrangement
W = np.exp(+1j * 2*np.pi/N * K)  # analysis matrix for DFT
```

```python
# visualize the content of the Fourier matrix
# we've already set up (use other N if desired):
# N = 8
# k = np.arange(N)
# mu = np.arange(N)
# W = np.exp(+1j*2*np.pi/N*np.outer(k, mu))  # set up Fourier matrix

fig, ax = plt.subplots(1, N)
fig.set_size_inches(6, 6)
fig.suptitle(
    r'Fourier Matrix for $N=$%d, blue: $\Re(\mathrm{e}^{+\mathrm{j} \frac{2\pi}{N} \mu k})$, orange: $\Im(\mathrm{e}^{+\mathrm{j} \frac{2\pi}{N} \mu k})$' % N)

for tmp in range(N):
    ax[tmp].set_facecolor('lavender')
    ax[tmp].plot(W[:, tmp].real, k, 'C0o-', ms=7, lw=0.5)
    ax[tmp].plot(W[:, tmp].imag, k, 'C1o-.', ms=7, lw=0.5)
    ax[tmp].set_ylim(N-1, 0)
    ax[tmp].set_xlim(-5/4, +5/4)
    if tmp == 0:
        ax[tmp].set_yticks(np.arange(0, N))
        ax[tmp].set_xticks(np.arange(-1, 1+1, 1))
        ax[tmp].set_ylabel(r'$\longleftarrow k$')
    else:
        ax[tmp].set_yticks([], minor=False)
        ax[tmp].set_xticks([], minor=False)
    ax[tmp].set_title(r'$\mu=$%d' % tmp)
fig.tight_layout()
fig.subplots_adjust(top=0.91)

fig.savefig('fourier_matrix.png', dpi=300)

# TBD: row version for analysis
```

## Fourier Matrix Properties

The DFT and IDFT basically solve two sets of linear equations, that are linked as forward and inverse problem.

This is revealed with the important property of the Fourier matrix

\begin{equation}
\mathbf{W}^{-1}
= \frac{\mathbf{W}^\mathrm{H}}{N}
= \frac{\mathbf{W}^\mathrm{*}}{N},
\end{equation}

the latter holds since the matrix is symmetric.

Thus, we see that by our convention, the DFT is the inverse problem (signal analysis) and the IDFT is the forward problem (signal synthesis)

\begin{align}
\text{DFT:   } & \mathbf{x}_\mu = \mathbf{W}^* \mathbf{x}_k \rightarrow \mathbf{x}_\mu = N \mathbf{W}^{-1} \, \mathbf{x}_k\\
\text{IDFT:   } & \mathbf{x}_k = \frac{1}{N} \mathbf{W} \mathbf{x}_\mu.
\end{align}

The occurrence of the $N$, $1/N$ factor is due to the prevailing convention in signal processing literature.

If the matrix is normalised as $\frac{\mathbf{W}}{\sqrt{N}}$, a so called unitary matrix results, for which the
important property
\begin{equation}
(\frac{\mathbf{W}}{\sqrt{N}})^\mathrm{H} \, (\frac{\mathbf{W}}{\sqrt{N}}) = \mathbf{I} =
(\frac{\mathbf{W}}{\sqrt{N}})^{-1} \, (\frac{\mathbf{W}}{\sqrt{N}})
\end{equation}
holds, i.e. the complex-conjugate, transpose is equal to the inverse
$(\frac{\mathbf{W}}{\sqrt{N}})^\mathrm{H} = (\frac{\mathbf{W}}{\sqrt{N}})^{-1}$
and due to the matrix symmetry also
$(\frac{\mathbf{W}}{\sqrt{N}})^* =
(\frac{\mathbf{W}}{\sqrt{N}})^{-1}$
is valid.

This tells that the matrix $\frac{\mathbf{W}}{\sqrt{N}}$ is **orthonormal**, i.e. the matrix spans a orthonormal vector basis (the best what we can get in linear algebra world to work with) of $N$ normalized DFT eigensignals.

So, DFT and IDFT is transforming vectors into other vectors using the vector basis of the Fourier matrix.

## Check DFT Eigensignals and -Frequencies

The columns of the Fourier matrix $\mathbf{W}$ contain the eigensignals of the DFT. These are
\begin{align}
w_\mu[k] = \cos(\frac{2\pi}{N} k \mu) + \mathrm{j} \sin(\frac{2\pi}{N} k \mu)
\end{align}
since we have intentionally set up the matrix this way.

The plot below shows the eigensignal for $\mu=1$, which fits again one signal period in the block length $N$.
For $\mu=2$ we obtain two periods in one block.

The eigensignals for $0\leq \mu \leq N-1$ therefore exhibit a certain digital frequency, the so called DFT eigenfrequencies.

What eigensignal corresponds to $\mu=0$?...

```python
tmpmu = 1  # column index

plt.stem(k, np.real(W[:, tmpmu]), label='real',
         markerfmt='C0o', basefmt='C0:', linefmt='C0:')
plt.stem(k, np.imag(W[:, tmpmu]), label='imag',
         markerfmt='C1o', basefmt='C1:', linefmt='C1:')
# note that connecting the samples by lines is actually wrong, we
# use it anyway for more visual convenience
plt.plot(k, np.real(W[:, tmpmu]), 'C0-', lw=0.5)
plt.plot(k, np.imag(W[:, tmpmu]), 'C1-', lw=0.5)
plt.xlabel(r'sample $k$')
plt.ylabel(r'DFT eigensignal = '+str(tmpmu+1)+'. column of $\mathbf{W}$')
plt.legend()
plt.grid(True)
```

The nice thing about the chosen eigenfrequencies, is that the eigensignals are **orthogonal**.

This choice of the vector basis is on purpose and one of the most important ones in linear algebra and signal processing.

We might for example check orthogonality with the **complex** inner product of some matrix columns.

```python
np.dot(np.conj(W[:, 0]), W[:, 0])  # same eigensignal, same eigenfrequency
# np.vdot(W[:,0],W[:,0])  # this is the suitable numpy function
```

```python
np.dot(np.conj(W[:, 0]), W[:, 1])  # different eigensignals
# np.vdot(W[:,0],W[:,1])  # this is the suitable numpy function
# result should be zero, with numerical precision close to zero:
```

## Initial Example: IDFT Signal Synthesis for N=8

Let us synthesize a discrete-time signal by using the IDFT in matrix notation for $N=8$.

The signal should contain a DC value, the first and second eigenfrequency with different amplitudes, such as

\begin{equation}
\mathbf{x}_\mu = [8, 2, 4, 0, 0, 0, 0, 0]^\text{T}
\end{equation}

using large `X_test` in code.

```python
if N == 8:
    X_test = np.array([8, 2, 4, 0, 0, 0, 0, 0])
    # x_test = 1/N*W@X_test  # >= Python3.5
    x_test = 1/N * np.matmul(W, X_test)

    plt.stem(k, np.real(x_test), label='real',
             markerfmt='C0o', basefmt='C0:', linefmt='C0:')
    plt.stem(k, np.imag(x_test), label='imag',
             markerfmt='C1o', basefmt='C1:', linefmt='C1:')
    # note that connecting the samples by lines is actually wrong, we
    # use it anyway for more visual convenience
    plt.plot(k, np.real(x_test), 'C0o-', lw=0.5)
    plt.plot(k, np.imag(x_test), 'C1o-', lw=0.5)
    plt.xlabel(r'sample $k$')
    plt.ylabel(r'$x[k]$')
    plt.legend()
    plt.grid(True)

    # check if results are identical with numpy ifft package
    print(np.allclose(ifft(X_test), x_test))
    print('DC is 1 as expected: ', np.mean(x_test))
```

This is a linear combination of the Fourier matrix columns, which are the DFT eigensignals, as

```python
if N == 8:
    x_test2 = X_test[0] * W[:, 0] + X_test[1] * W[:, 1] + X_test[2] * W[:, 2]
```

We don't need summing the other columns, since their DFT coefficients in `X_test` are zero.

Finally, normalizing yields the IDFT.

```python
if N == 8:
    x_test2 *= 1/N
    print(np.allclose(x_test, x_test2))  # check with result before
```

## Initial Example: DFT Spectrum Analysis for N=8

Now, let us calculate the DFT of the signal `x_test`. As result, we'd expect the DFT vector

\begin{equation}
\mathbf{x}_\mu = [8, 2, 4, 0, 0, 0, 0, 0]^\text{T}
\end{equation}

that we started from.

```python
if N == 8:
    # X_test2 = np.conj(W)@x_test  # >= Python3.5
    X_test2 = np.matmul(np.conj(W), x_test)  # DFT, i.e. analysis
    print(np.allclose(X_test, X_test2))  # check with result before
```

This looks good. It is advisable also to check against the `numpy.fft` implementation:

```python
if N == 8:
    print(np.allclose(fft(x_test), X_test))
```

Besides different quantization errors in range $10^{-15...-16}$ (which is prominent even with 64Bit double precision calculation)
all results produce the same output.

The analysis stage for the discrete-time signal domain, i.e. the DFT
can be reinvented by some intuition:
How 'much' of the reference signal $\mathbf{w}_{\text{column i}}$
(any column in $\mathbf{W}$)
is contained in the discrete-time signal $\mathbf{x}_k$ that is to be analysed.

In signal processing / statistic terms we look for the amount of correlation
of the signals
$\mathbf{w}_{\text{column i}}$ and $\mathbf{x}_k$.

In linear algebra terms we are interested in the projection of $\mathbf{x}_k$ onto
$\mathbf{w}_{\text{column i}}$, because the resulting length of this vector
reveals the amount of correlation, which is precisely one DFT coefficient $X[\cdot]$.

The complex inner products $\mathbf{w}_{\text{column i}}^\text{H} \cdot \mathbf{x}_k$
reveals these searched quantities.

```python
if N == 8:
    print(np.conj(W[:, 0])@x_test)
    print(np.conj(W[:, 1])@x_test)
    print(np.conj(W[:, 2])@x_test)
```

Doing this for all columns of matrix $\mathbf{W}$, all DFT coefficients are obtained, such as

\begin{align}
X[\mu=0] =& \mathbf{w}_{\text{column 1}}^\text{H} \cdot \mathbf{x}_k\\
X[\mu=1] =& \mathbf{w}_{\text{column 2}}^\text{H} \cdot \mathbf{x}_k\\
X[\mu=2] =& \mathbf{w}_{\text{column 3}}^\text{H} \cdot \mathbf{x}_k\\
X[\mu=3] =& \mathbf{w}_{\text{column 4}}^\text{H} \cdot \mathbf{x}_k\\
&\vdots\\
X[\mu=N-1] =& \mathbf{w}_{\text{column N}}^\text{H} \cdot \mathbf{x}_k.
\end{align}

Naturally, all operations can be merged to one single
matrix multiplication using the conjugate transpose of $\mathbf{W}$.

\begin{equation}
\mathbf{x}_\mu = \mathbf{W}^\text{H} \cdot \mathbf{x}_k = \mathbf{W}^* \cdot \mathbf{x}_k
\end{equation}

That's what we have performed with the single liner `X_test2 = np.matmul(np.conj(W), x_test)`

## Example: Plot the DFT Magnitude Spectrum

We should now be familiar with the DFT and IDFT basic idea.

Now, let us **return to our initially created signal** `x` at the very beginning of this notebook. We want to explore and learn to interpret the DFT magnitude spectrum of it. So, we'd perform a DFT first.

```python
X = fft(x)
# print(np.allclose(np.conj(W)@x, X))  # >=Python 3.5
print(np.allclose(np.matmul(np.conj(W), x), X))
```

Next, let us plot the magnitude of the spectrum over $\mu$.

- We should play around with the variable `tmpmu` when defining the input signal at the very beginning of the notebook. For example we can check what happens for `tmpmu = 1`, `tmpmu = 2` and run the whole notebook to visualize the actual magnitude spectra.

We should recognize the link of the 'energy' at $\mu$ in the magnitude spectrum with the chosen `tmpmu`.

- We can apply any real valued `tmpmu` for creating the input signal, for example
    - `tmpmu = N+1`, `tmpmu = N+2`
    - `tmpmu = 1.5`

We should explain what happens in these cases. Recall periodicity and eigenfrequencies/-signals as fundamental concepts.

```python
plt.stem(mu, np.abs(X)/N, markerfmt='C0o', basefmt='C0:', linefmt='C0:')
# plt.plot(mu, np.abs(X)/N, 'C0', lw=1)  # this is here a misleading plot and hence not used
plt.xlabel(r'DFT eigenfrequency $\mu$')
plt.ylabel(r'DFT spectrum magnitude $\frac{|X[\mu]|}{N}$')
plt.grid(True)
```

**Copyright**

The notebooks are provided as [Open Educational Resources](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebooks for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Frank Schultz, Digital Signal Processing - A Tutorial Featuring Computational Examples* with the URL https://github.com/spatialaudio/digital-signal-processing-exercises