# Orbit transfers

## Hohmann transfer

The Hohmann transfer orbit offers a solution to transferring between two circular orbits, via an elliptic transfer orbit,
using the least possible velocity change ($\Delta v$). It was first recognized/demonstrated by Hohmann in 1925,
before the rocket technology to achieve any orbit existed!

A typical use of the Hohmann transfer is to start at a lower-altitude circular "parking orbit", then use
the elliptical transfer orbit to reach a higher-altitude circular orbit. This involves two instantaneous
changes in velocity ($\Delta v$), one at the perigee of the elliptical orbit and one at the apogee (the "apogee kick").

:::{figure,myclass} fig-hohmann

Illustration of a Hohmann transfer orbit, from a smaller circular orbit to a larger circular orbit.
Source: [Leafnode](https://upload.wikimedia.org/wikipedia/commons/d/df/Hohmann_transfer_orbit.svg), [CC BY-SA 2.5](https://creativecommons.org/licenses/by-sa/2.5) via Wikimedia Commons.
:::

**The problem:** Given the original orbit's perigee and apogee distances ($r_{p1}$ and $r_{a1}$),
and the apogee distance of the desired elliptical orbit (which also may be the radius of the final circular orbit) $r_{a2}$,
find the required instantaneous change in velocity $\Delta v$ to enter the elliptical orbit.

The change in velocity needed is the difference in perigee velocities of the two
orbits. For orbit 1:

$$
\begin{align}
e_1 &= \frac{r_{a1} - r_{p1}}{r_{a1} + r_{p1}} \\
r_{p1} &= \frac{h_1^2 / \mu}{1 + e_1} \\
\rightarrow h_1 &= \sqrt{2 \mu} \sqrt{\frac{r_{p1} r_{a1}}{r_{p1} + r_{a1}}} \\
v_{p1} &= \frac{h_1}{r_{p1}} \;,
\end{align}
$$

where $v_{p1}$ is the velocity of orbit 1 at its perigee.

Similarly, for orbit 2:

$$
\begin{align}
h_2 &= \sqrt{2 \mu} \sqrt{\frac{r_{p2} r_{a2}}{r_{p1} + r_{a2}}} \\
v_{p2} &= \frac{h_2}{r_{p2}} \;,
\end{align}
$$

where $r_{p2} = r_{p1}$ is the perigee distance, shared between the orbits.
Then, the instantaneous velocity change needed is

$$
\Delta v = v_{p2} - v_{p1} \;.
$$

Orbit 2 may be an elliptical transfer orbit, used to reach a higher circular orbit via
an "apogee kick", which is a second velocity change at the apogee of the elliptical orbit.
We can determine the velocity change needed to enter a circular orbit with a radius
matching $r_{a2}$:

$$
\begin{align}
h_3 &= \sqrt{2 \mu} \sqrt{\frac{r_{a2} r_{a2}}{r_{a2} + r_{a2}}} = \sqrt{\mu r_{a2}} \\
v_{3} &= \frac{h_3}{r_{a2}} \;, \\
\Delta v &= v_3 - v_{a2}
\end{align}
$$

## Apse line rotation

A more general orbital transfer is an apse line rotation,
which involves an instantaneous velocity change applied at some location in one orbit
and leads to anew orbit with a different eccentricity and perigee/apogee distances.

**Problem:** Given initial perigee distance $r_{p1}$, initial apogee distance $r_{a1}$,
desired perigee distance $r_{p2}$, desired apogee distance $r_{a2}$,
and apse rotation angle $\eta$.

Find: true anomaly of the burn $\theta_1$, required instantaneous velocity change $\Delta v$,
and the required direction/angle of thrust from the local horizontal direction.

Solution steps:

1. Calculate $e_1$, $e_2$, $h_1$, $h_2$:

   $$
   \begin{align}
   e_1 &= \frac{r_{a1} - r_{p1}}{r_{a1} + r_{p1}} \\
   r_p &= \frac{h_1^2 / \mu}{1 + e_1} \\
   \rightarrow h_1 &= \sqrt{ r_p (1 + e_1) \mu} \;,
   \end{align}
   $$

   then repeat for $e_2$ and $h_2$.

2. Solve for the true anomalies in each orbit of the intercept, $\theta_1$ and $\theta_2$:

  $$
  \begin{align}
  \eta &= \theta_1 - \theta_2 \rightarrow \theta_2 = \theta_1 - \eta \\
  r_{1I} &= \frac{h_1^2 / \mu}{1 + e_1 \cos \theta_1} = r_{2I} = \frac{h_2^2 / \mu}{1 + e_2 \cos \theta_2}
  \end{align}
  $$

  Following some math, we can solve for the angles:

  $$
  \begin{align}
  \theta_1 &= \phi + \cos^{-1} \left( \frac{c}{a} \cos \phi \right) \\
  \theta_2 &= \theta_1 - \eta \\
  \text{where} \quad \phi &= \tan^{-1} \left( \frac{b}{a} \right) \\
  a &= e_1 h_2^2 - e_2 h_1^2 \cos \eta \\
  b &= -e_2 h_1^2 \sin \eta \\
  c &= h_1^2 - h_2^2 \;,
  \end{align}
  $$

  where $\phi$ gives the direction of the local horizontal.

3. Calculate the position and velocity in the initial orbit:

  $$
  \begin{align}
  v_{theta 1} &= \frac{h_1}{r_1} \\
  v_{r 1} &= \frac{\mu}{h_1} e_1 \sin \theta_1 \\
  v_1 &= \sqrt{ v_{r1}^2 + v_{\theta 1}^2 } \\
  \gamma_1 &= \tan^{-1} \left( \frac{v_{r1}}{v_{\theta 1}} \right)
  \end{align}
  $$

4. Calculate the position and velocity in the desired orbit:

  $$
  \begin{align}
  v_{theta 1} &= \frac{h_2}{r_2} \\
  v_{r 2} &= \frac{\mu}{h_2} e_2 \sin \theta_2 = \frac{\mu}{h_2} e_2 \sin (\theta_1 - \eta) \\
  v_2 &= \sqrt{ v_{r2}^2 + v_{\theta 2}^2 } \\
  \gamma_2 &= \tan^{-1} \left( \frac{v_{r2}}{v_{\theta 2}} \right)
  \end{align}
  $$

5. Calculate the instantaneous change in velocity needed:

  $$
  \Delta v = \sqrt{ v_1^2 + v_2^2 - 2 v_1 v_2 \cos \left( \theta_2 - \theta_1 \right)} \;,
  $$

  at the intercept location.

6. Calculate the direction/angle of thrust:

  $$
  I = \tan^{-1} \left( \frac{v_{r2} - v_{r1}}{v_{\theta 2} - v_{\theta 1}} \right) + 180^{\circ}
  $$

## Cost of transfers

The cost of an orbit transfer, in terms of the mass of propellant needed, depends on a few factors:

- propellant used
- thrust developed
- rate of fuel consumption ($\dot{m}$)

The **specific impulse**, given in units of seconds, is a performance parameter that quantifies the amount of thrust developed to the mass flow rate of propellant required, where higher values indicate (theoretically) better performance:

$$
I_{sp} = \frac{T}{\dot{m} g_0} \;,
$$

where $T$ is the thrust force (N), $\dot{m}$ is the mass flow rate (kg/s),
and $g_0$ is the acceleration due to gravity on the ground (9.81 m/s$^2$).
The mass flow rate is directly related to the time rate of change of vehicle mass:
$\dot{m} = -\frac{dm}{dt}$.

Based on a free-body diagram, we can write the equation of motion in the tangential (axial) direction:

$$
m a_t = T - D - m g \sin \gamma = m \frac{dV}{dt} \;,
$$

where in space we can neglect the effects of drag, or $D \approx 0$, due to the extremely rarified
atmosphere (for highly accurate calculations, we _will_ need to account for the perturbuations
due to atmospheric drag); we can also neglect the weight due to reduced gravity, so $m g \approx 0$.
We can also assume thrust is constant. Then:

$$
\begin{align}
T = m \frac{dV}{dt} &= I_{sp} g_0 \left( - \frac{dm}{dt} \right) \\
- I_{sp} g_0 \frac{dm}{dt} &= m \frac{dV}{dt} \\
\int_{m_0}^{m_f} - I_{sp} g_0 \frac{dm}{m} &= \int_0^{\Delta V} dV \\
\Delta v &= - I_{sp} g_0 \ln \left( \frac{m_f}{m_0} \right) \\
\rightarrow \Delta m &= m_0 \left( 1 - e^{-\Delta V / \left( I_{sp} g_0 \right)} \right) \;,
\end{align}
$$

where $\Delta m = m_0 - m_f$, which is the **ideal rocket equation**,
originally developed by Tsiolkovsky.
Given the specific impulse and initial mass of a rocket, this tells us the
change in mass needed (via exhaust of propellant) to achieve a particular change
in velocity.

## Finite burn times

While some burns can reasonably be approximated as instantaneous, other orbit transfers
will involve or require finite burn times.
These involve a burn of the engine starting at some time (and some true anomaly in the initial orbit), and leads to a different orbit, similar to an apse line rotation.

To determine how a finite burn time affects an object's orbit, we need to solve the orbital equation of motion, but with a thrust force:

$$
\vec{\ddot{r}} + \frac{\mu}{r^3} \vec{r} = \frac{\vec{F}_T}{m} = \frac{T \, \vec{v}}{m v} \;,
$$

assuming that the thrust force $\vec{F}_T$, with magnitude $T$, is in the same direction
as the velocity $\vec{v}$,
along with an equation for the changing mass of the vehicle, as it expels propellant:

$$
\frac{dm}{dt} = -\frac{T}{I_{sp} g_0} \;.
$$

This can be solved by decomposing into a system of first-order equations, and integrating numerically:

$$
\begin{align}
z_1 &= x \quad \dot{z}_1 = z_4 \\
z_2 &= y \quad \dot{z}_2 = z_5 \\
z_3 &= z \quad \dot{z}_3 = z_6 \\
z_4 &= \dot{x} \quad \dot{z}_4 = \frac{-\mu}{r^3} z_1 + \frac{T z_4}{z_7 v} \\
z_5 &= \dot{y} \quad \dot{z}_5 = \frac{-\mu}{r^3} z_2 + \frac{T z_5}{z_7 v} \\
z_6 &= \dot{z} \quad \dot{z}_6 = \frac{-\mu}{r^3} z_3 + \frac{T z_6}{z_7 v} \\
z_7 &= m \quad \dot{z}_7 = -\frac{T}{I_{sp} g_0} \;,
\end{align}
$$

where

$$
\begin{align}
r &= \sqrt{z_1^2 + z_2^2 + z_3^2} \\
v &= \sqrt{z_4^2 + z_5^2 + z_6^2} \;.
\end{align}
$$

To solve a problem where the burn time $\Delta t$ to reach a particular orbit is unknown,
an iterative approach can be used.

```python

```


#### _Speech Processing Labs 2020: Signals: Module 2_

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

import cmath
from math import floor
from matplotlib.animation import FuncAnimation
from IPython.display import HTML
plt.style.use('ggplot')

from dspMisc import *
```

# 2 Filtering the Source: Finite Impulse Response Filters

### Learning Outcomes
* Be able to describe what an FIR filter is
* Be able to explain what the impulse response of an FIR filter is
* See how an FIR filter can be used as a lowpass frequency filter.
* See how changing the coefficients of an FIR filter can change it's frequency response

### Need to know
* Topic Videos: Spectral Envelope, Filter, Impulse Train
* [Interpreting the DFT](./sp-m1-5-interpreting-the-dft.ipynb)
* [Building the source: impulse trains](sp-m2-1-impulse-as-source.ipynb)

Equation alert: If you're viewing this on github, please note that the equation rendering is not always perfect. You should view the notebooks through a jupyter notebook server for an accurate view.

## 2.0 Filters

We've seen in the past notebooks that sometimes our input signal isn't exactly what we want. There is a vast literature in signal processing about designing filters to transform one signal into another. In speech processing, our signals often includes some sort of noise that we'd like to get rid of.  However, we can also use filters to shape a simple input, like an impulse train, into something much more complicated, like a speech waveform.

In class you've seen two types of filters:

* Finite Impulse Response (FIR)
* Infinite Impulse Response (IIR)

Both perform a transform on an input sequence $x[n]$ to give us some desired output sequence $y[n]$.  The difference between the two types of filters is basically whether we only use the inputs to derive each output $y[n]$ (FIR), or whether we also use previous outputs (IIR).

In the following we'll illustrate some of the properties of FIR filters.

## 2.1  Finite Impulse Response Filters

Finite Impulse Response (FIR) filters have the following form:

$$
\begin{align}
y[n] &= b[0]x[n] + b[1]x[n-1] + \dots + b[K]x[n-K] \\
 &= \sum_{k=0}^K b(k) x[n-k]
\end{align}
$$

Here, we have:
* an input sequence $x[n]$ of length $N$
* a set of $K$ filter coefficients.

We can read the equation as saying that $n$th ouput of the filter, $y[n]$, is a weighted sum of the previous K inputs $x[n],...,x[n-K]$.

### Example
Let's plot the $b[k]\cdot x[n-k]$ terms where $x$ is sinusoid of 4 Hz and the filter is `b=[1.1, 1.2, 1.5, 1.2, 1.1]`

```python
## Plotting a filter input window

## Set the number of samples N, sampling rate f_s
N=64
f_s = 64
t_s = 1/f_s
print("sampling rate: f_s = %f\nsampling time: t_s: %f" % (f_s, t_s))

x, time_steps = gen_sinusoid(frequency=4, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)

## Plot the sinusoid we've just created:
fig, timedom = plt.subplots(figsize=(16, 4))
timedom.plot(time_steps, x, 'o', color='grey')
timedom.set_xlabel('Time (s)')
timedom.set_ylabel('Amplitude')

## Filter coefficients (you could try changing these)
b = [1.1, 1.2, 1.5, 1.2, 1.1]

## K is the number of filter coefficients
K=len(b)

## Let's see what happens when n=19 (or try changing this variable!)
n=19
print("filter using b at n=%d, K=%d\n" % (n, K))

## Plot the values that are input to the filter
## +1's because python indexing/slicing doesn't include the end point
timedom.plot(time_steps[n+1-K:n+1], x[n+1-K:n+1], 'o', color='red')

## Calculate the b[k]*x[n-k] terms and add them to a list
filtered_n = []
for k in range(len(b)):
    ## print out the variables here
    print("%d:, b[%d]=%f, x[%d-%d]=%f, b[%d]*x[%d-%d]=%f" % (n-k, k, b[k], n, k,  x[n-k], k, n, k, b[k]*x[n-k]))
    filtered_n.append(b[k]*x[n-k])

## reverse the list so that they're in time order
filtered_n.reverse()

## Plot the b[k]*x[n-k] terms
timedom.plot(time_steps[n+1-K:n+1], filtered_n, 'o', color='blue')

## Calculate the filter output (add up the product terms)
print("\ny[%d] = %f" % (n, sum(filtered_n)))
```

In the plot above, you should see:
* A cosine wave with frequency 4 Hz in grey.
* The inputs the the filter x[19],...,x[15] in red.
    * 5 input values for 5 filter coefficients
* The product $b[k]*x[n-k]$ for $n=19$, and $k=0,...,4$

So with `b = [1.1, 1.2, 1.5, 1.2, 1.1]` all the input values get scaled up, with the the middle of the filter window get biggest relative increase.

When we add all the product terms together, we get y[19] = 4.87 - a lot bigger than any of the input values!

## 2.2 An FIR moving average filter

A useful special case of an FIR filter is where each of the filter coefficients is just $1/K$. In this case our FIR equation looks like this:

 $$
 \begin{align}
     y[n] &= \sum_{k=0}^{K-1} \frac{1}{K} x[n-k] \\
          &= \frac{1}{K} \sum_{k=0}^{K-1}  x[n-k]
 \end{align}
 $$

This equation says that when we apply the filter, we step through the input. At each step, we output the average of the previous $K$ inputs. You might know this by another more intuitive name: a _moving average_.  You might also have seen this as method to 'smooth' an input.

Let's play around with this idea a bit and see how it relates to our notion of frequency response.

### Example:

Let's look at this 5-point moving average filter.  In this case all the filter coefficients $b[k] = 1/5$ for $k=0,..,K-1=4$

$$
\begin{align}
y[n] &= \frac{1}{5} \sum_{k=0}^4 x[n-k] \\
&= \frac{1}{5}x[n] + \frac{1}{5}x[n-1] + \frac{1}{5}x[n-2] + \frac{1}{5}x[n-3] + \frac{1}{5}x[n-3]
\end{align}
$$

Now, let's code this specific filter up and apply it to some sequences!

```python
# Apply a moving average filter of size K to input sequence x
def moving_average(x, K=5):
    ## There are quicker ways to do this in numpy, but let's just do it this way for now for transparency

    ## We know that we'll have as many outputs as inputs, so we can initialize y to all zeros
    N = len(x)
    y = np.zeros(N)

    ## Go through the input one step at a time
    for n in range(N):

        ## Add up the last K inputs (including the current one)
        for k in range(K):

            ## Exercise: why do we have to put this conditional in here?
            if n-k >= 0:
                y[n] = y[n] + x[n-k]

        ## Divide by the size of the input window to get an average
        y[n] = (1/K) * y[n]

    return y

```

### Generate a 'noisy' sinusoid

Let's generate a compound sinusoid with one low frequency and one high(er) frequency component.  We can take the higher frequency component as representing some sort of periodic noise in the signal.

```python
## Set the number of samples N, sampling rate f_s
N=64
f_s = 64
t_s = 1/f_s

print("sampling rate: f_s = %f\nsampling time: t_s: %f" % (f_s, t_s))
```

```python
## make some sinusoids:
## Since the sample rate and sequence length is the same, the generated time steps will match for
## x1 and x2
x1, time_steps = gen_sinusoid(frequency=4, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)
x2, time_steps = gen_sinusoid(frequency=24, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)

# add them up!
x_noisy = x1 + x2
```

```python
## Plot the compound sinusoid we've just created:
fig, timedom = plt.subplots(figsize=(16, 4))
timedom.plot(time_steps, x_noisy, color='magenta')
timedom.set_xlabel('Time (s)')
timedom.set_ylabel('Amplitude')
```

You should be able to see that the 4Hz cosine wave is perturbed by a 24 Hz cosine. So, this could represent a situation where there is high frequency noise in our signal that we'd like to get rid of.  Let's see if we can use our filter to smooth out this high frequency noise.

### Apply the moving average filter

```python
## Apply our moving average filter
K=5
y_mov = moving_average(x_noisy,K=K)
```

```python
## Plot the results
fig, timedom = plt.subplots(figsize=(16, 4))
timedom.set_xlabel('Time (s)')
timedom.set_ylabel('Amplitude')

## The input signal
timedom.plot(time_steps, x_noisy, color='magenta', label='Noisy input')

## The underlying 4Hz signal
timedom.plot(time_steps, x1, color='grey', label='low freq input component')

## After the moving average has been applied
timedom.plot(time_steps, y_mov, color='blue', label='filter output')

timedom.set_title('%d-point moving average filter applied to a noisy sinsuoid' % K)
timedom.legend()
```

You should see:
* the original signal in magenta
* the low frequency cosine component in grey
* the output of the filter in blue

### Exercise:

* From the plot, does it appear that the moving average recovered the original 4Hz signal in terms of:
    * frequency?
    * amplitude?
    * phase?
* Are there still high frequency components in the filter output?

Can you see some issues with applying this filter? What happens at the end points? Are they valid?

### Notes

```python

```

### Get the DFT of the filtered signal

We can apply the DFT to our output to check our observations from the time-domain output.

```python
## DFT of the original input
mags_x, phases_x = get_dft_mag_phase(x_noisy, N)

## DFT of the filter output
mags_y, phases_y = get_dft_mag_phase(y_mov, N)
```

```python
dft_freqs = get_dft_freqs_all(sample_rate=f_s, seq_len=N)

fig, mags = plt.subplots(figsize=(16, 4))
mags.set(xlim=(-1, N/2), ylim=(-1, N))
mags.scatter(dft_freqs, mags_x, color='magenta', label='noisy input')
mags.scatter(dft_freqs, mags_y, color='blue', label='filtered output')
mags.set_xlabel("Frequency (Hz)")
mags.set_ylabel("Magnitude")
mags.set_title("DFT Magnitude before and after moving average filter")
mags.legend()

#Let's not worry about phase right now, but feel free to uncomment and have a look!
#fig, phases = plt.subplots(figsize=(16, 4))
#phases.set(xlim=(-1, N/2), ylim=(-10, 10))
#phases.scatter(dft_freqs, phases_x, color='magenta', label='noisy input')
#phases.scatter(dft_freqs, phases_y, color='blue', label='filtered output')
#phases.set_xlabel("Frequency (Hz)")
#phases.set_ylabel("Phase (rad)")
#phases.set_title("DFT Phase before and after moving average filter")
#phases.legend()

```

### Exercise

* Based on the magnitude spectrum:
    * Did the filter get rid of the 24Hz component?
    * Do you see any signs of leakage?

* What happens if you change the frequency of the second sinusoid to something lower (e.g. 6Hz)?

### Notes

```python

```

## 2.3 FIR as convolution

An FIR Filter that takes $K$ previous elements of $x[n]$ as input has the following general form:

$$ y[n] = \sum_{k=0}^{K-1} b[k] x[n-k] $$

You might recognize this as a **convolution** of the two sequences $b$ and $x$  (i.e. $b * x$).

So, we can theoretically set our filter to coefficients to whatever we want.  Here's a function that generalizes our moving average filter to allow for this:

```python
def fir_filter(x, filter_coeffs):
    N = len(x)
    K = len(filter_coeffs)

    y = np.zeros(N)
    for n in range(N):
        for k in range(K):
            if n-k >= 0:
                #print("y[%d]=%f, b[%d]=%f, x[%d]=%f" % (n, y[n], k, filter_coeffs[k], n-k, x[n-k]))
                y[n] = y[n] + (filter_coeffs[k]*x[n-k])
        #print("y[%d]=%f" % (n, y[n]))
    return y

```

### Changing the filter coefficients

Let's try it out with different coefficient values, comparing our unweighted average `h_avg` filter with a weighted average `h_wavg` filter.

```python

## The 5-point moving average from before
h_avg = np.array([1/5, 1/5, 1/5, 1/5, 1/5])
y_avg = fir_filter(x_noisy, h_avg)

## A 5-point symmetrically weighted average
h_wavg = np.array([1/5, 1/3, 1, 1/3, 1/5])
y_wavg = fir_filter(x_noisy, h_wavg)
```

### Filter effects in the time domain

```python
## Plot the filter outputs
fig, timedom = plt.subplots(figsize=(16, 4))

## The original "noisy" input
timedom.plot(time_steps, x_noisy, color='magenta', label='input x_noisy')
timedom.scatter(time_steps, x_noisy, color='magenta')

## The 5-point moving average
timedom.plot(time_steps, y_avg, color='blue', label='unweighted average: y_avg')
timedom.scatter(time_steps, y_avg, color='blue')

## The 5-point weighted average
timedom.plot(time_steps, y_wavg, color='orange', label='weighted average: y_wavg')
timedom.scatter(time_steps, y_wavg, color='orange')
timedom.legend()
timedom.set_xlabel('Time (s)')
timedom.set_ylabel('Amplitude')
```

In this time vs amplitude graph, you should see:
* the 'noisy input' in magenta
* the output of the unweighted average filter in blue (`y_avg`)
* the output of the weighted average filter in orange (`y_wavg`)

### Exercise

Q: Why is the output of `y_wavg` more spikey than that of `y_avg`?

### Notes

```python

```

## 2.4 FIR Filters in the Frequency Domain

### The DFT of the filtered outputs

We can look at the effect of the two FIR filters defined above in the frequency domain by performing a DFT on the filter outputs.

```python
## DFT of the original input
mags_x, phases_x = get_dft_mag_phase(x_noisy, N)

## DFT after weighted average filter: h_wavg = np.array([1/5, 1/3, 1, 1/3, 1/5])
mags_wavg, phases_wavg = get_dft_mag_phase(y_wavg, N)

## DFT after unweighted average filter: h_avg = np.array([1/5, 1/5, 1/5, 1/5, 1/5])
mags_avg, phases_avg = get_dft_mag_phase(y_avg, N)

dft_freqs = get_dft_freqs_all(sample_rate=f_s, seq_len=N)

## Plot magnitude spectrums
fig, mags = plt.subplots(figsize=(16, 4))
mags.set(xlim=(-1, N/2), ylim=(-1, N))
mags.scatter(dft_freqs, mags_x, color='magenta', label='input')
mags.scatter(dft_freqs, mags_avg, color='blue', label='unweighted average')
mags.scatter(dft_freqs, mags_wavg, color='orange', label='weighted average')
mags.legend()

## Plot phase spectrums
fig, phases = plt.subplots(figsize=(16, 4))
phases.set(xlim=(-1, N/2), ylim=(-10, 10))
phases.scatter(dft_freqs, phases_x, color='magenta', label='input')
phases.scatter(dft_freqs, phases_avg, color='blue', label='unweighted average')
phases.scatter(dft_freqs, phases_wavg, color='orange', label='weighted average')
phases.legend()

```

### Exercise:

* Describe the difference between the different FIR filters based on the frequency magnitude and phases responses plotted above.

* Does the weighted average filter do as good a job at filtering out the higher frequency signals?

### Notes

```python

```

## 2.5 Convolution in Time,  Multiplication in Frequency

Now we get to the really cool bit.  We know that the DFT allows us to go from the time domain to the frequency domain (and the Inverse DFT let's us go back).  But it also has this very important property:

$$ h[k] * x[n] \mapsto \text{ DFT} \mapsto H(m) \cdot X(m) $$

That is, convolving an input sequence $x$ with a set of filter coefficients, $h$ in the time domain  ($h*x$) is the same as (pointwise) multiplication of the DFT of $h$ with the DFT of $x$.  So, if we know what type of frequency response $h$ has, we can treat this as apply a sort of mask to the DFT of $x$.  This property is known as the **convolution theorem**.

Another way to think about it is that if the DFT outputs of $h$ has zero magnitude, then applying that filter to some input will also zero out those frequencies in the filter output!

We'll see some visualizations of this shortly, but first, we can also note that you can go back the other way using the Inverse DFT:

$$ H(m) \cdot X(m) \mapsto \text{ IDFT} \mapsto h[k] * x[n] $$

And also it works for multiplication in time domain too:

$$ h[k] \cdot x[n] \mapsto \text{DFT} \mapsto H(m) * X(m) $$

This is useful for understanding why the leakage graph around each DFT bin looks the way it does (though we won't go into it here!).

In fact, we can use this to not only show that the moving average acts as a low pass filter, but also to be able to calculate exactly the type of filter it will be.

###  Plotting different FIR filter frequency responses

In order to multiply together the filter and the input frequency responses, we need to make sure the filter and the input have the same number of samples.  We can do this by just padding out the filter with zeros (you can see why in the optional extra material at the end of this notebook).  We can then look how changing the 'shape' of a filter changes it's frequency response. The following exercise shows some examples.

### Exercise
* Run the function plot_filter_freq_responses (defined in the next cell) to plot the frequency responses of filters with different shapes (e.g. `h_plat`, `h_tri`, `h_rect` in the cell below the next).
    * What's the difference in frequency response of the triangular filter and the rectangular filter?
    * What's the difference between the moving average of size 5 and one of size 9 (h_rect9)?
    * Try some other FIR filters!

```python
# Given a list of filters, the sample rate and a specific sequence length
# plot the DFT frequency response of each filter.
# Each filter should be defined as a list of coeffients (b[k])
def plot_filter_freq_responses(filters, sample_rate, seq_length):

    ## Get the set of DFT output frequencies given the sample rate and desired sequence length
    dft_freqs_filter = (sample_rate/seq_length) * np.arange(seq_length)

    ## Calculate the time steps for each filter value given the sample rate and sequence length
    time_steps = (1/sample_rate) * np.arange(seq_length)

    ## Set up some plots:
    # the filter itself (time v amplitude)
    fig_time, sinusoid = plt.subplots(figsize=(16, 4))

    # the frequency response (freq v magnitude)
    fig_freq, fresponse = plt.subplots(figsize=(16, 4))

    x_filters = {}
    ## For each filter:
    for i, h in enumerate(filters):

        # pad the filter coefficients with zeros until we get the desired sequence length
        x_zeros = np.zeros(seq_length - len(h))
        x = np.concatenate([h, x_zeros])

        # Get the DFT outputs
        mags, phases = get_dft_mag_phase(x, seq_length)

        # Plot the filter
        sinusoid.scatter(time_steps, x)
        sinusoid.plot(time_steps, x, label=repr(h))

        # plot the magnitude response
        fresponse.scatter(dft_freqs_filter, mags)
        fresponse.plot(dft_freqs_filter, mags, label=repr(h))
        fresponse.set(xlim=(-1,seq_length/2))

        # return the filters and the DFT responses just in case
        x_filters = {'x':x, 'mags':mags, 'phases':phases, 'coeffs':h}

    sinusoid.set_xlabel('Time(s)')
    sinusoid.set_ylabel('Amplitude')
    sinusoid.set_title('Zero padded filters of different shapes')
    sinusoid.legend()

    fresponse.set_xlabel('Frequency (Hz)')
    fresponse.set_ylabel('Magnitude')
    fresponse.set_title('DFT magnitude response of ero padded filters of different shapes')
    fresponse.legend()

    return x_filters

```

```python
h_plat = np.array([0.1, 0.2, 0.2, 0.2, 0.1])
h_tri = np.array([0.04, 0.12, 0.15, 0.12, 0.01])
h_rect = np.array([1/5, 1/5, 1/5, 1/5, 1/5])
h_rect9 = np.array([1/9, 1/9, 1/9, 1/9, 1/9, 1/9, 1/9, 1/9, 1/9])
## Try some others if you like!

N=64
f_s=64

filter_dict = plot_filter_freq_responses(filters=[h_tri, h_plat, h_rect, h_rect9], sample_rate=f_s, seq_length=N)
```

### Notes

```python

```

## 2.7 Applying an FIR filter to an impulse train

Now, finally we can look at the effect of applying an FIR filter to an impulse train.  Remember, we're using impulse trains to represent a sound source (i.e. vocal pulses at a specific frequency).  Eventually, we want to be able to define filters that capture the effects of the vocal tract.

First, here's a function that produces a sequence of pulses at a given frequency and sample rate. We'll then apply a triangular filter and look at what the frequency response is of the output is.

```python
## Let's make an impulse train
N=200
f_s = 8000
t_s = 1/f_s

frequency = 200

x_imp, time_steps = make_impulse_train(sample_rate=f_s, frequency=frequency, n_samples=N)

```

```python
## Plot the impulse train made in the cell before
fig, td = plt.subplots(figsize=(16, 4))
td.scatter(time_steps, x_imp)
td.plot(time_steps, x_imp)
td.set_xlabel('Time (s)')
td.set_ylabel('Amplitude')
td.set_title('Impulse train')
```

You should see an impulse train, with an impulse frequency of 200 Hz.  Let's see what happens when we apply the filter.

```python
## define our FIR filter coefficients
h_mov = np.array([0.1, 0.3, 0.5, 0.3, 0.1])

#try this later!
#h_mov = np.array([1])

## Apply the filter to our impulse train
y_mov = fir_filter(x_imp, h_mov)

## plot the impulse train and the filtered version of it
fig, td = plt.subplots(figsize=(16, 4))
td.scatter(time_steps, x_imp)
td.plot(time_steps, x_imp, label='impulse train')
td.plot(time_steps, y_mov, label='filtered impulse train')
td.set_xlabel('Time (s)')
td.set_ylabel('Amplitude')
td.legend()
td.set_title('Impulse train and filtered impulse train')
```

Here you should see our impulse train (red) and the filtered impulse train (blue).  The filtered version has smaller amplitude and a broader peak (more samples have non-zero value).  Also, the filtered peaks are slightly after the original impulses.

Now let's look at the DFT output:

```python
## Get the DFT the filter output
mags, phases = get_dft_mag_phase(y_mov, N)

## Plot the DFT frequencies rather than DFT output indices
dft_freqs = get_dft_freqs_all(f_s, N)

## Plot the magnitude spectrum
fig, ax = plt.subplots(figsize=(16, 4))
ax.scatter(dft_freqs[0:int(N/2)],mags[0:int(N/2)])
ax.plot(dft_freqs[0:int(N/2)], mags[0:int(N/2)])

```

### Exercises

* What do the spikes in the DFT outputs represent?
* What does the first spike after the 0 frequency one represent?
* What does this filter appear to do?
    * Remember applying the FIR filter in the time domain (via convolution) is the same as multiplying the DFT of the filter to the DFT of the input signal

### Notes

```python

```

### Exercises

* What happens when the impulse train frequency doesn't fall on one of the bins? e.g. `frequency = 130`, for 200 samples, with sampling rate 8000 samples/second
* Does the magnitude spectrum have the harmonic structure you'd expect?

### Notes

```python

```

## 2.6 (Extra Extension) The moving average filter as a rectangular function

This section (2.6) illustrates how we can use the convolution theorem understand why the moving average type filters act like low pass filters, and also the connection with leakage in the DFT magnitude response we saw previously.  This is optional extra material.

If you want to see an example of the convolution theorem working in the frequency domain, you can just run the code and have a look at the graph titled 'Frequency response after applying 5-point weighted average filter'

The convolution theorem tells us that, if we know the frequency response of an FIR filter, we know how it will affect the frequency response of it's input (we just multiply the individual frequency response together).

To understand what filter frequency response will look like, it's helpful to first observe that our unweighted moving average filter is pretty much a rectangular window function.  It's easy to see what this means when we plot it. The following function allows us to generate rectangular functions:

```python
def gen_rect_window(start_index, end_index, sample_rate=64, seq_length=64):
    nsteps = np.array(range(seq_length))
    t_s = 1/sample_rate
    time_steps = t_s * nsteps

    ## Let's make a rectangular window
    x_rect = np.zeros(seq_length)
    x_rect[start_index:end_index] = 1

    return x_rect, time_steps

```

###  Now, we make a rectangular window

```python
## Make rectangular window
N=64
K=16
f_s=64
start_index=24
end_index=start_index+K

x_rect, time_steps = gen_rect_window(start_index=start_index, end_index=end_index, sample_rate=f_s, seq_length=N)

fig, timedom = plt.subplots(figsize=(16, 4))
timedom.scatter(time_steps, x_rect, color='magenta')
timedom.plot(time_steps, x_rect, color='magenta')
timedom.set_xlabel('Time (s)')
timedom.set_ylabel('Amplitude')
timedom.set_title('a rectangular window')
```

You should see a sequence with 64 point where the mmiddle 16 points have value 1 and the rest have value 0  (i.e., it looks like a rectangle in the middle).

### Now, let's  look at the frequency response of the rectangular window

```python
## Now we do the DFT on the rectangular function:

## get the magnitudes and phases
mags_rect, phases_rect = get_dft_mag_phase(x_rect, N)

## the DFT output frequencies
dft_freqs_rect = get_dft_freqs_all(f_s, N)

## let's just look at the magnitudes
fig, fdom = plt.subplots(figsize=(16, 4))
fdom.set(xlim=(-1, N/2))
fdom.scatter(dft_freqs_rect, mags_rect)
fdom.set_xlabel("Frequency (Hz)")
fdom.set_ylabel('Magnitude')
fdom.set_title('Frequency response of rectangular window')
## Looks leaky!
```

### Leaky windows?

The plot of the frequency magnitude response of our rectangular window has the hallmarks of leakiness. That is, the frequency response looks scalloped, with the biggest peak occuring around 0Hz. That is, it looks like a low pass filter!

With a bit of algebra we can derive the frequency for any $m$ (not just the DFT output bins indices) to be the following:

If $x[n]$ is a rectangular function of N samples with $K$ continugous samples of value 1 (starting at index $n_0$), we can figure out what the DFT output will be:

$$X[m] = e^{i(2\pi m/N)(n_0-(K-1)/2)} . \frac{\sin(2\pi mK/2N)}{\sin(2\pi m /2N)}$$

This is called the **Dirichlet kernel**. It has the **sinc** shape we saw when we looked at spectral leakage.

How is this useful?  Since we know what the frequency response of a rectangular window is, we know what convolving this with different input sequences will look like in the frequency domain.  We just multiply the frequency magnitude responses together.

On a more general note, this sort of convolution with a (short) window is how we do frequency analysis of speech: we taking windows of speech (aka frames) through time and and apply the DFT to get a frequency response.  A rectangular window is the simplest type of window we can take.  The equation above tells us that the sinc shaped response is an inherent part of using this sort of window.  In fact, we can use other window types (e.g. Hanning) to make the main lobes shaper and the sidelobes flatter, but we never really get away from this sinc shape in real world applications.  This is a key component of this soft of short term analysis.

Let's write this up in a function:

```python
def gen_rect_response(n_0, K, N, stepsize=0.01, polar=True, amplitude=1):
    ms = np.arange(0.01, N, stepsize)
    qs = 2*np.pi*ms/N

    ## Infact, we can work the frequency response to be the Dirichlet Kernel:
    response = (np.exp(-1j*qs*(n_0-(K-1)/2)) * np.sin(qs*K/2))/np.sin(qs/2)

    if polar:
        response_polar = [cmath.polar(z) for z in response]
        mags = np.array([m for m, _ in response_polar]) * amplitude
        phases = np.array([ph if round(mag) > 0 else 0 for mag, ph in response_polar])
        return (mags, phases, ms)

    return response, ms

```

Now we can plot the dirichlet kernel with the leaky looking DFT magnitudes we calculated earlier for our rectangular window.

```python
## Overlay the dirichlet kernel onto the DFT magnitudes we calculated earlier
## You should be able to see that the DFT magnitudes appear as discrete samples of the Dirichlet Kernel

mags_rect, phases_rect = get_dft_mag_phase(x_rect, N)
mags_rect_sinc , _ , ms = response = gen_rect_response(start_index, K, N)

fig, ax = plt.subplots(figsize=(16, 4))
ax.scatter(dft_freqs_rect, mags_rect, label='rectangular window')
ax.plot((f_s/N)*ms, mags_rect_sinc, color='C2', label='dirichlet')
ax.set(xlim=(-1,N/2))
ax.set_xlabel('Frequency (Hz)')
ax.set_ylabel('Magnitude')
ax.set_title('Frequency response of a rectangular sequence, %d samples with %d contiguous ones' % (N, K))
```

You should be able to see that the DFT magnitudes appear as discrete samples of the sinc shaped Dirichlet Kernel

### The unweighted average filter as a rectangular function

We can think of our 5-point unweighted average filter as a 5-point input sequence with all values set to 1/5.  We can then deduce that the frequency response of the filter will have the same shape as the frequency response of a rectangular window of all ones, but scaled down by 1/5.

Now let's check:

```python
N_h=5
f_s=64
start_index=0
end_index=N_h - start_index

## A 5 point rectangular window of all ones
h_avg, time_steps = gen_rect_window(start_index=start_index, end_index=end_index, sample_rate=f_s, seq_length=N_h)
h_avg = h_avg/N_h

fig, td = plt.subplots(figsize=(16, 4))
td.scatter(time_steps, h_avg, color='magenta')
td.plot(time_steps, h_avg, color='magenta')
td.set_xlabel('Time (s)')
td.set_ylabel('Amplitude')
td.set_title('5 point unweighted average as a rectangular function')

## Not very exciting looking!
print("h_avg:", h_avg)
```

You should just see 5 point in a row, all with value 1/5.  Now, we can plot the DFT magnitude response, as well as it's idealized continuous version:

```python
## Get the frequency magnitude response for our rectangular function
mags_h_avg, phases_h_avg = get_dft_mag_phase(h_avg, N_h)

## Get the continuous
rect_mags_h_avg, _ , ms = gen_rect_response(start_index, N_h, N_h, amplitude=np.max(h_avg))

## x-axis as frequencies rather than indices
ms_freqs_h_avg = (f_s/N_h) * ms
dft_freqs_h_avg = (f_s/N_h) * np.arange(N_h)

## Plot the frequency magnitude response
fig, fd = plt.subplots(figsize=(16, 4))
fd.set(xlim=(-1, N/2))
fd.scatter(dft_freqs_h_avg, mags_h_avg)
fd.set_xlabel('Frequency (Hz)')
fd.set_ylabel('Magnitude')
fd.set_title('Frequency response of 5-point unweighter average filter')
#fd.scatter(dft_freqs_rect, mags_rect)

fd.plot(ms_freqs_h_avg, rect_mags_h_avg, color="C2")
```

You should see $floor(N/2) = 2$ points, with a main lobe peaking at 0 Hz, and side lobes peaking between each of the DFT output frequencies.
So,  DFT frequencies sit exactly at the zeros of this function when the windown size K is the same as the number of samples.

### Matching the filter and input size with zero padding

The theorem we saw above told us that we could calculate the frequency response of applying the FIR filter to an input sequence (via convolution), but multiply the DFT outputs of the filter and the input sequence.

Now, the x-axis range matches our that of our noisy input sequence because that is determined by the sampling rate. However, the filter frequency response we have above only has 5 outputs, while our input sample size was 64 because the number of DFT outputs is determined by the number of samples we put into the DFT.

To get things in the right form, we need to do some **zero padding** of the filter. We'll see that this basically gives us more samples of the Dirichlet Kernel corresponding to the filter frequency response.

```python
N=64
K=5
f_s=64
start_index=0
end_index=K

## Make a rectangular filter: K ones at the start
h_avg_pad, time_steps = gen_rect_window(start_index=start_index, end_index=end_index, sample_rate=f_s, seq_length=N)

## Divide by K to make it an average
h_avg_pad = h_avg_pad/K

## Plot the filter
fig, td = plt.subplots(figsize=(16, 4))
td.scatter(time_steps, h_avg_pad, color='magenta')
td.plot(time_steps, h_avg_pad, color='magenta')
td.set_xlabel('Time (s)')
td.set_title('5 point unweighted average FIR filter padded with zeros')

#print("N=%d, K=%d, start=%d, end=%d" % (N, K, start_index, end_index))

```

```python
## Get the frequency magnitude response for our rectangular function
mags_havg, phases_havg = get_dft_mag_phase(h_avg_pad, N)

## Plot the frequency magnitude response
## x-axis as actual frequencies rather that DFT indices
dft_freqs_havg = (f_s/N) * np.arange(N)

fig, fd = plt.subplots(figsize=(16, 4))
fd.set(xlim=(-1,N/2))
fd.scatter(dft_freqs_havg, mags_havg)
fd.set_xlabel('Frequency (Hz)')
fd.set_ylabel('Magnitude')
fd.set_title('Magnitude response of 5-point unweighter average filter zero padded to 64 samples')
```

You should be able to see more clearly in the frequency response graph that the zero padding doesn't change doesnt change the basic shape of the filter's frequency response, we just get a finer grained representation in terms of samples (red dots).

### Calculate the input and filter frequency responses

```python
## Now let's calculate frequency responses of the original input
mags, phases = get_dft_mag_phase(x_noisy, N)

## ...  the filter
mags_filter, phases_filter = get_dft_mag_phase(h_avg_pad, N)

## ... and the filtered output that we calculated above
mags_avg, phases_avg = get_dft_mag_phase(y_avg, N)

## Plot with actual frequencies on the x-axis
dft_freqs = get_dft_freqs_all(f_s, N)
```

```python
## plot frequency responses

fig, fd = plt.subplots(figsize=(16, 4))
fd.set(xlim=(-1,N/2), ylim=(-1, N))

# DFT(input)
fd.scatter(dft_freqs, mags, color='magenta', label='DFT(input)')

# DFT(filter) * DFT(input)
fd.scatter(dft_freqs, mags_filter*mags, color='blue', label='DFT(filter).DFT(input)')

# DFT(filtered input)
fd.scatter(dft_freqs, mags_avg, color='red', label='DFT(filter*input)')
fd.set_xlabel('Frequency (Hz)')
fd.set_ylabel('Magnitude')
fd.set_title('Frequency response after applying 5-point weighted average filter')
fd.legend()
```

You should see that the results from multiplying the DFT magnitudes from the input and the filter (blue) is (more or less) the same as the DFT of applying the filter in th time domain via convolution (red)

* Notice that there are some differences between the results from the time domain application of the filter (red) and the frequency domain multiplication (blue).  In particular there appears to be some leakage in the time-domain convolution case, possibly due to floating point errors.

### Exercise

* Try changing the frequency of the second cosine component of our compound wave in the code below.
    * Does the amount of attenuation of the high frequency component change as suggested by the DFT of the filter?
    * e.g. try 26 Hz vs 19 Hz
* What does this tell you about how well this low pass filter get's rid of high frequency noise?

```python
## Change the frequency of x2

x1, time_steps = gen_sinusoid(frequency=4, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)
x2, time_steps = gen_sinusoid(frequency=19, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)

# add them up!
x_noisy = x1 + x2

## Now let's calculate frequency responses of the original input
mags, phases = get_dft_mag_phase(x_noisy, N)

## ...  the filter
mags_filter, phases_filter = get_dft_mag_phase(h_avg_pad, N)

## Plot with actual frequencies on the x-axis
dft_freqs = get_dft_freqs_all(f_s, N)

## plot frequency responses

fig, fd = plt.subplots(figsize=(16, 4))

fd.set(xlim=(-1,N/2), ylim=(-1, N))

# DFT(input)
fd.scatter(dft_freqs, mags, color='magenta', label='DFT(input)')

# DFT(filter) * DFT(input)
fd.scatter(dft_freqs, mags_filter*mags, color='blue', label='DFT(filter)*DFT(input)')

fd.set_xlabel('Frequency (Hz)')
fd.set_ylabel('Magnitude')
fd.set_title('Frequency response after applying 5-point weighted average filter')
fd.legend()
```

### Notes

```python

```

```python

```


# Lecture 24: Gamma Distribution, Poisson Processes

## Stat 110, Prof. Joe Blitzstein, Harvard University

----

## Stirling's Approximation to the Factorial

\begin{align}
  n! &= n \, (n-1) \, (n -1) \, \dots \, 1 \\
     &\sim \sqrt{2 \pi n} \, \left( \frac{n}{e} \right)^{n}
\end{align}

where $\sim$ means that the ratio of the two number converges to 1 as $n$ approaches $\infty$.

This is fine when we are talking about $n \in \{1,2,3,\dots\}$, but have you ever wondered what $\pi!$ is?

For that, we need the Gamma function.

----

## Gamma Function

Just as an aside, the Beta and Gamma functions are closely related. The Gamma function should be amongst the Top 10 Functions of All Time (if there was ever such a list).

\begin{align}
  \Gamma(a) &= \int_{0}^{\infty} x^a \, e^{-x} \, \frac{dx}{x} \quad \text{for any real }a \gt 0 \\
  &= \int_{0}^{\infty} x^{a-1} \, e^{-x} \, dx \quad \text{(alternately)}
\end{align}

Note that if $a$ approaches 0 from the right, the $\frac{dx}{x}$ is what drives that integral since $x^0 = 1$ and $e^{-0}=1$.

But $\int \frac{dx}{x} = log(x)$ which blows up to $-\infty$, which is why we restrict $a \gt 0$.

### Properties of the Gamma Function

1. $\Gamma(n) = (n-1)!$ where $n \in \mathbb{Z}^{+}$

1. $\Gamma(x+1) = x \, \Gamma(x)$

1. $\Gamma(\frac{1}{2}) = \sqrt{\pi}$

----

## Gamma Distribution

How would we derive a PDF that is based on the Gamma distribution?

_By normalizing the Gamma function._

\begin{align}
  1 &= \int_{0}^{\infty} c \, x^a \, e^{-x} \, \frac{dx}{x} \\
  &= \int_{0}^{\infty} \frac{1}{\Gamma(a)} \, x^a \, e^{-x} \, \frac{dx}{x}
\end{align}

And so the PDF for a Gamma distribution would be

\begin{align}
 \operatorname{Gamma}(a,1) &= \frac{1}{\Gamma(a)} \, x^a \, e^{-x} \, \frac{dx}{x} \quad \text{for } x \gt 0 \\\\
 \\
 \text{More generally } Y &\sim \operatorname{Gamma}(a, \lambda) \\
 \text{Let } Y &= \frac{X}{\lambda} \text{, where } X \sim \operatorname{Gamma}(a,1) \\
 \rightarrow y &= \frac{x}{\lambda} \\
 x &= \lambda \, y \\
 \frac{dx}{dy} &= \lambda \\
 \\
 \Rightarrow f_Y(y) &= f_X(x) \, \frac{dx}{dy} \quad \text{ transforming } X \text{ to } Y \\
 &= \frac{1}{\Gamma(a)} \, (\lambda y)^a \, e^{-\lambda y} \, \frac{1}{\lambda y} \, \lambda \\
 &= \frac{1}{\Gamma(a)} \, (\lambda y)^a \, e^{-\lambda y} \, \frac{1}{y} \quad \text{for } y \gt 0
\end{align}

```python
import matplotlib
import numpy as np
import matplotlib.pyplot as plt

from matplotlib.ticker import (MultipleLocator, FormatStrFormatter,
                               AutoMinorLocator)
from scipy.stats import gamma

%matplotlib inline

plt.xkcd()

_, ax = plt.subplots(figsize=(12,8))

alpha_values  = [1.,2.,3.,4.,5.,7.5,9.,0.5]
lambda_values = [2.,2.,2.,1.,0.5,1.,1.,0.5]
x = np.linspace(0, 20, 1000)

# plot the distributions
#fig, ax = plt.subplots(figsize=(12, 6))

for a,l in zip(alpha_values, lambda_values):
    ax.plot(x, gamma.pdf(x, a, scale=l), lw=3.2, alpha=0.6, label=r'$\alpha$={}, $\lambda$={}'.format(a,l))

# legend styling
legend = ax.legend()
for label in legend.get_texts():
    label.set_fontsize('large')
for label in legend.get_lines():
    label.set_linewidth(1.5)

# y-axis
ax.set_ylim([0.0, 0.5])
ax.set_ylabel(r'$f(x)$')

# x-axis
ax.set_xlim([0, 20.0])
ax.set_xlabel(r'$x$')

# x-axis tick formatting
majorLocator = MultipleLocator(5.0)
majorFormatter = FormatStrFormatter('%0.1f')
minorLocator = MultipleLocator(1.0)
ax.xaxis.set_major_locator(majorLocator)
ax.xaxis.set_major_formatter(majorFormatter)
ax.xaxis.set_minor_locator(minorLocator)

ax.grid(color='grey', linestyle='-', linewidth=0.3)

plt.suptitle(r'Gamma Distribution $f(x) = \frac{1}{\Gamma(\alpha)} \, (\lambda x)^{\alpha} \, e^{-\lambda x} \, \frac{1}{x}$')

plt.show()
```

----

## Connecting the Gamma and Exponential, Poisson Distributions

### Poisson Processes

Recall Poisson processes, where the number of events happening in a certain time period $t$ is such that

\begin{align}
  N_t &= \text{number of events occuring up to time }t \\
      &\sim \operatorname{Pois}(\lambda, t)
\end{align}

where the number of events occuring in disjoint time intervals are independent.

Now earlier, we discussed how time $t$ is exponential. Consider $t_1$, the time until we observe the very first event.

\begin{align}
  P(t_1 \gt t) &= P(N_t = 0) &\quad \text{by definition} \\
  &= \frac{\lambda^0 \, e^{\lambda t}}{0!} \\
  &= e^{\lambda t} \\
  &= 1 - (1-e^{\lambda t}) &\quad \text{1 - CDF of } \operatorname{Expo}(\lambda) \\\\
  \\
  \Rightarrow &\text{time until first event } \sim \operatorname{Expo}(\lambda)
\end{align}

And so using this argument along with the memorylessness property, all of the other times between subsequent events $t_2, t_3, \dots , t_n$ are also $\operatorname{Expo}(\lambda)$

### Analogies: Geometric $\rightarrow$ Negative binomial (discrete); Exponential $\rightarrow$ Gamma (continuous)

So the interarrival time for events in a Poisson process are i.i.d. $\operatorname{Expo}(\lambda)$.

But what if we want to know $t_n$, the time of the $n^{th}$ arrival?

\begin{align}
  T_n &= \sum_{i=1}^{n} X_1, X_2, \dots , X_n &\quad \text{where } X_i \text{ is i.i.d. } \operatorname{Expo}(\lambda) \\\\
  &\sim \operatorname{Gamma}(n, \lambda) &\quad \text{ assuming } n \text{ is an integer}
\end{align}

Recall the relation between the geometric and negative binomial distributions:
* In geometric $\operatorname{Geom}(p)$, we count discrete time until $1^{st}$ success
* In negative binomial $\operatorname{NB}(n, p)$, we count discrete time until $n^{th}$ success ($\sum_{i=1}^{n} X_i \text{ where } X_i \text{ i.i.d., } X_i \sim \operatorname{Geom}(\lambda)$ )

Well, there's is something analogous between the exponential and Gamma distributions:
* In exponential $Expo(\lambda)$, we count continuous time until $1^{st}$ success
* In Gamma, we count continuous time until $n^{th}$ success ($\sum_{i=1}^{n}  X_i \text{ where } X_i \text{ i.i.d, } X_i \sim \operatorname{ Expo}(\lambda)$ )

----

## Moments of Gamma

#### $X \sim \operatorname{Gamma}(\alpha, 1)$

Let $X \sim \operatorname{Gamma}(\alpha,1)$

Directly using LOTUS:

\begin{align}
  \mathbb{E}(X^c) &= \int_{0}^{\infty} \frac{1}{\Gamma(a)} \,x^c \, x^{\alpha} \, e^{-x} \, \frac{dx}{x} \\
  &= \int_{0}^{\infty} \frac{1}{\Gamma(\alpha)} \, x^{a+c} \, e^{-x} \, \frac{dx}{x} \\
  &= \frac{\Gamma(\alpha+c)}{\Gamma(\alpha)} &\quad \text{, where } \alpha+x \gt 0 \\\\
  \\
  1^{st} \text{ moment (mean) of } \operatorname{Gamma}(n,1) &= \mathbb{E}(X) \\
  &= \frac{\Gamma(\alpha+1)}{\Gamma(\alpha)} \\
  &= \frac{\alpha \, \Gamma(\alpha)}{\Gamma(\alpha)} \\
  &= \boxed{ \alpha } \\\\
  \\
  2^{nd} \text{ moment } \operatorname{Gamma}(n,1) &= \mathbb{E}(X^2) \\
  &= \frac{\Gamma(\alpha+2)}{\Gamma(\alpha)} \\
  &= \frac{(\alpha+1) \, \Gamma(\alpha+1)}{\Gamma(\alpha)} \\
  &= \frac{(\alpha+1)\alpha \, \Gamma(\alpha)}{\Gamma(\alpha)} \\
  &= \alpha^2 + \alpha \\
  \\
  \Rightarrow \operatorname{Var} \text{ of } \operatorname{Gamma}(n,1) &=  \mathbb{E}(X^2) - \left( \mathbb{E}(X)^2 \right) \\
  &= \alpha^2 + \alpha - \alpha^2 \\
  &= \boxed{ \alpha  }
\end{align}

Let $X \sim \operatorname{Gamma}(\alpha, \lambda)$

Applying what we know from the case of above and using LOTUS:

\begin{align}
  \mathbb{E}(X^c) &= \int_{0}^{\infty} \frac{1}{\Gamma(\alpha)} \,x^c \, (\lambda x)^{\alpha} \, e^{-\lambda x} \, \frac{dx}{\lambda x} \\
  &= \int_{0}^{\infty} \frac{1}{\Gamma(\alpha)} \, x^{\alpha+c} \, \lambda^{\alpha} \, e^{-\lambda x} \, \frac{dx}{\lambda \, x} \\
  &= \int_{0}^{\infty} \frac{1}{\Gamma(\alpha)} \, (\lambda \, x)^{\alpha+c} \, e^{-\lambda x} \, \frac{dx}{\lambda \, x} \, \frac{1}{\lambda^c} \\
  &= \frac{\Gamma(\alpha+c)}{\Gamma(\alpha) \, \lambda^c} &\quad \text{, where } \alpha+x \gt 0 \\\\
  \\
  1^{st} \text{ moment (mean) of } \operatorname{Gamma}(\alpha,\lambda) &= \mathbb{E}(X) \\
  &= \frac{\Gamma(\alpha+1)}{\Gamma(\alpha) \, \lambda} \\
  &= \frac{\alpha \, \Gamma(\alpha)}{\Gamma(\alpha) \, \lambda} \\
  &= \boxed{ \frac{\alpha}{\lambda} } \\\\
  \\
  2^{nd} \text{ moment } \operatorname{Gamma}(\alpha,\lambda) &= \mathbb{E}(X^2) \\
  &= \frac{\Gamma(\alpha+2)}{\Gamma(\alpha) \, \lambda^2} \\
  &= \frac{(\alpha+1) \, \Gamma(\alpha+1)}{\Gamma(\alpha) \, \lambda^2} \\
  &= \frac{(\alpha+1)\alpha \, \Gamma(\alpha)}{\Gamma(\alpha) \, \lambda^2} \\
  &= \frac{(\alpha+1)\alpha}{\lambda^2} \\
  \\
  \Rightarrow \operatorname{Var} \text{ of } \operatorname{Gamma}\alpha,\lambda) &=  \mathbb{E}(X^2) - \left( \mathbb{E}(X)^2 \right) \\
  &= \frac{(\alpha+1)\alpha}{\lambda^2} - \left(  \frac{\alpha}{\lambda} \right)^2 \\
  &= \frac{\alpha + \alpha^2 - \alpha^2}{\lambda^2} \\
  &= \boxed{  \frac{\alpha}{\lambda^2} }
\end{align}

----

View [Lecture 24: Gamma distribution and Poisson process | Statistics 110](http://bit.ly/2CyadAf) on YouTube.


# Solving a Laplace problem with Dirichlet boundary conditions

### Background

In this tutorial, we will solve a simple Laplace problem inside the unit sphere with Dirichlet boundary conditions. Let $\Omega$ be the unit sphere with boundary $\Gamma$. Let $\nu$ be the outward pointing normal on $\Gamma$. The PDE and boundary conditions are given by

\begin{align}
\Delta u &= 0&&\text{in }\Omega,\\
u &= f&&\text{on }\Gamma.
\end{align}

The boundary data is a source $\hat{u}$ located at the point $(0.9,0,0)$.
$$
\hat{u}(\mathbf x)=\frac{1}{4\pi\sqrt{(x-0.9)^2+y^2+z^2}}.
$$

For this example, we will use a direct integral equation of the first kind. Let
$$
g(\mathbf x,\mathbf y) = \frac{1}{4\pi |\mathbf x-\mathbf y|}
$$
be the Green's function for Laplace in three dimensions. From Green's representation theorem, it follows that every harmonic function $u$ in $\Omega$ satisfies

$$
u(\mathbf x) = \int_{\Gamma} g(\mathbf x,\mathbf y)\frac{\partial u(\mathbf y)}{\partial \nu(\mathbf{y})}\mathrm{d}\mathbf y-\int_{\Gamma}\frac{\partial g(\mathbf x,\mathbf y)}{\partial \nu(\mathbf{y})}u(\mathbf y)\mathrm{d}\mathbf y,~\mathbf x\in\Omega\setminus\Gamma
$$

or equivalantly

$$
u(\mathbf x) = \left[\mathcal{V}\frac{\partial u(\mathbf y)}{\partial \nu(\mathbf{y})}\right] (\mathbf{x}) - \left[\mathcal{K}u\right] (\mathbf{x}),~\mathbf x\in\Omega\setminus\Gamma,
$$

where $\mathcal{V}$ and $\mathcal{K}$ are the single and double layer potential operators.

Taking the limit $\mathbf x\rightarrow \Gamma$ we obtain the boundary integral equation

$$
\left[\mathsf{V}\frac{\partial u}{\partial n}\right] (\mathbf x)=\left[(\tfrac12\mathsf{Id}+\mathsf{K})u\right] (\mathbf x),~\mathbf x\in\Gamma.
$$

Here, $\mathsf{V}$ and $\mathsf{K}$ are the single and double layer boundary operators.

### Implementation

We now demonstrate how to solve this problem with Bempp. We begin by importing Bempp and NumPy.

```python
import bempp.api
import numpy as np
```

We next define a mesh or grid. For this problem, we will use the built-in function `sphere` that defines a simple spherical grid. Details of other available grids can be found in the grids tutorial.

```python
grid = bempp.api.shapes.sphere(h=0.1)
```

We now define the spaces. For this example we will use two spaces: the space of continuous, piecewise linear functions; and the space of piecewise constant functions. The space of piecewise constant functions has the right smoothness for the unknown Neumann data. We will use continuous, piecewise linear functions to represent the known Dirichlet data.

```python
dp0_space = bempp.api.function_space(grid, "DP", 0)
p1_space = bempp.api.function_space(grid, "P", 1)
```

We can now define the operators. We need the identity, single layer, and double layer boundary operator.

```python
identity = bempp.api.operators.boundary.sparse.identity(
    p1_space, p1_space, dp0_space)
dlp = bempp.api.operators.boundary.laplace.double_layer(
    p1_space, p1_space, dp0_space)
slp = bempp.api.operators.boundary.laplace.single_layer(
    dp0_space, p1_space, dp0_space)
```

We now define the GridFunction object on the sphere grid that represents the Dirichlet data.

```python
@bempp.api.real_callable
def dirichlet_data(x, n, domain_index, result):
    result[0] = 1./(4 * np.pi * ((x[0] - .9)**2 + x[1]**2 + x[2]**2)**(0.5))

dirichlet_fun = bempp.api.GridFunction(p1_space, fun=dirichlet_data)
```

We next assemble the right-hand side of the boundary integral equation, given by $$(\tfrac12\mathsf{Id}+\mathsf{K})u.$$

```python
rhs = (.5 * identity + dlp) * dirichlet_fun
```

We now solve the linear system using a conjugate gradient (CG) method.

```python
neumann_fun, info = bempp.api.linalg.cg(slp, rhs, tol=1E-3)
```

We now want to provide a simple plot of the solution in the $(x,y)$ plane for $z=0$. We first define points at which to plot the solution.

```python
n_grid_points = 150
plot_grid = np.mgrid[-1:1:n_grid_points*1j, -1:1:n_grid_points*1j]
points = np.vstack((plot_grid[0].ravel(),
                    plot_grid[1].ravel(),
                    np.zeros(plot_grid[0].size)))
```

The variable `points` now contains in its columns the coordinates of the evaluation points. We can now use Green's representation theorem to evaluate the solution on these points. Note in particular the last line of the following code. It is a direct implementation of Green's representation theorem.

```python
slp_pot = bempp.api.operators.potential.laplace.single_layer(
    dp0_space, points)
dlp_pot = bempp.api.operators.potential.laplace.double_layer(
    p1_space, points)
u_evaluated = slp_pot * neumann_fun - dlp_pot * dirichlet_fun
```

We now plot the 2D slice of the solution. For a full three dimensional visualization, Bempp can export the data to Gmsh. Since the solution decays quickly, we use a logarithmic plot.

```python
# The next command ensures that plots are shown within the IPython notebook
%matplotlib inline

# Filter out solution values that are associated with points outside the unit circle.
u_evaluated = u_evaluated.reshape((n_grid_points,n_grid_points))
radius = np.sqrt(plot_grid[0]**2 + plot_grid[1]**2)
u_evaluated[radius>1] = np.nan

# Plot the image
import matplotlib
matplotlib.rcParams['figure.figsize'] = (5.0, 4.0)

from matplotlib import pylab as plt

plt.imshow(np.log(np.abs(u_evaluated.T)), extent=(-1,1,-1,1))
plt.title('Computed solution')
plt.colorbar()
```


# CasADi demo

## What is CasADi?

 * A tool for quick & efficient implementation of algorithms for dynamic optimization
 * Open source, LGPL-licensed,    casadi.org
 * C++ / C++11
 * Interfaces to Python, Haskell, (Matlab?)
 * Numerical backends: IPOPT, Sundials, ...
 * Developers in group of Moritz Diehl:
   * Joel Andersson
   * Joris Gillis
   * Greg Horn

## Outline of demo

 * Scalar expression (SX) graphs
 * Functions of SX graphs
 * Matrices of scalar expressions
 * Automatic differentiation (AD)
 * Integrators
 * Matrix expression (MX) graphs
 * Functions of MX graphs
 * Solving an optimal control problem

## Scalar expression (SX) graphs

```python
from __future__ import print_function
from pylab import *
from casadi import *
from casadi.tools import *  # for dotdraw
from IPython.display import Image, display
%matplotlib inline

def view_dot(graph):
   plt = Image(graph.create_png())
   display(plt)
```

```python
x = SX.sym("x")  # scalar symbolic primitives
y = SX.sym("y")

z = x*sin(x+y)   # common mathematical operators
```

```python
print(z)
```

```python
view_dot(dotgraph(z,direction="BT"))
```

```python
J = jacobian(z,x)
print(J)
```

```python
view_dot(dotgraph(J,direction="BT"))
```

> Note 1: subexpressions are shared.
>
>   Graph $\leftrightarrow$ Tree
>
>  Different from Maple, Matlab symbolic, sympy, ...

A (very) little bit of Computer Algebra

```python
print(x*y/x-y)
```

```python
H = hessian(z,x)
print(H)
```

## Functions of SX graphs

Sort graph into algorithm

```python
f = Function("f",[x,y],[z])

f.disp(True)
```

> Note 2: re-use of tape variables: live-variables

```python
print(f(1.2,3.4))
```

```python
print(f(1.2,x+y))
```

```python
f.generate("f")
print(open("f.c").read())
```

## Matrices of scalar expressions

```python
A = SX.sym("A",3,3)
B = SX.sym("B",3)
print(A)
```

```python
print(solve(A,B))
```

```python
print(trace(A))   # Trace
```

```python
print(mtimes(A,B))   # Matrix multiplication
#print(A @ B)        # Matrix multiplication in Python3
```

```python
print(norm_fro(A))  # Frobenius norm
```

```python
print(A[2,:])     # Slicing
```

> Rule 1: Everything is a matrix

```python
print(A.shape, z.shape)
```

```python
I = SX.eye(3)
print(I)
```

```python
Ak = kron(I,A)
print(Ak)
```

> Rule 1: Everything is a sparse matrix

```python
Ak.sparsity().spy()
```

```python
A.sparsity().spy()
```

```python
z.sparsity().spy()
```

## Automatic differentiation (AD)

Consider an ode:

\begin{equation}
\dot{p} = (1 - q^2)p-q+u
\end{equation}
\begin{equation}
\dot{q} = p
\end{equation}
\begin{equation}
\dot{c} = p^2+q^2+u^2
\end{equation}

```python
t = SX.sym("t")    # time
u = SX.sym("u")    # control
p = SX.sym("p")
q = SX.sym("q")
c = SX.sym("c")
x = vertcat(p,q,c) # state
```

```python
ode = vertcat((1 - q**2)*p - q + u, p, p**2+q**2+u**2)
print(ode, ode.shape)
```

```python
J = jacobian(ode,x)
print(J)
```

```python
f = Function("f",[t,u,x],[ode])

ffwd = f.forward(1)

fadj = f.reverse(1)

# side-by-side printing
print('{:*^24} || {:*^28} || {:*^28}'.format("f","ffwd","fadj"))
def short(f):
    import re
    return re.sub(r", a\.k\.a\. \"(\w+)\"",r". \1",f.str(True).replace(", No description available","").replace("Input ","").replace("Output ",""))
for l in zip(short(f).split("\n"),short(ffwd).split("\n"),short(fadj).split("\n")):
  print('{:<24} || {:<28} || {:<28}'.format(*l))
```

Performing **forward sweeps** gives the **columns** of J

```python
print(I)
```

```python
for i in range(3):
  print(ffwd(t,u,x, ode, 0,0,I[:,i]))
```

```python
print(J)
```

Performing **adjoint sweeps** gives the **rows** of J

```python
for i in range(3):
  print(fadj(t,u,x, ode, I[:,i])[2])
```

Often, you can do better than slicing with unit vectors

> Note 3: CasADi does graph coloring for efficient sparse jacobians

## Integrators

$\dot{x}=f(x,u,t)$ with $x = [p,q,c]^T$

```python
f = {'x':x,'t':t,'p':u,'ode':ode}
```

Construct an integrating block $x_{k+1} = \Phi(f;\Delta t;x_k,u_k)$

```python
tf = 10.0
N = 20
dt = tf/N
```

```python
Phi = integrator("Phi","cvodes",f,{"tf":dt})

x0 = DM([0,1,0])

print(Phi(x0=x0))
```

```python
x = x0
xs = [x]

for i in range(N):
  x = Phi(x0=x)["xf"]

  xs.append(x)
```

```python
plot(horzcat(*xs).T)
legend(["p","q","c"])
```

> Rule 2: Everything is a Function  (see http://docs.casadi.org)

## Matrix expression (MX) graphs

> Note 4: this is what makes CasADi stand out among AD tools

Recall

```python
n = 3

A = SX.sym("A",n,n)
B = SX.sym("B",n,n)
C = mtimes(A,B)
print(C)
view_dot(dotgraph(C,direction='BT'))
```

What if you **don't want** to expand into scalar operations?  ( avoid $O(n^3)$ storage)

```python
A = MX.sym("A",n,n)
B = MX.sym("B",n,n)
C = mtimes(A,B)
print(C)
view_dot(dotgraph(C,direction='BT'))
```

What if you **cannot** expand into matrix operations?  ( numerical algorithm )

```python
C = solve(A,B)
print(C)
view_dot(dotgraph(C,direction='BT'))
```

```python
X0 = MX.sym("x",3)

XF = Phi(x0=X0)["xf"]
print(XF)
```

```python
expr = sin(XF)+X0
view_dot(dotgraph(expr,direction='BT'))
```

## Functions of MX graphs

```python
F = Function("F",[X0],[  expr  ])
print(F)
```

```python
print(F(x0))
```

```python
J = Function("J",[X0],[  jacobian(expr,X0)  ])

print(J(x0))
```

This shows how an integrator-call can be embedded in matrix graph.

More possibilities:  external compiled library, a call to Matlab/Scipy

## Solving an optimal control problem

\begin{equation}
\begin{array}{cl}
\underset{p(.),q(.),u(.)}{\text{minimize}}  & \displaystyle \int_{0}^{T}{ p(t)^2 + q(t)^2 + u(t)^2 dt} \\\\
\text{subject to}
& \dot{p} = (1 - q^2)p-q+u   \\\\
& \dot{q} = p \\\\
& p(0) = 0, q(0) = 1 \\\\
&-1 \le u(t) \le 1
\end{array}
\end{equation}

Remember, $\dot{x}=f(x,u,t)$ with $x = [p,q,c]^T$

\begin{equation}
\begin{array}{cl}
\underset{x(.),u(.)}{\text{minimize}}  & c(T) \\\\
\text{subject to}
& \dot{x} = f(x,u) \\\\
& p(0) = 0, q(0) = 1, c(0)= 0 \\\\
&-1 \le u(t) \le 1
\end{array}
\end{equation}

Discretization with multiple shooting

\begin{equation}
\begin{array}{cl}
\underset{x_{\bullet},u_{\bullet}}{\text{minimize}}  & c_N \\\\
\text{subject to}
& x_{k+1} - \Phi(x_k,u_k) = 0 , \quad \quad k = 0,1,\ldots, (N-1) \\\\
& p_0 = 0, q_0 = 1, c_0 = 0 \\\\
&-1 \le u_k \le 1  , \quad \quad k = 0,1,\ldots, (N-1)
\end{array}
\end{equation}

Cast as NLP

\begin{equation}
\begin{array}{cl}
\underset{X}{\text{minimize}}  & F(X,P) \\\\
\text{subject to}
& \text{lbx} \le X \le \text{ubx} \\\\
& \text{lbg} \le G(X,P) \le \text{ubg} \\\\
\end{array}
\end{equation}

```python
X = struct_symMX([
     (
      entry("x", repeat=N+1, struct=struct(["p","q","c"]) ),
      entry("u", repeat=N)
     )
    ])
```

X is a symbolic matrix primitive, but with fancier indexing

```python
print(X.shape)
print((N+1)*3+N)
```

Demo: $\Phi(x_0,u_0)$

```python
Xf = Phi( x0=X["x",0],p=X["u",0] )["xf"]

print(Xf)
```

$ x_{k+1} - \Phi(x_k,u_k) = 0 , \quad \quad k = 0,1,\ldots, (N-1)$

```python
g = [] # List of constraint expressions

for k in range(N):
  Xf = Phi( x0=X["x",k],p=X["u",k] )["xf"]
  g.append( X["x",k+1]-Xf )
```

```python
obj = X["x",N,"c"] # c_N

nlp = {"x":X, "g": vcat(g), "f": obj}

print(nlp)
```

Block structure in the constraint Jacobian

```python
jacG = jacobian(nlp["g"],nlp["x"])

S = jacG.sparsity()

print(S.shape)

DM.ones(S)[:20,:20].sparsity().spy()
```

Recall

\begin{equation}
\begin{array}{cl}
\underset{X}{\text{minimize}}  & F(X,P) \\\\
\text{subject to}
& \text{lbx} \le X \le \text{ubx} \\\\
& \text{lbg} \le G(X,P) \le \text{ubg} \\\\
\end{array}
\end{equation}

```python
solver = nlpsol("solver","ipopt",nlp)

lbx = X(-inf)
ubx = X(inf)

lbx["u",:] = -1; ubx["u",:] = 1   #   -1 <= u(t) <= 1

lbx["x",0] = ubx["x",0] = x0      # Initial condition
```

```python
sol_out = solver(
           lbg = 0, # Equality constraints for shooting constraints
           ubg = 0, #    0 <= g <= 0
           lbx = lbx,
           ubx = ubx)
```

```python
print(sol_out["x"])
```

```python
sol = X(sol_out["x"])
```

```python
plot(horzcat(*sol["x",:]).T)
```

```python
step(range(N),sol["u",:])
```

## Wrapping up

Showcase: kite-power optimization by Greg Horn, using CasADi backend

```python
from IPython.display import YouTubeVideo
YouTubeVideo('tmjIBpb43j0')

```

```python
YouTubeVideo('SW6ZJzcMWAk')
```

Distinction with other software:

        ACADOtoolkit	CasADi

Black-box solver

Standard-form OCP

Good at small-scale real-time NMPC

Easy to get started

Write your own solver using a pool of building-blocks

No limitations on formulation

Good at large-scale OCP

Easy to extend

        Other operator-overloading AD tools	CasADi

Scalar graphs only, checkpointing

Scalar and matrix graphs

Batteries included: Ipopt, Sundials

Closest similarity: AMPL


```python
# Header starts here.
from sympy.physics.units import *
from sympy import *

# Rounding:
import decimal
from decimal import Decimal as DX
from copy import deepcopy
def iso_round(obj, pv, rounding=decimal.ROUND_HALF_EVEN):
    import sympy
    """
    Rounding acc. to DIN EN ISO 80000-1:2013-08
    place value = Rundestellenwert
    """
    assert pv in set([
        # place value   #  round to:
        1,              #  1
        0.1,            #  1st digit after decimal
        0.01,           #  2nd
        0.001,          #  3rd
        0.0001,         #  4th
        0.00001,        #  5th
        0.000001,       #  6th
        0.0000001,      #  7th
        0.00000001,     #  8th
        0.000000001,    #  9th
        0.0000000001,   # 10th
        ])
    objc = deepcopy(obj)
    try:
        tmp = DX(str(float(objc)))
        objc = tmp.quantize(DX(str(pv)), rounding=rounding)
    except:
        for i in range(len(objc)):
            tmp = DX(str(float(objc[i])))
            objc[i] = tmp.quantize(DX(str(pv)), rounding=rounding)
    return objc

# LateX:
kwargs = {}
kwargs["mat_str"] = "bmatrix"
kwargs["mat_delim"] = ""
# kwargs["symbol_names"] = {FB: "F^{\mathsf B}", }

# Units:
(k, M, G ) = ( 10**3, 10**6, 10**9 )
(mm, cm) = ( m/1000, m/100 )
Newton = kg*m/s**2
Pa     = Newton/m**2
MPa    = M*Pa
GPa    = G*Pa
kN     = k*Newton
deg    = pi/180

half = S(1)/2

# Header ends here.
#
# https://colab.research.google.com/github/kassbohm/wb-snippets/blob/master/ipynb/WB-Klein/5/5.9_cc.ipynb

pprint("\nSolution 1:")
A, F = var("A, F")
EI, l = var("EI, l")
sub_list = [
    (F,  1*Newton),
    (EI, 1*Newton*m**2),
    (l, 1*m),
    ]

l2 = l*l
l3 = l*l*l

pl, w, pr, A = var("psi_l, w, psi_r, A")

eq1 = Eq( EI/l3   * (  4*l2 *pl +  6*l * w) , 0 )
eq2 = Eq( EI/l3   * (  6*l  *pl + 12   * w) , + A + F/2 )
eq3 = Eq( EI/2/l3 * (  4*l2 *pr -  6*l * w) , 0 )
eq4 = Eq( EI/2/l3 * ( -6*l  *pr + 12   * w) , - A + F/2 )
eqns = [eq1, eq2, eq3, eq4]
unks = [pl, w, pr, A]
sol = solve(eqns, unks)
pprint(sol)
pl = sol[pl]
w = sol[w]
pr = sol[pr]
A = sol[A]

pprint("\nψₗ, ψᵣ / rad:")
for s in [pl, pr]:
    s = s.subs(sub_list)
    pprint(s)

pprint("\nw / m:")
w = w.subs(sub_list)
w /= m
pprint(w)

pprint("\nSolution 2:")

EI = 1 *Newton*m**2
l = 1 *m
l2 = l*l
l3 = l*l*l

K = EI/l**3
K *= Matrix([
[  18 ,  6*l, -3*l  ],
[  6*l, 4*l2,  0    ],
[ -3*l,   0,  2*l2  ],
])

# pprint("\nStiffness matrix / N:")
# pprint(K/Newton)

w2, p2m, p2p = var("w₂, ψ₂⁻, ψ₂⁺")
u = Matrix([w2, p2m, p2p])
f = Matrix([1*Newton,0,0])

eq = Eq(K*u , f)
sol = solve(eq, [w2, p2m, p2p])

pprint(sol)

# Solution 1:
# ⎧              2          2          3⎫
# ⎪   F      -F⋅l        F⋅l      2⋅F⋅l ⎪
# ⎨A: ─, φₗ: ──────, φᵣ: ────, w: ──────⎬
# ⎪   6       3⋅EI       3⋅EI      9⋅EI ⎪
# ⎩                                     ⎭
#
# φL, φR / rad:
# -1/3
# 1/3
#
# w / m:
# 2/9
#
# Solution 2:
#
# Stiffness matrix / N:
# ⎡  18                   ⎤
# ⎢─────     6       -3   ⎥
# ⎢meter                  ⎥
# ⎢                       ⎥
# ⎢  6    4⋅meter     0   ⎥
# ⎢                       ⎥
# ⎣ -3       0     2⋅meter⎦
# ⎧                         2⋅meter⎫
# ⎨p2m: -1/3, p2p: 1/3, w₂: ───────⎬
# ⎩                            9   ⎭

```