**Exercise set 13**
==============

>In this exercise set, we use the discrete the Fourier transform
>for analyzing some simple signals. Our focus here is on the practical
>use of the fast Fourier transform.

**Exercise 13.1**

In this exercise, we will investigate how the time resolution and the
length of the measured signal (in time) will influence the discrete Fourier transform.
For performing the Fourier transform, we will make use of the
[`scipy.fft`](https://docs.scipy.org/doc/scipy/reference/fft.html)
module which contains several methods for applying the fast Fourier transform (FFT).

Below, you will find some Python code for
calculating the FFT for a given signal.
```python
import numpy as np
from scipy.fft import fft, fftfreq
import matplotlib as mpl
from matplotlib import pyplot as plt
mpl.rcParams["savefig.format"] = 'pdf'
plt.style.use(['seaborn-talk', 'bmh'])

def run_fft(time, signal):
    """Do fft for the given signal.

    Parameters
    ----------
    time : numpy.array
        The times we have measured the signal at.
    signal : numpy.array
        The measured signal.

    Returns
    -------
    freq : numpy.array
        The positive frequencies.
    fourier : numpy.array
        The Fourier transform coefficients.
    amplitude : numpy.array
        The amplitude spectrum.
    power : numpy.array
        The power spectrum.

    """
    fourier = fft(signal)
    delta_t = time[1] - time[0]
    N = signal.size
    freq = fftfreq(N, delta_t)
    amplitude = np.abs(fourier / N)
    power = np.abs(fourier)**2
    # Normalize power:
    power = power / power.sum()
    return freq[:N//2], fourier, amplitude[:N//2], power[:N//2]

time = np.arange(0, 2, 0.01)
signal = np.sin(2.0 * 2.0 * np.pi * time)
freq, _, amplitude, _ = run_fft(time, signal)

fig, (ax1, ax2) = plt.subplots(constrained_layout=True, ncols=2)
ax1.plot(time, signal, lw=3)
ax2.plot(freq, amplitude, lw=3)
ax1.set(xlabel='Time (s)', ylabel='Signal amplitude')
ax2.set(xlabel='Frequency (Hz)', ylabel='Amplitude')
plt.show()
```

This method
returns several arrays of numbers:

*  The positive frequencies.

*  The Fourier transform coefficients (note that these are complex numbers).

*  The amplitude spectrum, obtained by taking the absolute value of the
complex numbers.

*  The power spectrum, obtained by taking the square of the amplitude spectrum.
When we analyze the Fourier transform, we are typically plotting the amplitude spectrum,
or the power spectrum, as a function of the frequencies.

**(a)** We will now consider the function,
\begin{equation}
y(t) = \sin (2 \times 2\pi t),
\label{eq:signal1}
\tag{1}\end{equation}

where $t$ is the time in seconds. This function has
a frequency of $2$ and a period of $\tfrac{1}{2}$.
Create three versions of this function by evaluating it for
different time ranges:

* (i)  from $0$ to $0.5$ (i.e. one period)
with a time step of $0.01$:
 `time = np.arange(0, 0.5, 0.01)`

* (ii) from $0$ to $1$ (i.e. two periods)
with a time step of $0.01$:
 `time = np.arange(0, 1, 0.01)`

* (iii) from $0$ to $5$ (i.e. $10$ periods)
with a time step of $0.01$:
 `time = np.arange(0, 5, 0.01)`

Create these three versions of the function in Eq. \eqref{eq:signal1},
calculate the FFT for each of them, and compare the amplitude spectra.
Based on your comparison, what is the effect of increasing the *length*
in time we sample the signal for?

```python
# Your code here
```

**Your answer to question 13.1(a):** *Double click here*

**(b)**  One "trick" we can do to increase the length of a signal is to
simply add zeros to the beginning and the end of the signal. This is
called "zero-padding", and we can do this with the method
[`pad`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.pad.html)
from `numpy`:
`zero_pad = np.pad(signal, pad_width=250)`.  This will add $250$ zeros to
the beginning of the signal, and $250$ zeros to the end of the signal. Consider again
the signal you created in point (i) in part **13.1(a)** of this
exercise. Pad this signal by adding $250$ zeros to the beginning and $250$ zeros to
the end
of the signal, and calculate the FFT for the padded signal. Comparing this to
the non-padded version, what seems to
be the effect when you pad the signal?

```python
# Your code here
```

**Your answer to question 13.1(b):** *Double click here*

**(c)**  We will continue with the signal in Eq. 1, but this time we will
investigate the effect of changing the sampling frequency (i.e. the time step/how often
we measure the signal in time).
Create $4$ new versions of this signal:

* (i) from $0$ to $0.5$ (i.e. one period)
with a time step of $0.01$:

  `time = np.arange(0, 0.5, 0.01)`

* (ii) from $0$ to $0.5$ (i.e. one period)
with a time step of $0.005$:

  `time = np.arange(0, 0.5, 0.005)`

* (iii) from $0$ to $1$ (i.e. two periods)
with a time step of $0.125$:

  `time = np.arange(0, 1, 0.125)`

* (iv) from $0$ to $10$ (i.e. $20$ periods)
with a time step of $0.3$:

  `time = np.arange(0, 10, 0.3)`

Create these $4$ signals, calculate the FFT for each of
them, and compare the amplitude spectra. Based on your results, what seems
to be the effect of changing the sampling frequency? Can you explain the
results you obtain for the $4$th signal (point (iv) above)?

```python
# Your code here
```

**Your answer to question 13.1(c):** *Double click here*

**(d)**  We will now consider a new signal which contains two frequencies:
\begin{equation}
y(t) = \sin (2 \times 2\pi t)
+ \sin (2.1 \times 2\pi t),
\label{eq:signal2}
\tag{2}\end{equation}

If you were to separate these two frequencies ($2$ and $2.1$) with the FFT method,
how long would you have to sample the signal for? Test this by
calculating the FFT of this signal (Eq. \eqref{eq:signal2}) and
plotting the amplitude spectrum.

```python
# Your code here
```

**Your answer to question 13.1(d):** *Double click here*

**Exercise 13.2**

The file `Data/unknown_signal.txt` contains a measurement
of a signal with noise. The first column are the times for which the signal was
measured, and the signal itself can be found in the second column. You know that this
signal, $y$, should have the following form as a function of time, $t$,

\begin{equation}
y(t) = a \sin (f_1 \times 2\pi t) + b \sin(f_2 \times 2\pi t),
\label{eq:signal2}
\tag{3}\end{equation}

where $a$ and $b$ are coefficients, and $f_1$ and $f_2$ are two
frequencies. Determine the coefficients and the two frequencies by
calculating the FFT of the given signal. Note: Since the FFT method will find
both negative and positive frequencies, you should also include the
negative frequencies. We can show that, in this case, this is effectively
the same as only
considering the positive frequencies, but doubling the
amplitudes.(This statement should be justified. One simple argument
is as follows. For Eq. \ref{eq:signal2}, the Fourier coefficients will be
purely imaginary. Further, the coefficients for a pair of frequencies, $f$ and $-f$,
will be complex conjugates of each other. The sum of the contributions for these
two frequencies will be $c_{f} \text{e}^{2\pi f t} + c_{-f} \text{e}^{-2 \pi f t}$,
where $c_f$ and $c_{-f}$ are the two coefficients.
Performing this summation results in  $2 A_f \sin (2 \pi f t)$,
where $A_f = \sqrt{||c_f||^2} = \sqrt{||c_{-f}||^2}$ is
the amplitude. Try to show this yourself, by using
[Euler's formula](https://en.wikipedia.org/wiki/Euler\%27s\_formula),
and assuming that $c_f=c_{-f}^\ast$ is purely imaginary (here "$^\ast$" denotes
the complex conjugate).)

```python
# Your code here
```

**Your answer to question 13.2:** *Double click here*

**Exercise 13.3**

In the lectures, we have mentioned the song
[Scar tissue](https://open.spotify.com/track/1G391cbiT3v3Cywg8T7DM1?autoplay=true&v=T)
by the Red Hot Chili Peppers (RHCP) and how one guitar string in the intro is apparently out of tune.
By nerdy inspection of RHCP music videos, we can see that the two first notes in the intro should be a
F note followed by a A note. However, the A note sounds slightly out of tune when comparing
a tuned guitar playing this note to the album version of the song. We
will here investigate if this is true and to what extent it is out of tune.

**(a)**  We will start by analyzing a known signal. The file `Data/440hz.txt.gz`
contains a sample of a A note at $440$ Hz. The first column of this file
is the time, and the second column is the sampled A note at the corresponding time.
Compute the FFT of this signal and verify that it contains the expected frequency.
Note: This file is compressed with
[gzip](https://en.wikipedia.org/wiki/Gzip) , but `numpy` can read this
file directly with the
[`loadtxt`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html)
method: `np.loadtxt("Data/440hz.txt.gz")`

```python
# Your code here
```

**Your answer to question 13.3(a):** *Double click here*

**(b)** A recording of the first few seconds of
[Scar tissue](https://open.spotify.com/track/1G391cbiT3v3Cywg8T7DM1?autoplay=true&v=T)
is shown in Fig 1. We will here focus on two parts of the signal, as
highlighted in the figure. In the first part of the signal, we expect to mainly
find the F note at $174.61$ Hz and harmonics which are multiples of this
frequency, for instance, $349.23\,\text{Hz} = 2 \times 174.61\,\text{Hz}$, and
$698.46\,\text{Hz} = 4 \times 174.61\,\text{Hz}$.(A table of notes and
frequencies can be found [here](https://pages.mtu.edu/suits/notefreqs.html))

In the second part of the signal, we expect to still find the F note, but also
the possible A note at $440$ Hz, as this is approximately when that note is
played. We will now try to identify the second note by comparing the
amplitude spectra obtained by taking the Fourier transform of the two parts separately.

* (i)  Calculate the FFT of the first part. The recording of this part
can be found in the file `Data/part1.txt.gz` where the first
column is the time and the second column is the amplitude. Can you find
the F note in this signal?

* (ii)  Calculate the FFT of the second part.
The recording of this part
can be found in the file `Data/part2.txt.gz` where the first
column is the time and the second column is the amplitude. Can you still find
the F note in this signal?

* (iii)  Compare the two amplitude spectra you have obtained. For your comparison,
it might be helpful to normalize the spectra so that the largest amplitude
is equal to $1$ in both spectra. When you compare the two spectra, can you
see a difference around $440$ Hz? Estimate the frequency
associated with the largest amplitude close to $440$ Hz in the amplitude
spectrum of part 2.

**Figure 1:** (Left) Amplitude in a recording of Scar Tissue as a function of time. (Right-top)
The first part (color red) of the signal we will analyze. (Right-bottom)
The second part (color red) of the signal we will analyze.

```python
# Your code here
```

**Your answer to question 13.3(b):** *Double click here*

**(c)**  Would you say that the frequency you identified above in part **13.3(b)**–(iii)
is out of tune with respect to the expected A note at $440$ Hz? Since we are, after all, at a
technical university, we can try to quantify this by calculating
the number of cents measuring the interval between the two frequencies.
The number of cents, $n$, is a unit of measure for the ratio
between two frequencies, $f_1$ and $f_2$,
\begin{equation}
n = 1200 \times \log_2 \frac{f_1}{f_2},
\tag{4}\end{equation}

and
[Wikipedia](https://en.wikipedia.org/wiki/Cent\_(music)\#Human_perception)  states that
we can distinguish a difference in pitch of about $5$–$6$ cents. Calculate the number
of cents for the two frequencies $f_1 = 440\,\text{Hz}$ and $f_2$ equal to the
frequency you identified above, and use this to support your conclusion.
Further, Wikipedia gives a [Just-noticeable difference](https://en.wikipedia.org/wiki/Just-noticeable_difference) of 3 Hz for frequencies below 500 Hz for
simple tones, and you can also use this to support your conclusion.

```python
# Your code here
```

**Your answer to question 13.3(c):** *Double click here*


# Basic Distributions

### A. Taylan Cemgil
### Boğaziçi University, Dept. of Computer Engineering

### Notebook Summary
* We review the notation and parametrization of densities of some basic distributions that are often encountered
* We show how random numbers are generated using python libraries
* We show some basic visualization methods such as displaying histograms

# Sampling From Basic Distributions

Sampling from basic distribution is easy using the numpy library.

Formally we will write

$x \sim p(X|\theta)$

where $\theta$ is the _parameter vector_, $p(X| \theta)$ denotes the _density_ of the random variable $X$ and $x$ is a _realization_, a particular draw from the density $p$.

The following distributions are building blocks from which more complicated processes may be constructed. It is important to have a basic understanding of these distributions.

### Continuous Univariate
* Uniform $\mathcal{U}$
* Univariate Gaussian $\mathcal{N}$
* Gamma $\mathcal{G}$
* Inverse Gamma $\mathcal{IG}$
* Beta $\mathcal{B}$

### Discrete
* Poisson $\mathcal{P}$
* Bernoulli $\mathcal{BE}$
* Binomial $\mathcal{BI}$
* Categorical $\mathcal{M}$
* Multinomial $\mathcal{M}$

### Continuous Multivariate (todo)
* Multivariate Gaussian $\mathcal{N}$
* Dirichlet $\mathcal{D}$

### Continuous Matrix-variate (todo)
* Wishart $\mathcal{W}$
* Inverse Wishart $\mathcal{IW}$
* Matrix Gaussian $\mathcal{N}$

## Sampling from the standard uniform $\mathcal{U}(0,1)$

For generating a single random number in the interval $[0, 1)$ we use the notation
$$
x_1 \sim \mathcal{U}(x; 0,1)
$$

In python, this is implemented as

```python
import numpy as np

x_1 = np.random.rand()

print(x_1)
```

    0.7605777935375794

We can also generate an array of realizations $x_i$ for $i=1 \dots N$,
$$
x_i \sim \mathcal{U}(x; 0,1)
$$

```python
import numpy as np

N = 5
x = np.random.rand(N)

print(x)

```

    [ 0.71384565  0.31621644  0.91739919  0.55668914  0.61991602]

For large $N$, it is more informative to display an histogram of generated data:

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

# Number of realizations
N = 50000
x = np.random.rand(N)

plt.hist(x, bins=20)
plt.xlabel('x')
plt.ylabel('Count')
plt.show()
```

$\newcommand{\indi}[1]{\left[{#1}\right]}$
$\newcommand{\E}[1]{\left\langle{#1}\right\rangle}$

We know that the density of the uniform distribution $\mathcal{U}(0,1)$ is

$$
\mathcal{U}(x; 0,1) = \left\{ \begin{array}{cc}  1 & 0 \leq x < 1 \\ 0 & \text{otherwise} \end{array}  \right.
$$
or using the indicator notation
$$
\mathcal{U}(x; 0,1) = \left[ x \in [0,1) \right]
$$

#### Indicator function

To write and manipulate discrete probability distributions in algebraic expression, the *indicator* function is useful:

$$ \left[x\right] = \left\{ \begin{array}{cc}
                                  1 & x\;\;\text{is true} \\
                                  0 & x\;\;\text{is false}
                                \end{array}
 \right.$$
This notation is also known as the Iverson's convention.

#### Aside: How to plot the density and the histogram onto the same plot?

In one dimension, the histogram is simply the count of the data points that fall to a given interval. Mathematically, we have
$j = 1\dots J$ intervals where $B_j = [b_{j-1}, b_j]$ and $b_j$ are bin boundries such that $b_0 < b_1 < \dots < b_J$.
$$
h(x) =  \sum_{j=1}^J \sum_{i=1}^N \indi{x \in B_j} \indi{x_i \in B_j}
$$
This expression, at the first sight looks somewhat more complicated than it really is. The indicator product just encodes the logical condition $x \in B_j$ __and__ $x_i \in B_j$. The sum over $j$ is just a convenient way of writing the result instead of specifying the histogram as a case by case basis for each bin. It is important to get used to such nested sums.

When the density $p(x)$ is given, the probability that a single realization is in bin $B_j$ is given by
$$
\Pr\left\{x \in B_j\right\} = \int_{B_j} dx p(x) = \int_{-\infty}^{\infty} dx \indi{x\in B_j} p(x) = \E{\indi{x\in B_j}}
$$
In other words, the probability is just the expectation of the indicator.

The histogram can be written as follows
$$
h(x) =  \sum_{j=1}^J \indi{x \in B_j} \sum_{i=1}^N \indi{x_i \in B_j}
$$

We define the counts at each bin as
$$
c_j \equiv \sum_{i=1}^N \indi{x_i \in B_j}
$$

If all bins have the same width, i.e., $b_j - b_{j-1} = \Delta$ for $\forall j$, and if $\Delta$ is sufficiently small we have

$$
\E{\indi{x\in B_j}} \approx p(b_{j-1}+\Delta/2) \Delta
$$

i.e., the probability is roughly the interval width times the density evaluated at the middle point of the bin. The expected value of the counts is

$$
\E{c_j} = \sum_{i=1}^N \E{\indi{x_i \in B_j}} \approx N \Delta p(b_{j-1}+\Delta/2)
$$

Hence, the density should be roughly

$$
p(b_{j-1}+\Delta/2) \approx \frac{\E{c_j} }{N \Delta}
$$

The $N$ term is intuitive but the $\Delta$ term is easily forgotten. When plotting the histograms on top of the corresponding densities, we should scale the normalized histogram ${ c_j }/{N}$ by dividing by $\Delta$.

```python
N = 1000

# Bin width
Delta = 0.02

# Bin edges
b = np.arange(0 ,1+Delta, Delta)

# Evaluate the density
g = np.ones(b.size)

# Draw the samples
u = np.random.rand(N)

counts,edges = np.histogram(u, bins=b)
plt.bar(b[:-1], counts/N/Delta, width=Delta)
#plt.hold(True)
plt.plot(b, g, linewidth=3, color='y')
#plt.hold(False)

plt.show()
```

The __plt.hist__ function (calling __np.histogram__) can do this calculation automatically if the option normed=True. However, when the grid is not uniform, it is better to write your own code to be sure what is going on.

```python
N = 1000
Delta = 0.05
b = np.arange(0 ,1+Delta, Delta)
g = np.ones(b.size)

u = np.random.rand(N)

#plt.hold(True)
plt.plot(b, g, linewidth=3, color='y')
plt.hist(u, bins=b, normed=True)
#plt.hold(False)

plt.show()
```

# Continuous Univariate Distributions
* Uniform $\mathcal{U}$
* Univariate Gaussian $\mathcal{N}$
$${\cal N}(x;\mu, v) =  \frac{1}{\sqrt{2\pi v}} \exp\left(-\frac12 \frac{(x - \mu)^2}{v}\right)  $$

* Gamma $\mathcal{G}$
$${\cal G}(\lambda; a, b) =  \frac{b^a \lambda^{a-1}}{\Gamma(a)} \exp( - b \lambda)$$

* Inverse Gamma $\mathcal{IG}$
$${\cal IG}(v; \alpha, \beta) =  \frac{\beta^\alpha}{\Gamma(\alpha) v^{\alpha+1}}  \exp(- \frac{\beta}{v}) $$

* Beta $\mathcal{B}$
$${\cal B}(r; \alpha, \beta) =  \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha) \Gamma(\beta) } r^{\alpha-1} (1-r)^{\beta-1}$$

In derivations, the distributions are often needed as building blocks. The following code segment prints the latex strings to be copied and pasted.

$\DeclareMathOperator{\trace}{Tr}$

```python
from IPython.display import display, Math, Latex, HTML
import notes_utilities as nut

print('Gaussian')
L = nut.pdf2latex_gauss(x=r'Z_{i,j}', m=r'\mu_{i,j}',v=r'l_{i,j}')
display(HTML(nut.eqs2html_table(L)))

print('Gamma')
L = nut.pdf2latex_gamma(x=r'u', a=r'a',b=r'b')
display(HTML(nut.eqs2html_table(L)))

print('Inverse Gamma')
L = nut.pdf2latex_invgamma(x=r'z', a=r'a',b=r'b')
display(HTML(nut.eqs2html_table(L)))

print('Beta')
L = nut.pdf2latex_beta(x=r'\pi', a=r'\alpha',b=r'\beta')
display(HTML(nut.eqs2html_table(L)))

```

    Gaussian

\begin{eqnarray}\mathcal{N}(Z_{i,j}; \mu_{i,j}, l_{i,j})\end{eqnarray}	\mathcal{N}(Z_{i,j}; \mu_{i,j}, l_{i,j})

\begin{eqnarray}\frac{1}{\sqrt{2\pi l_{i,j}} } \exp\left(-\frac12 \frac{(Z_{i,j} - \mu_{i,j} )^2}{l_{i,j}}\right)\end{eqnarray}	\frac{1}{\sqrt{2\pi l_{i,j}} } \exp\left(-\frac12 \frac{(Z_{i,j} - \mu_{i,j} )^2}{l_{i,j}}\right)

\begin{eqnarray}\exp\left(-\frac{1}{2}\frac{Z_{i,j}^2}{l_{i,j}} + \frac{Z_{i,j} \mu_{i,j} }{l_{i,j}} -\frac{1}{2}\frac{\mu_{i,j}^2}{l_{i,j}} -\frac{1}{2}\log(2{\pi}l_{i,j})    \right)\end{eqnarray}	\exp\left(-\frac{1}{2}\frac{Z_{i,j}^2}{l_{i,j}} + \frac{Z_{i,j} \mu_{i,j} }{l_{i,j}} -\frac{1}{2}\frac{\mu_{i,j}^2}{l_{i,j}} -\frac{1}{2}\log(2{\pi}l_{i,j})    \right)

\begin{eqnarray}-\frac{1}{2}\frac{Z_{i,j}^2}{l_{i,j}} + \frac{Z_{i,j} \mu_{i,j} }{l_{i,j}} -\frac{1}{2}\frac{\mu_{i,j}^2}{l_{i,j}} -\frac{1}{2}\log l_{i,j} -\frac{1}{2}\log 2\pi\end{eqnarray}	-\frac{1}{2}\frac{Z_{i,j}^2}{l_{i,j}} + \frac{Z_{i,j} \mu_{i,j} }{l_{i,j}} -\frac{1}{2}\frac{\mu_{i,j}^2}{l_{i,j}} -\frac{1}{2}\log l_{i,j} -\frac{1}{2}\log 2\pi

    Gamma

\begin{eqnarray}\mathcal{G}(u; a, b)\end{eqnarray}	\mathcal{G}(u; a, b)

\begin{eqnarray}\frac{ b^a u^{a-1}}{\Gamma(a)} \exp\left(-{b} {u}\right)\end{eqnarray}	\frac{ b^a u^{a-1}}{\Gamma(a)} \exp\left(-{b} {u}\right)

\begin{eqnarray}\exp(({a} - 1)\log u - {b}{u} - \log \Gamma({a}) + {a} \log {b})\end{eqnarray}	\exp(({a} - 1)\log u - {b}{u} - \log \Gamma({a}) + {a} \log {b})

\begin{eqnarray}({a} - 1)\log u - {b}{u} - \log \Gamma({a}) + {a} \log {b}\end{eqnarray}	({a} - 1)\log u - {b}{u} - \log \Gamma({a}) + {a} \log {b}

    Inverse Gamma

\begin{eqnarray}\mathcal{IG}(z; a, b)\end{eqnarray}	\mathcal{IG}(z; a, b)

\begin{eqnarray}\frac{ b^a }{\Gamma(a)z^{a+1}} \exp\left(-\frac{{b}}{{z}}\right)\end{eqnarray}	\frac{ b^a }{\Gamma(a)z^{a+1}} \exp\left(-\frac{{b}}{{z}}\right)

\begin{eqnarray}\exp(-({a} + 1)\log z - \frac{{b}}{{z}} - \log \Gamma({a}) + {a} \log {b})\end{eqnarray}	\exp(-({a} + 1)\log z - \frac{{b}}{{z}} - \log \Gamma({a}) + {a} \log {b})

\begin{eqnarray}-({a} + 1)\log z - \frac{{b}}{{z}} - \log \Gamma({a}) + {a} \log {b}\end{eqnarray}	-({a} + 1)\log z - \frac{{b}}{{z}} - \log \Gamma({a}) + {a} \log {b}

    Beta

\begin{eqnarray}\mathcal{B}(\pi; \alpha, \beta)\end{eqnarray}	\mathcal{B}(\pi; \alpha, \beta)

\begin{eqnarray}\frac{ \Gamma(\alpha+\beta) }{ \Gamma(\alpha) \Gamma(\beta) } \pi^{\alpha-1} (1-\pi)^{\beta-1}\end{eqnarray}	\frac{ \Gamma(\alpha+\beta) }{ \Gamma(\alpha) \Gamma(\beta) } \pi^{\alpha-1} (1-\pi)^{\beta-1}

\begin{eqnarray}\exp\left(({\alpha} - 1)\log \pi + ({\beta} - 1)\log (1-\pi) + \log\Gamma({\alpha}+{\beta}) - \log \Gamma({\alpha}) - \log \Gamma({\beta})\right)\end{eqnarray}	\exp\left(({\alpha} - 1)\log \pi + ({\beta} - 1)\log (1-\pi) + \log\Gamma({\alpha}+{\beta}) - \log \Gamma({\alpha}) - \log \Gamma({\beta})\right)

\begin{eqnarray}({\alpha} - 1)\log \pi + ({\beta} - 1)\log (1-\pi) + \log\Gamma({\alpha}+{\beta}) - \log \Gamma({\alpha}) - \log \Gamma({\beta})\end{eqnarray}	({\alpha} - 1)\log \pi + ({\beta} - 1)\log (1-\pi) + \log\Gamma({\alpha}+{\beta}) - \log \Gamma({\alpha}) - \log \Gamma({\beta})

We will illustrate two alternative ways for sampling from continuous distributions.

- The first method has minimal dependence on the numpy and scipy libraries. This is initially the preferred method. Only random variable generators and the $\log \Gamma(x)$ (__gammaln__) function is used and nothing more.

- The second method uses scipy. This is a lot more practical but requires knowing more about the internals of the library.

### Aside: The Gamma function $\Gamma(x)$
The gamma function $\Gamma(x)$ is the (generalized) factorial.
- Defined by
$$\Gamma(x) = \int_0^{\infty} t^{x-1} e^{-t}\, dt$$
- For integer $x$, $\Gamma(x) = (x-1)!$. Remember that for positive integers $x$, the factorial function can be defined recursively $x! = (x-1)! x $ for $x\geq 1$.
- For real $x>1$, the gamma function satisfies
$$
\Gamma(x+1) = \Gamma(x) x
$$
- Interestingly, we have
$$\Gamma(1/2) = \sqrt{\pi}$$
- Hence
$$\Gamma(3/2) = \Gamma(1/2 + 1) = \Gamma(1/2) (1/2) = \sqrt{\pi}/2$$
- It is available in many numerical computation packages, in python it is available as __scipy.special.gamma__.
- To compute $\log \Gamma(x)$, you should always use the implementation as __scipy.special.gammaln__. The gamma function blows up super-exponentially so numerically you should never evaluate $\log \Gamma(x)$ as
```python
import numpy as np
import scipy.special as sps
np.log(sps.gamma(x))  # Don't
sps.gammaln(x)        # Do
```
- A related function is the Beta function
$$B(x,y) = \int_0^{1} t^{x-1} (1-t)^{y-1}\, dt$$
- We have
$$B(x,y) = \frac{\Gamma(x)\Gamma(y)}{\Gamma(x+y)}$$
- Both $\Gamma(x)$ and $B(x)$ pop up as normalizing constant of the gamma and beta distributions.

#### Derivatives of $\Gamma(x)$
-   The derivatives of  $\log \Gamma(x)$ pop up quite often when fitting densities. The first derivative has a specific name,  often called the digamma function or the psi function.
$$
\Psi(x) \equiv \frac{d}{d x} \log \Gamma(x)
$$
- It is available as __scipy.special.digamma__ or __scipy.special.psi__

- Higher order derivatives of the $\log \Gamma(x)$ function (including digamma itself) are available as __scipy.special.polygamma__

```python
import numpy as np
import scipy.special as sps
import matplotlib.pyplot as plt

x = np.arange(0.1,5,0.01)

f = sps.gammaln(x)
df = sps.psi(x)

# First derivative of the digamma function
ddf = sps.polygamma(1,x)
# sps.psi(x) == sps.polygamma(0,x)

plt.figure(figsize=(8,10))
plt.subplot(3,1,1)
plt.plot(x, f, 'r')
plt.grid(True)
plt.xlabel('x')
plt.ylabel('log Gamma(x)')

plt.subplot(3,1,2)
plt.grid(True)
plt.plot(x, df, 'b')
plt.xlabel('x')
plt.ylabel('Psi(x)')

plt.subplot(3,1,3)
plt.plot(x, ddf, 'k')
plt.grid(True)
plt.xlabel('x')
plt.ylabel('Psi\'(x)')

plt.show()
```

#### Stirling's approximation

An important approximation to the factorial is the famous Stirling's approximation

\begin{align}
n! \sim \sqrt{2 \pi n}\left(\frac{n}{e}\right)^n
\end{align}

\begin{align}
\log \Gamma(x+1) \approx \frac{1}{2}\log(2 \pi) + x \log(x) - \frac{1}{2} \log(x)
\end{align}

```python
import matplotlib.pylab as plt
import numpy as np
from scipy.special import polygamma
from scipy.special import gammaln as loggamma
from scipy.special import psi

x = np.arange(0.001,6,0.001)
ylim = [-1,8]
xlim = [-1,6]
plt.plot(x, loggamma(x), 'b')
stir = x*np.log(x)-x +0.5*np.log(2*np.pi*x)
plt.plot(x+1, stir,'r')
plt.hlines(0,0,8)
plt.vlines([0,1,2],ylim[0],ylim[1],linestyles=':')
plt.hlines(range(ylim[0],ylim[1]),xlim[0],xlim[1],linestyles=':',colors='g')
plt.ylim(ylim)
plt.xlim(xlim)

plt.legend([r'\log\Gamma(x)',r'\log(x-1)'],loc=1)
plt.xlabel('x')
plt.show()
```

# Sampling from Continuous Univariate Distributions

## Sampling using numpy.random

```python

%matplotlib inline
import matplotlib.pyplot as plt

import numpy as np
from scipy.special import gammaln

def plot_histogram_and_density(N, c, edges, dx, g, title='Put a title'):
    '''
     N  : Number of Datapoints
     c  : Counts, as obtained from np.histogram function
     edges : bin edges, as obtained from np.histogram
     dx : The bin width
     g  : Density evaluated at the points given in edges
     title : for the plot
    '''
    plt.bar(edges[:-1], c/N/dx, width=dx)
#    plt.hold(True)
    plt.plot(edges, g, linewidth=3, color='y')
#    plt.hold(False)
    plt.title(title)

def log_gaussian_pdf(x, mu, V):
    return -0.5*np.log(2*np.pi*V) -0.5*(x-mu)**2/V

def log_gamma_pdf(x, a, b):
    return (a-1)*np.log(x) - b*x - gammaln(a) + a*np.log(b)

def log_invgamma_pdf(x, a, b):
    return -(a+1)*np.log(x) - b/x - gammaln(a) + a*np.log(b)

def log_beta_pdf(x, a, b):
    return - gammaln(a) - gammaln(b) + gammaln(a+b) + np.log(x)*(a-1) + np.log(1-x)*(b-1)

N = 1000

# Univariate Gaussian
mu = 2    # mean
V = 1.2   # Variance
x_normal = np.random.normal(mu, np.sqrt(V), N)

dx = 10*np.sqrt(V)/50
x = np.arange(mu-5*np.sqrt(V) ,mu+5*np.sqrt(V),dx)
g = np.exp(log_gaussian_pdf(x, mu, V))

#g = scs.norm.pdf(x, loc=mu, scale=np.sqrt(V))
c,edges = np.histogram(x_normal, bins=x)

plt.figure(num=None, figsize=(16, 5), dpi=80, facecolor='w', edgecolor='k')
plt.subplot(2,2,1)
plot_histogram_and_density(N, c, x, dx, g, 'Gaussian')

## Gamma
# Shape
a = 1.2
# inverse scale
b = 30

# Generate unit scale first than scale with inverse scale parameter b
x_gamma = np.random.gamma(a, 1, N)/b
dx = np.max(x_gamma)/500
x = np.arange(dx, 250*dx, dx)

g = np.exp(log_gamma_pdf(x, a, b))
c,edges = np.histogram(x_gamma, bins=x)

plt.subplot(2,2,2)
plot_histogram_and_density(N, c, x, dx, g, 'Gamma')

## Inverse Gamma
a = 3.5
b = 0.2

x_invgamma = b/np.random.gamma(a, 1, N)
dx = np.max(x_invgamma)/500
x = np.arange(dx, 150*dx, dx)
g = np.exp(log_invgamma_pdf(x,a,b))
c,edges = np.histogram(x_invgamma, bins=x)

plt.subplot(2,2,3)
plot_histogram_and_density(N, c, x, dx, g, 'Inverse Gamma')

## Beta
a = 0.5
b = 1
x_beta = np.random.beta(a, b, N)
dx = 0.01
x = np.arange(dx, 1, dx)
g = np.exp(log_beta_pdf(x, a, b))
c,edges = np.histogram(x_beta, bins=x)

plt.subplot(2,2,4)
plot_histogram_and_density(N, c, x, dx, g, 'Beta')

plt.show()
```

## Sampling using scipy.stats

```python
%matplotlib inline
import matplotlib.pyplot as plt

import numpy as np
import scipy.stats as scs

N = 2000

# Univariate Gaussian
mu = 2    # mean
V = 1.2   # Variance

rv_normal = scs.norm(loc=mu, scale=np.sqrt(V))
x_normal = rv_normal.rvs(size=N)

dx = 10*np.sqrt(V)/50
x = np.arange(mu-5*np.sqrt(V) ,mu+5*np.sqrt(V),dx)
g = rv_normal.pdf(x)

c,edges = np.histogram(x_normal, bins=x)

plt.figure(num=None, figsize=(16, 5), dpi=80, facecolor='w', edgecolor='k')
plt.subplot(2,2,1)
plot_histogram_and_density(N, c, x, dx, g, 'Gaussian')

## Gamma
a = 3.2
b = 30

# The following is equivalent to our parametrization of gamma, note the 1/b term
rv_gamma = scs.gamma(a, scale=1/b)
x_gamma = rv_gamma.rvs(N)
dx = np.max(x_gamma)/500
x = np.arange(0, 250*dx, dx)
g = rv_gamma.pdf(x)
c,edges = np.histogram(x_gamma, bins=x)

plt.subplot(2,2,2)
plot_histogram_and_density(N, c, x, dx, g, 'Gamma')

## Inverse Gamma
a = 3.5
b = 0.2

# Note the b term
rv_invgamma = scs.invgamma(a, scale=b)
x_invgamma = rv_invgamma.rvs(N)
dx = np.max(x_invgamma)/500
x = np.arange(dx, 150*dx, dx)
g = rv_invgamma.pdf(x)
c,edges = np.histogram(x_invgamma, bins=x)

plt.subplot(2,2,3)
plot_histogram_and_density(N, c, x, dx, g, 'Inverse Gamma')

## Beta
a = 0.7
b = 0.8

rv_beta = scs.beta(a, b)
x_beta = rv_beta.rvs(N)
dx = 0.02
x = np.arange(0, 1+dx, dx)
g = rv_beta.pdf(x)
c,edges = np.histogram(x_beta, bins=x)

plt.subplot(2,2,4)
plot_histogram_and_density(N, c, x, dx, g, 'Beta')

plt.show()
```

# Sampling from Discrete Densities
* Bernoulli $\mathcal{BE}$
$$
{\cal BE}(r; w)  =  w^r (1-w)^{1-r} \;\; \text{if} \; r \in \{0, 1\}
$$

* Binomial $\mathcal{BI}$
$${\cal BI}(r; L, w)  =   \binom{L}{r, (L-r)} w^r (1-w)^{L-r} \;\; \text{if} \; r \in \{0, 1, \dots, L\} $$

Here, the binomial coefficient is defined as
$$
\binom{L}{r, (L-r)} = \frac{N!}{r!(L-r)!}
$$

Note that
$$
{\cal BE}(r; w) = {\cal BI}(r; L=1, w)
$$

* Poisson $\mathcal{PO}$, with intensity $\lambda$
$${\cal PO}(x;\lambda) =  \frac{e^{-\lambda} \lambda^x}{x!} = \exp(x \log \lambda - \lambda - \log\Gamma(x+1)) $$

Given samples on nonnegative integers, we can obtain histograms easily using __np.bincount__.

```python
c = np.bincount(samples)
```

The functionality is equivalent to the following sniplet, while implementation is possibly different and more efficient.

```python
upper_bound = np.max()
c = np.zeros(upper_bound+1)
for i in samples:
    c[i] += 1
```

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

def plot_histogram_and_pmf(N, c, domain, dx, g, title='Put a title'):
    '''
     N  : Number of Datapoints
     c  : Counts, as obtained from np.bincount function
     domain : integers for each c, same size as c
     dx : The bin width
     g  : Density evaluated at the points given in edges
     title : for the plot
    '''
    plt.bar(domain-dx/2, c/N, width=dx)
#    plt.hold(True)
    plt.plot(domain, g, 'ro:', linewidth=3, color='y')
#    plt.hold(False)
    plt.title(title)

def log_poisson_pdf(x, lam):
    return -lam + x*np.log(lam) - gammaln(x+1)

def log_bernoulli_pdf(r, pr):
    return r*np.log(pr) + (1-r)*np.log(1 - pr)

def log_binomial_pdf(r, pr, L):
    return gammaln(L+1) - gammaln(r+1) - gammaln(L-r+1) + r*np.log(pr) + (L-r)*np.log(1 - pr)

N = 100
pr = 0.8

# For plots
bin_width = 0.3

# Bernoulli
L = 1
x_bern = np.random.binomial(n=L, p=pr, size=N)
c = np.bincount(x_bern, minlength=L+1)
g = np.exp(log_bernoulli_pdf(np.arange(L+1), pr))

plt.figure(figsize=(20,4))
plt.subplot(1,3,1)
plot_histogram_and_pmf(N, c, np.arange(L+1), bin_width, g, 'Bernoulli')
plt.xticks([0,1])

# Binomial
L = 10
pr = 0.7
x_binom = np.random.binomial(n=L, p=pr, size=N)
c = np.bincount(x_binom, minlength=L+1)
g = np.exp(log_binomial_pdf(np.arange(L+1), pr, L))

plt.subplot(1,3,2)
plot_histogram_and_pmf(N, c, np.arange(L+1), bin_width, g, 'Binomial')
plt.xticks(np.arange(L+1))

# Poisson
intensity = 10.5
x_poiss = np.random.poisson(intensity, size =N )
c = np.bincount(x_poiss)

x = np.arange(len(c))
g = np.exp(log_poisson_pdf(x, intensity))

plt.subplot(1,3,3)
plot_histogram_and_pmf(N, c, x, bin_width, g, 'Poisson')

```

## Bernoulli, Binomial, Categorical and Multinomial Distributions

The Bernoulli and Binomial distributions are quite simple and well known distributions on small integers, so it may come as a surprise that they have another, less obvious but arguably more useful representation as discrete multivariate densities. This representation makes the link to categorical distributions where there are more than two possible outcomes. Finally, all Bernoulli, Binomial or Categorical distributions are special cases of Multinomial distribution.

### Bernoulli
Recall the Bernoulli distribution $r \in \{0, 1\}$
$$
{\cal BE}(r; w)  =  w^r (1-w)^{1-r}
$$

We will define $\pi_0 = 1-w$ and $\pi_1 = w$, such that $\pi_0 + \pi_1 = 1$. The parameter vector is $\pi = (\pi_0, \pi_1)$

We will also introduce a positional encoding such that

\begin{eqnarray}
r = 0 & \Rightarrow & s = (1, 0) \\
r = 1 & \Rightarrow & s = (0, 1)
\end{eqnarray}

In other words $s = (s_0, s_1)$ is a 2-dimensional vector where
$$s_0, s_1 \in \{0,1\}\;\text{and}\; s_0 + s_1 = 1$$

We can now write the Bernoulli density

$$
p(s | \pi)  =  \pi_0^{s_0} \pi_1^{s_1}
$$

### Binomial
Similarly, recall the Binomial density where $r \in \{0, 1, \dots, L\}$

$${\cal BI}(r; L, w)  =   \binom{L}{r, (L-r)} w^r (1-w)^{L-r}  $$

We will again define $\pi_0 = 1-w$ and $\pi_1 = w$, such that $\pi_0 + \pi_1 = 1$. The parameter vector is $\pi = (\pi_0, \pi_1)$

\begin{eqnarray}
r = 0 & \Rightarrow & s = (L, 0) \\
r = 1 & \Rightarrow & s = (L-1, 1)\\
r = 2 & \Rightarrow & s = (L-2, 2)\\
\dots \\
r = L & \Rightarrow & s = (0, L)
\end{eqnarray}

where $s = (s_0, s_1)$ is a 2-dimensional vector where $$s_0, s_1 \in \{0,\dots,L\} \;\text{and}\; s_0 + s_1  = L$$

We can now write the Binomial density as
$$
p(s | \pi)  =  \binom{L}{s_0, s_1} \pi_0^{s_0} \pi_1^{s_1}
$$

### Categorical (Multinouilli)

One of the advantages of this new notation is that we can write the density even if the outcomes are not numerical. For example, the result of a single coin flip experiment when $r \in \{$ 'Tail', 'Head' $\}$ where the probability of 'Tail' is $w$ can be written as
$$
p(r | w)  =  w^{\indi{r=\text{'Tail'}}} (1-w)^{\indi{r=\text{'Head'}}}
$$

We define $s_0 = \indi{r=\text{'Head'}}$ and $s_1 = \indi{r=\text{'Tail'}}$, then the density can be written in the same form as
$$
p(s | \pi)  = \pi_0^{s_0} \pi_1^{s_1}
$$
where $\pi_0 = 1-w$ and $\pi_1 = w$.

More generally, when $r$ is from a set with $K$ elements, i.e., $r \in R = \{ v_0, v_1, \dots, v_{K-1} \}$ with probability of the event $r = v_k$ given as $\pi_k$, we define $s = (s_0, s_1, \dots, s_{K-1})$ for $k=0,\dots, K-1$

$$
s_k = \indi{r=v_k}
$$

Note that by construction, we have $\sum_k s_k = 1$.
The resulting density, known as the Categorical density, can be writen as

$$
p(s|\pi) = \pi_0^{s_0} \pi_1^{s_1} \dots \pi_{K-1}^{s_{K-1}}
$$

### Multinomial

When drawing from a categorical distribution, one chooses a single category from $K$ options with given probabilities. A standard model for this is placing a single ball into $K$ different bins. The vector $s = (s_0, s_1, \dots,s_k, \dots, s_{K-1})$ represents how many balls eack bin $k$ contains.

Now, place $L$ balls instead of one into $K$ bins with placing each ball idependently into bin $k$ where $k \in\{0,\dots,K-1\}$ with the probability $\pi_k$. The multinomial is the joint distribution of $s$ where $s_k$ is the number of balls placed into bin $k$.

The density will be denoted as

$${\cal M}(s; L, \pi)  =  \binom{L}{s_0, s_1, \dots, s_{K-1}}\prod_{k=0}^{K-1} \pi_k^{s_k} $$

Here $\pi \equiv [\pi_0, \pi_2, \dots, \pi_{K-1} ]$ is the probability vector and $L$ is referred as the _index parameter_.
Clearly, we have the normalization constraint $ \sum_k \pi_k = 1$ and realization of the counts $s$ satisfy
$ \sum_k s_k = L $.

Here, the _multinomial_ coefficient is defined as

$$\binom{L}{s_0, s_1, \dots, s_{K-1}} = \frac{L!}{s_0! s_1! \dots s_{K-1}!}$$

Binomial, Bernoulli and Categorical distributions are all special cases of the Multinomial distribution, with a suitable representation.

The picture is as follows:

~~~
|Balls/Bins | $2$ Bins | $K$ Bins |
|-------- | -------- | ---------|
| $1$ Ball  | Bernoulli ${\cal BE}$ | Categorical  ${\cal C}$ |
|-------- | -------- | ---------|
| $L$ Balls | Binomial ${\cal BI}$ | Multinomial ${\cal M}$ |
~~~

Murphy calls the categorical distribution ($1$ Ball, $K$ Bins) as the Multinoulli. This is non-standard but logical (and somewhat cute).

It is common to consider Bernoulli and Binomial as scalar random variables. However, when we think of them as special case of a Multinomial it is better to think of them as bivariate, albeit degenerate, random variables, as
illustrated in the following cell along with an alternative visualization.

```python
# The probability parameter
pr = 0.3

fig = plt.figure(figsize=(16,50), edgecolor=None)

maxL = 12
plt.subplot(maxL-1,2,1)
plt.grid(False)
# Set up the scalar binomial density as a bivariate density
for L in range(1,maxL):
    r = np.arange(L+1)
    p = np.exp(log_binomial_pdf(r, pr=pr, L=L))
    A = np.zeros(shape=(13,13))
    for s in range(L):
        s0 = s
        s1 = L-s
        A[s0, s1] = p[s]

    #plt.subplot(maxL-1,2,2*L-1)
#    plt.bar(r-0.25, p, width=0.5)
#    ax.set_xlim(-1,maxL)
#    ax.set_xticks(range(0,maxL))
    if True:
        plt.subplot(maxL-1,2,2*L-1)
        plt.barh(bottom=r-0.25, width=p, height=0.5)
        ax2 = fig.gca()
        pos = ax2.get_position()
        pos2 = [pos.x0, pos.y0, 0.04, pos.height]
        ax2.set_position(pos2)
        ax2.set_ylim(-1,maxL)
        ax2.set_yticks(range(0,maxL))
        ax2.set_xlim([0,1])
        ax2.set_xticks([0,1])
        plt.ylabel('s1')
        ax2.invert_xaxis()

    plt.subplot(maxL-1,2,2*L)
    plt.imshow(A,  interpolation='nearest', origin='lower',cmap='gray_r',vmin=0,vmax=0.7)
    plt.xlabel('s0')

    ax1 = fig.gca()

    pos = ax1.get_position()
    pos2 = [pos.x0-0.45, pos.y0, pos.width, pos.height]
    ax1.set_position(pos2)

    ax1.set_ylim(-1,maxL)
    ax1.set_yticks(range(0,maxL))
    ax1.set_xlim(-1,maxL)
    ax1.set_xticks(range(0,maxL))

plt.show()
```

The following cell illustrates sampling from the Multinomial density.

```python

# Number of samples
sz = 3

# Multinomial
p = np.array([0.3, 0.1, 0.1, 0.5])
K = len(p) # number of Bins
L = 20     # number of Balls

print('Multinomial with number of bins K = {K} and Number of balls L = {L}'.format(K=K,L=L))
print(np.random.multinomial(L, p, size=sz))

# Categorical
L = 1     # number of Balls

print('Categorical with number of bins K = {K} and a single ball L=1'.format(K=K))
print(np.random.multinomial(L, p, size=sz))

# Binomial
p = np.array([0.3, 0.7])
K = len(p) # number of Bins = 2
L = 20     # number of Balls

print('Binomial with two bins K=2 and L={L} balls'.format(L=L))
print(np.random.multinomial(L, p, size=sz))

# Bernoulli
L = 1     # number of Balls
p = np.array([0.3, 0.7])
K = len(p) # number of Bins = 2

print('Bernoulli, two bins and a single ball')
print(np.random.multinomial(L, p, size=sz))

```

    Multinomial with number of bins K = 4 and Number of balls L = 20
    [[ 6  3  2  9]
     [ 4  1  3 12]
     [ 7  1  5  7]]
    Categorical with number of bins K = 4 and a single ball L=1
    [[1 0 0 0]
     [0 0 1 0]
     [0 0 0 1]]
    Binomial with two bins K=2 and L=20 balls
    [[ 8 12]
     [ 4 16]
     [ 6 14]]
    Bernoulli, two bins and a single ball
    [[0 1]
     [0 1]
     [1 0]]


Fill in any place that says `# YOUR CODE HERE` or YOUR ANSWER HERE, as well as your name and collaborators below.
Grading for pre-lecture assignments is all or nothing. Partial credit is available for in-class assignments and checkpoints, but **only when code is commented**.

```python
NAME = ""
COLLABORATORS = ""
```

---

# Learning Objectives

This lecture will show you how to:
1. Solve 1st order ODEs with Euler's method
2. Solve 1st order ODEs with the Runge-Kutta method
3. Solve systems of 1st order ODEs
4. Solve 2nd order (and above) ODEs

```python
# imports
import numpy as np
import matplotlib.pyplot as plt

import grading_helper as _test
```

# Euler's Method

```python
%video _ulNf7dF1Fc
```

Summary:

Write a 1st order ODE in the form

$$\frac{dx}{dt}=f(x,t)\,.$$

Then, starting with **initial condition** $x_0=x(t_0)$, repeated apply

$$x(t+h)=x(t)+h\,f(x,t)\,,$$

with a small step size $h$.

```python

```

## Your Turn

Let
$$\frac{dx}{dt} = x\cos t\,,$$
with $x(0) = 1$.
1. Solve this equation by hand (separate variables and integrate). Plot $x(t)$ for $0 \le t \le 2\pi$.
2. Use Euler's method with $h=\pi/10$. Plot a dot at each value of $x$ that the method computes. Store these values in an array named `x`.

> The graphs will be similar, but not match exactly. The step size $h$ is pretty large.

```python
%%graded # 1 point

# YOUR CODE HERE
```

```python
%%tests

_test.equal(len(x), 21)
_test.equal(x[0], 1) # initial condition
_test.similar(x, [1.00, 1.31, 1.71, 2.14, 2.54, 2.78, 2.78, 2.51, 2.05, 1.53, 1.07,
                  0.735, 0.515, 0.384, 0.313, 0.283,  0.283,  0.310, 0.368, 0.46, 0.600])
_test.plot_shown()
```

# Runge-Kutta Method

```python
%video qUnPw5ZUuUs
```

Summary:

- **1st Order Runge-Kutta**<br>
We've already seen it. It's Euler's method.

- **2nd Order Runge-Kutta (A.K.A Midpoint Method)**

$$\begin{align} k_1 &= h\,f(x,t)\\
k_2 &= h\,f\big(x+\tfrac12 k_1, t+\tfrac12 h\big)\\
x(t+h) &= x(t) + k_2
\end{align}$$

 (The first line is used to estimate where the midpoint is, and the rest uses Euler's method to find the next value of $x$.)

- **4th Order Runge-Kutta**

$$\begin{align}
k_1 &= h\,f(x,t)\\
k_2 &= h\,f\big(x+\tfrac12 k_1, t+\tfrac12 h\big)\\
k_3 &= h\,f\big(x+\tfrac12 k_2, t+\tfrac12 h\big)\\
k_4 &= h\,f\big(x+k_3, t+h\big)\\
x(t+h) &= x(t) + \tfrac16\big(k_1+2k_2+2k_3+k_4\big)
\end{align}$$

- By far, the 4th order Runge-Kutta is the most popular of these methods. It hits a sweet spot between accuracy and complexity.

```python

```

## Your Turn

Write a function `RK4(f, x, t, h)` that takes one step of the 4th Order Runge-Kutta method for a function `f(x, t)`. You'll use this code for the in-class assignment.

```python
%%graded # 2 points

# YOUR CODE HERE
```

```python
%%tests

def _f(x, t): return x*np.cos(t) # function from previous part

_test.similar(RK4(_f, 1, 0, np.pi/10), 1.36) # 1st step
_test.similar(RK4(_f, 1.36, np.pi/10, np.pi/10), 1.80) # 2nd step
```

# Simultaneous 1st Order ODEs

```python
%video _aA9m8IKfVM
```

Summary:

- Write equations in the form

 $$\frac{dx}{dt}=f_x(x, y, t)\,,\qquad \frac{dy}{dt}=f_y(x, y, t)\,.$$

 With two equations, we need two initial conditions: $x_0=x(t_0)$ and $y_0=y(t_0)$.

- This set of equations can be written as a single vector equation:

 $$\frac{d\mathbf{r}}{dt}=f(\mathbf{r}, t)\,,$$

 where $\mathbf{r}=(x,y)$. Likewise, $\mathbf{r_0}=\mathbf{r}(t_0)$.

- Using arrays, we can generalize the 4th order Runge-Kutta method:

 $$\begin{align}
 \mathbf{k_1} &= h\,f(\mathbf{r},t)\\
 \mathbf{k_2} &= h\,f\big(\mathbf{r}+\tfrac12 \mathbf{k_1}, t+\tfrac12 h\big)\\
 \mathbf{k_3} &= h\,f\big(\mathbf{r}+\tfrac12 \mathbf{k_2}, t+\tfrac12 h\big)\\
 \mathbf{k_4} &= h\,f\big(\mathbf{r}+\mathbf{k_3}, t+h\big)\\
 \mathbf{r}(t+h) &= \mathbf{r}(t) + \tfrac16\big(\mathbf{k_1}+2\mathbf{k_2}+2\mathbf{k_3}+\mathbf{k_4}\big)
 \end{align}$$

 In fact, the code we wrote for this should work as-is!

```python

```

## Your Turn

The Lotka-Volterra equations are a simplified model of predator-prey population size. Let

$$\frac{dR}{dt} = R - \frac{FR}{2}$$

control the number of rabbits (in units of 1000 rabbits/year) and

$$\frac{dF}{dt} = \frac{FR}{2} - 2F$$

control the number of foxes (in units of 1000 foxes/year). Plot the numbers of rabbits and foxes over 10 years, starting with 2000 of each (i.e. $R_0=F_0=2$). Save the results in a pair of arrays named `R` and `F`.

```python
%%graded # 1 point

# YOUR CODE HERE
```

```python
%%tests

_test.similar(min(R), 2.00)
_test.similar(max(R), 7.03)
_test.similar(min(F), 0.715)
_test.similar(max(F), 4.31)
_test.plot_shown()
```

# 2nd Order ODEs

```python
%video dUmFpcDvQpk
```

Summary:

- Given

 $$\frac{d^2x}{dt^2}=f\bigg(x, \frac{dx}{dt}, t \bigg)\,,$$

 the trick is to define $v=dx/dt$, then we can write a pair of 1st order equations:

 $$\begin{aligned}
 \frac{dv}{dt} &=f(x, v, t)\\
 \frac{dx}{dt} &=v\,.
 \end{aligned}$$

- We can solve these using a 4th-order Runge Kutta or similar method.

- We need initial conditions for both $x$ and $v$.

```python

```

## Your Turn

A pendulum driven by an oscillating horizontal force is described using the equation

$$\frac{d^2\theta}{dt^2}=-\frac{g}{\ell}\sin\theta + C\cos\theta\sin(\omega_dt)\,,$$

where we'll use $\ell=10$ cm, $C=2$ s$^{-2}$ and $\omega_d=\sqrt{g/\ell}$.

Complete the code below to make a plot of $\theta(t)$ for $0<t<30$ s, with $\theta_0=0$ and $\omega_0=0$.

```python
%%graded # 1 point

g, l, C, wd = 9.8, 0.1, 2, np.sqrt(g/l)

# your job is to write this function
def f(r, t):
# YOUR CODE HERE

h = 0.01
time = np.arange(0, 30, h)
th, w = 0, 0
th_list = []
for t in time:
    th_list.append(th)
    th, w = RK4(f, (th, w), t, h) # you'll need a working version of RK4 from above

plt.plot(time, th_list)
plt.show()
```

```python
%%tests

_test.plot_shown() # the graph should look like a fish
```

# Additional Resources

- Textbook sections 8.1 - 8.4

Assignments/10.1 Lecture - Basic Differential Equations.ipynb


# Inferring parameters of SDEs using a Euler-Maruyama scheme

_This notebook is derived from a presentation prepared for the Theoretical Neuroscience Group, Institute of Systems Neuroscience at Aix-Marseile University._

```python
%pylab inline
import pymc3 as pm
import theano.tensor as tt
import scipy
from pymc3.distributions.timeseries import EulerMaruyama
```

    Populating the interactive namespace from numpy and matplotlib

## Toy model 1

Here's a scalar linear SDE in symbolic form

$ dX_t = \lambda X_t + \sigma^2 dW_t $

discretized with the Euler-Maruyama scheme

```python
# parameters
λ = -0.78
σ2 = 5e-3
N = 200
dt = 1e-1

# time series
x = 0.1
x_t = []

# simulate
for i in range(N):
    x += dt * λ * x + sqrt(dt) * σ2 * randn()
    x_t.append(x)

x_t = array(x_t)

# z_t noisy observation
z_t = x_t + randn(x_t.size) * 5e-3
```

```python
figure(figsize=(10, 3))
subplot(121)
plot(x_t[:30], 'k', label='$x(t)$', alpha=0.5), plot(z_t[:30], 'r', label='$z(t)$', alpha=0.5)
title('Transient'), legend()
subplot(122)
plot(x_t[30:], 'k', label='$x(t)$', alpha=0.5), plot(z_t[30:], 'r', label='$z(t)$', alpha=0.5)
title('All time');
tight_layout()
```

What is the inference we want to make? Since we've made a noisy observation of the generated time series, we need to estimate both $x(t)$ and $\lambda$.

First, we rewrite our SDE as a function returning a tuple of the drift and diffusion coefficients

```python
def lin_sde(x, lam):
    return lam * x, σ2
```

Next, we describe the probability model as a set of three stochastic variables, `lam`, `xh`, and `zh`:

```python
with pm.Model() as model:

    # uniform prior, but we know it must be negative
    lam = pm.Flat('lam')

    # "hidden states" following a linear SDE distribution
    # parametrized by time step (det. variable) and lam (random variable)
    xh = EulerMaruyama('xh', dt, lin_sde, (lam, ), shape=N, testval=x_t)

    # predicted observation
    zh = pm.Normal('zh', mu=xh, sd=5e-3, observed=z_t)
```

Once the model is constructed, we perform inference, i.e. sample from the posterior distribution, in the following steps:

```python
with model:

    # optimize to find the mode of the posterior as starting point for prob. mass
    start = pm.find_MAP(vars=[xh], fmin=scipy.optimize.fmin_l_bfgs_b)

    # "warm up" to transition from mode to prob. mass
    step = pm.NUTS(scaling=start)
    trace = pm.sample(1000, step, progressbar=True)

    # sample from the prob. mass
    step = pm.NUTS(scaling=trace[-1], gamma=.25)
    trace = pm.sample(2000, step, start=trace[-1], progressbar=True)
```

    100%|██████████| 1000/1000 [00:10<00:00, 93.65it/s]
    100%|██████████| 2000/2000 [00:11<00:00, 178.16it/s]

Next, we plot some basic statistics on the samples from the posterior,

```python
figure(figsize=(10, 3))
subplot(121)
plot(percentile(trace[xh], [2.5, 97.5], axis=0).T, 'k', label='$\hat{x}_{95\%}(t)$')
plot(x_t, 'r', label='$x(t)$')
legend()

subplot(122)
hist(trace[lam], 30, label='$\hat{\lambda}$', alpha=0.5)
axvline(λ, color='r', label='$\lambda$', alpha=0.5)
legend();
```

A model can fit the data precisely and still be wrong; we need to use _posterior predictive checks_ to assess if, under our fit model, the data our likely.

In other words, we
- assume the model is correct
- simulate new observations
- check that the new observations fit with the original data

```python
# generate trace from posterior
ppc_trace = pm.sample_ppc(trace, model=model)

# plot with data
figure(figsize=(10, 3))
plot(percentile(ppc_trace['zh'], [2.5, 97.5], axis=0).T, 'k', label=r'$z_{95\% PP}(t)$')
plot(z_t, 'r', label='$z(t)$')
legend()
```

Note that

- inference also estimates the initial conditions
- the observed data $z(t)$ lies fully within the 95% interval of the PPC.
- there are many other ways of evaluating fit

### Toy model 2

As the next model, let's use a 2D deterministic oscillator,
\begin{align}
\dot{x} &= \tau (x - x^3/3 + y) \\
\dot{y} &= \frac{1}{\tau} (a - x)
\end{align}

with noisy observation $z(t) = m x + (1 - m) y + N(0, 0.05)$.

```python
N, τ, a, m, σ2 = 200, 3.0, 1.05, 0.2, 1e-1
xs, ys = [0.0], [1.0]
for i in range(N):
    x, y = xs[-1], ys[-1]
    dx = τ * (x - x**3.0/3.0 + y)
    dy = (1.0 / τ) * (a - x)
    xs.append(x + dt * dx + sqrt(dt) * σ2 * randn())
    ys.append(y + dt * dy + sqrt(dt) * σ2 * randn())
xs, ys = array(xs), array(ys)
zs = m * xs + (1 - m) * ys + randn(xs.size) * 0.1

figure(figsize=(10, 2))
plot(xs, label='$x(t)$')
plot(ys, label='$y(t)$')
plot(zs, label='$z(t)$')
legend()
```

Now, estimate the hidden states $x(t)$ and $y(t)$, as well as parameters $\tau$, $a$ and $m$.

As before, we rewrite our SDE as a function returned drift & diffusion coefficients:

```python
def osc_sde(xy, τ, a):
    x, y = xy[:, 0], xy[:, 1]
    dx = τ * (x - x**3.0/3.0 + y)
    dy = (1.0 / τ) * (a - x)
    dxy = tt.stack([dx, dy], axis=0).T
    return dxy, σ2
```

As before, the Euler-Maruyama discretization of the SDE is written as a prediction of the state at step $i+1$ based on the state at step $i$.

We can now write our statistical model as before, with uninformative priors on $\tau$, $a$ and $m$:

```python
xys = c_[xs, ys]

with pm.Model() as model:
    τh = pm.Uniform('τh', lower=0.1, upper=5.0)
    ah = pm.Uniform('ah', lower=0.5, upper=1.5)
    mh = pm.Uniform('mh', lower=0.0, upper=1.0)
    xyh = EulerMaruyama('xyh', dt, osc_sde, (τh, ah), shape=xys.shape, testval=xys)
    zh = pm.Normal('zh', mu=mh * xyh[:, 0] + (1 - mh) * xyh[:, 1], sd=0.1, observed=zs)
```

As with the linear SDE, we 1) find a MAP estimate, 2) warm up and 3) sample from the probability mass:

```python
with model:

    # optimize to find the mode of the posterior as starting point for prob. mass
    start = pm.find_MAP(vars=[xyh], fmin=scipy.optimize.fmin_l_bfgs_b)

    # "warm up" to transition from mode to prob. mass
    step = pm.NUTS(scaling=start)
    trace = pm.sample(100, step, progressbar=True)

    # sample from the prob. mass
    step = pm.NUTS(scaling=trace[-1], gamma=.25)
    trace = pm.sample(2000, step, start=trace[-1], progressbar=True)
```

    100%|██████████| 100/100 [00:17<00:00,  6.02it/s]
    100%|██████████| 2000/2000 [01:38<00:00, 20.29it/s]

Again, the result is a set of samples from the posterior, including our parameters of interest but also the hidden states

```python
figure(figsize=(10, 6))
subplot(211)
plot(percentile(trace[xyh][..., 0], [2.5, 97.5], axis=0).T, 'k', label='$\hat{x}_{95\%}(t)$')
plot(xs, 'r', label='$x(t)$')
legend(loc=0)
subplot(234), hist(trace['τh']), axvline(τ), xlim([1.0, 4.0]), title('τ')
subplot(235), hist(trace['ah']), axvline(a), xlim([0, 2.0]), title('a')
subplot(236), hist(trace['mh']), axvline(m), xlim([0, 1]), title('m')
tight_layout()
```

Again, we can perform a posterior predictive check, that our data are likely given the fit model

```python
# generate trace from posterior
ppc_trace = pm.sample_ppc(trace, model=model)

# plot with data
figure(figsize=(10, 3))
plot(percentile(ppc_trace['zh'], [2.5, 97.5], axis=0).T, 'k', label=r'$z_{95\% PP}(t)$')
plot(zs, 'r', label='$z(t)$')
legend()
```


```python
import pandas as pd
import numpy as np
import seaborn as sns
```

```python
alphabet_sizes = np.arange(2, 21).astype(float)
alphabet_sizes

read_length = 50
ksize = 10
alphabet_size = 2
```

This assumes that k-mers are [iid](https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables)

$$
\begin{equation}
\left( \frac{1}{\left| \Sigma \right|^k \right)^{L - k + 1}
\begin{equation}
$$

```python
def probability_random_kmers_from_read_iid(read_length, ksize, alphabet_size):
    prob = (1/(alphabet_size ** ksize)) ** (read_length - ksize + 1)
    return prob

probability_random_kmers_from_read_iid(read_length, ksize, alphabet_size)
```

    3.7818280418450374e-124

```python
probability_random_kmers_from_read(50, 3, alphabet_sizes)
```

    array([4.48415509e-044, 1.97033159e-069, 2.01076468e-087, 2.23007452e-101,
           8.83527242e-113, 2.02247070e-122, 9.01658068e-131, 3.88220657e-138,
           1.00000000e-144, 1.09509889e-150, 3.96187318e-156, 3.90982454e-161,
           9.06907226e-166, 4.39398627e-170, 4.04317461e-174, 6.53665135e-178,
           1.74084164e-181, 7.23571517e-185, 4.48415509e-188])

Given the first k-mer can guess the second one with probability \(\frac{1}{|\Sigma|}\)

$$
\begin{equation}
\frac{1}{\left| \Sigma \right|^k * \left( \frac{1}{\left| \Sigma \right| \right)^{L - k + 1}
\begin{equation}
$$

```python
def probability_random_kmers_from_read_non_iid_v1(read_length, ksize, alphabet_size):
#     first_term = 1/(alphabet_size ** ksize)
    prob = 1/(alphabet_size ** ksize) * 1/(alphabet_size**(read_length - ksize))
    return prob

probability_random_kmers_from_read_non_iid_v1(read_length, ksize, alphabet_size)
```

    8.881784197001252e-16

```python
def probability_random_kmers_from_read_non_iid_v2(read_length, ksize, alphabet_size):
    prob = (read_length - ksize) / (alphabet_size ** (ksize + 1))
    return prob

probability_random_kmers_from_read_non_iid_v2(read_length, ksize, alphabet_size)
```

    0.01953125

```python
def probability_random_kmers_from_read_non_iid_v3(read_length, ksize, alphabet_size):
#     first_term = 1/(alphabet_size ** ksize)
    prob = 1/(alphabet_size**(read_length))
    return prob

probability_random_kmers_from_read_non_iid_v3(read_length, ksize, alphabet_size)
```

    8.881784197001252e-16

```python
probability_random_kmers_from_read_non_iid_v3(2, ksize, alphabet_size)
```

    0.25

```python

```


# Value-at-Risk for Stocks: Delta-Normal Approach, EWMA

### Lecture Notes by Jakov Ivan S. Dumbrique (jdumbrique@ateneo.edu)

MATH 100.2: Topics in Financial Mathematics II \
First Semester, S.Y. 2021-2022 \
Ateneo de Manila University

```python
import numpy as np # Numerical Computing
import pandas as pd # Data wrangling
import matplotlib.pyplot as plt # Plotting
from scipy.stats import norm # statistical analysis
from statistics import NormalDist # statistical analysis

%matplotlib inline
```

```python
def get_return(df):
    return np.log(df["close"]/df["previous"])
```

```python
def one_day_alpha_percent_VaR_single_stock_delta_normal(df, N, alpha):

    """Returns the one-day 100(alpha)% VaR of a single stock using Delta Normal Approach.

    Parameters
    ----------
    df : pandas.DataFrame
        has two columns: (1) dt [str] and (2) closing price [float]
        assumes the dates are arranged from newest to oldest, and the date today is the date on the first row
    N : int
        number of shares for the sole stock
    alpha : int
        the value to be used in calculting the 100(alpha)% VaR (e.g. 0.99, 0.95)

    Returns
    -------
    float (2 decimal places)
        one-day 100(alpha)% VaR of a single stock using Delta Normal Approach
    """

    S_0 = df.loc[0, "close"]

    # creates a new column that shifts the prices d values up (or -d values down)
    df["previous"] = df["close"].shift(-1)
    # get the daily returns
    df["return"] = df.apply(get_return, axis=1)

    # standard deviation of your stock returns
    sigma = df["return"].std()

    quantile = NormalDist().inv_cdf(alpha)
    var = N * S_0 * sigma * quantile

    return round(var, 2)

```

# EWMA for a single-asset portfolio
The one-day 99\% VaR on the portfolio is given by
		\begin{equation}
		\text{VaR}_{0.99} = |V| = NS_0\sigma_n\Phi^{-1}(0.99).
		\end{equation}
 The variance estimate $\sigma_n^2$ from the EWMA can be used as the variance of the random variable $R$, and so we assume that $R\sim N(0,\sigma_n^2)$.
 \begin{equation}
		\sigma_n^2 \approx \sum_{i=1}^{n-1}(1-\lambda)\lambda^{i-1}R_{n-i}^2,
\end{equation}
	giving us $\alpha_i = (1-\lambda)\lambda^{i-1}$, $i=1,2,\dots,n-1$.

$\alpha_i = \alpha_{j}= (1-\lambda)\lambda^{j}$, $j=0,1,2,\dots,n-2$

## Question:
Today is February 23, 2018. You are a portfolio risk manager who is assigned to analyze the market risk of a portfolio of 700 PLDT (TEL) shares. Assume that the one-day volatility of the stock follows an exponentially-weighted moving average model with parameter  $\lambda$=0.65
 . Determine the portfolio's one-day 99% VaR.

```python
def get_weights_df(df, d, ewma_par):
    """
    appends weights series to df
    """
    count_returns = len(df["close"])-d # gives the number of non-NaN (or non-empty) returns
    weight_lst = [(1-ewma_par)*(ewma_par**j) for j in range(count_returns)]
    df["weight"] = pd.Series(weight_lst)
    return df
```

```python
def d_day_alpha_percent_VaR_single_stock_delta_normal(df, d, N, alpha, ewma_par=None):

    """Returns the d-day 100(alpha)% VaR of a single stock using Delta Normal Approach.

    Parameters
    ----------
    df : pandas.DataFrame
        has two columns: (1) dt [str] and (2) closing price [float]
        assumes the dates are arranged from newest to oldest, and the date today is the date on the first row
    d : int
        value of d for the d-day VaR
    N : int
        number of shares for the sole stock
    alpha : int
        the value to be used in calculting the 100(alpha)% VaR (e.g. 0.99, 0.95)
    ewma_par :float
        the value of the lambda parameter in an EWMA model for the volatility of the stock
        assumes the value is in the range (0,1)

    Returns
    -------
    float (2 decimal places)
        d-day 100(alpha)% VaR of a single stock using Delta Normal Approach
    """

    # gets the most recent stock price
    S_0 = df.loc[0, "close"]

    # creates a new column that shifts the prices d values up (or -d values down)
    df["previous"] = df["close"].shift(-d)
    # get the d-day returns
    df["return"] = df.apply(get_return, axis=1)

    #Case 1: if I will use EWMA
    if (ewma_par != None) and (0 < ewma_par < 1):
        df = get_weights_df(df, d, ewma_par)
        variance = np.nansum(df["weight"] * (df["return"]**2))
        sigma = np.sqrt(variance)
    #Case 2: I'll use the normal VaR
    else:
        # standard deviation of your stock returns
        sigma = df["return"].std()

    quantile = NormalDist().inv_cdf(alpha)
    var = N * S_0 * sigma * quantile

    return round(var, 2)
```

```python
# read the data
TEL_df = pd.read_csv("https://raw.githubusercontent.com/ateneomathdept/math100.2_2021Sem1/main/data/lectures/TEL_2018.csv")
TEL_df
```

      dt
      close

      0
      2/23/18
      1488.74

      1
      2/22/18
      1510.86

      2
      2/21/18
      1513.72

      3
      2/20/18
      1536.65

      4
      2/16/18
      1476.37

      ...
      ...
      ...

      243
      3/2/17
      1430.29

      244
      3/1/17
      1408.94

      245
      2/28/17
      1394.35

      246
      2/27/17
      1374.93

      247
      2/24/17
      1367.68

248 rows × 2 columns

```python
get_weights_df(TEL_df, d=1, ewma_par=0.65)
```

      dt
      close
      weight

      0
      2/23/18
      1488.74
      3.500000e-01

      1
      2/22/18
      1510.86
      2.275000e-01

      2
      2/21/18
      1513.72
      1.478750e-01

      3
      2/20/18
      1536.65
      9.611875e-02

      4
      2/16/18
      1476.37
      6.247719e-02

      ...
      ...
      ...
      ...

      243
      3/2/17
      1430.29
      1.207852e-46

      244
      3/1/17
      1408.94
      7.851038e-47

      245
      2/28/17
      1394.35
      5.103175e-47

      246
      2/27/17
      1374.93
      3.317063e-47

      247
      2/24/17
      1367.68
      NaN

248 rows × 3 columns

```python
d_day_alpha_percent_VaR_single_stock_delta_normal(df=TEL_df, d=1, N=700, alpha=0.99, ewma_par=0.65)
```

    41212.93

```python
d_day_alpha_percent_VaR_single_stock_delta_normal(df=TEL_df, d=1, N=700, alpha=0.99)
```

    47587.79

```python
d_day_alpha_percent_VaR_single_stock_delta_normal(df=TEL_df, d=10, N=700, alpha=0.99, ewma_par=0.65)
```

    73320.42

```python

```


## Konvektiver + Diffusiver Transport = Burgersgleichung

### 1D-Burgersgleichung (Konvektion + Diffusion)

Die [Burgersgleichung](https://de.wikipedia.org/wiki/Burgersgleichung) entpsricht der Navier-Stokes-Gleichung ohne Quellterme bzw. der Addition aus konvektivem und diffusiven Transport. Im eindimensionalen Fall ergibt sich also folgende Gleichung:

$$\frac{\partial u}{\partial t} + u \frac{\partial u}{\partial x} = \nu \frac{\partial ^2u}{\partial x^2}$$

Die Burgersgleichung ist nicht nur für die Strömungsmechanik relevant, sondern taucht auch in anderen physikalischen Problemen auf, wie z.B. bei der Simulation des Straßenverkehrs.

Da wir die Diskretisierung der einzelnen Terme bereits durchgeführt haben, ist dieser Schritt trivial. Wir wenden wieder ein einseitiges (Vorwärts-)Differenzenverfahren für die Diskretisierung in der Zeit, ein einseitiges (Rückwärts-)Differenzenverfahren für die konvektiven Terme und ein Zentraldifferenzenverfahren für die zweiten Ableitungen der diffusiven Terme an und erhalten:

$$\frac{u_i^{n+1}-u_i^n}{\Delta t} + u_i^n \frac{u_i^n - u_{i-1}^n}{\Delta x} = \nu \frac{u_{i+1}^n - 2u_i^n + u_{i-1}^n}{\Delta x^2}$$

Wieder können wir nach den gesuchten Geschwindigkeiten zum neuen Zeitpunkt explizit auflösen:

$$u_i^{n+1} = u_i^n - u_i^n \frac{\Delta t}{\Delta x} (u_i^n - u_{i-1}^n) + \nu \frac{\Delta t}{\Delta x^2}(u_{i+1}^n - 2u_i^n + u_{i-1}^n)$$

#### Anfangs- und Randbedingungen

Eigentlich könnten wir die Gleichung nun sofort in Pyton umsetzen. Da die 1D-Burgersgleichung jedoch für bestimmte Anfangs- und Randbedingungen mit Hilfe einer [Cole-Hopf-Transformation](https://de.wikipedia.org/wiki/Hopf-Cole-Transformation) (mehr was für Mathematiker) analytisch lösbar ist, wollen wir die Gelegenheit nutzen und unsere numerische Lösung mit der analyitschen Lösung vergleichen.

Dazu müssen wir als Anfangsbedingung folgende Geschwindigkeitsverteilung realisieren:

$$u = -\frac{2 \nu}{\phi} \frac{\partial \phi}{\partial x} + 4 \quad\text{mit}\\\
\phi = \exp \bigg(\frac{-x^2}{4 \nu} \bigg) + \exp \bigg(\frac{-(x-2 \pi)^2}{4 \nu} \bigg)$$

und periodische Randbedingungen vorsehen:

$$u(0) = u(2\pi)$$

Für diesen Fall lautet die Lösung der Burgersgleichung:

$$u = -\frac{2 \nu}{\phi} \frac{\partial \phi}{\partial x} + 4 \quad\text{mit}\\\
\phi = \exp \bigg(\frac{-(x-4t)^2}{4 \nu (t+1)} \bigg) + \exp \bigg(\frac{-(x-4t -2 \pi)^2}{4 \nu(t+1)} \bigg)$$

Die Anfangsbedingung entspricht also der Lösung für den Zeitpunkt $t=0$.

##### Verwendung von SymPy

Um die Anfangsbedingung und die Lösung auszurechnen müssen wir die Ableitung von $\phi$ berechnen, was eigentlich nicht weiter schwer ist. Trotzdem soll an dieser Stelle [`SymPy`](http://www.sympy.org/en/index.html) eingeführt werden, mit dem symbolische Mathematik (ähnlich Wolfram Alpha, Mathematica oder Maple) in Python möglich ist.

Wir laden zunächst wieder die benötigen Bibliotheken. Neben `numpy` brauchen wir nun auch `sympy`:

```python
import numpy as np
import sympy as sp
```

Wenn man mit dem Jupyter-Notebook arbeitet, kann man SymPy so konfigurieren, dass es die Formeln mithifle von $\LaTeX$ rendert.

```python
from sympy import init_printing
init_printing(use_latex=True)
```

Zunächst müssen die Variablen, die in der Gleichung für $phi$ vorkommen als symbolische Variablen definiert werden. Anschließend können wir die Gleichung in SymPy zuweisen:

```python
x, nu, t = sp.symbols('x nu t')
phi = (sp.exp(-(x - 4 * t)**2 / (4 * nu * (t + 1))) +
       sp.exp(-(x - 4 * t - 2 * sp.pi)**2 / (4 * nu * (t + 1))))
phi
```

Damit ist es nun ganz einfach, die Ableitung von $phi$ nach dem Ort zu bilden:

```python
dphidx = phi.diff(x)
dphidx
```

Um die nicht-gerenderte version der Funktion zu bekommen, kann man diese einfach mit dem `print`-Befehl ausgeben:

```python
print (dphidx)
```

    -(-8*t + 2*x)*exp(-(-4*t + x)**2/(4*nu*(t + 1)))/(4*nu*(t + 1)) - (-8*t + 2*x - 4*pi)*exp(-(-4*t + x - 2*pi)**2/(4*nu*(t + 1)))/(4*nu*(t + 1))

Die Funktion ist in dieser Form allerdings noch nicht in Python aufrufbar. Wir könnten diese z.B. abtippen und in eine Python-Funktion umsetzen (mit `def`...).

Eleganter ist es, die `lambdify`-Funktion von SymPy zu nutzen, die genau das für uns macht, ohne dass wit etwas abtippen müssen:

```python
from sympy.utilities.lambdify import lambdify

# Der Ausdruck für u:
u_sym = -2 * nu * (dphidx / phi) + 4

# Und die daraus automatisch erzeugte Python-Funktion u
u_analytisch = lambdify((t, x, nu), u_sym)
print(u_analytisch(1, 4, 3))
```

    3.49170664206

Damit haben wir sowohl eine Funktion, mit der wir die Anfangsbedingung zum Zeitpunkt $t=0$ als auch die Lösung zu jedem beliebigen Zeitpunkt berechnen können.

Jetzt geht es an die Lösung der Burgersgleichung:

```python
import matplotlib.pyplot as plt
%matplotlib inline

# benötigte Variablen
L  = 2 * np.pi      # Länge unseres Rechengebiets
nx = 101             # Anzahl der diskreten Punkte
dx = L / (nx-1)     # Abstand zwischen den Punkten
t_end = 1.4        # Endzeitpunkt für die Berechnung
nu  = 0.07          # die kinematische Viskosität in m^2/s
CFL = 0.1
dt = CFL * dx * nu  # Zeitschrittweite

# Anfangsbedingung
xi = np.linspace(0, L, nx)
u_init = np.asarray([u_analytisch(0, x, nu) for x in xi])

u_alt = u_init.copy()  # wir übernehmen zunächst einmal die Anfangsbedingung
u_neu = u_alt.copy()
t = 0                  # der Anfangszeitpunkt ist 0
```

```python
plt.figure(figsize=(8, 4))
plt.plot(xi, u_init, marker='o', lw=2)
plt.xlim([0, L])
plt.ylim([0, 8]);
```

Die Anfangsverteilung ist nun eher eine Diskontinuität. Deutlich wird auch, dass diese periodisch verläuft, d.h. der Wert am Ende unseres Rechengebiets bei $x=2\pi$ entspricht dem Wert bei $x=0$.

#### Periodische Randbedingungen

Neu im Gegensatz zu den bisherigen Berechnungen ist die Verwendung von periodischen Randbedingungen. D.h. alles was aus dem rechten Rand hinausströmt muss am linken Rand wieder hineinströmen.

Betrachten wir die diskretisierte Burgersgleichung:

$$u_i^{n+1} = u_i^n - u_i^n \frac{\Delta t}{\Delta x} (u_i^n - u_{i-1}^n) + \nu \frac{\Delta t}{\Delta x^2}(u_{i+1}^n - 2u_i^n + u_{i-1}^n)$$

Was müssen wir für $u_{i+1}^n$ verwenden, wenn $i$ bereits am rechten Rand des Gebietes ist und was für $u_{i-1}^n$, wenn $i = 0$ ist?

```python
while t <= t_end:      # Schleife, solange bis Endzeitpunkt erreicht ist
    # dieses Mal mit innerer Schleife zum besseren Verständnis
    # ginge aber auch wieder direkt mit der zuvor eingeführten
    # Schreibweise
    for i in range(1, nx-1):
        u_neu[i] = (u_alt[i]
                    - u_alt[i] * dt / dx *(u_alt[i] - u_alt[i-1])
                    + nu * dt / dx**2
                    * (u_alt[i+1] - 2 * u_alt[i] + u_alt[i-1]))
    # periodische Randbedingungen:
    u_neu[0] = (u_alt[0]
                - u_alt[0] * dt / dx * (u_alt[0] - u_alt[-1])
                + nu * dt / dx**2
                * (u_alt[1] - 2 * u_alt[0] + u_alt[-1]))
    u_neu[-1] = (u_alt[-1]
                 - u_alt[-1] * dt / dx * (u_alt[-1] - u_alt[-2])
                 + nu * dt / dx**2
                 * (u_alt[0]- 2 * u_alt[-1] + u_alt[-2]))
    u_alt = u_neu.copy()
    t += dt
```

```python
u_analytisch_t_end = np.asarray([u_analytisch(t_end, x, nu) for x in xi])
```

```python
plt.figure(figsize=(8, 4))
plt.plot(xi, u_neu, marker='o', lw=2, label='FDM')
plt.plot(xi, u_analytisch_t_end, label='analytisch')
plt.xlim([0, L])
plt.ylim([0, 8])
plt.legend();
```

Experimentieren Sie ein wenig mit der Diskretisierung. Welche Auflösung benötigen wir, um die analytische Lösung genau zu treffen?

Wir sind jetzt schon sehr nahe an der Lösung der kompletten Navier-Stokes-Gleichung. Die 2D-Burgersgleichung sparen wir uns, da sie keine neuen Erkenntnisse bringt.

Im nächsten Notebook nehmen wir uns dafür die [2D-Poisson-Gleichung](TFD - 3.8 Kontinuumsstroemungen - Poisson.ipynb) vor, die wir zuvor bereits eingeführt haben.

---
Dieses Notebook folgt einer Idee von Lorena A. Barba und deren exzellenter Ausarbeitung in Python (["12 Steps to Navier Stokes"](http://lorenabarba.com/blog/cfd-python-12-steps-to-navier-stokes/)).

Der folgende Python-Code darf ignoriert werden. Er dient nur dazu, die richtige Formatvorlage für die Jupyter-Notebooks zu laden.

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('TFDStyle.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 120%;
    font-weight: 400;
    width:650px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Alegreya Sans', sans-serif;
    font-style:regular;
    font-weight: 300;
    font-size: 30pt;
    line-height: 100%;
    color:#CD2305;
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Alegreya Sans', sans-serif;
    font-style:regular;
    font-weight: 300;
    font-size: 30pt;
    line-height: 100%;
    color:#CD2305;
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h3 {
    font-family: 'Alegreya Sans', sans-serif;
    font-weight: 300;
    font-size: 24pt;
    line-height: 100%;
    color:#CD2305;
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h4 {
    font-family: 'Alegreya Sans', sans-serif;
    font-weight: 300;
    margin-top:12px;
	font-size: 20pt;
    color:#CD2305;
    margin-bottom: 3px;
    font-style: regular;
}

.text_cell_render h5 {
    font-family: 'Alegreya Sans', sans-serif;
    font-weight: 300;
    font-size: 16pt;
    margin-top: 12px;
    color:#CD2305;
    margin-bottom: 3px;
    font-style: regular;
}

.text_cell_render h6 {  /*Use this for small titles*/
    font-family: 'Alegreya Sans', sans-serif;
    font-weight: 300;
    font-size: 16pt;
    color: #CD2305;
    font-style: italic;
    margin-bottom: .5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h7 { /*use this for copyright note*/
    font-family: 'Source Code Pro', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/

    .warning{
        color: rgb( 240, 20, 20 )
        }

    div.highlight {
        background-color: #fcf2f2;
        border-color: #dFb5b4;
        /*border-left: 5px solid #dfb5b4;*/
        padding-top: 0.5em;
        padding-bottom: 0.5em
    }

</style>
/*

 */

TFD - 3.7 Kontinuumsstroemungen - Burgersgleichung.ipynb