###### Content under Creative Commons Attribution license CC-BY 4.0, code under BSD 3-Clause License © 2020 L.A. Barba, N.C. Clementi

# Untangle any waveform

Welcome to a new lesson of the _Engineering Computations_ series, where we use computing as our instrument to discover the world. In this lesson, we explore information in wave form, using sound as a prototypical example. We aim to bring to light how engineers and scientists are able to analyze this kind of data, which has a myriad interesting applications. Let's go!

## What is sound?

Physically, _sound_ means travelling vibrations in any medium: air, water, metal, anything that can support oscillations. When your vocal chords emit sound, they are literally vibrating and passing on that vibration to the air around them.

Imagine pressure vibrations in the air like longitudinal waves on a slinky, as shown on this gif (you can watch the full [video](https://www.youtube.com/watch?v=GIkeGBXqWW0) on YouTube).

#### Compression/expansion longitudinal wave on a slinky.

Every loop of the spiral just oscillates around a position of equilibrium, while the _wave_ of compressions and expansions travels the length of the slinky. In the air, sound also travels as longitudinal compression/rarefaction waves.

### Waveforms

A waveform is a schematic to understand sound waves. A fundamental sound is represented by a _sine wave_. Do explore the interactive guide to [waveforms](https://pudding.cool/2018/02/waveforms/) online: it presents both visually and audibly the concepts of displacement amplitude, wave frequency and more.

When you play on the middle of a piano the note of A, the piano string vibrates at 440 oscillations per second (Hz). That is the _frequency_ of the corresponding sine wave:

$$y(t) = A \sin(2\pi \, f \, t + \theta)$$

where $A$ is the wave amplitude, $f$ the frequency, and $\theta$ is the phase, i.e., the angle value of the sine at $t=0$. The corresponding angular frequency is $~\omega=2\pi f$, in radians per second.

Let's visualize this function using our favorite Python libraries. We first create a `time` array holding the discrete values of time to evaluate the function at. The array is defined using the length of time from zero to `length_sec` (in seconds) and a _sampling rate_, the number of time samples per second. Study the code below.

```python
import numpy
from matplotlib import pyplot
%matplotlib inline
```

```python
# set fonts for plot lables and axes
pyplot.rc('font', family='serif', size='14')
```

```python
sample_rate = 44100  #Hz
length_sec = 2       #seconds
time = numpy.linspace(0, length_sec, sample_rate * length_sec)
```

```python
freq = 440 #Hz
y = numpy.sin(2 * numpy.pi * freq * time)
```

The sampling rate used above, [44100 Hz](https://en.wikipedia.org/wiki/44,100_Hz) (44.1 kHz), is the standard value for digital audio recordings going back to the 1970s. It is the sampling rate used in CDs, and in .mp3 audio files.

A fundamental theorem in the field of digital signal processing states that to capture all the information of a continuous signal, the sampling rate needs to be more than twice the maximum frequency. It is the **Nyquist–Shannon sampling theorem**. Since our ears can perceive sound in a frequency range of 20–20,000 Hz (20 kHz), industry chose 44.1 kHz for digital audio.

Let's look at our sine waveform for the A4 note (that is the note of A in middle of the piano):

```python
pyplot.figure(figsize=(8,3))
pyplot.plot(time[:1000], y[:1000])
pyplot.xlabel('time')
pyplot.title('Sine wave with a 440 Hz frequency, $A=1$\n');
```

## Play with audio in Python

We can _play_ the sound from our sine waveform using the [display](https://ipython.org/ipython-doc/dev/api/generated/IPython.display.html) tools of IPython. Check out the documentation for details. When you use a NumPy array as the data, it needs the argument `rate` corresponding to the sampling rate.

```python
from IPython.display import Audio
```

```python
Audio(y, rate=sample_rate)
```

Isn't that cool? Let's now write a custom function that takes as arguments the frequency, duration of sound (and an optional phase angle) and it returns the time array and the signal, using 44.1 kHz as the sampling rate.

```python
def sine_signal(f, s, phase=0):
    '''Computes a sine signal of frquency f and duration s
    using a sample rate of 44.1kHz, and returns it along with
    the time array

    Arguments:
    ----------
    f: float, frequency
    s: float, duration of the audio in seconds
    phase: float, optional phase in radians

    Returns:
    --------
    wave: array, sinusoidal wave array of frequency f
    time: array, time samples

    '''
    samp_rate = 44100 #Hz
    time = numpy.linspace(0, s, samp_rate * s)

    wave = numpy.sin(2 * numpy.pi * f * time + phase)

    return wave, time
```

Let's create two sine waves of 2-second duration with frequencies of 440 Hz and 294 Hz, corresponding to the notes A4 and D4 on the piano, and save them to Python variables. On the second line, we save only the signal using the index `[0]`.

```python
A440, t2 = sine_signal(f=440, s=2)
D294 = sine_signal(f=294, s=2)[0]
```

Now play the sound for the D note alone, and for the sum of the two notes. Note that we can do this because the two signals have the same length and sampling rate!

```python
Audio(D294, rate=sample_rate)
```

```python
Audio(D294+A440, rate=sample_rate)
```

```python
f, (ax1, ax2, ax3) = pyplot.subplots(3, 1, sharex=True, figsize=(12.0, 9.0))

ax1.plot(t2[:1000], (A440+D294)[:1000], c='C0', label='A440 + D294')

ax2.plot(t2[:1000], D294[:1000], c='C1', label='D294')
ax2.set_ylabel('Amplitude')

ax3.plot(t2[:1000], A440[:1000], c='C2', label='A440')

ax3.set_xlabel('time [s]')

#plot all legends in one line
[ax.legend(loc='upper right') for ax in (ax1, ax2, ax3)];
```

##### Exercise

- Create two new sound waves of 2-second duration and frequencies of 349Hz and 523Hz, and name them F349 and C523, respectively.
- Create the audio widget to plays the sound that results from the sum of all the frequencies we have created.
- Plot all the sound waves and the sum of all of them in aplot like the one above.

### Read and write audio files

The SciPy library has many modules to read and write data in various file formats. To handle [WAV files](https://en.wikipedia.org/wiki/WAV), those written in the Waveform Audio File Format, we can use the `scipy.io.wavfile` module. After importing it, let's save the waveform data we have created into a file; notice the arguments we have passed to the [`wavfile.write()`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.wavfile.write.html#scipy.io.wavfile.write) function and take a look at the documentation page.

```python
from scipy.io import wavfile

sum_waves = A440 + D294

wavfile.write('sum_AD.wav', sample_rate, sum_waves) #name of file, sample rate, array
```

You can also read a WAV file, listen to it, and later analyze it or modify the data. Here, we download a WAV file with the tones of a C chord on a guitar (added in progression), and then use [`wavfile.read()`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.wavfile.read.html#scipy.io.wavfile.read)  to read the sound waveform, and to save it to a Python variable.
(C chord WAV data file from Ref. [2].)

```python
URL = 'https://matthewmcgonagle.github.io/assets/2018-01-05-output/cchord.wav'
from urllib.request import urlretrieve
urlretrieve(URL, 'cchord.wav')
```

```python
wavfile.read('cchord.wav')
```

Notice the return value of `wavfile.read()`: it is a tuple containing an integer, and an array. The first component is the sampling rate of this WAV file, and the second component is the data as a NumPy array. To play with them, we save them to Python variables.

```python
cchord_rate, cchord = wavfile.read('cchord.wav')
```

```python
Audio(cchord, rate=cchord_rate)
```

The values of time corresponding to the signal samples are computed using the integers from zero to the length of the data array, divided by the sampling rate. Let's create a time array for this signal, then plot the signal.

```python
ctimes = numpy.arange(len(cchord)) / cchord_rate
```

```python
pyplot.figure(figsize=(8,3))
pyplot.plot(ctimes, cchord)
pyplot.xlabel('time')
pyplot.title('C chord signal');
```

Interesting! But hard to make out what is going on there. What if you wanted to know the frequencies of the tones that make up this audio sample? _How might you untangle this waveform into its component sine waves?_

## Understanding the Fourier transform

The answer to the question above is _the Fourier transform_. Once again, we benefit from a wonderfully visual explanation by Grant Sanderson [3]. We highly recommend that you watch the [video](https://www.youtube.com/watch?v=spUNpyF58BY)! The rest of this lesson is based on it.

A key idea of the Fourier transform is that it mathematically treats each frequency component in a signal _differently_, so that it can identify them and show us what they are.

We will follow Ref. [3] closely. Let us start with a 2-second-long sine wave that is shifted by $\pi/2$ (i.e., it's the cosine wave), with a 3-Hz frequency. Create it with our custom function, then plot it.

```python
sinewave, time = sine_signal(f=3, s=2, phase=numpy.pi/2)
```

```python
pyplot.plot(time, sinewave)
pyplot.xlabel('Time [s]')
pyplot.ylabel('Amplitude')
pyplot.title('Sine wave with 3-Hz frequency and $\pi/2$ phase\n');
```

Now, complex numbers to the rescue! In [Lesson 0](https://go.gwu.edu/engcomp5lesson0) of this module, we reviewed complex numbers, played with them using Python, and learned about their geometry.

Imagine that our signal corresponds to the modulus of a complex number, while the time maps to its phase angle—it's like taking the horizontal time axis and wrapping it around a circle on the complex plane.

Remember that Euler's formula says that the exponential of $i\alpha$ lands on the complex number $(\cos\alpha, \sin\alpha)$ on the unit circle. We use a frequency value to define complex numbers with $\alpha = -2\pi i f t$—that is, as time moves forward, we rotate clockwise on the complex plane.

In the function below, we define a unit circle from a given frequency and time samples. In the next cell, we call it with a frequency of $1$ (corresponding to an angular frequency of $2\pi$ radians per second) and the `time` array, and plot the result.

```python
def comp_fourier_term(f, t):

    circ = numpy.exp(-2*numpy.pi*1j*f*t)

    return circ
```

```python
wrap = comp_fourier_term(1, time)

pyplot.figure(figsize=(4, 4))
pyplot.plot(wrap.real, wrap.imag)
pyplot.axis('scaled')
pyplot.axis([-1.1,1.1,-1.1,1.1])
pyplot.grid(True);
```

That took the `time` array and mapped it onto the unit circle on the complex plane.

Next, we _multiply_ each discrete value of our sine waveform with the corresponding complex number that maps time to the unit circle.
This is like winding our signal around the origin. The resulting curve passes through the complex numbers whose modulus is equal to the amplitude and whose phase is equal to $\alpha = -2\pi i f t$ for our signal.

By calling the function `comp_fourier_term()` with the argument `f=1`, we wrapped time around the unit circle once. But we can change this "winding" frequency to any value we want, and it will change how the signal is mapped onto the complex plane.

We are now dealing with two frequencies: that of the original sine waveform, and that of the winding of that signal around the unit circle on the complex plane.

Come on, try this with several values of `f`!

```python
wrap_sine = sinewave * comp_fourier_term(f=0.5, t=time)

pyplot.figure(figsize=(4, 4))
pyplot.plot(wrap_sine.real, wrap_sine.imag)
pyplot.axis('scaled')
pyplot.axis([-1.1,1.1,-1.1,1.1])
pyplot.grid(True);
```

If you experimented with lots of values of the winding frequency, you were probably amused by the pretty pictures, but our toy is also a powerful means for understanding the Fourier transform.

For now, consider just the real part of the wound-up signal, and look at the mean value for all sampled times, for a given winding frequency. This mean value tends to be small, as the plots show that the wound-up signal is often symmetric around the origin. But for some winding frequency, things change...

Let's explore. With a winding frequency of 0.5 the mean value of the real part is nearly zero. But let's re-compute the wound-up signal for an array of frequency values, and get the mean of the real parts, each time.
We choose 200 values of winding frequency, between 1 and 5, compute the wound-up signal with each one, then take the mean of the real parts of the resulting complex numbers, and save that in an array. Plotting this information is... interesting!

```python
numpy.mean(wrap_sine.real)
```

```python
# Winding frequencies
freqs = numpy.arange(1, 5, 0.02)
len(freqs)
```

```python
xmean = []
for i in range(len(freqs)):

    wrap = sinewave * comp_fourier_term(f=freqs[i], t=time)
    xmean.append(numpy.mean(wrap.real))

xmean = numpy.array(xmean)
```

```python
pyplot.figure(figsize=(8,4))
pyplot.plot(freqs, xmean)
pyplot.xlabel('frequency');
```

Starting with our sine wave, winding it around the unit circle on the complex planes at different frequencies, and computing the mean of the real parts of the resulting complex numbers—this transformed data—reveals the _frequency of the original signal_.

Why is there a marked peak on this data at just the winding frequency that matches the signal frequency?
We have a "helper" Python script that makes an animation with Matplotlib of the changing wound-up signal on the complex plane, and the plot of the transformed data with respect to frequency. This is not quite the Fourier transform, but it's pretty close. Grant Sanderson calls it "almost Fourier" transform [3], and so do we.

Here, we load Matplotlib's `animation` module, and our helper script. In the next cell, we call our helper function `create_animation()` using a coarser set of frequency values, to make things a little faster.

```python
from matplotlib import animation

# import helper to produce animation
import sys
sys.path.append('../scripts/')

import almost_fourier_helper as afh
```

```python
# This cell will take few seconds to run
# We use fewer frequencies otherwise the animation is too slow
freqs_coarse = numpy.arange(1, 5, 0.06)

afh.create_animation(sinewave, time, freqs_coarse)
```

Watch the animation and ponder how this "frequency detection machine" works: multiplying the signal amplitude by the complex number $e^{i\alpha}$, where $\alpha=-2\pi i f t$, with different values of $f$. This transforms the signal to a curve on the complex plane that for most frequencies has some symmetry, so the real parts average to near zero.
Except as it approaches the frequency of the signal, when the things line up just right and the curve on the complex plane averages a positive real part, revealing a peak at the signal frequency.
It's startling!

We can try again with the sum of two sine waveforms, say, with frequencies of 2 and 4 Hz, and see what happens now.

```python
sinewave_1, time = sine_signal(f=2, s=2, phase=numpy.pi/2)
sinewave_2, time = sine_signal(f=4, s=2, phase=numpy.pi/2)

xmean = []
for i in range(len(freqs)):

    wrap = (sinewave_1+sinewave_2) * comp_fourier_term(f=freqs[i], t=time)
    xmean.append(numpy.mean(wrap.real))

xmean = numpy.array(xmean)
```

```python
pyplot.figure(figsize=(8,4))
pyplot.plot(freqs, xmean)
pyplot.xlabel('frequency');
```

```python
#we need to normalize to have Amplitude max =1 and fit in the circle
afh.create_animation((sinewave_1 + sinewave_2)/max(sinewave_1 + sinewave_2), time, freqs_coarse)
```

Indeed, we see two frequency peaks: at 2 and 4 Hz.

Our final test will use the data corresponding to a guitar C chord, saved above from a WAV file into the variables `cchord` and `ctimes`. To know what frequency range we need to explore, we used a [table of the frequencies in Hz for musical pitches](https://www.seventhstring.com/resources/notefrequencies.html). It seems for the first 5 octaves, the frequencies are in the range from 15 to 1000 Hz.

```python
freqs = numpy.arange(15, 1000, 5)
len(freqs)
```

```python
xmean = []
for i in range(len(freqs)):

    wrap = cchord * comp_fourier_term(f=freqs[i], t=ctimes)
    xmean.append(numpy.mean(wrap.real))

xmean = numpy.array(xmean)
```

```python
pyplot.figure(figsize=(8,4))
pyplot.plot(freqs, abs(xmean))
pyplot.xlabel('frequency');
```

From Ref. [2], we know the frequencies used to generate the audio file were `[261.6, 329.6, 392.0, 523.3, 659.3]`, which from the table of frequencies correspond to C4, E4, G4, C5 and E5. Those frequencies look pretty close to the peaks seen in the plot above, if you squint!

That is how the Fourier transform works, in broad strokes. In this exploration, we looked at the real part only, and we took its mean value for the wound-up signal around a circle in the complex plane.

We have also worked with discrete signals throught this lesson. Sound is made up of _continuous waves_, but we always need to turn that into discrete values to store it digitally.

The Fourier transform takes a time-dependent signal $g(t)$ and computes:

$$\begin{equation}
     \hat{g}(f) = \int_{t_1}^{t_2} g(t) e^{-2\pi i f t} dt
\end{equation}
$$

This has real and imaginary parts, but we only looked at the real part. It is an integral—meaning, an infinite sum—while we looked at a mean value, which is a sum of discrete terms, divided by the number of terms. So we got close, but not quite. It is remarkable that the the power of the transform is still obvious even in this "almost" Fourier transform.

The transform converts the information in the time domain, to the frequency domain, but nothing is lost. In fact, you can recover a signal from its Fourier transform, using the _inverse_ Fourier transform.

## References

1. Josh Comeau (n.d.), [Waveforms](https://pudding.cool/2018/02/waveforms/), an interactive and visual guide.
2. Matthew McGonagle (2018), [Making an Audio .wav File of Cantor Tones](https://matthewmcgonagle.github.io/blog/2018/01/05/CantorTones), blog post.
3. Grant Sanderson (2018), [But what is the Fourier Transform? A visual introduction](https://www.youtube.com/watch?v=spUNpyF58BY), video on YouTube.

```python
# Execute this cell to load the notebook's style sheet, then ignore it
from IPython.core.display import HTML
css_file = '../style/custom.css'
HTML(open(css_file, "r").read())
```


-------------------
# Intorduction to `PyMC2`

#### Balint Szoke
-----------------------

Installation:

`>> conda install pymc`

```python
%matplotlib inline
import numpy as np
import scipy as sp
import pymc as pm
import seaborn as sb
import matplotlib.pyplot as plt
```

## Probabilistic model

Suppose you have a sample $\{y_t\}_{t=0}^{T}$ and want to characeterize it by the following probabilistic model; for $t\geq 0$

$$ y_{t+1} = \rho y_t + \sigma_x \varepsilon_{t+1}, \quad \varepsilon_{t+1}\stackrel{iid}{\sim}\cal{N}(0,1) $$

with the initial value $y_0 \sim {\cal N}\left(0, \frac{\sigma_x^2}{1-\rho^2}\right)$ and suppose the following (independent) prior beliefs for the parameters $\theta \equiv (\rho, \sigma_x)$
 - $\rho \sim \text{U}(-1, 1)$
 - $\sigma_x \sim \text{IG}(a, b)$

**Aim:** given the statistical model and the prior $\pi(\theta)$ we want to ''compute'' the posterior distribution $p\left( \theta \hspace{1mm} | \hspace{1mm} y^T  \right)$ associated with the sample $y^T$.

**How:** if no conjugate form available, sample from $p\left( \theta \hspace{1mm} | \hspace{1mm} y^T  \right)$ and learn about the posterior's properties from that sample

> **Remark:** We go from the prior $\pi$ to the posterior $p$ by using Bayes rule:
\begin{equation}
p\left( \theta \hspace{1mm} | \hspace{1mm} y^T  \right) = \frac{f( y^T \hspace{1mm}| \hspace{1mm}\theta) \pi(\theta) }{f( y^T)}
\end{equation}
The first-order autoregression implies that the likelihood function of $y^T$ can be factored as follows:

>$$ f(y^T \hspace{1mm}|\hspace{1mm} \theta) = f(y_T| y_{T-1}; \theta)\cdot f(y_{T-1}| y_{T-2}; \theta) \cdots f(y_1 | y_0;\theta )\cdot f(y_0 |\theta) $$
where for all $t\geq 1$
$$ f(y_t | y_{t-1}; \theta) = {\mathcal N}(\rho y_{t-1}, \sigma_x^2) =  {\mathcal N}(\mu_t, \sigma_x^2)$$

Generate a sample with $T=100$ for known parameter values:
$$\rho = 0.5\quad \sigma_x = 1.0$$

```python
def sample_path(rho, sigma, T, y0=None):
    '''
    Simulates the sample path for y of length T+1 starting from a specified initial value OR if y0
    is None, it initializes the path with a draw from the stationary distribution of y.

    Arguments
    -----------------
        rho (Float) : AR coefficient
        sigma (Float) : standard deviation of the error
        T (Int)       : length of the sample path without x0
        y0 (Float)    : initial value of X

    Return:
    -----------------
        y_path (Numpy Array) : simulated path

    '''
    if y0 == None:
        stdev_erg = sigma / np.sqrt(1 - rho**2)
        y0 = np.random.normal(0, stdev_erg)

    y_path = np.empty(T+1)
    y_path[0] = y0
    eps_path = np.random.normal(0, 1, T)

    for t in range(T):
        y_path[t + 1] = rho * y_path[t] + sigma * eps_path[t]

    return y_path

#-------------------------------------------------------

# Pick true values:
rho_true, sigma_x_true, T = 0.5, 1.0, 20

#np.random.seed(1453534)
sample = sample_path(rho_true, sigma_x_true, T)
```

## Probabilistic models in `pymc`

*Model instance* $\approx$ collection of random variables linked together according to some rules

### Linkages (hierarchical structure):
   * **parent**: variables that influence another variable
    - e.g. $\rho$ and $\sigma_x$ are parents of $y_0$, $a$ and $b$ are parents of $sigma_x$

   * **child**: variables that are affected by other variables (subjects of parent variables)
    - e.g. $y_t$ is a child of $y_{t-1}$, $\rho$ and $\sigma_x$

>*Why are they useful?*

> child variable's current value automatically changes whenever its parents' values change

### Random variables:
- have a `value` attribute producing the current internal value (given the values of the parents)
    - computed on-demand and cached for efficiency.
- other important attributes: `parents` (gives dictionary), `children` (gives a set)

Two main classes of random variables in `pymc`:

#### 1) Stochastic variable:
- variable whose value is not completely determined by its parents
- *Examples:*
    * parameters with a given distribution
    * observable variables (data) = particular realizations of a random variable (see below)
- treated by the back end as random number generators (see built-in `random()` method)
- `logp` attribute: evaluate the logprob (mass or density) at the current value; for vector-valued variables it returns the sum of the (joint) logprob
- *Initialization:*
    * define the distribution (built-in or your own) with `name` + params of the distribution (can be `pymc` variable)
    * optional flags:
     - `value`: for a default initial value; if not specified, initialized by a draw from the given distribution
     - `size`: for multivariate array of independent stochastic variables. (Alternatively: use array as a distribution parameter)

Initialize `stochastic variables`

```python
# Priors:
rho = pm.Uniform('rho', lower = -1, upper = 1)      # note the capitalized distribution name (rule for pymc distributions)
sigma_x = pm.InverseGamma('sigma_x', alpha = 3, beta = 1)
```

```python
# random() method
print('Initialization:')
print("Current value of rho = {: f}".format(rho.value.reshape(1,)[0]))
print("Current logprob of rho = {: f}".format(rho.logp))

rho.random()
print('\nAfter redrawing:')
print("Current value of rho = {: f}".format(rho.value.reshape(1,)[0]))
print("Current logprob of rho = {: f}".format(rho.logp))

```

    Initialization:
    Current value of rho = -0.615898
    Current logprob of rho = -0.693147

    After redrawing:
    Current value of rho =  0.630793
    Current logprob of rho = -0.693147

------------------
#### 2) Determinsitic variable:
- variable that is entirely determined by its parents
- ''exact functions'' of stochastic variables, however, we can treat them as a variable and not a Python function.
- *Examples:*
  * model implied restrictions on how the parameters and the observable variables are related
     - $\text{var}(y_0)$ is a function of $\rho$ and $\sigma_x$
     - $\mu_{t}$ is an exact function of $\rho$ and $y_{t-1}$
  * sample statistics, i.e. deterministic functions of the sample
- *Initialization:*
  * decorator form:
      - Python function of stochastic variables AND default values + the decorator `pm.deterministic`
  * elementary operations (no need to write a function or decorate): $+$, $-$, $*$, $/$
  * `pymc.Lambda`

Initialize `deterministic variables`:

(a) Standard deviation of $y_0$ is a deterministic function of $\rho$ and $\sigma$

```python
@pm.deterministic(trace = False)
def y0_stdev(rho = rho, sigma = sigma_x):
    return sigma / np.sqrt(1 - rho**2)

# Alternatively:
#y0_stdev = pm.Lambda('y0_stdev', lambda r = rho, s = sigma_x: s / np.sqrt(1 - r**2) )
```

(b) Conditional mean of $y_t$, $\mu_y$, is a deterministic function of $\rho$ and $y_{t-1}$

```python
# For elementary operators simply write
mu_y = rho * sample[:-1]
print(type(mu_y))

# You could also write, to generate a list of Determinisitc functions
#MU_y = [rho * sample[j] for j in range(T)]
#print(type(MU_y))
#print(type(MU_y[1]))
#MU_y = pm.Container(MU_y)
#print(type(MU_y))
```

    <class 'pymc.PyMCObjects.Deterministic'>

Let's see the parents of `y0_stdev`...

```python
y0_stdev.parents
```

    {'rho': <pymc.distributions.new_dist_class.<locals>.new_class 'rho' at 0x7f75561a3048>,
     'sigma': <pymc.distributions.new_dist_class.<locals>.new_class 'sigma_x' at 0x7f75561a39b0>}

Notice that this is a dictionary, so for example...

```python
y0_stdev.parents['rho'].value
```

    array(0.6307927826852653)

```python
rho.random()
y0_stdev.parents['rho'].value     # if the parent is a pymc variable, the current value will be always 'updated'
```

    array(-0.8067244934281792)

... and as we alter the parent's value, the child's value changes accordingly

```python
print("Current value of y0_stdev = {: f}".format(y0_stdev.value))

rho.random()
print('\nAfter redrawing rho:')
print("Current value of y0_stdev = {: f}".format(y0_stdev.value))
```

    Current value of y0_stdev =  0.951137

    After redrawing rho:
    Current value of y0_stdev =  0.564751

and similarly for `mu_y`

```python
print("Current value of mu_y:")
print(mu_y.value[:4])
rho.random()
print('\nAfter redrawing rho:')
print("Current value of mu_y:")
print(mu_y.value[:4])
```

    Current value of mu_y:
    [ 0.04004946  0.04786201  0.02833744  0.1285417 ]

    After redrawing rho:
    Current value of mu_y:
    [ 0.38506324  0.4601785   0.27245577  1.23588879]

### How to tell `pymc` what you 'know' about the data?

We define the data as a stochastic variable with fixed values and set the `observed` flag equal to `True`

For the sample $y^T$, depending on the question at hand, we might want to define
   - either $T + 1$ scalar random variables
   - or a scalar $y_0$ and a $T$-vector valued $Y$

In the current setup, as we fix the value of $y$ (observed), it doesn't really matter (approach A is easier). However, if we have an array-valued stochastic variable with mutable value, the restriction that we cannot update the values of stochastic variables' in-place becomes onerous in the sampling step (where the step method should propose array-valued variable). Straight from the pymc documentation:
>''In this case, it may be preferable to partition the variable into several scalar-valued variables stored in an array or list.''

#### (A) $y_0$ as a scalar and $Y$ as a vector valued random variable

```python
y0 = pm.Normal('y0', mu = 0.0, tau = 1 / y0_stdev, observed = True, value = sample[0])
Y = pm.Normal('Y', mu = mu_y, tau = 1 / sigma_x, observed=True, value = sample[1:])
```

```python
Y.value
```

    array([-0.49029625, -0.29028745, -1.31677523, -0.2966883 , -0.5337306 ,
           -3.36540298, -2.02043236, -1.0834821 ,  0.01594337, -0.57355078,
            0.92050047, -0.09611809, -0.21923504,  0.80092806,  0.0518965 ,
            0.60521111,  2.05571842,  0.5691191 ,  0.84095268,  0.4107615 ])

Notice that the value of this variable is fixed (even if the parent's value changes)

```python
Y.parents['tau'].value
```

    1.7791903326271949

```python
sigma_x.random()
print(Y.parents['tau'].value)
Y.value
```

    2.20473853378

    array([-0.49029625, -0.29028745, -1.31677523, -0.2966883 , -0.5337306 ,
           -3.36540298, -2.02043236, -1.0834821 ,  0.01594337, -0.57355078,
            0.92050047, -0.09611809, -0.21923504,  0.80092806,  0.0518965 ,
            0.60521111,  2.05571842,  0.5691191 ,  0.84095268,  0.4107615 ])

#### (B) $T+1$ scalar random variables

Define an array with `dtype=object`, fill it with scalar variables (use loops) and define it as a `pymc.Container` (this latter step is not necessary, but based on my experience Container types work much more smoothly in the blocking step when we are sampling).

```python
Y_alt = np.empty(T + 1, dtype = object)
Y_alt[0] = y0        # definition of y0 is the same as above

for i in range(1, T + 1):
    Y_alt[i] = pm.Normal('y_{:d}'.format(i), mu = mu_y[i-1], tau = 1 / sigma_x)

print(type(Y_alt))
Y_alt
```

    <class 'numpy.ndarray'>

    array([ <pymc.distributions.new_dist_class.<locals>.new_class 'y0' at 0x7f755613bf60>,
           <pymc.distributions.new_dist_class.<locals>.new_class 'y_1' at 0x7f7556220c50>,
           <pymc.distributions.new_dist_class.<locals>.new_class 'y_2' at 0x7f75562200f0>,
           <pymc.distributions.new_dist_class.<locals>.new_class 'y_3' at 0x7f7556220160>,
           <pymc.distributions.new_dist_class.<locals>.new_class 'y_4' at 0x7f7556220b70>,
           <pymc.distributions.new_dist_class.<locals>.new_class 'y_5' at 0x7f7556220b38>,
           <pymc.distributions.new_dist_class.<locals>.new_class 'y_6' at 0x7f75562204a8>,
           <pymc.distributions.new_dist_class.<locals>.new_class 'y_7' at 0x7f7556220390>,
           <pymc.distributions.new_dist_class.<locals>.new_class 'y_8' at 0x7f75562202b0>,
           <pymc.distributions.new_dist_class.<locals>.new_class 'y_9' at 0x7f75562207f0>,
           <pymc.distributions.new_dist_class.<locals>.new_class 'y_10' at 0x7f75560d4d68>,
           <pymc.distributions.new_dist_class.<locals>.new_class 'y_11' at 0x7f7556080400>,
           <pymc.distributions.new_dist_class.<locals>.new_class 'y_12' at 0x7f75560807f0>,
           <pymc.distributions.new_dist_class.<locals>.new_class 'y_13' at 0x7f75561a32e8>,
           <pymc.distributions.new_dist_class.<locals>.new_class 'y_14' at 0x7f75561a3780>,
           <pymc.distributions.new_dist_class.<locals>.new_class 'y_15' at 0x7f75561a3f98>,
           <pymc.distributions.new_dist_class.<locals>.new_class 'y_16' at 0x7f75561a37b8>,
           <pymc.distributions.new_dist_class.<locals>.new_class 'y_17' at 0x7f75561a3ef0>,
           <pymc.distributions.new_dist_class.<locals>.new_class 'y_18' at 0x7f75561a30b8>,
           <pymc.distributions.new_dist_class.<locals>.new_class 'y_19' at 0x7f75561a3fd0>,
           <pymc.distributions.new_dist_class.<locals>.new_class 'y_20' at 0x7f75561a3208>], dtype=object)

Currently, this is just a numpy array of `pymc.Deterministic` functions. We can make it a `pymc` object by using the `pymc.Container` type.

```python
Y_alt = pm.Container(Y_alt)
type(Y_alt)
```

    pymc.Container.ArrayContainer

and the pymc methods are applied element-wise.

### Create a `pymc.Model` instance
Remember that it is just a collection of random variables (`Stochastic` and `Deterministic`), hence

```python
ar1_model = pm.Model([rho, sigma_x, y0, Y, y0_stdev, mu_y])
```

```python
ar1_model.stochastics   # notice that this is an unordered set (!)
```

    {<pymc.distributions.new_dist_class.<locals>.new_class 'rho' at 0x7f75561a3048>,
     <pymc.distributions.new_dist_class.<locals>.new_class 'sigma_x' at 0x7f75561a39b0>}

```python
ar1_model.deterministics
```

    {<pymc.PyMCObjects.Deterministic 'y0_stdev' at 0x7f75561a3cf8>,
     <pymc.PyMCObjects.Deterministic '(rho_mul_[-0.41026485 -0.49029625 -0.29028745 -1.31677523 -0.2966883  -0.5337306
      -3.36540298 -2.02043236 -1.0834821   0.01594337 -0.57355078  0.92050047
      -0.09611809 -0.21923504  0.80092806  0.0518965   0.60521111  2.05571842
       0.5691191   0.84095268])' at 0x7f75561a3160>}

This object have very limited awareness of the structure of the probabilistic model that it describes and does not itslef possess methods for updating the values in the sampling methods.

----------------
# Fitting the model to the data (MCMC algorithm)

### MCMC algorithms

The joint prior distribution is sitting on an $N$-dimensional space, where $N$ is the number of parameters we are about to make inference on (see the figure below). Looking at the data through the probabilistic model deform the prior surface into the posterior surface, that we need to explore. In principle, we could naively search this space by picking random points in $\mathbb{R}^N$ and calculate the corresponding posterior value (Monte Carlo methods), but a more efficient (especially in higher dimensions) way is to do Markov Chain Monte Carlo (MCMC), which is basically an intelligent way of discovering the posterior surface.

MCMC is an iterative procedure: at every iteration, it proposes a nearby point in the space, then ask 'how likely that this point is close to the maximizer of the posterior surface?', it accepts the proposed point if the likelihood exceeds a particular level and rejects it otherwise (by going back to the old position). The key feature of MCMC is that it produces proposals by simulating a Markov chain for which the posterior is the unique, invariant limiting distribution. In other words, after a possible 'trasition period' (i.e. post converegence), it starts producing draws from the posterior.

### MCMC algorithm in `pymc`

By default it uses the *Metropolis-within-Gibbs* algorithm (in my oppinion), which is based on two simple principles:
1. **Blocking and conditioning:**
    - Divide the $N$ variables of $\theta$ into $K\leq N$ blocks and update every block by sampling from the conditional density, i.e. from the distribuition of the block parameters conditioned on all parameters in the other $K-1$ blocks being at their current values.
        * At scan $t$, cycle through the $K$ blocks
            $$\theta^{(t)} = [\theta^{(t)}_1, \theta^{(t)}_2, \theta^{(t)}_3, \dots, \theta^{(t)}_K] $$
        * Sample from the conditionals
    \begin{align}
    \theta_1^{(t+1)} &\sim f(\theta_1\hspace{1mm} | \hspace{1mm} \theta^{(t)}_2, \theta^{(t)}_3, \dots, \theta^{(t)}_K; \text{data}) \\
    \theta_2^{(t+1)} &\sim f(\theta_2\hspace{1mm} | \hspace{1mm} \theta^{(t+1)}_1, \theta^{(t)}_3, \dots, \theta^{(t)}_K; \text{data}) \\
    \theta_3^{(t+1)} &\sim f(\theta_3\hspace{1mm} | \hspace{1mm} \theta^{(t+1)}_1, \theta^{(t+1)}_2, \dots, \theta^{(t)}_K; \text{data}) \\
    \dots & \\
        \theta_K^{(t+1)} &\sim f(\theta_3\hspace{1mm} | \hspace{1mm} \theta^{(t+1)}_1, \theta^{(t+1)}_2, \dots, \theta^{(t+1)}_{K-1}; \text{data})
    \end{align}

2. **Sampling (choose/construct `pymc.StepMethod`):** if for a given block the conditional density $f$ can be expressed in (semi-)analytic form, use it, if not, use Metrololis-Hastings

  * Semi-closed form example: Foreward-backward sampler (Carter and Kohn, 1994):
  * Metropolis(-Hastings) algorithm:
      1. Start at $\theta$
      2. Propose a new point in the parameterspace according to some proposal density $J(\theta' | \theta)$ (e.g. random walk)
      3. Accept the proposed point with probability
      $$\alpha = \min\left( 1, \frac{p(\theta'\hspace{1mm} |\hspace{1mm} \text{data})\hspace{1mm} J(\theta \hspace{1mm}|\hspace{1mm} \theta')}{ p(\theta\hspace{1mm} |\hspace{1mm} \text{data})\hspace{1mm} J(\theta' \hspace{1mm}| \hspace{1mm}\theta)} \right) $$
          - If accept: Move to the proposed point $\theta'$ and return to Step 1.
          - If reject: Don't move, keep the point $\theta$ and return to Step 1.
      4. After a large number of iterations (once the Markov Chain convereged), return all accepted $\theta$ as a sample from the posterior

Again, a `pymc.Model` instance is not much more than a collection, for example, the model variables (blocks) are not matched with step methods determining how to update values in the sampling step. In order to do that, first we need to construct an MCMC instance, which is then ready to be sampled from.

MCMC‘s primary job is to create and coordinate a collection of **step methods**, each of which is responsible for updating one or more variables (blocks) at each step of the MCMC algorithm. By default, step methods are automatically assigned to variables by PyMC (after we call the sample method).

#### Main built-in `pymc.StepMethod`s
    * Metropolis
    * AdaptiveMetropolis
    * Slicer
    * Gibbs

you can assign step methods manually by calling the method `use_step_method(method, *args, **kwargs)`:

```python
M = pm.MCMC(ar1_model)
```

Notice that the step_methods are not assigned yet

```python
M.step_method_dict
```

    {<pymc.distributions.new_dist_class.<locals>.new_class 'y_12' at 0x7f75560807f0>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_19' at 0x7f75561a3fd0>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_16' at 0x7f75561a37b8>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_15' at 0x7f75561a3f98>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_14' at 0x7f75561a3780>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_4' at 0x7f7556220b70>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_5' at 0x7f7556220b38>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_17' at 0x7f75561a3ef0>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_13' at 0x7f75561a32e8>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_8' at 0x7f75562202b0>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_9' at 0x7f75562207f0>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_20' at 0x7f75561a3208>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'sigma_x' at 0x7f75561a39b0>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_3' at 0x7f7556220160>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_7' at 0x7f7556220390>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_10' at 0x7f75560d4d68>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_2' at 0x7f75562200f0>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_18' at 0x7f75561a30b8>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_6' at 0x7f75562204a8>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_1' at 0x7f7556220c50>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'rho' at 0x7f75561a3048>: [],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_11' at 0x7f7556080400>: []}

You can specify them now, or if you call the `sample` method, pymc will assign the step_methods automatically according to some rule

```python
# draw a sample of size 20,000, drop the first 1,000 and keep only every 5th draw
M.sample(iter = 50000, burn = 1000, thin = 5)
```

     [-----------------100%-----------------] 50000 of 50000 complete in 35.7 sec

... and you can check what kind of step methods have been assigned (the default in most cases is the Metropolis step method for non-observed stochastic variables, while in case of observed stochastics, we simply draw from the prior)

```python
M.step_method_dict
```

    {<pymc.distributions.new_dist_class.<locals>.new_class 'y_12' at 0x7f75560807f0>: [<pymc.StepMethods.DrawFromPrior at 0x7f7556229390>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_19' at 0x7f75561a3fd0>: [<pymc.StepMethods.DrawFromPrior at 0x7f7556229390>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_16' at 0x7f75561a37b8>: [<pymc.StepMethods.DrawFromPrior at 0x7f7556229390>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_15' at 0x7f75561a3f98>: [<pymc.StepMethods.DrawFromPrior at 0x7f7556229390>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_14' at 0x7f75561a3780>: [<pymc.StepMethods.DrawFromPrior at 0x7f7556229390>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_4' at 0x7f7556220b70>: [<pymc.StepMethods.DrawFromPrior at 0x7f7556229390>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_5' at 0x7f7556220b38>: [<pymc.StepMethods.DrawFromPrior at 0x7f7556229390>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_17' at 0x7f75561a3ef0>: [<pymc.StepMethods.DrawFromPrior at 0x7f7556229390>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_13' at 0x7f75561a32e8>: [<pymc.StepMethods.DrawFromPrior at 0x7f7556229390>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_8' at 0x7f75562202b0>: [<pymc.StepMethods.DrawFromPrior at 0x7f7556229390>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_9' at 0x7f75562207f0>: [<pymc.StepMethods.DrawFromPrior at 0x7f7556229390>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_20' at 0x7f75561a3208>: [<pymc.StepMethods.DrawFromPrior at 0x7f7556229390>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'sigma_x' at 0x7f75561a39b0>: [<pymc.StepMethods.Metropolis at 0x7f7556229470>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_3' at 0x7f7556220160>: [<pymc.StepMethods.DrawFromPrior at 0x7f7556229390>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_7' at 0x7f7556220390>: [<pymc.StepMethods.DrawFromPrior at 0x7f7556229390>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_10' at 0x7f75560d4d68>: [<pymc.StepMethods.DrawFromPrior at 0x7f7556229390>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_2' at 0x7f75562200f0>: [<pymc.StepMethods.DrawFromPrior at 0x7f7556229390>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_18' at 0x7f75561a30b8>: [<pymc.StepMethods.DrawFromPrior at 0x7f7556229390>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_6' at 0x7f75562204a8>: [<pymc.StepMethods.DrawFromPrior at 0x7f7556229390>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_1' at 0x7f7556220c50>: [<pymc.StepMethods.DrawFromPrior at 0x7f7556229390>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'rho' at 0x7f75561a3048>: [<pymc.StepMethods.Metropolis at 0x7f75562293c8>],
     <pymc.distributions.new_dist_class.<locals>.new_class 'y_11' at 0x7f7556080400>: [<pymc.StepMethods.DrawFromPrior at 0x7f7556229390>]}

The sample can be reached by the trace method (use the names you used at the initialization not the python name -- useful if the two coincide)

```python
M.trace('rho')[:20]
```

    array([ 0.38520033,  0.38572793,  0.24201839,  0.24201839,  0.25300562,
            0.49019388,  0.49019388,  0.58805794,  0.56849525,  0.56849525,
            0.56849525,  0.56849525,  0.72880629,  0.72880629,  0.51403447,
            0.66472121,  0.40543629,  0.40543629,  0.40543629,  0.40543629])

```python
M.trace('sigma_x')[:].shape
```

    (9800,)

Then this is just a numpy array, so you can do different sort of things with it. For example plot

```python
sigma_sample = M.trace('sigma_x')[:]
rho_sample = M.trace('rho')[:]

fig, ax = plt. subplots(1, 2, figsize = (15, 5))
ax[0].plot(sigma_sample)
ax[1].hist(sigma_sample)

```

Acutally, you don't have to waste your time on construction different subplots. `pymc`'s built-in plotting functionality creates pretty informative plots for you (baed on `matplotlib`). On the figure below
- Upper left subplot: trace,
- Lower left subplot: autocorrelation (try to resample the model with `thin=1`),
- Right subplot: histogram with the mean

```python
from pymc.Matplot import plot as fancy_plot
fancy_plot(M.trace('rho'))
```

For a non-graphical summary of the posterior use the `stats()` method

```python
M.stats('rho')

# Try also:
#M.summary()
```

    {'rho': {'95% HPD interval': array([ 0.1894716,  0.8630899]),
      'mc error': 0.0029385342710042278,
      'mean': 0.5298227172415475,
      'n': 9800,
      'quantiles': {2.5: 0.1767998875856974,
       25: 0.41720379051147455,
       50: 0.53323473221261153,
       75: 0.64675882923717398,
       97.5: 0.85677266144745734},
      'standard deviation': 0.17113578794137885}}

```python
N = len(rho_sample)
rho_pr = [rho.random() for i in range(N)]
sigma_pr = [sigma_x.random() for i in range(N)]

Prior = np.vstack([rho_pr, sigma_pr]).T
Posterior = np.vstack([rho_sample, sigma_sample]).T
```

```python
fig, bx = plt.subplots(1, 2, figsize = (17, 10), sharey = True)
sb.kdeplot(Prior, shade = True, cmap = 'PuBu', ax = bx[0])
bx[0].patch.set_facecolor('white')
bx[0].collections[0].set_alpha(0)
bx[0].axhline(y = sigma_x_true, color = 'DarkRed', lw =2)
bx[0].axvline(x = rho_true, color = 'DarkRed', lw =2)
bx[0].set_xlabel(r'$\rho$', fontsize = 18)
bx[0].set_ylabel(r'$\sigma_x$', fontsize = 18)
bx[0].set_title('Prior', fontsize = 20)

sb.kdeplot(Posterior, shade = True, cmap = 'PuBu', ax = bx[1])
bx[1].patch.set_facecolor('white')
bx[1].collections[0].set_alpha(0)
bx[1].axhline(y = sigma_x_true, color = 'DarkRed', lw =2)
bx[1].axvline(x = rho_true, color = 'DarkRed', lw =2)
bx[1].set_xlabel(r'$\rho$', fontsize = 18)
bx[1].set_ylabel(r'$\sigma_x$', fontsize = 18)
bx[1].set_title('Posterior', fontsize = 20)
plt.xlim(-1, 1)
plt.ylim(0, 1.5)
plt.tight_layout()
plt.savefig('beamer/prior_post.pdf')
```

```python
rho_grid = np.linspace(-1, 1, 100)
sigmay_grid = np.linspace(0, 1.5, 100)
U = sp.stats.uniform(-1, 2)
IG = sp.stats.invgamma(3)

fig2, cx = plt.subplots(2, 2, figsize = (17, 12), sharey = True)

cx[0, 0].plot(rho_grid, U.pdf(rho_grid), 'r-', lw = 3, alpha = 0.6, label = r'$\rho$ prior')
cx[0, 0].set_title(r"Marginal prior for $\rho$", fontsize = 18)
cx[0, 0].axvline(x = rho_true, color = 'DarkRed', lw = 2, linestyle = '--', label = r'True $\rho$')
cx[0, 0].legend(loc='best', fontsize = 16)
cx[0, 0].set_xlim(-1, 1)

sb.distplot(rho_sample, ax = cx[0,1], kde_kws={"color": "r", "lw": 3, "label": r"$\rho$ posterior"})
cx[0, 1].set_title(r"Marginal posterior for $\rho$", fontsize = 18)
cx[0, 1].axvline(x = rho_true, color = 'DarkRed',  lw = 2, linestyle = '--', label = r'True $\rho$')
cx[0, 1].legend(loc='best', fontsize = 16)
cx[0, 1].set_xlim(-1, 1)

cx[1, 0].plot(sigmay_grid, IG.pdf(sigmay_grid), 'r-', lw=3, alpha=0.6, label=r'$\sigma_y$ prior')
cx[1, 0].set_title(r"Marginal prior for $\sigma_y$", fontsize = 18)
cx[1, 0].axvline(x = sigma_x_true, color = 'DarkRed',  lw = 2, linestyle = '--', label = r'True $\sigma_y$')
cx[1, 0].legend(loc = 'best', fontsize = 16)
cx[1, 0].set_xlim(0, 3)

sb.distplot(sigma_sample, ax = cx[1,1], kde_kws={"color": "r", "lw": 3, "label": r"$\sigma_y$ posterior"})
cx[1, 1].set_title(r"Marginal posterior for $\sigma_y$", fontsize = 18)
cx[1, 1].axvline(x = sigma_x_true, color = 'DarkRed',  lw = 2, linestyle = '--', label = r'True $\sigma_y$')
cx[1, 1].legend(loc = 'best', fontsize = 16)
cx[1, 1].set_xlim(0, 3)
plt.tight_layout()
plt.savefig('beamer/marginal_prior_post.pdf')
```

```python

```

## Sources and further reading:

`pymc` official documentation: https://pymc-devs.github.io/pymc/index.html

Rich set of fun examples (very easy read) -- **Probabilistic Programming & Bayesian Methods for Hackers**
http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/

Nice example about `potential`: http://healthyalgorithms.com/2008/11/05/mcmc-in-python-pymc-to-sample-uniformly-from-a-convex-body/

Non-trivial example comparing the Gibbs and Metropolis algorithms:
https://github.com/aflaxman/pymc-examples/blob/master/gibbs_for_uniform_ball.ipynb

Another example: https://users.obs.carnegiescience.edu/cburns/ipynbs/PyMC.html


# Critical Points and Optimization
We've explored various techniques that we can use to calculate the derivative of a function at a specific *x* value; in other words, we can determine the *slope* of the line created by the function at any point on the line.

This ability to calculate the slope means that we can use derivatives to determine some interesting properties of the function.

## Function Direction at a Point
Consider the following function, which represents the trajectory of a ball that has been kicked on a football field:

\begin{equation}k(x) = -10x^{2} + 100x + 3 \end{equation}

Run the Python code below to graph this function and see the trajectory of the ball over a period of 10 seconds.

```python
%matplotlib inline

# Create function k
def k(x):
    return -10*(x**2) + (100*x)  + 3

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [k(i) for i in x]

# Set up the graph
plt.xlabel('x (time in seconds)')
plt.ylabel('k(x) (height in feet)')
plt.xticks(range(0,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

plt.show()
```

By looking at the graph of this function, you can see that it describes a parabola in which the ball rose in height before falling back to the ground. On the graph, it's fairly easy to see when the ball was rising and when it was falling.

Of course, we can also use  derivative to determine the slope of the function at any point. We can apply some of the rules we've discussed previously to determine the derivative function:

- We can add together the derivatives of the individual terms (***-10x2***, ***100x***, and ***3***) to find the derivative of the entire function.
- The *power* rule tells us that the derivative of ***-10x2*** is ***-10 • 2x***, which is ***-20x***.
- The *power* rule also tells us that the derivative of ***100x*** is ***100***.
- The derivative of any constant, such as ***3*** is ***0***.

So:

\begin{equation}k'(x) = -20x + 100 + 0 \end{equation}

Which of course simplifies to:

\begin{equation}k'(x) = -20x + 100 \end{equation}

Now we can use this derivative function to find the slope for any value of ***x***.

Run the cell below to see a graph of the function and its derivative function:

```python
%matplotlib inline

# Create function k
def k(x):
    return -10*(x**2) + (100*x)  + 3

def kd(x):
    return -20*x + 100

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [k(i) for i in x]

# Use the derivative function to get the derivative values
yd = [kd(i) for i in x]

# Set up the graph
plt.xlabel('x (time in seconds)')
plt.ylabel('k(x) (height in feet)')
plt.xticks(range(0,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the derivative
plt.plot(x,yd, color='purple')

plt.show()
```

Look closely at the purple line representing the derivative function, and note that it is a constant  decreasing value - in other words, the slope of the function is reducing linearly as x increases. Even though the function value itself is increasing for the first half of the parabola (while the ball is rising), the slope is becoming less steep (the ball is not rising at such a high rate), until finally the ball reaches its apogee and the slope becomes negative (the ball begins falling).

Note also that the point where the derivative line crosses 0 on the y-axis is also the point where the function value stops increasing and starts decreasing. When the slope has a positive value, the function is increasing; and when the slope has a negative value, the function is decreasing.

The fact that the derivative line crosses 0 at the highest point of the function makes sense if you think about it logically. If you were to draw the tangent line representing the slope at each point, it would be rotating clockwise throughout the graph, initially pointing up and to the right as the ball rises, and turning until it is pointing down and right as the ball falls. At the highest point, the tangent line would be perfectly horizontal, representing a slope of 0.

Run the following code to visualize this:

```python
%matplotlib inline

# Create function k
def k(x):
    return -10*(x**2) + (100*x)  + 3

def kd(x):
    return -20*x + 100

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [k(i) for i in x]

# Use the derivative function to get the derivative values
yd = [kd(i) for i in x]

# Set up the graph
plt.xlabel('x (time in seconds)')
plt.ylabel('k(x) (height in feet)')
plt.xticks(range(0,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the derivative
plt.plot(x,yd, color='purple')

# Plot tangent slopes for x = 2, 5, and 8
x1 = 2
x2 = 5
x3 = 8
plt.plot([x1-1,x1+1],[k(x1)-(kd(x1)),k(x1)+(kd(x1))], color='red')
plt.plot([x2-1,x2+1],[k(x2)-(kd(x2)),k(x2)+(kd(x2))], color='red')
plt.plot([x3-1,x3+1],[k(x3)-(kd(x3)),k(x3)+(kd(x3))], color='red')

plt.show()
```

Now consider the following function, which represents the number of flowers growing in a flower bed before and after the spraying of a fertilizer:

\begin{equation}w(x) = x^{2} + 2x + 7 \end{equation}

```python
%matplotlib inline

# Create function w
def w(x):
    return (x**2) + (2*x) + 7

def wd(x):
    return 2*x + 2

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(-10, 11))

# Use the function to get the y values
y = [w(i) for i in x]

# Use the derivative function to get the derivative values
yd = [wd(i) for i in x]

# Set up the graph
plt.xlabel('x (time in days)')
plt.ylabel('w(x) (flowers)')
plt.xticks(range(-10,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the derivative
plt.plot(x,yd, color='purple')

plt.show()
```

Note that the green line represents the function, showing the number of flowers for 10 days before and after the fertilizer treatment. Before treatment, the number of flowers was in decline, and after treatment the flower bed started to recover.

The derivative function is shown in purple, and once again shows a linear change in slope. This time, the slope is increasing at a constant rate; and once again, the derivative function line crosses 0 at the lowest point in the function line (in other words, the slope changed from negative to positive when the flowers started to recover).

## Critical Points
From what we've seen so far, it seems that there is a relationship between a function reaching an extreme value (a maximum or a minimum), and a derivative value of 0. This makes intuitive sense; the derivative represents the slope of the line, so when a function changes from a negative slope to a positive slope, or vice-versa, the derivative must pass through 0.

However, you need to be careful not to assume that just because the derivative is 0 at a given point, that this point represents the minimum or maximum of the function. For example, consider the following function:

\begin{equation}v(x) = x^{3} - 2x + 100 \end{equation}

Run the following Python code to visualize this function and its corresponding derivative function:

```python
%matplotlib inline

# Create function v
def v(x):
    return (x**3) - (2*x) + 100

def vd(x):
    return 3*(x**2) - 2

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(-10, 11))

# Use the function to get the y values
y = [v(i) for i in x]

# Use the derivative function to get the derivative values
yd = [vd(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('v(x)')
plt.xticks(range(-10,15, 1))
plt.yticks(range(-1000, 2000, 100))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the derivative
plt.plot(x,yd, color='purple')

plt.show()
```

Note that in this case, the purple derivative function line passes through 0 as the green function line transitions from a *concave downwards* slope (a slope that is decreasing) to a *concave upwards* slope (a slope that is increasing). The slope flattens out to 0, forming a "saddle" before the it starts increasing.

What we can learn from this is that interesting things seem to happen to the function when the derivative is 0. We call points where the derivative crosses 0 *critical points*, because they indicate that the function is changing direction. When a function changes direction from positive to negative, it forms a peak (or a *local maximum*), when the function changes direction from negative to positive it forms a trough (or *local minimum*), and when it maintains the same overall direction but changes the concavity of the slope it creates an *inflexion point*.

## Finding Minima and Maxima
A common use of calculus is to find minimum and maximum points in a function. For example, we might want to find out how many seconds it took for the kicked football to reach its maximum height, or how long it took for our fertilizer to be effective in reversing the decline of flower growth.

We've seen that when a function changes direction to create a maximum peak or a minimum trough, the derivative of the function is 0, so a step towards finding these extreme points might be to simply find all of the points in the function where the derivative is 0. For example, here's our function for the kicked football:

\begin{equation}k(x) = -10x^{2} + 100x + 3 \end{equation}

From this, we've calculated the function for the derivative as:

\begin{equation}k'(x) = -20x + 100 \end{equation}

We can then solve the derivative equation for an f'(x) value of 0:

\begin{equation}-20x + 100 = 0 \end{equation}

We can remove the constant by subtracting 100 to both sides:

\begin{equation}-20x = -100 \end{equation}

Multiplying both sides by -1 gets rid of the negative values (this isn't strictly necessary, but makes the equation a little less confusing)

\begin{equation}20x = 100 \end{equation}

So:

\begin{equation}x = 5 \end{equation}

So we know that the derivative will be 0 when *x* is 5, but is this a minimum, a maximum, or neither? It could just be an inflexion point, or the entire function could be a constant value with a slope of 0) Without looking at the graph, it's difficult to tell.

## Second Order Derivatives
The solution to our problem is to find the derivative of the derivative! Until now, we've found the derivative of a function, and indicated it as ***f'(x)***. Technically, this is known as the *prime* derivative; and it describes the slope of the function. Since the derivative function is itself a function, we can find its derivative, which we call the *second order* (or sometimes just *second*) derivative. This is indicated like this: ***f''(x)***.

So, here's our function for the kicked football:

\begin{equation}k(x) = -10x^{2} + 100x + 3 \end{equation}

Here's the function for the prime derivative:

\begin{equation}k'(x) = -20x + 100 \end{equation}

And using a combination of the power rule and the constant rule, here's the function for the second derivative:

\begin{equation}k''(x) = -20 \end{equation}

Now, without even drawing the graph, we can see that the second derivative has a constant value; so we know that the slope of the prime derivative is linear; and because it's a negative value, we know that it is decreasing. So when the prime derivative crosses 0, it we know that the slope of the function is decreasing linearly; so the point at *x=0* must be a maximum point.

Run the following code to plot the function, the prime derivative, and the second derivative for the kicked ball:

```python
%matplotlib inline

# Create function k
def k(x):
    return -10*(x**2) + (100*x)  + 3

def kd(x):
    return -20*x + 100

def k2d(x):
    return -20

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(0, 11))

# Use the function to get the y values
y = [k(i) for i in x]

# Use the derivative function to get the k'(x) values
yd = [kd(i) for i in x]

# Use the 2-derivative function to get the k''(x)
y2d = [k2d(i) for i in x]

# Set up the graph
plt.xlabel('x (time in seconds)')
plt.ylabel('k(x) (height in feet)')
plt.xticks(range(0,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot  k'(x)
plt.plot(x,yd, color='purple')

# Plot k''(x)
plt.plot(x,y2d, color='magenta')

plt.show()
```

Let's take the same approach for the flower bed problem. Here's the function:

\begin{equation}w(x) = x^{2} + 2x + 7 \end{equation}

Using the power rule and constant rule, gives us the prime derivative function:

\begin{equation}w'(x) = 2x + 2 \end{equation}

Applying the power rule and constant rule to the prime derivative function gives us the second derivative function:

\begin{equation}w''(x) = 2 \end{equation}

Note that this time, the second derivative is a positive constant, so the prime derivative (which is the slope of the function) is increasing linearly. The point where the prime derivative crosses 0 must therefore be a minimum. Let's run the code below to check:

```python
%matplotlib inline

# Create function w
def w(x):
    return (x**2) + (2*x) + 7

def wd(x):
    return 2*x + 2

def w2d(x):
    return 2

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(-10, 11))

# Use the function to get the y values
y = [w(i) for i in x]

# Use the derivative function to get the w'(x) values
yd = [wd(i) for i in x]

# Use the 2-derivative function to get the w''(x) values
y2d = [w2d(i) for i in x]

# Set up the graph
plt.xlabel('x (time in days)')
plt.ylabel('w(x) (flowers)')
plt.xticks(range(-10,15, 1))
plt.yticks(range(-200, 500, 20))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot w'(x)
plt.plot(x,yd, color='purple')

# Plot w''(x)
plt.plot(x,y2d, color='magenta')

plt.show()
```

## Critical Points that are *Not* Maxima or Minima
Of course, it's possible for a function to form a "saddle" where the prime derivative is zero at a point that is not a minimum or maximum. Here's an example of a function like this:

\begin{equation}v(x) = x^{3} - 6x^{2} + 12x + 2 \end{equation}

And here's its prime derivative:

\begin{equation}v'(x) = 3x^{2} - 12x + 12 \end{equation}

Let's find a critical point where v'(x) = 0

\begin{equation}3x^{2} - 12x + 12 = 0 \end{equation}

Factor the x-terms

\begin{equation}3x(x - 4) = 12 \end{equation}

Divide both sides by 3:

\begin{equation}x(x - 4) = 4 \end{equation}

Factor the x terms back again

\begin{equation}x^{2} - 4x = 4 \end{equation}

Complete the square, step 1

\begin{equation}x^{2} - 4x + 4 = 0 \end{equation}

Complete the square, step 2

\begin{equation}(x - 2)^{2} = 0 \end{equation}

Find the square root:

\begin{equation}x - 2 = \pm\sqrt{0}\end{equation}

\begin{equation}x - 2 = +\sqrt{0} = 0, -\sqrt{0} = 0\end{equation}

v'(2) = 0 (only touches 0 once)

Is it a maximum or minimum? Let's find the second derivative:

\begin{equation}v''(x) = 6x - 12\end{equation}

So

\begin{equation}v''(2) = 0\end{equation}

So it's neither negative or positive, so it's not a maximum or minimum.

```python
%matplotlib inline

# Create function v
def v(x):
    return (x**3) - (6*(x**2)) + (12*x) + 2

def vd(x):
    return (3*(x**2)) - (12*x) + 12

def v2d(x):
    return (3*(2*x)) - 12

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(-5, 11))

# Use the function to get the y values
y = [v(i) for i in x]

# Use the derivative function to get the derivative values
yd = [vd(i) for i in x]

# Use the derivative function to get the derivative values
y2d = [v2d(i) for i in x]

# Set up the graph
plt.xlabel('x')
plt.ylabel('v(x)')
plt.xticks(range(-10,15, 1))
plt.yticks(range(-2000, 2000, 50))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

# Plot the derivative
plt.plot(x,yd, color='purple')

# Plot the derivative
plt.plot(x,y2d, color='magenta')

plt.show()

print ("v(2) = " + str(v(2)))

print ("v'(2) = " + str(vd(2)))

print ("v''(2) = " + str(v2d(2)))

```

## Optimization
The ability to use derivatives to find minima and maxima of a function makes it a useful tool for scenarios where you need to optimize a function for a specific variable.

### Defining Functions to be Optimized
For example, suppose you have decided to build an online video service that is based on a subscription model. You plan to charge a monthly subscription fee, and you want to make the most revenue possible. The problem is that customers are price-sensitive, so if you set the monthly fee too high, you'll deter some customers from signing up. Conversely, if you set the fee too low, you may get more customers, but at the cost of reduced revenue.

What you need is some kind of function that will tell you how many subscriptions you might expect to get based on a given fee. So you've done some research, and found a formula to indicate that the expected subscription volume (in thousands) can be calculated as 5-times the monthly fee subtracted from 100; or expressed as a function:

\begin{equation}s(x) = -5x + 100\end{equation}

What you actually want to optimize is monthly revenue, which is simply the number of subscribers multiplied by the fee:

\begin{equation}r(x) = s(x) \cdot x\end{equation}

We can combine ***s(x)*** into ***r(x)*** like this:

\begin{equation}r(x) = -5x^{2} + 100x\end{equation}

### Finding the Prime Derivative
The function ***r(x)*** will return the expected monthly revenue (in thousands) for any proposed fee (*x*). What we need to do now is to find the fee that yields the maximum revenue. Fortunately, we can use a derivative to do that.

First, we need to determine the prime derivative of ***r(x)***, and we can do that easily using the power rule:

\begin{equation}r'(x) = 2 \cdot -5x + 100\end{equation}

Which is:

\begin{equation}r'(x) = -10x + 100\end{equation}

### Find Critical Points
Now we need to find any critical points where the derivative is 0, as this could indicate a maximum:

\begin{equation}-10x + 100 = 0\end{equation}

Let's isolate the *x* term:

\begin{equation}-10x = -100\end{equation}

Both sides are negative, so we can mulitply both by -1 to make them positive without affecting the equation:

\begin{equation}10x = 100\end{equation}

Now we can divide both sides by 10 to isolate *x*:

\begin{equation}x = \frac{100}{10}\end{equation}

So:

\begin{equation}x = 10\end{equation}

#### Check for a Maximum
We now know that with an *x* value of of **10**, the derivative is 0; or put another way, when the fee is 10, the slope indicating the change in subscription volume is flat. This could potentially be a point where the change in subscription volume has peaked (in other words, a maximum); but it could also be a minimum or just an inflexion point where the rate of change transitions from negative to positive.

To be sure, we can check the second order derivative. We can calculate this by applying the power rule to the prime derivative:

\begin{equation}r''(x) = -10\end{equation}

Note that the second derivative is a constant with a negative value. It will be the same for any point, including our critical point at *x=10*:

\begin{equation}r''(10) = -10\end{equation}

A negative value for the second derivative tells us that the derivative slope is moving in a negative direction at the point where it is 0, so the function value must be at a maximum.

In other words, the optimal monthly fee for our online video service is 10 - this will generate the maximum monthly revenue.

Run the code below to show the function ***r(x)*** as a graph, and verify that the maximum point is at x = 10.

```python
%matplotlib inline

# Create function s
def s(x):
    return (-5*x) + 100

# Create function r
def r(x):
    return s(x) * x

from matplotlib import pyplot as plt

# Create an array of x values to plot
x = list(range(0, 21))

# Use the function to get the y values
y = [r(i) for i in x]

# Set up the graph
plt.xlabel('x (monthly fee)')
plt.ylabel('r(x) (revenue in $,000)')
plt.xticks(range(0,22, 1))
plt.yticks(range(0, 600, 50))
plt.grid()

# Plot the function
plt.plot(x,y, color='green')

plt.show()
```

```python

```

```python

```

Basics of Calculus by Hiren/02-04-Critical Points and Optimization.ipynb


```julia
using CSV
using DataFrames
using PyPlot
using ScikitLearn # machine learning package
using StatsBase
using Random
using LaTeXStrings # for L"$x$" to work instead of needing to do "\$x\$"
using Printf

# (optional)change settings for all plots at once, e.g. font size
rcParams = PyPlot.PyDict(PyPlot.matplotlib."rcParams")
rcParams["font.size"] = 16

# (optional) change the style. see styles here: https://matplotlib.org/3.1.1/gallery/style_sheets/style_sheets_reference.html
PyPlot.matplotlib.style.use("seaborn-white")
```

## classifying breast tumors as malignant or benign

source: [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic))

> Features are computed from a digitized image of a fine needle aspirate (FNA) of a breast mass. They describe characteristics of the cell nuclei present in the image.

The mean radius and smoothness of the cell nuclei (the two features) and the outcome (M = malignant, B = benign) of the tumor are in the `breast_cancer_data.csv`.

```julia
df = CSV.read("breast_cancer_data.csv")
df[!, :class] = map(row -> row == "B" ? 0 : 1, df[:, :outcome])
first(df, 5)
```

	mean_radius	mean_smoothness	outcome	class

	Float64	Float64	String	Int64

5 rows × 4 columns

1	13.85	1.495	B	0

2	9.668	2.275	B	0

3	9.295	2.388	B	0

4	19.69	4.585	M	1

5	9.755	1.243	B	0

## visualize the two classes distributed in feature space

```julia
markers = Dict("M" => "x", "B" => "o")

figure()
xlabel("mean radius")
ylabel("mean smoothness")
for df_c in groupby(df, :outcome)
    outcome = df_c[1, :outcome]
    scatter(df_c[:, :mean_radius], df_c[:, :mean_smoothness], label="$outcome",
        marker=markers[outcome])
end
legend()
axis("equal")
```

## get data ready for classifiation in scikitlearn

scikitlearn takes as input:
* a feature matrix `X`, which must be `n_samples` by `n_features`
* a target vector `y`, which must be `n_samples` long (of course)

```julia
n_tumors = nrow(df)

X = zeros(n_tumors, 2)
y = zeros(n_tumors)
for (i, tumor) in enumerate(eachrow(df))
    X[i, 1] = tumor[:mean_radius]
    X[i, 2] = tumor[:mean_smoothness]
    y[i] = tumor[:class]
end
X # look at y too!
```

    300×2 Array{Float64,2}:
     13.85   1.495
      9.668  2.275
      9.295  2.388
     19.69   4.585
      9.755  1.243
     16.11   4.533
     14.78   2.45
     15.78   3.598
     15.71   1.972
     14.68   3.195
     13.71   3.856
     21.09   4.414
     11.31   1.831
      ⋮
     11.08   1.719
     18.94   5.486
     15.32   4.061
     14.25   5.373
     20.6    5.772
      8.671  1.435
     11.64   2.155
     12.06   1.171
     13.88   1.709
     14.9    3.466
     19.59   2.916
     14.81   1.677

## logistic regression

let $\mathbf{x} \in \mathbb{R}^2$ be the feature vector describing a tumor. let $T$ be the random variable that denotes whether the tumor is benign (0) or malignant (1). the logistic model is a probabilistic model for the probability that a tumor is malignant given its feature vector:

\begin{equation}
    \log \frac{Pr(T=1 | \mathbf{x})}{1-Pr(T=1 | \mathbf{x})} = \beta_0 + \boldsymbol \beta^\intercal \mathbf{x}
\end{equation}
where $\beta_0$ is the intercept and $\boldsymbol \beta \in \mathbb{R}$ are the weights for the features.

we will use scikitlearn to learn the  $\beta_0$ and $\boldsymbol \beta$ that maximize the likelihood.

```julia
@sk_import linear_model : LogisticRegression
```

    WARNING: redefining constant LogisticRegression

    PyObject <class 'sklearn.linear_model.logistic.LogisticRegression'>

```julia

```

prediction of the probability that a new tumor is 0 (benign) or 1 (malignant)

```julia

```

## visualize the learned model $Pr(T=1|\mathbf{x})$

```julia
radius = 5:0.25:30
smoothness = 0.0:0.25:20.0

lr_prediction = zeros(length(smoothness), length(radius))
for i = 1:length(radius)
    for j = 1:length(smoothness)
        x = [radius[i] smoothness[j]]
        lr_prediction[j, i] =
    end
end
```

```julia

```

## making decisions: the ROC curve

this depends on the cost of a false positive versus false negative. (here, "positive" is defined as testing positive for "malignant")

> "I equally value minimizing (1) false positives and (2) false negatives."

$\implies$ choose $Pr(T=1|\mathbf{x})=0.5$ as the decision boundary.

> "I'd rather predict that a benign tumor is malignant (false positive) than predict that a malignant tumor is benign (false negative)."

$\implies$ choose $Pr(T=1|\mathbf{x})=0.2$ as the decision boundary. Even if there is a relatively small chance that the tumor is malignant, we still take action and classify it as malignant...

the receiver operator characteristic (ROC) curve is a way we can evaluate a classification algorithm without imposing our values and specifying where the decision boundary should be.

```julia
@sk_import metrics : roc_curve
@sk_import metrics : auc
```

    WARNING: redefining constant roc_curve

    PyObject <function auc at 0x7fe80b1fb7b8>

```julia

```

```julia

```

```julia

```

tradeoff:
* threshold too small: classify all of the tumors as malignant, false positive rate very high
* threshold too large: classify all of the tumors as benign, false negative rate very high

somewhere in the middle (but still depending on the cost of a false positive versus false negative) is where we should operate.

the `auc`, area under the curve, has a probabilistic interpretation:
>  the area under the curve (often referred to as simply the AUC) is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one (assuming 'positive' ranks higher than 'negative') -[Wikipedia](https://en.wikipedia.org/wiki/Receiver_operating_characteristic)

**warning**: always split your data into test or train or do cross-validation to assess model performance. we trained on all data here to see the mechanics of fitting a logistic regression model to data, visualizing the model, and creating an ROC curve.


# Black-Scholes-Merton formula
## ${{C}_{0}}={{S}_{t}}N\left( {{d}_{1}} \right)-K\exp \left( -r\left( T-t \right) \right)N\left( {{d}_{2}} \right)$
## ${{d}_{1}}=\frac{\log \left( \frac{{{S}_{t}}}{K} \right)+\left( r+\frac{{{\sigma }^{2}}}{2} \right)\left( T-t \right)}{\sigma \sqrt{T-t}}$
## ${{d}_{2}}=\frac{\log \left( \frac{{{S}_{t}}}{K} \right)+\left( r-\frac{{{\sigma }^{2}}}{2} \right)\left( T-t \right)}{\sigma \sqrt{T-t}}={{d}_{1}}-\sigma \sqrt{T-t}$
## $\begin{align}
  & \Delta C=\frac{\partial C}{\partial S}\Delta S+\frac{\partial C}{\partial \sigma }\Delta \sigma +\frac{\partial C}{\partial t}\Delta t+\frac{1}{2}\frac{{{\partial }^{2}}C}{\partial {{S}^{2}}}{{\left( \Delta S \right)}^{2}}+\ldots  \\
 & \,\,\,\,\,\,\,\,=\text{Delta}\times \Delta S+\text{Vega}\times \Delta \sigma +\text{Theta}\times \Delta t+\frac{1}{2}\text{Gamma}\times {{\left( \Delta S \right)}^{2}}+\ldots  \\
\end{align}$
## ${{\Delta }_{c}}=\frac{\partial {{C}_{t}}}{\partial {{S}_{t}}}=N\left( {{d}_{1}} \right)$
## $\gamma =\frac{{{\partial }^{2}}{{C}_{t}}}{\partial S_{t}^{2}}=\frac{{{\partial }^{2}}{{P}_{t}}}{\partial S_{t}^{2}}=\left( \frac{n\left( {{d}_{1}} \right)}{\sigma {{S}_{t}}\sqrt{T-t}} \right)$
## ${{\theta }_{c}}=\frac{\partial {{C}_{t}}}{\partial t}=-rK\exp \left( -r\left( T-t \right) \right)N\left( {{d}_{2}} \right)-\frac{1}{2}{{\sigma }^{2}}S_{t}^{2}\gamma $
## ${{\kappa }_{c}}=\frac{\partial {{C}_{t}}}{\partial K}=\left[ N\left( -{{d}_{2}} \right)-1 \right]\exp \left( -r\left( T-t \right) \right)$
## ${{\rho }_{c}}=\frac{\partial {{C}_{t}}}{\partial r}=K\left( T-t \right)N\left( {{d}_{2}} \right)\exp \left( -r\left( T-t \right) \right)$
## $Veg{{a}_{c}}=Veg{{a}_{p}}=\frac{\partial {{C}_{t}}}{\partial \sigma }=KN\left( {{d}_{1}} \right)\sqrt{T-t}$

```python
# %load call_option.py
__version__ = '0.0.1'
__author__ = ' '
from scipy import stats
from math import log, sqrt, exp

class call_option:
    def __init__(self, S0, K, T, t, r, sigma):
        self.S0 = float(S0)
        self.K  = K
        self.T  = T
        self.t  = t
        self.r  = r
        self.sigma = sigma
        self.d1 = (log(S0/K)+(r+0.5*sigma**2)*(T-t))/(sigma*sqrt((T-t)))
        self.d2 = (log(S0/K)+(r-0.5*sigma**2)*(T-t))/(sigma*sqrt((T-t)))

    def __del__(self):
        class_name =self.__class__.__name__
        print(class_name, ':delete our call_option obj')

    def val(self):
        val=(self.S0*stats.norm.cdf(self.d1, 0.0, 1.0)-self.K*exp(-self.r*(self.T-self.t))*stats.norm.cdf(self.d2, 0.0, 1.0))
        return val

    def delta(self):
        delta=stats.norm.cdf(self.d1, 0.0, 1.0)
        return delta

    def gamma(self):
        gamma=stats.norm.pdf(self.d1, 0.0, 1.0)/(self.S0*self.sigma*sqrt(self.T-self.t))
        return gamma

    def theta(self,gamma):
        theta = -self.r*self.K*(sqrt(self.T-self.t))*stats.norm.cdf(self.d2)-0.5*gamma*(self.S0*self.sigma)*(self.S0*self.sigma)
        return theta

    def kappa(self):
        kappa= (stats.norm.cdf(-self.d2)-1.00)*exp(-self.r*(self.T-self.r))
        return kappa

    def rho(self):
        rho=self.K*(self.T-self.t)*stats.norm.cdf(self.d2)*exp(-self.r*(self.T-self.t))
        return rho

    def vega(self):
        vega = self.K * stats.norm.cdf(self.d1, 0.0, 1.0) * sqrt(self.T - self.t)
        return vega

    def impvol(self, C0, sigma_est= 0.2, N= 1000):
        opts = call_option(self.S0, self.K, self.T, self.t, self.r, sigma_est)
        for i in range(N):
            opts.sigma =opts.sigma-(opts.val()-C0)/opts.vega()
        return opts.sigma

    def prt_option_info(self):
        call_price=call_option(self.S0, self.K, self.T, self.t, self.r, self.sigma)
        value=call_price.val()
        veg=call_price.vega()
        delt=call_price.delta()
        gam=call_price.gamma()
        thet=call_price.theta(gam)
        kapp=call_price.kappa()
        rh=call_price.rho()
        iv=call_price.impvol(C0=value)
        print('Hello World!')
        print('When we assume \nS0=%f K=%f r=%f sigma=%f' %(self.S0, self.K, self.r, self.sigma))
        print('The length T\t=%f and the start time t\t=%f' %(self.T,self.t))
        print('The price of call option \t=\t %f' %value)
        print ('==============The Greek info================')
        print('The Delta of call option \t=\t %f' % delt)
        print('The Gamma of call option \t=\t %f' % gam)
        print('The Theta of call option \t=\t %f' % thet)
        print('The Kappa of call option \t=\t %f' % kapp)
        print('The Rho of call option \t\t=\t %f' % rh)
        print('The Vega of call option \t=\t %f' %veg)
        print('The implied volatility \t\t=\t %f' %iv)
        #return
```

# 下面我们来演示如何运行“主程序”
##  注意这种写代码的结构

```python
# %load demo_bsm_call_main.py
__version__ = '0.0.1'
__author__ = 'yf'

from call_option import *

def main():
    ST=50.00   #the price of underlying asset
    K=45.00    #the strike price
    T=1.0       #the length of call option contract
    t=0.0       #the initial time
    r=0.05      #risk-free rate
    sigma=0.25  #the volatility
    S0=ST*exp(-r*T)
    call_price=call_option(S0, K, T, t, r, sigma)
    call_price.prt_option_info()
    #del call_price

if __name__ == "__main__":
    main()
```

    call_option :delete our call_option obj
    When we assume
    S0=47.561471 K=45.000000 r=0.050000 sigma=0.250000
    The length T	=1.000000 and the start time t	=0.000000
    The price of call option 	=	 7.263615
    ==============The Greek info================
    The Delta of call option 	=	 0.707619
    The Gamma of call option 	=	 0.028898
    The Theta of call option 	=	 -3.430091
    The Kappa of call option 	=	 -0.587952
    The Rho of call option 		=	 26.391783
    The Vega of call option 	=	 31.842853
    The implied volatility 		=	 24.413612
    call_option :delete our call_option obj
    call_option :delete our call_option obj

# 蒙特卡罗模拟与欧式期权定价
## 蒙特卡罗计算欧式期权定价公式的思路其实非常简单，要点就是三点：
## 我们要模拟标的物，也就是股票价格的走势，我们将股票价格的走势假定为其满足对数正态分布，也就是股票价格满足以下式子：
## $\log {{S}_{t}}=\log {{S}_{t-\Delta t}}+\left( r-\frac{1}{2}{{\sigma }^{2}} \right)\Delta t+\sigma \sqrt{\Delta t}{{z}_{t}}$ 其中${{z}_{t}}\sim i.i.d\text{ N}\left( 0,1 \right)$
### 我们要建立一个期权定价的收益准则，即在何种情况下该期权应该获利。当然庆幸的是，我们研究的仅是欧式期权，我们知道欧式期权的收益准则非常简单，无非就是：
## ${{C}_{0}}={{e}^{-rT}}\max \left( {{S}_{T}}-K,0 \right)$
### 我们利用蒙特卡罗的模拟技术，模拟无数次未来标的物价格的走势（路径），然后根据欧式期权的收益准则，计算每一个模拟路径的可能的期权收益，然后求其均值，即：
## ${{C}_{0}}={{e}^{-rT}}\left( \frac{1}{N} \right)\sum\limits_{i=1}^{N}{\left\{ \max \left( {{S}^{i}}_{T}-K,0 \right) \right\}}$
### 我们简单写了一个Python的小程序来实现以上的量化定价策略，然后我们绘制了三个图，第一个图演示我们模拟的未来股价的走势，左下的图，我们绘制了模拟走势的股价的直方图，而右下模拟的是${{S}^{i}}_{T}-K$的分布，这个图说明啥呢？你可以看见有很多的结果出现在零处，也就是说，很多模拟结果显示，${{S}^{i}}_{T}-K$都是为零的，为零的情形都是表示不行权，也就是说，如果资产价格满足对数正态分布，很多时候当你买了期权，其实你能行权的概率并不高，很多时候都行不了权。

# 下面我们来演示蒙特卡罗方法的期权定价。数学原理完全一样。写代码的方法不一样。
## 第一种方法是基于 for-loop
## 第二种方法是基于 矢量化方法（vectorization）
### 下面我们看看第一种方法： for-loop 注意观察一下运行代码需要的时间。

```python
# %load mc_european_optionpricing_method1.py
from time import time
import math as mh
import random as rd

rd.seed(20000)
t0=time()
S0=50.00;K=55.00;T=1.0
r=0.05;sigma=0.2;M=50
dt=T/M;Iter=300000

S=[]
for i in range(Iter):
    path=[]
    for t in range(M+1):
        if t==0:
            path.append(S0)
        else:
            z = rd.gauss(0.0,1.0)
            St= path[t-1]*mh.exp((r-0.50*(sigma**2))*dt+sigma*mh.sqrt(dt)*z)
            path.append(St)
    S.append(path)

sum_val=0.0
for path in S:
    sum_val = sum_val+max(path[-1]-K,0)

C0=mh.exp(-r*T)*sum_val/Iter

time_comsume=time()-t0
print("European Option Price=",C0)
print("Time used=",time_comsume,"(seconds)")

```

    European Option Price= 3.0126656292980014
    Time used= 32.17306089401245 (seconds)

## 下面看看第二种方法，即基于矢量化方法。注意一下程序的运行时间。

```python
# %load mc_european_optionpricing_method2.py
%matplotlib inline
import math as mh
import numpy as np
from time import time
import matplotlib.pyplot as plt

np.random.seed(20000)
t0=time();
ST=50;K=45;T=1.0;
r=0.05;sigma=0.25;
S0=ST*np.exp(-r*T);
M=1000;
dt=T/M;I=100000

S=S0*np.exp(np.cumsum((r-0.5*sigma**2)*dt+sigma
                      *mh.sqrt(dt)*np.random.standard_normal((M+1,I)),axis=0))
S[0]=S0

C0=mh.exp(-r*T)*sum(np.maximum(S[-1]-K,0))/I

time_comsume=time()-t0
print('European Option Price=',C0)
print('Time used=',time_comsume,'(seconds)')

plt.subplot(2,2,1)
plt.subplot(2,2,2)
plt.subplot(2,1,1)
plt.plot(S[:,:100])
plt.grid(True)
plt.xlabel('Time Step')
plt.ylabel('Index Level')

plt.subplot(2,2,3)
plt.hist(S[-1],bins=50)
plt.grid(True)
plt.xlabel('Index Level')
plt.ylabel('Frequency')

plt.subplot(2,2,4)
plt.hist(np.maximum(S[-1]-K,0),bins=50)
plt.grid(True)
plt.xlabel('Index Level')
plt.ylabel('Frequency')

plt.show()

```

```python

```


# Laplacian filtering and combining of the methods

**Author: Uzhva Denis Romanovich**

**Lecturer: Soloviev Igor Pavlovich**

## Laplacian filtering
### Theory

Laplacian filtering is a type of spatial filtering aiming to sharpen an image.
Unlike box or weighted average filtering, which use the concept of an integral, laplacian is based on a derivatives.
Concerning the fact that we process discrete images, the derivatives thus become differences.
E.g., consider the 2-dimensional laplacian operator:
$$
\begin{equation}
\nabla^2 = \frac{\partial^2}{\partial x^2} + \frac{\partial^2}{\partial y^2}.
\tag{1}
\end{equation}
$$
Let $A \in \overline{0, 255}^{M \times N}$ be an $M \times N$ matrix of the image, containing values of intensity of corresponding pixels.
Let also $\widehat{A} \in \overline{0, 255}^{M + 2a \times N + 2b}$ be a padded $A$, i.e. $\widehat{A}_{i+a, j + a} = A_{i, j}$.
Then the discrete form of the laplacian would be defined as follows:
$$
\begin{equation}
\nabla^2 \widehat{A}_{i, j} = \widehat{A}_{i+1, j} + \widehat{A}_{i, j+1} + \widehat{A}_{i-1, j} + \widehat{A}_{i, j-1} - 4 \widehat{A}_{i, j},
\tag{2}
\end{equation}
$$
where $i \in \overline{2, M-1}$ and $j \in \overline{2, N-1}$.
Another definition is the following:
$$
\begin{equation}
\nabla^2 \widehat{A}_{i, j} = \widehat{A}_{i+1, j} + \widehat{A}_{i, j+1} + \widehat{A}_{i-1, j} + \widehat{A}_{i, j-1} + \widehat{A}_{i+1, j+1} + \widehat{A}_{i-1, j+1} + \widehat{A}_{i-1, j+1} + \widehat{A}_{i-1, j-1} - 8 \widehat{A}_{i, j},
\tag{3}
\end{equation}
$$
and the main advantage of it over the equation (2) is that it considers diagonals.

Finally, the result of application of the laplacian can be expressed by the following formula:
$$
\begin{equation}
B = A + c \nabla^2 \widehat{A},
\tag{4}
\end{equation}
$$
so that the result is the original image plus the extracted edges (which we want to highlight) times constant $c$, which may vary.

### Code

#### Laplacian filter

```python
import numpy as np

def apply_lapl(img, filt, c=1.):
    h = img.shape[0]
    w = img.shape[1]
    ker_size = filt.shape[0]
    pad_size = (ker_size - 1) // 2

    edg_ext = np.zeros_like(img, dtype=float)

    img_padded = np.zeros((h + 2 * pad_size, w + 2 * pad_size, channels))
    img_padded[pad_size:-pad_size, pad_size:-pad_size, :] = img

    for i in range(ker_size):
        for j in range(ker_size):
            ker_multiplier = filt[i, j]
            edg_ext = img_new + ker_multiplier * img_padded[i:i+h, j:j+h]

    edg_ext = c * edg_ext

    img_new = np.zeros_like(img, dtype=float)
    img_new = img + edg_ext
    img_new = 255 * img_new / np.max(img_new)
    img_new = img_new.astype(np.uint8)

    return img_new, edg_ext
```

#### Make laplacian kernel

```python
import numpy as np

def get_laplacian_kernel(diag=False):
    kernel = np.zeros((3, 3))
    if diag:
        kernel[:, :] = 1
        kernel[1, 1] = -8
    else:
        kernel[1, :] = 1
        kernel[:, 1] = 1
        kernel[1, 1] = -4

    return kernel
```

### Results

#### Processing + visualization

First of all, we need to load images for the further processing:

```python
import numpy as np
import matplotlib.pyplot as plt
from PIL import Image

# http://pages.cs.wisc.edu/~lizhang/projects/autoexpo/content/text.html
img_1 = Image.open('./text1.png')
# http://pages.cs.wisc.edu/~lizhang/projects/autoexpo/content/text.html
img_2 = Image.open('./text2.png')
```

Task4 - Laplacian filtering and combining of the methods/Laplacian filtering and combining of the methods.ipynb


```python
from sympy import *
from sympy.abc import r,x,y,z
from scipy.integrate import quad, nquad
import matplotlib.pyplot as plt
%matplotlib inline
init_printing()
```

# Energy of the Hydrogen Atom
The variational principle states a trial wavefunction will have an energy greater than or equal to the ground state energy.
$$\frac{\int \psi H \psi}{ \int \psi^2}  \ge E_0$$

First consider the hydogen atom. Let us use a trial wavefunction that is not the exact ground state.

```python
beta = Symbol('beta')
R_T = exp(-r - beta*r*r)
R_T
```

The Hamiltonian for this system is
$$-\frac{1}{2} \nabla^2 - \frac{1}{r}$$
The first term is the kinetic energy of the electron, and the second term is the Coulomb attraction between the electron and proton.

The first step is to compute the derivative of the trial wavefunction in spherical coordinates

```python
def del_spherical(e, r):
    """Compute Laplacian for expression e with respect to symbol r.
        Currently works only with radial dependence"""
    t1 = r*r*diff(e, r)
    t2 = diff(t1, r)/(r*r)
    return simplify(t2)
```

```python
del1 = del_spherical(R_T, r)

```

Construct $\psi H \psi$

```python
H = -1/S(2) * R_T * del1  - R_T*R_T/r
```

```python
simplify(H)
```

The integration occurs in 3D over the electron coordinates.  Because the integrand only has a dependence on $r$, it can be converted to spherical coordinates, and reduced to a 1D integral over $r$. (There should be an additional factor of $4 \pi$, but it will cancel since it occurs in the numerator and denominator)

```python
h1 = simplify(r*r*H)
```

Substitute a concrete value for $\beta$.

```python
h2 = h1.subs(beta, 0.1)
```

Perform the integral

```python
num = integrate(h2, (r, 0, oo)).evalf()
num
```

Also construct and integrate the denominator (the normalization).

```python
norm1 = r*r*R_T*R_T
norm2 = norm1.subs(beta, 0.1)
norm3 = simplify(norm2)
```

```python
denom = integrate(norm3, (r, 0, oo)).evalf()
simplify(denom).evalf()
```

```python
E = num/denom
E
```

And, as expected, energy is greater than the exact ground state energy of -0.5 Hartree.

## Find the minimum energy
Collect all the steps for computing the energy into a single function.   Even though this particular integral could be done symbolically, use numerical integration instead.

```python
def compute_energy(R_T, beta_val):
    """Energy given a value for beta"""

    # Normalization integrand (denominator)
    norm1 = r*r*R_T*R_T
    norm2 = norm1.subs(beta, beta_val)
    norm3 = simplify(norm2)

    # Integrand for the numerator
    del1 = del_spherical(R_T, r)
    # Construct psi * H * psi
    H = -1/S(2) * R_T * del1  - R_T*R_T/r
    h1 = simplify(r*r*H)
    h2 = h1.subs(beta, beta_val)

    lim = 20.0

    denom_func = lambdify([r], norm3)
    denom_res = quad(denom_func, 0.0, lim)

    num_func = lambdify([r], h2)
    num_res = quad(num_func, 0.0, lim)

    e = num_res[0]/denom_res[0]

    return e

```

Now the energy can be computed vs. $\beta$, and we can find the minimum energy.  In this case, the minimum occurs at $\beta = 0$, which we know is the exact wavefunction for the hydrogen atom.

```python
energies = []
betas = []
for i in range(10):
    beta_val = i*.01
    e = compute_energy(R_T, beta_val)
    betas.append(beta_val)
    energies.append(e)

plt.plot(betas, energies)
```

```python

```