```python
import numpy as np
import scipy
import scipy.stats
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
```

## Generazione dati
Generiamo $n$ samples da una distribuzione normale con valori dati di $\mu$ e  $\sigma$

```python
mu = 10
sigma = 3
n = 5000

mod=lambda t:np.random.normal(mu,sigma,t)

samples = mod(n)

fig = plt.figure(figsize=(12,10))
ax = fig.add_subplot(1,1,1)
ax.hist( samples,bins=35 ,)
ax.set_xlabel("Valore")
ax.set_ylabel("Frequenza")
ax.set_title("Distribuzione di {0:2d} sample da una distribuzione normale con $\mu={1:5.1f}$, $\sigma={2:5.1f}$".format(n,mu,sigma))
mu_obs=samples.mean()
```

## Modello bayesiano

Vogliamo stimare $\mu$ e $\sigma$ a partire dai dati, applicando MCMC. Nello specifico, vogliamo ottenere una sequenza di sample di

$$
p(\sigma,\mu|X)\propto p(X|\sigma,\mu)p(\sigma,\mu)=\prod_{i=1}^np(x_i|\sigma,\mu)p(\sigma,\mu)
$$

per semplicità, assumiamo di stimare $\mu$ come $$\overline{\mu}=\frac{1}{n}\sum_{i=1}^n x_i$$ che risulta pari a

```python
avg=samples.mean()
print(avg)
```

    10.029756798371398

Ne deriva che, limitatamente a $\sigma$, abbiamo

$$
p(\sigma|X,\overline{\mu})\propto p(X|\sigma,\overline{\mu})p(\sigma)=\prod_{i=1}^np(x_i|\sigma,\overline{\mu})p(\sigma)
$$

Assumendo sia la prior $p(\sigma)$ che la likelihood $p(x|\sigma,\overline{\mu})$ come gaussiane rispettivamente con media 0 e deviazione standard $\hat\sigma$ e media $\overline{\mu}$ e varianza $\sigma$, abbiamo che

\begin{equation}
p(\sigma|X,\overline{\mu})\propto \left(\prod_{i=1}^n\frac{1}{\sqrt{2\pi}\sigma}e^{-\dfrac{(x_i-\overline{\mu})^2}{2\sigma^2}}\right)\frac{1}{\sqrt{2\pi}\hat\sigma}e^{-\dfrac{\sigma^2}{2\hat\sigma^2}}\propto \sigma^{-n}e^{-\dfrac{\sigma^2}{2\hat\sigma^2}}\prod_{i=1}^ne^{-\dfrac{(x_i-\overline{\mu})^2}{2\sigma^2}}
\end{equation}

### Definizione della MC

Come distribuzione della probabilità di transizione $q(\sigma'|\sigma)$ assumiamo ad esempio una distribuzione normale con varianza unitaria

\begin{equation}
q(\sigma | \sigma') = \frac{1}{\sqrt{2\pi}}e^{-\dfrac{(\sigma-\sigma')^2}{2}}
\end{equation}

Si noti che $q(\sigma'|\sigma)=q(\sigma|\sigma')$, per cui possiamo applicare l'algoritmo di Metropolis

## Accettazione di $\sigma'$:
Applicando l'algoritmo di Metropolis $\sigma'$ è accettato:

- se $$
\frac{p(\sigma'|X,\overline{\mu})}{p(\sigma|X,\overline{\mu})}>1
$$

- con probabilità $$
\frac{p(\sigma'|X,\overline{\mu})}{p(\sigma|X,\overline{\mu})}
$$
altrimenti

## Passaggio ai logaritmi
La condizione di accettazione può essere espressa applicando il logaritmo ai termini in gioco e passando a prior e likelihood, ottenendo che $\sigma'$ è accettato:

- se $$
\log p(X|\sigma',\overline{\mu})+\log p(\sigma')-p(X|\sigma,\overline{\mu})-p(\sigma)>0
$$

- con probabilità $$
\frac{p(\overline{\sigma}|X,\overline{\mu})}{p(\sigma|X,\overline{\mu})}
$$
altrimenti

Ricordando le definizioni di likelihood e prior considerate, si ottiene quindi che la condizione da verificare è
$$
-n\log\sigma'-\sum_{i=1}^n\dfrac{(x_i-\overline{\mu})^2}{2\sigma'^2}-\dfrac{\sigma'^2}{2\hat\sigma^2}+n\log\sigma+\sum_{i=1}^n\dfrac{(x_i-\overline{\mu})^2}{2\sigma^2}+\dfrac{\sigma^2}{2\hat\sigma^2}>0
$$

```python
# Sampling dalla distribuzione di transizione

def transition(x):
    x_new = np.random.normal(x,1,(1,))[0]
    if x_new>0:
        return x_new
    else:
        return x

# Proporzionale al logaritmo della probabilità a priori
def log_prior(sigma):
    return -(sigma/sigma_hat)**2/2.0

# Proporzionale al logaritmo della likelihood dei dati
def log_likelihood(sigma,samples):
    n = len(samples)
    return -n*np.log(sigma)-np.sum((samples-avg)**2)/(2*sigma**2)

# Test di accettazione del nuovo valore
def acc(sigma_new, sigma):
    first_term = log_likelihood(sigma_new,samples)+log_prior(sigma_new)
    second_term = log_likelihood(sigma,samples)+log_prior(sigma)
    test_val = first_term-second_term
    if test_val>0:
        return True
    else:
        accept=np.random.uniform(0,1)
        return (accept < (np.exp(test_val)))

# Algoritmo di Metropolis
def metropolis(param_init,iterations):
    sigma = param_init
    values = []
    accepted_idx = []
    rejected_idx = []
    for i in range(iterations):
        sigma_new =  transition(sigma)
        values.append(sigma_new)
        if acc(sigma_new,sigma):
            sigma = sigma_new
            accepted_idx.append(i)
        else:
            rejected_idx.append(i)
    return np.array(values), np.array(accepted_idx), np.array(rejected_idx)
```

### Esegue l'algoritmo

```python
sigma_init = 5
iterations = 10000
sigma_hat = 5
values, accepted_idx, rejected_idx = metropolis(sigma_init, iterations)
```

```python
print('Numero valori generati: {0:3d}'.format(values.shape[0]))
print('Numero valori accettati: {0:3d}'.format(accepted_idx.shape[0]))
print('Numero valori respinti: {0:3d}'.format(rejected_idx.shape[0]))
```

    Numero valori generati: 10000
    Numero valori accettati: 364
    Numero valori respinti: 9636

### Traccia

```python
fig=plt.figure(figsize=(18,5))
plt.plot(values, alpha=.7)
plt.show()
```

```python
burn = int(0.1*iterations)

fig = plt.figure(figsize=(12,6))
ax = fig.add_subplot(1,1,1)
rejected_to_plot = rejected_idx[np.where(rejected_idx<burn)]
accepted_to_plot = accepted_idx[np.where(accepted_idx<burn)]
ax.plot( rejected_to_plot, values[rejected_to_plot], 'rx', label='Respinti',alpha=0.5)
ax.plot( accepted_to_plot, values[accepted_to_plot], 'b.', label='Accettati',alpha=0.5)
ax.set_xlabel("Iterazione")
ax.set_ylabel("$\sigma$")
ax.set_title("MCMC sampling di $\sigma$ con algoritmo di Metropolis. Primi {0:3d} valori di burn-in.".format(burn))
ax.grid()
ax.legend()
fig.tight_layout()
plt.show()
```

```python
fig = plt.figure(figsize=(12,6))
ax = fig.add_subplot(1,1,1)
rejected_to_plot_after_burn = rejected_idx[np.where(rejected_idx>burn)]
accepted_to_plot_after_burn = accepted_idx[np.where(accepted_idx>burn)]
ax.plot( rejected_to_plot_after_burn, values[rejected_to_plot_after_burn], 'rx', label='Respinti',alpha=0.5)
ax.plot( accepted_to_plot_after_burn, values[accepted_to_plot_after_burn], 'b.', label='Accettati',alpha=0.5)
ax.set_xlabel("Iterazione")
ax.set_ylabel("$\sigma$")
ax.set_title("MCMC sampling di $\sigma$ con algoritmo di Metropolis. {0:3d} valori successivi al burn.".format(iterations-burn))
ax.grid()
ax.legend()
fig.tight_layout()
plt.show()
```

```python
fig = plt.figure(figsize=(20,10))
ax = fig.add_subplot(1,1,1)
ax.hist(values[accepted_to_plot_after_burn], bins=20 ,density=True, alpha=.7)
ax.set_ylabel("Frequenza")
ax.set_xlabel("$\sigma$")
ax.set_title("Istogramma per $\sigma$")
fig.tight_layout()
plt.show()

```

```python
m = 5000
sigma_post=values[accepted_to_plot_after_burn].mean()
mod_emp=lambda t:np.random.normal(mu_obs,sigma_post,t)
observations = mod(m)
predictions = mod_emp(m)
fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(1,1,1)
ax.hist( observations,bins=70 ,label="Valori originali di {0:3d} nuovi elementi".format(m), alpha=.5)
ax.hist( predictions,bins=70 ,label="Valori predetti di {0:3d} elementi".format(m), alpha=.5)
ax.set_xlabel("Valore")
ax.set_ylabel("Frequenza")
ax.set_title("Predizioni")
ax.legend()
plt.show()
```

Vogliamo ora stimare entrambi $\mu$ e $\sigma$ a partire dai dati, applicando MCMC. La sequenza di sample sarà estratta da

$$
p(\mu,\sigma|X)\propto p(X|\mu,\sigma)p(\mu,\sigma)=\prod_{i=1}^np(x_i|\mu,\sigma)p(\mu,\sigma)
$$

Ne deriva che, facendo le ipotesi semplificative seguenti:
- la likelihood $p(x|\sigma,\mu)$ è una gaussiana
- la prior $p(\mu, \sigma)$ è una gaussiana bivariata, con medie $\mu_m, \mu_d=0$ e matrice di covarianza diagonale, con deviazioni standard $\sigma_m, \sigma_d$

abbiamo

\begin{equation}
p(\mu, \sigma|X)\propto \left(\prod_{i=1}^n\frac{1}{\sqrt{2\pi}\sigma}e^{-\dfrac{(x_i-\mu)^2}{2\sigma^2}}\right)\frac{1}{\sqrt{2\pi}\sigma_m}\propto \sigma^{-n}e^{-\dfrac{(\mu-\mu_m)^2}{2\sigma_m^2}}e^{-\dfrac{\sigma^2}{2\sigma_d^2}}\prod_{i=1}^ne^{-\dfrac{(x_i-\overline{\mu})^2}{2\sigma^2}}
\end{equation}

### Definizione della MC

Come distribuzione della probabilità di transizione $q([\mu',\sigma']|[\mu,\sigma])$ assumiamo ad esempio una distribuzione normale con covarianza nulla e varianza unitaria per le due variabili

\begin{equation}
q([\mu',\sigma']|[\mu,\sigma]) = \frac{1}{2\pi}e^{-\dfrac{(\sigma-\sigma')^2}{2}}e^{-\dfrac{(\mu-\mu')^2}{2}}
\end{equation}

Si noti che, ancora, $q([\mu',\sigma']|[\mu,\sigma])=q([\mu,\sigma]|[\mu',\sigma'])$, per cui possiamo applicare l'algoritmo di Metropolis

## Accettazione di $\mu',\sigma'$:
Applicando l'algoritmo di Metropolis il vettore $[\mu', \sigma']$ è accettato:

- se $$
\frac{p(\mu',\sigma'|X)}{p(\mu,\sigma|X)}>1
$$

- con probabilità $$
\frac{p(\mu',\sigma'|X)}{p(\mu,\sigma|X)}
$$
altrimenti

## Passaggio ai logaritmi
La condizione di accettazione può essere espressa applicando il logaritmo ai termini in gioco e passando a prior e likelihood, ottenendo che $\sigma'$ è accettato:

- se $$
\log p(X|\mu',\sigma')+\log p(\mu')+\log p(\sigma')-\log p(X|\mu,\sigma)-\log p(\mu)-\log p(\sigma)>0
$$

- con probabilità $$
\frac{p(\mu',\sigma'|X)}{p(\mu,\sigma|X)}
$$
altrimenti

```python
# Sampling dalla distribuzione di transizione

def transition(mu, sigma):
    sigma_new = np.random.normal(sigma,1,(1,))[0]
    if sigma_new>0:
        mu_new = np.random.normal(mu,1,(1,))[0]
        return mu_new, sigma_new
    else:
        return mu, sigma

# Proporzionale al logaritmo della probabilità a priori
def log_prior_d(sigma):
    return -(sigma/sigma_d)**2/2.0

def log_prior_m(mu):
    return -((mu-mu_m)/sigma_m)**2/2.0

# Proporzionale al logaritmo della likelihood dei dati
def log_likelihood(mu,sigma,samples):
    n = len(samples)
    return -n*np.log(sigma)-np.sum((samples-mu)**2)/(2*sigma**2)

# Test di accettazione del nuovo valore
def acc(mu_new, sigma_new, mu, sigma):
    first_term = log_likelihood(mu_new,sigma_new,samples)+log_prior(mu_new)+log_prior(sigma_new)
    second_term = log_likelihood(mu,sigma,samples)+log_prior(mu)+log_prior(sigma)
    test_val = first_term-second_term
    if test_val>0:
        return True
    else:
        accept=np.random.uniform(0,1)
        return (accept < (np.exp(test_val)))

# Algoritmo di Metropolis
def metropolis(mu_init,sigma_init,iterations):
    mu = mu_init
    sigma = sigma_init
    values = []
    accepted_idx = []
    rejected_idx = []
    i = 0
    for i in range(iterations):
        mu_new, sigma_new =  transition(mu, sigma)
        values.append([mu_new,sigma_new])
        if acc(mu_new, sigma_new, mu, sigma):
            sigma = sigma_new
            mu = mu_new
            accepted_idx.append(i)
        else:
            rejected_idx.append(i)
        i+=1
    return np.array(values), np.array(accepted_idx), np.array(rejected_idx)
```

```python
mu_init = 15
sigma_init = 3
iterations = 50000
mu_m = 5
sigma_m = 15
sigma_d = 15
values, accepted_idx, rejected_idx = metropolis(mu_init, sigma_init, iterations)
```

```python
fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(1,1,1)
rejected_to_plot = rejected_idx[np.where(rejected_idx>50)]
accepted_to_plot = accepted_idx[np.where(accepted_idx>50)]
ax.plot(values[rejected_to_plot,0], values[rejected_to_plot,1], 'rx', label='Rejected')
ax.plot(values[accepted_to_plot,0], values[accepted_to_plot,1], 'b', label="Path")
ax.set_xlabel("$\mu$")
ax.set_ylabel("$\sigma$")
ax.legend()
ax.set_title(r"MCMC sampling for $\sigma$ and $\mu$ with Metropolis algorithm")
```

```python
fig = plt.figure(figsize=(12,10))
ax = fig.add_subplot(1,1,1)
ax.hist( values[accepted_to_plot,0],bins=15 ,)
ax.set_xlabel("Valore")
ax.set_ylabel("Frequenza")
ax.set_title("Distribuzione di sample relativi a $\mu$")
plt.show()
```

```python
values[accepted_to_plot,0].mean()
```

    10.032302727162886

```python
fig = plt.figure(figsize=(12,10))
ax = fig.add_subplot(1,1,1)
ax.hist( values[accepted_to_plot,1],bins=15 ,)
ax.set_xlabel("Valore")
ax.set_ylabel("Frequenza")
ax.set_title("Distribuzione di sample relativi a $\sigma$")
plt.show()
```

```python
values[accepted_to_plot,1].mean()
```

    2.9554933531754166

```python

```


# Simple iPython Calculations

This notebook contains examples of simple calculations so there is a record of "how to get things done".

Start with common imports.

```python
%matplotlib inline
import numpy as np
import sympy as sp
import matplotlib.pyplot as plt
# To get equations the look like, well, equations, use the following two lines.
from sympy.interactive import printing
printing.init_printing(use_latex=True)
```

Take the derivative of a function.

```python
[x, y] = sp.symbols(['x', 'y'])
```

```python
sp.diff(x * y * sp.sin(x), x)
```

There are a few options on doing plots. **plot.ly** is a commercial product that has received good reviews. This is an older school way: create a function, a linear space of numbers, and plot it.

```python
def f(x):
    """A polynomial."""
    return x**5 - 2 * x**4 + 3 * x**2 - 5 * x + 1

z = np.linspace(-2, 2)
plt.plot(z, f(z));
```

It works. To share:
> git status   - what is going on, always RUN
> git commit file1 file2 -m "Made it all better."
> git push     - to the github repo you go!
> git pull     - get the stuff other folks worked on.

To MESS around at little cost
> git checkout -b mess_around
do your stuff
> git commit -a -m "Good work."

If GREAT:
> git checkout master
> git merge mess_around

If NOT:
> git checkout master
and ignore that branch.


# Continuous Signals

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Standard Signals

Certain [signals](https://en.wikipedia.org/wiki/Signal_%28electrical_engineering%29) play an important role in the theory and practical application of [signal processing](https://en.wikipedia.org/wiki/Signal_processing). They emerge from the theory of signals and systems, are used to characterize the properties of linear time-invariant (LTI) systems or frequently occur in practical applications. These standard signals are introduced and illustrated in the following. The treatise is limited to one-dimensional deterministic time- and amplitude-continuous signals.

### Complex Exponential Signal

The complex exponential signal over time $t$ is defined by the [complex exponential function](https://en.wikipedia.org/wiki/Exponential_function#Complex_plane)

\begin{equation}
x(t) = e^{s t}
\end{equation}

where $s = \sigma + j \omega$ denotes the complex frequency with $\sigma, \omega \in \mathbb{R}$ and $j$ the imaginary unit $(j^2=-1)$. The signal is often used as a generalized representation of harmonic signals. Using [Euler's formula](https://en.wikipedia.org/wiki/Euler's_formula) above definition can be reformulated as

\begin{equation}
x(t) = e^{(\sigma + j \omega) t} = e^{\sigma t} \cos(\omega t) + j e^{\sigma t} \sin(\omega t)
\end{equation}

The real/imaginary part of the exponential signal is given by a weighted cosine/sine with angular frequency $\omega = 2 \pi f$. For $t>0$, the time-dependent weight $e^{\sigma t}$ is

* exponentially decaying over time for $\sigma < 0$,
* constantly one for $\sigma = 0$,
* exponentially growing over time for $\sigma > 0$,

and vice-versa for $t<0$. The complex exponential signal is used to model harmonic signals with constant or exponentially decreasing/increasing amplitude.

**Example**

The following example illustrates the complex exponential signal and its parameters. The Python module [SymPy](http://docs.sympy.org/latest/index.html) is used for this purpose. It provides functionality for symbolic variables and functions, as well as their calculus. The required symbolic variables need to be defined explicitly before usage. In the example $t$, $\omega$ and $\sigma$ are defined as real-valued symbolic variables, followed by the definition of the exponential signal.

```python
import sympy as sym
%matplotlib inline
sym.init_printing()

t, sigma, omega = sym.symbols('t sigma omega', real=True)
s = sigma + 1j*omega
x = sym.exp(s*t)
x
```

Now specific values for the complex frequency $s = \sigma + j \omega$ are considered for illustration. For this purpose a new signal is defined by substituting both $\sigma$ and $\omega$ with specific values. The real and imaginary part of the signal is plotted for illustration.

```python
y = x.subs({omega: 10, sigma: -.1})

sym.plot(sym.re(y), (t, 0, 2*sym.pi), ylabel=r'Re{$e^{st}$}')
sym.plot(sym.im(y), (t, 0, 2*sym.pi), ylabel=r'Im{$e^{st}$}');
```

**Exercise**

* Try other values for `omega` and `sigma` to create signals with increasing/constant/decreasing amplitudes and different angular frequencies.

### Dirac Impulse

The Dirac impulse is one of the most important signals in the theory of signals and systems. It is used for the characterization of LTI systems and the modeling of impulse-like signals. The Dirac impulse is defined by way of the [Dirac delta function](https://en.wikipedia.org/wiki/Dirac_delta_function) which is not a function in the conventional sense. It is a generalized function or *distribution*. The Dirac impulse is denoted as $\delta(t)$. The Dirac delta function is defined by its effect on other functions. A rigorous treatment is beyond the scope of this course material. Please refer to the literature for a detailed discussion of the mathematical foundations of the Dirac delta distribution. Fortunately it is suitable to consider only certain properties for its application in signal processing. The most relevant ones are

1. **Sifting property**
    \begin{equation}
    \int_{-\infty}^{\infty} \delta(t) \cdot x(t) = x(0)
    \end{equation}
    where $x(t)$ needs to be differentiable at $t=0$. The sifting property implies $\int_{-\infty}^{\infty} \delta(t) = 1$.

2. **Multiplication**
    \begin{equation}
    x(t) \cdot \delta(t) = x(0) \cdot \delta(t)
    \end{equation}
    where $x(t)$ needs to be differentiable at $t=0$.

3. **Linearity**
    \begin{equation}
    a \cdot \delta(t) + b \cdot \delta(t) = (a+b) \cdot \delta(t)
    \end{equation}

4. **Scaling**
    \begin{equation}
    \delta(a t) = \frac{1}{|a|} \delta(t)
    \end{equation}
    where $a \in \mathbb{R} \setminus 0$. This implies that the Dirac impulse is a function with even symmetry.

5. **Derivation**
    \begin{equation}
    \int_{-\infty}^{\infty} \frac{d \delta(t)}{dt} \cdot x(t) \; dt = - \frac{d x(t)}{dt} \bigg\vert_{t = 0}
    \end{equation}

6. **Convolution**

    Generalization of the sifting property yields
    \begin{equation}
    \int_{-\infty}^{\infty} \delta(\tau) \cdot x(t - \tau)  \, d\tau = x(t)
    \end{equation}

    This operation is known as [convolution](https://en.wikipedia.org/wiki/Convolution) and will be introduced later in more detail. It may be concluded already here that the Dirac delta function constitutes the neutral element of the convolution.

It is important to note that the product $\delta(t) \cdot \delta(t)$ of two Dirac impulses is not defined.

**Example**

This example illustrates some of the basic properties of the Dirac impulse. Let's first define a Dirac impulse by way of the Dirac delta function

```python
delta = sym.DiracDelta(t)
delta
```

Now let's check the sifting property by defining an arbitrary signal (function) $f(t)$ and integrating over its product with the Delta impulse

```python
f = sym.Function('f')(t)
sym.integrate(delta*f, (t, -sym.oo, sym.oo))
```

**Exercise**

* Derive the sifting property for a shifted Dirac impulse $\delta(t-\tau)$ and check your results by modifying above example.

### Heaviside Signal

The Heaviside signal is defined by the [Heaviside step function](https://en.wikipedia.org/wiki/Heaviside_step_function)

\begin{equation}
\epsilon(t) = \begin{cases} 0 & t<0 \\  \frac{1}{2} & t=0 \\ 1 & t > 0 \end{cases}
\end{equation}

Note that alternative definitions exist, which differ with respect to the value of $\epsilon(t)$ at $t=0$. The Heaviside signal may be used to represent a signal that switches on at a specified time and stays switched on indefinitely. The Heaviside signal can be related to the Dirac impulse by

\begin{equation}
\epsilon(t) = \int_{-\infty}^{t} \delta(\tau) \; d\tau
\end{equation}

**Example**

In the following, a Heaviside signal $\epsilon(t)$ is defined and plotted. Note that `Sympy` denotes the Heaviside function by $\theta(t)$.

```python
step = sym.Heaviside(t)
step
```

```python
sym.plot(step, (t, -2, 2), ylim=[-0.2, 1.2], ylabel=r'$\epsilon(t)$');
```

Let's construct a harmonic signal $\cos(\omega t)$ with $\omega=2$ which is switched on at $t=0$. Considering the definition of the Heaviside function, the desired signal is given as

\begin{equation}
x(t) = \cos(\omega t) \cdot \epsilon(t)
\end{equation}

```python
x = sym.cos(omega*t) * sym.Heaviside(t)
sym.plot(x.subs(omega,2), (t, -2, 10), ylim=[-1.2, 1.2], ylabel=r'$x(t)$');
```

### Rectangular Signal

The rectangular signal is defined by the [rectangular function](https://en.wikipedia.org/wiki/Rectangular_function)

\begin{equation}
\text{rect}(t) = \begin{cases} 1 & |t| < \frac{1}{2} \\ \frac{1}{2} & |t| = \frac{1}{2} \\ 0 & |t| > \frac{1}{2} \end{cases}
\end{equation}

Its time limits and amplitude are chosen such that the area under the function is $1$.

Note that alternative definitions exist, which differ with respect to the value of $\text{rect}(t)$ at $t = \pm \frac{1}{2}$. The rectangular signal is used to represent a signal which has finite duration, respectively is switched on for a limited period of time. The rectangular signal can be related to the Heaviside signal by

\begin{equation}
\text{rect}(t) = \epsilon \left(t + \frac{1}{2} \right) - \epsilon \left(t - \frac{1}{2} \right)
\end{equation}

**Example**

The Heaviside function is used to define a rectangular function in `Sympy`. This function is then used as rectangular signal.

```python
class rect(sym.Function):

    @classmethod
    def eval(cls, arg):
        return sym.Heaviside(arg + sym.S.Half) - sym.Heaviside(arg - sym.S.Half)
```

```python
sym.plot(rect(t), (t, -1, 1), ylim=[-0.2, 1.2], ylabel=r'rect$(t)$');
```

**Exercise**

* Use $\text{rect}(t)$ to construct a harmonic signal $\cos(\omega t)$ with $\omega=2$ which is switched on at $t=-\frac{1}{2}$ and switched off at $t=+\frac{1}{2}$.

### Sign Signal

The sign signal is defined by the [sign/signum function](https://en.wikipedia.org/wiki/Sign_function) which evaluates the sign of its argument

\begin{equation}
\text{sgn}(t) = \begin{cases} 1 & t>0 \\ 0 & t=0 \\ -1 & t < 0  \end{cases}
\end{equation}

The sign signal is useful to represent the absolute value of a real-valued signal $x(t) \in \mathbb{R}$ by a multiplication

\begin{equation}
|x(t)| = x(t) \cdot \text{sgn}(x(t))
\end{equation}

It is related to the Heaviside signal by

\begin{equation}
\text{sgn}(t) = 2 \cdot \epsilon(t) - 1
\end{equation}

when following above definition with $\epsilon(0)=\frac{1}{2}$.

**Example**

The following example illustrates the sign signal $\text{sgn}(t)$. Note that the sign function is represented as $\text{sign}(t)$ in `Sympy`.

```python
sgn = sym.sign(t)
sgn
```

```python
sym.plot(sgn, (t, -2, 2), ylim=[-1.2, 1.2], ylabel=r'sgn$(t)$');
```

**Exercise**

* Check the values of $\text{sgn}(t)$ for $t \to 0^-$, $t = 0$ and $t \to 0^+$ as implemented in `SymPy`. Do they conform to above definition?

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


# Some Examples to Compare AM with AS

This notebook compares the methods of Active Subspaces and Active Manifolds for the visualization and approximation of scalar-valued functions which depend on some number of parameters.

In particular, we examine the "toy" functions

$$\begin{align}
    f(\mathbf{x}) &= |\mathbf{x}|^2, \\
    f(x,y) &= \sin(8x) + \sin(8y), \\
    f(x,y) &= \arctan{(y/x)},
\end{align}$$

as well as (real and fabricated) data for Magnetohydrodynamic (MHD) flow.  In particular, we consider the following model problems whose explanation and analysis follow [Glaws et al, 2016](https://arxiv.org/abs/1609.01255) and [Bridges et al, 2019](https://arxiv.org/abs/1904.133860).  See also [Paul Constantine's MHD notebook](https://github.com/paulcon/as-data-sets/blob/master/MHD/MHD.ipynb) for further explanation related to AS.

##### The Hartmann Problem

The Hartmann problem models laminar flow between parallel plates and admits closed-form analytical solutions in terms of its input parameters. Two quantities of interest will be examined: average flow velocity, $u_{avg}$, and induced magnetic field, $B_{ind}$, whose solutions are

$$
u_{avg} = -\frac{\partial p_0}{\partial x}\frac{\eta}{B_0^2}\left(1 - \frac{B_0l}{\sqrt{\eta\mu}}\coth\left(\frac{B_0l}{\sqrt{\eta\mu}}\right)\right), \\
B_{ind} = \frac{\partial p_0}{\partial x}\frac{l\mu_0}{2B_0}\left(1 - 2\frac{\sqrt{\eta\mu}}{B_0l}\tanh\left(\frac{B_0l}{2\sqrt{\eta\mu}}\right)\right).
$$

There are 5 input parameters to this problem: ($\mathbf x = \left[\begin{matrix}\log(\mu) & \log(\rho) & \log(\frac{\partial p_0}{\partial x}) & \log(\eta) & \log(B_0)\end{matrix}\right]^T$, since $l$ is fixed and $\mu_0$ is a physical constant). Samples are drawn randomly with uniform density, with ranges summarized below.

Variable | $\ $ Notation $\ $ | Range $\ \ \ \ \ $
:--------|:--------:|:------------
Fluid Viscosity | $\log(\mu)$ | [log(.05), log(.2)]
Fluid Density | $\log(\rho)$ | [log(1), log(5)]
Applied Pressure Gradient | $\log(\frac{\partial p_0}{\partial x})$ | [log(.5), log(3)]
Resistivity | $\log(\eta)$ | [log(.5), log(3)]
Applied Magnetic Field | $\log(B_0)$ | [log(.1), log(1)]

##### MHD Generator Problem

Following this, we consider a model that describes 3D duct flow in an idealized MHD generator. The variable input parameters are the same as those in the Hartmann problem (with uniform densities), but they have different ranges which are summarized below.

Variable | $\ $ Notation $\ $ | Range $\ \ \ \ \ $
:--------|:--------:|:------------
Fluid Viscosity | $\log(\mu)$ | [log(.001), log(.01)]
Fluid Density | $\log(\rho)$ | [log(.1), log(10)]
Applied Pressure Gradient | $\log(\frac{\partial p_0}{\partial x})$ | [log(.1), log(.5)]
Resistivity | $\log(\eta)$ | [log(.1), log(10)]
Applied Magnetic Field | $\log(B_0)$ | [log(.1), log(1)]

The quantities of interest are again $u_{avg}$ and $B_{ind}$. Note that each run of the model takes about 5 CPU-hours.

```python
# Path stuff
import sys
sys.path.insert(0, "/Users/anthonygruber/Desktop/Projects/AMv2/src")

# Some standard imports
import numpy as np
import pandas as pd
import scipy as sy
from sklearn import preprocessing

# Handwritten code
import basicFuncs as bf
import numethod as nm
```

## Toy functions

First, consider how well AM performs given an excess of data. This example predicts $f(x) = |x|^2$ given $30^3$ uniformly distributed points in the unit hypercube $[-1,1]^3$.  Compared to a 2-D Active Subspace, the errors are nearly 100 times smaller.

```python
## f(x) = |x|^2 with 3 dims, a uniform grid of 30^3 points, and 1/3 test fraction.
## Took 247 seconds for the paper, now takes ~5 due to the new approximation method.
mesh, fSamples, paths, realgrads = nm.build_uniform_data( 3, 30, bf.Squaresum, bf.gradSquaresum)
nm.manifoldEx( mesh, fSamples, realgrads, 0.05, 0.33, nm.get_random_init_pt(3))
nm.subspEx( mesh, fSamples, realgrads, 2, 0, .33, False, True)
```

Repeating this experiment with much less data, consider 500 randomly generated samples in $[-1,1]^5$.  AM still outperforms even a 4-D Active Subspace, although the errors are stil relatively large.

```python
# Experiment on f(x) = |x|^2 with 5 dimensions.
# Errors are not great, but they beat even a 4-D Active Subspace.
mesh, fSamples, paths, realgrads = nm.build_random_data( 5, 500, bf.Squaresum, bf.gradSquaresum)
nm.manifoldEx( mesh, fSamples, realgrads, 0.05, 50, nm.get_random_init_pt(5), 0, True)
nm.subspEx(mesh, fSamples, realgrads, 4, 0, .2, True)
```

We can also consider functions with many critical points, e.g. $f(x,y) = \sin(8x) + \sin(8y)$.  Given enough data, AM still performs reasonably well on this while AS struggles (even when not reducing the dimension at all).

```python
## Remember that function f(x,y) = sin(8x) + sin(8y) with a ton of critical points?
## Absolutely no problem now.
mesh, fSamples, paths, realgrads = nm.build_random_data( 2, 1000, bf.uhoh, bf.graduhoh)
nm.manifoldEx( mesh, fSamples, realgrads, 0.05, 0.2, nm.get_random_init_pt(2), 0)
nm.subspEx( mesh, fSamples, realgrads, 2, 0, .2, True, True)
```

Functions with singular gradient are even okay using AM, since everything is done locally.  Consider the argument function $f(x,y) = \arctan(y/x), which is well-approximated despite its singularity at the origin.  Once again, such function is not suitable for AS approximation.

On the other hand, the plot clearly indicates that the AM spirals several times, which is not really optimal -- something to be fixed eventually.

```python
mesh, fSamples, paths, realgrads = nm.build_random_data( 2, 100, bf.f4, bf.gradf4)
nm.manifoldEx( mesh, fSamples, realgrads, 0.1, 0.2, nm.get_random_init_pt(2), 0)
nm.subspEx( mesh, fSamples, realgrads, 2, 0, .2, True, False)
```

## Hartmann Simulation

Here we compare a 2-dimensional AS with the (necessarily) 1-dimensional AM on the Hartmann problem from before.

With AM we see large improvements in both cases, indicating that the behavior of these quantities is inherently nonlinear.

```python
## Can also do all the MHD stuff from the paper faster and more accurately.
## Takes maybe ~5s each (or less).
meshy = nm.make_mesh( 5, 5)
tree = sy.spatial.KDTree( meshy)
u, pathsu, rgu = bf.get_u_avg_data( meshy)
b, pathsb, rgb = bf.get_b_ind_data( meshy)

## Hartmann for u
nm.manifoldEx( meshy, u, rgu, 0.05, 0.2, nm.get_random_init_pt(5), 0)
nm.subspEx( meshy, u, rgu, 2, 0, 0.2, True, True)
# am,fvals = nm.build_AM_from_data( nm.get_random_init_pt(5), meshy, u, pathsu, rgu, 0.01)
# nm.splinePlot( am, fvals, 'u_{avg}', '{Hu}')

print(" ")

## Hartmann for B
nm.manifoldEx( meshy, b, rgb, 0.05, 0.2, nm.get_random_init_pt(5), 0)
nm.subspEx( meshy, b, rgb, 2, 0, .2, True, True)
# am,fvals = nm.build_AM_from_data( nm.get_random_init_pt(5), meshy, b, pathsb, rgb, 0.01)
# nm.splinePlot( am,fvals,'B_{ind}','{HB}')
```

## MHD Generator Problem

Here we consider the idealized MHD generator model from before.  The training data are 483 input samples which are randomly distributed across the 5-dimensional parameter space.

In this case, AM yields a small improvement on predicting $u_{avg}$ and a larger improvement on predicting $B_{ind}$.  This is not surprising given the eigenvalue plots, which do not decrease as sharply for $B_{ind}$.

```python
## Setup for real data MHD
# Load data from data files
data = pd.read_csv('/Users/anthonygruber/Desktop/Projects/AMv2/data/MHD/MHD_Generator_Data_Uavg.txt').values
Xu = data[:,1:6]; u = data[:,6]; du = data[:,7:]

data = pd.read_csv('/Users/anthonygruber/Desktop/Projects/AMv2/data/MHD/MHD_Generator_Data_Bind.txt').values
XB = data[:,1:6]; B = data[:,6]; dB = data[:,7:]

# New upper/lower bounds
lb = np.log(np.array([.001, .1, .1, .1, .1]))
ub = np.log(np.array([.01, 10, .5, 10, 1]))

# Scale gradients according to the chain rule, get normalized inputs
realdu = .5*(ub - lb)*Xu*du; XXu = 2*(np.log(Xu) - lb)/(ub - lb) - 1
realdB = .5*(ub - lb)*XB*dB; XXB = 2*(np.log(XB) - lb)/(ub - lb) - 1

## MHD for u
print("AM relative errors (u):")
nm.manifoldEx( XXu, u, realdu, 0.05, 0.2, nm.get_random_init_pt(5), 0)
nm.subspEx( XXu, u, realdu, 2, 0, .2, True, False)

print(" ")

## MHD for B
print("AM relative errors (B):")
nm.manifoldEx( XXB, B, realdB, 0.05, 97, nm.get_random_init_pt(5), 0)
nm.subspEx( XXB, B, realdB, 2, 0, .2, True, False)
```

```python

```


---
title: "Integration test for your Gibbs sampler"
excerpt: "Making your MCMC code ready for production."
---

This summer at [Civis Analytics](https://www.civisanalytics.com/), my project was to derive and implement a Gibbs sampler for one of the company's core algorithms. While I've written several Gibbs samplers, they are one-off projects that pay little attention to robustness and maintainability. Now that my code will be read by other engineers and used to make business decisions worth many (undisclosed number) of dollars, I think a lot more about how to productionize MCMC code.

Traditionally, the way a Gibbs sampler is taught in [lecture notes](https://people.duke.edu/~ccc14/sta-663/MCMC.html#gibbs-sampler) or [tutorials](https://theclevermachine.wordpress.com/2012/11/05/mcmc-the-gibbs-sampler/) is a giant loop iterating through each parameter's full conditional. It's no frill and fast, relying on `numpy` operations. It can be integration tested, but such a test is slow and non-deterministic, making it unsuitable for production.

In contrast, [Grosse and Duvenaud (2014)](https://arxiv.org/abs/1412.5218) proposes a modular way to write a Gibbs sampler that facilitates a unit testing approach. Written this way, the Gibbs sampler is easy to test, but slow to run due to all the class wrappers.

In this first post, I show how to write a fast, no-frill Gibbs sampler that can be integration tested. [In the second post]({% post_url 2017-08-16-productionizing-Gibbs-sampler-unit-test %}), I write a modular Gibbs sampler that can be unit tested, following [Grosse and Duvenaud (2014)](https://arxiv.org/abs/1412.5218). Then we can use the result from the modular, unit-tested Gibbs sampler to check the fast, no-frill Gibbs sampler. Once we're assured that the fast, no-frill Gibbs is correct, we can use it in production.

## Fast, no-frill Gibbs sampler

As an example throughout this series, I'll use the Gibbs sampler for the univariate normal model. It's a simple and common model, which you can find in other tutorials and textbooks to compare with my approach.

Below is the model's setup and its full conditionals. You can skip the math -- I write it here in case you want to check that it matches my code.

**Likelihood**

$$Data = Y_1, \dots, Y_n \sim i.i.d. N(\theta, \sigma^2)$$

**Prior**

$$
\begin{align}
p(\theta) &\sim N(\mu_0, \tau^2_0) \\
p(\sigma^2) &\sim Inverse-Gamma(\nu_0 / 2, \nu_0 \sigma^2_0 / 2)
\end{align}
$$

**Full conditional**

$$
\begin{align}
p(\theta | \tilde \sigma^2, Data) &= N(\mu_n, \tau^2_n) \\
p( \sigma^2 | \theta, Data) &= Inverse-Gamma(\frac{\nu_n}{2}, \frac{\nu_n \sigma_n^2(\theta)}{2})
\end{align}
$$

where

$$
\begin{align}
\tau_n^2 = \frac{1}{\frac{1}{\tau_0^2} + \frac{n}{\sigma^2}} \qquad &\text{and} \qquad \mu_n = \tau_n^2 \times \left( \frac{\mu_0}{\tau_0^2} + \frac{n\bar y}{\sigma^2} \right) \\
\nu_n = \nu_0 + n \qquad &\text{and} \qquad \sigma^2_n(\theta) = \frac{1}{\nu_n} \left[ \nu_0\sigma_0^2 + \sum (y_i - \theta)^2\right]
\end{align}
$$

We then draw \\( \theta \\) and \\( \sigma^2 \\) iteratively:

$$
\begin{align}
\theta &\sim p(\theta_s | \sigma_{s-1}^2, data) \\
\sigma^2 &\sim p(\sigma_{s}^2 | \theta_s, data)
\end{align}
$$

**We implement this iterative updating in a big loop as follows**

```python
import numpy as np
import scipy.stats
import pandas as pd

def gibbs_simple(S, y, prior, rng=None):
    """
    A no-frill Gibbs sampler

    Parameters
    ----------
    S : int, the number of samples
    y : data vector
    prior : dict, prior parameters
    rng : int, random seed
    """
    n = len(y)
    ybar = y.mean()

    # Initialize storage
    theta_samples = np.empty(S)
    sigma2_samples = np.empty(S)

    # Starting value as the sample variance and mean
    sigma2_samples[0] = y.var()
    theta_samples[0] = ybar

    # Big loop
    for s in range(1, S):
        # Update theta
        tau2_n = 1 / (1 / prior['tau2_0'] + n / sigma2_samples[s - 1])
        mu_n = tau2_n * (prior['mu_0'] / prior['tau2_0'] + n * ybar / sigma2_samples[s - 1])
        theta_samples[s] = scipy.stats.norm(mu_n, np.sqrt(tau2_n)).rvs(random_state=rng)

        # Update sigma2
        nu_n = prior['nu_0'] + n
        nu_sigma2_n = prior['nu_0'] * prior['sigma2_0'] + sum((y - theta_samples[s]) ** 2)
        sigma2_samples[s] = scipy.stats.invgamma(nu_n / 2, scale=nu_sigma2_n / 2).rvs(random_state=rng)

    return {'theta': theta_samples, 'sigma2': sigma2_samples}
```

## Integration test

This no-frill Gibbs sampler is fast, taking full advantage of `numpy` operations. We can test its correctness by generating data from known parameters, then check whether the Gibbs sampler produces posterior estimates that are "close enough" to the known parameter values. From a programming perspective, this can be considered an [integration test](http://softwaretestingfundamentals.com/integration-testing/).

```python
# Generate the data from known parameters
true_theta = 2
true_sigma2 = 3.5
y = np.random.normal(true_theta, np.sqrt(true_sigma2), size = 1000)
prior = {'mu_0': 2, 'tau2_0': 10000, 'nu_0': 1, 'sigma2_0': 3.5}

# Run the Gibbs sampler
samples_gibbs_simple = gibbs_simple(100, y, prior)
```

We can inspect the result visually, checking that the posterior distributions (the black density plot) covers the true parameter values (the red line).

```python
from plotnine import *
from plotnine import options

options.figure_size = (3, 3)
ggplot(data=pd.DataFrame(samples_gibbs_simple)) + \
    geom_density(aes(x='theta')) + \
    geom_vline(aes(xintercept=true_theta), color='red')
```

```python
options.figure_size = (3, 3)
ggplot(data=pd.DataFrame(samples_gibbs_simple)) + \
    geom_density(aes(x='sigma2')) + \
    geom_vline(aes(xintercept=true_sigma2), color='red')
```

We should also encode this visual check into a formal test, which can be put in an automated testing framework like `pytest` or `nose`:

```python
assert np.allclose(res_gibbs_simple['theta'].mean(), true_theta, rtol=0.99)
assert np.allclose(res_gibbs_simple['sigma2'].mean(), true_sigma2, rtol=0.99)
```

The integration test "works," but suffers from several drawbacks due to its reliance on a "close enough" behavior. In [Part 2](), I will discuss how to write unit tests that avoid these problems.


# Music Machine Learning - Probability basics

### Author: Philippe Esling (esling@ircam.fr)

In this course we will cover
1. An explanation on [simple probability](#probability) concepts
2. A review of the axioms and definitions of probability [theory](#theory)
3. An discussion on [conditional probability](#conditional) and independence

## Understanding probability

The field of probability aims to model random or uncertain events. We usually refer to random events when we cannot predict the outcome of a phenomenon with absolute certainty. Maybe the simplest and most iconic case of randomness that everyone has encoutered is to _flip a coin_. In that case, we are uncertain of the result, and might see randomness as a way of expressing this uncertainty.

Hence, a random variable $X$ denotes a quantity that is uncertain, such as the result of an experiment (flipping a coin) or the measurement of an uncertain property (the weather of tomorrow). However, if we can observe a sufficiently large set of _samples_ $\{\mathbf{x}_{i}\}_{i=1}$, it might take different values on each occasion, but some values may occur more often than others. Hence, we can start reasoning on these random events from a different perspective. If the coin is fair, and do not have access to any external information, we expect the _probability_ of the result to be heads 1 times out of 2, or equivalently 50\% of the times. Here, the use of percentage informs us on one of the interpretation of probability, as the _relative frequency_ of events. By repeatedly flipping the coin and observing the results, we can count the occurrence of each outcome.

To understand these concepts graphically, we will rely on `numpy` and `matplotlib`

```python
import random
import matplotlib.pyplot as plt
import numpy as np
```

## Definitions
Let's start with some basic definitions and principles.
- An ***experiment*** or ***trial*** is an action with an uncertain outcome, such as tossing a coin.
- A ***sample space*** is the set of all possible outcomes of an experiment. In a coin toss, there's a set of two possible oucomes (*heads* and *tails*).
- A ***sample point*** is a single possible outcome - for example, *heads*)
- An ***event*** is a specific outome of single instance of an experiment - for example, tossing a coin and getting *tails*.
- ***Probability*** is a value between 0 and 1 that indicates the likelihood of a particular event, with 0 meaning that the event is impossible, and 1 meaning that the event is inevitable. In general terms, it's calculated like this:

$$
\begin{equation}
\text{probability of an event} = \frac{\text{Number of sample points that produce the event}}{\text{Total number of sample points in the sample space}}
\end{equation}
$$

For example, the probability of getting *heads* when tossing as coin is 1/2 - there is only one side of the coin that is designated *heads*. and there are two possible outcomes in the sample space (*heads* and *tails*). So the probability of getting *heads* in a single coin toss is 0.5 (or 50% when expressed as a percentage).

```python
vals = ['heads', 'tails']
trials = 5
for t in range(trials):
    # Get a random 0 or 1
    toss = random.randint(0,1)
    # Print the result of our toss
    print('Coin is tossed on ' + vals[toss])
```

    Coin is tossed on tails
    Coin is tossed on tails
    Coin is tossed on tails
    Coin is tossed on tails
    Coin is tossed on tails

If we run the previous code several times, we will always get different (uncertain) results, showing the underlying randomness. However, if we start simulating (_sampling_) this _experiment_ a certain number of times, we can start to look at the _distribution_ of _events_

```python
plt.figure(figsize=(25,4))
# loop through 5,10,50,100,1000,10000 trials
for t_id, trials in enumerate([5,10,50,100,1000,10000]):
    # Keep track
    heads_or_tails = [0, 0]
    for t in range(trials):
        # Get a random 0 or 1
        toss = random.randint(0,1)
        # Increment the list element corresponding to the toss result
        heads_or_tails[toss] = heads_or_tails[toss] + 1
    # Show a pie chart of the results
    plt.subplot(1, 6, t_id+1)
    plt.pie(heads_or_tails, labels=['heads', 'tails'])
    plt.legend()
plt.show()
```

### A more complicated example

Let us look at another slightly more complicated example, where we have two dice, hoping to get a particular number. The dice throw itself is an *experiment*, as you will not know the outome until the dice has been thrown. In this case, the *sample space* of all possible outcomes is every combination of two dice - 36 *sample points*:

⚀+⚀	⚀+⚁	⚀+⚂	⚀+⚃	⚀+⚄	⚀+⚅

⚁+⚀	⚁+⚁	⚁+⚂	⚁+⚃	⚁+⚄	⚁+⚅

⚂+⚀	⚂+⚁	⚂+⚂	⚂+⚃	⚂+⚄	⚂+⚅

⚃+⚀	⚃+⚁	⚃+⚂	⚃+⚃	⚃+⚄	⚃+⚅

⚄+⚀	⚄+⚁	⚄+⚂	⚄+⚃	⚄+⚄	⚄+⚅

⚅+⚀	⚅+⚁	⚅+⚂	⚅+⚃	⚅+⚄	⚅+⚅

The *event* you want to happen is throwing a 7. There are 6 *sample points* that could produce this event:

⚀+⚅	⚁+⚄	⚂+⚃	⚃+⚂	⚄+⚁	⚅+⚀

The *probability* of throwing a 7 is therefore 6/36 which can be simplified to 1/6 or approximately 0.167 (16.7%).

## Probability theory

Probability theory defines the mathematical framework that allows to describe and analyze random phenomena, regardless of the interpretation of probability that we prefer. To rely on this framework, we starts by assuming few axioms of probability. We postulate a set of possible events $\Omega$ called the *sample space* (in our coin toss example, we would have $\Omega = \{Heads, Tails\}$. We can define a _probability measure_ $p(\omega)$ to an event $\omega\in\Omega$, which gives a value between 0 (very unlikely) and 1 (very probable) that shows how likely the event is. In our previous example, we could write that the probability of throwing a 7 is expressed as

$$
p(\omega = 7) = 0.167
$$

where our sample space is defined as $\Omega = \{2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12\}$

### Axioms of probability
The foundation of probability theory is defined in these axioms, on which we can develop tools and techniques to calculate probabilities of different events.

- For any event $\omega\in\Omega$, $0\leq p(\omega) \leq 1$.
- Probability of the sample space $\Omega$ is $p(\Omega)=1$.
- For a set of \textit{disjoint} events $\omega_1,\cdots,\omega_n$ we have $p(\bigcup_{i=1}^{n} \omega_i)=\sum_{i=1}^{n} \omega_{i}$

The first axiom is quite straightforward and states that probability is measured in the interval $[0,1]$. The second axiom states that the probability of the whole sample space is equal to one, as it contains all possible outcomes of our random experiment. For instance, in our coin toss, the outcome will always be either $Heads$ or $Tails$, and therefore will always be contained in $\Omega$. Finally, the third axioms state that for two disjoint events, the probability of one or the other to happen is the sum of their individual probabilities.

### Complement of an event
The *complement* of an event is the set of *sample points* that do ***not*** result in the event. If we follow our previous example, suppose that we want to know what is the probability of **not** throwing a 7. The *complement* of the event (throwing a 7) is all of the possible outcomes that **do not** result in throwing a 7. If we were to count all these events, we would find that there are 30 sample points in the complement, so the probability of the complement is 30/36 which is 5/6 or 0.833.

However, we can also note that the probability of an event and the probability of its complement ***always add up to 1***. This can be written as

$$
p(\bar{\omega}) = 1 - p(\omega)
$$

This fact can be useful in some cases. For example, suppose you throw two dice and want to know the probability of throwing more than 4. You *could* count all of the outcomes that would produce this result, but there are a lot of them. It might be easier to identify the ones that *do not* produce this result (in other words, the complement).

## Conditional Probability and dependence
Events can be:
- *Independent* (events that are not affected by other events)
- *Dependent* (events that are conditional on other events)
- *Mutually Exclusive* (events that can't occur together)

### Independent Events
Imagine you toss a coin. The sample space contains two possible outomes: heads or tails. The probability of getting *heads* is 1/2, and the probability of getting *tails* is also 1/2. Imagine we toss a coin and get *heads*. Now, we toss the coin again and get *heads* again. If we were to toss the coin a third time, what is the probability that we get *heads*?

Although you might be tempted to think that a *tail* is overdue, the fact is that each coin toss is an independent event. The outcome of the first coin toss does not affect the second coin toss (or the third, or any number of other coin tosses). For each independent coin toss, the probability of getting *heads* (or *tails*) remains 1/2, or 50%.

### Combining Independent Events
Now, we would like to know the probability of getting three *heads* in a row. Since the probability of a heads on each independent toss is 1/2, you might be tempted to think that the same probability applies to getting three *heads* in a row. But in reality, we need to understand that "getting three *heads* in a row" is a different random event, which is the _combination of three independent events_. To combine independent events like this, we need to multiply the probability of each event. As we know that $p(heads)$ = 1/2, we have

$p(heads, heads, heads)$ = 1/2 x 1/2 x 1/2

So the probability of tossing three *heads* in a row is 0.5 x 0.5 x 0.5, which is 0.125 (or 12.5%). You can simulate this behavior with the following code.

```python
import random
# Count the number of 3xHeads results
h3 = 0
# loop through 10000 trials
trials = 10000
for t in range(trials):
    trial = trial + 1
    # Flip three coins
    result = ['H' if random.randint(0,1) == 1 else 'T',
              'H' if random.randint(0,1) == 1 else 'T',
              'H' if random.randint(0,1) == 1 else 'T']
    # If it's three heads, add it to the count
    h3 = h3 + int(result == ['H','H','H'])
# What proportion of trials produced 3x heads
print ("Proportion of 3 heads : %.2f%%" % ((h3/trials)*100))
```

    Proportion of 3 heads : 13.07%

The output shows the percentage of times a trial resulted in three heads (which should be somewhere close to 12.5%).

#### Combined Event Probability Notation
When calculating the probability of combined events, we assign a letter such as **A** or **B** to each event, and we use the *intersection* (**∩**) symbol to indicate that we want the combined probability of multiple events. So we could assign the letters **A**, **B**, and **C** to each independent coin toss in our sequence of three tosses, and express the combined probability like this:

\begin{equation}p(A \cap B \cap C) = p(A) \times p(B) \times p(C) \end{equation}

Imagine you have created a new game that mixes coin-tossing and die-rolling. The objective of the game is to roll a die and get *6*, and toss a coin and get *heads*. On each turn of the game, a player rolls the die and tosses the coin. There are two independent events required to win: a die-roll of *6* (which we'll call event **A**), and a coin-toss of *heads* (which we'll call event **B**). Our formula for combined independent events is:

\begin{equation}p(A \cap B) = p(A) \times p(B) \end{equation}

The probablilty of rolling a *6* on a fair die is 1/6 or 0.167;  and the probability of tossing a coin and getting *heads* is 1/2 or 0.5:

\begin{equation}p(A \cap B) = 0.167 \times 0.5 = 0.083 \end{equation}

So on each turn, there's an 8.3% chance to win the game.

### Dependent events

In some cases, random events might be dependent. For instance, in the case of a dice, we can have the event $a=$"*getting an even number*" and $b=$"*rolling a 6*". If we *observe* event $a$, it will modify the probability of event $b$. This is given by the _conditional probability_ $p(b\vert a)$. We can also compute the full joint probability of both events as

$$
p(a,b)=p(a\vert b)p(b)
$$

Note that joint probabilities are symmetrical, which means that $p(a,b)=p(b,a)$ and also
$$
p(a,b) = p(a\vert b)p(b) = p(b\vert a)p(a) = p(b,a)
$$

In the case of our dice rolling, we have that $p(b=6 \vert a=even)=1/3$. Interestingly, this leads to
$$
p(b = 6, a = even) = p(b=6 \vert a=even)p(a=even) = 1/3 * 1/2 = 1/6
$$

which is our original probability of getting a 6. To summarize our discussion on dependence, you can note that _independent events_ will **not** be conditionned on each others, which means that $p(a \vert b) = p(a)$ and also $p(a, b) = p(a)p(b)$.


# Cálculo Diferencial e Integral

```python
from sympy import *
```

```python
x, y, z = symbols('x y z')
init_printing(use_unicode = True)
```

```python
# Derivadas:
```

```python
diff(5*x**3+cos(x))
```

```python
# ou:

diff(5*x**3+cos(x), x)
```

```python
diff(x**3) # primeira derivada
```

```python
diff(diff(x**3)) # segunda derivada
```

```python
# Integrada:
```

```python
integrate(15*x**2-sin(x))
```


The basic imports and the variables we'll be using:

```python
from __future__ import division
import sympy
from sympy import *
from sympy import Rational as frac
import simpletensors
from simpletensors import Vector, TensorProduct, SymmetricTensorProduct, Tensor
init_printing()

var('vartheta, varphi')
var('nu, m, delta, c, t')

# These are related scalar functions of time
var('r, v, Omega', cls=Function)
r = r(t)
v = v(t)
Omega = Omega(t)

# These get redefined momentarily, but have to exist first
var('nHat, lambdaHat, ellHat', cls=Function)
# And now we define them as vector functions of time
nHat = Vector('nHat', r'\hat{n}', [cos(Omega*t),sin(Omega*t),0,])(t)
lambdaHat = Vector('lambdaHat', r'\hat{\lambda}', [-sin(Omega*t),cos(Omega*t),0,])(t)
ellHat = Vector('ellHat', r'\hat{\ell}', [0,0,1,])(t)

# These are the spin functions -- first, the individual components as regular sympy.Function objects; then the vectors themselves
var('S_n, S_lambda, S_ell', cls=Function)
var('Sigma_n, Sigma_lambda, Sigma_ell', cls=Function)
SigmaVec = Vector('SigmaVec', r'\vec{\Sigma}', [Sigma_n(t), Sigma_lambda(t), Sigma_ell(t)])(t)
SVec = Vector('S', r'\vec{S}', [S_n(t), S_lambda(t), S_ell(t)])(t)
```

## Examples and tests

```python
nHat
```

```python
diff(nHat, t)
```

```python
diff(lambdaHat, t)
```

```python
diff(lambdaHat, t).components
```

```python
diff(lambdaHat, t).subs(t,0).components
```

```python
diff(lambdaHat, t, 2).components
```

```python
diff(lambdaHat, t, 2).subs(t,0).components
```

```python
diff(ellHat, t)
```

```python
diff(nHat, t, 2)
```

```python
diff(nHat,t, 3)
```

```python
diff(nHat,t, 4)
```

```python
diff(SigmaVec,t, 0)
```

```python
SigmaVec.fdiff()
```

```python
diff(SigmaVec,t, 1)
```

```python
diff(SigmaVec,t, 2)
```

```python
diff(SigmaVec,t, 2) | nHat
```

```python
T1 = TensorProduct(SigmaVec, SigmaVec, ellHat, coefficient=1)
T2 = TensorProduct(SigmaVec, nHat, lambdaHat, coefficient=1)
tmp = Tensor(T1,T2)
display(T1, T2, tmp)
```

```python
diff(tmp, t, 1)
```

```python
T1+T2
```

```python
T2*ellHat
```

```python
ellHat*T2
```

```python
T1.trace(0,1)
```

```python
T2*ellHat
```

```python
for k in range(1,4):
    display((T2*ellHat).trace(0,k))
```

```python
for k in range(1,4):
    display((T2*ellHat).trace(0,k).subs(t,0))
```

```python
T1.trace(0,1) * T2
```

Sympy can be a little tricky because it caches things, which means that the first implementation of this code silently changed tensors in place, without meaning to.  Let's just check that our variables haven't changed:

```python
display(T1, T2)
```

```python
T3 = SymmetricTensorProduct(SigmaVec, SigmaVec, ellHat, coefficient=1)
display(T3)
T3.trace(0,1)
```

```python
diff(T3, t, 1)
```

```python
T3.symmetric
```

    True

```python
T3*ellHat
```

```python
ellHat*T3
```

```python
T1+T3
```

```python
T1 = SymmetricTensorProduct(SigmaVec, SigmaVec, ellHat, nHat, coefficient=1)
display(T1)
display(T1.trace())
```

```python
T1*T2
```

```python
type(_)
```

    simpletensors.TensorProductFunction_73

```python
import simpletensors
isinstance(__, simpletensors.TensorProductFunction)
```

    True

```python
SymmetricTensorProduct(nHat, nHat, nHat).trace()
```

```python
diff(T1.trace(), t, 1)
```

```python
diff(T1.trace(), t, 2)
```

```python
diff(T1.trace(), t, 2).subs(t,0)
```

```python

```


```python
from sympy import *
from sympy.plotting import plot

x, A, B, T = symbols('x, A, B, T')
B = 2 * pi * (1/T)
kappa = (A*B**2*sin(B*x)) / ((1 + A**2 * B**2 * cos(B*x)**2)**(3/2))
kappa = abs(kappa)
kappa
```

$\displaystyle 4 \pi^{2} \left|{\frac{A \sin{\left(\frac{2 \pi x}{T} \right)}}{T^{2} \left(\frac{4 \pi^{2} A^{2} \cos^{2}{\left(\frac{2 \pi x}{T} \right)}}{T^{2}} + 1\right)^{1.5}}}\right|$

```python
subs = {}
subs['T'] = 15
subs['A'] = 6
subs['x'] = 15/4
kappa.evalf(subs=subs)
```

$\displaystyle 1.05275780278287$

```python
integral = Integral(kappa, (x, 0, 15))
integral
```

$\displaystyle \int\limits_{0}^{15} 4 \pi^{2} \left|{\frac{A \sin{\left(\frac{2 \pi x}{T} \right)}}{T^{2} \left(\frac{4 \pi^{2} A^{2} \cos^{2}{\left(\frac{2 \pi x}{T} \right)}}{T^{2}} + 1\right)^{1.5}}}\right|\, dx$

```python
subs = {}
subs['T'] = 15
subs['A'] = 6
integral.subs(subs).evalf()
```

$\displaystyle 3.7166$

```python
3.7166 / 15
```

    0.24777333333333335

```python
_ = plot(kappa.subs(subs), (x, 0, 15))
```

```python

```


# Bayesian optimization

*Selected Topics in Mathematical Optimization: 2017-2018*

**Michiel Stock** ([email](michiel.stock@ugent.be))

```python
import numpy as np
import sympy as sp
sp.init_printing()
from numpy import sin, exp, cos
from sklearn.preprocessing import normalize
from scipy.stats import norm
from scipy.integrate import odeint
import matplotlib.pyplot as plt
from sklearn.gaussian_process import GaussianProcessRegressor
%matplotlib inline
```

```python
np.random.seed(3)
```

```python
blue = '#264653'
green = '#2a9d8f'
yellow = '#e9c46a'
orange = '#f4a261'
red = '#e76f51'
black = '#50514F'
```

## Motivation

**Goal**:

Solve

$$
\mathbf{x}^* = \text{arg min}_\mathbf{x}\, f(\mathbf{x})
$$

when every evaluation of $f(\mathbf{x})$ is *expensive*.

**Example 1**:

*Dark art* of selecting the right hyperparameters for a machine learning model. Ditto for many optimization methods!

**Example 2**:

*Model calibration* to find the right parameters or structure of a mechanistic model.

**Example 3**:

Experimentation for improvement (drug design, product development, new cooking recipes...):

1. Gather data
2. Make model
3. Select new experiments for testing in the lab
4. Repeat

## Leading example

Damped spring mass system.

Model:

$$
m\ddot{y}(t) + \gamma \dot{y}(t) + k y(t) = 0\,,
$$

with:
- $m$: the mass (1 kg)
- $\gamma$: friction parameter (unknown)
- $k$: the spring constant (unknown)

Can be simulated (here exact solution known).

```python
def damped_oscillator(t_vals, g, k, m=1, sigma=0, x0=1):
    if g**2 > 4*k*m:
        l1 = (-g**2 + (g**2 - 4*k*m)**0.5) / (2 * m)
        l2 = (-g**2 - (g**2 - 4*k*m)**0.5) / (2 * m)
        c1, c2 = np.linalg.solve([[1, 1], [l1, l2]], [[x0], [0]]).flatten()
        y_vals = c1 * exp(t_vals * l1) + c2 * exp(t_vals * l2)
    elif g**2 == 4*k*m:
        l1 = - g / (2 * m)
        c1, c2 = np.linalg.solve([[1, 0], [l1, 1]], [[x0], [0]]).flatten()
        y_vals = c1 * exp(t_vals * l1) + c2 * t_vals * exp(t_vals * l1)
    else:
        alpha = - g / (2 * m)
        beta = (4 * m * k - g**2)**0.5 / (2 * m)
        c1, c2 = np.linalg.solve([[1, 0], [alpha, beta]], [[x0], [0]]).flatten()
        y_vals = exp(alpha * t_vals) * (c1 * cos(beta * t_vals) + c2 * sin(beta * t_vals))
    return y_vals + np.random.randn(len(t_vals)) * sigma
```

```python
fig, ax = plt.subplots()

t_vals = np.linspace(0, 5, num = 1000)

for g in np.logspace(-2, 2, num=4):
    for k in np.logspace(-2, 2, num=4):
        y_vals = damped_oscillator(t_vals, g=g, k=k)
        ax.plot(t_vals, y_vals)
ax.set_xlabel('$t$')
ax.set_ylabel('$y(t)$')
```

## Finding the best model parameters

We perform some experiments, and have 20 noisy measurements.

```python
# parameters
m=1  # known
kstar=7  # unknown
gstar=1  # unknown
```

```python
# noisy observations
t_meas = np.linspace(0, 5, num=20)
y_obs = damped_oscillator(t_meas, g=gstar, k=kstar, m=m, sigma=0.15)
```

```python
# true function
t_vals = np.linspace(0, 5, num = 1000)
y_vals = damped_oscillator(t_vals, g=gstar, k=kstar, m=m)
```

```python
fig, ax = plt.subplots()

ax.plot(t_vals, y_vals, c=blue, label='unknown parametrized function')
ax.scatter(t_meas, y_obs, c=red, label='noisy observations')
ax.set_xlabel('$t$')
ax.set_ylabel('$y(t)$')
ax.legend(loc=0)
```

For a given $\gamma$ and $k$, compute the mean squared error:

$$
MSE(\gamma, k) = \frac{1}{20} \sum_{i=1}^{20} (y(t_i) - \hat{y}(t_i, \gamma, k))^2
$$

Best parameters obtained by:

$$
\gamma^*, k^* = \text{arg min}_{\gamma, k} MSE(\gamma, k)
$$

```python
def get_mse(g, k):
    """
    Computes the mean squared error for a given gamma and k.
    """
    y_sim = damped_oscillator(t_meas, g=g, k=k)
    return np.log10(np.mean((y_sim - y_obs)**2))
```

Since gamma and kappa are positive values for which we do not know the scale, it makes sense to work with the logarithm of these paramaters.

```python
g_vals = np.logspace(-2, 2, num=100)
k_vals = np.logspace(-2, 2, num=100)

mse_pars = np.zeros((len(g_vals), len(k_vals)))
for i, g in enumerate(g_vals):
    for j, k in enumerate(k_vals):
        mse_pars[i, j] = get_mse(g=g, k=k)
```

Because the example is simple, we can search the complete space for the best parameters.

```python
G, K = np.meshgrid(g_vals, k_vals)

fig, ax = plt.subplots(figsize=(6, 6))
ax.contourf(K, G, mse_pars.T)
ax.set_xlabel('$k$')
ax.set_ylabel('$g$')
ax.scatter(kstar, gstar, c='y')
ax.loglog()
```

In general, it is *not* possible to sample the whole parameter space. If the model is really complex, even a single evalutation might be expensive! What if we only can test a limited number of parameter combinations?

## Grid search vs. random search

```python
param_grid = np.array([[g, k] for k in np.logspace(-1.5, 1.5, num=4) for g in np.logspace(-1.5, 1.5, num=4)])
param_random = 10**np.random.uniform(-2, 2, size=(16, 2))

fig, (ax0, ax1) = plt.subplots(ncols=2, figsize=(12, 5))

ax0.scatter(param_grid[:,0], param_grid[:,1], c=red, label='samples')
ax0.set_title('Grid')

ax1.scatter(param_random[:,0], param_random[:,1], c=red, label='samples')
ax1.set_title('Random')

for ax in (ax1, ax0):
    ax.scatter(kstar, gstar, c=yellow, label='optimal parameters')
    ax.loglog()
    ax.set_xlabel('$k$')
    ax.set_ylabel('$\gamma$')
    ax.set_ylim([1e-2, 1e2])
    ax.set_xlim([1e-2, 1e2])
    ax.legend(loc=0)
```

> **Don't use grid search!**

```python
# evaluate instances
simulated_parameters = param_random
#simulated_parameters = param_grid
mse_obs = np.array([get_mse(g=g, k=k) for g, k in simulated_parameters])
mse_obs
```

    array([-0.8542471 , -0.46689791, -0.33187908, -0.85548296, -0.94725825,
           -0.27965313, -0.8541473 , -0.18839734, -0.85414733, -0.29718357,
           -0.24694029, -0.8541491 , -0.95694289, -0.01830113, -0.22508889,
           -0.9303206 ])

## Surrogate modelling with Gaussian processes

Predict the performance of *new* parameter combinations using a surrogate model.

(From here on onwards, $\mathbf{x}\in\mathcal{X}$ is used to denote the parameters)

Gaussian process:
- learn a function of the form $f: \mathcal{X}\rightarrow \mathbb{R}$
- nonlinear model, uses a positive-definite covariance function $K:\mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$
- Bayesian method: prior on function and use Bayes' theorem to condition on observed data
- can be updated if more date becomes available (online learning)
- fully probabilistic, in theory no tuning!

The model return for a given instance $\mathcal{x}$:

- $\mu(\mathbf{x})$: expected value;
- $\sigma(\mathbf{x})$: standard deviation.

For a brief, but well-founded overview of Gaussian processes, consult the [chapter](http://www.inference.org.uk/mackay/gpB.pdf) in the book of MacKay.

```python
gaussian_process = GaussianProcessRegressor(alpha=1e-3)
# take logarithm of MSE and the parameters to obtain better scaling
gaussian_process.fit(np.log10(simulated_parameters), mse_obs)
```

    GaussianProcessRegressor(alpha=0.001, copy_X_train=True, kernel=None,
                 n_restarts_optimizer=0, normalize_y=False,
                 optimizer='fmin_l_bfgs_b', random_state=None)

```python
instance = np.log10([[3, 0.1]])

mu, sigma = gaussian_process.predict(instance, return_std=True)
print('Predicted MSE: {} ({})'.format(mu[0], sigma[0]))
```

    Predicted MSE: -0.8392304324239759 (0.34574779040748355)

Explore whole parameter space.

```python
mu_mse = np.zeros_like(mse_pars)
std_mse = np.zeros_like(mse_pars)

for i, g in enumerate(g_vals):
    for j, k in enumerate(k_vals):
        instance = np.log10([[g, k]])
        mu, sigma = gaussian_process.predict(instance, return_std=True)
        mu_mse[i, j] = mu[:]
        std_mse[i, j] = sigma[:]
```

```python
fig, (ax0, ax1) = plt.subplots(ncols=2, figsize=(12, 5))
ax0.contourf(K, G, mu_mse.T)
ax0.set_title('Estimated MSE')

ax1.contourf(K, G, std_mse.T)
ax1.set_title('Std MSE')

for ax in (ax0, ax1):
    ax.set_xlabel('$k$')
    ax.set_ylabel('$\gamma$')
    ax.loglog()
    ax.scatter(kstar, gstar, c=yellow)
    ax.scatter(simulated_parameters[:,0], simulated_parameters[:,1], c=red)
```

Slice for $k=1$.

```python
k_slice = np.column_stack((np.logspace(-2, 2, num=100), [1]*100))
# expected mse, std
mu, sigma = gaussian_process.predict(np.log10(k_slice), return_std=True)

fig, ax = plt.subplots()
ax.plot(k_slice[:,0], mu, c=green, label='expected value')
ax.plot(k_slice[:,0], mu-2*sigma, c=orange, ls='--', label='95% interval')
ax.plot(k_slice[:,0], mu+2*sigma, c=orange, ls='--')
ax.semilogx()
ax.set_xlabel('$\gamma$')
ax.set_ylabel('predicted MSE')
ax.legend(loc=0)
```

## Acquisition functions

How to choose points to simulate/test?

Acquisition function $a:\mathcal{X}\rightarrow\mathbb{R}^+$, determines how 'interesting' a point is to explore.

Choose

$$
\mathbf{x}_\text{next} = \text{arg max}_\mathbf{x} a(\mathbf{x})\,.
$$

Trade-off between *exploration* and *exploitation*.

**Probability of Improvement**

$$
a_\text{PI}(\mathbf{x}) = P(f(\mathbf{x})<f(\mathbf{x}_\text{best}) ) = \Phi(\gamma(\mathbf{x}))\,,
$$
with $\Phi(\cdot)$ the CDF of a standard normal distirbution and
$$
\gamma(\mathbf{x})= \frac{f(x_\text{best}) - \mu(\mathbf{x})}{\sigma(\mathbf{x})}\,.
$$

```python
def calculate_gamma(mu, sigma, fbest):
    return (fbest - mu) / sigma

def probability_improvement(mu, sigma, fbest=np.min(mse_obs)):
    """
    Calculates probability of improvement.
    """
    gamma_values = calculate_gamma(mu, sigma, fbest)
    return norm.cdf(gamma_values)
```

**Expected Improvement**

$$
a_\text{EI}(\mathbf{x}) = E[\max(f(\mathbf{x}) - f(\mathbf{x}_\text{best}), 0)] = \sigma(\mathbf{x})(\gamma(\mathbf{x})\Phi(\gamma(\mathbf{x})) + \phi(\gamma(\mathbf{x}))\,,
$$
with $\phi(\cdot)$ the PDF of a standard normal distirbution.

```python
def expected_improvement(mu, sigma, fbest=np.min(mse_obs)):
    """
    Calculates expected improvement.
    """
    gamma_values = calculate_gamma(mu, sigma, fbest)
    return sigma * (gamma_values * norm.cdf(gamma_values) + norm.pdf(gamma_values))
```

**GP Lower Confidence Limit**

$$
a_\text{LCB} = \mu(\mathbf{x}) - \kappa \sigma(\mathbf{x})\,,
$$
with $\kappa$ a parameter determining the tightness of the bound.

```python
def lower_confidence_bound(mu, sigma, kappa=2):
    """
    Calculates lower confidence bound.

    Made negative: maximizing acquisition function!
    """
    return - (mu - kappa * sigma)
```

```python
def plot_acquisitions():
    fig, (ax0, ax1, ax2, ax3) = plt.subplots(nrows=4, sharex=True, figsize=(8, 10))

    # plot mu and sigma
    ax0.plot(k_slice[:,0], mu, c=green, label='expected value')
    ax0.plot(k_slice[:,0], mu-2*sigma, c=orange, ls='--', label='95% interval')
    ax0.plot(k_slice[:,0], mu+2*sigma, c=orange, ls='--')
    ax0.semilogx()
    ax0.set_ylabel('predicted\nMSE')

    # plot information gains
    ax1.plot(k_slice[:,0], probability_improvement(mu, sigma), c=blue, label='PI')
    ax1.set_ylabel('PI')
    ax2.plot(k_slice[:,0], expected_improvement(mu, sigma), c=blue, label='EI')
    ax2.set_ylabel('EI')
    ax3.plot(k_slice[:,0], lower_confidence_bound(mu, sigma), c=blue, label='LCB')
    ax3.set_ylabel('LCB')
    ax3.set_xlabel('$\gamma$')

```

```python
plot_acquisitions()
```

```python
PI = np.zeros_like(mse_pars)
EI = np.zeros_like(mse_pars)
LCB = np.zeros_like(mse_pars)

for i, g in enumerate(g_vals):
    for j, k in enumerate(k_vals):
        instance = np.log10([[g, k]])
        mu, sigma = gaussian_process.predict(instance, return_std=True)
        mu = mu[:]
        sigma = sigma[:]
        PI[i,j] = probability_improvement(mu, sigma)
        EI[i,j] = expected_improvement(mu, sigma)
        LCB[i,j] = lower_confidence_bound(mu, sigma)
```

```python
def show_acquisition_contourf(ax0, ax1, ax2):
    ax0.contourf(K, G, PI.T)
    ax0.set_title('PI')
    ax0.scatter(simulated_parameters[:,0], simulated_parameters[:,1], c=red)

    ax1.contourf(K, G, EI.T)
    ax1.set_title('EI')
    ax1.scatter(simulated_parameters[:,0], simulated_parameters[:,1], c=red)

    ax2.contourf(K, G, LCB.T)
    ax2.set_title('LCB')
    ax2.scatter(simulated_parameters[:,0], simulated_parameters[:,1], c=red)

    for ax in (ax0, ax1, ax2):
        ax.set_xlabel('$k$')
        ax.set_ylabel('$\gamma$')
        ax.loglog()
        ax.scatter(kstar, gstar, c=yellow)
        ax.scatter(simulated_parameters[:,0], simulated_parameters[:,1], c=red)
```

```python
fig, (ax0, ax1, ax2) = plt.subplots(ncols=3, figsize=(13, 5))
show_acquisition_contourf(ax0, ax1, ax2)
```

Use Gaussian process model to select new points to evaluate!

## Gradient-based optimization for finding next point

If we assume that evaluating points from the Gaussian process surogate model is much cheaper than evaluating the true function, we can either use an exhaustive sampling of this space (if the dimension is not too large) or use a gradient-based optimizer to find a maximizer of the acquisition function.

```python
from scipy.optimize import minimize
```

```python
neg_acquisition = lambda parameter : -probability_improvement(*gaussian_process.predict(parameter.reshape((1,2)),
                                                                                return_std=True))
```

```python
# start from best previous point
f0, x0 = min(zip(mse_obs, simulated_parameters))

result = minimize(neg_acquisition, x0=x0,
                  bounds=[(-2, 2), (-2, 2)])

xnew = result.x

result
```

         nfev: 18
          fun: array([-0.36629826])
       status: 0
            x: array([ 2.        ,  1.06184265])
      success: True
          jac: array([ -1.83551085e-01,  -1.02140518e-06])
      message: b'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL'
     hess_inv: <2x2 LbfgsInvHessProduct with dtype=float64>
          nit: 5

```python
print('New parameters to try: {}'.format(10**xnew))
```

    New parameters to try: [ 100.           11.53035424]

## Concluding remarks

**Remark 1**

Gaussian process model usually continuous and differentiable w.r.t. $\mathbf{x}$:

- $\nabla \mu(\mathbf{x})$
- $\nabla \sigma(\mathbf{x})$

gradient-based optimization!

**Ramark 2**

Use of correlation between instances:

```python
param_random = 10**np.random.uniform(-2, 2, size=(20, 2))

mu, cov = gaussian_process.predict(np.log10(param_random), return_cov=True)

fig, (ax0, ax1) = plt.subplots(ncols=2, figsize=(8, 4))
ax0.plot(10**mu)
ax0.set_ylabel('estimated MSE')

ax1.imshow(cov, interpolation='nearest', cmap='hot')
ax1.set_title('Covariance')
```

Use coviance to select a *set* of informative instances to explore!

**Remark 3**

Some instances will differ in execution times:

- regularization size (machine learning)
- learning rate (machine learning)
- number of parameters
- grid size approximation (mechanistic modelling)

Better optimize *expected improvement per second*. Use a model of expected duration (second Gaussian process).

## References

Snoek, J., Larochelle, H., Adams, R. '*Practical Bayesian optimization of machine learning algorithms*'. Advances in Neural Information Processing Systems (2012)

Rasmussen, C., Williams, C., '*Gaussian Processes for Machine Learning*'. The MIT Press (2006)

```python

```


# Computational Astrophysics
## Optimization. Linear Fit

---
## Eduard Larrañaga

Observatorio Astronómico Nacional\
Facultad de Ciencias\
Universidad Nacional de Colombia

---

### About this notebook

In this notebook we present the method to define a straight line that fits a dataset.

---

Empirical relationships (e.g., the $M-\sigma$ relation for galaxies)
are typically established by taking experimental/observational data
and fitting an analytic function to them.

In this section, we will
introduce the most common curve fitting methods.

---
## Curve Fitting Criteria

$N$ data points $(x_i,y_i)$

Fit function $Y(x,\{a_j\})$

Parameters $\{a_j\}$ with $j=1,2,3,...,M$

### Least squares fit

\begin{equation}
\Delta_i = Y(x_i,\{a_j\}) - y_i\,\,,
\end{equation}

The goal is to minimize the function

\begin{equation}
\Delta(\{a_j\}) = \sum_{i=1}^N \Delta_i^2 = \sum_{i=1}^N \left( Y(x_i,\{a_j\}) - y_i \right)^2\,\,.
\end{equation}

The square of the difference is used because negative
and positive variations would otherwise partially or fully cancel out, leading to a wrong result.

### Data with y-uncertainties

Observational data having an estimated error as $y_i \pm \sigma_i$

Function to minimize: $\chi^2$

\begin{equation}
\chi^2(\{a_j\}) = \sum_{i=1}^N \left(\frac{\Delta_i}{\sigma_i} \right)^2 =
\sum_{i=1}^N \left( \frac{Y(x_i,\{a_j\}) - y_i}{\sigma_i} \right)^2\,\,
\label{eq:analysis_chi21}
\end{equation}

---
## Linear Regression

The simplest curve to fit some data is a stright
line (**linear regression**).

\begin{equation}
Y(x, \{a_1,a_2\}) = a_1 + a_2 x\,\,,
\end{equation}

The objective is to determine $a_1$ and $a_2$ such that

\begin{equation}
\chi^2(a_1,a_2) = \sum_{i=1}^N \frac{1}{\sigma_i^2} (a_1 + a_2 x_i - y_i)^2
\label{eq:analysis_chi22}
\end{equation}

is minimized. Differentiating this equation and setting the result to zero,

\begin{equation}
\begin{aligned}
\frac{\partial \chi^2}{\partial a_1} &= 2 \sum_{i=1}^N \frac{1}{\sigma_i^2}(a_1 + a_2x_i - y_i) = 0\,\,,\\
\frac{\partial \chi^2}{\partial a_2} &= 2 \sum_{i=1}^N \frac{1}{\sigma_i^2}(a_1 + a_2 x_i - y_i) x_i = 0\,\,.
\end{aligned}
\end{equation}

This system can be written as

\begin{equation}
\begin{aligned}
a_1 S + a_2 \Sigma x - \Sigma y &= 0\,\,,\\
a_1\Sigma x + a_2 \Sigma x^2 - \Sigma xy &=0\,\,,
\end{aligned}
\end{equation}

with

\begin{align}
S =& \sum_{i=1}^N \frac{1}{\sigma_i^2}\\
\Sigma x =& \sum_{i=1}^N \frac{x_i}{\sigma_i^2}\\
\Sigma y =& \sum_{i=1}^N \frac{y_i}{\sigma_i^2}\\
\Sigma x^2 =& \sum_{i=1}^N \frac{x_i^2}{\sigma_i^2}\\
\Sigma xy =& \sum_{i=1}^N \frac{x_i y_i}{\sigma_i^2}\\
\end{align}

Solving for the two unknowns $a_1$ and $a_2$,

\begin{equation}
a_1 = \frac{\Sigma y\Sigma x^2 - \Sigma x \Sigma xy}{S\Sigma x^2 - (\Sigma x)^2}
\,\,,\hspace{2em} a_2 = \frac{S\Sigma xy - \Sigma y \Sigma x}{S\Sigma x^2 - (\Sigma x)^2}\,\,.
\label{eq:analysis_chi23}
\end{equation}

**Notes**
- If all $\sigma_i$ are identical, they will cancel out of the
above equations and $a_1$ and $a_2$ will be independent of them.

- If the $\sigma_i$ are unknown, then one can still use the $\chi^2$ method and
just sets $\sigma_i = 1$.

---
### Data with x- and y-uncertainties

Observational data having an estimated error as $y_i \pm \sigma_i$ and $x_i \pm \sigma^x_i$

The uncertainty in $x_i$ can be incorporated in the $\chi^2$ fit by relating $\sigma^x_i$ into an additional error $\sigma^{\text{extra}}_i$ in $y_i$.

To first order, this can be done by writing

\begin{equation}
\sigma_{i,\text{extra}} = \left|\frac{\partial y}{\partial x} \right|_i \sigma^x_i\,\,,
\end{equation}

where one needs an approximation for the slope $\frac{\partial y}{\partial x}$.

- If both $\sigma_i$ and $\sigma_{i,\text{extra}}$
contribute significantly, one simply adds their squares:
$\sigma_{i,\text{total}}^2$ = $\sigma_i^2$ + $\sigma_{i,\text{extra}}^2$.

- If the error in $x_i$ or $y_i$ is asymmetric about $(x_i,y_i)$ one
could weigh by the maximum of the left and right error, or use
advanced techniques to incorporate this information.

---
### Associated Error to the fit parameters $a_j$

Associated error bar  $\sigma_{a_j}^2$  for the
curve fit parameter $a_j$.

Using first-order error propagation,
we have

\begin{equation}
\sigma^2_{a_j} = \sum_{i=1}^N \left(\frac{\partial a_j}{\partial y_i}
\right)^2 \sigma_i^2\,\,,
\end{equation}

from which we obtain

\begin{equation}
\sigma_{a_1} = \sqrt{\frac{\Sigma x^2}{S\Sigma x^2 - (\Sigma x)^2}}\,\,,
\hspace{2em} \sigma_{a_2} = \sqrt{\frac{S}{S\Sigma x^2 - (\Sigma x)^2}}\,\,.
\end{equation}

---
If the data set doesn't have an associated set of error bars, the error
 $\sigma_{a_j} = \sigma_0$ is estimated from the sample variance of the data,

\begin{equation}
\sigma_0^2 = \frac{1}{N-2} \sum_{i=1}^N \left(y_i - (a_1 + a_2 x_i) \right)^2\,\,.
\end{equation}

The normalization factor $N-2$ of the variance is due to the fact that
we have taken out two parameters ($a_1$ and $a_2$) from the
data.

## Non-Linear Fit

Many non-linear fitting problems may be transformed to linear
problems by a simple change of variables. \\
\bigskip

**Example**

Consider a power law

\begin{equation}
Z(t, \{\alpha,\beta\}) = \alpha t^{\beta}\,\,.
\end{equation}

This may be rewritten as $Y(x, \{a_1,a_2\}) = a_1 + a_2 x$ with

\begin{equation}
Y = \log Z\,\,,\hspace{1em} x = \log t\,\,\hspace{1em} a_1 = \log
\alpha\,\,,\hspace{1em} a_2 = \beta\,\,.
\end{equation}

**Example**

Consider an exponential

\begin{equation}
Z(t, \{\alpha, \beta\}) = \alpha e^{\beta x}\,\,.
\end{equation}

This may be rewritten as $Y(x, \{a_1,a_2\}) = a_1 + a_2 x$ with

\begin{equation}
Y = \ln Z\,\,,\hspace{1em} a_1 = \ln
\alpha\,\,,\hspace{1em} a_2 = \beta\,\,.
\end{equation}

```python

```