# Simular el Modelo  de HW en la Medida Forward

## Encontrar el Nuevo *Drift*

Primero se debe encontrar el nuevo drift $m^*\left(r_t,t\right)+\sigma_Z\left(r_t,t\right)\sigma\left(r_t,t\right)$ con

$$
\begin{equation}
\sigma_{Z}\left(r_t,t\right)=\frac{\partial Z}{\partial r}\frac{1}{Z}\sigma\left(r_t,t\right)
\end{equation}
$$

Recordar que en el modelo de HW:

$$
m^*\left(r_t,t\right)=\theta_t-\gamma^* r_t
$$

Y la fórmula para el valor de un bono cupón cero es:

$$
Z\left(r_t,t,T\right)=\exp\left[A\left(t,T\right)-B\left(t,T\right)r_t\right]
$$

Por lo tanto,

$$
\frac{\partial Z}{\partial r}=-B\left(t,T\right)Z
$$

Lo que implica que,

$$
\sigma_{Z}\left(r_t,t\right)=-B\left(t,T\right)\sigma
$$

Finalmente, el drift en la medida forward es:

$$
\theta_t-B\left(t,T\right)\sigma^2-\gamma^* r_t
$$

Para simular, puedo utilizar las funciones del notebook 13, considerando un nuevo $\theta_t$ sea éste $\overline{\theta_t}$, dado por la siguiente fórmula:

$$
\overline{\theta_t}=\theta_t-B\left(t,T\right)\sigma^2
$$

## Simular

```python
from scipy.interpolate import interp1d
import plotly.express as px
import pandas as pd
import numpy as np
import math
```

Se obtienen los valores de la curva cero cupón:

```python
curva = pd.read_excel('data/20201012_built_sofr_zero.xlsx')
```

```python
curva['t'] = curva['plazo'] / 365.0
```

```python
curva['rate'] = np.log(1 / curva['df'])/( curva['plazo'] / 365.0)
```

```python
curva.head()
```

      plazo
      tasa
      df
      t
      rate

      0
      1
      0.000811
      0.999998
      0.002740
      0.000811

      1
      7
      0.000841
      0.999984
      0.019178
      0.000841

      2
      14
      0.000780
      0.999970
      0.038356
      0.000780

      3
      21
      0.000774
      0.999955
      0.057534
      0.000774

      4
      33
      0.000781
      0.999929
      0.090411
      0.000781

La curva se obtiene interpolando linealmente en los factores de descuento. Esto es una novedad respecto al notebook 13.

```python
dfcurva = interp1d(curva['t'],
                  curva['rate'],
                  kind='cubic',
                  fill_value="extrapolate")
def zrate(t: float):
    return dfcurva(t)
```

```python
zrate(2)
```

    array(0.00056758)

Se definen las derivadas de la curva.

```python
def dzrate(t: float) -> float:
    delta = .0001
    return (zrate(t + delta) - zrate(t)) / delta

def d2zrate(t: float) -> float:
    delta = .0001
    return (dzrate(t + delta) - dzrate(t)) / delta
```

```python
t = 2
print(f'zrate({t:.2f}) = {zrate(t):.4%}')
print(f'dzrate({t:.2f}) = {dzrate(t):.4%}')
print(f'd2zrate({t:.2f}) = {d2zrate(t):.4%}')
```

    zrate(2.00) = 0.0568%
    dzrate(2.00) = -0.0066%
    d2zrate(2.00) = 0.0584%

```python
def fwd(t: float) -> float:
    return zrate(t) + t * dzrate(t)

def dfwd(t: float) -> float:
    return 2 * dzrate(t) + t * d2zrate(t)
```

```python
print(f'fwd({t:.2f}) = {fwd(t):.4%}')
print(f'dfwd({t:.2f}) = {dfwd(t):.4%}')
```

    fwd(2.00) = 0.0436%
    dfwd(2.00) = 0.1036%

```python
sigma = .015
gamma = .5

# Proxy de la tasa instantánea r(t) es una tasa entre t y t + dt (donde dt es un intervalo infitesimal)
r0 = zrate(1/264)
print(r0)
```

    0.0008182064199505745

```python
def b_hw(gamma: float, t: float, T: float) -> float:
    """
    Calcula el valor de la función B(t,T) que interviene en la fórmula
    para el valor de un bono cupón cero en el modelo de HW.

    params:

    - gamma: intensidad de reversión del modelo HW
    - t:
    - T:

    return:

    - valor de la función B(t, T)
    """
    aux = 1 - math.exp(- gamma * (T - t))
    return aux / gamma
```

```python
def theta_fwd_measure(t: float, T:float) -> float:
    aux = (sigma ** 2) / (2.0 * gamma) * (1 - math.exp(-2.0 * gamma * t))
    return dfwd(t) + gamma * fwd(t) + aux - b_hw(gamma, t, T)*sigma**2
```

```python
# Para el modelo de HW simular en la medida implica solamente un ajuste
# en la función Theta(t).
def sim_hw_many(gamma, sigma, theta_fwd_measure, T, r0, num_sim, num_steps, seed = None):
    """
    """
    dt = 1 / 264.0
    num_steps += 1

    # Calcula los números aleatorios
    alea = np.zeros((num_sim, num_steps))
    np.random.seed(seed)

    for i in range(0, num_sim):
        for j in range(0, num_steps):
            alea[i][j] = np.random.normal()

    # Calcula los valores de Theta. Theta sólo depende del tiempo, no de la simulación.
    theta_array = np.zeros(num_steps)
    tiempo = np.zeros(num_steps)
    for i in range(1, num_steps):
        tiempo[i] = i * dt
        theta_array[i] = theta_fwd_measure(i * dt, T)

    # Simula las trayectorias
    sqdt_sigma = math.sqrt(dt) * sigma
    gamma_dt = gamma * dt
    sim = np.zeros((num_sim, num_steps))
    for i in range(0, num_sim):
        sim[i][0] = r0
        r = r0
        for j in range(1, num_steps):
            r = r + theta_array[j - 1] * dt - gamma_dt * r + sqdt_sigma * alea[i][j - 1]
            sim[i][j] = r
    return tiempo, sim
```

```python
num_sim = 1000
num_steps = 264
seed = 1234
df = 0
T = 1
tiempo, s = sim_hw_many(gamma, sigma, theta_fwd_measure, T, r0, num_sim, num_steps, seed)
```

```python
which_sim = 999
result = [(z[0], z[1]) for z in zip(tiempo, s[which_sim][0:])]
df_sim = pd.DataFrame(result, columns=['plazo', 'tasa'])
```

```python
fig = px.line(
    df_sim,
    x='plazo',
    y='tasa',
    title=f'Sim HW en Medida Fwd'
)
fig.show()
```

## Valorizar una Ballena

La *ballena* es una especie de call sobre un bono cupón cero. Supongamos que:

- bono cero cupón subyacente es el que vence hoy en 2Y más.
- Valorizar para strikes: 98, 99, 100, 101, 102.
- Usar 1000 simulaciones.
- Los precios del bono cupón cero hay que multiplicarlos por 100 antes de calcular el payoff.

Para cada simulación, con la última tasa de la simulación,

- Calcular $z$ e insertarlo en el payoff de la ballena.
- Calcular el promedio de esos resultados.
- Traer el promedio a valor presente con el factor de descuento a 1Y de la curva.

```python
def ballena(z: float, strike: float) -> float:
    """
    z: es el precio del bono cupón cero al vencimiento
    strike: es el strike de la opción
    """
    return max(math.log(max(z - strike, .0001)), .0001)
```

```python
result = [(i, ballena(i, 95)) for i in range(90, 111)] # list comprehensions
df_ballena = pd.DataFrame(result, columns=['precio', 'valor'])
```

```python
fig = px.line(
    df_ballena,
    x='precio',
    y='valor',
    title=f'Payoff de una Ballena sobre un Bono Cero Cupón'
)
fig.show()
```

### Extrae la Última Tasa de Cada Simulación

```python
last_rates = [sim[-1] for sim in s]
```

```python
which_sim = 50
print(f'Última tasa de la simulación {which_sim}: {last_rates[which_sim]: .4%}')
```

    Última tasa de la simulación 50: -0.4078%

### Calcula el Precio del Bono Cupón Cero

Traemos la función $A$:

```python
def a_hw(
    zrate: float,
    fwd,
    gamma: float,
    sigma: float,
    t: float,
    T: float,
    verbose = False
) -> float:
    """
    verbose: cuando es True imprime los valores de c1, c2 y c3.
    """
    b = b_hw(gamma, t, T)
    dfT = math.exp(-zrate(T) * T)
    dft = math.exp(-zrate(t) * t)
    c1 = math.log(dfT / dft)
    c2 = b * fwd(t)
    c3 = (sigma**2) / (4 * gamma) * (b**2) * (1 - math.exp(-2 * gamma * t))
    if verbose:
        print("c1: " + str(c1))
        print("c2: " + str(c2))
        print("c3: " + str(c3))
    return c1 + c2 - c3
```

Y la función para el precio del bono cupón cero:

```python
def zero_hw(
        r: float,
        gamma: float,
        sigma: float,
        zrate: float,
        fwd: float,
        t: float,
        T: float) -> float:
    a = a_hw(zrate, fwd, gamma, sigma, t, T)
    b = b_hw(gamma, t, T)
    return math.exp(a - b * r)
```

```python
precios = [zero_hw(r, gamma, sigma, zrate, fwd, t=1, T=2) * 100 for r in last_rates]
```

```python
which_sim = 50
print(f'Valor bono cupón cero con la simulación {which_sim}: {precios[which_sim]: .4f}')
```

    Valor bono cupón cero con la simulación 50:  100.3226

### Calcula Payoffs

```python
strikes = [98, 99, 100, 101, 102]
```

```python
payoffs = {strike:[ballena(z, strike) for z in precios] for strike in strikes}
```

```python
payoffs[102][0:10]
```

    [0.0001,
     0.0001,
     0.0001,
     0.0001,
     0.0001,
     0.0001,
     0.0001,
     0.0001,
     0.0001,
     0.0001]

### Calcula Valores Presente

```python
t_ = 1.0 # 1 año que corresponde al vencimiento de la ballena.
```

```python
disc_factor = math.exp(-t_ * float(zrate(t_)))
valores = [disc_factor * np.average(payoffs[strike]) for strike in strikes]
```

```python
for v in valores:
    print(f'valor: {v:.8f}')
```

    valor: 0.63402728
    valor: 0.23224051
    valor: 0.04233673
    valor: 0.00351143
    valor: 0.00009993


# Lecture 10 - Priors on Function Spaces: Gaussian Processes

## Objectives:

+ Express prior knowledge/beliefs about model outputs using Gaussian process (GP)

+ Sample functions from the probability measure defined by GP

## Readings:

Please read the following before lecture:

+ [Chapter 1 from C.E. Rasmussen's textbook on Gaussian processes](http://www.gaussianprocess.org/gpml/chapters/RW1.pdf).

+ (Optional video lecture?) [Neil Lawrence's video lecture on Introduction to Gaussian processes](https://www.youtube.com/watch?v=ewJ3AxKclOg).

### Modeling prior knowledge in Gaussian processes

An experienced scientist or engineer typically has some knowledge about a function of interest $f(\cdot)$ even before observing it anywhere. For example, he/she might know that $f(\cdot)$ cannot exceed, or be smaller than, certain values or that it is periodic or that it shows translational invariance. Such knowledge is known as the *prior knowledge*.
Prior knowledge may be *precise*, e.g., the response is twice differentiable, or it may be vague, e.g., the probability that the periodicity is $T$ is $p(T)$. When one is dealing with vague prior knowledge, he/she may refer to it as *prior belief*. Almost always, prior knowledge a field quantity is a _prior belief_.

Prior beliefs about $f(\cdot)$ can be modeled by a probability measure on the space of functions from $\mathcal{X}$ to $\mathbb{R}$.
A Gaussian process (GP) is a great way to represent this probability measure.

### Introduction to Gaussian Processes.
In many engineering problems we have to deal with functions that are unknown.
For example, in oil reservoir modeling, the permeability tensor or the porosity  of
the ground are, generally, unknown quantities.
Therefore, we would like to treat them as if they where random.
That is, we have to talk about probabilities on function spaces.
Such a thing is achieved via the theory of *random fields*.
However, instead of developing the generic mathematical theory of random fields,
we concentrate on a special class of random fields, the *Gaussian random fields*
or *Gaussian processes*.

A Gaussian process (GP) is a generalization of a multivariate Gaussian distribution to
*infinite* dimensions.
It essentially defines a probability measure on a function space.
When we say that $f(\cdot)$ is a GP, we mean that it is a random variable that is actually
a function.
Mathematically, we write:
\begin{equation}
f(\cdot) | m(\cdot), k(\cdot, \cdot) \sim \mbox{GP}\left(f(\cdot) | m(\cdot), k(\cdot, \cdot) \right),
\end{equation}
where
$m:\mathbb{R}^d \rightarrow \mathbb{R}$ is the *mean function* and
$k:\mathbb{R}^d \times \mathbb{R}^d \rightarrow \mathbb{R}$ is the *covariance function*.
So, compared to a multivariate normal we have:

+ A random function $f(\cdot)$ instead of a random vector $\mathbf{x}$.
+ A mean function $m(\cdot)$ instead of a mean vector $\boldsymbol{\mu}$.
+ A covariance function $k(\cdot,\cdot)$ instead of a covariance matrix $\mathbf{\Sigma}$.

But, what does this definition actually mean? Actually, it gets its meaning from the multivariate Gaussian distribution. Here is how:

+ Let $\mathbf{x}_{1:n}=\{\mathbf{x}_1,\dots,\mathbf{x}_n\}$ be $n$ points in $\mathbb{R}^d$.
+ Let $\mathbf{f}\in\mathbb{R}^n$ be the outputs of $f(\cdot)$ on each one of the elements of $\mathbf{x}_{1:n}$, i.e.,
$$
\mathbf{f} =
\left(
\begin{array}{c}
f(\mathbf{x}_1)\\
\vdots\\
f(\mathbf{x}_n)
\end{array}
\right).
$$
+ The fact that $f(\cdot)$ is a GP with mean and covariance function $m(\cdot)$ and $k(\cdot,\cdot)$, respectively, *means* that the vector of outputs $\mathbf{f}$ at
the arbitrary inputs in $\mathbf{X}$ is the following multivariate-normal:
$$
\mathbf{f} | \mathbf{x}_{1:n}, m(\cdot), k(\cdot, \cdot) \sim \mathcal{N}\left(\mathbf{f} | \mathbf{m}(\mathbf{x}_{1:n}), \mathbf{K}(\mathbf{x}_{1:n}, \mathbf{x}_{1:n}) \right),
$$
with mean vector:
$$
\mathbf{m}(\mathbf{x}_{1:n}) =
\left(
\begin{array}{c}
m(\mathbf{x}_1)\\
\vdots\\
m(\mathbf{x}_n)
\end{array}
\right),
$$
and covariance matrix:
$$
\mathbf{K}(\mathbf{x}_{1:n},\mathbf{x}_{1:n}) = \left(
\begin{array}{ccc}
k(\mathbf{x}_1,\mathbf{x}_1) & \dots & k(\mathbf{x}_1, \mathbf{x}_n)\\
\vdots & \ddots & \vdots\\
k(\mathbf{x}_n, \mathbf{x}_1) & \dots & k(\mathbf{x}_n, \mathbf{x}_n)
\end{array}
\right).
$$

Now that we have defined a Gaussian process (GP), let us talk about we encode our prior beliefs into a GP.
We do so through the mean and covariance functions.

### Interpretation of the mean function.
What is the meaning of $m(\cdot)$?
Well, it is quite easy to grasp.
For any point $\mathbf{x}\in\mathbb{R}^d$, $m(\mathbf{x})$ should give us the value we believe is more probable for
$f(\mathbf{x})$.
Mathematically, $m(\mathbf{x})$ is nothing more than the expected value of the random variable $f(\mathbf{x})$.
That is:
\begin{equation}
m(\mathbf{x}) = \mathbb{E}[f(\mathbf{x})].
\end{equation}

The mean function can be any arbitrary function. Essentially, it tracks generic trends in the response as the input is varied. In practice, we try and make a suitable choice for the mean function that is easy to work with. Such choices include:

+ zero, i.e.,
$$
m(\mathbf{x}) = 0.
$$

+ a constant, i.e.,
$$
m(\mathbf{x}) = c,
$$
where $c$ is a parameter.

+ linear, i.e.,
$$
m(\mathbf{x}) = c_0 + \sum_{i=1}^dc_ix_i,
$$
where $c_i, i=0,\dots,d$ are parameters.

+ using a set of $m$ basis functions (generalized linear model), i.e.,
$$
m(\mathbf{x}) = \sum_{i=1}^mc_i\phi_i(\mathbf{x}),
$$
where $c_i$ and $\phi_i(\cdot)$ are parameters and basis functions.

+ generalized polynomial chaos (gPC), i.e.,
using a set of $d$ polynomial basis functions upto a given degree $\rho$
$m(\mathbf{x}) = \sum_{i=1}^{d}c_i\phi_i(\mathbf{x})$
where the basis functions $\phi_i$ are mutually orthonormal with respect to some
measure $\mu$:
$$
\int \phi_{i}(\mathbf{x}) \phi_{j}(\mathbf{x}) d\mu(\mathbf{x}) = \delta_{ij}
$$

+ and many other possibilities.

### Interpretation of the covariance function.
What is the meaning of $k(\cdot, \cdot)$?
This concept is considerably more challenging than the mean.

Let's try to break it down:

+ Let $\mathbf{x}\in\mathbb{R}^d$. Then $k(\mathbf{x}, \mathbf{x})$ is the variance of the random variable $f(\mathbf{x})$, i.e.,
$$
\mathbb{V}[f(\mathbf{x})] = \mathbb{E}\left[\left(f(\mathbf{x}) - m(\mathbf{x}) \right)^2 \right].
$$
In other words, we believe that there is about $95\%$ probability that the value of
the random variable $f(\mathbf{x})$ fall within the interval:
$$
\left((m(\mathbf{x}) - 2\sqrt{k(\mathbf{x}, \mathbf{x})}, m(\mathbf{x}) + 2\sqrt{k(\mathbf{x},\mathbf{x})}\right).
$$

+ Let $\mathbf{x},\mathbf{x}'\mathbb{R}^d$. Then $k(\mathbf{x}, \mathbf{x}')$ tells us how the random variable $f(\mathbf{x})$ and
$f(\mathbf{x}')$ are correlated. In particular, $k(\mathbf{x},\mathbf{x}')$ is equal to the covariance
of the random variables $f(\mathbf{x})$ and $f(\mathbf{x}')$, i.e.,
$$
k(\mathbf{x}, \mathbf{x}') = \mathbb{C}[f(\mathbf{x}), f(\mathbf{x}')]
= \mathbb{E}\left[
\left(f(\mathbf{x}) - m(\mathbf{x})\right)
\left(f(\mathbf{x}') - m(\mathbf{x}')\right)
\right].
$$

Essentially, a covariance function (or covariance kernel) defines a nearness or similarity measure on the input space. We cannot choose any arbitrary function of two variables as a covariance kernel. How we go about choosing a covariance function is discussed in great detail [here](http://www.gaussianprocess.org/gpml/chapters/RW4.pdf). We briefly discuss some properties of covariance functions here and then we shall move onto a discussion of what kind of prior beliefs we can encode through the covariance function.

### Properties of the covariance function

+ There is one property of the covariance function that we can note right away.
Namely, that for any $\mathbf{x}\in\mathbb{R}^d$, $k(\mathbf{x}, \mathbf{x}) > 0$.
This is easly understood by the interpretation of $k(\mathbf{x}, \mathbf{x})$ as the variance
of the random variable $f(\mathbf{x})$.

+ $k(\mathbf{x}, \mathbf{x}')$ becomes smaller as the distance between $\mathbf{x}$ and $\mathbf{x}'$ grows.

+ For any choice of points $\mathbf{X}\in\mathbb{R}^{n\times d}$, the covariance matrix: $\mathbf{K}(\mathbf{X}, \mathbf{X})$ has
to be positive-definite (so that the vector of outputs $\mathbf{f}$ is indeed a multivariate
normal distribution).

### Encoding prior beliefs in the covariance function.

+ **Modeling regularity**. The choice of the covariance function controls the regularity properties of the functions sampled from the probability induced by the GP. For example, if the covariance kernel chosen is the squared exponential kernel, which is infinitely differentiable, then the functions sampled from the GP will also be infinitely differentiable.

+ **Modeling invariance** If the covariance kernel is invariant w.r.t. a transformation $T$, i.e., $k(\mathbf{x}, T\mathbf{x}')=k(T\mathbf{x}, \mathbf{x}')=k(\mathbf{x}, \mathbf{x}')$ then samples from the GP will be invariant w.r.t. the same transformation.

+ Other possibilities include periodicity, additivity etc.

### Squared exponential covariance function

Squared expnential (SE) is the most commonly used covariance function.
Its formula is as follows:
$$
k(\mathbf{x}, \mathbf{x}') = v\exp\left\{-\frac{1}{2}\sum_{i=1}^d\frac{(x_i - x_i')^2}{\ell_i^2}\right\},
$$
where $v,\ell_i>0, i=1,\dots,d$ are parameters.
The interpretation of the parameters is as follows:

+ $v$ is known as the *signal strength*. The bigger it is, the more the GP $f(\cdot)$ will vary
about the mean.

+ $\ell_i$ is known as the *length scale* of the $i$-th input dimension of the GP.
The bigger it is, the smoother the samples of $f(\cdot)$ appear along the $i$-th input dimension.

Let's experiment with this for a while:

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns # Comment this out if you don't have it
sns.set_style('white')
sns.set_context('talk')
import GPy
# The input dimension
dim = 1
# The variance of the covariance kernel
variance = 1.
# The lengthscale of the covariance kernel
ell = 0.3
# Generate the covariance object
k = GPy.kern.RBF(dim, variance=variance, lengthscale=ell)
# Print it
print k
# and plot it
k.plot()
```

### Example 1: Plotting a covariance function
Remember:
> The covariance function $k(x,x')$ measures the similarity of $f(x)$ and $f(x')$.

The interactive tools provided, draw $k(\mathbf{x}, \mathbf{x}'=0)$ in one and two dimensions.
Use them to answer the following questions:
+ What is the intuitive meaning of $\ell$?
+ What is the intuitive meaning of $v$?
+ There are many other covariance functions that we could be using. Try changing ``RBF`` to ``Exponential``. What changes do you nottice.
+ Repeat the previous steps on a 2D covariance function.
+ If you still have time, try a couple of other covariances, e.g., ``Matern32``, ``Matern52``.
+ If you still have time, explore ``help(GPy.kern)``.

```python
from ipywidgets import interactive
def plot_kernel(variance=1., ell=0.3):
    k = GPy.kern.RBF(dim, variance=variance, lengthscale=ell)
    k.plot()
    plt.ylim(0, 10)
interactive(plot_kernel, variance=(1e-3, 10., 0.01), ell=(1e-3, 10., 0.01))
```

```python
from ipywidgets import interactive
def plot_kernel(variance=1., ell1=0.3, ell2=0.3):
    k = GPy.kern.RBF(2, ARD=True, variance=variance,
                     lengthscale=[ell1, ell2])  # Notice that I just changed the dimension here
    k.plot()
interactive(plot_kernel, variance=(1e-3, 10., 0.01), ell1=(1e-3, 10., 0.01), ell2=(1e-3, 10., 0.01))
```

### Example 2: Properties of the covariance matrix
Let $\mathbf{x}_{1:n}$ be an arbitrary set of input points. The covariance matrix $\mathbf{K}\in\mathbb{R}^{n\times n}$ defined by:
$$
\mathbf{K}\equiv\mathbf{K}(\mathbf{x}_{1:n}, \mathbf{x}_{1:n}) = \left(
\begin{array}{ccc}
k(\mathbf{x}_1,\mathbf{x}_1) & \dots & k(\mathbf{x}_1, \mathbf{x}_n)\\
\vdots & \ddots & \vdots\\
k(\mathbf{x}_n, \mathbf{x}_1) & \dots & k(\mathbf{x}_n, \mathbf{x}_n)
\end{array}
\right),
$$
must be [positive definite](https://en.wikipedia.org/wiki/Positive-definite_matrix). Mathematically this can be expressed in two equivalent ways:

+ For all vectors $\mathbf{v}\in\mathbb{R}^T$, we have:
$$
\mathbf{v}^t\mathbf{K}\mathbf{v} > 0,
$$
+ All the eigenvalues of $\mathbf{K}$ are positive.

Using the code provided:
+ Verify that the the sum of two covariance functions is a valid covariance function.
+ Verify that the product of two covariance functions is a valid covariance function.
+ Is the following function a covariance function:
$$
k(x, x') = k_1(x, x')k_2(x, x') + k_3(x, x') + k_4(x, x'),
$$
where all $k_i(x, x')$'s are covariance functions.
+ What about:
$$
k(x, x') = k_1(x, x') / k_2(x, x')?
$$

```python
# Number of dimensions
dim = 1

# Number of input points
n = 20

# The lengthscale
ell = .1

# The variance
variance = 1.

# The covariance function
k1 = GPy.kern.RBF(dim, lengthscale=ell, variance=variance)

# Draw a random set of inputs points in [0, 1]^dim
X = np.random.rand(n, dim)

# Evaluate the covariance matrix on these points
K = k1.K(X)

# Compute the eigenvalues of this matrix
eig_val, eig_vec = np.linalg.eigh(K)

# Plot the eigenvalues (they should all be positive)
print '> plotting eigenvalues of K'
print '> they must all be positive'
fig, ax = plt.subplots()
ax.plot(np.arange(1, n+1), eig_val, '.')
ax.set_xlabel('$i$', fontsize=16)
ax.set_ylabel('$\lambda_i$', fontsize=16)
```

```python
# Now create another (arbitrary) covariance function
k2 = GPy.kern.Exponential(dim, lengthscale=0.2, variance=2.1)

# Create a new covariance function that is the sum of these two:
k_new = k1 + k2

# Let's plot the new covariance
fig, ax = plt.subplots()
k1.plot(ax=ax, label='$k_1$')
k2.plot(ax=ax, label='$k_2$')
k_new.plot(ax=ax, label='$k_1 + k_2$')
plt.legend(fontsize=16);
```

```python
# If this is a valid covariance function, then it must
# be positive definite
# Compute the covariance matrix:
K_new = k_new.K(X)

# and its eigenvalues
eig_val_new, eig_vec_new = np.linalg.eigh(K_new)

# Plot the eigenvalues (they should all be positive)
print '> plotting eigenvalues of K'
print '> they must all be positive'
fig, ax = plt.subplots()
ax.plot(np.arange(1, n+1), eig_val_new, '.')
ax.set_xlabel('$i$', fontsize=16)
ax.set_ylabel('$\lambda_i$', fontsize=16);
```

### Example 3: Sampling from a Gaussian Process.

Samples from a Gaussian process are functions. But, functions are infinite dimensional objects?
We cannot sample directly from a GP....
However, if we are interested in the values of $f(\cdot)$ at any given set of test points $\mathbf{x}_{1:n} = \{\mathbf{x}_1,\dots,\mathbf{x}_b\}$, then we have that:
$$
\mathbf{f} | \mathbf{x}_{1:n}, m(\cdot), k(\cdot, \cdot) \sim \mathcal{N}\left(\mathbf{f} | \mathbf{m}(\mathbf{x}_{1:n}), \mathbf{K}(\mathbf{x}_{1:n}, \mathbf{x}_{1:n}) \right),
$$
where all the quantities have been introduced above.
This is
What we are going to do is pick a dense set of points $\mathbf{x}_{1:n}\in\mathbb{R}^{n\times d}$
sample the value of the GP, $\mathbf{f} = (f(\mathbf{x}_1),\dots,f(\mathbf{x}_n))$ on these points.
We saw above that the probability density of $\mathbf{f}$ is just a multivariate normal
with a mean vector that is specified from the mean function and a covariance matrix
that is specified by the covariance function.
Therefore, all we need to know is how to sample from the multivariate normal.
This is how we do it:
+ Compute the Cholesky of $\mathbf{L}$:
$$
\mathbf{K} = \mathbf{L}\mathbf{L}^T.
$$
+ Draw $n$ random samples $\mathbf{z} = (z_1,\dots,z_n)$ independently from a standard normal.
+ Get one sample by:
$$
\mathbf{f} = \mathbf{m} + \mathbf{L}\mathbf{z}.
$$

```python
# To gaurantee reproducibility
np.random.seed(123456)

# Number of test points
num_test = 10

# Pick a covariance function
k = GPy.kern.RBF(input_dim=1, variance=1., lengthscale=.1)

# Pick a mean function
mean_func = lambda(x): np.zeros(x.shape)

# Pick a bunch of points over which you want to sample the GP
X = np.linspace(0, 1, num_test)[:, None]

# Evaluate the mean function at X
m = mean_func(X)

# Compute the covariance function at these points
nugget = 1e-6 # This is a small number required for stability
C = k.K(X) + nugget * np.eye(X.shape[0])

# Compute the Cholesky of the covariance
# Notice that we need to do this only once
L = np.linalg.cholesky(C)

# Number of samples to take
num_samples = 3

# Take 3 samples from the GP and plot them:
fig, ax = plt.subplots()
# Plot the mean function
ax.plot(X, m)
for i in xrange(num_samples):
    z = np.random.randn(X.shape[0], 1)    # Draw from standard normal
    f = m + np.dot(L, z)                  # f = m + L * z
    ax.plot(X, f, color=sns.color_palette()[1], linewidth=1)
#ax.set_ylim(-6., 6.)
ax.set_xlabel('$x$', fontsize=16)
ax.set_ylabel('$y$', fontsize=16)
ax.set_ylim(-5, 5);
```

The solid line is the mean function and the dashed lines are 3 samples of   f . These don’t look like functions yet. This is because we have used only 10 test points to represent the GP.

#### Questions

1. Edit the code above changing the number of test points ``num_test`` to 20, 50, 100. Rerun the example. How do your samples of   f look like now? Do they look more like functions to you? Imagine that the true nature of the GP appears when these test points become infinitely dense.

2. Edit the code above and change the random seed to an arbitrary integer (just make up one). Rerun the example and notice how the sampled functions change.

3. Edit the code above and change the variance first to 0.1 and then to 5 each time rerunning the example. Notice the values on the vertical axis of the plot. What happens to the sampled functions as you do this? What does the variance parameter of the SE control?

4. Edit the code above and now change the length-scale parameter first to 0.05 and then to 1. What happens to the sampled functions as you do this? What does the length- scale parameter of the SE control?

5. Now set the variance and the length-scale back to their original values (1. and 0.1, respectively). Edit the code and change the mean function to:
```
     mean_fun = lambda(x): 5 * x
```
Re-run the example. What do you observe? Try a couple more. For example, try:
```
     mean_fun = lambda(x): np.sin(5 * np.pi * x)
```

6. So far, all the samples we have seen are smooth. There is this theorem that says that the samples of the GP will be as smooth as the covariance function we use. Since the SE covariance is infinitely smooth, the samples we draw are infinitely smooth. The [Matern 3-2 covariance function](https://en.wikipedia.org/wiki/Matérn_covariance_function) is twice differentiable. Edit the code and
change ``RBF`` to ``Matern32``. Rerun the example. How smooth are the samples now?

7. The exponential covariance function is continuous but not differentiable. Edit the code and change ``RBF`` to ``Exponential``. Rerun the example. How smooth are the samples now?

8. The covariance function can also be used to model invariances. The periodic exponential covariance function is... a periodic covariance function. Edit line 29 and change ``RBF`` to
```
k = GPy.kern.PeriodicMatern32(input_dim=1, variance=500., lengthscale=0.01, period=0.1)
```
Rerun the example. Do you notice the periodic pattern?

9. How can you encode the information that there are two lengthscales in $f(\cdot)$. There are many ways to do this.
Try summing or multiplying covariance functions.