# Bayesian Parametric Regression

    Notebook version: 1.4 (Sep 22, 2018)

    Author: Jerónimo Arenas García (jarenas@tsc.uc3m.es)
            Jesús Cid-Sueiro (jesus.cid@uc3m.es)

    Changes: v.1.0 - First version
             v.1.1 - ML Model selection included
             v.1.2 - Some typos corrected
             v.1.3 - Rewriting text, reorganizing content, some exercises.
             v.1.4 - Revised introduction

    Pending changes: * Include regression on the stock data

```python
# Import some libraries that will be necessary for working with data and displaying plots

# To visualize plots in the notebook
%matplotlib inline
from IPython import display

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import scipy.io       # To read matlab files
import pylab
import time
```

## A quick note on the mathematical notation

In this notebook we will make extensive use of probability distributions. In general, we will use capital leters
${\bf X}$, $S$, $E$ ..., to denote random variables, and lower-case letters ${\bf x}$, $s$, $\epsilon$ ..., to denote the values they can take.

In general, we will use letter $p$ for probability density functions (pdf). When necessary, we will use, capital subindices to make the random variable explicit. For instance, $p_{{\bf X}, S}({\bf x}, s)$ would be the joint pdf of random variables ${\bf X}$ and $S$ at values ${\bf x}$ and $s$, respectively.

However, to avoid a notation overload, we will omit subindices when they are clear from the context. For instance, we will use $p({\bf x}, s)$ instead of $p_{{\bf X}, S}({\bf x}, s)$.

## 1. Model-based parametric regression

### 1.1. The regression problem.

Given an observation vector ${\bf x}$, the goal of the regression problem is to find a function $f({\bf x})$ providing *good* predictions about some unknown variable $s$. To do so, we assume that a set of *labelled* training examples, $\{{\bf x}^{(k)}, s^{(k)}\}_{k=1}^K$ is available.

The predictor function should make good predictions for new observations ${\bf x}$ not used during training. In practice, this is tested using a second set (the *test set*) of labelled samples.

### 1.2. Model-based parametric regression

Model-based regression methods assume that all data in the training and test dataset have been generated by some stochastic process. In parametric regression, we assume that the probability distribution generating the data has a known parametric form, but the values of some parameters are unknown.

In particular, in this notebook we will assume the target variables in all pairs $({\bf x}^{(k)}, s^{(k)})$ from the training and test sets have been generated independently from some posterior distribution $p(s| {\bf x}, {\bf w})$, were ${\bf w}$ is some unknown parameter. The training dataset is used to estimate ${\bf w}$.

### 1.3. Model assumptions

In order to estimate ${\bf w}$ from the training data in a mathematicaly rigorous and compact form let us group the target variables into a vector
$$
{\bf s} = \left(s^{(0)}, \dots, s^{(K-1)}\right)^\top
$$
and the input vectors into a matrix
$$
{\bf X} = \left({\bf x}^{(0)}, \dots, {\bf x}^{(K-1)}\right)^\top
$$

We will make the following assumptions:

   * A1. All samples in ${\cal D}$ have been generated by the same distribution, $p({\bf x}, s \mid {\bf w})$
   * A2. Input variables ${\bf x}$ do not depend on ${\bf w}$. This implies that
$$
p({\bf X} \mid {\bf w}) = p({\bf X})
$$
   * A3. Targets $s^{(0)},\ldots, s^{(K-1)}$ are statistically independent, given ${\bf w}$ and the inputs ${\bf x}^{(0)},\ldots, {\bf x}^{(K-1)}$, that is:
$$
p({\bf s} \mid {\bf X}, {\bf w}) = \prod_{k=0}^{K-1} p(s^{(k)} \mid {\bf x}^{(k)}, {\bf w})
$$

## 2. Bayesian inference.

### 2.1. The Bayesian approach

The main idea of Bayesian inference is the following: assume we want to estimate some unknown variable $U$ given an observed variable $O$. If $U$ and $O$ are random variables, we can describe the relation between $U$ and $O$ through the following functions:

   * Prior distribution: $p_U(u)$. It describes our uncertainty on the true value of $U$ before observing $O$.
   * Likelihood function: $p_{O \mid U}(o \mid u)$. It describes how the value of the observation is generated for a given value of $U$.
   * Posterior distribution: $p_{U|O}(u \mid o)$. It describes our uncertainty on the true value of $U$ once the true value of $O$ is observed.

The major component of the Bayesian inference is the posterior distribution. All Bayesian estimates are computed as some of its central statistics (e.g. the mean, the median or the mode), for instance

   * Maximum A Posteriori (MAP) estimate: $\qquad{\widehat{u}}_{\text{MAP}} = \arg\max_u p_{U \mid O}(u \mid o)$
   * Minimum Mean Square Error (MSE) estimate: $\qquad\widehat{u}_{\text{MSE}} = \mathbb{E}\{U \mid O=o\}$

The choice between the MAP or the MSE estimate may depend on practical or computational considerations. From a theoretical point of view, $\widehat{u}_{\text{MSE}}$ has some nice properties: it minimizes $\mathbb{E}\{(U-\widehat{u})^2\}$ among all possible estimates, $\widehat{u}$, so it is a natural choice. However, it involves the computation of an integral, which may not have a closed-form solution. In such cases, the MAP estimate can be a better choice.

The prior and the likelihood function are auxiliary distributions: if the posterior distribution is unknown, it can be computed from them using the Bayes rule:
\begin{equation}
p_{U|O}(u \mid o) = \frac{p_{O|U}(o \mid u) \cdot p_{U}(u)}{p_{O}(o)}
\end{equation}

In the next two sections we show that the Bayesian approach can be applied to both the prediction and the estimation problems.

### 2.2. Bayesian prediction under a known model

Assuming that the model parameters ${\bf w}$ are known, we can apply the Bayesian approach to predict ${\bf s}$ for an input ${\bf x}$. In that case, we can take

   * Unknown variable: ${\bf s}$, and
   * Observations: ${\bf x}$

the MAP and MSE predictions become

* Maximum A Posterior (MAP): $\qquad\widehat{s}_{\text{MAP}} = \arg\max_s p(s| {\bf x}, {\bf w})$
* Minimum Mean Square Error (MSE): $\qquad\widehat{s}_{\text{MSE}} = \mathbb{E}\{S |{\bf x}, {\bf w}\}$

#### Exercise 1:

Assuming
$$
p(s\mid x, w) = \frac{s}{w x^2} \exp\left({-\frac{s^2}{2 w x^2}}\right), \qquad s \geq 0,
$$
compute the MAP and MSE predictions of $s$ given $x$ and $w$.

#### Solution:
<SOL>
\begin{align}
\widehat{s}_\text{MAP}
   &= \arg\min_s \left\{\frac{s}{w x^2} \exp\left({-\frac{s^2}{2 w x^2}}\right) \right\} \\
   &= \arg\min_s \left\{\log(s) - \log(w x^2) -\frac{s^2}{2 w x^2} \right\}     \\
   &= \sqrt{w}x
\end{align}
where the last step results from minimizing by differentiation.

\begin{align}
\widehat{s}_\text{MSE}
   &= \mathbb{E}\{s | x, w\}   \\
   &= \int_0^\infty \frac{s^2}{w x^2} \exp\left({-\frac{s^2}{2 w x^2}}\right)   \\
   &= \frac{1}{2} \int_{-\infty}^\infty \frac{s^2}{w x^2} \exp\left({-\frac{s^2}{2 w x^2}}\right)   \\
   &= \frac{\sqrt{2\pi}}{2\sqrt{w x^2}} \int_{-\infty}^\infty \frac{s^2}{\sqrt{2\pi w x^2}} \exp\left({-\frac{s^2}{2 w x^2}}\right)
\end{align}

Noting that the last integral corresponds to the variance of a Gaussian distribution, we get
\begin{align}
\widehat{s}_\text{MSE}
   &= \frac{\sqrt{2\pi}}{2\sqrt{w x^2}} w x^2   \\
   &= \sqrt{\frac{\pi w}{2}}x
\end{align}
</SOL>

#### 2.2.1. The Gaussian case

A particularly interesting case arises when the data model is Gaussian:

$$p(s|{\bf x}, {\bf w}) =
    \frac{1}{\sqrt{2\pi}\sigma_\varepsilon}
    \exp\left(-\frac{(s-{\bf w}^\top{\bf z})^2}{2\sigma_\varepsilon^2}\right)
$$

where ${\bf z}=T({\bf x})$ is a vector with components which can be computed directly from the observed variables. For a Gaussian distribution (and for any unimodal symetric distributions) the mean and the mode are the same and, thus,

$$
\widehat{s}_\text{MAP} = \widehat{s}_\text{MSE} = {\bf w}^\top{\bf z}
$$

Such expression includes a linear regression model, where ${\bf z} = [1; {\bf x}]$, as well as any other non-linear model as long as it can be expressed as a "linear in the parameters" model.

### 2.3. Bayesian Inference for Parameter Estimation

In a similar way, we can apply Bayesian inference to estimate the model parameters ${\bf w}$ from a given dataset, $\cal{D}$. In that case

   * the unknown variable is ${\bf w}$, and
   * the observation is $\cal{D} \equiv \{{\bf X}, {\bf s}\}$

so that

* Maximum A Posterior (MAP): $\qquad\widehat{\bf w}_{\text{MAP}} = \arg\max_{\bf w} p({\bf w}| {\cal D})$
* Minimum Mean Square Error (MSE): $\qquad\widehat{\bf w}_{\text{MSE}} = \mathbb{E}\{{\bf W} | {\cal D}\}$

## 3. Bayesian parameter estimation

NOTE: Since the training data inputs are known, all probability density functions and expectations in the remainder of this notebook will be conditioned on the data matrix, ${\bf X}$. To simplify the mathematical notation, from now on we will remove ${\bf X}$ from all conditions. For instance, we will write  $p({\bf s}|{\bf w})$ instead of  $p({\bf s}|{\bf w}, {\bf X})$, etc. Keep in mind that, in any case, all probabilities and expectations may depend on ${\bf X}$ implicitely.

Summarizing, the steps to design a Bayesian parametric regresion algorithm are the following:

1. Assume a parametric data model $p(s| {\bf x},{\bf w})$ and a prior distribution $p({\bf w})$.
2. Using the data model and the i.i.d. assumption, compute $p({\bf s}|{\bf w})$.
3. Applying the bayes rule, compute the posterior distribution $p({\bf w}|{\bf s})$.
4. Compute the MAP or the MSE estimate of ${\bf w}$ given ${\bf x}$.
5. Compute predictions using the selected estimate.

### 3.1. Bayesian Inference and Maximum Likelihood.

Applying the Bayes rule the MAP estimate can be alternatively expressed as
\begin{align}
\qquad\widehat{\bf w}_{\text{MAP}}
   &= \arg\max_{\bf w} \frac{p({\cal D}| {\bf w}) \cdot p({\bf w})}{p({\cal D})} \\
   &= \arg\max_{\bf w} p({\cal D}| {\bf w}) \cdot p({\bf w})
\end{align}

By comparisons, the ML (Maximum Likelihood) estimate has the form:
$$
\widehat{\bf w}_{\text{ML}} = \arg \max_{\bf w} p(\mathcal{D}|{\bf w})
$$

This shows that the MAP estimate takes into account the prior distribution on the unknown parameter.

Another advantage of the Bayesian approach is that it provides not only a point estimate of the unknown parameter, but a whole funtion, the posterior distribution, which encompasses our belief on the unknown parameter given the data. For instance, we can take second order statistics like the variance of the posterior distributions to measure the uncertainty on the true value of the parameter around the mean.

### 3.2. The prior distribution

Since each value of ${\bf w}$ determines a regression function, by stating a prior distribution over the weights we state also a prior distribution over the space of regression functions.

For instance, assume that the data likelihood follows the Gaussian model in sec. 2.2.1, with $T(x) = (1, x, x^2, x^3)$, i.e. the regression functions have the form
$$
w_0 + w_1 x + w_2 x^2 + w_3 x^3
$$

Each value of ${\bf w}$ determines a specific polynomial of degree 3. Thus, the prior distribution over ${\bf w}$ describes which polynomials are more likely before observing the data.

For instance, assume a Gaussian prior with zero mean and variance ${\bf V}_p$, i.e.,
$$
p({\bf w}) = \frac{1}{(2\pi)^{D/2} |{\bf V}_p|^{1/2}}
              \exp \left(-\frac{1}{2} {\bf w}^\intercal {\bf V}_{p}^{-1}{\bf w} \right)
$$
where $D$ is the dimension of ${\bf w}$. To abbreviate, we will also express this as
$${\bf w} \sim {\cal N}\left({\bf 0},{\bf V}_{p} \right)$$
The following code samples ${\bf w}$ according to this distribution for ${\bf V}_p = 0.002 \, {\bf I}$, and plots the resulting polynomial over the scatter plot of an arbitrary dataset.

You can check the effect of modifying the variance of the prior distribution.

```python
n_grid = 200
degree = 3
nplots = 20

# Prior distribution parameters
mean_w = np.zeros((degree+1,))
v_p = 0.002     ### Try increasing this value
var_w = v_p * np.eye(degree+1)

xmin = -1
xmax = 1
X_grid = np.linspace(xmin, xmax, n_grid)

fig = plt.figure()
ax = fig.add_subplot(111)

for k in range(nplots):

    #Draw weigths fromt the prior distribution
    w_iter = np.random.multivariate_normal(mean_w, var_w)
    S_grid_iter = np.polyval(w_iter, X_grid)
    ax.plot(X_grid, S_grid_iter,'g-')

ax.set_xlim(xmin, xmax)
ax.set_ylim(-1, 1)
ax.set_xlabel('$x$')
ax.set_ylabel('$s$')
plt.show()
```

The data observation will modify our belief about the true data model according to the posterior distribution. In the following we will analyze this in a Gaussian case.

## 4. Bayesian regression for a Gaussian model.

We will apply the above steps to derive a Bayesian regression algorithm for a Gaussian model.

### 4.1. Step 1: The Gaussian model.

Let as assume that the likelihood function is given by the Gaussian model described in Sec. 1.3.2.

$$
s~|~{\bf w} \sim {\cal N}\left({\bf z}^\top{\bf w}, \sigma_\varepsilon^2 \right)
$$

that is
$$p(s|{\bf x}, {\bf w}) =
    \frac{1}{\sqrt{2\pi}\sigma_\varepsilon}
    \exp\left(-\frac{(s-{\bf w}^\top{\bf z})^2}{2\sigma_\varepsilon^2}\right)
$$

Assume, also, that the prior is Gaussian

$$
{\bf w} \sim {\cal N}\left({\bf 0},{\bf V}_{p} \right)
$$

### 4.2. Step 2: Complete data likelihood

Using the assumptions A1, A2 and A3, it can be shown that

$$
{\bf s}~|~{\bf w} \sim {\cal N}\left({\bf Z}{\bf w},\sigma_\varepsilon^2 {\bf I} \right)
$$
that is
$$
p({\bf s}| {\bf w})
    = \frac{1}{\left(\sqrt{2\pi}\sigma_\varepsilon\right)^K}
      \exp\left(-\frac{1}{2\sigma_\varepsilon^2}\|{\bf s}-{\bf Z}{\bf w}\|^2\right)
$$

### 4.3. Step 3: Posterior weight distribution

The posterior distribution of the weights can be computed using the Bayes rule

$$p({\bf w}|{\bf s}) = \frac{p({\bf s}|{\bf w})~p({\bf w})}{p({\bf s})}$$

Since both $p({\bf s}|{\bf w})$ and $p({\bf w})$ follow a Gaussian distribution, we know also that the joint distribution and the posterior distribution of ${\bf w}$ given ${\bf s}$ are also Gaussian. Therefore,

$${\bf w}~|~{\bf s} \sim {\cal N}\left({\bf w}_\text{MSE}, {\bf V}_{\bf w}\right)$$

After some algebra, it can be shown that mean and the covariance matrix of the distribution are:

$${\bf V}_{\bf w} = \left[\frac{1}{\sigma_\varepsilon^2} {\bf Z}^{\top}{\bf Z}
                  + {\bf V}_p^{-1}\right]^{-1}$$

$${\bf w}_\text{MSE} = {\sigma_\varepsilon^{-2}} {\bf V}_{\bf w} {\bf Z}^\top {\bf s}$$

#### Exercise 2:

Consider the dataset with one-dimensional inputs given by

```python
# True data parameters
w_true = 3
std_n = 0.4

# Generate the whole dataset
n_max = 64
X_tr = 3 * np.random.random((n_max,1)) - 0.5
S_tr =  w_true * X_tr + std_n * np.random.randn(n_max,1)

# Plot data
plt.figure()
plt.plot(X_tr, S_tr, 'b.')
plt.xlabel('$x$')
plt.ylabel('$s$')
plt.show()
```

Fit a Bayesian linear regression model assuming ${\bf z}={\bf x}$ and

```python
# Model parameters
sigma_eps = 0.4
mean_w = np.zeros((1,))
sigma_p = 1e6
Var_p = sigma_p**2* np.eye(1)
```

To do so, compute the posterior weight distribution using the first $k$ samples in the complete dataset, for $k = 1,2,4,8,\ldots 128$. Draw all these posteriors along with the prior distribution in the same plot.

```python
# No. of points to analyze
n_points = [1, 2, 4, 8, 16, 32, 64]

# Prepare plots
w_grid = np.linspace(2.7, 3.4, 5000)   # Sample the w axis
plt.figure()

# Compute the prior distribution over the grid points in w_grid
# p = <FILL IN>
p = 1.0/(sigma_p*np.sqrt(2*np.pi)) * np.exp(-(w_grid**2)/(2*sigma_p**2))
plt.plot(w_grid, p,'g-')

for k in n_points:

    # Select the first k samples
    Zk = X_tr[0:k, :]
    Sk = S_tr[0:k]

    # Parameters of the posterior distribution
    # 1. Compute the posterior variance.
    #    (Make sure that the resulting variable, Var_w, is a 1x1 numpy array.)
    # Var_w = <FILL IN>
    Var_w = np.linalg.inv(np.dot(Zk.T, Zk)/(sigma_eps**2) + np.linalg.inv(Var_p))

    # 2. Compute the posterior mean.
    #    (Make sure that the resulting variable, w_MSE, is a scalar)
    # w_MSE = <FILL IN>
    w_MSE = (Var_w.dot(Zk.T).dot(Sk)/(sigma_eps**2)).flatten()

    # Compute the posterior distribution over the grid points in w_grid
    sigma_w = np.sqrt(Var_w.flatten())     # First we take a scalar standard deviation
    # p = <FILL IN>
    p = 1.0/(sigma_w*np.sqrt(2*np.pi)) * np.exp(-((w_grid-w_MSE)**2)/(2*sigma_w**2))

    plt.plot(w_grid, p,'g-')
    plt.fill_between(w_grid, 0, p, alpha=0.8, edgecolor='#1B2ACC', facecolor='#089FFF',
        linewidth=1, antialiased=True)
    plt.title('Posterior distribution after {} samples'.format(k))
    plt.xlim(w_grid[0], w_grid[-1])
    plt.ylim(0, np.max(p))
    plt.xlabel('$w$')
    plt.ylabel('$p(w|s)$')

    display.clear_output(wait=True)
    display.display(plt.gcf())
    time.sleep(2.0)

# Remove the temporary plots and fix the last one
display.clear_output(wait=True)
plt.show()
```

#### Exercise 3:

Note that, in the example above, the model assumptions are correct: the target variables have been generated by a linear model with noise standard deviation `sigma_n` which is exactly equal to the value assumed by the model, stored in variable `sigma_eps`. Check what happens if we take `sigma_eps=4*sigma_n` or `sigma_eps=sigma_n/4`.

* Does the algorithm fail in that cases?
* What differences can you observe with respect to the ideal case `sigma_eps=sigma_n`?

### 4.4. Step 4: Weight estimation.

Since the posterior weight distribution is Gaussian, both the MAP and the MSE estimates are equal to the posterior mean, which has been already computed in step 3:

$$\widehat{\bf w}_\text{MAP} = \widehat{\bf w}_\text{MSE} = {\sigma_\varepsilon^{-2}} {\bf V}_{\bf w} {\bf Z}^\top {\bf s}$$

### 4.5. Step 5: Prediction

Using the MSE estimate, the final predictions are given by
$$
\widehat{s}_\text{MSE} =  \widehat{\bf w}_\text{MSE}^\top{\bf z}
$$

#### Exercise 4:

Plot the minimum MSE predictions of $s$ for inputs $x$ in the interval [-1, 3].

```python
# <SOL>
x = np.array([-1.0, 3.0])
s_pred = w_MSE * x

plt.figure()
plt.plot(X_tr, S_tr,'b.')
plt.plot(x, s_pred)
plt.show()
# </SOL>
```

## 5. Maximum likelihood vs Bayesian Inference.

### 5.1. The Maximum Likelihood Estimate.

For comparative purposes, it is interesting to see here that the likelihood function is enough to compute the Maximum Likelihood (ML) estimate

\begin{align}
{\bf w}_\text{ML} &= \arg \max_{\bf w} p(\mathcal{D}|{\bf w}) \\
             &= \arg \min_{\bf w} \|{\bf s}-{\bf Z}{\bf w}\|^2
\end{align}
which leads to the Least Squares (LS) solution
$$
{\bf w}_\text{ML} = ({\bf Z}^\top{\bf Z})^{-1}{\bf Z}^\top{\bf s}
$$

ML estimation is prone to overfiting. In general, if the number of parameters (i.e. the dimension of ${\bf w}$) is large in relation to the size of the training data, the predictor based on the ML estimate may have a small square error over the training set but a large error over the test set. Therefore, in practice, some cross validation procedure is required to keep the complexity of the predictor function under control depending on the size of the training set.

By defining a prior distribution over the unknown parameters, and using the Bayesian inference methods, the overfitting problems can be alleviated

### 5.2 Making predictions

   - Following an ML approach, we retain a single model, ${\bf w}_{ML} = \arg \max_{\bf w} p({\bf s}|{\bf w})$. Then, the predictive distribution of the target value for a new point would be obtained as:

   $$p({s^*}|{\bf w}_{ML},{\bf x}^*) $$

   For the generative model of Section 3.1.2 (additive i.i.d. Gaussian noise), this distribution is:

   $$p({s^*}|{\bf w}_{ML},{\bf x}^*) = \frac{1}{\sqrt{2\pi\sigma_\varepsilon^2}} \exp \left(-\frac{\left(s^* - {\bf w}_{ML}^\top {\bf z}^*\right)^2}{2 \sigma_\varepsilon^2} \right)$$

      * The mean of $s^*$ is just the same as the prediction of the LS model, and the same uncertainty is assumed independently of the observation vector (i.e., the variance of the noise of the model).

      * If a single value is to be kept, we would probably keep the mean of the distribution, which is equivalent to the LS prediction.

   - Using Bayesian inference, we retain all models. Then, the inference of the value $s^* = s({\bf x}^*)$ is carried out by mixing all models, according to the weights given by the posterior distribution.

   \begin{align}p({s^*}|{\bf x}^*,{\bf s})
                         & = \int p({s^*}~|~{\bf w},{\bf x}^*) p({\bf w}~|~{\bf s}) d{\bf w}\end{align}

   where:

      * $p({s^*}|{\bf w},{\bf x}^*) = \displaystyle\frac{1}{\sqrt{2\pi\sigma_\varepsilon^2}} \exp \left(-\frac{\left(s^* - {\bf w}^\top {\bf z}^*\right)^2}{2 \sigma_\varepsilon^2} \right)$
      * $p({\bf w}~|~{\bf s})$: Is the posterior distribution of the weights, that can be computed using Bayes' Theorem.

In general the integral expression of the posterior distribution $p({s^*}|{\bf x}^*,{\bf s})$ cannot be computed analytically. Fortunately, for the Gaussian model, the computation of the posterior is simple, as we will show in the following section.

## 6. Posterior distribution of the target variable

In the same way that we have computed a distribution on ${\bf w}$, we can compute a distribution on the target variable for a given input ${\bf x}$ and given the whole dataset.

Since ${\bf w}$ is a random variable, the noise-free component of the target variable for an arbitrary input ${\bf x}$, that is, $f = f({\bf x}) = {\bf w}^\top{\bf z}$ is also a random variable, and we can compute its distribution from the posterior distribution of ${\bf w}$

Since ${\bf w}$ is Gaussian and $f$ is a linear transformation of ${\bf w}$, $f$ is also a Gaussian random variable, whose posterior mean and variance can be calculated as follows:

\begin{align}
\mathbb{E}\{f \mid {\bf s}, {\bf z}\}
   &= \mathbb{E}\{{\bf w}^\top {\bf z}~|~{\bf s}, {\bf z}\}
    = \mathbb{E}\{{\bf w} ~|~{\bf s}, {\bf z}\}^\top  {\bf z}    \\
   &=  \widehat{\bf w}_\text{MSE} ^\top {\bf z}     \\
%   &=  {\sigma_\varepsilon^{-2}} {{\bf z}}^\top {\bf V}_{\bf w} {\bf Z}^\top {\bf s}
\end{align}

\begin{align}
\text{Cov}\left[{{\bf z}}^\top {\bf w}~|~{\bf s}, {\bf z}\right]
   &= {\bf z}^\top \text{Cov}\left[{\bf w}~|~{\bf s}\right] {\bf z}    \\
   &= {\bf z}^\top {\bf V}_{\bf w} {{\bf z}}
\end{align}

Therefore,
$$
f^*~|~{\bf s}, {\bf x}
   \sim {\cal N}\left(\widehat{\bf w}_\text{MSE} ^\top {\bf z}, ~~
                      {\bf z}^\top {\bf V}_{\bf w} {\bf z} \right)
$$

Finally, for $s = f + \varepsilon$, the posterior distribution is
$$
s ~|~{\bf s}, {\bf z}^*
   \sim {\cal N}\left(\widehat{\bf w}_\text{MSE} ^\top {\bf z}, ~~
                      {\bf z}^\top {\bf V}_{\bf w} {\bf z} + \sigma_\varepsilon^2\right)
$$

#### Example:

The next figure shows a one-dimensional dataset with 15 points, which are noisy samples from a cosine signal (shown in the dotted curve)

```python
n_points = 15
n_grid = 200
frec = 3
std_n = 0.2

# Data generation
X_tr = 3 * np.random.random((n_points,1)) - 0.5
S_tr = - np.cos(frec*X_tr) + std_n * np.random.randn(n_points,1)

# Signal
xmin = np.min(X_tr) - 0.1
xmax = np.max(X_tr) + 0.1
X_grid = np.linspace(xmin, xmax, n_grid)
S_grid = - np.cos(frec*X_grid) #Noise free for the true model

# Compute matrix with training input data for the polynomial model
Z = []
for x_val in X_tr.tolist():
    Z.append([x_val[0]**k for k in range(degree+1)])
Z = np.asmatrix(Z)

# Plot data
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(X_tr,S_tr,'b.',markersize=10)

# Plot noise-free function
ax.plot(X_grid, S_grid, 'b:', label='Noise-free signal')

# Set axes
ax.set_xlim(xmin, xmax)
ax.set_ylim(S_tr[0] - 2, S_tr[-1] + 2)
ax.legend(loc='best')
plt.show()
```

Let us asssume that the cosine form of the noise-free signal is unknown, and we assume a polynomial model with a high degree. The following code plots the LS estimate

```python
degree = 12

# We plot also the least square solution
w_LS = np.polyfit(X_tr.flatten(), S_tr.flatten(), degree)
S_grid_LS = np.polyval(w_LS,X_grid)

# Plot data
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(X_tr,S_tr,'b.',markersize=10)

# Plot noise-free function
ax.plot(X_grid, S_grid, 'b:', label='Noise-free signal')
# Plot LS regression function
ax.plot(X_grid, S_grid_LS, 'm-', label='LS regression')

# Set axis
ax.set_xlim(xmin, xmax)
ax.set_ylim(S_tr[0] - 2, S_tr[-1] + 2)
ax.legend(loc='best')
plt.show()
```

The following fragment of code computes the posterior weight distribution, draws random vectors from $p({\bf w}|{\bf s})$, and plots the corresponding regression curves along with the training points. Compare these curves with those extracted from the prior distribution of ${\bf w}$ and with the LS solution.

```python
nplots = 6

# Prior distribution parameters
sigma_eps = 0.2
mean_w = np.zeros((degree+1,))
sigma_p = .5
Var_p = sigma_p**2 * np.eye(degree+1)

# Compute matrix with training input data for the polynomial model
Z = []
for x_val in X_tr.tolist():
    Z.append([x_val[0]**k for k in range(degree+1)])
Z = np.asmatrix(Z)

#Compute posterior distribution parameters
Var_w = np.linalg.inv(np.dot(Z.T,Z)/(sigma_eps**2) + np.linalg.inv(Var_p))
posterior_mean = Var_w.dot(Z.T).dot(S_tr)/(sigma_eps**2)
posterior_mean = np.array(posterior_mean).flatten()

# Plot data
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(X_tr,S_tr,'b.',markersize=10)

# Plot noise-free function
ax.plot(X_grid, S_grid, 'b:', label='Noise-free signal')
# Plot LS regression function
ax.plot(X_grid, S_grid_LS, 'm-', label='LS regression')

for k in range(nplots):

    # Draw weights from the posterior distribution
    w_iter = np.random.multivariate_normal(posterior_mean, Var_w)

    # Note that polyval assumes the first element of weight vector is the coefficient of
    # the highest degree term. Thus, we need to reverse w_iter
    S_grid_iter = np.polyval(w_iter[::-1], X_grid)
    ax.plot(X_grid,S_grid_iter,'g-')

# Set axis
ax.set_xlim(xmin, xmax)
ax.set_ylim(S_tr[0] - 2, S_tr[-1] + 2)
ax.legend(loc='best')
plt.show()
```

Not only do we obtain a better predictive model, but we also have confidence intervals (error bars) for the predictions.

```python
# Compute standard deviation
std_x = []
for el in X_grid:
    x_ast = np.array([el**k for k in range(degree+1)])
    std_x.append(np.sqrt(x_ast.dot(Var_w).dot(x_ast)[0,0]))
std_x = np.array(std_x)

# Plot data
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(X_tr,S_tr,'b.',markersize=10)

# Plot the posterior mean
# Note that polyval assumes the first element of weight vector is the coefficient of
# the highest degree term. Thus, we need to reverse w_iter
S_grid_iter = np.polyval(posterior_mean[::-1],X_grid)
ax.plot(X_grid,S_grid_iter,'g-',label='Predictive mean, BI')

#Plot confidence intervals for the Bayesian Inference
plt.fill_between(X_grid, S_grid_iter-std_x, S_grid_iter+std_x,
    alpha=0.4, edgecolor='#1B2ACC', facecolor='#089FFF',
    linewidth=2, antialiased=True)

#We plot also the least square solution
w_LS = np.polyfit(X_tr.flatten(), S_tr.flatten(), degree)
S_grid_iter = np.polyval(w_LS,X_grid)

# Plot noise-free function
ax.plot(X_grid, S_grid, 'b:', label='Noise-free signal')
# Plot LS regression function
ax.plot(X_grid, S_grid_LS, 'm-', label='LS regression')

# Set axis
ax.set_xlim(xmin, xmax)
ax.set_ylim(S_tr[0]-2,S_tr[-1]+2)
ax.legend(loc='best')
plt.show()
```

#### Exercise 5:

Assume the dataset ${\cal{D}} = \left\{ x^{(k)}, s^{(k)}\right\}_{k=0}^{K-1}$ containing $K$ i.i.d. samples from a distribution

$\qquad p(s|x,w) = w x \exp(-w x s), \; s>0, x> 0, w> 0$

We model also our uncertainty about the value of $w$ assuming a prior distribution for $w$ following a Gamma distribution with parameters $\alpha>0$ and $\beta>0$.

$ \qquad
w \sim \text{Gamma}\left(\alpha, \beta\right) = \frac{\beta^\alpha}{\Gamma(\alpha)} w^{\alpha-1} \exp\left(-\beta w\right), \; w>0
$

Note that the mean and the mode of a Gamma distribution can be calculated in closed-form as
$\qquad w_\text{MSE} = \mathbb{E}\left\{w\right\}=\frac{\alpha}{\beta}; \qquad
w_\text{MAP} = \max_w p(w) = \frac{\alpha-1}{\beta}$

**1.** Determine an expression for the likelihood function.

```python

```

**2.** Determine the maximum likelihood coefficient,  $\widehat{w}_{\text{ML}}$.

```python

```

**3.** Obtain the posterior distribution $p(w|{\bf s})$. Note that you do not need to calculate $p({\bf s})$ since the posterior distribution can be readily identified as another Gamma distribution.

```python

```

**4.** Determine the MSE and MAP a posteriori estimators of $w$: $w_\text{MSE}=\mathbb{E}\left\{w|{\bf s}\right\}$ and $w_\text{MAP} = \max_w p(w|{\bf s})$.

```python

```

**5.** Compute the following estimators of $S$:

$\qquad\widehat{s}_1 = \mathbb{E}\{s|w_\text{ML},x\}$

$\qquad\widehat{s}_2 = \mathbb{E}\{s|w_\text{MSE},x\}$

$\qquad\widehat{s}_3 = \mathbb{E}\{s|w_\text{MAP},x\}$

```python

```

## 7. Maximum evidence model selection

We have already addressed with Bayesian Inference the following two issues:

   - For a given degree, how do we choose the weights?

   - Should we focus on just one model, or can we use several models at once?

However, we still needed some assumptions: a parametric model (i.e., polynomial function and a priori degree selection) and several parameters needed to be adjusted.

Though we can recur to cross-validation, Bayesian inference opens the door to other strategies.

   - We could argue that rather than keeping single selections of these parameters, we could use simultaneously several sets of parameters (and/or several parametric forms), and average them in a probabilistic way ... (like we did with the models)

   - We will follow a simpler strategy, selecting just the most likely set of parameters according to an ML criterion

### 7.1 Model evidence

The evidence of a model is defined as

$$L = p({\bf s}~|~{\cal M})$$

where ${\cal M}$ denotes the model itself and any free parameters it may have. For instance, for the polynomial model we have assumed so far, ${\cal M}$ would represent the degree of the polynomia, the variance of the additive noise, and the a priori covariance matrix of the weights

Applying the Theorem of Total probability, we can compute the evidence of the model as

$$L = \int p({\bf s}~|~{\bf f},{\cal M}) p({\bf f}~|~{\cal M}) d{\bf f} $$

For the linear model $f({\bf x}) = {\bf w}^\top{\bf z}$, the evidence can be computed as

$$L = \int p({\bf s}~|~{\bf w},{\cal M}) p({\bf w}~|~{\cal M}) d{\bf w} $$

It is important to notice that these probability density functions are exactly the ones we computed on the previous section. We are just making explicit that they depend on a particular model and the selection of its parameters. Therefore:

   - $p({\bf s}~|~{\bf w},{\cal M})$ is the likelihood of ${\bf w}$

   - $p({\bf w}~|~{\cal M})$ is the a priori distribution of the weights

### 7.2 Model selection via evidence maximization

   - As we have already mentioned, we could propose a prior distribution for the model parameters, $p({\cal M})$, and use it to infer the posterior. However, this can be very involved (usually no closed-form expressions can be derived)

   - Alternatively, maximizing the evidence is normally good enough

   $${\cal M}_\text{ML} = \arg\max_{\cal M} p(s~|~{\cal M})$$

Note that we are using the subscript 'ML' because the evidence can also be referred to as the likelihood of the model

### 7.3 Example: Selection of the degree of the polynomia

For the previous example we had (we consider a spherical Gaussian for the weights):

   - ${\bf s}~|~{\bf w},{\cal M}~\sim~{\cal N}\left({\bf Z}{\bf w},~\sigma_\varepsilon^2 {\bf I} \right)$

   - ${\bf w}~|~{\cal M}~\sim~{\cal N}\left({\bf 0},~\sigma_p^2 {\bf I} \right)$

In this case, $p({\bf s}~|~{\cal M})$ follows also a Gaussian distribution, and it can be shown that

   - $L = p({\bf s}~|~{\cal M}) = {\cal N}\left({\bf 0},\sigma_p^2 {\bf Z} {\bf Z}^\top+\sigma_\varepsilon^2 {\bf I} \right)$

If we just pursue the maximization of $L$, this is equivalent to maximizing the log of the evidence

$$\log(L) = -\frac{M}{2} \log(2\pi) -{\frac{1}{2}}\log\mid\sigma_p^2 {\bf Z} {\bf Z}^\top+\sigma_\varepsilon^2 {\bf I}\mid - \frac{1}{2} {\bf s}^\top \left(\sigma_p^2 {\bf Z} {\bf Z}^\top+\sigma_\varepsilon^2 {\bf I}\right)^{-1} {\bf s}$$

where $M$ denotes the length of vector ${\bf z}$ (the degree of the polynomia minus 1).

The following fragment of code evaluates the evidence of the model as a function of the degree of the polynomia

```python
from math import pi

n_points = 15
frec = 3
std_n = 0.2
max_degree = 12

#Prior distribution parameters
sigma_eps = 0.2
mean_w = np.zeros((degree+1,))
sigma_p = 0.5

X_tr = 3 * np.random.random((n_points,1)) - 0.5
S_tr = - np.cos(frec*X_tr) + std_n * np.random.randn(n_points,1)

#Compute matrix with training input data for the polynomial model
Z = []
for x_val in X_tr.tolist():
    Z.append([x_val[0]**k for k in range(degree+1)])
Z=np.asmatrix(Z)

#Evaluate the posterior evidence

logE = []
for deg in range(max_degree):
    Z_iter = Z[:,:deg+1]
    logE_iter = -((deg+1)*np.log(2*pi)/2) \
                -np.log(np.linalg.det((sigma_p**2)*Z_iter.dot(Z_iter.T) + (sigma_eps**2)*np.eye(n_points)))/2 \
                -S_tr.T.dot(np.linalg.inv((sigma_p**2)*Z_iter.dot(Z_iter.T) + (sigma_eps**2)*np.eye(n_points))).dot(S_tr)/2
    logE.append(logE_iter[0,0])

plt.plot(np.array(range(max_degree))+1,logE)
plt.xlabel('Polynomia degree')
plt.ylabel('log evidence')
plt.show()

```

The above curve may change the position of its maximum from run to run.

We conclude the notebook by plotting the result of the Bayesian inference for M=6

```python
n_points = 15
n_grid = 200
frec = 3
std_n = 0.2
degree = 5 #M-1
nplots = 6

#Prior distribution parameters
sigma_eps = 0.1
mean_w = np.zeros((degree+1,))
sigma_p = .5 * np.eye(degree+1)

X_tr = 3 * np.random.random((n_points,1)) - 0.5
S_tr = - np.cos(frec*X_tr) + std_n * np.random.randn(n_points,1)
X_grid = np.linspace(-1,3,n_grid)
S_grid = - np.cos(frec*X_grid) #Noise free for the true model

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(X_tr,S_tr,'b.',markersize=10)

#Compute matrix with training input data for the polynomial model
Z = []
for x_val in X_tr.tolist():
    Z.append([x_val[0]**k for k in range(degree+1)])
Z=np.asmatrix(Z)

#Compute posterior distribution parameters
Sigma_w = np.linalg.inv(np.dot(Z.T,Z)/(sigma_eps**2) + np.linalg.inv(sigma_p))
posterior_mean = Sigma_w.dot(Z.T).dot(S_tr)/(sigma_eps**2)
posterior_mean = np.array(posterior_mean).flatten()

#Plot the posterior mean
#Note that polyval assumes the first element of weight vector is the coefficient of
#the highest degree term. Thus, we need to reverse w_iter
S_grid_iter = np.polyval(posterior_mean[::-1],X_grid)
ax.plot(X_grid,S_grid_iter,'g-',label='Predictive mean, BI')

#Plot confidence intervals for the Bayesian Inference
std_x = []
for el in X_grid:
    x_ast = np.array([el**k for k in range(degree+1)])
    std_x.append(np.sqrt(x_ast.dot(Sigma_w).dot(x_ast)[0,0]))
std_x = np.array(std_x)
plt.fill_between(X_grid, S_grid_iter-std_x, S_grid_iter+std_x,
    alpha=0.2, edgecolor='#1B2ACC', facecolor='#089FFF',
    linewidth=4, linestyle='dashdot', antialiased=True)

#We plot also the least square solution
w_LS = np.polyfit(X_tr.flatten(), S_tr.flatten(), degree)
S_grid_iter = np.polyval(w_LS,X_grid)
ax.plot(X_grid,S_grid_iter,'m-',label='LS regression')

ax.set_xlim(-1,3)
ax.set_ylim(S_tr[0]-2,S_tr[-1]+2)
ax.legend(loc='best')
plt.show()
```

We can check, that now the model also seems quite appropriate for LS regression, but keep in mind that selection of such parameter was itself carried out using Bayesian inference.

```python

```


# Supplementary material - Importance sampling example notebook

This notebook performs importance sampling on an example subset of the Monte Carlo energy cone outputs from Clarke et al. (in prep.). To run this notebook, various input files and the filing structure provided in the GitHub repository are required. This notebook requires the following files:

- input_weights.csv <-- this needs to be generated using the accompanying 'Input_weight_notebook'
- vent_opening_probabilites.csv <-- this needs to be generated using the accompanying 'Vent_opening_probabilities_notebook'
- output files from the energy cone modelling <-- a small subset of these are provided as an example
- grid_x.csv, grid_y.csv <-- UTM values of columns and rows of desired output .tiff file. Provided in 'source_files' folder

NB. as only a small subset of the output data is provided as an example (due to online storage constraints), the results are not representative of the probabilities indicated in the Clarke et al. (in prep.) paper, and so should not be used as a PVHA.

### Features:
1. Imports output data from the Monte Carlo energy cone model
2. Uses dask to efficiently weight the model outputs based on the source vent opening probability, and weight factor ($W_k$) associated with each energy cone input parameter pair ($\phi$ and $H_0$)
3. Combines these individual processed results to produce a final 2D array showing $P(PDC|eruption)$
4. Displays the result as a simple raster
5. Georeferences the raster to a suitable UTM format, ready for import into GIS software

### Specifically, the equations for importance sampling and calculating the probability are as follows:

The probability of PDC inundation in grid cell $j$, given an eruption of vent $i$ ($P(PDC_j|vent_i)$), is calculated using the following equation:

\begin{equation}
    P(PDC_j|vent_i) = \frac{1}{S} \times \sum\limits_{k=1}^N (W_k \times s_k)
\end{equation}

where $S$ is the total number of simulations from vent $i$ (in this case, 10,000), $k$ is the identity of each simulation, and $s$ is a binary 1 or 0 within each grid cell noting inundation versus no inundation. Effectively, this counts the proportion of weighted simulations in which a PDC indundated cell $j$ from vent $i$. The final step is to also consider the probability of the eruption coming from each vent; this gives the probability of inundation in grid cell $j$ given an eruption ($P(PDC_j)$), by the following equation:

\begin{equation}
    P(PDC_j) = \sum\limits_{i=1}^N P(PDC_j|vent_i) \times P(vent_i)
\end{equation}

where $P(vent_i)$ is the probability of vent $i$ being the source of the next eruption. This equation is applied to every grid cell, producing a map of $P(PDC|eruption)$.

### Notebook Author:
Ben Clarke

https://github.com/GeoBen93

https://www.researchgate.net/profile/Ben_Clarke6

https://twitter.com/PyroclasticBen

### How this is done in practice:

```python
import numpy as np # for array operations
import h5py # for importing .mat HDF5 files
import matplotlib.pyplot as plt # plotting
from scipy.misc import imsave, imshow # viewing as image
import pandas as pd # for importing weight data
import dask.array as da
import dask.dataframe as dd
import dask
```

## Step 1 - Conduct Importance sampling

This function takes each energy cone output file, consisting of results from a sinlge PDC parameter pair ($\phi$ and $H_0$) at every vent in a 3D array, and performs importance sampling. Each 3D array is in the form: (x location : y location : source vent number), stored as a compressed .hdf5 file (with a .mat alias).

The function employs 'dask' to speed up processing times, which essentially breaks the array into chunks, does the necessary calculations for each chunk in parallel, and joins the chunks back together.

The function requires:
1. inputs <-- the index of the energy cone output result to process, in this case, an integer from 1 to 10,000
2. vw     <-- a pandas dataframe of vent opening probabilities, the file: 'vent_opening_probability.csv' in 'notebook_output_files'. Generated by the 'vent_opening_probability_notebook'
3. iw     <-- a pandas dataframe of input weight factors ($W_k$), the file: 'input_weights.csv' in 'notebook_output_files'. Generated by the 'Input_weight_notebook'

What the function does:
1. Imports the result array from the energy cone model
2. Uses the einsum function to multiply each 2D slice (shape (x location : y location) ) of the 3D array, by the corresponding vent opening probability
3. Uses the einsum function to sum the 3D along the vent axis, creating a 2D array of shape (x location : y location)
4. Multiplies this 2D array by the input weight ($W_k$) associated with the paricular pair of $\phi$ and $H_0$
5. Saves the processed data as a compressed 2D array as a .hdf5 file. Found in the 'notebook_output_files/processed1_outputs' folder

```python
def array_processing_stage1(inputs, vw, iw):
    ##### inputs = number of input to process(ie. 1-10,000)
    ##### vw = vent weighting in the form of pandas dataframe of vent weights for each input
    ##### iw = input weighting in the form of pandas dataframe of input weights for each input
    ##### weightcode = number to distinguish this particular weighting procedure, must change each time different weights or methods are applied, must be a corresponding output folder

    ######----------IMPORT DATA--------------##########

    input_idx = inputs-1 # index of the input is 1 less than the input number as python starts counting from 0

    filename = 'source_files/example_results/invasion_input' + str(inputs) + '.mat'

    # import that file as a dask array in 1200 chunks (this appears to be fastest chunk size)
    f = h5py.File(filename)
    data = da.from_array(f["results_array"], chunks=(1200, 1305, 1305))

    ######----------PROCESSING--------#############

    # use the einsum function to multiply the vw with the vents and the output is the sum of vent weighted elements
    with dask.config.set(scheduler='threads'):
        collapsed = da.einsum('ijk, i -> jk', data, vw['Prob'][:1200]).compute()

    #apply the input weighting... because this is the same for every element of a single array, we can perform this after collapsing the array, saving processing time
    input_weight = iw['input_weight'][input_idx] # find weighting for this array according to input index
    collapsed = collapsed * input_weight # multiply the elements by this weight

    collapsed = collapsed.T # transpose the lat:long so they are the correct way around

    ######----------EXPORT-------------############

    #save this in a compressed HDF5 file format (about 50-100kb for 2d array shape (1305:1305))
    savefilename = 'notebook_output_files/processed1_outputs/input_' +str(inputs)+ '_p.h5'

    with h5py.File(savefilename, 'w') as h5f:
        h5f.create_dataset('dataset1', data=collapsed, compression='gzip', compression_opts=9)

    ######---------PROGRESS CHECK-------############

    #Create some outputs to check progress
    print('..........................')
    print('Done Input' ,inputs)
```

The following script runs the 'array_processing_stage1' function for the desired range of energy cone output data, saved in the 'notebook_output_files/example_results' folder. We have provided an example dataset of 20 outputs (1001 to 1019), as a test case. Each output array takes on the order of 20 seconds to process. The results are saved in the folder named 'processed1_outputs'.

*NB. These 20 example files are not uniformly distributed in terms of $\phi$ and $H_c$ input parameters.*

To run this function, define a start_index and end_index, where each is an integer between 1001 and 1019.

```python
#define start and end file indexes
start_index = 1001
end_index = 1019

#load in the input weights as a dataframe
input_filename =  'notebook_output_files/input_weights.csv' # define input weight file
iw = dd.read_csv(input_filename).compute() # load input weights into a pandas dataframe

#load in the vent opening probabilities as a dataframe
vw = dd.read_csv('notebook_output_files/vent_opening_probabilities.csv').compute()

for inputs in range(start_index, (end_index+1)):
    array_processing_stage1(inputs, vw, iw)
```

    ..........................
    Done Input 1001
    ..........................
    Done Input 1002
    ..........................
    Done Input 1003
    ..........................
    Done Input 1004
    ..........................
    Done Input 1005
    ..........................
    Done Input 1006
    ..........................
    Done Input 1007
    ..........................
    Done Input 1008
    ..........................
    Done Input 1009
    ..........................
    Done Input 1010
    ..........................
    Done Input 1011
    ..........................
    Done Input 1012
    ..........................
    Done Input 1013
    ..........................
    Done Input 1014
    ..........................
    Done Input 1015
    ..........................
    Done Input 1016
    ..........................
    Done Input 1017
    ..........................
    Done Input 1018
    ..........................
    Done Input 1019

## Step 2 - Combine processed arrays to produce final result

The next function combines the arrays generated by the 'array_processing_stage1' into a single 2D array with the shape (x location: y location), and the value in each cell is P(PDC|eruption).

The function requires:
1. start_input <-- integer of the first array to be combined
2. end_input   <-- integer of the last array to be combined

What the function does:
1. Imports the processed1_output arrays in the defined range
2. Adds each to a new cumulative array named: 'final_result_starting_(start_input) ending (end_input).h5'
3. Divides the final array by the number of iterations
4. Saves the final result to a file named: 'final_result_starting_(start_input) ending (end_input).h5'

```python
def combine_arrays(start_input, end_input):

    def hdf5_to_array(filename):
        with h5py.File(filename) as h5f:
            f = h5py.File(filename)
            dataset_name = list(f.keys())
            result = np.array(f[dataset_name[0]])
        return result

    def array_to_hdf5(array):
        cumfilename = 'final_result_starting_' + str(start_input) + '_ending_' + str(end_input) + '.h5'
        with h5py.File(cumfilename, 'w') as h5f:
            h5f.create_dataset('dataset1', data=array, compression='gzip', compression_opts=9)
        return

    #initialise the cumulative array by saving the start array as the cumulative array
    start_file = 'notebook_output_files/processed1_outputs/input_' + str(start_input) + '_p.h5'
    start_array = hdf5_to_array(start_file) #import start array

    cumfilename = 'notebook_output_files/final_result_starting_' + str(start_input) + '_ending_' + str(end_input) + '.h5'

    # load the cumulative array
    cumulative_array = start_array

    for file in range(start_input, end_input+1):

        # find the filename of the next array to be added to the cumulative array
        next_file = 'notebook_output_files/processed1_outputs/input_' + str(file) + '_p.h5'

        #import this as an array
        next_array = hdf5_to_array(next_file)

        #add the arrays together
        cumulative_array = cumulative_array + next_array

        print('........................')
        print('Added Input ' + str(file))

    niterations = end_input - start_input # calcualte number of iterations the result is based on

    cumulative_array = cumulative_array / niterations
    cumulative_array = np.flip(cumulative_array, axis = 0) # flip the y axis so north is up (don't know why this changed...)
    array_to_hdf5(cumulative_array) #save to hdf5

    return cumulative_array
```

To run this function, simply execute the next cell. It requires the start and end indexes defined earlier. To adjust which files to combine, simply enter the start and end integer into the function arguments.

```python
result = combine_arrays(start_index,end_index)
```

    ........................
    Added Input 1001
    ........................
    Added Input 1002
    ........................
    Added Input 1003
    ........................
    Added Input 1004
    ........................
    Added Input 1005
    ........................
    Added Input 1006
    ........................
    Added Input 1007
    ........................
    Added Input 1008
    ........................
    Added Input 1009
    ........................
    Added Input 1010
    ........................
    Added Input 1011
    ........................
    Added Input 1012
    ........................
    Added Input 1013
    ........................
    Added Input 1014
    ........................
    Added Input 1015
    ........................
    Added Input 1016
    ........................
    Added Input 1017
    ........................
    Added Input 1018
    ........................
    Added Input 1019

Now lets have a quick look at what the results look like (non-georeferenced, in relative geographical space).

```python
plt.imshow(result)
plt.title('P(PDC|eruption) overview')
plt.xlabel('X cell indexes')
plt.ylabel('Y cell indexes')
plt.colorbar(label='P(PDC|eruption)')
plt.show()
```

## Step 3 - Georeference the results to be viewed in GIS software

This step takes the array generated in step 2 and georeferences it, so that it can be used in GIS software as a standard raster file for map creation or analysis. This method requires a georeferenced grid to be made, which requires a list of the columns and rows (UTM coordinates) that describe the cells that form the results array generated in step 2. These are the 'grid_x.csv' and 'grid_y.csv' provided in the weighting_data folder.

The following code is adapted from https://gis.stackexchange.com/questions/37238/writing-numpy-array-to-raster-file.

What this step does:
1. Imports the x and y UTM column and row values, and creates a grid
2. Transforms this grid into a 'GeoTiff' file of the correct dimensions and resolution, with the specified EPSG coordinate system code
3. Applies the values of the results grid from step 2 to this GeoTiff as a raster band
4. Saves the GeoTiff as: 'inundation_map.tif' in the 'notebook_output_files' folder, which is ready to import into a GIS, such as QGIS or ArcGIS

```python
from osgeo import gdal
from osgeo import gdal_array
from osgeo import osr
import csv

yname = 'source_files/grid_y.csv'
xname = 'source_files/grid_x.csv'

def csv_to_float_list(filename):
    with open(filename, 'r', encoding='utf8') as f:
        reader = csv.reader(f)
        values = list(reader)

    values = values[0]
    values[0] = values[0][1:]
    for i in range(0,len(values)):
        values[i] = float(values[i])

    return values

lat = np.flip(np.array(csv_to_float_list(yname)))
lon = np.array(csv_to_float_list(xname))
lonG, latG = np.meshgrid(lon,lat)
array = result

xmin,ymin,xmax,ymax = [lon.min(),lat.min(),lon.max(),lat.max()]
nrows,ncols = np.shape(result)
xres = (xmax-xmin)/float(ncols)
yres = (ymax-ymin)/float(nrows)
geotransform=(xmin,xres,0,ymax,0, -yres)
# That's (top left x, w-e pixel resolution, rotation (0 if North is up),
#         top left y, rotation (0 if North is up), n-s pixel resolution)

name = 'notebook_output_files/inundation_map.tif'

output_raster = gdal.GetDriverByName('GTiff').Create(name,ncols, nrows, 1 ,gdal.GDT_Float32)  # Open the file
output_raster.SetGeoTransform(geotransform)  # Specify its coordinates
srs = osr.SpatialReference()                 # Establish its coordinate encoding
srs.ImportFromEPSG(32637)                    # This one specifies coordinate system

output_raster.SetProjection( srs.ExportToWkt() )   # Exports the coordinate system to the file
output_raster.GetRasterBand(1).WriteArray(result)   # Writes my array to the raster

output_raster.FlushCache()
```


```python
#Import relevant packages
import numpy as np
import sympy as sm
from sympy import symbols, Eq, solve
from scipy.optimize import fsolve
import math
import matplotlib.pyplot as plt
```

```python
#For pretty printing
sm.init_printing(use_unicode=True)
```

```python
par = {}

par['alpha'] = 5.76
par['h'] = 0.5
par['b'] = 0.5
par['phi'] = 0
par['gamma'] = 0.075
```

```python
#Define symbols
v_t = sm.symbols('v_t')
v_lag = sm.symbols('v_t-1')
y_t = sm.symbols('y_t')
y_lag = sm.symbols('y_t-1')
pi_t = sm.symbols('pi_t')
pi_lag = sm.symbols('pi_t-1')
s_t = sm.symbols('s_t')
s_lag = sm.symbols('s_t-1')
alpha = sm.symbols('alpha')
h = sm.symbols('h')
b = sm.symbols('b')
phi = sm.symbols('phi')
gamma = sm.symbols('gamma')
x_t = sm.symbols('x_t')
c_t = sm.symbols('c_t')
```

```python
#Define the AD- and the SRAS-curve
AD = sm.Eq(pi_t, (1/(h*alpha))*(v_t-(1+b*alpha)*y_t))
SRAS = sm.Eq(pi_t, (pi_lag+gamma*y_t-phi*gamma*y_lag+s_t-phi*s_lag))
AD, SRAS
```

```python
#Solve equilibrium
EQ = sm.solve((AD, SRAS), (pi_t,y_t))

#Display result
EQ
```

```python
#Vi skal finde ud af, om vi også vil vise den numeriske løsning her.
```

```python
#Use previously given parameter values
alpha = par['alpha']
h = par['h']
b = par['b']
phi = par['phi']
gamma = par['gamma']
```

**Question 2:** Find and illustrate the equilibrium when $y_{t-1} = \pi_{t-1} = v_t = s_t = s_{t-1} = 0$. Illustrate how the equilibrium changes when instead $v_t = 0.1$.

```python
#Define new AD-curve and SRAS-curve according to the new values for the variables
_AD = sm.Eq(pi_t, ((-1/h*alpha)*((1+b*alpha*y_t)))
_SRAS = sm.Eq(pi_t, (gamma*y_t))
```

```python
#Solve the new equilibrium
_EQ = sm.solve((_AD, _SRAS), (pi_t, y_t))

#Display result
_EQ
```

```python
#Set v_t equal to 0.1
v_t = 0.1

#Define the AD-curve according to new value for the demand disturbance
newAD = sm.Eq(pi_t, ((1/h*alpha)*(v_t-(1+b*alpha)*y_t)))

#Solve the new equilibrium
newEQ = sm.solve((newAD, _SRAS), (pi_t, y_t))

#Display result
newEQ
```

```python
#Defining functions in order to display result graphically
def SRAS(y_t):
    return gamma*y_t

def AD(y_t):
    return (-1/h*alpha)*((1+b*alpha)*y_t)

def newAD(y_t):
    return (1/h*alpha)*(v_t-(1+b*alpha)*y_t)

# Create best response to multiple q values
x = np.linspace(-0.05,0.05,100)
SRAS = SRAS(x)
y = np.linspace(-0.05,0.05,100)
AD = AD(y)
z = np.linspace(-0.05,0.05,100)
newAD = newAD(z)
```

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(x,SRAS,"m",label='SRAS')
ax.plot(y,AD,"c",label='AD')
ax.plot(z,newAD,"r",label='newAD')
ax.legend()
ax.grid()
ax.set_xlabel('Outputgap $y_t$')
ax.set_ylabel('Inflationgap $\pi_t$')
ax.set_title('Graphical illustration, Q2')
ax.set_xlim([-0.1,0.1])
ax.set_ylim([-0.1,0.1]);
```

```python

# Plot results
plt.plot(x,SRAS,"m",label='SRAS')
plt.plot(y,AD,"c",label='AD')
plt.plot(z,newAD,"r",label='newAD')
plt.legend()
plt.xlabel('Outputgap, $y_t$')
plt.ylabel('Inflationgap, $\pi_t$')
plt.title('Graphical solution, Q2')
plt.grid()
plt.show

#### Jeg kunne godt tænke mig at rangen på y-aksen er mindre, så SRAS-kurven ikke ligner den er lig 0
### Men det er som om jeg ikke rigtig kan få det fikset
### Hvis én vil kigge på det, vil det være fedt!
```

**Persistent disturbances:** Now, additionaly, assume that both the demand and the supply disturbances are AR(1) processes

$$ v_{t} = \delta v_{t-1} + x_{t} $$
$$ s_{t} = \omega s_{t-1} + c_{t} $$

where $x_{t}$ is a **demand shock**, and $c_t$ is a **supply shock**. The **autoregressive parameters** are:

```python
par['delta'] = 0.80
par['omega'] = 0.15
```

**Question 3:** Starting from $y_{-1} = \pi_{-1} = s_{-1} = 0$, how does the economy evolve for $x_0 = 0.1$, $x_t = 0, \forall t > 0$ and $c_t = 0, \forall t \geq 0$?

```python
#Define disturbances

def v_t(v_lag, x_t):
    return par['delta']*v_lag+x_t

def s_t(s_lag,c_t):
    return par['omega']*s_lag+c_t
```

```python

```


```python
import numpy as np
import numba
import matplotlib.pyplot as plt
plt.style.use('presentation.mplstyle')
%matplotlib notebook

def d2np(d):

    names = []
    numbers = ()
    dtypes = []
    for item in d:
        names += item
        if type(d[item]) == float:
            numbers += (d[item],)
            dtypes += [(item,float)]
        if type(d[item]) == np.float64:
            numbers += (d[item],)
            dtypes += [(item,float)]
        if type(d[item]) == np.ndarray:
            numbers += (d[item],)
            dtypes += [(item,np.float64,d[item].shape)]
    return np.array([numbers],dtype=dtypes)
```

```python
# [1] D. Ochoa and S. Martinez,
# “Fast-Frequency Response provided by DFIG-Wind Turbines and its impact on the grid,”
# IEEE Trans. Power Syst., vol. 8950, no. c, pp. 1–1, 2016.

@numba.jit(nopython=True, cache=True)
def wind_turbine_fit(struct,i,m):

    nu_w = struct[i]['nu_w']    # m/s
    omega_t = struct[i]['omega_t'] # rad/s
    beta = struct[i]['beta']       # deg

    C_1 = struct[i]['C_1']
    C_2 = struct[i]['C_2']
    C_3 = struct[i]['C_3']
    C_4 = struct[i]['C_4']
    C_5 = struct[i]['C_5']
    C_6 = struct[i]['C_6']

    #Omega_t_n = struct['Omega_t_n'] #
    Radio = struct[i]['Radio']
    Rho = struct[i]['Rho']

    lam = omega_t*Radio/nu_w
    inv_lam_i = 1/(lam + 0.08 * beta) - 0.035/(beta**3 + 1)
    c_p = C_1 * (C_2 * inv_lam_i - C_3 * beta - C_4) * np.exp(-C_5*inv_lam_i) + C_6 * lam

    p_w = 0.5*Rho*np.pi*Radio**2*nu_w**3*c_p

    tau_t = 0.001
    if omega_t>0.0:
        tau_t = p_w/omega_t

    struct[i]['p_w'] = p_w
    struct[i]['c_p'] = c_p
    struct[i]['lam'] = lam
    struct[i]['tau_t'] = tau_t

    return 0

@numba.jit(nopython=True, cache=True)
def mppt_fit(struct,i,m):

    omega_t = struct[i]['omega_t'] # rad/s

    C_1 = struct[i]['C_1']
    C_2 = struct[i]['C_2']
    C_3 = struct[i]['C_3']
    C_4 = struct[i]['C_4']
    C_5 = struct[i]['C_5']
    C_6 = struct[i]['C_6']
    lam_opt = struct[i]['lam_opt']
    beta = struct[i]['beta']

    #Omega_t_n = struct['Omega_t_n'] #
    Radio   = struct[i]['Radio']
    Rho = struct[i]['Rho']

    inv_lam_i = 1/(lam_opt + 0.08 * beta) - 0.035/(beta**3 + 1)
    c_p = C_1 * (C_2 * inv_lam_i - C_3 * beta - C_4) * np.exp(-C_5*inv_lam_i) + C_6 * lam_opt

    #lam = omega_t*R/nu_w
    nu_w = omega_t*Radio/lam_opt
    p_ref = 0.5*Rho*np.pi*Radio**2*nu_w**3*c_p

    struct[i]['p_mppt'] = p_ref
    return p_ref

@numba.jit(nopython=True, cache=True)
def wecs_mech_1(struct,i,m):

    omega_t = struct[i]['x'][0,0]  # rad/s
    tau_t   = struct[i]['tau_t']
    tau_r   = struct[i]['tau_r']

    J_t  = struct[i]['J_t']
    N_tr = struct[i]['N_tr']
    Dt   = struct[i]['Dt']

    domega_t = 1.0/J_t*(tau_t - N_tr*tau_r)

    omega_r = N_tr*omega_t

    struct[i]['f'][:,0] = domega_t

    struct[i]['omega_r'] = omega_r
    struct[i]['omega_t'] = omega_t

    return omega_t

```

```python
@numba.jit(nopython=True, cache=True)
def pitch(struct,i,m):
    Dt  = struct[i]['Dt']
    beta_ref  = struct[i]['beta_ref']
    beta      = struct[i]['beta']
    beta_rate = struct[i]['beta_rate']

    if beta<beta_ref:
        beta += beta_rate * Dt

    if beta>beta_ref:
        beta -= beta_rate * Dt

    struct[i]['beta'] = beta

    return beta
```

```python
Radio = 50.0
lam_opt = 8.0
Nu_w_n = 10.0
Omega_t_n = Nu_w_n*lam_opt/Radio*0.8
Omega_t_max = Omega_t_n*1.25
Omega_t_min = Omega_t_n*0.75
beta = 0.0
Rho = 1.225
C_1 = 0.5176
C_2 = 116.0
C_3 = 0.4
C_4 = 5.0
C_5 = 21.0
C_6 = 0.0068
inv_lam_i = 1/(lam_opt + 0.08 * beta) - 0.035/(beta**3 + 1)
c_p = C_1 * (C_2 * inv_lam_i - C_3 * beta - C_4) * np.exp(-C_5*inv_lam_i) + C_6 * lam_opt
P_t_n = 0.5*Rho*np.pi*Radio**2*Nu_w_n**3*c_p
P_t_n/1e6
```

    2.3080100601373608

```python
N = 100

#nu_w =np.linspace(0.1,15,N)
H = 4.5

# H = 0.5*J*Omega_t_n**2/S_b
S_b = 2.0e6
Omega_t_n = 1.5
J_t = 2*H*S_b/Omega_t_n**2

d = dict(Radio=Radio,
         Rho = 1.225,
         C_1 = 0.5176,
         C_2 = 116.0,
         C_3 = 0.4,
         C_4 = 5.0,
         C_5 = 21.0,
         C_6 = 0.0068,
         P_t_n = P_t_n,
         nu_w = Nu_w_n,
         omega_t = Omega_t_n,
         beta = 0.0,
         Omega_t_n = Omega_t_n,
         Omega_t_min = Omega_t_min,
         Omega_t_max = Omega_t_max,
         Nu_w_n = Nu_w_n,
         p_w = 0.0,
         c_p = 0.0,
         lam = 0.0,
         lam_opt = 8.0,
         J_t = J_t,
         N_tr = 20.0,
         x = np.ones((1,1)),
         f = np.zeros((1,1)),
         omega_r = 1.0,
         tau_t = 10.0,
         tau_r = 0.0,
         Dt = 10.0e-3,
         p_mppt = 1.0,
         beta_ref = 0.0,
         beta_rate = 0.1,
        )

struct = d2np(d)
i=0
m=2
wind_turbine_fit(struct,i,m)
mppt_fit(struct,i,m)
wecs_mech_1(struct,i,m)
pitch(struct,i,m)
```

    0.0

```python
%timeit pitch(struct,i,m)
```

    357 ns ± 3.76 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

```python
struct = d2np(d)

fig, axes = plt.subplots(nrows=1, ncols=1, figsize=(8, 5), sharex = True)
betas = np.linspace(0.0,24,5)
omegas = np.linspace(0.1,3.5,100)
for beta in betas:
    C_p = []
    Lam = []
    for omega in omegas:
        struct['beta'] = beta
        struct['omega_t'] = omega
        wind_turbine_fit(struct,0,2)
        C_p += [float(struct['c_p'])]
        Lam += [float(struct['lam'])]
    axes.plot(np.array(Lam),np.array(C_p))
    axes.set_ylabel('$\sf C_p ( \lambda,\\beta )$')
    axes.set_xlabel('$\sf \lambda$')
    axes.set_ylim([0,0.5])
    axes.set_xlim([0,22])
    struct[0]['c_p']

fig.savefig('lam_cp.svg', bbox_inches='tight')

```

    <IPython.core.display.Javascript object>

```python
struct = d2np(d)
plt.tight_layout()
fig, axes = plt.subplots(nrows=1, ncols=1, figsize=(8, 5), sharex = True)
struct['beta'] = 0.0
nu_ws = np.linspace(5,10,5)
omegas = np.linspace(0.1,3.5,100)
for nu_w in nu_ws:
    P_w = []
    Omega = []
    for omega in omegas:
        struct['nu_w'] = nu_w
        struct['omega_t'] = omega
        wind_turbine_fit(struct,0,2)
        P_w += [float(struct['p_w'])]
        Omega += [float(struct['omega_t'])]
    axes.plot(Omega,np.array(P_w)/1e6)

omega = 1
Nu = np.linspace(2,12,20)
omegas_opt = []
p_ref_opt = []
for nu in Nu:
    struct['nu_w'] = nu
    for it in range(500):
        struct['omega_t'] = omega
        mppt_fit(struct,0,2)
        wind_turbine_fit(struct,0,2)
        p_ref = float(struct['p_mppt'])
        p_t = float(struct['p_w'])
        omega += 0.0000001*(-p_ref + p_t)/omega
        if abs(-p_ref + p_t)<0.01:
            break

    omegas_opt += [float(struct['omega_t'])]
    p_ref_opt += [float(struct['p_w'])]

axes.set_ylabel('$\sf P_t$')
axes.set_xlabel('$\sf \omega_t$')
axes.set_ylim([0,2.5])
axes.set_xlim([0,3.0])

fig.savefig('w_p_t.svg', bbox_inches='tight')
axes.plot(np.array(omegas_opt),np.array(p_ref_opt)/1e6, 'r')

fig.savefig('w_p_t_mppt.svg', bbox_inches='tight')

```

    <IPython.core.display.Javascript object>

## without MPPT

```python
struct = d2np(d)
struct['nu_w'] = 5.0
for it in range(10000):
    wecs_mech_1(struct,0,2)
    struct[0]['x'] += 100.0e-3*struct[0]['f'][0,0]
print(struct['omega_t'])
print(struct['p_w']/1e6)
```

    [ 1.00124987]
    [ 0.]

## with MPPT

```python
struct['omega_t'] = 1.0
struct['omega_r'] = 1.0*20
struct['nu_w'] = 8.5
t = 0.0
Omega_t = []
P_w = []
T = []
Dt = 100.0e-3
for it in range(500):
    if t>10.0:
        struct['nu_w'] = 6.0

    wecs_mech_1(struct,0,2)
    mppt_fit(struct,0,2)
    struct['tau_t'] = struct['p_w']/struct['omega_t']
    struct['tau_r'] = struct['p_mppt']/struct['omega_r']
    wind_turbine_fit(struct,0,2)
    struct[0]['x'] += Dt*struct[0]['f'][0,0]
    struct['omega_t'] = float(struct[0]['x'])
    Omega_t += [float(struct['omega_t'])]
    P_w += [float(struct['p_w'])]
    T += [t]
    t+= Dt
P_w = np.array(P_w)
print(struct['omega_t'])
print(struct['p_w']/1e6)
```

    [ 0.96008068]
    [ 0.49853348]

```python
plt.tight_layout()
fig, axes = plt.subplots(nrows=1, ncols=1, figsize=(8, 5), sharex = True)

axes.plot(T,Omega_t)

```

    <IPython.core.display.Javascript object>

    [<matplotlib.lines.Line2D at 0x7ffa23a47588>]

```python
plt.tight_layout()
fig, axes = plt.subplots(nrows=1, ncols=1, figsize=(8, 5), sharex = True)

axes.plot(T,P_w/1e6)
```

    <IPython.core.display.Javascript object>

    [<matplotlib.lines.Line2D at 0x7ffa23980160>]

```python
struct['beta_ref'] = 0.0
struct['beta'] = 0.0
for it in range(1000):
    t = it*struct['Dt']
    if t>0.1:
        struct['beta_ref'] = 5.0

    pitch(struct,0,2)
   # print(t, struct['beta'])
```

```python
from ipywidgets import interact, interactive, widgets
from IPython.display import display
import matplotlib as mpl
from matplotlib import pyplot as plt
mpl.rc('animation', html='html5')
%matplotlib notebook

fig, axes = plt.subplots(nrows=1, ncols=1, figsize=(5, 4), sharex = True)
fig.set_animated(True)
omegas = np.linspace(0.1,3.5,300)
p_w_line = axes.plot([], '-', ms=8)
axes.plot([struct[0]['Omega_t_max']]*2,[-10,10], '--', ms=2, color='k')
axes.plot([struct[0]['Omega_t_min']]*2,[-10,10], '--', ms=2, color='k')
axes.plot([-10,10], [struct[0]['P_t_n']/1.0e6]*2, '--', ms=2, color='k')

p_mppt_line = axes.plot(np.array(omegas_opt),np.array(p_ref_opt)/1e6)
p_mppt_point = axes.plot(0.0,0.0, 'or', ms=12)
axes.set_xlabel('$\sf \omega_t$')
axes.set_ylabel('$\sf P_t$')
axes.grid(True)

axes.set_ylim([0,2.5])
axes.set_xlim([0,3.0])
def update(beta,nu_w,court):
    P_w = []
    P_mppt =[]
    Omega = []

    struct['nu_w'] = nu_w
    for omega in omegas:
        struct['beta'] = beta
        struct['omega_t'] = omega
        wind_turbine_fit(struct,0,2)
        P_w += [float(struct['p_w'])]
        Omega += [float(struct['omega_t'])]
        struct[0]['beta'] = 0.0
        struct[0]['omega_t'] = omega*(court)
        mppt_fit(struct,0,2)
        P_mppt +=[float(struct['p_mppt'])]
    Omega = np.array(Omega)
    P_w = np.array(P_w)
    P_mppt = np.array(P_mppt)
    sh = 20
    idx = np.argmin(np.abs(P_w[sh:]-P_mppt[sh:]))
    #axes.plot(Omega,np.array(P_w)/1e6)
    p_w_line[0].set_xdata(np.array(Omega))
    p_w_line[0].set_ydata(np.array(P_w)/1.0e6)
    p_mppt_line[0].set_xdata(np.array(Omega))
    p_mppt_line[0].set_ydata(np.array(P_mppt)/1.0e6)
    print('Pt = {:2.2f} MW  speed = {:2.2f} p.u.'.format(P_w[idx+sh]/1.0e6, Omega[idx+sh]/struct[0]['Omega_t_n']))
    p_mppt_point[0].set_xdata(Omega[idx+sh])
    p_mppt_point[0].set_ydata(P_w[idx+sh]/1.0e6)

sld_beta  = widgets.FloatSlider(min=0,max=20,step=0.1,value=0.0, width=1000)
sld_nu_w  = widgets.FloatSlider(min=0.1,max=20,step=0.1,value=struct['Nu_w_n'], width=1000)
sld_court = widgets.FloatSlider(min=0.1,max=1,step=0.02,value=1, width=1000)

interaction_avr = interactive(update,
                          beta=sld_beta,
                          nu_w=sld_nu_w,
                          court=sld_court,
                          continuous_update=True, width=1000)

```

    <IPython.core.display.Javascript object>

```python
display(interaction_avr)
```

    A Jupyter Widget

```python

```

```python

```

```python
from ipywidgets import interact, interactive, widgets
from IPython.display import display
import threading
import time

exitFlag = 0

class myThread (threading.Thread):
    def __init__(self, threadID, name, counter):
        threading.Thread.__init__(self)
        self.threadID = threadID
        self.name = name
        self.counter = counter
    def run(self):
        print ("Starting " + self.name)
        print_time(self.name, self.counter, 50)
        print("Exiting " + self.name)

def print_time(threadName, counter, delay):
    while 0:
        if exitFlag:
            threadName.exit()
        time.sleep(delay)
        print("%s: %s" % (threadName, time.ctime(time.time())))
        counter -= 1

# Create new threads
thread1 = myThread(1, "Thread-1", 1)
thread2 = myThread(2, "Thread-2", 2)

# Start new Threads
thread1.start()
thread2.start()

print("Exiting Main Thread")
```

    Starting Thread-1
    Exiting Thread-1
    Starting Thread-2
    Exiting Thread-2
    Exiting Main Thread

```python
x = np.array([1])
def update(value_1):
    print(value_1)
    x[0] = value_1

sld_1  = widgets.FloatSlider(min=0,max=20,step=0.1,value=0.0, width=1000)
interaction_1 = interactive(update,
                              value_1 = sld_1,
                              continuous_update=True, width=1000)
```

```python
display(interaction_1)
```

    0.0

```python
import numpy as np
import sympy as sym
from ipywidgets import interact, interactive, widgets
from IPython.display import display
import matplotlib as mpl
from matplotlib import pyplot as plt
mpl.rc('animation', html='html5')
%matplotlib notebook

fig, axes = plt.subplots(nrows=1, ncols=1, figsize=(8, 5), sharex = True)
fig.set_animated(True)
line_1 = axes.plot([], '-', ms=8)
axes.set_xlim([0.0,2*np.pi])
axes.set_ylim([-1.1,1.1])

class anima(object):

    def __init__(self):
        self.A = 1.0

def update(t):
    fobj = open('dato','r')
    A = float(fobj.read())
    fobj.close()
    x = np.linspace(0,2*np.pi,100)+ t
    line_1[0].set_xdata(x)
    line_1[0].set_ydata(A*np.sin(2*x))
    axes.set_xlim([0.0+t,2*np.pi+t])

def uparam(A):
    fobj = open('dato','w')
    fobj.write(str(A))
    fobj.close()

anima1 = anima()

sld_1  = widgets.FloatSlider(min=0,max=20,step=0.1,value=0.0, width=1000)
interaction_1 = interactive(uparam,
                              A = sld_1,
                              continuous_update=True, width=1000)

```

    <IPython.core.display.Javascript object>

```python

display(interaction_1)

import time
for t in np.linspace(0.0,10.0,100):
    update(t)
    time.sleep(0.1)
    fig.canvas.draw()
```

```python

```

```python
help(fig.draw)
```

    Help on method draw in module matplotlib.artist:

    draw(renderer, *args, **kwargs) method of matplotlib.figure.Figure instance
        Render the figure using :class:`matplotlib.backend_bases.RendererBase`
        instance *renderer*.

```python

```

```python
import matplotlib.pyplot as plt
import matplotlib.animation as anim

def plot_cont(fun, xmax):
    y = []
    fig = plt.figure()
    ax = fig.add_subplot(1,1,1)

    def update(i):
        yi = fun()
        y.append(yi)
        x = range(len(y))
        ax.clear()
        ax.plot(x, y)
        print( i, ': ', yi)

    a = anim.FuncAnimation(fig, update, frames=xmax, repeat=False)
    plt.show()
```

```python

```


 ILI285 - Computación Científica I  / INF285 - Computación Científica

 Generalized Minimal Residual Method

 [[S]cientific [C]omputing [T]eam](#acknowledgements)

 Version: 1.2

## Table of Contents
* [Introduction](#intro)
* [Short reminder about Least Squares](#LS)
* [GMRes](#GMR)
* [Theoretical Problems](#TP)
* [Practical Problems](#PP)
* [Acknowledgements](#acknowledgements)

```python
import numpy as np
import scipy as sp
from scipy import linalg as la
import matplotlib.pyplot as plt
import scipy.sparse.linalg
%matplotlib inline
#%load_ext memory_profiler
import matplotlib as mpl
mpl.rcParams['font.size'] = 14
mpl.rcParams['axes.labelsize'] = 20
mpl.rcParams['xtick.labelsize'] = 14
mpl.rcParams['ytick.labelsize'] = 14
# from scipy.interpolate import CubicSpline # Starting scipy 0.19.0
M=8
```

## Introduction

Welcome to another edition of our Jupyter Notebooks. A few notebooks back, we saw that the Conjugate Gradient Method, an iterative method, was very useful to solve $A\,\mathbf{x}=\mathbf{b}$ but it only worked when $A$ was positive definite and symmetric. So now we need an iterative method that works with nonsymmetric linear system of equations, and for that we have the Generalized Minimum Residual Method (GMRes). It works really well for finding the solution of large, sparse (and dense as well), nonsymmetric linear systems of equations. Of course, it will also have trouble for ill-conditioned linear system of equations. But it is really easy to add a left or right or both preconditioners!

## A quick review on Least Squares

Least Squares is used to solve overdetemined linear systems of equations $A\,\mathbf{x} = \mathbf{b}$. That is, for example, a linear system of equations where there are more equations than unknowns. It finds the _best_ $\overline{\mathbf{x}}$ so that it minimizes the euclidean length of $\mathbf{r} = \mathbf{b} - A\,\mathbf{x}$.

So, you might be wondering, what does Least Squares have to do with GMRes? WELL, since you're dying to know, I'll tell you: the backward error of the system in GMRes is minimized at each iteration step using a Least Squares formulation.

## GMRes

GMRes is a member of the family of Krylov methods. It finds an approximation of $\mathbf{x}$ restricted to _live_ on the Krylov sub-space $\mathcal{K_k}$,  where $\mathcal{K_k}=\{\mathbf{r}_0, A\,\mathbf{r}_0, A^2\,\mathbf{r}_0, \cdots, A^{k-1}\,\mathbf{r}_0\}$ and $\mathbf{r}_0 = \mathbf{b} - A\,\mathbf{x}_0$ is the residual vector of the initial guess.

The idea behind this method is to look for improvements to the initial guess $\mathbf{x}_0$ in the Krylov space. At the $k$-th iteration, we enlarge the Krylov space by adding $A^k\,\mathbf{r}_0$, reorthogonalize the basis, and then use least squares to find the best improvement to add to $\mathbf{x}_0$.

The algorithm is as follows:

`Generalized Minimum Residual Method`

$\mathbf{x}_0$ `= initial guess`

$\mathbf{r}$ `=` $\mathbf{b} - A\,\mathbf{x}_0$

$\mathbf{q}_1$ `=` $\mathbf{r} / \|\mathbf{r}\|_2$

`for` $k = 1, ..., m$

$\qquad \ \ \mathbf{y} = A\,\mathbf{q}_k$

$\qquad$ `for` $j = 1,2,...,k$

$\qquad \qquad$ $h_{jk} = \mathbf{q}_j^*\,\mathbf{y}$

$\qquad \qquad$ $\mathbf{y} = \mathbf{y} - h_{jk}\, \mathbf{q}_j$

$\qquad$ `end`

$\qquad \ h_{k+1,k} = \|y\|_2 \qquad$ `(If ` $h_{k+1,k} = 0$ `, skip next line and terminate at bottom.)`

$\qquad \ \mathbf{q}_{k+1} = \mathbf{y}/h_{k+1,k}$

$\qquad$ `Minimize` $\left\|\widehat{H}_k\, \mathbf{c}_k - [\|\mathbf{r}\|_2 \ 0 \ 0 \ ... \ 0]^T \right\|_2$ `for` $\mathbf{c}_k$

$\qquad$ $\mathbf{x}_k = Q_k \, \mathbf{c}_k + \mathbf{x}_0$

`end`

Now we have to implement it.

```python
# This is a very instructive implementation of GMRes.
def GMRes(A, b, x0=np.array([0.0]), m=10, flag_display=True, threshold=1e-12):
    n = len(b)
    if len(x0)==1:
        x0=np.zeros(n)
    r0 = b - np.dot(A, x0)
    nr0=np.linalg.norm(r0)
    out_res=np.array(nr0)
    Q = np.zeros((n,n))
    H = np.zeros((n,n))
    Q[:,0] = r0 / nr0
    flag_break=False
    for k in np.arange(np.min((m,n))):
        y = np.dot(A, Q[:,k])
        if flag_display:
            print('||y||=',np.linalg.norm(y))
        for j in np.arange(k+1):
            H[j][k] = np.dot(Q[:,j], y)
            if flag_display:
                print('H[',j,'][',k,']=',H[j][k])
            y = y - np.dot(H[j][k],Q[:,j])
            if flag_display:
                print('||y||=',np.linalg.norm(y))
        # All but the last equation are treated equally. Why?
        if k+1<n:
            H[k+1][k] = np.linalg.norm(y)
            if flag_display:
                print('H[',k+1,'][',k,']=',H[k+1][k])
            if (np.abs(H[k+1][k]) > 1e-16):
                Q[:,k+1] = y/H[k+1][k]
            else:
                print('flag_break has been activated')
                flag_break=True
            # Do you remember e_1? The canonical vector.
            e1 = np.zeros((k+1)+1)
            e1[0]=1
            H_tilde=H[0:(k+1)+1,0:k+1]
        else:
            H_tilde=H[0:k+1,0:k+1]
        # Solving the 'SMALL' least square problem.
        # This could be improved with Givens rotations!
        ck = np.linalg.lstsq(H_tilde, nr0*e1)[0]
        if k+1<n:
            x = x0 + np.dot(Q[:,0:(k+1)], ck)
        else:
            x = x0 + np.dot(Q, ck)
        # Why is 'norm_small' equal to 'norm_full'?
        norm_small=np.linalg.norm(np.dot(H_tilde,ck)-nr0*e1)
        out_res = np.append(out_res,norm_small)
        if flag_display:
            norm_full=np.linalg.norm(b-np.dot(A,x))
            print('..........||b-A\,x_k||=',norm_full)
            print('..........||H_k\,c_k-nr0*e1||',norm_small);
        if flag_break:
            if flag_display:
                print('EXIT: flag_break=True')
            break
        if norm_small<threshold:
            if flag_display:
                print('EXIT: norm_small<threshold')
            break
    return x,out_res
```

## A very simple example

```python
A = np.array([[1,1,0],[0,1,0],[0,1,1]])
b = np.array([1,2,3])
x0 = np.zeros(3)
```

```python
# scipy gmres
x_scipy = scipy.sparse.linalg.gmres(A,b,x0)[0]
# our gmres
x_our, _ = GMRes(A, b)
# numpy solve
x_np= np.linalg.solve(A,b)

# Showing the solutions
print('--------------------------------')
print('x_scipy',x_scipy)
print('x_our',x_our)
print('x_np',x_np)
```

    ||y||= 1.6475089421
    H[ 0 ][ 0 ]= 1.57142857143
    ||y||= 0.494871659305
    H[ 1 ][ 0 ]= 0.494871659305
    ..........||b-A\,x_k||= 1.1239029739
    ..........||H_k\,c_k-nr0*e1|| 1.1239029739
    ||y||= 0.786795792469
    H[ 0 ][ 1 ]= -0.659828879074
    ||y||= 0.428571428571
    H[ 1 ][ 1 ]= 0.428571428571
    ||y||= 2.77555756156e-17
    H[ 2 ][ 1 ]= 2.77555756156e-17
    flag_break has been activated
    ..........||b-A\,x_k||= 9.99200722163e-16
    ..........||H_k\,c_k-nr0*e1|| 8.89664082996e-16
    EXIT: flag_break=True
    --------------------------------
    x_scipy [-1.  2.  1.]
    x_our [-1.  2.  1.]
    x_np [-1.  2.  1.]

    /usr/local/lib/python3.4/dist-packages/numpy/core/fromnumeric.py:2699: VisibleDeprecationWarning: `rank` is deprecated; use the `ndim` attribute or function instead. To find the rank of a matrix see `numpy.linalg.matrix_rank`.
      VisibleDeprecationWarning)

## Another example, how may iteration does it need to converge?

```python
A = np.array([[0,0,0,1],[1,0,0,0],[0,1,0,0],[0,0,1,0]])
b = np.array([1,0,1,0])
x_our, _ = GMRes(A, b, m=10)
norm_full=np.linalg.norm(b-np.dot(A,x_our))
print(norm_full)
```

```python
A = np.random.rand(10,10)+10*np.eye(10)
b = np.random.rand(10)
x_our, out_res = GMRes(A, b, m=10,flag_display=True)
norm_full=np.linalg.norm(b-np.dot(A,x_our))
print(norm_full)
```

## Plotting the residual over the iterations

```python
plt.figure(figsize=(M,M))
plt.semilogy(out_res,'.k',markersize=20,label='residual')
plt.grid(True)
plt.xlabel(r'$k$')
plt.ylabel(r'$\|\mathbf{b}-A\,\mathbf{x}_k\|_2$')
plt.grid(True)
plt.show()
```

## Theoretical Problems

1. Prove that in GMRES method, the backward error $||b- Ax_k||$ decreases monotonically with k.
2. What would happen if we pass a singular matrix $A$ to the previous implementation of GMRes?
3. Prove that for
\begin{equation}
A=
\left[
\begin{array}{c|c}
I & C \\
\hline
0 & I
\end{array}
\right]
\end{equation}
and any $x_0$ and $b$, GMRES converges to the exact solution after **two** steps. Here $C$ is a $m_1 \times m_2$ submatrix, $0$ denotes the $m_2 \times m_1$ matrix of zeros, and $I$ denotes the appropiate-sized identity matrix.

## Practical Problems

1. A possible improvement to the present algorithm consists on taking out of the loop the least squares computations, since the _Krylov_ subspace spaned by $Q_k$ doesn't depend on previous least squares calculations.
    * Verify the truth of the above statement.
    * Verify if it is really an improvement.
    * Implement it.
    * Test both implementations using `%timeit`
1. The GMRES method is meant for huge $n\times n$ sparse matrices $A$. In most cases, the goal is to run the method for $k$ steps (with $k << n$), reducing the complexity of the subproblems (Least squares). Neverthless for $k$ values too small, the solution $x_k$ could be not as good as needed. So to keep the values $k$ small and avoid bad solutions, there exists a variation of the algorithm known as **Restarted GMRES**: If no enough progress is made toward the solution after $k$ iterations, discard $Q_k$ and start GMRES from the beginning, using the current best guess $x_k$ as the new $x_0$.
    * Implement the Restarted GMRES method. Introduce a tolerance parameter to stop restarting.
    * Compare the asymptotic operation count and storage requirements of GMRES and Restarted GMRES, for fixed $k$ and increasing $n$.
    * Execute it on a _huge_ linear system $A x = b$, and compare the solution with the solution of standard GMRES. Keep a value of $k$ small, and count how many times Restarted GMRES has to restart. Perform benchmarks using `%timeit` and `%memit` and verify the results.
    * Describe an example in which Restared GMRES can be expected to fail to converge, whereas GMRES succeds.

# Acknowledgements
* _Material created by professor Claudio Torres_ (`ctorres@inf.utfsm.cl`) _and assistants: Laura Bermeo, Alvaro Salinas, Axel Simonsen and Martín Villanueva. DI UTFSM. April 2016._
* _Material updated by professor Claudio Torres_ (`ctorres@inf.utfsm.cl`). DI UTFSM. June 2017.

```python

```


# Density operator and matrix

```
from sympy import *
from sympy.physics.quantum import *
from sympy.physics.quantum.density import *
from sympy.physics.quantum.spin import (
    Jx, Jy, Jz, Jplus, Jminus, J2,
    JxBra, JyBra, JzBra,
    JxKet, JyKet, JzKet,
)
from IPython.core.display import display_pretty
```

```
%load_ext sympyprinting
```

Create n density matrix using symbolic states

```
psi = Ket('psi')
phi = Ket('phi')
```

```
d = Density((psi,0.5),(phi,0.5)); d
```

$$\rho\left(\begin{pmatrix}{\left|\psi\right\rangle }, & 0.5\end{pmatrix},\begin{pmatrix}{\left|\phi\right\rangle }, & 0.5\end{pmatrix}\right)$$

```
display_pretty(d)
```

    ρ((❘ψ⟩, 0.5),(❘φ⟩, 0.5))

```
d.states()
```

$$\begin{pmatrix}{\left|\psi\right\rangle }, & {\left|\phi\right\rangle }\end{pmatrix}$$

```
d.probs()
```

$$\begin{pmatrix}0.5, & 0.5\end{pmatrix}$$

```
d.doit()
```

```
Dagger(d)
```

$$\rho\left(\begin{pmatrix}{\left|\psi\right\rangle }, & 0.5\end{pmatrix},\begin{pmatrix}{\left|\phi\right\rangle }, & 0.5\end{pmatrix}\right)$$

```
A = Operator('A')
```

```
d.apply_op(A)
```

$$\rho\left(\begin{pmatrix}A {\left|\psi\right\rangle }, & 0.5\end{pmatrix},\begin{pmatrix}A {\left|\phi\right\rangle }, & 0.5\end{pmatrix}\right)$$

Now create a density matrix using spin states

```
up = JzKet(S(1)/2,S(1)/2)
down = JzKet(S(1)/2,-S(1)/2)
```

```
d2 = Density((up,0.5),(down,0.5)); d2
```

$$\rho\left(\begin{pmatrix}{\left|\frac{1}{2},\frac{1}{2}\right\rangle }, & 0.5\end{pmatrix},\begin{pmatrix}{\left|\frac{1}{2},- \frac{1}{2}\right\rangle }, & 0.5\end{pmatrix}\right)$$

```
represent(d2)
```

$$\left[\begin{smallmatrix}0.5 & 0\\0 & 0.5\end{smallmatrix}\right]$$

```
d2.apply_op(Jz)
```

$$\rho\left(\begin{pmatrix}J_z {\left|\frac{1}{2},\frac{1}{2}\right\rangle }, & 0.5\end{pmatrix},\begin{pmatrix}J_z {\left|\frac{1}{2},- \frac{1}{2}\right\rangle }, & 0.5\end{pmatrix}\right)$$

```
qapply(_)
```

$$\rho\left(\begin{pmatrix}\frac{1}{2} \hbar {\left|\frac{1}{2},\frac{1}{2}\right\rangle }, & 0.5\end{pmatrix},\begin{pmatrix}- \frac{1}{2} \hbar {\left|\frac{1}{2},- \frac{1}{2}\right\rangle }, & 0.5\end{pmatrix}\right)$$

```
qapply((Jy*d2).doit())
```

## Evaluate entropy of the density matrices

```
entropy(d2)
```

```
entropy(represent(d2))
```

```
entropy(represent(d2,format="numpy"))
```

    (0.69314718056-0j)

```
entropy(represent(d2,format="scipy.sparse"))
```

    (0.69314718056-0j)

## Density operators with Tensor Products as args

```
from sympy.core.trace import Tr

A, B, C, D = symbols('A B C D',commutative=False)

t1 = TensorProduct(A,B,C)

d = Density([t1, 1.0])
d.doit()

t2 = TensorProduct(A,B)
t3 = TensorProduct(C,D)

d = Density([t2, 0.5], [t3, 0.5])
d.doit()
```

```
#mixed states
d = Density([t2+t3, 1.0])
d.doit()
```

## Trace operators on Density Operators with Spin States

```
from sympy.physics.quantum.density import Density
from sympy.physics.quantum.spin import (
    Jx, Jy, Jz, Jplus, Jminus, J2,
    JxBra, JyBra, JzBra,
    JxKet, JyKet, JzKet,
)
from sympy.core.trace import Tr

d = Density([JzKet(1,1),0.5],[JzKet(1,-1),0.5]);
t = Tr(d);

display_pretty(t)
print t.doit()
```

    Tr(ρ((❘1,1⟩, 0.5),(❘1,-1⟩, 0.5)))

    1.00000000000000

## Partial Trace on Density Operators with Mixed State

```
#Partial trace on mixed state
from sympy.core.trace import Tr

A, B, C, D = symbols('A B C D',commutative=False)

t1 = TensorProduct(A,B,C)

d = Density([t1, 1.0])
d.doit()

t2 = TensorProduct(A,B)
t3 = TensorProduct(C,D)

d = Density([t2, 0.5], [t3, 0.5])

tr = Tr(d,[1])
tr.doit()
```

$$0.5 A A^{\dagger} \mbox{Tr}\left(B B^{\dagger}\right) + 0.5 C C^{\dagger} \mbox{Tr}\left(D D^{\dagger}\right)$$

## Partial Trace on Density Operators with Spin states

```
from sympy.physics.quantum.density import Density
from sympy.physics.quantum.spin import (
    Jx, Jy, Jz, Jplus, Jminus, J2,
    JxBra, JyBra, JzBra,
    JxKet, JyKet, JzKet,
)
from sympy.core.trace import Tr

tp1 = TensorProduct(JzKet(1,1), JzKet(1,-1))

#trace out 0 index
d = Density([tp1,1]);
t = Tr(d,[0]);

display_pretty(t)
display_pretty(t.doit())

#trace out 1 index
t = Tr(d,[1])
display_pretty(t)
t.doit()
```

## Examples of qapply() on Density matrices with spin states

```
psi = Ket('psi')
phi = Ket('phi')

u = UnitaryOperator()
d = Density((psi,0.5),(phi,0.5)); d

display_pretty(qapply(u*d))

# another example
up = JzKet(S(1)/2, S(1)/2)
down = JzKet(S(1)/2, -S(1)/2)
d = Density((up,0.5),(down,0.5))

uMat = Matrix([[0,1],[1,0]])
display_pretty(qapply(uMat*d))

```

    ρ((O⋅❘ψ⟩, 0.5),(O⋅❘φ⟩, 0.5))

    ⎡                  0                    ρ((❘1/2,1/2⟩, 0.5),(❘1/2,-1/2⟩, 0.5))⎤
    ⎢                                                                            ⎥
    ⎣ρ((❘1/2,1/2⟩, 0.5),(❘1/2,-1/2⟩, 0.5))                    0                  ⎦

## Example of qapply() on Density Matrices with qubits

```
from sympy.physics.quantum.gate import UGate
from sympy.physics.quantum.qubit import Qubit

uMat = UGate((0,), Matrix([[0,1],[1,0]]))
d = Density([Qubit('0'),0.5],[Qubit('1'), 0.5])

display_pretty(d)

#after applying Not gate
display_pretty(qapply(uMat*d) )
```

    ρ((❘0⟩, 0.5),(❘1⟩, 0.5))

    ρ((❘1⟩, 0.5),(❘0⟩, 0.5))

```

```