# SUSA CX Kaggle Capstone Project
## Part 3: Hyperparameter Tuning, Decision Trees, Ensemble Learning

### Table Of Contents
* [Introduction](#section1)
* [Hyperparameters](#section2)
* [GridSearch](#section2i)
* [Decision Trees](#section3)
* [Random Forest](#section4)
* [Ensemble Learning](#section5)
* [Conclusion](#conclusion)
* [Additional Reading](#reading)

### Hosted by and maintained by the [Statistics Undergraduate Students Association (SUSA)](https://susa.berkeley.edu). Originally authored by [Patrick Chao](mailto:prc@berkeley.edu) & [Noah Gundotra](mailto:noah.gundotra@berkeley.edu).

# SUSA CX Kaggle Capstone Project

Welcome to the third week of the SUSA CX Kaggle Capstone! Now that you have an understanding of your data, a cleaned dataset, and a working model, now it's time to improve your model performance with some new tricks and techniques. Now that we have explored regularization techniques, PCA for feature selection and dimension reduction, and feature engineering, we will delve into more technical details and advanced models.

## Logistics

Most of the logistics are the same as last week, but we are repeating them here for your convenience. Please let us know if you or your teammates are feeling nervous about the pace of this project - remember that we are not grading you on your project, and we really try to make the notebooks relatively easy and fast to code through. If for any reason you are feeling overwhelmed or frustrated, please DM us or talk to us in person. We want all of you to have a productive, healthy, and fun time learning data science! If you have any suggestions or recommendations on how to improve, please do not hesitate to reach out!

## Datathon

For those of you that came to the Datathon, we hope you all had a great time! It seemed that many people got a lot of work done and learned a ton of new materials. We had a quick lecture by Patrick on the mathematical intuitions on regularization and PCA. If you couldn't make it or didn't understand all the details, feel free to reach out! Also if you felt that the lecture was very helpful and you would like to see more similar stuff, let us know!

### Mandatory Office Hours

Because this is such a large project, you and your team will surely have to work on it outside of meetings. In order to get you guys to seek help from this project, we are making it **mandatory** for you and your group to attend **two (2)** SUSA Office Hours over the next 4 weeks. This will allow questions to be answered outside of the regular meetings and will help promote collaboration with more experienced SUSA members.

The schedule of SUSA office hours are below:
https://susa.berkeley.edu/calendar#officehours-table

We understand that most of you will end up going to Arun or Patrick's office hours, but we highly encourage you to go to other people's office hours as well. There are many qualified SUSA mentors who can help and this could be an opportunity for you to meet them.

```python
# Import statements
from sklearn import tree # There are lots of other models from this module you can try!
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import Ridge, Lasso, LinearRegression, Ridge
import pandas as pd
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.grid_search import GridSearchCV
from sklearn.metrics import mean_squared_error, make_scorer
from sklearn.externals.six import StringIO
from IPython.display import Image
from sklearn.tree import export_graphviz
import pydotplus
sqrt=np.sqrt
```

    /usr/local/lib/python3.6/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
      "This module will be removed in 0.20.", DeprecationWarning)
    /usr/local/lib/python3.6/site-packages/sklearn/grid_search.py:42: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. This module will be removed in 0.20.
      DeprecationWarning)

# Hyperparameters and Tuning

As we have discussed in the project workflow, we are beginning to investigate more technical models. However, as we continue to explore more and more complex models, we may lose interpretability. Many problems in machine learning or data science eventually reduce to some parameters we are incapable of understanding or predicting ahead of time. These may be parameters inherent to the dataset or based on the model.

If we look back at k-Nearest Neighbors, based on different values of $k$ we obtain various models. For $k=1$, we obtain a low bias and high variance estimate. Our estimate is highly dependent on the data available. As $k$ increases, our bias increases but the variance decreases. We would like to minimize both bias and variance, thus the optimal value of $k$ lies somewhere between these values. How do we determine this value? Unfortunately there is no golden bullet or magical formula we may calculate to answer this question.

These mystical quantities we would like to evaluate are known as **hyperparameters**. Hyperparameters are values in our model that we cannot optimize directly, instead we may have to explore a variety of examples before determining the optimal value. In kNN, the value of $k$ characterizes the model. Thus we may describe this as a **model hyperparameter**. If we were using polynomial regression, the degree of the model would also be a model hyperparameter, as we can arbitrarily select the power and it characterizes our model.

Another example of a hyperparameter is the $\lambda$ used in Ridge and LASSO regression. There is essentially no method of evaluating the optimal value of $\lambda$ through calculations, instead we must experiment and emperically determine the best value. These do not determine the form of the model, but affect the optimization. Thus this is an **optimization hyperparameter**.

We may imagine hyperparameters as a bunch of individual knobs we may turn. Consider that we are visiting our friend and staying at her place. However, you did not realize that she is actually an alien and her house is filled with very strange objects. When you head to bed, you attempt to use her shower, but see that her shower is has a dozen of knobs that control the temperature of the water coming out! We only have a single output to work off of, but many different knobs or *parameters* to adjust. If the water is too hot, we can turn random knobs until it becomes cold, and learn a bit about our environment. We may determine that some knobs are more or less sensitive, just like hyperparameters. Each knob in the shower is equivalent to a hyperparameter we can tune in a model.

# Grid Search

Consider we have two hyperparameters $A$ and $B$ that range from $0$ to $100$.

> 1. Discuss with a group on possible methods of deciding the optimal value of $A$ and $B$?
> 2. Is it possible to find the exact best values of $A$ and $B$?

One of the most naive approaches is to simply create a grid of values for $A$ and $B$, hence the name grid search. We could consider choosing the values where $A$ or $B$ are equal to $0,50,100$, and determine the validation error on all pairs of hyperparameters. Since there are $3$ values for both $A$ and $B$, there are $9$ total possible choices for $(A,B)$.

Consider the following grid below.

The grid represents the validation error for all $(A,B)$ from $[0,50,100]$. It may seem that $(A,B)=(50,100)$ is the best value with an error of $7$, but our scale is quite coarse. We could instead have four values for $A$ and $B$, resulting in $16$ total calculations. This is shown below with $[0,33,66,100]$.

Now the optimal value shifts to $(A,B)=(66,33)$ with an error of $7$, but this will change again as our interval becomes more and more fine. However, a few issues immediately arise. First, the number of calculations increases quadratically with the number of values for each interval! This is quite bad, as quadratic functions grow quite quickly. Secondly, we can never really determine the exact optimal value unless we had infinitely fine intervals. Lastly, we only had a decrease in error from $7$ to $5$ with $9$ to $16$ calculations. Is this actually worthwhile? Sometimes performing more searches does not actually help. These are all various tradeoffs to decide, at a certain point the number of calculations is not worth the added benefit of well tuned parameters.

Now, this is only for the two variable case with $A$ and $B$. In practice, models may have many variables, even dozens! If we consider $10$ parameters, just doing two values for an interval results in $2^{10}=1024$ calculations! Keep in mind that each value in the grid requires a separate model to be trained using the parameters, which results in $1024$ different models to be trained with just a comparison of two values for each parameter! The number of models trained is equal to $m^n$ where $m$ is the number of values in the interval, and $n$ is the number of hyperparameters.

Grid search can be very challenging due to this issue, and suffers from the curse of dimensionality. There are other approaches that are used in practice. Instead of methodically selecting all pairs of values in the cartesian product, we could perform a [random search](https://en.wikipedia.org/wiki/Random_search) through parameters. Even more complex, we can create a model to estimate the optimal parameters. This leads into the field of [meta-learning](http://bair.berkeley.edu/blog/2017/07/18/learning-to-learn/), learning to learn.

# Decision Trees with Grid Search

Now that we have investigated how hyperparameters affect our models, we may apply it directly to our current problem of predicting housing prices. In this section, we will be going over decision trees. If you would like a refresher, please refer to this [notebook](../Machine%20Learning/BiasVarianceDecisionTrees/Bias%20Variance%20Decision%20Trees%20Ensemble%20Learning.ipynb)!

In the past, we talked about decision trees as purely classification. In practice, we can also use decision trees for regression! Rather than have a single class at the leaves and attempting to reduce the impurity or entropy of individual leaves, we have individual functions at the leaves of the decision tree.

First, we will load in the cleaned data and import the necessary libraries.

```python
def get_features(data, col_list, y_name):
    """
    Function to return a numpy matrix of pandas dataframe features, given k column names and a single y column
    Outputs X, a n X k dimensional numpy matrix, and Y, an n X 1 dimensional numpy matrix.
    This is not a smart function - although it does drop rows with NA values. It might break.

    data(DataFrame): e.g. train, clean
    col_list(list): list of columns to extract data from
    y_name(string): name of the column you to treat as the y column

    Ideally returns one np.array of shape (len(data), len(col_list)), and one of shape (len(data), len(col_list))
    """

    # keep track of numpy values
    feature_matrix = data[col_list + [y_name]].dropna().values
    np.random.shuffle(feature_matrix)
    return feature_matrix[:, :-1], feature_matrix[:, -1]

def get_loss(model, X,Y_true):
    """Returns square root of L2 loss (RMSE) from a model, X value input, and true y values

    model(Model object): model we use to predict values
    X: numpy matrix of x values
    Y_true: numpy matrix of true y values
    """
    Y_hat = model.predict(X)
    return get_RMSE(Y_hat,Y_true)

def get_RMSE(Y_hat,Y_true):
    """Returns square root of L2 loss (RMSE) between Y_hat and true values

    Y_true: numpy matrix of predicted y values
    Y_true: numpy matrix of true y values
    """
    return np.sqrt(np.mean((Y_true-Y_hat)**2))

def get_train_and_val(X,Y):
    """Given the X and Y data, return the training and validation based on the split variable

    X: numpy matrix of x values
    Y: numpy matrix of y values
    split: value between 0 and 1 for the training split
    """

    Y = Y.reshape(Y.shape[0],)

    train_index,_ = get_train_val_indices(X,Y)

    y_train = Y.reshape(Y.shape[0],)
    y_train = Y[:train_index]
    x_train = X[:train_index,:]

    x_val = X[train_index:,:]
    y_val = Y[train_index:]
    return (x_train,y_train),(x_val,y_val)

def get_train_val_indices(X,Y=None,split=0.7):
    train_index = (int)(X.shape[0]*split)
    test_index =X.shape[0]-1
    return train_index,test_index

def grid_search(model,grid_params,verbose=True):
    """ Given a model and a set of parameters, this will search through the grid of parameters

    model: model input ex: tree.DecisionTreeRegressor()
    grid_params: dictionary of parameter values, parameter to list
    verbose: boolean default to true, prints the value of all scores in the grid

    Modified from http://scikit-learn.org/0.15/auto_examples/grid_search_digits.html
    """

    clf=GridSearchCV(model,param_grid = grid_params,n_jobs=-1,cv=3,scoring=make_scorer(get_RMSE,greater_is_better=False))
    clf.fit(X,Y)
    if verbose:
        print("Grid scores on development set:")
        for params, mean_score, scores in clf.grid_scores_:
            print("%0.3f (+/-%0.03f) for %r"
                  % (-1*mean_score, scores.std() / 2, params))
        print()
    print("Best parameters set found:")
    print(clf.best_params_)
    print()
    print("Best score found:")
    print(-1*clf.best_score_)
    print()

def feature_importance(model,features,num_features = 10):
    importances = sorted(zip(list(model.feature_importances_),features),reverse=True,key=lambda x :x[0])
    return importances[:num_features]
```

Most of the code above is the same as last week with a few tweaks and added methods. A few important changes: now the training and validation set is randomized, so you may obtain different losses based on different random initalizations.

Replace `feature_cols` with the features you chose in the previous notebook. For decision trees, we would recommend having more features rather than fewer, recommended to have at least $10$.

We will first read in the data, select the relevant features, and separate into training and validation.

```python
clean = pd.read_csv('DATA/house-prices/train_cleaned.csv')
#feature_cols = ['LotArea', 'Utilities', 'OverallCond', 'BsmtFinSF1', 'MasVnrArea']

def select_columns_except(dframe, non_examples):
    """Returns all comlumns in dframe except those in non_examples."""
    all_cols = dframe.select_dtypes(include=[np.number]).columns.tolist()
    cond = lambda x: sum([x == col for col in non_examples]) >= 1
    return [x for x in all_cols if not cond(x)]

feature_cols = select_columns_except(clean, ['Id','SalePrice'])

X, Y = get_features(clean, feature_cols, 'SalePrice')
(x_train,y_train),(x_val,y_val) = get_train_and_val(X,Y)
```

Now using sklearn, we will train a decision tree regressor with all default parameters. Keep in mind since we have a small dataset and our decision tree may vary wildly with each optimization, the RMSE will also change greatly from each time you run it! This dataset is particularly small so it does not fair well with very deep trees.

```python
decisionTree = tree.DecisionTreeRegressor()
decisionTree = decisionTree.fit(x_train, y_train)
loss = get_loss(decisionTree, x_val,y_val)
print("Root Mean Squared Error loss of our model: {:.2f}".format(loss))
```

Now we will experiment with $3$ hyperparameters in the decision tree sklearn implementation. There are more than $3$ hyperparameters, but we will stick with three for simplicity.

The three we are using are `max_depth`, `min_samples_split`, and `min_samples_leaf`.
> `max_depth`: The maximum depth of any branch in the tree. As max_depth increases, our tree gets more and more complex and is more likely to overfit to the data.
> `min_samples_split`: Since each branch can continue splitting to obtain more and purer leaves, eventually it may not make sense to continue splitting as we may be overfitting. To avoid this, we may set a min_samples_split, which represents the minimum number of samples to split on.
> `min_samples_leaf`: We can continue our decision tree until the leaves become very small and we overfit. Another restraint could be to have a minimum value restricting the number of samples in a leaf.

The code below runs a grid search on the parameters chosen. Feel free to modify the ranges for all the hyperparameters! This is a crucial part of model selection.

The `grid_search` function takes in a model, a dictionary of a set of parameters, and a boolean flag for verbose. The model in our case will just be `tree.DecisionTreeRegressor()`, and the parameters is equal to `grid_params`. Feel free to change the verbose flag to True if you would like to view all the outputs.

```python
grid_params = [{'max_depth':[4,6,8,10,12],'min_samples_leaf':[4,6,8,10,12,14],'min_samples_split':[6,8,10,12,14,16]}]
grid_search(tree.DecisionTreeRegressor(),grid_params,verbose=False)
```

```python
decisionTree = tree.DecisionTreeRegressor(max_depth=8,min_samples_leaf=10,min_samples_split=4)
decisionTree = decisionTree.fit(x_train, y_train)
loss = get_loss(decisionTree, x_val,y_val)
print("Root Mean Squared Error loss of our model: {:.2f}".format(loss))
```

Another great thing about sklearn's decision tree implementation is that it can also tell you the relative feature importance! The code segment below

```python
feature_importance(decisionTree,feature_cols)
```

```python
dot_data = StringIO()
export_graphviz(decisionTree, out_file=dot_data,
                filled=True, rounded=True,
                special_characters=True,feature_names=feature_cols)
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
Image(graph.create_png())
```

# Random Forest

Now we have thoroughly investigated decision trees, but why stop there? We can collect a bunch of trees to form a **random forest**! A very quick reminder for what random forests are: collect a bunch of trees, and average their results. Averaging may be taking the majority in a classification problem, or the mean of the regression output of each tree. In order to prevent each of the trees being identical, we randomly sample portions of the data to train on and random features to use in our forest.

The code is very similar and we will take advantage of sklearn's implementation of random forests.

```python
randomForest = RandomForestRegressor()
randomForest = randomForest.fit(x_train, y_train)
loss = get_loss(randomForest, x_val,y_val)
print("Root Mean Squared Error loss of our model: {:.2f}".format(loss))
```

Wow! This random forest already performs far better than our other models. Can we tune some parameters as before? Now we may add the `n_estimators` parameter, which is basically how many trees we have in our ensemble.

**Warning**: This may take a long time to run! Do not try to search too much at once.

```python
grid_params = [{'max_depth':[12,14,16,18], 'min_samples_leaf':[1,2,4],'min_samples_split':[2,3,4,6,8,10],
                     'n_estimators':[20,30,40],'random_state':[0],'bootstrap':[False,True]}]
grid_search(RandomForestRegressor(),grid_params,verbose=False)
```

```python
randomForest = RandomForestRegressor(max_depth=14,min_samples_leaf=1,min_samples_split=4,
                                     n_estimators=50,random_state=0,bootstrap=True)
randomForest = randomForest.fit(x_train, y_train)
loss = get_loss(randomForest, x_val,y_val)
print("Root Mean Squared Error loss of our model: {:.2f}".format(loss))
```

We can look at the feature importances again as before.

```python
feature_importance(randomForest,feature_cols)
```

# Ensemble Learning
----

Ensemble learning is one of the most widely used techniques to improve model accuracy. Many automated, programmatic frameworks for simple modeling tasks lend themselves to being ensembled.

Ensembling is the process of training multiple models to do the same thing, and averaging their predictions. Ensemble models often train their models on different (overlapping) splits of the dataset. These datasets are picked randomly _**with**_ replacement. Intuitively, you can imagine training the same type of linear model and giving each model only 1 category of data to learn from. It's possible to imagine that each linear model as becoming an expert in its dataset, and can contribute unique information to other expert models. From a high level perspective, ensembling is merging the expert opinions of different models.

From this narrative, we can see how very different models produce a much more useful ensemble than identical models. Clearly, having 10 modes that detects numbers 0-9 is more useful to digit detection than having 10 models which all detect the number 1. This is known as independence of errors.

Along this note... There are good formal **statistical** and **mathematical** reasons to do ensembling. One of the easier things to prove about ensembling is that averaging model predictions actually reduces variance on error.

We just wanted to introduce you to what the math looks like, in case you're interested. The following is from the Deep Learning Book (Goodfellow 2016). Given $k$ models, with errors $\epsilon_i$ on each example, and errors with variance $v$, the expected loss of the model decreases as more models are ensembled. Specifically, this works when the correlation between the models, $c$, is less than 1.

$$
\begin{align}
\mathbb{E}\left[\left(\frac{1}{k} \sum_i \epsilon_i\right)^2\right] &= \frac{1}{k^2}\mathbb{E}\left[\sum_{i}\left(\epsilon_i^2 + \sum_{j\not=i} \epsilon_i\epsilon_j\right)\right] \\
&= \frac{1}{k}v + \frac{k-1}{k}c
\end{align}
$$

A second good reason to use ensembling is to improve the space of representations that the aggregate (total) model can learn. Understanding this line of reasoning requires the ability to visualize a model's decision boundary. Essentially, multiple single models can overfit to the data, creating complex, abstract hyperplanes to separate between classifications.

If you're interested in reading up on this, and learning where that figure came from, check out [MLWave's explanation.](https://mlwave.com/kaggle-ensembling-guide/)

### Do not worry about the math!

We do not need to understand it to understand how ensembling works. We'll begin our practical walk through below.

```python
# This is my favorite approach! Try everything lazy and grab all the low-hanging fruit!
# Brute force everything!

# Create `num_models` of models of decision trees with random max_depth
# And train them before appending to a list of models
# Creating LinearRegression models trained on different datasets have less correlated errors
# TODO: Why don't you try using this function with tree.DecisionTree? This is an alternate RandomForest implementation
# TODO: Can you extend this function to work accept any list of model functions?
def create_models(model_func, num_models, frac_dataset, x_train):
    """Trains `num_models` of model_func instances on `frac_dataset` percent of the x_train."""
    models = []
    for i in range(num_models):
        num_dataset = int(frac_dataset * len(x_train))
        indices = np.random.permutation(num_dataset)
        x_random, y_random = x_train[indices], y_train[indices]
        models.append(model_func().fit(x_random, y_random))
    return models

model_func = lambda : LinearRegression()
models = create_models(model_func, 10, 0.7, x_train)

def avg_model(ms):
    """Returns a lambda that can be used to predict over all the models."""
    def predict(x_input):
        """Ensembles model predictions using an unweighted average."""
        total = len(ms)
        return sum([model.predict(x_input) for model in models]) / total
    return predict

def get_loss_from_function(predict_function, X, Y_true):
    """Returns square root of L2 loss (RMSE) between Y_hat and true values

    predict_function(function): model we use to predict values
    X: numpy matrix of x values
    Y_true: numpy matrix of true y values
    """
    Y_hat = predict_function(X)
    return np.sqrt(np.mean((Y_true-Y_hat)**2))

# Now we can use the models we trained to create an averaged ensemble
avg_model_predict = avg_model(models)
loss = get_loss_from_function(avg_model_predict, x_train, y_train)
print("Train loss: {:.2f}".format(loss))
loss = get_loss_from_function(avg_model_predict, x_val, y_val)
print("Validation loss: {:.2f}".format(loss))
```

Yay! Clearly, the loss on this model is much better than the loss of the previous best decision tree!

## Smarter Ensembling - Weighted Averages

Some models are better than others, so we can train a model to learn how to weigh the predictions of the models. We can do this by saving the predictions of a bunch of models, and then learning a weighted average of them that predicts `y_train` (using a Linear Model). If using a linear model to learn the weights of the ensemble doesn't make sense - don't fret.

Let the ensembled model output be $f(x)$, and the output of model $i$ be $m_i$.
$$ f(x) = a_1*m_1(x) + a_2*m_2(x) + \ldots + a_n*m_n(x) $$
This is a **_linear_** transformation, which means a linear model can learn the weights $a_1, a_2, \cdots, a_n$.

```python
def get_ensemble_features(models, x_input):
    """Returns the features for an ensemble by passing the original input through the original base models."""
    model_predictions = np.concatenate([model.predict(x_input) for model in models]).reshape((-1, len(models)))
    return model_predictions

ensemble_features = get_ensemble_features(models, x_train)

linear_ensemble_model = Ridge().fit(ensemble_features, y_train)

loss = get_loss(linear_ensemble_model, get_ensemble_features(models, x_train), y_train)
print(loss)
loss = get_loss(linear_ensemble_model, get_ensemble_features(models, x_val), y_val)
print(loss)
```

```python
linear_ensemble_model.coef_ = np.array([1/10.0 for i in range(10)])
```

This clearly didn't work nearly as well as the model averaging! This is probably because the method by which Sci-Kit learn calculates least squares regression assumes very low correlation between features. It's possible that they leveraged this to produce a method which is not as good at learning a linear model over similar ensembles.

Other things to try would be performing least squares regression on the ensembles' outputs. That should probably work. In general, the practice of building models on top of other models is called **ensemble stacking.** This a very big subfield of statistical modeling that is responsible for the difference between a good model and an award-winning model. Specifically, **gradient-boosting** is a very widely used approach. See the [Additional Reading](#section) section for more.

# Conclusion

This ends the third of our four collaborative sessions on the Kaggle Housing Prices competition. This week contains a lot of technical material, but a large majority is running code chunks and thinking about what the models actually do. As always, please email [Arun Ramamurthy](mailto:contact@arun.run), [Patrick Chao](mailto:prc@berkeley.edu), or [Noah Gundotra](mailto:noah.gundotra@berkeley.edu) or with any questions or concerns whatsoever.

Next week, we will be wrapping things up by investigating how to apply DeEp lEArNinG with neural networks to this problem in Keras! This is the final culmination of all the work we have done so far, as well as finally submitting to compare your performances! Happy machine learning!

# Additional Reading

This notebook we covered a lot of different modeling tools. However, we have barely scratched the surface. Even though we have not presented as much math as in previous presentations, there is a lot of theory behind these tools.

The most widely used tools are variants of decision trees. One of the most popular variants of random forest modeling is called [Gradient Boosting](http://blog.kaggle.com/2017/01/23/a-kaggle-master-explains-gradient-boosting). The framework that implements this particularly well is called [XGBoost](https://github.com/dmlc/xgboost).

Sci-kit learn has a lot of ensemble methods built-in as well. Their documentation is probably some of the best out there, it's absolutely littered with examples. See the [ensemble docs](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.ensemble) for more.

If you're interested in different types of models to use in your ensemble, try out Logistic Regression in the sci-kit learn library. For the full list of models, see [their docs](http://scikit-learn.org/stable/supervised_learning.html


Continuous Data and the Gaussian Distribution
=======

### Preliminaries

- Goal
  - Review of information processing with Gaussian distributions in linear systems
- Materials
  - Mandatory
    - These lecture notes
  - Optional
    - Bishop pp. 85-93
    - [MacKay - 2006 - The Humble Gaussian Distribution](https://github.com/bertdv/BMLIP/blob/master/lessons/notebooks/files/Mackay-2006-The-humble-Gaussian-distribution.pdf) (highly recommended!)
    - [Ariel Caticha - 2012 - Entropic Inference and the Foundations of Physics](https://github.com/bertdv/BMLIP/blob/master/lessons/notebooks/files/Caticha-2012-Entropic-Inference-and-the-Foundations-of-Physics.pdf), pp.30-34, section 2.8, the Gaussian distribution
  - References
    - [E.T. Jaynes - 2003 - Probability Theory, The Logic of Science](http://www.med.mcgill.ca/epidemiology/hanley/bios601/GaussianModel/JaynesProbabilityTheory.pdf) (best book available on the Bayesian view on probability theory)

### Example Problem

Consider a set of observations $D=\{x_1,…,x_N\}$ in the 2-dimensional plane (see Figure). All observations were generated by the same process. We now draw an extra observation $x_\bullet = (a,b)$ from the same data generating process. What is the probability that $x_\bullet$ lies within the shaded rectangle $S$?

```julia
using Pkg;Pkg.activate("probprog/workspace/");Pkg.instantiate();
IJulia.clear_output();
```

```julia
using Distributions, PyPlot
N = 100
generative_dist = MvNormal([0,1.], [0.8 0.5; 0.5 1.0])
function plotObservations(obs::Matrix)
    plot(obs[1,:], obs[2,:], "kx", zorder=3)
    fill_between([0., 2.], 1., 2., color="k", alpha=0.4, zorder=2) # Shaded area
    text(2.05, 1.8, "S", fontsize=12)
    xlim([-3,3]); ylim([-2,4]); xlabel("a"); ylabel("b")
end
D = rand(generative_dist, N) # Generate observations from generative_dist
plotObservations(D)
x_dot = rand(generative_dist) # Generate x∙
plot(x_dot[1], x_dot[2], "ro");
```

### The Gaussian Distribution

- Consider a random (vector) variable $x \in \mathbb{R}^M$ that is "normally" (i.e., Gaussian) distributed. The _moment_ parameterization of the Gaussian distribution is completely specified by its _mean_ $\mu$ and _variance_ $\Sigma$ and given by
$$
p(x | \mu, \Sigma) = \mathcal{N}(x|\mu,\Sigma) \triangleq \frac{1}{\sqrt{(2\pi)^M |\Sigma|}} \,\exp\left\{-\frac{1}{2}(x-\mu)^T \Sigma^{-1} (x-\mu) \right\}\,.
$$
where $|\Sigma| \triangleq \mathrm{det}(\Sigma)$ is the determinant of $\Sigma$.

- Alternatively, the _canonical_ (a.k.a. _natural_  or _information_ ) parameterization of the Gaussian distribution is given by
$$\begin{equation*}
p(x | \eta, \Lambda) = \mathcal{N}_c(x|\eta,\Lambda)  = \exp\left\{ a + \eta^T x - \frac{1}{2}x^T \Lambda x \right\}\,.
\end{equation*}$$
  - $a = -\frac{1}{2} \left( M \log(2 \pi) - \log |\Lambda| + \eta^T \Lambda \eta\right)$ is the normalizing constant that ensures that $\int p(x)\mathrm{d}x = 1$.
  - $\Lambda = \Sigma^{-1}$ is called the *precision matrix*.
  - $\eta = \Sigma^{-1} \mu$ is the _natural_ mean or for clarity often called the *precision-weighted* mean.

### Why the Gaussian?

- Why is the Gaussian distribution so ubiquitously used in science and engineering? (see [Jaynes, section 7.1.4](http://www.med.mcgill.ca/epidemiology/hanley/bios601/GaussianModel/JaynesProbabilityTheory.pdf#page=250)).

- (1) Operations on probability distributions tend to lead to Gaussian distributions:
  - Any smooth function with single rounded maximum, if raised to higher and higher powers, goes into a Gaussian function. (Example: sequential Bayesian inference).
  - The [Gaussian distribution has higher entropy](https://en.wikipedia.org/wiki/Differential_entropy#Maximization_in_the_normal_distribution) than any other with the same variance.
    - Therefore any operation on a probability distribution that discards information but preserves variance gets us closer to a Gaussian.
    - As an example, see [Jaynes, section 7.1.4](http://www.med.mcgill.ca/epidemiology/hanley/bios601/GaussianModel/JaynesProbabilityTheory.pdf#page=250) for how this leads to the [Central Limit Theorem](https://en.wikipedia.org/wiki/Central_limit_theorem), which results from performing convolution operations on distributions.

- (2) Once the Gaussian has been attained, this form tends to be preserved. e.g.,
  - The convolution of two Gaussian functions is another Gaussian function (useful in sum of 2 variables and linear transformations)
  - The product of two Gaussian functions is another Gaussian function (useful in Bayes rule).
  - The Fourier transform of a Gaussian function is another Gaussian function.

###  Transformations and Sums of Gaussian Variables

- A **linear transformation** $z=Ax+b$ of a Gaussian variable $\mathcal{N}(x|\mu_x,\Sigma_x)$ is Gaussian distributed as
$$
p(z) = \mathcal{N} \left(z \,|\, A\mu_x+b, A\Sigma_x A^T \right) \tag{SRG-4a}
$$
  - In fact, after a linear transformation $z=Ax+b$, no matter how $x$ is distributed, the mean and variance of $z$ are always given by $\mu_z = A\mu_x + b$  and $\Sigma_z = A\Sigma_x A^T$, respectively (see   [probability theory review lesson](https://nbviewer.jupyter.org/github/bertdv/BMLIP/blob/master/lessons/notebooks/Probability-Theory-Review.ipynb#linear-transformation)). In case $x$ is not Gaussian, higher order moments may be needed to specify the distribution for $z$.

- The **sum of two independent Gaussian variables** is also Gaussian distributed. Specifically, if $x \sim \mathcal{N} \left(\mu_x, \Sigma_x \right)$ and $y \sim \mathcal{N} \left(\mu_y, \Sigma_y \right)$, then the PDF for $z=x+y$ is given by
$$\begin{align}
p(z) &= \mathcal{N}(x\,|\,\mu_x,\Sigma_x) \ast \mathcal{N}(y\,|\,\mu_y,\Sigma_y) \notag\\
  &= \mathcal{N} \left(z\,|\,\mu_x+\mu_y, \Sigma_x +\Sigma_y \right) \tag{SRG-8}
\end{align}$$

  - The sum of two Gaussian _distributions_ is NOT a Gaussian distribution. Why not?

### Example: Gaussian Signals in a Linear System

- Given independent variables
$x \sim \mathcal{N}(\mu_x,\sigma_x^2)$ and $y \sim \mathcal{N}(\mu_y,\sigma_y^2)$, what is the PDF for $z = A\cdot(x -y) + b$ ? (see [Exercises](http://nbviewer.jupyter.org/github/bertdv/BMLIP/blob/master/lessons/exercises/Exercises-The-Gaussian-Distribution.ipynb))

- Think about the role of the Gaussian distribution for stochastic linear systems in relation to what sinusoidals mean for deterministic linear system analysis.

### Bayesian Inference for the Gaussian

- Let's estimate a constant $\theta$ from one 'noisy' measurement $x$ about that constant.

- We assume the following measurement equations (the tilde $\sim$ means: 'is distributed as'):
$$\begin{align*}
x &= \theta + \epsilon \\
\epsilon &\sim \mathcal{N}(0,\sigma^2)
\end{align*}$$

- Also, let's assume a Gaussian prior for $\theta$
$$\begin{align*}
\theta &\sim \mathcal{N}(\mu_0,\sigma_0^2) \\
\end{align*}$$

##### Model specification

- Note that you can rewrite these specifications in probabilistic notation as follows:

$$\begin{align}
    p(x|\theta) &=  \mathcal{N}(x|\theta,\sigma^2) \tag{likelihood}\\
    p(\theta) &=\mathcal{N}(\theta|\mu_0,\sigma_0^2) \tag{prior}
\end{align}$$

- (**Notational convention**). Note that we write $\epsilon \sim \mathcal{N}(0,\sigma^2)$ but not $\epsilon \sim \mathcal{N}(\epsilon | 0,\sigma^2)$, and we write  $p(\theta) =\mathcal{N}(\theta|\mu_0,\sigma_0^2)$ but not $p(\theta) =\mathcal{N}(\mu_0,\sigma_0^2)$.

##### Inference

- For simplicity, we assume that the variance $\sigma^2$ is given and will proceed to derive a Bayesian posterior for the mean $\theta$. The case for Bayesian inference for $\sigma^2$ with a given mean is [discussed in the optional slides](#inference-for-precision).

- Let's do Bayes rule for the posterior PDF $p(\theta|x)$.
$$\begin{align*}
p(\theta|x)  &= \frac{p(x|\theta) p(\theta)}{p(x)} \propto p(x|\theta) p(\theta) \notag \\
    &= \mathcal{N}(x|\theta,\sigma^2) \mathcal{N}(\theta|\mu_0,\sigma_0^2)   \notag \\
    &\propto \exp \left\{   -\frac{(x-\theta)^2}{2\sigma^2} - \frac{(\theta-\mu_0)^2}{2\sigma_0^2} \right\} \notag \\
    &\propto \exp \left\{ \theta^2 \cdot \left( -\frac{1}{2 \sigma_0^2} - \frac{1}{2\sigma^2}  \right)  + \theta \cdot  \left( \frac{\mu_0}{\sigma_0^2} + \frac{x}{\sigma^2}\right)   \right\} \notag \\
    &= \exp\left\{ -\frac{\sigma_0^2 + \sigma^2}{2 \sigma_0^2 \sigma^2} \left( \theta - \frac{\sigma_0^2 x +  \sigma^2 \mu_0}{\sigma^2 + \sigma_0^2}\right)^2  \right\}
\end{align*}$$
which we recognize as a Gaussian distribution w.r.t. $\theta$.

- (Just as an aside,) this computational 'trick' for multiplying two Gaussians is called **completing the square**. The procedure makes use of the equality $$ax^2+bx+c_1 = a\left(x+\frac{b}{2a}\right)^2+c_2$$

- In particular, it follows that the posterior for $\theta$ is
$$\begin{equation*}
    p(\theta|x) = \mathcal{N} (\theta |\, \mu_1, \sigma_1^2)
\end{equation*}$$
where
$$\begin{align*}
  \frac{1}{\sigma_1^2}  &= \frac{\sigma_0^2 + \sigma^2}{\sigma^2 \sigma_0^2} = \frac{1}{\sigma_0^2} + \frac{1}{\sigma^2}  \\
  \mu_1   &= \frac{\sigma_0^2 x +  \sigma^2 \mu_0}{\sigma^2 + \sigma_0^2} = \sigma_1^2 \, \left(  \frac{1}{\sigma_0^2} \mu_0 + \frac{1}{\sigma^2} x \right)
\end{align*}$$

### (Multivariate) Gaussian Multiplication

- So, multiplication of two Gaussian distributions yields another (unnormalized) Gaussian with
  - posterior precision equals **sum of prior precisions**
  - posterior precision-weighted mean equals **sum of prior precision-weighted means**

- As we just saw, great application to Bayesian inference!

$$\begin{equation*}
\underbrace{\text{Gaussian}}_{\text{posterior}}
 \propto \underbrace{\text{Gaussian}}_{\text{likelihood}} \times \underbrace{\text{Gaussian}}_{\text{prior}}
\end{equation*}$$

- In general, the multiplication of two multi-variate Gaussians yields an (unnormalized) Gaussian:
$$\begin{equation*}
\boxed{\mathcal{N}(x|\mu_a,\Sigma_a) \cdot \mathcal{N}(x|\mu_b,\Sigma_b) = \underbrace{\mathcal{N}(\mu_a|\, \mu_b, \Sigma_a + \Sigma_b)}_{\text{normalization constant}} \cdot \mathcal{N}(x|\mu_c,\Sigma_c)} \tag{SRG-6}
\end{equation*}$$
where
$$\begin{align*}
\Sigma_c^{-1} &= \Sigma_a^{-1} + \Sigma_b^{-1} \\
\Sigma_c^{-1} \mu_c &= \Sigma_a^{-1}\mu_a + \Sigma_b^{-1}\mu_b
\end{align*}$$

- $\Rightarrow$ Note that Bayesian inference is trivial in the [_canonical_ parameterization of the Gaussian](#natural-parameterization), where we would get
$$\begin{align*}
 \Lambda_c &= \Lambda_a + \Lambda_b  \quad &&\text{(precisions add)}\\
 \eta_c &= \eta_a + \eta_b \quad &&\text{(precision-weighted means add)}
\end{align*}$$

### Code Example: Product of Two Gaussian PDFs

- Let's plot the exact product of two Gaussian PDFs as well as the normalized product according to the above derivation.

```julia
using PyPlot, Distributions
d1 = Normal(0, 1) # μ=0, σ^2=1
d2 = Normal(3, 2) # μ=3, σ^2=4

# Calculate the parameters of the product d1*d2
s2_prod = (d1.σ^-2 + d2.σ^-2)^-1
m_prod = s2_prod * ((d1.σ^-2)*d1.μ + (d2.σ^-2)*d2.μ)
d_prod = Normal(m_prod, sqrt(s2_prod)) # Note that we neglect the normalization constant.

# Plot stuff
x = range(-4, stop=8, length=100)
plot(x, pdf.(d1,x), "k")
plot(x, pdf.(d2,x), "b")
plot(x, pdf.(d1,x) .* pdf.(d2,x), "r-") # Plot the exact product
plot(x, pdf.(d_prod,x), "r:")          # Plot the normalized Gaussian product
legend([L"\mathcal{N}(0,1)",
        L"\mathcal{N}(3,4)",
        L"\mathcal{N}(0,1) \mathcal{N}(3,4)",
        L"Z^{-1} \mathcal{N}(0,1) \mathcal{N}(3,4)"]);
```

The solid and dotted red curves are identical up to a scaling factor $Z$.

### Bayesian Inference with multiple Observations

- Now consider that we measure a data set $D = \{x_1, x_2, \ldots, x_N\}$, with measurements
  $$\begin{align*}
x_n &= \theta + \epsilon_n \\
\epsilon_n &\sim \mathcal{N}(0,\sigma^2)
\end{align*}$$
and the same prior for $\theta$:
$$\begin{align*}
\theta &\sim \mathcal{N}(\mu_0,\sigma_0^2) \\
\end{align*}$$

- Let's derive a distribution for the next sample $x_{N+1}$.

##### inference

- Clearly, the posterior for $\theta$ is now
$$\begin{align*}
p(\theta|D) \propto  \underbrace{\mathcal{N}(\theta|\mu_0,\sigma_0^2)}_{\text{prior}} \cdot \underbrace{\prod_{n=1}^N \mathcal{N}(x_n|\theta,\sigma^2)}_{\text{likelihood}}
\end{align*}$$
which is a multiplication of $N+1$ Gaussians and is therefore also Gaussian distributed.

- Using the property that precisions and precision-weighted means add when Gaussians are multiplied, we can immediately write the posterior $$p(\theta|D) = \mathcal{N} (\theta |\, \mu_N, \sigma_N^2)$$ as
$$\begin{align*}
  \frac{1}{\sigma_N^2}  &= \frac{1}{\sigma_0^2} + \sum_n  \frac{1}{\sigma^2} \tag{B-2.142}\\
  \mu_N   &= \sigma_N^2 \, \left( \frac{1}{\sigma_0^2} \mu_0 + \sum_n \frac{1}{\sigma^2} x_n  \right) \tag{B-2.141}
\end{align*}$$

##### application: prediction of future sample

- We now have a posterior for the model parameters. Let's write down what we know about the next sample $x_{N+1}$.

$$\begin{align*}
p(x_{N+1}|D_N) &= \int p(x_{N+1}|\theta) p(\theta|D_{N})\mathrm{d}\theta \\
  &= \int \mathcal{N}(x_{N+1}|\theta,\sigma^2) \mathcal{N}(\theta|\mu_N,\sigma^2_N) \mathrm{d}\theta \\
  &= \mathcal{N}(x_{N+1}|\mu_N, \sigma^2_N +\sigma^2 ) \tag{use SRG-6}
\end{align*}$$

- Uncertainty about $x_{N+1}$ comprises both uncertainty about the parameter ($\sigma_N^2$) and observation noise $\sigma^2$.

### Maximum Likelihood Estimation for the Gaussian

- In order to determine the _maximum likelihood_ estimate of $\theta$, we let $\sigma_0^2 \rightarrow \infty$ (leads to uniform prior for $\theta$), yielding $ \frac{1}{\sigma_N^2} = \frac{N}{\sigma^2}$ and consequently
$$\begin{align*}
  \mu_{\text{ML}}  = \left.\mu_N\right\vert_{\sigma_0^2 \rightarrow \infty} = \sigma_N^2 \, \left(   \frac{1}{\sigma^2}\sum_n  x_n  \right) = \frac{1}{N} \sum_{n=1}^N x_n
  \end{align*}$$

- As expected, having an expression for the maximum likelihood estimate, it is now possible to rewrite the (Bayesian) posterior mean for $\theta$ as (exercise)
$$\begin{align*}
  \underbrace{\mu_N}_{\text{posterior}}   &= \frac{\sigma^2}{N} \, \left( \frac{1}{\sigma_0^2} \mu_0 + \sum_n \frac{1}{\sigma^2} x_n  \right) \\
  &= \underbrace{\mu_0}_{\text{prior}} + \underbrace{\underbrace{\frac{N \sigma_0^2}{N \sigma_0^2 + \sigma^2}}_{\text{gain}}\cdot \underbrace{\left(\mu_{\text{ML}} - \mu_0 \right)}_{\text{prediction error}}}_{\text{correction}}\tag{B-2.141}
\end{align*}$$

- Hence, the posterior mean always lies somewhere between the prior mean $\mu_0$ and the maximum likelihood estimate (the "data" mean) $\mu_{\text{ML}}$.

### Conditioning and Marginalization of a Gaussian

- Let $z = \begin{bmatrix} x \\ y \end{bmatrix}$ be jointly normal distributed as

$$\begin{align*}
p(z) &= \mathcal{N}(z | \mu, \Sigma)
  =\mathcal{N} \left( \begin{bmatrix} x \\ y \end{bmatrix} \left| \begin{bmatrix} \mu_x \\ \mu_y \end{bmatrix},
  \begin{bmatrix} \Sigma_x & \Sigma_{xy} \\ \Sigma_{yx} & \Sigma_y \end{bmatrix} \right. \right)
\end{align*}$$

- Since covariance matrices are by definition symmetric, it follows that $\Sigma_x$ and $\Sigma_y$ are symmetric and $\Sigma_{xy} = \Sigma_{yx}^T$.

- Let's factorize $p(z) = p(x,y)$ into $p(y|x)\cdot p(x)$ through conditioning and marginalization.

$$\begin{equation*}
\text{conditioning: }\boxed{ p(y|x) = \mathcal{N}\left(y\,|\,\mu_y + \Sigma_{yx}\Sigma_x^{-1}(x-\mu_x),\, \Sigma_y - \Sigma_{yx}\Sigma_x^{-1}\Sigma_{xy} \right)}
\end{equation*}$$

$$\begin{equation*}
\text{marginalization: } \boxed{ p(x) = \mathcal{N}\left( x|\mu_x, \Sigma_x \right)}
\end{equation*}$$

- **proof**: in Bishop pp.87-89

- Hence, conditioning and marginalization in Gaussians leads to Gaussians again. This is very useful for applications to Bayesian inference in jointly Gaussian systems.

- With a natural parameterization of the Gaussian $p(z) = \mathcal{N}_c(z|\eta,\Lambda)$ with precision matrix $\Lambda = \Sigma^{-1} = \begin{bmatrix} \Lambda_x & \Lambda_{xy} \\ \Lambda_{yx} & \Lambda_y \end{bmatrix}$,  the conditioning operation results in a simpler result, see Bishop pg.90, eqs. 2.96 and 2.97.

- As an exercise, interpret the formula for the conditional mean ($\mathbb{E}[y|x]=\mu_y + \Sigma_{yx}\Sigma_x^{-1}(x-\mu_x)$) as a prediction-correction operation.

### Code Example: Joint, Marginal, and Conditional Gaussian Distributions

- Let's plot of the joint, marginal, and conditional distributions.

```julia
using PyPlot, Distributions

μ = [1.0; 2.0]
Σ = [0.3 0.7;
     0.7 2.0]
joint = MvNormal(μ,Σ)
marginal_x = Normal(μ[1], sqrt(Σ[1,1]))

#Plot p(x,y)
subplot(221)
x_range = y_range = range(-2,stop=5,length=1000)
joint_pdf = [ pdf(joint, [x_range[i];y_range[j]]) for  j=1:length(y_range), i=1:length(x_range)]
imshow(joint_pdf, origin="lower", extent=[x_range[1], x_range[end], y_range[1], y_range[end]])
grid(); xlabel("x"); ylabel("y"); title("p(x,y)"); tight_layout()

# Plot p(x)
subplot(222)
plot(range(-2,stop=5,length=1000), pdf.(marginal_x, range(-2,stop=5,length=1000)))
grid(); xlabel("x"); ylabel("p(x)"); title("Marginal distribution p(x)"); tight_layout()

# Plot p(y|x)
x = 0.1
conditional_y_m = μ[2]+Σ[2,1]*inv(Σ[1,1])*(x-μ[1])
conditional_y_s2 = Σ[2,2] - Σ[2,1]*inv(Σ[1,1])*Σ[1,2]
conditional_y = Normal(conditional_y_m, sqrt.(conditional_y_s2))
subplot(223)
plot(range(-2,stop=5,length=1000), pdf.(conditional_y, range(-2,stop=5,length=1000)))
grid(); xlabel("y"); ylabel("p(y|x)"); title("Conditional distribution p(y|x)"); tight_layout()
```

As is clear from the plots, the conditional distribution is a renormalized slice from the joint distribution.

### Example: Conditioning of Gaussian

- Consider (again) the system
$$\begin{align*}
p(x\,|\,\theta) &= \mathcal{N}(x\,|\,\theta,\sigma^2)
\end{align*}$$
with a Gaussian prior for $\theta$:
$$
p(\theta) = \mathcal{N}(\theta\,|\,\mu_0,\sigma_0^2)
$$

- Let $z = \begin{bmatrix} x \\ \theta \end{bmatrix}$. The distribution for $z$ is then given by (Exercise)

$$
p(z) = p\left(\begin{bmatrix} x \\ \theta \end{bmatrix}\right) = \mathcal{N} \left( \begin{bmatrix} x\\
  \theta  \end{bmatrix}
  \,\left|\, \begin{bmatrix} \mu_0\\
  \mu_0\end{bmatrix},
         \begin{bmatrix} \sigma_0^2+\sigma^2  & \sigma_0^2\\
         \sigma_0^2 &\sigma_0^2
  \end{bmatrix}
  \right. \right)
$$

- Direct substitution of the rule for Gaussian conditioning leads to the posterior (derivation as an Exercise):
$$\begin{align*}
p(\theta|x) &= \mathcal{N} \left( \theta\,|\,\mu_1, \sigma_1^2 \right)\,,
\end{align*}$$
with
$$\begin{align*}
K &= \frac{\sigma_0^2}{\sigma_0^2+\sigma^2} \qquad \text{($K$ is called: Kalman gain)}\\
\mu_1 &= \mu_0 + K \cdot (x-\mu_0)\\
\sigma_1^2 &= \left( 1-K \right) \sigma_0^2
\end{align*}$$

- $\Rightarrow$ Moral: For jointly Gaussian systems, we can do inference simply in one step by using the formulas for conditioning and marginalization.

### Recursive Bayesian Estimation

- Consider the signal $x_t=\theta+\epsilon_t$, where $D_t= \left\{x_1,\ldots,x_t\right\}$ is observed _sequentially_ (over time).

- **Problem**: Derive a recursive algorithm for $p(\theta|D_t)$, i.e., an update rule for (posterior) $p(\theta|D_t)$ based on (prior) $p(\theta|D_{t-1})$ and (new observation) $x_t$.

##### Model specification
- Let's define the estimate after $t$ observations (i.e., our _solution_ ) as $p(\theta|D_t) = \mathcal{N}(\theta\,|\,\mu_t,\sigma_t^2)$.

- We define the joint distribution for $\theta$ and $x_t$, given background $D_{t-1}$, by

$$\begin{align*} p(x_t,\theta \,|\, D_{t-1}) &= p(x_t|\theta) \, p(\theta|D_{t-1}) \\
  &= \underbrace{\mathcal{N}(x_t\,|\, \theta,\sigma^2)}_{\text{likelihood}} \, \underbrace{\mathcal{N}(\theta\,|\,\mu_{t-1},\sigma_{t-1}^2)}_{\text{prior}}
\end{align*}$$

##### Inference

- Use Bayes rule,
$$\begin{align*}
p(\theta|D_t) &= p(\theta|x_t,D_{t-1}) \\
  &\propto p(x_t,\theta | D_{t-1}) \\
  &= p(x_t|\theta) \, p(\theta|D_{t-1}) \\
  &= \mathcal{N}(x_t|\theta,\sigma^2) \, \mathcal{N}(\theta\,|\,\mu_{t-1},\sigma_{t-1}^2) \\
  &= \mathcal{N}(\theta|x_t,\sigma^2) \, \mathcal{N}(\theta\,|\,\mu_{t-1},\sigma_{t-1}^2) \;\;\text{(note this trick)}\\
  &= \mathcal{N}(\theta|\mu_t,\sigma_t^2) \;\;\text{(use Gaussian multiplication formula SRG-6)}
\end{align*}$$
with
$$\begin{align*}
K_t &= \frac{\sigma_{t-1}^2}{\sigma_{t-1}^2+\sigma^2} \qquad \text{(Kalman gain)}\\
\mu_t &= \mu_{t-1} + K_t \cdot (x_t-\mu_{t-1})\\
\sigma_t^2 &= \left( 1-K_t \right) \sigma_{t-1}^2
\end{align*}$$

- This linear _sequential_ estimator of mean and variance in Gaussian observations is called a **Kalman Filter**.

- Note that the uncertainty about $\theta$ decreases over time (since $0<(1-K_t)<1$). Since we assume that the statistics of the system do not change (stationarity), each new sample provides new information.

- Recursive Bayesian estimation is the basis for **adaptive signal processing** algorithms such as Least Mean Squares (LMS) and Recursive Least Squares (RLS).

### Code Example: Kalman Filter

- Let's implement the Kalman filter described above. We'll use it to recursively estimate the value of $\theta$ based on noisy observations.

```julia
using PyPlot

N = 50                         # Number of observations
θ = 2.0                        # True value of the variable we want to estimate
σ_ϵ2 = 0.25                    # Observation noise variance
x = sqrt(σ_ϵ2) * randn(N) .+ θ # Generate N noisy observations of θ

t = 0
μ = fill!(Vector{Float64}(undef,N), NaN)    # Means of p(θ|D) over time
σ_μ2 = fill!(Vector{Float64}(undef,N), NaN) # Variances of p(θ|D) over time

function performKalmanStep()
    # Perform a Kalman filter step, update t, μ, σ_μ2
    global t += 1
    if t>1 # Use posterior from prev. step as prior
        K = σ_μ2[t-1] / (σ_ϵ2 + σ_μ2[t-1]) # Kalman gain
        μ[t] = μ[t-1] + K*(x[t] - μ[t-1])  # Update mean using (1)
        σ_μ2[t] = σ_μ2[t-1] * (1.0-K)      # Update variance using (2)
    elseif t==1 # Use prior
        # Prior p(θ) = N(0,1000)
        K = 1000.0 / (σ_ϵ2 + 1000.0) # Kalman gain
        μ[t] = 0 + K*(x[t] - 0)      # Update mean using (1)
        σ_μ2[t] = 1000 * (1.0-K)     # Update variance using (2)
    end
end

while t<N
    performKalmanStep()
end

# Plot the 'true' value of θ, noisy observations x, and the recursively updated posterior p(θ|D)
t = collect(1:N)
plot(t, θ*ones(N), "k--")
plot(t, x, "rx")
plot(t, μ, "b-")
fill_between(t, μ-sqrt.(σ_μ2), μ+sqrt.(σ_μ2), color="b", alpha=0.3)
legend([L"\theta", L"x[t]", L"\mu[t]"])
xlim((1, N)); xlabel(L"t"); grid()
```

The shaded area represents 2 standard deviations of posterior $p(\theta|D)$. The variance of the posterior is guaranteed to decrease monotonically for the standard Kalman filter.

###  Product of Normally Distributed Variables
- (We've seen that) the sum of two Gausssian distributed variables is also Gaussian distributed.

- Has the _product_ of two Gaussian distributed variables also a Gaussian distribution?

- **No**! In general this is a difficult computation. As an example, let's compute $p(z)$ for $Z=XY$ for the special case that $X\sim \mathcal{N}(0,1)$ and $Y\sim \mathcal{N}(0,1)$.
$$\begin{align*}
p(z) &= \int_{X,Y} p(z|x,y)\,p(x,y)\,\mathrm{d}x\mathrm{d}y \\
  &= \frac{1}{2 \pi}\int  \delta(z-xy) \, e^{-(x^2+y^2)/2} \, \mathrm{d}x\mathrm{d}y \\
  &=  \frac{1}{\pi} \int_0^\infty \frac{1}{x} e^{-(x^2+z^2/x^2)/2} \, \mathrm{d}x \\
  &= \frac{1}{\pi} \mathrm{K}_0( \lvert z\rvert )\,.
\end{align*}$$
where  $\mathrm{K}_n(z)$ is a [modified Bessel function of the second kind](http://mathworld.wolfram.com/ModifiedBesselFunctionoftheSecondKind.html).

### Code Example: Product of Gaussian Distributions

- We plot $p(Z=XY)$ and $p(X)p(Y)$ for $X\sim\mathcal{N}(0,1)$ and $Y \sim \mathcal{N}(0,1)$ to give an idea of how these distributions differ.

```julia
using PyPlot, Distributions, SpecialFunctions
X = Normal(0,1)
Y = Normal(0,1)
pdf_product_std_normals(z::Vector) = (besselk.(0, abs.(z))./π)
range1 = collect(range(-4,stop=4,length=100))
plot(range1, pdf.(X, range1))
plot(range1, pdf.(X,range1).*pdf.(Y,range1))
plot(range1, pdf_product_std_normals(range1))
legend([L"p(X)=p(Y)=\mathcal{N}(0,1)", L"p(X)*p(Y)",L"p(Z=X*Y)"]); grid()
```

- In short, Gaussian-distributed variables remain Gaussian in linear systems, but this is not the case in non-linear systems.

### Solution to Example Problem

We apply maximum likelihood estimation to fit a 2-dimensional Gaussian model ($m$) to data set $D$. Next, we evaluate $p(x_\bullet \in S | m)$ by (numerical) integration of the Gaussian pdf over $S$: $p(x_\bullet \in S | m) = \int_S p(x|m) \mathrm{d}x$.

```julia
using HCubature, LinearAlgebra# Numerical integration package
# Maximum likelihood estimation of 2D Gaussian
N = length(sum(D,dims=1))
μ = 1/N * sum(D,dims=2)[:,1]
D_min_μ = D - repeat(μ, 1, N)
Σ = Hermitian(1/N * D_min_μ*D_min_μ')
m = MvNormal(μ, convert(Matrix, Σ));

# # Contour plot of estimated Gaussian density
A = Matrix{Float64}(undef,100,100); B = Matrix{Float64}(undef,100,100)
density = Matrix{Float64}(undef,100,100)
for i=1:100
    for j=1:100
        A[i,j] = a = (i-1)*6/100 .- 2
        B[i,j] = b = (j-1)*6/100 .- 3
        density[i,j] = pdf(m, [a,b])
    end
end
c = contour(A, B, density, 6, zorder=1)
PyPlot.set_cmap("cool")
clabel(c, inline=1, fontsize=10)

# Plot observations, x∙, and the countours of the estimated Gausian density
plotObservations(D)
plot(x_dot[1], x_dot[2], "ro")

# Numerical integration of p(x|m) over S:
(val,err) = hcubature((x)->pdf(m,x), [0., 1.], [2., 2.])
println("p(x⋅∈S|m) ≈ $(val)")
```

### Summary

- A linear transformation of a Gaussian-distributed (potentially multivariate) variable remains Gaussian.

- Bayesian inference with Gaussian prior and likelihood leads to an analytically computable Gaussian posterior.

- Here's a nice [summary of Gaussian calculations](https://github.com/bertdv/AIP-5SSB0/raw/master/lessons/notebooks/files/RoweisS-gaussian_formulas.pdf) by Sam Roweis.

##  OPTIONAL SLIDES

### Inference for the Precision Parameter of the Gaussian

- Again, we consider an observed data set $D = \{x_1, x_2, \ldots, x_N\}$ and try to explain these data by a Gaussian distribution.

- We discussed earlier Bayesian inference for the mean with a given variance. Now we will derive a posterior for the variance if the mean is given. (Technically, we will do the derivation for a precision parameter $\lambda = \sigma^{-2}$, since the discussion is a bit more straightforward for the precision parameter).

##### model specification
- The likelihood for the precision parameter is
$$\begin{align*}
p(D|\lambda) &= \prod_{n=1}^N \mathcal{N}\left(x_n \,|\, \mu, \lambda^{-1} \right) \\
  &\propto \lambda^{N/2} \exp\left\{ -\frac{\lambda}{2}\sum_{n=1}^N \left(x_n - \mu \right)^2\right\} \tag{B-2.145}
\end{align*}$$

- The conjugate distribution for this function of $\lambda$ is the [_Gamma_ distribution](https://en.wikipedia.org/wiki/Gamma_distribution), given by
$$
p(\lambda\,|\,a,b) = \mathrm{Gam}\left( \lambda\,|\,a,b \right) \triangleq \frac{1}{\Gamma(a)} b^{a} \lambda^{a-1} \exp\left\{ -b \lambda\right\}\,, \tag{B-2.146}
$$
where $a>0$ and $b>0$ are known as the _shape_ and _rate_ parameters, respectively.

- (Bishop fig.2.13). Plots of the Gamma distribution $\mathrm{Gam}\left( \lambda\,|\,a,b \right) $ for different values of $a$ and $b$.

- The mean and variance of the Gamma distribution evaluate to $\mathrm{E}\left( \lambda\right) = \frac{a}{b}$ and $\mathrm{var}\left[\lambda\right] = \frac{a}{b^2}$.

##### inference

- We will consider a prior $p(\lambda) = \mathrm{Gam}\left( \lambda\,|\,a_0, b_0\right)$, which leads by Bayes rule to the posterior
$$\begin{align*}
p(\lambda\,|\,D) &\propto \underbrace{\lambda^{N/2} \exp\left\{ -\frac{\lambda}{2}\sum_{n=1}^N \left(x_n - \mu \right)^2\right\} }_{\text{likelihood}} \cdot \underbrace{\frac{1}{\Gamma(a_0)} b_0^{a_0} \lambda^{a_0-1} \exp\left\{ -b_0 \lambda\right\}}_{\text{prior}} \\
  &\propto \mathrm{Gam}\left( \lambda\,|\,a_N,b_N \right)
\end{align*}$$
with
$$\begin{align*}
a_N &= a_0 + \frac{N}{2} \tag{B-2.150} \\
b_N &= b_0 + \frac{1}{2}\sum_n \left( x_n-\mu\right)^2 \tag{B-2.151}
\end{align*}$$

- Hence the **posterior is again a Gamma distribution**. By inspection of B-2.150 and B-2.151, we deduce that we can interpret $2a_0$ as the number of a priori (pseudo-)observations.

- Since the most uninformative prior is given by $a_0=b_0 \rightarrow 0$, we can derive the **maximum likelihood estimate** for the precision as
$$
\lambda_{\text{ML}} = \left.\mathrm{E}\left[ \lambda\right]\right\vert_{a_0=b_0\rightarrow 0} = \left. \frac{a_N}{b_N}\right\vert_{a_0=b_0\rightarrow 0} = \frac{N}{\sum_{n=1}^N \left(x_n-\mu \right)^2}
$$

- In short, if we do density estimation with a Gaussian distribution $\mathcal{N}\left(x_n\,|\,\mu,\sigma^2 \right)$ for an observed data set $D = \{x_1, x_2, \ldots, x_N\}$, the maximum likelihood estimates for $\mu$ and $\sigma^2$ are given by
$$\begin{align*}
\mu_{\text{ML}} &= \frac{1}{N} \sum_{n=1}^N x_n \tag{B-2.121} \\
\sigma^2_{\text{ML}} &= \frac{1}{N} \sum_{n=1}^N \left(x_n - \mu_{\text{ML}} \right)^2 \tag{B-2.122}
\end{align*}$$

- These estimates are also known as the _sample mean_ and _sample variance_ respectively.

```julia
open("../../styles/aipstyle.html") do f
    display("text/html", read(f, String))
end
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>

```julia

```


```python
%matplotlib inline

import numpy as np; from matplotlib import pyplot as plt; import pyross
from multiprocessing import Pool; import cma; import time
from functools import partial
```

## Introduction

In the present example, a control protocol is optimised. The protocol consists of two durations, namely

* the duration $T_1$ of a full lockdown (during which the initial contact matrix reduced via a prefactor 1/4), and
* the duration $T_2$ a subsequent partial lockdown (during which the initial contact matrix reduced via a prefactor 3/4).

The cost of a protocol is defined as

\begin{equation}
C=exp\left[ \kappa\left(  T_1+ T_2/5 \right)\right] + \int_0^{t_f}\mathrm{d}t\,I_s(t),
\end{equation}

where $t_f = 1000$ days is the final time of the simulation, and $I_s(t)$ is the total number of symptomatic infectives at time $t$. The first term in the cost function can be thought of as a cost for social distancing, while the second terms models a cost of infection. We additionally set $C = \infty$ if at any time $I_s(t) > 2000$, which serves as a hard constraint that every protocol must keep the number of infectives below this threshold.

For an SIIR model with two age groups, we then minimise the cost as a function of $(T_1,T_2)$, considering three distinct scenarios: i) low cost of social distancing ($\kappa = 0.01$), ii) intermediate cost of social distancing ($\kappa = 0.1$), and iii) high cost of social distancing ($\kappa = 1$). Finally, we compare the time series corresponding to these optimal protocols.

*This is the Jupyter notebook version of an example from the PyRoss manual.*

## Define system parameters

```python
# 1. Define model
M  = 2               # the population has two age groups
N  =  5e4            # and this is the total population
# exact params of reference trajectory
beta  = 0.02         # infection rate
gIa   = 1./7         # removal rate of asymptomatic infectives
gIs   = 1./7         # removal rate of asymptomatic infectives
alpha = 0.2          # fraction of asymptomatic infectives
fsa   = 0.8          # the self-isolation parameter
# set the age structure
fi = np.array([0.25, 0.75])  # fraction of population in age age group
Ni = N*fi
# set up initial condition
Ia0 = np.array([10, 10])  # each age group has asymptomatic infectives
Is0 = np.array([10, 10])   # and also symptomatic infectives
R0  = np.array([0, 0])  # there are no recovered individuals initially
S0  = Ni - (Ia0 + Is0 + R0) # initial number of susceptibles
```

## Define function that generates two-step release protocol

```python
# 2. Define function that generates two-step release protocol
def get_events(protocol= [50,100]):
    end_phase_1, end_phase_2 = protocol
    C = np.array([[18., 9.], [3., 12.]])
    events = []; contactMatrices = []
    # For technical reasons, currently the event functions take a
    # vector argument "rp" instead of a tuple (S, Ia, Is).
    # When defining the event functions, use the following correspondences:
    # S  == rp[ :M]
    # Ia == rp[1*M:2*M]
    # Is == rp[2*M:3*M]
    # More generally, for any model the order of the populations
    # within the vector rp is the same as used for the initial
    # conditions in the model.simulate() function call.
    #
    # Event 0: dummy event used to set the initial contact matrix
    events.append( lambda t, rp: 1. )
    contactMatrices.append( C )
    # Event 1: contact matrix reduced by a factor of 4 once number
    # of symptomatic infectives exceeds a threshold ("start of full lockdown")
    lockdown_threshold_on = 1900
    def event1(t,rp):
        return np.sum(rp[2*M:3*M]) - lockdown_threshold_on
    event1.direction = +1 # need to pass threshold from below for event
    events.append(event1); contactMatrices.append( 0.25*C )
    # Event 2: contact matrix at 75% of original contact matrix after
    # a given time ("start of partial lockdown").
    def event2(t,rp):
        return t - end_phase_1
    events.append(event2); contactMatrices.append( 0.75*C )
    # Event 3: original contact matrix is restored after a given time
    # ("end of partial lockdown").
    def event3(t,rp):
        return t - end_phase_2
    events.append(event3); contactMatrices.append( C )
    return events, contactMatrices
```

## Define cost function, and function to evaluate cost of a given protocol

```python
# 3. Define cost function
maximal_value_for_Is = 2000
def evaluate_cost_function(t_arr,traj,protocol,events_occured,
                           sdc_prefactor=1.,
                           sdc_rate=1/7.):
    infinity = 1e300 # using np.inf would be more proper, but will
    # result in warnings throughout the minimization once a protocol
    # is probed that violates one of the hard constraints.
    #
    # hard constraints:
    # - times of protocol need to be non-negative
    if (np.array(protocol) < 0).any(): return infinity
    # - second stage of intervention needs to end after first stage ends
    if protocol[0] > protocol[1]: return infinity
    # - second stage of intervention needs to end before 700 days
    if protocol[1] > 700: return infinity
    # - number of symptomatic infectives should
    #   never exceed a given maximal value
    if np.max( np.sum( traj[2*M:3*M],axis=0 ) ) > maximal_value_for_Is:
        return infinity
    #
    # Evaluate results of protocol.
    # Which phases have occured in the simulation?
    have_lockdown_start = False
    have_lockdown_end_0 = False
    have_lockdown_end_1 = False
    for i,e in enumerate(events_occured):
        if e[1] == 1:
            lockdown_start = e[0]
            have_lockdown_start = True
        elif e[1] == 2:
            lockdown_end_0 = e[0]
            have_lockdown_end_0 = True
        elif e[1] == 3:
            lockdown_end_1 = e[0]
            have_lockdown_end_1 = True
    #
    # calculate cost function
    cost = 0.
    # cost of social distancing
    if have_lockdown_start:
        if have_lockdown_end_0:
            cost += sdc_prefactor* np.exp(sdc_rate* (lockdown_end_0-lockdown_start))
        else:
            cost += infinity
        #
        if have_lockdown_end_0 and have_lockdown_end_1:
            cost *=  np.exp(0.2*sdc_rate* (lockdown_end_1-lockdown_end_0))
    # cost of infection
    cost += np.trapz( np.sum( traj[2*M:3*M],axis= 0 ),t_arr)
    return cost

# 4. Define function that evaluates the cost of a given protocol
def evaluate_cost_of_protocol(model,protocol,
                             verbose=False,return_trajectory=False,
                              sdc_prefactor=10.,
                              sdc_rate=1/7.):
    # define events corresponding to given protocol
    events, contactMatrices = get_events(protocol)
    # run simulation
    Tf = 1000; Nt = (Tf +1)*10
    result = model.simulate(S0,  Ia0, Is0,
                            events=events,contactMatrices=contactMatrices,
                            Tf=Tf, Nf=Nt)
    # evaluate cost
    cost = evaluate_cost_function(t_arr=result['t'],
                                   traj=result['X'].T,
                                   protocol=protocol,
                                  events_occured=result['events_occured'],
                                  sdc_prefactor=sdc_prefactor,
                                  sdc_rate=sdc_rate)
    if return_trajectory:
        return cost, result['t'], result['X'], result['events_occured']
    else:
        return cost
```

## Define minimiser

```python
# 5. Define minimiser
def minimizing_function(sdc_prefactor,sdc_rate,
                    protocol):
    '''
    This is the function that will be minimised by the cma-es algorithm
    '''
    costs = evaluate_cost_of_protocol(model=model,
                                             protocol=protocol,
                                      sdc_prefactor=sdc_prefactor,
                                      sdc_rate=sdc_rate,
                                              return_trajectory=False)
    return costs

def find_optimal_protocol(initial_guess,sdc_prefactor,sdc_rate,
                          model,
                         verbose=True,initial_variance=5.):
    func = partial(minimizing_function, sdc_prefactor, sdc_rate)
    # set parameters for minimisation based on cma-es algorithm
    number_of_threads = 4
    number_iterations = 1000
    p = Pool(number_of_threads)
    options = cma.CMAOptions()
    options['popsize'] = 12
    # run minimisation
    start_time = time.time()
    optim = cma.CMAEvolutionStrategy(initial_guess, initial_variance,
                                     options)
    iteration = 0
    while not optim.stop() and iteration < number_iterations:
        positions = optim.ask()
        values = p.map(func, positions)
        optim.tell(positions, values)
        optim.disp()
        iteration += 1
    end_time = time.time()
    if verbose:
        print("Time: ", end_time - start_time)
        print("Value: ", optim.best.f)
        print("Parameters: ",optim.best.x)
    return optim.best.x
```

## Find optimal protocol for low-, intermediate- and high cost of social distancing

```python
# 6. Find optimal protocol for three different scenarios

# Initialise model
parameters = {'alpha':alpha,'beta':beta, 'gIa':gIa,'gIs':gIs,'fsa':fsa}
model = pyross.control.SIR(parameters, M, Ni)
initial_guess=[70,200] # use same initial guess for all minimisations
sdc_prefactor = 1. # means that social distancing cost and
# cost of infection have the same weight in our model

# Low, intermediate, and high cost rate for social distancing:
sdc_rates = [1e-2,1e-1,1.]
optimal_protocols = []
for i,sdc_rate in enumerate(sdc_rates):
    optimal_protocol = find_optimal_protocol(initial_guess=initial_guess,
                                           sdc_prefactor=sdc_prefactor,
                                           sdc_rate=sdc_rate,
                                              model=model)
    optimal_protocols.append(optimal_protocol)
```

    (6_w,12)-aCMA-ES (mu_w=3.7,w_1=40%) in dimension 2 (seed=729858, Fri Jul 30 16:26:28 2021)
    Iterat #Fevals   function value  axis ratio  sigma  min&max std  t[m:s]
        1     12 1.551522901619991e+05 1.0e+00 8.48e+00  9e+00  1e+01 0:00.1
        2     24 1.530396739480683e+05 1.2e+00 1.60e+01  2e+01  2e+01 0:00.1
        3     36 1.525286871792975e+05 1.1e+00 1.66e+01  2e+01  2e+01 0:00.2
       53    636 1.484973065739169e+05 9.3e+01 1.47e+01  2e-02  2e+00 0:03.2
      100   1200 1.484973047095792e+05 2.1e+00 7.77e+00  9e-04  3e-03 0:06.2
      171   2052 1.484973045851920e+05 4.7e+00 2.20e-01  8e-09  3e-08 0:11.2
      193   2316 1.484973045851915e+05 6.0e+00 3.74e-02  8e-11  4e-10 0:12.7
    Time:  12.72672986984253
    Value:  148496.56342598566
    Parameters:  [154.7295758  605.20315497]
    (6_w,12)-aCMA-ES (mu_w=3.7,w_1=40%) in dimension 2 (seed=821324, Fri Jul 30 16:26:40 2021)
    Iterat #Fevals   function value  axis ratio  sigma  min&max std  t[m:s]
        1     12 1.555910229721307e+05 1.0e+00 1.14e+01  1e+01  2e+01 0:00.1
        2     24 1.543717959775847e+05 1.7e+00 1.86e+01  2e+01  2e+01 0:00.1
        3     36 1.538711070378818e+05 1.3e+00 1.53e+01  1e+01  2e+01 0:00.2
       58    696 1.534246299007369e+05 7.7e+01 3.34e+00  3e-03  2e-01 0:03.2
      100   1200 1.534238328470859e+05 4.1e+00 2.84e+00  4e-04  1e-03 0:05.3
      197   2364 1.534237853960470e+05 3.4e+00 1.54e-02  1e-11  7e-11 0:10.4
      200   2400 1.534237853960474e+05 3.4e+00 1.31e-02  1e-11  3e-11 0:10.5
      215   2580 1.534237853960458e+05 9.2e+00 1.49e-02  1e-12  9e-12 0:11.3
    Time:  11.379749059677124
    Value:  153423.78539604548
    Parameters:  [ 59.64631831 236.18743131]
    (6_w,12)-aCMA-ES (mu_w=3.7,w_1=40%) in dimension 2 (seed=772511, Fri Jul 30 16:26:52 2021)
    Iterat #Fevals   function value  axis ratio  sigma  min&max std  t[m:s]
        1     12 8.736166571505890e+29 1.0e+00 7.58e+00  7e+00  8e+00 0:00.1
        2     24 3.304332163831881e+24 1.3e+00 1.41e+01  1e+01  2e+01 0:00.1
        3     36 1.267525431905797e+24 1.4e+00 1.76e+01  1e+01  3e+01 0:00.2
       51    612 2.778102227473433e+15 4.9e+00 8.50e-01  2e-03  1e-02 0:03.2
      100   1200 2.776126888070033e+15 3.6e+00 1.26e-02  1e-07  2e-07 0:06.4
      158   1896 2.776126316996960e+15 3.3e+00 4.04e-04  3e-12  9e-12 0:10.0
    Time:  10.013539791107178
    Value:  2776126316994518.5
    Parameters:  [45.0594465  98.41142971]

## Visualise results

```python
# 7. Visualise results
def plot_result(protocol,model,
                title='Optimal protocol',
                t_max=None):
    #
    cost, t_arr, traj, events_occured = evaluate_cost_of_protocol(
                                            model=model,
                                             protocol=protocol,
                                              return_trajectory=True)
    #
    Is = np.sum(traj[:,2*M:],axis=-1)
    #
    fig, ax = plt.subplots(1,1,figsize=(7,5))
    ax.set_title(title)
    ax.axhline(maximal_value_for_Is,color='red',ls='--',
              label=r'Constraint')
    #
    ax.axvspan(events_occured[0][0],events_occured[1][0],
              color='crimson',label='Intervention 1',alpha=0.2)
    ax.axvspan(events_occured[1][0],events_occured[2][0],
              color='orange',label='Intervention 2',alpha=0.2)
    ax.plot(t_arr,Is)
    if t_max == None:
        ax.set_xlim(np.min(t_arr),np.max(t_arr))
    else:
        ax.set_xlim(np.min(t_arr),t_max)
    ax.set_ylabel('Symptomatic infectives'); ax.set_xlabel(r'time [days]')
    ax.legend(loc='upper right',fontsize=12,framealpha=1.)
    #if title != 'Optimal protocol':
    #    fig.savefig(title+'.pdf',bbox_inches='tight')
    plt.show(); plt.close()

optimal_protocol_labels = ['Low cost of social distancing',
                          'Intermediate cost of social distancing',
                          'High cost of social distancing']
t_max = [700,250,250]

for i,optimal_protocol in enumerate(optimal_protocols):
    plot_result(protocol=optimal_protocol,model=model,t_max=t_max[i],
           title=optimal_protocol_labels[i])
```