# Practical Session 3: Ensemble Learning Techniques

*Notebook by Ekaterina Kochmar*

This practical will address the use of ensemble-based learning techniques. You will be working with the otherwise familiar settings of classification and regression tasks. In this practical, you will use the familiar datasets and will also learn how to generate artificial datasets.

## Learning objectives

In this practical you will learn about:
- simple voting classifiers using hard and soft voting strategies
- bagging and pasting ensembles
- boosting and early stopping
- popular ensemble-based learning algorithms including `RandomForests` and `AdaBoost`

In addition, you will get more practice with `sklearn's` machine learning routines, and you will learn how to implement ensembles both from scratch and using `sklearn's` implementation.

**Bibliography**: Aurelien Geron, *Hands-On Machine Learning with Scikit-Learn and TensorFlow*.

## Setup

First of all, let's import a few common modules, ensure `matplotlib` plots are inline and define the parameters for the figures. Feel free to add your own settings for the notebook here:

```python
# Common imports
import numpy as np
import os

# to make this notebook's output stable across runs
np.random.seed(42)

# To plot pretty figures
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
plt.rcParams['axes.labelsize'] = 14
plt.rcParams['xtick.labelsize'] = 12
plt.rcParams['ytick.labelsize'] = 12
```

## Simple voting classifiers: hard and soft voting strategies

[The wisdom of the crowd](https://en.wikipedia.org/wiki/Wisdom_of_the_crowd) is the collective opinion of a group of individuals rather than that of a single expert. It has been shown that if you ask a complex question to thousands of random people and then aggregate their answers, in many cases you are likely to get an answer that is more accurate than an expert's prediction. This effect is nowadays widely used in practice and can be said to provide the basis for crowdsourcing approaches.

By analogy, if you combine decisions of multiple diverse classifiers, the resulting vote will often be better than a vote of any single predictor. Such combination of classifiers is called _ensemble_. Perhaps somewhat counterintuitively, the aggregated vote of an ensemble often achieves a higher accuracy than the best classifier in the ensemble. This works even if each predictor in the ensemble is a _weak learner_ **(!)** (meaning that it works only slightly better than random guessing). The ensemble can still be a _strong learner_ achieving high accuracy, provided that there are:
- a sufficient number of weak learners, and
- their decisions are sufficiently diverse.

How is this possible? Here is a statistical analogy: suppose you have a slightly biased coin that has a $51\%$ chance of coming up heads and a $49\%$ chance of coming up tails. If you toss it $1000$ times, you will get around $510$ heads and $490$ tails, and hence a majority of heads. If you do the math or run simulations, you will find out that the probability of obtaining the majority of heads after $1000$ tosses of this coin approaches $73\%$. After $10K$ tosses, the probability of having a majority of heads is about $97\%$. This is explained by the [_law of large numbers_](https://en.wikipedia.org/wiki/Law_of_large_numbers): over a large number of tosses the ratio of heads gets closer to the probability of heads ($51\%$).

```python
import scipy, math
import scipy.special

def probability(p, n, x):
    binom = scipy.special.comb(n, x, exact=True) * math.pow(p, x) * math.pow((1-p), n-x)
    return binom

prob_maj = 0.0
for x in range(501, 1001):
    prob_maj += probability(0.51, 1000, x)

print(prob_maj)
```

    0.7260985557305037

The code below simulates $10$ independent series of coin tosses, and then plots the results. You can see in the figure below that as the number of tosses increases, all $10$ series end up close to $51\%$ and thus are consistently above $50\%$.

```python
np.random.seed(42)
heads_proba = 0.51
coin_tosses = (np.random.rand(10000, 10) < heads_proba).astype(np.int32)
cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)

plt.figure(figsize=(8,3.5))
plt.plot(cumulative_heads_ratio)
plt.plot([0, 10000], [0.51, 0.51], "k--", linewidth=2, label="51%")
plt.plot([0, 10000], [0.5, 0.5], "k-", label="50%")
plt.xlabel("Number of coin tosses")
plt.ylabel("Heads ratio")
plt.legend(loc="lower right")
plt.axis([0, 10000, 0.42, 0.58])
plt.show()
```

Similarly, if you combine $1000$ classifiers, each of which is individually correct only $51\%$ of the time, and then predict the majority voted class, you can get up to $73\%$ accuracy.

For this to work, the classifiers need to be independent and sufficiently diverse, i.e. they need to make **independent, uncorrelated errors**. In practice, this is hard to achieve because the classifiers are usually all trained on the same data.

Let's apply voting strategy to some data: this time, we'll be working with a synthetic (i.e., artificially generated) [`moons` dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html), that includes two interleaving half circles and provides a good 'toy' example for testing out classification strategies.

Let's first generate the data points and visualise them on a plot:

```python
from sklearn.datasets import make_moons
X, y = make_moons(n_samples=500, noise=0.30, random_state=42)

def plot_dataset(X, y, axes):
    plt.plot(X[:, 0][y==0], X[:, 1][y==0], "bs")
    plt.plot(X[:, 0][y==1], X[:, 1][y==1], "g^")
    plt.axis(axes)
    plt.grid(True, which='both')
    plt.xlabel(r"$x_1$", fontsize=20)
    plt.ylabel(r"$x_2$", fontsize=20, rotation=0)

plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
plt.show()
```

Next, let's split the dataset into training and test sets and train some classifiers on the training data:

```python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
```

The code below shows how to apply several classifiers to this data, with a couple of classifiers that we haven't used before in this course: [`RandomForestClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) and [`Support Vector Machines`](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html) (`SVC`) classifier. For the moment, don't worry about the particular details of the classifiers' implementation (if you are interested in learning more about them, check out `sklearn's` documentation; we'll discuss `RandomForestClassifier` in this practical; `Support Vector Machines` is covered in detail in the Part II [Machine Learning and Bayesian Inference](https://www.cl.cam.ac.uk/teaching/2021/MLBayInfer/) course). The point here is that we are combining *diverse* classifiers in an ensemble: in essence, `RandomForestClassifier` runs recursive partitioning on the data, while `Support Vector Machines` try to construct a linear boundary between the classes (as discussed in Lecture 3). Feel free to experiment with your own selection of classifiers from the [`sklearn's`](https://scikit-learn.org/stable/supervised_learning.html#supervised-learning) suite.

```python
from sklearn.ensemble import VotingClassifier

from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC

log_clf = LogisticRegression(random_state=42)
rnd_clf = RandomForestClassifier(random_state=42)
svm_clf = SVC(random_state=42)

voting_clf = VotingClassifier(
    estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
    voting='hard')
voting_clf.fit(X_train, y_train)
```

    VotingClassifier(estimators=[('lr', LogisticRegression(random_state=42)),
                                 ('rf', RandomForestClassifier(random_state=42)),
                                 ('svc', SVC(random_state=42))])

Let's look into how correlated the predictions of the classifiers are: first, get the predictions on the test data, next store them as a `pandas` DataFrame, and finally apply `.corr()` function:

```python
import pandas as pd

def get_predictions(clf):
    clf.fit(X_train, y_train)
    return clf.predict(X_test)

preds = {'lr': get_predictions(log_clf),
        'rf': get_predictions(rnd_clf),
        'svc': get_predictions(svm_clf)}
df = pd.DataFrame(data=preds)
df[:100]
```

      lr
      rf
      svc

      0
      1
      1
      0

      1
      0
      0
      0

      2
      0
      0
      0

      3
      1
      1
      1

      4
      1
      1
      1

      ...
      ...
      ...
      ...

      95
      1
      1
      1

      96
      0
      0
      0

      97
      1
      0
      0

      98
      0
      0
      0

      99
      1
      1
      1

100 rows × 3 columns

```python
df.corr()
```

      lr
      rf
      svc

      lr
      1.000000
      0.774252
      0.774595

      rf
      0.774252
      1.000000
      0.872101

      svc
      0.774595
      0.872101
      1.000000

Will this correlation in the individual classifiers' predictions be sufficient for an ensemble? Let's check this out by combining the votes with a *hard voting strategy*: the ensemble classifier will simply choose the majority class predicted by the three classifiers.

The code below prints out individual classifiers' accuracy scores along with the ensemble's accuracy score:

```python
from sklearn.metrics import accuracy_score

for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
```

    LogisticRegression 0.864
    RandomForestClassifier 0.896
    SVC 0.896
    VotingClassifier 0.912

As you can see, the voting classifier slightly outperforms each of the three individual classifiers, including the most accurate one(s). In addition, if individual classifiers can estimate class probabilities (i.e., if they have `predict_proba()` method), you can use _soft voting_ strategy, that is, estimate highest class probability averaged over individual classifiers. In comparison to the hard voting strategy, soft voting gives more weight to highly confident votes.

`SVC` classifier doesn't estimate class probabilities by default, so you need to set the `probability` hyperparameter to `True`.

```python
log_clf = LogisticRegression(random_state=42)
rnd_clf = RandomForestClassifier(random_state=42)
svm_clf = SVC(probability=True, random_state=42)

voting_clf = VotingClassifier(
    estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
    voting='soft')
voting_clf.fit(X_train, y_train)
```

    VotingClassifier(estimators=[('lr', LogisticRegression(random_state=42)),
                                 ('rf', RandomForestClassifier(random_state=42)),
                                 ('svc', SVC(probability=True, random_state=42))],
                     voting='soft')

Not let's estimate the accuracy of the voting classifier in this mode:

```python
from sklearn.metrics import accuracy_score

for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
```

    LogisticRegression 0.864
    RandomForestClassifier 0.896
    SVC 0.896
    VotingClassifier 0.92

## Bagging ensembles: bagging and pasting

One way to ensure that the classifiers are sufficiently diverse in their predicitons is to select different training algorithms. As we said before, often it is hard to ensure that the classifiers' errors are not highly correlated because the classifiers are ultimately trained on the same data.

So, another way to make sure you end up with diverse classifiers is to train them on different random subsets of the training set. There are two ways to do that:
- sampling _with replacement_ is called [_bagging_](https://link.springer.com/article/10.1023/A:1018054314350) – short for _bootstrap aggregating_
- sampling _without replacement_ is called [_pasting_](https://link.springer.com/article/10.1023/A:1007563306331)

Both bagging and pasting allow training instances to be sampled several times across multiple predictors, but only bagging allows training instances to be sampled several times for the same predictor.

When applied to a new, test instance the ensemble classifier aggregates the predictions of all predictors and estimates the statistical mode (i.e., most frequent prediction).

**Question**: Is this similar to the hard or the soft voting strategy?

**Ans**: This is similar to the _hard_ voting strategy, as it just picks the most frequest predicition.

Let's use `sklearn's BaggingClassifier` to train an ensemble of $500$ [`Decision Trees`](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html), each trained on $100$ training instances randomly selected from the training set with replacement (for bagging, but you can change the strategy to pasting by setting `bootstrap` to `False`). In addition, both bagging and pasting strategies are good for parallelisation: predictors can all be trained in parallel on different CPU cores or even different servers. Setting `n_jobs` parameter to $-1$ tells `sklearn` to use all available CPU cores:

```python
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

#use bootstrap=False for pasting
bag_clf = BaggingClassifier(
    DecisionTreeClassifier(random_state=42), n_estimators=500,
    max_samples=100, bootstrap=True, n_jobs=-1, random_state=42)
#n_jobs = use all of the available CPU cores
bag_clf.fit(X_train, y_train)
y_pred = bag_clf.predict(X_test)
```

**Question**: [`BaggingClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingClassifier.html) automatically performs soft voting when the base classifier can estimate class probabilities (i.e., has a `predict_proba()` method). Which strategy – hard or soft voting – does `BaggingClassifier` follow with the `DecisionTreeClassifier` as the base classifier?

**Ans**: The `DecisionTreeClassifier` has a `predict_proba()` with default settings. Hence, the `BaggingClassifier` will use soft voting.

Let's calculate the accuracy score for the bagging classifier that combines $500$ Decision Tree estimators with a prediction of a single Decision Tree trained on the same data:

```python
print(accuracy_score(y_test, y_pred))
```

    0.904

```python
tree_clf = DecisionTreeClassifier(random_state=42)
tree_clf.fit(X_train, y_train)
y_pred_tree = tree_clf.predict(X_test)
print(accuracy_score(y_test, y_pred_tree))
```

    0.856

To get more insight into the results, let's also plot the decision boundary for the two classifiers:

```python
from matplotlib.colors import ListedColormap

def plot_decision_boundary(clf, X, y, axes=[-1.5, 2.5, -1, 1.5], alpha=0.5, contour=True):
    x1s = np.linspace(axes[0], axes[1], 100)
    x2s = np.linspace(axes[2], axes[3], 100)
    x1, x2 = np.meshgrid(x1s, x2s)
    X_new = np.c_[x1.ravel(), x2.ravel()]
    y_pred = clf.predict(X_new).reshape(x1.shape)
    custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
    plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)
    if contour:
        custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])
        plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=0.8)
    plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo", alpha=alpha)
    plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs", alpha=alpha)
    plt.axis(axes)
    plt.xlabel(r"$x_1$", fontsize=18)
    plt.ylabel(r"$x_2$", fontsize=18, rotation=0)

plt.figure(figsize=(11,4))
plt.subplot(121)
plot_decision_boundary(tree_clf, X, y)
plt.title("Decision Tree", fontsize=14)
plt.subplot(122)
plot_decision_boundary(bag_clf, X, y)
plt.title("Decision Trees with Bagging", fontsize=14)
plt.show()
```

Even though the ensemble makes a comparable number of errors on the training set as a single Decision Tree, its decision boundary is less irregular. This suggests that ensemble's predictions will likely generalise better than the single `Decision Tree`'s predictions
when applied to the new data.

### Out-of-Bag evaluation

With bagging, some of the instances get sampled several times for any given predictor, while others may never be selected at all. By default, `BaggingClassifier` samples $m$ training instances with replacement, where $m$ is the size of the training set. As $m$ grows, the ratio of the training instances that are sampled on average for each predictor approaches $1 - exp(-1)$, i.e. around $63\%$. The remaining $37\%$ of the training instances that are never used by the predictors are called *out-of-bag* (*oob*) instances **(!)**.

Mathematical proof of the above: With $m$ training instances in the set, the probability of not picking an instance in a random draw is $\frac{m-1}{m}$. For sampling with replacement, the probability of not picking an instance in $m$ random draws is $(\frac{m-1}{m})^m$. In the limit of a large number of $m$, this becomes:

\begin{equation}
\lim_{m\to\infty} (1 - \frac{1}{m})^m = e^{-1} = 0.368 \approx 37\%
\end{equation}

Since a predictor never sees these instances during training, they can be used for evaluation without the need for an additional validation set or cross-validation experiments. You can evaluate the ensemble itself by averaging each predictor's performance on the oob instances.

The code below shows you how to do that: if you set `oob_score` to `True`, the bagging classifier will be evaluated on the oob instances after training, and you can output the score as `oob_score_`.

```python
bag_clf = BaggingClassifier(
    DecisionTreeClassifier(random_state=42), n_estimators=500,
    bootstrap=True, n_jobs=-1, oob_score=True, random_state=40)
bag_clf.fit(X_train, y_train)
bag_clf.oob_score_
```

    0.8986666666666666

This means that the bagging classifier is likely to achieve around $90\%$ accuracy on the test data, too. Let's check the results:

```python
from sklearn.metrics import accuracy_score
y_pred = bag_clf.predict(X_test)
accuracy_score(y_test, y_pred)
```

    0.912

Close enough! You can also take a look into the decision process by printing out the predictions of the classifier on the training instances. Since the base estimator has a `predict_proba()` method, what you see here are the class probabilities assigned by the classifier to negative and positive classes for each instance:

```python
bag_clf.oob_decision_function_
```

    array([[0.32275132, 0.67724868],
           [0.34117647, 0.65882353],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.09497207, 0.90502793],
           [0.31147541, 0.68852459],
           [0.01754386, 0.98245614],
           [0.97109827, 0.02890173],
           [0.97765363, 0.02234637],
           [0.74404762, 0.25595238],
           [0.        , 1.        ],
           [0.7173913 , 0.2826087 ],
           [0.85026738, 0.14973262],
           [0.97222222, 0.02777778],
           [0.0625    , 0.9375    ],
           [0.        , 1.        ],
           [0.97837838, 0.02162162],
           [0.94642857, 0.05357143],
           [1.        , 0.        ],
           [0.01704545, 0.98295455],
           [0.39473684, 0.60526316],
           [0.88700565, 0.11299435],
           [1.        , 0.        ],
           [0.97790055, 0.02209945],
           [0.        , 1.        ],
           [0.99428571, 0.00571429],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.62569832, 0.37430168],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.13402062, 0.86597938],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.38251366, 0.61748634],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.27093596, 0.72906404],
           [0.34146341, 0.65853659],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.00531915, 0.99468085],
           [0.98843931, 0.01156069],
           [0.91428571, 0.08571429],
           [0.97282609, 0.02717391],
           [0.98019802, 0.01980198],
           [0.        , 1.        ],
           [0.07361963, 0.92638037],
           [0.98019802, 0.01980198],
           [0.0052356 , 0.9947644 ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.97790055, 0.02209945],
           [0.8       , 0.2       ],
           [0.42424242, 0.57575758],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.66477273, 0.33522727],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.86781609, 0.13218391],
           [1.        , 0.        ],
           [0.56725146, 0.43274854],
           [0.1576087 , 0.8423913 ],
           [0.66492147, 0.33507853],
           [0.91709845, 0.08290155],
           [0.        , 1.        ],
           [0.16759777, 0.83240223],
           [0.87434555, 0.12565445],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.995     , 0.005     ],
           [0.        , 1.        ],
           [0.07878788, 0.92121212],
           [0.05418719, 0.94581281],
           [0.29015544, 0.70984456],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.83040936, 0.16959064],
           [0.01092896, 0.98907104],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.21465969, 0.78534031],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.94660194, 0.05339806],
           [0.77094972, 0.22905028],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.16574586, 0.83425414],
           [0.65306122, 0.34693878],
           [0.        , 1.        ],
           [0.02564103, 0.97435897],
           [0.50555556, 0.49444444],
           [1.        , 0.        ],
           [0.03208556, 0.96791444],
           [0.99435028, 0.00564972],
           [0.23699422, 0.76300578],
           [0.49509804, 0.50490196],
           [0.9947644 , 0.0052356 ],
           [0.00555556, 0.99444444],
           [0.98963731, 0.01036269],
           [0.26153846, 0.73846154],
           [0.92972973, 0.07027027],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.80113636, 0.19886364],
           [1.        , 0.        ],
           [0.0106383 , 0.9893617 ],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.98181818, 0.01818182],
           [1.        , 0.        ],
           [0.01036269, 0.98963731],
           [0.97752809, 0.02247191],
           [0.99453552, 0.00546448],
           [0.01960784, 0.98039216],
           [0.17857143, 0.82142857],
           [0.98387097, 0.01612903],
           [0.29533679, 0.70466321],
           [0.98295455, 0.01704545],
           [0.        , 1.        ],
           [0.00561798, 0.99438202],
           [0.75690608, 0.24309392],
           [0.38624339, 0.61375661],
           [0.40625   , 0.59375   ],
           [0.87368421, 0.12631579],
           [0.92462312, 0.07537688],
           [0.05181347, 0.94818653],
           [0.82802548, 0.17197452],
           [0.01546392, 0.98453608],
           [0.        , 1.        ],
           [0.02298851, 0.97701149],
           [0.9726776 , 0.0273224 ],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.01041667, 0.98958333],
           [0.        , 1.        ],
           [0.03804348, 0.96195652],
           [0.02040816, 0.97959184],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.94915254, 0.05084746],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.99462366, 0.00537634],
           [0.        , 1.        ],
           [0.39378238, 0.60621762],
           [0.33152174, 0.66847826],
           [0.00609756, 0.99390244],
           [0.        , 1.        ],
           [0.3172043 , 0.6827957 ],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.00588235, 0.99411765],
           [0.        , 1.        ],
           [0.98924731, 0.01075269],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.62893082, 0.37106918],
           [0.92344498, 0.07655502],
           [0.        , 1.        ],
           [0.99526066, 0.00473934],
           [1.        , 0.        ],
           [0.98888889, 0.01111111],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.06989247, 0.93010753],
           [1.        , 0.        ],
           [0.03608247, 0.96391753],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.02185792, 0.97814208],
           [1.        , 0.        ],
           [0.95808383, 0.04191617],
           [0.78362573, 0.21637427],
           [0.56650246, 0.43349754],
           [0.        , 1.        ],
           [0.18023256, 0.81976744],
           [1.        , 0.        ],
           [0.93121693, 0.06878307],
           [0.97175141, 0.02824859],
           [1.        , 0.        ],
           [0.00531915, 0.99468085],
           [0.        , 1.        ],
           [0.43010753, 0.56989247],
           [0.85858586, 0.14141414],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.00558659, 0.99441341],
           [0.        , 1.        ],
           [0.96923077, 0.03076923],
           [0.        , 1.        ],
           [0.21649485, 0.78350515],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.98477157, 0.01522843],
           [0.8       , 0.2       ],
           [0.99441341, 0.00558659],
           [0.        , 1.        ],
           [0.09497207, 0.90502793],
           [0.99492386, 0.00507614],
           [0.01714286, 0.98285714],
           [0.        , 1.        ],
           [0.02747253, 0.97252747],
           [1.        , 0.        ],
           [0.77005348, 0.22994652],
           [0.        , 1.        ],
           [0.90229885, 0.09770115],
           [0.98387097, 0.01612903],
           [0.22222222, 0.77777778],
           [0.20348837, 0.79651163],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.20338983, 0.79661017],
           [0.98181818, 0.01818182],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.98969072, 0.01030928],
           [0.        , 1.        ],
           [0.48663102, 0.51336898],
           [1.        , 0.        ],
           [0.00529101, 0.99470899],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.08379888, 0.91620112],
           [0.12352941, 0.87647059],
           [0.99415205, 0.00584795],
           [0.03517588, 0.96482412],
           [1.        , 0.        ],
           [0.39790576, 0.60209424],
           [0.05434783, 0.94565217],
           [0.53191489, 0.46808511],
           [0.51898734, 0.48101266],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.60869565, 0.39130435],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.24157303, 0.75842697],
           [0.81578947, 0.18421053],
           [0.08717949, 0.91282051],
           [0.99453552, 0.00546448],
           [0.82142857, 0.17857143],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.11904762, 0.88095238],
           [0.04188482, 0.95811518],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.89150943, 0.10849057],
           [0.19230769, 0.80769231],
           [0.95238095, 0.04761905],
           [0.00515464, 0.99484536],
           [0.59375   , 0.40625   ],
           [0.07692308, 0.92307692],
           [0.99484536, 0.00515464],
           [0.83684211, 0.16315789],
           [0.        , 1.        ],
           [0.99484536, 0.00515464],
           [0.95360825, 0.04639175],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.26395939, 0.73604061],
           [0.98461538, 0.01538462],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.00574713, 0.99425287],
           [0.85142857, 0.14857143],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.75301205, 0.24698795],
           [0.8969697 , 0.1030303 ],
           [1.        , 0.        ],
           [0.75555556, 0.24444444],
           [0.48863636, 0.51136364],
           [0.        , 1.        ],
           [0.92473118, 0.07526882],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.87709497, 0.12290503],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.74752475, 0.25247525],
           [0.09146341, 0.90853659],
           [0.42268041, 0.57731959],
           [0.22395833, 0.77604167],
           [0.        , 1.        ],
           [0.87046632, 0.12953368],
           [0.78212291, 0.21787709],
           [0.00507614, 0.99492386],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.02884615, 0.97115385],
           [0.96      , 0.04      ],
           [0.93478261, 0.06521739],
           [1.        , 0.        ],
           [0.50731707, 0.49268293],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.01604278, 0.98395722],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.96987952, 0.03012048],
           [0.        , 1.        ],
           [0.05172414, 0.94827586],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.99494949, 0.00505051],
           [0.01675978, 0.98324022],
           [1.        , 0.        ],
           [0.14583333, 0.85416667],
           [0.        , 1.        ],
           [0.00546448, 0.99453552],
           [0.        , 1.        ],
           [0.41836735, 0.58163265],
           [0.13095238, 0.86904762],
           [0.22110553, 0.77889447],
           [1.        , 0.        ],
           [0.97647059, 0.02352941],
           [0.21195652, 0.78804348],
           [0.98882682, 0.01117318],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.96428571, 0.03571429],
           [0.34554974, 0.65445026],
           [0.98235294, 0.01764706],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.99465241, 0.00534759],
           [0.        , 1.        ],
           [0.06043956, 0.93956044],
           [0.98214286, 0.01785714],
           [1.        , 0.        ],
           [0.03108808, 0.96891192],
           [0.58854167, 0.41145833]])

## Random Forests

[`RandomForestClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) is an ensemble of Decision Trees typically trained via the bagging method. The number of training instances (`max_samples`) is usually set to the total size of the training set. So, in fact, `RandomForestClassifier` will be roughly equivalent to the `BaggingClassifier` that takes Decision Trees as base estimators with the following parameters:

```python
bag_clf = BaggingClassifier(
    DecisionTreeClassifier(splitter="random", max_leaf_nodes=16, random_state=42),
    n_estimators=500, max_samples=1.0, bootstrap=True, n_jobs=-1, random_state=42)

bag_clf.fit(X_train, y_train)
y_pred = bag_clf.predict(X_test)
```

The key difference is that, rather than using `BaggingClassifier` and passing it a `DecisionTreeClassifier`, you can rely on the `sklearn's RandomForestClassifier` implementation, which is more convenient and optimised for Decision Trees:

```python
from sklearn.ensemble import RandomForestClassifier

rnd_clf = RandomForestClassifier(n_estimators=500, max_leaf_nodes=16,
                                 n_jobs=-1, random_state=42)
rnd_clf.fit(X_train, y_train)

y_pred_rf = rnd_clf.predict(X_test)
```

Let's estimate the difference between the two classifiers' predictions:

```python
np.sum(y_pred == y_pred_rf) / len(y_pred)  # see to what extent predictions are identical
```

    0.976

I.e., the two classifiers are almost identical in their predictions. The Random Forest algorithm introduces extra randomness when growing trees: instead of searching for the very best feature when splitting a node, it searches for the best feature among a random subset of features. As a result, the model is more diverse and in general yields better results.

Let's visualise the decision boundary for a random set of $15$ Decision Trees to get a better idea of where the diversity comes from:

```python
plt.figure(figsize=(6, 4))

for i in range(15):
    tree_clf = DecisionTreeClassifier(max_leaf_nodes=16, random_state=42 + i)
    indices_with_replacement = np.random.randint(0, len(X_train), len(X_train))
    tree_clf.fit(X[indices_with_replacement], y[indices_with_replacement])
    plot_decision_boundary(tree_clf, X, y, axes=[-1.5, 2.5, -1, 1.5],
                           alpha=0.02, contour=False)

plt.show()
```

**Extra material**: One further way to introduce randomness into the training process is to not only select a random subset of features when splitting a node, but also use a random threshold for each feature rather than searching for the best possible thresholds as traditional Decision Trees do. Since estimating best possible threshold for each feature at every node is also one of the most time-consuming tasks, this modification to the algorithm makes it more efficient. The algorithm that exploits this idea is called *Extremely Randomised Trees ensemble*, or *Extra-Trees*, and you can use it via [`sklearn's ExtraTreeClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.tree.ExtraTreeClassifier.html) implementation.

### Feature importance

Another useful property of the Random Forests classifier is that it can help you measure the relative importance of each feature by looking at how much the tree nodes that use this particular feature reduce impurity of the nodes on average (i.e., across all trees in the forest). The results are scaled so that the sum of all feature importances is equal to $1$.

Because features used in the *iris* and *digits* datasets are highly interpretable, let's estimate feature importances on these two datasets:

```python
from sklearn.datasets import load_iris
iris = load_iris()
rnd_clf = RandomForestClassifier(n_estimators=500, n_jobs=-1, random_state=42)
rnd_clf.fit(iris["data"], iris["target"])
for name, score in zip(iris["feature_names"], rnd_clf.feature_importances_):
    print(name, score)
```

    sepal length (cm) 0.11249225099876375
    sepal width (cm) 0.02311928828251033
    petal length (cm) 0.4410304643639577
    petal width (cm) 0.4233579963547682

```python
rnd_clf.feature_importances_
```

    array([0.11249225, 0.02311929, 0.44103046, 0.423358  ])

**Question**: What do these results suggest? Are they similar to your observations from the previous experiments on this dataset?

**Ans**: These results suggest that petal length and width are more important to distinguishing between different types of irises than sepal proportions. This is backed up by results from the second practical which showed that petal length and width are easily linearly separable but versicolor and virginica sepal dimensions are pretty mixed-together.

*Digits* dataset provides further interpretability of the results if we plot them. What does the figure below suggest?

```python
from sklearn import datasets
digits = datasets.load_digits()

rnd_clf = RandomForestClassifier(random_state=42)
rnd_clf.fit(digits["data"], digits["target"])
```

    RandomForestClassifier(random_state=42)

```python
def plot_digit(data):
    image = data.reshape(8, 8)
    plt.imshow(image, cmap = matplotlib.cm.hot,
               interpolation="nearest")
    plt.axis("off")

plot_digit(rnd_clf.feature_importances_)

cbar = plt.colorbar(ticks=[rnd_clf.feature_importances_.min(), rnd_clf.feature_importances_.max()])
cbar.ax.set_yticklabels(['Not important', 'Very important'])

plt.show()
```

## AdaBoost

*Boosting*, or *hypothesis boosting* is an ensemble method that can combine several weak learners into a strong learner. The general idea is to train predictors sequentially in such a way that each next predictor tries to correct its predecessor. The most popular among boosting methods are *AdaBoost* (which stands for *Adaptive Boosting*) and *Gradient Boosting*. Let's apply *AdaBoost* first.

The idea behind *AdaBoost* is as follows: in order to correct the predecessor's mistakes, the algorithm assigns higher weights to the training instances that the predecessor underfitted. I.e., the next step pays more attention to such instances and the algorithm focuses more on the hard cases.

Here is the step-by-step strategy:
- start with the first classifier (i.e., this can be a Decision Trees classifier)
- train it and use it to make predictions on the training set
- increase the relative weight of misclassified training instances
- train a second classifier with these new updated weights and make new predictions
- update weights using the new predictions
- continue until stopping criteria are satisfied.

For instance, suppose each instance's original weight $w^{(i)}$ is set to $\frac{1}{m}$, where $m$ is the number of instances. The first classifier is applied, and its error rate $r_1$ is computed on the training set using the equation below:

\begin{equation}
r_j = \frac{\sum_{\hat{y}^{(i)}_j \neq y^{(i)}} w^{(i)}}{\sum_{i=1}^m w^{(i)}}
\end{equation}

where $\hat{y}^{(i)}_j$ is the prediction of the $j$-th classifier on the $i$-th instance. The predictor's weight $\alpha_j$ is then estimated using:

\begin{equation}
\alpha_j = \eta log \frac{1-r_j}{r_j}
\end{equation}

where $\eta$ is the learning rate, a hyperparameter that defaults to $1$. The more accurate the predictor is, the higher its weight will be; if a predictor is guessing randomly, its weight will be close to $0$; and if it performs worse than random guessing it will get a high negative weight.

Next, the weights are updated and the misclassified instances are boosted as follows:

\begin{equation}
  w^{(i)}=\begin{cases}
    w^{(i)}, & \text{if $\hat y_j^{(i)} = y_j^{(i)}$}\\
    w^{(i)} exp(\alpha_j), & \text{if $\hat y_j^{(i)} \neq y_j^{(i)}$}
  \end{cases}
\end{equation}

Then all the instances weights are normalised (i.e., divided by $\sum_{i=1}^m w^{(i)}$). The new predictor is trained on the updated training instances, applied to the training set, its weight is computed, weights are updated again, and so on. The algorithm stops when either the predefined number of predictors is reached or a perfect predictor is found.

(This process should remind you about other sequential learning techniques we've discussed, e.g. Gradient Descent. The difference is, instead of tweaking a single predictor's parameters to minimise a cost function, AdaBoost adds predictors to the ensemble gradually making the prediction better.)

**Question**: We said earlier than one of the advantages of bagging and pasting strategies is that predictors can all be trained in parallel. Is the same applicable to AdaBoost?

**Ans**: No, the predictors need to be trained sequentially.

At prediction time, AdaBoost computes the predictions of all the predictors and weighs them according to $\alpha_j$. The predicted class is then the one that receives the majority of weighted votes, i.e.:

\begin{equation}
 \hat y(x) = argmax_{k} \sum_{j=1; \hat y_j(x)=k} ^{N} \alpha_j
\end{equation}

with $N$ being the number of predictors.

`sklearn's` implementation of AdaBoost uses *Stagewise Additive Modeling using a Multiclass Exponential loss function* (or *SAMME* for short) multiclass version of the algorithm, which in a binary case simply defaults to AdaBoost. In addition, if the base estimators can estimate class probabilities (have `predict_proba()` method), `sklearn's` algorithm relies on class probabilities rather than predictions, which generally performs better. This version is called *SAMME.R*, where *R* stands for "Real".

Let's train [`sklearn's AdaBoostClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html), for example using $200$ *Decision Stumps* (trees composed of a single decision node and two leaf nodes, `max_depth=1`):

```python
from sklearn.ensemble import AdaBoostClassifier

ada_clf = AdaBoostClassifier(
    DecisionTreeClassifier(max_depth=1), n_estimators=200,
    algorithm="SAMME.R", learning_rate=0.5, random_state=42)
ada_clf.fit(X_train, y_train)
```

    AdaBoostClassifier(base_estimator=DecisionTreeClassifier(max_depth=1),
                       learning_rate=0.5, n_estimators=200, random_state=42)

And let's plot the decision boundary of the algorithm:

```python
plot_decision_boundary(ada_clf, X, y)
```

To get a bit more insight into how the decision boundaries change from one step to another and with a different learning rate, let's plot decision boundaries for $5$ consecutive predictors, in this case using `SVC` as the estimator. Note how the first classifiers usually get many instances wrong, while the following predictors are gradually getting better. The plot on the right presents the very same $5$ consecutive classifiers, but assigns half the learning rate (i.e., the misclassified instances weights are only boosted half as much at every iteration):

```python
m = len(X_train)

plt.figure(figsize=(11, 4))
for subplot, learning_rate in ((121, 1), (122, 0.5)):
    sample_weights = np.ones(m)
    plt.subplot(subplot)
    for i in range(5):
        svm_clf = SVC(kernel="rbf", C=0.05, gamma="auto", random_state=42)
        svm_clf.fit(X_train, y_train, sample_weight=sample_weights)
        y_pred = svm_clf.predict(X_train)
        sample_weights[y_pred != y_train] *= (1 + learning_rate)
        plot_decision_boundary(svm_clf, X, y, alpha=0.2)
        plt.title("learning_rate = {}".format(learning_rate), fontsize=16)
    if subplot == 121:
        plt.text(1.70,  -0.90, "1", fontsize=14)
        plt.text(-0.40, -0.35, "2", fontsize=14)
        plt.text(-0.55, -0.05, "3", fontsize=14)
        plt.text(-0.70,  0.20, "4", fontsize=14)
        plt.text(-0.85,  0.45, "5", fontsize=14)

plt.show()
```

In addition, here is the full list of parameters and attributes of the algorithm:

```python
list(m for m in dir(ada_clf) if not m.startswith("_") and m.endswith("_"))
```

    ['base_estimator_',
     'classes_',
     'estimator_errors_',
     'estimator_weights_',
     'estimators_',
     'feature_importances_',
     'n_classes_',
     'n_features_in_']

E.g., you can check classification errors for each estimator in the boosted ensemble as follows:

```python
ada_clf.estimator_errors_
```

    array([0.18133333, 0.27512937, 0.28334096, 0.35488317, 0.3171361 ,
           0.38272512, 0.35630662, 0.33148186, 0.40484099, 0.29959284,
           0.38482533, 0.37793452, 0.4396849 , 0.41956551, 0.41977426,
           0.42276892, 0.44540522, 0.4592429 , 0.43517238, 0.4795022 ,
           0.46538072, 0.48000343, 0.43464541, 0.44653579, 0.47534577,
           0.48382716, 0.45804523, 0.46674707, 0.4374261 , 0.48598125,
           0.48769271, 0.4722106 , 0.46346519, 0.46165373, 0.44202223,
           0.44509139, 0.47099366, 0.45066309, 0.4881181 , 0.47269775,
           0.47884803, 0.46937918, 0.4663008 , 0.44912872, 0.45639441,
           0.46252216, 0.48048523, 0.49000957, 0.4780251 , 0.45748808,
           0.47390931, 0.4836773 , 0.45432253, 0.4816146 , 0.49129336,
           0.48627751, 0.48533895, 0.48977843, 0.4781294 , 0.45990675,
           0.45628973, 0.45333756, 0.48145777, 0.47828933, 0.46468214,
           0.46348201, 0.47643265, 0.471952  , 0.45673327, 0.47977324,
           0.46875606, 0.46464402, 0.45684007, 0.46146966, 0.46037637,
           0.47746753, 0.47412264, 0.47699586, 0.46442197, 0.47295429,
           0.46091199, 0.4563625 , 0.46459194, 0.47996521, 0.47721258,
           0.46659331, 0.46327864, 0.46068569, 0.47494239, 0.48438408,
           0.49138786, 0.47519606, 0.46263183, 0.48221231, 0.47845427,
           0.46707103, 0.46752551, 0.46045753, 0.46761742, 0.46499847,
           0.46395447, 0.49433102, 0.48645588, 0.48202713, 0.47469894,
           0.4653022 , 0.4673878 , 0.48326479, 0.49121441, 0.47963298,
           0.48471417, 0.46201554, 0.49488345, 0.48830469, 0.48244527,
           0.46889398, 0.47253576, 0.461846  , 0.46893359, 0.4663451 ,
           0.46250006, 0.46156224, 0.46978699, 0.47973597, 0.48335403,
           0.478635  , 0.47867593, 0.46810413, 0.46571099, 0.46988454,
           0.47102448, 0.48051697, 0.47955647, 0.48117019, 0.48996885,
           0.48676386, 0.46644644, 0.45875648, 0.49225164, 0.48657145,
           0.46831141, 0.46670826, 0.4642594 , 0.46138968, 0.46510823,
           0.46590595, 0.49260031, 0.48709267, 0.48131461, 0.47309791,
           0.47602392, 0.47498064, 0.47189274, 0.47859843, 0.48096745,
           0.48004431, 0.47278772, 0.46457035, 0.4791516 , 0.47009956,
           0.46682489, 0.46515405, 0.46253448, 0.46803148, 0.46881194,
           0.4695307 , 0.46808798, 0.48691669, 0.48859091, 0.4923833 ,
           0.4834733 , 0.47618303, 0.47679647, 0.48026735, 0.47879651,
           0.47578553, 0.47437284, 0.47017986, 0.4711307 , 0.46857867,
           0.46859443, 0.47158646, 0.47157599, 0.48294599, 0.48442876,
           0.47297868, 0.46764791, 0.4668408 , 0.46366048, 0.4665483 ,
           0.46686315, 0.46672805, 0.49566253, 0.49250423, 0.48836732,
           0.49223529, 0.49201652, 0.4806373 , 0.48783444, 0.48308099])

## Gradient Boosting

Gradient Boosting is another popular boosting algorithm. It also works sequentially adding new predictors to the ensemble, each one correcting the errors from its predecessor. Unlike AdaBoost, it's trying to fit each new predictor to the *residual errors* made by the previous predictor.

Let's this time apply the boosting algorithm to a regression task. First, generate a synthetic noisy quadratic training set:

```python
np.random.seed(42)
X = np.random.rand(100, 1) - 0.5
y = 3*X[:, 0]**2 + 0.05 * np.random.randn(100)
```

Next, let's fit a single [`DecisionTreeRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html):

```python
from sklearn.tree import DecisionTreeRegressor

tree_reg1 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg1.fit(X, y)
```

    DecisionTreeRegressor(max_depth=2, random_state=42)

Now train a second `DecisionTreeRegressor` on the residual errors made by the first predictor:

```python
y2 = y - tree_reg1.predict(X)
tree_reg2 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg2.fit(X, y2)
```

    DecisionTreeRegressor(max_depth=2, random_state=42)

Finally, let's train a third regressor on the residual errors made by the second predictor:

```python
y3 = y2 - tree_reg2.predict(X)
tree_reg3 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg3.fit(X, y3)
```

    DecisionTreeRegressor(max_depth=2, random_state=42)

That's it – you've just built an ensemble containing three trees. It can now be applied to new instances by making predictions based on **adding up the predictions** from all three trees:

```python
X_new = np.array([[0.8]])

y_pred = sum(tree.predict(X_new) for tree in (tree_reg1, tree_reg2, tree_reg3))

y_pred
```

    array([0.75026781])

Let's visualise the results by plotting the predictions from each tree on the left and the ensemble on the right. E.g., in the first row the ensemble contains only one tree, so its predictions are exactly the same as the first tree's predictions. In the second row, the second tree is trained on the residual errors from the first tree, and the ensemble's prediction is based on the sum of the predictions of the two trees. You can see how the predictions of the ensemble are gradually getting better:

```python
def plot_predictions(regressors, X, y, axes, label=None, style="r-", data_style="b.", data_label=None):
    x1 = np.linspace(axes[0], axes[1], 500)
    y_pred = sum(regressor.predict(x1.reshape(-1, 1)) for regressor in regressors)
    plt.plot(X[:, 0], y, data_style, label=data_label)
    plt.plot(x1, y_pred, style, linewidth=2, label=label)
    if label or data_label:
        plt.legend(loc="upper center", fontsize=16)
    plt.axis(axes)

plt.figure(figsize=(11,11))

plt.subplot(321)
plot_predictions([tree_reg1], X, y, axes=[-0.5, 0.5, -0.1, 0.8],
                 label="$h_1(x_1)$", style="g-", data_label="Training set")
plt.ylabel("$y$", fontsize=16, rotation=0)
plt.title("Residuals and tree predictions", fontsize=16)

plt.subplot(322)
plot_predictions([tree_reg1], X, y, axes=[-0.5, 0.5, -0.1, 0.8],
                 label="$h(x_1) = h_1(x_1)$", data_label="Training set")
plt.ylabel("$y$", fontsize=16, rotation=0)
plt.title("Ensemble predictions", fontsize=16)

plt.subplot(323)
plot_predictions([tree_reg2], X, y2, axes=[-0.5, 0.5, -0.5, 0.5],
                 label="$h_2(x_1)$", style="g-", data_style="k+", data_label="Residuals")
plt.ylabel("$y - h_1(x_1)$", fontsize=16)

plt.subplot(324)
plot_predictions([tree_reg1, tree_reg2], X, y, axes=[-0.5, 0.5, -0.1, 0.8],
                 label="$h(x_1) = h_1(x_1) + h_2(x_1)$")
plt.ylabel("$y$", fontsize=16, rotation=0)

plt.subplot(325)
plot_predictions([tree_reg3], X, y3, axes=[-0.5, 0.5, -0.5, 0.5],
                 label="$h_3(x_1)$", style="g-", data_style="k+")
plt.ylabel("$y - h_1(x_1) - h_2(x_1)$", fontsize=16)
plt.xlabel("$x_1$", fontsize=16)

plt.subplot(326)
plot_predictions([tree_reg1, tree_reg2, tree_reg3], X, y, axes=[-0.5, 0.5, -0.1, 0.8],
                 label="$h(x_1) = h_1(x_1) + h_2(x_1) + h_3(x_1)$")
plt.xlabel("$x_1$", fontsize=16)
plt.ylabel("$y$", fontsize=16, rotation=0)

plt.show()
```

A simpler way to train ensembles of *Gradient Boosted Regression Trees* is to use [`sklearn's GradientBoostingRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html) implementation. In addition to the typical hyperparameters that control the growth of the trees, it also includes hyperparameters that control the ensemble training, e.g. `n_estimators`:

```python
from sklearn.ensemble import GradientBoostingRegressor

gbrt = GradientBoostingRegressor(max_depth=2, n_estimators=3, learning_rate=1.0, random_state=42)
gbrt.fit(X, y)
```

    GradientBoostingRegressor(learning_rate=1.0, max_depth=2, n_estimators=3,
                              random_state=42)

**Question**: Another hyperparameter is `learning_rate`. What is it responsible for and what effect on the resulting ensemble does it have?

**Ans**: *learning rate shrinks the contribution of each tree by learning_rate. There is a trade-off between learning_rate and n_estimators.*

Feel free to check `sklearn's` documentation, as well as look into the comparison of two ensembles: the one with a lower number of estimators and a higher learning rate, and a 'slow learner' with a lower learning rate but a higher number of estimators:

```python
gbrt_slow = GradientBoostingRegressor(max_depth=2, n_estimators=200, learning_rate=0.1, random_state=42)
gbrt_slow.fit(X, y)
```

    GradientBoostingRegressor(max_depth=2, n_estimators=200, random_state=42)

```python
plt.figure(figsize=(11,4))

plt.subplot(121)
plot_predictions([gbrt], X, y, axes=[-0.5, 0.5, -0.1, 0.8], label="Ensemble predictions")
plt.title("learning_rate={}, n_estimators={}".format(gbrt.learning_rate, gbrt.n_estimators), fontsize=14)

plt.subplot(122)
plot_predictions([gbrt_slow], X, y, axes=[-0.5, 0.5, -0.1, 0.8])
plt.title("learning_rate={}, n_estimators={}".format(gbrt_slow.learning_rate, gbrt_slow.n_estimators), fontsize=14)

plt.show()
```

**Question**: What generalisation behaviour can you expect from each of these ensembles?

**Ans**: since the slow learner is more wriggly, I would bet that the higher learning rate estimator would fare better. 200 estimators is enough to fit to nearly all points exactly. It heavily depends on the kind of data we're working with though, i.e. the intrinsic properties of the data

### Gradient Boosting with Early Stopping

The examples above demonstrate ensembles with a different number of estimators. Which one is better? How can you quantitatively estimate the optimal number of predictors?

One technique that you can use is called *early stopping* – it allows your algorithm to stop as soon as the validation error reaches a minimum. An easy way to implement this with `sklearn` is to use `staged_predict()` method as the code below demonstrates:

```python
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

X_train, X_val, y_train, y_val = train_test_split(X, y, random_state=100)

gbrt = GradientBoostingRegressor(max_depth=2, n_estimators=120, random_state=42)
gbrt.fit(X_train, y_train)

errors = [mean_squared_error(y_val, y_pred)
          for y_pred in gbrt.staged_predict(X_val)]
bst_n_estimators = np.argmin(errors)

gbrt_best = GradientBoostingRegressor(max_depth=2,n_estimators=bst_n_estimators, random_state=42)
gbrt_best.fit(X_train, y_train)
```

    GradientBoostingRegressor(max_depth=2, n_estimators=68, random_state=42)

Let's visualise the validaton errors and the optimal number of estimators:

```python
min_error = np.min(errors)

plt.figure(figsize=(11, 4))

plt.subplot(121)
plt.plot(errors, "b.-")
plt.plot([bst_n_estimators, bst_n_estimators], [0, min_error], "k--")
plt.plot([0, 120], [min_error, min_error], "k--")
plt.plot(bst_n_estimators, min_error, "ko")
plt.text(bst_n_estimators, min_error*1.2, "Minimum", ha="center", fontsize=14)
plt.axis([0, 120, 0, 0.01])
plt.xlabel("Number of trees")
plt.title("Validation error", fontsize=14)

plt.subplot(122)
plot_predictions([gbrt_best], X, y, axes=[-0.5, 0.5, -0.1, 0.8])
plt.title("Best model (%d trees)" % bst_n_estimators, fontsize=14)

plt.show()
```

Finally, an alternative to training a large number of estimators and then looking back in order to find the optimal number of those is to allow the algorithm to stop training when the validation error does not improve over a number of consecutive iterations (e.g., $5$ in the code below). For that to work, set the `warm_start` to `True`.

`warm_start`: _When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just erase the previous solution._

```python
gbrt = GradientBoostingRegressor(max_depth=2, warm_start=True, random_state=42)

min_val_error = float("inf")
error_going_up = 0
for n_estimators in range(1, 120):
    gbrt.n_estimators = n_estimators
    gbrt.fit(X_train, y_train)
    y_pred = gbrt.predict(X_val)
    val_error = mean_squared_error(y_val, y_pred)
    if val_error < min_val_error:
        min_val_error = val_error
        error_going_up = 0
    else:
        error_going_up += 1
        if error_going_up == 5:
            break  # early stopping

print(gbrt.n_estimators)
```

    74

# Your tasks

1. Run the code in the notebook. During the practical session, be prepared to discuss the methods and answer the questions from this notebook:

    * When applied to a new, test instance bagging classifier aggregates the predictions of all predictors and estimates the statistical mode. Is this similar to the hard or the soft voting strategy?
        * **Ans**: This is similar to the _hard_ voting strategy, as it just picks the most frequest predicition.
    * [`BaggingClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingClassifier.html) automatically performs soft voting when the base classifier can estimate class probabilities (i.e., has a `predict_proba()` method). Which strategy – hard or soft voting – does `BaggingClassifier` follow with the `DecisionTreeClassifier` as the base classifier?
        * **Ans**: The `DecisionTreeClassifier` has a `predict_proba()` with default settings. Hence, the `BaggingClassifier` will use soft voting.
    * What do the feature importance weights obtained on the *iris* dataset suggest? Are they similar to your observations from the previous experiments on this dataset?
        * **Ans**: These results suggest that petal length and width are more important to distinguishing between different types of irises than sepal proportions. This is backed up by results from the second practical which showed that petal length and width are easily linearly separable but versicolor and virginica sepal dimensions are pretty mixed-together.
    * We said earlier than one of the advantages of bagging and pasting strategies is that predictors can all be trained in parallel. Is the same applicable to AdaBoost?
        * **Ans**: No, the predictors need to be trained sequentially.
    * One of the hyperparameters of the boosting algorithms is learning_rate. What is it responsible for and what effect on the resulting ensemble does it have?
        * **Ans**: *learning rate shrinks the contribution of each tree by learning_rate. There is a trade-off between learning_rate and n_estimators.*
    * What generalisation behaviour can you expect from each of the tree regressor ensembles (using different learning rates and different number of estimators – see above)?
        * **Ans**: since the slow learner is more wriggly, I would bet that the higher learning rate estimator would fare better. 200 estimators is enough to fit to nearly all points exactly. It heavily depends on the kind of data we're working with though, i.e. the intrinsic properties of the data

***

2. Apply ensemble techniques of your choice to one of the datasets you've worked on during the previous practicals and report your findings.

### Digits - Ensemble Techniques

```python
# Common imports
import numpy as np
import os
import scipy, math
import scipy.special
import pandas as pd

from sklearn.model_selection import train_test_split
from sklearn.model_selection import StratifiedShuffleSplit
from sklearn.ensemble import VotingClassifier
from sklearn.metrics import accuracy_score
from sklearn.metrics import mean_squared_error

from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
from sklearn.tree import ExtraTreeClassifier

from sklearn.ensemble import BaggingClassifier
from sklearn.ensemble import AdaBoostClassifier
from sklearn.ensemble import GradientBoostingRegressor

# to make this notebook's output stable across runs
np.random.seed(42)

# To plot pretty figures
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
plt.rcParams['axes.labelsize'] = 14
plt.rcParams['xtick.labelsize'] = 12
plt.rcParams['ytick.labelsize'] = 12
```

```python
from sklearn import datasets
digits = datasets.load_digits()
X, y = digits["data"], digits["target"]
```

```python
split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
split.get_n_splits(X, y)
print(split)

for train_index, test_index in split.split(X, y):
    print("TRAIN:", len(train_index), "TEST:", len(test_index))
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]

print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)
```

    StratifiedShuffleSplit(n_splits=1, random_state=42, test_size=0.2,
                train_size=None)
    TRAIN: 1437 TEST: 360
    (1437, 64) (1437,) (360, 64) (360,)

```python
def digit_importance(model):
    def plot_digit(data):
        image = data.reshape(8, 8)
        plt.imshow(image, cmap = matplotlib.cm.hot,
                   interpolation="nearest")
        plt.axis("off")

    try:
        feat_imp = model.feature_importances_
    except:
        feat_imp = np.mean([
            tree.feature_importances_ for tree in model.estimators_
        ], axis=0)
    plot_digit(feat_imp)

    cbar = plt.colorbar(ticks=[feat_imp.min(), feat_imp.max()])
    cbar.ax.set_yticklabels(['Not important', 'Very important'])

    plt.show()
```

### Evaluation

#### Voting Classifiers

```python
log_clf = LogisticRegression(max_iter=10000, random_state=42)
rnd_clf = RandomForestClassifier(random_state=42)
svm_clf = SVC(random_state=42)

voting_clf = VotingClassifier(
    estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
    voting='hard')
voting_clf.fit(X_train, y_train)

print("== Hard Voting ==")
for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
```

    == Hard Voting ==
    LogisticRegression 0.9611111111111111
    RandomForestClassifier 0.9611111111111111
    SVC 0.9916666666666667
    VotingClassifier 0.9861111111111112

```python
log_clf = LogisticRegression(max_iter=10000, random_state=42)
rnd_clf = RandomForestClassifier(random_state=42)
svm_clf = SVC(probability=True, random_state=42)

voting_clf = VotingClassifier(
    estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
    voting='soft')
voting_clf.fit(X_train, y_train)

print("== Soft Voting ==")
for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
```

    == Soft Voting ==
    LogisticRegression 0.9611111111111111
    RandomForestClassifier 0.9611111111111111
    SVC 0.9916666666666667
    VotingClassifier 0.9833333333333333

#### Bagging Ensembles

```python
bag_clf = BaggingClassifier(
    DecisionTreeClassifier(random_state=42), n_estimators=500,
    max_samples=100, bootstrap=True, n_jobs=-1, oob_score=True, random_state=42)

bag_clf.fit(X_train, y_train)
y_pred = bag_clf.predict(X_test)
print(f"500 DecisionTrees Bagging out-of-bag eval: {bag_clf.oob_score_}")
print(f"500 DecisionTrees Bagging eval on test set: {accuracy_score(y_test, y_pred)}")

digit_importance(bag_clf)
```

```python
bag_clf = BaggingClassifier(
    ExtraTreeClassifier(random_state=42), n_estimators=200,
    bootstrap=True, n_jobs=-1, oob_score=True, random_state=42)

bag_clf.fit(X_train, y_train)
y_pred = bag_clf.predict(X_test)
print(f"200 ExtraTrees Bagging out-of-bag eval: {bag_clf.oob_score_}")
print(f"200 ExtraTrees Bagging eval on test set: {accuracy_score(y_test, y_pred)}")

digit_importance(bag_clf)
```

```python
bag_clf = BaggingClassifier(
    DecisionTreeClassifier(random_state=42), n_estimators=500,
    max_samples=100, bootstrap=False, n_jobs=-1, random_state=42)

bag_clf.fit(X_train, y_train)
y_pred = bag_clf.predict(X_test)
print(f"500 DecisionTrees Pasting eval on test set: {accuracy_score(y_test, y_pred)}")

digit_importance(bag_clf)
```

```python
from sklearn.ensemble import RandomForestClassifier

rnd_clf = RandomForestClassifier(n_estimators=500, max_leaf_nodes=16,
                                 n_jobs=-1, random_state=42)
rnd_clf.fit(X_train, y_train)
y_pred = rnd_clf.predict(X_test)
print(f"RandomForest (n_est = 500) eval on test set: {accuracy_score(y_test, y_pred)}")

digit_importance(rnd_clf)
```

#### AdaBoost

```python
ada_clf = AdaBoostClassifier(
    DecisionTreeClassifier(max_depth=1), n_estimators=200,
    algorithm="SAMME.R", learning_rate=0.5, random_state=42)
ada_clf.fit(X_train, y_train)

y_pred = ada_clf.predict(X_test)
print(f"AdaBoostClassifier (200 decision stumps; class prob) eval on test set: {accuracy_score(y_test, y_pred)}")

digit_importance(ada_clf)
```

```python
ada_clf = AdaBoostClassifier(
    DecisionTreeClassifier(max_depth=1), n_estimators=200,
    algorithm="SAMME", learning_rate=0.5, random_state=42)
ada_clf.fit(X_train, y_train)

y_pred = ada_clf.predict(X_test)
print(f"AdaBoostClassifier (200 decision stumps; predictions) eval on test set: {accuracy_score(y_test, y_pred)}")

digit_importance(ada_clf)
```

#### Gradient Boosting

```python
gbrt = GradientBoostingRegressor(max_depth=16, warm_start=True, n_estimators=50, learning_rate=1.0, random_state=42)

min_val_error = float("inf")
error_going_up = 0
for n_estimators in range(1, 120):
    gbrt.n_estimators = n_estimators
    gbrt.fit(X_train, y_train)
    y_pred = gbrt.predict(X_test)
    val_error = mean_squared_error(y_test, y_pred)
    if val_error < min_val_error:
        min_val_error = val_error
        error_going_up = 0
    else:
        error_going_up += 1
        if error_going_up == 5:
            break  # early stopping

y_pred = np.round(gbrt.predict(X_test), 0)
print(f"Gradient Boosting (max_d=16, n_est={gbrt.n_estimators}, lr=1.0) eval on test set: {accuracy_score(y_test, y_pred)}")
```

    Gradient Boosting (max_d=16, n_est=7, lr=1.0) eval on test set: 0.7861111111111111

```python
gbrt = GradientBoostingRegressor(max_depth=8, warm_start=True, n_estimators=100, learning_rate=0.1, random_state=42)

min_val_error = float("inf")
error_going_up = 0
for n_estimators in range(1, 120):
    gbrt.n_estimators = n_estimators
    gbrt.fit(X_train, y_train)
    y_pred = gbrt.predict(X_test)
    val_error = mean_squared_error(y_test, y_pred)
    if val_error < min_val_error:
        min_val_error = val_error
        error_going_up = 0
    else:
        error_going_up += 1
        if error_going_up == 5:
            break  # early stopping

y_pred = np.round(gbrt.predict(X_test), 0)
print(f"Gradient Boosting (max_d=8, n_est={gbrt.n_estimators}, lr=0.1) eval on test set: {accuracy_score(y_test, y_pred)}")
```

    Gradient Boosting (max_d=8, n_est=119, lr=0.1) eval on test set: 0.6611111111111111

#### Conclusions
SVC perfoms best on the digits dataset with an accuracy of `0.992`.

Using a bagging classifier with bagging (ie bootstrap) with 200 `ExtraTreeClassifier` performs respectibly with an accuracy of `0.981`.

***

3. **Optional**: If you want more practice with these techniques, try implementing a [*stacking algorithm*](https://en.wikipedia.org/wiki/Ensemble_learning#Stacking). There is no available implementation of this approach in `sklearn` so it needs to be implemented from scratch. The idea is as follows:
    * Split a dataset of your choice into three subsets – training, validation and test.
    * Train a number of predictors on the training data and apply them to the validation data.
    * Treat the predictions of those predictors on the validation data to generate new training set: you can use the predictions of the estimators as new features (i.e., each training instance will have as many features as the number of predictors you originally used), and the validation data targets as the new training instances targets.
    * Now train a *blender* – a classifier of your choice – on the new training data created this way. Together with the original classifiers, the blender forms a stacking ensemble.
    * Finally, apply your original classifiers to the test set, feed their predictions to the blender, and use blender's output as the prediction on the test set.

```python
from sklearn import datasets
digits = datasets.load_digits()
```

```python
def stratified_test_val_train_split(X, y, seed=None):
    split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=seed)

    for train_index, test_index in split.split(X, y):
        print("TEST:", len(test_index))
        X_rem, X_test = X[train_index], X[test_index]
        y_rem, y_test = y[train_index], y[test_index]

    split = StratifiedShuffleSplit(n_splits=1, test_size=0.25, random_state=seed)
    for train_index, test_index in split.split(X_rem, y_rem):
        print("VAL:", len(test_index))
        print("TRAIN:", len(train_index))
        X_train, X_val = X[train_index], X[test_index]
        y_train, y_val = y[train_index], y[test_index]

    return X_train, y_train, X_val, y_val, X_test, y_test

X_train, y_train, X_val, y_val, X_test, y_test = stratified_test_val_train_split(digits["data"], digits["target"], 42)
```

    TEST: 360
    VAL: 360
    TRAIN: 1077

```python
from sklearn.naive_bayes import GaussianNB, MultinomialNB
from sklearn.linear_model import SGDClassifier, LogisticRegression
from sklearn.svm import SVC
from sklearn.multiclass import OneVsOneClassifier

def get_many_predictors(X, y, seed=None):
    np.random.seed(seed)
    ovo_log =  OneVsOneClassifier(LogisticRegression(max_iter=10000, random_state=seed))
    ovo_rnd =  OneVsOneClassifier(RandomForestClassifier(random_state=seed))
    ovo_svm =  OneVsOneClassifier(SVC(random_state=seed))
    ovo_sgd =  OneVsOneClassifier(SGDClassifier(max_iter=10000, random_state=seed, loss="perceptron",
                            eta0=1, learning_rate="constant", penalty=None))
    ovo_mnb = OneVsOneClassifier(MultinomialNB())
    ovo_gnb = OneVsOneClassifier(GaussianNB())

    predictors = [ovo_log, ovo_rnd, ovo_svm, ovo_sgd, ovo_mnb, ovo_gnb]
    for i, clf in enumerate(predictors):
        clf.fit(X, y)

    return predictors

def get_blender_set(X, predictors):
    train = np.empty((len(X), len(predictors)))
    for i, clf in enumerate(predictors):
        train[:,i] = clf.predict(X)
    return train

def get_blender_clf(X, y, predictors, seed=None):
    train = get_blender_set(X, predictors)

    blender = OneVsOneClassifier(SVC(random_state=seed))
    blender.fit(train, y)

    return blender

predictors = get_many_predictors(X_train, y_train, 42)
blender = get_blender_clf(X_val, y_val, predictors, 42)
X_blender_test = get_blender_set(X_test, predictors)
y_pred = blender.predict(X_blender_test)
print(f"Blender eval on test set: {accuracy_score(y_test, y_pred)}")
```

    Blender eval on test set: 0.9555555555555556

```python

```


```julia
using DataFrames, CSV
using JuMP, Gurobi
using LinearAlgebra, Random, Printf
using Plots

const GRB_ENV = Gurobi.Env()
```

    Academic license - for non-commercial use only - expires 2021-07-08

    Gurobi.Env(Ptr{Nothing} @0x00007f867c9e1400, false, 0)

```julia
Pb1_arcs.csv: A 90×5 matrix reporting, for each arc (column 1), the starting and ending
nodes (columns 2 and 3), the capacity (column 4) and the construction cost (column 5).
– Pb2_transport.csv: A 90×3 matrix reporting the unit transportation cost on each arc (rows)
for each customer (columns).
– Pb2_demand.csv: A 3×2 matrix reporting the demand from each customer (rows) in each
scenario (columns).
– Pb2_customer_OD.csv: A 3×2 matrix reporting, for each customer (rows), the origin (column
1) and the destination (column 2).
```

```julia
arcs = CSV.read("Pb1_arcs.csv", DataFrame)
transport = CSV.read("Pb1_transport.csv", DataFrame)
demand = CSV.read("Pb1_demand.csv", DataFrame)
customers = CSV.read("Pb1_customer_OD.csv", DataFrame)
arcs_ = convert.(Int64,arcs[!,2:3])
```

	x2	x3

	Int64	Int64

90 rows × 2 columns

1	2	1

2	3	1

3	4	1

4	5	1

5	6	1

6	7	1

7	8	1

8	9	1

9	10	1

10	1	2

11	3	2

12	4	2

13	5	2

14	6	2

15	7	2

16	8	2

17	9	2

18	10	2

19	1	3

20	2	3

21	4	3

22	5	3

23	6	3

24	7	3

25	8	3

26	9	3

27	10	3

28	1	4

29	2	4

30	3	4

⋮	⋮	⋮

```julia
customers
```

	x1	x2

	Int64	Int64

3 rows × 2 columns

1	1	9

2	4	2

3	8	2

## 1. Problem setup

We want to solve the following stochastic facility location problem:

$$\begin{align}
\min\quad & \sum_{i=1}^Nc_ix_i+\sum_{s=1}^Sp_s\left(\sum_{i=1}^N\sum_{j=1}^Mt_{ij}y_{ij}^s + \sum_{j=1}^Mq_jz_{j}^s\right)\\
\text{s.t.}\quad & \sum_{i=1}^Ny_{ij}^s+z_j^s\ge d_j^s & \forall j\in [M], s\in [S]\\
& \sum_{j=1}^My_{ij}^s \le C_ix_i&\forall i\in[N], s\in[S]\\
& \mathbf{y,z}\ge 0, \mathbf{x}\in\{0,1\}^N
\end{align}$$

Dual subproblem $s$:
$$\begin{align}
\max\quad &  \sum_{j=1}^M \mu_j d_j^s - \sum_{i=1}^N\lambda_iC_ix_i\\
\text{s.t.}\quad & \mu_j -\lambda_i \le t_{ij} & \forall i\in[N], j\in[M]\\
& \mu_j \le q_j &\forall j\in [M]\\
& \mathbf{\mu, \lambda}\ge 0
\end{align}$$

- If the dual subproblem is unbounded, we obtain an extreme ray $(\mu^*, \lambda^*)$ and add a feasibility cut:
$$\sum_{j=1}^M\mu^*_j d_j^s - \sum_{i=1}^N \lambda^*_i C_i x_i \le 0$$

- If the dual subproblem solves to optimality, we obtain an extreme point $(\mu^*, \lambda^*)$ and add an optimality cut:
$$θ_s \ge \sum_{j=1}^M\mu^*_j d_j^s - \sum_{i=1}^N\lambda^*_i C_i x_i$$

\begin{equation}
    \label{eq:subproblem-stochastic-dual}
    \begin{array}{ll}
        \max & \sum_{k \in \mathcal{K}} d_k^s ({\pi_{O_k, k}^s - \pi_{D_k,k}^s}) + \sum_{(i, j) \in \mathcal{A}} u_{i j} x_{i j} \mu_{ i j}^s \\
        \text { s.t. } & \pi_{i, k}^s - \pi_{j, k}^s + \mu_{i j}^s \leq f_{i j k}, \forall (i, j) \in \mathcal{A}, k \in \mathcal{K} \\
        & \pi_{i k}^s \in \mathbb{R}, \forall i \in \mathcal{N}, k \in \mathcal{K} \\
        & \mu_{i j}^s \leq 0,  \forall(i, j) \in \mathcal{A} \\
    \end{array}
\end{equation}

```julia
TIME_LIMIT = 90;
OPTIMALITY_GAP = 0.001;

"Solve problem using multi-cut Benders decomposition"
function solve_benders_multi(verbose::Bool=true)
    # define main problem
    MP = Model(() -> Gurobi.Optimizer(GRB_ENV));
    set_optimizer_attributes(MP, "TimeLimit" => 60, "MIPGap" => 1e-4, "OutputFlag" => 0, "DualReductions" => 0)
    A = size(arcs, 1); K = size(customers, 1); S = size(demand, 2)
    @variable(MP, x[1:A], Bin)
    @variable(MP, θ[1:S] >= 0)
    @objective(MP, Min, sum(arcs[i,5] * x[i] for i=1:A) + sum(1/S * θ[s] for s in 1:S))
#10
    lower_bound_all = []; upper_bound_all = []
    MP_time = []; SP_max_time = []; SP_time = []
    num_opt = 0; num_feas = 0
    while true
        # solve master problem
        push!(MP_time, @elapsed optimize!(MP))
        lower_bound_new = objective_value(MP)
        push!(lower_bound_all, lower_bound_new)
        x_MP = value.(MP[:x])
        # solve S subproblems
        obj_SP = zeros(S) #20
        SP_time_all = zeros(S)
        for s = 1:S

            SP_dual = Model(JuMP.optimizer_with_attributes(() -> Gurobi.Optimizer(GRB_ENV),
                "OutputFlag" => 0, "DualReductions" => 0, "TimeLimit" => 60))

            @variable(SP_dual, λ[1:10,1:K]);
            @variable(SP_dual, μ[1:A] <= 0);

            @objective(SP_dual, Max,
                       sum((λ[customers[k,1],k]-λ[customers[k,2],k]) * demand[k,s] for k in 1:K) +
                       sum(μ[i] * arcs[i,4] * x_MP[i] for i in 1:A))

            @constraint(SP_dual, [i in 1:A, k in 1:K],
                λ[arcs_[i,1],k] - λ[arcs_[i,2],k] + μ[i] <= transport[i,k])

            SP_time_all[s] = @elapsed optimize!(SP_dual)
            #obj_SP_dual = objective_value(SP_dual)
            λ_val = value.(SP_dual[:λ])
            μ_val = value.(SP_dual[:μ])
            if termination_status(SP_dual) == MOI.DUAL_INFEASIBLE # feasibility cut
                @constraint(MP, sum((λ_val[customers[k,1],k]-λ_val[customers[k,2],k]) * demand[k,s] for k in 1:K) +
                       sum(μ_val[i] * arcs[i,4] * x[i] for i in 1:A) <= 0)
                obj_SP[s] = 3200
                num_feas += 1
            elseif termination_status(SP_dual) == MOI.OPTIMAL
                @constraint(MP, θ[s] >= sum((λ_val[customers[k,1],k]-λ_val[customers[k,2],k]) * demand[k,s] for k in 1:K) +
                       sum(μ_val[i] * arcs[i,4] * x[i] for i in 1:A))
                obj_SP[s] = objective_value(SP_dual)
                num_opt += 1
            end
        end
        push!(SP_max_time, maximum(SP_time_all))
        push!(SP_time, sum(SP_time_all))
        upper_bound_new = sum(arcs[i,5] * x_MP[i] for i=1:A) + sum(1/S * obj_SP[s] for s in 1:S)
        push!(upper_bound_all, upper_bound_new)
        verbose && @printf("UB: %.2f - LB: %.2f\n", upper_bound_all[end], lower_bound_all[end])
        if sum(MP_time) + sum(SP_time) >= TIME_LIMIT ||
            (upper_bound_new-lower_bound_new)/lower_bound_new < OPTIMALITY_GAP
            break
        end
    end
    return upper_bound_all, lower_bound_all, MP_time, SP_time, SP_max_time, num_feas, num_opt
end
```

    solve_benders_multi

```julia
upper_bound_all, lower_bound_all, MP_time, SP_time, SP_max_time, num_feas, num_opt = solve_benders_multi()
```

    UB: 3200.00 - LB: 0.00
    UB: 3210.05 - LB: 10.05
    UB: 3210.55 - LB: 10.55
    UB: 3210.56 - LB: 10.56
    UB: 3212.90 - LB: 12.90
    UB: 3220.11 - LB: 20.11
    UB: 3220.74 - LB: 20.74
    UB: 3220.81 - LB: 20.81
    UB: 3221.25 - LB: 21.25
    UB: 3224.19 - LB: 24.19
    UB: 3230.80 - LB: 30.80
    UB: 3231.69 - LB: 31.69
    UB: 3232.80 - LB: 32.80
    UB: 3233.77 - LB: 33.77
    UB: 3352.03 - LB: 34.25
    UB: 3234.29 - LB: 34.29
    UB: 3234.74 - LB: 34.74
    UB: 2743.14 - LB: 34.75
    UB: 3365.28 - LB: 2152.35
    UB: 2756.38 - LB: 2152.85
    UB: 2743.14 - LB: 2743.14

    (Any[3200.0, 3210.054529448178, 3210.5504810699804, 3210.562429355936, 3212.898361166845, 3220.1136273655734, 3220.742821014491, 3220.807068585424, 3221.2507209222485, 3224.190843426695  …  3231.6882584851733, 3232.8006372684135, 3233.773629273072, 3352.0328794045035, 3234.286055186306, 3234.7413244966756, 2743.136793777782, 3365.27728915555, 2756.381203528829, 2743.136793777782], Any[0.0, 10.054529448177977, 10.550481069980382, 10.5624293559357, 12.89836116684521, 20.113627365573514, 20.742821014490755, 20.80706858542427, 21.250720922248476, 24.19084342669534  …  31.688258485173435, 32.80063726841344, 33.77362927307213, 34.24994134409088, 34.28605518630586, 34.74132449667572, 34.75327278263104, 2152.350838312871, 2152.854169751412, 2743.1367937777823], Any[0.001021085, 0.002473057, 0.000699174, 0.001314428, 0.001464564, 0.005083717, 0.00157934, 0.002331034, 0.001769716, 0.002026514  …  0.002012163, 0.002877235, 0.006162246, 0.003313874, 0.002469903, 0.002317671, 0.002410215, 0.002512856, 0.002510676, 0.003399851], Any[0.005480275, 0.004052873, 0.004486044999999999, 0.004129187, 0.006008623, 0.004787151, 0.005361995, 0.0046422600000000005, 0.004833864, 0.005578787  …  0.003817553, 0.005045758, 0.0038845250000000002, 0.0035313410000000003, 0.004731394, 0.004545399, 0.0036984210000000003, 0.003261647, 0.003215611, 0.004845708], Any[0.002845987, 0.002088845, 0.002348809, 0.002199612, 0.003261186, 0.002960261, 0.002686592, 0.002463351, 0.002689996, 0.003098361  …  0.001990776, 0.002670213, 0.00246172, 0.001961768, 0.002371545, 0.002298368, 0.002084203, 0.00166272, 0.001688792, 0.003139663], 32, 10)

```julia
plot([upper_bound_all lower_bound_all], label=["Upper bound" "Lower bound"], xlabel="Iteration")
```

```julia
@show length(upper_bound_all)
@show num_opt
@show num_feas
```

    length(upper_bound_all) = 21
    num_opt = 10
    num_feas = 32

    32


| |Pierre Proulx, ing, professeur|
|:---|:---|
|Département de génie chimique et de génie biotechnologique |** GCH200-Phénomènes d'échanges I **|

### Section 6.2, exemple 6.2-2

##### Dans cet exemple on traite de facon légèrement différente car on suppose une conduite lisse. Ceci explique que le débit soit plus élevé.

```python
#
# Pierre Proulx
#
# Préparation de l'affichage et des outils de calcul symbolique
#
import sympy as sp
from IPython.display import *
sp.init_printing(use_latex=True)
```

```python
# Paramètres, variables et fonctions
rho,L,dP,D,v_z,mu=sp.symbols('rho,L,dP,D,v_z,mu')
```

```python
f=1/4*(D/L)*(dP/(1/2*rho*v_z**2))            # equation définissant le facteur f
Re=rho*v_z*D/mu
f_L=16/Re                                # f calculé si Re<2100
f_T=0.0791/Re**0.25                      # si Re > 2100
```

```python
# Dictionnaire contenant les valeurs des paramètres
dico={'rho':1000,'D':7.981*2.54/100,'mu':0.001,'dP':3/14.7*101325,'L':1000*0.3048}
#
# a priori on ne sait pas si c'est laminaire ou pas,
# donc en premier on teste laminaire
#
eqL=sp.Eq(f-f_L)
v1=sp.solve((eqL,0),v_z)
Re1=Re.subs(v1)
Re1=Re1.subs(dico).evalf()
if Re1 > 2100:
    print(' Re estimé =',Re1)
    print(' Pas laminaire, on calcule en turbulent')
    eqT=sp.Eq(f-f_T)
    v2=sp.solve((eqT,0),v_z,dict=True)    # le dict=True est utilisé car il
    v2=v2.pop()                           # y a plusieurs solutions a cause de la forme de f(Re).
    display(v2)                           # le pop() va chercher la première racine
    Re2=Re.subs(v2)                       # qui est justement réelle
    Re2=Re2.subs(dico)
    print(' le Reynolds calculé avec Blasius pour conduite lisse')
    display(Re2.evalf())
    display(' Débit massique calculé en kg/sec')
    V=((v_z.subs(v2)).subs(dico)).evalf()        # evalf() car le résultat est symbolique
    W=rho*sp.pi*D**2/4*V                         # et on le veut en format numérique,
    display(W.subs(dico).evalf())                # c'est similaire à Matlab.
else:
    print(' le Reynolds calculé laminaire pour conduite lisse')
    display(Re1)
    V=((v_z.subs(v1)).subs(dico)).evalf()
    W=rho*sp.pi*D**2/4*V
    display(' Débit massique calculé en kg/sec')
    display(W.subs(dico).evalf())
```

```python

```


# Atmospheric Model

It should not come as much surprise that the core of our flight simulation models are derived from a model of the atmosphere that we're flying in. There are many such atmospheric models, but the one used most commonly in aviation is the International Standard Atmosphere.

...

### Symbols used

Since the primary source upon which the models in this package are built from is the ISA, I will use the symbols defined in the paper that defined it.

```python
from symbols import *

display(symbol_table)
```

Symbol | Description
-------|:-----------
$A$ | Area
$F$ | Force
$g$ | Acceleration due to gravity
$H$ | Geopotential height
$h$ | Geometric height
$P$ | Pressure
$P_{b}$ | Pressure at the base of an atmospheric layer
$M$ | Molecular mass
$m$ | Mass
$n$ | Amount of substance
$R^{*}$ | Universal gas constant
$R$ | Specific gas constant
$T$ | Thermodynamic temperature
$V$ | Volume
$\rho$ | Density

## Atmospheric Model

The basis of all of our calculations and simulations is a model of Earth's atmosphere, of which there are several.  The [International Standard Atmosphere](https://en.wikipedia.org/wiki/International_Standard_Atmosphere) (ISA) is used most commonl in the field of aviation, so that is what we will use.

The ISA is a mathematical model of the atmosphere that assumes a linear temperature gradient against altitude, and derives pressure and density from the [ideal gas law](https://en.wikipedia.org/wiki/Ideal_gas_law) and [hydrostatic equilibrium](https://en.wikipedia.org/wiki/Hydrostatic_equilibrium).

### Hydrostatic equation

Our model begins by noting that since the atmosphere is not floating away into space, nor sinking to the ground, it must be at equilibrium: all vertical forces must sum to zero. That is to say, it is at [hydrostatic equilibrium](https://en.wikipedia.org/wiki/Hydrostatic_equilibrium). For a column of air in the atmosphere, the 3 forces we'll model are from the surrounding air pressure - air pushing up from below, air pushing down from above - and the weight of the column of air itself:

$0 = F_{below} + F_{above} + F_{weight}$

Replacing with the definitions of pressure and weight gives us (For reasons that will become apparent, we will refer to the pressure at the base of the cylinder as $P_b$ and the pressure at the top simply as $P$):

```python
Eq((A*P_b) - (A*P) - (m*g), 0)
```

$\displaystyle - A P + A P_{b} - g m = 0$

```python
# The mass of a column of air is its volume multiplied by its density:
# m = V * rho
_.replace(m, V*rho)
```

$\displaystyle - A P + A P_{b} - V g \rho = 0$

```python
# The volume is its area multiplied by its height.
_.replace(V, A * h)
```

$\displaystyle - A P + A P_{b} - A g h \rho = 0$

```python
# We can simplify by removing the area:
_ / A
```

$\displaystyle P - P_{b} + g h \rho = 0$

```python
_.replace(P-P_b, diff(P))
```

$\displaystyle P - P_{b} + g h \rho = 0$

```python
z, z_1, z_2, g, R, T_v, dz = symbols('z z_1 z_2 g R T_v dz', real=True)
p_1, p_2, dp, p = symbols('p_1 p_2 dp p', real=True)

P = Symbol('P')
integrate(1/P, (P, p_1, p_2))
```

$\displaystyle - \log{\left(p_{1} \right)} + \log{\left(p_{2} \right)}$

### Ideal Gas Law

The first useful equation for us is the [ideal gas law](https://en.wikipedia.org/wiki/Ideal_gas_law), which relates the pressure and volume of an ideal gas to it's temperature. In short: as the temperature of a gas increases, it's pressure and/or volume must also increase.  It is commonly stated as $PV = nR^*T$, but since it can be hard to count the number of molecules in some volume of gas, it is useful to transform into the molar form through some simple substitutions. The molar form is convenient in atmoshpheric science as we can more easily measure air pressure, temperature, and density.

```python
# pV = nRT
ideal_gas_law = Eq(P * V, n * R_universal * T)
ideal_gas_law
```

$\displaystyle P V = R^{*} T n$

```python
# The amount of substance in a thing is equivalent to its mass / molecular mass:
# n = m / M
ideal_gas_law.subs(n, m / M)
```

$\displaystyle P V = \frac{R^{*} T m}{M}$

```python
# Density is defined as mass / volume:
# rho = m / V
_.solve(P).subs(m/V, rho)
```

```python
# Finally, the specific gas constant of a gas is defined as the universal gas constant / molecular mass:
# R_sp = R / M
molar_gas_law = _.subs(R_universal/M, R)
molar_gas_law
```

## Burst Altitude

The maximum diameter of a given balloon is given by its manufacturer.  If we know the volume of gas that we've put into the balloon, we can use the ideal gas law to determine the pressure at which the balloon will burst.

```python
r = symbols('r')

volume_of_sphere = Eq(V, Rational(4,3)*pi*r**3)
volume_of_sphere

foo = ideal_gas_law.subs(volume_of_sphere.lhs, volume_of_sphere.rhs)
display(foo)

solved = next(iter(solveset(foo, P)))

from sympy.printing.pycode import pycode

display(solved)
pycode(solved)
```

```python
from sympy import *

F_b, m, g = symbols('F_b m g')
rho, V, r = symbols('rho V r')
rho_air, rho_gas = symbols('rho_air rho_gas')

bouyancy = Eq(F_b, m*g)
bouyancy = bouyancy.subs({
    m: rho*V,
    rho: (rho_gas - rho_air),
    V: (Rational(4,3)*pi*r**3)
})

bouyancy
```

## References

* [ISO 2533-1975: Standard Atmosphere](https://cdn.standards.iteh.ai/samples/7472/c203e9121d4c40e5bdc98844b1a1e2f4/ISO-2533-1975.pdf)
* [Inspiration](https://github.com/cuspaceflight/cusf-burst-calc/blob/master/js/calc.js)
* [And](https://northstar-www.dartmouth.edu/~klynch/pmwiki-gc/uploads/BalloonCalulations.pdf)
* [And](https://amt.copernicus.org/articles/4/2235/2011/amt-4-2235-2011.pdf)

```python

```


```python
from sympy.physics.mechanics import *
import sympy as sp
mechanics_printing(pretty_print=True)
```

```python
m1,m2,l1,l2 = sp.symbols(r'm_1,m_2,l_1,l_2')
t,g,h = sp.symbols('t,g,h')
v1, v2 = dynamicsymbols(r'\theta_1 \theta_2')
dv1, dv2 = dynamicsymbols(r'\theta_1 \theta_2', 1)
```

```python
x1 = l1*sp.sin(v1)
y1 = -l1*sp.cos(v1)
x2 = x1 + l2*sp.sin(v2)
y2 = y1 + -l2*sp.cos(v2)

dx1 = x1.diff(t)
dy1 = y1.diff(t)
dx2 = x2.diff(t)
dy2 = y2.diff(t)
```

```python
V = m1*g*y1 + m2*g*y2
T = sp.Rational(1, 2)*m1*(dx1**2+dy1**2)+sp.Rational(1, 2)*m2*(dx2**2+dy2**2)
L = T - V
```

```python
LM = LagrangesMethod(L, [v1, v2])
```

```python
soln = LM.form_lagranges_equations()
```

```python
solvedsoln = sp.solve((sp.Eq(soln[0]),sp.Eq(soln[1])),(v1.diff(t,t),v2.diff(t,t)))
```

```python
sp.latex(solvedsoln)
```

    '\\left \\{ \\frac{d^{2}}{d t^{2}} \\theta_{1}{\\left (t \\right )} : - \\frac{g m_{1} \\sin{\\left (\\theta_{1}{\\left (t \\right )} \\right )} + \\frac{g m_{2} \\sin{\\left (\\theta_{1}{\\left (t \\right )} - 2 \\theta_{2}{\\left (t \\right )} \\right )}}{2} + \\frac{g m_{2} \\sin{\\left (\\theta_{1}{\\left (t \\right )} \\right )}}{2} + \\frac{l_{1} m_{2} \\sin{\\left (2 \\theta_{1}{\\left (t \\right )} - 2 \\theta_{2}{\\left (t \\right )} \\right )} \\left(\\frac{d}{d t} \\theta_{1}{\\left (t \\right )}\\right)^{2}}{2} + l_{2} m_{2} \\sin{\\left (\\theta_{1}{\\left (t \\right )} - \\theta_{2}{\\left (t \\right )} \\right )} \\left(\\frac{d}{d t} \\theta_{2}{\\left (t \\right )}\\right)^{2}}{l_{1} \\left(m_{1} - m_{2} \\cos^{2}{\\left (\\theta_{1}{\\left (t \\right )} - \\theta_{2}{\\left (t \\right )} \\right )} + m_{2}\\right)}, \\quad \\frac{d^{2}}{d t^{2}} \\theta_{2}{\\left (t \\right )} : \\frac{- \\left(m_{1} + m_{2}\\right) \\left(g \\sin{\\left (\\theta_{2}{\\left (t \\right )} \\right )} - l_{1} \\sin{\\left (\\theta_{1}{\\left (t \\right )} - \\theta_{2}{\\left (t \\right )} \\right )} \\left(\\frac{d}{d t} \\theta_{1}{\\left (t \\right )}\\right)^{2}\\right) + \\left(g m_{1} \\sin{\\left (\\theta_{1}{\\left (t \\right )} \\right )} + g m_{2} \\sin{\\left (\\theta_{1}{\\left (t \\right )} \\right )} + l_{2} m_{2} \\sin{\\left (\\theta_{1}{\\left (t \\right )} - \\theta_{2}{\\left (t \\right )} \\right )} \\left(\\frac{d}{d t} \\theta_{2}{\\left (t \\right )}\\right)^{2}\\right) \\cos{\\left (\\theta_{1}{\\left (t \\right )} - \\theta_{2}{\\left (t \\right )} \\right )}}{l_{2} \\left(m_{1} - m_{2} \\cos^{2}{\\left (\\theta_{1}{\\left (t \\right )} - \\theta_{2}{\\left (t \\right )} \\right )} + m_{2}\\right)}\\right \\}'

```python

```


### Calculating velocity distributions in verne

First, we'll load up some code that we need:

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.interpolate import interp1d

import verne
import MaxwellBoltzmann as MB
```

```python
verne.loadFFcorrections(1.0)
```

    >VERNE: Loading isotope data and density profiles...
    >VERNE: Calculating Form Factor corrections for m_x =  1.0  GeV...

Now let's try to calculate the velocity distribution $f(v)$ at one specific value of $v$, let's call it $v_1$:

```python
v1 = 200 #km/s
f_final = verne.CalcF(v1, gamma=0.0, depth=100, sigma_p=1e-26, m_x=1.0, target="SUF", vmax_interp=None, interaction="SI")
```

Okay, so the code won't let us use `None` :( So it turns out that `vmax_interp` needs to be some kind of function, which returns the maximum possible final speed at the detector $v_{f,\mathrm{max}}$, as a function of $\theta$, the incoming direction of the DM particle. What happens if we specify something very general?

```python
def vmax_interp1(theta):
    return 800 #km/s
```

```python
f_final = verne.CalcF(v1, gamma=0.0, depth=100, sigma_p=1e-30, m_x=1.0, target="SUF", vmax_interp=vmax_interp1, interaction="SI")
```

    /usr/local/lib/python3.9/site-packages/scipy/integrate/odepack.py:247: ODEintWarning: Excess work done on this call (perhaps wrong Dfun type). Run with full_output = 1 to get quantitative information.
      warnings.warn(warning_msg, ODEintWarning)

Hmmm...that seems to work - let's try doing it for a range of velocities:

```python
v_list = np.linspace(0, 800, 20)
%time f_final_list = [verne.CalcF(v, gamma=0.0, depth=100, sigma_p=1e-30, m_x=1.0, target="SUF", vmax_interp=vmax_interp1, interaction="SI") for v in v_list]
```

    CPU times: user 1min 10s, sys: 899 ms, total: 1min 11s
    Wall time: 1min 11s

```python
plt.figure()

plt.plot(v_list, MB.calcf_SHM(v_list), 'k--', label='SHM (unperturbed)')
plt.plot(v_list, f_final_list)

plt.show()
```

This works, but it's pretty slow. Let's calculate the maximum velocity as a function of the incoming angle $\theta$. We want to do this in advance (rather than letting `verne.CalcF` deal with it internally) because we need this function repeatedly (every time we call `verne.CalcF` for a different `v`, we don't want to have to recalculate!).

The maximum velocity at the detector corresponds to the final velocity of the DM particle which has the largest *initial* velocity. The maximum initial velocity depends on the incoming direction of the DM particle. Let's write $\mathbf{v}_g$ as the velocity of a DM particle in the galactic rest frame, $\mathbf{v}$ as the velocity of that same particle in the Earth's frame and $\mathbf{v}_e$ the velocity of the Earth with respect to the Galaxy. Then:

$$
\begin{align}
\mathbf{v}_g &= \mathbf{v} - \mathbf{v}_e\\
\Rightarrow v_g^2&= v^2 + v_e^2 - 2 \mathbf{v}\cdot \mathbf{v}_e
\end{align}
$$

To find the maximum speed in the Earth frame, we'll set $v_g = v_\mathrm{esc}$, the escape velocity in the Galactic rest frame:
$$
\begin{align}
v^2 - 2 \mathbf{v}\cdot \mathbf{v}_e + v_e^2 - v_\mathrm{esc}^2  = 0
\end{align}
$$

Based on the definitions of the angles $\gamma$ and $\theta$, you can show that the angle $\alpha$ between $\mathbf{v}$ and $\mathbf{v}_e$ is:
$$\cos\alpha = \sin\gamma\,\sin(\pi -\theta) + \cos\gamma\,\cos(\pi - \theta)$$
*(This is actually a nice exercise, to check that you understand what these angles mean).*

We finally have:
$$
\begin{align}
v^2 - 2 v\,v_e\cos\alpha + (v_e^2 - v_\mathrm{esc}^2)  = 0
\end{align}
$$
which can be solved to obtain $v$, the maximum initial velocity as a function of $\gamma$ and $\theta$.

```python
#Let's fix a value of gamma and calculate for a range of theta
gamma = 0.3

Nvals = 501
thetavals = np.linspace(0, np.pi, Nvals)

#Loading values from MaxwellBoltzmann module:
v_e = np.sqrt(2.0)*MB.sigmav
vesc = MB.vesc

#Solving using quadratic formula:
a = 1.0
b = 2*v_e*(-np.sin(gamma)*np.sin(np.pi-thetavals) + np.cos(gamma)*np.cos(np.pi-thetavals))
c = v_e**2 - vesc**2
v_initial_max = (-b + np.sqrt(b**2 - 4*a*c))/(2.0*a)

#Now we loop over values of theta, calculating the final velocity at the detector
        #Calculate the maximum final speed as a function of incoming angle theta
v_final_max = 0.0*v_initial_max
for i in range(Nvals):
    v_final_max[i] = verne.calcVfinal_full(v_initial_max[i], thetavals[i],  depth=100, sigma_p=1e-30, m_x=1.0, interaction="SI", target="SUF")

#Now generate an interpolation function for v_final(theta):
vmax_interp2 = interp1d(thetavals, v_final_max, kind='linear', bounds_error=False, fill_value=0)
```

```python
#This thing works just like any other function:
print(vmax_interp2(0.75*np.pi))
```

    363.3919867453854

**WARNING:** Note here that you should make sure that the parameters you use to calculate this interpolation function (depth, mass, cross section etc) agree with the parameters you'll use to calculate the velocity distribution. Otherwise things won't be consistent.

It's possible that there's a more refined way of doing all this, but for the moment, this saves the code having to store these things internally...

Now let's try calculating things again, using this more accurate function for v_max(theta):

```python
v_list = np.linspace(0, 800, 20)
%time f_final_list = [verne.CalcF(v, gamma=0.0, depth=100, sigma_p=1e-30, m_x=1.0, target="SUF", vmax_interp=vmax_interp2, interaction="SI") for v in v_list]
```

    /usr/local/lib/python3.9/site-packages/scipy/integrate/odepack.py:247: ODEintWarning: Excess work done on this call (perhaps wrong Dfun type). Run with full_output = 1 to get quantitative information.
      warnings.warn(warning_msg, ODEintWarning)

    CPU times: user 6.13 s, sys: 348 ms, total: 6.48 s
    Wall time: 6.19 s

```python
plt.figure()

plt.plot(v_list, MB.calcf_SHM(v_list), 'k--', label='SHM (unperturbed)')
plt.plot(v_list, f_final_list)

plt.show()
```

Great! That's now faster, because the code isn't wasting time calculating for speeds which are unphysically large!

```python

```