# Practical Session 3: Ensemble Learning Techniques

*Notebook by Ekaterina Kochmar*

This practical will address the use of ensemble-based learning techniques. You will be working with the otherwise familiar settings of classification and regression tasks. In this practical, you will use the familiar datasets and will also learn how to generate artificial datasets.

## Learning objectives

In this practical you will learn about:
- simple voting classifiers using hard and soft voting strategies
- bagging and pasting ensembles
- boosting and early stopping
- popular ensemble-based learning algorithms including `RandomForests` and `AdaBoost`

In addition, you will get more practice with `sklearn's` machine learning routines, and you will learn how to implement ensembles both from scratch and using `sklearn's` implementation.

**Bibliography**: Aurelien Geron, *Hands-On Machine Learning with Scikit-Learn and TensorFlow*.

## Setup

First of all, let's import a few common modules, ensure `matplotlib` plots are inline and define the parameters for the figures. Feel free to add your own settings for the notebook here:

```python
# Common imports
import numpy as np
import os

# to make this notebook's output stable across runs
np.random.seed(42)

# To plot pretty figures
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
plt.rcParams['axes.labelsize'] = 14
plt.rcParams['xtick.labelsize'] = 12
plt.rcParams['ytick.labelsize'] = 12
```

## Simple voting classifiers: hard and soft voting strategies

[The wisdom of the crowd](https://en.wikipedia.org/wiki/Wisdom_of_the_crowd) is the collective opinion of a group of individuals rather than that of a single expert. It has been shown that if you ask a complex question to thousands of random people and then aggregate their answers, in many cases you are likely to get an answer that is more accurate than an expert's prediction. This effect is nowadays widely used in practice and can be said to provide the basis for crowdsourcing approaches.

By analogy, if you combine decisions of multiple diverse classifiers, the resulting vote will often be better than a vote of any single predictor. Such combination of classifiers is called _ensemble_. Perhaps somewhat counterintuitively, the aggregated vote of an ensemble often achieves a higher accuracy than the best classifier in the ensemble. This works even if each predictor in the ensemble is a _weak learner_ (meaning that it works only slightly better than random guessing). The ensemble can still be a _strong learner_ achieving high accuracy, provided that there are:
- a sufficient number of weak learners, and
- their decisions are sufficiently diverse.

How is this possible? Here is a statistical analogy: suppose you have a slightly biased coin that has a $51\%$ chance of coming up heads and a $49\%$ chance of coming up tails. If you toss it $1000$ times, you will get around $510$ heads and $490$ tails, and hence a majority of heads. If you do the math or run simulations, you will find out that the probability of obtaining the majority of heads after $1000$ tosses of this coin approaches $73\%$. After $10K$ tosses, the probability of having a majority of heads is about $97\%$. This is explained by the [_law of large numbers_](https://en.wikipedia.org/wiki/Law_of_large_numbers): over a large number of tosses the ratio of heads gets closer to the probability of heads ($51\%$).

```python
import scipy, math
import scipy.special

def probability(p, n, x):
    binom = scipy.special.comb(n, x, exact=True) * math.pow(p, x) * math.pow((1-p), n-x)
    return binom

prob_maj = 0.0
for x in range(501, 1001):
    prob_maj += probability(0.51, 1000, x)

print(prob_maj)
```

    0.7260985557305037

The code below simulates $10$ independent series of coin tosses, and then plots the results. You can see in the figure below that as the number of tosses increases, all $10$ series end up close to $51\%$ and thus are consistently above $50\%$.

```python
np.random.seed(42)
heads_proba = 0.51
coin_tosses = (np.random.rand(10000, 10) < heads_proba).astype(np.int32)
cumulative_heads_ratio = np.cumsum(coin_tosses, axis=0) / np.arange(1, 10001).reshape(-1, 1)

plt.figure(figsize=(8,3.5))
plt.plot(cumulative_heads_ratio)
plt.plot([0, 10000], [0.51, 0.51], "k--", linewidth=2, label="51%")
plt.plot([0, 10000], [0.5, 0.5], "k-", label="50%")
plt.xlabel("Number of coin tosses")
plt.ylabel("Heads ratio")
plt.legend(loc="lower right")
plt.axis([0, 10000, 0.42, 0.58])
plt.show()
```

Similarly, if you combine $1000$ classifiers, each of which is individually correct only $51\%$ of the time, and then predict the majority voted class, you can get up to $73\%$ accuracy.

For this to work, the classifiers need to be independent and sufficiently diverse, i.e. they need to make independent, uncorrelated errors. In practice, this is hard to achieve because the classifiers are usually all trained on the same data.

Let's apply voting strategy to some data: this time, we'll be working with a synthetic (i.e., artificially generated) [`moons` dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html), that includes two interleaving half circles and provides a good 'toy' example for testing out classification strategies.

Let's first generate the data points and visualise them on a plot:

```python
from sklearn.datasets import make_moons
X, y = make_moons(n_samples=500, noise=0.30, random_state=42)

def plot_dataset(X, y, axes):
    plt.plot(X[:, 0][y==0], X[:, 1][y==0], "bs")
    plt.plot(X[:, 0][y==1], X[:, 1][y==1], "g^")
    plt.axis(axes)
    plt.grid(True, which='both')
    plt.xlabel(r"$x_1$", fontsize=20)
    plt.ylabel(r"$x_2$", fontsize=20, rotation=0)

plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])
plt.show()
```

Next, let's split the dataset into training and test sets and train some classifiers on the training data:

```python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
```

The code below shows how to apply several classifiers to this data, with a couple of classifiers that we haven't used before in this course: [`RandomForestClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) and [`Support Vector Machines`](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html) (`SVC`) classifier. For the moment, don't worry about the particular details of the classifiers' implementation (if you are interested in learning more about them, check out `sklearn's` documentation; we'll discuss `RandomForestClassifier` in this practical; `Support Vector Machines` is covered in detail in the Part II [Machine Learning and Bayesian Inference](https://www.cl.cam.ac.uk/teaching/2021/MLBayInfer/) course). The point here is that we are combining *diverse* classifiers in an ensemble: in essence, `RandomForestClassifier` runs recursive partitioning on the data, while `Support Vector Machines` try to construct a linear boundary between the classes (as discussed in Lecture 3). Feel free to experiment with your own selection of classifiers from the [`sklearn's`](https://scikit-learn.org/stable/supervised_learning.html#supervised-learning) suite.

```python
from sklearn.ensemble import VotingClassifier

from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC

log_clf = LogisticRegression(random_state=42)
rnd_clf = RandomForestClassifier(random_state=42)
svm_clf = SVC(random_state=42)

voting_clf = VotingClassifier(
    estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
    voting='hard')
voting_clf.fit(X_train, y_train)
```

    VotingClassifier(estimators=[('lr', LogisticRegression(random_state=42)),
                                 ('rf', RandomForestClassifier(random_state=42)),
                                 ('svc', SVC(random_state=42))])

Let's look into how correlated the predictions of the classifiers are: first, get the predictions on the test data, next store them as a `pandas` DataFrame, and finally apply `.corr()` function:

```python
import pandas as pd

def get_predictions(clf):
    clf.fit(X_train, y_train)
    return clf.predict(X_test)

preds = {'lr': get_predictions(log_clf),
        'rf': get_predictions(rnd_clf),
        'svc': get_predictions(svm_clf)}
df = pd.DataFrame(data=preds)
df[:100]
```

      lr
      rf
      svc

      0
      1
      1
      0

      1
      0
      0
      0

      2
      0
      0
      0

      3
      1
      1
      1

      4
      1
      1
      1

      ...
      ...
      ...
      ...

      95
      1
      1
      1

      96
      0
      0
      0

      97
      1
      0
      0

      98
      0
      0
      0

      99
      1
      1
      1

100 rows × 3 columns

```python
df.corr()
```

      lr
      rf
      svc

      lr
      1.000000
      0.774252
      0.774595

      rf
      0.774252
      1.000000
      0.872101

      svc
      0.774595
      0.872101
      1.000000

Will this correlation in the individual classifiers' predictions be sufficient for an ensemble? Let's check this out by combining the votes with a *hard voting strategy*: the ensemble classifier will simply choose the majority class predicted by the three classifiers.

The code below prints out individual classifiers' accuracy scores along with the ensemble's accuracy score:

```python
from sklearn.metrics import accuracy_score

for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
```

    LogisticRegression 0.864
    RandomForestClassifier 0.896
    SVC 0.896
    VotingClassifier 0.912

As you can see, the voting classifier slightly outperforms each of the three individual classifiers, including the most accurate one(s). In addition, if individual classifiers can estimate class probabilities (i.e., if they have `predict_proba()` method), you can use _soft voting_ strategy, that is, estimate highest class probability averaged over individual classifiers. In comparison to the hard voting strategy, soft voting gives more weight to highly confident votes.

`SVC` classifier doesn't estimate class probabilities by default, so you need to set the `probability` hyperparameter to `True`.

```python
log_clf = LogisticRegression(random_state=42)
rnd_clf = RandomForestClassifier(random_state=42)
svm_clf = SVC(probability=True, random_state=42)

voting_clf = VotingClassifier(
    estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
    voting='soft')
voting_clf.fit(X_train, y_train)
```

    VotingClassifier(estimators=[('lr', LogisticRegression(random_state=42)),
                                 ('rf', RandomForestClassifier(random_state=42)),
                                 ('svc', SVC(probability=True, random_state=42))],
                     voting='soft')

Not let's estimate the accuracy of the voting classifier in this mode:

```python
from sklearn.metrics import accuracy_score

for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    print(clf.__class__.__name__, accuracy_score(y_test, y_pred))
```

    LogisticRegression 0.864
    RandomForestClassifier 0.896
    SVC 0.896
    VotingClassifier 0.92

## Bagging ensembles: bagging and pasting

One way to ensure that the classifiers are sufficiently diverse in their predicitons is to select different training algorithms. As we said before, often it is hard to ensure that the classifiers' errors are not highly correlated because the classifiers are ultimately trained on the same data.

So, another way to make sure you end up with diverse classifiers is to train them on different random subsets of the training set. There are two ways to do that:
- sampling _with replacement_ is called [_bagging_](https://link.springer.com/article/10.1023/A:1018054314350) – short for _bootstrap aggregating_
- sampling _without replacement_ is called [_pasting_](https://link.springer.com/article/10.1023/A:1007563306331)

Both bagging and pasting allow training instances to be sampled several times across multiple predictors, but only bagging allows training instances to be sampled several times for the same predictor.

When applied to a new, test instance the ensemble classifier aggregates the predictions of all predictors and estimates the statistical mode (i.e., most frequent prediction).

**Question**: Is this similar to the hard or the soft voting strategy?

Let's use `sklearn's BaggingClassifier` to train an ensemble of $500$ [`Decision Trees`](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html), each trained on $100$ training instances randomly selected from the training set with replacement (for bagging, but you can change the strategy to pasting by setting `bootstrap` to `False`). In addition, both bagging and pasting strategies are good for parallelisation: predictors can all be trained in parallel on different CPU cores or even different servers. Setting `n_jobs` parameter to $-1$ tells `sklearn` to use all available CPU cores:

```python
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

#use bootstrap=False for pasting
bag_clf = BaggingClassifier(
    DecisionTreeClassifier(random_state=42), n_estimators=500,
    max_samples=100, bootstrap=True, n_jobs=-1, random_state=42)
#n_jobs = use all of the available CPU cores
bag_clf.fit(X_train, y_train)
y_pred = bag_clf.predict(X_test)
```

**Question**: [`BaggingClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingClassifier.html) automatically performs soft voting when the base classifier can estimate class probabilities (i.e., has a `predict_proba()` method). Which strategy – hard or soft voting – does `BaggingClassifier` follow with the `DecisionTreeClassifier` as the base classifier?

Let's calculate the accuracy score for the bagging classifier that combines $500$ Decision Tree estimators with a prediction of a single Decision Tree trained on the same data:

```python
print(accuracy_score(y_test, y_pred))
```

    0.904

```python
tree_clf = DecisionTreeClassifier(random_state=42)
tree_clf.fit(X_train, y_train)
y_pred_tree = tree_clf.predict(X_test)
print(accuracy_score(y_test, y_pred_tree))
```

    0.856

To get more insight into the results, let's also plot the decision boundary for the two classifiers:

```python
from matplotlib.colors import ListedColormap

def plot_decision_boundary(clf, X, y, axes=[-1.5, 2.5, -1, 1.5], alpha=0.5, contour=True):
    x1s = np.linspace(axes[0], axes[1], 100)
    x2s = np.linspace(axes[2], axes[3], 100)
    x1, x2 = np.meshgrid(x1s, x2s)
    X_new = np.c_[x1.ravel(), x2.ravel()]
    y_pred = clf.predict(X_new).reshape(x1.shape)
    custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
    plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)
    if contour:
        custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])
        plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=0.8)
    plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo", alpha=alpha)
    plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs", alpha=alpha)
    plt.axis(axes)
    plt.xlabel(r"$x_1$", fontsize=18)
    plt.ylabel(r"$x_2$", fontsize=18, rotation=0)

plt.figure(figsize=(11,4))
plt.subplot(121)
plot_decision_boundary(tree_clf, X, y)
plt.title("Decision Tree", fontsize=14)
plt.subplot(122)
plot_decision_boundary(bag_clf, X, y)
plt.title("Decision Trees with Bagging", fontsize=14)
plt.show()
```

Even though the ensemble makes a comparable number of errors on the training set as a single Decision Tree, its decision boundary is less irregular. This suggests that ensemble's predictions will likely generalise better than the single `Decision Tree`'s predictions
when applied to the new data.

### Out-of-Bag evaluation

With bagging, some of the instances get sampled several times for any given predictor, while others may never be selected at all. By default, `BaggingClassifier` samples $m$ training instances with replacement, where $m$ is the size of the training set. As $m$ grows, the ratio of the training instances that are sampled on average for each predictor approaches $1 - exp(-1)$, i.e. around $63\%$. The remaining $37\%$ of the training instances that are never used by the predictors are called *out-of-bag* (*oob*) instances.

Mathematical proof of the above: With $m$ training instances in the set, the probability of not picking an instance in a random draw is $\frac{m-1}{m}$. For sampling with replacement, the probability of not picking an instance in $m$ random draws is $(\frac{m-1}{m})^m$. In the limit of a large number of $m$, this becomes:

\begin{equation}
\lim_{m\to\infty} (1 - \frac{1}{m})^m = e^{-1} = 0.368 \approx 37\%
\end{equation}

Since a predictor never sees these instances during training, they can be used for evaluation without the need for an additional validation set or cross-validation experiments. You can evaluate the ensemble itself by averaging each predictor's performance on the oob instances.

The code below shows you how to do that: if you set `oob_score` to `True`, the bagging classifier will be evaluated on the oob instances after training, and you can output the score as `oob_score_`.

```python
bag_clf = BaggingClassifier(
    DecisionTreeClassifier(random_state=42), n_estimators=500,
    bootstrap=True, n_jobs=-1, oob_score=True, random_state=40)
bag_clf.fit(X_train, y_train)
bag_clf.oob_score_
```

    0.8986666666666666

This means that the bagging classifier is likely to achieve around $90\%$ accuracy on the test data, too. Let's check the results:

```python
from sklearn.metrics import accuracy_score
y_pred = bag_clf.predict(X_test)
accuracy_score(y_test, y_pred)
```

    0.912

Close enough! You can also take a look into the decision process by printing out the predictions of the classifier on the training instances. Since the base estimator has a `predict_proba()` method, what you see here are the class probabilities assigned by the classifier to negative and positive classes for each instance:

```python
bag_clf.oob_decision_function_
```

    array([[0.32275132, 0.67724868],
           [0.34117647, 0.65882353],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.09497207, 0.90502793],
           [0.31147541, 0.68852459],
           [0.01754386, 0.98245614],
           [0.97109827, 0.02890173],
           [0.97765363, 0.02234637],
           [0.74404762, 0.25595238],
           [0.        , 1.        ],
           [0.7173913 , 0.2826087 ],
           [0.85026738, 0.14973262],
           [0.97222222, 0.02777778],
           [0.0625    , 0.9375    ],
           [0.        , 1.        ],
           [0.97837838, 0.02162162],
           [0.94642857, 0.05357143],
           [1.        , 0.        ],
           [0.01704545, 0.98295455],
           [0.39473684, 0.60526316],
           [0.88700565, 0.11299435],
           [1.        , 0.        ],
           [0.97790055, 0.02209945],
           [0.        , 1.        ],
           [0.99428571, 0.00571429],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.62569832, 0.37430168],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.13402062, 0.86597938],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.38251366, 0.61748634],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.27093596, 0.72906404],
           [0.34146341, 0.65853659],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.00531915, 0.99468085],
           [0.98843931, 0.01156069],
           [0.91428571, 0.08571429],
           [0.97282609, 0.02717391],
           [0.98019802, 0.01980198],
           [0.        , 1.        ],
           [0.07361963, 0.92638037],
           [0.98019802, 0.01980198],
           [0.0052356 , 0.9947644 ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.97790055, 0.02209945],
           [0.8       , 0.2       ],
           [0.42424242, 0.57575758],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.66477273, 0.33522727],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.86781609, 0.13218391],
           [1.        , 0.        ],
           [0.56725146, 0.43274854],
           [0.1576087 , 0.8423913 ],
           [0.66492147, 0.33507853],
           [0.91709845, 0.08290155],
           [0.        , 1.        ],
           [0.16759777, 0.83240223],
           [0.87434555, 0.12565445],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.995     , 0.005     ],
           [0.        , 1.        ],
           [0.07878788, 0.92121212],
           [0.05418719, 0.94581281],
           [0.29015544, 0.70984456],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.83040936, 0.16959064],
           [0.01092896, 0.98907104],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.21465969, 0.78534031],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.94660194, 0.05339806],
           [0.77094972, 0.22905028],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.16574586, 0.83425414],
           [0.65306122, 0.34693878],
           [0.        , 1.        ],
           [0.02564103, 0.97435897],
           [0.50555556, 0.49444444],
           [1.        , 0.        ],
           [0.03208556, 0.96791444],
           [0.99435028, 0.00564972],
           [0.23699422, 0.76300578],
           [0.49509804, 0.50490196],
           [0.9947644 , 0.0052356 ],
           [0.00555556, 0.99444444],
           [0.98963731, 0.01036269],
           [0.26153846, 0.73846154],
           [0.92972973, 0.07027027],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.80113636, 0.19886364],
           [1.        , 0.        ],
           [0.0106383 , 0.9893617 ],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.98181818, 0.01818182],
           [1.        , 0.        ],
           [0.01036269, 0.98963731],
           [0.97752809, 0.02247191],
           [0.99453552, 0.00546448],
           [0.01960784, 0.98039216],
           [0.17857143, 0.82142857],
           [0.98387097, 0.01612903],
           [0.29533679, 0.70466321],
           [0.98295455, 0.01704545],
           [0.        , 1.        ],
           [0.00561798, 0.99438202],
           [0.75690608, 0.24309392],
           [0.38624339, 0.61375661],
           [0.40625   , 0.59375   ],
           [0.87368421, 0.12631579],
           [0.92462312, 0.07537688],
           [0.05181347, 0.94818653],
           [0.82802548, 0.17197452],
           [0.01546392, 0.98453608],
           [0.        , 1.        ],
           [0.02298851, 0.97701149],
           [0.9726776 , 0.0273224 ],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.01041667, 0.98958333],
           [0.        , 1.        ],
           [0.03804348, 0.96195652],
           [0.02040816, 0.97959184],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.94915254, 0.05084746],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.99462366, 0.00537634],
           [0.        , 1.        ],
           [0.39378238, 0.60621762],
           [0.33152174, 0.66847826],
           [0.00609756, 0.99390244],
           [0.        , 1.        ],
           [0.3172043 , 0.6827957 ],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.00588235, 0.99411765],
           [0.        , 1.        ],
           [0.98924731, 0.01075269],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.62893082, 0.37106918],
           [0.92344498, 0.07655502],
           [0.        , 1.        ],
           [0.99526066, 0.00473934],
           [1.        , 0.        ],
           [0.98888889, 0.01111111],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.06989247, 0.93010753],
           [1.        , 0.        ],
           [0.03608247, 0.96391753],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.02185792, 0.97814208],
           [1.        , 0.        ],
           [0.95808383, 0.04191617],
           [0.78362573, 0.21637427],
           [0.56650246, 0.43349754],
           [0.        , 1.        ],
           [0.18023256, 0.81976744],
           [1.        , 0.        ],
           [0.93121693, 0.06878307],
           [0.97175141, 0.02824859],
           [1.        , 0.        ],
           [0.00531915, 0.99468085],
           [0.        , 1.        ],
           [0.43010753, 0.56989247],
           [0.85858586, 0.14141414],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.00558659, 0.99441341],
           [0.        , 1.        ],
           [0.96923077, 0.03076923],
           [0.        , 1.        ],
           [0.21649485, 0.78350515],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.98477157, 0.01522843],
           [0.8       , 0.2       ],
           [0.99441341, 0.00558659],
           [0.        , 1.        ],
           [0.09497207, 0.90502793],
           [0.99492386, 0.00507614],
           [0.01714286, 0.98285714],
           [0.        , 1.        ],
           [0.02747253, 0.97252747],
           [1.        , 0.        ],
           [0.77005348, 0.22994652],
           [0.        , 1.        ],
           [0.90229885, 0.09770115],
           [0.98387097, 0.01612903],
           [0.22222222, 0.77777778],
           [0.20348837, 0.79651163],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.20338983, 0.79661017],
           [0.98181818, 0.01818182],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.98969072, 0.01030928],
           [0.        , 1.        ],
           [0.48663102, 0.51336898],
           [1.        , 0.        ],
           [0.00529101, 0.99470899],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.08379888, 0.91620112],
           [0.12352941, 0.87647059],
           [0.99415205, 0.00584795],
           [0.03517588, 0.96482412],
           [1.        , 0.        ],
           [0.39790576, 0.60209424],
           [0.05434783, 0.94565217],
           [0.53191489, 0.46808511],
           [0.51898734, 0.48101266],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.60869565, 0.39130435],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.24157303, 0.75842697],
           [0.81578947, 0.18421053],
           [0.08717949, 0.91282051],
           [0.99453552, 0.00546448],
           [0.82142857, 0.17857143],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [0.11904762, 0.88095238],
           [0.04188482, 0.95811518],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.89150943, 0.10849057],
           [0.19230769, 0.80769231],
           [0.95238095, 0.04761905],
           [0.00515464, 0.99484536],
           [0.59375   , 0.40625   ],
           [0.07692308, 0.92307692],
           [0.99484536, 0.00515464],
           [0.83684211, 0.16315789],
           [0.        , 1.        ],
           [0.99484536, 0.00515464],
           [0.95360825, 0.04639175],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.26395939, 0.73604061],
           [0.98461538, 0.01538462],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.00574713, 0.99425287],
           [0.85142857, 0.14857143],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.75301205, 0.24698795],
           [0.8969697 , 0.1030303 ],
           [1.        , 0.        ],
           [0.75555556, 0.24444444],
           [0.48863636, 0.51136364],
           [0.        , 1.        ],
           [0.92473118, 0.07526882],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.87709497, 0.12290503],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.74752475, 0.25247525],
           [0.09146341, 0.90853659],
           [0.42268041, 0.57731959],
           [0.22395833, 0.77604167],
           [0.        , 1.        ],
           [0.87046632, 0.12953368],
           [0.78212291, 0.21787709],
           [0.00507614, 0.99492386],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.02884615, 0.97115385],
           [0.96      , 0.04      ],
           [0.93478261, 0.06521739],
           [1.        , 0.        ],
           [0.50731707, 0.49268293],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.01604278, 0.98395722],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.96987952, 0.03012048],
           [0.        , 1.        ],
           [0.05172414, 0.94827586],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.99494949, 0.00505051],
           [0.01675978, 0.98324022],
           [1.        , 0.        ],
           [0.14583333, 0.85416667],
           [0.        , 1.        ],
           [0.00546448, 0.99453552],
           [0.        , 1.        ],
           [0.41836735, 0.58163265],
           [0.13095238, 0.86904762],
           [0.22110553, 0.77889447],
           [1.        , 0.        ],
           [0.97647059, 0.02352941],
           [0.21195652, 0.78804348],
           [0.98882682, 0.01117318],
           [0.        , 1.        ],
           [0.        , 1.        ],
           [1.        , 0.        ],
           [0.96428571, 0.03571429],
           [0.34554974, 0.65445026],
           [0.98235294, 0.01764706],
           [1.        , 0.        ],
           [0.        , 1.        ],
           [0.99465241, 0.00534759],
           [0.        , 1.        ],
           [0.06043956, 0.93956044],
           [0.98214286, 0.01785714],
           [1.        , 0.        ],
           [0.03108808, 0.96891192],
           [0.58854167, 0.41145833]])

## Random Forests

[`RandomForestClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) is an ensemble of Decision Trees typically trained via the bagging method. The number of training instances (`max_samples`) is usually set to the total size of the training set. So, in fact, `RandomForestClassifier` will be roughly equivalent to the `BaggingClassifier` that takes Decision Trees as base estimators with the following parameters:

```python
bag_clf = BaggingClassifier(
    DecisionTreeClassifier(splitter="random", max_leaf_nodes=16, random_state=42),
    n_estimators=500, max_samples=1.0, bootstrap=True, n_jobs=-1, random_state=42)

bag_clf.fit(X_train, y_train)
y_pred = bag_clf.predict(X_test)
```

The key difference is that, rather than using `BaggingClassifier` and passing it a `DecisionTreeClassifier`, you can rely on the `sklearn's RandomForestClassifier` implementation, which is more convenient and optimised for Decision Trees:

```python
from sklearn.ensemble import RandomForestClassifier

rnd_clf = RandomForestClassifier(n_estimators=500, max_leaf_nodes=16,
                                 n_jobs=-1, random_state=42)
rnd_clf.fit(X_train, y_train)

y_pred_rf = rnd_clf.predict(X_test)
```

Let's estimate the difference between the two classifiers' predictions:

```python
np.sum(y_pred == y_pred_rf) / len(y_pred)  # see to what extent predictions are identical
```

    0.976

I.e., the two classifiers are almost identical in their predictions. The Random Forest algorithm introduces extra randomness when growing trees: instead of searching for the very best feature when splitting a node, it searches for the best feature among a random subset of features. As a result, the model is more diverse and in general yields better results.

Let's visualise the decision boundary for a random set of $15$ Decision Trees to get a better idea of where the diversity comes from:

```python
plt.figure(figsize=(6, 4))

for i in range(15):
    tree_clf = DecisionTreeClassifier(max_leaf_nodes=16, random_state=42 + i)
    indices_with_replacement = np.random.randint(0, len(X_train), len(X_train))
    tree_clf.fit(X[indices_with_replacement], y[indices_with_replacement])
    plot_decision_boundary(tree_clf, X, y, axes=[-1.5, 2.5, -1, 1.5],
                           alpha=0.02, contour=False)

plt.show()
```

**Extra material**: One further way to introduce randomness into the training process is to not only select a random subset of features when splitting a node, but also use a random threshold for each feature rather than searching for the best possible thresholds as traditional Decision Trees do. Since estimating best possible threshold for each feature at every node is also one of the most time-consuming tasks, this modification to the algorithm makes it more efficient. The algorithm that exploits this idea is called *Extremely Randomised Trees ensemble*, or *Extra-Trees*, and you can use it via [`sklearn's ExtraTreeClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.tree.ExtraTreeClassifier.html) implementation.

### Feature importance

Another useful property of the Random Forests classifier is that it can help you measure the relative importance of each feature by looking at how much the tree nodes that use this particular feature reduce impurity of the nodes on average (i.e., across all trees in the forest). The results are scaled so that the sum of all feature importances is equal to $1$.

Because features used in the *iris* and *digits* datasets are highly interpretable, let's estimate feature importances on these two datasets:

```python
from sklearn.datasets import load_iris
iris = load_iris()
rnd_clf = RandomForestClassifier(n_estimators=500, n_jobs=-1, random_state=42)
rnd_clf.fit(iris["data"], iris["target"])
for name, score in zip(iris["feature_names"], rnd_clf.feature_importances_):
    print(name, score)
```

    sepal length (cm) 0.11249225099876375
    sepal width (cm) 0.02311928828251033
    petal length (cm) 0.4410304643639577
    petal width (cm) 0.4233579963547682

```python
rnd_clf.feature_importances_
```

    array([0.11249225, 0.02311929, 0.44103046, 0.423358  ])

**Question**: What do these results suggest? Are they similar to your observations from the previous experiments on this dataset?

*Digits* dataset provides further interpretability of the results if we plot them. What does the figure below suggest?

```python
from sklearn import datasets
digits = datasets.load_digits()

rnd_clf = RandomForestClassifier(random_state=42)
rnd_clf.fit(digits["data"], digits["target"])
```

    RandomForestClassifier(random_state=42)

```python
def plot_digit(data):
    image = data.reshape(8, 8)
    plt.imshow(image, cmap = matplotlib.cm.hot,
               interpolation="nearest")
    plt.axis("off")

plot_digit(rnd_clf.feature_importances_)

cbar = plt.colorbar(ticks=[rnd_clf.feature_importances_.min(), rnd_clf.feature_importances_.max()])
cbar.ax.set_yticklabels(['Not important', 'Very important'])

plt.show()
```

## AdaBoost

*Boosting*, or *hypothesis boosting* is an ensemble method that can combine several weak learners into a strong learner. The general idea is to train predictors sequentially in such a way that each next predictor tries to correct its predecessor. The most popular among boosting methods are *AdaBoost* (which stands for *Adaptive Boosting*) and *Gradient Boosting*. Let's apply *AdaBoost* first.

The idea behind *AdaBoost* is as follows: in order to correct the predecessor's mistakes, the algorithm assigns higher weights to the training instances that the predecessor underfitted. I.e., the next step pays more attention to such instances and the algorithm focuses more on the hard cases.

Here is the step-by-step strategy:
- start with the first classifier (i.e., this can be a Decision Trees classifier)
- train it and use it to make predictions on the training set
- increase the relative weight of misclassified training instances
- train a second classifier with these new updated weights and make new predictions
- update weights using the new predictions
- continue until stopping criteria are satisfied.

For instance, suppose each instance's original weight $w^{(i)}$ is set to $\frac{1}{m}$, where $m$ is the number of instances. The first classifier is applied, and its error rate $r_1$ is computed on the training set using the equation below:

\begin{equation}
r_j = \frac{\sum_{\hat{y}^{(i)}_j \neq y^{(i)}} w^{(i)}}{\sum_{i=1}^m w^{(i)}}
\end{equation}

where $\hat{y}^{(i)}_j$ is the prediction of the $j$-th classifier on the $i$-th instance. The predictor's weight $\alpha_j$ is then estimated using:

\begin{equation}
\alpha_j = \eta log \frac{1-r_j}{r_j}
\end{equation}

where $\eta$ is the learning rate, a hyperparameter that defaults to $1$. The more accurate the predictor is, the higher its weight will be; if a predictor is guessing randomly, its weight will be close to $0$; and if it performs worse than random guessing it will get a high negative weight.

Next, the weights are updated and the misclassified instances are boosted as follows:

\begin{equation}
  w^{(i)}=\begin{cases}
    w^{(i)}, & \text{if $\hat y_j^{(i)} = y_j^{(i)}$}\\
    w^{(i)} exp(\alpha_j), & \text{if $\hat y_j^{(i)} \neq y_j^{(i)}$}
  \end{cases}
\end{equation}

Then all the instances weights are normalised (i.e., divided by $\sum_{i=1}^m w^{(i)}$). The new predictor is trained on the updated training instances, applied to the training set, its weight is computed, weights are updated again, and so on. The algorithm stops when either the predefined number of predictors is reached or a perfect predictor is found.

(This process should remind you about other sequential learning techniques we've discussed, e.g. Gradient Descent. The difference is, instead of tweaking a single predictor's parameters to minimise a cost function, AdaBoost adds predictors to the ensemble gradually making the prediction better.)

**Question**: We said earlier than one of the advantages of bagging and pasting strategies is that predictors can all be trained in parallel. Is the same applicable to AdaBoost?

At prediction time, AdaBoost computes the predictions of all the predictors and weighs them according to $\alpha_j$. The predicted class is then the one that receives the majority of weighted votes, i.e.:

\begin{equation}
 \hat y(x) = argmax_{k} \sum_{j=1; \hat y_j(x)=k} ^{N} \alpha_j
\end{equation}

with $N$ being the number of predictors.

`sklearn's` implementation of AdaBoost uses *Stagewise Additive Modeling using a Multiclass Exponential loss function* (or *SAMME* for short) multiclass version of the algorithm, which in a binary case simply defaults to AdaBoost. In addition, if the base estimators can estimate class probabilities (have `predict_proba()` method), `sklearn's` algorithm relies on class probabilities rather than predictions, which generally performs better. This version is called *SAMME.R*, where *R* stands for "Real".

Let's train [`sklearn's AdaBoostClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html), for example using $200$ *Decision Stumps* (trees composed of a single decision node and two leaf nodes, `max_depth=1`):

```python
from sklearn.ensemble import AdaBoostClassifier

ada_clf = AdaBoostClassifier(
    DecisionTreeClassifier(max_depth=1), n_estimators=200,
    algorithm="SAMME.R", learning_rate=0.5, random_state=42)
ada_clf.fit(X_train, y_train)
```

    AdaBoostClassifier(base_estimator=DecisionTreeClassifier(max_depth=1),
                       learning_rate=0.5, n_estimators=200, random_state=42)

And let's plot the decision boundary of the algorithm:

```python
plot_decision_boundary(ada_clf, X, y)
```

To get a bit more insight into how the decision boundaries change from one step to another and with a different learning rate, let's plot decision boundaries for $5$ consecutive predictors, in this case using `SVC` as the estimator. Note how the first classifiers usually get many instances wrong, while the following predictors are gradually getting better. The plot on the right presents the very same $5$ consecutive classifiers, but assigns half the learning rate (i.e., the misclassified instances weights are only boosted half as much at every iteration):

```python
m = len(X_train)

plt.figure(figsize=(11, 4))
for subplot, learning_rate in ((121, 1), (122, 0.5)):
    sample_weights = np.ones(m)
    plt.subplot(subplot)
    for i in range(5):
        svm_clf = SVC(kernel="rbf", C=0.05, gamma="auto", random_state=42)
        svm_clf.fit(X_train, y_train, sample_weight=sample_weights)
        y_pred = svm_clf.predict(X_train)
        sample_weights[y_pred != y_train] *= (1 + learning_rate)
        plot_decision_boundary(svm_clf, X, y, alpha=0.2)
        plt.title("learning_rate = {}".format(learning_rate), fontsize=16)
    if subplot == 121:
        plt.text(1.70,  -0.90, "1", fontsize=14)
        plt.text(-0.40, -0.35, "2", fontsize=14)
        plt.text(-0.55, -0.05, "3", fontsize=14)
        plt.text(-0.70,  0.20, "4", fontsize=14)
        plt.text(-0.85,  0.45, "5", fontsize=14)

plt.show()
```

In addition, here is the full list of parameters and attributes of the algorithm:

```python
list(m for m in dir(ada_clf) if not m.startswith("_") and m.endswith("_"))
```

    ['base_estimator_',
     'classes_',
     'estimator_errors_',
     'estimator_weights_',
     'estimators_',
     'feature_importances_',
     'n_classes_',
     'n_features_in_']

E.g., you can check classification errors for each estimator in the boosted ensemble as follows:

```python
ada_clf.estimator_errors_
```

    array([0.18133333, 0.27512937, 0.28334096, 0.35488317, 0.3171361 ,
           0.38272512, 0.35630662, 0.33148186, 0.40484099, 0.29959284,
           0.38482533, 0.37793452, 0.4396849 , 0.41956551, 0.41977426,
           0.42276892, 0.44540522, 0.4592429 , 0.43517238, 0.4795022 ,
           0.46538072, 0.48000343, 0.43464541, 0.44653579, 0.47534577,
           0.48382716, 0.45804523, 0.46674707, 0.4374261 , 0.48598125,
           0.48769271, 0.4722106 , 0.46346519, 0.46165373, 0.44202223,
           0.44509139, 0.47099366, 0.45066309, 0.4881181 , 0.47269775,
           0.47884803, 0.46937918, 0.4663008 , 0.44912872, 0.45639441,
           0.46252216, 0.48048523, 0.49000957, 0.4780251 , 0.45748808,
           0.47390931, 0.4836773 , 0.45432253, 0.4816146 , 0.49129336,
           0.48627751, 0.48533895, 0.48977843, 0.4781294 , 0.45990675,
           0.45628973, 0.45333756, 0.48145777, 0.47828933, 0.46468214,
           0.46348201, 0.47643265, 0.471952  , 0.45673327, 0.47977324,
           0.46875606, 0.46464402, 0.45684007, 0.46146966, 0.46037637,
           0.47746753, 0.47412264, 0.47699586, 0.46442197, 0.47295429,
           0.46091199, 0.4563625 , 0.46459194, 0.47996521, 0.47721258,
           0.46659331, 0.46327864, 0.46068569, 0.47494239, 0.48438408,
           0.49138786, 0.47519606, 0.46263183, 0.48221231, 0.47845427,
           0.46707103, 0.46752551, 0.46045753, 0.46761742, 0.46499847,
           0.46395447, 0.49433102, 0.48645588, 0.48202713, 0.47469894,
           0.4653022 , 0.4673878 , 0.48326479, 0.49121441, 0.47963298,
           0.48471417, 0.46201554, 0.49488345, 0.48830469, 0.48244527,
           0.46889398, 0.47253576, 0.461846  , 0.46893359, 0.4663451 ,
           0.46250006, 0.46156224, 0.46978699, 0.47973597, 0.48335403,
           0.478635  , 0.47867593, 0.46810413, 0.46571099, 0.46988454,
           0.47102448, 0.48051697, 0.47955647, 0.48117019, 0.48996885,
           0.48676386, 0.46644644, 0.45875648, 0.49225164, 0.48657145,
           0.46831141, 0.46670826, 0.4642594 , 0.46138968, 0.46510823,
           0.46590595, 0.49260031, 0.48709267, 0.48131461, 0.47309791,
           0.47602392, 0.47498064, 0.47189274, 0.47859843, 0.48096745,
           0.48004431, 0.47278772, 0.46457035, 0.4791516 , 0.47009956,
           0.46682489, 0.46515405, 0.46253448, 0.46803148, 0.46881194,
           0.4695307 , 0.46808798, 0.48691669, 0.48859091, 0.4923833 ,
           0.4834733 , 0.47618303, 0.47679647, 0.48026735, 0.47879651,
           0.47578553, 0.47437284, 0.47017986, 0.4711307 , 0.46857867,
           0.46859443, 0.47158646, 0.47157599, 0.48294599, 0.48442876,
           0.47297868, 0.46764791, 0.4668408 , 0.46366048, 0.4665483 ,
           0.46686315, 0.46672805, 0.49566253, 0.49250423, 0.48836732,
           0.49223529, 0.49201652, 0.4806373 , 0.48783444, 0.48308099])

## Gradient Boosting

Gradient Boosting is another popular boosting algorithm. It also works sequentially adding new predictors to the ensemble, each one correcting the errors from its predecessor. Unlike AdaBoost, it's trying to fit each new predictor to the *residual errors* made by the previous predictor.

Let's this time apply the boosting algorithm to a regression task. First, generate a synthetic noisy quadratic training set:

```python
np.random.seed(42)
X = np.random.rand(100, 1) - 0.5
y = 3*X[:, 0]**2 + 0.05 * np.random.randn(100)
```

Next, let's fit a single [`DecisionTreeRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html):

```python
from sklearn.tree import DecisionTreeRegressor

tree_reg1 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg1.fit(X, y)
```

    DecisionTreeRegressor(max_depth=2, random_state=42)

Now train a second `DecisionTreeRegressor` on the residual errors made by the first predictor:

```python
y2 = y - tree_reg1.predict(X)
tree_reg2 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg2.fit(X, y2)
```

    DecisionTreeRegressor(max_depth=2, random_state=42)

Finally, let's train a third regressor on the residual errors made by the second predictor:

```python
y3 = y2 - tree_reg2.predict(X)
tree_reg3 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg3.fit(X, y3)
```

    DecisionTreeRegressor(max_depth=2, random_state=42)

That's it – you've just built an ensemble containing three trees. It can now be applied to new instances by making predictions based on adding up the predictions from all three trees:

```python
X_new = np.array([[0.8]])

y_pred = sum(tree.predict(X_new) for tree in (tree_reg1, tree_reg2, tree_reg3))

y_pred
```

    array([0.75026781])

Let's visualise the results by plotting the predictions from each tree on the left and the ensemble on the right. E.g., in the first row the ensemble contains only one tree, so its predictions are exactly the same as the first tree's predictions. In the second row, the second tree is trained on the residual errors from the first tree, and the ensemble's prediction is based on the sum of the predictions of the two trees. You can see how the predictions of the ensemble are gradually getting better:

```python
def plot_predictions(regressors, X, y, axes, label=None, style="r-", data_style="b.", data_label=None):
    x1 = np.linspace(axes[0], axes[1], 500)
    y_pred = sum(regressor.predict(x1.reshape(-1, 1)) for regressor in regressors)
    plt.plot(X[:, 0], y, data_style, label=data_label)
    plt.plot(x1, y_pred, style, linewidth=2, label=label)
    if label or data_label:
        plt.legend(loc="upper center", fontsize=16)
    plt.axis(axes)

plt.figure(figsize=(11,11))

plt.subplot(321)
plot_predictions([tree_reg1], X, y, axes=[-0.5, 0.5, -0.1, 0.8],
                 label="$h_1(x_1)$", style="g-", data_label="Training set")
plt.ylabel("$y$", fontsize=16, rotation=0)
plt.title("Residuals and tree predictions", fontsize=16)

plt.subplot(322)
plot_predictions([tree_reg1], X, y, axes=[-0.5, 0.5, -0.1, 0.8],
                 label="$h(x_1) = h_1(x_1)$", data_label="Training set")
plt.ylabel("$y$", fontsize=16, rotation=0)
plt.title("Ensemble predictions", fontsize=16)

plt.subplot(323)
plot_predictions([tree_reg2], X, y2, axes=[-0.5, 0.5, -0.5, 0.5],
                 label="$h_2(x_1)$", style="g-", data_style="k+", data_label="Residuals")
plt.ylabel("$y - h_1(x_1)$", fontsize=16)

plt.subplot(324)
plot_predictions([tree_reg1, tree_reg2], X, y, axes=[-0.5, 0.5, -0.1, 0.8],
                 label="$h(x_1) = h_1(x_1) + h_2(x_1)$")
plt.ylabel("$y$", fontsize=16, rotation=0)

plt.subplot(325)
plot_predictions([tree_reg3], X, y3, axes=[-0.5, 0.5, -0.5, 0.5],
                 label="$h_3(x_1)$", style="g-", data_style="k+")
plt.ylabel("$y - h_1(x_1) - h_2(x_1)$", fontsize=16)
plt.xlabel("$x_1$", fontsize=16)

plt.subplot(326)
plot_predictions([tree_reg1, tree_reg2, tree_reg3], X, y, axes=[-0.5, 0.5, -0.1, 0.8],
                 label="$h(x_1) = h_1(x_1) + h_2(x_1) + h_3(x_1)$")
plt.xlabel("$x_1$", fontsize=16)
plt.ylabel("$y$", fontsize=16, rotation=0)

plt.show()
```

A simpler way to train ensembles of *Gradient Boosted Regression Trees* is to use [`sklearn's GradientBoostingRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html) implementation. In addition to the typical hyperparameters that control the growth of the trees, it also includes hyperparameters that control the ensemble training, e.g. `n_estimators`:

```python
from sklearn.ensemble import GradientBoostingRegressor

gbrt = GradientBoostingRegressor(max_depth=2, n_estimators=3, learning_rate=1.0, random_state=42)
gbrt.fit(X, y)
```

    GradientBoostingRegressor(learning_rate=1.0, max_depth=2, n_estimators=3,
                              random_state=42)

**Question**: Another hyperparameter is `learning_rate`. What is it responsible for and what effect on the resulting ensemble does it have?

Feel free to check `sklearn's` documentation, as well as look into the comparison of two ensembles: the one with a lower number of estimators and a higher learning rate, and a 'slow learner' with a lower learning rate but a higher number of estimators:

```python
gbrt_slow = GradientBoostingRegressor(max_depth=2, n_estimators=200, learning_rate=0.1, random_state=42)
gbrt_slow.fit(X, y)
```

    GradientBoostingRegressor(max_depth=2, n_estimators=200, random_state=42)

```python
plt.figure(figsize=(11,4))

plt.subplot(121)
plot_predictions([gbrt], X, y, axes=[-0.5, 0.5, -0.1, 0.8], label="Ensemble predictions")
plt.title("learning_rate={}, n_estimators={}".format(gbrt.learning_rate, gbrt.n_estimators), fontsize=14)

plt.subplot(122)
plot_predictions([gbrt_slow], X, y, axes=[-0.5, 0.5, -0.1, 0.8])
plt.title("learning_rate={}, n_estimators={}".format(gbrt_slow.learning_rate, gbrt_slow.n_estimators), fontsize=14)

plt.show()
```

**Question**: What generalisation behaviour can you expect from each of these ensembles?

### Gradient Boosting with Early Stopping

The examples above demonstrate ensembles with a different number of estimators. Which one is better? How can you quantitatively estimate the optimal number of predictors?

One technique that you can use is called *early stopping* – it allows your algorithm to stop as soon as the validation error reaches a minimum. An easy way to implement this with `sklearn` is to use `staged_predict()` method as the code below demonstrates:

```python
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

X_train, X_val, y_train, y_val = train_test_split(X, y, random_state=100)

gbrt = GradientBoostingRegressor(max_depth=2, n_estimators=120, random_state=42)
gbrt.fit(X_train, y_train)

errors = [mean_squared_error(y_val, y_pred)
          for y_pred in gbrt.staged_predict(X_val)]
bst_n_estimators = np.argmin(errors)

gbrt_best = GradientBoostingRegressor(max_depth=2,n_estimators=bst_n_estimators, random_state=42)
gbrt_best.fit(X_train, y_train)
```

    GradientBoostingRegressor(max_depth=2, n_estimators=68, random_state=42)

Let's visualise the validaton errors and the optimal number of estimators:

```python
min_error = np.min(errors)

plt.figure(figsize=(11, 4))

plt.subplot(121)
plt.plot(errors, "b.-")
plt.plot([bst_n_estimators, bst_n_estimators], [0, min_error], "k--")
plt.plot([0, 120], [min_error, min_error], "k--")
plt.plot(bst_n_estimators, min_error, "ko")
plt.text(bst_n_estimators, min_error*1.2, "Minimum", ha="center", fontsize=14)
plt.axis([0, 120, 0, 0.01])
plt.xlabel("Number of trees")
plt.title("Validation error", fontsize=14)

plt.subplot(122)
plot_predictions([gbrt_best], X, y, axes=[-0.5, 0.5, -0.1, 0.8])
plt.title("Best model (%d trees)" % bst_n_estimators, fontsize=14)

plt.show()
```

Finally, an alternative to training a large number of estimators and then looking back in order to find the optimal number of those is to allow the algorithm to stop training when the validation error does not improve over a number of consecutive iterations (e.g., $5$ in the code below). For that to work, set the `warm_start` to `True`:

```python
gbrt = GradientBoostingRegressor(max_depth=2, warm_start=True, random_state=42)

min_val_error = float("inf")
error_going_up = 0
for n_estimators in range(1, 120):
    gbrt.n_estimators = n_estimators
    gbrt.fit(X_train, y_train)
    y_pred = gbrt.predict(X_val)
    val_error = mean_squared_error(y_val, y_pred)
    if val_error < min_val_error:
        min_val_error = val_error
        error_going_up = 0
    else:
        error_going_up += 1
        if error_going_up == 5:
            break  # early stopping

print(gbrt.n_estimators)
```

    74

# Your tasks

1. Run the code in the notebook. During the practical session, be prepared to discuss the methods and answer the questions from this notebook:

    * When applied to a new, test instance bagging classifier aggregates the predictions of all predictors and estimates the statistical mode. Is this similar to the hard or the soft voting strategy?
    * [`BaggingClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingClassifier.html) automatically performs soft voting when the base classifier can estimate class probabilities (i.e., has a `predict_proba()` method). Which strategy – hard or soft voting – does `BaggingClassifier` follow with the `DecisionTreeClassifier` as the base classifier?
    * What do the feature importance weights obtained on the *iris* dataset suggest? Are they similar to your observations from the previous experiments on this dataset?
    * We said earlier than one of the advantages of bagging and pasting strategies is that predictors can all be trained in parallel. Is the same applicable to AdaBoost?
    * One of the hyperparameters of the boosting algorithms is learning_rate. What is it responsible for and what effect on the resulting ensemble does it have?
    * What generalisation behaviour can you expect from each of the tree regressor ensembles (using different learning rates and different number of estimators – see above)?

2. Apply ensemble techniques of your choice to one of the datasets you've worked on during the previous practicals and report your findings.

```python
from sklearn import datasets
digits = datasets.load_digits()
```

```python
X, y = digits["data"], digits["target"]
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
```

```python
# Best model in Practical 2 - perceptron
from sklearn.linear_model import SGDClassifier

sgd = SGDClassifier(max_iter=5, tol=None, random_state=42,
                   loss="perceptron", eta0=1, learning_rate="constant", penalty=None)
sgd.fit(X_train, y_train)
y_pred = sgd.predict(X_test)
print(accuracy_score(y_test, y_pred))
```

    0.9311111111111111

```python
# Hard voting
log_clf = LogisticRegression(random_state=42, max_iter=5000) # Number of iterations is somewhere between 3000-4000
rnd_clf = RandomForestClassifier(random_state=42)
svm_clf = SVC(random_state=42)

voting_clf = VotingClassifier(
    estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
    voting='hard')

for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    print(clf.__class__.__name__, accuracy_score(y_test, y_pred))

preds = {'lr': get_predictions(log_clf),
        'rf': get_predictions(rnd_clf),
        'svc': get_predictions(svm_clf)}
df = pd.DataFrame(data=preds)
df.corr()
```

    LogisticRegression 0.9733333333333334
    RandomForestClassifier 0.9711111111111111
    SVC 0.9866666666666667
    VotingClassifier 0.98

      lr
      rf
      svc

      lr
      1.000000
      0.971966
      0.982963

      rf
      0.971966
      1.000000
      0.974460

      svc
      0.982963
      0.974460
      1.000000

```python
# Soft voting
log_clf = LogisticRegression(random_state=42, max_iter=5000)
rnd_clf = RandomForestClassifier(random_state=42)
svm_clf = SVC(probability=True, random_state=42)

voting_clf = VotingClassifier(
    estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],
    voting='soft')

for clf in (log_clf, rnd_clf, svm_clf, voting_clf):
    clf.fit(X_train, y_train)
    y_pred = clf.predict(X_test)
    print(clf.__class__.__name__, accuracy_score(y_test, y_pred))

preds = {'lr': get_predictions(log_clf),
        'rf': get_predictions(rnd_clf),
        'svc': get_predictions(svm_clf)}
df = pd.DataFrame(data=preds)
df.corr()
```

    LogisticRegression 0.9733333333333334
    RandomForestClassifier 0.9711111111111111
    SVC 0.9866666666666667
    VotingClassifier 0.9822222222222222

      lr
      rf
      svc

      lr
      1.000000
      0.971966
      0.982963

      rf
      0.971966
      1.000000
      0.974460

      svc
      0.982963
      0.974460
      1.000000

```python
# Single decision tree
tree_clf = DecisionTreeClassifier(random_state=42)
tree_clf.fit(X_train, y_train)
y_pred_tree = tree_clf.predict(X_test)
print(accuracy_score(y_test, y_pred_tree))
```

    0.8577777777777778

```python
# Bagging with 500 decision trees
bag_clf = BaggingClassifier(
    DecisionTreeClassifier(random_state=42), n_estimators=500,
    max_samples=100, bootstrap=True, n_jobs=-1, random_state=42)
bag_clf.fit(X_train, y_train)
y_pred = bag_clf.predict(X_test)
print(accuracy_score(y_test, y_pred))
```

    0.9377777777777778

```python
# Bagging with 500 decision trees - out-of-bag evaluation
bag_clf = BaggingClassifier(
    DecisionTreeClassifier(random_state=42), n_estimators=500,
    bootstrap=True, n_jobs=-1, oob_score=True, random_state=40)
bag_clf.fit(X_train, y_train)
bag_clf.oob_score_
```

    0.9502598366740905

```python
# Pasting with 500 decision trees
bag_clf = BaggingClassifier(
    DecisionTreeClassifier(random_state=42), n_estimators=500,
    max_samples=100, bootstrap=False, n_jobs=-1, random_state=42)
bag_clf.fit(X_train, y_train)
y_pred = bag_clf.predict(X_test)
print(accuracy_score(y_test, y_pred))
```

    0.9422222222222222

```python
# Bagging with decision trees vs. random forests
bag_clf = BaggingClassifier(
    DecisionTreeClassifier(splitter="random", max_leaf_nodes=16, random_state=42),
    n_estimators=500, max_samples=1.0, bootstrap=True, n_jobs=-1, random_state=42)

bag_clf.fit(X_train, y_train)
y_pred = bag_clf.predict(X_test)

rnd_clf = RandomForestClassifier(n_estimators=500, max_leaf_nodes=16,
                                 n_jobs=-1, random_state=42)
rnd_clf.fit(X_train, y_train)
y_pred_rf = rnd_clf.predict(X_test)

np.sum(y_pred == y_pred_rf) / len(y_pred)  # see to what extent predictions are identical
```

    0.9377777777777778

```python
# AdaBoost with early stopping
ada_clf = AdaBoostClassifier(
    DecisionTreeClassifier(max_depth=9), n_estimators=300,
    algorithm="SAMME.R", learning_rate=0.5, random_state=42)
ada_clf.fit(X_train, y_train)

y_pred = ada_clf.predict(X_test)
print(accuracy_score(y_test, y_pred))
```

    0.98

3. **Optional**: If you want more practice with these techniques, try implementing a [*stacking algorithm*](https://en.wikipedia.org/wiki/Ensemble_learning#Stacking). There is no available implementation of this approach in `sklearn` so it needs to be implemented from scratch. The idea is as follows:
    * Split a dataset of your choice into three subsets – training, validation and test.
    * Train a number of predictors on the training data and apply them to the validation data.
    * Treat the predictions of those predictors on the validation data to generate new training set: you can use the predictions of the estimators as new features (i.e., each training instance will have as many features as the number of predictors you originally used), and the validation data targets as the new training instances targets.
    * Now train a *blender* – a classifier of your choice – on the new training data created this way. Together with the original classifiers, the blender forms a stacking ensemble.
    * Finally, apply your original classifiers to the test set, feed their predictions to the blender, and use blender's output as the prediction on the test set.

```python

```

Practical 3 - Ensembles/DSPNP_practical3.ipynb


# Experiment Design Lecture 2A: Response Surface Methodology

Link: [http://charlesreid1.com/wiki/Response_Surface_Methodology](http://charlesreid1.com/wiki/Response_Surface_Methodology)

## Overview

A brief overview of response surface methodology (RSM) was given in [Lecture 1](ED_1_Introduction.ipynb), so let's review that again.

RSM is a set of methods for fitting a polynomial to a multi-input, multi-output function,

$$
Y = f(X)
$$

using a surrogate function to approximate:

$$
\hat{Y} = g(X)
$$

### Why Polynomials?

We learn in introdutory calculus that Maclaurin and Taylor series expansions are capable of representing any suitably well-behaved function in a suitably small neighborhood by expressing it in terms of an infinite sum of powers of x with coefficients.

Often, complicated functions are simplified (e.g., for integration or evaluation) by writing a Taylor series expansion, which is just a polynomial, and truncating all but a finite (hopefully few) terms.

### Why Not Polynomials?

Some care is required to select a system response vector $Y$ that lends itself well to being modeled with a polynomial. Rapidly-changing or overly-sensitive quantities can make $f(X)$ simply too complicated to model using a polynomial.

Another risk is in extrapolating outside of the experimental region: the fitted polynomial only describes responses in a smooth manner over the region where data were obtained.

Outside of this region, the polynomial values can quickly become spurious or non-physical.

Let's return to the Taylor series for $\sin{x}$ from the first lecture.

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import sympy
from sympy.functions import sin as sympy_sin

from pprint import pprint
from scipy.special import factorial
```

```python
def f(x):
    """Real function"""
    return np.sin(x)

def g(x):
    """Surrogate function from Taylor series about x=0,
    sin(x) ~ x - 1/3! x^3 + 1/5! x^5
    """
    x0 = 0
    dx = x - x0
    dim = np.shape(dx)[0]
    result = np.zeros(dim,)
    for i in range(dim):
        dxi = dx[i]
        val = dxi
        val -= (1/factorial(3))*pow(dxi,3)
        val += (1/factorial(5))*pow(dxi,5)
        result[i] = val

    return result
```

```python
# Taylor approximation at x0 of function f
def taylor(f,x,x0,n):
    """Function f should be a sympy function.
    """
    i = 0
    p = 0
    while i <= n:
        p = p + (f.diff(x,i).subs(x,x0))/(factorial(i))*(x-x0)**i
        i += 1
    return p
```

```python
def g_alt(x, n=5):
    """Taylor series
    """
    xv = sympy.Symbol('x')
    f = sympy_sin(xv)

    x0 = 0
    dim = np.shape(x)[0]

    r = taylor(f,xv,x0,n)

    result = np.zeros(dim,)

    for i in range(dim):
        result[i] = r.evalf(n=15, subs={xv: x[i]})

    return result
```

```python
the_x = np.linspace(-np.pi,np.pi,100)
big_x = np.linspace(-2*np.pi,2*np.pi,200)

fig, (ax1, ax2) = plt.subplots(1,2,figsize=(12,4))

for x, ax in zip([the_x,big_x], [ax1,ax2]):
    ax.plot(x, f(x), label='f(x) real')
    ax.plot(x, g(x), label='g(x) surr')
    ax.plot(x, g_alt(x), label='g(x) surr (sympy)')

    ax.set_xlabel(r'$x$')
    ax.set_ylabel(r'$y, \hat{y}$')
    ax.set_xlim([min(x), max(x)])
    ax.set_title('Real vs. Surrogate Function')

ax1.legend()
plt.show()
```

This surrogate function is only applicable in the neighbohood of $x=0$, so the further we get, the worse the approximation gets.

### Other Types of Linear Models

Are there other ways of using linear models besides RSM? Indeed there are. This is testament to the fact that linear models are surprisingly powerful, given their simplicity.

Here is a partial list of techniques that build linear models of sytem resopnses:

**Generalized Linear Models (GLM)** - These linear models can account for an arbitrary number of inputs and outputs. GLM models postulate that the error in the model representation $y - \hat{y}$ is Gaussian normal. Statistical techniques are then used to construct a linear model and associated error estimates.

**Multiple Linear Regression** - Linear regression technique used to construct models for a *single* response variable $y$ and multiple predictor variables $X$.

**Multivariate Linear Regression** - Extending multiple linear regression to the case of a multivariate response vector $Y$ is called multivariate linear regression.

**Polynomial Models (Univariate)** - In the simple case of a univariate input and univariate output, the system function $y = f(x)$ can be fit using a simple polynomial fitting procedure.

**Response Surface Models (Multivariate)** - We will cover this in more detail below.

# Strategy for Comprehensive Regression Analysis

Selecting a strategy for comprehensive regression analysis requires some care and consideration of the specifics of your system.

A proposal following the validation strategy in the NISS paper

Steps 1-3 provide information going into the surrogate model construction process

* Determine important variables (prior steps)
* Specify a model form
* Design experiments to determine function samples
* Define selection criteria and comparison metrics
* Calculate comparison metrics, plot data
* Assess if the model is good enough, assess model assumptions, select if satisfied (otherwise proceed to next step)
* Specify a new model form (that can incorporate already-gathered information!) and repeat until criteria met

# Selecting a Model

In order to select the model that is best for the indented use, two things must be done:

First, figure out what is wanted out of the model (the selection criteria).

Second, figure out how to select the model that is the best for that criteria (the comparison metrics).

Part of the difficulty in defining goals and selection criteria is that multivariate surfaces are very difficult to visualize in higher than 2 dimensions. Various selection criteria, i.e. numerical quantities related to error, curvature, best fit, etc., should be used to determine which surface is the best for the intended use.

## Pre-Selection Step: Experimental Design

Selecting the form of the surrogate model means selecting the experimental design you plan to use. The experimental design will give a set of experiments (inputs) to run to best regress a black box function to some particular functional form (e.g., a polynomial). Assuming a particular funtional form allows the number of experiments/samples to be minimized.

If a Monte Carlo simulation is being run, the cost is higher (thousands or millions more function samples are needed), but the method is very flexible - any model may be selected and its parameters fit to the data.

For more information on the experimental design step, see the Experimental Design Lecture.

## Selection Criteria

The most obvious criteria is minimization of the model's representation error,

$$
\min_{X} Y(X) - \hat{Y}(X)
$$

where $X$ is the input parameter vector, $Y$ is the system response vector, and $\hat{Y}$ is the surrogate model representation of the real response vector.

What experimental design is trying to accomplish for simulations and experiments is similar:

* Experiments are trying to create a simple and cheap model that can adequately represent the relevant features of a relevant physical system
* Simulations are trying to create a simple and cheap model that can adequately represent the result of a much more expensive and complicated model

However, the end use is often different:

* Simulations are trying to determine values of input parameters that lead to the best match between models and experiments
* Experiments are trying to create models that can be used to optimize or control a process, or find minimum/maximum points

### Least Squares for Linear Regression

Linear regression models a response $Y$ as a linear function of $N$ input variables $X$:

$$
Y_i = \beta_0 + \sum_{i=0}^{N} \beta_i + \epsilon_i
$$

Using least squares for a linear regression model approximates the coefficients of the linear model by minimizing the sum of the squared error residuals (SSE),

$$
\mbox{SS}_{E} = \sum r_i^2 = \sum \left( y_i - \hat{y} \right)^2
$$

The estimated coefficients that meet this criteria are then called least squares coefficient estimates.

To do this, the SSE equation above is differentiated with respect to each of the parameters $b_0, b_1, b_2, \dots$

All these derivatives are set equal to 0, and these equations are solved simultaneously. In practice, this is done using matrix algebra, which is typically wrapped with a high-level API in a computational implementation.

### Linear Regression with Scikit-Learn

We can use `scikit-learn` to do basic linear regression:

```python
from sklearn import linear_model
```

```python
def noisy_line(x):
    m = 4
    b = 1
    return m*x + b + 2*np.random.randn(*np.shape(x))

def noisy_line_model(x,y):
    lr = linear_model.LinearRegression()
    lr.fit(x,y)
    return lr

def noisy_line_fit(x,y):
    lr = noisy_line_model(x,y)
    return lr.predict(x)
```

```python
N = 30
x = np.linspace(0,5,N).reshape(-1,1)
y = noisy_line(x)
yhat = noisy_line_fit(x,y)

#print(yhat.shape)
fig, ax = plt.subplots(1,1,figsize=(4,4))

ax.plot(x,yhat,'g:',label='fit')
ax.plot(x,y,'go',label='data')

ax.legend()
plt.show()
```

```python
#print(help(linear_model.LinearRegression))
```

```python
m = noisy_line_model(x,y)

print("\nR^2 = %0.4f"%(m.score(x,y)))
print("\nCoefficient vector = %s"%(m.coef_))
```

### Interpretation

Cannot necessarily interpret approximated coefficients as "amount of change in $\hat{y}$ for a unit change in $x_j$"... This assumes that the coefficients are completely independent

In order to determine how good this assumption is, regress the input/predictor variable $x_j$ on all other variables (i.e. find $x_j$ as a function of all other input/predictor variables + constant)

The residuals from this fit are $r^{\star} = x_{ij} - \hat{x}_{ij}$

The coefficient estimate $b_j$ measures the change in the response due to a unit change in $r^{\star}$, not in $x_{ij}$.

If $x_j$ can't be predicted by other variables, then $r^{\star} \approx x_{ij} - \overline{x}_{ij}$ (where overline = average value)

In this case, $b_j$ can be interpreted in the way specified: i.e. measure of change in $\hat{y}$ for unit change in $x_j$.

### Significance

Coefficients cannot be used by themselves to determine relative significance of various terms in the linear model

To actually do this, you need to use normalized/weighted coefficients $\hat{\beta}_j$

Defined as:

$$
\hat{\beta}_j = b_j \left( \frac{s_{jj}}{s_{yy}} \right)^2
$$

where

$$
s_{jj} = \dfrac{ \sum (x_{ij} - \overline{x}_j)^2 }{ (n-1) } \\
s_{yy} = \dfrac{ \sum (y_i - \overline{y}_i)^2 }{ (n-1) }
$$

The overline indicates a sample mean.


## Introduction

```python
import numpy as np
import matplotlib.pyplot as plt
import pydicom
from pydicom.data import get_testdata_files

# Dataset
#A = np.array([[1, 0, 1, 0], [0, 1, 0, 0],[0, 0, 0, 0], [0,0,0,0]], dtype=float)
#B = np.array([[0, 0, 0, 0], [0, 0, 0, 2],[0, 0, 2, 0], [0,0,0,2]], dtype=float)

A = np.array([[1, 1, 0, 0], [0, 0, 0, 0],[0, 0, 0, 0], [0,0,0,0]], dtype=float)
B = np.array([[0, 0, 0, 0], [0, 0, 2, 0],[0, 0, 2, 0], [0,0,0,0]], dtype=float)

#filename = get_testdata_files("MR_small.dcm")[0]
#dataset = pydicom.read_file(filename)
#A = np.array(dataset.pixel_array, copy=True)
#rollx = 10
#rolly = 0
#B = np.copy(A)
#B = np.roll(B, rollx, axis=0)
#B = np.roll(B, rolly, axis=1)

# Plot images
ax = plt.subplot(2,2,1)
ax.imshow(A)

ax = plt.subplot(2,2,2)
ax.imshow(B)
```

# Optimization model
## Cross correlation
Normalized cross correlation ($NCC$). If $A$ is the reference image and $B$ is input image, the object of image registration is to bring the input image into alignment by applying a spartial transformation to the input image,

\begin{equation}
T^* = \underset{T} {\text{arg max}} S\{A,T[B]\}
\end{equation}

$S$ represents the measurement of similarity, $T$ represents the transformation matrix, The aim is to find $S$ similiarity between the images.

\begin{equation}
T =
\begin{bmatrix}
   \cos{\theta} & -\sin{\theta} & t_x \\
   \sin{\theta} & \cos{\theta} & t_y \\
   0 & 0 & 1
   \end{bmatrix}
\end{equation}

By using $NCC$ similarity and rigid transformation model (which consists of two translation parameters $t_x$ and $t_y$ and one rotation parameter $\theta$) the $NCC$ will look like

\begin{equation}
NCC(A,B) = \dfrac{1}{N} \sum_{(x,y)}{ \dfrac{(A-\mu_A)(B-\mu_B)}{\sigma_A \sigma_B}}
\end{equation}

where $N$ denotes the total number of pixels in image $A$ and $B$, $\mu_A$ and $\mu_B$ denote the average of the images, and $\sigma_A$ and $\sigma_B$ denote the standard deviation. Therefore, the model can be rewritten as

\begin{equation}
T^* = \underset{t_x, t_y, \theta} {\text{arg max}} NCC(A,T(B)) =
 \dfrac{1}{N} \underset{t_x, t_y, \theta} {\text{arg max}} \sum_{(x,y)}{ \dfrac{(A-\mu_A)(T(B)-\mu_{T(B)})}{\sigma_A \sigma_{T(B)}}}
\end{equation}

```python
def NCC_sum(A, B):
    assert(A.shape == B.shape)
    muA = np.mean(A)
    muB = np.mean(B)
    sigmaA = np.std(A)
    sigmaB = np.std(B)
    #A_vec = np.asarray(A).reshape(-1)
    #B_vec = np.asarray(B).reshape(-1)
    N =  np.size(A)
    return 1/N * np.sum((A-muA)*(B-muB)/(sigmaA*sigmaB))
    #ncc_1 = 1/N * (A_vec-muA).dot((B_vec-muB).T) / (sigmaA * sigmaB)
    #ncc = 0
    #for i in range(N):
    #    ncc = ncc + NCC(A_vec[i], B_vec[i], muA, muB, sigmaA, sigmaB)
    #return ncc / N

def NCC_mat(A, B):
    muA = np.mean(A)
    muB = np.mean(B)
    sigmaA = np.std(A)
    sigmaB = np.std(B)
    return (A-muA)*(B-muB)/(sigmaA*sigmaB)
#def NCC(a, b, muA, muB, sigmaA, sigmaB):
#    return (a-muA)*(b-muB)/(sigmaA * sigmaB)

```

## Spline pyramids

```python

```

## Optimization algorithm

### Firefly algorithm

Nature inspired algorithm based on flashing behaviors of firefly swarm. The primary purpose for the flash is to signal tha attract other fireflies.

Three rules:
1. All fireflies are unisex so that on firefly will be attacted to other fireflies regarding to their sex
2. An important and interesting behavior fo fireflies is to flow brigher mainly to attract pray an the share food with others.
3. Attractiveness is proportinal to their brigthness, thus agent firstly moves toward a neightbor that glows brighter.

The fireflies are randomly distributed in the search space. The fireflies carry a luminescence quality, called liciferin, which emits lighter propotional to quality. Each firefly is attracted by lighter glow of other approximated fireflies. The attractiveness decreases as their distance increases. If there is no brighter one within the scope of a firefly, it will move randomly in the search space. In this application the decision variables are the three spatial tranform parameters $t_x$, $t_y$ and $\theta$. The brightness is associated as the objective function which maximum the $NCC$. The procedure is typed into two phases, variation of light intensit and movement towards attractive flies.

#### Variation of light intensity

\begin{equation}
I_i = f(x_i), 1\leq i \leq n
\end{equation}

#### Movement towards attractive flies

\begin{equation}
\beta_r(i,j) = \beta_0 e^{-\gamma r(i.j)^2}
\end{equation}

\begin{equation}
r(i,j) = \lVert x_i - x_j \rVert
\end{equation}

\begin{equation}
x_i(t+1) = x(t) + \beta_0 e^{-\gamma r^2} (x_j - x_i) + \alpha \epsilon
\end{equation}

Here, $\alpha$ is the parameter controlling the step, and $\epsilon$ is a vector drawn from Gaussian distribution. The classical $FA$ chose $\beta_0 = 1$ (Yang's Comments) and $\alpha$ generated with uniform distribution in range $[0, 1]$

```python
import random

# Initialize
## Create the initial population of n fireflies within d-dimensional search space
X = np.asarray(np.copy(B)).reshape(-1)
d = len(X)
n = int(round(d*1))
A_vec = np.asarray(np.copy(A)).reshape(-1)

## Formulate light intensity of each firefly so as to be associated with the fitness value f(x)
I = NCC_mat(A_vec,X)
I_max = NCC_mat(A_vec, A_vec)
#print(X)
#print(A_vec)
## Define the parameters B_0 and gamma
B0 = 1.0
#r = 0.0
gamma = 0.0

# Perform
criteria = 1
success = False

def UpdateX(xi, xj, B0, r):
    alpha = random.uniform(0,1)
    #epsilon = 0.001
    epsilon = 0
    return xi + B0*np.exp(-gamma*r**2)*(xj-xi)+alpha*epsilon
print(X)
#while success:
for e in range(100):
    for i in range(n): # TODO: This is not only rows
        Ii = I[i]
        for j in range(n): # TODO: This is not only columns
            tempXi = np.copy(X)
            Ij = I[j]
            if Ij < Ii:
                r = np.sqrt(np.sum(X[i] - X[j])**2)
                xi1 = UpdateX(X[i], X[j], B0, r)
                #print("Updates. x_i: {0}, x_i1: {1}, x_j:{2}, a_i: {3}, lj: {4}, li: {5}".format(X[i], xi1, X[j], A_vec[i], Ij, Ii))
                X[i] = xi1
            I = NCC_mat(A_vec,X)
            #print(I)
    #Rank fireflies and find the current best
print(X)
print(A_vec)
newB = np.asarray(X).reshape(A.shape)
# Plot images
ax = plt.subplot(2,3,1)
ax.imshow(A)
ax = plt.subplot(2,3,2)
ax.imshow(B)
ax = plt.subplot(2,3,3)
ax.imshow(newB)

print(NCC_sum(A, newB))
print(NCC_sum(A, A))

```

```python
#Inialize: Identity transformation
a = 1
b = 0
tx = 0
ty = 0

# Repeat

for e in range(100):

# Compute gradient
# Make step in gradient direction
# Update mapping equation
# Remap image
# Until convergence

h = 1
alpha_grad = 1e-5
alpha = 1

I_max = NCC_sum(A, A)

for e in range(100):
    print("Values: tx={0}, ty={1}, theta={2}".format(tx, ty, theta))
    I = NCC_sum(A, DoTransformation(B, tx, ty, theta))
    tx_grad = (NCC_sum(A, DoTransformation(B, tx + h, ty, theta)) - NCC_sum(A, DoTransformation(B, tx - h, ty, theta))) / (2 * h)
    ty_grad = (NCC_sum(A, DoTransformation(B, tx, ty + h, theta)) - NCC_sum(A, DoTransformation(B, tx, ty-h, theta))) / (2 * h)
    theta_grad = (NCC_sum(A, DoTransformation(B, tx, ty, theta + h)) - NCC_sum(A, DoTransformation(B, tx, ty, theta - h))) / (2 * h)

    print("Gradients: tx={0}, ty={1}, theta={2}".format(tx_grad, ty_grad, theta_grad))
    tx = tx - alpha * tx_grad
    ty = ty - alpha * ty_grad
    theta = theta - alpha_grad * theta_grad
    print("Episode: {0}. Error: {1}. Tx: {2}".format(e, I_max - I, tx))
    print(tx_grad)

print(theta)
print(tx)
print(ty)
```

```python

```

```python
theta = 0
tx = 1
ty = 0
T = np.array([[np.cos(theta), -np.sin(theta), tx], [np.sin(theta), np.cos(theta), ty], [0, 0, 1]])

print(B)
C = DoTransformation(B, tx, ty, theta)
print(C)
```

    [[0. 0. 0. 0.]
     [0. 0. 2. 0.]
     [0. 0. 2. 0.]
     [0. 0. 0. 0.]]
    [[0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 2. 0.]
     [0. 0. 2. 0.]]

```python
def DoTransformation(B, tx, ty, theta):
    result = np.zeros(B.shape)
    for i in range(B.shape[0]):
        for j in range(B.shape[1]):
            value = B[i,j]
            #TODO Interpolate
            positionX = int(np.cos(theta)*i - np.sin(theta)*j + tx)
            positionY = int(np.sin(theta)*i + np.cos(theta)*j + ty)

            #TODO padding
            if positionX > result.shape[0] - 1 or positionX < 0:
                continue
            if positionY > result.shape[1] - 1 or positionY < 0:
                continue
            #print("X: {0} -> {1}, Y: {2} -> {3}, value: {4}".format(i, positionX, j, positionY, value))
            result[positionX,positionY] = value
    return result

```

```python

```

```python
x = 2
y = 3
positionX = int(np.cos(0)*x - np.sin(0)*y + -1)
positionY = int(np.sin(0)*x + np.cos(0)*y + 1)
if positionY > result.shape[1] - 1 or positionY < 0:
    print("Hej")
print(positionY)
```

    Hej
    4

```python
print(B[0,:])
print(B[1,:])
print(B[2,:])
print(B[3,:])
print(B)

```

    [0. 0. 0. 0.]
    [0. 0. 0. 2.]
    [0. 0. 2. 0.]
    [0. 0. 0. 2.]
    [[0. 0. 0. 0.]
     [0. 0. 0. 2.]
     [0. 0. 2. 0.]
     [0. 0. 0. 2.]]

Image registration using cross-correlation.ipynb