Copyright (c) 2015, 2016 [Sebastian Raschka](sebastianraschka.com)

2016 [Li-Yi Wei](http://liyiwei.org/)

https://github.com/1iyiwei/pyml

[MIT License](https://github.com/1iyiwei/pyml/blob/master/LICENSE.txt)

# Python Machine Learning - Code Examples

# Chapter 7 - Combining Different Models for Ensemble Learning

Note that the optional watermark extension is a small IPython notebook plugin that I developed to make the code reproducible. You can just skip the following line(s).

```python
%load_ext watermark
%watermark -a '' -u -d -v -p numpy,pandas,matplotlib,scipy,sklearn
```

    last updated: 2016-11-01

    CPython 3.5.2
    IPython 4.2.0

    numpy 1.11.1
    pandas 0.18.1
    matplotlib 1.5.1
    scipy 0.17.1
    sklearn 0.18

*The use of `watermark` is optional. You can install this IPython extension via "`pip install watermark`". For more information, please see: https://github.com/rasbt/watermark.*

### Overview

- [Learning with ensembles](#Learning-with-ensembles)
- [Implementing a simple majority vote classifier](#Implementing-a-simple-majority-vote-classifier)
  - [Combining different algorithms for classification with majority vote](#Combining-different-algorithms-for-classification-with-majority-vote)
- [Evaluating and tuning the ensemble classifier](#Evaluating-and-tuning-the-ensemble-classifier)
- [Bagging – building an ensemble of classifiers from bootstrap samples](#Bagging----Building-an-ensemble-of-classifiers-from-bootstrap-samples)
- [Leveraging weak learners via adaptive boosting](#Leveraging-weak-learners-via-adaptive-boosting)
- [Summary](#Summary)

```python
from IPython.display import Image
%matplotlib inline
```

```python
# Added version check for recent scikit-learn 0.18 checks
from distutils.version import LooseVersion as Version
from sklearn import __version__ as sklearn_version
```

# Learning with ensembles

Challenge:
* No single classifier best for all circumstances
* Hard to manually decide which classifier/model to use

Solution:
* Combine multipler classifiers for better performance than each individual classifier
* Need diverse, not just accurate, individual classifiers

#### Unanimity
* no voting necessary

#### Majority voting
* for binary classification

For sample $x$, each classifier $j$ decides whether the class $C_j(x)$ is $+1$ or $-1$

Majority voting is then:
$$
C(x) = sign\left[ \sum_j C_j(x) \right] =
\begin{cases}
+1 \; if \; \sum_j C_j(x) \geq 0\\
-1 \; else
\end{cases}
$$

#### Plurality voting
* extension of majority voting for multi-class setting

$$
C(x) = mode\{C_1(x), C_2(x), \cdots, C_m(x) \}
$$

Individual classifiers can be the same or different types
* decision tree, svm, logistic regression, etc.

Random forest
* combines multiple decision trees

# Why ensemble can better than individual classifiers

A simple math model:
* binary classification
* $n$ base classifiers
* each with error rate $\epsilon$
* classifiers make independent decisions

Probability that at least $K$ classifiers are wrong:
$$
\begin{align}
\epsilon_{ensemble} (K)
&=
\sum_{k=K}^n C\left(n, k\right) \epsilon^k \left(1-\epsilon\right)^{n-k}
\end{align}
$$
, where $C\left(n, k\right)$ is the combinatorial, i.e. binomial coefficient of n choosing k.

For the ensemble to be wrong, $K \geq \frac{n}{2}$:
$$
\epsilon_{ensemble} = \epsilon_{ensemble}\left( \left\lceil \frac{n}{2} \right\rceil \right)
$$

For example, if
* $n = 11$
* $\epsilon = 0.25$

$\epsilon_{ensemble} = 0.034$

The above assumes classifiers are making independent decisions!

Humans don't always make independent decisions
* elections
* financial markets

```python
# combinatorial C(n, k)
from scipy.misc import comb
import math

def ensemble_error(n_classifier, error):
    k_start = math.ceil(n_classifier / 2.0)
    probs = [comb(n_classifier, k) * error**k * (1-error)**(n_classifier - k)
             for k in range(k_start, n_classifier + 1)]
    return sum(probs)
```

**Note**

For historical reasons, Python 2.7's `math.ceil` returns a `float` instead of an integer like in Python 3.x. Although Although this book was written for Python >3.4, let's make it compatible to Python 2.7 by casting it to an it explicitely:

```python
from scipy.misc import comb
import math

def ensemble_error(n_classifier, error):
    k_start = int(math.ceil(n_classifier / 2.0))
    probs = [comb(n_classifier, k) * error**k * (1-error)**(n_classifier - k)
             for k in range(k_start, n_classifier + 1)]
    return sum(probs)
```

```python
ensemble_error(n_classifier=11, error=0.25)
```

    0.034327507019042969

```python
import numpy as np

error_range = np.arange(0.0, 1.01, 0.01)
ens_errors = [ensemble_error(n_classifier=11, error=error)
              for error in error_range]
```

```python
import matplotlib.pyplot as plt

plt.plot(error_range,
         ens_errors,
         label='Ensemble error',
         linewidth=2)

plt.plot(error_range,
         error_range,
         linestyle='--',
         label='Base error',
         linewidth=2)

plt.xlabel('Base error')
plt.ylabel('Base/Ensemble error')
plt.legend(loc='upper left')
plt.grid()
plt.tight_layout()
# plt.savefig('./figures/ensemble_err.png', dpi=300)
plt.show()
```

The ensemble outperforms individual classifiers as long as their base error $\epsilon < 0.5$.

## Excercise

Plot ensembe error versus different number of classifiers.
Does the ensemble error always go down with more classifiers? Why or why not?
Can you fix the issue?

# Ensemble bias and variance

$d_j$ are decisions made by $m$ individual base classifiers

$y$ is the ensemble decision via summation (not really voting for mathematical tractability):
$$
y = \frac{1}{m} \sum_{j=1}^m d_j
$$

For iid:
$$
\begin{align}
E(y) &= E\left( \frac{1}{m} \sum_{j=1}^m d_j \right) = E(d_j)
\\
Var(y) &= Var\left( \frac{1}{m} \sum_{j=1}^m d_j \right) = \frac{1}{m} Var(d_j)
\end{align}
$$
* The expected value E, and thus bias, remains the same.
* The variance reduces.

For general case:

$$
\begin{align}
Var(y) &= Var\left( \frac{1}{m} \sum_{j=1}^m d_j \right)
\\
& = \frac{1}{m^2} \sum_j Var(d_j)
+ \frac{1}{m^2} \sum_j \sum_{i \neq j} Cov(d_j, d_i)
\end{align}
$$

So $Var(y)$ increases/decreases for positively/negatively correlated base estimators.

# Diversity versus accuracy

Not possible to have all base estimators accurate and yet negatively correlated.

Why?
* accurate estimators tend to make correct predictions
* correct predictions, by definition, are positively correlated with themselves

Sometimes need intentionally non-optimal base learners for better ensemble performance

# How to achieve diversification and accuracy

Different models/algorithms
* base learners complement each other
* e.g. some parametric, some non-parametric

Different hyper-parameters for the same algorithm/model
* k in KNN
* threshold in decision tree
* kernel function in SVM
* initial weights for perceptron and neural networks

Different input representations of the same event
* sensor fusion, sound and mouth shape for speech recognition
* random subset of features (columns of the data matrix)

Different training sets
* random subset of all samples (rows of the data matrix) - bagging
* sequential training - boosting, cascading inaccurately classified samples

Base learners be reasonably instead of very accurate

# How to combine multiple classifiers

## Parallel: multi-expert combination

Base learners work in parallel

Global approach
* all base learners generate outputs for a given input: voting, bagging

Local approach
* select only a few base learners based on the input: gating for mixture of experts

## Sequential: multi-stage combination

Base learners work in series

Later models focus on datasets not well handled by early models: cascading

Start with simpler models, and increase model complexity only if necessary

# Implementing a simple majority vote classifier

A (multi-class) majority vote classifier can be implemented via the weighted sum of $m$ individual classifier prediction:
$$
\hat{y} = argmax_i \sum_{j=1}^m w_j \left( C_j(x) = i \right)
$$

Here, $C_j(x) = i$ is a boolean expression for classifier $j$ to predict the class of $x$ to be $i$.

The weights can come from confidence, accuracy, or Bayesian prior of each classifier.
$$
p(C(x) = i) = \sum_{j=1}^m p(C_j) p\left( C_j(x) = i \; | \; C_j \right)
$$

### Equal weighting

If the individual classifiers are weighed equally, the above would reduce to what we see earlier:
$$
\hat{y} = mode\{C_1(x), C_2(x), \cdots, C_m(x) \}
$$

$C_1(x) = 0$,
$C_2(x) = 0$,
$C_3(x) = 1$
$\rightarrow$
$\hat{y} = mode\{0, 0, 1\} = 0$

## Unequal weighting

$w_1 = 0.2$,
$w_2 = 0.2$,
$w_3 = 0.6$
$\rightarrow$
$
\begin{cases}
0.4 & class \; 0 \\
0.6 & class \; 1
\end{cases}
$
$\rightarrow$
$\hat{y} = 1$

```python
import numpy as np

np.argmax(np.bincount([0, 0, 1],
                      weights=[0.2, 0.2, 0.6]))
```

    1

## Weighted probability

If the individual classifiers can provide prediction probability
$$
p\left(C_j(x) = i\right)
$$
, such as predict_prob() in scikit-learn, we can also predict ensemble class probability:

$$
\begin{align}
p\left(C_{ensemble}(x) = i \right)
&= \sum_{j=1}^m w_j p\left( C_j(x) = i \right)
\end{align}
$$

From which we can decide the predicted class (i.e. predict() in scikit-learn):
$$
\hat{y} = argmax_i \; p\left(C_{ensemble}(x) = i \right)
$$

In the 2-class 3-classifier example above,

$w_1 = 0.2$,
$w_2 = 0.2$,
$w_3 = 0.6$

$$
\begin{align}
C_1(x) &= [0.9, 0.1]
\\
C_2(x) &= [0.8, 0.2]
\\
C_3(x) &= [0.4, 0.6]
\end{align}
$$

Then
$$
\begin{align}
p\left(C_{ensemble} = 0\right) &= 0.2 \times 0.9 + 0.2 \times 0.8 + 0.6 \times 0.4 = 0.58
\\
p\left(C_{ensemble} = 1 \right) &= 0.2 \times 0.1 + 0.2 \times 0.2 + 0.6 \times 0.6 = 0.42
\end{align}
$$

$$
\hat{y} = 0
$$

```python
ex = np.array([[0.9, 0.1],
               [0.8, 0.2],
               [0.4, 0.6]])

p = np.average(ex,
               axis=0,
               weights=[0.2, 0.2, 0.6])
p
```

    array([ 0.58,  0.42])

```python
np.argmax(p)
```

    0

# Majority classifier implementation

```python
from sklearn.base import BaseEstimator
from sklearn.base import ClassifierMixin
from sklearn.preprocessing import LabelEncoder
from sklearn.externals import six
from sklearn.base import clone
from sklearn.pipeline import _name_estimators
import numpy as np
import operator

# inherits from two classes to get some methods for free
class MajorityVoteClassifier(BaseEstimator,
                             ClassifierMixin):
    """ A majority vote ensemble classifier

    Parameters
    ----------
    classifiers : array-like, shape = [n_classifiers]
      Different classifiers for the ensemble

    vote : str, {'classlabel', 'probability'} (default='label')
      If 'classlabel' the prediction is based on the argmax of
        class labels. Else if 'probability', the argmax of
        the sum of probabilities is used to predict the class label
        (recommended for calibrated classifiers).

    weights : array-like, shape = [n_classifiers], optional (default=None)
      If a list of `int` or `float` values are provided, the classifiers
      are weighted by importance; Uses uniform weights if `weights=None`.

    """
    def __init__(self, classifiers, vote='classlabel', weights=None):

        self.classifiers = classifiers
        self.named_classifiers = {key: value for key, value
                                  in _name_estimators(classifiers)}
        self.vote = vote
        self.weights = weights

    def fit(self, X, y):
        """ Fit classifiers.

        Parameters
        ----------
        X : {array-like, sparse matrix}, shape = [n_samples, n_features]
            Matrix of training samples.

        y : array-like, shape = [n_samples]
            Vector of target class labels.

        Returns
        -------
        self : object

        """
        if self.vote not in ('probability', 'classlabel'):
            raise ValueError("vote must be 'probability' or 'classlabel'"
                             "; got (vote=%r)"
                             % self.vote)

        if self.weights and len(self.weights) != len(self.classifiers):
            raise ValueError('Number of classifiers and weights must be equal'
                             '; got %d weights, %d classifiers'
                             % (len(self.weights), len(self.classifiers)))

        # Use LabelEncoder to ensure class labels start with 0, which
        # is important for np.argmax call in self.predict
        self.lablenc_ = LabelEncoder()
        self.lablenc_.fit(y)
        self.classes_ = self.lablenc_.classes_
        self.classifiers_ = []
        for clf in self.classifiers:
            fitted_clf = clone(clf).fit(X, self.lablenc_.transform(y))
            self.classifiers_.append(fitted_clf)
        return self

    def predict(self, X):
        """ Predict class labels for X.

        Parameters
        ----------
        X : {array-like, sparse matrix}, shape = [n_samples, n_features]
            Matrix of training samples.

        Returns
        ----------
        maj_vote : array-like, shape = [n_samples]
            Predicted class labels.

        """
        if self.vote == 'probability':
            maj_vote = np.argmax(self.predict_proba(X), axis=1)
        else:  # 'classlabel' vote

            #  Collect results from clf.predict calls
            predictions = np.asarray([clf.predict(X)
                                      for clf in self.classifiers_]).T

            maj_vote = np.apply_along_axis(
                                      lambda x:
                                      np.argmax(np.bincount(x,
                                                weights=self.weights)),
                                      axis=1,
                                      arr=predictions)
        maj_vote = self.lablenc_.inverse_transform(maj_vote)
        return maj_vote

    def predict_proba(self, X):
        """ Predict class probabilities for X.

        Parameters
        ----------
        X : {array-like, sparse matrix}, shape = [n_samples, n_features]
            Training vectors, where n_samples is the number of samples and
            n_features is the number of features.

        Returns
        ----------
        avg_proba : array-like, shape = [n_samples, n_classes]
            Weighted average probability for each class per sample.

        """
        probas = np.asarray([clf.predict_proba(X)
                             for clf in self.classifiers_])
        avg_proba = np.average(probas, axis=0, weights=self.weights)
        return avg_proba

    def get_params(self, deep=True):
        """ Get classifier parameter names for GridSearch"""
        if not deep:
            return super(MajorityVoteClassifier, self).get_params(deep=False)
        else:
            out = self.named_classifiers.copy()
            for name, step in six.iteritems(self.named_classifiers):
                for key, value in six.iteritems(step.get_params(deep=True)):
                    out['%s__%s' % (name, key)] = value
            return out
```

```python
import numpy as np
foo = np.asarray([np.random.uniform(size=4) for k in range(3)])
foo.shape
```

    (3, 4)

## Combining different algorithms for classification with majority vote

Let's apply the class we wrote above for classification.

### Load the dataset

Use only 2 features for more challenge

Only last 100 samples in 2 classes

```python
from sklearn import datasets
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import LabelEncoder
if Version(sklearn_version) < '0.18':
    from sklearn.cross_validation import train_test_split
else:
    from sklearn.model_selection import train_test_split

iris = datasets.load_iris()
X, y = iris.data[50:, [1, 2]], iris.target[50:]
le = LabelEncoder()
y = le.fit_transform(y)

X_train, X_test, y_train, y_test =\
       train_test_split(X, y,
                        test_size=0.5,
                        random_state=1)
```

### Build individual classifiers

Mixed types
* logistic regression
* decision tree
* knn

```python
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline
if Version(sklearn_version) < '0.18':
    from sklearn.cross_validation import cross_val_score
else:
    from sklearn.model_selection import cross_val_score

clf1 = LogisticRegression(penalty='l2',
                          C=0.001,
                          random_state=0)

clf2 = DecisionTreeClassifier(max_depth=1,
                              criterion='entropy',
                              random_state=0)

clf3 = KNeighborsClassifier(n_neighbors=1,
                            p=2,
                            metric='minkowski')

pipe1 = Pipeline([['sc', StandardScaler()],
                  ['clf', clf1]])
pipe3 = Pipeline([['sc', StandardScaler()],
                  ['clf', clf3]])

clf_labels = ['Logistic Regression', 'Decision Tree', 'KNN']

print('10-fold cross validation:\n')
for clf, label in zip([pipe1, clf2, pipe3], clf_labels):
    scores = cross_val_score(estimator=clf,
                             X=X_train,
                             y=y_train,
                             cv=10,
                             scoring='roc_auc')
    print("ROC AUC: %0.2f (+/- %0.2f) [%s]"
          % (scores.mean(), scores.std(), label))
```

    10-fold cross validation:

    ROC AUC: 0.92 (+/- 0.20) [Logistic Regression]
    ROC AUC: 0.92 (+/- 0.15) [Decision Tree]
    ROC AUC: 0.93 (+/- 0.10) [KNN]

### Majority vote

```python
# Majority Rule (hard) Voting

mv_clf = MajorityVoteClassifier(classifiers=[pipe1, clf2, pipe3])

clf_labels += ['Majority Voting']
all_clf = [pipe1, clf2, pipe3, mv_clf]

for clf, label in zip(all_clf, clf_labels):
    scores = cross_val_score(estimator=clf,
                             X=X_train,
                             y=y_train,
                             cv=10,
                             scoring='roc_auc')
    print("ROC AUC: %0.2f (+/- %0.2f) [%s]"
          % (scores.mean(), scores.std(), label))
```

    ROC AUC: 0.92 (+/- 0.20) [Logistic Regression]
    ROC AUC: 0.92 (+/- 0.15) [Decision Tree]
    ROC AUC: 0.93 (+/- 0.10) [KNN]
    ROC AUC: 0.97 (+/- 0.10) [Majority Voting]

Notice the better performance of ensemble learning

# Evaluating and tuning the ensemble classifier

## Plot ROC curves for test data

TP (true positive) versus FP (false positive) rates

```python
from sklearn.metrics import roc_curve
from sklearn.metrics import auc

colors = ['black', 'orange', 'blue', 'green']
linestyles = [':', '--', '-.', '-']
for clf, label, clr, ls \
        in zip(all_clf,
               clf_labels, colors, linestyles):

    # assuming the label of the positive class is 1
    y_pred = clf.fit(X_train,
                     y_train).predict_proba(X_test)[:, 1]
    fpr, tpr, thresholds = roc_curve(y_true=y_test,
                                     y_score=y_pred)
    roc_auc = auc(x=fpr, y=tpr)
    plt.plot(fpr, tpr,
             color=clr,
             linestyle=ls,
             label='%s (auc = %0.2f)' % (label, roc_auc))

plt.legend(loc='lower right')
plt.plot([0, 1], [0, 1],
         linestyle='--',
         color='gray',
         linewidth=2)

plt.xlim([-0.1, 1.1])
plt.ylim([-0.1, 1.1])
plt.grid()
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')

# plt.tight_layout()
# plt.savefig('./figures/roc.png', dpi=300)
plt.show()
```

## Plot decision regions

```python
# for visualization purposes, not necessary for pipelines with standard scalar
sc = StandardScaler()
X_train_std = sc.fit_transform(X_train)
```

```python
from itertools import product

all_clf = [pipe1, clf2, pipe3, mv_clf]

x_min = X_train_std[:, 0].min() - 1
x_max = X_train_std[:, 0].max() + 1
y_min = X_train_std[:, 1].min() - 1
y_max = X_train_std[:, 1].max() + 1

xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.1),
                     np.arange(y_min, y_max, 0.1))

f, axarr = plt.subplots(nrows=2, ncols=2,
                        sharex='col',
                        sharey='row',
                        figsize=(7, 5))

for idx, clf, tt in zip(product([0, 1], [0, 1]),
                        all_clf, clf_labels):
    clf.fit(X_train_std, y_train)

    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)

    axarr[idx[0], idx[1]].contourf(xx, yy, Z, alpha=0.3)

    axarr[idx[0], idx[1]].scatter(X_train_std[y_train==0, 0],
                                  X_train_std[y_train==0, 1],
                                  c='blue',
                                  marker='^',
                                  s=50)

    axarr[idx[0], idx[1]].scatter(X_train_std[y_train==1, 0],
                                  X_train_std[y_train==1, 1],
                                  c='red',
                                  marker='o',
                                  s=50)

    axarr[idx[0], idx[1]].set_title(tt)

plt.text(-3.5, -4.5,
         s='Sepal width [standardized]',
         ha='center', va='center', fontsize=12)
plt.text(-10.5, 4.5,
         s='Petal length [standardized]',
         ha='center', va='center',
         fontsize=12, rotation=90)

plt.tight_layout()
# plt.savefig('./figures/voting_panel', bbox_inches='tight', dpi=300)
plt.show()
```

Notice the decision boundary from majority voting is a combination of the 3 base classifiers.

# Hyper-parameter tuning for ensemble learning

## Get the parameter names

```python
mv_clf.get_params()
```

    {'decisiontreeclassifier': DecisionTreeClassifier(class_weight=None, criterion='entropy', max_depth=1,
                 max_features=None, max_leaf_nodes=None,
                 min_impurity_split=1e-07, min_samples_leaf=1,
                 min_samples_split=2, min_weight_fraction_leaf=0.0,
                 presort=False, random_state=0, splitter='best'),
     'decisiontreeclassifier__class_weight': None,
     'decisiontreeclassifier__criterion': 'entropy',
     'decisiontreeclassifier__max_depth': 1,
     'decisiontreeclassifier__max_features': None,
     'decisiontreeclassifier__max_leaf_nodes': None,
     'decisiontreeclassifier__min_impurity_split': 1e-07,
     'decisiontreeclassifier__min_samples_leaf': 1,
     'decisiontreeclassifier__min_samples_split': 2,
     'decisiontreeclassifier__min_weight_fraction_leaf': 0.0,
     'decisiontreeclassifier__presort': False,
     'decisiontreeclassifier__random_state': 0,
     'decisiontreeclassifier__splitter': 'best',
     'pipeline-1': Pipeline(steps=[['sc', StandardScaler(copy=True, with_mean=True, with_std=True)], ['clf', LogisticRegression(C=0.001, class_weight=None, dual=False, fit_intercept=True,
               intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
               penalty='l2', random_state=0, solver='liblinear', tol=0.0001,
               verbose=0, warm_start=False)]]),
     'pipeline-1__clf': LogisticRegression(C=0.001, class_weight=None, dual=False, fit_intercept=True,
               intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
               penalty='l2', random_state=0, solver='liblinear', tol=0.0001,
               verbose=0, warm_start=False),
     'pipeline-1__clf__C': 0.001,
     'pipeline-1__clf__class_weight': None,
     'pipeline-1__clf__dual': False,
     'pipeline-1__clf__fit_intercept': True,
     'pipeline-1__clf__intercept_scaling': 1,
     'pipeline-1__clf__max_iter': 100,
     'pipeline-1__clf__multi_class': 'ovr',
     'pipeline-1__clf__n_jobs': 1,
     'pipeline-1__clf__penalty': 'l2',
     'pipeline-1__clf__random_state': 0,
     'pipeline-1__clf__solver': 'liblinear',
     'pipeline-1__clf__tol': 0.0001,
     'pipeline-1__clf__verbose': 0,
     'pipeline-1__clf__warm_start': False,
     'pipeline-1__sc': StandardScaler(copy=True, with_mean=True, with_std=True),
     'pipeline-1__sc__copy': True,
     'pipeline-1__sc__with_mean': True,
     'pipeline-1__sc__with_std': True,
     'pipeline-1__steps': [['sc',
       StandardScaler(copy=True, with_mean=True, with_std=True)],
      ['clf',
       LogisticRegression(C=0.001, class_weight=None, dual=False, fit_intercept=True,
                 intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
                 penalty='l2', random_state=0, solver='liblinear', tol=0.0001,
                 verbose=0, warm_start=False)]],
     'pipeline-2': Pipeline(steps=[['sc', StandardScaler(copy=True, with_mean=True, with_std=True)], ['clf', KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
                metric_params=None, n_jobs=1, n_neighbors=1, p=2,
                weights='uniform')]]),
     'pipeline-2__clf': KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
                metric_params=None, n_jobs=1, n_neighbors=1, p=2,
                weights='uniform'),
     'pipeline-2__clf__algorithm': 'auto',
     'pipeline-2__clf__leaf_size': 30,
     'pipeline-2__clf__metric': 'minkowski',
     'pipeline-2__clf__metric_params': None,
     'pipeline-2__clf__n_jobs': 1,
     'pipeline-2__clf__n_neighbors': 1,
     'pipeline-2__clf__p': 2,
     'pipeline-2__clf__weights': 'uniform',
     'pipeline-2__sc': StandardScaler(copy=True, with_mean=True, with_std=True),
     'pipeline-2__sc__copy': True,
     'pipeline-2__sc__with_mean': True,
     'pipeline-2__sc__with_std': True,
     'pipeline-2__steps': [['sc',
       StandardScaler(copy=True, with_mean=True, with_std=True)],
      ['clf',
       KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
                  metric_params=None, n_jobs=1, n_neighbors=1, p=2,
                  weights='uniform')]]}

## Grid search parameters

```python
if Version(sklearn_version) < '0.18':
    from sklearn.grid_search import GridSearchCV
else:
    from sklearn.model_selection import GridSearchCV

params = {'decisiontreeclassifier__max_depth': [1, 2],
          'pipeline-1__clf__C': [0.001, 0.1, 100.0]}

grid = GridSearchCV(estimator=mv_clf,
                    param_grid=params,
                    cv=10,
                    scoring='roc_auc')
grid.fit(X_train, y_train)

if Version(sklearn_version) < '0.18':
    for params, mean_score, scores in grid.grid_scores_:
        print("%0.3f +/- %0.2f %r"
              % (mean_score, scores.std() / 2.0, params))

else:
    cv_keys = ('mean_test_score', 'std_test_score','params')

    for r, _ in enumerate(grid.cv_results_['mean_test_score']):
        print("%0.3f +/- %0.2f %r"
              % (grid.cv_results_[cv_keys[0]][r],
                 grid.cv_results_[cv_keys[1]][r] / 2.0,
                 grid.cv_results_[cv_keys[2]][r]))
```

    0.967 +/- 0.05 {'pipeline-1__clf__C': 0.001, 'decisiontreeclassifier__max_depth': 1}
    0.967 +/- 0.05 {'pipeline-1__clf__C': 0.1, 'decisiontreeclassifier__max_depth': 1}
    1.000 +/- 0.00 {'pipeline-1__clf__C': 100.0, 'decisiontreeclassifier__max_depth': 1}
    0.967 +/- 0.05 {'pipeline-1__clf__C': 0.001, 'decisiontreeclassifier__max_depth': 2}
    0.967 +/- 0.05 {'pipeline-1__clf__C': 0.1, 'decisiontreeclassifier__max_depth': 2}
    1.000 +/- 0.00 {'pipeline-1__clf__C': 100.0, 'decisiontreeclassifier__max_depth': 2}

```python
print('Best parameters: %s' % grid.best_params_)
print('Accuracy: %.2f' % grid.best_score_)
```

    Best parameters: {'pipeline-1__clf__C': 100.0, 'decisiontreeclassifier__max_depth': 1}
    Accuracy: 1.00

Observations:
* decision tree depth doesn't matter
* lower regularization (larger $C$) better

**Note**
By default, the default setting for `refit` in `GridSearchCV` is `True` (i.e., `GridSeachCV(..., refit=True)`), which means that we can use the fitted `GridSearchCV` estimator to make predictions via the `predict` method, for example:

    grid = GridSearchCV(estimator=mv_clf,
                        param_grid=params,
                        cv=10,
                        scoring='roc_auc')
    grid.fit(X_train, y_train)
    y_pred = grid.predict(X_test)

In addition, the "best" estimator can directly be accessed via the `best_estimator_` attribute.

```python
grid.best_estimator_.classifiers
```

    [Pipeline(steps=[['sc', StandardScaler(copy=True, with_mean=True, with_std=True)], ['clf', LogisticRegression(C=100.0, class_weight=None, dual=False, fit_intercept=True,
               intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
               penalty='l2', random_state=0, solver='liblinear', tol=0.0001,
               verbose=0, warm_start=False)]]),
     DecisionTreeClassifier(class_weight=None, criterion='entropy', max_depth=1,
                 max_features=None, max_leaf_nodes=None,
                 min_impurity_split=1e-07, min_samples_leaf=1,
                 min_samples_split=2, min_weight_fraction_leaf=0.0,
                 presort=False, random_state=0, splitter='best'),
     Pipeline(steps=[['sc', StandardScaler(copy=True, with_mean=True, with_std=True)], ['clf', KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
                metric_params=None, n_jobs=1, n_neighbors=1, p=2,
                weights='uniform')]])]

```python
mv_clf = grid.best_estimator_
```

```python
mv_clf.set_params(**grid.best_estimator_.get_params())
```

    MajorityVoteClassifier(classifiers=[Pipeline(steps=[('sc', StandardScaler(copy=True, with_mean=True, with_std=True)), ('clf', LogisticRegression(C=100.0, class_weight=None, dual=False, fit_intercept=True,
              intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
              penalty='l2', random_state=0, solv...ski',
               metric_params=None, n_jobs=1, n_neighbors=1, p=2,
               weights='uniform'))])],
                vote='classlabel', weights=None)

```python
mv_clf
```

    MajorityVoteClassifier(classifiers=[Pipeline(steps=[('sc', StandardScaler(copy=True, with_mean=True, with_std=True)), ('clf', LogisticRegression(C=100.0, class_weight=None, dual=False, fit_intercept=True,
              intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
              penalty='l2', random_state=0, solv...ski',
               metric_params=None, n_jobs=1, n_neighbors=1, p=2,
               weights='uniform'))])],
                vote='classlabel', weights=None)

# Bagging -- Building an ensemble of classifiers from bootstrap samples

Use random subsets of samples for each base classifier.
* bootstrap samples: random sample with/without replacement of the entire training data
* rows (samples) or cols (features)

Parallel method
* base classifiers operate independently

7-sample training data example:

# Bagging example

Classify the wine data set

## Load the dataset

```python
import pandas as pd

remote_source = 'https://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.data'
local_source = '../datasets/wine/wine.data'

df_wine = pd.read_csv(local_source, header=None)

df_wine.columns = ['Class label', 'Alcohol', 'Malic acid', 'Ash',
                   'Alcalinity of ash', 'Magnesium', 'Total phenols',
                   'Flavanoids', 'Nonflavanoid phenols', 'Proanthocyanins',
                   'Color intensity', 'Hue', 'OD280/OD315 of diluted wines',
                   'Proline']

# drop 1 class
df_wine = df_wine[df_wine['Class label'] != 1]

y = df_wine['Class label'].values
X = df_wine[['Alcohol', 'Hue']].values
```

## Encoding and splitting

```python
from sklearn.preprocessing import LabelEncoder
if Version(sklearn_version) < '0.18':
    from sklearn.cross_validation import train_test_split
else:
    from sklearn.model_selection import train_test_split

le = LabelEncoder()
y = le.fit_transform(y)

X_train, X_test, y_train, y_test =\
            train_test_split(X, y,
                             test_size=0.40,
                             random_state=1)
```

## Bagging classifier

```python
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

tree = DecisionTreeClassifier(criterion='entropy',
                              max_depth=None, # allow over-fitting to enhance diversity
                              random_state=1)

bag = BaggingClassifier(base_estimator=tree,
                        n_estimators=500,
                        max_samples=1.0, # rows
                        max_features=1.0, # cols
                        bootstrap=True, # replace?
                        bootstrap_features=False,
                        n_jobs=1,
                        random_state=1)
```

```python
from sklearn.metrics import accuracy_score

tree = tree.fit(X_train, y_train)
y_train_pred = tree.predict(X_train)
y_test_pred = tree.predict(X_test)

tree_train = accuracy_score(y_train, y_train_pred)
tree_test = accuracy_score(y_test, y_test_pred)
print('Decision tree train/test accuracies %.3f/%.3f'
      % (tree_train, tree_test))

bag = bag.fit(X_train, y_train)
y_train_pred = bag.predict(X_train)
y_test_pred = bag.predict(X_test)

bag_train = accuracy_score(y_train, y_train_pred)
bag_test = accuracy_score(y_test, y_test_pred)
print('Bagging train/test accuracies %.3f/%.3f'
      % (bag_train, bag_test))
```

    Decision tree train/test accuracies 1.000/0.833
    Bagging train/test accuracies 1.000/0.896

Notice bagging has less over-fitting than decision tree.

## Visualize decision regions

Use meshgrid and isocontour as usual.

```python
import numpy as np
import matplotlib.pyplot as plt

x_min = X_train[:, 0].min() - 1
x_max = X_train[:, 0].max() + 1
y_min = X_train[:, 1].min() - 1
y_max = X_train[:, 1].max() + 1

xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.1),
                     np.arange(y_min, y_max, 0.1))

f, axarr = plt.subplots(nrows=1, ncols=2,
                        sharex='col',
                        sharey='row',
                        figsize=(8, 3))

for idx, clf, tt in zip([0, 1],
                        [tree, bag],
                        ['Decision Tree', 'Bagging']):
    clf.fit(X_train, y_train)

    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)

    axarr[idx].contourf(xx, yy, Z, alpha=0.3)
    axarr[idx].scatter(X_train[y_train == 0, 0],
                       X_train[y_train == 0, 1],
                       c='blue', marker='^')

    axarr[idx].scatter(X_train[y_train == 1, 0],
                       X_train[y_train == 1, 1],
                       c='red', marker='o')

    axarr[idx].set_title(tt)

axarr[0].set_ylabel('Alcohol', fontsize=12)
plt.text(10.2, -1.2,
         s='Hue',
         ha='center', va='center', fontsize=12)

plt.tight_layout()
# plt.savefig('./figures/bagging_region.png',
#            dpi=300,
#            bbox_inches='tight')
plt.show()
```

Notice the smoother decision boundaries of bagging than decision tree.

Bagging can reduce variance, but not bias.

Choose base classifiers with low bias, such as unpruned decision trees.

# Leveraging weak learners via adaptive boosting

AdaBoost for abbreviation

A collection of weak base learners (diversity over accuracy)
* sequential learning
* later classifiers focus on the weak parts (mis-classified samples) of the earlier classifiers

## Original boosting procedure [Schapire 1990]

Using 3 classifiers

Training:
* Train $C_1$ with a random subset of training data

* Train $C_2$ with a random subset of training data $\bigcup$ misclassified samples from $C_1$

* Train $C_3$ over the training samples for which $C_1$ and $C_2$ disagree

Testing/Usage:
* Combine $C_1$, $C_2$, $C_3$ via majority voting

## AdaBoost versus bagging

Bagging
* parallel
* random sample with replacement
* cannot reduce bias; same as base classifiers

AdaBoost
* sequential
* random sample without replacement
* can reduce bias

## AdaBoost example

Give previously misclassified samples higher weights

Example using decision tree stumps (i.e. very shallow trees, like just one level)

Steps:
1. equal weight training of all samples by $C_1$, two blue circles are misclassified
2. larger/lower weights to wrongly/correctly classified samples, train $C_2$
3. larger/lower weights to wrongly/correctly classified samples, train $C_3$
4. combine $C_1$, $C_2$, $C_3$ for weighted majority voting

## AdaBoost algorithm

Input data: $X$ and $y$ (label)

Initial equal weight vector $w$ to all samples, $\sum_i w_i = 1$

For j in m boosting rounds, do the following:
1. Train a weak learner: $C_j.train(X, y, w)$
2. Predict class labels: $\hat{y} = C_j.predict(X)$
3. Compute weighted error rate: $\epsilon = w . \left(\hat{y} \neq y \right)$, $0 \leq \epsilon < 0.5$
4. Compute re-weighting coefficients: $\alpha_j = 0.5 \log_e\left(\frac{1-\epsilon}{\epsilon}\right)$ $\geq0$ for $0 \leq \epsilon < 0.5$
 * $\epsilon \uparrow$ $\rightarrow$ $\alpha_j \downarrow$
 * $\epsilon = 0$ $\rightarrow$ $\alpha_j = \infty$ - equal weighting if no error
 * $\epsilon = 0.5$ $\rightarrow$ $\alpha_j = 0$ - no weight update if serious errors
5. Update weights: $w \leftarrow w \times \exp(-\alpha_j . \hat{y} \times y)$, $\times$ means element-wise product
 * correct/incorrect prediction will decrease/increase weight
6. Normalize weights: $w \leftarrow \frac{w}{\sum_i w_i}$

Compute final prediction via weighted ensemble voting:
$
\hat{y} = \sum_j \alpha_j \times C_j.predict(X)
$

```python
import matplotlib.pyplot as plt
import numpy as np

def alpha_func(x):
    return 0.5*np.log((1-x)/x)

delta = 0.01
x = np.arange(delta, 0.5, delta)
y = alpha_func(x)

plt.plot(x, y)
plt.xlabel(r'$\epsilon$')
plt.ylabel(r'$\alpha$')
plt.tight_layout()
plt.show()
```

10 sample example

$\epsilon = 0.3$

$\alpha = 0.5\log_e(0.7/0.3) = 0.424$

$w$ before normalization:
* correct prediction: $0.1 \times \exp(-0.424 \times 1 \times 1) = 0.066$
* incorrect prediction: $0.1 \times \exp(-0.424 \times 1 \times -1) = 0.153$

# AdaBoost code example

```python
from sklearn.ensemble import AdaBoostClassifier

tree = DecisionTreeClassifier(criterion='entropy',
                              max_depth=1,
                              random_state=0)

ada = AdaBoostClassifier(base_estimator=tree,
                         n_estimators=500,
                         learning_rate=0.1,
                         random_state=0)
```

```python
tree = tree.fit(X_train, y_train)
y_train_pred = tree.predict(X_train)
y_test_pred = tree.predict(X_test)

tree_train = accuracy_score(y_train, y_train_pred)
tree_test = accuracy_score(y_test, y_test_pred)
print('Decision tree train/test accuracies %.3f/%.3f'
      % (tree_train, tree_test))

ada = ada.fit(X_train, y_train)
y_train_pred = ada.predict(X_train)
y_test_pred = ada.predict(X_test)

ada_train = accuracy_score(y_train, y_train_pred)
ada_test = accuracy_score(y_test, y_test_pred)
print('AdaBoost train/test accuracies %.3f/%.3f'
      % (ada_train, ada_test))
```

    Decision tree train/test accuracies 0.845/0.854
    AdaBoost train/test accuracies 1.000/0.875

Better training performance by AdaBoost, but with more overfitting than bagging

```python
x_min, x_max = X_train[:, 0].min() - 1, X_train[:, 0].max() + 1
y_min, y_max = X_train[:, 1].min() - 1, X_train[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.1),
                     np.arange(y_min, y_max, 0.1))

f, axarr = plt.subplots(1, 2, sharex='col', sharey='row', figsize=(8, 3))

for idx, clf, tt in zip([0, 1],
                        [tree, ada],
                        ['Decision Tree', 'AdaBoost']):
    clf.fit(X_train, y_train)

    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)

    axarr[idx].contourf(xx, yy, Z, alpha=0.3)
    axarr[idx].scatter(X_train[y_train == 0, 0],
                       X_train[y_train == 0, 1],
                       c='blue', marker='^')
    axarr[idx].scatter(X_train[y_train == 1, 0],
                       X_train[y_train == 1, 1],
                       c='red', marker='o')
    axarr[idx].set_title(tt)

axarr[0].set_ylabel('Alcohol', fontsize=12)
plt.text(10.2, -1.2,
         s='Hue',
         ha='center', va='center', fontsize=12)

plt.tight_layout()
# plt.savefig('./figures/adaboost_region.png',
#           dpi=300,
#           bbox_inches='tight')
plt.show()
```

AdaBoost has more complex decision boundary than decision tree, as expected.

# Summary

Combine multiple (independent) base learners can improve performance
* majority voting
* bagging - parallel, can reduce variance but not bias
* AdaBoost - sequential, can reduce bias but might overfit

Choose diverse, not just, accurate base learners
* variance and bias trade off

Ensemble learning computationally more expensive

Netflix prize

# Reading

* PML Chapter 7
* IML Chapter 17.1-17.7


## Partial Fractions

In this notebook, we introduce another essential tool in the evaluation of integrals, namely the method of partial fractions.

### 1. Motivation

It is true that
$$
\frac{1}{x^2-16} = \frac{1}{(x-4)(x+4)} = \frac{1}{8} \left[ \frac{1}{x-4} - \frac{1}{x+4} \right].
$$

You can verify this by working backwards, that is, by starting from the right-hand side and making a common denominator (you should go ahead and do this - be sure to carefully keep track of negative signs).

Although we cannot readily integrate the left-hand side, we can readily integrate the right-hand side.  We then get
\begin{align}
\int \frac{dx}{x^2-16}
& = \frac{1}{8} \left[ \int \frac{dx}{x-4} - \int \frac{dx}{x+4} \right] \\
& = \frac{1}{8} \left[ \ln | x-4 | - \ln | x+4| \right] + C \\
& = \frac{1}{8} \ln \left| \frac{x-4}{x+4} \right| + C.
\end{align}

The key to making progress in the previous example was to write a 'complicated' rational function (in this case $\frac{1}{x^2-16})$ as a sum of simpler functions.

> Recall: A rational function is of the form $\displaystyle \frac{P(x)}{Q(x)}$, where both $P(x)$ and $Q(x)$ are polynomials.

Partial fraction decomposition, or simply 'partial fractions', is a systematic method to write a rational function as a sum of simpler functions.

### 2. Overview of Partial Fraction Decomposition

Consider a rational function $\displaystyle \frac{P(x)}{Q(x)}$ such that the degree of $P(x)$ is less than the degree of $Q(x)$.

Then the steps to determine the partial fraction decomposition are as follows:

1. Factor $Q(x)$ as far as possible.
> In MATH 136, we will restrict ourselves to the case where $Q(x)$ is a product of distinct linear factors.
2. Write the partial fraction decomposition using the rule that any linear factor $ax+b$ in $Q(x)$ gives rise to a partial fraction of the form $$\frac{A}{ax+b},$$ where $A$ is an unknown to be determined in step 3.
> Example: $$\frac{4x^2+14x-2}{(x-1)(x+1)(x+3)} = \frac{A}{x-1} + \frac{B}{x+1} + \frac{C}{x+3}.$$
3. Solve for the unknowns ($A$, $B$, $C$, etc.).

Once you have solved for the unknowns, you can proceed with any integration if needed.

### 3. One Example

In the next video, we will work through the example $\displaystyle \int \frac{4x^2 + 14x - 2}{x^3 + 3x^2-x-3} \ dx$.  Note that the degree of the polynomial in the numerator is 2, and the degree of the polynomial in the denominator is 3.  So the degree in the numerator is less than the degree in the denominator, and it is OK to proceed with partial fraction decomposition.

NOTE: The video was created before this notebook, and covers the introduction above.  You may skip through the first 2 minutes or so and start viewing at 2:44.

```python
from IPython.display import YouTubeVideo
YouTubeVideo('1pmFeBjLCKo')
```

### 4. A Second Method to Solve for the Unknowns

In the previous video, we showed one method to solve for the unknowns (let's refer to it as 'Tollo's Method').  It is a method that works really well for the case that $Q(x)$ is a product of distinct linear factors.  In the next video, we show a second method to solve for the unknowns ('Gerla's Method').

```python
from IPython.display import YouTubeVideo
YouTubeVideo('WKOJPPwSSRQ')
```

All right, so now we have two methods that we can use to solve for the unknowns.  You have a choice: you may use either Tollo's Method or Gerla's Method.  Which do you like best?

### 5. CAUTION! A Long Division Step May Be Needed Before Proceeding with a Partial Fraction Decomposition

Up to now, we have considered rational functions $\displaystyle \frac{P(x)}{Q(x)}$ where the degree of $P(x)$ is less than the degree of $Q(x)$.

This raises the question what to do when the degree of $P(x)$ is the same or larger than the degree of $Q(x)$.
> Example: The rational function $$\frac{4x^2+3x+2}{x^2+x-2}$$ has a polynomial with degree 3 in the numerator and a polynomial with degree 2 in the denominator.  Since the degree in the numerator is larger than the degree in the denominator, we cannot proceed immediately with the partial fraction decomposition.

In these cases, it is necessary to take the preliminary step of using long division to divide $Q(x)$ into $P(x)$ and obtain
$$
\frac{P(x)}{Q(x)} = S(x) + \frac{R(x)}{Q(x)},
$$
where $S(x)$ is a polynomial and $\displaystyle \frac{R(x)}{Q(x)}$ is a rational function such that the degree of $R(x)$ is less than the degree of $Q(x)$.  The partial fraction decomposition then can proceed for the rational function on the right-hand side, $\displaystyle \frac{R(x)}{Q(x)}$.

In the next video, we work through the example $\displaystyle \int \frac{4x^2 + 3x+2}{x^2+x-2} \ dx$.

```python
from IPython.display import YouTubeVideo
YouTubeVideo('uo-Ybafwoug')
```

### 6. Summary

Partial fraction decomposition is a systematic method to write a proper rational function as a sum of simpler rational functions.
> A proper rational function is a rational function such that the degree of the polynomial in the numerator is less than the degree of the polynomial in the denominator.
For example, $\displaystyle \frac{x}{x^3-1}$ is a proper rational function,
but neither $\displaystyle \frac{x^3+2x-5}{x^3-1}$
nor $\displaystyle \frac{x^4-3x}{x^3-1}$ are.

In general, the simpler rational functions are of the form
$$
\frac{A}{(ax+b)^n} \ \ \ \textrm{or} \ \ \ \frac{Bx+C}{(ax^2 + bx + c)^n},
$$
where $a$, $b$, $c$, $A$, $B$, and $C$ are constants and $n$ is a positive integer, and the quadratic $ax^2+bx+c$ is irreducible (cannot be factored further since $b^2 - 4 a c < 0$).

If a proper rational function is not already in one of these simpler forms, then the steps to determine the partial fraction decomposition are as follows:

1. Factor $Q(x)$ as far as possible.
 - In general, $Q(x)$ will be the product of linear factors and/or irreducible factors.
 - In MATH 136, we work with the restriction that $Q(x)$ is the product only of distinct linear factors.

2. Under the restriction on $Q(x)$ mentioned in Step 1, write the partial fraction decomposition using the rule that each linear factor $ax+b$ in $Q(x)$ gives rise to a partial fraction of the form $$\frac{A}{ax+b},$$ where $A$ is an unknown to be determined in step 3.

3. Solve for the unknowns ($A$, $B$, $C$, etc.).

Note: If the rational function $\displaystyle \frac{P(x)}{Q(x)}$ is not proper, then it is necessary to use long division first and obtain
$$
\frac{P(x)}{Q(x)} = S(x) + \frac{R(x)}{Q(x)},
$$
where $S(x)$ is a polynomial and $\displaystyle \frac{R(x)}{Q(x)}$ is a proper rational function.

### 7. Further Study

Please refer to Section 5.6 in the textbook for additional treatment of this topic.

### 8. Don't Forget

Don't forget to return to eClass to complete the pre-class quiz.

```python

```


# [OLD] Measurements

Lets say we are interested in the harmonic oscillator.  We can use methods from the HamiltonianTerms file to build a hamiltonian, each of these routines takes as input n, the size of the matrix it produces.  This is related to the truncation of the bosonic modes.

```python
import sys
sys.path.append('..')
import src.HamiltonianTerms as ham
import numpy as np
import sympy as sp
import math
from qiskit import ClassicalRegister

dim = 8
hmat = ham.number_op(dim) + 0.5*np.identity(dim)

import sys
sys.path.append('..')
from src.MatrixToQISKITCircuit import *
from src.BinaryEncodings import *

mtoqc = MatrixToQISKITCircuit(np.asarray(hmat), 1)
mtoqc.convert(standard_encode)
```

```python
mtoqc.circuit.draw(output="mpl")
```

```python
print(mtoqc.circuit.ancillas)
```

    []

```python
from qiskit import (
    QuantumCircuit,
    execute,
    Aer)
from qiskit.providers.aer.noise import NoiseModel
from qiskit import IBMQ
from qiskit.visualization import plot_histogram

provider = IBMQ.load_account()
```

    /home/chrisculver/miniconda3/envs/QuantumComputing/lib/python3.9/site-packages/qiskit/providers/ibmq/ibmqfactory.py:192: UserWarning: Timestamps in IBMQ backend properties, jobs, and job results are all now in local time instead of UTC.
      warnings.warn('Timestamps in IBMQ backend properties, jobs, and job results '

```python
noise_backend = provider.get_backend('ibmq_16_melbourne')
noise_model = NoiseModel.from_backend(noise_backend)
```

```python
simulator = Aer.get_backend('qasm_simulator')

circuit = QuantumCircuit(math.log(dim,2),math.log(dim,2))
print(mtoqc.circuit.qregs)
print(mtoqc.circuit.cregs)
print(circuit.qregs)
print(circuit.cregs)

circuit = circuit + mtoqc.circuit + mtoqc.circuit + mtoqc.circuit + mtoqc.circuit + mtoqc.circuit + mtoqc.circuit + mtoqc.circuit + mtoqc.circuit + mtoqc.circuit + mtoqc.circuit + mtoqc.circuit + mtoqc.circuit + mtoqc.circuit + mtoqc.circuit + mtoqc.circuit +mtoqc.circuit + mtoqc.circuit + mtoqc.circuit+mtoqc.circuit + mtoqc.circuit + mtoqc.circuit
```

    [QuantumRegister(3, 'q')]
    []
    [QuantumRegister(3, 'q')]
    [ClassicalRegister(3, 'c')]

```python
circuit.measure([i for i in range(round(math.log(dim,2)))], [i for i in range(round(math.log(dim,2)))])
```

    <qiskit.circuit.instructionset.InstructionSet at 0x7f893c311550>

```python
job = execute(circuit, simulator, shots=1000, noise_model=noise_model)
```

```python
result=job.result()
counts = result.get_counts(circuit)
plot_histogram(counts)
```

```python
dim = 8
adag = ham.adag(dim)

mtops = MatrixToPauliString(np.asarray(adag))
mtops.convert(standard_encode)
```

```python
sp.expand(mtops.sum_of_paulis)
```

$\displaystyle 0.951733762016657 I^{1} I^{2} X^{0} - 0.951733762016657 i I^{1} I^{2} Z^{0} - 0.268721060124438 I^{1} X^{0} Z^{2} + 0.268721060124438 i I^{1} Z^{0} Z^{2} + 0.482962913144534 I^{2} X^{0} X^{1} - 0.14271676764171 I^{2} X^{0} Z^{1} - 0.482962913144534 i I^{2} X^{0} Z^{1} + 0.482962913144534 i I^{2} X^{1} Z^{0} + 0.482962913144534 I^{2} Z^{0} Z^{1} + 0.14271676764171 i I^{2} Z^{0} Z^{1} + 0.25 X^{0} X^{1} X^{2} - 0.12940952255126 X^{0} X^{1} Z^{2} - 0.25 i X^{0} X^{1} Z^{2} + 0.25 i X^{0} X^{2} Z^{1} + 0.20970406574949 X^{0} Z^{1} Z^{2} + 0.12940952255126 i X^{0} Z^{1} Z^{2} + 0.25 i X^{1} X^{2} Z^{0} + 0.25 X^{1} Z^{0} Z^{2} - 0.12940952255126 i X^{1} Z^{0} Z^{2} - 0.25 X^{2} Z^{0} Z^{1} - 0.12940952255126 Z^{0} Z^{1} Z^{2} + 0.29029593425051 i Z^{0} Z^{1} Z^{2}$

```python
ptoqc = PauliStringsToQISKITCircuit(mtops.sum_of_paulis)
ptoqc.string_to_terms()
```

```python
pauli_dict={'paulis': []}
for k,v in ptoqc.pauli_terms.items():
    pauli_dict['paulis'].append({"coeff": {"real": v.real, "imag": v.imag}, "label":k})
```

```python

```


```python
%matplotlib inline
%load_ext autoreload
%autoreload 2

import matplotlib.pyplot as plt
import matplotlib.ticker as mticker
import numpy as np

import torch
from torch import nn

import os
import sys
import glob
import pickle
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

```python
SEED = 52
torch.manual_seed(SEED)

# Add parent dir to interpreter path
nb_dir = os.path.split(os.getcwd())[0]
if nb_dir not in sys.path:
    sys.path.append(nb_dir)
```

# Loading synthetic data

We will be loading simulated Hawkes data.

```python
SYNTH_DATA_FILES = glob.glob('../data/simulated/*.pkl')
SYNTH_DATA_FILES.sort()
print("Available files:")
for s in SYNTH_DATA_FILES:
    print(s)

from utils.load_synth_data import process_loaded_sequences, one_hot_embedding
```

    Available files:
    ../data/simulated/hawkes_sumxp_synthetic_exp_2d_20181210-002610.pkl
    ../data/simulated/hawkes_synthetic_exp_1d_20181205-142133.pkl
    ../data/simulated/hawkes_synthetic_exp_2d_20181206-160527.pkl
    ../data/simulated/hawkes_synthetic_exp_2d_20181207-220402.pkl
    ../data/simulated/hawkes_synthetic_exp_2d_20181209-211556.pkl

```python
# SET THESE PARAMETERS
process_dim = 2
print("Process dimension: {}".format(process_dim))
chosen_data_file = SYNTH_DATA_FILES[3]
print("Data file: {}".format(chosen_data_file))
```

    Process dimension: 2
    Data file: ../data/simulated/hawkes_synthetic_exp_2d_20181207-220402.pkl

```python
# Load data simulated using tick
print("Loading {}-dimensional Hawkes data.".format(process_dim))

with open(chosen_data_file, "rb") as f:
    loaded_hawkes_data = pickle.load(f)

print(loaded_hawkes_data.keys())

mu = loaded_hawkes_data['mu']
alpha = loaded_hawkes_data['alpha']
decay = loaded_hawkes_data['decay']
tmax = loaded_hawkes_data['tmax']
print("mu\t{:}".format(str(mu)))
print("alpha\t{:}".format(str(alpha)))
print("decay\t{:}".format(str(decay)))
print("tmax\t{:}".format(tmax))

seq_times, seq_types, seq_lengths = process_loaded_sequences(
    loaded_hawkes_data, process_dim, tmax)
print(seq_times.shape)
```

    Loading 2-dimensional Hawkes data.
    dict_keys(['mu', 'alpha', 'decay', 'tmax', 'timestamps', 'types', 'lengths'])
    mu	[0.1 0.1]
    alpha	[[0.1  0.01]
     [0.01 0.1 ]]
    decay	1
    tmax	3600
    torch.Size([4000, 921])

# Neural network

We are going for a recurrent neural structure that can learn from previous events. The intensity process of our neural model will be of the form

$$
    \lambda_t = f\left(W_{\lambda} h(t)\right)
$$

with $f$ a positive-valued activation function (such as ReLU) and $h(t)\in\mathbb R^D$ a continuous hidden state, computed as
\begin{equation}
    h(t) = h_{i}e^{-\delta_i(t-t_{i-1})},\quad \forall t\in(t_{i-1},t_i]
\end{equation}

Network update formulas: the jump and decay parameters on $(t_i,t_{i+1}]$ are given by
$$
\begin{align*}
    h_{i+1}      &= \tanh(W_{hh}h(t_i) + W_{xh}x_i + b_{h}) \\
    \delta_{i+1} &= f(W_{hd} h(t_i) + W_{xd}x_i + b_{d})
\end{align*}
$$

```python
from models.decayrnn import HawkesDecayRNN, HawkesRNNGen
from train_functions import train_decayrnn, plot_loss
```

```python
# define training data
train_size = 1000
print("Train size %d" % train_size)

train_times_tensor = seq_times[:train_size]
train_seq_types = seq_types[:train_size]
train_seq_lengths = seq_lengths[:train_size]
print("Train event times shape {}".format(train_times_tensor.shape))
print("Train event types shape {}".format(train_seq_types.shape))
print("Train seq lengths shape {}".format(train_seq_lengths.shape))
```

    Train size 1000
    Train event times shape torch.Size([1000, 152])
    Train event types shape torch.Size([1000, 152])
    Train seq lengths shape torch.Size([1000])

```python
process_dim = 4
hidden_size = 128
model = HawkesDecayRNN(process_dim, hidden_size)
optimizer = torch.optim.Adam(model.parameters(), lr=0.020)
```

```python
# Training parameters
BATCH_SIZE = 32
EPOCHS = 6

loss_hist, train_hist = train_decayrnn(
    model, optimizer, train_times_tensor,
    train_seq_types, train_seq_lengths, tmax,
    BATCH_SIZE, EPOCHS, use_jupyter=True)
```

```python
_ = plot_loss(EPOCHS, loss_hist)
```

# Model evaluation

```python
tmax = 3600

MODEL_FILES = glob.glob("../saved_models/HawkesDec*.pth")
MODEL_FILES.sort()
for p in MODEL_FILES:
    print(p)
```

    ../saved_models/HawkesDecayRNN-2d_hidden128-20181209-153942.pth
    ../saved_models/HawkesDecayRNN-2d_hidden128-20181209-212603.pth
    ../saved_models/HawkesDecayRNN-2d_hidden128-20181210-013110.pth

```python
load_filename = MODEL_FILES[0]
print("chosen file:", load_filename)

load_file_timestamp = "-".join(os.path.splitext(os.path.basename(load_filename))[0].split('-')[-2:])
print(load_file_timestamp)
```

    chosen file: ../saved_models/HawkesDecayRNN-2d_hidden128-20181209-153942.pth
    20181209-153942

```python
from models.decayrnn import HawkesDecayRNN, HawkesRNNGen
process_dim = 2
hidden_size = 128
print("Hidden size: {}".format(hidden_size))
model = HawkesDecayRNN(process_dim, hidden_size)

model.load_state_dict(torch.load(load_filename, map_location={"cuda:0": "cpu"}))
```

    Hidden size: 128

## Sequence generation

### Untrained model, random weights

```python
print("Real end time: {}".format(tmax))
```

    Real end time: 3600

```python
raw_model = HawkesDecayRNN(process_dim, hidden_size)
generator_base = HawkesRNNGen(raw_model)
```

    Process model dim:	2	Hidden units:	128

```python
sum(e.numel() for e in raw_model.parameters())
```

    33926

```python
tmax_simu = 60
generator_base.generate_sequence(tmax_simu, record_intensity=True)

fig = generator_base.plot_events_and_intensity(debug=False)
fig.tight_layout()
```

```python
fig.savefig("example_rnnplot2d_hidden{}.pdf".format(hidden_size))
```

### On trained model

```python
RESULT_BASEDIR = "../results"

MODEL_NAME = model.__class__.__name__
print(MODEL_NAME)
```

    HawkesDecayRNN

```python
generator = HawkesRNNGen(model, record_intensity=False)
```

    Process model dim:	2	Hidden units:	128

```python
tmax_simu = 120
generator.generate_sequence(tmax_simu, record_intensity=True)
fig_intensity_stream = generator.plot_events_and_intensity()
```

```python
fig_intensity_stream.savefig("../results/intensity_{}_{}d-sumexp_hidden{}_{}.pdf".format(
    MODEL_NAME, process_dim, hidden_size, load_file_timestamp))
```

## Length distribution

```python
from utils import evaluation
```

```python
n_gen_seq = 500
```

```python
gen_seq_lengths, gen_seq_types_lengths = evaluation.generate_multiple_sequences(generator, tmax, n_gen_seq=500)
```

    tmax: 1800
    Mean generated sequence length: 768.29
    Generated sequence length std. dev: 30.95942344424392

```python
if process_dim > 1:
    num_plots = process_dim + 1
else:
    num_plots = 1
fig, axes = plt.subplots(num_plots, 1, figsize=(8,4+(num_plots-1)*3), dpi=100)
ax0 = axes[0] if num_plots > 1 else axes
hist_opts = {
    "rwidth": 0.8,
    "alpha": 0.4,
    "density": True
}
gen_label = "Sequence lengths (generated by {:})".format(MODEL_NAME)
_ = ax0.hist(seq_lengths, bins=2*int(n_gen_seq**(1/3)),
            label="Sequence lengths (base model)",
            **hist_opts)
_ = ax0.hist(gen_seq_lengths, bins=2*int(n_gen_seq**(1/3)),
            label=gen_label,
            color='red', **hist_opts)
ax0.legend()
if num_plots > 1:
    ax0.set_title('no. of events (all types, including starter type)')
else:
    ax0.set_title('no. of  events')
ax0.set_ylabel('Proportion')
ax0.set_xlabel('Number of events $N_T$')
if num_plots > 1:
    axt = axes[1:]
    for i, ax in enumerate(axt):
        # get the sequence lengths for the event types
        typ_seq_lengths = (seq_types == i).sum(dim=1)
        gen_typ_seq_lengths = gen_seq_types_lengths[:, i]
        ax.hist(typ_seq_lengths, bins=2*int(typ_seq_lengths.size(0)**(1/3)),
                label="Sequence lengths (base model)",
                **hist_opts)
        ax.hist(gen_typ_seq_lengths,
                bins=2*int(gen_typ_seq_lengths.shape[0]**(1/3)),
                label=gen_label,
                color='red', **hist_opts)
        ax.set_xlabel("Number of events of type {}, $N_T^{}$".format(i, i))
        ax.set_title('no. of events of type $i={}$'.format(i))

fig.tight_layout()
```

```python
os.makedirs(RESULT_BASEDIR, exist_ok=True)
model_perf_indicator = "length_distrib"
distrib_plot_filepath = "{:}/{:}_{:}-{}-hidden_{}-{}.pdf".format(
    RESULT_BASEDIR, model_perf_indicator, MODEL_NAME,
    process_dim, model.hidden_size, load_file_timestamp)
fig.savefig(distrib_plot_filepath)
```

## Event prediction

The $L_2$-wise best estimator for the next event occurrence time is given by

$$
\begin{aligned}
    \hat{t}_i &= \mathbb{E}[t_i\mid\mathcal F_{t_{i-1}}] \\
    &= t_{i-1} + \int_0^\infty u p_i(u)\, du
\end{aligned}
$$

```python
import tqdm
from utils import evaluation
```

```python
dataset_size = seq_times.shape[0]
print(dataset_size)
```

    4000

```python
my_index = torch.randint(0, dataset_size, (1,)).int().item()
print(my_index)

_seq_data = (seq_times[my_index],
             seq_types[my_index],
             seq_lengths[my_index])

restup, fig = model.read_predict(*_seq_data, plot=True, hmax=90., print_info=True)
print(restup)
```

```python
fig.savefig('decayrnn_{}d_prediction_graphs_hidden{}_NEW.pdf'
            .format(process_dim, hidden_size))
```

```python
train_size = 1200
test_size = 1000

print("Train:", train_size)
print("Test:", test_size)
```

    Train: 1200
    Test: 1000

```python
test_data = (seq_times[train_size:(train_size+test_size)],
             seq_types[train_size:(train_size+test_size)],
             seq_lengths[train_size:(train_size+test_size)])
```

```python
incr_estimates, incr_errors, types_real, types_estimates = evaluation.predict_test(model, *test_data,
                                                                                hmax=90., use_jupyter=True)
```

    HBox(children=(IntProgress(value=0, max=1000), HTML(value='')))

```python

```

```python
avg_rmse = np.sqrt(np.mean(incr_errors), dtype=np.float64)
print("rmse", avg_rmse)

mse_var = np.var(incr_errors, dtype=np.float64)
print("mse var", mse_var)

delta_meth_stderr = 1 / test_size * mse_var / (4 * avg_rmse)
print("rmse var", delta_meth_stderr)
```

    rmse 3.7128522249871843
    mse var 1464.2632804715774
    rmse var 0.09859423374146217

```python
from sklearn.metrics import confusion_matrix, accuracy_score
```

```python
types_predict_score = accuracy_score(types_real, types_estimates)
print("Type prediction score:", types_predict_score)

print("Confusion matrix:\n", confusion_matrix(types_real, types_estimates))
```

    Type prediction score: 0.592
    Confusion matrix:
     [[  0 403]
     [  5 592]]

```python

```

```python
sum(e.numel() for e in model.parameters())
```

    33926

```python
predict_data = {
    "model_name": MODEL_NAME,
    "process_dim": process_dim,
    "hidden_size": hidden_size,
    "data_file": chosen_data_file,
    "avg_rmse": avg_rmse,
    "var_mse": mse_var,
    "std_rmse": delta_meth_stderr,
    "pred_score": types_predict_score,
    "model_timestamp": load_file_timestamp,
    "train_size": train_size,
    "test_size": test_size
}

predict_data
```

    {'model_name': 'HawkesDecayRNN',
     'process_dim': 2,
     'hidden_size': 128,
     'data_file': '../data/simulated/hawkes_sumxp_synthetic_exp_2d_20181210-002610.pkl',
     'avg_rmse': 3.7128522249871843,
     'var_mse': 1464.2632804715774,
     'std_rmse': 0.09859423374146217,
     'pred_score': 0.592,
     'model_timestamp': '20181210-013110',
     'train_size': 1200,
     'test_size': 1000}

```python
load_file_timestamp
```

    '20181210-013110'

```python
import json

predict_res_file = "predict_results_{}d-sumexp.json".format(process_dim)

predict_data.keys()

with open(predict_res_file, "r+", encoding='utf-8', newline='\n') as fp:
    try:
        data = json.load(fp)
    except json.JSONDecodeError:
        data = []
    data.append(predict_data)
    fp.seek(0)
    json.dump(data, fp, indent=4)
    fp.write('\n')
```

```python

```

```python

```

```python

```

```python

```