Table of Contents

1  ROC/AUC for Binary Classification

1.1  Sklearn Transformer

1.2  ROC curves

1.2.1  Implementation

1.2.2  AUC probabilistic interpretation

1.3  Precision Recall Curve

2  Thresholding via Cost

3  Reference

```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir(os.path.join('..', '..', 'notebook_format'))

from formats import load_style
load_style(plot_style=False)
```

```python
os.chdir(path)

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
# 4. magic to enable retina (high resolution) plots
# https://gist.github.com/minrk/3301035
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format='retina'

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, roc_curve, auc
from sklearn.metrics import precision_recall_curve, average_precision_score

%watermark -a 'Ethen' -d -t -v -p numpy,pandas,matplotlib,sklearn
```

    Ethen 2017-09-22 16:18:29

    CPython 3.5.2
    IPython 6.1.0

    numpy 1.13.1
    pandas 0.20.3
    matplotlib 2.0.0
    sklearn 0.18.1

# ROC/AUC for Binary Classification

For this documentation, we'll be working with a human resource dataset. Our goal is to find out the employees that are likely to leave in the future and act upon our findings, i.e. retain them before they choose to leave. This dataset contains 12000 observations and 7 variables, each representing :

- `S` The satisfaction level on a scale of 0 to 1
- `LPE` Last project evaluation by a client on a scale of 0 to 1
- `NP` Represents the number of projects worked on by employee in the last 12 month
- `ANH` Average number of hours worked in the last 12 month for that employee
- `TIC` Amount of time the employee spent in the company, measured in years
- `Newborn` This variable will take the value 1 if the employee had a newborn within the last 12 month and 0 otherwise
- `left` 1 if the employee left the company, 0 if they're still working here. This is our response variable

```python
filename = 'HR.csv'
data = pd.read_csv(filename)
print('dimensions: ', data.shape)
data.head()
```

    dimensions:  (12000, 7)

      S
      LPE
      NP
      ANH
      TIC
      Newborn
      left

      0
      0.38
      0.53
      2
      157
      3
      0
      1

      1
      0.80
      0.86
      5
      262
      6
      0
      1

      2
      0.11
      0.88
      7
      272
      4
      0
      1

      3
      0.72
      0.87
      5
      223
      5
      0
      1

      4
      0.37
      0.52
      2
      159
      3
      0
      1

To train and evaluate the model, we’ll perform a simple train/test split. 80 percent of the dataset will be used to actually train the model, while the rest will be used to evaluate the accuracy of this model, i.e. out of sample error. Note that the best practice is to split it in three ways train/validation/test split.

```python
label_col = 'left'
label = data[label_col].values
data = data.drop(label_col, axis = 1)
print('labels distribution:', np.bincount(label) / label.size)

test_size = 0.2
random_state = 1234
data_train, data_test, y_train, y_test = train_test_split(
    data, label, test_size = test_size, random_state = random_state, stratify = label)
```

    labels distribution: [ 0.83333333  0.16666667]

This probability table tells you that around 16 percent of the employees who became a staff member of yours have left! If those employees are all the ones that are performing well in the company, then this is probably not a good sign. We'll leave the exploratory analysis part to you ...

## Sklearn Transformer

We then convert perform some generic data preprocessing including standardizing the numeric columns and one-hot-encode the categorical columns (the "Newborn" variable is treated as a categorical variable) and convert everything into a numpy array that sklearn expects. This generic preprocessing step is written as a custom sklearn Transformer. You don't have to follow this structure if you prefer your way of doing it.

To roll out our own Transformer a adheres to the sklearn API, we need to

- Ensure that all arguments to the `__init__` method should be explicit: i.e. `*args` or `**kwargs` should be avoided, as they will not be correctly handled within cross-validation routines
- Subclass/Inherit [`BaseEstimator`](http://scikit-learn.org/stable/modules/generated/sklearn.base.BaseEstimator.html) to get some free stuff. It will give us class representations that are more informative when printing the class object. And provides us a `get_params` and `set_params` functions. These functionalities are used in sklearn's methods such as GridSearch and RandomSearch.
- Subclass/Inherit an appropriate class for your task (one of ClassifierMixin, RegressorMixin, ClusterMixin, TransformerMixin). In our case, we will be implementing a Transformer, thus we'll be subclassing [`TransformerMixin`](http://scikit-learn.org/stable/modules/generated/sklearn.base.TransformerMixin.html). For transformer, we need to implement a `.fit` method which fits some stuff on the training data and a `.transform` method that can perform transformation on both the training and test data. Note that we don't need to subclass [`TransformerMixin`](http://scikit-learn.org/stable/modules/generated/sklearn.base.TransformerMixin.html) this to work, but it does give the end-user the idea that this is a Transformer and we get the `.fit_transform` method that does the fitting and transformer on the training data in one shot for free
- In the fit implementation, you'll notice results that were learned during the `.fit` method is stored with a trailing underscore (e.g., self.colnames_). This is a convention used in sklearn so that we can quickly scan the members of an estimator and distinguish which members are fitting during training time

If you would like to read more on this topic. The following two link might be of interest to you.

- [Blog: Creating your own estimator in scikit-learn](http://danielhnyk.cz/creating-your-own-estimator-scikit-learn/)
- [scikit-learn Documentation: Rolling your own estimator](http://scikit-learn.org/dev/developers/contributing.html#rolling-your-own-estimator)

```python
from collections import defaultdict
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.preprocessing import LabelEncoder, OneHotEncoder, StandardScaler

class Preprocess(BaseEstimator, TransformerMixin):
    """
    Generic data preprocessing including:
    - standardize numeric columns
    - one-hot encode categorical columns

    Parameters
    ----------
    num_cols : list
        Numeric column's name

    cat_cols : list
        Categorical column's name

    Attributes
    ----------
    colnames_ : list
        Column name of the transformed numpy array

    label_encode_dict_ : dict of sklearn's LabelEncoder
        LabelEncoder that was used to encode the value
        of the categorical columns into with value between
        0 and n_classes-1. Categorical columns will go through
        this encoding process before being one-hot encoded

    cat_encode_ : sklearn's OneHotEncoder
        OneHotEncoder that was used to one-hot encode the
        categorical columns

    scaler_ : sklearn's StandardScaler
        Scaler that was used to standardize the numeric columns
    """

    def __init__(self, num_cols = None, cat_cols = None):
        self.num_cols = num_cols
        self.cat_cols = cat_cols

    def fit(self, data):
        """
        Fit the Preprocess Transformer

        Parameters
        ----------
        data : DataFrame
        """
        data = data.copy()

        # Label encoding across multiple columns in scikit-learn
        # https://stackoverflow.com/questions/24458645/label-encoding-across-multiple-columns-in-scikit-learn
        if self.cat_cols is not None:
            self.label_encode_dict_ = defaultdict(LabelEncoder)
            label_encoded = (data[self.cat_cols].
                             apply(lambda x: self.label_encode_dict_[x.name].fit_transform(x)))

            self.cat_encode_ = OneHotEncoder(sparse = False)
            self.cat_encode_.fit(label_encoded)

        if self.num_cols is not None:
            self.scaler_ = StandardScaler().fit(data[self.num_cols])

        # store the column names (numeric columns comes before the
        # categorical columns) so we can refer to them later
        if self.num_cols is not None:
            colnames = self.num_cols.copy()
        else:
            colnames = []

        if self.cat_cols is not None:
            for col in self.cat_cols:
                cat_colnames = [col + '_' + str(classes)
                                for classes in self.label_encode_dict_[col].classes_]
                colnames += cat_colnames

        self.colnames_ = colnames
        return self

    def transform(self, data):
        """
        Trasform the data using the fitted Preprocess Transformer

        Parameters
        ----------
        data : DataFrame
        """
        if self.cat_cols is not None:
            label_encoded = (data[self.cat_cols].
                             apply(lambda x: self.label_encode_dict_[x.name].transform(x)))
            cat_encoded = self.cat_encode_.transform(label_encoded)

        if self.num_cols is not None:
            scaled = self.scaler_.transform(data[self.num_cols])

        # combine encoded categorical columns and scaled numerical
        # columns, it's the same as concatenate it along axis 1
        if self.cat_cols is not None and self.num_cols is not None:
            X = np.hstack((scaled, cat_encoded))
        elif self.num_cols is None:
            X = cat_encoded
        else:
            X = scaled

        return X
```

```python
num_cols = ['S', 'LPE', 'NP', 'ANH', 'TIC']
cat_cols = ['Newborn']

preprocess = Preprocess(num_cols, cat_cols)
X_train = preprocess.fit_transform(data_train)
X_test = preprocess.transform(data_test)

print('colnames', preprocess.colnames_)
X_train
```

    colnames ['S', 'LPE', 'NP', 'ANH', 'TIC', 'Newborn_0', 'Newborn_1']

    array([[ 0.24997745,  0.61402599,  0.16885833, ...,  0.72182646,
             1.        ,  0.        ],
           [ 0.12545568, -0.27568766,  1.02655144, ..., -0.22007024,
             1.        ,  0.        ],
           [ 0.58203549, -1.34334405,  0.16885833, ..., -0.22007024,
             1.        ,  0.        ],
           ...,
           [-1.32729826, -0.75020161, -0.68883478, ..., -0.22007024,
             1.        ,  0.        ],
           [ 0.58203549, -0.21637342, -0.68883478, ..., -1.16196694,
             0.        ,  1.        ],
           [ 0.45751372,  0.25814053, -0.68883478, ..., -0.22007024,
             1.        ,  0.        ]])

```python
# pick your favorite classfication model
tree = RandomForestClassifier(max_depth = 4)
tree.fit(X_train, y_train)
```

    RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
                max_depth=4, max_features='auto', max_leaf_nodes=None,
                min_impurity_split=1e-07, min_samples_leaf=1,
                min_samples_split=2, min_weight_fraction_leaf=0.0,
                n_estimators=10, n_jobs=1, oob_score=False, random_state=None,
                verbose=0, warm_start=False)

After training our model, we need to evaluate whether its any good or not and the most straightforward and intuitive metric for a supervised classifier's performance is accuracy. Unfortunately, there are circumstances where simple accuracy does not work well. For example, with a disease that only affects 1 in a million people, a completely bogus screening test that always reports "negative" will be 99.9999% accurate. Unlike accuracy, ROC curves are less sensitive to class imbalance; the bogus screening test would have an AUC of 0.5, which is like not having a test at all.

## ROC curves

**ROC curve (Receiver Operating Characteristic)** is a commonly used way to visualize the performance of a binary classifier and AUC (Area Under the ROC Curve) is used to summarize its performance in a single number. Most machine learning algorithms have the ability to produce probability scores that tells us the strength in which it thinks a given observation is positive. Turning these probability scores into yes or no predictions requires setting a threshold; cases with scores above the threshold are classified as positive, and vice versa. Different threshold values can lead to different result:

- A higher threshold is more conservative about labeling a case as positive; this makes it less likely to produce false positive (an observation that has a negative label but gets classified as positive by the model) results but more likely to miss cases that are in fact positive (lower true positive rate)
- A lower threshold produces positive labels more liberally, so it creates more false positives but also generate more true positives

A quick refresher on terminology:

\begin{align}
[\text{true positive rate}]
&= \frac{[\text{# positive data points with positive predictions}]}{\text{[# all positive data points]}} \\
&= \frac{[\text{# true positives}]}{[\text{# true positives}] + [\text{# false negatives}]}
\end{align}

true positive rate is also known as **recall** or **sensitivity**

\begin{align}
[\text{false positive rate}]
&= \frac{[\text{# positive data points with positive predictions}]}{\text{[# all negative data points]}} \\
&= \frac{[\text{# false positives}]}{[\text{# false positives}] + [\text{# true negatives}]}
\end{align}

The ROC curve is created by plotting the true positive rate (when it's actually a yes, how often does it predict yes?) on the y axis against the false positive rate (when it's actually a no, how often does it predict yes?) on the x axis at various cutoff settings, giving us a picture of the whole spectrum of the trade-off we're making between the two measures.

If all these true/false positive terminology is confusing to you, consider reading the material at the following link. [Blog: Simple guide to confusion matrix terminology](http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/)

### Implementation

There are packages to plot ROC curves and to compute metrics from them, but it can still be worthwhile to work through how these curves are calculated from scratch to try and understand better what exactly are they showing us.

```python
def _binary_clf_curve(y_true, y_score):
    """
    Calculate true and false positives per binary classification
    threshold (can be used for roc curve or precision/recall curve);
    the calcuation makes the assumption that the positive case
    will always be labeled as 1

    Parameters
    ----------
    y_true : 1d ndarray, shape = [n_samples]
        True targets/labels of binary classification

    y_score : 1d ndarray, shape = [n_samples]
        Estimated probabilities or scores

    Returns
    -------
    tps : 1d ndarray
        True positives counts, index i records the number
        of positive samples that got assigned a
        score >= thresholds[i].
        The total number of positive samples is equal to
        tps[-1] (thus false negatives are given by tps[-1] - tps)

    fps : 1d ndarray
        False positives counts, index i records the number
        of negative samples that got assigned a
        score >= thresholds[i].
        The total number of negative samples is equal to
        fps[-1] (thus true negatives are given by fps[-1] - fps)

    thresholds : 1d ndarray
        Predicted score sorted in decreasing order

    References
    ----------
    Github: scikit-learn _binary_clf_curve
    - https://github.com/scikit-learn/scikit-learn/blob/ab93d65/sklearn/metrics/ranking.py#L263
    """

    # sort predicted scores in descending order
    # and also reorder corresponding truth values
    desc_score_indices = np.argsort(y_score)[::-1]
    y_score = y_score[desc_score_indices]
    y_true = y_true[desc_score_indices]

    # y_score typically consists of tied values. Here we extract
    # the indices associated with the distinct values. We also
    # concatenate a value for the end of the curve
    distinct_indices = np.where(np.diff(y_score))[0]
    end = np.array([y_true.size - 1])
    threshold_indices = np.hstack((distinct_indices, end))

    thresholds = y_score[threshold_indices]
    tps = np.cumsum(y_true)[threshold_indices]

    # (1 + threshold_indices) = the number of positives
    # at each index, thus number of data points minus true
    # positives = false positives
    fps = (1 + threshold_indices) - tps
    return tps, fps, thresholds
```

```python
# we'll work with some toy data so it's easier to
# show and confirm the calculated result
y_true = np.array([1, 0, 1, 0, 1])
y_score = np.array([0.45, 0.4, 0.35, 0.35, 0.8])

tps, fps, thresholds = _binary_clf_curve(y_true, y_score)
print('thresholds:', thresholds)
print('true positive count:', tps)
print('false positive count:', fps)
```

    thresholds: [ 0.8   0.45  0.4   0.35]
    true positive count: [1 2 2 3]
    false positive count: [0 0 1 2]

From the result above, we can see that the function will compute the true/false positive count for all unique threshold in the predicted score `y_score`. We can validate the result by hand to confirm that the calculation this in fact correct.

Recall that ROC curve plots that true positive rate on the y-axis and false positive rate on the x-axis. Thus all we need to do is to convert the count into rate and we have our ROC curve.

```python
# convert count to rate, append 0 to
# both true positive and false positive
# so the visualization will start from origin (0, 0)
tpr = np.hstack((0, tps / tps[-1]))
fpr = np.hstack((0, fps / fps[-1]))
print('true positive rate:', tpr)
print('false positive rate:', fpr)

plt.rcParams['figure.figsize'] = 8, 6
plt.rcParams['font.size'] = 12

fig = plt.figure()
plt.plot(fpr, tpr, marker = 'o', lw = 1)
plt.xlabel('false positive rate')
plt.ylabel('true positive rate')
plt.title('Receiver Operator Characteristic')
plt.show()
```

Now to calculate the AUC (Area Under the Curve) for the ROC curve, we need sum up the rectangular area and the triangular area under the curve. Depicted by the visualization below:

- For the rectangular area (the plot on the left illustrates one of them), the height are the TPR (true positive rate) and widths are in the difference in the FPR (false positive rate), so the total area of all the rectangles is the dot product of TPR and FPR's differences
- For the triangular area (the plot on the right illustrates one of them), the height are the difference in TPR (true positive rate) and widths are in the difference in the FPR (false positive rate), so the total area of all the rectangles is the dot product of both TPR's and FPR's difference. But only half the area of each rectangle is below its segment of the ROC curve, thus we divide the rectangle by 2 to obtain the triangular area

```python
import matplotlib.patches as patches

fig, ax = plt.subplots(1, 2, figsize = (10, 6))
fig.suptitle('Receiver Operator Characteristic', y = 1.02)

# this part is hard-coded for illustration purpose
fpr_diff = fpr[3] - fpr[0]
tpr_diff = tpr[3] - tpr[0]
rect1 = patches.Rectangle(xy = (0, 0), width = fpr_diff,
                          height = tpr_diff, alpha = 0.3)
ax[0].add_patch(rect1)

fpr_diff = fpr[-1] - fpr[-2]
tpr_diff = tpr[-1] - tpr[-2]
rect2 = patches.Rectangle(xy = (fpr[-2], tpr[-2]), width = fpr_diff,
                          height = tpr_diff, alpha = 0.3)
ax[1].add_patch(rect2)

for i in range(len(ax)):
    ax[i].plot(fpr, tpr, marker = 'o', lw = 1)
    ax[i].set_xlabel('false positive rate')
    ax[i].set_ylabel('true positive rate')

plt.tight_layout()
plt.show()
```

```python
def _roc_auc_score(y_true, y_score):
    """
    Compute Area Under the Curve (AUC) from prediction scores

    Parameters
    ----------
    y_true : 1d ndarray, shape = [n_samples]
        True targets/labels of binary classification

    y_score : 1d ndarray, shape = [n_samples]
        Estimated probabilities or scores

    Returns
    -------
    auc : float
    """

    # ensure the target is binary
    if np.unique(y_true).size != 2:
        raise ValueError('Only two class should be present in y_true. ROC AUC score '
                         'is not defined in that case.')

    tps, fps, _ = _binary_clf_curve(y_true, y_score)

    # convert count to rate
    tpr = tps / tps[-1]
    fpr = fps / fps[-1]

    # compute AUC using the trapezoidal rule;
    # appending an extra 0 is just to ensure the length matches
    zero = np.array([0])
    tpr_diff = np.hstack((np.diff(tpr), zero))
    fpr_diff = np.hstack((np.diff(fpr), zero))
    auc = np.dot(tpr, fpr_diff) + np.dot(tpr_diff, fpr_diff) / 2
    return auc
```

```python
auc_score = _roc_auc_score(y_true, y_score)
print('auc score:', auc_score)

# confirm with scikit-learn's result
auc_score = roc_auc_score(y_true, y_score)
print('package auc socre:', auc_score)
```

    auc score: 0.75
    package auc socre: 0.75

After working through the implementation of ROC curve and AUC score from sratch, we now pull back and visualize:

- The ROC curve of our original model
- Dotted line represents the ROC curve of a purely random classifier and a perfect classifier

```python
# calling the roc_curve, extract the probability of
# the positive class from the predicted probability
tree_test_pred = tree.predict_proba(X_test)[:, 1]
fpr, tpr, thresholds = roc_curve(y_test, tree_test_pred, pos_label = 1)

# AUC score that summarizes the ROC curve
roc_auc = auc(fpr, tpr)

plt.plot(fpr, tpr, lw = 2, label = 'ROC AUC: {:.2f}'.format(roc_auc))
plt.plot([0, 1], [0, 1],
         linestyle = '--',
         color = (0.6, 0.6, 0.6),
         label = 'random guessing')
plt.plot([0, 0, 1], [0, 1, 1],
         linestyle = ':',
         color = 'black',
         label = 'perfect performance')

plt.xlim([-0.05, 1.05])
plt.ylim([-0.05, 1.05])
plt.xlabel('false positive rate')
plt.ylabel('true positive rate')
plt.title('Receiver Operator Characteristic')
plt.legend(loc = "lower right")
plt.tight_layout()
plt.show()
```

The goal of visualizing the  ROC curve is to let us know how well can our classifier be expected to perform in general, at a variety of different baseline probabilities (percentage of the majority class)?

The diagonal line depicts a completely random classifier and ideally our model's ROC curve should be toward the top-left corner and stay as far away from the diagonal line as possible.

Side note: Apart from comparing the model's ROC curve against the ROC curve of a classifier that does random guessing, it's also useful to plot the ROC curve of different classifiers to compare performance against each other.

### AUC probabilistic interpretation

The probabilistic interpretation of the AUC metric is that if we randomly choose a positive case and a negative case, the probability that the positive case outranks the negative case according to the classifier's prediction. Hopefully, this is evident from the ROC curve figure, where plot is enumerating all possible combinations of positive and negative cases, and the fraction under the curve comprises of the area where the positive case outranks the negative one. I personally find this interpretation extremely useful when conveying what AUC is measuring to a non-technical audience.

```python
def auc_probability(y_true, y_score, size = 100000):
    """probabilistic interpretation of AUC"""
    labels = y_true.astype(np.bool)
    pos = np.random.choice(y_score[labels], size = size, replace = True)
    neg = np.random.choice(y_score[~labels], size = size, replace = True)
    auc = np.sum(pos > neg) + np.sum(pos == neg) / 2
    auc /= size
    return auc

# we want this be close the score returned
# by roc_auc_score
auc_probability(y_true = y_test, y_score = tree_test_pred)
```

    0.97633000000000003

## Precision Recall Curve

Apart from ROC curve, there is also the **precision recall curve**. Instead of plotting true positive rate (a.k.a recall) versus false positive rate. We now plot precision versus recall.

\begin{align}
[\text{precision}]
&= \frac{[\text{# positive data points with positive predicitions}]}{\text{[# all data points with positive predictions]}} \\
&= \frac{[\text{# true positives}]}{[\text{# true positives}] + [\text{# false positives}]}
\end{align}

```python
tree_test_pred = tree.predict_proba(X_test)[:, 1]
precision, recall, thresholds = precision_recall_curve(
    y_test, tree_test_pred, pos_label = 1)

# AUC score that summarizes the precision recall curve
avg_precision = average_precision_score(y_test, tree_test_pred)

label = 'Precision Recall AUC: {:.2f}'.format(avg_precision)
plt.plot(recall, precision, lw = 2, label = label)
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.title('Precision Recall Curve')
plt.legend()
plt.tight_layout()
plt.show()
```

A classifier with high recall but low precision flags many positive results, but most of its predicted labels are incorrect when compared to its corresponding labels. On the other hand, a classifier with high precision but low recall is just the opposite, returning very few results, but most of its predicted labels are correct when compared to the training labels. An ideal system with high precision and high recall will return many results, with all results labeled correctly.

Precision recall curve answers a fundamentally different question compared to ROC curve. By definition, precision directly answers the question, "What is the probability that this is a real hit given my classifier says it is? Thus it is useful in practice for needle-in-haystack type problems or problems where the "positive" class is more interesting than the negative class.

You can also think about it in the following way. ROC AUC looks at TPR and FPR, the entire confusion matrix for all thresholds. On the other hand, Precision-Recall AUC looks at Precision and Recall (TPR), it doesn't look at True Negative Rate (TNR). Because of that PR AUC can be a better choice when you care only about the "positive" while ROC AUC cares about both "positive" and "negative". Since PR AUC doesn't use TNR directly it can also be better for highly imbalanced problems. You may want to take a look at this [Blog: F1 Score vs ROC AUC vs Accuracy vs PR AUC: Which Evaluation Metric Should You Choose?](https://neptune.ml/blog/f1-score-accuracy-roc-auc-pr-auc).

Although ROC curve is presumably the more popular choice when evaluating binary classifiers, it is highly recommended to use precision recall curve as a supplement to ROC curves to get a full picture when evaluating and comparing classifiers. For more discussion on this topic, consider taking a look at another documentation. [Notebook: Evaluating Imbalanced Datasets](http://nbviewer.jupyter.org/github/ethen8181/machine-learning/blob/master/model_selection/imbalanced/imbalanced_metrics.ipynb)

# Thresholding via Cost

Lots of real world binary classification needs a threshold to convert the model into a business decision. i.e. All cases with a model score above the threshold get some sort of special treatment. For example:

- **Fraud Prevention**: We're a social network company and we'd like to delete fake accounts. We build a classifier that assigns a "fraud score" between 0 and 1 to each account and, after some research, decide that all accounts whose score is above 0.9 should be sent to our fraud team, which will review each case and delete the accounts that are actually fake
- **Response/Propensity Modeling**: We're at an enterprise software company and we'd like to improve our outbound sales program. We buy a large database of potential customers and build a classifier to predict which ones are likely buy our product if contacted by our sales team. We decide that all customers with a "response score" of above threshold 0.7 should get a call
- **Shopping Cart Abandonment**: We're at an e-commerce company, and we'd like to email a 10% off coupon to users who abandoned their shopping carts and won't return organically. (We don't want to give a 10% off coupon to users who are going to return anyway, since then you'd be losing 10% of the sale price.) We build a classifier to predict which users will never return to their carts, and decide that all users with a "abandonment score" above 0.85 should get the 10% off coupon
- Etc.

Thresholding is popular because of its simplicity and ease of implementation: We translate a continuous score to a binary yes/no decision, and act on it in a predetermined way. The biggest question for the thresholding pattern is: Where should I set the threshold point?

Up until this point, we've been using AUC to give us a single-number summary of classifier performance. This might be suitable in some circumstances, but for binary classifiers, evaluation metrics that take into account the actual costs of false positive and false negative errors may be much more appropriate than AUC. If we know these costs, we can use them not only to tie the evaluation metric more directly to the business value but also choose an appropriate final cutoff threshold for the classifier.

In real world application, the cost that comes along with making these two mistakes (false positive and false negative) are usually a whole lot different. Take our case for example, a false negative (FN) means an employee left our company but our model failed to detect that, while a false positive (FP) means an employee is still currently working at our company and our model told us that they will be leaving. The former mistake would be a tragedy, since, well, the employee left and we didn't do anything about it! As for conducting the latter mistake, we might be wasting like 20 minutes of a HR manager's time when we arrange a face to face interview with a employee, questioning about how the company can do better to retain him/her, while he/she is perfectly fine with the current situation.

In the code cell below, we assign a cost for a false negative (FN) and false positive (FP) to be 100 and 1000 respectively, given the cost associated with the two mistakes we will multiply them with the false negative and false positive rate at each threshold to figure out where's the best cutoff value. Note that the cost associated with the mistake can just be a back of the envelope number as long as we're sure about which one is more "expensive" than the other.

```python
fpr, tpr, thresholds = roc_curve(y_test, tree_test_pred, pos_label = 1)
fnr = tpr[-1] - tpr

# fn = false negative, meaning the employee left and we
# didn't do anything about it, in our case this will be
# substantially more expensive than a false positive
fp_cost = 100
fn_cost = 1000

costs = (fpr * fp_cost + fnr * fn_cost) * y_test.size
```

```python
best = np.argmin(costs)
best_cost = np.round(costs[best])
best_threshold = np.round(thresholds[best], 2)

plt.plot(thresholds, costs, label = 'cost ($)')
label = 'best cost: {} at threshold {}'.format(best_cost, best_threshold)
plt.axvline(best_threshold, linestyle = '--', lw = 2, color = 'black', label = label)
plt.xlabel('threshold')
plt.ylabel('$')
plt.title('Cost as a Function of Threshold')
plt.legend()
plt.tight_layout()
plt.show()
```

Just to hit the notion home, when executing on a project, if we are able to compute expected cost for each mistake, consider maximizing that directly instead of AUC or another general-purpose metrics.

# Reference

- [Scikit-learn Documentation: Precision Recall](http://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html)
- [StackExchange: ROC vs precision-and-recall curves](https://stats.stackexchange.com/questions/7207/roc-vs-precision-and-recall-curves)
- [Blog: Calculating AUC: the area under a ROC Curve](http://blog.revolutionanalytics.com/2016/11/calculating-auc.html)
- [Blog: F1 Score vs ROC AUC vs Accuracy vs PR AUC: Which Evaluation Metric Should You Choose?](https://neptune.ml/blog/f1-score-accuracy-roc-auc-pr-auc)
- [Blog: Machine Learning Meets Economics](http://blog.mldb.ai/blog/posts/2016/01/ml-meets-economics/)
- [Blog: Visualizing Machine Learning Thresholds to Make Better Business Decisions](http://blog.insightdatalabs.com/visualizing-classifier-thresholds/)


```python
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sns
sns.set(color_codes=True)
sns.set(font='SimHei')
#plt.rcParams['axes.grid'] = False

import numpy as np

import pandas as pd
pd.options.display.max_rows = 10

from IPython.display import Image
```

决策树原理与实现简介
=================

### 前言

为什么讲决策树?

1. 原理简单，直白易懂。
2. 可解释性好。
3. 变种在工业上应用多：随机森林、GBDT。

深化拓展

1. 理论，考古：ID3, C4.5, CART
2. 工程，实现细节：
   + demo
   + scikit-learn
   + spark
   + xgboost
3. 应用，调参分析
4. 演示

### 理论

算法：

1. ID3
2. C4.5
3. C5.0
4. **CART**
5. CHAID
6. MARS

#### 行业黑话

+ 分类问题 vs 回归问题

+ 样本 = （特征$x$，真实值$y$）

+ 目的：找到模型$h(\cdot)$，使得预测值$\hat{y} = h(x)$ $\to$ 真实值$y$

```python
from sklearn.datasets import load_iris
data = load_iris()

# 准备特征数据
X = pd.DataFrame(data.data,
                 columns=["sepal_length", "sepal_width", "petal_length", "petal_width"])

# 准备标签数据
y = pd.DataFrame(data.target, columns=['target'])
y.replace(to_replace=range(3), value=data.target_names, inplace=True)

# 组建样本 [特征，标签]
samples = pd.concat([X, y], axis=1, keys=["x", "y"])
samples.head(5)
```

      x
      y

      sepal_length
      sepal_width
      petal_length
      petal_width
      target

      0
      5.1
      3.5
      1.4
      0.2
      setosa

      1
      4.9
      3.0
      1.4
      0.2
      setosa

      2
      4.7
      3.2
      1.3
      0.2
      setosa

      3
      4.6
      3.1
      1.5
      0.2
      setosa

      4
      5.0
      3.6
      1.4
      0.2
      setosa

```python
samples["y", "target"].value_counts()
```

    versicolor    50
    virginica     50
    setosa        50
    Name: (y, target), dtype: int64

```python
samples["x"].describe()
```

      sepal_length
      sepal_width
      petal_length
      petal_width

      count
      150.000000
      150.000000
      150.000000
      150.000000

      mean
      5.843333
      3.054000
      3.758667
      1.198667

      std
      0.828066
      0.433594
      1.764420
      0.763161

      min
      4.300000
      2.000000
      1.000000
      0.100000

      25%
      5.100000
      2.800000
      1.600000
      0.300000

      50%
      5.800000
      3.000000
      4.350000
      1.300000

      75%
      6.400000
      3.300000
      5.100000
      1.800000

      max
      7.900000
      4.400000
      6.900000
      2.500000

#### 三分钟明白决策树

```python
Image(url="https://upload.wikimedia.org/wikipedia/commons/f/f3/CART_tree_titanic_survivors.png")
```

```python
Image(url="http://scikit-learn.org/stable/_images/iris.svg")
```

```python
Image(url="http://scikit-learn.org/stable/_images/sphx_glr_plot_iris_0011.png")
```

```python
samples = pd.concat([X, y], axis=1)
samples.head(3)
```

      sepal_length
      sepal_width
      petal_length
      petal_width
      target

      0
      5.1
      3.5
      1.4
      0.2
      setosa

      1
      4.9
      3.0
      1.4
      0.2
      setosa

      2
      4.7
      3.2
      1.3
      0.2
      setosa

### 工程

#### Demo实现

其主要问题是在每次决策时找到一个分割点，让生成的子集尽可能地纯净。这里涉及到四个问题:

1. 如何分割样本？
2. 如何评价子集的纯净度？
3. 如何找到单个最佳的分割点，其子集最为纯净？
4. 如何找到最佳的分割点序列，其最终分割子集总体最为纯净？

```python
Image(url="https://upload.wikimedia.org/wikipedia/commons/f/f3/CART_tree_titanic_survivors.png")
```

#### 1.0 如何分割样本
决策树的分割方法是取一个特征 $f$ 和阈值 $t$，以此为界将样本 $X$ 拆分为两个子集 $X_l, X_r$。其数学表达形同：

\begin{align}
    X = \begin{cases}
        X_l, \ \text{if } X[f] < t \\
        X_r, \ \text{if } X[f] \geq t
    \end{cases}
\end{align}

```python
def splitter(samples, feature, threshold):
    # 按特征 f 和阈值 t 分割样本

    left_nodes = samples.query("{f} < {t}".format(f=feature, t=threshold))
    right_nodes = samples.query("{f} >= {t}".format(f=feature, t=threshold))

    return {"left_nodes": left_nodes, "right_nodes": right_nodes}
```

```python
split = splitter(samples, "sepal_length", 5)

# 左子集
x_l = split["left_nodes"].loc[:, "target"].value_counts()
x_l
```

    setosa        20
    versicolor     1
    virginica      1
    Name: target, dtype: int64

```python
# 右子集
x_r = split["right_nodes"].loc[:, "target"].value_counts()
x_r
```

    versicolor    49
    virginica     49
    setosa        30
    Name: target, dtype: int64

#### 2. 如何评价子集的纯净度？

常用的评价函数正是计算各标签 $c_k$ 在子集中的占比 $p_k = c_k / \sum (c_k)$，并通过组合 $p_k$ 来描述占比集中或分散。

```python
def calc_class_proportion(node):
    # 计算各标签在集合中的占比

    y = node["target"]
    return y.value_counts() / y.count()
```

```python
calc_class_proportion(split["left_nodes"])
```

    setosa        0.909091
    versicolor    0.045455
    virginica     0.045455
    Name: target, dtype: float64

```python
calc_class_proportion(split["right_nodes"])
```

    versicolor    0.382812
    virginica     0.382812
    setosa        0.234375
    Name: target, dtype: float64

主要的评价函数有三种，它们评价的是集合的不纯度（值越大，集合越混杂）。

先做些数学定义以便于描述：
假设对于集合 $m$ 有 $N_m$ 个样本，可分割成 $R_m$ 子集。
若总的标签类别有 $K$ 种，则标签 $k$ 在此集合中的占比为：

\begin{equation}
    \hat{p}_{m k} = \frac{1}{N_m} \displaystyle \sum_{x_i \in R_m} I(y_i = k)
\end{equation}

且令标签 $k$ 是占比最大的标签，即 $k(m) = \operatorname{arg max}_k \hat{p}_{m k}$.

##### 1. Misclassification error
我们一般把集合的分类结果定义为占比最大的标签，那么落在此集合中的其它标签就是误分类。其比率是 $1 - \hat{p}_{m k}(m)$.

#### scikit-learn实现简介

### 应用

评价：过拟合

如何模型解释

参数的含义

### 演示

使用sklearn的决策树，进行一个小样本的分析

```python

```


```python
# %load ../../../preconfig.py
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sns
sns.set(color_codes=True)
#sns.set(font='SimHei')
#plt.rcParams['axes.grid'] = False

import numpy as np

import pandas as pd
pd.options.display.max_rows = 20

#import sklearn

#import itertools

import logging
logger = logging.getLogger()
```

决策树简介和 Python 实现
======================
参考：

+ [Building a decision tree from scratch - a beginner tutorial](http://www.patricklamle.com/Tutorials/Decision%20tree%20python/tuto_decision%20tree.html)

+ 9.2 Tree-Based Methods - The Elements of Statistical Learning

+ [Classification and Regression Trees (CART) Theory and Applications](http://edoc.hu-berlin.de/master/timofeev-roman-2004-12-20/PDF/timofeev.pdf)

#### 0. 基本介绍
本文主要是参照 Tree-Based Methods - The Elements of Statistical Learning 来实现一个简化版范例，其算法是 CART。

决策树的思想本身非常朴素，关于它的基本介绍在网上已经非常丰富，比如：

+ [算法杂货铺——分类算法之决策树(Decision tree)](http://www.cnblogs.com/leoo2sk/archive/2010/09/19/decision-tree.html)

其主要问题是在每次决策时找到一个分割点，让生成的子集尽可能地纯净。这里涉及到四个问题:

1. 如何分割样本？

2. 如何评价子集的纯净度？

3. 如何找到单个最佳的分割点，其子集最为纯净？

4. 如何找到最佳的分割点序列，其最终分割子集总体最为纯净？

接下来，围绕上述问题，一一概要说明，并加以演示。

#### 加载数据

古话说，「三军未动，粮草先行」。

我们先加载演示数据，使用的是 sklearn 自带的测试用例。

```python
from sklearn.datasets import load_iris
data = load_iris()
```

```python
# 准备特征数据
X = pd.DataFrame(data.data,
                 columns=["sepal_length", "sepal_width", "petal_length", "petal_width"])
X.head(2)
```

      sepal_length
      sepal_width
      petal_length
      petal_width

      0
      5.1
      3.5
      1.4
      0.2

      1
      4.9
      3.0
      1.4
      0.2

```python
# 准备标签数据
y = pd.DataFrame(data.target, columns=['target'])
y.replace(to_replace=range(3), value=data.target_names, inplace=True)
y.head(3)
```

      target

      0
      setosa

      1
      setosa

      2
      setosa

```python
# 组建样本 [特征，标签]
samples = pd.concat([X, y], axis=1) #, keys=["x", "y"])
samples.head(3)
```

      sepal_length
      sepal_width
      petal_length
      petal_width
      target

      0
      5.1
      3.5
      1.4
      0.2
      setosa

      1
      4.9
      3.0
      1.4
      0.2
      setosa

      2
      4.7
      3.2
      1.3
      0.2
      setosa

#### 1.0 如何分割样本
决策树的分割方法是取一个特征 $f$ 和阈值 $t$，以此为界将样本 $X$ 拆分为两个子集 $X_l, X_r$。其数学表达形同：

\begin{align}
    X = \begin{cases}
        X_l, \ \text{if } X[f] < t \\
        X_r, \ \text{if } X[f] \geq t
    \end{cases}
\end{align}

```python
def splitter(samples, feature, threshold):
    # 按特征 f 和阈值 t 分割样本

    left_nodes = samples.query("{f} < {t}".format(f=feature, t=threshold))
    right_nodes = samples.query("{f} >= {t}".format(f=feature, t=threshold))

    return {"left_nodes": left_nodes, "right_nodes": right_nodes}
```

```python
split = splitter(samples, "sepal_length", 5)

# 左子集
x_l = split["left_nodes"].loc[:, "target"].value_counts()
x_l
```

    setosa        20
    versicolor     1
    virginica      1
    Name: target, dtype: int64

```python
# 右子集
x_r = split["right_nodes"].loc[:, "target"].value_counts()
x_r
```

    virginica     49
    versicolor    49
    setosa        30
    Name: target, dtype: int64

#### 2. 如何评价子集的纯净度？

从常理来说，我们希望分割子集尽可能地纯净，最好是单个子集就只含有一类标签，从而保证决策结果精准。

那么什么样的评价函数，可以用来度量各子集的纯净度呢？

以刚才计算结果为例， $x_l$ 主要标签是 setosa，非常纯净，而 $x_r$ 则三种标签势均力敌，非常混杂。所以思路是，若一种标签在子集中占比非常大，则此子集就较纯净；若各标签占比差别不大，就较为混杂。

常用的评价函数正是计算各标签 $c_k$ 在子集中的占比 $p_k = c_k / \sum (c_k)$，并通过组合 $p_k$ 来描述占比集中或分散。

```python
def calc_class_proportion(node):
    # 计算各标签在集合中的占比

    y = node["target"]
    return y.value_counts() / y.count()
```

```python
calc_class_proportion(split["left_nodes"])
```

    setosa        0.909091
    versicolor    0.045455
    virginica     0.045455
    Name: target, dtype: float64

```python
calc_class_proportion(split["right_nodes"])
```

    virginica     0.382812
    versicolor    0.382812
    setosa        0.234375
    Name: target, dtype: float64

主要的评价函数有三种，它们评价的是集合的不纯度（值越大，集合越混杂）。

先做些数学定义以便于描述：
假设对于集合 $m$ 有 $N_m$ 个样本，可分割成 $R_m$ 子集。
若总的标签类别有 $K$ 种，则标签 $k$ 在此集合中的占比为：

\begin{equation}
    \hat{p}_{m k} = \frac{1}{N_m} \displaystyle \sum_{x_i \in R_m} I(y_i = k)
\end{equation}

且令标签 $k$ 是占比最大的标签，即 $k(m) = \operatorname{arg max}_k \hat{p}_{m k}$.

##### 1. Misclassification error
我们一般把集合的分类结果定义为占比最大的标签，那么落在此集合中的其它标签就是误分类。其比率是 $1 - \hat{p}_{m k}(m)$.

```python
def misclassification_error(node):
    p_mk = calc_class_proportion(node)

    return 1 - p_mk.max()
```

```python
misclassification_error(split["left_nodes"])
```

    0.090909090909090939

```python
misclassification_error(split["right_nodes"])
```

    0.6171875

对于二分类问题，

```python
binary_class = pd.Series(np.arange(0, 1.01, 0.01)).to_frame(name="p")
binary_class["1-p"] = 1 - binary_class["p"]
binary_class.head(3)
```

      p
      1-p

      0
      0.00
      1.00

      1
      0.01
      0.99

      2
      0.02
      0.98

误分类率和占比 $p$ 的关系可划图为：

```python
binary_class["misclass"] = binary_class.apply(lambda x: 1 - x.max(), axis=1)
binary_class.plot(x="p", y="misclass")
```

当 $p=0.5$，两种标签各占一半，不纯度最高；当 $p=0$ 或 $p=1$, 只含有其中一种标签时，不纯度最低。

##### 2. Gini index
这里的基尼系数并非是经济上测量分配公平的指标。

它的思路是从集合中随机抽取元素 $a \in K_p$，再以 $K_p$ 在集合中的分布为参考随机给 $a$ 分配标签，那么误分配率就是基尼系数。

具体到决策树的节点 $m$ 上，标签 $k_i$ 的占比为 $p_{k_i m}$。则抽中属于标签 $k_i$ 的元素概率是 $p_{k_i m}$，误分配到其它标签的概率是 $\sum_{k' \neq k_i} p_{k_i m} p_{k' m}$。对于整个集合的标签则是：

\begin{equation}
    G(m) = \displaystyle \sum_{k \neq k'} p_{k m} p_{k' m} \, \overset{乘法分配律}{=} \sum_{k = 1}^{K} p_{k m} (1 - p_{k m})
\end{equation}

```python
def gini_index(node):
    p_mk = calc_class_proportion(node)

    return (p_mk * (1 - p_mk)).sum()
```

```python
gini_index(split["left_nodes"])
```

    0.1694214876033058

```python
gini_index(split["right_nodes"])
```

    0.6519775390625

在二分类中，基尼系数和占比 $p$ 的关系可划图为：

```python
binary_class["gini"] = (binary_class["p"] * binary_class["1-p"] * 2)
binary_class.plot(x="p", y="gini")
```

##### 3. Cross-entropy
ref:
[Qualitively what is Cross Entropy](http://stats.stackexchange.com/questions/80967/qualitively-what-is-cross-entropy)

这个损失函数的思路来源于信息论：若某事件的发生概率是 $p$，则需至少 $\log_2 (1/p)$ 位编码。那么对于所有事件，其最优编码的平均字长为 $\sum_i p_i \log_2 (1 / p_i)$。

借用其思路，对于节点来说，其内容越混杂，就需要越多字长来区分。所以这里 cross-entropy 定义为：

\begin{equation}
    C(m) = \displaystyle \sum_{k=1}^K p_{m k} \log (1 / p_{m k}) \, = - \sum_{k=1}^K p_{m k} \log p_{m k}
\end{equation}

```python
def cross_entropy(node):
    p_mk = calc_class_proportion(node)

    return - (p_mk * p_mk.apply(np.log)).sum()
```

```python
cross_entropy(split["left_nodes"])
```

    0.36764947740014225

```python
cross_entropy(split["right_nodes"])
```

    1.075199711851601

在二分类中，cross-entropy 和占比 $p$ 的关系可划图为：

```python
x = binary_class[["p", "1-p"]]
binary_class["cross_entropy"] = -(x * np.log(x)).sum(axis=1)
binary_class.plot(x="p", y="cross_entropy")
```

在二分类问题中，三种评价函数的比较如图:

```python
binary_class.plot(x="p", y=["misclass", "gini", "cross_entropy"])
```

为了便于比较，我们将 cross_entropy 也放缩到 $(0.5, 0.5)$。

```python
binary_class["cross_entropy_scaled"] = binary_class["cross_entropy"] / binary_class["cross_entropy"].max() * 0.5
binary_class.plot(x="p", y=["misclass", "gini", "cross_entropy_scaled"], ylim=[0,0.55])
```

可以看到，识分类率在整个区间是均一的，而 cross_entropy 越靠近纯净，其值变化越剧烈。所以 cross_entropy 对纯净更敏感的特性，有利于让结果子集更纯净，其使用相对较多。

####  3. 如何找到单个最佳的分割点，其子集最为纯净？
单个最佳分割点，涉及三个问题：

1. 对于单次分割，分割前和分割后，集合的纯净度提升了多少？

2. 给定一个特征，纯净度提升最大的阈值是多少？

3. 对于多个特征，哪一个特征的最佳阈值对纯净度提升最大？

##### 3.1 对于单次分割，分割前和分割后，集合的纯净度提升了多少？
令测量不纯度的函数为 $G$,
对一个节点 $m$ 来说，若其按分割方法 $f$ 得到子集 $m_l$ 和 $m_r$，则总的不纯度减少量为：

\begin{equation}
    G(m) - G(m_l) - G(m_r)
\end{equation}

```python
def calc_impurity_measure(node, feathure, threshold, measure, min_nodes=5):
    child = splitter(node, feathure, threshold)
    left = child["left_nodes"]
    right = child["right_nodes"]

    if left.shape[0] <= min_nodes or right.shape[0] <= min_nodes:
        return 0

    impurity = pd.DataFrame([],
                       columns=["score", "rate"],
                       index=[])

    impurity.loc["all"] = [measure(node), node.shape[0]]
    impurity.loc["left"] = [-measure(left), left.shape[0]]
    impurity.loc["right"] = [-measure(right), right.shape[0]]

    impurity["rate"] /= impurity.at["all", "rate"]

    logger.info(impurity)

    return (impurity["score"] * impurity["rate"]).sum()
```

```python
calc_impurity_measure(samples, "sepal_length", 5, gini_index)
```

    0.08546401515151514

```python
calc_impurity_measure(samples, "sepal_length", 1, gini_index)
```

    0

##### 3.2. 给定一个特征，纯净度提升最大的阈值是多少？

对于一个给定的特征，理论上通过枚取所有可能的阈值，从中找到最大减少量的阈值点，就是此特征的最佳分隔点。

但现实中，很多特征是连续的，或者阈值点太多，全部穷尽并不现实，往往需要用到最优化的寻优方法。这里为了简易起见，我们对特征的值由小到大设了10个分位点，进行计算。

```python
def find_best_threshold(node, feature, measure):
    threshold_candidates = node[feature].quantile(np.arange(0, 1, 0.2))

    res = pd.Series([], name=feature)
    for t in threshold_candidates:
        res[t] = calc_impurity_measure(node, feature, t, measure)

    logger.info(res)

    if res.max() == 0:
        return None
    else:
        return res.argmax()
```

```python
find_best_threshold(samples, "sepal_width", gini_index)
```

    3.3999999999999999

```python
find_best_threshold(samples, "sepal_length", gini_index)
```

    5.5999999999999996

##### 3.3. 对于多个特征，哪一个特征的最佳阈值对纯净度提升最大？
显然，最暴力的方法是：每次分割，我们穷尽所有特征，即可找到对此节点最佳分割点

```python
def find_best_split(node, measure):
    if node["target"].unique().shape[0] <= 1:
        return None

    purity_gain = pd.Series([], name="feature")

    for f in node.drop("target", axis=1).columns:
        purity_gain[f] = find_best_threshold(node, f, measure)

    if pd.isnull(purity_gain.max()):
        return None
    else:
        best_split = {"feature": purity_gain.argmax(), "threshold": purity_gain.max()}
        best_split["child"] = splitter(node, **best_split)

        return best_split
```

```python
best_split = find_best_split(samples, gini_index)
[best_split[x] for x in ["feature", "threshold"]]
```

    ['sepal_length', 5.5999999999999996]

####  4. 如何找到最佳的分割点序列，其最终分割子集总体最为纯净？
搜索全局最优解在目前还没有有效的方法，所以退一步，我们用贪婪的思想，在每次分割时取最优，希望由局部最优的分割序列能够达到全局最优的效果。

我们使用递归的方法由上而下依次计算，在处理节点顺序时使用深度优先方法组建出决策树。

```python
class BinaryNode:
    def __init__(self, samples, max_depth, measure=gini_index):
        self.samples = samples
        self.max_depth = max_depth
        self.measure = measure

        self.is_leaf = False
        self.class_ = None

        self.left = None
        self.right = None

        self.best_split = None

    def split(self, depth):
        if depth > self.max_depth:
            self.is_leaf = True
            self.class_ = self.samples["target"].value_counts().argmax()
            return

        best_split = find_best_split(self.samples, self.measure)
        if pd.isnull(best_split):
            self.is_leaf = True
            self.class_ = self.samples["target"].value_counts().argmax()
            return

        self.best_split = best_split
        left = self.best_split["child"]["left_nodes"]
        self.left = BinaryNode(left.drop(best_split["feature"], axis=1), self.max_depth)

        right = self.best_split["child"]["right_nodes"]
        self.right = BinaryNode(right.drop(best_split["feature"], axis=1), self.max_depth)

        # 先序深度优先
        self.left.split(depth+1)
        self.right.split(depth+1)
```

```python
binaryNode = BinaryNode(samples, 3)
binaryNode.split(0)
```

```python
def show(node, depth):
    if node.left:
        show(node.left, depth+1)

    if node.is_leaf:
        print("{}{}".format("\t"*(depth+2), node.class_))
        return
    else:
        print("{}{}: {}".format("\t"*depth,
                                node.best_split["feature"],
                                node.best_split["threshold"]))
    if node.right:
        show(node.right, depth+1)
```

```python
show(binaryNode, 0)
```

    				versicolor
    	sepal_width: 2.8200000000000003
    					setosa
    		petal_length: 1.6
    						setosa
    			petal_width: 0.4
    						setosa
    sepal_length: 5.6
    					versicolor
    		sepal_width: 3.1
    					versicolor
    	petal_length: 4.8
    						versicolor
    			petal_width: 1.8
    						virginica
    		sepal_width: 2.9
    						virginica
    			petal_width: 2.0
    						virginica

观察可知，这颗树是有问题的，如 petal_width: 0.4 下两个叶子的类别均是 setosa。在树生成后，可以通过后续的剪枝操作，让整颗树更精简。这里不再详述。


# Priors: Theano Graph

```python
with gp.G3Model() as model:
    g3 = gp.G3GP(gp.Constant(), gp.SE(x,gp.ARD_L2), gp.Identity(y))
    g3.observed(x_obs,y_obs)
    mu, var, median, I_up, I_down, m1, m2 = g3.compilate_functions(model.vars,x)

model.use_prior()
gp.show_graph(model.logpt)
```

```python
gp.show_graph(model.fastlogp.f)
```

```python
model.use_prior(False)
gp.show_graph(model.logpt)
```

```python
gp.show_graph(model.fastlogp.f)
```

# Mapping

```python
with g3.Model() as model:
    mean = g3.Bias()
    ker = g3.SE(x)
    trans = g3.Identity(y_obs) @ g3.LogShifted(y_obs)
    tgp = g3.TGP(x, mean, ker, trans, noise=True, hidden=y)
    tgp.describe(str(k),'SUNACTIVITY','YEAR')
    tgp.observed(x_obs,y_obs)
    tgp.testing(x_test,y_test)
    tgp.compile()
#gp.plot_gp(gp.find_default(), samples=10)
```

# Transformation

```python
import numpy as np
from scipy.stats import gamma, norm
import matplotlib.pyplot as plt

fig, ax = plt.subplots(1, 1)
a = 2.0
b = 2.0
N = 10000
mean, var, skew, kurt = gamma.stats(a, moments='mvsk')
x = np.linspace(gamma.ppf(0.01, a,scale=b), gamma.ppf(0.99, a,scale=b), N)
rv = gamma(a,scale=b)
ax.plot(x, rv.pdf(x), 'k-', lw=2, label='pdf')
vals = gamma.ppf([0.001, 0.5, 0.999], a)
np.allclose([0.001, 0.5, 0.999], gamma.cdf(vals, a))
r = gamma.rvs(a, scale = b,size=N)
ax.hist(r,50, normed=True, histtype='stepfilled', alpha=0.2)
ax.legend(loc='best', frameon=False)

## transformar a gaussiana

g = r**(1.0/3.0)
s = np.std(g)
m = np.mean(g)
x = np.linspace(min(g), max(g), N)
fig, ax = plt.subplots(1, 1)
ax.plot(x, norm.pdf(x,m,s), 'k-', lw=2, label='pdf')
ax.hist(g,50, normed=True, histtype='stepfilled', alpha=0.2)
ax.legend(loc='best', frameon=False)

plt.show()

```

http://localhost:8888/notebooks/GaussianProcesses/BoxCoxGP/sunspots-GP-vs-TGP.ipynb

```python
hidden_var = np.linspace(-10,5,501)
params = {'ARD_L2_log_': np.array(2.5, dtype=np.float32),
         'COV_SE_log_': np.array([ 2, -3 ], dtype=np.float32),
         'Mean_Constant': np.array(3.0, dtype=np.float32),
         'T_Shift': np.array(0.0, dtype=np.float32),
         'T_Power_log_': np.array(-0.4, dtype=np.float32),
       }
trans, dtrans, dist_gp, dist_tgp = g3.compile_distributions(model.point_to_In(params))
params
```

```python
grilla = np.linspace(-5,15,501)
gp.style_normal()
plt.figure(1,(10,10))

desfase = 5

yyy = np.zeros_like(grilla)
i = 0
for xxx in grilla:
    yyy[i] = dist_gp(np.array([xxx]))
    i+=1
plt.plot(grilla,yyy*50-desfase,label='GaussianDist')

plt.plot(trans(grilla),grilla,label='BoxCoxTrans')

yyy = np.zeros_like(grilla)
i = 0
for xxx in trans(grilla):
    yyy[i] = dist_gp(np.array([xxx]))
    i+=1
plt.plot(yyy*30-desfase,grilla,label='BoxCox(Gaussian)')

i = 0
for xxx in grilla:
    yyy[i] = dtrans(np.array([xxx]))
    i+=1
plt.plot((yyy-np.min(yyy))-desfase,grilla,label='Jacobian(BoxCox)')

i = 0
for xxx in grilla:
    yyy[i] = dist_tgp(np.array([xxx]))
    i+=1
plt.plot(yyy*40-desfase,grilla,label='BoxCox-GaussianDist')

#location_gp = grilla[np.argmax(yyy*100)]
#min_gp = np.min(yyy*100)
#max_gp = trans(np.array([location_gp]))
#plt.plot(np.array([location_gp,location_gp]),np.array([min_gp,max_gp]))

gp.plot('Box-Cox Change of Variable','GP','Box-Cox GP',loc=1,ncol=1)
plt.axis([-5,12,-5,12])
gp.save('change_of_variable.pdf')
```

```python
import theano.sandbox.linalg as sT
trans = tt.log(sT.det(tt.jacobian(g3.mapping.inv(value),value)))
ftrans = th.function([value]+model.point_to_In(init),trans,allow_input_downcast=True)
yyy = np.zeros_like(grilla)
i = 0
for xxx in grilla:
    yyy[i] = ftrans(np.array([xxx]))
    i+=1
plt.plot(grilla,yyy)
```

```python
value,dist = gp.marginal_tgp(np.array([0.0]))
fdist = th.function([value]+gp_model.point_to_In(start_bfgs),dist,allow_input_downcast=True)
grilla = np.linspace(-200,200,301)
yyy = np.zeros_like(grilla)
i = 0
for xxx in grilla:
    yyy[i] = fdist(np.array([xxx]))
    i+=1
plt.plot(grilla,yyy)
```

# Plot Cov

```python
ker=LaplaceKernel(noise,h,np.array([l]))
plt.matshow(ker.cov(x_obs[:,np.newaxis]))
plt.matshow(sp.linalg.inv(ker.cov(x_obs[:,np.newaxis])))
plt.matshow(CovCholesky(ker.cov(x_obs[:,np.newaxis])))
plt.matshow(CovCholesky(ker.cov(x_obs[:,np.newaxis])).T)
```

```python
plt.imshow(Jxx,cmap=cm.seismic,vmax=v,vmin=-v)
```

```python
#Graficar Covarianza
plt.figure(1, figsize=(20,6))
Kxx=InfoMatrix(GaussKer,np.expand_dims(np.arange(-2,2,0.1),1))

plt.subplot(121)
v=np.max(np.abs(Kxx))
plt.imshow(Kxx,cmap=cm.seismic,vmax=v,vmin=-v)
plt.colorbar()

#Graficar Precision
plt.subplot(122)
Jxx=np.linalg.solve(Kxx,np.eye(len(Kxx)))
v=np.max(np.abs(Jxx))
plt.imshow(Jxx,cmap=cm.seismic,vmax=v,vmin=-v)
plt.colorbar()

plt.figure(2, figsize=(20,6))
plt.plot(Kxx[len(Kxx)/2,:],'b')
plt.plot(Jxx[len(Jxx)/2,:],'r')

#plt.axis([550,650,-0.1,14])
#show(np.linalg.norm(Jxx.dot(Kxx)- np.eye(len(Kxx))))
```

```python
_ = plt.plot(gp.compiles['covariance'](**gp.get_params())[0])
_ = plt.plot(gp.compiles['covariance'](**gp.get_params())[len(x)//2])
_ = plt.plot(gp.compiles['covariance'](**gp.get_params())[-1])
```

# GP 2D
http://localhost:8888/notebooks/GaussianProcesses/BoxCoxGP/Presentation-TGP.ipynb

```python
# Parametros grilla
x = np.linspace(0, 50, 51)
y = np.linspace(0,200, 201)

# Parametros GP
l = 30
sigma = 10*np.log(1.5)
gamma_x = 0.0*np.log(2.0/l**2)
gamma_y = 0.8*np.log(2.0/l**2)

x2d, y2d = np.meshgrid(x, y)
xy = np.zeros((len(x)*len(y),2))
for i in range(len(x)):
    for j in range(len(y)):
        xy[i*len(y)+j,:] = x[i],y[j]
x2d = x2d.T
y2d = y2d.T

Kerxy = GaussianKernel(-np.inf,sigma,np.array([gamma_x,gamma_y]))
fxy2d_hidden = Kerxy.sample(xy).reshape((len(x),len(y)))

fig = plt.figure(1,[20,10])
ax = fig.gca(projection='3d')
ax.plot_surface(x2d, y2d, fxy2d_hidden, alpha=0.4,cmap=cm.RdBu_r)
cset = ax.contourf(x2d, y2d, fxy2d_hidden, zdir='z', offset=np.min(fxy2d_hidden), cmap=cm.RdBu_r)
```

# Sympy

```python
%matplotlib inline
import sympy as sy
import seaborn as sb
import matplotlib.pyplot as plt
from IPython.display import display

from sympy import init_printing
from sympy import symbols,diff,log,sign,Abs
from sympy.plotting import plot

init_printing()
plt.rcParams['figure.figsize'] = (20, 10)
#init_session(quiet=True)

x,y = symbols('x,y',real=True)

```

```python
phi = sign(x)*log(Abs(x))
dphi = diff(phi,x)
phi_1 = sy.solve(phi-y,x,domain=sy.S.Reals)
```

```python
display(phi)
display(dphi)
display(phi_1)
```

```python
plot(phi,(x,-2,2))
```

```python
plot(dphi,(x,-2,2))
```

# Widgets

```python
tgp.widget_params()
```

# Traces

```python
traces = tgp.sample_hypers(start=tgp.get_params(), samples=5000,advi=False)
g3.save_trace(traces)
g3.style_seaborn()
traces.varnames.clear()
for v in gp.model.vars:
    traces.varnames.append(v.name)
traces.varnames
g3.traceplot(traces)
datatraces = g3.datatrace(model, traces)
datatraces.describe().T

items_ll = ['niter','ll']
items_mt = ['TGP_Bias_Constant','TGP_BoxCoxShifted_shift','TGP_SE_ARD_L2_Scales']
items_k = ['TGP_BoxCoxShifted_power', 'TGP_Noise_Var','TGP_SE_Var']

g3.plot_datatrace(datatraces,items_mt+items_k)
g3.plot_datatrace(datatraces,items_ll+items_mt)
g3.plot_datatrace(datatraces,items_ll+items_k)

g3.style_seaborn()
tgp.widget_trace(traces)
tgp.plot_tgp(tgp.get_params(), samples=15)
```

```python
items_ll = ['niter','ll']
items_k1 = ['GP_SM1_M','GP_SM1_S_log_','GP_SM1_Var_log_']
items_k2 = ['GP_SM3_M','GP_SM2_S_log_','GP_SM2_Var_log_']
items_m = ['GP_Bias_Constant','GP_Noise_Var_log_']

g3.plot_datatrace(datatraces,items_ll+items_k1)
g3.plot_datatrace(datatraces,items_ll+items_k2)
g3.plot_datatrace(datatraces,items_ll+items_m)

g3.plot_datatrace(datatraces,items_k1+items_k2)
g3.plot_datatrace(datatraces,items_k1+items_m)
g3.plot_datatrace(datatraces,items_k2+items_m)

```

```python
tgp.widget_trace(traces)
tgp.plot_tgp(tgp.get_params(), samples=10)
```

```python

```

```python

```


# Understanding moving time vs ellapsed time, distance and speed/pace calculations
> How the tracking apps analyse the GPS activity data

- toc: false
- badges: true
- comments: true
- author: Marcel Caraciolo
- categories: [calculation, gps, metrics, jupyter, features, tracking]
- image: images/post-lat-lon.png

Sport tracking applications and the social networks are quite popular among the runners nowadays. Everyone wants to perform the best run looking for a better pace, distance or time.  But have you wondered how these metrics are calculated ?

In this post we’ll use gpx-file downloaded from Strava social application, extract, and analyse the gpx data of one single route. We’ll start with extracting the data from the gpx-file into a convenient ``runpandas.Activity`` dataframe. From there we’ll explore the data and try to replicate the stats and graphs that the interface of one most popular running application provides us with.

## Getting the data

Most of the popular tracking apps allow you to download your activity as a gpx or tcx file. In this post, we will analyse a 12km run from [Strava](http://strava.com/). The gpx-file, short for GPS Exchange Format, can usually be obtained by clicking on export. The screenshot below shows you where you can download your gpx-file in Strava. You can download the file used in this article [here](https://github.com/corriporai/pandasrunner/blob/master/_notebooks/data/post-metrics.gpx).

Now, we want to load our gpx data into a ``runpandas.Activity``. Take a look at your start point and end point and make sure everything makes sense (i.e. start time < end time). If not, there might be something wrong with your data, or you might have forgotten about some tracks or segments.

```python
import runpandas
activity = runpandas.read_file('./data/post-metrics.gpx')
print('Start', activity.index[0],'End:', activity.index[-1])
print(activity.iloc[0]['lat'], activity.iloc[-1]['lat'])
```

    Start 0 days 00:00:00 End: 0 days 01:25:27
    -8.045075 -8.036119

The head of the activity (frame) should look like this:

```python
activity.head(8)
```

      cad
      alt
      hr
      lon
      lat

      time

      00:00:00
      84.0
      12.7
      174.0
      -34.893787
      -8.045075

      00:00:04
      84.0
      12.7
      174.0
      -34.893787
      -8.045075

      00:00:06
      84.0
      12.7
      174.0
      -34.893810
      -8.045115

      00:00:08
      84.0
      12.7
      174.0
      -34.893852
      -8.045168

      00:00:10
      84.0
      12.7
      174.0
      -34.893890
      -8.045216

      00:00:13
      84.0
      12.7
      174.0
      -34.893898
      -8.045266

      00:00:16
      85.0
      12.7
      174.0
      -34.893898
      -8.045282

      00:00:19
      86.0
      12.7
      174.0
      -34.893906
      -8.045329

It is important to notice that the time interval between two data points isn't constant. This happens because the device which records the data can't always provide the GPS-data due to connectivity problems or even to hardware limitation. In case of such a failure/limitation the data point is skipped (without an error) and the app will collect the data at the next time interval. So keep in mind that for further analysis we can't assume that the interval between all points is the same.

## Plotting the data

Now with the data loaded, we can explore it with some basic plots. Strava provides some interesting graphs to the runner such as the 2D map (longitude vs latitude) and elevation gain during the activity (altitude vs time). Using native plotting tools we can come to quite similar results.

```python
#hide
from IPython.display import set_matplotlib_formats
set_matplotlib_formats('png')
```

```python
import matplotlib.pyplot as plt
plt.plot(activity['lon'], activity['lat'])
```

```python
axis = activity.plot.area(y='alt')
axis.set_ylim(0,100)
```

## Calculating the total distance

Now let's going through the key metrics of the activity such as distance and speed. A quick look can misguide you into incorrect calculations. The first one is the distance between two LL-points (latitude, longitude) which it isn't a straight line, but spherical.

There are two main approaches to calculate the distance between two points on a spherical surface: the Haversine distance and the Vincenty distance. The two formulas take a different approach on calculating the distance, but this is outside the scoop of this post. The maths behind both approaches can be found [here](https://www.neovasolutions.com/2019/10/04/haversine-vs-vincenty-which-is-the-best/).

Another issue about the distance calculation is if the data have been calculated solely based on latitude and longitude, without taking into consideration the elevation gain or loss, then the distance covered can be underestimated. The easiest way to do this, is to compute the spherical 2d distance and then use the Euclidean formula to add the altitude measurements. The formula below shows this last step.

If the uncorrected distance covered at time point $t_{i}$ is $d_{2,i}$, to correct the distance covered between time point $t_{i-1}$ and time point $t_{i}$ to
\begin{equation}
d_{i} - d_{i-1} = \sqrt{ \left(d_{2,i} - d_{2,i-1} \right)^{\!\!2} + \left(a_{i} - a_{i-1} \right)^{\!\!2}}
\end{equation}
,
where $d_{i}$ and $a_{i}$ are the corrected cumulative distance and the altitude at time $t_{i}$, respectively.

Now with the theoretical background presented, we can start implementing these formulas in our code. We will apply in all data points all possible implementations of our distance formula (Haversine or Vicenty and 2d or 3d) and compute the total distance for every data point by using the dataframe function `cumsum`.

There are geodesic python libraries already available with Vicenty and Haversine formulas, so we will import them here for our distance calculations.

```python
#pip install haversine & pip install vincenty
from vincenty import vincenty as vn
from haversine import haversine as hv
from haversine import Unit
import numpy as np
```

```python
activity['shift_lon'] = activity.shift(1)['lon']
activity['shift_lat'] = activity.shift(1)['lat']
#compute the vincenty 2d distance
activity['distance_vin_2d'] = activity[['lat', 'lon', 'shift_lat', 'shift_lon']].apply(lambda x: vn( (x[0], x[1]),(x[2], x[3])), axis=1)
#transform the value in meters
activity['distance_vin_2d'] = activity['distance_vin_2d'] * 1000
#compute the haversine 2d distance
activity['distance_hav_2d'] = activity[['lat', 'lon', 'shift_lat', 'shift_lon']].apply(lambda x: hv((x[0], x[1]),(x[2], x[3]), unit=Unit.METERS), axis=1)
#compute the total distance
activity['distance_vin_no_alt'] = activity['distance_vin_2d'].cumsum()
activity['distance_hav_no_alt'] = activity['distance_hav_2d'].cumsum()
#compute the vincenty and haversine 3d
activity['shift_alt'] = activity.shift(1)['alt']
activity['alt_dif'] = activity[['alt', 'shift_alt']].apply(lambda x: x[1] - x[0], axis=1)
activity['distance_vin_3d'] = activity[['distance_vin_2d', 'alt_dif']].apply(lambda x: np.sqrt(x[0]**2 + (x[1])**2), axis=1)
activity['distance_hav_3d'] = activity[['distance_hav_2d', 'alt_dif']].apply(lambda x: np.sqrt(x[0]**2 + (x[1])**2), axis=1)

#compute the total distances for vincenty and haversined 3D
activity['distance_vin'] = activity['distance_vin_3d'].cumsum()
activity['distance_hav'] = activity['distance_hav_3d'].cumsum()

activity.drop(['shift_lon', 'shift_lat', 'shift_alt'], axis=1)
#present the results
activity[['distance_vin', 'distance_hav', 'distance_hav_3d', 'distance_vin_3d']]
```

      distance_vin
      distance_hav
      distance_hav_3d
      distance_vin_3d

      time

      00:00:00
      NaN
      NaN
      NaN
      NaN

      00:00:04
      0.000000
      0.000000
      0.000000
      0.000000

      00:00:06
      5.099000
      5.118162
      5.118162
      5.099000

      00:00:08
      12.568000
      12.609153
      7.490991
      7.469000

      00:00:10
      19.330000
      19.390883
      6.781730
      6.762000

      ...
      ...
      ...
      ...
      ...

      01:25:17
      12544.272150
      12564.172393
      6.437592
      6.421000

      01:25:21
      12559.222485
      12579.203834
      15.031441
      14.950334

      01:25:23
      12564.868485
      12584.874982
      5.671148
      5.646000

      01:25:25
      12570.232485
      12590.267705
      5.392723
      5.364000

      01:25:27
      12575.313485
      12595.361143
      5.093437
      5.081000

1697 rows × 4 columns

For futher convenice, we can extract the data in our previously created activity. Let's check the results with the following print comand.

```python
print('Vicenty 2D: ', activity['distance_vin_no_alt'].max())
print('Haversine 2D: ', activity['distance_hav_no_alt'].max())
print('Vincenty 3D: ', activity['distance_vin'].max())
print('Haversine 3D: ', activity['distance_hav'].max())
```

    Vicenty 2D:  12574.600999999993
    Haversine 2D:  12594.649752172338
    Vincenty 3D:  12575.313484827602
    Haversine 3D:  12595.361142855156

If we compare the distance calculations between the app and ours, we will see they are almost similar.

But why do we have 50m of difference between our internal calculations and the distance showed in the app ?  It is explained in an [article](https://support.strava.com/hc/en-us/articles/216919487-How-Distance-is-Calculated) from Strava's support team.

A flat surface is assumed, and vertical speed from topography is not accounted for. — Strava

In this scenario our 2d calculations are right and the we might conclude the app doesn’t take elevation into account. The difference between the distance proposed by the app and our estimate is about 50m (0.03%). Note that this difference will increase if you undertake more altitude-intense activities (mountain biking or hiking).

Now let's going forward to our next metric, the total activity time ellapsed x moving time.

## Calculating the time ellapsed x moving time

```python
def strfdelta(tdelta, fmt):
    d = {"days": tdelta.days}
    d["hours"], rem = divmod(tdelta.seconds, 3600)
    d["minutes"], d["seconds"] = divmod(rem, 60)
    return fmt.format(**d)

print('Total time: ' , strfdelta(activity.index[-1], "{hours} hour {minutes} min {seconds} sec"))
```

    Total time:  1 hour 25 min 27 sec

The total time elapsed totally agrees with our calculations, but the moving time seems to be diferent. Let us explain the basic concepts: **Elapsed time** is the duration from the moment you hit start on your device or phone to the moment you finish the activity. It includes stoplights, coffee breaks, bathroom stops and stopping for photos. **Moving time**, on the other hand, is a measure of how long you were active, this can be realistic when we have to slow down and stop for a traffic light, for example.

Let’s see if we can figure out which threshold Strava uses to stop the timer (and therefore boost our average speed). To do so, we need to create a new variable that calculates our movement in meters per second (and not just movement per data point, hence why we created the time difference variable). Let’s do this for our haversine 2d distance, since that’s the closest approximation of the distance proposed by the app.

```python
import pandas as pd
import numpy as np
activity['time'] = activity.index
activity['time_dif'] = (activity.time - activity.time.shift(1).fillna(activity.time.iloc[0]))/np.timedelta64(1,'s')
activity['distance_dif_per_sec'] = activity['distance_hav_2d'] / activity['time_dif']
activity['distance_dif_per_sec']
```

    time
    00:00:00         NaN
    00:00:04    0.000000
    00:00:06    2.559081
    00:00:08    3.745496
    00:00:10    3.390865
                  ...
    01:25:17    3.218796
    01:25:21    3.757777
    01:25:23    2.835574
    01:25:25    2.696362
    01:25:27    2.546719
    Name: distance_dif_per_sec, Length: 1697, dtype: float64

With this new variable we can iterate through a list of thresholds. Let's assume values between 50 cm and 1 meter, and try to evaluate which one adds up to a timer time-out closest to 640 seconds (~=10 minutes).

```python
for treshold in [0.5, 0.6, 0.7, 0.8, 0.9, 1]:
    print(treshold, 'm', ' : Time:',
          sum(activity[activity['distance_dif_per_sec'] < treshold]['time_dif']),
         ' seconds')
```

    0.5 m  : Time: 574.0  seconds
    0.6 m  : Time: 577.0  seconds
    0.7 m  : Time: 640.0  seconds
    0.8 m  : Time: 640.0  seconds
    0.9 m  : Time: 640.0  seconds
    1 m  : Time: 640.0  seconds

As we can see at the table above, the movement per second was less than 80-70 centimeters, the application didn't consider it as movement and discard those intervals (it's about 2.9 km/h , a speed far below that most people do in their walking pace). Since we don't have the algorithm used for the real calculation in the app,  we can get to an approximate moving time.

```python
total_time = activity['time_dif'].sum()
stopped_time =  sum(activity[activity['distance_dif_per_sec'] < 0.8]['time_dif'])
pd.Timedelta(seconds=total_time - stopped_time)

```

    Timedelta('0 days 01:14:47')

### Calculating the speed and pace

With the moving time and distance calculated, we can now calculate the pace and speed. **Speed** is calculated by dividing the distance traveled in meters by the time it took in seconds, and then converted to km/h.

```python
activity['speed'] = (activity['distance_hav_2d'] / activity
['time_dif']) * 3.6
```

Next we will filter out all the data where the movement per second is larger than 80 centimeters,  based on the threshold we evaluated above.

```python
activity_with_timeout = activity[activity['distance_dif_per_sec'] > 0.8]
```

Finally, we compute the weighted average speed and convert it to minutes and seconds per kilometers to get the **Pace metric**.

```python
def pace(speed, fmt):
    d = {"hours": 0}
    d["minutes"] = np.floor(60 / speed)
    d["seconds"] = round(((60 / speed - np.floor(60 / speed))*60), 0)
    return fmt.format(**d)

avg_km_h = (sum((activity_with_timeout['speed'] *
                 activity_with_timeout['time_dif'])) /
            sum(activity_with_timeout['time_dif']))

print('Speed:', avg_km_h , 'km/h')
print('Cadence:' , pace(avg_km_h, "{hours} hour {minutes} min {seconds} sec"))
```

    Speed: 10.065838469772745 km/h
    Cadence: 0 hour 5.0 min 58.0 sec

The results compared to the proposed by our app show similar values, with average speed of 5 minutes and 58 seconds per kilometer, a difference of only just 2 secs.

Let's plot our average speed for every 10 seconds to see our speed floats during the run. For this plot we will need the cumulative sum of our time differente to 10 seconds and plot the aggregated speed against it.

```python
activity['time_10s'] = list(map(lambda x: round(x, -1), np.cumsum(activity['time_dif'])))
plt.plot(activity.groupby(['time_10s']).mean()['speed'])
```

The result shown is a smooth line plot where we check the speed (km/h) vs the time in seconds.

### Calculating the elevation gain

The last metric we will explore is the elevation gain. According to the apps [documentation](https://support.strava.com/hc/en-us/articles/216919447-Elevation-for-Your-Activity), the cumulative elevation gain refers to the sum  of every gain elevation throughout an entire Activity.

Based on that, we can compute the elevation gain by mapping over our altitude difference column of our dataframe.

```python
activity_with_timeout.loc[activity_with_timeout['alt_dif'] > 0]['alt_dif'].sum()
```

    38.0

The elevation calculated is far away from what the app showed (about 25m). Checking the altitude difference column values we can see that there are measures down to 10 centimeters.  After reading the docummentation, we found out that the elevation data go through a noise correction algorithm. It is based on the a threshold where climbing needs to occur consistently for more than 10 meters for activities without strong barometric or two meters for an activity with the barometric data before it's added to the total elevation gain.

```python
elevations = zip(list(activity_with_timeout['alt']), list(activity_with_timeout['distance_hav_2d']))

THRESHOLD_METERS = 10
def calculate_elevation_gain(elevations):
    elevations = list(elevations)
    thresholdStartingPoint = elevations[0][0]
    diff_distance = 0
    count_diff = 0
    gain = 0
    valueAgainstThreshold = 0
    for elevation, distance in elevations:
        diff = elevation - thresholdStartingPoint
        diff_distance+=distance
        if diff > 0:
            valueAgainstThreshold += diff
            if abs(diff_distance) >= THRESHOLD_METERS:
                gain += valueAgainstThreshold
                diff_distance = 0
                valueAgainstThreshold = 0
        else:
            diff_distance = 0
            valueAgainstThreshold = 0
        thresholdStartingPoint = elevation

    return gain

#plt.plot(activity['alt_dif'])
print(calculate_elevation_gain(elevations))
```

    25.30000000000002

The new result is 25.2m, very close to the elevation propose by the app. Based on that, we could recalculate the 3rd distances to get these new elevation values into account to get the distance closest to the 2d distance.

## What's next ?

In this article we presented some popular metrics in runner's world such as moving time, cadence, pace and elevation gain. The calculations behind the metrics were also detailed and we gave a possible approximation of some of the algorithms used by a famous tracking running application, since we don't have access, of course,how they implemented it.

The next steps will be implement these metrics as properties, methods of the ``runpandas.Activity`` dataframe. We will release it soon with examples detailed at one of our future posts. Keep following here!

— Please feel free to bring any inconsistencies or mistakes by leaving a comment below.-


# 2. logistic_regression（逻辑回归）

# 备注
运行环境：python 3.6
现在我知道我应该考虑列向量，而Tensorflow对数据的形状非常挑剔。 但是在numpy中，正常的一维ndarray已经被表示为列向量。 如果我重新塑造$\mathbb{R}^n$ 为 $\mathbb{R}^{n\times1}$，它不再是列向量了，而是是1列的矩阵,那使用scipy会有麻烦。
*所以我们应该把TensorFlow的数据视为特殊情况。 我们继续使用numpy的惯例。

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
plt.style.use('fivethirtyeight')
import matplotlib.pyplot as plt
# import tensorflow as tf
from sklearn.metrics import classification_report#这个包是评价报告
```

# 准备数据

```python
data = pd.read_csv('ex2data1.txt', names=['exam1', 'exam2', 'admitted'])
data.head()#看前五行
```

      exam1
      exam2
      admitted

      0
      34.623660
      78.024693
      0

      1
      30.286711
      43.894998
      0

      2
      35.847409
      72.902198
      0

      3
      60.182599
      86.308552
      1

      4
      79.032736
      75.344376
      1

```python
data.describe()
```

      exam1
      exam2
      admitted

      count
      100.000000
      100.000000
      100.000000

      mean
      65.644274
      66.221998
      0.600000

      std
      19.458222
      18.582783
      0.492366

      min
      30.058822
      30.603263
      0.000000

      25%
      50.919511
      48.179205
      0.000000

      50%
      67.032988
      67.682381
      1.000000

      75%
      80.212529
      79.360605
      1.000000

      max
      99.827858
      98.869436
      1.000000

```python
#sns.set(context="notebook", style="darkgrid", palette=sns.color_palette("RdBu", 2))
sns.set(context="notebook", style="darkgrid", palette=sns.color_palette("RdBu", 2), color_codes=False)

sns.lmplot('exam1', 'exam2', hue='admitted', data=data,
           size=6,
           fit_reg=False,
           scatter_kws={"s": 50}
          )
plt.show()#看下数据的样子
```

```python
def get_X(df):#读取特征
#     """
#     use concat to add intersect feature to avoid side effect
#     not efficient for big dataset though
#     """
    ones = pd.DataFrame({'ones': np.ones(len(df))})#ones是m行1列的dataframe
    data = pd.concat([ones, df], axis=1)  # 合并数据，根据列合并
    #return data.iloc[:, :-1].as_matrix()  # 这个操作返回 ndarray,不是矩阵
    #return data.iloc[:, :-1].iloc[:, :].values  # 这个操作返回 ndarray,不是矩阵
    return data.iloc[:, :-1].values  # 这个操作返回 ndarray,不是矩阵

def get_y(df):#读取标签
#     '''assume the last column is the target'''
    return np.array(df.iloc[:, -1])#df.iloc[:, -1]是指df的最后一列

def normalize_feature(df):
#     """Applies function along input axis(default 0) of DataFrame."""
    return df.apply(lambda column: (column - column.mean()) / column.std())#特征缩放
```

```python
X = get_X(data)
print(X.shape)

y = get_y(data)
print(y.shape)
```

    (100, 3)
    (100,)

# sigmoid 函数
g 代表一个常用的逻辑函数（logistic function）为S形函数（Sigmoid function），公式为： \\[g\left( z \right)=\frac{1}{1+{{e}^{-z}}}\\]
合起来，我们得到逻辑回归模型的假设函数：
	\\[{{h}_{\theta }}\left( x \right)=\frac{1}{1+{{e}^{-{{\theta }^{T}}X}}}\\]

```python
def sigmoid(z):
    return 1 / (1 + np.exp(-z))
```

```python
fig, ax = plt.subplots(figsize=(8, 6))
ax.plot(np.arange(-10, 10, step=0.01),
        sigmoid(np.arange(-10, 10, step=0.01)))
ax.set_ylim((-0.1,1.1))
ax.set_xlabel('z', fontsize=18)
ax.set_ylabel('g(z)', fontsize=18)
ax.set_title('sigmoid function', fontsize=18)
plt.show()
```

# cost function(代价函数)
> * $max(\ell(\theta)) = min(-\ell(\theta))$
> * choose $-\ell(\theta)$ as the cost function

$$\begin{align}
  & J\left( \theta  \right)=-\frac{1}{m}\sum\limits_{i=1}^{m}{[{{y}^{(i)}}\log \left( {{h}_{\theta }}\left( {{x}^{(i)}} \right) \right)+\left( 1-{{y}^{(i)}} \right)\log \left( 1-{{h}_{\theta }}\left( {{x}^{(i)}} \right) \right)]} \\
 & =\frac{1}{m}\sum\limits_{i=1}^{m}{[-{{y}^{(i)}}\log \left( {{h}_{\theta }}\left( {{x}^{(i)}} \right) \right)-\left( 1-{{y}^{(i)}} \right)\log \left( 1-{{h}_{\theta }}\left( {{x}^{(i)}} \right) \right)]} \\
\end{align}$$

```python
theta = theta=np.zeros(3) # X(m*n) so theta is n*1
theta
```

    array([0., 0., 0.])

```python

```

```python
def cost(theta, X, y):
    ''' cost fn is -l(theta) for you to minimize'''
    return np.mean(-y * np.log(sigmoid(X @ theta)) - (1 - y) * np.log(1 - sigmoid(X @ theta)))

# X @ theta与X.dot(theta)等价
```

```python
cost(theta, X, y)
```

    0.6931471805599453

# gradient descent(梯度下降)
* 这是批量梯度下降（batch gradient descent）
* 转化为向量化计算： $\frac{1}{m} X^T( Sigmoid(X\theta) - y )$
$$\frac{\partial J\left( \theta  \right)}{\partial {{\theta }_{j}}}=\frac{1}{m}\sum\limits_{i=1}^{m}{({{h}_{\theta }}\left( {{x}^{(i)}} \right)-{{y}^{(i)}})x_{_{j}}^{(i)}}$$

```python
def gradient(theta, X, y):
#     '''just 1 batch gradient'''
    return (1 / len(X)) * X.T @ (sigmoid(X @ theta) - y)
```

```python
gradient(theta, X, y)
```

    array([ -0.1       , -12.00921659, -11.26284221])

# 拟合参数
> * 这里我使用 [`scipy.optimize.minimize`](http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html#scipy.optimize.minimize) 去寻找参数

```python
import scipy.optimize as opt
```

```python
res = opt.minimize(fun=cost, x0=theta, args=(X, y), method='Newton-CG', jac=gradient)
```

```python
print(res)
```

         fun: 0.20349771447597362
         jac: array([9.71219458e-06, 6.96602898e-04, 8.53722702e-04])
     message: 'Optimization terminated successfully.'
        nfev: 73
        nhev: 0
         nit: 29
        njev: 253
      status: 0
     success: True
           x: array([-25.17064404,   0.20630619,   0.20154693])

# 用训练集预测和验证

```python
def predict(x, theta):
    prob = sigmoid(x @ theta)
    return (prob >= 0.5).astype(int)
```

```python
final_theta = res.x
y_pred = predict(X, final_theta)

print(classification_report(y, y_pred))
```

                  precision    recall  f1-score   support

               0       0.87      0.85      0.86        40
               1       0.90      0.92      0.91        60

        accuracy                           0.89       100
       macro avg       0.89      0.88      0.88       100
    weighted avg       0.89      0.89      0.89       100

# 寻找决策边界
http://stats.stackexchange.com/questions/93569/why-is-logistic-regression-a-linear-classifier
> $X \times \theta = 0$  (this is the line)

```python
print(res.x) # this is final theta
```

    [-25.17064404   0.20630619   0.20154693]

```python
coef = -(res.x / res.x[2])  # find the equation
print(coef)

x = np.arange(130, step=0.1)
y = coef[0] + coef[1]*x
```

    [124.88725955  -1.02361365  -1.        ]

```python
data.describe()  # find the range of x and y
```

      exam1
      exam2
      admitted

      count
      100.000000
      100.000000
      100.000000

      mean
      65.644274
      66.221998
      0.600000

      std
      19.458222
      18.582783
      0.492366

      min
      30.058822
      30.603263
      0.000000

      25%
      50.919511
      48.179205
      0.000000

      50%
      67.032988
      67.682381
      1.000000

      75%
      80.212529
      79.360605
      1.000000

      max
      99.827858
      98.869436
      1.000000

> you know the intercept would be around 125 for both x and y

```python
sns.set(context="notebook", style="ticks", font_scale=1.5)

sns.lmplot('exam1', 'exam2', hue='admitted', data=data,
           size=6,
           fit_reg=False,
           scatter_kws={"s": 25}
          )

plt.plot(x, y, 'grey')
plt.xlim(0, 130)
plt.ylim(0, 130)
plt.title('Decision Boundary')
plt.show()
```

# 3- 正则化逻辑回归

```python
df = pd.read_csv('ex2data2.txt', names=['test1', 'test2', 'accepted'])
df.head()
```

      test1
      test2
      accepted

      0
      0.051267
      0.69956
      1

      1
      -0.092742
      0.68494
      1

      2
      -0.213710
      0.69225
      1

      3
      -0.375000
      0.50219
      1

      4
      -0.513250
      0.46564
      1

```python
sns.set(context="notebook", style="ticks", font_scale=1.5)

sns.lmplot('test1', 'test2', hue='accepted', data=df,
           size=6,
           fit_reg=False,
           scatter_kws={"s": 50}
          )

plt.title('Regularized Logistic Regression')
plt.show()
```

# feature mapping（特征映射）

polynomial expansion

```
for i in 0..i
  for p in 0..i:
    output x^(i-p) * y^p
```

```python
def feature_mapping(x, y, power, as_ndarray=False):
#     """return mapped features as ndarray or dataframe"""
    # data = {}
    # # inclusive
    # for i in np.arange(power + 1):
    #     for p in np.arange(i + 1):
    #         data["f{}{}".format(i - p, p)] = np.power(x, i - p) * np.power(y, p)

    data = {"f{}{}".format(i - p, p): np.power(x, i - p) * np.power(y, p)
                for i in np.arange(power + 1)
                for p in np.arange(i + 1)
            }

    if as_ndarray:
        #return pd.DataFrame(data).as_matrix()
        return pd.DataFrame(data).values
    else:
        return pd.DataFrame(data)

```

```python
x1 = np.array(df.test1)
x2 = np.array(df.test2)
```

```python
data = feature_mapping(x1, x2, power=6)
print(data.shape)
data.head()
```

    (118, 28)

      f00
      f10
      f01
      f20
      f11
      f02
      f30
      f21
      f12
      f03
      ...
      f23
      f14
      f05
      f60
      f51
      f42
      f33
      f24
      f15
      f06

      0
      1.0
      0.051267
      0.69956
      0.002628
      0.035864
      0.489384
      0.000135
      0.001839
      0.025089
      0.342354
      ...
      0.000900
      0.012278
      0.167542
      1.815630e-08
      2.477505e-07
      0.000003
      0.000046
      0.000629
      0.008589
      0.117206

      1
      1.0
      -0.092742
      0.68494
      0.008601
      -0.063523
      0.469143
      -0.000798
      0.005891
      -0.043509
      0.321335
      ...
      0.002764
      -0.020412
      0.150752
      6.362953e-07
      -4.699318e-06
      0.000035
      -0.000256
      0.001893
      -0.013981
      0.103256

      2
      1.0
      -0.213710
      0.69225
      0.045672
      -0.147941
      0.479210
      -0.009761
      0.031616
      -0.102412
      0.331733
      ...
      0.015151
      -0.049077
      0.158970
      9.526844e-05
      -3.085938e-04
      0.001000
      -0.003238
      0.010488
      -0.033973
      0.110047

      3
      1.0
      -0.375000
      0.50219
      0.140625
      -0.188321
      0.252195
      -0.052734
      0.070620
      -0.094573
      0.126650
      ...
      0.017810
      -0.023851
      0.031940
      2.780914e-03
      -3.724126e-03
      0.004987
      -0.006679
      0.008944
      -0.011978
      0.016040

      4
      1.0
      -0.513250
      0.46564
      0.263426
      -0.238990
      0.216821
      -0.135203
      0.122661
      -0.111283
      0.100960
      ...
      0.026596
      -0.024128
      0.021890
      1.827990e-02
      -1.658422e-02
      0.015046
      -0.013650
      0.012384
      -0.011235
      0.010193

5 rows × 28 columns

```python
data.describe()
```

      f00
      f10
      f01
      f20
      f11
      f02
      f30
      f21
      f12
      f03
      ...
      f23
      f14
      f05
      f60
      f51
      f42
      f33
      f24
      f15
      f06

      count
      118.0
      118.000000
      118.000000
      118.000000
      118.000000
      118.000000
      1.180000e+02
      118.000000
      118.000000
      118.000000
      ...
      118.000000
      1.180000e+02
      118.000000
      1.180000e+02
      118.000000
      1.180000e+02
      118.000000
      1.180000e+02
      118.000000
      1.180000e+02

      mean
      1.0
      0.054779
      0.183102
      0.247575
      -0.025472
      0.301370
      5.983333e-02
      0.030682
      0.015483
      0.142350
      ...
      0.018278
      4.089084e-03
      0.115710
      7.837118e-02
      -0.000703
      1.893340e-02
      -0.001705
      2.259170e-02
      -0.006302
      1.257256e-01

      std
      0.0
      0.496654
      0.519743
      0.248532
      0.224075
      0.284536
      2.746459e-01
      0.134706
      0.150143
      0.326134
      ...
      0.058513
      9.993907e-02
      0.299092
      1.938621e-01
      0.058271
      3.430092e-02
      0.037443
      4.346935e-02
      0.090621
      2.964416e-01

      min
      1.0
      -0.830070
      -0.769740
      0.000040
      -0.484096
      0.000026
      -5.719317e-01
      -0.358121
      -0.483743
      -0.456071
      ...
      -0.142660
      -4.830370e-01
      -0.270222
      6.472253e-14
      -0.203971
      2.577297e-10
      -0.113448
      2.418097e-10
      -0.482684
      1.795116e-14

      25%
      1.0
      -0.372120
      -0.254385
      0.043243
      -0.178209
      0.061086
      -5.155632e-02
      -0.023672
      -0.042980
      -0.016492
      ...
      -0.001400
      -7.449462e-03
      -0.001072
      8.086369e-05
      -0.006381
      1.258285e-04
      -0.005749
      3.528590e-04
      -0.016662
      2.298277e-04

      50%
      1.0
      -0.006336
      0.213455
      0.165397
      -0.016521
      0.252195
      -2.544062e-07
      0.006603
      -0.000039
      0.009734
      ...
      0.001026
      -8.972096e-09
      0.000444
      4.527344e-03
      -0.000004
      3.387050e-03
      -0.000005
      3.921378e-03
      -0.000020
      1.604015e-02

      75%
      1.0
      0.478970
      0.646562
      0.389925
      0.100795
      0.464189
      1.099616e-01
      0.086392
      0.079510
      0.270310
      ...
      0.021148
      2.751341e-02
      0.113020
      5.932959e-02
      0.002104
      2.090875e-02
      0.001024
      2.103622e-02
      0.001289
      1.001215e-01

      max
      1.0
      1.070900
      1.108900
      1.146827
      0.568307
      1.229659
      1.228137e+00
      0.449251
      0.505577
      1.363569
      ...
      0.287323
      4.012965e-01
      1.676725
      1.508320e+00
      0.250577
      2.018260e-01
      0.183548
      2.556084e-01
      0.436209
      1.859321e+00

8 rows × 28 columns

# regularized cost（正则化代价函数）
$$J\left( \theta  \right)=\frac{1}{m}\sum\limits_{i=1}^{m}{[-{{y}^{(i)}}\log \left( {{h}_{\theta }}\left( {{x}^{(i)}} \right) \right)-\left( 1-{{y}^{(i)}} \right)\log \left( 1-{{h}_{\theta }}\left( {{x}^{(i)}} \right) \right)]}+\frac{\lambda }{2m}\sum\limits_{j=1}^{n}{\theta _{j}^{2}}$$

```python
theta = np.zeros(data.shape[1])
X = feature_mapping(x1, x2, power=6, as_ndarray=True)
print(X.shape)

y = get_y(df)
print(y.shape)
```

    (118, 28)
    (118,)

```python
def regularized_cost(theta, X, y, l=1):
#     '''you don't penalize theta_0'''
    theta_j1_to_n = theta[1:]
    regularized_term = (l / (2 * len(X))) * np.power(theta_j1_to_n, 2).sum()

    return cost(theta, X, y) + regularized_term
#正则化代价函数
```

```python
regularized_cost(theta, X, y, l=1)
```

    0.6931471805599454

this is the same as the not regularized cost because we init theta as zeros...
因为我们设置theta为0，所以这个正则化代价函数与代价函数的值相同

# regularized gradient(正则化梯度)
$$\frac{\partial J\left( \theta  \right)}{\partial {{\theta }_{j}}}=\left( \frac{1}{m}\sum\limits_{i=1}^{m}{\left( {{h}_{\theta }}\left( {{x}^{\left( i \right)}} \right)-{{y}^{\left( i \right)}} \right)} \right)+\frac{\lambda }{m}{{\theta }_{j}}\text{ }\text{             for  j}\ge \text{1}$$

```python
def regularized_gradient(theta, X, y, l=1):
#     '''still, leave theta_0 alone'''
    theta_j1_to_n = theta[1:]
    regularized_theta = (l / len(X)) * theta_j1_to_n

    # by doing this, no offset is on theta_0
    regularized_term = np.concatenate([np.array([0]), regularized_theta])

    return gradient(theta, X, y) + regularized_term
```

```python
regularized_gradient(theta, X, y)
```

    array([8.47457627e-03, 1.87880932e-02, 7.77711864e-05, 5.03446395e-02,
           1.15013308e-02, 3.76648474e-02, 1.83559872e-02, 7.32393391e-03,
           8.19244468e-03, 2.34764889e-02, 3.93486234e-02, 2.23923907e-03,
           1.28600503e-02, 3.09593720e-03, 3.93028171e-02, 1.99707467e-02,
           4.32983232e-03, 3.38643902e-03, 5.83822078e-03, 4.47629067e-03,
           3.10079849e-02, 3.10312442e-02, 1.09740238e-03, 6.31570797e-03,
           4.08503006e-04, 7.26504316e-03, 1.37646175e-03, 3.87936363e-02])

# 拟合参数

```python
import scipy.optimize as opt
```

```python
print('init cost = {}'.format(regularized_cost(theta, X, y)))

res = opt.minimize(fun=regularized_cost, x0=theta, args=(X, y), method='Newton-CG', jac=regularized_gradient)
res
```

    init cost = 0.6931471805599454

         fun: 0.5290027297127542
         jac: array([-4.71952568e-08, -1.14457746e-08,  2.45828726e-08, -3.37674508e-08,
            1.50839836e-08, -6.26344651e-08,  1.47936000e-08,  1.44056576e-08,
            9.25470477e-09, -2.89079664e-08, -4.15440254e-08, -5.39192035e-09,
           -4.31597056e-08,  7.74514234e-09, -7.33884638e-08,  2.48858272e-09,
            1.14058975e-08, -1.27449854e-08, -8.06033750e-09,  1.32958062e-08,
           -4.66425956e-08, -3.00496891e-08,  6.99938544e-10, -1.90907362e-08,
           -5.59017882e-09, -1.83826946e-08,  2.74986858e-09, -7.07341836e-08])
     message: 'Optimization terminated successfully.'
        nfev: 7
        nhev: 0
         nit: 6
        njev: 66
      status: 0
     success: True
           x: array([ 1.27273914,  0.62527176,  1.18108862, -2.01996089, -0.91742465,
           -1.43166351,  0.12400766, -0.36553408, -0.35723919, -0.17512987,
           -1.4581576 , -0.05099003, -0.61555683, -0.27470695, -1.19281634,
           -0.24218765, -0.20600567, -0.04473145, -0.27778483, -0.29537771,
           -0.45635687, -1.04320334,  0.02777145, -0.29243213,  0.01556617,
           -0.32738012, -0.1438876 , -0.92465245])

# 预测

```python
final_theta = res.x
y_pred = predict(X, final_theta)

print(classification_report(y, y_pred))
```

                  precision    recall  f1-score   support

               0       0.90      0.75      0.82        60
               1       0.78      0.91      0.84        58

        accuracy                           0.83       118
       macro avg       0.84      0.83      0.83       118
    weighted avg       0.84      0.83      0.83       118

# 使用不同的 $\lambda$ （这个是常数）
# 画出决策边界
* 我们找到所有满足 $X\times \theta = 0$ 的x
* instead of solving polynomial equation, just create a coridate x,y grid that is dense enough, and find all those $X\times \theta$ that is close enough to 0, then plot them

```python
def draw_boundary(power, l):
#     """
#     power: polynomial power for mapped feature
#     l: lambda constant
#     """
    density = 1000
    threshhold = 2 * 10**-3

    final_theta = feature_mapped_logistic_regression(power, l)
    x, y = find_decision_boundary(density, power, final_theta, threshhold)

    df = pd.read_csv('ex2data2.txt', names=['test1', 'test2', 'accepted'])
    sns.lmplot('test1', 'test2', hue='accepted', data=df, size=6, fit_reg=False, scatter_kws={"s": 100})

    plt.scatter(x, y, c='R', s=10)
    plt.title('Decision boundary')
    plt.show()
```

```python
def feature_mapped_logistic_regression(power, l):
#     """for drawing purpose only.. not a well generealize logistic regression
#     power: int
#         raise x1, x2 to polynomial power
#     l: int
#         lambda constant for regularization term
#     """
    df = pd.read_csv('ex2data2.txt', names=['test1', 'test2', 'accepted'])
    x1 = np.array(df.test1)
    x2 = np.array(df.test2)
    y = get_y(df)

    X = feature_mapping(x1, x2, power, as_ndarray=True)
    theta = np.zeros(X.shape[1])

    res = opt.minimize(fun=regularized_cost,
                       x0=theta,
                       args=(X, y, l),
                       method='TNC',
                       jac=regularized_gradient)
    final_theta = res.x

    return final_theta
```

```python
def find_decision_boundary(density, power, theta, threshhold):
    t1 = np.linspace(-1, 1.5, density)
    t2 = np.linspace(-1, 1.5, density)

    cordinates = [(x, y) for x in t1 for y in t2]
    x_cord, y_cord = zip(*cordinates)
    mapped_cord = feature_mapping(x_cord, y_cord, power)  # this is a dataframe

    #inner_product = mapped_cord.as_matrix() @ theta
    inner_product = mapped_cord.values @ theta

    decision = mapped_cord[np.abs(inner_product) < threshhold]

    return decision.f10, decision.f01
#寻找决策边界函数
```

```python
draw_boundary(power=6, l=1)#lambda=1
```

```python
draw_boundary(power=6, l=0)  # no regularization, over fitting，#lambda=0,没有正则化，过拟合了
```

```python
draw_boundary(power=6, l=100)  # underfitting，#lambda=100,欠拟合
```

```python

```