```python
from preamble import *
%matplotlib inline
```

## Model Evaluation and Improvement

```python
from sklearn.datasets import make_blobs
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# create a synthetic dataset
X, y = make_blobs(random_state=0)
# split data and labels into a training and a test set
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
# instantiate a model and fit it to the training set
logreg = LogisticRegression().fit(X_train, y_train)
# evaluate the model on the test set
print("Test set score: {:.2f}".format(logreg.score(X_test, y_test)))
```

    Test set score: 0.88

### Cross-Validation

```python
mglearn.plots.plot_cross_validation()
```

#### Cross-Validation in scikit-learn

```python
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression

iris = load_iris()
logreg = LogisticRegression(max_iter=1000)

scores = cross_val_score(logreg, iris.data, iris.target)
print("Cross-validation scores: {}".format(scores))
```

    Cross-validation scores: [0.967 1.    0.933 0.967 1.   ]

```python
scores = cross_val_score(logreg, iris.data, iris.target, cv=5)
print("Cross-validation scores: {}".format(scores))
```

    Cross-validation scores: [0.967 1.    0.933 0.967 1.   ]

```python
print("Average cross-validation score: {:.2f}".format(scores.mean()))
```

    Average cross-validation score: 0.97

```python
from sklearn.model_selection import cross_validate
res = cross_validate(logreg, iris.data, iris.target, cv=5,
                     return_train_score=True)
display(res)
```

    {'fit_time': array([0.013, 0.017, 0.012, 0.013, 0.012]),
     'score_time': array([0.   , 0.   , 0.001, 0.   , 0.001]),
     'test_score': array([0.967, 1.   , 0.933, 0.967, 1.   ]),
     'train_score': array([0.967, 0.967, 0.983, 0.983, 0.975])}

```python
res_df = pd.DataFrame(res)
display(res_df)
print("Mean times and scores:\n", res_df.mean())
```

      fit_time
      score_time
      test_score
      train_score

      0
      0.01
      0.00e+00
      0.97
      0.97

      1
      0.02
      0.00e+00
      1.00
      0.97

      2
      0.01
      9.94e-04
      0.93
      0.98

      3
      0.01
      0.00e+00
      0.97
      0.98

      4
      0.01
      9.98e-04
      1.00
      0.97

    Mean times and scores:
     fit_time       1.34e-02
    score_time     3.98e-04
    test_score     9.73e-01
    train_score    9.75e-01
    dtype: float64

#### Benefits of Cross-Validation

### Stratified K-Fold cross-validation and other strategies

```python
from sklearn.datasets import load_iris
iris = load_iris()
print("Iris labels:\n{}".format(iris.target))
```

    Iris labels:
    [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
     1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2
     2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
     2 2]

```python
mglearn.plots.plot_stratified_cross_validation()
```

#### More control over cross-validation

```python
from sklearn.model_selection import KFold
kfold = KFold(n_splits=5)
```

```python
print("Cross-validation scores:\n{}".format(
      cross_val_score(logreg, iris.data, iris.target, cv=kfold)))
```

    Cross-validation scores:
    [1.    1.    0.867 0.933 0.833]

```python
kfold = KFold(n_splits=3)
print("Cross-validation scores:\n{}".format(
    cross_val_score(logreg, iris.data, iris.target, cv=kfold)))
```

    Cross-validation scores:
    [0. 0. 0.]

```python
kfold = KFold(n_splits=3, shuffle=True, random_state=0)
print("Cross-validation scores:\n{}".format(
    cross_val_score(logreg, iris.data, iris.target, cv=kfold)))
```

    Cross-validation scores:
    [0.98 0.96 0.96]

#### Leave-one-out cross-validation

```python
from sklearn.model_selection import LeaveOneOut
loo = LeaveOneOut()
scores = cross_val_score(logreg, iris.data, iris.target, cv=loo)
print("Number of cv iterations: ", len(scores))
print("Mean accuracy: {:.2f}".format(scores.mean()))
```

    Number of cv iterations:  150
    Mean accuracy: 0.97

#### Shuffle-split cross-validation

```python
mglearn.plots.plot_shuffle_split()
```

```python
from sklearn.model_selection import ShuffleSplit
shuffle_split = ShuffleSplit(test_size=.5, train_size=.5, n_splits=10)
scores = cross_val_score(logreg, iris.data, iris.target, cv=shuffle_split)
print("Cross-validation scores:\n{}".format(scores))
```

    Cross-validation scores:
    [0.947 0.987 0.973 0.96  0.933 0.96  0.987 0.947 0.947 0.933]

##### Cross-validation with groups

```python
mglearn.plots.plot_group_kfold()
```

```python
from sklearn.model_selection import GroupKFold
# create synthetic dataset
X, y = make_blobs(n_samples=12, random_state=0)
# assume the first three samples belong to the same group,
# then the next four, etc.
groups = [0, 0, 0, 1, 1, 1, 1, 2, 2, 3, 3, 3]
scores = cross_val_score(logreg, X, y, groups=groups, cv=GroupKFold(n_splits=3))
print("Cross-validation scores:\n{}".format(scores))
```

    Cross-validation scores:
    [0.75  0.6   0.667]

### Grid Search

#### Simple Grid Search

```python
# naive grid search implementation
from sklearn.svm import SVC
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, random_state=0)
print("Size of training set: {}   size of test set: {}".format(
      X_train.shape[0], X_test.shape[0]))

best_score = 0

for gamma in [0.001, 0.01, 0.1, 1, 10, 100]:
    for C in [0.001, 0.01, 0.1, 1, 10, 100]:
        # for each combination of parameters, train an SVC
        svm = SVC(gamma=gamma, C=C)
        svm.fit(X_train, y_train)
        # evaluate the SVC on the test set
        score = svm.score(X_test, y_test)
        # if we got a better score, store the score and parameters
        if score > best_score:
            best_score = score
            best_parameters = {'C': C, 'gamma': gamma}

print("Best score: {:.2f}".format(best_score))
print("Best parameters: {}".format(best_parameters))
```

    Size of training set: 112   size of test set: 38
    Best score: 0.97
    Best parameters: {'C': 100, 'gamma': 0.001}

#### The danger of overfitting the parameters and the validation set

```python
mglearn.plots.plot_threefold_split()
```

```python
from sklearn.svm import SVC
# split data into train+validation set and test set
X_trainval, X_test, y_trainval, y_test = train_test_split(
    iris.data, iris.target, random_state=0)
# split train+validation set into training and validation sets
X_train, X_valid, y_train, y_valid = train_test_split(
    X_trainval, y_trainval, random_state=1)
print("Size of training set: {}   size of validation set: {}   size of test set:"
      " {}\n".format(X_train.shape[0], X_valid.shape[0], X_test.shape[0]))

best_score = 0

for gamma in [0.001, 0.01, 0.1, 1, 10, 100]:
    for C in [0.001, 0.01, 0.1, 1, 10, 100]:
        # for each combination of parameters, train an SVC
        svm = SVC(gamma=gamma, C=C)
        svm.fit(X_train, y_train)
        # evaluate the SVC on the validation set
        score = svm.score(X_valid, y_valid)
        # if we got a better score, store the score and parameters
        if score > best_score:
            best_score = score
            best_parameters = {'C': C, 'gamma': gamma}

# rebuild a model on the combined training and validation set,
# and evaluate it on the test set
svm = SVC(**best_parameters)
svm.fit(X_trainval, y_trainval)
test_score = svm.score(X_test, y_test)
print("Best score on validation set: {:.2f}".format(best_score))
print("Best parameters: ", best_parameters)
print("Test set score with best parameters: {:.2f}".format(test_score))
```

    Size of training set: 84   size of validation set: 28   size of test set: 38

    Best score on validation set: 0.96
    Best parameters:  {'C': 10, 'gamma': 0.001}
    Test set score with best parameters: 0.92

#### Grid Search with Cross-Validation

```python
for gamma in [0.001, 0.01, 0.1, 1, 10, 100]:
    for C in [0.001, 0.01, 0.1, 1, 10, 100]:
        # for each combination of parameters,
        # train an SVC
        svm = SVC(gamma=gamma, C=C)
        # perform cross-validation
        scores = cross_val_score(svm, X_trainval, y_trainval, cv=5)
        # compute mean cross-validation accuracy
        score = np.mean(scores)
        # if we got a better score, store the score and parameters
        if score > best_score:
            best_score = score
            best_parameters = {'C': C, 'gamma': gamma}
# rebuild a model on the combined training and validation set
svm = SVC(**best_parameters)
svm.fit(X_trainval, y_trainval)
```

    SVC(C=10, gamma=0.1)

```python
mglearn.plots.plot_cross_val_selection()
```

```python
mglearn.plots.plot_grid_search_overview()
```

```python
param_grid = {'C': [0.001, 0.01, 0.1, 1, 10, 100],
              'gamma': [0.001, 0.01, 0.1, 1, 10, 100]}
print("Parameter grid:\n{}".format(param_grid))
```

    Parameter grid:
    {'C': [0.001, 0.01, 0.1, 1, 10, 100], 'gamma': [0.001, 0.01, 0.1, 1, 10, 100]}

```python
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
grid_search = GridSearchCV(SVC(), param_grid, cv=5,
                          return_train_score=True)
```

```python
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, random_state=0)
```

```python
grid_search.fit(X_train, y_train)
```

    GridSearchCV(cv=5, estimator=SVC(),
                 param_grid={'C': [0.001, 0.01, 0.1, 1, 10, 100],
                             'gamma': [0.001, 0.01, 0.1, 1, 10, 100]},
                 return_train_score=True)

```python
print("Test set score: {:.2f}".format(grid_search.score(X_test, y_test)))
```

    Test set score: 0.97

```python
print("Best parameters: {}".format(grid_search.best_params_))
print("Best cross-validation score: {:.2f}".format(grid_search.best_score_))
```

    Best parameters: {'C': 10, 'gamma': 0.1}
    Best cross-validation score: 0.97

```python
print("Best estimator:\n{}".format(grid_search.best_estimator_))
```

    Best estimator:
    SVC(C=10, gamma=0.1)

##### Analyzing the result of cross-validation

```python
import pandas as pd
# convert to Dataframe
results = pd.DataFrame(grid_search.cv_results_)
# show the first 5 rows
display(results.head())
```

      mean_fit_time
      std_fit_time
      mean_score_time
      std_score_time
      ...
      split3_train_score
      split4_train_score
      mean_train_score
      std_train_score

      0
      5.99e-04
      4.89e-04
      1.99e-04
      3.99e-04
      ...
      0.37
      0.36
      0.37
      5.58e-03

      1
      5.98e-04
      4.89e-04
      0.00e+00
      0.00e+00
      ...
      0.37
      0.36
      0.37
      5.58e-03

      2
      3.99e-04
      4.89e-04
      3.99e-04
      4.89e-04
      ...
      0.37
      0.36
      0.37
      5.58e-03

      3
      7.98e-04
      3.99e-04
      0.00e+00
      0.00e+00
      ...
      0.37
      0.36
      0.37
      5.58e-03

      4
      3.99e-04
      4.89e-04
      2.00e-04
      3.99e-04
      ...
      0.37
      0.36
      0.37
      5.58e-03

5 rows × 22 columns

```python
scores = np.array(results.mean_test_score).reshape(6, 6)

# plot the mean cross-validation scores
mglearn.tools.heatmap(scores, xlabel='gamma', xticklabels=param_grid['gamma'],
                      ylabel='C', yticklabels=param_grid['C'], cmap="viridis")
```

```python
fig, axes = plt.subplots(1, 3, figsize=(13, 5))

param_grid_linear = {'C': np.linspace(1, 2, 6),
                     'gamma':  np.linspace(1, 2, 6)}

param_grid_one_log = {'C': np.linspace(1, 2, 6),
                      'gamma':  np.logspace(-3, 2, 6)}

param_grid_range = {'C': np.logspace(-3, 2, 6),
                    'gamma':  np.logspace(-7, -2, 6)}

for param_grid, ax in zip([param_grid_linear, param_grid_one_log,
                           param_grid_range], axes):
    grid_search = GridSearchCV(SVC(), param_grid, cv=5)
    grid_search.fit(X_train, y_train)
    scores = grid_search.cv_results_['mean_test_score'].reshape(6, 6)

    # plot the mean cross-validation scores
    scores_image = mglearn.tools.heatmap(
        scores, xlabel='gamma', ylabel='C', xticklabels=param_grid['gamma'],
        yticklabels=param_grid['C'], cmap="viridis", ax=ax)

plt.colorbar(scores_image, ax=axes.tolist())
```

```python
param_grid = [{'kernel': ['rbf'],
               'C': [0.001, 0.01, 0.1, 1, 10, 100],
               'gamma': [0.001, 0.01, 0.1, 1, 10, 100]},
              {'kernel': ['linear'],
               'C': [0.001, 0.01, 0.1, 1, 10, 100]}]
print("List of grids:\n{}".format(param_grid))
```

    List of grids:
    [{'kernel': ['rbf'], 'C': [0.001, 0.01, 0.1, 1, 10, 100], 'gamma': [0.001, 0.01, 0.1, 1, 10, 100]}, {'kernel': ['linear'], 'C': [0.001, 0.01, 0.1, 1, 10, 100]}]

```python
grid_search = GridSearchCV(SVC(), param_grid, cv=5,
                          return_train_score=True)
grid_search.fit(X_train, y_train)
print("Best parameters: {}".format(grid_search.best_params_))
print("Best cross-validation score: {:.2f}".format(grid_search.best_score_))
```

    Best parameters: {'C': 10, 'gamma': 0.1, 'kernel': 'rbf'}
    Best cross-validation score: 0.97

```python
results = pd.DataFrame(grid_search.cv_results_)
# we display the transposed table so that it better fits on the page:
display(results.T)
```

      0
      1
      2
      3
      ...
      38
      39
      40
      41

      mean_fit_time
      0.0008
      0.0006
      0.0008
      0.0008
      ...
      0
      0
      0.0002
      0.001

      std_fit_time
      0.0004
      0.00049
      0.0004
      0.0004
      ...
      0
      0
      0.00041
      1.6e-05

      mean_score_time
      0.0002
      0.0004
      0.0002
      0.0002
      ...
      0.0008
      0
      0
      0

      std_score_time
      0.0004
      0.00049
      0.0004
      0.0004
      ...
      0.0004
      0
      0
      0

      param_C
      0.001
      0.001
      0.001
      0.001
      ...
      0.1
      1
      10
      100

      param_gamma
      0.001
      0.01
      0.1
      1
      ...
      NaN
      NaN
      NaN
      NaN

      param_kernel
      rbf
      rbf
      rbf
      rbf
      ...
      linear
      linear
      linear
      linear

      params
      {'C': 0.001, 'gamma': 0.001, 'kernel': 'rbf'}
      {'C': 0.001, 'gamma': 0.01, 'kernel': 'rbf'}
      {'C': 0.001, 'gamma': 0.1, 'kernel': 'rbf'}
      {'C': 0.001, 'gamma': 1, 'kernel': 'rbf'}
      ...
      {'C': 0.1, 'kernel': 'linear'}
      {'C': 1, 'kernel': 'linear'}
      {'C': 10, 'kernel': 'linear'}
      {'C': 100, 'kernel': 'linear'}

      split0_test_score
      0.35
      0.35
      0.35
      0.35
      ...
      1
      1
      1
      0.96

      split1_test_score
      0.35
      0.35
      0.35
      0.35
      ...
      0.91
      0.96
      1
      0.96

      split2_test_score
      0.36
      0.36
      0.36
      0.36
      ...
      1
      1
      1
      1

      split3_test_score
      0.36
      0.36
      0.36
      0.36
      ...
      0.91
      0.95
      0.91
      0.91

      split4_test_score
      0.41
      0.41
      0.41
      0.41
      ...
      0.95
      0.95
      0.95
      0.95

      mean_test_score
      0.37
      0.37
      0.37
      0.37
      ...
      0.96
      0.97
      0.97
      0.96

      std_test_score
      0.022
      0.022
      0.022
      0.022
      ...
      0.04
      0.022
      0.036
      0.029

      rank_test_score
      27
      27
      27
      27
      ...
      8
      1
      3
      8

      split0_train_score
      0.37
      0.37
      0.37
      0.37
      ...
      0.97
      0.99
      0.99
      0.98

      split1_train_score
      0.37
      0.37
      0.37
      0.37
      ...
      0.97
      0.98
      0.99
      0.99

      split2_train_score
      0.37
      0.37
      0.37
      0.37
      ...
      0.94
      0.98
      0.98
      0.99

      split3_train_score
      0.37
      0.37
      0.37
      0.37
      ...
      0.97
      0.99
      0.99
      1

      split4_train_score
      0.36
      0.36
      0.36
      0.36
      ...
      0.97
      0.99
      1
      1

      mean_train_score
      0.37
      0.37
      0.37
      0.37
      ...
      0.96
      0.98
      0.99
      0.99

      std_train_score
      0.0056
      0.0056
      0.0056
      0.0056
      ...
      0.0088
      0.0055
      0.007
      0.0084

23 rows × 42 columns

#### Using different cross-validation strategies with grid search

#### Nested cross-validation

```python
param_grid = {'C': [0.001, 0.01, 0.1, 1, 10, 100],
              'gamma': [0.001, 0.01, 0.1, 1, 10, 100]}
scores = cross_val_score(GridSearchCV(SVC(), param_grid, cv=5),
                         iris.data, iris.target, cv=5)
print("Cross-validation scores: ", scores)
print("Mean cross-validation score: ", scores.mean())
```

    Cross-validation scores:  [0.967 1.    0.967 0.967 1.   ]
    Mean cross-validation score:  0.9800000000000001

```python
def nested_cv(X, y, inner_cv, outer_cv, Classifier, parameter_grid):
    outer_scores = []
    # for each split of the data in the outer cross-validation
    # (split method returns indices of training and test parts)
    for training_samples, test_samples in outer_cv.split(X, y):
        # find best parameter using inner cross-validation
        best_parms = {}
        best_score = -np.inf
        # iterate over parameters
        for parameters in parameter_grid:
            # accumulate score over inner splits
            cv_scores = []
            # iterate over inner cross-validation
            for inner_train, inner_test in inner_cv.split(
                    X[training_samples], y[training_samples]):
                # build classifier given parameters and training data
                clf = Classifier(**parameters)
                clf.fit(X[inner_train], y[inner_train])
                # evaluate on inner test set
                score = clf.score(X[inner_test], y[inner_test])
                cv_scores.append(score)
            # compute mean score over inner folds
            mean_score = np.mean(cv_scores)
            if mean_score > best_score:
                # if better than so far, remember parameters
                best_score = mean_score
                best_params = parameters
        # build classifier on best parameters using outer training set
        clf = Classifier(**best_params)
        clf.fit(X[training_samples], y[training_samples])
        # evaluate
        outer_scores.append(clf.score(X[test_samples], y[test_samples]))
    return np.array(outer_scores)
```

```python
from sklearn.model_selection import ParameterGrid, StratifiedKFold
scores = nested_cv(iris.data, iris.target, StratifiedKFold(5),
                   StratifiedKFold(5), SVC, ParameterGrid(param_grid))
print("Cross-validation scores: {}".format(scores))
```

    Cross-validation scores: [0.967 1.    0.967 0.967 1.   ]

##### Parallelizing cross-validation and grid search

### Evaluation Metrics and Scoring
#### Keep the End Goal in Mind

#### Metrics for Binary Classification
##### Kinds of errors
##### Imbalanced datasets

```python
from sklearn.datasets import load_digits

digits = load_digits()
y = digits.target == 9

X_train, X_test, y_train, y_test = train_test_split(
    digits.data, y, random_state=0)
```

```python
from sklearn.dummy import DummyClassifier
dummy_majority = DummyClassifier(strategy='most_frequent').fit(X_train, y_train)
pred_most_frequent = dummy_majority.predict(X_test)
print("Unique predicted labels: {}".format(np.unique(pred_most_frequent)))
print("Test score: {:.2f}".format(dummy_majority.score(X_test, y_test)))
```

    Unique predicted labels: [False]
    Test score: 0.90

```python
from sklearn.tree import DecisionTreeClassifier
tree = DecisionTreeClassifier(max_depth=2).fit(X_train, y_train)
pred_tree = tree.predict(X_test)
print("Test score: {:.2f}".format(tree.score(X_test, y_test)))
```

    Test score: 0.92

```python
from sklearn.linear_model import LogisticRegression

dummy = DummyClassifier().fit(X_train, y_train)
pred_dummy = dummy.predict(X_test)
print("dummy score: {:.2f}".format(dummy.score(X_test, y_test)))

logreg = LogisticRegression(max_iter=1000, C=0.1).fit(X_train, y_train)
pred_logreg = logreg.predict(X_test)
print("logreg score: {:.2f}".format(logreg.score(X_test, y_test)))
```

    dummy score: 0.84
    logreg score: 0.98

    c:\users\t3kci\checkout\scikit-learn\sklearn\dummy.py:132: FutureWarning: The default value of strategy will change from stratified to prior in 0.24.
      "stratified to prior in 0.24.", FutureWarning)

##### Confusion matrices

```python
from sklearn.metrics import confusion_matrix

confusion = confusion_matrix(y_test, pred_logreg)
print("Confusion matrix:\n{}".format(confusion))
```

    Confusion matrix:
    [[402   1]
     [  6  41]]

```python
mglearn.plots.plot_confusion_matrix_illustration()
```

```python
mglearn.plots.plot_binary_confusion_matrix()
```

```python
print("Most frequent class:")
print(confusion_matrix(y_test, pred_most_frequent))
print("\nDummy model:")
print(confusion_matrix(y_test, pred_dummy))
print("\nDecision tree:")
print(confusion_matrix(y_test, pred_tree))
print("\nLogistic Regression")
print(confusion_matrix(y_test, pred_logreg))
```

    Most frequent class:
    [[403   0]
     [ 47   0]]

    Dummy model:
    [[362  41]
     [ 45   2]]

    Decision tree:
    [[390  13]
     [ 24  23]]

    Logistic Regression
    [[402   1]
     [  6  41]]

###### Relation to accuracy
\begin{equation}
\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}}
\end{equation}

##### Precision, recall and f-score
\begin{equation}
\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}
\end{equation}

\begin{equation}
\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}
\end{equation}
\begin{equation}
\text{F} = 2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}
\end{equation}

```python
from sklearn.metrics import f1_score
print("f1 score most frequent: {:.2f}".format(
    f1_score(y_test, pred_most_frequent)))
print("f1 score dummy: {:.2f}".format(f1_score(y_test, pred_dummy)))
print("f1 score tree: {:.2f}".format(f1_score(y_test, pred_tree)))
print("f1 score logistic regression: {:.2f}".format(
    f1_score(y_test, pred_logreg)))
```

    f1 score most frequent: 0.00
    f1 score dummy: 0.04
    f1 score tree: 0.55
    f1 score logistic regression: 0.92

```python
from sklearn.metrics import classification_report
print(classification_report(y_test, pred_most_frequent,
                            target_names=["not nine", "nine"]))
```

                  precision    recall  f1-score   support

        not nine       0.90      1.00      0.94       403
            nine       0.00      0.00      0.00        47

        accuracy                           0.90       450
       macro avg       0.45      0.50      0.47       450
    weighted avg       0.80      0.90      0.85       450

    c:\users\t3kci\checkout\scikit-learn\sklearn\metrics\_classification.py:1221: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
      _warn_prf(average, modifier, msg_start, len(result))

```python
print(classification_report(y_test, pred_dummy,
                            target_names=["not nine", "nine"]))
```

                  precision    recall  f1-score   support

        not nine       0.89      0.90      0.89       403
            nine       0.05      0.04      0.04        47

        accuracy                           0.81       450
       macro avg       0.47      0.47      0.47       450
    weighted avg       0.80      0.81      0.81       450

```python
print(classification_report(y_test, pred_logreg,
                            target_names=["not nine", "nine"]))
```

                  precision    recall  f1-score   support

        not nine       0.99      1.00      0.99       403
            nine       0.98      0.87      0.92        47

        accuracy                           0.98       450
       macro avg       0.98      0.93      0.96       450
    weighted avg       0.98      0.98      0.98       450

##### Taking uncertainty into account

```python
X, y = make_blobs(n_samples=(400, 50), cluster_std=[7.0, 2],
                  random_state=22)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
svc = SVC(gamma=.05).fit(X_train, y_train)
```

```python
mglearn.plots.plot_decision_threshold()
```

```python
print(classification_report(y_test, svc.predict(X_test)))
```

                  precision    recall  f1-score   support

               0       0.97      0.89      0.93       104
               1       0.35      0.67      0.46         9

        accuracy                           0.88       113
       macro avg       0.66      0.78      0.70       113
    weighted avg       0.92      0.88      0.89       113

```python
y_pred_lower_threshold = svc.decision_function(X_test) > -.8
```

```python
print(classification_report(y_test, y_pred_lower_threshold))
```

                  precision    recall  f1-score   support

               0       1.00      0.82      0.90       104
               1       0.32      1.00      0.49         9

        accuracy                           0.83       113
       macro avg       0.66      0.91      0.69       113
    weighted avg       0.95      0.83      0.87       113

##### Precision-Recall curves and ROC curves

```python
from sklearn.metrics import precision_recall_curve
precision, recall, thresholds = precision_recall_curve(
    y_test, svc.decision_function(X_test))
```

```python
# Use more data points for a smoother curve
X, y = make_blobs(n_samples=(4000, 500), cluster_std=[7.0, 2], random_state=22)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
svc = SVC(gamma=.05).fit(X_train, y_train)
precision, recall, thresholds = precision_recall_curve(
    y_test, svc.decision_function(X_test))
# find threshold closest to zero
close_zero = np.argmin(np.abs(thresholds))
plt.plot(precision[close_zero], recall[close_zero], 'o', markersize=10,
         label="threshold zero", fillstyle="none", c='k', mew=2)

plt.plot(precision, recall, label="precision recall curve")
plt.xlabel("Precision")
plt.ylabel("Recall")
plt.legend(loc="best")
```

```python
from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=100, random_state=0, max_features=2)
rf.fit(X_train, y_train)

# RandomForestClassifier has predict_proba, but not decision_function
precision_rf, recall_rf, thresholds_rf = precision_recall_curve(
    y_test, rf.predict_proba(X_test)[:, 1])

plt.plot(precision, recall, label="svc")

plt.plot(precision[close_zero], recall[close_zero], 'o', markersize=10,
         label="threshold zero svc", fillstyle="none", c='k', mew=2)

plt.plot(precision_rf, recall_rf, label="rf")

close_default_rf = np.argmin(np.abs(thresholds_rf - 0.5))
plt.plot(precision_rf[close_default_rf], recall_rf[close_default_rf], '^', c='k',
         markersize=10, label="threshold 0.5 rf", fillstyle="none", mew=2)
plt.xlabel("Precision")
plt.ylabel("Recall")
plt.legend(loc="best")
```

```python
print("f1_score of random forest: {:.3f}".format(
    f1_score(y_test, rf.predict(X_test))))
print("f1_score of svc: {:.3f}".format(f1_score(y_test, svc.predict(X_test))))
```

    f1_score of random forest: 0.610
    f1_score of svc: 0.656

```python
from sklearn.metrics import average_precision_score
ap_rf = average_precision_score(y_test, rf.predict_proba(X_test)[:, 1])
ap_svc = average_precision_score(y_test, svc.decision_function(X_test))
print("Average precision of random forest: {:.3f}".format(ap_rf))
print("Average precision of svc: {:.3f}".format(ap_svc))
```

    Average precision of random forest: 0.660
    Average precision of svc: 0.666

##### Receiver Operating Characteristics (ROC) and AUC
\begin{equation}
\text{FPR} = \frac{\text{FP}}{\text{FP} + \text{TN}}
\end{equation}

```python
from sklearn.metrics import roc_curve
fpr, tpr, thresholds = roc_curve(y_test, svc.decision_function(X_test))

plt.plot(fpr, tpr, label="ROC Curve")
plt.xlabel("FPR")
plt.ylabel("TPR (recall)")
# find threshold closest to zero
close_zero = np.argmin(np.abs(thresholds))
plt.plot(fpr[close_zero], tpr[close_zero], 'o', markersize=10,
         label="threshold zero", fillstyle="none", c='k', mew=2)
plt.legend(loc=4)
```

```python
fpr_rf, tpr_rf, thresholds_rf = roc_curve(y_test, rf.predict_proba(X_test)[:, 1])

plt.plot(fpr, tpr, label="ROC Curve SVC")
plt.plot(fpr_rf, tpr_rf, label="ROC Curve RF")

plt.xlabel("FPR")
plt.ylabel("TPR (recall)")
plt.plot(fpr[close_zero], tpr[close_zero], 'o', markersize=10,
         label="threshold zero SVC", fillstyle="none", c='k', mew=2)
close_default_rf = np.argmin(np.abs(thresholds_rf - 0.5))
plt.plot(fpr_rf[close_default_rf], tpr[close_default_rf], '^', markersize=10,
         label="threshold 0.5 RF", fillstyle="none", c='k', mew=2)

plt.legend(loc=4)
```

```python
from sklearn.metrics import roc_auc_score
rf_auc = roc_auc_score(y_test, rf.predict_proba(X_test)[:, 1])
svc_auc = roc_auc_score(y_test, svc.decision_function(X_test))
print("AUC for Random Forest: {:.3f}".format(rf_auc))
print("AUC for SVC: {:.3f}".format(svc_auc))
```

    AUC for Random Forest: 0.937
    AUC for SVC: 0.916

```python
y = digits.target == 9

X_train, X_test, y_train, y_test = train_test_split(
    digits.data, y, random_state=0)

plt.figure()

for gamma in [1, 0.05, 0.01]:
    svc = SVC(gamma=gamma).fit(X_train, y_train)
    accuracy = svc.score(X_test, y_test)
    auc = roc_auc_score(y_test, svc.decision_function(X_test))
    fpr, tpr, _ = roc_curve(y_test , svc.decision_function(X_test))
    print("gamma = {:.2f}  accuracy = {:.2f}  AUC = {:.2f}".format(
          gamma, accuracy, auc))
    plt.plot(fpr, tpr, label="gamma={:.3f}".format(gamma))
plt.xlabel("FPR")
plt.ylabel("TPR")
plt.xlim(-0.01, 1)
plt.ylim(0, 1.02)
plt.legend(loc="best")
```

#### Metrics for Multiclass Classification

```python
from sklearn.metrics import accuracy_score
X_train, X_test, y_train, y_test = train_test_split(
    digits.data, digits.target, random_state=0)
lr = LogisticRegression().fit(X_train, y_train)
pred = lr.predict(X_test)
print("Accuracy: {:.3f}".format(accuracy_score(y_test, pred)))
print("Confusion matrix:\n{}".format(confusion_matrix(y_test, pred)))
```

    Accuracy: 0.951
    Confusion matrix:
    [[37  0  0  0  0  0  0  0  0  0]
     [ 0 40  0  0  0  0  0  0  2  1]
     [ 0  1 40  3  0  0  0  0  0  0]
     [ 0  0  0 43  0  0  0  0  1  1]
     [ 0  0  0  0 37  0  0  1  0  0]
     [ 0  0  0  0  0 46  0  0  0  2]
     [ 0  1  0  0  0  0 51  0  0  0]
     [ 0  0  0  1  1  0  0 46  0  0]
     [ 0  3  1  0  0  0  0  0 43  1]
     [ 0  0  0  0  0  1  0  0  1 45]]

    c:\users\t3kci\checkout\scikit-learn\sklearn\linear_model\_logistic.py:762: ConvergenceWarning: lbfgs failed to converge (status=1):
    STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

    Increase the number of iterations (max_iter) or scale the data as shown in:
        https://scikit-learn.org/stable/modules/preprocessing.html
    Please also refer to the documentation for alternative solver options:
        https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
      extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)

```python
scores_image = mglearn.tools.heatmap(
    confusion_matrix(y_test, pred), xlabel='Predicted label',
    ylabel='True label', xticklabels=digits.target_names,
    yticklabels=digits.target_names, cmap=plt.cm.gray_r, fmt="%d")
plt.title("Confusion matrix")
plt.gca().invert_yaxis()
```

```python
print(classification_report(y_test, pred))
```

                  precision    recall  f1-score   support

               0       1.00      1.00      1.00        37
               1       0.89      0.93      0.91        43
               2       0.98      0.91      0.94        44
               3       0.91      0.96      0.93        45
               4       0.97      0.97      0.97        38
               5       0.98      0.96      0.97        48
               6       1.00      0.98      0.99        52
               7       0.98      0.96      0.97        48
               8       0.91      0.90      0.91        48
               9       0.90      0.96      0.93        47

        accuracy                           0.95       450
       macro avg       0.95      0.95      0.95       450
    weighted avg       0.95      0.95      0.95       450

```python
print("Micro average f1 score: {:.3f}".format(
    f1_score(y_test, pred, average="micro")))
print("Macro average f1 score: {:.3f}".format(
    f1_score(y_test, pred, average="macro")))
```

    Micro average f1 score: 0.951
    Macro average f1 score: 0.952

#### Regression metrics

### Using evaluation metrics in model selection

```python
# default scoring for classification is accuracy
print("Default scoring: {}".format(
    cross_val_score(SVC(), digits.data, digits.target == 9, cv=5)))
# providing scoring="accuracy" doesn't change the results
explicit_accuracy =  cross_val_score(SVC(), digits.data, digits.target == 9,
                                     scoring="accuracy", cv=5)
print("Explicit accuracy scoring: {}".format(explicit_accuracy))
roc_auc =  cross_val_score(SVC(), digits.data, digits.target == 9,
                           scoring="roc_auc", cv=5)
print("AUC scoring: {}".format(roc_auc))
```

    Default scoring: [0.975 0.992 1.    0.994 0.981]
    Explicit accuracy scoring: [0.975 0.992 1.    0.994 0.981]
    AUC scoring: [0.997 0.999 1.    1.    0.984]

```python
res = cross_validate(SVC(), digits.data, digits.target == 9,
                     scoring=["accuracy", "roc_auc", "recall_macro"],
                     return_train_score=True, cv=5)
display(pd.DataFrame(res))
```

      fit_time
      score_time
      test_accuracy
      train_accuracy
      test_roc_auc
      train_roc_auc
      test_recall_macro
      train_recall_macro

      0
      0.02
      8.98e-03
      0.97
      0.99
      1.00
      1.0
      0.89
      0.97

      1
      0.02
      7.98e-03
      0.99
      1.00
      1.00
      1.0
      0.96
      0.98

      2
      0.02
      7.98e-03
      1.00
      1.00
      1.00
      1.0
      1.00
      0.98

      3
      0.02
      8.98e-03
      0.99
      1.00
      1.00
      1.0
      0.97
      0.98

      4
      0.02
      7.98e-03
      0.98
      1.00
      0.98
      1.0
      0.90
      0.99

```python
X_train, X_test, y_train, y_test = train_test_split(
    digits.data, digits.target == 9, random_state=0)

# we provide a somewhat bad grid to illustrate the point:
param_grid = {'gamma': [0.0001, 0.01, 0.1, 1, 10]}
# using the default scoring of accuracy:
grid = GridSearchCV(SVC(), param_grid=param_grid)
grid.fit(X_train, y_train)
print("Grid-Search with accuracy")
print("Best parameters:", grid.best_params_)
print("Best cross-validation score (accuracy)): {:.3f}".format(grid.best_score_))
print("Test set AUC: {:.3f}".format(
    roc_auc_score(y_test, grid.decision_function(X_test))))
print("Test set accuracy: {:.3f}".format(grid.score(X_test, y_test)))
```

    Grid-Search with accuracy
    Best parameters: {'gamma': 0.0001}
    Best cross-validation score (accuracy)): 0.976
    Test set AUC: 0.992
    Test set accuracy: 0.973

```python
# using AUC scoring instead:
grid = GridSearchCV(SVC(), param_grid=param_grid, scoring="roc_auc")
grid.fit(X_train, y_train)
print("\nGrid-Search with AUC")
print("Best parameters:", grid.best_params_)
print("Best cross-validation score (AUC): {:.3f}".format(grid.best_score_))
print("Test set AUC: {:.3f}".format(
    roc_auc_score(y_test, grid.decision_function(X_test))))
print("Test set accuracy: {:.3f}".format(grid.score(X_test, y_test)))
```

    Grid-Search with AUC
    Best parameters: {'gamma': 0.01}
    Best cross-validation score (AUC): 0.998
    Test set AUC: 1.000
    Test set accuracy: 1.000

```python
from sklearn.metrics import SCORERS
print("Available scorers:")
print(sorted(SCORERS.keys()))
```

    Available scorers:
    ['accuracy', 'adjusted_mutual_info_score', 'adjusted_rand_score', 'average_precision', 'balanced_accuracy', 'completeness_score', 'explained_variance', 'f1', 'f1_macro', 'f1_micro', 'f1_samples', 'f1_weighted', 'fowlkes_mallows_score', 'homogeneity_score', 'jaccard', 'jaccard_macro', 'jaccard_micro', 'jaccard_samples', 'jaccard_weighted', 'max_error', 'mutual_info_score', 'neg_brier_score', 'neg_log_loss', 'neg_mean_absolute_error', 'neg_mean_gamma_deviance', 'neg_mean_poisson_deviance', 'neg_mean_squared_error', 'neg_mean_squared_log_error', 'neg_median_absolute_error', 'neg_root_mean_squared_error', 'normalized_mutual_info_score', 'precision', 'precision_macro', 'precision_micro', 'precision_samples', 'precision_weighted', 'r2', 'recall', 'recall_macro', 'recall_micro', 'recall_samples', 'recall_weighted', 'roc_auc', 'roc_auc_ovo', 'roc_auc_ovo_weighted', 'roc_auc_ovr', 'roc_auc_ovr_weighted', 'v_measure_score']

### Summary and Outlook

```python

```


```python
!pip install qiskit
from qiskit import QuantumCircuit, assemble, Aer
from qiskit.visualization import plot_histogram, plot_bloch_vector, array_to_latex
```

    Requirement already satisfied: qiskit in /usr/local/lib/python3.7/dist-packages (0.34.2)
    Requirement already satisfied: qiskit-aer==0.10.3 in /usr/local/lib/python3.7/dist-packages (from qiskit) (0.10.3)
    Requirement already satisfied: qiskit-ignis==0.7.0 in /usr/local/lib/python3.7/dist-packages (from qiskit) (0.7.0)
    Requirement already satisfied: qiskit-ibmq-provider==0.18.3 in /usr/local/lib/python3.7/dist-packages (from qiskit) (0.18.3)
    Requirement already satisfied: qiskit-terra==0.19.2 in /usr/local/lib/python3.7/dist-packages (from qiskit) (0.19.2)
    Requirement already satisfied: scipy>=1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.10.3->qiskit) (1.7.3)
    Requirement already satisfied: numpy>=1.16.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-aer==0.10.3->qiskit) (1.21.5)
    Requirement already satisfied: python-dateutil>=2.8.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (2.8.2)
    Requirement already satisfied: requests-ntlm>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (1.1.0)
    Requirement already satisfied: requests>=2.19 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (2.23.0)
    Requirement already satisfied: urllib3>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (1.24.3)
    Requirement already satisfied: websocket-client>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-ibmq-provider==0.18.3->qiskit) (1.3.1)
    Requirement already satisfied: retworkx>=0.8.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ignis==0.7.0->qiskit) (0.11.0)
    Requirement already satisfied: setuptools>=40.1.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-ignis==0.7.0->qiskit) (57.4.0)
    Requirement already satisfied: ply>=3.10 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (3.11)
    Requirement already satisfied: tweedledum<2.0,>=1.1 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (1.1.1)
    Requirement already satisfied: symengine>=0.8 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (0.9.2)
    Requirement already satisfied: python-constraint>=1.4 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (1.4.0)
    Requirement already satisfied: stevedore>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (3.5.0)
    Requirement already satisfied: sympy>=1.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (1.7.1)
    Requirement already satisfied: dill>=0.3 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (0.3.4)
    Requirement already satisfied: psutil>=5 in /usr/local/lib/python3.7/dist-packages (from qiskit-terra==0.19.2->qiskit) (5.4.8)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.8.0->qiskit-ibmq-provider==0.18.3->qiskit) (1.15.0)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (2.10)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (2021.10.8)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19->qiskit-ibmq-provider==0.18.3->qiskit) (3.0.4)
    Requirement already satisfied: cryptography>=1.3 in /usr/local/lib/python3.7/dist-packages (from requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.3->qiskit) (36.0.2)
    Requirement already satisfied: ntlm-auth>=1.0.2 in /usr/local/lib/python3.7/dist-packages (from requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.3->qiskit) (1.5.0)
    Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.7/dist-packages (from cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.3->qiskit) (1.15.0)
    Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.12->cryptography>=1.3->requests-ntlm>=1.1.0->qiskit-ibmq-provider==0.18.3->qiskit) (2.21)
    Requirement already satisfied: importlib-metadata>=1.7.0 in /usr/local/lib/python3.7/dist-packages (from stevedore>=3.0.0->qiskit-terra==0.19.2->qiskit) (4.11.2)
    Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from stevedore>=3.0.0->qiskit-terra==0.19.2->qiskit) (5.8.1)
    Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra==0.19.2->qiskit) (3.10.0.2)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=1.7.0->stevedore>=3.0.0->qiskit-terra==0.19.2->qiskit) (3.7.0)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy>=1.3->qiskit-terra==0.19.2->qiskit) (1.2.1)

```python
qc = QuantumCircuit(1,1)
qc.h(0)
qc.draw()
```

     ┌───┐
  q: ┤ H ├
     └───┘
c: 1/═════

```python
from math import sqrt
initial_state = [1/sqrt(2), -1/sqrt(2)]
qc.initialize(initial_state,0)
##qc.measure(0,0)
sim = Aer.get_backend("aer_simulator")
qc.save_statevector()
```

    <qiskit.circuit.instructionset.InstructionSet at 0x7f46286e3500>

```python
from qiskit import execute
result = execute(qc, sim).result()
##array_to_latex(result.get_statevector())
```

```python
counts = result.get_counts()
plot_histogram(counts)
```

```python

import numpy as np

qc = QuantumCircuit(1,1) # create quantum circuit with one qubit and one classical bit
initial_state = [1,1]/np.sqrt(2) # define our initial state as |+> (change to [1,-1]/np.sqrt(2) for |->)
qc.initialize(initial_state, 0) # add initialization operation to qubit 0
qc.measure(0,0) # measure qubit 0 and store the results in classical bit 0

backend = Aer.get_backend('qasm_simulator') # Tell Qiskit how to simulate our circuit
counts = execute(qc,backend).result().get_counts() # Do the simulation, returning the counts
plot_histogram(counts) # Display the results in a histogram

```


```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import scipy.stats as stats
from tabulate import tabulate
```

# 数理统计基础
> 数据样本分析从分析数据样本分布开始

## 数据分布分析

### 样本

| Name     | Salary      |
|----------|-------------|
| Dan      | 50,000      |
| Joann    | 54,000      |
| Pedro    | 50,000      |
| Rosie    | 189,000     |
| Ethan    | 55,000      |
| Vicky    | 40,000      |
| Frederic | 59,000      |

#### 均值

\begin{equation}\bar{x} = \frac{\displaystyle\sum_{i=1}^{n}x_{i}}{n}\end{equation}

\begin{equation}\bar{x} = \frac{50000+54000+50000+189000+55000+40000+59000}{7} = 71000 \end{equation}

```python
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

df['Salary'].mean()
```

    71000.0

#### 中值

把数据进行排序

| Salary      |
|-------------|
| 40,000      |
| 50,000      |
| 50,000      |
| 54,000      |
| 55,000      |
| 59,000      |
| 189,000     |

取中间的位置${\frac{n+1}{2} = 4}$

| Salary      |
|-------------|
| 40,000      |
| 50,000      |
| 50,000      |
|***>54,000*** |
| 55,000      |
| 59,000      |
| 189,000     |

如果是奇数个元素，则

\begin{equation}\frac{n}{2} \;\;\;\;and \;\;\;\; \frac{n}{2} + 1\end{equation}

```python
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

df['Salary'].median()
```

    54000.0

#### 众数

如果选出现频率最高的数字时

| Salary      |
|-------------|
| 40,000      |
|***>50,000***|
|***>50,000***|
| 54,000      |
| 55,000      |
| 59,000      |
| 189,000     |

应该是***50,000****

```python

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

df['Salary'].mode()
```

    0    50000
    dtype: int64

##### 多个众数
很显然，众数可能有多个

| Salary      |
|-------------|
| 40,000      |
|***>50,000***|
|***>50,000***|
| 54,000      |
|***>59,000***|
|***>59,000***|
| 189,000     |

```python
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,59000,40000,59000]})

df['Salary'].mode()
```

    0    50000
    1    59000
    dtype: int64

### 分布密度

找到中心点之后，希望获得更多关于分布形状的信息。

先从最大值和最小值开始：

```python
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})
s = df['Salary']
print(tabulate([['min','mode','media','mean','max'],
                [s.min(),s.mode()[0],s.median(),s.mean(),s.max()]]))
```

    -----  -----  -------  -------  ------
    min    mode   media    mean     max
    40000  50000  54000.0  71000.0  189000
    -----  -----  -------  -------  ------

##### 可视化

```python
%matplotlib inline
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

salary = df['Salary']
salary.plot.hist(title='Salary Distribution', color='lightblue', bins=25)
plt.axvline(salary.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(salary.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()
```

***均值*** 和 ***中值*** 用绿色和紫色虚线标出:

- 直方图的高度是对应的样本数
- 可见，大部分样本在40000和55000之间
- 均值高于中值

直方图给出了分布密度的信息，在上图基础上绘制密度曲线：

> [高斯概率密度估计](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gaussian_kde.html?highlight=gaussian_kde#scipy.stats.gaussian_kde)文档

```python
%matplotlib inline
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

salary = df['Salary']
density = stats.gaussian_kde(salary)
n, x, _ = plt.hist(salary, histtype='step', density=True, bins=25)
plt.plot(x, density(x)*5)
plt.axvline(salary.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(salary.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()

```

>概率密度曲线的形状并不对称

- 峰值在左边，右边有一个长尾
  因为右侧的值把均值给拉向右边，所以叫右偏态分布

- 再看一个左偏态分布的例子

| Name     | Hours |
|----------|-------|
| Dan      | 41    |
| Joann    | 40    |
| Pedro    | 36    |
| Rosie    | 30    |
| Ethan    | 35    |
| Vicky    | 39    |
| Frederic | 40    |

```python
%matplotlib inline

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Hours':[41,40,36,30,35,39,40]})

hours = df['Hours']
density = stats.gaussian_kde(hours)
n, x, _ = plt.hist(hours, histtype='step', normed=True, bins=25)
plt.plot(x, density(x)*7)
plt.axvline(hours.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(hours.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()
```

- 下面是一个正态分布的例子

| Name     | Grade |
|----------|-------|
| Dan      | 50    |
| Joann    | 50    |
| Pedro    | 46    |
| Rosie    | 95    |
| Ethan    | 50    |
| Vicky    | 5     |
| Frederic | 57    |

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Grade':[50,50,46,95,50,5,57]})

grade = df['Grade']
density = stats.gaussian_kde(grade)
n, x, _ = plt.hist(grade, histtype='step', normed=True, bins=25)
plt.plot(x, density(x)*7.5)
plt.axvline(grade.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(grade.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()
```

一个对称的钟形曲线 The ***均值***, ***中值***, 和众值重合.

这就是最著名的正态分布（高斯分布）

#### 偏度和峰度

通过测量这两个指标获取关于数据分布形状的认识：

```python
%matplotlib inline

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

numcols = ['Salary', 'Hours', 'Grade']
for col in numcols:
    print(df[col].name + ' skewness: ' + str(df[col].skew()))
    print(df[col].name + ' kurtosis: ' + str(df[col].kurt()))
    density = stats.gaussian_kde(df[col])
    n, x, _ = plt.hist(df[col], histtype='step', density=True, bins=25)
    plt.plot(x, density(x)*6)
    plt.show()
    print('\n')
```

### 测量变化

#### 范围

先用最大值减最小值

```python
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

numcols = ['Salary', 'Hours', 'Grade']
for col in numcols:
    print(df[col].name + ' range: ' + str(df[col].max() - df[col].min()))
```

    Salary range: 149000
    Hours range: 11
    Grade range: 90

#### 百分比 四分位数

##### 百分比
例如

\begin{equation}\frac{5}{7} \times 100 \approx 71.4\end{equation}

```python
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(stats.percentileofscore(df['Grade'], 57, 'strict')) # 和下面的weak比较
```

    71.42857142857143

根据是否包含被统计的样本本身，还有另外一种算法

\begin{equation}\frac{6}{7} \times 100 \approx 85.7\end{equation}

```python
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(stats.percentileofscore(df['Grade'], 57, 'weak'))
```

    85.71428571428571

如果要测量50分组的等级，50分有三个样本，把其中一个归类到高分组，把另一个归类到低分组。

\begin{equation}(\frac{4}{7}) \times 100 \approx 57.14\end{equation}

这样可以算出50分组的等级

```python
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(stats.percentileofscore(df['Grade'], 50, 'rank')) # 单独计算得分组里面的一个
print(stats.percentileofscore(df['Grade'], 50, 'weak'))
```

    57.142857142857146

```python
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(tabulate([['strict','weak','rank'],
                [stats.percentileofscore(df['Grade'], 50, 'strict'),
                 stats.percentileofscore(df['Grade'], 50, 'weak'),
                 stats.percentileofscore(df['Grade'], 50, 'rank')]]))
```

    71.42857142857143
    -----------------  -----------------  ------------------
    strict             weak               rank
    28.57142857142857  71.42857142857143  57.142857142857146
    -----------------  -----------------  ------------------

##### 四分位数

以25%，50%，75%为界，把所有样本划分到4个组里：

```python
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})
df['Hours'].quantile([0.25, 0.5, 0.75])
```

    0.25    35.5
    0.50    39.0
    0.75    40.0
    Name: Hours, dtype: float64

```python
%matplotlib inline
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

df['Hours'].plot(kind='box', title='Weekly Hours Distribution', figsize=(10,8))
plt.show()
```

箱线图

- 矩形（箱）代表 ${\frac{1}{4} \to \frac{1}{2} + \frac{1}{2} \to \frac{3}{4}}$ 两部分
- 箱上、下的线分别代表 ${0 \to \frac{1}{4}}$ 和 ${\frac{3}{4} \to 1}$区间
- 中间的绿线代表中值

#### 例外

> 例外往往格外引起注意

```python
%matplotlib inline

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

df['Salary'].plot(kind='box', title='Salary Distribution', figsize=(10,8))
plt.show()
```

解释

例外是距离中值点太远的反常数据。例外总需要解释，

- 检查数据源有没有错
- 排除例外
- 接受例外，接受反常是存在的

```python
%matplotlib inline

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

df['Salary'].plot(kind='box', title='Salary Distribution', figsize=(10,8), showfliers=False)
plt.show()
```

```python
%matplotlib inline

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})
# 考试分数的巨大差别是可以解释的
df['Grade'].plot(kind='box', title='Grade Distribution', figsize=(10,8))
plt.show()
```

```python
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

# 人多了，“例外”就不突出了
df['Grade'].plot(kind='box', title='Grade Distribution', figsize=(10,8))
plt.show()
```

#### 方差和标准差

##### 方差
>方差测量偏离期望的程度

\begin{equation}\sigma^{2} = \frac{\displaystyle\sum_{i=1}^{N} (X_{i} -\mu)^{2}}{N}\end{equation}

```python
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

df['Grade'].var()
```

    685.6190476190476

##### 标准差

\begin{equation}\sigma = \sqrt{\frac{\displaystyle\sum_{i=1}^{N} (X_{i} -\mu)^{2}}{N}}\end{equation}

- 平方可以去掉负值
- 平方根可以统一量纲

```python
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

df['Grade'].std()
```

    26.184328282754315

#### 正态分布的标准差

正态分布是最重要的分布，标准差是一个正态分布不可缺少的测量指标：期望、方差。

```python
%matplotlib inline

Udf = pd.DataFrame(np.random.randn(100000, 1), columns=['Grade'])

grade = df['Grade']
density = stats.gaussian_kde(grade)
n, x, _ = plt.hist(grade, color='lightgrey', density=True, bins=100)
plt.plot(x, density(x))

s = df['Grade'].std()
m = df['Grade'].mean()

x1 = [m-s, m+s]                                # 1 𝛔
y1 = [0.25, 0.25]
plt.plot(x1,y1, color='magenta')
plt.annotate('1s (68.26%)', (x1[1],y1[1]))

x2 = [m-(s*2), m+(s*2)]                        # 2 𝛔
y2 = [0.05, 0.05]
plt.plot(x2,y2, color='green')
plt.annotate('2s (95.45%)', (x2[1],y2[1]))

x3 = [m-(s*3), m+(s*3)]                        # 3 𝛔
y3 = [0.005, 0.005]
plt.plot(x3,y3, color='orange')
plt.annotate('3s (99.73%)', (x3[1],y3[1]))

plt.axvline(grade.mean(), color='grey', linestyle='dashed', linewidth=1)
plt.show()
```

百分之六十八，九十五和九十九
> 看起来很难记住，其实很容易

#### 归一化

\begin{equation}Z = \frac{x - \mu}{\sigma}\end{equation}

比如

\begin{equation}Z = \frac{95 - 50.43}{26.184} = 1.702\end{equation}

### 总结数据分布
>函数名：describe

```python
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})
print(df.describe())
```

                  Salary      Hours      Grade
    count       7.000000   7.000000   7.000000
    mean    71000.000000  35.428571  50.428571
    std     52370.475143   8.423324  26.184328
    min     40000.000000  17.000000   5.000000
    25%     50000.000000  35.500000  48.000000
    50%     54000.000000  39.000000  50.000000
    75%     57000.000000  40.000000  53.500000
    max    189000.000000  41.000000  95.000000

# 数据比较

通过比较发现趋势和关系

## 单变量数据

```python
%matplotlib inline

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

plt.figure()
df['Grade'].plot( kind='box', title='Grade Distribution')
plt.figure()
df['Grade'].hist(bins=9)
plt.show()
print(df.describe())
print('median: ' + str(df['Grade'].median()))
```

## 多变量数据

```python

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

df[['Name', 'Salary', 'Hours', 'Grade']]
```

      Name
      Salary
      Hours
      Grade

      0
      Dan
      50000
      41
      50

      1
      Joann
      54000
      40
      50

      2
      Pedro
      50000
      36
      46

      3
      Rosie
      189000
      17
      95

      4
      Ethan
      55000
      35
      50

      5
      Vicky
      40000
      39
      5

      6
      Frederic
      59000
      40
      57

      7
      Jimmie
      42000
      45
      42

      8
      Rhonda
      47000
      41
      26

      9
      Giovanni
      78000
      35
      72

      10
      Francesca
      119000
      30
      78

      11
      Rajab
      95000
      33
      60

      12
      Naiyana
      49000
      38
      40

      13
      Kian
      29000
      47
      17

      14
      Jenny
      130000
      24
      85

### 数据归一化

>要使数据有可比性

将数据转换到0和1之间，这样才能比较不同的特征。

```python
def MinMaxScaler(v):
    min = v.min()
    max = v.max()
    scale = max - min
    return [(i - min) / scale for i in v]

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

# 归一化
df['Salary'], df['Hours'], df['Grade'] = tuple(map(MinMaxScaler, [df['Salary'], df['Hours'], df['Grade']]))

df.plot(kind='box', title='Distribution', figsize = (10,8))
plt.show()
```

### 对于双变量数据画散点图

```python
%matplotlib inline

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

df.plot(kind='scatter', title='Grade vs Hours', x='Grade', y='Salary')
plt.show()

```

在成绩和收入之间，我们看到了某种趋势。

```python
%matplotlib inline

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

df.plot(kind='scatter', title='Grade vs Salary', x='Grade', y='Salary')

# 用一条直线来模拟这种趋势
plt.plot(np.unique(df['Grade']), np.poly1d(np.polyfit(df['Grade'], df['Salary'], 1))(np.unique(df['Grade'])))

plt.show()
```

### 相关性

让我们用一个统计指标来描述相关性

- 这是个-1到1之间的数值
- 大于0代表正相关
- 小于0代表负相关
- 越接近0相关程度越低
- 0代表不相关

\begin{equation}r_{x,y} = \frac{\displaystyle\sum_{i=1}^{n} (x_{i} -\bar{x})(y_{i} -\bar{y})}{\sqrt{\displaystyle\sum_{i=1}^{n} (x_{i} -\bar{x})^{2}(y_{i} -\bar{y})^{2}}}\end{equation}

```python
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

# 调用方法 corr
print(df['Grade'].corr(df['Salary']))
```

    0.8149286388911882

0.8多，说明成绩和收入之间存在很大的正相关性。

那么成绩和学习时间呢？

```python
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

r = df['Grade'].corr(df['Hours'])
print('Correlation: ' + str(r))

df.plot(kind='scatter', title='Grade vs Hours', x='Grade', y='Hours')

plt.plot(np.unique(df['Grade']), np.poly1d(np.polyfit(df['Grade'], df['Hours'], 1))(np.unique(df['Grade'])))
plt.show()
```

学习时间越长，成绩越好，照理如此。

请看直线下方的两个异常点，第一名的同学其实和中上等的同学学习时间一样长，我们用有天赋解释。

那最后一名呢？他好像根本没有学习，所以取得这样的成绩并不意外。

## 最小二乘拟合

让我们看看*numpy.**polyfit***

对方程
\begin{equation}y = mx + b\end{equation}

可解释为：x和y是两个变量，m是斜率，b是截距。假设我们猜测一条直线，如何计算它和散点之间的差距呢？我们可以让m是使得差距最小的值。

\begin{equation}m = \frac{n(\sum{xy}) - (\sum{x})(\sum{y})}{n(\sum{x^{2}})-(\sum{x})^{2}}\end{equation}

然后根据m计算b

\begin{equation}b = \frac{\sum{y} - m(\sum{x})}{n}\end{equation}

对于数据

| Name     | Study | Grade |
|----------|-------|-------|
| Dan      | 1     | 50    |
| Joann    | 0.75  | 50    |
| Pedro    | 0.6   | 46    |
| Rosie    | 2     | 95    |
| Ethan    | 1     | 50    |
| Vicky    | 0.2   | 5     |
| Frederic | 1.2   | 57    |

```python
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

df['x2'] = df['Grade']**2
df['xy'] = df['Grade'] * df['Salary']
x = df['Grade'].sum()
y = df['Salary'].sum()
x2 = df['x2'].sum()
xy = df['xy'].sum()
n = df['Grade'].count()
m = ((n*xy) - (x*y))/((n*x2)-(x**2))  # 公式
b = (y - (m*x))/n
df['fx'] = (m*df['Grade']) + b
df['error'] = df['fx'] - df['Salary']

print('slope: ' + str(m))
print('y-intercept: ' + str(b))

df.plot(kind='scatter', title='Grade vs Salary Regression', x='Grade', y='Salary', color='red')

plt.plot(df['Grade'],df['fx'])

plt.show()

# df[['Grade', 'Salary', 'fx', 'error']]
```

```python
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

# 线形回归方法
m, b, r, p, se = stats.linregress(df['Grade'], df['Salary'])

df['fx'] = (m*df['Grade']) + b
df['error'] = df['fx'] - df['Salary']

df.plot(kind='scatter', title='Grade vs Salary Regression', x='Grade', y='Salary', color='red')

plt.plot(df['Grade'],df['fx'])

plt.show()
```

```python
df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

df = df[(df['Grade'] > 5) & (df['Grade'] < 95)] # 去掉异常

m, b, r, p, se = stats.linregress(df['Grade'], df['Salary'])

df['fx'] = (m*df['Grade']) + b
df['error'] = df['fx'] - df['Salary']

print('slope: ' + str(m))
print('y-intercept: ' + str(b))

df.plot(kind='scatter', title='Grade vs Salary Regression', x='Grade', y='Salary', color='red')

plt.plot(df['Grade'],df['fx'])

plt.show()
```

可以用拟合出来的曲线进行预测，比如一位同学的成绩是62，他最有可能的收入大概是8万多

\begin{equation}f(x) = (1424.50\times62) - 7822.24 \approx 80,497 \end{equation}

# 概率

## 基本概念

- 实验，结果不确定
- 样本空间，所有实验的所有可能结果
- 事件，包含若干样本的实验
- 概率，事件发生的概率

\begin{equation}\text{事件的概率} = \frac{\text{事件样本数}}{\text{样本空间样本数}} \end{equation}

### 记号

\begin{equation}P(A) = 0.167 \end{equation}

### 补

\begin{equation}P(A) = 1 - P(A') \end{equation}

### 偏

A 晴, B 阴, C 雨

\begin{equation}P(A)=0.6\;\;\;\; P(B)=0.3\;\;\;\; P(C)=0.1 \end{equation}

**A** 的补：不请

\begin{equation}P(A') = 1 - P(A) = 1 - 0.6 = 0.4 \end{equation}

或者

\begin{equation}P(A') = P(B) + P(C) = 0.3 + 0.1 = 0.4 \end{equation}

## 条件概率和独立

事件之间的依赖关系可以是：
- *独立*
- *依赖*
- *互斥*

### 独立事件

抛硬币实验

```python
import random

heads_tails = [0,0]

# 模拟抛硬币10000次
trials = 10000
trial = 0
while trial < trials:
    trial = trial + 1
    toss = random.randint(0,1)
    heads_tails[toss] = heads_tails[toss] + 1

print (heads_tails)

from matplotlib import pyplot as plt
plt.figure(figsize=(5,5))
plt.pie(heads_tails, labels=['heads', 'tails'])
plt.legend()
plt.show()
```

### 独立事件乘法律

❂ = 1/2

❂❂ = 1/2 x 1/2

❂❂❂ = 1/2 x 1/2 x 1/2

```python
import random
h3 = 0
results = []
trials = 10000
trial = 0

while trial < trials:
    trial = trial + 1
    result = ['H' if random.randint(0,1) == 1 else 'T',
              'H' if random.randint(0,1) == 1 else 'T',
              'H' if random.randint(0,1) == 1 else 'T']
    results.append(result)
    h3 = h3 + int(result == ['H','H','H'])

print ("%.2f%%" % ((h3/trials)*100))

# print (results)
```

    12.70%

#### 概率树

                         ____H(0.5)  : 0.5 x 0.5 x 0.5 = 0.125
                        /
                   ____H(0.5)
                  /     \____T(0.5)  : 0.5 x 0.5 x 0.5 = 0.125
                 /
              __H(0.5)   ____H(0.5)  : 0.5 x 0.5 x 0.5 = 0.125
             /   \      /
            /     \____T(0.5)
           /            \____T(0.5)  : 0.5 x 0.5 x 0.5 = 0.125
          /
    _____/              _____H(0.5)  : 0.5 x 0.5 x 0.5 = 0.125
         \             /
          \        ___H(0.5)
           \      /    \_____T(0.5)  : 0.5 x 0.5 x 0.5 = 0.125
            \    /
             \__T(0.5)  _____H(0.5)  : 0.5 x 0.5 x 0.5 = 0.125
                 \     /
                  \___T(0.5)
                       \_____T(0.5)  : 0.5 x 0.5 x 0.5 = 0.125
                                                         _____
                                                          1.0

#### 交集和并集

##### 交

\begin{equation}P(A \cap B) = P(A) \times P(B) \end{equation}

##### 并

\begin{equation}P(A \cup B) = P(A) + P(B) - P(A \cap B)\end{equation}

### 条件概率

\begin{equation}P(B|A) = \frac{P(A \cap B)}{P(A)}\end{equation}

或者

\begin{equation}P(A \cap B) = P(A) \times P(B|A)\end{equation}

### 互斥事件不独立

\begin{equation}P(A \cap B) = 0\end{equation}

\begin{equation}P(A \cup B) = P(A) + P(B)\end{equation}

## 二项分布

### 二项式系数记号${C^{k}_{n}}$ 或者 ${\begin{pmatrix} n \\ k\end{pmatrix}}$

\begin{equation}\begin{pmatrix} n \\ k\end{pmatrix} = \frac{n!}{k!(n-k)!}\end{equation}

一枚硬币扔三次，求概率分布

\begin{equation}P(x=2) = \frac{C^{2}_{3}}{8} =  \frac{3!}{2!(3 - 2)!8} = \frac{3}{8}\end{equation}

```python
from scipy import special as sps

trials = 3

possibilities = 2**trials
x = np.array(range(0, trials+1))

p = np.array([sps.comb(trials, i, exact=True)/possibilities for i in x])

plt.xlabel('Successes')
plt.ylabel('Probability')
plt.bar(x, p)
plt.show()
```

#### 假设事件发生的概率不是${\frac{1}{2}}$ （有偏）

可以把二项式分布记为：

\begin{equation}P(x=k)  = \begin{pmatrix} n \\ k\end{pmatrix}p^{k}(1-p)^{(n-k)}\end{equation}

```python
from scipy.stats import binom

n = 5
p = 0.25
x = np.array(range(0, n+1))

prob = np.array([binom.pmf(k, n, p) for k in x])

plt.xlabel('x')
plt.ylabel('Probability')
plt.bar(x, prob)            # 可以在图中看到偏离的情况
plt.show()
```

```python
n = 100
p = 0.25
x = np.array(range(0, n+1))

prob = np.array([binom.pmf(k, n, p) for k in x])

plt.xlabel('x')
plt.ylabel('Probability')
plt.bar(x, prob)
plt.show()
```

### 二项式分布的统计特征 （期望和方差）

\begin{equation}\mu  = np\end{equation}

\begin{equation}\mu  = 100 \times 0.25 = 25\end{equation}

\begin{equation}\sigma^{2}  = np(1-p)\end{equation}

\begin{equation}\sigma  = \sqrt{np(1-p)}\end{equation}

\begin{equation}\sigma  = \sqrt{18.75} \approx 4.33 \end{equation}

```python
from scipy.stats import binom

n = 100
p = 0.25

print(binom.mean(n,p))
print(binom.var(n,p))
print(binom.std(n,p))
```

    25.0
    18.75
    4.330127018922194

# 采样和分布

\begin{equation}\mu_{\hat{p}} = \hat{p}\end{equation}

\begin{equation}\sigma_{\hat{p}} = \sqrt{\hat{p}(1-\hat{p})}\end{equation}

```python
searches = np.array([0,1,0,0,1,0,0,0,0,0,0,0,1,0,0,0])

plt.xlabel('Search Results')
plt.ylabel('Frequency')
plt.hist(searches)
plt.show()
print('Mean: ' + str(np.mean(searches)))
print('StDev: ' + str(np.std(searches)))
```

### 对数据

| Sample | Result |
|--------|--------|
| p̂1| 0.1875 |
| p̂2| 0.2500 |
| p̂3| 0.3125 |
| p̂4| 0.1875 |
| p̂5| 0.1250 |
| p̂6| 0.3750 |
| p̂7| 0.2500 |
| p̂8| 0.1875 |
| p̂9| 0.3125 |
| p̂10| 0.2500 |
| p̂11| 0.2500 |
| p̂12| 0.3125 |

```python

searches = np.array([0.1875,0.25,0.3125,0.1875,0.125,0.375,0.25,0.1875,0.3125,0.25,0.25,0.3125])

plt.xlabel('Search Results')
plt.ylabel('Frequency')
plt.hist(searches)
plt.show()
```

#### 中心极限定理

对于很多样本数的情况，二项式分布趋近于正态分布。

再看一个例子

```python
n, p, s = 100, 0.25, 10000
df = pd.DataFrame(np.random.binomial(n,p,s)/n, columns=['p-hat'])

means = df['p-hat']
means.plot.hist(title='Simulated Sampling Distribution')
plt.show()
print ('Mean: ' + str(means.mean()))
print ('Std: ' + str(means.std()))
```

### 样本的均值和方差

\begin{equation}\sigma_{\hat{p}} = \sqrt{\frac{p(1-p)}{n}}\end{equation}

把样本数从16增加为100

\begin{equation}\sigma_{\hat{p}} = \sqrt{\frac{0.25 \times 0.75}{16}} \approx 0.11\end{equation}

\begin{equation}\sigma_{\hat{p}} = \sqrt{\frac{0.25 \times 0.75}{100}} \approx 0.043\end{equation}

有了均值和方差，就可以应用百分之六十八九十五和九十九

比如95.4%的样本都会在范围0.25 ± 0.086内， 称为置信区间。

```python
n, p, s = 100, 0.25, 10000
df = pd.DataFrame(np.random.binomial(n,p,s)/n, columns=['p-hat'])

means = df['p-hat']
m = means.mean()
sd = means.std()
moe1 = m - (sd * 2)
moe2 = m + (sd * 2)

means.plot.hist(title='Simulated Sampling Distribution')

plt.axvline(m, color='red', linestyle='dashed', linewidth=2)
plt.axvline(moe1, color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(moe2, color='magenta', linestyle='dashed', linewidth=2)
plt.show()
```

### 用样本均值生成样本分布

下例展示多个随机变量组合的效果

| Sample | Weights |
|--------|---------|
| 1      | [4.020992,2.143457,2.260409,2.339641,4.699211] |
| 2      | [3.38532,4.438345,3.170228,3.499913,4.489557] |
| 3      | [3.338228,1.825221,3.53633,3.507952,2.698669] |
| 4      | [2.992756,3.292431,3.38148,3.479455,3.051273] |
| 5      | [2.969977,3.869029,4.149342,2.785682,3.03557] |
| 6      | [3.138055,2.535442,3.530052,3.029846,2.881217] |
| 7      | [1.596558,1.486385,3.122378,3.684084,3.501813] |
| 8      | [2.997384,3.818661,3.118434,3.455269,3.026508] |
| 9      | [4.078268,2.283018,3.606384,4.555053,3.344701] |
| 10     | [2.532509,3.064274,3.32908,2.981303,3.915995] |
| 11     | [4.078268,2.283018,3.606384,4.555053,3.344701] |
| 12     | [2.532509,3.064274,3.32908,2.981303,3.915995] |

计算均值：

| Sample | Mean Weight |
|--------|---------|
| x̄1 | 3.092742  |
| x̄2 | 3.7966726 |
| x̄3 | 2.98128   |
| x̄4 | 3.239479  |
| x̄5 | 3.36192   |
| x̄6 | 3.0229224 |
| x̄7 | 2.6782436 |
| x̄8 | 3.2832512 |
| x̄9 | 3.5734848 |
| x̄10 | 3.1646322 |
| x̄11 | 3.5734848 |
| x̄12 | 3.1646322 |

```python

meanweights = np.array([3.092742,
                        3.7966726,
                        2.98128,
                        3.239479,
                        3.36192,
                        3.0229224,
                        2.6782436,
                        3.2832512,
                        3.5734848,
                        3.1646322,
                        3.5734848,
                        3.1646322])

plt.xlabel('Mean Weights')
plt.ylabel('Frequency')
plt.hist(meanweights, bins=6)
plt.show()

print('Mean: ' + str(meanweights.mean()))
print('Std: ' + str(meanweights.std()))
```

```python
mu, sigma, n = 3.2, 1.2, 500
samples = list(range(0, 10000))   # 比较久运行时间，几分钟

data = np.array([])

sampling = np.array([])

for s in samples:
    sample = np.random.normal(mu, sigma, n)
    data = np.append(data,sample)
    sampling = np.append(sampling,sample.mean())

df = pd.DataFrame(sampling, columns=['mean'])

means = df['mean']
means.plot.hist(title='Simulated Sampling Distribution', bins=100)
plt.show()

print('Sample Mean: ' + str(data.mean()))
print('Sample StdDev: ' + str(data.std()))
print ('Sampling Mean: ' + str(means.mean()))
print ('Sampling StdErr: ' + str(means.std()))
```

## 置信区间

| 置信度  | 标准分数 |
|-------------|---------|
| 90%         | 1.645   |
| 95%         | 1.96    |
| 99%         | 2.576   |

\begin{equation}MoE = 0.053 \times 1.96 = 0.10388 \end{equation}

当样本趋近无限大时，辨准差会趋近0

\begin{equation}\lim_{n \to \infty} \frac{\sigma}{\sqrt{n}} = 0\end{equation}

```python
mu, sigma, n = 3.2, 1.2, 500
samples = list(range(0, 10000))

data = np.array([])

sampling = np.array([])

for s in samples:
    sample = np.random.normal(mu, sigma, n)
    data = np.append(data,sample)
    sampling = np.append(sampling,sample.mean())

df = pd.DataFrame(sampling, columns=['mean'])

means = df['mean']
m = means.mean()
sd = means.std()
ci = stats.norm.interval(0.95, m, sd)

means.plot.hist(title='Simulated Sampling Distribution', bins=100)
plt.axvline(m, color='red', linestyle='dashed', linewidth=2)
plt.axvline(ci[0], color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(ci[1], color='magenta', linestyle='dashed', linewidth=2)
plt.show()

print ('Sampling Mean: ' + str(m))
print ('Sampling StdErr: ' + str(sd))
print ('95% Confidence Interval: ' + str(ci))
```

# 假设检验

## 单样本单侧测试

让学生对老师的授课打分，从-5 到 5.

生成50个随机样本：

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

np.random.seed(123)
lo = np.random.randint(-5, -1, 6)
mid = np.random.randint(0, 3, 38)
hi = np.random.randint(4, 6, 6)
sample = np.append(lo,np.append(mid, hi))
print("Min:" + str(sample.min()))
print("Max:" + str(sample.max()))
print("Mean:" + str(sample.mean()))

plt.hist(sample)
plt.show()
```

看起来评价是正面的，怎么知道这代表了全体同学的意见呢？

我们把假设分成正面和负面两个

\begin{equation}H_{0}: \mu \le 0 \\ H_{1}: \mu > 0 \end{equation}

假设 ${H_{0}}$ 是正确的，看看它的t检验是否落在置信区间里。

```python
pop = np.random.normal(0, 1.15, 100000)
plt.hist(pop, bins=100)
plt.axvline(pop.mean(), color='yellow', linestyle='dashed', linewidth=2)
plt.show()
```

\begin{equation}t = \frac{\bar{x} - \mu}{\frac{s}{\sqrt{n}}} \end{equation}

**x̄** 样本均值, **μ** 均值, **s** 标准差, and **n** 样本数. 当样本数大于30时，t-score接近z-score

```python
# T-Test
t,p = stats.ttest_1samp(sample, 0)

p1 = '%f' % (p/2)
print ("t-statistic:" + str(t))
print("p-value:" + str(p1))

ci = stats.norm.interval(0.90, 0, 1.15) # 分布（0, 1.15) 90%置信区间
plt.hist(pop, bins=100)

plt.axvline(pop.mean(), color='yellow', linestyle='dashed', linewidth=2)

plt.axvline(ci[1], color='red', linestyle='dashed', linewidth=2)

plt.axvline(pop.mean() + t*pop.std(), color='magenta', linestyle='dashed', linewidth=2)
plt.show()
```

黄线是均值，红线是临界线，紫线是P-Value

紫线出了置信区间，所以假设${H_{0}}$不成立。

## 双侧检验

对假设：

\begin{equation}H_{0}: \mu = 0 \\ H_{1}: \mu \neq 0 \end{equation}

```python
t,p = stats.ttest_1samp(sample, 0)
print ("t-statistic:" + str(t))

print("p-value:" + '%f' % p)

ci = stats.norm.interval(0.95, 0, 1.15)
plt.hist(pop, bins=100)

plt.axvline(pop.mean(), color='yellow', linestyle='dashed', linewidth=2)

plt.axvline(ci[0], color='red', linestyle='dashed', linewidth=2)
plt.axvline(ci[1], color='red', linestyle='dashed', linewidth=2)

plt.axvline(pop.mean() - t*pop.std(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(pop.mean() + t*pop.std(), color='magenta', linestyle='dashed', linewidth=2)
plt.show()
```

同样的，假设${H_{0}}$不成立。

## 双样本检验

假设有两组学生，一组已经上过课程，另一组没有，有如下假设：

\begin{equation}H_{0}: \mu_{1} \le \mu_{2} \\ H_{1}: \mu_{1} > \mu_{2} \end{equation}

```python
np.random.seed(123)
nonMath = np.random.normal(66.0, 1.5, 100)
math = np.random.normal(66.55, 1.5, 100)
print("non-math sample mean:" + str(nonMath.mean()))
print("math sample mean:" + str(math.mean()))

t,p = stats.ttest_ind(math, nonMath)

p1 = '%f' % (p/2)
print("t-statistic:" + str(t))
print("p-value:" + str(p1))

pop = np.random.normal(nonMath.mean(), nonMath.std(), 100000)

ci = stats.norm.interval(0.90, nonMath.mean(), nonMath.std())
plt.hist(pop, bins=100)

plt.axvline(pop.mean(), color='yellow', linestyle='dashed', linewidth=2)

plt.axvline(ci[1], color='red', linestyle='dashed', linewidth=2)

plt.axvline(pop.mean() + t*pop.std(), color='magenta', linestyle='dashed', linewidth=2)
plt.show()
```

结论是上过课的人并不比没上过的强。

## 配对检测

上例的两组数据是无关的，如果他们相互依赖呢？

比如不是两组不同学生，而是期中和期末考试。

```python
np.random.seed(123)
midTerm = np.random.normal(59.45, 1.5, 100)
endTerm = np.random.normal(60.05, 1.5, 100)

t,p = stats.ttest_rel(endTerm, midTerm)

p1 = '%f' % (p/2)
print("t-statistic:" + str(t))
print("p-value:" + str(p1))

pop = np.random.normal(midTerm.mean(), midTerm.std(), 100000)

ci = stats.norm.interval(0.90, midTerm.mean(), midTerm.std())
plt.hist(pop, bins=100)

plt.axvline(pop.mean(), color='yellow', linestyle='dashed', linewidth=2)

plt.axvline(ci[1], color='red', linestyle='dashed', linewidth=2)

plt.axvline(pop.mean() + t*pop.std(), color='magenta', linestyle='dashed', linewidth=2)
plt.show()
```

期末成绩好于期中成绩。


```julia
using ForwardDiff
using PyPlot
using NLopt
using Ipopt
using JuMP
using FixedSizeArrays
using ProfileView
using GeometryTypes
using LCMGL
import ForwardDiff: value
```

```julia
import SoftRobots
```

    WARNING: New definition
        +(MultiPoly.MPoly...) at /Users/rdeits/.julia/v0.4/MultiPoly/src/mpolyarithmetic.jl:2
    is ambiguous with:
        +(Lazy.List...) at /Users/rdeits/.julia/v0.4/Lazy/src/Lazy.jl:24.
    To fix, define
        +()
    before the new definition.
    INFO: Recompiling stale cache file /Users/rdeits/.julia/lib/v0.4/SpatialFields.ji for module SpatialFields.

```julia
reload("SpatialFields")
reload("SoftRobots")
```

    INFO: Recompiling stale cache file /Users/rdeits/.julia/lib/v0.4/SpatialFields.ji for module SpatialFields.
    WARNING: replacing module SpatialFields
    WARNING: Method definition convert(Type{Array{#T<:Any, 2}}, Array{FixedSizeArrays.Point{#N<:Any, #T<:Any}, 1}) in module SpatialFields at /Users/rdeits/.julia/v0.4/SpatialFields/src/polynomial.jl:46 overwritten in module SpatialFields at /Users/rdeits/.julia/v0.4/SpatialFields/src/polynomial.jl:48.
    WARNING: replacing module SoftRobots
    WARNING: Method definition convert(Type{Array{#T<:Any, 2}}, Array{FixedSizeArrays.Point{#N<:Any, #PointType<:Any}, 1}) in module SoftRobots at /Users/rdeits/.julia/v0.4/SoftRobots/src/drawing.jl:2 overwritten in module SoftRobots at /Users/rdeits/.julia/v0.4/SoftRobots/src/drawing.jl:2.
    WARNING: Method definition convert(Type{Array{#T<:Any, 2}}, Array{GeometryTypes.Face{#N<:Any, #FaceType<:Any, #Offset<:Any}, 1}) in module SoftRobots at /Users/rdeits/.julia/v0.4/SoftRobots/src/drawing.jl:12 overwritten in module SoftRobots at /Users/rdeits/.julia/v0.4/SoftRobots/src/drawing.jl:12.

```julia
robot, state = SoftRobots.tetrahedron()
world = SoftRobots.World3D()
SoftRobots.update_barrier!(robot, state)
push!(world.objects, robot)
world_state = Dict(robot => state)
```

    Dict{SoftRobots.SoftRobot{Int64,0},SoftRobots.SoftRobotState{Float64}} with 1 entry:
      SoftRobots.SoftRobot{In… => SoftRobots.SoftRobotState{Float64}(4-elementArray…

```julia
function potential_energy{T}(robot, state::SoftRobots.SoftRobotState{T})
    energy = zero(T)
    for edge in robot.edges
        energy += 0.5 * edge.stiffness *
         (norm(state.positions[edge.parent] - state.positions[edge.child]) -
          edge.rest_length)^2
    end
    energy
end

Base.convert{T}(::Type{SoftRobots.SoftRobotState{T}}, x::Vector{T}) = SoftRobots.SoftRobotState(SoftRobots.Point{3, T}[x[i:i+2]
    for i in 1:3:(3*length(robot.nodes)-2)],
    [SoftRobots.Point{3, T}(0) for i in 1:length(robot.nodes)])

potential_energy{T}(robot, x::Vector{T}) = potential_energy(robot, convert(SoftRobots.SoftRobotState{T}, x))

draw{T}(robot, x::Vector{T}) = SoftRobots.draw(robot, convert(SoftRobots.SoftRobotState{T}, x))

value(x::Real) = x
```

    value (generic function with 5 methods)

\begin{align}
f(x) \approx f(x_0) + \nabla f(x) |_{x_0} (x - x_0) + \frac{1}{2} (x - x_0)^\top H(x) |_{x_0} (x - x_0)
\end{align}

\begin{align}
f(x) \approx f(x_0) + \nabla f(x) |_{x_0} (x - x_0) + \frac{1}{2} x^\top H(x)|_{x_0} x - x^\top H(x)|_{x_0} x_0 + \frac{1}{2} x_0^\top H(x)|_{x_0} x_0
\end{align}

$$
\nabla f(x) \approx \nabla f(x)|_{x_0} + (x - x_0)^\top H(x)|_{x_0}
$$

$$
\nabla f(x) = 0 \implies (x - x_0)^\top H(x)|_{x_0} = -\nabla f(x)|_{x_0}
$$

$$
x - x_0 = -H^{-1} \nabla f(x)|_{x_0}
$$

$$
x = x_0 - H^{-1} \nabla f(x)|_{x_0}
$$

```julia
function two_body_energy(x)
    0.5 * (norm(x[1] - x[2]) - 1.0)^2 + 0.5 * (x[1] - 0.0)^2
end

# .5 x1^2 - x1 x2 + .5 x2^2

f = two_body_energy
h = ForwardDiff.hessian(f)
g = ForwardDiff.gradient(f)
x = rand(2)
xs = []
energies = []
steps = 1:10
for i = steps
    push!(xs, x)
    push!(energies, f(x))
    gi = g(x)
    hi = h(x)
    x -= inv(hi) * gi
end
plot([x[1] for x in xs], [x[2] for x in xs], "bo:")
```

```julia
x0 = 0.1*rand(3 * (length(robot.nodes)));
```

```julia
g = ForwardDiff.gradient(x -> potential_energy(robot, x))
x = copy(x0)
energies = []
step = 1e-4
for i = 1:50
    gi = g(x)
    x -= step * gi
    draw(robot, x)
    push!(energies, potential_energy(robot, x))
end
plot(energies)
```

```julia
h = ForwardDiff.hessian(x -> potential_energy(robot, x))
g = ForwardDiff.gradient(x -> potential_energy(robot, x))
x = copy(x0)
energies = []
for i = 1:50
    gi = g(x)
    hi = h(x)
    x -= inv(hi) * gi
    draw(robot, x)
    push!(energies, potential_energy(robot, x))
end
plot(energies)
```

```julia
function tower()
    k = 10.
    b = 4.0
    l = 1.0
    m = 0.1
    nodes = SoftRobots.PointMass[]
    positions = SoftRobots.Point{3, Float64}[]

    for z = [1;2]
        x = 0
        y = 0
        push!(nodes, SoftRobots.PointMass(m))
        push!(positions, SoftRobots.Point{3, Float64}(x, y, z))
    end

    for z = 0:1:3
        bulge = 1 - 0.1*(z - 1.5)^2
        for x = -0.5:0.5
            for y = -0.5:0.5
                push!(nodes, SoftRobots.PointMass(m))
                push!(positions, SoftRobots.Point{3, Float64}(x*bulge, y*bulge, z))
            end
        end
    end
    for z in [-0.1; 3.1]
        for x in [-0.1; 0.1]
            for y in [-0.1; 0.1]
#     for (x,y,z) in Any[(0,0,-0.1), (0,0,3.1)]
                push!(nodes, SoftRobots.PointMass(m))
                push!(positions, SoftRobots.Point{3, Float64}(x, y, z))
            end
        end
    end

    num_nodes = length(nodes)
    edges = SoftRobots.DampedSpring[]
    for i = 1:num_nodes
        for j = (i+1):num_nodes
            rest_length = norm(positions[i] - positions[j])
            push!(edges, SoftRobots.DampedSpring(i, j, k, b, rest_length))
        end
    end
    faces = SoftRobots.convex_hull(positions)
    robot = SoftRobots.SoftRobot(nodes, edges, faces)
    velocities = [SoftRobots.Point{3, Float64}(0) for n in robot.nodes]
    state = SoftRobots.SoftRobotState(positions, velocities)
    robot, state
end

```

    tower (generic function with 1 method)

```julia
robot, state = tower()
SoftRobots.draw(robot, state)
```

```julia
# point = [0.0; 0.0; 2.5]
point = [0.25; 0.0; 2.0]

function potential_energy_fixed_base{T}(robot, state::SoftRobots.SoftRobotState{T})
    energy = zero(T)
    for edge in robot.edges
        energy += 0.5 * edge.stiffness *
         (norm(state.positions[edge.parent] - state.positions[edge.child]) -
          edge.rest_length)^2
    end
    for i = 1:2
        energy += 0.5 * 1000 * (state.positions[i][3]-(i))^2
    end

    energy
end

fd_states = []

function cost{T}(x::Vector{T})
    state = convert(SoftRobots.SoftRobotState{T}, x)
    energy = potential_energy_fixed_base(robot, state)
    push!(fd_states, state)
    SoftRobots.update_barrier!(robot, state)
    v = SoftRobots.evaluate(state.barrier, point)
    if any(isnan, v)
        @show v
        @show point
    end
    if v < 0
        energy += 0.5 * 100 * v^2
    end
    energy
end

x0 = collect(destructure(state.positions))
f = cost
g = ForwardDiff.gradient(f, ForwardDiff.AllResults)
x = copy(x0)
energies = []
step = 1e-4
# Profile.clear()
@time for i = 1:100
    gi, results = g(x)
    if any(isnan, gi)
        @show gi
        error("nan in gradient")
    end
    x -= step * gi
    draw(robot, map(value, x))

    state = convert(SoftRobots.SoftRobotState{Float64}, map(value, x))
    mesh = HomogenousMesh(state.positions, robot.faces)

    LCMGLClient("normals") do lcmgl
        normals = decompose(GeometryTypes.Normal{3, Float64}, mesh)
        color(lcmgl, 0,0,0)
        begin_mode(lcmgl, LCMGL.LINES)
        for i = 1:length(state.positions)
            vertex(lcmgl, state.positions[i]...)
            vertex(lcmgl, (state.positions[i] + Point(normals[i]))...)
        end
        end_mode(lcmgl)
        switch_buffer(lcmgl)
    end
    push!(energies, value(results))
end
# ProfileView.view()
plot(energies)

```

```julia
x
```

    78-element Array{Float64,1}:
      2.06042e-16
      1.43884e-16
      1.00293
      1.71986e-15
      2.59783e-15
      1.93762
     -0.385647
     -0.385647
      0.000309194
     -0.385647
      0.385647
      0.000309194
      0.385647
      ⋮
     -0.109675
     -0.109675
      3.09302
     -0.109675
      0.109675
      3.09302
      0.109675
     -0.109675
      3.09302
      0.109675
      0.109675
      3.09302

```julia

```


(cli_editor_tutorial)=
# Tutorial

We will here consider a problem we have already solved (in the Algebra
{ref}`algebra_tutorial`) but use a different interface to do so than Jupyter.
The code itself will be the same. The way we run it will differ.

```{admonition} Problem
Rationalise the denominator of $\frac{1}{\sqrt{2} + 1}$
```

Open a command line tool:

1. On **Windows** search for `Anaconda Prompt` (it should be available to you
   after installing Anaconda). See
   {ref}`fig:starting_the_notebook_server_windows`.
2. On **OS X** search for `terminal`. See
   {ref}`fig:starting_the_notebook_server`.

Whether or not your are on Windows or OS X changes the commands you need to
type. We will first list the directory we are currently in:

````{panels}
On Windows:

```bash
$ dir
```

---

On OS X:

```bash
$ ls
```

````

```{tip}
Throughout this book, when there are commands to be typed in a command line
tool I will prefix them with a `$`. Do not type the `$`.
```

This is similar to using your file explorer to view the contents in a given
directory. Similarly to the way we click on a directory in the file explorer we
can navigate to a directory in the command line.

To do this we use the same command on both operating systems:

````{panels}
On Windows:

```bash
$ cd <target_directory_name>
```

---

On OS X:

```bash
$ cd <target_directory_name>
```

````

We will do this to navigate to our `cfm` directoy. For example if, as in the
Algebra {ref}`algebra_tutorial` the `cfm` directory was on the `Desktop`
directory we would run the following:

````{panels}
On Windows:

```bash
$ cd Desktop
$ cd cfm
```

---

On OS X:

```bash
$ cd Desktop
$ cd cfm
```

````

```{attention}
The two statements are written under each other to denote that they are run
one after the other.
```

We will now create a new directory:

````{panels}
On Windows:

```bash
$ mkdir scripts
```

---

On OS X:

```bash
$ mkdir scripts
```

````

Inside this directory we will run the same command as before to see the
contents:

````{panels}
On Windows:

```bash
$ dir
```

---

On OS X:

```bash
$ ls
```

````

If you have followed the same steps described in {ref}`using_notebooks` then you
will see something similar to:

```{figure} ./img/output_of_dir/main.png
---
width: 75%
name: fig:output_of_dir
---
The contents of our directory on Windows
```

```{figure} ./img/output_of_ls/main.png
---
width: 75%
name: fig:output_of_ls
---
The contents of our directory on OS X
```

We will come back to this directory shortly but now we are going to install a
powerful code editor.

1. Navigate to <https://code.visualstudio.com>.
2. Download the installer making sure it is the correct one for your operating
   system (Windows, Mac OSX, Linux).
3. Run the installer.

This code editor will offer us a different way to write Python code.

Open VS code and create a new file.

In it write the following (which corresponds to the solution of our problem):

```python
"""
This script displays the solution to the problem considered.
"""
import sympy as sym

print("Question 1:")
expression = 1 / (sym.sqrt(2) + 1)
print(sym.simplify(expression))
```

This is shown:

```{figure} ./img/code_in_vscode/main.png
---
width: 75%
name: fig:code_in_vscode
---
The code above in VS code
```

We will now save this as `algebra.py` inside the `scripts` directory we created
earlier.

```{figure} ./img/saving_file_in_vscode/main.png
---
width: 75%
name: fig:saving_file_in_vscode
---
Saving `algebra.py` in VS code.
```

VScode now recognises the Python language and adds syntax colouring. It also
suggests a plugin specific for the Python language. There is more information
about plugins in the rest of this chapter.

```{figure} ./img/syntax_colouring_and_plugin_suggestion/main.png
---
width: 75%
name: fig:syntax_colouring_and_plugin_suggestion
---
Syntax colouring and plugin suggestion for the now recognised Python file.
```

All we have done so far is write the code. We now need to tell Python to run it.
To do this we will use the command line.

Navigate to the `scripts` directory that we created earlier:

````{panels}
On Windows:

```bash
$ cd scripts
```

---

On OS X:

```bash
$ cd scripts
```

````

Now confirm that the `algebra.py` file is in that directory:

````{panels}
On Windows:

```bash
$ dir
```

---

On OS X:

```bash
$ ls
```

````

Now we will run the python code in that script:

```bash
$ python algebra.py
```

When doing that you should see the following output:

```{figure} ./img/output_of_running_script/main.png
---
width: 75%
name: fig:output_of_running_script
---
Output of running the `algebra.py` script on OS X.
```