### Decision Trees and Random Forests

#### Packages

```python
import sys
import sklearn
import graphviz
import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.pyplot as plt

%matplotlib inline

from sklearn import tree
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split, GridSearchCV

print('The Python version is {}.\n'.format(sys.version))
print('The Numpy version is {}.\n'.format(np.__version__))
print('The Pandas version is {}.\n'.format(pd.__version__))
print('The Matplotlib version is {}.\n'.format(mpl.__version__))
print('The Scikit-Learn version is {}.\n'.format(sklearn.__version__))
```

    The Python version is 3.8.5 (default, Jul 28 2020, 12:59:40)
    [GCC 9.3.0].

    The Numpy version is 1.19.1.

    The Pandas version is 1.1.0.

    The Matplotlib version is 3.3.3.

    The Scikit-Learn version is 0.24.0.

### Decision Trees

Decision trees and the machine learning models that are based on them, in particular random forests and gradient boosted trees, are fundamentally different types of models than generalized linear models, such as logistic regression. GLMs are rooted in the theories of classical statistics, which have a long history. The mathematics behind linear regression were originally developed at the beginning of the 19th century, by Legendre and Gauss. Because of this, the normal distribution is also called the Gaussian.

Decision trees have an intuitive structure and mimic the way that logical decisions might be made by humans. Therefore, they are a highly interpretable type of mathematical model, which can be a particularly desirable property depending on the audience.

Decision trees are able to capture non-linear effects of the features, as opposed to a linear relationship.

In general, it should be clear that in a tree with $n$ levels, where every node before the final level is split,
there will be $2n$ leaf nodes. This is important to bear in mind as the maximum depth is one of the hyperparameters that you can set for a decision tree classifier in scikit-learn

#### Growing a tree for the study case

Let us load the data and select the desired columns:

```python
df = pd.read_csv('data/default_of_credit_card_clients_cleaned.csv')

features_response = df.columns.tolist()
to_remove = ['id', 'sex'] + ['pay_{}'.format(i) for i in range(2,7)] + features_response[-6:]

features_response = [feature for feature in features_response if feature not in to_remove]
```

Let us split the dataset into train and test sets:

```python
X_train, X_test, y_train, y_test = train_test_split(df[features_response].values,
                                                    df.default_payment_next_month.values,
                                                    test_size=0.2, random_state=24)
```

Now, let us instantiate the decision tree model:

```python
dt = tree.DecisionTreeClassifier(max_depth=2)

dt.fit(X_train, y_train)
```

    DecisionTreeClassifier(max_depth=2)

Using the graphviz library, let display a graphical representation of the tree:

```python
dot_data = tree.export_graphviz(dt, out_file=None, filled=True,
                                rounded=True, feature_names=features_response,
                                proportion=True, class_names=['Not Defaulted', 'Defaulted'])
```

```python
graph = graphviz.Source(dot_data)
graph
```

The training decision tree worked by starting with all the training samples in the initial node at the top of the tree, and then splited these into two groups based on a **threshold** in one of the features. The cut point was represented by a Boolean condition in the top **pay_1 <= 1.5** node. In the top node, there were all the samples ("samples = 100.0%"). Following the first split, 89.5% of the samples got sorted into the node on the left, while the remaining 10.5% went into the node on the right.

Let us confirm the proportions stated by the model:

```python
(features_response.index('pay_1'),
 X_train.shape)
```

    (4, (21331, 17))

```python
sum(X_train[:, features_response.index('pay_1')] <= 1.5)/X_train.shape[0]
```

    0.8946134733486475

Let us confirm the values line (the class proportion for the first node):

```python
np.mean(y_train)
```

    0.223102526838873

The class fractions were also colored: those with a higher proportion of the negative class than the positive class were orange, with darker orange signifying higher proportions, while those with a higher proportion of the positive class had a similar scheme using a blue color.

The line starting with "class" indicates how the decision tree made predictions from a given node, if that node were a leaf node.

Decision trees for classification make predictions by determining which leaf node a sample will be sorted in to, given the values of the features, and then predicting the class of the majority of the training samples in that leaf node.

#### Node Impurity

The goal of training decision trees is to make splits so that the next two nodes after the split have a higher purity, or, in other words, are closer to containing either only positive or only negative samples.

In practice, decision trees are actually trained using the inverse of purity, or node impurity. This is some measure of how far the node is from having 100% of the training samples belonging to one class and is analogous to the concept of a cost function, which signifies how far a given solution is from a theoretical perfect solution. For a given node $m$, the misclassification rate is simply the proportion of the less common class in that node, since all these samples will be misclassified when the majority class in that node is taken as the prediction.

The fraction of the positive class for a node $m$ and a class $k$ is given by:

$p_{mk+1} = 1 - p_{mk}$

Let us look at the missclassification distribution:

```python
p_m0 = np.linspace(0.01, 0.99, 101)
p_m1 = 1 - p_m0

misclassification_rate = np.minimum(p_m0, p_m1)

plt.plot(p_m0, misclassification_rate, label='Misclassification rate')
plt.xlabel('$p_{m0}$')
plt.legend()
```

Every time a node is split when growing a decision tree, two new nodes are created. Since the prediction from either of these new nodes is simply the majority class, an important goal will be to reduce the misclassification rate. Therefore, we will want to find a feature, from all the possible features, and a value of this feature at which to make a cut point, so that the misclassification rate in the two new nodes will be as low as possible when averaging over all the classes.

The algorithm then considers each possible threshold for every candidate feature and chooses the one that results in the lowest impurity, calculated as the average impurity across the two possible new nodes, weighted by the number of samples in each node. This process is repeated until a stopping criterion of the tree, such as max_depth.

While the misclassification rate is an intuitive measure of impurity, it happens that there are better measures that can be used to find splits during the model training process. The two options that are available in scikit-learn for the impurity calculation, which you can specify with the criterion keyword argument, are the Gini impurity and the cross-entropy options.

#### Gini

The Gini impurity is calculated for a node $m$ using the following formula:

$$
\begin{equation}
\begin{split}
Gini = \sum_{k} p_{mk} (1 - p_{mk})
\end{split}
\end{equation}
$$

Here, the summation is taken over all classes. In the case of a binary classification problem, there are only two classes, and we can write
this programmatically as follows

#### Cross Entropy

Cross entropy is calculated using this formula:

$$
\begin{equation}
\begin{split}
\text{cross entropy} = - \sum_{k} p_{mk} log(p_{mk})
\end{split}
\end{equation}
$$

Let us see these two distributions:

```python
def gini_impurity(**kwargs):
    """
    Returns the gini impurity for a given array (or arrays)

    Arguments:
        **kwargs (array): Numpy Arrays
    Returns:
        Numpy Array
    """
    return sum([i*(1-i) for i in kwargs.values()])

def cross_entropy_impurity(**kwargs):
    """
    Returns the cross entropy a given array (or arrays)

    Arguments:
        **kwargs (array): Numpy Arrays
    Returns:
        Numpy Array
    """
    return - sum([i*np.log(i) for i in kwargs.values()])
```

```python
gini = gini_impurity(x1=p_m0, x2=p_m1)
cross_entropy = cross_entropy_impurity(x1=p_m0, x2=p_m1)
```

```python
plt.plot(p_m0, misclassification_rate, label='Misclassification rate')
plt.plot(p_m0, gini, label='Gini impurity')
plt.plot(p_m0, cross_entropy, label='Cross entropy')
plt.xlabel('$p_{m0}$')
plt.legend()
```

Like the misclassification rate, both the Gini impurity and the cross entropy are highest when the class fractions are equal at 0.5, and they decrease as the node becomes purer – in other words, when they contain a higher proportion of just one of the classes. However, the Gini impurity is somewhat steeper than the misclassification rate in certain regions of the class fraction, which enables it to more effectively find the best split. Cross-entropy looks yet steeper.

It is necessary to consider both impurity metrics in a cross-validation search for hyperparameters in order to
determine the appropriate one. Note that in scikit-learn, Gini impurity can be specified with the criterion argument using the 'gini' string, while cross entropy is just referred to as 'entropy'.

#### Greedy Algorithm

There is no guarantee that a decision tree trained by the process described previously will be the best possible decision tree for finding leaf nodes with the lowest impurity. This is because the algorithm used to train decision trees is what is called a greedy algorithm.

The reason why the algorithm is used this way is because it takes substantially longer to consider all possible splits in a way that enables finding the truly optimal tree. Despite this shortcoming of the decision tree training process, there are methods that you can use to reduce the possible harmful effects of the greedy algorithm. Instead of searching for the best split at each node, the splitter keyword argument to the decision tree class can be specified as random in order to choose a random feature to make a split on.

### Decision Trees Pros

#### No need to scale data:

It is not an algorithms based on gradient descent, neither use L1 or L2 regularization to penalize coefficients. The algorithms that use the previous estimations need to scale the features to balance their weights during the training phase.

#### Non-linear relationships and interactions

Because each successive split in a decision tree is performed on a subset of the samples resulting from previous split(s), decision trees can describe complex non-linear relationships of a single feature, as well as interactions between features.

#### Predicted Probabilities

A decision tree makes predictions based on the majority of class of the leaf nodes. The probability is based on the proportion of the majority class in the leaf node. If the leaf node consisted 75% of the positive class, for
example, the prediction for that node will be the positive class and the predicted probability will be 0.75. The predicted probabilities from decision trees are not considered to be as statistically rigorous as those from generalized linear models, but they are still commonly used to measure the performance of models by methods that depend on varying the threshold for classification, such as the ROC curve or the precision-recall curve.

### Finding the best hyperparameters for a D3 model

Let us see the number of samples in the training study case set:

```python
X_train.shape
```

    (21331, 17)

With 21,331 training samples and 4-fold cross-validation, there will be three-fourth of the samples, or about 16,000 samples, in each training fold.

A theoretical limitation is that at least one sample in each leaf is needed. The depth of the tree relates
to the number of leaves, such that a split at every node n, implies 2n leaf nodes. Therefore, a tree with $L$ leaf nodes has a depth of approximately $log_{2}(L)$. In the limiting case, if this tree grow deep enough, so that every leaf node has one training sample for a given fold, the depth will be $log_{2}(16,000) \approx 14$. So, 14 is the theoretical limit to the depth of a tree that could grow in this case.

The depth of 14 would certainly imply overfitting, so this depth is not desired. Let us define different values for the max depth parameter:

```python
params = {'max_depth': [1, 2, 4, 6, 8, 10, 12]}
```

And now instantiate the GridSearch object:

```python
cv = GridSearchCV(dt, param_grid=params, scoring='roc_auc',
                  n_jobs=None, refit=True, cv=4, verbose=1,
                  pre_dispatch=None, error_score=np.nan, return_train_score=True)
```

The other options were the ROC AUC metric (scoring='roc_auc'), the 4-fold cross-validation (cv=4), and to calculate training scores (return_train_score=True) to assess the bias-variance trade-off.

Let us fit the data:

```python
cv.fit(X_train, y_train)
```

    Fitting 4 folds for each of 7 candidates, totalling 28 fits

    GridSearchCV(cv=4, estimator=DecisionTreeClassifier(max_depth=2),
                 param_grid={'max_depth': [1, 2, 4, 6, 8, 10, 12]},
                 pre_dispatch=None, return_train_score=True, scoring='roc_auc',
                 verbose=1)

And save the dictionary of results in a DataFrame object:

```python
cv_results_df = pd.DataFrame(cv.cv_results_)
cv_results_df
```

      mean_fit_time
      std_fit_time
      mean_score_time
      std_score_time
      param_max_depth
      params
      split0_test_score
      split1_test_score
      split2_test_score
      split3_test_score
      mean_test_score
      std_test_score
      rank_test_score
      split0_train_score
      split1_train_score
      split2_train_score
      split3_train_score
      mean_train_score
      std_train_score

      0
      0.032530
      0.007054
      0.004060
      0.000895
      1
      {'max_depth': 1}
      0.639514
      0.643398
      0.651891
      0.650753
      0.646389
      0.005136
      7
      0.648680
      0.647384
      0.644553
      0.644934
      0.646388
      0.001712

      1
      0.051850
      0.001237
      0.003802
      0.000087
      2
      {'max_depth': 2}
      0.695134
      0.699022
      0.713376
      0.699510
      0.701761
      0.006917
      5
      0.704034
      0.702700
      0.698113
      0.702535
      0.701845
      0.002232

      2
      0.101508
      0.009389
      0.003910
      0.000102
      4
      {'max_depth': 4}
      0.732720
      0.741078
      0.746946
      0.743731
      0.741119
      0.005276
      2
      0.756882
      0.752421
      0.749368
      0.753055
      0.752932
      0.002673

      3
      0.143802
      0.002191
      0.004772
      0.000942
      6
      {'max_depth': 6}
      0.743836
      0.746910
      0.751932
      0.740517
      0.745798
      0.004201
      1
      0.782202
      0.780044
      0.775228
      0.774776
      0.778062
      0.003158

      4
      0.188230
      0.005357
      0.004744
      0.000730
      8
      {'max_depth': 8}
      0.727023
      0.732562
      0.750618
      0.731040
      0.735311
      0.009066
      3
      0.812061
      0.808245
      0.803415
      0.802370
      0.806523
      0.003890

      5
      0.225156
      0.002811
      0.004701
      0.000228
      10
      {'max_depth': 10}
      0.706871
      0.708786
      0.717045
      0.712074
      0.711194
      0.003857
      4
      0.848864
      0.853907
      0.841430
      0.836463
      0.845166
      0.006704

      6
      0.259489
      0.005075
      0.004958
      0.000604
      12
      {'max_depth': 12}
      0.676996
      0.661479
      0.676430
      0.675982
      0.672722
      0.006501
      6
      0.887915
      0.903416
      0.885891
      0.875493
      0.888179
      0.009980

The DataFrame had one row for each combination of hyperparameters in the grid. As only one hyperparameter is being investigated, there was one row for each of the seven values that were searched for. A lot of output for each row can be seen, such as the mean and standard deviation of the time in seconds that each of the four folds took for both training (fitting) and testing (scoring).

```python
cv_results_df.columns
```

    Index(['mean_fit_time', 'std_fit_time', 'mean_score_time', 'std_score_time',
           'param_max_depth', 'params', 'split0_test_score', 'split1_test_score',
           'split2_test_score', 'split3_test_score', 'mean_test_score',
           'std_test_score', 'rank_test_score', 'split0_train_score',
           'split1_train_score', 'split2_train_score', 'split3_train_score',
           'mean_train_score', 'std_train_score'],
          dtype='object')

Generally speaking, the "best" combination of hyperparameters was the one with the highest average testing score. This was an estimation of how well the model, fit using these hyperparameters, could perform when scored on new data.

Let us display how the average testing score varies with the max_depth hyperparameter, including the standard deviations of the 4-fold training and testing scores as error bars:

```python
ax = plt.axes()
ax.errorbar(cv_results_df['param_max_depth'],
            cv_results_df['mean_train_score'],
            yerr=cv_results_df['std_train_score'],
            label='Mean $\pm$ 1 SD training scores')
ax.errorbar(cv_results_df['param_max_depth'],
            cv_results_df['mean_test_score'],
            yerr=cv_results_df['std_test_score'],
            label='Mean $\pm$ 1 SD testing scores')
ax.legend()
plt.xlabel('max_depth')
plt.ylabel('ROC AUC')
```

The standard deviations of the training and testing scores were shown as vertical lines at each value of max_depth that was tried; the distance above and below the average score was 1 standard deviation (they are also in the same units - it would be different when using the variance, which is the std squared).

The error bars indicated how variable the scores were across folds. There was not much variability between the folds, since the standard deviations were no high, so this was not an issue.

It is possible to see that as the tree's depth grown deeper and deeper, the model fitted the training data better and better. As noted previously, if the model grew trees deep enough so that each leaf node had just one training sample, a model that was very specific to the training data was spotted. In fact, it would fit the training data
perfectly. This model can be said to have an extremely high variance.

The apparent effect of increasing max_depth only increased testing scores up to a point, after which deeper trees had lower testing performance. This is another example of how the bias-variance trade-off can be used to create a better predictive model. In summary, shallower trees had more bias, since they were not fitting the training data as well. This is okay as some bias is acceptable so long a better performance on the testing data is verified.

The best depth for the study case was 6.

Comparing the results with the previous - logistic regression - model, the D3 showed a better performance with an ROC CURVE 0.745, the best performance for the logistic regression - with feature engineering - was 0.740.

### Random Forests

Random forests are examples of what are called ensemble models, because they are formed by combining other models. By combining the predictions of many models, it is possible to improve upon the deficiencies of any given one of them.

Random forests are just ensembles of many decision trees; all the models in this kind of ensemble have the same mathematical form. One hyperparameter, **n_estimators**, needs to be specified when building a random forest model. This parameter counts the number of D3 estimators in the forest. Generally speaking, the more trees, the better. As the number of trees increases, the variance of the overall ensemble will decrease. This should result in the random forest model having better generalization to new data, which will be reflected in increased testing scores.

The variance diminishes when using random forests mainly because of two principles:

#### The number of features considered in each split

By using all the features to fit the training data, overfitting is possible. By limiting the number of features considered at each split, some of the decision trees in a random forest will potentially find better splits. his is because, although they are still greedily searching for the best split, they are doing it with a limited selection of features.

#### The samples used to grow different trees

The other way that the trees in a random forest differ from each other is that they are usually grown with different training samples. To do this, a statistical procedure known as bootstrapping is used, which means to generate new synthetic datasets from the original data. The synthetic datasets are created by randomly selecting samples from the original dataset using replacement. Here, "replacement" means that if we select a certain sample, we will continue to consider it for selection, that is, it is "replaced" to the original dataset after we've sampled it.

The procedure of using random sampling to create synthetic datasets, and training models on them separately, is called bagging,which is short for bootstrapped aggregation.

### Selecting the Best Feautures

After model training, classification trees will take an input sample and a produce a predicted class, for example, whether or not a credit account in our case study problem will default. One intuitive approach to combining the predictions of these trees into the ultimate prediction of the forest is to take a majority vote. Sikit-learn uses a somewhat different approach: adding up the predicted probabilities for each class and then choosing the one with the highest probability. This captures more information from each tree than just the predicted class.

####  Predictions and Interpretability

Random forests are difficult to understand and interpret besides the ranking of the features (that is why it is used as a feature selection sometimes). Although the rank is make available, how the rank was done is difficult to know.

Let us train a Random Forest model:

```python
rf = RandomForestClassifier(n_estimators=10, criterion='gini', max_depth=3,
                            min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0,
                            max_features='auto', max_leaf_nodes=None, min_impurity_decrease=0.0,
                            min_impurity_split=None, bootstrap=True, oob_score=False, n_jobs=None,
                            random_state=4, verbose=0, warm_start=False, class_weight=None)
```

```python
rf_params = {'n_estimators': list(range(10, 110, 10))}
```

Let us instantiate the cv GridSearch object:

```python
cv_rf = GridSearchCV(rf, param_grid=rf_params, scoring='roc_auc',
                     n_jobs=None, refit=True, cv=4, verbose=1,
                     pre_dispatch=None, error_score=np.nan, return_train_score=True)
```

And train the model:

```python
cv_rf.fit(X_train, y_train)
```

    Fitting 4 folds for each of 10 candidates, totalling 40 fits

    GridSearchCV(cv=4,
                 estimator=RandomForestClassifier(max_depth=3, n_estimators=10,
                                                  random_state=4),
                 param_grid={'n_estimators': [10, 20, 30, 40, 50, 60, 70, 80, 90,
                                              100]},
                 pre_dispatch=None, return_train_score=True, scoring='roc_auc',
                 verbose=1)

```python
cv_rf_results_df = pd.DataFrame(cv_rf.cv_results_)
cv_rf_results_df
```

      mean_fit_time
      std_fit_time
      mean_score_time
      std_score_time
      param_n_estimators
      params
      split0_test_score
      split1_test_score
      split2_test_score
      split3_test_score
      mean_test_score
      std_test_score
      rank_test_score
      split0_train_score
      split1_train_score
      split2_train_score
      split3_train_score
      mean_train_score
      std_train_score

      0
      0.165941
      0.006219
      0.009214
      0.000139
      10
      {'n_estimators': 10}
      0.739183
      0.746599
      0.772127
      0.771322
      0.757308
      0.014656
      10
      0.767973
      0.767158
      0.760550
      0.763683
      0.764841
      0.002955

      1
      0.310278
      0.004693
      0.014494
      0.000648
      20
      {'n_estimators': 20}
      0.745612
      0.753712
      0.777616
      0.771737
      0.762169
      0.012998
      9
      0.772819
      0.771499
      0.765944
      0.765395
      0.768914
      0.003284

      2
      0.449935
      0.012662
      0.020149
      0.002076
      30
      {'n_estimators': 30}
      0.747259
      0.754154
      0.777954
      0.771825
      0.762798
      0.012524
      7
      0.774655
      0.772100
      0.765026
      0.766697
      0.769620
      0.003910

      3
      0.595663
      0.013271
      0.025158
      0.002189
      40
      {'n_estimators': 40}
      0.746682
      0.754553
      0.777709
      0.773134
      0.763020
      0.012813
      4
      0.774300
      0.772642
      0.765051
      0.765876
      0.769467
      0.004057

      4
      0.720641
      0.005543
      0.028281
      0.000115
      50
      {'n_estimators': 50}
      0.748009
      0.755878
      0.779465
      0.771250
      0.763651
      0.012379
      1
      0.775241
      0.774025
      0.765732
      0.766947
      0.770486
      0.004191

      5
      0.852607
      0.001184
      0.033063
      0.000228
      60
      {'n_estimators': 60}
      0.748437
      0.755604
      0.779478
      0.770978
      0.763624
      0.012252
      2
      0.775419
      0.773615
      0.766133
      0.767032
      0.770550
      0.004031

      6
      0.993426
      0.003056
      0.038019
      0.000113
      70
      {'n_estimators': 70}
      0.747491
      0.754465
      0.779104
      0.771023
      0.763021
      0.012620
      3
      0.774420
      0.773379
      0.765960
      0.766174
      0.769983
      0.003934

      7
      1.143173
      0.005024
      0.043118
      0.001129
      80
      {'n_estimators': 80}
      0.747424
      0.754488
      0.778753
      0.770872
      0.762884
      0.012502
      5
      0.774234
      0.773257
      0.765978
      0.766220
      0.769922
      0.003840

      8
      1.279748
      0.004862
      0.049479
      0.003227
      90
      {'n_estimators': 90}
      0.747097
      0.753794
      0.778507
      0.770426
      0.762456
      0.012571
      8
      0.774095
      0.772973
      0.766022
      0.765748
      0.769709
      0.003846

      9
      1.416619
      0.002662
      0.052569
      0.000116
      100
      {'n_estimators': 100}
      0.747221
      0.753924
      0.779195
      0.770937
      0.762819
      0.012811
      6
      0.774340
      0.773598
      0.766103
      0.766914
      0.770238
      0.003750

Let us display the performance for each gridsearch params:

```python
fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(10, 4))

axs[0].plot(cv_rf_results_df['param_n_estimators'],
            cv_rf_results_df['mean_fit_time'],
            '-o')
axs[0].set_xlabel('Number of trees')
axs[0].set_ylabel('Mean fit time (seconds)')

axs[1].errorbar(cv_rf_results_df['param_n_estimators'],
                cv_rf_results_df['mean_test_score'],
                yerr=cv_rf_results_df['std_test_score'])
axs[1].set_xlabel('Number of trees')
axs[1].set_ylabel('Mean testing ROC AUC $\pm$ 1 SD ')
plt.tight_layout()
```

It is possible to see that by using a random forest, the model performance increased on the cross-validation testing folds compared to the previous models. While no attempt to tune the random forest hyperparameters to achieve the best model performance was done, the results indicated that a random forest was higher in performance compared to previous efforts.

However, along with these higher model testing scores, notice that there was also more variability between the folds saw with the decision tree; this variability was visible as larger standard deviations in model testing scores across the folds.

The left plot shows that there was a linear increase in training time as more trees were added to the forest. This was expected as more trees were evaluated. The additional computation time did not compensate in terms of performance, as beyond 20 trees it was not clear that adding more trees reliably improves testing performance. While the model with 50 trees had the highest score, the fact that adding more trees actually decreased the testing score somewhat indicated that the gain in ROC AUC for 50 trees might just was due to randomness.

Let us see the best hyperparameter as well as the best params estimated by the model:

```python
cv_rf.best_params_
```

    {'n_estimators': 50}

```python
feat_imp_df = pd.DataFrame({
    'Feature name':features_response,
    'Importance':cv_rf.best_estimator_.feature_importances_
})

feat_imp_df.sort_values('Importance', ascending=False)
```

      Feature name
      Importance

      4
      pay_1
      0.609609

      11
      pay_amt1
      0.094123

      0
      limit_bal
      0.079265

      13
      pay_amt3
      0.047067

      12
      pay_amt2
      0.035393

      6
      bill_amt2
      0.022422

      14
      pay_amt4
      0.021164

      15
      pay_amt5
      0.015962

      10
      bill_amt6
      0.014293

      8
      bill_amt4
      0.013454

      7
      bill_amt3
      0.013321

      16
      pay_amt6
      0.011831

      9
      bill_amt5
      0.011264

      5
      bill_amt1
      0.008574

      1
      education
      0.001630

      3
      age
      0.000459

      2
      marriage
      0.000170

The top 5 most important features from the random forest were the same as the top 5 chosen by an ANOVA F-test analysis previously done, though in some different order. This was a good confirmation about the importance of these features as they were evaluated by different methods.

### Cross-Validation GridSearch with Random Forest on two hyperparameters

Let us extend the previous exercise and conduct a grid search over the maximum depth of a tree. First, let us create a dictionary representing the grid for **n_estimators** and **max_depth**:

```python
rf_params = {'n_estimators': [10, 50, 100, 200],
             'max_depth': [3, 6, 9, 12]}
```

And instantiate the GridSearch object using the same options that were applied before and train the predictor (to use parallel computing set **n_jobs=-1** that will match the number of cpus available automatically):

```python
cv_rf = GridSearchCV(rf, param_grid=rf_params, scoring='roc_auc',
                     n_jobs=None, refit=True, cv=4, verbose=2,
                     pre_dispatch=None, error_score=np.nan, return_train_score=True)

cv_rf.fit(X_train, y_train)
```

    Fitting 4 folds for each of 16 candidates, totalling 64 fits
    [CV] END .......................max_depth=3, n_estimators=10; total time=   0.2s
    [CV] END .......................max_depth=3, n_estimators=10; total time=   0.2s
    [CV] END .......................max_depth=3, n_estimators=10; total time=   0.2s
    [CV] END .......................max_depth=3, n_estimators=10; total time=   0.2s
    [CV] END .......................max_depth=3, n_estimators=50; total time=   0.8s
    [CV] END .......................max_depth=3, n_estimators=50; total time=   0.8s
    [CV] END .......................max_depth=3, n_estimators=50; total time=   0.8s
    [CV] END .......................max_depth=3, n_estimators=50; total time=   0.8s
    [CV] END ......................max_depth=3, n_estimators=100; total time=   1.5s
    [CV] END ......................max_depth=3, n_estimators=100; total time=   1.5s
    [CV] END ......................max_depth=3, n_estimators=100; total time=   1.5s
    [CV] END ......................max_depth=3, n_estimators=100; total time=   1.5s
    [CV] END ......................max_depth=3, n_estimators=200; total time=   3.0s
    [CV] END ......................max_depth=3, n_estimators=200; total time=   3.0s
    [CV] END ......................max_depth=3, n_estimators=200; total time=   2.9s
    [CV] END ......................max_depth=3, n_estimators=200; total time=   2.9s
    [CV] END .......................max_depth=6, n_estimators=10; total time=   0.3s
    [CV] END .......................max_depth=6, n_estimators=10; total time=   0.3s
    [CV] END .......................max_depth=6, n_estimators=10; total time=   0.3s
    [CV] END .......................max_depth=6, n_estimators=10; total time=   0.3s
    [CV] END .......................max_depth=6, n_estimators=50; total time=   1.3s
    [CV] END .......................max_depth=6, n_estimators=50; total time=   1.3s
    [CV] END .......................max_depth=6, n_estimators=50; total time=   1.3s
    [CV] END .......................max_depth=6, n_estimators=50; total time=   1.2s
    [CV] END ......................max_depth=6, n_estimators=100; total time=   2.5s
    [CV] END ......................max_depth=6, n_estimators=100; total time=   2.5s
    [CV] END ......................max_depth=6, n_estimators=100; total time=   2.5s
    [CV] END ......................max_depth=6, n_estimators=100; total time=   2.5s
    [CV] END ......................max_depth=6, n_estimators=200; total time=   5.0s
    [CV] END ......................max_depth=6, n_estimators=200; total time=   5.0s
    [CV] END ......................max_depth=6, n_estimators=200; total time=   5.0s
    [CV] END ......................max_depth=6, n_estimators=200; total time=   5.0s
    [CV] END .......................max_depth=9, n_estimators=10; total time=   0.4s
    [CV] END .......................max_depth=9, n_estimators=10; total time=   0.4s
    [CV] END .......................max_depth=9, n_estimators=10; total time=   0.4s
    [CV] END .......................max_depth=9, n_estimators=10; total time=   0.4s
    [CV] END .......................max_depth=9, n_estimators=50; total time=   1.7s
    [CV] END .......................max_depth=9, n_estimators=50; total time=   1.7s
    [CV] END .......................max_depth=9, n_estimators=50; total time=   1.7s
    [CV] END .......................max_depth=9, n_estimators=50; total time=   1.7s
    [CV] END ......................max_depth=9, n_estimators=100; total time=   3.4s
    [CV] END ......................max_depth=9, n_estimators=100; total time=   3.4s
    [CV] END ......................max_depth=9, n_estimators=100; total time=   3.4s
    [CV] END ......................max_depth=9, n_estimators=100; total time=   3.4s
    [CV] END ......................max_depth=9, n_estimators=200; total time=   6.9s
    [CV] END ......................max_depth=9, n_estimators=200; total time=   6.9s
    [CV] END ......................max_depth=9, n_estimators=200; total time=   6.9s
    [CV] END ......................max_depth=9, n_estimators=200; total time=   6.9s
    [CV] END ......................max_depth=12, n_estimators=10; total time=   0.4s
    [CV] END ......................max_depth=12, n_estimators=10; total time=   0.4s
    [CV] END ......................max_depth=12, n_estimators=10; total time=   0.4s
    [CV] END ......................max_depth=12, n_estimators=10; total time=   0.4s
    [CV] END ......................max_depth=12, n_estimators=50; total time=   2.1s
    [CV] END ......................max_depth=12, n_estimators=50; total time=   2.1s
    [CV] END ......................max_depth=12, n_estimators=50; total time=   2.1s
    [CV] END ......................max_depth=12, n_estimators=50; total time=   2.2s
    [CV] END .....................max_depth=12, n_estimators=100; total time=   4.3s
    [CV] END .....................max_depth=12, n_estimators=100; total time=   4.3s
    [CV] END .....................max_depth=12, n_estimators=100; total time=   4.2s
    [CV] END .....................max_depth=12, n_estimators=100; total time=   4.3s
    [CV] END .....................max_depth=12, n_estimators=200; total time=   8.7s
    [CV] END .....................max_depth=12, n_estimators=200; total time=   8.5s
    [CV] END .....................max_depth=12, n_estimators=200; total time=   8.5s
    [CV] END .....................max_depth=12, n_estimators=200; total time=   8.6s

    GridSearchCV(cv=4,
                 estimator=RandomForestClassifier(max_depth=3, n_estimators=10,
                                                  random_state=4),
                 param_grid={'max_depth': [3, 6, 9, 12],
                             'n_estimators': [10, 50, 100, 200]},
                 pre_dispatch=None, return_train_score=True, scoring='roc_auc',
                 verbose=2)

Now, let store the results into a pandas DataFrame object:

```python
cv_rf_results_df = pd.DataFrame(cv_rf.cv_results_)
cv_rf_results_df
```

      mean_fit_time
      std_fit_time
      mean_score_time
      std_score_time
      param_max_depth
      param_n_estimators
      params
      split0_test_score
      split1_test_score
      split2_test_score
      split3_test_score
      mean_test_score
      std_test_score
      rank_test_score
      split0_train_score
      split1_train_score
      split2_train_score
      split3_train_score
      mean_train_score
      std_train_score

      0
      0.160572
      0.010295
      0.009034
      0.000137
      3
      10
      {'max_depth': 3, 'n_estimators': 10}
      0.739183
      0.746599
      0.772127
      0.771322
      0.757308
      0.014656
      15
      0.767973
      0.767158
      0.760550
      0.763683
      0.764841
      0.002955

      1
      0.739119
      0.014972
      0.028264
      0.000112
      3
      50
      {'max_depth': 3, 'n_estimators': 50}
      0.748009
      0.755878
      0.779465
      0.771250
      0.763651
      0.012379
      12
      0.775241
      0.774025
      0.765732
      0.766947
      0.770486
      0.004191

      2
      1.433841
      0.013277
      0.052308
      0.000071
      3
      100
      {'max_depth': 3, 'n_estimators': 100}
      0.747221
      0.753924
      0.779195
      0.770937
      0.762819
      0.012811
      13
      0.774340
      0.773598
      0.766103
      0.766914
      0.770238
      0.003750

      3
      2.848571
      0.006656
      0.100138
      0.000516
      3
      200
      {'max_depth': 3, 'n_estimators': 200}
      0.747454
      0.753057
      0.777644
      0.771729
      0.762471
      0.012550
      14
      0.774453
      0.773534
      0.766139
      0.767404
      0.770382
      0.003653

      4
      0.252126
      0.002849
      0.010875
      0.000209
      6
      10
      {'max_depth': 6, 'n_estimators': 10}
      0.752905
      0.758904
      0.780036
      0.774017
      0.766466
      0.010980
      11
      0.795911
      0.796476
      0.793695
      0.796446
      0.795632
      0.001141

      5
      1.219600
      0.008659
      0.036014
      0.000334
      6
      50
      {'max_depth': 6, 'n_estimators': 50}
      0.760089
      0.765251
      0.788567
      0.778669
      0.773144
      0.011193
      8
      0.807081
      0.805109
      0.799176
      0.801229
      0.803149
      0.003113

      6
      2.418893
      0.007093
      0.067700
      0.000225
      6
      100
      {'max_depth': 6, 'n_estimators': 100}
      0.760964
      0.765515
      0.788793
      0.778936
      0.773552
      0.011004
      7
      0.808194
      0.806130
      0.800846
      0.803123
      0.804573
      0.002808

      7
      4.845598
      0.009769
      0.131047
      0.000598
      6
      200
      {'max_depth': 6, 'n_estimators': 200}
      0.761481
      0.765319
      0.787854
      0.779745
      0.773600
      0.010682
      6
      0.808276
      0.806853
      0.800682
      0.804006
      0.804954
      0.002906

      8
      0.345823
      0.001968
      0.012519
      0.000102
      9
      10
      {'max_depth': 9, 'n_estimators': 10}
      0.757853
      0.755444
      0.781544
      0.773823
      0.767166
      0.010899
      10
      0.854161
      0.856197
      0.850317
      0.852039
      0.853178
      0.002211

      9
      1.677115
      0.008941
      0.045137
      0.000182
      9
      50
      {'max_depth': 9, 'n_estimators': 50}
      0.763956
      0.764420
      0.786690
      0.780255
      0.773830
      0.009908
      4
      0.867242
      0.871380
      0.868213
      0.867390
      0.868557
      0.001672

      10
      3.346243
      0.009928
      0.086113
      0.000312
      9
      100
      {'max_depth': 9, 'n_estimators': 100}
      0.763872
      0.765788
      0.789016
      0.781199
      0.774969
      0.010530
      3
      0.872099
      0.874119
      0.870537
      0.871196
      0.871988
      0.001350

      11
      6.705547
      0.007114
      0.167002
      0.000740
      9
      200
      {'max_depth': 9, 'n_estimators': 200}
      0.764963
      0.767541
      0.790255
      0.782034
      0.776199
      0.010402
      1
      0.873807
      0.874909
      0.871185
      0.871246
      0.872787
      0.001619

      12
      0.426109
      0.005192
      0.014213
      0.000121
      12
      10
      {'max_depth': 12, 'n_estimators': 10}
      0.754513
      0.751116
      0.764061
      0.759223
      0.757228
      0.004884
      16
      0.922544
      0.919157
      0.926191
      0.914636
      0.920632
      0.004263

      13
      2.093706
      0.010107
      0.053497
      0.000099
      12
      50
      {'max_depth': 12, 'n_estimators': 50}
      0.763750
      0.763663
      0.780781
      0.778086
      0.771570
      0.007921
      9
      0.940170
      0.943000
      0.942297
      0.937049
      0.940629
      0.002315

      14
      4.155320
      0.017221
      0.107935
      0.004810
      12
      100
      {'max_depth': 12, 'n_estimators': 100}
      0.765665
      0.765061
      0.783413
      0.781119
      0.773815
      0.008493
      5
      0.942902
      0.946968
      0.946109
      0.942448
      0.944607
      0.001962

      15
      8.371203
      0.092187
      0.208209
      0.005270
      12
      200
      {'max_depth': 12, 'n_estimators': 200}
      0.765910
      0.766963
      0.785611
      0.783513
      0.775499
      0.009101
      2
      0.945018
      0.949378
      0.946415
      0.944689
      0.946375
      0.001851

Now, let us plot a three dimensional space into a two dimensional plot using a Checkerboard Graph. A Checkerboard Graph plots a two-dimensional grid with colored squares (or other shapes on it) as a way to show three dimensions of data. First, it is necessary to create a meshgrid of the *x*-axis and *y*-axis:

```python
xx_rf, yy_rf = np.meshgrid(range(5), range(5))
```

Let us set the colour map scale and create the axes:

```python
cm = plt.cm.jet

plt.figure(figsize=(20,10))
ax = plt.axes()

pcolor_graph = ax.pcolormesh(xx_rf,
                             yy_rf,
                             cv_rf_results_df['mean_test_score'].values.reshape((4,4)),
                             cmap=cm)

plt.colorbar(pcolor_graph, label='Average testing ROC AUC')

ax.set_aspect('equal')
ax.set_xticks([0.5, 1.5, 2.5, 3.5])
ax.set_yticks([0.5, 1.5, 2.5, 3.5])
ax.set_xticklabels([str(tick_label) for tick_label in rf_params['n_estimators']])
ax.set_yticklabels([str(tick_label) for tick_label in rf_params['max_depth']])
ax.set_xlabel('Number of trees')
ax.set_ylabel('Maximum depth')
```

There was certainly an advantage to using trees with a depth of more than three. Of the parameter combinations that were tried, **max_depth=9** with **200 trees** yielded the best average testing score, with ROC AUC = 0.776.

This was the best model found so far (compared to the individual D3 and logistic regression).


```python
from sympy import init_session
init_session()
```

```python
init_printing?
```

```python
# this is for rac[3,1]
a, b, d, k, l0 = symbols('a b d k, l0')
```

```python
f=l0*(k**2 +2*a**2*k + a**4+b**2)*(1+d**2*k)/(a**4 + b**2 + k*(2*a**2 + d**2*(a**4+b**2)))
f
```

$\displaystyle \frac{l_{0} \left(d^{2} k + 1\right) \left(a^{4} + 2 a^{2} k + b^{2} + k^{2}\right)}{a^{4} + b^{2} + k \left(2 a^{2} + d^{2} \left(a^{4} + b^{2}\right)\right)}$

```python
f.factor()
```

$\displaystyle \frac{l_{0} \left(d^{2} k + 1\right) \left(a^{4} + 2 a^{2} k + b^{2} + k^{2}\right)}{a^{4} d^{2} k + a^{4} + 2 a^{2} k + b^{2} d^{2} k + b^{2}}$

```python
diff(f,l0)
```

$\displaystyle \frac{\left(d^{2} k + 1\right) \left(a^{4} + 2 a^{2} k + b^{2} + k^{2}\right)}{a^{4} + b^{2} + k \left(2 a^{2} + d^{2} \left(a^{4} + b^{2}\right)\right)}$

```python
da=diff(f,a)
da.factor()
```

$\displaystyle - \frac{4 a k^{2} l_{0} \left(d^{2} k + 1\right) \left(a^{4} d^{2} + a^{2} d^{2} k + a^{2} - b^{2} d^{2} + k\right)}{\left(a^{4} d^{2} k + a^{4} + 2 a^{2} k + b^{2} d^{2} k + b^{2}\right)^{2}}$

```python
db=diff(f,b)
db.factor()
```

$\displaystyle - \frac{2 b k^{2} l_{0} \left(d^{2} k + 1\right) \left(2 a^{2} d^{2} + d^{2} k + 1\right)}{\left(a^{4} d^{2} k + a^{4} + 2 a^{2} k + b^{2} d^{2} k + b^{2}\right)^{2}}$

```python
dd=diff(f,d)
dd.factor()
```

$\displaystyle \frac{4 a^{2} d k^{2} l_{0} \left(a^{4} + 2 a^{2} k + b^{2} + k^{2}\right)}{\left(a^{4} d^{2} k + a^{4} + 2 a^{2} k + b^{2} d^{2} k + b^{2}\right)^{2}}$

```python
printing.sstrrepr(f.factor())
```

    'l0*(d**2*k + 1)*(a**4 + 2*a**2*k + b**2 + k**2)/(a**4*d**2*k + a**4 + 2*a**2*k + b**2*d**2*k + b**2)'

```python
printing.sstrrepr(diff(f,l0).factor())
```

    '(d**2*k + 1)*(a**4 + 2*a**2*k + b**2 + k**2)/(a**4*d**2*k + a**4 + 2*a**2*k + b**2*d**2*k + b**2)'

```python
printing.sstrrepr(da.factor())
```

    '-4*a*k**2*l0*(d**2*k + 1)*(a**4*d**2 + a**2*d**2*k + a**2 - b**2*d**2 + k)/(a**4*d**2*k + a**4 + 2*a**2*k + b**2*d**2*k + b**2)**2'

```python
printing.sstrrepr(db.factor())
```

    '-2*b*k**2*l0*(d**2*k + 1)*(2*a**2*d**2 + d**2*k + 1)/(a**4*d**2*k + a**4 + 2*a**2*k + b**2*d**2*k + b**2)**2'

```python
printing.sstrrepr(dd.factor())
```

    '4*a**2*d*k**2*l0*(a**4 + 2*a**2*k + b**2 + k**2)/(a**4*d**2*k + a**4 + 2*a**2*k + b**2*d**2*k + b**2)**2'

```python

```


###### Content under Creative Commons Attribution license CC-BY 4.0, code under BSD 3-Clause License © 2017 L.A. Barba, N.C. Clementi

# Linear regression with real data

## Earth temperature over time

In this lesson, we will apply all that we've learned (and more) to analyze real data of Earth temperature over time.

Is global temperature rising? How much? This is a question of burning importance in today's world!

Data about global temperatures are available from several sources: NASA, the National Climatic Data Center (NCDC) and the University of East Anglia in the UK. Check out the [University Corporation for Atmospheric Research](https://www2.ucar.edu/climate/faq/how-much-has-global-temperature-risen-last-100-years) (UCAR) for an in-depth discussion.

The [NASA Goddard Space Flight Center](http://svs.gsfc.nasa.gov/goto?3901) is one of our sources of global climate data. They produced the video below showing a color map of the changing global surface **temperature anomalies** from 1880 to 2015.

The term [global temperature anomaly](https://www.ncdc.noaa.gov/monitoring-references/faq/anomalies.php) means the difference in temperature with respect to a reference value or a long-term average. It is a very useful way of looking at the problem and in many ways better than absolute temperature. For example, a winter month may be colder than average in Washington DC, and also in Miami, but the absolute temperatures will be different in both places.

```python
from IPython.display import YouTubeVideo
YouTubeVideo('gGOzHVUQCw0')
```

How would we go about understanding the _trends_ from the data on global temperature?

The first step in analyzing unknown data is to generate some simple plots using **Matplotlib**. We are going to look at the temperature-anomaly history, contained in a file, and make our first plot to explore this data.

We are going to smooth the data and then we'll fit a line to it to find a trend, plotting along the way to see how it all looks.

Let's get started!

## Step 1: Read a data file

We took the data from the [NOAA](https://www.ncdc.noaa.gov/cag/) (National Oceanic and Atmospheric Administration) webpage. Feel free to play around with the webpage and analyze data on your own, but for now, let's make sure we're working with the same dataset.

We have a file named `land_global_temperature_anomaly-1880-2016.csv` in our `data` folder. This file contains the year on the first column, and averages of land temperature anomaly listed sequentially on the second column, from the year 1880 to 2016. We will load the file, then make an initial plot to see what it looks like.

##### Note:

If you downloaded this notebook alone, rather than the full collection for this course, you may not have the data file on the location we assume below. In that case, you can download the data if you add a code cell, and execute the following code in it:

```Python
from urllib.request import urlretrieve
URL = 'http://go.gwu.edu/engcomp1data5?accessType=DOWNLOAD'
urlretrieve(URL, 'land_global_temperature_anomaly-1880-2016.csv')
```
The data file will be downloaded to your working directory, and you will then need to remove the path information, i.e., the string `'../../data/'`, from the definition of the variable `fname` below.

Let's start by importing NumPy.

```python
import numpy
```

To load our data from the file, we'll use the function [`numpy.loadtxt()`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html), which lets us immediately save the data into NumPy arrays. (We encourage you to read the documentation for details on how the function works.) Here, we'll save the data into the arrays `year` and `temp_anomaly`.

```python
fname = '../../data/land_global_temperature_anomaly-1880-2016.csv'

year, temp_anomaly = numpy.loadtxt(fname, delimiter=',', skiprows=5, unpack=True)
```

##### Exercise

Inspect the data by printing `year` and `temp_anomaly`.

## Step 2: Plot the data

Let's first load the **Matplotlib** module called `pyplot`, for making 2D plots. Remember that to get the plots inside the notebook, we use a special "magic" command, `%matplotlib inline`:

```python
from matplotlib import pyplot
%matplotlib inline
```

The `plot()` function of the `pyplot` module makes simple line plots. We avoid that stuff that appeared on top of the figure, that `Out[x]: [< ...>]` ugliness, by adding a semicolon at the end of the plotting command.

```python
pyplot.plot(year, temp_anomaly);
```

Now we have a line plot, but if you see this plot without any information you would not be able to figure out what kind of data it is! We need labels on the axes, a title and why not a better color, font and size of the ticks.
**Publication quality** plots should always be your standard for plotting.
How you present your data will allow others (and probably you in the future) to better understand your work.

We can customize the style of our plots using **Matplotlib**'s [`rcParams`](https://matplotlib.org/api/matplotlib_configuration_api.html#matplotlib.rcParams). It lets us set some style options that apply for all the plots we create in the current session.
Here, we'll make the font of a specific size and type. You can also customize other parameters like line width, color, and so on (check out the documentation).

```python
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
```

We'll redo the same plot, but now we'll add a few things to make it prettier and **publication quality**. We'll add a title, label the axes and, show a background grid. Study the commands below and looks at the result!

```python
#You can set the size of the figure by doing:
pyplot.figure(figsize=(10,5))

#Plotting
pyplot.plot(year, temp_anomaly, color='#2929a3', linestyle='-', linewidth=1)
pyplot.title('Land global temperature anomalies. \n')
pyplot.xlabel('Year')
pyplot.ylabel('Land temperature anomaly [°C]')
pyplot.grid();
```

Better, no? Feel free to play around with the parameters and see how the plot changes. There's nothing like trial and error to get the hang of it.

## Step 3: Least-squares linear regression

In order to have an idea of the general behavior of our data, we can find a smooth curve that (approximately) fits the points. We generally look for a curve that's simple (e.g., a polynomial), and does not reproduce the noise that's always present in experimental data.

Let $f(x)$ be the function that we'll fit to the $n+1$ data points: $(x_i, y_i)$, $i = 0, 1, ... ,n$:

$$
    f(x) = f(x; a_0, a_1, ... , a_m)
$$

The notation above means that $f$ is a function of $x$, with $m+1$ variable parameters $a_0, a_1, ... , a_m$, where $m < n$. We need to choose the form of $f(x)$ a priori, by inspecting the experimental data and knowing something about the phenomenon we've measured. Thus, curve fitting consists of two steps:

1. Choosing the form of $f(x)$.
2. Computing the parameters that will give us the "best fit" to the data.

### What is the "best" fit?

When the noise in the data is limited to the $y$-coordinate, it's common to use a **least-squares fit**, which minimizes the function

$$
\begin{equation}
    S(a_0, a_1, ... , a_m) = \sum_{i=0}^{n} [y_i - f(x_i)]^2
\end{equation}
$$

with respect to each $a_j$. We find the values of the parameters for the best fit by solving the following equations:

$$
\begin{equation}
    \frac{\partial{S}}{\partial{a_k}} = 0, \quad k = 0, 1, ... , m.
\end{equation}
$$

Here, the terms $r_i = y_i - f(x_i)$ are called residuals: they tell us the discrepancy between the data and the fitting function at $x_i$.

Take a look at the function $S$: what we want to minimize is the sum of the squares of the residuals. The equations (2) are generally nonlinear in $a_j$ and might be difficult to solve. Therefore, the fitting function is commonly chosen as a linear combination of specified functions $f_j(x)$,

$$
\begin{equation*}
    f(x) = a_0f_0(x) + a_1f_1(x) + ... + a_mf_m(x)
\end{equation*}
$$

which results in equations (2) being linear. In the case that the fitting function is polynomial, we have have $f_0(x) = 1, \; f_1(x) = x, \; f_2(x) = x^2$, and so on.

### Linear regression

When we talk about linear regression we mean "fitting a straight line to the data." Thus,

$$
\begin{equation}
    f(x) = a_0 + a_1x
\end{equation}
$$

In this case, the function that we'll minimize is:

$$
\begin{equation}
    S(a_0, a_1) = \sum_{i=0}^{n} [y_i - f(x_i)]^2 = \sum_{i=0}^{n} (y_i - a_0 - a_1x_i)^2
\end{equation}
$$

Equations (2) become:

$$
\begin{equation}
    \frac{\partial{S}}{\partial{a_0}} =  \sum_{i=0}^{n} -2(y_i - a_0 - a_1x_i) = 2 \left[ a_0(n+1) + a_1\sum_{i=0}^{n} x_i - \sum_{i=0}^{n} y_i \right] = 0
\end{equation}
$$

$$
\begin{equation}
    \frac{\partial{S}}{\partial{a_1}} =  \sum_{i=0}^{n} -2(y_i - a_0 - a_1x_i)x_i = 2 \left[ a_0\sum_{i=0}^{n} x_i + a_1\sum_{i=0}^{n} x_{i}^2 - \sum_{i=0}^{n} x_iy_i \right] = 0
\end{equation}
$$

Let's divide both equations by $2(n+1)$ and rearrange terms.

Rearranging (6) and (7):

$$
\begin{align}
    2 \left[ a_0(n+1) + a_1\sum_{i=0}^{n} x_i - \sum_{i=0}^{n} y_i \right] &= 0 \nonumber \\
    \frac{a_0(n+1)}{n+1} + a_1 \frac{\sum_{i=0}^{n} x_i}{n+1} - \frac{\sum_{i=0}^{n} y_i}{n+1} &= 0 \\
\end{align}
$$

$$
\begin{align}
    a_0  = \bar{y} - a_1\bar{x}
\end{align}
$$

where $\bar{x} = \frac{\sum_{i=0}^{n} x_i}{n+1}$ and $\bar{y} = \frac{\sum_{i=0}^{n} y_i}{n+1}$.

Rearranging (7):

$$
\begin{align}
    2 \left[ a_0\sum_{i=0}^{n} x_i + a_1\sum_{i=0}^{n} x_{i}^2 - \sum_{i=0}^{n} x_iy_i \right] &= 0 \\
    a_0\sum_{i=0}^{n} x_i + a_1\sum_{i=0}^{n} x_{i}^2 - \sum_{i=0}^{n} x_iy_i &=0 \\
\end{align}
$$

Now, if we replace $a_0$ from equation (8) into (9) and rearrange terms:

$$
\begin{align*}
    (\bar{y} - a_1\bar{x})\sum_{i=0}^{n} x_i  + a_1\sum_{i=0}^{n} x_{i}^2 - \sum_{i=0}^{n} x_iy_i  &= 0 \\
\end{align*}
$$

Replacing the definitions of the mean values into the equation,

$$
\begin{align*}
    \left[\frac{1}{n+1}\sum_{i=0}^{n} y_i  - \frac{a_1}{n+1}\sum_{i=0}^{n} x_i \right]\sum_{i=0}^{n} x_i  + a_1\sum_{i=0}^{n} x_{i}^2 - \sum_{i=0}^{n} x_iy_i  &= 0  \\
     \frac{1}{n+1}\sum_{i=0}^{n} y_i \sum_{i=0}^{n} x_i - \frac{a_1}{n+1}\sum_{i=0}^{n} x_i \sum_{i=0}^{n} x_i  + a_1\sum_{i=0}^{n} x_{i}^2 - \sum_{i=0}^{n} x_iy_i  &= 0  \\
\end{align*}
$$

Leaving everything in terms of $\bar{x}$,

$$
\begin{align*}
    \sum_{i=0}^{n} y_i \bar{x} - a_1\sum_{i=0}^{n} x_i \bar{x} + a_1\sum_{i=0}^{n} x_{i}^2 - \sum_{i=0}^{n} x_iy_i  = 0
\end{align*}
$$

Grouping the terms that have $a_1$ on the left-hand side and the rest on the right-hand side:

$$
\begin{align*}
    a_1\left[ \sum_{i=0}^{n} x_{i}^2 - \sum_{i=0}^{n} x_i \bar{x}\right] &= \sum_{i=0}^{n} x_iy_i - \sum_{i=0}^{n} y_i \bar{x} \\
    a_1 \sum_{i=0}^{n} (x_{i}^2 - x_i \bar{x}) &= \sum_{i=0}^{n} (x_iy_i -  y_i \bar{x}) \\
    a_1 \sum_{i=0}^{n} x_{i}(x_{i} -\bar{x}) &= \sum_{i=0}^{n} y_i(x_i - \bar{x})
\end{align*}
$$

Finally, we get that:

$$
\begin{align}
    a_1 = \frac{ \sum_{i=0}^{n} y_{i} (x_i - \bar{x})}{\sum_{i=0}^{n} x_i (x_i - \bar{x})}
\end{align}
$$

Then our coefficients are:

$$
\begin{align}
    a_1 = \frac{ \sum_{i=0}^{n} y_{i} (x_i - \bar{x})}{\sum_{i=0}^{n} x_i (x_i - \bar{x})} \quad , \quad a_0  = \bar{y} - a_1\bar{x}
\end{align}
$$

### Let's fit!

Let's now fit a straight line through the temperature-anomaly data, to see the trend over time. We'll use least-squares linear regression to find the slope and intercept of a line

$$y = a_1x+a_0$$

that fits our data.

In our case, the `x`-data corresponds to `year`, and the `y`-data is `temp_anomaly`. To calculate our coefficients with the formula above, we need the mean values of our data. Sine we'll need to compute the mean for both `x` and `y`, it could be useful to write a custom Python _function_ that computes the mean for any array, and we can then reuse it.

It is good coding practice to *avoid repeating* ourselves: we want to write code that is reusable, not only because it leads to less typing but also because it reduces errors. If you find yourself doing the same calculation multiple times, it's better to encapsulate it into a *function*.

Remember the _key concept_ from [Lesson 1](http://go.gwu.edu/engcomp1lesson1): A function is a compact collection of code that executes some action on its arguments.

Once *defined*, you can *call* a function as many times as you want. When we *call* a function, we execute all the code inside the function. The result of the execution depends on the *definition* of the function and on the values that are *passed* into it as *arguments*. Functions might or might not *return* values in their last operation.

The syntax for defining custom Python functions is:

```python
def function_name(arg_1, arg_2, ...):
    '''
    docstring: description of the function
    '''
    <body of the function>
```

The **docstring** of a function is a message from the programmer documenting what he or she built. Docstrings should be descriptive and concise. They are important because they explain (or remind) the intended use of the function to the users. You can later access the docstring of a function using the function `help()` and passing the name of the function. If you are in a notebook, you can also prepend a question mark `'?'` before the name of the function and run the cell to display the information of a function.

Try it!

```python
?print
```

Using the `help` function instead:

```python
help(print)
```

    Help on built-in function print in module builtins:

    print(...)
        print(value, ..., sep=' ', end='\n', file=sys.stdout, flush=False)

        Prints the values to a stream, or to sys.stdout by default.
        Optional keyword arguments:
        file:  a file-like object (stream); defaults to the current sys.stdout.
        sep:   string inserted between values, default a space.
        end:   string appended after the last value, default a newline.
        flush: whether to forcibly flush the stream.

Let's define a custom function that calculates the mean value of any array. Study the code below carefully.

```python
def mean_value(array):
    """ Calculate the mean value of an array

    Arguments
    ---------
    array: Numpy array

    Returns
    -------
    mean: mean value of the array
    """
    sum_elem = 0
    for element in array:
            sum_elem += element # this is the same as sum_elem = sum_elem + element

    mean = sum_elem / len(array)

    return mean

```

Once you execute the cell above, the function`mean_value()` becomes available to use on any argument of the correct type. This function works on arrays of any length. We can try it now with our data.

```python
year_mean = mean_value(year)
print(year_mean)
```

    1948.0

```python
temp_anomaly_mean = mean_value(temp_anomaly)
print(temp_anomaly_mean)
```

    0.0526277372263

Neat! You learned how to write a Python function, and we wrote one for computing the mean value of an array of numbers. We didn't have to, though, because NumPy has a built-in function to do just what we needed: [`numpy.mean()`](https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.mean.html).

##### Exercise

Calculate the mean of the `year` and `temp_anomaly` arrays using the NumPy built-in function, and compare the results with the ones obtained using our custom `mean_value` function.

```python

```

Now that we have mean values, we can compute our coefficients by following equations (12). We first calculate $a_1$ and then use that value to calculate $a_0$.

Our coefficients are:

$$
    a_1 = \frac{ \sum_{i=0}^{n} y_{i} (x_i - \bar{x})}{\sum_{i=0}^{n} x_i (x_i - \bar{x})} \quad , \quad a_0  = \bar{y} - a_1\bar{x}
$$

We already calculated the mean values of the data arrays, but the formula requires two sums over new derived arrays. Guess what, NumPy has a built-in function for that: [`numpy.sum()`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.sum.html). Study the code below.

```python
a_1 = numpy.sum(temp_anomaly*(year - year_mean)) / numpy.sum(year*(year - year_mean))
```

```python
print(a_1)
```

    0.0103702839435

```python
a_0 = temp_anomaly_mean - a_1*year_mean
```

```python
print(a_0)
```

    -20.1486853847

##### Exercise

Write a function that computes the coefficients, call the function to compute them and compare the result with the values we obtained before. As a hint, we give you the structure that you should follow:

```python
def coefficients(x, y, x_mean, y_mean):
    """
    Write docstrings here
    """

    a_1 =
    a_0 =

    return a_1, a_0
```

We now have the coefficients of a linear function that best fits our data. With them, we can compute the predicted values of temperature anomaly, according to our fit. Check again the equations above: the values we are going to compute are $f(x_i)$.

Let's call `reg` the array obtined from evaluating $f(x_i)$ for all years.

```python
reg = a_0 + a_1 * year
```

With the values of our linear regression, we can plot it on top of the original data to see how they look together. Study the code below.

```python
pyplot.figure(figsize=(10, 5))

pyplot.plot(year, temp_anomaly, color='#2929a3', linestyle='-', linewidth=1, alpha=0.5)
pyplot.plot(year, reg, 'k--', linewidth=2, label='Linear regression')
pyplot.xlabel('Year')
pyplot.ylabel('Land temperature anomaly [°C]')
pyplot.legend(loc='best', fontsize=15)
pyplot.grid();
```

## Step 4: Apply regression using NumPy

Above, we coded linear regression from scratch. But, guess what: we didn't have to because NumPy has built-in functions that do what we need!

Yes! Python and NumPy are here to help! With [`polyfit()`](https://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.polyfit.html), we get the slope and $y$-intercept of the line that best fits the data. With [`poly1d()`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.poly1d.html), we can build the linear function from its slope and $y$-intercept.

Check it out:

```python
# First fit with NumPy, then name the coefficients obtained a_1n, a_0n:
a_1n, a_0n = numpy.polyfit(year, temp_anomaly, 1)

f_linear = numpy.poly1d((a_1n, a_0n))
```

```python
print(a_1n)
```

    0.0103702839435

```python
print(a_0n)
```

    -20.1486853847

```python
print(f_linear)
```

    0.01037 x - 20.15

```python
pyplot.figure(figsize=(10, 5))

pyplot.plot(year, temp_anomaly, color='#2929a3', linestyle='-', linewidth=1, alpha=0.5)
pyplot.plot(year, f_linear(year), 'k--', linewidth=2, label='Linear regression')
pyplot.xlabel('Year')
pyplot.ylabel('Land temperature anomaly [°C]')
pyplot.legend(loc='best', fontsize=15)
pyplot.grid();
```

## "Split regression"

If you look at the plot above, you might notice that around 1970 the temperature starts increasing faster that the previous trend. So maybe one single straight line does not give us a good-enough fit.

What if we break the data in two (before and after 1970) and do a linear regression in each segment?

To do that, we first need to find the position in our `year` array where the year 1970 is located. Thankfully, NumPy has a function called  [`numpy.where()`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.where.html) that can help us. We pass a condition and `numpy.where()` tells us where in the array the condition is `True`.

```python
numpy.where(year==1970)
```

    (array([90]),)

To split the data, we use the powerful instrument of _slicing_ with the colon notation. Remember that a colon between two indices indicates a range of values from a `start` to an `end`. The rule is that `[start:end]` includes the element at index `start` but excludes the one at index `end`. For example, to grab the first 3 years in our `year` array, we do:

```python
year[0:3]
```

    array([ 1880.,  1881.,  1882.])

Now we know how to split our data in two sets, to get two regression lines. We need two slices of the arrays `year` and `temp_anomaly`, which we'll save in new variable names below. After that, we complete two linear fits using the helpful NumPy functions we learned above.

```python
year_1 , temp_anomaly_1 = year[0:90], temp_anomaly[0:90]
year_2 , temp_anomaly_2 = year[90:], temp_anomaly[90:]

m1, b1 = numpy.polyfit(year_1, temp_anomaly_1, 1)
m2, b2 = numpy.polyfit(year_2, temp_anomaly_2, 1)

f_linear_1 = numpy.poly1d((m1, b1))
f_linear_2 = numpy.poly1d((m2, b2))
```

```python
pyplot.figure(figsize=(10, 5))

pyplot.plot(year, temp_anomaly, color='#2929a3', linestyle='-', linewidth=1, alpha=0.5)
pyplot.plot(year_1, f_linear_1(year_1), 'g--', linewidth=2, label='1880-1969')
pyplot.plot(year_2, f_linear_2(year_2), 'r--', linewidth=2, label='1970-2016')

pyplot.xlabel('Year')
pyplot.ylabel('Land temperature anomaly [°C]')
pyplot.legend(loc='best', fontsize=15)
pyplot.grid();
```

We have two different curves for two different parts of our data set. A little problem with this and is that the end point of our first regression doesn't match the starting point of the second regression. We did this for the purpose of learning, but it is not rigorously correct. We'll fix in in the next course module when we learn more about different types of regression.

## We learned:

* Making our plots more beautiful
* Defining and calling custom Python functions
* Applying linear regression to data
* NumPy built-ins for linear regression
* The Earth is warming up!!!

## References

1. [_Essential skills for reproducible research computing_](https://barbagroup.github.io/essential_skills_RRC/) (2017). Lorena A. Barba,  Natalia C. Clementi, Gilbert Forsyth.
2. _Numerical Methods in Engineering with Python 3_ (2013). Jaan Kiusalaas. Cambridge University Press.
3. _Effective Computation in Physics: Field Guide to Research with Python_ (2015). Anthony Scopatz & Kathryn D. Huff. O'Reilly Media, Inc.

```python
# Execute this cell to load the notebook's style sheet, then ignore it
from IPython.core.display import HTML
css_file = '../../style/custom.css'
HTML(open(css_file, "r").read())
```

<link href="https://fonts.googleapis.com/css?family=Merriweather:300,300i,400,400i,700,700i,900,900i" rel='stylesheet' type='text/css'>
<link href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:300,300i,400,400i,700,700i" rel='stylesheet' type='text/css'
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 800px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.5em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Source Sans Pro', sans-serif;
    line-height: 140%;
    font-size: 110%;
    width:680px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Merriweather', serif;
    font-style:regular;
    font-weight: bold;
    font-size: 250%;
    line-height: 100%;
    color: #004065;
    margin-bottom: 1em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Merriweather', serif;
    font-weight: bold;
    font-size: 180%;
    line-height: 100%;
    color: #0096d6;
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Merriweather', serif;
	font-size: 150%;
    margin-top:12px;
    margin-bottom: 3px;
    font-style: regular;
    color: #008367;
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Merriweather', serif;
    font-weight: 300;
    font-size: 100%;
    line-height: 120%;
    text-align: center;
    margin-top: 1em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Source Sans Pro', sans-serif;
    font-weight: regular;
    font-size: 130%;
    color: #e31937;
    font-style: italic;
    margin-bottom: .5em;
    margin-top: 1em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'Source Code Pro', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>


# PRÁCTICA 02.
# Perceptrón Simple.

### OBJETIVO:

Que el alumno implemente el perceptrón simple y lo aplique en distintos conjuntos de datos aplicando variaciones en la forma de aprendizaje del perceptrón.

```python
%matplotlib inline

import matplotlib.pyplot as plt
import matplotlib.cm as cm
from mpl_toolkits.mplot3d import Axes3D

import json, matplotlib
from IPython.core.pylabtools import figsize
figsize(11, 5)

from ipywidgets import interact, interact_manual, interactive, fixed
import ipywidgets as widgets
from IPython.display import display
import numpy as np

```

```python
class Perceptron:
    def __init__(self, num_inputs, weights = None, seed = 0):
        if weights: self.weights = np.copy(weights)
        else:
            np.random.seed(seed)
            self.weights = np.random.rand(num_inputs)

    def predict(self, input):
        ponderate_sum = np.dot(input, self.weights)
        result = 1.0 if ponderate_sum > 0 else 0.0

        return np.float64(result), ponderate_sum

    def train(self, input, desired_output, learning_rate):
        prediction, _ = self.predict(input)
        if prediction == desired_output: return

        add_or_substract = 1.0 if (prediction == 0.0 and desired_output == 1.0)  else -1.0
        self.weights = self.weights + (add_or_substract * input)

    def train_epoch(self, input_set, output_set, learning_rate, may_print = True):
        for index, input in enumerate(input_set):
            output = output_set[index]
            if may_print: print(f"input={input} index={index}")

            prediction, ponderate = self.predict(input)
            if may_print: print(f"prediction={prediction} real={output} ponderate={ponderate} ")

            self.train(input, output, learning_rate)

            prediction, ponderate = self.predict(input)
            if may_print: print(f"prediction={prediction} real={output} ponderate={ponderate} ")
            if may_print: print()

```

### PERCEPTRÓN

Un perceptrón simple es una unidad computacional con un umbral $\theta$, que al recibir $X \text{ de }n$ entradas reales $X = x_1, x_2, \cdots, x_n$ a través de sus pesos asociados $W = w_1, w_2, \cdots, w_n$, da una salida de valor 1 si la desigualdad $\sum_{i=1}^{n}w_ix_i \geq \theta$ se cumple, en otro caso la salida es 0.

La ley de aprendizaje se define como: $w_i(t+1) = w_i (t) + \alpha · d(x) · x_i$

$$
\
d(x)=
\begin{cases}
1, & \text{si } x \in A;\\
-1, & \text{si } x \in B
\end{cases}
\
$$

y $\alpha$ la tasa de aprendizaje. Los pesos no se modifican si $x$ está bien clasificado.

1.- Implementa la clase Perceptron que contenga lo siguiente:
   * Inicialización del perceptrón dados los pesos iniciales.
   * Función **evalua_escalon** que reciba X y evalúe la salida utilizando la función escalón en el intervalo [0,1].
   * Función **entrena**, debe entrenar al perceptrón: modificación de los pesos en una iteración cada que se llame a esta función dada una tasa de aprendizaje.

2.- Crea un *Perceptron* de pesos iniciales aleatorios y entrénalo con el conjunto de datos datosAND con inicialización de pesos aleatorios.

 * Completa la función **entrenaPerceptronAND** y utiliza tu función **entrena** para que en cada paso de entrenamiento se dibuje el conjunto de datos y el plano que divide al conjunto, tal como se muestra en la figura (utiliza colores para diferenciar al conjunto de datos)

HINT1: Recuerda que el vector de pesos W contiene el vector normal al plano.

HINT2: Puedes utilizar la función scatter y plot_surface

```python
and_input = np.array([[1, 0, 0, 0],
                     [1, 0 ,0, 1],
                     [1, 0, 1, 0],
                     [1, 0, 1, 1],
                     [1, 1, 0, 0],
                     [1, 1, 0, 1],
                     [1, 1, 1, 0],
                     [1, 1, 1, 1]]).astype(np.float64)

and_output = np.array([0,0,0,0,0, 0, 0, 1]).astype(np.float64)
```

```python
def show_line_with_perceptron(perceptron, title):
    fig = plt.figure()
    plane = fig.add_subplot(111, projection='3d')

    plane.set_xlabel('input_1')
    plane.set_ylabel('input_2')
    plane.set_zlabel('input_3')
    plane.set_xlim(-0.2, 1.2)
    plane.set_ylim(-0.2, 1.2)
    plane.set_zlim(-0.5, 2)

    plane.scatter(*data_and[:-1, 1:].T, c="red", label="0")
    plane.scatter(*data_and[-1, 1:].T, c="blue", label="1")

    d, a, b, c = perceptron.weights
    normal = np.array([a, b, c])
    point  = np.array([-d/a, 0, 0])
    d_p = -np.sum(point * normal)
    xx, yy = np.meshgrid(np.linspace(0, 1), np.linspace(0, 1))

    z3 = (-normal[0]*xx - normal[1]*yy - d_p)*1./normal[2]

    plane.plot_surface(xx,yy,z3, color='cyan')
    plane.set_title(title)

```

```python
# train
perceptron_and = None

@interact_manual(epochs=widgets.FloatSlider(min=1, max=10, step=1, value=6))
def entrenaPerceptronAND(epochs):
    global perceptron_and

    perceptron_and = Perceptron(4)
    epochs = int(epochs)
    for epoch in range(epochs):
        perceptron_and.train_epoch(and_input, and_output, 0.1, False)
        show_line_with_perceptron(perceptron_and, f"Epoch # {epoch + 1}")
        print(perceptron_and.predict([1, 0, 0, 0]))

```

    interactive(children=(FloatSlider(value=6.0, description='epochs', max=10.0, min=1.0, step=1.0), Button(descri…

```python
print(perceptron_and.predict([1,  1, 1, 1]))
```

### MÍNIMOS CUADRADOS

La meta es encontrar los pesos $w$ que mejor mapean los datos de entrada en los datos objetivo. Con ese fin, se plantea la ecuación de error:

\begin{equation}
    E(w) = \frac{1}{2} \| \bar{y} − \bar{t} \|^2
\end{equation}

Con $\bar{t}$ el vector de datos objetivo, $\bar{y}$ el vector de salidas asociadas $ y(x) = X^Tw $

A partir de la ecuación anterior obtenemos:

\begin{equation}
E(w) = \frac{1}{2}(\bar{y}-\bar{t})^T(\bar{y}-\bar{t})
\end{equation}

Derivando respecto a $w$

\begin{equation}
\frac{dE(w)}{dw} = \phi(\bar{y}-\bar{t})
\end{equation}

con $\phi= [x^1  \cdots x^N] \text{ y }x^i$ el i−ésimo vector de datos de entrada a la red. $N$ el número de datos/registros de entrenamiento.

De lo anterior tenemos:

\begin{equation}
w = (\phi \phi^T)^{-1}\phi \bar{t}
\end{equation}

### ENTRENAMIENTO Y PRUEBA

Hay dos fases en el modelado con redes neuronales:
* **Fase de entrenamiento:** Se utiliza un conjunto de datos o patrones de entrenamiento
para determinar los pesos que definen el modelo. Se calculan de manera iterativa con el objeto de minimizar el error cometido entre la salida obtenida por la red neuronal y la salida deseada.

* **Fase de prueba:** Se utiliza para estimar el error de generalización del modelo.

1. Dado el conjunto de datos datos.txt anexo, entrena un perceptrón con el 70% de los datos (elegidos aleatoriamente). El otro 30% será el conjunto de prueba.

2. Para el mismo conjunto de entrenamiento, calcula los pesos $w$ con el método de mínimos cuadrados.

2. Grafica y compara los resultados de $w$ obtenidos del entrenamiento del perceptrón y el de mínimos cuadrados. La salida dece ser similar a la figura siguiente:

3. Utiliza el conjunto de prueba para obtener la exactitud de los modelos.

OBS:  El archivo datos.txt contiene 3 columnas separadas por comas, las dos primeras columnas son el valor del conjunto $X = \{x_1, x_2\}$; la tercer columna son los datos objetivo $\bar{t}$

El criterio de parada tómalo como un número fijo de iteraciones o hasta que el conjunto esté  totalmente clasificado y fija la tasa de aprendizaje a una constante.

```python
# Split data
data = np.loadtxt(open("data.csv", "rb"), delimiter=",").astype(np.float64)
np.random.shuffle(data)
number_of_points = len(data)
split_index = int(0.7 * number_of_points)

training, test = data[ : split_index], data[split_index : ]

training_input, training_output = training[:, [0, 1]], training[:, 2]
test_input, test_output = test[:, [0, 1]], test[:, 2]

bias = np.ones((len(training_input), 1))
training_input = np.concatenate( (bias, training_input), axis=1)

bias = np.ones((len(test_input), 1))
test_input = np.concatenate( (bias, test_input), axis=1)

training_pure_data = training_input[:, 1:].T
test_pure_data = test_input[:, 1:].T
```

```python
# Train perceptron
perceptron2 = Perceptron(3)

for i in range(20):
    perceptron2.train_epoch(training_input, training_output, 0.08, False)

b, w1, w2 = perceptron2.weights
x = -b / w1
y = -b / w2

d = y
c = -y / x

line_x_perceptron = np.array([0, 6])
line_y_perceptron = c * line_x_perceptron + d
```

```python
# Get Least Squares
phi = training_input
t = np.copy(training_output)
t[t == 0.0] = -1

beta = np.linalg.inv(phi.T @ phi) @ (phi.T @ t)

line_x_least = np.linspace(0, 5, 20)
line_y_least = -beta[0] / beta[2] - (beta[1] / beta[2]) * line_x_least
```

```python
#Compare
plt.plot(line_x_perceptron, line_y_perceptron, label='Perceptron')
plt.plot(line_x_least, line_y_least, label='Least Squares')

plt.legend()
plt.scatter(*test_pure_data, c=test_output)

plt.show()
```


```python
from sympy import *

init_printing(use_unicode=True)
```

# Matriz con respecto a la base canónica de la proyección ortogonal sobre un subespacio

$$
\left[P_{gen\{G^{-1}w\}}\right]^{E}_{E}
$$

```python
def matriz_de_proyeccion_ortogonal_en_base_canonica(S, dimension, G=None):
    '''
        w: es el vector que genera al subespacio del cuál queremos obtener la matriz respecto a la base canónica de
           la proyección ortogonal si el producto interno fuese el canónico
            Ejemplo: si S = {x en R^3 : x1 + x2 + x3 = 0} => w = [1, 1, 1]
        dimension: es la dimension del espacio euclídeo en cuestión
        G: es la matriz de Gram del espacio euclídeo en cuestión
    '''
    if S.rank() > 2:
        raise NotImplementedError("Esta función sólo funciona para subespacios de dimensión <= 2")

    if not G:
        G = eye(dimension)

    if S.rank() == 2:
        S = S.col(0).cross(S.col(1))

    G_inv = G.inv()
    return  (1 / (S.T * G_inv * S)[0]) * G_inv * S * S.T
```

# Ejemplo

En $\mathbb{R}^{3}$ con el producto interno $\langle\cdot, \cdot\rangle$ definido por

$$
\langle x, y \rangle = y^{T} \begin{bmatrix}2 & -2 & 0\\ -2 & 5 & 4\\ 0 & 4 & 6\end{bmatrix} x
$$

se consideran los subespacios
$$
\mathbb{S}_{1} = \{x \in \mathbb{R}^{3} : x_{1} + x_{2} + x_{3} = 0\} \text{ y } \mathbb{S}_{2} = \{x \in \mathbb{R}^{3} : x_{1} - x_{3} = 0\}
$$

3.7 (a) Hallar las matrices con respecto a la base canónica de las proyecciones ortogonales de $\mathbb{R}^{3}$ sobre $\mathbb{S}_{1}^{\perp}$ y sobre $\mathbb{S}_{2}^{\perp}$

```python
S1_ort = Matrix([1, 1, 1])
S2_ort = Matrix([1, 0, -1])
G = Matrix([[2, -2, 0], [-2, 5, 4], [0, 4, 6]])
dim = 3

P_S1_ort = matriz_de_proyeccion_ortogonal_en_base_canonica(S1, dim, G)
P_S2_ort = matriz_de_proyeccion_ortogonal_en_base_canonica(S2_ort, dim, G)
display(P_S1_ort)
display(P_S2_ort)
```

# Distancia de un vector a un subespacio

$$
\text{Esta función computa }||P_{\mathbb{S}^{\perp}}(x)|| \text{ a partir de un subespacio } \mathbb{S} \text{ con } dim(\mathbb{S}) \leq 2, \text{ o a partir de } \mathbb{S}^{\perp} \text{ con } dim(\mathbb{S}) \leq 2.
$$

```python
def distancia_de_un_vector_a_un_subespacio(x: Matrix, S, dimension: int, G: Matrix=None):
    '''
        x: es el vector que queremos proyectar sobre el subespacio ortogonal de dimensión 1
        S: es el subespacio que queremos saber a qué distancia de x se encuentra
        dimension: es la dimension del espacio euclídeo en cuestión
        G: es la matriz de Gram del espacio euclídeo en cuestión

        Esta función calcula la distancia del vector x al subespacio S, sea S de dimensión 1, o 2
    '''
    if S.rank() > 2:
        raise NotImplementedError("Esta función sólo funciona para subespacios de dimensión <= 2")

    if not G:
        G = eye(dimension)

    if S.rank() == 2:
        S = S.col(0).cross(S.col(1))

    return S.T * x / sqrt((S.T * G.inv() * w)[0])
```

## Ejemplo

Sea $(\mathbb{V}, \langle\cdot, \cdot\rangle)$ un $\mathbb{R}$-espacio euclídeo de dimsensión 3, y sea $\{v_{1}, v_{2}, v_{3}\}$ un base de $\mathbb{V}$ cuya matriz de Gram es

$$
\begin{bmatrix}
    \frac{1}{5} & 0 & \frac{1}{3}\\
    0 & \frac{1}{3} & 0\\
    \frac{1}{3} & 0 & 1\\
\end{bmatrix}
$$

La distancia del vector $2v_{1} + 5v_{2} + 3v_{3}$ al subespacio gen$\{v_{1}, v_{2}\}$ es

```python
r = Rational # Misma función, nombre más corto - Es para que las fracciones no aparezcan en decimal
G_B = Matrix([[r('1/5'), 0, r('1/3')], [0, r('1/3'), 0], [r('1/3'), 0, 1]])
q_B = Matrix([2, 5, 3])
w = Matrix([0, 0, 1])

distancia_de_un_vector_a_un_subespacio(x=q_B, S=w, dimension=3, G=G_B)
```

notebooks_varios/Proyecciones Ortogonales y Distancia de un Vector a un Subespacio.ipynb