# Practical Session 2: Classification algorithms

*Notebook by Ekaterina Kochmar*

## 0.1 Your task

In practical 1, you worked with the housing prices and bike sharing datasets on the tasks that required you to predict some value (e.g., price of a house) or amount (e.g., the count of rented bikes, or the number of registered users) based on a number of attributes – age of the house, number of rooms, income level of the house owners for the house price prediction (or weather conditions and time of the day for the prediction of the number of rented bikes). That is, you were predicting some continuous value.

This time, your task is to predict a particular category the instance belongs to based on its characteristics. This type of tasks is called *classification*.

## 0.2 Dataset

First you will look into the famous [*Iris dataset*](https://en.wikipedia.org/wiki/Iris_flower_data_set), which was first introduced by the British statistician and biologist Ronald Fisher in his 1936 paper *The use of multiple measurements in taxonomic problems*. The dataset contains $4$ characteristics (sepal length and width, and petal length and width) for $3$ related species of irises – *setosa*, *versicolor* and *virginica*. Your task is to learn to predict, based on these $4$ characteristics, the type of an iris.

For further reference, see the original paper: R. A. Fisher (1936). *The use of multiple measurements in taxonomic problems*. Annals of Eugenics. 7 (2): 179–188.

## 0.3 Learning objectives

In this practical you will learn about:
- binary and multiclass classification
- linearly separable data
- the use of a number of classifiers, including Naive Bayes, Logistic Regression, and Perceptron
- kernel trick
- ways to evaluate the performance of a classification model, including accuracy, precision, recall and F$_1$ measure
- precision-recall trade-off and the ways to measure it, including ROC curves and AUC

In addition, you will learn about the dataset uploading routines with `sklearn`.

## Step 1: Uploading and inspecting the data

As before, let's start by uploading and looking into the data. In the previous practical, you worked with the data collected independently and stored in a comma-separated file, as the data in a real-life data science project might be. In this practical, you will learn how to use `sklearn`'s data uploading routines.

`sklearn` has a number of datasets to practice your ML skills on, and the `iris` dataset is one of them. Here is how you can access the dataset through `sklearn`. Note that such data fields as *data* and *target* are already pre-defined for you:

```python
from sklearn import datasets
iris = datasets.load_iris()
list(iris.keys())
```

Take a look into what is contained in *data*. Remember that each instance of an iris is described in terms of $4$ variables – *sepal length*, *sepal width*, *petal length*, and *petal width*:

```python
iris.data
```

To find out what variables are contained in `data`, check the `feature_names` data field.

```python
iris.feature_names
```

What about the target values?

```python
iris.target
```

There are $3$ classes of irises – *setosa*, *versicolor*, and *virginica*, and they are already converted into numerical values for you (recall, that when the dataset is not already preprocessed this way, and the target or any of the arrtibutes are represented as text or categorical data, you need to convert them into numerical data). If you want to check what each numerical label corresponds to in the original data, you can do so accessing the `target_names` data field:

```python
iris.target_names
```

Remember, that for further ML experiments, we need to have two data structures: the instance-by-attributes matrix $X$ and the target labels vector $y$. For instance, in the previous practical the regression algorithm learned the vector of weights $w$ to predict the target variable $\hat y^{(i)}$ for each instance $i$ so that its prediction would be maximally close to the actual label $y^{(i)}$. Since the labels $y^{(i)}$ were continuous (i.e., amount, number, or value), you measured the performance of your regressor by the distance between the predictions $\hat y$ and actual labels $y$. In this practical, you will need to work with $X$ and $y$, too, but the vector of $y$ this time will contain discrete values – classes $[0, 1, 2]$ for the different types of the flower.

As you might have already figured out, you need to initialise $X$ and $y$ with the `data` and `target` fields of your iris dataset:

```python
X, y = iris["data"], iris["target"]
print(X.shape)
print(y.shape)
```

Let's look closer into the data to get a feel for what is contained in it. As before, let's use visualisations with `matplotlib`, and in particular, plot one attribute against another for the three types of irises using scatterplot. E.g., let's plot sepal length vs. sepal width:

```python
%matplotlib inline
#so that the plot will be displayed in the notebook
import numpy as np
np.random.seed(42)

import matplotlib
from matplotlib import pyplot as plt

# visualise sepal length vs. sepal width
X = iris.data[:, :2]
y = iris.target

scatter_x = np.array(X[:, 0])
scatter_y = np.array(X[:, 1])
group = np.array(y)
cmap = matplotlib.cm.get_cmap('jet')
cdict = {0: cmap(0.1), 1: cmap(0.5), 2: cmap(0.9)}
labels = iris.target_names

fig, ax = plt.subplots(figsize=(8, 6))
for g in np.unique(group):
    ix = np.where(group == g)
    ax.scatter(scatter_x[ix], scatter_y[ix], c=np.array([cdict[g]]), #c = cdict[g],
               label = labels[g], s = 100, marker = "H",
               linewidth=2, alpha = 0.5)
ax.legend()
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')

plt.show()
```

It looks like *setosa* is quite clearly distiguishable from the other two types of irises with these two features. What about petal length and width?

```python
# visualise petal length vs. petal width
X = iris.data[:, 2:]
y = iris.target

scatter_x = np.array(X[:, 0])
scatter_y = np.array(X[:, 1])
group = np.array(y)
cmap = matplotlib.cm.get_cmap('jet')
cdict = {0: cmap(0.1), 1: cmap(0.5), 2: cmap(0.9)}
labels = iris.target_names

fig, ax = plt.subplots(figsize=(8, 6))
for g in np.unique(group):
    ix = np.where(group == g)
    ax.scatter(scatter_x[ix], scatter_y[ix], c=np.array([cdict[g]]), #c = cdict[g],
               label = labels[g], s = 100, marker = "H",
               linewidth=2, alpha = 0.5)
ax.legend()
plt.xlabel('Petal length')
plt.ylabel('Petal width')

plt.show()
```

This plot shows an even clearer separation between the class of *setosa* irises and the other two classes. In fact, with respect to these two attributes, it might be possible to clearly separate not only *setosas* from the other two classes, but also, with certain success, *versicolors* from *virginicas*.

When the data can be separated by a straight line (or a single decision surface) as in the example above, it is called *linearly separable*. This property of the data is successfully exploited by ML models that try to learn a linear separation boundary between the classes. In fact, there is a whole set of lines that you can use to separate the class of *setosas* from the other two classes, *versicolors* and *virginicas*, in this example. Some linear models explicitly allow you to select the *best* separation boundary by maximising the distance between the boundary and the closest instances of the two classes. Such are, for example, [Support Vector Machines](http://scikit-learn.org/stable/modules/svm.html), which will be covered in the [Part II Machine Learning and Bayesian Inference course](https://www.cl.cam.ac.uk/teaching/2021/MLBayInfer/):

```python
# visualise petal length vs. petal width
X = iris.data[:, 2:]
y = iris.target

scatter_x = np.array(X[:, 0])
scatter_y = np.array(X[:, 1])
group = np.array(y)
cmap = matplotlib.cm.get_cmap('jet')
cdict = {0: cmap(0.1), 1: cmap(0.5), 2: cmap(0.9)}
labels = iris.target_names

fig, ax = plt.subplots(figsize=(8, 6))
for g in np.unique(group):
    ix = np.where(group == g)
    ax.scatter(scatter_x[ix], scatter_y[ix], c=np.array([cdict[g]]), #c = cdict[g],
               label = labels[g], s = 100, marker = "H",
               linewidth=2, alpha = 0.5)
ax.legend()
plt.xlabel('Petal length')
plt.ylabel('Petal width')

for i in range(0, 9):
    plt.plot([3 + 0.5*i, 0], [0, 3-0.25*i], 'k-', color=cmap(0.1*(i+1)))

plt.show()
```

For the sake of consistency, let's plot all the pairs of features against each other. Note that all plots confirm that *setosas* are linearly separable from the other two types of irises (you might notice an occasional outlier, though), while *versicolors* are linearly separable from *virginicas* with respect to some attributes only:

```python
fig = plt.figure(figsize=(16, 12))
fig.subplots_adjust(hspace=0.2, wspace=0.2)

X = iris.data
y = iris.target
labels = iris.target_names
index = 1

for i in range(0, X.shape[1]):
    for j in range(0, X.shape[1]):
        scatter_x = np.array(X[:, i])
        scatter_y = np.array(X[:, j])
        group = np.array(y)
        cmap = matplotlib.cm.get_cmap('jet')
        cdict = {0: cmap(0.1), 1: cmap(0.5), 2: cmap(0.9)}

        ax = fig.add_subplot(X.shape[1], X.shape[1], index)
        index+=1

        for g in np.unique(group):
            ix = np.where(group == g)
            ax.scatter(scatter_x[ix], scatter_y[ix], c=np.array([cdict[g]]), #c = cdict[g],
                       label = labels[g], s = 50, marker = "H",
                       linewidth=1, alpha = 0.5)

plt.show()
```

## Step 2: Splitting the data into training and test subsets

Before applying the classifiers, let's split the dataset into the training and test sets. Recall, that when building an ML model, all further data exploration, feature selection and scaling, model selection and fine-tuning should be done on the training data, and the test data should only be used at the final step to evaluate the best estimated model:

```python
from sklearn.model_selection import train_test_split

train_set, test_set = train_test_split(X, test_size=0.2)
print(len(train_set), "training instances +", len(test_set), "test instances")
```

As before, you want your training and test data to contain enough representative examples of each class, that is, you should rather apply `StratifiedShuffleSplit` and not random splitting:

```python
from sklearn.model_selection import StratifiedShuffleSplit

split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
split.get_n_splits(X, y)
print(split)

for train_index, test_index in split.split(X, y):
    print("TRAIN:", len(train_index), "TEST:", len(test_index))
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]

print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)
```

Let's check the class proportions in the original dataset, and training and test subsets:

```python
import pandas as pd

# def original_proportions(data):
#     props = {}
#     for value in set(data["target"]):
#         data_value = [i for i in data["target"] if i==value]
#         props[value] = len(data_value) / len(data["target"])
#     return props

def subset_proportions(subset):
    props = {}
    for value in set(subset):
        data_value = [i for i in subset if i==value]
        props[value] = len(data_value) / len(subset)
    return props

compare_props = pd.DataFrame({
    "Overall": subset_proportions(iris["target"]),
    "Stratified tr": subset_proportions(y_train),
    "Stratified ts": subset_proportions(y_test),
})
compare_props["Strat. tr %error"] = 100 * compare_props["Stratified tr"] / compare_props["Overall"] - 100
compare_props["Strat. ts %error"] = 100 * compare_props["Stratified ts"] / compare_props["Overall"] - 100

compare_props.sort_index()
```

The original dataset is well-balanced – it contains exactly $50$ examples for each class. With the stratified data splits, you get equal proportions of each type of the irises in the training and test sets, too.

Now, let's first approach the classification task in a simpler setting: let's start with *binary classification* and try to predict whether an iris is of a particular type: e.g., *setosa* vs. *not-a-setosa*, or *versicolor* vs. *not-a-versicolor*.

## Case 1: Binary classification

Let's start by separating the data that describes *setosa* from other data.

```python
y_train_setosa = (y_train == 0) # will return True when the label is 0 (i.e., setosa)
y_test_setosa = (y_test == 0)
y_test_setosa
```

`y_test_setosa` returns a boolean vector of $30$ test instances: it contains `True` for the test instances that are *setosas*, and `False` otherwise. Let's pick one example of a *setosa* – for instance, the first one from the test set, `X_test[0]`, for further evaluations.

```python
setosa_example = X_test[0]
```

As you've noticed above, *setosas* are linearly separable from the other two classes, so it would be reasonable to apply a linear model to the task of separating *setosas* from *not-setosas*.

### Perceptron

A (single-layer) perceptron is a simple linear classifier that tries to learn the set of weights $w$ for the input vectors $X$ in order to predict the output binary class values $y$. In particular:

\begin{equation}
  \hat y^{(i)}=\begin{cases}
    1, & \text{if $w \cdot x^{(i)} + b > 0$}\\
    0, & \text{otherwise}
  \end{cases}
\end{equation}

where $w \cdot x^{(i)}$ is the dot product of weight vector $w$ and the feature vector $x^{(i)}$ for the instance $i$, $\sum_{j=1}^{m} w_{j}x_{j}^{(i)}$, and $b$ is the bias term.

`sklearn` has a perceptron implementation through its linear [`SGDClassifier`](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html#sklearn.linear_model.SGDClassifier) with the following parameter settings:

```python
from sklearn.linear_model import SGDClassifier

sgd = SGDClassifier(max_iter=5, tol=None, random_state=42,
                   loss="perceptron", eta0=1, learning_rate="constant", penalty=None)
sgd.fit(X_train, y_train_setosa)
sgd.predict([setosa_example])
```

The perceptron correctly predicts that `X_test[0]` is a *setosa* flower. However, as you've seen above, not all types of irises are linearly separable. Let's select a more challenging example of a *versicolor* (class $1$) for comparison.

```python
y_train_versicolor = (y_train == 1) # True when the label is 1 (i.e., versicolor)
y_test_versicolor = (y_test == 1)
y_test_versicolor
```

Select one of the examples to try your classifier on (array indexing in python starts with $0$, so you can pick indexes $2$, $3$, $5$ and so on):

```python
versicolor_example = X_test[17]
print("Class", y_test[17], "(", iris.target_names[y_test[17]], ")")

sgd.fit(X_train, y_train_versicolor)
print(sgd.predict([versicolor_example]))
```

Looks like perceptron indeed cannot predict the class of the *versicolor* example correctly. Let's see if another linear classifier – `Logistic Regression` – can do a better job.

### Logistic Regression

You used `Linear Regression` model in practical 1 to predict continuous values. In contrast, `Logistic Regression` is used for binary classification, that is, to predict a discrete value of $0$ or $1$. In particular, it estimates the probability that an instance belongs to a particular class (e.g., that $X\_test[0] \in {setosa}$). If the probability is greater than $50\%$, the instance is classified as *setosa* (positive class, labelled $1$ or $True$). Otherwise, it is classified as *not-a-setosa* (negative class, labelled $0$ or $False$).

Similarly to `Linear Regression`, `Logistic Regression` computes a weighted sum using the input features $w \cdot X$ plus an intercept ($w_0$), but instead of outputting the result as `Linear Regression` does, it further applies a *sigmoid function* to this result:

\begin{equation}
\hat p = \sigma (w \cdot X)
\end{equation}

The sigmoid function outputs a value between $0$ and $1$:

\begin{equation}
\sigma (t) = \frac{1}{1 + exp(-t)}
\end{equation}

Once the Logistic Regression model has estimated the probability $\hat p$, the label $\hat y$ is predicted as follows:

\begin{equation}
  \hat y=\begin{cases}
    1, & \text{if $\hat p \geq 0.5$}\\
    0, & \text{otherwise}
  \end{cases}
\end{equation}

Note that the above is equivalent to:

\begin{equation}
  \hat y=\begin{cases}
    1, & \text{if $t \geq 0$}\\
    0, & \text{otherwise}
  \end{cases}
\end{equation}

Let's apply a `Logistic Regression` model to our setosa example:

```python
from sklearn.linear_model import LogisticRegression

log_reg = LogisticRegression()
log_reg.fit(X_train, y_train_setosa)
```

```python
log_reg.predict([setosa_example])
```

The model correctly predicts that the test example is indeed a setosa. What about the versicolor example?

```python
log_reg.fit(X_train, y_train_versicolor)
log_reg.predict([versicolor_example])
```

Finally, for comparison, let's introduce one more classifier, that some of you might have come across before.

### Naive Bayes

If you did [Part IA Machine Learning and Real-World Data](https://www.cl.cam.ac.uk/teaching/1920/MLRD/) in the past, you might recall that you have already come across classification tasks: for example, you were asked to build a classifier that identifies sentiment in text, and you used `Naive Bayes` for that. `Naive Bayes` makes different assumptions about the data. In particular, it doesn't assume linear separability but makes the predictions based on the prior and the updated belief about the data.

To remind you, on a two class problem (e.g., distinguishing between classes $0$ for *not-setosas* and $1$ for *setosas*), a Naive Bayes model will predict:

\begin{equation}
  \hat y^{(i)} = argmax_{c \in (0, 1)} p(y=c | x^{(i)}) = \begin{cases}
    1, & \text{if $\hat p(y=1 | x^{(i)}) > \hat p(y=0 | x^{(i)}$})\\
    0, & \text{otherwise}
  \end{cases}
\end{equation}

where the probabilities are conditioned on the feature vector for $x^{(i)}$, i.e. $(f^{(i)}_{1}, ..., f^{(i)}_{n})$. In practice, it is impossible to estimate these probabilities exactly, and one uses the Naive Bayes theorem so that:

\begin{equation}
\hat p(y=c | x^{(i)}) = \frac{p(c) p(x^{(i)} | c)}{p(x^{(i)})}
\end{equation}

where $c \in \{0, 1\}$ is the class to be predicted. Since the denominator is the same for both estimates of $\hat p(y=1 | x^{(i)})$ and $\hat p(y=0 | x^{(i)})$, it can be omitted. Therefore, the estimate can be simplified to:

\begin{equation}
\hat y^{(i)} = argmax_{c \in (0, 1)} p(y=c | x^{(i)}) = argmax_{c \in (0, 1)} p(c) p(x^{(i)} | c)
\end{equation}

where $p(c)$ is the *prior belief* of the classifier about the distribution of the classes in the data, and $p(x^{(i)} | c)$ is the *posterior probability*. Both can be estimated from the training data using *maximum a posteriori (MAP)* estimation. Moreover, the "naive" independence assumption of this learning algorithm allows you to estimate $p(x^{(i)} | c)$ as a product of feature probabilities taken independently of each other, i.e.:

\begin{equation}
p(x^{(i)} | c) = p(f^{(i)}_{1}, ..., f^{(i)}_{n} | c) \approx p(f^{(i)}_{1} | c) \times ... \times p(f^{(i)}_{n} | c)
\end{equation}

`sklearn` has a number of implementations of a [`Naive Bayes`](http://scikit-learn.org/stable/modules/naive_bayes.html) algorithm which mainly differ from each other with respect to how they estimate the conditional probability above using the data and what believes they hold about the distribution of this data, e.g.:

```python
from sklearn.naive_bayes import GaussianNB, MultinomialNB

gnb = MultinomialNB() # or:
gnb = GaussianNB()
gnb.fit(X_train, y_train_setosa)
gnb.predict([setosa_example])
```

```python
gnb.fit(X_train, y_train_versicolor)

gnb.predict([versicolor_example])
```

As you can see, not all classifiers perform equally well. How do you measure their performance in a more comprehensive way?

## Step 3: Evaluation

### Performance measures

The most straightforward way to evaluate a classifier is to estimate how often its predictions are correct. This estimate is called *accuracy* and it is calculated as follows:

\begin{equation}
ACC = \frac{num(\hat y == y)}{num(\hat y == y) + num(\hat y != y)} = \frac{num(\hat y == 1 \& y == 1) + num(\hat y == 0 \& y == 0)}{num(\hat y == 1 \& y == 1) + num(\hat y == 0 \& y == 0) + num(\hat y == 1 \& y == 0) + num(\hat y == 0 \& y == 1)}
\end{equation}

E.g., for the *setosa* classification example, an accuracy of the classifier is the ratio of correctly identified *setosas* and correctly identified *not-setosas* to the total number of examples.

You can either import accuracy metric using `from sklearn.metrics import accuracy_score` or measure accuracy across multiple cross-validation folds (refer to practical 1, if you need to remind yourself what cross-validation does):

```python
from sklearn.model_selection import cross_val_score

print(cross_val_score(log_reg, X_train, y_train_setosa, cv=5, scoring="accuracy"))
print(cross_val_score(gnb, X_train, y_train_setosa, cv=5, scoring="accuracy"))
print(cross_val_score(sgd, X_train, y_train_setosa, cv=5, scoring="accuracy"))
```

All three classifiers are precfectly accurate in their prediction on the *setosa* example. What about the *versicolor* example?

```python
print(cross_val_score(log_reg, X_train, y_train_versicolor, cv=5, scoring="accuracy"))
print(cross_val_score(gnb, X_train, y_train_versicolor, cv=5, scoring="accuracy"))
print(cross_val_score(sgd, X_train, y_train_versicolor, cv=5, scoring="accuracy"))
```

The numbers differ but they still do not tell you much about the performance of a classifier in general. E.g., an accuracy of $\approx 0.71$ is far from perfect, but exactly how bad or how acceptable is it?

Let's implement a brute-force algorithm that will simply predict *not-versicolor* for every instance in the *versicolor* detection case (or *not-setosa* in the *setosa* detection case). Here is how well it will perform:

```python
from sklearn.base import BaseEstimator

class NotXClassifier(BaseEstimator):
    def fit(self, X, y=None):
        pass
    def predict(self, X):
        return np.zeros((len(X), 1), dtype=bool)

notversicolor_clf = NotXClassifier()
cross_val_score(notversicolor_clf, X_train, y_train_versicolor, cv=5, scoring="accuracy")
```

This gives you a very clear benchmark for comparison. The above represents a *majority class baseline*: for each split in cross-validation splits, it measures the proportion of the majority class (*not-versicolor* in this case). That is, if the classifier does nothing and simply returns the majority class label every time, this is how "well" it will perform. Obviously, you want the actual classifier that you build to do better than that.

### Confusion matrix

So now you can compare the accuracy (e.g., the proportion of correctly identified *setosas* and *not-setosas* in the dataset) to the baseline system. However, this doesn't help you understand *where* your classifier goes wrong. For example, does the low accuracy of the classifiers in the *versicolor* identification case suggest that they miss some *versicolors* and classify them as other types of irises, or does it suggest that they mistake other types of irises for *versicolors*? Or, perhaps, it's a combination of two types of mistakes?

The accuracy score itself doesn't allow you to make any of these conclusions. What you need to do is to look into the number of correctly and incorrectly classified instances, and the data representation that helps you do that is called *confusion matrix*. A confusion matrix is simply a table that compares the number of actual instances of type $c$ to the number of predicted instances of type $c$, i.e.:

|                  | predicted $\hat c=0$| predicted $\hat c=1$ |
|  -------------   |   :-------------:   |    :-------------:   |
| **actual $c=0$** | TN                  |   FP                 |
| **actual $c=1$** | FN                  |   TP                 |

The instances that are classified correctly as well as those that are misclassified have certain importance for evaluation of your classification algorithms. Here is the terminology:

- `TP` stands for *true positives*. These are the actual instances of class $1$ that are correctly classified as class $1$ (*setosas* identified as *setosas*);
- `TN` stands for *true negatives*. These are the actual instances of class $0$ that are correctly classified as class $0$ (*not-setosas* identified as *not-setosas*);
- `FN` stands for *false negatives*. These are the actual instances of class $1$ that are incorrectly classified as class $0$ (*setosas* identified as *not-setosas*, i.e. missed by the classifier);
- and finally, `FP` are *false positives*. These are the actual instances of class $0$ that are incorrectly classified as class $1$ (*not-setosas* identified as *setosas*, i.e. other types of flower mistaken for setosas by the classifier).

Now you can re-interpret the accuracy score as:

\begin{equation}
ACC = \frac{TP + TN}{TP + TN + FP + FN}
\end{equation}

So in order to maximise accuracy, you'll need to maximise on the number of true positives and true negatives.

The confusion matrix also tells you what exactly is misclassified by the classifier. For example, let's look into the confusion matrices for *setosa* identification:

```python
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import confusion_matrix

y_train_pred = cross_val_predict(log_reg, X_train, y_train_setosa, cv=5)
confusion_matrix(y_train_setosa, y_train_pred)
```

```python
y_train_pred = cross_val_predict(gnb, X_train, y_train_setosa, cv=5)
confusion_matrix(y_train_setosa, y_train_pred)
```

You know that the accuracy of these classifiers on *setosa* identification equals to $1$, so all *setosas* ($40$ instances) and *not-setosas* ($80$ instances) are classified correctly, and the confusion matrices show exactly that. What about the mistakes that the classifiers make on the *versicolor* example?

```python
y_train_pred = cross_val_predict(log_reg, X_train, y_train_versicolor, cv=5)
confusion_matrix(y_train_versicolor, y_train_pred)
```

```python
y_train_pred = cross_val_predict(gnb, X_train, y_train_versicolor, cv=5)
confusion_matrix(y_train_versicolor, y_train_pred)
```

The matrices above show that the `Logistic Regression` classifier correctly identifies $71$ *not-versicolors* (`TN`) and $15$ *versicolors* (`TP`), and it also misses $25$ *versicolors* (`FN`) and mistakes $9$ other flowers for *versicolors*. Now you can see that the bigger source of error for the `Logistic Regression` classifier is that it's not good enough in identifying *versicolors* as it misses more of them that it identifies ($25$ vs $15$). In contrast, `Naive Bayes` identifies more *versicolors* in general ($36 + 5$) and more of them correctly ($36$), while it misses only $4$ *versicolor* instances.

Apart from making such observations, you might want to be able to make sense of the number of correctly and incorrectly classified examples and compare the performance of the different classifiers in an aggregated way. The following set of measures will help you do so: *precision* ($P$), *recall* ($R$), and $F_{\beta}$*-measure* (e.g., $F_1$).

- *Precision* measures how reliable or trustworthy your classifier is. It tells you how often when the classifier predicts that a flower is a *versicolor* (class $1$) it actually is a *versicolor*. It relies on the number of $TP$'s and $FP$'s:
\begin{equation}
P = \frac{TP}{TP+FP}
\end{equation}

- *Recall* measures the coverage of your classifier. It tells you how many of the actual instances of *versicolor* your classifier can detect at all. It relies on the number of $TP$'s and $FN$'s:
\begin{equation}
R = \frac{TP}{TP+FN}
\end{equation}

- Finally, $F_1$*-score* combines the two measures above to give you an overall idea of your classifier's performance. $F_1$*-score* is estimated as follows:
\begin{equation}
F_1 = 2 \times \frac{P \times R}{P+R}
\end{equation}

It is the *harmonic mean* of the two measures. However, if you want to highlight the importance of precision in your task (and we'll talk about when you might want to do so later in this practical) or the importance of recall, you can do so by changing the $\beta$ coefficient of the *F-score*. In fact, the $F_1$*-score* is the most commonly used case of a more general $F_{\beta}$*-measure*, which is estimated as follows:
\begin{equation}
F_{\beta} = (1 + \beta^2) \times \frac{P \times R}{\beta^2 \times P+R}
\end{equation}

$\beta < 1$ puts more emphasis on precision than on recall, and $\beta > 1$ weighs recall higher than precision.

```python
from sklearn.metrics import precision_score, recall_score, f1_score

y_train_pred = cross_val_predict(gnb, X_train, y_train_versicolor, cv=5)
precision = precision_score(y_train_versicolor, y_train_pred) # == 36 / (36 + 5)
recall = recall_score(y_train_versicolor, y_train_pred) # == 36 / (36 + 4)
f1 = f1_score(y_train_versicolor, y_train_pred)
print(precision, recall, f1)

y_train_pred = cross_val_predict(log_reg, X_train, y_train_versicolor, cv=5)
precision = precision_score(y_train_versicolor, y_train_pred) # == 15 / (15 + 9)
recall = recall_score(y_train_versicolor, y_train_pred) # == 15 / (15 + 25)
f1 = f1_score(y_train_versicolor, y_train_pred)
print(precision, recall, f1)
```

The numbers above tell you that the `Naive Bayes` classifier has pretty good recall ($0.9$, or it identifies $90\%$ of *versicolor* instances) as well as precision ($\approx 0.88$, or in approximately $88\%$ of the cases you can trust its *versicolor* prediction). The two numbers are close to each other in range, so the $F_{1}$ score is high, too. The `Logistic Regression` classifier is less precise (you can trust it about $62.5\%$ of the time) and it misses many *versicolor* examples (identifying only $37.5\%$ of those).

Obviously, if your algorithm gets it all right, you will end up with perfect accuracy, precision and recall. In practice, however, the algorithms often don't get it all perfectly correct, and depending on the task at hand, you might decide that you are actually mostly interested in high precision of your algorithm, or in high recall.

For example, imagine you are working on a machine learning classifier that detects cancerous cases in the data coming from patients' analyses. The task of your algorithm is to detect whether the patient needs further tests and closer analysis based on the preliminary tests. The cost of *false negatives* (missed cancerous cases) in this task is very high and it's better to administer further tests for a patient about which the algorithm has doubts than to ignore the case altogether. In this case, you should prioritise recall over precision.

On the other hand, imagine you work for a pharmaceutical company trying to detect whether a particular drug will be applicable to a particular condition or a particular group of patients using data science and machine learning. Or, you want to learn in a data-driven way [when to change the drug dosage for patients in a hospital](http://www.cs.cornell.edu/people/tj/publications/morik_etal_99a.pdf). In these cases, the cost of *false positives*, i.e. deciding that the drug is suitable for a particular patient when it is not, or deciding to intervene in patients' treatment when the dosage should be kept as before, is more costly. These are the cases when you should prioritise precision over recall.

### Precision-recall trade-off

Often, you can achieve perfect precision if you lower the recall, and other way around. For example, by always predicting *versicolor* you will reach perfect recall (all *versicolor* instances will be covered by such prediction), but very low precision (because in most of the cases such prediction will be unreliable as your classifier will predict *versicolor* for all setosas and all virginicas as well). On the other hand, by trying to maximise precision, you will often need to constrain your classifier so that it returns less false positives and therefore is more conservative.

A good way to understand how a classifier makes its predictions is to look into its `decision_function` – the confidence score of the classifier's predictions. For `Logistic Regression` it returns a signed distance to the separation hyperplane selected by the model. Let's check the confidence score of the `Logistic Regression` model on the *versicolor* example:

```python
log_reg.fit(X_train, y_train_versicolor)

y_scores = log_reg.decision_function([versicolor_example])
y_scores
```

When this confidence score is higher than the predefined model's threshold, the model assigns the instance to the positive class, and otherwise it assigns the instance to the negative class. The `Logistic Regression`'s default threshold is $0$, so the example above is classified as *versicolor*:

```python
threshold = 0
y_versicolor_pred = (y_scores > threshold)
y_versicolor_pred
```

However, what would happen if you change the threshold to, for example, $0.7$? As you expect, this particular instance will be classified as *not-a-versicolor*. What effect will it have on the precision and recall of your classifier as a consequence?

```python
threshold = 0.7
y_versicolor_pred = (y_scores > threshold)
y_versicolor_pred
```

The following will return the confidence scores for each of the training set instances:

```python
y_scores = cross_val_predict(log_reg, X_train, y_train_versicolor, cv=5, method="decision_function")
y_scores
```

Let's use `sklearn`'s [`precision_recall_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html) functionality and plot the precision and recall values against the different threshold values. This shows you how the measures will be affected by changing the confidence threshold:

```python
from sklearn.metrics import precision_recall_curve

precisions, recalls, thresholds = precision_recall_curve(y_train_versicolor, y_scores)

def plot_pr_vs_threshold(precisions, recalls, thresholds):
    plt.plot(thresholds, precisions[:-1], "b--", label="Precision")
    plt.plot(thresholds, recalls[:-1], "g--", label="Recall")
    plt.xlabel("Threshold")
    plt.legend(loc="upper right")
    plt.ylim([0, 1])

plot_pr_vs_threshold(precisions, recalls, thresholds)
plt.show()
```

As you can see, the recall has a general tendency of decreasing when you increase the threshold – that is, the more conservative the classifier becomes, the more instances it is likely to miss. At the same time, precision curve is bumpier: with certain changes in the threshold, it might drop as well as the classifier misidentifies some other types of irises as *versicolor*.

You can also plot precision values against recall values and track the changes in precision against recall:

```python
def plot_precision_vs_recall(precisions, recalls):
    plt.plot(recalls, precisions, "b-", linewidth=2)
    plt.xlabel("Recall")
    plt.ylabel("Precision")
    plt.axis([0, 1, 0, 1])

plot_precision_vs_recall(precisions, recalls)
plt.show()
```

### The Receiver Operating Characteristic (ROC)

Another widely used way to inspect and present the results is to plot the receiver operating characteristic (ROC) curve. Similarly to the precision / recall curve above, it shows how the performance of the classifier changes at different threshold values. In particular, it plots the *true positive rate (TPR)* against the *false positive rate (FPR)*. Here is some more machine learning terminology:

- *True positive rate (TPR)* is simply another term for recall. Alternatively, it is also called *sensitivity*, or *probability of detection*;
- *False positive rate (FPR)* is also referred to as *fall-out* or *probability of false alarm*, and it can be calculated as $(1 − specificity)$. *Specificity*, in its turn, is estimated as:

\begin{equation}
specificity = \frac{TN}{TN + FP}
\end{equation}

In other words, *specificity* expresses the probability that the classifier correctly identifies *non-versicolors* in the *versicolor* identification example. *FPR*, therefore, shows how many incorrect *versicolor* votes (false alarms) the classifier makes for all *non-versicolor* examples while testing.

The ROC curve shows how the sensitivity of the classifier increases as a function of (i.e., at the expense of) the fall-out. A perfect classifier would have $100\%$ sensitivity (no false negatives) and $100\%$ specificity (no false positives), which in the ROC space can be illustrated by the point in the upper left corner with the coordinate $(0,1)$. The closer the actual ROC curve gets to this point, the better is the classifier. A random guess would give a point along a diagonal line from the left bottom to the top right corners. Points above the diagonal represent good classification results (better than random); points below the line represent bad results (worse than random).

To plot the ROC curve, let's first estimate the *FPR* and *TPR* for different threshold values using `sklearn`'s `roc_curve` functionality, and then plot *FPR* and *TPR* using `matplotlib`:

```python
from sklearn.metrics import roc_curve

fpr, tpr, thresholds = roc_curve(y_train_versicolor, y_scores)

def plot_roc_curve(fpr, tpr, label=None):
    plt.plot(fpr, tpr, linewidth=2, label=label)
    plt.plot([0, 1], [0, 1], "k--")
    plt.axis([0, 1, 0, 1.01])
    plt.xlabel("False positive rate (fpr)")
    plt.ylabel("True positive rate (tpr)")

plot_roc_curve(fpr, tpr)
plt.show()
```

Another characteristic of the ROC curve is the *area under the curve (AUC)*, which for a perfect classifier will equal $1$ and for a random one will equal $0.5$. A estimate on a actual classifier, thus, usually lies between these two values:

```python
from sklearn.metrics import roc_auc_score
roc_auc_score(y_train_versicolor, y_scores)
```

Now, the `Logistic Regression` classifier does not perform very well on the *versicolor* identification task, and the best classifier among the three so far is `Naive Bayes`. Let's compare its performance to the `Logistic Regression` and visualise the difference using the ROC curve. Note that `Naive Bayes` classifier implementation in `sklearn` doesn't use `decision_function`; [`predict_proba`](https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.GaussianNB.html#sklearn.naive_bayes.GaussianNB.predict_proba) is the equivalent of that for `Naive Bayes`:

```python
y_probas_gnb = cross_val_predict(gnb, X_train, y_train_versicolor, cv=3, method="predict_proba")
y_scores_gnb = y_probas_gnb[:, 1] # score = proba of the positive class
fpr_gnb, tpr_gnb, thresholds_gnb = roc_curve(y_train_versicolor, y_scores_gnb)

plt.plot(fpr, tpr, "b:", label="Logistic Regression")
plot_roc_curve(fpr_gnb, tpr_gnb, "Gaussian Naive Bayes")
plt.legend(loc="lower right")
plt.show()
```

The curve for `Naive Bayes` is much closer to the upper left corner, which shows that the classifier is better than `Logistic Regression` on this task. Let's estimate the AUC as well:

```python
roc_auc_score(y_train_versicolor, y_scores_gnb)
```

## Step 4: Data transformations

So far, you have trained three classifiers on two classification tasks – identification of *setosas* and *versicolors*. You've seen that all three classifiers perform well on the *setosas* identification example as the data is linearly separable, and only `Naive Bayes` performs well in the *versicolors* case, because the data with the given features is not linearly separable.

For example, consider a decision boundary that a `Logistic Regression` classifier would learn for the linearly separable class of *setosas* (based on the first two features):

```python
X = iris.data[:, :2] # consider the first two features for plotting (as in Step 1)
y = iris.target

scatter_x = np.array(X[:, 0])
scatter_y = np.array(X[:, 1])
group = np.array(y)
cmap = matplotlib.cm.get_cmap('jet')
cdict = {0: cmap(0.1), 1: cmap(0.5), 2: cmap(0.9)}
labels = iris.target_names

fig, ax = plt.subplots(figsize=(8, 6))
for g in np.unique(group):
    ix = np.where(group == g)
    ax.scatter(scatter_x[ix], scatter_y[ix], c=np.array([cdict[g]]), #c = cdict[g],
               label = labels[g], s = 100, marker = "H",
               linewidth=2, alpha = 0.5)

log_reg.fit(X_train[:, :2], y_train_setosa) # train the classifier for setosas, using the first two features only
w = log_reg.coef_[0]
i = log_reg.intercept_[0]
xx = np.linspace(4, 8) # generate some values for feature1 (sepal length) in the appropriate range of values
yy = -(w[0]*xx + i)/w[1] # estimate the value for feature2 (sepal width) based on the learned weights and intercept

plt.plot(xx, yy, 'b--')

ax.legend()
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')

plt.show()
```

and compare it to the case of non-linearly separable class of *versicolors*:

```python
fig, ax = plt.subplots(figsize=(8, 6))
for g in np.unique(group):
    ix = np.where(group == g)
    ax.scatter(scatter_x[ix], scatter_y[ix], c=np.array([cdict[g]]), #c = cdict[g],
               label = labels[g], s = 100, marker = "H",
               linewidth=2, alpha = 0.5)

log_reg.fit(X_train[:, :2], y_train_versicolor)
w = log_reg.coef_[0]
i = log_reg.intercept_[0]
xx = np.linspace(4, 8)
yy = -(w[0]*xx + i)/w[1]

plt.plot(xx, yy, 'g--')

ax.legend()
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')

plt.show()
```

The original representation of the data that you've been using so far is simply not expressive enough. But what if you could transform this data in such a way that it could be linearly separated? In fact, there is a "trick" for this problem that involves transforming the original data using a number of new dimensions in such a way that in this new high-dimensional space the classes become linearly separable. This "trick" is commonly known as the *kernel trick* or *kernel method*.

### Kernel trick  and approximate kernel maps

You will most commonly hear about this method in the context of Support Vector Machines (SVMs), for example in the [Part II Machine Learning and Bayesian Inference course](https://www.cl.cam.ac.uk/teaching/2021/MLBayInfer/). However, some other linear classifiers, including [perceptron](https://en.wikipedia.org/wiki/Kernel_perceptron), allow for the kernel methods to be used as well.

The general motivation behind the kernel trick is as follows: when the data is not linearly separable, i.e. there is no clear dividing boundary between the two classes, the kernel trick allows you to transform the data using a number of additional dimensions that would allow for such clear dividing boundary to be learned. Kernel methods require a user-specified kernel, i.e. a function that will transform the original data into a higher dimensional space. Polynomial and Gaussian (also known as *radial-basis function*) transformations are among the most widely used kernel functions. These data transformations might remind you of the polynomial feature transformation used with the linear regression in practical 1: recall that you cast the original features from the space where the relation between the features and the output cannot exactly be captured by a linear function, into a higher dimensional feature space using  polynomial function. Recall also that this allowed you to apply a linear function to this new feature space leading to an improved result.

You will learn more about kernel functions and their application to SVMs in more detail in the Part II Machine Learning and Bayesian Inference course. For the task at hand, you will use an `sklearn`'s [approximate kernel map](http://scikit-learn.org/stable/modules/kernel_approximation.html) in combination with the perceptron implementation of the `SGDClassifier`:

```python
from sklearn.kernel_approximation import RBFSampler

rbf_features = RBFSampler(gamma=1, random_state=42)
X_train_features = rbf_features.fit_transform(X_train)
print(X_train.shape, "->", X_train_features.shape)
sgd_rbf = SGDClassifier(max_iter=100, random_state=42, loss="perceptron",
                        eta0=1, learning_rate="constant", penalty=None)
sgd_rbf.fit(X_train_features, y_train_versicolor)

sgd_rbf.score(X_train_features, y_train_versicolor)
```

The output above shows that the classifier tries to learn the separation boundary in a highly-dimensional transformed feature space, and the accuracy of this learning on the training set is over $0.99$. Let's test this classifier in a 5-fold cross-validation manner, and compare precision, recall and F$_1$ scores to the linear classifier trained on the original data:

```python
y_train_pred = cross_val_predict(sgd, X_train, y_train_versicolor, cv=5)
precision = precision_score(y_train_versicolor, y_train_pred)
recall = recall_score(y_train_versicolor, y_train_pred)
f1 = f1_score(y_train_versicolor, y_train_pred)
print(precision, recall, f1)

y_train_pred = cross_val_predict(sgd_rbf, X_train_features, y_train_versicolor, cv=5)
precision = precision_score(y_train_versicolor, y_train_pred)
recall = recall_score(y_train_versicolor, y_train_pred)
f1 = f1_score(y_train_versicolor, y_train_pred)
print(precision, recall, f1)
```

Looks like the kernel trick helped improve the results on the originally non-linearly separable data significantly!

## Case 2:  Multi-class classification

Now remember that your actual goal is to build a three-way classifier that can predict *setosa*, *versicolor* and *virginica* classes, and not just tell whether an instance is an $X$ (*setosa* or *versicolor*) or not. Actually, you are already half way there, and here is why.

Some classifiers are capable of handling multiple classes directly. For example, `Naive Bayes` learns about the probabilities of the classes in the data irrespective of the number of classes. Therefore, the binary examples above can naturally be extended to the 3-class classification scenario: you simply provide the classifier with the data on the $3$ rather than $2$ classes.

In contrast, such classifiers as perceptron (`SDGClassifier`) and `Logistic Regression`, which seek to learn a linear separation boundary, are inherently binary classifiers: they try to learn a single separation boundary between two classes at a time. However, they can also very easily be extended to handle more than $2$ classes. Multi-class classification with such linear classifiers generally follows one of the two routes:

- with the *one-vs-all* (*OvA*, or *one-vs-rest*, *OvR*) strategy you train $n$ classifiers (e.g., a setosa detector, a versicolor detector and a virginica detector). For a new instance, you apply all of the classifiers and predict the class that gets the highest score returned by the classifiers;
- with the *one-vs-one* (*OvO*) strategy, you train a binary classifier for each pair of classes in your data and select the class that wins most of the duels.

There are pros and cons to each of these approaches. E.g., with the *OvO* strategy, you end up training $n \times (n-1) / 2$ classifiers. I.e. for the iris dataset you will have $3$ classifiers (exactly as with the *OvA* strategy) but on a $10$-class problem this will amount to $45$ classifiers. On the other hand, the training sets with the *OvO* will be much smaller and more balanced. Some classifiers scale poorly with the size of the training set when it is imbalanced, so *OvO* for them is preferable (e.g., such are SVMs), but most of the linear classifiers use *OvA* instead.

The nice thing about `sklearn` is that it implements the above strategies under the hood, so to perform a multi-class classification with the `SDGClassifier`, all you need to do is to provide it with the data and labels on $3$ classes, and it will train $3$ binary *OvA* classifiers and output the class with the highest score, i.e.:

```python
sgd.fit(X_train, y_train) # i.e., all instances, not just one class
print(sgd.predict([setosa_example]))
print(sgd.predict([versicolor_example]))
```

Recall that the *versicolor* class label is $1$, so the classifier's output is correct this time. Let's also check the result with the RBF kernel:

```python
sgd_rbf.fit(X_train_features, y_train) # i.e., all instances, not just one class
X_test_features = rbf_features.transform(X_test)
setosa_rbf_example = X_test_features[0] # note that you need to transform the test data in the same way, too
versicolor_rbf_example = X_test_features[17]

print(sgd_rbf.predict([setosa_rbf_example]))
print(sgd_rbf.predict([versicolor_rbf_example]))
```

This classifier classified both test examples correctly. Let's see what logic the classifier followed:

```python
setosa_scores = sgd_rbf.decision_function([setosa_rbf_example])
print(setosa_scores)

# check which class gets the maximum score
prediction = np.argmax(setosa_scores)
print(prediction)
# check which class this corresponds to in the classifier
print(sgd_rbf.classes_[prediction])
print(iris.target_names[sgd_rbf.classes_[prediction]])
```

This shows that *setosa* class got a much higher score than the other two. What about the versicolor example?

```python
versicolor_scores = sgd_rbf.decision_function([versicolor_rbf_example])
print(versicolor_scores)
prediction = np.argmax(versicolor_scores)
print(prediction)
print(iris.target_names[sgd_rbf.classes_[prediction]])
```

For comparison, let's see what the original `SGDClassifier` (without the RBF kernel) predicted:

```python
versicolor_scores = sgd.decision_function([versicolor_example])
print(versicolor_scores)
prediction = np.argmax(versicolor_scores)
print(prediction)
print(iris.target_names[sgd.classes_[prediction]])
```

Now, if you'd like to use the *OvO* strategy, you can enforce the `sklearn` to do so creating an instance of a `OneVsOneClassifier` (similarly, `OneVsRestClassifier` for *OvA*). Note that you're essentially using the same classifier as before, and it is just the framework within which the predictions are made:

```python
from sklearn.multiclass import OneVsOneClassifier

ovo_clf = OneVsOneClassifier(SGDClassifier(max_iter=100, random_state=42, loss="perceptron",
                                           eta0=1, learning_rate="constant", penalty=None))
ovo_clf.fit(X_train_features, y_train)
ovo_clf.predict([versicolor_rbf_example])
```

```python
len(ovo_clf.estimators_)
```

Now let's look into the `NaiveBayes` performance on the $3$-class problem:

```python
gnb.fit(X_train, y_train)
gnb.predict([versicolor_example])
```

It correctly classifies the *versicolor* example, so let's check how confident it is about this prediction (remember that you should use `predict_proba` with `NaiveBayes` and `decision_function` with the `SGDClassifier`):

```python
gnb.predict_proba([versicolor_example])
```

Let's look into the cross-validated performance of the classifiers:

```python
print(cross_val_score(sgd_rbf, X_train_features, y_train, cv=5, scoring="accuracy"))
print(cross_val_score(ovo_clf, X_train_features, y_train, cv=5, scoring="accuracy"))
print(cross_val_score(gnb, X_train, y_train, cv=5, scoring="accuracy"))
```

Finally, recall that in practical 1 you used further transformations on the data, e.g. scaling. Let's apply scaling and compare the results (note that you can use `np.mean()` to get the average accuracy score across all splits if you wish to get a single aggregated accuracy score instead of $5$ separate ones):

```python
from sklearn.preprocessing import StandardScaler, MinMaxScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train.astype(np.float64))
X_train_features_scaled = scaler.fit_transform(X_train_features.astype(np.float64))

print(cross_val_score(sgd_rbf, X_train_features_scaled, y_train, cv=5, scoring="accuracy"))
print(cross_val_score(ovo_clf, X_train_features_scaled, y_train, cv=5, scoring="accuracy"))
print(cross_val_score(gnb, X_train_scaled, y_train, cv=5, scoring="accuracy"))
```

## Step 5: Error analysis

Before applying the classifiers to the test data, let's gain a bit more insight into what the classifiers get wrong. Recall, that earlier you used confusion matrices to learn about the classification errors:

```python
y_train_pred = cross_val_predict(sgd_rbf, X_train_features_scaled, y_train, cv=3)
conf_mx = confusion_matrix(y_train, y_train_pred)
conf_mx
```

Let's visualise the classifier decisions:

```python
plt.imshow(conf_mx, cmap = "jet")
plt.show()
```

And if you'd like to highlight only the most salient errors, you can do so as follows (the `"jet"` color scheme uses red spectrum for higher numbers):

```python
row_sums = conf_mx.sum(axis=1, keepdims=True)
norm_conf_mx = conf_mx / row_sums
np.fill_diagonal(norm_conf_mx, 0)
plt.imshow(norm_conf_mx, cmap = "jet")
plt.show()
```

## Final step – evaluating on the test set

The `SGDClassifier` with the RBF kernel:

```python
from sklearn.metrics import accuracy_score

X_test_features_scaled = scaler.transform(X_test_features.astype(np.float64))
y_pred = sgd_rbf.predict(X_test_features_scaled)
accuracy_score(y_test, y_pred)
```

```python
precision = precision_score(y_test, y_pred, average='weighted')
recall = recall_score(y_test, y_pred, average='weighted')
f1 = f1_score(y_test, y_pred, average='weighted')
print(precision, recall, f1)
```

The *OvO* SGD classifier:

```python
from sklearn.metrics import accuracy_score

X_test_features_scaled = scaler.transform(X_test_features.astype(np.float64))
y_pred = ovo_clf.predict(X_test_features_scaled)
accuracy_score(y_test, y_pred)
```

```python
precision = precision_score(y_test, y_pred, average='weighted')
recall = recall_score(y_test, y_pred, average='weighted')
f1 = f1_score(y_test, y_pred, average='weighted')
print(precision, recall, f1)
```

The `NaiveBayes` classifier:

```python
gnb.fit(X_train, y_train)
y_pred = gnb.predict(X_test)
accuracy_score(y_test, y_pred)
```

```python
precision = precision_score(y_test, y_pred, average='weighted')
recall = recall_score(y_test, y_pred, average='weighted')
f1 = f1_score(y_test, y_pred, average='weighted')
print(precision, recall, f1)
```

#  Assignment: Handwritten digits dataset

The dataset that you will use in this assignment is the [*digits* dataset](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html) which contains $1797$ images of $10$ hand-written digits. The digits have been preprocessed so that $32 \times 32$ bitmaps are divided into non-overlapping blocks of $4 \times 4$ and the number of on pixels are counted in each block. This generates an input matrix of $8 \times 8$ where each element is an integer in the range of $[0, ..., 16]$. This reduces dimensionality and gives invariance to small distortions.

For further information on NIST preprocessing routines applied to this data, see M. D. Garris, J. L. Blue, G. T. Candela, D. L. Dimmick, J. Geist, P. J. Grother, S. A. Janet, and C. L. Wilson, *NIST Form-Based Handprint Recognition System*, NISTIR 5469, 1994.

As before, use the `sklearn`'s data uploading routines to load the dataset and get the data fields:

```python
from sklearn import datasets
digits = datasets.load_digits()
list(digits.keys())
```

```python
digits
```

```python
X, y = digits["data"], digits["target"]
X.shape
```

```python
y.shape
```

You can access the digits and visualise them using the following code (feel free to select another digit):

```python
some_digit = X[3]
some_digit_image = some_digit.reshape(8, 8)

plt.imshow(some_digit_image, cmap=matplotlib.cm.binary, interpolation="nearest")
plt.axis("off")
plt.show()
```

```python
y[3]
```

For the rest of the practical, apply the data preprocessing techniques, implement and evaluate the classification models on the digits dataset using the steps that you applied above to the iris dataset.

```python

```


# Geometric Series for Elementary Economics

## Contents

- [Geometric Series for Elementary Economics](#Geometric-Series-for-Elementary-Economics)
  - [Overview](#Overview)
  - [Key Formulas](#Key-Formulas)
  - [Example: The Money Multiplier in Fractional Reserve Banking](#Example:-The-Money-Multiplier-in-Fractional-Reserve-Banking)
  - [Example: The Keynesian Multiplier](#Example:-The-Keynesian-Multiplier)
  - [Example: Interest Rates and Present Values](#Example:-Interest-Rates-and-Present-Values)
  - [Back to the Keynesian Multiplier](#Back-to-the-Keynesian-Multiplier)

## Overview

The lecture describes important ideas in economics that use the mathematics of geometric series.

Among these are

- the Keynesian **multiplier**
- the money **multiplier** that prevails in fractional reserve banking
  systems
- interest rates and present values of streams of payouts from assets

(As we shall see below, the term **multiplier** comes down to meaning **sum of a convergent geometric series**)

These and other applications prove the truth of the wise crack that

> “in economics, a little knowledge of geometric series goes a long way “

Below we’ll use the following imports:

```python
%matplotlib inline
import matplotlib.pyplot as plt
plt.rcParams["figure.figsize"] = (11, 5)  #set default figure size
import numpy as np
import sympy as sym
from sympy import init_printing, latex
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D
```

## Key Formulas

To start, let $ c $ be a real number that lies strictly between
$ -1 $ and $ 1 $.

- We often write this as $ c \in (-1,1) $.
- Here $ (-1,1) $ denotes the collection of all real numbers that
  are strictly less than $ 1 $ and strictly greater than $ -1 $.
- The symbol $ \in $ means *in* or *belongs to the set after the symbol*.

We want to evaluate geometric series of two types – infinite and finite.

### Infinite Geometric Series

The first type of geometric that interests us is the infinite series

$$
1 + c + c^2 + c^3 + \cdots
$$

Where $ \cdots $ means that the series continues without end.

The key formula is

$$
1 + c + c^2 + c^3 + \cdots = \frac{1}{1 -c } \tag{1}
$$

To prove key formula [(1.1)](#equation-infinite), multiply both sides  by $ (1-c) $ and verify
that if $ c \in (-1,1) $, then the outcome is the
equation $ 1 = 1 $.

### Finite Geometric Series

The second series that interests us is the finite geometric series

$$
1 + c + c^2 + c^3 + \cdots + c^T
$$

where $ T $ is a positive integer.

The key formula here is

$$
1 + c + c^2 + c^3 + \cdots + c^T  = \frac{1 - c^{T+1}}{1-c}
$$

**Remark:** The above formula works for any value of the scalar
$ c $. We don’t have to restrict $ c $ to be in the
set $ (-1,1) $.

We now move on to describe some famous economic applications of
geometric series.

## Example: The Money Multiplier in Fractional Reserve Banking

In a fractional reserve banking system, banks hold only a fraction
$ r \in (0,1) $ of cash behind each **deposit receipt** that they
issue

- In recent times
  - cash consists of pieces of paper issued by the government and
    called dollars or pounds or $ \ldots $
  - a *deposit* is a balance in a checking or savings account that
    entitles the owner to ask the bank for immediate payment in cash
- When the UK and France and the US were on either a gold or silver
  standard (before 1914, for example)
  - cash was a gold or silver coin
  - a *deposit receipt* was a *bank note* that the bank promised to
    convert into gold or silver on demand; (sometimes it was also a
    checking or savings account balance)

Economists and financiers often define the **supply of money** as an
economy-wide sum of **cash** plus **deposits**.

In a **fractional reserve banking system** (one in which the reserve
ratio $ r $ satisfies $ 0 < r < 1 $), **banks create money** by issuing deposits *backed* by fractional reserves plus loans that they make to their customers.

A geometric series is a key tool for understanding how banks create
money (i.e., deposits) in a fractional reserve system.

The geometric series formula [(1.1)](#equation-infinite) is at the heart of the classic model of the money creation process – one that leads us to the celebrated
**money multiplier**.

### A Simple Model

There is a set of banks named $ i = 0, 1, 2, \ldots $.

Bank $ i $’s loans $ L_i $, deposits $ D_i $, and
reserves $ R_i $ must satisfy the balance sheet equation (because
**balance sheets balance**):

$$
L_i + R_i = D_i \tag{2}
$$

The left side of the above equation is the sum of the bank’s **assets**,
namely, the loans $ L_i $ it has outstanding plus its reserves of
cash $ R_i $.

The right side records bank $ i $’s liabilities,
namely, the deposits $ D_i $ held by its depositors; these are
IOU’s from the bank to its depositors in the form of either checking
accounts or savings accounts (or before 1914, bank notes issued by a
bank stating promises to redeem note for gold or silver on demand).

Each bank $ i $ sets its reserves to satisfy the equation

$$
R_i = r D_i \tag{3}
$$

where $ r \in (0,1) $ is its **reserve-deposit ratio** or **reserve
ratio** for short

- the reserve ratio is either set by a government or chosen by banks
  for precautionary reasons

Next we add a theory stating that bank $ i+1 $’s deposits depend
entirely on loans made by bank $ i $, namely

$$
D_{i+1} = L_i \tag{4}
$$

Thus, we can think of the banks as being arranged along a line with
loans from bank $ i $ being immediately deposited in $ i+1 $

- in this way, the debtors to bank $ i $ become creditors of
  bank $ i+1 $

Finally, we add an *initial condition* about an exogenous level of bank
$ 0 $’s deposits

$$
D_0 \ \text{ is given exogenously}
$$

We can think of $ D_0 $ as being the amount of cash that a first
depositor put into the first bank in the system, bank number $ i=0 $.

Now we do a little algebra.

Combining equations [(1.2)](#equation-balance) and [(1.3)](#equation-reserves) tells us that

$$
L_i = (1-r) D_i \tag{5}
$$

This states that bank $ i $ loans a fraction $ (1-r) $ of its
deposits and keeps a fraction $ r $ as cash reserves.

Combining equation [(1.5)](#equation-fraction) with equation [(1.4)](#equation-deposits) tells us that

$$
D_{i+1} = (1-r) D_i  \ \text{ for } i \geq 0
$$

which implies that

$$
D_i = (1 - r)^i D_0  \ \text{ for } i \geq 0 \tag{6}
$$

Equation [(1.6)](#equation-geomseries) expresses $ D_i $ as the $ i $ th term in the
product of $ D_0 $ and the geometric series

$$
1, (1-r), (1-r)^2, \cdots
$$

Therefore, the sum of all deposits in our banking system
$ i=0, 1, 2, \ldots $ is

$$
\sum_{i=0}^\infty (1-r)^i D_0 =  \frac{D_0}{1 - (1-r)} = \frac{D_0}{r} \tag{7}
$$

### Money Multiplier

The **money multiplier** is a number that tells the multiplicative
factor by which an exogenous injection of cash into bank $ 0 $ leads
to an increase in the total deposits in the banking system.

Equation [(1.7)](#equation-sumdeposits) asserts that the **money multiplier** is
$ \frac{1}{r} $

- An initial deposit of cash of $ D_0 $ in bank $ 0 $ leads
  the banking system to create total deposits of $ \frac{D_0}{r} $.
- The initial deposit $ D_0 $ is held as reserves, distributed
  throughout the banking system according to $ D_0 = \sum_{i=0}^\infty R_i $.

## Example: The Keynesian Multiplier

The famous economist John Maynard Keynes and his followers created a
simple model intended to determine national income $ y $ in
circumstances in which

- there are substantial unemployed resources, in particular **excess
  supply** of labor and capital
- prices and interest rates fail to adjust to make aggregate **supply
  equal demand** (e.g., prices and interest rates are frozen)
- national income is entirely determined by aggregate demand

### Static Version

An elementary Keynesian model of national income determination consists
of three equations that describe aggregate demand for $ y $ and its
components.

The first equation is a national income identity asserting that
consumption $ c $ plus investment $ i $ equals national income
$ y $:

$$
c+ i = y
$$

The second equation is a Keynesian consumption function asserting that
people consume a fraction $ b \in (0,1) $ of their income:

$$
c = b y
$$

The fraction $ b \in (0,1) $ is called the **marginal propensity to
consume**.

The fraction $ 1-b \in (0,1) $ is called the **marginal propensity
to save**.

The third equation simply states that investment is exogenous at level
$ i $.

- *exogenous* means *determined outside this model*.

Substituting the second equation into the first gives $ (1-b) y = i $.

Solving this equation for $ y $ gives

$$
y = \frac{1}{1-b} i
$$

The quantity $ \frac{1}{1-b} $ is called the **investment
multiplier** or simply the **multiplier**.

Applying the formula for the sum of an infinite geometric series, we can
write the above equation as

$$
y = i \sum_{t=0}^\infty b^t
$$

where $ t $ is a nonnegative integer.

So we arrive at the following equivalent expressions for the multiplier:

$$
\frac{1}{1-b} =   \sum_{t=0}^\infty b^t
$$

The expression $ \sum_{t=0}^\infty b^t $ motivates an interpretation
of the multiplier as the outcome of a dynamic process that we describe
next.

### Dynamic Version

We arrive at a dynamic version by interpreting the nonnegative integer
$ t $ as indexing time and changing our specification of the
consumption function to take time into account

- we add a one-period lag in how income affects consumption

We let $ c_t $ be consumption at time $ t $ and $ i_t $ be
investment at time $ t $.

We modify our consumption function to assume the form

$$
c_t = b y_{t-1}
$$

so that $ b $ is the marginal propensity to consume (now) out of
last period’s income.

We begin with an initial condition stating that

$$
y_{-1} = 0
$$

We also assume that

$$
i_t = i \ \ \textrm {for all }  t \geq 0
$$

so that investment is constant over time.

It follows that

$$
y_0 = i + c_0 = i + b y_{-1} =  i
$$

and

$$
y_1 = c_1 + i = b y_0 + i = (1 + b) i
$$

and

$$
y_2 = c_2 + i = b y_1 + i = (1 + b + b^2) i
$$

and more generally

$$
y_t = b y_{t-1} + i = (1+ b + b^2 + \cdots + b^t) i
$$

or

$$
y_t = \frac{1-b^{t+1}}{1 -b } i
$$

Evidently, as $ t \rightarrow + \infty $,

$$
y_t \rightarrow \frac{1}{1-b} i
$$

**Remark 1:** The above formula is often applied to assert that an
exogenous increase in investment of $ \Delta i $ at time $ 0 $
ignites a dynamic process of increases in national income by successive amounts

$$
\Delta i, (1 + b )\Delta i, (1+b + b^2) \Delta i , \cdots
$$

at times $ 0, 1, 2, \ldots $.

**Remark 2** Let $ g_t $ be an exogenous sequence of government
expenditures.

If we generalize the model so that the national income identity
becomes

$$
c_t + i_t + g_t  = y_t
$$

then a version of the preceding argument shows that the **government
expenditures multiplier** is also $ \frac{1}{1-b} $, so that a
permanent increase in government expenditures ultimately leads to an
increase in national income equal to the multiplier times the increase
in government expenditures.

## Example: Interest Rates and Present Values

We can apply our formula for geometric series to study how interest
rates affect values of streams of dollar payments that extend over time.

We work in discrete time and assume that $ t = 0, 1, 2, \ldots $
indexes time.

We let $ r \in (0,1) $ be a one-period **net nominal interest rate**

- if the nominal interest rate is $ 5 $ percent,
  then $ r= .05 $

A one-period **gross nominal interest rate** $ R $ is defined as

$$
R = 1 + r \in (1, 2)
$$

- if $ r=.05 $, then $ R = 1.05 $

**Remark:** The gross nominal interest rate $ R $ is an **exchange
rate** or **relative price** of dollars at between times $ t $ and
$ t+1 $. The units of $ R $ are dollars at time $ t+1 $ per
dollar at time $ t $.

When people borrow and lend, they trade dollars now for dollars later or
dollars later for dollars now.

The price at which these exchanges occur is the gross nominal interest
rate.

- If I sell $ x $ dollars to you today, you pay me $ R x $
  dollars tomorrow.
- This means that you borrowed $ x $ dollars for me at a gross
  interest rate $ R $ and a net interest rate $ r $.

We assume that the net nominal interest rate $ r $ is fixed over
time, so that $ R $ is the gross nominal interest rate at times
$ t=0, 1, 2, \ldots $.

Two important geometric sequences are

$$
1, R, R^2, \cdots \tag{8}
$$

and

$$
1, R^{-1}, R^{-2}, \cdots \tag{9}
$$

Sequence [(1.8)](#equation-geom1) tells us how dollar values of an investment **accumulate**
through time.

Sequence [(1.9)](#equation-geom2) tells us how to **discount** future dollars to get their
values in terms of today’s dollars.

### Accumulation

Geometric sequence [(1.8)](#equation-geom1) tells us how one dollar invested and re-invested
in a project with gross one period nominal rate of return accumulates

- here we assume that net interest payments are reinvested in the
  project
- thus, $ 1 $ dollar invested at time $ 0 $ pays interest
  $ r $ dollars after one period, so we have $ r+1 = R $
  dollars at time$ 1 $
- at time $ 1 $ we reinvest $ 1+r =R $ dollars and receive interest
  of $ r R $ dollars at time $ 2 $ plus the *principal*
  $ R $ dollars, so we receive $ r R + R = (1+r)R = R^2 $
  dollars at the end of period $ 2 $
- and so on

Evidently, if we invest $ x $ dollars at time $ 0 $ and
reinvest the proceeds, then the sequence

$$
x , xR , x R^2, \cdots
$$

tells how our account accumulates at dates $ t=0, 1, 2, \ldots $.

### Discounting

Geometric sequence [(1.9)](#equation-geom2) tells us how much future dollars are worth in terms of today’s dollars.

Remember that the units of $ R $ are dollars at $ t+1 $ per
dollar at $ t $.

It follows that

- the units of $ R^{-1} $ are dollars at $ t $ per dollar at $ t+1 $
- the units of $ R^{-2} $ are dollars at $ t $ per dollar at $ t+2 $
- and so on; the units of $ R^{-j} $ are dollars at $ t $ per
  dollar at $ t+j $

So if someone has a claim on $ x $ dollars at time $ t+j $, it
is worth $ x R^{-j} $ dollars at time $ t $ (e.g., today).

### Application to Asset Pricing

A **lease** requires a payments stream of $ x_t $ dollars at
times $ t = 0, 1, 2, \ldots $ where

$$
x_t = G^t x_0
$$

where $ G = (1+g) $ and $ g \in (0,1) $.

Thus, lease payments increase at $ g $ percent per period.

For a reason soon to be revealed, we assume that $ G < R $.

The **present value** of the lease is

$$
\begin{aligned} p_0  & = x_0 + x_1/R + x_2/(R^2) + \ddots \\
                 & = x_0 (1 + G R^{-1} + G^2 R^{-2} + \cdots ) \\
                 & = x_0 \frac{1}{1 - G R^{-1}} \end{aligned}
$$

where the last line uses the formula for an infinite geometric series.

Recall that $ R = 1+r $ and $ G = 1+g $ and that $ R > G $
and $ r > g $ and that $ r $ and $ g $ are typically small
numbers, e.g., .05 or .03.

Use the Taylor series of $ \frac{1}{1+r} $ about $ r=0 $,
namely,

$$
\frac{1}{1+r} = 1 - r + r^2 - r^3 + \cdots
$$

and the fact that $ r $ is small to approximate
$ \frac{1}{1+r} \approx 1 - r $.

Use this approximation to write $ p_0 $ as

$$
\begin{aligned}
 p_0 &= x_0 \frac{1}{1 - G R^{-1}} \\
 &= x_0 \frac{1}{1 - (1+g) (1-r) } \\
 &= x_0 \frac{1}{1 - (1+g - r - rg)} \\
 & \approx x_0 \frac{1}{r -g }
\end{aligned}
$$

where the last step uses the approximation $ r g \approx 0 $.

The approximation

$$
p_0 = \frac{x_0 }{r -g }
$$

is known as the **Gordon formula** for the present value or current
price of an infinite payment stream $ x_0 G^t $ when the nominal
one-period interest rate is $ r $ and when $ r > g $.

We can also extend the asset pricing formula so that it applies to finite leases.

Let the payment stream on the lease now be $ x_t $ for $ t= 1,2, \dots,T $, where again

$$
x_t = G^t x_0
$$

The present value of this lease is:

$$
\begin{aligned} \begin{split}p_0&=x_0 + x_1/R  + \dots +x_T/R^T \\ &= x_0(1+GR^{-1}+\dots +G^{T}R^{-T}) \\ &= \frac{x_0(1-G^{T+1}R^{-(T+1)})}{1-GR^{-1}}  \end{split}\end{aligned}
$$

Applying the Taylor series to $ R^{-(T+1)} $ about $ r=0 $ we get:

$$
\frac{1}{(1+r)^{T+1}}= 1-r(T+1)+\frac{1}{2}r^2(T+1)(T+2)+\dots \approx 1-r(T+1)
$$

Similarly, applying the Taylor series to $ G^{T+1} $ about $ g=0 $:

$$
(1+g)^{T+1} = 1+(T+1)g(1+g)^T+(T+1)Tg^2(1+g)^{T-1}+\dots \approx 1+ (T+1)g
$$

Thus, we get the following approximation:

$$
p_0 =\frac{x_0(1-(1+(T+1)g)(1-r(T+1)))}{1-(1-r)(1+g) }
$$

Expanding:

$$
\begin{aligned} p_0 &=\frac{x_0(1-1+(T+1)^2 rg -r(T+1)+g(T+1))}{1-1+r-g+rg}  \\&=\frac{x_0(T+1)((T+1)rg+r-g)}{r-g+rg} \\ &\approx \frac{x_0(T+1)(r-g)}{r-g}+\frac{x_0rg(T+1)}{r-g}\\ &= x_0(T+1) + \frac{x_0rg(T+1)}{r-g}  \end{aligned}
$$

We could have also approximated by removing the second term
$ rgx_0(T+1) $ when $ T $ is relatively small compared to
$ 1/(rg) $ to get $ x_0(T+1) $ as in the finite stream
approximation.

We will plot the true finite stream present-value and the two
approximations, under different values of $ T $, and $ g $ and $ r $ in Python.

First we plot the true finite stream present-value after computing it
below

```python
# True present value of a finite lease
def finite_lease_pv_true(T, g, r, x_0):
    G = (1 + g)
    R = (1 + r)
    return (x_0 * (1 - G**(T + 1) * R**(-T - 1))) / (1 - G * R**(-1))
# First approximation for our finite lease

def finite_lease_pv_approx_1(T, g, r, x_0):
    p = x_0 * (T + 1) + x_0 * r * g * (T + 1) / (r - g)
    return p

# Second approximation for our finite lease
def finite_lease_pv_approx_2(T, g, r, x_0):
    return (x_0 * (T + 1))

# Infinite lease
def infinite_lease(g, r, x_0):
    G = (1 + g)
    R = (1 + r)
    return x_0 / (1 - G * R**(-1))
```

Now that we have defined our functions, we can plot some outcomes.

First we study the quality of our approximations

```python
def plot_function(axes, x_vals, func, args):
    axes.plot(x_vals, func(*args), label=func.__name__)

T_max = 50

T = np.arange(0, T_max+1)
g = 0.02
r = 0.03
x_0 = 1

our_args = (T, g, r, x_0)
funcs = [finite_lease_pv_true,
        finite_lease_pv_approx_1,
        finite_lease_pv_approx_2]
        ## the three functions we want to compare

fig, ax = plt.subplots()
ax.set_title('Finite Lease Present Value $T$ Periods Ahead')
for f in funcs:
    plot_function(ax, T, f, our_args)
ax.legend()
ax.set_xlabel('$T$ Periods Ahead')
ax.set_ylabel('Present Value, $p_0$')
plt.show()
```

Evidently our approximations perform well for small values of $ T $.

However, holding $ g $ and r fixed, our approximations deteriorate as $ T $ increases.

Next we compare the infinite and finite duration lease present values
over different lease lengths $ T $.

```python
# Convergence of infinite and finite
T_max = 1000
T = np.arange(0, T_max+1)
fig, ax = plt.subplots()
ax.set_title('Infinite and Finite Lease Present Value $T$ Periods Ahead')
f_1 = finite_lease_pv_true(T, g, r, x_0)
f_2 = np.ones(T_max+1)*infinite_lease(g, r, x_0)
ax.plot(T, f_1, label='T-period lease PV')
ax.plot(T, f_2, '--', label='Infinite lease PV')
ax.set_xlabel('$T$ Periods Ahead')
ax.set_ylabel('Present Value, $p_0$')
ax.legend()
plt.show()
```

The graph above shows how as duration $ T \rightarrow +\infty $,
the value of a lease of duration $ T $ approaches the value of a
perpetual lease.

Now we consider two different views of what happens as $ r $ and
$ g $ covary

```python
# First view
# Changing r and g
fig, ax = plt.subplots()
ax.set_title('Value of lease of length $T$')
ax.set_ylabel('Present Value, $p_0$')
ax.set_xlabel('$T$ periods ahead')
T_max = 10
T=np.arange(0, T_max+1)

rs, gs = (0.9, 0.5, 0.4001, 0.4), (0.4, 0.4, 0.4, 0.5),
comparisons = ('$\gg$', '$>$', r'$\approx$', '$<$')
for r, g, comp in zip(rs, gs, comparisons):
    ax.plot(finite_lease_pv_true(T, g, r, x_0), label=f'r(={r}) {comp} g(={g})')

ax.legend()
plt.show()
```

This graph gives a big hint for why the condition $ r > g $ is
necessary if a lease of length $ T = +\infty $ is to have finite
value.

For fans of 3-d graphs the same point comes through in the following
graph.

If you aren’t enamored of 3-d graphs, feel free to skip the next
visualization!

```python
# Second view
fig = plt.figure()
T = 3
ax = fig.gca(projection='3d')
r = np.arange(0.01, 0.99, 0.005)
g = np.arange(0.011, 0.991, 0.005)

rr, gg = np.meshgrid(r, g)
z = finite_lease_pv_true(T, gg, rr, x_0)

# Removes points where undefined
same = (rr == gg)
z[same] = np.nan
surf = ax.plot_surface(rr, gg, z, cmap=cm.coolwarm,
    antialiased=True, clim=(0, 15))
fig.colorbar(surf, shrink=0.5, aspect=5)
ax.set_xlabel('$r$')
ax.set_ylabel('$g$')
ax.set_zlabel('Present Value, $p_0$')
ax.view_init(20, 10)
ax.set_title('Three Period Lease PV with Varying $g$ and $r$')
plt.show()
```

We can use a little calculus to study how the present value $ p_0 $
of a lease varies with $ r $ and $ g $.

We will use a library called [SymPy](https://www.sympy.org/).

SymPy enables us to do symbolic math calculations including
computing derivatives of algebraic equations.

We will illustrate how it works by creating a symbolic expression that
represents our present value formula for an infinite lease.

After that, we’ll use SymPy to compute derivatives

```python
# Creates algebraic symbols that can be used in an algebraic expression
g, r, x0 = sym.symbols('g, r, x0')
G = (1 + g)
R = (1 + r)
p0 = x0 / (1 - G * R**(-1))
init_printing(use_latex='mathjax')
print('Our formula is:')
p0
```

    Our formula is:

$\displaystyle \frac{x_{0}}{- \frac{g + 1}{r + 1} + 1}$

```python
print('dp0 / dg is:')
dp_dg = sym.diff(p0, g)
dp_dg
```

    dp0 / dg is:

$\displaystyle \frac{x_{0}}{\left(r + 1\right) \left(- \frac{g + 1}{r + 1} + 1\right)^{2}}$

```python
print('dp0 / dr is:')
dp_dr = sym.diff(p0, r)
dp_dr
```

    dp0 / dr is:

$\displaystyle - \frac{x_{0} \left(g + 1\right)}{\left(r + 1\right)^{2} \left(- \frac{g + 1}{r + 1} + 1\right)^{2}}$

We can see that for $ \frac{\partial p_0}{\partial r}<0 $ as long as
$ r>g $, $ r>0 $ and $ g>0 $ and $ x_0 $ is positive,
so $ \frac{\partial p_0}{\partial r} $ will always be negative.

Similarly, $ \frac{\partial p_0}{\partial g}>0 $ as long as $ r>g $, $ r>0 $ and $ g>0 $ and $ x_0 $ is positive, so $ \frac{\partial p_0}{\partial g} $
will always be positive.

## Back to the Keynesian Multiplier

We will now go back to the case of the Keynesian multiplier and plot the
time path of $ y_t $, given that consumption is a constant fraction
of national income, and investment is fixed.

```python
# Function that calculates a path of y
def calculate_y(i, b, g, T, y_init):
    y = np.zeros(T+1)
    y[0] = i + b * y_init + g
    for t in range(1, T+1):
        y[t] = b * y[t-1] + i + g
    return y

# Initial values
i_0 = 0.3
g_0 = 0.3
# 2/3 of income goes towards consumption
b = 2/3
y_init = 0
T = 100

fig, ax = plt.subplots()
ax.set_title('Path of Aggregate Output Over Time')
ax.set_xlabel('$t$')
ax.set_ylabel('$y_t$')
ax.plot(np.arange(0, T+1), calculate_y(i_0, b, g_0, T, y_init))
# Output predicted by geometric series
ax.hlines(i_0 / (1 - b) + g_0 / (1 - b), xmin=-1, xmax=101, linestyles='--')
plt.show()
```

In this model, income grows over time, until it gradually converges to
the infinite geometric series sum of income.

We now examine what will
happen if we vary the so-called **marginal propensity to consume**,
i.e., the fraction of income that is consumed

```python
bs = (1/3, 2/3, 5/6, 0.9)

fig,ax = plt.subplots()
ax.set_title('Changing Consumption as a Fraction of Income')
ax.set_ylabel('$y_t$')
ax.set_xlabel('$t$')
x = np.arange(0, T+1)
for b in bs:
    y = calculate_y(i_0, b, g_0, T, y_init)
    ax.plot(x, y, label=r'$b=$'+f"{b:.2f}")
ax.legend()
plt.show()
```

Increasing the marginal propensity to consume $ b $ increases the
path of output over time.

Now we will compare the effects on output of increases in investment and government spending.

```python
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(6, 10))
fig.subplots_adjust(hspace=0.3)

x = np.arange(0, T+1)
values = [0.3, 0.4]

for i in values:
    y = calculate_y(i, b, g_0, T, y_init)
    ax1.plot(x, y, label=f"i={i}")
for g in values:
    y = calculate_y(i_0, b, g, T, y_init)
    ax2.plot(x, y, label=f"g={g}")

axes = ax1, ax2
param_labels = "Investment", "Government Spending"
for ax, param in zip(axes, param_labels):
    ax.set_title(f'An Increase in {param} on Output')
    ax.legend(loc ="lower right")
    ax.set_ylabel('$y_t$')
    ax.set_xlabel('$t$')
plt.show()
```

Notice here, whether government spending increases from 0.3 to 0.4 or
investment increases from 0.3 to 0.4, the shifts in the graphs are
identical.