# Practical Session 2: Classification algorithms

*Notebook by Ekaterina Kochmar*

## 0.1 Your task

In practical 1, you worked with the housing prices and bike sharing datasets on the tasks that required you to predict some value (e.g., price of a house) or amount (e.g., the count of rented bikes, or the number of registered users) based on a number of attributes – age of the house, number of rooms, income level of the house owners for the house price prediction (or weather conditions and time of the day for the prediction of the number of rented bikes). That is, you were predicting some continuous value.

This time, your task is to predict a particular category the instance belongs to based on its characteristics. This type of tasks is called *classification*.

## 0.2 Dataset

First you will look into the famous [*Iris dataset*](https://en.wikipedia.org/wiki/Iris_flower_data_set), which was first introduced by the British statistician and biologist Ronald Fisher in his 1936 paper *The use of multiple measurements in taxonomic problems*. The dataset contains $4$ characteristics (sepal length and width, and petal length and width) for $3$ related species of irises – *setosa*, *versicolor* and *virginica*. Your task is to learn to predict, based on these $4$ characteristics, the type of an iris.

For further reference, see the original paper: R. A. Fisher (1936). *The use of multiple measurements in taxonomic problems*. Annals of Eugenics. 7 (2): 179–188.

## 0.3 Learning objectives

In this practical you will learn about:
- binary and multiclass classification
- linearly separable data
- the use of a number of classifiers, including Naive Bayes, Logistic Regression, and Perceptron
- kernel trick
- ways to evaluate the performance of a classification model, including accuracy, precision, recall and F$_1$ measure
- precision-recall trade-off and the ways to measure it, including ROC curves and AUC

In addition, you will learn about the dataset uploading routines with `sklearn`.

## Step 1: Uploading and inspecting the data

As before, let's start by uploading and looking into the data. In the previous practical, you worked with the data collected independently and stored in a comma-separated file, as the data in a real-life data science project might be. In this practical, you will learn how to use `sklearn`'s data uploading routines.

`sklearn` has a number of datasets to practice your ML skills on, and the `iris` dataset is one of them. Here is how you can access the dataset through `sklearn`. Note that such data fields as *data* and *target* are already pre-defined for you:

```python
from sklearn import datasets
iris = datasets.load_iris()
list(iris.keys())
```

    ['data',
     'target',
     'frame',
     'target_names',
     'DESCR',
     'feature_names',
     'filename']

Take a look into what is contained in *data*. Remember that each instance of an iris is described in terms of $4$ variables – *sepal length*, *sepal width*, *petal length*, and *petal width*:

```python
iris.data
```

    array([[5.1, 3.5, 1.4, 0.2],
           [4.9, 3. , 1.4, 0.2],
           [4.7, 3.2, 1.3, 0.2],
           [4.6, 3.1, 1.5, 0.2],
           [5. , 3.6, 1.4, 0.2],
           [5.4, 3.9, 1.7, 0.4],
           [4.6, 3.4, 1.4, 0.3],
           [5. , 3.4, 1.5, 0.2],
           [4.4, 2.9, 1.4, 0.2],
           [4.9, 3.1, 1.5, 0.1],
           [5.4, 3.7, 1.5, 0.2],
           [4.8, 3.4, 1.6, 0.2],
           [4.8, 3. , 1.4, 0.1],
           [4.3, 3. , 1.1, 0.1],
           [5.8, 4. , 1.2, 0.2],
           [5.7, 4.4, 1.5, 0.4],
           [5.4, 3.9, 1.3, 0.4],
           [5.1, 3.5, 1.4, 0.3],
           [5.7, 3.8, 1.7, 0.3],
           [5.1, 3.8, 1.5, 0.3],
           [5.4, 3.4, 1.7, 0.2],
           [5.1, 3.7, 1.5, 0.4],
           [4.6, 3.6, 1. , 0.2],
           [5.1, 3.3, 1.7, 0.5],
           [4.8, 3.4, 1.9, 0.2],
           [5. , 3. , 1.6, 0.2],
           [5. , 3.4, 1.6, 0.4],
           [5.2, 3.5, 1.5, 0.2],
           [5.2, 3.4, 1.4, 0.2],
           [4.7, 3.2, 1.6, 0.2],
           [4.8, 3.1, 1.6, 0.2],
           [5.4, 3.4, 1.5, 0.4],
           [5.2, 4.1, 1.5, 0.1],
           [5.5, 4.2, 1.4, 0.2],
           [4.9, 3.1, 1.5, 0.2],
           [5. , 3.2, 1.2, 0.2],
           [5.5, 3.5, 1.3, 0.2],
           [4.9, 3.6, 1.4, 0.1],
           [4.4, 3. , 1.3, 0.2],
           [5.1, 3.4, 1.5, 0.2],
           [5. , 3.5, 1.3, 0.3],
           [4.5, 2.3, 1.3, 0.3],
           [4.4, 3.2, 1.3, 0.2],
           [5. , 3.5, 1.6, 0.6],
           [5.1, 3.8, 1.9, 0.4],
           [4.8, 3. , 1.4, 0.3],
           [5.1, 3.8, 1.6, 0.2],
           [4.6, 3.2, 1.4, 0.2],
           [5.3, 3.7, 1.5, 0.2],
           [5. , 3.3, 1.4, 0.2],
           [7. , 3.2, 4.7, 1.4],
           [6.4, 3.2, 4.5, 1.5],
           [6.9, 3.1, 4.9, 1.5],
           [5.5, 2.3, 4. , 1.3],
           [6.5, 2.8, 4.6, 1.5],
           [5.7, 2.8, 4.5, 1.3],
           [6.3, 3.3, 4.7, 1.6],
           [4.9, 2.4, 3.3, 1. ],
           [6.6, 2.9, 4.6, 1.3],
           [5.2, 2.7, 3.9, 1.4],
           [5. , 2. , 3.5, 1. ],
           [5.9, 3. , 4.2, 1.5],
           [6. , 2.2, 4. , 1. ],
           [6.1, 2.9, 4.7, 1.4],
           [5.6, 2.9, 3.6, 1.3],
           [6.7, 3.1, 4.4, 1.4],
           [5.6, 3. , 4.5, 1.5],
           [5.8, 2.7, 4.1, 1. ],
           [6.2, 2.2, 4.5, 1.5],
           [5.6, 2.5, 3.9, 1.1],
           [5.9, 3.2, 4.8, 1.8],
           [6.1, 2.8, 4. , 1.3],
           [6.3, 2.5, 4.9, 1.5],
           [6.1, 2.8, 4.7, 1.2],
           [6.4, 2.9, 4.3, 1.3],
           [6.6, 3. , 4.4, 1.4],
           [6.8, 2.8, 4.8, 1.4],
           [6.7, 3. , 5. , 1.7],
           [6. , 2.9, 4.5, 1.5],
           [5.7, 2.6, 3.5, 1. ],
           [5.5, 2.4, 3.8, 1.1],
           [5.5, 2.4, 3.7, 1. ],
           [5.8, 2.7, 3.9, 1.2],
           [6. , 2.7, 5.1, 1.6],
           [5.4, 3. , 4.5, 1.5],
           [6. , 3.4, 4.5, 1.6],
           [6.7, 3.1, 4.7, 1.5],
           [6.3, 2.3, 4.4, 1.3],
           [5.6, 3. , 4.1, 1.3],
           [5.5, 2.5, 4. , 1.3],
           [5.5, 2.6, 4.4, 1.2],
           [6.1, 3. , 4.6, 1.4],
           [5.8, 2.6, 4. , 1.2],
           [5. , 2.3, 3.3, 1. ],
           [5.6, 2.7, 4.2, 1.3],
           [5.7, 3. , 4.2, 1.2],
           [5.7, 2.9, 4.2, 1.3],
           [6.2, 2.9, 4.3, 1.3],
           [5.1, 2.5, 3. , 1.1],
           [5.7, 2.8, 4.1, 1.3],
           [6.3, 3.3, 6. , 2.5],
           [5.8, 2.7, 5.1, 1.9],
           [7.1, 3. , 5.9, 2.1],
           [6.3, 2.9, 5.6, 1.8],
           [6.5, 3. , 5.8, 2.2],
           [7.6, 3. , 6.6, 2.1],
           [4.9, 2.5, 4.5, 1.7],
           [7.3, 2.9, 6.3, 1.8],
           [6.7, 2.5, 5.8, 1.8],
           [7.2, 3.6, 6.1, 2.5],
           [6.5, 3.2, 5.1, 2. ],
           [6.4, 2.7, 5.3, 1.9],
           [6.8, 3. , 5.5, 2.1],
           [5.7, 2.5, 5. , 2. ],
           [5.8, 2.8, 5.1, 2.4],
           [6.4, 3.2, 5.3, 2.3],
           [6.5, 3. , 5.5, 1.8],
           [7.7, 3.8, 6.7, 2.2],
           [7.7, 2.6, 6.9, 2.3],
           [6. , 2.2, 5. , 1.5],
           [6.9, 3.2, 5.7, 2.3],
           [5.6, 2.8, 4.9, 2. ],
           [7.7, 2.8, 6.7, 2. ],
           [6.3, 2.7, 4.9, 1.8],
           [6.7, 3.3, 5.7, 2.1],
           [7.2, 3.2, 6. , 1.8],
           [6.2, 2.8, 4.8, 1.8],
           [6.1, 3. , 4.9, 1.8],
           [6.4, 2.8, 5.6, 2.1],
           [7.2, 3. , 5.8, 1.6],
           [7.4, 2.8, 6.1, 1.9],
           [7.9, 3.8, 6.4, 2. ],
           [6.4, 2.8, 5.6, 2.2],
           [6.3, 2.8, 5.1, 1.5],
           [6.1, 2.6, 5.6, 1.4],
           [7.7, 3. , 6.1, 2.3],
           [6.3, 3.4, 5.6, 2.4],
           [6.4, 3.1, 5.5, 1.8],
           [6. , 3. , 4.8, 1.8],
           [6.9, 3.1, 5.4, 2.1],
           [6.7, 3.1, 5.6, 2.4],
           [6.9, 3.1, 5.1, 2.3],
           [5.8, 2.7, 5.1, 1.9],
           [6.8, 3.2, 5.9, 2.3],
           [6.7, 3.3, 5.7, 2.5],
           [6.7, 3. , 5.2, 2.3],
           [6.3, 2.5, 5. , 1.9],
           [6.5, 3. , 5.2, 2. ],
           [6.2, 3.4, 5.4, 2.3],
           [5.9, 3. , 5.1, 1.8]])

To find out what variables are contained in `data`, check the `feature_names` data field.

```python
iris.feature_names
```

    ['sepal length (cm)',
     'sepal width (cm)',
     'petal length (cm)',
     'petal width (cm)']

What about the target values?

```python
iris.target
```

    array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
           2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
           2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

There are $3$ classes of irises – *setosa*, *versicolor*, and *virginica*, and they are already converted into numerical values for you (recall, that when the dataset is not already preprocessed this way, and the target or any of the arrtibutes are represented as text or categorical data, you need to convert them into numerical data). If you want to check what each numerical label corresponds to in the original data, you can do so accessing the `target_names` data field:

```python
iris.target_names
```

    array(['setosa', 'versicolor', 'virginica'], dtype='<U10')

Remember, that for further ML experiments, we need to have two data structures: the instance-by-attributes matrix $X$ and the target labels vector $y$. For instance, in the previous practical the regression algorithm learned the vector of weights $w$ to predict the target variable $\hat y^{(i)}$ for each instance $i$ so that its prediction would be maximally close to the actual label $y^{(i)}$. Since the labels $y^{(i)}$ were continuous (i.e., amount, number, or value), you measured the performance of your regressor by the distance between the predictions $\hat y$ and actual labels $y$. In this practical, you will need to work with $X$ and $y$, too, but the vector of $y$ this time will contain discrete values – classes $[0, 1, 2]$ for the different types of the flower.

As you might have already figured out, you need to initialise $X$ and $y$ with the `data` and `target` fields of your iris dataset:

```python
X, y = iris["data"], iris["target"]
print(X.shape)
print(y.shape)
```

    (150, 4)
    (150,)

Let's look closer into the data to get a feel for what is contained in it. As before, let's use visualisations with `matplotlib`, and in particular, plot one attribute against another for the three types of irises using scatterplot. E.g., let's plot sepal length vs. sepal width:

```python
%matplotlib inline
#so that the plot will be displayed in the notebook
import numpy as np
np.random.seed(42)

import matplotlib
from matplotlib import pyplot as plt

# visualise sepal length vs. sepal width
X = iris.data[:, :2]
y = iris.target

scatter_x = np.array(X[:, 0])
scatter_y = np.array(X[:, 1])
group = np.array(y)
cmap = matplotlib.cm.get_cmap('jet')
cdict = {0: cmap(0.1), 1: cmap(0.5), 2: cmap(0.9)}
labels = iris.target_names

fig, ax = plt.subplots(figsize=(8, 6))
for g in np.unique(group):
    ix = np.where(group == g)
    ax.scatter(scatter_x[ix], scatter_y[ix], c=np.array([cdict[g]]), #c = cdict[g],
               label = labels[g], s = 100, marker = "H",
               linewidth=2, alpha = 0.5)
ax.legend()
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')

plt.show()
```

It looks like *setosa* is quite clearly distiguishable from the other two types of irises with these two features. What about petal length and width?

```python
# visualise petal length vs. petal width
X = iris.data[:, 2:]
y = iris.target

scatter_x = np.array(X[:, 0])
scatter_y = np.array(X[:, 1])
group = np.array(y)
cmap = matplotlib.cm.get_cmap('jet')
cdict = {0: cmap(0.1), 1: cmap(0.5), 2: cmap(0.9)}
labels = iris.target_names

fig, ax = plt.subplots(figsize=(8, 6))
for g in np.unique(group):
    ix = np.where(group == g)
    ax.scatter(scatter_x[ix], scatter_y[ix], c=np.array([cdict[g]]), #c = cdict[g],
               label = labels[g], s = 100, marker = "H",
               linewidth=2, alpha = 0.5)
ax.legend()
plt.xlabel('Petal length')
plt.ylabel('Petal width')

plt.show()
```

This plot shows an even clearer separation between the class of *setosa* irises and the other two classes. In fact, with respect to these two attributes, it might be possible to clearly separate not only *setosas* from the other two classes, but also, with certain success, *versicolors* from *virginicas*.

**When the data can be separated by a straight line (or a single decision surface) as in the example above, it is called *linearly separable*.** This property of the data is successfully exploited by ML models that try to learn a linear separation boundary between the classes. In fact, there is a whole set of lines that you can use to separate the class of *setosas* from the other two classes, *versicolors* and *virginicas*, in this example. Some linear models explicitly allow you to select the *best* separation boundary by maximising the distance between the boundary and the closest instances of the two classes. Such are, for example, [Support Vector Machines](http://scikit-learn.org/stable/modules/svm.html), which will be covered in the [Part II Machine Learning and Bayesian Inference course](https://www.cl.cam.ac.uk/teaching/2021/MLBayInfer/):

```python
# visualise petal length vs. petal width
X = iris.data[:, 2:]
y = iris.target

scatter_x = np.array(X[:, 0])
scatter_y = np.array(X[:, 1])
group = np.array(y)
cmap = matplotlib.cm.get_cmap('jet')
cdict = {0: cmap(0.1), 1: cmap(0.5), 2: cmap(0.9)}
labels = iris.target_names

fig, ax = plt.subplots(figsize=(8, 6))
for g in np.unique(group):
    ix = np.where(group == g)
    ax.scatter(scatter_x[ix], scatter_y[ix], c=np.array([cdict[g]]), #c = cdict[g],
               label = labels[g], s = 100, marker = "H",
               linewidth=2, alpha = 0.5)
ax.legend()
plt.xlabel('Petal length')
plt.ylabel('Petal width')

for i in range(0, 9):
    plt.plot([3 + 0.5*i, 0], [0, 3-0.25*i], 'k-', color=cmap(0.1*(i+1)))

plt.show()
```

For the sake of consistency, let's plot all the pairs of features against each other. Note that all plots confirm that *setosas* are linearly separable from the other two types of irises (you might notice an occasional outlier, though), while *versicolors* are linearly separable from *virginicas* with respect to some attributes only:

```python
fig = plt.figure(figsize=(16, 12))
fig.subplots_adjust(hspace=0.2, wspace=0.2)

X = iris.data
y = iris.target
labels = iris.target_names
index = 1

for i in range(0, X.shape[1]):
    for j in range(0, X.shape[1]):
        scatter_x = np.array(X[:, i])
        scatter_y = np.array(X[:, j])
        group = np.array(y)
        cmap = matplotlib.cm.get_cmap('jet')
        cdict = {0: cmap(0.1), 1: cmap(0.5), 2: cmap(0.9)}

        ax = fig.add_subplot(X.shape[1], X.shape[1], index)
        index+=1

        for g in np.unique(group):
            ix = np.where(group == g)
            ax.scatter(scatter_x[ix], scatter_y[ix], c=np.array([cdict[g]]), #c = cdict[g],
                       label = labels[g], s = 50, marker = "H",
                       linewidth=1, alpha = 0.5)

plt.show()
```

## Step 2: Splitting the data into training and test subsets

Before applying the classifiers, let's split the dataset into the training and test sets. Recall, that when building an ML model, all further data exploration, feature selection and scaling, model selection and fine-tuning should be done on the training data, and the test data should only be used at the final step to evaluate the best estimated model:

```python
from sklearn.model_selection import train_test_split

train_set, test_set = train_test_split(X, test_size=0.2)
print(len(train_set), "training instances +", len(test_set), "test instances")
```

    120 training instances + 30 test instances

As before, you want your training and test data to contain enough representative examples of each class, that is, you should rather apply `StratifiedShuffleSplit` and not random splitting:

```python
from sklearn.model_selection import StratifiedShuffleSplit

split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
split.get_n_splits(X, y)
print(split)

for train_index, test_index in split.split(X, y):
    print("TRAIN:", len(train_index), "TEST:", len(test_index))
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]

print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)
```

    StratifiedShuffleSplit(n_splits=1, random_state=42, test_size=0.2,
                train_size=None)
    TRAIN: 120 TEST: 30
    (120, 4) (120,) (30, 4) (30,)

Let's check the class proportions in the original dataset, and training and test subsets:

```python
import pandas as pd

def original_proportions(data):
    props = {}
    for value in set(data["target"]):
        data_value = [i for i in data["target"] if i==value]
        props[value] = len(data_value) / len(data["target"])
    return props

def subset_proportions(subset):
    props = {}
    for value in set(subset):
        data_value = [i for i in subset if i==value]
        props[value] = len(data_value) / len(subset)
    return props

compare_props = pd.DataFrame({
    "Overall": subset_proportions(iris["target"]),
    "Stratified tr": subset_proportions(y_train),
    "Stratified ts": subset_proportions(y_test),
})
compare_props["Strat. tr %error"] = 100 * compare_props["Stratified tr"] / compare_props["Overall"] - 100
compare_props["Strat. ts %error"] = 100 * compare_props["Stratified ts"] / compare_props["Overall"] - 100

compare_props.sort_index()
```

      Overall
      Stratified tr
      Stratified ts
      Strat. tr %error
      Strat. ts %error

      0
      0.333333
      0.333333
      0.333333
      -1.421085e-14
      -1.421085e-14

      1
      0.333333
      0.333333
      0.333333
      -1.421085e-14
      -1.421085e-14

      2
      0.333333
      0.333333
      0.333333
      -1.421085e-14
      -1.421085e-14

The original dataset is well-balanced – it contains exactly $50$ examples for each class. With the stratified data splits, you get equal proportions of each type of the irises in the training and test sets, too.

Now, let's first approach the classification task in a simpler setting: let's start with *binary classification* and try to predict whether an iris is of a particular type: e.g., *setosa* vs. *not-a-setosa*, or *versicolor* vs. *not-a-versicolor*.

## Case 1: Binary classification

Let's start by separating the data that describes *setosa* from other data.

```python
y_train_setosa = (y_train == 0) # will return True when the label is 0 (i.e., setosa)
y_test_setosa = (y_test == 0)
y_test_setosa
```

    array([ True, False, False, False,  True, False,  True,  True, False,
           False, False, False, False, False,  True,  True,  True, False,
           False, False,  True, False, False, False, False, False, False,
            True, False,  True])

`y_test_setosa` returns a boolean vector of $30$ test instances: it contains `True` for the test instances that are *setosas*, and `False` otherwise. Let's pick one example of a *setosa* – for instance, the first one from the test set, `X_test[0]`, for further evaluations.

```python
setosa_example = X_test[0]
```

As you've noticed above, *setosas* are linearly separable from the other two classes, so it would be reasonable to apply a linear model to the task of separating *setosas* from *not-setosas*.

### Perceptron

A (single-layer) perceptron is a simple linear classifier that tries to learn the set of weights $w$ for the input vectors $X$ in order to predict the output binary class values $y$. In particular:

\begin{equation}
  \hat y^{(i)}=\begin{cases}
    1, & \text{if $w \cdot x^{(i)} + b > 0$}\\
    0, & \text{otherwise}
  \end{cases}
\end{equation}

where $w \cdot x^{(i)}$ is the dot product of weight vector $w$ and the feature vector $x^{(i)}$ for the instance $i$, $\sum_{j=1}^{m} w_{j}x_{j}^{(i)}$, and $b$ is the bias term.

`sklearn` has a perceptron implementation through its linear [`SGDClassifier`](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html#sklearn.linear_model.SGDClassifier) with the following parameter settings:

```python
from sklearn.linear_model import SGDClassifier

sgd = SGDClassifier(max_iter=5, tol=None, random_state=42,
                   loss="perceptron", eta0=1, learning_rate="constant", penalty=None)
sgd.fit(X_train, y_train_setosa)
sgd.predict([setosa_example])
```

    array([ True])

The perceptron correctly predicts that `X_test[0]` is a *setosa* flower. However, as you've seen above, not all types of irises are linearly separable. Let's select a more challenging example of a *versicolor* (class $1$) for comparison.

```python
y_train_versicolor = (y_train == 1) # True when the label is 1 (i.e., versicolor)
y_test_versicolor = (y_test == 1)
y_test_versicolor
```

    array([False, False,  True,  True, False,  True, False, False, False,
            True, False, False, False,  True, False, False, False,  True,
            True, False, False, False,  True, False, False,  True,  True,
           False, False, False])

Select one of the examples to try your classifier on (array indexing in python starts with $0$, so you can pick indexes $2$, $3$, $5$ and so on):

```python
versicolor_example = X_test[17]
print("Class", y_test[17], "(", iris.target_names[y_test[17]], ")")

sgd.fit(X_train, y_train_versicolor)
print(sgd.predict([versicolor_example]))
```

    Class 1 ( versicolor )
    [False]

Looks like perceptron indeed cannot predict the class of the *versicolor* example correctly. Let's see if another linear classifier – `Logistic Regression` – can do a better job.

### Logistic Regression

You used `Linear Regression` model in practical 1 to predict continuous values. In contrast, `Logistic Regression` is used for binary classification, that is, to predict a discrete value of $0$ or $1$. In particular, it estimates the probability that an instance belongs to a particular class (e.g., that $X\_test[0] \in {setosa}$). If the probability is greater than $50\%$, the instance is classified as *setosa* (positive class, labelled $1$ or $True$). Otherwise, it is classified as *not-a-setosa* (negative class, labelled $0$ or $False$).

Similarly to `Linear Regression`, `Logistic Regression` computes a weighted sum using the input features $w \cdot X$ plus an intercept ($w_0$), but instead of outputting the result as `Linear Regression` does, it further applies a *sigmoid function* to this result:

\begin{equation}
\hat p = \sigma (w \cdot X)
\end{equation}

The sigmoid function outputs a value between $0$ and $1$:

\begin{equation}
\sigma (t) = \frac{1}{1 + exp(-t)}
\end{equation}

Once the Logistic Regression model has estimated the probability $\hat p$, the label $\hat y$ is predicted as follows:

\begin{equation}
  \hat y=\begin{cases}
    1, & \text{if $\hat p \geq 0.5$}\\
    0, & \text{otherwise}
  \end{cases}
\end{equation}

Note that the above is equivalent to:

\begin{equation}
  \hat y=\begin{cases}
    1, & \text{if $t \geq 0$}\\
    0, & \text{otherwise}
  \end{cases}
\end{equation}

Let's apply a `Logistic Regression` model to our setosa example:

```python
from sklearn.linear_model import LogisticRegression

log_reg = LogisticRegression()
log_reg.fit(X_train, y_train_setosa)
```

    LogisticRegression()

```python
log_reg.predict([setosa_example])
```

    array([ True])

The model correctly predicts that the test example is indeed a setosa. What about the versicolor example?

```python
log_reg.fit(X_train, y_train_versicolor)
log_reg.predict([versicolor_example])
```

    array([ True])

Finally, for comparison, let's introduce one more classifier, that some of you might have come across before.

### Naive Bayes

If you did [Part IA Machine Learning and Real-World Data](https://www.cl.cam.ac.uk/teaching/1920/MLRD/) in the past, you might recall that you have already come across classification tasks: for example, you were asked to build a classifier that identifies sentiment in text, and you used `Naive Bayes` for that. `Naive Bayes` makes different assumptions about the data. In particular, it doesn't assume linear separability but makes the predictions based on the prior and the updated belief about the data.

To remind you, on a two class problem (e.g., distinguishing between classes $0$ for *not-setosas* and $1$ for *setosas*), a Naive Bayes model will predict:

\begin{equation}
  \hat y^{(i)} = argmax_{c \in (0, 1)} p(y=c | x^{(i)}) = \begin{cases}
    1, & \text{if $\hat p(y=1 | x^{(i)}) > \hat p(y=0 | x^{(i)}$})\\
    0, & \text{otherwise}
  \end{cases}
\end{equation}

where the probabilities are conditioned on the feature vector for $x^{(i)}$, i.e. $(f^{(i)}_{1}, ..., f^{(i)}_{n})$. In practice, it is impossible to estimate these probabilities exactly, and one uses the Naive Bayes theorem so that:

\begin{equation}
\hat p(y=c | x^{(i)}) = \frac{p(c) p(x^{(i)} | c)}{p(x^{(i)})}
\end{equation}

where $c \in \{0, 1\}$ is the class to be predicted. Since the denominator is the same for both estimates of $\hat p(y=1 | x^{(i)})$ and $\hat p(y=0 | x^{(i)})$, it can be omitted. Therefore, the estimate can be simplified to:

\begin{equation}
\hat y^{(i)} = argmax_{c \in (0, 1)} p(y=c | x^{(i)}) = argmax_{c \in (0, 1)} p(c) p(x^{(i)} | c)
\end{equation}

where $p(c)$ is the *prior belief* of the classifier about the distribution of the classes in the data, and $p(x^{(i)} | c)$ is the *posterior probability*. Both can be estimated from the training data using *maximum a posteriori (MAP)* estimation. Moreover, the "naive" independence assumption of this learning algorithm allows you to estimate $p(x^{(i)} | c)$ as a product of feature probabilities taken independently of each other, i.e.:

\begin{equation}
p(x^{(i)} | c) = p(f^{(i)}_{1}, ..., f^{(i)}_{n} | c) \approx p(f^{(i)}_{1} | c) \times ... \times p(f^{(i)}_{n} | c)
\end{equation}

`sklearn` has a number of implementations of a [`Naive Bayes`](http://scikit-learn.org/stable/modules/naive_bayes.html) algorithm which mainly differ from each other with respect to how they estimate the conditional probability above using the data and what believes they hold about the distribution of this data, e.g.:

```python
from sklearn.naive_bayes import GaussianNB, MultinomialNB

gnb = MultinomialNB() # or:
gnb = GaussianNB()
gnb.fit(X_train, y_train_setosa)
gnb.predict([setosa_example])
```

    array([ True])

```python
gnb.fit(X_train, y_train_versicolor)

gnb.predict([versicolor_example])
```

    array([ True])

As you can see, not all classifiers perform equally well. How do you measure their performance in a more comprehensive way?

## Step 3: Evaluation

### Performance measures

The most straightforward way to evaluate a classifier is to estimate how often its predictions are correct. This estimate is called *accuracy* and it is calculated as follows:

\begin{equation}
ACC = \frac{num(\hat y == y)}{num(\hat y == y) + num(\hat y != y)} = \frac{num(\hat y == 1 \& y == 1) + num(\hat y == 0 \& y == 0)}{num(\hat y == 1 \& y == 1) + num(\hat y == 0 \& y == 0) + num(\hat y == 1 \& y == 0) + num(\hat y == 0 \& y == 1)}
\end{equation}

E.g., for the *setosa* classification example, an accuracy of the classifier is the ratio of correctly identified *setosas* and correctly identified *not-setosas* to the total number of examples.

You can either import accuracy metric using `from sklearn.metrics import accuracy_score` or measure accuracy across multiple cross-validation folds (refer to practical 1, if you need to remind yourself what cross-validation does):

```python
from sklearn.model_selection import cross_val_score

print(cross_val_score(log_reg, X_train, y_train_setosa, cv=5, scoring="accuracy"))
print(cross_val_score(gnb, X_train, y_train_setosa, cv=5, scoring="accuracy"))
print(cross_val_score(sgd, X_train, y_train_setosa, cv=5, scoring="accuracy"))
```

    [1. 1. 1. 1. 1.]
    [1. 1. 1. 1. 1.]
    [1. 1. 1. 1. 1.]

All three classifiers are precfectly accurate in their prediction on the *setosa* example. What about the *versicolor* example?

```python
print(cross_val_score(log_reg, X_train, y_train_versicolor, cv=5, scoring="accuracy"))
print(cross_val_score(gnb, X_train, y_train_versicolor, cv=5, scoring="accuracy"))
print(cross_val_score(sgd, X_train, y_train_versicolor, cv=5, scoring="accuracy"))
```

    [0.75       0.70833333 0.66666667 0.75       0.70833333]
    [0.875      0.95833333 1.         0.875      0.91666667]
    [0.33333333 0.66666667 0.66666667 0.66666667 0.66666667]

The numbers differ but they still do not tell you much about the performance of a classifier in general. E.g., an accuracy of $\approx 0.71$ is far from perfect, but exactly how bad or how acceptable is it?

Let's implement a brute-force algorithm that will simply predict *not-versicolor* for every instance in the *versicolor* detection case (or *not-setosa* in the *setosa* detection case). Here is how well it will perform:

```python
from sklearn.base import BaseEstimator

class NotXClassifier(BaseEstimator):
    def fit(self, X, y=None):
        pass
    def predict(self, X):
        return np.zeros((len(X), 1), dtype=bool)

notversicolor_clf = NotXClassifier()
cross_val_score(notversicolor_clf, X_train, y_train_versicolor, cv=5, scoring="accuracy")
```

    array([0.66666667, 0.70833333, 0.66666667, 0.70833333, 0.58333333])

This gives you a very clear benchmark for comparison. The above represents a *majority class baseline*: for each split in cross-validation splits, it measures the proportion of the majority class (*not-versicolor* in this case). That is, if the classifier does nothing and simply returns the majority class label every time, this is how "well" it will perform. Obviously, you want the actual classifier that you build to do better than that.

### Confusion matrix

So now you can compare the accuracy (e.g., the proportion of correctly identified *setosas* and *not-setosas* in the dataset) to the baseline system. However, this doesn't help you understand *where* your classifier goes wrong. For example, does the low accuracy of the classifiers in the *versicolor* identification case suggest that they miss some *versicolors* and classify them as other types of irises, or does it suggest that they mistake other types of irises for *versicolors*? Or, perhaps, it's a combination of two types of mistakes?

The accuracy score itself doesn't allow you to make any of these conclusions. What you need to do is to look into the number of correctly and incorrectly classified instances, and the data representation that helps you do that is called *confusion matrix*. A confusion matrix is simply a table that compares the number of actual instances of type $c$ to the number of predicted instances of type $c$, i.e.:

|                  | predicted $\hat c=0$| predicted $\hat c=1$ |
|  -------------   |   :-------------:   |    :-------------:   |
| **actual $c=0$** | TN                  |   FP                 |
| **actual $c=1$** | FN                  |   TP                 |

The instances that are classified correctly as well as those that are misclassified have certain importance for evaluation of your classification algorithms. Here is the terminology:

- `TP` stands for *true positives*. These are the actual instances of class $1$ that are correctly classified as class $1$ (*setosas* identified as *setosas*);
- `TN` stands for *true negatives*. These are the actual instances of class $0$ that are correctly classified as class $0$ (*not-setosas* identified as *not-setosas*);
- `FN` stands for *false negatives*. These are the actual instances of class $1$ that are incorrectly classified as class $0$ (*setosas* identified as *not-setosas*, i.e. missed by the classifier);
- and finally, `FP` are *false positives*. These are the actual instances of class $0$ that are incorrectly classified as class $1$ (*not-setosas* identified as *setosas*, i.e. other types of flower mistaken for setosas by the classifier).

Now you can re-interpret the accuracy score as:

\begin{equation}
ACC = \frac{TP + TN}{TP + TN + FP + FN}
\end{equation}

So in order to maximise accuracy, you'll need to maximise on the number of true positives and true negatives.

The confusion matrix also tells you what exactly is misclassified by the classifier. For example, let's look into the confusion matrices for *setosa* identification:

```python
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import confusion_matrix

y_train_pred = cross_val_predict(log_reg, X_train, y_train_setosa, cv=5)
confusion_matrix(y_train_setosa, y_train_pred)
```

    array([[80,  0],
           [ 0, 40]], dtype=int64)

```python
y_train_pred = cross_val_predict(gnb, X_train, y_train_setosa, cv=5)
confusion_matrix(y_train_setosa, y_train_pred)
```

    array([[80,  0],
           [ 0, 40]], dtype=int64)

You know that the accuracy of these classifiers on *setosa* identification equals to $1$, so all *setosas* ($40$ instances) and *not-setosas* ($80$ instances) are classified correctly, and the confusion matrices show exactly that. What about the mistakes that the classifiers make on the *versicolor* example?

```python
y_train_pred = cross_val_predict(log_reg, X_train, y_train_versicolor, cv=5)
confusion_matrix(y_train_versicolor, y_train_pred)
```

    array([[71,  9],
           [25, 15]], dtype=int64)

```python
y_train_pred = cross_val_predict(gnb, X_train, y_train_versicolor, cv=5)
confusion_matrix(y_train_versicolor, y_train_pred)
```

    array([[75,  5],
           [ 4, 36]], dtype=int64)

The matrices above show that the `Logistic Regression` classifier correctly identifies $71$ *not-versicolors* (`TN`) and $15$ *versicolors* (`TP`), and it also misses $25$ *versicolors* (`FN`) and mistakes $9$ other flowers for *versicolors*. Now you can see that the bigger source of error for the `Logistic Regression` classifier is that it's not good enough in identifying *versicolors* as it misses more of them that it identifies ($25$ vs $15$). In contrast, `Naive Bayes` identifies more *versicolors* in general ($36 + 5$) and more of them correctly ($36$), while it misses only $4$ *versicolor* instances.

Apart from making such observations, you might want to be able to make sense of the number of correctly and incorrectly classified examples and compare the performance of the different classifiers in an aggregated way. The following set of measures will help you do so: *precision* ($P$), *recall* ($R$), and $F_{\beta}$*-measure* (e.g., $F_1$).

- *Precision* measures how reliable or trustworthy your classifier is. It tells you how often when the classifier predicts that a flower is a *versicolor* (class $1$) it actually is a *versicolor*. It relies on the number of $TP$'s and $FP$'s:
\begin{equation}
P = \frac{TP}{TP+FP}
\end{equation}

- *Recall* measures the coverage of your classifier. It tells you how many of the actual instances of *versicolor* your classifier can detect at all. It relies on the number of $TP$'s and $FN$'s:
\begin{equation}
R = \frac{TP}{TP+FN}
\end{equation}

- Finally, $F_1$*-score* combines the two measures above to give you an overall idea of your classifier's performance. $F_1$*-score* is estimated as follows:
\begin{equation}
F_1 = 2 \times \frac{P \times R}{P+R}
\end{equation}

It is the *harmonic mean* of the two measures. However, if you want to highlight the importance of precision in your task (and we'll talk about when you might want to do so later in this practical) or the importance of recall, you can do so by changing the $\beta$ coefficient of the *F-score*. In fact, the $F_1$*-score* is the most commonly used case of a more general $F_{\beta}$*-measure*, which is estimated as follows:
\begin{equation}
F_{\beta} = (1 + \beta^2) \times \frac{P \times R}{\beta^2 \times P+R}
\end{equation}

$\beta < 1$ puts more emphasis on precision than on recall, and $\beta > 1$ weighs recall higher than precision.

```python
from sklearn.metrics import precision_score, recall_score, f1_score

y_train_pred = cross_val_predict(gnb, X_train, y_train_versicolor, cv=5)
precision = precision_score(y_train_versicolor, y_train_pred) # == 36 / (36 + 5)
recall = recall_score(y_train_versicolor, y_train_pred) # == 36 / (36 + 4)
f1 = f1_score(y_train_versicolor, y_train_pred)
print(precision, recall, f1)

y_train_pred = cross_val_predict(log_reg, X_train, y_train_versicolor, cv=5)
precision = precision_score(y_train_versicolor, y_train_pred) # == 15 / (15 + 9)
recall = recall_score(y_train_versicolor, y_train_pred) # == 15 / (15 + 25)
f1 = f1_score(y_train_versicolor, y_train_pred)
print(precision, recall, f1)
```

    0.8780487804878049 0.9 0.888888888888889
    0.625 0.375 0.46875

The numbers above tell you that the `Naive Bayes` classifier has pretty good recall ($0.9$, or it identifies $90\%$ of *versicolor* instances) as well as precision ($\approx 0.88$, or in approximately $88\%$ of the cases you can trust its *versicolor* prediction). The two numbers are close to each other in range, so the $F_{1}$ score is high, too. The `Logistic Regression` classifier is less precise (you can trust it about $62.5\%$ of the time) and it misses many *versicolor* examples (identifying only $37.5\%$ of those).

Obviously, if your algorithm gets it all right, you will end up with perfect accuracy, precision and recall. In practice, however, the algorithms often don't get it all perfectly correct, and depending on the task at hand, you might decide that you are actually mostly interested in high precision of your algorithm, or in high recall.

For example, imagine you are working on a machine learning classifier that detects cancerous cases in the data coming from patients' analyses. The task of your algorithm is to detect whether the patient needs further tests and closer analysis based on the preliminary tests. The cost of *false negatives* (missed cancerous cases) in this task is very high and it's better to administer further tests for a patient about which the algorithm has doubts than to ignore the case altogether. In this case, you should prioritise recall over precision.

On the other hand, imagine you work for a pharmaceutical company trying to detect whether a particular drug will be applicable to a particular condition or a particular group of patients using data science and machine learning. Or, you want to learn in a data-driven way [when to change the drug dosage for patients in a hospital](http://www.cs.cornell.edu/people/tj/publications/morik_etal_99a.pdf). In these cases, the cost of *false positives*, i.e. deciding that the drug is suitable for a particular patient when it is not, or deciding to intervene in patients' treatment when the dosage should be kept as before, is more costly. These are the cases when you should prioritise precision over recall.

### Precision-recall trade-off

Often, you can achieve perfect precision if you lower the recall, and other way around. For example, by always predicting *versicolor* you will reach perfect recall (all *versicolor* instances will be covered by such prediction), but very low precision (because in most of the cases such prediction will be unreliable as your classifier will predict *versicolor* for all setosas and all virginicas as well). On the other hand, by trying to maximise precision, you will often need to constrain your classifier so that it returns less false positives and therefore is more conservative.

A good way to understand how a classifier makes its predictions is to look into its `decision_function` – the confidence score of the classifier's predictions. For `Logistic Regression` it returns a signed distance to the separation hyperplane selected by the model. Let's check the confidence score of the `Logistic Regression` model on the *versicolor* example:

```python
log_reg.fit(X_train, y_train_versicolor)

y_scores = log_reg.decision_function([versicolor_example])
y_scores
```

    array([0.51527727])

When this confidence score is higher than the predefined model's threshold, the model assigns the instance to the positive class, and otherwise it assigns the instance to the negative class. The `Logistic Regression`'s default threshold is $0$, so the example above is classified as *versicolor*:

```python
threshold = 0
y_versicolor_pred = (y_scores > threshold)
y_versicolor_pred
```

    array([ True])

However, what would happen if you change the threshold to, for example, $0.7$? As you expect, this particular instance will be classified as *not-a-versicolor*. What effect will it have on the precision and recall of your classifier as a consequence?

```python
threshold = 0.7
y_versicolor_pred = (y_scores > threshold)
y_versicolor_pred
```

    array([False])

The following will return the confidence scores for each of the training set instances:

```python
y_scores = cross_val_predict(log_reg, X_train, y_train_versicolor, cv=5, method="decision_function")
y_scores
```

    array([-6.87063685e-01,  3.66847885e-01, -4.03787021e-02, -9.65747614e-01,
            3.36786297e-01,  1.40865169e-01,  1.10007621e-01,  1.88980949e-01,
           -7.63784085e-01, -1.59596588e+00, -1.07020726e-01, -3.67974371e-01,
            1.34388703e+00,  3.59991938e-01,  2.09532653e-01, -1.14131953e+00,
           -2.50492853e+00,  7.73381874e-01, -4.39212328e-01, -1.99957095e+00,
            5.67154901e-01, -2.30329749e+00, -1.62365949e+00, -8.63889607e-01,
            5.93952591e-01, -1.18477717e-01, -1.11462881e-01, -2.02148096e+00,
           -4.51606178e-01, -2.32088275e+00, -2.69914465e+00, -4.84783847e-01,
           -5.30495759e-01, -4.22683730e+00, -1.19176293e+00, -1.76041842e+00,
           -1.72375525e-03, -1.87426494e-01, -1.74140496e+00, -1.15501264e+00,
           -1.92389291e+00, -1.13290142e+00, -3.32787156e+00, -3.28823258e-02,
           -1.14825799e+00, -2.27905845e+00, -6.29092642e-02, -1.27116762e+00,
            5.19367774e-01, -6.57491762e-01, -8.51079964e-01, -8.08943262e-01,
           -1.53482930e+00, -3.73688660e-02, -1.77871956e+00, -6.50609208e-01,
           -2.31174827e-01, -1.82907392e+00, -1.12323814e+00, -1.42948390e-01,
           -1.87819755e+00,  7.32050757e-01,  1.19028174e-01, -1.30095947e+00,
            1.21920665e-01,  4.30952523e-01,  9.58727271e-02, -6.12665974e-01,
           -2.84558571e-01, -1.49592538e+00, -1.53303531e+00, -1.77415284e+00,
           -2.18869247e+00,  1.23749403e+00, -4.68075651e-02, -2.22874435e-01,
            6.23532545e-01, -1.31572262e+00, -2.41919592e-01, -1.56651067e+00,
           -1.11597938e+00, -8.58489866e-01,  8.34639431e-01, -2.32223993e+00,
           -1.41552794e+00, -3.03410667e+00, -2.23650600e+00, -1.34849671e-02,
           -9.22891227e-01, -1.01852499e+00, -4.35738423e-01, -1.36844084e+00,
           -2.49397355e+00, -9.12546644e-01, -1.26233035e+00,  4.04232274e-01,
           -8.45995578e-01,  7.75748184e-01, -4.57950899e-01, -2.99883607e+00,
           -1.16691876e+00, -3.64358148e-01, -7.04677590e-01, -6.62591065e-01,
           -2.51723514e+00, -5.32129504e-01,  1.10365347e-01, -4.96667761e-01,
           -6.36832978e-01, -1.03424298e+00, -2.10496992e+00, -2.84415990e-01,
            1.02859743e+00, -8.55483071e-01, -1.73572784e+00, -2.17986898e+00,
           -1.52309396e+00, -3.60064685e+00, -8.57916124e-01, -1.17007870e+00])

Let's use `sklearn`'s [`precision_recall_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html) functionality and plot the precision and recall values against the different threshold values. This shows you how the measures will be affected by changing the confidence threshold:

```python
from sklearn.metrics import precision_recall_curve

precisions, recalls, thresholds = precision_recall_curve(y_train_versicolor, y_scores)

def plot_pr_vs_threshold(precisions, recalls, thresholds):
    plt.plot(thresholds, precisions[:-1], "b--", label="Precision")
    plt.plot(thresholds, recalls[:-1], "g--", label="Recall")
    plt.xlabel("Threshold")
    plt.legend(loc="upper right")
    plt.ylim([0, 1])

plot_pr_vs_threshold(precisions, recalls, thresholds)
plt.show()
```

As you can see, the recall has a general tendency of decreasing when you increase the threshold – that is, the more conservative the classifier becomes, the more instances it is likely to miss. At the same time, precision curve is bumpier: with certain changes in the threshold, it might drop as well as the classifier misidentifies some other types of irises as *versicolor*.

You can also plot precision values against recall values and track the changes in precision against recall:

```python
def plot_precision_vs_recall(precisions, recalls):
    plt.plot(recalls, precisions, "b-", linewidth=2)
    plt.xlabel("Recall")
    plt.ylabel("Precision")
    plt.axis([0, 1, 0, 1])

plot_precision_vs_recall(precisions, recalls)
plt.show()
```

### The Receiver Operating Characteristic (ROC)

Another widely used way to inspect and present the results is to plot the receiver operating characteristic (ROC) curve. Similarly to the precision / recall curve above, it shows how the performance of the classifier changes at different threshold values. In particular, it plots the *true positive rate (TPR)* against the *false positive rate (FPR)*. Here is some more machine learning terminology:

- *True positive rate (TPR)* is simply another term for recall. Alternatively, it is also called *sensitivity*, or *probability of detection*;
- *False positive rate (FPR)* is also referred to as *fall-out* or *probability of false alarm*, and it can be calculated as $(1 − specificity)$. *Specificity*, in its turn, is estimated as:

\begin{equation}
specificity = \frac{TN}{TN + FP}
\end{equation}

In other words, *specificity* expresses the probability that the classifier correctly identifies *non-versicolors* in the *versicolor* identification example. *FPR*, therefore, shows how many incorrect *versicolor* votes (false alarms) the classifier makes for all *non-versicolor* examples while testing.

The ROC curve shows how the sensitivity of the classifier increases as a function of (i.e., at the expense of) the fall-out. A perfect classifier would have $100\%$ sensitivity (no false negatives) and $100\%$ specificity (no false positives), which in the ROC space can be illustrated by the point in the upper left corner with the coordinate $(0,1)$. The closer the actual ROC curve gets to this point, the better is the classifier. A random guess would give a point along a diagonal line from the left bottom to the top right corners. Points above the diagonal represent good classification results (better than random); points below the line represent bad results (worse than random).

To plot the ROC curve, let's first estimate the *FPR* and *TPR* for different threshold values using `sklearn`'s `roc_curve` functionality, and then plot *FPR* and *TPR* using `matplotlib`:

```python
from sklearn.metrics import roc_curve

fpr, tpr, thresholds = roc_curve(y_train_versicolor, y_scores)

def plot_roc_curve(fpr, tpr, label=None):
    plt.plot(fpr, tpr, linewidth=2, label=label)
    plt.plot([0, 1], [0, 1], "k--")
    plt.axis([0, 1, 0, 1.01])
    plt.xlabel("False positive rate (fpr)")
    plt.ylabel("True positive rate (tpr)")

plot_roc_curve(fpr, tpr)
plt.show()
```

Another characteristic of the ROC curve is the *area under the curve (AUC)*, which for a perfect classifier will equal $1$ and for a random one will equal $0.5$. A estimate on a actual classifier, thus, usually lies between these two values:

```python
from sklearn.metrics import roc_auc_score
roc_auc_score(y_train_versicolor, y_scores)
```

    0.8109375000000001

Now, the `Logistic Regression` classifier does not perform very well on the *versicolor* identification task, and the best classifier among the three so far is `Naive Bayes`. Let's compare its performance to the `Logistic Regression` and visualise the difference using the ROC curve. Note that `Naive Bayes` classifier implementation in `sklearn` doesn't use `decision_function`; [`predict_proba`](https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.GaussianNB.html#sklearn.naive_bayes.GaussianNB.predict_proba) is the equivalent of that for `Naive Bayes`:

```python
y_probas_gnb = cross_val_predict(gnb, X_train, y_train_versicolor, cv=3, method="predict_proba")
y_scores_gnb = y_probas_gnb[:, 1] # score = proba of the positive class
fpr_gnb, tpr_gnb, thresholds_gnb = roc_curve(y_train_versicolor, y_scores_gnb)

plt.plot(fpr, tpr, "b:", label="Logistic Regression")
plot_roc_curve(fpr_gnb, tpr_gnb, "Gaussian Naive Bayes")
plt.legend(loc="lower right")
plt.show()
```

The curve for `Naive Bayes` is much closer to the upper left corner, which shows that the classifier is better than `Logistic Regression` on this task. Let's estimate the AUC as well:

```python
roc_auc_score(y_train_versicolor, y_scores_gnb)
```

    0.9828125000000001

## Step 4: Data transformations

So far, you have trained three classifiers on two classification tasks – identification of *setosas* and *versicolors*. You've seen that all three classifiers perform well on the *setosas* identification example as the data is linearly separable, and only `Naive Bayes` performs well in the *versicolors* case, because the data with the given features is not linearly separable.

For example, consider a decision boundary that a `Logistic Regression` classifier would learn for the linearly separable class of *setosas* (based on the first two features):

```python
X = iris.data[:, :2] # consider the first two features for plotting (as in Step 1)
y = iris.target

scatter_x = np.array(X[:, 0])
scatter_y = np.array(X[:, 1])
group = np.array(y)
cmap = matplotlib.cm.get_cmap('jet')
cdict = {0: cmap(0.1), 1: cmap(0.5), 2: cmap(0.9)}
labels = iris.target_names

fig, ax = plt.subplots(figsize=(8, 6))
for g in np.unique(group):
    ix = np.where(group == g)
    ax.scatter(scatter_x[ix], scatter_y[ix], c=np.array([cdict[g]]), #c = cdict[g],
               label = labels[g], s = 100, marker = "H",
               linewidth=2, alpha = 0.5)

log_reg.fit(X_train[:, :2], y_train_setosa) # train the classifier for setosas, using the first two features only
w = log_reg.coef_[0]
i = log_reg.intercept_[0]
xx = np.linspace(4, 8) # generate some values for feature1 (sepal length) in the appropriate range of values
yy = -(w[0]*xx + i)/w[1] # estimate the value for feature2 (sepal width) based on the learned weights and intercept

plt.plot(xx, yy, 'b--')

ax.legend()
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')

plt.show()
```

and compare it to the case of non-linearly separable class of *versicolors*:

```python
fig, ax = plt.subplots(figsize=(8, 6))
for g in np.unique(group):
    ix = np.where(group == g)
    ax.scatter(scatter_x[ix], scatter_y[ix], c=np.array([cdict[g]]), #c = cdict[g],
               label = labels[g], s = 100, marker = "H",
               linewidth=2, alpha = 0.5)

log_reg.fit(X_train[:, :2], y_train_versicolor)
w = log_reg.coef_[0]
i = log_reg.intercept_[0]
xx = np.linspace(4, 8)
yy = -(w[0]*xx + i)/w[1]

plt.plot(xx, yy, 'g--')

ax.legend()
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')

plt.show()
```

The original representation of the data that you've been using so far is simply not expressive enough. But what if you could transform this data in such a way that it could be linearly separated? In fact, there is a "trick" for this problem that involves transforming the original data using a number of new dimensions in such a way that in this new high-dimensional space the classes become linearly separable. This "trick" is commonly known as the *kernel trick* or *kernel method*.

### Kernel trick  and approximate kernel maps

You will most commonly hear about this method in the context of Support Vector Machines (SVMs), for example in the [Part II Machine Learning and Bayesian Inference course](https://www.cl.cam.ac.uk/teaching/2021/MLBayInfer/). However, some other linear classifiers, including [perceptron](https://en.wikipedia.org/wiki/Kernel_perceptron), allow for the kernel methods to be used as well.

The general motivation behind the kernel trick is as follows: when the data is not linearly separable, i.e. there is no clear dividing boundary between the two classes, the kernel trick allows you to transform the data using a number of additional dimensions that would allow for such clear dividing boundary to be learned. Kernel methods require a user-specified kernel, i.e. a function that will transform the original data into a higher dimensional space. Polynomial and Gaussian (also known as *radial-basis function*) transformations are among the most widely used kernel functions. These data transformations might remind you of the polynomial feature transformation used with the linear regression in practical 1: recall that you cast the original features from the space where the relation between the features and the output cannot exactly be captured by a linear function, into a higher dimensional feature space using  polynomial function. Recall also that this allowed you to apply a linear function to this new feature space leading to an improved result.

You will learn more about kernel functions and their application to SVMs in more detail in the Part II Machine Learning and Bayesian Inference course. For the task at hand, you will use an `sklearn`'s [approximate kernel map](http://scikit-learn.org/stable/modules/kernel_approximation.html) in combination with the perceptron implementation of the `SGDClassifier`:

```python
from sklearn.kernel_approximation import RBFSampler

rbf_features = RBFSampler(gamma=1, random_state=42)
X_train_features = rbf_features.fit_transform(X_train)
print(X_train.shape, "->", X_train_features.shape)
sgd_rbf = SGDClassifier(max_iter=100, random_state=42, loss="perceptron",
                        eta0=1, learning_rate="constant", penalty=None)
sgd_rbf.fit(X_train_features, y_train_versicolor)

sgd_rbf.score(X_train_features, y_train_versicolor)
```

The output above shows that the classifier tries to learn the separation boundary in a highly-dimensional transformed feature space, and the accuracy of this learning on the training set is over $0.99$. Let's test this classifier in a 5-fold cross-validation manner, and compare precision, recall and F$_1$ scores to the linear classifier trained on the original data:

```python
y_train_pred = cross_val_predict(sgd, X_train, y_train_versicolor, cv=5)
precision = precision_score(y_train_versicolor, y_train_pred)
recall = recall_score(y_train_versicolor, y_train_pred)
f1 = f1_score(y_train_versicolor, y_train_pred)
print(precision, recall, f1)

y_train_pred = cross_val_predict(sgd_rbf, X_train_features, y_train_versicolor, cv=5)
precision = precision_score(y_train_versicolor, y_train_pred)
recall = recall_score(y_train_versicolor, y_train_pred)
f1 = f1_score(y_train_versicolor, y_train_pred)
print(precision, recall, f1)
```

    0.4 0.4 0.4000000000000001
    0.9230769230769231 0.9 0.9113924050632911

Looks like the kernel trick helped improve the results on the originally non-linearly separable data significantly!

## Case 2:  Multi-class classification

Now remember that your actual goal is to build a three-way classifier that can predict *setosa*, *versicolor* and *virginica* classes, and not just tell whether an instance is an $X$ (*setosa* or *versicolor*) or not. Actually, you are already half way there, and here is why.

Some classifiers are capable of handling multiple classes directly. For example, `Naive Bayes` learns about the probabilities of the classes in the data irrespective of the number of classes. Therefore, the binary examples above can naturally be extended to the 3-class classification scenario: you simply provide the classifier with the data on the $3$ rather than $2$ classes.

In contrast, such classifiers as perceptron (`SDGClassifier`) and `Logistic Regression`, which seek to learn a linear separation boundary, are inherently binary classifiers: they try to learn a single separation boundary between two classes at a time. However, they can also very easily be extended to handle more than $2$ classes. Multi-class classification with such linear classifiers generally follows one of the two routes:

- with the *one-vs-all* (*OvA*, or *one-vs-rest*, *OvR*) strategy you train $n$ classifiers (e.g., a setosa detector, a versicolor detector and a virginica detector). For a new instance, you apply all of the classifiers and predict the class that gets the highest score returned by the classifiers;
- with the *one-vs-one* (*OvO*) strategy, you train a binary classifier for each pair of classes in your data and select the class that wins most of the duels.

There are pros and cons to each of these approaches. E.g., with the *OvO* strategy, you end up training $n \times (n-1) / 2$ classifiers. I.e. for the iris dataset you will have $3$ classifiers (exactly as with the *OvA* strategy) but on a $10$-class problem this will amount to $45$ classifiers. On the other hand, the training sets with the *OvO* will be much smaller and more balanced. Some classifiers scale poorly with the size of the training set when it is imbalanced, so *OvO* for them is preferable (e.g., such are SVMs), but most of the linear classifiers use *OvA* instead.

The nice thing about `sklearn` is that it implements the above strategies under the hood, so to perform a multi-class classification with the `SDGClassifier`, all you need to do is to provide it with the data and labels on $3$ classes, and it will train $3$ binary *OvA* classifiers and output the class with the highest score, i.e.:

```python
sgd.fit(X_train, y_train) # i.e., all instances, not just one class
print(sgd.predict([setosa_example]))
print(sgd.predict([versicolor_example]))
```

    [0]
    [1]

Recall that the *versicolor* class label is $1$, so the classifier's output is correct this time. Let's also check the result with the RBF kernel:

```python
sgd_rbf.fit(X_train_features, y_train) # i.e., all instances, not just one class
X_test_features = rbf_features.transform(X_test)
setosa_rbf_example = X_test_features[0] # note that you need to transform the test data in the same way, too
versicolor_rbf_example = X_test_features[17]

print(sgd_rbf.predict([setosa_rbf_example]))
print(sgd_rbf.predict([versicolor_rbf_example]))
```

    [0]
    [1]

This classifier classified both test examples correctly. Let's see what logic the classifier followed:

```python
setosa_scores = sgd_rbf.decision_function([setosa_rbf_example])
print(setosa_scores)

# check which class gets the maximum score
prediction = np.argmax(setosa_scores)
print(prediction)
# check which class this corresponds to in the classifier
print(sgd_rbf.classes_[prediction])
print(iris.target_names[sgd_rbf.classes_[prediction]])
```

    [[ 0.45557008 -0.99104538 -1.37659362]]
    0
    0
    setosa

This shows that *setosa* class got a much higher score than the other two. What about the versicolor example?

```python
versicolor_scores = sgd_rbf.decision_function([versicolor_rbf_example])
print(versicolor_scores)
prediction = np.argmax(versicolor_scores)
print(prediction)
print(iris.target_names[sgd_rbf.classes_[prediction]])
```

    [[-1.50838342  2.76324654 -1.76976171]]
    1
    versicolor

For comparison, let's see what the original `SGDClassifier` (without the RBF kernel) predicted:

```python
versicolor_scores = sgd.decision_function([versicolor_example])
print(versicolor_scores)
prediction = np.argmax(versicolor_scores)
print(prediction)
print(iris.target_names[sgd.classes_[prediction]])
```

    [[-14.59   2.38 -21.34]]
    1
    versicolor

Now, if you'd like to use the *OvO* strategy, you can enforce the `sklearn` to do so creating an instance of a `OneVsOneClassifier` (similarly, `OneVsRestClassifier` for *OvA*). Note that you're essentially using the same classifier as before, and it is just the framework within which the predictions are made:

```python
from sklearn.multiclass import OneVsOneClassifier

ovo_clf = OneVsOneClassifier(SGDClassifier(max_iter=100, random_state=42, loss="perceptron",
                                           eta0=1, learning_rate="constant", penalty=None))
ovo_clf.fit(X_train_features, y_train)
ovo_clf.predict([versicolor_rbf_example])
```

    array([1])

```python
len(ovo_clf.estimators_)
```

    3

Now let's look into the `NaiveBayes` performance on the $3$-class problem:

```python
gnb.fit(X_train, y_train)
gnb.predict([versicolor_example])
```

    array([1])

It correctly classifies the *versicolor* example, so let's check how confident it is about this prediction (remember that you should use `predict_proba` with `NaiveBayes` and `decision_function` with the `SGDClassifier`):

```python
gnb.predict_proba([versicolor_example])
```

    array([[8.75885105e-64, 9.99994982e-01, 5.01792481e-06]])

Let's look into the cross-validated performance of the classifiers:

```python
print(cross_val_score(sgd_rbf, X_train_features, y_train, cv=5, scoring="accuracy"))
print(cross_val_score(ovo_clf, X_train_features, y_train, cv=5, scoring="accuracy"))
print(cross_val_score(gnb, X_train, y_train, cv=5, scoring="accuracy"))
```

    [0.91666667 0.95833333 0.95833333 0.95833333 0.95833333]
    [0.875      0.95833333 0.95833333 0.91666667 0.95833333]
    [0.91666667 0.95833333 1.         0.95833333 0.95833333]

Finally, recall that in practical 1 you used further transformations on the data, e.g. scaling. Let's apply scaling and compare the results (note that you can use `np.mean()` to get the average accuracy score across all splits if you wish to get a single aggregated accuracy score instead of $5$ separate ones):

```python
from sklearn.preprocessing import StandardScaler, MinMaxScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train.astype(np.float64))
X_train_features_scaled = scaler.fit_transform(X_train_features.astype(np.float64))

print(cross_val_score(sgd_rbf, X_train_features_scaled, y_train, cv=5, scoring="accuracy"))
print(cross_val_score(ovo_clf, X_train_features_scaled, y_train, cv=5, scoring="accuracy"))
print(cross_val_score(gnb, X_train_scaled, y_train, cv=5, scoring="accuracy"))
```

    [0.91666667 1.         0.95833333 0.91666667 0.95833333]
    [0.91666667 0.95833333 0.95833333 0.95833333 0.95833333]
    [0.91666667 0.95833333 1.         0.95833333 0.95833333]

## Step 5: Error analysis

Before applying the classifiers to the test data, let's gain a bit more insight into what the classifiers get wrong. Recall, that earlier you used confusion matrices to learn about the classification errors:

```python
y_train_pred = cross_val_predict(sgd_rbf, X_train_features_scaled, y_train, cv=3)
conf_mx = confusion_matrix(y_train, y_train_pred)
conf_mx
```

    array([[40,  0,  0],
           [ 0, 37,  3],
           [ 0,  4, 36]], dtype=int64)

Let's visualise the classifier decisions:

```python
plt.imshow(conf_mx, cmap = "jet")
plt.show()
```

And if you'd like to highlight only the most salient errors, you can do so as follows (the `"jet"` color scheme uses red spectrum for higher numbers):

```python
row_sums = conf_mx.sum(axis=1, keepdims=True)
norm_conf_mx = conf_mx / row_sums
np.fill_diagonal(norm_conf_mx, 0)
plt.imshow(norm_conf_mx, cmap = "jet")
plt.show()
```

## Final step – evaluating on the test set

The `SGDClassifier` with the RBF kernel:

```python
from sklearn.metrics import accuracy_score

X_test_features_scaled = scaler.transform(X_test_features.astype(np.float64))
y_pred = sgd_rbf.predict(X_test_features_scaled)
accuracy_score(y_test, y_pred)
```

    0.9

```python
precision = precision_score(y_test, y_pred, average='weighted')
recall = recall_score(y_test, y_pred, average='weighted')
f1 = f1_score(y_test, y_pred, average='weighted')
print(precision, recall, f1)
```

    0.9023569023569024 0.9 0.8997493734335841

The *OvO* SGD classifier:

```python
from sklearn.metrics import accuracy_score

X_test_features_scaled = scaler.transform(X_test_features.astype(np.float64))
y_pred = ovo_clf.predict(X_test_features_scaled)
accuracy_score(y_test, y_pred)
```

    0.9666666666666667

```python
precision = precision_score(y_test, y_pred, average='weighted')
recall = recall_score(y_test, y_pred, average='weighted')
f1 = f1_score(y_test, y_pred, average='weighted')
print(precision, recall, f1)
```

    0.9696969696969696 0.9666666666666667 0.9665831244778613

The `NaiveBayes` classifier:

```python
gnb.fit(X_train, y_train)
y_pred = gnb.predict(X_test)
accuracy_score(y_test, y_pred)
```

    0.9666666666666667

```python
precision = precision_score(y_test, y_pred, average='weighted')
recall = recall_score(y_test, y_pred, average='weighted')
f1 = f1_score(y_test, y_pred, average='weighted')
print(precision, recall, f1)
```

    0.9696969696969696 0.9666666666666667 0.9665831244778613

#  Assignment: Handwritten digits dataset

The dataset that you will use in this assignment is the [*digits* dataset](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html) which contains $1797$ images of $10$ hand-written digits. The digits have been preprocessed so that $32 \times 32$ bitmaps are divided into non-overlapping blocks of $4 \times 4$ and the number of on pixels are counted in each block. This generates an input matrix of $8 \times 8$ where each element is an integer in the range of $[0, ..., 16]$. This reduces dimensionality and gives invariance to small distortions.

For further information on NIST preprocessing routines applied to this data, see M. D. Garris, J. L. Blue, G. T. Candela, D. L. Dimmick, J. Geist, P. J. Grother, S. A. Janet, and C. L. Wilson, *NIST Form-Based Handprint Recognition System*, NISTIR 5469, 1994.

As before, use the `sklearn`'s data uploading routines to load the dataset and get the data fields:

## Step 1: Inspecting Data

```python
from sklearn import datasets
%matplotlib inline
import numpy as np
np.random.seed(42)

import matplotlib
from matplotlib import pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.model_selection import StratifiedShuffleSplit
import pandas as pd
from sklearn.linear_model import SGDClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB, MultinomialNB
from sklearn.model_selection import cross_val_score
from sklearn.base import BaseEstimator
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import confusion_matrix
from sklearn.metrics import precision_score, recall_score, f1_score, accuracy_score
from sklearn.kernel_approximation import RBFSampler
from sklearn.multiclass import OneVsOneClassifier
from sklearn.preprocessing import StandardScaler, MinMaxScaler

digits = datasets.load_digits()
list(digits.keys())
```

    ['data', 'target', 'frame', 'feature_names', 'target_names', 'images', 'DESCR']

```python
digits.feature_names
```

    ['pixel_0_0',
     'pixel_0_1',
     'pixel_0_2',
     'pixel_0_3',
     'pixel_0_4',
     'pixel_0_5',
     'pixel_0_6',
     'pixel_0_7',
     'pixel_1_0',
     'pixel_1_1',
     'pixel_1_2',
     'pixel_1_3',
     'pixel_1_4',
     'pixel_1_5',
     'pixel_1_6',
     'pixel_1_7',
     'pixel_2_0',
     'pixel_2_1',
     'pixel_2_2',
     'pixel_2_3',
     'pixel_2_4',
     'pixel_2_5',
     'pixel_2_6',
     'pixel_2_7',
     'pixel_3_0',
     'pixel_3_1',
     'pixel_3_2',
     'pixel_3_3',
     'pixel_3_4',
     'pixel_3_5',
     'pixel_3_6',
     'pixel_3_7',
     'pixel_4_0',
     'pixel_4_1',
     'pixel_4_2',
     'pixel_4_3',
     'pixel_4_4',
     'pixel_4_5',
     'pixel_4_6',
     'pixel_4_7',
     'pixel_5_0',
     'pixel_5_1',
     'pixel_5_2',
     'pixel_5_3',
     'pixel_5_4',
     'pixel_5_5',
     'pixel_5_6',
     'pixel_5_7',
     'pixel_6_0',
     'pixel_6_1',
     'pixel_6_2',
     'pixel_6_3',
     'pixel_6_4',
     'pixel_6_5',
     'pixel_6_6',
     'pixel_6_7',
     'pixel_7_0',
     'pixel_7_1',
     'pixel_7_2',
     'pixel_7_3',
     'pixel_7_4',
     'pixel_7_5',
     'pixel_7_6',
     'pixel_7_7']

```python
X, y = digits["data"], digits["target"]
X.shape
```

    (1797, 64)

```python
y.shape
```

    (1797,)

You can access the digits and visualise them using the following code (feel free to select another digit):

```python
some_digit = X[3]
some_digit_image = some_digit.reshape(8, 8)

plt.imshow(some_digit_image, cmap=matplotlib.cm.binary, interpolation="nearest")
plt.axis("off")
plt.show()
```

```python
y[3]
```

    3

Looking at the data, I feel like it makes little sense to try to visualise the individual features and see any visual patterns between them. there are too many features to do this manually.

## Step 2: Subset Generation
Use stratified shuffle split to ensure distributions between train, test, and overall sets match

```python
split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
split.get_n_splits(X, y)
print(split)

for train_index, test_index in split.split(X, y):
    print("TRAIN:", len(train_index), "TEST:", len(test_index))
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]

print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)
```

    StratifiedShuffleSplit(n_splits=1, random_state=42, test_size=0.2,
                train_size=None)
    TRAIN: 1437 TEST: 360
    (1437, 64) (1437,) (360, 64) (360,)

Checking the proportions of the split subsets. The percentage errors seem reasonable.

```python
def subset_proportions(subset):
    props = {}
    for value in set(subset):
        data_value = [i for i in subset if i==value]
        props[value] = len(data_value) / len(subset)
    return props

compare_props = pd.DataFrame({
    "Overall": subset_proportions(digits["target"]),
    "Stratified tr": subset_proportions(y_train),
    "Stratified ts": subset_proportions(y_test),
})
compare_props["Strat. tr %error"] = 100 * compare_props["Stratified tr"] / compare_props["Overall"] - 100
compare_props["Strat. ts %error"] = 100 * compare_props["Stratified ts"] / compare_props["Overall"] - 100

compare_props.sort_index()
```

      Overall
      Stratified tr
      Stratified ts
      Strat. tr %error
      Strat. ts %error

      0
      0.099054
      0.098817
      0.100000
      -0.239263
      0.955056

      1
      0.101280
      0.101601
      0.100000
      0.316594
      -1.263736

      2
      0.098497
      0.098817
      0.097222
      0.324357
      -1.294727

      3
      0.101836
      0.101601
      0.102778
      -0.231584
      0.924408

      4
      0.100723
      0.100905
      0.100000
      0.179933
      -0.718232

      5
      0.101280
      0.100905
      0.102778
      -0.370506
      1.478938

      6
      0.100723
      0.100905
      0.100000
      0.179933
      -0.718232

      7
      0.099610
      0.099513
      0.100000
      -0.097969
      0.391061

      8
      0.096828
      0.096729
      0.097222
      -0.101984
      0.407088

      9
      0.100167
      0.100209
      0.100000
      0.041754
      -0.166667

## Step 3: Evaluation

The digit problem lends itself nicely to multi-class classification. We will be using four different models - `SDGClassifier`, `Logistic Regression`, and `Naive Bayes` (Multinomial and Gaussian) - and comparing results between them.

```python
sgd = SGDClassifier(max_iter=10000, random_state=42, loss="perceptron",
                        eta0=1, learning_rate="constant", penalty=None)
log_reg = LogisticRegression(max_iter=10000)
mnb = MultinomialNB()
gnb = GaussianNB()

ovo_sgd = OneVsOneClassifier(SGDClassifier(max_iter=10000, random_state=42, loss="perceptron",
                        eta0=1, learning_rate="constant", penalty=None))
ovo_log_reg = OneVsOneClassifier(LogisticRegression(max_iter=10000))
ovo_mnb = OneVsOneClassifier(MultinomialNB())
ovo_gnb = OneVsOneClassifier(GaussianNB())
```

```python
models = [("SGD\t", sgd), ("Log Reg\t", log_reg), ("Multi NB", mnb), ("Gaus NB\t", gnb), ("ovo_SGD\t", ovo_sgd), ("ovo_Log Reg", ovo_log_reg), ("ovo_Mult NB", ovo_mnb), ("ovo_Gaus NB", ovo_gnb)]
```

Note that micro-averaging is used owning to the explanation presented [here](https://datascience.stackexchange.com/a/24051).

```python
print("name\t accuracy\t precision\t recall\t f1")
for name, model in models:
    y_train_pred = cross_val_predict(model, X_train, y_train, cv=5)
    acc = np.round(accuracy_score(y_train, y_train_pred), 2)
    pre = np.round(precision_score(y_train, y_train_pred, average='micro'), 2)
    rec = np.round(recall_score(y_train, y_train_pred, average='micro'), 2)
    f1 = np.round(f1_score(y_train, y_train_pred, average='micro'), 2)
    print(f"{name}\t{acc}\t{pre}\t{rec}\t{f1}")
```

    name	 accuracy	 precision	 recall	 f1
    SGD		0.95	0.95	0.95	0.95
    Log Reg		0.97	0.97	0.97	0.97
    Multi NB	0.9	0.9	0.9	0.9
    Gaus NB		0.83	0.83	0.83	0.83
    ovo_SGD		0.96	0.96	0.96	0.96
    ovo_Log Reg	0.98	0.98	0.98	0.98
    ovo_Mult NB	0.9	0.9	0.9	0.9
    ovo_Gaus NB	0.83	0.83	0.83	0.83

## Step 4: Data Transformations

```python
from sklearn_extra.kernel_approximation import Fastfood
from sklearn.kernel_approximation import RBFSampler
from sklearn.preprocessing import StandardScaler, MinMaxScaler
from sklearn.pipeline import Pipeline
from sklearn.base import BaseEstimator, TransformerMixin
import math

# generate more non-overlapping averaged blocks, 3 layers deep (so 2x2 and 4x4 and 8x8)
class KernelConvolver(BaseEstimator, TransformerMixin):
    def __init__(self):
        pass
    def fit(self, X, y=None):
        return self
    def convolve(self, X, kernels, kernel_size, side_len):
        for i in range(side_len // kernel_size):
            for j in range(side_len // kernel_size):
                summ = None
                for ii in range(kernel_size):
                    for jj in range(kernel_size):
                        if summ is None:
                            summ = np.array(X[:, (i * kernel_size + ii) * side_len + j * kernel_size + jj], copy=True)
                        else:
                            summ += X[:, (i * kernel_size + ii) * side_len + j * kernel_size + jj]
                summ = np.array(summ) / (kernel_size ** 2)
                kernels = summ if kernels is None else np.c_[kernels, summ]
        return kernels
    def transform(self, X, y=None):
        kernels = None
        side_len = int(math.sqrt(X.shape[1]))

        kernels = self.convolve(X, kernels, 2, side_len)
        kernels = self.convolve(X, kernels, 4, side_len)
        kernels = self.convolve(X, kernels, 8, side_len)

        return np.c_[X, kernels]

pipeline = Pipeline([
    ('kernel', KernelConvolver()),
    ('minmax_scaler', MinMaxScaler()),
])

X_train_features = pipeline.fit_transform(X_train)
print(X_train.shape, "->", X_train_features.shape)
```

    (1437, 64) -> (1437, 85)

```python
print("name\t accuracy\t precision\t recall\t f1")
for name, model in models:
    y_train_pred = cross_val_predict(model, X_train_features, y_train, cv=5)
    acc = np.round(accuracy_score(y_train, y_train_pred), 2)
    pre = np.round(precision_score(y_train, y_train_pred, average='micro'), 2)
    rec = np.round(recall_score(y_train, y_train_pred, average='micro'), 2)
    f1 = np.round(f1_score(y_train, y_train_pred, average='micro'), 2)
    print(f"{name}\t{acc}\t{pre}\t{rec}\t{f1}")
```

    name	 accuracy	 precision	 recall	 f1
    SGD		0.94	0.94	0.94	0.94
    Log Reg		0.96	0.96	0.96	0.96
    Multi NB	0.89	0.89	0.89	0.89
    Gaus NB		0.85	0.85	0.85	0.85
    ovo_SGD		0.96	0.96	0.96	0.96
    ovo_Log Reg	0.97	0.97	0.97	0.97
    ovo_Mult NB	0.89	0.89	0.89	0.89
    ovo_Gaus NB	0.85	0.85	0.85	0.85

The data is already heavily pre-processed. Running a simple kernel-averaging scheme slightly worsens performance across the board with the exception of the Gaussian NB.

## Step 5: Error Analysis

```python
fig, axs = plt.subplots(2,4)
fig.tight_layout()
plt.setp(axs, xticks=list(range(10)), yticks=list(range(10)))
i = 0
for name, model in models:
    y_train_pred = cross_val_predict(model, X_train, y_train, cv=5)
    conf_mx = confusion_matrix(y_train, y_train_pred)
    row_sums = conf_mx.sum(axis=1, keepdims=True)
    norm_conf_mx = conf_mx / row_sums
    np.fill_diagonal(norm_conf_mx, 0)
    axs[i%2, i // 2].imshow(norm_conf_mx, cmap = "jet")
    axs[i%2, i // 2].set_title(name.replace("\t", ""))
    i += 1
fig.text(0.5, -0.04, 'Actual', ha='center')
fig.text(0.02, 0.5, 'Predicted', va='center', rotation='vertical')
plt.show()
```

It seems that Log Reg and Mult NB both confuse $9$ for $5$ which is reasonable. SGD seems to have a completely judgement and mixes up $(1, 8)$ and $(3, 8)$.

```python
fig, axs = plt.subplots(2,4)
fig.tight_layout()
plt.setp(axs, xticks=list(range(10)), yticks=list(range(10)))
i = 0
for name, model in models:
    y_train_pred = cross_val_predict(model, X_train_features, y_train, cv=5)
    conf_mx = confusion_matrix(y_train, y_train_pred)
    row_sums = conf_mx.sum(axis=1, keepdims=True)
    norm_conf_mx = conf_mx / row_sums
    np.fill_diagonal(norm_conf_mx, 0)
    axs[i%2, i // 2].imshow(norm_conf_mx, cmap = "jet")
    axs[i%2, i // 2].set_title(name.replace("\t", ""))
    i += 1
fig.text(0.5, -0.04, 'Actual', ha='center')
fig.text(0.02, 0.5, 'Predicted', va='center', rotation='vertical')
plt.show()
```

The data preprocessing seems to have lowered extreme errors in log reg and SGD.

## Step 6: Test set Evaluation

```python
print("name\t accuracy\t precision\t recall\t f1")

for name, model in models:
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    acc = np.round(accuracy_score(y_test, y_pred), 2)
    pre = np.round(precision_score(y_test, y_pred, average='weighted'), 2)
    rec = np.round(recall_score(y_test, y_pred, average='weighted'), 2)
    f1 = np.round(f1_score(y_test, y_pred, average='weighted'), 2)
    print(f"{name}\t{acc}\t{pre}\t{rec}\t{f1}")
```

    name	 accuracy	 precision	 recall	 f1
    SGD		0.92	0.93	0.92	0.92
    Log Reg		0.96	0.96	0.96	0.96
    Multi NB	0.89	0.89	0.89	0.89
    Gaus NB		0.81	0.85	0.81	0.82
    ovo_SGD		0.97	0.97	0.97	0.97
    ovo_Log Reg	0.98	0.98	0.98	0.97
    ovo_Mult NB	0.89	0.89	0.89	0.89
    ovo_Gaus NB	0.81	0.85	0.81	0.82

```python
print("name\t accuracy\t precision\t recall\t f1")

X_test_features = pipeline.fit_transform(X_test)
for name, model in models:
    model.fit(X_train_features, y_train)
    y_pred = model.predict(X_test_features)
    acc = np.round(accuracy_score(y_test, y_pred), 2)
    pre = np.round(precision_score(y_test, y_pred, average='weighted'), 2)
    rec = np.round(recall_score(y_test, y_pred, average='weighted'), 2)
    f1 = np.round(f1_score(y_test, y_pred, average='weighted'), 2)
    print(f"{name}\t{acc}\t{pre}\t{rec}\t{f1}")
```

    name	 accuracy	 precision	 recall	 f1
    SGD		0.93	0.93	0.93	0.93
    Log Reg		0.96	0.96	0.96	0.96
    Multi NB	0.88	0.87	0.88	0.87
    Gaus NB		0.83	0.85	0.83	0.83
    ovo_SGD		0.94	0.94	0.94	0.94
    ovo_Log Reg	0.98	0.98	0.98	0.97
    ovo_Mult NB	0.88	0.87	0.88	0.87
    ovo_Gaus NB	0.83	0.85	0.83	0.83

Data transforming did not seem to improve performance much.

```python

```


## OIQ Exam Question 2

Question from OIQ Technical Exam, obviously meant to be solved using moment distribution, but
here we see how easy it is using slope deflection instead.  This version users a newer version of 'sdutil'
that computes end-shears as well as moments.

```python
from IPython import display
display.SVG('oiq-frame-1.svg')
```

### Solution 1
Actually, solution 1 does not use any special library modules at all (except for 'sympy').

```python
from sympy import *
```

```python
var('EI ta tb tc td')
Mab = (EI/4)*(4*ta + 2*tb) - 24*4**2/12
Mba = (EI/4)*(2*ta + 4*tb) + 24*4**2/12
Mbc = (EI/2/4)*(4*tb + 2*tc)
Mcb = (EI/2/4)*(2*tb + 4*tc)
Mcd = (EI/4)*(4*tc + 2*td) - 72*4/8
Mdc = (EI/4)*(2*tc + 4*td) + 72*4/8
```

```python
eqns = [ta, Mba+Mbc, Mcb+Mcd, Mdc]
soln = solve( eqns, [ta, tb, tc, td] )
soln
```

    {ta: 0.0,
     tb: -29.5172413793103/EI,
     tc: 49.1034482758621/EI,
     td: -60.551724137931/EI}

```python
[m.subs(soln).n(4) for m in [Mab,Mba,Mbc,Mcb,Mcd,Mdc]]
```

    [-46.76, 2.483, -2.483, 17.17, -17.17, 0]

```python
Rd = (Mcd + 72*2)/4
Ra = (-(Mab + Mba) + 24*4*2)/4
Rc = 24*4 + 72 - Ra - Rd
Hd = (Mbc + 72*2 - Rd*4)/4
[r.subs(soln).n(4) for r in [Ra,Rc,Rd,Hd]]
```

    [59.07, 77.22, 31.71, 3.672]

### Solution 2
Here it is again, using the SD utilities:

```python
from sympy import *
init_printing(use_latex='mathjax')
from sdutil2 import SD,FEF
```

```python
var('EI theta_a theta_b theta_c theta_d')
M_ab, M_ba, V_ab, V_ba = SD(4,EI,theta_a,theta_b) + FEF.udl(4,24)
M_bc, M_cb, V_bc, V_cb = SD(4,EI/2,theta_b,theta_c)
M_cd, M_dc, V_cd, V_dc = SD(4,EI,theta_c,theta_d) + FEF.p(4,72,2)
```

```python
eqns = [theta_a, M_ba+M_bc, M_cb+M_cd, M_dc]
soln = solve( eqns, [theta_a, theta_b, theta_c, theta_d] )
soln
```

$$\left \{ \theta_{a} : 0.0, \quad \theta_{b} : - \frac{29.5172413793103}{EI}, \quad \theta_{c} : \frac{49.1034482758621}{EI}, \quad \theta_{d} : - \frac{60.551724137931}{EI}\right \}$$

End moments and shears:

```python
[m.subs(soln).n(4) for m in [M_ab,M_ba,M_bc,M_cb,M_cd,M_dc]]
```

$$\left [ -46.76, \quad 2.483, \quad -2.483, \quad 17.17, \quad -17.17, \quad 0\right ]$$

```python
[m.subs(soln).n(4) for m in [V_ab,V_ba,V_bc,V_cb,V_cd,V_dc]]
```

$$\left [ 59.07, \quad -36.93, \quad -3.672, \quad -3.672, \quad 40.29, \quad -31.71\right ]$$

Reactions:

```python
R_d = -V_dc
R_c = V_cd - V_ba
R_a = V_ab
H_d = -V_cb
H_a = V_cb
M_a = M_ab
[r.subs(soln).n(4) for r in [R_a,R_c,R_d,H_d,H_a,M_a]]
```

$$\left [ 59.07, \quad 77.22, \quad 31.71, \quad 3.672, \quad -3.672, \quad -46.76\right ]$$

#### Equilibrium check

```python
## sum vertical forces
(R_a + R_c + R_d - 24*4 - 72).subs(soln)
```

$$0$$

```python
## sum horizontal forces
(H_a+H_d).subs(soln)
```

$$0$$

```python
## sum moments about a
(M_a + 24*4*2 - R_c*4 + 72*6 - R_d*8 - H_d*4).subs(soln)
```

$$0$$

```python

```

```python

```


# Full-Waveform Inversion (FWI)

This notebook is the third in a series of tutorial highlighting various aspects of seismic inversion based on Devito operators. In this second example we aim to highlight the core ideas behind seismic inversion, where we create an image of the subsurface from field recorded data. This tutorial follows on the modelling tutorial and will reuse the modelling and velocity model.

## Inversion requirement

Seismic inversion relies on two known parameters:

- **Field data** - or also called **recorded data**. This is a shot record corresponding to the true velocity model. In practice this data is acquired as described in the first tutorial. In order to simplify this tutorial we will fake field data by modelling it with the true velocity model.

- **Initial velocity model**. This is a velocity model that has been obtained by processing the field data. This model is a rough and very smooth estimate of the velocity as an initial estimate for the inversion. This is a necessary requirement for any optimization (method).

## Inversion computational setup

In this tutorial, we will introduce the gradient operator. This operator corresponds to the imaging condition introduced in the previous tutorial with some minor modifications that are defined by the objective function (also referred to in the tutorial series as the *functional*, *f*) and its gradient, *g*. We will define this two terms in the tutorial too.

## Notes on the operators

As we already describe the creation of a forward modelling operator, we will only call an wrapped function here. This wrappers already contains all the necessary operator for seismic modeling, imaging and inversion, however any new operator will be fully described and only used from the wrapper in the next tutorials.

```python
import numpy as np
%matplotlib inline

from devito import configuration
configuration['log_level'] = 'WARNING'
```

## Computational considerations

As we will see in this tutorial, FWI is again very computationally demanding, even more so than RTM. To keep this tutorial as light-wight as possible we therefore again use a very small demonstration model. We also define here a few parameters for the final example runs that can be changed to modify the overall runtime of the tutorial.

```python
nshots = 9  # Number of shots to create gradient from
nreceivers = 101  # Number of receiver locations per shot
fwi_iterations = 8  # Number of outer FWI iterations
```

# True and smooth velocity models

As before, we will again use a very simple model domain, consisting of a circle within a 2D domain. We will again use the "true" model to generate our synthetic shot data and use a "smooth" model as our initial guess. In this case the smooth model is very smooth indeed - it is simply a constant background velocity without any features.

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import demo_model, plot_velocity, plot_perturbation

# Define true and initial model
shape = (101, 101)  # Number of grid point (nx, nz)
spacing = (10., 10.)  # Grid spacing in m. The domain size is now 1km by 1km
origin = (0., 0.)  # Need origin to define relative source and receiver locations

model = demo_model('circle-isotropic', vp=3.0, vp_background=2.5,
                    origin=origin, shape=shape, spacing=spacing, nbpml=40)

model0 = demo_model('circle-isotropic', vp=2.5, vp_background=2.5,
                     origin=origin, shape=shape, spacing=spacing, nbpml=40)

plot_velocity(model)
plot_velocity(model0)
plot_perturbation(model0, model)
```

## Acquisition geometry

In this tutorial, we will use the easiest case for inversion, namely a transmission experiment. The sources are located on one side of the model and the receivers on the other side. This allow to record most of the information necessary for inversion, as reflections usually lead to poor inversion results.

```python
#NBVAL_IGNORE_OUTPUT
# Define acquisition geometry: source
from examples.seismic import RickerSource, Receiver

# Define time discretization according to grid spacing
t0 = 0.
tn = 1000.  # Simulation lasts 1 second (1000 ms)
dt = model.critical_dt  # Time step from model grid spacing
nt = int(1 + (tn-t0) / dt)  # Discrete time axis length
time = np.linspace(t0, tn, nt)  # Discrete modelling time

f0 = 0.010  # Source peak frequency is 10Hz (0.010 kHz)
src = RickerSource(name='src', grid=model.grid, f0=f0, time=np.linspace(t0, tn, nt))
src.coordinates.data[0, :] = np.array(model.domain_size) * .5
src.coordinates.data[0, 0] = 20.  # 20m from the left end

# We can plot the time signature to see the wavelet
src.show()
```

```python
#NBVAL_IGNORE_OUTPUT
# Define acquisition geometry: receivers

# Initialize receivers for synthetic data
rec = Receiver(name='rec', grid=model.grid, npoint=nreceivers, ntime=nt)
rec.coordinates.data[:, 1] = np.linspace(0, model.domain_size[0], num=nreceivers)
rec.coordinates.data[:, 0] = 980. # 20m from the right end

# Plot acquisition geometry
plot_velocity(model, source=src.coordinates.data,
              receiver=rec.coordinates.data[::4, :])
```

## True and smooth data

We can generate shot records for the true and smoothed initial velocity models, since the difference between them will again form the basis of our imaging procedure.

```python
# Compute synthetic data with forward operator
from examples.seismic.acoustic import AcousticWaveSolver

solver = AcousticWaveSolver(model, src, rec, space_order=4)
true_d, _, _ = solver.forward(src=src, m=model.m)
```

```python
# Compute initial data with forward operator
smooth_d, _, _ = solver.forward(src=src, m=model0.m)
```

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import plot_shotrecord

# Plot shot record for true and smooth velocity model and the difference
plot_shotrecord(true_d.data, model, t0, tn)
plot_shotrecord(smooth_d.data, model, t0, tn)
plot_shotrecord(smooth_d.data - true_d.data, model, t0, tn)
```

# Full-Waveform Inversion

## Formulation

Full-waveform inversion (FWI) aims to invert an accurate model of the discrete wave velocity, $\mathbf{c}$, or equivalently the square slowness of the wave, $\mathbf{m} = \frac{1}{\mathbf{c}^2}$, from a given set of measurements of the pressure wavefield $\mathbf{u}$. This can be expressed as the following optimization problem [1, 2]:

\begin{aligned}
	\mathop{\hbox{minimize}}_{\mathbf{m}} \Phi_s(\mathbf{m})&=\frac{1}{2}\left\lVert\mathbf{P}_r
	\mathbf{u} - \mathbf{d}\right\rVert_2^2 \\
	\mathbf{u} &= \mathbf{A}(\mathbf{m})^{-1} \mathbf{P}_s^T \mathbf{q}_s,
\end{aligned}

where $\mathbf{P}_r$ is the sampling operator at the receiver locations, $\mathbf{P}_s^T$ is the injection operator at the source locations, $\mathbf{A}(\mathbf{m})$ is the operator representing the discretized wave equation matrix, $\mathbf{u}$ is the discrete synthetic pressure wavefield, $\mathbf{q}_s$ is the corresponding pressure source and $\mathbf{d}$ is the measured data. It is worth noting that $\mathbf{m}$ is the unknown in this formulation and that multiple implementations of the wave equation operator $\mathbf{A}(\mathbf{m})$ are possible.

We have already defined a concrete solver scheme for $\mathbf{A}(\mathbf{m})$ in the first tutorial, including appropriate implementations of the sampling operator $\mathbf{P}_r$ and source term $\mathbf{q}_s$.

To solve this optimization problem using a gradient-based method, we use the
adjoint-state method to evaluate the gradient $\nabla\Phi_s(\mathbf{m})$:

\begin{align}
 \nabla\Phi_s(\mathbf{m})=\sum_{\mathbf{t} =1}^{n_t}\mathbf{u}[\mathbf{t}] \mathbf{v}_{tt}[\mathbf{t}] =\mathbf{J}^T\delta\mathbf{d}_s,
\end{align}

where $n_t$ is the number of computational time steps, $\delta\mathbf{d}_s = \left(\mathbf{P}_r \mathbf{u} - \mathbf{d} \right)$ is the data residual (difference between the measured data and the modelled data), $\mathbf{J}$ is the Jacobian operator and $\mathbf{v}_{tt}$ is the second-order time derivative of the adjoint wavefield solving:

\begin{align}
 \mathbf{A}^T(\mathbf{m}) \mathbf{v} = \mathbf{P}_r^T \delta\mathbf{d}.
\end{align}

We see that the gradient of the FWI function is the previously defined imaging condition with an extra second-order time derivative. We will therefore reuse the operators defined previously inside a Devito wrapper.

## FWI gradient operator

To compute a single gradient $\nabla\Phi_s(\mathbf{m})$ in our optimization workflow we again use `solver.forward` to compute the entire forward wavefield $\mathbf{u}$ and a similar pre-defined gradient operator to compute the adjoint wavefield `v`. The gradient operator provided by our `solver` utility also computes the correlation between the wavefields, allowing us to encode a similar procedure to the previous imaging tutorial as our gradient calculation:

- Simulate the forward wavefield with the background velocity model to get the synthetic data and save the full wavefield $\mathbf{u}$
- Compute the data residual
- Back-propagate the data residual and compute on the fly the gradient contribution at each time step.

This procedure is applied to multiple source positions and summed to obtain a gradient image of the subsurface. We again prepare the source locations for each shot and visualize them, before defining a single gradient computation over a number of shots as a single function.

```python
#NBVAL_IGNORE_OUTPUT

# Prepare the varying source locations sources
source_locations = np.empty((nshots, 2), dtype=np.float32)
source_locations[:, 0] = 30.
source_locations[:, 1] = np.linspace(0., 1000, num=nshots)

plot_velocity(model, source=source_locations)
```

```python
# Create FWI gradient kernel
from devito import Function, clear_cache

def fwi_gradient(m_in):
    # Important: We force previous wavefields to be destroyed,
    # so that we may reuse the memory.
    clear_cache()

    # Create symbols to hold the gradient and residual
    grad = Function(name="grad", grid=model.grid)
    residual = Receiver(name='rec', grid=model.grid,
                        ntime=nt, coordinates=rec.coordinates.data)
    objective = 0.

    for i in range(nshots):
        # Update source location
        src.coordinates.data[0, :] = source_locations[i, :]

        # Generate synthetic data from true model
        true_d, _, _ = solver.forward(src=src, m=model.m)

        # Compute smooth data and full forward wavefield u0
        smooth_d, u0, _ = solver.forward(src=src, m=m_in, save=True)

        # Compute gradient from data residual and update objective function
        residual.data[:] = smooth_d.data[:] - true_d.data[:]
        objective += .5*np.linalg.norm(residual.data.reshape(-1))**2
        solver.gradient(rec=residual, u=u0, m=m_in, grad=grad)

    return objective, grad.data
```

Having defined our FWI gradient procedure we can compute the initial iteration from our starting model. This allows us to visualize the gradient alongside the model perturbation and the effect of the gradient update on the model.

```python
#NBVAL_IGNORE_OUTPUT

# Compute gradient of initial model
ff, update = fwi_gradient(model0.m)
print('Objective value is %f ' % ff)
```

    Objective value is 57237.215627

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import plot_image

# Plot the FWI gradient
plot_image(update, vmin=-1e4, vmax=1e4, cmap="jet")

# Plot the difference between the true and initial model.
# This is not known in practice as only the initial model is provided.
plot_image(model0.m.data - model.m.data, vmin=-1e-1, vmax=1e-1, cmap="jet")

# Show what the update does to the model
alpha = .05 / np.max(update)
plot_image(model0.m.data - alpha*update, vmin=.1, vmax=.2, cmap="jet")
```

We see that the gradient and the true perturbation have the same sign, therefore, with an appropriate scaling factor, we will update the model in the correct direction.

```python
# Define bounding box constraints on the solution.
def apply_box_constraint(m):
    # Maximum possible 'realistic' velocity is 3.5 km/sec
    # Minimum possible 'realistic' velocity is 2 km/sec
    return np.clip(m, 1/3.5**2, 1/2**2)
```

```python
#NBVAL_SKIP

# Run FWI with gradient descent
history = np.zeros((fwi_iterations, 1))
for i in range(0, fwi_iterations):
    # Compute the functional value and gradient for the current
    # model estimate
    phi, direction = fwi_gradient(model0.m)

    # Store the history of the functional values
    history[i] = phi

    # Artificial Step length for gradient descent
    # In practice this would be replaced by a Linesearch (Wolfe, ...)
    # that would guarantee functional decrease Phi(m-alpha g) <= epsilon Phi(m)
    # where epsilon is a minimum decrease constant
    alpha = .005 / np.max(direction)

    # Update the model estimate and inforce minimum/maximum values
    model0.m.data[:] = apply_box_constraint(model0.m.data - alpha * direction)

    # Log the progress made
    print('Objective value is %f at iteration %d' % (phi, i+1))
```

    Objective value is 57237.215627 at iteration 1
    Objective value is 39749.780543 at iteration 2
    Objective value is 26149.841154 at iteration 3
    Objective value is 16186.648824 at iteration 4
    Objective value is 9285.255823 at iteration 5
    Objective value is 4789.281485 at iteration 6
    Objective value is 2008.388293 at iteration 7
    Objective value is 800.837208 at iteration 8

```python
#NBVAL_IGNORE_OUTPUT

# First, update velocity from computed square slowness
nbpml = model.nbpml
model0.vp = np.sqrt(1. / model0.m.data[nbpml:-nbpml, nbpml:-nbpml])

# Plot inverted velocity model
plot_velocity(model0)
```

```python
#NBVAL_SKIP
import matplotlib.pyplot as plt

# Plot objective function decrease
plt.figure()
plt.loglog(history)
plt.xlabel('Iteration number')
plt.ylabel('Misift value Phi')
plt.title('Convergence')
plt.show()
```

## References

[1] _Virieux, J. and Operto, S.: An overview of full-waveform inversion in exploration geophysics, GEOPHYSICS, 74, WCC1–WCC26, doi:10.1190/1.3238367, http://library.seg.org/doi/abs/10.1190/1.3238367, 2009._

[2] _Haber, E., Chung, M., and Herrmann, F. J.: An effective method for parameter estimation with PDE constraints with multiple right hand sides, SIAM Journal on Optimization, 22, http://dx.doi.org/10.1137/11081126X, 2012._

This notebook is part of the tutorial "Optimised Symbolic Finite Difference Computation with Devito" presented at the Intel® HPC Developer Conference 2017.