# Homework - Logistic Regression

This homework is part of a series of exercises for the CIVIL-226 Introduction to Machine Learning for Engineers course at EPFL. Copyright (c) 2022 [VITA](https://www.epfl.ch/labs/vita/) lab at EPFL
Use of this source code is governed by an MIT-style license that can be found in the LICENSE file or at https://www.opensource.org/licenses/MIT

**Author(s):** [David Mizrahi](mailto:david.mizrahi@epfl.ch), [Tom Winandy](mailto:tom.winandy@epfl.ch) and [Luc Reveyron](mailto:luc.reveyron@epfl.ch)

**2022 Revision:** [Brian Sifringer](mailto:brian.sifringer@epfl.ch), [Bastien Van Delft](mailto:bastien.vandelft@epfl.ch), [Taylor Mordan](mailto:taylor.mordan@epfl.ch)

### Name and SCIPER

**Name:** YOUR NAME HERE

**SCIPER:** YOUR SCIPER HERE

### Homework info
**Released:** Wednesday March 23, 2022
**Submission**: Thursday March 31, 2022 (before 11:59PM) on Moodle
**Grade weight:** 20% of the overall grade
**This is an individual assignment, the exchange of code between students is forbidden.**

This homework is composed of 4 parts:
- Part 1: Logistic regression (45 pts)
- Part 2: Regularization (20 pts)
- Part 3: Cross-validation (20 pts)
- Part 4: Evaluation metrics (15 pts)

**Total:** 100 pts

**Work through this exercise in order, as functions implemented early on can be used for later parts.**

**Tips:**
- Use the Table of Contents extension to quickly navigate to each part.
- In order to ensure that there are no hidden states in your notebook, frequently restart the kernel using: **Kernel -> Restart Kernel and Run all Selected Cells...**

Good luck!

### Imports

```python
# Function to align all tables to the left (useful for later on)
```

```python
%%html
<style>
table {float:left}
</style>
```

```python
from typing import Tuple, List

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import helpers
```

# Part 1: Logistic regression

The first part of this exercise consists of implementing logistic regression for a binary classification problem. In this part, you will use a slightly modified version of the Palmer Penguins dataset.

## 1.1. Dataset

### Loading the data & normalization

First, you'll work on a relatively simple dataset: a modified version of the Palmers Penguins dataset with only two features (bill length and flipper length), and two species (Gentoo and Chinstrap). Using `helpers.preprocess_data()` (check helpers.py for more info), we'll obtain a training and test set.

```python
penguins = pd.read_csv('data/penguins.csv')
penguins.head(5)
```

```python
X_train_penguins, y_train_penguins, X_test_penguins, y_test_penguins, feature_names, label_map = helpers.preprocess_data(penguins, label="species", train_size=0.70, seed=42)
```

Next, you need to normalize this data with the mean and standard deviation from the training set and keep the original variable name. For example, the normalized `X_train_penguins` should be called `X_train_penguins`.

**Task: Normalize the training and test data using `helpers.normalize()`**

**Note:** Remember that you should not use any of the knowledge you get from the test data when implementing a model. This includes the normalization step, where you should use the mean and standard deviation of the **training set** to normalize both the **training** and **test** set.

```python
# Normalize features of the training and test set using the mean and std of the training set features
### START CODE HERE ###
...
### END CODE HERE ###
```

```python
# These assertions check that your normalization didn't go completely wrong.
# Passing these assertions does not mean you will automatically get all points for that question
# We may use other tests to check the correctness of your implementation
assert np.allclose(X_train_penguins.mean(axis=0), 0)
assert np.allclose(X_train_penguins.std(axis=0), 1)
```

Let's preview the arrays for the training and test set.

```python
# Preview of X_train_penguins and y_train_penguins (separation of the features and the labels)
print('Training set features:')
print(f'X_train_penguins: \n {X_train_penguins[:3]}')

print('\nTraining set labels:')
print(f'y_train_penguins: \n {y_train_penguins[:3]}')
```

**Expected output:**

|                 |                                                  |
|-----------------|--------------------------------------------------|
| **X_train_penguins** | [[ 0.75973314  1.26418483]
[ 0.62957146 -0.72722878]
[-1.84350033 -1.87612124]] |
| **y_train_penguins**  | [1 0 0]

```python
# Preview of X_train_penguins and y_train_penguins (separation of the features and the labels)
print('Test set features:')
print(f'X_test_penguins: \n {X_test_penguins[:3]}')

print('\nTraining set labels:')
print(f'y_test_penguins: \n {y_test_penguins[:3]}')
```

**Expected output:**

|                 |                                                  |
|-----------------|--------------------------------------------------|
| **X_test_penguins** |  [[-1.58317698  0.1918852 ]
[-1.48555573  1.11099917]
[ 0.59703104  0.57484935]] |
| **y_test_penguins**  | [1 1 1]

```python
# Show shapes
print('Training set shape:')
print(f'X: {X_train_penguins.shape}, y: {y_train_penguins.shape}')

print('\nTest set shape:')
print(f'X: {X_test_penguins.shape}, y: {y_test_penguins.shape}')
```

### Notation

Now that we have pre-processed our dataset, here's how it looks:

- features: $\boldsymbol{X} \in \mathbb{R}^{N \times D}$, $\forall \ \boldsymbol{x}^{(i)} \in \boldsymbol{X}: \boldsymbol{x}^{(i)} \in \mathbb{R}^{D}$
- labels: $\boldsymbol{y} \in \mathbb{R}^{N}$, $\forall \ y^{(i)} \in \boldsymbol{y}: y^{(i)} \in \{0, 1\}$

 where $N$ is the number of examples in our dataset, and $D$ is the number of features per example

For the weights, we have:

 - weights: $\mathbf{w} \in \mathbb{R}^{D}$
 - bias: $b \in \mathbb{R}$

 **Note:**

 $\boldsymbol{X}$ is called the design matrix, where $\boldsymbol{X}_{i, :}$ denotes $\boldsymbol{x}^{(i)}$.
 Note that a single example $\boldsymbol{x}^{(i)}$ is a column vector of shape $(D \times 1)$, while the design matrix $\boldsymbol{X}$ is of shape $(N \times D)$, where each row represents an example and each column represents a feature.

## 1.2. Diving into logistic regression

In this section, you are going to implement the different functions needed for logistic regression.

### Sigmoid

A key element of logistic regression is the sigmoid function. This function takes any real-valued input and outputs a value in [0, 1]. The sigmoid function is defined as:
$$\sigma(z)= \frac {e^z}{1+e^z}= \frac{1}{1+e^{-z}}$$

Here's a plot of the sigmoid:

**Task: Implement `sigmoid()`**

**Hint:** Use `np.exp(x)` to take the exponential of a number. You can find the documentation for this method [here](https://numpy.org/doc/stable/reference/generated/numpy.exp.html).

```python
def sigmoid(z: np.ndarray) -> np.ndarray:
    """ Sigmoid function

    Args:
        z: Input data of shape (N, )

    Returns:
        np.ndarray: Sigmoid of z of shape (N, ), where each value is in [0, 1].
    """
    ### START CODE HERE ###
    ...
    ### END CODE HERE ###
    return s
```

```python
# Verify implementation
a = sigmoid(np.array([3, 0.5, -1]))
print(f'a: {np.round(a, 4)}')
```

**Expected output:**

|   |                                                  |
|---|--------------------------------------------------|
| **a** | [0.9526 0.6225 0.2689] |

### Logistic output

Now that you have a function that computes the sigmoid function, you can implement a function that gives the logistic output. As a reminder, this function outputs the estimated probability that y = 1 for an input $\boldsymbol{x}$, that is: $ \hat{y}^{(i)} = p(y^{(i)} = 1  | \boldsymbol{x}^{(i)})$.
The logistic output (of a single example) is defined as:
$$\hat{y}^{(i)} = \sigma(\mathbf{w}^{T} \boldsymbol{x}^{(i)} + b) = \frac{1}{1 + e^{-(\mathbf{w}^{T}  \boldsymbol{x}^{(i)} + b)}}$$

For all examples, the output is defined as:
$$\mathbf{\hat{y}} = \sigma(\mathbf{X} \mathbf{w} + b \mathbf{1})$$
where $\mathbf{1}$ is a vector of ones of shape $(N \times 1)$, and the sigmoid is applied element-wise.

**Task: Implement `logistic_output()`**

**Hint:** To vectorize this operation , use `np.matmul(a, b)` (or equivalently `a @ b`) for matrix multiplication.
You can find the documentation for this method [here](https://numpy.org/doc/stable/reference/generated/numpy.matmul.html).

**Hint:** Remember that you have already coded the sigmoid function. This might help you in the logistic output implementation.

```python
def logistic_output(X: np.ndarray, w: np.ndarray, b: float) -> np.ndarray:
    """ Output of logistic regression

    Args:
        X: Dataset of shape (N, D)
        w: Weights of logistic regression model of shape (D, )
        b: bias, a scalar
    Returns:
        y_hat (np.ndarray): Output of logistic regression of shape (N, )
    """
    ### START CODE HERE ###
    ...
    ### END CODE HERE ###
    return y_hat
```

```python
# Verify implementation
X = X_train_penguins[:3]
w = np.array([1, 1])
b = 1

y_hat = logistic_output(X, w, b)
print(f'y_hat: {np.round(y_hat, decimals=4)}')
```

**Expected output:**

|                 |                                                  |
|-----------------|--------------------------------------------------|
| **y_hat** | [0.9536 0.7114 0.0618] |

### Binary cross-entropy loss

In order to train your model, you also need a loss function, which penalizes outputs based on how far off they are from the ground-truth. Here, we'll use the logistic loss / binary cross-entropy loss.

It is defined as:

$$\text{BCE}(\mathbf{w}, b) = - \frac{1}{N}\sum^{N}_{i=1} y^{(i)} \log(\hat{y}^{(i)}) + (1-y^{(i)}) \log(1- \hat{y}^{(i)})$$ where $\log(x)$ refers to the natural logarithm of $x$.

**Task: Implement `bce_loss()`**

**Hint:** Use `np.log(x)` to take the natural logarithm of x. You can find the documentation for this method [here](https://numpy.org/doc/stable/reference/generated/numpy.log.html).

Due to floating-point arithmetic, values very close to 0 will get rounded to 0 (likewise for values very close to 1).

However, the natural logarithm log(x) is undefined for values equal to 0. An easy way to fix this is to add a small term $\epsilon$ to the logarithm. For example, write `np.log(x + epsilon)` instead of `np.log(x)`.

In order to get full points (and avoid problems in later parts of this homework), add  `epsilon` (set to $10^{-9}$) whenever you call `np.log()`.

```python
def bce_loss(X: np.ndarray,  y: np.ndarray, w: np.ndarray, b: float) -> float:
    """ Binary cross-entropy loss function

    Args:
        X: Dataset of shape (N, D)
        y: Labels of shape (N, ).
        w: Weights of logistic regression model of shape (D, )
        b: bias, a scalar

    Returns:
        float: binary cross-entropy loss.
    """
    # Add the epsilon term to the np.log() in your implementation (e.g. do np.log(x + epsilon) instead of np.log(x))
    # Epsilon is there to avoid log(0)
    epsilon = 1e-9

    ### START CODE HERE ###
    ...
    ### END CODE HERE ###

    return loss
```

```python
# Verify implementation
X = X_train_penguins[:3]
y  = y_train_penguins[:3]
w = np.array([1, 1])
b = 1

loss = bce_loss(X, y, w, b)
print(f'Loss: {loss:.3f}')
```

**Expected output:**

|                 |                                                  |
|-----------------|--------------------------------------------------|
| **Loss** | 0.451|

### Binary cross-entropy loss gradient

After having computed the loss, you need to update the parameters of the model in order to decrease it. This can be done with gradient descent, and for that, you'll need the gradient of your loss function.

In class, we have seen the gradient of the binary cross-entropy loss defined as:
$$\frac{\partial \text{BCE}(\mathbf{w},b)}{\partial \mathbf{w}} = \frac{1}{N}\sum^{N}_{i=1}\boldsymbol{x}^{(i)} (\hat{y}^{(i)} - y^{(i)}) $$
$$ \frac{\partial \text{BCE}(\mathbf{w},b)}{\partial b} = \frac{1}{N}\sum^{N}_{i=1}\hat{y}^{(i)} - y^{(i)}$$

A vectorized implementation of that first expression is the following:

$$\frac{\partial \text{BCE}(\mathbf{w},b)}{\partial \mathbf{w}} = \frac{1}{N} \ \mathbf{X}^T (\mathbf{\hat{y}} - \mathbf{y})$$

**Task: Implement `bce_gradient()`**

```python
def bce_gradient(X: np.ndarray,  y: np.ndarray, w: np.ndarray, b: float) -> Tuple[np.ndarray, float]:
    """ Gradient of the binary-cross entropy loss

    Args:
        X: Dataset of shape (N, D)
        y: Labels of shape (N, )
        w: Weights of logistic regression model of shape (D, )
        b: bias, a scalar

    Returns:
        dw (np.ndarray) gradient of the loss with respect to w of shape (D, )
        db (float) gradient of the loss with respect to b, a scalar
    """
    ### START CODE HERE ###
    ...
    ### END CODE HERE ###

    return dw, db

```

```python
# Verify implementation
X = X_train_penguins[:3]
y  = y_train_penguins[:3]
w = np.array([1, -1])
b = -0.5

dw, db = bce_gradient(X, y, w, b)
print(f'dw: {np.round(dw, decimals=4)}')
print(f'db: {db:.4f}')

```

**Expected output:**

|    |                                   |
|----|-----------------------------------|
| **dw** | [-0.2748 -0.7195] |
| **db** | 0.1184                           |

### Classification

The output of logistic regression estimates $ \hat{y}^{(i)} = P(y^{(i)} = 1  | \boldsymbol{x}^{(i)})$.

Based on this output, you'll want to classify each example in one of the 2 categories. To do so, you'll need to set $\text{predicted label}^{(i)}$ to 1 when $ \hat{y}^{(i)} \gt 0.5$ (as it means it is more likely to be of class 1 than of class 0), and to 0 otherwise.

**Task: Implement `classify()`**

**Hint:** Use `np.where(condition, [x, y])`. You can find the documentation for this method [here](https://numpy.org/doc/stable/reference/generated/numpy.where.html).

```python
def classify(y_hat: np.ndarray) -> np.ndarray:
    """ Classification function for binary class logistic regression.

    Args:
        y_hat (np.array): Output of logistic regression of shape (N, ).
    Returns:
        np.array: Label assignments of data of shape (N, )
    """
    ### START CODE HERE ###
    ...
    ### END CODE HERE ###
    return labels_pred
```

```python
y_hat = np.array([0.25, 0.75])
print(classify(y_hat))
```

**Expected output:** [0 1]

### Accuracy

To measure how well your model is doing, we'll first consider the accuracy as our metric. It corresponds to the fraction of predictions our model got right.

$$\text{Accuracy} = \frac{\text{Number of correct predictions}}{\text{Total number of predictions}}$$

**Task: Implement `accuracy()`**

```python
def accuracy(labels_gt: np.ndarray, labels_pred: np.ndarray) -> float:
    """Computes the accuracy.

    Args:
        labels_gt: labels (ground-truth) of shape (M, ).
        labels_pred: Predicted labels of shape (M, ).

    Returns:
        float: Accuracy, in range [0, 1].
    """
    ### START CODE HERE ###
    ...
    ### END CODE HERE ###
```

```python
# Check that output is in [0, 1] (ensures that output is not a percentage)
assert 0.0 <= accuracy(np.array([1, 0]), np.array([1, 1])) <= 1.0

```

## 1.3. Training
You will now implement the training process by putting together all the functions you have implemented so far.

Here are the different steps of the training process:
- Compute the output of logistic regression (`y_hat`)
- Compute the loss of the model (`loss`)
- Compute the derivates w.r.t to the weights `dw` and the bias `db`
- Update `w` and `b`

Recall that a gradient step is:
$$ \mathbf{w}  := \mathbf{w} - \alpha \frac{\partial J}{\partial \mathbf{w}} $$

$$b  := b - \alpha \frac{\partial J}{\partial b} $$ where $J$ is the loss function and $\alpha$ is the learning rate.

**Task: Complete the following `train_logistic_regression()` function.**

```python
def train_logistic_regression(X: np.ndarray,
                              y: np.ndarray,
                              max_iters: int = 101,
                              lr: float = 0.5,
                              loss_freq: int = 0) -> Tuple[np.ndarray, float, dict]:
    """ Training function for binary class logistic regression using gradient descent

    Args:
        X: Dataset of shape (N, D).
        y: Labels of shape (N, ).
        max_iters: Maximum number of iterations. Default : 100
        lr: The learning rate of  the gradient step. Default : 1
        loss_freq : Prints the loss every `loss_freq` iterations. Default : 0

    Returns:
        w: weights of shape (D, )
        b: scalar
        viz_d: dict used for visualizations
    """

    # Initialize weights
    np.random.seed(0)
    w = np.random.normal(0, 1, size=(X.shape[1], ))
    b = 0

    # Initialize dict with lists to keep track of loss, accuracy, weight and bias evolution
    logger = {'loss': [],
             'acc': [],
             'w': [],
             'b': []
            }

    for i in range(max_iters):
        # Compute loss, dw, db and update w and b
        ### START CODE HERE ###
        ...
        ### END CODE HERE ###

        # Keep track of parameter, loss and accuracy values for each iteration
        logger['w'].append(w)
        logger['b'].append(b)
        logger['loss'].append(loss)
        y_hat = logistic_output(X, w, b)
        logger['acc'].append(accuracy(y, classify(y_hat)))

        if (loss_freq !=0) and i % loss_freq == 0:
            print(f'Loss at iter {i}: {loss:.5f}')

    if (loss_freq != 0):
        print('\nFinal loss: {:.5f}'.format(logger['loss'][-1]))

    return w, b, logger
```

Now, run the following cell with the default hyperparameters (max_iters=101, lr=0.5) to train your model parameters (`w` & `b`).

```python
w, b, logger = train_logistic_regression(X_train_penguins, y_train_penguins, max_iters=101, lr=0.5, loss_freq=5)
```

**Expected output:**

|            |         |
|------------|---------|
| **Final loss**  | 0.10350 |

## 1.4. Results

### Model accuracy
Great, your model is trained! But how well does it perform? To find out, run the following cells to get the accuracy of your trained model on the train and test set.

```python
# Train acc
y_hat = logistic_output(X_train_penguins, w, b)
acc = accuracy(y_train_penguins, classify(y_hat))
print(f'Train accuracy: {100 * acc:.2f}%')
```

```python
# Test acc
y_hat = logistic_output(X_test_penguins, w, b)
acc = accuracy(y_test_penguins, classify(y_hat))
print(f'Test accuracy: {100*acc:.2f}%')
```

**Expected output:**

|            |         |
|------------|---------|
| **Train accuracy**  | 95.79% |
| **Test accuracy**  | 97.56% |

### Visualization of the training
Let's observe how the training went. The first graph plots the evolution of the loss during the training while the second graph plots the evolution of the accuracy during the training. If everything was implemented correctly, you'll notice that as training goes on, the loss decreases and the training accuracy tends to increase.

```python
# Plot the evolution of loss during training
def plot_loss(loss_list):
    fig = plt.figure(figsize=(6, 6))
    ax = fig.add_subplot(111)
    step = np.arange(1, len(loss_list)+1)
    plt.plot(step, loss_list)
    plt.title('Evolution of the loss during the training')
    plt.xlabel('iteration')
    plt.ylabel('Training loss')
    plt.show()

plot_loss(logger["loss"])
```

```python
# Plot the evolution of accuracy during training
def plot_acc(acc_list):
    fig = plt.figure(figsize=(6, 6))
    ax = fig.add_subplot(111)
    step = np.arange(1, len(acc_list)+1)
    plt.plot(step, acc_list)
    plt.title('Evolution of the accuracy during the training')
    plt.xlabel('iteration')
    plt.ylabel('Training accuracy')

plot_acc(logger["acc"])
```

### Plotting results

We also implemented functions in helpers.py which can help you visualize the decision boundary for this model, and visualize which points are correctly classified, and which aren't. This step is important in the process of developing your algorithm as it allows you to make sense of the mathematical result you obtained.

```python
class_names = list(label_map.values())
ax_titles = ["Bill length (normalized)", "Flipper length (normalized)"]

helpers.plot_boundaries(X=X_train_penguins,
                        y=y_train_penguins,
                        w=w, b=b,
                        output_func=logistic_output,
                        class_names=class_names,
                        ax_titles=ax_titles,
                        train=True)
```

```python
helpers.plot_boundaries(X=X_test_penguins,
                        y=y_test_penguins,
                        w=w,
                        b=b,
                        output_func=logistic_output,
                        class_names=class_names,
                        ax_titles=ax_titles,
                        train=False)
```

If everything was implemented correctly, you'll notice that your model fits the training data with a good accuracy and also generalizes to the test data. Generalizing to new and unknown data points is the ultimate goal of any machine learning algorithm. In our case, the visualisation highlights this property.

### Training visualization

We alo implemented an interactive function which you can use to visualize the decision boundaries at different iterations.
Notice how the decision boundary gradually improves during training?

```python
helpers.interactive_boundaries(X_train_penguins,
                               y_train_penguins,
                               X_test_penguins,
                               y_test_penguins,
                               logger["w"],
                               logger["b"],
                               logistic_output,
                               class_names,
                               ax_titles,
                               total_steps=50)
```

Good job on completing the first part of this exercise! In the next parts, we'll implement a slightly modified version of logistic regression and train it on a different dataset. The functions implemented so far will prove to be very useful later on, so make sure that they are correct!

# Part 2: Regularization

The second part of this exercise consists of adding regularization to improve logistic regression's performance on a higher dimensional dataset.

## 2.1. Connectionist Bench Dataset

In this part, you'll use the Connectionist Bench Dataset, where the task is to train a model to discriminate between sonar signals bounced off a mine (metal cylinder) and those bounced off a roughly cylindrical rock. Each sample is composed of 60 features and a label: "S" if it is a stone and "M" if it is Metal (mine).

```python
sonar = pd.read_csv("data/sonar_2022.csv")
```

```python
print(f"Shape: {sonar.shape}")

sonar.head(5)
```

We use `helpers.preprocess_data()` to extract a training and test set from this data. The label "M" (metal) is mapped to 0 and the label "S" (stone) is mapped to 1.

```python
X_train_sonar, y_train_sonar, X_test_sonar, y_test_sonar, feature_names, label_map = helpers.preprocess_data(sonar, label="material", train_size=0.75, seed=42)
```

```python
label_map
```

You also need to normalize the data, as was done in the previous part. You can use `helpers.normalize()` to do so.

**Task: Normalize `X_train_sonar` and `X_test_sonar` using `helpers.normalize()`**

```python
# Normalize features of the training and test set using the mean and std of the training set features
# Use helpers.normalize()
### START CODE HERE ###
...
### END CODE HERE ###
```

```python
assert np.allclose(X_train_sonar.mean(axis=0), 0)
assert np.allclose(X_train_sonar.std(axis=0), 1)
```

```python
# Show shapes
print('Training set shape:')
print(f'X: {X_train_sonar.shape}, y: {y_train_sonar.shape}')

print('\nTest set shape:')
print(f'X: {X_test_sonar.shape}, y: {y_test_sonar.shape}')
```

Now that you've processed the data, let's check how well logistic regression performs without regularization.

```python
w, b, _ = train_logistic_regression(X_train_sonar, y_train_sonar, max_iters=1001, lr=0.5, loss_freq=100)
```

```python
# Train acc
y_hat = logistic_output(X_train_sonar, w, b)
acc = accuracy(y_train_sonar, classify(y_hat))
print(f'Train accuracy: {100 * acc:.2f}%')
```

```python
# Test acc
y_hat = logistic_output(X_test_sonar, w, b)
acc = accuracy(y_test_sonar, classify(y_hat))
print(f'Test accuracy: {100*acc:.2f}%')
```

## 2.2. Penalized logistic regression

### Penalized binary cross-entropy loss

Logistic regression can overfit when there are too many parameters compared to training examples, as it can find weights for which the decision boundary perfectly separates all the training examples. When such overfitting occurs, the weights are often set to large values. One way to reduce such overfitting is to prevent weights from becoming so large, which can be done using L2 regularization, which consists of changing the training objective to penalize "large" weights. The name L2 regularization comes from the fact that weights are penalized using the L2-norm.

We will call this new training objective (new loss function) the penalized binary-cross entropy loss (or PBCE), it is expressed as:

$$
\begin{align}
\text{PBCE}(\mathbf{w}, b) &= \text{BCE}(\mathbf{w}, b) + \lambda \|\mathbf{w}\|_{2}^{2} \\
&= [- \frac{1}{N}\sum^{N}_{i=1} y^{(i)} \log(\hat{y}^{(i)}) + (1-y^{(i)}) \log(1- \hat{y}^{(i)}) \hspace{0.2cm}] + \lambda\|\mathbf{w}\|_{2}^{2}
\end{align}$$

As you can see, this loss function consists of adding a penalty term to BCE, which **penalizes the weights but not the bias term**. The hyper-parameter $\lambda$ controls overfitting. The larger its value, the more the weights are penalized for being large, which makes the model less flexible.

**Task: Implement `penalized_bce_loss()`.**

**Note:** `lambda` is a reserved keyword in Python (used for lambda expressions), so no variables can be named this way. In our function, we use `lambd` as a variable name instead.

```python
def penalized_bce_loss(X: np.ndarray,  y: np.ndarray, w: np.ndarray, b: float, lambd: float) -> float:
    """ Penalized binary cross-entropy loss function

    Args:
        X: Dataset of shape (N, D)
        y: Labels of shape (N, ).
        w: Weights of logistic regression model of shape (D, )
        b: bias, a scalar
        lambd: regularization coefficient (named this way as lambda is a reserved keyword in python)

    Returns:
        float: binary cross-entropy loss.
    """
    ### START CODE HERE ###
    ...
    ### END CODE HERE ###
    return loss
```

```python
X = X_train_sonar[:3]
y  = y_train_sonar[:3]
w = np.ones(X.shape[1])
b = 1

loss = penalized_bce_loss(X, y, w, b, lambd=0.1)
assert(isinstance(loss, float))
```

### Penalized binary cross-entropy loss gradient

In order to use the penalized binary cross-entropy loss, you need its gradient. This time, we won't give you the gradient formula, you'll need to figure it out yourself!

**Task: Figure out the gradient for the penalized binary cross-entropy loss and implement `penalized_bce_gradient()`.**

**Hint:** Here are two useful resources for matrix calculus:
- http://www.matrixcalculus.org/
- https://en.wikipedia.org/wiki/Matrix_calculus

```python
def penalized_bce_gradient(X: np.ndarray,
                           y: np.ndarray,
                           w: np.ndarray,
                           b: float,
                           lambd: float) -> Tuple[np.ndarray, float]:
    """ Gradient of the penalized binary-cross entropy loss

    Args:
        X: Dataset of shape (N, D)
        y: Labels of shape (N, )
        w: Weights of logistic regression model of shape (D, )
        b: bias, a scalar
        lambd: regularization coefficient (named this way as lambda is a reserved keyword in python)

    Returns:
        dw (np.ndarray) gradient of the loss with respect to w of shape (D, )
        db (float) gradient of the loss with respect to b, a scalar
    """
    ### START CODE HERE ###
    ...
    ### END CODE HERE ###

    return dw, db

```

```python
X = X_train_sonar[:3]
y  = y_train_sonar[:3]
w = np.ones(X.shape[1])
b = 1

dw, db = penalized_bce_gradient(X, y, w, b, lambd=0.1)
assert(dw.shape == w.shape)
assert(isinstance(db, float))
```

### Training

Now that you've implemented the loss function and its gradient, you can use it to train your model.

**Task: Implement `train_penalized_logistic_regression()`.**

**Hint:** This training function is very similar to the one implemented in Part 1.

```python
def train_penalized_logistic_regression(X: np.ndarray,
                                        y: np.ndarray,
                                        lambd: float,
                                        max_iters: int = 1001,
                                        lr: float = 0.5,
                                        loss_freq: int = 0) -> Tuple[np.ndarray, float, dict]:
    """ Training function for binary class penalized logistic regression using gradient descent

    Args:
        X: Dataset of shape (N, D).
        y: Labels of shape (N, ).
        lambd: regularization coefficient (named this way as lambda is a reserved keyword in python)
        max_iters: Maximum number of iterations.
        lr: The learning rate of  the gradient step.
        loss_freq : Prints the loss every `loss_freq` iterations.

    Returns:
        w: weights of shape (D, )
        b: scalar
        viz_d: dict used for visualizations
    """

    # Initialize weights
    np.random.seed(0)
    w = np.random.normal(0, 1, size=(X.shape[1], ))
    b = 0

    # Initialize dict with lists to keep track of loss, accuracy, weight and bias evolution
    logger = {'loss': [],
             'acc': [],
            }

    for i in range(max_iters):
        # Compute loss, dw, db and update w and b
        ### START CODE HERE ###
        ...
        ### END CODE HERE ###

        # Keep track of loss and accuracy values for each iteration
        logger['loss'].append(loss)

        y_hat = logistic_output(X, w, b)
        logger['acc'].append(accuracy(y, classify(y_hat)))

        if (loss_freq !=0) and i % loss_freq == 0:
            print(f'Loss at iter {i}: {loss:.5f}')

    if (loss_freq != 0):
        print('\nFinal loss: {:.5f}'.format(logger['loss'][-1]))

    return w, b, logger
```

Let's now the check that there are no huge mistakes in your implementation by training your model for two different values of $\lambda$ and checking the training accuracy. Given that regularization reduces overfitting, you should expect the training accuracy to decrease when $\lambda$ increases.

```python
w, b, _ = train_penalized_logistic_regression(X_train_sonar, y_train_sonar, lambd=0, max_iters=1001, lr=0.5, loss_freq=100)
y_hat = logistic_output(X_train_sonar, w, b)
acc = accuracy(y_train_sonar, classify(y_hat))
print(f'Train accuracy: {100*acc:.2f}%')
```

```python
w, b, _ = train_penalized_logistic_regression(X_train_sonar, y_train_sonar, lambd=0.1, max_iters=1001, lr=0.5, loss_freq=100)
y_hat = logistic_output(X_train_sonar, w, b)
acc = accuracy(y_train_sonar, classify(y_hat))
print(f'Train accuracy: {100*acc:.2f}%')
```

Now that you've implemented penalized logistic regression, you may wonder which value of $\lambda$ to pick. This is what we'll try to figure out in part 3, using cross-validation.

# Part 3: Cross-validation

In this class, you saw that a dataset is usually split into 3 parts: One training set, one validation set and one test set. The training set is used as training data, the validation set is used for tuning hyper-parameters and the test set is held out for final evaluation. However, by partitioning data this way, we reduce the number of samples available for training the model, and the results on the validation depend on a particular random choice for the training and validation sets. For datasets with a small amount of training examples, this can be especially problematic.

This is where cross-validation comes into play. With cross-validation, a test set will still be held out for final evaluation, but there is no need for a designated validation set. Here, you will implement a non-exhaustive cross-validation technique known as k-fold cross-validation.

## 3.1. k-Fold Cross-validation

In k-fold cross-validation, the training data is randomly partitioned into $k$ equal sized subsamples. Of the $k$ subsamples, a single subsample is used as the validation data, and the remaining $k − 1$ subsamples are used as training data. This process is repeated $k$ times, with each of the $k$ subsamples used exactly once as the validation data. The $k$ results are then averaged to produce a single estimation.

This process is illustrated below:

We have implemented the function `k_fold_indices()` for you, which generates indices for k-fold cross-validation. You can see its implementation and an example usage in the cell below.

```python
def k_fold_indices(num_examples: int, k: int = 4) -> List[Tuple[np.ndarray, np.ndarray]]:
    """Generates indices for k-fold cross-validation

    Args:
        num_examples: Number of training examples
        k: Number of folds

    Returns:
        List of tuples containing the training indices and validation indices

    """
    indices = np.arange(num_examples)
    split_size = num_examples // k
    val_indices = [indices[k * split_size : (k + 1) * split_size] for k in range(k)]
    both_indices = [(np.delete(indices, val_ind), val_ind) for val_ind in val_indices]
    return both_indices

# Example usage
for train_index, val_index in k_fold_indices(num_examples=8, k=4):
    # Do something with the indices
    print(f"{train_index} {val_index}\n")
```

**Task: With the help of `k_fold_indices()` and  previously implemented functions, implement the function `cross_val_penalized_logistic_regression()` according to its documentation.**

```python
def cross_val_penalized_logistic_regression(X: np.ndarray,
                                            y: np.ndarray,
                                            lambd: float = 0,
                                            max_iters: int = 1001,
                                            lr: float = 0.5,
                                            loss_freq: int = 0,
                                            k: int = 4) -> float:
    """
    Performs k-fold cross-validation for penalized logistic regression and returns the mean validation accuracy

    Args:
        X: Dataset of shape (N, D).
        y: Labels of shape (N, ).
        lambd: regularization coefficient (named this way as lambda is a reserved keyword in python)
        max_iters: Maximum number of iterations.
        lr: The learning rate of  the gradient step.
        loss_freq : Prints the loss every `loss_freq` iterations.
        k: Number of folds

    Returns:
        Mean validation accuracy

    """
    val_accs = []

    # Hint: Use a for-loop to iterate over all k-fold indices
    ### START CODE HERE ###
    ...
    ### END CODE HERE ###

    return np.mean(val_accs)
```

```python
mean_cv_acc = cross_val_penalized_logistic_regression(X_train_sonar, y_train_sonar)
print(f"Mean CV acc for default settings: {mean_cv_acc}")
```

## 3.2. Finding a good regularization parameter

You'll now use cross-validation to find a good $\lambda$ for penalized logistic regression.

**Task: Find the best value of $\lambda$**

To do so:
- Suppose that `max_iters` and `lr` are set to the **default values** of `cross_val_penalized_logistic_regression()`, and that you can only modify the hyper-parameter `lambd`.
- Compute the 4-fold cross-validation accuracy for $\lambda \in \{0, 0.001, 0.01, 0.1, 1\}.$
- Set `best_cv_acc` to the best cross-validation accuracy obtained, and `best_lambda` to the $\lambda$ associated with the best cross-validation accuracy.

```python
# Only search for values in this list, do not modify  it
lambdas = [0, 0.001, 0.01, 0.1, 1]
k = 4

# cv_accs should contain the mean cross-validation accuracy for each value of lambda
cv_accs = []

### START CODE HERE ###
...
### END CODE HERE ###

for lambd, acc in zip(lambdas, cv_accs):
    print(f"Lambda: {lambd}")
    print(f"Cross-val acc: {acc}")
    print()

### START CODE HERE ###
...
### END CODE HERE ###
```

```python
print(f"Best lambda: {best_lambda}")
print(f"Best CV acc: {best_cv_acc}")
```

Now that you've settled on a value for $\lambda$, you can use it to train our model using our entire training set, and find out how well it performs on the test set.

```python
w, b, _ = train_penalized_logistic_regression(X_train_sonar, y_train_sonar, lambd=best_lambda, max_iters=1001, lr=0.5, loss_freq=100)

# Test acc
print()
y_hat = logistic_output(X_test_sonar, w, b)
acc = accuracy(y_test_sonar, classify(y_hat))
print(f'Test accuracy: {100*acc:.2f}%')
```

Assuming that penalized logistic regression, cross-validation and the hyperparameter search were implemented correctly, your test accuracy should be slightly higher than what was obtained for standard logistic regression in Part 2.1. While this is a modest increase, regularization can lead to much more significant increases in accuracy when working with complex machine learning models such as neural networks.

# Part 4: Evaluation metrics

Accuracy is not the only metric which can be used to measure how well your model is doing. In fact, accuracy can sometimes be a very poor metric to judge model performance. For example, suppose that you are building a machine learning model to detect whether or not someone has a disease. This disease is quite rare, so you estimate that only 1% of the people tested actually have the disease. In such a scenario, a model that always classifies a patient as negative would have an accuracy of 99%, and yet that model would be completely useless (and potentially very harmful), as it would fail to detect any positive case.

You'll now implement other metrics, which are usually preferable over accuracy when working on class-imbalanced datasets.

In this section, always suppose that:
- a ground truth label equal to **1** corresponds to a **positive example**
- a ground truth label equal to **0** corresponds to a **negative example**

Refer to this confusion matrix when implementing metrics:

**Confusion Matrix**

|                    | Actual Positive (1)    | Actual Negative (0)    |
|--------------------|---------------------|---------------------|
| **Predicted Positive (1)** | True Positive (TP)  | False Positive (FP) |
| **Predicted Negative (0)** | False Negative (FN) | True Negative (TN)  |

## 4.1. Precision

Precision attempts to answer the following question: *What proportion of positive identifications was actually correct?*

It is defined as: $$\text{Precision} = \frac{TP}{TP + FP}$$

**Task: Implement `precision()`**

```python
def precision(labels_gt: np.ndarray, labels_pred: np.ndarray) -> float:
    """Computes precision.

    Args:
        labels_gt: labels (ground-truth) of shape (M, ).
        labels_pred: Predicted labels of shape (M, ).

    Returns:
        float: Precision, in range [0, 1].
    """

    ### START CODE HERE ###
    ...
    ### END CODE HERE ###
```

```python
# Check that output is in [0, 1] (ensures that output is not a percentage)
assert 0.0 <= precision(np.array([1, 0]), np.array([1, 1])) <= 1.0
```

## 4.2. Recall

Recall attempts to answer the following question: *What proportion of actual positives was identified correctly?*

It is defined as: $$\text{Recall} = \frac{TP}{TP + FN}$$

**Task: Implement `recall()`**

```python
def recall(labels_gt: np.ndarray, labels_pred: np.ndarray) -> float:
    """Computes recall.

    Args:
        labels_gt: labels (ground-truth) of shape (M, ).
        labels_pred: Predicted labels of shape (M, ).

    Returns:
        float: Recall, in range [0, 1].
    """

    ### START CODE HERE ###
    ...
    ### END CODE HERE ###

```

```python
# Check that output is in [0, 1] (ensures that output is not a percentage)
assert 0.0 <= recall(np.array([1, 0]), np.array([1, 1])) <= 1.0
```

## 4.3. F1-score

The F1-score (or F-measure) is the harmonic mean of precision and recall:

$$F_1 = 2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} +  \text{recall}}$$

**Task: Implement `f1_score()`**

```python
   def f1_score(labels_gt: np.ndarray, labels_pred: np.ndarray) -> float:
    """Computes the F1-score.

    Args:
        labels_gt: labels (ground-truth) of shape (M, ).
        labels_pred: Predicted labels of shape (M, ).

    Returns:
        float: F1 Score, in range [0, 1].
    """

    ### START CODE HERE ###
    ...
    ### END CODE HERE ###
```

```python
# Check that output is in [0, 1] (ensures that output is not a percentage)
assert 0.0 <= f1_score(np.array([1, 0]), np.array([1, 1])) <= 1.0
```

## 4.4 Balanced Accuracy

The balanced accuracy is calculated as the average of the proportion of correctly classified examples for each class, it can be a good alternative to accuracy for heavily imbalanced datasets. It is defined as: $$\text{Balanced accuracy} = \frac{TPR + TNR}{2} = \frac{\frac{TP}{TP + FN} + \frac{TN}{TN + FP}}{2}$$

**Task: Implement `balanced_accuracy()`**

```python
def balanced_accuracy(labels_gt: np.ndarray, labels_pred: np.ndarray) -> float:
    """Computes the balanced accuracy

    Args:
        labels_gt: labels (ground-truth) of shape (M, ).
        labels_pred: Predicted labels of shape (M, ).

    Returns:
        float: Balanced accuracy, in range [0, 1].
    """
    ### START CODE HERE ###
    ...
    ### END CODE HERE ###
```

```python
# Check that output is in [0, 1] (ensures that output is not a percentage)
assert 0.0 <= balanced_accuracy(np.array([1, 0]), np.array([1, 1])) <= 1.0
```

## 4.5. Model evaluation

```python
print("Performance of penalized logistic regression:")
# Acc
acc = accuracy(y_test_sonar, classify(y_hat))
print(f'Test accuracy: {100 * acc:.2f}%')
# Precision
prec = precision(y_test_sonar, classify(y_hat))
print(f'Test precision: {100 * prec:.2f}%')
# Recall
rec = recall(y_test_sonar, classify(y_hat))
print(f'Test recall: {100 * rec:.2f}%')
# F1-Score
f1 = f1_score(y_test_sonar, classify(y_hat))
print(f'Test F1-score: {100 * f1:.2f}%')
# Balanced accuracy
balanced_acc = balanced_accuracy(y_test_sonar, classify(y_hat))
print(f'Test balanced accuracy: {100 * f1:.2f}%')
```

When using penalized logistic regression on the sonar dataset, all these metrics should give relatively similar results. The main reason for this is that this dataset is relatively well-balanced, and a lot of the implemented metrics are mostly very useful on class-imbalanced datasets.

Whew, you reached the end of this homework! The only remaining step is to submit it, so make sure to read the next section carefully.

# Submitting your homework

### Restarting the kernel

Before submitting, make sure that there are no [hidden states](https://github.com/vita-epfl/introML-2021/blob/main/exercises/00-setup/jupyter.md#the-kernel) by restarting your kernel and running all cells. Check that the code runs without errors.

```python
# Restart your kernel and run all cells, make sure that you can reach this cell
print("Ran all cells :)")
```

### Renaming this notebook

Rename this notebook by preprending your SCIPER (name it **SCIPER_logistic_regression.ipynb**). To do so, right click on the notebook in the left sidebar and select **"Rename"**.

For example, if your SCIPER is 123456, then name your notebook *123456_logistic_regression.ipynb* .

### Submitting on Moodle

Finally, make sure to save your notebook (File -> Save Notebook), so that it has your most recent changes. Then:
- If you are working on a **local** environment, you can directly upload this notebook (i.e. the file you are currently working on) to Moodle.
- If you are using a **cloud-based** environment such as **EPFL Noto**, you'll first need to get a local copy by right clicking on the notebook in the sidebar and clicking on **"Download"**, then upload the downloaded notebook to Moodle.


```python
%reset -f
%matplotlib notebook
from sympy.interactive import printing
printing.init_printing(use_latex=True)
import matplotlib.pyplot as plt
from sympy import *
import sympy as sp
import numpy as np

V, Δt, d, D, ω_v, ω_Δt = symbols('V Δt d D ω_v ω_Δt')

Q = V/Δt

ω_Q = sqrt((Q.diff(V)*ω_v)**2+(Q.diff(Δt)*ω_Δt)**2)

Q + ω_Q, Q - ω_Q.subs([(V,1),(Δt,2)])
```

**2) Sabe-se, pelas medições, que $\frac{ω_v}{V}$ = 5% e $\frac{ω_{Δt}}{Δt}$ = 1%, calcule $\frac{ω_Q}{Q}$.**

A partir das respostas do exercício anterior, pela lei do produtório, tem-se que:

```python

```

**Calculadoras**

```python
#Calculadora de incerteza:

%reset -f
from uncertainties import ufloat
from math import pi

d = ufloat(3.906, 0.084, "diâmetro")
h = ufloat(6.119, 0.019, "altura")
t = ufloat(84.80, 0.2, "tempo")

vazao = pi*h*d**2/(4*t)

print(vazao.format('.6f'))
```

    0.864647+/-0.037342

```python
#Calculadora t de Student:

%reset -f
import scipy.stats

conf = 0.9
σ = 1.180411478

def t(alpha, gl):
    return scipy.stats.t.ppf(1-(alpha/2), gl)

print(t(1 - conf,13)*σ)
```

    2.0904301073963363

```python
%reset -f
from CoolProp.CoolProp import PropsSI
from uncertainties import ufloat

P = 96.550 #kPa
w_P = 0.150
T = 28.5 + 273.15 #K
w_T = 0.5

R = 8.31451
MM = PropsSI('M','P', P*1000, 'T', T, 'Air')*1000

Px = ufloat(P, w_P)
Tx = ufloat(T, w_T)

ρx = Px*MM/(R*Tx)

print(ρx.format('.6f'))
```

    1.115046+/-0.002533


# Problem Set 2 : Extra problems
## Learning from data [TIF285], Chalmers, Fall 2019

Last revised: 30-Sep-2019 by Christian Forssén [christian.forssen@chalmers.se]

## Instructions

- See deadline on the course web page
- This problem set is performed individually (but collaboration is encouraged) and contains a number of basic and extra problems; you can choose which and how many to work on.
- See examination rules on the course web page.
- Hand-in is performed through the following **two** actions:
  - Upload of your solution in the form of a jupyter notebook, or python code, via Canvas.
  - Answer the corresponding questions on OpenTA.

  Note that the hand-in is not complete, and will not be graded, if any of those actions is not performed.

Fill your personal details
- Name: **Lastname, Firstname**
- Personnummer: **YYMMDD-XXXX**
  <br/>
  (civic registration number)

## Problem 5 (extra): Good-data, bad-data
### Task (a): 3 extra points; Task (b): 1 extra point

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import seaborn; seaborn.set("talk") # for plot formatting
```

Let's start by defining some data that we will fit with a straight line.  The following data is measured velocities and distances for a set of galaxies. We will assume that there is a constant standard deviation of $\sigma = 200$ km/sec on the $y$ values and no error on $x$.

```python
# Data from student lab observations;
# x = Galaxy distances in MPc
# y = Galaxy velocity in km/sec
x = np.array([10.1 ,45.2, 19.7, 31.2, 31.9, 44.0,
       14.9, 35.1,  39.9  ])
y = np.array([1507.9, 2937.5,  930.4, 2037.1, 2131.1,
       2795.6, 1061.8, 2464.8,
       1971.1])
```

### Task (a)

The question that we will be asking is:
> What value would you infer for the Hubble constant, i.e. the slope of the velocity versus distance relation, given this data?

We will make the prior assumption that the data can be fitted with a straight line. But we note that we are actually not interested in the offset of the straight line, but just its slope.

We will try three different approaches:
1. Maximum likelihood estimate
1. Full Bayesian analysis
1. Full Bayesian analysis incorporating a fully Bayesian approach to good-data-bad-data

As a subsequent task, we will also explore how the posterior belief from this analysis can feed into a second data analysis.

#### Statistical model
Here we are given data with simple error bars (constant $\sigma$), which implies that the probability for any *single* data point is a normal distribution about the true value. That is,

$$
y_i \sim \mathcal{N}(y_M(x_i;\theta), \sigma),
$$

with $y_M(x) = mx + b$.
Or, in other words,

$$
p(x_i,y_i\mid\theta) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(\frac{-\left[y_i - y_M(x_i;\theta)\right]^2}{2\sigma^2}\right)
$$

where $\sigma$ is the (known) measurement error indicated by the error bars.

#### Step 1: Maximum likelihood estimate

```python
# Insert code here
#
```

#### Step 2: Bayesian analysis
For the Bayesian analysis we would recommend a Gaussian prior (mean=0, standard deviation=200) for the intercept, and a symmetric prior
$$
p(m|I) \propto \frac{1}{(1+m^2)^{3/2}}
$$
for the slope (the latter was used also in the fitting straight line example).

```python
# Insert your code here
#
```

#### Step 3: Bayesian Approach to Outliers (good-data/bad-data)

There are several Bayesian approaches to accounting for outliers. They generally involve *modifying the statistical model*. For this data, it is abundantly clear that a simple straight line is not a good fit to our data. So let's propose a more complicated model that has the flexibility to account for outliers where the experimental error bar might have been underestimated.

Specifically we will assume that there is a chance that the error distribution is represented by a Lorentzian distribution rather than a Gaussian. The peaks of these two distributions are very similar in shape, but the Lorentzian has much heavier tails.

We will use a statistical model in which we allow each individual data point $(x_i,y_i)$ to be described by
a mixture between a Gaussian and a Lorentzian error model:

$$
\begin{array}{ll}
p(\{x_i\}, \{y_i\},\{e_i\}~|~\theta,\{g_i\},\sigma) = & g_i \frac{1}{\sqrt{2\pi e_i^2}}\exp\left[\frac{-\left(y_M(x_i~|~\theta) - y_i\right)^2}{2e_i^2}\right] \\
&+ (1 - g_i) \frac{1}{\sqrt{2}\pi \sigma} \frac{1}{1 + \left[ y_M (x_i~|~\theta) - y_i\right]^2 / (2\sigma^2)}
\end{array}
$$

What we've done is expanded our model with some nuisance parameters: $\{g_i\}$ is a series of weights which range from 0 to 1 and encode for each point $i$ the degree to which it fits the different error models.

Our model is much more complicated now: it has one new parameter per data point, but these can be considered nuisance parameters that can be marginalized-out in the end.  You can use a uniform prior $U(0,1)$ for these parameters. Your task is to implement this likelihood and to use, e.g., the [emcee](http://dan.iel.fm/emcee/current/) MCMC package to explore the parameter space.

### Task (b): Error propagation

The Bayesian approach offers a straight-forward approach for dealing with (known) systematic uncertainties; namely by marginalization.

The Hubble constant acts as a galactic ruler as it is used to measure astronomical distances according to $v = H_0 x$. An error in this ruler will therefore correspond to a systematic uncertainty in such measurements.

Suppose that a particular galaxy has a measured recessional velocity $v_\mathrm{measured} = (100 \pm 5) \times 10^3$ km/sec. Also assume that the Hubble constant $H_0$ is known from the analysis performed above in Step 3. Determine the posterior pdf for the distance to the galaxy assuming:
1. A fixed value of $H_0$ corresponding to the posterior mean of the previous analysis.
1. Using the full, sampled posterior pdf for $H_0$ from the same analysis.

In this analysis we will set the intercept $m=0$ so that $v_\mathrm{theory} = H_0 x$.

```python
vm=100000
sig_vm=5000
```

We assume that we can write
$$
v_\mathrm{measured} = v_\mathrm{theory} + \delta v_\mathrm{exp},
$$
where $v_\mathrm{theory}$ is the recessional velocity according to our model, and $\delta v_\mathrm{exp}$ represents the noise component of the measurement. We that $\delta v_\mathrm{exp}$ can be described by a Gaussian pdf with mean 0 and standard deviation $\sigma_v = 5 \times 10^3$ km/sec. Note that we have also assumed that our model is perfect, i.e. $\delta v_\mathrm{theory}$ is negligible.

In the following, we also assume that the error in the measurement in $v$ is uncorrelated with the uncertainty in $H_0$.

Through application of Bayes' rule we can readily evaluate the posterior pdf $p(x|D,I)$ for the distance $x$ to the galaxy.

#### Case 1: Fixed $H_0$

\begin{align}
p(x | D,I) & \propto p(D | x, I) p(x|I) \\
& = \frac{1}{\sqrt{2\pi}\sigma_v} \exp \left( - \frac{(v_\mathrm{measured} - v_\mathrm{theory})^2}{2\sigma_v^2} \right) p(x|I)\\
&= \left\{ \begin{array}{ll} \frac{1}{\sqrt{2\pi}\sigma_v} \exp \left( - \frac{(v_\mathrm{measured} - H_0 x)^2}{2\sigma_v^2} \right) & \text{with }x \in [x_\mathrm{min},x_\mathrm{max}] \\
0 & \text{otherwise},
\end{array} \right.
\end{align}

where $p(x|I)$ is the prior for the distance which we have assumed to be uniform, i.e. $p(x|I) \propto 1$ in some (possibly large) region $[x_\mathrm{min},x_\mathrm{max}]$.

#### Case 2: Using the inferred pdf for $H_0$

Here we use marginalization to obtain the desired posterior pdf $p(x|D,I)$ from the joint distribution of $p(x,H_0|D,I)$
$$
p(x|D,I) = \int_{-\infty}^\infty dH_0 p(x,H_0|D,I).
$$

Using Bayes' rule, the product rule, and the fact that $H_0$ is independent of $x$ we find that
$$
p(x|D,I) \propto p(x|I) \int dH_0 p(H_0|I) p(D|x,H_0,I),
$$
which means that we have expressed the quantity that we want (the posterior for $x$) in terms of quantities that we know.

The pdf $p(H_0 | I)$ is known via its $N$ samples $\{H_{i}\}$ generated by the MCMC sampler.

This means that we can approximate
$$
p(x |D,I) = \int dH_0 p(H_0|I) p(D|x,H_0,I) \approx \frac{1}{N} \sum_{i=1}^N p(D | x, H_i, I)
$$
where we have used $p(x|I) \propto 1$.

```python

```

## Problem 6 (extra): Auto-correlation time of the MCMC sampler
### (2 extra points)

A challenge when doing MCMC sampling is that the collected samples can be *correlated*. This can be tested by computing the *autocorrelation function* and extacting the correlation time.

We can define the autocorrelation function $\rho(h)$ from the list of samples $X$. Say that $X_t$ is sample $t$ and $X_{t+h}$ is a shifted version of $X_t$ with $h$ being called the *lag*
$$
\rho(h) = \frac{\sum_\mathrm{overlap} \left[ (X_t - \bar{X}) (X_{t+h} - \bar{X})\right]}
{\sqrt{ \sum_\mathrm{overlap} (X_t - \bar{X})^2 } \sqrt{ \sum_\mathrm{overlap} (X_{t+h} - \bar{X})^2 }}
$$
The summation is carried out over the subset of samples that overlap.

* Try to understand what the autocorrelation time measures and why it is large (small) when the samples are correlated (not correlated).

* It is often observed that $\rho(h)$ is roughly exponential so that we can define an autocorrelation time $\tau$ according to
$$
\rho(h) \sim \exp(-h/\tau).
$$

#### Task
Repeat the MCMC sampling of a Lorentzian pdf using the random walk Metropolis algorithm from Problem 4.

Extract both the autocorrelation time and the acceptance ratio $\tau$ for some different choices of the proposal width; e.g. 0.1, 0.5, 1.0, 2.0. What is a good choice for sampling this distribution?

```python

```


# Under- and overfitting, model selection

## Preliminaries

In the first set of exercises you had to implement the training and evaluation of the linear regression and $k$-NN methods from scratch in order to practice your `numpy` skills. From this set of exercises onward, you can use the implementations provided in `scikit-learn` or other higher-level libraries. We start this set of exercises by demonstrating some of the features of `scikit-learn`.

For example, implementation of linear regression model fitting with an analytical solution for the parameters is provided by the class `sklearn.linar_model.LinearRegression`. You can train a linear regression model in the following way:

```python
import numpy as np
from sklearn import datasets, linear_model

# load the diabetes dataset
diabetes = datasets.load_diabetes()

# use only one feature
X = diabetes.data[:, np.newaxis, 2]
y = diabetes.target

# split the data into training/testing sets
X_train = X[:-20]
X_test = X[-20:]

# split the targets into training/testing sets
y_train = y[:-20]
y_test = y[-20:]

# create linear regression object
model = linear_model.LinearRegression()

# train the model using the training dataset
model.fit(X_train, y_train)
```

    LinearRegression()

Let's visualize the training dataset and the learned regression model.

```python
%matplotlib inline
import matplotlib.pyplot as plt

fig = plt.figure()
plt.plot(X_train, y_train, 'r.', markersize=12)
X_edge = np.array([np.min(X_train, 0), np.max(X_train, 0)])
plt.plot(X_edge, model.predict(X_edge), 'b-')
plt.legend(('Data', 'Linear regression'), loc='lower right')
plt.title('Linear regression')
plt.xlabel('$x$')
plt.ylabel('$y$')
plt.show()
```

Once trained, the model can be used to make predictions on the test data:

```python
# Make predictions using the testing dataset
prediction = model.predict(X_test)
```

The next step (not shown here) is to evaluate the performance of the trained model.

Note that the `scikit-learn` interface works by first initializing an object from the class that implements the machine learning model (linear regression in this case) and then fitting the initialized model using the data in the training set. Finally, the trained (fitted) model can be used to make predictions on unseen data. In fact, all models implemented in this library follow the same *initialize-fit-predict* programming interface. For example, a $k$-NN classifier can be trained in the following way:

```python
from sklearn.model_selection import train_test_split
from sklearn import datasets, neighbors

breast_cancer = datasets.load_breast_cancer()

X = breast_cancer.data
y = breast_cancer.target

# make use of the train_test_split() utility function instead
# of manually dividing the data
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=40)

# initialize a 3-NN classifier
model = neighbors.KNeighborsClassifier(n_neighbors=3)

# train the model using the training dataset
model.fit(X_train, y_train)

# make predictions using the testing dataset
prediction = model.predict(X_test)
```

Note that the features in the breast cancer dataset have different scales (some have on average very small absolute values, and some very large), which means that the distance metric used by $k$-NN will me dominated by the features with large values. You can use any of the number of feature transformation methods implemented in `scikit-learn` to scale the features. For example, you can use the `sklearn.preprocessing.StandardScaler` method to transform all features to a have a zero mean and unit variance:

```python
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
scaler.fit(X_train)

X_train_scaled = scaler.transform(X_train)
```

The scaler has its own parameters which are the means and standard deviations of the features estimated from the training set. If you train a model with the scaled features, you will have to remember to also apply the scaling transformation every time you make a prediction on new unseen and unscaled data. This is somewhat prone to error. One option for making the code more robust is to create a processing pipeline that includes the scaling and $k$-NN models in a sequence:

```python
from sklearn.pipeline import Pipeline

knn = neighbors.KNeighborsClassifier(n_neighbors=3)

model = Pipeline([
                 ("scaler", scaler),
                 ("knn", knn)
                ])

# train the model using the training dataset
model.fit(X_train, y_train)

# make predictions using the testing dataset
prediction = model.predict(X_test)

```

If you are curious, more information about the design of the `scikit-learn` application programming interface (API) can be found [in this paper](https://arxiv.org/pdf/1309.0238.pdf).

## Exercises

### Bias-variance decomposition

Show that the mean squared error of the estimate of a parameter can be decomposed into an expression that includes both the bias and variance (Eq. 5.53-5.54 in "Deep learning" by Goodfellow et al.).

First defining the bias as:

\begin{align}
Bias(\hat{\theta}_{m}) &\overset{\textrm{def}} = \textrm{E}\left[ \hat{\theta}_{m} \right] - \theta
\label{bias}
\tag{1}
\end{align}

Then defining the variance as:

\begin{align}
Var(\hat{\theta}_{m}) &\overset{\textrm{def}}= \textrm{E}[(\hat{\theta}_{m} - \textrm{E}[\hat{\theta}_{m}])^{2}]\nonumber
\end{align}

And rewriting the variance by virtue of linearity of expected value at $*$, to:

\begin{align}
Var(\hat{\theta}_{m}) &= \textrm{E}\left[(\hat{\theta}_{m} - \textrm{E}[\hat{\theta}_{m}])^{2}\right]]\nonumber \\
&= \textrm{E}\left[\hat{\theta}_{m}^{2} -2\hat{\theta}_{m}\textrm{E}[\hat{\theta}_{m}] + \textrm{E}[\hat{\theta}_{m}]^{2}\right]\nonumber \\
&\overset{*}= \textrm{E}\left[\hat{\theta}_{m}^{2}\right] - \textrm{E}\left[2\hat{\theta}_{m}\textrm{E}[\hat{\theta}_{m}]\right] + \textrm{E}\left[\textrm{E}[\hat{\theta}_{m}]^{2}\right]\nonumber \\
&\overset{*}= \textrm{E}\left[\hat{\theta}_{m}^{2}\right] - 2\textrm{E}\left[\hat{\theta}_{m}\right]\textrm{E}\left[\hat{\theta}_{m}\right] + E\left[\hat{\theta}_{m}\right]^{2}\nonumber \\
&= \textrm{E}\left[\hat{\theta}_{m}^{2}\right] - 2\textrm{E}\left[\hat{\theta}_{m}\right]^{2} + \textrm{E}\left[\hat{\theta}_{m}\right]^{2}\nonumber \\
&= \textrm{E}\left[\hat{\theta}_{m}^{2}\right] - \textrm{E}\left[\hat{\theta}_{m}\right]^{2}
\label{variance}
\tag{2}
\end{align}

And last, defining the MSE and rewriting using linearity of expected value at $*$ and substitution at \eqref{bias} and \eqref{variance}:

\begin{align}
MSE &\overset{\textrm{def}}= \textrm{E}\left[(\hat{\theta}_{m} - \theta)^{2}\right]\nonumber \\
    &= \textrm{E}\left[\hat{\theta}_{m}^{2} - 2\hat{\theta}_{m}\theta + \theta^{2}\right]\nonumber \\
    &\overset{*}= \textrm{E}\left[\hat{\theta}_{m}^{2}\right] - \textrm{E}\left[2\hat{\theta}_{m}\theta\right] + \textrm{E}\left[\theta^{2}\right]\nonumber \\
    &\overset{*}= \textrm{E}\left[\hat{\theta}_{m}^{2}\right] - 2\theta\textrm{E}\left[\hat{\theta}_{m}\right] + \theta^{2}\nonumber \\
    &= \textrm{E}\left[\hat{\theta}_{m}^{2}\right] - 2\theta\textrm{E}\left[\hat{\theta}_{m}\right] + \theta^{2}\nonumber \\
    &= \textrm{E}\left[\hat{\theta}_{m}^{2}\right] - \textrm{E}\left[\hat{\theta}_{m}\right]^{2} + \textrm{E}\left[\hat{\theta}_{m}\right]^{2} - 2\theta\textrm{E}\left[\hat{\theta}_{m}\right] + \theta^{2}\nonumber \\
    &\overset{(1)}= Var(\hat{\theta}_{m}) + \textrm{E}\left[\hat{\theta}_{m}\right]^{2} - 2\theta\textrm{E}\left[\hat{\theta}_{m}\right] + \theta^{2}\nonumber \\
    &= Var(\hat{\theta}_{m}) + (\textrm{E}\left[\hat{\theta}_{m}^{2}\right] - \textrm{E}\left[\hat{\theta}_{m}\right]^{2})^2\nonumber \\
    &\overset{(2)}= Var(\hat{\theta}_{m}) + Bias(\hat{\theta}_{m})^{2}\nonumber
\end{align}

### Polynomial regression

For this exercise we will be using generated data to better show the effects of the different polynomial orders.
The data is created using the make_polynomial_regression function.

```python
%matplotlib inline

def generate_dataset(n=100, degree=1, noise=1, factors=None):
    # Generates a dataset by adding random noise to a randomly
    # generated polynomial function.

    x = np.random.uniform(low=-1, high=1, size=n)

    factors = np.random.uniform(0, 10, degree+1)

    y = np.zeros(x.shape)

    for idx in range(degree+1):
        y += factors[idx] * (x ** idx)

    # add noise
    y += np.random.normal(-noise, noise, n)

    return x, y

# load generated data
np.random.seed(0)

X, y = generate_dataset(n=100, degree=4, noise=1.5)

plt.plot(X, y, 'r.', markersize=12)
```

Implement polynomial regression using the `sklearn.preprocessing.PolynomialFeatures` transformation. Using the `sklearn.grid_search.GridSearchCV` class, perform a grid search of the polynomial order hyperparameter space with cross-validation and report the performance on an independent test set.

Plot a learning curve that show the validation accuracy as a function of the polynomial order.

Which models have a high bias, and which models have high variance? Motivate your answer.

Repeat this experiment, this time using the diabetes dataset instead of the generated data.

```python
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.model_selection import GridSearchCV, validation_curve, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error

# add an extra dimension to the array
X = X[:, np.newaxis]
y = y[:, np.newaxis]

#split training and test data.
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=33)
```

```python
# the range of polynominal degrees. 1-20
degrees = np.arange(1, 20)

# A parameter library with settings
param_grid = {'polynomialfeatures__degree': degrees,
              'linearregression__fit_intercept': [True, False],
              'linearregression__normalize': [True, False]}

# Function to make a pipeline for polynomial regression
def PolynomialRegression(degree=2, **kwargs):
    return make_pipeline(PolynomialFeatures(degree), LinearRegression(**kwargs))

# Setup GridSearchCV
poly_grid = GridSearchCV(PolynomialRegression(), param_grid,
                         cv=10,
                         scoring='neg_mean_squared_error',
                         verbose=1)

# fit with all degrees
poly_grid = poly_grid.fit(X_train, y_train)

# Select model with best degree
model = poly_grid.best_estimator_
print(model)

# find train scores and validation scores
train_scores, validation_scores = validation_curve(
    model, X_train, y_train,
    param_name='polynomialfeatures__degree',
    param_range=degrees, cv = 10, scoring = 'neg_mean_squared_error')

# predicts the test data with the best found degree
prediction = model.predict(X_test)

# error calculation using mean squared error
mse = mean_squared_error(y_test, prediction)
print("Independent test set MSE: "+ str(mse))

# evaluates score by cross-validation
scores = cross_val_score(model, X, y,
                scoring="neg_mean_squared_error", cv=10)
print('Cross-validation MSE: ' + str(-scores.mean()))

#plot figure cross validation
plt.figure()
plt.plot(degrees, -validation_scores.mean(axis=1), lw=2,
         label='cross-validation')
plt.plot(degrees, -train_scores.mean(axis=1), lw=2, label='training')

plt.legend(loc='best')
plt.xlabel('degree of fit')
plt.ylabel('mean squared error')
plt.title('Validation curve')

#plot figure polynomial regression model on test data
plt.figure()
plt.plot(X_train, y_train, 'r.', markersize=12)
X_edge = np.arange(np.min(X_train, 0), np.max(X_train, 0), 0.01)
plt.plot(X_edge, model.predict(X_edge[:,np.newaxis]), 'b-')
plt.legend(('Data', 'Polynomial regression'))
plt.title('Polynomial regression')
plt.xlabel('$x$')
plt.ylabel('$y$')
plt.show()
```

```python
# selected degrees to plot
degrees = [1, 4, 16]

# for loop for plotting the degrees
plt.figure(figsize=(14, 5))
for i in range(len(degrees)):
    ax = plt.subplot(1, len(degrees), i + 1)

    # fitting the model with the selected degree
    model = PolynomialRegression(degree=degrees[i])
    model.fit(X, y)

    # Evaluating the score of the selected model
    scores = cross_val_score(model, X, y,
                             scoring="neg_mean_squared_error", cv=10)

    # plotting the results
    plt.plot(X_train, y_train, 'r.', markersize=12)
    X_edge = np.arange(np.min(X_train, 0), np.max(X_train, 0), 0.01)
    plt.plot(X_edge, model.predict(X_edge[:,np.newaxis]), 'b-')
    plt.legend(('Data', 'Polynomial regression'))
    plt.title('Polynomial regression')
    plt.xlabel('$x$')
    plt.ylabel('$y$')
    plt.title("Degree {}\nMSE = {:.2e}".format(
        degrees[i], -scores.mean()))
plt.show()
```

The models with a degree less than 2 have a high bias. This is seen in the validation curve where the MSE is relatively high of both the training and validation set. Models using a polynomial degree that's 13 or higher show signs of overfitting. This is also seen in the validation curve and visualized in the plots with the data and polynomial regression. These models have a high variance, since the error on the training data is low but the model does not generalize on new data. Meaning that the error on the validation set is high.

#### Diabetes Dataset

```python
# set up diabetes dataset
X = diabetes.data
y = diabetes.target[:, np.newaxis]
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=33)
```

```python
# the range of polynominal degrees. 1-5
degrees = np.arange(1, 5)

# A parameter library with settings
param_grid = {'polynomialfeatures__degree': degrees,
              'linearregression__fit_intercept': [True, False],
              'linearregression__normalize': [True, False]}

# Function to make a pipeline for polynomial regression
def PolynomialRegression(degree=2, **kwargs):
    return make_pipeline(PolynomialFeatures(degree), LinearRegression(**kwargs))

# Setup GridSearchCV
poly_grid = GridSearchCV(PolynomialRegression(), param_grid,
                         cv=10,
                         scoring='neg_mean_squared_error',
                         verbose=1)
# fit with all degrees
poly_grid = poly_grid.fit(X_train, y_train)

# Select model with best degree
model = poly_grid.best_estimator_
print(model)

# find train scores and validation scores
train_scores, validation_scores = validation_curve(
    model, X, y,
    param_name='polynomialfeatures__degree',
    param_range=degrees, cv = 10, scoring = 'neg_mean_squared_error')

# predicts the test data with the best found degree
prediction = model.predict(X_test)
mse = mean_squared_error(y_test, prediction)
print("Independent test set MSE: "+ str(mse))

# error calculation using mean squared error
scores = cross_val_score(model, X, y,
                scoring="neg_mean_squared_error", cv=10)
print('Cross-validation MSE: ' + str(-scores.mean()))

# plot validation curve
plt.figure()
plt.plot(degrees, -validation_scores.mean(axis=1), lw=2,
         label='cross-validation')
plt.plot(degrees, -train_scores.mean(axis=1), lw=2, label='training')

plt.legend(loc='best')
plt.xlabel('degree of fit')
plt.ylabel('mean squared error')
plt.title('Validation curve')
```

### ROC curve analysis
A common method to evaluate binary classifiers is the receiver operating characteristic (ROC) curve. Similar to the week one practicals, implement a $k$-NN classifier on the breast cancer dataset, however, this time use the $k$-NN pipeline from the preliminary. Train the model for different values of $k$ and evaluate their respective performance with an ROC curve, use the `sklearn.metrics.roc_curve` function.

```python
from sklearn.metrics import roc_curve,auc

# load breast cancer data
X = breast_cancer.data
y = breast_cancer.target

X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=40)

k_neighbors=list(range(1,5))+list(range(5,40,4))
evaluate=[]
# initialize a 3-NN classifier
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(14, 7))
fig.suptitle('Receiver Operating Characteristic', fontsize=16)

for k in k_neighbors:
    knn=neighbors.KNeighborsClassifier(n_neighbors=k)
    scaler = StandardScaler()
    model = Pipeline([
                 ("scaler", scaler),
                 ("knn", knn)
                ])
    model.fit(X_train, y_train)
    # Determine false positives and true positives
    prediction = model.predict_proba(X_test)
    fpr, tpr, threshold = roc_curve(y_test, prediction[:, 1])#second colomn: true pos prediction
    evaluate.append([k,auc(fpr, tpr)])

    ax[0].plot(fpr, tpr, label = 'k ={}'.format(k))
    ax[0].legend(loc = 'lower right')
    ax[0].plot([0, 1], [0, 1],'r--')
    ax[0].set_xlim([-0.05, 1])
    ax[0].set_ylim([0, 1.05])
    ax[0].set_ylabel('True Positive Rate')
    ax[0].set_xlabel('False Positive Rate')
    ax[0].set_title('ROC Curve of kNN')
    #Zoom
    ax[1].plot(fpr, tpr, label = 'k ={}'.format(k))
    ax[1].legend(loc = 'lower right')
    ax[1].set_xlim([-0.01, 0.1])
    ax[1].set_ylim([0.8, 1.01])
    ax[1].set_ylabel('True Positive Rate')
    ax[1].set_xlabel('False Positive Rate')
    ax[1].set_title('ROC Curve of kNN zoomed')

plt.show()

evaluate=np.array(evaluate)
max_AUC_idx=np.argmax(evaluate[:,1])
plt.plot(evaluate[:,0],evaluate[:,1],'*')
plt.title('AUC values of ROC curves')
plt.xlabel('k-Value')
plt.ylabel('AUC')
plt.show
print('Max AUC is {:4f} for k={}'.format(evaluate[max_AUC_idx,1],int(evaluate[max_AUC_idx,0])))

```

Up to approximately the maximum Aera Under the Curve (AUC) at k = 9, the AUG values increase for increasing k values. For k greater than 9, the AUC values seem to fluctuate around a value just below the max AUC: after the maximum AUC the model becomes more complex, but does not seem to perform better at distinguishing between breast cancer and non-breast cancer.

### $F_1$ score and Dice similarity coefficient

The Dice similarity coefficient is a very popular evaluation measure for image segmentation applications. Assuming that $A$ is the ground truth segmentation of an object represented as a binary image, and $B$ is the binary output of an image segmentation method, the Dice similarity coefficient is computed as:

$\text{Dice}(A,B) = \frac{2|A\cap B|}{|A| + |B|}$

where $|\cdot|$ represents the cardinality of the objects (e.g. $|A|$ is the number of non-zero pixels in the ground truth segmentation).

For example, the Dice similarity can be computed in the following way:

```python
# generate some test objecys
A = np.zeros((32, 32))
A[10:-10, 10:-10] = 1
B = np.zeros((32, 32))
B[5:-15, 5:-15] = 1

dice = 2*np.sum(A*B)/(np.sum(A)+np.sum(B))

# display the results
plt.plot()
plt.imshow(A)
plt.imshow(B, alpha=0.7)

print(dice)
```

Show that the $F_1$ score, which is the harmonic mean of precision and recall, is equivalent to the Dice similarity coefficient

$F_{1} = 2*\frac{precision * recall}{precision + recall}$

$precision = \frac{TP}{TP+FP}$

$recall = \frac{TP}{TP+FN}$

$F_{1} = 2*\frac{\frac{TP}{TP+FP} * \frac{TP}{TP+FN}}{\frac{TP}{TP+FP} + \frac{TP}{TP+FN}} = 2*\frac{TP * \frac{TP}{TP+FN}}{TP + \frac{TP(TP+FP)}{TP+FN}} = 2*\frac{TP * TP}{TP(TP+FN) + TP(TP+FP)} $

$= 2*\frac{TP}{TP+FN + TP+FP} = \frac{2TP}{2TP+FN+FP}$

$\text{Dice}(A,B) = \frac{2|A\cap B|}{|A| + |B|} = \frac{2TP}{TP+FN + TP+FP} = \frac{2TP}{2TP+FN+FP}$


```python
%matplotlib inline
```

Neural Transfer with PyTorch
============================
**Author**: `Alexis Jacq <https://alexis-jacq.github.io>`_

Introduction
------------

Welcome! This tutorial explains how to impletment the
`Neural-Style <https://arxiv.org/abs/1508.06576>`__ algorithm developed
by Leon A. Gatys, Alexander S. Ecker and Matthias Bethge.

Neural what?
~~~~~~~~~~~~

The Neural-Style, or Neural-Transfer, is an algorithm that takes as
input a content-image (e.g. a tortle), a style-image (e.g. artistic
waves) and return the content of the content-image as if it was
'painted' using the artistic style of the style-image:

.. figure:: /_static/img/neural-style/neuralstyle.png
   :alt: content1

How does it work?
~~~~~~~~~~~~~~~~~

The principle is simple: we define two distances, one for the content
($D_C$) and one for the style ($D_S$). $D_C$ measures
how different the content is between two images, while $D_S$
measures how different the style is between two images. Then, we take a
third image, the input, (e.g. a with noise), and we transform it in
order to both minimize its content-distance with the content-image and
its style-distance with the style-image.

OK. How does it work?
^^^^^^^^^^^^^^^^^^^^^

Well, going further requires some mathematics. Let $C_{nn}$ be a
pre-trained deep convolutional neural network and $X$ be any
image. $C_{nn}(X)$ is the network fed by $X$ (containing
feature maps at all layers). Let $F_{XL} \in C_{nn}(X)$ be the
feature maps at depth layer $L$, all vectorized and concatenated
in one single vector. We simply define the content of $X$ at layer
$L$ by $F_{XL}$. Then, if $Y$ is another image of same
the size than $X$, we define the distance of content at layer
$L$ as follow:

\begin{align}D_C^L(X,Y) = \|F_{XL} - F_{YL}\|^2 = \sum_i (F_{XL}(i) - F_{YL}(i))^2\end{align}

Where $F_{XL}(i)$ is the $i^{th}$ element of $F_{XL}$.
The style is a bit less trivial to define. Let $F_{XL}^k$ with
$k \leq K$ be the vectorized $k^{th}$ of the $K$
feature maps at layer $L$. The style $G_{XL}$ of $X$
at layer $L$ is defined by the Gram produce of all vectorized
feature maps $F_{XL}^k$ with $k \leq K$. In other words,
$G_{XL}$ is a $K$\ x\ $K$ matrix and the element
$G_{XL}(k,l)$ at the $k^{th}$ line and $l^{th}$ column
of $G_{XL}$ is the vectorial produce between $F_{XL}^k$ and
$F_{XL}^l$ :

\begin{align}G_{XL}(k,l) = \langle F_{XL}^k, F_{XL}^l\rangle = \sum_i F_{XL}^k(i) . F_{XL}^l(i)\end{align}

Where $F_{XL}^k(i)$ is the $i^{th}$ element of
$F_{XL}^k$. We can see $G_{XL}(k,l)$ as a measure of the
correlation between feature maps $k$ and $l$. In that way,
$G_{XL}$ represents the correlation matrix of feature maps of
$X$ at layer $L$. Note that the size of $G_{XL}$ only
depends on the number of feature maps, not on the size of $X$.
Then, if $Y$ is another image *of any size*, we define the
distance of style at layer $L$ as follow:

\begin{align}D_S^L(X,Y) = \|G_{XL} - G_{YL}\|^2 = \sum_{k,l} (G_{XL}(k,l) - G_{YL}(k,l))^2\end{align}

In order to minimize in one shot $D_C(X,C)$ between a variable
image $X$ and target content-image $C$ and $D_S(X,S)$
between $X$ and target style-image $S$, both computed at
several layers , we compute and sum the gradients (derivative with
respect to $X$) of each distance at each wanted layer:

\begin{align}\nabla_{	extit{total}}(X,S,C) = \sum_{L_C} w_{CL_C}.\nabla_{	extit{content}}^{L_C}(X,C) + \sum_{L_S} w_{SL_S}.\nabla_{	extit{style}}^{L_S}(X,S)\end{align}

Where $L_C$ and $L_S$ are respectivement the wanted layers
(arbitrary stated) of content and style and $w_{CL_C}$ and
$w_{SL_S}$ the weights (arbitrary stated) associated with the
style or the content at each wanted layer. Then, we run a gradient
descent over $X$:

\begin{align}X \leftarrow X - \alpha \nabla_{	extit{total}}(X,S,C)\end{align}

Ok. That's enough with maths. If you want to go deeper (how to compute
the gradients) **we encourage you to read the original paper** by Leon
A. Gatys and AL, where everything is much better and much clearer
explained.

For our implementation in PyTorch, we already have everything
we need: indeed, with PyTorch, all the gradients are automatically and
dynamically computed for you (while you use functions from the library).
This is why the implementation of this algorithm becomes very
comfortable with PyTorch.

PyTorch implementation
----------------------

If you are not sure to understand all the mathematics above, you will
probably get it by implementing it. If you are discovering PyTorch, we
recommend you to first read this :doc:`Introduction to
PyTorch </beginner/deep_learning_60min_blitz>`.

Packages
~~~~~~~~

We will have recourse to the following packages:

-  ``torch``, ``torch.nn``, ``numpy`` (indispensables packages for
   neural networks with PyTorch)
-  ``torch.optim`` (efficient gradient descents)
-  ``PIL``, ``PIL.Image``, ``matplotlib.pyplot`` (load and display
   images)
-  ``torchvision.transforms`` (treat PIL images and transform into torch
   tensors)
-  ``torchvision.models`` (train or load pre-trained models)
-  ``copy`` (to deep copy the models; system package)

```python
from __future__ import print_function

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

from PIL import Image
import matplotlib.pyplot as plt

import torchvision.transforms as transforms
import torchvision.models as models

import copy
```

Cuda
~~~~

If you have a GPU on your computer, it is preferable to run the
algorithm on it, especially if you want to try larger networks (like
VGG). For this, we have ``torch.cuda.is_available()`` that returns
``True`` if you computer has an available GPU. Then, we can set the
``torch.device`` that will be used in this script. Then, we will use
the method ``.to(device)`` that moves a tensor or a module to the desired
device. When we want to move back this tensor or module to the
CPU (e.g. to use numpy), we can use the ``.cpu()`` method.

```python
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
```

Load images
~~~~~~~~~~~

In order to simplify the implementation, let's start by importing a
style and a content image of the same dimentions. We then scale them to
the desired output image size (128 or 512 in the example, depending on gpu
availablity) and transform them into torch tensors, ready to feed
a neural network:

.. Note::
    Here are links to download the images required to run the tutorial:
    `picasso.jpg <http://pytorch.org/tutorials/_static/img/neural-style/picasso.jpg>`__ and
    `dancing.jpg <http://pytorch.org/tutorials/_static/img/neural-style/dancing.jpg>`__.
    Download these two images and add them to a directory
    with name ``images``

```python
# desired size of the output image
imsize = 512 if torch.cuda.is_available() else 128  # use small size if no gpu

loader = transforms.Compose([
    transforms.Resize(imsize),  # scale imported image
    transforms.ToTensor()])  # transform it into a torch tensor

def image_loader(image_name):
    image = Image.open(image_name)
    # fake batch dimension required to fit network's input dimensions
    image = loader(image).unsqueeze(0)
    return image.to(device, torch.float)

style_img = image_loader("images/picasso.jpg")
content_img = image_loader("images/dancing.jpg")

assert style_img.size() == content_img.size(), \
    "we need to import style and content images of the same size"
```

Imported PIL images has values between 0 and 255. Transformed into torch
tensors, their values are between 0 and 1. This is an important detail:
neural networks from torch library are trained with 0-1 tensor image. If
you try to feed the networks with 0-255 tensor images the activated
feature maps will have no sense. This is not the case with pre-trained
networks from the Caffe library: they are trained with 0-255 tensor
images.

Display images
~~~~~~~~~~~~~~

We will use ``plt.imshow`` to display images. So we need to first
reconvert them into PIL images:

```python
unloader = transforms.ToPILImage()  # reconvert into PIL image

plt.ion()

def imshow(tensor, title=None):
    image = tensor.cpu().clone()  # we clone the tensor to not do changes on it
    image = image.squeeze(0)      # remove the fake batch dimension
    image = unloader(image)
    plt.imshow(image)
    if title is not None:
        plt.title(title)
    plt.pause(0.001) # pause a bit so that plots are updated

plt.figure()
imshow(style_img, title='Style Image')

plt.figure()
imshow(content_img, title='Content Image')
```

Content loss
~~~~~~~~~~~~

The content loss is a function that takes as input the feature maps
$F_{XL}$ at a layer $L$ in a network fed by $X$ and
return the weigthed content distance $w_{CL}.D_C^L(X,C)$ between
this image and the content image. Hence, the weight $w_{CL}$ and
the target content $F_{CL}$ are parameters of the function. We
implement this function as a torch module with a constructor that takes
these parameters as input. The distance $\|F_{XL} - F_{YL}\|^2$ is
the Mean Square Error between the two sets of feature maps, that can be
computed using a criterion ``nn.MSELoss`` stated as a third parameter.

We will add our content losses at each desired layer as additive modules
of the neural network. That way, each time we will feed the network with
an input image $X$, all the content losses will be computed at the
desired layers and, thanks to autograd, all the gradients will be
computed. For that, we just need to make the ``forward`` method of our
module returning the input: the module becomes a ''transparent layer''
of the neural network. The computed loss is saved as a parameter of the
module.

Finally, we define a fake ``backward`` method, that just call the
backward method of ``nn.MSELoss`` in order to reconstruct the gradient.
This method returns the computed loss: this will be useful when running
the gradient descent in order to display the evolution of style and
content losses.

```python
class ContentLoss(nn.Module):

    def __init__(self, target,):
        super(ContentLoss, self).__init__()
        # we 'detach' the target content from the tree used
        # to dynamically compute the gradient: this is a stated value,
        # not a variable. Otherwise the forward method of the criterion
        # will throw an error.
        self.target = target.detach()

    def forward(self, input):
        self.loss = F.mse_loss(input, self.target)
        return input
```

.. Note::
   **Important detail**: this module, although it is named ``ContentLoss``,
   is not a true PyTorch Loss function. If you want to define your content
   loss as a PyTorch Loss, you have to create a PyTorch autograd Function
   and to recompute/implement the gradient by the hand in the ``backward``
   method.

Style loss
~~~~~~~~~~

For the style loss, we need first to define a module that compute the
gram produce $G_{XL}$ given the feature maps $F_{XL}$ of the
neural network fed by $X$, at layer $L$. Let
$\hat{F}_{XL}$ be the re-shaped version of $F_{XL}$ into a
$K$\ x\ $N$ matrix, where $K$ is the number of feature
maps at layer $L$ and $N$ the lenght of any vectorized
feature map $F_{XL}^k$. The $k^{th}$ line of
$\hat{F}_{XL}$ is $F_{XL}^k$. We let you check that
$\hat{F}_{XL} \cdot \hat{F}_{XL}^T = G_{XL}$. Given that, it
becomes easy to implement our module:

```python
def gram_matrix(input):
    a, b, c, d = input.size()  # a=batch size(=1)
    # b=number of feature maps
    # (c,d)=dimensions of a f. map (N=c*d)

    features = input.view(a * b, c * d)  # resise F_XL into \hat F_XL

    G = torch.mm(features, features.t())  # compute the gram product

    # we 'normalize' the values of the gram matrix
    # by dividing by the number of element in each feature maps.
    return G.div(a * b * c * d)
```

The longer is the feature maps dimension $N$, the bigger are the
values of the Gram matrix. Therefore, if we don't normalize by $N$,
the loss computed at the first layers (before pooling layers) will have
much more importance during the gradient descent. We dont want that,
since the most interesting style features are in the deepest layers!

Then, the style loss module is implemented exactly the same way than the
content loss module, but it compares the difference in Gram matrices of target
and input

```python
class StyleLoss(nn.Module):

    def __init__(self, target_feature):
        super(StyleLoss, self).__init__()
        self.target = gram_matrix(target_feature).detach()

    def forward(self, input):
        G = gram_matrix(input)
        self.loss = F.mse_loss(G, self.target)
        return input
```

Load the neural network
~~~~~~~~~~~~~~~~~~~~~~~

Now, we have to import a pre-trained neural network. As in the paper, we
are going to use a pretrained VGG network with 19 layers (VGG19).

PyTorch's implementation of VGG is a module divided in two child
``Sequential`` modules: ``features`` (containing convolution and pooling
layers) and ``classifier`` (containing fully connected layers). We are
just interested by ``features``:
Some layers have different behavior in training and in evaluation. Since we
are using it as a feature extractor. We will use ``.eval()`` to set the
network in evaluation mode.

```python
cnn = models.vgg19(pretrained=True).features.to(device).eval()
```

Additionally, VGG networks are trained on images with each channel normalized
by mean=[0.485, 0.456, 0.406] and std=[0.229, 0.224, 0.225]. We will use them
to normalize the image before sending into the network.

```python
cnn_normalization_mean = torch.tensor([0.485, 0.456, 0.406]).to(device)
cnn_normalization_std = torch.tensor([0.229, 0.224, 0.225]).to(device)

# create a module to normalize input image so we can easily put it in a
# nn.Sequential
class Normalization(nn.Module):
    def __init__(self, mean, std):
        super(Normalization, self).__init__()
        # .view the mean and std to make them [C x 1 x 1] so that they can
        # directly work with image Tensor of shape [B x C x H x W].
        # B is batch size. C is number of channels. H is height and W is width.
        self.mean = torch.tensor(mean).view(-1, 1, 1)
        self.std = torch.tensor(std).view(-1, 1, 1)

    def forward(self, img):
        # normalize img
        return (img - self.mean) / self.std
```

A ``Sequential`` module contains an ordered list of child modules. For
instance, ``vgg19.features`` contains a sequence (Conv2d, ReLU,
MaxPool2d, Conv2d, ReLU...) aligned in the right order of depth. As we
said in *Content loss* section, we wand to add our style and content
loss modules as additive 'transparent' layers in our network, at desired
depths. For that, we construct a new ``Sequential`` module, in which we
are going to add modules from ``vgg19`` and our loss modules in the
right order:

```python
# desired depth layers to compute style/content losses :
content_layers_default = ['conv_4']
style_layers_default = ['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5']

def get_style_model_and_losses(cnn, normalization_mean, normalization_std,
                               style_img, content_img,
                               content_layers=content_layers_default,
                               style_layers=style_layers_default):
    cnn = copy.deepcopy(cnn)

    # normalization module
    normalization = Normalization(normalization_mean, normalization_std).to(device)

    # just in order to have an iterable access to or list of content/syle
    # losses
    content_losses = []
    style_losses = []

    # assuming that cnn is a nn.Sequential, so we make a new nn.Sequential
    # to put in modules that are supposed to be activated sequentially
    model = nn.Sequential(normalization)

    i = 0  # increment every time we see a conv
    for layer in cnn.children():
        if isinstance(layer, nn.Conv2d):
            i += 1
            name = 'conv_{}'.format(i)
        elif isinstance(layer, nn.ReLU):
            name = 'relu_{}'.format(i)
            # The in-place version doesn't play very nicely with the ContentLoss
            # and StyleLoss we insert below. So we replace with out-of-place
            # ones here.
            layer = nn.ReLU(inplace=False)
        elif isinstance(layer, nn.MaxPool2d):
            name = 'pool_{}'.format(i)
        elif isinstance(layer, nn.BatchNorm2d):
            name = 'bn_{}'.format(i)
        else:
            raise RuntimeError('Unrecognized layer: {}'.format(layer.__class__.__name__))

        model.add_module(name, layer)

        if name in content_layers:
            # add content loss:
            target = model(content_img).detach()
            content_loss = ContentLoss(target)
            model.add_module("content_loss_{}".format(i), content_loss)
            content_losses.append(content_loss)

        if name in style_layers:
            # add style loss:
            target_feature = model(style_img).detach()
            style_loss = StyleLoss(target_feature)
            model.add_module("style_loss_{}".format(i), style_loss)
            style_losses.append(style_loss)

    # now we trim off the layers after the last content and style losses
    for i in range(len(model) - 1, -1, -1):
        if isinstance(model[i], ContentLoss) or isinstance(model[i], StyleLoss):
            break

    model = model[:(i + 1)]

    return model, style_losses, content_losses
```

.. Note::
   In the paper they recommend to change max pooling layers into
   average pooling. With AlexNet, that is a small network compared to VGG19
   used in the paper, we are not going to see any difference of quality in
   the result. However, you can use these lines instead if you want to do
   this substitution:

   ::

       # avgpool = nn.AvgPool2d(kernel_size=layer.kernel_size,
       #                         stride=layer.stride, padding = layer.padding)
       # model.add_module(name,avgpool)

Input image
~~~~~~~~~~~

Again, in order to simplify the code, we take an image of the same
dimensions than content and style images. This image can be a white
noise, or it can also be a copy of the content-image.

```python
input_img = content_img.clone()
# if you want to use a white noise instead uncomment the below line:
# input_img = torch.randn(content_img.data.size(), device=device)

# add the original input image to the figure:
plt.figure()
imshow(input_img, title='Input Image')
```

Gradient descent
~~~~~~~~~~~~~~~~

As Leon Gatys, the author of the algorithm, suggested
`here <https://discuss.pytorch.org/t/pytorch-tutorial-for-neural-transfert-of-artistic-style/336/20?u=alexis-jacq>`__,
we will use L-BFGS algorithm to run our gradient descent. Unlike
training a network, we want to train the input image in order to
minimise the content/style losses. We would like to simply create a
PyTorch  L-BFGS optimizer ``optim.LBFGS``, passing our image as the
Tensor to optimize. We use ``.requires_grad_()`` to make sure that this
image requires gradient.

```python
def get_input_optimizer(input_img):
    # this line to show that input is a parameter that requires a gradient
    optimizer = optim.LBFGS([input_img.requires_grad_()])
    return optimizer
```

**Last step**: the loop of gradient descent. At each step, we must feed
the network with the updated input in order to compute the new losses,
we must run the ``backward`` methods of each loss to dynamically compute
their gradients and perform the step of gradient descent. The optimizer
requires as argument a "closure": a function that reevaluates the model
and returns the loss.

However, there's a small catch. The optimized image may take its values
between $-\infty$ and $+\infty$ instead of staying between 0
and 1. In other words, the image might be well optimized and have absurd
values. In fact, we must perform an optimization under constraints in
order to keep having right vaues into our input image. There is a simple
solution: at each step, to correct the image to maintain its values into
the 0-1 interval.

```python
def run_style_transfer(cnn, normalization_mean, normalization_std,
                       content_img, style_img, input_img, num_steps=300,
                       style_weight=1000000, content_weight=1):
    """Run the style transfer."""
    print('Building the style transfer model..')
    model, style_losses, content_losses = get_style_model_and_losses(cnn,
        normalization_mean, normalization_std, style_img, content_img)
    optimizer = get_input_optimizer(input_img)

    print('Optimizing..')
    run = [0]
    while run[0] <= num_steps:

        def closure():
            # correct the values of updated input image
            input_img.data.clamp_(0, 1)

            optimizer.zero_grad()
            model(input_img)
            style_score = 0
            content_score = 0

            for sl in style_losses:
                style_score += sl.loss
            for cl in content_losses:
                content_score += cl.loss

            style_score *= style_weight
            content_score *= content_weight

            loss = style_score + content_score
            loss.backward()

            run[0] += 1
            if run[0] % 50 == 0:
                print("run {}:".format(run))
                print('Style Loss : {:4f} Content Loss: {:4f}'.format(
                    style_score.item(), content_score.item()))
                print()

            return style_score + content_score

        optimizer.step(closure)

    # a last correction...
    input_img.data.clamp_(0, 1)

    return input_img
```

Finally, run the algorithm

```python
output = run_style_transfer(cnn, cnn_normalization_mean, cnn_normalization_std,
                            content_img, style_img, input_img)

plt.figure()
imshow(output, title='Output Image')

# sphinx_gallery_thumbnail_number = 4
plt.ioff()
plt.show()
```