# Convolutional Neural Networks

## Preliminaries

In this practical session you will implement and train several Convolutional Neural Networks (CNNs) using the Keras framework with a Tensorflow backend. If you are not already familiar with Keras, you can go over the [following tutorial](https://github.com/tueimage/essential-skills/blob/master/keras.md). More detailed information on the different functionalities can be found in the [Keras library documentation](https://keras.io/).

Note that for this set of exercise CPU-only Tensorflow is sufficient (i.e. GPU-support is not required but it will make your experiments run faster). If you use the Anaconda Python distribution, it is recommeded to install Tensorflow via the conda package manager: `conda install tensorflow-cpu`.

You are also required to use the `gryds` package for data augmentation that you can install directly from git: `pip install git+https://github.com/tueimage/gryds/`.

## MNIST
We will first train a simple CNN to classify handwritten digits using the MNIST dataset. This dataset is often referred to as the "Hello world!" example of deep learning because it can be used to quickly illustrate a small neural network in action (and obtain a decent classification accuracy in the process). More information on it can be found [here](http://yann.lecun.com/exdb/mnist/).

First, let's load the dataset and visualize some images.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from keras.datasets import mnist
from keras.utils import to_categorical

# load the MNIST the dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()

# scale the image intensities to the 0-1 range
x_train = (x_train / 255.0).astype(np.float32)
x_test = (x_test / 255.0).astype(np.float32)

# convert the data to channel-last
x_train = np.expand_dims(x_train, axis=-1)
x_test = np.expand_dims(x_test, axis=-1)

# convert the labels to one-hot encoded
y_train = to_categorical(y_train, num_classes=10)
y_test = to_categorical(y_test, num_classes=10)

def plot_images(images, dim=(10, 10), figsize=(10, 10), title=''):

    plt.figure(figsize=figsize)

    for i in range(images.shape[0]):
        plt.subplot(dim[0], dim[1], i+1)
        plt.imshow(images[i], interpolation='nearest', cmap='gray_r')
        plt.axis('off')

    plt.tight_layout()
    plt.suptitle(title)
    plt.show()

plot_images(x_train[np.random.randint(0, x_train.shape[0], size=100)].reshape(100, 28, 28))
```

The MNIST classification task is quite simple: given an image, predict the digit that it contains. Thus, this is a 10-class classification problem.

Let's define a simple network for the handwritten digit classification.

```python
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D, Dropout, Flatten, Dense

model = Sequential()
model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(10, activation='softmax'))

model.summary()
```

    Model: "sequential_1"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    conv2d (Conv2D)              (None, 26, 26, 32)        320
    _________________________________________________________________
    conv2d_1 (Conv2D)            (None, 24, 24, 64)        18496
    _________________________________________________________________
    max_pooling2d (MaxPooling2D) (None, 12, 12, 64)        0
    _________________________________________________________________
    dropout (Dropout)            (None, 12, 12, 64)        0
    _________________________________________________________________
    flatten (Flatten)            (None, 9216)              0
    _________________________________________________________________
    dense (Dense)                (None, 128)               1179776
    _________________________________________________________________
    dropout_1 (Dropout)          (None, 128)               0
    _________________________________________________________________
    dense_1 (Dense)              (None, 10)                1290
    =================================================================
    Total params: 1,199,882
    Trainable params: 1,199,882
    Non-trainable params: 0
    _________________________________________________________________

Compile and train the network (note that this could take a while).

```python
from keras.losses import categorical_crossentropy
from keras.optimizers import Adam

model.compile(loss=categorical_crossentropy, optimizer=Adam(), metrics=['accuracy'])

model.fit(x_train, y_train,
          batch_size=128,
          epochs=12,
          verbose=1,
          validation_data=(x_test, y_test))

score = model.evaluate(x_test, y_test, verbose=0)

print('Test loss:', score[0])
print('Test accuracy:', score[1])
```

    Train on 60000 samples, validate on 10000 samples
    Epoch 1/12
    60000/60000 [==============================] - 118s 2ms/sample - loss: 0.2438 - accuracy: 0.9266 - val_loss: 0.0539 - val_accuracy: 0.9823
    Epoch 2/12
    60000/60000 [==============================] - 118s 2ms/sample - loss: 0.0860 - accuracy: 0.9739 - val_loss: 0.0405 - val_accuracy: 0.9861
    Epoch 3/12
    60000/60000 [==============================] - 124s 2ms/sample - loss: 0.0658 - accuracy: 0.9799 - val_loss: 0.0337 - val_accuracy: 0.9893
    Epoch 4/12
    60000/60000 [==============================] - 114s 2ms/sample - loss: 0.0518 - accuracy: 0.9846 - val_loss: 0.0305 - val_accuracy: 0.9897
    Epoch 5/12
    60000/60000 [==============================] - 121s 2ms/sample - loss: 0.0457 - accuracy: 0.9860 - val_loss: 0.0302 - val_accuracy: 0.9902
    Epoch 6/12
    60000/60000 [==============================] - 113s 2ms/sample - loss: 0.0388 - accuracy: 0.9878 - val_loss: 0.0343 - val_accuracy: 0.9895
    Epoch 7/12
    60000/60000 [==============================] - 116s 2ms/sample - loss: 0.0341 - accuracy: 0.9891 - val_loss: 0.0286 - val_accuracy: 0.9914
    Epoch 8/12
    60000/60000 [==============================] - 116s 2ms/sample - loss: 0.0328 - accuracy: 0.9894 - val_loss: 0.0284 - val_accuracy: 0.9903
    Epoch 9/12
    60000/60000 [==============================] - 114s 2ms/sample - loss: 0.0298 - accuracy: 0.9907 - val_loss: 0.0289 - val_accuracy: 0.9903
    Epoch 10/12
    60000/60000 [==============================] - 120s 2ms/sample - loss: 0.0246 - accuracy: 0.9922 - val_loss: 0.0338 - val_accuracy: 0.9912
    Epoch 11/12
    60000/60000 [==============================] - 134s 2ms/sample - loss: 0.0234 - accuracy: 0.9922 - val_loss: 0.0302 - val_accuracy: 0.9913
    Epoch 12/12
    60000/60000 [==============================] - 133s 2ms/sample - loss: 0.0225 - accuracy: 0.9926 - val_loss: 0.0312 - val_accuracy: 0.9914
    Test loss: 0.03122303096586777
    Test accuracy: 0.9914

Now that the model is trained, we can evaluate its performance on the independent test dataset.

```python
score = model.evaluate(x_test, y_test, verbose=0)

print('Test loss:', score[0])
print('Test accuracy:', score[1])
```

    Test loss: 0.03122303096586777
    Test accuracy: 0.9914

## U-Net
The U-Net convolutional neural network architecture was first developed for biomedical image segmentation and is to this day one of the most widely used methods for image segmentation. The details of the architecture can be found in the [original paper](https://arxiv.org/abs/1505.04597). In this practical we will build and train a U-Net network that is able to segment blood vessels in retinal images.

### Loading and visualizing the data
The data for this task is taken from the [DRIVE](https://www.isi.uu.nl/Research/Databases/DRIVE/index.php) database. It consists of photographs of the retina, where the goal is to segment the blood vessels within. The dataset has a total of 40 photographs, divided in 20 images for training and 20 for testing.

Let's load the training set and visualize an image with the corresponding blood vessel segmentation. For training we will divide the data in a training and a validation set to monitor the model during training.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from glob import glob
import sys
sys.path.append('code/')
from unet_utils import load_data

# location of the DRIVE dataset
data_folder = './data/'
train_paths = glob(data_folder + 'training/images/*.tif')
images, masks, segmentations = load_data(train_paths)

# print the shape of image dataset
print(images.shape)

plt.figure(figsize=(10, 8))

plt.subplot(1, 2, 1)
plt.title("Image of the retina")
plt.axis('off')
plt.imshow(images[0])

plt.subplot(1, 2, 2)
plt.title("Ground truth vessel segmentation")
plt.axis('off')
plt.imshow(segmentations[0][:, :, 0])

plt.show()

# divide in training and validation
train_images, val_images, train_masks, val_masks, train_segmentations, val_segmentations = train_test_split(
    images, masks, segmentations, test_size=0.2, random_state=7)

# print the shape of the training and valudation datasets
print(train_images.shape)
print(train_masks.shape)
print(train_segmentations.shape)
print(val_images.shape)
print(val_masks.shape)
print(val_segmentations.shape)
```

### Training a U-Net model

You are already provided with implementation of the U-Net architecture in `unet_model.py`. This is a modular implementation and can be used to generate U-Net architectures with a variety of hyperparameters such as depth and number of feature maps. Before using the model, examine the code and documentation and make sure that you understand all the details.

We will train a U-Net model using smaller patches extracted from the training images. Training the images on smaller patches requires less computation power and results in a more varied training dataset (it has the effect of data augmentation by image translation). Because a U-Net is a fully convolutional network it can be evaluated on inputs of different size (the output size will change according to the input size). Thus, although the model will be trained on smaller patches it can still be used to segment larger images with one pass through the model.

```python
from unet_utils import extract_patches, preprocessing

# work with 32x32 patches
patch_size = (32, 32)

# 200 patches per image
patches_per_im = 200

# visualize a couple of patches as a visual check
patches, patches_segmentations = extract_patches(train_images, train_segmentations, patch_size, patches_per_im=1, seed=7)

print(patches.shape)

fig, axes = plt.subplots(nrows=3, ncols=4, figsize=(12, 12))

for i in range(0, 3):
    axes[i, 0].axis('off')
    axes[i, 0].imshow(patches[i])
    axes[i, 1].axis('off')
    axes[i, 1].imshow(patches_segmentations[i][:, :, 0])
    axes[i, 2].axis('off')
    axes[i, 2].imshow(patches[i+5])
    axes[i, 3].axis('off')
    axes[i, 3].imshow(patches_segmentations[i+5][:, :, 0])

plt.show()
```

```python
# pad the validation data to fit the U-Net model
# images of shape (584, 565) shape result in concatenation error due to the odd number of columns

print("Old shape:", val_images.shape)

val_images, val_masks, val_segmentations = preprocessing(
    val_images,
    val_masks,
    val_segmentations,
    desired_shape=(584, 584))

print("New shape:", val_images.shape)
```

    Old shape: (4, 584, 565, 3)
    New shape: (4, 584, 584, 3)

```python
import os
sys.path.append('code/')
from unet_model import unet
from unet_utils import datagenerator
from keras.models import load_model
from keras.optimizers import Adam, SGD
from keras.callbacks import EarlyStopping, ModelCheckpoint, ReduceLROnPlateau

# use a single training image, to better demonstrate the effects of data augmentation
X_train, y_train = np.expand_dims(train_images[0], axis=0), np.expand_dims(train_segmentations[0], axis=0)

print(X_train.shape)
print(y_train.shape)

# hyperparameters
depth = 3
channels = 32
use_batchnorm = True
batch_size = 64
epochs = 250
steps_per_epoch = int(np.ceil((patches_per_im * len(train_images)) / batch_size))

# work with 32x32 patches
patch_size = (32, 32)
# 200 patches per image
patches_per_im = 200

# initialize model
model = unet(input_shape=(None, None, 3), depth=depth, channels=channels, batchnorm=use_batchnorm)

# print a summary of the model
model.summary(line_length=120)

# compile the model
model.compile(loss='binary_crossentropy', optimizer=Adam(), metrics=['accuracy'])

# stop the training if the validation loss does not increase for 15 consecutive epochs
early_stopping = EarlyStopping(monitor='val_loss', patience=15, restore_best_weights=True)

# train the model with the data generator, and save the training history
history = model.fit_generator(datagenerator(X_train, y_train, patch_size, patches_per_im, batch_size),
                              validation_data=(val_images, val_segmentations),
                              steps_per_epoch=steps_per_epoch, epochs=epochs, verbose=2,
                              callbacks=[early_stopping])
```

    (1, 584, 565, 3)
    (1, 584, 565, 1)
    Model: "model"
    ________________________________________________________________________________________________________________________
    Layer (type)                           Output Shape               Param #       Connected to
    ========================================================================================================================
    input_1 (InputLayer)                   [(None, None, None, 3)]    0
    ________________________________________________________________________________________________________________________
    conv2d (Conv2D)                        (None, None, None, 32)     896           input_1[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization (BatchNormalizatio (None, None, None, 32)     128           conv2d[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_1 (Conv2D)                      (None, None, None, 32)     9248          batch_normalization[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_1 (BatchNormalizat (None, None, None, 32)     128           conv2d_1[0][0]
    ________________________________________________________________________________________________________________________
    max_pooling2d (MaxPooling2D)           (None, None, None, 32)     0             batch_normalization_1[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_2 (Conv2D)                      (None, None, None, 64)     18496         max_pooling2d[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_2 (BatchNormalizat (None, None, None, 64)     256           conv2d_2[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_3 (Conv2D)                      (None, None, None, 64)     36928         batch_normalization_2[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_3 (BatchNormalizat (None, None, None, 64)     256           conv2d_3[0][0]
    ________________________________________________________________________________________________________________________
    max_pooling2d_1 (MaxPooling2D)         (None, None, None, 64)     0             batch_normalization_3[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_4 (Conv2D)                      (None, None, None, 128)    73856         max_pooling2d_1[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_4 (BatchNormalizat (None, None, None, 128)    512           conv2d_4[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_5 (Conv2D)                      (None, None, None, 128)    147584        batch_normalization_4[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_5 (BatchNormalizat (None, None, None, 128)    512           conv2d_5[0][0]
    ________________________________________________________________________________________________________________________
    max_pooling2d_2 (MaxPooling2D)         (None, None, None, 128)    0             batch_normalization_5[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_6 (Conv2D)                      (None, None, None, 256)    295168        max_pooling2d_2[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_6 (BatchNormalizat (None, None, None, 256)    1024          conv2d_6[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_7 (Conv2D)                      (None, None, None, 256)    590080        batch_normalization_6[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_7 (BatchNormalizat (None, None, None, 256)    1024          conv2d_7[0][0]
    ________________________________________________________________________________________________________________________
    up_sampling2d (UpSampling2D)           (None, None, None, 256)    0             batch_normalization_7[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_8 (Conv2D)                      (None, None, None, 128)    131200        up_sampling2d[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_8 (BatchNormalizat (None, None, None, 128)    512           conv2d_8[0][0]
    ________________________________________________________________________________________________________________________
    concatenate (Concatenate)              (None, None, None, 256)    0             batch_normalization_5[0][0]
                                                                                batch_normalization_8[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_9 (Conv2D)                      (None, None, None, 128)    295040        concatenate[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_9 (BatchNormalizat (None, None, None, 128)    512           conv2d_9[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_10 (Conv2D)                     (None, None, None, 128)    147584        batch_normalization_9[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_10 (BatchNormaliza (None, None, None, 128)    512           conv2d_10[0][0]
    ________________________________________________________________________________________________________________________
    up_sampling2d_1 (UpSampling2D)         (None, None, None, 128)    0             batch_normalization_10[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_11 (Conv2D)                     (None, None, None, 64)     32832         up_sampling2d_1[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_11 (BatchNormaliza (None, None, None, 64)     256           conv2d_11[0][0]
    ________________________________________________________________________________________________________________________
    concatenate_1 (Concatenate)            (None, None, None, 128)    0             batch_normalization_3[0][0]
                                                                                batch_normalization_11[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_12 (Conv2D)                     (None, None, None, 64)     73792         concatenate_1[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_12 (BatchNormaliza (None, None, None, 64)     256           conv2d_12[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_13 (Conv2D)                     (None, None, None, 64)     36928         batch_normalization_12[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_13 (BatchNormaliza (None, None, None, 64)     256           conv2d_13[0][0]
    ________________________________________________________________________________________________________________________
    up_sampling2d_2 (UpSampling2D)         (None, None, None, 64)     0             batch_normalization_13[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_14 (Conv2D)                     (None, None, None, 32)     8224          up_sampling2d_2[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_14 (BatchNormaliza (None, None, None, 32)     128           conv2d_14[0][0]
    ________________________________________________________________________________________________________________________
    concatenate_2 (Concatenate)            (None, None, None, 64)     0             batch_normalization_1[0][0]
                                                                                batch_normalization_14[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_15 (Conv2D)                     (None, None, None, 32)     18464         concatenate_2[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_15 (BatchNormaliza (None, None, None, 32)     128           conv2d_15[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_16 (Conv2D)                     (None, None, None, 32)     9248          batch_normalization_15[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_16 (BatchNormaliza (None, None, None, 32)     128           conv2d_16[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_17 (Conv2D)                     (None, None, None, 1)      33            batch_normalization_16[0][0]
    ========================================================================================================================
    Total params: 1,932,129
    Trainable params: 1,928,865
    Non-trainable params: 3,264
    ________________________________________________________________________________________________________________________
    Epoch 1/250
    50/50 - 200s - loss: 0.5374 - accuracy: 0.8079 - val_loss: 0.5276 - val_accuracy: 0.9211
    Epoch 2/250
    50/50 - 188s - loss: 0.3277 - accuracy: 0.9485 - val_loss: 0.3127 - val_accuracy: 0.9211
    Epoch 3/250
    50/50 - 195s - loss: 0.2220 - accuracy: 0.9580 - val_loss: 0.2666 - val_accuracy: 0.9211
    Epoch 4/250
    50/50 - 202s - loss: 0.2032 - accuracy: 0.9560 - val_loss: 0.3032 - val_accuracy: 0.9171
    Epoch 5/250
    50/50 - 195s - loss: 0.1840 - accuracy: 0.9523 - val_loss: 0.2984 - val_accuracy: 0.9215
    Epoch 6/250
    50/50 - 184s - loss: 0.1556 - accuracy: 0.9574 - val_loss: 0.3050 - val_accuracy: 0.9194
    Epoch 7/250
    50/50 - 184s - loss: 0.1293 - accuracy: 0.9608 - val_loss: 0.2672 - val_accuracy: 0.9222
    Epoch 8/250
    50/50 - 174s - loss: 0.1437 - accuracy: 0.9582 - val_loss: 0.2738 - val_accuracy: 0.9238
    Epoch 9/250
    50/50 - 126s - loss: 0.1228 - accuracy: 0.9611 - val_loss: 0.2738 - val_accuracy: 0.9261
    Epoch 10/250
    50/50 - 110s - loss: 0.1375 - accuracy: 0.9599 - val_loss: 0.2659 - val_accuracy: 0.9246
    Epoch 11/250
    50/50 - 131s - loss: 0.1245 - accuracy: 0.9588 - val_loss: 0.2337 - val_accuracy: 0.9338
    Epoch 12/250
    50/50 - 120s - loss: 0.1088 - accuracy: 0.9637 - val_loss: 0.2437 - val_accuracy: 0.9335
    Epoch 13/250
    50/50 - 117s - loss: 0.1006 - accuracy: 0.9650 - val_loss: 0.1634 - val_accuracy: 0.9453
    Epoch 14/250
    50/50 - 119s - loss: 0.1091 - accuracy: 0.9634 - val_loss: 0.1636 - val_accuracy: 0.9461
    Epoch 15/250
    50/50 - 124s - loss: 0.1072 - accuracy: 0.9635 - val_loss: 0.3248 - val_accuracy: 0.8587
    Epoch 16/250
    50/50 - 115s - loss: 0.0999 - accuracy: 0.9643 - val_loss: 0.1578 - val_accuracy: 0.9417
    Epoch 17/250
    50/50 - 118s - loss: 0.0934 - accuracy: 0.9655 - val_loss: 0.1997 - val_accuracy: 0.9162
    Epoch 18/250
    50/50 - 115s - loss: 0.0934 - accuracy: 0.9661 - val_loss: 0.1581 - val_accuracy: 0.9436
    Epoch 19/250
    50/50 - 120s - loss: 0.0971 - accuracy: 0.9664 - val_loss: 0.2689 - val_accuracy: 0.8929
    Epoch 20/250
    50/50 - 119s - loss: 0.0995 - accuracy: 0.9656 - val_loss: 0.2179 - val_accuracy: 0.9288
    Epoch 21/250
    50/50 - 119s - loss: 0.1000 - accuracy: 0.9649 - val_loss: 0.2689 - val_accuracy: 0.8758
    Epoch 22/250
    50/50 - 113s - loss: 0.0983 - accuracy: 0.9648 - val_loss: 0.1758 - val_accuracy: 0.9303
    Epoch 23/250
    50/50 - 114s - loss: 0.0986 - accuracy: 0.9653 - val_loss: 0.1844 - val_accuracy: 0.9376
    Epoch 24/250
    50/50 - 114s - loss: 0.0960 - accuracy: 0.9665 - val_loss: 0.2825 - val_accuracy: 0.8835
    Epoch 25/250
    50/50 - 117s - loss: 0.0863 - accuracy: 0.9678 - val_loss: 0.1767 - val_accuracy: 0.9418
    Epoch 26/250
    50/50 - 113s - loss: 0.0910 - accuracy: 0.9676 - val_loss: 0.2698 - val_accuracy: 0.8942
    Epoch 27/250
    50/50 - 116s - loss: 0.0899 - accuracy: 0.9668 - val_loss: 0.2694 - val_accuracy: 0.9041
    Epoch 28/250
    50/50 - 112s - loss: 0.0867 - accuracy: 0.9685 - val_loss: 0.2212 - val_accuracy: 0.9168
    Epoch 29/250
    50/50 - 129s - loss: 0.0862 - accuracy: 0.9686 - val_loss: 0.2125 - val_accuracy: 0.9144
    Epoch 30/250
    50/50 - 131s - loss: 0.0826 - accuracy: 0.9690 - val_loss: 0.1770 - val_accuracy: 0.9294
    Epoch 31/250
    50/50 - 113s - loss: 0.0846 - accuracy: 0.9685 - val_loss: 0.2402 - val_accuracy: 0.9038

### Visualizing the segmentation results

```python
# run the model on one test image and show the results
from unet_utils import preprocessing

# test data paths
impaths_test = glob(data_folder + 'test/images/*.tif')

# load data
test_images, test_masks, test_segmentations = load_data(impaths_test, test=True)

# pad the data to fit the U-Net model
test_images, test_masks, test_segmentations = preprocessing(test_images, test_masks, test_segmentations,
                                                            desired_shape=(584, 584))

# use a single image to evaluate
X_test, y_test = np.expand_dims(test_images[0], axis=0), np.expand_dims(test_masks[0], axis=0)

# predict test samples
test_prediction = model.predict(X_test, batch_size=4)

# visualize the test result
plt.figure(figsize=(12, 10))

plt.subplot(1, 3, 1)
plt.title("Image of the retina")
plt.axis('off')
plt.imshow(test_images[0])

plt.subplot(1, 3, 2)
plt.title("Ground truth vessel segmentation")
plt.axis('off')
plt.imshow(test_segmentations[0][:, :, 0])

plt.subplot(1, 3, 3)
plt.title("Predicted vessel segmentation")
plt.axis('off')
plt.imshow(test_prediction[0, :, :, 0])

plt.show()
```

# Exercises

## Number of parameters

The first convolutional layer in the MNIST example has 320 parameters. The first fully connected layer has 1179,776 parameters. What do these parameters correspond to?

These parameters corresponds to the weights and the bias of that layer.

The input of the first convolutional layer is the image, so the input has 1 dimension. The filter size of the first concolutional layer is n=3 and m=3, or 3x3. Each filter contains 9 weights.(When the input has already s filers, then the filter size is nxmxs). Each filter has a bias, so the total of parameters of each filter is 9 + 1 = 10. There are 32 filter, so the total parameters are 10 * 32 = 320.

The first fully connected layer has a high number of paramaters, beachse every neuron is connected to every other neuron. The number of neurons of the previous layer are 9216. When we added the bias, it makes 9217. The number of neurons of the current layer are 128. The total number of parameters are 9217 * 128 = 1179,776.

<font color='#770a0a'>What is the general expression for the number of parameters of 1) a convolutional layer and 2) a fully-connected layer?</font>

The general expressions are:
1) convolutional layer
\begin{align}
Number\;of\;Parameters = ((m * n * s) + 1) * k
\end{align}
Where m is the shape of the width of the filer, n is the shape op the height of the filer, s is the number of input feature maps  and k is the number of output feature maps.
We addes 1 because of the bias term for each filter

1) Fully connected layer
\begin{align}
Number\;of\;Parameters = ((Neurons\;of\;previous\;layer + 1) * Neurons\;of\;current\;layer)
\end{align}

## Fully-convolutional MNIST model

Modify the model in the MNIST example in such a way that it only contains convolutional layers while keeping the same number of parameters. If you do the modification correctly, the two models will have the same behaviour (i.e. they will represent the same model, only with different implementation). Show this experimentally.

```python
import numpy as np
import matplotlib.pyplot as plt
from tensorflow.keras.datasets import mnist
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dropout, Flatten, Dense
from tensorflow.keras.models import load_model

model2 = Sequential()
model2.add(Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))
model2.add(Conv2D(64, (3, 3), activation='relu'))
model2.add(MaxPooling2D(pool_size=(2, 2)))
model2.add(Dropout(0.25))
model2.add(Conv2D(128, (12,12), activation = 'relu', padding='valid'))
model2.add(Dropout(0.5))
model2.add(Conv2D(10, (1,1), activation='softmax'))
model2.add(Flatten())

model2.summary()
```

    Model: "sequential_14"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    conv2d_55 (Conv2D)           (None, 26, 26, 32)        320
    _________________________________________________________________
    conv2d_56 (Conv2D)           (None, 24, 24, 64)        18496
    _________________________________________________________________
    max_pooling2d_14 (MaxPooling (None, 12, 12, 64)        0
    _________________________________________________________________
    dropout_30 (Dropout)         (None, 12, 12, 64)        0
    _________________________________________________________________
    conv2d_57 (Conv2D)           (None, 1, 1, 128)         1179776
    _________________________________________________________________
    dropout_31 (Dropout)         (None, 1, 1, 128)         0
    _________________________________________________________________
    conv2d_58 (Conv2D)           (None, 1, 1, 10)          1290
    _________________________________________________________________
    flatten_7 (Flatten)          (None, 10)                0
    =================================================================
    Total params: 1,199,882
    Trainable params: 1,199,882
    Non-trainable params: 0
    _________________________________________________________________

```python
import numpy as np
import matplotlib.pyplot as plt
from tensorflow.keras.datasets import mnist
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.losses import categorical_crossentropy
from tensorflow.keras.optimizers import Adam

# load the MNIST the dataset
(x_train, y_train), (x_test, y_test) = mnist.load_data()

# scale the image intensities to the 0-1 range
x_train = (x_train / 255.0).astype(np.float32)
x_test = (x_test / 255.0).astype(np.float32)

# convert the data to channel-last
x_train = np.expand_dims(x_train, axis=-1)
x_test = np.expand_dims(x_test, axis=-1)

# convert the labels to one-hot encoded
y_train = to_categorical(y_train, num_classes=10)
y_test = to_categorical(y_test, num_classes=10)

model2.compile(loss=categorical_crossentropy, optimizer=Adam(), metrics=['accuracy'])

model2.fit(x_train, y_train,
          batch_size=128,
          epochs=12,
          verbose=1,
          validation_data=(x_test, y_test))

score = model2.evaluate(x_test, y_test, verbose=0)

print('Test loss:', score[0])
print('Test accuracy:', score[1])

```

    Train on 60000 samples, validate on 10000 samples
    Epoch 1/12
    60000/60000 [==============================] - 225s 4ms/sample - loss: 0.2441 - accuracy: 0.9251 - val_loss: 0.0531 - val_accuracy: 0.9830
    Epoch 2/12
    60000/60000 [==============================] - 232s 4ms/sample - loss: 0.0875 - accuracy: 0.9744 - val_loss: 0.0386 - val_accuracy: 0.9878
    Epoch 3/12
    60000/60000 [==============================] - 223s 4ms/sample - loss: 0.0634 - accuracy: 0.9807 - val_loss: 0.0390 - val_accuracy: 0.9863
    Epoch 4/12
    60000/60000 [==============================] - 228s 4ms/sample - loss: 0.0551 - accuracy: 0.9830 - val_loss: 0.0304 - val_accuracy: 0.9891
    Epoch 5/12
    60000/60000 [==============================] - 226s 4ms/sample - loss: 0.0458 - accuracy: 0.9862 - val_loss: 0.0315 - val_accuracy: 0.9891
    Epoch 6/12
    60000/60000 [==============================] - 213s 4ms/sample - loss: 0.0399 - accuracy: 0.9873 - val_loss: 0.0278 - val_accuracy: 0.9910
    Epoch 7/12
    60000/60000 [==============================] - 264s 4ms/sample - loss: 0.0363 - accuracy: 0.9886 - val_loss: 0.0276 - val_accuracy: 0.9914
    Epoch 8/12
    60000/60000 [==============================] - 259s 4ms/sample - loss: 0.0316 - accuracy: 0.9897 - val_loss: 0.0301 - val_accuracy: 0.9910
    Epoch 9/12
    60000/60000 [==============================] - 264s 4ms/sample - loss: 0.0310 - accuracy: 0.9900 - val_loss: 0.0251 - val_accuracy: 0.9923
    Epoch 10/12
    60000/60000 [==============================] - 267s 4ms/sample - loss: 0.0253 - accuracy: 0.9920 - val_loss: 0.0291 - val_accuracy: 0.9922
    Epoch 11/12
    60000/60000 [==============================] - 5323s 89ms/sample - loss: 0.0235 - accuracy: 0.9923 - val_loss: 0.0264 - val_accuracy: 0.9919
    Epoch 12/12
    60000/60000 [==============================] - 251s 4ms/sample - loss: 0.0233 - accuracy: 0.9923 - val_loss: 0.0276 - val_accuracy: 0.9927
    Test loss: 0.02756877655624703
    Test accuracy: 0.9927

## U-Net architecture

<font color='#770a0a'> What is the role of the skip connections in the U-Net neural network architecture? Will it be possible to train the exact same architecture with the skip connections omitted? If yes, what would be the expected result? If no, what would be the cause of the error?</font>

In the architecture of the U-net neural network you first start with downsampling the medical images. After a while you are going to upsample the image(s) again and now it is important to have these skip connections. Because it gives the opportunity to skip features of the contracting part to the expanding path and so it would be possible to recover some spatial information that would be lost during the downsampling procedure.
Another benefit of skip connections in neural networks is that it can help the system with transfer information in a faster way into the deep neural network.
What you can see is that Gradient information could get lost when passing through multiple layers of the DeepNN and this would give difficulties considering the gradient descent. With the skip connections you can pass some information to deeper layers and because of it classifying in more detail becomes easier for the system.

It would be possible to train the exact same architecture without these skip connection, however as mentioned above, some of the spatial information can get loss during the downsampling procedure. When this information is not restored it would affect the result of the model in a negative way. The accuracy of the model will become lower for example.

## Data augmentation

<font color='#770a0a'>Why does data augmentation result in less overfitting? Can data augmentation be applied to the test samples? If yes, towards what goal? If no, what is preventing that?</font>

_Data augmentation is normally applied when there is not much training data available. Normally, when a model is be trained on a small dataset, overfitting occurs. However, when enlarging the training dataset by ‘creating’ new data that is similar to the existing data, the model can be trained on a bigger training set, so the model would not be overfitted._

_Data augmentation applied to the test set is called ‘test time augmentation’. Test time augmentation has the goal to make better predictions of the test set and is not involved in fitting a model. In test time augmentation, the test set is augmented multiple times and the output is averaged over the augmented versions of each test sample to obtain the final prediction._

Implement random brightness augmentation of the image data by adding a random offset to the image intensity before passing them trough the network at training time. Train a model with random brightness augmentation and compare it to the baseline above.

```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from glob import glob
import sys
import os
from tensorflow.keras.models import load_model
from tensorflow.keras.optimizers import Adam, SGD
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, ReduceLROnPlateau
import gryds
sys.path.append('code/')
from unet_model import unet
from unet_utils import load_data, datageneratorbrn, preprocessing

# location of the DRIVE dataset
data_folder = './data/'
train_paths = glob(data_folder + 'training/images/*.tif')
images, masks, segmentations = load_data(train_paths)

# divide in training and validation
train_images, val_images, train_masks, val_masks, train_segmentations, val_segmentations = train_test_split(
    images, masks, segmentations, test_size=0.2, random_state=7)

val_images, val_masks, val_segmentations = preprocessing(
    val_images,
    val_masks,
    val_segmentations,
    desired_shape=(584, 584))

# use a single training image, to better demonstrate the effects of data augmentation
X_train, y_train = np.expand_dims(train_images[0], axis=0), np.expand_dims(train_segmentations[0], axis=0)

patch_size = (32, 32)
patches_per_im = 200

# hyperparameters
depth = 3
channels = 32
use_batchnorm = True
batch_size = 64
epochs = 50
steps_per_epoch = int(np.ceil((patches_per_im * len(train_images)) / batch_size))

# initialize model
modelbrn = unet(input_shape=(None, None, 3), depth=depth, channels=channels, batchnorm=use_batchnorm)

# print a summary of the model
modelbrn.summary(line_length=120)

# compile the model
modelbrn.compile(loss='binary_crossentropy', optimizer=Adam(), metrics=['accuracy'])

# stop the training if the validation loss does not increase for 15 consecutive epochs
early_stopping = EarlyStopping(monitor='val_loss', patience=15, restore_best_weights=True)

# train the model with the data generator, and save the training history
history = modelbrn.fit_generator(datageneratorbrn(X_train, y_train, patch_size, patches_per_im, batch_size),
                              validation_data=(val_images, val_segmentations),
                              steps_per_epoch=steps_per_epoch, epochs=epochs, verbose=2,
                              callbacks=[early_stopping])

```

    Model: "model"
    ________________________________________________________________________________________________________________________
    Layer (type)                           Output Shape               Param #       Connected to
    ========================================================================================================================
    input_1 (InputLayer)                   [(None, None, None, 3)]    0
    ________________________________________________________________________________________________________________________
    conv2d (Conv2D)                        (None, None, None, 32)     896           input_1[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization (BatchNormalizatio (None, None, None, 32)     128           conv2d[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_1 (Conv2D)                      (None, None, None, 32)     9248          batch_normalization[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_1 (BatchNormalizat (None, None, None, 32)     128           conv2d_1[0][0]
    ________________________________________________________________________________________________________________________
    max_pooling2d (MaxPooling2D)           (None, None, None, 32)     0             batch_normalization_1[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_2 (Conv2D)                      (None, None, None, 64)     18496         max_pooling2d[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_2 (BatchNormalizat (None, None, None, 64)     256           conv2d_2[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_3 (Conv2D)                      (None, None, None, 64)     36928         batch_normalization_2[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_3 (BatchNormalizat (None, None, None, 64)     256           conv2d_3[0][0]
    ________________________________________________________________________________________________________________________
    max_pooling2d_1 (MaxPooling2D)         (None, None, None, 64)     0             batch_normalization_3[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_4 (Conv2D)                      (None, None, None, 128)    73856         max_pooling2d_1[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_4 (BatchNormalizat (None, None, None, 128)    512           conv2d_4[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_5 (Conv2D)                      (None, None, None, 128)    147584        batch_normalization_4[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_5 (BatchNormalizat (None, None, None, 128)    512           conv2d_5[0][0]
    ________________________________________________________________________________________________________________________
    max_pooling2d_2 (MaxPooling2D)         (None, None, None, 128)    0             batch_normalization_5[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_6 (Conv2D)                      (None, None, None, 256)    295168        max_pooling2d_2[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_6 (BatchNormalizat (None, None, None, 256)    1024          conv2d_6[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_7 (Conv2D)                      (None, None, None, 256)    590080        batch_normalization_6[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_7 (BatchNormalizat (None, None, None, 256)    1024          conv2d_7[0][0]
    ________________________________________________________________________________________________________________________
    up_sampling2d (UpSampling2D)           (None, None, None, 256)    0             batch_normalization_7[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_8 (Conv2D)                      (None, None, None, 128)    131200        up_sampling2d[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_8 (BatchNormalizat (None, None, None, 128)    512           conv2d_8[0][0]
    ________________________________________________________________________________________________________________________
    concatenate (Concatenate)              (None, None, None, 256)    0             batch_normalization_5[0][0]
                                                                                batch_normalization_8[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_9 (Conv2D)                      (None, None, None, 128)    295040        concatenate[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_9 (BatchNormalizat (None, None, None, 128)    512           conv2d_9[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_10 (Conv2D)                     (None, None, None, 128)    147584        batch_normalization_9[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_10 (BatchNormaliza (None, None, None, 128)    512           conv2d_10[0][0]
    ________________________________________________________________________________________________________________________
    up_sampling2d_1 (UpSampling2D)         (None, None, None, 128)    0             batch_normalization_10[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_11 (Conv2D)                     (None, None, None, 64)     32832         up_sampling2d_1[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_11 (BatchNormaliza (None, None, None, 64)     256           conv2d_11[0][0]
    ________________________________________________________________________________________________________________________
    concatenate_1 (Concatenate)            (None, None, None, 128)    0             batch_normalization_3[0][0]
                                                                                batch_normalization_11[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_12 (Conv2D)                     (None, None, None, 64)     73792         concatenate_1[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_12 (BatchNormaliza (None, None, None, 64)     256           conv2d_12[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_13 (Conv2D)                     (None, None, None, 64)     36928         batch_normalization_12[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_13 (BatchNormaliza (None, None, None, 64)     256           conv2d_13[0][0]
    ________________________________________________________________________________________________________________________
    up_sampling2d_2 (UpSampling2D)         (None, None, None, 64)     0             batch_normalization_13[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_14 (Conv2D)                     (None, None, None, 32)     8224          up_sampling2d_2[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_14 (BatchNormaliza (None, None, None, 32)     128           conv2d_14[0][0]
    ________________________________________________________________________________________________________________________
    concatenate_2 (Concatenate)            (None, None, None, 64)     0             batch_normalization_1[0][0]
                                                                                batch_normalization_14[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_15 (Conv2D)                     (None, None, None, 32)     18464         concatenate_2[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_15 (BatchNormaliza (None, None, None, 32)     128           conv2d_15[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_16 (Conv2D)                     (None, None, None, 32)     9248          batch_normalization_15[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_16 (BatchNormaliza (None, None, None, 32)     128           conv2d_16[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_17 (Conv2D)                     (None, None, None, 1)      33            batch_normalization_16[0][0]
    ========================================================================================================================
    Total params: 1,932,129
    Trainable params: 1,928,865
    Non-trainable params: 3,264
    ________________________________________________________________________________________________________________________
    Epoch 1/50
    50/50 - 145s - loss: 0.5472 - accuracy: 0.8020 - val_loss: 0.4381 - val_accuracy: 0.9211
    Epoch 2/50
    50/50 - 140s - loss: 0.3439 - accuracy: 0.9466 - val_loss: 0.4688 - val_accuracy: 0.9216
    Epoch 3/50
    50/50 - 156s - loss: 0.2334 - accuracy: 0.9639 - val_loss: 0.2722 - val_accuracy: 0.9211
    Epoch 4/50
    50/50 - 141s - loss: 0.1769 - accuracy: 0.9667 - val_loss: 0.3350 - val_accuracy: 0.9211
    Epoch 5/50
    50/50 - 145s - loss: 0.1491 - accuracy: 0.9654 - val_loss: 0.2802 - val_accuracy: 0.9210
    Epoch 6/50
    50/50 - 150s - loss: 0.1231 - accuracy: 0.9697 - val_loss: 0.3033 - val_accuracy: 0.9213
    Epoch 7/50
    50/50 - 140s - loss: 0.1057 - accuracy: 0.9706 - val_loss: 0.2912 - val_accuracy: 0.9218
    Epoch 8/50
    50/50 - 159s - loss: 0.1021 - accuracy: 0.9704 - val_loss: 0.3028 - val_accuracy: 0.9259
    Epoch 9/50
    50/50 - 142s - loss: 0.0939 - accuracy: 0.9707 - val_loss: 0.2987 - val_accuracy: 0.9278
    Epoch 10/50
    50/50 - 140s - loss: 0.1168 - accuracy: 0.9660 - val_loss: 4.9800 - val_accuracy: 0.4772
    Epoch 11/50
    50/50 - 162s - loss: 0.0980 - accuracy: 0.9690 - val_loss: 0.2086 - val_accuracy: 0.9321
    Epoch 12/50
    50/50 - 170s - loss: 0.0857 - accuracy: 0.9723 - val_loss: 0.2276 - val_accuracy: 0.9337
    Epoch 13/50
    50/50 - 143s - loss: 0.0868 - accuracy: 0.9712 - val_loss: 0.2118 - val_accuracy: 0.9348
    Epoch 14/50
    50/50 - 152s - loss: 0.0792 - accuracy: 0.9731 - val_loss: 0.1975 - val_accuracy: 0.9386
    Epoch 15/50
    50/50 - 155s - loss: 0.0802 - accuracy: 0.9720 - val_loss: 0.1915 - val_accuracy: 0.9400
    Epoch 16/50
    50/50 - 143s - loss: 0.0749 - accuracy: 0.9739 - val_loss: 0.2132 - val_accuracy: 0.9396
    Epoch 17/50
    50/50 - 138s - loss: 0.0727 - accuracy: 0.9740 - val_loss: 0.1860 - val_accuracy: 0.9446
    Epoch 18/50
    50/50 - 138s - loss: 0.0701 - accuracy: 0.9750 - val_loss: 0.1829 - val_accuracy: 0.9460
    Epoch 19/50
    50/50 - 139s - loss: 0.0707 - accuracy: 0.9741 - val_loss: 0.1657 - val_accuracy: 0.9480
    Epoch 20/50
    50/50 - 138s - loss: 0.0697 - accuracy: 0.9748 - val_loss: 0.1682 - val_accuracy: 0.9460
    Epoch 21/50
    50/50 - 138s - loss: 0.0682 - accuracy: 0.9750 - val_loss: 0.1605 - val_accuracy: 0.9478
    Epoch 22/50
    50/50 - 138s - loss: 0.0649 - accuracy: 0.9760 - val_loss: 0.1712 - val_accuracy: 0.9462
    Epoch 23/50
    50/50 - 138s - loss: 0.0639 - accuracy: 0.9759 - val_loss: 0.1976 - val_accuracy: 0.9411
    Epoch 24/50
    50/50 - 140s - loss: 0.0628 - accuracy: 0.9764 - val_loss: 0.2120 - val_accuracy: 0.9425
    Epoch 25/50
    50/50 - 138s - loss: 0.0648 - accuracy: 0.9753 - val_loss: 0.1613 - val_accuracy: 0.9457
    Epoch 26/50
    50/50 - 150s - loss: 0.0606 - accuracy: 0.9767 - val_loss: 0.1871 - val_accuracy: 0.9360
    Epoch 27/50
    50/50 - 229s - loss: 0.0596 - accuracy: 0.9769 - val_loss: 0.2028 - val_accuracy: 0.9328
    Epoch 28/50
    50/50 - 226s - loss: 0.0570 - accuracy: 0.9779 - val_loss: 0.1812 - val_accuracy: 0.9433
    Epoch 29/50
    50/50 - 238s - loss: 0.0586 - accuracy: 0.9770 - val_loss: 0.2662 - val_accuracy: 0.9238
    Epoch 30/50
    50/50 - 233s - loss: 0.0565 - accuracy: 0.9779 - val_loss: 0.2764 - val_accuracy: 0.8981
    Epoch 31/50
    50/50 - 239s - loss: 0.0571 - accuracy: 0.9774 - val_loss: 0.2300 - val_accuracy: 0.9137
    Epoch 32/50
    50/50 - 233s - loss: 0.0535 - accuracy: 0.9788 - val_loss: 0.1633 - val_accuracy: 0.9488
    Epoch 33/50
    50/50 - 227s - loss: 0.0565 - accuracy: 0.9775 - val_loss: 0.2037 - val_accuracy: 0.9383
    Epoch 34/50
    50/50 - 228s - loss: 0.0535 - accuracy: 0.9788 - val_loss: 0.1810 - val_accuracy: 0.9432
    Epoch 35/50
    50/50 - 231s - loss: 0.0523 - accuracy: 0.9790 - val_loss: 0.1722 - val_accuracy: 0.9450
    Epoch 36/50
    50/50 - 230s - loss: 0.0492 - accuracy: 0.9803 - val_loss: 0.1942 - val_accuracy: 0.9347

    <Figure size 1200x1000 with 3 Axes>

```python
# run the model on one test image and show the results
# test data paths
impaths_test = glob(data_folder + 'test/images/*.tif')

# load data
test_images, test_masks, test_segmentations = load_data(impaths_test, test=True)

# pad the data to fit the U-Net model
test_images, test_masks, test_segmentations = preprocessing(test_images, test_masks, test_segmentations,
                                                            desired_shape=(584, 584))

# use a single image to evaluate
X_test, y_test = np.expand_dims(test_images[0], axis=0), np.expand_dims(test_masks[0], axis=0)

# predict test samples
test_prediction = modelbrn.predict(X_test, batch_size=4)

# visualize the test result
plt.figure(figsize=(12, 10))

plt.subplot(1, 3, 1)
plt.title("Image of the retina")
plt.axis('off')
plt.imshow(test_images[0])

plt.subplot(1, 3, 2)
plt.title("Ground truth vessel segmentation")
plt.axis('off')
plt.imshow(test_segmentations[0][:, :, 0])

plt.subplot(1, 3, 3)
plt.title("Predicted vessel segmentation")
plt.axis('off')
plt.imshow(test_prediction[0, :, :, 0])

plt.show()
```

It is expected that with an added augmentation, the model would improve. However, when the result from the brightness augmentation is compared to the previous result without brightness augmentation, not much of a difference is observed.

Implement data augmentation procedure that in addition to brightness augmentation also performs b-spline geometric augmentation using the [`gryds`](https://github.com/tueimage/gryds) package (you can look at the documentation of the package for an example on how to do that). Compare the new model with the baseline and the model that only performs brightness augmentation.

```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from glob import glob
import sys
import os
from tensorflow.keras.models import load_model
from tensorflow.keras.optimizers import Adam, SGD
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, ReduceLROnPlateau
import gryds
sys.path.append('code/')
from unet_model import unet
from unet_utils import load_data, datagen_brn_bspline, preprocessing

# location of the DRIVE dataset
data_folder = './data/'
train_paths = glob(data_folder + 'training/images/*.tif')
images, masks, segmentations = load_data(train_paths)

# divide in training and validation
train_images, val_images, train_masks, val_masks, train_segmentations, val_segmentations = train_test_split(
    images, masks, segmentations, test_size=0.2, random_state=7)

val_images, val_masks, val_segmentations = preprocessing(
    val_images,
    val_masks,
    val_segmentations,
    desired_shape=(584, 584))

# use a single training image, to better demonstrate the effects of data augmentation
X_train, y_train = np.expand_dims(train_images[0], axis=0), np.expand_dims(train_segmentations[0], axis=0)

patch_size = (32, 32)
patches_per_im = 200

# hyperparameters
depth = 3
channels = 32
use_batchnorm = True
batch_size = 64
epochs = 20
steps_per_epoch = int(np.ceil((patches_per_im * len(train_images)) / batch_size))

# initialize model
modelb_spline = unet(input_shape=(None, None, 3), depth=depth, channels=channels, batchnorm=use_batchnorm)

# print a summary of the model
modelb_spline.summary(line_length=120)

# compile the model
modelb_spline.compile(loss='binary_crossentropy', optimizer=Adam(), metrics=['accuracy'])

# stop the training if the validation loss does not increase for 15 consecutive epochs
early_stopping = EarlyStopping(monitor='val_loss', patience=15, restore_best_weights=True)

# train the model with the data generator, and save the training history
history = modelb_spline.fit_generator(datagen_brn_bspline(X_train, y_train, patch_size, patches_per_im, batch_size),
                              validation_data=(val_images, val_segmentations),
                              steps_per_epoch=steps_per_epoch, epochs=epochs, verbose=2,
                              callbacks=[early_stopping])

```

    Model: "model_2"
    ________________________________________________________________________________________________________________________
    Layer (type)                           Output Shape               Param #       Connected to
    ========================================================================================================================
    input_3 (InputLayer)                   [(None, None, None, 3)]    0
    ________________________________________________________________________________________________________________________
    conv2d_36 (Conv2D)                     (None, None, None, 32)     896           input_3[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_34 (BatchNormaliza (None, None, None, 32)     128           conv2d_36[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_37 (Conv2D)                     (None, None, None, 32)     9248          batch_normalization_34[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_35 (BatchNormaliza (None, None, None, 32)     128           conv2d_37[0][0]
    ________________________________________________________________________________________________________________________
    max_pooling2d_6 (MaxPooling2D)         (None, None, None, 32)     0             batch_normalization_35[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_38 (Conv2D)                     (None, None, None, 64)     18496         max_pooling2d_6[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_36 (BatchNormaliza (None, None, None, 64)     256           conv2d_38[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_39 (Conv2D)                     (None, None, None, 64)     36928         batch_normalization_36[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_37 (BatchNormaliza (None, None, None, 64)     256           conv2d_39[0][0]
    ________________________________________________________________________________________________________________________
    max_pooling2d_7 (MaxPooling2D)         (None, None, None, 64)     0             batch_normalization_37[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_40 (Conv2D)                     (None, None, None, 128)    73856         max_pooling2d_7[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_38 (BatchNormaliza (None, None, None, 128)    512           conv2d_40[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_41 (Conv2D)                     (None, None, None, 128)    147584        batch_normalization_38[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_39 (BatchNormaliza (None, None, None, 128)    512           conv2d_41[0][0]
    ________________________________________________________________________________________________________________________
    max_pooling2d_8 (MaxPooling2D)         (None, None, None, 128)    0             batch_normalization_39[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_42 (Conv2D)                     (None, None, None, 256)    295168        max_pooling2d_8[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_40 (BatchNormaliza (None, None, None, 256)    1024          conv2d_42[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_43 (Conv2D)                     (None, None, None, 256)    590080        batch_normalization_40[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_41 (BatchNormaliza (None, None, None, 256)    1024          conv2d_43[0][0]
    ________________________________________________________________________________________________________________________
    up_sampling2d_6 (UpSampling2D)         (None, None, None, 256)    0             batch_normalization_41[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_44 (Conv2D)                     (None, None, None, 128)    131200        up_sampling2d_6[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_42 (BatchNormaliza (None, None, None, 128)    512           conv2d_44[0][0]
    ________________________________________________________________________________________________________________________
    concatenate_6 (Concatenate)            (None, None, None, 256)    0             batch_normalization_39[0][0]
                                                                                batch_normalization_42[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_45 (Conv2D)                     (None, None, None, 128)    295040        concatenate_6[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_43 (BatchNormaliza (None, None, None, 128)    512           conv2d_45[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_46 (Conv2D)                     (None, None, None, 128)    147584        batch_normalization_43[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_44 (BatchNormaliza (None, None, None, 128)    512           conv2d_46[0][0]
    ________________________________________________________________________________________________________________________
    up_sampling2d_7 (UpSampling2D)         (None, None, None, 128)    0             batch_normalization_44[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_47 (Conv2D)                     (None, None, None, 64)     32832         up_sampling2d_7[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_45 (BatchNormaliza (None, None, None, 64)     256           conv2d_47[0][0]
    ________________________________________________________________________________________________________________________
    concatenate_7 (Concatenate)            (None, None, None, 128)    0             batch_normalization_37[0][0]
                                                                                batch_normalization_45[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_48 (Conv2D)                     (None, None, None, 64)     73792         concatenate_7[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_46 (BatchNormaliza (None, None, None, 64)     256           conv2d_48[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_49 (Conv2D)                     (None, None, None, 64)     36928         batch_normalization_46[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_47 (BatchNormaliza (None, None, None, 64)     256           conv2d_49[0][0]
    ________________________________________________________________________________________________________________________
    up_sampling2d_8 (UpSampling2D)         (None, None, None, 64)     0             batch_normalization_47[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_50 (Conv2D)                     (None, None, None, 32)     8224          up_sampling2d_8[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_48 (BatchNormaliza (None, None, None, 32)     128           conv2d_50[0][0]
    ________________________________________________________________________________________________________________________
    concatenate_8 (Concatenate)            (None, None, None, 64)     0             batch_normalization_35[0][0]
                                                                                batch_normalization_48[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_51 (Conv2D)                     (None, None, None, 32)     18464         concatenate_8[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_49 (BatchNormaliza (None, None, None, 32)     128           conv2d_51[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_52 (Conv2D)                     (None, None, None, 32)     9248          batch_normalization_49[0][0]
    ________________________________________________________________________________________________________________________
    batch_normalization_50 (BatchNormaliza (None, None, None, 32)     128           conv2d_52[0][0]
    ________________________________________________________________________________________________________________________
    conv2d_53 (Conv2D)                     (None, None, None, 1)      33            batch_normalization_50[0][0]
    ========================================================================================================================
    Total params: 1,932,129
    Trainable params: 1,928,865
    Non-trainable params: 3,264
    ________________________________________________________________________________________________________________________
    Epoch 1/20
    50/50 - 395s - loss: 0.6380 - accuracy: 0.7521 - val_loss: 0.5306 - val_accuracy: 0.9211
    Epoch 2/20
    50/50 - 352s - loss: 0.4915 - accuracy: 0.9060 - val_loss: 0.4085 - val_accuracy: 0.9211
    Epoch 3/20
    50/50 - 371s - loss: 0.3844 - accuracy: 0.9097 - val_loss: 0.3237 - val_accuracy: 0.9211
    Epoch 4/20
    50/50 - 3069s - loss: 0.3244 - accuracy: 0.9099 - val_loss: 0.2899 - val_accuracy: 0.9211
    Epoch 5/20
    50/50 - 385s - loss: 0.3059 - accuracy: 0.9096 - val_loss: 0.2854 - val_accuracy: 0.9211
    Epoch 6/20
    50/50 - 357s - loss: 0.3081 - accuracy: 0.9089 - val_loss: 0.3355 - val_accuracy: 0.9166
    Epoch 7/20
    50/50 - 2851s - loss: 0.3157 - accuracy: 0.9061 - val_loss: 0.6004 - val_accuracy: 0.6025
    Epoch 8/20
    50/50 - 408s - loss: 0.2980 - accuracy: 0.9073 - val_loss: 0.4422 - val_accuracy: 0.7894
    Epoch 9/20
    50/50 - 425s - loss: 0.3099 - accuracy: 0.9105 - val_loss: 2.9626 - val_accuracy: 0.4017
    Epoch 10/20
    50/50 - 359s - loss: 0.3095 - accuracy: 0.9079 - val_loss: 5.9285 - val_accuracy: 0.3924
    Epoch 11/20
    50/50 - 374s - loss: 0.2995 - accuracy: 0.9104 - val_loss: 9.0382 - val_accuracy: 0.3779
    Epoch 12/20
    50/50 - 356s - loss: 0.3120 - accuracy: 0.9080 - val_loss: 9.2764 - val_accuracy: 0.3508
    Epoch 13/20
    50/50 - 373s - loss: 0.2987 - accuracy: 0.9079 - val_loss: 9.3900 - val_accuracy: 0.3276
    Epoch 14/20
    50/50 - 249s - loss: 0.3075 - accuracy: 0.9081 - val_loss: 9.0984 - val_accuracy: 0.3769
    Epoch 15/20
    50/50 - 236s - loss: 0.2878 - accuracy: 0.9128 - val_loss: 9.7371 - val_accuracy: 0.2114
    Epoch 16/20
    50/50 - 240s - loss: 0.3001 - accuracy: 0.9075 - val_loss: 9.6621 - val_accuracy: 0.2986
    Epoch 17/20
    50/50 - 263s - loss: 0.3165 - accuracy: 0.9074 - val_loss: 9.7547 - val_accuracy: 0.2955
    Epoch 18/20
    50/50 - 255s - loss: 0.3167 - accuracy: 0.9038 - val_loss: 9.8780 - val_accuracy: 0.2582
    Epoch 19/20
    50/50 - 251s - loss: 0.2933 - accuracy: 0.9101 - val_loss: 10.1090 - val_accuracy: 0.1268
    Epoch 20/20
    50/50 - 257s - loss: 0.3174 - accuracy: 0.9038 - val_loss: 9.9277 - val_accuracy: 0.1358

```python
# run the model on one test image and show the results
# test data paths
impaths_test = glob(data_folder + 'test/images/*.tif')

# load data
test_images, test_masks, test_segmentations = load_data(impaths_test, test=True)

# pad the data to fit the U-Net model
test_images, test_masks, test_segmentations = preprocessing(test_images, test_masks, test_segmentations,
                                                            desired_shape=(584, 584))

# use a single image to evaluate
X_test, y_test = np.expand_dims(test_images[0], axis=0), np.expand_dims(test_masks[0], axis=0)

# predict test samples
test_prediction = modelb_spline.predict(X_test, batch_size=4)

# visualize the test result
plt.figure(figsize=(12, 10))

plt.subplot(1, 3, 1)
plt.title("Image of the retina")
plt.axis('off')
plt.imshow(test_images[0])

plt.subplot(1, 3, 2)
plt.title("Ground truth vessel segmentation")
plt.axis('off')
plt.imshow(test_segmentations[0][:, :, 0])

plt.subplot(1, 3, 3)
plt.title("Predicted vessel segmentation")
plt.axis('off')
plt.imshow(test_prediction[0, :, :, 0])

plt.show()
```

Training the model here was done for 20 epochs to reduce running time (as a trade-off with model quality). It was expected that this model, that contained three types of data augmentation; 200 patches from a single image as an input, random shift augmentation and b-spline deformation, would give the most accurate segmentation. The more augmentation is applied, the more accurate the model should become. In this case, this is not what happened, the segmentation failed completely. Several reasons for this might be the low nummber of epochs that the model was run. From the summary per epoch, it is also seen that the validation accuracy decreased with every epoch, which also shouldn't have happened. The opposite should occur, such that the model would improve with every epoch and the validation error should get close to 1.

Bspline was executed on the whole image right before it got a random brightness shift. The image variable (X_train) at this point had 4 dimensions, the first one was was the number of images (which was 1), the second and third dimensions were the x and y pixels of the image and the fourth dimension had 3 layers in which all (3)RGB values of the image pixels were stored. Due to this dimensionality the bspline may not have worked properly, therefore causing this faulty prediction.

```python

```


# PIT Summary

# Purpose
There has been a lot done in the parameter identification techniques (PIT) in this project, this notebook is a summary.

# Setup

```python
# %load imports.py
# %load imports.py
%matplotlib inline
%load_ext autoreload
%autoreload 2
%config Completer.use_jedi = False  ## (To fix autocomplete)

## External packages:
import pandas as pd
pd.options.display.max_rows = 999
pd.options.display.max_columns = 999
pd.set_option("display.max_columns", None)

import numpy as np
import os
import matplotlib.pyplot as plt
#if os.name == 'nt':
#    plt.style.use('presentation.mplstyle')  # Windows

import plotly.express as px
import plotly.graph_objects as go

import seaborn as sns
import sympy as sp
from sympy.physics.mechanics import (dynamicsymbols, ReferenceFrame,
                                      Particle, Point)
from sympy.physics.vector.printing import vpprint, vlatex
from IPython.display import display, Math, Latex
from src.substitute_dynamic_symbols import run, lambdify

#import pyro

import sklearn
import pykalman
from statsmodels.sandbox.regression.predstd import wls_prediction_std
import statsmodels.api as sm

from scipy.integrate import solve_ivp

## Local packages:
from src.data import mdl
#import src.models.nonlinear_martin_vmm as vmm
#import src.nonlinear_martin_vmm_equations as eq

#import src.models.linear_vmm as vmm
import src.linear_vmm_equations as eq

#import src.models.linear_vmm as model
from src.symbols import *
from src.parameters import *
import src.symbols as symbols
from src import prime_system
from src.models import regression
from src.visualization.plot import track_plot
from src.equation import Equation
```

```python
Math(vlatex(eq.X_eq))
```

```python
Math(vlatex(eq.Y_eq))
```

```python
Math(vlatex(eq.N_eq))
```

```python
Math(vlatex(eq.X_eq.rhs-eq.X_eq.lhs))
```

```python
Math(vlatex(eq.Y_eq.rhs-eq.Y_eq.lhs))
```

```python
Math(vlatex(eq.N_eq.rhs-eq.N_eq.lhs))
```

## Load test

```python
#id=22773
#id=22616
id=22774
#id=22770

df, units, meta_data = mdl.load(id=id, dir_path='../data/processed/kalman')
df.index = df.index.total_seconds()
df = df.iloc[0:-100].copy()
df.index-=df.index[0]
df['t'] = df.index
df.sort_index(inplace=True)
df['-delta'] = -df['delta']
df['V'] = np.sqrt(df['u']**2 + df['v']**2)

df['thrust'] = df['Prop/PS/Thrust'] + df['Prop/SB/Thrust']
df['U'] = df['V']
df['beta'] = -np.arctan2(df['v'],df['u'])
```

```python
meta_data['rho']=1000
meta_data['mass'] = meta_data['Volume']*meta_data['rho']
```

```python
from src.visualization.plot import track_plot
fig,ax=plt.subplots()
#fig.set_size_inches(10,10)
track_plot(df=df, lpp=meta_data.lpp, x_dataset='x0', y_dataset='y0',  psi_dataset='psi', beam=meta_data.beam, ax=ax);
df.plot(y='u')
```

# Ship parameters

```python
T_ = (meta_data.TA + meta_data.TF)/2
L_ = meta_data.lpp
m_ = meta_data.mass
rho_ = meta_data.rho
B_ = meta_data.beam
CB_ = m_/(T_*B_*L_*rho_)
I_z_ = m_*meta_data.KZZ**2
#I_z_=839.725

ship_parameters = {
        'T' : T_,
        'L' : L_,
        'CB' :CB_,
        'B' : B_,
        'rho' : rho_,
        #'x_G' : meta_data.lcg,  # motions are expressed at CG
        'x_G' : 0,  # motions are expressed at CG

        'm' : m_,
        'I_z': I_z_,
        'volume':meta_data.Volume,
    }

ps = prime_system.PrimeSystem(**ship_parameters)  # model

scale_factor = meta_data.scale_factor
ps_ship = prime_system.PrimeSystem(L=ship_parameters['L']*scale_factor, rho=meta_data['rho'])  # ship
ship_parameters_prime = ps.prime(ship_parameters)
```

```python
I_z_+m_*meta_data.lcg**2  # Steiner rule...
```

```python
I_z_
```

```python
ship_parameters
```

```python
ship_parameters_prime
```

## Prime system

```python
interesting = ['x0','y0','psi','u','v','r','u1d','v1d','r1d','U','t','delta','thrust','beta']
df_prime = ps.prime(df[interesting], U=df['U'])
df_prime.set_index('t', inplace=True)
```

```python
fig,ax=plt.subplots()
#fig.set_size_inches(10,10)
track_plot(df=df_prime, lpp=ship_parameters_prime['L'], beam=ship_parameters_prime['B'],
           x_dataset='x0', y_dataset='y0',  psi_dataset='psi', ax=ax);

df_prime.plot(y='u')
```

```python
df.index
```

```python
df_prime.index
```

```python
t_ = np.array([0,1,2])
U_ = np.array([1,2,2])
t_prime = ps._prime(t_, unit='time', U=U_)
t_prime
```

```python
ps._unprime(t_prime, unit='time', U=U_)
```

# Brix parameters

```python
def calculate_prime(row, ship_parameters):
    return run(function=row['brix_lambda'], inputs=ship_parameters)

mask = df_parameters['brix_lambda'].notnull()
df_parameters.loc[mask,'brix_prime'] = df_parameters.loc[mask].apply(calculate_prime, ship_parameters=ship_parameters, axis=1)

df_parameters.loc['Ydelta','brix_prime'] = 0.0004  # Just guessing
df_parameters.loc['Ndelta','brix_prime'] = -df_parameters.loc['Ydelta','brix_prime']/4  # Just guessing

df_parameters['brix_prime'].fillna(0, inplace=True)
#df_parameters['brix_SI'].fillna(0, inplace=True)

```

## Simulate with Brix

```python
X_eq = eq.X_eq.copy()
Y_eq = eq.Y_eq.copy()
N_eq = eq.N_eq.copy()

subs=[
    #(x_G,0),
    (eq.p.Xvdot,0),
    (eq.p.Xrdot,0),
    (eq.p.Yudot,0),
    (eq.p.Yrdot,0),
    (eq.p.Nudot,0),
    (eq.p.Nvdot,0),
]

X_eq = X_eq.subs(subs)
Y_eq = Y_eq.subs(subs)
N_eq = N_eq.subs(subs)

eqs = [X_eq, Y_eq, N_eq]
solution = sp.solve(eqs, u1d, v1d, r1d, dict=True)

## Decouple the equations:
u1d_eq = sp.Eq(u1d, solution[0][u1d])
v1d_eq = sp.Eq(v1d, solution[0][v1d])
r1d_eq = sp.Eq(r1d, solution[0][r1d])

## Lambdify:
subs = {value:key for key,value in eq.p.items()}
u1d_lambda = lambdify(u1d_eq.subs(subs).rhs)
v1d_lambda = lambdify(v1d_eq.subs(subs).rhs)
r1d_lambda = lambdify(r1d_eq.subs(subs).rhs)

```

```python
from scipy.spatial.transform import Rotation as R

def step(t, states, parameters, ship_parameters, control):

    u,v,r,x0,y0,psi = states
    states_dict = {
        'u':u,
        'v':v,
        'r':r,
        'x0':x0,
        'y0':y0,
        'psi':psi,

        }

    inputs = dict(parameters)
    inputs.update(ship_parameters)
    inputs.update(states_dict)

    if isinstance(control, pd.DataFrame):
        index = np.argmin(np.array(np.abs(control.index - t)))
        control_ = dict(control.iloc[index])
    else:
        control_ = control
    inputs.update(control_)

    inputs['U'] = np.sqrt(u**2 + v**2)  #Instantanious velocity
    u1d = run(function=u1d_lambda, inputs=inputs)
    v1d = run(function=v1d_lambda, inputs=inputs)
    r1d = run(function=r1d_lambda, inputs=inputs)
    rotation = R.from_euler('z', psi, degrees=False)
    w = 0
    velocities = rotation.apply([u,v,w])
    x01d = velocities[0]
    y01d = velocities[1]
    psi1d = r
    dstates = [
        u1d,
        v1d,
        r1d,
        x01d,
        y01d,
        psi1d,
    ]
    return dstates
```

```python
fig,ax=plt.subplots()
df_prime.plot(y='delta', ax=ax)
df_cut_prime = df_prime.iloc[2000:12000]
df_cut_prime.plot(y='delta', ax=ax, style='--', label='cut')
```

```python
def simulate(df_, parameters, ship_parameters):

    t = df_.index
    t_span = [t.min(),t.max()]
    t_eval = np.linspace(t.min(),t.max(),len(t))

    #control = df_[['delta','thrust']]
    control = {
        'delta': df_[['delta']].mean()
    }

    df_0 = df_.iloc[0:100].median(axis=0)
    y0 = {
        'u' : df_0['u'],
        'v' : df_0['v'],
        'r' : df_0['r'],
        'x0' : df_0['x0'],
        'y0' : df_0['y0'],
        'psi' : df_0['psi']
        }

    solution = solve_ivp(fun=step, t_span=t_span, y0=list(y0.values()), t_eval=t_eval,
                args=(parameters, ship_parameters, control))

    columns = list(y0.keys())
    df_result = pd.DataFrame(data=solution.y.T, columns=columns)
    df_result.index=t[0:len(df_result)]

    df_result['beta'] = -np.arctan2(df_result['v'],df_result['u'])

    return solution, df_result
```

```python
solution, df_result_brix = simulate(df_cut_prime, parameters = df_parameters['brix_prime'], ship_parameters=ship_parameters_prime)
```

```python
fig,ax=plt.subplots()
track_plot(df=df_cut_prime, lpp=ship_parameters_prime['L'], beam=ship_parameters_prime['B'],ax=ax, label='model test')
track_plot(df=df_result_brix, lpp=ship_parameters_prime['L'], beam=ship_parameters_prime['B'],ax=ax, label='simulation', color='green')
ax.legend()

for key in df_result_brix:
    fig,ax = plt.subplots()
    df_cut_prime.plot(y=key, label='model test', ax=ax)
    df_result_brix.plot(y=key, label='simulation', ax=ax)
    ax.set_ylabel(key)
```

## Back to SI

```python
fig,ax=plt.subplots()
ax.plot(df.index,df_prime.index)
```

```python
U_ = ship_parameters['L']*df_prime.index/df.index
df_unprime = ps.unprime(df_prime, U=U_)
df_unprime.index = ps._unprime(df_prime.index,unit='time',U=U_)
```

```python
fig,ax=plt.subplots()
#fig.set_size_inches(10,10)
track_plot(df=df, lpp=meta_data.lpp, x_dataset='x0', y_dataset='y0',  psi_dataset='psi', beam=meta_data.beam, ax=ax);
track_plot(df=df_unprime, lpp=meta_data.lpp, x_dataset='x0', y_dataset='y0',  psi_dataset='psi', beam=meta_data.beam, ax=ax);

fig,ax=plt.subplots()
df.plot(y='u',ax=ax)
df_unprime.plot(y='u', style='--', ax=ax)

fig,ax=plt.subplots()
df.plot(y='v',ax=ax)
df_unprime.plot(y='v', style='--', ax=ax)
```

# VCT regression

## Load VCT data

```python
df_VCT_all = pd.read_csv('../data/external/vct.csv', index_col=0)
df_VCT_all.head()
```

```python
df_VCT = df_VCT_all.groupby(by=['model_name']).get_group('V2_5_MDL_modelScale')
```

```python
df_VCT['test type'].unique()
```

# Subtract the resistance

```python
df_resistance = df_VCT.groupby(by='test type').get_group('resistance')

X = df_resistance[['u','fx']].copy()
X['u**2'] = X['u']**2
y = X.pop('fx')

model_resistance = sm.OLS(y,X)
results_resistance = model_resistance.fit()

X_pred = pd.DataFrame()
X_pred['u'] = np.linspace(X['u'].min(), X['u'].max(), 20)
X_pred['u**2'] = X_pred['u']**2
X_pred['fx'] = results_resistance.predict(X_pred)

fig,ax=plt.subplots()
df_resistance.plot(x='u', y='fx', style='.', ax=ax)
X_pred.plot(x='u', y='fx', style='--', ax=ax);

```

```python
df_VCT_0_resistance = df_VCT.copy()
df_VCT_0_resistance['u**2'] = df_VCT_0_resistance['u']**2
df_VCT_0_resistance['fx']-= results_resistance.predict(df_VCT_0_resistance[['u','u**2']])
```

## VCT to prime system

```python
interesting = [
    'u',
    'v',
    'r',
    'delta',
    'fx',
    'fy',
    'mz',
    'thrust',
]
df_VCT_prime = ps_ship.prime(df_VCT_0_resistance[interesting], U=df_VCT_0_resistance['V'])
```

```python
from statsmodels.sandbox.regression.predstd import wls_prediction_std
def show_pred_vct(X,y,results, label):

    display(results.summary())

    X_ = X.copy()
    X_['y'] = y
    X_.sort_values(by='y', inplace=True)

    y_ = X_.pop('y')

    y_pred = results.predict(X_)

    prstd, iv_l, iv_u = wls_prediction_std(results, exog=X_, alpha=0.05)
    #iv_l*=-1
    #iv_u*=-1

    fig,ax=plt.subplots()
    #ax.plot(X_.index,y_, label='Numerical gradient from model test')
    #ax.plot(X_.index,y_pred, '--', label='OLS')

    ax.plot(y_,y_pred, '.')
    ax.plot([y_.min(),y_.max()], [y_.min(),y_.max()], 'r-')

    ax.set_ylabel(f'{label} (prediction)')
    ax.set_xlabel(label)

    ax.fill_between(y_, y1=iv_l, y2=iv_u, zorder=-10, color='grey', alpha=0.5, label=r'5% confidence')
    ax.legend();
```

## N

```python
eq.N_qs_eq
```

```python
label = sp.symbols('N_qs')
N_eq_ = eq.N_qs_eq.subs(N_qs,label)
diff_eq_N = regression.DiffEqToMatrix(ode=N_eq_, label=label, base_features=[delta,u,v,r])
```

```python
Math(vlatex(diff_eq_N.acceleration_equation))
```

```python
X = diff_eq_N.calculate_features(data=df_VCT_prime)
y = diff_eq_N.calculate_label(y=df_VCT_prime['mz'])

model_N = sm.OLS(y,X)
results_N = model_N.fit()

show_pred_vct(X=X,y=y,results=results_N, label=r'$N$')
```

## Y

```python
eq.Y_qs_eq
```

```python
label = sp.symbols('Y_qs')
Y_eq_ = eq.Y_qs_eq.subs(Y_qs,label)
diff_eq_Y = regression.DiffEqToMatrix(ode=Y_eq_, label=label, base_features=[delta,u,v,r])
```

```python
Math(vlatex(diff_eq_Y.acceleration_equation))
```

```python
X = diff_eq_Y.calculate_features(data=df_VCT_prime)
y = diff_eq_Y.calculate_label(y=df_VCT_prime['fy'])

model_Y = sm.OLS(y,X)
results_Y = model_Y.fit()

show_pred_vct(X=X,y=y,results=results_Y, label=r'$Y$')
```

## X

```python
eq.X_qs_eq
```

```python
label = sp.symbols('X_qs')
X_eq_ = eq.X_qs_eq.subs(X_qs,label)
diff_eq_X = regression.DiffEqToMatrix(ode=X_eq_, label=label, base_features=[delta,u,v,r])
```

```python
Math(vlatex(diff_eq_X.acceleration_equation))
```

```python
X = diff_eq_X.calculate_features(data=df_VCT_prime)
y = diff_eq_X.calculate_label(y=df_VCT_prime['fx'])

model_X = sm.OLS(y,X)
results_X = model_X.fit()

show_pred_vct(X=X,y=y,results=results_X, label=r'$X$')
```

```python
results_summary_X = regression.results_summary_to_dataframe(results_X)
results_summary_Y = regression.results_summary_to_dataframe(results_Y)
results_summary_N = regression.results_summary_to_dataframe(results_N)
```

## Add the regressed parameters
Hydrodynamic derivatives that depend on acceleration cannot be obtained from the VCT regression. They are however essential if a time simulation should be conducted. These values have then been taken from Brix semi empirical formulas for the simulations below.

```python
df_parameters_all = df_parameters.copy()
for other in [results_summary_X, results_summary_Y, results_summary_N]:
    df_parameters_all = df_parameters_all.combine_first(other)

df_parameters_all.rename(columns={'coeff':'regressed'}, inplace=True)
df_parameters_all.drop(columns=['brix_lambda'], inplace=True)

df_parameters_all['prime'] = df_parameters_all['regressed'].combine_first(df_parameters_all['brix_prime'])  # prefer regressed
```

```python
fig,ax=plt.subplots()
fig.set_size_inches(15,5)

mask = ((df_parameters_all['brix_prime']!=0) |
        (pd.notnull(df_parameters_all['regressed'])))

df_parameters_all_plot = df_parameters_all.loc[mask]
df_parameters_all_plot.plot.bar(y=['brix_prime','regressed'], ax=ax);
```

## Simulate

```python
solution, df_result_VCT = simulate(df_cut_prime, parameters = df_parameters_all['prime'], ship_parameters=ship_parameters_prime)
```

```python
fig,ax=plt.subplots()
track_plot(df=df_cut_prime, lpp=ship_parameters_prime['L'], beam=ship_parameters_prime['B'],ax=ax, label='model test')
track_plot(df=df_result_VCT, lpp=ship_parameters_prime['L'], beam=ship_parameters_prime['B'],ax=ax, label='simulation', color='green')
ax.legend()

for key in df_result_VCT:
    fig,ax = plt.subplots()
    df_cut_prime.plot(y=key, label='model test', ax=ax)
    df_result_VCT.plot(y=key, label='simulation', ax=ax)
    ax.set_ylabel(key)
```

# Time series PIT

```python
from statsmodels.sandbox.regression.predstd import wls_prediction_std
def show_pred(X,y,results, label):

    display(results.summary())

    X_ = X
    y_ = y
    y_pred = results.predict(X_)

    prstd, iv_l, iv_u = wls_prediction_std(results, exog=X_, alpha=0.05)
    #iv_l*=-1
    #iv_u*=-1

    fig,ax=plt.subplots()
    ax.plot(X_.index,y_, label='Numerical gradient from model test')
    ax.plot(X_.index,y_pred, '--', label='OLS')
    ax.set_ylabel(label)

    ax.fill_between(X_.index, y1=iv_l, y2=iv_u, zorder=-10, color='grey', alpha=0.5, label=r'5\% confidence')
    ax.legend();
```

## N

```python
N_eq_ = N_eq.copy()
N_eq_ = N_eq_.subs([
    (x_G,0),  # Assuming or moving to CG=0
#    #(I_z,1),  # Removing inertia
#    #(eq.p.Nrdot,0),  # Removing added mass
#    #(eq.p.Nvdot,0),  # Removing added mass
#    #(eq.p.Nudot,0),  # Removing added mass
#
])

solution = sp.solve(N_eq_,r1d)[0]
inertia_ = (I_z-eq.p.Nrdot)
N_eq_ = sp.Eq(r1d*inertia_, solution*inertia_)

```

```python
Math(vlatex(N_eq_))
```

```python
label_N = N_eq_.lhs
diff_eq_N = regression.DiffEqToMatrix(ode=N_eq_, label=label_N, base_features=[delta,u,v,r])
```

```python
Math(vlatex(diff_eq_N.acceleration_equation))
```

```python
Math(vlatex(diff_eq_N.acceleration_equation_x))
```

```python
Math(vlatex(diff_eq_N.eq_y))
```

```python
diff_eq_N.eq_beta
```

```python
Math(vlatex(diff_eq_N.eq_X))
```

```python
diff_eq_N.y_lambda
```

```python
X = diff_eq_N.calculate_features(data=df_prime)
y = run(function=diff_eq_N.y_lambda, inputs=df_prime, **ship_parameters_prime, **df_parameters_all['brix_prime'])

model_N = sm.OLS(y,X)
results_N = model_N.fit()

show_pred(X=X,y=y,results=results_N, label=r'$%s$' % vlatex(label_N))
```

## Y

```python
Y_eq_ = Y_eq.copy()
Y_eq_ = Y_eq.subs([
    (x_G,0),  # Assuming or moving to CG=0
#    #(I_z,1),  # Removing inertia
#    #(eq.p.Nrdot,0),  # Removing added mass
#    #(eq.p.Nvdot,0),  # Removing added mass
#    #(eq.p.Nudot,0),  # Removing added mass
#
])

solution = sp.solve(Y_eq_,v1d)[0]
inertia_ = (eq.p.Yvdot-m)
Y_eq_ = sp.simplify(sp.Eq(v1d*inertia_-U*m*r, solution*inertia_-U*m*r))

Math(vlatex(Y_eq_))
```

```python
label_Y = Y_eq_.rhs
diff_eq_Y = regression.DiffEqToMatrix(ode=Y_eq_, label=label_Y, base_features=[delta,u,v,r])
```

```python
X = diff_eq_Y.calculate_features(data=df_prime)
y = run(function=diff_eq_Y.y_lambda, inputs=df_prime, **ship_parameters_prime, **df_parameters_all['brix_prime'])

model_Y = sm.OLS(y,X)
results_Y = model_Y.fit()

show_pred(X=X,y=y,results=results_Y, label=r'$%s$' % vlatex(label_Y))
```

## X

```python
X_eq_ = X_eq.copy()
X_eq_ = X_eq_.subs([
    (x_G,0),  # Assuming or moving to CG=0
#    #(I_z,1),  # Removing inertia
#    #(eq.p.Nrdot,0),  # Removing added mass
#    #(eq.p.Nvdot,0),  # Removing added mass
#    #(eq.p.Nudot,0),  # Removing added mass
#
])

solution = sp.solve(X_eq_,u1d)[0]
inertia_ = m-eq.p.Xudot
X_eq_ = sp.simplify(sp.Eq(u1d*inertia_, solution*inertia_))

Math(vlatex(X_eq_))
```

```python
label_X = X_eq_.lhs
diff_eq_X = regression.DiffEqToMatrix(ode=X_eq_, label=label_X, base_features=[delta,u,v,r])
```

```python
X = diff_eq_X.calculate_features(data=df_prime)
y = run(function=diff_eq_X.y_lambda, inputs=df_prime, **ship_parameters_prime, **df_parameters_all['brix_prime'])

model_X = sm.OLS(y,X)
results_X = model_X.fit()

show_pred(X=X,y=y,results=results_X, label=r'$%s$' % vlatex(label_X))
```

```python
results_summary_X = regression.results_summary_to_dataframe(results_X)
results_summary_Y = regression.results_summary_to_dataframe(results_Y)
results_summary_N = regression.results_summary_to_dataframe(results_N)
```

## Add regressed parameters

```python
results = pd.concat([results_summary_X, results_summary_Y, results_summary_N],axis=0)
df_parameters_all['PIT'] = results['coeff']
df_parameters_all['PIT'] = df_parameters_all['PIT'].combine_first(df_parameters_all['brix_prime'])  # prefer regressed
```

```python
fig,ax=plt.subplots()
fig.set_size_inches(15,5)

mask = ((df_parameters_all['brix_prime']!=0) |
        (pd.notnull(df_parameters_all['regressed'])) |
        (df_parameters_all['PIT']!=0)

       )

df_parameters_all_plot = df_parameters_all.loc[mask]
df_parameters_all_plot.plot.bar(y=['brix_prime','regressed','PIT'], ax=ax);
```

## Simulate

```python
parameters = df_parameters_all['PIT'].copy()
#parameters['Xv']=0
#parameters['Xr']=0
#parameters['Xu']=0
#parameters['Xdelta']=0
#parameters['Nv']*=-1

solution, df_result_PIT = simulate(df_cut_prime, parameters = parameters, ship_parameters=ship_parameters_prime)
```

```python
fig,ax=plt.subplots()
track_plot(df=df_cut_prime, lpp=ship_parameters_prime['L'], beam=ship_parameters_prime['B'],ax=ax, label='model test')
track_plot(df=df_result_PIT, lpp=ship_parameters_prime['L'], beam=ship_parameters_prime['B'],ax=ax, label='simulation', color='green')
ax.legend()

for key in df_result_PIT:
    fig,ax = plt.subplots()
    df_cut_prime.plot(y=key, label='model test', ax=ax)
    df_result_PIT.plot(y=key, label='simulation', ax=ax)
    ax.set_ylabel(key)
```

```python

```