# Practical 1: Deep Feed-forward Networks

## Introduction
In this practical, we implement and train a feed-forward neural network (also known as an "MLP" for "multi-layer perceptron") on a dataset called "Fashion MNIST", consisting of small greyscale images of items of fashion. We consider the practical issues around generalisation to out-of-sample data and introduce some important techniques for addressing this.

## Learning Objectives
* Understand how to use **Tensorflow Eager** and **Keras Layers** to build a neural network architecture
* Understand how a model is trained and evaluated
* Be able to explain why there is a difference between **in-sample** and **out-of-sample** model performance
* Understand the concept of **train/validation/test split** and why it's useful
* Research at least 1 technique that can be used to improve model **generalisation**

```
#@title Imports (RUN ME!) { display-mode: "form" }
from __future__ import absolute_import, division, print_function

import tensorflow as tf
import tensorflow.contrib.eager as tfe
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

try:
  tf.enable_eager_execution()
  print('Running in Eager mode.')
except ValueError:
  print('Already running Eagerly')
```

    Running in Eager mode.

## The Data

In this practical, we use the Fashion MNIST dataset consisting of 70,000 greyscale images and their labels. The dataset is divided
 into 60,000 training images and 10,000 test images. The idea is to train a **classifier** to identify the class value (what type of fashion item it is) given the image. We train and *tune* a model on the 60,000 training images and then evaluate how well it classifies the 10,000 test images that the model did not see during training. This task is an example of a **supervised learning** problem, where we are given both input and labels (targets) to learn from. This is in contrast to **unsupervised learning** where we only have inputs from which to learn patterns or **reinforcement learning** where an agent learns how to maximise a reward signal through interaction with its environment.

###Aside: Train/Validation/Test Split

When we build machine learning models, the goal is to build a model that will perform well on *future* data that we have not seen yet. We say that we want our models to be able to **generalise** well from whatever training data we can collect and do have available, to whatever data we will be applying them to in future. To do this, we split whatever data we have available into a **training set, a validation set and a test set**. The idea is that we train our model and use the performance on the validation set to make any adjustments to the model and its hyperparameters, but then we report the final accuracy on the test set. The test set (which we never train on), therefore acts as a proxy for our future data.

```
# Tensorflow has convenient modules for loading a number of standard datasets
fashion_mnist = tf.keras.datasets.fashion_mnist
(train_and_validation_images, train_and_validation_labels), (test_images, test_labels) = fashion_mnist.load_data()

text_labels = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']
```

### Construct a validation set
Next, we remove 10,000 images and labels from the training set to use as a validation set. We will come back to the validation set later!

```
# Construct a validation set from the last 10000 images and labels from
# train_and_validation_images and train_and_validation_labels
validation_images = train_and_validation_images[-10000:, :, :]
validation_labels = train_and_validation_labels[-10000:]

# Construct a training set from the first 50000 images and labels.
train_images = train_and_validation_images[:50000, :, :]
train_labels = train_and_validation_labels[:50000]
```

### What does the data look like?
Each image in the dataset consists of a 28 x 28 matrix of greyscale pixels. The values are between 0 and 255 where 0 represents black, 255 represents white and there are many shades of grey in-between. Each image is assigned a corresponding numerical label, so the image in ```train_images[i]``` has its corresponding label stored in ```train_labels[i]```. We also have a lookup array called ```text_labels``` to associate a text description with each of the numerical labels. For example, the label 1 is associated with the text description "Trouser".

The 2 cells below demonstrate how to visualise the input data. Make sure you understand what's happening, particularly how the indices correspond to individual items in the dataset.

```
# We use the Matplotlib plotting library to visualise an image selected at random from the training set
plt.figure()
random_index = np.random.randint(0, len(train_images))
plt.imshow(train_images[random_index], cmap='gray')
plt.colorbar()
numerical_label = train_labels[random_index]
text_description = text_labels[numerical_label]
plt.title('True Class: {} ("{}")'.format(numerical_label, text_description))

plt.gca().grid(False)
```

```
# Another view, showing 25 randomly selected images at a time
plt.figure(figsize=(10,10))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid('off')

    img_index = np.random.randint(0, 50000)
    plt.imshow(train_images[img_index], cmap=plt.cm.gray)
    plt.xlabel(text_labels[train_labels[img_index]])
```

### Exploratory Task
Run the above cells multiple times to get a good sense of the dataset. Look at the general structure of the images and their corresponding labels. Can you spot some classes that might be difficult for a classifier to distinguish? If so, why do you think they would be difficult to distinguish? Chat to your neighbour about this.

### Preparing the data with TensorFlow
At the moment, our training data consists of two large tensors. The images are stored in a tensor of shape $[50000, 28, 28]$, consisting of all the $28 \times 28$ images matrices stacked together. The labels are stored in a 1D vector of shape $[50000]$. We wish to train a model using **mini-batch stochastic gradient descent**. In order to do so, we need to shuffle the data and split it into smaller (mini-)batches. We also convert the data from numpy arrays to TensorFlow Tensors.

#### Questions
* Can you explain what the difference is between normal stochastic gradient descent and "batched" stochastic gradient descent? What is the purpose of chunking data into batches during training? Are there any trade-offs? What are they? **HINT**: Speak to your neighbours and tutors about this. You can also read [this blog post](https://machinelearningmastery.com/gentle-introduction-mini-batch-gradient-descent-configure-batch-size/) for more details.
* Why is it important to randomise the data before chunking it into batches?

In order to do this batching (and shuffling) we will use the Tensorflow [Dataset API](https://www.tensorflow.org/api_docs/python/tf/data/Dataset), which is a set of simple reusable components that allow you to build efficient data pipelines. Data is said to "stream" through the pipeline, meaning that when something at the output of the pipeline wants data, the pipeline will provide that data as soon as it has enough, rather than waiting to process all the data. This allows you to easily build pipelines that work on large datasets without having to load it all into memory in one go!

We build this pipeline step-by-step:

We being by defining the ```batch_size``` hyperparameter of our model. This hyperparameter controls the sizes of the mini-batches (chunks) of data that we use to train the model. The value you use will affect the memory usage, speed of convergence and potentially also the performance of the model. It also interacts with the *learning rate* used in gradient descent.

```
batch_size = 128
```

The first component we add will group the image and label tensors together into a tuple and then split them into individual entries - ie. 50000 tuples containing a (28, 28) dimensional image and 1D label associated with that image. The following line adds this splitting component to the pipeline.

```
train_ds = tf.data.Dataset.from_tensor_slices((train_images, train_labels))
```

The next thing we do is apply a map operation. This lets us run an arbitrary function on each element. The function we provide returns the image values divided by 255 and converted ('cast') to a float and the label converted to a 32-bit integer.

**NOTE**: "Lambda" functions are just one-line, anonymous functions. In this case, it defines a function that takes arguments x and y (before the colon), and outputs their manipulated values (after the colon).

```
# Divide image values and cast to float so that they end up as a floating point number between 0 and 1
train_ds = train_ds.map(lambda x, y: (tf.cast(x, tf.float32) / 255.0, tf.cast(y, tf.int32)))
```

Then we add a **shuffle** component. This returns a random element from the pipeline. Can you spot a potential problem here?

**QUESTION**:  What might happen if the shuffle component didn't have a *buffer*?  **HINT**: Refer back to the discussion about the data pipeline and "data streaming through it" above.

```
# Shuffle the examples.
train_ds = train_ds.shuffle(buffer_size=batch_size * 10)
```

The final component in our pipeline is the batch component. This just requests `batch_size` elements from the previous pipeline component, groups them together into a single tensor and returns that.

```
# Now "chunk" the examples into batches
train_ds = train_ds.batch(batch_size)

# The output of this pipeline will be tuples of tensors containing images and labels.
# The images will be of shape (batch_size, 28, 28) and the labels of shape (batch_size, )
```

Finally, we apply the same pre-processing (converting to Tensors and normalising the values to lie between 0 and 1) to the validation set. Here we don't need to use the Dataset API (but could if we wanted to!) because we don't need batching or shuffling.

```
# Apply the same preprocessing to the validation set
# Ignore this for now - it will be used later!
validation_images = tf.convert_to_tensor(validation_images, dtype=tf.float32) / 255.0
validation_labels = tf.convert_to_tensor(validation_labels, dtype=tf.int32)
```

## Define the model

In this section we'll build a classifier. A **classifier** is a function that takes an object's characteristics (or "features") as inputs and outputs a prediction of the class (or group) that the object belongs to. It may make a single prediction for each input or it may output some score (for example a probability) for each of the possible classes. Specifically, we will build a classifier that takes in (a batch of) 28 x 28 Fashion MNIST images as we've seen above, and outputs predictions about which class the image belongs to.

For each (batch of) input images, we will use a **feed-forward neural network** to compute un-normalised scores (also known as **logits**) for each of the 10 possible classes that the image could belong to. We can then **classify** the image as belonging to the class which receives the highest score, or we can quantify the model's "confidence" about the classifications by converting the scores into a probability distribution.

A feed-forward neural network consisting of $N$ layers, applied to an input vector $\mathbf{x}$ can be defined as:

\begin{equation}
\mathbf{f_0} = \mathbf{x} \\
\mathbf{f_i} = \sigma_i(\mathbf{W_if_{i-1}} + \mathbf{b_i}) \ \ \ i \in [1, N]
\end{equation}

Each layer has a particular number, $m_i$, of neurons. The parameters of a layer consist of a matrix $\mathbf{W_i} \in \mathbb{R}^{m_i \times m_{i-1}}$ and bias vector $\mathbf{b_i} \in \mathbb{R}^{m_i}$. Each layer also has a non-linear activation function $\sigma_i$.

**QUESTION**: Why do you think the activation functions need to be *non-linear*? What would happen if they were *linear*? **HINT**: If you're stuck, consider the very simplest case of an identity activation (which essentially does nothing) and ignore the biases.

### Aside: Activation functions

Activation functions are a core ingredient in deep neural networks. In fact they are what allows us to make use of multiple layers in a neural network. There are a number of different activation functions, each of which are more or less useful under different circumstances. The four activation functions that you are most likely to encounter are, arguably, [ReLU](https://www.tensorflow.org/api_docs/python/tf/keras/layers/ReLU), [Tanh](https://www.tensorflow.org/api_docs/python/tf/keras/activations/tanh), [Sigmoid](https://www.tensorflow.org/api_docs/python/tf/keras/activations/sigmoid), and [Softmax](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Softmax).

ReLU, has in recent years, become the default choice for use in MLPs and Convolutional Neural Networks (CNNs). ReLU has two advantages over Tanh and Sigmoid: it is computationally much more efficient, and, it allows us to use deeper networks because it does not suffer from [vanishing gradients](https://en.wikipedia.org/wiki/Vanishing_gradient_problem). As a result of their success, a number of ReLU variants, such as [LeakyRelu](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LeakyReLU) and [PReLU](https://www.tensorflow.org/api_docs/python/tf/keras/layers/PReLU), have been developed.

Sigmoid and Softmax activations are often found after the last layer in binary and multi-class classification networks, respectively, as they transform the outputs of the network in such a way that we can interpret them as class probabilities.

Both Tanh and Sigmoid are found in LSTM and GRU recurrent neural networks, which we will find out more about in the coming days. They are also useful in MLPs and CNNs where we want the output to be bounded between -1 and 1 (Tanh) or 0 and 1 (Sigmoid).

Read more about activation functions [here](https://towardsdatascience.com/activation-functions-neural-networks-1cbd9f8d91d6).

### Configure the feed-forward neural network
We configure the feed-forward neural network part of our classifier using the [Keras Layers API](https://www.tensorflow.org/api_docs/python/tf/keras/layers). This API consists of various reusable building-blocks that allow us to define many different neural network architectures (similar to how we defined a data pipeline earlier!).

Here we use the [Sequential](https://www.tensorflow.org/api_docs/python/tf/keras/Sequential) component which allows us to wrap together a sequence of layers. An important point to note here is that we are **configuring** our neural network architecture as a pipeline. We can think of the resulting ```model`` variable as a *function* that takes a batch of images as inputs and outputs a batch of logits.

```
model = tf.keras.Sequential([
    # Convert the 28x28 image into a flat vector of 28x28 = 784 values
    tf.keras.layers.Flatten(input_shape=(28, 28), name='flatten_input'),
    # Create a "hidden" layer with 256 neurons and apply the ReLU non-linearity
    tf.keras.layers.Dense(256, activation=tf.nn.relu, name='input_to_hidden1'),
    # Create another hidden layer with 128 neurons
    tf.keras.layers.Dense(128, activation=tf.nn.relu, name='hidden1_to_hidden2'),
    # Create an "output layer" with 10 neurons
    tf.keras.layers.Dense(10, name='hidden_to_logits'),
])
```

The following summary shows how many parameters each layer is made up of (the number of entries in the weight matrics and bias vectors). Note that a value of ```None``` in a particular dimension of a shape means that the shape will dynamically adapt based on the shape of the inputs. In particular, the output shape of the ```flatten_input``` layer will be $[N, 784]$ when the batch of inputs passed to the model has shape $[N, 28, 28]$

```
model.summary()
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    flatten_input (Flatten)      (None, 784)               0
    _________________________________________________________________
    input_to_hidden1 (Dense)     (None, 256)               200960
    _________________________________________________________________
    hidden1_to_hidden2 (Dense)   (None, 128)               32896
    _________________________________________________________________
    hidden_to_logits (Dense)     (None, 10)                1290
    =================================================================
    Total params: 235,146
    Trainable params: 235,146
    Non-trainable params: 0
    _________________________________________________________________

**QUESTION**: Note that the flattening operation has no parameters. However, the Dense (fully-connected) layer mapping from the 784-D output of the flatten operation to the 256-D hidden layer has 200,960 parameters! Can you explain why the dense layer has 200,960 parameters? **HINT**: Refer back to the equations of a feed-forward network.

### Define the loss
As we did in the previous practical, we need to specify a loss function for our classifier. This tells us how good our model's predictions are compared to the actual labels (the targets), with a lower loss meaning better predictions. The standard loss function to use with a **multi-class classifier** is the **cross-entropy loss** also known as the "negative log likelihood". Suppose we have a classification problem with $C$ classes. A classifier is trained to predict a probability distribution $p(y | X_i)$ for each input $X_i$ from a batch of $N$ examples. The vector $p(y|X_i)$ is $C$ dimensional, sums to one, and we use $p(y|X_i)_c$ to denote the $c$th component of  $p(y|X_i)$. The true class for example $i$ in the batch is $y_i$ and we define the indicator function $\mathbb{1}[y_i=c]$ to be 1 whenever $y_i = c$ and $0$ otherwise. This classifier has a cross-entropy loss of

$- \frac{1}{N}\sum_{i=1}^N \sum_{c=1}^C log( p(y|X_i)_c) \mathbb{1}[y_i=c]$

**NOTE**: The indicator is one for the true class label, and zero everywhere else. So in that sum, the indicator just "lifts out" the $log(p(y|X_i))$ values for all true classes. So the above expression is minimised (note the negative at the front) when the model places all its probability mass on the true labels for all the examples. Remember that  log(1)=0 , thus the closer all probabilities of $y_i = c$ are to one, the lower the loss will be and the better the model will be performing.

**QUESTION**:
* Why do you think this is a *good* loss function?
* Can you think of any potential issues with this loss function?

Fortunately we don't need to write this function ourselves as Tensorflow provides a version called

```tf.nn.sparse_softmax_cross_entropy_with_logits```.

**NOTE**: This function actually computes the cross entropy loss directly from the un-normalised logits, rather than from the probability distribution for numerical stability.

By the way, for training data in which the labels are themselves distributions rather than exact values, this definition of cross-entropy still works, where the indicator function is replaced with the corresponding probability of each class for that example. This might be important when we are not sure whether the training data has been labelled correctly, or when the data was labelled by a human who gave their answer along with a degree of confidence that the answer was correct

## Train the model
Now that we have our data, data processing pipeline, our neural network architecture and the corresponding loss that we want to minimise, we need to **train** the model using batched stochastic gradient descent. We do this in multiple **epochs**, which is a single iteration through the entire training dataset. Briefly, in each epoch we loop over all the batches of images and labels, and for each batch we perform the following steps:
* Get the **predictions** of the model on the current batch of images
* Compute the **average loss** values across the batch, telling us how good these predictions are / how close they are to the true targets.
* Compute the **gradient of the average loss** (or the average gradient of the losses in the batch) with respect to each of the model's parameters: This tells us the direction to move in "parameter space" to decrease the loss value
* **Adjust the parameters** by taking a small step in the direction of each component of the gradient (where the learning rate controls the size of the step)

During training we also track some metrics, such as the loss and accuracy to see how well the classifier is doing. Note that the cell below may take a few minutes to run!

###ASIDE: Computing gradients in Eager mode

In graph mode, TensorFlow builds up a "computation graph" which captures all operations of the model and their dependencies. For training, the gradient can then be computed by traversing backwards from every node through its dependents and applying the "chain rule" of differentiation.

However, in Eager mode, we don't have the concept of the computation graph anymore. Operations are performed imperatively (in the order in which they were executed). TensorFlow therefore uses a mechanism called the "GradientTape" for computing gradients in Eager mode. Basically, the gradient tape records the order of all operations as they are executed, and can then be "run backwards" (traversed from last to first operation) for computing the gradients.

```
# Create an optimizer
# The optimizer is responsible for controlling the learning rate
optimizer = tf.train.AdamOptimizer()

step_counter = tf.train.get_or_create_global_step()  # Just a variable that keeps track of how many training steps we've run
num_epochs = 50  # The number of epochs to run

# Lists to store the loss and accuracy of every epoch
epoch_losses = []
epoch_accuracies = []

for epoch in range(num_epochs):
  # Tensorflow provides a convenient API for tracking a number of metrics during training/evaluation
  loss_avg = tfe.metrics.Mean()
  accuracy = tfe.metrics.Accuracy()

  # Loop over our data pipeline
  for step, (image_batch, label_batch) in enumerate(train_ds):

    # Initialise a GradientTape to track the operations
    with tf.GradientTape() as tape:
      # Compute the logits (un-normalised scores) of the current batch of examples
      # using the neural network architecture we defined earlier
      logits = model(image_batch, training=True)
      # Compute the cross-entropy loss of the classification outputs on this batch
      loss_value = tf.nn.sparse_softmax_cross_entropy_with_logits(
          logits=logits, labels=label_batch)
      # Compute the average loss over the batch
      loss_value = tf.reduce_mean(loss_value)

      # Add current batch loss to our loss metric tracker - note the function call semantics
      loss_avg(loss_value)
      # Compare most likely predicted label to actual label
      accuracy(tf.argmax(logits, axis=1, output_type=tf.int32), label_batch)

    # Play the tape backwards and get the gradient of the loss of the current batch
    # Note we're now outside the scope of the with-block above
    grads = tape.gradient(loss_value, model.variables)
    # Use the optimizer to apply the gradients to the model parameters along with
    # its internal learning rate
    optimizer.apply_gradients(
        zip(grads, model.variables), global_step=step_counter)

  # Get the average loss and accuracy for the epoch
  epoch_loss = loss_avg.result()
  epoch_losses.append(epoch_loss)
  epoch_accuracy = accuracy.result()
  epoch_accuracies.append(epoch_accuracy)
  print("Epoch {:03d}: Loss: {:.3f}, Accuracy: {:.3%}".format(epoch, epoch_loss, epoch_accuracy))

# Plot the loss for all epochs using Matplotlib
plt.figure()
plt.plot(range(num_epochs), epoch_losses)
plt.title('Loss vs epochs')

# Plot the accuracy for all epochs using Matplotlib
plt.figure()
plt.plot(range(num_epochs), epoch_accuracies)
plt.title('Accuracy vs epochs')
```

The above code block shows a typical training loop. There's actually an easier way to do this using Tenosrflow and Keras, which we'll see in the next prac, but for now we expand it so you can see what's going on.

Lets visualise some of the model's prediction on the training set

```
images, labels = next(train_ds.make_one_shot_iterator())  # Get a batch of images and labels

_logits = model(images, training=False)  # Pass the images to the model function and get its output logits
predicted_labels = tf.argmax(_logits, axis=1, output_type=tf.int32)

img_indexs = np.arange(images.numpy().shape[0])
np.random.shuffle(img_indexs)

plt.figure(figsize=(10,10))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid('off')

    img_index = img_indexs[i]
    predicted_label = int(predicted_labels[img_index])

    plt.imshow(images[img_index], cmap=plt.cm.gray)

    actual_label = int(labels[img_index].numpy())
    plt.xlabel("Actual: {} ({})\n Predicted: {} ({})".format(
        actual_label, text_labels[actual_label], predicted_label, text_labels[predicted_label]
    ))

plt.tight_layout()
plt.show()
```

 ### Aside: Optimisation schemes

You might have noticed that we are using the Adam optimizer to train our neural networks. Adam is a variant of stochastic gradient descent which often performs well in practice. In fact, there is a whole range of variations on stochastic gradient descent that are often used. Here is an illustration showing how a few of these methods perform on a toy problem:

For a detailed description of various optimization methods read [this](http://ruder.io/optimizing-gradient-descent/) article. For a great visual discussion on how these methods work see [this](https://distill.pub/2017/momentum/) article.

## Testing

After 50 epochs of training on our training set, we obtain a loss of around 0.061 and accuracy of 97.7%. That's pretty good, right?

It is important to distinguish between data that is **in-sample** and **out-of-sample**. Our training data is all in-sample and we would expect any resonably powerful model (like an MLP) to be very accurate and have a low loss on this (in fact, with a sufficiently large MLP and enough training epochs, we can get the loss arbitrarily close to zero!). This is not a good thing though, because pushing the loss close to zero may mean that the model has fit the **noise** in the training data rather than the true signal. If this is the case, it will not perform well on out-of-sample data that it has not seen before (which is what we really care about!). To assess this, we evaluate our trained model on the held-out test set (but we don't update the parameters of the model when testing it, that would be cheating!)

```
# We need to apply the same pre-processing to the test set as we did to the training set
# Since we don't need batching or shuffling, we can do this directly instead of
# building a tf.Dataset pipeline

tf_test_images = tf.convert_to_tensor(test_images, dtype=tf.float32) / 255.0
tf_test_labels = tf.convert_to_tensor(test_labels, dtype=tf.int32)
```

```
test_accuracy = tfe.metrics.Accuracy()

test_logits = model(tf_test_images, training=False)

# Compute the average cross-entropy loss of the classification over the entire test set
test_loss = tf.nn.sparse_softmax_cross_entropy_with_logits(
    logits=test_logits, labels=tf_test_labels)
test_loss = tf.reduce_mean(test_loss)

# Compare predicted labels to actual labels
test_accuracy(tf.argmax(test_logits, axis=1, output_type=tf.int32), tf_test_labels)

print('Completed testing on', tf_test_images.shape[0], 'examples...')
print('Loss: {:.3f}, Accuracy: {:.3%}'.format(test_loss, test_accuracy.result()))
```

    Completed testing on 10000 examples...
    Loss: 0.607, Accuracy: 88.920%

And again we visualise some of the model's predictions:

```
test_predictions = tf.argmax(test_logits, axis=1, output_type=tf.int32)

plt.figure(figsize=(10,10))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid('off')

    img_index = np.random.randint(0, 10000)
    plt.imshow(test_images[img_index], cmap=plt.cm.gray)

    actual_label = int(test_labels[img_index])
    predicted_label = int(test_predictions[img_index])

    plt.xlabel("Actual: {} ({})\n Predicted: {} ({})".format(
        actual_label, text_labels[actual_label], predicted_label, text_labels[predicted_label]
    ))

plt.tight_layout()
plt.show()
```

What happened? We got 97.7% accuracy on the training set, but only 88.9% on the test set. This is an example of how a model's in-sample performance differs from its out-of-sample performance. It may even be considered **overfitting** (where the model failing to generalise its in-sample performance to unseen out-of-sample data). There are a number of ways we can address this, but first, it's important to realise that we should try and keep the number of evaluations on the test set to a minimum.

**QUESTION**: Can you think of WHY we should limit the number of evaluations on the test set? This is an important point, so spend a few minutes thinking about it, discuss it with your neighbour and ask the tutors if you get stuck!

The first important method is called **validation**. Using validation, we reserve a portion of our training dataset and call it the validation set. After each epoch of training (or every K epochs where K can be a fraction), we evaluate our model on this validation set (without updating the model parameters). This gives us a good sense of how well we expect our model to do on out-of-sample data. In this way, the validation set acts as a surrogate for the test set which we can safely use during model development and training. We can also use this validation set to tune our model by selecting hyperparameters such as the batch size, sizes of layers or the network architecture.

### Aside: Underfitting

In addition to the problem of _overfitting_ (in which the model is too complex and as a result is able to memorize the training data rather than learning a general pattern) there is the problem of _underfitting_ which is the opposite. Underfitting occurs when the model does not have enough complexity (also known as capacity) to learn a general pattern. While overfitting is characterised by diverging test/validation and training scores, underfitting is characterised by test/validation and training scores that very slowly continue to improve.

Unlike overfitting which has a number of solutions, such as collecting more data, data augmentation, dropout, and L1/L2 regularization, underfitting has one simple solution: just make your model incrementally more complex until it no longer underfits.

##TASKS:

1. [**ALL**] Implement **Validation** by adding some code to the loop in the "Train the model" section (**TIP**: The code for evaluating on the validation set will be very similar to evaluating on the test set!):
 * Evaluate your model on the validation set at the end of each training epoch and print the loss and accuracy on the validation set. We have already defined a validation set and done the necessary preprocessing for you.
  * Evaluate how closely the loss and accuracy on the validation set matches the eventual performance on the test set.

2. [**ALL**] Implement DROPOUT (see [Tensorflow documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dropout) and [research paper](https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf) ) to improve the model's generalisation.

3. [**ADVANCED**] Implement BATCH NORMALISATION ([Tensorflow documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/BatchNormalization) and [research paper](http://proceedings.mlr.press/v37/ioffe15.pdf))to improve the model's generalisation.

4. [**OPTIONAL**] Remove the shuffling from the dataset pipeline and compare the generalisation performance of the model with and without shuffling.

5. [**OPTIONAL**] Adjust the model architecture by adding layers and modifying other parameters to improve the performance on your **validation** set. Once you've settled on an architecture and hyperparameters, evaluate your model on the **test** set. See the Tensorflow documentation for examples of parameters you can change in the model.

## Important Tip
If you want to re-train your model, you need to rerun all the code in the "Define the model" section first. This is to ensure that the Tensorflow variables containing the model parameters get reset. If you don't do this, you're effectively running more epochs on an already-trained model.


# Сегментация изображений

## Сегментация изображений с помощью порогов

1. Построить гистограммы по трем (RGB) составляющим изображения. По оси абсцисс откладывается значение составляющей, а по оси ординат количество пикселов, имеющих соответствующее значение составляющей. При необходимости провести сглаживание гистограммы:
$$
\hat{g}_i=\frac{1}{n} \sum_{j=-\frac{n}{2}}^{\frac{n}{2}}g_{i+j}
$$
где $\hat{g}_i$ - новое значение столбца гистограммы, $n$ - ширина фильтра сглаживания.

2. По гистограмме определить значение порогов сегментации:
значения порогов по каждой составляющей определяются по локальным минимумам гистограмм;
значения составляющих изображения, лежащие в пределах между двумя минимумами гистограммы можно отнести к одной области (сегменту) изображения.

3. Однородные области изображения окрасить в псевдоцвета.

### Пример

Загрузка изображения

```python
import matplotlib.image as imr
im = imr.imread('files2/test_image1.jpg')
im.setflags(write=1)
import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams["figure.figsize"] = (10,7)
plt.imshow(im)
plt.show()
```

Построение гистограмм

```python
import numpy as np

def histogram(im, bins):
    res = np.zeros(bins)
    for i in im: res[i] = res[i] + 1
    return res

h1 = histogram(im[:,:,0], 256)
h2 = histogram(im[:,:,1], 256)
h3 = histogram(im[:,:,2], 256)
plt.subplot(131)
plt.scatter(np.arange(256), h1)
plt.subplot(132)
plt.scatter(np.arange(256), h2)
plt.subplot(133)
plt.scatter(np.arange(256), h3)
```

Сглаживание и нахождение локальных минимумов

```python
def smooth (g, n):
    res=np.zeros_like(g)
    #import pdb; pdb.set_trace()
    for i in range(g.shape[0]):
        s=0
        for j in range(-n//2, n//2):
            inx = i+j
            if inx >= 0 and inx<g.shape[0]:
                s=s+g[inx]
        res[i]=s//n
    return res

n=20 # Ширина окна сглаживания
_h1 = smooth(h1,n)
_h2 = smooth(h2,n)
_h3 = smooth(h3,n)
plt.subplot(131)
plt.scatter(np.arange(256),_h1)

plt.subplot(132)
plt.scatter(np.arange(256),_h2)
plt.subplot(133)
plt.scatter(np.arange(256),_h3)

def locmin(g):
    res=[]
    for i in range(1, g.shape[0]-1):
        if g[i-1]>g[i] and g[i]<g[i+1]:
             res.append(i)
    res.append(255)
    return res
# Поиск локальных минимумов
l1 = locmin(_h1)
l2 = locmin(_h2)
l3 = locmin(_h3)
```

```python
l3

```

    [255]

Раскрашивание изображение в псевдоцвета

```python
def colored(h, lm):
    for i in range(h.shape[0]):
        for j in range(h.shape[1]):
            for lc in lm:
                if h[i,j]<lc:
                    h[i,j]=lc
                    break
    return h
im[:,:,0]=colored(im[:,:,0],l1)
im[:,:,1]=colored(im[:,:,1],l2)
im[:,:,2]=colored(im[:,:,2],l3)
plt.imshow(im)
plt.show()
```

# Динамическая сегментация изображений

Суть метода сегментации изображения с динамически изменяющимся порогом заключается в следующем:
1. В первую очередь необходимо определить приемлемый диапазон изменений $P$ цвето-яркостных характеристик элементов изображения (пикселов), который будет служить критерием принадлежности пикселов одному сегменту.
2. Разбиение на сегменты начинается с того, что фиксируется определенный пиксел, который рассматривается как элемент сегмента.
3. Затем рассматриваются элементы, граничащие с сегментом (на первом проходе сегмент состоит из единственного фиксированного пиксела).
4. Вычисляется так называемая "базовая яркость" или "базовый цвет" по формуле:
$$
I_{i}^{b}=\frac{1}{i}\sum_{j=1}^{i}I_{j},
$$
где $I_{j}$ - яркость (цветовая характеристика) пиксела $j$.
 На первом шаге - это значение яркости (цветовой характеристики) первого  пиксела.
5. Вычисляется критерий принадлежности очередного граничного элемента к сегменту:
$$
\begin{equation}
|I_{i}^{b} - I_{i}| \leq P \tag{1}
\end{equation}
$$
Если неравенство выполняется, то рассматриваемый элемент изображения  приписывается сегменту.
6. Пункты 3-5 выполняются до тех пор, пока существуют граничащие с сегментом элементы изображения, удовлетворяющие критерию (1). Формирование сегмента завершается, если не окажется ни одного такого элемента.
7. Затем начинается формирование нового сегмента с фиксации пиксела, не принадлежащего ни одному из вновь образованных сегментов. Предыдущие шаги выполняются до тех пор, пока не будут рассмотрены все пикселы изображения.
8. Для цветного изображения может быть применен следующий критерий объединения пикселов в сегменты:
$$
\sqrt{(R_C - R)^{2}+(G_C - G)^{2}+(B_C - B)^{2}} \leq P
$$
где  $R_c G_c B_c$ – базовый цвет сегмента и $RGB$ - цвет тестируемого пиксела.

```python
# Функция, возвращающая случайный цвет
def random_color():
    h,s,l = random.random(), 0.5 + random.random()/2.0, 0.4 + random.random()/5.0
    r,g,b = [int(256*i) for i in colorsys.hls_to_rgb(h,l,s)]
    return [r, g, b]

# Метрика (пункт 8)
def metric(x1, x2):
    return np.sqrt(np.sum((x1-x2)**2))

# Рекурсивная функция добавления пикселя в сегмент
# Pc - приемлемый диапазон, x - координаты добавляемого писеля,
# seg - матрица сегментов, im - изображение,
# P - сумма яркостей пикселей в сегменте,
# n - количество пикселей в сегменте, n - номер сегмента
def append(Pc, x, seg, im, P, n, i):
    # Добаляемый пиксель помечается принадлежащим текущему сегменту
    seg[x[0],x[1]]=i
    n=n+1 # Увеличивается количество пикселей в сегменте
    P=P+im[x[0],x[1],:] #Яркость добавленого пикселя добавляется к сумме яркостей
    for w in range(-1,2): # Циклы, проходящие
        for h in range(-1,2): # восемь соседей добавленного пикселя
            if w==0 and h==0: continue # но сам пиксел не является своим соседом
            cx=x[0]+w; cy=x[1]+h # Координаты n-го соседа
            # и если они выходят за пределы изображения,
            # то переходим к следующему соседу
            if cx<0 or cy<0 or cx>=im.shape[0] or cy>= im.shape[1]: continue
            #Если n-й сосед уже входит в какой-либо сегмент,
            if seg[cx, cy] !=0: continue # то он пропускается
            # Находим расстояние между базовой яркостью сегмента
            Pi=metric(P/n, im[cx, cy, :]) # и яркостью n-го соседа
            if Pi<=Pc: # и если оно меньше или равно приемлемому
                try:
                    # добавляем n-ный сосед в текущий сегмент
                    seg, P, n = append(Pc, [cx, cy], seg, im, P, n, i)
                except Exception:
                    print("Достигнут лимит вызова рекурсий!")
    # возвращается матрица сегментов, сумма яркостей пикселей в сегменте
    return(seg, P, n) # и количество пикселей в сегменте

# Функция динамической сегментации
# im - изображение, Pc - приемлемый диапазон (пункт 1)
def din_seg(im, Pc):
    # создается матрица сегментов
    seg = np.zeros([im.shape[0], im.shape[1]])
    si=1 # счетчик сегментов
    while True:
        #Ищется первый пиксель, не принадлежащий
        x=np.where(seg==0) # какому-либо сегменту
        # Если такого нет
        if len(x[0])==0 or len(x[1]) == 0: break # то выходим из цикла
        # Если есть, то добавляем его в текущий сегмент
        seg, _, _ = append(Pc,[x[0][0],x[1][0]], seg, im,[0.,0.,0.], 0, si)
        cl = random_color() # Случайный цвет
        for i in range (im.shape[0]): # которым мы раскрашиваем
            for j in range (im.shape[1]): # каждый пиксель изображения
                # который принадлежит
                if seg[i,j]==si: im[i,j]=cl # текущему сегменту
        si=si+1
    # возвращаем количество сегментов
    return si

import resource, sys
import random
import colorsys
im = imr.imread('files2/test_image1.jpg')
im.setflags(write=1)
# Запрашивается большой стек
resource.setrlimit(resource.RLIMIT_STACK, (2**29,-1))
# и большой лимит вызова рекурсий
sys.setrecursionlimit(10**6)
# Динамическая сегментация
P = 90. # с приемлемым диапазоном равным
seg_count = din_seg(im, P)
print ("Всего сегментов: "+str(seg_count))
plt.imshow(im)
plt.show()
```

# *Задания для самостоятельной работы:*

1. Реализуйте динамическую сегментацию без использования рекурсии, только с помощью циклов.
2. Реализуйте алгоритм, объединяющий смежные сегменты в один сегмент.
3. Реализуйте алгоритм, берущий в прямоугольную рамочку каждый сегмент.


```python
from matplotlib.pylab import *
from matplotlib.patches import Rectangle
from matplotlib.collections import PatchCollection
from matplotlib.lines import Line2D
π = pi
```

```python
style.use(['dark_background', 'bmh'])
%matplotlib notebook
```

Car-trailer diagram (inverted image `res/car-trainer-k.png` available as well):

Car-trailer equation:
\begin{align}
\dot x &= s \cos \theta_0 \\
\dot y &= s \sin \theta_0 \\
\dot \theta_0 &= \frac{s}{L} \tan \phi \\
\dot \theta_1 &= \frac{s}{d_1} \sin(\theta_1 - \theta_0)
\end{align}
where $s$: signed speed, $\phi$: negative steering angle,

```python
class Truck:
    def __init__(self, display=False):

        self.W = 1  # car and trailer width, for drawing only
        self.L = 1 * self.W  # car length
        self.d = 4 * self.L  # d_1
        self.s = -0.1  # speed
        self.display = display

        self.box = [0, 40, -10, 10]
        if self.display:
            self.f = figure(figsize=(10, 5), num='The truck backer-upper', facecolor='none')
            self.ax = self.f.add_axes([0.01, 0.01, 0.98, 0.98], facecolor='black')
            self.patches = list()

            self.ax.axis('equal')
            b = self.box
            self.ax.axis([b[0] - 1, b[1], b[2], b[3]])
            self.ax.set_xticks([], []); self.ax.set_yticks([], [])
            self.ax.axhline(); self.ax.axvline()

        self.reset()

    def reset(self, ϕ=0):
        self.ϕ = ϕ  # car initial steering angle

        # self.θ0 = deg2rad(30)  # car initial direction
        # self.θ1 = deg2rad(-30)  # trailer initial direction
        # self.x, self.y = 20, -5  # initial car coordinates

        self.θ0 = random() * 2 * π  # 0 <= ϑ₀ < 2π
        self.θ1 = (random() - 0.5) * π / 2 + self.θ0  # -π/4 <= ϑ₁ - ϑ₀ < π/4
        self.x = (random() * .75 + 0.25) * self.box[1]
        self.y = (random() - 0.5) * (self.box[3] - self.box[2])

        # If poorly initialise, then re-initialise
        if not self.valid():
            self.reset(ϕ)

        # Draw, if display is True
        if self.display: self.draw()

    def step(self, ϕ=0, dt=1):

        # Check for illegal conditions
        if self.is_jackknifed():
            print('The truck is jackknifed!')
            return

        if self.is_offscreen():
            print('The car or trailer is off screen')
            return

        self.ϕ = ϕ
        x, y, W, L, d, s, θ0, θ1, ϕ = self._get_atributes()

        # Perform state update
        self.x += s * cos(θ0) * dt
        self.y += s * sin(θ0) * dt
        self.θ0 += s / L * tan(ϕ) * dt
        self.θ1 += s / d * sin(θ0 - θ1) * dt

        return (self.x, self.y, self.θ0, *self._traler_xy(), self.θ1)

    def state(self):
        return (self.x, self.y, self.θ0, *self._traler_xy(), self.θ1)

    def _get_atributes(self):
        return (
            self.x, self.y, self.W, self.L, self.d, self.s,
            self.θ0, self.θ1, self.ϕ
        )

    def _traler_xy(self):
        x, y, W, L, d, s, θ0, θ1, ϕ = self._get_atributes()
        return x - d * cos(θ1), y - d * sin(θ1)

    def is_jackknifed(self):
        x, y, W, L, d, s, θ0, θ1, ϕ = self._get_atributes()
        return abs(θ0 - θ1) * 180 / π > 90

    def is_offscreen(self):
        x, y, W, L, d, s, θ0, θ1, ϕ = self._get_atributes()

        x1, y1 = x + 1.5 * L * cos(θ0), y + 1.5 * L * sin(θ0)
        x2, y2 = self._traler_xy()

        b = self.box
        return not (
            b[0] <= x1 <= b[1] and b[2] <= y1 <= b[3] and
            b[0] <= x2 <= b[1] and b[2] <= y2 <= b[3]
        )

    def valid(self):
        return not self.is_jackknifed() and not self.is_offscreen()

    def draw(self):
        if not self.display: return
        if self.patches: self.clear()
        self._draw_car()
        self._draw_trailer()
        self.f.canvas.draw()

    def clear(self):
        for p in self.patches:
            p.remove()
        self.patches = list()

    def _draw_car(self):
        x, y, W, L, d, s, θ0, θ1, ϕ = self._get_atributes()
        ax = self.ax

        x1, y1 = x + L / 2 * cos(θ0), y + L / 2 * sin(θ0)
        bar = Line2D((x, x1), (y, y1), lw=5, color='C2', alpha=0.8)
        ax.add_line(bar)

        car = Rectangle(
            (x1, y1 - W / 2), L, W, 0, color='C2', alpha=0.8, transform=
            matplotlib.transforms.Affine2D().rotate_deg_around(x1, y1, θ0 * 180 / π) +
            ax.transData
        )
        ax.add_patch(car)

        x2, y2 = x1 + L / 2 ** 0.5 * cos(θ0 + π / 4), y1 + L / 2 ** 0.5 * sin(θ0 + π / 4)
        left_wheel = Line2D(
            (x2 - L / 4 * cos(θ0 + ϕ), x2 + L / 4 * cos(θ0 + ϕ)),
            (y2 - L / 4 * sin(θ0 + ϕ), y2 + L / 4 * sin(θ0 + ϕ)),
            lw=3, color='C5', alpha=1)
        ax.add_line(left_wheel)

        x3, y3 = x1 + L / 2 ** 0.5 * cos(π / 4 - θ0), y1 - L / 2 ** 0.5 * sin(π / 4 - θ0)
        right_wheel = Line2D(
            (x3 - L / 4 * cos(θ0 + ϕ), x3 + L / 4 * cos(θ0 + ϕ)),
            (y3 - L / 4 * sin(θ0 + ϕ), y3 + L / 4 * sin(θ0 + ϕ)),
            lw=3, color='C5', alpha=1)
        ax.add_line(right_wheel)

        self.patches += [car, bar, left_wheel, right_wheel]

    def _draw_trailer(self):
        x, y, W, L, d, s, θ0, θ1, ϕ = self._get_atributes()
        ax = self.ax

        x, y = x - d * cos(θ1), y - d * sin(θ1) - W / 2
        trailer = Rectangle(
            (x, y), d, W, 0, color='C0', alpha=0.8, transform=
            matplotlib.transforms.Affine2D().rotate_deg_around(x, y + W/2, θ1 * 180 / π) +
            ax.transData
        )
        ax.add_patch(trailer)

        self.patches += [trailer]
```

```python
truck = Truck(display=True)
```

```python
ϕ = deg2rad(-35)  # positive left, negative right
truck.step(ϕ)
truck.draw()
```

```python
truck.reset()
```

```python
import torch
import torch.nn as nn
from torch.optim import SGD
from tqdm import tqdm
```

```python
# Build expert data set

episodes = 10
inputs = list()
outputs = list()
# truck = Truck(); episodes = 10_000  # uncooment for creating the data set

for episode in tqdm(range(episodes)):

    truck.reset()

    while truck.valid():
        initial_state = truck.state()
        ϕ = (random() - 0.5) * π / 2
        inputs.append((ϕ, *initial_state))
        outputs.append(truck.step(ϕ))
        truck.draw()
```

```python
len(inputs), len(outputs)
```

```python
state_size = 6
steering_size = 1
hidden_units_e = 45

emulator = nn.Sequential(
    nn.Linear(steering_size + state_size, hidden_units_e),
    nn.ReLU(),
    nn.Linear(hidden_units_e, state_size)
)

optimiser_e = SGD(emulator.parameters(), lr=0.005)
criterion = nn.MSELoss()
```

```python
tensor_inputs = torch.Tensor(inputs)
tensor_outputs = torch.Tensor(outputs)
```

```python
mean = tensor_inputs.mean(0)
std = tensor_inputs.std(0)
tensor_inputs = (tensor_inputs - mean) / std
tensor_outputs = (tensor_outputs - mean[1:]) / std[1:]
```

```python
# Split the data into 80:20 for test:train.
test_size = int(len(tensor_inputs) * 0.8)
print(len(tensor_inputs), test_size)

train_inputs = tensor_inputs[:test_size]
train_outputs = tensor_outputs[:test_size]
test_inputs = tensor_inputs[test_size:]
test_outputs = tensor_outputs[test_size:]
```

```python
len(train_inputs)
```

```python
# Emulator training
cnt = 0
for i in torch.randperm(len(train_inputs)):
    ϕ_state = train_inputs[i]
    next_state_prediction = emulator(ϕ_state)

    next_state = train_outputs[i]
    loss = criterion(next_state_prediction, next_state)

    optimiser_e.zero_grad()
    loss.backward()
    optimiser_e.step()

    if cnt == 0 or (cnt + 1) % 1000 == 0:
        print(f'{cnt + 1:4d} / {len(train_inputs)}, {loss.item():.10f}')
    cnt += 1
```

```python
# Test
total_loss = 0
with torch.no_grad():
    for idx, ϕ_state in enumerate(test_inputs):
        next_state_prediction = emulator(ϕ_state)

        next_state = test_outputs[idx]
        total_loss += criterion(next_state_prediction, next_state).item()

ave_test_loss = total_loss/test_size
print(f'Test loss: {ave_test_loss:.10f}')
```

```python
# Here you need to insert the code for training the controller
# by using the emulator for backpropagation

# If you succeed, feel free to send a PR
```


[Table of Contents](http://nbviewer.ipython.org/github/rlabbe/Kalman-and-Bayesian-Filters-in-Python/blob/master/table_of_contents.ipynb)

# Designing Nonlinear Kalman Filters

```python
#format the book
%matplotlib inline
from __future__ import division, print_function
from book_format import load_style
load_style()
```

<style>
@import url('http://fonts.googleapis.com/css?family=Source+Code+Pro');
@import url('http://fonts.googleapis.com/css?family=Lora');

//@import url('http://fonts.googleapis.com/css?family=Open+Sans');
//@import url('http://fonts.googleapis.com/css?family=Vollkorn');
//@import url('http://fonts.googleapis.com/css?family=Karla');
//@import url('http://fonts.googleapis.com/css?family=Poppins');
//@import url('http://fonts.googleapis.com/css?family=Arimo');
//@import url('http://fonts.googleapis.com/css?family=Roboto');
//@import url('http://fonts.googleapis.com/css?family=Lato');
//@import url('http://fonts.googleapis.com/css?family=Domine');
//@import url('http://fonts.googleapis.com/css?family=Chivo');
//@import url('http://fonts.googleapis.com/css?family=Cardo');
//@import url('http://fonts.googleapis.com/css?family=Arvo');
//@import url('http://fonts.googleapis.com/css?family=Crimson+Text');
//@import url('http://fonts.googleapis.com/css?family=Ubuntu');
//@import url('http://fonts.googleapis.com/css?family=Fontin');
//@import url('http://fonts.googleapis.com/css?family=Raleway');
//@import url('http://fonts.googleapis.com/css?family=Merriweather');

.CodeMirror pre {
    font-family: 'Source Code Pro', Consolas, monocco, monospace;
}
    div.cell{
        //width: 950px;
        margin-left: 0% !important;
        margin-right: auto;
    }
    div.text_cell_render{
        font-family: 'Lora';
        //font-family: 'Open Sans';
        //font-family: 'Karla',verdana,arial,sans-serif;
        //font-family: 'Roboto',verdana,arial,sans-serif;
        //font-family: 'Lato',verdana,arial,sans-serif;
        //font-family: 'Domine',verdana,arial,sans-serif;
        //font-family: 'Chivo',verdana,arial,sans-serif;
        //font-family: 'Cardo',verdana,arial,sans-serif;
        //font-family: 'Arvo',verdana,arial,sans-serif;
        //font-family: 'Poppins',verdana,arial,sans-serif;
        //font-family: 'Ubuntu',verdana,arial,sans-serif;
        //font-family: 'Fontin',verdana,arial,sans-serif;
        //font-family: 'Raleway',verdana,arial,sans-serif;
        //font-family: 'Merriweather',verdana,arial,sans-serif;
        //font-family: 'Crimson Text', verdana,arial,sans-serif;
        //font-family: verdana,arial,sans-serif;
        //font-family: arial,sans-serif;
        line-height: 125%;
        font-size: 130%;
        text-align: justify;
        text-justify:inter-word;
    }
    div.text_cell code {
        background: transparent;
        color: #000000;
        font-weight: 400;
        font-size: 12pt;
        //font-style: bold;
        font-family:  'Source Code Pro', Consolas, monocco, monospace;
   }
    h1 {
        font-family: 'Open sans',verdana,arial,sans-serif;
	}

    div.input_area {
        background: #F6F6F9;
        border: 1px solid #586e75;
    }

    .text_cell_render h1 {
        font-weight: 200;
        font-size: 30pt;
        line-height: 100%;
        color:#c76c0c;
        margin-bottom: 0.5em;
        margin-top: 1em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h2 {
        font-family: 'Open sans',verdana,arial,sans-serif;
        text-align: left;
    }
    .text_cell_render h2 {
        font-weight: 200;
        font-size: 16pt;
        font-style: italic;
        line-height: 100%;
        color:#c76c0c;
        margin-bottom: 0.5em;
        margin-top: 1.5em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h3 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }
    .text_cell_render h3 {
        font-weight: 200;
        font-size: 14pt;
        line-height: 100%;
        color:#d77c0c;
        margin-bottom: 0.5em;
        margin-top: 2em;
        display: block;
        white-space: wrap;
        text-align: left;
    }
    h4 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }
    .text_cell_render h4 {
        font-weight: 100;
        font-size: 14pt;
        color:#d77c0c;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
        white-space: nowrap;
    }
    h5 {
        font-family: 'Open sans',verdana,arial,sans-serif;
    }

    .text_cell_render h5 {
        font-weight: 200;
        font-style: normal;
        color: #1d3b84;
        font-size: 16pt;
        margin-bottom: 0em;
        margin-top: 0.5em;
        display: block;
        white-space: nowrap;
    }
    div.output_subarea.output_text.output_pyout {
        overflow-x: auto;
        overflow-y: scroll;
        max-height: 50000px;
    }
    div.output_subarea.output_stream.output_stdout.output_text {
        overflow-x: auto;
        overflow-y: scroll;
        max-height: 50000px;
    }
    div.output_wrapper{
        margin-top:0.2em;
        margin-bottom:0.2em;
}

    code{
        font-size: 6pt;

    }
    .rendered_html code{
    background-color: transparent;
    }
    ul{
        margin: 2em;
    }
    ul li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
    }
    ul li li{
        padding-left: 0.2em;
        margin-bottom: 0.2em;
        margin-top: 0.2em;
    }
    ol{
        margin: 2em;
    }
    ol li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
    }
    ul li{
        padding-left: 0.5em;
        margin-bottom: 0.5em;
        margin-top: 0.2em;
    }
    a:link{
       color:#447adb;
    }
    a:visited{
       color: #1d3b84;
    }
    a:hover{
       color: #1d3b84;
    }
    a:focus{
       color:#447adb;
    }
    a:active{
       font-weight: bold;
       color:#447adb;
    }
    .rendered_html :link {
       text-decoration: underline;
    }
    .rendered_html :hover {
       text-decoration: none;
    }
    .rendered_html :visited {
      text-decoration: none;
    }
    .rendered_html :focus {
      text-decoration: none;
    }
    .rendered_html :active {
      text-decoration: none;
    }
    .warning{
        color: rgb( 240, 20, 20 )
    }
    hr {
      color: #f3f3f3;
      background-color: #f3f3f3;
      height: 1px;
    }
    blockquote{
      display:block;
      background: #fcfcfc;
      border-left: 5px solid #c76c0c;
      font-family: 'Open sans',verdana,arial,sans-serif;
      width:680px;
      padding: 10px 10px 10px 10px;
      text-align:justify;
      text-justify:inter-word;
      }
      blockquote p {
        margin-bottom: 0;
        line-height: 125%;
        font-size: 100%;
      }
</style>

## Introduction

** Author's note: I was initially planning to have a design nonlinear chapter that compares various approaches. This may or may not happen, but for now this chapter has no useful content and I suggest not reading it. **

We see that the Kalman filter reasonably tracks the ball. However, as already explained, this is a silly example; we can predict trajectories in a vacuum with arbitrary precision; using a Kalman filter in this example is a needless complication.

### Kalman Filter with Air Drag

I will dispense with the step 1, step 2, type approach and proceed in a more natural style that you would use in a non-toy engineering problem. We have already developed a Kalman filter that does excellently at tracking a ball in a vacuum, but that does not incorporate the effects of air drag into the model. We know that the process model is implemented with $\textbf{F}$, so we will turn our attention to that immediately.

Notionally, the computation that $\textbf{F}$ computes is

$$x' = Fx$$

With no air drag, we had

$$
\mathbf{F} = \begin{bmatrix}
1 & \Delta t & 0 & 0 & 0 \\
0 & 1 & 0 & 0 & 0 \\
0 & 0 & 1 & \Delta t & \frac{1}{2}{\Delta t}^2 \\
0 & 0 & 0 & 1 & \Delta t \\
0 & 0 & 0 & 0 & 1
\end{bmatrix}
$$

which corresponds to the equations

$$
\begin{aligned}
x &= x + v_x \Delta t \\
v_x &= v_x \\
\\
y &= y + v_y \Delta t + \frac{a_y}{2} {\Delta t}^2 \\
v_y &= v_y + a_y \Delta t \\
a_y &= a_y
\end{aligned}
$$

From the section above we know that our new Euler equations must be

$$
\begin{aligned}
x &= x + v_x \Delta t \\
v_x &= v_x \\
\\
y &= y + v_y \Delta t + \frac{a_y}{2} {\Delta t}^2 \\
v_y &= v_y + a_y \Delta t \\
a_y &= a_y
\end{aligned}
$$

## Realistic 2D Position Sensors

The position sensor in the last example are not very realistic. In general there is no 'raw' sensor that provides (x,y) coordinates. We have GPS, but GPS already uses a Kalman filter to create a filtered output; we should not be able to improve the signal by passing it through another Kalman filter unless we incorporate additional sensors to provide additional information. We will tackle that problem later.

Consider the following set up. In an open field we put two transmitters at a known location, each transmitting a signal that we can detect. We process the signal and determine  how far we are from that signal, with some noise. First, let's look at a visual depiction of that.

```python
import matplotlib.pyplot as plt

circle1=plt.Circle((-4, 0), 5, color='#004080',
                   fill=False, linewidth=20, alpha=.7)
circle2=plt.Circle((4, 0), 5, color='#E24A33',
                   fill=False, linewidth=5, alpha=.7)

fig = plt.gcf()
ax = fig.gca()

plt.axis('equal')
plt.xlim((-10, 10))
plt.ylim((-10, 10))

plt.plot ([-4, 0], [0, 3], c='#004080')
plt.plot ([4, 0], [0, 3], c='#E24A33')
plt.text(-4, -.5, "A", fontsize=16, horizontalalignment='center')
plt.text(4, -.5, "B", fontsize=16, horizontalalignment='center')

ax.add_artist(circle1)
ax.add_artist(circle2)
plt.show()
```

Here I have attempted to show transmitter A, drawn in red, at (-4,0) and a second one B, drawn in blue, at (4,0). The red and blue circles show the range from the transmitters to the robot, with the width illustrating the effect of the $1\sigma$ angular error for each transmitter. Here I have given the blue transmitter more error than the red one. The most probable position for the robot is where the two circles intersect, which I have depicted with the red and blue lines. You will object that we have two intersections, not one, but we will see how we deal with that when we design the measurement function.

This is a very common sensor set up. Aircraft still use this system to navigate, where it is called DME (Distance Measuring Equipment). Today GPS is a much more common navigation system, but I have worked on an aircraft where we integrated sensors like this into our filter along with the GPS, INS, altimeters, etc. We will tackle what is called *multi-sensor fusion* later; for now we will just address this simple configuration.

The first step is to design our state variables. We will assume that the robot is traveling in a straight direction with constant velocity. This is unlikely to be true for a long period of time, but is acceptable for short periods of time. This does not differ from the previous problem - we will want to track the values for the robot's position and velocity. Hence,

$$\mathbf{x} =
\begin{bmatrix}x\\v_x\\y\\v_y\end{bmatrix}$$

The next step is to design the state transition function. This also will be the same as the previous problem, so without further ado,

$$
\mathbf{x}' = \begin{bmatrix}1& \Delta t& 0& 0\\0& 1& 0& 0\\0& 0& 1& \Delta t\\ 0& 0& 0& 1\end{bmatrix}\mathbf{x}$$

The next step is to design the control inputs. We have none, so we set ${\mathbf{B}}=0$.

The next step is to design the measurement function $\mathbf{z} = \mathbf{Hx}$. We can model the measurement using the Pythagorean theorem.

$$
z_a = \sqrt{(x-x_A)^2 + (y-y_A)^2} + v_a\\[1em]
z_b = \sqrt{(x-x_B])^2 + (y-y_B)^2} + v_b
$$

where $v_a$ and $v_b$ are white noise.

We see an immediate problem. The Kalman filter is designed for linear equations, and this is obviously nonlinear. In the next chapters we will look at several ways to handle nonlinear problems in a robust way, but for now we will do something simpler. If we know the approximate position of the robot than we can linearize these equations around that point. I could develop the generalized mathematics for this technique now, but instead let me just present the worked example to give context to that development.

Instead of computing $\mathbf{H}$ we will compute the partial derivative of $\mathbf{H}$ with respect to the robot's position $\mathbf{x}$. You are probably familiar with the concept of partial derivative, but if not, it just means how $\mathbf{H}$ changes with respect to the robot's position. It is computed as the partial derivative of $\mathbf{H}$ as follows:

$$\frac{\partial \mathbf{h}}{\partial \mathbf{x}} =
\begin{bmatrix}
\frac{\partial h_1}{\partial x_1} & \frac{\partial h_1}{\partial x_2} &\dots \\
\frac{\partial h_2}{\partial x_1} & \frac{\partial h_2}{\partial x_2} &\dots \\
\vdots & \vdots
\end{bmatrix}
$$

Let's work the first partial derivative. We want to find

$$\frac{\partial }{\partial x} \sqrt{(x-x_A)^2 + (y-y_A)^2}
$$

Which we compute as

$$
\begin{aligned}
\frac{\partial h_1}{\partial x} &= ((x-x_A)^2 + (y-y_A)^2))^\frac{1}{2} \\
&= \frac{1}{2}\times 2(x-x_a)\times ((x-x_A)^2 + (y-y_A)^2))^{-\frac{1}{2}} \\
&= \frac{x_r - x_A}{\sqrt{(x_r-x_A)^2 + (y_r-y_A)^2}}
\end{aligned}
$$

We continue this computation for the partial derivatives of the two distance equations with respect to $x$, $y$, $dx$ and $dy$, yielding

$$\frac{\partial\mathbf{h}}{\partial\mathbf{x}}=
\begin{bmatrix}
\frac{x_r - x_A}{\sqrt{(x_r-x_A)^2 + (y_r-y_A)^2}} & 0 &
\frac{y_r - y_A}{\sqrt{(x_r-x_A)^2 + (y_r-y_A)^2}} & 0 \\
\frac{x_r - x_B}{\sqrt{(x_r-x_B)^2 + (y_r-y_B)^2}} & 0 &
\frac{y_r - y_B}{\sqrt{(x_r-x_B)^2 + (y_r-y_B)^2}} & 0 \\
\end{bmatrix}
$$

That is pretty painful, and these are very simple equations. Computing the Jacobian can be extremely difficult or even impossible for more complicated systems. However, there is an easy way to get Python to do the work for you by using the SymPy module [1]. SymPy is a Python library for symbolic mathematics. The full scope of its abilities are beyond this book, but it can perform algebra, integrate and differentiate equations, find solutions to differential equations, and much more. We will use it to compute our Jacobian!

First, a simple example. We will import SymPy, initialize its pretty print functionality (which will print equations using LaTeX). We will then declare a symbol for NumPy to use.

```python
import sympy
from sympy import init_printing
init_printing(use_latex='png')

phi, x = sympy.symbols('\phi, x')
phi
```

Notice how we use a latex expression for the symbol `phi`. This is not necessary, but if you do it will render as LaTeX when output. Now let's do some math. What is the derivative of $\sqrt{\phi}$?

```python
sympy.diff('sqrt(phi)')
```

We can factor equations.

```python
sympy.factor('phi**3 -phi**2 + phi - 1')
```

SymPy has a remarkable list of features, and as much as I enjoy exercising its features we cannot cover them all here. Instead, let's compute our Jacobian.

```python
from sympy import symbols, Matrix
phi = symbols('\phi')
phi

x, y, xa, xb, ya, yb, dx, dy = symbols('x y x_a x_b y_a y_b dx dy')

H = Matrix([[sympy.sqrt((x-xa)**2 + (y-ya)**2)],
            [sympy.sqrt((x-xb)**2 + (y-yb)**2)]])

state = Matrix([x, dx, y, dy])
H.jacobian(state)
```

In a nutshell, the entry (0,0) contains the difference between the x coordinate of the robot and transmitter A's x coordinate divided by the distance between the robot and A. (2,0) contains the same, except for the y coordinates of the robot and transmitters. The bottom row contains the same computations, except for transmitter B. The 0 entries account for the velocity components of the state variables; naturally the range does not provide us with velocity.

The values in this matrix change as the robot's position changes, so this is no longer a constant; we will have to recompute it for every time step of the filter.

If you look at this you may realize that this is just a computation of x/dist and y/dist, so we can switch this to a trigonometic form with no loss of generality:

$$\frac{\partial\mathbf{h}}{\partial\mathbf{x}}=
\begin{bmatrix}
-\cos{\theta_A} & 0 & -\sin{\theta_A} & 0 \\
-\cos{\theta_B} & 0 & -\sin{\theta_B} & 0
\end{bmatrix}
$$

However, this raises a huge problem. We are no longer computing $\mathbf{H}$, but $\Delta\mathbf{H}$, the change of $\mathbf{H}$. If we passed this into our Kalman filter without altering the rest of the design the output would be nonsense. Recall, for example, that we multiply $\mathbf{Hx}$ to generate the measurements that would result from the given estimate of $\mathbf{x}$ But now that $\mathbf{H}$ is linearized around our position it contains the *change* in the measurement function.

We are forced, therefore, to use the *change* in $\mathbf{x}$ for our state variables. So we have to go back and redesign our state variables.

>Please note this is a completely normal occurrence in designing Kalman filters. The textbooks present examples like this as *fait accompli*, as if it is trivially obvious that the state variables needed to be velocities, not positions. Perhaps once you do enough of these problems it would be trivially obvious, but at that point why are you reading a textbook? I find myself reading through a presentation multiple times, trying to figure out why they made a choice, finally to realize that it is because of the consequences of something on the next page. My presentation is longer, but it reflects what actually happens when you design a filter. You make what seem reasonable design choices, and as you move forward you discover properties that require you to recast your earlier steps. As a result, I am going to somewhat abandon my **step 1**, **step 2**, etc.,  approach, since so many real problems are not quite that straightforward.

If our state variables contain the velocities of the robot and not the position then how do we track where the robot is? We can't. Kalman filters that are linearized in this fashion use what is called a *nominal trajectory* - i.e. you assume a position and track direction, and then apply the changes in velocity and acceleration to compute the changes in that trajectory. How could it be otherwise? Recall the graphic showing the intersection of the two range circles - there are two areas of intersection. Think of what this would look like if the two transmitters were very close to each other - the intersections would be two very long crescent shapes. This Kalman filter, as designed, has no way of knowing your true position from only distance measurements to the transmitters. Perhaps your mind is already leaping to ways of working around this problem. If so, stay engaged, as later sections and chapters will provide you with these techniques. Presenting the full solution all at once leads to more confusion than insight, in my opinion.

So let's redesign our *state transition function*. We are assuming constant velocity and no acceleration, giving state equations of
$$
\dot{x}' = \dot{x} \\
\ddot{x}' = 0 \\
\dot{y}' = \dot{y} \\
\dot{y}' = 0$$

This gives us the the *state transition function* of

$$
\mathbf{F} = \begin{bmatrix}0 &1 & 0& 0\\0& 0& 0& 0\\0& 0& 0& 1\\ 0& 0& 0& 0\end{bmatrix}$$

A final complication comes from the measurements that we pass in. $\mathbf{Hx}$ is now computing the *change* in the measurement from our nominal position, so the measurement that we pass in needs to be not the range to A and B, but the *change* in range from our measured range to our nominal position.

There is a lot here to take in, so let's work through the code bit by bit. First we will define a function to compute $\frac{\partial\mathbf{h}}{\partial\mathbf{x}}$ for each time step.

```python
from math import sin, cos, atan2

def H_of(pos, pos_A, pos_B):
    """ Given the position of our object at 'pos' in 2D, and two
    transmitters A and B at positions 'pos_A' and 'pos_B', return
    the partial derivative of H
    """

    theta_a = atan2(pos_a[1] - pos[1], pos_a[0] - pos[0])
    theta_b = atan2(pos_b[1] - pos[1], pos_b[0] - pos[0])

    return np.array([[0, -cos(theta_a), 0, -sin(theta_a)],
                     [0, -cos(theta_b), 0, -sin(theta_b)]])
```

Now we need to create our simulated sensor.

```python
from numpy.random import randn

class DMESensor(object):
    def __init__(self, pos_a, pos_b, noise_factor=1.0):
        self.A = pos_a
        self.B = pos_b
        self.noise_factor = noise_factor

    def range_of(self, pos):
        """ returns tuple containing noisy range data to A and B
        given a position 'pos'
        """

        ra = math.sqrt((self.A[0] - pos[0])**2 + (self.A[1] - pos[1])**2)
        rb = math.sqrt((self.B[0] - pos[0])**2 + (self.B[1] - pos[1])**2)

        return (ra + randn()*self.noise_factor,
                rb + randn()*self.noise_factor)
```

Finally, we are ready for the Kalman filter code. I will position the transmitters at x=-100 and 100, both with y=-20. This gives me enough space to get good triangulation from both as the robot moves. I will start the robot at (0,0) and move by (1,1) each time step.

```python
import code.book_plots as bp
from filterpy.kalman import KalmanFilter
import math
import numpy as np

pos_a = (100, -20)
pos_b = (-100, -20)

f1 = KalmanFilter(dim_x=4, dim_z=2)

f1.F = np.array ([[0, 1, 0, 0],
                  [0, 0, 0, 0],
                  [0, 0, 0, 1],
                  [0, 0, 0, 0]], dtype=float)

f1.R *= 1.
f1.Q *= .1

f1.x = np.array([[1, 0, 1, 0]], dtype=float).T
f1.P = np.eye(4) * 5.

# initialize storage and other variables for the run
count = 30
xs, ys = [], []
pxs, pys = [], []

# create the simulated sensor
d = DMESensor(pos_a, pos_b, noise_factor=3.)

# pos will contain our nominal position since the filter does not
# maintain position.
pos = [0, 0]

for i in range(count):
    # move (1,1) each step, so just use i
    pos = [i, i]

    # compute the difference in range between the nominal track
    # and measured ranges
    ra,rb = d.range_of(pos)
    rx,ry = d.range_of((pos[0] + f1.x[0, 0], pos[1] + f1.x[2, 0]))
    z = np.array([[ra - rx], [rb - ry]])

    # compute linearized H for this time step
    f1.H = H_of (pos, pos_a, pos_b)

    # store stuff so we can plot it later
    xs.append(f1.x[0, 0]+i)
    ys.append(f1.x[2, 0]+i)
    pxs.append(pos[0])
    pys.append(pos[1])

    # perform the Kalman filter steps
    f1.predict()
    f1.update(z)

bp.plot_filter(xs, ys)
bp.plot_track(pxs, pys)
plt.legend(loc=2)
plt.show()
```

## Linearizing the Kalman Filter

Now that we have seen an example of linearizing the Kalman filter we are in a position to better understand the math.

We start by assuming some function $\mathbf f$

## Example: A falling Ball

**author's note: ignore this section for now. **

In the **Designing Kalman Filters** chapter I first considered tracking a ball in a vacuum, and then in the atmosphere. The Kalman filter performed very well for vacuum, but diverged from the ball's path in the atmosphere. Let us look at the output; to avoid littering this chapter with code from that chapter I have placed it all in the file `ekf_internal.py'.

```python
import code.ekf_internal as ekf
ekf.plot_ball()
```

We can artificially force the Kalman filter to track the ball by making $Q$ large. That would cause the filter to mistrust its prediction, and scale the kalman gain $K$ to strongly favor the measurments. However, this is not a valid approach. If the Kalman filter is correctly predicting the process we should not 'lie' to the filter by telling it there are process errors that do not exist. We may get away with that for some problems, in some conditions, but in general the Kalman filter's performance will be substandard.

Recall from the **Designing Kalman Filters** chapter that the acceleration is

$$a_x = (0.0039 + \frac{0.0058}{1+\exp{[(v-35)/5]}})*v*v_x \\
a_y = (0.0039 + \frac{0.0058}{1+\exp{[(v-35)/5]}})*v*v_y- g
$$

These equations will be *very* unpleasant to work with while we develop this subject, so for now I will retreat to a simpler one dimensional problem using this simplified equation for acceleration that does not take the nonlinearity of the drag coefficient into account:

$$\begin{aligned}
\ddot{y} &= \frac{0.0034ge^{-y/20000}\dot{y}^2}{2\beta} - g \\
\ddot{x} &= \frac{0.0034ge^{-x/20000}\dot{x}^2}{2\beta}
\end{aligned}$$

Here $\beta$ is the ballistic coefficient, where a high number indicates a low drag.

This is still nonlinear, so we need to linearize this equation at the current state point. If our state is position and velocity, we need an equation for some arbitrarily small change in $\mathbf{x}$, like so:

$$ \begin{bmatrix}\Delta \dot{x} \\ \Delta \ddot{x} \\ \Delta \dot{y} \\ \Delta \ddot{y}\end{bmatrix} =
\large\begin{bmatrix}
\frac{\partial \dot{x}}{\partial x} &
\frac{\partial \dot{x}}{\partial \dot{x}} &
\frac{\partial \dot{x}}{\partial y} &
\frac{\partial \dot{x}}{\partial \dot{y}} \\
\frac{\partial \ddot{x}}{\partial x} &
\frac{\partial \ddot{x}}{\partial \dot{x}}&
\frac{\partial \ddot{x}}{\partial y}&
\frac{\partial \dot{x}}{\partial \dot{y}}\\
\frac{\partial \dot{y}}{\partial x} &
\frac{\partial \dot{y}}{\partial \dot{x}} &
\frac{\partial \dot{y}}{\partial y} &
\frac{\partial \dot{y}}{\partial \dot{y}} \\
\frac{\partial \ddot{y}}{\partial x} &
\frac{\partial \ddot{y}}{\partial \dot{x}}&
\frac{\partial \ddot{y}}{\partial y}&
\frac{\partial \dot{y}}{\partial \dot{y}}
\end{bmatrix}\normalsize
\begin{bmatrix}\Delta x \\ \Delta \dot{x} \\ \Delta \dot{y} \\ \Delta \ddot{y}\end{bmatrix}$$

The equations do not contain both an x and a y, so any partial derivative with both in it must be equal to zero. We also know that $\large\frac{\partial \dot{x}}{\partial x}\normalsize = 0$ and that  $\large\frac{\partial \dot{x}}{\partial \dot{x}}\normalsize = 1$, so our matrix ends up being

$$\mathbf{F} = \begin{bmatrix}0&1&0&0 \\
\frac{0.0034e^{-x/22000}\dot{x}^2g}{44000\beta}&0&0&0
\end{bmatrix}$$

$$\begin{aligned}\ddot{x} &= -\frac{1}{2}C_d\rho A \dot{x}\\
\ddot{y} &= -\frac{1}{2}C_d\rho A \dot{y}-g\end{aligned}$$

```python
from sympy.abc import *
from sympy import *

init_printing(pretty_print=True, use_latex='mathjax')

x1 = (0.0034*g*exp(-x/22000)*((x)**2))/(2*b) - g

x2 = (a*g*exp(-x/c)*(Derivative(x)**2))/(2*b) - g

#pprint(x1)
#pprint(Derivative(x)*Derivative(x,n=2))
#pprint(diff(x2, x))
```

** orphan text
This approach has many issues. First, of course, is the fact that the linearization does not produce an exact answer. More importantly, we are not linearizing the actual path, but our filter's estimation of the path. We linearize the estimation because it is statistically likely to be correct; but of course it is not required to be. So if the filter's output is bad that will cause us to linearize an incorrect estimate, which will almost certainly lead to an even worse estimate. In these cases the filter will quickly diverge. This is where the 'black art' of Kalman filter comes in. We are trying to linearize an estimate, and there is no guarantee that the filter will be stable. A vast amount of the literature on Kalman filters is devoted to this problem. Another issue is that we need to linearize the system using analytic methods. It may be difficult or impossible to find an analytic solution to some problems. In other cases we may be able to find the linearization, but the computation is very expensive. **

## References

[1] http://sympy.org