# Decision Trees and the Iterative Dichtomiser 3 (ID3) Algorithm
By Marco A. Peyrot for course 15-388, Carnegie Mellon University

Image obtained from [source](https://littleml.files.wordpress.com/2013/04/traditional.png). _No rights over it owned_.

## Introduction

In this tutorial, you will learn about the Iterative Dichtomiser 3 (ID3) Algorithm through an interactive example. The tutorial is divided into sections that will lead you through all the necessary steps to use ID3; from data gathering to the application of the algorithm, to the final decision tree it returns.

### What is a decision tree?

A decision tree is a regression or classification model in the form of a tree (not strictly binary) that is used to determine if individual _predictor_ values will lead to a particular decision. Each branch in this data structure represents a possible choice between several alternatives, and each leaf node accounts for a decision. Therefore, it finds common usage in aiding decision making, where a decision is a product of traversing the tree from top to bottom.

> A decision tree represents the learned functions obtained from approximating discrete-valued target functions.
> -- <cite>Tom M. Mitchell,  Machine Learning (1997)</cite>.

A decision tree learning algorithm involves processing some data set through some criteria (for example Gini impurity or entropy) to generate nodes. Non-leaf nodes will be processed further until only leaf nodes remain.

### What is the ID3 algorithm?

The ID3 is a traditional _Decision Tree Learning Algorithm_ for creating decision trees based on some observable data. The algorithm was invented by Ross Quinlan in 1983 and operates by _dichtomizing_ (to divide or separate into two parts) data instances into branches recursively, hence the name.

This algorithm uses a top-down greedy search to generate the decision tree. It is one of the simplest tree learning algorithms and serves as the predecessor for the C4.5 algorithm (click [here](http://saiconference.com/Downloads/SpecialIssueNo10/Paper_3-A_comparative_study_of_decision_tree_ID3_and_C4.5.pdf) to learn about this algorithm). It is often one of the first tree learning algorithms taught in machine learning courses because it makes the classification process self-evident.

There is a limitation to consider: it can only operate on discrete values for both predictors and targets; that is, it cannot operate on continuous data. There exist several modifications and improvements of the algorithm to cope with these limitations. However, they are out of scope for the present tutorial.

Without further delay, let's start!

```python
import pandas as pd
import math
import graphviz as gv
from IPython.display import Image
from IPython.core.display import HTML
import numpy as np

# used to render the graphs in the center of the notebook
HTML("""
<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: middle;
}
</style>
""")
```

<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: middle;
}
</style>

You may have to manually install with [pip](https://pypi.python.org/pypi/pip) the `Pandas`, `Graphviz`, and `IPython` modules before continuing with the tutorial.

## The training data set

For this tutorial, we will be using a training data set from the Kaggle website about global shark attacks. This data was collected as a CSV file and is located in the same directory as this notebook. The file can be downloaded from this [source](https://www.kaggle.com/teajay/global-shark-attacks). The contents of the archive are:

In this tutorial we will determine based on the information in this [website](http://www.theinertia.com/surf/10-shark-attack-beaches/), _upon which factors is it likely that a person will suffer an attack from a Bull or a Tiger shark?_ From this data our prediction label (y) will be the Species of Sharks. We will create a pandas `DataFrame` to hold this data and start processing it. Notice that some of this values are not discrete or irrelevant to the problem, so we must take them out of the Data Frame or classify them into categories.

The `get_target` function takes a label column and returns a `target` list that will be our prediction; in this case 1 for a Tiger and 0 for a Bull shark.

```python
def get_target(df, target_name, pos_value):
    """ Given a DataFrame, create the target list for the data set as a series of 1s for positive outcome
        and 0s for negative.

        Args:   df(pandas.DataFrame): the dataframe from which the target data will be obtained,
                target_name(String): the column name that contains the target data,
                pos_value(String): the value that is considered a positive outcome,
        Return: a list with the target values."""

    return [1 if x == pos_value else 0 for x in df.loc[:,target_name]]
```

Let's gather our data:

```python
# pass the data from the CSV file to a Data Frame
training_df = pd.read_csv("attacks.csv", na_filter=False)

# we do not care about the case nuber for this problem
training_df.drop('CaseNumber', axis=1, inplace=True)
# the date is also irrelevant, because it is too specific
training_df.drop('Date', axis=1, inplace=True)
# The following fields are irrelevant for this example because they are too specific or hard to classify into
# concrete categories. Feel free to play with them to get different results.
training_df.drop('Area', axis=1, inplace=True)
training_df.drop('Location', axis=1, inplace=True)
training_df.drop('Name', axis=1, inplace=True)
training_df.drop('Age', axis=1, inplace=True)
training_df.drop('Injury', axis=1, inplace=True)
training_df.drop('Time', axis=1, inplace=True)
training_df.drop('InvestigatororSource', axis=1, inplace=True)
training_df.drop('Year', axis=1, inplace=True)
training_df.drop('Sex', axis=1, inplace=True)
training_df.drop('Fatal(Y/N)', axis=1, inplace=True)
training_df.drop('Type', axis=1, inplace=True)

# The data may have symbols outside the range supported by ASCII, therefore we remove this 'illegal' characters
training_df.Activity.replace({r'[^\x00-\x7F]+':''}, regex=True, inplace=True)
training_df.Species.replace({r'[^\x00-\x7F]+':''}, regex=True, inplace=True)
# The data sometimes adds some additional information about the shark (like length), we want this fields to also
# be accounted for in our analysis.
training_df.Species.replace({r'[\s\S]*\bBull shark[\s\S]*':'Bull shark'}, regex=True, inplace=True)
training_df.Species.replace({r'[\s\S]*\bTiger shark[\s\S]*':'Tiger shark'}, regex=True, inplace=True)

# discard rows with missing values
for col in training_df.columns:
    training_df = training_df[~training_df[col].isin([''])]

training_df = training_df.loc[training_df['Country'].isin(['MEXICO', 'USA', 'AUSTRALIA', 'SOUTH AFRICA','BRAZIL', 'EGYPT'])]
training_df = training_df.loc[training_df['Activity'].isin(['Surfing', 'Swimming', 'SCUBA diving','Scuba diving', 'Fishing'])]
training_df = training_df.loc[training_df['Species'].isin(['Tiger shark', 'Bull shark'])]

training_target = get_target(training_df, "Species", "Tiger shark")
print training_df.head()
```

           Country      Activity      Species
    7          USA      Swimming  Tiger shark
    13         USA       Surfing   Bull shark
    60         USA      Swimming   Bull shark
    107     BRAZIL  Scuba diving  Tiger shark
    115  AUSTRALIA       Surfing   Bull shark

Notice that we are only considering the countries where most shark attacks are reported, alongside the most common practices people do when attacked. You may change these parameters to investigate some other feature of the dataset.

## Entropy

Now that we have our data in a flexible format let's create a function to calculate the entropy of each value. But first, let's explain what entropy is:

> Entropy is the amount of _impurity_ in the information that some value can provide to make a decision. On the other hand, _information_ measures the _purity_ of that value.

For example, tossing a fair two-sided coin in the air has an entropy of 0.5 and an information gain of 0.5, because there is a 50% chance that either side finishes upward. In contrast, a coin with a single side will have an entropy of 0 and an information gain of 1.

To find the entropy of a value in a data set we use the following formula:

\begin{equation}
    Entropy(S) = - P(positive)\log_2{P(positive)} - P(negative)\log_2{P(negative)}
\end{equation}

where, $P$ is the proportion of positive or negative examples in $S$. $P$ is depicted as:

\begin{equation}
    P(S) = \frac{\sum\limits_{x=1}^n x[x \equiv S]}{\sum\limits_{x=1}^n x}
\end{equation}

where, $S$ is a category of the data (e.g. Area), and $x$ are the distinct values of $S$.

If we take `Sex` for example; we first calculate $P(positive)$ for the 'Masculine' value by dividing the number of occurrences of this value that lead to a 'Yes' in the target (Tiger Shark) by the total number of events of 'Masculine'. Then we do the same, but for the ones that lead to a 'No', which will give us some values that sum up to one respectively. We repeat the process for 'Feminine'. With these values, we can now calculate $Entropy(S)$ for each value in the category.

## Entropy in the ML context

And how does Entropy relate to the context of a Machine Learning algorithm? Well, it is the _probabilistic loss function_ where _y_ is considered to be a Bernoulli random variable. If you look carefully, you will notice the similarities it has with the _negative log probability_ function used for maximum likelihood estimation:

\begin{equation}
    loss(h_\theta(x), y) = - y\log{h_\theta(x)} - (1 - y)\log{1 - h_\theta(x)}
\end{equation}

where $h_\theta(x)$ is the probability of the attacked being provoked by a Tiger Shark.

Now that all the theory has been explained let's write some code. We require two functions; the first will calculate the entropy for the target values in the data set. The second will calculate the entropy and information gain for a determined category.

```python
def set_entropy(target):
    """ Given a target list for a data set, calculate the entropy for it.
        Args:
            target(array-like): the target values from the data set.
        Return:
            the entropy of the data set target values. """
    neg_target = [0 if x is 1 else 1 for x in target]

    pos = sum(target) / float(len(target))
    neg = sum(neg_target) / float(len(neg_target))

    entropy = 0

    # if a value has no entropy for either the negative or positve case, omit it from the entropy
    # calculation.
    if pos != 0:
        entropy -= pos * math.log(pos, 2)
    if neg != 0:
        entropy -= neg * math.log(neg, 2)

    return entropy

def category_entropy(df, target, target_entropy, category):
    """ Given a pandas DataFrame, calculate the entropy and information gain for a given category.
        Args:
            df(pandas.DataFrame): the dataframe from which the entropy will be calculated.
            target(array-like): the target values from the data set.
            target_entropy(numeric-value): the entropy for the target data set.
            category(String): the category for which to calculate the entropy, must be contained as a column in df.
        Return:
            A tuple with two elements:  1. The information gain of the category.
                                        2. a dictionary from every value in the category to its
                                           corresponding entropy alongside the value of the positive entropy. """
    values = df.loc[:,category].unique()
    neg_target = [0 if x is 1 else 1 for x in target]
    result_dict = {}

    for value in values:
        value_list = [1 if x is value else 0 for x in df.loc[:,category]]
        if sum(value_list) != 0:
            pos = sum([a*b for a,b in zip(value_list,target)]) / float(sum(value_list))
        else:
            pos = 0

        if sum(value_list) != 0:
            neg = sum([a*b for a,b in zip(value_list,neg_target)]) / float(sum(value_list))
        else:
            neg = 0

        entropy = 0

        # if a value has no entropy for either the negative or positve case, omit it from the entropy
        # calculation.
        if pos != 0:
            entropy -= pos * math.log(pos, 2)
        if neg != 0:
            entropy -= neg * math.log(neg, 2)

        result_dict[value] = (entropy, pos)

    result = target_entropy

    for key,value in result_dict.iteritems():
        result -= sum([1 if x is key else 0 for x in df.loc[:,category]]) / float(len(target)) * value[0]

    return (category,result,result_dict)
```

_Information gain_ consists of subtracting to the entropy of the target values the occurrences of each value times its entropy. It is given by the formula:

\begin{equation}
    Gain(category) = Entropy(target) - \frac{\sum\limits_{x=1}^n x[x \equiv value]}{{\sum\limits_{x=1}^n x}} * Entropy(value)
\end{equation}

The information gain will be the target value minus the number of occurrences of each category of the entries in the data set times its entropy. From an ML perspective, this is how we will minimize the loss function.

With these two functions, we can calculate the entropy for the target values in the data set and each category. Let's do it for the `country` category:

```python
data_set_entropy = set_entropy(training_target)
category, gain,entropy = category_entropy(training_df, training_target, data_set_entropy, "Country")

print category + " Information Gain: " + str(gain)

for key,value in entropy.iteritems():
    print str(key) + " Entropy: " + str(value[0])
```

    Country Information Gain: 0.0597117399496
    BRAZIL Entropy: 1.0
    AUSTRALIA Entropy: 0.998363672594
    SOUTH AFRICA Entropy: 0.0
    USA Entropy: 0.928839146188
    MEXICO Entropy: 0.811278124459

The next step is to apply the previous functions for all the categories in the dataset, let's create a function that does this for us:

```python
def entropies(df, target, target_entropy):
    """ Given a pandas DataFrame, calculate the entropy and information gain for all categories.
        Args:
            df(pandas.DataFrame): the dataframe from which the entropies will be calculated.
            target(array-like): the target values from the data set.
            target_entropy(numeric-value): the entropy for the target data set.
        Return:
            A list with the information gain and entropies for each category. """
    result = []
    for category in df.columns:
        value = category_entropy(df, target, target_entropy, category)
        result.append(value)

    return result
```

```python
entropy_results = entropies(training_df.drop("Species", 1), training_target, data_set_entropy)

for category, gain, entropy in entropy_results:
    print category + " Information Gain: " + str(gain)
    for key,value in entropy.iteritems():
        print str(key) + " Entropy: " + str(value[0])
    print
```

    Country Information Gain: 0.0597117399496
    BRAZIL Entropy: 1.0
    AUSTRALIA Entropy: 0.998363672594
    SOUTH AFRICA Entropy: 0.0
    USA Entropy: 0.928839146188
    MEXICO Entropy: 0.811278124459

    Activity Information Gain: 0.0986360631097
    SCUBA diving Entropy: 0.0
    Scuba diving Entropy: 0.0
    Surfing Entropy: 0.746736607483
    Fishing Entropy: 1.0
    Swimming Entropy: 0.99777247209

The final step of the ID3 algorithm is to get the category with the highest information gain. In this case, it is the activity category. Activity will be the root node of the decision tree and its different values its children. If one of these nodes has an entropy of zero, it is considered a leaf-node. On the other case, the node will act as the root node for the next subtree. Let's create a function to see it visually.

### Graphviz

To plot the decision tree generated by the ID3 algorithm, we will make use of the Graphviz module. It is a Python library that lets you create directed and undirected graphs with some level of customization. A short tutorial can be found [here](http://matthiaseisen.com/articles/graphviz/).

The first function we need to create is one to get the category with the highest information gain. Its result will get passed to our tree function to generate the corresponding tree for that category.

```python
def find_largest(entropy_results):
    """ Given the result generated by the entropies function, find the element with the highest information gain
        value
        Args:
            entropy_results(list of tuples): the list generated by the entropies function, each element represents a
                                             category of the data set. The format of each element is defined as
                                             a tuple composed of:

                                             1. the category name
                                             2. the information gain for that category
                                             3. a dictionary from each value of the category to their entropy
        Return:
            The element from the entropy_results list with the biggest information gain. """
    max_gain = 0

    for category, gain, entropy in entropy_results:
        if gain >= max_gain:
            result = (category, entropy)
            max_gain = gain

    return result
```

To modularize and simplify ID3. We will create a function that only creates a tree for a given category. The ID3 function will call it per category and join the resulting sub-trees to form the final decision tree. Since `Graphviz` identifies each node by name and there can be no values in the data set that share a common name, but the target values.

```python
def sub_tree(graph, category, entropy, parent):
    """ Given a category, its entropy and its parent, create a tree structure that has the category as the parent
        its different values.
        Args:
            graph: a graphviz object, this is were the subtree will form part of.
            category: the category for which the tree will be made of.
            entropy: the differnt values that the category can have, each one of these will be a leaf of this tree.
            parent: the parent of the category node, if None the category node passed to this function will be the new
            root of the tree. """
    graph.node(category)

    if parent is not None:
        graph.edge(parent, category)

    for key, value in entropy.iteritems():
        graph.node(key)
        graph.edge(category, key)
```

Now that we have a way to create a tree for each category let's create one for outlook. Notice that we first have to generate and save the graph as a .png file to show it.

```python
category, entropy = find_largest(entropy_results)
# creates a directed graph
g1 = gv.Digraph(format='png')
# creates the tree for the outlook category (since it is the one with the biggest information gain)
sub_tree(g1, category, entropy, None)
# saves the graph as a .png image in the img directory with name g1
filename = g1.render(filename='img/g1')
# displays the tree
Image("img/g1.png")
```

## The ID3 Algorithm

Now that we have all the utility functions we need, creating the ID3 algorithm is quite simple. We pass our graph object and all the information we require to call the previous methods. The `decision_tree` will call itself recursively for every node that has an entropy greater than 0. Notice that there is a need to create a new `DataFrame` for each recursive call. Each `DataFrame` will only contain the rows of the previous one that contains the value of the parent node (with the column where this value is located removed).

```python
def decision_tree(graph, entropy_results, parent, df, target_name, pos_value, neg_value):
    """ Creates a decision tree recursively based on the ID3 algorithm. This version only supports a binary output
        in the data set. It returns a new tree that is directly inserted into the graph argument.
    """
    category, entropy = find_largest(entropy_results)
    sub_tree(graph, category, entropy, parent)

    for key, value in entropy.iteritems():
        if value[0] != 0:
            new_df = df[df[category] == key]
            new_target = get_target(new_df, target_name, pos_value)
            new_data_set_entropy = set_entropy(new_target)
            new_entropy_results = entropies(new_df.drop(target_name, 1), new_target, new_data_set_entropy)
            decision_tree(graph, new_entropy_results, key, new_df, target_name, pos_value, neg_value)
        else:
            if value[1] == 1:
                graph.node(pos_value)
                graph.edge(key, pos_value)
            else:
                graph.node(neg_value)
                graph.edge(key, neg_value)
```

Calling our final function will generate the complete decision tree for the dataset.

```python
g1 = gv.Digraph(format='png')
decision_tree(g1, entropy_results, None, training_df, "Species", "Tiger", "Bull")
filename = g1.render(filename='img/g1')
Image("img/g1.png")
```

This tree is quite interesting; it appears that if an individual is scuba diving, it is likely that he could be attacked by a Tiger Shark. On the other hand, our data shows that Surfing, Fishing, and Swimming attacks are attributed more to Bull sharks than Tiger sharks.

Did you notice that the `country` feature is completely missing from the tree? This is due to _pruning_. Pruning is a technique that is employed by the ID3 algorithm, among several others, to reduce the complexity or depth of the resulting tree. In summary, while ID3 is iterating through the different features, it determines which sections of the tree provide little additional information and removes them. That is why we create the subtrees of the features with more information gain first, and as a result, we get less _overfitting_ in our predictions. For more information take a look at this article from [Wikipedia](https://en.wikipedia.org/wiki/Pruning_(decision_trees)).

## Summary of the ID3 Algorithm

Congratulations! You have a fully functional ID3 algorithm that will work on any data set given to it (with the restrictions specified above). Let's take an overall look at the algorithm:

1. Find the target entropy of the data set.
2. For every category, locate the entropy for each value it can have. For example: for `Sex`, `Feminine` and `Masculine`.
3. For every category, find the information gain using the entropy values calculated in step 2.
4. Find the largest information gain of the categories.
5. Create a sub-tree for that category (if it is the first one, it will become the root node of the tree).
6. For every node in that category:
    1. If it has an entropy of 0, point to the final decision node (positive or negative) depending on its `pos_value`.
    2. If the entropy is greater than 0:
        1. Create a new dataset that only contains the rows with the value of that node.
        2. Find the target values for the new data set.
        3. Repeat recursively from step 1.

And that's it! Thank you very much for following this tutorial.

## References

* Dr. Saed Sayad. Decision Tree - Classification. (2010-2016). _An Introduction to Data Mining_. Obtained from [source](http://www.saedsayad.com/decision_tree.htm).
* scikit-learn. Decision Trees. (2010-2016). _Machine Learning in Python_. Obtained from [source](http://scikit-learn.org/stable/modules/tree.html).
* David Ramos. Decision Trees and the Iterative Dictomizer Algorithm. (2006). Obtained from: www.cs.sjsu.edu/faculty/lee/cs157b/David_Ramos.ppt
* zel buenaobra. ID3 decision tree play baseball example. (2016). _YouTube_. Obtained from [source](https://www.youtube.com/watch?v=czPBoxmI7_I).
* Soetam Rizky. ID3 algorithm. (2011). _YouTube_. Obtained from [here](https://www.youtube.com/watch?v=wL9aogTuZw8).
* Tom Mitchell. (1997). Chapter 3, Decision Tree Learning. _Machine Learning_, McGraw Hill. Obtained from [source](http://web.arch.usyd.edu.au/~wpeng/DecisionTree2.pdf).
* Wei Peng, Juhua Chen and Haiping Zhou. An Implementation of ID3 - Decision Tree Learning Algorithm. _University of New South Wales, School of Computer Science & Engineering, Sydney, Australia_. Obtained from [source](http://web.arch.usyd.edu.au/~wpeng/DecisionTree2.pdf).
* Amro. (2009). What is "entropy and information gain"?. _Stackoverflow_. Obtained from [source](http://stackoverflow.com/questions/1859554/what-is-entropy-and-information-gain).

```python

```


# Solutions: Theorem 0.0.2 (Approximate Caratheodory's Theorem)

*These are **solutions** to the worksheet on Theorem 0.0.2. Please **DO NOT LOOK AT IT** if you haven't given the worksheet a fair amount of thought.*

In this worksheet we run through the proof of approximate Caratheodory, keeping an example to work with as we go. Here is the theorem (slightly generalized) for reference:

**Theorem 0.0.2** (Generalized)**.** *Consider a set $T \subset \mathbb{R}^n$. Then, for every point $x \in \text{conv}(T)$ and every integer $k$, one can find points $x_1,\ldots,x_k \in T$ such that*
$$ \left\lVert x - \frac{1}{k}\sum_{j=1}^k x_j \right\rVert \leq \frac{\text{diam}(T)}{\sqrt k}$$

```python
# Some useful imports
import matplotlib.pyplot as plt
import numpy as np
import random as rand
import math
```

For the purposes of the code, let us fix a set $T \subset \mathbb R^n$ and an integer $k$ to work with.

```python
# Choose a dimension to work in
n = 3

# Take some set of points with the points listed as np arrays of length n.
T = [np.array(arr) for arr in
    [[1,0,0],
    [0,1,0],
    [0,0,1],
    [0,0,0]]]

# Pick a k, any k
k = 5

```

Let $m = |T|$.

```python
m = len(T) # Set m to the size of T
```

Our task is to find some $x \in \text{conv}(T)$ and express it as a convex combination of vectors in $T$. Recalling the definition of the convex hull, this amounts to finding an assignment of coefficients $\lambda_1, \ldots, \lambda_m$, so that $\lambda_1 + \cdots + \lambda_m = 1$ and $\lambda_1,\ldots,\lambda_m \geq 0$.

Below we present one way of doing this, but it would be nice to be able to sample more uniformly from $\text{conv}(T)$. See if you can improve it!

```python
# Find random coefficients lambda_1,...,lambda_m as necessary.
# To recap, these satisfy:
# 1. lambda_1 + ... + lambda_m = 1
# 2. lambda_1, ... , lambda_m >= 0

coefficients = [0]*m # initialize list of coefficients
perm = np.random.permutation(m) # get a random permutation of 0,...,m-1
accum = 1.0
for i in perm[:-1]:
    # sample from viable range
    coefficients[i] = np.random.uniform(0,accum)
    accum = accum - coefficients[i]
coefficients[perm[-1]] = accum

coefficients
```

    [0.029193369882891777,
     0.009157147599174303,
     0.9155946928216051,
     0.04605478969632881]

We can now get $x \in \text{conv}(T)$ with $x = \lambda_1 z_1 + \cdots + \lambda_m z_m$ where $z_1, \ldots, z_m$ are the elements of $T$.

```python
# Construct x
x = np.zeros(n)
for i in range(m):
    x = x + coefficients[i] * T[i]

x
```

    array([0.02919337, 0.00915715, 0.91559469])

Now, we interpret the definition of convex combination probabilistically, with $\lambda_i$ taking the roles of probabilities. Specifically, we can define a random vector $Z$ that takes values $z_i$ with probabilities $\lambda_i$:
$$
\mathbb P \{Z = z_i\} = \lambda_i, \ \ \ i = 1, \ldots, m.
$$

```python
# We can sample from the distribution of this random variable as follows:
Z_sample = rand.choices(T, weights = coefficients)
print(Z_sample)
```

    [array([0, 0, 1])]

Consider independent copies $Z_1, \ldots, Z_k$ of $Z$. Then we are interested in the random variable
$$\left\lVert x - \frac{1}{k} \sum_{j=1}^k Z_j \right\rVert_2^2$$.

```python
# Sample from the distribution of the random variable specified above, and return
# the sampled value along with the assignments to Z_1,...,Z_k
def sample_from_distribution():
    vectors = rand.choices(T, weights = coefficients, k = k)
    vec_sum = np.zeros(len(T[0]))
    for vec in vectors:
        vec_sum = vec_sum + vec
    distance = np.sum((x - vec_sum/k)**2)
    return distance, vectors
```

In the proof of the theorem it is shown that
\begin{equation}
\mathbb E \left\lVert x - \frac{1}{k} \sum_{j=1}^k Z_j \right\rVert_2^2 \leq \frac{\text{diam}^2(T)}{k} \tag{1}
\end{equation}

Prove that this generalization to an arbitrary set is true.

We now find $\text{diam}(T)$. Recall $\text{diam}(T) = \sup_{x,y \in T} \lvert x - y \rvert $

```python
# Compute diam(T)
diamT = 0
for i in range(len(T)):
    for j in range(i+1, len(T)):
        d_xixj = np.sqrt(np.sum((T[i] - T[j])**2))
        diamT = d_xixj if (d_xixj > diamT) else diamT
```

We can now verify inequality $(1)$ for our set $T$.

```python
numSamples = 100000
samples = []

bestVal = float("inf")
best = None
for i in range(numSamples):
    sample, vectors = sample_from_distribution()
    if sample < bestVal:
        best = vectors
        bestVal = sample
    samples.append(sample)
```

```python
# plotting the example
num_bins = 10
plt.hist(samples, num_bins, facecolor='blue', alpha=0.5)
plt.axvline((diamT**2)/k, color='k', linestyle='dashed', linewidth=1.5, label = "$diam^2(T)/k$")
plt.axvline(np.mean(samples), color='r', linestyle='dashed', linewidth=1.5, label = "mean" )
plt.legend()
plt.show()
print("Best assignment (with value",bestVal, "): ")
print(best)
```

Copyright (c) 2020 TRIPODS/GradStemForAll 2020 Team


# Time Series Forecasting using the NOAA Weather Data of JFK Airport (New York)

This notebook relates to the NOAA Weather Dataset - JFK Airport (New York). The dataset contains 114,546 hourly observations of 12 local climatological variables (such as temperature and wind speed) collected at JFK airport. This dataset can be obtained for free from the IBM Developer [Data Asset Exchange](https://developer.ibm.com/exchanges/data/all/jfk-weather-data/).

In this notebook we explore approaches to predicting future temperatures by using the time-series dataset.

### Table of Contents:
* [1. Read the Cleaned Data](#cell1)
* [2. Explore Baseline Models](#cell2)
* [3. Train Statistical Time-series Analysis Models](#cell3)
* [Authors](#cell4)

#### Import required modules

Import and configure the required modules.

```python
!pip install statsmodels
!pip install sklearn
!pip install matplotlib
```

    Requirement already satisfied: statsmodels in /opt/conda/lib/python3.7/site-packages (0.11.1)
    Requirement already satisfied: patsy>=0.5 in /opt/conda/lib/python3.7/site-packages (from statsmodels) (0.5.1)
    Requirement already satisfied: pandas>=0.21 in /opt/conda/lib/python3.7/site-packages (from statsmodels) (1.1.2)
    Requirement already satisfied: scipy>=1.0 in /opt/conda/lib/python3.7/site-packages (from statsmodels) (1.5.2)
    Requirement already satisfied: numpy>=1.14 in /opt/conda/lib/python3.7/site-packages (from statsmodels) (1.19.2)
    Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from patsy>=0.5->statsmodels) (1.15.0)
    Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from pandas>=0.21->statsmodels) (2.8.1)
    Requirement already satisfied: pytz>=2017.2 in /opt/conda/lib/python3.7/site-packages (from pandas>=0.21->statsmodels) (2020.1)
    Processing /home/jovyan/.cache/pip/wheels/46/ef/c3/157e41f5ee1372d1be90b09f74f82b10e391eaacca8f22d33e/sklearn-0.0-py2.py3-none-any.whl
    Requirement already satisfied: scikit-learn in /opt/conda/lib/python3.7/site-packages (from sklearn) (0.23.2)
    Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from scikit-learn->sklearn) (2.1.0)
    Requirement already satisfied: joblib>=0.11 in /opt/conda/lib/python3.7/site-packages (from scikit-learn->sklearn) (0.16.0)
    Requirement already satisfied: scipy>=0.19.1 in /opt/conda/lib/python3.7/site-packages (from scikit-learn->sklearn) (1.5.2)
    Requirement already satisfied: numpy>=1.13.3 in /opt/conda/lib/python3.7/site-packages (from scikit-learn->sklearn) (1.19.2)
    Installing collected packages: sklearn
    Successfully installed sklearn-0.0
    Requirement already satisfied: matplotlib in /opt/conda/lib/python3.7/site-packages (3.2.2)
    Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib) (1.2.0)
    Requirement already satisfied: python-dateutil>=2.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib) (2.8.1)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib) (2.4.7)
    Requirement already satisfied: numpy>=1.11 in /opt/conda/lib/python3.7/site-packages (from matplotlib) (1.19.2)
    Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.7/site-packages (from matplotlib) (0.10.0)
    Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.7/site-packages (from python-dateutil>=2.1->matplotlib) (1.15.0)

```python
import numpy as np
import pandas as pd
from sklearn.metrics import mean_squared_error
from statsmodels.tsa.statespace.sarimax import SARIMAX
from matplotlib import pyplot as plt
```

### 1. Read the Cleaned Data

We start by reading the cleaned dataset that was created in notebook `Part 1 - Data Cleaning`.

**Note:** if you haven't yet run this notebook, run it first; otherwise the cells below will not work.

```python
data = pd.read_csv('data/noaa-weather-data-jfk-airport/jfk_weather_cleaned.csv', parse_dates=['DATE'])
```

```python
# Set date index
data = data.set_index(pd.DatetimeIndex(data['DATE']))
data.drop(['DATE'], axis=1, inplace=True)
data.head()
```

      visibility
      dry_bulb_temp_f
      wet_bulb_temp_f
      dew_point_temp_f
      relative_humidity
      wind_speed
      station_pressure
      sea_level_pressure
      precip
      altimeter_setting
      wind_direction_sin
      wind_direction_cos
      pressure_tendency_incr
      pressure_tendency_decr
      pressure_tendency_const

      DATE

      2010-01-01 01:00:00
      6.0
      33.0
      32.0
      31.0
      92.0
      0.0
      29.97
      29.99
      0.01
      29.99
      0.0
      1.0
      0
      1
      0

      2010-01-01 02:00:00
      6.0
      33.0
      33.0
      32.0
      96.0
      0.0
      29.97
      29.99
      0.02
      29.99
      0.0
      1.0
      0
      1
      0

      2010-01-01 03:00:00
      5.0
      33.0
      33.0
      32.0
      96.0
      0.0
      29.97
      29.99
      0.00
      29.99
      0.0
      1.0
      0
      1
      0

      2010-01-01 04:00:00
      5.0
      33.0
      33.0
      32.0
      96.0
      0.0
      29.95
      29.97
      0.00
      29.97
      0.0
      1.0
      0
      1
      0

      2010-01-01 05:00:00
      5.0
      33.0
      32.0
      31.0
      92.0
      0.0
      29.93
      29.96
      0.00
      29.95
      0.0
      1.0
      0
      1
      0

For purposes of time-series modeling, we will restrict our analysis to a 2-year sample of the dataset to avoid overly long model-training times.

```python
sample = data['2016-01-01':'2018-01-01']
sample.info()
```

    <class 'pandas.core.frame.DataFrame'>
    DatetimeIndex: 17568 entries, 2016-01-01 00:00:00 to 2018-01-01 23:00:00
    Data columns (total 15 columns):
     #   Column                   Non-Null Count  Dtype
    ---  ------                   --------------  -----
     0   visibility               17568 non-null  float64
     1   dry_bulb_temp_f          17568 non-null  float64
     2   wet_bulb_temp_f          17568 non-null  float64
     3   dew_point_temp_f         17568 non-null  float64
     4   relative_humidity        17568 non-null  float64
     5   wind_speed               17568 non-null  float64
     6   station_pressure         17568 non-null  float64
     7   sea_level_pressure       17568 non-null  float64
     8   precip                   17568 non-null  float64
     9   altimeter_setting        17568 non-null  float64
     10  wind_direction_sin       17568 non-null  float64
     11  wind_direction_cos       17568 non-null  float64
     12  pressure_tendency_incr   17568 non-null  int64
     13  pressure_tendency_decr   17568 non-null  int64
     14  pressure_tendency_const  17568 non-null  int64
    dtypes: float64(12), int64(3)
    memory usage: 2.1 MB

#### Create Training/Validation/Test Splits

Before we attempt any time-series analysis and prediction, we should split the dataset into training, validation and test sets. We use a portion of the data for training, and a portion of _future_ data for our validation and test sets.

If we instead trained a model on the full dataset, the model would learn to be very good at making predictions on that particular dataset, essentially just copying the answers it knows. However, when presented with data the model _has not seen_ , it would perform poorly since it has not learned how to generalize its answers.

By training on a portion of the dataset and testing the model's performance on another portion of the dataset (which data the model has not seen in training), we try to avoid our models "over-fitting" the dataset and make them better at predicting temperatures given unseen, future data. This process of splitting the dataset and evaluating a model's performance on the validation and test sets is commonly known as [cross-validation](https://en.wikipedia.org/wiki/Cross-validation_(statistics)).

By default here we use 80% of the data for the training set and 10% each for validation and test sets.

```python
def split_data(data, val_size=0.1, test_size=0.1):
    """
    Splits data to training, validation and testing parts
    """
    ntest = int(round(len(data) * (1 - test_size)))
    nval = int(round(len(data) * (1 - test_size - val_size)))

    df_train, df_val, df_test = data.iloc[:nval], data.iloc[nval:ntest], data.iloc[ntest:]

    return df_train, df_val, df_test

# Create data split
df_train, df_val, df_test = split_data(sample)

print('Total data size:      {} rows'.format(len(sample)))
print('Training set size:    {} rows'.format(len(df_train)))
print('Validation set size:  {} rows'.format(len(df_val)))
print('Test set size:        {} rows'.format(len(df_test)))
```

    Total data size:      17568 rows
    Training set size:    14054 rows
    Validation set size:  1757 rows
    Test set size:        1757 rows

### 2. Explore Baseline Models

In this section, we'll create a few simple predictive models of temperature, using shifting and rolling averages. These will serve as a baseline against which we can compare more sophisticated models.

Using values at recent timesteps (such as the most recent timestep `t-1` and second-most recent timestep `t-2`) to predict the current value at time `t` is what's known as persistence modeling, or using the last observed value to predict the next following value. These preceding timesteps are often referred to in time-series analysis as `lags`. So, the value at time `t-1` is known as the `1st lag` and the value at time `t-2` is the `2nd lag`.

We can also create baselines based on rolling (or moving) averages. This is a time-series constructed by averaging each lagged value up to the selected lag. For example, a 6-period (or 6-lag) rolling avearge is the average of the previous 6 hourly lags `t-6` to `t-1`.

Our baseline models will be:
1. `1st lag` - i.e. values at `t-1`
2. `2nd lag` - i.e. values at `t-2`
3. `6-lag` rolling average
4. `12-lag` rolling average

```python
# define the column containing the data we wish to model - in this case Dry Bulb Temperature (F)
Y_COL = 'dry_bulb_temp_f'

# Use shifting and rolling averages to predict Y_COL (t)
n_in = 2
n_out = 1
features = [Y_COL]
n_features = len(features)

# create the baseline on the entire sample dataset.
# we will evaluate the prediction error on the validation set
baseline = sample[[Y_COL]].loc[:]
baseline['{} (t-1)'.format(Y_COL)] = baseline[Y_COL].shift(1)
baseline['{} (t-2)'.format(Y_COL)] = baseline[Y_COL].shift(2)
baseline['{} (6hr rollavg)'.format(Y_COL)] = baseline[Y_COL].rolling('6H').mean()
baseline['{} (12hr rollavg)'.format(Y_COL)] = baseline[Y_COL].rolling('12H').mean()
baseline.dropna(inplace=True)
baseline.head(10)
```

      dry_bulb_temp_f
      dry_bulb_temp_f (t-1)
      dry_bulb_temp_f (t-2)
      dry_bulb_temp_f (6hr rollavg)
      dry_bulb_temp_f (12hr rollavg)

      DATE

      2016-01-01 02:00:00
      43.0
      43.0
      44.0
      43.333333
      43.333333

      2016-01-01 03:00:00
      42.0
      43.0
      43.0
      43.000000
      43.000000

      2016-01-01 04:00:00
      43.0
      42.0
      43.0
      43.000000
      43.000000

      2016-01-01 05:00:00
      42.0
      43.0
      42.0
      42.833333
      42.833333

      2016-01-01 06:00:00
      41.0
      42.0
      43.0
      42.333333
      42.571429

      2016-01-01 07:00:00
      41.0
      41.0
      42.0
      42.000000
      42.375000

      2016-01-01 08:00:00
      41.0
      41.0
      41.0
      41.666667
      42.222222

      2016-01-01 09:00:00
      40.0
      41.0
      41.0
      41.333333
      42.000000

      2016-01-01 10:00:00
      41.0
      40.0
      41.0
      41.000000
      41.909091

      2016-01-01 11:00:00
      41.0
      41.0
      40.0
      40.833333
      41.833333

Next, we will plot data from our validation dataset to get a sense for how well these baseline models predict the next hourly temperature. Note thatd we only use a few days of data in order to make the plot easier to view.

```python
# plot first 7 days of the validation set, 168 hours
start = df_val.index[0]
end = df_val.index[167]
sliced = baseline[start:end]
```

```python
# Plot baseline predictions sample
cols = ['dry_bulb_temp_f', 'dry_bulb_temp_f (t-1)', 'dry_bulb_temp_f (t-2)', 'dry_bulb_temp_f (6hr rollavg)', 'dry_bulb_temp_f (12hr rollavg)']
sliced[cols].plot()

plt.legend(['t', 't-1', 't-2', '6hr', '12hr'], loc=2, ncol=3)
plt.title('Baselines for First 7 Days of Validation Set')
plt.ylabel('Temperature (F)')
plt.tight_layout()
plt.rcParams['figure.dpi'] = 100
plt.show()
```

#### Evaluate baseline models

As you can perhaps see from the graph above, the _lagged_ baselines appear to do a better job of forecasting temperatures than the _rolling average_ baselines.

In order to evaluate our baseline models more precisely, we need to answer the question _"how well do our models predict future temperature?"_. In regression problems involving prediction of a numerical value, we often use a measure of the difference between our predicted value and the actual value. This is referred to as an error measure or error metric. A common measure is the Mean Squared Error (MSE):

\begin{equation}
MSE = \frac{1}{n} \sum_{i=1}^{n}{(y_i - \hat y_i)^{2}}
\end{equation}

This is the average of the squared differences between predicted values $ \hat y $ and actual values $ y $.

Because the MSE is in "units squared" it can be difficult to interpet, hence the Root Mean Squared Error (RMSE) is often used:

\begin{equation}
RMSE = \sqrt {MSE}
\end{equation}

This is the square root of the MSE, and is in the same units as the values $ y $. We can compare the RMSE (and MSE) values for different models and say that the model that has the lower MSE is better at predicting temperatures, all things equal. Note that MSE and RMSE will grow large quickly if the differences between predicted and actual values are large. This may or may not be a desired quality of your error measure. In this case, it is probably a good thing, since a model that makes large mistakes in temperature prediction will be much less useful than one which makes small mistakes.

Next, we calculate the RMSE measure for each of our baseline models, on the full validation set.

```python
# Calculating baseline RMSE
start_val = df_val.index[0]
end_val = df_val.index[-1]
baseline_val = baseline[start_val:end_val]

baseline_y = baseline_val[Y_COL]
baseline_t1 = baseline_val['dry_bulb_temp_f (t-1)']
baseline_t2 = baseline_val['dry_bulb_temp_f (t-2)']
baseline_avg6 = baseline_val['dry_bulb_temp_f (6hr rollavg)']
baseline_avg12 = baseline_val['dry_bulb_temp_f (12hr rollavg)']

rmse_t1 = round(np.sqrt(mean_squared_error(baseline_y, baseline_t1)), 2)
rmse_t2 = round(np.sqrt(mean_squared_error(baseline_y, baseline_t2)), 2)
rmse_avg6 = round(np.sqrt(mean_squared_error(baseline_y, baseline_avg6)), 2)
rmse_avg12 = round(np.sqrt(mean_squared_error(baseline_y, baseline_avg12)), 2)

print('Baseline t-1 RMSE:            {0:.3f}'.format(rmse_t1))
print('Baseline t-2 RMSE:            {0:.3f}'.format(rmse_t2))
print('Baseline 6hr rollavg RMSE:    {0:.3f}'.format(rmse_avg6))
print('Baseline 12hr rollavg RMSE:   {0:.3f}'.format(rmse_avg12))
```

    Baseline t-1 RMSE:            1.690
    Baseline t-2 RMSE:            2.880
    Baseline 6hr rollavg RMSE:    3.080
    Baseline 12hr rollavg RMSE:   5.010

The RMSE results confirm what we saw in the graph above. It is clear that the _rolling average_ baselines perform poorly. In fact, the `t-2` lagged baseline is also not very good. It appears that the best baseline model is to simply use the current hour's temperature to predict the next hour's temperature!

Can we do better than this simple baseline using more sophisticated models?

### 2. Train Statistical Time-series Analysis Models

In the previous section, we saw that a simple `lag-1` baseline model performed reasonably well at forecasting temperature for the next hourly time step. This is perhaps not too surprising, given what we know about hourly temperatures. Generally, the temperature in a given hour will be quite closely related to the temperature in the previous hour. This phenomenon is very common in time-series analysis and is known as [autocorrelation](https://en.wikipedia.org/wiki/Autocorrelation) - that is, the time series is `correlated` with previous values of itself. More precisely, the values at time `t` are correlated with lagged values (which could be `t-1`, `t-2` and so on).

Another thing we saw previously is the concept of _moving averages_. In this case the moving-average baseline was not that good at prediction. However it is common in many time-series for a moving average to capture some of the underlying structure and be useful for prediction.

In order to make our model better at predicting temperature, ideally we would want to take these aspects into account. Fortunately, the statistical community has a long history of analyzing time series and has created many different forecasting models.

Here, we will explore one called SARIMAX - the **S**easonal **A**uto**R**egressive **I**ntegrated **M**oving **A**verage with e**X**ogenous regressors model.

This sounds like a very complex name, but if we look at the components of the name, we see that it includes `autocorrelation` (this is what auto regressive means) and `moving averages`, which are the components mentioned above.

The SARIMAX model also allows including a _seasonal_ model component as well as handling *exogenous* variables, which are external to the time-series value itself. For example, for temperature prediction we may wish to take into account not just previous temperature values, but perhaps other weather features which may have an effect on temperature (such as humidity, rainfall, wind, and so on).

For the purposes of this notebook, we will not explore modeling of seasonal components or exogenous variables.

If we drop the "S" and "X" from the model, we are left with an [ARIMA model](https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average) (Auto-regressive Integrated Moving Average). This is a very commonly used model for time-series analysis and we will use it in this notebook by only specifying the relevant model components of the full SARIMAX model.

#### 2.1 Replicating a baseline model

As a starting point, we will see how we can use SARIMAX to create a simple model that in fact replicates one of the baselines we created previously. Auto-regression, as we have seen, means using values from preceding time periods to predict the current value. Recall that one of our baseline models was the `1st lag` or `t-1` model. In time-series analysis this is referred to as an **AR(1)** model, meaning an **A**uto-**R**egressive model for `lag 1`.

Technically, the AR(1) model is not exactly the same as our baseline model. A statistical time series model like SARIMAX learns a set of `weights` to apply to each component of the model. These weights are set so as to best fit the dataset. We can think of our baseline as setting the `weight` for the `t-1` lag to be exactly `1`. In practice, our time-series model will not have a weight of exactly `1` (though it will likely be very close to that), hence the predictions will be slightly different.

Now, lets fit our model to the dataset. First, we will set up the model inputs by taking the temperature column of our dataframe. We do this for training and validation sets.

```python
X_train = df_train[Y_COL]
X_val = df_val[Y_COL]
X_both = np.hstack((X_train, X_val))
```

Here we created a variable called `X_both` to cover both the training and validation data. This is required later when we forecast values for our SARIMAX model, in order to give the model access to all the datapoints for which it must create forecasts. Note that the forecasts themselves will only be based on the _model weights_ learned from the training data (this is important for over-fitting as we have seen above)!

The SARIMAX model takes an argument called `order`: this specifies the components of the model and itself has 3 parts: `(p, d, q)`. `p` denotes the lags for the AR model and `q` denotes the lags for the MA model. We will not cover the `d` parameter here. Taken together this specifies the parameters of the [ARIMA](https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average) model portion of SARIMAX.

To create an AR(1) model, we set the `order` to be `(1, 0, 0)`. This sets up the AR model to be a `lag 1` model. Then, we fit our model on the training data and inspect a summary of the trained model.

```python
order = (1, 0, 0)
model_ar1 = SARIMAX(X_train, order=order)
results_ar1 = model_ar1.fit()
results_ar1.summary()
```

    /opt/conda/lib/python3.7/site-packages/statsmodels/tsa/base/tsa_model.py:162: ValueWarning: No frequency information was provided, so inferred frequency H will be used.
      % freq, ValueWarning)
    /opt/conda/lib/python3.7/site-packages/statsmodels/tsa/base/tsa_model.py:162: ValueWarning: No frequency information was provided, so inferred frequency H will be used.
      % freq, ValueWarning)

SARIMAX Results

  Dep. Variable:	    dry_bulb_temp_f	   No. Observations:  	    14054

  Model:	           SARIMAX(1, 0, 0)	   Log Likelihood     	 -27650.431

  Date:	            Mon, 28 Sep 2020	   AIC                	  55304.863

  Time:	                05:17:59	       BIC                	  55319.964

  Sample:	             01-01-2016	      HQIC               	  55309.889

  	                   - 08-08-2017

  Covariance Type:	        opg

     	       coef	     std err	      z	      P>|z|	  [0.025	    0.975]

  ar.L1	      0.9996	     0.000	  4277.653	  0.000	     0.999	     1.000

  sigma2	     2.9937	     0.024	   125.518	  0.000	     2.947	     3.040

  Ljung-Box (Q):	          16360.93	   Jarque-Bera (JB):  	 3656.68

  Prob(Q):	                  0.00	     Prob(JB):          	  0.00

  Heteroskedasticity (H):	   0.97	     Skew:              	  0.15

  Prob(H) (two-sided):	      0.26	     Kurtosis:          	  5.48

Warnings:
[1] Covariance matrix calculated using the outer product of gradients (complex-step).

There's quite a lot of information printed out in the model summary above. Much of it is related to the statistical properties of our model.

The most important thing for now is to look at the second table, where we can see a `coef` value of `0.9996` for the weight `ar.L1`. This tells us the model has set a weight for the `1st lag` component of the AR model to be `0.9996`. This is almost `1` and hence we should expect the prediction results to indeed be close to our `t-1` baseline.

Let's create our model forecast on the validation dataset. We will then plot a few data points like we did with our baseline models (using 7 days of validation data) and compute the RMSE value based on the full validation set.

```python
full_data_ar1 = SARIMAX(X_both, order=order)
model_forecast_ar1 = full_data_ar1.filter(results_ar1.params)
```

```python
start = len(X_train)
end = len(X_both)
forecast_ar1 = model_forecast_ar1.predict(start=start, end=end - 1, dynamic=False)

# plot actual vs predicted values for the same 7-day window for easier viewing
plt.plot(sliced[Y_COL].values)
plt.plot(forecast_ar1[:168], color='r', linestyle='--')
plt.legend(['t', 'AR(1)'], loc=2)
plt.title('AR(1) Model Predictions for First 7 Days of Validation Set')
plt.ylabel('Temperature (F)')
plt.tight_layout()
plt.show()
```

We can see that the plot looks almost identical to the plot above, for the `t` and `t-1 baseline` values.

Next, we compute the RMSE values.

```python
# compute print RMSE values
rmse_ar1 = np.sqrt(mean_squared_error(baseline_val[Y_COL], forecast_ar1))
print('AR(1) RMSE:                   {0:.3f}'.format(rmse_ar1))
print('Baseline t-1 RMSE:            {0:.3f}'.format(rmse_t1))
```

    AR(1) RMSE:                   1.692
    Baseline t-1 RMSE:            1.690

We can see that the RMSE values for the validation set also almost identical.

#### 2.2 Create a more complex model

One of our baseline models was a `lag 2` model, i.e. `t-2`. We saw that it performed a lot worse than the `t-1` baseline. Intuitively, this makes sense, since we are throwing away a lot of information about the most recent lag `t-1`. However, the `t-2` lag still provides some useful information. In fact, for temperature prediction it's likely that the last few hours can provide some value.

Fortunately, our ARIMA model framework provides an easy way to incorporate further lag information. We can construct a model that includes _both_ the `t-1` and `t-2` lags. This is an **AR(2)** model (meaning an auto-regressive model up to lag `2`). We can specify this with the model order parameter `p=2`.

```python
order = (2, 0, 0)
model_ar2 = SARIMAX(X_train, order=order)
results_ar2 = model_ar2.fit()
results_ar2.summary()
```

    /opt/conda/lib/python3.7/site-packages/statsmodels/tsa/base/tsa_model.py:162: ValueWarning: No frequency information was provided, so inferred frequency H will be used.
      % freq, ValueWarning)
    /opt/conda/lib/python3.7/site-packages/statsmodels/tsa/base/tsa_model.py:162: ValueWarning: No frequency information was provided, so inferred frequency H will be used.
      % freq, ValueWarning)

SARIMAX Results

  Dep. Variable:	    dry_bulb_temp_f	   No. Observations:  	    14054

  Model:	           SARIMAX(2, 0, 0)	   Log Likelihood     	 -26867.232

  Date:	            Mon, 28 Sep 2020	   AIC                	  53740.465

  Time:	                05:18:01	       BIC                	  53763.117

  Sample:	             01-01-2016	      HQIC               	  53748.005

  	                   - 08-08-2017

  Covariance Type:	        opg

     	       coef	     std err	      z	      P>|z|	  [0.025	    0.975]

  ar.L1	      1.3242	     0.006	   220.175	  0.000	     1.312	     1.336

  ar.L2	     -0.3248	     0.006	   -53.852	  0.000	    -0.337	    -0.313

  sigma2	     2.6779	     0.020	   136.374	  0.000	     2.639	     2.716

  Ljung-Box (Q):	          4350.76	   Jarque-Bera (JB):  	 7579.33

  Prob(Q):	                 0.00	     Prob(JB):          	  0.00

  Heteroskedasticity (H):	  0.99	     Skew:              	  -0.01

  Prob(H) (two-sided):	     0.75	     Kurtosis:          	  6.60

Warnings:
[1] Covariance matrix calculated using the outer product of gradients (complex-step).

This time, the results table indicates a weight for variable `ar.L1` _and_ `ar.L2`. Note the values are now quite different from `1` (or `0.5` say, for a simple equally-weighted model). Next, we compute the RMSE on the validation set.

```python
full_data_ar2 = SARIMAX(X_both, order=order)
model_forecast_ar2 = full_data_ar2.filter(results_ar2.params)

start = len(X_train)
end = len(X_both)
forecast_ar2 = model_forecast_ar2.predict(start=start, end=end - 1, dynamic=False)

# compute print RMSE values
rmse_ar2 = np.sqrt(mean_squared_error(baseline_val[Y_COL], forecast_ar2))
print('AR(2) RMSE:                   {0:.3f}'.format(rmse_ar2))
print('AR(1) RMSE:                   {0:.3f}'.format(rmse_ar1))
print('Baseline t-1 RMSE:            {0:.3f}'.format(rmse_t1))
```

    AR(2) RMSE:                   1.526
    AR(1) RMSE:                   1.692
    Baseline t-1 RMSE:            1.690

We've improved the RMSE value by including information from the first two lags.

In fact, you will see that if you continue to increase the `p` parameter value, the RMSE will continue to decrease, indicating that a few recent lags provide useful information to our model.

#### 2.3 Incorporate moving averages

Finally, what if we also include moving average information in our model? The ARIMA framework makes this easy to do, by setting the order parameter `q`. A value of `q=1` specifies a **MA(1)** model (including the first lag `t-1`), while `q=6` would include all the lags from `t-1` to `t-6`.

Note that the moving average model component is a little different from the simple moving or rolling averages computed in the baseline models. The [definition of the MA model](https://en.wikipedia.org/wiki/Moving-average_model) is rather technical, but conceptually you can think of it as using a form of weighted moving average (compared to our baseline which would be a simple, unweighted average).

Let's add an MA(1) component to our AR(2) model.

```python
order = (2, 0, 1)
model_ar2ma1 = SARIMAX(X_train, order=order)
results_ar2ma1 = model_ar2ma1.fit()
results_ar2ma1.summary()
```

    /opt/conda/lib/python3.7/site-packages/statsmodels/tsa/base/tsa_model.py:162: ValueWarning: No frequency information was provided, so inferred frequency H will be used.
      % freq, ValueWarning)
    /opt/conda/lib/python3.7/site-packages/statsmodels/tsa/base/tsa_model.py:162: ValueWarning: No frequency information was provided, so inferred frequency H will be used.
      % freq, ValueWarning)

SARIMAX Results

  Dep. Variable:	    dry_bulb_temp_f	   No. Observations:  	    14054

  Model:	           SARIMAX(2, 0, 1)	   Log Likelihood     	 -26667.060

  Date:	            Mon, 28 Sep 2020	   AIC                	  53342.120

  Time:	                05:18:04	       BIC                	  53372.322

  Sample:	             01-01-2016	      HQIC               	  53352.172

  	                   - 08-08-2017

  Covariance Type:	        opg

     	       coef	     std err	      z	      P>|z|	  [0.025	    0.975]

  ar.L1	      1.6697	     0.015	   114.448	  0.000	     1.641	     1.698

  ar.L2	     -0.6700	     0.015	   -45.898	  0.000	    -0.699	    -0.641

  ma.L1	     -0.3825	     0.016	   -23.348	  0.000	    -0.415	    -0.350

  sigma2	     2.6026	     0.019	   139.868	  0.000	     2.566	     2.639

  Ljung-Box (Q):	          2762.35	   Jarque-Bera (JB):  	 8834.61

  Prob(Q):	                 0.00	     Prob(JB):          	  0.00

  Heteroskedasticity (H):	  1.00	     Skew:              	  -0.04

  Prob(H) (two-sided):	     0.92	     Kurtosis:          	  6.88

Warnings:
[1] Covariance matrix calculated using the outer product of gradients (complex-step).

We see the results table shows an additional weight value for `ma.L1`, our MA(1) component. Next, we compare the RMSE to the other models and finally plot all the model forecasts together - _note_ we use a much smaller 48-hour window to make the plot readable for illustrative purposes.

```python
full_data_ar2ma1 = SARIMAX(X_both, order=order)
model_forecast_ar2ma1 = full_data_ar2ma1.filter(results_ar2ma1.params)

start = len(X_train)
end = len(X_both)
forecast_ar2ma1 = model_forecast_ar2ma1.predict(start=start, end=end - 1, dynamic=False)

# compute print RMSE values
rmse_ar2ma1 = np.sqrt(mean_squared_error(baseline_val[Y_COL], forecast_ar2ma1))
print('AR(2) MA(1) RMSE:             {0:.3f}'.format(rmse_ar2ma1))
print('AR(2) RMSE:                   {0:.3f}'.format(rmse_ar2))
print('AR(1) RMSE:                   {0:.3f}'.format(rmse_ar1))
print('Baseline t-1 RMSE:            {0:.3f}'.format(rmse_t1))
```

    AR(2) MA(1) RMSE:             1.491
    AR(2) RMSE:                   1.526
    AR(1) RMSE:                   1.692
    Baseline t-1 RMSE:            1.690

```python
# plot actual vs predicted values for a smaller 2-day window for easier viewing
hrs = 48
plt.plot(sliced[Y_COL][:hrs].values)
plt.plot(forecast_ar1[:hrs], color='r', linestyle='--')
plt.plot(forecast_ar2[:hrs], color='g', linestyle='--')
plt.plot(forecast_ar2ma1[:hrs], color='c', linestyle='--')
plt.legend(['t', 'AR(1)', 'AR(2)', 'AR(2) MA(1)'], loc=2, ncol=1)
plt.title('ARIMA Model Predictions for First 48 hours of Validation Set')
plt.ylabel('Temperature (F)')
plt.tight_layout()
plt.show()
```

We've again managed to reduce the RMSE value for our model, indicating that adding the MA(1) component has improved our forecast!

Congratulations! You've applied the basics of time-series analysis for forecasting hourly temperatures. See if you can further improve the RMSE values by exploring the different values for the model parameters `p`, `q` and even `d`!

### Authors

This notebook was created by the [Center for Open-Source Data & AI Technologies](http://codait.org).

Copyright © 2019 IBM. This notebook and its source code are released under the terms of the MIT License.

pipelines/hello_world/Part 3 - Time Series Forecasting.ipynb


$\newcommand{\vct}[1]{\boldsymbol{#1}}
\newcommand{\mtx}[1]{\mathbf{#1}}
\newcommand{\tr}{^\mathrm{T}}
\newcommand{\reals}{\mathbb{R}}
\newcommand{\lpa}{\left(}
\newcommand{\rpa}{\right)}
\newcommand{\lsb}{\left[}
\newcommand{\rsb}{\right]}
\newcommand{\lbr}{\left\lbrace}
\newcommand{\rbr}{\right\rbrace}
\newcommand{\fset}[1]{\lbr #1 \rbr}
\newcommand{\pd}[2]{\frac{\partial #1}{\partial #2}}$

# Single layer models

In this lab we will implement a single-layer network model consisting of solely of an affine transformation of the inputs. The relevant material for this was covered in [the slides of the first lecture](http://www.inf.ed.ac.uk/teaching/courses/mlp/2016/mlp01-intro.pdf).

We will first implement the forward propagation of inputs to the network to produce predicted outputs. We will then move on to considering how to use gradients of an error function evaluated on the outputs to compute the gradients with respect to the model parameters to allow us to perform an iterative gradient-descent training procedure. In the final exercise you will use an interactive visualisation to explore the role of some of the different hyperparameters of gradient-descent based training methods.

#### A note on random number generators

It is generally a good practice (for machine learning applications **not** for cryptography!) to seed a pseudo-random number generator once at the beginning of each experiment. This makes it easier to reproduce results as the same random draws will produced each time the experiment is run (e.g. the same random initialisations used for parameters). Therefore generally when we need to generate random values during this course, we will create a seeded random number generator object as we do in the cell below.

```python
import numpy as np
seed = 27092016
rng = np.random.RandomState(seed)
```

## Exercise 1: linear and affine transforms

Any *linear transform* (also called a linear map) of a finite-dimensional vector space can be parametrised by a matrix. So for example if we consider $\vct{x} \in \reals^{D}$ as the input space of a model with $D$ dimensional real-valued inputs, then a matrix $\mtx{W} \in \reals^{K\times D}$ can be used to define a prediction model consisting solely of a linear transform of the inputs

\begin{equation}
    \vct{y} = \mtx{W} \vct{x}
    \qquad
    \Leftrightarrow
    \qquad
    y_k = \sum_{d=1}^D \lpa W_{kd} x_d \rpa \quad \forall k \in \fset{1 \dots K}
\end{equation}

with here $\vct{y} \in \reals^K$ the $K$-dimensional real-valued output of the model. Geometrically we can think of a linear transform doing some combination of rotation, scaling, reflection and shearing of the input.

An *affine transform* consists of a linear transform plus an additional translation parameterised by a vector $\vct{b} \in \reals^K$. A model consisting of an affine transformation of the inputs can then be defined as

\begin{equation}
    \vct{y} = \mtx{W}\vct{x} + \vct{b}
    \qquad
    \Leftrightarrow
    \qquad
    y_k = \sum_{d=1}^D \lpa W_{kd} x_d \rpa + b_k \quad \forall k \in \fset{1 \dots K}
\end{equation}

In machine learning we will usually refer to the matrix $\mtx{W}$ as a *weight matrix* and the vector $\vct{b}$ as a *bias vector*.

Generally rather than working with a single data vector $\vct{x}$ we will work with batches of datapoints $\fset{\vct{x}^{(b)}}_{b=1}^B$. We could calculate the outputs for each input in the batch sequentially

\begin{align}
    \vct{y}^{(1)} &= \mtx{W}\vct{x}^{(1)} + \vct{b}\\
    \vct{y}^{(2)} &= \mtx{W}\vct{x}^{(2)} + \vct{b}\\
    \dots &\\
    \vct{y}^{(B)} &= \mtx{W}\vct{x}^{(B)} + \vct{b}\\
\end{align}

by looping over each input in the batch and calculating the output. However in general loops in Python are slow (particularly compared to compiled and typed languages such as C). This is due at least in part to the large overhead in dynamically inferring variable types. In general therefore wherever possible we want to avoid having loops in which such overhead will become the dominant computational cost.

For array based numerical operations, one way of overcoming this bottleneck is to *vectorise* operations. NumPy `ndarrays` are typed arrays for which operations such as basic elementwise arithmetic and linear algebra operations such as computing matrix-matrix or matrix-vector products are implemented by calls to highly-optimised compiled libraries. Therefore if you can implement code directly using NumPy operations on arrays rather than by looping over array elements it is often possible to make very substantial performance gains.

As a simple example we can consider adding up two arrays `a` and `b` and writing the result to a third array `c`. First lets initialise `a` and `b` with arbitrary values by running the cell below.

```python
size = 1000
a = np.arange(size)
b = np.ones(size)
```

Now let's time how long it takes to add up each pair of values in the two array and write the results to a third array using a loop-based implementation. We will use the `%%timeit` magic briefly mentioned in the previous lab notebook specifying the number of times to loop the code as 100 and to give the best of 3 repeats. Run the cell below to get a print out of the average time taken.

```python
%%timeit -n 100 -r 3
c = np.empty(size)
for i in range(size):
    c[i] = a[i] + b[i]
```

    100 loops, best of 3: 2.18 ms per loop

And now we will perform the corresponding summation with the overloaded addition operator of NumPy arrays. Again run the cell below to get a print out of the average time taken.

```python
%%timeit -n 100 -r 3
c = a + b
```

    The slowest run took 4.89 times longer than the fastest. This could mean that an intermediate result is being cached.
    100 loops, best of 3: 3.42 µs per loop

The first loop-based implementation should have taken on the order of milliseconds ($10^{-3}$s) while the vectorised implementation should have taken on the order of microseconds ($10^{-6}$s), i.e. a $\sim1000\times$ speedup. Hopefully this simple example should make it clear why we want to vectorise operations whenever possible!

Getting back to our affine model, ideally rather than individually computing the output corresponding to each input we should compute the outputs for all inputs in a batch using a vectorised implementation. As you saw last week, data providers return batches of inputs as arrays of shape `(batch_size, input_dim)`. In the mathematical notation used earlier we can consider this as a matrix $\mtx{X}$ of dimensionality $B \times D$, and in particular

\begin{equation}
    \mtx{X} = \lsb \vct{x}^{(1)} ~ \vct{x}^{(2)} ~ \dots ~ \vct{x}^{(B)} \rsb\tr
\end{equation}

i.e. the $b^{\textrm{th}}$ input vector $\vct{x}^{(b)}$ corresponds to the $b^{\textrm{th}}$ row of $\mtx{X}$. If we define the $B \times K$ matrix of outputs $\mtx{Y}$ similarly as

\begin{equation}
    \mtx{Y} = \lsb \vct{y}^{(1)} ~ \vct{y}^{(2)} ~ \dots ~ \vct{y}^{(B)} \rsb\tr
\end{equation}

then we can express the relationship between $\mtx{X}$ and $\mtx{Y}$ using [matrix multiplication](https://en.wikipedia.org/wiki/Matrix_multiplication) and addition as

\begin{equation}
    \mtx{Y} = \mtx{X} \mtx{W}\tr + \mtx{B}
\end{equation}

where $\mtx{B} = \lsb \vct{b} ~ \vct{b} ~ \dots ~ \vct{b} \rsb\tr$ i.e. a $B \times K$ matrix with each row corresponding to the bias vector. The weight matrix needs to be transposed here as the inner dimensions of a matrix multiplication must match i.e. for $\mtx{C} = \mtx{A} \mtx{B}$ then if $\mtx{A}$ is of dimensionality $K \times L$ and $\mtx{B}$ is of dimensionality $M \times N$ then it must be the case that $L = M$ and $\mtx{C}$ will be of dimensionality $K \times N$.

The first exercise for this lab is to implement *forward propagation* for a single-layer model consisting of an affine transformation of the inputs in the `fprop` function given as skeleton code in the cell below. This should work for a batch of inputs of shape `(batch_size, input_dim)` producing a batch of outputs of shape `(batch_size, output_dim)`.

You will probably want to use the NumPy `dot` function and [broadcasting features](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html) to implement this efficiently. If you are not familiar with either / both of these you may wish to read the [hints](#Hints:-Using-the-dot-function-and-broadcasting) section below which gives some details on these before attempting the exercise.

```python
def fprop(inputs, weights, biases):
    """Forward propagates activations through the layer transformation.

    For inputs `x`, outputs `y`, weights `W` and biases `b` the layer
    corresponds to `y = W x + b`.

    Args:
        inputs: Array of layer inputs of shape (batch_size, input_dim).
        weights: Array of weight parameters of shape
            (output_dim, input_dim).
        biases: Array of bias parameters of shape (output_dim, ).

    Returns:
        outputs: Array of layer outputs of shape (batch_size, output_dim).
    """
    return inputs.dot(weights.T) + biases
```

Once you have implemented `fprop` in the cell above you can test your implementation by running the cell below.

```python
inputs = np.array([[0., -1., 2.], [-6., 3., 1.]])
weights = np.array([[2., -3., -1.], [-5., 7., 2.]])
biases = np.array([5., -3.])
true_outputs = np.array([[6., -6.], [-17., 50.]])

if not np.allclose(fprop(inputs, weights, biases), true_outputs):
    print('Wrong outputs computed.')
else:
    print('All outputs correct!')
```

    All outputs correct!

### Hints: Using the `dot` function and broadcasting

For those new to NumPy below are some details on the `dot` function and broadcasting feature of NumPy that you may want to use for implementing the first exercise. If you are already familiar with these and have already completed the first exercise you can move on straight to [second exercise](#Exercise-2:-visualising-random-models).

#### `numpy.dot` function

Matrix-matrix, matrix-vector and vector-vector (dot) products can all be computed in NumPy using the [`dot`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.dot.html) function. For example if `A` and `B` are both two dimensional arrays, then `C = np.dot(A, B)` or equivalently `C = A.dot(B)` will both compute the matrix product of `A` and `B` assuming `A` and `B` have compatible dimensions. Similarly if `a` and `b` are one dimensional arrays then `c = np.dot(a, b)` / `c = a.dot(b)` will compute the [scalar / dot product](https://en.wikipedia.org/wiki/Dot_product) of the two arrays. If `A` is a two-dimensional array and `b` a one-dimensional array `np.dot(A, b)` / `A.dot(b)` will compute the matrix-vector product of `A` and `b`. Examples of all three of these product types are shown in the cell below:

```python
# Initiliase arrays with arbitrary values
A = np.arange(9).reshape((3, 3))
B = np.ones((3, 3)) * 2
a = np.array([-1., 0., 1.])
b = np.array([0.1, 0.2, 0.3])
print(A.dot(B))  # Matrix-matrix product
print(B.dot(A))  # Reversed product of above A.dot(B) != B.dot(A) in general
print(A.dot(b))  # Matrix-vector product
print(b.dot(A))  # Again A.dot(b) != b.dot(A) unless A is symmetric i.e. A == A.T
print(a.dot(b))  # Vector-vector scalar product
```

    [[  6.   6.   6.]
     [ 24.  24.  24.]
     [ 42.  42.  42.]]
    [[ 18.  24.  30.]
     [ 18.  24.  30.]
     [ 18.  24.  30.]]
    [ 0.8  2.6  4.4]
    [ 2.4  3.   3.6]
    0.2

#### Broadcasting

Another NumPy feature it will be helpful to get familiar with is [broadcasting](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html). Broadcasting allows you to apply operations to arrays of different shapes, for example to add a one-dimensional array to a two-dimensional array or multiply a multidimensional array by a scalar. The complete set of rules for broadcasting as explained in the official documentation page just linked to can sound a bit complex: you might find the [visual explanation on this page](http://www.scipy-lectures.org/intro/numpy/operations.html#broadcasting) more intuitive. The cell below gives a few examples:

```python
# Initiliase arrays with arbitrary values
A = np.arange(6).reshape((3, 2))
b = np.array([0.1, 0.2])
c = np.array([-1., 0., 1.])
print(A + b)  # Add b elementwise to all rows of A
print((A.T + c).T)  # Add b elementwise to all columns of A
print(A * b)  # Multiply each row of A elementise by b
```

    [[ 0.1  1.2]
     [ 2.1  3.2]
     [ 4.1  5.2]]
    [[-1.  0.]
     [ 2.  3.]
     [ 5.  6.]]
    [[ 0.   0.2]
     [ 0.2  0.6]
     [ 0.4  1. ]]

## Exercise 2: visualising random models

In this exercise you will use your `fprop` implementation to visualise the outputs of a single-layer affine transform model with two-dimensional inputs and a one-dimensional output. In this simple case we can visualise the joint input-output space on a 3D axis.

For this task and the learning experiments later in the notebook we will use a regression dataset from the [UCI machine learning repository](http://archive.ics.uci.edu/ml/index.html). In particular we will use a version of the [Combined Cycle Power Plant dataset](http://archive.ics.uci.edu/ml/datasets/Combined+Cycle+Power+Plant), where the task is to predict the energy output of a power plant given observations of the local ambient conditions (e.g. temperature, pressure and humidity).

The original dataset has four input dimensions and a single target output dimension. We have preprocessed the dataset by [whitening](https://en.wikipedia.org/wiki/Whitening_transformation) it, a common preprocessing step. We will only use the first two dimensions of the whitened inputs (corresponding to the first two principal components of the inputs) so we can easily visualise the joint input-output space.

The dataset has been wrapped in the `CCPPDataProvider` class in the `mlp.data_providers` module and the data included as a compressed file in the data directory as `ccpp_data.npz`. Running the cell below will initialise an instance of this class, get a single batch of inputs and outputs and import the necessary `matplotlib` objects.

```python
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from mlp.data_providers import CCPPDataProvider
%matplotlib notebook

data_provider = CCPPDataProvider(
    which_set='train',
    input_dims=[0, 1],
    batch_size=5000,
    max_num_batches=1,
    shuffle_order=False
)

input_dim, output_dim = 2, 1

inputs, targets = data_provider.next()
```

Here we used the `%matplotlib notebook` magic command rather than the `%matplotlib inline` we used in the previous lab as this allows us to produce interactive 3D plots which you can rotate and zoom in/out by dragging with the mouse and scrolling the mouse-wheel respectively. Once you have finished interacting with a plot you can close it to produce a static inline plot using the <i class="fa fa-power-off"></i> button in the top-right corner.

Now run the cell below to plot the predicted outputs of a randomly initialised model across the two dimensional input space as well as the true target outputs. This sort of visualisation can be a useful method (in low dimensions) to assess how well the model is likely to be able to fit the data and to judge appropriate initialisation scales for the parameters. Each time you re-run the cell a new set of random parameters will be sampled

Some questions to consider:

  * How do the weights and bias initialisation scale affect the sort of predicted input-output relationships?
    * Magnitude of weights initialisation scale determines how steep along the two input directions the plane predictions lie on is. The magnitude of the bias initialisation scale determines the typical offset of the plane from the `output = 0.` plane.
  * Does the linear form of the model seem appropriate for the data here?
    * While it appears a linear-model will not be able to fully capture the input-output relationship evident in the data with there some degree of non-linearity seeming to be present, as a first approximation a linear model seems a reasonable choice as a simple model for the data.

```python
weights_init_range = 0.5
biases_init_range = 0.1

# Randomly initialise weights matrix
weights = rng.uniform(
    low=-weights_init_range,
    high=weights_init_range,
    size=(output_dim, input_dim)
)

#  Randomly initialise biases vector
biases = rng.uniform(
    low=-biases_init_range,
    high=biases_init_range,
    size=output_dim
)
# Calculate predicted model outputs
outputs = fprop(inputs, weights, biases)

# Plot target and predicted outputs against inputs on same axis
fig = plt.figure(figsize=(8, 8))
ax = fig.add_subplot(111, projection='3d')
ax.plot(inputs[:, 0], inputs[:, 1], targets[:, 0], 'r.', ms=2)
ax.plot(inputs[:, 0], inputs[:, 1], outputs[:, 0], 'b.', ms=2)
ax.set_xlabel('Input dim 1')
ax.set_ylabel('Input dim 2')
ax.set_zlabel('Output')
ax.legend(['Targets', 'Predictions'], frameon=False)
fig.tight_layout()
```

    <IPython.core.display.Javascript object>

## Exercise 3: computing the error function and its gradient

Here we will consider the task of regression as covered in the first lecture slides. The aim in a regression problem is given inputs $\fset{\vct{x}^{(n)}}_{n=1}^N$ to produce outputs $\fset{\vct{y}^{(n)}}_{n=1}^N$ that are as 'close' as possible to a set of target outputs $\fset{\vct{t}^{(n)}}_{n=1}^N$. The measure of 'closeness' or distance between target and predicted outputs is a design choice.

A very common choice is the squared Euclidean distance between the predicted and target outputs. This can be computed as the sum of the squared differences between each element in the target and predicted outputs. A common convention is to multiply this value by $\frac{1}{2}$ as this gives a slightly nicer expression for the error gradient. The error for the $n^{\textrm{th}}$ training example is then

\begin{equation}
    E^{(n)} = \frac{1}{2} \sum_{k=1}^K \lbr \lpa y^{(n)}_k - t^{(n)}_k \rpa^2 \rbr.
\end{equation}

The overall error is then the *average* of this value across all training examples

\begin{equation}
    \bar{E} = \frac{1}{N} \sum_{n=1}^N \lbr E^{(n)} \rbr.
\end{equation}

*Note here we are using a slightly different convention from the lectures. There the overall error was considered to be the sum of the individual error terms rather than the mean. To differentiate between the two we will use $\bar{E}$ to represent the average error here as opposed to sum of errors $E$ as used in the slides with $\bar{E} = \frac{E}{N}$. Normalising by the number of training examples is helpful to do in practice as this means we can more easily compare errors across data sets / batches of different sizes, and more importantly it means the size of our gradient updates will be independent of the number of training examples summed over.*

The regression problem is then to find parameters of the model which minimise $\bar{E}$. For our simple single-layer affine model here that corresponds to finding weights $\mtx{W}$ and biases $\vct{b}$ which minimise $\bar{E}$.

As mentioned in the lecture, for this simple case there is actually a closed form solution for the optimal weights and bias parameters. This is the linear least-squares solution those doing MLPR will have come across.

However in general we will be interested in models where closed form solutions do not exist. We will therefore generally use iterative, gradient descent based training methods to find parameters which (locally) minimise the error function. A basic requirement of being able to do gradient-descent based training is (unsuprisingly) the ability to evaluate gradients of the error function.

In the next exercise we will consider how to calculate gradients of the error function with respect to the model parameters $\mtx{W}$ and $\vct{b}$, but as a first step here we will consider the gradient of the error function with respect to the model outputs $\fset{\vct{y}^{(n)}}_{n=1}^N$. This can be written

\begin{equation}
    \pd{\bar{E}}{\vct{y}^{(n)}} = \frac{1}{N} \lpa \vct{y}^{(n)} - \vct{t}^{(n)} \rpa
    \qquad \Leftrightarrow \qquad
    \pd{\bar{E}}{y^{(n)}_k} = \frac{1}{N} \lpa y^{(n)}_k - t^{(n)}_k \rpa \quad \forall k \in \fset{1 \dots K}
\end{equation}

i.e. the gradient of the error function with respect to the $n^{\textrm{th}}$ model output is just the difference between the $n^{\textrm{th}}$ model and target outputs, corresponding to the $\vct{\delta}^{(n)}$ terms mentioned in the lecture slides.

The third exercise is, using the equations given above, to implement functions computing the mean sum of squared differences error and its gradient with respect to the model outputs. You should implement the functions using the provided skeleton definitions in the cell below.

```python
def error(outputs, targets):
    """Calculates error function given a batch of outputs and targets.

    Args:
        outputs: Array of model outputs of shape (batch_size, output_dim).
        targets: Array of target outputs of shape (batch_size, output_dim).

    Returns:
        Scalar error function value.
    """
    return 0.5 * ((outputs - targets)**2).sum() / outputs.shape[0]

def error_grad(outputs, targets):
    """Calculates gradient of error function with respect to model outputs.

    Args:
        outputs: Array of model outputs of shape (batch_size, output_dim).
        targets: Array of target outputs of shape (batch_size, output_dim).

    Returns:
        Gradient of error function with respect to outputs.
        This will be an array of shape (batch_size, output_dim).
    """
    return (outputs - targets) / outputs.shape[0]
```

Check your implementation by running the test cell below.

```python
outputs = np.array([[1., 2.], [-1., 0.], [6., -5.], [-1., 1.]])
targets = np.array([[0., 1.], [3., -2.], [7., -3.], [1., -2.]])
true_error = 5.
true_error_grad = np.array([[0.25, 0.25], [-1., 0.5], [-0.25, -0.5], [-0.5, 0.75]])

if not error(outputs, targets) == true_error:
    print('Error calculated incorrectly.')
elif not np.allclose(error_grad(outputs, targets), true_error_grad):
    print('Error gradient calculated incorrectly.')
else:
    print('Error function and gradient computed correctly!')
```

    Error function and gradient computed correctly!

## Exercise 4: computing gradients with respect to the parameters

In the previous exercise you implemented a function computing the gradient of the error function with respect to the model outputs. For gradient-descent based training, we need to be able to evaluate the gradient of the error function with respect to the model parameters.

Using the [chain rule for derivatives](https://en.wikipedia.org/wiki/Chain_rule#Higher_dimensions) we can write the partial deriviative of the error function with respect to single elements of the weight matrix and bias vector as

\begin{equation}
    \pd{\bar{E}}{W_{kj}} = \sum_{n=1}^N \lbr \pd{\bar{E}}{y^{(n)}_k} \pd{y^{(n)}_k}{W_{kj}} \rbr
    \quad \textrm{and} \quad
    \pd{\bar{E}}{b_k} = \sum_{n=1}^N \lbr \pd{\bar{E}}{y^{(n)}_k} \pd{y^{(n)}_k}{b_k} \rbr.
\end{equation}

From the definition of our model at the beginning we have

\begin{equation}
    y^{(n)}_k = \sum_{d=1}^D \lbr W_{kd} x^{(n)}_d \rbr + b_k
    \quad \Rightarrow \quad
    \pd{y^{(n)}_k}{W_{kj}} = x^{(n)}_j
    \quad \textrm{and} \quad
    \pd{y^{(n)}_k}{b_k} = 1.
\end{equation}

Putting this together we get that

\begin{equation}
    \pd{\bar{E}}{W_{kj}} =
    \sum_{n=1}^N \lbr \pd{\bar{E}}{y^{(n)}_k} x^{(n)}_j \rbr
    \quad \textrm{and} \quad
    \pd{\bar{E}}{b_{k}} =
    \sum_{n=1}^N \lbr \pd{\bar{E}}{y^{(n)}_k} \rbr.
\end{equation}

Although this may seem a bit of a roundabout way to get to these results, this method of decomposing the error gradient with respect to the parameters in terms of the gradient of the error function with respect to the model outputs and the derivatives of the model outputs with respect to the model parameters, will be key when calculating the parameter gradients of more complex models later in the course.

Your task in this exercise is to implement a function calculating the gradient of the error function with respect to the weight and bias parameters of the model given the already computed gradient of the error function with respect to the model outputs. You should implement this in the `grads_wrt_params` function in the cell below.

```python
def grads_wrt_params(inputs, grads_wrt_outputs):
    """Calculates gradients with respect to model parameters.

    Args:
        inputs: array of inputs to model of shape (batch_size, input_dim)
        grads_wrt_to_outputs: array of gradients of with respect to the model
            outputs of shape (batch_size, output_dim).

    Returns:
        list of arrays of gradients with respect to the model parameters
        `[grads_wrt_weights, grads_wrt_biases]`.
    """
    grads_wrt_weights = grads_wrt_outputs.T.dot(inputs)
    grads_wrt_biases = grads_wrt_outputs.sum(0)
    return [grads_wrt_weights, grads_wrt_biases]
```

Check your implementation by running the test cell below.

```python
inputs = np.array([[1., 2., 3.], [-1., 4., -9.]])
grads_wrt_outputs = np.array([[-1., 1.], [2., -3.]])
true_grads_wrt_weights = np.array([[-3., 6., -21.], [4., -10., 30.]])
true_grads_wrt_biases = np.array([1., -2.])

grads_wrt_weights, grads_wrt_biases = grads_wrt_params(
    inputs, grads_wrt_outputs)

if not np.allclose(true_grads_wrt_weights, grads_wrt_weights):
    print('Gradients with respect to weights incorrect.')
elif not np.allclose(true_grads_wrt_biases, grads_wrt_biases):
    print('Gradients with respect to biases incorrect.')
else:
    print('All parameter gradients calculated correctly!')
```

    All parameter gradients calculated correctly!

## Exercise 5: wrapping the functions into reusable components

In exercises 1, 3 and 4 you implemented methods to compute the predicted outputs of our model, evaluate the error function and its gradient on the outputs and finally to calculate the gradients of the error with respect to the model parameters. Together they constitute all the basic ingredients we need to implement a gradient-descent based iterative learning procedure for the model.

Although you could implement training code which directly uses the functions you defined, this would only be usable for this particular model architecture. In subsequent labs we will want to use the affine transform functions as the basis for more interesting multi-layer models. We will therefore wrap the implementations you just wrote in to reusable components that we can build more complex models with later in the course.

  * In the [`mlp.layers`](/edit/mlp/layers.py) module, use your implementations of `fprop` and `grad_wrt_params` above to implement the corresponding methods in the skeleton `AffineLayer` class provided.
  * In the [`mlp.errors`](/edit/mlp/errors.py) module use your implementation of `error` and `error_grad` to implement the `__call__` and `grad` methods respectively of the skeleton `SumOfSquaredDiffsError` class provided. Note `__call__` is a special Python method that allows an object to be used with a function call syntax.

Run the cell below to use your completed `AffineLayer` and `SumOfSquaredDiffsError` implementations to train a single-layer model using batch gradient descent on the CCCP dataset.

```python
from mlp.layers import AffineLayer
from mlp.errors import SumOfSquaredDiffsError
from mlp.models import SingleLayerModel
from mlp.initialisers import UniformInit, ConstantInit
from mlp.learning_rules import GradientDescentLearningRule
from mlp.optimisers import Optimiser
import logging

# Seed a random number generator
seed = 27092016
rng = np.random.RandomState(seed)

# Set up a logger object to print info about the training run to stdout
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.handlers = [logging.StreamHandler()]

# Create data provider objects for the CCPP training set
train_data = CCPPDataProvider('train', [0, 1], batch_size=100, rng=rng)
input_dim, output_dim = 2, 1

# Create a parameter initialiser which will sample random uniform values
# from [-0.1, 0.1]
param_init = UniformInit(-0.1, 0.1, rng=rng)

# Create our single layer model
layer = AffineLayer(input_dim, output_dim, param_init, param_init)
model = SingleLayerModel(layer)

# Initialise the error object
error = SumOfSquaredDiffsError()

# Use a basic gradient descent learning rule with a small learning rate
learning_rule = GradientDescentLearningRule(learning_rate=1e-2)

# Use the created objects to initialise a new Optimiser instance.
optimiser = Optimiser(model, error, learning_rule, train_data)

# Run the optimiser for 5 epochs (full passes through the training set)
# printing statistics every epoch.
stats, keys, run_time = optimiser.train(num_epochs=10, stats_interval=1)

# Plot the change in the error over training.
fig = plt.figure(figsize=(8, 4))
ax = fig.add_subplot(111)
ax.plot(np.arange(1, stats.shape[0] + 1), stats[:, keys['error(train)']])
ax.set_xlabel('Epoch number')
ax.set_ylabel('Error')
```

    Epoch 1: 0.00s to complete
        error(train)=1.67e-01, cost(param)=0.00e+00
    Epoch 2: 0.00s to complete
        error(train)=9.30e-02, cost(param)=0.00e+00
    Epoch 3: 0.00s to complete
        error(train)=7.95e-02, cost(param)=0.00e+00
    Epoch 4: 0.00s to complete
        error(train)=7.71e-02, cost(param)=0.00e+00
    Epoch 5: 0.00s to complete
        error(train)=7.66e-02, cost(param)=0.00e+00
    Epoch 6: 0.00s to complete
        error(train)=7.65e-02, cost(param)=0.00e+00
    Epoch 7: 0.00s to complete
        error(train)=7.65e-02, cost(param)=0.00e+00
    Epoch 8: 0.00s to complete
        error(train)=7.65e-02, cost(param)=0.00e+00
    Epoch 9: 0.00s to complete
        error(train)=7.63e-02, cost(param)=0.00e+00
    Epoch 10: 0.00s to complete
        error(train)=7.64e-02, cost(param)=0.00e+00

    <IPython.core.display.Javascript object>

    <matplotlib.text.Text at 0x7fd88df72350>

Using similar code to previously we can now visualise the joint input-output space for the trained model. If you implemented the required methods correctly you should now see a much improved fit between predicted and target outputs when running the cell below.

```python
data_provider = CCPPDataProvider(
    which_set='train',
    input_dims=[0, 1],
    batch_size=5000,
    max_num_batches=1,
    shuffle_order=False
)

inputs, targets = data_provider.next()

# Calculate predicted model outputs
outputs = model.fprop(inputs)[-1]

# Plot target and predicted outputs against inputs on same axis
fig = plt.figure(figsize=(8, 8))
ax = fig.add_subplot(111, projection='3d')
ax.plot(inputs[:, 0], inputs[:, 1], targets[:, 0], 'r.', ms=2)
ax.plot(inputs[:, 0], inputs[:, 1], outputs[:, 0], 'b.', ms=2)
ax.set_xlabel('Input dim 1')
ax.set_ylabel('Input dim 2')
ax.set_zlabel('Output')
ax.legend(['Targets', 'Predictions'], frameon=False)
fig.tight_layout()
```

    <IPython.core.display.Javascript object>

## Exercise 6: visualising training trajectories in parameter space

Running the cell below will display an interactive widget which plots the trajectories of gradient-based training of the single-layer affine model on the CCPP dataset in the three dimensional parameter space (two weights plus bias) from random initialisations. Also shown on the right is a plot of the evolution of the error function (evaluated on the current batch) over training. By moving the sliders you can alter the training hyperparameters to investigate the effect they have on how training procedes.

Some questions to explore:

  * Are there multiple local minima in parameter space here? Why?
    * In this case there is a single unique global minima, as suggested by the fact random parameter initialisations consistently converge to the same point in parameter space. As mentioned previously there is a closed form solution for the optimal weights and biases for this simple single-layer affine model <a href='https://en.wikipedia.org/wiki/Linear_least_squares_(mathematics)'>linear least squares</a>) and the error function is [convex](https://en.wikipedia.org/wiki/Convex_function).
  * What happens to learning for very small learning rates? And very large learning rates?
    * For very small learning rates, the training proceeds very slowly and the parameters do not tend to converge to the global optimum unless a lot of training epochs are used. For very large learning rates, the gradient descent dynamic becomes increasingly instable, leading to large oscillations or at extreme values divergence in the parameter space.
  * How does the batch size affect learning?
    * Smaller batch sizes generally lead to quicker initial learning as the parameters are updated more frequently (more batches in an epoch) however as the batch becomes smaller the error estimate calculated from the batch and its gradients become increasingly noisy estimates for the true error function / gradients. This can be observed by the less smooth trajectories in parameter space for lower batch sizes and greater noise in the batch error curves.

**Note:** You don't need to understand how the code below works. The idea of this exercise is to help you understand the role of the various hyperparameters involved in gradient-descent based training methods.

```python
from ipywidgets import interact
%matplotlib inline

def setup_figure():
    # create figure and axes
    fig = plt.figure(figsize=(12, 6))
    ax1 = fig.add_axes([0., 0., 0.5, 1.], projection='3d')
    ax2 = fig.add_axes([0.6, 0.1, 0.4, 0.8])
    # set axes properties
    ax2.spines['right'].set_visible(False)
    ax2.spines['top'].set_visible(False)
    ax2.yaxis.set_ticks_position('left')
    ax2.xaxis.set_ticks_position('bottom')
    ax2.set_yscale('log')
    ax1.set_xlim((-2, 2))
    ax1.set_ylim((-2, 2))
    ax1.set_zlim((-2, 2))
    #set axes labels and title
    ax1.set_title('Parameter trajectories over training')
    ax1.set_xlabel('Weight 1')
    ax1.set_ylabel('Weight 2')
    ax1.set_zlabel('Bias')
    ax2.set_title('Batch errors over training')
    ax2.set_xlabel('Batch update number')
    ax2.set_ylabel('Batch error')
    return fig, ax1, ax2

def visualise_training(n_epochs=1, batch_size=200, log_lr=-1., n_inits=5,
                       w_scale=1., b_scale=1., elev=30., azim=0.):
    fig, ax1, ax2 = setup_figure()
    # create seeded random number generator
    rng = np.random.RandomState(1234)
    # create data provider
    data_provider = CCPPDataProvider(
        input_dims=[0, 1],
        batch_size=batch_size,
        shuffle_order=False,
    )
    learning_rate = 10 ** log_lr
    n_batches = data_provider.num_batches
    weights_traj = np.empty((n_inits, n_epochs * n_batches + 1, 1, 2))
    biases_traj = np.empty((n_inits, n_epochs * n_batches + 1, 1))
    errors_traj = np.empty((n_inits, n_epochs * n_batches))
    # randomly initialise parameters
    weights = rng.uniform(-w_scale, w_scale, (n_inits, 1, 2))
    biases = rng.uniform(-b_scale, b_scale, (n_inits, 1))
    # store initial parameters
    weights_traj[:, 0] = weights
    biases_traj[:, 0] = biases
    # iterate across different initialisations
    for i in range(n_inits):
        # iterate across epochs
        for e in range(n_epochs):
            # iterate across batches
            for b, (inputs, targets) in enumerate(data_provider):
                outputs = fprop(inputs, weights[i], biases[i])
                errors_traj[i, e * n_batches + b] = error(outputs, targets)
                grad_wrt_outputs = error_grad(outputs, targets)
                weights_grad, biases_grad = grads_wrt_params(inputs, grad_wrt_outputs)
                weights[i] -= learning_rate * weights_grad
                biases[i] -= learning_rate * biases_grad
                weights_traj[i, e * n_batches + b + 1] = weights[i]
                biases_traj[i, e * n_batches + b + 1] = biases[i]
    # choose a different color for each trajectory
    colors = plt.cm.jet(np.linspace(0, 1, n_inits))
    # plot all trajectories
    for i in range(n_inits):
        lines_1 = ax1.plot(
            weights_traj[i, :, 0, 0],
            weights_traj[i, :, 0, 1],
            biases_traj[i, :, 0],
            '-', c=colors[i], lw=2)
        lines_2 = ax2.plot(
            np.arange(n_batches * n_epochs),
            errors_traj[i],
            c=colors[i]
        )
    ax1.view_init(elev, azim)
    plt.show()

w = interact(
    visualise_training,
    elev=(-90, 90, 2),
    azim=(-180, 180, 2),
    n_epochs=(1, 5),
    batch_size=(100, 1000, 100),
    log_lr=(-3., 1.),
    w_scale=(0., 2.),
    b_scale=(0., 2.),
    n_inits=(1, 10)
)

for child in w.widget.children:
    child.layout.width = '100%'
```