$$ \LaTeX \text{ command declarations here.}
\newcommand{\R}{\mathbb{R}}
\renewcommand{\vec}[1]{\mathbf{#1}}
\newcommand{\X}{\mathcal{X}}
\newcommand{\D}{\mathcal{D}}
\newcommand{\d}{\mathrm{d}}
$$

```python
from __future__ import division

# plotting
%matplotlib inline
from matplotlib import pyplot as plt;
import seaborn as sns
import pylab as pl
from matplotlib.pylab import cm
import pandas as pd

# scientific
import numpy as np;

# ipython
from IPython.display import Image
```

# EECS 445:  Machine Learning
## Lecture 11:  Information Theory and Decision Trees
* Instructor:  **Jacob Abernethy**
* Date:  October 12, 2016

## Outline

- Information Theory
    - Information, Entropy, Maximum Entropy Distributions
    - Entropy and Encoding, Cross Entropy, Relative Entropy
    - Mutual Information & Collocations

- Decision Trees
    - Definition of a decision tree
    - Metrics for selecting decision stumps

## Reading List

- **[PRML]**, §1.6: Information Theory
- **[MLAPP]**, §2.8: Information Theory

## Other References

- Information Theory:
    - **[Shannon 1951]** Shannon, Claude E.. [*The Mathematical Theory of Communication*](http://worrydream.com/refs/Shannon%20-%20A%20Mathematical%20Theory%20of%20Communication.pdf).  1951.
    - **[Pierce 1980]** Pierce, John R..  [*An Introduction to Information Theory:  Symbols, Signals, and Noise*](http://www.amazon.com/An-Introduction-Information-Theory-Mathematics/dp/0486240614).  1980.
    - **[Stone 2015]** Stone, James V..  [*Information Theory:  A Tutorial Introduction*](http://jim-stone.staff.shef.ac.uk/BookInfoTheory/InfoTheoryBookMain.html).  2015.

> This lecture some basics of information theory and will be introduced. These will provide some important background for **Probabilistic Graphical Models**, which is a big topic that we will cover for several following lectures. For information theory, some definitions like information, entropy, cross entropy, relative entropy, etc. are to be introduced. We could see how entropy is related to compression theory. As for applications, we will show how information theory can help us select features and find most frequent collocations in a novel.

## Information Theory

> Uses material from **[MLAPP]** §2.8, **[Pierce 1980]**, **[Stone 2015]**, and **[Shannon 1951]**.

### Information Theory

- Information theory is concerned with
    - **Compression:**  Representing data in a compact fashion
    - **Error Correction:**  Transmitting and storing data in a way that is robust to errors

- In machine learning, information-theoretic quantities are useful for
    - manipulating probability distributions
    - interpreting statistical learning algorithms

### What is Information?

- Can we measure the amount of **information** we gain from an observation?
    - Information is measured in *bits* ( don't confuse with *binary digits*, $0110001\dots$ )
    - Intuitively, observing a fair coin flip should give 1 bit of information
    - Observing two fair coins should give 2 bits, and so on...

### Information:  Definition

- The **information content** of an event $E$ with probability $p$ defined as
    $$
    I(E) = I(p) = - \log_2 p = \log_2 \frac{1}{p} \geq 0
    $$

    - Information theory is about *probabilities* and *distributions*
    - The "meaning" of events doesn't matter.
    - Using bases other than 2 yields different units (Hartleys, nats, ...)

### Information Example:  Fair Coin—$P(\text{Head})=0.5$

- **One Coin:**  If we observe one head, then
    $$
    I(\text{Head}) = - \log_2 P(\text{Head}) = 1 \;\mathrm{bit}
    $$

- **Two Coins:** If we observe two heads in a row,
    $$
    \begin{align}
    I(\text{Head},\text{Head})
    &= -\log_2 P(\text{Head}, \text{Head}) \\
    &= -\log_2 P(\text{Head})P(\text{Head}) \\
    &= -\log_2 P(\text{Head}) - \log_2 P(\text{Head}) = 2 \;\mathrm{bits}
    \end{align}
    $$

### Information Example:  Unfair Coin

- Suppose the coin has two heads, so $P(\text{Head})=1$.  Then,
    $$
    I(\text{Head}) = - \log_2 1 = 0
    $$
    - We will gain no information!
- On the contrary, if we observe tail
    $$
    I(\text{Tail}) = - \log_2 0 = + \infty
    $$
    - We will gain *infinite* information because we observe an impossible thing!

- Information is a measure of how **surprised** we are by an outcome.

### Entropy:  Definition

- The **entropy** of a discrete random variable $X$ with distribution $p$ is
    $$
    H[X] = E[I(p(X))] = - \sum_{x \in X} p(x) \log p(x)
    $$
    - Entropy is the expected information received when we sample from $X$.
    - Entropy measures how *surprised* we are on average
    - When $X$ is continuous random variable, summation is replaced with integral

### Entropy:  Coin Flip

- If $X$ is binary, entropy is
    $$
    H[X] = -p \log p + (1-p) \log (1-p)
    $$

- Entropy is highest when $X$ is close to uniform.
    - Large entropy $\iff$ high uncertainty, more information from each new observation
    - Small entropy $\iff$ more knowledge about possible outcomes

- The farther from uniform $X$ is, the smaller the entropy.

### Maximum Entropy Principle

- Suppose we sample data from an unknown distribution $p$, and
    - we collect statistics (mean, variance, etc.) from the data
    - we want an *objective* or unbiased estimate of $p$
    The **Maximum Entropy Principle** states that:

> We should choose $p$ to have maximum entropy $H[p]$ among all distributions satisfying our constraints.

- Some examples of maximum entropy distributions:

Constraints	Maximum Entropy Distribution

Min $a$, Max $b$	Uniform $U[a,b]$

Mean $\mu$, Support $(0,+\infty)$	Exponential $Exp(\mu)$

Mean $\mu$, Variance $\sigma^2$	Gaussian $\mathcal{N}(\mu, \sigma^2)$

- Later, **Exponential Family Distributions** will generalize this concept.

### Entropy and Encoding: Communication Channel

- Now let's see how entropy is related to encoding theory
- **Communication channel** can be characterized as:
    - **[Source]** generates messages.
    - **[Encoder]** converts the message to a **signal** for transmission.
    - **[Channel]** is the path along which signals are transmitted, possibly under the influence of **noise**.
    - **[Decoder[** attempts to reconstruct the original message from the transmitted signal.
    - **[Destination]** is the intended recipient.

### Entropy and Encoding: Encoding

- Suppose we draw messages from a distribution $p$.
    - Certain messages may be more likely than others.
    - For example, the letter **e** is most frequent in English

- An **efficient** encoding minimizes the average code length,
    - assign *short* codewords to common messages
    - and *longer* codewords to rare messages

- Example: **Morse Code**

### Entropy and Encoding: Source Coding Theorem

- Claude Shannon proved that for discrete noiseless channels:

> It is impossible to encode messages drawn from a distribution $p$ with fewer than $H[p]$ bits, on average.

- Here, *bits* refers to *binary digits*, i.e. encoding messages in binary.

> $H[p]$ measures the optimal code length, in bits, for messages drawn from $p$

### Cross Entropy & Relative Entropy

- Consider different distributions $p$ and $q$
    - What if we use a code optimal for $q$ to encode messages from $p$?

- For example, suppose our encoding scheme is optimal for German text.
    - What if we send English messages instead?
    - Certainly, there will be some waste due to different letter frequencies, umlauts, ...

### Cross Entropy & Relative Entropy

- **Cross entropy** measures the average number of bits needed to encode messages drawn from $p$ when we use a code optimal for $q$:
    $$
    H(p,q) = -\sum_{x \in \X} p(x) \log q(x)
    = - E_p[\log q(x)]
    $$

- Intuitively, $H(p,q) \geq H(p)$.

- **Relative entropy** is the difference $H(p,q) - H(p)$.

- Relative entropy, aka **Kullback-Leibler divergence**, of $q$ from $p$ is
    $$
    \begin{align}
    D_{KL}(p \| q)
    &= H(p,q) - H(p) \\
    &= \sum_{x \in X} p(x) \log \frac{p(x)}{q(x)} \\
    \end{align}
    $$

> Measures the number of *extra* bits needed to encode messages from $p$ if we use a code optimal for $q$.

### Mutual Information:  Definition

- **Mutual information** between discrete variables $X$ and $Y$ is
    $$
    \begin{align}
    I(X; Y)
    &= \sum_{y\in Y} \sum_{x \in X} p(x,y) \log\frac{p(x,y)}{p(x)p(y)} \\
    &= D_{KL}( p(x,y) \| p(x)p(y) )
    \end{align}
    $$

    - If $X$ and $Y$ are independent, $p(x,y)=p(x)p(y)$ and $I(X; Y)=0$
    - So, $I(X;Y)$ measures how *dependent* $X$ and $Y$ are!
    - Related to correlation $\rho(X,Y)$

### Mutual Information: Example of Feature Selection

- Mutual information can also be used for **feature selection**.
    - In classification, features that *depend* most on the class label $C$ are useful
    - So, choose features $X_k$ such that $I(X_k ; C)$ is large
    - This helps to avoid *overfitting* by ignoring irrelevant features!

> See **[MLAPP]** §3.5.4 for more information

### Pointwise Mutual Information

- A **collocation** is a sequence of words that co-occur more often than expected by chance.
    - fixed expression familiar to native speakers (hard to translate)
    - meaning of the whole is more than the sum of its parts
    - See [these slides](https://www.eecis.udel.edu/~trnka/CISC889-11S/lectures/philip-pmi.pdf) for more details

- Substituting a synonym sounds unnatural:
    - "fast food" vs. "quick food"
    - "Great Britain" vs. "Good Britain"
    - "warm greetings" vs "hot greetings"

- How can we find collocations in a corpus of text?

### Pointwise Mutual Information

- The **pointwise mutual information (PMI)** between words $x$ and $y$ is
    $$
    \mathrm{pmi}(x;y) = \log \frac{p(x,y)}{p(x)p(y)}
    $$

    - $p(x)p(y)$ is how frequently we **expect** $x$ and $y$ to co-occur, if $x$ and $y$ are independent.
    - $p(x,y)$ measures how frequently $x$ and $y$ **actually** occur together

- **Idea:**  Rank word pairs by $\mathrm{pmi}(x,y)$ to find collocations!
    - $\mathrm{pmi}(x,y)$ is large if $x$ and $y$ co-occur more frequently together than expected

- **Example:** Let's try it on the novel *Crime and Punishment*!
    - Pre-computed unigram and bigram counts are found in the `collocations/data` folder

## Break time!

```python
### Requirements: PyDotPlus, Matplotlib, Scikit-Learn, Pandas, Numpy, IPython (and possibly GraphViz)

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import Imputer
from sklearn import preprocessing
from sklearn.tree import DecisionTreeRegressor, DecisionTreeClassifier
from sklearn.ensemble import AdaBoostClassifier
from sklearn.ensemble import BaggingRegressor
from sklearn.ensemble import BaggingClassifier
from sklearn import tree
import sklearn
import sklearn.metrics as skm
from scipy import misc
from sklearn.externals.six import StringIO
import pydotplus
from IPython.display import Image, YouTubeVideo

def visualize_tree(tree, feature_names, class_names):
    dot_data = StringIO()
    sklearn.tree.export_graphviz(tree, out_file=dot_data,
                         filled=True, rounded=True,
                         feature_names=feature_names,
                         class_names=class_names,
                         special_characters=True)
    graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
    return graph.create_png()
```

### Decision Trees Overview

In this lecture we will:
- Look at a class of ML algorithms that are particularly good in the sense that the learned model is in general interpretable. (***Decision Trees***)
- **Later**: some "meta-algorithms" that combine learned models (***Ensemble Learning***), particularly for:
    - Improving stability and reducing variance (***Bagging***) and,
    - Learning from some set of "weak" models. (***Boosting***)

### Motivating Example: 20 Questions (1)
(Example: http://en.akinator.com)

- Famous game where a player thinks of something and another player must guess what that thing is.
 - Many variants: People, Animals, Music, ...
- Premise: I am thinking of a person. Ask me up to 20 yes/no questions to determine who this person is that I am thinking about.

### Motivating Example: 20 Questions (2)
#### (Things to think about)
- The questions (and possibly their order as well) matter a lot!
 - Which ones should you choose? (These will determine features!)
 - When should you ask them?
- With 20 questions each with a binary answer, one can distringuish at the most $2^{20} = 1,048,576$ people.
 - Still far less than the number of people anyone really knows. Does this mean playing this game and winning is in some sense easy?
- If you were to develop such an A.I. what kind of algorithms/techniques would you choose? (Decision Trees are just one possibility!)

### An "obvious" representation: Classification Decision Trees (1)

- Observations are classified into two or more classes, coded by a response variable $Y$ taking values $\in \{1, 2, . . . , K\}$.
- We have a feature vector $\mathbf{x} = (x_1, x_2, ..., x_p)$, and we hope to build a classification rule $C(\mathbf{x})$ to assign a class label to $\mathbf{x}$.
- We have a sample of pairs $(\mathbf{x}_i, y_i), i = 1, ..., n$. Note that each of the $\mathbf{x}_i$ are vectors $\mathbf{x}_i = (x_{i1},x_{i2}, ..., x_{ip})$.
- An Example: $Y$ indicates whether an E-mail is spam or not. $\mathbf{x}$ is bag-of-words representation of the email.

### Classification Decision Trees (2)

- As the name suggests, the main data structure is a Tree.
- Each node represents a value query on one of the variables — e.g. “Is $x_3$ > 0.4”. If the answer is “Yes”, go right, else go left.
- The leaf nodes are the decision nodes where a classification is made.
- New observations $\mathbf{x}$ are passed from the root down to a terminal node of the tree, and then the class label is determined at the terminal node.

```python
# Decision Tree Example: Poisonous/Edible Mushroom Classification
# (from the UCI Repository)
data = pd.read_csv('agaricus-lepiota.data', header=None)
# Preprocessing (Note: Label Encoder is a very useful tool to convert categorical data!)
le = preprocessing.LabelEncoder()
# Change columns from labels to integer categories
#(See agaricus-lepiota.data for the initial labels and
# agaricus-lepiota.names for a full description of the columns)
data = data.apply(le.fit_transform)
# Use a Decision Tree with maximum depth = 5
dt_classifier = DecisionTreeClassifier(max_depth=5)
dt = dt_classifier.fit(data.ix[:,1:], data.ix[:,0])
```

```python
# Visualize the decision tree with class names and feature names (See the first cell for the function)
Image(visualize_tree(dt, feature_names =
                     open('agaricus-lepiota.feature-names').readlines(),
                     class_names = ['edible', 'poisonous']))
```

### Decision Tree Example: Fruits
#### Note: This example illustrates that Decision Trees don't have to be binary trees or involve binary classification.

### Decision Tree Construction Ideas

- The general idea behind decision trees like any other classification algorithm is to partition the space of inputs based on their corresponding classes.
- Moreover, we are looking for the "best split" at each level.
 - Often explained using an "impurity" measure or metric which tells us how certain a classification can be made using a particular query and all previously gained information.
- We repeat the "best splitting" procedure recursively in a Divide and Conquer fashion.

### Metrics: Information Gain (Mutual Information)
#### Used by the ID3, C4.5 and C5.0 tree-generation algorithms.

**Recall** Information Gain = Entropy(Parent) -  Weighted Sum of Entropy(Children)

$$IG(P,a) = H(P) - H(P|a)$$

Remember, the entropy is $\text{H(p)} = -\sum \limits_{i = 1}^{k_m} p_i \log_2 p_i$ where the query is concerned with some feature $x_k$ and the query has $k_m$ responses.

### A question to illustrate the Information Gain metric

Note:
 - The $[x+, y-]$ indicate the number of samples belonging to the two classes, say positive and negative.
 - The topmost one denotes the number of positive and negative samples in the dataset before "any partitioning."

### Answer

- Entropy of the dataset, say $D$:
$H(D) = -\left(\frac{29}{29 + 35} \log_2 \frac{29}{29 + 35} + \frac{35}{29 + 35} \log_2 \frac{35}{29 + 35}\right) = 0.99365071169$. So:

    - $H(D|A1) =  -\left(\frac{21}{21 + 5} \log_2 \frac{21}{21 + 5} +  \frac{5}{21 + 5} \log_2 \frac{5}{21 + 5}\right) = 0.70627408918$
    - $H(D| \neg A1) = -\left(\frac{8}{8 + 20} \log_2 \frac{8}{8 + 30} + \frac{30}{8 + 30} \log_2 \frac{30}{8 + 30}\right) = 0.74248756954$

### Answer (Continued): Calculation of Weighting for Children when using Attribute $A1$

- Weighting for $A1 = $ $\frac{21 + 5}{29 + 35} = \frac{26}{64} = 0.40625$
- Weighting for $\neg A1 = $ $\frac{8 + 30}{29 + 35} = \frac{38}{64} = 0.59375$

### Answer (Continued): Weighted Sum of Entropy and Final Information Gain Calculation
- Weighted Sum of Entropy of Children = $0.70627408918 \cdot 0.40625 + 0.74248756954 \cdot 0.59375 = 0.7277758431$
- Therefore, the information gain using attrbiute $A1 = 0.99365071169 - 0.7277758431 = 0.2658748686$

### Answer (Continued): Final Answer

Repeating the process shown for attribute $A2$, we get that the information gain for attribute $A2 = 0.1214$ < the information gain for attribute $A1 = 0.2658$.

### So, what is Information Gain?

- Intuitively, Information Gain captures:
 - The mutual information that there is between an attribute and the class labels, or, equivalently,
 - The reduction in entropy gained by observing an attribute.
- Another intersting note:
 - Mutual information (i.e., Information Gain) and KL-divergence are connected: $IG(X, Y) = D_{KL}(p(x, y) \mid \mid p(x)p(y))$.

### Metrics: Misclassification Error

$I(N) = 1 - \max \limits_{i} P(i \mid N)$ for a query node $N$

- Measures misclassification error made by a node.
- Maximum $(1 - 1/K)$ when samples are equally distributed among all classes, implying node $N$ provides really no information. (Note: $K$ is the number of classes)
- Minimum $(0.0)$ when all samples belong to one class, implying node $N$ provides enough information to be fully certain in the classification.

###  Metrics: Gini Impurity
#### Used by the CART (Classification and Regression Tree) Algorithm.
- A measure of how often a randomly chosen element from the set would be incorrectly labeled if it were randomly labeled according to the distribution of labels in the subset.
- To compute Gini impurity for a set of  m  items, suppose $i \in \{1, 2, ..., m\}$, and let $f_i$ be the fraction of items labeled with value $i$ in the set.
    $$I_{G}(f) = \sum_{i=1}^{m} f_i (1-f_i) = \sum_{i=1}^{m} (f_i - {f_i}^2) = \\ \sum_{i=1}^m f_i - \sum_{i=1}^{m} {f_i}^2 = 1 - \sum^{m}_{i=1} {f_i}^{2} = \sum_{i\neq k}f_i f_k$$

```python
%matplotlib inline

def gini(p):
    return (p) * (1 - p) + (1 - p) * (1 - (1 - p))

def entropy(p):
    return -(p * np.log2(p) + (1 - p) * np.log2(1 - p))

def error(p):
    return 1 - np.max([p, 1 - p])

def plot_impurity_metrics():
    x = np.arange(0.0, 1.0, 0.01)
    entropy_t = [entropy(p) if p != 0 else None for p in x]
    entropy_scaled_t = [0.5 * i if i != None else None for i in entropy_t]
    gini_t = [gini(p) for p in x]
    error_t = [error(p) for p in x]

    fig = plt.figure()
    ax = plt.subplot(111)

    for i, lab, ls, c in zip([entropy_t, entropy_scaled_t, gini_t, error_t],
                             ['Entropy', 'Entropy (Scaled)', 'Gini', 'Misclassification Error'],
                             ['-', '-', '--', '-.'], ['black', 'lightgray',
                             'red', 'lightgreen', 'cyan']):
        line = ax.plot(x, i, label = lab, linestyle = ls, color = c, lw = 2)
    ax.legend(loc = 'upper center', ncol = 3, bbox_to_anchor = (0.5, 1.15), fancybox = True, shadow = False)
    ax.axhline(y = 1, linewidth = 1, color = 'k', linestyle = '--')
    ax.axhline(y = 0.5, linewidth = 1, color = 'k', linestyle = '--')
    plt.ylim([0, 1.1])
    plt.xlabel('p(Attribute = True)')
    plt.ylabel('Impurity Index')
```

```python
# The Metrics Illustrated in a Graph For a 2-Class Problem
plot_impurity_metrics()
# Note: The Gini Impurity Index is in between the misclassifcation
#       error curve and the scaled Entropy Curve!
```

### Choosing an Impurity Measure
All three measures are similar, and often construct identical trees. However:
- Entropy and Gini are differentiable, and hence more subject to numerical optimization.
- Entropy and Gini are more sensitive to changes in the node probabilities than Misclassification error.
- Entropy maybe slow to compute.
- Entropy is suggested for exploratory analysis as it is based on mutual information and thus may reveal insights about the data.
- Gini Impurity is suggested to minimize misclassification (see the note on the previous slide.)

### Pseudocode Iterative Dichotomizer 3 (ID3)

* ID3 was first "decision tree" alg.
* Precursor to C4.5 Alg. [More here](https://en.wikipedia.org/wiki/ID3_algorithm)

### The Value of Labeled Data
- Labeled data is often hard to come by, various approaches can be taken to mitigate this however.
- In general, we can use many semi-supervised techniques.
- Active Learning is a particularly notable one, used in for example 20-Questions like games.

```python
### Decision Tree Active Learning Example: Animal Game

import string

class node :
    "Node objects have a question, and  left and right pointer to other nodes"
    def __init__ (self, question, left=None, right=None) :
        self.question = question
        self.left     = left
        self.right    = right

def yes (ques) :
    "Force the user to answer 'yes' or 'no' or something similar. Yes returns true"
    while 1 :
        ans = raw_input (ques)
        ans = string.lower(ans[0:1])
        if ans == 'y' : return 1
        elif ans == 'n' : return 0

knowledge = node("bird")

def active_learning_example (suppress=True) :
    "Guess the animal. Add a new node for a wrong guess."
    first = True
    while not suppress and (first or yes("Continue? (y/n)")):
        if first:
            first = False
        print
        if not yes("Are you thinking of an animal? (y/n)") : break
        p = knowledge
        while p.left != None :
            if yes(p.question+"? ") : p = p.right
            else                    : p = p.left

        if yes("Is it a " + p.question + "? ") : continue
        animal   = raw_input ("What is the animals name? ")
        question = raw_input ("What question would distinguish a %s from a %s? "
                                            % (animal, p.question))
        p.left     = node(p.question)
        p.right    = node(animal)
        p.question = question

        if not yes ("If the animal were %s the answer would be? (y/n)" % animal) :
            (p.right, p.left) = (p.left, p.right)
```

```python
# Interactive Active Learning Example
# Change suppress to False and run this cell to see demonstration
active_learning_example(suppress=True)
```

### Generalizations of Decision Trees (1)
- Decision Trees in their simplest form involve:
 - A 2-class scenario, with,
 - Binary Features, and use a,
 - Binary Tree (every node has at most 2 children).

However, generalizations are possible.

### Generalizations of Decision Trees (2)
- Categorical Features
 - Can use $n$-ary trees rather than binary search trees
   - Fruits Tree Example: Color? asked at the root had three choices: Red, Yellow or Green (Ternary Tree)
 - Can use questions such as "$x_i$ = $l$?" or even $"2 \leq x_i \leq l$?", where $x_i \in \{1, ..., l, ..., K\}$?
   - The Mushroom Classification Tree was an example of this (Binary Tree with comparative rather than only equality checking conditions).

### Generalizations of Decision Trees (3)
- Categorical Output/Multiclass Scenario (the Fruits Example)
 - One way to handle categorical outputs is to pose a question such as "Is it a bird?" when "Yes" is expected. This keeps the outputs binary.
    Moreover, if the answer is no, one can then use active learning as shown before to improve the tree.
 - Another way is to simply use a one-hot encoding for the output (Ex: Bird corresponds to $[0, 0, 0, 0, 0, 1, 0]$)
- Real-Valued Response (Output) - Owing to the nature of trees, regression usually involves discretizing the output space either in some way. Once discretized, everything is the same.

### Inductive Bias

- What is the policy by which a particular decision tree algorithm generalizes from observed training examples to classify unseen instances?

- ***Definition:*** The set of assumptions that, together with the training data, deductively justify the classifications assigned by the learner to future instances.

- We can also think of this bias as an algorithms preference.

[More here](https://en.wikipedia.org/wiki/Inductive_bias)

### Inductive Bias of Decision Tree Algorithms

- When a decision tree is built, it is almost always not the only tree that will perfectly classify the training set!
- Finding the inductive bias of a decision tree algorithm is basically trying to find the type of tree that the algorithm favors in general.
- It turns out that two of the common decision tree algorithms (ID3 and C4.5) have the same approximate inductive bias:
 - Prefers shorter trees over larger trees, and,
 - Trees that place high information gain attributes close to the root over those that do not.

### Occam's Razor

- If a larger tree classifies the training set just as well as the shorter hypothesis, then we would logically want to use the shorter tree as it performs the same, is smaller, and is quicker to build.
- But is it always the case that shorter, more simpler hypotheses are preferred over larger ones?
- This is a question that has been debated for centuries, and the answer still remains unclear.
- This bias of preferring the simplest hypothesis that fits the data is known as Occam's razor. That is, "the simpler the better."

### Decision Tree Limitations (1)

Decision Trees in general perform well with lots of data, are robust to violations of assumptions, and probably most strikingly are easy to understand and interpret. However:
 - The problem of Learning an optimal Decision Trees is NP-Complete under several definitions of optimal.
  - Standard algs are "greedy", make myopic decisions that may not be globally optimal.
 - There are concepts that are hard to learn using Decision Trees, which are also generally hard for other linear classifiers, as the Decision Tree learned is prohibitively large. These include "toy" problems such as:
  - XOR, Parity or Multiplexer based Problems


# Maximum Mean Discrepancy drift detector on CIFAR-10

### Method

The [Maximum Mean Discrepancy (MMD)](http://jmlr.csail.mit.edu/papers/v13/gretton12a.html) detector is a kernel-based method for multivariate 2 sample testing. The MMD is a distance-based measure between 2 distributions *p* and *q* based on the mean embeddings $\mu_{p}$ and $\mu_{q}$ in a reproducing kernel Hilbert space $F$:

\begin{align}
MMD(F, p, q) & = || \mu_{p} - \mu_{q} ||^2_{F} \\
\end{align}

We can compute unbiased estimates of $MMD^2$ from the samples of the 2 distributions after applying the kernel trick. We use by default a [radial basis function kernel](https://en.wikipedia.org/wiki/Radial_basis_function_kernel), but users are free to pass their own kernel of preference to the detector. We obtain a $p$-value via a [permutation test](https://en.wikipedia.org/wiki/Resampling_(statistics)) on the values of $MMD^2$. This method is also described in [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://arxiv.org/abs/1810.11953).

### Backend

The method is implemented in both the *PyTorch* and *TensorFlow* frameworks with support for CPU and GPU. Various preprocessing steps are also supported out-of-the box in Alibi Detect for both frameworks and illustrated throughout the notebook. Alibi Detect does however not install PyTorch for you. Check the [PyTorch docs](https://pytorch.org/) how to do this.

### Dataset

[CIFAR10](https://www.cs.toronto.edu/~kriz/cifar.html) consists of 60,000 32 by 32 RGB images equally distributed over 10 classes. We evaluate the drift detector on the CIFAR-10-C dataset ([Hendrycks & Dietterich, 2019](https://arxiv.org/abs/1903.12261)). The instances in
CIFAR-10-C have been corrupted and perturbed by various types of noise, blur, brightness etc. at different levels of severity, leading to a gradual decline in the classification model performance. We also check for drift against the original test set with class imbalances.

```python
from functools import partial
import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf

from alibi_detect.cd import MMDDrift
from alibi_detect.models.tensorflow.resnet import scale_by_instance
from alibi_detect.utils.fetching import fetch_tf_model
from alibi_detect.utils.saving import save_detector, load_detector
from alibi_detect.datasets import fetch_cifar10c, corruption_types_cifar10c
```

### Load data

Original CIFAR-10 data:

```python
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.cifar10.load_data()
X_train = X_train.astype('float32') / 255
X_test = X_test.astype('float32') / 255
y_train = y_train.astype('int64').reshape(-1,)
y_test = y_test.astype('int64').reshape(-1,)
```

For CIFAR-10-C, we can select from the following corruption types at 5 severity levels:

```python
corruptions = corruption_types_cifar10c()
print(corruptions)
```

    ['brightness', 'contrast', 'defocus_blur', 'elastic_transform', 'fog', 'frost', 'gaussian_blur', 'gaussian_noise', 'glass_blur', 'impulse_noise', 'jpeg_compression', 'motion_blur', 'pixelate', 'saturate', 'shot_noise', 'snow', 'spatter', 'speckle_noise', 'zoom_blur']

Let's pick a subset of the corruptions at corruption level 5. Each corruption type consists of perturbations on all of the original test set images.

```python
corruption = ['gaussian_noise', 'motion_blur', 'brightness', 'pixelate']
X_corr, y_corr = fetch_cifar10c(corruption=corruption, severity=5, return_X_y=True)
X_corr = X_corr.astype('float32') / 255
```

We split the original test set in a reference dataset and a dataset which should not be rejected under the *H0* of the MMD test. We also split the corrupted data by corruption type:

```python
np.random.seed(0)
n_test = X_test.shape[0]
idx = np.random.choice(n_test, size=n_test // 2, replace=False)
idx_h0 = np.delete(np.arange(n_test), idx, axis=0)
X_ref,y_ref = X_test[idx], y_test[idx]
X_h0, y_h0 = X_test[idx_h0], y_test[idx_h0]
print(X_ref.shape, X_h0.shape)
```

    (5000, 32, 32, 3) (5000, 32, 32, 3)

```python
# check that the classes are more or less balanced
classes, counts_ref = np.unique(y_ref, return_counts=True)
counts_h0 = np.unique(y_h0, return_counts=True)[1]
print('Class Ref H0')
for cl, cref, ch0 in zip(classes, counts_ref, counts_h0):
    assert cref + ch0 == n_test // 10
    print('{}     {} {}'.format(cl, cref, ch0))
```

    Class Ref H0
    0     472 528
    1     510 490
    2     498 502
    3     492 508
    4     501 499
    5     495 505
    6     493 507
    7     501 499
    8     516 484
    9     522 478

```python
n_corr = len(corruption)
X_c = [X_corr[i * n_test:(i + 1) * n_test] for i in range(n_corr)]
```

We can visualise the same instance for each corruption type:

```python
i = 4

n_test = X_test.shape[0]
plt.title('Original')
plt.axis('off')
plt.imshow(X_test[i])
plt.show()
for _ in range(len(corruption)):
    plt.title(corruption[_])
    plt.axis('off')
    plt.imshow(X_corr[n_test * _+ i])
    plt.show()
```

We can also verify that the performance of a classification model on CIFAR-10 drops significantly on this perturbed dataset:

```python
dataset = 'cifar10'
model = 'resnet32'
clf = fetch_tf_model(dataset, model)
acc = clf.evaluate(scale_by_instance(X_test), y_test, batch_size=128, verbose=0)[1]
print('Test set accuracy:')
print('Original {:.4f}'.format(acc))
clf_accuracy = {'original': acc}
for _ in range(len(corruption)):
    acc = clf.evaluate(scale_by_instance(X_c[_]), y_test, batch_size=128, verbose=0)[1]
    clf_accuracy[corruption[_]] = acc
    print('{} {:.4f}'.format(corruption[_], acc))
```

    Test set accuracy:
    Original 0.9278
    gaussian_noise 0.2208
    motion_blur 0.6339
    brightness 0.8913
    pixelate 0.3666

Given the drop in performance, it is important that we detect the harmful data drift!

### Detect drift with TensorFlow backend

First we try a drift detector using the *TensorFlow* framework for both the preprocessing and the *MMD* computation steps.

We are trying to detect data drift on high-dimensional (*32x32x3*) data using a multivariate MMD permutation test. It therefore makes sense to apply dimensionality reduction first. Some dimensionality reduction methods also used in [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://arxiv.org/pdf/1810.11953.pdf) are readily available: a randomly initialized encoder (**UAE** or Untrained AutoEncoder in the paper), **BBSDs** (black-box shift detection using the classifier's softmax outputs) and **PCA** (using `scikit-learn`).

#### Random encoder

First we try the randomly initialized encoder:

```python
from tensorflow.keras.layers import Conv2D, Dense, Flatten, InputLayer, Reshape
from alibi_detect.cd.tensorflow import preprocess_drift

tf.random.set_seed(0)

# define encoder
encoding_dim = 32
encoder_net = tf.keras.Sequential(
  [
      InputLayer(input_shape=(32, 32, 3)),
      Conv2D(64, 4, strides=2, padding='same', activation=tf.nn.relu),
      Conv2D(128, 4, strides=2, padding='same', activation=tf.nn.relu),
      Conv2D(512, 4, strides=2, padding='same', activation=tf.nn.relu),
      Flatten(),
      Dense(encoding_dim,)
  ]
)

# define preprocessing function
preprocess_fn = partial(preprocess_drift, model=encoder_net, batch_size=512)

# initialise drift detector
cd = MMDDrift(X_ref, backend='tensorflow', p_val=.05,
              preprocess_fn=preprocess_fn, n_permutations=100)

# we can also save/load an initialised detector
filepath = 'my_path'  # change to directory where detector is saved
save_detector(cd, filepath)
cd = load_detector(filepath)
```

    WARNING:tensorflow:No training configuration found in save file: the model was *not* compiled. Compile it manually.

    WARNING:tensorflow:No training configuration found in save file: the model was *not* compiled. Compile it manually.
    WARNING:alibi_detect.cd.base:`sigma` is specified for the kernel and `configure_kernel_from_x_ref` is set to True. `sigma` argument takes priority over `configure_kernel_from_x_ref` (set to False).

Let's check whether the detector thinks drift occurred on the different test sets and time the prediction calls:

```python
from timeit import default_timer as timer

labels = ['No!', 'Yes!']

def make_predictions(cd, x_h0, x_corr, corruption):
    t = timer()
    preds = cd.predict(x_h0)
    dt = timer() - t
    print('No corruption')
    print('Drift? {}'.format(labels[preds['data']['is_drift']]))
    print(f'p-value: {preds["data"]["p_val"]:.3f}')
    print(f'Time (s) {dt:.3f}')

    if isinstance(x_corr, list):
        for x, c in zip(x_corr, corruption):
            t = timer()
            preds = cd.predict(x)
            dt = timer() - t
            print('')
            print(f'Corruption type: {c}')
            print('Drift? {}'.format(labels[preds['data']['is_drift']]))
            print(f'p-value: {preds["data"]["p_val"]:.3f}')
            print(f'Time (s) {dt:.3f}')
```

```python
make_predictions(cd, X_h0, X_c, corruption)
```

    No corruption
    Drift? No!
    p-value: 0.680
    Time (s) 2.217

    Corruption type: gaussian_noise
    Drift? Yes!
    p-value: 0.000
    Time (s) 6.074

    Corruption type: motion_blur
    Drift? Yes!
    p-value: 0.000
    Time (s) 6.031

    Corruption type: brightness
    Drift? Yes!
    p-value: 0.000
    Time (s) 6.019

    Corruption type: pixelate
    Drift? Yes!
    p-value: 0.000
    Time (s) 6.010

As expected, drift was only detected on the corrupted datasets.

#### BBSDs

For **BBSDs**, we use the classifier's softmax outputs for black-box shift detection. This method is based on [Detecting and Correcting for Label Shift with Black Box Predictors](https://arxiv.org/abs/1802.03916). The ResNet classifier is trained on data standardised by instance so we need to rescale the data.

```python
X_ref_bbsds = scale_by_instance(X_ref)
X_h0_bbsds = scale_by_instance(X_h0)
X_c_bbsds = [scale_by_instance(X_c[i]) for i in range(n_corr)]
```

Initialisation of the drift detector. Here we use the output of the softmax layer to detect the drift, but other hidden layers can be extracted as well by setting *'layer'* to the index of the desired hidden layer in the model:

```python
from alibi_detect.cd.tensorflow import HiddenOutput

# define preprocessing function
preprocess_fn = partial(preprocess_drift, model=HiddenOutput(clf, layer=-1), batch_size=128)

# initialise drift detector
cd = MMDDrift(X_ref_bbsds, backend='tensorflow', p_val=.05,
              preprocess_fn=preprocess_fn, n_permutations=100)
```

```python
make_predictions(cd, X_h0_bbsds, X_c_bbsds, corruption)
```

    No corruption
    Drift? No!
    p-value: 0.440
    Time (s) 3.072

    Corruption type: gaussian_noise
    Drift? Yes!
    p-value: 0.000
    Time (s) 7.701

    Corruption type: motion_blur
    Drift? Yes!
    p-value: 0.000
    Time (s) 7.754

    Corruption type: brightness
    Drift? Yes!
    p-value: 0.000
    Time (s) 7.760

    Corruption type: pixelate
    Drift? Yes!
    p-value: 0.000
    Time (s) 7.732

Again drift is only flagged on the perturbed data.

### Detect drift with PyTorch backend

We can do the same thing using the *PyTorch* backend. We illustrate this using the randomly initialized encoder as preprocessing step:

```python
import torch
import torch.nn as nn

# set random seed and device
seed = 0
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(device)
```

    cuda

Since our *PyTorch* encoder expects the images in a *(batch size, channels, height, width)* format, we transpose the data:

```python
def permute_c(x):
    return np.transpose(x.astype(np.float32), (0, 3, 1, 2))

X_ref_pt = permute_c(X_ref)
X_h0_pt = permute_c(X_h0)
X_c_pt = [permute_c(xc) for xc in X_c]
print(X_ref_pt.shape, X_h0_pt.shape, X_c_pt[0].shape)
```

    (5000, 3, 32, 32) (5000, 3, 32, 32) (10000, 3, 32, 32)

```python
from alibi_detect.cd.pytorch import preprocess_drift

# define encoder
encoder_net = nn.Sequential(
    nn.Conv2d(3, 64, 4, stride=2, padding=0),
    nn.ReLU(),
    nn.Conv2d(64, 128, 4, stride=2, padding=0),
    nn.ReLU(),
    nn.Conv2d(128, 512, 4, stride=2, padding=0),
    nn.ReLU(),
    nn.Flatten(),
    nn.Linear(2048, encoding_dim)
).to(device).eval()

# define preprocessing function
preprocess_fn = partial(preprocess_drift, model=encoder_net, device=device, batch_size=512)

# initialise drift detector
cd = MMDDrift(X_ref_pt, backend='pytorch', p_val=.05,
              preprocess_fn=preprocess_fn, n_permutations=100)
```

```python
make_predictions(cd, X_h0_pt, X_c_pt, corruption)
```

    No corruption
    Drift? No!
    p-value: 0.730
    Time (s) 0.478

    Corruption type: gaussian_noise
    Drift? Yes!
    p-value: 0.000
    Time (s) 1.104

    Corruption type: motion_blur
    Drift? Yes!
    p-value: 0.000
    Time (s) 1.066

    Corruption type: brightness
    Drift? Yes!
    p-value: 0.000
    Time (s) 1.065

    Corruption type: pixelate
    Drift? Yes!
    p-value: 0.000
    Time (s) 1.066

The drift detector will attempt to use the GPU if available and otherwise falls back on the CPU. We can also explicitly specify the device. Let's compare the GPU speed up with the CPU implementation:

```python
device = torch.device('cpu')
preprocess_fn = partial(preprocess_drift, model=encoder_net.to(device),
                        device=device, batch_size=512)

cd = MMDDrift(X_ref_pt, backend='pytorch', preprocess_fn=preprocess_fn, device='cpu')
```

```python
make_predictions(cd, X_h0_pt, X_c_pt, corruption)
```

    No corruption
    Drift? No!
    p-value: 0.670
    Time (s) 14.282

    Corruption type: gaussian_noise
    Drift? Yes!
    p-value: 0.000
    Time (s) 32.061

    Corruption type: motion_blur
    Drift? Yes!
    p-value: 0.000
    Time (s) 32.060

    Corruption type: brightness
    Drift? Yes!
    p-value: 0.000
    Time (s) 32.459

    Corruption type: pixelate
    Drift? Yes!
    p-value: 0.000
    Time (s) 35.935

Notice the over **30x acceleration** provided by the GPU.

Similar to the *TensorFlow* implementation, *PyTorch* can also use the hidden layer output from a pretrained model for the preprocessing step via:

```python
from alibi_detect.cd.pytorch import HiddenOutput
```


# Applying BlueFog on Deep Learning problem(High Level API Introduction)

All previous sections we focused on the low-level API in the BlueFog, which is great for flexible algorithm design and research. Give a quick summary here:

- Basic static topology propertis and how to manipulate it
- Collective communication such as broadcast and allreduce
- Topology based Neighborhood communication such as neighbor_allreduce.
- Blocking versus non-blocking operation
- Dynamic topology and its corresponding
- Asynchronous operation through window object
- Several decentralized algorithms and their performance under different scenarios
- etc.

However, it can be boilerplate if you want to apply one certain algorithm on different tasks. Further, it is also tricky to write a efficiency code combining above mentioned concepts. This becomes even worse in the deep learning problem. Backpropagation property of neural network makes that the gradient can be efficiently calculated. Backpropagation also implied that the gradient is calculated (approximately) in layer-by-layer style, in contrast to one global stochastic (sub-)gradient we encountered in the optimization. Further, this layer-wise computation provides a great opportunity to overlap the communication and comptuation for minimizing the trainning time, which is a crucial. Clearly, writing the code to address these concern correctly and efficient is not easy. Hence, BlueFog further provides the high level APIs, which can be directly applied on the `torch.Optimizer` directly.

In this section, we will focus in applying high level APIs of BlueFog on Deep Learning problem, mainly the decentralized trainning task.
Before we demystify how we implement the High-Level API in BlueFog, let's see the example of using them to write distributed trainning of ResNet-18 model over CIFAR-10 easily. (*Note, although this example is relative small, it still can be time-consuming and drain tons of computation resources if you want to train them on CPUs.*)

```
import os
import time

import ipyparallel as ipp
import networkx as nx
import numpy as np
import matplotlib.pyplot as plt
import torch
from torchvision import datasets, transforms, models

%matplotlib inline
```

```
rc = ipp.Client(profile="bluefog")
rc.block = True
rc.ids
```

    [0, 1, 2, 3]

## Prepared the data

```
# Down the CIFAR10 Dataset if not available
# Since the dataset is smaller enough, we just load it in-memory.
train_dataset = datasets.CIFAR10(
    os.path.join(os.getcwd(), "..", "data"),
    train=True,
    download=True,
    transform=transforms.Compose(
        [
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
        ]
    ),
)

val_dataset = datasets.CIFAR10(
    os.path.join(os.getcwd(), "..", "data"),
    train=False,
    transform=transforms.Compose(
        [
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
        ]
    ),
)
```

    Files already downloaded and verified

```
# Distribute the data into each worker.
# Note we push the full dataset into each worker is just for simplicity.
# Each worker only read the partial of dataset later.
_ = rc[:].push({"train_dataset": train_dataset, "val_dataset": val_dataset})
```

```
%%px
import math
import torch
import bluefog.torch as bf
from bluefog.common import topology_util

seed = 2021
bf.init()
torch.manual_seed(seed)
run_on_cuda = torch.cuda.is_available()
if run_on_cuda:
    print("using cuda.")
    # Bluefog: pin GPU to local rank.
    device_id = (
        bf.local_rank()
        if bf.nccl_built()
        else bf.local_rank() % torch.cuda.device_count()
    )
    torch.cuda.set_device(device_id)
    torch.cuda.manual_seed(seed)
else:
    print("using cpu")
```

    [stdout:0] using cuda.
    [stdout:1] using cuda.
    [stdout:2] using cuda.
    [stdout:3] using cuda.

```
%%px
print(torch.__file__)
```

    [stdout:0] /home/kun/anaconda3/envs/bluefog/lib/python3.7/site-packages/torch/__init__.py
    [stdout:1] /home/kun/anaconda3/envs/bluefog/lib/python3.7/site-packages/torch/__init__.py
    [stdout:2] /home/kun/anaconda3/envs/bluefog/lib/python3.7/site-packages/torch/__init__.py
    [stdout:3] /home/kun/anaconda3/envs/bluefog/lib/python3.7/site-packages/torch/__init__.py

```
%%px
# Prepare the distributed loader for dataset.
batch_size = 32
val_batch_size = 1024
kwargs = {"num_workers": 4, "pin_memory": True} if run_on_cuda else {}

train_sampler = torch.utils.data.distributed.DistributedSampler(
    train_dataset, num_replicas=bf.size(), rank=bf.rank()
)
train_loader = torch.utils.data.DataLoader(
    train_dataset, batch_size=batch_size, sampler=train_sampler, **kwargs
)

val_sampler = torch.utils.data.distributed.DistributedSampler(
    val_dataset, num_replicas=bf.size(), rank=bf.rank()
)
val_loader = torch.utils.data.DataLoader(
    val_dataset, batch_size=val_batch_size, sampler=val_sampler, **kwargs
)
```

## Set up standard torch optimizer

```
%%px
from torchvision import models

model = models.resnet18(num_classes=10)
if run_on_cuda:
    model.cuda()

# Scale learning rate by the number of GPUs.
base_lr = 0.0125
momentum = 0.9
weight_decay = 0.00005

optimizer = torch.optim.SGD(
    model.parameters(),
    lr=(base_lr * bf.size()),
    momentum=momentum,
    weight_decay=weight_decay,
)
```

## Wrap the torch standard optimizer into BlueFog distributed one

```
%%px
atc_style = False
base_dist_optimizer = (
    bf.DistributedAdaptThenCombineOptimizer
    if atc_style
    else bf.DistributedAdaptWithCombineOptimizer
)
optimizer = base_dist_optimizer(
    optimizer, model=model, communication_type=bf.CommunicationType.neighbor_allreduce
)

# Bluefog: broadcast parameters & optimizer state.
bf.broadcast_parameters(model.state_dict(), root_rank=0)
bf.broadcast_optimizer_state(optimizer, root_rank=0)
```

## Define several helper functions for training and validation

```
%%px
def accuracy(output, target):
    # get the index of the max log-probability
    pred = output.max(1, keepdim=True)[1]
    return pred.eq(target.view_as(pred)).cpu().float().mean()

class Metric(object):
    def __init__(self, name):
        self.name = name
        self.sum = torch.tensor(0.0)  # pylint: disable=not-callable
        self.n = torch.tensor(0.0)  # pylint: disable=not-callable

    def update(self, val):
        self.sum += bf.allreduce(val.detach().cpu(), name=self.name)
        self.n += 1

    @property
    def avg(self):
        return self.sum / self.n

dynamic_neighbor_allreduce_gen = topology_util.GetDynamicOnePeerSendRecvRanks(
    bf.load_topology(), bf.rank()
)

def dynamic_topology_update(epoch, batch_idx):
    send_neighbors, recv_neighbors = next(dynamic_neighbor_allreduce_gen)
    optimizer.send_neighbors = send_neighbors
    optimizer.neighbor_weights = {
        r: 1 / (len(recv_neighbors) + 1) for r in recv_neighbors
    }
    optimizer.self_weight = 1 / (len(recv_neighbors) + 1)

def adjust_learning_rate(epoch, batch_idx):
    if epoch < 5:  # warmup_epochs
        epoch += float(batch_idx + 1) / len(train_loader)
        lr_adj = 1.0 / bf.size() * (epoch * (bf.size() - 1) / 5 + 1)
    elif epoch < 30:
        lr_adj = 1.0
    elif epoch < 60:
        lr_adj = 1e-1
    elif epoch < 80:
        lr_adj = 1e-2
    else:
        lr_adj = 1e-3
    for param_group in optimizer.param_groups:
        param_group["lr"] = base_lr * bf.size() * lr_adj
```

```
%%px
# publish_data is useful for monitoring the status of workers
from ipyparallel.datapub import publish_data

def train(epoch):
    model.train()
    train_sampler.set_epoch(epoch)
    train_loss = Metric("train_loss")
    train_accuracy = Metric("train_accuracy")

    for batch_idx, (data, target) in enumerate(train_loader):
        adjust_learning_rate(epoch, batch_idx)
        dynamic_topology_update(epoch, batch_idx)

        if run_on_cuda:
            data, target = data.cuda(), target.cuda()
        optimizer.zero_grad()
        output = model(data)
        train_accuracy.update(accuracy(output, target))
        loss = torch.nn.functional.cross_entropy(output, target)
        train_loss.update(loss)
        # Average gradients among sub-batches
        loss.div_(math.ceil(float(len(data)) / batch_size))
        loss.backward()
        # Gradient is applied across all ranks
        optimizer.step()
        publish_data(
            dict(
                batch_idx=batch_idx,
                loss=train_loss.avg.item(),
                accuracy=100.0 * train_accuracy.avg.item(),
            )
        )

def validate(epoch):
    model.eval()
    val_loss = Metric("val_loss")
    val_accuracy = Metric("val_accuracy")

    with torch.no_grad():
        for batch_idx, (data, target) in enumerate(val_loader):
            if run_on_cuda:
                data, target = data.cuda(), target.cuda()
            output = model(data)

            val_loss.update(torch.nn.functional.cross_entropy(output, target))
            val_accuracy.update(accuracy(output, target))
            publish_data(
                dict(
                    batch_idx=batch_idx,
                    loss=val_loss.avg.item(),
                    accuracy=100.0 * val_accuracy.avg.item(),
                )
            )
```

### A fake example of monitoring the status of workers

The trainning of deep learning model can be time consuming. In order to monitor the
status of workers, you can use `ipyparallel.datapub.publish_data` to push the data
of the workers to the notebook. Following is a simple fake example to help you understand
how to use publish_data and [tqdm](https://tqdm.github.io/) library.

```
%%px
# publish_data is useful for monitoring the status of workers
import time
from ipyparallel.datapub import publish_data
import numpy as np

def fake_training():
    for i in range(n_iter):
        loss, accuracy = np.random.rand(2)
        publish_data(dict(i=i, loss=loss, accuracy=accuracy))
        time.sleep(0.5)
```

```
from tqdm import tqdm

dview = rc[:]
dview.block = True
n_iter = 50
dview["n_iter"] = n_iter
ar = dview.apply_async(lambda: fake_training())
epoch = 1

with tqdm(total=n_iter - 1, desc="Train Epoch     #{}".format(epoch + 1)) as t:
    while not ar.ready():
        data = ar.data[0]
        t.set_postfix(
            {
                "loss": data.get("loss", 0),
                "accuracy": f"{data.get('accuracy', 0)*100:.2f}%",
            }
        )
        t.update(data.get("i", 0) - t.n)
        time.sleep(0.5)
```

    Train Epoch     #2: 100%|██████████| 49/49 [00:25<00:00,  1.95it/s, loss=0.846, accuracy=84.98%]

## Start decentralized trainning

```
import time

# from tqdm.notebook import tqdm
from tqdm import tqdm

epochs = 2
for epoch in range(epochs):
    rc[:].push({"epoch": epochs}, block=True)

    # Training
    with tqdm(
        total=len(rc[0]["train_loader"]) - 1,
        desc="Train Epoch     #{}".format(epoch + 1),
    ) as t:
        ar_train = rc[:].apply_async(lambda: train(epoch))
        while not ar_train.ready() or t.n < t.total:
            data = ar_train.data[0]
            t.set_postfix(
                {
                    "loss": data.get("loss", 0),
                    "accuracy": f"{data.get('accuracy', 0):.2f}%",
                }
            )
            t.update(data.get("batch_idx", 0) - t.n)
            time.sleep(0.5)
    # Validation
    with tqdm(
        total=len(rc[0]["val_loader"]) - 1, desc="Validate Epoch  #{}".format(epoch + 1)
    ) as t:
        ar_val = rc[:].apply_async(lambda: validate(epoch))
        while not ar_val.ready() or t.n < t.total:
            data = ar_val.data[0]
            t.set_postfix(
                {
                    "loss": data.get("loss", 0),
                    "accuracy": f"{data.get('accuracy', 0):.2f}%",
                }
            )
            t.update(data.get("batch_idx", 0) - t.n)
            time.sleep(0.2)
```

    Train Epoch     #1: 100%|██████████| 390/390 [00:32<00:00, 12.12it/s, loss=1.72, accuracy=41.46%]
    Validate Epoch  #1: 100%|██████████| 2/2 [00:01<00:00,  1.96it/s, loss=1.51, accuracy=52.60%]
    Train Epoch     #2: 100%|██████████| 390/390 [00:32<00:00, 12.12it/s, loss=1.23, accuracy=57.31%]
    Validate Epoch  #2: 100%|██████████| 2/2 [00:00<00:00,  2.40it/s, loss=1.75, accuracy=58.47%]

# Demystify the BlueFog High Level APIs

Here we show more details about two common BlueFog high level APIs -- `DistributedAdaptWithCombineOptimizer` and `DistributedAdaptThenCombineOptimizer`. They are corresponding to ATC and AWC algorithm we introduced before. After understanding how these works, you should be able to quickly grasp main idea of other high level APIs or even build your own high level APIs with different decentralized algorithm.

From the algorithm part, there is nothing new happened under the these two distributed optimizer. But we update our notation slightly since in neural network it is common to denote the gradient layer by layer:

\begin{align}
    grad_{l}(w_{k,i})
     =& \frac{\partial f_k(w_{k,i})}{\partial w_{k,i}[l]} \\
     =& \frac{\partial f_k(w_{k,i})}{\partial w_{k,i}[l+1]}\frac{\partial w_{k,i}[l+1]}{\partial w_{k,i}[l]} \\
     =& grad_{l+1}(w_{k,i}) \frac{\partial w_{k,i}[l+1]}{\partial w_{k,i}[l]}
\end{align}
where we use $l$ for $l-$ layer and $[l]$ means the parameter related to layer $l$. In this notation, it is closer to reflect how torch works. Except that, the AWC and ATC are applied in the same way. However, in the engineering part,
it is not the same as we did before. The main difference is that we no longer wait for the gradient computation of whole model finished to start the communication. Instead, we try to maximize the communication and computation through layer-wise style.

First, let's review the typical deep learning training under Torch framework:
```python
loss = metric(model(data), target)
loss.backward()  <--- # it starts to compute the gradient
optimizer.step() <--- # it applies the computed gradient on the weights
```
The `backwards()` function will inversely traverse the neural networks and compute the gradients of each components in the model. The earliest time of executing communication is either at the *forward* or the *backward* step depending on the communication vector required gradient information or not. The latest time of finishing communication is at the `step()` step.
A typical communication and computation timeline for Adapt-With-Combine optimizer and Adapt-Then-Combine optimizer is shown in the following figure:

To achieve that the key mechanism is `hook` pattern or so-called [Template method pattern](https://en.wikipedia.org/wiki/Template_method_pattern). For those who are not familar with this design pattern, we provide a simple example:

```
class TriviaHookPattern:
    def __init__(self):
        pass

    def register_pre_hook(self, f):
        self._pre_hook = f

    def register_post_hook(self, f):
        self._post_hook = f

    def execute(self):
        self._pre_hook()
        print("Execute some function")
        self._post_hook()

    def _pre_hook(self):
        raise NotImplementedError

    def _post_hook(self):
        raise NotImplementedError
```

```
h = TriviaHookPattern()
h.register_pre_hook(lambda: print("I am pre hook"))
h.register_post_hook(lambda: print("I am post hook"))
h.execute()
```

    I am pre hook
    Execute some function
    I am post hook

There are three types of hooks provided by torch APIs.

- `register_forward_hook` for one `torch.nn` module
- `register_backward_hook` for one `torch.nn` module
- `register_hook()` in gradient computation of one `torch.Tensor`

We implement distributed optimizers with some proper combination of these hooks. We would not list the full source code due to too many engineering details. But we demonstrate a one-layer neural network example to illustrate how to combine it with bluefog non-blocking communication operations.

```
%%px
import torch.nn as nn

class LinearNet(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(LinearNet, self).__init__()
        self.fc = nn.Linear(input_dim, output_dim)
        self.num_parameters = input_dim * output_dim

    def forward(self, x):
        return self.fc(x)
```

```
%%px
# register for communication hook for model
data, target = torch.rand(5), torch.rand(1)
model = LinearNet(5, 1)

handles = []

def make_hook(model):
    def hook(model, *unused):
        for name, parameter in model.named_parameters():
            print("Make neighbor_allreduce for ", name)
            handle = bf.neighbor_allreduce_nonblocking(parameter.data)
            handles.append(handle)

    return hook

hook = make_hook(model)
_ = model.register_forward_hook(hook)
```

```
%%px
# Start nonblocking communication at first time.
# Wait until it finished at last step.
y = model(data)
loss = nn.MSELoss()(y, target)
loss.backward()
print(handles)
tensors = [bf.wait(handle) for handle in handles]
handles.clear()
print(tensors)
```

    [stdout:0]
    Make neighbor_allreduce for  fc.weight
    Make neighbor_allreduce for  fc.bias
    [50250, 50251]
    [tensor([[ 0.0330, -0.3186,  0.2665,  0.2265, -0.3765]]), tensor([0.3806])]
    [stdout:1]
    Make neighbor_allreduce for  fc.weight
    Make neighbor_allreduce for  fc.bias
    [50250, 50251]
    [tensor([[ 0.0330, -0.3186,  0.2665,  0.2265, -0.3765]]), tensor([0.3806])]
    [stdout:2]
    Make neighbor_allreduce for  fc.weight
    Make neighbor_allreduce for  fc.bias
    [50250, 50251]
    [tensor([[ 0.0330, -0.3186,  0.2665,  0.2265, -0.3765]]), tensor([0.3806])]
    [stdout:3]
    Make neighbor_allreduce for  fc.weight
    Make neighbor_allreduce for  fc.bias
    [50250, 50251]
    [tensor([[ 0.0330, -0.3186,  0.2665,  0.2265, -0.3765]]), tensor([0.3806])]

Section 7/Applying BlueFog on Deep Learning problem(High Level API Introduction).ipynb


# Vibration Reduction of Flexible Beam with Bayesian Input shaper

## Zsolt Pásztori
## CERN/ SMM-MRO-Robotics

```python
#Imports
from IPython.core.display import Image, display
from IPython.core.display import HTML

import numpy as np
from numpy import genfromtxt
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
plt.rcParams["figure.figsize"] = (20,10)

from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import Matern, RBF

import json
import matplotlib.animation as animation
from matplotlib.animation import FuncAnimation
%matplotlib inline
#
```

## Introduction

I work in the CERN Robotic Team as a PhD student. One of the experiments has asked us to design a mobile robot for inspection of their facilities.
The robot has to fit 2 main criteria:

* has to be able to pass under 20 cm obstacles
* must be able to perform visual inspection between 20 cm and 1.5 meters

To ensure these requirements are met, we decided to fit a pan-tilt camera on top of a flexible beam.
The beam can be raised with a motor up to 2 meters high. It can also be lowered to the base of the robot.

The beam has a low weight so it can be moved quickly.
This however means that the beam is not stiff. It vibrates when the robot is moving. This results in poor video feed.
The vibration is induced by the linear acceleration of the robot and the unevenness of the surface.

Bellow you can see the video taken on top of the flexible beam.
It can be observed that the oscillation of the camerea, is mainly caused by the initial acceleration of the robot.
It dies down very quickly when it is moving with constant velocity tough.

```python
HTML('')
```

    /home/robotronics/.local/lib/python3.5/site-packages/IPython/core/display.py:694: UserWarning: Consider using IPython.display.IFrame instead
      warnings.warn("Consider using IPython.display.IFrame instead")

## Brain Storming of Solutions

To get rid of the vibration of the camera, I can see 4 possible solutions.

### 1. Drive slowly
* limits the movement of the robot
* no need to implement anything

### 2. Balance with accelerating/breaking the robot
* needs a sensor to give feedback of the beam position
* needs perfect control of the robotic base
* the beam movement must be modeled
* no need for additional actuator
* easy to implement the hardware, medium hard software

### 3. Corrigate the video camera with a gimbal
* needs a sensor to give feedback of the beam position
* needs 2 additional gimbal motors
* an additonal software needed for the control of the gimbal
* no need to change the code of the robot base
* medium hard hardware, medium hard software

### 4. Use reaction wheels to stabilize the top of the beam
* needs a sensor to give feedback of the beam position
* needs 2 custom machined reaction wheels (very expensive)
* needs 2 additional motors to rotate the wheels
* complicated control to stabilize the beam
* the high weight makes the beam movement slower
* no need to change the code of the robot base
* very hard hardware, very hard software

I decided to go with solution number 2. The main reason is that it doesn't require additional hardware components, just a simple sensor.

In my experience in a system the bigger the complexity of the hardware the higher the cahnce to fail during extended operation.

## Experiment Set UP

### The Hardware

The robot base is equipped with 4 Maxon EC motors. The motors are controlled with the EPOS2 servo controller.
The brain of the robot is a commercially available NUC computer with Ubuntu on it.
To get the position feedback from the beam I have placed a VMU931 IMU sensor on the top.
The sensor has built in accelerometer, gyroscope and magnetometer.
I only used the gyroscope measurements, because they have lower noise compared to the accelerometer.
Since we do not yet have the camera, I have placed 420 g weight (biggest wrench at hand) on top of the beam to emulate it.

### The Software

The control of the robot is handled by the CERN Robotic Framework (CRF).
This is a framework developed by the members of the CERN Robotics team.
It is written in C++, runs on Ubuntu.

For my application I needed to be able to control the movement of the robot precisely.
I have modified our interface to the motor drivers to be able to set both the velocity and acceleration of the motors.
Reviewed the control loop of the 4 robots. Originally they needed on average 20 ms to change the state of the 4 robots.
I managed to optimize the code, and cut it down to an average of 6 ms. Additionally the motor movements are synchronized to a 2 ms window. This makes the robot movement much smoother, and also there is less torque ripple between the 4 motors.

The motors of the robot are controlled by a current PI loop and a PI velocity loop with an additional feedforward term.
The controllers were originally tuned by the Maxon software, while the motors were freerunning.
This meant that each motor had slightly different parameters. Their movement was slightly different.
Since the robot has a considerable weight the controller was undertuned, it was unable to reach the desired speed and acceleration.
I manually retuned them, to have a better performance.

```python
# Plotting the velocity curves
motor_bad = genfromtxt("robotBaseLog_bad.csv", delimiter=";")
motor_good = genfromtxt("robotBaseLog_good.csv", delimiter=";")

fig, (ax0, ax1) = plt.subplots(ncols=2, constrained_layout=True)
ax0.set_xlabel('Time [msec]', size=15)
ax0.set_ylabel('Motor Velocity [relvalue]', size=15)

time0 = motor_bad[:,0] - motor_bad[0,0]
time0 = np.rint(time0 / 1e6)
ax0.plot(time0, motor_bad[:,1:4])
ax0.set_xlim([min(time0), max(time0)])
ax0.set_ylim([0,16])
ax0.grid()
ax0.set_title("Automatically tuned controller")

time1 = motor_good[:,0] - motor_good[0,0]
time1 = np.rint(time1 / 1e6)
ax1.plot(time1, motor_good[:,1:4])
ax1.set_xlim([min(time1), max(time1)])
ax1.set_ylim([0,16])
ax1.grid()
ax1.set_xlabel('Time [msec]', size=15)
ax1.set_title("Manually tuned controller")

fig.set_size_inches(12, 6, forward=True)

plt.show()
#
```

Above you can see the velocities of each motor before and after tuning the controllers.

The VMU931 communicates with the computer through the serial port.
The CRF already contained an interface for it.
I rewrote major part of it, to make it more robust and increase the signal acquisition frequency.

Finally to make experiments and the plotting of results easier, I wrote a python wrapper for the code with Boost.Python.
This allows the user to control the robot speed and acceleration from Jupyter Notebook.
The notebook recieves the motor torques and velocities, the gyroscope measurements and the robot position.
This data can quickly be plotted with Matplotlib in the notebook.

## Input shaping

Input shaping is an open-loop control method, which strives to reduce the vibration of flexible mechanisms.
It is mainly used in cranes and flexible link robotic arms in practice.
Input shaping can be used to control linear asymptotically stable flexible systems.
The main idea of input shaping lies in the superposition principle.
The flexible system starts vibrating when an acceleration is introduced to this system, which resuslts in the vibration around zero position.
When accelerating the structure, instead of using a single impulse, several impulses are applied.
If the impulses are applied out of phase of the vibration, they simply cancel each other out.

Let us look at a second order harmonic oscillator. The physical realization of such systems is for example a pendulum.

Bellow the speed of the pendulum can be seen related to time.

\begin{align}
y_{o}(t) & = \left[ \frac{A_{0} \omega}{\sqrt{1 - \xi^2}} e^{-\xi \omega \left( t - t_{0} \right)} \right] sin\left( \omega \sqrt{1 - \xi^2} \left( t - t_{0} \right) \right)
\end{align}

\begin{align}
\omega_{d} & = \omega \sqrt{1 - \xi^2}
\end{align}

```python
# Second order harmonic oscillator impulse response
def dampedVibration(x, A0, omega, damping, x0):
    if (damping >1):
        return np.inf
    coeff = np.sqrt(1 - damping*damping)
    y = ((A0 * omega) / coeff) * np.exp(-damping * omega * (x - x0)) * np.sin(omega * coeff* (x - x0))
    return y

x = np.linspace(0, 10, 200)
y = dampedVibration(x, 7, 2*np.pi*0.5, 0.15, 0)
fig, ax = plt.subplots(figsize=(10,5))
ax.plot(x, y)
ax.set_xlim([0, 10])
ax.set_ylim([-20,20])
ax.grid()
ax.set_xlabel('Time [sec]', size=15)
ax.set_ylabel('Velocity [1]', size=15)
ax.set_title("Under Damped Oscillation")

plt.show()
#
```

If we have several impulses we can sum the effects together, since the system is linear.

\begin{align}
y_{\sum} \left( t \right) & = \sum_{i=1}^{n}\left[ \frac{A_{0} \omega}{\sqrt{1 - \xi^2}} e^{-\xi \omega \left( t - t_{0} \right)} \right] sin\left( \omega_{d} \left( t - t_{0} \right) \right)
\end{align}

```python
# Superposition of impulses
x = np.linspace(0, 10, 200)
y1 = dampedVibration(x, 7, 2*np.pi*0.5, 0.15, 0)
y2 = -dampedVibration(x, 7, 2*np.pi*0.5, 0.15, 0)
y2[:int(1/10*200)] *= 0
y3 = y1 + y2
fig, ax = plt.subplots(figsize=(10,5))
p1 = ax.plot(x, y1)
p2 = ax.plot(x, y2)
p3 = ax.plot(x, y3, linewidth=4.0)
ax.set_xlim([0, 10])
ax.set_ylim([-20,20])
ax.grid()
ax.set_xlabel('Time [sec]', size=15)
ax.set_ylabel('Position [1]', size=15)
ax.legend((p1[0],p2[0],p3[0]),
          ("1st impulse response","2nd impulse response","Superpoisiton of the two"),
            prop={'size': 13})
ax.set_title("Superposition of impulses")
plt.show()
#
```

```python
# Superposition of impulses
x = np.linspace(0, 10, 200)
y1 = dampedVibration(x, 7, 2*np.pi*0.5, 0.15, 0)
y2 = -dampedVibration(x, 7, 2*np.pi*0.5, 0.15, 0)
y2[:int(1/10*200)] *= 0
y3 = y1 + y2
fig, ax = plt.subplots(figsize=(10,5))
p1 = ax.plot(x, y1)
p2 = ax.plot(x, y2)
p3 = ax.plot(x, y3, linewidth=4.0)
ax.set_xlim([0, 10])
ax.set_ylim([-20,20])
ax.grid()
ax.set_xlabel('Time [sec]', size=15)
ax.set_ylabel('Position of the Camera', size=15)
ax.legend((p1[0],p2[0],p3[0]),
          ("Normal Movement","Counter Balancing","The result of the two together"),
            prop={'size': 13})
ax.set_title("How to get rid of the camera vibration?")
plt.show()
#
```

As can be seen on the figure above the method gets rid of the oscillation in the ideal case.
Unfortunately, there is still a bit of oscillation, since the first half wave is not eliminated.

The method has 3 sets of parameters: number of impulses, the time location of impulses and the proportion of the impulses.
In the simplest case there is only a pair of impulses, this is called Zero Vibration shaper.

Bellow you can see how to calculate the amplitudes ($A_{i}$) and the time position ($T_i$) of the controller.
$$
\begin{bmatrix}
    A_{i}        \\
    t_{i}
\end{bmatrix} =
\begin{bmatrix}
    \frac{1}{1 + K}  & \frac{K}{1 + K}     \\
    0  & \frac{\pi}{\omega \sqrt{1 - \xi^{2}}}
\end{bmatrix}
$$

$$
K = \exp^{\left( \frac{- \xi \pi}{\sqrt{1 - \xi^{2}}} \right)}
$$

For the method to work only the natural frequency and the damping factor of the beam is needed. There is no need for complicated models.

Normally the beam equations are modelled by partial differential equation , either by Bernouilli or Timoshenko. These equations are hard to solve, and their precision highly depends on the precision of the initial conditions.

I applied an impulse to the flexible beam and recorded the angular velocity of the tip with the gyroscope.
Then fitted the second order harmonic scillator equation to the beam velocity data.
As can be seen the plot bellow, the model fits the data pretty well.
Unfortunately, there is a phase shift in the later periods. This might be because the data gets noisy around the peaks, for the first few periods. Since the goodness of the fit is mostly influenced by the first few oscillations.

```python
# Plot of rod natural frequency
gyroscope = genfromtxt("vmulog_free.csv", delimiter=";")
y = gyroscope[785:,1]
x = np.linspace(0, y.shape[0] * 0.005, y.shape[0])
fig, ax = plt.subplots(figsize=(10,5))
p1  = ax.plot(x, y)

p0 = [30, 1, 0.1, 0]
popt, pcov = curve_fit(dampedVibration, x, y, p0)
p2 = ax.plot(x, dampedVibration(x, popt[0], popt[1], popt[2], popt[3]))
ax.set_xlim([0, max(x)])
ax.set_ylim([min(y) * 1.2 ,max(y)*1.2])
ax.grid()
ax.set_xlabel('Time [sec]', size=15)
ax.set_ylabel('Velocity [1]', size=15)
ax.legend((p1[0],p2[0]), ("Gyroscope Data","Secod order fit"), prop={'size': 15})
ax.set_title("Natural Frequency of the rod")

ax.text(0.9, 0.2,'$\\omega$ = {:.2f} \n $\\xi$ = {:.2f}'.format( popt[1], popt[2]),
     horizontalalignment='center',
     verticalalignment='center',
     transform = ax.transAxes, size = 20)
plt.show()
#
```

```python
# Calculate K and T1
damping = popt[2]
K = np.exp((-damping * np.pi) / (1- damping* damping))
print("The K factor is : {:.2f}".format(K))
print("The first impulse is {:.2f}%, the second impulse is {:.2f}% of the acceleration.".format(1/(1+K)*100, K/(1+K)*100))
omega = popt[1]
# This is in VMU measurement units, which is 5 millisec
t_i = np.pi / (omega * np.sqrt(1-damping* damping))
print("The second impulse is at : {:.3f} msec".format(t_i))
#
```

    The K factor is : 0.59
    The first impulse is 62.82%, the second impulse is 37.18% of the acceleration.
    The second impulse is at : 0.858 msec

There is one big problem with this method of vibration reduction.
Input shaping gives no guidelines for evaluating how well the shaper works.
It does not take into account the real system. The shaper is ideal only if the actuators and controllers of the robot can produce the required impulses at just the right time.
In real world however there is always some timedelays and overshoots in the controllers.
In my case since I was working with a heavy mobile robot, which has 4 motors I wanted to know if I could do better.

# Cost Functions

One way to get a better input shaper is to try different parameters on the real robot and determine which one is better.
From this feedback we could create an optimization/learning algorithm to find the optimal parameters.

The first step is to create a metric which will show if one run of the parameters were better than the other. I will call this metric loss function.
The loss function has to satisfy the following:
1. the value should not vary between consecutive trials with same parameters
2. the value should change between different sets of parameters

I decided to use the root-mean-square error (RMSE) of the gyroscope values. For the mean I have substitued zero, since the values should be 0 in the longer constant speeds. (The mean of the measurement would be very different from zero, since the oscillation is not symmetric.)

$
RMSE = \sqrt{\frac{\sum{ \left( y_{i} - 0 \right)^{2}}}{N}}
$

Since the robot wheels have no suspensions, the uneven ground will introduce noise to the gyroscope signal.
It is important to get rido of this, otherwise the loss function will be very noisy.

I have recorded a longer run of the robot with constant speed. In constant speed after the inital period the gyroscope should measure values around zero.

In the figure bellow the measurement values can be seen. As expected the measurements are centered around zero, and they have symmetrical stochastic noise.

```python
#Steady state
gyroscope = genfromtxt("vmulog_constantv.csv", delimiter=";")
y = gyroscope[:,1]
x = np.linspace(0, y.shape[0] * 0.005, y.shape[0])
fig, ax = plt.subplots(figsize=(15,5))
p1  = ax.plot(x, y)
ax.set_xlim([0, max(x)])
ax.set_ylim([min(y) * 1.2 ,max(y)*1.2])
ax.grid()
ax.set_xlabel('Time [sec]', size=15)
ax.set_ylabel('Velocity [1]', size=15)
ax.set_title("Gyroscope readings during constant speed movement")

plt.show()
#
```

To better see the noise spectrum I have also plotted the fast fourrier transformation (FFT) of the signal.
There are 3 peaks in the signal. From the previous experiment it can be seen that the beam's natural frequency is around 1 Hz. We can apply a Digital Butterworth filter to get rid of the components higher than 4-5 Hz, to make the loss calculations less noisy.

Bellow the filtered and the original signal can be seen right after the acceleration phase.

```python
# Fourrier Transform of steady state
import scipy.fftpack
T = 0.005 # sampling time period
N = y.shape[0]
yf = scipy.fftpack.fft(y)
xf = np.linspace(0.0, 1.0/(2.0*T), N/2)

fig, ax = plt.subplots(figsize=(10,5))
ax.plot(xf, 2.0/N * np.abs(yf[:N//2]))
ax.set_xlim([0, max(xf)])
ax.set_ylim([min(2.0/N * np.abs(yf[:N//2])) * 1.2 ,max(2.0/N * np.abs(yf[:N//2]))*1.2])
ax.grid()
ax.set_xlabel('Frequency [Hz]', size=15)
ax.set_ylabel('Amplitude [rad/sec]', size=15)
ax.set_title("Fourrier Transformation of the Gyroscope signal")
plt.show()
#
```

```python
#Filtered Signal
y = genfromtxt("vmulog_accel.csv", delimiter=";")[:,1]
x = np.linspace(0, y.shape[0] * 0.005, y.shape[0])
from scipy import signal
fs = 200 #sampling freq
fc = 4
w = fc / (fs / 2)
b, a = signal.butter(5, w, 'low')
output = signal.filtfilt(b, a, y)

fig, ax = plt.subplots(figsize=(10,5))
ax.scatter(x, y, marker = ".", alpha = 0.3)
p0 = ax.plot(x, y, marker = ".", alpha = 0.3)
p1 = ax.plot(x, output, linewidth=4)
ax.set_xlim([0, max(x)])
ax.set_ylim([min(y) * 1.2 ,max(y)*1.2])
ax.grid()
ax.set_xlabel('Time [sec]', size=15)
ax.set_ylabel('Velocity [1]', size=15)
ax.set_title("Comparison of the raw and filtered measurements")
ax.legend((p0[0],p1[0]), ("Gyroscope Data","Filtered Data"), prop={'size': 15})

plt.show()
#
```

# Bayesian Optimization

Now that we have a metric to evaluate which control parameters are better we need to choose an algorithm to propose new parameters.

Machine learning in robotics have several difficulties compared to computer vision:
- there are a limited number of robots, learning can not be parallelized
- each control function evaluation takes long time
- real world robot control and sensors are very noisy

On the other hand when we do not use computer vision, just low dimensional sensor data, the models are much simpler. A gyroscope and the 4 wheels represent 7 dimensions. A small resolution image with 64 x 64 x 3 pixels represents 12288 dimensions.

The available algorithms are also constrained because we have no access to the gradient of the control parameters.

After careful consideration I have chosen Bayesian optimization for this problem. This can be also considered a reinforcement learning algorithm.

Bayesian optimization is a global optimization method. Theoretically it finds the best set of parameters.
It does not need the gradients of the loss function. Works with relatively small amount of data.

It is mostly used for finding the optimum in cases where the cost function is very expensive to evaluate. Such as predicting which point is ideal for oil drilling (Krieging), or optimizing machine learning model hyperparameters.

Bayesian optimization works by creating a model of the loss surface based on the previously evaluated points, than proposing a new set of parameters to evaluate.

This loss surface is an $\mathbb{R}^{n} \rightarrow  \mathbb{R}$ function, where n is the number of model parameters. The most common model to use are Gaussian Processes. Other common choice is using decision trees.

Gaussian Processes (GP) are probabilistic curves, where the finite collection of points form a multivariate normal distribution. To put it simpler this is just a different way to fit a curve, can be compared to line fitting with OLS. Instead of fitting a parametric model, such as a line or a sine wave, the function value is determined by key points, and a kernel function which shows how the function value changes between the key points.

Unlike parametric models GPs model the function in a discrete interval. When using Bayesian Optimization we have to set the range where we are looking for the parameter.

GPs are probabilistic models. They do not represent a single function, but a collection of stochastic functions. In practice we look at the mean, most likely, values of the function. From the standard deviaton of these functions we can calculate how certain is a Gaussian process in the function value in a given point. As a rule of thumb the further away we are from the key points, the less certain the model is.

```python
# Fitting a Gaussian Process to the Sine function

def f(x):
    return x * np.sin(x)

x = np.linspace(0,10, 100)
y = f(x)

number_of_samples = 12
gp =GaussianProcessRegressor(
            kernel=Matern(),
            normalize_y=True,
            random_state=5
        )
x_star = np.random.uniform(min(x), max(x), number_of_samples)

y_star = f(x_star)
gp.fit(x_star.reshape(-1, 1), y_star)

m, v = gp.predict(x.reshape(-1, 1), return_std=True)

plt.figure(figsize=(15,8))

p0 = plt.plot(x,y)
fill = plt.fill(np.concatenate([x, x[::-1]]),
         np.concatenate([m - 1.9600 * v,
                        (m + 1.9600 * v)[::-1]]),
         alpha=.1, fc='b', ec='None', label='95% confidence interval')
sc = plt.scatter(x_star, y_star)
p1 = plt.plot(x,m)

plt.legend((p0[0],p1[0], fill[0], sc),
           ("f(x)","Gaussian Process Mean","95% confidence interval", "Random Samples" ),
           prop={'size': 15})
plt.title("Fitting a Gaussian Process to y = x * sin(x)")
plt.show()
#
```

Based on the model of the loss at each iteration we need to pick a new set of parameters to probe.
Acquisition function is responsible to suggesting the new location.

There are three common acquisition functions : Expected Improvement (EI), Upper Confidence Bound (UCB) and Probabilty of Improvement (POI).

They suggest new points based on the mean and the standard deviation of the loss. Usually they have a parameter where we can choose the trade off between exploration and explotiation.

If we prefere exploration the acquisition function will prefere points in sample space, where the standard deviation is high. This means that the model is uncertain about the loss values in the given point. By exploring the space better, we ensure that we will find the global imptimum of the problem. On the other hand the optimization will take longer.

If we focus on exploitiation, then the new parameters will be close to points where the mean is minimal. It will make the optimization converge faster to a minumum. However since the algoirthm evaluates smaller part of the parameter space it risks getting stuck in a local optimum.

The trade of between the two have to be set manually. It must be based on how much time we have to evaluate the algorithm.

# Experiments

For the Bayesian Optimization I have tried out several libraries.
I chose to use fmfn/BayesianOptimization because it has very nice logging functionalities.

I run several trials with Matern and RBF kernels of the GaussianProcess.
Tried out the 3 previously mentioned acquisition functions.

The goal of the optimization was to find the optimal shaper parameters.
To have as much vibration as possible I have fixed the first acceleratioin impulse close to the maximum acceleration of the robot. This way only the time location of the second impulse ($t_{2}$), and the amplitude ($A_{2}$) of the second impulse had to be found.

The robot was running in the lab. There are 2 bumpers in the runway, which add noise to the trials.
To make each execution as similar as possible, I did one run forward and one backward. Calculated the mean of the two runs. Each trial took around 15 seconds. An experiment consisted of 5 initial random values, and 30 optimization steps.

Bellow you can see a video of one optimization process. The posterior mean, corresponds to the predicted value of the loss function. The higher the value, the better the parameters are. The posterior standard deviation shows the uncertainty of the predicted mean. It can be observed that at points which were evaluated the standard deviation is close to zero. At points far from evaluations the standard deviation is higher.

```python
# Optimization video
gp =GaussianProcessRegressor(
            kernel=Matern(length_scale=[5, 10]),#RBF(length_scale=[0.05, 1]),
            alpha=1e-6,
            normalize_y=True,
            n_restarts_optimizer=5,
            random_state=5
        )
#Load data
log = "logs_kappa5.json"
x = []
y = []
with open(log, "r") as f:
    for line in f:
        line = json.loads(line)
        y.append(line["target"])
        x.append([line["params"]["a1"],line["params"]["t1"]])
x = np.array(x)

label_x = "A1"
label_y = "T1"
bounds = [[0, 1.5],[300,1000]]

X1 = np.linspace(0, 1.5, 200)
X2 = np.linspace(300, 1000, 200)
x1, x2 = np.meshgrid(X1, X2)
X = np.hstack((x1.reshape(200*200,1),x2.reshape(200*200,1)))

fig = plt.figure(figsize=(13,5))

def update(i):
    fig.clear()
    ax1 = fig.add_subplot(121)
    ax2 = fig.add_subplot(122)

    gp.fit(x[:i],y[:i])
    m, v = gp.predict(X, return_std=True)

    cf1 = ax1.contourf(X1, X2, m.reshape(200,200),100)
    ax1.plot(x[:i-1,0], x[:i-1,1], 'r.', markersize=10, label=u'Observations')
    ax1.plot(x[i,0], x[i,1], 'r.', markersize=25, label=u'New Point')
    cb1 = fig.colorbar(cf1, ax=ax1)
    ax1.set_xlabel(label_x)
    ax1.set_ylabel(label_y)

    ax1.set_title('Posterior mean')
    ##
    ax2.plot(x[i,0], x[i,1], 'r.', markersize=25, label=u'New Point')
    ax2.plot(x[:i-1,0], x[:i-1,1], 'r.', markersize=10, label=u'Observations')
    cf2 = ax2.contourf(X1, X2, np.sqrt(v.reshape(200,200)),100)
    cb2 = fig.colorbar(cf2, ax=ax2)
    ax2.set_xlabel(label_x)
    ax2.set_ylabel(label_y)

    ax2.set_title('Posterior sd.')
    return ax1, ax2
#
```

```python
anim = FuncAnimation(fig, update, frames=np.arange(3, x.shape[0]), interval=500)
anim.save('line.gif', dpi=80, writer='imagemagick')
HTML(anim.to_html5_video())
```

On the figure bellow you can see the comparison of the different shapers. No shaping case is where the robot is left to acclerate from standing position to the desired velocity directly.

The analytic shaper was created simply using the beam natural frequency and damping. It is clearly superior to the no shaper case.

Finally, the Bayesian Shaper is the result of the optimization method. It has out performed the analytical shaper in all 10 trials.

```python
#Comparison of methods
y = genfromtxt("b_s_n.csv", delimiter=";")
x = np.linspace(1, y.shape[0], y.shape[0])
print(x.shape)
fig, ax = plt.subplots(figsize=(10,5))
ax.scatter(x, y[:,0], marker = "X")
ax.scatter(x, y[:,1], marker = "X")
ax.scatter(x, y[:,2], marker = "X")
ax.set_xlim([0, max(x)+1])
ax.set_ylim([5 ,10])
ax.grid()
ax.set_xlabel('Number of Trial', size=15)
ax.set_ylabel('Loss Function (smaller better)', size=15)
plt.legend(("Bayesian Shaper","Analytic Shaper", "No Shaping"))
ax.set_title("Comparison between the methods")
plt.show()
#
```

On the video bellow the camera feed can be seen from the top of the beam.
There is considerably less vibration in the video then during the original control of the robot.

```python
HTML('')
```

    /home/robotronics/.local/lib/python3.5/site-packages/IPython/core/display.py:694: UserWarning: Consider using IPython.display.IFrame instead
      warnings.warn("Consider using IPython.display.IFrame instead")

# Discussion

In my experience Bayesian Optimization is a very reliable algorithm. It can find the global optima at each trial, and does overperform the analytic case very stably.

The biggest failure points of the method is setting the the initial parameters of the Gaussian Process and finding a good exploration-exploitation trade off.

The frameworks I have looked at all initialize the Gaussian Process with a single dimensional length scale parameter. If the parameters however are not in the same order of magnitude, the optimization method will disregard one of the dimensions.

The exploration parameter can be set, by trying a few values. If the optimization quickly converges to local minima, after 5-10 trials, we should increase the exploration term.

In the future this method could be extanded to work with different flexible beam heights. The length of the beam could be considered a parameter, which the algorithm can not change. This type of optimization is called contextual Bayesian Optimization.

Finally, instead of using a 2 impulse shaper Bayesian Optimization could be used to predicta whole velocity trajectory. This would increase the dimensionality of the problem, but I believe it could work very well.