# SLU13 - Tree-based models: Learning notebook

## Imports

```python
from math import log2

import pandas as pd
import numpy as np

from sklearn.datasets import load_boston
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.metrics import mean_squared_error
from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor
from IPython.display import Image

from utils.utils import (
    make_data,
    separate_target_variable,
    process_categorical_features,
    visualize_tree
)
```

# Table of Contents

1. [Decision trees](#decision-trees)
2. [Tree-based ensembles](#tree-based-ensembles)
3. [Bagging with random forests](#bagging-with-random-forests)
4. [Boosting with gradient boosting](#boosting-with-gradient-boosting)

# Decision trees

Knowledge-based systems represent knowledge explicitly, as a set of rules, rather than being implicit in algorithms.

Imagine we want to decide whether to go hiking based on the weather. Our acquired knowledge can be represented as:

*Fig. 1: A simple decision-tree used for classification (Quinlan, 1986).*

This particular structure, a **decision tree**, displays a flow of conditional statements (i.e., `if Condition then Outcome`) where:

* Each **node** represents a test on a feature
* Each **branch** represents the outcome of the test
* Each **leaf** represents an outcome or decision.

The paths from root to leaf represent the rules, e.g., `if Outlook is Sunny and Humidity is Normal then Go Hiking`.

Pioneering knowledge-based systems codified rules coming from domain specialists, at the rate of a few rules per man day.

But complex tasks require creating and maintaining thousands of such rules, turning this specialist-based approach into a bottleneck.

## Learning sets of rules as decision trees

The need to discover and maintain sets of rules to make decisions stimulated the investigation of general-purpose machine learning methods to build knowledge-based systems.

The resulting systems address the underlying task of learning simple rules, by building decision trees from data examples.

Typically, a decision tree is developed from the top-down (known as top-down induction), and there are several algorithms that can be used to build them, including:
* **Iterative Dichotomiser 3 (ID3)**, for classification using categorical features
* **C4.5**, extends ID3 to support non-categorical features
* **Classification and Regression Trees (CART)**, generalizes C4.5 to support regression.

We follow ID3 ([Quinlan, 1986][1]), for binary classification using categorical features with a small set of possible values (i.e. low cardinality).

[1]: http://hunch.net/~coms-4771/quinlan.pdf "Quinlan, J. 1986. Induction of Decision Trees. Machine Learning 1: 81-106."

```python
data = make_data()
```

### Algorithm overview

Take an arbitrary training set $C = \{\textbf{x}_i, y_i\}_{i=1}^m$, a collection of labeled observations, with $y_i \in \{L_1, \dots, L_v\}$.

The features vector is represented as $\textbf{x}_i = \{x_i^{(1)}, \dots, x_i^{(n)}\}$, where $x_i^{(j)}$ is the value of the $j$-th feature for the $i$-th observation.

In ID3, $x_i^{(j)}$ can take on one of a fixed number of possible values $x_i^{(j)} \in \{A_1^{(j)}, \dots, A_w^{(j)}\}$.

$$X = \begin{bmatrix}
  x_1^{(1)} & x_1^{(2)} & \dots & x_1^{(n)} \\
  x_2^{(1)} & x_2^{(2)} & \dots & x_2^{(n)} \\
  \dots     & \dots     & \dots & \dots     \\
  x_m^{(1)} & x_m^{(2)} & \dots & x_m^{(n)}
 \end{bmatrix}$$

```python
X, y = separate_target_variable(data)

X
```

      Outlook
      Temperature
      Humidity
      Windy

      0
      sunny
      hot
      high
      false

      1
      sunny
      hot
      high
      true

      2
      overcast
      hot
      high
      false

      3
      rain
      mild
      high
      false

      4
      rain
      cool
      normal
      false

      5
      rain
      cool
      normal
      false

      6
      overcast
      cool
      normal
      false

      7
      sunny
      mild
      high
      false

      8
      sunny
      cool
      normal
      false

      9
      rain
      mild
      normal
      false

      10
      sunny
      mild
      normal
      true

      11
      overcast
      mild
      high
      true

      12
      overcast
      hot
      normal
      false

      13
      rain
      mild
      high
      true

This is a binary classification problem, where $y_i \in \{0, 1\}.$

$$y = \begin{bmatrix}
  y_1   \\
  y_2   \\
  \dots \\
  y_3
 \end{bmatrix}$$

```python
y
```

    0     0
    1     0
    2     1
    3     1
    4     1
    5     0
    6     1
    7     0
    8     1
    9     1
    10    1
    11    1
    12    1
    13    0
    Name: Class, dtype: int64

Now, imagine a test, $T$, on $x_i$, with possible outcomes $O_1, O_2, \dots, O_w$.

It can be a condition on the features, for example.

$T$ produces a partition $\{C_1, C_2, \dots, C_w\}$ of $C$, where $C_k$ contains those observations having outcome $O_k$.

*Fig. 2: Partitioning of the objects in $C$ with a test $T$ (Quinlan, 1986).*

We can recursively apply different tests to each of these partitions, creating smaller and smaller sub-partitions, until we get homogenous (i.e., single-class) leaves. The result of this process is a decision tree for all of $C$.

This tree-building algorithm can be described in the following way:
```
ID3 (Data, Target, Attributes)
    If all examples are positive, Return the single-node tree Root, with label = 1.
    If all examples are negative, Return the single-node tree Root, with label = 0.
    Otherwise Begin
        A <- Pick the Attribute that best classifies examples.
        Decision tree for Root = A.
        For each possible value, O_i, of A,
            Add a new tree branch, corresponding to the test A = O_i.
            Let Data(v_i) be the subset of examples that have the value O_i for A.
            Below this new branch add the subtree ID3 (Data(O_i), Target, Attributes – {A}).
    End
    Return Root
```

But what is the **attribute that best classifies examples** at each step? As we will soon see, this is related with the concepts of **information gain** and **entropy**.

## Attribute selection

A test will have a high **information gain** if it generates partitions with low [entropy](https://en.wikipedia.org/wiki/Entropy_(information_theory)). In general: the greater the homogeneity of a partition, the lower its entropy.

In this case, a partition is said to be very homogeneous if the majority of its elements belong to a single class.

Consider the problem of binary classification (it can be extended to support multi-class settings). Take $p$ as the probability of the positive class, i.e., the proportion of positive cases in the set. In this case, the entropy is given by:

$$I(p) = - p \log_2 p - (1 - p) \log_2 (1-p)$$

It ranges between 0 and 1:

* 0, when $p=1$ or $p=0$. We are in a situation of low entropy/high purity, because there is no "surprise" in the outcome. We say the information is minimum, because observing the class of an instance will not give us any new information - we already know what it will be.
* 1, when $p=\frac{1}{2}$. We are in a situation of high entropy/low purity, because the "uncertainty" of the outcome is maximum. This is equivalent to saying that the information is maximum.

Out of curiosity, entropy is measured in **bits** (hence the $log_2$ in the formula, since it tells you the number of digits you would need to represent a number in base 2).

```python
def entropy(p):
    if 0 < p < 1:
        return -p * log2(p) - (1 - p) * log2(1 - p)
    else:
        return 0
```

Let's compute the entropy for the entire data set:

```python
def compute_probability(data):
    n = data.shape[0]
    f = (data['Class'] == 1).sum()

    return f / n

p = compute_probability(data)
```

```python
entropy(p)
```

    0.9402859586706309

We have a fairly high entropy for the entire data set.

This means that substantial proportions of the data set belong to each of the classes, as confirmed by the value of $p$ - the data is not homogenous.

```python
p
```

    0.6428571428571429

Starting from a set with high entropy, we would like to select a test that divides it into partitions that are as homogeneous as possible. In the ideal case, each partition would be a leaf node, containing only instances from a single class.

A test is restricted to branching on an attribute $A$ with values $\{A_1, A_2, \dots, A_v\}$, thus partitioning $C$ into $\{C_1, C_2, \dots, C_v\}$.

The expected entropy of the test is obtained as a weighted average of the entropy of the resulting groupings:

$$E(A) = \sum_{i=1}^v \frac{\|C_i\|}{\|C\|}I(p_i)$$

In short, and as expected, we are measuring an attribute's ability to generate homogeneous groupings.

Let's calculate the expected information of the `Outlook` feature:

```python
def mean_entropy(data, attribute):
    c_norm = data.shape[0]
    information = 0
    values = data[attribute].unique()
    for value in values:
        group = data[data[attribute] == value]

        c_i_norm = group.shape[0]

        w = c_i_norm / c_norm

        p = compute_probability(group)
        e = entropy(p)

        information += w * e

    return information

mean_entropy(data, 'Outlook')
```

    0.6935361388961918

Given this result, the `Outlook` feature yields partitions that have, on average, a much lower entropy than the entire data set. It is a good candidate for the first test!

We call this loss of entropy the **information gain** of branching on an attribute $A$, and it is given by:

$$IG(A) = I(p) - E(A)$$

where $I(p)$ is the entropy on the node over which the test is being performed, and $E(A)$ is the expected entropy of the test.

Let's compute it.

```python
def information_gain(data, attribute):
    p = compute_probability(data)
    i = entropy(p)

    e = mean_entropy(data, attribute)

    return i - e

information_gain(data, 'Outlook')
```

    0.2467498197744391

To generate the simplest possible tree, it is a good choice to branch, at each step, on the attribute with the highest information gain. Let's find out which:

```python
def examine_candidate_attributes(data, attributes):
    return {
        attribute:information_gain(data, attribute)
        for attribute
        in attributes
    }

attributes = [c for c in data.columns if c is not 'Class']
examine_candidate_attributes(data, attributes)
```

    {'Outlook': 0.2467498197744391,
     'Temperature': 0.029222565658954647,
     'Humidity': 0.15183550136234136,
     'Windy': 0.0250781735058504}

So, we would start by branching on the values of the `Outlook` attribute.

```python
outlook_values = data['Outlook'].unique()
outlook_values
```

    array(['sunny', 'overcast', 'rain'], dtype=object)

We repeat the process in each partition generated by the `Outlook` attribute:

```python
for i, value in enumerate(outlook_values):
    partition = data[data['Outlook'] == value]
    results = examine_candidate_attributes(partition, attributes)
    print("\n---\n")
    print("Partition: Outlook == {}. \nResults:\n{}.".format(value, results))
```

    ---

    Partition: Outlook == sunny.
    Results:
    {'Outlook': 0.0, 'Temperature': 0.5709505944546686, 'Humidity': 0.9709505944546686, 'Windy': 0.01997309402197489}.

    ---

    Partition: Outlook == overcast.
    Results:
    {'Outlook': 0.0, 'Temperature': 0.0, 'Humidity': 0.0, 'Windy': 0.0}.

    ---

    Partition: Outlook == rain.
    Results:
    {'Outlook': 0.0, 'Temperature': 0.01997309402197489, 'Humidity': 0.01997309402197489, 'Windy': 0.3219280948873623}.

Interestingly enough, none of the attributes provides any information gain in the `Outlook == overcast`  case. Let's try to understand why:

```python
data[data['Outlook'] == 'overcast']
```

      Outlook
      Temperature
      Humidity
      Windy
      Class

      2
      overcast
      hot
      high
      false
      1

      6
      overcast
      cool
      normal
      false
      1

      11
      overcast
      mild
      high
      true
      1

      12
      overcast
      hot
      normal
      false
      1

As it turns out, this grouping is already homogeneous (all instances are positive) - it is already a leaf. We figured out our first rule:

```
if Outlook is Overcast then Go Hiking
```

Finally, we would continue partitioning recursively until all the resulting groupings are homogeneous (we obtain pure leaves), or until we have exhausted our features (in which case, we assign to the final leaves the majority class in their partition).

## Using decision trees

The `sklearn` implementation uses an optimized version of the CART algorithm. This means that the resulting decision tree might be different than if you had used ID3.

We won't go into this in detail, but don't worry - the principles behind it are very similar to what you've learned.

```python
X_ = process_categorical_features(X)

clf = DecisionTreeClassifier(criterion='entropy', random_state=101)
clf.fit(X_, y)
```

    DecisionTreeClassifier(criterion='entropy', random_state=101)

Above, we do a trick to convert categorical variables into something more `sklearn`-friendly.

(ignore it for now - you will learn all about it in due time.)

Then, we train a `DecisionTreeClassifier` (refer to the [docs](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html) for more information).

Let's visualize the resulting tree:

```python
class_names = ["Don't go hiking!", "Go hiking!"]
t = visualize_tree(clf, X_.columns, class_names)
Image(t)
```

On each node, we can see:

* two different paths, indicated by each result of the test `(feature_value <= 0.5)`, applied to the current node. `True` means **all instances in this node for which `feature` IS NOT equal to `value`**. The left path is always True, and the right path is always False.
* the entropy in that node.
* the `value` array, which indicates how many samples reaching that node belong to each class (the classes are in ascending numerical order)
* the name of the majority class in the node (in case of a tie, the first class in numerical order is taken)

The CART algorithm can also be used for regression problems. In that case, you should use the `DecisionTreeRegressor` (refer to the [docs](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html)). To visualize a simple example, check out [this page](https://scikit-learn.org/stable/auto_examples/tree/plot_tree_regression.html).

## Feature importance

A great characteristic of decision trees is that they allow us to compute **feature importance**: a score for each feature, based on how useful they are at predicting the target variable.

The importance of a feature is given by the decrease in node impurity (according to the criterion being used - in this case, the entropy), weighted by the proportion of samples reaching the node.

We can calculate the feature importance in the following way:

```python
feature_importances = pd.Series(data=clf.feature_importances_, index=X_.columns)
feature_importances.sort_values(ascending=False)
```

    Outlook_overcast    0.283411
    Humidity_high       0.249079
    Outlook_sunny       0.211799
    Windy_true          0.179147
    Temperature_mild    0.076563
    Windy_false         0.000000
    Humidity_normal     0.000000
    Temperature_hot     0.000000
    Temperature_cool    0.000000
    Outlook_rain        0.000000
    dtype: float64

By inspecting the visual representation of the tree, we can see that the `Windy_true` attribute has the highest information gain (from 1 to 0 on both leaves it generate), and therefore the highest decrease in node impurity.

However, since not many samples reach that node (only 2 in the training set), this is not the most important feature, since it is not as relevant to predicting the class in the overall dataset. Compare it with the feature with the highest importance, `Outlook_overcast`, placed right at the root of the tree: despite not having as high of an information gain, it immediately allows us to classify 4 samples as "Go Hiking".

## Pros and cons

### Pros

Decision trees are a straightforward way to represent rules that:

* are simple to understand, interpret and visualize
* require little to no data preparation (for example, don't require data scaling and normalization)
* are able to handle numerical and categorical variables
* are a white-box model, all decisions are replicable and easily explainable.

### Cons

Decision trees are extremely flexible and prone to creating over-complex trees that don't generalize:

* Overfitting
* Overfitting
* Overfitting.

(Repetition makes perfect.)

Mechanisms such as pruning (removing sections of the tree) and setting the maximum depth of the tree help with overfitting. Let's see how we can configure the maximum tree depth with `sklearn`:

```python
clf = DecisionTreeClassifier(
    criterion='entropy',
    max_depth=2,
    random_state=101
)
clf.fit(X_, y)
```

    DecisionTreeClassifier(criterion='entropy', max_depth=2, random_state=101)

```python
t = visualize_tree(clf, X_.columns, class_names)
Image(t)
```

We can also set the minimum number of samples required to split a node, in order to avoid fully-grown trees.

```python
clf = DecisionTreeClassifier(
    criterion='entropy',
    min_samples_split=5,
    random_state=101
)
clf.fit(X_, y)
```

    DecisionTreeClassifier(criterion='entropy', min_samples_split=5,
                           random_state=101)

```python
t = visualize_tree(clf, X_.columns, class_names)
Image(t)
```

These all help with controlling overfitting, but they might simply not be enough; and in fact, some times this approach is too heavy-handed, and might lead to underfitting. We will need to find a smarter way to control overfitting, while still allowing our trees to represent complex rules.

Some final notes:

* if the attributes are adequate, it is always possible to construct a decision-tree that correctly classifies every training instance.
* attributes are **inadequate** if the data contains two objects that have identical values for each attribute and yet belong to different classes.
* the key takeaway: **decision trees overfit like hell**.

## Tree-based ensembles

**Ensemble methods** combine the predictions of several models, known as **base learners** or **base estimators**. Each individual estimator in the ensemble is often a "weak learner" (i.e. has slightly better accuracy than random). However, combining the predictions of the entire ensemble, we often get better results that are less prone to overfitting, when compared to using a single, larger model.

*Fig. 3: A simple ensemble model, using many trained models to generate a single prediction.*

There are homogeneous and heterogenous ensembles, based on whether the base learners are all of the same type or not.

We focus on homogenous ensembles of decision-trees, particularly:

* Building several independent trees and then average their predictions, so that variance is reduced (i.e., **bagging**)
* Building trees sequentially as to reduce the bias of the combined estimator (i.e., **boosting**).

# Bagging with random forests

Bootstrap aggregating, also known as bagging, consists in:

1. Creating several independent data sets
2. Training a model in each data set
3. Aggregating individual predictions.

Bagging can be seen as training several independent models in parallel and averaging the predictions.

Imagine the following example: you are experiencing a strange feeling in your nose, and want to get a diagnosis. You have the option of going to a single, extremely-specialized podiatrist (that's a foot doctor, in case you didn't know, and in this analogy, it represents a single large model), or you could get your diagnosis from an association of 500 3rd year university students of medicine (which in this analogy represent a bag of simpler, shallower models).

Now, the podiatrist is very specialized: one might even say that he might have **overfit** to foot diseases. He might be able to extend his knowledge to your particular affliction and give a good diagnosis, but odds are he will diagnose you with something totally unrelated.
The group of 3rd year medicine students, on the other hand, might lack individual expertise; but if each of them has at least attended *some* classes, and *if they haven't all attended the same classes*, it's very likely that their collective knowledge surpasses the podiatrist's, and that their majority vote will be the right diagnosis.

This example illustrates the main strength of bagging, which is that several weak models can cancel each other's weaknesses, provided that they are highly independent and were exposed to different information.

But how does this relate to decision trees, you might ask? Well, decision-trees don't simply overfit, they are also highly unstable: small variations in the data result in wildly different trees.

This makes them particularly suited for bagging.

## Bagging

Suppose you have a sequence of data sets $\{D_1, D_2, \dots, D_k\}$, with observations from the same underlying distribution $\mathcal{D}$.

We can obtain an **ensemble of models**, $\{h_1, h_2, \dots, h_k\}$, by training a model in each data set.

*Fig. 4: In bagging different data sets are generated from the main one, models are trained in parallel and predictions averaged.*

By running the models in parallel, we get $\{\hat{y}_1\, \hat{y}_2\, \dots, \hat{y}_k\}$, a list of predictions from our ensemble.

To obtain a single prediction we can then evaluate all models and aggregate the results by:
* Averaging the $k$ predictions (regression)
* Using majority voting to predict a class (classification).

More often than not, however, we don't have multiple data sets. In this situation, we can do bootstrapping.

## Bootstrapping

We can take repeated random samples with replacement $\{C^1, C^2, \dots, C^b\}$ from $C$, our full dataset:

```python
def make_bootstrap_data(data, b):
    n = data.shape[0]
    return [data.sample(n=n, replace=True) for i in range(b)]

bootstrap_data = make_bootstrap_data(data, 2)
```

Typically, the bootstrapped data sets are the same size as the original data set.

```python
bootstrap_data[0]
```

      Outlook
      Temperature
      Humidity
      Windy
      Class

      4
      rain
      cool
      normal
      false
      1

      0
      sunny
      hot
      high
      false
      0

      10
      sunny
      mild
      normal
      true
      1

      6
      overcast
      cool
      normal
      false
      1

      13
      rain
      mild
      high
      true
      0

      12
      overcast
      hot
      normal
      false
      1

      1
      sunny
      hot
      high
      true
      0

      9
      rain
      mild
      normal
      false
      1

      0
      sunny
      hot
      high
      false
      0

      3
      rain
      mild
      high
      false
      1

      6
      overcast
      cool
      normal
      false
      1

      8
      sunny
      cool
      normal
      false
      1

      6
      overcast
      cool
      normal
      false
      1

      9
      rain
      mild
      normal
      false
      1

```python
bootstrap_data[1]
```

      Outlook
      Temperature
      Humidity
      Windy
      Class

      12
      overcast
      hot
      normal
      false
      1

      1
      sunny
      hot
      high
      true
      0

      8
      sunny
      cool
      normal
      false
      1

      4
      rain
      cool
      normal
      false
      1

      12
      overcast
      hot
      normal
      false
      1

      1
      sunny
      hot
      high
      true
      0

      2
      overcast
      hot
      high
      false
      1

      8
      sunny
      cool
      normal
      false
      1

      12
      overcast
      hot
      normal
      false
      1

      13
      rain
      mild
      high
      true
      0

      7
      sunny
      mild
      high
      false
      0

      13
      rain
      mild
      high
      true
      0

      0
      sunny
      hot
      high
      false
      0

      7
      sunny
      mild
      high
      false
      0

We would now train a different decision tree in each data set and use voting to predict whether or not to go hiking.

Bagging works very well in controlling overfitting and the generalization error in unstable models. If one of the models in the bag overfits to a certain noisy observation, it's very likely that its vote will be drowned out by the rest of the models in the ensemble.

## Random forests

A **random forest** is an ensemble learning method created by bagging multiple decision trees.
In random forests, bagging is used in tandem with random feature selection:

* Datasets are generated from the original data, using bootstrapping (row sampling)
* Then a tree is grown on each bootstrapped data set using random feature selection (column sampling).

Random feature selection means that only a random subset of the features is available at each split.

Randomizing features acts as a kind of regularization, further mitigating overfitting, because it forces each individual classifier to be as good as possible, while having access to limited information. This increases diversity inside the ensemble, which is often very beneficial.

Fortunately, `sklearn` implements all this intricate logic for us.

```python
rf = RandomForestClassifier(
    n_estimators=10,
    criterion='entropy',
    max_features=2,
    bootstrap=True
)

rf.fit(X_, y)
```

    RandomForestClassifier(criterion='entropy', max_features=2, n_estimators=10)

For detailed information about the models, check the documentation:
* RandomForestClassifier ([docs](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html))
* RandomForstsRegressor ([docs](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html)).

Bagging is a great technique to reduce overfitting, and as such, to reduce the variance of our final predictions. This comes at the cost of interpretability - it is much easier to interpret the rules contained within a single decision tree, than it is to analyze the driving forces in a majority vote of hundreds or even thousands of models.

# Boosting with gradient boosting

**Boosting** is a different ensemble learning technique. In this case, instead of training several base learners in parallel and averaging their predictions, we train them sequentially; and the input of each model is the residual error of the previous model.

The general idea is to build strong ensembles by combining base learners sequentially, each of them correcting previous errors (and thus reducing bias).

**Note:** this section can be pretty intimidating at first, but don't despair! It will be worth it in the end.

## Boosting

Trees are grown sequentially, and each tree is created using information derived from the previous tree. In each iteration:

1. Errors and misclassifications of the past model are given increased weight in a new training data set
2. The current model is trained on the new training set, fitting the residual errors of the previous model.

As such, each model specializes in correcting past mistakes and misclassifications.

Intuitively, this produces an ensemble of models that are good in different "parts" of the training data, as they sequentially correct each other's mistakes. The final prediction is obtained by summing the predictions of each model in the ensemble.

Boosting is less robust than bagging against overfitting, and as such, it is recommended to control the number of estimators used, the strength of each individual estimator, the learning rate, and use other regularization techniques.

In this example we will use the [boston](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html) house-prices data set (regression).

```python
def prepare_boston():
    boston = load_boston()
    X = pd.DataFrame(data=boston.data, columns=boston.feature_names)
    y = pd.Series(data=boston.target, name='price')

    return X, y

X_boston, y_boston = prepare_boston()
```

## What do we mean by "gradient" boosting

To understand the meaning of **gradient boosting**, we need to review a couple of concepts.

### Loss function

A loss function quantifies how bad our predictions are. An example is the squared error, seen below:

$$L_i = (y_i - h(x_i))^2$$

As we can see, the square error increases with the square of the difference between each of our predictions, and the real value. The worse this error across all data samples, the worse will the overall model performance be.

The training of an individual machine learning model is driven by the minimization the total loss, over all observations in the training set, with respect to the model's parameters $\theta$:

$$\min_{\theta} J = \sum_{i=1}^N L(y_i, h(x_i, \theta))$$

$h(x_i, \theta)$ represents our model's predictions, or $\hat{y}$.

### Gradient descent

The gradient of a loss function ($\nabla$), at each point, is its inclination or slope.

It's the fancy term for the multi-variable generalization of the derivative, i.e., the partial derivatives of function $f$ at a given point.

When training a model by gradient descent, we calculate, at each training step, the derivative of the loss $L$ with respect to the model's parameters, and then update the parameters by giving a small step in the opposite direction of this gradient - the direction in which the loss is decreasing:

$$\theta^{n+1} = \theta^{n} - \eta \nabla_{\theta} L(y, \hat{y})$$

As such, at each step, we are trying to gradually move our model to a location in parameter space where the loss function has a lower value.

## Gradient boosting in detail

We now have all pieces required to understand gradient boosting. Let's consider the case of a numerical target variable.

We have our first model in the boosting ensemble, $F^{(1)}(x, \theta)$. This model achieves a certain loss, given by $L(\textbf{y}, F^{(1)}(x, \theta))$.

We now want to add a second model, $h$, to improve our first model in a way that

$$
F^{(1)}(x_1) + h(x_1) = y_1 \\
F^{(1)}(x_2) + h(x_2) = y_2 \\
...
$$

or equivalently,

$$
h(x_1) = y_1 - F^{(1)}(x_1) \\
h(x_2) = y_2 - F^{(1)}(x_2) \\
...
$$

Maybe there isn't a single model that can achieve this. But perhaps some regression tree can do it approximately. We can try to fit it to the following data points:

$$
(x_1, y_1 - F^{(1)}(x_1)) \\
(x_2, y_2 - F^{(1)}(x_2)) \\
...
$$

$y_i - F^{(1)}(x_i)$ are called **residuals**. The role of $h$ is to be strong where the initial model was weak. And if the new model $F^{(2)} = F^{(1)} + h$ still isn't good enough, we can fit an additional regression tree to its residuals, and so on and so forth.

Taking a step back to our first model, imagine we are using the squared error loss function:

$$
L(\textbf{y},F_1(x)) = (\textbf{y} - F^{(1)}(x))^2
$$

We want to minimize $J = \sum_i{L(y_i, F^{(1)}(x_i))}$ by adjusting $F^{(1)}(x_1)$, $F^{(1)}(x_2)$, ...

Now, what we are truly adjusting during learning are the parameters $\theta$ of $F^{(1)}$. But if we take a functional perspective, we can observe that $F^{(1)}(x_i)$ are simply numbers - we can treat them as parameters and take derivatives:

$$
\begin{align}
\frac{\partial J}{\partial F^{(1)}(x_i)} & = \frac{\partial \sum_{j=0}^{n}{L(y_j, F^{(1)}(x_j))}}{\partial F^{(1)}(x_i)} \\
& = \frac{\partial L(y_i, F^{(1)}(x_i))}{\partial F^{(1)}(x_i)} \\
& = -2 (y_i - F^{(1)}(x_i)) \\
& = -2*residuals
\end{align}
$$

As you may have noticed, when the loss function is the squared error, the residuals are proportional to the negative gradients of the loss function!

Thus arises the similarity with gradient descent:

$$
\begin{align}
F^{(2)}(x_i) & = F^{(1)}(x_i) + h(x_i) \\
& = F^{(1)}(x_i) + y_i - F^{(1)}(x_i) \\
& = F^{(1)}(x_i) - \frac{1}{2}\frac{\partial J}{\partial F^{(1)}(x_i)} \\
\end{align}
$$

and

$$
\theta^{n+1} = \theta^{n} - \eta \frac{\partial J}{\partial F(\theta_i)}
$$

To summarize: for regression with **square loss**,

* the residual <=> negative gradient
* fitting h to the residual <=> fitting h to the negative gradient
* adding a new estimator based on the residual <=> adding a new estimator based on the negative gradient

When adding a new estimator to our boosting ensemble, we are actually minimizing a global loss function by using gradient descent, in function space!

The final thing to understand here is that the concept of gradients is more useful than the concept of residuals, because it allows us to generalize to loss functions other than the squared error. As long as we fit each additional model to the negative gradients of the current global model, we will be minimizing our desired global loss function.

## In practice

Let's implement one step of a boosting algorithm, for the case where the loss function is the mean squared error, based on what we have discussed so far!

### Initialization

We initialize all $F^{(1)}(x_i)$ to a sensible constant, $F^{(1)}(x_i) = \gamma$.

Since our goal is to minimize the mean squared error, we will use the mean value of the target variable in the training set.

```python
y_boston_h0_pred = np.repeat(y_boston.mean(), y_boston.size)
```

We compute the mean squared error of this initial set of predictions.

```python
mean_squared_error(y_boston, y_boston_h0_pred)
```

    84.41955615616556

### Iterations

#### Generating a new data set

We build a new training set at the start of each iteration, by recomputing the target variable to be the gradient of the previous model.

More concretely, at the $j$-th iteration we compute a new target variable $r^{(j)}_i$, corresponding to the gradient of the past iteration:

$$r^{(j)}_{i} = \frac{\partial L(y_i, F^{(j-1)}(x_i))}{\partial F^{(j-1)}(x_i)} = -2(r^{(j-1)}_{i} - F^{(j-1)}(x_i))$$

For the first iteration:

* $r^{(j-1)}_i = y_i$, the original targets
* $F^{(j-1)}(x_i)) = F^{(1)}(x_i) = \gamma$

```python
def compute_gradient(y, y_pred):
    return -2 * (y - y_pred)

y_boston_h0_residual = compute_gradient(y_boston, y_boston_h0_pred)
```

#### Training a decision-tree

We now fit a new decision-tree to the negative gradient that we just calculated, $r^{(j)}_i$:

```python
dt = DecisionTreeRegressor(max_depth=1)
dt.fit(X_boston, y_boston_h0_residual)

y_boston_h0_residual_pred = dt.predict(X_boston)
```

Typically, we will want to train the simplest, shallowest decision-trees as base learners, to avoid overfitting (which is why we are setting the maximum depth of the tree to one.)

#### Updating the global model

We now want to add the predictions made by this model (over the residuals) to the predictions made by the base model. The update rule goes as follows, where $\eta$ is a **learning rate** that controls the magnitude of the update and acts as a regularizer:

$$F^{(j)}(x_i)) = F^{(j-1)}(x_i)) - \eta \cdot \frac{\partial L(y_i, F^{(j-1)}(x_i))}{\partial F^{(j-1)}(x_i)}$$

The learning rate is also known as the *shrinkage factor*, as it shrinks the impact of the corrections if $\eta$ between 0 and 1.

Since we don't know the true value of $y_i$ at prediction time, we plug-in the prediction of the decision-tree we  fit in the previous step:

$$F^{(j)}(x_i)) \approx F^{(j-1)}(x_i)) - \eta \cdot \hat{r}^{(j)}_i(x_i) $$

If everything goes well, with each step we are moving closer and closer to the function that minimizes the squared error.

```python
def update_predictions(previous_prediction, residual_prediction, lr):
    return previous_prediction - lr * residual_prediction

y_boston_h1_pred = update_predictions(y_boston_h0_pred, y_boston_h0_residual_pred, lr=0.1)
```

Now, we compute the mean squared error for the updated predictions.

```python
mean_squared_error(y_boston, y_boston_h1_pred)
```

    70.660188943705

Hurrah! The error decreased. We would continue the process until the error stopped decreasing, or until we detected signs of overfitting.

### Putting it all together

Suppose we have $m$ individual decision-trees (boosting stages)

The final prediction will be given by the sum of the initial constant and all the gradient-based corrections:

$$F^m_i(x_i) = \gamma + \sum_{j=1}^m \eta \cdot \hat{r}^{(j)}_i(x_i)$$

Where $\hat{r}^{(j)}_i(x_i)$ is the output of a decision-tree that predicts the negative gradient of the previous iteration.

A final note: don't worry if you didn't understand all the details of previous section - it was quite heavy on the mathematical side. Just make sure you understand the basics of boosting and the idea behind it, and `sklearn` has got you covered!

## With sklearn

Enough with theory: `sklearn` to the rescue! Let's create a `GradientBoostingClassifier`, which can be used for classification problems:

```python
gb = GradientBoostingClassifier(
    learning_rate=.1,
    n_estimators=10
)

gb.fit(X_, y)
```

    GradientBoostingClassifier(n_estimators=10)

#### Include sampling

Some implementations provide the ability to sample observations and features at each iteration, similarly to what happens with Random Forests. Unsurprisingly, this reduces overfitting at the expense of increased bias.

```python
gb = GradientBoostingClassifier(
    learning_rate=.1,
    n_estimators=10,
    subsample=.5
)
gb.fit(X_, y)
```

    GradientBoostingClassifier(n_estimators=10, subsample=0.5)

# Conclusion

You are now armed with the power of bagging and boosting, two techniques you can easily use to obtain models with a lot of expressive power that also generalize better! Hopefully, you also got some insight into the world of more advanced machine learning techniques.

Make sure to review this notebook well, and when you're ready, go solve the exercises. Good luck!

S01 - Bootcamp and Binary Classification/SLU13 - Tree-Based Models/Learning notebook.ipynb


```python
%matplotlib inline
```

```python
import matplotlib.pyplot as plt
import numpy as np
import numpy.polynomial.polynomial as p
from turtle import *
```

# Basic Algebra Exercise
## Functions, Polynomials, Complex Numbers. Applications of Abstract Algebra

### Problem 1. Polynomial Interpolation
We know that if we have a set of $n$ data points with coordinates $(x_1; y_1), (x_2; y_2), \dots, (x_n; y_n)$, we can try to figure out what function may have generated these points.

Please note that **our assumptions about the data** will lead us to choosing one function over another. This means that our results are as good as our data and assumptions. Therefore, it's extremely important that we write down our assumptions (which sometimes can be difficult as we sometimes don't realize we're making them). It will be better for our readers if they know what those assumptions and models are.

In this case, we'll state two assumptions:
1. The points in our dataset are generated by a polynomial function
2. The points are very precise, there is absolutely no error in them. This means that the function should pass **through every point**

This method is called *polynomial interpolation* (*"polynomial"* captures assumption 1 and *"interpolation"* captures assumption 2).

It can be proved (look at [Wikipedia](https://en.wikipedia.org/wiki/Polynomial_interpolation) for example) that if we have $n$ data points, there is only one polynomial of degree $n-1$ which passes through them. In "math speak": "the vector spaces of $n$ points and polynomials of degree $n-1$ are isomorphic (there exists a bijection mapping one to the other)".

There are a lot of ways to do interpolation. We can also write the function ourselves if we want but this requires quite a lot more knowledge than we already covered in this course. So we'll use a function which does this for us. `numpy.polyfit()` is one such function. It accepts three main parameters (there are others as well, but they are optional): a list of $x$ coordinates, a list of $y$ coordinates, and a polynomial degree.

Let's say we have these points:
```python
points = np.array([(0, 0), (1, 0.8), (2, 0.9), (3, 0.1), (4, -0.8), (5, -1.0)])
```

First, we need to "extract" the coordinates:
```python
x = points[:, 0]
y = points[:, 1]
```

Then, we need to calculate the interpolating polynomial. For the degree, we'll set $n-1$:
```python
coefficients = np.polyfit(x, y, len(points) - 1)
poly = np.poly1d(coefficients)
```

After that, we need to plot the function. To do this, we'll create a range of $x$ values and evaluate the polynomial at each value:
```python
plot_x = np.linspace(np.min(x), np.max(x), 1000)
plot_y = poly(plot_x)
```

Finally, we need to plot the result. We'll plot both the fitting polynomial curve (using `plt.plot()`) and the points (using `plt.scatter`). It's also nice to have different colors to make the line stand out from the points.
```python
plt.plot(plot_x, plot_y, c = "green")
plt.scatter(x, y)
plt.xlabel("x")
plt.ylabel("y")
plt.show()
```
Don't forget to label the axes!

Your task now is to **wrap the code in a function**. It should accept a list of points, the polynomial degree, min and max value of $x$ used for plotting. **Be extremely careful to ensure that the function uses its parameters!**

We'll use this function to try some other cases.

```python
def interpolate_polynomial(points, degree, min_x, max_x, is_shown = True):
    """
    Interpolates a polynomial of the specified degree through the given points and plots it
    points - a list of points (x, y) to plot
    degree - the polynomial degree
    min_x, max_x - range of x values used to plot the interpolating polynomial
    """

    x_coordinates = points[:, 0]
    y_coordinates = points[:, 1]

    coefficients = np.polyfit(x_coordinates, y_coordinates, degree)
    poly = np.poly1d(coefficients)

    plot_x = np.linspace(min_x, max_x, 5000)
    plot_y = poly(plot_x)

    plt.plot(plot_x, plot_y, c = "green")
    plt.scatter(x_coordinates, y_coordinates)
    plt.xlabel("x")
    plt.ylabel("y")
    if is_shown:
        plt.show()
```

```python
points = np.array([(0, 0), (1, 0.8), (2, 0.9), (3, 0.1), (4, -0.8), (5, -1.0)])
interpolate_polynomial(points, len(points) - 1, np.min(points[:, 0]), np.max(points[:, 0]))
```

We see this is a very nice fit. This is expected, of course. Let's try to expand our view a little. Let's try to plot other values of $x$, further than the original ones. This is **extrapolation**.

```python
interpolate_polynomial(points, len(points) - 1, -5, 10)
```

Hmmm... it seems our polynomial goes a little wild outside the original range. This is to show how **extrapolation can be quite dangerous**.

Let's try a lower polynomial degree now. We used 4, how about 3, 2 and 1?
**Note:** We can add titles to every plot so that we know what exactly we're doing. Te title may be passed as an additional parameter to our function.

```python
interpolate_polynomial(points, 3, np.min(points[:, 0]), np.max(points[:, 0]))
interpolate_polynomial(points, 2, np.min(points[:, 0]), np.max(points[:, 0]))
interpolate_polynomial(points, 1, np.min(points[:, 0]), np.max(points[:, 0]))
```

```python

```

We see the fitting curves (or line in the last case) struggle more and more and they don't pass through every point. This breaks our assumptions but it can be very useful.

Okay, one more thing. How about increasing the degree? Let's try 5, 7 and 10. Python might complain a little, just ignore it, everything is fine... sort of :).

```python
interpolate_polynomial(points, 5, np.min(points[:, 0]), np.max(points[:, 0]))
interpolate_polynomial(points, 7, np.min(points[:, 0]), np.max(points[:, 0]))
interpolate_polynomial(points, 10, np.min(points[:, 0]), np.max(points[:, 0]))
```

Those graphs look pretty much the same. But that's the point exactly. I'm being quite sneaky here. Let's try to expand our view once again and see what our results really look like.

```python
interpolate_polynomial(points, 5, -10, 10)
interpolate_polynomial(points, 7, -10, 10)
interpolate_polynomial(points, 10, -10, 10)
```

Now we see there are very wild differences. Even though the first two plots look quite similar, look at the $y$ values - they're quite different.

So, these are the dangers of interpolation. Use a too high degree, and you get "the polynomial wiggle". These are all meant to represent **the same** data points but they look insanely different. Here's one more comparison.

```python
interpolate_polynomial(points, len(points) - 1, -2, 7)
interpolate_polynomial(points, len(points) + 1, -2, 7)
```

Now we can see what big difference even a small change in degree can make. This is why we have to choose our interpolating functions very carefully. Generally, a lower degree means a simpler function, which is to be preferred. See [Occam's razor](https://en.wikipedia.org/wiki/Occam%27s_razor).

And also, **we need to be very careful about our assumptions**.

```python
points = np.array([(-5, 0.03846), (-4, 0.05882), (-3, 0.1), (-2, 0.2), (-1, 0.5), (0, 1), (1, 0.5), (2, 0.2), (3, 0.1), (4, 0.05882), (5, 0.03846)])
interpolate_polynomial(points, len(points) - 1, np.min(points[:, 0]), np.max(points[:, 0]))
```

This one definitely looks strange. Even stranger, if we remove the outermost points... ($x = \pm 5$), we get this

```python
points = np.array([(-4, 0.05882), (-3, 0.1), (-2, 0.2), (-1, 0.5), (0, 1), (1, 0.5), (2, 0.2), (3, 0.1), (4, 0.05882)])
interpolate_polynomial(points, len(points - 1), np.min(points[:, 0]), np.max(points[:, 0]))
```

This is because the generating function is not a polynomial. It's actually:
$$ y = \frac{1}{1 + x^2} $$

Plot the polynomial interpolation and the real generating function **on the same plot**. You may need to modify the original plotting function or just copy its contents.

```python
points = np.array([(-4, 0.05882), (-3, 0.1), (-2, 0.2), (-1, 0.5), (0, 1), (1, 0.5), (2, 0.2), (3, 0.1), (4, 0.05882)])
min_x = np.min(points[:, 0])
max_x = np.max(points[:, 0])
interpolate_polynomial(points, len(points - 1), min_x, max_x, False)
x_plot_non_pol = np.linspace(min_x, max_x, 5000)
y_plot_non_pol = [1 / (1 + x**2) for x in x_plot_non_pol]

plt.plot(x_plot_non_pol, y_plot_non_pol)
plt.show()
```

### Problem 2. Complex Numbers as Vectors
We saw that a complex number $z = a + bi$ is equivalent to (and therefore can be represented as) the ordered tuple $(a; b)$, which can be plotted in a 2D space. So, complex numbers and 2D points are equivalent. What is more, we can draw a vector from the origin of the coordinate plane to our point. This is called a point's **radius-vector**.

Let's try plotting complex numbers as radius vectors. Don't forget to label the real and imaginary axes. Also, move the axes to the origin. Hint: These are called "spines"; you'll need to move 2 of them to the origin and remove the other 2 completely. Hint 2: You already did this in the previous lab.

We can use `plt.quiver()` to plot the vector. It can behave a bit strangely, so we'll need to set the scale of the vectors to be the same as the scale on the graph axes:
```python
plt.quiver(0, 0, z.real, z.imag, angles = "xy", scale_units = "xy", scale = 1)
```

Other than that, the main parameters are: $x_{begin}$, $y_{begin}$, $x_{length}$, $y_{length}$ in that order.

Now, set the aspect ratio of the axes to be equal. Also, add grid lines. Set the axis numbers (called ticks) to be something like `range(-3, 4)` for now.
```python
plt.xticks(range(-3, 4))
plt.yticks(range(-3, 4))

```

If you wish to, you can be a bit more clever with the tick marks. Find the minimal and maximal $x$ and $y$ values and set the ticks according to them. It's a good practice not to jam the plot too much, so leave a little bit of space. That is, if the actual x-range is $[-2; 2]$, set the plotting to be $[-2.5; 2.5]$ for example. Otherwise, the vector heads (arrows) will be "jammed" into a corner or side of the plot.

```python
def plot_complex_number(z):
    """
    Plots the complex number z as a radius vector in the 2D space
    """
    ax = plt.gca()
    ax.spines["bottom"].set_position("zero")
    ax.spines["left"].set_position("zero")
    ax.spines["top"].set_visible(False)
    ax.spines["right"].set_visible(False)
    plt.xlabel("real")
    plt.ylabel("imaginary")
    plt.quiver(0, 0, z.real, z.imag, angles = "xy", scale_units = "xy", scale = 1)
    rounded_x = int(round(z.real))
    rounded_y = int(round(z.imag))
    plt.xticks(range(-rounded_x, rounded_x + 2))
    plt.yticks(range(-rounded_y, rounded_y + 2))
    plt.show()

plot_complex_number(2 + 3j)
```

How about many numbers? We'll need to get a little bit more creative. First, we need to create a 2D array, each element of which will be a 4-element array: `[0, 0, z.real, z.imag]`. Next, `plt.quiver()` can accept a range of values. Look at [this StackOverflow post](https://stackoverflow.com/questions/12265234/how-to-plot-2d-math-vectors-with-matplotlib) for details and adapt your code.

```python
def plot_complex_numbers(numbers, colors):
    """
    Plots the given complex numbers as radius vectors in the 2D space
    """
    coordinates = [[0, 0, n.real, n.imag] for n in numbers]
    X, Y, U, V = zip(*coordinates)
    ax = plt.gca()
    ax.spines["bottom"].set_position("zero")
    ax.spines["left"].set_position("zero")
    ax.spines["top"].set_visible(False)
    ax.spines["right"].set_visible(False)
    plt.xlabel("real")
    plt.ylabel("imaginary")
    ax.set_aspect("equal")
    ax.quiver(X, Y, U, V, angles='xy', scale_units='xy', scale=1, color= colors)
    max_x = max([int(round(n.real)) for n in numbers])
    max_y = max([int(round(n.imag)) for n in numbers])
    max_xy = max(max_x, max_y)
    ax.set_xlim([-0.5 -max_x, max_x + 0.5])
    ax.set_ylim([-0.5 -max_y, max_y + 0.5])
    plt.show()

plot_complex_numbers([2 + 3j, -2 - 1j, -3, 2j], ["green", "red", "blue", "orange"])
```

Now let's see what the operations look like. Let's add two numbers and plot the result.

```python
z1 = 2 + 3j
z2 = 1 - 1j
plot_complex_numbers([z1, z2, z1 + z2], ["red", "blue", "green"])
```

We can see that adding the complex numbers is equivalent to adding vectors (remember the "parallelogram rule"). As special cases, let's try adding pure real and pure imaginary numbers:

```python
z1 = 2 + 3j
z2 = 2 + 0j
plot_complex_numbers([z1, z2, z1 + z2], ["red", "blue", "green"])
```

```python
z1 = 2 + 3j
z2 = 0 + 2j
plot_complex_numbers([z1, z2, z1 + z2], ["red", "blue", "green"])
```

How about multiplication? First we know that multiplying by 1 gives us the same vector and mulpiplying by -1 gives us the reversed version of the same vector. How about multiplication by $\pm i$?

```python
z = 2 + 3j
plot_complex_numbers([z, z * 1], ["red", "blue"])
plot_complex_numbers([z, z * -1], ["red", "blue"])
plot_complex_numbers([z, z * 1j], ["red", "blue"])
plot_complex_numbers([z, z * -1j], ["red", "blue"])
```

So, multiplication by $i$ is equivalent to 90-degree rotation. We can actually see the following equivalence relationships between multiplying numbers and rotation about the origin:

| Real | Imaginary | Result rotation |
|------|-----------|-----------------|
| 1    | 0         | $0^\circ$       |
| 0    | 1         | $90^\circ$      |
| -1   | 0         | $180^\circ$     |
| 0    | -1        | $270^\circ$     |

Once again, we see the power of abstraction and algebra in practice. We know that complex numbers and 2D vectors are equivalent. Now we see something more: addition and multiplication are equivalent to translation (movement) and rotation!

Let's test the multiplication some more. We can see the resulting vector is the sum of the original vectors, but *scaled and rotated*:

```python
z1 = 2 + 3j
z2 = 1 - 2j
plot_complex_numbers([z1, z2, z1 * z2], ["red", "blue", "green"])
```

### Problem 3. Recursion and Fractals

> "To understand recursion, you first need to understand recursion."

There are three main parts to a recursive function:
1. Bottom - when the recursion should finish
2. Operation - some meaningful thing to do
3. Recursive call - calling the same function
4. Clean-up - returning all data to its previous state (this reverses the effect of the operation)

Let's do one of the most famous recursion examples. And I'm not talking about Fibonacci here. Let's draw a tree using recursive functions.

The figure we're going to draw is called a **fractal**. It's self-similar, which means that if you zoom in on a part of it, it will look the same. You can see fractals everywhere in nature, with broccoli being one of the prime examples. Have a look:

First, we need to specify the recursive part. In order to draw a tree, we need to draw a line of a given length (which will be the current branch), and then draw two more lines to the left and right. By "left" and "right", we should mean "rotation by a specified angle".

So, this is how to draw a branch: draw a line and prepare to draw two more branches to the left and right. This is going to be our recursive call.

To make things prettier, more natural-looking (and have a natural end to our recursion), let's draw each "sub-branch" a little shorter. If the branch becomes too short, it won't have "child branches". This will be the bottom of our recursion.

There's one more important part of recursion, and this is **"the clean-up"**. After we did something in the recursive calls, it's very important to return the state of everything as it was **before** we did anything. In this case, after we draw a branch, we go back to our starting position.

Let's first import the most import-ant (no pun intended...) Python drawing library: `turtle`! In order to make things easier, we'll import all methods directly.
```python
from turtle import *
```

You can look up the docs about turtle if you're more interested. The basic things we're going to use are going forward and backward by a specified number of pixels and turning left and right by a specified angle (in degrees).

Let's now define our recursive function:
```python
def draw_branch(branch_length, angle):
    if branch_length > 5:
        forward(branch_length)
        right(angle)
        draw_branch(branch_length - 15, angle)
        left(2 * angle)
        draw_branch(branch_length - 15, angle)
        right(angle)
        backward(branch_length)
```

And let's call it:
```python
draw_branch(100, 20)
```

We need to start the tree not at the middle, but toward the bottom of the screen, so we need to make a few more adjustments. We can wrap the setup in another function and call it. Let's start one trunk length below the center (the trunk length is the length of the longest line).

**Note:** It's important to call `done()` after the drawing is finished. If you miss it, the window with the turtle drawing will freeze and throw an exception.

```python
def draw_tree(trunk_length, angle):
    speed("fastest")
    left(90)
    up()
    backward(trunk_length)
    down()
    draw_branch(trunk_length, angle)
    done()
```

Note that the graphics will show in a separate window. Also note that sometimes you might get bugs. If you do, go to Kernel > Restart.

```python
def draw_branch(branch_length, angle):
    if branch_length > 5:
        forward(branch_length)
        right(angle)
        draw_branch(branch_length - 15, angle)
        left(2 * angle)
        draw_branch(branch_length - 15, angle)
        right(angle)
        backward(branch_length)

def draw_tree(trunk_length, angle):
    speed("fastest")
    left(90)
    up()
    backward(trunk_length)
    down()
    draw_branch(trunk_length, angle)
    done()
```

```python
draw_tree(100, 20)
```

Experiment with different lengths and angles. Especially interesting angles are $30^\circ$, $45^\circ$, $60^\circ$ and $90^\circ$.

```python
draw_tree(100, 30)
```

```python
draw_tree(100, 45)
```

```python
draw_tree(100, 90)
```

Now modify the original function a little. Draw the lines with different thickness. Provide the trunk thickness at the initial call. Similar to how branches go shorter, they should also go thinner.

```python
def draw_branch(branch_length, angle):
    if branch_length > 5:
        forward(branch_length)
        right(angle)
        draw_branch(branch_length - 15, angle)
        left(2 * angle)
        draw_branch(branch_length - 15, angle)
        right(angle)
        backward(branch_length)

def draw_tree(trunk_length, angle, thickness):
    speed("fastest")
    pensize(thickness)
    left(90)
    up()
    backward(trunk_length)
    down()
    draw_branch(trunk_length, angle)
    done()
```

```python
draw_tree(100, 90, 10)
```

#### * Optional problem
Try to draw another kind of fractal graphic using recursion and the `turtle` library. Two very popular examples are the "Koch snowflake" and the "Sierpinski triangle". You can also modify the original tree algorithm to create more natural-looking trees. You can, for example, play with angles, number of branches, lengths, and widths. The Internet has a lot of ideas about this :). Hint: Look up **"L-systems"**.

### Problem 4. Run-length Encoding
One application of algebra and basic math can be **compression**. This is a way to save data in less space than it originally takes. The most basic form of compression is called [run-length encoding](https://en.wikipedia.org/wiki/Run-length_encoding).

Write a function that encodes a given text. Write another one that decodes.

We can see that RLE is not very useful in the general case. But it can be extremely useful if we have very few symbols. An example of this can be DNA and protein sequences. DNA code, for example, has only 4 characters.

Test your encoding and decoding functions on a DNA sequence (you can look up some on the Internet). Measure how much your data is compressed relative to the original.

```python
def encode(text):
    """
    Returns the run-length encoded version of the text
    (numbers after symbols, length = 1 is skipped)
    """
    textLength = len(text)
    result = ""
    previousChar = ""
    continuityCounter = 1
    for i, char in enumerate(text):
        if i == (textLength - 1):
            if char == previousChar:
                continuityCounter += 1
            else:
                result += get_char_num(previousChar, continuityCounter)
                previousChar = char
                continuityCounter = 1
            result += get_char_num(previousChar, continuityCounter)
        elif char == previousChar:
            continuityCounter += 1
        else:
            result += get_char_num(previousChar, continuityCounter)
            previousChar = char
            continuityCounter = 1
    return result

def get_char_num(char, counter):
    return char + (str(counter) if counter != 1 else "")

def decode(text):
    """
    Decodes the text using run-length encoding
    """
    lastIndex = len(text)
    result = ""
    i = 0
    while i < len(text):
        char = text[i]
        if i == lastIndex - 1:
            result += char
            break
        nextChar = text[i + 1]
        if nextChar.isdigit():
            result += char * int(nextChar)
            i += 1
        else:
            result += char
        i += 1
    return result
```

```python
# Tests
# Test that the functions work on their own
assert encode("AABCCCDEEEE") == "A2BC3DE4"
assert decode("A2BC3DE4") == "AABCCCDEEEE"

# Test that the functions really invert each other
assert decode(encode("AABCCCDEEEE")) == "AABCCCDEEEE"
assert encode(decode("A2BC3DE4")) == "A2BC3DE4"
```

### * Problem 5. Function Invertibility and Cryptography
As we already saw, some functions are able to be inverted. That is, if we know the output, we can see what input generated it directly. This is true if the function is **one-to-one correspondence** (bijection).

However, not all functions are created the same. Some functions are easy to compute but their inverses are extremely difficult. A very important example is **number factorization**. It's relatively easy (computationally) to multiply numbers but factoring them is quite difficult. Let's run an experiment.

We'll need a function to generate random n-bit numbers. One such number can be found in the `Crypto` package
```python
from Crypto.Util import number
random_integer = number.getRandomNBitInteger(n_bits)
```

We could, of course, write our factorization by hand but we'll use `sympy`
```python
from sympy.ntheory import factorint
factorint(1032969399047817906432668079951) # {3: 2, 79: 1, 36779: 1, 7776252885493: 1, 5079811103: 1}
```

This function returns a `dict` where the keys are the factors, and the values - how many times they should be multiplied.

We'll also need a tool to accurately measure performance. Have a look at [this one](https://docs.python.org/3/library/time.html#time.time) for example.

Specity a sequence of bit lengths, in increasing order. For example, you might choose something like `[10, 20, 25, 30, 32, 33, 35, 38, 40]`. Depending on your computer's abilities you can go as high as you want. For each bit length, generate a number. See how much time it takes to factor it. Then see how much time it takes to multiply the factors. Be careful how you measure these. You shouldn't include the number generation (or any other external functions) in your timing.

In order to have better accuracy, don't do this once per bit length. Do it, for example, five times, and average the results.

Plot all multiplication and factorization times as a function of the number of bits. You should see that factorization is much, much slower. If you don't see this, just try larger numbers :D.

```python
# Write your code here
```

### * Problem 6. Diffie - Hellman Simulation
As we already saw, there are functions which are very easy to compute in the "forward" direction but really difficult (computationally) to invert (that is, determine the input from the output). There is a special case: the function may have a hidden "trap door". If you know where that door is, you can invert the function easily. This statement is at the core of modern cryptography.

Look up **Diffie - Hellman key exchange** (here's a [video](https://www.youtube.com/watch?v=cM4mNVUBtHk) on that but feel free to use anything else you might find useful).

Simulate the algorithm you just saw. Generate large enough numbers so the difference is noticeable (say, factoring takes 10-15 seconds). Simulate both participants in the key exchange. Simulate an eavesdropper.

First, make sure after both participants run the algotihm, they have *the same key* (they generate the same number).

Second, see how long it takes for them to exchange keys.

Third, see how long it takes the eavesdropper to arrive at the correct shared secret.

You should be able to see **the power of cryptography**. In this case, it's not that the function is irreversible. It can be reversed, but it takes a really long time (and with more bits, we're talking billions of years). However, if you know something else (this is called a **trap door**), the function becomes relatively easy to invert.

```python
# Write your code here
```

### ** Problem 7. The Galois Field in Cryptography
Research about the uses of the Galois field. What are its properties? How can it be used in cryptography? Write a simple cryptosystem based on the field.

You can use the following questions to facilitate your research:
* What is a field?
* What is GF(2)? Why is it an algebraic field?
* What is perfect secrecy? How does it relate to the participants in the conversation, and to the outside eavesdropper?
* What is symmetrical encryption?
* How to encrypt one-bit messages?
* How to extend the one-bit encryption system to many buts?
* Why is the system decryptable? How do the participants decrypt the encrypted messages?
* Why isn't the eavesdropper able to decrypt?
* What is a one-time pad?
    * How does the one-time pad achieve perfect secrecy?
* What happens if we try to use a one-time pad many times?
    * Provide an example where you break the "many-time pad" security
* What are some current enterprise-grade applications of encryption over GF(2)?
* Implement a cryptosystem based on GF(2). Show correctness on various test cases

### ** Problem 8. Huffman Compression Algorithm
Examine and implement the **Huffman algorithm** for compressing data. It's based on information theory and probiability theory. Document your findings and provide your implementation.

This algorithm is used for **lossless compression**: compressing data without loss of quality. You can use the following checklist:

* What is the difference betwenn lossless and lossy compression?
* When can we get away with lossy compression?
* What is entropy?
* How are Huffman trees constructed?
    * Provide a few examples
* How can we get back the uncompressed data from the Huffman tree?
* How and where are Huffman trees stored?
* Implement the algorithm. Add any other formulas / assumptions / etc. you might need.
* Test the algorithm. A good meaure would be percentage compression: $$\frac{\text{compressed}}{\text{uncompressed}} * 100\%$$
* How well does Huffman's algorithm perform compared to other compression algorithms (e.g. LZ77)?


# *FEniCS NB:* Laser Pulse Crossing a Crystal
### *File:*    fn_simp_06c.py — propagate the laser pulse, now with a static crystal, _i.e._ $\Delta \neq 0, \partial_t\Delta=0$

The Frantz-Nodvik equations
$$
\begin{align}
  \partial_t n + c\partial x &= \sigma c n \Delta, \\
  \partial_t \Delta          &= -\gamma \sigma c n \Delta, \\
\end{align}
$$
are a pair of coupled, first-order PDEs that describe
the propagation of a one-dimensional laser pulse across a
slab of laser gain material composed of two-level atoms.
Here, $x$ and $t$ denote respectively distance along the beam axis and
the time; $n(x,t)$ denotes photon number density in the medium;
and $\Delta(x,t)$ denotes the “population inversion”, $N_2 - N_1$,
giving the difference between the number density of atoms
in the excited state as compared to those in the ground state.
In addition, $c$ denotes the speed of light in the medium,
$\sigma$ the resonance absorption cross-section,
and $\gamma$ a factor related to the relative degeneracy
of the ground and excited states.

This problem has both spatial and temporal boundary conditions (BCs).
For the spatial BC, we apply Dirichlet to the left-hand edge,
and leave the right-hand edge free.
For the temporal BC, we specify the initial population inversion
across the domain of interest, and the initial profile of the
incident laser pulse.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

# import os
# import sys
# from IPython.display import Image
# from IPython.core.interactiveshell import InteractiveShell
# InteractiveShell.ast_node_interactivity = "all"
```

```python
from fenics import *
from mshr import *
```

Describe the simulation: parameters, domain, boundary conditions, initial conditions.

```python
# set simulation parameters: domain, crystal, laser, time step
# -- simulation domain
Ld     = 1.00          # length of simulation domain (we force, see below, domain to [0,1])
mesh_d = 226           # mesh density
ds     = Ld / mesh_d   # size of individual mesh cells
# -- laser pulse
p_w  = (50 - 4) * ds   # laser pulse width
p_hw = p_w / 2         # laser pulse half-width
p_x0 = p_hw + 2 * ds   # laser pulse starting point
p_n0 = 1.00e02         # incident photon density
cl = 1.00              # speed of light in vacuum, β_0
cm = 163 * ds          # speed of light in crystal, β_c
# -- crystal
c_w   = 70 * ds        # crystal width
c_hw  = c_w / 2        # crystal half-width
c_x0  = 85 * ds        # crystal center
c_D0  = 1.00e06        # initial population inversion density
sigma = 2.70e-6        # resonance absorption cross-section
# -- time step
T       = 183 * ds     # total simulation time
n_steps =  800         # number of time steps
dt      = T / n_steps  # size of time step
nip     = 100          # number of intervals between plots

# create domain: a 1D mesh on the unit interval [0,1]
mesh = UnitIntervalMesh(mesh_d)
V = FunctionSpace(mesh, "P", 2)

# define boundary functions: left and right ends
def on_L(x, on_boundary):
    return (on_boundary and near(x[0], 0.))

def on_R(x, on_boundary):
    return (on_boundary and near (x[0], 1.))

# specify boundary conditions
# (leaving RH boundary free)
bc_L = DirichletBC(V, Constant(0), on_L)
bc_R = DirichletBC(V, Constant(0), on_R)
bc = [bc_L, bc_R]
# bc = bc_L

# specify initial conditions
# -- define profile of incident laser pulse: half wave near left edge of domain
nph_init = Expression('abs(x[0] - p_x0) < p_hw ? p_n0 * cos(pi * (x[0] - p_x0) / p_w) : 0.',
                       degree=2, p_w=p_w, p_hw=p_hw, p_x0=p_x0, p_n0=p_n0)
# -- define profile of crystal's initial population inversion
Dcr_init = Expression('abs(x[0] - c_x0) < c_hw ? c_D0 * cos(pi * (x[0] - c_x0) / c_w) : 0.',
                    degree=2, c_w=c_w, c_hw=c_hw, c_x0=c_x0, c_D0=c_D0)
# -- initialize nph_j = nph_init
nph_j = interpolate(nph_init, V)
# -- initialize Dc_j = nph_init
Dcr_j = interpolate(Dcr_init, V)
```

Set up the variational problem.

```python
# define variational problem
nph = TrialFunction(V)
# Dcr = TrialFunction(V)
v   = TestFunction(V)
F = ( nph - nph_j + cm * dt * nph.dx(0) - sigma * cm * dt * Dcr_init * nph) * v * dx  # ∂n/∂t + c ∂n/∂x = σ c Δ n
# F = ( (Dcr - Dcr_j) / dt ) * v * dx  # ∂Δ/∂t = 0
a, L = lhs(F), rhs(F)
```

Execute simulation.

```python
# initializations
nph = Function(V)
# Dcr= Function(V)
t = 0
plt.figure(figsize=(13,8))
plt.title("population inversion density")

# time steps
for ii in range(n_steps):

    # update time
    t += dt

    # compute solution
    solve(a == L, nph, bc)

    # update plot at nip-step intervals
    if ii % nip == 0:
        plot(nph)

    # update previous solution
    nph_j.assign(nph)

# final envelope of laser pulse (if necessary)
if ii % nip != 0:
    plot(nph)

# finish plot
# plt.xlim([0.00, 0.20])
plt.ylim([0.00, 175.0])
# plt.axvline(c_x0 - c_hw,   c='b', lw=1.2)
# plt.axvline(c_x0 + c_hw,   c='b', lw=1.2)
plt.axvline(p_x0,          c='g', lw=1.2)
plt.axvline(p_x0 + cm * T, c='r', lw=1.2)
plt.axhline(p_n0 * 0.970,  c='b', lw=0.5)

plot(Dcr_j / (c_D0 / p_n0), ls='-.', c='b', lw=1.2)
plt.grid(True)
plt.show()
plt.close()
```

The following approach is useful for 2D and 3D systems,
when overlaying plots may not make sense.

```python
# define time-evolution function
def evolve():

    # initialization
    nph = Function(V)
    t = 0

    # time steps
    for ii in range(n_steps):

        # update current time
        t += dt

        # compute solution
        solve(a == L, nph, bc)

        # report solution at nip-step intervals
        if ii % nip == 0:
            yield nph

        # update previous solution
        nph_j.assign(nph)

    # final envelope of laser pulse (if necessary)
    if ii % nip == 0:
        yield nph
```

```python
n_rows = 3
n_cols = 5
fig_wd = 15
# default sizing here yields unit aspect ratio
plt.figure(figsize = (fig_wd, fig_wd * n_rows // n_cols))
plt.subplots_adjust(wspace=0.3, hspace=0.3)

idx = 0
for u in evolve():
    idx += 1
    plt.subplot(n_rows, n_cols, idx)
    plot(u)
#     plot(u, vmin=u_min, vmax=u_max)
```

```python

```


# Task 1: Introduction

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
```

# Task 2: Dataset

Real estate agent table:

|Area|Distance|Price|
|---|---|---|
|70|3|21200|
|50|1|22010|
|120|9|24305|
|100|2|31500|

You can write the relationship with a 2-variable linear equation:

$
\begin{equation}
y = b + w_1.x_1 + w_2.x_2
\end{equation}
$

In a vector form:

$
\begin{equation}
y = b + (w_1 w_2).\binom{x_1}{x_2}
\end{equation}
$

Where
$
\begin{equation}
W = (w_1 w_2)
\end{equation}
$
and
$
\begin{equation}
X = \binom{x_1}{x_2}
\end{equation}
$

```python
def generate_example(num = 1000):
    W = [1.0, -3.0] #vector value lets asssume actual value for w is this, 1 row 2 columns, making it 2 dimentional

    b = 1.0 #scalar value

    W = np.reshape(W, (2, 1)) # shape = (2,1), 2 for number of elements,w1, w2 and 1 for total w elements in its ,i.e.  number of elements in it. 2 coln 1 rows

    X = np.random.randn(num, 2) # 2 because we have x1 and x2, x1= Area, x2= Distance

    y = b + np.dot(X, W) + np.random.randn() # dot product , return 1000 values, add some noise to add some real data

    y = np.reshape(y, (num, 1)) #to make sure we explicitely set the dimentions , y will be list of 1000 numbers, we are adding a dimention here so that we dont get any error later one as we do more of matric computations

    return X, y

```

```python
X, y = generate_example()
```

```python
print(X.shape, y.shape)
#(1000,2) (1000,1) ,i.e. 1000 examples n 2 features/independent variable
#and 1000 examples and 1 dependent variable

```

    (1000, 2) (1000, 1)

```python
print(X[99], y[0])
```

    [-1.20532326 -0.36443401] [1.58045968]

# Task 3: Initialize Parameters

The loss over **m** examples:

$
\begin{equation}
J = \frac{1}{2m} \sum_{i=1}^{m} (y - \hat{y})^2
\end{equation}
$

The objective of the gradient descent algorithm is to minimize this loss value.

Gradient Descent Objective is to
$
\begin{equation}
min(J)
\end{equation}
$

```python
class Model:
    def __init__(self, num_features):
        self.num_features = num_features
        self.W = np.random.randn(num_features, 1)
        self.b = np.random.randn()

```

```python
model = Model(2)
print(model.W)
print(model.b)
```

    [[-0.05828935]
     [ 0.08746248]]
    -1.3758653757662547

# Task 4: Forward Pass

The gradient descent algorithm can be simplified in 4 steps:

1. Get predictions y_hat for X with current values of W and b.
2. Compute the loss between y and y_hat
3. Find gradients of the loss with respect to parameters W and b
4. Update the values of W and b by subtracting the gradient values obtained in the previous step

Let's simplify our linear equation a bit more for an example:
$
\begin{equation}
y = wx
\end{equation}
$

Let's plot J as a function of w

The gradients of loss with respect to w:

\begin{equation}
\frac{dJ}{dw} = \frac{\delta{J}}{\delta{w}} = \lim_{\epsilon \to 0} \frac{J(w + \epsilon) - J(w)}{\epsilon}
\end{equation}

```python
class Model(Model):
    def forward_pass(self, X):
        y_hat = self.b + np.dot(X, self.W)
        return y_hat

```

```python
y_hat = Model(2).forward_pass(X) # 2 features , area and distance
print(y_hat.shape[0], y_hat[0])
```

    1000 [0.9267226]

# Task 5: Compute Loss

The loss over **m** examples:

$
\begin{equation}
J = \frac{1}{2m} \sum_{i=1}^{m} (y - \hat{y})^2
\end{equation}
$

```python
class Model(Model):
    def compute_loss(self, y_hat, y_true):
        return  np.sum(np.square(y_hat - y_true))/(2*y_hat.shape[0]) # y_hat.shape[0] is 1000
```

```python
model = Model(2)
y_hat = model.forward_pass(X)  # return predction
loss = model.compute_loss(y_hat, y)
```

```python
loss
```

    7.4322017765225175

# Task 6: Backward Pass

The gradient of loss with respect to bias can be calculated with:

$
\begin{equation}
\frac{dJ}{db} = \frac{1}{m} \sum_{i=1}^{m} (\hat{y^{(i)}} - y^{(i)})
\end{equation}
$

$
\begin{equation}
\frac{dJ}{dW_j} = \frac{1}{m} \sum_{i=1}^{m} (\hat{y^{(i)}} - y^{(i)}).x_j^{(i)}
\end{equation}
$

```python
class Model(Model):
    def backward_pass(self, X, y_true, y_hat):
        m = y_true.shape[0]
        db = (1/m) * np.sum(y_hat - y_true)
        dw = (1/m) * np.sum(np.dot(np.transpose(y_hat - y_true), X), axis = 0)
        return dw, db

```

```python
model = Model(2)
x, y = generate_example()
y_hat = model.forward_pass(X)
dw,db = model.backward_pass(X, y, y_hat)
```

```python
print(dw, db)
```

    [0.69264906 0.52019447] -2.8254536853922665

# Task 7: Update Parameters

```python
class Model(Model):
    def update_params(self, dw, db, lr): # learning rate
        self.W = self.W - lr * np.reshape(dw, (self.num_features, 1)) # should hv same shape as W
        self.b = self.b - db # nothing to do with shape as its scalar value

```

# Task 8: Training Loop

```python
class Model(Model):
    def train(self, x_train, y_train, iterations, lr):
        losses = []
        for i in range(0, iterations):
            y_hat = self.forward_pass(x_train)
            loss = self.compute_loss(y_hat, y_train)
            dw, db = self.backward_pass(x_train, y_train, y_hat)
            self.update_params(dw, db, lr)
            losses.append(loss)
            if i%int(iterations/10) == 0:
                print('Iter: {}, Loss: {:.4f}'.format(i, loss))
        return losses
```

model = Model(2)

```python
model = Model(2)
```

```python
x_train, y_train = generate_example()
```

```python
losses = model.train(x_train, y_train, 1000, 3e-3)
```

    Iter: 0, Loss: 0.0103
    Iter: 100, Loss: 0.0055
    Iter: 200, Loss: 0.0030
    Iter: 300, Loss: 0.0016
    Iter: 400, Loss: 0.0009
    Iter: 500, Loss: 0.0005
    Iter: 600, Loss: 0.0002
    Iter: 700, Loss: 0.0001
    Iter: 800, Loss: 0.0001
    Iter: 900, Loss: 0.0000

```python
plt.plot(losses)
```

# Task 9: Predictions

```python
model_untrain = Model(2)
x_test, y_test = generate_example(500)
print(x_test.shape, y_test.shape)

```

    (500, 2) (500, 1)

```python
preds_untrained = model_untrain.forward_pass(x_test)
preds_trained = model.forward_pass(x_test)
```

```python
plt.figure(figsize =(6,6))
plt.plot(preds_untrained, y_test, 'rx', label = 'Untrained')
plt.plot(preds_trained, y_test, 'b.', label = 'Trained')
plt.legend()
plt.xlabel('Predictions')
plt.ylabel('Ground Truth')
plt.show()
```

```python

```


# Bayesian Inference of the brain and the derivation of the Free Energy Principle

The idea of the Free Energy Principle has its foundations in Bayesian Inference, specifically Variational Bayesion Methods, as well as the theory of predictive coding. 'Inference' in this case means infering the probability of a hidden state (or hypothesis) given an observation (evidence). This brings us to Bayes' formula:

$$p(H|E) = \displaystyle\frac{p(E|H)p(H)}{p(E)}$$

To see how this equation works in practice, let's introduce an example, taken from [Bogacz's paper](https://www.sciencedirect.com/science/article/pii/S0022249615000759).

Consider a simple life form with one photo-sensitive receptor. This life form scuttles around trying to determine the size of objects around it based on reflective light. Let the size of an object = $v$. This simple life form has a difficult time perceiving the correct light intesnity as its photo-receptor provides a noisy input. Therefore when the size of an object is $v$, the organism perceives a normally distributed light intensity $p(u|v) =N(g(v), \sum_u)$ where $g$ is a mapping of size to light intensity approximated by the organism. For now let $g(v) = v^2$. Addtionally, the organism has a prior assumption on the sizes of objects around it. For simplicity, let's imagine this prior as a normal distribution with mean $v_p$ and variance $\sum_p$. Therefore $p(v) = N(v_p,\sum_p)$. We now have all the components necessary to construct Bayes' theorem with $u$ as the evidence and $v$ as the hypothesis:

$p(v|u) = \displaystyle\frac{p(u|v)p(v)}{p(u)}$

This, however, presents a problem. Calculating the numerator is easy, as it is described by the multiplication of two known normal distributions (stated above). The denominator, however, by the law of total probability, is expressed as: $p(u) = \int p(u|v)p(v)dv$. In many scenarios, these integrals are intractable, and are especially impossible for organisms utilising neuronal connections for calculation. Therefore, the posterior distribution $p(v|u)$ needs to be calculated some other way. This is where Variational Bayes comes in. We might not be able to calculate the exact posterior, but we can try approximate it. To start, lets just consider a very coarse approximation of determining the mode of the posterior. In the case of our organism, the mode is important as it represents the most likely $v$ given the observed $u$ and the prior. It is reasonable to think that it is realistic and useful for organisms to only entertain most likely hypotheses rather than the probability of all possible hypotheses. Therefore we want to find the value of $v$ which is the maximum point of the posterior. Let this $v = \phi = \argmax_v p(v|u)$. The important part about estimating just one value of $v$ is that the denominator $p(u) = \int p(u|v)p(v)$ does not depend on $\phi$ and is constant for any value $v$ of which we want to calculate the probability. Finding $\phi$ then simply becomes a case of finding the value of $v$ which maximises the numerator: $p(u|v)p(v)$ putting this all together, we have:

$max_vp(v|u) = p(u|\phi)p(\phi)$

Now from here all we need do is iteratively determine $\phi$. This can be achieved via gradient ascent. We take the logarithm of the numerator because the maximum of it is the same as the maximum of $p(u|v)p(v)$ and is an easier function to work with as calculating $p(u|v)p(v)$ involves exponentiation:

$$
\begin{equation}F = ln(p(u|v)p(v))
\end{equation}
$$
$$
\begin{equation}= ln(pu|v) + ln p(v)
\end{equation}
$$

$= ln \bigg[\displaystyle\frac{1}{\sqrt{2\pi\Sigma_p}}\bigg(-]\displaystyle\frac{(\phi - v_p)^2}{2\Sigma_p}\bigg)\bigg] + ln \bigg[\displaystyle\frac{1}{\sqrt{2\pi\Sigma_u}}exp\bigg(-\displaystyle\frac{(u - g(\phi))^2}{2\Sigma_u}\bigg)\bigg]$

$...$

$...$

$ = \displaystyle\frac{1}{2}\bigg(-\displaystyle\frac{(\phi - v_p)^2}{\Sigma_p}- \displaystyle\frac{(u-g(\phi))^2}{\Sigma_u}\bigg) + C$

In the last line, all the constant terms not involving $\phi$ have been incorporated into a constant $C$ as they will disappear when the derivative is taken:

$ \displaystyle\frac{\partial F}{\partial \phi} = \displaystyle\frac{v_p - \phi}{\Sigma_p} + \displaystyle\frac{u-g(\phi)}{\Sigma_u}g'(\phi)$

With this equation we can pefrom gradient ascent until $\theta$ converges on a value. This will be the value which maximises $F$ and therefore maximises $p(v|u)$.
Looking at the form of the equation, it is evident that the gradient of $F$ is being influenced in two different ways. One by how much the approximated hypothesis $\phi$ is different from the prior mean hypothesis and the other by how different the observation, $u$, is from the expected observation given the approxmated hypothesis $g(\phi)$. There seems to be a tradeoff happening here. The prior and the [likelihood](https://en.wikipedia.org/wiki/Likelihood_function) are each pulling the posterior toward their mean values, with it ultimately being a weighted average between the two. The weighting here is exactly determined by the variance of the two terms respectively, with higher variance resulting in a less 'reliable' contribution to the posterior. Ths makes sense, as the more noisy the prior or observation is, the less one would want it to contribute to the inference of a posterior hypothesis.

To frame this in terms of neuronal activity, let us denote these two terms in the derviative of $F$ as:

$\displaystyle\epsilon_p = \frac{\phi - v_p}{\Sigma_p}$
$\displaystyle\epsilon_u = \frac{u - g(\phi)}{\Sigma_u}$

As alluded to above, these can be viewed as **weighted prediction errors** and could be realised in a simple neuronal structure as follows:

Here lines with arrows denote excitatory connections while lines with circles denote inhibitory connections. The circular connection between the prediction error nodes and the inference node allows for iterative update of all three as the maximum posterior is calculated. The update equations of the three inner nodes are as follows:

$\displaystyle\phi_{new} = \phi_{old} + a(\epsilon_ug'(\phi) - \epsilon_p)$

$\displaystyle\epsilon_{u_{new}} = \epsilon_{u_{old}} + a(u - g(\phi_{new}) - \epsilon_{u_{old}}\Sigma_u)$

$\displaystyle\epsilon_{p_{new}} = \epsilon_{p_{old}} + a(\phi_{new} - v_p - \epsilon_{p_{old}}\Sigma_p)$

where $0<a<1$ is some constant to allow for better convergence. Note that in the update equation for $\phi$, the quantity from the excitatory 'evidence' neuron $\epsilon_ug'(\phi)$ is being added, while the quantity from the inhibitory 'prior' neuron $\epsilon_p$ is being subtracted. This exemplified the two componenets 'pulling' the posterior in different directions as stated above.
Also note that the error neurons are being self-inhibited by their respective variances, which represent the weighting on the components used in calculating the inferenced hypothesis.

The next step for the organism would be to shift its prior and likelihood calculation so as to better represent the new observation. In this case, it would mean changing the $v_p, \Sigma_p$ and $\Sigma_u$ as well as sometimes the transformation function $g$. Essentailly, the organsim would like to maximise $p(u)$ on average. If it is constantly inferring hypotheses that it gives little prior probability to, $p(v)$, and little likelihood to, $p(u|v)$ perhaps it should adjust the parameters that determine these two factors. In bayesian Inference, this is what model evidence is, referred to in frequentist statistics as marginal likelihood. However trying to maximise $p(u)$ once again leaves us with the problem of the intractable marginalisation: $p(u) = \int p(u|v)p(v)dv$. To avoid this we can turn to maximising a related and familiar expression: $p(u, \phi)$. At first glance it might not seem clear why maximising this term, rather the whole model evidence, is useful. Remember that $p(u)$ essentially represents a weighted average of all possible likelihoods. However given that we are not necessarily attempting to calculate the entire posterior distribution, rather just one point of the posterior, it is reasonable to only care about maximising model evidence with respect to this point. Note that $p(u,\phi) = p(u|\phi)p(\phi) = F$. Therefore by maximising $F$ with respect to the various parameters mentioned above will maximise the model evidence for the point of maximum likelihood, $\phi$. The update to these parameters would therefore be proportional to the derivative of $F$ with respect to each:

$\displaystyle\frac{\partial F}{\partial v_p} = \displaystyle\frac{\phi - v_p}{\Sigma_p}$

$\displaystyle\frac{\partial F}{\partial \Sigma_p} = \displaystyle\frac{1}{2}\bigg(\displaystyle\frac{(\phi - v_p)^2}{\Sigma_p^2} - \displaystyle\frac{1}{\Sigma_p}\bigg)$

$\displaystyle\frac{\partial F}{\partial \Sigma_u} = \displaystyle\frac{1}{2}\bigg(\displaystyle\frac{(u - g(\phi))^2}{\Sigma_u^2} - \displaystyle\frac{1}{\Sigma_u}\bigg)$

Additionally, the translation, $g$ may be incorrect. In its most complex form, updating this function would require the approximating power of a separate neural network. In its simplest form, the function form is static but weighted by a parameter $\theta$ which can represent the strength of the connection between the neurons which pass the translated data - in this case between the neurons computing $\epsilon_u$ and $\phi$ (see picture above). The update update to this weighting would therefore be:
$\displaystyle\frac{\partial F}{\partial \theta} = \epsilon_u\phi$

### Free Energy

All this calculation has been for the purpose of two main things: Approximating the posterior distribution and maximising model evidence. We will now see that this can be approached from another angle.

Consider the goal of approximating the posterior. For the sake of simplicity, and we will continue on that vein, but this time treating that single value as a distribution, specifically a delta distribution with all its mass concentrated at one point which we will denote as $\phi$ (therefore has a 0 probability at every other point). This fits well with what we did above, however it is important to remember that we could use another known distribution, such as the normal, to approximate the posterior, however for now we will stick with the delta distribution.

Let $q(v)$ = the distribution to approximate the posterior (in this example remember q(v) is the delta distribution)
We can take the KL-divergence between the two distributions to measure their similarity (with a KL-divergence of 0 meaning they are identical):

$KL(q(v), p(v|u)) = \int q(v)ln\displaystyle\frac{q(v)}{p(v|u)}dv$

substituting in the definition that $p(v|u) = \large\frac{p(u,v)}{p(u)}$ we have:

$KL(q(v), p(v|u)) = \int q(v)ln\displaystyle\frac{q(v)p(u)}{p(v,u)}dv$

$= \int q(v)ln\displaystyle\frac{q(v)}{p(v,u)}dv + \int q(v)dvlnp(u)$

$= \int q(v)ln\displaystyle\frac{q(v)}{p(v,u)}dv + lnp(u)$

Going from the second to the third line we note that the integral of $q(v)$ is 1 and $lnp(u)$ is not a function of $v$

Let $F = \int q(v)ln\displaystyle\frac{q(v)}{p(v,u)}dv$. This term is known as negative Free Energy and is related to the KL-divergence by:

$KL(q(v), p(v|u)) = -F + lnp(u)$

Now because $lnp(u)$ does not depend on $q$, the minimisation of the KL divergence between the two distributions can be achieved by maximising $F$. In this case, the value of $\phi$ that minimises the distance between $q(v)$ and $p(v|u)$ is the value of $\phi$ which maximises $F$. To go full circle, we will evaluate $F$ a little more:

$F = \int q(v)ln\displaystyle\frac{q(v)}{p(v,u)}dv$

$  = \int q(v)lnp(u,v)dv - \int q(v)lnp(v)dv$

Using the property of the delta function which states that for $\delta(x)$ with center $\phi$, for any function $h(x)$, the integral of $\delta(x)h(x)$ is equal to $h(\phi)$. Therefore:

$F = lnp(u,\phi) + C$.

We can change the second integral in the expression to a constant as it does not depend on $\phi$ and will cancel when we take the derivative.
This expression is exactly the same as $F$ calculated in the previous section, and it should be, as approximating a distribution with the delta distribution is the same as approximating it with a single point ($\phi$).
Also note that $lnp(u) = F + KL(q(v), p(v|u))$.
Since the KL-divergence is non-negative, the minimum the log model evidence can be is $F$. Therefore $F$ acts as a lower bound to model evidence, and maximising it via model parameters (as we did in the previous section) maximises the lower bound on model evidence.

Thus maximising $F$ serves to both maximise the approximate distribution, and optimise model parameters.

```python

```


# Initialization

```python
%matplotlib inline
%config InlineBackend.figure_format = 'svg'

import numpy as np
import scqubits as scq
```

# Transmon qubit

The transmon qubit and the Cooper pair box are described by the Hamiltonian
\begin{equation}
H_\text{CPB}=4E_\text{C}(\hat{n}-n_g)^2+\frac{1}{2}E_\text{J}\sum_n(|n\rangle\langle n+1|+\text{h.c.}),
\end{equation}
expressed in the charge basis. Here,  $E_C$ is the charging energy, $E_J$ the Josephson energy, and $n_g$ the offset charge. Internal representation of the Hamiltonian proceeds via the charge basis with charge-number cutoff specified by `ncut`, which must be chosen sufficiently large for convergence.

An instance of the transmon qubit is initialized as follows:

```python
tmon = scq.Transmon(
    EJ=30.02,
    EC=1.2,
    ng=0.3,
    ncut=31
)
```

Or, alternatively, we can use the graphical user interface (if the `ipywidgets` package is installed)

```python
tmon = scq.Transmon.create()
```

    HBox(children=(VBox(children=(HBox(children=(Label(value='EJ'), FloatText(value=15.0)), layout=Layout(justify_…

    Output()

## Calculating and plotting energy levels and eigenfunctions

The energy eigenvalues of the transmon Hamiltonian for the given set of model parameters are obtained by calling the `eigenvals()` method. The optional parameter `evals_count` specifies the sought number of eigenenergies.

```python
tmon.eigenvals(evals_count=12)
```

    array([-12.07703386,  -6.39445819,  -1.05664942,   3.89594699,
             8.34157217,  12.54032139,  14.69273601,  20.71739651,
            20.85721955,  30.96770401,  30.96906144,  43.86230239])

To plot eigenenergies as a function of one of the qubit parameters (`EJ`, `EC`, or `ng`), we generate an array of values for the desired parameter and call the method `plot_evals_vs_paramvals`:

```python
ng_list = np.linspace(-2, 2, 220)
tmon.plot_evals_vs_paramvals('ng', ng_list, evals_count=6, subtract_ground=False);
```

    HBox(children=(FloatProgress(value=0.0, description='Spectral data', max=220.0, style=ProgressStyle(descriptio…

```python
tmon.plot_n_wavefunction(esys=None, which=0, mode='real');
```

```python
tmon.plot_wavefunction(esys=None, which=0, mode='real');
```

```python
tmon.plot_phi_wavefunction(which=[0, 1, 4, 5], mode='abs_sqr');
```

## Calculating and visualizing matrix elements

Matrix elements can be calculated by referring to the `Transmon` operator methods in string form. For instance, `.n_operator` yields the charge operator:

```python
tmon.matrixelement_table('n_operator', evals_count=3)
```

    array([[ 0.2999574 , -0.90286728, -0.00126536],
           [-0.90286728,  0.30195231, -1.21191324],
           [-0.00126536, -1.21191324,  0.26241099]])

Visualizing matrix elements is accomplished by calling the `.plot_matrixelements` method:

```python
tmon.plot_matrixelements('n_operator', evals_count=10);
```

```python
tmon.plot_matrixelements('cos_phi_operator', evals_count=10, show3d=False, show_numbers=True);
```

```python
fig, ax = tmon.plot_matelem_vs_paramvals('n_operator', 'ng', ng_list, select_elems=4, filename='./data/test');
```

    HBox(children=(FloatProgress(value=0.0, description='Spectral data', max=220.0, style=ProgressStyle(descriptio…

    HBox(children=(FloatProgress(value=0.0, max=220.0), HTML(value='')))

## Tunable Transmon

An important modification of the transmon qubit is the replacement of the Josephson junction by a SQUID loop of two Josephson junctions. A flux threaded through this loop can then be used to change the effective Josephson energy of the circuit and thus make the transmon tunable. The resulting Hamiltonian is
\begin{equation}
H_\text{CPB}=4E_\text{C}(\hat{n}-n_g)^2+\frac{1}{2}E_\text{J,eff}(\Phi_\text{ext})\sum_n(|n\rangle\langle n+1|+\text{h.c.}),
\end{equation}
expressed in the charge basis. Here,  parameters are as above except for the effective Josephson energy $E_\text{J,eff}(\Phi_\text{ext}) = E_{\text{J,max}} \sqrt{\cos^2(\pi\Phi_\text{ext}/\Phi_0)+ d^2 \sin^2 (\pi\Phi_\text{ext}/\Phi_0)}$, where $E_\text{J,max} = E_\text{J1} + E_\text{J2}$ is the maximum Josephson energy, and $d=(E_\text{J1}-E_\text{J2})/(E_\text{J1}+E_\text{J2})$ is the relative junction asymmetry.

An instance of a tunable transmon qubit is obtained like this:

```python
tune_tmon = scq.TunableTransmon(
    EJmax=50.0,
    EC=0.5,
    d=0.01,
    flux=0.0,
    ng=0.0,
    ncut=30
)
```

Alternatively, the `.create` method can again be used if `ipywidgets` is available:

```python
tune_tmon = scq.TunableTransmon.create()
```

    HBox(children=(VBox(children=(HBox(children=(Label(value='EJmax'), FloatText(value=30.0)), layout=Layout(justi…

    Output()

```python
flux_list = np.linspace(-1.1, 1.1, 220)
tune_tmon.plot_evals_vs_paramvals('flux', flux_list, subtract_ground=True);
```

    HBox(children=(FloatProgress(value=0.0, description='Spectral data', max=220.0, style=ProgressStyle(descriptio…

```python

```